dataesr/scientific-paragraphs-categorization huggingface dataset:
1138 In supervised classification, labeled datasets are split into 1139 training, validation, and test datasets. Datasets can be 1140 separated chronologically or randomly. However, temporal 1141 partitioning better matches the real world scenario, when 1142 a classifier is trained on existing data and then tested on 1143 new data. Despite the cost of labeling and the difficulty of 1144 detecting new devices not included in training, supervised 1145 classification techniques are commonly employed in IoT 1146 device classification literature (84% of reviewed papers) due 1147 to their high accuracy and ease of implementation.
Regressing the distribution of different sub-populations from a batch of images with learning algorithms is not a trivial task, as models tend to make errors that are unequally distributed across the different sub-populations. Obviously, the baseline is forming a histogram from the batch after having characterized each image independently. However, we show that this approach can be strongly improved by making the model aware of the ultimate task thanks to a density loss for both sub-populations related to classes (on three public datasets of image classification) and subpopulations related to size (on two public datasets of object detection in image). For example, class distribution was improved two-fold on the EUROSAT dataset and size distribution was improved by 10% on the PASCAL VOC dataset with both RESNET and VGG backbones.
Introduction Image classification performance has risen thanks to deep learning approaches  [1, 2]. Yet, data driven algorithms are still far from being perfect, in particular when testing data are sampled from a different distribution than the training data (also called dataset drift), e. g. , out-of-distribution detection  [3] , transfer learning  [4] , domain adaptation  [5] , and adversarial attacks  [6]. In addition, those deep learning models tend to be unfair toward some sub-populations of the distribution, as highlighted in numerous papers in the literature (see  [7]  for a review). Of course, a detector with no error is fair. Thus, in critical use cases such as autonomous driving, fairness is not really an issue because the models should have an almost perfect detection rate for all sub-populations of pedestrians regardless (independently from color of skin, age, size, gait, clothes, presence of medical walker, etc.
The offered pipelines are presented by Figures  1  and 2  for class and size sub-populations, respectively. Importantly, it is rather obvious that making an algorithm more aware of the ultimate task will improve the ultimate performance. However, here we obtain significant improvement of the density estimation. For example, on the EUROSAT dataset, we halve the class distribution error of the baseline using our density loss, and on the PASCAL VOC dataset, we improve the quality of the size density estimation by 10%. The amplitude of this improvement shows that this problem should receive more attention from the community, as all methods are clearly not equivalent. To summarize, the contributions of this paper are: • The consideration of batch distribution estimation for class and size sub-populations.
, (x K , s K )) = t → ∑ k, f (x k )=1 f (x k )G(t -s k ) ∑ k, f (x k )=1 f (x k )G(τ -s k )dτ (4) where f (x k ) ∈ [0, 1] is extracted from logit of f (we consider the mean of softmax and softplus in our experiments). Interestingly, it is known that small objects (e. g. , included in 32 × 32 bounding boxes) are harder to detect than larger ones. For example, in most detection datasets, small objects are counted separately. Yet, the exact purpose of the pipeline is to improve the predicted distribution by asking the network to perform as well for small objects as for large ones. Potentially, this may be achieved by simply worsening detection for large objects. Yet, it may still improve the predicted distribution. In addition, in the case of size distribution estimation, the intermediary algorithm may process each image after resizing, i.
Experiments The two previous sections have introduced five methods (mainly five different losses) which aim to extract the proportion of sub-populations in a batch. This section presents experiments on multiple datasets which evaluate the behavior of all five algorithms on different datasets/backbones. Indeed, the offered losses are independent from the underlying backbones; thus, we report results for VGG13, VGG16  [25] , RESNET34, and RESNET50  [26]  backbones. Yet, the results are consistent across these backbones. These experiments mainly employ moderately sized datasets. However, we think that these are the use cases where our algorithm would be relevant. Indeed, in very large datasets, there are probably methods of filtering the predicted distribution using information from the entire test set.
Datasets We chose to rely on standard classification datasets for evaluating the capacity to predict class proportion in batches of data. We relied on the CIFAR  [27] , SVHN  [28] , and EUROSAT  [29]  datasets. All three datasets have native pytorch data loaders. For CIFAR and SVHN, we relied on the standard training/testing split. For EUROSAT, we used the pytorch loader, which is deterministic. Then, images with IDs that are multiples of three were used for testing while the others were used for training (i. e. , 66% training and 33% testing).
Results The results are presented in Tables 1-3 for the CIFAR, SVNH, and EUROSAT datasets. First, as pointed in the introduction, simple regression offers very poor performances: processing the batch directly without taking into account the intermediary image structure seems to lead to the problem of an excessively high number of dimensions. Baseline+fairness is the best on CIFAR but has a poor performance on SVHN and is only average on the EUROSAT dataset. Baseline+Dice is inversely the best on EUROSAT but has a poor performance on CIFAR. Baseline+rejection improves slightly upon the baseline on average. This last result shows that discarding ambiguous samples is not necessarily a good idea when regressing a proportion. Indeed, wrong but overconfident samples are here more harmful than ambiguous ones.
Influence of Batch Size Batch size is a critical parameter of all these experiments. Indeed, if batch size is one, then the problem collapses into simple classification. Inversely, if the batch size becomes very large, then all proportions tend to 1  |Y| where |Y| is the number of classes, making all the methods converge toward the same value. Thus, the offered algorithm is mostly relevant for moderate batch sizes. However, moderate batch sizes are the ones found in deep learning as excessively small batches lead to time wastage when moving data from the CPU to the GPU and excessively large batches cannot be loaded in the GPU. However, in order to see the sensitivity to batch size, we considered batch sizes of 128, 256, and 512 on EUROSAT for the baseline and our density loss. Those results (which are not averaged over five trials, contrary to the other results from this paper) are summarized in Table  4.
We can see that results are consistent across these values, stressing that the experiments presented in this paper are probably valid for batch sizes ranging from 128 to 512 and not only for a batch size of 256. We chose to rely on detection datasets for evaluating the capacity to predict distribution of sizes of objects of interest. We relied on the Munich Dataset for Vehicle Detection (MDVD)  [31]  and PASCAL VOC  [32]. We tackled those detection datasets with the box proposal + rejection paradigm  [33]. We acknowledge that the box proposal is clearly a deprecated paradigm for object detection, but it seems acceptable, as we only want to test the impact of different methods of estimating sub-population distributions. For MDVD, we use 4K0G0060, 4K0G0070, 4K0G0080, 4K0G0090, and 4K0G0100 as the training set and 4K0G0010, 4K0G0020, 4K0G0030, 4K0G0040, and 4K0G0051 as the testing set.
Implementation Details Specifically, we relied on selective search (opencv implementation  [34] ) for both MDVD and PASCAL VOC. Then, the classifier has only to decide if the box seems to be an object of interest. This classification is performed after resizing the image to 32 × 32. Yet, the true size (before resizing) is kept to regress the size distributions of accepted objects (as presented in Section 3. 2. 2) and to compare this predicted distribution to the true one. Indeed, the ultimate task is not to correctly accept/reject boxes but to ensure that the predicted size distribution matches the true one. Importantly, as PASCAL is clearly larger than the other datasets, we performed the PASCAL experiment only once (instead using an average of five experiments). Furthermore, simple regression is not presented, as it performs even worse than in class sub-population experiments.
Results The results are presented in Tables  5  and 6  for MDVD and PASCAL datasets. The offered density loss is consistently better than the three baselines (i. e. , baseline with or without fairness or rejection) on all backbones/datasets (except for RESNET34 on MDVD where baseline+fairness is 0. 0015 higher than the offered loss). The consistent improvement brought by density loss can be explained by the fact that baselines are even less aware of the ultimate task than in the class sub-population case. Indeed, in both cases, a perfect classification still leads to a perfect distribution estimation. However, in the class case, the classifier outputs directly matches distributions, while in the size case, the distribution is extracted by combining classifier outputs and sizes that are unknown by the classifier.
Conclusions This paper tackles the problem of estimating the distribution of different sub-populations within a batch of images. This problem has received almost no direct attention from the community, despite the fact that one can see the link with fairness, selective classification, multiple instance learning, and unstructured data classification (such as point cloud classification). The main contribution of this paper is the demonstration, in five datasets, that baselines can be significantly improved by our approach based on the comparison of densities, allowing the intermediate classifier to be made aware of the ultimate task. The implementation of this loss (which allows an efficient propagation of the gradient within the comparison of densities) is carefully presented and the code is made available for the community.
In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators scanpaths, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content.
Understanding the mechanisms underlying the visual attention on movies can also be of help for computational models related to movie production, such as automated camera placement, automated editing or 3D animated scenes design. In this paper, we extend the work of  Breeden and Hanrahan (2017)  by proposing a new eye-tracking database on 20 different movie clips, of duration 2 to 7 minutes each. For each clip, we provide cinematographic features annotations drawn from  Wu et al. (2017) , such as the camera movement and angle, the framing size, and the temporal location of cuts and edits. Alongside with a comprehensive analysis of the collected data, we expose several strong correlations between high-level cinematographic features and gaze patterns, that can be easily used to improve visual attention modeling.
(2014) , and managed to outperform most models of the state of the art at the time. Later on, several deep learning models using transfer-learning were proposed, where features learned on large-scale classification datasets were used, like DeepFix  (Kruthiventi et al. , 2017) , SALICON  (Huang et al. , 2015) , DeepNet  (Pan et al. , 2016)  or Deep Gaze II  (Kümmerer et al. , 2017). More recently, the emergence of large-scale fixation datasets allowed for end-to-end approaches, in order to learn features more specific to visual saliency. These new models, like SalGan  (Pan et al. , 2017) , SAM-VGG and SAM-Resnet  (Cornia et al. , 2018) , or MSI-Net  (Kroner et al. , 2020) , exhibit great predictive behaviors, and constitute a very strong baseline for modeling human visual attention. Dynamic models followed the same path towards deep learning, with models such as DeepVS  (Jiang et al.
Movies eye-tracking datasets In the field of visual attention modeling for videos, a majority of the large-scale databases used to train various models contain mostly non-cinematographic stimuli. As we show in Section 6, this leads to consistent errors when saliency models are used on film sequences. Moreover, most studies involving visual attention and movies use their own collected eye-tracking data, as the experimental setups are often very specific to the characteristics studied. However, there exists a few available eye-tracking databases on movie scenes, that can be general enough for modeling purposes. Hollywood-2 (Mathe and Sminchisescu, 2015) includes 1707 movie clips, from 69 Hollywood movies, as well as fixation data on those clips from 19 observers. Observers were split into three groups, each with a different task (3 observers free-viewing, 12 observers with an action recognition task, and 4 observers with a context recognition task).
Each group being relatively small, the common way to use this data for visual attention modeling is by merging those groups, thus introducing potential biases. The large scale of this dataset (around 20 hours of video) is well fit for training deep saliency models, however few conclusions regarding gaze patterns on movies can be drawn from the data itself, since it mainly focuses on task-driven viewing mode, and that each clip is only around 15 seconds long. SAVAM  (Gitman et al. , 2014)  includes 41 highdefinition videos, 28 of which are movie sequences (or use movie-like realisation, like commercials for instance). Eye fixations are recorded from 50 observers, in a free viewing situation. As for Hollywood-2, the each clip is quite short, only 20 seconds on average. Breeden and Hanrahan (2017)  proposed eye-tracking data from 21 observers, on 15 clips from 13 films, for a total of 38 minutes of content.
Each clip is between 1 and 4 minutes. Alongside this data, they also provide high-level feature annotations, such as the camera movements in shots, the temporal location and types of edits, the presence or absence of faces on screen, and whether or not the characters are speaking. However, the main limitations of this dataset are the relatively low precision of the eye-tracking device used, and the duration of the total content of the base itself. It follows that the saliency modeling community, as well as cinematographic studies, would greatly benefit from an extension of Breeden and Hanrahan's work, i. e. a relatively large-scale eye-tracking database on movies sequences, including a large diversity of editing styles, genres and epochs, alongside with high-level features annotations, related to different film-making parameters.
Films and clips selection In  Wu et al. (2017) , the authors proposed a language called Film Editing Patterns (FEP) to annotate the production and edition style of a film sequence. Alongside this formalization of cinematographic rules, they present an open database of annotations on several film sequences, for pattern analysis purposes. In order to simplify the annotation process of our dataset, we decided to use the same clips. We selected 20 clips, extracted from 17 different movies. The movies span different times (from 1966 to 2012) and genres, and are from different directors and editors, in order to eliminate bias coming from individual style. Table  1  gives an overview of the selected clips. The sequences were selected as they were the most memorable or famous sequences from each movie, based on scenes that users uploaded to YouTube, indicating popularity and interest to the general public.
A narrator comments a part of the sequence. Various camera movements, angles and shot sizes. • The Shining: Dialogue scene between two characters. Very low frequency of edits, and abundant presence of the color red in the scene. • The Help (1): Flashback scene, dialogue between two characters. • The Help (2): Flashback scene, in between a dialogue scene between two characters. A lot of faces and colored clothing. The length of the clips varies from 1 minute 30 to 7minutes. This length is voluntarily higher than in the other datasets presented in Section 2. 3, in order to allow the observer to feel immersed in the sequence, and thus exhibiting more natural gaze patterns. In total, the dataset contains.
Faces As explained by  Cerf et al. (2008) , the presence of faces in images is a very important high-level information to take into account when studying visual attention. We then provide bounding boxes delimiting each face on each frame. Recent state of the art face detection models show that deep learning models extract this information very well. It is then probable that deep visual attention models are also great at extracting faces features, making this hand-crafted feature redundant. However, we include it as it permits an easier analysis of the editing style: for instance, continuity edits will often display faces on the same area of the image, while shot/reverse shots often display faces on opposite sides of the image. 4 Eye-tracking data collection.
Participants and experimental conduct We have collected eye-tracking data from 24 volunteers (11 female and 13 male), aged 19 to 56 (average 28. 8). Participants were split into two groups, each group watching half of the videos. A few observers were part of both groups, and viewed the whole dataset. In total, we acquired exploitable eye fixation data for 14 participants for each video. Details of the sequences viewed by each group can be found in supplementary material. Viewers were ask to fill an explicit consent form, and to perform a pre-test form. The objective of the pre-test form was to detect any kind of visual impairment that could interfere with the conduct of the experiment (colourblindess, or strabism, for instance), as well as ensuring that they could understand English language well enough, as sequences were extracted from the English version of the movies.
After the break, before viewing the five last clips, participants were asked to validate the previous calibration, and to recalibrate if necessary. After recording the data for all participants, we used the following cleaning procedure. First, we ensured that every participant had a gaze sampling rate of more than 90% (i. e. more than 90% of the sampled points were considered as valid). We then kept only points that were flagged as fixations, eliminating tracking errors due to blinks or other factors, as well as points recorded during saccades. This choice was motivated by the relatively low frequency rate of the eye-tracker, making the analysis of saccadic data impossible. Then, we discarded all points that fell in the letterboxing or outside the screen. Finally, we used the position of the remaining raw points to construct binary fixation maps : for each frame, we create an image the same size of the frame, where we give the value 1 to each pixel where a fixation point was flagged during the time the frame was on screen (i.
Exploring the effects of film making patterns on gaze In this section, we explore several characteristics throughout our database, and analyze underlying relationships between editing patterns and eye fixations patterns. In the following, we will often refer to fixation maps and saliency maps. For each frame, the fixation map is the binary matrix where each pixel value is 1 if a fixation occurred at the pixel location during the frame, and 0 if not, as described previously. Saliency maps are obtained by convolving the fixation maps with a 2-D Gaussian kernel, which variance is set to one degree of visual angle (in our case, one degree of visual angle equals to roughly 45 pixels), in order to approximate the size of the fovea.
Editing-induced visual biases Studying the average of the saliency maps usually reveals strong attentional biases. For instance, on static images,  Tatler (2007)  showed that humans tend to look at the center of the frame. That center bias is also commonly used as a lower baseline for saliency models. In order to avoid recording this bias too much, we did not take into account for our analysis the first 10 frames of each clip, as people tend to look in the middle of the screen before each stimulus. This center bias is also strong on video stimuli: for instance, Fig. 4  (a) and (b) shows the average saliency map on our dataset and on the DHF1K dataset  (Wang et al. , 2019)  respectively. However, the latter is composed of Youtube videos, with a great diversity in the content, and no cinematographic scenes, which might cause a different viewing bias.
Fig. 4  (a) shows a peak density slightly above the center of the frame, which would indicate that filmmakers use a different composition rule. Fig. 4 (c ) shows a centered Gaussian map, often used as a baseline for centered bias. Correlation between the average saliency map on our dataset and this centered Gaussian is 0. 81, whereas the correlation between the average map on DHF1K and the centered Gaussian is 0. 84, which highlights this position discrepancy between the two average saliency maps. This is consistent with the findings of  Breeden and Hanrahan (2017) , and is most likely due to the rule of thirds  (Brown, 2016)  stating that in cinematography, important elements of the scene should be placed on thirds lines, i. e. lines dividing the frame in thirds horizontally and vertically.
Inter-observer visual congruency Inter-observer congruency (IOC) is a measure of the dispersion of gaze patterns between several observers watching the same stimulus. In other words, it measures how well gaze patterns from a subset of the observers is predictive of the whole set of observers. Thus, it has been used in saliency modeling as an upper baseline. This characteristic is very similar to attentional synchrony  (Smith and Henderson, 2008) , and many methods have been proposed to measure it. For instance,  Goldstein et al. (2007)  measure how well gaze points are fitted by a single ellipsoid, while  Mital et al. (2011)  use Gaussian mixture models, associating low cluster covariance to high attentional synchrony. In their work,  Breeden and Hanrahan (2017)  use the area of the convex hull of the fixation points, for each frame of their dataset.
We then can build a saliency map using the fixation points of the N -1 observers, by convolving the fixation map with a 2-D Gaussian kernel, and use any saliency metric to compare it to the fixation map (or saliency map) of the left out observer. The process is then iterated and averaged over all observers. To compare the saliency map of the N -1 observers to the left-out one, we used the Normalized Scanpath Saliency metric (NSS); more details about can be found in Le  Meur and Baccino (2013). A high value of this score will mean that people tend to look in the same region, and a low value will indicate a higher dispersion in the fixations. The main drawback of this way of computing IOC, especially for large-scale datasets, is its computational cost, as the process is iterated over every observers, and every n-frame window of the dataset.