dataesr/scientific-paragraphs-categorization huggingface dataset:
• KITTI: The videos of the KITTI Dataset 14  were obtained from a moving VM station wagon that was recording for 6 h at 10-100 Hz, in the city of Karlsruhe, Germany  (Geiger et al. , 2013). Several sensors were involved for the recording such as a color and a grayscale stereo camera, and a Velodyne 3D laser scanner. Color and grayscale images are stored under the 8-bit PNG files format. The dataset is used, inter alia, for object detection and tracking. Eight classes of objects can be tracked and detected in challenging conditions, because the objects may be static or dynamic, and the camera is constantly moving. These eight classes are: 'Car', '  Van', 'Truck', 'Pedestrian', 'Person (sitting)', 'Cyclist', 'Tram' and 'Misc'. The provided annotations are in the form of 3D bounding boxes tracklets.
The total size of the provided data is 180 GB. • Toulouse Campus Surveillance: The Toulouse Campus Surveillance Dataset 15  can be used, among other things, for object detection/tracking and audio event recognition  (Malon et al. , 2018). The dataset results from a multi-camera setup, and thereby, can be useful for assessing Multi-source fusion methods. Some cameras are fixed, others are mobile. The setup gave birth to 50 videos fairly broken down into two scenarios. Each video is depicted by three resolution qualities: 1920 × 1080, 960 × 540, 640 × 360. The Ground Truth annotations for detection and tracking are only provided for the first scenario. • The PathTrack MOT: Assisted by the crowd-sourcing permitted by Amazon Mechanical Turk, where approximately 80 AMT workers were involved,  Manen et al.
(2017)  propose the large-scale PathTrack MOT dataset. 16 This dataset was released following the creation of the PathTrack annotator. It contains the annotation of 720 sequences depicting pedestrian movements from videosurveillance scenes that were captured by either static or moving cameras. The sequences include 16. 287 annotated trajectories. • Optical Flow Dataset: Also named TUB CrowdFlow Dataset, the Optical Flow Dataset 17  was created by  Schröder et al. (2018). The authors graphically generated the dataset by using Unreal Engine. They simulate crowd motion in five different situations. Moreover, the crowd is captured by a static and a dynamic camera. The dataset contains 10 sequences whose lengths range from 300 to 450 frames. The sequences are characterized by a frame-rate of 25 Hz and a HD resolution.
The scenes contain between 371 and 1451 individuals. The authors also verified that the results that state-of-the-art methods obtain on this dataset can be transferred to real-world generated datasets such as UCF crowd tracking  (Ali & Shah, 2008). Within the dataset, the ground truth data provided is for optical flow, and dense and sparse trajectories of individuals. • UCF crowd tracking: The UCF crowd tracking dataset 18  was created by  Ali and Shah (2008). The dataset is composed of three sequences called Marathon-1, Marathon-2, and Marathon-3. The length of these sequences ranges from 333 to 492 frames. To test their methods, the authors annotated the trajectories of respectively 199 individuals, 120 individuals, and 50 individuals on the three sequences.
Pedestrian detection • INRIA Person: The Inria Person Dataset 19  is used for pedestrian detection. The dataset contains bounding-boxes annotations of original images, resized positive images of pedestrians in 64 × 128 format with original negative images. The dataset contains 1805 images of people in challenging situations: different poses and orientations, and within a wide range of backgrounds  (Dalal & Triggs, 2005). • Caltech Pedestrian: The Caltech Dataset 20  is commonly used for Pedestrian Detection  (Dollar et al. , 2012). It can also be used for occlusion handling. The dataset contains 10 h of 640 × 480 pixels video recorded from a mono-camera mounted on a vehicle moving in the street. These 10 h video are also depicted by 250. 000 frames. The provided Ground Truth is for 2300 different pedestrians, in a form of 350.
000 bounding boxes. • COCO Common Object in Context:  Lin et al. (2014)  propose the COCO dataset. 21 It is mainly used for object and keypoint detection, and image segmentation. In addition to pedestrians, 79 other objects can be detected. The dataset contains 330 K images and more than 200 K are labeled. (Chan et al. , 2008). The anomalies are linked to abnormal elements appearing in the video clip such as the circulation of non-pedestrians. However, anomalies pertain also to pedestrians adopting abnormal motion patterns. The ground truth indicate the occurrence of an anomaly on frame via a binary flag. The bounding-box localization of the abnormal element is also provided. • Agoraset: Created by  Allain et al. (2012) , this dataset can be used for pedestrian tracking, abnormal event analysis and density estimation.
The simulations have been generated using a model based on the Lagrangian forces proposed by  Helbing, Farkas, and Vicsek (2000). The dataset consists of 7 scenarios that can be broken down into several video clips by changing the crowd state (soft motion or panic) and/or the illumination rendering (shading or no shading). (Hassner et al. , 2012). The dataset, that consists of 246 videos, is mainly collected from Youtube. The video frames were resized to 320 × 240 pixels. The average video duration is of 3. 60 s. To assess their dataset, they compared their technique to two other existing ones: an interest-point driven method  (Laptev, 2005) , and a framebased descriptor, the LTP (Local Trinary Patterns)  (Yeffet & Wolf, 2009). Their technique is based on the use of Optical Flow and Support Vector Machines.
They also developed for the need of their method the ViF (Violent Flows) descriptor. The code to compute the violence descriptors is publicly available. a dataset dedicated to group detection and behavior analysis, and more precisely for the study of free-standing conversational groups (FCGs). Within the dataset, two scenarios are depicted in an indoor setup: a poster presentation and a cocktail party. In each of them, 18 participants are involved and their behaviors were not scripted. The recorded scenes last for 60 min. The challenges presented by this dataset are the low resolution, the lighting variations, and occlusions. The provided annotations are about each individual's personality as well as their position, head and body orientation, the group F-formation information. The individual's personality annotation are scores related to Extraversion, Agreeableness, Emotional Stability, and Creativity.
Conclusion on the existing sources of data As you can observe from the range of datasets presented in this section, most of the publicly available datasets can be used for individualscene action detection/recognition, pedestrian detection/tracking and sometimes group detection/tracking, crowd counting and density estimation, crowded scene classification, few cases of anomaly detection, and very few cases of group activity recognition. However, there is a lack of datasets reproducing crowd motions, in roads or avenues of populated cities. Besides, to the best of our knowledge, there is not any 40  dataset that can be used for massive upstream crowd behavior analysis and motion forecasting that can be utilized to train and test a deep Neural Network. The access to the video data, provided by live video-surveillance presented in Section 5.
Annotators As you may have observed from the precedent section, data scarcity, and more precisely the lack of relevant labeled data, is one of the major problems of crowd analysis. One solution to this problem is the use of annotators assisted with massive crowd-sourcing. In this section, we provide a small list of annotators that can be used for crowd analysis-related tasks. Table  6  summarizes their characteristics. Russell, Torralba, Murphy, and Freeman (2008)  propose LabelME, a publicly available web-based tool intended for image annotation. Its purpose is to create ground truthed image datasets intended for object detection and recognition tasks.
This reduces the workload of the human annotator. For instance, the authors compared their use of both of Fluid Annotation and LabelMe, and concluded that annotation time is reduced by a factor of 3× when using their own tool. Despite being three times quicker than LabelMe, and more handy than VIA, Fluid Annotation does not seem to perform video annotation contrary to VIA. Bartoli et al. (2017)  propose PACE an open-source collaborative tool used for the annotation of crowded scenes, that is an improvement of WATSS  (Bartoli, Seidenari et al. , 2015) , and is targeted to indoor multicamera setups and group behavior understanding. Developed using HTML5 and Javascript, it has two back-ends: a PHP-based that can mainly be summarized to a relational database, and a Python-based REST server that is used for computer vision tasks.
Tracking annotators Manen  et al. (2017)  propose PathTrack, a realtime trajectory annotator, and aspire to enrich MOTChallenge-like datasets. The annotations are produced by users watching a video-surveillance scene and following targets with a cursor. The annotations are then turned into frame-by-frame bounding boxes. The use of PathTrack halved the misclassification rate on a person matching method trained on Multiple Object Tracking Challenge 2015  (Leal-Taixé et al. , 2015) , and improved NOMT performance reducing the identity switches by 18% and the fragments by 5%. As aforementioned, the crowd-sourced use of Pathrack allowed the creation of a public dataset, named the PathTrack MOT dataset. This annotator does not seem to be publicly available. Mettes et al. (2016)  propose Spot On, a spatio-temporal action annotator in video clips.
We saw recent studies pertaining to pedestrian and group detection, as well as on the branches and several sub-branches of crowd analysis. We enumerated the sources of video/image data we came across, and due to the paucity of datasets, we found it relevant to talk about annotators. These latter are somehow neglected by the research community in some subtopics of crowd analysis which are: crowd statistics, action recognition, and crowd behavior analysis. This review allowed us to find out that group analysis-related tasks are not widely explored using Deep Learning methods, despite their widespread use in crowd analysis. Moreover, upstream massive crowd analysis for motion tracking and/or anomaly detection is not widely explored by the Deep Learning literature, due to the non-existence of relevant datasets.
's and Marsden, McGuinness, Little, and O'Connor (2016a)'s in crowd counting on the UCF Crowd 50, slightly by Shao et al. (2015)'s in violence identification on the WWW Crowd test set, and slightly in anomaly detection by Li, Mahadevan, and Vasconcelos (2014)'s and Marsden, McGuinness, Little, and O'Connor (2016b)'s on the UMN dataset.
Benfold and Reid (2011)cv. ucf. edu/data/ucf-cc-50/. 10 FLICKR: https://www. flickr. com/. 11cript hook v. http://www. dev-c. com/gtav/scripthookv/. The Town Center Dataset, proposed byBenfold and Reid (2011), is a video recorded from a video-surveillance mono-camera recording the pedestrians walking in a city center. The dataset is mainly used for tracking. At each frame, there is at around 16 pedestrians. Th  video quality is good, 1920 × 1080 pixels for 25 FPS.
Karpathy et al. (2014)opose HMDB-51. 24Thedatasetconsists of 6766 annotated video clips gleaned from various sources such as Youtube or Movies. In terms of resolution, the videos share all the same 240 pixels height. However, the width of each video clip is re-sized so as to maintain its proper aspect ratio. The frame-rate of all the clips is of 30 FPS. The videos are categorized into 51 action classes. Meta-information within the dataset details for each video clip a range of information such as camera viewpoint, occlusion, occurrence of camera motion (that concerns 2∕3's of the database), video quality (which isKay et al. (2017)propose the Kinetics dataset, 25 a dataset that is similar to UCF-101 and HMDB-51, and that is mainly used for video classification. It consists of 400 classes of different actions performed by a wide spectrum of different persons.
Thre are from 400 to 1150 video clips for each class. Each video clip is taken from Youtube and lasts for approximately 10 s. These actions cover human-human and humanobject interactions. The challenges presented by this dataset are illumination changes, background clutter, camera motion and vibrations, shadows, etc. The annotation process relied on the Amazon Mechanical Turk (MTurk). 26Theclips are not exhaustively annotated. Soe of them may incorporate more than one action, but the authors made sure that each clip is labeled with the name of, at least, one of occurring actions. •Sprts-1m:Karpathyetal. (2014)propose the Sports-1 m dataset. 27Thisdataset contains 1 million Youtube videos that lasts for 5 min and 36 s on average. As he name of the dataset suggests it, the videos are solely about sports-related activities.
Blunsden and Fisher (2010) 32 can be used for pedestrian and group behavior analysis in a mildly crowded scene, but mainly for anomaly detection. All the scenes were scripted(ED, 2003). , scared, sad, and neutral. The dataset is exhaustively annotated. The scenes are recorded from a camera fixed at a height oriented to the ground to capture people walking. The video resolution is of 554 × 235 pixels and consists of 30 frames per second (FPS). The density of the scenes varies from an intermediate to a high concentration of individuals. •UCFCrime:TheUCFCrimedataset34was created by Sultani et al. (2018). The dataset consists of 1900 long and untrimmed surveillance videos, that totalized 128 h of recording, and that were acquired from Youtube and Liveleak 35 via text queries expressed in different languages.
UCF Crime can be used for two tasks: General anomaly detection as we can find normal and abnormal behaviors; and anomalous activities classification, because the abnormal behaviors can be classified into 13 anomalous activities which are: Abuse, Arrest, Arson, Assault, Road Accident, Burglary, Explosion, Fighting, Robbery, Shooting, Stealing, Shoplifting, and Vandalism. When the authors evaluated methods on this dataset, they could fix the frame rate to 30 FPS and the resolution to 240 × 320 pixels. •CCTV-Fights:TheCCTV-Fightsdataset36wascreated byPerez et al. (2019). The dataset consists of 1000 temporally annotated videos of real-world fights that required more than 17 h of camera recording. These videos are made up of 280 CCTV videos, whose duration varies between 5 s to 12 min, in addition to 720 videos that come from Non-CCTV sources, whose duration varies between 3 s to 7 min.
The resolutions of the videos are diverse. Bazzani et al. (2012)propose The Friends Meet Dataset 37 for their Decentralized Particle Filter technique used for joint pedestrian-group tracking. The Friends Meet is suitable for the development of bottom-up group tracking and detection approaches. The dataset contains 53 sequences broken down into 16286 frames, involving 3 to 16 individuals per frame. The dataset contains two scenarios, the first one depicts individual visitors watching artworks, and the second one depicts groups of visitors watching artworks. Provided annotations are in the form of Bounding Boxes containing the visitors. If a person is occluded, its visible part is surrounded by a second Bounding Box. The Ground Truth furnishes also a unique identifier for each Group and each Pedestrian in all the frames.
Seven detectors, pre-trained either on the Caltech pedestrian or the INRIA pedestrian datasets, were assessed on the dataset and improved their miss rate at 10 -1 false positive per image (FPPI) for Pedestrian and Group detection scenarios. •Behave:Blunsdenand Fisher (2010)observed that most of the available ground truthed datasets are used for target detection, tracking, and individual behavior analysis. Consequently, they proposed the Behave Dataset. 40 The dataset is used for pedestrian detection, bottom-up group behavior recognition, and abnormal behavior analysis. It consists of 4 WMV video clips, depicted by 76800 frames. Qualitatively, the resolution is at 640 × 480 at a frame-rate of 25 FPS. There are 125 individuals involved in the actions within the scenes. The detected pedestrians are surrounded by rectangular bounding boxes, which results in 83545 bounding boxes.
The annotations are yielded using the Viper-GT 41 tool. The group behavioral interactions are categorized in 10 classes. The annotation policy in this dataset considers unique pedestrian as the smallest group unit. A group may include many pedestrians that are involved in the same activities and showing a certain proximity. A group activity is labeled by an activity name that happens between two persons. The challenges presented by this dataset are light changes and the recurrent occlusions. A Hidden Markov Model (HMM) classifier had been tested on the dataset. The authors varied several times the size of the window centered on a current frame. Each frame is represented by a feature vector. For a window size of 100, the reached performance is at around 93. 67%. • SALSA: Alameda-Pineda et al.
''?DL'' column: DL stands for Deep Learning, this column means if the paper used Deep Learning approaches (DL) or not (¬DL). Reference Date Research axis Used datasets ?DL Source code Ge, Collins, and Ruback (2012) 2012 Pedestrian & Group detection Self-made ¬DL Unavailable Li, Wu, and Zhang (2016) 2016 Pedestrian detection INRIA Person DL Unavailable Tian, Luo, Wang, and Tang (2015) 2015 Pedestrian detection Caltech, KITTI DL Unavailable Angelova, Krizhevsky, Vanhoucke, Ogale, and Ferguson (2015) 2015 Pedestrian detection Caltech DL Available Shao, Dong, and Zhao (2018) 2018 Group detection Student003, GVEII, MPT-20X100 ¬DL Unavailable Wang et al. (2018) 2018 Group detection CUHK Crowd ¬DL Unavailable Voon, Mustapha, Affendey, and Khalid (2019) 2019 Group detection CUHK Crowd ¬DL Unavailable.
Web Dataset 2009 Vid MonoCam 20 Vid's, 8 for abnormal events and 12 for normal ones MOT Challenge 2014 (Updated 2016) Vid MonoCam 14 Vid's Multi Task Crowd 2017 Img MonoCam 100 Img's, 50/50 violent/non-violent crowds, 0 to +150 ped's/img Agoraset (Simulated) 2012 Vid MonoCam 7 scenarios subjected to different simulations CUHK Crowd (Updated 2014) Vid MonoCam 474 videos on 215 crowded scenes KITTI 2013 Vid MonoCam 180 GB data. 6 h recording Toulouse Campus Surveillance 2018 Vid MultiCam 50 vid's of 2 scenarios. GT for 1 scenario. UCF-101 2012 Vid MonoCam 13320 vid's, 27 h recording depicting 101 classes of actions HMDB-51 2011 Vid MonoCam 6766 annotated vid's, 51 classes of actions PathTrack MOT 2017 Vid MonoCam 720 sequences, 16,287 GT INRIA Person 2005 Img MonoCam 1805 Img's Kinetics 2017 Vid Several 400 actions, 400-1150 vid's for each action, ∼10s duration each Caltech Pedestrian 2009 (Updated 2018) Vid MonoCam 10 h recording, appearance of ∼2300 ped's, 350,000 GT BBxs Violent Flows (ViF) 2012 Vid MonoCam 246 vid's, ∼3.
6s duration each Sports-1m 2014 Vid MonoCam 1 m vid's, ∼5min 36s duration, 487 action classes, 1K-3K vid's/class Caviar 2003 (Updated 2004) Vid Several 28 vid's from MonoCam, 26 vid's from MultiCam Behave 2010 Vid MultiCam 4 vid's, 76800 frames, ∼125 ped's, 83545 GT The Friends Meet (FM) 2012 Vid MonoCam 53 vid's, 16286 frames, 3-16 ped's involved MuseumVisitors 2015 Vid MultiCam 2 scenarios SALSA 2016 Vid MonoCam 2 scenarios, 18 ped's, vid's lasting for 60 min COCO 2015 Img MonoCam 330K img's, 80 classes of detectable objects, 200K GT Crowd-11 2017 Vid MonoCam 6,272 clips, 11 classes, all clips labeled Motion Emotion 2016 Vid MonoCam 31 vids; 44. 000 frames; classes: 5 motions, 6 emotions; GT provided UCF Crime 2018 Vid MonoCam 1900 vids, 13 classes of crimes, GT provided CCTV Fights 2019 Vid MonoCam 1000 vids, the 2 classes normal/abnormal, GT provided Crowd-Flow 2018 Vid MonoCam 10 vids, OF and trajectories GT provided UCF crowd tracking 2008 Vid MonoCam 3 vids, trajectories GT provided GTA5 Crowd Counting (GCC) 2019 Images MonoCam 15,212 images, GT provided, very diverse scenes presentation of datasets (Part 1).
(2009) Web Dataset Not mentioned Abnormal behavior detection/recognition Public Mehran et al. (2009) MOT Challenge Not mentioned Group/Pedestrian detection and tracking Public Milan, Leal-Taixé, Reid, Roth, and Schindler (2016) Multi Task Crowd Not mentioned Abnormal behavior recognition/Crowd Statistics Private Marsden et al. (2017) Agoraset (Simulated) Not mentioned Group/Pedestrian detection/tracking/behavior analysis Public Allain et al. (2012) CUHK Crowd Not mentioned Group/anomaly detection Unavailable Shao et al. (2014) KITTI 8-bit PNG frames Object detection/tracking Public Geiger, Lenz, Stiller, and Urtasun (2013) Toulouse Campus Surveillance 1920 × 1080 px Object detection/tracking Public Malon et al. (2018) UCF-101 320 × 240 px, 25 FPS Action detection and recognition Public Soomro, Zamir, and Shah (2012) HMDB-51 240 px height, 30 FPS Action detection and recognition Public Kuehne, Jhuang, Garrote, Poggio, and Serre (2011) PathTrack MOT Not mentioned Pedestrian detection and tracking Public Manen, Gygli, Dai, and Van Gool (2017) INRIA Person 64 × 128 px Pedestrian detection, re-identification Public Dalal and Triggs (2005) Kinetics Not mentioned Action detection and recognition Public Kay et al.
5 Crowd Counting (GCC) dataset: Proposed by Benfold and Reid (2011)cv. ucf. edu/data/ucf-cc-50/. 10 FLICKR: https://www. flickr. com/. 11cript hook v. http://www. dev-c. com/gtav/scripthookv/. The Town Center Dataset, proposed byBenfold and Reid (2011), is a video recorded from a video-surveillance mono-camera recording the pedestrians walking in a city center. The dataset is mainly used for tracking. At each frame, there is at around 16 pedestrians. Th  video quality is good, 1920 × 1080 pixels for 25 FPS. 5. 2. 2. Tracking and motion analysis • Town Center: • i-Lids for AVSS 2007: The i-Lids dataset 12 contains two types of videos: one recording a train station (Task 1), and an other recording road vehicle traffic (Task 2). The sequences we are interested in are those of the train station.
They can be used for Pedestrian detection, tracking, Abnormal behavior detection (in this case: abandoning a luggage). The resolution of the videos are of 720 × 576 pixels for 25 Hz. • MOTChallenge Dataset: The 2016 MOTChallenge Dataset 13 is an extension of the precedent version of 2015 proposed by Leal-Taixé et al. (2015) 8 • UCF CROWD 50: Used for Crowd Counting, UCF Crowd 50 9 dataset consists of 50 images of densely crowded scenes. The pictures that were collected from FLICKR 10 contain a population ranging from 94 to 4543 individuals. • GTA Wang et al. (2019), this synthetic dataset is created using the Script Hook V 11 C++ library applied on the Grand Theft Auto 5 (GTA 5) Rockstar game. The scenes come from 100 different indoor and outdoor locations in Los Santos (fictional city inspired by Los Angeles).
The people in the scenes are diverse and generated from 265 different person models. However, the scenes in GTA 5 are limited to 256 individuals. The GCC dataset has 15,212 images. The resolution of each image is 1080 × 1920. The scenes of the dataset have got 7 different weather conditions: clear, clouds, rain, foggy, thunder, overcast and extra sunny. They are captured at any time of the day. 7 For the sake of completeness; we invite you to explore these three websites: • http://homepages. inf. ed. ac. uk/rbf/CVonline/Imagedbase. htm. • http://riemenschneider. hayko. at/vision/dataset/index. php. • https://www. di. ens. fr/~miech/datasetviz/. 8 WWW Crowd Dataset: https://computervisiononline. com/dataset/ 1105138602. 9.