dataesr/scientific-paragraphs-categorization huggingface dataset:
They propose an approach based on the extraction of optical flow from a video clip, using the Farnebäck technique  (Farnebäck, 2003) , on which they apply the Thermal Diffusion Process (TDP)  (Wang et al. , 2014)  to make it more coherent. After that, the moving particles are restricted to individuals on which they apply a modified version of the Social Force Model (M-SFM) to understand the interactions between individuals. At the end of this process, they end up with a continuous dynamic system that describes the motion flow field from which they extract the first order and the second order derivatives to identify the crowd behavior. Their method was validated on the benchmark dataset  (Solmaz, Moore, & Shah, 2012)  and the UCD dataset  (Ullah & Conci, 2012) , evaluated using the average F1 score metric, and compared to  Solmaz et al.
(2012) 's method where it has difficulties with the lane class. Ullah et al. (2019)  work on crowd video classification. They propose a two-stream convolutional architecture to achieve this. A stream extracts spatial features from an RGB frame and a second stream extracts temporal features from a trajectory-based descriptors  (Wang, Kläser, Schmid, & Liu, 2011)  computed on a motion flow field of a certain number of consecutive frames. The streams are initialized with weights pretrained on the ImageNet dataset. Their method is evaluated on the CUHK crowd dataset and compared to reference methods on terms of average accuracy and confusion matrices: tensor learning classification (TLC)  (Zhang, Liu, & Jiang, 2018) , spatio-temporal classification (STC)  (Li, Liang, & Jin, 2016) , and energy-based features  (Zhang, Zhang, Hu, Guo, & Yu, 2018).
After that, they compute the Finite Time Lyapunov Exponent (FLTE) field from the spatial gradient tensor of the Flow Map. This operation yields the Lagrangian Coherent Structures (LCS) which allows to divide the crowd into several groups. The approach is tested on clips taken from Getty-Images, Photo-Search, Google Videos and Inside Mecca a National Geographic documentary. However, the approach was neither tested using proper metrics nor compared to other methods. The code source of this approach is publicly available in github. 3 n another work,  Ali and Shah (2008)  propose a probabilistic approach to track individuals in a highly dense crowded scene based on the scene structure force model. The constraints of this force model are determined by three floor fields inspired from the evacuation dynamics domain.
The three fields are (1) the Static Floor Field (SFF): which represents the locations the crowd is heading to, (2) Dynamic Floor Field (DFF): which represents the interacting forces between the individual and the surrounding crowd. Here, the optical flow, the use of particles advection and the social force model is involved, (3) Boundary Floor Field (BFF): which represents the limits of the crowded environment that the crowd cannot infringe and that are found through the computation of the Finite Time Lyapunov Exponent (FLTE) field. The authors divide the scene image into a grid of cells where each cell represents a particle or a pedestrian. On each particle they calculate the three constraints: DFF, SFF, BFF. These constraints help to predict the future locations of the pedestrians. Ali and Shah undertook their experiments on the three sequences of the Marathon dataset.
It achieves a good accuracy score, making it close to state-of-the-art trackers. However, SORT considers long-term occlusions as an unimportant issue, making it failing to perform in person re-identification which worsens the score of identity switches. In the same line,  Wojke et al. (2017)  developed deepSORT, an improved version of SORT that includes a deep pre-trained association metric augmented with appearance and motion information. This makes it facing better long-term occlusions while maintaining a similar speed. The association metric is pre-learned on the MARS person reidentification dataset  (Zheng et al. , 2016)  using a Convolutional Neural Network. Specifically, the key changes occur at the assignment stage. More precisely, the SORT's formulation of the assignment problem is kept: the Kalman filter predicts states.
The density of each flow is analyzed to detect local overcrowdedness. Their method is trained and tested on UCF crowd dataset, Collective Motion, 4  and Violent Flows. However, their results were compared in terms of Jaccard similarity, F-score, and Mean Absolute Error (MAE) to non-recent methods, which makes the proposed method perform well. Li et al. (2019)  propose a top-bottom clustering algorithm based on pedestrians trajectories to detect groups. First, they extract the pedestrians trajectories. After that, they leverage distance and velocity difference between two trajectories in order to measure their similarity. Secondly, they use a combination between density peak clustering and a greedy algorithm to perform top-level coarse-grained clustering that relies on the mean distance between trajectories based on the Euclidean distance.
They evaluate their method in terms of Area Under Curve (AUC) and Receiver Operating Characteristic (ROC) curve on the UCF crowd dataset, CUHK Crowd dataset, and a combination of UCF and CUHK crowded scenes. They compared to other baseline methods, where they showed that it performs well on all the classes, except on the bottleneck class. Alahi et al. (2016)  propose Social-LSTM a Recurrent Neural Network which contains Long Short-Term Memory units, to learn pedestrians movements and jointly predict their future locations by taking into account the social context of each pedestrian's trajectory. From a video clip of a crowded scene, the authors obtain the positions of the pedestrians on all the frames. After that, they use a separate LSTM model to learn each pedestrian's trajectory. The LSTMs are connected to each other via a social pooling layer, so that spatially close LSTMs share information about each other and inputs it to the next step.
They train their model following a leave-one-out validation strategy and evaluate it on UCY  (Lerner, Chrysanthou, & Lischinski, 2007)  and ETH  (Pellegrini, Ess, Schindler, & Van Gool, 2009)  datasets. Social-LSTM is tested in terms of Average Displacement Error (ADE), Final Displacement Error (FDE), Average non-linear displacement error (ANLDE). They compare their method to the Kalman Filter, a Collision avoidance based on the Social force model  (Yamaguchi, Berg, Ortiz, & Berg, 2011) , the social force model, the iterative Gaussian process (IGP)  (Peter, Richard, Murray, & Krause, 2013) , a vanilla LSTM model without the social pooling layer, and a simpler version of their model that contains only the occupancy maps (O-LSTM) which captures the positions of the neighbors at time t without taking into account the previous positions.
The methods learn the trajectories for 3, 2 s and try to predict the 4, 8 following seconds. The overall results show that Social-LSTM outperforms other methods on the test sequences of both of the datasets. However, it is sometimes outperformed by IGP because it knows the ground truth final destination of each pedestrian contrary to the other methods, and O-LSTM because it performs well on less crowded scenes. Bartoli, Lisanti, Ballan, and Del Bimbo (2018)  propose a contextaware Recurrent Neural Network that uses Long Short-Term Memory units to learn and predict pedestrians trajectories. They extend the Social-LSTM and the O-LSTM models proposed by  Alahi et al. (2016)  by incorporating to them a context-aware pooling that takes into account the human-human interactions as well as the static objects that are located in the vicinity of pedestrians.
This approach needs a prior knowledge of the positions of the static objects. They evaluate their method, in terms of Average Displacement Error (ADE), on the UCY dataset  (Lerner et al. , 2007)  and the MuseumVisits dataset  (Bartoli et al. , 2018). As in  Alahi et al. (2016) , the methods learn the trajectories for 3, 2 s and try to predict the 4, 8 following seconds. The authors found out that context-aware O-LSTM performs well on the MuseumVisits because the persons move in group and are only interested in artworks, but context-aware Social-LSTM performs well in the UCY sequences because there the persons move alone and the prior knowledge of entry and exit points are fed to the model. A lot of research has been done for motion tracking  (Bewley et al. , 2016; Wojke et al. , 2017)  and motion analysis  (Lamba & Nain, 2019; Li et al.
The Collective Transition approach is a clustering algorithm that is used for group detection. The authors start by extracting the tracklets of each pedestrian using the KLT tracker. The initial clusters of tracklets are found by using the Coherent Filtering method  (Zhou et al. , 2012). After that, anchor tracklets are elected to represent the centers of the clusters (groups) or individuals and tracklets are grouped using the Expectation-Maximization (EM) algorithm. Intra-group and inter-group properties which are stability, collectiveness, uniformity, and conflict, are verified through the use of graph K-Nearest Neighbors algorithm. Experiments had been undertaken on the self-made CUHK Crowd dataset. The authors' method was compared to a mixture of dynamic textures (DTM)  (Chan & Vasconcelos, 2008) , hierarchical clustering (HC)  (Ge et al.
They evaluated their method on the collective activity dataset  (Choi, Shahid, & Savarese, 2009) , and compared it to six other state-of-the-art approaches. The author's approach that relies on GRU units outperforms all the others on terms of accuracy. Shu et al. (2017)  propose a Confidence-Energy Recurrent Neural Network (CERN) using Long Short-Term Memory (LSTM) units, to recognize individual human actions, interactions and group activities. They developed two variants of their algorithm: CERN-1 and CERN-2. They tested these two variants on the Collective Activity  (Choi et al. , 2009)  and Volleyball  (Ibrahim, Muralidharan, Deng, Vahdat, & Mori, 2016)  datasets. Compared to other methods, CERN-2 outperforms all the methods in terms of accuracy. Zitouni et al. (2020)  adopt a mesoscopic approach to crowd behavior analysis, where they perceive the crowd as a compound of two types of elements individuals and groups.
In this context, groups are divided into simple groups which demonstrate a homogeneous behaviors and compound groups which demonstrate heterogeneous behaviors. They propose a pipeline where pedestrians, heads, and groups are detected and then tracked. Afterwards, a Gaussian Mixture Model of Dynamic Textures detection technique (GMM-of-DT)  (Zitouni, Bhaskar, & Al-Mualla, 2016)  based on a Kalman filter is utilized to categorize crowd behaviors into 4 classes: individual, group, leader-follower, and social interaction. The method was validated on 6 sequences from the PETS dataset  (Ferryman & Shahrokni, 2009) , and tested on sequences from Parking Lot  (Dehghan, Modiri Assari, & Shah, 2015)  and Town Center  (Benfold & Reid, 2011). The evaluation were undertaken using the F-measure. Bisagno et al.
(2018)  propose Group LSTM, a method to detect groups and predict their trajectories. First, they use the coherent filtering approach  (Zhou et al. , 2012)  to cluster trajectories and form groups based on the individuals trajectories, and by taking into account their surroundings: pedestrians walking in the same direction belong to the same group. After that, they use an extended version of Social-LSTM  (Alahi et al. , 2016)  to predict the group trajectories. While Social-LSTM predicts solely pedestrians' trajectories by associating an LSTM for each of them, Bisagno et al. introduce a social pooling hidden layer to extend the interest for all the group of pedestrians, and then predict the trajectory of the group. They evaluate their method on UCY and ETH datasets, using the Average Displacement Error (ADE) and the Final Displacement Error (FDE).
Their method is compared to Social-LSTM  (Alahi et al. , 2016) , to its variant  (Gupta, Johnson, Fei-Fei, Savarese, & Alahi, 2018) , and to the Kalman Filter. In terms of ADE, they obtain the best results, but in terms of FDE, they sometimes fail to find the final position of a group, contrary to other methods. Research in group behavior analysis does not attract the attention of researchers as it does for motion tracking or action recognition in individual scenes. This may be due to the lack of datasets where group activities can be found. However, we can observe during the recent years a rising interest for this research axis. Recent papers  (Bisagno et al. , 2018; Zitouni et al. , 2020)  bypass data paucity and perform group behavior analysis on datasets that are not originally meant for this task such as ETH and UCY, or PETS dataset.
Anomaly detection Anomaly detection and forecasting can be done for any subtopic of crowd analysis. This research axis is getting more and more attention in crowd analysis because of its various applications in video surveillance and crowd monitoring  (Zhan et al. , 2008). Inspired by the effectiveness of the Mixture of Dynamic Textures (MDT) in video modeling and video clustering  (Chan & Vasconcelos, 2008) ,  Mahadevan et al. (2010)  propose an unsupervised framework based on MDT to model normalcy in crowded scenes. This model considers temporal and spatial outliers as anomalies. Temporal anomalies represent events that are unlikely to happen or that happen rarely, and spatial anomalies are detected using discriminant saliency. Metrics used to evaluate the model on the UCSD anomaly dataset are the ROC curve when the comparison between ground-truth and the detected anomaly is done at the frame level and the pixel level.
A Force Flow vector field is extracted, relying on the laws of the Social Force model which is based on the interactions that occur between the particles in the scene. From this vector field, a bag of words is constructed to represent the different behaviors of a crowd. The Latent Dirichlet Allocation (LDA), a Natural Language Processing technique, is trained to recognize normal behaviors. Finally, the Expectation-Maximization (EM) method is used to differentiate between normal and abnormal behaviors. Abnormal behavior is found in regions where high force flow occurs. The method is tested on the UMN and the Web datasets. The ROC curve metric is used to test the method. The authors show that the Social force-based approach outperforms methods based solely on Optical Flow. The major drawback of this approach is its inability to recognize new examples taken from a different camera angle or position than the camera angle and position of the examples the model was trained on.
Hassner et al. (2012)  propose violent flows descriptors (ViF) to detect violence in crowded scenes. These descriptors are extracted from the fluctuations in the magnitudes of optical flow vectors. The authors propose to use a Bag-of-Features representation of the scenes with the ViF descriptor and then to train Support Vector Machines (SVM) to classify the scenes in either violent or non-violent. For the classification task, the metrics that were used are the mean prediction accuracy (ACC) ±standard deviation (SD) and the area under the ROC curve (AUC). The authors propose a violence detection dataset on which they test their approach and compare it to other state-of-the-art features extraction techniques. Their descriptor is efficient in classifying violent and non-violent scenes. However, as shown by the authors, it is not as well performing on the Hockey Fights dataset where it is outperformed by other Space-Time Interest Points (STIP)-based descriptors  (Laptev, 2005).
The DSS outperforms a State-of-the-Art approach (Penmetsa, Minhuj,  Singh, & Omkar, 2014 ) by over 10% accuracy on the proposed Aerial Violent Individual (AVI) dataset. Following these results, it would be interesting to see the performance of this method on other publicly available datasets. Ravanbakhsh et al. (  2016 ) developed a measure-based unsupervised approach that detects local abnormality by combining motion information with appearance. They get motion and appearance information using a Convolutional Neural Network (CNN). Their approach consists in three steps: after feeding a Binary Fully Convolutional Network (BFCN) with input frames, they extract from them binary maps. They use these binary maps to compute a measure, inspired from the commotion measure  (Mousavi, Nabi, Kiani, Perina, & Murino, 2015) , which they called the Temporal CNN Pattern (TCP).
Mixed with Optical Flow, they yield refined motion segments. The Binary Fully Convolutional Network is composed of a Fully Convolutional Network and a Binary Quantization Layer (BQL). Although the weights of the FCN are obtained from a pre-trained AlexNet model  (Krizhevsky et al. , 2012) , the weights of the BQL are obtained from a hashing method. They tested their methods in terms of frame-level anomaly detection and pixel-level anomaly detection. When the former assesses the ability of the model to identify correctly an anomalous frame, the latter ensures that it localizes it accurately. The experiments were undertaken on the UMN SocialForce, and UCSD's Ped1 and Ped2 datasets. The evaluation metrics are the Receiver Operating Characteristic (ROC) curve and the Area Under Curve (AUC). Most of the time, the method performs well.
After that, they train a Self-Organizing Map (SOM) on the ABC's population, food storage and centroids to find particular events relying on behavior patterns similarity. ABC's population, stock and centroids represent respectively movement's spread, density and center. During the process, it is noteworthy to mention that ABC's population is subjected to the Darwinian natural selection method, and that the fitness function relies on the food stock. Using Optical Flow to extract motion layers reinforces the method's resilience to noise and light changes. Experiments were undertaken on the UMN SocialForce dataset. For decent values of bacteria population and neurons number for the SOM map, the approach outperforms other runs quite quickly, processing each frame for 0. 033 s on average. It outperforms  Mehran et al.
(2009) 's method AUC (Area Under Curve) score by 18%. It would have been interesting to observe the performance of the author's method compared to more recent approaches. As a future work, the authors propose to use  Mehran et al. (2009) 's Social Force Model instead of Optical Flow, and to extrapolate the improvements to the use of ABC to optimize spatio-temporal volumes. Singh et al. (2020)  propose Aggregation of Ensembles (AOE), an aggregation of four classifiers over sub-ensembles of three fine-tuned Convolutional Neural Nets (CNNs) on crowd datasets to detect anomalies in videos of crowded scenes using the majority vote. One classifier is applied for each sub-ensemble. A sub-ensemble of CNNs is made up of the following 3 pretrained models: AlexNet  (Krizhevsky et al. , 2012)  on CIFAR-10, GoogleNet  (Hu, Huang, Gao, Luo, & Duan, 2018) , VGGNet  (Simonyan & Zisserman, 2014) , both on ImageNet.
The CNNs were used as feature extractors that feed three variants of Support Vector Machines (SVM): a Linear SVM, a Quadratic SVM, a Cubic SVM ; and a Softmax classifier. The features are extracted from a batch of frames selected from a video. If more than 10% of the frames of the batch are classified as anomalous, the video is considered as anomalous. Models finetuning, AOE training and evaluation were undertaken on the following datasets: UCSD Ped 1, UCSD Ped 2, and the Avenue dataset. 5 asim and Bhatti (2019) propose a three-dimensional descriptor made up of three features, based on Optical Flow (OF) extracted from videos. Most of the time, 7 consecutive frames are selected from a video. The 3 features are: 1. sum of thresholded OF magnitude, 2. joint entropy of the OF magnitude of 2 consecutive frames.
The joint entropy helps to detect sudden changes related to a rapid dispersion (for example), 3. variance of a space-time cuboid obtained using the history of OF field magnitude. On top of this descriptor a Support Vector Machine (SVM) classifier is used to detect anomalies in videos. Qasim and Bhatti's purpose is to propose a good trade-off between accuracy and real time performance. Their method is evaluated in terms of accuracy on the UMN dataset. Hao et al. (2019)  propose an abnormal behavior detector based on a Gabor-filtered extracted spatio-temporal texture. On a raw video clip, a spatio-temporal volume (STV)  (Adelson & Bergen, 1985)  is constructed, from which a Spatio-temporal textures (STT) are extracted. STTs are vertical or horizontal slices of STVs along the time axis. Gabor filtering is applied on the STT for background subtraction and noise removal.
The comparison between TAMURA features and GLCM features, held on the UMN dataset, show that Hao et al. 's GLCM strategy is good for detecting panic situations, however TAMURA is good for describing congestion situations. Lin et al. (2019)  propose Social Multiple Instance Learning (Social MIL) framework coupled with a dual branch network to detect anomalies in crowded scenes. Their approach is made up of a Two-Stream Neural Network: 1. a spatio-temporal stream, and 2. an interactive dynamic stream. The first stream is a spatio-temporal branch, that is fed with RGB video clips. The video clip is converted to a video segment then fed through a 3D ConvNets (C3D) pre-trained model on Kinetics and UCF-101 for features extraction. The features are fed to a one dimensional dependency attention capturing module which output is inputted to a fully connected network.
The second stream is fed with social force maps. Social forces maps are obtained using the social force model  (Mehran et al. , 2009)  that describe the interactions that occur within a scene. Their method is trained using a MIL ranking loss function, and evaluated on the UCF Crime dataset  (Sultani et al. , 2018) , using the receiver operating characteristic (ROC) curve and area under the curve (AUC) metrics. During the evaluation process, it is compared to 4 other methods. Xie et al. (2019)  propose crowd abnormal behavior recognition algorithm based on the computation of optical flow and the use of the social force model. First, they compute optical flow from a video clip using Lucas-Kanade method. After that, optical flow is mapped to 2D geospatial space using the camera parameters. This operation helps them to extract particle points from a video clip.
Datasets We do not intend to be exhaustive about the existing datasets. 7 Most of the datasets we introduce, in this subsection, are frequently used for tracking and pedestrian detection such as KITTI and MotChallenge, action recognition such as UCF-101 and HMDB-51, solely pedestrian detection such as Inria Person and Caltech Pedestrian. The datasets that tend more to be used for crowd analysis-related tasks are rare to find. We listed the most relevant and popular ones. Tables  4  and 5  summarize the characteristics of these datasets.
Crowd statistics • Multi Task Crowd: This dataset was developed by  Marsden et al. (2017). They used it to train their ResnetCrowd architecture to perform three tasks: crowd counting, density estimation and violence detection. The dataset is composed of a set of images. These images are obtained from the WWW Crowd videos dataset. (Milan et al. , 2016). It consists of 14 sequences, containing crowded scenarios such as the famous PETS09-S2L1. The dataset presents some challenging conditions like a dynamic camera, illumination changes, and various viewpoints. The annotated objects are pedestrians, sitting persons, vehicles and occluding objects, etc. This dataset encourages the use of the CLEAR metrics  (Kasturi et al. , 2009) , in addition to other metrics such as those evoked in  Huang, Nevatia, and Li (2009) , to evaluate methods used for object detection and tracking.