A Comprehensive Review on Vision-based Violence Detection in Surveillance Videos

Recent advancements in intelligent surveillance systems for video analysis have been a topic of great interest in the research community due to the vast number of applications to monitor humans(cid:1745) activities. The growing demand for these systems aims towards automatic violence detection (VD) systems enhancing and comforting human lives through artificial neural networks (ANN) and machine intelligence. Extremely overcrowded regions such as subways, public streets, banks, and the industries need such automatic VD system to ensure safety and security in the smart city. For this purpose, researchers have published extensive VD literature in the form of surveys, proposals, and extensive reviews. Existing VD surveys are limited to a single domain of study, i.e., coverage of VD for non-surveillance or for person-to-person data only. To deeply examine and contribute to the VD arena, we survey and analyze the VD literature into a single platform that highlights the working flow of VD in terms of machine learning strategies, neural networks (NNs)-based patterns analysis, limitations in existing VD articles, and their source details. Further, we investigate VD in terms of surveillance datasets and VD applications and debate on the challenges faced by researchers using these datasets. We comprehensively discuss the evaluation strategies and metrics for VD methods. Finally, we emphasize the recommendations in future research guidelines of VD that aid this arena with respect to trending research endeavors. VD and VD, datasets, researchers, metrics for VD, and future research directions. Discussion of both deep learning and traditional methods for VD with working flow, visual presentation, and challenges. Index of impact analysis for VD and future research direction with recommendation is provided. of a large collection of local features, where each vector is invariant to any scale or changes, while SURF is based on principles of the SIFT descriptor. The SURF descriptor [98] is more suitable for processing video frames for VD [106] due to its efficiency. Another feature descriptor known as ORB is an alternative to the SURF and SIFT algorithms. It is the combination of FAST [101] and BRISK [99] and has been proven to be invariant in rotation and scale, robust to affine transformation and noise. Furthermore, the BoW [107] represents the VD frames as a single vector that reduces the sensitivity and variability of noise, thus enabling the classifier for training. It builds the vocabulary of the visual words obtained from the frames, i.e., chooses the most representative features for classification purposes. Estimating optical flow calculates the velocities of objects and enables the classifier for training the video sequences. The key point descriptors only determine the area of interest, while optical flow uses multiple video frames and finds the movement difference among them. has firstly used IFV as an extension, whereby the local features and positions of spatiotemporal information are used to represent complete video. They applied the sliding window technique to detect violence. Furthermore, the other classifiers have distinct classification procedures that are outside the scope of this survey. Learning techniques include various classifiers, such as SVM [144], Naïve Bayes [120], KNN [181] and decision trees [115]. features that are based on Lagrangian-directed fields. They used the background motion information, appearance, and long-term motion. BoW was applied in a late fusion manner, ensuring the spatial and temporal scale. They demonstrated that the temporal scale obtained through Lagrangian is crucial for VD and showed how they correlated to the spatial scale of event characteristics. [189]. LSTM networks are the RNN extended form that have the ability to find the spatiotemporal patterns in the violence activity data. In this way, they show the loss of the initial dependencies present in the sequence. The LSTM includes mainly three kinds of gates, namely the input gate, output gate, and forget gate. These gates assist in learning the violence sequences. These gates are adjusted by a sigmoid function that learns the opening and closing in training. less focused. In this way, the installation of surveillance cameras and automating them through computer vision and AI techniques will further play a vital role in society. After deep analysis, several guidelines in the form of reviews and surveys have been found to guide the VD community. For this purpose, we comprehensively surveyed the existing methods from 2011 to date with a focus on NNs.