ANOMALY DETECTION WITH FRAME-GROUP ATTENTION IN SURVEILLANCE VIDEOS

Abstract: The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds, such as running, bicycling, throwing from a height. The algorithm forms continuous video frames into a frame group and uses the frame-group feature extractor to obtain the spatio-temporal information. The implicit vector based attention mechanism will work on the extracted frame-group features to highlight the important features. We use fully connected layers to transform the space and reduce the computation. Finally, the group-pooling maps the processed frame-group features to the abnormal scores. The network input is flexible to cope with the form of video streams, and the network output is the abnormal score. The designed compound loss function will help the model improve the classification performance. This paper arranges several commonly used anomaly detection datasets and tests the algorithms on the integrated dataset. The experimental results show that the proposed algorithm has significant advantages in many objective metrics compared with other anomaly detection algorithms.

Reviews and author responses:

This paper proposes a novel neural network that maps short chunks of eight video frames to a final probability score (0-1) of abnormality. The network is trained and evaluated in a supervised fashion, using the Avenue, UMN and UCSD data (all three datasets capture semantic anomalies e.g. walking in opposite directions of pedestrians, etc.). The network architecture comprises of various architectural concepts such as CNN, multi-level features, ConvLSTM, attention & transformer, etc. Basically the idea is to design a spatiotemporal feature from the chunk of video frames in an embedding where the features are easily being discriminated to 0/1 (normality/abnormality). Anomaly detection is important for various applications in surveillance. I like the structure of the paper and the related work. However, there are significant and weak points in this paper the authors need to address, which hinders me to accept the paper.

Pros:
- the idea to consider the whole network in two steps: the spatiotemporal feature of short video and then the relaxed mapping to [0,1] as a decision function
- building on existing anomaly datasets

Cons:
- the whole network is not sufficiently described in the paper so that the reader can understand the design choices in all details. For example, an LSTM is used on top of the spatial feature maps and it is not clear to me why a simple unrolled but full spatiotemporal convolutional neural network was not used. the chunk size is rather low with eight. I would have expected a careful analysis in the empirical evaluation, but the authors did not provide such.

- the same is true with other important design patterns. the authors perform simple tests in section 4.2 but the reader is not able to understand these design choices. For me the empirical methodology is not sufficient enough to clearly state the empirical contributions the paper tries to claim.

- there is reasonable literature in anomaly detection using unsupervised learning. E.g. perhaps starting with Johnson, Neil and David C. Hogg. “Learning the distribution of object trajectories for event recognition.” Image Vis. Comput. 14 (1996): 609-615. I miss the comparison between this work (and the related work) to work on unsupervised learning. When the reader thinks about anomalies one would first think about a potential pdf of "normality" and to detect anomaly such a pdf needs to be approximated from lots of unlabelled data. The authors should give a comment in the rebuttal why this task of learning 0/1 anomalies on visual data should not be considered as unsupervised learning task. Overall, I find the paper not elaborated enough in terms of the empirical methodology to bring objective insight into the design choices of the network based on the used data. Ablation studies are very much needed, otherwise insight by empirical evaluation is marginal. 

The paper presents an algorithm to detect abnormal events in video sequences. The proposed algorithm decomposes a video sequence into groups of consecutive frames ("frame groups"), and uses ConvLSTM to extract features from frame groups. A group level attention module is applied to focus on some most relevant portions of the extracted features, before feeding to fully connected layers for final anomaly score prediction. The algorithm is evaluated on a curated dataset accumulated from three popular benchmarks.     
 The proposed method looks like a routine application of well established approaches for video anomaly detection. It is like a compilation of off-shelf methods: frame groups are standard in video modeling literature, attention mechanism has been exploited for some time too (e.g., Cluster Attention Contrast for Video Anomaly Detection in ACM Multimedia 2020), etc. I have a very hard time finding any novelty in the reported method. 

On the other hand, the definition of anomaly in crowded scenes is kind of subjective. In the most popular setting, an anomaly is defined to be an event that deviates from the reference normalcy events in the training set, with the nature of anomaly inferred implicitly by the target data, as opposed to some manually specified events like “Abnormal behavior is defined as rapid movements in slow moving crowds, such as cycling, running, throwing from a height, etc.” This seems to significantly limit the applicability of the proposed method. How does the proposed method work in other types of anomaly events? A more thorough evaluation is expected for this question.

The paper also fails to compare to many state-of-the-art methods on the curated dataset. To name a few: 
- Abnormal Event Detection in Videos using Generative Adversarial Nets, ICIP 2017
- Learning Normal Dynamics in Videos with Meta Prototype Network, CVPR 2021
- Few-shot Scene-adaptive Anomaly Detection, ECCV 2020
 A routine practice of existing video anomaly detection solutions without novelty and convincing empirical justification. 


The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds. The group-feature extractor is adopted. The proposed algorithm is tested on the integrated dataset. The motivation of the paper is not explained clearly in the introduction. No descriptions of the figures are provided for Fig.2 and Fig.3 in the title below the images. All the figures are not vector illustration. The structure of Fig.2 are hard to understand, since In_t(0;Spa;Tem) are not exist in Fig.2. Moreover, the paper is hard to understand.
In the experiments, no latest methods are compared, the newest compared method is from 2018. There are too few comparing methods. It's quited confusing that it integrated the other three datasets to form a new dataset. Maybe the author has to explain in the introduction why it is different from the traditional anomaly detection task. It's  a clear reject.

This paper proposes a method for video anomaly detection. It basically trains a binary classifier to recognize anomaly vs normal.  This paper is clearly below the threshold for ICLR.

First of all, the novelty of the paper is extremely small. Although the title of the paper says "anomaly detection", the method in the paper actually has nothing to do with anomaly detection. It is just a standard binary classification model. There is nothing in the method specifically designed to take into account of the specifics of the anomaly detection problem.

This paper also completely changed the definition of the "anomaly detection". Most recent work in video-based anomaly detection assumes that you only have normal videos during training, but you may encounter anomaly during testing. This is how standard anomaly detection datasets (e.g. Avenue, UCSD) are constructed. However, this paper completely changed the problem definition. In the setting considered in this paper, you actually have both normal and abnormal samples during training. In this case, the problem is much easier than other standard anomaly detection papers (where you only have normal videos during training). The setting in this paper is nothing but a standard binary classification problem. 

The proposed method is a combination of fairly standard techniques (e.g. spatial-temporal features, attention, etc). I do not see any technical innovation.

Since this paper considers a different setting from standard anomaly detection.  As a consequence, this paper also completely re-did the standard datasets. This makes it impossible to compare directly with any previous published work. The only comparison with previous works is Table 3. However, since this paper uses a non-standaed dataset setup, I have no idea how the numbers for other methods in table 3 are obtained.

Also the setting of this paper is similar to [Sultani CVPR'2018]. Then why cannot you directly compare with [Sultani CVPR'2018] on the large-scale surveillance dataset proposed in [Sultani CVPR'2018]? Why do you need to create your own dataset by merging Avenue, UMN, UCSD datasets? Also note that [Sultani CVPR'2018] is already 3 years old and already has a lot of follow-up work. The paper should compare with more recent methods.
 This paper is lacking in terms of novelty. It also did not follow standard dataset setup and experiment protocol, so the experiment is fairly weak.

Meta-review: This paper handles anomaly detection in surveillance videos. The authors proposed to use a frame-group attention method for handling this task. However, all the reviewers have concerns about the novelty, clarity, and experimental evaluations of this work. Moreover, no rebuttal is provided by the authors.

Decision: Reject