# An Efficient Anomaly Detection System for Crowded Scenes Using Variational Autoencoders
## 3 Method
### Anomaly Detection and Localization  
In the training phase, all the frames \(\{I_{t}|t=1,2,\ldots,T\}\) and the frame series \(\{\mathcal{I}_{t}|t=3,4,\ldots,T-2\}\) are free of abnormal events and thus the features extraced from them are considered as normal features. As mentioned above, the appearance and motion patterns in each receptive field of a video sequence \(\mathcal{I}_{t}\) are represented using a feature vector \(\mathbf{f}_{t}(i,j)=[f_{t}(i,j,1),f_{t}(i,j,2),\ldots,f_{t}(i,j,c_{l})]^{\top}\) from the encoder of our fully convolutional network, where \((i,j)\) is corresponding location on the feature map and \(i\in[1,h_{t}],\ j\in[1,w_{l}]\).  
To model all the normal features that form the training frame series and to check whether the upcoming features are abnormal or not, Gaussians are constructed to fit the normal features. Different from other works that used only one model to fit all the feature vectors, we construct different Gaussian models to fit the normal feature vectors extracted from each receptive field. Besides this, considering the residual correlation among the elements of a feature vector, the multivariate Gaussian is adopted to improve the accuracy. Specifically, for each location \((i,j)\) of the feature tensor, an exclusive multivariate Gaussian model \(G_{ij}\) with \(c_{l}\) variables is fitted to all normal feature vectors \(\{\mathbf{f}_{t}(i,j)|t=3,4,\ldots,T-2\}\) extracted from all frame series \(\{\mathcal{I}_{t}|t=3,4,\ldots,T-2\}\), where \(T\) is the number of the normal frames:  
\[\mathbf{f}_{t}(i,j)\sim\mathcal{N}(\mathbf{\mu}_{ij}^{G},\Sigma_{ij}^{G}) \tag{8}\]  
where \(\mathbf{\mu}_{ij}^{G}\in\mathbb{R}^{c_{l}}\) is the mean vector, \(\Sigma_{ij}^{G}\in\mathbb{R}^{c_{l}\times c_{l}}\) is the covariance matrix and \(c_{l}\) is the dimension of \(\mathbf{f}_{t}(i,j)\). Therefore, we define these Gaussian models \(\{G_{ij}|i=1,2,\ldots,h;j=1,2,\ldots,w\}\) as our reference models for normal appearance and motion patterns.  
In the anomaly detection phase, input sequences are pushed into the encoder network first. Then \(h_{l}\times w_{l}\) feature vectors are extracted by the encoder network. These feature vectors are varified by the corresponding Gaussian model \(G_{ij}\). Observations of low probability under these Gaussian model are declared anomalies. In our work, the log-likelihood of a feature vector \(f_{t}^{test}(i,j)\) under the Gaussian model \(G_{ij}\) is used to measure the anomaly so that the abnormality map at location \((i,j)\) is the negative log-likelihood of the feature vector \(f_{t}^{test}(i,j)\):  
\[\mathcal{A}(i,j)=-\text{log}(G_{ij}(f_{t}^{test}(i,j)|\mathbf{\mu}_{ij}^{G}, \Sigma_{ij}^{G})) \tag{9}\]  
In addition, the location of the anomalies can be mapped from the feature maps layer to the original frame. According to the convolution and pooling processes of CNN, all the feature vectors \(f_{t}^{test}\) are extracted from the corresponding receptive fields that overlap each other in sequence \(\mathcal{I}_{t}^{test}\) for considering the \(t\)th frame. As the kernel of each convolution and pooling processes has the fixed size, therefore, the abnormality of each receptive field in the original \(t\)th frame \(I_{t}^{test}\) can be backward mappedby the abnormality at each location in the abnormality map. For the regions where the receptive fields overlap in the original frame, an averaging operation is used to calculate the final anomaly score.