# An Efficient Anomaly Detection System for Crowded Scenes Using Variational Autoencoders
## 3 Method
### Convolutional VAE Architecture and Feature Extraction  
The network architecture of our convolution based variational auto-encoder (VAE) [18] is shown in Figure 2. In our work, a frame sequence is represented by a set of regional appearance and motion feature vectors, which are extracted densely from the corresponding receptive fields by the following convolution based VAE. Taking both appearance and motion patterns into account, a series of frames is used as the input. Specifically, suppose we have \(T\) frames in the training dataset and all the frames are free of abnormal events, the pixel-wise average of frame \(I_{t-1}\) and previous frame \(I_{t-2}\) denoted by \(I_{t-1}^{(-1)}\) and the same as the frame \(I_{t+1}\), with the next frame \(I_{t+2}\) denoted by \(I_{t+1}^{(+1)}\), are used,  
\[I_{t-1}^{(-1)} =\frac{I_{t-2}+I_{t-1}}{2} \tag{1}\] \[I_{t+1}^{(+1)} =\frac{I_{t+1}+I_{t+2}}{2} \tag{2}\]  
where \(I_{t}\) is the \(t\)th frame in the video. Thus, the sequence \(\mathcal{I}_{t}=\{I_{t-1}^{(-1)},\;I_{t},\;I_{t+1}^{(+1)}\}\) is used as as a multichannel image to detect anomalies in frame \(I_{t}\) and the sequence set is \(\mathcal{I}=\{\mathcal{I}_{t}|t=3,4,\ldots,T-2\}\). In the feature description step, our convolution based VAE has a deep fully convolutional architecture that contains an encoder network and a decoder network. The encoder can be divided  
Figure 2: The network architecture of the proposed convolution-based VAE. Three blocks with different convolution kernels marked green, blue, and red constitute the encoder. The mean map \(\mathbf{\mu}_{t}\) output by the encoder is defined as the feature matrix that consists of the feature vector \(\mathbf{f}_{t}\) from each position. into three blocks according to the size of the convolution kernel. The first block consists of three convolutional layers with the same kernel size \(7\times 7\) and the same stride \(1\times 1\), and a max pooling layer with a \(2\times 2\) kernel and \(2\times 2\) stride. The second block has two convolutional layers with kernel size \(3\times 3\) and stride \(1\times 1\). The third block contains three convolutional layers with the same kernel size \(1\times 1\) and stride \(1\times 1\). Then according to the VAE, which assumes the posterior distribution of each latent variable takes on an approximate Gaussian form, the encoder outputs a mean map \(\mathbf{\mu}\) and a standard deviation map \(\mathbf{\sigma}\), which are the parameters for the posterior distributions of the latent variables. The reparameterization trick is used to generate samples from the posterior distributions of the latent variables, \(\mathbf{\mu}+\mathbf{\sigma}\otimes\mathbf{\epsilon}\), where \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) and \(\otimes\) as an element-wise product. Next, the samples are pushed into the following decoder network. Compare with the encoder, the followed decoder has the same reversed structure with the deconvolutional layers instead of the convolutional layers. Specifically, for the \(t\)th sequence \(\mathcal{I}_{t}\) with the resolution \(h_{0}\times w_{0}\), our encoder network outputs a mean tensor \(\mathbf{\mu}_{t}\) and a standard deviation tensor \(\mathbf{\sigma}_{t}\), where \(\mathbf{\mu}_{t},\mathbf{\sigma}_{t}\in\mathbb{R}^{h_{t}\times w_{t}\times c_{l}}\), \(h\), \(w\) are the height and width of the tensor map, \(c\) is the number of channels, and \(l\) is the layer of the network. Following the work in VAE, the loss function is mainly comprised of two parts. Here we give the equations used for the calculation as follows.