# Masked Convolutions Within Skip Connections for Video Anomaly Detection
## 3 Methodology
### MaskedConv2D  
To facilitate a more tangible understanding of our methodology, we initially apply the concept of masked convolutional layers to a 2D image dataset as a proof of concept, before diving into the complexities of video anomaly detection with MaskedConv3D layers. Understanding the idea of a masked convolutional layer in 2D is easier prior to delving into the 3D variant. This approach underscores the versatility and adaptability of our technique. A traditional Conv2D kernel, for instance a \(7\times 7\) one, is made up of 49 weights, denoted by \(\{w_{i,j}|i,j\in\{0,..,6\}\}\), where \(i,j\) are identifiers for the rows and columns of the kernel, respectively. This kernel moves over an image, and the weights operate as multipliers for each corresponding pixel, establishing a linear combination between the weights and pixels. The outcome of this calculation is then linked to the central pixel position of the kernel, in our case \(w_{33}\). If we were to momentarily employ a single Conv2D layer to depict the entire reconstruction procedure, it would be evident that a Conv2D layer, when trained solely on normal samples, would be proficient at reconstructing any image with near precision. This would hold for both normal and anomalous samples, with a graphic depiction of this scenario visible on the left side of Figure 2.  
We could envision that most of the kernel weights would have values close to zero and the central weight, \(w_{33}\), would have a value approximating 1, implying negligible reliance on the surrounding pixels' information. However, if we apply a mask to some of the central pixels of the kernel, assigning them zero values (as we do in our MaskedConv2D layer, shown in Figure 3), the central pixel would have to be deduced in the context of its surrounding pixels. In a two-dimensional space, these masked-out pixels form a sort of \"ring\", a boundary within which the central pixel lies. The task of the MaskedConv2D layer then becomes figuring out the value of this central pixel based on the information from the pixels that exist outside this zero-value ring. This changes the operation of the layer from a simple linear combination of all pixels within the kernel to a process of spatial inference, where the values of the surrounding pixels provide cues to what the central pixel might be. Hence, a MaskedConv2D layer trained on normal data would find it difficult to infer anomalies from these boundary pixels and is more likely to substitute the anomaly with a pattern that resembles normal data. This phenomenon can be seen on the right side of Figure 2.  
This presents a perspective not typically accounted for in traditional autoencoders. It is generally thought that the decoder will find it difficult to reconstruct anomalous samples, but this is not always the case. The encoder maps patterns to the latent space, and the decoder rebuilds these patterns. Ideally, normal and anomalous patterns would differ significantly, but if there is significant overlap between normal and anomalous samples, autoencoders may fall short. MaskedConv2D layers introduce the idea of extrapolating a normal pattern from its neighbouring patterns. It is worth noting a key differentiator of our method. Whereas other methodologies might mask the image input directly [38; 39], leading to a potential loss of information, our MaskedConv2D layer masks the convolutional kernels. This preserves the original data while altering the convolution operation, pushing the architecture to focus on the surrounding context rather than directly altered data. Figure 2: This figure visually illustrates the impact of the MaskedConv2D layer in the reconstruction process. The left side shows the reconstruction using a traditional Conv2D layer, where both normal and anomalous samples are reconstructed with near precision. However, on the right side, with the application of the MaskedConv2D layer, the reconstruction is influenced by the surrounding pixels due to the zero-value mask.