# Masked Convolutions Within Skip Connections for Video Anomaly Detection
## 1 Introduction
These connections allow the network to carry detailed spatial information from the encoder directly to the decoder, thereby facilitating the preservation and reconstruction of intricate spatial patterns often lost in the bottleneck of traditional autoencoders. However, UNets have a significant limitation in that they can proficiently reconstruct anomalous samples as well as normal samples due to their skip connections from earlier layers. This proficiency can obstruct anomaly detection and is often resolved by designing large complexity around the UNet, as is later demonstrated in Section 5.1 (Figure 7). In our research, we tackle the challenge of UNet's anomaly reconstruction by implementing special convolution layers, the MaskedConv3D, between the skip connections. While several methodologies in the realm of anomaly detection focus on masking the image or data directly, our approach diverges significantly. Instead of operating on the image, we introduce masking at the convolutional layer level, resulting in a more nuanced and informed mechanism for anomaly identification. This strategic departure offers a unique lens through which anomaly detection can be more effectively understood and implemented. Our research presents several key innovations that distinguish it from existing methods:  
* Unlike traditional approaches that **mask the image or data directly**, often requiring multiple random masks to account for unknown anomaly locations, our method **masks at the convolutional layer level**. This avoids the risk of obscuring critical normal features or missing anomalous ones. Moreover, by masking the kernel instead of the image, our approach enables efficient inference without requiring multiple masked versions of the input, significantly reducing computational overhead [12; 13]. * Our research introduces **MaskedConv3D layers** that are strategically placed **between the skip connections** of the UNet architecture. By placing the MaskedConv3D layers between skip connections, we apply feature masking and feature inference on various granularities of detail. This addition tackles the challenge of UNet's ability to reconstruct anomalies of various sizes [14; 15]. * By implementing masked convolutions between skip connections, our method infers normal spatio-temporal features to be passed to the decoder. Instead of passing forward the anomalies across the reconstruction process [6; 7], **using MaskedConv3D** layers means our **model needs to infer normal features** from surrounding spatio-temporal features. Section 2 of this paper reviews existing methods in video anomaly detection, highlighting their strengths and limitations. In Section 3, we introduce our proposed Masked-SkipUNet architecture and explain its unique components, such as the MaskedConv3D layer. Section 4 describes the three datasets used for evaluation: the Ped2, CUHK Avenue, and ShanghaiTech datasets. In Section 5, we present the performance of the Masked-SkipUNet, demonstrating its advantages over other frame-based methods. Finally, Section 6 provides an evaluation of the effectiveness of the MaskedSkipUNet architecture by comparing it across various perumataions of architecture components. The source code is publicly available at [[https://github.com/demetrislappas/MaskedSkipUNet](https://github.com/demetrislappas/MaskedSkipUNet)]([https://github.com/demetrislappas/MaskedSkipUNet](https://github.com/demetrislappas/MaskedSkipUNet)) (accessed on 1 April 2025).