<figure><img src="image_1.png" /></figure>

HAL
open science

# Unsupervised deep learning for spatio-temporal representations of videos: application to video surveillance

Devashish Lohani

► To cite this version:

Devashish Lohani. Unsupervised deep learning for spatio-temporal representations of videos: application to video surveillance. Computer Vision and Pattern Recognition [cs.CV]. Université Lumière Lyon 2, 2023. English. NNT: tel-04108165

HAL Id: tel-04108165

https://theses.hal.science/tel-04108165

Submitted on 26 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Université Lumière Lyon 2

Ecole Doctorale : ED 512 Informatique et Mathématiques

LIRIS

THÈSE

pour obtenir le grade de

DOCTEUR EN INFORMATIQUE

Unsupervised deep learning for
spatio-temporal representations of videos:
application to video surveillance

Devashish LOHANI

Informatique

Sous la direction de Laure TOUGNE RODET
Co-encadrée par Carlos CRISPIM-JUNIOR

Soutenue publiquement le 3 Avril 2023

Composition du jury :

Nicolas THOME, Professeur des Universités, CNAM Paris, Rapporteur

Thierry CHATEAU, Professeur des Universités, Université Clermont Ferrand II, Rapporteur

Jenny BENOIS-PINEAU, Professeur des Universités, Université de Bordeaux, Examinatrice

François BRÉMOND, Directeur de recherche, Inria Sophia Antipolis, Président

Laure TOUGNE RODET, Professeur des Universités, Université Lyon 2, Directrice de thèse

Carlos CRISPIM-JUNIOR, Maître de conférences, Université Lyon 2, Co-encadrant de thèse

Lionel ROBINAULT, Docteur (HDR), Foxstream, Tuteur entreprise
Acknowledgments

First of all, I would like to thank my thesis supervisors: Laure TOUGNE RODET and Carlos CRISPIM-JUNIOR. They not only guided me with their scientific expertise, but also supported me morally during difficult times. I am specially grateful to them for their valuable feedbacks on our papers and this manuscript. I would also like to thank my tutor of the enterprise Lionel ROBINAULT and members of the R&I team: Quentin BARTHÉLEMY and Sarah BERTRAND. They followed my work from the start of the thesis and we had very insightful scientific discussions, which helped me immensely for my research work. Without the above mentioned people, I cannot imagine where my thesis work would be. I am sincerely very grateful to all five of you.

I would also like to thank all the jury members for accepting to consider the scientific
contributions presented in this thesis. My gratitude also goes for all the members of
LIRIS and FOXSTREAM, with whom I had great discussions during coffee breaks. A
big thanks to all my family and friends, who always supported me and helped me stayed
sane and motivated.

Finally, I would like to thank ANRT, LIRIS and FOXSTREAM, for financing the work
of this CIFRE thesis.
Abstract

In the last two decades, we have witnessed a massive increase of surveillance cameras in our surroundings. One of the most important uses of these cameras is to detect suspectful or abnormal behaviors, e.g., a moving truck in a pedestrian zone or an intruder entering a prohibited site. These abnormal events occur very rarely and thus it is an extremely tedious and difficult task for professionals to attentively monitor the video constantly for finding these events. Therefore, an automatic video analysis system is essential. Traditional systems suffer to generalize across different types of anomalies, often rely on handcrafted rules and cannot adapt to abnormal events that they have never seen before. In the past few years, we have seen a tremendous progress in deep learning based video surveillance systems. These systems learn representative features from the data itself, generalize across different scenes and anomalies. That is why, in this thesis, we explore deep learning based methods. Majority of these methods in automatic video analysis are supervised, i.e., they require a large volume of labeled data. But since abnormal events depend on context and are rare, it is very difficult to have labeled anomalous data beforehand, and even if there is some annotated data for abnormal events, it will always be a small portion compared to normal data. Furthermore, one cannot annotate every possible event that might occur in future. So, we require approaches that can work without labeled data. Since these events occur in videos, they can have both spatial and temporal dimensions. Therefore, the approach must be able to learn pertinent spatio-temporal representations to differentiate abnormal and normal events.

Thus, in this PhD, prepared at LIRIS laboratory and in collaboration with the Foxstream enterprise, we aim to learn spatio-temporal representations from unlabeled videos to detect abnormal events. Precisely, we address the task of video anomaly detection and its sub-task, perimeter intrusion detection. We provided mathematical definitions to these tasks because they were not clearly defined in the literature. The definitions have a direct impact on the evaluation and therefore, we proposed new suitable evaluation schemes. Concerning spatio-temporal representation learning without annotations, we proposed two approaches. In the first approach, we designed a strided 3D convolutional autoencoder network and it was used for the perimeter intrusion detection task. The main idea is to learn normal representation from training data without intrusions (or anomalies) and detect intrusions (or anomalies) as they deviate from learned normality. It worked well in small-length videos but suffered in long videos, which have changes in scene dynamics like weather, lighting, etc. To address this problem, we introduced an adaptive thresholding approach using moving z-score. Our extensive experiments showed the viability of our approach in comparison with other existing methods. To further improve the spatio-temporal comprehension of normality, we introduced our second approach. It consisted of a framework that leverages unsupervised and self-supervised learning in an autoencoder. To be precise, we proposed multiple, carefully designed tasks (unsupervised and self-supervised) that are performed in a single autoencoder. This method is also
trained in an end-to-end and joint manner, where training data is without anomalies or
intrusions. For detecting anomalies (or intrusions), each of the tasks provide an anomaly
score and the combined score is used for final detection. This approach is generic and was
applied to the two tasks. We obtained state-of-the-art results in all major public datasets
for both video anomaly detection and perimeter intrusion detection task.

**Keywords:** deep learning, computer vision, unsupervised learning, self-supervised learning, video surveillance, video anomaly detection, perimeter intrusion detection
# Résumé

Au cours des deux dernières décennies, nous avons assisté à une augmentation massive des caméras de surveillance dans notre environnement. L'une des utilisations les plus importantes de ces caméras est de détecter des comportements suspects ou anormaux, par exemple un camion en mouvement dans une zone piétonne ou un intrus pénétrant dans un site interdit. Ces événements anormaux se produisent très rarement et c'est donc une tâche extrêmement fastidieuse pour les professionnels de surveiller manuellement la vidéo en permanence pour trouver ces événements. Par conséquent, une analyse vidéo automatique est essentielle. Les systèmes traditionnels ont du mal à se généraliser à différents types d'anomalies, s'appuient souvent sur règles et ne peuvent pas s'adapter à des événements anormaux qu'ils n'ont jamais vus auparavant. Au cours des dernières années, nous avons constaté d'énormes progrès dans les systèmes de vidéosurveillance basés sur l'apprentissage profond. Ces systèmes apprennent des caractéristiques représentatives à partir des données elles-mêmes et généraliser sur différentes scènes et anomalies. C'est pourquoi, dans cette thèse, nous explorons méthodes basées sur l'apprentissage profond. La majorité des travaux de la littérature en analyse vidéo automatique sont supervisés, c'est-à-dire qu'ils nécessitent un grand volume de données étiquetées pour obtenir des résultats pertinents. Mais comme les événements anormaux dépendent du contexte et sont rares, il est très difficile d'avoir des données anomaliques étiquetées à l'avance, et même s'il existe des données annotées pour les événements anormaux, ce sera toujours une petite partie par rapport aux données normales. De plus, on ne peut pas annoter tous les événements possibles qui pourraient se produire. Nous avons donc besoin d'approches qui peuvent fonctionner directement sur les vidéos, sans nécessiter d'annotations. Puisque ces événements se produisent dans des vidéos, ils ont à la fois des dimensions spatiales et temporelles. Par conséquent, l'approche doit pouvoir apprendre des représentations spatio-temporelles pertinentes pour différencier les événements anormaux et normaux.

Ainsi, dans cette thèse, préparée au sein du laboratoire LIRIS et en collaboration avec l'entrepr ise Foxstream, nous visons à apprendre des représentations spatio-temporelles à partir de vidéos non étiquetées pour détecter des événements anormaux. Plus précis ment, nous abordons la tâche de détection d'anomalie vidéo et sa sous-tâche, la détec tion d'intrusion périmétrique. Comme ces tâches n'étaient pas clairement définies, nous en avons proposé des définitions mathématiques. Ces définitions ont un impact direct sur l'évaluation et nous avons donc proposé de nouveaux schémas d'évaluation adaptés. Concernant l'apprentissage des représentations spatio-temporelles sans annotations, nous avons mis en place deux approches. Dans la première approche, nous avons conçu un réseau d'auto-encodeur convolutif 3D. L'idée principale est d'apprendre la représentation normale à partir de données d'entraînement sans intrusions (ou anomalies) et détecter les intrusions (ou anomalies) lorsqu'elles s'écartent de la normalité apprise. Il a été utilisé pour la tâche de détection d'intrusion périmétrique. Cela a bien fonctionné dans les vidéos de petite durée, mais moins bien dans les longues vidéos, qui ont des changements dans
la dynamique de la scène comme la météo, l'éclairage, etc. Pour résoudre ce problème,
nous avons introduit une approche de seuillage adaptatif utilisant le z-score mobile. Nos
nombreuses expérimentations ont montré la viabilité de notre approche par rapport aux
autres méthodes existantes. Pour encore améliorer la compréhension spatio-temporelle
de la normalité, nous avons introduit notre deuxième approche. Il consistait en un cadre
qui exploite l'apprentissage non supervisé et auto-supervisé dans un auto-encodeur. Pour
être précis, nous avons proposé de multiples tâches bien conçues (non supervisées et auto-
supervisé) qui sont exécutés dans un seul auto-encodeur. Cette méthode est également
entraînée de bout en bout et de manière conjointe, où les données d'entraînement sont
sans anomalies ou intrusions. Pour la détection d'anomalies (ou d'intrusions), chacune
des tâches fournit une anomalie score et le score combiné est utilisé pour la détection
finale. Cette approche est générique et a été appliquée aux deux tâches. Nous avons
obtenu des résultats au-delà de l'état de l'art pour les deux tâches, dans les principaux
jeux de données publics.

**Mot clés :** apprentissage profond, vision par ordinateur, apprentissage non supervisé, apprentissage auto-supervisé, vidéosurveillance, détection d'anomalie vidéo, détection d'intrusion périmétrique
# Contents

<table><tr><td>Introduction</td><td>1</td></tr><tr><td>1 Video surveillance: Data and challenges</td><td>5</td></tr><tr><td>1.1 Introduction</td><td>6</td></tr><tr><td>1.2 Data Acquisition in surveillance systems</td><td>6</td></tr><tr><td>1.3 Open Challenges</td><td>9</td></tr><tr><td>1.4 Datasets</td><td>12</td></tr><tr><td>1.5 Conclusion</td><td>18</td></tr><tr><td>2 Literature review</td><td>21</td></tr><tr><td>2.1 Introduction</td><td>23</td></tr><tr><td>2.2 Background</td><td>23</td></tr><tr><td>2.3 Spatio-temporal feature learning from videos</td><td>34</td></tr><tr><td>2.4 Unsupervised spatio-temporal video understanding</td><td>40</td></tr><tr><td>2.5 Video anomaly detection (VAD) methods</td><td>42</td></tr><tr><td>2.6 Perimeter intrusion detection (PID) methods</td><td>53</td></tr><tr><td>2.7 Conclusion</td><td>62</td></tr><tr><td>3 Defining tasks and their evaluations</td><td>65</td></tr><tr><td>3.1 Introduction</td><td>66</td></tr><tr><td>3.2 Defining video anomaly detection</td><td>66</td></tr><tr><td>3.3 Defining perimeter intrusion detection</td><td>70</td></tr><tr><td>3.4 Evaluation protocols</td><td>73</td></tr><tr><td>3.5 Evaluation metrics</td><td>77</td></tr><tr><td>3.6 Conclusion</td><td>81</td></tr><tr><td>4 Unsupervised autoencoder and adaptive detection</td><td>83</td></tr><tr><td>4.1 Introduction</td><td>84</td></tr><tr><td>4.2 Unsupervised and Adaptive Perimeter Intrusion Detector</td><td>84</td></tr><tr><td>4.3 Experiments and results</td><td>88</td></tr><tr><td>4.4 Conclusion</td><td>95</td></tr><tr><td>5 Leveraging Unsupervised and Self-Supervised Learning</td><td>99</td></tr><tr><td>5.1 Introduction</td><td>100</td></tr><tr><td>5.2 Proposed approach</td><td>100</td></tr><tr><td>5.3 Application to tasks</td><td>106</td></tr><tr><td>5.4 VAD: Experiments and results</td><td>110</td></tr><tr><td>5.5 PID: Experiments and results</td><td>115</td></tr><tr><td>5.6 Conclusion</td><td>118</td></tr></table>
CONTENTS

<table><tr><td>Conclusion and perspectives</td><td>121</td></tr><tr><td>Annexes</td><td>127</td></tr><tr><td>References</td><td>154</td></tr><tr><td>Table of figures</td><td>160</td></tr><tr><td>Liste of tables</td><td>161</td></tr></table>
# Introduction

## General context

Over the years, the video surveillance systems have evolved from simple video acquisition and display systems to intelligent autonomous systems (Ibrahim, 2016). The systems of today use some of the most sophisticated video analysis and decision-making algorithms to function. The massive installations of cameras in different sites, from banks to supermarkets and in prominent streets, have further helped in developing and testing these systems.

Visually surveying a site can include various tasks, such as object detection, object tracking, object segmentation, abnormal event detection, etc (Valera et Velastin, 2005). Out of these tasks, the abnormal event (anomaly) detection task has gained a major attention of the computer vision community (Ramachandra et al., 2022). Anomalies are patterns in data that do not follow a well-defined notion of normal behaviour (Chandola et al., 2009). Based on context and nature of input data, anomalies can refer to different patterns, such as abnormal sections in a time-series data, abnormal patches in an image, abnormal spatio-temporal volumes in a video, etc., as pictured in Figure 1.

<figure><img src="image_3.png" /><figcaption>Figure 1: Venn diagram to illustrate the taxonomy of tasks in anomaly detection.</figcaption></figure>

For data in video form, video anomaly detection (VAD) refers to the detection of unusual appearance or motion attributes in the video (Feng et al., 2021; Li et al., 2022). For example, for a video surveilled site where only pedestrians are authorized, all the vehicles are anomalies. Depending on the context, video anomaly detection can be further classified into specific tasks, such as detecting abandoned object (Luna et al., 2018), loitering (Ramachandra et Jones, 2020), illegally parked vehicle (Xie et al., 2017b), etc. The task of perimeter intrusion detection (PID) also falls into this category (Saligrama et al., 2010; Sodemann et al., 2012; Nayak et al., 2021). It aims at detecting the presence of an intrusion in a secured perimeter. Intrusions are a particular type of anomalies, classified
as point and contextual anomalies by (Chandola et al. (2009), Section 3.5). Moreover, the notions of perimeter, intruder movement and site protection time are crucial for the PID task, i.e., anomalous/unauthorized objects present in the video are intruders only if they are in movement inside the designated perimeter when the site is being surveyed.

In this thesis, we tackle the tasks of video anomaly detection and perimeter intrusion detection. Furthermore, both the tasks are carried out in the outdoor environment. The fact that it is an outdoor environment is very important here as it induces challenges such as changing weather and light conditions, obtrusion due to insects, animals, vegetation, etc., contrary to an indoor environment (Matern et al., 2013; Villamizar et al., 2018). Since the input to both tasks are videos, i.e., data with spatial and temporal dimensions. Anomalies or intrusion in videos too, can have the spatio-temporal nature. Therefore, it is essential to understand the spatio-temporal representation in videos correctly, in order to detect anomalies or intrusions. Some works address spatial (frames) and temporal dimensions of video separately and then process them sequentially. But these approaches do not learn the spatio-temporal nature of videos aptly (Hasan et al., 2016; Chong et Tay, 2017; Simonyan et Zisserman, 2014). Thus, we seek to develop approaches that learn the spatio-temporal nature of videos in a joint and cohesive manner.

Traditionally, most of the works used trajectory-based anomaly detection (Popoola et
Wang, 2012). The main idea is if the objects of interest are not following the learned
normal trajectories, then they are considered anomalous. This approach suffers from oc-
clusions as it needs to monitor continuously the objects of interest. Some other approaches
used low-level appearance, motion and texture features to enhance the comprehension of
normality. Other approaches can be found in the works of (Li et Cai, 2016; Kaur et al.,
2018). All these traditional approaches suffer to generalize across datasets and anomaly
types, often rely on handcrafted features, and are unable to adapt to anomalies that
they have never seen before (Hu et al., 2016; Medel et Savakis, 2016). For a sterile zone
with very few distractions, these approaches can function to some extent for the PID
task but in the case of VAD, these approaches perform poorly. Over the past few years,
deep learning has surpassed traditional approaches in many major computer vision tasks.
Concerning video anomaly detection too, deep learning based approaches have outper-
formed their traditional counterpart. This is because, given enough data, deep learning
based approaches can learn the features themselves, instead of using handcrafted or cho-
sen features. These approaches generalize with different scenes and types of anomalies.
Therefore, in this thesis, we consider developing deep learning based approaches. Most
of the existing works for spatio-temporal representation learning in video rely on huge
volume of labeled data to obtain relevant results. But since in VAD and PID task, the
examples of anomalies or intrusions are rare, it is extremely difficult to obtain annotated
data. Thus, the goal of this thesis is to learn pertinent spatio-temporal representation
from unlabeled videos in order to detect abnormal behaviors like anomaly or intrusion.
Furthermore, the proposed methods should be suitable to run in videos of long length
(hours, even days), in order to test if they are robust to real-world challenges like change
in weather, luminosity, etc.

Industrial context

This thesis is carried out within the framework of a CIFRE (Convention Industrielle de
Formation par la REcherche) partnership between the laboratory LIRIS (Imagine team)
and the company Foxstream (CIFRE n°2019/1709). The Imagine team of laboratory LIRIS (Laboratoire d'InfoRmatique en Image et Systèmes d'information) is specialized in computer vision, machine learning and pattern recognition. On the other hand, the company Foxstream is specialized in the analysis and automatic processing of real time video content. The video analysis algorithms developed by the company (perimeter intrusion detection, object segmentation, object tracking, etc.) use "classic" signal processing or image analysis algorithms such as modeling of the static components of the scene or using low level descriptors such as SIFT, Haar wavelets, HOG, etc. With this thesis, they want to explore deep learning based solutions for perimeter intrusion detection and video anomaly detection tasks. Furthermore, since intrusions and anomalies are rare events, the company cannot provide labeled data as one cannot anticipate all possible anomalies beforehand. Therefore the proposed methods should be able to work directly in unlabeled data. Overall, the work from this thesis would help the company to move towards deep learning approaches in many of its major applications and that too, without additional cost of manual annotation.

# Plan of manuscript

This manuscript is composed of 5 chapters:

* Chapter 1 details various data acquisition schemes and associated challenges in video surveillance. It also describes various datasets used in this work along with their challenges and shortcomings.

* Chapter 2 corresponds to a literature review of various topics: spatio-temporal representation learning in videos with and without annotations, video anomaly detection and perimeter intrusion detection. The basic concepts needed to understand this manuscript are also defined there.

* Chapter 3 is concerned with formally defining the tasks of video anomaly definition and perimeter intrusion definition, and revisiting their evaluation frameworks. We mathematically define the tasks and discuss existing evaluation protocols. We also propose a new, more suitable evaluation protocol.

* Chapter 4 focuses on the task of perimeter intrusion detection. For this, we propose an unsupervised autoencoder with adaptive thresholding mechanism. We provide extensive experimental results of our method in comparison with existing methods, using different evaluation protocols.

* Chapter 5 introduces a new approach for spatio-temporal representation learning from unlabeled videos. It leverages unsupervised and self-supervised learning and is applicable to both VAD and PID tasks. We perform exhaustive experiments on major public VAD and PID datasets and provide comparative analysis of our method with existing methods.

# Contributions

The scientific contributions made during this thesis and detailed in the chapters of this manuscript have been published in international peer-reviewed conferences and journals.
Our first article was dedicated to the task of fall detection and perimeter intrusion detection. It was published in the Reproducible Research in Pattern Recognition (RRPR) workshop of ICPR 2021 conference (Lohani et al., 2021). Both these tasks are subsets of VAD task and do not contain training labels. In this work, we extended the original work to perimeter intrusion detection task. We also proposed to use precision-recall curve and it provided insightful results on the two tasks. Our next article was published in the special issue "Unusual Behavior Detection Based on Machine Learning" of the Sensors 2022 journal (Lohani et al., 2022b). In this article, we provided mathematical definition of the PID task, along with a review of existing methods and proposed a new suitable evaluation scheme. The work of adaptive PIDS was published in the IEEE International Conference on Image Processing (ICIP) 2022 (Lohani et al., 2022a). In our final contribution, we proposed multiple tasks (unsupervised and self-supervised) for video anomaly detection. We proposed these tasks on a single autoencoder and the main idea is to enrich the spatio-temporal understanding of normality. Experiments were performed in major public VAD datasets and we obtained state-of-the-art results in each of them. This work has been accepted in the International Conference on Computer Vision Theory and Applications (VISAPP), 2023.
# Chapter 1

# Video surveillance: Data and challenges

In this chapter, we first explain how using different acquisition schemes, the data can be acquired in a surveillance system. Then, we explain various forms of challenges associated either with scene capturing, objects of interest or complexity of the scene. Finally, we detail the datasets that we use along with their difficulties and shortcomings.

## Contents

<table><tr><td>1.1</td><td>Introduction</td><td>6</td></tr><tr><td>1.2</td><td>Data Acquisition in surveillance systems</td><td>6</td></tr><tr><td>1.2.1</td><td>Visual Camera</td><td>6</td></tr><tr><td>1.2.2</td><td>Infrared Camera</td><td>6</td></tr><tr><td>1.2.3</td><td>Thermal Camera</td><td>7</td></tr><tr><td>1.2.4</td><td>Other Types of Acquisitions</td><td>8</td></tr><tr><td>1.3</td><td>Open Challenges</td><td>9</td></tr><tr><td>1.3.1</td><td>Challenges in capturing the scene</td><td>9</td></tr><tr><td>1.3.2</td><td>Challenges related to the complexity of the scene</td><td>10</td></tr><tr><td>1.3.3</td><td>Challenges related to the objects of interest</td><td>11</td></tr><tr><td>1.4</td><td>Datasets</td><td>12</td></tr><tr><td>1.4.1</td><td>VAD datasets</td><td>13</td></tr><tr><td>1.4.2</td><td>PID datasets</td><td>16</td></tr><tr><td>1.5</td><td>Conclusion</td><td>18</td></tr></table>
## 1.1 Introduction

Main steps of a modern surveillance system are data acquisition, data pre-processing, data analysis and returning an output signal (like an alarm). In this chapter, we focus on the data acquisition step. This step is very important since the input of analysis part is obtained from here. It is the basis of any surveillance system. The data acquired depends on the type of camera used like visual camera, infrared camera, thermal camera, etc. Each of these cameras have their strengths and weaknesses, and therefore they should be used carefully. Thus, in this chapter, we first explain various data acquisition schemes. Since we are interested in outdoor video surveillance, the cameras are placed in the outdoor environment. In outdoor environment, they face various challenges, e.g., challenges due to change in environmental conditions like rain, dust, or bad weather (Hu et al., 2018). Therefore, there are various challenges associated with data in video surveillance and they will be also detailed in this chapter. Finally, we also explore different existing VAD and PID datasets, along with their associated challenges.

## 1.2 Data Acquisition in surveillance systems

In this section, we describe the various data acquisition systems with their associated advantages and drawbacks. The area to be surveyed is observed with the help of cameras. All cameras can acquire the video stream, but the nature of data depends on the type of camera used. Broadly, the following categories of video capture devices are used.

### 1.2.1 Visual Camera

It is also called as the ‘classic’ or ‘color’ camera as it captures the visible light in form of grey-scale or RGB images and was the first type of cameras to be used for the video surveillance. The advantage is that it renders an image visually closer to the naked eye. However, for proper functioning, it needs a certain level of brightness in the scene from external light source like sun and furthermore it is sensitive to illumination changes (Van De Sande et al., 2009; Ibrahim, 2016). During night, it needs additional lighting for the sensor to generate a sufficiently contrasted image (Gade et Moeslund, 2014). Adverse weather conditions such as fog, rain, snow, etc., further limit the observation of objects to a short distance from the camera (Robinault, 2021), and thus make detection difficult. Even after all these drawbacks, these cameras have been used extensively in video surveillance systems (Valera et Velastin, 2005; Kim et al., 2011).

### 1.2.2 Infrared Camera

To address the issue of scene brightness, the infrared cameras were proposed as they capture near-infrared emissions from objects (Ibrahim, 2016). To be precise, the term infrared camera is a bit of misnomer because the classic visual camera can also capture the near-infrared emissions, but it is restricted to visible bandwidth using a filter (Robinault, 2021). Therefore, the infrared camera is nothing, but a visual camera coupled with an infrared (IR) lighting. The IR lighting illuminates the scene with near infrared radiation (0.7-1.4 µm) and the camera captures the reflected radiation from objects in the scene in both the visible and near infrared spectrum (Gade et Moeslund, 2014). Thus, it can work during night too and can provide better contrast when an object moves past the
camera (Haritaoglu et al., 2000). However, it is difficult to detect objects during rain in
this camera as rain drops appear as thick stripes in front of the camera (Liang et al.,
2012). Furthermore, due to the heat from IR lighting, this camera attracts flying insects
and spiders that can raise false alarms and, consequently, impact detection. Figure 1.1
demonstrates these issues with infrared camera.

<figure><img src="image_4.png" /><figcaption>Figure 1.1: Issues with the infrared camera: low visibility in rain (left) and insect on camera (right).</figcaption></figure>

1.2.3 Thermal Camera

Unlike the visual and infrared camera, the thermal camera does not require an external energy source for illuminating the scene. In fact, it relies on the infrared radiation emitted by all objects with a temperature above absolute zero. The thermal camera captures this radiation which is in the mid and long infrared wavelength spectrum (3-14 µm) (Gade et Moeslund, 2014). Since it does not depend on illuminating the scene, it can work even in complete darkness (Ibrahim, 2016). The thermal image is rendered grayscale where the high intensity pixels relate to high temperature. In other words, the higher the temperature of an object in the scene, the brighter it will appear in the thermal image. Advantages of this camera include lower sensibility to weather conditions and object shadows (Liang et al., 2012), and long-range detection (Xu et al., 2016; Hu et al., 2018). However, the main drawback of the thermal camera is that it is difficult to distinguish an object from its background when both have almost the same temperature (Robinault, 2021).

Figure 1.2 shows examples of data acquisition by color and thermal camera. During the day, we can observe that three persons and vehicles are visible in both the cameras. While thermal camera provide better contrast for first two persons and is less affected by shadows, the third person is better visible in color camera since in thermal camera, the object is camouflaged with the car behind it. During night, we cannot see much from the color camera, just the light emitted by an electric scooter. In complete contrast to this, the thermal camera shows the complete scene with details and the person with its scooter is clearly visible.
<figure><img src="image_5.png" /><figcaption>Figure 1.2: Data acquisition on a site by color (left) and thermal camera (right). Each camera captures two scenes at the same instant, one during day (top row) and the other during night (bottom row).</figcaption></figure>

### 1.2.4 Other Types of Acquisitions

The depth information of an object can be a useful information for its detection (Vil-lamizar et al., 2018; Aravamuthan et al., 2020). This depth information can be added to any camera type using an additional depth sensor. It determines the perspective size of an object in the scene. However, using a depth sensor also has several problems, such as mixed, lost, and noisy pixels in the depth image (Kim et al., 2013; Shao et al., 2014).

Another important information in the scene is the motion information. The camera type specializing in this are known as the event camera (Lichtsteiner et al., 2008). It generates events at the microsecond resolution. An event is triggered each time a single pixel detects a change in intensity value. It finds its application in motion detection, object segmentation, pose estimation, motion tracking, etc. (Gallego et al., 2020). Advantages include low latency (no motion blur), high temporal resolution, high dynamic range, and ultra-low power consumption. However, it cannot capture static motion and absolute intensity, and therefore is often used together with other camera types. Event cameras are not used for tasks such as video anomaly detection and perimeter intrusion detection yet, where both spatial and temporal information are essential. The main reasons for this are its extremely high cost and inability to capture visual features, such as texture and color.

Finally, we can also have acquisitions from multi-camera systems where camera types
can be homogeneous or heterogeneous. Intuitively, combining multiple cameras can pro-
vide more accurate information about the targeted object and may help to overcome
occlusions. Surveillance systems with multiple cameras has been studied extensively (Cai
et Aggarwal, 1999; Hu et al., 2004; Wang et al., 2009). Recently, an intelligent perime-
ter intrusion detection system was proposed using an integrated image acquisition device
that combines visual and thermal camera (Kim et al., 2018b). However, a multi-camera
system also brings new challenges, such as camera installation, camera calibration, object
matching, automated camera switching and data fusion (Hu et al., 2004).

To conclude, no acquisition scheme is perfect. Visual camera is still largely used but
it does not work well when the scene illumination is low. The infrared camera can work
in low illumination but it suffers in bad weather conditions, particularly during rain, and
furthermore infrared lighting attracts insects, which can occlude the view. Even though
scene illumination does not affect thermal cameras, but it has other problems like an ob-
ject is indistinguishable from the background if their temperature is similar. Additional
sensors, like the depth sensor, can also be added to any of the camera types to improve
scene comprehension. Also, a multi-camera system can be made using various camera
types. But even these configurations have their drawbacks. Therefore, the video acqui-
sition scheme should be chosen according to the requirements of the site and concerned
task.

1.3 Open Challenges

In the context of video surveillance, the cameras are generally fixed in a static position to monitor a site (Valera et Velastin, 2005; Bouwmans, 2011). This site can be either indoors like a bank or agency, or outdoors like an industrial site, private property, or public places such as a city square. We focus on outdoor sites and they are more challenging due to environmental conditions like rain, dust, or bad weather (Hu et al., 2018). The system must be operational continuously for many days and encounter changing light and weather (i-LIDS Team, 2006). Several authors have identified different challenges in the outdoor video surveillance (Bouwmans, 2011; Brutzer et al., 2011). We can group all these challenges into three main categories. The first category is specific to the data acquisition. The second category corresponds to the captured scene as a whole and the third is linked to the objects of interest.

1.3.1 Challenges in capturing the scene

These challenges are related to the acquisition and transmission of the video signal.

a) **Noise:** It is characterized by a random variation in pixel intensities or color components (Xu *et al.*, 2016). In video surveillance, there are two main sources of noise (Brutzer *et al.*, 2011). Noise related to the quality of the sensor and noise related to compression. In the case of the sensor, noise is often present when the brightness of the scene is low. This noise is more frequent on thermal cameras (Gade et Moeslund, 2014). Compression noise is related to encoding and depends on the bandwidth available on the network.

b) **Automatic adjustments:** Cameras try, as far as possible, to maximize the dynamics of their sensor to present a correctly contrasted image. They accomplish this with
functionalities like auto focus, automatic gain control, automatic white balance, and auto
brightness control (Bouwmans, 2014). These adjustments results in an overall change in
brightness and color levels among different frames in the video. This phenomenon is more
important on thermal cameras, where the sensors also auto-adjust depending on the scene
temperature (Gade et Moeslund, 2014).

**c) Vibrations or jitters:** It refers to all the untimely movements of the camera which cause a displacement of the optical center. We are in the case of a fixed camera where the optical center and all the shooting conditions must not be modified during the acquisition of the video. However, it is not uncommon that the camera experiences some vibrations due to wind or other factors (Xu *et al.*, 2016; Sehairi *et al.*, 2017). These vibrations cause false motions in the scene (Bouwmans, 2014). Mechanical or software solutions can be used to correct this defect (Bouwmans, 2011). If these solutions prove to be insufficient, the same pixel coordinates on two successive images no longer represents the same structure in the scene.

**d) Dirty or Misadjusted lens:** When the cameras are outdoors, projections of rain, dust or bad weather can gradually degrade the quality of the image (Hu *et al.*, 2018; Zou *et al.*, 2019). Likewise, the camera lens adjusted during installation may become out of focus over time. As a result, the video acquired by the camera gradually lose sharpness and becomes blurry (Liang *et al.*, 2012). This problem should not be taken lightly because it corresponds to many real cases. A blurred image causes a significant loss of the information (both texture and color) contained in the scene (Tsakanikas et Dagiuklas, 2018).

This list is not exhaustive. We can add the case of so-called day / night cameras which transmit color frames when the scene brightness is sufficient and switch to grayscale frames when the brightness is low. We can also add the case of cameras with integrated infrared lighting that we have briefly presented previously or the problems of dazzling by vehicle headlights.

1.3.2 Challenges related to the complexity of the scene

This second category of challenges is linked to the complexity of the scene, regardless of
the presence or absence of objects of interest. In the context of outdoor video surveillance,
we can be confronted with a multitude of situations which can range from a parking lot
without trees, to a field bordered by tall trees and even a riverbank. In addition, the al-
gorithms must also consider different climatic conditions and different lighting conditions.
These different situations and conditions can be summarized by the following set of cases.

a) **Progressive illumination changes:** These variations are generally caused by the change in light conditions which progressively evolve over time and by the course of the sunlight (Bouwmans *et al.*, 2008; Xu *et al.*, 2016). For example, it can be caused by the passage of cloud shadows in the scene.

b) **Sudden illumination changes:** It corresponds to a sudden change in illumination between two successive images (Brutzer *et al.*, 2011; Xu *et al.*, 2016). This change can be either in part of the image or in the full image (Cermeño *et al.*, 2018). It is generally
caused by the activation or deactivation of exterior lighting, but it can also be linked to
the sudden introduction of a massive object into the scene (Bouwmans, 2011).

c) **Dynamic background:** Sometimes, some parts of the scene are dynamic and contain movements of irrelevant objects. Examples include tree branches or flag driven by the wind, the reflection of moving clouds on the surface of river, snowflakes, rain, etc. This can confuse the system to make false detections (Liang *et al.*, 2012; Bouwmans, 2014; Cermeño *et al.*, 2018).

d) **Shadows:** Managing shadows is another big challenge (i-LIDS Team, 2006; Al-Najdawi *et al.*, 2012). Shadows casted by moving objects of interest or static objects of scene can lead to false detections (specially in grayscale mode) (Liang *et al.*, 2012). Unlike color cameras, the thermal cameras do not have this problem of moving object shadows (Gade et Moeslund, 2014).

e) **Perspective:** This challenge is linked to the fact that we want to detect objects of interest both close and far from the camera. However, the same object does not have the same size depending on its distance from the camera (see Figure 1.3) (Hu *et al.*, 2004). Besides, weather conditions can attenuate the contrast between the object of interest and the background of the scene. This, along with object's perspective, makes detection more difficult (Buch et Velastin, 2014). Furthermore, as the apparent object size decreases with the distance, the apparent object speed also decreases.

<figure><img src="image_6.png" /><figcaption>Figure 1.3: Effect of perspective: three people (with blue bounding boxes) in the same image with different apparent size. Image source: ShanghaiTech dataset (Luo *et al.*, 2017a).</figcaption></figure>

### 1.3.3 Challenges related to the objects of interest

This last category of challenges concerns more specifically the objects that one seeks to
detect in the scene. These challenges are therefore much more linked to the problem we
are trying to solve. Most video surveillance tasks face the following challenges.
a) **Camouflage:** Camouflage, whether intentional or not, occurs when the difference in appearance between the object of interest and the background is small (Maddalena et Petrosino, 2008; Brutzer *et al.*, 2011). It is a particularly difficult challenge when the scene is full of different sized objects as shown in Figure 1.4. It is often possible to tune the parameters of an algorithm to detect objects on a short camouflaged video sequence (Cermeño *et al.*, 2018; Kim *et al.*, 2018b). We can therefore manage to detect camouflaged objects by increasing the sensitivity of the algorithm. However, when these very sensitive parameters are used for monitoring a site 24 hours per day, the number of false alarms can become impossible to manage.

<figure><img src="image_7.png" /><figcaption>Figure 1.4: Camouflaged image with three persons to detect. Source: (Robinault, 2021).</figcaption></figure>

b) **Hole in the object of interest (Foreground aperture):** This problem occurs when objects having insignificant texture difference with the scene background, move slowly in the scene or in the axis of the camera. Due to this, the object pixels are difficult to detect because they can be embedded in the scene background (Maddalena et Petrosino, 2008; Bouwmans, 2014). Thus, the same object may appear as several small objects, as if there are holes in the object.

Among other challenges, we have the speed of objects and their manner of movement (i-LIDS Team, 2006). Very slow or very fast moving objects can be difficult to detect. Furthermore, the manner in which the object moves can create ambiguity, e.g., person crawling or imitating an animal.

## 1.4 Datasets

Since we investigate the problem of abnormal event detection in videos, we explore the concerning datasets. Concretely, we describe the datasets for video anomaly detection (VAD) and perimeter intrusion detection (PID). As mentioned previously (refer Section 1.1), we are interested in systems that operate in the outdoor environment and consequently we choose datasets that follow this criterion.
1.4.1 VAD datasets

Anomalies in video can be caused by anything like riding a bicycle, fighting, dancing, playing, running, etc. In other words, they depend on context and can differ from one dataset to another. A formal definition for the VAD task will be presented in Chapter 3.

VAD methods rely on a large number of datasets in the literature such as: subway entrance (Adam et al., 2008), subway exit (Adam et al., 2008), UMN (Mehran et al., 2009), UCF-Crime (Sultani et al., 2018), UBnormal (Acsintoae et al., 2022), UCSD Pedestrian (Li et al., 2013), CUHK Avenue (Lu et al., 2013), ShanghaiTech (Luo et al., 2017a), etc. However, most of recent works only use the last three datasets of the list for unsupervised video anomaly detection (Zhao et al., 2017; Lee et al., 2018; Liu et al., 2018; Gong et al., 2019; Ionescu et al., 2019; Park et al., 2020; Astrid et al., 2021b; Liu et al., 2021b; Lv et al., 2021; Park et al., 2022). The reasons for not using the other datasets are the following: (a) Subway entrance and subway exit dataset: the scenes are in indoor environment rather than outdoors. No clear training sets, for instance, Cheng et al. (2015) use the first 15 minute videos for training, while Lu et al. (2013) use more than half of the videos for training; (b) UMN: some scenes are in indoor environment, no clear training and testing split (Ramachandra et al., 2022) and the dataset is already saturated (performance above 99% by many methods) (Acsintoae et al., 2022); (c) UCF-Crime: used for weakly or semi supervised anomaly detection since it contains both annotated anomalies and normal events in the training set (Ramachandra et al., 2022; Acsintoae et al., 2022); (d) UBnormal: a recent dataset but it consists of synthetic data instead of real anomalies.

We detail below the three commonly used VAD datasets, i.e., UCSD Pedestrian, CUHK Avenue, and ShanghaiTech, respectively.

1.4.1.1 UCSD Pedestrian Anomaly Dataset

It is one of the most widely used datasets for video anomaly detection (Kiran et al., 2018). This dataset consists of video clips recorded with a stationary color camera with grayscale frames, overlooking pedestrian walkways on the UCSD campus (Li et al., 2013). It has two distinct scenes, leading to two subsets of the dataset, called as Ped1 and Ped2. The difference between the two subsets is the walking direction (towards and away from the camera in Ped1, parallel to the camera plane in Ped2) and frame resolution (158 × 238 in Ped1 and 240 × 360 in Ped2). Most works use only the Ped2 dataset (Gong et al., 2019; Ionescu et al., 2019; Feng et al., 2021; Georgescu et al., 2021a; Liu et al., 2021b; Cho et al., 2022; Park et al., 2022) because Ped1 has a considerably low video resolution (Hinami et al., 2017; Doshi et Yilmaz, 2021; Ouyang et Sanchez, 2021; Le et Kim, 2022), is suitable for pixel-level and not frame-level anomaly detection (Xu et al., 2015a; Luo et al., 2019) and has ambiguity in anomaly annotation (some events labeled as normal in training set are considered as anomalies in testing set) (Nguyen et Meunier, 2019). For all these reasons, we also only focus on the UCSD Ped2 dataset.

There is a constant movement of people in this dataset, and it is considered normal. The abnormal events are due to the circulation of non-pedestrian entities, i.e., bikers, skaters, carts, and wheelchairs. All these events occur naturally, i.e., they were not staged for dataset collection. Figure 1.5 shows illustrative normal and anomalous frames of this dataset. This dataset contains 28 videos in total, with 120 to 200 frames per video. The training set has 16 videos (2,550 frames) without anomalies and the testing set comprises of 12 videos (2,010 frames) with anomalous events. Both pixel-level and frame-level annotations are provided with this dataset. The following challenges are present in this
<figure><img src="image_8.png" /><figcaption>Figure 1.5: Example frames from the UCSD Ped2 dataset. First row contains frames without any anomaly, with increasing crowd volume (left to right order). Second row shows anomalous frames with anomalies highlighted in red bounding boxes.</figcaption></figure>

dataset: dynamic background, shadows, perspective, and camouflage.

Despite being widely used, this dataset has various shortcomings. First, it is one of the smallest datasets in terms of videos, number of frames, total anomalies, and number of anomaly types. Therefore, most methods have almost perfect results on this dataset. Second, all the anomalies can be detected by only analyzing a single frame at a time. In other words, the anomalies have a dominant appearance / spatial difference from normality and it is easy to detect them based on spatial features, without requiring the temporal characteristics.

### 1.4.1.2 CUHK Avenue Dataset

It consists of one scene captured in the Chinese University of Hong Kong (CUHK) campus avenue with a fixed color camera (Lu *et al.*, 2013). People walking towards various pre-defined directions is considered as a normal event. The abnormal events are due to either non-pedestrian objects like bicycle or unexpected pedestrian behavior like running, loitering, jumping, throwing bag, walking in prohibited direction, etc.. We must highlight that this was the first dataset to introduce loitering (static) as an anomaly, which is important for surveillance applications. Unlike UCSD Ped2, anomalies are mostly staged in this dataset. Some illustrative normal and anomalous frames of this dataset are shown in Figure 1.6

This dataset contains 16 video clips (15,328 frames) for training and 21 for testing (15,324 frames), with a frame resolution of 360 × 640. The training videos consist of normal events with the exception of few videos with small occurrences of anomalies. The testing videos have both normal and abnormal events. Like UCSD Ped2, we have both the pixel-level and frame-level annotations. This dataset presents the following challenges: camera shakes, perspective, dynamic background, and progressive illumination changes.

Avenue dataset also has still some deficiencies. Like UCSD Ped2, we have a very small amount of data for deep neural networks and some authors argue that the amount of normal events is not sufficient for training (Li et Chang, 2019). Furthermore, the annotations are ambiguous for some test videos, leading to their exclusion by some works (Hinami *et al.*, 2017; Ionescu *et al.*, 2019; Ouyang et Sanchez, 2021).
<figure><img src="image_9.png" /><figcaption>Figure 1.6: Example frames from the CUHK Avenue dataset. First row contains frames without any anomaly but with different scene dynamics. Second row shows the following anomalies in red bounding boxes (left to right order): bicycle, throwing bag and running person, loitering and person walking in wrong direction.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 1.7: Some frames from the ShanghaiTech dataset, each from different scene. First row contains frames without any anomaly but with different scene dynamics. Second row depicts the following anomalies in red bounding boxes (left to right order): chasing, small truck, brawling and two people jumping.</figcaption></figure>

1.4.1.3 ShanghaiTech Campus Dataset

All previous datasets had a single camera view (or scene) in training and testing set. Having only a single view might lead some models to overfit by memorizing the fixed background. That is why it is essential to have different views in the dataset, so that the proposed detection method is aware of different scenes and generalizes to them. The ShanghaiTech Campus dataset is one of the most challenging datasets for video anomaly detection and includes 13 different views with various lighting conditions and camera angles (Luo et al., 2017a). The videos are filmed at the ShanghaiTech university campus using color cameras with the frame resolution of 480 × 856. Pedestrians walking in the campus with regular pace is considered as a normal event. Abnormal events are caused either by the movement of unauthorized objects like bicycle, skates, strollers, etc. or by irregular pedestrian behaviors like brawling, chasing, jumping, throwing objects, robbery, etc. Figure 1.7 illustrates various normal and abnormal events in this dataset.

It is one of the biggest anomaly detection datasets with 330 training (274,515 frames) and 107 test videos (42,883 frames). Like other datasets, training videos only contain normal events and testing videos have both normal and abnormal events. For each anomalous event both pixel-level and frame-level ground-truth annotations are available. It is worth noting that recently Zhong et al. (2019) proposed a reorganization of this dataset to also have annotated anomalies during training for weakly supervised learning. We focus on
<table><caption>Table 1.1: Description of video anomaly detection datasets.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="5">Number of frames</th><th rowspan="2">#Anomalies</th><th rowspan="2">#Anomaly types</th></tr><tr><th>Total</th><th>Train</th><th>Test</th><th>Normal</th><th>Abnormal</th></tr></thead><tbody><tr><td>UCSD Ped2</td><td>4,560</td><td>2,550</td><td>2,010</td><td>2,924</td><td>1,636</td><td>23</td><td>5</td></tr><tr><td>Avenue</td><td>30,652</td><td>15,328</td><td>15,324</td><td>26,832</td><td>3,820</td><td>47</td><td>5</td></tr><tr><td>ShanghaiTech</td><td>317,398</td><td>274,515</td><td>42,883</td><td>300,308</td><td>17,090</td><td>158</td><td>11</td></tr></tbody></table>

the original dataset setting as we focus on developing methods without annotations. This dataset presents the following challenges: noise, camera shakes, shadows, perspective, dynamic background, progressive and sudden illumination changes.

We still have the following deficiencies in this dataset. Despite being a very large dataset, we have only a small number of anomalies (158) belonging to only 11 anomaly types. Therefore, it lacks diversity and number of anomalies. Similarly, since there are overall 13 scenes, we have a small training set for each scene. This small per-scene training set is not fully representative of all the normal activities (Ramachandra et al., 2022). Therefore, many works tend to use some form of data augmentation techniques to have enough representative data (Astrid et al., 2021a; Feng et al., 2021).

Table 1.1 summarizes the three datasets with frame-level details. It must be noted that none of these datasets have a validation set, which makes it difficult to choose hyperparameters while training any model.

## 1.4.2 PID datasets

A perimeter intrusion detection system (PIDS) concerns with the detection of intrusions in a secured perimeter. Like VAD, the intrusions depend on the context and vary from one dataset to another. A mathematical definition for PID task will be presented in Chapter 3. PID can include various subtasks such as detection and tracking, therefore historically algorithms were tested on the datasets of these subtasks. For example, CAVIAR (Crowley et al., 2005), PETS2006 (Thirde et al., 2006) and AVSS2007 (AVSS dataset, 2007) have been used to test the tracking module of the PIDS (Vijverberg et al., 2010, 2013; Nayak et al., 2019). Concerning the PID task, most works use their private datasets. One recent work introduces a new dataset, called SIC (Cermeño et al., 2018), but it is available under strict conditions and, without annotations, thus omissions cannot be evaluated. Finally, there is only one public dataset, dedicated for the PID task: the i-LIDS sterile zone dataset (i-LIDS Team, 2006) and it has been extensively used in the literature (Buch et Velastin, 2008; Vijverberg et al., 2013, 2014; Buch et Velastin, 2014; Cermeño et al., 2018).

### 1.4.2.1 i-LIDS sterile zone dataset

The imagery library for the intelligent detection systems (i-LIDS) consists a library of closed-circuit television (CCTV) video footage datasets for benchmarking video analytics systems. Concerning the PID task, they propose a sterile zone dataset, known as the i-LIDS sterile zone dataset (i-LIDS Team, 2006). This dataset comes with a proper annotation and evaluation procedure to ensure its relevance for the industrial application.
<figure><img src="image_11.png" /><figcaption>Figure 1.8: Example frames drawn from i-LIDS sterile zone dataset with various intrusion (intruders in red boxes) and non-intrusion frames. The color and black and white frames belong to view 1 and view 2 respectively. The time of the day in four columns are dawn, dusk, night, and day. The distractions (in blue boxes) in non-intrusion row (left to right order) are a rabbit, birds with rain, fox and an insect on camera respectively. In intrusion row (left to right order), we have two intruders, intruder with ladder during snow, log rolling intruder and crawling intruder respectively.</figcaption></figure>

The PID objective in this dataset is to detect the presence of people in a sterile zone. It consists of two sites monitored by two different cameras (view 1 in color/monochrome and view 2 in monochrome) as shown in Figure 1.8. There is a security fence in each site (view) and the aim is to detect intrusion as soon as it enters the site. Intruders try to breach the fence in various ways. For example, people may walk, run, crawl, or roll towards the fence and, on occasion, may carry climbing aids, such as a ladder. The intruders can be at three different distances from the camera: close, middle, and far. The cameras recordings are over many days, capturing different times of the day, such as dawn, day, dusk, and night. They also include various weather conditions, such as cloudy, rainy, snowy, and foggy. Furthermore, there are various distractions that might trigger false alerts, like plastic bag or paper moving due to wind, bats, birds, foxes, insects, rabbits, squirrels, shadows through the fence, etc. Figure 1.8 illustrates various distractions and intrusion examples. All these factors like different weather conditions, different times of the day, numerous distractions, and various ways in which the intruder approaches the fence make it a very challenging and realistic dataset. Finally, the challenges present in this dataset are noise, automatic adjustments (switch from color to monochrome images and vice-versa), vibrations, progressive / sudden illumination changes, shadows, perspective, and camouflage.

It has two separate disks for training and testing. The PID system is developed using the training disk data and its performance is evaluated with the testing disk data. Each disk contains the two camera views, with over 20 hours of video recorded in the various previously cited situations. All videos are taken at 25 FPS with 720 × 576 frame size resolution. Each disk contains annotations, distractions, and other information for each video. The annotation provided is the time interval of each intrusion event in the video i.e., the entry and exit time of people in the respective scene.

Table 1.2 summarizes the i-LIDS dataset where view 1 and view 2 corresponds to
<table><caption>Table 1.2: i-LIDS Sterile Zone dataset description.</caption><thead><tr><th rowspan="2">View</th><th rowspan="2">Set</th><th rowspan="2">#Videos</th><th rowspan="2">#Intrusions</th><th colspan="7">Number of videos per intrusion count</th></tr><tr><th>0</th><th>1</th><th>10</th><th>13</th><th>15</th><th>17</th><th>31</th></tr></thead><tbody><tr><td rowspan="2">View 1</td><td>train</td><td>123</td><td>113</td><td>10</td><td>113</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>test</td><td>17</td><td>113</td><td>10</td><td>0</td><td>2</td><td>1</td><td>1</td><td>2</td><td>1</td></tr><tr><td rowspan="2">View 2</td><td>train</td><td>113</td><td>103</td><td>10</td><td>103</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>test</td><td>16</td><td>103</td><td>10</td><td>0</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td></tr></tbody></table>

the color camera and monochrome camera respectively. Each view has videos with and without intrusions. In both training and testing disk, we have 10 videos from each view without any intrusions. In training disk, we have 216 short videos (≈ 3 minutes), with single intrusion per video. These videos can be used for training and/or validation for example. The testing disk have 7 and 6 very long videos in view 1 and view 2. These long videos range from 36 minutes to 92 minutes and contain between 10 and 31 intrusions per video. Even though the total number of intrusions are also 216 in testing disk, the long-length videos makes it more challenging. All i-LIDS videos contain various distractions and/or intruders but these long videos also have a drastic change in weather and luminosity. This makes this testing dataset even more relevant from practical perspective. Furthermore, the distractions, intrusions and other information is very well distributed in both the views and thus we do not have bias for the choice of a camera view. It must be noted that like VAD datasets, it does not contain an official validation set. However, in unsupervised context, the intrusion videos from the training set can be used as a form of validation set.

Although we have extensive real-life elements in i-LIDS dataset, it has some major drawbacks. First, we only have people as intrusion. This is a limitation because in real-life, intrusion can very well be some other object like a car, a bike, a truck, etc.. Consequently, any good human detector system might work very well in this dataset but that does not necessarily mean that it is a good PIDS. The second important limitation is that we have only two views with very similar settings. This makes it easy for the algorithms (particularly supervised learning based) to learn the scene. A multi-view dataset would have added an additional difficulty in the PID task.

## 1.5 Conclusion

In this chapter, we explored the data acquisition schemes in the surveillance systems, followed by the common challenges and finally the datasets used for the VAD and PID tasks.

Visual camera or color camera is still one of the most used cameras for video surveillance, even though it does not perform well when the scene has low illumination. The infrared camera eliminates this problem of visual camera with infrared lighting, but this lighting has other problems, like it attracts insects and performs poor in rainy condition. Thermal camera relies on the infrared radiation emitted by the objects and hence it does not require an external lighting, but it also has problems like we cannot distinguish an object from its background if their temperature is similar. Since none of the video acquisition device is perfect, often an additional sensor like the depth sensor is used, or multiple
camera types are used as a multi-camera system, but even these configurations have their drawbacks. To conclude, one should use the video acquisition scheme depending on the requirements of the site and concerned task.

We highlighted various challenges for surveillance systems into three categories: concerning capturing the scene, concerning complexity of the scene, and concerning objects of interest. The first category comprises challenges like noisy frames, camera vibrations and gain adjustments. The second category includes challenges like illumination changes, dynamic background, scene perspective and shadows. Finally, the last category includes camouflage and foreground aperture as challenges.

We explored various datasets for the VAD and PID tasks with their difficulties and shortcomings. Concerning VAD task, three datasets are mainly used. Out of these three datasets, UCSD Ped2 and CUHK Avenue are single scene dataset, i.e., they have only one scene for training and testing. Ped2 is the smallest and easiest dataset, having anomalies with distinguishable appearance like bike, skate, etc. Avenue dataset is slightly more difficult as it also have anomalies where the normal object has temporal irregularity, like person throwing a bag, running, etc. Finally, the ShanghaiTech dataset is the biggest and most challenging dataset among the three, with 13 different scenes captured by color cameras. There is a good variety of anomalies in this dataset and contains both appearance and temporally irregular anomalies. The fact that the same model needs to understand 13 different scenes and detect anomalies is challenging. However, none of these datasets are perfect: they do not have a validation set, they lack proper anomaly description and correct test annotations (some normal events like ice-cream eating are present in test set but not the training set, causing ambiguity), they lack diversity and have small amount of anomalies, and they contain only short video clips. In real-life, the VAD system needs to protect a site for a long time (atleast few days or weeks) and must face changing scene dynamics like weather, lighting, etc., therefore longer videos are essential. Ideally, we need a new multi-view dataset of long videos, having appropriate anomaly description and annotations, containing diverse anomalies, and accompanied with a validation set.

For the PID task, we have only one public dataset, i.e., the i-LIDS sterile zone dataset and it has just two camera scenes. It is challenging as intruders enter the scene in various ways (like crawling), and there are distractions caused by animals/insects, along with changing scene dynamics like rain, snow, etc.. There is no official validation set in this dataset. It also lacks intrusion types and has only person as intruder. Furthermore, the two scenes are very similar to each other, they are like mirror images of each other. Therefore, we require more datasets for the PID task, each with a validation set. They should have multiple scenes and diverse intruder types like cars, bikes, etc.
# Chapter 2

## Literature review

This chapter presents a review of existing works for spatio-temporal feature learning in videos, with main focus on video anomaly detection and perimeter intrusion detection task. We first present all necessary background concepts needed for understanding the chapter. We next explore the existing spatio-temporal feature learning approaches, followed by unsupervised approaches. In the next two sections, we review existing methods concerning video anomaly detection and perimeter intrusion detection. Finally, we conclude this chapter with takeaway points along with our chosen direction to tackle these problems.

## Contents

<table><tr><td>2.1</td><td>Introduction</td><td>23</td></tr><tr><td>2.2</td><td>Background</td><td>23</td></tr><tr><td>2.2.1</td><td>Feedforward neural network</td><td>23</td></tr><tr><td>2.2.2</td><td>Convolutional neural networks</td><td>27</td></tr><tr><td>2.2.3</td><td>Recurrent Neural Networks</td><td>30</td></tr><tr><td>2.2.4</td><td>Autoencoder</td><td>32</td></tr><tr><td>2.2.5</td><td>Generative adversarial network</td><td>34</td></tr><tr><td>2.3</td><td>Spatio-temporal feature learning from videos</td><td>34</td></tr><tr><td>2.3.1</td><td>Modeling spatial and temporal dimensions independently, processing them sequentially</td><td>35</td></tr><tr><td>2.3.2</td><td>Two-stream modeling</td><td>36</td></tr><tr><td>2.3.3</td><td>Jointly modeling spatio-temporal dimensions</td><td>37</td></tr><tr><td>2.3.4</td><td>Hybrid approaches</td><td>39</td></tr><tr><td>2.4</td><td>Unsupervised spatio-temporal video understanding</td><td>40</td></tr><tr><td>2.4.1</td><td>Reconstruction models</td><td>40</td></tr><tr><td>2.4.2</td><td>Predictive models</td><td>41</td></tr><tr><td>2.4.3</td><td>Generative models</td><td>41</td></tr><tr><td>2.5</td><td>Video anomaly detection (VAD) methods</td><td>42</td></tr><tr><td>2.5.1</td><td>Types of input</td><td>42</td></tr><tr><td>2.5.2</td><td>Types of tasks</td><td>43</td></tr></table>
<table><tr><td>2.5.3</td><td>Auxiliary components for enhancing VAD</td><td>48</td></tr><tr><td><b>2.6</b></td><td><b>Perimeter intrusion detection (PID) methods</b></td><td><b>53</b></td></tr><tr><td>2.6.1</td><td>Pre-Processing</td><td>54</td></tr><tr><td>2.6.2</td><td>Detection</td><td>55</td></tr><tr><td>2.6.3</td><td>Tracking</td><td>59</td></tr><tr><td>2.6.4</td><td>Joint Detection and Tracking</td><td>59</td></tr><tr><td>2.6.5</td><td>Post-Processing</td><td>59</td></tr><tr><td>2.6.6</td><td>Alarm</td><td>60</td></tr><tr><td>2.6.7</td><td>System Deployment</td><td>60</td></tr><tr><td><b>2.7</b></td><td><b>Conclusion</b></td><td><b>62</b></td></tr></table>
## 2.1 Introduction

In this chapter, we present the literature review of existing works for spatio-temporal feature learning on videos, with particular focus on two unsupervised tasks, i.e., video anomaly detection and perimeter intrusion detection. A video consists of spatial and temporal dimensions. We need to understand these space-time characteristics of the video, in order to have a meaningful interpretation of it, which helps in performing various tasks and applications like video object recognition, segmentation, anomalous behaviour detection, etc. Many works address this, but they use labeled data. However for some tasks, there is no available annotated data, such as for video anomaly detection and perimeter intrusion detection. Therefore, we need methods that can understand the spatio-temporal characteristics of videos without labeled data. In this chapter, we will first present some background concepts. Then, we will explore various ways for space-time feature learning from video, followed by reviewing unsupervised approaches. We will then consider methods of literature concerning video anomaly detection and perimeter intrusion detection respectively.

## 2.2 Background

This section aims to present in a concise manner all necessary background concepts required for understanding the subsequent sections. These concepts are on different types of Artificial Neural Networks (ANNs), which aims at learning patterns within data (Bishop et Nasrabadi, 2006). ANNs originated from various attempts to represent the human brain from a mathematical perspective (McCulloch et Pitts, 1943; Rosenblatt, 1957). The underlying hope was that such representation would provide with pertinent learning capabilities.

The main idea of the neural networks is to construct a model parameterized by weights (from a few hundreds up to a trillion (Fedus et al., 2022)) in order to fit a given target function. The learning of models is done with gradient descent and more specifically by using the backpropagation algorithm (Rumelhart et al., 1986). These models are constructed by stacking layers and various types of layers were proposed to handle different types of inputs. For example, convolutional layers (Fukushima et Miyake, 1982) were introduced to process images. They have a translational invariance property, which is highly useful for tasks such as object detection in images. For sequential data, recurrent layers were introduced (Jordan, 1990). These layers keep in an internal state, the memory of previous inputs, to help future outcomes.

In the following subsections, we begin reviewing the core concepts of neural networks via the standard feedforward network. Then, we explain the convolutional neural network, followed by recurrent neural network, autoencoder and finally the generative adversarial network.

### 2.2.1 Feedforward neural network

ANNs use artificial neurons that were designed to loosely mimic the biological human neuron. A human neuron receives inputs through axons and dendrites and, depending on the intensity of the electrical impulse received, in turn generates an impulse to subsequent neurons. For ANN, such functioning is formalized (McCulloch et Pitts, 1943) using the following equation:
$$f(x, w) = \sigma \left( \sum_{i} w_{i}x_{i} + b \right) \qquad (2.1)$$

Each input signal $x_i$ is multiplied by a dedicated weight $w_i$, the resulting signals are summed along with a bias term and fed to an activation function $\sigma$ that decides if the neuron should activate or not, and propagate the signal to subsequent neurons. Any function can serve as activation function, as long as it is continuous and non-linear. Typical activation functions are described in next subsection.

<figure><img src="image_12.png" /><figcaption>Figure 2.1: Pictorial representation of a feed-forward neural network. Source: (Nielsen, 2017)</figcaption></figure>

The neurons are stacked to make a layer and stacked layers form a neural network model. The first layer is the raw input data (for example, pixel values of an image or video) and subsequent layers, called hidden layers, extract more abstract information of the input until one reaches the final layer, called the output layer. Figure 2.1 depicts a model with four layers. It is a feedforward neural network, where the term feedforward means that the information flows strictly in a forward direction from the input to the output. It is a fully connected network (or dense network) since each neuron on a layer is connected to all the neurons of the preceding layer. In other words, the activation value of a neuron depends exclusively on the activation values of the neurons of the preceding layer. Concretely, the activations of the neurons in the $l^{th}$ layer, i.e., $x^{(l)}$, are a linear combination of the values of the preceding layer $x^{(l-1)}$, passed through a nonlinear function $\sigma$:

$$x^{(l)} = \sigma \left( W^{(l)}x^{(l-1)} + b^{l} \right) \qquad (2.2)$$

where $W^{(l)}$ and $b^{l}$ are the weights matrix and the bias vector respectively. These are the parameters that are tuned during the training stage of the neural network. Usually, $x^{(0)}$ is referred to as input, $x^{(N)}$ (N being the number of layers in a model) as prediction or output and the remaining $x^{(i)}$ ($i = 1, ..., N - 1$) as hidden layers.

### 2.2.1.1 Activation functions

An activation function defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the neural network (Haykin et Network, 2004; Goodfellow et al., 2016). The choice of activation function has a crucial impact on the capability and performance of the neural network, and different activation functions may be
used in different parts of the network. Typically, all hidden layers use the same activation function, while the activation function for the output layer depends upon the type of prediction required. Since neural networks are typically trained using the backpropagation algorithm (refer next subsection) that requires the derivative of prediction error in order to update the weights, therefore activation functions are also typically differentiable.

<figure><img src="image_13.png" /><figcaption>Figure 2.2: Common activation functions.</figcaption></figure>

Some commonly used activation functions are shown in Figure 2.2 and are summarized below:

* **Sigmoid.** The sigmoid or logistic function squashes the input in the range 0 to 1.

$$ \sigma(v) = \frac{1}{1 + e^{-v}} \qquad (2.3) $$

As shown in Figure 2.2, the large values of input incline towards the value 1 in output, while the small values tends closer to 0. This property of sigmoid output can be seen as a probability distribution, and thus it is used in binary classification (with single output neuron) to predict whether a certain example belongs to a fixed pre-defined class or not. However, there are two main issues with this activation function. The first issue is of the vanishing gradient. Since the output of sigmoid saturates (i.e. the curve becomes parallel to x-axis) for a large positive or large negative number, the gradients at these regions is almost zero. When n hidden layers use an activation like the sigmoid function, n small derivatives are multiplied together. Thus, the gradient decreases exponentially as we propagate down to the initial layers. A small gradient means that the weights and biases of the initial layers will not be updated effectively with each training iteration, thus making the learning inefficient. That is why, sigmoid function is not used in hidden layers. The second issue is that the sigmoid outputs are not zero-centered and it is undesirable because it can indirectly introduce zig-zagging dynamics in the gradient updates for the weights.

* **Hyperbolic tangent.** It is also known as the tanh activation and it squashes the input to [-1, +1] range.

$$ \sigma(v) = \frac{e^v - e^{-v}}{e^v + e^{-v}} \qquad (2.4) $$

It is very similar to the sigmoid activation and even has the same S-shape as shown in Figure 2.2. Since the output is centered at zero, we do not have the problem of zig-zagging dynamics in gradient updates like sigmoid. We still have the same issue of vanishing gradient but the range of inputs that do not saturate are larger here.
Like sigmoid, it is not suitable for hidden layers but can be used in output layer for binary classification. Since the output of tanh is symmetric around zero, it leads to faster convergence of the neural network.

* **ReLU.** The rectified linear activation function (ReLU) is a piecewise linear function that outputs the input if it is positive, otherwise, it outputs zero. It is defined as:

$$ \sigma(v) = max(0, v) \qquad (2.5) $$

It is less susceptible to the vanishing gradient problem since the gradient for all inputs greater than 0 is 1. This is why it is one of the most used activation functions and is mostly used in hidden layers. The main issue with this activation function is called the dying ReLU. It occurs when the neuron gets stuck in the negative side and constantly outputs zero. Because gradient of 0 is also 0, it's unlikely for the neuron to ever recover. This happens when the learning rate is too high or negative bias is quite large. To address this issue, Leaky ReLU is proposed, which allows small negative values when the input is less than zero. ReLU or its variants are usually not in the output layer.

* **Softmax.** It is not a classic activation function that can be applied on a single neuron. The softmax function turns a vector of k real values into a vector of k real values that sum to 1. The input values can be positive, negative, zero, but the softmax transforms them into values between 0 and 1, so that they can be interpreted as probabilities. It is used as the activation function in the output layer of neural network models that predict a multinomial probability distribution (Bishop et Nasrabadi, 2006). It can be seen as a generalization of the sigmoid function which was used to represent a probability distribution over a binary variable (Goodfellow et al., 2016). Since softmax converts the input to a normalized probability distribution (between 0 and 1) that sum to 1, it is used in output layer of networks that do multi-class classification. It is not used for hidden layer activations. Furthermore, it should not be used for multi-label classification, i.e., when an example have more that one labels like a dog and a bone. This is because softmax simply cannot produce more than one label with values close to 1. In this case, sigmoid function is used.

We presented above some of the most common activation functions, with particular focus to functions that are later used in this thesis. There are also other activation functions like maxout, gaussian linear unit (GLU), exponential linear unit (ELU), etc. For hidden layers, ReLU activation function is the most used. However, the application and architecture of the neural network often dictates which functions to use, for example in LSTM (a type of recurrent neural network), sigmoid and tanh functions are used as a ReLU would make learning unstable and maybe even impossible. Finally, for binary class classification with single output value, sigmoid function is used in output layer. Similarly, for multi-class classification, softmax function is the preferred activation function in output layer.

### 2.2.1.2 Training

So far we detailed the general architecture of the neural networks along with the various activation functions, we now review the training procedure used to estimate the target function. The aim is to train a model $M: \mathcal{X} \to \mathcal{Y}$ to approximate a unknown target
function, where $\mathcal{X}$ is called the *input space* and $\mathcal{Y}$ the *output space*. The training is done to extract the statistical properties of the inputs in order to maximize the posterior probability $p(y|x)$ of getting the correct $y \in \mathcal{Y}$ given $x \in \mathcal{X}$. This maximization is done through the minimization of an objective function:

$$ \min_{W} \mathbb{E}_{x,y} [ \mathcal{L}(\mathcal{M}(x, W), y) ], \qquad (2.6) $$

where $\mathbb{E}$ is the expectation, $\mathcal{M}$ is the deep network, $W$ represents the weights of $\mathcal{M}$ to be set and $\mathcal{L}$ is the loss function measuring how close $\mathcal{M}(x, W)$ is to the true output. To do this minimization, gradient descent is usually used. The standard way to do such an optimization is Stochastic Gradient Descent (SGD) (Bottou et al., 1991). It computes the average gradient

$$ \Delta W_{k-1} = \frac{1}{|B|} \sum_{i \in B} \Delta W_{k-1} \mathcal{L}(y_i, \mathcal{M}(x_i, W_{k-1})), \qquad (2.7) $$

where $(x_i, y_i)$ is the $i$-th element of the current batch $B$ sampled from the training data, and updates the weights using the backpropagation algorithm (Rumelhart et al., 1986) as:

$$ W_k = W_{k-1} - \eta \Delta W_{k-1}, \qquad (2.8) $$

where $\Delta W_{k-1}$ is the gradient at iteration $k$ and $\eta$ is called the learning rate. For such an optimization method, the speed and convergence towards a minima is highly dependent on the shape of the surface of the objective function with respect to the weights. As shown in Figure 2.3a, the optimization process may bounce back and forth, as the slope is very steep (therefore creating a strong gradient). Multiple update rules have been further developed to prevent such problem: momentum (Qian, 1999) (Figure 2.3b), Adagrad (Duchi et al., 2011), RMSprop (Tieleman et Hinton, 2012), Adam (Kingma et Ba, 2015), Adadelta (Zeiler, 2012), to name the most common.

<figure><img src="image_14.png" /><figcaption>Figure 2.3: SGD optimization with and without momentum. On the left, without momentum the optimization process bounces between the ravine's slopes whereas, on the right, with momentum, optimization is smoother. Source: (Orr et al., 1999).</figcaption></figure>

### 2.2.2 Convolutional neural networks

In the last subsection, we detailed the basic architecture of a neural network as well as the training procedure used to optimize it. However, so far, we have only considered one type of layer, i.e., the dense layer, where each output neuron is connected to each input through a weight. While this layer is functional, it is not best suited for all types of input. For instance, for processing images, each output neuron would be connected to each pixel,
therefore requiring a tremendous amount of weights. These weights would not only be hard to optimize but can also lead to overfitting. Furthermore, the dense layer is not invariant to translations. Translation invariance is a very desirable property for image processing and it refers to the ability to ignore positional shifts, or translations, of the target in the image. For example, a cat is still a cat regardless of whether it appears in the left half or the right half of the image.

To address these problems, layers that operate in a sliding window fashion can be used. As the window slides over the input image, weights are re-used at every target position. Therefore, translated inputs result in feature maps with translated activated neurons. Moreover, as the computations of each neuron within a feature map are independent, one can leverage the parallelism to speed up processing. Layers with such behavior are called convolutional layers.

a) **Convolutional layer** A convolution operation can be mathematically defined as:

$$o(t) = (a * b)(t) = \int a(z)b(t - z)dz \qquad (2.9)$$

where a is input, b is kernel and o is the resultant output obtained by convolving input with kernel. This can be interpreted as a weighted operation applied around the neighborhood of the point t. We can easily extend this convolution operation to multiple dimensions. For image data, we have discrete data as input in the form of pixels. Equation (2.9) can be extended for image data as:

$$O(i, j) = (I * K)(i, j) = (K * I)(i, j) = \sum_{m} \sum_{n} I(i + m, j + n)K(m, n) \qquad (2.10)$$

where I is input image, K is kernel and O is the convolved output at pixel (i, j). This output is also known as the feature map. Since convolution is commutative, we can observe that in the equation (I * K) = (K * I). The region in the input space (here image) that the kernel is looking at the current point is known as the receptive field. Finally, the displacement in the image that the kernel performs after each convolution operation is termed as the stride. Figure 2.4 shows an example of convolution operation applied on an 7 × 7 image using a kernel of size 3 × 3 and stride 1 × 1. The orange region in the image shows the receptive field of the kernel and green unit in feature map shows the corresponding convolved output.

<figure><img src="image_15.png" /><figcaption>Figure 2.4: Convolution of an image I with kernel K using stride 1 × 1.</figcaption></figure>
It is to be noted that various modifications to this classic convolution can be obtained by increasing kernel stride, or padding the input image, or using dilation. The reader can refer to the work of Dumoulin et Visin (2016).

**b) Pooling layer** This layer is also known as the downsampling layer. It is often placed after obtaining the feature map and it summarizes the most pertinent values of it. In other words, convolutional layers detect features, while pooling layers aggregate them. Pooling layers reduce the amount of weights or parameters in the network, thus speeding up the computation and reducing the chance of overfitting the training data. Like convolution, this layer also have a kernel or filter, which performs the pooling operation over the feature map and then this filter is slided with a certain stride. Two most common pooling operations are max and average, i.e., taking maximum and average of the feature map for a given filter size. Figure 2.5 demonstrates these two operations. The feature map and filter size are 4 × 4 and 2 × 2 respectively. The filter is first placed on the top left of feature map (shown with yellow color) to perform the pooling operation (max or average), resulting in value (yellow color) shown in right side of the figure. Then, the filter displaces with stride two to reach the top right (shown in blue color) and performs the pooling operation again. This pooling process continues until the full feature map is covered. Finally, we can observe the overall results of max-pooling and average pooling layers in the figure.

<figure><img src="image_16.png" /><figcaption>Figure 2.5: Examples of max pooling and average pooling operations with filter size and stride as two.</figcaption></figure>

Convolutional neural network (CNN) is a type of neural network that uses convolution in at least one of its layers. However, given a model with a single convolutional layer, the context (receptive field) a given neuron can gather is limited to the size of the kernel. Increasing context information requires to increase the size of the kernel and therefore the computation needed as well as the number of weights. A simple solution to this problem is to stack multiple convolutional layers. By doing so, the receptive field will naturally increase, while keeping the desired translational invariance property. Therefore, typical CNNs are composed of many convolutional layers stacked on top of each other, where each layer is followed by an activation function like a standard ANN. As we advance in the layers, the size of feature map should generally decrease. For example, for image classification task, we need a single output from the image. To do so, generally pooling layers are used after each convolutional layer. Alternatively, strided convolutional layers can also be used as they decrease the feature map size. Finally, at the end of last layer in CNN, often a fully-connected (dense) layer is used and it reduces the final output to desired dimensions (like dimension 1 for image binary classification). The basic mechanism of training a CNN remains the same as for regular neural networks. Like weights of the
feed-forward network, in CNN, the kernel parameters are meant to be learned. The initial weights of an untrained CNN are randomly chosen. Then given the training data and task, a loss function is used to calculate the error. This error is backpropagated to update the kernel parameters. Like the regular ANN, the training is continued until the model converges.

To conclude, CNNs are extremely versatile tools. Thanks to their limited number of weights, they help mitigate over-parameterization problems inherent to dense layers. Furthermore, they are also translation invariant and allow for easy parallelisation by design. As a result, convolutional layers are used nowadays throughout the scientific literature. For instance, 1D-CNNs have been used for automatic speech recognition (Kiranyaz et al., 2021) and 2D-CNNs are the standard for image processing (Karpathy et al., 2014; Goodfellow et al., 2016; Liu et al., 2016; Krizhevsky et al., 2017).

### 2.2.3 Recurrent Neural Networks

For the moment we described networks that takes as input spatial signals such as images. However in this manuscript, we are also interested in working with sequential data since we want to extract information from videos. In this case the input data $x = (x_1, ..., x_t, ..., x_T)$ is a sequence composed of T elements.

By design, the feedforward and convolutional neural networks are not suited to model long sequential data like a time series, speech, or text. Besides, the CNN can only gather information up to the receptive field size. Although such behavior might be sufficient for entries of fixed shape, for entries of varying length, it would be better to manage to implement some form of memory that can gather a potentially unlimited amount of information. For this purpose, there is another neural network variant, called recurrent neural network (RNN) (Jordan, 1990). RNNs employ hidden vector denoted $h$ which is recursively updated at each timestep using the element from the input, and from which the output $y$ is predicted as:

$$h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h) \qquad (2.11)$$

$$y_t = \sigma_y(W_y h_t + b_y) \qquad (2.12)$$

where $\sigma_h$ and $\sigma_y$ are activations function and $W_h, W_y, b_h, b_y, U_h$ are parameters matrices and biases.

<figure><img src="image_17.png" /><figcaption>Figure 2.6: Pictorial representation of a simple RNN and an LSTM. Source: (MathWorks, 2022).</figcaption></figure>
As shown in Figure 2.6a, in an RNN, the information feeds cells through a loop. When it makes a decision, it takes into consideration the current input and also what it has learned from the inputs it received previously. This is also demonstrated from Equations (2.11 and 2.12). The RNN has two inputs, the present and the recent past. This is important because the sequence of data contains crucial information about future information. It allows the previous inputs to affect the current output. Due to this internal memory, they have a particular advantage to form a much deeper understanding of a sequence and its context. Classical RNN suffer from two major problems: exploding and vanish gradient (Le Cun et al., 1997). The first problem refers when algorithm assigns high importance to the weights, without much reason. The second issue occurs when the values of a gradient are too small and the model stops learning.

These problems were solved with the introduction of the Long Short-Term Memory (LSTM) by Hochreiter et Schmidhuber (1997). LSTM is a special kind of RNN, capable of learning long-term dependencies. In standard RNNs, the main unit is composed by a simple structure, such as a single *tanh* layer. However, LSTM introduces a structure based on four gates, which are used to remember important information, to forget irrelevant information and to select which type of information is used in each iteration during the learning. This is illustrated in Figure 2.6b. This mechanism allows the gradient to backpropagate more easily essentially by smoothing out the update of the hidden vector *h* at each timestep by using activation functions. Finally, another way to solve the problems of a standard RNN is to use a Gated Recurrent Unit (GRU) (Chung et al., 2014). The GRU is a simplified version of LSTM with only two gates: update gate and reset gate. The update gate is responsible for determining the amount of previous information that needs to pass along the next state. The reset gate is used from the model to decide how much of the past information is needed to neglect; in short, it decides whether the previous cell state is important or not. Finally, these two gates decide what information should be passed to the output. In comparison to LSTM, GRU have lesser training parameters and therefore uses smaller memory and executes faster. However, LSTM can provide better accuracy in larger datasets. Therefore, the choice of RNN should depend on the task and memory constraints.

While extremely effective, the LSTM or GRU networks are not suited for sequence of images or videos as they only process vectors. Therefore, to handle such data, one must first generate a meaningful vectorial representation and then apply the classical LSTM architecture. This representation has the major drawback of not using the spatial information for temporal processing. In order to correct this issue, Convolutional Long Short-Term Memory (ConvLSTM) networks were introduced by (Shi et al., 2015). This architecture uses the convolutions in both the input-to-state and state-to-state transitions and adds peephole connections that allow the network to look at the cell state to make decisions. Due to both convolutions and recurrent neural network in its design, it models well the spatiotemporal relationships in a video. Similarly, ConvGRU (Ballas et al., 2016) also have a similar functioning.

To conclude, RNNs facilitate the modeling of long-term dependencies through their potentially infinite memory. Comparatively, CNNs are limited to a fixed receptive field, which is cumbersome in case of sequences. However, the main drawback of the RNNs formulation is its limitation for parallel computations. As the previous state needs to be computed to output the next one, it becomes mandatory to run the calculations sequentially. Yet, despite such limitation, RNNs have been used with great success for natural language processing (Lipton et al., 2015; Yonghui et al., 2016), speech recognition
(Graves et al., 2013; Sak et al., 2015), video understanding tasks (Ballas et al., 2016; Dwibedi et al., 2018; Kim et al., 2018a; Li et al., 2018), etc.

### 2.2.4 Autoencoder

The Autoencoder (AE) is a type of feed-forward networks which possesses the auto-association property (Hinton et Zemel, 1993; Wang et al., 2016b). It is an unsupervised learning algorithm and provides an alternative to dimensionality reduction techniques like principal component analysis. It projects the data from a higher dimension to a lower dimension using linear transformations and tries to preserve the important features of the data while removing the non-essential parts. The main goal of this type of networks is to learn how to reconstruct the data from a lower dimensional space representation. Figure 2.7 depicts the classical architecture for AE. It tries to learn an approximation of the identity function, to output $\hat{x}$ that is similar to $x$. The identity function seems a trivial solution for this, but it can be avoided by placing constraints on the network, such as by limiting the number of hidden units.

<figure><img src="image_18.png" /><figcaption>Figure 2.7: Classic autoencoder architecture. Source: (Wikipedia contributors, 2022)</figcaption></figure>

AE can be divided into three parts: encoder, code and decoder. The encoder compresses or downsamples the input into a lower dimensional representation. The space represented by this new dimensionality is often called the latent-space or bottleneck and contains the semantic representation or the code of the input. The decoder intends to reconstruct the input using only the encoding of the input. AEs have generalized the idea of encoder and decoder beyond deterministic functions to stochastic mappings $f_{encoder}(h|x)$ and $g_{decoder}(\hat{x}|h)$. The goal is to minimize $\arg\min_{f,g} \|x - (f \circ g)(x)\|^2$. The traditional AE and PCA are not suitable for images or video frames as they ignore the spatial structure and location of pixels, which is termed as being permutation invariant. Furthermore, it is important to note that when working with images, even with a small size like 100 × 100, these methods introduce large redundancy in network parameters due to dense connections. Therefore, for spatial data of images, we need a new form of autoencoder, which is presented below.
### 2.2.4.1 Convolutional autoencoder

The CNN successfully understands spatial characteristics of images, with many times less parameters than the fully connected networks due to convolution and pooling operations. Inspired from this, these operations are used in the autoencoder to obtain a convolutional autoencoder (CAE) (Masci et al., 2011). CAEs have fewer parameters on account of their kernels being shared across many input locations/pixels. Like CNN, they use convolutional and pooling layers, or strided convolutional layers, for encoding the input image. Similarly, for decoding, the deconvolutional layer or unpooling layer, or both are used. Finally, the mean squared error loss between input and reconstructed images are back-propagated for learning. We present below the deconvolutional and unpooling layers.

#### Deconvolutional layer and unpooling

A deconvolutional layer in a neural network is a layer which is able to obtain a dense map from downsampled and course input (Dumoulin et Visin, 2016). A more appropriate name is the transposed convolutional layer, as the term deconvolution is misleading since deconvolutional layers also perform convolutions. A transposed convolution has a transformation going in the opposite direction of a normal convolution, i.e., from something that has the shape of the output of some convolution to something that has the shape of its input while maintaining a connectivity pattern that is compatible with said convolution. Pooling layers in convolutional networks are required in order to decrease the number of network parameters. Unpooling layers perform the reverse of pooling layers (see Figure 2.8).

<figure><img src="image_19.png" /><figcaption>Figure 2.8: Schematic representation of pooling, unpooling, convolution and deconvolution. Source: (Noh et al., 2015).</figcaption></figure>

The location of the maximum activation in the pooling layer is recorded in switch variables (Noh et al., 2015), in the unpooling layer it is placed back. The output of such an unpooling layer is sparse, as it is an enlarged version of the input map. The deconvolutional layer can produce a dense output map from the unpooling layer output. In summary, convolutional layers map multiple activations in a receptive field to a single activation, deconvolutional layers map one single activation to a field or window of multiple
activations. Convolutional layers learn a filter which map a receptive field to one value, consequently deconvolutional layers learn filters that perform the opposite.

Concerning video data, CAE can be used in each image of the sequence but it will not take into account the temporal dimension. Therefore, Convolutional LSTM based autoencoder (CLSTM-AE) was proposed (Chong et Tay, 2017). It uses first few convolutional layers for encoding, then later in deeper layers, it uses the ConvLSTM layers. Similarly for deconding, it first uses the ConvLSTM layers, followed by deconvolutional layers to get the final video reconstruction. In conclusion, autoencoder is an easy to use, unsupervised learning algorithm, and can be applied to vector, images, videos, etc.

### 2.2.5 Generative adversarial network

A Generative adversarial network (GAN) is a category of generative models, first proposed by Goodfellow et al. (2014), that consists of two sub-networks in competition, a generator and a discriminator network, as shown in Figure 2.9. During the learning phase, the generator try to generate convincing data to fool the discriminator who in turn tries to detect whether the data is real or fake. In this way we obtain two trained networks, one to generate realistic data and the other to distinguish between real data and generated data.

<figure><img src="image_20.png" /><figcaption>Figure 2.9: Illustrating Generative adversarial network (GAN). Source: (Hayes et al., 2017)</figcaption></figure>

The generator network is often an autoencoder which generates some input for the discriminator, given a noisy input. The discriminator is usually a fully connected network that takes the real and generated input and predicts whether the generated input is fake or not. Like last subsection, for image data, convolutional autoencoders are use as generators and CNNs are used as discriminators. Similarly, for sequences of images, ConvLSTM or ConvGRU can be a part of the GAN architecture. Finally, the ability of GANs to generate meaningful representations has gained a major interest in many applications like object detection (Li et al., 2017), data generation (Ehsani et al., 2018), super resolution (Ledig et al., 2017), etc.

## 2.3 Spatio-temporal feature learning from videos

In this section, we present different methods for learning spatio-temporal features from a video stream. Since videos have both spatial and temporal dimensions, we explore in this section, various ways to extract this spatio-temporal information. The methods presented below are supervised as they were the first to be developed for spatio-temporal feature learning and furthermore, they act as a baseline for unsupervised approaches. As
a reminder, in supervised learning we have as input the data with its label for training the neural network. We present below various approaches, based on how they treat the spatial and temporal dimensions of video.

### 2.3.1 Modeling spatial and temporal dimensions independently, processing them sequentially

One of the first works to obtain features from video was to use image-based 2D CNN extractors on each frame of the video and then pooling their predictions across the whole video (Karpathy et al., 2014). It is very convenient to use, but it ignores the temporal structure of the video, e.g., these models potentially cannot distinguish between opening and closing of a gate. Therefore, it is essential to model temporal data in the video stream.

<figure><img src="image_21.png" /><figcaption>Figure 2.10: Illustrative example of 2D CNN for spatial feature extraction and an LSTM to model this spatial information over time, applied to action recognition task.</figcaption></figure>

One way to continue using image-based extractors without compromising temporal information is to first extract spatial feature vector by using a 2D CNN shared over all frames and then modeling the temporal representation of this sequence of spatial features using a recurrent neural network (Donahue et al., 2015; Yue-Hei Ng et al., 2015). Since traditional vanilla RNN suffers from the vanishing gradient problem, most of the existing methods adopt gated RNN architectures like LSTM, to model the long-term temporal dynamics in video sequences (Du et al., 2017; Sun et al., 2017; Perrett et Damen, 2019). Figure 2.10 demonstrates this approach, where first the spatial information is extracted from each video frame using a 2D CNN, then this sequence of spatial information is fed into an LSTM network which temporally relates it, and finally the overall information is used for the action recognition task. Some other notable works are described as follows. Wu et al. (2019a) leveraged two LSTMs operating on coarse-scale and fine-scale CNN features cooperatively. Majd et Safabakhsh (2020) proposed the C²LSTM model which incorporates convolution and cross-correlation operators to learn motion and spatial features while modeling temporal dependencies. To learn the temporal information from both the past and future, some works adopt the Bi-directional LSTM consisting of two independent LSTMs, called the forward and backward LSTM respectively (Ullah et al., 2017; Zhao et Jin, 2020). The forward LSTM is used with past frame sequence like a single LSTM network, while the backward LSTM takes a future frame sequence as input. Finally, the information from both LSTMs are fused together to perform the target task (like action recognition).

This approach of modeling spatio-temporal data also benefited from the introduction of attention mechanisms, whether it is spatial attention (Sharma et al., 2016; Ge et al., 2019; Sudhakaran et al., 2019), temporal attention (Meng et al., 2019; Wu et al., 2019b), or spatio-temporal attention (Li et al., 2018; Liu et al., 2020). Sharma et al.
(2016) designed a multi-layer LSTM model, which recursively outputs attention maps weighting the input features of the next frame to focus on the important spatial features. Sudhakaran *et al.* (2019) introduced a recurrent unit with built-in spatial attention to spatially localize the discriminative information across a video sequence. Li *et al.* (2018) proposed a Video-LSTM incorporating convolutions and motion-based attention into the soft-attention LSTM (Xu *et al.*, 2015b), to better capture both spatial and motion information. Compared to LSTM, Gated Recurrent Unit (GRU) has fewer gates, leading to fewer model parameters. Therefore, many works use GRU instead, for modeling spatio-temporal data in videos (Ballas *et al.*, 2016; Dwibedi *et al.*, 2018; Kim *et al.*, 2018a).

Some drawbacks of the presented approach to model spatio-temporal are as follows. Since this approach is sequential, if the 2D CNN fails or performs poorly, the complete approach will fail. Networks like LSTM are often costly in terms of memory and time. Finally, this approach has difficulties to model fine-grained action since local motion is generally hard to model with it. This is because the recurrent network models sequence of spatial information extracted for the whole frames, thus it captures the propagation of global frame level spatial information over time, thus not paying special attention to local motion.

## 2.3.2 Two-stream modeling

Different from the last approach, in this approach two streams, *i.e.*, appearance (spatial) and motion (temporal) are used independent of each other and later fused to obtain the overall spatio-temporal features. It usually works better than the last approach since it does not process space and time dimensions sequentially. Simonyan et Zisserman (2014) first introduced the two-stream network consisting of two parallel networks, *i.e.*, spatial and temporal network. The spatial network accepts raw video frames while the temporal network gets multi-frame-based optical flows as input. The final score is obtained by fusing scores from both streams. Optical flow provides an effective motion representation in the scene and can effectively remove the non-moving background information (Horn et Schunck, 1981). The proposed network had very high performance on existing benchmarks, while being very efficient to train and test. Figure 2.11 shows the two-stream concept of Simonyan et Zisserman (2014).

<figure><img src="image_22.png" /><figcaption>Figure 2.11: Illustration of Two-stream model for video action recognition proposed by Simonyan et Zisserman (2014).</figcaption></figure>

In recent years, the two-stream modeling has seen big improvements and is being used massively in many video tasks. Regarding neural networks, deeper networks have been introduced for both streams and they improve the performance (Wang *et al.*, 2015b, 2016a). But deeper networks tend to overfit on the video dataset (especially smaller
datasets) (Wang et al., 2015a) and therefore Wang et al. (2015b) introduced a series of good practices, including cross-modality initialization, synchronized batch normalization, corner cropping, data augmentation, large dropout ratio, etc. to prevent it. Since there are two streams, there is a stage called spatio-temporal fusion, where results from both the networks are merged to obtain the final score. The standard way is late fusion, which is a weighted average of final scores from both streams (Simonyan et Zisserman, 2014; Wang et al., 2015b). Many works claim that late fusion is not the optimal solution (Feichtenhofer et al., 2016, 2017). For example, Feichtenhofer et al. (2016) shows that a fusion at an intermediate layer not only improves the performance but also reduces the number of parameters significantly as compared to original work of Simonyan et Zisserman (2014). As video is a temporal sequence of frames, researchers have also explored various recurrent neural networks with two-stream networks like LSTM (Sun et al., 2017; Li et al., 2018), bi-directional LSTM (Ullah et al., 2017), hierarchical multi-granularity LSTM (Li et al., 2016), etc. Li et al. (2018) proposed VideoLSTM, a two-stream network with an LSTM-based spatial attention mechanism and a lightweight motion-based attention mechanism. It improves results on action recognition task and furthermore, the learned attention can be used for action localization as well. Other relevant works are Lattice-LSTM (Sun et al., 2017), ShuttleNet (Shi et al., 2017), FASTER (Zhu et al., 2020), etc. Two-stream networks still cannot capture long-range temporal information. To address this issue, Wang et al. (2016a) proposed a temporal segment network which first divides a video into uniformly distributed segments, then randomly selects one video frame within each segment and feeds them to the network which shares weights for input frames from all the segments. In the end, a segmental consensus is performed with one of the operators like average pooling, max pooling, weighted average etc., to aggregate information from the sampled video frames. Since this network uses the whole video as input and provides video-level prediction, it models long-range temporal structure. Other relevant works are DVOF (Lan et al., 2017), TLE (Diba et al., 2017b), TRN (Zhou et al., 2018), TSM (Lin et al., 2019), etc.

To conclude, the two-stream approach is better than the last approach since it takes care of both spatial and temporal dimensions of the video stream. This approach does not treat the spatial and temporal dimensions jointly but instead relies on an external component to extract the motion information. If the motion information is correctly extracted, the spatial and temporal networks can learn the features of video, which are later fused together to make the final decision. This approach has seen a big evolution since its beginning and is still being largely used in many video tasks. If the joint spatio-temporal information is not very important for an application and one can rely on external detectors for motion information, then this approach provides a good alternative.

### 2.3.3 Jointly modeling spatio-temporal dimensions

A conceptually simple way to understand a video is to consider it as a 3D tensor with two spatial and one temporal dimension. One of the simplest ways to jointly model spatial and temporal dimensions is to extend 2D convolutions into three dimensions. It is called a 3D convolution and it is an extension of 2D convolution which consists of learning space-time kernel filters instead of space kernel filters only. Figure 2.12 demonstrates the difference between 2D and 3D convolutions.
<figure><img src="image_23.png" /><figcaption>Figure 2.12: Comparison of 2D convolution (left image) and 3D convolution (right image) operations. Figure reproduced from (Tran *et al.*, 2015).</figcaption></figure>

A 3D CNN is basically a network like 2D CNN where 2D convolution and pooling operations are replaced by 3D convolution and pooling operations. The seminal work for using 3D CNNs is of Baccouche et al. (2011). While inspiring, the network was not deep enough to show its potential. Tran et al. (2015) extended it to a deeper 3D network, named C3D. C3D follows the modular design and learns the spatio-temporal features from raw videos in an end-to-end learning framework. It showed strong generalization capability and encouraging performance on video action recognition task but was computationally more expensive than its 2D counterpart. Figure 2.13 demonstrates the schema for video action recognition task using 3D CNN. We can observe that the 3D CNN takes a video volume as input and jointly understand the spatio-temporal features (like 2D CNN do for images) to predict the output. Important characteristics of 3D CNNs are that they directly create hierarchical representations of spatio-temporal data and are very powerful in modeling discriminative features. However, the main issue is that they have many more parameters than 2D CNNs because of the additional kernel dimension, and this makes them harder and longer to train.

<figure><img src="image_24.png" /><figcaption>Figure 2.13: Illustration of 3D CNN for video action recognition.</figcaption></figure>

Since 3D CNN needs high memory space and running time (Tran *et al.*, 2015; Qiu *et al.*, 2017), some works propose to factorize 3D convolutions (Xie *et al.*, 2017a; Tran *et al.*, 2018). Specifically, they decompose 3D convolution kernels (e.g., 3 × 3 × 3) into 2D spatial kernels (e.g., 1 × 3 × 3) followed by a 1D temporal kernels (e.g., 3 × 1 × 1), called as (2+1)D kernel. This allows to reduce the number of trainable parameters significantly while still modeling space-time features. Another alternative to reduce the complexity of 3D CNN is to use 2D convolutions in place of some 3D convolutions in the same network. A temporal shift module is proposed by Lin *et al.* (2019), which shifts a part of the channels along the temporal dimension to perform temporal interaction between the features from adjacent frames. Different from these parameter-free temporal shift operations, Sudhakaran *et al.* (2020) introduced a lightweight Gate-Shift Module (GSM), which uses learnable spatial gating blocks for spatial-temporal decomposition of 3D convolutions. S3D (Xie *et al.*, 2018) combines the merits of approaches mentioned above. It replaces the 3D convolutions at the bottom of the network with 2D kernels and factorizes the remaining 3D kernels into (2+1)D kernels. It demonstrates that this kind
of top-heavy network has higher recognition accuracy, along with reduction in model size and training complexity.

The 3D CNN based methods generally perform spatio-temporal processing over fixed time intervals via the window-based 3D convolutional operations, where each convolution operation only attends to relatively short-term context in videos (Baccouche et al., 2011; Ji et al., 2012; Tran et al., 2015). Thus, by design and due to high computational requirements, they are not suitable for long-range spatio-temporal relations in videos. Hence, several approaches focused on modeling these long-term dependencies. Diba et al. (2017a) proposed temporal 3D convolutional networks, where the temporal transition layer models variable temporal convolution kernel depths. It can efficiently capture the appearance and temporal information at short, middle, and long terms. They reinforced this in their subsequent work (Diba et al., 2018), with the introduction of a new block with residual connections, which can model the inter-channel correlations of a 3D CNN with respect to the temporal and spatial features. A long-term temporal convolution framework was proposed by Varol et al. (2017) to model the long-term temporal information in videos. They increase the temporal extents of 3D convolutional layers at the cost of reducing the spatial resolution. Finally, Hussein et al. (2019) proposed Timeception, a multi-scale temporal-only convolutional network to account for large variations and tolerate a variety of temporal extents in complex and long actions.

This approach is a good alternative to the last two approaches. The most important quality of this approach is that it learns jointly the spatio-temporal features of the video. This is an essential point since it depends only the video in hand to learn its feature and does not require external supervision. Even though this approach has seen significant improvement throughout the years, it still remains computationally expensive and cannot treat a large video sequence at once. As a concluding remark, this approach should be used when we want to model the spatio-temporal dimension correctly in the video and when the long range temporal dependencies are not important.

### 2.3.4 Hybrid approaches

To further enhance the spatio-temporal comprehension of videos, several works have investigated using 3D CNN based models with two-stream or multi-stream designs (Carreira et Zisserman, 2017; Wang et al., 2017; Feichtenhofer et al., 2019; Li et al., 2020). Carreira et Zisserman (2017) introduced the two-stream inflated 3D CNN (I3D), made by inflating the convolutional and pooling kernels of a 2D CNN with an additional temporal dimension. Similarly, Wang et al. (2017) integrated a two-stream 3D CNN with an LSTM model to capture the long-range temporal dependencies. To better recognize actions with low inter-class variability, Martin et al. (2020) introduced a new siamese network with two streams using 3D convolutions. SlowFast network (Feichtenhofer et al., 2019) extend the idea of two-stream approaches but without the need of optical flow as input. This 3D CNN based two-stream network have a slow pathway and a fast pathway that operate on frames at low and high frame rates to capture semantic and motion, respectively. At each layer of this network, the features of the fast and slow pathways are fused (by summation or concatenation) to share the motion and semantic information among sub networks. Finally, Li et al. (2020) introduced a two-stream spatio-temporal deformable 3D CNN with attention mechanisms to capture the long-range temporal and long-distance spatial dependencies. Recently, a new neural network, called the Transformer was introduced (Vaswani et al., 2017). It mainly utilizes the self-attention mechanism (Bahdanau et al.,
2015) to extract intrinsic features and shows great potential for extensive use in AI appli-
cations. It was first applied to NLP for machine translation tasks (Vaswani et al., 2017;
Kenton et Toutanova, 2019). Only recently, it has been applied in vision tasks (Doso-
vitskiy et al., 2020; Chen et al., 2020), but it has shown a very strong potential with
performance equal or better than CNN based approaches for various tasks (Zeng et al.,
2020; Khan et al., 2022). Transformers also have various drawbacks like they require very
massive datasets for large-scale training, extremely heavy computational requirements,
and furthermore generalization and robustness of transformers is still an open issue (Han
et al., 2022).

In this section, we studied different types of approaches to learn spatio-temporal fea-
tures from video. We observed that two-stream, 3D CNN and hybrid approaches all rest
viable for effective learning in video. While two-stream approaches require an external
detector for motion and does not treat the spatial and temporal dimensions jointly, the 3D
CNN based approach requires high computational power. Finally, the hybrid approaches
takes best of the both worlds but their drawbacks does accompany these approaches, like
they remain computationally expensive. Finally, the choice of approach depends on the
task at hand and requirements. In this thesis, we want to learn spatio-temporal features
jointly in video and furthemore we want to use minimum supervision possible. For the
applications, i.e., video anomaly detection and perimeter intrusion detection, we do not
require long spatio-temporal information. For example, an intrusion that was detected 10
minutes ago, does not have any direct relation in detecting the current intrusion. There-
fore, we choose the 3D CNN based approach in this thesis. The next section will present
how can we use these approaches in an unsupervised context.

## 2.4 Unsupervised spatio-temporal video understanding

This section presents various approaches for unsupervised learning of spatio-temporal
features from the video stream. To recall, in unsupervised learning, we do not have labels
with data during training the neural network. In the last section, we explored various
approaches to obtain spatio-temporal features. All the approaches, whether 2D CNN
with a recurrent network, two-stream modeling or 3D CNN, can be used in the context of
unsupervised learning. We just need to employ them either in reconstructive, predictive,
or generative models, that are presented below.

2.4.1 Reconstruction models

The main idea here is to reconstruct a given input, in order to learn pertinent features
of the input data. The input data can be a vector, image or a video sequence. In our
case, it is the video sequence. These models include methods such as auto encoders (refer
Section 2.2.4) or sparse coding, that are used to extract different linear and non-linear
representations of appearance (image) or motion (stream) or both, to model meaningful
patterns of unlabeled videos. We can simply extend the idea of CAE for images to
videos and use many of the video feature extraction approaches described in last section.
Hasan et al. (2016) proposed a spatio-temporal stacked autoencoder with a video clip
of ten frames as input. It consists of a stack of 2D convolutions and 2D deconvolutions
for encoding and decoding, respectively. Similarly, Chong et Tay (2017) introduced a
convolutional LSTM based autoencoder, with the same input of ten frames. They used
2D convolutions with LSTM for encoding and LSTM with 2D deconvolutions for decoding.
By using LSTM, they were able to better understand the temporal characteristics of video stream. Similarly, 3D convolutional autoencoder (Zhao *et al.*, 2017) encodes and decodes using 3D convolutions and 3D deconvolutions respectively. In theory, any approach of learning spatio-temporal features from videos described in Section 2.3 can be used with an autoencoder. More works on unsupervised deep learning on videos will be detailed in upcoming sections for video anomaly detection and perimeter intrusion detection tasks.

### 2.4.2 Predictive models

Another approach based on unsupervised deep learning tends to use predictive modeling. It is well-known in time series analysis under auto-regressive models. Different from reconstructive models where the objective is to train a model to reconstruct the input data, predictive models try to predict a current sequence of input using the previous sequences. Concerning video data, the objective is to model output of current frame (or future frame) as a function of past *t* frames. Since it concerns temporal sequence, traditionally recurrent neural networks like LSTM were used (Srivastava *et al.*, 2015; Luo *et al.*, 2017b). Recently, there have also been attempts to perform efficient video prediction using convolutional autoencoder networks. The function of an AE can be determined by considering its output values. When the output values are only the reconstruction of the inputs, the AE is a reconstructive model. When the output values are the values after the input values in the time axis, the model is said to be predictive. Medel et Savakis (2016) introduced a ConvLSTM-based AE where the encoder extracts representations from an input sequence, a first decoder that uses these representations to reconstruct the input sequence, and a second decoder that uses the same representations to predict the next frame. Similarly, Zhao *et al.* (2017) proposed a network made up of an encoder and two decoders, the first for reconstruction and the second for prediction. In this network, 3D convolution layers are used instead of ConvLSTMs, for the learning of spatiotemporal representations. More works will be discussed with VAD as the application in later sections.

### 2.4.3 Generative models

Generative models like Generative Adversarial Networks (GAN), Adversarially trained AutoEncoders (AAE) and Variational Autoencoders (VAE) are used for the purpose of modeling the likelihood of video samples in unlabeled data. These models are also used massively in unsupervised video learning tasks like video anomaly detection (Ravanbakhsh *et al.*, 2017; Kiran *et al.*, 2018). For example, Lee *et al.* (2018) proposed a generative adversarial network, called STAN (spatio-temporal adversarial networks) for this VAD task. They use a video clip as input. The middle frame of the video clip is removed and the resultant clip is fed to a spatio-temporal generator. The generator is a convolutional autoencoder with convolutions, ConvLSTMs and deconvolutions. It generates the missing middle frame, which is added to its position in the video clip. The spatio-temporal discriminator, made using 3D convolution layers, takes the generated video clip and original video clip as input and tries to distinguish the clips. Once the two networks are trained, the detection of abnormal events is done using the losses from both generator and discriminator. More unsupervised generative works are detailed in later sections.

In this section, we presented various ways to unsupervisely learn spatio-temporal features of video. We can observe that all of these approaches are different and none of them is more suitable than others. In fact, they can also be combined together if required,
and can potentially lead to better generalization. All of these approaches have been ex-
tensively developed in the last few years. Choosing the correct approach, depends on
the video task being tackled, its requirements, and the spatio-temporal feature learner
(like a 2D CNN, 3DCNN, etc.) being used. Our chosen 3D CNN based approach trans-
lates to 3D CAE in unsupervised learning. This 3DCAE can be used as a reconstructive
model, predictive model and even as the generator of a generative model. In this thesis,
we explore the first two ways and even combine them. The generative model requires
extra computation on top of the generator via discriminator, therefore we avoided that
approach to be computationally light.

## 2.5 Video anomaly detection (VAD) methods

Given a video stream, this task deals with the detection of anomalies. The anomalies are context dependent and vary from one dataset to another as described in Section 1.4.1. Since anomaly is a rare event and the datasets are usually not annotated. Therefore, most methods use unsupervised deep learning, with or without external supervision from pre-trained detectors. We focus only on these methods. It must be noted that there are other methods which rearrange the current VAD datasets to have weak labels and thus use weakly supervised learning. Some methods use semi-supervised learning by using a portion of labeled data for training. Finally, there are also a few methods that reorganize VAD datasets to use supervised learning. We do not review these kind of methods as they are well out of the scope of this study.

To systematically review the VAD methods, we first explore several types of input
that they can take, followed by different proxy tasks that they can perform and finally
some auxiliary components that they use to enhance their performance.

2.5.1 Types of input

To understand different approaches for video anomaly detection, we must first consider
the different types of input they use. Even though the task is detecting anomalies in
video, the methods can use frames of video, frame patches, objects in frame, objects in
video snippet, etc. as input. We explain below different input types used for VAD.

1. Full Frame (FF): This input means using a single frame as input. This means that the video is read frame by frame and each frame is an input to the VAD method. It can be used with any VAD task but has been predominantly used in the reconstruction task (Ravanbakhsh et al., 2017; Nguyen et Meunier, 2019; Ye et al., 2019). One major issue with using FF as input is that we miss to capture the temporal information of video sequence.

2. Frame patch (FP): This input is made by dividing the video frame into smaller patches. The video is read as frames which are converted into smaller patches and each patch is considered as a separate input. If an anomaly is detected in one of the frame patches, then the frame is labeled as anomalous (Sabokrou et al., 2018). We have similar problem here as we lose the temporal dimension and due to patches, we lose part of the spatial information.

3. Frame object (FO): It refers to the objects in the video frame. A pre-trained object detector is first used to detect all possible objects of interest and then these objects
are used as the input to the method (Ionescu et al., 2019; Georgescu et al., 2021b). Since the input is object, rather than the whole frame, we do not have background noise. The problem is that it is highly biased to the external dataset where the object detector was trained, it assumes that objects of interest causing anomalies are known in advance and finally if the object detector fails, the VAD will fail too.

4. Video clip (VC): It refers to a video snippet with a fix number of consecutive frames. It is one of the most widely used input for VAD task (Hasan et al., 2016; Lee et al., 2018; Dong et al., 2020; Chang et al., 2020; Cho et al., 2022). Since it is a video, we have both spatial and temporal dimensions to exploit. It has been used with all the proxy tasks for VAD. However, it takes more time and resources to process video clips and furthermore, it is comparatively difficult to extract pertinent information since often the anomalies occupy a small spatio-temporal volume in the video clip.

5. Video clip patch (VCP): It corresponds to 3-dimensional patches extracted from a video clip. Unlike frame patch, here we include the temporal dimension into the input. Various methods propose different patch extraction techniques (Tran et Hogg, 2017; Park et al., 2022). Since not the whole video clip is used for creating these 3D patches, it works faster than video clip as input. These patches are generally created on parts of the video clip where foreground objects occur (assuming anomaly is in foreground), therefore they either use moving object detector as a pre-processing, or they use some heuristics to create patches only in certain parts of the video clip, where foreground objects should be present (like Park et al. (2022) exclude a margin of 12.5 percent from the top and bottom in each frame of the video clip).

6. Video clip object (VCO): It refers to a video clip composed of objects detected by an object detector. It is an extension of frame object to video clip. It is being used by many recent methods as it is robust to background noise and have both spatio-temporal dimension like a video clip (Yu et al., 2020; Ouyang et Sanchez, 2021; Liu et al., 2021b; Georgescu et al., 2021a). The problem with this input is same as that of frame object due to its dependence on pre-trained object detector.

### 2.5.2 Types of tasks

Since anomaly is a rare event, there is lack of anomaly examples for training a supervised two-class classifier. Due to this, most works address the VAD task using a proxy task like frame reconstruction, prediction, etc. These tasks are called proxy tasks since they address the main task (VAD is binary classification task) indirectly. We describe below the classification of approaches based on the proxy task that they use.

#### 2.5.2.1 Reconstruction task

This task concerns the reconstruction of an input entity like frame, video clip, object, etc. The aim is to learn normality by trying to reconstruct entities in normal videos (without any anomaly). Since the representations were learned from normal videos, it is assumed that anomalies will be harder to reconstruct and thus they can be separated from correctly reconstructed normal entities. This task is usually performed using a convolutional autoencoder (Masci et al., 2011) or an adversarial network like a GAN (Goodfellow et al., 2014). Autoencoder (AE) encodes the input data into the latent space through an encoder and then reconstructs it using a decoder. The anomaly measure is
<figure><img src="image_25.png" /><figcaption>Figure 2.14: The architecture of the convolutional autoencoder proposed in Hasan *et al.* (2016) for video anomaly detection.</figcaption></figure>

the reconstruction error, which is assumed to be high when anomalies occur since the training of AE is only done on normal videos.

Figure 2.14 presents an example of convolutional autoencoder proposed by Hasan *et al.* (2016), using reconstruction task for video anomaly detection. They use a video clip of 10 frames as input and reconstructs it. They use 2D convolutions and pooling layers for encoding and 2D deconvolutions and unpooling layers, for decoding. The mean squared error (MSE) between input clip and its reconstruction is used as the reconstruction error (anomaly measure). By using only 2D convolutions, the temporal features of the video clip cannot be taken into account. To address this, Chong et Tay (2017) proposed to add convolutional long-short term memory (LSTM) layers in both encoder and decoder. Taking it one step further, Zhao *et al.* (2017) proposed a 3D convolutional autoencoder with 3D convolutions and 3D deconvolutions for VAD. Unlike 2d convolutions plus LSTM, the 3D convolutional autoencoder jointly captures the spatio-temporal features from normal videos. Recently, many works used some AE variant for VAD, using FF, VC or VCO as input (Ye *et al.*, 2019; Ouyang et Sanchez, 2021; Astrid *et al.*, 2021b; Cho *et al.*, 2022).

One of the earliest works for VAD using GANs, with reconstruction as proxy task, was proposed by Ravanbakhsh *et al.* (2017). Two conditional GANs are trained, with input as pairs of frames and noise vectors, which generate corresponding frames of a different modality (raw frames to optical flows and vice versa in the two GANs). The discriminators are asked to classify pairs of input and generated representations of frames as real or fake. Assuming that anomalies are not reconstructed well, they fuse reconstruction errors from both modalities as an anomaly score. Some works use both AEs and GANs for the reconstruction task (Nguyen et Meunier, 2019; Ye *et al.*, 2019). One such work is proposed by Nguyen et Meunier (2019), where they learn a correspondence between common object appearances and their associated motions in a two-stream model. Using an FF as input, they use a single encoder coupled with two decoders, one that predicts motion and another that reconstructs the input frame. This entire network is considered as a generator in a conditional GAN setup, where the discriminator is another network that distinguishes between pairs of input frames and corresponding real/estimated flow fields. For testing, they calculate loss scores at a patch-level. For other VAD works with reconstruction tasks, the reader can refer to (Kiran *et al.*, 2018; Ramachandra *et al.*, 2022). The main drawback of using this task is that sometimes the neural network generalizes too well and even reconstructs the anomalies very well. In that case, the difference between normal and abnormal sample is negligible and thus anomaly cannot be detected.
### 2.5.2.2 Prediction task

This task deals with the prediction of an unknown entity, given a known entity. For example, predicting a future frame, given a video clip with present and previous frames. Concerning VAD, the future frame prediction task is one of the most used tasks (Liu et al., 2018; Lv et al., 2021; Park et al., 2020). Other prediction tasks for VAD include future object prediction (Liu et al., 2021b) or missing bounding box prediction (Georgescu et al., 2021a). The input entities used are VC, VCO and VCP, with VC being the most common (Lee et al., 2018; Dong et al., 2020). Models performing prediction task are also trained only on normal videos, with the assumption that they can precisely predict the entities in normal test sequences but will fail to correctly predict in anomalous test sequences. This requires comprehension of how normal spatio-temporal patterns propagate along the video clip. The anomaly score for this task can be calculated by measuring the difference between real and predicted entities or by calculating the conditional probability of a new observation based on the previous samples (Kiran et al., 2018). Autoencoders, GANs and their combination are typically used for this task.

<figure><img src="image_26.png" /><figcaption>Figure 2.15: The future frame prediction architecture proposed by Liu *et al.* (2018) for video anomaly detection.</figcaption></figure>

Figure 2.15 presents an example of future frame prediction architecture proposed by Liu *et al.* (2018). They propose a GAN, where the encoder is a U-Net style network (Ronneberger *et al.*, 2015) that takes training video clips of length *t* as input and predicts the future frame *t* + 1. The discriminator network checks if the future frame is real (ground truth) or predicted, by minimizing the intensity and gradient loss. Furthermore, they add motion (temporal) constraint by using FlowNet (Dosovitskiy *et al.*, 2015) to estimate pairs of optical flow maps between the frame at *t* and real or predicted frame at *t* + 1. The anomaly score is composed of L1 score between flow maps and intensity plus gradient scores from frame prediction. Some other prominent VAD works that use GANs for prediction task are (Lee *et al.*, 2018; Dong *et al.*, 2020), while works using AE for prediction tasks are (Park *et al.*, 2020; Lv *et al.*, 2021; Le et Kim, 2022; Park *et al.*, 2022). Since prediction task depends on the given input to learn the features in order to predict the unknown entity, it can fail when the input contains only stationary anomalies. For example, for the future frame prediction task, when the input clip contains a stationary car (anomaly), the model is likely to predict this car in future frame too. Thus, the prediction error would be too small to differentiate it from the normality.
### 2.5.2.3 Self-supervised task

This type of task uses self-supervisory signals from the data itself and does not require external annotations. In other words, identify any hidden part of the data from any unhidden part of the data, e.g., predict the missing patch of an image, given the image without patch. This learning paradigm is known as self-supervised learning. In recent years, the self-supervised learning is being used massively for different applications in various domains (Liu et al., 2021a) like natural language processing (NLP), robotics, computer vision: applied to image and video analysis, etc. It is typically used as a pre-training step (pretext task) to enrich a learning module, which is later used for supervised downstream tasks like video classification, detection, etc. (Jing et Tian, 2020). Some well-known self-supervised tasks for video representation learning are video playback rate perception (Yao et al., 2020), video pace or speed prediction (Benaim et al., 2020; Wang et al., 2020), relative speed perception (Chen et al., 2021), video cloze procedure (Luo et al., 2020), etc.

<figure><img src="image_27.png" /><figcaption>Figure 2.16: Comparison of VAD tasks: reconstruction (left), frame prediction (middle) and self-supervised: video event completion (right) (Yu et al., 2020). DNN refers to a deep neural network and video event is the VCO input.</figcaption></figure>

Concerning VAD, there are only two works that use self-supervised tasks. The first work is by Yu et al. (2020), where they propose a new self-supervised VAD task, called video event completion. This task is an adaptation of video cloze procedure task (Luo et al., 2020). In this task, first VCO is extracted from a video clip using a pre-trained object detector and a series of appearance and motion-based operations. The VCO is referred as video event here and contains image patches with objects of interest like human. From each video event, one patch is erased randomly and a DNN (U-Net style autoencoder) predicts it using the rest of incomplete video event, known as appearance completion. The MSE between predicted and actual patch is used as an anomaly score. Similarly, motion completion is also done on the same video events: optical flow for each video patch is extracted using a pre-trained network and then similar pipeline like appearance completion is followed using the same U-Net style autoencoder. Again, MSE is used as an anomaly score here. The final anomaly score is a weighted sum of appearance and motion anomaly scores. Figure 2.16 demonstrates this self-supervised video event completion task along with a comparison with reconstruction and prediction tasks. The second work is proposed by Georgescu et al. (2021a), where they perform multiple self-supervised tasks using the same 3D CNN in a multi-task learning paradigm. They use VCO as input, using YOLOv3 object detector (Redmon et Farhadi, 2018). The proposed tasks are: (i) arrow of time prediction (discriminating forward and backward moving objects), (ii) motion irregularity detection (distinguishing objects captured in consecutive frames versus objects captured in intermittent frames), (iii) middle bounding box prediction (given objects in preceding and succeeding frames), (iv) model distillation: estimating normality-specific
class probabilities by distilling pre-trained classification (ImageNet (Russakovsky et al., 2015)) and detection (MS COCO (Lin et al., 2014)) teachers. Both these works show promising results but they rely heavily on external supervision using object detectors, which inhibits their generalizing capability as discussed in Section 2.5.1.

### 2.5.2.4 Classification task

Despite the fact that video anomaly datasets are imbalanced and that only normal classes are present during training, there are still some methods that use a classifier to detect anomalies. Given only normal videos, they either perform one-class classification (Tran et Hogg, 2017; Sabokrou et al., 2018) or do multi-class classification by adding dummy anomalies (Ionescu et al., 2019; Georgescu et al., 2021b).

<figure><img src="image_28.png" /><figcaption>Figure 2.17: One-class classification VAD proposed by Sabokrou et al. (2018). $\mathcal{R}$ and $\mathcal{D}$ are generator and discriminator modules of the adversarially learned GAN. $X$, $\tilde{X}$, $Z$ and $X'$ refers to input, input with noise, latent space and reconstruction respectively, where target class 1 represents normal (non-anomaly) class.</figcaption></figure>

Figure 2.17 presents an example of one-class classification approach for VAD. The input of this approach are patches from video frames, i.e., FP rather than the whole frames. It adopts a GAN where the discriminator ($\mathcal{D}$) is tasked with distinguishing original image patches from reconstructions of noisy patches obtained from a denoising auto-encoder network ($\mathcal{R}$) which plays the role of generator. Since $\mathcal{R}$ is trained only on image patches from training data, it deteriorates reconstruction of outliers and thus enables $\mathcal{D}$ to distinguish an anomalous image patch from its reconstruction easily.

Concerning VAD with multi-class classification, Ionescu et al. (2019) proposed one of the first approaches. Their input is FO, which is obtained by using a pre-trained single-shot detector (SSD) (Lin et al., 2017) on each frame of the video. They train convolutional auto-encoders on appearance and gradient features of these objects to learn latent representations and then perform k-means clustering followed by training of k one-class SVMs to make binary one-versus-rest classifications. Each cluster represents a different type of normality. At test time, they simply use the inverse of the maximum of k classification scores as an anomaly score, meaning if a sample does not belong to any cluster, it is considered as anomalous. The classification task fails when the anomaly resemble too much with normality in the latent space, thus making it difficult to separate anomalies from normality.
### 2.5.2.5 Multiple task learning

Since no task is perfect for VAD, many approaches use multiple tasks to benefit from advantages of different tasks (Zhao *et al.*, 2017; Tang *et al.*, 2020; Liu *et al.*, 2021b).

<figure><img src="image_29.png" /><figcaption>Figure 2.18: Network architecture proposed by Zhao *et al.* (2017) for reconstructing present frames and predicting future frames.</figcaption></figure>

Figure 2.18 presents the work of Zhao *et al.* (2017) where they combine reconstruction and prediction tasks. They propose an encoder consisting of 3D convolutions (Baccouche *et al.*, 2011) and two decoders (one for each task) consisting of 3D deconvolutions (Zeiler *et al.*, 2010). They train on normal data to reconstruct the current video clip and predict the future frame video clip. During testing, they use only the reconstruction error of the test video clip as the anomaly measure. They demonstrated that using these tasks together boosts the detection of video anomalies. Even though multiple task learning benefits from advantages of different tasks, it is often difficult to combine various tasks and furthermore this makes the model computationally very expensive. So, the tasks must be chosen wisely with taking into consideration the gain in performance (if there is) in relation to the expense of memory and computation time.

## 2.5.3 Auxiliary components for enhancing VAD

To enhance the detection of anomalies, often many auxiliary components are used along with the VAD tasks. We describe below some of the most used components.

### 2.5.3.1 Optical flow

Since moving objects can cause anomalies, it is important to reinforce motion analysis for VAD systems. Many motion analysis tasks employ optical flow as a fundamental basis upon which more semantic interpretation is built (Fortun *et al.*, 2015). Optical flow is defined as the estimation of a dense motion field, corresponding to the displacement of each pixel (Beauchemin et Barron, 1995). Concerning VAD, approaches use optical flow to enrich the input with motion information before feeding it to the deep neural network (Tran et Hogg, 2017; Zhao *et al.*, 2017; Ravanbakhsh *et al.*, 2017; Liu *et al.*, 2018; Nguyen et Meunier, 2019; Dong *et al.*, 2020; Yu *et al.*, 2020; Liu *et al.*, 2021b; Georgescu *et al.*, 2021b).

### 2.5.3.2 Pre-trained feature extractor

Instead of learning features from anomaly datasets directly, many methods use a pre-trained feature extractor trained on an external dataset. Pre-trained extractors can be
used to extract features (of frames / video clips), to extract objects in the video (Ionescu et al., 2019; Ouyang et Sanchez, 2021; Georgescu et al., 2021a; Le et Kim, 2022), or to extract optical flow (Zhao et al., 2017; Ravanbakhsh et al., 2017; Liu et al., 2018; Nguyen et Meunier, 2019; Dong et al., 2020; Yu et al., 2020; Liu et al., 2021b; Georgescu et al., 2021b). The advantage of features extractors for VAD is that they save computational capacity, so bigger architectures can be made, and they can add additional information like optical flow or object information. The main disadvantage is that they are biased towards the external dataset and an incorrect feature/object/optical flow estimation can lead to failure of the whole system.

### 2.5.3.3 Data transformation

Data transformation refers to the modification of input data in various ways so that it helps in better anomaly detection. The transformed data is used in two ways: as an augmentation of data to the original input of the model, or as an auxiliary input along with the original input of the model. Some major works using data transformation in the first way are (Hasan et al., 2016; Chong et Tay, 2017; Zhao et al., 2017). Hasan et al. (2016) and Chong et Tay (2017) increase the size of their input data by generating video clips with different strides (stride-2 and stride-3) between frames. Zhao et al. (2017) instead augments the data by creating video clips with various transformations like random cropping, changing brightness and Gaussian blurring. The main motivation behind this type of data transformation is to train the model with a sufficiently large dataset with different input variations so that the model becomes adaptive and retain the most pertinent features of normal input.

Few important VAD works using data transformation in the second way are (Georgescu et al., 2021a,b; Astrid et al., 2021a,b; Park et al., 2020). Georgescu et al. (2021a) transforms data into various ways for different tasks like reading frames backwards or not for arrow of time detection, or skipping frames in video clip for irregular/regular motion task, etc. Georgescu et al. (2021b) and Astrid et al. (2021a,b) transform data to form pseudo anomalies in order to use normal and pseudo-abnormal data while training.

<figure><img src="image_30.png" /><figcaption>Figure 2.19: Autoencoder based VAD work of Astrid et al. (2021a) with normal input and pseudo anomaly input, where probability $p$ regulates quantity of pseudo anomalous input.</figcaption></figure>

Figure 2.19 shows an example of VAD work that use pseudo anomalies synthesized from normal data as an auxiliary input to the AE. The pseudo anomalies are formed by temporally striding the input to create temporal incoherence in video clip, which simulates
an anomaly. The motivation is to train the AE in order to reduce the reconstruction loss for normal inputs while increasing it for pseudo anomaly inputs, thus making model sensitive to anomalies with temporally incoherent behavior.

### 2.5.3.4 Clustering

It is the task of grouping or segmenting a collection of entities into subsets or “clusters”, such that the entities within each cluster are more closely related to one another than to those assigned to other clusters (Hastie et al., 2009). Concerning VAD, Chang et al. (2020) clusters the encoder output of an AE with reconstruction as proxy task. They propose a deep k-means clustering in encoder to obtain a more compressed data representation and extract the common factors of variation within the normal dataset. By minimizing the distance between the data representation and cluster centers, normal examples are closely mapped to the cluster center while anomalous examples are mapped away from the cluster center. The cluster centers can be deemed as a certain kind of normality within the training dataset. Since the model is only trained on normal events, the distance between cluster and abnormal representations is much higher than between the normal patterns. Reconstruction error and the cluster distance are together used as an anomaly score. With a similar strategy, Ouyang et Sanchez (2021) proposes to cluster the latent manifolds of the autoencoder using expectation maximization. Different from these approaches, Ionescu et al. (2019) first clusters the latent space of an autoencoder into different types of normality, and then train a binary classifier following the one-versus-rest scheme to separate normality clusters from one another. In the inference phase, a test sample is labeled as abnormal if the highest classification score is negative, i.e., the sample is not attributed to any normality cluster.

### 2.5.3.5 Memory, attention and other units

One of the main challenges with CNN based networks like autoencoder or GAN is that they have very powerful representational capacity which can even hinder to distinguish between normality and abnormality, if used naively. For example, reconstruction-based VAD methods using AE may even reconstruct the abnormal frames well (Zong et al., 2018; Liu et al., 2018). Hence, the assumption that reconstruction error is comparatively high for abnormal test frames might be violated.

One popular approach to handle this issue is to use memory units with the network. The prototypical patterns of normal data are recorded into the memory during training, whereas while testing on a test input, the most relevant elements from the memory are retrieved to perform the given task (reconstruction, prediction, etc.). Figure 2.20 shows one such approach, namely MemAE (Gong et al., 2019). MemAE or memory-augmented autoencoder is a reconstruction task-based VAD method, which learns and updates the memory contents during training, to store the prototypical elements of the normal data. In the test phase, the memory is fixed, and reconstruction is performed using items selected from the memory. Taking this work forward, Park et al. (2020) propose a similar strategy to enhance AE for VAD by an improved memory module. They separate stored memory items explicitly using feature compactness and separateness losses, which enables using a small number of memory items compared to MemAE (10 vs 2,000 for MemAE). Furthermore, they also update the memory at test time, while discriminating anomalies simultaneously, suggesting their model memorizes normal patterns of test data. Overall, their model better records diverse and discriminative normal patterns for VAD. Finally, a
recent work by Liu *et al.* (2021b) uses multiple memory units attached in different parts of the AE and they propose an ideal arrangement of these memory units along with skip connections for the VAD task.

<figure><img src="image_31.png" /><figcaption>Figure 2.20: Proposed schema of MemAE (Gong *et al.*, 2019). The encoder takes the input **x** to produce encoding **z**, which is taken as a query by the memory addressing unit to obtain the soft addressing weights **w**. The memory slots can be used to model the whole encoding or the features on one pixel (as shown in the figure). The updated encoding $\hat{z}$ passes via decoder to produce the reconstruction $\hat{x}$.</figcaption></figure>

Another important approach to enhance neural network is to use attention modules (Zagoruyko et al., 2016; Vaswani et al., 2017; Wang et al., 2019). In simple words, attention is a technique that imitates human cognitive attention, enhancing a part of input, such as an object and neglecting the remaining parts (Zhou et al., 2019). Concerning VAD, Le et al. (2022) proposed an attention-based residual autoencoder with future frame prediction as the proxy task. To exploit channel dependency of features, they propose a channel attention module made of two convolutional layers and it is applied in each layer of the decoder. Similarly, Chang et al. (2020) proposed a cluster driven autoencoder with future frame prediction task, where a variance-based attention module is designed to assign an importance to moving part of video clips, thus improving detection of anomalies with large temporal movements like person running, jumping, etc. Different from these approaches, Lv et al. (2021) proposed an AE-based VAD method with a dynamic prototype unit. They learn diverse patterns of the normal data in the form of prototypes. Prototype is a compact representation of pertinent normal data. A novel attention operation on the AE encoding map assigns a normalcy weight to each pixel location to form a normalcy map. Then, prototypes are obtained as an ensemble of the local encoding vectors under the guidance of normalcy weights. Multiple parallel attention operations are applied to generate a pool of prototypes, which represent diverse and compact dynamics of the normal patterns. Finally, the AE encoding map is aggregated with the normalcy encoding reconstructed by prototypes for latter frame prediction.

Recently, some other types of units have been proposed to tackle with the issue of powerful representational capacity of CNN based VAD networks. Szymanowicz *et al.* (2022) proposed a vector quantized autoencoder for detecting video anomalies using the future frame prediction task. They store features of the encoder in a learnable codebook using a vector quantization module. The vector quantization leads to discretization of high-level features into embeddings, which are used by decoder to predict the future frame. Since decoder has no direct access to input features but only the discrete embeddings of
codebook, it is unable to correctly predict the anomalous frames, thus improving the video anomaly detection. Similarly, Cho *et al.* (2022) proposed normal density estimation (NDE) unit which estimate the density of normality in their AE based VAD method.

<table><caption>Table 2.1: Review of major video anomaly detection methods.</caption><thead><tr><th>Method</th><th>Input</th><th>Task</th><th>Principal compo.</th><th>Auxiliary compo.</th></tr></thead><tbody><tr><td>Hasan *et al.* (2016)</td><td>VC</td><td>Recon</td><td>AE</td><td>DT</td></tr><tr><td>Chong et Tay (2017)</td><td>VC</td><td>Recon</td><td>AE, CLSTM</td><td>DT</td></tr><tr><td>Zhao *et al.* (2017)</td><td>VC</td><td>Pred, Recon</td><td>AE</td><td>OF, pre., DT</td></tr><tr><td>Ravanbakhsh *et al.* (2017)</td><td>FF</td><td>Recon</td><td>Adv</td><td>OF, pre.</td></tr><tr><td>Tran et Hogg (2017)</td><td>VCP</td><td>Classification</td><td>AE, SVM</td><td>OF</td></tr><tr><td>Lee *et al.* (2018)</td><td>VC</td><td>Pred</td><td>AE, CLSTM, Adv</td><td><b>X</b></td></tr><tr><td>Sabokrou *et al.* (2018)</td><td>FP</td><td>Classification</td><td>Adv</td><td><b>X</b></td></tr><tr><td>Liu *et al.* (2018)</td><td>VC</td><td>Pred</td><td>AE, Adv</td><td>OF, pre.</td></tr><tr><td>Nguyen et Meunier (2019)</td><td>FF</td><td>Recon</td><td>AE, Adv</td><td>OF, pre.</td></tr><tr><td>Ye *et al.* (2019)</td><td>FF</td><td>Recon</td><td>AE, Adv</td><td><b>X</b></td></tr><tr><td>Gong *et al.* (2019)</td><td>VC</td><td>Recon</td><td>AE</td><td>mem.</td></tr><tr><td>Ionescu *et al.* (2019)</td><td>FO</td><td>Classification</td><td>AE, SVM</td><td>clust., pre.</td></tr><tr><td>Dong *et al.* (2020)</td><td>VC</td><td>Pred</td><td>AE, Adv</td><td>OF, pre.</td></tr><tr><td>Tang *et al.* (2020)</td><td>VC</td><td>Pred, Recon</td><td>AE, Adv</td><td><b>X</b></td></tr><tr><td>Yu *et al.* (2020)</td><td>VCO</td><td>SST</td><td>AE</td><td>OF, pre.</td></tr><tr><td>Chang *et al.* (2020)</td><td>VC</td><td>Recon</td><td>AE</td><td>clust., att.</td></tr><tr><td>Park *et al.* (2020)</td><td>VC</td><td>Pred</td><td>AE</td><td>mem.</td></tr><tr><td>Ouyang et Sanchez (2021)</td><td>VCO</td><td>Recon</td><td>AE, GMM</td><td>pre., clust.</td></tr><tr><td>Liu *et al.* (2021b)</td><td>VCO</td><td>Pred, Recon</td><td>AE</td><td>OF, pre., mem.</td></tr><tr><td>Georgescu *et al.* (2021a)</td><td>VCO</td><td>SST</td><td>AE, FCN</td><td>pre., DT</td></tr><tr><td>Georgescu *et al.* (2021b)</td><td>FO</td><td>Classification</td><td>AE, Adv</td><td>OF, pre., DT</td></tr><tr><td>Astrid *et al.* (2021a)</td><td>VC</td><td>Recon</td><td>AE</td><td>DT</td></tr><tr><td>Astrid *et al.* (2021b)</td><td>VC</td><td>Recon</td><td>AE</td><td>DT</td></tr><tr><td>Lv *et al.* (2021)</td><td>VC</td><td>Pred</td><td>AE</td><td>att.</td></tr><tr><td>Le et Kim (2022)</td><td>VC</td><td>Pred</td><td>AE</td><td>att., pre.</td></tr><tr><td>Cho *et al.* (2022)</td><td>VC</td><td>Recon</td><td>AE</td><td>NDE</td></tr><tr><td>Szymanowicz *et al.* (2022)</td><td>VC</td><td>Pred</td><td>AE</td><td>codebook</td></tr><tr><td>Park *et al.* (2022)</td><td>VCP</td><td>Pred</td><td>AE</td><td>DT</td></tr></tbody></table>

Table 2.1 presents a review summary of various VAD methods. The first column refers
to the publication associated with the method and the second column shows various input data (refer Section 2.5.1). In third column, the proxy tasks are mentioned where Pred, Recon and SST refer to prediction, reconstruction, and self-supervised task, respectively. The fourth column refers to main learning component of the method, where the undeclared terms Adv, CLSTM, GMM and FCN refer to adversial unit, convolutional LSTM, gaussian mixture modeling and fully connected network. Final column is of the auxiliary component, where the abbreviated terms are: DT - data transformation, OF - optical flow, pre. - pre-trained feature extractor, clust. - clustering, mem. - memory unit, att. - attention unit and NDE - normal density estimation. We can observe the following from Table 2.1. Most of the methods use video clip based inputs like VC, VCP and VCO and only a very few methods use the frame based input. This signifies that rather than using just the spatial data through a single frame, spatio-temporal data via video clips is essential for VAD. Regarding proxy tasks, reconstruction task is the most preferred, followed by the prediction task, and they are often combined together. The classification task has been used only four times and it uses patch or object type inputs, requiring pre-processing through an external detector or data transformation. This signifies that it is not easy to perform classification task directly on the raw frame or video clip. The SST task has been recently used for VAD and there are only two methods that use them. Both of them use VCO as input, i.e., they need an external object detector to first detect objects and then perform the task. They obtain excellent performances as they do not suffer from scene background and associated noise. However, one must note that in real life, we cannot always detect all the possible anomalous object and furthermore we might not even know which objects will cause anomaly (unlike offline datasets where we know which objects are potentially anomalous). The most used learning component is the autoencoder (2D CAE, 3D CAE, etc.), but other components like GAN, CLSTM, etc., are also used. The choice of learning component depends on the chosen task, input and architecture design. For auxiliary components, pre-trained detector, optical flow and data transformations are mostly used. These components ensure the input data needed for the methods concern. Rest of the auxiliary components like memory, attention, clustering, codebook, etc., are used to reinforce the learning component. This signifies that most learning components do require this auxiliary help to better detect anomalies. To conclude, reconstruction, prediction, SST, or their combinations should be used for VAD, with minimum or no auxiliary component. Ideally, video clip should be used as input, since it is the spatio-temporal data and does not require any processing or external supervision of any sort, thus it is an unbiased input.

## 2.6 Perimeter intrusion detection (PID) methods

In this section, we review the existing perimeter intrusion detection methods. It must be noted that only a few methods tackle the complete PID task. Unlike VAD, the PID task has not gained enough scientific attention in recent years, even though private enterprises kept on improving their intrusion detection systems, but they did not share their works publicly. One simple evidence to support this claim is that there is only one public dataset available for this task, unlike the VAD with many datasets. Nevertheless, we review the few existing public methods for this task.
<figure><img src="image_32.png" /><figcaption>Figure 2.21: Typical pipeline of a PIDS, composed of an optional offline training (left part) and an online detection (middle part). An illustrative example of different steps in online detection is shown in right part of the figure (best viewed in color).</figcaption></figure>

PID is closely related to other computer vision tasks like motion detection, object
detection, tracking, etc. (Valera et Velastin, 2005). Indeed, these tasks are part of the
typical PIDS pipeline as shown in Fig. 2.21. In order to examine PID methods, we must
review methods that tackled one or several of these tasks with the aim to improve the
PIDS pipeline. We describe below various parts of the pipeline, along with some major
methods. It should be noted that here we do not describe data acquisition schemes since
it has been already explored in Section 1.2.

2.6.1 Pre-Processing

In the case of protecting a perimeter, the cameras are usually CCTV and therefore they are
not necessarily of the highest quality. It is because there are several cameras for each site
and it is economically not viable for most people to invest too much in expensive cameras.
Also, the task is just to detect the presence of an intrusion and not necessarily do profound
analysis like face detection of an intruder (which would require high quality cameras). This
low-quality camera sensor and adverse environmental conditions such as snow, fog, rain,
extreme sunshine, etc., may produce highly noisy video streams. We cannot directly feed
this noisy data into the detection algorithm. Therefore, video enhancement is needed to
remove noise and improve the visual appearance of the video. We can classify the existing
video enhancement methods into two broad categories (Agaian et al., 2007; Rao et Chen,
2012): spatial domain enhancement and transform domain enhancement.

Spatial domain video enhancement deals directly with pixels, i.e., it makes a direct
manipulation of pixels in video frames. It is conceptually simple and has a low time
complexity, which favors real-time implementation but lacks robustness. Some surveys
on this method can be found in (Bennett et McMillan, 2005; Mittal et al., 2006; Rao et al., 2011). In most PIDS, some standard spatial enhancement is carried out on raw frames (Buch et Velastin, 2008, 2014; Cermeño et al., 2018), such as image resizing, image normalization, mean centering, color space conversion (RGB to grayscale or vice versa), histogram equalization, etc.

Contrary to spatial domain, the transform domain video enhancement operates on the transform coefficients of the video frame, such as Fourier transform, discrete wavelet transform and discrete cosine transform (Ali, 2007; Rao et al., 2011). The video quality is enhanced by manipulating these coefficients. This category of methods has a low computational complexity with the ease of manipulating the frequency composition of the video frame. Some major examples of PIDS using these techniques are (Buch et Velastin, 2008, 2014). They first use fast Fourier transform (FFT) on video frame patches and then decrease noise by removing very low or high frequencies from the FFT.

Apart from video enhancement, some other pre-processing can be conducted depend-
ing on the PIDS. Buch et Velastin (2014) first define patches of 16-pixel squares in each
video frame. Then, they designate two regions per frame: grass and fence area for seg-
regating the scene into an authorized and unauthorized zone. Their pre-processing is
demonstrated in Figure 2.22. Similarly, another common pre-processing is to have a fixed
spatial perimeter in each frame of the video (Cermeño et al., 2018; Kim et al., 2018b;
Nayak et al., 2019). This helps the PIDS to focus only on this region of the scene and to
ignore activities outside this perimeter.

<figure><img src="image_33.png" /><figcaption>Figure 2.22: Pre-processing proposed by Buch et Velastin (2014), applied on the i-LIDS dataset. After normalization of video frame, patches of 16 × 16 pixels with 20% overlap are constructed on the region of interest, i.e., ground and fence. These patches are used as input for their PIDS instead of the whole video frame.</figcaption></figure>

2.6.2 Detection

This is one of the most important steps of the pipeline since the goal of a PIDS is to
detect certain categories of objects that might cause an intrusion. Detection in video
can belong to one of the two families: (i) detection of blobs, analyzing the pixel motion,
and (ii) detection of objects, analyzing the image appearance with the localization and
classification of objects. We describe below various detection strategies and associated
PID methods.

### 2.6.2.1 Motion Detection

Intrusion is caused by a moving object in a protected perimeter during an unauthorized time. Therefore, motion detection is essential in a PIDS. The principal approaches can be classified into three categories.

**a) Optical Flow** It literally refers to the displacements of intensity patterns. It is an approximation of image motion defined as the projection of velocities of 3D surface points onto the imaging plane of a visual sensor (Beauchemin et Barron, 1995). Optical-flow-based methods use partial derivatives with respect to the spatial and temporal coordinates to calculate the motion between video frames. One key benefit of using optical flow is that it is robust to multiple and simultaneous cameras and camera shakes. However, most optical flow methods are computationally complex, very sensitive to noise and tough to implement in real-time settings. Some surveys on optical flow approaches are Beauchemin et Barron (1995); Stiller et Konrad (1999). Concerning PIDS, Kim et al. (2018b) implemented an optical flow based detection to compare it with their proposed detection method.

**b) Temporal Differencing** It uses pixel-wise differences among consecutive video frames to extract moving regions. This technique is adaptive to dynamic environments and has a low computational complexity. However, it can fail to extract all of the relevant pixels and can leave holes in regions. Some important studies can be found in (Sehairi et al., 2017; Dimitriou et al., 2017; Tsakanikas et Dagiuklas, 2018; Cermeño et al., 2018). The works of Buch et Velastin (2008, 2014) use simple inter-frame difference followed by some morphological operations for motion detection in their PIDS.

**c) Background Subtraction** This is one of the most popular and key techniques for detecting moving objects in video. Background subtraction detects moving regions by taking the difference between the current frame and the reference frame, often referred to as the 'background model'. The detection ability depends on the adaptiveness of the background model. Some popular background subtraction methods are: running Gaussian average (RGA) (Wren et al., 1997), Gaussian mixture model (GMM) (Stauffer et Grimson, 1999), kernel density estimator (KDE) (Elgammal et al., 2000) and visual background extractor (ViBe) (Barnich et Van Droogenbroeck, 2009). Background subtraction mainly suffers from illumination changes, dynamic background, shadows, camouflage, video noise, etc. (Bouwmans, 2011). These effects can cause a background object to appear as a false foreground object and vice-versa. There are dedicated datasets, like the BMC dataset (Vacavant et al., 2013), for testing these methods. Most comprehensive surveys on background subtraction-based methods are (Hu et al., 2004; Bouwmans, 2011; Xu et al., 2016; Garcia-Garcia et al., 2020).

Concerning PIDS, Buch et Velastin (2008, 2014) use Gaussian background modeling to discriminate people (intruder) from background. The study of Vijverberg et al. (2014) uses background subtraction to extract object blobs from the video frame, which are later used for tracking. The work of Kim et al. (2018b) detects moving objects by comparing a background model with an input video frame in real-time. Figure 2.23 illustrates the
<figure><img src="image_34.png" /><figcaption>Figure 2.23: Moving object detection using background subtraction proposed by Kim *et al.* (2018b). The first column represents frames from a thermal and color camera respectively, with objects of interest in white bounding boxes. The second column demonstrates the output after background subtraction and image binarization, where white regions represent detected moving objects.</figcaption></figure>

performance of their method. Finally, Cermeño *et al.* (2018) detect the potential intruder object using the RGA method.

As already stated, background subtraction also has some drawbacks. In Figure 2.24, we demonstrate this using two examples proposed by Vijverberg *et al.* (2013). The first example, depicted in Figure 2.24a, shows false detections due to snowfall. Since the background subtraction models scene background and detects any moving object, it produces false alarms for these moving snow flakes. These are termed as false alarms due to the dynamic background. In Figure 2.24b, the second example shows split-object detections due to camouflage effect as object appearance matches with background (both have similar white texture).

### 2.6.2.2 Object Detection

In the last decade, object detection, i.e., object localization and classification, has been a field of intensive research (Zou *et al.*, 2019). Intrusion detection is closely related to it as intruders belong to certain categories of objects, such as people, car, bike, etc., to be
detected in a protected area. Even though intrusion can be caused by vehicles, animals, etc., the state of art mostly focuses on detecting people as intruders.

<figure><img src="image_35.png" /><figcaption>(a) False detections due to snowfall in a private dataset. Yellow blobs show detections along with their index number in blue.</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>(b) Object split into extra detections in i-LIDS dataset. Blue blobs show split-object detections along with their index number.</figcaption></figure>

Figure 2.24: Examples of incorrect detections using background subtraction method proposed by Vijverberg et al. (2013).

<figure><img src="image_37.png" /><figcaption>Figure 2.25: Moving object classification architecture proposed by Kim et al. (2018b). Input frame of size 224 × 224 × 3 goes through a series of 2D convolutions, local response normalization and max-pooling units, followed by fully connected layers to finally give predictions of length ten, where each unit depicts probability of a particular class from a list of human (intruder) and wild animal classes.</figcaption></figure>

Object detection methods can be classified into traditional and deep-learning-based detectors (Zou et al., 2019). Some traditional object detectors are the Viola-Jones detector (Viola et Jones, 2001), histograms of oriented gradients (HOG) detector (Dalal et Triggs, 2005) and deformable part-based model (DPM) (Felzenszwalb et al., 2008). Deep learning based methods such as Faster R-CNN (Ren et al., 2015) and YOLO (Redmon et Farhadi, 2017), have achieved remarkable performances in object detection domain. Still
human detection can be challenging, especially in scenes with an atypical human pose, such as crawling or creeping, occluded scenes (Patino et al., 2021) and scenes with low luminosity, such as during night. In regard to PIDS, Nayak et al. (2019) uses a pre-trained YOLO network for intruder object detection. Kim et al. (2018b) first detects moving objects with background modeling and then classifies them into one of the ten classes, containing wild animals and human (intruder). For this object classification, they propose a 2D CNN-based supervised classifier as shown in Figure 2.25.

### 2.6.3 Tracking

It is essential for a PIDS to keep track of a detected intruder object, otherwise it will be re-detected (or not) and may cause unnecessary alarms. Furthermore, perimeter protection solutions may use this information to impose detection constraints. For example, leaving an area can be allowed, but not entering it. We can also think of raising an intrusion alarm only if an object is inside the area for a specified amount of time.

Buch et Velastin (2008) proposed to use the Kalman filter on the texture of objects with a motion mask to build object tracks. Particle filters have also been used to track intruders (Cermeño et al., 2018; Kim et al., 2018b). In (Buch et Velastin, 2014), an intruder is tracked by logging positions of foreground objects over time. The work of (Vijverberg et al., 2013) proposes a tracking algorithm based on tracklet clustering. Finally, Nayak et al. (2019) uses the simple on-line and real-time tracking (SORT) algorithm (Bewley et al., 2016) for tracking intruders.

### 2.6.4 Joint Detection and Tracking

As video stream has two components, spatial and temporal, it is usually analyzed in two steps. The first step captures spatial patterns by using detection on each frame (see Section 2.6.2), whereas the second step uses tracking to apprehend temporal coherence (see Section 2.6.3). This approach creates the hypothesis that spatial and temporal dimensions are independent and can be processed sequentially.

Recent approaches jointly model spatial and temporal dimensions using 3D convolutions and improve results in video analysis (Tran et al., 2015; Jiao et al., 2021). Like anomaly detection, an autoencoder or GAN based on 3D convolutions/deconvolutions can lead to an implicit joint detection and tracking (Zhao et al., 2017; Sun et al., 2020).

### 2.6.5 Post-Processing

A major failure for the PIDS is when it misses to detect intrusions. In fact, ideally a system should always try to detect as much as possible, even at the cost of some false alarms (Cermeño et al., 2018). These false alarms need to be filtered, which is why we need some form of post-processing. Even though this step is crucial in a PIDS, there are few publications on this topic because companies prefer to keep their post-processing confidential. However, despite this, we can list several post-processing techniques.

One of the most common post-processing is to filter objects of interest outside the chosen perimeter (Cermeño et al., 2018; Kim et al., 2018b; Nayak et al., 2019). Sometimes, blobs are inconsistent across time, such as rain drops, and a filter can check the coherence of the blob trajectory. Detected objects can also be filtered with a minimum threshold on the blob size. For example, Cermeño et al. (2018) filters all the objects with a size less
than four pixels. Since foreground objects are bigger than background ones, perspective calibration learns the dimension of object of interest as a function of its position in the scene. This allows to filter objects with a size smaller than the expected size of the object of interest at the same position in the scene (AXIS Communications, 2020; FOXSTREAM Smart Video Analytics, 2021).

### 2.6.6 Alarm

In a perimeter intrusion detection system, it is essential to send an alarm signal to the security personnel to indicate the potential occurrence of an intrusion. This alarm signal alerts the client to verify if there is an intrusion in the site and then to take appropriate action against it. This alarm may also indicate a false alert in case there is no intrusion in the site. It is also very important to have this alarm as soon as the intrusion enters the surveilled place. The reason for this is that if the intrusion has already been inside the perimeter for long time, it would have already done the damage to the protected site. For all these reasons, in a PIDS, it is essential to send alarms at right time intervals (at the beginning of intrusions). The detected and tracked objects are transformed into alarms by using some sensibility thresholds to set the omissions-false alarms trade-off (Buch et Velastin, 2008; Kim et al., 2018b; Nayak et al., 2019; Vijverberg et al., 2014). Generally, these thresholds are manually tuned during the actual deployment of the PIDS. Furthermore, some high-level rules can be applied to trigger the alarm. In the work of Buch et Velastin (2014), the alarm is triggered if the intruder moves towards the target for a minimum time of 2 seconds. In (Kim et al., 2018b; Nayak et al., 2019), the alarm is generated as long as the intruder is inside the protected perimeter. Figure 2.26 demonstrates this for the work of Nayak et al. (2019). Taking this forward, Cermeño et al. (2018) adds an extra constraint that the object must be tracked for at least three frames to give an alarm.

### 2.6.7 System Deployment

The deployment of a perimeter intrusion detection system is realized in three stages: an optional offline model training, then an online initialization of the system, and, finally, online execution.

#### 2.6.7.1 Model Training

There are some PIDS which require an offline training on part of the dataset for their detection or tracking steps (Vijverberg et al., 2014; Kim et al., 2018b; Nayak et al., 2019). This training can be supervised, requiring labeled videos (tagging intrusion frames or events); or, it can be unsupervised, under the assumption that there is no annotated data. The supervised training can also be done on an external dataset and then the trained detectors are used for detection or tracking in PIDS. Vijverberg et al. (2014) proposed a classifier model trained as a multiple instance learning problem by employing image-based features to distinguish intruder objects from moving vegetation and other distractions. The works of (Kim et al., 2018b; Nayak et al., 2019) use supervised object detectors for detection step. So far, we do not have any work which use the unsupervised learning for perimeter intrusion detection.
<figure><img src="image_38.png" /><figcaption>Figure 2.26: Working illustration of the PIDS proposed by Nayak et al. (2019) on CAVIAR (Crowley et al., 2005) (first row) and a private dataset (Nayak et al., 2019) (second row). The first column presents video frames from two different sites with green zones depicting the areas to protect. In the second column, potential intruder objects detected by YOLO detector are in yellow bounding boxes and since some of them are inside the protected zone, the alarm is raised (shown with red zone).</figcaption></figure>

### 2.6.7.2 System Initialization

When a PIDS is installed in a new site, it may need several seconds to set the internal state of the system in alignment with the new scene. For example, to initialize the mean and standard deviation of a GMM (Buch et Velastin, 2014; Cermeño et al., 2018). This is called the system initialization. This online initialization must be well distinguished from the offline model training. Furthermore, a PIDS can have sensibility thresholds, which are manually tuned by the installer during the deployment.

### 2.6.7.3 System Execution

The final stage is the online execution of the PIDS. It includes all the steps of the pipeline as illustrated in the right part of Figure 2.21. To provide a reliable protection, most of the systems work between 5 and 25 frames per second (Buch et Velastin, 2014; Cermeño et al., 2018).

Table 2.2 summarizes major PIDS works with various methods used in different steps of the pipeline. It can be observed that most systems use the visual camera for data
acquisition and only one system is multi-camera based. Most traditional systems use background modeling for detection, whereas recent deep-learning-based models use 2D CNN or YOLO detector. Concerning tracking, the Kalman filter, particle filter, tracklet-based tracking and SORT algorithm are used. For alarms, most systems have their own method-specific rules. Three systems use supervised training and the rest do not have a training step.

<table><caption>Table 2.2: PIDS reviewed in chronological order, where columns represent steps of the pipeline and model training needs. ✗ denotes unavailability of the step, whereas ✓ denotes that the step is available but not detailed.</caption><thead><tr><th>Publication</th><th>Data Acquisition</th><th>Pre-processing</th><th>Detection</th><th>Tracking</th><th>Post-processing</th><th>Alarm</th><th>Model training</th></tr></thead><tbody><tr><td>Buch et Velastin (2008)</td><td>Visual camera</td><td>frame patches, FFT</td><td>inter-frame differencing</td><td>Kalman filter</td><td>✗</td><td>rule-based</td><td>none</td></tr><tr><td>Vijverberg et al. (2013)</td><td>Visual camera</td><td>✓</td><td>background subtraction</td><td>tracklet tracking</td><td>✗</td><td>✗</td><td>none</td></tr><tr><td>Buch et Velastin (2014)</td><td>Visual camera</td><td>frame patches, FFT</td><td>Gaussian background modeling</td><td>Kalman filter</td><td>✗</td><td>rule-based</td><td>none</td></tr><tr><td>Vijverberg et al. (2014)</td><td>Visual camera</td><td>✓</td><td>background subtraction</td><td>tracklet tracking</td><td>✓</td><td>rule-based</td><td>supervised</td></tr><tr><td>Kim et al. (2018b)</td><td>Multi-camera: visual and thermal</td><td>resize, calibration, perimeter</td><td>2D CNN</td><td>particle filter</td><td>outside perimeter</td><td>rule-based</td><td>supervised</td></tr><tr><td>Cermeño et al. (2018)</td><td>Visual camera</td><td>perimeter</td><td>RGA</td><td>particle filter</td><td>object size rule</td><td>rule-based</td><td>none</td></tr><tr><td>Nayak et al. (2019)</td><td>Visual camera</td><td>perimeter</td><td>YOLO v2</td><td>SORT</td><td>✗</td><td>rule-based</td><td>supervised</td></tr></tbody></table>

## 2.7 Conclusion

We showed that concerning spatio-temporal feature learning, there are broadly four approaches. The first type of approach uses 2D CNN for learning spatial features, followed by an RNN-type network for learning temporal features. The second approach has two network streams to learn different and independent features like appearance and motion, which are finally fused together to obtain the overall space-time features. The third approach uses 3D convolution based networks and jointly learns spatio-temporal features. Finally, the fourth approach is a hybrid approach, which include a combination of above approaches. In this thesis, we want to learn spatio-temporal features jointly in video, with minimum supervision possible. Therefore, the 3D convolution-based approaches are ideal for us, with new variants being also fast and low on computational complexity. We do not want to rely on external supervision for optical flow, etc., thus two-stream or hybrid approaches are not suitable.
To move from these supervised spatio-temporal approaches towards unsupervised approaches, three principal modeling approaches can be used, i.e., reconstructive, predictive and generative. Reconstruction models principally use auto-encoder with the aim to reconstruct the input video stream. Predictive models try to predict the future frames or sequences, given observed sequence. They use LSTM-based models and autoencoder variants. Finally, the generative models principally use generative adversarial networks to model the likelihood of video samples in an unlabeled dataset. We found all three approaches are extensively used in the literature and none of them is more suitable than others. The choice of approach depends on the video task being tackled, its requirements, and the spatio-temporal feature learner (like a 2D CNN, 3DCNN, etc.) being used. Our chosen 3D convolution-based approach can be used as 3D CAE in unsupervised learning. It can be used as a reconstructive model, predictive model and even as the generator of a generative model. In this thesis, we explore the first two approaches and also combine them. The generative model requires extra computation on top of the generator, via the discriminator, therefore we avoided that approach to be computationally light.

Concerning video anomaly detection, we found that most methods use video clip based inputs like VC, VCP and VCO as input, thus signifying the importance of spatio-temporal input data. Majority of methods use the reconstruction or prediction as proxy task and they accept any type of input. The classification and SST tasks are used less and they require the help of external object detector or data transformation for obtaining the input to their model. This external supervision is not only biased to the external dataset but also in real life, we cannot detect all the possible anomalous objects beforehand (unlike offline datasets where we know which objects are potentially anomalous). The most used learning component is the autoencoder but the choice of learning component depends on the chosen task, input and architecture design. To better detect anomalies, the learning components often require reinforcement from auxiliary components like memory, attention, clustering, codebook, etc. To conclude, reconstruction, prediction, SST, or their combinations should be used for VAD, with minimum or no auxiliary component. Ideally, video clip should be used as input, since it is the spatio-temporal data and does not require any processing or external supervision of any sort.

Like VAD, the task of perimeter intrusion detection concerns with detection of rare and unknown event, i.e., intrusion. In real life PIDS, we do not have labeled data for these intrusions and thus we need to use an unsupervised approach. Irrespective of this, none of the methods in state of the art uses unsupervised learning. The simple answer to this is the lack of public dataset, i.e., there is only one dataset called i-LIDS (i-LIDS Team, 2006). This dataset was released in 2006, so most traditional methods used non-deep learning based approaches. Furthermore, the i-LIDS dataset provide annotation in training set for developing supervised PIDS. This is potentially because during its launch time (2006), deep learning was not explored much, even less unsupervised deep learning.

In our PIDS review, we found that majority of works use their private datasets and mostly the visual camera is used for data acquisition and only one system is multi-camera based. Most traditional systems use background modeling for detection, whereas recent deep-learning-based models use 2D CNN or YOLO detector. Concerning tracking, the Kalman filter, particle filter, tracklet-based tracking and SORT algorithm are used. For alarms, most systems have their own method-specific rules. Three systems use supervised training and the rest do not have a deep learning based training step. Since no two systems use the same dataset or same part of the dataset and many use different evaluation strategies, it is impossible to quantitatively compare the existing methods. Overall, we
observe that even though there are only human intruders in the i-LIDS dataset, it still has the potential because of various distractions, weather conditions and intruder approach manners. Therefore, still an unsupervised deep learning based approach can be evaluated on this public dataset.

Since in this thesis, we explore for unsupervised deep learning approaches for both VAD and PID tasks. We will use 3D CAE as our initial point, as stated above. In Chapter 4, we propose a 3D CAE for PID with reconstruction as the proxy task. It has an adaptive thresholding mechanism to adapt with changing scene dynamics like weather, light conditions, *etc.*, and it will be discussed later in that chapter. In Chapter 5, we enrich our 3D CAE with different unsupervised and self-supervised tasks. This new model shows promising results for both VAD and PID tasks, without need of external supervision or memory modules.
# Chapter 3

# Defining tasks and their evaluations

In this chapter, we are concerned with formally defining the two tasks that we tackle in this thesis, i.e., video anomaly definition and perimeter intrusion definition. We mathematically describe these two tasks in the first two sections. Then, we explain various evaluation protocols for these tasks. Since the existing protocols are not fully suitable for the PID task, we propose a new edge-level evaluation protocol for it. Next, we describe various evaluation metrics used for these tasks, with a proper explanation concerning which metrics should be used for each of the tasks. Part of this work was published in the special issue “Unusual Behavior Detection Based on Machine Learning” of the Sensors 2022 journal (Lohani et al., 2022b).

## Contents

<table><tr><td>3.1</td><td>Introduction</td><td>66</td></tr><tr><td>3.2</td><td>Defining video anomaly detection</td><td>66</td></tr><tr><td>3.2.1</td><td>VAD with anomaly information</td><td>67</td></tr><tr><td>3.2.2</td><td>VAD without anomaly information</td><td>69</td></tr><tr><td>3.3</td><td>Defining perimeter intrusion detection</td><td>70</td></tr><tr><td>3.3.1</td><td>Object in the Video</td><td>70</td></tr><tr><td>3.3.2</td><td>Intrusion event</td><td>71</td></tr><tr><td>3.3.3</td><td>Intrusion interval</td><td>72</td></tr><tr><td>3.3.4</td><td>PIDS</td><td>73</td></tr><tr><td>3.4</td><td>Evaluation protocols</td><td>73</td></tr><tr><td>3.4.1</td><td>Frame-level evaluation</td><td>74</td></tr><tr><td>3.4.2</td><td>i-LIDS evaluation</td><td>75</td></tr><tr><td>3.4.3</td><td>Edge-level evaluation</td><td>76</td></tr><tr><td>3.5</td><td>Evaluation metrics</td><td>77</td></tr><tr><td>3.5.1</td><td>Threshold dependent metrics</td><td>78</td></tr><tr><td>3.5.2</td><td>Threshold independent metrics</td><td>79</td></tr><tr><td>3.6</td><td>Conclusion</td><td>81</td></tr></table>
## 3.1 Introduction

In this thesis, we are focused principally on video anomaly detection (VAD) and perimeter intrusion detection (PID) task. Since past few years, both these tasks have attained a major attention in the computer vision research (Chandola et al., 2009; Saligrama et al., 2010; Kiran et al., 2018; Aravamuthan et al., 2020; Nayak et al., 2021; Ramachandra et al., 2022). Even after all this attention, their definitions still lack clarity, since they are not defined formally (Vijverberg et al., 2014; Kim et al., 2018b; Kiran et al., 2018; Aravamuthan et al., 2020; Ramachandra et al., 2022). It is essential to mathematically define these tasks as it has a direct impact on the evaluation. For example, when a PID system detects an intrusion, it sends a short video to the security post, where a human operator validates the alarm as a true intrusion or otherwise. This short video, composed of several frames before and after the suspected intrusion, must contain the intruder so that the operator can decide to act on it. The end of the intrusion event is generally not relevant for this application. In practice, we would like to detect intrusion as soon as it occurs; thus, we must obey time constraints. This requires a suitable evaluation protocol that takes these constraints into account. Thus, in this chapter, we first mathematically define the two tasks. Then we describe the existing evaluation protocols and metrics, with a correct explanation of which protocols / metrics to be used for the two tasks. We also propose a new evaluation protocol for the PID task that conforms correctly with its definition and requirements.

## 3.2 Defining video anomaly detection

Video anomaly detection is the task of localizing anomalies in space and/or time in a video. In literature, anomalies are described simply as activities that are out of the ordinary (Chandola et al., 2009; Saligrama et al., 2010; Sodemann et al., 2012). Anomalies are also known as abnormalities, novelties, and outliers among other similar terms. Examples of studied anomalies range from fighting and falling detection, to unattended baggage at airports, to a person loitering outside a building. According to Saligrama et al. (2010), video anomalies can be thought of as the occurrence of unusual appearance or motion attributes or the occurrence of usual appearance or motion attributes in unusual locations or times.

One simple implication of this definition is that video anomalies are context dependent. This signifies that an activity considered anomalous in one context or scene may be normal in another and vice-versa. For example, in one scene, skating is considered a normal activity, while in another, it is anomalous. Therefore, real-world anomalous events are complicated and diverse, and it is difficult to list all the possible anomalous events. It is indeed desirable that the anomaly detection algorithm does not rely on a big prior information about the events, as it can mislead the algorithm to detect/ignore certain types of normal/abnormal events.

Overall, the most fundamental question here is to determine what are the events which can be designated as anomalous. Anomaly definition has always been debated in the literature due to its subjective nature and the complexity of human behaviors (Jiang et al., 2009; Saligrama et al., 2010; Sodemann et al., 2012; Kiran et al., 2018). In general, an event is considered abnormal if it deviates from observed or learned ordinary events (i.e., the event with low occurrence or statistical representation in the learned model) or