the event is not known, or it is outstanding. As aforementioned, another important aspect about the definition of anomalies is the context or environment of the scene. Besides, the difficulty of defining anomalous events increases as the semantics of events grow. In other words, the greater the semantics of an event, more complex situations must be described to determine the anomalous events out of the normal.

To determine whether an event is anomalous, there are two main scenarios: (i) the expected anomalies are known, (ii) the expected anomalies are unknown. In the first scenario, we know some information of anomalies in the data, *i.e.*, we have some sort of labels for anomalies available during training. It is exceptional to have this since anomalies are rarely labeled beforehand. In the second scenario, we just have the data without any anomaly information. It is assumed that this data is only normal, *i.e.*, without anomalies. This is a more plausible setting since anomalies are rare and it is not possible to have all potential normal and abnormal samples available during training. The main challenge here is to understand pertinent features of normal events from a diversity of scenes and contexts, which would lead to distinguish anomalous events from the normal events (Sodemann *et al.*, 2012; Kiran *et al.*, 2018).

Please note that throughout the manuscript anomaly and abnormality denotes the same thing, *i.e.*, they are synonyms for each other. Similarly, their adjective forms anomalous and abnormal are used interchangeably.

We define below the two cases concerning video anomaly detection.

### 3.2.1 VAD with anomaly information

It refers to video anomaly detection when normal examples and at-least some abnormal examples are known. We can also say that in this case we have at-least some supervision via anomalous and normal examples during training.

#### 3.2.1.1 Fully Supervised VAD

Conceptually, the most direct case is fully supervised VAD having well-defined explanation of normality and abnormality, *i.e.*, training data have labels for both normal and abnormal class. This case is particularly common for some VAD sub-tasks like fall detection (Vishwakarma *et al.*, 2007; Kong *et al.*, 2019), fighting detection (Esen *et al.*, 2013) and traffic violation detection (Fu *et al.*, 2005; Zen et Ricci, 2011). Concerning VAD, since most datasets do not have a training set with anomalies, some works try to re-arrange the datasets to add anomalies in the training set and provide their annotations. For example, He *et al.* (2018) re-organized UCSD Ped datasets (Li *et al.*, 2013) and provided annotations. Similarly, Landi *et al.* (2019) and Liu et Ma (2019) provided bounding box annotations in UCF-Crime dataset. Given both normal and abnormal classes during training, a supervised binary classifier can efficiently make a distinction between normal and abnormal videos during testing. Some works on supervised video anomaly detection are (Huo *et al.*, 2012; He *et al.*, 2018; Liu et Ma, 2019; Landi *et al.*, 2019).

Even though fully supervised VAD can produce good results for a dataset, its outcome is not generalizable to other datasets. Furthermore, the big assumption of having labeled data is rarely feasible in real life, especially for mid to large length datasets, which would require extensive manual or semi-automated annotations. Since the portion of abnormal samples is extremely small in comparison to the normal samples (as anomaly is a rare event), there is a huge imbalance of classes in VAD, and this makes it even more difficult for a supervised classifier to perform well (Kim *et al.*, 2020). Moreover, the big diversity
of anomalies disturbs the proper training procedure and practically makes it infeasible to consider all anomaly types. Thus, supervised VAD has a very limited applicability in real-life applications.

### 3.2.1.2 Semi-Supervised VAD

Since anomalies are very diverse and rare, it is implausible and computationally expensive to access the complete range of anomalous and normal events for training. However, using both normal and abnormal samples during training provides a better anomaly understanding to the model. Therefore, some works propose semi-supervised VAD, where they use a very small portion of abnormal samples along with the usual training dataset (with no anomalies) (Akcay et al., 2018; Ruff et al., 2019; Liu et al., 2019).

### 3.2.1.3 Weakly Supervised VAD

Another, more realistic form of supervision in VAD is weak supervision. It comprises of video-level labels (as opposed to video clip or frame-level labels), i.e., a label indicating whether a video is normal or contains anomaly somewhere, but we do not know where (Sultani et al., 2018). UCF-Crime dataset contains these kinds of labels by default (Sultani et al., 2018). Researchers also re-arranged the well-known ShanghaiTech (Luo et al., 2017a) dataset to have weakly labeled anomalous videos during training. It is known as re-organized ShanghaiTech (Zhong et al., 2019) dataset. Some notable works on weakly supervised video anomaly detection are (Sultani et al., 2018; Zhong et al., 2019; Majhi et al., 2021; Feng et al., 2021; Tian et al., 2021; Li et al., 2022).

### 3.2.1.4 Formalism: VAD with anomaly information

We assume $x \in \mathcal{X}$ and $y \in \mathcal{Y}$ respectively as the input and the output of a VAD system. $\mathcal{X}$ and $\mathcal{Y}$ represent respectively the input and output space. In the case of video anomaly detection, $x$ can be either a video clip, a video frame or something else like object in frames, depending on the method (refer Section 2.5.1 for more details), and $y \in \{0, 1\}$ is the associated label where 1 indicates anomalous class and 0 otherwise.

We assume a training set $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N \subset \mathcal{X} \times \mathcal{Y}$ composed of $N$ training samples. The goal is to learn a mapping function that can correctly predict the label $y$ given the input $x$. In VAD with supervision, the anomalies are simply defined by labels given in the labeled dataset.

In general, a discriminative model, represented by function $f$ and composed of parameters $\theta$ is supervisely trained as:

$$ \theta^* = \arg \min_{\theta} \mathbb{E}_{x \sim \mathcal{X}} [ \mathcal{L}_{\text{dis}}(f_{\theta}(x), y) ], \quad (3.1) $$

where $\mathcal{L}_{\text{dis}}$ is a discriminative loss function. This can be easily used for the VAD task with a dedicated loss function.

Like the usual supervised binary classification setup, during testing phase, an input sample $x$ can produce a two-dimensional output (or a single dimensional output) $\hat{y} = f_{\theta}(x)$, where each dimension corresponds to the probability of normal and abnormal classes in $[0, 1]$. The class with higher probability is predicted.
### 3.2.2 VAD without anomaly information

In this case, we have training data with no labels. The main assumption is that the training data has none to very negligible amount of anomalies, which is true since anomalies are rare. In other words, the training data principally have normal events, i.e., only one class. Based on this, a popular paradigm for anomaly detection is one-class classification, i.e., to encode the usual pattern with only normal training samples (Kiran et al., 2018; Ramachandra et al., 2022). Then, the distinctive encoded patterns are detected as anomalies. In this technique, anomalies are detected solely based on intrinsic properties of normal data instances. The main idea is: “Something that has not being seen before is considered anomalous”. Since in unsupervised VAD, we do not have strict assumptions about the training data, it leads to generalizability, especially for applications in which there is not a precise definition for anomalous events (Mohammadi et al., 2021).

#### 3.2.2.1 Formalism

The formalism proposed here is inspired by probably approximately correct (PAC) learning framework (Valiant, 1984). In unsupervised VAD, the training set $\mathcal{D}$ does not contain labels, i.e., $\mathcal{D} = \{x_i\}_{i=1}^N \subset \mathcal{X}$ with $N$ training samples. Usually, it is equivalent to assume that all videos of $\mathcal{D}$ are normal videos, i.e., their labels $y = 0$. Hence, video clips follow a distribution $\mathcal{N}$, usually called normality.

A reconstructive model (Kiran et al., 2018), like an autoencoder, is trained as:

$$ \theta^* = \arg\min_{\theta} \mathbb{E}_{x \sim \mathcal{N}} [ \mathcal{L}_{\text{rec}}(f_{\theta}(x), x) ], \quad (3.2) $$

where $f$ is a reconstruction function composed of parameters $\theta$ and $\mathcal{L}_{\text{rec}}$ is a reconstructive loss function. During testing, a reconstruction error $\hat{y}$ is calculated for an input sample $x$ as:

$$ \hat{y} = R(\mathcal{L}_{\text{rec}}(f_{\theta}(x), x)), \quad (3.3) $$

where $R$ is a common re-scaling function to transform the error into a probability in $[0, 1]$. Then, when the reconstruction error is low, i.e., it is inferior to a threshold $\epsilon \ll 1$, the input sample is classified as normal with a high confidence, denoted $1 - \delta$ where $\delta \ll 1$:

$$ p(x \sim \mathcal{N} | \hat{y} \le \epsilon) \ge 1 - \delta. \quad (3.4) $$

Conversely, when the reconstruction error is high, i.e., it is superior to the threshold $1 - \epsilon$, the input sample is classified as abnormal with a high confidence:

$$ p(x \sim \mathcal{A} | \hat{y} \ge 1 - \epsilon) \ge 1 - \delta, \quad (3.5) $$

where $\mathcal{A}$ is the distribution followed by abnormal videos.

Finally, we can define abnormality as the complement of the distribution of training set: $\mathcal{A} = \mathcal{X} \setminus \mathcal{N}$. It must be well noted that this representation takes care of complicated or debated cases for VAD using the uncertainty parameter $\delta \ll 1$. It denotes in Equations (3.4 and 3.5) that we can never be completely sure that if the reconstruction error is small, it is normal and if it is large, it is an anomaly. It other words, we cannot be sure that all representations far from distribution followed by training data should be anomaly. For example, in ShanghaiTech dataset, the normal activities like picking up stones and eating ice-creams are present only in the testing set and not the training set. Therefore,
even if the reconstruction error is high for these activities, we cannot be fully certain that they are anomaly due to the uncertainty parameter $\delta$. Like reconstructive models, this definition of anomalies can be easily extended to generative models also.

## 3.3 Defining perimeter intrusion detection

In simple words, the task of perimeter intrusion detection (PID) aims to detect the presence of an unauthorized moving object in a protected site during a certain time. As described before, PID is a specific type of VAD task. Furthermore, intrusions are a particular type of anomalies, classified as point and contextual anomalies (Chandola et al., 2009). But contrary to VAD, we can more concretely define this task as it has certain unique attributes like perimeter, intruder object type, movement constraints and site protection time.

The PID task has been defined in various ways in the state-of-the-art. Vijverberg et al. (2014) defined a perimeter intrusion detection system (PIDS) as a monitoring system that identifies the presence of humans or devices in a pre-defined field of view. In (Kim et al., 2018b), it is defined as a system that detects physical intrusions on a site having a protective barrier to isolate it from outside. Aravamuthan et al. (2020) described it as a system that detects the movements of intruders attempting to breach a security wall or region and alert security. However, all these definitions lack clarity and formalization; for example, the following questions need to be addressed: “what are intruders?”, “does moving intruder cause intrusion?” and “is a perimeter necessary?”. To answer all these questions concretely, we must mathematically define a PIDS. Before defining a PIDS, we need a definition of intrusion. Since objects cause intrusion, we first define an object in the video.

### 3.3.1 Object in the Video

We define a video $V$ acquired for $n$ frames during the interval $T = [1, n]$ as:

$$V = \{I_t \in \mathbb{R}^{H \times W \times D} \}_{t \in T}, \qquad (3.6)$$

where $I_t$ denotes the frame at the time instant $t$, with height $H$, width $W$ and number of channels $D$. To define an object in the video, we must first specify the object definition at frame-level. An object in a frame or image is defined with a spatial specification and a class that distinguishes one family of objects from another (such as humans, animals, or cars). The spatial specification can be either on pixel-level by allocating each pixel to an object or background, or on area-level by encapsulating the object in a bounding box. We choose the bounding box specification as it has been used in the literature extensively (Zou et al., 2019). It should be noted that the choice of spatial specification (bounding box or otherwise) cannot have an impact on the intrusion definition. Thus, we define an object at frame-level with a class and a bounding box. To define an object in the video, we consider all of the frames where it is present. Therefore, an object $o_i$ in the video is defined as:

$$o_i = (\{b_{i,t}\}_{t \in T}, c_i \in \mathcal{C}), \qquad (3.7)$$

where $c_i$ is the class of the object from the set of object classes $\mathcal{C}$, and $b_{i,t}$ is its bounding box at time instant $t$, which is defined as:

$$b_{i,t} = \{g_{i,t}, w_{i,t}, h_{i,t}\}, \qquad (3.8)$$
where $g_{i,t}$, $w_{i,t}$ and $h_{i,t}$ are the center, width and height of the bounding box, respectively. The center is defined by its coordinates as $g_{i,t} = (x_{i,t}, y_{i,t}) \in I_t$. Note that, instead of the bounding box center, it is also possible to choose other points, such as the bounding box bottom, as reference. We illustrate these definitions in Figure 3.1.

<figure><img src="image_39.png" /><figcaption>Figure 3.1: Video with $n$ frames of height $H$, width $W$ and channels $D=1$. Two objects, shown in orange and green bounding boxes, are defined as $o_1 = (\{b_{1,1}, b_{1,2}, b_{1,3}\}, c_1)$ and $o_2 = (\{b_{2,3}\}, c_2)$, where $c_1, c_2 \in C$ are the object classes. Here, $\{b_{1,1}, b_{1,2}, b_{1,3}\}$ are bounding boxes of object 1 on first three frames and $\{b_{2,3}\}$ represents bounding box of object 2 at frame 3.</figcaption></figure>

### 3.3.2 Intrusion event

For the protection of a site, some parameters must be defined to qualify objects as non-authorized (na), i.e., intruders.

* $S_{na} \subseteq \mathbb{R}^{H \times W}$: the subset of the frame/image, defining the surface to protect.

* $\mathcal{T}_{na} \subseteq \mathcal{T}$: the time interval during which the surface must be protected (e.g., protection only during night).

* $C_{na} \subseteq C$: the set of non-authorized classes, such as person, car, truck, etc. These classes of objects are considered as possible intruders and can be different from one site to another.

Since $C_{na}$ is a non-finite set (it is impossible to make the exhaustive list of non-authorized objects, exposing the system to omissions), it is easier to explicitly define a short list of authorized objects $C_a$ (such as small animals), which leads to $C_{na} = C \setminus C_a$.

An object causes an intrusion event if it belongs to a non-authorized class and is moving in a protected area during a prohibited time interval. Our hypothesis is that the object should be moving and not stationary to cause an intrusion event. It is to tackle the case where the unauthorized object enters and stays for a long period of time in the site. For example, if a car enters the site, it causes an intrusion event (assuming car belongs to non-authorized class) but later if the car stays stationary in the site for many days, it is not necessarily causing an intrusion event. We are interested in knowing if the intruder has entered the site, thus our hypothesis of moving object. We define the intrusion event caused by an object $o_i$ as:

$$ \mathcal{IE}(o_i) = \{ I_t \text{ s.t. } c_i \in C_{na} \text{ and } t \in \mathcal{T}_{na} \text{ and } \|\text{grad } \vec{g}_{i,t}\| > 0 \text{ and } g_{i,t} \in S_{na} \}_{t \in \mathcal{T}} , \quad (3.9) $$

where $\|\text{grad } \vec{g}_{i,t}\|$ is the gradient of object $o_i$ at instant $t$ and it being non-zero signifies that the object is in motion. Thus, the intrusion event caused by object $o_i$ is a collection
of all the frames $I_t$ such that $t \in \mathcal{T}_{na}$, object class $c_i \in \mathcal{C}_{na}$, the gradient is non-zero and the bounding box center lies in the protected area.

Figure 3.2 illustrates the intrusion event caused by an object. The surface to protect $S_{na}$ is depicted with a yellow trapezoid in each frame, and we assume that we want to protect it during the entire video. One object is present in the video, and it is shown with a rectangular bounding box plus a center. The object is in motion from the second frame until the eighth frame. While in motion, the center of object lies in $S_{na}$ from fourth to seventh frame, causing an intrusion event. Thus, this object triggers an intrusion event for four frames.

<figure><img src="image_40.png" /><figcaption>Figure 3.2: Illustration of an intrusion event caused by a single object. Video with n frames, where $S_{na}$ is shown with yellow surface and object with a green bounding box plus center. The object causes an intrusion event for four frames from frame $I_4$ to $I_7$, colored in red.</figcaption></figure>

Since a video can have more than one object causing intrusion events, we define the intrusion events of the whole video containing j objects as:

$$ \mathcal{IE}(V) = \bigcup_{i=1}^{j} \mathcal{IE}(o_i). \qquad (3.10) $$

### 3.3.3 Intrusion interval

Figure 3.3 shows three intrusion events caused by three objects in the video. We can observe that intrusion events of objects 1 and 2 overlap in time for two frames. It means that for those two frames, there were two objects causing intrusion events simultaneously. The intrusion events of this video is a collection of all the intrusion frames, marked by 1 (see Figure 3.3). In the context of video surveillance, we are concerned with whether there is an intrusion event, regardless of whether one object or many objects are causing it. Therefore, we are interested in an interval of a contiguous sequence of intrusion frames. We term this as an intrusion interval, and the task of intrusion detection is focused on detecting them. Formally, an intrusion interval $II \subseteq \mathcal{IE}(V)$ is defined on a closed interval as:

$$ II = \{ I_t \in \mathcal{IE}(V) \text{ with } t \in [t_{start}, t_{end}] \text{ s.t. } I_{t_{start}-1} \notin \mathcal{IE}(V) \text{ and } I_{t_{end}+1} \notin \mathcal{IE}(V) \}, (3.11) $$

where $t_{start}$ and $t_{end}$ denote the first and last frames of an intrusion interval. In other words, an intrusion interval is a contiguous sequence of frames of maximal size derived from $\mathcal{IE}(V)$. Figure 3.3 depicts two intrusion intervals of the video.
<figure><img src="image_41.png" /><figcaption>Figure 3.3: Illustration of intrusion events of the video and intrusion intervals. Objects $o_1$, $o_2$ and $o_3$ cause intrusion events $\mathcal{IE}(o_1)$, $\mathcal{IE}(o_2)$ and $\mathcal{IE}(o_3)$, marked with value 1. $\mathcal{IE}(V)$ is collection of all the frames with value 1. Two intrusion intervals $II_1$ and $II_2$ are shown in red intervals.</figcaption></figure>

### 3.3.4 PIDS

Given a precise definition of intrusion, we can now define a perimeter intrusion detection system (PIDS). Given a video $V$ and intrusion parameters $(S_{na}, T_{na}, C_{na})$, the prediction of a PIDS can be defined as:

$$ \mathcal{P}(V, S_{na}, T_{na}, C_{na}) = \{ \hat{p}_t \in \{0, 1\} \}_{t \in \mathcal{T}_{na}}, \qquad (3.12) $$

where $\hat{p}_t$ is a binary prediction for each frame $t$ of video $V$ for time $T_{na}$, with 1 denoting a frame predicted as an intrusion, and 0 otherwise. Therefore, a PIDS classifies each frame into an intrusion frame or otherwise. This type of output is useful when we want to evaluate a PIDS at frame-level (refer Section 3.4.1). In a real-life surveillance system, the system sends an alarm signal to surveillance personnel as soon as there is a transition from a normal to intrusion state (i-LIDS Team, 2006). The output of the PIDS $\mathcal{O}(\mathcal{P})$ can be derived from $\mathcal{P}$ as follows:

$$ \mathcal{O}(\mathcal{P}) = \{ \hat{p}_t \in \mathcal{P} \text{ s.t. } \hat{p}_{t-1} \in \mathcal{P} \text{ with } \hat{p}_{t-1} = 0 \text{ and } \hat{p}_t = 1 \}_{t \in \mathcal{T}_{na}}. \qquad (3.13) $$

This is a set of intrusion alarms created by the system, marked by the rising edge, i.e., the transition of the system state from non-alarm to alarm. These alarms alert the surveillance personnel about a suspicious activity. For each alarm, a mini clip is sent containing some frames before the alarm and some frames after the alarm. The surveillance personnel visually analyze this mini-clip and decide whether it is an actual intrusion activity or a false alarm. Therefore, we need an evaluation scheme that takes into account this real-life constraint.

## 3.4 Evaluation protocols

Given a video or set of videos and the requested task, the system can detect intrusions or anomalies. To evaluate the performance, we need to compare the predicted output with ground truth annotations. The way this evaluation is carried out is essential as each evaluation scheme quantifies different aspects of the system and depending on the task, one should wisely choose the correct evaluation methodology. The following subsections present different evaluation protocols. It must be noted that only frame-level evaluation is used in both the tasks and other evaluation protocols are only used for the PID task.
### 3.4.1 Frame-level evaluation

As the name suggests, in this type of evaluation, we are interested in checking whether each frame of the video is correctly classified or not.

#### 3.4.1.1 VAD task

Irrespective of the VAD types and formalisms defined in Section 3.2, every system outputs whether a frame is normal or abnormal. Since per-frame anomaly ground truth is normally available in VAD datasets (refer Section 1.4.1), the predictions can be easily compared with it. It is simply the binary classification evaluation of each frame of the video (Sokolova et al., 2006). First, the elements related to the confusion matrix, i.e., true positive (TP), false negative (FN), false positive (FP) and true negative (TN), are computed with anomaly as the objective class. Then, the performance of VAD is evaluated using the metrics, such as the precision, recall, AUROC, AUPR, etc., as defined in Section 3.5.

In frame level evaluation, the anomalies are evaluated at the lowest level, i.e., frames. This means that here we lose the notion of anomalous events. For example, if we have a good frame level score, it does not necessarily mean that all anomalous events are well detected and maybe a few long anomalous events are contributing a large portion of the overall score while many small anomalous events are missed or incorrectly detected. In short, a good frame level anomaly score just gives an overall idea for a VAD performance and does not guarantee if most anomalous events are correctly detected. Even after these drawbacks, the VAD community still mostly use the frame-level evaluation protocol (Popoola et Wang, 2012; Kiran et al., 2018; Nayak et al., 2021; Ramachandra et al., 2022). In this work, we continue with this evaluation so that we can fairly compare with other VAD works.

#### 3.4.1.2 PID task

For a video $V$ and given intrusion parameters ($S_{na}$, $T_{na}$, $C_{na}$), the frame-level ground truth is defined as:

$$ \mathcal{G}(V, S_{na}, T_{na}, C_{na}) = \{ p_t \in \{0, 1\} \text{ s.t. } p_t = 1 \text{ if } I_t \in \mathcal{IE}(V) \}_{t \in T_{na}} \quad (3.14) $$

where $p_t$ is the ground truth label for each frame $t$ of the video $V$ at time $T_{na}$; value 1 denotes an intrusion class, and 0 otherwise.

Given ground truth $\mathcal{G}$ and prediction $\mathcal{P}$ (see Equation (3.12)), the frame-level intrusion evaluation is computed like frame-level evaluation of VAD task by computing confusion matrix elements and using metrics like precision, recall, $F_1$ score, etc.

In this type of evaluation, each frame contributes equally to the overall score. Thus, it can provide the same overall score for an algorithm that fails to detect multiple intrusion events altogether, i.e., detecting no frames in those events, versus an algorithm that detects all intrusion events but does not detect some intrusion frames in multiple intrusion events. This is an undesirable evaluation in the case of intrusion detection because we cannot afford to have omissions of intrusion events. In reality, we are more interested in knowing if the system is able to detect the intrusion events correctly as a whole. This demands an event-level evaluation. In other words, we want to detect intrusion intervals (IIs) from the video. More specifically, we are interested in evaluating whether the beginnings of these intrusion intervals are detected correctly. This is because, if an intrusion event is detected too late, then that detection is not very useful (i-LIDS Team,
2006; Buch et Velastin, 2014). The idea is to detect each intrusion interval as soon as it occurs and, thus, we need an evaluation scheme that takes this into account.

### 3.4.2 i-LIDS evaluation

To correctly evaluate PID systems, the i-LIDS dataset is provided with an evaluation protocol (i-LIDS Team, 2006). It focuses on evaluating intrusion at the event-level rather than frame-level. To be precise, an intrusion is considered correctly detected if there is at least one system alarm within 10 seconds from the start of the intrusion event.
For an *II* of the video and alarms *O(P)*, the rules of the i-LIDS evaluation protocol are as follows:

1. TP: if there is at least one alarm within 10 seconds from the beginning of the *II*. If there are multiple alarm candidates, the first one is taken, and the rest are ignored.

2. FN: if there is no alarm within 10 seconds from the beginning of the *II*.

3. FP: if there is an alarm but not within 10 seconds from the beginning of the *II*. If there are consecutive FPs within a 5-seconds gap among them, only the first one is considered and the rest are ignored.

Apart from these, one rule is specific to the i-LIDS-dataset: all *II*s and alarms that start within 5 minutes from the beginning of the video are ignored. This means that they wanted to give a preparation time to the system. This evaluation scheme is not generic and has several drawbacks, as illustrated in Figure 3.4. It penalizes an alarm as an FP after 10 seconds from the beginning of an *II* without considering the duration of intrusion. If the *II* has a long duration (such as an hour) and we have an alarm at the 11th second, it is not ideal to mark it as an FP. From a practical point of view, the surveillance personnel will receive a mini clip as soon as the alarm is triggered and, if the intrusion is present, then it is not sensible to mark this as an FP. Instead, this alarm should be ignored as it is not detected within 10 seconds. Similarly, each alarm after 10 seconds but within *II* is considered as an FP, and this strongly penalizes the system precision. Instead, these extra alarms should be counted without assigning them as an FP.

<figure><img src="image_42.png" /><figcaption>Figure 3.4: Illustration of i-LIDS evaluation protocol, highlighting its drawback on an intrusion example starting at 9th second. Since no alarm has been raised in the first 10 seconds of the intrusion, a false negative (FN) is counted. Following alarms, at 22nd and 37th second, are marked as false positive (FP) because they do not occur within 10 seconds from the beginning of the intrusion.</figcaption></figure>
### 3.4.3 Edge-level evaluation

To appropriately evaluate a PIDS while considering the real-world aspects, we propose a new evaluation protocol. An intrusion event begins with a transition from a non-intrusion to intrusion state, i.e., we have a rising edge as shown in Figure 3.5. Similarly, an intrusion event stops by a reverse transition, i.e., a falling edge. We are interested in detecting intrusion within a few frames from the rising edge. Since we focus on this rising edge, we call this the edge-level evaluation. In other words, we emphasize detecting the beginning of intrusion intervals. We first define the following terms from an intrusion interval of the video (see Figure 3.5).

The intrusion interval neighborhood IN is an expanded interval defined by $m_{\text{pre}}$ frames before and $m_{\text{post}}$ frames after the *II*:

$$ \text{IN}(\text{II}, m_{\text{pre}}, m_{\text{post}}) = [t_{\text{pre}}, t_{\text{post}}] \text{ s.t. } t_{\text{pre}} = t_{\text{start}}(\text{II}) - m_{\text{pre}} \text{ and } t_{\text{post}} = t_{\text{end}}(\text{II}) + m_{\text{post}}. $$

These $m_{\text{pre}}$ and $m_{\text{post}}$ frames are in the range of one to five (less than 1/5 seconds for a video at 25 FPS) and are added in order to take into account the error of annotation. This error is because it is difficult to mark the exact frame at which the intrusion starts or ends. This tolerance further permits not strictly penalizing the system when an intrusion event is detected a few frames before the actual event or when the system detects a few more intrusion frames after the actual event is finished. These cases arise often when the intrusion object is in the scene but not inside the surface to protect. Therefore, IN is an interval where the actual intrusion activity takes place, and an alarm given by a PIDS in this interval can be counted as either TP or ignored. An alarm given outside IN must be a false alarm and should be counted as an FP.

<figure><img src="image_43.png" /><figcaption>Figure 3.5: The top subfigure is an illustration of the definitions of edge-level evaluation terms on ground truth, with time in abscissa and non-intrusion (0) and intrusion (1) class for each frame of the video in ordinate. The next two subfigures represent examples of alarms and possible outcomes (TP, FP, FN) for two different PIDSs evaluated by the edge-level protocol.</figcaption></figure>

The intrusion beginning neighborhood IBN is an interval comprising $m_{\text{pre}}$ frames before
and *m* frames from the beginning of *II*:

$$IBN(II, m_{\text{pre}}, m) = [t_{\text{pre}}, t_{\text{m}}] \text{ s.t. } t_{\text{pre}} = t_{\text{start}}(II) - m_{\text{pre}} \text{ and } t_{\text{m}} = t_{\text{start}}(II) + m.$$

This interval highlights the importance of the initial frames of an *II*, where an intruder has just entered the protected area, and it is in this interval where we ideally want the PIDS to raise an alarm. An alarm raised in IBN must be a TP.

For an *II* and alarms $O(P)$, the possible outcomes at edge-level are defined as:

1. TP: if there is at least one alarm in IBN. For multiple alarms in IBN, only the first one is considered, and the rest are ignored.

2. FN: if there is no alarm in IBN.

3. FP: if an alarm is outside IN. Each alarm outside of IN is counted as an FP.

In this evaluation scheme, alarms lying outside IBN but inside IN are ignored. This means that we neither adversely penalize these alarms as an FP nor count them as a TP. In event-level evaluation, whether i-LIDS or this scheme, we do not define a true negative (TN). A TN is when a normal (non-intrusion) event is detected as such; in other words, how well we are classifying a normal event as normal. However, this is not the aim of intrusion detection; indeed, it is the opposite. Furthermore, the calculation of TN is ambiguous. We cannot generalize what length of the non-intrusion video should be considered as a TN. For example, a non-intrusion video clip of 5 minutes cannot be considered as similar to a non-intrusion video clip of 5 days.

These rules are for individual *II*s, but how we deal with scenarios where the intrusion neighborhoods are so close that they intersect one another is another matter. If INs of two or more *II* intersect one another, then we merge them into a single IN. The new IN consists of $m_{\text{pre}}$ frames of the first *II* and $m_{\text{post}}$ frames of the last *II*, and all the frames in between are merged as an *II*. Algorithm 1 summarizes the protocol to evaluate a video at edge-level.

<caption>**Algorithm 1:** Edge-Level Evaluation of a PIDS</caption>

1. Initialize variables *m*, $m_{\text{pre}}$ and $m_{\text{post}}$.

2. Calculate IN for all *II*s of the video.

3. If two or more INs intersect, merge them into a single expanded IN.

4. Calculate intrusion beginning neighbourhood IBN for each *II*.

5. Obtain alarms $O(P)$ from the PIDS.

6. Calculate TP, FN and FP.

7. Calculate precision, recall and other metrics.

## 3.5 Evaluation metrics

Since both PID and VAD tasks are finally a binary classification task, i.e., to classify a frame or event as an intrusion / anomaly (positive) or otherwise (negative), we can naturally apply common metrics to obtain a quantitative performance measure. We describe below some widely used metrics in the literature. They are classified into two categories based on the threshold dependency.
### 3.5.1 Threshold dependent metrics

Most of the binary classification methods irrespective of application to VAD or PID task, produce a continuous output value within a pre-defined range. As an example, reconstruction-based VAD methods produce an output in [0, 1] range, indicating the probability for an input to be anomalous (refer Section (2.5.2.1 and 3.2). To convert this continuous output value into discrete predictions {0, 1}, we need to choose a threshold. This threshold simply creates a boundary and values above and below it are converted to 1 and 0 respectively.

Given ground truth values, with these binary predictions, we can compute various metrics as defined below. They also work for methods where the output is not continuous but discrete values. When any system (PID or VAD) is deployed in a site, we need to set the threshold and therefore, these metrics are essential in a real-life online scenario. Concerning their use, the PID community always use these metrics for evaluation (Buch et Velastin, 2014; Vijverberg et al., 2014; Kim et al., 2018b), while in video anomaly detection, these are rarely used and instead the offline threshold independent metrics are used.

#### 3.5.1.1 Precision

The precision is the percentage of correctly predicted output out of the total predictions, as defined in the Equation (3.15). It is particularly useful when we want to measure how false alarms are affecting a VAD or PID system. A high value of precision denotes that we have very low false alarms.

$$ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}. \qquad (3.15) $$

#### 3.5.1.2 Recall

It is also known as sensitivity or true positive rate (TPR). It is the percentage of correct predictions out of the total predictions, as defined in the Equation (3.16). It is useful when the cost of the false negatives is high, i.e., when we cannot afford to have omissions of intrusions or anomalies. Usually, in any system, the preference is to have the minimum omissions possible. This means that we need to ideally maximize recall.

$$ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}. \qquad (3.16) $$

#### 3.5.1.3 False positive rate (FPR)

It is also known as the false alarm rate. As shown in the Equation (3.17), it is the ratio between false alarms or false positives and total ground truth negatives (non-intrusion or non-anomaly in our case). True negatives (TN) have a major impact in FPR.

$$ \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}. \qquad (3.17) $$

Concerning VAD and PID, true negatives refer to the case where normal frames or events (without anomaly or intrusion) are correctly detected. Both these tasks usually have abundance of normal frames since anomaly or intrusions are rare events. Therefore,
this metric is not very useful while evaluating systems for these tasks. However, popular VAD datasets like UCSD Pedestrian, CUHK Avenue, etc. have almost balanced testing set (see Section 1.4.1). Thus, FPR is computed for them and it is used to calculate AUROC score (more details in Section 3.5.2.1).

### 3.5.1.4 F<sub>β</sub> Score

To take into account both omissions (FN) and false alarms (FP), we need a way to combine precision and recall. The F<sub>β</sub> score combines them with a bias parameter β, which provides more or less importance to recall or precision.

$$F_{\beta} = \frac{(1 + \beta^2) \times \text{Precision} \times \text{Recall}}{(\beta^2 \times \text{Precision}) + \text{Recall}}. \qquad (3.18)$$

The most common values used for β are 0.5, 1 and 2. With β = 1, we obtain the F<sub>1</sub> score. The F<sub>1</sub> score is the harmonic mean of precision and recall and it is widely used (Vijverberg et al., 2013, 2014; Sehairi et al., 2017). The choice of β depends on the precision versus recall demand. For the i-LIDS dataset, there are two system roles with different bias values (i-LIDS Team, 2006). The roles are called ‘Operational Alert’ and ‘Event Recording’, with β as 0.81 and 0.87, respectively (in (i-LIDS Team, 2006), α is used instead of β, where α = β<sup>2</sup>). The former role is designed for real-time intrusion detection and, therefore, has a lower β value to give more importance to precision, as false alarms are essential here. The latter role is for non-real-time systems, where videos are recorded and analyzed on an offline basis. It has a higher β value, as we cannot afford omissions in this case. Most PID systems (Buch et Velastin, 2008, 2014; Cermeño et al., 2018) use this F<sub>β</sub> metric for evaluating on the i-LIDS dataset.

## 3.5.2 Threshold independent metrics

Often in threshold dependent metrics, it is not clear how the threshold is chosen (Saito et Rehmsmeier, 2015). Therefore, systems tend to choose a threshold to maximize the final score, e.g., Buch et Velastin (2014) chose a high detection threshold to eliminate false alarms, as the metric used is F<sub>β</sub> with β = 0.81, which favors precision.

Alternatively, a system can be evaluated over a range of all possible thresholds, thus avoiding bias towards the choice of a fixed single threshold. We present below these types of threshold independent metrics. The main requirement of these metrics is that the system output should be continuous, and they are not adapted for discrete output systems. Contrary to threshold dependent metrics, these are not used in online system deployment but in offline system evaluation instead. These metrics are rarely used in PID community but are very prevalent for the video anomaly detection task (Kiran et al., 2018; Nayak et al., 2021; Ramachandra et al., 2022).

### 3.5.2.1 AUROC

It stands for area under the receiver operating characteristics (ROC) curve. The ROC curve shows the trade-off between sensitivity and specificity (Fawcett, 2006). It is created by plotting the TPR against the FPR, where each point corresponds to a threshold from the list of possible thresholds. This is depicted in Figure 3.6. In ROC curve, a classifier with random performance is shown via a straight diagonal line from (0, 0) to (1, 1), and this line is the baseline of ROC, with AUROC value of 0.5. The best possible value of
AUROC is 1, which signifies perfect classification. Since this score contains FPR, it is influenced by true negatives (refer Section 3.5.1.3). Therefore, AUROC is not suitable for imbalanced datasets (Saito et Rehmsmeier, 2015; Sokolova et al., 2006). Since anomalies or intrusions are rare events, there are often very few anomalies or intrusions compared to the normal events. Therefore, theoretically AUROC should not be used for these tasks.

<figure><img src="image_44.png" /><figcaption>Figure 3.6: Illustration of ROC (left sub figure) and PR curve (right sub figure). Each point in dotted lines represent values ( (TPR, FPR) for ROC and (Precision, Recall) for PR) corresponding to a threshold. The area under the ROC or PR curve, i.e., AUROC or AUPR is depicted with colored areas.</figcaption></figure>

However, recent VAD datasets have balanced dataset for testing phase and therefore AUROC is primarily used there (Kiran et al., 2018; Nayak et al., 2021). Given the VAD test set, the ROC curve is obtained by varying the thresholds on the frame level predictions across the whole test set. It is called as “AUROC on all videos” or micro-averaged AUROC. However, some VAD works compute a “AUROC per video” and report the average, also called macro-averaged AUC (Georgescu et al., 2021a,b; Ristea et al., 2022). In this metric, the succession of thresholds to estimate the ROC curve is not common to all test videos. Since thresholds are adapted to each video, ROC curve is in risk to be overfitted, providing overly optimistic performances. Consequently, AUROC should always be measured as “AUROC on all videos”, computed on the whole test set with thresholds common to all test videos concatenated together (Fawcett, 2006).

3.5.2.2 AUPR

It refers to area under the precision-recall (PR) curve. The precision-recall plot shows precision values for corresponding sensitivity (recall) values. Like the ROC curve, it is model-wide evaluation with complete set of possible thresholds, as shown in Figure 3.6. The AUPR values range from 0 to 1, where 0 and 1 are for the worst and perfect classification, respectively. It is particularly useful when the datasets are highly imbalanced, such as in PID or VAD (Saito et Rehmsmeier, 2015). This is because it uses precision and recall which are focused on predictions of correct class (intrusion or anomaly), rather than that of incorrect class (normal class), i.e., no dependency on true negatives unlike ROC. Concerning VAD, it has been used in some works (Kiran et al., 2018) and it must be used if future datasets are imbalanced (which theoretically they should be).
## 3.6 Conclusion

In this chapter, we defined the two tasks that we tackle in this thesis, i.e., the video anomaly detection and the perimeter intrusion detection. We provided proper formalism to both these tasks. This clear mathematical formulation is essential for properly evaluating works in these tasks. We explained the usual evaluation protocols and since they were not suitable according to the definition and task requirements, we provided a new evaluation protocol. We also studied various evaluation metrics and explained which ones are suitable for which tasks.

Concerning video anomaly detection, we are interested in detecting the occurrence of unusual appearance or motion attributes in unusual locations in the video. There are two principal ways to define an anomaly: either with some labeled anomaly examples during model training, or with only normal (non-anomaly) examples. The anomalous examples can provide weak, semi or full supervision. This first case of VAD with supervision is not practical for real-life use. The two main problems with it are: (1) since anomaly occur so rarely, it is practically impossible to have annotations for all possible future anomalous activities in any natural scene, and (2) even if all possible anomalous activities were available for supervision, the task itself would reduce to binary video classification where the anomalous class is “known”. This vanquish the spirit of video anomaly detection where the ultimate goal in practice is to detect any deviation from normality. In contrary to this, in the second case, the normality is learned from the data. We have data with only normal scenes in the case. Hence, the main idea is to understand normality from this data and then the distribution outside this normality is defined as the anomaly. We propose a formalism for this using the PAC learning framework, which takes into account the complicated cases where normality and abnormality are conceptually very close to each other. For VAD evaluation, frame-level evaluation is followed, therefore the concept of anomalous events is neglected. Furthermore, the threshold-independent metrics are used instead of online metrics. In short, frame-level AUROC metric is used as an evaluation measure in most VAD systems.

Perimeter intrusion detection is a special sub-task of VAD with several specificities like perimeter, intruder object type, movement constraints, etc. It has never been systematically defined before. We first mathematically define how an object can cause an intrusion event, followed by the definition of intrusion intervals in the video. We proposed that the task of perimeter intrusion detection is basically to detect the beginning of these intrusion intervals. Since these intrusion intervals are nothing but events, so we first explained the existing i-LIDS evaluation protocol. We found several drawbacks in this evaluation scheme. Therefore, we proposed a new evaluation protocol for PIDS, called edge level evaluation. It is completely coherent with the PID definition and basically focuses on evaluating a PIDS based on the rising edge from non-intrusion to intrusion state. In other words, this protocol helps evaluating if an intrusion was detected within the first few seconds of its beginning. Finally, similar to i-LIDS evaluation protocol, the threshold-dependent metrics like precision, recall and $F_1$ score can be used with the proposed edge level evaluation.
# Chapter 4

## Unsupervised autoencoder and adaptive detection

In this chapter, we introduce an unsupervised perimeter intrusion detector. We first introduce how we learn spatio-temporal normality from videos with an end-to-end trainable neural network, followed by how intrusions are detected using an adaptive mechanism that handles long-term variations in scene dynamics. We then provide a thorough comparative analysis with other methods on i-LIDS dataset, using different evaluation protocols. Part of this work was published in IEEE International Conference on Image Processing (ICIP) 2022 (Lohani et al., 2022a).

## Contents

<table><tr><td>4.1</td><td>Introduction</td><td>84</td></tr><tr><td>4.2</td><td>Unsupervised and Adaptive Perimeter Intrusion Detector</td><td>84</td></tr><tr><td>4.2.1</td><td>Learning spatio-temporal normality</td><td>84</td></tr><tr><td>4.2.2</td><td>Detecting Intrusions</td><td>86</td></tr><tr><td>4.3</td><td>Experiments and results</td><td>88</td></tr><tr><td>4.3.1</td><td>Implementation details</td><td>88</td></tr><tr><td>4.3.2</td><td>Comparison of Evaluation Protocols for PIDS</td><td>89</td></tr><tr><td>4.3.3</td><td>Effect of adaptive thresholding</td><td>94</td></tr><tr><td>4.3.4</td><td>Working illustration of AE-Adapt</td><td>95</td></tr><tr><td>4.4</td><td>Conclusion</td><td>95</td></tr></table>
## 4.1 Introduction

In this chapter, we introduce an unsupervised deep learning method for perimeter intruder detection. As already stated in previous chapters, intrusions are rare by nature and therefore it is very difficult to have large amounts of annotated examples for training a supervised model. We saw in the review of existing PID methods (refer to Section 2.6) that most methods do not use deep learning. The ones that use deep learning rely on externally trained supervised object detectors and thus their functioning is biased on the external dataset, and furthermore they assume that intruder object class is known beforehand. Therefore, there is a need of methods which learn from the data itself, without requiring need of annotation or external dataset. We propose such an approach in this chapter.

Inspired from the methods of video anomaly detection, this work is based on video clip reconstruction proxy task for PID. The same principle as VAD is followed here: learning normality while training and detecting intrusions as deviations from normality. One major challenge for any PIDS is to adapt to changing weather, light, and environmental conditions. Standard reconstruction based methods are sensitive to these dynamic conditions (Chalapathy et Chawla, 2019; Kiran et al., 2018). To address this limitation, we propose an adaptive strategy so that the presented unsupervised PIDS works well irrespective of scene dynamics.

In the rest of the chapter, we present our proposed method and provide an extensive comparative analysis of it with other methods.

## 4.2 Unsupervised and Adaptive Perimeter Intrusion Detector

In this section, we present our proposed AE-Adapt (AutoEncoder with Adaptive thresholding) method. It is the first unsupervised deep learning approach for perimeter intrusion detection. It has two main steps: learning spatio-temporal normality using an autoencoder and detecting intrusions using this learned representation via an adaptive mechanism. These two steps are described in following subsections.

Before presenting them, we must define the input to our method. We use video clip as the input unit. We mathematically define it as follows. Given a video $V$ with $n$ frames $\{I_1, I_2, \dots, I_n\}$, a video clip $V_{l,s}$ of length $l$ and temporal gap $s$ between frames is defined as:

$$V_{l,s} = \{I_1, I_{1+s}, \dots, I_{1+(l-1)s}\} = \{I_{1+xs}\}_{0 \le x < l}, \quad (4.1)$$

where for simplicity, it is assumed that the clip starts from the 1st frame.

### 4.2.1 Learning spatio-temporal normality

We propose a strided 3D convolutional autoencoder (S3DCAE) as autoencoder for our PIDS as shown in Figure 4.1. Given a video, we use a sliding window to extract video clips. Each video clip is fed to the S3DCAE, which reconstructs it. The idea is to train the autoencoder with only normal videos (without intrusions) and minimize the error between input and reconstructed video clips. The intuition is that the trained autoencoder should correctly reconstruct normal video clips and badly reconstruct the video clips with intrusions. In other words, it will learn the normality and reconstruction error between the input and reconstructed clip will help to distinguish between non-intrusion and intrusion.
<figure><img src="image_45.png" /><figcaption>Figure 4.1: Learning normality: A sliding window (in yellow) extracts video clips of fixed length. Each video clip is fed to S3DCAE, which reconstructs it. The mean squared error (MSE) between input and reconstructed clip is backpropagated to update learnable parameters of the autoencoder.</figcaption></figure>

### 4.2.1.1 S3DCAE architecture

The main components of this architecture are 3D convolutions and 3D deconvolutions. They were used since they jointly model spatio-temporal information and are better suited for feature learning from videos than 2D convolutions, 2D convolutions with LSTM, etc. (Section 2.3).

Let us denote the input as $X$, where $X$ is a video clip of length $T$ and temporal gap of 1, i.e., $X = V_{l=T,s=1} = \{I_1, I_2, \dots, I_T\}$. The proposed autoencoder takes $X$ and outputs a reconstructed video clip $\hat{X}$ of same dimensions. The error between these clips is the training loss function (MSE), defined as:

$$ \mathcal{L} = \frac{1}{T \times D \times H \times W} \|\hat{X} - X\|_F^2, \qquad (4.2) $$

where $D$, $H$ and $W$ denotes channels (depth), height and width of each frame and $\| \cdot \|_F$ denotes the Frobenius norm.

Figure 4.2 shows the architecture of S3DCAE. Each encoder layer consists of a strided 3D convolution while each decoder layer uses a strided 3D deconvolution (Zeiler et al., 2010), with kernel size of $5 \times 3 \times 3$ (shape: temporal length, height, width). A stride of $2 \times 2 \times 2$ is used in each operation except the last 3D deconvolution where $1 \times 1 \times 1$ stride is used. This signifies that each encoder layer reduces the input volume into half, while each decoder layer (except the last) does the opposite. Encoder is composed of two layers containing (16, 8) filters, while decoder also have two layers with (8, 16) filters respectively. The ReLU activation function is used between each layer, except the final layer, which uses the hyperbolic tangent (tanh) activation. A dropout layer with dropout probability of 0.25 is applied after the first layer. Overall, we have a light architecture with only 15, 889 parameters (1011 MB size). Finally, this model is trained using the Adadelta optimizer (Zeiler, 2012).
<figure><img src="image_46.png" /><figcaption>Figure 4.2: S3DCAE architecture for PID with encoder and decoder. Conv3D and De-conv3D refer to 3D convolution and 3D deconvolution respectively, where s is stride, and the number that follows – denotes number of filters. The rightmost column refers to the data shape (channels, temporal length, height, width).</figcaption></figure>

## 4.2.2 Detecting Intrusions

Once we have a trained model, we use it during testing phase with the hypothesis that the intrusion frames will be badly reconstructed, i.e., with a high reconstruction error. Figure 4.3 illustrates how intrusions are detected in our system. A video clip is obtained using a sliding window, the learned S3DCAE takes it as input and reconstructs it. Then, the frame level reconstruction error is computed using the input and reconstructed clip. This reconstruction error is used by an adaptive strategy with moving z-score to produce an intrusion alarm. These steps are explained in detail below.

<figure><img src="image_47.png" /><figcaption>Figure 4.3: Detecting intrusion using AE-Adapt: an input video clip is reconstructed after going through the S3DCAE. The reconstruction error is calculated between these clips and fed to a moving z-score module, the output of which is thresholded to raise an intrusion alarm.</figcaption></figure>

### 4.2.2.1 Frame level reconstruction error

From a test video, we obtain video clips using the sliding window like training. A test video clip of length T frames and temporal gap s = 1, i.e., {I₁, I₂, ..., I_T} goes through S3DCAE to produce a reconstructed clip {Î₁, Î₂, ..., Î_T}. The reconstruction error can then be calculated as:

$$r = \frac{1}{T \times D \times H \times W} \sum_{k=1}^{T} \left\| \hat{I}_k - I_k \right\|_F^2 . \qquad (4.3)$$
For real-time applications, we need a reconstruction error for each frame that the system encounters. Our method is designed for video clip level and not frame level. The video clips of length T are formed with temporal gap of one between each video clip. This means that if we have a video with n frames {$I_1, I_2, ..., I_n$} and T = 8, the first video clip is {$I_1, I_2, ..., I_8$}, the second video clip is {$I_2, I_3, ..., I_9$}, etc. Since each video clip provides a single scalar value of reconstruction error (see Equation 4.3). To convert this video clip level reconstruction error to frame level error, we assume that the value of each video clip reconstruction error corresponds to the last frame of the video clip. This implies that for first T - 1 frames of a video, there are no frame level reconstruction errors, i.e., $r_x$ with $x \ge T$.

### 4.2.2.2 Adaptive thresholding using moving z-score

A PIDS installed in a site records videos for many days. This long video sequence has very high scene dynamics with changes in light, weather, temperature, scene background, etc. Since reconstruction error changes with scene dynamics, naively thresholding it can provide undesirable outcomes (Ribeiro et al., 2018). To resolve this issue, most work using unsupervised autoencoder try to standardize the per-frame reconstruction error for each video (Ribeiro et al., 2018; Ramachandra et al., 2022). They usually use the min-max (MM) rescaling which force the error values to be in the [0,1] range, where 0 and 1 represents normal and intrusion case, respectively. Thresholding on this rescaled reconstruction error is dangerous because this standardization will force the lower values of reconstruction errors (e.g., values of night) to be considered as normal (since forced to be near 0), even if there is an intrusion.

To address this issue of adjusting to scene dynamics, we propose an adaptive mechanism which follows the reconstruction error along time and trigger an alarm as soon as it deviates from the normal behavior and then continue adapting to the new values. It is also well aligned with perimeter intrusion detection definition, where the goal is to detect just the beginning of intrusions and not every intrusion frame in the video. We compute the moving z-score (MZ) of reconstruction errors to provide a temporal standardization of values (Barthélemy et al., 2019). Once standardized, values can be easily compared to a fixed threshold $z_{th}$. Using frame level reconstruction errors for first t frames, i.e., {$r_1, r_2, ..., r_t$}, the system is initialized by computing the mean and standard deviation of them, denoted as $\mu_t$ and $\sigma_t$, respectively. For the $(t+1)^{th}$ frame, z-score of reconstruction error $r_{t+1}$ is computed as:

$$z_{t+1} = \frac{r_{t+1} - \mu_t}{\sigma_t}, \qquad (4.4)$$

If this value is greater than a fixed chosen threshold, i.e., $z_{t+1} \ge z_{th}$, the system raises an intrusion alarm. Regardless of whether an alarm is raised or not, the moving mean and standard deviation are updated as:

$$\mu_{t+1} = \alpha r_{t+1} + (1 - \alpha) \mu_t \qquad (4.5)$$

$$\sigma_{t+1} = \sqrt{\alpha (r_{t+1} - \mu_{t+1})^2 + (1 - \alpha) \sigma_t^2},$$

where $\alpha \in [0, 1]$ defines the speed of the exponential update. If $\alpha$ is close to 1, then it implies that the system adapts itself quickly with the current value of reconstruction error, without taking much into account the evolution of error. Contrary to this, when $\alpha$
is close to 0, the system relies on evolution of error historically and the current value of error plays a smaller role. This process with Equation (4.4) and Equation (4.5) is repeated for each new frame of the video.

## 4.3 Experiments and results

In this section, we first describe our experimental setup with data, methods and evaluation protocols. Next, we present various results comparing our approach different existing PID systems.

### 4.3.1 Implementation details

Here, we present the data used in experiments, the implementation details of our approach and other methods, and finally the evaluation protocols used.

#### 4.3.1.1 Data

We used the two cameras views of i-LIDS sterile zone dataset as described in Table 1.2. The frame resolution of i-LIDS dataset is 720x576. The training set contains 10 non-intrusion videos per view. The testing set contains 17 and 16 videos of view 1 and view 2, with 7 and 6 videos containing intrusions respectively (from 36 to 92 minutes in length). For more details on i-LIDS dataset, please refer to Section 1.4.2.

#### 4.3.1.2 Methods

To compare our method with others using different evaluation protocols, we need their source code. From the list of PID methods (refer Table 2.2), only one method, i.e., Nayak et al. (2019), provides an implementation of their method. Therefore, we select it for our PIDS comparison. Since no classical PIDS (non-deep learning based) provides source code, we use GOFPID (Barthélemy, 2022), which allows to build the traditional PIDS pipeline. With GOFPID, we can have a representative method for the classical PIDS, and thus we include it in our experiments.

The method of Nayak et al. (2019) follows the typical PIDS pipeline (see Figure 2.21), along with some deep-learning-based components. Given a video, frames are extracted and the user is asked to draw a perimeter in the frame for protection. Then, a potential intruder object should be chosen from a list of pre-defined classes, such as person, car, etc. The system detects the intruder object with bounding box using the pre-trained YOLO v2 network (Redmon et Farhadi, 2017), having a detection threshold of 0.25 for class human. If the detected intruder object is inside the pre-defined perimeter, then it is considered as an intrusion. Finally, the object is tracked using simple online and real-time tracking (SORT) algorithm (Bewley et al., 2016). For testing it in i-LIDS, we first drew a protection perimeter following the fences for each view. We then chose person as an intruder class. Then, frames of each video are fed into the system, and we obtained a binary intrusion/non-intrusion prediction for each frame.

GOFPID (Barthélemy, 2022) helps to define the traditional PIDS pipeline. It also provides suitable configuration for the i-LIDS dataset. We use this default i-LIDS configuration as the classical PIDS method, and it can be described as follows. First, the user is asked to draw a perimeter in the first frame of the video. Along with perimeter, the
perspective size of the intruder object (here person) at different distances from the camera is also requested. We use the same perimeter like last method and use the perspective settings provided for i-LIDS with GOFPID. Given these settings, the input video frame is first denoised by spatial blurring using a Gaussian filter. Then, the foreground detection is performed using an improved adaptive Gaussian mixture model (Zivkovic, 2004). The obtained foreground mask is then denoised by mathematical morphology. The created foreground blob is then checked for two conditions: whether it is inside the perimeter and if it obeys the minimum object size according to the perspective setting. If these conditions are met, then the blob is tracked using blob tracking. If the blob is tracked for a minimum of 3 frames and it displaces for a minimum distance (set beforehand), then an intrusion alarm is raised. Testing this method on i-LIDS dataset was straightforward and a binary intrusion/non-intrusion output is obtained for each frame.

Concerning the proposed AE-Adapt method, we trained and tested our model in each view (view 1 and view 2) of i-LIDS dataset. Test uses the same protection perimeter as defined for the last two methods, signifying that all activities (intrusion or otherwise) outside this perimeter are ignored. Non-intrusion videos from training set of each view are used for training, using one Nvidia RTX 3090 GPU, with a batch size of 32. Each frame is converted to grayscale, pre-processed using histogram equalization and pixels were rescaled to [-1, 1]. For each video, input video clip is constructed using 8 frames with temporal stride of one frame, leading to an input of shape 8 × 720 × 576 × 1. Adaptive thresholding is initialized with reconstruction errors of first 10 frames, moving z-score is used with $z_{th} = 4.5$ and $\alpha = 0.01$, chosen from the validation set.

### 4.3.1.3 Evaluation Protocols

We used the three evaluation protocols studied in Section 3.4: frame-level (FL), i-LIDS and edge-level (EL) evaluations. For edge-level evaluation, we set the tolerance variables $m_{pre}$ and $m_{post}$ to 3, i.e., less than 1 second of tolerance. For variable $m$, the following values were chosen: 5, 10 and 50. This signifies that we tested the methods to detect within 1, 2 and 10 seconds from the beginning of intrusion (refer Section 3.4.3). The parameters for 1 and 2 seconds were selected to satisfy the demands of a real-time PIDS, whereas the parameter for 10 second was chosen in order to be comparable with i-LIDS evaluation protocol. For each evaluation protocol, we present the results in terms of precision, recall and $F_1$ score as defined in Section 3.5.

## 4.3.2 Comparison of Evaluation Protocols for PIDS

Here, we present overall results of different methods on view 1 and view 2 of i-LIDS dataset, using different evaluation protocols.

### 4.3.2.1 Results concerning View 1

Figure 4.5 shows the results of the three methods on view 1 of the i-LIDS test dataset through different evaluation protocols. To better understand results, we must first comprehend how these methods predict intrusions. Figure 4.4 shows predictions using these methods on portions of the video taken from the i-LIDS test dataset.

Observe how each method differs in intrusion detection behavior. It can be observed that method of Nayak et al. (2019) has the highest number of correct frame predictions, followed by GOFPID and our method has the least frame predictions. Similarly, the
same trend follows for omissions, since Nayak *et al.* (2019) leaves least number of frames undetected and our method detects only few frames per intrusion. Both these observations explain the frame level recall scores of the three methods as seen in Figure 4.5. All the three methods have few false detections as seen in Figure 4.4, but the ratio of correctly predicted frames to false detections is highest in Nayak *et al.* (2019), followed by GOFPID and finally our proposed method. This explains the frame level precision scores of the three methods, with Nayak *et al.* (2019) having the best score. Overall, this leads to the frame level $F_1$ scores, as shown in the left subfigure of Figure 4.5.

<figure><img src="image_48.png" /><figcaption>Figure 4.4: Example of per-frame predictions of the three methods on portions of video taken from i-LIDS test dataset. The intrusion intervals (ground truth) are shown in light red strips, the abscissa represents frames and ordinate shows prediction, where 1 signifies intrusion and 0 otherwise.</figcaption></figure>

The i-LIDS evaluation protocol functions on event level and concerns with intrusion alarms rather than frame-level binary prediction. To recall, an intrusion alarm is raised as soon as a system changes its state from 0 to 1. For the i-LIDS evaluation protocol, if there is at least one alarm in the first 10 seconds of intrusion (ground truth), it is counted as a true positive (TP). However, an alarm after 10 seconds is counted as a false positive (FP), as already explained in Section 3.4.2. It can be observed that Nayak *et al.* (2019) has a high number of intrusion alarms from the beginning to end of each intrusion interval. These alarms at the end are considered as false detections, even though they happen during the intrusion. This explains their poor precision score in the i-LIDS evaluation protocol as shown in Figure 4.5. Since this method rarely misses producing alarms within the first 10 seconds, it has a very low number of false negatives (FNs), and this is depicted with a high recall score. For GOFPID, we can observe in Figure 4.4 that it produces alarms in beginning of almost all intrusion intervals, this leads to its excellent
recall score in i-LIDS evaluation protocol (see Figure 4.5). But this method also have
alarms at the end of intrusion intervals and even outside them (see Figure 4.4), and these
are counted as FP, which is aligned with their poor precision score. Our method has a
lower number of alarms, and most of them occur at the beginning of intrusions as shown
in Figure 4.4. This leads to a smaller FP due to fewer late predictions, and we have a
good precision value as seen in Figure 4.5. We also have few omissions within the first 10
seconds; therefore, we observe a good recall value in terms of i-LIDS evaluation.

<figure><img src="image_49.png" /><figcaption>Figure 4.5: The results on view 1 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak *et al.* (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F₁ score.</figcaption></figure>

In the edge-level evaluation protocol (Figure 4.5), if an alarm is not raised within the first *m* seconds of an intrusion, then it is considered as an omission or false negative. But alarms raised within the intrusion, even though after *m* seconds from beginning are not counted as FP like i-LIDS evaluation protocol. Since it is a difficult task to raise the alarm in 1st second of an intrusion in comparison to 2nd or 10th second, we observe an increase in recall for all methods from EL (1 s) to EL (10 s). Nayak *et al.* (2019) and GOFPID have a lower recall than our method in EL (1 s) and EL (2 s) because they often raise alarms later than 2 seconds. Regarding precision, here, the excessive late alarms of these two methods are not penalized as FPs. Therefore, in contrast to i-LIDS evaluation, we observe a better precision value for Nayak *et al.* (2019) and GOFPID in edge-level evaluation. Our method raises fewer false alarms than both of them, thus, has a better precision in edge-level evaluation.

Overall, we observe that predictions of Nayak *et al.* (2019) fluctuates frequently be-
tween non-intrusion and intrusion prediction, thus producing a high number of alarms.
This is because it is focused on human detection and it often loses track of the human from
one frame to another, and then re-detects it, causing more intrusion alarms. Furthermore,
it has very few predictions outside intrusions. That is why it has a good precision and re-
call value at frame-level. However, for i-LIDS and edge-level, we observe that their method
has a low precision value. This is because, from this high number of alarms, only few are
relevant, i.e., the ones at the beginning of intrusions. Other alarms are either treated as false alarms or discarded depending on the protocol. The aim of intrusion detection is to detect the intrusion as soon as possible and not just to classify each frame; therefore, it is important to use the right evaluation protocol that emphasizes these characteristics.

Unlike the method of Nayak et al. (2019), GOFPID does not have depend on person detection for PIDS. It first detects blobs. The blobs are then filtered out with the perspective size of intruder objects. The resultant blobs are tracked irrespective of the blob information, i.e., whether it is an intruder blob (human for example), or a non-intruder blob (camera noise or insects). An intrusion alarm is raised as soon as tracked blobs have minimum displacement. When an actual intruder enters the scene in i-LIDS dataset, it is moving and thus the alarm is raised by this system. After some time, either the intrusion stops (as there is a fence) or blob is too small, thus leading to no alarm state. Finally, often the intrusion starts moving again (like for going on other side of fence), which causes another alarm as seen in Figure 4.4. There are a few alarms which are raised not for intrusion, for example, when there is sudden change in light in the scene, or due to climatic conditions like snow. This explains the high precision value and low recall value in frame level. But in reality, the intrusion alarms were raised in beginning of intrusions and both i-LIDS and edge-level protocols captured it with high recall values. The i-LIDS evaluation highly penalized alarms end of intrusion alarms (leading to poor precision score), while edge-level evaluation addressed this issue. Thus, edge-level protocol helped in correctly evaluating this system

For our method, it can be observed that we have a lower number of alarms. It focuses more on the beginning of intrusion events and, due to the z-score, it quickly adapts itself with the scene dynamics. Since only beginning of intrusion are detected, we have a very poor frame-level recall. Our method is sensitive to flickering light or moving light conditions, therefore it has some false alarms. The relation between few correctly predicted frames and some false alarm frames leads to poor frame level precision. Therefore, we have a poorer frame level score. However, in both i-LIDS and at edge-level, we obtain a good score because the proposed method raises the alarm as soon as the intrusion occurs, as it is expected from a PIDS.

### 4.3.2.2 Results concerning View 2

The results of view 2 are shown in Figure 4.6. The GOFPID method follows similar results like in View 1. The recall values are slightly decreased in all protocols because few difficult intrusions were not detected like person rolling far away from camera with same clothes as background. The precision values have slightly increases in event level metrics because there is no camera switch from color to monochrome (since only monochrome videos) and there are less distractions like animals etc. Therefore, the GOFPID method detects lesser number of irrelevant blobs as intrusions here. Thus, it obtains overall better results than View 1. We can observe that in all evaluation protocols, our method has a similar trend of results as in view 1, but with a slight decrease in metric values. This slight decrease in values is due to an increase in FPs and FNs, which is linked to more instances of light fluctuation and intrusions at a far distance. Nayak et al. (2019) obtain similar results to view 1 in frame-level and i-LIDS-level evaluation. In edge-level evaluation, we observe a perfect precision score of 1. This is because this method did not predict any false detection outside the intrusion interval in view 2. This perfect precision score augments their overall score of in edge-level evaluation. This important detail of no false detections was not captured by the i-LIDS-level protocol, and we observe a poor value of precision
for them.

<figure><img src="image_50.png" /><figcaption>Figure 4.6: The results on view 2 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak *et al.* (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F₁ score.</figcaption></figure>

Overall from this subsection, we can conclude that the edge-level protocol is the most suited for evaluating any PIDS. It can be seen as an evolved version of i-LIDS evaluation protocol and thus adheres to real-life constraints, like detecting in beginning of intrusion event. Furthermore, using its parameters, we can evaluate at different lengths from the beginning of intrusions, like 1 second, 2 seconds, etc. Concerning compared methods, we found that their performances were very close to one another in edge-level evaluation. The GOFPID method is representative of the classical PIDS. After extracting blob, it filters out objects with irregular perspective size and tracks the resultant blobs for atleast three frames and checks whether the blob moves a minimum distance. These well formed post-processing operations allows it to filter many irrelevant blobs of objects. Since this dataset has only person as intruder without too much distractions, this method performs very well to detect intruders with a high precision. It only fails in complicated cases where the object is too far and too similar with the background. Some false alarms are caused due to fast moving light in the scene or due to snow. The method of Nayak *et al.* (2019) uses pre-trained person detector. Since this dataset has only person as intruder, this method works very well. It rarely detects noise due to snow or lighting conditions as intrusions, thus it has a very good precision value in edge-level evaluation. When intruder enters the scene, it detects it but then loses it in tracking and have re-detections. Also, it has problem in detecting when the person in crawling or log-rolling in the far away distance. It detects most easily when the person is standing. Due to this, it often have intrusion alarms late, around the end of intrusion, and this leads to a lower recall value for this method. Finally, our unsupervised and adaptive PID system performed equally well in comparison to last two methods. The false alarms are mainly caused due to changing light conditions and insects on camera. But these are very few in number, thus we have an overall good precision score in edge-level evaluation. Our method detects intrusions
<figure><img src="image_51.png" /><figcaption>Figure 4.7: Qualitative comparison of thresholding strategies on a video, with ground truth (GT) in blue, thresholds in green and alarms in red. Fixed threshold on reconstruction error of S3DCAE (top); fixed threshold on min-max (MM) rescaled reconstruction error (middle); adaptive threshold by moving z-score (MZ) of reconstruction error (bottom).</figcaption></figure>

and adapts itself, we rarely miss an intrusion event, therefore we have a very good recall value. The missed intrusions include cases where intruders are very far from the camera, having clothes similar to scene background. Our method has still a room for improvement and we will present a new method in the next chapter. Since i-LIDS dataset is not very difficult, having only person as intruder, we also test methods on a private dataset which is detailed in annex Section A.

### 4.3.3 Effect of adaptive thresholding

In this subsection, we explore the impact of adaptive thresholding in the proposed method. Figure 4.7 shows the impact of various thresholding strategies. We can observe that the reconstruction error varies a lot from one intrusion to another. Therefore, trying to select an optimal threshold on it can lead to unsatisfactory results, e.g., false alarms before and after 1st intrusion. After min-max rescaling, we obtain a similar conclusion as only the range of values have changed to [0, 1], without any significant difference in threshold choosing strategy. We can see that the moving z-score adjusts itself with the reconstruction error. It detects the beginning of each intrusion and then updates itself.

In Table 4.1, we quantitatively compare the thresholding strategies on proposed S3DCAE. We observe that the direct outcome from the autoencoder leads to a poor performance. As expected, it is difficult to choose a threshold when the error varies due to scene dynamics. Rescaled reconstruction error using min-max (MM) scheme improves the overall result by only 5%. Since i-LIDS test-set videos are very long, the rescaling scheme failed to handle changes in reconstruction error caused by changing scene dynamics. We can
<table><caption>Table 4.1: Quantitative comparison of thresholding strategies, on View 1 of i-LIDS dataset. S3DCAE stands for proposed autoencoder with frame reconstruction error, MM for min-max scaling, and MZ for moving z-score. EL (10 s) is used as evaluation protocol.</caption><thead><tr><th>Methods</th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>S3DCAE</td><td>0.54</td><td>0.44</td><td>0.49</td></tr><tr><td>S3DCAE + MM</td><td>0.49</td><td>0.59</td><td>0.54</td></tr><tr><td>S3DCAE + MZ</td><td><b>0.92</b></td><td><b>0.91</b></td><td><b>0.92</b></td></tr></tbody></table>

clearly observe that adding an adaptive thresholding with moving z-score (MZ) almost doubles the overall performance from $F_1$ of 0.49 to 0.92. These results strongly support our proposition that an adaptive component is necessary for the deployment of a PIDS in real-life scenes.

### 4.3.4 Working illustration of AE-Adapt

In this subsection, we demonstrate the working of our proposed method, shown in Figure 4.8. For ease of comprehension, we continue with the example of Figure 4.7.

Since it is continuously snowing (distraction) in the scene, there are small zigzag effects in reconstruction error. We can observe in images that the intruders are at different distances from camera and the magnitude of reconstruction error depends on it. For instance, the intruders far from camera occupy a smaller area in the image and thus have smaller reconstruction error. We also observe a high local RE in the beginning of intrusions (see images in top row). This is because when an intruder enters the scene, there is a high spatio-temporal change, which for autoencoder is abnormal (normal training videos in i-LIDS dataset have extremely low number of examples for where objects move) and thus it is badly reconstructed. Two images in bottom row show that even though the intruder is present in the scene, the reconstruction error is relatively lower. The reason behind is the same, i.e., spatio-temporal change. The spatio-temporal change is comparatively lower in these cases, since the intruders are hardly moving (trying to break in). Finally, we can observe via the green curve that the moving z-score just detects the beginning of these intrusions and then adapts itself with the reconstruction error. This is what we expect ideally from a PIDS.

## 4.4 Conclusion

In this chapter, we introduced a new unsupervised method called AE-Adapt, for detecting intrusions in a video stream. We proposed a strided 3D convolutional autoencoder that learns spatio-temporal normality while training only on normal videos. To detect intrusions during testing, the error between input and reconstructed video clips is used. This error is thresholded to detect intrusions. But this error depends on the changing scene dynamics like weather, temperature, etc. We show that naively thresholding this error, especially on long videos, performs poorly. Therefore, we proposed an adaptive thresholding strategy using moving z-score on this error. We found that this adaptive
<figure><img src="image_52.png" /><figcaption>Figure 4.8: AE-Adapt working illustration on a portion of video of i-LIDS test set. In the curve, reconstruction error (RE) is shown in black, threshold in green and alarm in red. GT stands for ground truth per frame, where blue region indicates intrusion. Some illustrative frames are shown above and below the plot, where the blue bounding box shows intruder.</figcaption></figure>

mechanism almost doubles the results of our proposed PIDS.

We also compared our PIDS with two existing methods, using different evaluation protocols. We showed that intrusion detection should be evaluated at a higher level than frame level, because frame level evaluation cannot tell if the beginning of intrusion events is detected or not, which is essential for a PIDS. That is why almost all methods in the state-of-the-art use i-LIDS evaluation. However, the major problem with this evaluation is that it penalizes all alarms after 10 seconds, even if they are well within the intrusion interval. It is indeed too strict in counting FPs, and this hits the overall score negatively, although the system has detected intrusion well within first 10 seconds. We proposed edge-level evaluation to address this issue, where we can parameterize the time after intrusion beginning for the evaluation. This helps in testing the system for different practical time settings and testing its robustness. More importantly, the alarms after a specified time are not counted as false positives if they are within the intrusion. We found that irrespective of method, edge level evaluation should be used for PIDS evaluation.

Concerning compared methods, we found that all the three methods have equivalent
performance on i-LIDS dataset, with edge-level evaluation. The method of Nayak et al. (2019) depends on a pre-trained person detector and i-LIDS have only people as intruders, it was an easy task for this method. GOFPID uses traditional PIDS pipeline and also manually drawn perspective to filter object blobs of irregular size. It performs well in this dataset, since there are very few non-intruder moving objects. Our proposed method first learned normality from training set and then detected intrusions as deviations from it. Theoretically, our method can detect intrusions caused by other objects since it does not use beforehand information about intruder class. In fact, for edge-level evalution for 1 and 2 seconds, our method outperformed other methods. This means it is suitable for rapid real time detection as well. Nevertheless, our proposed method still has drawbacks. That is why, in the next chapter, we try to enhance the spatio-temporal comprehension of normality, to better detect intrusions. Also, since i-LIDS dataset was comparatively easy, we also test these methods on a private dataset, available in annex.
# Chapter 5

## Leveraging Unsupervised and Self-Supervised Learning

In this chapter, we introduce a new approach for spatio-temporal video representation learning. It leverages unsupervised and self-supervised learning and is applicable to both VAD and PID tasks. In the coming sections, we first explain our generic approach. Next, we reveal how it is applied to VAD and PID tasks. Then, the following two sections show experiments and results for the two tasks. Major part of this work was accepted in International Conference on Computer Vision Theory and Applications (VISAPP), 2023.

## Contents

<table><tr><td>5.1 Introduction</td><td>100</td></tr><tr><td>5.2 Proposed approach</td><td>100</td></tr><tr><td>5.2.1 Learning normal spatio-temporal patterns using multiple tasks</td><td>101</td></tr><tr><td>5.2.2 Testing phase</td><td>103</td></tr><tr><td>5.3 Application to tasks</td><td>106</td></tr><tr><td>5.3.1 VAD task</td><td>106</td></tr><tr><td>5.3.2 PID task</td><td>108</td></tr><tr><td>5.4 VAD: Experiments and results</td><td>110</td></tr><tr><td>5.4.1 Implementation details</td><td>110</td></tr><tr><td>5.4.2 Quantitative results</td><td>110</td></tr><tr><td>5.4.3 Qualitative results</td><td>112</td></tr><tr><td>5.4.4 Ablation studies</td><td>113</td></tr><tr><td>5.5 PID: Experiments and results</td><td>115</td></tr><tr><td>5.5.1 Implementation details</td><td>116</td></tr><tr><td>5.5.2 Results</td><td>116</td></tr><tr><td>5.5.3 Importance of adaptiveness in proposed PID methods</td><td>117</td></tr><tr><td>5.5.4 Testing proposed PIDS in a more challenging real-world scenario</td><td>118</td></tr><tr><td>5.6 Conclusion</td><td>118</td></tr></table>
## 5.1 Introduction

In this chapter, we propose a new approach to address the task of video anomaly detection (VAD) and perimeter intrusion detection (PID). As described in Section 2.5, one of the most highly successful approaches to tackle VAD is to use a deep autoencoder (AE) (Hasan et al., 2016; Luo et al., 2017a; Ramachandra et al., 2022) and it also has the least assumptions on data. We also showed in Chapter 4 that the same principle of AE for VAD can be also used for the task of PID. Therefore in this chapter, we proceed with the AE based approaches, proposing a method that leverages unsupervised and self-supervised learning on a single AE. To this end, we devise multiple tasks to enhance the normal spatio-temporal understanding of the AE by training it only on the normal data. Each task has its specific objective: (i) video clip reconstruction (VCR) to learn spatio-temporal characteristics of the normal videos; (ii) future frame prediction (FFP) to learn how normal spatio-temporal patterns propagate along the videos; (iii) playback rate prediction (PRP) to strengthen the playback speed perception of the encoder. Our method is end-to-end trainable and is jointly trained on the three tasks. While testing, the anomaly or intrusion is detected if the combined score of the three tasks is high. Most of the current methods use error measures like mean squared error (MSE) or peak signal to noise ratio (PSNR) for comparing input and reconstructed frames (Astrid et al., 2021a,b; Gong et al., 2019; Park et al., 2020; Lv et al., 2021). But these measures are prone to noise (Gudi et al., 2022; Sinha et Russell, 2011) and therefore we introduce a new measure, called the blur pooled error (BPE). It is locally sensitive and keeps only relevant pixels for error calculation. Most works apply a min-max rescaling to anomaly scores per video (Liu et al., 2018; Gong et al., 2019; Park et al., 2020; Astrid et al., 2021a,b). It is sensitive to extreme values and to address this issue, we propose a robust rescaling of scores for VAD.

Overall, in this chapter, we propose a new approach for spatio-temporal learning in videos. It is generic and we applied it to VAD and PID tasks. In the following sections, we will present our method in general, then its application to VAD and PID tasks, followed by respective experiments and results.

## 5.2 Proposed approach

In this section, we present our proposed LUSS-AE (Leveraging Unsupervised and Self-Supervised AutoEncoder) method. The main idea is to learn normal spatio-temporal features in order to better discriminate anomalies or intrusions from normal patterns. To this end, we propose to train a 3D convolutional autoencoder (3D CAE) on normal videos using carefully designed tasks in a unsupervised and self-supervised manner. The video clip reconstruction task learns spatio-temporal characteristics of normal videos. The future frame prediction task is designed to learn the propagation of spatio-temporal patterns in the normal videos. Finally, the playback rate prediction task strengthens the speed understanding of the encoder. The autoencoder is jointly trained on all these tasks. During testing, each of these branches provides a score to distinguish anomalous or intrusion frames from the normal ones.
<figure><img src="image_53.png" /><figcaption>Figure 5.1: Overall schema of the proposed LUSS-AE method.</figcaption></figure>

Figure 5.1 illustrates overall schema of our approach. A video clip of $T+1$ consecutive frames is extracted from the video. The first $T$ frames of this clip goes through the 3D CAE, which reconstructs it. This reconstruction, after passing through an activation function, is compared with original clip to perform the video clip reconstruction task. The same reconstruction (before activation) is fed to the $H_{FFP}$ head, which predicts a future frame. This frame is compared with the $(T+1)^{th}$ frame of the input clip, thus performing the future frame prediction task. Another video clip of $T$ frames is extracted from the video. It is either accelerated or kept at the original playback rate. Then, this video clip is fed to the encoder of the 3D CAE. This output is fed to the $H_{PRP}$ head, which produces a prediction for playback rate of the input clip. This prediction is compared with the actual playback rate of the video clip, to perform the playback rate prediction task. All these tasks are performed jointly and the weighted sum of errors is backpropagated together in the network.

In the following subsections, we first detail the role of each task and how they are trained together, followed by how these tasks act during the testing phase.

### 5.2.1 Learning normal spatio-temporal patterns using multiple tasks

In this subsection, we explain how the proposed tasks help in learning normal characteristics during training. We describe each task with its role, followed by details on how all these tasks are trained in a joint manner.

#### 5.2.1.1 Video Clip Reconstruction (VCR)

Reconstructing a video clip is one of the most popular tasks for unsupervised VAD (Astrid et al., 2021a,b; Gong et al., 2019; Liu et al., 2021b; Zhao et al., 2017). We explored its principle and some major VAD methods in Section 2.5.2.1. In the last chapter, we proposed a video clip reconstruction based PID method and we demonstrated its usefulness for the task. Concretely, the VCR task aims to reconstruct an input video clip, mostly
using an autoencoder (AE) type network. The AE is trained only on normal video clips with the learning objective of minimizing the mean-squared error (MSE) between the input and reconstructed clips. The main hypothesis is that the abnormal or intrusion clips will be badly reconstructed during testing.

Using Eq. (4.1), a non-strided video clip of length $T + 1$ frames can be defined as:

$$V_{T+1,1} = \{I_1, I_2, \dots, I_T, I_{T+1}\} . \qquad (5.1)$$

The first $T$ frames of this clip is used for the VCR task and we denote it as $X_{VCR}$, i.e., $X_{VCR} = \{I_1, I_2, \dots, I_T\}$. This video clip goes through the autoencoder followed by an activation function to produce a reconstructed clip $\hat{X}_{VCR}$ as:

$$\hat{X}_{VCR} = \tanh(\mathrm{Dec}(\mathrm{Enc}(X_{VCR}))), \qquad (5.2)$$

where Enc and Dec stand for encoder and decoder networks, respectively, and tanh is the activation function.

The loss function can then be defined as:

$$\mathcal{L}_{VCR} = \frac{1}{T \times D \times H \times W} \left\| \hat{X}_{VCR} - X_{VCR} \right\|_{F}^{2}, \qquad (5.3)$$

where $D$, $H$ and $W$ denotes channels (depth), height and width of each frame and $\| \cdot \|_F$ denotes the Frobenius norm.

### 5.2.1.2 Future Frame Prediction (FFP)

Predicting a future frame is also a well-spread task for unsupervised VAD (Liu et al., 2018, 2021b; Lv et al., 2021; Park et al., 2020) and it was discussed thoroughly in Section 2.5.2.2. To our knowledge, it has not yet been applied for the PID task. FFP aims to predict an unseen future frame, given an input video clip. This requires comprehension of how normal spatio-temporal patterns propagate along the video clip. Like VCR task, the objective is to minimize the MSE, but between the predicted and actual future frame. Since the AE is trained only on normal videos, it should predict the anomalous or intrusion frames poorly.

This task uses the same input of VCR task, i.e., $X_{VCR}$. After passing through the AE, the video clip $X_{VCR}$ goes through the prediction head $H_{FFP}$ to predict the future frame as:

$$\hat{X}_{FFP} = H_{FFP}(\mathrm{Dec}(\mathrm{Enc}(X_{VCR}))). \qquad (5.4)$$

This frame is compared with the actual future frame, i.e., frame $T + 1$ of $V_{T+1,1}$ (see Eq. (5.1)) denoted as $X_{FFP}$, where $X_{FFP} = I_{T+1}$. The loss function is then defined as:

$$\mathcal{L}_{FFP} = \frac{1}{D \times H \times W} \left\| \hat{X}_{FFP} - X_{FFP} \right\|_{F}^{2} . \qquad (5.5)$$

### 5.2.1.3 Self-supervised Playback Rate Prediction (PRP)

Section 2.5.2.3 exhibited that self-supervised learning is very recent for the VAD task. Like FFP task, it has never been studied for the PID. We propose here a self-supervised task, i.e., playback rate prediction (PRP), applicable to both VAD and PID tasks. The PRP task does exist in self-supervised representation learning and it is used as a pretext task to learn transferable semantic spatio-temporal features for downstream tasks like
action recognition (Liu et al., 2021a). In other words, first PRP task is performed and later the learned model is adapted to downstream tasks. Contrary to them, we adapt the PRP task, and train it via a single AE with two other tasks, all done simultaneously in a joint and end to end manner.

The original PRP task generates speed labels for video clip sampled at different rates and aims at predicting them (Benaim et al., 2020; Wang et al., 2020; Yao et al., 2020). Since we know that the training videos are normal in our case, we adapt this task to generate two speed-rate labels: original playback rate (implying normal behaviour) and accelerated playback rate with 2x to 5x speed (implying abnormal / intrusion behaviour). The motive is to enforce the encoder with motion comprehension of normal videos. During testing, we hypothesize that the encoder would detect anomalies or intrusions caused by irregular and abrupt motion.

Concretely, given a video clip, this task aims to predict its playback rate. The clip with default playback rate of the video is termed as the original playback rate clip and the clip formed by skipping 1 (2x), 2 (3x), 3 (4x) or 4 (5x) frames is designated as an accelerated playback rate clip. The input $X_{PRP}$ is a video clip of length $T$, chosen between an original playback rate (class $c = 1$) and an accelerated playback rate (class $c = 2$) with equal chance (50% probability each):

$$X_{PRP} = \begin{cases} V_{T,1} & \text{when } c = 1 \\ V_{T,s \in \{2,3,4,5\}} & \text{when } c = 2 \end{cases}, \qquad (5.6)$$

where the accelerated playback rate clip, when $c = 2$, is a temporally strided video clip with temporal gap $s$ randomly chosen between 2 and 5. This input $X_{PRP}$ goes through the encoder, playback rate prediction head $H_{PRP}$ and softmax activation to produce prediction as:

$$\hat{y} = \mathrm{softmax}(H_{\mathrm{PRP}}(\mathrm{Enc}(X_{\mathrm{PRP}}))) \in \mathbb{R}^2 \qquad (5.7)$$

The loss function for this classification task is the binary cross-entropy (BCE), defined as:

$$\mathcal{L}_{\mathrm{PRP}} = - \sum_{c=1,2} y[c] \log (\hat{y}[c]) , \qquad (5.8)$$

where $y$ is the one-hot encoding of the ground truth classes for $X_{PRP}$.

### 5.2.1.4 Training objective

A single autoencoder is trained with the above-mentioned tasks and the overall training loss is defined as the weighted sum of individual loss functions:

$$\mathcal{L} = \lambda_1 \mathcal{L}_{\mathrm{VCR}} + \lambda_2 \mathcal{L}_{\mathrm{FFP}} + \lambda_3 \mathcal{L}_{\mathrm{PRP}} , \qquad (5.9)$$

where $\lambda_1$, $\lambda_2$ and $\lambda_3$ are the weights in $[0, 1]$ that regulate the importance of each task.

## 5.2.2 Testing phase

In this subsection, we describe how the LUSS-AE is used during testing. Given a test video clip, each of the three tasks provides a score and the overall score is a weighted sum of these scores. Depending on the aimed task (VAD or PID), this score is post-processed accordingly, e.g., for VAD, it is normalized. After post-processing, the high values of the score represents anomaly or intrusion. We first define below some image or video error measures and then describe how these measures help to calculate the overall score in inference phase.
### 5.2.2.1 Image error measures

To quantitatively assess how well a future frame is predicted or how well a video clip is reconstructed, we need to compare them with the appropriate ground truth using an error measure. For this, the most widely used measure is MSE (Gong et al., 2019; Liu et al., 2018; Lv et al., 2021; Zhao et al., 2017). Given two images $J, \hat{J} \in \mathbb{R}^{H \times W \times D}$ (in our case, original and reconstructed frame), the MSE is calculated as:

$$ \text{MSE}(J, \hat{J}) = \frac{1}{D \times H \times W} \left\| \hat{J} - J \right\|_{F}^{2}. \qquad (5.10) $$

Since last few years, many works use the peak signal to noise ratio (PSNR) measure (Astrid et al., 2021a,b; Park et al., 2022; Ye et al., 2019). But PSNR also depends on MSE as can be seen in its mathematical formulation. Both MSE and PSNR integrate errors on the whole image and therefore are prone to random and incoherent noise (Gudi et al., 2022; Sinha et Russell, 2011). To overcome this, a new measure called the proximally sensitive error (PSE) is proposed by Gudi et al. (2022). It is defined as:

$$ \text{PSE}(J, \hat{J}) = \frac{1}{D \times H \times W} \left\| (\hat{J} - J) * G_{(\sigma,k)} \right\|_{F}^{2}, \qquad (5.11) $$

where * is the convolution operator and $G_{(\sigma,k)}$ is a 2D Gaussian kernel with size $k$ and standard deviation $\sigma$, given by:

$$ G_{(\sigma,k)}[i,j] = \frac{1}{2\pi\sigma^2} e^{-\frac{i^2+j^2}{2\sigma^2}}, \qquad (5.12) $$

where $i$ and $j$ are the pixel coordinates centered in the kernel. Thanks to the Gaussian convolution, PSE smooths incoherent noise and is locally sensitive. However, an anomaly or intrusion generating an important error in some pixels can disappear in the noise of all other pixels of the high-dimensional image.

To address this, we take this idea one step further and introduce the blur pooled error (BPE), defined as:

$$ \text{BPE}(J, \hat{J}) = \frac{1}{D \times H \times W} \left\| S_b((\hat{J} - J) * B_k) \right\|_{F}^{2}, \qquad (5.13) $$

where $B_k$ is a generic 2D low-pass filter with kernel size $k$, and $S_b$ signifies a subsampling with stride $b$ (Zhang, 2019). Using a low-pass filter smooths incoherent noise, like in PSE. Then, the subsampling keeps only the most pertinent values from it. Thus, this error has an increased sensitivity to anomalies / intrusions than PSE. All these measures can easily be extended to video clip by applying them to its constituent frames.

### 5.2.2.2 Test score

Here, we will focus on how different tasks of our approach act during testing phase and provide scores. Concretely, we will see how these scores are calculated and how the final score is obtained. Figure 5.2 shows the schema of our method during testing phase. Given a test video, we first generate non-strided clip of length $T + 1$. The first $T$ frames of this clip, i.e., $I_{1:T}$ is the input to our method. This input first goes through the encoder and the resultant latent space is input to two branches, i.e., decoder and $H_{PRP}$ head. The one that goes through $H_{PRP}$ head produces predictions $\hat{y}$ which are used to compute the $A_{PRP}$
score. While the one that goes through decoder bifurcates again into two branches, i.e., reconstruction and $H_{FFP}$ head. With reconstruction branch, the input clip is reconstructed as $\hat{I}_{1:T}$ and the $A_{VCR}$ score is computed. The branch of $H_{FFP}$ head produces the future frame $\hat{I}_{T+1}$, which is compared with frame $I_{T+1}$ of the input to calculate the $A_{FFP}$.

<figure><img src="image_54.png" /><figcaption>Figure 5.2: Overall schema of the proposed LUSS-AE method during testing. A window of $T+1$ consecutive frames is drawn sequentially from the test video. The first $T$ frames from this window are the input of the system and the output for each task is computed on it. The anomaly score is determined for each task, i.e., $A_{VCR}$, $A_{FFP}$ and $A_{PRP}$, and the final anomaly score is their weighted sum.</figcaption></figure>

The anomaly scores for the three tasks are described below:

(i) the VCR anomaly score is defined as:

$$A_{VCR} = \frac{1}{T} \sum_{t=1}^{T} E(\hat{I}_t, I_t), \qquad (5.14)$$

where $I_t$, $\hat{I}_t$ are the $t^{th}$ frame and its reconstruction respectively, and $E$ can be one of three error measures, i.e., MSE, PSE or BPE respectively.

(ii) the FFP anomaly score is computed as:

$$A_{FFP} = E(\hat{I}_{T+1}, I_{T+1}). \qquad (5.15)$$

(iii) the PRP anomaly score is defined as the probability of accelerated class ($c = 2$):

$$A_{PRP} = \hat{y}[2], \qquad (5.16)$$

where $\hat{y}$ is the output of PRP branch as defined in Eq. (5.8).

The final score is defined as:

$$A = \alpha_1 A_{VCR} + \alpha_2 A_{FFP} + \alpha_3 A_{PRP}, \qquad (5.17)$$

where $\alpha_1, \alpha_2, \alpha_3$ are the weights in $[0, 1]$ for the three scores.
## 5.3 Application to tasks

So far, we introduced the main idea of LUSS-AE. It is generic and can be theoretically applied to any unsupervised video learning task. In this section, we explain concretely how it is applied to video anomaly detection and perimeter intrusion detection task, and what is the final measure that leads to detection of anomalies and intrusions.

### 5.3.1 VAD task

We recall that in this manuscript, we tackle the problem of video anomaly detection without anomaly information (refer Section 3.2.2). In other words, we have training data with no labels, and it is considered safe to assume that it contains only normal events. The idea is to understand these normal events during model training to detect anomalies as a deviation to normality during testing. We use our proposed LUSS-AE with video clip (VC) as the input unit. As explained in Section 2.5.1, video clip is an ideal input for VAD since it contains both spatial and temporal information and does not have bias towards external detectors. Our motivation is to propose a generic VAD method without extra supervision or bias. We present below the network architecture we use in LUSS-AE for VAD and the final anomaly score to detect anomalies.

#### 5.3.1.1 Network architecture

In this subsection, we detail the network architecture of different components of our method. Precisely, the components are autoencoder, the future frame prediction head and the playback rate prediction head.

**a) Autoencoder** We do not propose a new autoencoder architecture and instead use a widely popular architecture for VAD. Like this, we can study the viability of our method and understand how exactly it is better or worse in comparison to the existing works. We use the strided 3D convolutional autoencoder proposed by Gong *et al.* (2019), and it is also used by other works (Astrid *et al.*, 2021a,b). The reason for choosing this architecture is because there are 3D convolutions and 3D deconvolutions, which help in learning spatio-temporal features jointly from a video stream (refer to Sections 2.3.3 and 4.2.1.1 for more information).

In Figure 5.3, we demonstrate this architecture. It takes a video clip of 16 frames as input, where each frame is 256 x 256 in size. The encoder uses a series of 3D convolutions with kernel size of 3 x 3 x 3 (shape: temporal length, height, width). The stride in first convolution is of shape 1 x 2 x 2, signifying it reduces only the spatial dimension by half. Rest of the convolutions diminish all the three dimensions by half using 2 x 2 x 2 stride. The decoder uses 3D deconvolutions or transposed convolutions with the same kernel size of 3 x 3 x 3. The stride is 2 x 2 x 2, except for the last deconvolution which has 1 x 2 x 2 as stride. Each layer is followed by 3D batch normalization and LeakyReLU activation, except for the last layer. Finally, the output is of the same shape as the input.

**b) Future frame prediction head** Figure 5.4 shows the proposed architecture of $H_{FFP}$ head for VAD task. It takes the output of decoder as its input. This input is passed through a 3D convolution with kernel size of 3 x 3 x 3 and stride 16 x 1 x 1, followed by tanh activation function to produce the future frame.
<figure><img src="image_55.png" /><figcaption>Figure 5.3: Autoencoder architecture for VAD with encoder and decoder. Conv3D and Deconv3D refers to 3D convolution and 3D deconvolution respectively, and the number that follows them, denotes amount of filters. The rightmost column refers to the data shape (channels or filters, temporal length, frame height, frame width).</figcaption></figure>

<figure><img src="image_56.png" /><figcaption>Figure 5.4: Architecture of FFP head i.e., $H_{FFP}$, for VAD task. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively. The rightmost column refers to the data shape.</figcaption></figure>

c) **Playback rate prediction head** The proposed architecture of $H_{PRP}$ head for VAD is shown in Figure 5.5. It takes the encoder output as its input, i.e., an input of size (256, 2, 16, 16). The input is reduced to a smaller dimension using a series of max-pooling and 2D convolution operations. This reduced input is then flattened to a vector of 512 dimensions. Then, three consecutive fully connected (FC) layers reduce the vector dimension to 2, where the first two FC layers are followed by ReLU activation function. Finally, this vector is converted into probabilities using the softmax activation function.

### 5.3.1.2 Anomaly score

Most VAD works apply a min-max rescaling to scores per video (Liu et al., 2018; Gong et al., 2019; Park et al., 2020; Astrid et al., 2021a,b). This scaling bounds values to interval [0, 1] where the minimum and maximum values are forced to be 0 and 1 respectively. Due to this, it is prone to outliers with extreme values. To address this issue, we propose a robust rescaling per video. For a test video with n frames, the rescaled anomaly score for frame t is defined as:

$$ \tilde{A}_t = \frac{A_t - \text{med}(\{A_i\})}{\text{iqr}_{1-99}(\{A_i\})}, \qquad (5.18) $$

where med(·) and iqr$_{1-99}$(·) are respectively the median and the interquantile range (between 1st and 99th percentiles) of scores {$A_i$}$_{i=1}^n$. Finally, like previous methods, the higher scores correspond to anomalies.
<figure><img src="image_57.png" /><figcaption>Figure 5.5: Architecture of PRP head i.e., $H_{PRP}$, for VAD task. The different layers are depicted in middle column with violet color and the number that follows them, denotes amount of filters or units. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape (channels, temporal length, frame height, frame width).</figcaption></figure>

## 5.3.2 PID task

For PID task, we follow the same intuition as the last chapter. In other words, we train LUSS-AE on training data with no labels to understand normality, so that intrusion can be detected during testing as they would follow a deviation from normality. Like VAD and previous PID method (see Section 4.2.1.1), we use video clip as input. We present below the network architecture we propose in LUSS-AE for PID and the final intrusion score to detect intrusions.

### 5.3.2.1 Network architecture

In this subsection, we detail the network architecture of different components of our method for PID.

**a) Autoencoder** For autoencoder, we use the same 3DCAE architecture that we proposed in last chapter. In this way, we can investigate how well or worse the LUSS-AE performs in comparison to PID method proposed in the last chapter. As shown in Figure 4.1, this autoencoder takes an input video clip of 8 frames, where each frame is 720 × 576 in size. The encoder uses a series of strided 3D convolutions to produce a latent space of (8, 2, 180, 144). The decoder takes this latent space as input and uses a series 3D deconvolutions to produce an output with same size as that of the input video clip. Refer Section 4.2.1.1 for more details. Even though autoencoder architecture for both VAD and PID have strided 3D convolutions and deconvolutions, the architecture of PID takes only 8 frames as input instead of 16, has a much higher frame resolution than VAD, and smaller amount of filters and layers, making it computationally fast.

**b) Future frame prediction head** The proposed architecture $H_{FFP}$ for PID is shown in Figure 5.6. It takes the output of decoder as its input. A 3D convolution with kernel size of 3 × 3 × 3 and stride 8 × 1 × 1 is first applied to this input, and then a tanh activation function is used to produce the future frame.
<figure><img src="image_58.png" /><figcaption>Figure 5.6: Architecture of FFP head i.e., $H_{FFP}$, for perimeter intrusion detection. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively. The rightmost column refers to the data shape (channels, temporal length, frame height, frame width).</figcaption></figure>

**c) Playback rate prediction head** The proposed architecture of $H_{PRP}$ head for PID is shown in Figure 5.7. The encoder output of size (8, 2, 180, 144) is its input. This input is reduced to a smaller dimension using a series of max-pooling and 2D convolution operations. It is then flattened to a vector of 828 dimensions. Then, three consecutive fully connected (FC) layers reduce the vector dimension to 2, where the first two FC layers are followed by ReLU activation function. Finally, this vector is converted to probabilities using a softmax activation function.

<figure><img src="image_59.png" /><figcaption>Figure 5.7: Architecture of PRP head i.e., $H_{PRP}$, for perimeter intrusion detection. The different layers are depicted in middle column with violet color. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape.</figcaption></figure>

### 5.3.2.2 Adaptive Intrusion score

To finally detect intrusion, we need a score that describes it. The overall testing score $A$ of Equation (5.17) can be used in three ways as an intrusion score: (i) directly using $A$ as intrusion score; (ii) using normalized $A$ (like in VAD task) as intrusion score; (iii) using adaptive mechanism of moving z-score (MZ) on $A$ (refer to Section 4.2.2.2). We concluded in the last chapter that adaptive thresholding is essential for a PID system, therefore we choose the third way for our experiments with LUSS-AE. We follow exactly the same method of adaptive thresholding as described in last chapter. We just replace $r$ values with $A$ values in Equations (4.4 and 4.5), respectively. Since we use LUSS-AE
with adaptive thresholding, we name this method as LUSS-AE-Adapt, in correspondence
to the nomenclature of the last chapter.

5.4 VAD: Experiments and results

We perform experiments on three publicly available benchmark datasets: UCSD Ped2 (Li et al., 2013), CUHK Avenue (Lu et al., 2013), and ShanghaiTech (Luo et al., 2017a). Each dataset has a standard training / test division, where the training set consists of only normal videos while testing set has videos with one or more anomalous events (refer to Section 1.4.1). We evaluate with the frame-level area under the receiver operating characteristic (AUROC) metric. Precisely, we use “AUROC on all videos” (micro-averaged AUROC) computed on the whole test set with thresholds common to all test videos. Refer to Section 3.5.2.1 for more details.

5.4.1 Implementation details

For both training and testing in each dataset, the frames are resized to 256 × 256 and
pixels are rescaled to the range [-1, 1]. The autoencoder takes a video clip of 16 frames in
grayscale as input, i.e., T=16, D=1, H=256 and W=256 respectively. While training, the
input clip for PRP task is chosen between original playback rate and accelerated playback
rate with equal chance (50% probability for each). For each batch of accelerated playback
rate, the value of s is chosen randomly from (2, 3, 4, 5) with equal chance for the four
values. The balance weights in the training objective function are set to λ₁=0.6, λ₂=0.4
and λ₃=1, respectively. The whole model is trained end-to-end using the Adam optimizer
(Kingma et Ba, 2015) with a learning rate of 10⁻⁴ and a batch size of 16. While testing,
a single video clip is used for the three tasks (see Figure 5.2). We use different measures
like MSE, PSE and BPE for the anomaly score. For PSE and BPE, we use σ=1, b=2
while keeping the same kernel size of k=5. For blur kernel, we use a Gaussian filter. After
grid search, we set the optimal weights for anomaly score (α₁, α₂, α₃) as (0.1, 0.8, 1), (0.1,
1, 0.1) and (0.2, 0.2, 0.9) for Ped2, Avenue and ShanghaiTech dataset, respectively.

5.4.2 Quantitative results

In this subsection, we present a quantitative comparison of our method with existing
state of the art methods. Table 5.1 shows this quantitative results. The listed methods
belong to the category of VAD without training labels, with and without external object
detectors. As explained in Section 2.5, methods with external supervision via pre-trained
object detectors should not be directly compared with fully unsupervised VAD methods
and most work respect this notion (Astrid et al., 2021a; Park et al., 2022). Therefore,
even though we list object-centric methods for comparison purpose, our discussion below
will solely focus on fully unsupervised methods.

In the fully unsupervised section of the table, the methods marked with * denote non-
reproducible results, i.e., the originally claimed results were found to be incorrect. These
works have been re-implemented by Menon et Stephen (2021) and Lu et al. (2022), and
the new results have been updated in the table.
<table><caption>Table 5.1: Quantitative comparison with the existing state of the art methods: AUROC (%) for VAD is computed on Ped2, Avenue and Shanghai datasets. Numbers in bold indicate the best performance, and * indicate non-reproducible results.</caption><thead><tr><th></th><th>Methods</th><th>Ped2</th><th>Avenue</th><th>Shanghai</th></tr></thead><tbody><tr><td rowspan="5">Object-centric</td><td>OCAE (Ionescu et al., 2019)</td><td>94.30</td><td>87.40</td><td>78.70</td></tr><tr><td>Any-Shot (Doshi et Yilmaz, 2020a)</td><td>97.80</td><td>86.40</td><td>71.62</td></tr><tr><td>VEC (Yu et al., 2020)</td><td>97.30</td><td>89.60</td><td>74.80</td></tr><tr><td>SSMTL (Georgescu et al., 2021a)</td><td>92.4</td><td>91.50</td><td>82.40</td></tr><tr><td>Back-Agnostic (Georgescu et al., 2021b)</td><td><b>98.70</b></td><td><b>92.30</b></td><td><b>82.70</b></td></tr><tr><td rowspan="15">Fully unsupervised</td><td>AnoPCN (Ye et al., 2019)</td><td>96.80</td><td>86.20</td><td>73.60</td></tr><tr><td>MemAE (Gong et al., 2019)</td><td>94.10</td><td>83.30</td><td>71.20</td></tr><tr><td>UNet-inte (Tang et al., 2020)</td><td>96.30</td><td>85.10</td><td>73.00</td></tr><tr><td>Cluster AE (Chang et al., 2020)</td><td>96.50</td><td>86.00</td><td>73.30</td></tr><tr><td>MNAD (Park et al., 2020) *</td><td><b>97.00*</b></td><td><b>88.50*</b></td><td><b>70.50*</b></td></tr><tr><td>MNAD (Menon et Stephen, 2021)</td><td>96.33</td><td>87.91</td><td>67.81</td></tr><tr><td>STEAL Net (Astrid et al., 2021b)</td><td>98.40</td><td>87.10</td><td>73.70</td></tr><tr><td>LNTRA (Astrid et al., 2021a)</td><td>96.50</td><td>84.91</td><td>75.97</td></tr><tr><td>MPN (Lv et al., 2021) *</td><td><b>96.90*</b></td><td><b>89.50*</b></td><td><b>73.80*</b></td></tr><tr><td>MPN (Lu et al., 2022)</td><td>96.13</td><td>83.90</td><td>73.00</td></tr><tr><td>ITAE (Cho et al., 2022)</td><td>97.30</td><td>85.80</td><td>74.70</td></tr><tr><td>VQU-Net (Szymanowicz et al., 2022)</td><td>89.20</td><td>88.30</td><td>-</td></tr><tr><td>FastAno (Park et al., 2022)</td><td>96.30</td><td>85.30</td><td>72.20</td></tr><tr><td>LUSS-AE [ours]</td><td><b>98.52</b></td><td><b>89.04</b></td><td><b>81.21</b></td></tr></tbody></table>

We can observe that our method outperforms all the other methods across all the datasets. The performance gain in Ped2 and Avenue datasets is less significant as in the Shanghai dataset. In fact, it has been suggested to not use the Ped2 dataset as the performance in it is almost saturated (Acsintoae et al., 2022). The Shanghai dataset is considered one of the most difficult dataset and our high performance gain of 5.24% in comparison to the best method in state of the art, demonstrates the viability of our method. Furthermore, the fact that our method works on all the datasets, irrespective of the type of anomalies, shows its generalizing ability. Even though, we use the same architecture for autoencoder like many other methods (Astrid et al., 2021a,b; Gong et al., 2019), still our proposed method outperforms them, without using any sort of external memory or supervision. Even though our method focus on end to end detection without external supervision, still we obtain competitive performance, equivalent to object-centric methods. All these points demonstrate the strength and effectiveness of our LUSS-AE method.
5.4.3 Qualitative results

In this part, we discuss the strengths and weaknesses of our method via visual examples.

<figure><img src="image_60.png" /><figcaption>Figure 5.8: LUSS-AE working illustration on video 11_0176 of Shanghai test set. Anomaly scores (*A*<sub>VCR</sub>, *A*<sub>FFP</sub>, *A*<sub>PRP</sub>, *A*) are plotted per video frame; red regions depict anomalous events and some illustrative frames are shown above the plot, where the yellow and red bounding boxes exhibit objects of interest and anomalies respectively.</figcaption></figure>

Figure 5.8 demonstrates an example of our method tested on a video with two anomalous events, both containing movement of bikes. Here, the people move with relatively normal speed while bikes move with fast speed. Also, the number of people are relatively less in this example and bike does occupy a big area, which means its displacement causes a big spatio-temporal change. We can observe that as soon as the bike enters the scene, we have a high jump in *A*<sub>PRP</sub> and it remains high until the bike exits the scene. It jumps up again in next scene and have highest value when two bikes move in the scene. Regarding *A*<sub>VCR</sub> and *A*<sub>FFP</sub>, the scores remain high when bikes are in the scene. Overall, our method detects both anomalous events in this example.

<figure><img src="image_61.png" /><figcaption>Figure 5.9: LUSS-AE working illustration on video 05_0023 of Shanghai test set. Anomaly scores (*A*<sub>VCR</sub>, *A*<sub>FFP</sub>, *A*<sub>PRP</sub>, *A*) are plotted per video frame; red regions depict ground truth anomalous events, and some illustrative frames are shown above the plot, where the yellow and red bounding boxes exhibit objects of interest and anomalies respectively.</figcaption></figure>

Figure 5.9 demonstrates another example containing three different anomalous events:
person turning in wrong direction and person jumping, brawling/chasing action, and stone picking. We can observe that the problem is difficult in this example as not only the anomalies are spatio-temporal but they also depend on context. In the first anomalous event, $A_{VCR}$ and $A_{FFP}$ have higher values than $A_{PRP}$ in the beginning. The anomaly here consists of person turning in wrong direction which is well captured by the VCR and FFP task. Later, when the person jumped, $A_{PRP}$ suddenly increases, indicating its sensitivity to abrupt motion. During the second anomalous event, we observe that $A_{PRP}$ has higher values than $A_{VCR}$ and $A_{FFP}$, thus contributing primarily to detect the anomaly. The $A_{PRP}$ starts to increase just before the start of this event because the person in red starts to suddenly approach the other person. We then observe a first peak of $A_{PRP}$ as one person pushes other to the ground. We later observe a big second peak of $A_{PRP}$ and this relates to fast movement chasing. However, the third event is very rare (picking up stones) and does not contain large spatio-temporal movement in the scene and thus our method fails to detect it. In fact, the score in later frames is slightly higher than the third event because there are three people in close proximity, trying to change directions, which is considered anomalous for Shanghai dataset. Overall, the VCR and FFP tasks work better in anomalies without abrupt motions and PRP task addresses the anomalies with sudden motions. There is still a room to improve the spatio-temporal comprehension of AE for VAD, possibly with a data augmentation technique as the datasets lack more examples of targetted scenarios.

### 5.4.4 Ablation studies

In this subsection, we study the importance of different parts of our proposed method.

#### 5.4.4.1 Are all tasks useful ?

In this ablation study, we analyze the impact of different tasks on the autoencoder for detecting anomalies. Table 5.2 shows combinations of different tasks and their respective AUROC performances on Avenue and Shanghai datasets. Concretely, for each of these combinations, we train and test AE using the chosen tasks, and use the introduced BPE for computing anomaly score during inference.

<table><caption>Table 5.2: Influence of different tasks (VCR, FFP and FRP) used during training and testing of AE on Avenue and Shanghai datasets, in terms of AUROC (%).</caption><thead><tr><th colspan="3">Tasks</th><th colspan="2">AUROC (%)</th></tr><tr><th>VCR</th><th>FFP</th><th>PRP</th><th>Avenue</th><th>Shanghai</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td>82.72</td><td>73.11</td></tr><tr><td>✓</td><td>✓</td><td></td><td>87.95</td><td>76.35</td></tr><tr><td>✓</td><td></td><td>✓</td><td>87.43</td><td>79.20</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td><strong>89.04</strong></td><td><strong>81.21</strong></td></tr></tbody></table>

We observe that when AE is trained only with the VCR task, the VAD performance is the least, i.e., 82.72% and 73.11%. We consider this as the baseline since this a standard task in VAD (Kiran et al., 2018) and we combine it with other tasks to assess their impact. Using VCR and FFP tasks together, boost the performance with a gain of 5.23%
and 3.24% over the baseline, indicating the importance of learning propagation of normal spatio-temporal patterns via the FFP task. Similarly, when PRP task is trained together with the VCR task, we observe an increase of 4.71% and 6.09% in comparison to baseline, validating that indeed playback speed perception enriches the comprehension of normality for anomaly detection. Finally, when all the tasked are used together, we observe a substantial yield in performance with 6.32% and 8.10% over the baseline, demonstrating the effectiveness of our proposed approach to train AE by leveraging unsupervised and self-supervised tasks.

### 5.4.4.2 Is the autoencoder enriched by FFP and PRP?

In this work, we employ a 3D CAE used in many previous works (Astrid et al., 2021a,b; Gong et al., 2019). All these works used it with the VCR task. In this study, we first reproduced their work by training and testing 3D CAE with the VCR task. Then, we trained it with the proposed tasks, i.e., VCR, FFP and PRP, and tested using only the VCR task. This way, we can assess the impact of these tasks on AE's comprehension of normality during training to detect anomalies during testing.

<table><caption>Table 5.3: Influence of tasks (VCR, FFP and PRP) used during training of autoencoder on Avenue and Shanghai datasets, when tested only with VCR, in terms of AUROC (%).</caption><thead><tr><th rowspan="2">Training tasks</th><th colspan="2">Testing only with VCR</th></tr><tr><th>Avenue</th><th>Shanghai</th></tr></thead><tbody><tr><td>VCR</td><td>82.72</td><td>73.11</td></tr><tr><td>VCR + FFP + PRP</td><td><strong>86.48</strong></td><td><strong>75.37</strong></td></tr></tbody></table>

Table 5.3 exhibit this ablation study. It can be observed by training the same autoencoder with proposed tasks, a gain of 3.76% and 2.26% is obtained on Avenue and Shanghai datasets, respectively. Since the three tasks are used only during training and only VCR is used while testing, this gain in performance reveals that these tasks enriched the normality comprehension of the 3D CAE.

### 5.4.4.3 Are error measures equivalent?

The goal of this ablation study is to see the effect of different error measures, i.e., MSE, PSE and BPE, on the VAD task. Since these measures apply to VCR and FFP tasks, PRP task will not be considered here.
<figure><img src="image_62.png" /><figcaption>Figure 5.10: Illustration of different error measures on a test frame of the Avenue dataset. From left to right: actual frame (ground truth), predicted frame, error frame for MSE, for PSE and for BPE. (Best viewed in color)</figcaption></figure>

Figure 5.10 shows an visual example using the FFP task to better understand these measures. We can observe that AE did not correctly predict this frame, where dropping bag is an anomaly. The error frame for MSE highlights this anomalous region of the image but it also captures the background noise of the frame. The PSE error frame has a more visible region of anomaly, and it smooths some background noise. Finally, the BPE error frame has a principal focus on anomalous region and has the least amount of background noise. Furthermore, the BPE frame is smaller than other maps as we remove irrelevant pixels via subsampling.

<table><caption>Table 5.4: Influence of error measure (MSE, PSE and BPE) on each task (VCR and FFP) during testing of AE on Avenue dataset, in terms of AUROC (%).</caption><thead><tr><th rowspan="2">Error measure</th><th colspan="2">Testing tasks</th></tr><tr><th>VCR</th><th>FFP</th></tr></thead><tbody><tr><td>MSE</td><td>84.95</td><td>85.38</td></tr><tr><td>PSE</td><td>85.96</td><td>86.61</td></tr><tr><td>BPE</td><td><strong>86.48</strong></td><td><strong>87.00</strong></td></tr></tbody></table>

Table 5.4 shows the quantitative impact of these error measures. To be precise, we train our method with the three tasks and test it with the respective tasks and measures shown in the table. We can observe that the PSE improves the performance in both tasks with 1.01% and 1.23% respectively, signifying the importance of proximity error and noise reduction. BPE provides a significant boost in results with 1.53% and 1.62% performance improvement over MSE in the two tasks. This validates the importance of our proposed BPE for the VAD task.

## 5.5 PID: Experiments and results

We perform experiments on the i-LIDS sterile zone dataset with the two cameras views as described in Table 1.2. Using only non-intrusion videos, model was trained on view 1 of i-LIDS training set and tested on view 1 of i-LIDS test set (same procedure for view 2). Test set uses the same protection perimeter as defined in the last chapter. Since, in the last chapter, we concluded the importance of edge-level (EL) protocol for PID evaluation, we use only this protocol in this chapter. Finally, we do not demonstrate the
ablation studies like in VAD (refer Section 5.4.4) because we had similar observations and thus explanations would be repetitive. In the next subsections, we first describe the implementation details of LUSS-AE-Adapt for PID, followed by overall results and how it differs from PIDS proposed in the last chapter. Finally, in the last subsection we provide a summary of how our method performs in a real-world PID case.

## 5.5.1 Implementation details

Each frame is converted to grayscale, pre-processed using histogram equalization and pixels were rescaled to [-1, 1]. The autoencoder we use takes a video clip of 8 frames as input, i.e., $T=8$, $D=1$, $H=576$ and $W=720$ respectively. Like VAD, the input clip for PRP task is chosen between original playback rate and accelerated playback rate with equal chance. For accelerated playback rate, s is chosen randomly from (2, 3, 4, 5) with equal chance for the four values. Concerning training objective function, the balance weights are experimentally set to $\lambda_1=0.6$, $\lambda_2=0.4$ and $\lambda_3=1$ respectively. The whole model is trained end-to-end using the Adam optimizer (Kingma et Ba, 2015) with a learning rate of $10^{-4}$ and a batch size of 24.

While testing, we used the BPE measure for the overall test score. For blur kernel, we use a Gaussian filter with $\sigma=1$, $b=2$ and kernel size $k$ of 5. After grid search, we set the optimal weights for final score ($\alpha_1, \alpha_2, \alpha_3$) as (1, 1, 0.1). Finally, the adaptive thresholding is done on moving z-score of A. We use values of first 10 frames for initialization, moving z-score is used with $z_{th} = 4.5$ and $\alpha = 0.01$, like work in previous chapter. Following the last chapter, for edge-level evaluation, we set the tolerance variables $m_{pre}$ and $m_{post}$ to 3, i.e., less than 1 second of tolerance. For variable $m$, the following values were chosen: 5, 10 and 50. This signifies that we tested the methods to detect within 1, 2 and 10 s from the beginning of intrusion.

## 5.5.2 Results

Tables 5.5 and 5.6 show the comparative results for PID in View 1 and View 2 of i-LIDS datasets respectively. The first three methods mentioned here are well defined in Section 4.3.1.2 and were used in evaluation in the last chapter. Their results shown in these tables are taken from the last chapter.

<table><caption>Table 5.5: PID methods comparison based on edge-level (EL) evaluation at 1, 2 and 10 seconds for View 1 of i-LIDS dataset.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">EL (1 s)</th><th colspan="3">EL (2 s)</th><th colspan="3">EL (10 s)</th></tr><tr><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>GOFPID</td><td>0.69</td><td>0.61</td><td>0.65</td><td>0.76</td><td>0.87</td><td>0.81</td><td>0.78</td><td>0.98</td><td>0.87</td></tr><tr><td>Nayak et al. (2019)</td><td>0.62</td><td>0.52</td><td>0.57</td><td>0.70</td><td>0.74</td><td>0.72</td><td>0.74</td><td>0.93</td><td>0.83</td></tr><tr><td>AE-Adapt</td><td>0.90</td><td>0.74</td><td>0.81</td><td>0.92</td><td>0.90</td><td>0.91</td><td>0.92</td><td>0.91</td><td>0.92</td></tr><tr><td>LUSS-AE-Adapt</td><td>0.91</td><td>0.78</td><td><strong>0.84</strong></td><td>0.93</td><td>0.91</td><td><strong>0.92</strong></td><td>0.95</td><td>0.93</td><td><strong>0.94</strong></td></tr></tbody></table>
<table><caption>Table 5.6: PID methods comparison based on edge-level (EL) evaluation at 1, 2 and 10 seconds for View 2 of i-LIDS dataset.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">EL (1 s)</th><th colspan="3">EL (2 s)</th><th colspan="3">EL (10 s)</th></tr><tr><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>GOFPID</td><td>0.80</td><td>0.59</td><td>0.68</td><td>0.83</td><td>0.73</td><td>0.78</td><td>0.86</td><td>0.94</td><td>0.90</td></tr><tr><td>Nayak et al. (2019)</td><td>0.99</td><td>0.47</td><td>0.64</td><td>0.99</td><td>0.69</td><td>0.82</td><td>0.99</td><td>0.94</td><td><b>0.96</b></td></tr><tr><td>AE-Adapt</td><td>0.83</td><td>0.64</td><td>0.73</td><td>0.89</td><td>0.80</td><td>0.84</td><td>0.90</td><td>0.92</td><td>0.91</td></tr><tr><td>LUSS-AE-Adapt</td><td>0.85</td><td>0.70</td><td><b>0.77</b></td><td>0.90</td><td>0.82</td><td><b>0.86</b></td><td>0.94</td><td>0.93</td><td>0.93</td></tr></tbody></table>

We can observe that the proposed LUSS-AE-Adapt outperforms AE-Adapt in both views. In View 1, we obtain the state-of-the-art results with a good margin in terms of $F_1$ scores in every edge-level metric. Concerning View 2, we still outperform other methods in 1 and 2 seconds of edge-level evaluation and we are just slightly behind Nayak et al. (2019) in 10 seconds. This means that we detect better than other methods in online scenario, i.e., 1 or 2 seconds. It can also be observed that LUSS-AE-Adapt have constantly better precision and recall scores than AE-Adapt, regardless of edge-level evaluation parameters or camera view. This signifies that indeed using our proposed LUSS-AE-Adapt for PID increases the spatio-temporal comprehension of normality, in order to better detect intrusions.

### 5.5.3 Importance of adaptiveness in proposed PID methods

Even though the same 3DCAE is used in both AE-Adapt and LUSS-AE-Adapt, we would like to explore where exactly these two methods differ in terms of PID performance. Table 5.7 shows the comparative performances of these two methods, with and without moving z-score, on View 1 of i-LIDS dataset. Please note that without moving z-score, AE-Adapt or LUSS-AE-Adapt loses the adaptiveness (Adapt) and boils down to just autoencoders.

<table><caption>Table 5.7: Comparison (EL (10 s)) of proposed unsupervised PID methods on i-LIDS dataset, taking into account the effect of moving z-score.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">Without moving z-score</th><th colspan="3">With moving z-score</th></tr><tr><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>AE-Adapt</td><td>0.54</td><td>0.44</td><td>0.49</td><td>0.92</td><td>0.91</td><td>0.92</td></tr><tr><td>LUSS-AE-Adapt</td><td>0.62</td><td>0.51</td><td><b>0.56</b></td><td>0.95</td><td>0.93</td><td><b>0.94</b></td></tr></tbody></table>

We can observe that LUSS-AE-Adapt is 7% better than AE-Adapt when no moving z-score is used. This means that using carefully chosen tasks increases the intrusion detection ability of the autoencoder. When moving z-score is used, both the methods get massive gains in performances. Moreover, the gap in their performances lowers down drastically. This indicates that adaptive thresholding is essential for a PID when long time-length videos are present. Even though autoencoder can detect the intrusions locally (for example, if small videos are taken like in VAD datasets), thresholding it's reconstruction error globally for long-length videos leads to poor performance. This is because even
though reconstruction error is high locally for intrusions in different scene dynamics like
day and night, but the gap between these errors is very large and thus one cannot find
a fixed optimal threshold to detect all the intrusions. Thus, the adaptive component is a
must and it increases the ability of detecting intrusion in both PID systems.

5.5.4 Testing proposed PIDS in a more challenging real-world scenario

Until now, all experiments concerning intrusion detection were done in i-LIDS dataset. As already explained in previous chapters, even though this dataset pose some interesting challenges, it still lacks in many ways, like it only has people as intruders, the scenes are simple and most of the time if something is moving, it is an intrusion (thus lack of distractions). To address this, we performed experiments on a private industrial dataset. This dataset has sheeps roaming in a perimeter continuously and the aim is to detect human intruder which can walk, run, crawl, etc. Moreover, the intruder is present at various distances from the camera and at a very far distance, it appears so small that it is very difficult to recognize. We tested our best proposed method, i.e., LUSS-AE-Adapt, an industrial PIDS (an evoloved form of GOFPID) and method of Nayak et al. (2019) on this. We found that our method outperforms the two methods with a huge margin of 26% and 31% respectively. This proves the viability of our method which learns from the data itself in an end-to-end and unsupervised manner. The performance in this dataset validates that our approach is very suitable for real-world perimeter intrusion detection problem. Please refer to Annex A for detailed dataset description, methods analysis and results.

5.6 Conclusion

In this chapter, we tackled the problem of detecting video anomalies and intrusions with-
out annotations. To address this, we proposed a novel scheme that leverages unsupervised
and self-supervised learning on a single autoencoder. Our method is end-to-end trained
on the normal data and jointly learns to discriminate anomalies / intrusions from normal-
ity using three chosen tasks: (i) unsupervised video clip reconstruction; (ii) unsupervised
future frame prediction; (iii) self-supervised playback rate prediction. The video clip re-
construction task is meant to learn spatio-temporal characteristics of normal videos. The
future frame prediction task is designed to learn the propagation of spatio-temporal pat-
terns in the normal videos. Finally, the playback rate prediction task is for strengthening
the speed understanding of the encoder. It was the first time when the PRP was adapted
for VAD or PID tasks. To focus on anomalous / intrusional regions in the video, we also
proposed a new error measure, called the blur pooled error (BPE) and a robust rescaling
of anomaly scores.

When the proposed LUSS-AE is applied to VAD task, we outperformed the state-of-
the-art methods in three major datasets. The high performance gain in comparison to
previous methods on the Shanghai dataset proved the significance of our method. This
is because the Shanghai dataset is considered one of most difficult VAD dataset with
thirteen different scenes and various anomaly types. Our experiments demonstrate that
the chosen tasks enriched the spatio-temporal comprehension of the autoencoder to better
understand the normality for detecting anomalies. Our ablation study demonstrated the
importance of each task and we found that the PRP task plays a major role in LUSS-AE. Furthermore, a significant boost in performance with respect to MSE, showed the significance of the BPE as it removes the background noise by keeping only the pertinent pixels.

Concerning application of LUSS-AE to PID, we found that we outperform all other methods in the i-LIDS dataset. We found through an ablation study that indeed LUSS-AE leads to a significant improvement in autoencoder performance for PID, but this effect is less visible when we use the adaptive thresholding using moving z-score. This indicates that adaptive thresholding is essential for PIDS when there are long-length videos. We further tested our proposed approach on a real-world industrial dataset having sheeps as distractors and we found that our approach outperforms existing approaches with a huge margin. All this validated the viability of our approach for detecting intrusions.

Overall, we can conclude that our proposed LUSS-AE is generic and can be successfully applied to both VAD and PID tasks and leads to superior performances in both tasks. However, there is still a room for improvement. Concerning VAD, there are still complicated cases (like anomaly at far distance or anomaly with low spatio-temporal movement), where our method fails. We need to further improve our approach by proposing new proxy tasks, or by using modules like attention. Training network using BPE also needs to be investigated. Concerning the PID task, we have similar shortcomings. Therefore, we need to further improve our approach for obtaining an even better VAD / PID system.
# Conclusion and perspectives

In this thesis, we explored unsupervised deep learning based approaches for learning spatio-temporal representations of videos. These approaches were developed for application in video surveillance domain and in particular for video anomaly detection task and perimeter intrusion detection task. We first formally defined these tasks in chapter 3, along with suitable evaluation protocols. Then, in chapter 4, we proposed an unsupervised autoencoder based method for perimeter intrusion detection. It was coupled with an adaptive thresholding strategy to handle changing scene dynamics. Finally, we also proposed a new generic method in chapter 5. With carefully chosen tasks, it leverages unsupervised and self-supervised learning in a single autoencoder. This approach is applicable to both video anomaly detection and perimeter intrusion detection task. All the proposed approaches were compared with state of the art methods on standard datasets and obtained superior performances.

## Importance of tasks definition and suitable evaluation

Before solving any problem, it is essential to correctly understand it first and this requires defining it as precisely as possible. That is why, in chapter 3, we proposed mathematical definitions for the two main tasks that we address in this thesis along with corresponding evaluation protocols.

Concerning perimeter intrusion definition (PID), we first defined how an intrusion event is triggered by an object. An object causes an intrusion event if it belongs to a non-authorized class (like person, car, etc.) and is moving in a protected area during a prohibited time interval. The classes considered as non-authorized, the protected area and the prohibited time interval depend on the end user or client and can vary from one site to another. We can have multiple objects causing intrusion events and these events can also overlap. For example, if a car causes an intrusion event and then after some time, a person also triggers an intrusion event (when car is still there), then these two events overlap. In the case of PID, at any instant, we are concerned if there is an intrusion event, regardless of whether one object or many objects are causing it. Therefore, it is safe to merge intrusion events when they overlap. Consequently, the intrusion event of the whole video is just the union of its intrusion events. The constituent frames of the intrusion event of video are called the intrusion frames. The main concern in PID are the time intervals in video where intrusions occur. We define these time intervals using intrusion intervals, where an intrusion interval corresponds to a contiguous sequence of intrusion frames. The goal of PID task is to detect the beginning of intrusions intervals of the video. The beginning is essential, because if an intrusion is detected too late, then that detection is not very useful in terms of PID requirements (for client/user, the intruder is too close or inside the protected site) (i-LIDS Team, 2006; Buch et Velastin, 2014). This
real-life constraint must be considered by the evaluation protocol. The i-LIDS evaluation protocol adhered to this constraint and counted a detection as valid if it is within the first 10 seconds of actual intrusion. The main problem with this evaluation protocol is that it strongly penalizes the system. It treats detections as false alarms if they occur after 10 seconds from intrusion beginning, even if the alarms are within the intrusion. And while doing so, it does not consider the duration of intrusion. For example, if an intruder enters a site and stay there for 5 minutes and an alarm is triggered at 12th second after the beginning of intrusion. Then, the i-LIDS evaluation protocol will count it as a false alarm, even though the alarm is triggered during the intrusion which would stay in site for almost 5 minutes. We addressed drawbacks of this protocol and introduced an edge level evaluation protocol. Our evaluation protocol adheres with real-life constraints of detecting the beginning of intrusions. Moreover, it has tolerance limits of few frames before and after the intrusion, which permits not to adversely penalize a system if an intrusion is detected a little bit in advance (like a system might detect a person as soon as it steps on the scene, while annotation is when the whole person is in the scene) or when the alarm takes some time to stop after the intrusion.

Defining video anomaly detection is not trivial since anomalies depend on context and they differ from one dataset to another. In this thesis, we defined VAD on the basis of available annotations. If annotations of both normal and abnormal events are present during training, then the VAD problem boils down to binary classification, i.e., understanding the two classes (normal and anomalies) with training and then detecting anomalies during testing. However, realistically, it is very difficult to have annotations during training since anomaly is a rare event and it is practically impossible to have annotations for all possible future anomalous activities. Therefore, the second case is more realistic, where it is considered that training data primarily contains normal (non-anomalous) videos. We mathematically describe this case using the probably approximately correct (PAC) learning framework and consequently anomalies are defined as the complement of normality distribution present in the training data. This definition takes into account the anomaly context, since anomalies are defined according to the normal distribution provided with the dataset. Furthermore, our definition takes care of debated cases for VAD with the uncertainty parameter of PAC framework. For example, in ShanghaiTech dataset, normal activities like eating ice-creams are present only in the testing set and not the training set. Therefore, even though there is high probability for them to be abnormal, we cannot be fully certain that they are anomaly due to the uncertainty parameter. For evaluating VAD methods, we used the frame level binary class evaluation. This is done to be comparable with the state-of-the-art methods, all of which use this evaluation protocol.

Even though our PAC formulation addressed some complicated cases in VAD, there is still a need for a new dataset with more precise anomaly annotation (during testing). The training data should be representative of normalities and any deviation from it should define some degree of anomaliness. Concerning evaluation, anomalies are also events and like PID, we want to finally know if these events are detected or not. Frame-level evaluation does not provide much information for anomalous event detection. For example, with frame level evaluation, we cannot differentiate between a method that detected some frames of all anomalous events, with the one that detected only few events having large number of frames. Therefore, an event level evaluation should also be used in VAD. For this, datasets like Shanghai and Avenue should be manually annotated at event level. Concerning event level evaluation, we focused on detecting the beginning of event, but our proposed protocol can be extended for detecting the complete event if required. In
some cases, researchers might want to detect if the alarm is raised during the whole event. In that case, our protocol cannot work since it is dedicated to detect rising edges and not the frames of an event. Therefore, a suitable evaluation protocol should be developed in that case. Similarly, we did not penalize re-detections in our evaluation as long as it is during the event. But one can penalize them to some extent, depending on the system output requirements.

Unsupervised spatio-temporal video understanding

In this thesis, we explored spatio-temporal learning in videos for unsupervised tasks. We first proposed a strided 3D convolutional autoencoder (S3DCAE) for this, in chapter 4. To learn spatio-temporal features in a joint manner from videos, we used only 3D convolutions and 3D deconvolutions. The encoder layers generally use pooling operation for compressing information, while decoder layers use the unpooling operation. We instead proposed to use strided convolutions and deconvolutions, therefore our architecture was light in memory and fast in execution. The proposed autoencoder learns normality during training as it is trained only on normal videos and detects intrusions as they are badly reconstructed (having high reconstruction error) during testing. Therefore, it is capable to understand spatio-temporal normality and detect intrusions (or even anomalies) in an unsupervised way.

The second method that we proposed, called LUSS-AE, leverages unsupervised and self-supervised learning using a single autoencoder for spatio-temporal feature learning. It was proposed in Chapter 5. Like the previous approach, the aim is to learn normality during training, for detecting anomalies/intrusions during testing. The main idea is to enrich the spatio-temporal comprehension of an autoencoder using different tasks. We proposed three tasks: video clip reconstruction (VCR), future frame prediction (FFP) and playback rate perception (PRP). VCR is meant to learn spatio-temporal characteristics, FFP is designed to learn the propagation of spatio-temporal patterns, and PRP is intended to strengthen the speed understanding of the encoder. We demonstrated that these tasks enhanced the spatio-temporal comprehension of the autoencoder. With successful application on both VAD and PID tasks, we showed its effectiveness and generic behavior.

Notice that one issue with LUSS-AE is that the PRP task has only two classes: original and accelerated. We group different speeds like 2x, 3x and 4x into a single class (accelerated). Semantically, these different speeds have different spatio-temporal information and therefore they should be in different classes. We can further add cases like slow speed (0.5x) and even negative speed, implying reverse playback. This will make LUSS-AE even more speed aware. Another idea could be to use speed regression instead of speed classification. In real life, often the speed cannot be discretely categorized as 2x or 3x, etc., but can be represented by a continuous value. In speed classification, each speed like 2x or 3x, were treated independently but there is a semantic relation among speeds, e.g., speed of 2x and 3x are more closer to each other than speed of 5x. The speed regression will permit the network to understand the semantics among different speeds and it should enrich its spatio-temporal comprehension by making it even more speed aware. Finally, more tasks can be added to our AE like backward / forward video clip prediction, masking input and predicting it, input flip or rotation prediction, etc. Attention modules can further enhance the spatio-temporal comprehension, but they do require extra computational / memory capacity. There is also a new family of methods that use a transformer network
(Bertasius et al., 2021; Neimark et al., 2021). These methods are now used extensively in spatio-temporal video learning, especially in self-supervised learning, as a pre-training step. However, they need a very large amount of GPU / CPU computational capacity and long time to train. For instance, video transformer network (Neimark et al., 2021) has 114 million parameters with 4,218 GFLOPs, while our proposed LUSS-AE for VAD has 6.12 million parameters with 16.30 GFLOPs only. Even though smaller and quicker variants of this new family of methods are coming, we are still far to have a model that runs for real-time tasks like video anomaly detection and perimeter intrusion detection.

## Adaptiveness for online outdoor event detection

Since we focused on detecting intrusion or anomalies in the outdoor environment, the obvious challenge is to face changing scene dynamics like weather, temperature, lighting conditions, etc. Any detection system must adapt itself to these changes, otherwise it will underperform. This challenge is particularly dominant for long length videos, like we have in the i-LIDS dataset. To address this challenge, we proposed an adaptive mechanism using a moving z-score on the reconstruction error of the autoencoder. It detected intrusions as soon as they occur and then adapted itself with the scene dynamics. It is generic and brought massive gains in performance to both the network architectures we proposed in this thesis. The LUSS-AE with adaptive thresholding (LUSS-AE-Adapt) was also tested in a private dataset. This dataset is very challenging since majority of time, there are sheeps moving in the scene, which is considered a normal behavior and intrusions are caused by human intruders entering the scene in various ways like crawling, rolling, etc. It was filmed with thermal camera, which makes it even more difficult to distinguish intrusions with sheeps. Our adaptive LUSS-AE method outperformed an object detector based method and a private industrial method by a large margin. This validated the importance of adaptiveness.

Notice that when a VAD or PID system is deployed in real-life, it does need to face these changing scene dynamics. To develop these adaptive approaches, we need appropriate datasets. Concerning PID, the i-LIDS dataset does conform to this need, but it lacks different types of intrusions like cars, trucks, skateboard, etc. Furthermore, the two views of this dataset are very similar to each other, they are like mirror images of each other. Also, the amount of distractions is very low in this dataset and thus any person detector system can work well. Therefore, we need more public PID datasets which address the above mentioned weaknesses. Even though we also tested our adaptive approach on a challenging dataset, unfortunately it cannot be made public. In regard to VAD, it is unfortunate that we do not have long length test videos in any current VAD dataset. This would have allowed the development of approaches that adapts themselves with time. Thus, for development of more adaptive approaches, we would require more realistic VAD and PID datasets. Furthermore, different types of camera sensors like thermal, depth, etc. and multiple viewpoints could further help both VAD and PID approaches.

## Towards more generic and deployable models

Our proposed approaches learn in offline mode from training data to detect events during testing. When these approaches are installed in a site, they need to update themselves with the newly observed data, i.e., learning new normalities on the go. Naively updating model
weights with newly observed data can lead to catastrophic forgetting (Kirkpatrick et al., 2017). Another solution is to gather data and re-train the models but this would quickly become infeasible due to computational and storage needs. Thus, we need to preserve the initial knowledge of models without reusing the initial data, and also acclimate the model to newly observed data. To accomplish this, continual learning can be used. Recently, it has been successfully used for VAD (Doshi et Yilmaz, 2020b, 2022). The approaches we proposed are trained on each dataset separately. But how can we use a model trained on one dataset to work on another? Ideally, we would like to just use few frames of the new dataset to adapt the current model. This is precisely what few shot learning does (Lake et al., 2015; Finn et al., 2017). Recently, some works have used it in VAD too (Lu et al., 2020; Lv et al., 2021). This property is extremely desirable since then, we can use a model trained on an offline dataset to a number of sites and adapt to each one of them. As already mentioned, we can improve our model with more suitable tasks. However, the new trend in computer vision is to use transformer like model due to their immense generalization capacity. In future, smaller and faster transformer like models can be used in video surveillance.
# A Testing PIDS in a real-world case

So far, we have evaluated different methods on the i-LIDS dataset. As already mentioned before, this dataset is interesting but it still has some major drawbacks, which makes it an easy task for any simple person detection. The intrusion in real-life are not so easy to detect and furthermore they can be very well non-human entity like wild animals or cars. To address this, we evaluate methods on a private dataset that depicts more challenging conditions like very dynamic background, different distances from camera, change in light and scene dynamics and hence it is more inline with recent challenges in video surveillance.

## Sheeps dataset

This dataset contains videos of a site surveilled by a thermal camera that principally contains sheep. The videos are recorded for many days and therefore present varying light conditions in the scene. Some illustrative frames are shown in Figure A.1.

<figure><img src="image_63.png" /><figcaption>Figure A.1: Example frames from the Sheeps dataset without intrusions. The images show video frames from different time of the day. The sheeps present are moving in random manner. Also, observe the sheeps at the top-right end of the third image.</figcaption></figure>

The intrusions in this dataset are very rare and caused by people. In other words, when some person enters the scene, it needs to be detected. The human intruders enter in various ways like running, walking, crawling, etc. Furthermore, they are situated at different distances from the camera like near, middle and far. Overall, the task is hard since the PIDS method need to ignore sheeps but at the same time detect human intruders who often act like sheep. Some illustrative examples of intrusions in this dataset are shown in Figure A.2.

<figure><img src="image_64.png" /><figcaption>Figure A.2: Example frames from the Sheeps dataset with intrusions shown in red-bounding boxes. The images are shown in increasing level of intrusion detection difficulty.</figcaption></figure>
The videos in this dataset are captured at 5 FPS with frame resolution of 384x288. Since intrusions are very rare in this dataset, the training set does not contain intrusions. The training set contains 50 videos (approx. 20,000 frames) with sheeps at different configurations and having different lighting conditions. In test set, each video contain at least one intrusion. The intrusion can be at different distance from the camera and depicts human crawling, running, etc., accompanied with varied lighting conditions and sheeps. We have in total 36 videos (11,189 frames) of intrusions (7,077 frames). Since each video has different level of difficulty for detecting intrusion, the dataset also comes with video level label from 1 to 5, where 1 is the easiest and 5 is the most difficult case.

## Methods and experimental setup

We compare three methods on this dataset: LUSS-AE-Adapt, Nayak *et al.* (2019) and an industrial PIDS. We chose LUSS-AE-Adapt because it has enhanced spatio-temporal comprehension and performs better than AE-Adapt. Instead of using GOFPID, we use a private industrial PIDS. This is because their functioning is similar (follows similar traditional pipeline) but industrial PIDS performs better than GOFPID and thus is closer to the state of the art practiced by industry. For LUSS-AE-Adapt, we use the same architecture as proposed in Chapter 5. Just the input frame size is updated to 384x288. Other settings remain the same as mentioned in Section 5.5.1. For (Nayak *et al.*, 2019) also, the same setting like before (see Section 4.3.1) as used with human as the intrusion class. The industrial PIDS was also used with appropriate settings. For evaluation, we use the proposed edge-level protocol (EL (10 s)), as we concluded in Chapter 4 that it should be used for evaluating PID methods. For better comprehension, we also show results for some levels of difficulty.

## Results

Table 5.8 presents results of the three methods on the whole Sheeps dataset. We can observe that our proposed method has outperformed the other two methods with a large margin. The $F_1$ score is almost double in comparison to Nayak *et al.* (2019) and has a huge margin **0.26** in comparison to industrial PIDS. We can also observe that our method also has the best precision and recall values. This is important because Nayak *et al.* (2019) also has a good recall value but it is at the cost of high number of false alarms, depicted by the poor precision value.

<table><caption>Table 5.8: PID methods comparison based on edge-level (EL) evaluation at 10 seconds in the Sheeps dataset.</caption><thead><tr><th>Methods</th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Nayak <i>et al.</i> (2019)</td><td>0.23</td><td>0.78</td><td>0.36</td></tr><tr><td>Industrial PIDS</td><td>0.32</td><td>0.57</td><td>0.41</td></tr><tr><td>LUSS-AE-Adapt</td><td><strong>0.57</strong></td><td><strong>0.81</strong></td><td><strong>0.67</strong></td></tr></tbody></table>

To better understand the results, we also provide results at different difficulty levels. For sake of simplicity, we provide the results of only three difficulties, i.e., 1, 2 and 5. The other difficulty have similar results, meaning when difficulty increases performance often
decreases. The difficulty 1 and 2 correspond to easier cases, while the difficulty of level 5 corresponds to the most difficult case (shown in rightmost image of Figure A.2).

<table><caption>Table 5.9: PID methods comparison at different difficulty levels in the Sheeps dataset, where edge-level (EL) evaluation at 10 seconds is used.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">Difficulty - 1</th><th colspan="3">Difficulty - 2</th><th colspan="3">Difficulty - 5</th></tr><tr><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th><th>Precision</th><th>Recall</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Nayak et al. (2019)</td><td>0.21</td><td>1</td><td>0.35</td><td>0.25</td><td>1</td><td>0.40</td><td>0.20</td><td>1</td><td>0.33</td></tr><tr><td>Industrial PIDS</td><td>0.45</td><td>0.83</td><td>0.58</td><td>0.50</td><td>0.77</td><td>0.60</td><td>0.10</td><td>0.22</td><td>0.14</td></tr><tr><td>LUSS-AE-Adapt</td><td>0.86</td><td>1</td><td><strong>0.92</strong></td><td>0.73</td><td>0.89</td><td><strong>0.80</strong></td><td>0.71</td><td>0.56</td><td><strong>0.63</strong></td></tr></tbody></table>

Table 5.9 shows the methods comparison based on different difficulty. We observe that the method of Nayak *et al.* (2019) performs similar for the three distances. In fact, the person object detector seems to fail for distinguishing between person and sheep. That is why, there is a recall of 1 and very low precision. In other words, their system is always raising intrusion alarms (thus perfect recall), irrespective of intruder or sheep, and thus have a very high number of false alarms, depicted with very low precision value. The industrial PIDS performs better than Nayak *et al.* (2019). For difficulty 1 and 2, their precision values are better than the last method, meaning they have fewer false alarms. For difficulty 5, their method performs poorly and is not able to differentiate between intruders and sheeps. For our proposed LUSS-AE-Adapt method, we observe that it performs better than the two methods irrespective of the difficulty. We obtain an excellent $F_1$ score of **0.92** for difficulty 1, which is very interesting for this difficult PID dataset. We also observe that the $F_1$ value decreases with increasing difficulty. The main component of this decrease in scores is due to the recall values. This means that with increase in difficulty, we detect lesser intrusions but we do not raise a lot of false alarms.

## Conclusion

In this annex, we tested our proposed PIDS with different existing methods on a private dataset. This dataset is very interesting because it contains constant movement of sheeps (distractors) in a scene, where human intruder tries to enter in various ways. The change in lighting conditions, arbitrary movement of sheeps, different distances of intruder from the camera, and various ways in which the intruder enters the scene, makes this dataset very challenging. The method of Nayak *et al.* (2019), which has excellent performance in i-LIDS dataset, performs poorly here. This is precisely because their object detector failed to detect human intruder among sheeps. To be precise, their detector mostly detected some sheep as human, thus causing massive false alarms. Therefore, here we observed the risk of using pre-trained detectors for PIDS. Concerning the private industrial PIDS, it performed a bit better than Nayak *et al.* (2019). However, it still fails to detect intrusions at difficult cases and often confuse sheeps with intruders. We found that our proposed LUSS-AE-Adapt method outperformed the two other methods by a good margin. Since our method learned normality, it was able to better differentiate between sheeps and humans. Our adaptive method has an excellent result for easier cases and even with increase in difficulty, our method does not trigger a large number of false alarms. One of the most important aspect is that our method is completely without supervision and learns
to detect intrusions just from the unlabeled data. Overall, we clearly see the viability of
our method in real-world perimeter intrusion detection task, provided we have a dataset
to train the method beforehand.
Bibliography

Andra ACSINTOAE, Andrei FLORESCU, Mariana-Iuliana GEORGESCU, Tudor MARE, Paul SUMEDREA, Radu Tudor IONESCU, Fahad Shahbaz KHAN et Mubarak SHAH : Ubnormal: New benchmark for supervised open-set video anomaly detection. *In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 20143–20153, 2022.

Amit ADAM, Ehud RIVLIN, Ilan SHIMSHONI et Daviv REINITZ : Robust real-time unusual event detection using multiple fixed-location monitors. *IEEE transactions on pattern analysis and machine intelligence*, 30(3):555–560, 2008.

Sos S AGAIAN, Blair SILVER et Karen A PANETTA : Transform coefficient histogram-based image enhancement algorithms using contrast entropy. *IEEE Trans Image Process*, 16:741–758, 2007.

Samet AKCAY, Amir ATAPOUR-ABARGHOUEI et Toby P BRECKON : Ganomaly: Semi-supervised anomaly detection via adversarial training. *In Asian conference on computer vision*, pages 622–637. Springer, 2018.

Nijad AL-NAJDAWI, Helmut E. BEZ, Jyoti SINGHAI et Eran.A. EDIRISINGHE : A survey of cast shadow detection algorithms. *Pattern Recognition Letters*, 33(6):752–764, 2012. ISSN 0167-8655. URL https://www.sciencedirect.com/science/article/pii/S0167865511004326.

Walid S Ibrahim ALI : Real time video sharpness enhancement by wavelet-based lumi-nance transient improvement. *In ISSPA*, pages 1–4, 2007.

G ARAVAMUTHAN, P RAJASEKHAR, RK VERMA, SV SHRIKHANDE, S KAR et Suresh BABU : Physical intrusion detection system using stereo video analytics. *In CVIP*, pages 173–182, 2020.

M ASTRID, M Z ZAHEER, J-Y LEE et S-I LEE : Learning not to reconstruct anomalies. *In BMVC*, 2021a.

M ASTRID, M Z ZAHEER et S-I LEE : Synthetic temporal anomaly guided end-to-end video anomaly detection. *In ICCV*, pages 207–214, 2021b.

AVSS DATASET : http://www.elec.qmul.ac.uk/staffinfo/andrea/avss2007_d.html, 2007.

AXIS COMMUNICATIONS : AXIS Perimeter Defender - User Manual M14.3, 2020.
Moez BACCOUCHE, Franck MAMALET, Christian WOLF, Christophe GARCIA et Atilla BASKURT : Sequential deep learning for human action recognition. *In International workshop on human behavior understanding*, pages 29–39. Springer, 2011.

Dzmitry BAHDANAU, Kyung Hyun CHO et Yoshua BENGIO : Neural machine translation by jointly learning to align and translate. *In 3rd International Conference on Learning Representations, ICLR 2015*, 2015.

Nicolas BALLAS, Li YAO, Chris PAL et Aaron C COURVILLE : Delving deeper into convolutional networks for learning video representations. *In ICLR (Poster)*, 2016.

Olivier BARNICH et Marc VAN DROOGENBROECK : ViBe: a powerful random technique to estimate the background in video sequences. *In ICASSP*, pages 945–948, 2009.

Q BARTHÉLEMY, L MAYAUD, D OJEDA et M CONGEDO : The Riemannian potato field: a tool for online signal quality index of EEG. *IEEE Trans Neural Syst Rehabil Eng*, 27:244–255, 2019.

Quentin BARTHÉLEMY : PyGOFPID is a Python package for good old fashioned perimeter intrusion detection systems for video protection. https://github.com/qbarthelemy/PyGOFPID, 2022.

Steven S. BEAUCHEMIN et John L. BARRON : The computation of optical flow. *ACM Comput Surv*, 27(3):433–466, 1995.

S BENAIM, A EPHRAT, O LANG, I MOSSERI, W T FREEMAN, M RUBINSTEIN, M IRANI et T DEKEL : SpeedNet: Learning the speediness in videos. *In CVPR*, pages 9922–9931, 2020.

Eric P BENNETT et Leonard MCMILLAN : Video enhancement using per-pixel virtual exposures. *In ACM SIGGRAPH*, pages 845–852. 2005.

Gedas BERTASIUS, Heng WANG et Lorenzo TORRESANI : Is space-time attention all you need for video understanding? *In ICML*, volume 2, page 4, 2021.

Alex BEWLEY, Zongyuan GE, Lionel OTT, Fabio RAMOS et Ben UPCROFT : Simple online and realtime tracking. *In ICIP*, pages 3464–3468, 2016.

Christopher M BISHOP et Nasser M NASRABADI : *Pattern recognition and machine learning*, volume 4. Springer, 2006.

Léon BOTTOU et al. : Stochastic gradient learning in neural networks. *Proceedings of Neuro-Nimes*, 91(8):12, 1991.

Thierry BOUWMANS : Recent advanced statistical background modeling for foreground detection-a systematic survey. *Recent Patents on Computer Science*, 4:147–176, 2011.

Thierry BOUWMANS : Traditional and recent approaches in background modeling for foreground detection: An overview. *Computer science review*, 11:31–66, 2014.

Thierry BOUWMANS, Fida EL BAF et Bertrand VACHON : Background modeling using mixture of gaussians for foreground detection-a survey. *Recent patents on computer science*, 1(3):219–237, 2008.
Sebastian BRUTZER, Benjamin HÖFERLIN et Gunther HEIDEMANN : Evaluation of background subtraction techniques for video surveillance. *In CVPR*, pages 1937–1944, 2011.

N BUCH et SA VELASTIN : Human intrusion detection using texture classification in real-time. *Tracking Humans for the Evaluation of their Motion in Image Sequences*, page 1, 2008.

Norbert BUCH et Sergio A VELASTIN : Local feature saliency classifier for real-time intrusion monitoring. *Opt Eng*, 53:073108, 2014.

Quin CAI et Jake K AGGARWAL : Tracking human motion in structured environments using a distributed-camera system. *IEEE Trans Pattern Anal Mach Intell*, 21:1241–1247, 1999.

Joao CARREIRA et Andrew ZISSERMAN : Quo vadis, action recognition? a new model and the kinetics dataset. *In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 6299–6308, 2017.

Eduardo CERMEÑO, Ana PÉREZ et Juan Alberto SIGÜENZA : Intelligent video surveillance beyond robust background modeling. *Expert Syst Appl*, 91:138–149, 2018.

R CHALAPATHY et S CHAWLA : Deep learning for anomaly detection: A survey. *arXiv preprint arXiv:1901.03407*, 2019.

Varun CHANDOLA, Arindam BANERJEE et Vipin KUMAR : Anomaly detection: A survey. *ACM Comput Surv*, 41(3):1–58, 2009.

Y CHANG, Z TU, W XIE et J YUAN : Clustering driven deep autoencoder for video anomaly detection. *In ECCV*, pages 329–345, 2020.

Mark CHEN, Alec RADFORD, Rewon CHILD, Jeffrey WU, Heewoo JUN, David LUAN et Ilya SUTSKEVER : Generative pretraining from pixels. *In International conference on machine learning*, pages 1691–1703. PMLR, 2020.

Peihao CHEN, Deng HUANG, Dongliang HE, Xiang LONG, Runhao ZENG, Shilei WEN, Mingkui TAN et Chuang GAN : Rspnet: Relative speed perception for unsupervised video representation learning. *In Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, pages 1045–1053, 2021.

Kai-Wen CHENG, Yie-Tarng CHEN et Wen-Hsien FANG : Video anomaly detection and localization using hierarchical feature representation and gaussian process regression. *In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 2909–2917, 2015.

MyeongAh CHO, Taeoh KIM, Woo Jin KIM, Suhwan CHO et Sangyoun LEE : Unsupervised video anomaly detection via normalizing flows with implicit latent features. *Pattern Recognit*, 129:108703, 2022.

Yong Shean CHONG et Yong Haur TAY : Abnormal event detection in videos using spatiotemporal autoencoder. *In International Symposium on Neural Networks*, pages 189–196. Springer, 2017.
Junyoung CHUNG, Caglar GULCEHRE, Kyunghyun CHO et Yoshua BENGIO: Empirical evaluation of gated recurrent neural networks on sequence modeling. *In NIPS 2014 Workshop on Deep Learning, December 2014*, 2014.

James L CROWLEY, P REIGNIER et S PESNEL: Caviar context aware vision using image-based active recognition. Rapport technique, University of Edinburgh, 2005.

Navneet DALAL et Bill TRIGGS: Histograms of oriented gradients for human detection. *In CVPR*, volume 1, pages 886-893, 2005.

Ali DIBA, Mohsen FAYYAZ, Vivek SHARMA, M Mahdi ARZANI, Rahman YOUSEFZADEH, Juergen GALL et Luc VAN GOOL: Spatio-temporal channel correlation networks for action classification. *In Proceedings of the European Conference on Computer Vision (ECCV)*, pages 284-299, 2018.

Ali DIBA, Mohsen FAYYAZ, Vivek SHARMA, Amir Hossein KARAMI, Mohammad Mahdi ARZANI, Rahman YOUSEFZADEH et Luc VAN GOOL: Temporal 3d convnets: New architecture and transfer learning for video classification. *arXiv preprint arXiv:1711.08200*, 2017a.

Ali DIBA, Vivek SHARMA et Luc VAN GOOL: Deep temporal linear encoding networks. *In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition*, pages 2329-2338, 2017b.

Nikolaos DIMITRIOU, George KIOUMOURTZIS, Anargyros SIDERIS, Georgios STAVROPOULOS, Evdoxia TAKA, Nikolaos ZOTOS, George LEVENTAKIS et Dimitrios TZOVARAS: An integrated framework for the timely detection of petty crimes. *In EISIC*, pages 24-31, 2017.

Jeffrey DONAHUE, Lisa ANNE HENDRICKS, Sergio GUADARRAMA, Marcus ROHRBACH, Subhashini VENUGOPALAN, Kate SAENKO et Trevor DARRELL: Long-term recurrent convolutional networks for visual recognition and description. *In Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2625-2634, 2015.

Fei DONG, Yu ZHANG et Xiushan NIE: Dual discriminator generative adversarial network for video anomaly detection. *IEEE Access*, 8:88170-88176, 2020.

Keval DOSHI et Yasin YILMAZ: Any-shot sequential anomaly detection in surveillance videos. *In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 934-935, 2020a.

Keval DOSHI et Yasin YILMAZ: Continual learning for anomaly detection in surveillance videos. *In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops*, pages 254-255, 2020b.

Keval DOSHI et Yasin YILMAZ: Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate. *Pattern Recognition*, 114:107865, 2021.

Keval DOSHI et Yasin YILMAZ: Rethinking video anomaly detection-a continual learning approach. *In Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pages 3961-3970, 2022.
Alexey DOSOVITSKIY, Lucas BEYER, Alexander KOLESNIKOV, Dirk WEISSENBORN, Xiaohua ZHAI, Thomas UNTERTHINER, Mostafa DEHGHANI, Matthias MINDERER, Georg HEIGOLD, Sylvain GELLY et al. : An image is worth 16x16 words: Transformers for image recognition at scale. *In International Conference on Learning Representations*, 2020.

Alexey DOSOVITSKIY, Philipp FISCHER, Eddy ILG, Philip HAUSSER, Caner HAZIRBAS, Vladimir GOLKOV, Patrick VAN DER SMAGT, Daniel CREMERS et Thomas BROX : Flownet: Learning optical flow with convolutional networks. *In Proceedings of the IEEE international conference on computer vision*, pages 2758–2766, 2015.

Wenbin DU, Yali WANG et Yu QIAO : Rpan: An end-to-end recurrent pose-attention network for action recognition in videos. *In Proceedings of the IEEE international conference on computer vision*, pages 3725–3734, 2017.

John DUCHI, Elad HAZAN et Yoram SINGER : Adaptive subgradient methods for online learning and stochastic optimization. *Journal of machine learning research*, 12(7), 2011.

Vincent DUMOULIN et Francesco VISIN : A guide to convolution arithmetic for deep learning. *stat*, 1050:23, 2016.

Debidatta DWIBEDI, Pierre SERMANET et Jonathan TOMPSON : Temporal reasoning in videos using convolutional gated recurrent units. *In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops*, pages 1111–1116, 2018.

Kiana EHSANI, Roozbeh MOTTAGHI et Ali FARHADI : Segan: Segmenting and generating the invisible. *In Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 6144–6153, 2018.

Ahmed ELGAMMAL, David HARWOOD et Larry DAVIS : Non-parametric model for background subtraction. *In ECCV*, pages 751–767, 2000.

Ersin ESEN, Mehmet Ali ARABACI et Medeni SOYSAL : Fight detection in surveillance videos. *In 2013 11th International Workshop on Content-Based Multimedia Indexing (CBMI)*, pages 131–135. IEEE, 2013.

T FAWCETT : An introduction to ROC analysis. *Pattern Recognit Lett*, 27:861–874, 2006.

William FEDUS, Barret ZOPH et Noam SHAZEER : Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. *Journal of Machine Learning Research*, 23(120):1–39, 2022.

Christoph FEICHTENHOFER, Haoqi FAN, Jitendra MALIK et Kaiming HE : Slowfast networks for video recognition. *In Proceedings of the IEEE/CVF international conference on computer vision*, pages 6202–6211, 2019.

Christoph FEICHTENHOFER, Axel PINZ et Richard P WILDES : Spatiotemporal multiplier networks for video action recognition. *In Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 4768–4777, 2017.

Christoph FEICHTENHOFER, Axel PINZ et Andrew ZISSERMAN : Convolutional two-stream network fusion for video action recognition. *In Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1933–1941, 2016.
Pedro FELZENSZWALB, David MCAllester et Deva RAMANAN : A discriminatively trained, multiscale, deformable part model. In CVPR, pages 1-8, 2008.

J-C FENG, F-T HONG et W-S ZHENG : MIST: Multiple instance self-training framework for video anomaly detection. In CVPR, pages 14009-14018, 2021.

Chelsea FINN, Pieter ABBEEL et Sergey LEVINE : Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.

Denis FORTUN, Patrick BOUTHEMY et Charles KERVRANN : Optical flow modeling and computation: A survey. Computer Vision and Image Understanding, 134:1-21, 2015.

FOXSTREAM SMART VIDEO ANALYTICS : FoxVigi 3.71.0 software user guide, 2021.

Zhouyu FU, Weiming HU et Tieniu TAN : Similarity based vehicle trajectory clustering and anomaly detection. In IEEE International Conference on Image Processing 2005, volume 2, pages II-602. IEEE, 2005.

Kunihiko FUKUSHIMA et Sei MIYAKE : Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.

Rikke GADE et Thomas B MOESLUND : Thermal cameras and applications: a survey. Mach Vis Appl, 25:245-262, 2014.

Guillermo GALLEGO, Tobi DELBRÜCK, Garrick ORCHARD, Chiara BARTOLOZZI, Brian TABA, Andrea CENSI, Stefan LEUTENEGGER, Andrew J DAVISON, Jörg CONRADT, Kostas DANILIDIS et al. : Event-based vision: A survey. IEEE Trans Pattern Anal Mach Intell, 44:154-180, 2020.

Belmar GARCIA-GARCIA, Thierry BOUWMANS et Alberto Jorge Rosales SILVA : Background subtraction in real applications: Challenges, current models and future directions. Computer Science Review, 35:100204, 2020.

Hongwei GE, Zehang YAN, Wenhao YU et Liang SUN : An attention mechanism based convolutional lstm network for video action recognition. Multimedia Tools and Applications, 78(14):20533-20556, 2019.

M-I GEORGESCU, A BARBALAU, R T IONESCU, F S KHAN, M POPESCU et M SHAH : Anomaly detection in video via self-supervised and multi-task learning. In CVPR, pages 12742-12752, 2021a.

M-I GEORGESCU, R IONESCU, F S KHAN, M POPESCU et M SHAH : A background-agnostic framework with adversarial training for abnormal event detection in video. IEEE Trans Pattern Anal Mach Intell, 2021b.

D GONG, L LIU, V LE, B SAHA, M R MANSOUR, S VENKATESH et A van den HENGEL : Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In ICCV, pages 1705-1714, 2019.

Ian GOODFELLOW, Yoshua BENGIO et Aaron COURVILLE : Deep learning mit press (2016). In Conference on information and communication systems (ICICS), pages 151-156, 2016.
Ian GOODFELLOW, Jean POUGET-ABADIE, Mehdi MIRZA, Bing XU, David WARDE-FARLEY, Sherjil OZAIR, Aaron COURVILLE et Yoshua BENGIO : Generative adversarial nets. *Advances in neural information processing systems*, 27, 2014.

Alex GRAVES, Navdeep JAITLY et Abdel-rahman MOHAMED : Hybrid speech recognition with deep bidirectional lstm. *In 2013 IEEE workshop on automatic speech recognition and understanding*, pages 273–278. IEEE, 2013.

A GUDI, F BÜTTNER et J van GEMERT : Proximally sensitive error for anomaly detection and feature learning. *arXiv preprint arXiv:2206.00506*, 2022.

Kai HAN, Yunhe WANG, Hanting CHEN, Xinghao CHEN, Jianyuan GUO, Zhenhua LIU, Yehui TANG, An XIAO, Chunjing XU, Yixing XU et al. : A survey on vision transformer. *IEEE transactions on pattern analysis and machine intelligence*, 2022.

Ismail HARITAOGLU, David HARWOOD et Larry S. DAVIS : W4: real-time surveillance of people and their activities. *IEEE Trans Pattern Anal Mach Intell*, 22:809–830, 2000.

Mahmudul HASAN, Jonghyun CHOI, Jan NEUMANN, Amit K ROY-CHOWDHURY et Larry S DAVIS : Learning temporal regularity in video sequences. *In Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 733–742, 2016.

Trevor HASTIE, Robert TIBSHIRANI, Jerome H FRIEDMAN et Jerome H FRIEDMAN : *The elements of statistical learning: data mining, inference, and prediction*, volume 2. Springer, 2009.

Jamie HAYES, Luca MELIS, George DANEZIS et Emiliano DE CRISTOFARO : Logan: Evaluating privacy leakage of generative models using generative adversarial networks. 05 2017.

Simon HAYKIN et N NETWORK : A comprehensive foundation. *Neural networks*, 2 (2004):41, 2004.

Chengkun HE, Jie SHAO et Jiayu SUN : An anomaly-introduced learning method for abnormal event detection. *Multimedia Tools and Applications*, 77(22):29573–29588, 2018.

Ryota HINAMI, Tao MEI et Shin'ichi SATOH : Joint detection and recounting of abnormal events by learning deep generic knowledge. *In Proceedings of the IEEE international conference on computer vision*, pages 3619–3627, 2017.

Geoffrey E HINTON et Richard ZEMEL : Autoencoders, minimum description length and helmholtz free energy. *Advances in neural information processing systems*, 6, 1993.

Sepp HOCHREITER et Jürgen SCHMIDHUBER : Long short-term memory. *Neural computation*, 9(8):1735–1780, 1997.

Berthold KP HORN et Brian G SCHUNCK : Determining optical flow. *Artificial intelligence*, 17(1-3):185–203, 1981.

Weiming HU, Tieniu TAN, Liang WANG et Steve MAYBANK : A survey on visual surveillance of object motion and behaviors. *IEEE Trans Syst Man Cybern C Appl Rev*, 34:334–352, 2004.
Xianwei HU, Tie LI, Zongzhi WU, Xuan GAO et Zhiqiang WANG: Research and application of intelligent intrusion detection system with accuracy analysis methodology. *Infrared Phys Technol*, 88:245-253, 2018.

Xing HU, Shiqiang HU, Yingping HUANG, Huanlong ZHANG et Hanbing WU: Video anomaly detection using deep incremental slow feature analysis network. *IET Computer Vision*, 10(4):258-267, 2016.

Jing HUO, Yang GAO, Wanqi YANG et Hujun YIN: Abnormal event detection via multi-instance dictionary learning. *In International conference on intelligent data engineering and automated learning*, pages 76-83. Springer, 2012.

Noureldien HUSSEIN, Efstratios GAVVES et Arnold WM SMEULDERS: Timeception for complex action recognition. *In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 254-263, 2019.

I-LIDS TEAM: Imagery library for intelligent detection systems (i-LIDS); a standard for testing video based detection systems. *In IET Conference on Crime and Security*, pages 445-448, 2006.

Sutrisno Warsono IBRAHIM: A comprehensive review on intelligent surveillance systems. *Communications in Science and Technology*, 1, 2016.

Radu Tudor IONESCU, Fahad Shahbaz KHAN, Mariana-Iuliana GEORGESCU et Ling SHAO: Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. *In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 7842-7851, 2019.

Shuiwang JI, Wei XU, Ming YANG et Kai YU: 3d convolutional neural networks for human action recognition. *IEEE transactions on pattern analysis and machine intelligence*, 35(1):221-231, 2012.

Fan JIANG, Ying WU et Aggelos K KATSAGGELOS: Detecting contextual anomalies of crowd motion in surveillance video. *In 2009 16th IEEE International Conference on Image Processing (ICIP)*, pages 1117-1120. IEEE, 2009.

L. JIAO, R. ZHANG, F. LIU, S. YANG, B. HOU, L. LI et X. TANG: New generation deep learning for video object detection: A survey. *IEEE Trans Neural Netw Learn Syst*, Early Access:1-21, 2021.

Longlong JING et Yingli TIAN: Self-supervised visual feature learning with deep neural networks: A survey. *IEEE transactions on pattern analysis and machine intelligence*, 43(11):4037-4058, 2020.

Michael I JORDAN: Attractor dynamics and parallelism in a connectionist sequential machine. *In Artificial neural networks: concept learning*, pages 112-127. 1990.

Andrej KARPATHY, George TODERICI, Sanketh SHETTY, Thomas LEUNG, Rahul SUKTHANKAR et Li FEI-FEI: Large-scale video classification with convolutional neural networks. *In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition*, pages 1725-1732, 2014.
Phulpreet KAUR, M GANGADHARAPPA et Shalu GAUTAM : An overview of anomaly detection in video surveillance. *In 2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)*, pages 607-614. IEEE, 2018.

Jacob Devlin Ming-Wei Chang KENTON et Lee Kristina TOUTANOVA : Bert: Pre-training of deep bidirectional transformers for language understanding. *In Proceedings of NAACL-HLT*, pages 4171-4186, 2019.

Salman KHAN, Muzammal NASEER, Munawar HAYAT, Syed Waqas ZAMIR, Fahad Shah-baz KHAN et Mubarak SHAH : Transformers in vision: A survey. *ACM computing surveys (CSUR)*, 54(10s):1-41, 2022.

Jong Sun KIM, Dong Hae YEOM et Young Hoon JOO : Fast and robust algorithm of tracking multiple moving objects for intelligent video surveillance systems. *IEEE Trans Consum Electron*, 57:1165-1170, 2011.

Junbong KIM, Kwanghee JEONG, Hyomin CHOI et Kisung SEO : Gan-based anomaly detection in imbalance problems. *In European Conference on Computer Vision*, pages 128-145. Springer, 2020.

Pil-Soo KIM, Dong-Gyu LEE et Seong-Whan LEE : Discriminative context learning with gated recurrent unit for group activity recognition. *Pattern Recognition*, 76:149-161, 2018a.

Seung Hyun KIM, Su Chang LIM *et al*. : Intelligent intrusion detection system featuring a virtual fence, active intruder detection, classification, tracking, and action recognition. *Ann Nucl Energy*, 112:845-855, 2018b.

Sung-Yeol KIM, Manbae KIM et Yo-Sung HO : Depth image filter for mixed and noisy pixel removal in RGB-D camera systems. *IEEE Trans Consum Electron*, 59:681-689, 2013.

D P KINGMA et J BA : Adam: A method for stochastic optimization. *In ICLR*, 2015.

B R KIRAN, D M THOMAS et R PARAKKAL : An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. *J Imaging*, 4:36, 2018.

Serkan KIRANYAZ, Onur AVCI, Osama ABDELJABER, Turker INCE, Moncef GABBOUJ et Daniel J INMAN : 1d convolutional neural networks and applications: A survey. *Mechanical systems and signal processing*, 151:107398, 2021.

James KIRKPATRICK, Razvan PASCANU, Neil RABINOWITZ, Joel VENESS, Guillaume DESJARDINS, Andrei A RUSU, Kieran MILAN, John QUAN, Tiago RAMALHO, Agnieszka GRABSKA-BARWINSKA *et al.* : Overcoming catastrophic forgetting in neural networks. *Proceedings of the national academy of sciences*, 114(13):3521-3526, 2017.

Yongqiang KONG, Jianhui HUANG, Shanshan HUANG, Zhengang WEI et Shengke WANG : Learning spatiotemporal representations for human fall detection in surveillance video. *Journal of Visual Communication and Image Representation*, 59:215-230, 2019.
Alex KRIZHEVSKY, Ilya SUTSKEVER et Geoffrey E HINTON : Imagenet classification with deep convolutional neural networks. *Communications of the ACM*, 60(6):84-90, 2017.

Brenden M LAKE, Ruslan SALAKHUTDINOV et Joshua B TENENBAUM : Human-level concept learning through probabilistic program induction. *Science*, 350(6266):1332-1338, 2015.

Zhenzhong LAN, Yi ZHU, Alexander G HAUPTMANN et Shawn NEWSAM : Deep local video feature for action recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition workshops*, pages 1–7, 2017.

Federico LANDI, Cees GM SNOEK et Rita CUCCHIARA : Anomaly locality in video surveillance. *arXiv preprint arXiv:1901.10364*, 2019.

Viet-Tuan LE et Yong-Guk KIM : Attention-based residual autoencoder for video anomaly detection. *Applied Intelligence*, pages 1–15, 2022.

Yann LE CUN, Leon BOTTOU et Yoshua BENGIO : Reading checks with multilayer graph transformer networks. In *1997 IEEE International Conference on Acoustics, Speech, and Signal Processing*, volume 1, pages 151–154. IEEE, 1997.

Christian LEDIG, Lucas THEIS, Ferenc HUSZÁR, Jose CABALLERO, Andrew CUNNINGHAM, Alejandro ACOSTA, Andrew AITKEN, Alykhan TEJANI, Johannes TOTZ, Zehan WANG *et al.* : Photo-realistic single image super-resolution using a generative adversarial network. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 4681–4690, 2017.

Sangmin LEE, Hak Gu KIM et Yong Man RO : Stan: Spatio-temporal adversarial networks for abnormal event detection. In *ICASSP*, pages 1323–1327. IEEE, 2018.

Jianan LI, Xiaodan LIANG, Yunchao WEI, Tingfa XU, Jiashi FENG et Shuicheng YAN : Perceptual generative adversarial networks for small object detection. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1222–1230, 2017.

Jun LI, Xianglong LIU, Mingyuan ZHANG et Deqing WANG : Spatio-temporal deformable 3d convnets with attention for action recognition. *Pattern Recognition*, 98:107037, 2020.

Nanjun LI et Faliang CHANG : Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder. *Neurocomputing*, 369:92–105, 2019.

Qing LI, Zhaofan QIU, Ting YAO, Tao MEI, Yong RUI et Jiebo LUO : Action recognition by learning deep multi-granular spatio-temporal video representation. In *Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval*, pages 159–166, 2016.

Shuo LI, Fang LIU et Licheng JIAO : Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection. In *AAAI*, 2022.

W LI, V MAHADEVAN et N VASCONCELOS : Anomaly detection and localization in crowded scenes. *IEEE Trans Pattern Anal Mach Intell*, 36:18–32, 2013.
Xiaoli LI et Ze-min CAI: Anomaly detection techniques in surveillance videos. In *2016 9th International congress on image and signal processing, BioMedical engineering and informatics (CISP-BMEI)*, pages 54–59. IEEE, 2016.

Zhenyang LI, Kirill GAVRILYUK, Efstratios GAVVES, Mihir JAIN et Cees GM SNOEK: Videolstm convolves, attends and flows for action recognition. *Computer Vision and Image Understanding*, 166:41–50, 2018.

KM LIANG, HW HON, MJ KHAIRUNNISA, TL CHOONG et HZ KHAIRIL: Real time intrusion detection system for outdoor environment. In *ISCAI*, pages 147–152, 2012.

P LICHTSTEINER, C POSCH et T DELBRUCK: A 128x128 120 db 15 µs latency asynchronous temporal contrast vision sensor. *IEEE J Solid-State Circuits*, 43:566–576, 2008.

Ji LIN, Chuang GAN et Song HAN: Tsm: Temporal shift module for efficient video understanding. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 7083–7093, 2019.

Tsung-Yi LIN, Piotr DOLLÁR, Ross GIRSHICK, Kaiming HE, Bharath HARIHARAN et Serge BELONGIE: Feature pyramid networks for object detection. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2117–2125, 2017.

Tsung-Yi LIN, Michael MAIRE, Serge BELONGIE, James HAYS, Pietro PERONA, Deva RAMANAN, Piotr DOLLÁR et C Lawrence ZITNICK: Microsoft coco: Common objects in context. In *European conference on computer vision*, pages 740–755. Springer, 2014.

Zachary C LIPTON, John BERKOWITZ et Charles ELKAN: A critical review of recurrent neural networks for sequence learning. *arXiv preprint arXiv:1506.00019*, 2015.

Kun LIU et Huadong MA: Exploring background-bias for anomaly detection in surveillance videos. In *Proceedings of the 27th ACM International Conference on Multimedia*, pages 1490–1499, 2019.

W LIU, W LUO, D LIAN et S GAO: Future frame prediction for anomaly detection-a new baseline. In *CVPR*, pages 6536–6545, 2018.

Wei LIU, Dragomir ANGUELOV, Dumitru ERHAN, Christian SZEGEDY, Scott REED, Cheng-Yang FU et Alexander C BERG: Ssd: Single shot multibox detector. In *European conference on computer vision*, pages 21–37. Springer, 2016.

Wen LIU, Weixin LUO, Zhengxin LI, Peilin ZHAO, Shenghua GAO *et al*. : Margin learning embedded prediction for video anomaly detection with a few anomalies. In *IJCAI*, pages 3023–3030, 2019.

X LIU, F ZHANG, Z HOU, L MIAN, Z WANG, J ZHANG et J TANG: Self-supervised learning: Generative or contrastive. *IEEE Trans Knowl Data Eng*, 2021a.

Z LIU, Y NIE, C LONG, Q ZHANG et G LI: A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In *ICCV*, pages 13588–13597, 2021b.
Zhenbing LIU, Zeya LI, Ruili WANG, Ming ZONG et Wanting JI: Spatiotemporal saliency-based multi-stream networks with attention-aware lstm for action recognition. *Neural Computing and Applications*, 32(18):14593-14602, 2020.

Devashish LOHANI, Carlos CRISPIM-JUNIOR, Quentin BARTHÉLEMY, Sarah BERTRAND, Lionel ROBINAULT et Laure Tougne RODET: Unsupervised and adaptive perimeter intrusion detector. *In 2022 IEEE International Conference on Image Processing (ICIP)*, pages 3621-3625. IEEE, 2022a.

Devashish LOHANI, Carlos CRISPIM-JUNIOR, Quentin BARTHÉLEMY, Sarah BERTRAND, Lionel ROBINAULT et Laure TOUGNE: Spatio-temporal convolutional autoencoders for perimeter intrusion detection. *In RRPR*, pages 47-65, 2021.

Devashish LOHANI, Carlos CRISPIM-JUNIOR, Quentin BARTHÉLEMY, Sarah BERTRAND, Lionel ROBINAULT et Laure TOUGNE RODET: Perimeter intrusion detection by video surveillance: A survey. *Sensors*, 22(9), 2022b. ISSN 1424-8220. URL https://www.mdpi.com/1424-8220/22/9/3601.

C LU, J SHI et J JIA: Abnormal event detection at 150 fps in Matlab. *In ICCV*, pages 2720-2727, 2013.

Yiwei LU, Frank YU, Mahesh Kumar Krishna REDDY et Yang WANG: Few-shot scene-adaptive anomaly detection. *In European Conference on Computer Vision*, pages 125-141. Springer, 2020.

Yue LU, Congqi CAO, Yifan ZHANG et Yanning ZHANG: Learnable locality-sensitive hashing for video anomaly detection. *IEEE Transactions on Circuits and Systems for Video Technology*, 2022.

Elena LUNA, Juan Carlos SAN MIGUEL, Diego ORTEGO et José María MARTÍNEZ: Abandoned object detection in video-surveillance: Survey and comparison. *Sensors*, 18 (12), 2018.

Dezhao LUO, Chang LIU, Yu ZHOU, Dongbao YANG, Can MA, Qixiang YE et Weiping WANG: Video cloze procedure for self-supervised spatio-temporal learning. *In Proceedings of the AAAI conference on artificial intelligence*, volume 34, pages 11701-11708, 2020.

W LUO, W LIU et S GAO: A revisit of sparse coding based anomaly detection in stacked RNN framework. *In ICCV*, pages 341-349, 2017a.

Weixin LUO, Wen LIU et Shenghua GAO: Remembering history with convolutional lstm for anomaly detection. *In 2017 IEEE International Conference on Multimedia and Expo (ICME)*, pages 439-444. IEEE, 2017b.

Weixin LUO, Wen LIU, Dongze LIAN, Jinhui TANG, Lixin DUAN, Xi PENG et Shenghua GAO: Video anomaly detection with sparse coding inspired deep neural networks. *IEEE transactions on pattern analysis and machine intelligence*, 43(3):1070-1084, 2019.

H LV, C CHEN, Z CUI, C XU, Y LI et J YANG: Learning normal dynamics in videos with meta prototype network. *In CVPR*, pages 15425-15434, 2021.