# Three-Stage Deep Learning Framework for Video Surveillance
## 3 Proposed Method
### Transformer Mechanism  
Before the advent of Transformers, the \"Seq2 Seq model\" [36; 37; 38; 39] based on RNNs was recognized as the dominant methodology for the processing of sequence data.  
However, these RNN models, like the Seq2 Seq model, have several limitations. First, the model must process the inputs sequentially, which limits computation parallelization. Second, the window size of the model is fixed, which renders it difficult to learn the relationships between items that are far apart in a sequence, particularly because of the problem of gradient decay or explosion.  
A proposed solution to this problem is an attention mechanism. Recently, improved RNNs, such as LSTM and GRU, have improved the window size to increase the relationships between distant items to a certain extent; however, the window size of these models is still limited. In contrast, the attention mechanism can theoretically have an infinite window size under the premise that computing resources are available, it can be parallelized for fast learning speeds, and it can be effectively applied to very large datasets.  
In this attention mechanism, self-attention plays a crucial role in considering the relationships between words in the input sequence. Self-attention introduces the concepts of positional encoding: the query, key, and value. Since self-attention does not explicitly consider sequential order information, it adds vectors containing positional information of the same dimension as the input used in the model. This is known as positional encoding.  
Once positional encoding is completed, self-attention utilizes three elements: the query, key, and value. The query extracts information about the given input and is used to evaluate the relationships with other input elements. The key determines how much attention each element of the input sequence should receive, and it combines the query and key to calculate attention scores, which represent how much attention each element should receive. These scores contain the actual information and, by weighting according to attention scores, form the final output. The value of each element is weighted based on its importance and attention, and these values are combined to generate the final output of attention.  
The self-attention mechanism is particularly well suited for dealing with sequential data where information changes over time or where correlations are important, such as videos, which are the type of data used in this study.  
However, in practice, the data that we deal with have complex structures with many different attributes and perspectives. This diversity cannot be fully captured by a single attentional mechanism because it operates on a limited set of information and is unlikely to reflect the interactions between the various attributes of complex data. To address these issues, the multi-head attention technique is used, which employs multiple attention heads in parallel. The advantage of this structure is that each attention head captures information from an independent perspective, and these diverse perspectives are integrated to form a richer representation.  
Transformers are architectures that are not based on an RNN structure but introduce self-attention and multi-head attention to the encoder and decoder parts. This integrates the advantages of self-attention and multi-head attention and has exhibited excellent performance in various natural language processing tasks, such as translation, document summarization, question and answer, and time series data processing. The basic structure of the Transformer is illustrated in Figure 2.