## 3 Methodology
### Self-Attention Based on Euclidean Distance  
In sequence modeling, learning the relationships between tokens and contextual information is crucial. The Transformer model employs a self-attention mechanism to capture the global context of sequences, demonstrating exceptional capabilities in processing text sequences. However, videos differ from sentences in that every token in a sentence can be converted into a word vector using embeddings, with finite vocabularies constraining the feature representation of words. In contrast, each frame of a video has an infinite number of representations within a specific space, resulting in a gap between the application of Transformers to videos and text. First, we analyze the limitations of Transformers in processing video sequences. Then, we propose an improved approach to make them more adaptable for perceiving and learning contextual information within visual sequences, leveraging Euclidean distance for enhanced attention. The self-attention in vanilla Transformer is formulated as Equation (5):  
\[Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{5}\]  
The crucial part involves calculating the attention map by multiplying query \(Q\in R^{t\times c}\) and key \(K\in R^{t\times c}\). We assume \(Q=[q_{1},q_{2},\dots,q_{t}]^{T}\) and \(K=[k_{1},k_{2},\dots,k_{t}]^{T}\), so we can obtain the following inference:  
Figure 3: Multi-scaled features integration and fusion. The spatial encoder outputs feature maps of sizes 3 and 7, respectively. These feature maps are first flattened into 1D vectors. Then, a special token \([cls]\) is added to the head of the vector, similar to ViT. Next, the flattened vectors are added with trainable position embedding and then utilize the Transformer encoder to obtain the global context information of both feature maps. Finally, the two \([cls]\) are concatenated to achieve multi-scale feature fusion. \[ \begin{split} QK^{T}&=\left[\begin{array}{c}q_{1}^{T}\\ q_{2}^{T}\\ \vdots\\ q_{T}^{T}\end{array}\right]\\ &=\left[\begin{array}{cccc}q_{1}^{T}k_{1}&q_{1}^{T}k_{2}&\cdots&q_{1}^{T}k_{ t}\\ q_{2}^{T}k_{1}&q_{2}^{T}k_{2}&\cdots&q_{2}^{T}k_{t}\\ \vdots&\vdots&\ddots&\vdots\\ q_{t}^{T}k_{1}&q_{t}^{T}k_{2}&\cdots&q_{t}^{T}k_{t}\end{array}\right]\\ \\ \alpha_{i,j}=q_{i}^{T}k_{j}=\parallel q_{i}\parallel\cdot\parallel k_{j} \parallel\textit{-cos}\theta_{ij}\end{split}  \tag{6}\]  
where, in Equations (6) and (7), \(\alpha_{i,j}\) presents the value of row \(i\) and column \(j\) in the attention map, and \(\theta_{ij}\) means the angle of vector \(q_{i}\) and \(k_{j}\). According to Equation (5), the \(QK^{T}\) necessitates softmax operation in the row direction. In each row, the query vector \(q_{i}\) is the same, indicating that the attention weights are influenced by the magnitude and cosine similarity. However, our prior research reveals that an exceedingly large vector may disrupt the attention mechanism, causing it to focus excessively on a specific position while disregarding more global information (as shown in Figure 4). \[EDAttention(Q,K,V)=Softmax(ED(Q,K))V \tag{8}\]  
\[ \begin{split} ED(Q,K)=\left[\begin{array}{cccc}<q_{1},k_{1}>&< q_{1},k_{2}>&\cdots&<q_{1},k_{t}>\\ <q_{2},k_{1}>&<q_{2},k_{2}>&\cdots&<q_{2},k_{t}>\\ \vdots&\vdots&\ddots&\vdots\\ <q_{t},k_{1}>&<q_{t},k_{2}>&\cdots&<q_{t},k_{t}>\end{array}\right]\\ \\ <q_{i},k_{j}>=-\parallel q_{i}-k_{j}\parallel_{2}^{2}=-\parallel q_{i} \parallel_{2}^{2}-\parallel k_{j}\parallel_{2}^{2}+2q_{i}k_{j}\end{split}  \tag{9}\]  
Therefore, to address the issue, we propose EDAttention, which replaces the dot product with the Euclidean distance between vectors as attention weights. Specifically, the  
Figure 4: The demo for the attention map of vanilla Transformer. The heatmap denotes the attention scores, and the bar above the heatmap is the magnitude of key vectors.