## 3 The Proposed Framework
### Temporally Adaptive Cross-Modal Attention  
The attention mechanism was first proposed to deal with natural language processing tasks, and it is very capable of modeling the complex relationships within serialized tokens [52]. By considering features from each segment in the video as tokens, the attention module can also be modified to learn cross-modal relationships in AVE [8; 10; 11; 12; 13]. Furthermore, since task-related information is often contained in only some of the tokens, a threshold-based approach can better model the relationships between modalities sparsely by obtaining all the semantically similar feature pairs at all times [19]. However, unlike sentences, where words that are far apart may still be highly correlated, the changing state of events in the video is often continuous, namely, the audio (visual) signal at one moment is primarily correlated with its synchronized visual (audio) signal. Therefore, this paper designs a TACMA that makes one modality adaptively assign attention to useful information in another modality by considering their temporal difference. In such a way, the multimodal information is integrated along the timeline in a more fine-grained manner. The detailed structure of our adaptive attention is depicted in Figure 4.  
Given the fixed \(\mathbf{f}_{a}\) and \(\mathbf{f}_{y}\) output from feature encoders as input, the first step is to compute the attention matrices \(\mathbf{M}^{\text{tr}}\in\mathbb{R}^{T\times T}\) and \(\mathbf{M}^{\text{va}}\in\mathbb{R}^{T\times T}\) by the scaled dot-product along the feature dimension:  
\[\mathbf{M}^{\text{tr}}=(\mathbf{f}_{a}\mathbf{W}_{a}^{\text{l}})(\mathbf{f}_{y}\mathbf{W}_{y}^{ \text{l}})^{T}/\sqrt{d_{l}},\ \ \mathbf{M}^{\text{va}}=(\mathbf{M}^{\text{tr}})^{T}, \tag{3}\]  
where \(\mathbf{W}_{a}^{\text{l}}\) and \(\mathbf{W}_{y}^{\text{l}}\in\mathbb{R}^{d_{l}\times d_{h}}\) are learnable weight matrices implemented as linear layers. Note that we adopt a similar strategy as in [19] that uses the same set of _queries_ and _keys_ (which are both the linear projections of input features) in the computation of the attention matrices. This can help to reduce computational costs and ensure efficiency. ReLU activation is applied on \(\mathbf{M}^{\text{tr}}\) and \(\mathbf{M}^{\text{va}}\) afterward to filter out negatively correlated audio and  
Figure 4: Structures of the proposed TACMA and classification layer. visual feature pairs. Then, the softmax function is performed along each row of \(\mathbf{M}^{av}\) and \(\mathbf{M}^{va}\) to rescale the values between 0 and 1. The larger the elements in \(\mathbf{M}^{av}\) and \(\mathbf{M}^{va}\) are, the more event-related information that is shared between their corresponding audio and visual features. To weaken the connections between features that share no event-related information or are temporally asynchronous, we weigh the scores of all the feature pairs in the attention matrices. Particularly, given \(M^{av}_{i,j}\) and \(M^{va}_{i,j}\) as the elements in the \(i\)-th row and the \(j\)-th column of the attention matrices, where \(i\), \(j\) also index the time dimension of audio and visual features, respectively, their corresponding weights \(W^{mv}_{i,j}\) and \(W^{ma}_{i,j}\) are  
\[\left\{ \begin{aligned} W^{mv}_{ij}&\triangleq exp \Big{(}-\theta|i-j|/M^{av}_{i,j}\Big{)}\;,\\ W^{ma}_{ij}&\triangleq exp\Big{(}-\theta|i-j|/M^{ va}_{i,j}\Big{)}\;,\end{aligned} \right. \tag{4}\]  
where \(\theta\) is a hyperparameter and the magnitudes of \(W^{mv}_{i,j}\) and \(W^{ma}_{i,j}\) are adjusted by the ratio of \(\theta\) to the corresponding element in \(\mathbf{M}^{av}\) and \(\mathbf{M}^{va}\). Hence, the overall weighting method is expressed as the element-wise product, written as:  
\[\left\{ \begin{aligned} W^{av}&=W^{mv}\odot\mathbf{M}^{av }\;,\\ W^{va}&=W^{ma}\odot\mathbf{M}^{va}\;,\end{aligned} \right.