Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment (2024)

Abstract: Transformer-based methods have gone mainstream in multimodal sequential learning. The intra and inter modality interactions are captured by the query-key associations of multihead attention. In this way, the calculated multimodal contexts (attentional results) are expected to be relevant to the query modality. However, in existing literature, the alignment degree between different calculated attentional results of the same query are under-explored. Based on this concern, we propose a new constrained scheme called Multimodal Contextual Contrast (MCC), which could align the multiple attentional results from both local and global perspectives, making the information capture more efficient. Concretely, the calculated attentional results of different modalities are mapped into a common feature space, those attentional vectors with the same query are considered as a positive group and the remaining sets are negative. From local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context and keeping the other stay the same. From coarse global perspective, we divide all the contextual groups into two sets (i.e., aligned and unaligned), making the total score of aligned group relatively large. We extend the vectorial inner product operation for more input and calculate the aligned score for each multimodal group. Considering that the computational complexity scales exponentially to the number of modalities, we adopt stochastic expectation approximation (SEA) for the real process. The extensive experimental results on several tasks reveal the effectiveness of our contributions.

heading: 1 Introduction
text: Multimodal sequential learning, which aims to process and understand the semantic information from multiple modalities (e.g., vision, language, audio)
∗† denotes corresponding author.
with machine learning skills, has drawn increasing attention recently. Many endeavors (Gabeur et al., 2020; Pham et al., 2019; Zadeh et al., 2018a,b) are devoted to the design of multimodal interactive mode and effective individual representation learning. Transformer-based multimodal interaction methods (Gabeur et al., 2020; Tsai et al., 2019) occupy the mainstream position in multimodal interaction area. Compared with vanilla methods, Transformer-based methods could achieve relatively superior performances with deep stacked attention blocks (Vaswani et al., 2017) and a suitable number of training samples. Concretely, by treating one modality as query (e.g., text) and the other modalities as keys (e.g., visual, audio), the multimodal contextual sequences (attentional results) can be obtained by query-key associated mechanism. In this way, the calculated attentional results are expected to be relevant to the query modality. However, the alignment degree between different calculated attentional results of the same query are under-explored. Specifically, if we have three modalities (i.e., text, visual, audio), we employ Transformer to correlate text-visual and textaudio interaction respectively. As shown in Fig. 1, when the text query is “rather”, the attentional
5247
results (“shorter”) of text-visual pair has obvious problems, while the audio localization (“rather”) is correct. During training, the mistakes of query-key association are inevitable. At this time, constraining the alignment of audio and visual attentional results can polish the capture mistakes indirectly.
Motivated by the observations, in this paper, we propose a new constrained strategy called Multimodal Contextual Contrast (MCC), which could align multimodal attentional results from both local and global perspectives, making the attentional capture more efficient. Specifically, the multimodal attentional results of different modalities are calculated with multi-head attention first and then mapped into a common feature space. The sequence lengths of the multimodal attentional results are same as that of query modality, we denote the multiple attentional sequences as ci ∈ Rta×d(i ∈ [n]), where d is the feature dimension and n is the number of modalities. Those attentional vectors with the same query (at the same sequential step t ∈ [ta]) are considered as a positive group and the remaining groups (at least one of the attentional vectors is with different query) are negative. From the local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context vector and keeping the other stay the same. Totally, the number of negative groups is n(ta − 1) for a positive group. From the global perspective, we divide all the groups into two sets (i.e., aligned and unaligned), making the total score of aligned groups relatively large. For the implementation of contrastive constraints, we extend the vectorial inner product operation for more input and compute the aligned score for each multimodal group. Considering that the computational complexity of relevance scores scales exponentially to the number of modalities, we adopt stochastic expectation approximation for the real process. We conduct extensive experiments on three tasks, the experimental results show that MCC could achieve competitive results compared with the state-of-the-art methods. To sum up, the contributions of our work are four-folded:
• We propose a constrained strategy called Multimodal Contextual Contrast (MCC) for multimodal sequential learning, which conducts contrastive constraints for the multiple calculated attentional results. To the best of our knowledge, it is the first time to conduct con-
trastive scheme for the calculated attentional results.
• We develop the contrastive mechanism from both fine-grained local and coarse-grained global perspectives, making the attentional capture more accurate indirectly.
• Considering that the computational complexity of relevance scores scales exponentially to the number of modalities, we adopt stochastic expectation approximation (SEA) for the real process.
• We conduct extensive experiments on three tasks, multimodal sentiment analysis (with video, audio, text modalities), speaker traits recognition (with video, audio, text modalities), and video retrieval (with motion, scene, OCR, audio, speech, face, appearance modalities). The experimental results show that MCC could achieve competitive results compared with state-of-the-art methods.

heading: 2 Related Work
text: Multimodal Interaction. Existing multimodal interaction methods could be categorized into Transformer-based and Non-Transformer-based methods. As for the former, Zadeh et al. (2016b) proposes to train the model on simply concatenated multimodal features for prediction. Poria et al. (2017) correlates multiple modalities with a context-dependent fusion method. Zadeh et al. (2018b) explicitly accounts for both interactions in a neural architecture and continuously models them through time. As for the latter, Tsai et al. (2019) proposes multimodal transformer to boost the interactions between multiple modalities. LMFMulT (Sahay et al., 2020) combines the LMF and MulT to process multimodal sequential information, First LMF-MulT aligns different modalities at sequential-step level, fusing different modalities. The results are input to the Transformer module. Li et al. (2022) proposes the modal-order-aware network to integrate the three modalities in a certain order to distinguish the importance of different modalities. Yang et al. (2023) conducts contrastive feature decomposition from the sample level.
Contrastive Learning. According to the modality of data, existing methods can be divided into two categories, i.e., single-modality based and multi-modality based contrastive learning.
Considering the single-modality based methods, Wu et al. (2018) uses a memory bank which stores previously-computed representations and noise-contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010) to tackle the computational challenges imposed by the large number of instance classes. MoCo (He et al., 2020) further improves such a scheme by storing representations from a momentum encoder in dynamic dictionary with a queue. As for the multi-modality based methods (Hager et al., 2023; Mustafa et al., 2022; Yang et al., 2023), the common strategy is to explore the natural correspondences among different modalities and use contrastive learning to learn representations by pushing modalities describing the same scene closer, while pushing modalities of different scenes apart. However, different from the existing multimodal contrastive strategies, we consider the problem from the view of attentional results and their time steps, instead of input features.

heading: 3 Contrastive Contextual Alignment
text: In this section, we would introduce MCC with the following scheme. First, we give simple illustration of the definition and background of some symbols. Second, we would introduce the local contrastive alignments and global contrastive alignments in detail (as shown in Fig. 2). Third, considering the complexity, we adopt the theoretical approximation for computation reduction. To be more intuitive, we further provide the complexity analysis of vanilla MCC and Improved MCC.

heading: 3.1 Multimodal Contextual Sequences
text: Suppose that there exists n modalities with sequential representation vi ∈ Rti×d, i ∈ [n], where ti denotes the sequence length of i-th modality, d denotes the feature dimension of all the modalities. For convenience, we choose one modality va ∈ Rta×d as “anchor (query)”. As we know, there are two forms of interactions among different modalities: modality-specific interactions and cross-modal interactions. Considering the former interactions, we treat the va as “key (value)”. Considering the latter interactions, we treat vi(i 6= a) as “keys (values)”. In this paper, we employ the mainstream Transformer structure for multimodal sequential learning. Specifically, the modalityspecific and cross-modal interactions of va can be
expresses as follows:
ca = Self_ATT(va)
ci = Cross_ATT(va, vi) (1)
where ca ∈ Rta×d denotes the modality-specific attentional results and ci ∈ Rta×d denotes the cross-modal attentional results. The only difference between functions “Self_ATT” and “Cross_ATT” is the key (value). To be more intuitive, we can further rewrite these two functions as follows:
Self_ATT(va) =ATT(va, va, va)
Cross_ATT(va, vi) = ATT(va, vi, vi) (2)
where “ATT” denotes the multi-head attention mechanism (Vaswani et al., 2017) which is widely used in computer vision/natural language processing/multimodal analysis community. In the following sections, we merge the contexts ca and ci(i 6= a) and employ ci ∈ Rta×d(i ∈ [n]) for illustration. In practice, we implement multiple projection layers following the attentional sequences. For convenience, we still utilize d to denote the common feature dimension.

heading: 3.2 Local Contrastive Alignments
text: The basic of multi-head attention mechanism is the inner-product operation for query-key similarity, while most of the existing methods do not consider the relevance of different contextual sequences of the same query. For example, the context (attentional) vectors corresponding to the t-th time step (vta ∈ Rd) of query modality are cti ∈ Rd where i ∈ [n]. According to the calculation rules of inner-product operation, each cti would be related to vta. However, the alignments between the context vectors (cti, i ∈ [n]) are not strictly evaluated, which may influence the attentional capture. Inspired by the widely-studied contrastive learning techniques, we divide the groups of context vectors into positive sets and negative sets. Note each context group contains n vectors correspond to n random sequential steps of n modalities, thus, the number of total groups is (ta)n. We facilitate the contrasts from two perspectives: local alignments and global alignments, which are complementary to each other. In this section, we introduce the local contrastive alignments in detail. Following the illustration above, the context vectors of the same query (at the same step) are relevant and treated as positive (totally ta groups). The main challenge
is to sample some negative groups. We start by introducing the alignments for two contextual sequences and then employ the conclusion for the generalization of more contextual sequences.

heading: 3.2.1 The Alignments for Two Contextual Sequences
text: Suppose that there are two contextual sequences c1 and c2 ∈ Rta×d. The primary objective of the training is to maximize the alignment degree between the positive pairs (i.e., ct1 and c t 2). Thus, we first define the alignment function by using the normalized inner product as:
A(ct1, c t 2) = 〈ct1, ct2〉 ||ct1||||ct2||
(3)
where 〈, 〉 denotes the inner product operation. The range of function A() is [−1, 1]. We hope that the scores of positive pairs are close to 1. However, merely optimizing the alignment of positive pairs ignores the important positive-negative relation knowledge (Mikolov et al., 2013b). To make the training process more informative, we reform the overall objective in the contrastive learning manner (Arora et al., 2019; Van den Oord et al., 2018) with Noise Contrastive Estimation (NCE) loss (Mnih and Teh, 2012; Mikolov et al., 2013b). Specifically, we consider the fact that one context vector is more related to the context vector with the same query (at the same sequential step among all the steps). Then, we can formulate the overall NCE objective as follows:
Ll = − ta∑
t=1
log [ exp
( A(ct1, c t 2) )
exp ( A(ct1, c t 2) ) + µ
]
µ =
ta∑
t′=1,t′ 6=t
( exp ( A(ct1, c t′ 2 ) ) + exp ( A(ct ′ 1 , c t 2) ))
(4)
where A(ct1, c t 2) denotes the positive pair of the query at t-th sequential step, A(ct1, c t′ 2 ) and A(ct ′ 1 , c t 2) denote the negative pairs according to the natural facts stated above. Such objective in Eq. 4 explicitly encourages the alignment of positive pair while separates the negative pairs.

heading: 3.2.2 Generalization for More Contextual Sequences
text: With the conclusion for two contextual sequences, we discuss the condition of more contextual sequences. One simple idea is to treat the contextual sequences as multiple two-sequence pairs and utilize the existing conclusion of Eq. 4. However, this way neglects the correlation among all the modalities. Thus, we consider the contrastive constraints from a more general perspective by jointly processing all the contextual sequences. Based on one specific positive group {cti ∈ Rd|i ∈ [n]} of the same query at t-th sequential step, we try to change some sequential steps of the attentional vectors and analyze the relative correlation of all the records. After the multiple replacements, we obtain two conclusions.
We first define the relevance scores for the nvector sets based on those of two-vector pairs. The detailed process is shown as follows:
A({cti|i ∈ [n]}) = n∑
i=1
n∑
j=1,j<i
A(cti, c t j) (5)
Proposition 1: Suppose that we have the group of contexts for the t-th sequential step cti ∈ Rd where i ∈ [n]. When we randomly change the sequential step of one specific context, the obtained new group has less relevance than the original positive group. Further, if we treat the score of positive
n-vector group consisting of the scores of multiple (n(n−1)2 ) positive two-vector pairs like Eq. 5, the scores of negative n-vector groups (only changing one specific context) also contain the scores of all the negative two-vector pairs. The detailed analysis is in the appendix (Sec. B).
Proposition 2: Only changing the sequential step for the query vector of one modality can be considered as hard negative mining. Following the nature of contrastive learning (Robinson et al., 2020), to enhance the alignment of the positive set, we can improve the difficulty of the negative set. Based on a specific positive set, if we change the step of one modality, the number of irrelevant twovector pairs increases by O(n). When we change the steps of s modalities, the complexity is O(sn).
Based on the above observations, we create the negative groups for the specific positive groups. Concretely, we sample the negative groups with only one different sequential step. Totally, the number of the negative groups is n(ta−1) for a specific postive group. The local contrastive constraints can be expressed as follows:
Ll = − ta∑
t=1
log [ exp
( A({cti|i ∈ [n]}) )
exp ( A({cti|i ∈ [n]}) ) + µ
]
µ = ∑
∗ exp
( A({ct ′ i i |i ∈ [n]})
) (6)
where ∗ denotes the sampling condition of negative groups (i.e., only one of {t′i|i ∈ [n]} is not equal to t, the others are equal to t). Although the local contrastive constraints can align the multimodal contexts at a fine granularity, most of the negative sets are discarded. Relying on this loss function alone, we may obtain a locally optimal solution. To solve this concern, we propose a complementary contrastive constraint, which align the context vectors from a coarse global perspective.

heading: 3.3 Global Contrastive Alignments
text: To make full use of the negative context (attentional results) sets, we propose a complementary global contrastive strategy. Specifically, we divide all the context groups into two sets, one includes all the positive groups and the other includes all the negative groups. From a coarse-grained perspective, the scores of positive groups should be close to n(n−1)2 and the scores of negative sets are less than n(n−1)2 . The optimization goal is to make the more relevant
set dominate, which fits our intuition. The formula expression is shown as follows:
Lg =− log [ ∑ta t=1 exp ( A({cti|i ∈ [n]}) )
∑ta t=1 exp ( A({cti|i ∈ [n]}) ) + µ
]
µ = ∑
∗ exp
( A({ct ′ i i |i ∈ [n]})
) (7)
where ∗ denotes the sampling condition of negative groups (i.e., not all of {t′i|i ∈ [n]} are equal), Lg denotes the global contrastive loss, which can control the relative distributions of positive relevance and negative relevance, to some extent.

heading: 3.4 Stochastic Expectation Approximation
text: When the number of modalities increases, the computational complexity would become unexpectedly large. Partial complexity comes from combinational summation operation for relevance score. Inspired by the kernel approximation skills (Kar and Karnick, 2012), we develop an efficient method called Stochastic Expectation Approximation (SEA) to calculate the relevance score. Following the assumptions above, the exp-based relevance score of a context group is expressed as:
exp ( n∑
i=1
n∑
j=1,j<i
A(cti, c t j) )
= exp ( n∑
i=1
n∑
j=1,j<i
〈 cti ||cti|| , ctj ||ctj ||
〉) (8)
The main challenges of the approximation are twofolded: First, we should reduce the square-level complexity. Second, we should consider the reconstruction of non-linear function exp(). The Stochastic Expectation Approximation is the extension of binary kernel reconstruction, which can be expressed as follows:
exp ( n∑
i=1
n∑
j=1,j<i
〈vi, vj〉 ) =
Ew∼N (0,Id)
[ n∏
k=1
exp ( w>vk −
‖vk‖2 2
)] (9)
where we utilize vi to denote cti ||cti||
for convenience. E and N (0, Id) denote expectation operation of
different random features w and the sampling distribution of w, respectively. The detailed derivations can be found in the appendix (Sec. C). Note that we also provide an analysis of the tradeoff between the sampling number (computational complexity) and performance in the appendix Eq. 18 and Sec. D.

heading: 3.5 Complexity Analysis
text: In this section, we detailedly analyze the alignment complexity before and after the approximation. We divide the analysis into two parts, global contrastive alignments and local contrastive alignments. The total complexity of the global contrastive alignments is exponential O(n2(ta)n), the square complexity O(n2) arises from the combinational addition operations and the exponential complexity O((ta)
n) arises from a large number of context sets. We argue that the SEA approximation can reduce the square complexity to a linear level O(n). Specifically, the relevance score of a set can be calculated with continuous multiplication operation according to the Eq. 9. As for the exponential part, we calculate the sum of all the relevance scores like∑
i ∑ j ∑ k aibjck = ( ∑ i ai)( ∑ j bj)( ∑ k ck) (from exponentialO((ta)n) to linearO(nta)). The detailed analysis is shown in the appendix (Sec. E). Therefore, the complexity with approximation changes from O(n2(ta)n) to O(n2ta).
The total complexity of the local contrastive alignments is O(n3(ta)2), where the square term O(n2) also arises from the combinational addition operation, the term O(n(ta)2) arises from the number of negative sets in the local contrastive constraints. This term can be easily reduced to O(nta) with vanilla summation for multiple sequential steps. Thus, the complexity is also O(n2ta). With the SEA approximation, the total computational complexity changes from O(n3(ta)2) to O(n2ta).

heading: 3.6 Training
text: We evaluate MCC on three tasks, including multimodal sentiment analysis, speaker traits recognition, and video retrieval. MCC is treated as an auxiliary constraint for these tasks. Suppose that Lt denotes the loss of the original task, the final optimization goal can be expressed as follows:
L = Lt + λ1Ll + λ2Lg (10)
where Lt can be MAE loss (Liu et al., 2018) for multimodal sentiment analysis and contrastive loss
(Gabeur et al., 2020) for video retrieval.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate the performance of MCC on three challenging tasks, including multimodal sentiment analysis, multimodal speaker traits recognition, and multimodal video retrieval.
CMU-MOSI(Zadeh et al., 2016a): The goal of multimodal sentiment analysis is to identify a speaker’s sentiment based on the speaker’s display of verbal and nonverbal behaviors. There are a total of 2199 data points (opinion utterances) within CMU-MOSI datasets. The dataset has realvalued sentiment intensity annotations in the range [−3,+3]. It is considered a challenging dataset due to speaker diversity (1 video per distinct speaker), topic variations and low-resource setup.
POM(Pérez-Rosas et al., 2013): The dataset contains 963 movie review videos, it is designed for speaker traits recognition based on communicative behavior of a speaker. There are 16 different speaker traits in total. Each video is annotated for various personality and speaker traits.
MSR-VTT(Xu et al., 2016): MSR-VTT is composed of 10k videos. Each video is 10 to 30s long, and is paired with 20 natural sentences describing it. We report results on the train/test splits introduced in (Gabeur et al., 2020) that uses 9000 videos for training and 1000 for testing.

heading: 4.2 Experiments for Sentiment Analysis and Speaker Traits Recognition
text: Data Preprocessing: We extract three modalities for CMU-MOSI and POM, including textual (Pennington et al., 2014), visual (iMotions, 2017), and audio modalities (Degottex et al., 2014).
Experimental Details: Transformer-based multimodal interaction methods have gone mainstream in recent years. With the flexible multi-head attention mechanism, the self-modal interactions and cross-modal interactions can be implemented easily. We implement MCC based on Transformerbased backbone and make comparisons with existing methods. Our Transformer backbone named SC-Transformer (The detailed structure is shown in Appendix Fig. 6) is similar to MulT (Tsai et al., 2019) (Appendix Fig. 5), MulT is a commonlyused baseline that first introduces Transformer structure into the multimodal sequential learning. However, MulT separately processes the intra (self)
and inter (cross) interactions in tandem, which does not fit in with the precondition of MCC (i.e., in parallel). Thus, we simply add a self-attention module in the cross-modal interaction block, making the self and cross attention parallel. The hyperparameters include Adam learning rate 0.001, the structure of projection network (where the hidden size is 40, the size of common space is 16, the number of random features is 64). λ1 and λ2 are set to 0.1 and 0.01. The temperature for contrastive learning is set to 0.2. We conduct the experiments on RTX 3080Ti GPUs.
Compared Baselines: We mainly compare MCC with the baseline methods that utilize the same features (i.e. Glove, FACET, COVAREP) for fairness. We reproduce the experimental results that do not be conducted on CMU-MOSI and POM by ourselves.
Experimental Results: The results are shown in Table 1. We can find that MCC consistently gains the best performance across all the baseline methods. We provide a detailed analysis for such observations. MTGAT, which develops graph convolutional networks to capture the multimodal interactions, performs worse than MCC as it pays more attention to the complexity reduction. As for the tensor based multimodal fusion methods, LMF neglects the fine-grained temporal interaction which includes rich structured information for multimodal modeling. Further, MCC outperforms SCTransformer, LMF-MulT, HGraph, AMOA, ConFEDE with a better overall performance. In general, the best performances of MCC are attribute to the local fine-grained contrastive constraints and the complementary global coarse-grained contrastive constraints which make the attention capture more
accurate1. Table 2 shows the experimental results of different methods on speaker traits recognition dataset, POM, where we report the multi-class accuracy of all the traits. A similar observation is found from the table, MCC achieves competitive performances compared with all the baseline methods on most of the traits. Particularly, the performance of MCC increases the average multi-class accuracy from 42.0 to 44.3 compared to the best counterparts. Further, the improvement on POM is larger than CMU-MOSI. At first, we think the reason is the size of dataset. However, when we try the larger datasets like CMU-MOSEI (The results are shown in section I of appendix), the improvement is similar to CMU-MOSI. Thus, we think the improvement of POM is caused by the multitask training mechanism, since we train 16 traits together, which is different from CMU-MOSI.
Ablation Study: We set some control experiments on CMU-MOSI and POM to verify the effectiveness of MCC and the results are shown in Tables 3, 4, and 5, where “w/o. Global” denotes the model without global contrastive constraints, “w/o. Local” denotes the model without local contrastive constraints, “w/o. SEA” denotes the model without SEA approximation. We could observe that “w/o. SEA” performs similarly to MCC, since the approximation mechanism mainly focus on the reduction of computational complexity. Besides, MCC and “w/o. SEA” perform much better than “w/o. Global” and “w/o. Local”, since the proposed global and local contrastive constraints are com-
1Due to the space limit, we put the attention visualization results in the appendix (Sec. G).
plementary to each other, only one of them can not lead to large improvement. We evaluate the stability of SEA approximation and the results are shown in the appendix (Sec. D). We could find that, when the number of random features is big enough, MCC achieves competitive performances of MA close to “w/o. SEA”. We also provide some visualization results to show the effectiveness of modality alignments in the appendix (Sec. G). We mainly compare the results of SC-Transformer and MCC. Each word in the sentence can more accurately attend to the visual and audio modalities when using contrastive alignment constraints1.

heading: 4.3 Experiments for Multimodal Retrieval
text: For evaluating the scalability of MCC, we conduct experiments on MSR-VTT with 7 modalities and larger size. Please refer to appendix (Sec. H).

heading: 5 Conclusion
text: In this paper, we propose a generalizable constrained scheme for multimodal sequential learning, which could align attentional sequences from both local and global perspectives. Concretely, the multimodal contexts at the same sequential step are considered as a positive set and the remaining sets are negative. We adopt additional random feature mechanism to approximate the real process, reducing the complexity. We conduct extensive experiments on several traditional tasks, the experimental results reveal the effectiveness of our contributions. In the future, we would focus on how to sample negative groups more effectively and simulate the constrained process without approximation error.
Achknowledgement
This work was supported by National Key R&D Program of China under Grant No.2022ZD0162000.
Limitations and Future Work
We conclude the limitations of MCC as follows: (1) Although we adopt SEA to approximate the contrastive learning process to reduce the complexity, the approximation error always exists. In the future, we would try to devise a more effective unbiased estimation. The goal of unbiased estimation is to make the approximation value close to the real value, avoiding the influence to performance. We think that it need more complicated theoretical derivations to achieve the goal. For examples, borrowing the ideas of effective linear Transformer may be a good direction (Kitaev et al., 2020; Wang et al., 2020). (2) MCC can only be used for sequential input, thus, if the features are not sequential, MCC cannot generalize to. (3) the complexity is related to the sequence length of input, thus, MCC is not compatible with extremely long-sequence input.
In the future, we would focus on how to sample negative groups more effectively and simulate the constrained process without approximation error. (1), Specifically, during the local constraints, we only sample the negative groups where only one time step changes, thus, many negative groups with more changes are wasted. We decide to study more theoretical derivations to make full use of the negative groups. (2), As for the approximation error, it is equivalent to unbiased estimation, The goal of unbiased estimation is to make the approximation value close to the real value, avoiding the influence to performance. We think that it need more complicated theoretical derivations to achieve the goal. For examples, borrowing the ideas of effective linear Transformer may be a good direction (Kitaev et al., 2020; Wang et al., 2020).
Broader Impact
MCC has some real-world applications. First, MCC is plug-and-play and can enhance the alignment of different modalities when conducting multimodal interaction. In fact, many scenarios need the multimodal interaction, like the E-commerce live interaction and short video interaction. Second, MCC can check whether the different modalities
are aligned (For example, MCC can be used for detecting fake titles of the videos when the titles not correspond to the videos).

heading: A The Detailed Structure of SC-Transformer
text: To implement MCC, we employ the variant SCTransformer of MulT. The main difference between these two structure is the usage of self-attention in the cross-modal interaction stage. The comparison is shown in Figs. 5 and 6. In detail, “A→B” denotes multi-head attention mechanism (Vaswani et al., 2017) where B is the query and A denotes key and value.

heading: B The Proof of Proposition 1
text: Proposition 1: Suppose that we have the group of contexts for the t-th sequential step cti ∈ Rd where i ∈ [n]. When we randomly change the sequential step of one specific context, the obtained new group has less relevance than the original positive group. Further, if we treat the score of positive n-vector group consists of the scores of multiple (n(n−1)2 ) positive two-vector pairs, the scores of negative n-vector groups (only changing one specific context) also contain the scores of all the negative twovector pairs.
Proof 1: The fact in the first three lines is obvious. Thus, we mainly prove the proposition in the last three lines. We list the negative two-vector pairs obtained by the two methods, simultaneously. Concretely, we select one sequential step t. First, we separately calculate the scores of negative twovector pairs of n(n−1)2 positive two-vector pairs. We could obtain the following equation:
G =
n∑
i=1
n∑
j=1,j<i
ta∑
t′=1,t′ 6=t (A(cti, c
t′ j ) +A(c t′ i , c t j))
(11)
where {i, j} denotes one modality pair and there are 2(ta − 1) negative two-vector pairs for each {i, j} modality pair. Second, we list the negative two-vector pairs in the negative n-vector
groups and we only change the sequential step of one specific context. Suppose we change the sequential step of k-th modality, the relevance scores of new groups are A({ct
′ i i |i ∈ [n]}) =∑n
i ∑n j=1,j<iA(c t′i i , c t′j j ) , where t ′ i = t for i ∈ [n], i 6= k and t′k 6= t. During the process, multiple negative two-vector pairs appear and can be expressed as:
g′ = n∑
i=1,i 6=k
ta∑
t′k=1, 6=t A(cti, c
t′k k ) (12)
We can obtain corresponding negative twovector pairs by changing the sequential steps of other modalities. The total set can be expressed as:
G′ = n∑
k=1
n∑
i=1,i 6=k
ta∑
t′k=1, 6=t A(cti, c
t′k k ) (13)
We can easily find that sets G and G′ are equal. Each element in G can also be found in G′, vice versa. Thus, the proposition is proved.
C The Derivations of Eq. 9 of Main Paper
exp(
n∑
i=1
n∑
j=1,<i
〈vi, vj〉)
= exp( ‖v1 + ...+ vn‖2
2 ) ·
n∏
i=1
exp(−‖vi‖ 2
2 )
(14)
Next, let w ∈ Rd. We use the fact that:
(2π)−d/2 ∫ exp ( −‖w − c‖22/2 ) dw = 1 (15)
for any c ∈ Rd and derive:
exp( ‖v1 + ...vn‖2
2 ) = (2π)−d/2 exp( ‖v1 + ...vn‖2 2 )·
· ∫ exp( −‖w − (v1 + ...vn)‖2
2 )dw
= (2π)−d/2 ∫ exp(−‖w‖ 2
2 +w>(v1+...vn)
−‖v1+...vn‖ 2 2 + ‖v1 + ...vn‖2 2 )dw
= (2π)−d/2 ∫ exp(−‖w‖ 2
2 + w>(v1 + ...vn))dw
= (2π)−d/2 ∫ exp(
−‖w‖2 2 )· n∏
i=1
exp(w>vi)dw
= Ew∼N (0,Id)[ n∏
i=1
exp(w>v1)]
(16)
That completes the proof. We then provide the estimation error of the approximation. Suppose that the number of random features is H ,
exp(
n∑
i=1
n∑
j=1,<i
〈vi, vj〉)
= exp ( −‖v1‖
2 + ...+ ‖vn‖2 2
) ·
· Ew∼N (0,1d) [ exp ( w>z )]
(17)
where z = v1 + v2 + ... + vn, based on the fact Ew∼N (0,Id) [ exp ( w>z )] = exp ( ‖z‖2 2 ) , then we
can obtain:
MSE(expH( n∑
i=1
n∑
j=1,<i
〈vi, vj〉))
= 1
H exp (−β)Var
( exp ( w>z ))
= 1 H exp(−β)(E[exp(2w>z)]−(E[exp(w>z)])2) = 1
H exp (−β)
( exp ( 2‖z‖2 ) −exp ( z4 ))
= 1
H exp (−β) exp
( ‖z‖2 ) ( exp ( ‖z‖2 ) −1 )
= 1
H γ exp(
n∑
i=1
n∑
j=1,<i
〈vi, vj〉)2 ( 1− 1
γ
)
β = ‖v1‖2+‖v2‖2...+ ‖vn‖2 γ = exp ( ‖z‖2 ) (18) where H denotes the number of random features. It is obvious that improving H can help reduce the approximation error.
To further reduce the variance of the estimator, we entangle different random weights w1, . . . , wH to be exactly orthogonal. This can be done while maintaining unbiasedness whenever isotropic distributions N (0, Id) are used by standard GramSchmidt renormalization procedure (Choromanski et al., 2017). ORFs is a well-known method and can be applied to reduce the variance of softmax/Gaussian kernel estimators for any dimensionality d rather than just asymptotically for large enough d and leads to the first exponentially small bounds on large deviations probabilities that are strictly smaller than for non-orthogonal methods. The ORF mechanism requires H ≤ d, if H > d, ORFs still can be used locally within each d × d block.

heading: D Ablation Study of the Number of Random Features
text: According to Eq. 18, the approximation error is related to the sampling number, increasing the sampling number can reduce the error.
The ablation results are shown in Fig. 4. The red line denotes the method without SEA approximation, the complexity is O(105) as shown in Table 5, while MCC with much lower complexity O(104) (smaller sampling number) can achieve similar or better results than the model without SEA, which also means the approximation error is small.
E Illustration of Complexity Analysis
We detailedly analyze the alignment complexity before and after the approximation. The total complexity of the global contrastive alignments is O(n2(ta)n), the square complexity O(n2) arises from the combinational addition operations and the exponential complexity O((ta)n) arises from the large number of context sets. We argue that the SEA approximation can reduce the square complexity to a linear level O(n). Specifically, the relevance score of a set can be calculated with continuous multiplication operation according to the Eq. 9 (main paper). As for the exponential part, we calculate the sum of all the relevance scores like ∑ i ∑ j ∑ k aibjck =
( ∑ i ai)( ∑ j bj)( ∑
k ck). Therefore, the complexity with approximation becomes O(n2ta).
According to Eq. 9 (main paper), we can obtain the following equation:
exp(A({cti|i ∈ [n]})) = Mean( n∏
i=1
(cti) ′) (19)
where (cti) ′ denotes the transformation of cti with Eq. 9 (main paper) and Mean() denotes the mean operation for the elements in the vector. Therefore, the sum of the scores of all the n-vector group is as
follows:
ta∑
t′1=1
ta∑
t′2=1
...
ta∑
t′n=1
exp(A({ct ′ i i |i ∈ [n]}))
=
ta∑
t′1=1
ta∑
t′2=1
...
ta∑
t′n=1
Mean( n∏
i=1
(c t′i i ) ′)
= Mean( ta∑
t′1=1
ta∑
t′2=1
...
ta∑
t′n=1
n∏
i=1
(c t′i i ) ′)
= Mean( n∏
i=1
(
ta∑
t′i
(c t′i i ) ′))
(20)
Therefore, we change the complexity from O(n2(ta) n) to O(n2ta).

heading: F The Complete Results of POM
text: We shown the results of all the 16 traits in Table 6.

heading: G Attention Visualization
text: We also provide the visualization results in Fig. 7 and Fig. 8, which show that MCC can help the multimodal interaction occurs between the more related segments of different modalities. We take the word “fighting” as an example, MCC can accurately locate the word in the corresponding positions of visual and audio modalities, while SCTransformer can not.

heading: H Experiments for Multimodal Video Retrieval
text: Data Preprocessing: we follow (Gabeur et al., 2020)and use multiple pre-trained models for extracting features. Concretely, we utilize following 7 experts: Motion embeddings are extracted from S3D (Xie et al., 2018) trained on the kinetics dataset. Scene embeddings are extracted with DenseNet-161 (Huang et al., 2017) trained on the Place365 dataset (Zhou et al., 2017). OCR embeddings are encoded with a word2vec embedding. Audio embeddings are obtained with a VGGish model, trained on the YouTube-8m dataset. Speech features are extracted using the Google Cloud speech API, to extract word tokens from the audio stream, which are then encoded via pretrained word2vec (Mikolov et al., 2013a) embeddings. Face features are extracted by ResNet-50 (He et al., 2016) trained on the VGGFace2 dataset. Appearance features are extracted by SENet-154 (Hu et al., 2018) trained on ImageNet.
Experimental Details: We implement MCC based on the Transformer based backbone and make comparisons with existing methods. We utilize MMT as backbone. We implement MCC by normalizing the attention weights along the sequential steps of corresponding modalities in parallel. The hyperparameters of MCC include Adam learning rate 5× 10−5, which we decay by a multiplicative factor 0.95 every 1000 optimization steps, the
structure of projection network (where the hidden size is 512, the size of common space is 64, the number of random features is 512). λ1 and λ2 are set to 0.1 and 0.01. The temperature for contrastive learning is set to 0.2. We conduct all the experiments on RTX 3080Ti GPUs (10GB).
Compared Baselines: For fairness, we mainly compare MCC with the baseline methods CE, Dual Enc, FIT, CLIPBERT, CERT, MMT that do not utilize large-scale pre-training with HowTo100M (Miech et al., 2019) dataset.
Experimental Results: We report the evaluation results of MCC and the competing text-video retrieval methods on MSR-VTT (Table 7). Our MCC performs constantly better than other baselines (Dual Enc, FIT, CLIPBERT, CERT, ConFEDE, MMT). Specifically, the R@5 score of MCC can reach 56.4% and 57.8% of text-to-video and video-to-text tasks, making the relative improvement over the best competitor by 1.2% and 1.6%. As expected, MCC utilizing both global and local contrastive constraints exhibits better performance than that only using the indirect alignments of attention mechanism.

heading: I More Experiments on Other Datasets
text: I.1 CMU-MOSEI
The CMU-MOSEI dataset (Zadeh et al., 2018c) is the next generation of CMU-MOSI. Compared with CMU-MOSI, it has much larger training sam-
ples and more variations in speakers and video topics. Specifically, CMU-MOSEI contains 23453 manually annotated utterance-level video segments from 1000 distinct speakers and 250 different topics.
The experimental details of CMU-MOSEI are similar to CMU-MOSI. We obtain the results in Table 8. MCC still outperform all the compared methods with a better overall performance.
I.2 LSMDC LSMDC contains 118081 short video clips (about 45s) extracted from 202 movies. Each clip is annotated with a caption, extracted from either the movie script or the audio description. The test set is composed of 1000 videos, from movies not presented in the training set.
The experimental details of LSMDC are similar to MSR-VTT. We obtain the results in Table 9. MCC still outperform all the compared methods with a better overall performance.

heading: Future Work
text: In the future, we would focus on how to sample negative groups more effectively and simulate the constrained process without approximation error. Achknowledgement
This work was supported by National Key R&D Program of China under Grant No. 2022ZD0162000. Limitations and Future Work
We conclude the limitations of MCC as follows: (1) Although we adopt SEA to approximate the contrastive learning process to reduce the complexity, the approximation error always exists. In the future, we would try to devise a more effective unbiased estimation. The goal of unbiased estimation is to make the approximation value close to the real value, avoiding the influence to performance. We think that it need more complicated theoretical derivations to achieve the goal. For examples, borrowing the ideas of effective linear Transformer may be a good direction (Kitaev et al. , 2020; Wang et al. , 2020). (2) MCC can only be used for sequential input, thus, if the features are not sequential, MCC cannot generalize to. (3) the complexity is related to the sequence length of input, thus, MCC is not compatible with extremely long-sequence input. In the future, we would focus on how to sample negative groups more effectively and simulate the constrained process without approximation error. (1), Specifically, during the local constraints, we only sample the negative groups where only one time step changes, thus, many negative groups with more changes are wasted. We decide to study more theoretical derivations to make full use of the negative groups. (2), As for the approximation error, it is equivalent to unbiased estimation, The goal of unbiased estimation is to make the approximation value close to the real value, avoiding the influence to performance. We think that it need more complicated theoretical derivations to achieve the goal. For examples, borrowing the ideas of effective linear Transformer may be a good direction (Kitaev et al. , 2020; Wang et al. , 2020)