## 1 Introduction
### Multimodal Joint Representation Learning
#### 2.2.1 Orthogonal Projection  
Our orthogonal projection can be described in the following steps (here, we set vibration as the target modality):  
* First, L2 normalization is applied to encode each sequence, obtaining \(\mathbf{z}_{\mathrm{v}}^{\mathrm{t}}\), \(\mathbf{z}_{\mathrm{a}}^{\mathrm{t}}\), and \(\mathbf{z}_{\mathrm{i}}^{\mathrm{t}}\), as well as \(\mathbf{z}_{\mathrm{v}}^{\mathrm{t}}\), \(\mathbf{z}_{\mathrm{a}}^{\mathrm{t}}\), and \(\mathbf{z}_{\mathrm{i}}^{\mathrm{t}}\). Then, we measure the correlation between the corresponding positions of two modality vectors by computing the dot product, as follows: \[\mathrm{Corr}_{\mathrm{va}}^{\mathrm{t}}=\mathbf{z}_{\mathrm{v}}^{\mathrm{t} }\cdot\mathbf{z}_{\mathrm{a}}^{\mathrm{t}}\] (7) \[\mathrm{Corr}_{\mathrm{vi}}^{\mathrm{t}}=\mathbf{z}_{\mathrm{v}}^{\mathrm{t} }\cdot\mathbf{z}_{\mathrm{i}}^{\mathrm{t}}\] (8)
* Second, we use the SoftMax operation to approximate the correlation between modality v and modality a (or i) at corresponding positions, as well as the correlation between each element in \(\mathrm{Corr}_{\mathrm{va}}\)(or \(\mathrm{Corr}_{\mathrm{vi}}\)) and modality v (or i). \[\mathrm{Corr}_{\mathrm{va}}^{\mathrm{t}}=\mathrm{SoftMax}(\mathrm{Corr}_{ \mathrm{va}}^{\mathrm{t}})\] (9) \[\mathrm{Corr}_{\mathrm{vi}}^{\mathrm{t}}=\mathrm{SoftMax}(\mathrm{Corr}_{ \mathrm{vi}}^{\mathrm{t}})\] (10)
* Third, we perform a \(1-\mathrm{x}\) operation on the correlation values (for each input x), returning \(1-\mathrm{x}\), performing subtraction at each position to obtain a dissimilarity vector. This vector measures the degree of difference between the two modality representations at corresponding positions. Then, we multiply the original representations (\(\mathbf{z}_{\mathrm{a}}^{\mathrm{t}}\) and \(\mathbf{z}_{\mathrm{i}}^{\mathrm{t}}\)) with these dissimilarity vectors to obtain the information components that are orthogonal to the target modality. These components eliminate redundant information between other modalities and the target modality, thereby preserving only the parts that are orthogonal to the target modality vector for further processing. \[\mathrm{Orth}_{\mathrm{va}}^{\mathrm{t}}=\mathbf{z}_{\mathrm{a}}^{\mathrm{t} }\cdot(1-\mathrm{Corr}_{\mathrm{va}}^{\mathrm{t}})\] (11) \[\mathrm{Orth}_{\mathrm{vi}}^{\mathrm{t}}=\mathbf{z}_{\mathrm{i}}^{\mathrm{t} }\cdot(1-\mathrm{Corr}_{\mathrm{vi}}^{\mathrm{t}})\] (12)
* Then, these orthogonal components are added to the target modality, obtaining the fused latent adaptation from modality a and i to v. In this process, the original representation of modality v is preserved while incorporating complementary information from other modalities. \[\mathrm{M}_{\mathrm{v}}^{\mathrm{t}}=\mathrm{OP}(\mathbf{z}_{\mathrm{v}}^{ \mathrm{t}},\mathbf{z}_{\mathrm{a}}^{\mathrm{t}},\mathbf{z}_{\mathrm{i}}^{ \mathrm{t}})=\mathbf{z}_{\mathrm{v}}^{\mathrm{t}}+\mathrm{Orth}_{\mathrm{va}} ^{\mathrm{t}}+\mathrm{Orth}_{\mathrm{vi}}^{\mathrm{t}}\] (13)
* After this, \(\mathrm{M}_{\mathrm{v}}^{\mathrm{t}}\) can participate in the cross-attention process of the t-th encoder of modality v (serving as the source for K and V).