## 1 Introduction
### Multimodal Joint Representation Learning
#### 2.2.2 Joint Representation Learning Layer  
With the aid of the orthogonal model, information from the other two modalities is incorporated (see Section 2.2.1). In this process, Q retains the information from the current modality, while the orthogonal projection model enriches it with information from the other modalities to form K and V. This joint representation layer facilitates the interaction of multimodal information at the same level and can be stacked multiple times to generate more hierarchical representations. For each modality, there are N cross-attention layers, and we define all modalities at the same hierarchical level as the joint representation layer, as illustrated in Figure 3.  
To process inputs from the three modalities (vibration, sound, and image), we use a 1D temporal convolution layer to capture sequential information and adjust its dimension to fit the subsequent encoder. Then, the representations pass through N (N = 6) stacked joint representation layers to acquire additional information from other modalities. The forward process of our joint representation layer (t = 0, 1,..., N - 1) can be described as follows:  
\[\mathrm{z}_{\mathrm{v}}^{\mathrm{t+1}}=\mathrm{Basic\_B}(\mathrm{z}_{\mathrm{v}} ^{\mathrm{t}}\mathrm{W}_{\mathrm{Q}_{\mathrm{v}}}^{\mathrm{t}},\mathrm{M}_{ \mathrm{v}}^{\mathrm{t}}\mathrm{W}_{\mathrm{K}_{\mathrm{v}}}^{\mathrm{t}}, \mathrm{M}_{\mathrm{v}}^{\mathrm{t}}\mathrm{W}_{\mathrm{V}_{\mathrm{v}}}^{ \mathrm{t}}),\mathrm{M}_{\mathrm{v}}^{\mathrm{t}}=\mathrm{OPM}(\mathrm{z}_{ \mathrm{v}}^{\mathrm{t}},\mathrm{z}_{\mathrm{a}}^{\mathrm{t}},\mathrm{z}_{ \mathrm{i}}^{\mathrm{t}}) \tag{14}\]  
\[\mathrm{z}_{\mathrm{a}}^{\mathrm{t+1}}=\mathrm{Basic\_B}(\mathrm{z}_{\mathrm{a}} ^{\mathrm{t}}\mathrm{W}_{\mathrm{Q}_{\mathrm{a}}}^{\mathrm{t}},\mathrm{M}_{ \mathrm{a}}^{\mathrm{t}}\mathrm{W}_{\mathrm{K}_{\mathrm{a}}}^{\mathrm{t}}, \mathrm{M}_{\mathrm{a}}^{\mathrm{t}}\mathrm{W}_{\mathrm{V}_{\mathrm{a}}}^{ \mathrm{t}}),\mathrm{M}_{\mathrm{a}}^{\mathrm{t}}=\mathrm{OPM}(\mathrm{z}_{ \mathrm{a}}^{\mathrm{t}},\mathrm{z}_{\mathrm{v}}^{\mathrm{t}},\mathrm{z}_{ \mathrm{i}}^{\mathrm{t}}) \tag{15}\]  
\[\mathrm{z}_{\mathrm{i}}^{\mathrm{t+1}}=\mathrm{Basic\_B}(\mathrm{z}_{\mathrm{i}} ^{\mathrm{t}}\mathrm{W}_{\mathrm{Q}_{\mathrm{i}}}^{\mathrm{t}},\mathrm{M}_{ \mathrm{i}}^{\mathrm{t}}\mathrm{W}_{\mathrm{K}_{\mathrm{i}}}^{\mathrm{t}}, \mathrm{M}_{\mathrm{i}}^{\mathrm{t}}\mathrm{W}_{\mathrm{V}_{\mathrm{i}}}^{ \mathrm{t}}),\mathrm{M}_{\mathrm{i}}^{\mathrm{t}}=\mathrm{OPM}(\mathrm{z}_{ \mathrm{i}}^{\mathrm{t}},\mathrm{z}_{\mathrm{v}}^{\mathrm{t}},\mathrm{z}_{ \mathrm{a}}^{\mathrm{t}}) \tag{16}\]  
where \(\mathrm{W}_{\mathrm{Q}_{\mathrm{m}}}^{\mathrm{t}}\), \(\mathrm{W}_{\mathrm{K}_{\mathrm{m}}}^{\mathrm{t}}\), and \(\mathrm{W}_{\mathrm{V}_{\mathrm{m}}}^{\mathrm{t}}\) (m \(\in\left(\mathrm{v},\mathrm{a},\mathrm{i}\right)\)) are the weights, and the input of \(\mathrm{Basic\_B}\) corresponds sequentially to the query (Q), key (K), and value (V) in the cross-attention mechanism.  
In the final joint representation layer, the hidden states of the three modalities are input into three self-attention encoders, respectively, to obtain the final fused representation for each modality, containing both multimodal and self-attention information.