## 4 Common Space Fusion through Similarity Learning
### Detailed Method Description  
In Figure 7, we can see an overview of the proposed multimodal fusion pipeline. Given an input batch containing the text and visual modality features, we extracted the text and visual representations using two unimodal encoders. For training, we used a joint contrastive loss and classification loss. The contrastive loss brings the corresponding text and image features in the batch close in the multimodal space, while the classification loss is used to solve the task. The model was jointly trained using both losses through multitask learning. In detail, given an input batch \(B\), we first passed each image \(v_{k}\) in the batch through the visual encoder \(g_{v}\) and each textual description \(t_{k}\) through the text encoder \(g_{t}\), to obtain intermediate representations \(V_{k}\) and \(T_{k}\):  
\[V_{k}=g_{v}(v_{k}),\qquad T_{k}=g_{t}(t_{k}) \tag{8}\]  
We created the shared projection space using a contrastive loss \(L_{CO}\). Our contrastive loss is inspired by the one used in CLIP [79]. To obtain \(L_{CO}\), we first created the cross-modal cosine distance matrix \(A\)  
Figure 7: The multimodal architecture used in our experiments. \[A= \begin{pmatrix}s_{1}^{t}&s_{2}^{t}&s_{i}^{t}&\cdots&s_{|B|}^{t}\\ c(T_{1},V_{1})&c(T_{2},V_{1})&\cdots&\cdots&c(T_{|B|},V_{1})\\ \vdots&\ddots&&\vdots&\vdots\\ c(T_{1},V_{i})&\cdots&c(T_{i},V_{i})&\cdots&c(T_{|B|},V_{i})\\ \vdots&&&&\ddots&\vdots\\ c(T_{1},V_{|B|})&\cdots&\cdots&&c(T_{|B|},V_{|B|})\end{pmatrix}\begin{matrix}s_{1}^ {v}\\ \vdots\\ \vdots\\ s_{i}^{v}\\ \vdots\\ \vdots\\ s_{|B|}^{v}\end{matrix}  \tag{9}\]  
of all the encoded pairs \(T_{i},V_{j}\) in the batch, where \(i,j\in[1,|B|]\). Each row \(s_{i}^{v}\) is a vector of the pairwise cosine similarities of the visual features \(V_{i}\) with every textual feature vector in the batch \(T_{j},j\in[1,|B|]\). Similarly, each column \(s_{i}^{t}\) is a vector of pairwise cosine similarities of the textual features \(T_{i}\) with every visual feature vector in the batch \(V_{i},j\in[1,|B|]\). For the first part of the contrastive loss, we constructed the n-pair loss [64; 65] over the columns of \(A\), using the identity matrix \(\mathbb{I}_{|B|}\) as labels, as in Equation (10)  
\[L_{1}=-\frac{1}{|B|}\sum_{i=1}^{|B|}\sum_{j=1}^{|B|}\Bigl{[}(\mathbb{I}_{|B|})_ {ij}log(s_{i}^{t})_{j}\Bigr{]}. \tag{10}\]  
Similarly, the second part of the contrastive loss is taken over the rows of \(A\):  
\[L_{2}=-\frac{1}{|B|}\sum_{i=1}^{|B|}\sum_{j=1}^{|B|}\Bigl{[}(\mathbb{I}_{|B|})_ {ij}log(s_{i}^{v})_{j}\Bigr{]} \tag{11}\]  
The contrastive loss \(L_{\text{CO}}\) is given by Equation (12):  
\[L_{\text{CO}}=\frac{1}{2}(L_{1}+L_{2}). \tag{12}\]  
The goal of the contrastive loss is that the diagonal elements of \(A\) take values close to 1, while all other elements take values close to 0. This creates a shared multimodal space \(\mathcal{S}\), where image representations \(V_{i}\in\mathcal{S}\) and text representations \(T_{i}\in\mathcal{S}\) that correspond to the same sample are close, while text and image representations \(V_{i},T_{j}\in\mathcal{S},i\
eq j\) that correspond to different samples are further away from each other. For the task loss, we created the multimodal representation in Equation (13) for each sample \(k\) in the batch:  
\[m_{k}=v_{k}\oplus t_{k}\oplus\left(\frac{1}{2}(V_{k}+T_{k})\right) \tag{13}\]  
where \(\bigoplus\) denotes the vector concatenation. The representation \(m\) is based on both the shared space representations \(V_{k}\) and \(T_{k}\) and the individual modality representations \(v_{k}\) and \(t_{k}\), to capture both unimodal and bimodal interactions. The vector \(m\) was then fed through a classification network, described in Equation (14)  
\[ \begin{split}\alpha_{k}&=f_{\theta}(m_{k})\\ \ o_{k}^{\prime}&=a(o_{k})\\ \hat{y}_{k}&=f_{\theta^{\prime}}(o_{k}^{\prime})\end{split}  \tag{14}\]  
where \(f_{\theta}\) denotes a feedforward layer and \(a\) is the GELU activation [86].