# Multimodal Framework for Long-Tailed Recognition
## 2 Method
### Double-Branch Multimodal Recognition Framework  
In this section, a simple and effective two-branch multimodal long-tailed data recognition framework is introduced. The two branches of the framework are the image-text branch and the image branch. In this context, the image-text branch uses both image information and text information for judgment, while the image branch only uses image information. In the previous section, \(S\) text features \(\mathbf{F}_{T}\in\mathbb{R}^{S\times C\times D}\) and images for each category have been obtained as features \(\mathbf{F}_{I}\in\mathbb{R}^{D}\). The formula defining the logical output for computing the image-text branch is  
\[\begin{split} O_{I,T}&=\sigma(Attention(\mathbf{F}_{I},\mathbf{F}_{T},\mathbf{F}_{T})\cdot\mathbf{F}_{I})\\ &=\sigma(\sigma(\frac{(\mathbf{W}_{I}^{\top}\mathbf{F}_{I})^{\top }(\mathbf{W}_{T}^{\top}\mathbf{F}_{T})}{\sqrt{d_{T}}})\mathbf{F}_{T}\cdot \mathbf{F}_{I})\end{split} \tag{10}\]  
where \(\mathbf{W}_{I}\) and \(\mathbf{W}_{T}\) represent the linear transformation matrix of image features and text features, respectively; \(\sigma(\cdot)\) represents the Softmax Function; and \(d_{T}\) represents the latitude of the text feature.  
The formula for the image branch logic output is defined as  
\[O_{I}=\sigma(\mathbf{W}_{I}^{\prime\top}\mathbf{F}_{I}) \tag{11}\]  
where \(\mathbf{W}_{I}^{\prime}\) represents the linear transformation matrix of the image branch.  
The formulation of the final logistic output and loss function is defined as  
\[L_{cls}=-\sum_{c=1}^{C}y^{c}\times log(O_{I,T})-\sum_{c=1}^{C}y^{c}\times log(O _{I}) \tag{12}\]  
where \(y^{c}\) represents the label of the \(c\)th category. Algorithm 1 gives the data processing flow of the model.  
```
0: Randomly select batch size \(n\) of images \(\{I_{i}\}_{i=1}^{n}\) and text \(\{T_{i}\}_{i=1}^{n}\) and the hyperparameters \(E\) and \(E_{max}\).
0: Classification result \(p_{i}\).
1: Initialize hyperparameters \(\lambda\) according to Equation (6).
2:for\(t=1\) to \(T\)do
3:\(F_{I_{i}}=E_{I}(I_{i})\)
4:\(F_{T_{i}}=E_{T}(T_{i})\)
5: Calculate \(rel_{t,i}\) according to Equation (8).
6: Calculate \(score_{t,c}\) according to Equation (9).
7: Select the best text feature for each category based on the score.
8: Normalize \(F_{I_{i}}\) and \(F_{T_{i}}\) according to Equation (1).
9: Calculate the similarity loss of image features relative to text features \(L_{v}\) according to Equation (2).
10: Calculate the similarity loss of text features relative to image features \(L_{I}\) according to Equation (3).
11: Calculate \(L_{KL}\) according to Equation (4).
12: Calculate the overall loss function of the contrastive learning phase \(L_{cl}\) according to Equation (5).
13: Optimize \(E_{I}\) and \(E_{T}\) to \(L_{cl}\).
14: Calculate the logical output of the image-text branch \(O_{I,T}\) according to Equation (10).
15: Calculate the image branch logic output \(O_{I}\) according to Equation (11).
16: Calculate the final classification loss \(L_{cls}\) according to Equation (12).
17: Optimize Dot-Head and Classifier according to \(L_{cls}\).
18:endfor
19: Calculate the classification results \(p_{i}\) based on Dot-Head and Classifier.
```  
**Algorithm 1** Long-tailed recognition algorithm.