# Multimodal Framework for Long-Tailed Recognition
## 2 Method
### Image-Text Contrastive Model  
Image-text contrastive learning models usually adopt a dual-encoder architecture, where one is an image encoder \(E_{I}\) and the other is a text encoder \(E_{T}\), such as CLIP [20] and CoOp [21]. The main idea is to encode images and text separately through two independent encoders and then align the features of these two modalities. Figure 2 shows the most relevant text feature extraction process. Usually, the learning goal of this kind of model is the contrastive loss; the purpose is to bring the features described by the image and the corresponding text closer and, at the same time, push away features for which the image does not match the text in the feature space. For example, in the process of training, images \(\{l_{i}\}_{i=1}^{n}\) and sentences \(\{T_{i}\}_{i=1}^{n}\) of \(n\) batch size are randomly selected from the samples to obtain image features \(\mathbf{F}_{l_{i}}=E_{I}(l_{i})\in\mathbb{R}^{D_{V}}\) and text features \(\mathbf{F}_{T_{i}}=E_{T}(T_{i})\in\mathbb{R}^{D_{L}}\), respectively, in the multimodal space. Then, the features are passed through two transformation matrices:  
\[\mathrm{v}_{i}=\frac{\mathrm{W}_{V}^{\top}\cdot\mathbf{F}_{l_{i}}}{\|\mathrm{ W}_{V}^{\top}\cdot\mathbf{F}_{l_{i}}\|},l_{i}=\frac{\mathrm{W}_{L}^{\top} \cdot\mathbf{F}_{T_{i}}}{\|\mathrm{W}_{L}^{\top}\cdot\mathbf{F}_{T_{i}}\|} \tag{1}\]  
Here, \(\mathrm{v}_{i}\) and \(\mathrm{l}_{i}\) are the D-dimensional image and text features of the \(i\)th image normalized in the multimodal space, respectively. The goal of the original contrastive learning is to bring the image features and text features of a single image closer while pulling the image and all other text features away. In order to make features of the same category as compact as possible in the feature space, features of the same category should be excluded when pulling features out. In other words, when training image features, text features that are the same as the image category should be pulled closer, while text features that are different from the image category should be pulled away. In the same way, when training  
Figure 1: An overview of the proposed framework. The dashed line above in the figure represents the image–text branch, while the dashed line below represents the image-classifier branch. The term cls-text denotes the textual database describing each category. The nums-cls graph illustrates several classes in the training set along with their corresponding sample counts, displaying a long-tailed distribution curve. The image encoder and text encoder are utilized separately to extract features from their respective modalities. Due to potential noise and interference in the textual descriptions for each category, such as ambiguous or entirely incorrect descriptions, in the second phase of training, we employ filters to retain the most representative textual features for each class. Each small square in the figure represents a feature extracted by the feature extractor for each sample, forming a vector. Different colors indicate different categories. For example, the three yellow squares represent three distinct samples, but they share the same class label. text images, image features that are the same as the text category should be pulled closer, and image features that are different from the text category should be pulled away. \[L_{v}=-\frac{1}{n}\frac{n}{i}\frac{1}{|i_{l}^{+}|}\sum_{p\in i_{l}^{+}}\log(\frac {\exp(v_{i}^{\top}l_{p}/\tau)}{\sum_{j=1}^{n}\exp(v_{i}^{\top}l_{j}/\tau)}) \tag{2}\]  
\[L_{l}=-\frac{1}{n}\frac{n}{i}\frac{1}{|i_{v}^{+}|}\sum_{p\in i_{v}^{+}}\log( \frac{\exp(l_{i}^{\top}v_{p}/\tau)}{\sum_{j=1}^{n}\exp(l_{i}^{\top}v_{j}/\tau)}) \tag{3}\]  
Specifically, \(L_{v}\) and \(L_{l}\) represent the similarity loss of image features relative to text features and the similarity loss of text features relative to image features, respectively, and \(i_{l}^{+}\) indicates that it belongs to \(i\) in the current batch.