# Multimodal Framework for Long-Tailed Recognition
## 3 Experiment Details
### Ablation Experiment  
Table 5 shows the results of the ablation study, where our primary objective is to verify the contributions of different modules and strategies to the overall performance of the multimodal approach. The details of the experiments are described below.  
Experiment 1: Only the image encoder was used, without any textual information, aiming to evaluate the performance with a single visual modality. Experiment 2: Building on Experiment 1, text feature filtering was added to assess the improvement brought by the text feature filtering module in image classification. Experiment 3: Both the image encoder and text encoder were used for contrastive learning without subsequent multimodal classification to evaluate the impact of contrastive learning on the model. Experiment 4: Only the image branch of the multimodal model was used, meaning that in the second-stage classification, only image features were relied upon to assess the performance of the image branch. Experiment 5: This was the complete multimodal framework, which includes image-text contrastive learning, text feature filtering, and a dual-branch classification model (image and image-text), aiming to evaluate the contribution of all combined modules to the overall model performance.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Dataset** & \multicolumn{4}{c}{**ImageNet-LT**} & \multicolumn{4}{c}{**iNaturalist 2018**} \\  & **Many-Shot** & **Medium-Shot** & **Few-Shot** & **All** & **Many-Shot** & **Medium-Shot** & **Few-Shot** & **All** \\ \hline cRT * & 62.5 & 47.4 & 29.5 & 50.3 & 73.2 & 68.8 & 66.1 & 68.2 \\ LWS * & 61.8 & 48.6 & 33.5 & 51.2 & 71.0 & 69.8 & 68.8 & 69.5 \\ MiSLAS * & 61.7 & 51.3 & 35.8 & 52.7 & 73.2 & 72.4 & 70.4 & 71.6 \\ RIDE & 65.8 & 51.0 & 34.6 & 54.4 & 70.2 & 71.3 & 71.7 & 71.4 \\ \hline MMLTR (ours) & **77.5** & **67.4** & **50.5** & **70.1** & **73.5** & **74.8** & **74.2** & **74.1** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracy of many-shot (over 100 training images), medium-shot (20\(\sim\)100 training images), and few-shot class (fewer than 20 training images) on ImageNet-LT and iNaturalist 2018. The backbone is ResNet-50. * Means the result from MiSLAS [27].  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Datasets** & **ImageNet-LT** & **iNaturalist 2018** \\
**Backbone** & **ResNet-50** & **ResNet-50** \\ \hline LDAM-DRW * & - & 66.12 \\ BBN * & - & 69.62 \\ CAM & - & 70.87 \\ Remix-DRW & - & 70.49 \\ DisAlign & 52.90 & 70.60 \\ RIDE & 54.40 & 71.40 \\ \hline MMLTR (ours) & **70.13** & **74.16** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top-1 accuracy on ImageNet-LT and iNaturalist 2018 datasets. The backbone networks for training on the ImageNet-LT dataset are ResNet-10 and ResNet-50, and the backbone network for training on iNaturalist 2018 is ResNet-50. * Represents the result from BBN. Other results are derived from the original paper.  
The results of ablation experiments on ImageNet-LT and iNaturalist2018 datasets are reported. Contrastive learning means that the image-text comparison model is used for image and text feature extraction in the first stage, and the most relevant text feature extraction means that the most relevant features are selected for each category to prepare for the classification task model in the second stage. In the second stage of the multimodal recognition framework, the results of using one of the branches alone and using the two branches at the same time are obtained. It can be seen from the table that when all the methods proposed in this paper are used, the best results can be achieved. The results show the good effect and thus effectiveness of each part proposed in this article, as well as the whole.