Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards
Authors: Xiaoyu Yang
Jie Lu
En Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Concept Drift
Multi-modal Large Language Model
Data Defect
Primary area: learning theory

Abstract: Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have increasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MLLMs in concept drift settings remain largely underexplored. In this paper, we reveal the susceptibility and vulnerability of Vision-Language (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training. To effectively address these challenges, we propose a unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL model to unpredictable distribution changes. Additionally, a T-distribution based drift adapter is proposed to effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distribution modeling. Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL models, particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model's ability to adapt to the long-tailed open world. Furthermore, we create a set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open-world setting, to validate our findings. To foster the development of the multi-modal community, we have made both OpenMMlo datasets and our code publicly available at: https://github.com/XiaoyuYoung/ConceptDriftMLLMs.

Published as a conference paper at ICLR 2025
ADAPTING MULTI-MODAL LARGE LANGUAGE MODEL TO CONCEPT DRIFT FROM PRE-TRAINING ONWARDS
Xiaoyu Yang, Jie Lu, En Yu Australian Artificial Intelligence Institute (AAII), Faulty of Engineering and Information Technology,
University of Technology Sydney, Australia. xi xiaoyu. yang student. uts. edu. au; (jie.lux en. yu- edu, au*
ABSTRACT
Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have in- creasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MILMs in concept drift settings remain largely underexplored. In this paper, we reveal the susceptibility and vulnerability of Languae (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training To effectively address these challenges, we pro- pose unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL. model to unpredictable distribu- tion changes. Additionally, T-distribution based drift adapter is proposed effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distri- bution modeling. Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL mod- els. particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model's ability to adapt to the long-tailed open world. Furthermore, we create set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open-world setting, to validate our findings. To foster the development of the multi-modal commu- nity, we have made both OpenMMlo datasets and our code publicly available at: http://hhhhtt.aoooo/ooooooo/nnnn Dwtpr//oooooooooeoeerreeeMi
INTRODUCTION
The rapid expansion of data availability has created significant challenges for multi-modal large language models (MLLMs), particularly in addressing concept drift, which predominantly manifests as gradual drift and sudden drift Lu et al. (2019). Among them, tailed drift represents a classic illustration of gradual drift, emerging due to severe data imbalance, where the distributions of long- tail categories evolve because of their intrinsic sparsity and noise. Concurrently, sudden drift is mainly represented by OOD drift, as the model encounters new, previously unseen concepts, resulting in distributional shifts that disrupt its ability to generalize in an open-world context. While the issues of long-tailed recognition and concept drift in open- world settings have been extensively studied in visual models Liu et al. (2022b) and language models Kandpal et al. (2023), their impact on MLLMs, particularly vision-language (VL) models, remains largely unexplored. In this work, we aim to bridge this gap by providing systematic analysis of how tailed drift and OOD drift affect VL models during both pre-training and fine- tinitt phases. Our findings highlight critical vulnerabilities of current VL models in adapting these challenges, underscoring the need for novel strategies to enhance their robustness in dynamic, open-world environments.
"Correspondence Jie Lu and En Yu

---

Published as a conference paper at ICLR 2025
Pre-training: As illustrated in Figure la a comparison of the VL model trained on the balanced dataset ImageNet Russakovsky et al. (2015)) and the imbalanced dataset ImageNet-LT Liu et al. (2022b) is conducted. Due to the implicit feature centers of each category, we approximate them by averaging unit image and text features obtained by samples on the test set. To assess the intra-class compactness, the cosine distance between the image feature center and the text feature center from the same category is calculated and expressed as degrees. It is evident that training on the imbalanced dataset leads to a higher degree, indicating worse intra-class compactness brought by the tailed drift. Besides, with the tail drift intensifies, it results in deterioration of the image-text alignment performance in tailed categories. Beyond the deterioration in tailed categories, the tailed drift also affects the image-text alignment in head categories with abundant training samples, which means that leads to an overall performance degradation, not just the tailed categories. From the perspective of inter-class separability, we measure the average cosine distance from an image feature center to text feature centers of different categories. Figure Ia depicts that the VL model trained on the imbalanced dataset has lower inter-class separability. What's more, utilize KNN to extract 100 image and text feature centers of OOD samples to verify the impact of OOD drift on the pre-training of the VL model. Compared to training on the balanced dataset, the VL model trained under an imbalanced scenario is harder to distinguish between ID samples and OOD samples from the open world due to their similar inter-class separability. The undistinguished OOD drift will bias the underlying distribution of the feature space in the VL model, further disturbing the image-text alignment in the pre- pre-training.
Degrev
Degree
Entes Class Spannbillyy
90
000 vs Sasorahiitt
Categories
40
60-
20
Long Tailed
OOD
Categories
110
120
Nom ofaumpls
140-
(a) Image-Text Alignment in Pre-training
(b) Feature Space Allocation in Fine-tuning
The impacts of tailed drift and OOD drift the vision language model in the stages of pre-training and fine-tuning, respectively. (a) In terms of the pre-training, we visualize the alignment results pre-trained on both a balanced dataset (denoted as BL) and imbalanced dataset (as LT) without OOD samples, under the same balanced test set. The cosine metric is used to measure the distances between unit image and text features across various categories including OOD samples, which is expressed as degrees. A smaller degree indicates a higher level of similarity between the features. Thus, it provides feature-level visualization of the intra-class compactness and inter-class separability in the vision language model. (b) In the context of fine-tuning in imbalance datasets, the mutual cosine distance between the centers of each category in the classifier is directly visualized to illustrate the feature space of the classifier, denoted as blue bars. Besides, the average cosine distance between each category center and OOD samples is calculated, which is represented as orange bars.
Figure I:
We explicitly leverage the weights of the embedding layer in the VL model to visualize its feature space. The average cosine distance between each category and others is calculated as exhibited in the blue bars of Figure [b] With the decreasing of the training samples, a smaller degree means a worse inter-class separability. It verified that tail drift leads to compression of the feature space for tailed categories with limited number of training samples, while head categories with abundant samples dominate the overall feature space of the VL model. Moreover, the average cosine distance between each category and unit features extracted by OOD samples is applied to reveal the OOD separability as denoted as orange bars in Figure [b] Since head categories occupy most of the feature space, OOD samples are closer to the center of the head categories compared to the tail categories, implying that in the stage of ane-ttning, it is difficult for the VL model to distinguish between OOD samples under imbalanced scenarios.
Fine-tuning:
2

---

Published as a conference paper at ICLR 2025
To effectively address tailed drift and OOD drift within unified framework, which often occurs simultancously, we encapsulate them using the concept drift theory. Therefore, summarizing the above challenges of vision language models in the long-tailes open world, it raises the important question:
How to adapt multi-modal large language model to concept drift in the long-tailed open world?
Remark 1.1. Research Objective: Our focus lies in addressing the drift that MLLMs exhibit when confronted with the long-tailed open world, rather than leveraging MILMMs to enhance classification performance on longetailed open datasets. The classification is exploited as the downstream task to provide a more intuitive visualization of the concept drift suffered by the MLLMs, which could have other downstream tasks.
Therefore, we propose a concept drift-aware multi-modal large language model, mitigating the tail drift and OOD drift encountered in the long-tailed open world. Firstly, we introduce the concept drift theory to the multi-modal domain, which provides a more holistic perspective to explain tailed drift and OOD drift. Then, the T-distributed adapter is proposed be embedded in the hyperspherical feature space. It aligns image-text features for contrastive learning in pre-training. The desirable light-tailed property of the proposed T-distributed spherical metric (Thp) prevents the compression of tailed categories and mitigates the crowding of feature space caused by tailed drift. Besides, in fine-tuning, the adapter projects the features to the decision boundary and detects OOD samples at the feature level based on the underlying distribution. The proposed T-hp distribution explicitly models the feature space with concrete feature centers, optimizing large inter-class margins and yielding more desirable hyperspherical embeddings. And a simple non-parametric KNN is adopted to distinguish the OOD sample based on the T-hp distribution. Finally, we construct a group of multi-moda long- sailed open datasets to support our claims.
In summary, our paper mainly makes the following contributions:
1. We are the pioneers in revealing the unexplored impacts of concept drift to multi-modal large language models, especially in the image-text alignment in the pre-training and feature space allocation in the fine-tuning. This allows future research more comprehensively study the impact of defect data on MLLMs.
2. The concept drift theory is introduced and extended to multi-modal, integrating the tailed drift and OOD drift in unified framework. And the T-distributed spherical adapter is proposed to perform the tailed adaptation and OOD detection in the pre-training and fine- tuning stage of the VL model.
3. Extensive experiments evaluate the performance of our method under the long-tailed open world. Compared to specialized models, ours demonstrates superior performance in down- stream tasks of long-tailed classification and OOD detection. Crucially, our model effectively addresses drift in image-text alignment, facilitating large-scale pre-training of MLLMs.
4. We build a group of multi-modal datasets OpenMMlo under the long-tailed open world by extending existing image-based long-tailed open datasets. It contains about 740k image- caption pairs with related category annotations. To support and encourage the community focused on multi-modall we have made both the OpenMMlo and our code public.
2 METHODOLOGY
2.1 MUL.TI-MODAL CONCEPT DRIFT THFORY
Concept drift is phenomenon in which the statistical properties of a target domain change over time in an arbitrary way Lu et al. (2019). Formally, given a set of examples denoted as the data stream So.t = (dg, ..., ). where d; (X,, yi) is one data instance, X, and y; respectively denote the fea- ture vector and the label, and f represents the timestamp of the instance in the data stream. So. follows a certain distribution Fo,((X; v). The concept drift is formalized as: 3t P((X, y) o P++ (X,y), y). where the joint probability Pa(X. y) can be decomposed the concept drift due to tailed and OOD data often co-occur, they are fundamentally distinct phe- nomena. The tailed concept drift foucus on the drift in (X),, while (ı = P.+1 (g) X) remains unchanged. However, the OOD drift from the unknown categories triggers the drift of both X)) and P. (X), that (x( # P (y) X) and Px(X) * (X). Therefore, concept drift theory
P.(X, = P. (X) x P((y(X). Although
3

---

Published as a conference paper at ICLR 2025
provides unified framework harmonize the tailed shift and OOD shift that often occur together, enabling more robust and adaptive deep learning models.
In the context of multi-modal vision language models, extend the concept drift theory from single data stream to multiple data streams. Each modality is associated with distinct data stream. Thereby, the multi-modal concept drift framework robustly handle the complex, heterogeneous data distributions inherent to vision language models. Therefore, we formally define multi-modal concept drift as follows:
Definition 2.1. Assume that there are N data streams corresponding to N modalities, given a set of examples denoted as So. = {Su....S., .... Sch. where S; ($1....,,j... 8N) and s; = (Xij. yi) one data instance from single j-th data stream, Xij is the feature vector, y: is the label and is the timestamp of the data stream. So.t follows certain distribution Fo,x(Si, the multi-modal concept drift occurs timestamp L 1, if (S.) + Pet1,ca(S,) denoted as:
36: P.(..) + Pst1(S.).
(1)
Sample
T-distributed Adapter
Encoder
Image
Projector
Brtt Aduaice
Vision feature Space Alignment
Image groun ded decader
Image
1.
Ner plive nlowrr tmands
pack dovag powe Hook seglins outloste
Text Encoder
Projec rnjjecton to
Str Stream
Hign The buter is chooe Aandl soone Ronsed spuraci- harls
Hopr aper-spherical Embedding
Toxt
Stage & Pre-training Adaption
Text decader
Projector
T-Router
FFN
FFN
Image Encoder
grounded
FFN
Stage il: Fine tuning on Dwinstreem Tasks
Figure 2: The workflow of our methodology, which consists of stages: the pre-training of the vision-language model and the fine-tuning on downstream tasks. Within the data streaming, a drift adaptation window slides detect changes data distribution and subsequently update the model, in both pre-training and fine-tuning. In the pre-training, the T-distributed adapter aligns visual and textual feature space by image-text contrastive learning, with large inter-class margin. Coupled with the language model loss, they drive the training of all modules. In the downstream task, the image encoder and the text decoder are frozen out training, with a lincar projector fusing image-text features. Additionally, mixture of expert modules is leveraged with the T-distributed adapter as the router, allowing it to effectively adapt tail drift and perform OOD drift detection based on the distribution.
2.2 T-DISTRIBUTED ADAPTER FOR CONCEPT DRIFT
To adapt the vision language model to concept drift, it is essential to adapt the model to align with the evolving data distribution, which can be formally defined as:
147 (())))))))))))
min
where f() denotes the vision language model trained by the data stream St-A.11 from the drift
adaption window with the size of And the model is driven by the target metric continuously to adapt the drift in given time period [t, + 7]] Thus, one prevalent method for detecting and adapting
4

---

Published as a conference paper at ICLR 2025
to concept drift involves designing metries based on data distributions that can effectively counteract the impacts of sudden and gradual changes within the time window Diao et al. (002)): : Yu et al. (2024). Building upon this thinking, integrate directional statistics into distribution -asee based drift detection and adaptation, proposing a T-distribbted adapter to alleviate in the vision language model. Firstly,
we provide an overview of directional statistics in the Appendix A.2.1 Then, we will introduce the T-distributed adapter.
Inspired by the T-SNE Van der Maaten & Hinton (2008), design a T-distribution based metric in hypersphere (T-hp), which follows the density:
2
Px (xl1) n(1 p. x(1)'
where 20()) E g denotes the unit feature vector, / sd-1 represents the center of category and α 2 0 symbolizes the concentration of the distribution, with higher values indicating greater concentration around the center μ. Accordingly, we can get the marginal normalizer:
2 Jz KnT x(0 de 2- [(a)[(3) [()
Nr d)
T(a + B)
where a 1-1 B = 433
2. and T(-) represents the gamma function. Combined with Eq. 9 in
Appendix A.2. the normalizer Nx (d) of density px(x; is:
2
r(a)
Nx d) d) Ad-2
(5)
T(a+ + +)
Thus, the probability density function of the proposed Thp distribution is as follows:
p(z(1) (x x(1 n² "'')
2
x1 Thp(u).
(6)
The detailed derivation process is provided in Appendix A.2
In terms of adapting to tailed concept drift, the Thp metric with a large concentration exhibits light-tailed property, wherein the probability density function exhibits a faster rate of decay as the values increase, relative to the vMF metric, trated in Figure 3 The high kurtosis of Thp is characterized that it yields high confidence only when the feature vector is sufficiently close to the center of the category, thereby min- imizing the influence of head category samples on the tail category centers. Formally, Tamp (1), notes the T-distributed metric on hypersphere, where non-zero value setting to 1 to avoid the denominator of O, and Lime z) exp(xp"> z)) represents the vMF metric. Given an unit feature vector Whend from the head categories, the gra- dient of the metric I, over the tailed category center Pual is (L(Men
illus-
=(I=""=0)++ de-
2
The proposed T- distributed spherical metric with various and the classical vMF metric when
Figure
Due to T Thead E [-,, when k > 1, it is readily obtain that aLag .Laat
Optii
Consequently, the light-tailed Thp distribution effectively counteracts the squeezing of tail
Bjuu "ppuil
categories caused by an overwhelming number of head samples, thereby alleviating the bias induced by the tailed concept drift.
Likewise, the Thp metric is directly applied to detect the OOD concept drift. A sample with a unit feature vector is deemed out-of-distributiin if it lies at a relatively large distance from the in-distribution (ID) data the eigenspace. Following Sun et al. (2022). a simple non- non-parametric KNN is adopted to partition the data into two sets (ID vs. OOD), which does not impose any distributional assumption on the feature space. Here the distance is the Thp metric with respect to the k-th nearest neighbor.
2.3 T-DISTRIBUTED VISION LANGUAGE MODEL FOR THE CONCEPT DRIFT
As illustrated in Figure 2 our proposed vision language model contains two stages, the pre-training and the fine-tuning on the downstream task. Specifically, the classification task is chosen to visualize
5

---

Published as a conference paper at ICLR 2025
the impact of the bias caused by the long-tailed open world on the model. The molti-modal concept drift theory offers a unified framework for integrating gradual drift adaptation and sudden drift detection, where heterogeneous image and text imputs are treated as distinct data streams.
The proposed vision language model follows the encoder-decoder mixture architecture of the Blip Li et al. (2022c), containing an image encoder, text encoder and an image-grounded text decoder. With an input image I E ## the visual features Time are extracted by the image encoder Eing and further projected the spherical eigenspace by the L2 normalizer Pnom:
(7) where n. is the number of visual features and d represents the feature dimension. The image encoder Fing can be any common visual backbones, such as Vit-Base Dosoviiskiy et al. (202T), Vit-Large Dosovitskiy et al. (202T) and ResNeXt-50 et al. (2017). In terms of the text encoder Eva. with a processed input text sequence Tú, text features are extracted by the language encoder and further projected to the spherical eigenspace by the L2 normalizer P
Zing = normm (Eing (I)) E R"xd
Tiu Paorm (Exa (T;)) R
(8)
where n is the number of input tokens and d represents the feature dimension. our case, Bert Devlin et al. (2019a) is used as the language encoder, where a [CLS] token is added to the start of the text input for sentence summarization. Additionally, an image-grounded text decoder is employed to produce textual description corresponding to a provided image. Utilizing the input visual features Timg and text features Ztat, we initially create fused multi-modal representations by merging the image and text feature embeddings. These combined features act as the keys and values within the cross-attention blocks in the predicted partial sequence Viej. the decoder iteratively forecasts the token position j, effectively producing textual descriptions corresponding across modalities.
text decoder. Through conditioning on the already
In the pre-training of the vision language model, the T-distributed adapter aligns image and text encoders by contrastive learning. It seeks to align visual and textual transformer feature spaces by promoting similar representations for positive pairs and dissimilar representations for negative pairs. More importantly, our approach circumvents model bias stemming from the long-tailed distribution of data. Specifically, given mini-batch with N image-text feature pairs, we calculate the N x N Thp similarity of the cross between image and text features. N correct pairs are recognized as positive samples to maximize the Thp similarity, whereas the rest of N - N are negative samples to minimize the similarity. And we follow the ALBEF Li et al. (2021) to use soft labels from a momentum encoder as training targets to account for the potential positives in the negative pairs.
Additionally, coupled with the T-distributed adapter, language modeling loss is utilized to activate the mmage-grounding text encoder for generating coherent and detailed captions based on the image, further propelling the training of all three modules. Driven by language modeling loss, the model is trained to optimize a cross-entropy loss with label smoothing, to maximize the likelihood of the generated text in an autoregressive manner.
In the downstream classification task, we leverage the T-distributed adapter as the router to distribute features to various FFNs as experts. Furthermore, by explicitly modeling the feature space. the T-router enables a straightforward application of non-parametric KNN to effectively partition the data into ID and OOD samples. Besides, following the Blip Li et al. (2022c), the image encoder and the text decoder are frozen out of training during fine-tuning. A head of the classifier with two linear layers embeds features into the spherical eigenspace trained.
2.4 BUILDING MULTI- DATASET OPENMMLO FOR THE LONG-TAILED OPEN WORLD
As the parameters of large models continue to expand, the demand for extensive training data also escalates. However, due the inherent challenge of obtaining images and related captions, most multi-modal datasets struggle to be balanced in an open world, while cleaning the data requires huge costs. Thus, our aspiration is for the model to adeptly acclimate to the imbalanced dataset by itself, acquiring abundant knowledge with more and more data but not exhibiting bias. In this context, more realistic training dataset for vision language models is required to validate their potential to be trained under the long-tailed open world. Recognizing the demand for higher-quality mudti-modal data with long-taiil tailed distribution in an open world, we developed a group of datasets called Open Multi-mode Long-Tailes OOD Datasets (OpenMMlo).

---

Published as a conference paper at ICLR 2025
We extend the open-source datasets, namely ImageNet-LL Liu et al. (2019), iNatualist2018 Van Horn et al. (2018) and Places-LT Liu et al. (2019). ImageNet-LT has 1,000 classes and contains 115.8k samples, with maximum of 1,280 samples and minimum of 5 samples for category. Besides, it consists of 18k images for OOD detection. Places-LTT has 184.5K samples from 365 classes, with class samples ranging from 4, 4,980 to 5. The iNaturalist 2018 is large-scale species dataset collected in the natural world with 437.55 samples for 8, 8,142 classes We use the InstructBLIP Dai et al.) al. (2023) to generate the related caption of the image, with the prompt of "What does this picture describe? Please describe in detail its size, location, color, and its relationship to the surroundings." " And, we define long-tailed tailed data in image-caption pairs according to the image categories, which are provided in open-source image datasets. Concerning related captions based on images, we counted the word frequencies and found that their distribution is similar to the image categories distribution, which is imbalanced. For more details about OpenMMlo, please refer to Appendix A.4
EXPERIMENTS
In this section, we first present the performance in downstream long-tailed classification and OOD detection tasks, which is induced by tail drift and OOD drift on MLLMs. Then, we evaluate the interior feature space of the VL model and further demonstrate our method alleviates the crowding and bias problems caused by the tail drift and OOD drift. The constructed long-tailed multi-modal dataset OpenMMlo is utilized for training and validating. In terms of the OOD drift detection, we follow the setting of CIDER Ming al. 2023). The model is trained on CIFAR LIFAR100-LT1 100- Krizheskkyy & Hinton (2009) with an imbalance ratio of 100, and validated on external OOD datasets including SVHNNetzee et al. 20II), (rrcess65//hhuu et al. (2017). LSUN Yu al.) iSUN Xu et al. and Texture Cimpoi et al. 2014). More detailed experimental implementations are given in Appendix A.3
3.1 TAMING THE TAILED DRIFT AND OOD DRIFT FOR ROBUST FINE-TUNING
We compare our proposed vision-language model with other models to explicitly demonstrate its superior performance in long-tailed open-world scenarios. As shown in Table [ our model demonstrates exceptional overall performance in long-tailed classification across two large-scale datasets, namely the ImageNet-LT and iNaturalist 2018. To ensure more equitable comparison, we opted to conduct pre- training using ImageNet and iNaturalist datasets separately, rather than pre-training the entire OpenMMlo. Besides, it is worth noting that training from scratch means that our method uses the imbalanced dataset for pre-training instead of utilizing the pre-trained model, such as clip Radford et al. (202T) pre-trained by the large WTT dataset. The results validate the robustness of our vision language model against biases arising from tailed drift, particularly when leveraging large-scale data for both pre-training and fine-tuning.
As shown in Table 1 compared to other methods trained from scratch, especially the ViT model, our model demonstrates a notable lead across all metries, indicating our effective mitigation of concept drift during the pre-training and providing robust pre- trained models for downstream tasks. Furthermore, to compare the current long-tailed methods, we apply the same setup as LIFT SShi et al. (2024), i.e., using the pre-trained model of the clip and only fine-tuning. Only fine-tuning means that the method does not pre-train the model on the long-tail dataset, while directly using the parameters of CLIP pre-trained on high-quality and large-scale WIT dataset. And they only fine-tune the model on long-tailed datasets. It is worth noting that, most vision language models only focus on the impact of tailed drift in the fine-tuning, such as DONG et al. (2023). BALLAD Ma al. (22TT) Decoder Wang et al. (2024), VL-LTR Tian et al. (202)) and LIFT et al. (2024). We are the pioneers in revealing the unexplored impacts of concept drift from pre- pre-training onwards. The superior results on medium and few splits of ImageNet-L.T demonstrate the adaptability and robustness of our model in dealing with the gradual drift caused by tail data. Besides, although we are slightly behind LIFT et al. (2024) in the few split of iNatualist2018, we still surpass it overall, exhibiting that our method does not compromise the accuracy of the head category to improve the tailed.
Beyond that, we compare our method with the CLIP Radford et al. (2021) 1 under zero-shot, linear probing and fine-tuning, where CLIP results are from the Decoder Wang et al. 2024).. Based on the zero-shot results, it is evident that CLIP, even trained
large-scale and high-quality WIT datasets, struggles to address the issue of tailed drift. CLIP only achieves 5.5% accuracy on iNaturalist2019 18, and the accuracy variance between many-shot and medium-shot scenarios is 1 11.6% on ImageNet-L.T.
7

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results of long-tailes classification on ImageNet-LT and iNatualist2018. The best-performing models are highlighted in red. Many, Medium and Few denote the evaluated splits of many-shot (> 100 training samples), medium-shot (20- 100 samples) and few-shot (<20 samples). Top-1 accuracy is applied to evaluate the performance of different methods. Additionally, means the model trained with the resolution of 384 x 384. Besides, ZS denotes zero-shot results of the CLIP model, LP represents the linear probing results, and FT means the fine-tuning results.
ImageNet-LT
iNaturalist 2018
Methods
Backbones
Many Medium Few
All Many Medium Few All
Training from seratch
cRT
Kang al. 2019)
61.8
46.2
27.3
49.6
69.0
66.0
63.2
65.2
LWS Kang et al. (2019)
60.2
47.2
30.3
49.9
65.0
66.3
65.5
65.9
MiSLAS Zhong et (2021
62.9
50.7
34.3
52.7
73.2
72.4
70.4
71.6
BALMS Ren et al. (2020)
64.1
48.2
33.4
52.3
70.6
LADE Huang ct al. (2016)
ResNet-50
64.4
47.7
34.3
52.3
69.3
ACE a1 et al. 2022
56.6
72.9
RIDE Wang et al. (2020)
68.2
53.8
36.0
56.9
70.9
72.4
73.1
72.6
PaCo Cui et al.
2021
68.2
58.7
41.0
60.0
70.3
73.2
73.6
73.2
NCL
L1 ct al.
2022b)
57.4
72.0
74.9
74.2 73.8
VIT Dosovitskiy et al.
20021
50.5
23.5
6.9
31.6
65.4
55.3
50.9 54.6
MAE He et (022)
74.7
48.2
19.4
54.5
79.6
70.8
65.0 69.4
DeiT
Touvron et al. (2022)
70.4
40.9
12.8
48.4
72.9
62.8
55.8 61.0
LiVI XVTXuex et 2023
VIT-B/16
73.6
56.4
41.0
60.9
78.9
76.5
74.8 76.1
LiVT Xu al. 2023
76.4
59.7
42.7
63.8
83.2
81.5
79.7 81.0
Ours
76.4
66.2
48.9
67.6
82.5
79.8
77.1 78.9
Ours
77.2
68.6
51.3
69.4
83.3
82.1
80.5
81.5
Only Fine-tuning
CLIP+ZS Radford
1021
82.0
70.4
69.6
70.5
9.9
5.3
4.6
5.5
CLIP+LP Radford et al.
2021
87.3
65.1
19.0
67.4
62.4
7.1
0.1
10.0
CLIP+FT Radford
2021
83.0
65.0
39.9
68.5
79.4
67.6
59.1
65.4
DPT et al.
20233
62.1
76.2
79.3
76.1
VIT-B/16 (w/ Pretrained
BALLAD Ma et al. (202
79.1
74.5
69.8
75.7
Decoder Wang al. 2024
73.2
59.2
Clip)
VL-LTR Tian et al. 2022
84.5
74.6
59.3
77.2
81.0
LIFT Shi et al. 2024
80.2
76.1
71.5
77.0
72.4
79.0
81.1
79.1
Ours
79.5
76.5
74.1
77.2
83.5
82.2
80.7
81.7
Ours f
79.9
76.8
74.5
77.6
84.
82.7
81.0
82.1
Our method significantly outperforms CLIP in dealing with tailed drift, especially on iNaturalist2018 also indicates that training on a high-quality balanced dataset alone cannot effectively mitigate the bias induced by taig drift. Furthermore, the results of linear probing and fine-tuning demonstrate that imbalanced datasets can induce pronounced tailed drift MLLMs, ultimately degrading model performance. The CLIP accuracy in Few-shot is only 39.9% under fine-tuning ImageNet-LT: much lower than the 59.6% under zero-shot. It further verifies the challenges brought by imbalanced data in the training of MLLMS and the superiority of our method in adapting the MLLM concept drift from pre-training onwards.
In terms of OOD drift detection, our proposed vision language model, trained CIFARI 10-LTT in an-distribution dataset, demonstrates exceptional performance across four diverse OOD datasets, as shown in Table 2 Our approach stands with two significant advancements. Firstly, the training of our model does not incorporate any additional data from the open world to delineate the decision boundary between samples and OOD samples. Secondly, our proposed model detects OOD drift based on the hyperspherical distribution, without the need for any specialized modules. The proposed methodology offers convenience training and inference for large models.
8

---

Published as a conference paper at ICLR 2025
Table Evaluation results of OOD detection with the OOD datasets SVHN, LSUN, iSUN and Texture. ResNet-34 is selected as the image encoder. The best- pest-performing method is highlighted in red. FPR, and AUROC1 are applied evaluate the performance of different methods.
SVHN
LSUN
iSUN
Texture
Methods
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
ProxyAnchor Kim et al. (2020)
87.2
82.4
37.2
91.7
70.0
85.0
65.6
85.0
CE + SimCLK Winkens et al.
24.8
94.5
56.4
89.0
66.5
83.8
63.7
82.0
CSI
Tack et al. (0200)
44.5
92.7
75.6
83.8
76.6
85.0
61.6
86.5
SSD+ Sehwag al. (2021)
31.2
94.2
79.4
85.2
80.9
84.1
66.6
86.2
KNN+ Sun et al. 2022)
39.2
92.8
49.0
89.3
75.0
82.7
57.2
88.4
CIDER Ming
2023
12.6
97.8
30.2
92.8
46.0
88.9
35.6
92.3
Ours
8.3
98.7
20.3
97.5
32.5
95.2
45.1
96.3
Moreover, we evaluate the gen- eralizability of our method in the domain generalization set- ting. Experiments are conducted on ImageNet-Sketch Wang al. (2019) with ImageNet Rus- sakovsky et al. (2015a) as the source dataset, shown in the Table From the experiment results, method achieves perior performance with an accu- racy of 50.2% on ImageNet-Sketch, attributed the robustness of the T-distribution-baser drift adapter. It further verifies the generalization ability of our model in the open world.
Table 3: Evaluation results of generalization on ImageNet-Skeut
Wang et al. (2019) with ImageNet Russakovsky et al. (2015a)
as the source dataset. We compare our methods with other VL
models, including zero-shot CLIP Radford et al. (202T) linear probing CLIP Radford et al. (2021). CoOp /houu et al. 20022)
ViT al. (2022) and DAPT al. (2023).
CLIP+ZS
CLIP+LP
CoOp
VPT
DAPT
Ours 50.2
46.1
36.0
47.1
47.7
48.3
3.2 CONCEPT DRIFT-AWARE IMAGE-TEXT ALIGNMENT FOR EFFECTIVE PRE-TRAINING
Moreover, we verified at the feature level that the proposed T-distributed adapter significantly alleviates the bias from tailed drift and OOD drift the pre-training. As exhibited in Table the degree of ID intra-class compactness re- duces from 49.2 in L.Tcossine to 36.2 in LT/Thp. thereby validates the effec- tiveness of the proposed T-distributed adapter in enhancing feature extraction in oong-tilled scenarios. Notably, the decrease in standard deviation demon- strates that the model considerably mit- igates the bias induced by tail drift. In addition, our method achieves remark- able inter-class separability within in- distribution categories under long-tailed scenarios, even surpassing the perfor- mance of cosine achieved under the bal- anced dataset. It confirms the effective- ness of the proposed high kurtosis method in enhancing the alignment between images and text the pre-train pre-training stage. Concerning the separability between ID and OOD, achieve superior results even than the balanced condition, attributed to the inherent light-tailed property of the T-distributed adapter. It ensures our approach performs robustly for OOD drift detection.
Table 4: Evaluation results of image-text alignment of different contrastive learning strategies the stage of pre-training, from three perspectives: ID intra-class pactness, ID inter-class separability and the separability between ID and OOD categories. The cosine metric is utilized to measure these distances, which is expressed average degrees with standard deviation in brackets. We compare our proposed Thp with classical cosine loss, under balanced scenario (BL, ImageNet) and imbalanced scenarios (LT, ImageNet-L.T), respectively.
ID Intra-class ID Inter-class ID vs. OOD Separability f
Pre-training Compactness
Separability
BL Cosine
33.0 (=2.85)
84.3 (#1.64
98.4 (#:0.98)
LT / Cosine
49.2 (=4.95)
76.5 (#:2116
80.7 (#1.9))
LT / Thp
36.2 (+3.53)
85.6 (++1.88)
101.3 (+11.25

---

Published as a conference paper at ICLR 2025
3.3 ABLATION EXPERIMENTS
3.3.1 T-DISTRIBUTED SPHERICAL EMBEDDING IN THE PRE-TRAINING AND FINE-TUNING
Firstly, we conduct ablation experiments to verify the improved performance of the T-distributed adapter in the stage of pre-training and fine-tuning, respectively, As demonstrated in Table 5 our proposed T-distributed adapter exhibits improvements in both the pre-training and fine-tuning stages of the vision language model. It worth highlighting that the T-distribution adapter plays a more prominent role during the fine-tuning stage. We argue that it is due to different characteristics of the pre- training and fine-tuning. During fine-tuning, the model directly involved in specific downstream tasks, and plicit category centers are present in the classifier. In con- trast, pre- tre-training primarily focuses on aligning image- features, where the implicit information of categories is embedded. As result, the T-distribution adapter's 's impact is more pronounced in the fine-tuning stage compared to pre-training.
Table 5: Ablation evaluation results with or without the T-distributed adapter in the pre-training or fine-tuning. The v denotes the stage is trained with the T- distributed adapter. The results are based on the ImageNet-LT with the Vit-base. Top-I accuracy (Acc) used as the met- ric.
Pre-training
Fine-tuning
Acc
56.0
58.7
v
65.1
v
69.4
3.3.2 VARIOUS CONCENTRATION
T-ADAPTER
Furthermore, we conduct ablation experiments to examine the impact of concentrations of the T- distributed adapter on the overall performance of the VL models in Table 6 Four fixed degrees are in- volved, namely x 4. 16, 64 and 128. The greater the degree of concentration, the greater the kurtosis of the Thp metric. Besides, the concentration can also be utilized as trainable parameter joining in the training, with the initial setting of 16. Table 6 setting the concentration parameter x 16 yields superior results. We argue that a smaller concentra- tion makes challenging effectively mitigate the biases introduced by tail drift and OOD drift in the vision language model. In terms of the bigger con- centration, the model is hard to train due to the high kurtosis of the Thp metric. the context of concentration as a trainable parameter with the initializa- tion of 16, there is slight reduction in model performance, accompanied by marginal increase of the concentration parameter to 16.37. We assert that the introduction of the new parameter increases the model's complexity, thereby making the training process more challenging.
Table Ablation evaluation results of various concentrations of parameter on the long- tailed classification task. "Training" denotes the involved the training as a parameter of the model with an initial setting of 16. The results are based on the ImageNet-LT with the Vit- Top-1 accuracy (Acc) is used as the metric.
16
64
128
Training
Acc
67.2
69.4
64.9
61.1
68.3
4 CONCLUSIONS AND OUTLOOK
Our findings indicate that visual- languae models significantly affected by biases introduced during both pre- vre-trannin and fine-tuning in long-tailed open- spennworld scenarios. To address this, we propose concept drift-aware unified framework for visual-language models. This framework incorporates a T-distributed adapter designed to mitigate biases arising from both tailed drift and out-of-distribution (OOD) drift. Additionally, we introduce comprehensive set of multi-modal datasets (OpenMMMo) tailored to the long-tailed open world, which includes images, captions and related category annotations.
Finally, we hope that our work will inspire future advancements in multi mudti-modal large language models, specifically addressing the mitigation of biases originating from real-world data challenges. such as tailed drift and OOD drift.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The work was supported by the Australian Research Council (ARC) under Laureate project FL190100149.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716
23736, 2022.
Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the Unit Hypersphere using von Mises-Fisher Distributions. Journal of Machine Learning Research, 6(9): 1382, 2005.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-taile tailed recognition in one-shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 112- 2021.
Jiarui Cai, Yizhou Wang. Hung-Min Hsu, Jenq-Neng Hwang, Kelsey Magrane, and Craig S. Rose. LUNA: Localizing Unfamiliarity Near Acquaintance for Open-Set Long-Taile Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):131-139) 2022. ISSN
2374-3468. doi: 10.1609/aaii vii.ia.iii.ii..11888
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.091 2023.
Eulrang Cho, Jooyeon Kim, and Hyunwoo J. Kim. Distribution-Awar Prompt Tuning for Vision- Language Models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22004-22013, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint vrXiv:2204.023111 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc Le, and Jason Wei. Scaling Instruction-Finetuned Language Models. URL http:://arils / abs 221.2220..1111 11416
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing Textures in the Wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric Contrastive Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 715-724, 2021.
Wenliang Dai, Junnan Li, Dongxu L., Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N. Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Advances in Neural Information Processing Systems, 36:49250- 49267, 2023,
Nicola De Cao and Wilker Aziz. The power spherical distribution. arXiv preprint arXiv:2006.04437, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019b. doi: 1/865//11/11914222
Bowen DONG, Pan ZHOU, Shuicheng YAN, and Wangmeng ZUO, LPT: Long-tailed prompt tuning for image classification. pp. 1-20, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021. URL http: http//axiv.coo/ org abs 201 0 11929
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: Graph Out-of-Distributin Bench- mark. Advances in Neural Information Processing Systems, 55:2059-20777 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning Deep Representation for Imbalanced Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5375-5384, 2016.
Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, pp. 709-727. Springer Nature Switzerland, 2022. ISBN 978-3-031-19827-4, doi: 10. 10.10077997--3-011-199777 41.
Botao Jiao, Yinan Guo, Shengxiang Yang, Jiayang Pu, and Dunwei Gong. Reduced-space multi- stream classification based on multi-objective evolutionary optimization. IEEE Transactions on Evolutionary Computation, 2022.
Botao Jiao, Yinan Guo, Dunwei Gong, and Qiuju Chen. Dynamic ensemble selection for imbalanced data streams with concept drift. IEEE Transactions on Neural Networks and Learning Systems, 35 (1):1278-1291, 2024.
doi: 10. 1 10911ONNS 2022.318322222
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large Language Mod- els Struggle to Learn Long-Taie Tail Knowledge. In Proceedings of the 40th International Conference on Machine Learning, pp. 15696-1555 15707. PMLR, 2023.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long-Taile Tailed Recognition. In Eighth International Conference on Learning Representations (ICLR), 2019.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy Anchor Loss for Deep Met- ric Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3238 3244, 2020.
Takumi Kobayashi. T-vMF Similarity For Regularizing Intra-Class Feature Distribution. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6612-6621. IEEE. ISBN 978-1-66544-509-2 doi: 10.1 D0.1199/CPP44437 200.R46637.222.1.. 2021.00655.
Lukasz Korycki and Bartosz Krawezyk. Concept Drift Detection from Multi-Class Imbalanced Data Streams. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 1068-1079. doi: 10.1 109//DCE
1ECDE 299.9992222... 2021.00097.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
URL
http:// cs utoronto. -kriz/learning Seearninnn-ffttureeeeee features 2009-0000000 TR
fdf
12

---

Published as a conference paper at ICLR 2025
Bolian Li, Zongbo Han, Haining Li, Huazhu Fu, and Changqing Zhang. Trustworthy Long- TaileTailed Classification. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 6970-6977 2022a.
Jun Li, Zichang Tan. Jun Wan, Zhen Lei, and Guodong Guo. Nested Collaborative Learning for Long- Tailed Visual Recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6949-6958, 2022b.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation.
2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Iman Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning, pp. 12980. PMLR, 2022c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang. Lu Yuan, Lei Zhang, Jenq-Neneee -Nag Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. 2022d.
Wendi Li, Xiao Yang, Weiqing Liu, Yingce Xia, and Jiang Bian. DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation. 36(4):4092-4100, ISSN 2374-3468. doi:
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Wei Liu, Xiaodong Yue, Yufci Chen, and Thierry Denoeux. Trusted multi-view deep learning with opinion aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7585-7595 2022a.
Wei Liu, Yufci Chen, Xiaodong Yue, Changqing Zhang, and Shaorong Xie. Safe multi-view deep classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8870-8878, 2023b.
Wei Liu, Yufci Chen, and Xiaodong Yue. Building trust in decision with conformalized multi-view deep classification. Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7278 7778-7287,1 2024.
Weike Liu, Hang Zhang, Zhaoyun Ding, Qingbao Liu, and Cheng Zhu. A comprehensive active learning method for multiclass imbalanced data streams with concept drift. Knowledge-Based Systems, 215:106778, 2021. ISSN 0950-7051. doi: 10.10166).mooyy.2221...o
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-Scale Long- Tailed Recognition in an Open World. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. doi: 10.1109/CVPR.22 2019.001.0024
IEEE, 2019. ISBN 978-1-72813-293-8.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Open Long- Tailed Recognition A Dynamic World. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-15, 2022b. ISSN 1939-3539. doi: 10.1 00.1000/PAMI.222223200999
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. Learning under Concept Drift: A Review. IEEE Transactions Knowledge and Data Engineering, 31(12)22346-23 2363, 2019. ISSN 1558-2191. doi: 10. 109/TKDE 288768877
Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Simple Long- Tailed Recognition Baseline via Vision- LanguuLe Model, 2021. URL
http:://are http://wwxvv.oorg///////11
13

---

Association for Published as a conference paper at ICLR 2025
K. V. Mardia and Peter E. Jupp. Directional Statistics. Wiley Series in Probability and Statistics. J. Wiley, 2000. ISBN 978-0-471-95333-3
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into Out-of- Distribution Detection with Language Representations. Advances in Neural Information Processing Systems, 35:35087-35102, 2022.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspheriia embeddings for out-of-distribution detection?, 2023. Utthtpp://araet www.wpwwwwwww.../p/p//aapppaap 2203 04450
embedddnn
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. volume 2011, pp. 7. Granada, Spain, 2011.
OpenAL. Gpt-4v(ision) system card, 2023. GPTV SS__SSysemmmmmmmm Crrsse
URL
ope.ai.nnooo.......oooomoaa.aa.. com cowppapers////cccccooccc..
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. On Long-Tailed Phenomena in Neural Machine Translation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
Computational Linguistics. doi: wt..1663///2220.fimm
Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced Meta-Softmax for Long- -ailed Visual Recognition. In Advances in Neural Information Processing Systems, volume 33, pp.
Curran Associates, Inc., 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. 1 (5(3):22112522) 2015a.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211 13):211-252, 2015b. doi: 10.1007/s11266 0153--16--816.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
Vikash Sehwag, Mung Chiang, and Prateck Mittal. SSD: A Unified Framework for Self-Supervised Outlier Detection. In International Conference on Learning Representations, 2021. URL https
//openreview.oo net / wwrummmiiii wid-uauiiiaiiiii [Xp
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yar Han, and Yu-Feng Li. Long- Learning with Foundation Model: Heavy Fine- Fine-Tuning Hurts. In forty-First International Conference Machine Learning, 2024. URL
net forum? d=ccSSKT7911X
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Nearest Neighbors. In Proceedings of the 39th International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Detection with Deep
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances. In Advances in Neural Information Processing Systems, volume 33, pp. 1183--11852 Curran Associates, Inc., 2020.
Jalil Taghia, Zhanyu Ma, and Ame Leijon. Bayesian Estimation of the von-Mises Fisher Mixture Model with Variational Inference. 36(9):1701-175.. ISSN 1939-3539. doi: 10.1109/TPAM..201. 2306426.
14

---

Published as a conference paper at ICLR 2025
Hui Tang, Xiatian Zhu, Ke Chen, Kui Jia, and C. L. Philip Chen. Towards Uncovering the Intrin- sic Data Structures for Unsupervised Domain Adaptation Using Structurally Regularized Deep Clustering. 44(10):6517-6533 ISSN 1939-3539. doi: 10.1 10.1099TPAM112201:009955
Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao. VL-LTR:
Learning Class-wise Visual-Linguistie Representation for Lang-ailedd Visual Recognition. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 73-91. Springer Nature Switzerland, 2022. ISBN 978-3-031-19806-1 doi: 100007/978-3031-11 19806-99
Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the VIT. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, pp. 516-533. Springer Nature Switzerland, 2022. ISBN doi: 00000779778-3001-20055555
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The INaturalist Species Classification and Detection Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8769-8778, 2018.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representa- tions by Penalizing Local Predictive Power. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http/tsts///rrrreeddiiss
cc/
Partial and Asymmetric Contrastive Learning for htt.://eeeaaaaaa..aaaattt....e.e Abst:a55t-btttatt..ttt. Recognition. In Proceedings of the 39th International Conference on Machine Learning, pp.
2appr//013/haah//eeeeeeee wtt.eeeeeeecbeeeeeeeeeeeeeeeeee.
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J. Smola, and Zhangyang Wang. Detection in Loil-Tailed
23458. PMLR, 2022.
Min Wang, Lei Zhou, Qian Li, and An-an Zhang. Open world long-tailed data classification through active distribution optimization. Expert Systems with Applications, 0957-4174. doi: (0.1016fj.esa..222... 199054.
2023. ISSN
Xudong Wang. Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed Recognition by Rout- ing Diverse Distribution Aware Experts. In International Conference on Learning Representations, 2020. URL
net/ tene/fformmii forum? waeruummiiDDD11. httt:/oorummm..-.133rrrr--- 3dr.---9933d22BB
Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. Exploring Vision-Language Models for Imbalanced Learning. 132(1):224-237, 2024. ISSN 1573-1405. doi: 10. 1000/0//111 121-100-11---0---8---------
Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-Sampling: Exploring Out-of- Distribution data for Re-balancing Long-tailed datasets. In Proceedings of the 39th International Conference on Machine Learning, pp. 23615-23638 PMLR, 2022.
Tong Wei, Bo-Lin Wang, and Min-Ling Zhang. EAT: Towards Long-Tailes
Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 38(14)115777-15999 2024. ISSN 2374-3468. doi: 10. wtt.aa.aiii.i.........
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive Training for Improved Diitrri-iitiitoi Detection. http: http://wxxvvvoooraaa////s/o
org /200 05566
15

---

Published as a conference paper at ICLR 2025
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492-1500, 2017.
Pingmei Xu, Krista A. Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking. URL http:
7 //rx/vv.rgg/bb//500...
Zhengzhuo Xu, Ruikang Liu, Shuo Yang. Zenghao Chai, and Chun Yuan. Learning Imbalanced Data With Vision Transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15793-15803, 2023.
Xiaoyu Yang. Yufei Chen, Xiaodong Yue, Shaoxun Xu, and Chao Ma. T-distributed Spherical Feature Representation for Imbalanced Classification. Proceedings of the AAAI Conference Artificial Intelligence, (7(0))00855-0088
2023. ISSN 2374-3468. doi: 10.1609/aaai, v3729.26284
Xiaoyu Yang, Lijian Xu, Hao Sun, Hongsheng Li, and Shaoting Zhang. Enhancing visual grounding and generalization: A multi-task cycle training approach for vision-language models. arXiv preprint arXiv:2311.
2024.
22111122171
Xiaoyu Yang, Jie Lu, and En Yu. Causal-informed contrastive learning: Towards bias-resilient pre-training under concept drift, arXiv preprint arXiv:2502.07620, 2025a. URL https://
arxiv. org/ abs 2250207622
Xiaoyu Yang, Lijian Xu, Hongsheng Li, and Shaoting Zhang. One leaf reveals the season: Occlusion- based contrastive learning with semantic-aware views for efficient visual representation. arXiv preprint arXiv:2411.088 100958,, 2025b. (RRpattss//arxii org. abs /411. 09858
En Yu, Jie Lu, Bin Zhang, and Guangquan Zhang. Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16522-16530, 2024.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of Large-scale Image Dataset using Deep Learning with Humans in the Loop. URL htttp://arxlv.orr//bbb//// wwwp:/ogg//bbbsbeeeeeoo
03365
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Xuefei Zhe, Shifeng Chen, and Hong Yan. Directional statistics- batistics-based deep metric learning for image classification and retrieval. 93:1 13-133.. ISSN 0031-3203. doi: ht.10166j.pptcooe pattcoog22199000 20.99.04.055.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 452-1444, 2017.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to Prompt for Vision- Language Models. 13399222377-3348, 2022. ISSN 1573-1405. doi: 10.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
RELATED WORKS
A.1.1 MULTI-MODAL LARGE LANGUAGE MODEL
Large Language Models (LLMs) have recently significantly impacted the field of natural language processing. Through alignment techniques such as supervised learning and reinforcement learning with human feedback, LLMs can effectively generalize to perform a wide range of tasks, even with limited training data. A remarkable application of LLM is ChatGPT, which presents an amazing ability to interact with humans. OpenAl's ChatGPT and GPT4 are prime examples of the impact that Al can have, and there have been extensive open-source efforts to replicate their success, such as OPT
Zhang ct al. (2022). BLOOM Scao et al. 2022). PALM (howwdeerrr et al. (0022. LLaMA Touvron al. (2023).
Multi-modal large language models have further promoted the development of the vision-language models Radford et al. (202I); Li et al. (2022d); Alayrac et al. (2022); Li et al. 2023);; Zhu al. (2023): Liu et al. 20 Chen et al. 2023);] Yang et al. (2024) 2025b). CLIP|Radord et al. (2021) was introduced to separately extract features from the visual encoder and the text encoder, and combine them using contrastive learning. CLIP supports a variety of downstream tasks, including image retrieval, image classification tasks and especially zero-shot classification tasks. But, it cannot generate detailed captions based on images due to the lack of a text decoder. In contrast, our model primarily addresses the concept drift issue within multi-modal large language models, since an grouudee text decoder is employed to generate text based on the images. Besides, CLIP requires large- scale and high-quality WIT dataset to be driven, that contains 37.6 million entity image-text samples with 11.5 million unique images across 108 Wikipedia languages. Whereas, our method is validated under the extended ImageNet-LT, which consists of only 115.8K imbalanced images-text pairs.
Building on CLIP, GLIP et al. (2022d) was developed to learn object-level, language-awarc and semantic-rich visual representations, unifying object detection and phrase grounding for pre-training, Different from the contrastive method, Flamingo Allgora et al. (2022) aligned a pre-trained vision encoder and language model using gated cross-attention demonstrating impressive few-shot learning capabilities. BLIP2 Li al. (2023) was subsequently introduced, and it employed a Flan-T5 Flan-TS/Chuug let al. along with Q-Former to effectively align visual features with the language model. MiniGPT4
Zhu et al." (2023), the most recent development in the field is the PaLM-E model, which features 562 billion parameters and is designed to integrate real-world continuous sensor modalities into an LLM. thereby establishing a connection between real-world perceptions and human languages. Based on Visual Fundamental Models like BLIP mentioned above, Visual ChatGPT adopts ChatGPT as the central component for interacting with users. It integrates multiple visual foundation models and utilizes prompt engineering, also known as Prompt Manager, to instruct CharGPT about the usage, input-output format, and capabilities of each foundation model. This enables ChatGPT to determine how to invoke these models to fulfill the user's requirements. Besides, GPT-4V(ision) OpenAl (2023) and GPT-40(mni) have recently shown unprecedented ability in understanding and processing an arbitrary mix of input images and texts.
A.1.2 LONG-TAILED OPEN WORLD
In vision tasks, significant efforts have been devoted to mitigating the challenges posed by the long-tailed open world. Two prominent research directions have emerged: long-tailed classification under open-world settings, exemplified by approaches like OLTR+ Liu al. (2019 2022b), LUNA Cai et al. (202)), DALC
| [aang et al. (2023), Open-sampling Wei et al. (2022) and TLC et al. (2022a), and OOD detection in long-tailed recognition, as seen in methods such as PASCL Wang et al. (2022). BATTee et al. (2044). +LTR++ Liu et al. (2019) 2022b) proposed an ensemble algorithm, consisting of dynamic meta-embedding to improve the recognition of tail categories and active learning for open categories detection. LUNA Cai et al. (2022) presented distributionn-ensitive loss to weigh more on the tail classes and a local-density- based metric to measure the novelty of OOD samples. DALC Wang et al. (2023) designed an active distribution optimization algorithm for clustering, querying and classification to balance the classification bias. seee-sam Wei et al. (2022) rebalances class priors by sampling labels from a complementary distribution for each open-set
17

---

Published as a conference paper at ICLR 2025
(202a)) utilizes the Dempster- Evidence Theory in a multi-expert framework for uncertainty estimation of tail and OOD samples. PASCL Wang al. 2022) applied supervised contrastive learning to explicitly boost the model to distinguish between fail-class in-distribution samples and OOD samples. EAT Wei et al. (2024) introduces abstention classes for clear decision boundaries and augmenting tail classes with context-rich OOD data to focus on discriminative features. MCM[Ming et al. (2022) pioneers the integration of vision language models into OOD detection, enabling zero-shot OOD by aligning visual features with text concepts through a proposed maximum concept matching approach.
instance, mitigating class imbalance. et
In addition, more and more VL methods have gained attention in the long tail domain, such as LPT DONG et al. (2023), BALLAD Ma et al. (202I). Decoder Wang et al. (2024), VL-LTR Tian et al. (2022) and LIFT Shi et al. (2024). However, most of them pay attention to the fine-tuning of the vision language model under long-tailed scenarios. They directly use the pre-trained CLIP model, which is pre-trained using the high-quality and large-scale WIT dataset. In contrast, we are more concerned about the impact of long-tail open data on the whole model training from pre-training onwards, including pre-training and fine-tuning.
Additionally, in the domain of the language model, Kandpal et al. (0233) corroborates that large language models (LLMs) also struggle to learn long-tailed knowledge. While larger models are better at absorbing long-tailed knowledge, they estimate that current models must be scaled by many orders of magnitude to reach competitive performance. Besides, [aanna et al.) alleviates the long-tail problem in neural machine translation by quantifying token classification and sequence generation, and introduces an anti-focus loss that incorporates beam search inductive biases to better adapt model training conditional text generation.
A.1.3 CONCEPT DRIFT
In the review Lu et al. (2019), the algorithms related to concept drift are categorized into three groups: error rate-based, data distribution-basee and multiple hypothesis-based. Our proposed algorithm belongs to the distribution-base concept drift algorithms not only accurately detect drift through explicit distributions but also analyze the drift to identify its happening timing, location, and severity.
concept drift detection and adaptation method. Distribution-bar based
Besides, Korriik/oryykki & Krawezyk proposes a novel trainable concept drift detector based on Restricted Boltzmann Machine, to solve the concept drift in multi-class imbalanced data streams. Meanwhile, DDG-DA [i al.) initially trains a predictor to estimate future data distribution with concept drift, utilizes this information to create training samples, and subsequently trains models on the generated data. Furthermore, CALMID Liu et al. (202T) proposes comprehensive active learning method for multiclass imbalanced streaming data with concept drift, including an ensemble classifier, a drift detector, and a variable threshold uncertainty strategy. Subsequently, DES-ICD [iao et al. 2024h is dynamic ensemble selection method for imbalanced data streams with concept drift. It considers the local performances of base classifiers and addresses class imbalance using a novel synthetic minority oversampling technique. Moreover, GOOD Gui al. (2022) develops graph OOD benchmark, which explicitly distinguishes between covariate and concept shifts and designs data splits that accurately capture these different shifts. Beyond that, ResilientCL Yang et al. (2025a) introduces a causal framework that integrates concept drift adaptation with structural causal modeling. By decoupling spurious correlations via causal graphs and enforcing counterfactual invariance, addresses distributional biases in streaming training data. Besides, Liu et al. (2022a 2023b) 2024) propose a multi-view -iew uncertainty framework that addresses concept drift across heterogeneous data streams through set-valued prediction generation, effectively consolidating probabilistic outputs into deterministic categorical representations.
Remark A.I. Differences: Concept Drift vs. Data Drift (Covariate Drift) Data drift entails changes solely in the distribution of inputs P(z), while concept drift involves alterations in both input and output distributions, i.e., P(z) and P(y). leading changes in the decision boundary. Furthermore, data drift predominantly stems from internal factors like data collection and processing. whereas concept drift typically arises from external factors, reflecting real- world changes.
18

---

Published as a conference paper at ICLR 2025
1.1.4 HYPERSPHERICAL DISTRIBUTION MODELLING
The Bayesian estimation of the vMF mixture model with variational inference is addressed in Taghia et al. The learning task in VI consists of the optimization of the variational posterior distribution. Besides, deep metric learning model for image classification and retrieval is presented in Zhe et al.) which utilizes the vMF distribution to define the loss function and introduces effective alternative learning algorithm by updating class centers. The model captures global information in the embedding space and approximates the class distribution during training, leading to improved performance in image tasks. Kobayashi extends the vMF distribution to regularize the intra-class feature distribution for imbalanced, small-scale and noisy data. Yang al. (2023) focus using hyperspherical embedding to alleviate the crowding problem arisen by the imbalanced data. [ing et al. (2023) utilizes hyperspherical embeddings for OOD detection in representation learning. consisting of two losses, a dispersion loss to increase angular distances between different class prototypes, and a compactness loss ensure samples closer their respective class prototypes. Besides, H-SRDC Tang et al. enhances intra-class compactness by combining target data clustering with a domain domain-shared classifier and cluster centroid learning, enhancing deep clustering by minimizing Ballback-Leible divergence between network predictions and an auxiliary distribution.
A.2 THE T-DISTRIBUTED DISTRIBUTION ON HYPERSPHERE
A.2.1 DIRECTIONAL STATISTICS
Directional statistics primarily focus on the distribution of eigenvector angles, while neglecting the impact of eigenvector module lengths. Given the unit feature vector X,j E $é-1, where gd-1 { E Rd ||æ:|| = 1} denotes the (d - 1)-dimensional hyperspherical set. A key idea in directional distribution is the tangent-normal decomposition. Any unit vector can be decomposed as:
1u (1 13) L [-1,
with v Sd-2 a tangent to gd-1 at μ Mardia & Jupp (000); De Cao & Aziz (2020), where v and I are independent and v is uniform on 30-2 Thus, the intersection of S with the hyperplane through i/s and normal to j is a (d 2)ddimensia sphere of radius v1 12, that f has density as following:
pr(t: d) x (1 ,t
(10)
Therefore, through the marginal density pr and pe. we can estimate the density of the entire spherical distribution. One prominent instance the von Mises-Fisher distribution (vMF) Banerjee et al.) (2005), which can be interpreted as probability distribution over the cosine similarity between unit vector æ and a fixed mean direction μu, following the density:
Px (#: M4, R) xx exp (sp² x),
(11)
where x > 0 denotes the concentration and exp represents the exponential function. Therefore, combined with the Eq. 9 and Eq. [0] the density of vMF is:
p(x) = Cx(1s, d) exp (nqi" x)) r~ vMF(p,
(x(Kdd) d) (2T) (2T) 1/2////
(12)
k4/2-1
where I, denotes the modified Bessel function of the first kind at order m.
A.2.2 DERIVATION OF THE T-DISTRIBUTED DISTRIBUTION ON HYPERSPHERE
Given the unit feature vector X,y E gd-1, where gd-1 fz e Rd ||ا|| 1} denotes the (d - di-dieensional hyperspherical set. The proposed T-distribution metric on hypersphere (Thp) follows the density:
Px α s(1 N x)
(13)
where E Sd-1, direction - and concentration R2o. Let T bet random variable that denotes the dot-product 1¹ xr then 7 27 1, with Z ~ Beta(o, (3), where d=ه and -
19

---

Published as a conference paper at ICLR 2025
Proof. Given Eq. 10 the marginal distribution of the dot-product t is
2 t k(1 - 1) (1 8 "7
(14)
So, its normalizer is:
(r d)
2
(1 (- 2-3 dt
J84-1 1i(1 - 1)
k<1 - (1 + d-3
(1 - 1) ق-3
dc
(15)
/ (1 + - (1 1) - dt.
Given the useful integral function:
f (1+ (1+ 2) = 2 1 B x (a + +, + 1) +
(16)
So, its normalizer is:
d-1 d 3-3
d-1 d 1-3 ))
(x) 3 (B1
Bo
16
2
2
(17)
d
d-1
1-1d-3
d-3
2 B
K
2
The Beta function:
B(a,b) {(a)T(b)
(18)
I(a+b)
So, the normalizer is
Nr d) = 1 -
(19)
K
3))
where, a d-1 and B d-3 It follows that the probability density function of the marginal 2
2
distribution of the dot product is.
pr(ti d) Nr(K, d)
2
(1- -
x(1 - 1)
NN((K
(1 +1 133
(1 - t)² 111
(20)
= =Nr(Ni (2z) 2 -1-1(2 22) -
2 B(a, 12, (1 8-1
k
d-1 where, 2 and B = d-3 2
Due to the surface area of the hyper-sphere gd-1 is:
2x
Ad-1
(21)
[(
The T-distributed spherical distribution is expressed via the tangent normal decomposition as a joint distribution between T ~ prt;K, K, d and V ~ Since T 1L V. the Thp normalizer (2 k) is the product of the normalizer of pr (t; k, d) and the uniform distribution on dd-2 is:
(x d) = Nr d) Ad-2
2n
2 = .)
kI
(22)
20+B [(a)
I(a + 9)
Thus,
px(=; = (x(x,
(23)
x(1 n1² x)
20

---

Published as a conference paper at ICLR 2025
A.3 IMPLEMENTATION DETAILS
For our language-guided image tokenizer, we leverage the strengths of both BERT Devlin et al. (20T96) and ViT as our text encoder, text decoder and visual encoder, respectively.
We employ VīT-Bae as our visual encoder, which consists of 12 transformer encoder layers and an FFN intermediate size of 3,072. The imput image size is set to 384 x 384, with a patch size of 16 x 16. The hidden dimensions of the ViT-Base are 768, with 12 attention heads. And, the number of parameters about 86 million. Besides, we also use ResNe to perform ablation experiments. In addition, ResNeXt-50 has 16 residual blocks with 50 layers. Each block has 3 convolutional layers with the kernel size of 3, the stride of and the padding of 1. The batch normalization and max pooling are utilized to connect the convolutional layers. The classification head hidden dimensions are 2,048.
Additionally, BERT as the language model in our vision-language model, has 12 transformer layers with 768 hidden dimensions and 3,078 intermediate dimensions. The number of attention heads is 12, with the input sequence length of 512. It has approximately 1 110 million parameters.
In terms of the pre-training progress, the hyperparameters are presented Table 7 We utilize the AdamW optimizer, which is configured with a cosine annealing schedule as the learning policy. The initial learning rate set to 2 x 10 and the Adam optimizer is employed with hyperparameters set the weight decay to 0.05 and the dropout rate to 0.1. During
(0.9, 0.98). Additionally, the first 1,000 warm-up steps, the learning rate increases to x 10-5, and subsequently decays to 10-7. Unless otherwise specified, the pre-training of our vision language model consists of 800,000 steps, executed on 2 x 2 NVIDIA A100 GPUs. And the pre-training experiments are conducted in the manner of different stages, namely gradual drifts with long-tailed data and sudden drifts with OOD data. It is mainly to compare with different methods with the same setup.
Table 7: The training hyperparameters of our vision language model.
Pre-training
Fine-tuning
Training Steps
400,000
Training Steps
18,000
Warmup Steps
1,000
Warmup Steps
0
Optimizer
Adam W le-4
Optimizer
AdamW 2e-5
Learning Rate
Learning Rate
Learning Rate Decay
Cosine (0.9, 0.98) 0.05
Learning Rate Decay
Cosine (0.9, 0.98)
Adam &
Adam 3
Weight Decay
Weight Decay
0.05
Batch Size
50
Batch Size
400
While in the fine-tuning the downstream task of classification, the initial learning rate is reduced to 10-6 without the warmup. The visual encoder and text decoder are frozen out of the training. Thus, the batch size can be increased 400. The fine-tuning consists of 18,000 steps, executed on 2 NVIDIA A100 GPUs. Other training parameters are the same as the pre-training. Besides, under the only fine-tuning settings, the image encoder and the text encoder are frozen with the CLIP pre-trained parameters, while the image-grove grounded text decoder is trained during the fine-tuning.
When evaluating the performance of our VL model under the long-tailes open world, we use the top-1 accuracy metric on the downstream classification task. In particular, the categories are split into three groups: many-shot (with more than 100 training samples), medium-shot (with 20- 100 training samples), and few-shot (with fewer than 20 training samples). The Top-1 accuracies are computed for each group to evaluate the performance mitigating the bias introduced by the long-tail distribution, respectively. Furthermore, in order to assess the capability of detecting the OOD drift, we employ two metrics: FPR95 which measures the false positive rate of OOD samples when the true positive rate of ID samples reaches 95%, and AUROC providing the area under the receiver operating characteristic curve. Besides, cosine distance is exploited to measure the distances between features and centers in the feature space of the VI. model.
21

---

Published as a conference paper at ICLR 2025
The image depicts a [mask], also known as [mask], sitting on a branch of a free. The [mask] is holding leaf in its mouth, which suggests that it might be eating chew- ing on the plant. This behavior is typical of [mask]s, as they primarily feed on bamboo shoots, leaves, fruits, and insects. In the wild, [mask]s are found in the mountainous regions of southern and southwestern China, Myanmar, and India
Caption:
Annotation: lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens
(a) Sample Training Set
AGAADS
The picture depicts a young man sitting on bench, holding a [mask] in his hand. This suggests that he is either playing the [mask] or contemplating playing it. The [mask] is musical instrument that is commonly asso- ciated with blues and folk music, and can be used to create melodic and rhythmic sounds. The presence of the [mask] in the image adds a musical element to the scene.
Caption:
Annotation: harmonica, mouth organ, harp,
mouth harp
(b) Sample in Test Set
The main object in the picture is an open suitcase, which is a type of luggage. It is red in color and appears to be medium-sized. The suit- case located on the floor of a room. The suitcase is partially filled with clothing items, including shirts. pants, and socks. It appears that the suitcase still in the process of being packed or unpacked, as some items are visible on top of the suitcase while others are spilling out it. The suitcase is open, allowing easy access to the clothing items inside. Overall, the picture provides glimpse into the process of preparing for trip or organizing one's belongings.
Caption:
(c) Sample in Open Set
Figure 4: Samples of OpenMMlo in training set, test set and open set.
A.4 BUILDING MULTI-MODAL LONG-TAILED OOD DATASETS GROUP OPENMMLLO
Figure showcases the samples utilized for training and validation our study. To intuitively verify the impact of loig-tail open- world scenarios multi-modaa large language models, we employ classification as our downstream task. When matching images and texts, we strategically mask words that are directly related to category names. This approach ensures the accuracy and reliability of
22

---

Published as a conference paper at ICLR 2025
experimental results. As depicted in Figure 4 comprehensive descriptions of the image provided through long-form text, encompassing details such as size, position, color, relationships, and other relevant information about the objects present in the image. This ensures detailed and information-rich depiction of the visual content. We have publicly released the datasets used for training and validation, well the original unmasked datasets.
23


Reviews:
[{"review_id": "Af8aA123d8", "reviewer": "Reviewer_8prm", "summary": "This paper proposed a T-distribution based drift adapter for the vision language model to reduce the potential impact of concept drift in real-world data streams. Meanwhile, a multi-modal dataset OpenMMlo is built in a long tail open world setting.", "questions": "1. The current experiment, which introduces additional data, larger resolution, and additional modules, makes it impossible to effectively evaluate the effectiveness of the method, which is my main concern in giving a rating.\n2. The text decoder used in the fine-tuning stage is trained from the pretrain stage, so how could the method only fine-tuning using the pretrained CLIP model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This model has good performance on both long-tail and OOD datasets.\n2. The motivation of the paper is clear and detailed experiments and pictures are provided as support.", "weaknesses": "1. The introduction and evaluation of the image-grounded decoder and text modeling loss are lacking. The author did not introduce and evaluate this module in the methods and experiments. What is the motivation and function of introducing this module?\n2. The setup of the experiment is inappropriate. Although OpenMMlo is presented as part of the paper contribution, it is not sufficient to show only the training results of OpenMMlo, which makes comparisons with other methods unfair and meaningless. The authors are encouraged to align the training strategy with the remaining methods, and the current experiments make it impossible to evaluate both the method and the new data set.\n3. The resolution used in the paper is 336x336, and the performance of LIVT should be compared with that of the corresponding resolution.\n4. Typos in Fig. 1, 'LL' should be 'LT'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpuy4ORQfm", "reviewer": "Reviewer_JLLb", "summary": "This paper addresses the challenges faced by Multi-modal Large Language Models (MLLMs) due to concept drift in real-world streaming data, which includes gradual drift from long-tailed data and sudden drift from Out-Of-Distribution (OOD) data. While previous research has focused on these issues within the separate domains of vision or language, their effects on Vision-Language (VL) models remain underexplored. The authors propose a unified framework that adapts concept drift theory to the multi-modal context, enhancing the adaptability of VL models to unpredictable distribution changes. They introduce a T-distribution based drift adapter to mitigate biases from gradual drift and improve identification of sudden changes. Extensive experiments demonstrate that their approach enhances image-text alignment accuracy during pre-training and improves performance on various downstream tasks. Additionally, the authors create a new multi-modal dataset, OpenMMlo, designed for long-tailed open-world scenarios to validate their findings.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- *Novel Contribution to Concept Drift in MLLMs*: The paper extends concept drift theory to the multi-modal domain, providing a new perspective on how VL models can adapt to unpredictable changes in data distributions, which is a significant advancement in the field.\n\n- *Effective Mitigation Strategies*: The introduction of a T-distribution based drift adapter effectively addresses biases from gradual drift while distinguishing sudden distribution changes, enhancing the robustness and reliability of VL models under concept drift conditions.\n\n- *Empirical Validation and New Dataset*: The extensive experiments demonstrate the proposed method's effectiveness in improving image-text alignment and performance on downstream tasks. The creation of the OpenMMlo dataset specifically tailored for long-tailed open-world settings adds value to the research community by providing a resource for further studies in this area.", "weaknesses": "- The authors raise important points regarding the presence of long-tail and Out-Of-Distribution (OOD) issues during the pre-training phase of Multi-modal Large Language Models (MLLMs). However, it would be beneficial for them to clarify why these issues are particularly relevant in the context of pre-training.\n\nSpecifically, they should address the following questions:\n\nRelevance of Long-Tail and OOD Issues in Pre-Training: Why do long-tail and OOD issues emerge during the pre-training phase? How do these issues affect the learning process of the model?\n\nConsideration of Sub-Distribution in Training Data: Should the pre-training process take into account the different sub-distributions present in the training data? What implications does this have for the model's performance and adaptability?\n\nImpact of Long-Tail Issues with Large Datasets: Is the long-tail problem still a concern when the pre-training dataset is large? If so, how does the size of the dataset influence the severity of the long-tail issue, and what strategies can be employed to mitigate its effects?\n\nAddressing these questions would strengthen this paper by providing deeper insights into the challenges faced during pre-training and the rationale behind the proposed solutions.\n\n- The used intra-class compactness and T-distribution are somewhat similar to those used in [a]. It would be beneficial to provide a more detailed discussion of the connections between them.\n\n[a] Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering. TPAMI, 2022.\n\n- In Figure 1a, it is unclear whether the test sets used to evaluate the BL (Balanced) or LT (Long-Tailed) trained VL (Vision-Language) models are the same. Additionally, it is important to know if these test sets are balanced in terms of class distribution. The answers to these questions could significantly impact the empirical evaluation results and the conclusions drawn from the study. I recommend that the authors clarify this aspect, as it is crucial for understanding the validity and reliability of the comparisons made between the two pretraining setups.\n\n- In Figure 1a, it is unclear whether the Out-Of-Distribution (OOD) samples are included in the pre-training of the Vision-Language (VL) model or if they are used solely for evaluation purposes. Additionally, the authors should clarify how OOD drift impacts the pre-training of the VL model.\n\nIt would be helpful to understand whether the performance is assessed on both In-Distribution (ID) test samples and OOD samples after incorporating OOD samples into the training process. This clarification is essential for comprehensively evaluating the model's robustness and adaptability in the presence of OOD drift. \n\n- In Figure 1b, it is unclear whether the Vision-Language (VL) model used for evaluation has been fine-tuned on a downstream class-imbalanced dataset. Additionally, the authors should discuss whether fine-tuning on a balanced dataset can mitigate the side effects of pre-training on an imbalanced dataset. \n\n- The writing and overall flow of the article should be further improved. Enhancing clarity and coherence will help convey the authors' ideas more effectively.\n\n- In Fig. 1a, 'LL' in the green rounded rectangle should be \"LT\"?\n\n- In Abstract, “distribution unpredictable changes” should be \"unpredictable distribution changes”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfRuA4KTJv", "reviewer": "Reviewer_zKrC", "summary": "Tailed drift and OOD drift are known issues in the vision and language community. While such problems have seen extensive studies on vision and language separately, it has not been studied jointly in multimodal learning. This paper attempts to bridge this gap. The authors show that pretraining on an imbalanced dataset worsens inter and intra-class as well as OOD features in terms of separability. These drifts further degrade performance in FT. A T distribution adapter is subsequently adopted to mitigate both drifts. The authors release a new long-tailed dataset pertaining. Extensive experiments on FT classification and OOD detection demonstrate the effectiveness of the proposed method.", "questions": "See Weakness. My question mainly lies in the clarity of the proposed method as well as dataset, the experiments are alright. While I believe this manuscript has its clear merit, I feel several confounding parts bar readers from a clear understanding and practitioners from adopting the method. I'm inclined to increase my score if the authors can answer my questions and make good efforts to modify their manuscript.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. the issue caused by tailed drift and OOD drift (concept shift) is illustrated clearly. \n2. The proposed method seems effective across most settings, barring a few in INat which is reasonable.\n3. the proposed method is properly theoretically motivated --I checked most math and they seem to be correct.", "weaknesses": "1. The manuscript lacks clarity on how their proposed method or model family compares to popular models like CLIP. The related work section of MLLM only discusses the common MLLMs but fails to draw a clear comparison to the proposed method. \n2. There lacks motivation on exactly how the proposed method work and why they are designed in this way. The entire section 2 is on defining concept shift as well as T-distribution adapter how ever it doesn't address the architecture. While the authors mention that \"drives the training of all three modules\", more details are needed here. I also think the authors should clearly state how each loss is calculated, for example, on which labels are the CE loss calculated. This helps reader understand, compare and contrast the proposed method with existing solutions. \n3. What advantage would training on OpenMMlo have compared to training on WIT? Are there any other benefits it brings to the table besides supporting the long-tailed study of this paper? -- since naturally we would want balanced dataset in the real world.\n4. In the related work section on MLLM, the authors mentions that \"However, most of them used the clip pre-trained model, that it is\npre-trained using high-quality and large-scale WIT dataset. And the VL models will also encounter\nconcept drift caused by data defects in the pre-training, which is the focus of this paper\", can you elaborate more on a) would pretraining on WIT mostly already alleviate the said issue and b) what \"also encounter concept drift\" mean here, odes it refers to pretraining on WIT?\n5. How do you define long-tailed data in image-caption pairs? Since no class label is given here how do we define these concepts?\n6. The authors mentioned that both gradual and sudden drifts arise from evolving distributions, I wonder if the pre-training experiments are conducted in the manner of different stages?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jn7L77LyF", "reviewer": "Reviewer_M73e", "summary": "This paper proposes a new unified framework for Multi-modal Large Language Models (MLLMs) to allviate the concept drift. The objective is to enhance MLLMs to enhance classification performance on long-tailed open datasets. The concept drift theory is introduced and extensive experiments demonstrates the method superior performance in downstream tasks. Finally, a multi-modal datasets named OpenMMlo is built for long-tailed open world setting.", "questions": "Besides tailed drift and OOD drift, domain shift is also a common problem in open-world environments, especially in the medical field. Can a T-distribution-based drift adapter be robust to domain shift? It would be better to provide experiments to evaluate this on ImageNet-Sketch.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and easy to follow.\n\nThe paper conducts extensive experiments, including long-tailed classification and OOD detection.\n\nThe paper's results show substantial improvements over other state-of-the-art (SOTA) baselines.", "weaknesses": "All downstream tasks only test on vision models, lacking exploration of the overall impact of MLLMs on concept drift. It would be better to provide more experiments, such as long-tailed classification and OOD detection, based on VL models.\n\nIt would be better to provide more ablations studies of the T-distributed spherical adapter component, it's not clear that which parts adapt to concept drift and be beneficial to downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)