Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

## Abstract

Abstract Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have increasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MLLMs in concept drift settings remain largely underexplored.
In this paper, we reveal the susceptibility and vulnerability of Vision-Language (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training.
To effectively address these challenges, we propose a unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL model to the distribution unpredictable changes.
Additionally, a T-distribution based drift adapter is proposed to effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distribution modeling.
Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL models, particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model’s ability to adapt to long-tailed open world.
Furthermore, we create a set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open world settings, to validate our findings. To foster the development of the multi-modal community, we have made both OpenMMlo datasets and our code publicly available at: https://github.com/Anonymous0Knight/ConceptDriftMLLMs .

## 1 Introduction

The rapid expansion of data availability has created significant challenges for multi-modal large language models (MLLMs), particularly in addressing concept drift, which predominantly manifests as gradual drift and sudden drift [].
Among them, tailed drift represents a classic illustration of gradual drift, emerging due to severe data imbalance, where the distributions of long-tail categories evolves because of their intrinsic sparsity and noise.
Concurrently, sudden drift is mainly represented by OOD drift, as the model encounters new, previously unseen concepts, resulting in distributional shifts that disrupt its ability to generalize in an open-world context. While the issues of long-tailed recognition and concept drift in open-world settings have been extensively studied in visual models [] and language models [], their impact on MLLMs, particularly vision-language (VL) models, remains largely unexplored. In this work, we aim to bridge this gap by providing a systematic analysis of how tailed drift and OOD drift affect VL models during both pre-training and fine-tuning phases. Our findings highlight critical vulnerabilities of current VL models in adapting to these challenges, underscoring the need for novel strategies to enhance their robustness in dynamic, open-world environments.

Pre-training: As illustrated in Figure 1(a), a comparison of the VL model trained on the balanced dataset ImageNet [] and the imbalanced dataset ImageNet-LT [] is conducted.
Due to the implicit feature centers of each category, we approximate them by averaging unit image and text features obtained by samples on the test set. To assess the intra-class compactness, the cosine distance between the image feature center and the text feature center from the same category is calculated and expressed as degrees. It is evident that training on the imbalanced dataset leads to a higher degree, indicating worse intra-class compactness brought by the tailed drift. Besides, with the tail drift intensifies, it results in a deterioration of the image-text alignment performance in tailed categories. Beyond the deterioration in tailed categories, the tailed drift also affects the image-text alignment in head categories with abundant training samples, which means that it leads to an overall performance degradation, not just the tailed categories. From the perspective of inter-class separability, we measure the average cosine distance from an image feature center to text feature centers of different categories. Figure 1(a) depicts that the VL model trained on the imbalanced dataset has lower inter-class separability. What’s more, we utilize KNN to extract 100 image and text feature centers of OOD samples to verify the impact of OOD drift on the pre-training of the VL model. Compared to training on the balanced dataset, the VL model trained under an imbalanced scenario is harder to distinguish between ID samples and OOD samples from the open world due to their similar inter-class separability. The undistinguished OOD drift will bias the underlying distribution of the feature space in the VL model, further disturbing the image-text alignment in the pre-training.

Figure: (a) Image-Text Alignment in Pre-training
Refer to caption: x1.png

Fine-tuning: We explicitly leverage the weights of the embedding layer in the VL model to visualize its feature space. The average cosine distance between each category and others is calculated as exhibited in the blue bars of Figure 1(b). with the decreasing of the training samples, a smaller degree means a worse inter-class separability. It is verified that tail drift leads to a compression of the feature space for tailed categories with a limited number of training samples, while head categories with abundant samples dominate the overall feature space of the VL model.
Moreover, the average cosine distance between each category and unit features extracted by OOD samples is applied to reveal the OOD separability as denoted as orange bars in Figure 1(b). Since head categories occupies most of the feature space, OOD samples are closer to the center of the head categories compared to the tail categories, implying that in the stage of fine-tuning, it is difficult for the VL model to distinguish between OOD samples under imbalanced scenarios.

To effectively address tailed drift and OOD drift within a unified framework, which often occur simultaneously, we encapsulate them using the concept drift theory. Therefore, summarizing the above challenges of vision language models in the long-tailed open world, it raises the important question:

How to adapt multi-modal large language model to concept drift in the long-tailed open world?

###### Remark 1.1 .

Research Objective:
Our focus lies in addressing the drift that MLLMs exhibit when confronted with the long-tailed open world, rather than leveraging MLLMs to enhance classification performance on long-tailed open datasets. The classification is exploited as the downstream task to provide a more intuitive visualization of the concept drift suffered by the MLLMs, which could have other downstream tasks.

Therefore, we propose a concept drift-aware multi-modal large language model, mitigating the tail drift and OOD drift encountered in the long-tailed open world. Firstly, we introduce the concept drift theory to the multi-modal domain, which provides a more holistic perspective to explain tailed drift and OOD drift.
Then, the T-distributed adapter is proposed to be embedded in the hyperspherical feature space. It aligns image-text features for contrastive learning in pre-training. The desirable light-tailed property of the proposed T-distributed spherical metric (Thp) prevents the compression of tailed categories and mitigates the crowding of feature space caused by tailed drift.
Besides, in fine-tuning, the adapter projects features to decision boundary and detects OOD samples at feature-level based on the underlying distribution. The proposed T-hp distribution explicitly models the feature space with concrete feature centers, optimizing large inter-class margins and yielding more desirable hyperspherical embeddings.
And a simple non-parametric KNN is adopted to distinguish the OOD sample based on the T-hp distribution. Finally, we construct a group of multi-modal long-tailed open datasets to support our claims.

In summary, our paper mainly makes the following contributions:

- 1.
We are the pioneers in revealing the unexplored impacts of concept drift to multi-modal large language models, especially in the image-text alignment in the pre-training and feature space allocation in the fine-tuning. This allows future research to more comprehensively study the impact of defect data on MLLMs.
- 2.
The concept drift theory is introduced and extended to multi-modal, integrating the tailed drift and OOD drift in a unified framework. And the T-distributed spherical adapter is proposed to perform the tailed adaptation and OOD detection in the pre-training and fine-tuning stage of the VL model.
- 3.
Extensive experiments evaluate the performance of our method under the long-tailed open world. Compared to specialized models, ours demonstrates superior performance in downstream tasks of long-tailed classification and OOD detection. Crucially, our model effectively addresses drift in image-text alignment, facilitating large-scale pre-training of MLLMs.
- 4.
We build a group of multi-modal datasets OpenMMlo under the long-tailed open world, by extending existing image-based long-tailed open datasets. It contains about 740k image-caption pairs with related category annotations. To support and encourage the community focused on multi-modal, we have made both the OpenMMlo and our code public.

###### Remark 1.1 .

## 2 Methodology

### 2.1 Multi-modal Concept Drift Theory

Concept drift is a phenomenon in which the statistical properties of a target domain change over time in an arbitrary way []. Formally, given a set of examples denoted as the data stream $S_{0,t}=\{d_{0},...,d_{t}\}$, where $d_{i}=(X_{i},y_{i})$ is one data instance, $X_{i}$ and $y_{i}$ respectively denote the feature vector and the label, and $t$ represents the timestamp of the instance in the data stream. $S_{0,t}$ follows a certain distribution $F_{0,t}(X,y)$. The concept drift is formalized as: $\exists t:P_{t}(X,y)\neq P_{t+1}(X,y)$, where the joint probability $P_{t}(X,y)$ can be decomposed as $P_{t}(X,y)=P_{t}(X)\times P_{t}(y|X)$. Although the concept drift due to tailed and OOD data often co-occur, they are fundamentally distinct phenomena. The tailed concept drift foucus on the drift in $P_{t}(X)$, while $P_{t}(y|X)=P_{t+1}(y|X)$ remains unchanged. However, the OOD drift from the unknown categories triggers the drift of both $P_{t}(y|X)$ and $P_{t}(X)$, that $P_{t}(y|X)\neq P_{t+1}(y|X)$ and $P_{t}(X)\neq P_{t+1}(X)$.
Therefore, concept drift theory provides a unified framework to harmonize the tailed shift and OOD shift that often occur together, enabling more robust and adaptive deep learning models.

In the context of multi-modal vision language models, we extend the concept drift theory from a single data stream to multiple data streams. Each data modal is associated with a distinct data stream. Thereby, the multi-modal concept drift framework can robustly handle the complex, heterogeneous data distributions inherent to vision language models. Therefore, we formally define multi-modal concept drift as follows:

###### Definition 2.1 .

Assume that there are $N$ data streams corresponding to $N$ modalities, given a set of examples denoted as $S_{0,t}=\{S_{0},...,S_{i},...,S_{t}\}$, where $S_{i}=(s_{1},...,s_{j},...,s_{N})$ and $s_{j}=(X_{ij},y_{i})$ is one data instance from a single $j$-th data stream, $X_{ij}$ is the feature vector, $y_{i}$ is the label and $t$ is the timestamp of the data stream. $S_{0,t}$ follows a certain distribution $F_{0,t}(S_{i})$, the multi-modal concept drift occurs at timestamp $t+1$, if $P_{0,t}(S_{i})\neq P_{t+1,\infty}(S_{i})$, denoted as:

$$ $\exists t:P_{t}(S_{i})\neq P_{t+1}(S_{i})$ (1) $$

###### Remark 2.1 .

Differences: Concept Drift vs. Data Drift (Covariate Drift)
Data drift entails changes solely in the distribution of inputs $P(x)$, while concept drift involves alterations in both input and output distributions, i.e., $P(x)$ and $P(y)$, leading to changes in the decision boundary. Furthermore, data drift predominantly stems from internal factors like data collection and processing, whereas concept drift typically arises from external factors, reflecting real-world changes.

Figure: Figure 2: The workflow of our methodology, consisting of two stages: the pre-training of the vision-language model and the fine-tuning on downstream tasks. Within the data streaming, a drift adaptation window slides to detect changes in data distribution and subsequently update the model, in both pre-training and fine-tuning. In the pre-training, the T-distributed adapter aligns visual and textual feature space by image-text contrastive learning, with a large inter-class margin. Coupled with the language model loss, they drive the training of all modules. In the downstream task, the image encoder and the text decoder are frozen out of training, with a linear projector fusing image-text features. Additionally, a mixture of experts module is leveraged with the T-distributed adapter as the router, allowing it to effectively adapt tail drift and perform OOD drift detection based on the distribution.
Refer to caption: x3.png

###### Definition 2.1 .

###### Remark 2.1 .

### 2.2 T-distributed Adapter for Concept Drift

To adapt the vision language model to concept drift, it is essential to adapt the model to align with the evolving data distribution, which can be formally defined as

$$ $\min_{f^{(t)},f^{(t+1)},...,f^{(t+\tau)}}\sum_{i=t}^{t+\tau}L(f^{(i)}(x^{(i)}) ,y^{(i)})$ (2) $$

where $f^{(t)}$ denotes the vision language model trained by the data stream $S_{t-k,t-1}$ from the drift adaption window with the size of $k$. And the model is driven by the target metric $L$ continuously to adapt the drift in a given time period $[t,t+\tau]$. Thus, one prevalent method for detecting and adapting to concept drift involves designing metrics based on data distributions that can effectively counteract the impacts of sudden and gradual changes within the time window [, ].
Building upon this thinking, we integrate directional statistics into distribution-based drift detection and adaptation, proposing a T-distributed adapter to alleviate it in the vision language model. Firstly, we provide an overview of directional statistics in the Appendix LABEL:appendix:ds. Then, we will introduce the T-distributed adapter.

Inspired by the T-SNE [], we design a T-distribution based metric in hypersphere (T-hp), which follows the density:

$$ $p_{X}(x^{(i)})\propto\frac{2}{\kappa(1-\mu^{T}x^{(i)})}$ (3) $$

where $x^{(i)}\in\mathbb{S}^{d-1}$ denotes the unit feature vector, $\mu\in\mathbb{S}^{d-1}$ represents the center of category and $\kappa\geq 0$ symbolizes the concentration of the distribution, with higher values indicating a greater concentration around the center $\mu$. Accordingly, we can get the marginal normalizer:

$$ $N_{T}(\kappa,d)=\int_{\mathbb{S}^{d-1}}{\frac{2}{1-\kappa\mu^{T}x^{(i)}}} \mathrm{d}x=\frac{1}{\kappa}2^{\alpha+\beta-1}\frac{\Gamma(\alpha)\Gamma(\beta )}{\Gamma(\alpha+\beta)}$ (4) $$

where $\alpha=\frac{d-1}{2}$, $\beta=\frac{d-3}{2}$, and $\Gamma(\cdot)$ represents the gamma function. Combined with Eq. LABEL:eq:2, the normalizer $N_{X}(d)$ of density $p_{X}(x;\mu)$ is:

$$ $N_{X}(\kappa,d)=N_{T}(\kappa,d)\cdot A_{d-2}=\frac{2^{\alpha+\beta}\pi^{\beta} }{\kappa}\frac{\Gamma(\alpha)}{\Gamma(\alpha+\beta)}$ (5) $$

Thus, the probability density function of the proposed Thp distribution is as follows:

$$ $p(x^{(i)})=N_{X}(d)^{-1}\frac{2}{\kappa(1-\mu^{T}x^{(i)})},\quad x^{(i)}\sim \text{Thp}(\mu)$ (6) $$

The detailed derivation process is provided in Appendix LABEL:appendix:Tsp.

Figure: Figure 3: The proposed T-distributed spherical metric with various $\kappa$ and the classical vMF metric when $\kappa=1$.
Refer to caption: x4.png

In terms of adapting to tailed concept drift, the Thp metric with a large concentration exhibits a light-tailed property, wherein the probability density function exhibits a faster rate of decay as the values increase, relative to the vMF metric, as illustrated in Figure 3. The high kurtosis of Thp is characterized that it yields high confidence only when the feature vector is sufficiently close to the center of the category, thereby minimizing the influence of head category samples on the tail category centers. Formally, $L_{\text{thp}}(\mu,x^{(i)})=\frac{2}{\kappa(1-\mu^{T}x^{(i)})+\epsilon}$ denotes the T-distributed metric on hypersphere, where $\epsilon$ is a non-zero value setting to $1$ to avoid the denominator of 0, and $L_{\text{vmf}}(\mu,x)=\exp(\kappa\mu^{T}{x})$ represents the vMF metric. Given an unit feature vector $x_{\text{head}}$ from the head categories, the gradient of the metric $L$ over the tailed category center $\mu_{\text{tail}}$ is $\frac{\partial L(\mu_{\text{tail}},x_{\text{head}})}{\partial\mu_{\text{tail}}}$. Due to $\mu^{T}_{\text{tail}}x_{\text{head}}\in[-1,1]$, when $\kappa\geqslant 1$, it is readily obtain that $\frac{\partial L_{\text{thp}}}{\partial\mu_{\text{tail}}}<\frac{\partial L_{
\text{vmf}}}{\partial\mu_{\text{tail}}}$. Consequently, the light-tailed Thp distribution effectively counteracts the squeezing of tail categories caused by an overwhelming number of head samples, thereby alleviating the bias induced by tailed concept drift.

Likewise, the Thp metric is directly applied to detect the OOD concept drift. A sample with a unit feature vector $x$ is deemed out-of-distribution if it lies at a relatively large distance from the in-distribution (ID) data in the eigenspace. Following [], a simple non-parametric KNN is adopted to partition the data into two sets (ID vs. OOD), which does not impose any distributional assumption on the feature space. Here the distance is the Thp metric with respect to the k-th nearest neighbor.

### 2.3 T-distributed Vision Language Model for the Concept Drift

As illustrated in Figure 2, our proposed vision language model contains two stages, the pre-training and the fine-tuning on the downstream task. Specifically, the classification task is chosen to visualize the impact of the bias caused by the long-tailed open world on the model. The multi-modal concept drift theory offers a unified framework for integrating gradual drift adaptation and sudden drift detection, where heterogeneous image and text inputs are treated as distinct data streams.

The architecture of the proposed vision language model contains the image encoder, text encoder and text decoder. With an input image $I_{i}\in\mathbb{R}^{H\times W}$, the visual features $x_{\text{img}}$ are extracted by the image encoder $E_{\text{img}}$ and further projected to the spherical eigenspace by the L2 normalizer $P_{\text{norm}}$:

$$ $x_{\text{img}}=P_{\text{norm}}(E_{\text{img}}(I_{i}))\in\mathbb{R}^{n\times d}$ (7) $$

where $n$ is the number of visual features and $d$ represents the feature dimension. The image encoder $E_{\text{img}}$ can be any common visual backbones, such as Vit-Base [], Vit-Large [] and ResNeXt-50 []. In terms of the text encoder $E_{\text{txt}}$, with a processed input text sequence $T_{i}$, text features are extracted by the language encoder and further projected to the spherical eigenspace by the L2 normalizer $P_{\text{norm}}$:

$$ $x_{\text{txt}}=P_{\text{norm}}(E_{\text{txt}}(T_{i}))\in\mathbb{R}^{n\times d}$ (8) $$

where $n$ is the number of input tokens and $d$ represents the feature dimension. In our case, Bert [] is used as the language encoder.

In the pre-training of the vision language model, the T-distributed adapter aligns image and text features by contrastive learning. It seeks to align visual and textual transformer feature spaces by promoting similar representations for positive pairs and dissimilar representations for negative pairs. More importantly, our approach circumvents model bias stemming from the long-tailed distribution of data. It couples with the language modelling loss to drives the training of all three modules.

In the downstream classification task, we leverage the T-distributed adapter as the router to distribute features to various FFNs as experts.
Furthermore, by explicitly modelling the feature space, the T-router enables a straightforward application of non-parametric KNN to effectively partition the data into ID and OOD samples. Besides, following the Blip [], the image encoder and the text decoder are frozen out of training during fine-tuning. A head of the classifier with two linear layers embeds features into the spherical eigenspace is trained.

### 2.4 Building Multi-modal Dataset OpenMMlo for the Long-Tailed Open World

Due to the inherent challenge of obtaining images and related captions, most multi-modal datasets struggle to follow the long-tailed distribution in the open world.
Recognizing the demand for higher-quality multi-modal data with long-tailed distribution in an open world, we developed a group of datasets called Open Multi-modal Long-Tailed OOD Datasets (OpenMMlo).

We extend the open-source datasets, namely ImageNet-LT [], iNatualist2018 [] and Places-LT []. ImageNet-LT has 1,000 classes and contains 115.8k samples, with a maximum of 1,280 samples and a minimum of 5 samples for a category. Besides, it consists of 18k images for OOD detection.
Places-LT has 184.5K samples from 365 classes, with class samples ranging from 4,980 to 5. The iNaturalist 2018 is a large-scale species dataset collected in the natural world with 437.5K samples for 8,142 classes. We use the InstructBLIP [] to generate the related caption of the image, with the prompt of "What does this picture describe? Please describe in detail its size, location, color, and its relationship to the surroundings.". For more details about OpenMMlo, please refer to Appendix LABEL:appendix:details.

## 3 Experiments

In this section, we first present the performance in downstream long-tailed classification and OOD detection tasks, which is induced by tail drift and OOD drift on MLLMs. Then, we evaluate the interior feature space of the VL model and further demonstrate our method alleviates the crowding and bias problems caused by the tail drift and OOD drift. The constructed long-tailed multi-modal datasets OpenMMlo is utilized for training and validating.
In terms of the OOD drift detection, we follow the setting of CIDER []. The model is trained on CIFAR100-LT [] with an imbalance ratio of 100, and validated on external OOD datasets including
SVHN [],
Places365 [],
LSUN [],
iSUN [] and
Texture [].
More detailed experimental implementations are given in Appendix LABEL:appendix:expdetails.

### 3.1 Taming the Tailed Drift and OOD Drift for Robust Fine-tuning

We compare our proposed vision-language model with other models to explicitly demonstrate its superior performance in long-tailed open-world scenarios.
As shown in Table 1, our model demonstrates exceptional overall performance in long-tailed classification across two large-scale datasets, namely the ImageNet-LT and iNaturalist 2018. Especially, training from scratch means that our method uses the imbalanced dataset for pre-training instead of utilizing the pre-trained model, such as clip [] pre-trained by the large WIT dataset.
The results validate the robustness of our vision language model against biases arising from tailed drift, particularly when leveraging large-scale data for both pre-training and fine-tuning.

**Table 1: Evaluation results of long-tailed classification on ImageNet-LT and iNatualist2018. The best-performing models are highlighted in red. Many, Medi. and Few denote the evaluated splits of many-shot ($>$100 training samples), medium-shot (20-100 samples) and few-shot ($<$20 samples). Top-1 accuracy is applied to evaluate the performance of different methods.**
|  |  | ImageNet-LT | iNaturalist 2018 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Backbones | Many | Medium | Few | All | Many | Medium | Few | All |
| Training from scratch |  |  |  |  |  |  |  |  |  |
| cRT [] |  | 61.8 | 46.2 | 27.3 | 49.6 | 69.0 | 66.0 | 63.2 | 65.2 |
| LWS [] |  | 60.2 | 47.2 | 30.3 | 49.9 | 65.0 | 66.3 | 65.5 | 65.9 |
| MiSLAS [] |  | 62.9 | 50.7 | 34.3 | 52.7 | 73.2 | 72.4 | 70.4 | 71.6 |
| BALMS [] |  | 64.1 | 48.2 | 33.4 | 52.3 | - | - | - | 70.6 |
| LADE [] |  | 64.4 | 47.7 | 34.3 | 52.3 | - | - | - | 69.3 |
| ACE [] |  | - | - | - | 56.6 | - | - | - | 72.9 |
| RIDE [] |  | 68.2 | 53.8 | 36.0 | 56.9 | 70.9 | 72.4 | 73.1 | 72.6 |
| PaCo [] |  | 68.2 | 58.7 | 41.0 | 60.0 | 70.3 | 73.2 | 73.6 | 73.2 |
| NCL [] | ResNet-50 | - | - | - | 57.4 | 72.0 | 74.9 | 73.8 | 74.2 |
| ViT [] |  | 73.7 | 46.5 | 15.6 | 52.4 | 65.4 | 55.3 | 50.9 | 54.6 |
| MAE [] |  | 74.7 | 48.2 | 19.4 | 54.5 | 79.6 | 70.8 | 65.0 | 69.4 |
| DeiT [] |  | 70.4 | 40.9 | 12.8 | 48.4 | 72.9 | 62.8 | 55.8 | 61.0 |
| LiVT [] | ViT-B/16 | 73.6 | 56.4 | 41.0 | 60.9 | 78.9 | 76.5 | 74.8 | 76.1 |
| Ours |  | 77.2 | 68.6 | 51.3 | 69.6 | 83.3 | 82.1 | 80.5 | 81.5 |
| Only Fine-tuning |  |  |  |  |  |  |  |  |  |
| LPT [] |  | - | - | - | - | 62.1 | 76.2 | 79.3 | 76.1 |
| BALLAD [] |  | 79.1 | 74.5 | 69.8 | 75.7 | - | - | - | - |
| Decoder [] |  | - | - | - | 73.2 | - | - | - | 59.2 |
| VL-LTR [] |  | 84.5 | 74.6 | 59.3 | 77.2 | - | - | - | 81.0 |
| LIFT [] | ViT-B/16<br>(w/ Pretrained<br>Clip) | 80.2 | 76.1 | 71.5 | 77.0 | 72.4 | 79.0 | 81.1 | 79.1 |
| Ours |  | 79.9 | 76.8 | 74.5 | 77.6 | 84.1 | 82.7 | 81.0 | 82.1 |

Compared to other methods trained from scratch, especially the ViT model, our model demonstrates a notable lead across all metrics, indicating our effective mitigation of concept drift during the pre-training, and providing robust pre-trained models for downstream tasks. Furthermore, to compare the current long-tailed methods, we the same setup as LIFT[], i.e., using the pre-trained model of clip and only fine-tuning. The superior results on medium and few splits of ImageNet-LT demonstrate the adaptability and robustness of our model in dealing with the gradual drift caused by tail data. Besides, although we are slightly behind LIFT[] in the few split of iNatualist2018, we still surpass it overall, exhibiting that our method does not compromise the accuracy of the head category to improve the tailed.

**Table 2: Evaluation results of OOD detection on CIFAR100-LT(ID) with the OOD datasets of SVHN, LSUN, iSUN and Texture. ResNet-34 is selected as the image encoder. The best-performing method is highlighted in red. FPR$\downarrow$ and AUROC$\uparrow$ are applied to evaluate the performance of different methods.**
| Methods | SVHN | LSUN | iSUN | Texture |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FPR | AUROC | FPR | AUROC | FPR | AUROC | FPR | AUROC |  |
| ProxyAnchor [] | 87.2 | 82.4 | 37.2 | 91.7 | 70.0 | 85.0 | 65.6 | 85.0 |
| CE + SimCLR [] | 24.8 | 94.5 | 56.4 | 89.0 | 66.5 | 83.8 | 63.7 | 82.0 |
| CSI [] | 44.5 | 92.7 | 75.6 | 83.8 | 76.6 | 85.0 | 61.6 | 86.5 |
| SSD+ [] | 31.2 | 94.2 | 79.4 | 85.2 | 80.9 | 84.1 | 66.6 | 86.2 |
| KNN+ [] | 39.2 | 92.8 | 49.0 | 89.3 | 75.0 | 82.7 | 57.2 | 88.4 |
| CIDER [] | 12.6 | 97.8 | 30.2 | 92.8 | 46.0 | 88.9 | 35.6 | 92.3 |
| Ours | 8.3 | 98.7 | 20.3 | 97.5 | 32.5 | 95.2 | 45.1 | 96.3 |

In terms of OOD drift detection, our proposed vision language model, trained on CIFAR100-LT as an in-distribution dataset, demonstrates exceptional performance across four diverse OOD datasets, as shown in Table 2. Our approach stands out with two significant advancements. Firstly, the training of our model does not incorporate any additional data from the open world to delineate the decision boundary between ID samples and OOD samples. Secondly, our proposed model detects OOD drift based on the hyperspherical distribution, without the need for any specialized modules. The proposed methodology offers the convenience of training and inference for large models.

### 3.2 Concept Drift-Aware Image-Text Alignment for Effective Pre-training

**Table 3: Evaluation results of image-text alignment of different contrastive learning strategies in the stage of pre-training, from three perspectives: ID intra-class compactness, ID inter-class separability and the separability between ID and OOD categories. The cosine metric is utilized to measure these distances, which is expressed as average degrees with standard deviation in brackets. We compare our proposed Thp with classical cosine loss, under balanced scenario (BL, ImageNet) and imbalanced scenarios (LT, ImageNet-LT), respectively.**
| Pre-training | ID Intra-class<br>Compactness $\downarrow$ | ID Inter-class<br>Separability $\uparrow$ | ID vs. OOD<br>Separability $\uparrow$ |
| --- | --- | --- | --- |
| BL / Cosine | 33.0 ($\pm$2.85) | 84.3 ($\pm$1.64) | 98.4 ($\pm$0.98) |
| LT / Cosine | 49.2 ($\pm$4.95) | 76.5 ($\pm$2.16) | 80.7 ($\pm$1.90) |
| LT / Thp | 36.2 ($\pm$3.53) | 85.6 ($\pm$1.88) | 101.3 ($\pm$1.25) |

Moreover, we verified at the feature-level that the proposed T-distributed adapter significantly alleviates the bias from tailed drift and OOD drift in the pre-training. As exhibited in Table 3, the degree of ID intra-class compactness reduces from 49.2 in LT/cosine to 36.2 in LT/Thp. It thereby validates the effectiveness of the proposed T-distributed adapter in enhancing feature extraction in long-tailed scenarios. Notably, the decrease in standard deviation demonstrates that the model considerably mitigates the bias induced by tail drift. In addition, our method achieves remarkable inter-class separability within in-distribution categories under long-tailed scenarios, even surpassing the performance of cosine achieved under the balanced dataset. It confirms the effectiveness of the proposed high kurtosis method in enhancing the alignment between images and text in the pre-training stage. Concerning the separability between ID and OOD, we achieve superior results even than the balanced condition, attributed to the inherent light-tailed property of the T-distributed adapter. It ensures our approach performs robust for OOD drift detection based on the spherical distribution robustly.

### 3.3 Ablation Experiments

#### 3.3.1 T-distributed Spherical Embedding in the Pre-training and Fine-tuning

Firstly, we conduct ablation experiments to verify the improved performance of the T-distributed adapter in the stage of pre-training and fine-tuning, respectively. As demonstrated in Table 4, our proposed T-distributed adapter exhibits improvements in both the pre-training and fine-tuning stages of the vision language model. It is worth highlighting that the T-distribution adapter plays a more prominent role during the fine-tuning stage. We argue that it is due to different characteristics of the pre-training and fine-tuning.
During fine-tuning, the model is directly involved in specific downstream tasks, and explicit category centers are present in the classifier. In contrast, pre-training primarily focuses on aligning image-text features, where the implicit information of categories is embedded. As a result, the T-distribution adapter’s impact is more pronounced in the fine-tuning stage compared to pre-training.

**Table 4: Ablation evaluation results with or without the T-distributed adapter in the pre-training or fine-tuning. The ✓denotes the stage is trained with the T-distributed adapter. The results are based on the ImageNet-LT with the Vit-base. Top-1 accuracy (Acc) is used as the metric.**
| Pre-training | Fine-tuning | Acc |
| --- | --- | --- |
| - | - | 56.0 |
| ✓ | - | 58.7 |
| - | ✓ | 65.1 |
| ✓ | ✓ | 69.6 |

#### 3.3.2 Various Concentration Parameter κ 𝜅 \kappa italic_κ

Furthermore, we conduct ablation experiments to examine the impact of different concentrations of the T-distributed adapter on the overall performance of the vision language model in Table 5. Four fixed degrees of concentration are involved, namely $\kappa=4$ 16, 64 and 128. The greater the degree of concentration, the greater the kurtosis of the T-distributed spherical metric. Besides, the concentration can also be utilized as a trainable parameter joining in the training, with the initial setting of 16.

**Table 5: Ablation evaluation results of various concentration parameter $\kappa$ on long-tailed classification task. "Training" denotes the $\kappa$ involved in the training as a parameter of the model with an initial setting of 16. The results are based on the ImageNet-LT with the Vit-base. Top-1 accuracy (Acc) is used as the metric.**
| $\kappa$ | 4 | 16 | 64 | 128 | Training |
| --- | --- | --- | --- | --- | --- |
| Acc | 67.2 | 69.6 | 64.9 | 61.1 | 68.3 |

In Table 5, setting the concentration parameter to $\kappa=16$ yields superior results. We argue that, a smaller concentration makes it challenging to effectively mitigate the biases introduced by tail drift and OOD drift in the vision language model. In terms of the bigger concentration, the model is hard to train due to the high kurtosis of the Thp metric. In the context of concentration as a trainable parameter with the initialization of 16, there is a slight reduction in model performance, accompanied by a marginal increase of the concentration parameter to 16.37. We assert that, the introduction of the new parameter increases the model’s complexity, thereby making the training process more challenging. Additionally, the low learning rate hinders the effective training of the concentration parameters, leading its less change compared with the initialization.

## 4 Conclusions And Outlook

Our findings indicate that visual-language models are significantly affected by biases introduced during both pre-training and fine-tuning in long-tailed open-world scenarios. To address this, we propose a concept drift-aware unified framework for visual-language models. This framework incorporates a T-distributed adapter designed to mitigate biases arising from both tailed drift and out-of-distribution (OOD) drift. Additionally, we introduce a comprehensive set of multi-modal datasets (OpenMMlo) tailored to the long-tailed open world, which includes images, captions and related category annotations.

We hope that our work will inspire future advancements in multi-modal large language models, specifically addressing the mitigation of biases originating from real-world data challenges, such as tailed drift and OOD drift.
In future research, we will leverage concept drift theory to analyze the temporal, intensity, and regional aspects of the various drifts.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. The current experiment, which introduces additional data, larger resolution, and additional modules, makes it impossible to effectively evaluate the effectiveness of the method, which is my main concern in giving a rating.
2. The text decoder used in the fine-tuning stage is trained from the pretrain stage, so how could the method only fine-tuning using the pretrained CLIP model?
soundness: 3
strengths: 1. This model has good performance on both long-tail and OOD datasets.
2. The motivation of the paper is clear and detailed experiments and pictures are provided as support.
summary: This paper proposed a T-distribution based drift adapter for the vision language model to reduce the potential impact of concept drift in real-world data streams. Meanwhile, a multi-modal dataset OpenMMlo is built in a long tail open world setting.
weaknesses: 1. The introduction and evaluation of the image-grounded decoder and text modeling loss are lacking. The author did not introduce and evaluate this module in the methods and experiments. What is the motivation and function of introducing this module?
2. The setup of the experiment is inappropriate. Although OpenMMlo is presented as part of the paper contribution, it is not sufficient to show only the training results of OpenMMlo, which makes comparisons with other methods unfair and meaningless. The authors are encouraged to align the training strategy with the remaining methods, and the current experiments make it impossible to evaluate both the method and the new data set.
3. The resolution used in the paper is 336x336, and the performance of LIVT should be compared with that of the corresponding resolution.
4. Typos in Fig. 1, 'LL' should be 'LT'.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: - *Novel Contribution to Concept Drift in MLLMs*: The paper extends concept drift theory to the multi-modal domain, providing a new perspective on how VL models can adapt to unpredictable changes in data distributions, which is a significant advancement in the field.

- *Effective Mitigation Strategies*: The introduction of a T-distribution based drift adapter effectively addresses biases from gradual drift while distinguishing sudden distribution changes, enhancing the robustness and reliability of VL models under concept drift conditions.

- *Empirical Validation and New Dataset*: The extensive experiments demonstrate the proposed method's effectiveness in improving image-text alignment and performance on downstream tasks. The creation of the OpenMMlo dataset specifically tailored for long-tailed open-world settings adds value to the research community by providing a resource for further studies in this area.
summary: This paper addresses the challenges faced by Multi-modal Large Language Models (MLLMs) due to concept drift in real-world streaming data, which includes gradual drift from long-tailed data and sudden drift from Out-Of-Distribution (OOD) data. While previous research has focused on these issues within the separate domains of vision or language, their effects on Vision-Language (VL) models remain underexplored. The authors propose a unified framework that adapts concept drift theory to the multi-modal context, enhancing the adaptability of VL models to unpredictable distribution changes. They introduce a T-distribution based drift adapter to mitigate biases from gradual drift and improve identification of sudden changes. Extensive experiments demonstrate that their approach enhances image-text alignment accuracy during pre-training and improves performance on various downstream tasks. Additionally, the authors create a new multi-modal dataset, OpenMMlo, designed for long-tailed open-world scenarios to validate their findings.
weaknesses: - The authors raise important points regarding the presence of long-tail and Out-Of-Distribution (OOD) issues during the pre-training phase of Multi-modal Large Language Models (MLLMs). However, it would be beneficial for them to clarify why these issues are particularly relevant in the context of pre-training.

Specifically, they should address the following questions:

Relevance of Long-Tail and OOD Issues in Pre-Training: Why do long-tail and OOD issues emerge during the pre-training phase? How do these issues affect the learning process of the model?

Consideration of Sub-Distribution in Training Data: Should the pre-training process take into account the different sub-distributions present in the training data? What implications does this have for the model's performance and adaptability?

Impact of Long-Tail Issues with Large Datasets: Is the long-tail problem still a concern when the pre-training dataset is large? If so, how does the size of the dataset influence the severity of the long-tail issue, and what strategies can be employed to mitigate its effects?

Addressing these questions would strengthen this paper by providing deeper insights into the challenges faced during pre-training and the rationale behind the proposed solutions.

- The used intra-class compactness and T-distribution are somewhat similar to those used in [a]. It would be beneficial to provide a more detailed discussion of the connections between them.

[a] Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering. TPAMI, 2022.

- In Figure 1a, it is unclear whether the test sets used to evaluate the BL (Balanced) or LT (Long-Tailed) trained VL (Vision-Language) models are the same. Additionally, it is important to know if these test sets are balanced in terms of class distribution. The answers to these questions could significantly impact the empirical evaluation results and the conclusions drawn from the study. I recommend that the authors clarify this aspect, as it is crucial for understanding the validity and reliability of the comparisons made between the two pretraining setups.

- In Figure 1a, it is unclear whether the Out-Of-Distribution (OOD) samples are included in the pre-training of the Vision-Language (VL) model or if they are used solely for evaluation purposes. Additionally, the authors should clarify how OOD drift impacts the pre-training of the VL model.

It would be helpful to understand whether the performance is assessed on both In-Distribution (ID) test samples and OOD samples after incorporating OOD samples into the training process. This clarification is essential for comprehensively evaluating the model's robustness and adaptability in the presence of OOD drift. 

- In Figure 1b, it is unclear whether the Vision-Language (VL) model used for evaluation has been fine-tuned on a downstream class-imbalanced dataset. Additionally, the authors should discuss whether fine-tuning on a balanced dataset can mitigate the side effects of pre-training on an imbalanced dataset. 

- The writing and overall flow of the article should be further improved. Enhancing clarity and coherence will help convey the authors' ideas more effectively.

- In Fig. 1a, 'LL' in the green rounded rectangle should be "LT"?

- In Abstract, “distribution unpredictable changes” should be "unpredictable distribution changes”.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: See Weakness. My question mainly lies in the clarity of the proposed method as well as dataset, the experiments are alright. While I believe this manuscript has its clear merit, I feel several confounding parts bar readers from a clear understanding and practitioners from adopting the method. I'm inclined to increase my score if the authors can answer my questions and make good efforts to modify their manuscript.
soundness: 3
strengths: 1. the issue caused by tailed drift and OOD drift (concept shift) is illustrated clearly. 
2. The proposed method seems effective across most settings, barring a few in INat which is reasonable.
3. the proposed method is properly theoretically motivated --I checked most math and they seem to be correct.
summary: Tailed drift and OOD drift are known issues in the vision and language community. While such problems have seen extensive studies on vision and language separately, it has not been studied jointly in multimodal learning. This paper attempts to bridge this gap. The authors show that pretraining on an imbalanced dataset worsens inter and intra-class as well as OOD features in terms of separability. These drifts further degrade performance in FT. A T distribution adapter is subsequently adopted to mitigate both drifts. The authors release a new long-tailed dataset pertaining. Extensive experiments on FT classification and OOD detection demonstrate the effectiveness of the proposed method.
weaknesses: 1. The manuscript lacks clarity on how their proposed method or model family compares to popular models like CLIP. The related work section of MLLM only discusses the common MLLMs but fails to draw a clear comparison to the proposed method. 
2. There lacks motivation on exactly how the proposed method work and why they are designed in this way. The entire section 2 is on defining concept shift as well as T-distribution adapter how ever it doesn't address the architecture. While the authors mention that "drives the training of all three modules", more details are needed here. I also think the authors should clearly state how each loss is calculated, for example, on which labels are the CE loss calculated. This helps reader understand, compare and contrast the proposed method with existing solutions. 
3. What advantage would training on OpenMMlo have compared to training on WIT? Are there any other benefits it brings to the table besides supporting the long-tailed study of this paper? -- since naturally we would want balanced dataset in the real world.
4. In the related work section on MLLM, the authors mentions that "However, most of them used the clip pre-trained model, that it is
pre-trained using high-quality and large-scale WIT dataset. And the VL models will also encounter
concept drift caused by data defects in the pre-training, which is the focus of this paper", can you elaborate more on a) would pretraining on WIT mostly already alleviate the said issue and b) what "also encounter concept drift" mean here, odes it refers to pretraining on WIT?
5. How do you define long-tailed data in image-caption pairs? Since no class label is given here how do we define these concepts?
6. The authors mentioned that both gradual and sudden drifts arise from evolving distributions, I wonder if the pre-training experiments are conducted in the manner of different stages?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Besides tailed drift and OOD drift, domain shift is also a common problem in open-world environments, especially in the medical field. Can a T-distribution-based drift adapter be robust to domain shift? It would be better to provide experiments to evaluate this on ImageNet-Sketch.
soundness: 3
strengths: The paper is well-structured and easy to follow.

The paper conducts extensive experiments, including long-tailed classification and OOD detection.

The paper's results show substantial improvements over other state-of-the-art (SOTA) baselines.
summary: This paper proposes a new unified framework for Multi-modal Large Language Models (MLLMs) to allviate the concept drift. The objective is to enhance MLLMs to enhance classification performance on long-tailed open datasets. The concept drift theory is introduced and extensive experiments demonstrates the method superior performance in downstream tasks. Finally, a multi-modal datasets named OpenMMlo is built for long-tailed open world setting.
weaknesses: All downstream tasks only test on vision models, lacking exploration of the overall impact of MLLMs on concept drift. It would be better to provide more experiments, such as long-tailed classification and OOD detection, based on VL models.

It would be better to provide more ablations studies of the T-distributed spherical adapter component, it's not clear that which parts adapt to concept drift and be beneficial to downstream tasks.

Meta-review: The paper addresses the problem of concept drift for MLLMs with a new adapter-based approach. The paper received four reviews with 2x accept and 2x borderline accept ratings. The reviews are generally positive: the reviewers appreciated the idea and the results; they also found the paper well-motivated and the writing easy to follow. On the negative side, the reviewers mainly criticized that the paper lacks clarity in several places. The rebuttal had properly addressed the reviewers' questions.

Decision: Accept (Poster)