Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
Authors: Adyasha Maharana
Jaehong Yoon
Tianlong Chen
Mohit Bansal
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Instruction Tuning
Continual Learning
Data Selection
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Visual instruction datasets from various distributors are released at different times and often contain a significant number of semantically redundant text-image pairs, depending on their task compositions (i.e., skills) or reference sources. This redundancy greatly limits the efficient deployment of continually adaptable multimodal large language models, hindering their ability to refine existing skills and acquire new competencies over time. To address this, we reframe the problem of lifelong Instruction Tuning (LiIT) via data selection, where the model automatically selects beneficial samples to learn from earlier and new datasets based on the current state of acquired knowledge in the model. Based on empirical analyses that show that selecting the best data subset using a static importance measure is often ineffective for multi-task datasets with evolving distributions, we propose Adapt-$\infty$, a new multi-way and adaptive data selection approach that dynamically balances sample efficiency and effectiveness during LiIT. We first construct pseudo-skill clusters by grouping gradient-based sample vectors. Next, we select the best-performing data selector for each skill cluster from a pool of selector experts, including our newly proposed scoring function, Image Grounding score. This data selector samples a subset of the most important samples from each skill cluster for training. To prevent the continuous increase in the size of the dataset pool during LIT, which would result in excessive computation, we further introduce a cluster-wise permanent data pruning strategy to remove the most semantically redundant samples from each cluster, keeping computational requirements manageable. We validate the effectiveness and efficiency of Adapt-$\infty$ over a sequence of various multimodal instruction tuning datasets with various tasks, including (Knowledge) VQA, multilingual, grounding, reasoning, language-only, and multi-image comprehension tasks. Training with samples selected by Adapt-$\infty$ alleviates catastrophic forgetting, especially for rare tasks, and promotes forward transfer across the continuum using only a fraction of the original datasets.

Published as a conference paper at ICLR 2025
ADAPT-00: SCALABLE CONTINUAL MULTIMODAL IN- STRUCTION TUNING VIA DYNAMIC DATA SELECTION
Adyasha Maharana"
Jaehong Yoon"
Tianlong Chen
Mohit Bansal
Department of Computer Science, UNC Chapel Hill
ABSTRACT
Visual instruction datasets from various distributors are released at different times and often contain significant number of semantically redundant text-image pairs, depending on their task compositions (i.e., skills) or reference sources. This re- dundancy greatly limits the efficient deployment of continually adaptable mul- timodal large language models, hindering their ability to refine existing skills and acquire new competencies over time. To address this, we reframe the problem of lifelong Instruction Tuning (LilT) via data selection, where the model auto- matically selects beneficial samples learn from earlier and new datasets based on the current state of acquired knowledge in the model. Based on empirical analyses that show that selecting the best data subset using static importance measure is often ineffective for multi-task datasets with evolving distributions, we propose Adapt-oo, a new multi-way and adaptive data selection approach that dynamically balances sample efficiency and effectiveness during LilT. We first construct pseudo-skill clusters by grouping gradient- we select the best- sest-perfrming data selector for each skill cluster from pool of selector experts, including our newly proposed scoring function, Image Ground- ing score. This data selector samples subset of the most important samples from each skill cluster for training. To prevent the continuous increase in the size of the dataset pool during LIT, which would result in excessive computation, we fur- ther introduce cluster-wise permanent data pruning strategy to remove the most semantically redundant samples from each cluster, keeping computational require- ments manageable. We validate the effectiveness and efficiency of Adapt-co over a sequence of various multimodal instruction tuning datasets with various tasks, including (Knowledge) VQA, multilingual, grounding, reasoning, language-only, and multi-image comprehension tasks. Training with samples selected by Adapt- alleviates catastrophic forgetting, especially for rare tasks, and promotes for- ward transfer across the continuum using only a fraction of the original datasets."
sample vectors. Next,
INTRODUCTION
Multimodal instruction tuning (Liu et al., 2023b; Zhang et al., 2023a; Liu al., 2023a; Gan al., 2024; Yoon et al., 2024) has been actively explored to enhance visual reasoning or the generation ability of Multimodal Large Language Models (MLLMs) (Zhu et al., 2023; Li et al., 2023a; Tang et al.. 2023; Team et al., 2023; Munasinghe et al., 2023; Yu et al., 2024; Zhang et al., 2024) fol- lowing user intents by training models on human or machine-generater multi-task visual instruction tuning datasets (Li et al., 2023c; Yin et al., 2023; Xu et al., 2024; Chen et al., 2024). While many distributors continue to release new high-quality instruction-tuning tasks and datasets, continually adapting large models to these massive multi-task datasets over time is prohibitively costly and inef- ficient. Given a pre-trained MLLM and the continuous expansion of the datasel pool with a stream of multi-task instruction-tuninn the challenge lies in developing an ever-evolving, instruction instruction-folloving MLLM in the most data- and compatation-efficient manner. This research question poses realistic, sustainable instruction
datasets, commonly observed in the research community today,
"Equal contribution.
Code released https: : / //gi //gitttt thub. htt.t..cmmmayymaaaraaaaaaaaaaata

---

Published as a conference paper at ICLR 2025
Temporal stream of instruction tuning datasets
Step Pseudo-Task Clustering
t-0
[-]
[=]
Gradient Vectors
k-means
Training Data Pool
Step Redundancy-bbsed
Stop 2: Multi-way Data Selection
Lifelong Multimodal LLM
Permanent Pruning
So
from similer samples
Selection of scoring function and data samples
R Retain more from samples
training
Figure 1: Illustration of Adapt-co. When new dataset is incorporated into the data pool the beginning of each timestep, Adapt-so extracts sample vectors forms pseudo-task clusters based on their similarity. Using a set of scoring functions, Adapt-ox predicts the most suitahle scoring function for each cluster and trains an MI.LM on the selected samples. To prevent excessive computation the pool size grows, introduce dataset compression by permanently removing redundant samples.
tuning scenario for MLLMs, distinct from conventional continual learning (Zenke et al., 2017; Yoon al., 2018; Van de Ven & Tolias, 2019), which focuses on learning a sequence of disjoint tasks. Specifically, at each time step, we assume new multimodal multi-task instruction-tuming dataset added the existing training pool, which already contains previous datasets. The model aims to train on samples from this continually expanding dataset pool, a scenario we refer to Lifelong Instruction Tuning (LilT). This scenario is critical as recent datasets tend be exhaustively large to variety of skills uni- and multimodal tasks. It poses the unique challenges of (1) data redundancies over time due to significant overlap between datasets, (2) imbalanced task represen- tations due to over-representation of simple tasks like captioning, (3) learning rare tasks, and (4) learning new output modalities.
Our initial experiments with sequential multimodal instruction tuning, i.e., training sequence of instruction tuning datasets, show catastrophic forgetting, particularly when new output modalities such bounding boxes and key points are introduced. While experience replay small subset of past datasets helps mitigate forgetting, is insufficient for retaining rare or unique tasks that appear only once due the skewed task distribution. This challenge is further compounded by the loosely defined nature of 'tasks' in instruction tuning (e.g., multi-turn conversations across multiple tasks on the same image) and the lack of sample-wise task labels. To address this, we explore LilT from a data selection perspective, enabling the model to learn skills in a balanced manner over time and avoid overfitting to dominant tasks. Specifically, we explore how to prune both past and incoming datasets to create a balanced training set at each time step, considering the model's 's current state.
To build an efficient, lifelong-evolving MILM,, we introduce Adapt-co: Adaptive Multi-way Prun- ing, efficient and dynamic multimodal data selection strategy. At each time step. Adapt-oo selects the most beneficial samples for the current model from the data pool, adapting to the model's 's evolv- ing knowledge and changing dataset distributions. This is crucial for LilT, sample importance shifts over time. Adapt-oc operates in two major steps: (1) Task-based clustering. When introduc- ing new dataset, we integrate it into the training data pool and create pseudo-task clusters using gradient vectors to represent data samples. (2) Cluster-wise data selection. We select influential samples from each cluster for model training. Motivated by the observation that different score functions (Paul et al., 2021; Toneva et al.; Coleman et al.) define sample importance differently, we propose new multi-way data selection approach that chooses the best scoring function from pool of experts based on its discriminativeness (measured by entropy). It selects skill- ball-eb subset of highly influential samples from the current training data pool. addition, to better assess the influence of multimodal samples, we also propose new scoring function, image grounding score (IG), which measures the change sample perplexity when grounded by visual information. This metric prioritizes samples that enhance MLLM multimodal skills and serves lector in Adapt-oo. As the data pool grows, computing scores and representations in Adapt-co, like other data selection strategies, becomes costly. To ensure diversity while managing computational costs, Adapt-oc adds (3) Permanent data pruning that removes semantically redundant samples from the data pool at the end of each time step, thereby continually controlling its size during LilT.
effective data
2

---

Published as a conference paper at ICLR 2025
We design an experimental setup for this previously unexplored scenario of lifelong multimodal instruction tuning using the pre-trained LLaVA 1.5 (Liu et al., 2023a) model on a stream of five visual instruction tuning datasets. As discussed earlier, experience replay alone is insufficient to mitigate forgetting, however, selecting a random subset from the data pool of past and new datasets reduces the forgetting rate from 26% to nearly 2%. Score-based data-selection strategies largely fail in this setting due to their inability to select task-balanced data subsets from multi-task datasets. Adapt-oo not only minimizes the forgetting to a mere 0.9% using only a fraction of the training data pool, but also promotes forward transfer of skills and consistently achieves > 100% relative gains.
We conduct extensive ablations of Adapt- to identify best-performing settings. Our key finding is that hidden layer outputs capture semantics, while gradient vectors represent skills, making them more effective for pseudo-task clustering (see examples in Figures 2 and 4). Further, we find that gradients from the middle layer of the model lead to the best performance, suggesting that skill retention could be localized to few layers in LLMss. We also find that zero-order gradients (Hinton, 2022; Sung et al., 2024) are promising, computationally cheaper alternatives to backpropagated gradients for pseudo-task clustering (See Table 5). Lastly, skill-wise analysis shows language-only skills are easiest to retain, while multilingual multimodal skills suffer significant forgetting in LilT.
In summary, to the best of our knowledge, we are the first to explore the realistic setting of life- long multimodal instruction tuning where the temporal stream of datasets may contain new skills, overlapping or rare tasks, and redundant samples. This scenario necessitates dynamic data selec- tion strategies for efficient and effective learning. Our proposed method, Adapt-oc, demonstrates superior retention as well as forward transfer of skills over time.
2 RELATED WORK
Multimodal Instruction Tuning Datasets. While multimodal data, such as image-text pairs, has increased significantly, multimodal instroction-followinng data remains relatively scarce due to the cone-conn nature of human data collection. To address this, recent works (Liu et al., 2023b; Zhang et al., 2023a; Chen et al., 2023; Zhu et al., 2023; He et al., 2024a) have leveraged generative models to collect such data from existing image datasets. Li et al. (2023c) introduce the M3IT dataset with 2.4 million instances and 400 task instructions, translated into 80 languages. Xu et al. (2024) develop VISION-FLAN, a large-scale dataset of 187 tasks with expert-written instructions, ensuring diversity through iterative refinement. MultiInstruct (Xu et al., 2023) features 62 tasks across 10 categories, sourced from 21 open datasets.
Continual Instruction Tuning. In the era of multimodal LLMs, instructional datasets have raised several timely research problems. Therefore, it crucial to develop sustainable models that can ad- dress emerging data and real-world challenges. Inspired by continual learning ( (Van de Ven & Tolias, 2019; Srinivasan et al., 2022; Lee et al., 2024b), paradigm focused on enabling models to adapt to non-i.i.d, time-variant tasks, continual instruction tuning (CIT) (Chen et al., 2024; Zhang et al., 2023b) has recently been studied for (multimodal) LLMs that allows the model adapt to multiple instruction tuning datasets sequentially without costly retraining. KPIG (He et al., 2024b) introduces a new CIT method that helps L.LMs capture spec-fie information and avoid overfitting general instructions by computing key-part information gain on masked parts to replay data and refine train- ing dynamically. EProj (He et al., 2023) and Wwwd-Prompt (Zheng et al., 2024) expand the CIT to the training of large multimodal models. EProj introduces new regularization and model expansion methods based on task correlations for continual instruction tuning of LMMs. Twd-Prompt proposes prompt-basee based approach that projects the prompt gradient into the residual space to minimize task interference while utilizing pre-trained knowledge, reducing negative forward transfer.
Data Selection. Data selection has been explored in the form of coreset selection in many works (Welling, 2009; Chen et al., 2010; Feldman et al., 201 1). Uncertaintylossserroo-bas
methods es- timate the difficulty of a sample from model confidence (Swayamdipta et al., 2020) or its training dynamics (Toneva et al.; Paul et al., 2021; Bachem et al., 2015). Zheng et al. (2023) address catas- trophic accuracy drop at high pruning rates; Maharana et al. (2024) represent datasets as undirected graphs and employ message passing to select the best subset. Gadre et al. (2024) investigate data selection for CLIP Radford et al. (2021) models. Evans et al. (2024) use learnability score (Minder- mann al., 2022) to accelerate training of CLIP models.
3

---

Published as a conference paper at ICLR 2025
A. Importance score distributions heavily overiap across tasks
C. importance score functions emphasize different data quality attributas
cane 9d
1G
Parpleaity Scores
EL2N Scores
Entrory Scores
the cabloet
rign the Loft
the Japtipp
vectors more separable by task than feature vectors (right)
heed?
fott
greanded
Image
taaton tol
eritbin
righ
15KE dimens
-SHE dimeeelon 0
Praocarrigur
Figure 2: A: Sample distributions for different visual language tasks in M3IT (Li et al.. 2023b) based on two importance scores, EL2N and entropy. B: t-SNE visualization of sample vectors based on their gradients and fcatures. C: Histogram of Perplexity and Image Grounding scores. We visualize few samples from M3IT with prompts (black) and ground-trutt trrnt answers (blue)
3. LIMITATIONS OF SCORE-BASED DATA SELECTION IN LIFELONG MULTIMODAL INSTRUCTION TUNING
3.1 LOCALITY OF THE SAMPLE IMPORTANCE: DATA SELECTION DEPENDS ON THE DATA
Score-based selection methods are widely used to assess the importance of training samples across modalities (Zheng et al., 2023; Liu et al., 2024; Marion et al., 2023; Evans et al., 2024; Gadre al., 2024). We analyze two importance scores in multimodal instruction tuning: the EL2N score (Paul et al., 2021) and entropy score (Coleman al.). EL2N measures the L2-norm of the output error vector, while entropy reflects the uncertainty in the output probabilities. Using the M3IT dataset (Li al,, 2023b), which includes eight tasks: captioning, multilingual (Chinese), classification, gener- ation, knowledge VQA (kvqa), reasoning, videoQA, and VQA, we compute score distributions. As shown in Figure 2A, relying on single importance score metric, such as EL2N or entropy, in- sufficient to differentiate meaningful samples across a diverse range of tasks. For instance, selecting higher EL.NN scores tends to favor for generalization (Paul et al., 2021), captioning samples over kuga, leading to skewed dataset.
In addition, the effectiveness of different importance scores varies based on the task and dataset at hand. The perplexity score is effective for filtering out low-quality samples in VQA that generally occur the tail end of its distribution (see Figure 2C). Tasks such as kvya (in purple) and captioning (in blue) are more separable via their entropy scores than EL2N scores. Morcover, Zheng et al. (2023); Swayamdipta et al. (2020) show that the most effective training subset contains a balanced mix of casy, difficult, and ambiguous samples. A biased score estimator may assign higher or lower scores to too many samples, making it hard to select the most effective subset. Thus, we need a different, generalizable strategy for assessing sample importance across multiple datasets during multimodal instruction tuning.
3.2 IMPORTANCE OF VISION-LANGUAGE DATA WITH IMAGE GROUNDING SCORE
The perplexity score measures the likelihood of given sequence of tokens as predicted by an autoregressive generative model and has been used for selecting samples with higher instruction following difficulty in language datasets (Li et al., 2024). For MLLMs, the perplexity function additionally conditions on the input image tokens within the model. A lower perplexiy score implies that the model assigns higher probability to the output tokens. We compute the perplexity of
4

---

Published as a conference paper at ICLR 2025
multimodal data instance 2, for MLLM with weights 0 as:
PPL(z.) = exp(
Σ NLL(e,)), where NLL(e,) -og((e)les 1:0).
(1)
where (e)((e)) indicates the negative log-likelihood of token ej in the sequence 2; comprising of image I and tokens e. As discussed in the previous section, perplexity is useful for detecting low- quality multimodal samples (see Figure 2C top). Motivated by the effectiveness and generalizability of the perplexity score (Marion et al., 2023; et al., 2024), we further modify this scoring func- tion to distill the importance of the image in a multimodal data instance. We compute the image grounding score of multimodal data instance 2; with image I and tokens e as:
(PL(e)
1Z,),) = PPL(e,
(2)
A higher IG score is assigned when the model assigns a higher probability to the text when condi- tioned on the image, compared when the image is absent. Conversely, lower IG score indicates that the image has little to no effect on the text's probability. As shown in Figure 2C bottom, an image-query pair with higher IG score requires the model to carefully understand the visual scene (e.g., reading the text on a sign in the image). In contrast, examples where the model can predict the answer without seeing the images represent lower IG scores. Thus, the IG scoring function allows us to discard multimodal samples that do not leverage the multimodal functionality of MLLMs.
4 LIFELONG MULTIMODAL INSTRUCTION TUNING VIA MULTI-WAY DATA SELECTION
4.1 PROBLEM STATEMENT
This paper tackles the problem of LilT over a sequence of multiple large datasets. Let Du.... Dr-1 be a set of accessible datasets where D, = ( ( p1 denotes the dataset released in the timestep composed of N; image-text pairs. Formally, observed datasets for a given computational budget, such as FLOPs or training iterations. Given the model f parameterized by 0, the training objective at time step T is formulated follows:
aim train multimodal model over multiple
T
N
arg min
Σ
Σ
c (1). 0.) s.t. T- -
(3)
T
1=
where Ñ; T E N. N. S No. and r is the computational budget. r(-) denotes a decay function conditioned T and т, and ÿ indicates the ground truth answer corresponding to the input data sample. Here, we constrain the minibatch iterations for multimodal instruction tuning per training timestep, by subsampling D = (æ., p: ) where D C D. D ~~ P(D D. TT). When a new multimodal instruction tuning dataset Dr is released, {D.Y is (re-) drawn for finetuning 0.
We propose the Adapt-oo data selection method for the problem of lifelong multimodal instruction tuning and describe each of its steps in detail in the following sections.
4.2 PSEUDO-TASK CLUSTERING VIA GRADIENTS
In the LilT scenario, the model continuously updates its weights to incorporate new knowledge and refine its capabilities in specific tasks by training on unseen, meaningful data. The relative importance of each data sample evolves with changes in the model's state and expansion of the data pool at each time step. Therefore, adjusting the relative importance of samples within the data pool over time is crucial to faster and better optimization under restricted conditions. We accomplish this by first using gradient vectors from the model's current state to estimate skill clusters within the training data pool. As shown in Figure 2B and Figure we find that gradient vectors significantly more separable by skills than hidden state outputs of the model.
For model with weights 0, for layer I, compute the gradients of 9; for sample (z ys) ((z,, Pc). 0:) using backpropagation. We obtain the data representation for the sample by concate- nating weight gradient vectors Voz, [Vo z;; Vo, Z.i...; Vo.- 2i], where T. denotes the number
5

---

Published as a conference paper at ICLR 2025
of layers, and construct pseudo-task clusters in the data pool by performing k-means clustering over data representations of the seen and unseen datasets (see Figure 1). However, in practice, we find that not all gradients are necessary for distinguishing samples into multiple meaningful skills. Fol- lowing Yoon et al. (2022), we cluster samples based on the gradients of essential layers, which leads to better performance compared to using gradients from all layers and is more memory-efficiet as well (see discussion in Section 6).
4.3 ENTROPY-BASED MULTI-WAL WAY DATA SELECTION
As discussed in Section 3.1, different scoring functions provide distinct advantages when select- ing meaningful samples from wide range of vision-language instruction tuning tasks, such as in-domain VQA, open-ended VQA, multilingual QA, visual grounding, commonsense reasoning, scientific/physial understanding, and OCR. To address the limitations of relying on single data selection (pruning) method and promote collaborative recision-making when evaluating sample im- portance, based on the model's current state and data diversity, we propose a novel and versatile, multi-way data pruning strategy. First, we construct a function pool S = (=o(-),...,88% (-) where s.(.)) denotes rith sample selection operation based on the corresponding importance score func- tion. Here, we aim to identify the function & that maximizes the entropy of the distribution of scores over the samples of each pseudo-task cluster. For the k-th cluster C₃ with m samples, first ex- tract the corresponding scores for each 8,, using model weights @ at time step t. We approximate the distribution of scores, Pe, by binning the range of normalized scores and calculating densities ph over the BB(h) bins. Omitting cluster index k for brevity, the selection of $ is formulated as:
g(k) arg max H (P%) where Pad (1% (x)), Ab ((k) and x: E Cs-
(4)
Se A score function that yields higher average entropy in its distribution compared other functions indicates a better ability to assess the uncertainty of the system (i.e., the model 0 at timestep ()- Moreover, we employ the CCS (Zheng et al., 2023) sampling strategy that aims select balanced representation of seen and unseen samples as well as frequent and rare tasks by sampling uniformly over the range of score function. Hence, is even more imperative to use score function that is discriminative over its entire range. Since the resulting pseudo-task clusters may vary in size, we define training budget T and divide it uniformly over the l clusters. The leftover data budget from clusters with sizes [c.] smaller than T/k are equally distributed over other clusters. This results in the selection of skill-balanced subset from the pool of multi-task datasets in lifelong learning. Importantly, our proposed multi-way approach is highly flexible since the scoring function pool can be seamlessly extended with new scoring functions based on users' needs.
4.4 COMBINED PERMANENT- TEMPORAL DATA PRUNING BY REMOVING REDUNDANCY
The steps of Adapt-co outlined in the previous sections enable the model to effectively select the most beneficial samples for each pseudo-task (i.e., skill) and train on them adaptively, based on the current model and evolving dataset distributions. However, release of new instruction-tuning datasets, the computational burden of updating gradient-based sample vectors and ranking their importance increases. To keep the computational requirements manageable over time, we further implement permanent data pruning strategy to iteratively reduce the size of the entire dataset pool. At the end of training each timestep, we measure pairwise cosine similarities between samples within each cluster and prune those with maximum similarities, as they are highly redundant and contain overlapping knowledge. The similarity is computed in the semantic space, which is well represented using hidden layer outputs of the model as shown in Figure (A (Abbas al., 2023; Sorscher et al., 2022). Larger clusters are prioritized for pruning to fit a predefined data pool budget D until all clusters retain a uniform number of samples. We refer to the version of Adapt-oc with this combined permanent-temporal pruning as -dat-Ada
the data pool grows with the
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Model. We conduct our experiments using the LLaVA 1.5 multimodal large language model (Liu al., 2023a). is trained on top of the Vicuna LLM (Chiang et al., 2023) using corpus of approx-

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of multimodal instruction tuning datasets. Distribution of various skills in the datasets. RE: Referring Expression, OD: Object Detection, KD: Keypoint Detection.
Skills
Truining Dulasets
Dutuset Size
VQA
Kanwkelge VQA
Captioning
Muli- lingual
Non-izut Outpu
Video QA
Couplex Reasening
Only
LLaVA 1.5 (Lin et 202363
065K 2.IM
RE
MEIT clict ual, 2005c]
x
x
MiniGPT4 (2hu es 20233
3K
x
x
x
MANTIS Diung stl 30144
665K
X
LaMM (Vin al. 2023)
250K
X
OD & KD
X
X
VistnFLAN DS0 ot 2004g
191K
imately 600K image-text caption pairs for pretraining of the vision projectors from the pre-trained CLIP visual encoder ViT-L/14 (Radford et al., 2021). Further, it is trained on the LLaVA instruc- tion tuning dataset consisting of 665K text-only and vision-language instances. We adopt LoRA finetuning (Hu et al., 2021) of the LLaVA-1.5-7B model with the recommended hyperparameters.
Datasets. For training, in addition to the LLaVA-665K instruction tuning dataset at t 0, consider the following order of datasets: M3IT (Li et al., 2023c), MiniGPT4 (Zhu et al., 2023), MANTIS (Jiang et al., 2024), LAMM (Yin et al., 2023) and VisionFLAN (Xu et al., 2024). Each dataset's temporal order, size, and skill composition are summarized in Table 1. We select standard evaluation datasets to measure performance on the skills enumerated in Table 1. These datasets and their corresponding task-specifie evaluation metrics are listed in Table 4.
Metries. We report various evaluation metrics from existing literature designed for understand- ing the continual learning phenomena in machine learning. The Average Accuracy (acc) at final timestep t = T is the average of the model's ' performance across all skills (and across datasets within each skill). The Relative Gain (r) metric (Scialom al., 2022) is the average of skill
performances as a % of the respective upper bounds i.c., ," = 3 ES bound X 100%. We consider the best performances in each skill group in the sequential learning setting to be the upper bound in performances and report r for the final time step T. We also report the Forget- ting Rate(f) which is the % drop in performance averaged across all skills and timesteps i.e.,
5xT E.=EE
min( p' pt .0) x 100%.
Adapt-co Setup. The optimal value of k in the pseudo-task clustering step is computed from grid search over values of between and 50, and selected based on the WSS value of clusters. In the score-based sample selection step,
bin size of 50 and discard the top and bottom 5% of samples for computing entropy as well as for CCS sampling, to remove outliers, low-quality sam- ples, and uninformative data (Zheng al., 2023). We perplexity, image grounding (Section 3.2), EL2N (Paul et al., 2021) and entropy (Coleman et al.) score functions for S throughout the paper.
Baselines. We present baseline numbers (1) Sequential and Multi-task training, (2) Random selection for experience replay (10% of past datasets), (3) Score-based selection methods, including Random, EL,2N (Paul et al., 2021). Entropy (Coleman et al.), Perplexity (Marion al., 2023), and (3) recent competitive data pruning baselines: SemDeDup (Abbas et al., 2023), Density-based Pruning (Abbas et al., 2024), and COINCIDE (Lee et al., 2024a). See Appendix for details.
5.2 MAIN RESULTS
We present the main experimental results in Table 2 and show breakdown of skill-wise accuracies at each time step for various methods in Figure 3. Our main findings are as follows:
Sequential training leads to catastrophic forgetting of skills. Multiple skills learned at t=0 (LLaVA) are forgotten at t=1,2 upon training the model on datasets containing a different set of skills. The M3IT dataset (I=1) does not contain grounding tasks and results in large drops in per- formance for the same. Similarly, the MiniGPT4 dataset (1=2) predominantly contains high-quality captions and causes forgetting of all other skills. The MANTIS dataset (t=3) improves performance on the MMMU dataset because it contains training instances with documents, charts, and visualiza- tion images. At /=4, the LAMM dataset contains object detection and keypoint detection tasks that
httpps: / /gi thub. wwt.//oo//.tae.cocoooooo haotian- 1 uu/ L.aa r/tee//mma /
Within the sum of squares (WSS) is of squared distance between samples and their cluster centroids.
7

---

Published as a conference paper at ICLR 2025
Table Overall results for lifelong multimodal instruction tuning. Comparison of performance of LLaVA models trained on datasets selected using Adapt-co vs. other data selection methods.
Pruning Strategy
Data size at
Relative % (7)
Forgotting % (1)
Aug. Acc. (
Sequemial
Full
68.D
26.0
$2.0
Multi-task
Full
92.5
46.1
Experience Roplay
Full
89.5
43.4
Random
25k
95.3
2.1
47.2
EL2N (Paul et all 2021)
25k
82.4
12.2
43.9
Entropy
25k
79.6
15.1
41.6
Perplexity (Marion et al., 2023)
25k
91.4
9.6
45.2
Image-g groue-ggnnding (ours)
25k
92.3
5.6
45.6
SumDeDup (Abbas all, 2023)
25k
76.4
6.4
38.5
Density-based (Abbas ol. 2024)
25k
78.1
5.1
39.6
COINCIDE (l.ce al., 2024g)
25k
89.5
3.9
44.7
Adapt-o
25k
102.3
0.9
50.5
Adapt-o0
SUk
107.2
51.7
Adupt-co
100k
109.7
0.4
52.5
Lerr-Adapp-co
25k
99.7
1.3
49.6
VOA
Know V0g
Mult linuual (CM)
Rondein
10
Groundnng
KNNU
SENUU
50
33
10
AUU
Lavgy
Average accuracies per skill over time. Comparison of average accuracies over time for each skill in our evaluation suite using various data selection methods. Higher is better.
Figure 3:
promote recovery of the referring comprehension skill learned at t = 0. presumably due to a simi- larity in their non-textual output modalities. Finally, VisionFLAN (t=5) recovers performance on all skills except grounding and MMMU. Surprisingly, VisionFLAN also induces recovery of multilin- gual and unimodal skills (MMLU) despite not containing those tasks in its dataset composition. This method incurs nearly 26% forgetting across all timesteps and retains only 68% of its all-time high performances at the final timestep (see Table 2). Models trained using Adapt-oo also outperform other models in multi-turn visual conversations (see Figs. 6, 7 and 8).
Random experience replay and pruning alleviate forgetting. Random experience replay using 10% data from past datasets significantly improves skills retention over sequential training, bringing down the forgetting rate from 26% to 6.6%. Random pruning of the combined set of past and incoming datasets results in better retention of skills with only a 2. % forgetting rate using a fraction of the datasets for training as seen in Figure 3. This result demonstrates the need for applying data selection methods to the combined datasets rather than in isolation and serves as strong baseline for lifelong multimodal instruction tuning. Notably, random pruning also improves the language-only skill of the underlying LLM in LLaVA. However, random pruming achieves a relative gain of 95%, falling short of reaching the best performance seen during sequential training.
Adapt-co minimizes forgetting and promotes forward transfer. Adapt-cc outperforms all exist- ing data selection methods for retaining and learning skills via instruction tuning over time. Score- based data selection methods generally fall short of random pruning because selecting samples from multi-task datasets based on scores leads to a skewed representation of tasks in the subset (see Fig- ure 2). Our proposed scoring function, Image grounding, prioritizes the data samples where the outputs are strongly grounded in images (see Fig. 2C) and achieves relatively higher performance for multimodal tasks. SemDeDup (Abbas et al., 2023), DBP (Abbas al., 2024). and COINCIDE (Lee et al., 2024a) rely on hidden layer outputs from the model to represent and prune data samples which can lead to imbalanced task distributions in the selected subset.
The use of gradient vectors in Adapt-oc to pool samples into pseudo-task clusters before pruning ensures that all skills are well-represented each time step, resulting in the lowest forgetting rate
8

---

Published as a conference paper at ICLR 2025
Nearest-reighbors in gradient vector space ditt-31
Nesrestneeighoors in hidden state space
Query
dim=0
dim-0
dm-115
dim=31
What is around the basebell Helld?
Hox earry players are standing and watching! answer tao.
Describe the trage- baseball playen is swargäng at pitch by wall that says France.
Translate from English to Crech. A person playing crtcket in an all green outfit.. hraje
Describe the image. running baseball crowd.
type venicie background?? The answer bus.
What is on the grean chain? answer
The answor is ferce.
giraiffes
ranyy cats thwre?
How, mony COwS arw there?
giraffes In the yee are
Describe the image. giraffe
Describe the image. ho
there?
are
to p4
trying to eat
standing ins idei
throe giraffes
Therw, are Strae deeen here.
rood harggtig wooden pole
fanced
girafre.
Figure 4: Nearest neighbor samples for query samples from VQA (top) and Chinese VQA (bottom)
tasks in the gradient (left) and feature spaces (right).
Table Ablation results for Adapt- Comparison of performance of the Adapt-co method to its ablated versions for lifelong multimodal instruction tuning.
Ablution Type
Values
Relative Gain % (f)
Forgetting Rate & (L)
Avg. Ace. (1)
Within-cluster Pruning
Multi-way
102.3
0.9
50.5
Image Grounding Score
96.3
2.7
48.8
EL2N
97.4
1.5
49.1
Data Representations
Gradients (middle layor)
102.3
50.5
Gradients (fiest layer)
97.2
1.8
47.9
Gradicuts (last layer)
101.5
50.1
Gradients (all layers)
98.9
2.5
49.1
Hidden layer outputs (all layers)
96.5
4.3
47.4
Cluster Budget
Uniform
102.3
0.9
50.5
Density-based
101.7
0.5
49,5
iee, 0.9%. Further, the multi-wal way score- score-based selection of important samples within those clusters promotes forward transfer skills and results > 100% relative gain, unlike any other method in Table 2. The relative gain increases by nearly 5% on doubling the data budget from 25K to 50K samples and shows signs of plateauing with further increase data.
Semantic deduplication for managing data complexity over time is effective. LITE-Adapt-co employs semantic deduplication of the training pool at the end each timestep to reduce the com- putation complexity of extracting data representations at the next step. For the deduplication size of 100K samples (4x training budget) across all timesteps, LITE-Adapt-oo suffers minor drops in performance and forgetting compared Adapt-co.
6 ANALYSIS
Relative gains per skill. We present the skill-wise breakdown of relative gains of each method discussed in Table 2, in Figure 5A. The largest increases are observed for the language-only skill across all methods, except the image-groume grounigg score which deprioritizes unimodal samples. This result is especially striking because none of the datasets in our experimental setting contain samples similar those in the MML.U evaluation dataset. It suggests that the underlying L.LM LLaVV recover this skill from similar multimodal samples. The next highest gains knowledge VQA dataset, using our mmaag--grondding
for the score and the Adapt-co method. Multilingual skills appear to be the hardest skills to learn and retain over time, potentially because they utilize different part of the model's vocabulary than other tasks and are included the MBIT dataset only.
Multi way pruning tions (or pruner experts) for each cluster to select a representative subset of importance samples from the cluster. This works better than using any single metric across all clusters shown in
single pruner. Adapt-co
among various importance
func-

---

Published as a conference paper at ICLR 2025
Brouledge/OOA
Iqual
laparo
Grandin
MMLLI
114
Relative data selection
1-NE plots gradient vectors from diff. layers
Figure 5: A: Relative gain % for different skills using various data selection methods in the lifelong multimodal instruction tuning setting. B: t-SNE visualization of gradient vectors of the M3IT dataset from layers of varying depth in the LLaVA model.
Table 3. Importantly, this serves flexible framework for swapping adding advanced 'expert' pruners, e.g., learnability score (Mindermann et al., 2022; Evans et al., 2024).
Source of data representations. Various data selection works propose different ways of represent- ing data samples in a high-dimensional space. Methods designed for pruning pretraining datasets or single-task finetuning can effectively use semantic representations such sentence embeddings (Abbas et al., 2023) and hidden state outputs (Sorscher et al., 2022; Maharana et al., 2024). However, we observe subpar performance with the use of semantic representations for pseudo-tass clustering, as reported in Table 3. Gradient vectors are better representations of the skill component of data sample as we show in Figure 4. Further, we experimented with gradients from all layers (similar to Xia et al. (2024); Liu et al. (2024)) as well as individual layers of the model. We observed the best performance with gradients from the middle layer only. Gradients from the last layer also work relatively well with Adapt-oo but those from the first layer work poorly. This discrepancy correlates with the compactness of task clusters in the t-SNE plots of gradients from the corresponding layers, as demonstrated in Figure 5B. Sampling budget across clusters. As outlined in Section 4, we sample a subset with an equal number of samples from each pscudo-task cluster in Adapt-oc. The budgets leftover from smaller clusters are distributed equally across the remaining clusters. We experimented with more intuitive budgets for each cluster i.e., based on the density of the cluster members (Abbas al., 2024). However, did not result in significant changes in the performance.
Efficiency. Most data selection methods require extra steps to select an influential data subset and incur computational costs. Adapt-oo expends additional memory and inference-time compute to ef- fectively select data. We experiment with three methods to reduce this cost: (1) Zero-order gradients (Hinton, 2022), (2) Varying size of data pool in LITE-Adapt-oo and (3) gradients from a smaller model i.c., TinyLLaVA (Zhou al., 2024). Using zero-order gradients instead of full gradients leads to 2% drop in average accuracy and a 2.4% drop in relative gains. TinyLLaVA gradients demonstrate a similar drop, in addition to a higher forgetting rate. Conversely, the performance of LITE-Adapt-so improves with increasing size of the data pool post-deduplication (see Table 5).
See Appendix for additional results using LLMs, analysis on efficiency and recovery of skills.
CONCLUSION
Valuable visual instruction tuning datasets from various sources are released over time and often contain overlapping text-image pairs. To efficiently train lifelong adaptive MLLMs on these growing datasets, scenario we call Lifelong Instruction Tuning (LilT), we reformulate data selection so the model automatically selects meaningful samples from both old and new datasets, maintaining balance when incorporating new data. We observe that assessing sample informativeness with a static importance measure is challenging in LiIT, as it depends on the model's evolving capabilities and the shifting dataset distribution. To address this, we propose a scalable lifelong multimodal instruction tuning approach that dynamically balances sample efficiency and effectiveness through temporal multi way data pruning. We show that training with samples selected by this method reduces catastrophic forgetting and enhances forward gain, using only a fraction of the original dataset, particularly for rare tasks with limited resources.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank the reviewers for their valuable feedback and input on the paper. This work was sup- ported by the National Institutes of Health (NIH) under other transactions 10T20D038045-01, NSF-CAREER Award 1846185, DARPA ECOLE Program No. HR001 1230006,,, NSF-AI Engage Institute DRL-21 1635.. DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031 ARO Award W91 INF21 10220, ONR Grant N00014-23-1-2356. The views contained in this article are those of the authors and not of the funding agency.
ETHICS STATEMENT
The intended use of Adapt-oc is to enhance the vision-language instruction-following capabilities of MLLMs by training them on an integrated, ever-growing instructional dataset over time. This system does not pose any specific potential for misuse beyond the general risks associated with AI technology. However, instruction-tuned frameworks, including Adapt-oo, are required to carefully consider the selection of training datasets and the intended purpose of usage to ensure the devel- opment of a reliable and trustworthy languala inference system that supports stable, reliable, and safe AI.
REPRODUCIBILITY STATEMENT
This paper fully discloses all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions. To maximize repro- ducibility, we have included our code in the supplementary material. Also, we report of our hyperparameter settings and model details in the Appendix.
REFERENCES
Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Amro Kamal Mohamed Abbas, Evgenia Rusak, Kushal Tirumala, Wieland Brendel, Kamalika Chaudhuri, and Ari S Morcos. Effective pruning of web-scale datasets based on complexity of concept clusters. In The Twelfth International Conference Learning Representations, 2024.
Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation-the case of de-means. In International Conference on Machine Learning, pp. 209-217. PMLR, 2015.
Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Lianli Gao, and Jingkuan Song. Coin: A benchmark of continual instruction tuning for multimodel large language model. arXiv preprint 0030052403.08350, 2024.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-mode models with better captions. arXiv preprint arXiv:2311.1279) 2023.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pp. 109-116, 2010.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang. Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations.

---

Published as a conference paper at ICLR 2025
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14223, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier Henaff. Data curation via joint example selection further accelerates multimodal learning. arXiv preprint arXiv:2406.1771 17711, 2024.
Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. Advances in neural information processing systems, 24, 2011.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Yulu Gan, Sungwoo Park, Alexander Schubert, Amthnyy Philippakis, and Ahmed M Alaa. In- structev: Instruction-tuned text-to-image diffusion models vision generalists. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:2311.16200 2023.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.1 /530, 2024a.
Yongquan He, Xuancheng Huang, Minghao Tang, Lingxun Meng, Xiang Li, Wei Lin, Wenyuan Zhang, and Yifu Gao. Don't half-listen: Capturing key-part information in continual instruction tuning. arXiv preprint arXiv:2403. 10056, 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint a3X:v:2212.13345, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 21X.:2200600985.. 2021.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9700-6709, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi malti-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798. 2014.
Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability- bbase data selection for large vision-language models. arXiv preprint arXiv:2406. 10995, 2024a.
12

---

Published as a conference paper at ICLR 2025
Jaewoo Lee, Jachong Yoon, Wonjae Kim, Yunji Kim, and Sung Ju Hwang. Stella: Continual audio- video pre-traimn trainini with spatiotemporal localized alignment. In Proceedings of the International Conference on Machine Learning (ICML), 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the Inter- national Conference on Machine Learning (ICML), 2023a.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. Mßi: A large-scale dataset towards multi- modal multilingual instruction tuning. arXiv preprint arXiv:2306.0438 2023b.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruc- tion tuning. arXiv preprint arXiv:2306. 04387, 2023c.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers). pp. 7595-7628, 2024.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-Rong Wen. Less is more: Data value estimation for visual instruction tuning. arXiv preprint arXiv:2403.09559 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Kei Chang, Gong-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balanc- ing diversity and difficulty in data pruning. In Proceedings of the International Conference on Learning Representations (ICI.), 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/eyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint arXiv:2309.0455 045444 2023.
Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Win- nie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Confer- ence on Machine Learning, pp. 15630-15649, PMLR, 2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1470-3487, 2022.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-XXin Yong. Hailcy Schoelkopf. et al. Crosslingual gen- eralization through multitask finetuning. arXiv preprint arXiv:22/ 1 1.0/786, 2022.
Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-Ilava Pixel grounding large video- Iado-aan models. arXiv preprint 231 2023.
13

---

Published as a conference paper at ICLR 2025
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning, pp- 27074-27113, PMLR, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596-20607, 2021.
Zheng Lin Qingyi Si. Alpaca-cot: An instruction fine-tuning platform with instruction data col- lection and unified large language models interface. btttps://githe com Phtn///hheuss////e
a alpaca-Coof CoT, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-6122, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-1953, 2022.
Tejas Srinivasan, Ting- Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and-languae tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yi-lin Sung, Jachong Yoon, and Mohit Bansal. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models. In International Conference on Learning Representations, 2024.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275 9293, 2020.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sorieut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 190400990 2019.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah Smith. Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 747766 2023.
Max Welling. Herding dynamical weights to leam. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128, 2009.
14

---

Published as a conference paper at ICLR 2025
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning. 2024.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinnsruuc: Improving multi-modal zero-shot learning via instruction tuning, In Proceedings of the Association for Computational Linguistics (ACL), 2023.
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang, Vision-flan: Scaling human-labeled tasks in visual instruction tuning, arXiv preprint arXiv:2402.11690, 2024.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang. Zhiy- ong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assister multi-modal instruction-tuiing dataset, framework, and benchmark. In Advances Neural Information Processing Systems (NeurIPS), 2023.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL
httpp:://ppenreee
n forum?dd=Skkkms
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Jaehong Yoon, Shoubin Yu, and Mohit Bansal. Raccoon: Remove, add, and change video content with auto-generated narratives. arXiv preprint arXiv:2405. 10.0005.8440 18406, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part I1 14, pp. 69-85. Springer, 2016.
Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Multimodal compositional video reasoning via efficient modular adaptation and fusion. arXiv preprint arXiv:2402.05889 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the International Conference on Machine Learning (ICML). 2017.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306. 17 17107, 2023a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model. hotps: / //llava-vl, github. bt//loog/02a/
2024.
Zihan Zhang, Meng Fang, Ling Chen, and Mohammad-Redd Namazi- Rad. Citb: A benchmark for continual instruction tuning. arXiv preprint arXiv:2310.145100 2023b.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high
pruning rates. In Proceedings of the International Conference Learning Representations (ICLR), 2023.
Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, and Huawen Feng. Beyond anti-forgetting: Multimodal continual instruction tuning with positive forward transfer. arXiv preprint arXiv:2401.0918 2024.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
15

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usion-langual understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
16

---

Published as a conference paper at ICLR 2025
Table Evaluation datasets used for measuring the performance of the trained model at each time step in our continual learning experiments.
Skill
Evaluation Datasets
Evaluation Metric
VQA
GQA (Hudson & Manning, 2019), LLaVA-
Accuracy
Bench (Liu et al., 2023a)
Knowledge VQA
ScienceQA (Lu et al., 2022), OK-VQA (Marino al., 2019), A-OK-VQU VQA (Schwenk et al.
Accuracy
2022)
Multilingual (Chinese)
COCO-CN, Flickr-8K-CN (Li et al., 2023b)
BLEU Score
Grounding
RefCOCO, RefCOCO+, (Kazemzadeh et al., 2014; Yu al,, 2016)
RefCOCOg
IOU
Reasoning
MMMU (Yue et al., 2024)
Accuracy
Language
MMLU (Ilendrycks et al., 202
Accuracy
APPENDIX
A BASELINES
Sequential Training. In this method, the model is naively trained on the stream of instruction tuning datasets without any experience replay. Generally, this method sets a lower bound on the performance of the model on each evaluation task.
Multi-task Training. This method comprises pooling all of the datasets across time steps and training the model this pooled dataset in one go. Generally, this method sets upper bound on the performance of the model on various skill sets, However, if there are low-resource tasks in the dataset, it can lead low performance on those tasks.
Coverage-based Coreset Selection (CCS). Zheng et al. (2023) introduces the method CCS where they divide range of difficulty scores into equal-sized bins and randomly data samples from each bin with a uniform budget. This approach is motivated by maximizing coverage over the semantic space while ensuring equal distribution of easy and difficult samples in the selected subset. Easy samples promote learning whereas difficult samples are information dense and promote generaliza- tion. We use this method in Adapt-co for score-based selection.
Score-based Selection, Multiple importance score functions have been proposed over the years for various data types. We select the following for baseline experiments: (1) Perplexity: This metric is widely used for filtering language corpora (Marion et al., 2023), (2) E1.2N (Paul et al., 2021): I): This metric is the L2-norm of the output error vector and is effective low pruning ratios (or high retention rates), (3) Entropy (Coleman et al.): This score function is the entropy value of the output probability vector.
Embedding- based Selection. SemDeDup (Abbas et al., 2023) extracts semantic embeddings for pertaining datasets using universal embedding transformer such CLIP (Radford et al., 2021) for image-text pairs or Sentence Transformer' for natural language corpora and performs deduplication within k-means clusters. DBP (Abbas al., 2024) assigns pruning budget the clusters in SemD- eDup using a cluster complexity score. COINCIDE (Lee et al., 2024a) clusters feature vectors into large number of cluster e.g., 1=10,000 to identify skill-coneept clusters and samples non-uniformly from the clusters using difficulty score metric.
'htpps: / wwwww//www//.c/////////// htw.://wwww...oooooo//o/ooooooaa hub un i versal _encoder
17

---

Published as a conference paper at ICLR 2025
Table 5: Efficiency results. Comparison of performance of efficient versions of the Adapt-co method for lifelong multimodal instruction tuning. T = Training data size; |D] Size of data pool after permanent pruning LiTE-Adap--o.
Method
Relative Gain % (1)
Forgetting Rate % (1)
Avg. Acc. (1)
Adapt-co (T=25k)
102.3
0.9
50.5
TinyLLaVA gradients (Zhou et al,, 2024)
99.6
2.9
48.1
Zeco-order gradients (Hinton, 2022)
99.9
1.5
48.5
LITE-Adapt-so 100k)
99.7
1.3
49.6
LITE-Adapt- (20kk)
101.8
50.1
LITE-Adapt-x (50k))
102.5
0.7
50.8
Table 6: Efficiency analysis. Comparison of total time taken for various data selection methods
across all time steps in our experiments with 25k training samples at each t. Numbers are in hours.
Method
Random Selection
Scoring-based
LESS oli, 2024)
COINCIDE (Love el 2024g)
Adapt-cc
LiTE-Alapt-o
Dula scoring
48
92
48
92
34
I-mrcns clastering
3.5
3,5
1.4
Training
21
Total
21
113
72.5
116.5
56.4
B EXPERIMENTAL SETUP
Additional details Adapt-co setup. We random projections (Park et al., 2023; Xia al., 2024) to reduce the dimensionality of gradient vectors extracted for the pseudo-task clustering step. We use constant projection dimension of 8192 throughout our experiments.
C ADDITIONAL RESULTS
Efficiency analysis. We present comparison of the time taken by various data selection methods for our experimental setting (for training with 25k samples each time step) in Table 6. Results are presented for 8 A100 GPUs. The total time taken to train the model without any methodical data se- lection (i.e., random pruning) is approximately 21 hours. Scoring-base based selection methods generally require a forward pass of the model to extract the score (e.g., EL2N, entropy), which takes nearly 48 hours on 8A100 GPUs for all datasets in our experiments. The COINCIDE (Lee al., 2024a) takes similar amount of time since it uses forward pass to extract hidden layer outputs from the models as data representations. LESS (Xia et al., 2024) and our proposed method Adapt-oc require longer time i.e., 92 hours, perform backward pass over the model and extract gradients. However, with LITE-Adapt-oo, we are able to significantly reduce this time due to systematic compression of the dataset each timestep.
Ablations with scoring functions. We present ablation results for the different types of scoring functions used in Adapt-ox. The results presented in Table for Adapt- include four scoring functions outlined in Sec. 5.1 i.e., perplexity, image grounding (Section 3.2), EL.2N (Paul al., 2021) and entropy (Coleman et al.) score functions. We refer to this set of functions as {S}. We performed additional experiments where used (a) only the image grounding score as the scoring function, and (b) all scoring functions except the mmage-groundingg Table We find that the image grounding score serves as good scoring function when used as standalone scorer in Adapt-co. In addition, Adapt-oo performs well even when the image grounding score is removed from the set of best performing score functions. This suggests that performance with our method can improve with better scoring functions, making Adapt-co a versatile method.
score. Results are reported in
Visual chat skill. Multimodal LLMs are equipped with chat-style multi-turn conversational skills due to the chat capabilities of the underlying LLM as well as instruction tuning data tailored to the skill. The LLaVA instruction tuning dataset contains such data; hence. the LLa VA model can hold multi-turn chats with a user. However, the multi-turn visual chat task not well-defined and lacks quantitative evaluation metrics benchmarks. Nevertheless, we interested in seeing if
18

---

Published as a conference paper at ICLR 2025
Rustes regetables: cerrets, broccais,
Must ia babind the apolos?
then flavtrfal cide stck. Vegetazie Conbine the vegetasiac with brots, and acd terbe and solces for conterting and teallty vegetabla 50a3.
lebörd the spples, grapes:
rectpe
the refrlgeratar:
Eeuits
Ingredienta:
what are the meals 1 con cook with these?
intrigerater.
orange,, pested aegranted
green apales, paeled and citiced
to
selad.
gopes,
With the varietyy of suults and vegecables in the outrigerator, you cas prepore a ronge of heal my ant del tihous mada, Somu comples inclade:
apple
(oottons1))
(ontiane))
chopped fresh laaver (optiona3)
spsles located
Instructions
the top shalf of the refrigerator, aleng with the other fouite and vegersbles.
In a largs t361, conbire the slsced applas, sognested aranges, talved gropes. If destred, driceie the fruit vith tang Lumn jaice awest Lengr
havanss
refrottng and intiitions trult salad.
Please precids the basttiin bos
Vegtiblle stir try: thep the broccoli, carrots, and pagpars Lato nite-stted pleces and saute the pun wish some garlic, and your sevsonings.
coorcinate of the ragion this serience describes: gress appla.
flavens.
If using wint lasess, and tver to the tow and toss again,
Serve the feult salad extiled at race tomperature.
hononar with creaue rutritionn and delicimus smoctie.
atte
This recioe veruetile hexithy optice refrwaking
Figure 6: Visual chat with LLaVA at /=0 (Liu et al., 2023a). We use representative chat example to evaluate the visual chat capability of MILMs trained using various methods in our experiments.
0
Whst are the maste CAT COOC with Theee?
What are the maste CAN cook with theee?
salad
sulad
uould coot saces, Where the Ssage?
would like to coot the saizd. Khere spple tnago?
spple
Left
top right
the this senteace deseribes:
senteace decrribec:
What behind the asplis??
What behlidd the apples?
necipe.
necipe.
Fruit salae
necipe
Figure 7: Visual chat with LLaVA at t= using sequential training (left) and random pruning (right) baselines in experiments. The conversational ability of the LLaVA model is greatly diminished over time using these baseline data selection methods.
Table 7: Ablations results for scoring functions in Adapt-oc for multimodal models. Compar- ison of performance of LLaVA model trained on datasets selected using Adapt-oc under different combinations of scoring functions.
Scoring Functions
Relative Gain (t)
Forgetting Rate % (L)
Avg. Acc. (f)
(S)
109.7
0.4
52.5
(S) Image Grounding Score
106.9
0.3
51.2
Image Grounding Score only
92.3
5.6
45.6
the multi-turn chat skill is retained in the LLaVA model during lifelong learning, especially since none of the fine-tuning datasets, except for LLa VA-65K,,,, contain conversational data. We perform qualitative evaluation of the visual chat skill using a single representative example demonstrated in
19

---

Published as a conference paper at ICLR 2025
would like to cook the fruit calad.
1 Excuabee
Whrre is the upple located in the lmage
Ranch brescoka
Runch carreta
The apple lacated on the top sheir refrigeratar, along with other
blue vater bettlet
cap of white
Instructtons:
vish all fruits and segetables thorneghly.
cosrelnate the this dotribo: apile.
Cat the green apple into atte-stted places.
Paek the enange and cut segeatta.
pepper
wtat ane the seals 1 can cook with these?
and the lettuce, zocuaber,
brocooll.
With the strtety of frutce and iogelassla eviiiable Use refrigerator, you cxn prepare a rarge hoaltty and delicizes wals. tome stinstry, coloeful and vegetable smoctvis, vegetable swe, or a frem vegetahle and Fruit calss, There moate zar he enjoyet se siandaaare siahes or is side distes Wo copplaeett other eate currest.
6. Pael the caneete and cut than into bite-sited pieces.
large bost, combine apple, orange,
Nut bekind the apoles?
bell puspery lettuse, cocumer, broccoll,
add grases
anted the there
ooe frutt calas, and ate well.
grapes.
buttles
Gtve to the rectoe.
Cover the losl and refiiigirate the frust talad for at least 10 strutes to allow the flavors to sold tagether.
uting the frutte avatlable
10. Serve the feult salad chillee.
retriprater:
fruits and vegetanles provide sixerse
Ingrsllests:
eejoy refrtgerater. You can cxctontte ingrediants and and adlitiione fruits vegetables you prefer.
rectpe etapte
range
gresn apple
nutelants, saking then lece
orange
chaics for creating nctratious cnt stylng wals.
red papeer
Figure 8: Visual chat with LLaVA at (=5 using our proposed dynamic data selection method Adapt-oo. Unlike other methods (see Fig. 7), the Adapt-oo method identifies conversation as a distinct task in the pseudo-task clustering step and retains sufficient samples from this task each time step to prevent forgetting the skill.
Fig. 6. In this example, the model is queried for VQA, knowledge VQA, and referring expression comprehension tasks based on single image. As seen in Fig. 6, the LLaVA model (at (=0 in our experiments) can perform all these tasks effectively in a multi-turn chat scenario, except for fine- grained visual reasoning. The models trained using sequential learning and random pruning in our experiments lose this skill, as shown in Fig. 7. The models' answers become less verbose and they provide inaccurate answers for most tasks. We present results from our method in Fig. 8. Adapt-oo identifies the multi-turn chat as a distinct task during the pseudo-task task clustering step and selects samples from this cluster for training at each time step. Thus, it can retain this skill effectively even after many steps of fine-tuning on other datasets.
What is important for multimodal skill recovery? As noted in Section 5.2, a model can recover previ- ously learned skills without being re-trained on data from that skill. Sequential training of the LLaVA model in our experimental setting shows that the model at (=4 has poor multilingual skills (introduced at (=1), and recovers those skills at 1=5 without be- ing trained on multilingual data (See the result of Sequential on Multilingual (CN) task in Figure 3). We use this phenomenon to understand what aspects of the training data induce skill recovery in multi- modal model. To this end, we train the model from /=4 timestep on different subsets of the Vision- FLAN dataset (=5 and evaluate its performance on the multilingual skill. Specifically, in (=5, we train the model on an equal number of (a) randomly selected samples, most similar samples in (b) LoRA gradient space, (c) vision projection layer gradient space, (d) hidden layer output space and (c) projection layer output space. We compute the average cosine similarity of each sample in the VisionFLAN dataset with the multilingual subset of the M3IT dataset (!=1). Gradients and outputs are extracted from the vision projector layers and the decoder layers to represent the visual and mul- timodal content of the samples, respectively. We report the relative gain of the trained model as a measure of skill recovery. As presented in Figure 9, the best multilingual performance is achieved by (b) training on VisionFLAN samples that have similar gradients in the LoRA modules, followed by (c) samples with similar outputs from the visual projection layer. Since we have already estab- lished that gradients represent the skill whereas hidden layer outputs represent the semantics (see Section 4.2), these results suggest that multimodal models can retain as well as recover previously
Skill Recovery
[a] Random
(b) LoRA grads
92.5
(c) Proj, grads
(d) Layer outputs
90.0
(e) Proj. outputs
87.5
85.0
Figure 9: Recovery of multilingual skill
with various training data selection methods.
20

---

Published as a conference paper at ICLR 2025
Table 8: Overall results for lifelong instruction tuning in language models. Comparison of performance of LLaMa3-8B models trained on datasets selected using Adapt-co vs. other data selection methods.
Method
Relative Gain % (t)
Forgetting Rate % (4)
Avg. Acc. (f)
Sequential Random
18.1
36.8
91.6
5.6
41.8
LITE-Adapt-oo 97.8
2.3
43.1
learned skills through reminding indirect representations of the skill in training datasets. We also observe this phenomenon at t=4 in the sequential setting where training the LLaVA model on object and keypoint detection tasks leads to recovery of the referring comprehension skill. These results provide further evidence for the effectiveness of gradient-based clustering of datasets in Adapt-co.
Data selection for language-only models. We performed experiments for lifelong instruction tun- ing (LilT) of language models explore the efficacy of our proposed approach for language-ona data. For this purpose, we use the LLaMA3-8B (Dubey et al., 2024) model and five multi-task language datasets in the following training order: Natural Instructions (Mishra al., 2022), Alpaca- CoT (Qingyi Si, 2023), xP3 (Muennighoff et al., 2022), UltraChat (Ding et al., 2023) and FireFly. We use the set of evaluation benchmarks used in Wang et al. (2023) to evaluate our models (MMLU, GSM. BBH, TydiQA, CodexEval, AlpacaEval) and compute the following metrics based on perfor- mance across these benchmarks: Average accuracy, Relative gain, and Forgetting rate (see Sec. 5.1 in main text). The model is trained on 25k samples at each time step, similar to the MILM experi- ments in our paper. We report results for sequential training, random selection and LITE-Adapt-oo on this setting in Table 8. We see up 18% forgetting rate in the sequential setting, which comes down to 5.6% and 2.3% with random pruning and Adapt-oo pruning. The average accuracy is high- est with our method the final time step. These results suggest that lifelong learning from multi-task instruction tuning datasets posed method can significantly alleviate forgetting of skills over time in the language domain, as well as preserve the maximum accuracy that can be achieved from each dataset.
significant problem in the language-only domain as well. Our pro-
21


Reviews:
[{"review_id": "9FbXCfZPLp", "reviewer": "Reviewer_9s1q", "summary": "This is an interesting article that addresses the issue of forgetting in multimodal large models within multi-task and continual learning scenarios from the perspective of instruction dataset selection and proposes effective strategies. Traditional training methods often lead to the model forgetting previously learned skills when new tasks are introduced. However, the article introduces the LAMP (Lifelong and Adaptive Multi-way Pruning) method, which uses gradient vectors for pseudo-task clustering and combines an entropy-maximization strategy for sample selection, achieving effective sample filtering and diversity balance. LAMP employs semantic redundancy detection and coverage-based sampling strategies to prune the data pool, control its size, and prevent unrestrained data pool expansion, while ensuring the representativeness and diversity of tasks. Moreover, training budgets are allocated reasonably across task clusters, allowing the model to be sufficiently trained on each task, thereby reducing forgetting and enhancing generalization capabilities. Overall, LAMP performs well in multi-task and continual learning environments, ensuring that the model retains previously learned skills when training on new tasks, even with limited computational resources, thus achieving efficient and balanced continual learning.", "questions": "please refer to the weakness part", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This article starts from the current situation where the instruction-tuning datasets used in fine-tuning multimodal large models are continuously increasing and points out the redundancy problem among these datasets. It then proposes a solution to address the issue of continual learning, demonstrating significant practical relevance. In the design of the proposed solution, the authors first analyze the problems and limitations of existing methods, then detail how the new solution effectively addresses these issues, showcasing clear research objectives and a logical structure. The narrative of the paper reflects the author’s deep understanding of the field and direction, providing readers with extensive knowledge and unique insights. Moreover, the article verifies the correctness and validity of the proposed solution through extensive experiments, opening up new avenues for future research and demonstrating considerable i", "weaknesses": "Although it is good to see the paper as a whole, there still exist some minor drawbacks in this work.\n1.\tIn Figure 2, the meanings of the axes in subfigure C are unclear, and the explanatory content for subfigure A is insufficient, failing to adequately explain the issue that the figure is intended to illustrate.\n2.\tThe text descriptions in all the images are not very clear and appear somewhat distorted.\n3.\tIn the paper, clustering operations on datasets based on gradients and subsequent pruning of redundant data use k-means and pair-wise cosine similarity, respectively. This computational approach likely incurs significant time and space costs in such large-scale instruction datasets. The question arises as to whether there are corresponding measures to address these challenges.\n4.\tDuring the data selection process, a pool of function tools is used, and the function that produces higher average entropy is ultimately chosen. In this process, combining multiple functions can be considered to achieve more robust data selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LC3TrE1NF9", "reviewer": "Reviewer_bvXZ", "summary": "This paper introduces Lifelong Multimodal Instruction Tuning, which differs from continual Multimodal Instruction Tuning in that the previous datasets are still included in the dataset pool. To effectively select the data at each timestep, the authors propose a framework that adaptively  selects data samples based on their importance and data balance. The experiments show that the method is more effective than baselines.", "questions": "the t-SNE visualization has been shown to have high stochasticity across seeds. Have you manually tuned for your method and other baselines throughout the paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The considered lifelong setting is more practical than the traditional continual learning setting for multimodal LLMs, since currently the most important aspect of large models is to use all data available better. \n- The proposed framework considers the key difficulties of data sampling in adapting multimodal LLMs with progressively growing datasets and reasonably tackles the problem with carefully designed pipelines. I find sections 3 and 4 meaningful with rigorous thoughts and experiment analysis.\n- The experiments are thorough and clearly show the effectiveness of the proposed framework over the considered baselines.\n- The paper is written very well.", "weaknesses": "- The proposed method, except for the image grounding score, seems not to be specifically designed for multimodal LLMs and can be potentially applied to broader domains like pure LLMs. However, the paper only considers multimodal LLMs which lowers the impact and importance of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkzBEpYYfG", "reviewer": "Reviewer_VSGx", "summary": "This paper introduces a novel approach, LAMP, for optimizing Lifelong Instruction Tuning (LiIT) of Multimodal Large Language Models (MLLMs). Traditional visual instruction datasets, which are frequently redundant and task-specific, hinder MLLMs' ability to continuously adapt and learn new skills effectively. LAMP addresses this by dynamically selecting and pruning data to maximize training efficiency while minimizing computational demands. It groups data into pseudo-skill clusters, applying adaptive selection to prioritize relevant samples for each skill and using a cluster-wise pruning method to reduce redundancies. This approach mitigates catastrophic forgetting, enhances knowledge transfer, and improves performance on rare tasks, all while leveraging only a fraction of the original dataset. LAMP's adaptive tuning demonstrates significant benefits for sustaining long-term model growth in evolving multimodal learning environments.", "questions": "Please focus on Weaknesses, and I encourage the authors to provide further discussion and clarification on these points.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method mitigates catastrophic forgetting and supports forward transfer, enabling the model to retain skills and adapt to new tasks seamlessly.\n2. LAMP’s cluster-wise pruning manages dataset growth by removing redundancies, keeping training scalable and resource-efficient.\n3. This paper is well-writing and its organization is logical.", "weaknesses": "1. A primary concern regarding the proposed sequential data structure and the lifelong-based instruction tuning strategy is their practical applicability, as the current MLLMs typically rely on the integration of diverse tasks together for effective multitask training instead of sequential learning.\n2. The paper mentioned that (lines 301-302) \"our proposed multi-way approach can be seamlessly extended with new scoring functions based on users’ needs\", however, the proposed method only employs four scoring functions in practice and lacks ablation experiments for additional scoring functions. I suggest the authors conduct related experiments to demonstrate the proposed method’s flexibility and scalability.\n3. In Tab.2, the effectiveness of the random pruning strategy is notably highlighted. The computational costs and complexities associated with other sophisticated pruning techniques do not appear to correspond proportionately to the performance improvements achieved. This raises questions regarding the efficiency of the proposed pruning methods in comparison to simpler alternatives.\n4. In Tab.2, although LITE-LAMP utilizes 4 times less training data (25k vs. 100k), its relative improvement dropped significantly when compared to LAMP, i.e., 99.7 vs. 109.7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3hVC7Jkzn7", "reviewer": "Reviewer_eQ3c", "summary": "This paper proposes a new scenario of lifelong instruction tuning, where Multimodal Large Language Models (MLLM) continuously learns from new datasets that include both new and redundant data samples. They then propose LAMP, a adaptive data selection approach designed for this context. The authors claims that the proposed method can select beneficial samples to adapt the current model's knowledge to the continuously changing dataset distribution.", "questions": "- How is \"accuracy\" measured on LLaVA-Bench?\n- In table 2, why does multi-task learning performs worse than random pruning which simply uses fewer data (e.g. accuracy 46.1% VS 47.2%) ? Can pruning more data lead to better result?\n- Can you provide numerical comparisons on the computation cost of the proposed method against the random pruning baseline? Additionally, how does the computation cost of data selection compare with the training cost at step $t$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The setting of lifelong instruction tuning introduced in the paper is practical in real-world applications.\n- The paper is overall well organized and easy to follow.\n- The proposed method achieves convincing results in experiments, and comprehensive ablation studies are provided to validate the effectiveness of individual components.", "weaknesses": "- The method does not appear very novel to me, as both gradient-based clustering and ensemble scoring functions for data selection have been explored in previous works.\n- In evaluation, the paper focuses on tasks with short answers but omits open-ended visual chat, where multimodal large language models (MLLMs) generate long responses. However, visual chat is essential for assessing the comprehensive capabilities of MLLMs.\n- Compared to random pruning, which already serves as a strong baseline, the proposed method (including the LITE or efficient version) incurs significantly higher computational and time costs, and no efficiency analysis on the computational cost is provided.\n- Clarification Issues. In Table 2, the authors do not clearly explain what \"data size at *t*\" refers to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)