LESS IS MORE: HIGH-VALUE DATA SELECTION FOR VISUAL INSTRUCTION TUNING

## Abstract

Abstract Visual instruction tuning is the key to building large vision language models (LVLMs), which can greatly improve the task generalization and solving capabilities by learning a mixture of instruction data from diverse visual tasks.
Previous work mostly collects multiple existing visual instruction datasets via heuristic ways for training (even more than a million instructions), which may introduce data redundancy and enlarge the training cost.
To investigate this issue, we conduct a series of empirical studies, which reveal a significant redundancy within the visual instruction datasets, and show that greatly reducing the amount of instructions from several tasks even do not affect the performance.
Based on the findings, we propose a high-value data selection approach TIVE , to eliminate redundancy within the visual instruction data and reduce the training cost.
In TIVE, we first estimate the instance influence score on its corresponding task, and the task difficulty score, based on the gradient-based influence functions.
Then, we leverage the two kinds of scores to determine the task proportion within the selected visual instruction subset, and select high-value instances for each task, respectively.
Experiments on various LVLMs show that our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks, even surpassing it on four of the benchmarks. Our code and data will be publicly released.

## 1 Introduction

The advent of large language models (LLMs) (Brown et al., ; Ouyang et al., ; Touvron et al., ; Zhao et al., ) has marked significant advancements in the field of artificial intelligence (AI), exhibiting excellent capabilities in human instruction following, world knowledge utilization, and complex reasoning. A surge of recent studies (Zhu et al., ; Liu et al., ; Dai et al., ; Liu et al., ) equip LLMs with the vision encoder to empower the capability of processing visual information.
Through vision-language alignment pre-training and visual instruction tuning, *Large Vision Language Models (LVLMs)* are created to extend the application of LLMs into multimodal tasks and scenarios.

Visual instruction tuning (Liu et al., ; Dai et al., ) is the key technique for improving the task generalization and instruction following capabilities of LVLMs, which relies on a set of visual instructions for fine-tuning.
Therefore, the construction of visual instruction datasets is very crucial for LVLMs.
Typically, there are two widely used ways to construct visual instructions: synthesizing instructions based on LLMs (Liu et al., ) or transforming existing vision-language datasets into visual instructions (Dai et al., ; Liu et al., ). To achieve better performance, existing LVLMs generally combine a mixture of visual instructions from different domains or tasks, to compose a large-scale visual instruction dataset.
The LVLMs fine-tuned on these mixtures of visual instructions have shown remarkable performance on massive downstream multimodal benchmarks.

Figure: Figure 1: A comparison of TIVE-8B with other open-source models in terms of the instruction data scale and average benchmark performance on MME, SEED-Bench, MMBench, ScienceQA.
Refer to caption: x1.png

However, such a mixture of instructions may also introduce significant data redundancy, leading to increased training costs and potentially overfitting risk.
To investigate the redundancy issue, we first conduct an empirical study on the visual instruction dataset of state-of-the-art open-source LVLM, *i.e.*, LLaVA-1.5 (Liu et al., ), by reducing the instruction amount of a certain task and then evaluating the performance.
The results show that the reduction of instruction data only leads to slight or even no performance decline across most benchmarks, indicating that there exists redundancy within the used visual instructions.
Therefore, it is promising to mitigate this redundancy by selecting a small set of representative data samples.
Furthermore, we also find that the degree of redundancy varies across different tasks. It suggests that the contribution of each task should be considered when performing the redundancy elimination.

To this end, in this paper, we propose a data selection approach for visual instruction tuning, namely TIVE, based on *Task and Instance Value Estimation*.
The key motivation is to estimate the value of each instance and then select the high-value ones, based on its influence on LVLM fine-tuning process.
According to the influence function theory (Pruthi et al., ), the influence of an instance on the training process can be estimated by its gradient similarities with other instances.
However, due to the large-scale parameters of LVLMs, the computation of gradient similarity may cause unaffordable cost.
Besides, since the goal of visual instruction tuning is to learn the solving capability for diverse tasks, it is necessary to measure the influence of task learning (Pruthi et al., ; Xia et al., ), instead of only cross-instance influence.

In TIVE, we adjust the gradient computation and influence estimation strategies, to better adapt into visual instruction tuning of LVLMs.
To reduce the cost, we only leverage the gradients of the LoRA (Hu et al., ) matrices from LLM for influence estimation. These parameters are the key components for learning visual understanding and instruction following capabilities, hence their gradients would be informative features.
To focus on task learning, we estimate the contribution of each instance to its corresponding task, by computing the average influence of each instance on all other in-task instances, namely *instance influence score* to help distinguish the most useful instances.
Then, we measure the difficulty of each task for LVLM to learn, by computing the average self-influence of all its contained data instances, as the *task difficulty score* to help determine the task data proportion.
Guided by the above scores, we can select the high-value instances to fine-tune the LVLM, for efficiently and effectively learning all the involved tasks within the visual instruction dataset.

To demonstrate the effectiveness of our approach, we apply our data selection method into several SOTA LVLMs and widely-used instruction datasets, and perform evaluation on eight benchmarks.
By only using the selected 15% subset from the visual instruction dataset, the fine-tuned LVLMs can achieve comparable performance to the full-data fine-tuned model, even outperforms it on four benchmarks.
As shown in Figure 1, our TIVE-8B (based on LLaVA-LLaMA3-8B) can reach the SOTA performance with much fewer instructions than SOTA methods.

## 2 Redundancy Analysis on Visual Instruction Data

In this section, we conduct an empirical study to examine: (1) whether data redundancy exists in existing visual instruction datasets, and (2) whether the degree of redundancy differs in different task instructions.

Figure: (a) MME-P
Refer to caption: x2.png

### 2.1 Analysis Setup

Given a mixture of visual instruction datasets for training LVLMs, we prune the amount of visual instructions from a certain task and then examine the performance change after fine-tuning with the adjusted instruction dataset.
In this experiment, we mainly study the used instruction dataset for training the SOTA open-source LVLM, LLaVA-1.5 (Liu et al., ).

#### Backbone Model.

We choose the LLaVA-1.5 (Liu et al., ) model after cross-modal alignment training as the backbone model (without instruction-tuning), which has been trained on more than 500k image-text pairs.
It incorporates CLIP (Radford et al., ) as the visual encoder and Vicuna-v1.5 (Chiang et al., ) as the LLM, and further leverages two linear layers for mapping the encoded visual features to the latent space of LLM.

#### Visual Instruction Dataset.

LLaVA-1.5 has been fine-tuned on a mixture of instruction datasets from different tasks. To ensure internal consistency across different tasks, only one dataset for each type of task will be selected. We select VQAv2 (Goyal et al., ) dataset for
Open-Ended Visual Question Answering (OE-VQA), A-OKVQA dataset (Schwenk et al., ) for Multi-Choice Visual Question Answering (MC-VQA), RefCOCO (Mao et al., ; Kazemzadeh et al., ) dataset for Referring Expression Comprehension (REC), LLaVA-1.0 (Liu et al., ) dataset for Visual Conversation (VC), CC3M (Sharma et al., ) dataset for Image Caption (IC), and ShareGPT (Zheng et al., ) dataset for Textual Conversation (TC). Appendix A contains details about the datasets.

To investigate the redundancy issue in visual instruction datasets, we gradually halve the number of instructions from each task, then fine-tune the backbone model on the new instruction set and finally compare the performance change.
For all experiments, we follow the default experimental configuration of LLaVA-1.5.

#### Evaluation Benchmark.

To conduct a comprehensive empirical analysis, we evaluate the fine-tuned LVLMs on the three commonly-used benchmarks: MME-P (Fu et al., ), ScienceQA (Lu et al., ), and MMBench (Liu et al., ). Detailed descriptions
of these benchmarks are available in Appendix B.

### 2.2 Results and Findings

According to the results in Figure 2, we list the main findings as follows:

First, *there exists a significant redundancy in visual instruction datasets*. We can observe that decreasing the amount of instruction data only leads to slight performance drop in most cases.
For example, reducing the number of VC would not significantly affect the model’s performance across all benchmarks, and even lead to improvement on ScienceQA using 50% of data.
It indicates that not all the used instruction datasets are indispensable.

Second, *for each task, the redundancy degree of different instruction datasets differs*.
For OE-VQA and MC-VQA, reducing their instruction number leads to relatively significant performance degradation, *e.g.* 8% on MME-P and 7% on MMBench using a pruning ratio of 87.5%, respectively.
While pruning task instructions from VC leads to minimal decline on most of the benchmarks. It indicates that different task instructions contribute to the model’s final performance differently. Therefore, it is necessary to estimate the value of each task, for helping set a more proper pruning ratio and mixing proportion for all the tasks.

## 3 Approach

In this section, we present our approach TIVE, to reduce the redundancy of visual instruction data.
Based on the findings in section 2, it is necessary to consider the contribution degree to the learning of diverse tasks during fine-tuning LVLMs.
Specially, we consider measuring both task difficulty and instance influence scores for helping select visual instruction data.
Based on the two kinds of estimated scores, we design the data selection process, to sample a small high-value visual instruction subset for efficiently and effectively fine-tuning LVLMs.
We show the details of TIVE in Figure 3.

### 3.1 Problem Formulation

The elimination of dataset redundancy aims to select a high-quality subset from a large dataset suffering the redundancy issue. The selected subset should contain relatively few but informative samples, to ensure the performance of the models trained on it.
In this work, we focus on reducing the redundancy of the visual instruction data pool $\mathcal{D}=\left\{D_{1},...,D_{n}\right\}\$, which is a mixture of multiple highly diverse instruction datasets from different tasks. Each dataset comprises a set of instruction samples, denoted as $D_{i}=\left\{s_{1},...,s_{n}\right\}\$ .
Our goal is to select a data subset $\mathcal{D}_{T}$ from the visual instruction data pool for fine-tuning LVLMs. We use $|\mathcal{D}_{T}|$ to denote the target size of the selected subset.

Specially, we select the data subset from two perspectives, with the help of a pre-learned reference model trained on the sampled small set of the visual instruction data.
First, we estimate the value of each task and rely on its difficulty to determine their proportions within the final subset $\mathcal{D}_{T}$.
Second, we estimate the value of each instance within each task $D_{i}$ to select the most useful instances for this task.

Figure: Figure 3: The illustration of our proposed approach. We utilize the gradient vectors from the LoRA parameters of the LLM, to compute the task difficulty and instance influence scores. Then, these scores are leveraged to determine the task data proportion and instance selection probability.
Refer to caption: x5.png

### 3.2 Estimating Task Difficulty and Instance Influence

In this part, we present how we measure the task difficulty and instance influence scores based on the influence on fine-tuning LVLMs.
According to the influence formulation (Pruthi et al., ), the influence of a training instance $s$ on the another instance $s^{\prime}$ can be denoted as:

$$ $\displaystyle\texttt{Inf}(s,s^{\prime})=\nabla l(s,\theta)\cdot\nabla l(s^{ \prime},\theta),$ (1) $$

where $\theta$ and $\nabla l(s,\theta)$ denote the parameters of the LVLM and their gradients, respectively.
Based on it, we devise two formulations for estimating the influence of each instance on learning its corresponding task, and measuring the difficulty of learning each task, respectively.

#### Instance Influence Estimation.

To efficiently learn each task during visual instruction tuning, we aim to obtain the contribution of each task instance, to help select a small proportion of training samples which are highly important for the task learning.
Our motivation is that if an instance has a higher positive influence on the learning of all other instances within the task, it can be regarded as a higher-value instance for helping learn the task and should be selected.
Therefore, given an instance $s$ from task set $D_{i}$. we compute the average influence of the instance on all other instances from its affiliated task, denoted as:

$$ $\displaystyle v^{i}_{s}=\frac{1}{|D_{i}|}\sum_{s^{\prime}\in D_{i}\setminus s} \frac{\nabla l(s,\theta)\cdot\nabla l(s^{\prime},\theta)}{|\nabla l(s,\theta)| |\nabla l(s^{\prime},\theta)|}.$ (2) $$

We normalize the gradients to mitigate the impact caused by abnormally large gradient values.
By this way, we can compare the influence of different instances within each task, and select the high-value ones for training.

#### Task Difficulty Estimation

According to our findings in section 2, the impact of pruning different task instruction amount also differs in the LVLM performance.
It is because not all the involved tasks are so hard that require such number of training instances, and it is promising to prune their data amount for reducing redundancy.
Therefore, we aim to measure the difficulty of all the tasks within the visual instruction dataset, to adjust their proportion in the selected subset.
Concretely, we employ the average self-influence score of all the in-task instance, to measure the task difficulty.
Self-influence is to estimate the influence of training an instance on learning itself, denoted as $\nabla l(s,\theta)\cdot\nabla l(s,\theta)$.
A higher self-influence score indicates that the instance is hard to learn (Bejan et al., ), as it leads to large gradient values.
By averaging the self-influence scores of all instances from each task, we can estimate the overall difficulty of a task as:

$$ $\displaystyle v_{i}^{t}=\frac{1}{|D_{i}|}\sum_{s\in D_{i}}\nabla l(s,\theta) \cdot\nabla l(s,\theta).$ (3) $$

Based on the task difficulty score, we can determine the proportion of all the task data within the selected visual instruction subset.
In this way, the difficult task should be assigned with a larger proportion of selected data, while the redundant data within the easy tasks should be removed more.

### 3.3 Data Subset Selection

In this section, we introduce how we obtain the gradient features and select a small data subset based on the proposed data value measurements.

#### Gradient Features Computation.

Firstly, to efficiently compute the gradient features, we train a reference model with LoRA (Hu et al., ) using a small amount of instruction data. In this way, the reference model can be warm-up to learn the visual instruction following capability, and has not overfitted to the distribution of the whole visual instruction dataset.
Thus, the gradients from the reference model can store useful information about visual instruction tuning for following influence estimation.
After training the reference model, we can obtain the gradient features through backward propagation. To save storage and computation, we follow existing work (Pruthi et al., ) to reduce feature dimensions with random projection. Such projection often preserves the inner products (Johnson, ), ensuring the effectiveness of the projected gradient features.

#### Selecting Data based on Estimated Values.

After obtaining the task-level and instance-level data values, we can select the subset from the visual instruction data pool.
First, we use the task-level value to determine the proportion for each task in the data subset.
The target data subset $\mathcal{D}_{T}=\{D_{1}^{{}^{\prime}},...,D_{n}^{{}^{\prime}}\}\$ contains the same number of task datasets as the original data pool, but changes the total amount and task proportion.
For each task subset $D_{i}^{{}^{\prime}}$, we compute its data proportion within the target data subset as $p_{i}^{{}^{\prime}}=\frac{v^{t}_{i}}{\sum_{j=1}^{n}v^{t}_{j}}$.
where $v^{t}_{i}$ is the estimated task-level value. Then, we rely on the instance-level value to sample $|D_{i}^{{}^{\prime}}|$ instances from the original visual instruction dataset.
Here, we directly employ the softmax function to map the instance-level value to a sampling weight distribution. We use a hyperparameter $\lambda$ to control the temperature of the weight distribution. For all the tasks, we sample the instances based on the above weight distribution, and merge all the datasets to compose our final selected data subset.

## 4 Experiments

**Table 1: A comparison between TIVE and other baseline approaches for data selection on several downstream benchmarks. Benchmark names are abbreviated due to space limits. MME-P: MME-Perception, MME-C: MME-Cognition, SEED-I: SEED-Bench (Image), MMB: MMBench, MMB-CN: MMBench (Chinese), SQA: ScienceQA, SQA-I: ScienceQA (Image). * indicates our reimplemented results. Rel. represents the average relative performance compared to baseline model. Improvement over best represents the relative improvement of TIVE over the best performance among other baseline approaches. Bold and underline fonts indicate the best and second best performance on the task.**
| Method | # Ins | MME-P | MME-C | SEED-I | MMB | MMB-CN | SQA | SQA-I | POPE | Rel. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 | - | 1293.8 | - | - | - | - | - | 61.0 | 85.3 | - |
| InstructBLIP-7B | 1.2M | - | - | - | 36.0 | 23.7 | - | 60.5 | - | - |
| Shikra | 5.5M | - | - | - | 58.8 | - | - | - | - | - |
| IDEFICS-80B | 1M | - | - | - | 54.5 | 38.1 | - | - | - | - |
| Qwen-VL | 50M | - | - | - | 38.2 | 7.4 | - | 67.1 | - | - |
| Qwen-VL-Chat | 50M | 1487.5 | - | - | 60.6 | 56.7 | - | 68.2 | - | - |
| InstructionGPT-4 | 0.2K | 463.3 | - | - | 31.4 | - | - | - | - | - |
| SELF-FILTER | 25K | 955.6 | - | 47.5 | 38.5 | - | 59.4 | - | - | - |
| Backbone model |  |  |  |  |  |  |  |  |  |  |
| LLaVA-1.5 | 665K | 1510.7 | 311.9* | 66.1 | 64.3 | 58.3 | 69.4* | 66.8 | 85.9 | 100.0% |
| Our experiment |  |  |  |  |  |  |  |  |  |  |
| Random | 100K | 1386.5 | 271.3 | 61.9 | 61.8 | 54.5 | 69.8 | 68.4 | 83.9 | 95.2% |
| Length | 100K | 1413.0 | 266.1 | 61.2 | 59.3 | 53.9 | 71.1 | 69.2 | 83.3 | 94.8% |
| Perplexity | 100K | 1393.3 | 260.7 | 61.3 | 62.3 | 55.0 | 70.5 | 67.9 | 83.6 | 94.9% |
| GraNd | 100K | 1400.5 | 287.1 | 62.3 | 62.9 | 54.3 | 71.4 | 68.4 | 82.5 | 96.3% |
| EL2N | 100K | 1356.5 | 294.7 | 61.9 | 61.6 | 56.1 | 70.2 | 66.2 | 84.6 | 95.5% |
| TIVE (ours) | 100K | 1433.0 | 322.1 | 63.2 | 65.0 | 58.2 | 72.2 | 70.6 | 85.6 | 100.3% |
| Improve over best | - | 1.4% | 9.3% | 1.4% | 3.3% | 3.7% | 1.1% | 2.0% | 1.2% | 4.0% |

### 4.1 Experiment Setup

We conduct extensive experiments on TIVE across various models and datasets. The models include LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-Phi-3-4B, and LLaVA-LLaMA3-8B. The datasets include LLaVA-1.5 instructions, SVIT-Mix (Zhao et al., ) instructions and, Mini-Gemini (Li et al., ) instructions. Since LLaVA-1.5-7B and LLaVA-1.5-13B takes Vicuna-7B and Vicuna-13B as their LLM backbone, we denote these two models as LLaVA-Vicuna-7B and LLaVA-Vicuna-13B in some experiments. More information about the training datasets, evaluation benchmarks, and implementation details are presented in Appendix A, Appendix B, and Appendix D, respectively.

### 4.2 Baselines

We compare our methods with several baselines for data selection: (1) *Random Selection* selects data randomly; (2) *Instruction Length* utilizes length of instruction to determine the importance of an instruction sample; (3) *Perplexity* computes the perplexity score of an instruction sample to measure its importance; (4) *GraNd* (Paul et al., ) measures the importance of each sample by the L2-norm of the gradient caused by each sample; (5) *EL2N* (Paul et al., ) measures the importance of each sample by the L2-norm of the error vector of each sample.
The EL2N scores are primarily used for estimating sample importance in image classification tasks. To adapt it for visual instruction tuning, we compute the error vector for each token in each sample, and then compute the final EL2N score by averaging norms of all error vectors.

### 4.3 Main Results

We present the comparison of TIVE with other baseline methods on LLaVA-1.5 in Table 1, the results of TIVE across different LVLMs in  Table 2, and the results of TIVE across different instruction datasets in  Table 3.
We present analyses of the results as follows:

#### Comparison of TIVE with other baseline methods.

In  Table 1, we compare TIVE with several baseline methods on 8 benchmarks. First, we observe that the traditional data selection approaches (GraNd and EL2N) perform slightly better than random selection. A possible reason is that these approaches indeed select valuable data, but are also more vulnerable to the data noise, resulting in a limited improvement. For the data selection approaches used in LLM instruction tuning (Length and Perplexity), the performances across several benchmarks are even worse than random selection. We discover that these approaches mostly focus on selecting samples which have a high influence on improving the model’s generation ability, which leads to minor enhancement on the model’s ability on visual understanding. It is clear that our approach significantly outperforms all other baselines and achieves consistently promising results across all benchmarks under a limited data setting. With only *15% of the instruction data*, our approach can achieve *100.3%* average performance on all benchmarks compared to the LLaVA-1.5 model, even surpass the performance of LLaVA-1.5 in four benchmarks. These results show that our proposed approach can effectively address the issues of data redundancy within LLaVA-1.5 instructions.

**Table 2: The performance of TIVE across different LVLMs. # Samp indicates the sampling ratio.**
| Model | Method | # Samp | MME-P | MME-C | SEED-I | MMB | SQA | SQA-I | POPE | Rel. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-<br>Vicuna-7B | - | 100% | 1510.7 | 311.9 | 66.1 | 64.3 | 69.4 | 66.8 | 85.9 | 100.0% |
|  | Random | 15% | 1386.5 | 271.3 | 61.9 | 61.8 | 69.8 | 68.4 | 83.9 | 95.2% |
|  | TIVE | 15% | 1433.0 | 322.1 | 63.2 | 65.0 | 72.0 | 70.6 | 85.6 | 100.3% |
|  | TIVE | 30% | 1467.2 | 309.8 | 64.4 | 66.5 | 71.4 | 70.1 | 85.2 | 100.6% |
| LLaVA-<br>Vicuna-13B | - | 100% | 1531.3 | 295.4 | 68.2 | 67.7 | 74.4 | 71.6 | 85.9 | 100.0% |
|  | Random | 15% | 1456.6 | 307.1 | 63.4 | 64.9 | 73.5 | 69.4 | 85.5 | 96.6% |
|  | TIVE | 15% | 1502.9 | 336.1 | 65.3 | 66.1 | 74.5 | 72.2 | 86.3 | 100.5% |
|  | TIVE | 30% | 1545.4 | 298.6 | 65.6 | 68.8 | 74.2 | 72.2 | 86.5 | 100.1% |
| LLaVA-<br>Phi-3-4B | - | 100% | 1440.8 | 301.6 | 66.7 | 67.9 | 81.0 | 73.6 | 85.1 | 100.0% |
|  | Random | 15% | 1329.1 | 295.4 | 63.1 | 64.0 | 80.2 | 71.2 | 82.8 | 95.7% |
|  | TIVE | 15% | 1386.9 | 306.4 | 63.9 | 66.0 | 81.2 | 73.5 | 84.1 | 98.0% |
|  | TIVE | 30% | 1425.0 | 338.2 | 65.1 | 68.5 | 81.8 | 74.3 | 83.8 | 100.9% |
| LLaVA-<br>LLaMA3-8B | - | 100% | 1569.4 | 338.6 | 68.8 | 71.2 | 77.2 | 73.5 | 85.7 | 100.0% |
|  | Random | 15% | 1495.8 | 318.2 | 65.2 | 67.9 | 80.4 | 75.4 | 83.3 | 97.4% |
|  | TIVE | 15% | 1511.4 | 331.1 | 67.4 | 69.8 | 81.6 | 75.7 | 84.9 | 99.5% |
|  | TIVE | 30% | 1560.3 | 322.9 | 68.1 | 72.0 | 80.5 | 74.1 | 84.6 | 100.2% |

#### Performance of TIVE across different LVLMs.

Table 2 shows the performance of TIVE on different LVLMs. We find that under the same sampling ratio (15%), our approach significantly outperforms the random baseline across all LVLMs on all benchmarks, achieving an average improvement of at least 2.3%. Simultaneously, when the sampling ratio is increased to 30%, our approach achieves better average performance than full data performance across all models, proving that TIVE successfully eliminates redundancy in visual instruction data and is effective across different LVLMs. Furthermore, we discover that under a low sampling ratio (15%), LLaVA-Vicuna-13B achieves the best average relative performance (100.5%), while LLaVA-Phi-3-4B achieves the worst (98.0%). This indicates that LVLMs with a larger LLM backbone have a relatively better average performance under less data, which is consistent with the results for LLM on language instruction tuning scenarios.

#### Performance of TIVE across different instruction datasets.

We present the results of TIVE on two other instruction datasets in Table 3. We observe that TIVE remains effective on different instruction datasets. On the SVIT-Mix dataset, it significantly outperforms other baselines in five out of six benchmarks, and surpasses the full data performance in three out of the six benchmarks. On the Mini-Gemini dataset, TIVE shows more advantage over the other baseline methods, and the average performance of TIVE on these benchmarks is better than the full data performance. Considering that the Mini-Gemini dataset has a larger number of instructions, TIVE may be more effective at eliminating redundancy when dealing with a substantial amount of instructions. These results demonstrate the effectiveness of TIVE across different instruction datasets.

**Table 3: The performance of TIVE across different instruction datasets.**
| Method | # Samp | MME-P | MME-C | MMB | SQA | SQA-I | POPE | Rel. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-1.5 |  |  |  |  |  |  |  |  |
| Baseline | 100% | 1510.7 | 311.9 | 64.3 | 69.4 | 66.8 | 85.9 | 100.0% |
| TIVE | 15% | 1433.0 | 322.1 | 65.0 | 72.0 | 70.6 | 85.6 | 100.3% |
| Random | 15% | 1386.5 | 271.3 | 61.8 | 69.8 | 68.4 | 83.9 | 95.2% |
| Length | 15% | 1413.0 | 266.1 | 59.3 | 71.1 | 69.2 | 83.3 | 94.8% |
| SVIT-Mix |  |  |  |  |  |  |  |  |
| Baseline | 100% | 1443.5 | 306.1 | 67.3 | 70.2 | 68.0 | 85.3 | 100.0% |
| TIVE | 15% | 1391.7 | 306.8 | 65.8 | 72.3 | 71.2 | 84.3 | 99.8% |
| Random | 15% | 1402.9 | 288.8 | 60.2 | 69.6 | 65.7 | 83.8 | 96.0% |
| Length | 15% | 1366.5 | 301.1 | 61.3 | 70.2 | 67.1 | 84.2 | 96.8% |
| Mini-Gemini |  |  |  |  |  |  |  |  |
| Baseline | 100% | 1538.4 | 324.9 | 68.1 | 72.0 | 69.9 | 85.1 | 100.0% |
| TIVE | 15% | 1506.9 | 345.4 | 67.9 | 72.6 | 71.1 | 85.4 | 101.2% |
| Random | 15% | 1404.8 | 305.4 | 62.2 | 71.1 | 69.2 | 84.9 | 95.7% |
| Length | 15% | 1403.3 | 313.2 | 62.1 | 70.3 | 67.9 | 83.7 | 95.4% |

### 4.4 More Detailed Analysis

#### Effectiveness of Data Value Measurements.

We conduct a series of ablation studies to validate the efficacy of our proposed data value on both levels. Initially, to verify the effectiveness of task value estimation, we standardize the weight of all tasks to 1 and then conduct data selection based on instance influence only. Subsequently, to verify the effectiveness of instance value estimation, we calculate task weights based on task difficulty, but select instances within task instructions randomly. We present our results in Table 4.

We discover that data selection based on task value alone or instance value alone can both boost the performance on all three benchmarks. And selecting data based on both instance influence and task difficulty achieve the best results than all other baseline methods on all of the benchmarks, which proves the effectiveness of both values.

**Table 4: The ablation of the effectiveness of different data values.**
| Benchmarks | Ours (Both) | $\neg$ Instance-level | $\neg$ Task-level | Neither |
| --- | --- | --- | --- | --- |
| SQA-I | 70.6 | 69.8 | 68.2 | 68.4 |
| MMB | 65.0 | 63.7 | 62.9 | 62.5 |
| SEED-I | 63.2 | 62.7 | 62.9 | 62.2 |

#### Model Performance with Different Sampling ratio.

To explore the trend of model performance as data size changes, we conduct a series of experiments with different data sampling ratio. In all experiments, we maintain consistency in the data selection approach as well as model training configuration. Our experimental results are presented in 4(a).

As we can observe, the model’s performance continuously improves with the increasing amount of data yet, the trend of this enhancement varies across different tasks. The model’s performance on MME-P rapidly increases as the data size increases. However, on MMBench and SQA-I, the model’s performance increases at first and then stabilizes. A possible reason for this is that MME-P tends to evaluate the model’s ability on visual recognition while the other two benchmarks focus on the model’s general reasoning capability. Furthermore, We find that the model can maintain a certain level of performance under the minimal data size, indicating that models can acquire basic capability for downstream tasks even with a minimal amount of data.

**Table 5: The ablation of different warm-up data size. # Avg indicates the average performance on the benchmarks. We normalize the scores on MME-P and MME-C for computing average performance.**
| # Samp | MME-P | MME-C | SEED-I | MMB | SQA | SQA-I | POPE | # Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2% | 1424.9 | 284.6 | 63.1 | 65.0 | 72.1 | 70.4 | 85.4 | 59.6 |
| 4% | 1441.9 | 321.4 | 63.4 | 64.6 | 71.3 | 69.3 | 85.3 | 59.6 |
| 8% | 1433.0 | 322.1 | 63.2 | 65.1 | 72.2 | 70.6 | 85.6 | 59.9 |
| 16% | 1434.3 | 281.8 | 63.4 | 64.8 | 72.3 | 70.1 | 84.4 | 59.5 |
| 32% | 1431.3 | 317.1 | 63.1 | 64.7 | 72.0 | 70.1 | 85.2 | 59.7 |

#### Influence of Different Warm-up Data Size.

We design a series of experiments to investigate the influence of different warm-up data size on the performance of TIVE. We simply change the sampling ratio for warm-up data and maintain consistency in other parts of TIVE selection. The results are presented in Table 5.

As we can observe, as the sampling ratio increases, the model performance initially exhibits a slight increase trend. Then, it begins to oscillate when the sampling ratio reaches 8%. Even so, the performance differences between various sampling ratios are quite minimal. This implies that a reference model trained with a minimal amount of warm-up data is already effective for TIVE, making the selection process more efficient.

#### Influence of Different Hyperparameter λ 𝜆 \lambda italic_λ .

To achieve a balanced choice between data effectiveness and data diversity, we introduce a hyperparameter $\lambda$ to control the temperature of weight distribution. We study the influence of different $\lambda$ on the quality of final selected data. We set $\lambda$ to different values and evaluate the model’s performance on downstream benchmarks.

The evaluation results on MME-P, MMBench and SQA-I are shown in 4(b). We can observe a consistent slight increase in the model’s performance on MME-P benchmark as $\lambda$ increases, indicating that the MME-P benchmark is highly sensitive to instruction diversity, which is consistent with previous conclusions. On the other hand, the performance on SQA-I and MMBench initially increases with the escalation of $\lambda$, then shows a decline once the $\lambda$ reaches $1e3$. The results demonstrate that our approach with $\lambda=1e3$ is an optimal data selection strategy that balances data effectiveness and data diversity for the model’s consistent optimal performance across all downstream tasks.

Figure: (a) The ablation of different sampling ratio.
Refer to caption: x6.png

## 5 Related Work

#### Visual Instruction Tuning.

Visual instruction tuning is a crucial part of the construction of LVLMs, which aims to enhance the model’s ability on instruction following. The collection of visual instructions is essential for visual instruction tuning. Early studies often employ LLMs to synthesize visual instructions. LVLMs trained on these instructions demonstrate promising capabilities in visual conversation and instruction following, but fail to achieve satisfactory performance on academic benchmark (Goyal et al., ; Schwenk et al., ; Marino et al., ). Subsequent studies (Liu et al., ; Luo et al., ; Dai et al., ) have usually mixed the synthesized visual instructions and instructions from existing academic datasets together as the final instruction data. LVLMs trained on these mixtures of instructions demonstrate exceptional performance in both understanding and generation scenarios. Despite the success, these efforts solely combine all instructions in a simple way, neglecting the potential redundancy within the instructions from different tasks. We investigate the redundancy in existing visual instruction datasets and propose a measurement for data value based on instance influence and task difficulty to reduce redundancy.

#### Data Selection for Instruction Tuning.

With the advancement of LLMs, the significance of data selection has become increasingly prominent due to the high training costs. As for instruction tuning, LIMA (Zhou et al., ) is the first to demonstrate that instruction tuning can be accomplished with only a small amount of data. Chen et al. () further explores the potential of low data usage in task-specific models. Subsequent efforts focus on estimating the importance of an instruction sample. The importance can be estimated based on certain prior characteristics (*e.g.* length, complexity, diversity) (Liu et al., ; Cao et al., ), with the assistance of language models (Jain et al., ; Liu et al., ; Li et al., ), by human efforts (Zhuo et al., ; Muennighoff et al., ), or using the gradient-based influence estimation on the validation set of the target benchmark (Xia et al., ). Compared to the data selection approach for language instruction tuning, our approach doesn’t only rely on prior characteristics of texts, but considers the importance of visual instructions from a holistic perspective of both image and text. Compared to LESS (Xia et al., ), our approach doesn’t require data from downstream benchmark, thereby achieving better generalization ability.

#### Data Selection for Visual Instruction Tuning.

Fewer studies have been focusing on data-efficient visual instruction tuning. To the best of our knowledge, there are only two studies currently conducted in this area. Among these studies, InstructionGPT-4 (Wei et al., ) selects high-quality instructions based on several metrics designed in their studies and SELF-FILTER (Chen et al., ) proposes selecting instruction data with higher diversity and difficulty by training a score-net. Compared to these studies, We are the first to study data selection for a highly complex mixture of visual task instructions, which provides much better results than the candidate datasets from these studies. To handle such complex visual instructions, we propose a gradient-based approach to estimate data value for efficient and effective task learning. With our approach, we accomplish better results compared to previous studies on data selection for visual instruction tuning with our selected data.

## 6 Conclusion

In this work, we focus on the redundancy issue within a mixture of visual instruction datasets that have been widely used for fine-tuning LVLMs. Through our empirical studies, we find that a significant redundancy exists in the mixed visual instruction datasets, with varying redundancy degrees across different task instructions. To eliminate redundancy, we design a novel method namely TIVE, which first estimates data value based on instance influence and task difficulty, then determines the instruction task proportion and selects representative instances to compose a smaller visual instruction subset for training.
Experimental results indicate that, with the help of our data selection method, using only about 15% data can achieve comparable performance as the full-data fine-tuned model across eight benchmarks, even surpassing it on some of the benchmarks.

## Appendix A Training Dataset

### A.1 Training Dataset for empirical analysis

The visual instruction dataset used for our empirical analysis is a subset of the original LLaVA-1.5 instructions. We select one dataset for each type of task. The details of these selected datasets are as followed:

- •
Open-Ended Visual Question Answering (OE-VQA): it requires a model to generate natural language answers without predefined options. We select VQAv2 (Goyal et al., ) since it’s one of the most commonly-used OE-VQA dataset.
- •
Multi-Choice Visual Question Answering (MC-VQA): it also requires a model to answer visual questions, but only selects the answer from the provided candidate choices. We select the A-OKVQA dataset (Schwenk et al., ).
- •
Referring Expression Comprehension (REC): it requires a model to generate the regional description of the given object or select the correct object based on the given description. We select RefCOCO dataset (Mao et al., ; Kazemzadeh et al., ).
- •
Visual Conversation (VC): it requires a model to generate long conversations based on visual content. We select the VC data from instructions of LLaVA-1.0 (Liu et al., ).
- •
Image Caption (IC): it requires a model to provide an description of the given image. We select CC3M dataset (Sharma et al., ) as it is already used for cross-modal alignment training of LLaVA-1.5 (Liu et al., ).
- •
Textual Conversation (TC): it requires the model to generate conversation in a text-only setting. We select ShareGPT (Zheng et al., ), as it has been widely used in training LLMs.

The statistics of our base dataset are presented in Table 6.

**Table 6: Statistics of base training data for empirical studies.**
| Task | MC-VQA | OE-VQA | REC | VC | Caption | TC |
| --- | --- | --- | --- | --- | --- | --- |
| Numbers. | 60K | 80K | 120K | 40K | 100K | 40K |

### A.2 Training Dataset for main experiments

We conduct experiments on three datasets. In the experiment of evaluating TIVE against other baseline methods, we adopt the LLaVA-1.5 instruction datasets. In the experiment of evaluating the transferability of TIVE across different datasets, we additionally use the Mini-Gemini and SVIT-Mix instruction datasets. LLaVA-1.5 and SVIT-Mix contains over 600K instructions and Mini-Gemini contains over 1.4M instructions. All these datasets encompass at least nine sub-task datasets. We exclude the caption data from the selection process since it’s already been trained during the LLaVA’s pre-training stage.

## Appendix B Evaluation Benchmarks

To comprehensively evaluate the efficacy of our approach, we evaluate TIVE across various benchmarks. The details of these benchmarks are as followed:

- •
MME: (Fu et al., ) it evaluates LVLM’s reasoning ability from the two dimensions of perception and cognition. Each instance in MME includes an image and two binary questions. We evaluate TIVE on both splits.
- •
MMBench: (Liu et al., ) it is a systematically-constructed dataset for evaluating the capacity of LVLMs. It encompasses an evaluation of 20 fine-grained capabilities of LVLMs. The evaluation is performed through its official website. We evaluate TIVE on both english split and chinese split to test its multilingual capability.
- •
SEED-Bench: (Li et al., ) it develops a comprehensive set of multimodal evaluation tasks across twelve dimensions with the assistence of GPT-4. SEED-Bench encompasses assessments of both image and video understanding capabilities. In our experiments, we only utilize the image benchmark of SEED-Bench.
- •
ScienceQA: (Lu et al., ) it is a benchmark constructed around various science topics, encompassing both pure text-based questions and image-related text questions. In our experiment, we assess ScienceQA under both multi-modal and uni-modal setting.
- •
POPE: (Li et al., )
it designs a polling-based query approach for the evaluation of object hallucination. It contains 3000 binary questions and support four evaluation metrics. In our experiment, we report the results of accuracy.

For simplicity, we only adopt MME-Perception, MMBench, and ScienceQA-Image during our empirical analysis.

## Appendix C Baseline Models

We compare TIVE with other baseline models in Figure 1 and Table 1. These models include: BLIP-2 (Li et al., ), InstructBLIP-7B (Dai et al., ), Shikra (Chen et al., ), IDEFICS-80B (Laurençon et al., ), Qwen-VL (Bai et al., ), Qwen-VL-Chapt (Bai et al., ), InstructionGPT-4 (Wei et al., ), SELF-FILTER (Chen et al., ), Yi-VL-34B (Young et al., ), LLaVA-Next-8B (Liu et al., ), Libra (Xu et al., ), and DeepSeek-VL (Lu et al., ).

## Appendix D Implementation Details

We utilize Bunny (He et al., ) to construct LVLM with different LLM backbone.
We follow the training settings of LLaVA-1.5 across all
experiments. During fine-tuning, the learning rate is set to 2e-5 and the batch
size is set to 16. All models are trained for one epochs. The training settings
for reference models are the same as the previous settings. For all experiments, we sample 8% of the total instructions and train the reference model on the sampled data for one epoch. We provide a detailed description of our approach in Algorithm 1 and Algorithm 2.

Figure: Algorithm 1 Estimating Task Difficulty and Instance Influence.

Figure: Algorithm 2 Data Selection Based on Data Value.

## Appendix E Additional Experiment Details

### E.1 Calculated Task Proportion

We present the task proportion calculated via the task-level data value for LLaVA-1.5 instructions in Table 7.

We find that tasks which require precise answers to visually related questions have a relatively higher proportions in the final selected subset. The potential reason is that these tasks often require models to possess a higher level of visual reasoning ability, which contributes more to the enhancement of the model’s performance on downstream tasks. Furthermore, task-level values computed by TIVE are consistent with the findings presented in section 2, proving the effectiveness of our method.

**Table 7: Statistics of calculated task proportion.**
| Task | VQAv2 | GQA | OCRVQA | A-OKVQA | VG | RefCOCO | ShareGPT | LLaVA-1.0 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Proportion | 20.0% | 12.7% | 12.8% | 35.3% | 4.1% | 7.3% | 2.2% | 5.5% |

### E.2 Scaling Instruction Numbers across all models

To further explore the trend of model performance as data size changes, we conduct the experiments of scaling selected instructions on more models. The data selection approach is still consistent with the previous experiments. The results are presented in Figure 5.

Figure: Figure 5: The experiment of scaling instruction number on different LVLMs.
Refer to caption: x8.png

We find that the performance of different models follow a similar trend with the increase in instruction number. When the sampling rate is low (less than 15%), the performance of all models significantly improves with the increase in instruction number. However, when the sampling rate reaches 15%, the model’s performance gradually stabilizes, scaling instruction number will have minimal effect to the model’s performance. Meanwhile, when the sampling rate exceeds 60%, increasing the number of instructions can even have a negative impact on some models. These experimental results indicate that visual instruction redundancy is clearly present in different models and can potentially have a significant side effect.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weakness part. The authors are encouraged to answer such questions.
1. Regarding weakness 1, the authors are encouraged to provide the actual time cost for TIVE and fair comparisons with full training for LLaVA-1.5.
2. The accuracy drop for TIVE is significant compared with the baseline. 
3. The authors could explain the design of gradient inference in detail and show the relations with the original training data and the backbone.
Therefore, I recommend rejecting this manuscript in its current version. I would like to increase my score if the authors could address the issues above or provide more results against approaches with similar targets.
soundness: 2
strengths: 1. The experiments are complete across various MLLM backbones, including Vicuna, Phi, and LLaMA3, and architectures, including LLaVA-1.5, SVIT-Mix, and Mini-Gemini. The authors also show comparisons with baselines / advanced MLLMs. 
2. The performance meets the full baselines with only 10% to 30% training data, which shows the effectiveness of TIVE.
3. The paper is well written and the formulation periods are clear.
summary: The paper concentrates on reducing the data redundancy of instruction-following MLLMs. The authors show that pruning a certain ratio of specific training data has a slight influence on the overall accuracy. Based on the observation, the authors present a data selection approach named TIVE. The data selection strategy is based on estimating task difficulty and instance influence. Then the gradient features are used for selection. The authors integrate the approach on several MLLM backbones including LLaVA-1.5, LLaVA-LLaMA3, Mini-Gemini, etc. Experiments on multimodal benchmarks show an increase compared with random selection.
weaknesses: 1. The main weakness lies in the design of the approach, especially regarding the computation costs. In my recognition, the inference operation based on the gradients and other selection operations are costly, even meets the original training cost. This makes the contribution of the pruning method weak. 
2. The selection based on gradients is a posterior probability, which means choosing the hard samples as prior knowledge. This may be unfair for the comparisons against baselines.
3. The overall performance in Table 1 against backbone models is weak, only shows significant improvement on the SciQA benchmark, and gains accuracy drop or fair on other benchmarks (may be due to experiment uncertainty). This may mean the selection approach is sub-optimal.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. In algorithm 1, the task influence is calculated in a nested for loop, with a overall complexity $O(|D_i|^2)$ for each task. A question is, could the author first use one pass to aggregate the average of the normalized gradients and then use another pass to calculate the score? This will reduce the complexity to linear. Will this cause numerical instability or it doesn't? Originally, was the gradients stored or re-computed?
2. Are the influence scores' gradient of a sample computed over all tokens in it and do average, or only on outputs part? 
3. In the line 301, $\lambda$ is introduced as "We use a hyperparameter $\lambda$ to control the temperature of the weight distribution". However, how actually it is used is presented in Line 871 in appendix. The ablation of $\lambda$ appears before readers know how actually it is used. The ordering of this part needs further consideration.
soundness: 3
strengths: The observation of the dataset redundancy problem aligns with the community's observations.
The proposed TIVE method sounds reasonable.
The authors conduct extensive experiments with detailed analysis to demonstrate the effectiveness of the method and its components.
summary: This work studied the redundancy problem in the visual instruction tuning dataset for LVLMs. It proposed a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. HIVE can effectively reduce the training data size of different VLM instruction tuning datasets across different models without compromising the overall performance.
weaknesses: 1. Though the experiments are comprehensive, there are several points to further discuss or clarify in the method. See questions.
2. The authors need to provide a further discussion on the overall cost of the method: as TIVE needs the reference model trained with warmup data, the selection of TIVE is generally model-specific. TIVE needs to compute the LoRA gradient over all samples in the pool, then this cost is close to training on all of the data with LoRA. Tuning the hyper-parameters of HIVE would give another dimension of complexity if there are no default hyper-parameters. From this perspective, this method may fail to reduce the overall training costs. If so, it needs to target improving the final performance (without insisting on 15% of data) and discuss more about how to achieve this (what proportion of data is the best?). If not, the corresponding additional cost should be discussed.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to weakness part
soundness: 2
strengths: 1.  The paper introduces a well-justified and innovative method, TIVE, that addresses data redundancy in visual instruction datasets for LVLMs. 
2. The motivation for addressing redundancy is well explained, and the proposed solution is logically developed based on detailed empirical findings.
3. The authors provide thorough empirical evidence demonstrating the existence of redundancy within current visual instruction datasets, supporting the motivation for their approach.
summary: The paper investigates redundancy within visual instruction datasets used for fine-tuning large vision-language models (LVLMs). It presents an empirical analysis which reveals that reducing the instruction data does not significantly impact model performance, suggesting the potential for data reduction. To address this, the authors propose TIVE, a novel method that selects high-value data based on task difficulty and instance influence using gradient-based techniques. Experiments demonstrate that TIVE can achieve comparable or even superior results to full-data models while using only 15% of the dataset. The proposed method provides a more efficient approach to visual instruction tuning by minimizing training costs and redundancy.
weaknesses: 1. The paper does not sufficiently discuss the potential limitations of the TIVE approach, such as its scalability to even larger datasets or its applicability to different types of multimodal tasks.
2. I have some concerns regarding the data selection approach. In the earlier stages of machine learning, data and feature selection were widely popular. However, recent trends show that using larger models with bigger datasets tends to yield remarkable generalization capabilities. I hope the authors can address this concern in their rebuttal.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Is there any example analysis to see how the samples that were filtered out compare to those that were retained?
soundness: 2
strengths: 1. The paper addresses a crucial problem in the field of LVLMs data redundancy in visual instruction tuning. This is an important issue as it can lead to increased training costs and potential overfitting.
2. The authors conduct a series of experiments to demonstrate the existence of data redundancy and the effectiveness of their proposed method. The analysis includes pruning the amount of instructions from different tasks and evaluating the performance on various benchmarks, providing strong evidence for their claims.
3. The proposed TIVE method is innovative, considering both instance influence and task difficulty scores for data selection. This holistic approach takes into account the characteristics of different tasks and instances, making it more effective than traditional data selection methods.
summary: This paper focuses on the issue of data redundancy in visual instruction tuning for building large vision language models (LVLMs). Through empirical studies, it reveals significant redundancy within visual instruction datasets and proposes a high-value data selection approach named TIVE. TIVE estimates instance influence scores and task difficulty scores based on gradient-based influence functions to select a representative subset of data, reducing training costs while achieving comparable performance to full-data fine-tuned models on multiple benchmarks.
weaknesses: 1. Lack of in-depth analysis of task characteristics: Although the paper considers task difficulty in its method, it could provide a more in-depth analysis of the characteristics of different tasks and how they affect data redundancy and model performance.
2. Why does Instance Influence consider the gradients of other samples (Formula 1)? Is it for the normalized comparison of all samples in Formula 2?
3. A significant limitation of the current study is the lack of ablation experiments to evaluate the relative importance of instance selection versus task selection. The paper would be strengthened by comparing the proposed method against two baseline scenarios: one using only instance influence for global selection without task-level grouping, and another applying task selection first followed by random sampling or established methods like GraNd within each selected task. Such comparisons would help quantify the individual contributions of these two selection mechanisms and provide stronger justification for the proposed two-stage approach.

Meta-review: ### **Summary**
This paper introduces **TIVE**, a method designed to reduce redundancy in visual instruction datasets by leveraging **task difficulty** and **instance influence** scores calculated via gradient-based techniques. The approach is evaluated on multiple vision-language models (LVLMs) and benchmarks, claiming comparable or superior performance to full-data fine-tuning with only 15% of the dataset.

### **Strengths**
1. **Problem Relevance**:
   - The issue of data redundancy in visual instruction datasets is a timely and important topic in scaling LVLMs.
2. **Empirical Validation**:
   - The authors provide extensive experimental results across various LVLM architectures and benchmarks.
3. **Efficiency Improvements**:
   - The introduction of **TIVE-efficient** shows an attempt to reduce computational costs, which is a positive step toward scalability.

### **Weaknesses**
1. **Insufficient Novelty and Impact**:
   - The approach, while novel in combining task- and instance-level selection, does not offer significant improvements over baseline methods.
   - In some benchmarks, performance gains are marginal or nonexistent, raising concerns about the practical utility of such aggressive data pruning.

2. **Scalability and Practicality**:
   - The computational overhead of TIVE remains high, especially for larger datasets, even with TIVE-efficient. The applicability to datasets larger than those tested (e.g., >10M samples) is unclear and largely speculative.
   - The reliance on gradient-based influence functions makes the approach resource-intensive, potentially negating the claimed efficiency benefits in realistic scenarios.

3. **Limited Task Insights**:
   - Although the method incorporates task difficulty, the paper does not provide a deep understanding of how task characteristics influence redundancy or the success of TIVE.
   - The empirical analysis lacks qualitative insights into the differences between selected and filtered data, leaving questions about the robustness of the selection criteria.

4. **Methodological Concerns**:
   - The proposed method relies heavily on prior knowledge of task labels and task-specific gradients, which may not generalize to more diverse or unlabeled datasets.
   - Comparisons to baselines are not entirely convincing, as the method’s advantage appears to depend on specific configurations and datasets.

5. **Presentation Issues**:
   - Key concepts (e.g., gradient-based influence, task difficulty) are not clearly explained, making the methodology difficult to follow.
   - The ablation studies, while improved during the rebuttal, do not provide enough evidence to justify the two-stage approach over simpler alternatives.

### **Recommendation**
The proposed solution lacks sufficient novelty, scalability, and practical impact to justify acceptance. The marginal improvements and methodological limitations do not demonstrate a compelling advancement over existing methods. Addressing these issues in future work, particularly by improving scalability and providing deeper insights into task-specific effects, would strengthen the paper’s contribution.

Decision: Reject