Mosaic-IT: Free Compositional Data Augmentation Improves Instruction Tuning

Authors: Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, YuPeng Hou, Fuxiao Liu, Tianyi Zhou
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Large Language Model, Instruction Tuning, Supervised Finetuning, Data Augmentaion

Abstract:
Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data augmentation method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and a $80\%$ reduction in training costs compared with original instruction tuning. Our codes and data are available at https://anonymous.4open.science/r/mosaic-955B.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729922876996

thread_type:
official_review

<thread_object>
creator: reviewer_7wng

<content>
summary:
This paper proposes Mosaic-IT, a data augmentation method for the instruction following. It proposes Primary and advanced mosaic strategies. It also includes format, permute, and mark out.

strengths:
The paper deploys the method into several evaluation benchmarks and different model structures. It also includes several analyses to study the method's aspects.

weaknesses:
1.	The core problem of the proposed method is the lack of a detailed explanation of the core reasons for the proposed method. It does not provide justifiable and experimental explanations for the effectiveness of the proposed method. The primary motivation of the proposed method should be further cleared here. 
2.	From the experiments, it seems that the proposed method does not improve the multi-turn data for MT-Bench. Would there be any explanations for this?
3.	Many instruction-following methods and literatures focus on data augmentation. There are no comparisons with those baselines. In addition, how would mask methods be different from the other dropout, etc. methods?

questions:
1. Would there be any justifiable and experimental explanations for demonstrating the method's effectiveness?

2. Would it be possible to show more experiments on multi-turn benchmarks?

3. For the comparison baselines, would it be possible to add more baselines related to data augmentation for instruction turning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647858063

thread_type:
official_review

<thread_object>
creator: reviewer_cpom

<content>
summary:
This paper introduces a data augmentation method, Mosaic-IT, for instruction-tuning large language models (LLMs) without human or model dependency. Unlike traditional approaches that rely on human intervention or teacher models to generate instruction-response pairs, the proposed method works by combining existing instructions into composite multi-instruction samples. They propose four ways to do the composition - primary, format, permute and maskout. By doing so, the paper shows that LLMs trained with this method develop a higher level of instruction-following capacity and format adherence. The proposed method, which reduces training time by approximately 80%, holds promise as a scalable solution for instruction tuning without extensive resource requirements.

strengths:
1. The paper is well-structured, progressing logically from the motivation behind Mosaic-IT to the methodology, followed by experiments and results. Each section builds on the last, making the paper easy to follow and understand.
2. The figures do a great job of clearly summarizing the idea. 
3. The experiments are comprehensive for the scope the paper setup - they have explored different datasets and model families and explored different sampling procedures for the composition

weaknesses:
1. The paper lacks a theoretical basis for why random concatenation should improve instruction-following abilities; structured or semantically grouped concatenations could offer further insights.
2. Randomly concatenated instructions may introduce noise, potentially impacting training stability. An analysis of this effect on model perplexity would strengthen the work.

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678849031

thread_type:
official_review

<thread_object>
creator: reviewer_mjbt

<content>
summary:
This paper studies instruction-tuning methods in LLMs by augmenting training data with three different templates, Format, Permute, and Maskout strategies. These techniques may reduce the over-fitting or memorization. The proposed method, Mosaic-IT, achieves consistent performance improvements over various benchmarks.

strengths:
- [S1] The experimental results seems to be solid by demonstrating consistent improvement against a no-augmentation baseline.

weaknesses:
- [W1] The techniques to prevent over-fitting and memorization by preparing various formatted templates and order randomization has been well studied and widely known approach; such as pioneering work of Instruction Tuning (Wei et al., 2021, Flan-T5 paper). From that time, the input/output pairs for instruction tuning are not always fixed and dynamically randomized. Considering these literatures, I think this paper is a kind of re-invention of those techniques, and the technical novelty and contribution seems to be limited.
- [W2] Figure 3 is unclear to me. Could you clarify what is a “mixture count”? While “Fix” strategy is adopted, the number of “mixture count” seems to be distributed among 1-10 (not fixed?). Why do you use Uniform as a default despite its not the best performance?
- [W3] In Figure 4, we can see that Mosaic-iT accelerates its training, but the performance at the convergence seems to be the same or even worse than the baselines, which is contradictory to your main results that improves the performance against baselines. Could you clarify the relationship between the convergence performance and the logic of performance improvement.
- [W4] In Table 4 (a) (ablation of Mosaic-IT), you tried Format, Permute, Maskout, and Permute/Maskedout. Why didn’t you try all the combinations?
Also, your best performance came from Maskout, but the adopted variant for Table 1 seems Permute/Maskedout. Why didn’t you use Maskout only?
  
**Reference**
- Wei et al., 2021. Finetuned Language Models Are Zero-Shot Learners. https://arxiv.org/abs/2109.01652

questions:
See the weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706179639

thread_type:
official_review

<thread_object>
creator: reviewer_w54h

<content>
summary:
The paper argues that acquiring instruction-tuning data from a teacher model or humans is resource-intensive. In addition, it suggests that the complexity of single instruction can be limited for many instances which limits the instruction-following capabilities. To address this, the authors propose Mosaic-IT, a data augmentation strategy where the model is trained to follow multiple instructions via a meta instruction. Specifically, the paper considers multiple mosaic strategies including primary, maskout, permute, and format. Finally, the paper shows good improvements across models, instruction-tuning datasets and evaluation methods.

strengths:
- The paper proposes an interesting way to stack multiple instructions to teach more complex instruction-following capabilities to them. It is encouraging that the authors consider many ways in which the instruction-response data can be stacked.
- The paper performs a diverse set of experiments across many base language models, instruction-tuning datasets, and evaluation methods. 
- The paper performs several ablation studies to understand the usefulness of different experimental components.  The paper further analyzes the usefulness of the method using the smoothness of the learning dynamics.

weaknesses:
- Motivation: how much of instruction tuning data acquisition is a bottleneck? There are several papers that show that a small number of instruction tuning data is enough to enable strong instruction-following capabilities. With the rise of powerful small language models (e.g., 4o-mini, Gemini-Flash, Haiku), getting a lot of instruction tuning data is not a bottleneck in terms of resources. In addition, I do not understand the connection between instruction tuning and Dense and Aligned Captions paper from the VL literature. The authors should rethink the motivation in the introduction. It is unclear whether this strategy scales with data i.e., having more Mosaic-IT data beneficial or not. 
- The absolute performance on Alpaca2-LC seems too low. According to the original leaderboard [1], the AlpacaEval LC performance of Alpaca 7B (w/ LLama-1) is 5.9%, and Vicuna is 6.3%. However, the paper indicates that the baseline performance with much stronger base models (Mistral and LLaMA-3-8B) and datasets (Alpaca-GPT4, Wizard-70K, Vicuna, Magpie) is quite low. This makes me wonder if the models have been instruction tuned properly or not.
- Table 2 suggests that baseline methods have better 2-round MT-Bench scores than Mosaic-IT. Shouldn’t the second round MT-Bench scores improve with Mosaic-IT augmentation? Mosaic-IT shares similarity with multi-turn chats in the sense that both require answering multiple instructions in the given context. 

[1] https://tatsu-lab.github.io/alpaca_eval/

questions:
Mentioned in the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732420012800

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely appreciate the time and effort the reviewers had taken to evaluate our manuscript and provide valuable feedback. In the following, we will respond to the major concerns. 

>Q1: Why our method works?

1. 
**Mosaic-IT trains LLMs to follow meta-instructions for compositional reasoning.** 

Previous methods train LLMs to produce a response for a single instruction or query. Instead, our method produces compositional data augmentations to **train LLMs to generate multiple responses for multiple instructions in diverse forms** (e.g., order, mask, format) specified by different meta-instructions. It also enforces LLMs to partition the input context correctly and manage the interference and dependencies among multiple instructions. These are critical to developing and improving the compositional reasoning capabilities of LLMs, which have not been covered by mainstream instruction-tuning frameworks. 

2. 
**Mosaic-IT creates more challenging and complex instructions to further improve LLMs’ instruction-following capabilities.** 

Mosaic-IT’s composition of multiple instructions and the diverse meta-instructions create more challenging and complex instruction-tuning data for LLMs. Moreover, since we do not rely on data synthesis using LLMs but solely apply some rules to existing data, the correctness and quality of the augmented data are guaranteed. As shown in Section 5.1, even powerful LLMs like GPT4 can not follow concatenated instructions. **It has been widely accepted that such challenging and complex instructions improve LLMs’ instruction-following capability [1-8].  Mosaic-IT follows this intuition by making the instruction more challenging and complex** in order to improve LLMs. Different from previous methods relying on humans or stronger teacher LLMs to create the challenging samples, Mosaic-IT does not require any humans/models to create the augmentations.  

To quantitatively evaluate the difficulty and complexity of instruction-tuning data, [2] proposes a ChatGPT-based method (Number of InsTag), while [5] proposes a perplexity-based Instrutcion-Following Difficulty (IFD) score. We compute these two metrics on the Alpaca and WizardLM70k datasets to verify the effectiveness of our method in improving the difficulty/complexity:   

**Number of InsTag [2]:** 
The number of InsTag is used to measure the complexity of the instructions. A larger value of the Number of InsTag indicates the intentions of the instruction are complex and benefit the LLM instruction tuning process. For the experiments below, we prompt GPT4o with the exact prompt provided in [2] to generate the Instags. 

Average InsTag (Alpaca): 2.62

Average InsTag (Alpaca-Mosaic): 9.75

Average InsTag (WizardLM): 4.20

Average InsTag (WizardLM-Mosaic): 10.93

Mosaic-IT largely increases the average number of InsTag, indicating a large increase in instruction intention complexity, further leading to better performance. 

**IFD score [5]:** 
IFD score is a perplexity-based metric used to evaluate the instruction-following difficulty of a given instruction-response pair. A higher IFD score indicates that it is hard for the current model to build a connection between the instruction and the corresponding response, so it can be used to select training data beneficial for LLM instruction tuning. For the experiments below, we utilized the IFD score computed on GPT2. 

Average IFD (Alpaca): 0.60

Average IFD (Alpaca-Mosaic): 0.76

Average IFD (WizardLM): 0.67

Average IFD (WizardLM-Mosaic): 0.79

Mosaic-IT increases IFD scores, indicating an increase in the instruction-following difficulty, which leads to an improvement in performance. 

[1] LIMA: Less Is More for Alignment. (NeurIPS’23)

[2] #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models. (ICLR’24)

[3] WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions. (ICLR’24)

[4]What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. (ICLR’24)

[5] From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. (NAACL’24)

[6] Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. (ACL’24)

[7] Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. (ACL’24)

[8] Instruction Fusion: Advancing Prompt Evolution through Hybridization. (ACL’24)

>Q2: Novelty and contribution

To the best of our knowledge, Mosaic-IT is the **first cost-free compositional data augmentation for instruction tuning of LLMs**. It reduces the training cost and simultaneously improves the performance. In contrast, most existing data-enhancement methods for instruction tuning rely on human supervision or additional LLMs to generate new data.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a data augmentation method for instruction-tuning, the most common post pretraining stage that makes LLMs most useful. The authors propose Mosaic-IT, which can create diverse augmentations from samples of existing instruction tuning datasets. The method works by randomly concatenating multiple instructions into one and requiring the LLM to follow meta-instructions. 
The method overcomes the issue of requiring a strong(er) teacher model to rewrite instruction datasets and shows reductions in training time. The idea is also clear and simple and motivated from computer vision's mosaic augmentation strategy for object detection. 
However, the paper in its current form has severe weaknesses: the mosaic strategies are somewhat arbitrary, the training time reduction in instruction tuning is typically not essential, as this step is very short (~13h) on 4 GPUs and similarly whether the current issue of IT data is indeed the quantity (which if it were the case, would motivate Mosaic IT). While the AC disagrees with w54c that the ubiquity of industrial (private) IT datasets counteracts the value of the proposed paper, the point does stand. Moreover, specifying the augmentation format to single-turn conversation does seem to lead to a decrease in performances in 2-round MT Bench.