Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuning
Authors: Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, YuPeng Hou, Fuxiao Liu, Tianyi Zhou
Accepted at: acl25

Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuning

Ming Li1, Pei Chen2, Chenguang Wang3, Hongyu Zhao1, Yijun Liang1
Yupeng Hou2, Fuxiao Liu1, Tianyi Zhou1
1University of Maryland   2Texas A&M University   3Stony Brook University
minglii@umd.edu  tianyi.david.zhou@gmail.com

Abstract
Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data synthesis method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and an 80% reduction in training costs compared with original instruction tuning. Our codes and data are available at https://github.com/tianyi-lab/Mosaic-IT.

1 Introduction
The emergence of Large Language Models (LLMs) (Scao et al., 2022; OpenAI, 2023; Touvron et al., 2023a) along with their remarkable performance in downstream tasks (Zhao et al., 2023; Xu et al., 2024a), has revolutionized the domains of Artificial Intelligence and Natural Language Processing. A key component of the recipe to unlock the exceptional ability of LLMs in understanding and following instructions is the technique of Instruction Tuning (IT) (Mishra et al., 2021; Wei et al., 2022; Chung et al., 2022), which involves the fine-tuning of LLMs on datasets comprising corresponding instruction-response pairs.

![The brief illustration of our Mosaic-IT with different strategies. Given the original dataset, our method randomly samples and concatenates them together into more complex samples, simulating the multi-instruction-following scenarios in a cost-free manner.](page_1012_370_393_246.png)

Figure 1: The brief illustration of our Mosaic-IT with different strategies. Given the original dataset, our method randomly samples and concatenates them together into more complex samples, simulating the multi-instruction-following scenarios in a cost-free manner.

To ensure the quality of instruction tuning data, earlier efforts (Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023a; Jiang et al., 2023) carefully curate extensive, diverse, and high-quality datasets manually. Although these datasets encompass a wide range of instructions to improve instruction tuning, they require the responses to be meticulously curated by human experts (Khashabi et al., 2020; Ye et al., 2021; Wei et al., 2022; Wang et al., 2022; Du et al., 2022). Alternatively, some approaches (Wang et al., 2023b; Taori et al., 2023; Xu et al., 2023; Li et al., 2023a) leverage more capable teacher LLMs to reduce the labor-intensive process of data generation. For example, the Alpaca (Taori et al., 2023) utilizes self-instruct (Wang et al., 2023b) to automatically generate diverse instruction tuning datasets. Building on this trend and the widely acknowledged notion that more complicated instructions are more beneficial for LLMs’ instruction-following ability (Xu et al., 2023; Zhao et al., 2024), numerous strategies (Zhao et al., 2024; Wu et al., 2024; Ding et al., 2023; Li et al., 2023a; Liu et al., 2023a; Li et al., 2024c,b;
Guo et al., 2024; Xu et al., 2024a) have been proposed to further diversify and complexity the instruction-response pairs, utilizing teacher models like ChatGPT-3.5 and GPT-4 (OpenAI, 2023).

Despite the enhanced performance in instruction-following ability offered by these existing methods, they face Two major issues: (1) They heavily rely on teacher models or human annotators to rewrite instruction-response pairs, which highlights the resource-intensive nature and their constraints on scalability; (2) They only increase the complexity within the scope of a single instruction, which limits the potential improvement in LLMs’ instruction-following capabilities. Motivated by the Dense and Aligned Captions (Doveh et al., 2023) proposed for vision language (VL) models and the mosaic data augmentation proposed in Yolov4 (Bochkovskiy et al., 2020), we hypothesize that denser instructions benefit the LLM alignment, i.e. the process of instruction tuning should not be constrained by one single instruction but be extended to follow several instructions at a time, which represents a higher level of instruction-following ability that is beneficial to the training process. A similar concept during the inference phase is proposed by batch prompting (Cheng et al., 2023; Lin et al., 2024), where multiple samples are grouped in one batch, allowing LLMs to generate multiple responses at one inference, while its performance is sub-optimal.

As orthogonal to the existing instruction tuning methods, we introduce Mosaic Instruction Tuning (Mosaic-IT), an innovative and model/human-free compositional approach that augments existing instruction tuning datasets, which concurrently improves the LLM performances and lowers the training expenses. As shown in Figure 1, in our method, multiple instructions and corresponding responses from the original dataset are concatenated into a single sample for fine-tuning, simulating the multi-instruction-following scenarios at no cost. Without applying any additional strategies, we term this simple process as the Primary Mosaic Strategy. We posit that this mosaic strategy process significantly improves the complexity and density of the original instructions, learning from which directly benefits LLMs in their instruction-following ability. Additionally, this method offers the advantage of directly reducing the total count of instruction-response pairs, thereby cutting down on training iterations, and accelerating the training process significantly by approximately 80% reduction.

Though effective, the Primary Mosaic strategy constrains LLMs in responding to the instructions in the original order and format, potentially limiting their further potential. Thus, we further introduce three Advanced Mosaic Strategies aimed at enhancing the diversity and complexity of the mosaicked instruction-response pairs: Format, Permute, and Maskout, in which an additional meta-instruction is provided as a higher-level guideline for LLMs to follow the given instructions. Illustrative examples are presented in Figure 2. Specifically, in the Format strategy, some arbitrary parsing formats will be defined in the meta-instruction, thus forcing LLMs to follow these formats, which notably enhances the LLMs’ capacity to follow formats. In the Permutation strategy, an arbitrary permuted order is defined, thus forcing LLMs to respond in a desired order. In the Maskout strategy, some arbitrary instructions are sampled, which meta-instruction forces LLMs to ignore. Moreover, the use of these Advanced strategies not only boosts the performance in several evaluation metrics but also keeps our method free of additional costs.

In summary, our primary contributions can be illustrated as follows:

• We propose the first cost-free data Synthesis method, Mosaic-IT, which extends existing instruction tuning from handling one single instruction at a time to following multiple instructions in diverse forms. This approach significantly enhances the potential utilization of existing high-quality datasets.

• Mosaic-IT improves the instruction-following abilities of LLMs compared to training on original data, as evidenced by consistent performance gains across a wide range of benchmarks, model families, and datasets, demonstrating strong generalization capabilities.

• Mosaic-IT substantially increases training efficiency by reducing the required number of training iterations, resulting in an approximate 80% reduction in training time, as confirmed by experimental results.

2 Methodology

2.1 Preliminaries

The instruction tuning dataset, defined as \( D \), consists of \( n \) data samples, each represented by a triplet \((Instruction, Input, Response)\). For simplicity, we define \( x = map(Instruction, Input) \) as
Figure 2: Illustrative examples of Mosaic-IT. Given 3 simple data instances, our method concatenates them into data samples with diverse forms. Texts in red represent the meta-instructions that define the formats or orders for LLMs to respond. Texts in yellow are major response differences of each strategy. The Primary Strategy only concatenates data. The Format Strategy requires LLMs to respond in predefined formats. The Permute Strategy requires LLMs to respond in specific orders, and the Maskout Strategy requires LLMs to ignore some instructions.

the unified instruction, and \( y \) as the corresponding response. Therefore, \( D \) can be represented as \( (x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n) \), denoting a set of \( n \) instruction-response pairs. Let \( p_\theta(\cdot) \) denote the LLMs to be trained, with parameters \( \theta \). In the instruction tuning setting, \( p_\theta \) is typically fine-tuned by maximizing the following objective on each data \( (x_i, y_i) \). \( y_{i,j} \) represents the \( j \)th token of response \( y_i \), \( y_{i,-j} \) represents the tokens prior to \( y_{i,j} \), and \( l_i \) represents the token length of \( y_{i,j} \):

\[
\max_\theta \sum_{j=1}^{l_i} \log p_\theta \left( y_{i,j} | x_i, y_{i,-j} \right),
\]

2.2 Mosaic-IT

Motivated by the success of the existing data-centric instruction tuning methods, a line of approaches is proposed to further enhance the instruction-response pairs utilizing extra teacher LLMs (Xu et al., 2024a). Though effective, all existing methods for instruction tuning restrict training samples to just one instruction, which severely limits the potential of the existing high-quality data and the instruction-following ability of the models to be trained. Motivated by the Dense and Aligned Captions (Doveh et al., 2023) for VL, we hypothesize that denser instructions benefit the LLM alignment, thus the process of instruction tuning should not be constrained by one single instruction but be extended to follow several instructions at a time, which represents a higher level of instruction-following ability that is beneficial to the training process. Thus, we propose the cost-free data synthesis method, Mosaic Instruction Tuning (Mosaic-IT), as shown in Figure 1.

2.2.1 Primary Mosaic strategy

Exploring the concept of concatenating random instruction-response pairs into a unified instruction-response pair for training remains largely unexplored. The primary challenge lies in crafting a coherent overall instruction and obtaining its corresponding response. Most existing methods utilize a strong teacher model to rewrite the instructions with prompting techniques and generate corresponding responses, introducing more cost by actually re-generating new data samples. To harness the full potential of existing data rather than directly discarding it, we introduce a simple cost-free compositional approach as shown in Figure 2, in which instructions are randomly concatenated with serial
digits to form an overall instruction. The concatenated overall instruction is denoted as \([x_1, ..., x_k]\), with the corresponding overall response concatenated as \([y_1, ..., y_k]\). Here, \(k\) denotes the number of original data samples integrated into each overall sample.

In this framework, the fundamental instruction-following capability is triggered by the existing instruction-response pairs, and the mosaic strategy extends this capability to a higher level in which LLMs are forced to follow multiple instructions. It represents a much more complicated scenario that benefits LLMs compared with traditional single-task instructions. Consequently, the objective function for each concatenated overall data sample can be formulated as follows:

\[
\max_{\theta} \sum_{j=1}^{l} \log p_\theta \left( [y_1, ..., y_k]_j | [x_1, ..., x_k], [y_1, ..., y_k]_{<j} \right),
\]

Here, \([y_1, ..., y_k]_j\) denotes the \(j\)th token of the overall response, \([y_1, ..., y_k]_{<j}\) denotes the tokens prior to \(j\)th token, and \(l\) represents the length of overall response. This formulation encapsulates the essence of our approach, optimizing the model parameters \(\theta\) to maximize the likelihood of generating the correct sequence of responses for the given overall instruction.

2.2.2 Advanced Mosaic Strategies
Though effective, this primary mosaic strategy constrains LLMs in responding to the instructions with the original order and format, potentially limiting their generalization and practical usage. In our method, the instructions and corresponding responses from the original dataset can be viewed as atomic components, and our method randomly combines these elements to form new instructions and responses. This nature allows us to further complicate this process with fancier strategies, thus forcing LLMs to follow more complicated overall instructions. Hence, we propose three Advanced Mosaic Strategies to complicate and diversify the mosaicked samples as shown in Figure 2, including Format, Permute, and Maskout, with meta-instructions guiding them. These strategies are still purely rule-based (Li et al., 2024a) and do not incorporate the additional human/LLM generation.

Format In the Format strategy, some arbitrary formats are defined in the meta-instruction to force LLMs to follow these formats in the response. The formats mainly contain two categories: 1) Serial Digit Format and (2) Response Parsing Format. The serial digits establish the initial instruction order that guides LLMs to follow sequentially. We manually define 10 types of serial digit format, which will be randomly sampled during each mosaic process. For response parsing, we simulate the scenario where the users try to extract specific information from the responses. We define 27 types of parsing brackets and 17 types of parsing text pairs, which will be randomly sampled and assembled during each mosaic process. Examples can be found in Appendix G, which can be easily extended for customized training settings. We denote responses with specific formats as \(y'_i = wrap(y_i, s_{format})\), and \(l\) as the token length of the overall response. An additional meta-instruction \(s_{format}\) specifying the required format will be included in the overall instruction. Thus, the objective function for each mosaic data point:

\[
\max_{\theta} \sum_{j=1}^{l} \log p_\theta \left( [y'_1, ..., y'_k]_j | x_1, x_2, ..., x_k, s_{format}, [y_1, ..., y_k]_{<j} \right)
\]

Permute and Maskout Building upon the Format strategy, we further introduce two strategies for our Mosaic-IT, Permutation and Maskout.

In the Permute strategy, an arbitrary permuted order is defined in the meta-instructions, forcing LLMs to follow. Moreover, several high-level rules are defined to ensure the complexity and diversity of meta-instructions, e.g., forcing LLMs to respond to each instruction in the randomly generated permutation list, forcing LLMs to respond in the alphabetical order of each instruction, forcing LLMs to respond according to the length of instructions, etc. The detailed rule types and descriptions are depicted in Appendix G. These various meta-instructions not only provide higher-level guidelines for LLMs to follow multiple instructions but also inherently enhance the instruction perception ability of LLMs. In our settings, LLMs are required to generate responses selectively conditioned on some critical parts of the overall instruction, forcing them to first understand the formats and other requirements, indicating a more comprehensive understanding of the context given. The meta-instruction is denoted as \(s_{permute}\) and is included in the overall instruction. The permuted response list is denoted as \([y'_1, ..., y'_k] = Permute([y_1', ..., y_k'], s_{permute})\). Thus, the objective function can be formulated as below:
\[
\max_{\theta} \sum_{j=1}^{l} \log p_\theta \left( [y_1', ..., y_{k'}']_j \middle| x_1, x_2, ..., x_k, s_{format}, s_{permute}, [y_1', ..., y_{k'}']_{<j} \right)
\] (4)

In the Maskout strategy, some arbitrary instructions are selected in the meta-instructions, forcing LLMs to ignore them. Several high-level rules are also defined similarly to the permute strategy, including forcing LLMs to ignore the instructions with given random digits, forcing LLMs to ignore the longest one/several instructions, forcing LLMs to ignore odd-numbered instructions, etc. The details are provided in Appendix G. Similarly, the meta-instruction is denoted as \( s_{maskout} \) and the response list is denoted as \( [y_1', ..., y_m'] = Maskout([y_1', ..., y_k'], s_{maskout}) \), where m is the count of responses after masking out. Thus, the objective function can be formulated as below:

\[
\max_{\theta} \sum_{j=1}^{l} \log p_\theta \left( [y_1', ..., y_m']_j \middle| x_1, x_2, ..., x_k, s_{format}, s_{maskout}, [y_1', ..., y_m']_{<j} \right)
\] (5)

It’s important to note that our mosaic strategies entail no supervision cost, and the predefined rules are flexible and have the potential for further extension. We utilize the version with three Advanced strategies as our default Mosaic-IT.

How to decide the Number of Instructions k: Number of Instructions denotes the number of original data samples that are integrated into an overall sample. In addition to the detailed mosaic strategies being used, this count also dramatically affects the effect of Mosaic-IT. Our experiments reveal that larger and more diverse numbers of instructions will benefit LLM training. By default, we set the maximum number of instructions as \( k_{max} = 10 \), and randomly sample an integer that is smaller than or equal to \( k_{max} \) under a uniform distribution. If the number causes the data sample to be longer than the max length, it will be automatically reduced to the max number, which remains the sample length within the limits.

3 Experimental Results

3.1 Main Results

In this section, we present the evaluation results comparing our methods with the baseline methods on 6 baseline models (Mistral-7B (Jiang et al., 2023), Llama2-7B (Touvron et al., 2023b), Llama2-13B, Llama-3-8B (Dubey et al., 2024), Phi-3 (Abdin et al., 2024), Gemma2-2B (Team et al., 2024)) and 4 instruction tuning datasets (Alpaca-GPT4 (Peng et al., 2023), Alpaca (Taori et al., 2023), WizardLM-70k (Xu et al., 2023)), Magpie (Xu et al., 2024b), on 5 commonly used evaluation metrics and additional Human Evaluation. Detailed experimental setup and descriptions of evaluation metrics can be found in Appendix B.

Table 1 shows the results on 2 general evaluation settings (Pair-Wise Comparison and Open LLM leaderboard). Pair-wise Winning Score indicates the result directly comparing our models with the corresponding baseline models, which is calculated as \( (Num(Win) - Num(Lose))/Num(All) +1 \). These values that are greater than 1.0 represent better responses generated by our models. The performances on the Huggingface Open LLM Leaderboard are also presented, and we bold the greater average values for each comparison. The consistently outperforming results on different base models and datasets represent the effectiveness and robustness of our methods. Results on more advanced baseline models and datasets can be found in Table 2. Our method shows consistent improvements compared with baseline models. The performance gains in pair-wise comparison indicate our method helps LLMs generate more detailed and accurate responses, and the performance gains in the open leaderboard indicate our method helps alleviate harm to the intrinsic capabilities of base LLMs.

To better understand how our method improves the instruction-following abilities of LLMs, we further compare the performance on the other 3 benchmarks for fine-grained analysis as shown in Table 2. On the Alpaca Eval 2 benchmark, our method has a consistent improvement with or without the Length Control (LC), indicating that the improvement of response qualities does not directly originate from the length of responses. On the MT-Bench, the 1-round scores of our method are consistently higher, while the 2-round scores slightly fluctuate, indicating that our method mainly improves the response quality for single-round conversations since our meta instructions only focus on single-round scenarios in this version. On the IF Eval benchmark, our method consistently improves the performance on different settings, both Prompt-level and Instruction-level. Compared with the previous benchmarks, IF Eval mainly focuses on the constraint-following ability of LLMs. The
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Pair-wise ↑ Winning Score</th>
    <th colspan="5">Huggingface Open LLM Leaderboard ↑</th>
  </tr>
  <tr>
    <th>Average</th>
    <th>Winning Score</th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
  </tr>
  <tr>
    <td rowspan="3">Mistral-7B</td>
    <td rowspan="3">Alpaca-GPT4</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>59.70</td>
    <td>55.03</td>
    <td>78.87</td>
    <td>56.01</td>
    <td>48.88</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.349</b></td>
    <td><b>63.65</b></td>
    <td>59.04</td>
    <td>81.85</td>
    <td>60.09</td>
    <td>53.62</td>
  </tr>
  <tr>
    <td>Alpaca</td>
    <td>1.000</td>
    <td>55.15</td>
    <td>51.96</td>
    <td>74.61</td>
    <td>52.85</td>
    <td>41.20</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td rowspan="2">Wizard-70k</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>57.86</td>
    <td>51.88</td>
    <td>77.93</td>
    <td>53.76</td>
    <td>47.89</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.161</b></td>
    <td><b>61.11</b></td>
    <td>57.85</td>
    <td>82.13</td>
    <td>57.42</td>
    <td>47.08</td>
  </tr>
  <tr>
    <td rowspan="3">Llama2-7B</td>
    <td rowspan="3">Alpaca-GPT4</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>58.71</td>
    <td>54.69</td>
    <td>80.05</td>
    <td>47.89</td>
    <td>52.21</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.073</b></td>
    <td><b>58.84</b></td>
    <td>54.18</td>
    <td>80.54</td>
    <td>47.92</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td>Alpaca</td>
    <td>1.000</td>
    <td>55.25</td>
    <td>54.35</td>
    <td>78.65</td>
    <td>47.02</td>
    <td>40.98</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td rowspan="2">Wizard-70k</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>57.09</td>
    <td>54.18</td>
    <td>79.25</td>
    <td>46.93</td>
    <td>48.02</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.197</b></td>
    <td><b>57.41</b></td>
    <td>54.69</td>
    <td>79.69</td>
    <td>48.11</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td rowspan="3">Llama2-13B</td>
    <td rowspan="3">Alpaca-GPT4</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>61.47</td>
    <td>58.70</td>
    <td>83.12</td>
    <td>54.13</td>
    <td>49.92</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.110</b></td>
    <td><b>63.26</b></td>
    <td>58.87</td>
    <td>83.54</td>
    <td>55.75</td>
    <td>54.87</td>
  </tr>
  <tr>
    <td>Alpaca</td>
    <td>1.000</td>
    <td>57.63</td>
    <td>57.25</td>
    <td>81.23</td>
    <td>54.13</td>
    <td>37.91</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td rowspan="2">Wizard-70k</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>61.24</td>
    <td>57.04</td>
    <td>83.39</td>
    <td>55.76</td>
    <td>48.78</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.078</b></td>
    <td><b>61.50</b></td>
    <td>58.70</td>
    <td>83.69</td>
    <td>56.44</td>
    <td>47.18</td>
  </tr>
</table>

Table 1: The performance comparison on the Pair-wise Comparison Winning Score and the Open LLM Leaderboard, on 3 different base models and 3 different instruction tuning datasets.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Pair-wise ↑ Score</th>
    <th colspan="2">Open LLM ↑ Average</th>
    <th colspan="2">Alpaca Eval 2 ↑ Rate (LC)</th>
    <th colspan="2">MT-Bench ↑ 1-round</th>
    <th colspan="2">IF Eval ↑ P(L)</th>
  </tr>
  <tr>
    <th>Winning Score</th>
    <th>Average</th>
    <th>Rate</th>
    <th>Rate</th>
    <th>1-round</th>
    <th>2-round</th>
    <th>P(L)</th>
    <th>I(L)</th>
  </tr>
  <tr>
    <td rowspan="2">Llama-3-8B</td>
    <td rowspan="2">Magpie</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>56.15</td>
    <td>9.22</td>
    <td>13.74</td>
    <td>8.10</td>
    <td>7.08</td>
    <td>35.67</td>
    <td>47.72</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.133</b></td>
    <td><b>60.13</b></td>
    <td><b>12.23</b></td>
    <td><b>16.05</b></td>
    <td><b>8.36</b></td>
    <td><b>7.49</b></td>
    <td><b>40.67</b></td>
    <td><b>52.76</b></td>
  </tr>
  <tr>
    <td rowspan="2">Phi-3</td>
    <td rowspan="2">Magpie</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>62.90</td>
    <td>13.82</td>
    <td><b>17.68</b></td>
    <td><b>7.78</b></td>
    <td><b>6.42</b></td>
    <td>44.36</td>
    <td>55.52</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.014</b></td>
    <td><b>63.54</b></td>
    <td><b>14.04</b></td>
    <td><b>17.67</b></td>
    <td><b>7.89</b></td>
    <td><b>6.16</b></td>
    <td><b>50.83</b></td>
    <td><b>62.35</b></td>
  </tr>
  <tr>
    <td rowspan="2">Gemma2-2B</td>
    <td rowspan="2">Magpie</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>46.37</td>
    <td>5.35</td>
    <td>7.77</td>
    <td>4.57</td>
    <td>3.23</td>
    <td>21.81</td>
    <td>32.49</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.032</b></td>
    <td><b>48.36</b></td>
    <td><b>5.66</b></td>
    <td><b>8.54</b></td>
    <td><b>5.16</b></td>
    <td><b>3.96</b></td>
    <td><b>22.18</b></td>
    <td><b>34.77</b></td>
  </tr>
  <tr>
    <td rowspan="3">Mistral 7B</td>
    <td rowspan="3">Alpaca-GPT4</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>59.70</td>
    <td>3.98</td>
    <td>7.28</td>
    <td>6.44</td>
    <td><b>5.26</b></td>
    <td>35.86</td>
    <td>45.92</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>1.349</b></td>
    <td><b>63.65</b></td>
    <td><b>5.00</b></td>
    <td><b>7.81</b></td>
    <td><b>7.11</b></td>
    <td>4.69</td>
    <td><b>38.08</b></td>
    <td><b>50.23</b></td>
  </tr>
  <tr>
    <td>Wizard-70k</td>
    <td>Baseline</td>
    <td>1.000</td>
    <td>57.86</td>
    <td>4.13</td>
    <td>6.46</td>
    <td>6.21</td>
    <td><b>4.70</b></td>
    <td>41.96</td>
    <td>53.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Mosaic-IT</td>
    <td><b>1.161</b></td>
    <td><b>61.11</b></td>
    <td><b>4.44</b></td>
    <td><b>7.56</b></td>
    <td><b>6.95</b></td>
    <td><b>4.32</b></td>
    <td><b>45.47</b></td>
    <td><b>56.47</b></td>
  </tr>
</table>

Table 2: The performance comparison across multiple model families and datasets on five evaluation metrics. Rate(LC) in Alpaca Eval represents length-controlled win rates. In IF Eval, P(L) and I(L) represent Prompt-level and Instruction-level accuracy in the Loose setting.

consistent improvement in this benchmark represents that our method not only improves the response qualities of the LLMs but also improves their controllability regarding formats.

Further Human Evaluations are conducted on Mistral-7B with the Alpaca-GPT4 and WizardLM datasets. For the comparison on (1) Alpaca-GPT4: the model using Mosaic-IT wins on 68 out of 100 instruction, ties on 3, and losses on 29 instructions; on (2) WizardLM: the model using Mosaic-IT wins on 63 out of 100 instruction, ties on 6, and losses on 31 instructions. This human evaluation also further verifies the effectiveness of our Mosaic-IT.

To conclude, our Mosaic-IT shows consistent improvement in instruction-following and constraint-following ability and response quality with a reduction of approximately 80% of the training time cost. Given that our method is a cost-free data synthesis technique that does not rely on any additional human/LLM generation, the observed improvements are remarkable.

3.2 Ablation Studies

Detailed ablations are in Appendix C.

Ablation on Mosaic Strategies investigates how different mosaic strategies affect LLM performances. We find out that further implementing Advanced Strategies (Format, Permute, Maskout) improves LLM performance as they largely diversify and complicate the instructions, only implementing the Primary Strategy. Ablation on the Max Number of Instructions investigates the trend between the number of instructions that are concatenated
<table>
  <tr>
    <th>Settings</th>
    <th>Baseline</th>
    <th>Fix</th>
    <th>Exponential</th>
    <th>Pareto</th>
    <th>Log-normal</th>
    <th>Logistic</th>
    <th>Uni-2</th>
    <th>Uni-4</th>
    <th>Uni-6</th>
    <th>Uni-8</th>
    <th>Uni-10</th>
    <th>Uni-12</th>
  </tr>
  <tr>
    <td>Time (min)</td>
    <td>827</td>
    <td>121</td>
    <td>129</td>
    <td>133</td>
    <td>133</td>
    <td>143</td>
    <td>716</td>
    <td>426</td>
    <td>305</td>
    <td>245</td>
    <td>202</td>
    <td>173</td>
  </tr>
  <tr>
    <td>Time Ratio</td>
    <td>100.0%</td>
    <td>14.6%</td>
    <td>15.6%</td>
    <td>16.1%</td>
    <td>16.1%</td>
    <td>17.3%</td>
    <td>86.6%</td>
    <td>51.5%</td>
    <td>36.9%</td>
    <td>29.6%</td>
    <td>24.4%</td>
    <td>20.9%</td>
  </tr>
  <tr>
    <td>Winning Score</td>
    <td>1.000</td>
    <td>0.982</td>
    <td>0.995</td>
    <td>1.417</td>
    <td>1.431</td>
    <td>1.417</td>
    <td>0.989</td>
    <td>1.142</td>
    <td>1.303</td>
    <td>1.294</td>
    <td>1.349</td>
    <td>1.376</td>
  </tr>
</table>

Table 3: The training time comparison of different settings, and the pair-wise winning scores are also provided for better illustration. “Uni-2” represents uniform distribution with max count as 2. **Mosaic-IT reduces the training time to 16% – 25% while achieving better performance.**

![Three line plots comparing loss curves for different instruction tuning processes and datasets](page_184_370_1082_246.png)

Figure 3: The training loss curve comparisons between the original instruction tuning process and our Mosaic-IT with w datasets on (a) Mistral-7B, (b) Llama2-7B, and (c) Llama2-13B. The “stair-like” loss curves for the original training process indicate potential memorizing effects, while our loss curves are smoother. All the training settings are kept the same between the baseline models and Mosaic-IT models, including the Learning Rate, Warm-up Ratio, Learning Rate Schedule (Cosine), Batch Size, etc.

together and the LLMs’ performances. We find out that in the uniform distribution setting, more of the instructions are concatenated together, as this process makes the concatenated instruction complex. **Ablation on the Distribution of Number of Instructions** investigates how different distributions affect the LLMs’ performance. It is revealed that, except for the maximum number of instructions concatenated together, the distribution is also important. **Ablation on Semantic Grouping** investigates how semantic grouping, i.e., grouping instructions with similar semantic meaning rather than pure random grouping, affects performance. We show that semantic grouping has its own benefits, while random selection is much more convenient.

4 Further Discussion

4.1 Improving Efficiency

One of the benefits of our method is the efficiency of the training process. Given an existing dataset, our mosaic processes largely decrease the number of total overall instructions and the total number of gradient descents, leading to a reduction in the training process. The detailed comparison is shown in Table 3, which is based on the Mistral-7B model on the Alpaca-GPT4 dataset. The time is calculated based on four NVIDIA A100 Graphic Cards. As shown, our method greatly decreases the training time to approximately 16% to 25% while achieving better performances, especially when there are mosaic samples with larger permutation counts. Please note that the time reduction is *ratio-based*, when larger datasets or models are trained, the absolute time reduction gap between baseline methods and our methods will be much more obvious.

4.2 Alleviating Memorizing

In the original instruction tuning process, each data sample will be trained several times for LLMs without changes to the instructions and responses. This training process poses risks to the potential memorizing effects on training samples, which can be partially indicated by the “*stair-like*” training loss curves as shown in Figure 3. In the figure, all the training settings are kept the same between the baseline models and Mosaic-IT models, including the Learning Rate, Warm-up Ratio, Learning Rate Schedule (Cosine), Batch Size, etc. For the baseline methods, the training loss hardly decreases within each epoch of training but drops dramatically when the LLMs meet the same training samples again, which indicates a potential memorizing effect of training samples and potential overfitting. However, when utilizing our method, the random mosaics of original instructions with diverse and complex meta-instructions largely diversify the overall training
instructions. Although each original data sample will still be seen by LLMs several times during training, the overall context varies dramatically as each original sample is only an atomic element of the overall mosaic sample, indicating that there will be no identical overall instructions during the whole training process. Thus, this augmentation largely alleviates the potential memorizing and overfitting problems as shown in the figure, where the training loss decreases smoothly, representing the gradual learning process.

5 Why It Works

5.1 Preliminaries: Performance Degradation

The motivation of our Mosaic-IT is also rooted in the observation that when handling multiple instructions simultaneously, a performance degradation will incurred for even strong LLMs like GPT-4-turbo. While LLMs generally perform well when responding to single instructions, their capability to follow multiple instructions at once tends to decline noticeably. BatchPrompt has shown the uncertainty when LLMs are requested to answer multiple formatted questions at one time. Moreover, in some cases, e.g for general open-domain instructions, LLMs might directly ignore some of the instructions, especially when the LLMs are required to respond to the instructions in a random pre-defined order, which is exactly simulating our Permute strategy.

To quantitatively analyze this phenomenon, experiments using GPT-3.5-turbo and GPT-4-turbo are conducted on the WizardLM test set. Specifically, we compare the models’ performance when responding to multiple instructions concurrently versus responding to a single instruction at each time, by utilizing LLM-based Pair-Wise comparison, as shown in Table 4. All the win rates are lower than 1.0, demonstrating a clear and significant reduction in response quality when these models are required to respond to multiple instructions at one time. Moreover, the possibility of missing instructions (Miss Rate) increases further when they are required to respond to the instructions in a predefined random order rather than a sequential order. These results clearly demonstrate the difficulties of following several instructions at a time and why it can be regarded as a higher level of instruction-following capability.

5.2 Further Analysis

Mosaic-IT trains LLMs to follow meta-instructions for compositional reasoning.

Previous methods train LLMs to produce a response for a single instruction or query. Instead, our method produces a compositional data synthesis method to train LLMs to generate multiple responses for multiple instructions in diverse forms (e.g., order, mask, format) specified by different meta-instructions. It also enforces LLMs to partition the input context correctly and manage the interference and dependencies among multiple instructions. These are critical to developing and improving the compositional reasoning capabilities of LLMs, which have not been covered by mainstream instruction-tuning frameworks.

Mosaic-IT creates more challenging and complex instructions to further improve LLMs’ instruction-following capabilities.

Mosaic-IT’s composition of multiple instructions and the diverse meta-instructions create more challenging and complex instruction-tuning data for LLMs. Moreover, since we do not rely on data synthesis using LLMs but solely apply some rules to existing data, the correctness and quality of the augmented data are guaranteed. As shown in Table 4, even powerful LLMs like GPT4 can not follow concatenated instructions. Besides, it has been widely accepted that such challenging and complex instructions improve LLMs’ instruction-following capability (Zhao et al., 2024; Wu et al., 2024; Ding et al., 2023; Li et al., 2023a; Liu et al., 2023a; Li et al., 2024c,b; Guo et al., 2024). Mosaic-IT follows this intuition by making the instruction more challenging and complex in order to improve LLMs. Different from previous methods relying on humans or stronger teacher LLMs to create the challenging samples, Mosaic-IT does not require any humans/models to create the augmentations.

To quantitatively evaluate the difficulty and complexity of instruction-tuning data, InsTag (Lu et al., 2023) proposes a ChatGPT-based method (Number of InsTag), while Cherry (Li et al., 2024e) proposes a perplexity-based Instruction-Following Difficulty (IFD) score. We compute these two metrics on the Alpaca and WizardLM70k datasets to verify the effectiveness of our method in improving the difficulty/complexity:

Number of InsTag: The number of InsTag is used to measure the complexity of the instructions.
<table>
  <tr>
    <th rowspan="2">Pair-Wise (Multi vs. Single)</th>
    <th colspan="2">3 Instructions</th>
    <th colspan="2">5 Instructions</th>
    <th colspan="2">7 Instructions</th>
  </tr>
  <tr>
    <th>Win Rate ↑</th>
    <th>Miss Rate ↓</th>
    <th>Win Rate ↑</th>
    <th>Miss Rate ↓</th>
    <th>Win Rate ↑</th>
    <th>Miss Rate ↓</th>
  </tr>
  <tr>
    <td>GPT-3.5-turbo (Sequential)</td>
    <td>0.357</td>
    <td>0.014</td>
    <td>0.336</td>
    <td>0.055</td>
    <td>0.303</td>
    <td>0.064</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo (Random)</td>
    <td>0.315</td>
    <td>0.124</td>
    <td>0.330</td>
    <td>0.156</td>
    <td>0.198</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>GPT-4-turbo (Sequential)</td>
    <td>0.176</td>
    <td>0.000</td>
    <td>0.137</td>
    <td>0.000</td>
    <td>0.140</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>GPT-4-turbo (Random)</td>
    <td>0.139</td>
    <td>0.000</td>
    <td>0.153</td>
    <td>0.014</td>
    <td>0.101</td>
    <td>0.005</td>
  </tr>
</table>

Table 4: Pair-wise win rate of performances when responding to multiple instructions concurrently versus responding to a single instruction each time, and miss rate when responding to multiple instructions concurrently. “3 Instructions” represents the setting where 3 random instructions are concatenated together for inference. “Sequential” and “Random” represents the setting where the models are asked to respond to each instruction sequentially, or in a random pre-defined order.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Ins Tag</th>
    <th colspan="2">IFD</th>
  </tr>
  <tr>
    <th>Alpaca</th>
    <th>Wizard-70k</th>
    <th>Alpaca</th>
    <th>Wizard-70k</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>2.62</td>
    <td>4.20</td>
    <td>0.60</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td>9.75</td>
    <td>10.93</td>
    <td>0.76</td>
    <td>0.79</td>
  </tr>
</table>

Table 5: The comparison between the original dataset and Mosaic-IT enhanced dataset with respect to the Number of InsTag and the IFD score.

A larger value of the Number of InsTag indicates the intentions of the instruction are complex and benefit the LLM instruction tuning process. For the experiments below, we prompt GPT4o with the exact prompt provided in the paper to generate the Instags. As shown in Table 5, Mosaic-IT largely increases the average number of InsTag, indicating a large increase in instruction intention complexity, further leading to better performance.

IFD score: IFD score is a perplexity-based metric used to evaluate the instruction-following difficulty of a given instruction-response pair. A higher IFD score indicates that it is hard for the current model to build a connection between the instruction and the corresponding response, so it can be used to select training data beneficial for LLM instruction tuning. For the experiments below, we utilized the IFD score computed on GPT2. As shown in Table 5, Mosaic-IT increases IFD scores, indicating an increase in the instruction-following difficulty, which leads to an improvement in performance.

5.3 Qualitative Example

To better understand the differences between the responses generated by the baseline model and the Mosaic-IT model, a pair of qualitative examples are presented in Figure 4 (baseline model) and Figure 5 (Mosaic-IT model). The example instruction is selected from the WizardLM test sets and the models are Mistral-7B trained on Alpaca-GPT4 with or without our method.

Some of the real-world instructions can be complex and hard to answer all at once but require LLMs to “decompose” the original overall instruction into pieces to respond. Due to the lack of this kind of difficult data in the training set, the capability of most LLMs is largely restricted as shown in Figure 4. The baseline LLM directly ignores the “sub-query” of explaining “protocols and standards such as TCP/IP, HTTP, FTP, DNS, DHCP, and ARP”. On the contrary, as a cost-free data synthesis method, Mosaic-IT simulates this kind of instruction by using original easy instructions, thus equipping LLMs with the capability to respond to these difficult instructions without ignoring some parts of it, as shown in Figure 5.

The better performances on Alpaca Eval 2 (LC) show that the performance gains of Mosaic-IT do not purely originate from longer responses. On the contrary, Mosaic-IT actually makes the lengths of training data more diverse compared with the original data, e.g., some of the samples are the concatenation of only 1 original sample, while some of them are the concatenation of 10 samples. This ensures diverse generation lengths according to the given instructions rather than being dominated by the original length distribution.

6 Conclusion

We introduce Mosaic Instruction Tuning (Mosaic-IT), a novel, human/model-free method to enhance instruction tuning for LLMs. By concatenating multiple instruction-response samples and using higher-level meta-instructions, Mosaic-IT improves multi-step and format-following capabilities. Our evaluations show superior performance and an 80% reduction in training costs compared to the original methods. Mosaic-IT’s simplicity and efficiency make it a scalable solution for improving LLMs without extensive human intervention or resource-intensive teacher models. Our results highlight the potential of innovative data synthesis techniques in advancing LLM capabilities.
Limitations

The potential limitations of our work: (1) Currently, three Advanced Mosaic Strategies with corresponding high-level rules are proposed and utilized in our method, however, we believe more strategies and predefined rules can be further introduced. (2) The optimal distribution of the number of instructions for the mosaic process still needs further justification in future studies. (3) It is unknown whether the inclusion of extra models or careful curation/selection of instructions for concatenation will further improve the performance of Mosaic-IT largely.

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Jun-heng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iser, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Pierre Kaufmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilchenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Linden, Xihui Lin, Ziqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weisheng Liu, Xiaodong Liu, Chong Luo, Piush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilic, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambuddha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Said, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiran Zhou. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. Preprint, arXiv:2004.10934.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Alexander Bukharin and Tuo Zhao. 2023. Data diversity matters for robust instruction tuning. Preprint, arXiv:2311.14736.

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2023. Alpagasus: Training a better alpaca with fewer data. Preprint, arXiv:2307.08701.

Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model APIs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792–810, Singapore. Association for Computational Linguistics.

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharar Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai shin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question
answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Preprint, arXiv:2205.14135.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.

Ning Ding, Yulin Chen, Bokai Xu, Yuja Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.

Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Al-fassy, Rameswar Panda, Raja Giryes, Rogerio Feris, Shimon Ullman, and Leonid Karlinsky. 2023. Dense and aligned captions (DAC) promote compositional reasoning in VL models. In Thirty-seventh Conference on Neural Information Processing Systems.

Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. Mods: Model-oriented data selection for instruction tuning. Preprint, arXiv:2311.15653.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhang Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, Dublin, Ireland. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahleb, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Harshorn, Aoob Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolidakis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, and Ishan Misra. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca-farm: A simulation framework for methods that learn from human feedback. Preprint, arXiv:2305.14387.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.

Weidong Guo, Jiuding Yang, Kaitong Yang, Xiangyang Li, Zhuwei Rao, Yu Xu, and Di Niu. 2024. Instruction fusion: Advancing prompt evolution through hybridization. Preprint, arXiv:2312.15692.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio RenardLavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Computational Linguistics.

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.

Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look at the first sentence: Position bias in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1109–1121, Online. Association for Computational Linguistics.

Ming Li, Han Chen, Chenguang Wang, Dang Nguyen, Dianqi Li, and Tianyi Zhou. 2024a. Ruler: Improving llm controllability by rule-based data recycling. arXiv preprint arXiv:2406.15938.

Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. 2024b. Can LLMs speak for diverse people? tuning LLMs via debate to generate controllable controversial statements. In Findings of the Association for Computational Linguistics ACL 2024, pages 16160–16176, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Ming Li, Lichang Chen, Juhai Chen, Shuai He, Jiuxiang Gu, and Tianyi Zhou. 2024c. Selective reflection-tuning: Student-selected data recycling for LLM instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024, pages 16189–16211, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Ming Li, Lichang Chen, Juhai Chen, Shuai He, and Tianyi Zhou. 2023a. Reflection-tuning: Recycling data for better instruction-tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.

Ming Li, Yanhong Li, Ziyue Li, and Tianyi Zhou. 2025. How instruction and reasoning data shape post-training: Data quality through the lens of layer-wise gradients. arXiv preprint arXiv:2504.10766.

Ming Li, Yong Zhang, Shuai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024d. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14255–14273, Bangkok, Thailand. Association for Computational Linguistics.

Ming Li, Yong Zhang, Zhitao Li, Juhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024e. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7595–7628, Mexico City, Mexico. Association for Computational Linguistics.

Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023b. Self-alignment with instruction back-translation. arXiv preprint arXiv:2308.06259.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023c. AlpacaEval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.

Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. 2024. Batchprompt: Accomplish more with less. In The Twelfth International Conference on Learning Representations.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023a. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. Preprint, arXiv:2303.16634.

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. Preprint, arXiv:2308.07074.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773.

OpenAI. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. Preprint, arXiv:2304.03277.

Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Dawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etxabe, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris E. Emeze, Christopher Klamm, Colin Leong, Daniel Alexander van Strien, David Ifeoluwa Adekani, Dragomir R. Radev, Eduardo Gonz’alez Ponferrada, Efrat Levkovitz, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady ElSahar, Hamza Benyamina, Hieu Trung Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Josephine L. Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro von Werra, Leon Weber, Long Phan, Loubna Ben Allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, Mar’ia Grandury, Mario vSavsko, Max Huang, Maximin Coavoux, and Mayank Singh. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100.

Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation metrics in the era of GPT-4: Reliably evaluating large language models on sequence to sequence tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8776–8788, Singapore. Association for Computational Linguistics.
Rohan Taori, Ishan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvensky, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Milligan, Lars Lowe Sjøesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Worth, Michael Sharman, Nikolai Chineav, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahlitnez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimentenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioibu, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023a. Large language models are not fair evaluators. Preprint, arXiv:2305.17926.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormalabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi La, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.

Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. 2024. Lamini-Im: A diverse herd of distilled models from large-scale instructions. Preprint, arXiv:2304.14402.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. Preprint, arXiv:2304.12244.

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng
Tao, and Tianyi Zhou. 2024a. A survey on knowledge distillation of large language models. ArXiv, abs/2402.13116.

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. Preprint, arXiv:2406.08464.

Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163–7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruixiang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. Preprint, arXiv:2303.18223.

Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L. Zhang. 2024. A preliminary study of the intrinsic relationship between complexity and alignment. Preprint, arXiv:2308.05696.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024a. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. Preprint, arXiv:2309.11998.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024b. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. Preprint, arXiv:2305.11206.
A  Qualitative Examples

To better understand the differences between the responses generated by the baseline model and the Mosaic-IT model, a pair of qualitative examples are presented in Figure 4 (baseline model) and Figure 5 (Mosaic-IT model).
Identify some of the main components of a computer network such as hosts, routers, switches, hubs, links, and interfaces. Explain how they communicate and coordinate with each other using protocols and standards such as TCP/IP, HTTP, FTP, DNS, DHCP, and ARP. Describe how data is transmitted and received over a network using concepts such as packets, frames, headers, addresses, ports, sockets, and checksums.

A computer network is composed of several components that work together to facilitate communication and data transfer between devices. These components include:
1. Hosts: A host is a device that is connected to the network, such as a computer, smartphone, or printer. Each host has a unique IP address, which is used to identify it on the network.
2. Routers: A router is a device that connects multiple networks together and directs traffic between them. It uses protocols such as IP to route data packets to their destination.
3. Switches: A switch is a device that connects multiple devices together on a single network. It uses MAC addresses to direct data packets to their destination.
4. Hubs: A hub is a device that connects multiple devices together on a single network. It broadcasts data packets to all connected devices, which allows them to communicate with each other.
5. Links: A link is the physical connection between two devices on a network. It can be a cable, a wireless connection, or a combination of both.
6. Interfaces: An interface is the part of a device that connects it to the network. It can be a network card, a USB port, or a wireless card.

All of these components communicate and coordinate with each other using protocols and standards such as TCP/IP, HTTP, FTP, DNS, DHCP, and ARP. These protocols define how data is transmitted and received over a network, how devices are addressed, and how traffic is routed.
Data is transmitted over a network in packets or frames. Each packet or frame contains a header, which includes information such as the destination IP address, the source IP address, and the type of data being transmitted. The data is then transmitted over the network, and the receiving device uses the information in the header to identify the data and direct it to the appropriate application or program.

In summary, a computer network is composed of several components that work together to facilitate communication and data transfer. These components communicate and coordinate with each other using protocols and standards, and data is transmitted over the network in packets or frames, using headers to identify the destination, source, and type of data.

Figure 4: An example from the WizardLM test set with the response generated by the Baseline model.
A computer network is made up of several components that work together to facilitate communication and data transfer between devices. Some of the main components include:
1. Hosts: These are the devices that are connected to the network and use its resources, such as computers, printers, and servers.
2. Routers: Routers are devices that direct traffic between networks, and are used to connect multiple networks together.
3. Switches: Switches are used to direct traffic within a network, and are used to connect multiple devices together on a single network.
4. Hubs: Hubs are devices that connect multiple devices together on a single network, and are used to distribute data to multiple devices at once.
5. Links: Links are the physical connections between devices on a network, such as Ethernet cables or wireless connections.
6. Interfaces: Interfaces are the points of connection between the network and the devices on it, such as network cards or wireless adapters.

For these components to communicate and coordinate with each other, they use a variety of protocols and standards. Protocols are sets of rules that dictate how data is transmitted and received, while standards are agreed-upon methods or formats that allow for interoperability between different systems. Some common protocols and standards used in computer networking include:
- TCP/IP: This is a suite of protocols that is used to communicate between devices on a network. It includes protocols for addressing, routing, and transporting data.
- HTTP: This is a protocol used for accessing and transferring data over the web. It is used to communicate between web servers and clients.
- FTP: This is a protocol used for transferring files over a network. It is used to communicate between FTP servers and clients.
- DNS: This is a protocol used for translating domain names to IP addresses. It is used to coordinate communication between devices on a network.
- DHCP: This is a protocol used for assigning IP addresses to devices on a network. It is used to coordinate communication between devices on a network.
- ARP: This is a protocol used for finding the MAC address of a device on a network. It is used to coordinate communication between devices on a network.

When data is transmitted over a network, it is broken down into small packets, each of which contains information about where it needs to go and what it contains. These packets are sent over the network, hopping from one device to the next until they reach their destination. At each hop, the device examines the packet’s header to see where it needs to go next, and then forwards it along the appropriate link. Once the packet reaches its destination, the receiving device examines the header to see if it is the intended recipient. If it is, the device will extract the data from the packet and use it, otherwise, the packet will be discarded.
To ensure that the data in the packets is not corrupted during transmission, checksums are used. A checksum is a mathematical calculation performed on the data in the packet, which produces a value that can be used to verify that the data has not been altered during transmission. If the checksum of the data in a packet is incorrect when it arrives at its destination, it is discarded, and the sender is notified of the error.

In summary, data is transmitted over a network by being broken down into small packets, each of which contains information about where it needs to go and what it contains. These packets are sent hopping from one device to the next, using a variety of protocols and standards to coordinate their transmission and reception. Checksums are used to ensure that the data in the packets has not been corrupted during transmission, and any errors are detected and corrected.
B Experimental Setup

B.1 Implementation Details
For the three base pre-trained models, Llama2-7B, Llama2-13B (Touvron et al., 2023b), and Mistral-7B (Jiang et al., 2023), we utilize the prompt and code base from Vicuna (Chiang et al., 2023) and flash attention (Dao et al., 2022). The overall training arguments are aligned with the common training configuration. The Adam optimizer (Kingma and Ba, 2017) is utilized with the batch size of 128 and with the max token length of 2048. When training the baseline models Llama2-7B and Llama2-13B, the maximum learning rate is set to \(2 \times 10^{-5}\) with the warmup rate as 0.03 for 3 epochs. When training the baseline models on Mistral-7B, the maximum learning rate is set to \(1 \times 10^{-5}\) with the warmup rate as 0.1 for 3 epochs. For the three models, Llama-3-8B (Dubey et al., 2024), Phi-3 (Abdin et al., 2024), and Gemma2-2B (Team et al., 2024), we utilize the code base from LLaMA-Factory (Zheng et al., 2024b). The max token length is set with 4096 following the modern settings and we train the model for 2 epochs. Other parameters are kept the same as the above.

When training with Mosaic-IT, we run the mosaic process \(n\) times for each experiment to simulate \(n\) epochs of training, \(n\) represents the number of epochs trained on baseline models, to ensure the alignment of overall data sample counts. Then these augmented data are mixed together and used for training 1 epoch while all other configurations are kept the same as baselines.

B.2 Training Dataset
The Alpaca dataset (Taori et al., 2023) comprises 52,000 instruction-following samples and is constructed utilizing the self-instruct paradigm (Wang et al., 2023b). This dataset was produced by employing OpenAI’s text-davinci-003 model. Characterized as a classical dataset with moderate quality attributes, the Alpaca dataset serves as an initial platform to validate our methodology. To further substantiate our approach using a dataset of inherently high quality, we also applied our method to the Alpaca-GPT4 dataset (Peng et al., 2023), which features responses generated by GPT4. The WizardLM dataset (Xu et al., 2023) is also utilized in our method, which contains 70,000 samples created by the evolution algorithm proposed by them. With ChatGPT-3.5 utilized, the data quality on WizardLM is largely guaranteed. The Vicuna 1M dataset (Zheng et al., 2024a) is a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. Due to the computation budget, 300k instances are randomly sampled for our experiments. Magpie dataset (Xu et al., 2024b) is the most recent SOTA synthetic dataset with 300k samples.

B.3 Evaluation Metrics
Pair-wise Comparison by using powerful LLMs like GPT-4 is recently widely accepted and becoming a common practice (Touvron et al., 2023b; Chiang et al., 2023; Dettmers et al., 2023; Liu et al., 2023b; Chiang and Lee, 2023). The evaluation of responses from LLMs, especially in open-domain contexts where definitive ground truth is hard to establish, continues to be an intricate and evolving research domain. Recent studies, however, have indicated a notable alignment between GPT-4’s performance evaluations and human assessments (Zheng et al., 2023; Li et al., 2023c; Sottana et al., 2023), thereby establishing a credible foundation for this evaluative methodology. We adopted test instruction sets from WizardLM (Xu et al., 2023), comprising 218 diverse, human-curated instructions for pair-wise comparison. We directly follow the evaluation framework proposed by (Chen et al., 2023; Li et al., 2024e), which evaluates responses on a scale spanning from 1 to 10 across multiple dimensions. To further address positional bias, as discussed by (Ko et al., 2020; Wang et al., 2023a), the comparison is conducted in two distinct sequences, LLM1’s response first and then LLM2’s response first, ensuring a fair assessment of model performance. Evaluation outcomes are categorized into ‘win-tie-loss’ for each instruction. The detailed evaluation prompt is provided in Appendix H.

Open LLM Leaderboard, employing the evaluation framework from Eval Harness (Gao et al., 2021), offers a detailed and systematic approach to assessing the capabilities of generative language models through a set of diverse evaluation tasks. This methodology zeroes in on four pivotal benchmarks: ARC (Clark et al., 2018), Hellaswag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and TruthfulQA (Lin et al., 2022). These benchmarks collectively provide a comprehensive evaluation of the models’ reasoning abilities, their grasp of common-sense knowledge, and their accuracy in presenting factual information. Consequently, the leaderboard presents valuable insights.
Alpaca-Eval Leaderboard, leveraging the AlpacaFarm evaluation dataset, presents a dependable and efficient automated evaluation tool for LLMs (Li et al., 2023c; Dubois et al., 2023). This tool benchmarks the responses generated by LLMs against those from Davinci003, focusing on the models’ ability to adhere to generic user instructions.

MT-Bench (Multi-turn Benchmark) (Zheng et al., 2023) is a benchmark tool designed for automated evaluating LLMs in multi-turn dialogue settings. It focuses on analyzing conversation flow and the model’s ability to follow instructions with 80 high-quality, multi-turn questions.

IFEval (Instruction-Following Eval) (Zeng et al., 2024) is a straightforward and easy-to-produce evaluation benchmark focusing on a set of “verifiable instructions”. It contains 25 types of verifiable instructions and 541 prompts, with each prompt containing one or multiple verifiable instructions.

Human Evaluation is further implemented to substantiate the superiority of our approach based on the WizardLM test set. The test set contains 100 samples randomly sampled from the original WizardLM test set. Three human evaluators were tasked with comparing the outputs generated by the models under consideration, using the same criteria as in the previous pairwise evaluation. Each evaluator was presented with three response options: Win, Tie, and Loss. The final outcomes were determined by a majority vote.
C  Detailed Ablation Studies

In this section, extensive ablation experiments are conducted on Mistral-7B using the Alpaca-GPT4 dataset to verify our method. We utilize Pair-wise comparison for evaluation.

C.1  Ablation on Mosaic Strategies

Ablation on Mosaic Strategies is presented in Table 6a. “Primary” represents the Primary Mosaic Strategy. The winning score of this setting is greater than 1.0, indicating a better performance compared with the baseline method. This comparison directly verifies the effectiveness of the idea of introducing multiple instructions during training, which complicates the instructions at no cost and improves the instruction-following ability of LLMs. “Format” represents the Format Strategy. Although the winning score is only slightly greater than the naive version, this version makes it possible for LLMs to follow the customized user-defined formats, indicating great potential for the controllability of LLMs. Moreover, the format version can be easily used with other types of meta instructions, showing great extensibility. “Permute” represents the Permute Strategy that builds on the Format Strategy with a probability of 1/2, similar to “Maskout”. “Permute/Maskout” represents our default setting, where the Permute or Maskout Strategies are utilized together with the Format Strategy with a probability of 1/3. All these 3 settings show higher performance than the format version, indicating the effectiveness of Advanced Mosaic Strategies, which define more complicated meta instructions.

<table>
  <tr>
    <th></th>
    <th>Winning Score</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Lose</th>
  </tr>
  <tr>
    <td>Primary</td>
    <td>1.261</td>
    <td>110</td>
    <td>55</td>
    <td>53</td>
  </tr>
  <tr>
    <td>Format</td>
    <td>1.284</td>
    <td>109</td>
    <td>62</td>
    <td>47</td>
  </tr>
  <tr>
    <td>Permute</td>
    <td>1.334</td>
    <td>118</td>
    <td>55</td>
    <td>45</td>
  </tr>
  <tr>
    <td>Maskout</td>
    <td>1.376</td>
    <td>121</td>
    <td>58</td>
    <td>39</td>
  </tr>
  <tr>
    <td>Permute/Maskout</td>
    <td>1.349</td>
    <td>123</td>
    <td>48</td>
    <td>47</td>
  </tr>
</table>

(a) Ablation on Mosaic-IT strategies.

<table>
  <tr>
    <th></th>
    <th>Winning Score</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Lose</th>
  </tr>
  <tr>
    <td>Max Count = 2</td>
    <td>0.989</td>
    <td>70</td>
    <td>75</td>
    <td>73</td>
  </tr>
  <tr>
    <td>Max Count = 4</td>
    <td>1.142</td>
    <td>92</td>
    <td>65</td>
    <td>61</td>
  </tr>
  <tr>
    <td>Max Count = 6</td>
    <td>1.303</td>
    <td>111</td>
    <td>62</td>
    <td>45</td>
  </tr>
  <tr>
    <td>Max Count = 8</td>
    <td>1.294</td>
    <td>112</td>
    <td>58</td>
    <td>48</td>
  </tr>
  <tr>
    <td>Max Count = 10</td>
    <td>1.349</td>
    <td>123</td>
    <td>48</td>
    <td>47</td>
  </tr>
  <tr>
    <td>Max Count = 12</td>
    <td>1.376</td>
    <td>124</td>
    <td>52</td>
    <td>42</td>
  </tr>
</table>

(b) Ablation on the Max Number of Instructions.

Table 6: Ablation on (a) Mosaic-IT strategies and (b) Max Number of Instructions.

and maskout strategies, which benefit LLMs’ instruction-following capability.

<table>
  <tr>
    <th></th>
    <th>Winning Score</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Lose</th>
    <th>Mix ≤ 5</th>
  </tr>
  <tr>
    <td>Fix</td>
    <td>0.982</td>
    <td>90</td>
    <td>34</td>
    <td>94</td>
    <td>2.39%</td>
  </tr>
  <tr>
    <td>Exponential</td>
    <td>0.995</td>
    <td>94</td>
    <td>29</td>
    <td>95</td>
    <td>2.58%</td>
  </tr>
  <tr>
    <td>Pareto</td>
    <td>1.417</td>
    <td>129</td>
    <td>51</td>
    <td>38</td>
    <td>8.94%</td>
  </tr>
  <tr>
    <td>Log-normal</td>
    <td>1.431</td>
    <td>136</td>
    <td>40</td>
    <td>42</td>
    <td>6.83%</td>
  </tr>
  <tr>
    <td>Logistic</td>
    <td>1.417</td>
    <td>123</td>
    <td>49</td>
    <td>46</td>
    <td>15.84%</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>1.349</td>
    <td>123</td>
    <td>48</td>
    <td>47</td>
    <td>51.45%</td>
  </tr>
</table>

Table 7: Ablation on the Distribution of Number of Instructions. The distribution formula and data counts for different settings are shown in Appendix E. “Mix ≤ 5” represents the percentage of samples with the number of instructions less or equal to 5.

![Ablation on the Distribution of Number of Instructions, the visualization of distributions.](page_1012_1342_388_180.png)

Figure 6: Ablation on the Distribution of Number of Instructions, the visualization of distributions.

C.2  Ablation on the Max Number of Instructions

Ablation on the Max Number of Instructions is presented in Table 6b, including the pair-wise comparison values. As shown in the table, when the max number is set as 2, i.e., at most 2 instructions/responses are concatenated together, the performance is almost the same as the baseline, indicating the ineffectiveness. However, when the max number grows, the corresponding winning scores also grow consistently. This trend shows that the more instructions concatenated together, the better the instruction-following ability. We hypothesize that, with the growth of the number of instructions, the overall instruction becomes much harder to follow, especially for the permute

C.3  Ablation on the Distribution of Number of Instructions

Ablation on the Distribution of Number of Instructions is presented, including the pair-wise comparison values in Table 7 and detailed number distribution comparisons in Figure 6, which aims at identifying how this count distribution affects the performance of our method. The detailed
distribution formula and data counts are provided in the Appendix E. “Fix” represents the setting where all the overall instructions are concatenated with a fixed number of instructions, which we set as 10 unless the overall instructions exceed the max length limit. “Exponential” represents the setting where the number of instructions is sampled following the exponential distribution. Under these two settings, less than 3% of the overall instructions are concatenated by less than or equal to 5 original instructions. The lack of few-instruction concatenated samples negatively affects the LLMs’ ability to follow the single instruction, which is employed by most of the existing evaluation methods, leading to worse performance. “Pareto”, “Log-normal”, and “Logistic” represents the corresponding distribution that are utilized for sampling. Different from the above two settings, approximately 10% of the overall instructions are composed of fewer original instructions, thus ensuring the LLMs are trained with samples with sufficiently diverse lengths, resulting in optimal performances. “Uniform” is our default setting, representing using the uniform distribution where different numbers are sampled evenly. In this situation, the LLMs are trained with samples with the most diverse lengths, thus avoiding the LLMs overfitting to simple lengthy responses.

C.4 Ablation on Semantic Grouping

To demonstrate that including other concatenation methods could further strengthen our argument, we further conduct experiments using a semantic grouping approach based on the Alpaca-GPT4 dataset to fine-tune Mistral.

Specifically, we utilize a sentence transformer model all-mpnet-base-v2\(^1\) to obtain the semantic embedding for each sample in the dataset, and then we apply the K-means algorithm to group these data samples into multiple clusters. To ensure enough samples per cluster, we set K=52 as the dataset contains 52k samples in total. Given the clusters, each concatenated sample is composed of multiple samples randomly drawn from the same cluster. We keep using the same training hyperparameters as in our main experiments. As shown in Table 8, we report the performance on two evaluation metrics: pair-wise comparison and Alpaca Eval.

\(^1\)https://huggingface.co/sentence-transformers/all-mpnet-base-v2

The semantic concatenation can still outperform the non-mosaic baseline by a large margin, indicating the effectiveness and potential of our Mosaic-IT augmentations and tasks. The semantic concatenation method has a slightly lower performance than the pure-random concatenation method, on pair-wise comparison and Alpaca Eval 2 scores. However, it achieves a much higher Alpaca Eval 2 (LC) score. This result suggests that the response quality of the model trained with semantic concatenation is on par with pure-random, but the response length is shorter and more condensed. We found that semantic grouping leads to clusters with highly different average lengths of samples: The largest average length is 316.7 tokens, while the smallest is 31.4 tokens. This discrepancy makes the lengths of Mosaic-IT concatenated samples more diverse, resulting in a better trade-off between the quality and length of the responses.
<table>
  <tr>
    <th>Method</th>
    <th>Alpaca Eval 2 (LC)</th>
    <th>Alpaca Eval 2</th>
    <th>Pair-wise Compare (with non-mosaic)</th>
    <th>Pair-wise Compare (with pure-random)</th>
  </tr>
  <tr>
    <td>Pure-random Concatenation</td>
    <td>5.00</td>
    <td><b>7.81</b></td>
    <td><b>1.349</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Concatenation with Semantic Groups</td>
    <td><b>7.80</b></td>
    <td>6.51</td>
    <td>1.275</td>
    <td>0.936</td>
  </tr>
</table>

Table 8: Comparison with Semantic grouping
D Multi-Instruction Evaluation

To verify our trained models’ capability to follow multiple instructions and meta-instructions in one inference, we create a test set of compositional instructions from WizardLM test sets using Mosaic-IT. For simplicity, we name this new test setting as Mosaic Task, which evaluates LLMs’ capability to follow multiple instructions with additional diverse constraints (meta-instructions). One example of Mosaic Task is shown as follows.

Example of Mosaic Task

System Prompt
You are a helpful and precise assistant for providing the answer.

User Prompt
Respond to each of the following instructions in reverse of the original order.
[Ins1]
[Ins2]
[Ins3]

Figure 7: The prompt we used to request GPT4-Turbo to evaluate the responses.

We use the success rate(%) to evaluate the performance of models on the Mosaic task. A response is successful if it follows the meta-instruction and no instruction is ignored (unless the meta-instruction masks it). In the table below, we report the success rate (%) of LLMs following three meta-instruction strategies, i.e., Format, Permute, and Maskout, on compositional augmentations of different numbers of instructions (i.e., 3, 5, 7 instructions). We report the success rates of GPT4o, two base models, and their Mosaic-IT finetuned versions, as shown in Table 9.

The results expose the weaknesses of existing LLMs on Mosaic-IT tasks and show that training on Mosaic-IT augmentations can significantly improve performance. Specifically, existing LLMs, even GPT4o, can not perfectly follow multiple instructions with diverse constraints, not to mention other open-source models like Llama3 finetuned on datasets such as Magpie. These results further demonstrate the difficulty and complexity of Mosaic-IT tasks for existing LLMs, indicating the novelty of our method. The compositional reasoning capability required by Mosaic-IT tasks cannot be covered by the capabilities of base LLMs and existing instruction-tuning datasets. For example, the success rates of Mistral + Alpaca-GPT4 (baseline) and Llama3 + Magepie (baseline) are similar, although Llama3 + Magepie has relatively better general instruction-following capabilities among them.

Our method can bridge the significant gap and enhance LLMs’ capability to follow multiple instructions with diverse constraints. Moreover, our data augmentation is cost-free and does not take any effort from humans or models.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">3 Instructions</th>
    <th colspan="3">5 Instructions</th>
    <th colspan="3">7 Instructions</th>
  </tr>
  <tr>
    <th>Format</th>
    <th>Permute</th>
    <th>Maskout</th>
    <th>Format</th>
    <th>Permute</th>
    <th>Maskout</th>
    <th>Format</th>
    <th>Permute</th>
    <th>Maskout</th>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>59.17</td>
    <td>55.05</td>
    <td>41.46</td>
    <td>56.88</td>
    <td>51.38</td>
    <td>26.13</td>
    <td>29.82</td>
    <td>37.16</td>
    <td>24.27</td>
  </tr>
  <tr>
    <td>Mistral + Alpaca-GPT4 (baseline)</td>
    <td>20.18</td>
    <td>3.67</td>
    <td>3.25</td>
    <td>10.09</td>
    <td>2.75</td>
    <td>5.41</td>
    <td>7.34</td>
    <td>0.92</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>Mistral + Alpaca-GPT4 (mosaic)</td>
    <td>98.32</td>
    <td>66.51</td>
    <td>69.11</td>
    <td>95.87</td>
    <td>60.55</td>
    <td>67.57</td>
    <td>97.25</td>
    <td>64.68</td>
    <td>66.02</td>
  </tr>
  <tr>
    <td>Llama3 + Magepie (baseline)</td>
    <td>16.06</td>
    <td>8.26</td>
    <td>7.32</td>
    <td>9.63</td>
    <td>1.38</td>
    <td>5.41</td>
    <td>5.50</td>
    <td>2.75</td>
    <td>3.88</td>
  </tr>
  <tr>
    <td>Llama3 + Magepie (mosaic)</td>
    <td>97.71</td>
    <td>79.82</td>
    <td>84.55</td>
    <td>94.95</td>
    <td>72.94</td>
    <td>77.48</td>
    <td>76.61</td>
    <td>61.01</td>
    <td>85.44</td>
  </tr>
</table>

Table 9: Performance comparison across multiple instruction settings.
E  Detailed Distribution for Ablation on Mixture Distribution

E.1  Distribution description
The detailed distribution descriptions and formulas are provided below.

Exponential Distribution2: The exponential distribution is a continuous probability distribution used to model the time or space between events in a Poisson process. The probability density function (PDF) of the exponential distribution is:

\[
f(x; \lambda) = \lambda e^{-\lambda x} \quad \text{for } x \geq 0,
\]

where \( \lambda = 1 \) by default in our setting. We will resample with this distribution if the sampled value \( x_{sample} \) is greater than \( k_{max} \).

Log-normal Distribution3: The log-normal distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. It is often used to model variables that are positively skewed, such as income, stock prices, and other financial data. The probability density function (PDF) for a log-normal distribution is given by:

\[
f(x; \mu, \sigma) = \frac{1}{x \sigma \sqrt{2\pi}} \exp \left( -\frac{(\ln x - \mu)^2}{2\sigma^2} \right) \quad \text{for } x > 0
\]

where \( \mu = 0 \) and \( \sigma = 0 \) by default in our setting. We will resample with this distribution if the sampled value \( x_{sample} \) is greater than \( k_{max} \).

Logistic Distribution4: The logistic distribution is a continuous probability distribution used in various fields, including logistic regression, modeling growth, and in some cases as an alternative to the normal distribution due to its heavier tails. The probability density function (PDF) for the logistic distribution is given by:

\[
f(x; \mu, s) = \frac{e^{-(x-\mu)/s}}{s (1 + e^{-(x-\mu)/s})^2}
\]

where \( \mu = 0 \) and \( s = 2 \) by default in our setting. We will resample with this distribution if the sampled value \( x_{sample} \) is greater than \( k_{max} \).

Pareto Distribution5: The Pareto II or Lomax distribution is a shifted Pareto distribution. It can be considered a simplified version of the Generalized Pareto distribution, with the scale set to one and the location set to zero. The probability density function (PDF) for the Pareto distribution is:

\[
f(x; \alpha) = \frac{\alpha m^\alpha}{x^{\alpha+1}} \quad \text{for } x \geq m,
\]

where \( m = 1 \) and \( \alpha = 1 \) by default in our setting. We will resample with this distribution if the sampled value \( x_{sample} - 1 \) is greater than \( k_{max} \).

After getting \( x_{sample} \), a floor function will be utilized to get the corresponding integer and the final concatenation count \( k = k_{max} - \lfloor x_{sample} \rfloor \).

E.2  Distribution visualization
The detailed data counts for different distributions are provided in Figure 8.
(a) Fix Max Number

(b) Exponential Distribution

(c) Log-normal Distribution

(d) Logistic Distribution

(e) Pareto Distribution

(f) Uniform Distribution

Figure 8: Bar plots of detailed data counts for different distributions in the Ablation on the Numbers of Instructions: (a) Fix Max Number, (b) Exponential Distribution, (c) Log-normal Distribution, (d) Logistic Distribution, (e) Pareto Distribution, (f) Uniform Distribution.
F  Related Work

Earlier research in instruction tuning primarily centered on constructing expansive, high-quality datasets through intensive curation by human experts, a process both time-consuming and labor-intensive (Khashabi et al., 2020; Ye et al., 2021; Wei et al., 2022; Wang et al., 2022; Du et al., 2022). Motivated by the success of Alpaca (Taori et al., 2023), recent studies have explored automated approaches for developing instruction-tuning datasets.

Instruction Data Improvement: WizardLM (Xu et al., 2023) first proposes an Evol Algorithm to complicate the existing data and reach supreme performance. LaMini-LM (Wu et al., 2024) innovatively generates "Topic-Guided" instructions utilizing Wiki data. Tree-Instruct (Zhao et al., 2024) preliminarily explores the relationship between instruction complexity and Alignment and proposes adding nodes to complicate the instruction. UltraChat (Ding et al., 2023) establishes broad thematic scopes, systematically generating numerous instructions within each. Reflection-Tuning (Li et al., 2023a) sequentially refines both instructions and responses by focusing on specific evaluative criteria. DEITA (Liu et al., 2023a) utilizes ChatGPT to diversify and then select the data. Selective Reflection-Tuning (Li et al., 2024c) proposes a teacher-student collaborative pipeline to improve and select the data. Instruction Fusion (Guo et al., 2024) proposes to utilize ChatGPT4 to merge two distinct instructions for further complexity enhancement. These advancements showcase a shift towards automating the generation and refinement of datasets, reducing reliance on human labor.

Instruction Data Selection: It is widely accepted that "quality is all you need" (Touvron et al., 2023b; Zhou et al., 2023) for instruction tuning. LIMA (Zhou et al., 2023) demonstrates that merely 1,000 human-carefully-curated, high-quality training instances can substantially enhance the instruction-following performance. InsTag (Lu et al., 2023) employs the proprietary model, ChatGPT, to tag instruction data and select data with complex tags. Alpagasus (Chen et al., 2023) utilizes proprietary LLMs chatGPT and Claude2 to directly assess the quality of instruction tuning data. Cherry LLM (Li et al., 2024e) proposes the Instruction-Following Difficulty (IFD) scores to assess the difficulty of the instructions, which is a self-guided method in which no extra LLMs are utilized. Motivated by Humpback (Li et al., 2023b), Selective Reflection-Tuning (Li et al., 2024c) extends the IFD score to a reverse version, focusing on the feasibility of responses. (Du et al., 2023) and (Bukharin and Zhao, 2023) utilize reward models as the base scores for measuring data quality. DEITA (Liu et al., 2023a) experiments on several different data selection metrics and builds a dataset with high quality. Superfiltering (Li et al., 2024d) reveals the consistency between weak and strong language models in perceiving instruction difficulty, making the filtering process much more efficient. A recent work (Li et al., 2025) tries to unify the different data evaluation metrics through the lens of gradients. All these works are devoted to distinguishing and selecting good data samples from bad ones for instruction tuning.
G   Predefined Rules

Examples of predefined formats can be found in Table 10 and detailed predefined rule descriptions can be found in Table 11.
<table>
  <tr>
    <th>Serial Digit</th>
    <th>Parsing Bracket</th>
    <th>Parsing Text</th>
    <th>Assembled Examples</th>
  </tr>
  <tr>
    <td>i</td>
    <td>(text)</td>
    <td>BEGIN, END</td>
    <td>1. (BEGIN)<i>response</i>(END)</td>
  </tr>
  <tr>
    <td>(i)</td>
    <td>[text]</td>
    <td>START, END</td>
    <td>(1). [START<i>response</i>[END]</td>
  </tr>
  <tr>
    <td>[i]</td>
    <td>[text]</td>
    <td>RESPONSE, END</td>
    <td>[1]. (RESPONSE)<i>response</i>(END)</td>
  </tr>
  <tr>
    <td>{i}</td>
    <td><text>></td>
    <td>RESPONSE, END OF RESPONSE</td>
    <td>{1}. <<RESPONSE>><i>response</i><<<END OF RESPONSE>></td>
  </tr>
  <tr>
    <td><i>i</i></td>
    <td>text</td>
    <td>OPEN, CLOSE</td>
    <td><i>i</i>>. IOPEN<i>response</i>CLOSEI</td>
  </tr>
  <tr>
    <td>###</td>
    <td>[text]</td>
    <td>OPEN RESPONSE, CLOSE</td>
    <td>###!. [OPEN RESPONSE]<i>response</i>[CLOSEI</td>
  </tr>
  <tr>
    <td>#i</td>
    <td>[text]</td>
    <td>INITIATE, TERMINATE</td>
    <td>#i!. (INITIATE)<i>response</i>(TERMINATE)</td>
  </tr>
  <tr>
    <td>####</td>
    <td>#text#</td>
    <td>START POINT, END POINT</td>
    <td>####. #START POINT#<i>response</i>#END POINT#</td>
  </tr>
  <tr>
    <td>lil</td>
    <td>*text*</td>
    <td>RES_START, RES_END</td>
    <td>lil!. *RES_START*<i>response</i>*RES_END*</td>
  </tr>
  <tr>
    <td>lll</td>
    <td>@text@</td>
    <td>RES, /RES</td>
    <td>lll!. @RES@<i>response</i>@/RES@</td>
  </tr>
</table>

Table 10: Examples of predefined formats, including the Serial Digit formats and Response Parsing formats. “i” represents the real number serial number, “text” represents the replaceable parsing text, and “response” represents the real response of the concatenated overall instructions/responses. The response parsing formats are composed of the parsing bracket and text. In each mosaic process, random formats will be sampled simulating the real-world user-defined formats. The last column represents the assembled examples using the formats in the same row.

<table>
  <tr>
    <th>Strategy</th>
    <th>Rule Name</th>
    <th>Rule Description</th>
  </tr>
  <tr>
    <td rowspan="8">Permute</td>
    <td>FIX</td>
    <td>Respond in the order of a provided list.</td>
  </tr>
  <tr>
    <td>REVERSE</td>
    <td>Respond in reverse of the original order.</td>
  </tr>
  <tr>
    <td>ALPHA</td>
    <td>Respond in the alphabetical order of the first letter of tasks.</td>
  </tr>
  <tr>
    <td>REVERSE_ALPHA</td>
    <td>Respond in the reverse alphabetical order of the first letter of tasks.</td>
  </tr>
  <tr>
    <td>LENGTH_WORD</td>
    <td>Respond according to the length (words) of tasks, respond to short ones first.</td>
  </tr>
  <tr>
    <td>REVERSE_LENGTH_WORD</td>
    <td>Respond according to the length (words) of tasks, respond to long ones first.</td>
  </tr>
  <tr>
    <td>LENGTH_CHAR</td>
    <td>Respond according to the length (characters) of tasks, respond to short ones first.</td>
  </tr>
  <tr>
    <td>REVERSE_CHAR_WORD</td>
    <td>Respond according to the length (characters) of tasks, respond to long ones first.</td>
  </tr>
  <tr>
    <td rowspan="2">Permute</td>
    <td>ODD_EVEN</td>
    <td>First respond to the odd-numbered tasks, then the even-numbered ones.</td>
  </tr>
  <tr>
    <td>EVEN_ODD</td>
    <td>First respond to the even-numbered tasks, then the odd-numbered ones.</td>
  </tr>
  <tr>
    <td rowspan="5">Maskout</td>
    <td>FIX</td>
    <td>Ignore the tasks provided in the list.</td>
  </tr>
  <tr>
    <td>WORD_LONG</td>
    <td>Ignore the longest one/several task(s) according to the word count.</td>
  </tr>
  <tr>
    <td>WORD_SHORT</td>
    <td>Ignore the shortest one/several task(s) according to the word count.</td>
  </tr>
  <tr>
    <td>ODD</td>
    <td>Ignore the odd-numbered tasks.</td>
  </tr>
  <tr>
    <td>EVEN</td>
    <td>Ignore the even-numbered tasks.</td>
  </tr>
</table>

Table 11: Predefined rules for the Permute and Maskout strategy. A random rule will be sampled for each mosaic process, which largely complicates and diversifies the mosaicked instructions.
H  Prompt for Evaluation

The detailed pair-wise comparison prompt for the pair-wise comparison is in Figure 9.

Prompt for Performance Evaluation

System Prompt
You are a helpful and precise assistant for checking the quality of the answer.

User Prompt
[Question]
Question
[The Start of Assistant 2’s Answer]
Answer 2
[The End of Assistant 2’s Answer]
[The Start of Assistant 2’s Answer]
Answer 2
[The End of Assistant 2’s Answer]

We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above.
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.

Figure 9: The prompt we used to request GPT4-Turbo to evaluate the responses.
1  Detailed Performance Scores on Llama3, Phi3 and Gemma2

The detailed performance scores on the Open LLM Leaderboard and IFEval, for Llama-3-8B, Phi-3, and Gemma2-2B.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="5">Open LLM Leaderboard ↑</th>
    <th colspan="4">IF Eval ↑</th>
  </tr>
  <tr>
    <th>Average</th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Prompt (S)</th>
    <th>Inst (S)</th>
    <th>Prompt (L)</th>
    <th>Inst (L)</th>
  </tr>
  <tr>
    <td rowspan="2">Llama-3-SB</td>
    <td>Vicuna</td>
    <td>Baseline</td>
    <td>52.51</td>
    <td>44.54</td>
    <td>70.66</td>
    <td>49.68</td>
    <td>45.18</td>
    <td>19.04</td>
    <td>30.70</td>
    <td>21.26</td>
    <td>33.45</td>
  </tr>
  <tr>
    <td></td>
    <td>Mosaic-IT</td>
    <td><b>55.62</b></td>
    <td>47.78</td>
    <td>73.77</td>
    <td>56.11</td>
    <td>44.83</td>
    <td><b>29.76</b></td>
    <td><b>43.17</b></td>
    <td><b>31.42</b></td>
    <td><b>45.56</b></td>
  </tr>
  <tr>
    <td rowspan="2">Phi-3</td>
    <td>Maggie</td>
    <td>Baseline</td>
    <td>56.15</td>
    <td>50.09</td>
    <td>71.29</td>
    <td>54.40</td>
    <td>48.84</td>
    <td>29.39</td>
    <td>40.76</td>
    <td>35.67</td>
    <td>47.72</td>
  </tr>
  <tr>
    <td></td>
    <td>Mosaic-IT</td>
    <td><b>60.13</b></td>
    <td>53.58</td>
    <td>76.62</td>
    <td>60.82</td>
    <td>49.52</td>
    <td><b>38.08</b></td>
    <td><b>49.64</b></td>
    <td><b>40.67</b></td>
    <td><b>52.76</b></td>
  </tr>
  <tr>
    <td rowspan="2">Gemma2-2B</td>
    <td>Vicuna</td>
    <td>Baseline</td>
    <td>62.06</td>
    <td>58.96</td>
    <td>76.48</td>
    <td>64.89</td>
    <td>47.89</td>
    <td>28.47</td>
    <td>40.29</td>
    <td>30.50</td>
    <td><b>43.17</b></td>
  </tr>
  <tr>
    <td></td>
    <td>Mosaic-IT</td>
    <td><b>62.30</b></td>
    <td>58.45</td>
    <td>77.66</td>
    <td>65.24</td>
    <td>47.87</td>
    <td><b>30.13</b></td>
    <td>39.57</td>
    <td><b>32.35</b></td>
    <td>41.85</td>
  </tr>
  <tr>
    <td rowspan="2">Maggie</td>
    <td>Baseline</td>
    <td>62.90</td>
    <td>59.30</td>
    <td>75.07</td>
    <td>65.89</td>
    <td>51.35</td>
    <td>39.56</td>
    <td>50.84</td>
    <td>44.36</td>
    <td>55.25</td>
  </tr>
  <tr>
    <td></td>
    <td>Mosaic-IT</td>
    <td><b>63.54</b></td>
    <td>60.23</td>
    <td>76.30</td>
    <td>66.14</td>
    <td>51.50</td>
    <td><b>42.33</b></td>
    <td><b>53.60</b></td>
    <td><b>50.83</b></td>
    <td><b>62.35</b></td>
  </tr>
  <tr>
    <td rowspan="2">Vicuna</td>
    <td>Baseline</td>
    <td>48.90</td>
    <td>43.43</td>
    <td>64.20</td>
    <td>41.50</td>
    <td>46.46</td>
    <td>20.51</td>
    <td>32.61</td>
    <td>23.66</td>
    <td>35.61</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>51.31</b></td>
    <td>46.33</td>
    <td>69.32</td>
    <td>44.29</td>
    <td>45.31</td>
    <td><b>21.44</b></td>
    <td><b>33.57</b></td>
    <td><b>24.03</b></td>
    <td><b>36.93</b></td>
  </tr>
  <tr>
    <td rowspan="2">Maggie</td>
    <td>Baseline</td>
    <td>46.37</td>
    <td>39.59</td>
    <td>60.71</td>
    <td>35.46</td>
    <td>49.75</td>
    <td>19.78</td>
    <td>29.74</td>
    <td>21.81</td>
    <td>32.49</td>
  </tr>
  <tr>
    <td>Mosaic-IT</td>
    <td><b>48.36</b></td>
    <td>39.33</td>
    <td>64.10</td>
    <td>39.87</td>
    <td>50.16</td>
    <td><b>19.78</b></td>
    <td><b>31.65</b></td>
    <td><b>22.18</b></td>
    <td><b>34.77</b></td>
  </tr>
</table>

Table 12: The performance comparison on more model families and datasets on all five automatic evaluation metrics. In IF Eval, P and I represent Prompt-level and Instruction-level accuracy.

Reviewer paper summaries: This paper introduces Mosaic Instruction Tuning (Mosaic-IT), a cost-free compositional data synthesis approach for improving the instruction-following abilities of Large Language Models (LLMs). Mosaic-IT generates complex, multi-step instructions by concatenating existing instruction-response pairs and applying various compositional strategies (Format, Permute, and Maskout) guided by meta-instructions. Experiments demonstrate that Mosaic-IT significantly enhances instruction-following performance while reducing training costs by about 80%, showcasing improvements across multiple LLM architectures and benchmarks without requiring additional human annotation or costly model-generated data.
Summary of strengths: Novel and Cost-Effective Approach: Mosaic-IT creatively synthesizes compositional data from existing instructions, significantly improving complexity and diversity without additional cost or reliance on expensive teacher models or human annotators.

Strong Empirical Results: The method consistently demonstrates substantial performance gains across multiple evaluation benchmarks and LLM architectures (e.g., Mistral-7B, Llama models, Phi-3, Gemma-2B), validating its generalization capability and effectiveness.

Efficiency and Scalability: Mosaic-IT achieves significant computational savings (~80% reduction in training costs), emphasizing its practical scalability, especially beneficial in resource-constrained scenarios.

Reduced Memorization and Overfitting: By diversifying instructions through compositional synthesis, Mosaic-IT addresses memorization issues commonly encountered in traditional instruction tuning, as indicated by smoother training loss curves.
Summary of weaknesses: •	The paper claims that existing instruction datasets lack diversity, but it fails to concretely illustrate the specific shortcomings of these datasets. Without providing explicit examples or analysis to support this claim, the argument regarding dataset insufficiency lacks conviction.
	•	The backbone models chosen for validation (e.g., Mistral-7B, Llama2-7B) are relatively outdated. The absence of experiments conducted on state-of-the-art, more powerful instruction-tuned models such as Qwen2.5-7B-Inst limits the generalizability and practical applicability of the findings.
Comments, suggestions and typos: Nothing
Overall assessment scores: 3
Metareview score: 3.5
Metareview: This paper proposes Mosaic Instruction Tuning (Mosaic-IT), a lightweight method of data augmentation for instruction-tuning LLMs. Mosaic-IT combines multiple instruction-response pairs into single, multi-instruction prompts with added meta-instructions that control formatting, ordering, and selective masking. The paper claims improvements in instruction-following performance and training efficiency, without requiring any additional human or LLM-generated data. The reviews were overall positive, citing novelty and practical impact but noting some experimental and clarity gaps.
Summary of reasons to publish: - **Computational Efficiency** The primary strength of Mosaic-IT lies in its simplicity and cost-efficiency. As multiple reviewers highlight, the method avoids reliance on expensive data generation pipelines by simply reusing existing data pairs in novel compositions.  The authors argue that training costs are reduced significantly, not by cutting down token volume but by reducing the number of training iterations. 

- **Empirical Breadth and Generalizability** Multiple reviewers commend the comprehensive experiments across diverse model architectures and Mosaic-IT demonstrates consistent gains in single-turn tasks across evaluation suites. While some reviewers wished for newer models like Qwen2.5-7B-Inst to be included, the existing coverage is still reasonably strong.