Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting

## Abstract

Abstract With the proliferation of large language models (LLMs), the comprehensive alignment of such models across multiple tasks has emerged as a critical area of research. Existing alignment methodologies primarily address single task, such as multi-turn dialogue, coding, mathematical problem-solving, and tool usage. However, AI-driven products that leverage language models usually necessitate a fusion of these abilities to function effectively in real-world scenarios. Moreover, the considerable computational resources required for proper alignment of LLMs underscore the need for a more robust, efficient, and encompassing approach to multi-task alignment, ensuring improved generative performance. In response to these challenges, we introduce a novel technique termed Mixture-of-Instructions (MoI), which employs a strategy of instruction concatenation combined with diverse system prompts to boost the alignment efficiency of language models. We have also compiled a diverse set of seven benchmark datasets to rigorously evaluate the alignment efficacy of the MoI-enhanced language model. Our methodology was applied to the open-source Qwen-7B-chat model, culminating in the development of Qwen-SFT-MoI. This enhanced model demonstrates significant advancements in generative capabilities across coding, mathematics, and tool use tasks.

## 1 Introduction

The rise of large language models (LLMs) has underscored the importance of effective training techniques, encompassing pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF), to expand their knowledge and ensure alignment with human values for greater reliability. However, despite advancements, LLMs often falter in specialized tasks such as coding, mathematics, and reasoning. With the growing access to high-quality, domain-specific datasets(Luo et al., ; Li et al., ; CodeFuse, ; Wei et al., ; Yue et al., ; Yu et al., ; Luo et al., ; Zeng et al., ; Yang et al., ; Qin et al., ), this study investigates how targeted SFT can boost LLM capabilities in these areas while preserving existing competencies.

We examined a case where the Qwen-7B-chat (Bai et al., )model failed to learn Python code for the Boyer-Moore majority vote algorithm. Training with standard code dataset proved ineffective, with the model producing suboptimal solutions due to prompt-induced knowledge conflicts. To our surprise, we found that simply changing to a different system prompt and retraining could easily overcome this issue, indicating that altering prompts can resolve knowledge conflicts and improve task performance across tasks.

Leveraging our initial findings, we developed a strategy that assigns unique system prompts to different tasks and integrates these varying prompts into a unified instruction set for model alignment. We introduce the Mixture of Instruction (MoI) methodology, a structured approach to amalgamate open-source datasets for comprehensive multi-task training, spanning dialogue, code, math, and agent-based tasks, each with bespoke prompts. This not only recasts training as a multi-task learning (MTL) endeavor but also tackles MTL challenges such as dataset bias, where improvement in one task could detriment another. Through MoI, we adeptly use diverse prompts to ensure balanced task performance, effectively navigating the complexities of MTL.

Ultimately, we integrated the MoI with multi-turn dialogue, code , mathematics and agent datasets for comprehensive training, resulting in a significantly enhanced language model—Qwen-SFT-MoI. This model exhibited substantial improvement across a range of benchmarks, including MT-Bench(Zheng et al., ), HumanEval(Chen et al., ), MBPP(Austin et al., ), MATH(Hendrycks et al., ), GSM8K(Cobbe et al., ), MMLU(Hendrycks et al., ), and T-EVAL(Chen et al., ), compared to the Qwen-7B-Chat model, validating our MoI strategy for further alignment on an already SFT model.

This work makes the following contributions:

- •
We identified and addressed conflicts between new and old knowledge during Supervised Fine-Tuning (SFT), by introducing modified system prompts that aid in the integration of new knowledge. This resulted in improved alignment with and performance on code generation task.
- •
Our Mixture of Instruction (MoI) method facilitates joint multi-task training, effectively reducing dataset bias and preserving conversational abilities. The efficacy of the MoI approach was substantiated through extensive experimentation.
- •
By applying MoI alongside high-quality datasets on the Qwen-7B-chat model, we developed the Qwen-SFT-MoI model. This model demonstrates superior performance in mathematical reasoning, code generation, tool usage, and chat benchmarks, thereby highlighting the effectiveness of MoI in enhancing SFT models.

## 2 Related work

### 2.1 LLMs for mathematical reasoning

Solving complex mathematical tasks is challenging for open-source LLMs. Research focuses on fine-tuning open-source models (e.g., Touvron et al. (; ); Bai et al. ()) with insights from advanced proprietary LLMs (OpenAI, ; ). WizardMath (Luo et al., ) employs a reinforced evol-instruct method, combining supervised fine-tuning with PPO training to enhance reasoning. Meanwhile, MAmmoTH (Yue et al., ) uses Chain of Thought and Program-of-Thought to enable LLMs to work with external tools, like Python interpreters, for mathematical problem-solving.

### 2.2 LLMs for code generation

LLMs have been specifically designed for code generation tasks, such as StarCoder (Li et al., ) and DeepSeek-Coder (Guo et al., ). Alternatively, code-generating LLMs can be derived from fine-tuning general-purpose models, exemplified by CodeLlama (Rozière et al., ) and WizardCoder(Luo et al., ). Our approach aligns with the latter, fine-tuning general-purpose LLMs to enhance code generation.

### 2.3 LLM for tool usage.

LLMs in real-world scenarios must adeptly select and apply tools from numerous APIs. Gorilla (Patil et al., ) pairs LLMs with a broad API set, while ToolAlpaca (Tang et al., ) documents 3938 tool uses from over 400 real-world APIs in 50 categories. ToolLLM (Qin et al., ) further contributes with ToolBench, a rich dataset featuring 16464 real-world APIs across 49 categories from RapidAPI Hub, accommodating diverse single and multi-tool use cases.

### 2.4 Multi-task learning

During multi-task learning, dataset bias (Hadsell et al., ) is a phenomenon where two or more tasks pull the model parameters in different directions, thus impacting the multi-task learning performance.
Mixture-of-Experts (MoE) framework (Shazeer et al., ; Fedus et al., ; Lepikhin et al., )provides a way to model this mechanism. Such architectures are designed to support conditional computation in which only certain weights of the network are activated per input as governed by a gating mechanism.In our work, we propose a method to mitigate dataset bias that focuses on the data rather than the model.

## 3 Mixture of instructions

### 3.1 System prompt matters

Our study extends an existing aligned model by advancing its proficiency in programming, mathematical reasoning, and tool utilization. Initially, we investigated the performance of the Qwen-7B-chat(Bai et al., ) model on the MT-Bench(Zheng et al., ) dataset, a multifaceted benchmark suite that evaluates models’ efficacy in eight diverse domains through GPT-4 scoring. Analysis of the Qwen-7B-chat model on MT-Bench’s coding questions revealed consistent yet suboptimal scores, particularly to specific question ID 127——”Write a function to find the majority element in a given integer array using the Boyer-Moore Voting Algorithm.”. Although Qwen-7B-chat successfully identified the majority element, its approach resulted in a higher time complexity of O(nlog(n)) and a space complexity of O(n), deviating from the expected O(n) time and O(1) space complexities of the algorithm.Subsequently, we discovered that Question ID 127 corresponds to an original problem from the LeetCode repository. Consequently, we downloaded 2612 LeetCode problems along with their solutions and trained the Qwen-7B-chat model using SFT. However, we observed that the model, even after SFT, still failed to answer this particular question.

Figure: Figure 2: The figure illustrates attention maps for the responses of three models to Question ID 127 in the MT-Bench, where greener regions signify higher attention levels. The maps show attention distribution across the system prompt, the question, and each model’s answer. Model (a) Qwen-7B-chat heavily focuses on the prompt and question but erroneously associates ’Boyer’ with frequency tracking, misrepresenting the Boyer-Moore algorithm. Model (b), post-SFT on coding data, still pays undue attention to the prompt and question, disregarding ’Boyer’ and fixating on iterative element search. Model (c), after SFT with an updated prompt, shifts attention towards generating an answer, correctly hinting at the use of the Boyer-Moore algorithm, which fundamentally tracks a candidate element.

As depicted in Figure 2, analysis of attention maps for both the original Qwen-7B-chat and SFT-enhanced models when processing a specific question revealed that the original Qwen-7B-chat model favored array traversal and frequency counting, divergent from the Boyer-Moore algorithm’s approach. The SFT model also disproportionately focused on array operations and unduly on the system prompt. This suggests that a consistent default prompt (”You are a helpful assistant”) during SFT may have inadvertently anchored the model’s attention, preventing it from engaging more relevant computational strategies. Introducing an atypical system prompt during SFT could recalibrate focus and facilitate the integration of novel knowledge.

To redirect the attention of a large language model, we replaced its system prompt with ’You are a programmer’ in the SFT regime. This change, as shown in Figure 2, led to a realigned attention map that concentrated on the key ’Boyer-Moore Algorithm’ principles, prompting accurate descriptions of the algorithm’s iteration and candidate tracking. The shift in focus from the prompt to the algorithmic core problem and solution was significant. Additionally, we evaluated the effects of varying SFT-driven system prompts on model performance using the HumanEval, MBPP, and MT-Bench coding benchmark. Table 3 conveys that the ’You are a programmer’ prompt engendered a notable uplift in code generation proficiency.

### 3.2 Balanced sampling diverse system prompting instructions

Upon validating that modifying the system prompt can enhance a model’s coding capabilities, we pondered whether a unique system prompt could stimulate a model’s distinct abilities. Consequently, we speculated whether combining datasets from various tasks according to their respective system prompt directives could enable a model to concurrently acquire competencies from different fields. To explore this, we gathered open-source datasets for four tasks: multi-turn dialogue, code, math, and agent, providing a specific system prompt for each. However, when we aggregated datasets from multiple tasks for joint training, we observed that the model trained on combined tasks did not perform as well as models trained on individual tasks separately2. This revealed a typical phenomenon in multi-task learning, where training on combined domain data can cause model bias towards a particular domain due to dataset bias, leading to improved performance in one domain at the expense of another.

An effective solution to the dataset bias within the multi-task learning field is the ’mixture of experts’ (MoE) approach. This method utilizes a gating mechanism and multiple expert feedforward neural networks (FFNs) to learn domain-specific knowledge, which is a significant research direction in language models. The principle of MoE is to use a mixture of models to learn a single instruction. We wondered whether a similar effect could be achieved by using a single model to learn a mixture of instructions. To address this, we proposed a balanced sampling scheme, a method that concatenates instructions in the order of their tasks. The structure of the balanced sampling method is illustrated in the figure 4 which sequentially collects instructions from datasets of various tasks, decodes these instructions into tokens, and combines them.

$$ $\mathcal{L}_{\text{SFT}}(\pi_{\theta})=-\mathbb{E}_{(x,y)\sim\mathcal{D}}[\log \pi_{\theta}(y\mid x)]$ (1) $$

The expression of 1 reveals that during the SFT phase. The equation represents the negative expected value of the log likelihood for the model $(\pi_{\theta})$ to correctly predict (y) given (x) under a specified data distribution $(\mathcal{D})$. Our task is to pre-partition datasets $\mathcal{D}_{\text{chat}}$, $\mathcal{D}_{\text{code}}$, $\mathcal{D}_{\text{math}}$, $\mathcal{D}_{\text{agent}}$ for each tasks, enabling the model to optimize all tasks simultaneously during a single back propagation pass.Figure 5 demonstrates the effectiveness of the balanced sampling approach in multi-task learning, achieving stable learning performance across all tasks.

Figure: Figure 3: Attention maps from a Qwen-7B-chat model’s first, middle, and last layers when processing mixture of instructions. The model attends not only to the current system prompt but also to initial tokens in concatenated instructions. By prefixing mixed instructions with the default system prompt, the model implicitly revisits it during learning of diverse prompts, thereby facilitating cross-domain knowledge transfer to the domain indicated by the default prompt.

Figure: Figure 4: (a). ”Sequence of Instructions” involves extracting instructions from a dataset, tokenizing them, and using padding tokens to reach a fixed maximum length for SFT in an LLM. (b). ”Concatenation of Instructions” merges several instructions into a longer single instruction, minimizing the need for padding and enhancing training efficiency. (c). ”Mixture of Instructions” employs balanced sampling from various datasets and concatenates instructions to meet a maximum token length, then prioritizes the instruction with the default system prompt at the start.

### 3.3 Instructions reordering with a default system prompt

While a balanced sampling scheme can mitigate dataset biases and foster domain-specific learning in models, users often seek a language model adept across tasks with just a default system prompt. To merge content from diverse system prompts into a cohesive domain, we suggest a novel method inspired by attention sinks (Xiao et al., ) and our attention mask performance analyses.

Our findings, depicted in Figure 3, show that language models from intermediate to final layers heavily focus on initial tokens when encountering new tokens. Leveraging this, we propose an innovative technique: reordering mixed instructions to position the instruction with the default system prompt ’You are a helpful assistant.’ at the start. This design is grounded on the idea of making the model, during SFT, explicitly recognize its domain-specific prompts while concurrently responding to a universal, default prompt.

This approach seamlessly integrates learning progress from various tasks under a singular, default domain framework. This enables the model to maximally utilize its reasoning abilities, even in the absence of domain-specific system prompts, providing a unified and effective learning trajectory.

### 3.4 Mixture of instructions

Ultimately, we propose a novel SFT scheme, Mixture of Instructions (MoI), comprising our balanced sampling and instruction reordering strategies.

Beyond the MoI approach, common methods for training datasets from multiple tasks include sequence instructions and instruction concatenation. Sequence instructions method entails feeding individual data entries into the language model for training, padding each entry to meet the prescribed context length required for the model’s training if its length is insufficient, with padding tokens excluded from the loss function calculation. On the other hand, instruction concatenation (also known as instruction packing) addresses the efficiency loss in sequence instructions caused by excessive padding when individual instructions are too short. By concatenating multiple instructions into a single entry while preserving the format of both single and multi-turn dialogues, instruction concatenation allows the language model to process more instructions per training instance, thus enhancing training efficiency without significantly impacting model performance.

Subsequent experiments 2 comparing MoI with concatenation of instructions validated the effectiveness of our approach. It alleviates dataset bias in multi-task learning and efficiently transfers knowledge learned from various system prompts to a default prompt, significantly enhancing model alignment and performance.

## 4 Experiments

### 4.1 Experiment setup

All of our experiments are performed on 8x80G A100 GPUs, using the Qwen-7B-chat model as the base for each independent experiment.We utilized the Huggingface Transformers(Wolf et al., ) library to execute our training and employed DeepSpeed(Rasley et al., ) Zero3 optimization to enhance memory efficiency.

### 4.2 Training Dataset

For training data, in our experiments, we exclusively used open-source datasets to validate the effectiveness of our method. The data we utilized comprises solely open-source datasets, which is shown in Table 1

**Table 1: All of our training data are sourced from open datasets downloaded from Huggingface, categorized into four tasks: multi-turn dialogue, code, mathematics, and tools usage. Custom system prompts were individually crafted for the dataset of each task to ensure precise alignment with their respective content.**
| Dataset | Task | System Prompt |
| --- | --- | --- |
| ultrachat200k (Ding et al., ) | Chat | You are a helpful assistant. |
| MagicCoder110k (Wei et al., ) | Code | You are a programmer. |
| CodeFeedback157k (Zheng et al., ) | Code | You are a programmer. |
| EvolInstructCode80k (Luo et al., ) | Code | You are a programmer. |
| MathInstruct262k (Yue et al., ) | Math | You are a mathematics expert. |
| MetaMath395k (Yu et al., ) | Math | You are a mathematics expert. |
| GPT4Tools75k (Yang et al., ) | Agent | You are an Agent with a lot of tools. |
| ToolbenchG3 (Qin et al., ) | Agent | You are an Agent with a lot of tools. |
| AgentInstruct (Zeng et al., ) | Agent | You are an Agent with a lot of tools. |

### 4.3 Evaluation Benchmark

Mathematic reasoning. GSM8K (Cobbe et al., ) presents diverse grade school math word problems, while the MATH dataset (Hendrycks et al., ) focuses on evaluating mathematical reasoning abilities with a collection of problems.

Code generation. HumanEval (Hendrycks et al., ) assesses code generation models’ problem-solving skills through Python programming challenges, and MBPP (Austin et al., ) targets models’ execution-based performance on Python problems.

Tool usage.T-EVAL (Chen et al., ) breaks down the assessment of tool usage into several sub-domains, shedding light on LLMs’ comprehensive and specific competencies.

STEM. MMLU (Hendrycks et al., ) covers 57 subjects from STEM to humanities and social sciences, testing models on a wide range of difficulty levels and world knowledge. Its broad and detailed subject coverage serves as an excellent tool for spotting models’ blind spots.

Multi-turn dialogue. MT-Bench (Zheng et al., ) features 80 questions across eight domains, including writing, roleplay, reasoning, and more, with each domain offering a two-round query-response evaluation using GPT-4.

**Table 2: Performance of SFT models on widely-used benchmarks. We report zero-shot performance of the models.For all benchmark computations, we consistently used the system prompt ”You are a helpful assistant.” to ensure the effectiveness of our methodology.When computing the average score, we multiplied the mt-bench scores by 10 to ensure numerical validity.**
| Model | MMLU | MBPP | HumanEval | GSM8K | MATH | T-Eval | MT-Bench | MT-Bench | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 0-shot | pass@1 | pass@1 | 0-shot | 0-shot | overall | 1st turn | 2nd turn |  |
| Qwen-7B-chat | 57.0 | 31.8 | 37.2 | 50.3 | 6.8 | 60.7 | 5.65 | 4.51 | 49.3 |
| Qwen-SFT-code | 41.5 | 53.0 | 56.1 | 34.0 | 0.0 | 22.3 | 5.70 | 3.13 | 47.9 |
| Qwen-SFT-math | 45.9 | 36.8 | 22.5 | 71.8 | 27.8 | 14.0 | 5.58 | 3.36 | 51.0 |
| Qwen-SFT-agent | 47.7 | 15.0 | 20.7 | 16.1 | 0.1 | 47.1 | 4.41 | 2.87 | 34.6 |
| Qwen-SFT-chat | 57.8 | 12.6 | 22.5 | 54.5 | 0.01 | 60.8 | 5.73 | 4.45 | 44.5 |
| Qwen-SFT-seq | 59.2 | 46.6 | 50.6 | 69.2 | 30.3 | 69.6 | 5.64 | 4.38 | 61.0 |
| Qwen-SFT-concat | 59.7 | 49.0 | 56.1 | 71.3 | 30.3 | 67.1 | 5.65 | 4.56 | 62.4 |
| Qwen-SFT-balanced | 59.4 | 51.6 | 55.5 | 71.9 | 30.0 | 66.9 | 5.66 | 4.31 | 62.2 |
| Qwen-SFT-MoI | 59.3 | 61.0 | 60.4 | 72.4 | 30.0 | 68.2 | 5.68 | 4.62 | 65.0 |

### 4.4 SFT experiments

In our experiments, the Qwen-7B-chat model was fine-tuned using domain-specific datasets over three epochs at a learning rate of 2e-5, except as noted. Table 3 lists the system prompts utilized for each dataset.

Qwen-SFT-code. Trained on MagicCoder110k, CodeFeedback157k, and EvolInstructCode80k.

Qwen-SFT-math. Utilized MathInstruct262k and MetaMath395k datasets.

Qwen-SFT-agent. Fine-tuned with GPT4Tools75k, AgentInstruct, and ToolbenchG3.

Qwen-SFT-chat. Focused on ultrachat200k for dialogue interactions.

Qwen-SFT-seq. Combined datasets, following sequence instruction strategy.

Qwen-SFT-concat. Merged datasets with concatenation strategy, learning rate of 4e-5.

Qwen-SFT-balanced. Balanced sampling from all datasets, learning rate of 4e-5.

Qwen-SFT-MoI. Employed the MoI scheme using ”You are a helpful assistant” as the default system prompt, learning rate of 4e-5,.

### 4.5 Ablation study and analysis

System prompt. As shown in Table 3, by modifying system prompts for SFT, we can comprehensively enhance the alignment performance of code-related models. Furthermore, employing the MoI method has also enabled us to acquire new knowledge under the default system prompts.

**Table 3: Performance of SFT models on widely-used code generation benchmarks. We report both zero-shot and few-shot performance of the models.**
| Model | Inference system prompt | MBPP | HumanEval | MT-Bench-coding |
| --- | --- | --- | --- | --- |
|  |  | pass@1 | pass@1 | 1st turn |
| Qwen | You are a helpful assistant. | 31.8 | 37.2 | 3.8 |
| Qwen-SFT-code | You are a helpful assistant. | 53.0 | 56.1 | 4.1 |
| Qwen-SFT-code | You are a programmer. | 55.2 | 60.4 | 5.0 |
| Qwen-SFT-MoI | You are a helpful assistant. | 61.0 | 60.4 | 4.7 |

Figure: Figure 5: (a) The model’s performance using two different methods across various iteration steps on the MBPP dataset. (b) Comparative performance analysis of the model, utilizing two distinct approaches, over multiple iteration stages on the GSM8K dataset. (c) An evaluation of the model’s performance, employing two methods, through successive iteration steps on the HumanEval dataset. (d) The performance trajectory of the model, under two methodologies, across a range of iteration steps on the MATH dataset.

Dataset bias. Figure 5 shows our ablation study compared Qwen-SFT-concat and Qwen-SFT-balanced, demonstrating that balanced sampling rapidly improves GSM8K accuracy and stabilizes code generation performance during training. In contrast, concat showed no substantial late-stage gains in GSM8K and a decline in code capability. With identical datasets and hyperparameters, these differences highlight balanced sampling superior handling of multi-task learning in language models.

Code,Math,Agent. Our comparative ablation study in table 2 revealed that while models trained on math, and code data respectively enhance performance in their specific tasks, this specialization comes at the cost of diminished multi-turn conversation ability, as evidenced by a significant drop in MT-Bench 2nd turn scores for all models. Thus, models trained on these datasets are recommended for domain-specific tasks rather than as conversational assistants.Additionally, models trained on the agent dataset did not show improvement in any metric, suggesting that the T-Eval dataset serves as a comprehensive test scheme to assess model capabilities.

**Table 4: Scores on HumanEval, MBPP, GSM8K, MATH. The pass@1 scores of our models are computed with greedy decoding. Models are evaluated in 0-shot on all datasets.**
| Model | Params | HumanEval | MBPP | GSM8K | MATH |
| --- | --- | --- | --- | --- | --- |
|  |  | pass@1 | pass@1 | Accuracy(%) | Accuracy(%) |
| GPT-3.5(OpenAI, ) | - | 73.2 | 81.7 | 80.8 | 37.8 |
| GPT-4(OpenAI, ) | - | 86.6 | 83.0 | 97.0 | 69.7 |
| StarCoder Base(Li et al., ) | 7B | 24.4 | 33.1 | - | - |
| Code Llama(Rozière et al., ) | 7B | 33.5 | 41.4 | - | - |
| Code Llama-Instruct(Rozière et al., ) | 7B | 34.7 | 44.4 | - | - |
| Code Llama-Python(Rozière et al., ) | 7B | 38.4 | 47.6 | - | - |
| WizardCoder(Luo et al., ) | 7B | 48.2 | 56.6 | - | - |
| Magicoder(Wei et al., ) | 7B | 60.4 | 64.2 | - | - |
| WizardMath(Luo et al., ) | 7B | - | - | 54.9 | 10.7 |
| MetaMath(Yu et al., ) | 7B | - | - | 66.4 | 19.4 |
| MAmmoTH(Yue et al., ) | 7B | - | - | 59.4 | 33.4 |
| Llama2(Touvron et al., ) | 7B | 12.2 | 20.8 | 16.7 | 3.3 |
| Mistral(Jiang et al., ) | 7B | 27.4 | 38.6 | 47.5 | 11.3 |
| Qwen(Bai et al., ) | 7B | 37.2 | 35.8 | 52.5 | 10.3 |
| Qwen-SFT-MoI(ours) | 7B | 60.4 | 61.0 | 72.4 | 30.0 |

Comparison with open-source models. Table 4 shows that despite the lack of extensive code pre-training, our model, through high-quality SFT and the MoI method, matches the performance of contemporary code models with similar parameters. Comparisons with open-source math reasoning models indicate our model’s math skills are on par with those fine-tuned solely on math datasets. Additionally, our model exhibits improved multi-turn dialogue and common sense reasoning capabilities.

## 5 Conclusion

In conclusion, our study highlights the significant role of system prompts in enhancing language model performance during SFT. By refining prompts, we mitigated issues in code generation with the Qwen-7B-chat model, leading to better handling of complex tasks. Our innovative multi-task learning strategy, which utilizes curated datasets with tailored prompts, successfully tackled the dataset bias seen in multi-task training. The Mixture of Instruction (MoI) method further ensured consistent performance across tasks. Our finalized Qwen-SFT-MoI model showcased substantial improvements over the original, underscoring MoI’s potential for improving language model adaptability and proficiency.

## Appendix A Appendix

### A.1 Mixture of instructions details

During the training of our language model, we employed the classic ChatML format:

In the example above, the portion labeled with ’system’ is what we have been discussing as the ’system prompt’ in our paper.

Figure 6 illustrates the effects of the Mixture of Instruction (MoI) framework. In the initial phase, models subjected to Supervised Fine-Tuning (SFT) with the default system prompt strive to replicate the canonical implementation of the Boyer-Moore algorithm. However, as the process advances, they divert their approach towards devising an algorithm based on iterative querying. Subsequent to retraining with a modified system prompt, the model acquires the capability to accurately script the Boyer-Moore algorithm, demonstrating a clear understanding of its logic and structure. Utilizing the Mixture of Instructions (MoI) approach for further training enables the seamless transference of knowledge acquired in the ’You are a programmer’ domain to applications within the ’You are a helpful assistant’ scenario, showcasing the versatility and adaptability of the learned skills across different contexts.

Figure: Figure 6: The function of the system prompt and the mapping of knowledge learned from all system prompts onto the default system prompt following the application of the Mixture of Instructions method.

### A.2 System prompt for mathematics task

We replicated the experiment of changing the system prompt from our code generation task in the context of mathematical reasoning tasks, altering the default system prompt to ’You are a mathematics expert.’ The experimental results in Table 5 indicate that this role-setting remains effective in anchoring mathematical knowledge.

**Table 5: Performance of SFT models on mathematics reasoning benchmark. We report zero-shot performance of the models.**
| Model | Inference system prompt | GSM8K | MATH | MT-Bench-math |
| --- | --- | --- | --- | --- |
|  |  | accuracy | accuracy |  |
| Qwen-7B-chat | You are a helpful assistant. | 50.3 | 6.8 | 2.65 |
| Qwen-SFT-math | You are a helpful assistant. | 71.8 | 27.8 | 4.9 |
| Qwen-SFT-math | You are a mathematics expert. | 72.1 | 30.0 | 5.0 |
| Qwen-SFT-MoI | You are a helpful assistant. | 72.4 | 30.0 | 5.0 |

### A.3 MT-bench results

Figure 7 shows that employing the MoI technology in language models concurrently boosts mathematical reasoning and coding skills, paralleling the proficiency achieved by models trained solely on individual tasks. This consistency underscores the robustness of our proposed approach.

Figure: Figure 7: Detailed scoring charts on MT-Bench for Qwen-7B-chat, Qwen-SFT-code, Qwen-SFT-math, and Qwen-SFT-MoI models.

### A.4 T-EVAL results

Table 6 indicates that the agent capabilities within language models exhibit a direct association with their chat functionalities. Subjecting the models to SFT using datasets composed exclusively of code, math, or agent-specific content can significantly impair their agent abilities. The Sequence of Instructions approach outperforms other methodologies in agent-related tasks, which we surmise is due to its compliance with the input format used during the initial SFT, thus maximally preserving the consistency of the language model. Furthermore, our MoI technique secures the highest scores in techniques involving instruction concatenation, suggesting that our method also aims to retain the original competencies of the language model to the greatest possible extent.

**Table 6: Detailed scores of SFT models on T-EVAL. We report zero-shot performance of the models.**
| Model | Instruct | Plan | Reason | Retrieve | Understand | Review | overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen-7B-chat | 67.6 | 56.3 | 46.9 | 66.1 | 66.5 | 60.6 | 60.7 |
| Qwen-SFT-code | 0.8 | 0.9 | 35.7 | 29.8 | 52.4 | 14.0 | 22.3 |
| Qwen-SFT-math | 0.3 | 9.2 | 21.5 | 11.2 | 30.3 | 11.7 | 14.0 |
| Qwen-SFT-agent | 44.1 | 22.8 | 37.8 | 58.7 | 63.7 | 55.2 | 47.1 |
| Qwen-SFT-chat | 64.0 | 61.1 | 51.3 | 63.4 | 74.4 | 50.3 | 60.8 |
| Qwen-SFT-seq | 83.7 | 65.8 | 54.4 | 77.1 | 78.4 | 58.3 | 69.6 |
| Qwen-SFT-concat | 83.6 | 62.9 | 51.2 | 72.7 | 75.6 | 56.3 | 67.1 |
| Qwen-SFT-balanced | 82.6 | 64.3 | 51.0 | 73.0 | 75.9 | 54.6 | 66.9 |
| Qwen-SFT-MoI | 78.1 | 65.3 | 53.9 | 76.8 | 77.8 | 57.5 | 68.2 |

### A.5 Case Study

**Table 7: Our MoI model demonstrated performance on the MT-Bench math, and we proceeded to inquire about the third and fourth rounds.**
| [Turn #1] Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies of a sci-fi novel, each priced at $20, 3 copies of a history book priced at $30 each, and 2 copies of a philosophy book for $45 each. What was the total cost of his purchases? |
| --- |
| Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total cost of the sci-fi novels is 5 * $20 = $100. He also bought 3 copies of a history book, each priced at $30, so the total cost of the history books is 3 * $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at $45, so the total cost of the philosophy books is 2 * $45 = $90. The total cost of his purchases is $100 + $90 + $90 = $280. |
| 280 |
| The answer is 280 |
| [Turn #2] Suppose Benjamin decides to sell each of these books at a 25% markup from the price he purchased them. What would be his total revenue if he sold all the books he bought? |
| Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total cost of the sci-fi novels is 5 * $20 = $100. He also bought 3 copies of a history book, each priced at $30, so the total cost of the history books is 3 * $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at $45, so the total cost of the philosophy books is 2 * $45 = $90. The total cost of his purchases is $100 + $90 + $90 = $280. If Benjamin decides to sell each of these books at a 25% markup from the price he purchased them, the markup would be 25% of $280, which is 0.25 * $280 = $70. So, the total revenue Benjamin would get if he sold all the books he bought is $280 + $70 = $350. |
| 350 |
| The answer is 350 |
| [Turn #3]After achieving his profit goal by selling books at a markup, Benjamin decides to donate a portion of his total revenue to a local library. If he chooses to donate 15% of his total revenue, how much money will the local library receive? |
| Benjamin’s total revenue is $350. If he chooses to donate 15% of his total revenue, the amount he will donate is 15/100 * $350 = $52.50. So, the local library will receive $52.50. |
| 52.5 |
| The answer is 52.5 |
| [Turn #4]If Benjamin aims to achieve a profit of $100 over the initial cost of his purchases by selling all the books, what percentage markup should he apply to each book’s selling price? |
| Benjamin aims to achieve a profit of $100 over the initial cost of his purchases, which is $280. To calculate the markup percentage, we need to find the difference between the selling price and the cost price, and then divide that by the cost price. The selling price is the cost price plus the markup, so the selling price is $280 + $100 = $380. The markup is the difference between the selling price and the cost price, which is $380 - $280 = $100. To find the markup percentage, we divide the markup by the cost price and multiply by 100. The markup percentage is ($100 / $280) * 100 = 35.71%. |
| 35.71 |
| The answer is 35.71 |

| [Turn #1]Write a simple website in HTML. When a user clicks the button, it shows a random joke from a list of 4 jokes. |
| --- |
| Sure, here is a simple HTML code that does what you asked for: |
| <!DOCTYPE html> |
| <html> |
| <head> |
| <title>Joke Generator</title> |
| <style> |
| button { |
| margin: 10px; |
| padding: 10px; |
| font-size: 18px; |
| } |
| </style> |
| </head> |
| <body> |
| <h1>Joke Generator>/h1> |
| <button onclick=”displayJoke()”>Generate Joke</button> |
| <p id=”joke”></p> |
| <script > |
| const jokes = [ |
| ”Why don’t scientists trust atoms? Because they make up everything!”, |
| ”Why did the scarecrow win an award? Because he was outstanding in his field!”, |
| ”Why don’t eggs tell jokes? Because they’d crack each other up!”, |
| ”Why don’t programmers like nature? Because they prefer to create their own world!” |
| ]; |
| function displayJoke() { |
| const randomJoke = Math.floor(Math.random() * jokes.length); |
| document.getElementById(”joke”).innerText = jokes[randomJoke]; |
| } |
| </script > |
| </body > |
| </html > |
| This code creates a simple webpage with a button that, when clicked, generates a random joke from the list of jokes and displays it in a paragraph element. |
| [Turn #2]How to use CSS to change the color of jokes to red? |
| You can use CSS to change the color of the jokes. Here is how you can do it: |
| <!DOCTYPE html> |
| <html> |
| <head> |
| <title>Random Joke Generator</title> |
| <style> |
| body { |
| font-family: Arial, sans-serif; |
| } |
| button { |
| margin-top: 20px; |
| } |
| #joke { |
| color: read; |
| } |
| </style> |
| </head> |
| <body> |
| <h1>Random Joke Generator</h1> |

**Table 8: Our MoI model demonstrated performance on the MT-Bench code, and we proceeded to inquire about the third round.**
| <button onclick=”displayJoke()”>Generate Joke</button> |
| --- |
| <p id=”joke”></p> |
| <script > |
| const jokes = [ |
| ”Why don’t scientists trust atoms? Because they make up everything!”, |
| ”Why did the scarecrow win an award? Because he was outstanding in his field!”, |
| ”Why don’t eggs tell jokes? Because they’d crack each other up!”, |
| ”Why don’t programmers like nature? Because they prefer to create their own world!” |
| ]; |
| function displayJoke() { |
| var randomJoke = jokes[Math.floor(Math.random() * jokes.length)]; |
| document.getElementById(”joke”).innerHTML = randomJoke; |
| } |
| </script > |
| </body > |
| </html > |
| In this code, the CSS rule p { color: red; } is used to change the color of all paragraph elements to red. |
| [TURN #3]How to dynamically retrieve a list of jokes from an external file or API |
| You can dynamically retrieve a list of jokes from an external file or API using JavaScript’s fetch function. Here is an example: |
| <!DOCTYPE html> |
| <html> |
| <head> |
| <title>Random Joke Generator</title> |
| <style> |
| body { |
| font-family: Arial, sans-serif; |
| } |
| button { |
| margin-top: 20px; |
| } |
| #joke { |
| color: read; |
| } |
| </style> |
| </head> |
| <body> |
| <h1>Random Joke Generator</h1> |
| <button onclick=”displayJoke()”>Generate Joke</button> |
| <p id=”joke”></p> |
| <script > |
| function displayJoke() { |
| fetch(’https://your-api-url.com/jokes’) |
| .then(response =>response.json()) |
| .then(jokes =>{ |
| var randomJoke = jokes[Math.floor(Math.random() * jokes.length)]; |
| document.getElementById(”joke”).innerHTML = randomJoke; |
| }) |
| .catch(error =>console.error(’Error:’, error)); |
| } |
| </script > |
| </body > |
| </html > |

| Questions: |
| --- |
| You have access to the following API:<br>[ |
| [ |
| ’AirbnbSearch.get_property_details’, { |
| ’name’: ’AirbnbSearch.get_property_details’, |
| ’description’: ’This function retrieves the details of a property given its ID.’, |
| ’required_parameters’: [ |
| {’name’: ’propertyId’,’type’: ’NUMBER’, ’description’: ’The ID of the property.’}], |
| ’optional_parameters’: [], |
| ’return_data’: [ |
| { ’name’: ’title’, ’description’: ’The title of the property.’}, |
| {’name’: ’propertyType’, ’description’: ’The type of the property.’}, |
| {’name’: ’personCapacity’, ’description’: ’The capacity of the property.’}, |
| {’name’: ’rating’, ’description’: ’The rating of the property.’}, |
| {’name’: ’latitude’, ’description’: ’The latitude of the property.’}, |
| {’name’: ’longitude’, ’description’: ’The longitude of the property.’}]} |
| ], |
| [ |
| ’AirbnbSearch.get_property_reviews’, { |
| ’name’: ’AirbnbSearch.get_property_reviews’, |
| ’description’: ’This function retrieves the reviews of a property given its ID.’, |
| ’required_parameters’: [ |
| {’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the property.’}], |
| ’optional_parameters’: [], |
| ’return_data’: [ |
| {’name’: ’reviews’, ’description’: ’The reviews of the property, containing comment, rating, and date. At most 3 reviews are returned.’}]} |
| ], |
| [ |
| ’AirbnbSearch.check_availability’,{ |
| ’name’: ’AirbnbSearch.check_availability’, |
| ’description’: ’This function checks the availability of a property given its ID.’, |
| ’required_parameters’: [ |
| {’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the property.’}], |
| ’optional_parameters’: [ |
| {’name’: ’date’, ’type’: ’STRING’, ’description’: ’The date to check, it must in the format of YYYY-MM-DD.’}], |
| ’return_data’: [ |
| {’name’: ’availability’, ’description’: ’The availability of the property for the given date or the next 10 days.’}]} |
| ], |
| [ |
| ’AirbnbSearch.search_property_by_place’, { |
| ’name’: ’AirbnbSearch.search_property_by_place’, |
| ’description’: ’This function takes various parameters to search properties on Airbnb.’, |
| ’required_parameters’: [ |
| {’name’: ’place’, ’type’: ’STRING’, ’description’: ’The name of the destination.’}], |
| ’optional_parameters’: [], |
| ’return_data’: [ |
| {’name’: ’property’, ’description’: ’a list of at most 3 properties, containing id, name, and address.’}]} |
| ] |
| ] |
| Please generate a plan for answer user’s questions, which should be a list of actions with the following format: |

**Table 9: Our MoI model demonstrated performance on the T-EVAL plan.**
| “‘ |
| --- |
| [ |
| { |
| // id of the action |
| ”id”: number; |
| // the name of the action |
| ”name”: string; |
| // input params required by this action |
| ”args”: ”str(Record<string, any>)”; |
| }, |
| … |
| ] |
| “‘ |
| You can imagine args when you plan the action, and these instructions will be executed sequentially.For example, if you want to call ‘api1‘ with ‘arg1‘ and ‘arg2‘, you can write the following plan: |
| “‘ |
| [ |
| { |
| ”id”: 0, |
| ”name”: ”api1”, |
| ”args”: ”{’arg1’: ’value1’, ’arg2’: ’value2’, …}”, |
| }, |
| … |
| ] |
| “‘ |
| The args should be a dictionary in string format. PLEASE use ” in args dictionary and use ”” in other places, DO NOT print args with value None or null.You should only generate a list in json format. The list should be the full planning list without ‘…‘. DO NOT generate any text to explain the json. |
| Here is the instruction: |
| I’m helping a client find the first available rental property in Los Angeles. Please find a property in Los Angeles. Then, check the availability of the property for the coming week. Next, get the details of that property. Finally, retrieve the reviews of the property. |
| Answer: |
| [ |
| { |
| ”id”: 0, |
| ”name”: ”AirbnbSearch.search_property_by_place”, |
| ”args”: ”{’place’: ’Los Angeles’}” |
| }, |
| { |
| ”id”: 1, |
| ”name”: ”AirbnbSearch.check_availability”, |
| ”args”: ”{’propertyId’:<property_id>, ’date’: ’2022-01-01’}” |
| }, |
| { |
| ”id”: 2, |
| ”name”: ”AirbnbSearch.get_property_details”, |
| ”args”: ”{’propertyId’: <property_id>}” |
| }, |
| { |
| ”id”: 3, |
| ”name”: ”AirbnbSearch.get_property_reviews”, |
| ”args”: ”{’propertyId’: <property_id>}” |
| } |
| ] |

### A.6 Solve math problem with code interpreter

In this section, we demonstrate that models trained using the Mixture of Instructions (MoI) approach can perform functions akin to the code interpreter capabilities of GPT-4. This case necessitates a comprehensive evaluation of the model’s mathematical reasoning, code writing, and tool utilisation proficiencies.

We feed prompts into the Qwen-SFT-MoI system in the following format, adhering to the ReACT standard during tool use—prompting the model to comply with a Thought, Action, Action Input sequence to generate the tool’s output:

This interaction informs the model to employ a code interpreter tool to address mathematical queries. A model proficient in following instructions should, upon encountering the user’s question, compose code that meets the formatting requirements. For this functionality, we have developed an entire pipeline. It filters and extracts the relevant substance from the model’s code output, then passes it into a Python code interpreter to obtain execution results, which are subsequently compared with the standard outcomes.

Our model’s efficacy is validated against the GSM8K benchmark, where our Qwen-SFT-MoI model achieves an accuracy of 69.14% under a zero-shot setting. Contrastingly, when the model infers separately on the problems, the accuracy reaches 72%. This case illustrates that our model not only possesses commendable mathematical reasoning, code writing, and tool utilisation capacities but can also synergise these abilities to resolve problems encountered in practical processes.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: LN038: How did you come to the conclusion that the model generates less than ideal solutions due to conflicts in prompt-induced knowledge? Is there any research or experimental evidence to back this up?

LN040: Can you explain why modifying prompts can help in resolving knowledge conflicts?

LN085: What is the approach you use to identify conflicts between new and old knowledge during SFT? I couldn't find this information in section 2.

LN099: Are there any previous studies that highlight the significance of system prompts? If so, you should reference them and compare your method to theirs to demonstrate its superiority. If you're the first to do this, it's also worth mentioning.

LN470: The main focus of this study is multi-task learning, but I don't see a thorough discussion or comparison with existing methods.
soundness: 3
strengths: 1) The authors effectively identify a gap in existing alignment methodologies that typically focus on single task and propose a multi-task solution that is particularly relevant for AI-driven products. 
2) The authors also provide a comprehensive evaluation of their method using seven benchmark datasets, demonstrating its effectiveness. The resulting model, Qwen-SFT-MoI, shows significant improvements in generative capabilities across multiple tasks. 
3) The paper is well-written and clear in its presentation of the methodology and results. The work is significant as it offers a practical solution to a key challenge in the field of AI, namely the efficient alignment of LLMs for multi-task performance.
summary: The paper introduces a technique known as Mixture-of-Instructions (MoI) for improving the alignment efficiency of large language models (LLMs) across multiple tasks. The technique uses instruction packing and diverse system prompts to enhance the model's performance. The authors applied this methodology to the Qwen-7B-chat model, resulting in the development of Qwen-SFT-MoI, which showed significant improvements in tasks like coding, mathematics, and tool use. The paper contributes by identifying and addressing conflicts between new and old knowledge during SFT, introducing the MoI method for joint multi-task training, and demonstrating the effectiveness of the MoI in enhancing SFT models.
weaknesses: The primary concern revolves around the originality. The suggested MoI method incorporates several techniques, such as a distinct system prompt for each task, balanced sampling, and chunk-based attention. However, these techniques have already been introduced in previous research. For instance, the significance of the prompt for SFT has been highlighted by "Keeping llms aligned after fine-tuning: The crucial role of prompt templates, Lyu et al., 2024" and "Reducing the cost: Cross-prompt pre-finetuning for short answer scoring, Funayama et al., 2023". The technique of balanced sampling has been employed by "Sampling bias and class imbalance in maximum-likelihood logistic regression, Oommen et al., 2011", and the concept of chunk-based attention has been put forth by "Shifted chunk encoder for transformer based streaming end-to-end ASR, Wang et al., 2022" and "Statistically defined visual chunks engage object-based attention, Lengyel et al., 2021".

Considering these techniques are not innovative, the method seems more of an engineering blend if it fails to offer fresh perspectives or intriguing discoveries. Nevertheless, the positive outcomes and comprehensive experiments render the work robust and beneficial for the industrial sector.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - Does the idea work for existing open SFT tuned models like Llama
- Does the idea apply to all model sizes and task categories?
soundness: 4
strengths: This is a clever and practical idea. The impact of this is clearly presented and I appreciate the comprehensive paper with many analyses.
summary: This paper addresses a practical issue on multi-task supervised fine tuning (SFT). In order the model to be trained propoerly on all the tasks they propose packing the instructions belonging to different tasks in a balanced fashion. This results in superior performance on a number of tasks.
weaknesses: However I feel like it may not constitute a full ICLR paper. Instead in the model system card, this could have been mentioned in a paragraph. For example Llama or Olmo model system cards have similar technical details like this.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See above.
soundness: 3
strengths: 1. The proposed strategy is straightforward and easy to follow yet achieves competitive performance against many open-source models. It can be effective guidance for SFT training and data utilization.

2. The experiments and ablation studies are comprehensive, which can clearly reflect the effectiveness of the method and explain how it works.
summary: The paper introduces a Mixture-of-Instructions (MOI) strategy, which combines instruction packing with diverse system prompts to enhance multiple downstream tasks. The multi-task training technique is applied to a Qwen-7B-Chat model, which demonstrates good performance in mathematical reasoning, code generation, tool usage, and chat benchmark.
weaknesses: 1. The novelty is limited. The idea of Mixture of Instructions has already been explored by the works of FLAN serious. Packing multiple samples into a single one and balanced sampling are also common strategies employed by many works, as cited in the paper. The main innovation of this work is the chunk-based packing and attention mask. However, the effectiveness and generalization of this strategy are still not clear when the model size and task numbers increase in the view of the scaling law.

2. In each chunk, the instructions are packed and reordered. One concern is whether the models would be sensitive to the order of instructions. Have the authors ever explored the influence of different orders of the packed instructions in the chunk? 

3. The extensibility of this strategy is not clear. The work has tried four tasks, namely mathematical reasoning, code generation, tool usage, and daily chat. When the task number increases, does this strategy still work? If the length of packed instructions exceeds the maximum sequence length, e.g., 4096 in llama, how can we solve this issue?

4. The description of the system prompt is somewhat confusing. In Section 2.4, the authors say that unique system prompts for each domain-specific task are set to ensure tailored guidance. However, in Table 1, they highlight that for all benchmark computations, they consistently use the system prompt "You are a helpful assistant." How can these conflict statements be understood? Does that mean the system prompts are different in training and testing?

5. In section 2.2, the authors mention that training on combined data can lead to model bias toward certain tasks, enhancing performance in some while degrading it in others. This is related to the studies of catastrophic forgetting [1,2], which should be discussed in the related work or compared as baselines.

[1] An empirical study of catastrophic forgetting in large language models during continual fine-tuning.

[2] Understanding catastrophic forgetting in language models via implicit inference.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. See the weakness section. Maybe elaborate more on your opinions about point 1.
soundness: 3
strengths: **1. Important Research Questions**

The paper addresses the critical challenge of multi-task alignment in LLMs, which is increasingly relevant given the diverse applications of AI-driven products. By focusing on harmonizing various capabilities—such as coding, mathematical reasoning, and multi-turn dialogues—the study tackles the complex interplay between different task domains, which is essential for creating versatile and reliable language models.

**2. Rich Experiments and Evaluations**

The authors conduct extensive experiments across seven diverse benchmark datasets, including MT-Bench, HumanEval, MBPP, MATH, GSM8K, MMLU, and T-EVAL. This comprehensive evaluation framework ensures that the proposed MoI methodology is rigorously tested across multiple domains, demonstrating its efficacy in improving generative performance. Additionally, ablation studies and comparisons with existing models underscore the robustness of the approach.

**3. Carefully Crafted Implementation Details**

The introduction of chunk-based attention masking is a meticulous enhancement that addresses the issue of attention cross-contamination during multi-task training. Coupled with balanced sampling techniques, these implementation strategies effectively mitigate dataset biases and ensure that the model maintains high performance across all tasks. The detailed explanation of the loss function modifications further showcases the depth of the methodological advancements.
summary: This paper introduces Mixture-of-Instructions (MoI), a novel methodology aimed at enhancing the alignment of large language models (LLMs) across multiple tasks through a combination of instruction balanced packing and diverse system prompts. Traditional alignment methods focus on single tasks, which limits the effectiveness of LLMs in real-world, multi-faceted applications. To address this, the authors propose MoI, which assigns unique system prompts to different tasks and integrates them into a unified instruction set, facilitating comprehensive multi-task training. The approach is evaluated using seven benchmark datasets encompassing mathematics, programming, tool usage, common sense, and dialogue, demonstrating significant improvements in generative performance. The methodology is applied to the open-source Qwen-7B-chat model, resulting in the enhanced Qwen-SFT-MoI model, which outperforms existing models across various benchmarks.
weaknesses: **1. Lack of Explanation on MoI's Absence of System Leading to Performance Degradation**

The paper briefly touches upon the performance decline observed when not using the chunk-based attention masking but fails to delve deeply into the underlying reasons. A more thorough analysis or theoretical justification for why the absence of a system prompt within MoI leads to performance degradation would provide greater clarity and strengthen the argument for the proposed solution.

**2. Insufficient Exploration of Long Input and Output Scenarios**

While the study effectively evaluates MoI on various tasks, it overlooks scenarios that involve prolonged inputs and outputs, such as Retrieval-Augmented Generation (RAG) and long-form text creation. Including these cases would offer a more comprehensive understanding of how MoI performs under conditions that demand handling of extensive contextual information and sustained generation over longer sequences.

**3. Lack of Evaluation on Larger-Scale Models**

The experiments are primarily conducted on the Qwen-7B-chat model and smaller variants. Testing the MoI methodology on larger models with more parameters (like Qwen-72B or Qwen-MoE) would provide insights into its scalability and effectiveness in more powerful architectures. This omission leaves a gap in understanding how MoI can be generalized to state-of-the-art LLMs with significantly higher capacities.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How do you control balance when the numbers of samples of different tasks are different? And according to Table 9, the numbers of samples are indeed unbalanced. How do you order them? Conversely, if the balanced packing required the same number of samples across all tasks, that would bring out a great limitation.

2. The system prompt part is confusing for me. Why do you always adopt the unified "Assistant" system prompt for all tasks in MoI model rather than different system prompts? What system prompt do you use for other baseline models in training and inference?  I strongly recommand a table to present the system prompts in training and inference stages for each model.

3. What is the difference between "Isolated attention mask" in Table 8 and "Qwen-SFT-balanced" in Table 1? Why are the results not the same?
soundness: 2
strengths: 1. The research scope of this work, how to perform multi-task learning in SFT is indeed key to LLMs.

2. The proposed method is simple and easy to implement. Researchers would love to replicates this work if they believe the reported results of this work.

3. This paper employs some benchmarks and visualization analysis to demonstrate its effectiveness, which enhances the readers' comprehensive understanding of this work.
summary: This paper proposes a method called Mixture-of-Instructions (MoI), which employs a strategy of instruction packing combined with diverse system prompts to boost the alignment efficiency of language models. The methods mainly include balanced instruction packing and chunk-based attention. The experimental results show that the methods can enhance the ability of cross-domain tasks by multi-task learning.
In addition, this authors also conduct some experiments on the impact of system prompt for different tasks and find that system prompt matters.
weaknesses: 1. The chunk-based attention, packing the data and unmasking attention across different tasks, is quite unreasonable from intuition. Isolated attention is for removing unrelated contexts and full attention is for efficiency. So what is the motivation of the chunk-based attention? Moreover, some related works (Zhao et al, 2024) that this paper has even cited have pointed out that similar samples should be packed together to enhance the final performance. However, this work, chooses the opposite way, unmask the attention across different tasks in a chunk. What is the motivation and why is it better? This concern could become more important to me when I see the coding ability gets greatly enhanced by MoI compared to balanced packing, which is kindof magic to me.

* PS: I don't think the paragraph "Why is MoI effective?" can convince me. The paragraph only points out that the attention of MoI in SFT is significantly different from one in Chat, which is quite natural and normal (in my opinion) if we unmask the attention between different instructions. I expect more essential reasons, that is, why the altered attention distribution is more useful rather than more useless?

2. The contribution (or motivation) listed in line 085-087 seems contradictive to the actual implementation in MoI, which uses a unified "Assistant" system prompt. The authors emphasize that the system prompt is important (and conduct some experiments to prove this) but finally adopt a simple "Assistant" system prompt in their proposed model. What is the connection between them?

3. In fact, many SFT training samples have their own specific system prompts rather than a simple sentence "You are ....". Moreover, the huge impact (sensitivity) of system prompt seems well-known by LLM/NLP community. From my perspective, the exploration about system prompt in this work is kind of shallow and the possible insight is limited.

Meta-review: This paper proposes an approach, Mixture-of-Instructions (MoI), that aims to align a model with several tasks simultaneously. The approach is based on instruction packing with diverse system prompts to represent and improve multiple downstream tasks during supervised fine-tuning. Authors test the approach on a new benchmark they compiled from existing benchmarks, and create the MoI version of the Qwen-7B-chat model. Reviewers highlight the contributions as the ease of applying the approach and comprehensive experimental results. The weaknesses include the limited novelty based on the incremental nature of the approach and limited experimentation, for example, the analysis on the single model.

Decision: Reject