Exploring Format Consistency for Instruction Tuning
Authors: Shihao Liang
Runchu Tian
Kunlun Zhu
Yujia Qin
Huadong Wang
Xin Cong
Zhiyuan Liu
Xiaojiang Liu
Maosong Sun
Venue: Accepted by TMLR ()

Abstract: Instruction tuning has emerged as a promising approach to enhancing large language models in following human instructions. It is shown that increasing the diversity and number of instructions in the training data can consistently enhance generalization performance, which facilitates a recent endeavor to collect various instructions and integrate existing instruction tuning datasets into larger collections. However, different users have their unique ways of expressing instructions, and there often exist variations across different datasets in the instruction styles and formats, i.e., format inconsistency. In this work, a framework named Unified Instruction Tuning (UIT) is proposed, which calls OpenAI APIs for automatic format transfer among different instruction tuning datasets such as PromptSource, FLAN and CrossFit. With the framework, we (1) demonstrate the necessity of maintaining format consistency in instruction tuning; (2) improve the generalization performance on unseen instructions on T5-LM-xl; (3) provide a novel perplexity-based denoising method to reduce the noise of automatic format transfer to make the UIT framework more practical and a smaller offline model based on GPT-J that achieves comparable format transfer capability to OpenAI APIs to reduce costs in practice. Further analysis regarding variations of targeted formats and other effects is intended. The code and trained models will soon be available.

Published in Transactions on Machine Learning Research (12/2023)
Exploring Format Consistency for Instruction Tuning
Shihao Liang*
shikaaokkang(08220mmaaa..
Deportment of Computer Science Tsinghua University
Runchu Tian"
nr.200maiis.tsiigghaae cr
Department of Computer Science Tsinghua University
Kunlun Zhu'
zhukkun Thullunnmaall.sssuu...uuu
Deportment of Computer Science Tsinghua University
Yujia Qin
mpj200@mmiiistiinnnuuuu edu.ccm
Department of Computer Science Tsinghua University
Huadong Wang
huadw2012@116 16112/111..... com
ModelBest Inc.
Xin Cong
congrin 195 edu.ccm
Deportment of Computer Science Tsinghua University
Zhiyuan Liut
liuzyttsinghua.co edu.ccn
Deportment of Computer Science Tsinghua University
Xiaojiang Liu Apple
miaojiang lia@apple.com
Maosong Sunt Deportment of Computer Science Tsinghua University
sms@ttsinghua edu.
Reviewed OpenReview: https: //
https://openrevieuu openreview. net/ forum? id= f66
Abstract
Instruction tuning has emerged as promising approach to enhancing large language models in following human instructions. It is shown that increasing the diversity and number of instructions in the training data can consistently enhance generalization performance, which facilitates a recent endeavor to collect various instructions and integrate existing instruction luning datasets into larger collections. However, different users have their unique ways of expressing instructions, and there often exist variations across different datasets in the instruction styles and formats, i.c., format inconsistency. In this work, propose framework named "Unified Instruction Tuning" (UIT), which calls OpenAl APIs for automatic format transfer among different instruction tuning datasets such as PromptSource, FLAN and FrossPit. With the framework, we (1) demonstrate the necessity of maintaining
Indicates equal contribution. Corresponding author.

---

format consistency in instruction tuning; (2) improve the generalization performance unseen instructions on T5-LM-xl; (3) provide a novel perplexity-based denoising method to reduce the noise of automatic format transfer to make the UIT framework more practical and a smaller offline model based on GPT-J that achieves comparable format transfer capability to OpenAT APIs to reduce costs in practice. Further analysis regarding variations of targeted formats and other effects is intended. The code and trained models are publicly available at Published in Transactions on Machine Learning Research (12/2023)
1 Introduction
Recently, instruction tuning has gained considcrable attention as a potent strategy for enhancing large language models (LLMs) in following human instructions and generating appropriate responses. For instance, by reformulating various NLP tasks with an instruction template, models trained on the converted dataset. exhibit powerful capabilities of zero-shot generalization on unseen tasks (Wei et al. 2021). Later studies have demonstrated that instruction tuning is critical to facilitating LLMs in grounding their inner knowledge diverse real-world scenarios (Ouyang et al.
2022 Ding et al., 2023). Up to now, considerable efforts have been dedicated to creating datasets for instruction tuning (Honovich et al.
2022 Tyer et. al. 2022 Chung et. al.
2022a Bach et al. 2022 Wei el al. 2021 Wang et al.
2022 and researchers find that increasing the task diversity (i.e., the number of unique tasks) of the training data can consistently enhance generalization performance (Wang et al. 2022b Lyer ct al.
2022b
Aribandi et al.
2022 Longpre ct al. 2023). Therefore, the
community has witnessed a growing endeavor to collect various instructions and integrate existing instruction
tuning datasets into larger collections (Lyer et al. 2022 Longpre et al, 2023 Chung et al. 2022 Zhou et al.
2023
While previous works strive to increase task diversity and merge existing instruction tuning datasets, they typically ignore the format consistency among these datasets. More specifically, different users have their unique ways of expressing instructions, even if these instructions correspond to the same intent. Hence, there often exist variations across different datasets in the instruction styles and formats, which is dubbed as the format inconsistency issue. Take the case of a summarization task, the instruction can be as detailed as "In this task, you are given a conversation, and your task is to generate a summary... Input: ... Output: ..." in Ni-v2 (Wang al.) 2022b) or simply composed of few keywords, e.g., "Summarize: ... in CrossFit (Ye) et al. 2021b). Due to the format inconsistency issue, fine-tuned LLMs may have difficulty in handling unseen instructions in a different format at the test time, exhibiting poor out-of-distributiio (OOD) generalization. Hence, before directly merging diverse datasets from various sources and performing multi-task training (i.e., the common practice), it is essential to conduct a comprehensive study of how format inconsistency may impact the performance of instruction tuning and whether mitigating such inconsistency could enhance the gencralization.
However, unifying the format across different datasets is not easy. First,, instructions are inherently diverse and nuanced, and the vast range of possible expressions makes it challenging to devise fixed rule for format transfer. Sccond, standardizing formats can sometimes inadvertently change the mcaning of the original instructions. This is particularly problematic for complex lasks where the instruction's wording and style are crucial to correctly guiding the model behavior. In this paper, we introduce a format transfer framework, Unified Instruction Tuning (UIT) (Figure [ to explore the effects of format consistency. Specifically, we use OpenAI GPT3.5 for automatic instruction format transfer. Leveraging its powerful in-context learning capabilities, GPT3.5 can successfully transfer the instruction from source format to a target format. based on only few handcrafted examples. Then we analyze how format inconsistency could affect generalization under two settings: (1) Lesting-time setting, which simulates the format inconsistency between the training data and the testing data, and (2) trainingttme setting, which simulates the format inconsistency among different sources of instructions in the training data. We perform analysis across five benchmarks and show that our method successfully mitigates the format inconsistency issue and improves the generalization performance on unseen instructions in both settings
https://platform.. openai. com/ con//bocs/nodel//ggttt
2

---

Published in Transactions on Machine Learning Research (12/2023)
Training-Time Time Transfer
Testing-Time Transfer
Training data
Test data
Performance
Source Formats (Fs)
SFT
Source Format 73
Performance
LLM
Score,
[Dialogue]. Summary:
Score,
Sy
Better than
Format tramsfer Jg - Jt
Format transfer
Test data with format Fc
Better than
Fay
F2
SFT
Target Format Fr
SFT
Target Format Fr
Training data with format Fr
LLM
Score
Please summarize...
Input: (Dinlogue).
Figure 1: The proposed format transfer framework applied to two settings: testing-timg time transfer and training-timi time transfer. $1."1' && denote the training data in the original instruction format, " ,N denote all the transferred training data in larget format.
Despite its simplicity and performance, the above framework encounters two practical challenges. To begin with, the converted instructions are not as perfect as written ones and sometimes involve noise. For instance, an auto-converted instruction may express a slightly different meaning than the original one. To address this issue, we propose a novel perplexity-based denoising strategy that samples multiple possible conversions of a source instruction and then filters those low-quality ones based perplexity. Experimental results reveal that this strategy effectively reduces the noise of format transfer and improves robustness and performance. Second, converting large-scale instructions via OpenAl API can result in substantial costs for API calls, which is infeasible practice. To this end, we propose to learn an offline model for format transfer by distilling from GPT3.5. We demonstrate that with a few examples generated by GPT3.5, a much smaller model can be trained to achieve almost equivalent performance in format transfer, which saves the costs for API calls in practice. In general, our findings shed light on an essential but previously overlooked aspect, i.e., format consistency, for instruction tuning. We envision rescarch could inspire more efforts in advancing the instruction luning methodologies for LLMs.
Table 1: A comparison of representative instruction tuning datasets of different instruction formats. "Num.", "Cate.", "Exp.", "Inst.", " "Unnat-Inst", refer to Number, Category, Example, Instruction, and unnatural- instructions respectively.
Resource
Task Num.
Cate. Num.
Total Exp.
Inst. format
Ni-v2 (Wang et al.
2022b
1616
5M
task-level instance-lever
Flan 2021 (eii
202
62
4.4M
CrossFit Ye al. 2021a)
159
13
7.1M
keywords-level
P3 Bach et al.
2022
62
13
12M
instance-leved
Unnat- Ins
Honovich et al.
2022a
117
64k
task-level
OPT-IML
Lyer et al. 2022
1545
93
17.9M
mixed
Flan 2022
Longpre et al. 2023
1836
162
15M
mixod
2 Related Work
Instruction Tuning Instruction tuning regulates LLMs to accurately comprehend and interpret natural language instructions. Prior works in this field focus on reformulating NLP tasks using the templates of instructions. Wei et al. (2021) pioneered to show that fine-tuning LLMs on large collections of tasks formatted in instructions enables the model to generalize to unseen tasks in a zero-shot manner. Since then, there has been a surge of interest in manually constructing high-quality instruction datascts by first reformulating the formats of existing NLP datasets and then merging them (Mishra
2022 Bach et al. 2022; Ye al.,
2021b Ouyang et al. 2022 Another line of study [Longpre et al. 2023 [err et al. 2022 demonstrates
3

---

Published in Transactions on Machine Learning Research (12/2023)
that scaling the number of training tasks and task diversity can further enhance the model's generalization performance. However, all these works directly mix all the existing instruction datasets while ignoring the potential issue of format inconsistency. Instead of investigating the number and diversity of training
instructions, we instead explore an under-explored facet, i.e., the instruction format of instruction tuning, and investigate its impact on generalization.
Data Augmentation Besides manually curating instruction tuning datasets, Honovich et al. (2022a)
show that fine-tuning LLMs with machine-generatee instruction tuning data achieves excellent performance compared with human-written data, indicating that data augmentation is effective method to enhance the data quantity and task diversity, which overcomes the time-consuming issues of human annotation. Recently, Taori et al. (2023); Peng et al. (2023 Ding et al. (2023) adopt
method (Wang et al. 2022al to generate real-world human instructions (rather than instructions that describe NLP tasks) and model responses based on powerful LLMs such as ChatGPT, Similarly, in this paper, we also leverage LLMs for automatic formal transfer and data augmentation. Since real-world instructions are quite diverse and hard to annotate their formats, we instead focus on instructions that describe NLP tasks to rigorously study the effects of instruction format. We believe the derived findings potentially be applied to real-world instructions in the future.
Synthetic Data Denoising Generative models are commonly utilized for data augmentation TTori et al. 2023). However, these synthelic datasets are not always as reliable as those human-annolated ones, and
filterigg out noisy examples can boost the model performance (Le Bras et al. 2020). Recent studies have suggested different approaches for denoising. For instance, Yang et al. (2020); ; Fang et al. (022) adopted
influence functions (Koh & Liang, 2017) to evaluate the quality of the synthetic data; Wang et al. (202cc)
employ the NLU Consistency Filtering (Anaby-Tavor et al. 2020) to filter out low-quality samples. In
research, utilized LLMs for instruction format transfer, which may introduce noise throughout the process. To overcome this challenge, adopted simple and effective perplexity scoring strategy to denoise our auto-constructed dataset (section
3 Instruction Format Inconsistency
As outlined in Tyer et al. (002)). existing instruction formats exhibit variations across different datasets, which can be classified into three distinct hierarchical levels: Task-level format, Instance-level format, and Keywords-level format (as illustrated in Figure We present an overview of existing instruction tuning datasets based on instruction formats in Table
Task-level Format encompasses comprehensive definition of task and may include supplementary information such as positive or negative examples and explanations of the examples. Representative datasets are Ni-v2 (Wang et. al. 2022b). Unnatural Instructions (Honovich et al. 2022a). and
Alpaca (Taori et al. 2023).
Instance-level Format employs succinct templates that are customized for each individual example and is occasionally structured in cloze-style format to elicit the intended output. Representative datasets are Flan (Wei el al. 2021) and PromptSource (Bach et al. 2022
Keywords-level Format closely resembles the instance-level format, but it limits the instruction templates exclusively keywords. CrossFit (Ye et al. 2021b) serves as representative example of keywords-level dataset.
Compared with task diversity, the effect of format consistency is poorly understood in instruction tuning. We contend that successful instruction understanding and generalization are influenced by both task diversity and format consistency. Task diversity can be enhanced by incorporating as many tasks into the training data (c.g., merging existing instruction tuning datasots) as possible. However, crucial to note that when merging different datasets for training, the training data originating from different sources often present variations in the instruction formats. When confronted with instructions of unseen inconsistent. formats
4

---

Published in Transactions on Machine Learning Research (12/2023)
Source Datasets with Diverse Instruction Formats ((Fs,))
Keywords-level Format:
Instance-level Format: Flan, PromptSouree
Task-level Format:
CrossFit
DiversePrompt
Dialogue:
Answer the question:
Write paraphrase of the input sentence, but formal style:
Ali: Nice to meet you
[Question]: What animals does
Kune: Mc, too.
mahout work with?
Input: What's up?
Summary: Mecting
(Answer]: Elephants
Output: How are you doing?
Seed parallel data
Example 1: [5p, fl
Example [8t, 4g
Unified Instruction Format Transfer (Fs, F.)
Example A: [5, 4.
Definition: Please summary dialogue.
Definition:
Here question.
Definition: this task, you need rewrite the informal sentence formal style.
Please provide an appropriate
Positive Examples: Input:
the question.
Output: Now please complete this example:
Positive Examples: Input: Output: ... Now please complete this example:
Positive Examples: Input: Output: Now please complete this example:
Input: Ali: Nice to meet you.
Kane: Mc, too.
Input: What animals
Input: What's ' up?
Output: Meeting
mahout with?
Output: How are you doing?
Output: Elephants
Target Dataset with Unified Instruction Format (Fe)
Instruction Tuning for LLMs
Figure 2: Transferring instruction formats with UIT. The existing instruction formats exhibit variations across different datasets, which can be classified into three distinct hierarchical formats: Task level, Instance level, and Keywords level. UTT leverages seed parallel data to conduct format transfer across different formats automatically.
at the test time, the trained model may fail to generalize well and comprehend the intent behind different instructions, showing poor OOD generalization.
4 Framework and Experiments
To mitigate format inconsistency, we propose a format transfer framework, Unified Instruction Tuning (UIT), to convert the instruction formats of existing datasets into unified format.
4.1 Unified Instruction Format Transfer
Denote the target unified instruction format as F. and the original instruction format of source dataset. F&, aim to convert F. into F& to alleviate the OOD generalization in the instruction format. Taking inspiration from Honovich et al. (2022a), we rely on the LLM's in-context lcarning ability to conduct format transfer in an automatic manner. Specifically, we manually select k seed parallel data t1,, [Sk. tx]), where 8; and & are the same instance (task) expressed in format F, and F, respectively.
Given a new instance 8new with format F., we transfer its instruction format into the unified instruction format F, via in-context learning as follows:
Inea LLM (Smear, (31,4), ... [sk, 4x)),
(1)
where knew refers to the transferred instance with F,, We choose text-davinci-003 (GPT3.5) as the LLM for format transfer. Details of the prompt for format transfer shown in Figure
4.2 Experiments
Settings To simulate the format inconsistency problem, design two experimental settings:
5

---

Published in Transactions on Machine Learning Research (12/2023)
Example
Source Instruction: Translate Spanish
Target Instruction: In this task, you given sentence in the English language. Your job is to translate the English sentence into the Spanish language.
Example
Source Instruction: output the opposite of
Target Instruction: Given an adjective, generate its antonym. An antonym a word is word opposite in meaning to
Example
Source Instruction: Output whether the sentiment of the input sentence is positive or negative.
Target Instruction: Given review, you need predict whether the review is good or bad. A negative review is review, and posifiveineutral reviews good reviews.
Example
Source Instruction: Rate from (definitely not) (perfecily) degree in which both sentences describe same event.
Target Instruction: Given two sentences related to a specific event or scenario, assign a score from 0 (definitely not) to 5 (perfectly) to indicate the degree in which both sentences describe the same event.
Figure 3: An example of format transfer using GPT3.5, where we prompt the model with parallel examples to generate the target instruction for the 4-th example.
the training data is formatted in Fe, while the test data is formatted in F.. To mitigate the format inconsistency, we convert the instruction format of the test data into Fe, without modifying the training data. This setting is designed to explore the format inconsistency impact between training data and the test data in the inference phase.
Testing-time Format. Transfer:
Training-time Format Transfer: the training data is mixed with different formats (e-g., both F, and Fi), and the testing data is in the format of F, Instead of modifying the testing data, here we convert the training data from format F, to Fz. This setting is designed to simulate the format inconsistency of different sources of the training data.
For both scttings, we choose T5-LM-xl as our model and use Exact Match (EM) and Rouge-L as evaluation metrics.
Datasets For the resting-time setting, we select Ni-v2 /Wang al. 2022b) the training dataset and use DiversePrompt (Honovich et al. 2022b), Flan (Wei al. 2021 CrossFit (Ye et al.) 2021a), and PromptSource (Bach et al., 2022) the test dataset. We evaluate the tasks that do not appear in the training stage. These tasks are the same or similar to those in Ni-v2 test set. In Ni-v2, the instruction format incorporates four components: (1) task definition (D), (2) positive example (P) for demonst ration instances with the ground-truth label, (3) negative examples (N) for demonstration instances with a false label, and (4) explanations (E) that provide detailed explanations for the examples. Different formats refer to distinct combinations of the above components. For example, the DP format includes the task definition and positive examples information. In our experiments, we consider four primary formats, namely DP, DPN, DPE, and DPNE as the unified instruction format, respectively.
For the training time setting, we use the training set of Ni-v2 together with Flan, CrossFit, and P3 respectively for training and use the test set of Ni-v2 for evaluation. As Flan, CrossFit, and P3 may contain instances that exist the test set of Ni-v2, to prevent data leakage we filter the overlapped data in Flan, CrossFit, and P3 and use the remaining data for training. In this setting, we choose DP as the unified instruction format.
Baselines We construct two baselines: (1) Raw does not involve any modifications on the instruction format for both training and testing. For instance, directly test an instance from Flan in its original format using model trained with Ni-v2 in DPN format. (2) Heuristic applies mamally-designed rules to transfer different instruction formats into the unified one. If the information from the original format matches the corresponding field in the unified format, we fill the unified format with that information. Otherwise, we
hhttps://mugginggaace http:///wwwgggggtcccco/////o//e/ co/google//eeeeeeoooooocooo 1m-adapt

---

Published in Transactions on Machine Learning Research (12/2023)
leave the respective field in the unified format blank. For instance, instance from Flan can be transferred to the DPE format. by leaving the Definition and Erplanations fields blank and filling the Posilive Examples field with randomly selected instances from the Flan training set.
Table 2:
Testing-time format transfer experiment with four target unified instruction formats (DP, DPE, DPN, DPNE), respectively. We evaluate three methods: (1) raw instructions, transferred instructions based on (2) heuristic rules and (3) our proposed UIT. The training is conducted Ni-v2 while the testing is conducted on DiversePrompt, FLAN, CrossFit, and PromptSource, respectively.
Diverse Prompt
FLAN
Cross
PromptSource Rouge-I.
Average
Format
Method
EM
Rouge-L,
EM
Ronge-1.
RM
Rouge-I,
EM
EM
Rouge-I.
raw
0.1
11.6
20.8
3.9
6.6
13.6
4.6
10.8
DP
heuristic
34.7
45.1
31.4
44.8
43.7
56.0
27.3
32.7
34.3
44.6
unified
34.2
45.4
32.6
46.3
49.1
60.1
29.2
34.7
36.3
46.6
TaW
0.1
5.0
18.1
27.8
0.3
4.4
14.4
19.2
8.2
14.1
DPE heuristic
32.5
43.4
32.0
45.3
41.3
54.2
26.6
31.1
33.1
43.8
unified
32.9
44.8
33.5
46.9
58.4
27.8
32.8
35.5
46.0
Taw 0.2
5.5
12.0
22.6
0.2
4.5
5.3
11.6
4.4
11.1
DPN
houristic
30.6
31.9
45.3
55.5
29.0
33.9
33.9
unified
31.5
44.3
34.8
48.3
50.3
60.4
32.4
38.3
37.5
48.2
raw heuristic
0.1
5.2
15.2
25.3
3.8
19.2
23.7
8.7
14.5
DPNE
30.6
43.4
30.7
43.6
42.8
54.6
29.1
33.7
33.4
44.1
unified
32.2
43.4
35.0
48.0
48.6
59.3
29.8
34.9
36.6
46.7
Results and Analyses Testing-ting timenng-tmme format transfer results are shown in Table 2 and find that: (1) transferring the instruction format either through the heuristic rule UIT significantly improves the performance than the vanilla baseline (i.e., raw), demonstrating the necessity of maintaining format consistency in instruction luning; (2) our UIT consistently oulperforms the heuristic method across all benchmarks and almost all formats in Ni-v2. Compared with the heuristic method, UIT fully utilizes the semantic understanding and generation abilities of GPT3.5 to derive better transferred instructions; (3) the DPN formal demonstrates the highest average performance and exhibits the largest improvements with UIT.
Training-time format transfer results are shown in Table 3 which shows that format transfer also brings performance improvements compared to raw baseline and performs slightly better than the heuristic method. This again demonstrates that UIT can improve the generalization performance by unifying the instruction format. However, the improvements in the training-time setting are not. as significant as those in the testing-time setting. We conjecture this may be because the format inconsistency issue is more evident in our testing-ting time setting than in the training-tine time setting. Overall, the results under both settings validate our hypothesis that mitigating the instruction format conduces improved generalization.
Limitations in Practice Despite the favorable performance, the proposed framework still has some limitations: first, automatic format transfer sometimes involves noise or even errors in the generated data, which may produce adverse effects; second, the proposed method heavily relics on OpenAI API calls, which entail substantial costs especially for large-scale instruction datasets. Both issues would limit UTT's real-world deployment. In the following, we discuss potential solutions for the above limitations by proposing denoising strategy (section 5 and training offline transfer model (section 6 respectively.
5 Denoising for Format Transfer
Empirically, transferring format via LLMs will introduce noise unavoidably. The transferred instances may contain errors like critical changes task definition or hallucinatory restrictions. Intuitively, utilizing erroneous instructions would impair the model's generalization performance. To this end, we propose a perplexity-based denoising strategy to filter low-quality instances.
7

---

Published in Transactions on Machine Learning Research (12/2023)
Table Training- time format transfer experiment with DP format. We compare our UIT with two baselines: instructions and instructions transferred by the hcuristic rulc. The training dataset is Ni-v2 combined with CrossFit, Flan, or P3.
CrossFit
+FLAN
+P3
Average
Method
EM
Rouge-L
EM
Rouge-L
EM
Rouge-L
EM
Rouge-L
raw
37.5
56.0
38.4
56.9
38.8
56.7
38.2
56.5
houristic
37.7
55.9
38.9
57.4
39.9
58.1
38.8
57.1
unified
37.9
56.0
38.9
57.5
39.4
57.3
38.7
56.9
Denoising test on T5-LM
Testing time
Testing time Rouge-L.
Training time
Training time Rouge-L
34.00
40.00
33.15
30.75
30.5
51.5
33.50
36.50
11.25
26.25
27.0
Rouge-1
28.75
56.0
32.50
38.50
20.55
38.25
37.5
05.9
10.00
28.50
37.0
55.0
Sample Number
Samole Number
Sample Number
Sample Number
Figure 4: The performance of the denoising stralegy at the lesting and training time with different number of samples. Detailed results in the form of table are presented in Section B of the appendices.
Perplexity-based Denoising Strategy Perplexity (PPL) is a widely used metric for evaluating the semantic coherence and certainty of language models. We assume that noisy instructions can reduce the certainly of LLMs accurately predicting the correct output token sequence leading higher perplexity. As a result, perplexity serves as a useful metric for assessing the quality of transferred instructions. Hence, we propose to sample multiple times from LLM to obtaín multiple transferred instructions. Then calculato the perplexity for each instruction. Specifically, concatenate the transferred instruction and the input query, then predict the annotated label and calculate its perplexity, and filter those with high perplexity.
We employ GPT3.5 with lemperature 1.0 to perform sampling for N times with different random seeds, where N is chosen from (1, 4, 8, 16, 32}. Then, we sort the generated instructions based on perplexity using GPT-J (Wang & Komatsuzaki 2021) and select the sample with the lowest perplexity. We compare our method with the baseline that only samples once from GPT3.5. We conduct experiments on Ni-v2 and PromptSource under both the testing- time and training- time settings. For the former, select the transferred instruction samples with the lowest perplexity; while for the latter, incorporate multiple transferred results with lower perplexity as the training data.
Results As shown in figure 4 our proposed denoising strategy stably improves the performance at the lesting time, and this improvement continues to increase when more instructions are sampled, which shows our method can successfully filter out those low-quality instructions to reduce noise during format transfer. In addition, the method can also improve performance the training-time setting but the improvement is not more evident when more transferred instructions are included in the training data. It reveals that the model less sensitive to noise during the training phase.
6 Training Offline Model for Format Transfer
Converting large-scale instructions via OpenAT APT can cause substantial costs for APT calls. To alleviate the reliance on OpenAl API, it is necessary to derive an offline model that has comparable format transfer
We merely use the positive example (P) as mentioned section
noise assessment..

---

Published in Transactions on Machine Learning Research (12/2023)
performance to GPT3.5 but involves fewer costs. Hence propose to distill the format transfer ability of GPT3.5 into small-scale models.
Finc-tuned Offline Model with Knowledge Distillation Compared with larger models, small offline models are less capable of completing format transfer directly through in-context learning without training. Therefore, we strive to enhance small-scale models via knowledge distillation (Hinton et al., 2015). In pursuit of higher quality, we always make GPT3.5 convert the relatively complex and informative instruction format. (e.g., Ni-v2) into simpler and less informative one (e.g,, PromptSource). In this way, we obtain parallel data and use to fine-tune GPT-J for format transfer. We use the generated PromptSource-style instructions the source and the original Ni-v2 instructions the target to construct dataset of approximately 3,000 instances. To assess the quality of GPT-J's transfer results, we compare them with the hcuristic bascline and GPT3.5's conversion results in the lesting-time setting with two formats (DP and DPN)
Table 4: Results of training offline model (GPT-J) for format transfer at testing time. We compare the transferred instructions using heuristic rules, GPT3.5, or our fine-tuned GPT-J. Other settings are similar to those in Table
DiversePromp
Flan
CrossFit
PromptSource Rouge-L
Average
Format Method
EM
Rouge-L
EM
Rouge-L
EM
Rouge-L
EM
EM
Ronge-I.
heuristic
34.7
45.1
31.4
44.8
43.7
56.0
27.3
32.7
34.3
44.6
DP
GPT3.5
34.2
45.4
32.6
46.3
49.1
60.1
29.2
34.7
36.3
46.6
GPT-J
35.2
45.6
33.5
46.6
43.6
54.5
31.6
36.4
36.0
45.8
henristic
30.6
43.5
31.9
45.3
43.5
55.5
29.0
33.9
33.9
44.8
DPN
GPT3.5 31.5
44.3
34.8
48.3
50.3
60.4
30.8
36.1
37.1
47.6
GPT-J 34.7
45.7
34.8
48.4
46.0
55.5
31.4
36.5
36.7
46.5
Results As exhibited in Table 4 the fine-tuned GPT-J performs much better than the heuristic baseline but slightly worse than GPT3.5. This shows that our method can distill the format transfer ability into small-scale models, which saves the costs in practice. Additionally, the performance is highly correlated with the similarity of the source and largel formats. For instance, for DiversePrompt whose instruction format is similar the target format, the transfer process is less challenging. As a result, the fine-tuned model demonstrates comparable or even superior performance than GPT3.5. Conversely, for CrossFit which only describes keywords and lacks natural language instructions, it is more difficult for small models to produce high-quality instructions resulting in inferior performance.
7 Further Analysis
Effects of the Target Unified Format In previous experiments, we mainly use Ni-v2 as the target instruction format. To verify the versatility of UIT for various target instruction formats, we select Flan, an instance-level dataset as the larget datasel and conduct Lesting-time transfer experiments. Results are shown in Figure 6 from which we find that testing-ting time format transfer brings even more significant performance improvements than the scenario when Ni-v2 is selected as target dataset. This again validates our hypothesis that format consistency is essential to OOD generalization for instruction tuning, matter which target format is.
Effects of Model Scaling As observed in previous works [Iyer et al. 2022), larger models tend to perform better in following human instructions. We also conduct model scaling experiments in the Lesting-time setting with T5 (Raffel et al. 2020), with the model size ranging from 5 million (T5-small) to 10 billion (T5-XXL). Results presented in Figure demonstrate that in general, the performance tends to improve as the model size increases. These findings suggest that instruction format consistency is consistently essential language models of various sizes.

---

Published in Transactions on Machine Learning Research (12/2023)
Transfer Results
Scaling test on T5-LM
55
Flan Urified
Rouge-L
43.0 0
CrossFit Unifeed
50
Unified
Percentage
Flan Heuristic
45
EM (96
Heuristic
.00
25
15.4
25
small
base
large
XXL
Model Size
unified
Figure 5: Results of T5-LM of different model sizes on the tosting-time transfer sctting.
Figure 6: Testing time transfer experiment results when Flan selected as the target dataset and Ni- v2 as the source dataset. Transferring Ni-v2 to the target format brings significant, performance improve- ments during inference when training is conducted with target format. Results in the form of table are presented in Section C of the appendices.
Table 5: Experiments for task diversity and format consistency. For task diversity, we the training datasct to sace, dicf same diff. For format consistency, either the raw format. use the unified format.
Method
src s same
src+diff
src+ t
EM
Rouge-L
EM
Rouge-L
EM
Rouge-L
raw unified
29.3
46.3
28.3
45.3
29.1
45.8
30.8
47.6
30.7
47.7
31.0
47.8
Task Diversity v.s. Format Consistency We show that both task diversity and format consistency have impacts on the generalization performance for instruction tuning. As task diversity can only a variable during the training stage, we only conduct training-time time transfer experiments. Specifically, we choose Ni-v2 as the target dataset with DP as target format and P3 as the source dataset. We first randomly select 20 tasks from Ni-v2 (denoted as src). Then choose the same 20 training tasks from P3, denoted as same, and 20 different lasks from P3, which is denoted as diff. We treat whether to integrate same diff to the training set (src) as variable and evaluate on the original Ni-v2 test set.
As shown in Table 5 no matter which tasks are chosen as the training data, our UIT always performs better than the vanilla baseline (raw), which again demonstrates the importance of format consistency. We can also observe that without format unification, src+ performs better than sre+diiff, which indicates that increasing task diversity may be inferior without format consistency. Besides, sa performs the best among all combinations, suggesting that increasing task diversity and maintaining format consistency at the same time is the best practice for merging datasets in instruction tuning. We believe this finding can guide practitioners to better prepare the datasets for instruction tuning in the future.
with UIT
8 Conclusion
In this paper, we propose the unified instruction-ttuning framework (UIT), a standardized approach to enhancing the generalization ability for instruction tuning by unifying the format of existing instruction tuming datasets and enabling format transfer between them with LLMs like GPT-3.5. With the framework, we (1) exhibit the significance of format consistency in instruction tuning: (2) enhance the generalization
10

---

Published in Transactions on Machine Learning Research (12/2023)
performance (9.3% in Exact Match, 7.6% in Rouge-L) various datasets such as PromptSource, FLAN and CrossFit on T5- (3) propose denoising method and an offline model training method to make our CIT more feasible in practice.
In general, we study an under-explored facet, i.., the format consistency, for instruction tuning, and we hope our work could facilitate more attempts relevant areas.
9 Limitation
While our proposed UIT framework and format transferer offer promising approach to enhancing the generalization performance of instruction-tuned LLMs, several limitations should be acknowledged. Firstly, our method relies the assumption that the user knows the target instruction format in advance, which may not always be the case. Secondly, we focus instruction tuning for NLP tasks, instead of broader settings (e.g., real-world instructions (Taori et al.) 2023 Chiang et al. 2023) where formats are hard to define. We expect future works to explore whether our UIT framework can be applied to broader scenarios.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (Grant No. 62306159).
References
Ateret Anaby-Tavor Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep lcarning to the rescue! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7383 7383-7390, 2020.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupla, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In ICLR. OpenReview.ned, 2022.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93-104, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liammin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URI. hhttps://mms/:: or owggbloggglooooooooooooooooooggo
2023-03-30-vicunel
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fieetuee language models. arXiv preprint arXiv:2210. 11416, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint Xiv:22305. 14233, 2023.
Tianqing Fang, Quyet V Do, Hongming Zhang, Yangqiu Song, Ginny Y Wong, and Simon See. Pseudoreasoner: Leveraging pscudo labels for commonsense knowledge base population. arXiv preprint arXim:2210.0r: 07988, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXio preprint 2015.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022a. URL hhtttp::/aardv..or//bb/////1///
11

---

Published in Transactions on Machine Learning Research (12/2023)
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. CoRR, htss/2205.1077 10782, 2022b. doi: 4..5550//2X1.2225511777 URL https://doi.org// 400.10//85///.i. www.:2205...00 2205.10782
Srinivasan Iyer, Xi Victoria. Lin, Ramakanth Pasumuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprinl Xix:2221 12017, 2022.
Pang Wei Koh and Percy Liang. Understanding black- box predictions via influence functions. In International conference on machine learning, pp. 1885-1894 PMLR, 2017.
Ronan Le Bras, Swabha Swayamdipla, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference on Machine Learning, pp. 1078-1088 PMLR, 2020.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint 2rX1:22001.1388. 2023.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470 3487, 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprinl ar urXi::2203.021 02155, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.
J. Mach. Learn. Res., 21:140:1-140:6; 2020.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: //t::://gttttttt wwtp://wgggolll//////aaaaaaaaaaa
jax. May 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10500 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashii Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, al. huperrnaturaliistrurtte Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022b.
Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. PromDA: Prompt-based data augmentation for low-resource NLU tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4242-4255, Dublin, Treland, May 2022c. Association for Computational Linguistics. doi: 10.18653/v1 www...acloooog..220... URL https://ac.lanthologg...o
ac1-long. 292
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint 0tte2199.01652 2021.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang. Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1008 1025, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v /1/////11222/// http//2...iniinn/pe// findings-emnlp. 90. URL htttp:://aclarthologgy org/2020. http://22..finniiiiee findings-ennlp. 90
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shol learning challenge for cross-lask generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163-7189, Online and Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 10.18653/vl 23653/v1/202111 2etp00..mmmpmmmmii557.. URL 2021. 1021eemmlp-maiee 572.
.rgggg org/
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163 7189, 2021b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini lyer, Jiao Sun, Yuming Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arviv preprint arXiv:2305. 1 11206, 2023.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Appendices
A Case Study
We list some examples of our format transfer process in this scction. You can find examples of Defintion, Positive examples, Negative examples and Explanation in these cases.
Transferring from P3 to Ni-v2 (Task: trec):
Instruction: You are given a sentence that contains a question and a possible answer type. Your task is to identify the correct answer type from the suggested options. You may need to read the sentence and its context carefully order to determine the correct answer type.
Positive Examples:
Input: What do bee hives do in cranberry bogs Msls this asking about Description, Entity, Abbreviation, Person, Quantity, Location?
Output: Description
Explanation: The question is asking for a description of what bee hives do in cranberry bogs. So the correct answer type is Description.
Negative Examples:
Input: What golfing accessory was patented by George Grant on December nIs this asking about Description, Entity, Abbreviation, Person, Quantity, Location?
Output: Quantity
Explanation: The sentence is asking about a golfing accessory. These types of questions lypically require an answer about an entity (i.e. specific object or thing), so the correct answer type is 'Entity' rather than 'Quantity'.
Original P3(trec):
Instruction: {Input} (n (aa DDescriptos: Description, Entity, Abbreviation, Person, Quantity, Loca- n'neet Descriptor?'n
Positive Examples:
Input: What do bee hives do in cranberry bogs ?n
Output: \nDescription
Transferring from Flan Ni-v2 (Task: wsc)
Instruction: In this task, you given two sentences (sentence 1 and sentence 2). If sentence 1 implics that sentence 2 is true, answer "Yes", otherwise 'No".
Positive Examples:
Input: sentence 1: As Ollic carried Tommy up the long winding steps, his logs ached. sentence 2: Tommy's legs ached. options: no yes.
Output: no
Explanation: The sentence does not imply that Tommy's 's legs ached. So, the output should be "No". Negative Examples:
Input: Sentence 1: Paul couldn't find his car keys, so he had to walk. Sentence 2: Paul had to walk.
Output: Yes
Explanation: Sentence implies that sentence 2 is true, so the correct output should be 'Yes'.
Original Flan (wsc):
Instruction: If "(sentencel)" can we conclude that "(sentence2) (Ssentence)""muooptiiii " (optiooss
Positive Examples:
Input: If As Ollie carried Tommy up the long winding steps, his legs ached.", conclude that 'Tommy's 's legs ached." "nlodd"" no'nn- yes.
Output: no
14

---

Published in Transactions on Machine Learning Research (12/2023)
Transferring from CrossFit Ni-v2 (Task: imdb)
Instruction: You are given a sentence a paragraph describing a particular topic. Your task is to classify the sentiment of the sentence/
ventence/paragraph paragraph either negative positive. Label the sentiment in the output as per the given parameters without incorporating any additional information into your answer. Positive Examples:
Input: T am a big fan of Faerie Tale Theatre and I've seen them all and this is one the best! It's funny, romantic, and a classic. recommend this for all ages, It's great for little kids because it's well, Cinderella and great for adults and teens because it's funny and not over the top. watched it when was little and I still watch it now. It has great lines that my family and I quote all the time. The acting is great and it never gets old. If you like fairy tales and romances you will love this. I've watched many Cinderella movie in my time and this is the best of them all. (Sorry Disney) highly recommend this movie and all the Faerie Tale Theatre shows. They all appeal to all ages and are all unique and very entertaining. Output: positive
Explanation: The sentence describes a very positive opinion on the chosen topic. The opinion supported by facts, like the uniqueness of the show, lasting values, great acting, and so on. Hence, the sentiment of the sentence is classified as positive.
Negative Examples:
Input: know few things that are worst. A few. It had couple of funny scenes. It is movie not appropriate for kids but, only child would find this movie hilarious. This is definitely movie that you would like to use a free rental coupon for. Don't waste your money just to laugh a couple of times.
Output: Positive
Explanation: The given sentence mainly negative in nature as suggests not waste money on the movie. The words \ ''definetly', and \'a couple of funny scenes). are used in the sentence provide a bit of contrast, yet it does not make the overall sentiment of the sentence positive. Therefore, the correct answer should be 'neegatiee)" instead of ''positive\..
Original CrossFit (imdb)
Instruction:
Positive Examples:
Input: I am a big fan of Facric Tale Theatre and I've seen them all and this one the best! It's funny, romantic, and classic. recommend this for all ages. It's great for little kids because it's well, Cinderella and great for adults and teens because it's funny and not I still watch it now. It has great lines that family and I quote all the time. The acting is great and it never gets old. If you like fairy tales and romances you will love this. I've watched many a Cinderella movie in my time and this is the best of them all. (Sorry Disney) highly recommend this movie and all the Faerie Tale Theatre shows. They all appeal to all ages and are all unique and very entertaining. Output: positive
the top. I watched it when I was little and
Transferring from Ni-v2 to Flan (Task: dialogre):
Instruction: (input) Identify the name of one of the speakers in the given dialog.
Original Ni-v2 (ddaloogggr
Instruction: You are given dialog between 2 or more individuals. Within the dialog, there will be clues to the names of the speakers. You will be asked at the end of the dialog to identify the name of one of the speakers.
B Detailed Results of the Denoising Strategy
This is the detailed results of the performance of the denoising strategy with different. number of samples.
15

---

Published in Transactions on Machine Learning Research (12/2023)
Table 6: of samples.
The performance of the denoising strategy the testing and training time with different numbers
Testing time
Training time
Sample Num
EM
Rouge-L
EM
Rouge-L
32.4
38.3
37.4
56.1
2
33.0
38.9
39.4
57.6
33.5
39.4
38.2
56.5
8
33.7
39.7
38.8
57.0
16
33.8
39.8
38.5
56.3
C Results with Flan Selected as the Target Dataset
This is the detailed results of Testing time transfer experiment results when Flan selected the target dataset and Ni-v2 the source dataset.
Table 7: Testing time transfer experiment results when Flan selected as the target dataset and Ni-v2 the source dataset. Transferring Ni-v2 to the larget format brings significant performance improvements during inference when training is conducted with target format.
Source
Target
Method
EM
Rouge-L
Ni-v2
Flan
heuristic
18.4
28.3
Ni-v2
Flan
unified
29.3
43.0
Seed Data
Example 1
Task description A: Review: (sentence) Is this movie review sentence negative or positive? {options }
Task description B: In this task, you are given sentences from movie reviews. The task is to classify a sentence as 'POS" the sentiment the sentence is positive or as "NEG" if the sentiment the sentence is negative
Positive examples: Input b positive 1: It lovely film with lovely performances by Buy and Accorsi. Output b positive 1: POS Explanation b positive 1: The sentiment of the sentence is positive. Hence, the label is POS.
Input b positive 2: Here's yet another studio horror franchise mucking up its storyline with glitches casual fans could correct in their sleep. Output b positive 2: NEG Explanation positive 2: The sentiment of the sentence is negative. Hence, the label is 'NEG:
Negative examples: Input b negative 1: A smart, witty follow-up. Output b negative NEG Explanation b negative 1: Although the sentiment of the sentence positive, the label is 'NEG' Hence, the label should be PPOS:
Input b negative 2: Ultimately feels empty and unsatisfying, like swallowing a Communion wafer without the wine. Output negative 2: POS Explanation b negative 2: Although the sentiment of the sentence is positive, the label is 'POS. Hence, the label should be 'NEG:
Example 2
Task description A: (question1) (question2) Would you say that these questions the same? (options
16

---

Published in Transactions on Machine Learning Research (12/2023)
Task description B: Here are two questions (Question1 and Question2). If these questions have the same meaning and same answer, answer "Yes", otherwise "No".
Positive examples: Input b positive 1: Questionl: How do get into my Instagram if I forgot my email and my Facebook password?, Question2: forgot my password and also my email password. how can I get back that account? Output positive Yes Explanation positive These questions have the meaning and the same answer. So, the output should be 'Yes'.
Input b positive 2: Questionl: Why don't Hong Kong residents emigrate from their cramped & stressful city, like to places such as Australia?, Question2: Why made Hong Kong so attractive to Britain as colony given that it was the last of Britain's colonies and Britain does not profit from taxing Hong Kong? Output b positive 2: No Explanation b positive 2: The first question is about the emigration of Hong Kong residents and the second question is about the attraction of Hong Kong. So, they don't 't have the same meaning.
Negative examples: Input b negative 1: Questionl: Why are there so many accidents on I-8807, Question2: Were there accidents in outer space? Output. b negative 1: Yes Explanation b negalive 1: Questionl asks about the cause of the accidents, while question2 inquires about their existence. So, they are different and the correct output should be 'No'.
Input b negative 2: Question How do you determine the number of neutrons of an element or its ion?, Question2: How do you find the number of neutrons an element? What are some examples? Output b negative 2: They are the same. Explanation negative 2: Note that you need to answer with "Yes" or "No' and other answers are not acceptable.
Example 3
Task description A: {context) Generate question about the above context.
Task description B: Based on the given context, craft a common-sense question, especially those that are LONG, INTERESTING, and COMPLEX. The goal to write questions that are easy for humans and hard for AT machines! To create such questions, here are some suggestions: A. What may (or may not) be the plausible reason for an event? B. What may (or may not) happen before (or after, or during) an event? C. What may (or may not) be plausible fact about someone (or something)? D. What may (or may not) happen if an event happens (or did not happen)? You can also create other types of questions. DO NOT make your question answerable without looking at the context, or question of which the correct answer can be directly extracted from the context. DO NOT ask a question that requires very specialized knowledge that is not common sense. DO NOT ask too simple or too short questions. Your question must be related to the context and answerable with common sense. Try to add more variations and complexity to the questions.
Positive examples: Input b positive Context: I was told, in person over the phone, that my shoes were their way. They have my money. I have shoes. Output b positive 1: What may happen before called them? Explanation b positive 1: The question can not be answered directly from context. and requires commonsense.
Input potitive 2: Context: you see at my age relationship is kind of important and i thought i got the one after all these years noticed that once again was wrong i was good simply because was good was caring. helping, supportive, bla bla blaaa Output b potitive 2: What may happen to me? Explanation b positive 2: The question can not be answered directly from context and requires commonsense.
Negative examples: Input b negative 1: Context: was told, in person over the phone, that my shoes were their way. They have my money. have no shoes. Output b negative 1: What is on the way to my home? Explanation b negative 1: It can be directly answered with a span of the context and does not require any commonsense reasoning.
Input negative 2: Context: GPS technology dates back to the time when first ever satellite was launched in the sky in 1979. The era of global positioning started then. Output b negative 2: What was launched in the sky in 1979? Explanation b nogative 2: It can be directly answered with a span of the context and docs not require any commonsense reasoning.
17

---

Published in Transactions on Machine Learning Research (12/2023)
E
Model Implementation Details
The hyper-parameters for training include a maximum source data length of 1024, maximum target data length of 128, cap of 100 instances per task for both training and evaluation, batch size of 16 for training, a learning rate of 0.00001, a total of training epochs, linear learning rate scheduling, and a warm-up period consisting of 1000 steps.
18


Reviews:
[{"review_id": "akhMTWOmn3", "reviewer": "Reviewer_nGZc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Approach is intuitive and can likely be used broadly due by being focused on fixing the data itself.\n* Sections 1-3 were easy to follow.\n* Grammar and spelling seem good.\n\nWeaknesses:\n* The paper is written in a way that perhaps naturally follows in the development of research but is awkward to read, starting at Section 4. These later sections continue to expand on the base idea presented in the earlier sections, namely by focusing on the dataset dimensions D/P/N/E.\n    * Section 4 claims success but Section 5 proposes mitigations to limitations in Section 4. An alternative way to structure the experiments would be to combine Section 4 and 5 together, where “raw”, “heuristic”, “unified” were compared alongside “unified_denoised”.\n    * Similarly, Table 5 is overlapping data from Table 2.\n    * Section 7 seems to not have a coherent story, making it hard to follow. Additionally, the experiments and corresponding results seem more limited in scope compared to the rest of the paper. \n* There are key areas with missing details for how the idea of the work maps to a concrete implementation (e.g., what is the data, program, manual intervention required):\n    * Can you provide more information on the training/testing setup? The only relevant details I saw were that the model is t5-LM-xl. For example, it’s not clear how pre-training, fine-tuning, and in-context learning fit into the evaluation, which is itself split between a training and testing evaluation. How does it compare to Tk-INSTRUCT in the Ni-v2 work? Section 6 similarly has more training details that could have been included (loss curves, hyper parameters, etc.).\n    * Can you provide more information on the manual labeling for GPT-3.5 exemplars? I don’t see any methodology of how this is done.\n    * Can you provide more information on the heuristic approach? It similarly requires manually designed rules. Furthermore, it seems quite competitive to Unified Instruction Tuning and doesn’t need the mitigations (e.g., denoising) that are required for using an LLM.\n* Summary parts of the work speak in general terms but the reader will want to know what models were evaluated, what was the range of improvements, etc.\n    * Abstract and conclusion don’t have any specific results\n    * Nearly all the results are complex tables (e.g., data is not naturally ordered), so the reader has to compare every table entry with every other table entry to get a sense of trends. Even Tables with a natural ordering, such as the N variable in Table 4, are presented in table format rather than something like a line plot. Color, annotations with percentage improvement to a baseline, and analysis in the caption can help reduce the burden on the reader.\n* It’s not clear where the approach’s generality breaks down. How specific is the framing of D/P/N/E in the experiment setup to Ni-v2? While the approach seems intuitive, the evaluation is difficult to understand because the approach is described in a general manner (before the evaluation) but the evaluation is specific to aspects of particular datasets (e.g., Ni-v2). The evaluation’s focus makes it seem like it would be easier to explain Figure 2 and Figure 3 by incorporating the references to D/P/N/E inside Figure 2 and 3 explicitly. For example, “we chose DP as the unified instruction format” would seem to make more sense if Figure 2 had “Definition” and “Positive Examples” highlighted accordingly. As of now, I am struggling to see how the specifically defined D/P/N/E fields flow into exemplars that are fed into GPT-3.5 which are then fed into training or testing. It also seems difficult to understand the paper without reading the Ni-v2 work first.\n    * In the Ni-v2 work, the explanation seems to be a field of the positive (P) and negative (N) examples, but the current framing puts Examples (E) in the same logical level as D/P/N (e.g., “siblings” rather than “children”). This is potentially confusing, because P and N each have their own E.\n    * The unified approach as well as the heuristic could also be explained with a few examples. The examples in Figure 2, Figure 3, and appendix don’t seem to clearly state their relationship with the D/P/N/E framework.\n    * What is the behavior (e.g., hallucinations) if more fields are used in the Unified Instruction Format than allowed by a source dataset (e.g., if some of the fields in D/P/N/E are missing)? Section 6 mentions intentionally going from complex formats to simpler ones, which seems related.\n    * What makes training so different from testing, given the differences noticed in the results?\n    * Table 6 shows results generalizing past Ni-v2. Can you add an additional dataset other than flan?\n* Some of the hyperparameters aren’t motivated. What is the significance of the four unified instruction formats selected (e.g., Table 2)? One of the observations is that DPN performed the best, but do we have any explanation for why this may be the case?\n    * Can you provide examples of samples from Table 2 (with analysis) so the reader gains intuition?\n* Cost for Section 6 is not motivated. Can you quantify how many LLM queries are required to train a model (Section 6)?\n\nNits:\n* Table 1 seems squished due to abbreviations, but it’s still not clear what the reader is supposed to learn from it. The “P3” with “Total Exp. Entry” entry is inconsistently italicized.\n* Table 2 “Average” column should be marked as different from the other columns representing datasets.\n* Figure 2 has “theinformal” as one word.\n* Some typos: “hypothsis”, “noises”\n* “Without loss of generality” shouldn’t be used unless the results really hold in all of generality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Motivated by reduced performance in out-of-distribution settings, this paper describes a method for automatically transferring datasets of one format into a dataset of a different format. The method uses either heuristics or LLMs to perform the transfer. The transfer can be applied either at training time or testing time. The paper also introduces methods for denoising the samples from the LLM and distilling potentially cheaper models from the LLM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n* Provide details on how experiments were run (e.g., model details, manual intervention, heuristics) as mentioned in weaknesses.\n* Summarize and clarify the idea of the work and the analysis of results as mentioned in weaknesses. Ideally, key takeaways of the paper and every figure are clear (e.g., including summary statistics or general trends).\n\nSuggested:\n* Add ablations and study on the variability of the results as mentioned in the weaknesses.\n* Reorganizing and merging Section 4, 5, 6, 7 to reduce overlap and fragmentation as mentioned in the weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t0qc0gGwHP", "reviewer": "Reviewer_NsUB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is well-organized and the proposed framework is easy to follow.\n2. The studied OOD generalization problem among diverse datasets can be valuable.\n3. The evaluation on various benchmarks demonstrates the effectiveness of the proposed format transfer.\n\nCons:\n1. The major contribution lies in the format transfer from source format $\\mathcal{F}_s$ to a unified one $\\mathcal{F}_t$. Although the authors provide several variants of unified formats in Table 2, some of the comparisons are unclear or missing. For example, how is the baseline \"Heuristic\" constructed? How about directly transferring the format of source $\\mathcal{F}_s$ to the one in test $\\mathcal{F}_t$ for each test set instead of a unified format across all the test sets?\n2. The cost of LLM's in-context learning in practice as well as the ratio of failure cases are unclear.\n3. The explanations (E) is excluded in Table 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to tackle the OOD generalization of large language models on diverse datasets from the perspective of instruction format tuning. Specifically, a framework called \"Unified Instruction Tuning\" is introduced, which utilizes the LLM's in-context learning to perform instruction format transformation into a unified format for diverse datasets. Experiments on various benchmarks as well as different settings demonstrate the effectiveness of the proposed framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please clarify the results in Table 2 by providing more details of the baseline \"Heuristic\" and discussing the scenario of format transformation for each test set.\n2. Please discuss the cost of in-context learning and the failure cases.\n3. Please include explanations (E) in Table 5 or provide the reason why it is excluded.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EV6SLEommW", "reviewer": "Reviewer_HFkM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n\n+ To my knowledge, this is the first work that explicitly models and considers the format consistency problem for instruction tuning. But I would like to admit that I am not an expert in this area so I cannot accurately assess the technical value of this submission.\n+ The paper is well-written. I can follow it easily. And the discussions on limitations are included.\n+ Comprehensive quantitative results are provided with good illustrations for concept demonstration.\n\nAfter reading, I’m satisfied with most of the contents in this submission, except for some minor points:\n\n**Cons:**\n\n+ The proposed method looks to be ad-hoc. Specifically, the OpenAI GPT 3.5 based framework, the format denoising and the offline model training appear to be three different improvement tricks instead of one single solution based on one point. I am looking forward to a more principled solution. But I guess it is just a problem of personal taste.\n+ Table 1-7 mainly report the quantitative comparisons for the ablated alternatives. I am curious whether the format consistency could help the existing instruction tuning methods. I guess the format consistency is perpendicular to their technical contribution and it is not difficult to implement? For example, what happens when the format consistency is incorporated into the state-of-the-art models which are trained over a set of datasets? Could it significantly improve the empirical results?\n+ Except the reported quantitative scores, can the authors also show some qualitative comparisons? For example, can the authors show a pair of outputs where one is from the full model with the format consistency but another is not equipped with? Can the authors show some comparison results for several different SOTA baselines?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors explicitly model the concept of format consistency for instruction tuning which is important to a number of down-streamed LLM-based applications. Specifically, a format transfer framework named Unified Instruction Tuning (UIT) is developed. Together with the denoising strategy and offline model training, empirical improvements are shown in some experiments to support their efficacy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness section for the requested changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper models and discusses format consistency for instruction tuning, which is important to LLM-based applications. The focused problem is interesting and comprehensive quantitative results verify the effectiveness of the method. Though reviewers raise some concerns (e.g., the method's motivation) , most of them have been resolved in n the response phase. All the reviewers agree that the proposed method fit the TMLR acceptance criteria and post positive comments. 

Overall, I recommend accepting this paper.
Decision: Accept (poster)