#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models
Authors: Keming Lu
Hongyi Yuan
Zheng Yuan
Runji Lin
Junyang Lin
Chuanqi Tan
Chang Zhou
Jingren Zhou
Venue: ICLR 2024 poster (2024)
Keywords: Data Analysis
Supervised Fine-tuning for Human Alignment
Large Language Model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Pre-trained large language models (LLMs) can understand and align with human instructions by supervised fine-tuning (SFT).
It is commonly believed that diverse and complex SFT data are of the essence to enable good instruction-following abilities.
However, such diversity and complexity are obscure and lack quantitative analyses.
In this work, we propose InsTag, an open-set instruction tagging method, to identify semantics and intentions of human instructions by tags that provide access to definitions and quantified analyses of instruction diversity and complexity.
We obtain 6.6K fine-grained tags to describe instructions from popular open-sourced SFT datasets comprehensively.
We find that the abilities of aligned LLMs benefit from more diverse and complex instructions in SFT data.
Based on this observation, we propose a data sampling procedure based on InsTag, and select 6K diverse and complex samples from open-source datasets for SFT.
The resulting models, TagLM, outperform open-source models based on considerably larger SFT data evaluated by MT-Bench, echoing the importance of instruction diversity and complexity and the effectiveness of InsTag.
InsTag has robust potential to be extended to more applications beyond the data selection as it provides an effective way to analyze the distribution of instructions.

Published as a conference paper at ICLR 2024
#INSTAG: INSTRUCTION TAGGING FOR ANALYZ- ING SUPERVISED FINE-TUNING OF LARGE LANGUAGE MODELS
Keming Lu' & Hongyi Yuan" && Zheng Yuan & Runji Lin
Alibaba DAMO Academy [lukeming. 1km, yww.yuuggnnnnnnommmm yhy, yuanzheng.
trranzhengyyuazhhen,. yuanzuun, Miaanzhen,Linuujji..rrir http:///jj...ieelllbabbbbb-....o
Junyang Lin & Chuanqi Tan & Chang Zhou & Jingren Zhou
Alibaba DAMO Academy [junyang. 1jy, chuanqi. tcq, ericzhou. zc, zc. eat.rrennnooooealaaaaaea com
ABSTRACT
Pre-trained large language models (LLMs) understand and align with human instructions by supervised fine-tuning (SFT). is commonly believed that diverse and complex SFT data are of the essence to enable good instruction-followe abilities. However, such diversity and complexity are obscure and lack quantita- tive analyses. In this work, propose INSTAG, an open-set instruction tagging method, to identify semantics and intentions of human instructions by tags that provide access definitions and quantified analyses of instruction diversity and complexity. We obtain 6.6K fine-grained tags to describe instructions from pop- ular open-sourced SFT datasets comprehensively. We find that the abilities of aligned LLMs benefit from more diverse and complex instructions in SFT data. Based on this observation, we propose data sampling procedure based on IN- STAG, and select 6K diverse and complex samples from open-source datasets for SFT. The resulting models, TAGLM, outperform open-source models based on considerably larger SFT data evaluated by MT Bench, echoing the importance of instruction diversity and complexity and the effectiveness INSTAG. INSTAG has robust potential to be extended more applications beyond the data selection as it provides an effective way to analyze the distribution instructions.
INTRODUCTION
The contemporary chatbots, such as GPT-4 (OpenAl 2023), ) have brought to the forefront ar- tificial generative intelligence with their superior and versatile abilities in real-world task solving. Such abilities are unlocked by fine-tuning pre-trained large language models (LLMs) to align human preference, and well-aligned LLMs can precisely recognize human intentions and properly formalize responses expressed in natural languages. There have been proposed various techniques to achieve such alignment of enabling pre-trained models to comprehend and execute diverse instructions fectively, including supervised fine-tuning (SFT) (Taori et al.] 2023 Chiang et al. 2023), rejection sampling (Yuan et al. 2023b Song et al. 2023 Rafailov et al. 2023), and reinforcement learning
with human feedback (RLHF) (Bai et al. 2022a
Ouyang et al. 2022 Touvron et al. 2023b).
Especially, SFT for alignment is widely studied by recent research, which generally formalized in multi-turn utterance manner, and each turn is composed of a human query and corresponding response well-aligned with human preference (Wang et al. 2023d). Achieving alignment with hu- man preference through SFT necessitates collecting a broad range of training data which is typically gathered through crowd-sourcing (Ouyang et al. 2022 Bai et al. 2022a; Touvron et al.) (2023)) or by distilling from other LLMss (Taori et al.) 2023 Ding al. 2023). Recent research indicates that such training data for alignment should be diverse and complex, covering various domains, tasks,
Eqqully Contributed. Order determined by swapping the one in Yuan et al. 2023a).
Work done during internships at Alibaba DAMO Academy.

---

Published as a conference paper at ICLR 2024
semantics, and formats (Xu et al., 2023a Mukherjee et al. 2023 Wang et al. 2023b). Such diversity and complexity are mainly determined by the query formation. Various methods are proposed and claimed to improve the diversity and complexity of the queries and advance the performance of the SFT-aligned LLMSs (Wang et al. 2023c] Xu et al. 2023a Ding et al. [003] inter alia). However, how to quantify the diversity and complexity of queries is significantly understudied.
To shed light on this topic, we propose using a tagging system to feature and categorize samples in SFT datasets. Given the versatile tasks that the aligned LLMs are expected to handle, an equally versatile tag system is necessary to distinguish open-world human queries. However, building an open, fine-grained tagging system manually is infeasible scale for large datasets. To this end, we propose INSTAG, an automatic INStruction TAGging method empowered by proprietary high- performing chatbot ChatGPT. Leveraging such well-aligned chatbot, INSTAG designs a framework to automatically prompt ChatGPT to assign tags to training sample queries. INSTAG achieves the increased quality of the tag assignment by deliberately prompting ChatGPT to explain each tag as- signed and including systematic tag normalization procedure. We apply INSTAG to an extensive collection of open-source SFT datasets and build open-set, fine-grained tags which, as we observed, can reflect the semantics and intentions beneath human queries in SFT datasets. Through the scope of tags, we conduct a detailed and quantified analysis of existing open-source datasets, providing in- sights into query distributions in terms of diversity and complexity. Such analyses reveal that diverse and complex queries induce high alignment performance through SFT. Following this insight, we propose a data selector based on INSTAG, including a complexity-fing diverse sampling method that can extract the most complex queries in a diverse distribution. LLMs fine-tuned with data selected by the INSTAG selector perform well on the popular benchmark MT-Bench (Zheng et al. 2023 supporting our previous query distribution insights.
The contributions of this work are mainly fhrde-fold. Firstly, we propose using open-set fine- fine-graine grained intention tags as instruction diversity and complexity metrics. To this end. we develop INS INSTAG, an automated annotator that leverages the citrructionfoollowiiii abilities of proprietary chatbots and employs tag normalization methods. Secondly, we analyze existing open-source SFT datasets and provide insights into query diversity and complexity. Finally, we design a data selector based on INSTAG and apply it to the latest open-source datasets. The resulting best LLMs, TAGLM-13b- v1.0 and TAGLM- PAGL.M-13b-22 13b-v2.0 respectively based on LLaMA (Touvron et al. 2023a) and LLaMA-2 Touvron et al. 2023b), trained with selected data achieve scores of 6.44 and 6.55 on the benchmark MT-Bench, respectively, surpassing group of LLMs aligned with considerably more SFT data. Our contributions are verified with rich experiments and multifaceted analysis. Most notably, INSTAG exhibits its robust potential to offer deeper insights into LLMs alignment, extending beyond the data selection introduced in our work.
2 RELATED WORKS
LLMs with Human Alignment. Through supervised fine-tuning (SFT), rejection sampling, or re- inforcement learning (Ouyang et al.) 2022 Bai et al. 2022a b Yuan et al. 2023b Rafailov et al. 2023 Song et al. 2023 Touvron et al. (2023)). LLMs can obtain versatile abilities for understanding and following diversified human queries expressed in natural languages, aligning with human inten- tions. Recent research mainly focused on SFT to align LLMs with human intentions and has con- tributed essential practices to developing open-resourced well-aligned LLMs, which is adequately summarized by ZZho et al. (2023). Several prominent works collected SFT data through human annotated demonstrations (Ouyang et al. 2022 Touvron et al. (023b), online user logs of propri- etary L.MSs (Chiang et al. 2023 Wang et al. 2023a Köpf et al. 2023), or prompting proprietary high-performing LLMSs (OpenAl 2023) to generate or rewrite samples (Taoi et al. 2023; Ding et al.)
2023 Xu et al. 2023a Mukherjee et al. 2023 inter alia). Different LLMs fine-tuned on the datasets have aligned with human preference and exhibited good performance in various real-world tasks.
Data for Human Alignment. It has been highlighted that the performance of aligned LLMs is affected by the quality of the SFT data. Such data quality pertains to the level of responses (Peng et al. 2023 Chiang et al. 2023), the difficulty of tasks presented (Mukherjee et al. (023), the com- plexity of queries (Xu et al. 2023a). the diversity of semantics (Ding et al. 2023] Taori et al. 2023), and the scale of sample amounts (Zhou et al. 2023). Taori et al. (2023) used Self-Instrun (Wang ett al., 2023c) to generate diversified queries for SFT and Xu et al. (2023a) proposed Evol-Instruct to complexify simple queries for better human alignment. Mukherjee et al. (2023) used propri-
2

---

Published as a conference paper at ICLR 2024
Open-source
Dataset
Open Tagging
SharedPT
MizardLN
Dully
OpenChat
DNCC
ULtraChat
Alpaca
FLAW
WATH
LIPA
Skarserit
Writt rlack rustes
Bevtl
Iote eurr meseicii healescare orhriders
sreate 1a JoM.
onner statas
the aoctsurry to the aboss sode.
cartee the mn wirs
astistted wiih
to
Tag lorsslization
catit Seama data lyis progr clac Ficatin
Writing softaare divelspentt
Quality Evaluation
Freqercy Filterlay
Coas develspeant
Annotators
Netrics
hule Appropation
nathenat Lics
prob Solve
Precision
Sessitic eeattion
incoonendationn Lin
Consistency
Association Augs spatiion
infor ofr
Saalling
eques
Codl Soool Tetrieve
Figure Overview of INSTAG. We use ChatGPT to annotate fine-grained tags for a series of open-source datasets. This figure presents two cases of open tagging annotations from ShareGPT and UltraChat. A tag normalization, including multiple denoising and aggregation methods, is then applied to the original tagging results. Finally, the quality of the tag set, as shown in the word cloud, is evaluated by human and L.M annotators, focusing on the tagging precision and consistency.
etary high-performing LLMs to rewrite the queries and responses of samples from FLAN collection (Longpre et al. (023) and observed improvement of LLMs in conventional NLP task solving. Ding eet al. (2023) proposed UltraCaat using manually designed diverse anchor concepts and entities to generate multi-turn data by inducing conversations in ChatGPT. OpenChat (Wang et al. 2023a)
and Vicuna (Chiang et al.] 2023) are both current open-sourced L.MMs with cutting-edge instruc- tion following abilities, and both models are trained on the user logs of GPT-4 from ShareGPT. As evaluated in Wang et al. (2023b), the success of fine-tuning on ShareGPT demonstrates that queries from user logs are of higher diversity and the responses generated from GPT-4 are of better quality, resulting in superior instruction following the abilities. Zhoou et al. (2023) found that a small amount of high-quality data is sufficient for LLMs to excel at human alignment.
Although current research proposed more diversified and complex SFT data and made significant progress in developing well-aligned LLMSs with human intentions, existing works have yet to discuss how to quantify the diversity and complexity of queries. Taking advantage of the ChatGPT, we annotate existing data samples with tags. We quantify the diversity and complexity of the training data for the first time and study the data mixture for better alignment.
INSTAG
This section presents an automatic instruction tagging method INSTAG and its preliminary analyses. We first define fine-gun ane-grained intention tags and present the open tagging process with LLMs ($3.I). Then, we design a systematic normalization method to denoise the raw tags from previous anno- tations (33.2). We also fully evaluate the tagging quality to ensure INSTAG generates precise and consistent intention tags (33)). Finally, we use INSTAG to analyze open-source SFT datsets ($3.4)
3.1 OPEN-SET FINE-GRAINED TAGGING
Instructions, queries in prompting modern chatbots, serve as expressions of user intentions, which can often be multifaceted and highly intricate. To illustrate, we showcase an instruction from the ShareGPT dataset in Fig. I where the user submits coding request specifying desired output for- mats and error- handlign methods. To better parse such instructions, employing fine-grained tags to identify fine-grate fine-grained intentions rather than relying generalized, coarse- coarse-grained classes is essen- tial. However, although fine- mee-grinedd intention tags offer a more detailed understanding of instruction distribution, they also present challenges in annotation and normalization. Therefore, we propose
3

---

Published as a conference paper at ICLR 2024
Inconsistency
Examples
Output
Lexical Noise
Information Retrieval, information retrieval, in-
information retrieval
Rule Aggregation
fomation retrieve
information request, request for information, request for additional information, request for more information, additional information re- quest, specific information request
Uncontrolled Granularity Semantic Aggregation
information request
Spurious Correlations Association Aggregation
(mathematics, math problem), (loop, for loop)
mathematics, for loop
Table 1:
Inconsistency in intention tagging results from open-set annotations. Inconsistencies can be addressed with three aggregation methods described $3.2
an open-set tagging with ChatGPT without providing a predefined ontology of tags and a normal- ization technique address these issues. We prefer an open setting since a closed set is not flexible enough to cover versatile intentions in open chatting. Our prompt for tagging is shown in Tab. We provide few-shot examples in the prompt to hint ChatGPT to provide tags in specific JSON format for accurate parsing. As shown in Fig. I we separately annotate each query in chat session and require ChatGPT explain tags for the convenience of quality evaluation briefly.
3.2 TAG NORMALIZATION
The production of intention tags through ChatGPT in an open setting presents a challenge in ensuring consistency, as no predefined ontology provided, resulting in noise in the raw tagging outcomes. The number of original raw tags for open-sourced datasets is over 12,000, showing ChatGPT can provide diverse and fine-gratd are-grained amnotations. However, we notice the original tagging results con- tain noticeable noises, including inconsistent word format and granularity. Therefore, we design systematic method to normalize the open-set tagging. We have identified three significant types of noise, detailed in Tab. I Lexical Noise, arises from the instability of ChatGPT in adhering to output format instructions and can be mitigated through stemming as a post-processing step: Uncontrolled Granularity refers to the potential for ChatGPT to produce overly specific tags; Spurious Cor- relations refer to tags that often appear together due to the bias of ChatGPT or data distributions. Such tag groups should be merged to form an atomic tag. These issues must be addressed to en- sure that intentions are accurately identified and utilized in downstream processes. Therefore, we normalize open-set tagging results by various aspects, including frequency, format, semanties, and associations. Specifically, we clean the raw tagging with the following normalization procedure:
We first filter out long-tail tags appearing less than a times in the whole annotated dataset. a hyperparameter related the scale of the dataset.
Frequency Filtering:
Rule Aggregation: We transform all tags into lower characters to avoid the influence of capi- talization and replace all special characters with spaces. Finally, we apply stemming to each tag with the support of NLTK (Bird et al. 2009).
Semantic Aggregation: We employ text embedding models to obtain the semantics of tags. We use PHRASEBERT (Wang et al. 2021). a BERT-based model designed explicitly for embed- ding phrases, such as titles of tags. Other embedding methods, such OpenAl embeddings or DENSEPURASE (Lee et al. 2020). can also be adopted as alternatives. After obtaining the se- mantic embeddings of tags, we DBSCAN algorithm (Hahsler et al. 2019) to cluster tags with a given threshold of semantic similarity. Similarly, other density clustering methods can be used instead of DBSCAN for the same denoising purpose. Semantic aggregation controls the granularity of tags in terms of semantic similarities.
Association Aggregation: We notice ChatGPT tends to provide highly associated tags that are ex- pected to be considered as atomic tag as whole, mainly occurring in mathematics and coding queries. Therefore, we analyze all raw tagging results and employ the FP-Growth algorithm (Han et al., 2000) mine association rules between tags. We then recursively merge associated tags based on the above association rules and reduce verbosity.
We apply INSTAG on 17 widely-used open-source SFT datasets introduced in Appx. $ Over 100 thousand original unique tags are generated following the ChatGPT annotation. To filter out long-
4

---

Published as a conference paper at ICLR 2024
Agreement (r:)
Metric
GPT-4 Annotation Human Annotation (1 ())
Humann-Huma Human-GPT
Tag Precision 16.1
100
0.47
0.92
Tag Consistency
86.6
100
0.73
0.75
Table 2: Evaluation for the tagging quality of INSTAG. We design two metrics, tagging precision and consistency, for evaluating INSTAG. Moreover, we also employ three human annotators to annotate 1% cases and report their majority voting. We report agreement between human annotators in Fleiss-kappa scores and agreement between majority voting and GPT-4 in Cohen's 's kappa scores.
tail cases, we implement Frequency Filtering with 20, resulting in the retention of 8,541 tags. We apply the rule aggregation to address lexical noise, which reduces tags to 7,157. Then, semantic aggregation with a minimum semantic similarity 0.05 reduces the count 6,587 tags. Finally, we employed the association aggregation with a minimum support of 40 times and a minimum confi- dence of 99%, producing 1,772 association rules transform tag groups into atomic tags. These measures were essential in streamlining the tagging process and ensuring the quality of downstream processes. An overview of frequent tags in Appx. $ We also train local specialized tagging LLM, INSTAGGER, to distill such annotation abilities into smaller LLMs, shown in Appx. SC
3.3 QUALITY EVALUATION
We employ both GPT-4 and human annotators provide judgments in tagging quality. The quality of the normalized tagging dataset is evaluated in precision and consistency:
Precision Precision whether tags assigned specific query correctly relate to query inten- tions. Tag precision is essential since fie-grained tags should be precisely expressed as part of query intentions. For example, given a case (q, 7) where q is the query and T is tags assigned to it, we employ annotators to identify any incorrect tags in T. We consider it negative case if any tag in T is annotated as incorrect. Otherwise, it a precise tagging case.
Consistency To form consistent tag ontology, we naturally require that the semantics of spe- cific tag will not shift across queries. An annotation case in consistency (t, contains a tag t and set of randomly selected instructions I annotated with such tag. Annotators are required to identify any semantic changes in tags across all instructions.
Specifically, we randomly sample 4,000 cases for GPT-4 annotation, 2,000 each for precision and consistency. Then, we hire three annotators to manually label 40 cases (1%) selected from the above set. Manual annotations provide judgments and reveal confidence of GPT-4 annotation. The eval- uation results are shown in Tab. 2] GPT-4 provides 96.1% 1% and 86.6% accuracy in tag precision and consistency, respectively. Meanwhile, we also report the majority voting of human annotators, which suggests a hundred percent correctness among both tasks. We notice the Fleiss-kappa be- tween human annotators reaches the basic agreement. In contrast, Cohen's 's kappa between majority voting and GPT-4 reaches more than 0.7, suggesting solid agreement between human and GPT-4 annotators. To eliminate the possibility that such results contain robust false positive annotations, specifically design counterfactual annotation experiments shown in Tab. 9 and proof that both human and GPT-4 are capable of precisely recalling incorrect cases. Therefore, tags provided by INSTAG are of good quality regarding precision and consistency for downstream analyses.
3.4 PRELIMINARY ANALYSIS
We present the analysis of open-source datasets through normalized tags in Fig. 2 To start with, we introduce the diversity and complexity attributes of SFT datasets induced by tagging results:
Diversity is used to access the range of intentions and semantics covered by queries in a dataset. According to the tagging results, dataset considered more diverse it covers more individual tags. The attribute is quantified as the unique tag coverage rate for the overall tag set.
Complexity aims to measure the number of intentions complicating queries. We assume a more complex query would be assigned more tags. The attribute is quantified as the average tag number assigned to queries in a dataset.
We first depict the overall assessments regarding the axis diversity and complexity as shown in Fig. 2a Each dataset is represented dot whose size indicates the sample size, and color indicates
5

---

Published as a conference paper at ICLR 2024
StarcGif
Vltak paseplodi
preti
Worgi MAMAA
Diversity (Tag Caverage Tate 9%)
Alpan 13
0.6
Absaa
DparChaa (V2)
0,01
0001
5:00
830
830
6:0
606
038
027
PEAN
0.1
FLAN
WitaSE (Alpax)
OpenAhai(yd)
6:5 830
paot
WandEM (Slax((F1)
UtraChai (umpled
x31 soh
uendtcall
SkareGrt
Jina
gensk
sasd
RK-45
Opeec botter)
Alpuss
wooal (atjoco Whst (iheertrry U grmpla sheestr
plgz
groce guReper
Math
Coke
Complexity (Arg. (Tag)
(a) Diversity and Complexity based on Tags
(b) Dataset Correlation (Column recalls Row)
Dataset analysis based on tags. Fig. 2a shows diversities and complexities based on tags, where data scales and AlpacaEval scores are marked in dot sizes and colors respectively. Datasets without AlpacaEval scores are marked in circles. Fig. 26 shows correlations among datasets based on the recalls of tags. Numbers are recalls using tags in the column against tags in the row.
Figure 2:
the performance of LLMs fine-tuned thereon and tested on AlpacaEval (Li et al. 2023). As shown, (1) Tag-based metries well presents diversity and complexity. WizardLM (A Alpaa is created by complicating the queries from Alpaca datasets using Evol-Instreut (Xu et al. (023a). We can see that WizardI.M (Alpaca) has a larger coverage rate and average tag number than the Alpaca dataset. This observation demonstrates the complexity and diversity of an SFT dataset can be well quantified by tags. (2) The larger size, the more diverse and more complex. On both axes, the larger datasets naturally contain human queries of higher diversity and complexity, except for mathematical reason- ing and code generation. (3) Math and Code show different trends. The datasets for mathematical reasoning (MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al. (021)) and code generation (DMCC (Li et al. 2022). MBPP (Austin et al. 2021). DrRepair (Yasunaga & Liang 2020)) focus on specific downstream abilities and result in low diversity, while such datasets have relatively high complexity, (4) Diverse and complex data induces higher performance. ShareGPT, UltraChat (Ding et al. 2023). and OpenChat-vl (Wang al.) 2023a) datasets lay the upper-right corner of Fig. 2a having both high diversity and complexity. Vicuna (Chiang et al. 2023). UltraChat, and Openchat, respectively fine-tuned on the datasets, achieve cutting-edge performance among open- sourced models, as evaluated by public leaderboards (e.g., AlpacaEval). This scenario verifies that LLMs can benefit from fine-tuning more diverse and complex data for alignment.
We display the correlations between datasets regarding tag recalls to understand the correlations between open-source SFT datasets. As depicted in Fig. 26. we use the tag sets of the datasets on each column to calculate the recall to the tag sets of the datasets from the figure that (I) Tags can identify different tasks. Datascts for mathematical reasoning and code generation tasks exhibit higher tag recalls within the tasks. This demonstrates that the tags can identify the uniqueness of mathematical reasoning and code generation datasets compared to more general-purpose datasets. (2) Few cover all. WizardLM (Alpaca), WizardLM (ShareGPT), UltraChat, and ShareGPT have higher tag recalls for other datasets. These four datasets contain more diversified queries and cover other datasets, consistent with the results in Fig. 2a
each row. We conclude
Overall, INSTAG provides a tool for analyzing SFT datasets through the perspective of tagging. Ex- isting SFT datasets differ in diversity and complexity as evaluated by the tagging results. However, we also notice two outliers in these figures. The dataset for Alpaca seems to have a large data size filtered
while resulting in inferior performance and low complexity. The dataset for OpenChat-vi from ShareGPT, resulting in high query diversity and complexity while having only 8K multi-turn conversations, which suggests a considerably small data scale with high query complexity and di- versity can potentially result in better performance. We give more analysis on data size in
INSTAG FOR DATA SELECTION
As analyses in $3.4, we notice fine-tuning LLMs on more diverse and complex datasets may benefit alignment performance. Therefore, we present data selection method supported by INSTAG in this

---

Published as a conference paper at ICLR 2024
Model
Data Size MT--Bench AlpacaEval
Proprictary Models
GPT-4
8.99
95.3
GPT-3.5-turbo
7.91
86.1
Claude-vI
7.90
88.4
LLaMA-2 Based Open-source Models
Llama-2- 2-1ha-----b-hat Touvron al. 2023b TAGLM-13b-v2.00
6.65
81.1
6.55
80.9
LLaMA Based Open-source Models
Alpaca-133b Taori al. 2023)
52K
4.53
21.9
OpenChat-130e 13b-v1 Wang
2023a
8K
5.22
80.9
Baize-v2-130 13b Xuu et al. 2023b
56K
5.75
67.0
(ecuna-13b-vll 1.1 (C hane ei al.
2023
70K
6.31
70.4
Wizard [Xu ct al. 202.5a)
70K
6.35
75.3
Vicuna- 1aauaa-13h 13b-v1.3 (Chang al.
2023
125K
6.39
82.1
TAGLM-13b-vL.0
6K
6.44:0.04
75.8
Table 3: Main results of TAGLM. Standard deviations are derived under three GPT-4 judgments and obtain results for other baselines from the official MT-Bench and AlpacaEval leaderboard. Dashes in the data column denote unknown data sizes. Detailed results are presented in Tab.
section and align LLMs with selected data to show the effectiveness of INSTAG. We introduced ex- perimental setup 4..1 results ($4.2). and analyses related to query diversity and complexity ($4.3)
4.1 EXPERIMENTAL SETUP
Data Pool. Based on the normalized tagging results and the preliminary analyses of open-source datasets as presented in Figure 2 we conduct fine-grained experiments to discuss the impact of data complexity and diversity through controlling- aaallinn--f----vriallll methods. Under the correlation analyses (TzarrLM(SSharGPPP) UltraChat, and ShareGPT maintains large tag recalls regarding other datasets. The four datasets also have the largest average tag numbers shown in Figure 2a] These results indicate that the four datasets have high coverage for pool the four datasets and create different subsets for
in Figure 2b each dataset of MaatdddMMAAlppcca
other datasets regarding tags. Therefore, data complexity and diversity analysis. The pooled dataset contains 306,044 samples with tag set size 6,398 and an average tag number of 4.48. Detailed datasets are in Appx. $
Configuration. We use the dataset of 6K samples to align the 13B version of LLaMA (Tourron al. 2023a) and LLaMA-2 (Touvron et al., 2023b) with human preference via SFT, and dub both aligned LLMs TAGLM--33-- 1.0 and TAG L---LM--12200 respectively. All the models fine-tuned in the following analyses are based on 13B version LLMs of either LLaMA (Touvron al., 2023a) or I.L.MA-2 Touvrro et al. 2023b). If not specified otherwise, we fine-tune the model for five epochs with the batch size set to 128 and the learning rate set to 2 x 10-5, The Vicuna-style template applied to concatenate queries and responses during fine-tuning. We evaluate each fine-tuned model on MT-Bench (Zheng et al. 2023) strictly following its recipe using GPT4 as a judge to demonstrate the alignment performance, comparison to other LLMs, and conduct analyses.
Data Sampling. LLMs can benefit from datasets with higher diversity and complexity according to the analyses in $3.4 We sample data subset of 6K samples from the pooled dataset with the highest sample complexity of an average tag number 16.56 and tag coverage 100%. We propose Complexity-first Diverse Sampling procedure (Cf-D, Alg. to obtain the datasets.
Baselines. We compare our models to two sets of baselines. We first use proprictary GPT-4, GPT- 3.5, and Claude-V V1 as strong baselines, and then include strong cutting-edge open-sourced aligned LLMs. Vicuma (Chiang et al.) 2023), WizardLM (Xu et al. 2023a), Baize (Xu et al.) (2023)). Open-
Chat (Wang et al. 2023a), and Alpaca (Taori et al. 2023).
Details are left to Appx. $F

---

Published as a conference paper at ICLR 2024
5.00
MT-Banch Score
Mi-Bana
575
Jog. Buselic
2.79
340
Kandam Besclin
Co Copplntess
Tag Complexit (Ang TTyg Session)
Tag Diversity (Tag Coverage Rate
%)
(a) Performance under different Tag Complexities
(b) Performance under different Tag Diversities
Figure 3: Analysis results of model performance terms of different tag complexities and diversi- ties. Fig. 3a shows MT-Bench scores over different tag complexities defined average number of tags per session. Fig. 3boshows scores over different tag diversities defined as coverage rates over all tags. We include random baseline in both figures as shown in red triangles.
4.2 RESULTS
TAGLM-13b-vll outperforms all the open-sourced aligned LLMs, achieving a 6.44 average score on MT-Bench, although it is only fine-tuned based on LLaMA on 6K samples, far less than those of other LLMs. We report the average three GPT-4 judgments and provide the standard deviation of scores we notice randomness in GPT-4 judgments. This result illustrates that diversity and complexity matter in human alignment by SFT. Our INSTAG provides a decent tool for accessing and quantifying both attributes. AAGLM-13b-v/e fine-tuned based on LLaMA-2 achieves even higher results while lagging behind LLaMA-2-chat by only 0. which is aligned with human preference via RLHF. Compared proprietary geth-performing LLMs, especially GPT-4, the performance falls behind on MT-Bench. We also present more detailed results on MT-Bench in terms of eight tasks in Appx. S We also report evaluation results on AlpacaEval and wit- ness a similar conclusion as MT-Bench. TAGLM-1330 11-vv11l.... 1.0 outperforms most of the baselines that are aligned with much more data and achieves a 75.8 win rate compare with text-davinci-003. but falls behind OpenChat- 13b-v1 and Vicuna- 1aamna-1bb-v... 113.. MGGLM-13b-v22 also achieves comparable performance on AlpacaFval with Llama-2-1 1-ma-22-3b--chat 13b-chat.
As shown in Tab. 3
4.3 DECOUPLED ANALYSIS
We primarily discuss how SFT data size relates the alignment performance and give an ablation study on the sampling procedure according to tags. The results are shown in Tab. 4] Using the sampling procedure in Alg. [ the alignment performance achieves the best score with 6K data, and the performance degrades when the data size increases 10K and 16K. As compared SFT with all and half of the whole pooled data, the performance re- mains superior. These results empirically verified there exists small-scale subset with high diversity and com- plexity that can lead to excellent alignment performance. The finding is consistent with LIMA (Zhou et al.) (2023). Comparing Alg. [ to random sampling with the same 6K samples, the proposed sampling procedure results in significantly better performance than random sampling, largely surpassing by 0.68 MT-Bench.
Selection
Data Size
MT-Bench
3K
5.92
5K
6.33
CI-D
6K
6.44
10K
6.34
16K
6.31
6K
5.76
Random
10K
6.27
153K(half)
6.23
306K(all)
6.21
Table Results for different SFT data sizes and sampling procedures. Cf-D represents the complexity-firs diverse sampling in Alg,
We then provide decoupled analyses of complexity and diversity to demonstrate how they influence alignment performance given the same SFT data size.
Complexity. To decouple and focus on the data complexity, we sample different data subsets of diverse averaged tag numbers. Different sampled subsets share the same sample size of 6k and have
8

---

Published as a conference paper at ICLR 2024
the same tag coverage of 100%, implying the largest data diversity. In the sampling procedure, all the data samples are first sorted by the tag numbers in descending order. Then for each data subset, start from the sample in the whole dataset with the largest tag number. The sample that can expand the tag set size of the current sampled data will be extracted and removed from the whole dataset. If the tag of the current sampled subset covers the whole tag set and the sample number is still less than 6k, we repeat the sampling procedure until the sample numbers reach 6k. This sampling procedure is similar to Alg. [ We leave the detail in Appx. SH
We sample 10 different data subsets, and the average tag numbers of the subsets range from 6.7 to 16.6. As shown Figure Baj the overall performance trend on MT-Bencl is increasing along with the growth of average tag numbers. This trend may not be significant on the fine- -g--gnrann level of average tag numbers where the number difference between subsets is small, Compared to the randomly sampled datasets, the average tag number of around 4.5, all the 10 data subsets can lead to superior fine-tuned model performance than the randomly sampled subset baseline. To summarize, on coarse-grained level of data complexity, the downstream performance is positively correlated to the average tag number, while on a fine-grained level, such a phenomenon becomes less evident. This may be partly because ChatGPT does not recall all the possible tags for each query, some tags are filtered out during the tag normalization procedure, resulting in a less accurate tag number.
Diversity. For diversity, sample different data subsets spanning various tag coverage rates garding the whole tag set. Different subsets share the same sample scale of 6k and the same average tag number, implying the same data complexity. The average tag number is set to 5.0. For data subset sampling, we first draw samples that can expand the tag set size of the current sampled data until the target tag coverage rate. Then, we traverse the remaining samples and extract samples that do not expand the tag coverage and can keep the current average tag number of the subset around 5.0. We leave the detailed sampling algorithm for diversity analysis Appx. $
We can observe in Figure 36 that as the tag coverage increases, the fine-tuned model can achieve higher MT-Bench scores. Randomly sampled data subsets of tag coverage 71.9% result in similar model performance with the sampled subset of tag coverage 70%. This demonstrates that the fine- tuned models may benefit from the more diverse datasets through the scope of tags. The trend not strictly linear, and there seems to be plateau ranging from 50% to 90% coverage. This could be due to the tags assigned may not share the sample importance for diversity.
5 INSTAGGER: LOCAL TAGGER BY DISTILLATION
INSTAG depends on advanced chatbots that are expensive on large-scale applications. As fine- grained tags benefit SFT data selection and other applications, we naturally propose INSTAGGER, which is equipped with the tagging ability of these high- seghhperformin chatbots by distillation of de- noised tagging results with significantly fewer budgets and higher efficiency. Distilling is effee- tive method to inject a smaller model with specialized abilities (Fu et al. (023)). We use our INSTAG results open-sourced SFT datasets to fine-tune 7B version LLaMA-2 model. The detailed im- plementation is described in Appx. $C We validate the model on our validation set. The tag-level Fl score based on exact match (EM) and semantic-based fuzzy match are 31.8% and 73,4 73.4%. As this is an unconstrained open generated tagging. EM is rigorous metric for annotating over six thousand tags. Therefore, we also calculate the fuzzy match by PhraseBERT, which considers a predicted tag is correct if it has over 0.8 cosine similarity in semantics with any gold tag.
6 CONCLUSION
In this paper, we introduced INSTAG, an open-set tagging method leveraging the instruction- following ability of ChatGPT for SFT data analysis. We apply INSTAG on open-source SFT datasets, showing diverse and complex data leads to better alignment performance. We designed a complexity-fire diverse sampling method to select 6K samples, and TAGLM fine-tuned on this selected dataset outperforms other open-source models aligned with considerably more data. More- over, further decoupled analyses revealed that model performance increases with fine-tuning on more diverse and complex SFT data, respectively. In summary, our proposed INSTAG provides a novel aspect for a deeper understanding of query distribution in the alignment of LLMs. It has robust potential to be extended to more applications beyond the data selection shown in this work, such as creating comprehensive evaluations and tag-based self-instruct.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham. Timothy Telleen-Lawien Tom Con- erly. Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield-Dodds. Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022b.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkir. " O'Reilly Media, Inc.", 2009.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% " chatgpt quality, March 2023. URL https:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. 14168, 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Iim. 2023. URL. https: //www wwtp://wwaabbbickcooo folly-firss-opee
cotts//////llg/0/////111// instruntiinnnnnoom tuned-
cowmercially caleissttopoo-ommrrriaaallllll
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:23305. 14233, 2023.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning, 2023.
Michael Hahsler, Matthew Piekenbrock, and Derek Doran. dbscan: Fast density-based clustering with r. Journal of Statistical Software, 91:1-30, 2019.
Jiawei Han, Jian Pei, and Yiwen Yin. Mining frequent patterns without candidate generation. ACM sigmod record, 112 2000.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:22/2.096 2022.
10

---

Published as a conference paper at ICLR 2024
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rut Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations-demne democratizing large language model alignment. arXiv preprint arXtv:2304.073227, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam. Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc. Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arna Dantuluri, Andrew Maguire. Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model align- ment, 2023.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. arXiv preprint arXiv:2012.126. 2020.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloingg 2023.
models. htttps://gittee wwt.c//tttttlla/aalaacaae..om
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov. James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, (092 URL https://tww wcwesce. org /ww//bs////bbe/////ee/aae 100 science. abq1158
2022. doi: 10. 2..116/science.abgg 158.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal. Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Baolin Peng, Chunyuan Li, Pengeheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint 2rXhv:2304.03277, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang- Preference ranking optimization for human alignment, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following, llama model. alpaca| 2023.
https://gi com/ wtwc
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom,

---

Published as a conference paper at ICLR 2024
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Guan Wang, Sijie Cheng, Qiying Yu, and Changling Liu. OpenChat: Advancing Open-source Lan- guage Models with Imperfect Data, 7 2023a. URI. h/tptt:/t///gthbbb http://tth....cooomeeeeeeeeeeeee cot/ com/immneoi/openccaat
Shufan Wang, Laure Thompson, and Mohit lyyer. Phrase-bert: Improved phrase embeddings from bert with an application to corpus exploration. arXiv preprint arXiv:2109 109006344 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprinl arXiv:22122 10560, 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. How far can camels go? exploring the state instruction tuning on open resources, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023c.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966. 2023d.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Jiazhan Jiang. Wizardlm: Empowering large language models follow complex instructions, 2023a.
Jiazhan Feng, Chongyang Tao, and Daxin
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.00 196, 2023b.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning (ICML), 2020.
Hongyi Yuan, Keming Lu, and Zheng Yuan. Exploring partial knowledge base inference in biomed- ical entity linking. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pp. 37-49, Toronto, Canada, July 2023a. Association for Computa- tional Linguisties. doi: 10.18653/v1/2033 binn:l...innn-1..33 1.3. URL URLLhttpp://acaaathoo.o orgg/2/3....// bionlp-1.3
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian Yun Nie and Ji-Rong Wen. A survey of large language models, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuming Mao, Xuezhe Ma, Avia Efrat, Ping Yu. Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
12

---

Published as a conference paper at ICLR 2024
solve
analysis
ingulry
st
ecsarance
retrieval
Inquiry
loop
natne matics
Fregiam
treattue
data
errive
operration
siaton
Nuterica
request
operation
calculation
information
declaration
Vanbbll
creation
Inguit
speling
handle
and
dev levelopment entt
Be eloomee
handling
de vellommenn
Kor
slatement
ua magggul
io.i3
Figure 4: The sunburst plot of all tags. We plot with the first two words of each tag and the size is proportional the frequency of the tag.
APPENDIX
A LIMITATIONS
Our conclusions mainly rely on MT-Bench and AlpacaEval for model evaluations, which may miss some influence caused by SFT data. Besides, we notice MT-Bench shows instabilities in terms of the randomness of GPT-4 judgments, so provide random ablations comprehensively show the statistical significance of our results, including reporting standard variance MT-Bench scores. Furthermore, our analysis of SFT datasets is mainly focused on English, so our claims may not be directly extended to multi-lingual scenarios.
possible to
B TAG REVIEW
We present a sunburst plot of all tags in Fig. 4 showing the most frequent tags is about information- related, data manipulations, and coding queries. We plot with the first two words of each tag and the size is proportional to the frequency of the tag. We only plot with tags that have frequencies larger than 2000 in our data pool.
C LOCAL TAGGER
We use the following template to concatenate queries to tag and tagging results:
You are helpful assistant. Please identify tags of user intentions in the following user query and explain each tag. Please respond in the JSON format ("tag": str, "explanation", str). Query: <query- to-tag> Assistant: <lagging-results
13

---

Published as a conference paper at ICLR 2024
You are a tagging system that provides useful tags instruction inten- tions to distinguish instructions for helpful Al assistant. Below is instruction:
[begin]
(instruction)
[end]
Please provide coarse-grained tags, such as "Spelling and Grammar Check" and "Cosplay", identify main intentions of above instruction. Your answer should be list including titles of tags and a brief explana- tion of each tag. Your response have strictly follow this JSON format: [("tag": str, "explanation": s. Please response in English.
Table 5: ChatGPT prompt template for annotating intention tags of given queries.
You experienced judge for intention tags of instructions. You will
be provided query and a list of tags describing intentions of the query as followed:
[query]: {query)
(tags)
Please provide feedback about whether all tags precisely describe an in- tention of the instruction. Please identify all incorrect tags and provide their indices in the JSON format as your response. The JSON format for your response a list of JSON dictionary and the JSON dictionary has only one key to identify the index of each incorrect tag: [f"idx": ). For example, if [tag 0] and [tag 2] are incorrect, you should response as id" 0}, ["idx", 2}1. If all tags are correct, please response an empty list [].
Table 6: GPT-4 prompt template for evaluating tagging precision.
We also include the explanation in the tagging results to make the fine-tuned model obtain better tagging performance. The overall sample size for fine-tuning 773,511, where we randomly sample 1,000 samples for validation. The model is fine-tuned with 512 batch size for one epoch since we empirically find that training for more than one epoch will lead over-fitting
D PROMPT TEMPLATES FOR CHATGPT
We preset prompt for ChatGPT for annotation Tab. 5) precision evaluation Tab. 6), and con-
sistency evaluation Tab. 7
E DATASETS
We apply INSTAG to 17 open-source SFT datasets for intention tagging:
ShareGP1P refers to the multi-turn chatting histories used by VICUNA (Chiang et al. 2023 ShareGPT includes numan-written queries and responses from ChatGPT and other chatbots.
OpenChat Wang et al. 2023a) subset ShareGPT containing only chat histories with GPT-
responses
a reproduced version from (https://shareegtt.com has been released. We instead http:///gwaa..ootttttatattaaaaaa imffittereed/tre///aiiiniie hwtph/Meeeeeeeeeeeattttaaeeeeee and follow Vicuna prepracess.
'Exact dataset of ShareGPT
We use the dataset with 8,000 GPT-4 responses denoting OpenChat v1.0 in https: htttps:/ http::///hugginffaee
co/ o0//ttastts/oeencc///eenaaaa sharegpt4 dataset
14

---

Published as a conference paper at ICLR 2024
You are an experienced judge consistency of intention tags for in- structions. You will be provided tag and a list of instructions labeled with this tag as followed:
[tag]: {tag}
(instructions]
Please provide feedback about whether the meaning of this tag is consis- tent among all instructions. Please identify all inconsistent instructions and provide their indices in the JSON format your response. The JSON format for your response is a list of JSON dictionary: [f"idx": int]]. For example, if the meaning of tags in [instruction OJ and [instruc- tion 2] are inconsistent, you should response as [["idx": 0}, ("idx": 2]1. If the meaning of tag is consistent in all instructions, please re- sponse empty list as [].
Table 7: GPT-4 prompt template for evaluating tagging consistency.
UltraChat (Ding et al. 2023 is systematically designed, diverse, informative, large-scale dataset of multi-turn instructional conversations without involving human queries 3)
Alpaca (Taori al., 2023) is a dataset generated by the modified SELF-INSTRUCT method (Wang et al. (022), containing 52,000 instruction- folllooinnn demonstrations generated from OpenAl's
lext-davinci-00, model
WizardIM (Xu et al. 2023a) is instruction dataset built with the INSRLUN method. EVOL-INSTRUCT utilizes ChatGPT to augment the complexity of the same queries in Alpaca and ShareGPT. We denote these two subsets as WizardLMM(Alpcca) and WizardLM(ShareGPT for clarification
FLAN (Wei et al.) is a series of data from NLP tasks formatted in instruction tuning. The queries in FLAN are generated by templates for each NLP task.
Dolly (Conover et al. 2023) contains 15,000 high-quality luman-generated prompt and response
pairs for instruction tuning of LLMs.
OAssist (Köpf al. 2023) is crowdsourced human-annan -nnotated dataset about multi- multi-lingual con-
versations.
Unnatural (Honovich et al. 2022 contains queries generated by prompting DAVINCI-002
Lima (Zhou et al. 2023) contains only 1,000 carefully human-curated prompts and responses.
Math Collections: We involve a set of math datasets including GSMSK (Cobbe et al. (021)) and
MATH (Hendrycks et al. 2021) to prompt INSTAG generating fine-grained mathemetical tags.
Code Collections: We also involve a set of code datasets including DMCC (Li et al. 2022).
MBPP (Austin al. (002I). and DrRepair (Yasunaga Liang 2020) for the same purpose as introducing mathematical datasets.
F BASELINE LLMS
We give introductions to the LLM baselines for human alignment.
2023) the first open-sourced LLM aligned with human preference. Alpaca is fine-tuned on SFT data of 52K samples generated from text-davince-003 using Self-Instruct
Alpaca (Taori et al.
(Wang et al. 2023c).
WizardLM (Xu al. 2023a is fine-tuned on the SFT data enhanced with novel technique named Evol-Instruct It complexifies the Alpaca SFT data using ChatGPT and achieves better
alignment performance.
Vieuna (Chiang al. 2023) is aligned LLM fine-tuned collected user chatting logs of proprietary high-performing chatbots on ShareGPT.
ace. http///ttttttttttigggggggg/teeee htter/////ggggggoooooooo//eetttt
/https://hugging: ace.
We collect the Alpaca dataset along with Dolly, OAssist, and Unnatural from the sharing of Wang al.
2023b https:://gtthhn
com/al comm/alleaii/pen. innttruuuttissttttt
http://wwwotto////oooooooooooooe http://www.........t....//taaaaa evol instruct V2 196k
We use the V2 version of WizardLM in https:
15

---

Published as a conference paper at ICLR 2024
MT-Bench Scores
baize-v2-122
coding
wizardim- 13b
openchat v1
vicuna-13b-v1.
TagLM-v1
math
humanities
6
reasoning
writing
extraction
stem
roleplay
Figure 5: Radar plot showing detailed scores of TAGLM-13b-v1.0 and major baselines on eight subtasks of MT-Bench. Detailed numbers can be viewed in Tab.
OpenChat (Wang et 2023a) fine-tuned on subset of ShareGPT with only the chatting logs with GPT-4.
Baize (Xu al. 2023b 100K dialogues generated by self-chatting of ChatGPT. It also includes Alpaca's data for SFT.
LLaMA-2 Chat (Touvron et al. 2023b) differs from the above-mentions LLMs in (1) being based on per-trained LLaMA-2 instead of LLaMA (Touvron et al. 2023a); (2) being aligned with human preference by both SFT and RLHE
G DETAILED MT-BENCH SCORES IN CATEGORIES
As shown in Fig. 5 we present detailed number results in Tab. 8) TAGLM-13b-vi outperforms all other baselines on stem and extraction, and achieves comparable performances on humanities with Vicuna, suggesting these tasks may rely on few data for alignment. TAGLM-13b-V ranks the second on math, coding, and writing, but falls short on roleplay and reasoning. These detailed results show that some tasks may require diverse but only few alignment data, while tasks about reasoning and writing may continually benefit from large-scale data.
H SAMPLING ALGORITHM FOR DECOUPLED ANALYSIS
We calculate complexity and diversity with bas-based metries described in $3.4 We first sort all samples by the query complexity (the query tag number) and then pick distinct queries according to tags achieve high sample diversity (tag coverage). The selection criterion at each time is to select query with large tag numbers and can increase the tag set of the selected subset data. The algorithm is detailed in Alg.
We present our sampling algorithm for decoupled analysis of complexity and diversity in Alg. and Alg. respectively.
16

---

Published as a conference paper at ICLR 2024
MT-Bench Scores
Average
Model
Data
code extraction humanitics math reason roleplay stem writing
all w/o C&M
gpt-4
8.55
9.38
9.95
6.8
9.0
8.9
9.7
9.65
8,99
9.43
gpt-3.5
6.9
8.85
9.55
6.3
5.65
8.4
8.7
9.2
7.94
8.39
claude-vl
6,25
8.8
9.7
4.8
5,95
8.5
9.7
9.5
7.9
8.69
Hoama-2-13b-aee
3.0
6.92
9.75
3.45
5.1
7.5
8.62
8.85
6..5
7.79
(1)
6K
3.75
6.5
9.55
2.1
5.3
7.95
8.5
8.75
6,55
7.76
(2)
6K
3.7
6.2
9.52
2.15
5.35
8.1
8.4
8.95
6.55
7.75
TAGL.M-13h-v2.0 (3)
6K
3.4
7.35
9.6
2.15
5.9
7.45
8.28
8.0
6.52
7.76
TAGLM-v1.0-13b (1)
6K
3.8
6.45
9.55
3.0
4.9
6.9
8.75
8.55
6,,99
7.52
TAGLM-v1.0-13b (2)
6K
3.45
2.95
4.95
7.15
8.65
8.5
6,46
7.54
TAGL.-vv1.0-330 (3)
6K
3.4
6.45
9.45
2.85
5.05
7.05
8.43
8.4
6.38
7.47
vicuna-13b-ve 13b-v1.3
125K
3.25
5.55
9.45
2.6
5.85
7.18
7.98
9.25
6.39
7.54
ficuna-13b-vl.1
70K
2.95
6.4
9,45
2.9
4.65
7.5
8.55
8.05
6.31
7.43
wizardlm-13b
70K
4.0
4.9
9.7
3.75
5.25
7.4
7.7
8.12
6.35
7.18
baize-v2-13b
56K
3.0
9.02
1.8
7.72
7.65
5.75
6.87
nous-hermes-133
300K
2.45
5.05
9.0
2.65
3.8
6.38
7.02
7.75
5.51
6.5
ept4all-13b-snoozy
900K
3.0
4.8
8.85
1.2
4.2
7.0
6.9
7.35
5,41
6.52
koala- 13b
472K
2.9
4.15
8.45
1.9
4.0
6.85
7.2
7.35
5.35
6.33
openchat- 13b-vl
8K
2.35
2.75
7.55
7.7
7.05
5.22
6.24
alpaca-13h
52K
2.35
4.15
1.05
3.5
5.45
5.2
6.7
4.53
5.48
Table 8: Main results of INSTAG. We present MT-Bench scores of both proprietary and open-source baselines in similar scales. We also provide average scores overall categories and categories without code and math (w/o C&M). Dashes in the data column denote unknown data scales. Parentheses mark the three different rounds GPT-4 judgments.
Algorithm 1: Complexity first Diverse Sampling
Data: The Whole Pooled Dataset D. Sub-Dataset Size N
Result: The Sampled Sub-Dataset D.
Initialize Empty D.;
Sorting Queries in D by tag number in descending:
while D. < N do
Tag Set T:B +- 0:
foreach Query q D do
if Query Tags T, IT, UT,
>
IT:21 then
D. - D2 {q):
T B + T TTi
D D {at:
10
[D. equals to then
Break:
12 return D,
I COUNTERFACTUAL EVALUATION
To test how well annotators can evaluate tag quality, we created counterfactual cases for two tasks. In the tag precision task, we substituted some tags with similar ones in terms of semantics. In the tag consistency task, we used inconsistent instructions to replace the original instructions. Both humans and GPT-4 able recognize most of the counterfactual cases. And humans better at tag precision, while GPT-4 is better at tag consistency. This analysis shows that annotators have low false positive rates and proof confidence of their judgments in the original tagging results.
17

---

Published as a conference paper at ICLR 2024
Algorithm 2: Data Sampling for Complexity Analysis
Data: The Whole Pooled Dataset D
Suusett Size N Result: The Sampled
of Different Complexity
= {Dili ,,...,m}
Sorting Queries in D by tag number in descending;
Initialize D = list();
foreach i in (1....... n do
Initialize Empty D
while Di N do
Tag Set T. +- 0;
foreach Query q E D do
if Query Tags Tg [To
UT/ > |T then
D: D U (g): T T
+ D \ {q):
if [ = N then
+ D appends D
Break;
16 return I)
Algorithm 3: Data Sampling for Diversity Analysis
Data: The Whole Pooled Dataset D, Preset Coverage Rate = {r' ........... n
Subset Size N Result: The Sampled Sub-Datase of Different Diversity
{D] i 1,...,n
Initialize D list();
foreach i in {1..... n do
Initialize Empty D 0:
Tag Set Ta + 0;
foreach Query 9 E D do
if Query Tags T TTa U Tg
|Tal then
Drj < Di U (q);
Ta - Ta Tgi
- D {g}:
10
if Tal/TT > then
12
Break;
13
while D < N do foreach Query q E D do
14
if Query Tags Tg Ta UT
[Tal then
D D D - D {g):
U {q):
18
2 = then
Break;
D+ D appends Dir;;
29
21 return
J CASE FROM INSTAG
We present a sample case in Tab. 10 to illustrate the behavior of INSTAG.
18

---

Published as a conference paper at ICLR 2024
GPT-4 Annotation
Human Annotation (1%)
Metric
Original
Countcrfactual
Original Counterfactual
Tag Precision
96.1
100
Tag Consistency
86.6
7.8
100
14.3
Table 9: Evaluation for the tagging quality of INSTAG. We design metrics, tagging precision and consistency, for evaluating INSTAG. We employ GPT-4 to label 4,000 tagging results. And we also employ three human annotators annotate 1% cases and report their majority voting. We also create counterfactual cases to probe the judgment abilities of different annotators.
Sample Case
Utterance
Human: LLM:
List 3 breakfast foods that healthy.
1. 1.Oatmeal it's high fiber, protein, and antioxidants, and can lower cholesterol levels.
2. Grere yogurt it's packed with protein, calcium, and probiotics, which support gut health.
3.Eggs they 'r rich in protein, vitamins D and B12, and minerals like selenium, and can help you feel full longer.
INSTAG Results
Tag food recommendation quantity
Explanation
The instruction asks for a list of breakfast foods that are healthy. The instruction specifies that the list should include three breakfast foods.
health
The instruction emphasizes the need for the breakfast foods to be healthy.
Table 10: One case from SFT data of WizardLM tagged by INSTAG. INSTAG accurately captures the semantics (food, health) and intentions (recommendation quantity) in the human query.
19


Reviews:
[{"review_id": "JWoXQgCtoG", "reviewer": "Reviewer_jmr3", "summary": "This paper presents InsTag, a method to quantify and select/prune sft dataset from a large dataset pool. The authors propose to utilize ChatGPT to tag the intent/topic of the data samples, and use the number of tags to represent complexity and tag coverage for diversity. These metrics are simple from both concept and practice perspectives. The authors give interesting analysis of existing datasets by adopting these two metrics. In the experiments, InsTag automatically selects 6K sft examples and the resulting model achieves comparable performance to WizardLM and Vicuna that are trained on 10x more data examples.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed data measurement and data selection methods are simple, both concept-wise and practice-wise. \n2. The experimental results are strong – TagLM is the only model that achieves such high performance on MT-bench with <10K data examples as far as I know.  \n3. The proposed approach can be utilized to measure existing datasets, as shown in Figure 2 which are interesting.  \n4. Table 4 indicates that more sft data does not necessarily give better performance and InsTag is able to select the effective ones.", "weaknesses": "1. Evaluation is a bit weak, MT-Bench scores are the only metric used across entire paper – other datasets such as AlpacaEval and human evaluation could further strengthen the claims in the paper.  \n2. While I appreciate that the authors distill a local tagger in Section 5, it is unknown how much SFT performance would be sacrificed by using the tags from this local tagger. The strong results on Table 3 is from ChatGPT tagger if I understand correctly, and it may be too expensive to use ChatGPT in practice when we have a large data pool to measure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15E4wpMZSn", "reviewer": "Reviewer_32G5", "summary": "This paper explores the diversity and complexity of supervised data in the alignment process of large language models. It proposes InsTag, an open-set instruction tagging method, which can conveniently evaluate the diversity and complexity of human instructions. Based on this, the authors design a method of selecting human instructions, which can make the model achieve better performance with less supervised training data.", "questions": "The description of InsTagger in Section 5 seems too brief. Can you provide more detailed explanations, such as why it was designed and further influences.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes an automatic method to evaluate the diversity and complexity of human instructions. \n2. Based on the proposed InsTag, this paper further presents a method of selecting human instructions, which can potentially reduce the cost of the alignment phase of large language models.", "weaknesses": "1. Selecting more diverse and complex samples is an existing idea, and this article is more similar to the implementation and application of this idea.\n2. The definition of complexity seems a little strange. Is there any explanation from other papers? If not, I hope the authors can give a more detailed explanation for it.\n3. The author is recommended to verify the performance of the proposed method on more benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wxgOxtE7Lk", "reviewer": "Reviewer_jYjU", "summary": "This study illustrates how GPT-4's open-ended topic tagging capability can be instrumental in quantitatively assessing the diversity and complexity of various Instruction Fine-Tuning (IFT) datasets. \nSpecifically, the complexity of an IFT dataset is gauged by the average number of tags per data point, while its diversity is measured by the proportion of unique tags in the dataset relative to the total number of tags recognized by GPT-4. \nUtilizing these novel quantitative metrics, the researchers compared distinct IFT datasets to discern their unique features. \nThe research further suggests that employing these metrics to filter data can enhance the effectiveness of instruction fine-tuning.\nExperimental results in the paper demonstrate the importance of enhancing the diversity and complexity of data points when tuning language models to align with human instructions.", "questions": "- The proposed method appears to be sufficiently versatile to extend beyond instruction fine-tuning applications. Could you provide additional examples where the efficacy of this method could be demonstrated?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is straightforward, intuitive, and simple to implement.\n- The study demonstrates that metrics driven by the statistical analysis of automatically generated tags—specifically complexity and diversity—can effectively probe the characteristics of IFT datasets.\n- This work provides further empirical evidence that the quality, rather than the quantity, of IFT datasets is crucial for aligning language models successfully.", "weaknesses": "- The primary mechanism of the proposed method is dependent on the automated, open-ended tagging capability of GPT-4; therefore, there is a risk that the analysis in this paper might be influenced by any inherent biases present within GPT-4.\n- Further examination of data instances categorized by the proposed metrics would be advantageous. Specifically, exploring the semantic or syntactic traits defining IFT datasets identified as diverse and complex by these metrics would be informative.\n- The presented work is largely empirical, which may raise concerns within the community regarding the foundational grounding of its results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxWsaXyKKX", "reviewer": "Reviewer_NPyy", "summary": "The authors propose a method for automatically tagging instruction data with labels based on query intent, using ChatGPT to generate queries and applying various rules to normalize and aggregate related tags together. The authors analyse existing datasets using the tagging method, finding that a higher average number of tags, and higher diversity correlate with good performance, matching prior work examining instruction datasets. They then use the tags as a data selection method (maximising complexity and diversity based on the tags), and find they can train a strong model with as few as 6,000 examples. Finally, they ablate their data selection method and find that increasing diversity and complexity improves performance, while increasing dataset size only improves performance up to a point (around 6,000 examples).", "questions": "- Does the average number of tags per instance correlate well with human intuitions of complexity? \n- How well does your distilled tagger perform if used as a tagger for data selection? Did you test with different models?\n- How does TagLM perform over other benchmarks (e.g. MMLU, HumanEval)? Are there any particular capabilities or skills it seems to underperform in compared to baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The tagging method is interesting, and well-validated both as an analysis and as a selection method. The use of ChatGPT as a tagger, and the rules used to simplify the generated tags, are novel and effective.\n- The data selection method appears well-validated, and achieving strong results with only 6,000 examples is impressive. The method outperforms a random selection baseline and other popular open models such as Vicuna and WizardLM.\n- The analyses of performance against complexity, diversity, and dataset size are useful, providing useful guidelines for future researchers in data collection and selection.", "weaknesses": "- The method’s reliance on a strong tagging model (in this case, ChatGPT) is not explored, despite a very brief mention of training a distilled tagger model at the end of the work. It would be interesting and useful to see how well this method works with openly available models, or over a variety of different quality models (e.g. how does using GPT-4 as a tagger compare? How about Vicuna? etc.).\n- The tag-based analysis is somewhat restrictive, as discussed in section 4.3. It would be interesting to take the semantics of the tags themselves into account somehow, since (a) some tags may be closer to each other and so overlap in terms of diversity, and (b) certain tags may express queries that are naturally more complex than others (e.g. ‘solve’ vs ‘inquiry’ tags).\n- MT-bench evaluation involves a relatively small number of questions (80), which may be easier to cover with 6,000 examples. I wonder how well the TagLM model would perform with more questions (e.g. the alpacaEval setting, which has 800 examples), or on more traditional benchmarks such as MMLU, Big Bench, etc. In general, it would be interesting to see if the selection method is biased towards certain capabilities compared to others.\n\nOverall, I think this work is strong, and my main qualm is that the evaluation of the data selection method is somewhat weak, only exploring one evaluation setting. However, the method proposed is interesting and novel, and produces useful insights for data selection in instruction tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)