WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training
Authors: Benjamin Feuer
Chinmay Hegde
Venue: ICML 2025 poster (2025)
Keywords: deep learning
machine learning
foundation models
llms
large language models
datasets
sft
post-training
Primary area: deep_learning->foundation_models
TL;DR: We introduce WildChat-50m, the largest public chat dataset to date, and use it to create ReWild, an SFT mix which outperforms even strong baselines, such as the recent Tulu 3 SFT mixture from Allen AI.

Abstract: Language model (LLM) post-training can refine behaviors and unlock new skills, but the open science supporting these post-training techniques is still in its infancy. One limiting factor has been the difficulty of conducting large-scale comparative analyses of synthetic data generating models and LLM judges. To close this gap, we introduce WildChat-50M, the largest public chat dataset to date. We extend the existing WildChat dataset to include responses not only from GPT, but from over 50 different open-weight models, ranging in size from 0.5B to 104B parameters. We conduct an extensive comparative analysis and demonstrate the potential of this dataset by creating Re-Wild, our own public SFT mix, which outperforms the recent Tulu-3 SFT mixture from Allen AI with only 40% as many samples.

WILDCHAT-50M:
A Deep Dive Into the Role of Synthetic Data in Post- cest-Trainng
Benjamin Feuer'
Chinmay Hegde
Abstract
erful tool enabling reasoning models (DeepSeck-Al et al., 2025). Unfortunately, the open source ecosystem supporting post-training in general, and data curation in particular, is in its infancy, with industry labs' capabilities far outstripping that of most academic labs (Weber et al., 2024; Feuer et al., 2024b; Ivison al., 2023).
Language model (LLM) post-training, from DPO to distillation, can refine behaviors and unlock new skills, but the open science supporting these post-training techniques is still in its infancy. One limiting factor has been the difficulty of conduct- ing large-scale comparative analyses of synthetic data generating models and LLM judges. To close this gap, we introduce WILDCHAT-5OM. the largest public chat dataset to date. We ex- tend the existing WildChat dataset to include re- sponses not only from GPT, but from over 50 dif- ferent open- weight models, ranging in size from 0.5B to 104B parameters. We conduct exten- sive comparative analysis and demonstrate the potential of this dataset by creating RF- RI.-WILD, our own public SFT mix, which outperforms the recent Tulu-3 SFT mixture from Allen AI with only 40% as many samples. Our dataset, samples and code are available https: / //g tiuu.b. com./penfever/wwlle. wwwpheettwwwcaaeaaeeeeeeaaaaeeel 50m.
A stark challenge for smaller labs, especially in academia, has been the difficulty of acquiring publicly available syn- thetic datasets at large scale. This has posed barriers for researchers who are interested in conducting careful com- parative analyses of the synthetic data quality (SDQ) of data generating models (DGMs), as measured by standard academic ground-truth and LLM-as-a-judde benchmarks.
To close this gap and better understand the downstream effects of DGM choice on synthetic data quality, we develop WILDCHAT-5OM, which is the largest and most diverse publicly available dataset of chat transcripts to date. We also show that WILDCHAT- 50M is a particularly effective source for post-training data for L.Mss. Our core contributions in this work are as follows:
1. We introduce WILDCHAT-50M.. the largest publicly available dataset of chat transcripts. Our dataset consists a vast corpus of synthetically generated chat transcripts using 50 different open-weight models. ranging in size from 0.5B to 104B parameters, each participating in over 1M multi-turn conversations. Each model participates in 2 3 turns per conversation on average, resulting in approximately dataset comprising over 125 million chat transcripts in aggregate.
1. Introduction
Large language model (LLM) post-training encompasses a broad suite of algorithmic techniques, and is an active area of current research. Improvements in L.LM post-training have led to many breakthrough accomplishments. ranging from recent developments in test-time scaling from OpenAl and Deepseek (OpenAl et al., 2024; DeepSeek-Ar Al et al., 2025) to new algorithms for efficiently aligning LLMS to human preferences (Rafailov et al., 2024). All of them rely on synthetic data during post-training sometimes in the form of judgments through LLM judges or pairwise comparative outputs. More recently, simple SFT on large model outputs (also called distillation) has proven pow-
2. We conduct thorough comparative analysis on the run- time and VRAM efficiency of these models, as well as analyze the distinctive qualitics of different model out- puts. Our analysis may inform researchers on how to scale up post-training data even further in the future. 3. We demonstrate the power of our dataset by using it as the basis of RE- RE-WILD, novel data mix for supervised fine-tuning (SFT) of LLMs. When we fine-tune Llama-
Department of Computer Science and Engineering, New York University, New York City, USA. Correspondence to: Benjamin Feuer <bf996@nyu.cdu.>
3.1 8B Base on RE- WILD, we show that our models outperform the SET mix proposed in Tulu-3 (Lambert et al., 2024), along with several other existing, strong SFT baselines on range of post-training benchmarks.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

WildChat-SOM
The rest of this paper is organized as follows. Sec- tion 2.1 describes high-level details of how the WILDCHAT- 50M dataset was constructed. Section 2.2 provides deep dive into the generation process of this dataset, along with technical aspects such as model response similarity and throughput efficiency. Section 3 lists and analyzes the re- sults of our SFT experiments. Section overviews related work, and Section gives a concluding discussion.
work (Jin et al., 2024).
2.2. Dataset Analysis
WILDCHAT-50M collects data from 19 unique pre-trained models (each of which is post-trained) and 35 post-trained model variants (with non-unique pre-trained models). This yields a total of 54 DGMs represented. With the exception of the responses in the original WildChat dataset, which are sourced from various GPT checkpoints, all responses in WILDCHAT-50M are derived from models sourced from HuggingFace; the release dates range from July 2023 to November 2024, and the parameter counts range from 0.5B to 104B. For comprehensive list of all the L.MSs we used in the study, please refer to Sec. C. We attempted to select a diverse set of models; our main limiting factor was com- patibility with our hardware setup, and with VLLM as an inference engine. The resulting public artifact is more than 50 times larger than the next largest public chat datasets of which we are aware, WildChat- IM and LMSys-Chat Chat-lM.
2. The WILDCHAT-50N Dataset
2.1. Data Collection
We begin with a brief description of the data collection pro- cess for WILDCHAT-50M detailing our source of prompts and the technical details of how responses were collected.
Although, to our knowledge, there are no large-scale diverse chat transcripts dataset of synthetically generated LLM re- sponses. there are at least two recent large chat datasets available on which one could potentially base such dataset: WildChat-IM IM from AllenAI, and LMSSs-Chat-- 1M from LMSys (Zhao et al., 2024b; Zheng et al., 2024). Although both have their strengths, we chose to focus on the former because of its rich variety of use-cases (including those which are potentially toxic), its diverse regional and tempo- ral dimensions, and its relatively low levels of contamination for commonly used test sets (Zhao et al., 2024b; Lambert et al., 2024).
Naming Conventions. In order make this paper more readable, we will employ certain naming conventions for the models and datasets generated using DGMs described in this paper. The aforementioned conventions are enumerated below.
We will sometimes utilize abbreviations for some par- ticularly common model names: Qwen2.5-72B-Insteee
= Q72, Llama-3.1 1--1-88-IInsterr == L8I,, Llama-
Technical details. Our data collection process was con- ducted over a period of approximately two months on a 12x8 H100 shared research cluster. We estimate the total GPU costs of our data collection at 10,000 H100-hours. The homogeneity of the nodes in our study and codebase allow us to make controlled comparisons on important consider-
== Q7, SachereCoommnde = 1.70, Qwen2-7B-Instnne
Plus-104B == CRP, 111-JJmma-Mini-....5
:= JMB.
Our model names will follow the following general con-
vention: {SFT target : DGM}.
Sometimes we will not specify the SFT target model name; in that case, will always be Llama-3.1-8B-
ations, such as the VRAM efficiency and runtime of each model.
= -L8B.
Several times, we just report benchmarks for model as is and not do any model post-training; in this case, the naming convention is just { Model name }: None.
All responses and judgments are generated using VLLM (Kwon al., 2023). a highly performant and stable frame- work for L.LM inference. Models are distributed across up to 8 GPUs; we do not conduct infererence using more than
Most of our experiments were conducted on SFT models trained on 250,000 (250k) conversations. If we used a quantity other than 250k, we note in the model name.
one node for any model. We first minimize the number of GPUs required per model, and then heuristically maximize the size of the context window given that number of GPUs and the capacity of the model, resulting in wide range of context windows, depending on the model architecture (2048 tokens to 20,000 tokens).
Analysis of Throughput Efficiency. Our first comparison describes the relative efficiency of inference across the mod- els in our study. We consider two measures of throughput efficiency: average combined input and output tokens per second (Tok/s), and average time elapsed in seconds per 1000 conversations processed (Time). We compute our averages over a random subset of 5000 conversations.
The largest models in our data collection process were queried using FP8 quantization, with checkpoints provided by Neural Magic (Kurtic et al., 2023). All other models were run in bfloat16, using their native checkpoints. We do not further ablate the effect of this quantization on out- put quality, as this has been studied and reported in prior
The slowest model in our study is 10een2.5-72B-nnstruue with context window length of 20,000, averaging 3,163 Tok/s, and the fastest is Llama-2-7B-Chat with a con- text window of 2, 2,048, averaging 37,357 Tok/s, more than
2

---

WildChat-SOM
10 times faster. Input is significantly faster than output;
Source
Num. Convs
the mean ratio over all unique pretrained models is 4.68 to 1, with large standard deviation of 3.3. Both Time (o = 0.90, p 0.73) and Tok/s (a = -0.41, p = -0.80) are strongly correlated with a simple proxy for model effi ciency, the product of context window length and number of parameters.
WildChat-Q73
246,750
MMLU Auxiliary Train
99,800
Tulu 3 Persona Hub Algebra
20,000
Table 1. Data biending in RE-WILD. Our data blend is sim- pler than Talu 3. consisting of just three sources, and is around 40% the size of the Tulu 3 SFT blend. The datascts were chosen heuristically to emphasize complementary skillsets (math, world- knowledge, and chat/instruction following). MMLU Auxiliary Train data is from Hendrycks et al. (2021). Tulu 3 Persona Hub Algebra is from Lambert et al. (2024).
Analysis of Response Similarity. To the best of our knowl- edge, the degree of similarity between diverse human re- spondents to LLM chat prompts has not been rigorously quantified at scale. However, this problem has been stud- ied in the domain of abstractive summarization, where it can be assumed that the similarity would be considerably higher (Maynez et al., 2020; Iskender et al., 2021; Lin & Hovy, 2002; van Halteren & Teufel, 2003; Jing et al., 1998). For summarization tasks, there is no "one truth", evidenced by low agreement between humans in producing gold standard summaries by sentence selection, low overlap mea- sures between humans when gold standard summaries are created by reformulation in the summarizers' own words, and assigning information overlap between them.
quality DGM that generates the responses and datasets de- signed to boost performance on world knowledge bench- marks. Later in this section, we describe the empirical process by which we determined which DGMs had high SDQ, and how.
The specific composition of our mix can be found in Tab. 1; The datasets in this composition were chosen heuristi- cally to emphasize complementary skillsets (math, world- knowledge, and chat/instruction following).
It would be natural to assume, therefore, that LLMs that do not entirely share either pre-training post-training data would likewise produce substantively different responses to prompts. However, our results show that this does not appear to be the case; LLM responses are unusually similar to one another. See Sec. A for a deeper analysis of this result, as well as every score (with associated standard deviation) for every model.
Training, We conduct our SFT experiments using a modi- fied version of the Axolotl framework (Lian, 2025). We use the Adam optimizer (Loshchilov & Hutter, 2017) with learning rate of 2e-5, single epoch, and a cosine learning rate scheduler, with eight steps of gradient accumulation, in 6 precision. We also utilize several techniques to opti- mize training speed, such as gradient checkpointing, flash attention, and in some cases, FSDP (full shard, autowrap). The base model trained is always Ilama 3.1 8B (for and haselines). All artifacts are available on our Hub repo Each of our SFT runs utilizes one 4xH100 node. The av- erage time to fine-tune a model for 250,000 conversations takes approximately 5.5 hours.
3. SFT Experiments
We now show that WILDCHAT-50M can be leveraged by researchers as a very valuable dataset for studying data curation strategies for LLMs post-training. Our core experi- ments focus on the SFT stage of post-training (also referred to as instruction tuning). While other forms of post-training (such tuning to human preferences) are also interesting, we leave their thorough analysis to future work.
Evaluation. Benchmarking LLM alignment is a challeng- ing task, because of the open-ended nature of the objective and the large number of potential confounds (Lambert et al., 2024), well as the fact that evaluation hyperparameters are not generally standardized across reported results. To deal with the latter issue, we employ Evalchemy, a recently introduced evaluation framework that is standardized, pop- ular, reliable, and validated by reproduction reports for all benchmarks (Guha et al., 2024). Evalchemy itself utilizes the LM Evaluation Harness from Eleuther AI (Gao et al.. 2024).
Following recent work such as Lambert et al. (2024), we focus on curating an SFT data mixture using a human-in in---in-tttt- the- loop process, in contrast with an automated curation process such as that of Xu et al. (2023a). Unlike both of those works, we do not curate new prompts; only new responses them.
Our dataset, samples and code are available at https: / //github. www/poonfeeeeecccccccc http//wwlddhhtteee
3.1. RE-WIID: A new data mixture for SFT
In order to make comparisons with past and future work easier, we select benchmarks which are popular and promi- nent in the recent research literature. We break down the concept of alignment into subcategories such as generalist chat capability, world knowledge, and instruction follow-
Following the recent work of Lambert et al. (2024), we de- sign our SFT data mixture, that we call RE-WILD, using a combination of WildChat data with a particularly high
3

---

WildChat-S0M
Comparison
Ninmen
Ultrahhal [Aug 0.206)
0.8
Phara Dual
Re-Wild (hvg: 0.378)
Tulu (Aug 0.335)
0.5
0.4
0.3
MMIU
MUSI
96T1
Figure 1. RE-WILD outperforms strong baselines, average, nine benchmarks. In particular, it exhibits strong performance on generalist chat and instruction following benchmarks. MT Bench scores here are divided by 10, so that the scale is similar to our other evaluations. For the exact numeric for all models, please refer GitHub repository. Figure best viewed in color.
Model
MTBench
AlpacaFval
BBH
GPQA
MATH
MUSR
IFEyal
MMLU Pro
MixEval
Q72
6,86
41.00
0.48 + 0.01
0.01
0.29
0.06 + 0.00
0.40 + 0.04
0.37 0.04
0.30 + 0.01
0.65 0.01
LSI
6.26
21.12
0.06
0.04
0.04 0.00
0.37=0.03
0.38 + 0.04
0.33 0.01
0,65 0.01
L70
6.23
24.91
0.00
0.04
0.04 0.00
0.00
0.34 0.04
0.31 0..01
0.65 0.01
Q7
6.03
17.26
0.01
0.30 0.04
0.03 103
0.42 0.04
0.29 0.04
0.00
0.61 0.02
CRP
6.05
13.44
0.01
0.04
0.00
0.39 0.044
0.28 0.04
0.32
0.60 0.02
JMB
6.05
25.14
0.07
0.28 0.04
0.00
0.38 1.38.0.00 0.04
0.26 0.04
0.29
0.57 0.02
Table The choice of data generating model has strong and unpredictahle effects on downstream benchmark performance. We compare the performance of six different DGMs from four different model families, ranging in size from 0.5B to 104B parameters, each fine-tuned 250k samples from DGM. We find large degree variance benchmark performance, no one model dominating.

---

WildChat-SOM
ing. Following recent work, select mix of ground-truth benchmarks and LLMM-jude benchmarks, in order bal- ance out the potential confounds inherent to each evalua- tion method (Feuer et al., 2024a; White et al., 2024). For generalist chat capabilities, we use MixEval, A ApaccEEva22, and MTBench with GPT-4o-mini as the judge LLM (Ni et al., 2024; Dubois et al., 2024; Zheng et al., 2023). some cases, we also report the average score over all of our benchmarks. For AlpacaEval2, we report length-controlled win rate. For instruction following and world knowledge, we utilize the recent HuggingFace OpenI..M Leaderboard 2 (Fourrier et al., 2024). For IFEval, we report prompt-level strict accuracy because is more challenging and therefore better exhibits separation between models; however, we also include instance-level loose accuracy in our artifacts, where we observe generally similar trends. Finally, in some figures and tables, report the average performance for all benchmarks (MixEval, AlpacaEval2-L.C, MTBench / 10, OpenLLM LB 2) as Avg. Where possible, we include 95% confidence intervals using the normal approximation method.
Quantity
100k
250k
500k
0.36
Avg
0.33
GPT
L70
LBI
Q72
Figure 2. Data scaling improves SFT performance. The effect is, however, somewhat dependent SDQ for DGMs such GPT 3.5, the benefits taper off relatively quickly, but for the other three DGMs consider, they continue increase. Avg is the average performance over (MixEval, A/pacaFval22-.C, MTBench 10, OpenLLM LB 2).
Baselines. Following (Lambert al., 2024), we utilize strong baseline checkpoints also utilized in that paper; Tulu 3 SFT, Magpie Align SFT from Xu et al. (2024), and Ultra- chat from (Cui et al., 2023).
How much does the choice of data generating model im- pact downstream performance? Can we be certain that it not the prompts, or perhaps some quirk of training procedure, that have led to improved SFT performance us- ing
3.2. Key Findings
To evaluate this concern, we compare the performance of six unique pretrained models from four dis- tinct model families, including Qwen-2.5 22wen-2.5-7BB 2.5-72B-Instruch from Alibaba, Llama-3.3- 3.3-7-3 3.3-700B-Instrut from Meta, Command-E R- Plus from Cohere, and Jamba- Mini from A121 (Qwen Team, 2024; Dubey al., 2024; Gomez; Lieber al., 2024). The models range from 7B to 104B active parameters. The choice of DGM has large effect, even when controlling for potential confounds such as the number of parameters for the model and the size of the context window. Fur- thermore, we find that no model dominates the benchmark, and that parameter count not perfect indicator of data quality. On three of the nine benchmarks consider, the best performing model has fewer than 10B parameters. See Tab. 2.
RE- BE-WILD is strong SFT data mix. Our first result is that RE- WILD constitutes particularly attactive data mixture for SFT. In the spider chart shown in Fig. 1, we show that RE- WILD outperforms several strong SFT baselines on aggregate; in particular, excels on generalist chat tasks as well as instruction following tasks. Because prior work has shown that LLM judges can introduce implicit biases into their judgments, include a mix of ground truth and LLM judged benchmarks (Feuer et al., 2024a). We find that RE- WILD performs well on both measures, indicating robust support for the claim that RF.WWIL WIID provides a superior model for generalist chat and instruction following.
How much data from WILDCHAT-50M should be used? Scaling up dataset size is generally acceptable method for improving SFT model performance in most settings. But what sort of scaling laws apply for partially synthetic datasets such as WILDCHAT-50M? We ablate the effect of data scaling at 100k, 250k and 500k samples across four DGMs. In Fig. 2, we see that average performance steadily improves with scale, as expected, for most models. In this figure, GPT is used denote the original WildChat dataset, which were generated using blend of different GPT checkpoints, primarily 3.5. The upper asymptote for performance (if it exists) is beyond the maximum we have encountered in our experiments.
How much does the length of the context window im- pact performance? Qwen-2.5-72B supports a large context window over 131K tokens, which we were able to take advantage of during data generation. We experiment with truncating all Qwen-2.5-72BB responses so that they are no longer than Llama-3.3 3.B responses (with context win- dow of 8, 1,99 tokens compared to 20,000 for Qwen in our experiments). Surprisingly, we find that the effect on the SFT model is slightly positive (.404 vs .400 averaged over 9 benchmarks). This is perhaps because we use a context win- dow of 8, 192 tokens in our SFT base model, Llama-3. 1-8B.
5

---

WildChat-SOM
Do models benefit from blending DGMs? One impetus behind creating very large prompt-response dataset like that of Zhao et al. (2024b) is the intuition that more samples and more interactions will generally lead to better models. In the case of prompt diversity, this appears to be true (Feuer et al., 2024a), perhaps because it makes the model more ro- bust to inconsistencies in prompting. But do the benefits of heterogeneous scaling extend to responses? In other words, do models trained on blended DGM responses outperform the sum of their parts? We conduct experiments to test this hypothesis, the results of which can be found in Sec. D, summarized here. We find that blending offers no benefit; the whole behaves almost exactly as the sum of its parts. This finding indicates that the dependencies on prompt di- versity discovered in prior work do not extend to the space of model responses even for large set of benchmarks. It is therefore most effective to optimize, rather than generalize, responses.
Llama, this score is 1.05. When we compare the SFTs to each other, by contrast, it is 2.61, indicating that the responses are much less similar.
Do models learn better from DGMs in the same model family? Recent work from Tajwar et al. (2024) has shown that approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i.e., employ a "negative gradient") outperform offline and maximum likelihood objectives. In this section, we inquire whether this finding extends not only to direct on-policy sampling, but something we might call approximate on-policy sam- pling, where a pretrained base model is fine-tuned on its own post-trained outputs, or those of a model from a similar family. Indeed, when we experiment with Qwen-2-7B and Llama-3-8B, we find that this approach produces stronger benchmark results when controlling for dataset size, and appears also to extend to larger post-trained checkpoint in the same model family; see Sec. B for the complete results. We acknowledge that our experiments here are limited in scope, and consider this an important area of future study.
Are models with strong performance on certain bench- marks better teachers for those benchmarks? We evalu- ate this question by comparing a Llama-3.188. 188-BBse model that is fine-tuned on Qwen-2.5-72B responses, to Llama- 3.1 8B- 3B-Base model fine-tuned on Llama-3.3-70B responses, and then comparing the two source models directly. We report the agreement rate between model checkpoints; we say that there is agreement if the fine-tuned model and the base model are both better or both worse than their coun- terparts, and there is not agreement otherwise. Across six benchmarks, we report an agreement rate of .5, which is at chance level, indicating that fine-tuned models do not necessarily inherit the strengths and weaknesses of their synthetic data generators.
3.3. Why do some models outperform others as sources of synthetic data?
We showed above that the choice of DGM can have a dra- matic effect on the performance of downstream LLMS fine- tuned on their responses. We now explore possible expla- nations for why SDQ varies so dramatically, even between superficially similar models.
We begin by eliminating some of the more obvious explana- tions. For example, we showed above that model parameter count and response length is not not reliably predictive of perfor- mance, so these are likely not primary factors contributing to data quality.
Is styling inherited during SFT? Effective use of presen- tational styling elements such as HTML tags, attributes and inline properties can have a strong effect on a text's readability and clarity. We investigate whether such formal styling behaviors are inherited from the DGM during the SFT process. We select subset of 80 turns from MTBench and examine the behavior of two DGMs (Qwen 2.5 72B and Llama 3.3 70B) and their finetunes. We convert the model responses from Markdown (which they commonly use in their responses) to HTML and report the absolute and proportional frequency of each styling tag, Absolute frequency is the raw count for each feature (see Tab. rows 1:4). Proportional frequency, for rows A, B, and feature F, is given by A[F]= + B FF. The closer this quantity is to 1, the more similar the model responses.
Inspection of the benchmark results in Tab. 2 shows greater variance in performance on chat-quality and instruction fol- lowing benchmarks (such as IFEval, MixEval, MTBench, and A ApacaEval). A natural explanation for this observation would be that SDQ is domain-specific, and is inherited from the DGM. If this was the case however, then benchmark performance of fine-tunes would generally agree with those of the DGMs those benchmarks. In reality, benchmark performance is not reliably inherited from the DGM. On the AlpacaEval leaderboard, Qwen 2.5-72B and Llama 3.3-70B are essentially tied; but Qwen 2.5-72B is a superior DGM (as measured by AlpacaEvval performance) (Dubois et al., 2024).
We report the raw results from this experiment in Tab. 3. Overall, we observe that SFTs track the styling of their DGMs very closely indeed; across all style features, the mean proportional frequency (MPF) of Qwen SFT com- pared to Qwen-2.5-72B is .91 across all features, and for
A potential resolution to this phenomenon lies in noting that LLM judges utilize range of judgment criteria, both explicit and implicit (Feuer et al., 2024a). Therefore, it is possible that supervised fine-tuning can significantly im- prove a model's (generalist) benchmark score by improving
6

---

WildChat-SOM
Model
Strong
Em
Ol
h1
h2
h3
p
len
Q72: None
741
53
83
230
24
135
1217
6492
L70: None
406
38
76
116
30
38
22
4
1222
6411
L8B: L70
331
34
75
109
27
13
32
3
1210
6207
L8B: Q72
808
60
77
260
20
149
33
1237
6585
PF L8B: L70. L70: None
0.815
0.895
0.987
0.940
0.900
0.342
1,455
0.750
0.990
0.968
L8B: Q72, Q72: None
1.090
1.132
0.928
1.130
0.833
1.000
1.104
1.269
1.016
1.014
PF L8B: : Q72. L8B : L70
2,441
1.765
1.027
2.385
0.741
0.000
4.656
11.000
1.022
1.061
Table 3. SFT models strongly inherit formal stylistic elements from their DGMs. This table indicates how frequently certain Markdown stylistic elements appear LLM responses (converted to HTML lags for greater clarity). The columns are names of HTML tags and inline properties, and the cells are frequeney counts. PF stands for proportional frequency, the ratio of the first and second model listed (order here is presumed to be arbitrary).
on only one or two criteria (e.g., comprehensiveness and readability), while its DGM, whose overall score may be already higher, comparatively underperforms because of limitations in some other criteria such as factuality.
score. In other words, even though the style of L8B : Q72 is still generally preferred by the judge, the superior factuality of L70: None drives an overall change in rank.
We analyze the same phenomenon through another lens in Fig. 3 and Fig. 4). Here, we visualize the frequency of common words in the judgments where the score differed be- tween models. Common negations of words were counted separately (although these were rare). We discover that L8B: : L70 responses are more commonly associated with words such as "lacks", "few", "misleading" and "concise", whereas L8B Q72 responses are more commonly associ- ated with words such ""apropriate", "comprehensive", "complete" and "detailed".
In order to determine whether this is indeed the case in our evaluations, we conduct experiment using 80 turns from MT Bench drawn from four models- - (i) Q72: None, (ii) L70: : None (DGMs), (iii) LSB Q72, and (iv) L8B L,0 each fine-tuned on 500k samples from WILDCHAT-5OM. We provide the complete conversations in the appendix for reference.
In Tab. 4, provide raw results in the form of MTBench score well as PrefRt, which stands for Preference Rate, or the rate at which a model's responses are preferred over another's s (100 means always preferred, 0 means never pre- ferred). L8L is the rate at which a model's response is pre- ferred over I.8B 1.70, L8Q is the rate at which a model's response is preferred over L8B Q72, L70B is the rate at which model's response is preferred over L70: None, and Q72B is the rate for Q72: None.
One limitation of this analysis is that it fails to capture common context that tends to accompany these words in the judgments; for example, the reason that "clearer" is more common in LSB I.70 judgments is that the judge frequently employed semantic variations of phrases like "could have been clearer", not because the responses themselves were "clcarer". The same applics to "critical".
We can see that L8B: Q72 outperforms L8B L70, and Q72 : None outperforms L70: : None, both in terms of average MTBench score, and in terms of win rate. Interestingly, the proportional improvement is similar in both comparisons, suggesting degree of heritability LLM judge When we manually inspected random sample of model
Key Words More Common in Llama Judgments
practical
function Jentsstt sucherreee
cul tur Cural prem
süch Sentral
more
lack
conc Cire
K
S
Cionctful
Lear
e
Iitipla
mimil aar
judge preferences.
&
good!
convoluted
distinct
receppablee
misleading
L
evocative
mult
alternative
outputs, we often agreed with the judge (the reader may form their own opinions by referring to examples in Sec. E). The MT bench prompts, responses and complete judgments are available in our GitHub repository.
Fund
basic
Tange The
tor starttt
tivee
Tacceptaa
Fam
magiiah
a
ownn
uunfami
a
ala
nex
white
Cher
b
Pepeet
un
oma
overall
came
specifas.
When we include the DGMs themselves in the comparisons, the conclusions are somewhat different. L.00 None out- performs L8B: Q72 both metrics. In partieular, on 5 of the 80 turns, L70 None flips the ranking (rather than simply breaking tie), compared to L8B L70. We note first that such judgment flips are quite rare. We manually inspect these 5 flips and find that on 4 of the 5, the LLM judge cited factuality as key reason for L8BB L.70's low
Mpglogical Jus
bene
deneficia
Tew
Luc descriptive commendable
Figure 3. Key words more common in L.8B I.70 judgments. The more negative tone of these judgments emphasizes words like clearer (as in, "could have been clearer"), lacks, convoluted and repetitive.
From these results, it seems that SDQ on a set of generalist
7

---

WildChat-SOM
Key
Words More
Common
in
Owen
judgmentts
try labs (Chiang al., 2024). Models trained on open data underperform those trained on closed data, both in the pretraining and in the post-training stage. Numerous algorithmic advances have been introduced into the recent literature which attempt to serve as scalable instruction tun- ing methods (Tunstall et al., 2023; Xu et al., 2023b; Zhou et al., 2023; Yasunaga et al., 2024).
mlll
accurate
e
familial
informative
correcet ect Cor
viee
true
tetelial
a
ppp
initial
opp
stylistic
Thigh
crucial
a
aate
lous
only
notable
greater
first
necessary
third
Commonactua
if
aract
likely
comp
complete
Sa
sever
eathenatig
tcall
a
mxinrr
unique fails
s t rong
sub sequent
n
consis seen
comprehenst
other
helpful
important
valid
cant
enough
Despite the variety (and abundance) of algorithmic ap- proaches, many fail to scale to the high-data regime (Feuer et al., 2024a). Therefore, in this work we turn to the more basic question of the quality of datasets used in post- bost-training. While there is still considerable ground to be covered, we hope that the size and diversity of our dataset, combined with its strong performance on SFT benchmarks, will en- courage researchers use basis for future work.
Ve
we wellstructured theroughettiieed
varied final
solid
Figure Key words more common in L8B Q72 judgments.
These judgments tended to be more positive; emphasis was placed on words like appropriate, necessary, comprehensive and accurate.
Recently, Zhao et al. (2024a) introduced WildChat- 1M, dataset containing 1 million ChatGPT interaction logs col- lected in real-world settings. Their work takes a different approach to ours by focusing on capturing authentic human- AI interactions rather than generating synthetic conversa- tions. WildChat- 1M leverages multiple GPT model variants, with 76% of conversations using the GPT-3.5-Turbo API (including versions 3.5 3.5-turbo-0613, 3.5-turbo-0301. and 3.5- turbo-0125) and 24% utilizing the GPT-4 API (including 4-1 -00eppr, 4-0314, and 1..012--preieee For human prompt acquisition, they deployed chatbot services on Hug- ging Face Spaces and implemented a two-step consent pro- cess, collecting anonymized data from users who explicitly opted into the research. While WildChat- 1M offers valuable insights into real-world chatbot interactions across multiple languages, our approach with WildChat-50m explores the complementary direction of high-volume synthetic data for post-training.
Model
MTBench
PrefRt-LSL
PrefRiLLS
PrefRe-L200
PrefR-QQ72
FlipCt
LSD:
8.75
15
L8B Q72
6.72
40
NIA
16:25
13:75
N/A
1.70: None 7.64
52.5
18.75
N/A
21:25
Q72: None
7.83
61.25
43.75
25
NAA
Table 4. LLM judge preferences for DGMs and SFTs. For column name definitions, we refer the reader to the main text, section See. 3. Overall, we observe that Qwen responses generally preferred by LLM judges, and that the rate at which they preferred is similar from DGM to SFT. Last but not least, note that reversals of judgment from SFT to DGM (PlipCt) are uncommon.
chat prompts such as WildChat is largely a function of donain-agnostic factors, such as the comprehensiveness of the model response, the clarity of the structure, and the tone and stylistic tendencies of the language.
4. Related Work
5. Limitations
Our work explores LLM post-training, a research area which has witnessed dynamic growth since 2022. SFT, sometimes referred to as instruction tuning (Mishra et al., 2022; Wei et al., 2022; Sanh al., 2022: Wang al., 2022; Long- pre et al., 2023), in which language models are trained samples including task instructions and their corresponding responses, has been shown to allow LLMs to generalize better to unseen tasks. Initially those samples were drawn from traditional NLP tasks with verified ground truth an- swers (Wang et al., 2023b). However, over time, it has become clear that a more heterogeneous approach, both to prompt and model responses, tends to lead to superior out- comes, and that combining instruction tuning datasets, either strategically or randomly, can lead to strong results (Taori et al., 2023; Conover et al., 2023; Wang et al., 2023a).
There are a few key limitations of this work which we wish to highlight here. Because of practical constraints, we were not able to report results using other post-training approaches other than SFT. It is likely that the relative effect of DGM choice would differ depending on the post- pest-training regime. Although our benchmark suite is standardized, bal- anced and large, it does not encompass all use cases. In particular, we did not evaluate performance on highly spe- cialized tasks, such coding or legal reasoning. We con- sider both of these limitations as pointing towards useful directions for future work. It might also be beneficial to consider the differential advantages and disadvantages of diversifying DGMs on smaller, more focused datasets, with and without ground truth.
Unfortunately, bost-training is an area where open science continues to trail closed advances made in frontier indus-
8

---

WildChat-SOM
6. Conclusions
resources, services, and staff. The authors also gratefully acknowledge compute support provided by the National Ar- tificial Intelligence Research Resource (NAIRR) Pilot, Groq LPU Inference Engine, and the Empire AI Consortium.
In this work, we make several valuable practical and empir- ical contributions the field's understanding of synthetic data. In particular, we provide robust empirical evidence that the choice of DGM is an extremely important factor in downstream SFT model performance on generalist chat benchmarks; simply by selecting good DGM, we compen- sate for a small dataset size and outperform more complex methods and carefully curated SFT mixes.
References
Adler, B., Agarwal, N., Aithal, A., Anh, D. H., Bhattacharya, P., Brundyn, A., Casper, J., Catanzaro, B., Clay, S., Co- hen, J., et al. Nemotron-4 340b technical report. arXiv preprint, 2024.
Equally important, we provide novel insight into why certain DGMs produce much higher SDQ than others; our exper- iments indicate that comprehensiveness, clarity, tone and prompt responsiveness are highly heritable during the SFT process, even on generalist data, unlike skills such as world knowledge or mathematics, which are heritable only when data is curated for that particular purpose.
AI, Young, A., Chen, B., Li, C.,, Huang, C., Zhang, G., Zhang, G., Wang. G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang. S., Xie,
W., Huang. W., Hu, X., Ren, X., Niu, X., Nie, P., Li, Y., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by Ol. 2025. URL https: htt//arxivv.o//// org/ abs 403. 0 04652.
Finally, we provide novel comparative insights into LLMs, reporting high degree of similarity in the prompt responses of diverse LLMs. Taken together with the prior observation, the distinction between high and low SDQ may be subtle, and worthy of future research.
Banerjee, S. and Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Goldstein, J., Lavie, A., Lin, C.-Y., and Voss, C. (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Ar- bor, Michigan, June 2005. Association for Computational Linguistics. URL https: http://aaaaattoooooooooo org/ W05- 0909/.
Impact Statement
This paper presents work whose goal advance the field of machine learning. There are many potential societal con- sequences of our work. In particular, we wish to acknowl- edge the fact that we are building upon a dataset (WildChat) which is known to contain examples of user inputs on po- tentially upsetting topics, including but not limited to sex, violence, and bigoted claims (Zhao et al., 2024b). Following the authors of that work, we will require approval before re- searchers are allowed to access our datasets, and will release our data under the A12 ImpACT License, which explicitly forbids certain use cases for this data.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z.., Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang, T., Jiang, T. Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, I.., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong. J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L.. Ma, R., Ma. Z., Ning, W., Ouyang, I., Qiu, J., Qu, Y., Shang, F,, Shao, Y., Song, D.. Song. Z., Sui, Z., Sun, P., Sun, Y.. Tang, H., Wang, B., Wang, G., Wang, J., Wang. J., Wang, R., Wang, Y., Wang, Z., Wei, X., Weng, Q., Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying. H., Yu, J., Yu, J., Zang, Y., Zhang, C., Zhang, L., Zhang, P., Zhang. P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang. W.. Zhang, X., Zhang, X., Zhao, H., Zhao, Q.. Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internlm2 technical report, 2024. URL https::///// / arxiv. org abs /24.2.
In addition, our data will be subject to any additional re- strictions from the licenses of the models we utilize. We also preserve, without modification, the original user in- puts from WildChat (Zhao al., 2024b), and therefore in our work there remains the possibility that users may have inadvertently included personal information within their conversations which was not detected by the (considerable) safeguards put in place by the WildChar authors. We will make this clear to users upon dataset release.
Acknowledgements
Chiang. W.-L.,, Zheng, I.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H. Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint, 2024.
This work is supported in part by the AI Research Institutes program supported by NSF and USDA-NIFA under Award No. 2021-67021-35329 a Google Cyber NYC gift grant, and NSF grant# # 2154119. The authors gratefully acknowl- edge the support of NYU IT's High Performance Computing
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan,

---

WildChat-SOM
J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tuned llm. URL, 2023.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https: /arxiv. org/ abs abs/ 04475.
Feuer, B., Goldblum, M., Datta, T., Nambiar, S., Be- saleli, R., Dooley, S., Cembalest, M., and Dickerson, J. P. Style outweighs substance: Failure modes of Ilm judges in aligmment benchmarking. 2024a. URL https:/ //a//axxvvv/.//// wwwp:ii..rr/ab////00000 ..04409.12220 15268.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint, 2023.
Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal 1lms, 2024. URL https: / aaxiv.iiivvv.... org/ abs/ /bs/2409 2209 11402.
Feuer, B., Xu, J., Cohen, N., Yubeaton, P., Mittal, G., and Hegde, C. Select: A large-scale benchmark of data cu-
ration strategies for image classification, 2024b. URL
DeepSeek-AI, Guo, D., Yang, D., Zhang. H., Song. J., Zhang, R., Xu, R., Zhu, Q. Ma. S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng. B., Lu, C., Zhao, C., Deng, C., Zhang. C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, F., Lin, F., Dai, F, Luo, F. Hao, G., Chen, G., Li, G., Zhang. H., Bao, H., Xu, H. Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H. Guo, J., Li, J.. Wang. J., Chen, J., Yuan, J. Qiu, J., Li, J., Cai, J. I.., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L., Wang. L., Zhang, L., Xu, L., Xia, I., Zhang, M., Zhang, M., Tang, M., Li, M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang. Q., Chen, Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J.. Jin, R. L.. Chen, R., Lu. S., Zhou, S., Chen, S., Ye, S., Wang. S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S.. Wu, S., Ye, S., Yun, T. Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W. Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q. Jin, X., Shen, X., Chen, X., Sun, X., Wang. X., Song, X., Zhou, X., Wang. X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, Y. X., Zhang, Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang. Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y. Ou, Y., Wang, Y., Gong, Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao,
Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open Ilm leaderboard 2. https://huuiinggaaaae htt.g/ggagggg.c.e.e/o//c co/ soospaces/ https: open 1ew-lmmmmmeeeeeee Leaderboard/blog, 2024. Accessed: 2025-01-13.
Ganesan, K. Rouge 2.0: Updated and improved mea- sures for evaluation of summarization tasks, 2018. URL
http:///aaaaa..rraa////3.... 01937.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Noac h, A. L., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L.,, Tang, E., Thite, A., Wang. B., Wang. K., and Zou, A. framework for few-shot language model evaluation, 07 2024. URL, 2024. https://zenodo.orgg www..//oodoo///o///////oooo/o/// htttrrecoords/1100000000 12086602.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang. H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Sun, J., Li, J., Zhao, L., Wu, L., Zhong, L., Liu, M., Huang, M.. Zhang, P., Zheng, Q.. Lu, R., Duan, S., Zhang, S., Cao, S., Yang, S., Tam, W. L., Zhao, W. Liu, X., Xia, X., Zhang, X., Gu, X., Lv, X., Liu, X., Liu, X., Yang, X., Song, X., Zhang, X., An, Y., Xu, Y., Niu, Y., Yang, Y., Li, Y., Bai, Y., Dong, Y., Qi, Z., Wang, Z., Yang, Z., Du, Z., Hou, Z., and Wang, Z. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024. URL https: pttpss//arxiv..
abs 22406.46..2 12793.
Z., Ma, Z., Yan, Z., Wu. Z., Gu, Z., Zhu, Z., Liu, Z., Li,
Z. Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z. and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: // /atwxvvivvv////////////////aaav/ org/ abs/////////// 2501 12948.
Gomez, A. Introducing Command R+: A scalable LLM built for business. URL https://cohere.ce blog/ commmand-r-lus--aa www.miioooooooooooooocc ctt.:pllu-mmccoooootttaoaaa
Guha, E., Raoof, N., Mercat, J., Frankel, E., Keh, S., Grover, S., Smyrnis, G., Vu, T., Marten, R., Saad-Falcon, J., Choi, C.,, Arora, K., Merrill, M., Deng, Y., Suvarna, A., Bansal, H., Nezhurina, M., Choi, Y., Heckel, R., Oh, S.,
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint, 2024.
10

---

WildChat-SOM
Hashimoto, T., Jitsev, J., Shankar, V., Dimakis, A., Sathi- amoorthy, M., and Schmidt, L.. Evalchemy, November 2024.
Lian, W. e. a. Axolotl. https://github.
pott-ai-cloudd/ 2025-01-13.
2025. Accessed:
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding, 2021. URL https: //arxiv. org/
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi-
gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., Abend. O.. Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Rat- N., Rozen, N., Shwartz, E., Zusman, M., and Shoham, Y. Jamba: A hybrid transformer-mamba lan- guage model, 2024. URL https://arxive 20s/2403.1998
03300.
Iskender, N., Polzehl, T., and Möller, S. Reliability of hu- man evaluation for text summarization: Lessons learned and challenges ahead, In Belz, A., Agarwal, S., Gra- ham, Y., Reiter, E., and Shimorina, A. (eds.), Proceed- ings of the Workshop on Human Evaluation of NLP Sys- tems (HumEval), pp. 86-96, Online, April 2021. Asso- ciation for Computational Linguistics. URL https: //aclaannn //aclanthologyy 2021 hwmeval-1.101
Lin, C.-Y. and Hovy, E. Manual and automatic evaluation of summaries. In Proceedings of the Document Under- standing Conference (DUC), 2002.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q V., Zoph, B., Wei, J., al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint, 2023.
Ivison, H., Wang. Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I. al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint, 2023.
Loshchilov, I. and Hutter, F. Fixing weight decay regular- ization adam. CoRR, abs/1711.05101, 2017. URL http: htt///axxivv... wreiii.org//bbee 1711. 05101.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., d. Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint, 2023.
Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summa- rization. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Compu- tational Linguistics. doi: 10.18653/ acl- 173. URL a acl ma . 173/.
Jin, R., Du, J., Huang, W., Liu, W., Luan, J., Wang. B., and Xiong, D. A comprehensive evaluation of quantization strategies for large language models, 2024. URL https: // //arxiv.. org/ abs / 222/2242.2222 16775.
Jing, H., Barzilay, R., McKeown, K. R., and Elhadad, M. Summarization evaluation methods: Experiments and analysis. In Radev, D. R. and Hovy, E. H. (eds.), Working Notes of the AAAI Spring Symposium on Intelligent Text Summarization, pp. 60-68, 1998.
2020.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward, arXiv preprint, 2024.
Kurtic, E., Kuznedelev, D., Frantar, E., Goin, M.. and Alistarh, D. Sparse fine-tuning for inference acceler- ation of large language models, 2023. URL https: //arxiv. org/ abs/ 2210223000000 06927.
Mishra, S., Khashabi, D.,, Baral, C., and Hajishirzi, H. Cross- task generalization via natural language crowdsourcing instructions. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1, pp. 3470-3487, Dublin, Ireland, Association for Com- putational Linguistics. doi: 10.116663/11/22222.eell 244. URL, May 2022. https:/accaamtooogg..o/222222220 long.244.
Kwon, W., Li, 7., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Lambert, N., Morrison, J., Pyatkin, V., Huang. S., Ivison, H., Brahman, F., Miranda, L. V., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang. J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing frontiers in open language model post-training, 2024. URL https::////......... / arrxiv. org/ abs///ss11111 151244.
Mistral, I. Ministraux: Pushing the boundaries of effi- cient transformer design. URL, 2024. Retrieved 2024-11- 17. https:/mmismalaaiieessiiitnttii
NexusFlow AL Athene: Pioneering the Future of AI Workflows, 2025. URL hitps:/ http/exxufoooo bl
Accessed: 2025-01-29.

---

WildChat-SOM
http://wwwm.gguuuu.bboo/o//////e J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neubig, G., and You, Y. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures, 2024. URL https:// arxiv.org www.///.......oo. 06565.
Degry, T., Dimson, T., Zheng, T., Garipov, T., Stasi, T., Bansal, T., Creech, T., Peterson, T., Floundou, T., Qi, V., Kosaraju, V., Monaco, V., Pong, V., Fomenko, V., Zheng, W., Zhou, W., McCabe, W., Zaremba, W. Dubois, Y., Lu, Y., Chen, Y., Cha, Y., Bai, Y., He, Y., Zhang, Y., Wang, Y., Shao, Z., and Li, Z. Openai ol system card, 2024. URL https: /ppss///axxan www.iiiwgv.aaa...aa.aaaoaaoo /twpab////21...
OpenAI, Jacch, A.,, Kalai, A., Lerer, A.,, Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beu- tel, A., Carney, A., Iftimie, A., Karpenko, A., Passos, A. T., Neitz, A., Prokofiev, A., Wei, A., Tam, A., Bennett, A., Kumar, A., Saraiva, A., Vallone, A., Duberstein, A., Kondrich, A., Mishchenko, A., Applebaum, A., Jiang, A., Nair, A., Zoph, B., Ghorbani, B., Rossen, B., Sokolowsky, B., Barak, B., McGrew, B., Minaiev, B., Hao, B., Baker, B., Houghton, B., McKinzie, B., Eastman, B., Lugaresi, C., Bassin, C., Hudson, C., Li, C. M., de Bourcy, C., Voss,
Qwen Team. Qwen2.5: A party of founda- Alibaba, 2024.
tion models, September 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
C., Shen, C., Zhang, C., Koch, C., Orsinger, C., Hesse, C., Fischer, C., Chan, C., Roberts, D., Kappler, D., Levy, D. Selsam, D.. Dohan, D., Farhi, D., Mely, D., Robinson, D., Tsipras D., Li, D., Oprica, D., Freeman, E., Zhang. E., Wong, E., Proehl, E., Cheung, E., Mitchell, E., Wal- lace, E., Ritter, E., Mays, E., Wang, F., Such, F. P., Raso, F. Leoni, F., Tsimpourlas, F., Song, F., von Lohmann, F. Sulit, F., Salmon, G., Parascandolo, G., Chabot, G., Zhao, G., Brockman, G., Leclere, G., Salman, H., Bao, H., Sheng, H., Andrin, H., Bagherinezhad, H., Ren, H., Lightman, H., Chung, H. W., Kivlichan, I., O'Connell, I. Osband, I., Gilaberte, I. C., Akkaya, I., Kostrikov, I,, Sutskever, I., Kofman, I., Pachocki, J., Lennon, J., Wei, J., Harb, J., Twore, J., Feng, J., Yu, J., Weng, J., Tang. J., Yu, Candela, J. Q. Palermo, J., Parish, J., Heidecke, J. Hallman, J., Rizzo, J., Gordon, J., Uesato, J., Ward, J., Huizinga, J., Wang, J., Chen, K., Xiao, K., Singhal, K., Nguyen, K., Cobbe, K., Shi, K., Wood, K., Rimbach, K., Gu-L.emberg, K., Liu, K., Lu, K., Stone, K., Yu, K., Ahmad, L., Yang, L., Liu, L., Maksin, L., Ho, L., Fedus, L., Weng, L., Li, L., McCallum, L., Held, L.. Kuhn, L., Kondraciuk, L.,, Kaiser, L.,, Metz, L., Boyd, M., Trebacz, M., Joglekar, M., Chen, M., Tintor, M., Meyer, M., Jones, M., Kaufer, M., Schwarzer, M., Shah, M., Yatbaz, M., Guan, M. Y., Xu, M., Yan. M., Glaese, M., Chen, M., Lampe, M., Malek, M., Wang, M., Fradin, M., McClay, M., Pavlov, M., Wang, M., Wang, M., Murati, M., Bavar- ian, M., Rohaninejad, M., McAleese, N., Chowdhury, N., Chowdhury, N., Ryder, N., Tezak, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Chao, P., Ashbourne, P., Izmailov, P., Zhokhov, P., Dias, R., Arora, R., Lin, R., Lopes, R. G., Gaon, R., Miyara, R., Leike, R., Hwang, R., Garg, R., Brown, R., James, R., Shu, R., Cheu, R., Greene, R., Jain, S., Altman, S., Toizer, S., Toyer, S., Miserendino, S., Agarwal, S., Hernande7, S., Baker, S., McKinney, S. Yan, S., Zhao, S., Hu, S., Santurkar, S., Chaudhuri, S. R., Zhang, S., Fu, S., Papay, S., Lin, S., Bal- aji, S., Sanjeev, S., Sidor, S., Broda, T., Clark, A., Wang, T., Gordon, T., Sanders, T., Patwardhan, T., Sottiaux, T.,
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szezechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-.., Wang. H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj. T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, M. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of I1ms should leverage suboptimal, on-policy data, 2024. URL ht //ar:ivaaaivi///aaa org/ abs//4bs//// 14367.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An citructtoo-folloiing Ilama model, 2023.
Team, T. M. G., Hardin, C., Dadashi, R., Bhupatiraju, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., and Hussenot, L. and et al. Gemma, 10:34740, 2024. https//www..aggllc.oom//////3
Teknium, R., Quesnelle, J., and Guang, C. Hermes 3 techni- cal report. arXiv preprint, 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint, 2023.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint, 2023.
12

---

WildChat-SOM
van Halteren, H. and Teufel, S. Examining the consen- between human summaries: initial experiments with factoid analysis. In Proceedings of the HLT-NAACL 03
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng. J., Tao, C., and Jiang. D. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint, 2023a.
Text Summarization Workshop, pp. 57-64, 2003. URL ht https: / / acl antho wwtchhlooy..roo wtgg///rrg///.. /003 0508/.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions, 2023b. URL https: arxiv org/ abs / 2.334.122240
Wang, Y., Mishra, S., Alipoormolabashi. P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A.. Stap. D., Pathak, E., Karamanolakis, G., Lai, H., Purohit, I,, Mondal, I., Anderson, J., Kuz- nia, K., Doshi, K., Pal. K. K., Patel, M., Morad- shahi, M., Parmar, M., Purohit, M., Varshney, N., Kaza, P. R., Verma, P., Puri, R. S., Karia, R., Doshi, S., Sampat, S. K., Mishra, S., S. Reddy, A., Patro, S., Dixit, T., and Shen, X. Super-Naturallnssrccion Generalization via declarative instructions on 1600+ NLP tasks. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pp. 5085-5109, United Arab Emirates. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 1022.emmlp-mainn34000 URI., December 2022. Abu Dhabi. http:///waa.tool....////////////
Xu, Z., Jiang, F, Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Align- ment data synthesis from scratch by prompting aligned Ilms with nothing, ArXiv, abs/2406.08464. 2024. URL https: /api. sew.isseaanttiii iimmantcscchlar.. http:/CoopSID::22330000
Yasunaga, M., Shamis, L., Zhou, C., Cohen, A., Weston, J., Zettlemoyer, I.,, and Ghazvininejad, M. Alma: Align- ment with minimal annotation, 2024. URL https: //arxiv. org/ abs 22112003303
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint, 2024a.
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neu- ral Information Processing Systems, 36:77764-777766, 2023a.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild, 2024b. URL https: /tas://axee http///axvvvvvo/// /abs/ $405.01470,
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu. Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging Ilm-as-a-judge with me- and chatbot arena, 2023. URL https: / //arxiv. orroor/g///oo abs 2306 05685.
Wang, Y., Kordi, Y., Mishra, S.. Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self- gelf-generated instructions, 2023b. URL https: / /arxiv...vvv////i///aaa///aaavvv org/ abs 2212.2222222 10560.
Zheng, L.,, Chiang, W.-L.,, Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E.. Stoica, I., and Zhang, H. Lmsys-chat- lmm A large- scale real-world Ilm conversation dataset, 2024. URL https: / wrwaa. http://wwwv..rrr..///// abs 2309. 11998.
Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkum, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Red- pajama: an open dataset for training large language mod- els. 2024. URL https: / wwwwaaiivvvoooo org/ abs wtwgoa////aa...../ 12372.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlmoyer, L., and Levy, O. Lima: Less is more for alignment, 2023. URL https: //t.//avorrgg////eeee///// 11206.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners, 2022. URL https:// arxiv. org/ abs abs///09......//.//////////baa/b 01652.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., and Goldblum, M. Livebench: A challenging, contamination- free Ilm benchmark, 2024. URL https: //axxiv. org/ /bsg/ab//22 /a/////s//2.... 2406. 1.314.
13

---

WildChat-SOM
Model
avg-rougel
std rougel
avg-rougeL
std_rougeL
avg.meteor
meteor
Mixtral-8x7B..
Si-tal--x7B-Instrree Instruct
0.37
0.11
0.19
0.06
0.20
0.05
1amma-3.-NNmotrroe Instrrccttt
0.37
0.06
0.23
0.05
0.20
0.05
Dwen2.5-72B-Insttne
0.34
0.09
0.17
0.05
0.17
0.05
Mistral-7BB-wizarddlm
0.34
0.07
0.22
0.06
0.16
0.05
Mistral-7B-siee
Mistral-7B-sharegt---voo
0.34
0.06
0.18
0.03
0.18
0.04
Baaral-7B-Bas-SFTAAA
0.33
0.11
0.17
0.05
0.19
0.06
1ilemim2.55200-chatt
0.33
0.08
0.15
0.03
0.20
0.06
0.33
0.10
0.16
0.05
0.16
0.06
Llama-3.3-708-Instmee
0.33
0.09
0.17
0.04
0.20
0.05
Llama-2-7b-chatt
0.32
0.09
0.19
0.07
0.20
0.06
Missea--7BB-Bas--SFTCPPO
0.32
0.07
0.17
0.04
0.18
0.05
Qwen2-77B-Instruet
0.32
0.08
0.15
0.04
0.17
0.06
Hhama-3-8BB-SharGGPP-1112
0.31
0.09
0.18
0.07
0.15
0.06
Qwee2255-Coder-22B--nsss
0.31
0.10
0.15
0.05
0.18
0.05
Liama3--B--aggii-P---7----0-
0.30
0.15
0.18
0.09
0.17
0.09
google
0.30
0.10
0.16
0.06
0.14
0.06
Al21-Jamba-1.--Mii
0.29
0.13
0.17
0.08
0.14
0.06
OpenHermes-2-Mistral--12
0.28
0.09
0.16
0.06
0.15
0.07
LAma-3-BAsc-BB--FT-OOROQ
0.27
0.07
0.14
0.03
0.22
0.05
google-gemmma-2227biit
0.27
0.04
0.13
0.02
0.15
0.03
OpenlHemmss2255MMisrral----
0.27
0,06
0.15
0.04
0.13
0.04
0.26
0.13
0.14
0.07
0.13
0.07
Mistral-7B- -aass-S-TT-
0.25
0.05
0.13
0.04
0.11
0.04
Ministral-8B-Ine SB-nnstruct-2410
0.24
0.13
0.12
0.07
0.13
0.09
0.23
0,06
0.13
0.03
0.19
0.03
BBara--7BBBBs
0.19
0.06
0.08
0.02
0.16
0.04
Table 5. Intra-LIM response similarity in RE-WILD. Here report intra-IIm response similarity scores. The method we use to obtain
these scores is described in Sec. 2.2.
A. Intra-LLM Response Similarity
In our experiment, we collect 500 responses from 25 randomly selected models and compute their similarity to set of reference responses (randomly sampled from other models in the set of 25) using three traditional NLP similarity metrics: R0UGE-1, ROUGE-L, and METEOR (Banerjee & Lavie, 2005; Ganesan, 2018). These metrics are computed as F1 Fi-score over unigrams, Fl-score using the longest common subsequence, and weighted Fi-score giving 9:1 weightage for precision over recall with chunking penalty. In Tab. 5, we include the extended results for response similarity. To see relevant citation for any partieular model in this table, please refer Sec. C.
Overall, we find high similarity across models, albeit with fairly high degree of variance; for example, Mixtral-8x7BB Instruct has high ROUGE-1 similarity of 0.37, while Ministral- Betrral-B--nnsruuce In--Intruut--4100 has 0.24, among the lowest scores. Considering the diversity of both prompts and models, this level of similarity suggests that L.LMs produce much more regular and predictable output than humans. On model-by-model level, we can interpret measures like these as signal as to "how generic" and "how consensus-driven" any particular LLM's response is. also observe that the larger models tend to generate more similar responses; some sense, they are closer consensus response to the prompt.
B. Ablation on the Effect of On-Policy DGMs
In Tab. 6, we provide extended results on the effects of fine-tuning using DGMs that are highly similar to the fine-tune targets (and therefore, some limited sense, on-policy). For the analysis of the results here, please refer to our main paper, Sec. 3. The naming convention used this table is described Sec. 2.2, with one new abbreviation; Qwen-2-78 Been-22-B-Bass (B-Base := Q7B.
14

---

WildChat-SOM
Model
MTBench
AlpacaEval
BBH
GPQA
MATH
MUSR
IFEval
MMLU
MixEval
Avg
L8B: L81
6.26
21.12
0.00
0.00
0.04 # 0.00
0.37 = 0.03
0.38 = 0.04
0.33
0.65 = 0.01
0.36
L8B Q7
6.03
17.26
0.40
0.04
0.03 + 0.00
0
0.29 0.04
0.30
0.02
0.35
L8B 1.70
6.23
24.91
0.01
0.04
0.04 + 0.00
0.39
0.34 0.04
0
0.65
0.36
Q7B L81
6.51
15.87
0.01
0.07 0.04
0.43
0.04
0.39
0.01
0.39
Q7B: Q7
27.09
0.31 0.04
0.09
0.45
0.00
0.42
0.09
0.43
Q7B: Q72
7.25
36.68
0.01
0.04
0.21 0.01
0.43
0.04
0.01
0.05
0.42
Q7: None
7.17
33.14
#.001
0.33 = 0.04
0.19 0.00
0.04
0.42
0.40
0.03
0.44
L8I: : None
7.20
30.84
0.51 0.00
0.33 0.04
0.12 à 0.01
0,40 0.03
0.42 * 0.04
0.38 #.01
0.74 0.01
0,41
Table Models learn more effectively from highly similar DGMs. In this table. we report the complete, extended results from our experiments on the effect of diversifying both DGM and SFT-target. Both Ilama and Qwen benefit from more similar upstream models.
C. List of All LLMs in the Study, with Citations
We attempt to use model naming conventions consistent with those on HuggingFace; that way, in order to find any particular model, it should only be necessary to Google its name. We do not include the complete HuggingFace links because they might aeeennony this work. Where available, we include a citation to the work where the model was first introduced into the literature.
NVLM-D-72B, from Dai al. (2024)
JJama-3.3-70B-Imat Instruct,, Llama-3.1 3.1--B-Instruu Llama-3.1 3.1--00B 7700-IIntruee from Dubey et al. (2024)
Yi-1.5-34B- from Al al. (2025)
chai-ommaand-rplus--88--2 2024, from Gomez
mistral-7b-sft-Be from Tunstall et al. (2023)
Llama---B--aagiieAAAinn--00 Llama-3-8B-
-ara----aaii--ro-------0000 Llama-3-8B-O000 S--ee---------eeeeee------- 243K,, Llama-
H-BB-ShaneGPT-112K Llama-3-88 1aana-----Tuull---333- Tulu-330K, Llama-3-88 -attn------Urrhcatttt
Ultrachtt-20000 Llama-3-80 -aaeaa-------iiii 3B-WildChat, Llama-3-8B.
WizardI.M-196K, from Xu al. (2024)
Athene-70B, from NexusFlow (2025)
Qwen2-7B-Instre Instruct, 1innn..5-14BIInttttee Qwcn2.5-72B-Insn Instruct, Qwen2.5-Coder- 32---InstuccIc from Qwen Team (2024)
glm-4-9b-chat from GLM et al. (2024)
A221-Jamba-1.5-Mim from Lieber et al. (2024)
gemma-2-27b-it, gemma-2-9b-it from Team et al. (2024)
internmm..5200bchall from Cai et al. (2024)
1lama-2-7b-chat-hh 1amma----13bchhthhhe from Touvron et al. (2023)
Ministral-8B-IInstrre---21111 MistaalNNemo-Instrueeeaa00 Mattral-x7B-Innstrc---. from Mistral (2024); Jiang et al. (2023)
--t----eoooooo------eeeeeee from Adler et al. (2024)
Histral-7B-magpie-V/ Missral-7B-sharegpe-iieeee Mistral-7B-tulu,
Mistral-7--wizardln--11 Misraal-BBulll.iiitta////eaaaaa from Feuer
et al. (2024a)
---------e------SB Baaeee--------B-SF-T-- Laam-3--Bac-SD--FFARDDP
Llama-3-Base-9--FFT-- Llama-3-
Mistral-7B-Bant Basc-SFT-
CPO, Baatral-BB-Bas-SFT----
Bareall-BBBBaa--SFF-T- Baad---B------F--KTT Settal--B-Baas-S55
RDPO, Mistral-7B- Boatral-B-Bass-SS5 Base-SFT-RRHEF Mistral-7B- 7----B--BB---- SFT-SLi--HF, Mistral- -B-rall Mase-SFF-SimPO from Meng et al.
(2024)
15

---

WildChat-S0M
Openleerees-2-Mistra--17 OpenlHermes225-Missnaa---- from Teknium et al. (2024)
D. blations on Blending Data-Generating Models
Our extended results on the effect of blending DGMs can be found at Tab. 7. The analysis of this table can be found in our main paper, as well as explanations of the abbreviation conventions for model names.
Model
MTBench
Alpaca Eval (LC)
BBH
GPQA
MATH
MUSR
IFEval
MMLU
MixEval
Avg
L8B: (500k)
6.33
0.48 0.01
0.28=0.04
0.33
0.01
0.64
L8B L81 (500k)
6.52
21.03
0.46 0.01
0.04
0.05 1000
0.39 0.03
0.42
0.32 1.3200001
0.66
0.37
L8B: L8I + Q7 (500k)
6.43
18.57
0.47 0.01
0.28 0.04
0.05 0.00
0.00
0.32 0.04
0.64
0,36
Q72 (500))
6,51
41.67
0.48 0.01
0.29 0.04
# 0.00
0.39 0.03
0.39 0.04
0,30 # 0.01
0.66
L70 (500k)
6.39
0.46
0.31
0.32
0.63
L8B: Q72 L70 (500k)
6.82
39.93
0.48 a: 0.01
0.29 0.04
0.04 0.00
0.38 a: 0.03
0.38 0.04
0.31 0.01
0.65 4
0.36
Table Effeetss of blending DGMs. Where we blend models, we always draw random sample approximately equal size from each DGM, and denote the mixture as DGM DGM B. c.g., L8I Q7. In this table, it can be seen that blends benchmark between their DGMs, not above them (as measured by Avg).
E. Output Samples
In this appendix section, we provide complete sample responses from a pair of DGMs in conveniently human-readable Instructt are the models shown here. For more samples, including sample judgments, please refer to our GitHub repository. The responses may be more convenient to read in the repository than they are this document, as they are HTML formatted; however, we include them here as not to inconvenience the reader.
format, along with the prompt(s) the conversation; Llama-3.3-70B and Qwen-2.5-72B 2-sstutt-nntttu
Please note that some of the output samples may contain statements that are offensive, toxic otherwise distasteful to some readers. Please exercise caution when reading both prompts and responses. The content of the prompts and responses does not necessarily reflect the beliefs of the authors.
16

---

Shared Prompt:
Describe gas giant with gothic colors. Including chemistry composition.
allenai_WildCChee 1amm-
allenai_WildChat-1NM WiiCChha-11---..l--eeeeenn Instruct
Instruct
ASSISTANT (Lang: English Finish: stop)
ASSISTANT (Finish: stop)
somber, mystical realm of gas giant. Shrouded eerie, dark ombré,
Certainly! Imagine gas giant that embodies the essence gothic aesthetics, celestial body that is both majestic and foreboding, shrouded in deep, dark hues and enveloped in a mysterious, almost sinister atmosphere.
like the mourning vell of bereaved widow. The hue the sky deep,
foreboding indigo, gradating into sickly, yellowish-green towards horizon, if tainted by the faint glow of luminescent decay.
Appearance
The gas giant's atmosshere is a hauntingly beautiful tapestry of gothic colors. The sky is dominated by deep purples, rich blacks, and dark blues,
The atmosphere itself noxious brew hydrogen (H2) and helium (He), with
wispy tendrils methane (CH4) ammonia (NH3) wafting through the upper
occasional streaks of blood-red and silver. The clouds, thick and swirling, are tinged with murky, almost metallic sheen. Lightning storms, rare but intense,
strata, like the ghostly whispers of forgotten souls. These gases writhe and
twist in torment, forming swirling starm systems that churn with maddening
illuminate the atmosphere with flashes electric blue green, casting eerie shadows across the gas giant's surface.
fury, as if fueled by the essence of despair.
As delve deeper into the heart of the gas giant, the pressure and temperature
escalate, forging environment that beautiful and terrifying. chemistry composition shifts, with hydrogen and helium
both
Chemistry Composition
1. Hydrogen (H,): The primary component gas giant's atmosphere, hydrogen makes up approximately

---

succumbing to the crushing and heavier elements such carbon, nitrogen, and oxygen
90% of the atmosphere by volume. appears as a deep, almost black hue due the presence various compounds and the thick, absorbing nature of the upper atmosphere.
beginning to assert their dominance. Water, ammonia, and methane ices form, like the frozen tears of the damned, suspended in sea of liquid hydrogen and helium.
2. Helium (He): The second most abundant element, helium comprises about 9%
of the atmosphere. It contributes to the overall density opacity
The planet's core, glowing, hellish ember, pulsates with otherworldlyy
atmosphere, enhancing the gothic, shadowy appearance.
3. Methane (CH,): Methane, present in lesser amounts, gives the planet its deep blue and purple The methane molecules absorb red light, leaving the atmosphere with predominantly blue and purple coloration. In the upper layers,
energy, if fueled by the very essence darkness itself.
The magnetic field that surrounds is twisted, nightmarish latticework, crackling with
electrical energy, and weaving spell of eerie, gothic power.
In this forsaken realm, the winds howl like a chorus the doomed, while the lightning flashes with an eldritch, greenish-blue glow, casting an eerie,
methane condenses into clouds, contributing to the gothic, stormy appearance.
Ammonia (NH,): Ammonia present the form of particles and
macabre light the churning clouds. The gas giant world of twisted, gothic horror, place where the very fabric of reality to be warped and
clouds, adding the planet's dark, stormy appearance. These clouds often tinged with dark red and black, creating moody and dramatic effect.
distorted, and where darkness reigns supreme.
5. Water (H,0): Water vapor is also present, contributing to the planet's
Chemical Composition:
Hydrogen (H2): 75-80%
atmospheric complexity. Water ice and vapor form high- altitude clouds that appear as ghostly, translucent layers, enhancing the gothic aesthetic.
Helium (He):: 20-25%
Methane (CH4): 1-5%
Ammonia (NH3): 0.1-1%
Water (H20): 0.1-1%
Carbon, nitrogen, and oxygen compounds: 0.01-0.1%
Sulfur Compounds: Sulfur dioxide (SO,) and hydrogen

---

Colors:
sulfide (H₃S) are minor significant components. These compounds contribute to the planet's
Deep indigo (#3B3F54) sickly yellowish-green
(#889467)
Faint, glowing blue-green (#34A8FF) of luminescent decay
reddish and brownish hues, creating contrast adds depth and forebroadness to the atmosphere.
Hellish, ember-like glow of the core (#FF9900)
Eerie, greenish-blue lightning flashes (#33CCFF)
This gas giant is place of dark, gothic beauty, where the
7. Phosphine (PH,): Phosphine is trace gas that can create spectacular,
boundaries between
reality and nightmare are blurred, and the essence of the universe seems to be twisted and distorted.
bioluminescent clouds and lightning storms. During these storms, the phosphine reacts with other chemicals, producing flashes of electric blue and green, adding the eerie, gothic ambiance.
8. Metallic Hydrogen: Deep within planet, under immense pressure and temperature, hydrogen exists in metallic state. This metallic hydrogen
responsible for the planet's strong magnetic field and contributes
overall energy and dynamism.
Additional Features
The gas giant is encircled by dark, narrow rings compased of small, rocky debris and ice particles. These rings reflect the deep, moody colors of planet, adding to the gothic atmosphere.
Rings:
Moons: The planet has several moons, each with its own gothic charm. Some moons are shrouded dark, icy surfaces, while others have volcanic activity that produces plumes sulfur and other volatiles, creating stark, surreal landscape.
This gas giant, with gothic colors and complex chemistry, is mesmerizing

---

and haunting sight, true marvel of the universe.

---

Shared Prompt:
[Response must be written short blurb.]
As 2023, what the copyright starus of Oswald the Lucky Rabbit?
allenai WMldCChat----Fullmett---lle Llama-3.3-70B- Instruct
allenai WildChaa-IN-FFll--eennn Qwe.r.5-2ee...222-.. Instruct
ASSISTANT (Lang: English| Finish: stop)
ASSISTANT (Finish: stop)
As of 2023, Oswald the Lucky Rabbit is now in the public domain in the United
As of 2023, Oswald the Lucky Rabbit, created by Walt Disney and Ub lwerks, is in the public domain. This that anyone character without infringing on copyright laws, providing a wide range of creative opportunities
States, having been created 1927 and original copyright expiring after 95
years.
USER (Lang: English)
for artists and producers.
What the different between tax queen and welfare queen? Is tax queen less
USER
racially/ethnically pejorative term?
What the different between tax queen welfare queen? Is tax queen less
ASSISTANT (Lang: English| Finish: stop)
raciallyfethnicclly pejorative term?
A "welfare queen" typically refers pejorative stereotype a person, often
ASSISTANT (Finish: stop)
of color, who allegedly abuses the welfare system by collecting
The "tax queen" "welfare queen" both used describe individuals who allegedly manipulate the system to their advantage, but they refer
excessive benefits while not working. contrast, a "tax queen" is not commonly used term, but it could potentially refer to someone who takes
different aspects of public assistance and have different connotations.
advantage of tax loopholes or deductions to minimize their

---

tax liability.
Welfare Queen:
Definition: "welfare queen" is derogatory used to describe someone, often woman, who is perceived to be gamnng the welfare system to receive benefits fraudulently excessively.
While "tax queen' may seem like less racially or ethnically pejorative term,
it's essential to consider the context in which it's used. If the term is
The term has a strong negative and often racially charged history, particularly the United States, where it has been used to stereotype and stigmatize poor, often Black, women. gained prominence the 1970s has been associated with racial and class biases.
Connotations:
applied primarily to individuals from marginalized groups, it could still
perpetuate negative stereotypes. However, used to describe individuals of
background who exploit tax laws, it might seen as more neutral. Nonetheless,
Tax Queen:
the term "welfare queen" has well-documented history of being used stigmatize and demean low-income communities, particularly communities of color,
Definition: "tax queen" more recent and less common that typically refers individual, often wealthy person, who uses legal loopholes and strategies to minimize avoid paying taxes.
making more loaded and problematic term.
The term "tax queen" is generally not racially or ethnically charged as "welfare queen." is more likely used financial or political contexts criticize the exploitation of tax laws by the wealthy.
Connotations:
USER (Lang: English)
Is illegal in a British-style constitutional monarchy for monarch use their royal status avoid paying taxes?
Comparison:
Racial and Ethnic Connotations: "Welfare queen" is heavily laden with racial and ethnic stereatypes, particularly targeting minority groups. "Tax queen," while still derogatory, is less likely to carry the same racial overtones is more focused on the manipulation of tax laws by affluent individuals.
ASSISTANT (Lang: English| Finish: stop)
In British-style constitutional monarchy, the monarch's tax complex, While the monarch is not above the law, they may be exempt from certain
"Welfare queen" often used to advocate for stricter welfare regulations and to stigmatize recipients, whereas "tax queen" used to highlight issues of tax fairness and inequality, often in the context of advocating for tax reform.
Social and Political Context:
tax obligations due to historical customary arrangements. However, using
royal deliberately avoid paying taxes would be considered unethical
and patentially illegal. the UK, example, the Queen

---

voluntarily pays
In summary, while both terms are pejorative, "tax queen" generally less
income and capital gains her private income, but the extent of her
racially and ethnically charged compared to "welfare queen."
tax liability not publicly disclosed. monarch were abuse their
USER
position to evade taxes, could be seen as breach of their constitutional
Is illegal British-style constitutional monarchy for a monarch
duties and potentially lead to parliamentary scrutiny even legal action.
their royal status to avoid paying taxes?
ASSISTANT (Finish: stop)
USER (Lang: English)
In British-style constitutional monarchy, monarch is typically subject to the same tax laws as other citizens, although there are some specific
the President of the United States obliged pay taxes?
ASSISTANT (Lang: English| Finish: stop)
The President of the United States not exempt from paying taxes. Like
arrangements and historical precedents. For example, in the United Kingdom, the monarch is technically exempt from some taxes by tradition and historical
U.S. citizens, the President is required income taxes on their earnings, including their presidential salary. President's returns are subject to audit by Internal Revenue Service (IRS), just like any other taxpayer.
privilege, Queen Elizabetth and King Charles have both voluntarily
paid income tax and capital gains tax their personal income and assets since 1993.
In fact, since the 1970s, U.S. presidents have voluntarily released their tax returns, although this legal requirement. This practice is intended to demonstrate transparency and accountability, as the President is public figure with significant influence over the country's tax policies.
However, would generally be considered inappropriate and potentially illegal for monarch to use their royal status to avoid paying taxes that are normally
applicable to all citizens. Any such action could lead legal and
constitutional challenges, well as significant public backlash. The monarch's role uphold the law and serve as a symbol of national
It's worth noting that the President's tax situation can be complex, as they may

---

have various sources of income, such as investments, book royalties, other business interests. Additionally, the President's s tax returss may be subject additional scrutiny due to their public position and potential conflicts interest.
unity, and engaging
in avoidance evasion would undermine this and principles of a constitutional monarchy.
USER
Is the President of the United States obliged to pay taxes?
USER (Lang: English)
ASSISTANT (Finish: stop)
Does the Tenno pay taxes?
Yes, the President af the United States is legally required to pay federal and
ASSISTANT (Lang: English| Finish: stop)
state taxes, just like any other citizen. The U.S. tax system is based on the principle that individuals and entities must and pay taxes on their income, and this includes President. Internal Revenue Code, which governs federal taxation, applies equally to all individuals regardless their position. Presidents are subject to the same tax laws and regulations as other Americans, and they required to report their income and pay applicable taxes.
The Emperor of Japan, also known as the Tenno, has a unique financial situation.
The Imperial Family's expenses are covered by the Japanese government through
the Imperial Households' budget. This budget is used fund the official duties and maintenance of the Imperial Palace, well as personal expenses of the Imperial Family.
terms of taxation, Emperor and Imperial Family do not pay income taxes in the conventional sense, as their income is derived
from the Imperial
USER
Households' budget and is considered part of the national treasury. However,
Does the Tenno pay taxes?
they do pay other types of taxes, such property taxes their private
ASSISTANT (Finish: stop)
The specifics of their tax obligations can be complex, and are subject Japanese tax laws and regulations.
The Emperor of Japan, known as the Tenno, is not required to pay under
Japanese law. This exemption is part of the broader legal and

---

constitutional
USER (Lang: English)
framework that establishes the Emperor's status symbol of the state and the unity of the Japanese people. The Emperor's official duties and the maintenance
Did the General Secretary of the Soviet Union pay taxes?
ASSISTANT (Lang: English Finish: stop)
of the Imperial Household are funded by the government through the annual
the Soviet Union, the General Secretary of Communist Party, who was the de facto head of state and govemment, did not pay taxes the way that ordinary citizens did. The Saviet Union had system where high-ranking
budget, which is approved by the Diet (the national legislature Japan).
Therefore, while the Emperor does not pay personal taxes, the financial support for Imperial Family and their duties indirectly provided by the Japanese taxpayers.
officials, including the General Secretary, received various
perks and benefits, such
exclusive stores, vacation homes, and
other privileges, which
USER
were not subject taxation.
Did the General Secretary of the Soviet Union pay taxes?
These benefits were considered part their compensation package for their
ASSISTANT (Finish: stop)
service to the state and the Communist Party. Additionally, the concept taxes
The General Secretary of the Soviet Union, as the leader of the Communist Party,
understood in capitalist systems did not apply in the same way the Soviet
subject to the Soviet Union's tax laws. However, the tax system the Soviet Union quite different from that capitalist countries. The Soviet
Union, where the state controlled most aspects of the economy.
It's reported that high-ranking officials, including General Secretaries
state-controlled economy, and income was generally lower and more uniform across the population. Taxes were typically lower and
Joseph Stalin, Leonid Brezhnev, and Mikhail Gorbachev, received generous compensation packages that taxed, and they often lived in state-
straightforward, often taking the form of social insurance payments taxes on goods and services.
provided housing and enjoyed other benefits that were not

---

subject to income tax.
The General Secretary, like other citizens, would have paid whatever
The details of their compensation benefits publicly disclosed, and the lack transparency made it difficult to determine exact nature of their tax obligations or exemptions.
required by law. However, due to their high-ranking position, they likely had higher income and access various state-provided benefits and perquisites,
which might have reduced their need for personal expenses. These benefits could include housing, healthcare, travel, which were often provided part of their position rather than as direct income.
USER (Lang: English)
How imperial Chinese taxation system work?
specificity, let's use
the Qing dynasty as reference.
In practice, the financial details of high-ranking Soviet officials, including the General Secretary, were not transparent, and specific records their tax payments widely available. Nonetheless, they were technically subject to the laws other Soviet citizens.
ASSISTANT (Lang: English Finish: stop)
Qing dynasty's taxation system, which lasted from 1644 to 1912, was complex
multifaceted, with various taxes imposed cifferent levels of society.
Here's an overview of how worked:
1. Land Tax (Diding): The primary source revenue for Qing government was land tax, which was based on the amount land owned
USER
productivity.
How did the Imperial Chinese taxation system work? For specificity, let's use
Landowners were required pay proportion of their
crop yields, usually kind (grain) or in silver.
2. Head Tax (Ding): The head was poll tax levied on all adult males, regardless of their social status or occupation. This tax was used to finance local government expenses, such as maintaining irrigation systems, roads, and public buildings.
the Qing dynasty reference.
ASSISTANT (Finish: stop)
During the Qing dynasty (1644-1912), the Chinese taxation system was complex
underwent several reforms the Here the key aspects the Qing dynasty's taxation system:
3. Salt Tax: Qing government monopolized the salt industry imposed
on salt production distribution. Salt merchants
were required
1. Tax (HE, tián fü): This was primary source of

---

purchase licenses from government, which generated significant
revenue for the Qlng government. Landowners were required pay based the size and productivity of their land. The land tax was usually collected in the form rice or other grains, especially the early periods. Later, was often paid in silver.
Customs Duties: The Qing government collected
customs duties imported
and exported goods, including tea, silk, other luxury items.
Commercial Taxes: Merchants traders
2. Poll (T fR, diig shui): This per-capita tax levied on adult
required pay
their businesses, including permits, licenses, and taxes.
males. initially fixed amount, regardiess social status.
Tribute: The Qing government received tributes from vassal states, such
However, the poll tax was often criticized being regressive and unfair the poor. In the 18th century, the Yongzheng Emperor introduced the "Ding-Yin
Mongolia, Tibet, and Korea, the form goods services.
Administration:
The taxation system was administered through network of provincial,
Zhibiao" (ME T AE, tän diing rů mū) reform, which merged poll tax the land
prefectural, and county-level governments, well as local magistrates and tax
making the system more equitable.
collectors. The central government set taxation policies, while local officials
3. Salt Tax li yán shui): The revenue source.
significant
were responsible for collection enforcement.
The government maintained monopoly on salt production and distribution, and merchants who sold salt had to purchase a government- issued license (125), yán
Exemptions and Privileges:
Certain groups, such as Buddhist monks, Taoist priests, and imperial family
yín). The salt tax was often source of corruption, as local officials merchants sometimes colluded evade taxes.
members, exempt from taxation. Additionally, regions, like the
Imperial Hunting Grounds, were also exempt from taxation.
4. Commercial Taxes (19 shäng shui): These included taxes on trade and
Reforms and Changes:
The Qing taxation system underwent changes throughout its
Various fees levied on goods marketplaces, ports, and along
history, including the
introduction taxes, such the likin tax goods in transit), and
major trade routes. The Qing government also imposed customs duties imported
reforms aimed at reducing corruption and increasing

---

efficiency. However, the system remained largely intact until the fall of the Qing dynasty in 1912.
and exported goods.
5. Miscellaneous Taxes: There were numerous other taxes fees, including
Keep in mind that this is simplified overview, and the actual taxation system
those property transactions, funerals, and even the sale of livestock. Local
complex, with regional variations and exemptions. Nonetheless, this
governments often imposed additional levies to fund local projects and administration.
gives you a general idea the Qing dynasty's taxation system worked.
6. Corvée Labor (1811, yáo yi): Although not monetary corvée labor
USER (Lang: English)
required peasants to provide labor for public works projects, building roads, canals, and fortifications. This system was gradually phased out replaced with monetary payments.
[Response must be written as a short blurb.]
What the main
for European mixed
economies and social
democracies?
ASSISTANT (Lang: English| Finish: stop)
7. Reforms and Challenges: The Qing government faced
significant challenges
European mixed economies and social democracies, the main source of revenue
collecting taxes, especially in times natural disasters, rebellion, and economic hardship. The late 19th century saw increasing the tax system due to the need to fund military modernization reforms. The Boxer Rebellion and the subsequent Boxer Indemnity further strained the government's finances.
typically income tax and social insurance contributions from citizens, along
with value-added tax (VAT) on goods services, corporate taxes
businesses. These countries often have progressive tax system, where higher incomes are taxed at higher rate, to fund public services, social welfare programs, and infrastructure development.
The Qing taxation system characterized mix of direct and indirect
USER (Lang: English)
taxes, with strong emphasis land and poll taxes. However, the system often inefficient and corrupt, leading to widespread evasion
[Response must be written as a short blurb.]

---

What is an income tax? Who is paying for what service?
and resistance.
Despite reforms, the system remained significant source of social and
ASSISTANT (Lang: English| Finish: stop)
income type direct tax levied by governments individuals and
economic tension throughout the dynasty.
USER
businesses based on their earnings. In essence, it's fee paid by citizens the government in exchange for public goods and services such education, healthcare, infrastructure, defense, and social welfare programs, The taxpayer
[Response must be written short blurb.]
What is the main source of revenue for European mixed economies social democracies?
(individual business) pays the government for these services, which are then
ASSISTANT (Finish: stop)
In European mixed economies and social democracies, the main of revenue
allocated and distributed benefit the public large.
is typically income tax, followed by value-added tax (VAT) and social security contributions. These taxes fund extensive public services and social welfare
programs, ensuring robust safety net for citizens.
USER
[Response must be written as a short blurb.]
What an income tax? Who paying what service?
ASSISTANT (Finish: stop)
income tax is levied the income earned individuals and
businesses. typically progressive, meaning that rate increases the taxable income increases. Individuals and corporations pay income tax the government, which uses the revenue to fund public services

---

and infrastructure,
such education, healthcare, defense, and social welfare programs.

---

WildChat-SOM
F. Common Pitfalls and Best Practices in Synthetic Data Generation
Synthetic data generation for large language model training presents numerous challenges that can significantly impact downstream performance. Through our experience developing WILDCHAT-500 we identified several critical pitfalls that researchers should avoid when creating large-scale synthetic datasets.
One of the most consequential errors is bias transfer from data-generating models (DGMs). When a DGM has been trained on biased data or exhibits systemic failures in specific domains, these issues propagate into the synthetic dataset and may be further amplified when models are trained on this data. In our work, we observed that using a diverse mix of DGMs helps mitigate this problem by ensuring no single model's biases dominate the dataset. Quantitative bias evaluation should be performed on sample generations before committing to full-scale data generation, as discovering bias issues after generating millions of examples can necessitate discarding entire dataset subsets- tremendously wasteful outcome in terms of both computational resources and research timelines.
Technical configuration issues equally undermine synthetic data quality. Inadequate context window sizing, improper temperature settings, poor prompt engineering, and insufficient runtime validation can all produce fatally flawed datasets. We found that temperature settings substantially influence output diversity and quality-femmeertuures that are too low (i 0.3) lead to repetitive, generic responses while settings that are too high ( 1.2) frequently introduce hallucinations, grammatical errors, and incoherence. During our data generation, we witnessed how improper prompt formatting with missing or inconsistent system instructions caused several models to produce unusable outputs, requiring us restart entire batches. Another critical technical consideration is tokenization differences between models; in our collection process, observed that using identical maximum token limits across architecturally diverse models led to dramatically different response lengths, necessitating model-specific calibration.
Based on our experience, we recommend the following best practices: (1) Conduct small-scale pilot runs with diverse inputs before full production to identify potential issues; (2) Implement comprehensive runtime validation including token count verification, response coherence checks, and completion confirmation; (3) Meticulously document all hyperparameters used in generation across different model architectures; (4) Employ robust error handling and logging systems that can gracefully recover from inference failures without losing progress; and (5) Include a diverse array of DGMs prevent any single model's 's limitations from dominating the resulting dataset. Following these practices can substantially improve synthetic data quality while avoiding costly regeneration cycles that consume valuable computational resources.
31


Reviews:
[{"review_id": "P00uxez0L0", "reviewer": "Reviewer_a9Br", "summary": "The paper proposes a new synthetic chat dataset, WildChat-50M, which consists of generated responses from 50+ open weight models. The authors then created a new SFT datamix, Re-Wild, by combining WildChat-50M with two other datasets (MMLU Auxiliary Train, Tulu 3 Persona Hub Algebra). Main contribution of the paper:\n- Introduction of new datasets WildChat-50M and Re-Wild and their source codes\n- Ablation showing its SFT performance beats other open datamix on 9 benchmarks\n- Analysis on the effect of data generating models (DGM) efficiency and their impact on the synthetic data quality (SDQ), and hypothesis why certain DGMs outperforms the rest on certain benchmarks\n- Misc empirical insights on what tricks matter and what don't, e.g. choice of DGM, diversity of DGM, context window, ...", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.\n\nTo your point about our inability to include RLHF in this work (and therefore to answer your question about whether stronger SFT leads to stronger post-trained models), we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.\n\nWe agree that a section documenting common pitfalls and best practices in synthetic data generation, including configurations that might ruin the entire dataset, would be valuable; we do not include it here because of space constraints, however, if the paper is accepted, we will add such a section to the appendix in the camera-ready version.\n\nWe are happy to confirm that Table 1, Row 1 is in fact not a typo – the name refers to samples with WildChat-1M prompts and Qwen 2.5 72B Instruct responses, following the convention introduced in Sec. 2.2. However, as we only explicitly define naming conventions for models and here implicitly use it to name a dataset, there could be confusion -- we will clarify that the names can apply to datasets generated using DGMs in the camera-ready draft.\n\nThanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.", "claims_and_evidence": "The major claim of the paper:\n- WildChat-50M is a useful dataset, evidenced by its derived SFT datamix, Re-Wild outperforming other open SFT datamix baselines in finetuning llama3 8b.\n- Various factors contributed to its source of effectiveness, including data volume, DGM category, etc.\n- The choice of DGM is critical and highly diversified across different benchmarks.\n\nEvidence: \n\nFigure 1-2 showing that\n- Overall Re-Wild outperforms strongly over other baselines. \n- SFT performance improves as the data volume increases.\n\nTable 2 showing that the choice of DGM differs drastically across benchmarks", "methods_and_evaluation_criteria": "This is an empirical paper. It is well written and straightforward to follow through. The figures and tables are well presented and the introduction, related work provide thorough context on the related literature. While the paper is not technically novel, I'm leaning toward acceptance because\n- It introduces a new SFT dataset that's open source and useful\n- The design of this datamix is backed by extensive study and comprehensive analysis is included to inform the reader of key design choices\n- Efficiency is included and key parameters are transparently disclosed\n\nThoughts on evaluation\n- While the reviewer acknowledges the performance gain after SFT, how well it carries over to RLHF is unknown -- would a stronger SFT base from finetuning on this synthetic data result in a stronger RLHF candidate? In an industry setup, usually SFT is followed by RLHF and the model quality after RLHF is what truly matters.\n\nThoughts on ablations\n- The SFT datamix shows strong performance -- and there might have been considerable iteration cycles before it finally worked. The reviewer imagines the synthetic data generation to not be something straightforward and there are many decisions crucial to the success. Would be helpful to callout any common fallacies and summarize the best practices in synthetic data generation, e.g. is there some config that is easily overlooked but would ruin the entire dataset if not carefully engineered. Such empirical insights would add great benefits to the industrial community.\n\nMinor issues\n- Tab 1 / row 1 -- should it be WildChat-50M instead of WildChat-Q72?", "theoretical_claims": "no theoretical claim was proposed", "experimental_designs_or_analyses": "see above", "relation_to_broader_scientific_literature": "The following are related:\n- LLM post training\n- supervised finetuning\n- synthetic data generation", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "nZum2DuyYi", "reviewer": "Reviewer_86kE", "summary": "Authors are proposing a new synthetic dataset: 'WILDCHAT-50M'\nCompare to other open datasets, 'WILDCHAT-50M' is much larger and includes synthetic data generated from many open source models other than GPT.\n\n-WILDCHAT-50M is the largest public chat dataset to date\n-It includes responses from over 50 different open-weight models\n-A comparative analysis was conducted using this dataset\n-RE-WILD, a public SFT mix, was created and outperformed a recent mixture from Allen AI", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.\n\nYou are correct that \"wehow\" is a typo and should read “we show”. We apologize for the inconvenience, and will remedy this in the camera-ready draft if we are accepted.\n\nYou are also correct that our benchmark suite contains no examples of highly specialized coding or legal reasoning tasks. This exclusion was motivated by our findings about the limited usefulness of generalist chat data for such tasks (see our conclusions at the end of 3.3).\n\nTo your point about our inability to include other post-training approaches in this work, we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.\n\nThanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.", "claims_and_evidence": "Claim - The choice of DATA Generative Model (DGM) significantly impacts downstream model performance on generalist chat benchmarks. Selecting a good DGM can compensate for small dataset size and outperform more complex methods and carefully curated SFT mixes.\nEvidence:  compare the\nperformance of six unique pretrained models from four distinct model families, including Qwen-2.5-72B-Instruct from\nAlibaba, Llama-3.3-70B-Instruct from Meta, Command-RPlus from Cohere, and Jamba-1.5-Mini from AI21. \nBenchmarked on:  MTBench AlpacaEval BBH GPQA MATH MUSR IFEval MMLU Pro MixEval\nShowed that the results have large variance and is unpredictable.\n- Claim: There is no benefit in generating data generation models. Certain DGMs produce higher Synthetic Data Quality (SDQ) due to factors such as: Comprehensiveness, Clarity, Tone, Prompt responsiveness. These factors are highly heritable during the SFT process, even on generalist data.\n- Skills like world knowledge or mathematics are only heritable when data is curated for that specific purpose.\n- Large Language Models (LLMs) exhibit a high degree of similarity in prompt responses, suggesting a subtle distinction between high and low SDQ.", "methods_and_evaluation_criteria": "For benchmarking, author used a few different benchmarks.\n\nMTBench AlpacaEval BBH GPQA MATH MUSR IFEval MMLU Pro MixEval", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiment setups are straight forward. \nTraining framework: Axolotl\nEval framework: Evalchemy\nWith all standard settings", "relation_to_broader_scientific_literature": "Post-training techniques for language models: The paper builds upon prior work on post-training techniques, such as Supervised Fine-Tuning (SFT) , which has been shown to improve the performance of language models. The authors' contribution lies in exploring the effect of different Data Generating Models (DGMs) on SFT.\nImportance of dataset diversity: The paper's focus on creating a large and diverse dataset (WILDCHAT-50M) is in line with previous research highlighting the importance of dataset diversity for training robust language models. The authors' contribution is in providing a standardized benchmark suite that can be used to evaluate the performance of different DGMs.\nComparative analysis of language models: The paper's comparative analysis of different DGMs is similar to previous studies that have compared the performance of various language models on specific tasks . However, the authors' contribution is in providing a comprehensive evaluation of multiple DGMs on a large and diverse dataset.\nOpen science and reproducibility: The paper's commitment to open science and reproducibility is in line with recent efforts to promote transparency and accountability in AI research. The authors' decision to release their data, artifacts, and code publicly aligns with these values.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n- The authors presented a large synthetic dataset that is benefit for the open source community.\n\nWeakness:\n- Most of the claims in the paper are well established by past publications. There is little novelty in the claims.\n- Limited post-training approaches: Only SFT (Supervised Fine-Tuning) was used, and results may differ with other post-training methods.\n- Benchmark suite limitations: The benchmark suite is standardized, balanced, and large, but does not cover all use cases, particularly:\nHighly specialized tasks (e.g., coding, legal reasoning)", "other_comments_or_suggestions": "Check wording:\n\n When we fine-tune Llama-3.1 8B Base on RE-WILD, 'wehow' that our models outperform the SFT mix proposed in Tulu-3\n\nShould be 'we show'? I suggest use grammarly.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fs6JrXZkOi", "reviewer": "Reviewer_hY6u", "summary": "This paper constructed a large set of 50M synthetic conversations by using the initial human prompts from WildChat and various pretrained language models to generate responses and following turns. Based on the data, they further selected a subset and mixed with two other sources of data (MMLU Auxiliary Train and Tulu 3 Persona Hub Algebra) to construct form a dataset for instruction tuning base language models. This work also performed extensive analyses to study the effects of various choices in their dataset construction pipeline, such as whether blending different models helps.\n\n### update after rebuttal\nAuthors mentioned that they'll consider adding a section addressing some of my concerns. I'm leaning positively about this paper and am keeping my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.\n\nRegarding experiments to determine if synthetic data can surpass the performance of smaller-scale, higher-quality human-written chat responses: unfortunately, we know of no such publicly available chat dataset. The original WildChat-1M dataset contained GPT 3.5 and GPT-4 synthetic responses (we did train models on that data; the results of those experiments are available in our artifacts). The closest comparisons of which we are aware are the large-scale post-training runs exclusively on FLAN data described in (https://arxiv.org/abs/2409.15268). FLAN has human prompts and human responses, but is not a chat dataset; that work found that FLAN-trained models significantly underperformed WildChat-trained ones, controlling for dataset size. We agree that more research needs to be done in this area.\n\nAs to whether the primary benefit observed in our mix is due solely to the strong performance of a single DGM (Qwen 2.5 72B), our key contributions are (1) discovering which DGMs tend to produce higher quality responses and (2) providing experimental evidence on why they perform better.\n\nWe agree that we could have more thoroughly explained the contributions of the original WildChat-1M paper, which included the acquisition of the human multi-turn conversations used in this work. We will add more information on this to our camera-ready draft.\n\nFor the camera-ready version, we will also correct the issue of too much white space on page 4.", "claims_and_evidence": "1. A critical question is whether the advantage of increased dataset scale (50x larger than the original WildChat) compensates for the potentially lower quality of synthetically-generated responses compared to human-written responses. The paper does not address whether this large-scale synthetic data genuinely surpasses smaller-scale, higher-quality human-written responses. I strongly suggest the authors conduct experiments to determine if synthetic data can effectively compensate for presumed quality deficits at larger scales. For example, the authors can add a comparison to Figure 2, where in addition to existing bars, add results for simply subsampling original WildChat to 100K, 250K, and 500K, and compare results. I'd hope to see that at smaller scales original data is better, but beyond a certain scale original data is no longer available and synthetic data finally catches up.\n2. Section E Table 7 shows that blending responses from multiple language models yields no benefit (compared to using the strongest teacher). This result appears to undermine the motivation of exploring diverse synthetic responses, raising the question: is the primary benefit observed here due solely to the strong performance of a single DGM (Qwen 2.5 72B)?", "methods_and_evaluation_criteria": "1. Important details about how human turns within multi-turn conversations are generated remain unclear. Since pretrained language models typically generate only assistant responses, it is unclear how human dialogue turns were synthesized, how many turns were included per conversation, and whether such multi-turn interactions improve downstream performance. Clarifying these points would substantially strengthen the paper.\n2. The ReWild dataset, although performing well, incorporates external datasets (MMLU Auxiliary Train, Tulu 3 Persona Hub Algebra), somewhat diluting the primary contribution of the WildChat-50M dataset itself.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "WildChat-50M is a substantial resource, significantly expanding the availability of synthetic conversational data for training or fine-tuning language models, greatly benefiting the research community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "On page 4 there's too much white space.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hX7KjS432B", "reviewer": "Reviewer_uUWy", "summary": "- The paper introduces WildChat-50M, the largest public chat dataset to date, featuring responses from 50+ different open-weight models (0.5B-104B parameters) participating in over 1M multi-turn conversations each.\n\n- The authors created Re-Wild, a new supervised fine-tuning (SFT) data mix that outperforms Allen AI's Tulu-3 mixture while using only 40% as many samples.\n\n- Key findings show that the choice of data generating model (DGM) significantly impacts downstream performance, sometimes more than model size or parameter count.\n\n- The research demonstrates that SFT models inherit stylistic elements and response patterns from their DGMs, with LLM judge preferences transferring from DGM to fine-tuned model.\n\n- Analysis reveals scaling laws for synthetic data, showing consistent performance improvements with larger dataset sizes.\n\n- Technical investigations found high similarity between responses from different models, suggesting LLMs produce more predictable outputs than humans.\n\n- The authors observed that models learn more effectively from DGMs in the same model family (approximate on-policy sampling).\n\n- Blending responses from multiple DGMs showed no performance benefits; individual DGM quality was more important than diversity.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.\n\nTo your point about confidence intervals; we agree that these would be valuable. Therefore, if our paper is accepted, we will add 95% confidence intervals using the normal approximation method to the main paper, where they are absent. We hope that this resolves your concern in this area.\n\nWe agree that direct size comparisons with other datasets should be included. The largest such datasets we are aware of are WildChat-1M and LMSys-Chat-1M. Our dataset is more than 50x larger than either of those datasets. We will include this information in our camera-ready.\n\nWe will improve our explanation of the derivation of chat transcripts by more thoroughly explaining the contributions of the original WildChat-1M, which included the acquisition of the human multi-turn conversations used in this work.\n\nWe agree that our claim about models learning better from same-family DGMs would benefit from more extensive cross-family experiments; we will be happy to reword the claim to be more modest in scope in the camera-ready draft.\n\nThe raw quantitative data used to generate Figures 3-4 will be made available in our repository for the camera ready release. To your point about the selection of which judgments to analyze potentially introducing bias, we agree, which is why we report results over all of MTBench without subselection.\n\nOur heuristic choice of SFT mix is consistent with prior work (Tulu 3); as of right now, automated methods for subselection trail human performance (Tulu 3), and it is a relatively low-cost annotation task for human experts.\n\nWe agree that the response similarity analysis would benefit from a small-scale human study; if our paper is accepted, we will conduct such a study.\n\nWe did conduct experiments varying only the DGM while keeping dataset size fixed; these can be found in our supplementary material ZIP attached to the submission.\n\nSadly, at this time we can offer no theoretical basis for our findings; they are purely empirical. We consider this a useful direction for future work.\n\nWe did perform experiments with Qwen as well as Llama; the results can be found in Appendix Table 6.\n\nTo your point about RLHF; we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.\n\nThanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.", "claims_and_evidence": "**Well-supported claims:**\n- The RE-WILD data mix outperforming Tulu-3 with fewer samples is convincingly demonstrated in Figure 1 and through detailed benchmark comparisons across 9 different benchmarks.\n- The claim that DGM choice significantly impacts downstream performance is well-supported by Table 2, showing substantial variance across different models.\n- The scaling effects shown in Figure 2 provide clear evidence that performance improves with increased dataset size.\n\n**Adequately supported claims:**\n- LLM judge preferences transferring from DGMs to fine-tuned models is supported by Table 4 and word frequency analysis, though the selection of which judgments to analyze could introduce bias.\n\n**Claims needing stronger evidence:**\n- The assertion that Wildchat-50M is the \"largest public chat dataset\" lacks direct size comparisons with other datasets.\n- The derivation of chat transcripts isn't fully explained.\n- The claim about models learning better from same-family DGMs would benefit from more extensive cross-family experiments.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally appropriate for studying synthetic data in post-training, with several strengths and some limitations:\n\n**Strengths:**\n\n- The benchmark selection balances LLM-judge metrics (MTBench, AlpacaEval2, MixEval) with ground-truth benchmarks (MMLU, BBH, GPQA), which mitigates biases inherent to either approach.\n\n- The diverse set of 50+ models spanning different sizes and architectures supports generalizable conclusions about DGM quality.\n\n- Their multi-faceted analysis (scaling laws, DGM comparison, style inheritance) effectively addresses different aspects of synthetic data quality.\n\n**Limitations:**\n\n- The evaluation focuses heavily on Llama-3.1-8B Base as the SFT target, which may not generalize to other model families or sizes.\n\n- The response similarity analysis relies on automated metrics (ROUGE, METEOR) without additional evaluation to validate perceived similarity.\n\n- The set of 9 benchmarks, while diverse, still primarily measures general capabilities rather than fully capturing the range of potential LLM applications.\n\n- Limited analysis of how specific prompt types or domains benefit differently from various DGMs.", "theoretical_claims": "I'm not aware of any theoretical claims made by this paper.", "experimental_designs_or_analyses": "**Re-Wild Data Mix Evaluation:**\n- Sound: Compared against established baselines (Tulu-3, Magpie, Ultrachat) on multiple metrics\n- Issue: The composition of Re-Wild mix (Table 1) was \"chosen heuristically\" rather than through systematic optimization\n\n**DGM Comparison Analysis (Table 2):**\n- Sound: Controlled experiments across 6 models from 4 families (7B-104B parameters)\n- Sound: Comprehensive evaluation across 9 benchmarks\n- Issue: No statistical significance testing for performance differences between models, especially since some numbers seem quite close\n\n**Scaling Experiments (Figure 2):**\n- Sound: Consistent methodology across 100k, 250k, and 500k dataset sizes\n- Sound: Tested with multiple DGMs to validate trends\n- Issue: Did not extend to saturation points\n\n**Style Inheritance Analysis (Table 3):**\n- Sound: Quantitative measurement of stylistic elements using HTML tag frequency\n- Issue: Limited to analyzing only 80 conversations from MTBench\n\n**General Issues:**\n- Limited target model diversity (primarily using Llama-3.1-8B Base)\n- Fixed hyperparameters throughout experiments without extensive tuning\n- Limited evaluation of generalization to different domains or specific prompt types\n- General lack of indicators of statistical significance", "relation_to_broader_scientific_literature": "The paper's contributions connect to several established research threads in the LLM community:\n\n- Extends the original WildChat dataset (Zhao et al., 2024) and parallels LMSys-Chat-1M (Zheng et al., 2024)\n\n- Their DGM choice findings complement recent work on teacher-student model alignment\n\n- Uses MT-Bench (Zheng et al., 2023) while examining biases in LLM judges (Feuer et al., 2024)\n\n- Investigates scaling relationships specifically for synthetic data, complementing general scaling law research", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- **Multi-faceted Analysis:** Comprehensive investigation combining style inheritance, scaling laws, and model comparison in a single study.\n\n- **Effective Visualizations:** Figure 1's spider chart elegantly illustrates comparative model performance across multiple dimensions.\n\n- **Practical Impact:** Re-Wild demonstrates that careful DGM selection can achieve better performance with fewer samples, democratizing high-quality model training.\n\n**Weaknesses:**\n\n- **Limited Theoretical Framework:** The paper lacks a cohesive theoretical model explaining why certain DGMs perform better than others.\n\n- **Narrow Application Focus:** Focuses exclusively on SFT without exploring how findings might extend to RLHF or other post-training techniques.\n\n- **Presentation Issues:** Some figures (like Figure 3-4) would benefit from quantitative scales rather than relative word sizes.\n\n- **Terminology Inconsistency:** The paper introduces several abbreviations and naming conventions that can be difficult to track/understand without reading in depth.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Did you conduct experiments varying only the DGM while keeping dataset size fixed versus varying dataset size while keeping the DGM fixed?\n\n2. What theoretical explanation do you propose for why same-family models perform better as DGMs? Is it architectural similarity, tokenization consistency, or some other factor?\n\n3. Your main experiments use Llama-3.1-8B as the target model. Did you perform any experiments with radically different architecture families (e.g., Mistral, Qwen) as targets to verify your conclusions generalize?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xV4bQaZSfr", "reviewer": "Reviewer_XCnA", "summary": "This paper constructs a larger and more high-quality post-training chat dataset (called WildChat-50M) by getting responses to prompts from more than just one \"data-generating model\" (DGM). The authors get responses to prompts from the WildChat-1M dataset from 50 open-weight models. The dataset contains over 1M multi-turn conversations (2-3 turns on average). The authors curate a supervised fine-tuning dataset from WildChat-50M using a human-in-the-loop strategy and show that when Llama-3.1 8B is SFT-ed on it, it improves on 2 out of 9 benchmarks used for evalaution (length-controlled AlpacaEval and IF eval) over baselines (Tulu 3, Magpie Align, UltraChat). For the other 7/9 benchmarks, the model achieves similar performance as baselines, despite using 40% of the data used for Tulu 3. The authors show some additional empirical results, like that there is a strong effect of which DGM is used to construct the finetuning dataset on downstream performance.\n\n## Update after rebuttal\nMost of my points have been addressed (except the claim in the paper about teaching performance of models with better benchmark performance), but I remain my score as I still believe there are remaining questions about the broader usefulness/quality of rewild as a posttraining mixture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful response. To the best of our ability, given the 5000 character limit, we address your comments and questions below.\n\nWe agree that our claim about SDQ depending primarily on prompt diversity could be worded more carefully. In the camera-ready draft, we will re-scope the claim to be more in line with our findings, e.g., by inserting phrases such as “in our experimental setting”.\n\nWhen we wrote that LLMs “do not share pre- and post-training data”, we meant that they do not entirely share it (that at least some data differs between the models trained); we will add this qualifier to the camera-ready.\n\nTo your point about being upfront about the performance of our mix in the text, the caption of figure 1 reads: “RE-WILD outperforms strong baselines, on average, across nine benchmarks. In particular, it exhibits strong performance on generalist chat and instruction following benchmarks.” We are glad your assessment that this is a strong result agrees with ours, hence, we refer to it as a strong mix.\n\nIn Sec 2.2, we state: Sometimes we will not specify the SFT target model name; in that case, it will always be Llama-3.1-8B-Base := L8B. If you would like us to repeat this information in another section, we can.\n\nIn Appendix D, we write “we have attached several relevant artifacts to this submission which would not have incorporated well into the body of the paper.” The artifacts in question are in the Supplementary Material ZIP attached to our submission.\nHow does Re-Wild do for other base models compared to existing SFT datasets? This would be an interesting ablation – however, to conduct it, we would have had to retrain not only our own model but also every baseline, because the prior work does not. How we trained our baselines could in turn raise its own set of methodological questions. With limited resources, we feel consistency with prior work is sufficient.\n\nHow does it interact with scale? We provide scaling experiments at 100k, 250k, and 500k in our paper (Fig. 2) and we have more in our repository, which we will share as part of the camera-ready.\n\nWhat is the performance improvement over simply using WildChat-1M? The improvement is significant; the results are available in our artifacts. We did not include this in the main paper out of respect for the WildChat-1M authors, who felt it would not be a fair comparison unless we improved their work by sampling every response from GPT-4, which was prohibitively expensive. However, we are prepared to reverse that choice and include it in the appendix if you feel it is vital.\n\nWe cannot release all configs and logs because some of them were corrupted on our cluster. We will release all uncorrupted ones.\nIn the camera-ready draft, we will be happy to remove the word “may” from line 117-118l, change the citation for Cohere command-r plus to the model announcement the Cohere website, and highlight the best score for each group of base LMs in Table 6.\n\nRegarding Figures 3 and 4, we will make available the detailed responses from the LLM judges in our repository associated with the camera-ready release – it is, therefore, a straightforward matter to recover the context around the words, should anyone wish to do so. We note, however, that many of these words are easily interpretable even out of context.\n\nTo clarify the pretrained versus post-trained model variants distinction; all of our DGMs were post-trained. However, many of the post-trained variants started from the same pretrained checkpoint. Therefore, we have 19 unique pre-trained models (each of which is post-trained) and 35 post-trained model variants (with non-unique pre-trained models). We will add this clarification to our camera-ready.\n\n**Why is DGM Q72 outperforming others on Math (table 2) but in Figure 1 the model trained on Re-Wild is not?** This is a good question – the reason is that the baselines in Figure 1 are trained on different data than the models compared in Table 2. In particular, the Tulu 3 SFT mix, which outperforms ReWild SFT mix on MATH, contains significantly more math data in the mix. \n\nWe will update the camera-ready draft to link to Table 3, rows 1:4; thanks for this correction.\n\nThanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.", "claims_and_evidence": "Not all claims made in this submission are supported by clear and convincing evidence. The authors do a lot of experiments, which is good, but the paper would've been stronger if there were less experiments and each experiment was given more attention. Now, for many claims there are confounders that could explain the result. I will detail issues with each claim below.\n\n**Main claim: Re-Wild is a strong SFT mixture**.   \nThe claim that Re-Wild is a strong SFT mixture is based on fine-tuning one model on it and showing it improves over 3 baselines on 2 benchmarks, and stays the same for others or gets worse (for Math and MUSR). While this is still on average an improvement, it would be great to be more upfront about this in the text. The result is still strong in my opinion, given that you outperform on two benchmarks with 40% of the data. This experiment is essentially the only one that gives some insight into the quality of WildChat-50M over other SFT datasets, which leaves many questions open; how does Re-Wild do for other base models compared to existing SFT datasets, how does it interact with scale? Additionally, what is the performance improvement over simply using WildChat-1M? \n\n**Blending DGMs doesn't benefit models**.   \nAlthough you can say that for the two blending runs you do it did not improve performance, you can't claim from that that it won't work in general. For this claim in the main paper: *\"This finding indicates that SDQ depends primarily on prompt diversity, and it is most effective to optimize, rather than generalize, responses\"*, you would require more experiments. All you can say now is that it does not always help to blend different DGMs, but it might for other blends.\n\n**Models with strong benchmark performance are not better teachers for that benchmark**.   \nAgain, there are confounders here, like maybe llama is a better teacher for another model from the same family, not because of benchmark performance (as you mention yourself further down the paper).\n\n**LLMs that do not share pre- and post-training data still have very similar outputs**.   \nHow do you know these randomly selected models do not share pre- and post-training data?", "methods_and_evaluation_criteria": "Most methods and evaluation criteria make sense for the problem at hand, except one:\n\nIn order to investigate the effect of context window you truncate responses from a model with a larger context window. This does not make a lot of sense to me, as the truncated responses might not make sense anymore. Additionally, the average performance over benchmarks is not a good metric to look at in this experiment, and the authors do not report benchmark scores except the aggregate. Perhaps long-context is important for some not other and therefore truncating deteriorates performance on some not others? I would also like to see the full results of that experiment in the main paper as opposed to just the average over benchmarks.", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "As mentioned in the claims section above, the experimental design has flaws, where sometimes claims are made that are not properly supported by the experiments due to confounders being present.", "relation_to_broader_scientific_literature": "The paper positions itself well within literature, but it's sometimes difficult to understand to what extent the contribution is an improvement over very related existing papers (like WildBench-1M), because they are not compared against. Some things can be described clearer w.r.t. prior research, e.g. it would be good to mention more clearly for the first experiment in section 3.1 that the base model trained is always llama 3.1 8B, also for the other baselines like tulu 3 etc (in the caption of Figure 1 for example).", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper contributes an important dataset, which is a strong contribution given that there is a lot of opacity surrounding post-training datasets\n- The paper does a lot of experiments aimed at understanding more about post-training on their dataset.\n\n**Weaknesses**\nAll mentioned above in response to questions.", "other_comments_or_suggestions": "- Your citation for Cohere command-r plus is a policy primer that has not much to do with the model itself and the authors you cite are not from cohere. Should probably just refer to the model announcement on their website.\n\n- Nit: line 117-118l should remove the word \"may\" or change phrasing\n\n- In Appendix D, it would be useful to refer to where those things can be found.\n\n- Figure 3 and 4 are hard to interpret and do not add much, and it's completely unclear what the words meant in the original judgement by the models because the context is gone (as you mention yourself in line 366-375l)\n\n- Table 6; would be useful if you can highlight the best score for each group of base LMs that are finetuned.", "questions_for_authors": "- I'm confused about the pretrained versus post-trained model variants distinction; which models do you consider pretrained? You say you use 19 pretrained models (which I already wonder how they would respond to instructions out of the box unless theyve been trained on some instruction data in pretraining), and then in section 3.2 you call a bunch of post-trained models pretrained?\n\n- Why release most of complete configuration files and not all? (line 124r)\n\n- Why is DGM Q72 outperforming others on Math (table 2) but in Figure 1 the model trained on Re-Wild is not? Or is in table 2 the model trained only on WildChat-50M data and in figure 1 on the full Re-Wild split?\n\n- Styling experiment: you refer to rows 1:4 but its unclear where.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces WildChat-50M, a synthetic dataset generated by 50+ open-weight models, and Re-Wild, a supervised fine-tuning (SFT) datamix that outperforms existing baselines with fewer samples. Key contributions include empirical insights into 
1) impact of data-generating model (DGM) on performances,
2) data scaling
3) models inherit stylistic patterns from DGMs

While reviewers praised the dataset’s scale and practical utility, concerns centered on 
1) Limited comparison to human-written data (e.g., those from WildChat) to validate synthetic data quality.
2) more evidence needed around the claim about mixing DGM, especially when some are shown superior to others as claimed in the paper.
3) lack of details on multi-turn conversation synthesis (human vs. model-generated turns).
4) generalizability of the claims to other model sizes/families. All experiments validating this new synthetic dataset is based on supervised finetuning a llama3 8b model.

WildChat-50M and Re-Wild provide valuable resources for the LLM community. The paper also offers valuable insights into SFT data and DGM choices.

However there are also some concerns (including those highlighted above) not fully addressed and remain open. Given that this work offers a foundation for open research on data drive post training, I'm leaning towards acceptance when all the concerns listed above are fully addressed.
Decision: Accept (poster)