Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
Authors: Yuxin Jiang, Yufei Wang, Chuhan Wu, Xinyi Dai, Yan Xu, Weinan Gan, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
Accepted at: acl25

Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction

Yuxin Jiang1,2*, Yufei Wang3, Chuhan Wu3, Xinyi Dai3, Yan Xu3, Weinan Gan3, Yasheng Wang3, Xin Jiang3, Lifeng Shang3, Ruiming Tang3, Wei Wang1,2
The Hong Kong University of Science and Technology (Guangzhou)1,
The Hong Kong University of Science and Technology2, Huawei Noah’s Ark Lab3
yjiangcm@connect.ust.hk, wang.yufei1@huawei.com, weiwcs@ust.hk

Abstract
The improvement of LLMs’ instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm—Web as Instruction and Web as Response—where each web document is designated as either an instruction or a response to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort.

1 Introduction
Large language models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Dubey et al., 2024) have become integral across a myriad of applications, demonstrating exceptional performance on diverse tasks by effectively following instructions (OpenAI, 2022; Achiam et al., 2023). Their remarkable performance largely stems from supervised fine-tuning (SFT) (Wei et al., 2022; Mishra et al., 2022) on instruction-response pairs. This process empowers LLMs to produce customized outputs when provided with specific instructions, facilitating their adaptation to novel tasks without prior exposure.

A fundamental challenge in advancing the instruction-following capabilities of LLMs lies in the collection of high-quality instruction-tuning (IT) data. Early approaches primarily rely on human experts to manually generate and curate IT data (Wang et al., 2022; Conover et al., 2023), which is both time-intensive and resource-heavy. To mitigate these limitations, Semi-Automated Synthetic Methods (Wang et al., 2023; Taori et al., 2023; Xu et al., 2024a) leverage LLMs to expand small, human-annotated seed datasets using few-shot prompting techniques. While effective, the performance of these methods is highly sensitive to prompt engineering and the careful selection of seed examples (Xu et al., 2024b). More recently, Fully Automated Synthetic Methods, such as WebInstruct (Yue et al., 2024) and instruction back-translation (Li et al., 2024d), have emerged as scalable alternatives that eliminate human involvement by synthesizing IT data based on web-scraped doc-

![Our proposed Web Reconstruction method surpasses previous techniques by being (1) fully automated, eliminating the need for manual intervention or seed data; (2) minimally reliant on assumptions about the structure and content of web documents; and (3) capable of generating high-quality IT data.](page_1012_370_388_312.png)

Figure 1: Our proposed Web Reconstruction method surpasses previous techniques by being (1) fully automated, eliminating the need for manual intervention or seed data; (2) minimally reliant on assumptions about the structure and content of web documents; and (3) capable of generating high-quality IT data.
uments. These methods, however, often operate under strong assumptions about the structure and content of raw web data, such as the availability of explicit question-answer pairs or minimal irrelevant content. Consequently, they can only handle a limited scope of web documents, restricting their diversity and leading to suboptimal performance across various tasks.

To overcome these limitations, we propose Web Reconstruction (WebR)—a novel framework that synthesizes high-quality IT data from raw web documents with minimal assumptions on web and no reliance on human annotations, enabling broader adaptability and improved performance. Unlike backtranslation (Li et al., 2024d), which directly treats web content as a response, or WebInstruct (Yue et al., 2024), which extracts QA pairs, WebR introduces a novel paradigm by conceptualizing web reconstruction as an instruction-tuning data synthesis task. At its core, WebR aims to transform raw, noisy web documents into human-preferred, response-like outputs through a dual-perspective paradigm. Each web document is designated as either an instruction or a response, triggering the reconstruction process: (1) Web as Instruction introduces a first-of-its-kind web rewriting approach in IT data synthesis, where raw web document is concatenated with a synthesized rewrite request to serve as a complete instruction; (2) Web as Response enhances backtranslation by introducing a novel rollout and refinement process, mitigating reliance on strong web content assumptions. Crucially, we show that these two perspectives are complementary (refer to Table 3): Web as Instruction enhances reasoning and understanding tasks, while Web as Response improves instruction-following and question-answering tasks.

We apply WebR to the Llama3-70B-Instruct and GPT-4o-mini models, creating two 100k-sample IT datasets: WebR-Basic and WebR-Pro. To validate their effectiveness, we train various LLMs, including Llama3-8B-base and Qwen2.5-1.5/3/7/14B-base, and evaluate them on over ten widely used benchmarks. Our experiments provide key contributions and insights into IT data synthesis:

• Efficacy: WebR is the first web-based IT synthetic method to consistently surpass current IT datasets with human annotations;

• Compatibility: Merging WebR with existing IT datasets yields further performance gains;

• Data Efficiency: The performance of WebR improves linearly relative to the logarithmic growth of training data;

• Scalability: WebR scales with LLM size, consistently boosting larger models;

• Domain Adaptability: WebR achieves domain adaption by simply adjusting the proportion of source web documents.

2 Related Work

The synthesis of high-quality instruction-tuning (IT) data (Zhou et al., 2023a; Jiang et al., 2023, 2024a; Xu et al., 2024a) can be broadly classified into three main categories:

Human-Crafted Method primarily involves employing professionals to create instructions, as seen in datasets like SUPER-NI (Wang et al., 2022) and DOLLY (Conover et al., 2023). While these datasets offer high-quality content, their size is constrained by the significant costs associated with manual creation. Alternatively, approaches like ShareGPT (Chiang et al., 2023) and WildChat (Zhao et al., 2024) leverage user interaction logs with LLMs to collect human-generated instructions. However, this method risks incorporating toxic or undesirable content (Zhao et al., 2024).

Semi-Automated Synthetic Method uses LLMs to generate synthetic IT datasets by starting with a small set of human-annotated seed data and expanding them through few-shot prompting. Notable methods include Self-Instruct (Wang et al., 2023), Alpaca (Taori et al., 2023), and Evol-Instruct (Xu et al., 2024a). While these techniques enable large-scale data generation, the diversity of the synthesized data is often constrained by the quality and variety of seed examples (Li et al., 2024a).

Fully Automated Synthetic Method utilizes LLMs to synthesize IT data from scratch, drawing from web-scraped documents. For instance, WebInstruct (Yue et al., 2024) extracts question-answer (QA) pairs from web documents to construct instruction-response datasets. Nevertheless, this approach depends on the explicit presence of QA pairs within the raw web corpus, which is not always guaranteed. Similarly, backtranslation (Li et al., 2024d; Nguyen et al., 2024) treats web documents as natural responses and employs LLMs to infer the corresponding latent user instructions.
Figure 2: Overview of the proposed Web Reconstruction (WebR) framework. Leveraging an off-the-shelf LLM, WebR transforms raw web documents into high-quality instruction-response pairs. It strategically assigns each document as either an instruction or a response to trigger the process of web reconstruction.

However, web documents often contain irrelevant content or unsuitable expressions, making them suboptimal as response candidates.

3 Web Reconstruction

Prior fully automated synthetic methods often rely on strong assumptions about the structure and content of raw web documents—such as the presence of explicit question-answer pairs, minimal irrelevant content, or appropriate expressions—necessitating complex preprocessing steps like retrieval and filtering. In contrast, we introduce the Web Reconstruction (WebR) framework, which leverages a powerful, off-the-shelf LLM to overcome these limitations by directly reconstructing unstructured and noisy web content into high-quality, response-like outputs. As shown in Figure 2, WebR comprises two core strategies: (1) Web as Instruction, where raw web content is concatenated with a synthesized rewrite request to serve as a complete instruction, guiding the generation of a reorganized, coherent response; (2) Web as Response, where a latent instruction is inferred by treating raw web content as a response, enabling reconstruction through the LLM’s initial rollout and subsequent refinement. By adopting this dual-branch approach, WebR efficiently generates high-quality instruction-response pairs, ensuring contextually appropriate outputs while eliminating the need for extensive preprocessing.

3.1 Web as Instruction

Raw web documents often contain disorganized or irrelevant information that hinders direct usability. Even when dealing with well-structured content, further refinement is often required to meet human-preferred formats and stylistic conventions. A natural approach to reconstructing web content is to rewrite it according to specific requirements, such as style, format, structure, etc. To ensure diverse and realistic rewriting scenarios, we leverage a powerful LLM to generate a detailed rewrite request tailored to the original document’s content (See prompt in Figure 7). The request, along with the raw web content, are concatenated to form a comprehensive instruction. In addition to whole-document transformations, we further enhance task diversity by randomly (50% probability) generating rewrite requests that target specific sections of the web content rather than the entire document, as shown in Figure 8. This simulates real-world text manipulation scenarios where users may need to
extract and modify only certain portions of a text. The curated instructions are then processed by the LLM to produce reconstructed web content. Notably, the complexity of rewrite requests naturally encompasses various NLP tasks, such as summarization, information extraction, and semantic understanding. Addressing these tasks requires LLM to demonstrate advanced reasoning and comprehension abilities, thereby enhancing its proficiency in instruction-following, contextual understanding, and reasoning (as verified in Table 3).

3.2 Web as Response

Inspired by instruction backtranslation (Li et al., 2024d), we propose an alternative approach to reconstruct web content by treating the web as a response. Specifically, we utilize a LLM to predict a latent instruction for which the raw web content would serve as an ideal response, as illustrated in Figure 9. To further enhance diversity, specific segments of web content are treated as responses (with a 50% probability), as depicted in Figure 10. Unlike traditional back-translation methods, which directly treat latent instructions and raw web content as instruction-response pairs, our approach introduces a two-stage refinement process. First, we generate an initial response by rolling out an LLM prediction for the latent instruction. Next, we refine this response using both the raw web content and the latent instruction to produce a more accurate and comprehensive output, as shown in Figure 11. The initial rollout ensures that the response exhibits human-like fluency and natural language style, while the subsequent refinement step integrates critical information from the raw web, ensuring that the final response is both precise and thorough. This dual-stage process significantly enhances the LLM’s performance in knowledge acquisition and question-answering tasks, as demonstrated by the improvements reported in Table 3. The generated instruction as well as the refined response are finally paired as IT data.

3.3 Dataset Construction Details

Following prior work (Li et al., 2024d; Yue et al., 2024), we construct our dataset by sampling raw web documents from three diverse and representative domains: 70% from the English subset of Common Crawl (Computer, 2023) (general domain), 15% from OpenWebMath (Paster et al., 2024) (math domain), and 15% from GitHub (Computer, 2023) (code domain). To enable large-scale creation of diverse synthetic data for various scenarios, we adopt a persona-driven instruction synthesis strategy inspired by Ge et al. (2024). Initially, an LLM generates personas for the raw web documents (see template in Figure 6), which guide the subsequent instruction synthesis for our proposed Web Reconstruction process. The ratio of Web as Instruction to Web as Response is set to 2:1, following insights from the ablation study presented in Table 3. To enhance diversity and eliminate redundancy, we apply MinHash (Broder, 1997) deduplication based on n-gram features of instructions. We configure the signature size to 128 and the similarity threshold to 0.7. The final synthesized dataset comprises 100,000 instruction-response pairs.

To evaluate the effectiveness of WebR in generating high-quality IT datasets, we use WebR to construct datasets with two LLMs: the open-source Llama3-70B-Instruct (Dubey et al., 2024) (temperature=0.6, top-p=0.9) and the proprietary GPT-4o-mini (Achiam et al., 2023) (temperature=0.7, top-p=1.0). The resulting datasets, WebR-Basic (from Llama3) and WebR-Pro (from GPT-4o-mini), differ in their generative capabilities and quality. A comparative analysis of the average token lengths is presented in Appendix C, while a detailed cost analysis of WebR is provided in Appendix D. Notably, the overall expenditure for calling GPT-4o-mini API is $38.57.

4 Experimental Setup

4.1 Baselines

We compare the family of IT datasets generated by WebR with ten state-of-the-art (SOTA) open-source IT datasets, categorized as follows: (1) Human-crafted data: ShareGPT (Chiang et al., 2023) and WildChat (Zhao et al., 2024) are exemplary human-written datasets containing 112K and 652K high-quality multi-round conversations between humans and GPT, respectively. (2) Semi-automated synthetic data: Alpaca (Taori et al., 2023) and Evol-Instruct (Xu et al., 2024a) represent widely-used synthetic datasets generated with semi-automated techniques. (3) Mixed data: Tulu V2 Mix (Ivison et al., 2023) and Open-Hermes 2.5 (Teknium, 2023) are crowd-sourced datasets that aggregate diverse open-source IT datasets, featuring 326K and 1M conversations, respectively. (4) Fully automated synthetic data: Magpie (Xu et al., 2024b) synthesizes IT data by prompting Llama3-70B-Instruct with its chat tem-
<table>
  <tr>
    <th>IT Data</th>
    <th>#Data</th>
    <th>Human Effort</th>
    <th>Response Generator</th>
    <th>Alpaca Eval 2</th>
    <th>Arena Hard</th>
    <th>MT Bench</th>
    <th>MT Pr. (S.)</th>
    <th>MT Ins. (S.)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>None (w/o fine-tuning)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.18</td>
    <td>0.31</td>
    <td>1.78</td>
    <td>16.26</td>
    <td>18.01</td>
    <td>7.31</td>
  </tr>
  <tr>
    <td>ShareGPT</td>
    <td>112k</td>
    <td>High</td>
    <td>ChatGPT</td>
    <td>9.89</td>
    <td>6.49</td>
    <td>6.34</td>
    <td>38.52</td>
    <td>42.26</td>
    <td>22.70</td>
  </tr>
  <tr>
    <td>WildChat</td>
    <td>652k</td>
    <td>High</td>
    <td>GPT-3.5 & 4</td>
    <td>14.62</td>
    <td>8.73</td>
    <td>6.60</td>
    <td>39.53</td>
    <td>45.66</td>
    <td>23.03</td>
  </tr>
  <tr>
    <td>Tutu V2 Mix</td>
    <td>326k</td>
    <td>Mid</td>
    <td>Mix</td>
    <td>9.91</td>
    <td>5.41</td>
    <td>5.76</td>
    <td>37.69</td>
    <td>41.05</td>
    <td>19.96</td>
  </tr>
  <tr>
    <td>OpenHermes 2.5</td>
    <td>1M</td>
    <td>Mid</td>
    <td>Mix</td>
    <td>12.89</td>
    <td>8.20</td>
    <td>6.51</td>
    <td>38.82</td>
    <td>43.52</td>
    <td>21.99</td>
  </tr>
  <tr>
    <td>Alpaca</td>
    <td>52k</td>
    <td>Low</td>
    <td>Davinci-003</td>
    <td>4.21</td>
    <td>1.24</td>
    <td>3.75</td>
    <td>20.21</td>
    <td>23.56</td>
    <td>10.59</td>
  </tr>
  <tr>
    <td>Evol Instruct</td>
    <td>143k</td>
    <td>Low</td>
    <td>ChatGPT</td>
    <td>7.19</td>
    <td>5.58</td>
    <td>5.77</td>
    <td>39.00</td>
    <td>44.25</td>
    <td>20.36</td>
  </tr>
  <tr>
    <td>WebInstruct</td>
    <td>100k</td>
    <td>No</td>
    <td>Llama3-70B</td>
    <td>3.43</td>
    <td>1.69</td>
    <td>5.35</td>
    <td>18.99</td>
    <td>20.56</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Backtranslation</td>
    <td>100k</td>
    <td>No</td>
    <td>Llama3-70B</td>
    <td>5.24</td>
    <td>2.81</td>
    <td>3.74</td>
    <td>26.85</td>
    <td>29.61</td>
    <td>13.65</td>
  </tr>
  <tr>
    <td>DoG-Instruct</td>
    <td>100k</td>
    <td>No</td>
    <td>Llama3-70B</td>
    <td>11.75</td>
    <td>8.07</td>
    <td>5.92</td>
    <td>36.60</td>
    <td>41.87</td>
    <td>20.84</td>
  </tr>
  <tr>
    <td>Magpie</td>
    <td>100k</td>
    <td>No</td>
    <td>Llama3-70B</td>
    <td>23.62</td>
    <td>13.98</td>
    <td>6.26</td>
    <td>33.83</td>
    <td>43.07</td>
    <td>24.15</td>
  </tr>
  <tr>
    <td>WebR-Basic</td>
    <td>100k</td>
    <td>No</td>
    <td>Llama3-70B</td>
    <td><b>25.33</b></td>
    <td><b>16.50</b></td>
    <td><b>6.95</b></td>
    <td><b>41.40</b></td>
    <td><b>50.69</b></td>
    <td><b>28.17</b></td>
  </tr>
  <tr>
    <td>IT Mix</td>
    <td>100k</td>
    <td>Mid</td>
    <td>GPT-4o-mini</td>
    <td>30.39</td>
    <td>28.03</td>
    <td>7.36</td>
    <td>43.30</td>
    <td>47.38</td>
    <td>31.29</td>
  </tr>
  <tr>
    <td>Magpie</td>
    <td>100k</td>
    <td>No</td>
    <td>GPT-4o-mini</td>
    <td>32.61</td>
    <td>27.97</td>
    <td>7.26</td>
    <td>36.81</td>
    <td>45.07</td>
    <td>29.95</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td>100k</td>
    <td>No</td>
    <td>GPT-4o-mini</td>
    <td><b>34.36</b></td>
    <td><b>31.10</b></td>
    <td><b>7.57</b></td>
    <td><b>43.79</b></td>
    <td><b>51.76</b></td>
    <td><b>33.71</b></td>
  </tr>
  <tr>
    <td>(IT + WebR-Pro) Mix</td>
    <td>100k</td>
    <td>Mid</td>
    <td>GPT-4o-mini</td>
    <td>35.00</td>
    <td>34.23</td>
    <td>7.50</td>
    <td>48.06</td>
    <td>53.23</td>
    <td>35.60</td>
  </tr>
  <tr>
    <td>(IT + WebR-Pro) Merge</td>
    <td>200k</td>
    <td>Mid</td>
    <td>GPT-4o-mini</td>
    <td><b>35.40</b></td>
    <td><b>35.12</b></td>
    <td><b>7.59</b></td>
    <td><b>49.72</b></td>
    <td><b>53.97</b></td>
    <td><b>36.36</b></td>
  </tr>
</table>

Table 1: Instruction-following performance comparison of various instruction-tuning (IT) data, based on Llama3-8B.

plate, from which we sample 100k examples. To ensure a fair and controlled comparison, we reproduce several representative web-based IT synthesis methods—namely WebInstruct (Yue et al., 2024), Backtranslation (Li et al., 2024d), and DoG-Instruct (Chen et al., 2024)—using the same source web data as our proposed WebR. All methods are implemented based on the LLaMA3-70B Instruct model, thereby aligning model capacity and input sources across approaches.

4.2 Models and Training Settings

For instruction tuning (IT), we train Llama3-8B-base (Dubey et al., 2024) and Qwen2.5-1.5/3/7/14B-base (Qwen Team, 2024) on various IT datasets. For each IT dataset, we fine-tune models with five different random seeds and report the average performance. We adhere to the official instruction templates provided by each model. To ensure a fair comparison, we use consistent training hyper-parameters across different baseline datasets. The comprehensive implementation details are listed in Appendix A.

4.3 Evaluation Benchmarks and Metrics

We evaluate the performance of the fine-tuned models using four widely adopted instruction-following benchmarks: AlpacaEval 2 (Li et al., 2023), Arena-Hard (Li et al., 2024c), MT-Bench (Zheng et al., 2023), and IFEval (Zhou et al., 2023b). For AlpacaEval 2, we report the length-controlled win rate (LC), which ensures robustness against verbosity. For Arena-Hard, we report the win rate (WR) against the baseline model. For MT-Bench, we provide the average score, using GPT-4-turbo as the evaluation judge. For IFEval, we report two metrics: prompt-level strict accuracy (\( Pr.\ (S.) \)) and instruction-level strict accuracy (\( Ins.\ (S.) \)). More evaluation details are listed in Appendix B.

5 Experimental Results

5.1 Main Results

WebR Outperforms Existing Baselines. Table 1 highlights the performance of Llama3-8B-base fine-tuned with datasets generated by WebR, compared to those fine-tuned with baseline datasets. A general trend emerges: IT datasets requiring higher human effort tend to exhibit better performance than those with lower or no human effort. Nevertheless, our WebR-Basic, which entirely eliminates human effort in dataset creation, significantly and consistently surpasses the SOTA Magpie dataset across all four benchmarks with a \( 16.65\% \) average improvement. To ensure a fair and more challenging comparison, we deduplicated and randomly sampled 100k instructions from baseline datasets of varying human effort levels (high, mid, and low) and generated responses using GPT-4o-mini, naming this synthesized strong baseline "IT Mix." We also generate responses using GPT-4o-mini for Magpie and compare with our proposed method. Even under the same response generator, WebR-Pro consistently outperforms IT Mix and Magpie by \( 7.73\% \) and \( 12.55\% \), respectively. These results validate that datasets generated by WebR possess
<table>
  <tr>
    <th>IT Data</th>
    <th>MMLU</th>
    <th>ARC</th>
    <th>WinoGrande</th>
    <th>MATH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>None (w/o fine-tuning)</td>
    <td>60.56</td>
    <td>73.52</td>
    <td>52.14</td>
    <td>19.62</td>
    <td>56.16</td>
    <td>39.08</td>
    <td>50.18</td>
  </tr>
  <tr>
    <td>WildChat</td>
    <td>58.46</td>
    <td>72.62</td>
    <td>49.43</td>
    <td>19.34</td>
    <td>60.25</td>
    <td>42.55</td>
    <td>50.44</td>
  </tr>
  <tr>
    <td>OpenHermes 2.5</td>
    <td>60.08</td>
    <td>75.65</td>
    <td>51.22</td>
    <td>24.18</td>
    <td>64.70</td>
    <td>44.43</td>
    <td>53.38</td>
  </tr>
  <tr>
    <td>Magpie</td>
    <td>58.58</td>
    <td>71.53</td>
    <td>51.93</td>
    <td>16.12</td>
    <td>57.39</td>
    <td>40.85</td>
    <td>49.40</td>
  </tr>
  <tr>
    <td>WebR-Basic</td>
    <td>60.85</td>
    <td>76.27</td>
    <td>52.91</td>
    <td>20.28</td>
    <td>55.57</td>
    <td>40.10</td>
    <td>51.00</td>
  </tr>
  <tr>
    <td>IT Mix</td>
    <td>57.44</td>
    <td>73.56</td>
    <td>50.36</td>
    <td>22.00</td>
    <td>61.87</td>
    <td>45.12</td>
    <td>51.73</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td><b>61.15</b></td>
    <td>74.92</td>
    <td><b>53.20</b></td>
    <td>24.94</td>
    <td>60.69</td>
    <td>48.73</td>
    <td>53.94</td>
  </tr>
  <tr>
    <td>(IT + WebR-Pro) Mix</td>
    <td>60.69</td>
    <td><b>77.63</b></td>
    <td>50.67</td>
    <td>26.34</td>
    <td>64.90</td>
    <td><b>50.61</b></td>
    <td>55.14</td>
  </tr>
  <tr>
    <td>(IT + WebR-Pro) Merge</td>
    <td>61.02</td>
    <td>76.27</td>
    <td>52.72</td>
    <td><b>28.36</b></td>
    <td><b>66.41</b></td>
    <td><b>50.61</b></td>
    <td><b>55.90</b></td>
  </tr>
</table>

Table 2: Performance comparison of downstream tasks (Knowledge, Reasoning, Math, Code) based on Llama3-8B.

<table>
  <tr>
    <th>Setting</th>
    <th>Alpaca Eval 2</th>
    <th>MT Bench</th>
    <th>IFEval Pr. (S)</th>
    <th>Avg.</th>
    <th>MMLU</th>
    <th>ARC</th>
    <th>MATH</th>
    <th>HumanEval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td>34.17</td>
    <td>7.50</td>
    <td>43.55</td>
    <td><b>28.41</b></td>
    <td>61.15</td>
    <td>74.92</td>
    <td>24.94</td>
    <td>48.73</td>
    <td><b>52.43</b></td>
  </tr>
  <tr>
    <td>-w/o Persona</td>
    <td>33.30</td>
    <td>6.93</td>
    <td>44.69</td>
    <td>28.31</td>
    <td>60.98</td>
    <td>74.58</td>
    <td>24.03</td>
    <td>48.50</td>
    <td>52.02</td>
  </tr>
  <tr>
    <td>-w/o Part</td>
    <td>33.89</td>
    <td>7.53</td>
    <td>42.60</td>
    <td>28.01</td>
    <td>61.05</td>
    <td>72.53</td>
    <td>22.73</td>
    <td>48.41</td>
    <td>51.18</td>
  </tr>
  <tr>
    <td>-w/o Refinement</td>
    <td>31.61</td>
    <td>7.42</td>
    <td>44.73</td>
    <td>27.92</td>
    <td>59.83</td>
    <td>74.92</td>
    <td>24.36</td>
    <td>48.61</td>
    <td>51.93</td>
  </tr>
  <tr>
    <td>-w/o MinHash</td>
    <td>32.43</td>
    <td>7.29</td>
    <td>43.02</td>
    <td>27.58</td>
    <td>60.69</td>
    <td>74.92</td>
    <td>24.82</td>
    <td>47.15</td>
    <td>51.90</td>
  </tr>
  <tr>
    <th colspan="10">Ratio of Web as Instruction to Web as Response (2 : 1 in WebR)</th>
  </tr>
  <tr>
    <td>1 : 0</td>
    <td>29.15</td>
    <td>7.10</td>
    <td>39.56</td>
    <td>25.27</td>
    <td>58.79</td>
    <td>74.58</td>
    <td>25.74</td>
    <td>50.00</td>
    <td>52.28</td>
  </tr>
  <tr>
    <td>1 : 1</td>
    <td>33.16</td>
    <td>7.39</td>
    <td>43.26</td>
    <td>27.94</td>
    <td>60.60</td>
    <td>73.22</td>
    <td>25.18</td>
    <td>48.78</td>
    <td>51.95</td>
  </tr>
  <tr>
    <td>1 : 2</td>
    <td>32.99</td>
    <td>7.33</td>
    <td>42.85</td>
    <td>27.72</td>
    <td>57.76</td>
    <td>72.61</td>
    <td>25.26</td>
    <td>50.00</td>
    <td>51.41</td>
  </tr>
  <tr>
    <td>0 : 1</td>
    <td>33.41</td>
    <td>6.68</td>
    <td>42.54</td>
    <td>27.54</td>
    <td>52.68</td>
    <td>72.90</td>
    <td>23.30</td>
    <td>46.95</td>
    <td>48.96</td>
  </tr>
</table>

Table 3: Ablation study based on Llama3-8B.

superior quality, enabling significantly enhanced instruction-following performance.

Compatibility of WebR. To explore the potential synergy between WebR and existing datasets, we merged IT Mix and WebR-Pro using two strategies: (1) random sampling of 50k data points from each dataset and (2) direct concatenation. As shown in Table 1, both merged datasets deliver further performance improvements over their individual components, establishing new SOTA results. This can be attributed to the complementary strengths of the datasets: IT Mix offers broader data coverage, while WebR-Pro provides higher quality and more challenging instructions, as evidenced in Figure 3.

Performance on Downstream Benchmarks. We evaluate the impact of various instruction-tuning datasets on downstream task performance across multiple domains\footnote{Evaluation settings are aligned with https://opencompass.org.cn.}: (1) Knowledge: MMLU (Hendrycks et al., 2021a); (2) Reasoning: ARC (Clark et al., 2018) and WinoGrande (Sakaguchi et al., 2019); (3) Math: MATH (Hendrycks et al., 2021b) and GSM8K (Cobbe et al., 2021); (4) Code: HumanEval (Chen et al., 2021). As shown in Table 2, models fine-tuned on the WebR datasets outperform those trained on other baselines, demonstrating their effectiveness in improving generalization across diverse downstream tasks, especially in challenging benchmarks like ARC and WinoGrande. Furthermore, the combination of WebR-Pro and IT Mix further validates the complementary strengths of WebR data in aligning models with complex task requirements.

5.2 Ablation Study

Table 3 compares the LLM performance using different settings to construct WebR-Pro.
• w/o Persona: removing the author’s persona information during instruction generation leads to performance declines across almost all benchmarks.
• w/o Part: creating instructions solely from the entire web content, rather than using specific parts, causes notable performance degradation, particularly on IFEval and reasoning-intensive tasks like ARC and MATH.
• w/o Refinement: skipping the refinement step for Web as Response—by directly adopting the rollout response as the final output—results in a substantial drop in instruction-following performance.
• w/o MinHash: eliminating MinHash-based
<table>
  <tr>
    <th>IT Data</th>
    <th>Human Effort</th>
    <th>Avg. Score</th>
    <th>Diversity</th>
  </tr>
  <tr>
    <td>WildChat</td>
    <td>high</td>
    <td>23.03</td>
    <td><b>0.93</b></td>
  </tr>
  <tr>
    <td>OpenHermes</td>
    <td>mid</td>
    <td>21.99</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>Evol Instruct</td>
    <td>low</td>
    <td>20.36</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>WebInstruct</td>
    <td>no</td>
    <td>9.79</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Magpie</td>
    <td>no</td>
    <td>24.15</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>WebR-Basic</td>
    <td>no</td>
    <td>28.17</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td>no</td>
    <td><b>33.58</b></td>
    <td><b>0.93</b></td>
  </tr>
</table>

Table 4: Comparison of embedding diversity.

![Bar chart showing statistics on instruction quality and difficulty](page_184_370_495_312.png)

Figure 3: Statistics of instruction quality and difficulty.

deduplication decreases performance across all benchmarks, highlighting the importance of maintaining dataset diversity.

*Ratio of Web as Instruction to Web as Response*: varying the ratio of *Web as Instruction* to *Web as Response* data synthesis reveals that each component contributes uniquely to model capabilities. Specifically, *Web as Instruction* enhances reasoning and understanding tasks (e.g., ARC and MATH), while *Web as Response* primarily improves instruction-following and question-answering tasks (e.g., IFEval and AlpacaEval 2). The optimal balance is achieved at a ratio of 2:1, which delivers the best overall performance.

6 Analysis

6.1 Dataset Analysis of WebR

Diversity. We utilize a quantitative measure of diversity: (1) we randomly sample \( N = 10,000 \) instructions from each dataset and encode them using the all-mpnet-base-v2\(^2\) embedding model; (2) we compute the average cosine similarity between all embedding pairs and define embedding diversity as \( 1 - \frac{1}{C(N,2)} \sum_{i<j} \cos(e_i, e_j) \), where higher values indicate greater diversity. Table 4 demonstrates that WebR-Pro achieves the highest diversity score (0.93), matching that of WildChat, which involves high human effort. Notably, WebR-Pro surpasses all other datasets—including those requiring human annotation like OpenHermes (0.87) and Evol Instruct (0.88)—indicating its strong capability to generate diverse instructions automatically. Furthermore, it outperforms previous automatic baselines such as WebInstruct (0.84) and Magpie (0.92), highlighting its effectiveness in promoting diversity without human intervention.

Quality and Difficulty. Following Magpie (Xu et al., 2024b), we use the Qwen2.5-72B-Instruct model to evaluate the quality and difficulty of each instruction, categorizing them into five levels. As depicted in Figure 3, synthetic data generally demonstrates higher quality and greater difficulty compared to human-crafted instructions. In particular, WebR-Basic exhibits superior distributions in both quality and difficulty metrics, surpassing existing baselines in these aspects.

6.2 Data Efficiency of WebR

Figure 4 illustrates the impact of training data scale on model performance. The results clearly underscore the superior data efficiency of WebR-Pro compared to IT Mix: (1) With only 10k training samples, WebR-Pro achieves a striking **40.26%** performance improvement over IT Mix, highlighting its exceptional capability to elicit latent potential from LLMs even with limited data. (2) WebR-

![Line chart showing the impact of training data scale on average instruction-following performance](page_728_120_377_246.png)

Figure 4: The impact of training data scale on the average instruction-following performance.

\footnotetext{2https://huggingface.co/sentence-transformers/all-mpnet-base-v2}
<table>
  <tr>
    <th>Base LLM</th>
    <th>IT Data</th>
    <th>AlpacaEval 2</th>
    <th>Arena-Hard</th>
    <th>MT-Bench</th>
    <th>IFEval/Pr. (S)</th>
    <th>IFEval/Ins. (S)</th>
  </tr>
  <tr>
    <td rowspan="2">Qwen2.5-1.5B</td>
    <td>IT Mix</td>
    <td>10.98</td>
    <td><b>15.10</b></td>
    <td>6.03</td>
    <td>29.57</td>
    <td>33.27</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td><b>11.00 (+0.02)</b></td>
    <td>14.03 (-1.07)</td>
    <td>5.92 (-0.11)</td>
    <td><b>29.57 (+0.00)</b></td>
    <td>32.16 (-1.11)</td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2.5-3B</td>
    <td>IT Mix</td>
    <td><b>22.36</b></td>
    <td>26.54</td>
    <td>6.95</td>
    <td><b>43.07</b></td>
    <td><b>44.73</b></td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td>22.29 (-0.07)</td>
    <td><b>28.13 (+1.59)</b></td>
    <td><b>7.03 (+0.08)</b></td>
    <td>42.38 (-0.69)</td>
    <td>44.71 (-0.02)</td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2.5-7B</td>
    <td>IT Mix</td>
    <td>32.59</td>
    <td>45.10</td>
    <td>7.45</td>
    <td>49.35</td>
    <td>52.68</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td><b>34.90 (+2.31)</b></td>
    <td><b>45.66 (+0.56)</b></td>
    <td><b>7.62 (+0.17)</b></td>
    <td><b>50.55 (+1.20)</b></td>
    <td><b>53.35 (+0.67)</b></td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2.5-14B</td>
    <td>IT Mix</td>
    <td>42.07</td>
    <td>59.00</td>
    <td>8.10</td>
    <td>58.04</td>
    <td>60.63</td>
  </tr>
  <tr>
    <td>WebR-Pro</td>
    <td><b>46.19 (+4.12)</b></td>
    <td><b>62.13 (+2.13)</b></td>
    <td><b>8.39 (+0.29)</b></td>
    <td><b>60.23 (+2.19)</b></td>
    <td><b>64.88 (+4.25)</b></td>
  </tr>
</table>

Table 5: Performance comparison across varied scales of base LLMs.

<table>
  <tr>
    <th>Data Proportion</th>
    <th>AlpacaEval 2</th>
    <th>MATH</th>
    <th>HumanEval</th>
    <th>MedQA</th>
    <th>FinBen</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>IT Mix</td>
    <td>30.19</td>
    <td>22.00</td>
    <td>45.12</td>
    <td>38.88</td>
    <td>29.20</td>
    <td>33.08</td>
  </tr>
  <tr>
    <td>WebR-Pro (4.7 gen : 1 math : 1 code)</td>
    <td>34.17</td>
    <td>24.94</td>
    <td>48.73</td>
    <td>47.31</td>
    <td>29.56</td>
    <td>36.94</td>
  </tr>
  <tr>
    <td>- 1 gen</td>
    <td>34.40</td>
    <td>22.52</td>
    <td>44.78</td>
    <td>44.94</td>
    <td>28.97</td>
    <td>35.12</td>
  </tr>
  <tr>
    <td>- 1 gen : 1 math</td>
    <td>34.25</td>
    <td><b>28.09</b></td>
    <td>48.23</td>
    <td>46.59</td>
    <td>29.77</td>
    <td>37.39</td>
  </tr>
  <tr>
    <td>- 1 gen : 1 math : 1 code</td>
    <td>34.59</td>
    <td>27.10</td>
    <td><b>51.39</b></td>
    <td>46.83</td>
    <td>29.34</td>
    <td><b>37.85</b></td>
  </tr>
  <tr>
    <td>- 1 gen : 1 math : 1 code : 1 med</td>
    <td>32.75</td>
    <td>26.22</td>
    <td>49.68</td>
    <td><b>49.98</b></td>
    <td>29.01</td>
    <td>37.53</td>
  </tr>
  <tr>
    <td>- 1 gen : 1 math : 1 code : 1 med : 1 fin</td>
    <td>33.03</td>
    <td>25.38</td>
    <td>48.17</td>
    <td>45.64</td>
    <td><b>30.22</b></td>
    <td>36.49</td>
  </tr>
</table>

Table 6: Domain adaptation based on Llama3-8B, with the domain improvements marked in green.

Pro exhibits a more consistent and pronounced linear performance increase with respect to the logarithmic growth in training data, consistently outperforming IT Mix across all data scales. These results strongly validate the efficacy of WebR in efficiently leveraging training data to unlock and enhance the capabilities of LLMs.

6.3 Scalability of WebR

Table 5 highlights the impact of base LLM scale on the performance of our proposed WebR method. While WebR-Pro slightly underperforms IT Mix at the 1.5B model scale, its advantages become increasingly pronounced as the model size grows. For instance, WebR-Pro achieves an average performance improvement of 2.86% over IT Mix with Qwen2.5-7B and an even more substantial improvement of 5.55% with Qwen2.5-14B. These results suggest that the advanced synthesis paradigm of WebR better aligns with larger models’ capacity to capture complex patterns and utilize reasoning-intensive data. In contrast, smaller models with limited capacity may struggle to fully exploit WebR’s potential.

6.4 Domain Adaptability of WebR

We explore the potential of our proposed WebR framework for domain adaptation by simply adjusting the proportion of source web documents. Starting with general-domain content, we progressively add domain-specific materials from math, code, medicine, and finance, assessing performance across relevant benchmarks. For the medical and financial domains, we utilize raw web documents from IndustryCorpus2 (Shi et al., 2024), and evaluate using MedQA (Jin et al., 2021) and FinBen (Xie et al., 2024) benchmarks. As shown in Table 6, WebR demonstrates strong adaptability across domains. Compared to the IT Mix baseline, incorporating domain-specific data consistently improves performance, with math and code data yielding significant gains in MATH (28.09) and HumanEval (51.39), and medical and financial domains showing strong results on MedQA (49.98) and FinBen (30.22). These results highlight WebR’s ability to incorporate specialized knowledge while maintaining competitive general-domain performance. Furthermore, the process of collecting domain-specific web documents is straightforward, underscoring WebR’s practicality.

7 Conclusion

In this paper, we present Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) datasets. Harnessing the richness of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm—Web as Instruction and Web as Response—where each web document is des-
gnated as either an instruction or a response to trigger the reconstruction process. Extensive experiments show that WebR-generated datasets consistently outperform state-of-the-art baselines across four instruction-following benchmarks and six diverse downstream tasks. Furthermore, WebR exhibits exceptional compatibility, data efficiency, and scalability with existing datasets, underscoring its potential as a versatile tool for advancing instruction-driven LLM training.

Limitations

While WebR can already obtain satisfactory performance, there are several areas for improvement and future exploration. Firstly, the current implementation of WebR focuses on single-turn data synthesis. Expanding this framework to support multi-turn conversations could further enhance its applicability to complex, interactive tasks. Secondly, due to constraints in time and computational resources, the size of the constructed WebR-Basic and WebR-Pro datasets is currently limited to 100k samples. However, given the vast availability of web documents—numbering in the trillions—the WebR framework has significant potential for scaling to create large-scale IT datasets, which could further boost performance. Thirdly, WebR does not incorporate advanced data selection techniques, such as Instruction Following Difficulty (IFD) (Li et al., 2024b), as part of its post-processing pipeline. Incorporating such techniques could refine data quality and further enhance the instruction-following capabilities of LLMs (Jiang et al., 2024b). Finally, advanced post-training techniques such as reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Huang et al., 2025) and direct preference optimization (DPO) (Rafailov et al., 2023; Jiang et al., 2025) can be applied to the synthesized WebR data to further enhance alignment performance, which we leave for future work.

Ethics Statement

This study adheres strictly to the ethical principles established by the research community. The utilized IT datasets are reported to be safe and free of content that may contain discrimination, personally identifiable information, or any other undesirable behaviors. We have meticulously designed and curated our instructions for LLMs to ensure that all tasks are restricted to web reconstruction. This approach minimizes the risk of generating content that could raise ethical concerns.

Acknowledgments

Wei Wang was supported by the Guangdong Provincial Key Laboratory of Integrated Communication, Sensing, and Computation for Ubiquitous Internet of Things (Grant No. 2023B1212010007), the Guangzhou Municipal Science and Technology Project (Grant Nos. 2023A03J00003, 2023A03J0013, and 2024A03J0621), and the Institute of Education Innovation and Practice Project (Grant Nos. G01RF000012 and G01RF000017).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder,
Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.

Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. 2024. DoG-instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4125–4135, Mexico City, Mexico. Association for Computational Linguistics.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewon Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. NeurIPS.

Bo Huang, Yuxin Jiang, Mingyang Chen, Yi Wang, Hongyang Chen, and Wei Wang. 2025. When evolution strategy meets language models tuning. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5333–5344, Abu Dhabi, UAE. Association for Computational Linguistics.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. Preprint, arXiv:2311.10702.

Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3134–3154, Singapore. Association for Computational Linguistics.

Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, and Wei Wang. 2025. Bridging and modeling correlations in pairwise data for direct preference optimization. In The Thirteenth International Conference on Learning Representations.

Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, and Wei Wang. 2024a. Learning to edit: Aligning llms with knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 4689–4705. Association for Computational Linguistics.

Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024b. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 4667–4688. Association for Computational Linguistics.

Di Jin, Eileen Pan, Nassim Ouafatole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. 2024a. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064.

Ming Li, Yong Zhang, Zhitao Li, Juhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and
Jing Xiao. 2024b. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7602–7635, Mexico City, Mexico. Association for Computational Linguistics.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024c. From live data to high-quality benchmarks: The arena-hard pipeline.

Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024d. Self-alignment with instruction back-translation. In The Twelfth International Conference on Learning Representations.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.

Thao Nguyen, Jeffrey Li, Sewoo Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, and Xian Li. 2024. Better alignment with instruction back-and-forth translation. Preprint, arXiv:2408.04614.

TB OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in neural information processing systems, pages 27730–27744.

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2024. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Rafael Rafaïlov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641.

Xiaofeng Shi, Lulu Zhao, Hua Zhou, and Donglin Hao. 2024. Industrycorpus2.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Teknium. 2023. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karla, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijjing Xu, Haoqiang Kang, Ziyuan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhwei Liu, GUOJUN XIONG, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Huang Jiajia, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, and Jimin Huang. 2024. Finben: An holistic financial benchmark for large language models. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024a. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
A Implementation Details

Our implementation is based on the alignment-handbook repo3. The training procedure was executed on 4 NVIDIA A800 GPUs, each equipped with 80GB of memory. The duration required to train a single instance of the model, specifically the Llama3-8B-base, was approximately 9 hours. The specific hyperparameters used during training are detailed in Table 7. Notably, all models were trained using the same set of hyperparameters, except for the maximum sequence length, which was set to 2048 for the 14B LLMs to mitigate computational bottlenecks.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Max length</td>
    <td>4096 (2048 for 14B LLMs)</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Scheduler</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Weight decay</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Warmup ratio</td>
    <td>0.1</td>
  </tr>
</table>

Table 7: Training hyperparameters for Llama3-8B-base and Qwen2.5-1.5/3/7/14B-base.

B Evaluation Details

Table 8 lists the evaluation details for AlpacaEval 2 (Li et al., 2023), Arena-Hard (Li et al., 2024c), MT-Bench (Zheng et al., 2023), and IFEval (Zhou et al., 2023b). AlpacaEval 2 comprises 805 questions from 5 datasets, and MT-Bench spans 8 categories with a total of 80 questions. Arena-Hard is an enhanced version of MT-Bench, featuring 500 well-defined technical problem-solving queries. IFEval consists of 541 samples, each containing 1 to 3 verifiable constraints. Evaluation metrics are reported in accordance with each benchmark’s protocol.

C Dataset Analysis

Statistics including token lengths of instructions and responses are illustrated in Figure 5. Tokens are counted using the tiktoken library4. For WebR-Basic, the average token lengths of instructions and responses are 441.41 and 381.28, respectively. For WebR, the average token lengths of instructions and responses are 439.88 and 457.34, respectively.

![Histograms showing input and output length distributions for WebR-Basic and WebR-Pro](page_1012_120_388_1012.png)

Figure 5: Lengths of instructions and responses in WebR-Basic and WebR-Pro.

D Cost Analysis

Here we analyze the cost-effectiveness of our proposed Web Reconstruction framework. For context, we estimated the budget for data synthesis using the GPT-4o-mini API, based on the Batch API’s pricing of $0.075 per 1M input tokens and $0.3 per 1M output tokens. Table 9 lists the breakdown of the estimated costs for each step, which demonstrates that the overall expenditure ($38.57) is both reasonable and manageable.

Additionally, our main experiment in Table 1 demonstrates that the open-source Llama3-70B-Instruct model can achieve satisfactory performance for our proposed Web Reconstruction, significantly outperforming previous IT datasets. Notably, Llama3-70B-Instruct can be deployed on only 2 NVIDIA-3090 GPUs, with
<table>
  <tr>
    <th>Benchmark</th>
    <th># Exs.</th>
    <th>Baseline Model</th>
    <th>Judge Model</th>
    <th>Scoring Type</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>AlpacaEval 2</td>
    <td>805</td>
    <td>GPT-4 Turbo</td>
    <td>GPT-4 Turbo</td>
    <td>Pairwise comparison</td>
    <td>Length-controlled win rate</td>
  </tr>
  <tr>
    <td>Arena-Hard</td>
    <td>500</td>
    <td>GPT-4-0314</td>
    <td>GPT-4 Turbo</td>
    <td>Pairwise comparison</td>
    <td>Win rate</td>
  </tr>
  <tr>
    <td>MT-Bench</td>
    <td>80</td>
    <td>-</td>
    <td>GPT-4/GPT-4 Turbo</td>
    <td>Single-answer grading</td>
    <td>Rating of 1-10</td>
  </tr>
  <tr>
    <td>IFEval</td>
    <td>541</td>
    <td>-</td>
    <td>-</td>
    <td>Rule-based verification</td>
    <td>Accuracy</td>
  </tr>
</table>

Table 8: Evaluation details for AlpacaEval 2 (Li et al., 2023), Arena-Hard (Li et al., 2024c), MT-Bench (Zheng et al., 2023), and IFEval (Zhou et al., 2023b). The baseline model refers to the model compared against.

<table>
  <tr>
    <th># of Samples</th>
    <th>Avg. Input Token Length</th>
    <th>Avg. Output Token Length</th>
    <th>Cost ($)</th>
  </tr>
  <tr>
    <td>Generate author’s persona</td>
    <td>100,000</td>
    <td>523</td>
    <td>32</td>
    <td>4.88</td>
  </tr>
  <tr>
    <td>Web as Instruction (instruction)</td>
    <td>66,667</td>
    <td>711</td>
    <td>123</td>
    <td>6.02</td>
  </tr>
  <tr>
    <td>Web as Instruction (rollout response)</td>
    <td>66,667</td>
    <td>611</td>
    <td>392</td>
    <td>10.90</td>
  </tr>
  <tr>
    <td>Web as Response (instruction)</td>
    <td>33,333</td>
    <td>645</td>
    <td>91</td>
    <td>2.52</td>
  </tr>
  <tr>
    <td>Web as Response (rollout response)</td>
    <td>33,333</td>
    <td>91</td>
    <td>522</td>
    <td>5.45</td>
  </tr>
  <tr>
    <td>Web as Response (refined response)</td>
    <td>33,333</td>
    <td>1,155</td>
    <td>591</td>
    <td>8.80</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>38.57</td>
  </tr>
</table>

Table 9: Estimated budget for data synthesis using the GPT-4o-mini API.

the option to further reduce hardware requirements through low-bit quantization\footnote{https://github.com/ollama/ollama}. This provides an economical alternative for our proposed WebR. In conclusion, our framework demonstrates robustness in leveraging diverse LLMs for data synthesis, confirming its adaptability and effectiveness.

E Prompt Template

Figure 6 shows the prompt template for generating the author persona according to the web content. Figure 7 shows the prompt template for generating the rewrite request based on the whole web content. Figure 8 shows the prompt template for generating the rewrite request based on a specific part of the web content. Figure 9 shows the prompt template for generating the latent instruction corresponding to the whole web content. Figure 10 shows the prompt template for generating the latent instruction corresponding to a specific part of the web content. Figure 11 shows the prompt template for generating a refined response based on the raw web and the instruction.
Prompt Template for Author Persona

[Text]
{web}

[Instruction]
The text above is from an English webpage. According to the text, please infer the author's profile (within 30 words).

Figure 6: Prompt template for generating author persona.

Prompt Template for Web as Instruction (all)

[Text]
{web}

[Author of the Text]
{persona}

[Instruction]
The text above is from an English webpage. Imagine that you are a user of an AI assistant, please provide a rewrite request specifically designed based on the text content, to create a new version of the text. You can ask for the rewrite to follow constraints including word/sentence/paragraph length, style, format, structure, etc. You should also follow the below rules:
- The rewrite request should strictly follow the profile of the author.
- The rewrite request should be based on the above text, rather than an isolated instruction.
- The constraints should be detailed and specific.
- Output only the request.
- Do **not** directly use the keyword 'rewrite' and 'new version' in the generated request.
- Make sure the generated request is within {len_limit} words.

Figure 7: Prompt template for Web as Instruction (generating the rewrite request based on the whole web content).

Prompt Template for Web as Instruction (part)

[Text]
{web}

[Author of the Text]
{persona}

[Instruction]
The text above is from an English webpage. Imagine that you are a user of an AI assistant, please provide a rewrite request specifically designed based on the text content, to create a new version of the text focusing on a specific part of information, rather than global information, in the given text above. You can ask for the rewrite to follow constraints including word/sentence/paragraph length, style, format, structure, etc. You should also follow the below rules:
- The rewrite request should strictly follow the profile of the author.
- The rewrite request should be based on the above text, rather than an isolated instruction.
- The constraints should be detailed and specific.
- Output only the request.
- Do **not** directly use the keyword 'rewrite', 'new version', and 'specific part information' in the generated request.
- Make sure the generated request is within {len_limit} words.

Figure 8: Prompt template for Web as Instruction (generating the rewrite request based on the specific part of the web content).
Prompt Template for Web as Response (all)

[Text]
{web}

[Author of the Text]
{persona}

/Instruction]
Imagine that you are a user of an AI assistant, please provide the most likely request to which the text above would be a great answer. You should also follow the below rules:
- The request should strictly follow the profile of the author.
- Ensure your request is detailed, specific (including the style, format, and structure of the text), clear, and concise.
- Output only the request.
- Make sure the generated request is within {len_limit} words.

Figure 9: Prompt template for Web as Response (generating the latent instruction based on the whole web content).

Prompt Template for Web as Response (part)

[Text]
{web}

[Author of the Text]
{persona}

/Instruction]
Imagine that you are a user of an AI assistant, please provide the most likely request to which **a specific part of the text above** would be a great answer. You should also follow the below rules:
- The request should strictly follow the profile of the author.
- Ensure your request is detailed, specific (including the style, format, and structure of the text), clear, and concise.
- Output only the request.
- Make sure the generated request is within {len_limit} words.

Figure 10: Prompt template for Web as Response (generating the latent instruction based on the specific part of the web content).

Prompt Template for Web as Response (Answer Refinement)

Based on the Provided Information, please improve the Answer to the Question, so that the improved answer is of high quality and factually correct. Only output the improved answer.

[Provided Information]
{web}

[Question]
{request}

[Answer]
{answer}

Figure 11: Prompt template for Web as Response (answer refinement).

Reviewer paper summaries: This paper proposes a novel framework for synthesizing data from raw web pages, named "Web as Instruction" and "Web as Response." The authors leverage the LLM to generate corresponding requests based on the text content of the web pages, and then conduct multi-step rewriting and refining to synthesize the final instruction-response pairs.

This paper proposes Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data from raw web documents without requiring seed prompts, human annotations, or strong structural assumptions. WebR introduces a novel dual-perspective approach—Web as Instruction and Web as Response—to transform unstructured web content into coherent instruction-response pairs. WebR also demonstrates strong data efficiency, domain adaptability, and compatibility with existing datasets, making it a practical and effective solution for enhancing the instruction-following abilities of large language models. A potential misunderstanding is that WebR is not a retrieval or QA extraction system; instead, it reconstructs new, meaningful instruction-response pairs from noisy input, enabling broader and more flexible application.

This paper proposed a method for generating instruction-tuning data with no human effort. The proposed method uses web documents to build instruction data in two ways: Web as Instruction and Web as Response. The paper shows that the model fine-tuned using the proposed data outperformed the baseline datasets.
Summary of strengths: Data is the most important factor for the performance of LLM training. The lack of high-quality data is  still a significant issue in the community. It is very meaningful to synthesize instruction tuning datasets from large-scale web corpora. This paper provides a good demonstration of the entire process.

1. This paper aims to  eliminate the need for human-labeled seed data or prompt engineering.
2. Propose Novel “Web as Instruction” and “Web as Response” approach enhances instruction diversity and quality.

* A method is proposed for generating instruction tuning data with no human effort.
* Good ablation study to show the effectiveness of the components of the proposed method
Summary of weaknesses: Synthesizing data directly from web is not a new concept to the community. [1] Such practices have become very common in the LLM industry. This paper does not offer many new insights.

For the experiments in Figure 3 and Figure 4, Alpaca, ShareGPT, and UltraChat are not high baselines. As a general data synthesis method, it should be compared with more recent datasets in terms of diversity and quality, such as OpenHermes, Magpie, and WebInstruct.

In Figure 3, the diversity of the dataset presented in this paper does not appear to be significantly better than that of the Alpaca dataset which is from two years ago. I hypothesize that during this data synthesis process, the model introduces too much subjective bias, resulting in the synthesized dataset having too much GPT style, which affects the diversity of the original web data distribution.

It is very, very important to show the benchmark scores of Llama3-8B before any SFT in Table 1 and Table 2 as a comparison.

[1] Yue, X., Zheng, T., Zhang, G., & Chen, W. (2024). MAmmoTH2: Scaling Instructions from the Web. ArXiv, abs/2405.03548.

1. The novelty is not strong enough. This paper focuses on the prompt engineering and the main idea is inspired by the Back-translation[1].
2. This paper involves the synthesis the instruction tuning data from web data, which is similar to the related paper[1][2]. The experimental results should align the datasource of web data to demonstrate the effectiveness to compare with the above baselines. 


[1] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024d. Self-alignment with instruction back- translation. In The Twelfth International Conference on Learning Representations.
[2] Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. 2024. Dog-instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4125–4135

* The improvement against the baseline (IT-Mix) is marginal, and the author failed to show the difference is significant.
* The effectiveness of the web document quality should be analyzed. For example, when using Common Crawl only, what will the performance of the proposed method change?
Comments, suggestions and typos: In Figure 3 and Figure 4, the paper should be compared with more recent datasets in terms of diversity and quality, such as OpenHermes, Magpie, and WebInstruct.

In Table 1 and Table 2,  the benchmark scores of Llama3-8B before any SFT should be presented.

1. Adding the more experimental results, including aligning the experimental setting to compare the performance of baselines[1][2]. It can further verify the effectiveness of proposed method.


[1] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024d. Self-alignment with instruction back- translation. In The Twelfth International Conference on Learning Representations.
[2] Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. 2024. Dog-instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4125–4135

* In order to show the improvement against the baseline (IT-Mix) is significant, there are some suggestions.
  - Use different random seeds to sample IT-Mix and report performances with std based on different IT-Mix samples
  - Generate responses using GPT-4o-mini for datasets with human level = no (or simply Magpie datasets) and compare with the proposed method
  - Train models a few times with the proposed method/baseline with fixed examples and report performances with std
Overall assessment scores: 2.5

2.5

3.5
Metareview score: 3
Metareview: This paper proposes an instruction data synthesis framework named WebR, which aims to automatically produce high-quality instruction data without relying on human effort. The framework consists of two parts: web as instruction and web as response, distinguished by whether web content is used as the starting point for instruction or response generation. While these synthesis approaches have been individually explored in prior studies, the authors introduce additional steps to improve the quality of synthesized data, such as rewriting web page content before the synthesis process.The experimental results demonstrate that the synthesized datasets, WebR-Pro and WebR-Basic, outperform previous open-source instruction datasets in terms of quality and exhibit good compatibility with other open-source datasets. However, the lack of controlled experimental settings makes it difficult to quantify the specific contribution of the proposed method.
Summary of reasons to publish: 1. The authors propose an instruction data synthesis framework that incorporates both synthesis directions (web as instruction and web as response) and introduces additional processing steps to ensure data quality.
2. The experimental results are good, demonstrating that the constructed instruction datasets (WebR-Pro and WebR-Basic) improve models' instruction-following capabilities.