Title: TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Synthesis

Tree-Guided Space Partitioning

Data Diversity Enhancement
Abstract: Model customization necessitates high-quality and diverse datasets, but acquiring such data remains time-consuming and labor-intensive. 
Despite the great potential of large language models (LLMs) for data synthesis, current approaches are constrained by limited seed data, model biases and low-variation prompts, resulting in limited diversity and biased distribution with the increase of data scales.
To tackle this challenge, we introduce TreeSynth, a tree-guided subspace-based data synthesis approach inspired by decision trees. 
It constructs a spatial partitioning tree to recursively divide a task-specific full data space (i.e., root node) into numerous atomic subspaces (i.e., leaf nodes) with mutually exclusive and exhaustive attributes to ensure both distinctiveness and comprehensiveness, before synthesizing samples within each atomic subspace. 
This globally divide-and-synthesize method finally collects subspace samples into a comprehensive dataset, effectively circumventing repetition and space collapse to ensure the diversity of large-scale data synthesis.
Furthermore, the spatial partitioning tree enables sample allocation into atomic subspaces, allowing the re-balancing of existing datasets for more balanced and comprehensive distributions.
Empirically, extensive experiments across diverse benchmarks consistently validates the superior data diversity, model performance, and robust scalability of TreeSynth compared to both human-crafted datasets and peer data synthesis methods, with the average performance gain reaching 10%.
Besides, the consistent improvements of TreeSynth-balanced datasets highlight its efficacious application to redistribute existing datasets for more comprehensive coverage and the induced performance enhancement. The code is available at https://github.com/cpa2001/TreeSynth.


Peer reviews:
comments: SUMMARY:
This paper proposes TreeSynth, a method to generate data synthetically by formulating concepts into a tree format. Nodes are concepts, where deeper nodes are more fine-grained concepts. They show results on Llama and Qwen 70B models with math and code generation benchmarks, showing improved performance.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper provides enough mathematical formulation to make it easier to follow
- Experimentation is extensive

Weaknesses:
- This methodology reminds me of the InstructLab paper (LAB: Large-Scale Alignment for ChatBots) that also uses a tree-like data synthesis method. Providing a comparison (results would be nice, but a discussion will suffice for now) would be great.
- I might have missed any human studies or automatic studies to measure the quality of the synthetic data, but that information would also help.

I'd be more than happy to update my scores once my concerns are addressed.

QUESTIONS:
1. How did you choose N to be 10? Were there any hyperparameter studies done?
2. The manuscript mentions "another LLM, proficient in identifying inter-sample distinctions, determines exactly one core criterion" on lines 116-117. It wasn't immediately clear to me what this LLM is - is it fine-tuned, few-shot augmented, or a base model? I might have missed where those details were mentioned.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TreeSynth: a novel method for synthesis of text data aided by Decision Trees and Large Language Models (LLMs).\nTreeSynth leverages an LLM to generate a decision tree to exhaustively enumerate all possible subspaces of the dataset (also called criteria).\nThis enumeration is then used to enhance diversity and comprehensiveness of the starting dataset, leading to improved downstream ML performance of models trained with such augmented data.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n+ **The proposed method showcases consistent improvements with respect to the state-of-the-art.** These improvements are consistently measured over several experiments spanning three LLMs and three datasets.\n+ **The authors tackled the issue of data diversity in text data.** This is a significant problem in Machine Learning.\n+ **Extensive information to reproduce the results.** The authors provided comprehensive information to reproduce the results. Including the prompts used for each experiment.\n\n## Weaknesses\n- **Some statements in the main paper are not properly discussed.** For instance, at lines 116-119: “Subsequently, another LLM, proficient in identifying inter-sample distinctions, determines exactly one core criterion δ (e.g., Type of Mathematical Operation). This criterion $\\delta$ optimally partitions $X$ into mutually exclusive attribute values $V_X^\\delta = {v_j^\\delta |j = 1, 2, ..., m}$” lacks a proof or an informal explanation of why this is the case, nor any experimental evidence is shown in support of this statement. A reference should point to such a source if this has been done in the literature.\n- **Need for extensive evaluation of TreeSynth and TreeSynth-guided data balancing.** TreeSynth generation quality is only measured by means of data samples’ cosine similarity. Further insights on the data generated by such a method would strengthen the paper’s statements and contributions. Tables that give insights into the generated trees (e.g., how many criteria are generated, depth of the trees, examples of generated criteria) would strengthen the paper as well. TreeSynth-guided data balancing is evaluated in one experiment only (Table 3), in which it is showcased to outperform all other methods. This method could have been analysed more extensively, showcasing the strength of the proposed method.\n- **Need for more qualitative analysis.** In papers related to language models, in which quantitative/formal analysis is often infeasible, it is important to provide qualitative analysis of the produced data. The paper does not provide any example of text generated by the TreeSynth-aided models, nor does it provide examples that showcase how improvements of these generated texts are related to the proposed method.\n\nQUESTIONS:\n- Could you provide further numerical insights that demonstrate the increased diversity of data generated by TreeSynth?\n- Could you provide some qualitative examples of answers generated by one of the models with TreeSynth data and with another baseline which showcase how the performance improvements are related with the diversity issue tackled by TreeSynth? (For instance, TreeSynth-aided LLMs can answer questions about domains that are underrepresented in the PersonaHub-aided ones) \n- Could you provide an informal justification (or a source) in support of the statement made in lines 116-119? Is it the case that the LLM can possibly generate criteria that do not satisfy such a statement? If so, how often does this issue happen? How can it be identified? To what extent does it affect TreeSynth performance? And how can you address it?\n- Enumerating all the possible criteria of a dataset can be prohibitive as the size of the decision tree built by TreeSynth grows exponentially in the number of attributes\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for generating synthetic data that is more diverse, inspired by Decision Trees. This method aims to create criteria for subdividing the data space into exclusive regions. To achieve this, they prompt a large language model (LLM) with a data description to generate N diverse samples. Another LLM, proficient in identifying distinctions between samples, determines the core criteria. This process yields a set of instructions used to guide an LLM in generating diverse synthetic data. The authors evaluated their method by starting with popular benchmarking datasets and employing well-known LLMs (both open-source and proprietary) to create synthetic data following the proposed approach. They then fine-tuned smaller models on this generated data, demonstrating performance gains. The baseline for comparison was established using other methods for generating diverse synthetic data. The results indicate that their method outperforms the baseline.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and clearly explained, featuring a robust evaluation method and an extensive presentation of results, including effective representations like t-SNE visualizations. \n\nThe baseline is effectively presented alongside other similar methods (such as de Evol-Instruct and Persona), and the choice of evaluation method (fine-tuning smaller models) is a strong approach for assessing the performance of the proposed method. \n\nHowever, I have some concerns regarding the use cases for generating this type of synthetic data. Essentially, this process resembles model distillation, as the method relies entirely on a \"teacher\" LLM to create diversity within a specific data space. The approach involves a chain of prompts that generate samples and criteria for constructing the Tree, which is then used as guidance to prompt the \"teacher\" model for generating synthetic data. This dependency is evident in the varying results produced by different base models used in the tests. While the method clearly illustrates model distillation, the introduction may imply that it could generalize and \"enhance their specific capabilities.\" \n\nI recommend that the authors clarify the scope of utility for these models to align with the demonstrated use case of model distillation. \n\nModel distillation indeed offers several advantages, enabling smaller models to be deployed on devices or in fog computing systems, thereby conserving computational resources.\n\nQUESTIONS:\nI also have several questions: \n\n- Is it clear whether the context provided to the LLM is input by the user? The criterion may vary significantly depending on the input. \n- What exactly is this \"proficient LLM\" in identifying inter-sample distinctions that determine the core criteria? \n- Evol-Instruct also shows similar diversity in the t-SNE visualizations, and results demonstrate diversity comparable to TreeSynth; however, TreeSynth achieves better outcomes. What explains this discrepancy?\n\nLIMITATIONS:\nThe authors stated a set of limitations and the limitations sections.\n\nIn my point of view, the main limitations is the dependency of the teacher LLM to generated synthetic data, and I would suggest the authors to better scope the utility of the models to the demostrated use cases as model distillation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TreeSynth, a method for generating synthetic training data for large language models (LLMs). The core idea is to sample from the intermediate nodes of a tree of LLM generations, which are conditioned on a common root instruction. This tree structure allows TreeSynth to capture both diversity and relevance across multiple levels of reasoning, going beyond traditional sampling techniques like nucleus or top-k sampling. TreeSynth is evaluated by training LLaMA 3.1B and Qwen2.5-7B models on TreeSynth-generated data across four tasks: summarization, closed-book QA, math reasoning, and code generation. The results show that TreeSynth outperforms conventional sampling strategies and even some strong baselines like Self-Instruct or Evol-Instruct in various settings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n- Novel sampling mechanism: The tree-based approach introduces an elegant way to traverse and utilize the latent space of LLM generations, capturing a mix of diverse and high-quality examples by harvesting from both leaf and intermediate nodes.\n- Empirical Results: Models trained on TreeSynth data outperform those trained on standard instruction-tuned datasets generated by top-k/top-p sampling or previous bootstrapping techniques.\n-Comprehensive evaluation across tasks: The paper evaluates performance on summarization, QA, math reasoning, and coding tasks, providing a good sense of TreeSynth's general applicability within the fine-tuning context.\n- Clear methodology: The design of the tree, including beam width, scoring strategies, and node selection policies, is well-explained and easy to follow. Ablation studies on node scoring and position further strengthen the technical argument.\n\n\n## Weaknesses\n\n- Limited model diversity: The experiments are restricted to only two models: LLaMA 3.1B and Qwen2.5-7B. It remains unclear how TreeSynth generalizes to other model architectures and/or to larger/smaller model sizes. This limits the generality of the claims.\n- Narrow scope of data synthesis: The method is only tested on domain- and task-specific data generation (e.g., math reasoning). Despite TreeSynth’s generally task/domain agnostic nature, the paper does not explore its utility for general-purpose instruction tuning or pretraining-style data generation. If the method is capable of broad data synthesis, it should be evaluated on more open-ended tasks or datasets.\n- Compute requirements for tree sampling: TreeSynth involves generating a beam of responses at each node recursively, which could be costly at scale. The paper doesn't discuss this overhead or propose methods to make it more efficient.\n\nQUESTIONS:\n1. Why were only LLaMA 3.1B and Qwen2.5-7B chosen for evaluation? Can TreeSynth generalize to other model architectures or sizes? Experiments on at least one more model family would increase confidence in generality.\n\n2. TreeSynth appears capable of producing a broad range of data types, yet it is only applied to very task-specific settings. Could the authors demonstrate its use on general-purpose instruction tuning data (e.g., FLAN, OIG, or Dolly-style datasets)?\n\n3. What are the computational costs of generating and pruning large generation trees? Could this method scale to hundreds of thousands or millions of examples?\n\n4. Could the relevance scoring function be better calibrated across domains? Are there task-specific biases introduced by using a static scoring model?\n\n5. Is TreeSynth applicable to other stages of LLM development, such as reward model training or RLHF-style pipelines?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}