PANGEA: Projection-Based Augmentation with Non-Relevant General Data for Enhanced Domain Adaptation in LLMs
Authors: Seungyoo Lee
Giung Nam
Moonseok Choi
Hyungi Lee
Juho Lee
Venue: NeurIPS 2025 poster ()
Keywords: synthetic data generation
large language models
domain adaptation
Primary area: deep_learning
TL;DR: This paper introduces PANGEA, a method that leverages general-purpose data to generate diverse and high-quality synthetic data, improving LLM performance on domain-specific tasks.

Abstract: Modern large language models (LLMs) achieve competitive performance across a wide range of natural language processing tasks through zero-shot or few-shot prompting. However, domain-specific tasks often still require fine-tuning, which is frequently hindered by data scarcity, i.e., collecting sufficient domain-specific data remains a practical challenge. A widely adopted solution is to generate synthetic data using LLMs by augmenting a small set of available domain-specific examples. In this work, we first identify fundamental limitations of such approach in terms of both data diversity and quality, particularly when relying on only a handful of domain-specific examples. We then propose our method, PANGEA, which leverages large-scale, publicly available general-purpose data---entirely unrelated to the target domain---to generate more diverse and higher-quality synthetic data. Our extensive experiments on domain-specific benchmarks, including GSM8K, MedQA, and FinQA, as well as a custom domain-specific language task, validate the effectiveness of our approach.

PANGEA: Projection-Based Augmentation With Non-Relevant General Data for Enhanced Domain Adaptation in LLMs
Seungyoo Lee KAIST
Giung Nam KAIST
Moonseok Choi KAIST
punctuate@kaist.cco ac.kr
giungekaist.ac.uu ac.
ms as.cho@@kaist.acs ac.kt kr
Hyungi Leet
Juho Leef
Kookmin University Lhk27056kookmin..a0
KAIST
juholee@kaist.an ac.kr
Abstract
Modern large language models (LLMs) achieve competitive performance across a wide range of natural language processing tasks through zero-shot or few-shot prompting. However, domain-specific tasks often still require fine-tuning, which is frequently hindered by data scarcity. i.e., collecting sufficient domain-specific data remains a practical challenge. A widely adopted solution is generate syn- thetic data using LLMs by augmenting a small set of available domain-specific ex- amples. In this work, we first identify fundamental limitations of such approach terms of both data diversity and quality, particularly when relying only a hand- ful of domain-spectie leverages large-scale, publicly available general-purpose data entirely unrelated to the target domain-to generate more diverse and higher-quality synthetic data. Our extensive experiments on domain-specific benchmarks, including GSM8K, MedQA, and FinQA, as well custom domain-specific language task, validate the effectiveness of our approach.
examples. We then propose our method, PANGEA, which
Introduction
Large language models (LLMs) have achieved strong general-purpose capabilities via large-scale pre-training, handling tasks from natural language understanding to
CDSL
40
Tuned
35.36%
Naive
complex reasoning [11, 12, 31], yet they often strug- gle on domain-specifie tasks which demand expert-level knowledge such as those in the medical or legal sec- tors [3, 13, 28]. A key bottleneck is the scarcity of high- quality domain data, as many real-world datasets (e.g.. patient records or proprietary financial documents) are often inaccessible due to privacy constraints [1, 18, 19, annotated datasets is
Accutacy (%)
30
Evel-Instruct
PANGFA (ours)
20
10-
6.96%
0.57%
30k
120k
41]. Moreover, constructing expert-annotated costly and time-consuming due to the necessity for the limited availability of qualified annotators. As a result, niche domains often have far less training data than is available for web-scale corpus [6, 15, 47, 48].
Figure 1: Performance by data scale. Result on customized dataset (cf. $ 4.1.
Equal corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In practice one may resort to fine-tuning APIs on small domain-spectt datasets as an initial at- tempt. However, such direct tuning on small amount of data is fundamentally prone to overfitting and catastrophic forgetting of previously learned general knowledge, particularly as LLM sizes con- tinue to grow exponentially [17, 25J. Instead of updating the model weights, another line of work equips models with external knowledge at inference. Retrieval-augmented generation (RAG) tech- niques incorporate relevant documents from external knowledge base into the context, allowing LLMs to address domain-specific queries using up-to-date information [28, 46]. This can mitigate the scarcity of training-date by leveraging large text collections on the fly, though it still requires large retrievable domain knowledge source. Another direction involves multi-stage alignment or self-refinement pipelines, often guided by human feedback [22, 29, 43]. These approaches enable gradual domain adaptation, though they typically induce training instability, limiting scalability and practical deployment when given a small number of data.
Recently, a promising trend has emerged which leverages L.MSs themselves to synthesize new train- ing data, reducing reliance on scarce human annotations [7, 20, 33, 35, 37]. Pioneering works showed that LLMs can bootstrap their instruction-folllwing ability by creating large set of syn- thetic instructions and answer pairs [21, 33]. Subsequent efforts extend to diverse domain-specifit tasks, data modalities, and integration with external tools [8, 37, 42]. Across such diverse appli- cations, synthetic data generation has proven to be remarkably effective for extending LLM capa- bilities. Despite ongoing advances, one major limitation persists: the synthetic data distribution is skewed or repetitive, as models tend to generate simpler examples which cannot capture the full complexity of genuine domain-speciffe data distribution [9, 37].
Our key contributions are summarized as follows:
We present PANGEA (Projection-basee Augmentation with Non-relevant General data for Enhanced domain Adaptation). a fully automated framework for domain-specific data gener- ation that enhances diversity without any additional annotation cost.
Unlike existing LLM-based synthetic data generators that rely manually crafted seeds or prompts, our method leverages a large-scale general-purposs dataset De to bootstrap the syn- thesis of high-quality domain-speciff examples. In essence, PANGEA extracts diversity from De and integrates it into the structure of a small comaiin-pecifue dataset D,, yielding a synthetic dataset that inherits the diversity of Dg while preserving the domain relevance of D..
As synthetic dataset size increases, our method maintains and even more improves diversity, outperforming existing approaches in both coverage and quality. In this paper, we validate PANGEA domain-specific benchmarks spanning mathematics, medicine, finance, and cus- tom domain-specific language task, demonstrating consistently strong performance.
2 Related Works
Recent efforts adapt LLMs to domain expertise using specialized datasets, requiring the collection of sufficiently large domain-specifec data. This section summarizes key methodologies for constructing and leveraging such datasets for domain adaptation. See & C.1 for additional related works.
Synthetic data generation. Recent works such as WizardLM [36], WizardMath [20], and Wizard- Coder [21] generate synthetic data for prompt- based reinforcement learning and instruction tuning. While effective, these methods depend exclusively on LLM generation guided only by source data, often resulting in off-domain or low-quality samples. Other methods like Ti [8] and Mag- pie [38] demonstrate efficient generation pipelines, but are limited by structural simplicity or lack of domain specificity. aetrieval-augment
techniques such as CRAFT [49] and DataTune [9, 27] enhance generation via external data, yet their performance deteriorates without sufficient in-domain data. ELTEX [23] further emphasizes iterative extraction from raw data, but similarly presupposes the presence of relevant domain signals.
Scalable fine-tuning for domaín adaptation. Domain adaptation typically involves fine-tuning LLMs on curated comain--peciff datasets. Early scalable approaches introduced domain- speccifc instruction tuning to improve generalization across tasks [4, 26, 34j. However, acquiring high- quality and sufficiently diverse data remains a significant bottleneck, particularly in specialized do- mains. Later works involve actively collecting broader domain-specifie datasets or constructing new datasets [22, 30, 33, 44], yet such approaches are costly and labor-intensive.

---

OE D
sIL
<I>
Two space explorers are Rlynn around through... the volcanic cruat on Sarahn or Timan hotter?
Marhh
Medical
Finance
CDSL
+
Small-scale, Domain-Specifee
Large-scale, Nomsin-Unrelater General Data De) )
Source Data
Dy
Synth- Block (r)
88
Sarahn and Timon discovered two planets... What is the difference in their temperatures?
Synth-Profillng
06
Plan: (1) Celoulate...
d, 1->
a
Denegraphiot:
ceplorers...
36-yyea-oold space explorer... of the following the most likeely pathophysiologic mechanism...
0 06
cstract
Scanardo...
Plan...
high-rssk, high copprrsmar....
Firenctel scmarii:: Fanane ts
gaartnigroooaptio onlif....
Extrost,
peppgraniic.....
Stellar Frotiers Inc., ploneering space exploration fir.... ecoplanets...
profit...
Finaccial scasarda.... Format...
<
"Sararc hot; Timon: cold". reverse the order of the entries, swaup the separators...
Petract stanget stringg....
stränge....
<EOP
CORD
Specific Profile (9)
Prompt-Writer (dg. 7)
Projection (r,d,) deyn.
Large-scale, nomain-Speciffe Synthetic Data (Day
Figure 2: Overview PANGEA framework. effectively synthesizes domain specifcc data using domain-unrelated open-source data, thus even applicable to areas where similar data is scarce.
3 Methodology
In this section, introduce method, called PANGEA, which is designed to augment high- quality synthetic data while addressing challenges in domain adaptation and data diversity. Tradi- tional augmentation methods relying on LLMs are limited because they depend solely on the LLM's performance without proper guidance. When applied to augment scarce source data for domain- specific tasks, these methods suffer from issues such as quality degradation, reduced diversity, and high redundancy in the generated synthetic data. Moreover, techniques like Aetrieval-Augment Generation (RAG), which rely on augmenting data with similar or in-domain data, cannot be applied when such relevant data is unavailable, which is the case for most domain-specine tasks.
PANGEA, the other hand, leverages easily accessible out-of-domain general data supplement limited domain-specific data by applying guidance distilled from the source data. This strategy ef- fectively enhances the diversity, complexity, and quality of the synthesized dataset, as demonstrated in the following sections. An overview of the proposed methodology is illustrated in Figure 2, which shows how domain-inelevant data used generate synthetic domain-specific data.
3.1 Motivation: deconstructing step-by-step prompts for synthetic data generation
We explore a simple step-by-step prompting strategy as a proof of concept to leverage out-of-domain general data for augmenting domain-specift, data. The process involves guiding the model gen- erate data through the following steps, formatted as a single prompt (see $ A. for details):
Step 1: General data analysis. In this step, the model is instructed to analyze the given general data and extract relevant details, numerical information, and entities that can be mapped to the target domain. Even if the data itself is domain-irreleven components of synthetic domain -spacffc data. Which details from the general data should be used is determined based the target domain, guided by patterns extracted from the source data. These elements are then identified for potential application to the domain.
such elements may still hold potential as
Step 2: Referencing existing domain data structure. By referencing the existing data structure and details, the model encouraged to generate data that does not replicate the original data but instead follows the problem's 's structure, format, length, and difficulty level.

---

Step 3: Projection and transformation of data. The elements extracted in Step I are projected onto the target domain. Guided by the references crafted in Step 2, this step ensures that the projected data is contextually rich and aligned with the target domain.
While this step-by-step prompting approach proved to be effective for generating domain-speciff synthetic data using out-of-domaii general data, its fixed-format prompt posed challenges in certain domains, such as MedQA. As illustrated in Fig. 3, the model either failed to extract the necessary elements from the general data data-nesulting in non-informative synthetic outputs-or generated data that lacked sufficient domain--seciff information, leading to content irrelevant to the target domain.
As an ablation study, we present the results in Table 1. Compared to the full prompt, (1) simplifying Step 1, (2) simplifying Step 2, and (3) simplifying Step 3 each signif- icantly degrade performance and data quality. We found that the first step analzzigg general data and mapping to the domain had a significant impact on perfor- mance across various domains. Building on this insight,
Table 1:
Ablation study. Importance of each step measured by performance drop at 30k data scale when removed.
GSM8K
MedQA
FinQA
Avg.
(1)
29.11
37.75
36.70
34,52
(2)
32.22
39.59
38.27
36.69
our PANGEA methodology introduces the Prompt Writer component to further strengthen Step 1, which focuses on leveraging general data.
(3)
31.84
38.89
37.75
36.16
Full 35.48
39.12
40.19
38.26
3.2 PANGEA: three-stage generation process using the Prompt Writer
Stage 1: Synth profiling. When utilizing irrelevant gen- eral data augment scarce dpomain-peecfic datasets, it is crucial to first systematically identify and extract mean- ingful domain-specific information. Without explicitly characterizing domain-specific attributes upfront, aug- mented data may suffer from poor alignment with the tar- get domain and reduced effectiveness in the target task.
Algorithm 1 PANGEA
Require: Source dataset Dj and gen-
cral dataset Dg. where [Dg < (Dgl-
Ensure: Synthetic dataset Dsyn.
1: D + 0
2: f- LLMpoodinng (D,)
To address this challenge, we start with an initial stage within our framework termed synth profiling. When the available source dataset is extremely limited-i.e., con-
3:
while enough-data generated -eoege-d do
4:
Sample dg E D, and dg E Dg- x- LLL prompt writer (dg, 7) dayn +- L.L. projecttiioonn (T, ds)
5:
taining fewer than 100 examples-it becomes feasible to analyze the entire dataset by an L.M human ex- pert. Such analysis can provide valuable guidance on what kinds of information should be extracted from gen-
6:
7: Dayn <- Dayn U ((dyn))
8:
end while
eral datasets to generate synthetic data. Therefore, through our synth profiling framework, we aim to identify the critical informational elements in the target dataset-such as narrative structures key entities and use this insight to guide the generation of diverse synthetic data by leveraging various combinations and information scattered across general data sources.
Although information extraction can be performed by a human expert, in our experiments, we utilize state-of-the-art LLM,, OpenAT's ol [12]. Formally, given small amount of source dataset D,, the synth profiling step can be formulated as follows:
17 LLMpronlling (Ds),
where 1) represents the structured domain-speciff profile derived through the systematic analysis of the source data and I.I.Mposiling denotes the ol model which we used. And because ni is derived from each specific source dataset, the resulting profiles can vary significantly. For example, in the GSM8K dataset, 17 encompasses scenario descriptions, symbolic representations, equations, and re- lated reasoning structures. In contrast, for the MedQA dataset, 7) includes demographic information, clinical timelines, key medical events, laboratory results, and other domain-specific elements. These specialized profiles help guide the generation of more relevant and informative synthetic data. Refer to A.2 to see how we prompt LLMprodling to generate 7) from D,.
By explicitly conducting the profiling stage, we provide a clear blueprint for how general data should be transformed for effective synthetic data generation. Here, the structured guidance derived from facilitates the creation of high-quality synthetic samples, substantially improving both the relevance and diversity of the generated data.

---

Domain-Unrelated General Data
explorers decide
around through unknown
they discover planets. Given that planets
after themselvea. The Surahn currently going through geagraphic Many of continents planet currently breaking spart moving. stable planet with continental moving for quite awhile now, Givan the paragraph above, plaass anawer correctly the following question: the volcanic crust on Sarahn Timon hotter?
Low- www-Quality Synthetic Case
A A-45-yaar-old goologist, who has been studying the planetury movemnnnta of Sarahn and Timon, presents with symptoms of heat exhauation after spending several hours the surface one these planets, Given the peological actrvity these planets, which esposed accerstve leading Choose the following:
Synth-Guide Block
following the Litely planet where the peologist
[- DDemggraphics: Two space
ARRO
explorers, planets named Sarahn
Timellin: Current
Timon
Projection
B. Serahn, due to its current geographic shift and incressed volcanic activity
Timan, due its stable tectonic plates
volcanic activity
geographic
(dg. d,) dayn
C. Timon, due to its high concentration of greenhouse gases the stmosphere
stable period on Timon
D. Sarahn, due its low altitude and proximity to the sun
specified duration.
Key olinical events: Continental
breal--aaar and movement on Sarahn
High-Quality Synthetic Case
LLab imaging patters: None explicitty stated, Implied tectonic activity Sarahn Pathophysiologie principles:
+
Sarahn presents with 2-week hisstory following recent continental break apart event. patient's ooygen saturation RB% on reom air, and chest my shres hilateral infiltraees Which of the followeing is the most likely pathophysiologis mechanism contr contributing to the patient's is symptoms?
333-year-old space explorer
worsening respiratory symptoms,
cough
Goologic processes such as plart tactonica and volcanic activity, potentially leading increased heat and volcanic Sarahn
Choose ona of the following::
A. Incressed volcanic cruat tempenature leading sulfar dioside gas Decreased
Projection
pressure due to planetary grogmphic
crust compared Timon.
(r, d,) dayn
Inhalation particulate
toctonic activity
Hypobario hypoxia resulting from high- altitude terrain formation
Figure 3: End-to-end flow diagram of PANGEA Stages and 3. A general data passage is first distilled into a structured Synth- Synth-Guide Block (T), and then projected into a domain-spectio question format (e.g., MedQA). A naive projection of general data lacks clinical realism, whereas data generated through Stage exhibits high realism and domain relevance.
Stage 2: General data analysis with the Prompt Writer. After extracting the domain-specific profile 7) in Stage 1, we employ frozen LLM, referred as the Prompt Writer promproopt-rrrioor) to systematically extract structured, domain-specific Synth-Guide Blocks (r) from irrelevant, out- of-domain general data instances (dg), using 7) as guidance. Each instance dg is drawn from the irrelevant general dataset D₂ (i.e.. dg E Dz)). Simply put, T can be understood as the information within each dg that either corresponds to is relevant to 1,, organized into a prompt-like format. Since T is generated from a wide range of different de samples, numerous and diverse combinations of r are produced, which in turn ensures the diversity of the resulting synthetic data. Specifically, this process can be written as follow:
LLM wropt (dz, 1), where dg E DD
(2)
Here, we utilized Llama3. . 3- 70B- Instruct [10] as for LLMprompt in our experiments. As described in Stage I, since the profile 1) is closely tied to the characteristics of each source dataset, it can vary significantly across tasks. Consequently, the 7 extracted from the general dataset is tailored to reflect the specific requirements of each target domain. This flexibility sets our method apart from traditional synthetic data generation approaches that rely on source-similar data [9, 49]. Instead, our framework leverages single, broad general dataset to support wide range of domain-speciff tasks, offering a more scalable and adaptable solution.
Also, this explicit structuring of information- rather than relying on implicit prompting- offers clear and consistent guidance for data generation. As a result, this approach helps transform each dg into data that is more suitable for learning in the target domain, reducing the risk of generating irrelevant content as observed in the failure cases discussed in $ 3.1. It consequently improves both the efficiency and quality of the generated synthetic dataset. Refer to $ A.3 to see how we prompt LLMppongt-wttterr to generate т.
Stage 3: Domain-specific synthetic data generation. In the final stage-Stae 3-we generate synthetic data using the Synth-Guide Blocks 7 produced in Stage 2. As previously mentioned, each is a summarized prompt that contains key information required for the target domain. By using the information in to generate data in the appropriate domain-specifie format, we can produce synthetic samples that are more aligned with the learning needs of the downstream task. To ensure
5

---

Table Main results comparing synthetic data generation frameworks. Accuracy (%) on four benchmarks is shown for increasing amounts of synthetic data (10k, 30k, and 120k). Our proposed PANGEA consistently outperforms the baselines across all data scales, with especially large margins at higher data volumes. All evaluations are conducted in zero-shot setting.
Benchmarks
# Synthetic
Method
GSM8K (1)
McdQA (1)
FinQA (1)
CDSL (1)
Avg. (impe)
Pre-trained
5.69
28.91
6.02
0.00
10.16
Instruction-tuned
45.03
37.31
26.68
0.57
27.40
Naive
26.91
35,42
24.06
3.20
22.40 (+12.24)
10k
Evol-Instruct
27.36
36.29
26.68
5.22
23.89 (+13.73)
PANGEA (ours)
32.52
37.78
36.44
11.30
29.51 (+19.35)
Naive
34.72
34,24
27.46
5.51
25.48 (+15.32)
30k
Evol- lool-Instrcct
32.51
38.09
29.64
9.57
27.45 (+17.29)
PANGEA (ours)
38.36
39.98
41.41
17.68
34.36 (+24.20)
Naive
42.68
38.02
32,43
6.96
30,02 (+19.86)
120k
Evol- Iostrnntttct
38.73
42.34
33.74
13.04
31.96 (+21.80)
PANGEA (ours)
48.61
44.62
50.22
35.36
44.70 (+34.54)
that the generated data follows the correct format, we project each T using both the domain-specific source data d, Dd and large language model, denoted as LLMppjection- At this point, maximize the data diversity and improve training efficiency, we partition the collection of r during the projec- tion process so that each domain-speedt specific sample is assigned with N = [Pal different general data instances. Then, for each ds with its partition Ps containing the N Synth-Guide Blocks, we perform projection using LLM projection as follows:
D.
M.MMoooectioon
dyyn LLM projection (Ty. ds), where Tj Ps and dg E D..
(3)
In experiments, utilized Llama3. 3-70B- as for LLMprojection- After we generate a partial synthetic dataset using each dę and its corresponding partition, we gather all of them to make our new synthetic dataset Dsyn to train a domain-speeifie task. Fig. illustrates examples of generated in Stage 2, as well as qualitative comparison between synthetic data generated with and without the use of in Stage The figure highlights how leveraging т leads more relevant and structured outputs, while directly using dg without 7 often results in misaligned synthetic data. As we proceed with instruction fine-tuning using Daya. we additionally generate corresponding la- bels for each synthetic instance. Unless otherwise specified, we reuse LLM projection to annotate the dataset. The entire process of the PANGEA framework summarized in Algorithm Additionally, example prompts used in our experiments are provided in $ A.4 and A.5.
LLMprojsctionn
4 Experiments
4.1 Main results
To evaluate the effectiveness of our proposed PANGEA framework, we conduct experiments using established benchmark datasets (GSM8K [5], MedQA [13], FinQA [3]). We consider the following scenarios: 1) A dalu-scarce setting where only 100 domain-specific examples are available. To simulate this, we randomly select 100 samples from the training split. 2) A setting where the general data is entirely unrelated to the target domain. To construct this, manually exclude all math-, medical-, and finance-related entries from the CoT-Collection database [14]. We further explore an extreme domain-specinf scenario by introducing a Custom Domain-Specife Language (CDSL), an artificially constructed programming language; the accuracy of pre-trained models on this language indeed zero. Following the same design procedure in the previous experiments, generate synthetic data using 100 domain-specifie examples along with a domain-unrelated general dataset. Please refer B.2 for more details our experimental setup.
Table summarizes the comparative results of synthetic data generation frameworks using the 1,1 ama 3. 2 2B model. When fine-tuned on 10k, 30k, and 120k synthetic data generated by each

---

GSM8K
MedQA
FinQA
CDSL
24
19
Eval-Instruct 51
433
59
18
21
55
29
PANGEA (ours)
67
24
66
28
55
29
16
46
11
High (95-100)
Medium (90-95)
Low (0-90)
Figure 4: o1 scores for data quality analysis. The proportion (%) of generated data evaluated as High (95-100), Medium (90-95), or Low (0-90) quality by OpenAl's ol model.
Naive
Evol-Instruct
PANGEA (ours)
GSMBK
MedQA
FinQA
CDSL
1.08
0.88
1.36
1.44
Donsky
0.81
0.66
1.02
1.08
0.54
0.44
0.68
0.72
0.27
0.22
0.34
0.36
0.00
0.00
0.00
0.00
-14.5
-13.5
-12.5
-14.5
-13.0
-11.5
-14.5
-13.5
-12.5
-13.5
-12.5
-11.5
Influence score
Influence score
Influence score
Influence score
Figure 5: Influence scores for data quality analysis. Each histogram shows the distribution of data points for each augmentation method over influence score ranges. kernel density estimation curve is overlaid as a solid line for better visualization, and the vertical dashed line denotes the mean.
methodology, our proposed PANGEA framework consistently outperforms the baselines across all benchmarks and data scales. At the 10k scale, PANGEA already demonstrates substantial improve- ments the Naive and Evol-Instruct methods, with PANGEA achieving the highest performance. As the data scale increases to 30k and 120k, both PANGEA and PANGEA continue to outperform the GSM8K and FinQA benchmarks, showing notable performance
the baselines, particularly gains. Notably, the L. amam333 2-1B model fine-tuned with our framework (35.36) even surpasses the instruction-tuned 1.1 ama3 3-B (33.62) and Qwen2 5-7BB (23.76) models the CDSL. benchmark, demonstrating its effectiveness in building domain-specinf small language models.
Having demonstrated the strong performance of PANGEA, now turn to analysis of its success along two key aspects: 1) quality, which assesses how realistic and high-quality the generated syn- thetic data is (cf. 4.2), and 2) diversity, which measures how novel the generated data is compared to the original sources (cf. $ 4.3).. Ideally, synthetic data should be both distinct from the original data and of high quality; extensive experiments confirm that PANGEA excels in both aspects.
Analysis synthetic data quality
We conduct three complementary analyses: qualitative review of key data characteristics, quan- titative influence-score analysis of training impact, and human evaluation.
ol score. We randomly sample 5,000 synthetic examples generated by each augmentation method and evaluated them using OpenAl's ol model across five criteria: difficulty, formatting, problem relevance, language quality, and clarity. Consequently, each sample received quality from to 100, providing detailed view of the overall quality of the generated data (cf. $ B.3 for more experimental details). Fig. 4 summarizes the results across three score ranges: High (95-100), Medium (90-95), and Low (0-90). clearly demonstrates that PANGEA consistently achieves higher scores across all benchmarks compared the baseline methods, suggesting that ol judged the generated data to be higher domain domain-specific quality. The difference especially pronounced in FinQA, where the data structure is more complex.
ranging
Influence score. We also measure the influence score [16], which quantifies the impact of each data point the model's training. It allows us measure how much each data point contributed to

---

Source
Naive
E Instruct
PANGEA (aurs)
Two-dimenniooall t-SNE visualizations (MedQA)
Cosine distance
CDSL
FinQA
MedOA
GSMBK
Nalve
Evol-instruct
PANGEA (ours)
0.0
0.2
0.4
0.6
Figure 6: Sentence embeddings for diversity analysis. The first three plots show t-SNE visual- izations data generated by the Naive, Evol-Instruct, and PANGEA methods (colored dots), which augmented from the source data (white dots), on the MedQA dataset. The bar plot on the right summarizes the average pairwise cosine distance generated samples across datasets.
model performance. Fig. 5 illustrates the distribution of influence scores in a sign-log scale for each augmentation method across the datasets (GSM8K, MedQA, FinQA, and CDSL). Our proposed PANGEA approach consistently produces data points with higher influence scores, particularly in more specialized domains like CDSL, compared to the Naive and Evol-Instruct methods. It suggests that our framework generates data points that have greater impact on model training, leading to more effective learning and improved performance in downstream tasks.
Human evaluation. To complement automatic metrics, we ran human preference study with 57 graduate-level annotators. For each domain (GSM8K, FinQA, MedQA. CDSL), annotators ranked three synthetic candidates (Naive, Evol-Instruct, PANGEA) as Best/Second/Wone on 100 randomly sampled items. PANGEA received the highest Best preference on all four domains (avg. 50.6%). and differences were statistically significant by both x² and Friedman tests (p < 0.01). Inter-annotator agreement was moderate (Krippendorff's a 0.61). Full protocol, per-domain distributions, and statistics are provided in B.6.
Analysis synthetic data diversity
PANGEA augments source domain-speciff data by leveraging irrelevant general data, enabling it to generate more diverse synthetic data compared to augmentation methods that rely solely on limited source-domain samples. Recognizing the critical role of data diversity model performance [2], we assess the diversity of the generated synthetic data using the all all-mpnet - base-v2 embedding model from the Sentence Transformers library [24].
t-SNE visualization. We begin by qualitatively examining the diversity of the synthetic data through t-SNE [32] visualization of the embeddings. Fig. 6 shows two-dimensional embeddings of the syn- thetic data generated by each augmentation method (colored dots), alongside the original domain- specific source data (white dots), to assess how broadly the synthetic data points distributed rel- ative to their source. For the Naive and Evol-Instruct baselines, the synthetic data points are mostly concentrated around the source data, indicating limited diversity. In contrast, PANGEA produces more widely dispersed embeddings, reflecting a higher degree diversity in synthetic data.
Cosine similarity. To quantitatively evaluate data diversity, compute the average pairwise sine distance between embeddings of the synthetic data. For each of the 100 source instances, we generate 100 synthetic samples and caleulate the pairwise cosine similarity among them. As summa- rized in Fig. 6, PANGEA consistently achieves significantly higher average cosine distance than the baselines, indicating superior semantic diversity across all benchmark datasets. As result, the data generated by PANGEA is not only more diverse but also of high quality (as previously discussed in 4.2), demonstrating the overall effectiveness of the method.
4.4 Scalability & Robustness
Scalability & Robustness. We evaluate PANGEA along two axes, data scarcity and model capacity, to verify both robustness and scalability. First, to test robustness under extreme scarcity, we vary

---

Table 3: Model and seed scale results. (a) shows how performance changes the number of seed data increases from 10 to 100 under a fixed 10k synthetic set, while (b) reports the 8B-scale results with L1 allam-33. 1-8B, where PANGEA outperforms all baseline models.
(a) Seed-size ablation. Robust down to 20-40; drop only 10 duc to an under-specified profile.
(b) 8B-scale setup. With 8B scale, PANGEA steadily surpasses all basclines across every benchmark.
Method (fiseed)
GSMSK
MedQA
FinQA
CDSL
Avg.
Method
GSMSK
MedQA
FinQA
CDSL
Naive (100)
26.91
35.42
24.06
Pre-trained
48.75
39.31
25.63
1.61
Evol-Instruct (100)
27.36
36.29
26.68
5.22
23.89
Instruct-tuned
85.62
64.10
64.95
2.32
PANGEA (100)
32.52
37.78
36.44
11.30
29.51
PANGEA (80)
32.91
37.34
36.37
11.01
29.41
Naive
81.35
55.93
48.78
15.65
PANGEA (40)
31.84
36.10
35.21
10.15
28.33
Evol- Ivot-Instruct
79.08
60.02
53.88
17.42
PANGEA (20)
31.21
35.79
34.83
8.70
27.63
PANGEA (10)
29.28
33.27
28.31
3.77
23.66
PANGEA 86.47
64.51
65.31
25.91
the number of seed data from 100 to 10 while keeping the synthetic set fixed 10k. As shown in Table 3a, PANGEA remains strong with as few as 20 seeds and degrades gracefully only at 10, confirming that the quality of Synth-Profilligg governs performance under limited seed data.
Second, to assess scaling with backbone capacity, we use L. allam--3 3-1-B and find that PANGEA yields substantial gains over the pre-trained model and consistently surpasses all baselines on every benchmark (Table 3b), showing that the framework remains effective even for larger models.
Domain-agnose prompting & multilingual transfer. We replace task-specific prompts with uni- fied, domain-agnostic templates across all stages and find that PANGEA (agnostic) continues to outperform con-projection baselines, showing only small drop compared to the original domain- tailored prompts (Table 4). To evaluate cross- lesss-l robustness, we project English D, into Korean to synthesize 10k CSAT items from official seeds. PANGEA achieves the highest overall score and is the only method that successfully solves Hard items (Table 5), demonstrating that the profil- gr--uccroojoetioonn
ing-guide- projection pipeline remains effective even when the source and target languages differ.
Table 4: Dgnosic result. Unified tem-
Table 5: Korean CSAT. PANGEA achieves the best total and uniquely solves Hard items.
plates beat baselines, with a small gap original.
Method
GSMSK MedQA
FinQA
CDSL
Method
Easy
Normal
Hard
Total
Pre-trained
5.69
28.91
6.02
0.00
Pre-trained
40.00
13.64
0.00
13
Instruction-tuner
45.03
37.31
26.68
0.57
Instruction-tuned
60.00
31.82
0.00
27
Naive
26.91
35.42
24.06
3.20
Evol-Instruct
27.36
36.29
26.68
5.22
Naive
60.00
13.64
0.00
15
PANGFA (agnostic)
31.70
36.39
34.74
9.25
Evol- Iost-Insttuct
40.00
31.82
0.00
22
PANGFA (original)
32.52
37.78
36.44
11.30
PANGEA
60.00
36.36
5.26
34
4.5
Discussion
Ablation of prompt writer. As part of ablation study, we investigate the impact of the prompt writer component within the PANGEA pipeline, aiming to show that it plays a crucial role in effec- tively utilizing task-irrelevant general data. To this end, we evaluate PANGEA without the prompt writer, i.c., the proof-of-concept model introduced in $ 3.1. Even without the prompt writer, lever- aging general data itself yields strong performance, surpassing both the Naive and Evol-Instruct baselines that do not utilize general data. Moreover, integrating the prompt writer brings an addi- tional gain. In the case of our 120k-scale experimental setup in Table 2, PANGEA improves average performance from 38. 25% to 44.70% further enhancing our core idea of leveraging general data by providing guideline for effectively synthesizing domain-specific data from it. Details in $ B.2.
Generalization across architectures. To verify the generalization capability of PANGEA across different model architectures, we conducted additional experiments using Gemma Qwen2. 5- 5-1.5B 5B [39], and the DeepSeek -atSeeee-1--- -il--iiiiiilllseeee model is evaluated under the 10k-scale setup, while the others are evaluated under the 30k-scale setup. As shown in Table 6, PANGEA consistently delivers substantial improvements over the pre- trained weights across all backbones.
[31], [11]. The reasoning

---

Table 6: Architecture generalization. Benchmark results under our 30k-scale experimental setup (reasoning at 10k scale). PANGEA consistently displays significant performance improvements across all architectures. In particular, the reasoning model utilizes the DeepSeek- -R1 for labeling- The average performance improvement (+) is also shown for each model.
Benchmarks
Model
Method
GSM8K
MedQA (f)
FinQA
CDSL (f)
Avg. timpe)
Pre-trained
1.66
27.81
7.23
0.28
9.00
Gemmaa-2B
PANGEA
37.16
37.47
15.87
32.74 (+23.74)
Pre-trained
3.98
26.45
27.83
0.00
14.57
Qwen2.5-1.5B
PANGEA
41.92
32.25
40.84
16.48
32.87 (+18.30)
Pre-trained
5.69
28.91
6.02
0.00
10,16
Llama3.2-1B
PANGRA
38,36
39.98
41.41
17.68
34,36 (424.20)
DeepSeck-Rt Ree--RR--Diie
Pre-trained
85.29
57.09
61.29
6.96
52.66
PANGEA
88.91
66.53
69.75
28.70
63.47 (+1081)
Comparison to domain-specialized methods. To fur- ther highlight PANGEA's strength, we present compar- ative results with MuMath [40] and UltraMedical [45] datasets, which were constructed using domain-specifie strategies the mathematics and medical domains, re- spectively. Table 7 summarizes the results under our 30k-scale experimental setup. Notably, the proposed sur- passes such domain -peeiin-peeeillizedd baselines, further validat- ing the high quality and diversity of the data it generates.
Table 7: Comparative results. Bench- mark results on GSM8K and MedQA.
Method
GSM8K
MedQA
MuMath UltraMedical
30.25
36.71
PANGEA (ours)
32.52
37.78
then-rassoor approaches. It is worth noting that our ex- treme data searcity scenario, combined with reliance on out-of-domain general data, poses signifi- cant challenges for generating domain-specifie data. While we already conducted comprehensive evaluation against the Naive and Evol-Instruct baselines in our main experiments, there exists an- other line of work that also leverages large-scale general- purpose data, similar to our approach. For example, DataTTune [9] improves automatic dataset generation by utilizing existing, publicly available datasets. However, their method assumes that the general dataset is task-relevant but mis- Dg from the outset. This assumption aligned: that is, they retrieve examples from a domain-relewee domain--unelateer unrelated, making their approach less does not hold in our scenario, where D3 is entirely domain applicable. Indeed, on MedQA under our 10k-scale experimental setup, Data'Tune achieved only worse than 35.42% of the Naive baseline. This clearly highlights the limi- tations of existing retrieval-Be based synthetic data generation methods in our problem setting- where D, is extremely scarce and there is relevant examples in Dz- In contrast, our approach proves to be significantly more effective under such challenging conditions.
Further comparison with
34.95% performing
Conclusion
In this paper, we propose the PANGEA framework, which effectively generates high-quality syn- thetic data even in domain-specinn environments with extremely scarce data. By transforming unrel- evant general data into comainn-peccife synthetic data through a three-stage framework, PANGEA achieves superior diversity and quality compared to existing methods. Extensive benchmarking and quality evaluation experiments demonstrate that the data generated by PANGEA consistently main- tains high quality and diversity, enabling efficient synthetic data generation and improved model performance even highly specialized domain settings.
Limitation and future direction. Currently, PANGEA requires carefully designed prompts across its three stages, which entails certain level of human effort. Reducing this manual overhead while evolving the framework into more efficient and flexible solution for synthetic data generation presents promising direction for future work.
10

---

Acknowledgement
This work was partly supported by Institute of Information communications Technology Plan- ning & (IT))))ion grant funded by the Korea government(MSIT) (NN..RS-2199-119900 Artificial Intelligence Graduate School Program(KAIST); No.RS-2024-00 RS-20024-00009279, Global AI Fron- tier Lab; No.2022-0-00184 Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethies; No.RS-2025-0221113
AI Star Fellowship(Kooknin University); the MSIT(Ministry of Science. ICT), Korea, under the Global Research Support Program in the supervised by the (ITt(tntee for Infor-
Digital Field mation & Communications Technology Planning & Evaluation); and the National Research Foun- dation of Korea(NRF) grant funded by the Korea government( (SSIT) (RS-2021-NR0569) 2022RIA5A 708390812) 12). This work supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City.
NRF-
References
[1] Yung-Chieh Chan, George Pu, Apaar Shanker, Parth Suresh, Penn Jenks, John Heyer, and Sam Denton. Balancing cost and effectiveness of synthetic data generatioo strategies for lims. arXiv preprint arXiv:2409. 19759, 2024.
[2] Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. On the diversity of synthetic data and its impact on training large language models. arXiv preprini arXiv:2410. 242.2226,.,. 2024.
[3] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697-3711, 2021.
[4] Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnerructionfinetuun language models. Journal of Machine Learning Research, 25(70):1-53 2024.
[5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, and et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[6] Mike
Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm,
Shah,
2023.
URL https:/ htt:://wwwwt www. databricks eellle-nnsstee rnstiucttoontttt
[7] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
[8] Ronen Eldan and Yuanzhi Li. Tinystories: How small language models be and still speak coherent english? arXiv preprint (tt.::2205.0077,22000
[9] Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets. In Findings of the Association for Computational Linguisties ACI. 2024, pages 6453-6466, 2024.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[11] Daya Guo, Dejian Yang. Haowei Zhang Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
11

---

[12] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprini 2rXiv:2412. 2024.
[13] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng. Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
[14] Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: D D7omx8QyFP.
https://ppenzeiiew /tpp///wppeee.../e////e/ee net forum? ?id-
[15] Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant large language model alignment. Advances in Neural Information Processing Systems, 36:47699-47681, 2023.
[16] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[17] Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. arXiv preprint arXiv:2406.04886 2024.
[18] Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.1877 2023.
[19] Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang- On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406. 15126, 2024.
[20] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Em- powering mathematical reasoning for large language models via reinforced evol-instruct. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //openreview.. net/ ////fooo for Waridddd-MMMWOd
[21] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language mod- els with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpa:::///ppeevieieee net / forum? wtwaat//orrmmiiduuuusss...... UdUnUUSIIKK5..
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[23] Arina Razmyslovich, Kseniia Murasheva, Sofia Sedlova, Julien Capitaine, and Eugene Dmitriev. Eltex: A framework for domain-driven synthetic data generation. arXiv preprint arXiv:2503. 15055, 2025.
[24] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: /trpp:://axiv... org/abs/ 190011911111 10084.
[25] Weijieying Ren, Xinlong Li, Lei Wang, Tianxiang Zhao, and Wei Qin. Analyzing and reducing catastrophic forgetting in parameter efficient tuning, arXiv preprint arXiv:2402. /8000. 2024.
12

---

[26] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnad Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted train- ing enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations, 2022.
[27] Minju Sco, Jinheon Back, James Thome, and Sung Ju Hwang. Retrieval-augmented data augmentation for low-resource domain tasks. arXiv preprint ar arXhv:2402.13482, 2024.
[28] Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. Inject- ing domain-specifee knowledge into large language models: a comprehensive survey. arXiv preprint a0Xiv:2502.10708, 2025.
[29] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yim- ing Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36: 2023.
[30] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model. 2023.
[31] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.00118. 2024.
[32] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):5579-2055, 2008. URL http: // jmmr. wttppopers////pppppaaaooppppesst papers/ v9 19//aandermateen08.. html.
[33] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self seff-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 13484-13508, 2023.
[34] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[35] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
[36] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Represen- tations, 2024. URL
net forum wawrrrmmmddcchheeee wwwpicchhcnnnegpe.e.c
[37] Yiming Xu, Chenghao Zheng, Jialu Liu, and al. Magpie: Synthetic data for instruction tuning with no human labels. arXiv preprint wxXiv:2402.08525 2024.
[38] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing, 2024. URL https: /arxivv..v. org/ abs /2406. 08464.
[39] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1.Xi::4112.15115. 2024.
[40] Weihao You, Shuo Yin, Xudong Zhao, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath: Multi-perspective data augmentation for mathematical reasoning in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2932-2958, 2024.
13

---

[41] Xiao Yu, Zexian Zhang, Feifei Niu, Xing Hu, Xin Xia, and John Grundy. What makes a high- quality training dataset for large language models: A practitioners' perspective. In Proceedings of the 39th IEEF/ACM International Conference Automated Software Engineering, pages 2024.
[42] Arthur Yuan and et al. arXiv:2404. 09850,9 2024.
Craft: Distilling tool use for Ilm reasoning. arXiv preprint
[43] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302. 2023.
[44] Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, et al. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:2339. 11325, 2023.
[45] Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding. Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, al. Ultramedical: Building specialized generalists in biomedicine. Conference on Neural Information Processing Systems, 2024.
[46] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang. Jie Jiang, and Bin Cui. Retrieval-augmenne generation for ai-generated content: A survey. arXiv preprint arXiv:2402. 19473, 2024.
[47] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: Im chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024.
[48] Liammin Zheng. Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. Lmsys-chat-Im: A large-scale real-world Ilm conversation dataset. arXiv preprint arXiv:2309. /998, 2023.
[49] Ingo Ziegler, Abdullatif Köksal, Desmond Elliott, and Hinrich Schütze. Craft your dataset: Tpsk-specific synthetic dataset generation through corpus retrieval and augmentation. arXiv preprint (xXi:24409002998 2024.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The primary claims outlined in $ precisely align with our paper's contribu- tions and scope, as substantiated by comprehensive empirical results and analyses detailed in $ 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss the limitations of our method in $ 5, particularly the requirement of human effort for precise prompt design, and we suggest directions for future improvements.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper primarily focuses on proposing and validating novel synthetic data generation methodology through empirical experiments, without involving theoretical proofs or formal theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all necessary information for reproducing the main perimental results, including dataset descriptions, experimental setup, evaluation metrics, hyperparameters, and prompt details across the three-stage framework, clearly presented in $ and $ B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide detailed experimental settings, including prompt specifications in S A, comprehensive methodological and algorithmic details as well as data descriptions used in our main experiments $ B.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: : //tps/ips...... cc/
public/ /t/gguidees Code ss wiioooollcoo (omllicc) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nis. cc cttpcccccpcbbccccccccce public c/ gu quuid Codee Submi s s ciono onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
All experiments details are provided in $ B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments do not explicitly include error bars or statistical significance metries; instead, we focus primarily on comparative performance improvements and data quality evaluations across different methods and benchmarks, detailed in 4 and $ B.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Computational resource details are provided
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::///// http:////eurippeeeeee
Answer: [Yes]
Justification:
Our research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

Justification: Potential positive and negative societal impacts of our work are discussed in $ 5 and $ B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We describe appropriate safeguards for responsible use and release of our syn- thetic data generation framework, including controlled access and clear usage guidelines, detailed in $ 5 and C.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All datasets and models utilized are clearly listed in B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
Our research does not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our study does not include crowdsourcing or human-subject experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurlPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study involves no human subjects, IRB approval is not applicable.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our proposed PANGEA framework leverages LLMs for synthetic data gen- eration. For writing, LLMs were solely used for grammar correction; all original ideas and core methodologies presented this paper are entirely our own.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: neur ps cct.cc//ooooeeecssss///.o/os//// 2025 //LM) for what should should not be described.
21

---

A
Supplementary materials
We provide detailed prompts utilized throughout the PANGEA framework. We first present the step- by-step prompts used in our initial motivation study ($ 3.1). Subsequently, we provide the structured prompts employed within our three-stage framework, leveraging the Prompt Writer ($ 3.2). Finally, we include the annotation prompts for each benchmark dataset evaluated in our experiments.
A.1
Step-by-step prompts for motivation study ($ 3.1)
Our initial proof-of-coneept (Motivation) prompts were structured as follows. In this stage, data was generated in an end-to-end manner, where dg, represents the irrelevant general data instance and ds, denotes the domain-speciffe source data instance.
GSM8K prompt
You are an expert at transforming general questions into domain-speci fic, math-related questions, Your task is to generate only the transformed math question without including any answers or solutions.
Analyze and Inderstand the General Question Fully understand the general question's context, key concepts, and quantitative elements.
### STEP 1:
Icentify core topics (e.g., measurement, comparison, probability) and specific details that inspire mathematical transformations.
WH STEF 2: Refer to Domain-Specific Question (for Inspiration Only) Use the demain-specific questien to understand common formulations in mathematical contexts.
Extract core mathematical concepts and focus on transformation, not duplication.
Frame questions encouraging mathematical reasoning (e.g., multi-step calculations, logic, real-world applications).
### STEP 3:
Generate the Transformed Math Question
Create an original math questien retaining general question context. Integrate scenario, characters, objects clearly. Introauce mathenatical challenges explicitly requiring calculations, reasoning. Avoid solutions or numerical answers entirely.
comparisons, or probabilistic/ogge
Response Rule: Generate only the fransformed Domain questionMithout any answer, explanation, solution.
Output Format:
Transformed Domain Question: [Write your transformed question here.]
Task: General Question:
(dg;)
Domain Question: .**
(ds,
Figure 7: Motivation GSM8K prompt. Prompt used for GSM8K Benchmark Generation.
22

---

MedQA prompt
You are expert in transforming General Data and existing MedoR-style clinical queations into high-quallln diverse, and complex MeagA style clinical questions. Generate reslistic USXLE style multiple-shooe questions.
Step
Analysis Clinical Scenario Transformation
Convert General pata into clinical elements (demographics, lab values, timelines, events, exposeres, physiclogical proceases). Transform key events into realistic elinical scenarios (aceidents, expoaures, Lempozal shifts).
clinicsi contexts (symptoms, history, diagnostics). Ensure clinical relevencs lemergency medicine, infecticus disease, cardiology). Step Referenzing Existing Domain Question Structure Refer to existing MedgA questions (structure, relevance, complexity). Recreate di varsity scenario, symptom prograssion, diagnoses. Introduce new events, mechaniams, injuries, exposures, patient history. Step 3: Generate Transformed Data (Scenario (CQe) Develop Complex clinical Scenoric
Reinterpret
Marga Genersl nata with MedDA style for realistic patient scanarios tage, gender, travel, medication, family history, exposures). Add disease progresaion, comorbleicias, side effects, diagnestic errors. Create New Multippe-CChote Questions
Integrate General Dale and MedQA elinical elements for unique scenario. Provide answer optious (1 correct, 3 reievant distractors). Step Final Output (Restrictions)
Fraure acouracy, originality, logical conaistency.
Strictly output New clinical question
options only. No explanations, comments, anawers, reagoning.
output Format:
New Clinical Questien:
Question: <clinicel question here>
Choose of tha following:
A. <oation 1>
<option
c. <option 3>
<Cption
#+ key Considerations:
Convert Ganeral Data into medical (t'melines, labs, exposures).
Reference existing Xedga questions out create original scenario.
Diversify question types idiagnosta, treatment, prognosta, machonisms).
Ensure distractors are realiscie and clinically relevant.
Task
Caneral Question:
.**
(dg,
.**
existing KedOA style clinical question:
(do,
Figure 8: Motivation MedQA prompt. Prompt used for MedQA Benchmark Generation.
23

---

FinQA prompt
You are expert in transforming Ceneral Data (general knowledge everyday
questions, or non Tinancial tex:) into structured Pinancial Data and generating qugliquality, diverse, and complex financis. ressoning problems.
Your Enhanced Approach
Include;
#+* Step 1. Generel Data rinancial Context Mapping
Convart Caneral
a financo-ariented scenario.
Tdentify relevant financial events, busineas implications, investment scenarios, oz econemic contexts that relate to the General Data.
Ensure financial scenario logically consistent, reslistic, and distinct from Existing FingA-style financial reasoning quastion.
Step Pinancisl Centext Structured Pinancial Create realistic financial daza (revenue, costs, tax, investments, stecks). Trclude least two different financial variahles for complexity,
Generation
Generate data solely Irom Tinancial context derived from General Datá.
#+ Step 3, Pinancial Data Unique PingA-style Questien Generation kormulate multi-step coleslations, financial analysis, market evalustion, investment deciston-moking, or risk assessment question.
knsure complete originality and copying from existing singA questiens. Use new business scamorios derived from Genera: Data.
#4 Output Restriction:
Strictly generate New Financial Context and Question only. Meintain similar length and
Existing FingA-style question. no WT reference existing company names or finarcial detafls. pata.
Ensure financial scenario derived entirely No explanations, comments, or notes.
#* Output Fermat:
New Vinancial Context and Vssstion
please <Generated financial context with relevant numerical table data> reasoning question>
given financial question based the context.
Context:
Question: <Cenerated
#+ Eahanced Data Processing Flow:
Extract key themes from Seneral
Map themes to financial events icorperste, market, investmenta).
Generate ructuuee Financtal Data (revenue, costa, indi indicatora).
Create unique Linancial ressoning problem (complexity, originalityl.
#4 Strict Constraints:
DO NOT copy company names or figures from Existing FinQA question.
All data derived solely from General
Ersure completa originality.
Diversity numerical relationships and economic insighta.
General Data:
Id
Existing FinoA Linancial reasoning question:
(d..)
Figure 9: Motivation FinQA prompt. Prompt used for FinQA Benchmark Generation.
24

---

CDSL prompt
Step-by-Ster Prompt Example for Custom DSL: You are an expert converting Ceneral Data and Reference string-Manipulatae Task into fresh benchmark item that natches the reference style while remaining fully original.
Inputs:
General Data:
Idg,
Reference String Task:
(ds,
### STEP General Data Analysis & Ruilding-Block Construction (keep
private)
1. Invent the initial string -- 5-50 chars, quoted, contains >
separator.
2. Write one Geal sentence describing the intended transformation of
Select 3-6 operations (split, join, reverse, st) reverse_tokens, replace, substr, upper, lower, append, prepend).
3.
4.
Draft the chain (op (arg) in order).
5.
Compute the target string by mentally executing the chain.
Keep S, Goal, Ops, Chain, and hidden; never expose ther.
### STEP 2 Referencing Existing Task Structure
Examine Reference Task for leading verb/phrase and sentence length tone.
Craft one instruction sentence S words, starting with that leading verb/phrase, quotes S exactly once, uses vivid verbs, no ops or intermediate math.
### STEP Projection & Final Benchmark Generation
Goal: Publish final benchmark item in strict txo-line format. Header: New string-Transformatt
Task:
Problem-statement sentence (verbatim from Step 2)
Desired Output: "<т>" (directly after problem sentence), No Extras: Exactly two lines shown, no comments or explanations.
### HARD RULES
Reveal nothing from Step 1. Exactly three printed 1ines (header two content lines). Task requires 3-6 implicit operations. Never copy wording or digits from the reference; mirror only opener and sentence length.
Figure 10: Motivation CDSL prompt. Prompt used for CDSI. Benchmark Generation.
25

---

A.2 PANGEA: Profiling prompt ($ 3.2)
The prompt used in the initial profiling stage our three-stage PANGEA framework ($ 3.2) is presented below. In this stage, we provided our scarce domain-specinf source data (D,) to OpenAl's ol model and requested a detailed analysis of the dataset. Specifically, the model was instructed to identify key characteristics, structural patterns, and thematic elements of the provided data.
Iteration 1:
Initial analysis
"<Insert domain-speciin source dataset DD here>"
User:
You are an analyst tasked with transforming irrelevant general-domain data into {do- main_name) specific data. Carefully analyze the 100 provided domain-specific source ex- amples. Identify their key characteristics, structural patterns, and thematic elements. Suggest what types of information should be extracted or transformed from unrelated general datasets to create useful domain name data.
ol:
<model's 's initial analysis and suggestions>
Iteration 2: Refinement based on generated data
User:
Following your suggested approach, generated this synthetic example:
"<first synthetic data example>"
Given this outcome, what additional information or refinements do you suggest extracting or transforming from general data better align with the domain name domain?
o1:
<model's refined analysis and further suggestions>
(The iterative process continues until desired data quality is achieved.)
Figure 11: Synth-proflling prompt. Example dialogue illustrating the iterative profiling process used in the PANGEA framework.
illustrative example of the profiling prompt dialogue presented Figure Based on the initial analysis, the model generated structured guidelines describing how irrelevant general-domain data could be effectively transformed into high-quality, domain-specifte synthetic data. This profil- ing process was iterative: synthetic examples were first generated following the guidelines provided by the model, their quality was evaluated, and instructions were progressively refined based on feed- back if the generated data did not sufficiently meet the desired criteria. This iterative, conversational approach ensured continuous improvement of instructions and consistent generation high-quality synthetic data.
26

---

A.3 PANGEA: Prompt writer prompts ($ 3.2)
The prompt used in the second stage of the PANGEA framework ($ 3.2), which employs the Prompt Writer to generate structured Synth-Guie Blocks (r), is presented below. Only irrelevant general data provided as input to the Prompt Writer.
GSM8K Prompt-Writer's prompt
You are an expert at distilling everyday descriptions into concise, rsMeK-ready
quantitative building blocks. Your output feeds next stage, converting these
blocks into 2-4 sentence word preblem solvable by 5th- to 8th-grader in
multiple arithmetic or logical steps.
HF Inputs
General Question (raw text) (dg,) an*
### Choose Kid-Friendly Scenaric
Restate one everyday clause students can picture (shopping, chores, snacks, simple travel).
Remove unnecessary details.
### Select 3-5 Meaningful Numbers
Prefer numbers in General Questicn.
Invent realistic values if needed, supporting clear multi-step solution.
Use everyday units (dollars, minutes, km, items, "c, simple percents).
### Define Symbols Units
Assign symbols B, C, (D, B) with brief label and explicti icit unit.
#+ write 4-5 Simple Equations
Use only +, -, 4, &, single-step unit conversions.
No numbers;
chosen symbols intermediate results.
#*# State the Target
Specify final symbol solve and its numeric meaning.
#*# Outline step-by-Step Plan
Provice one bullet per equation, in order (4-5 steps).
### Output Format (use EXACTLY this template)
Quantitative Building Blocks:
Scenario <one kid-friendly clause>
Symbols - <A value unit label>, <B = ...?, <C = ...>l, <D = ...>l, <E ...>1l
Equations (1)
(2)
(3)
(4)
[55
Target <T description of numeric goal>
Flan (1) ... (2) ... (3)
(4)
[- (5) ...]
### Hard Restrictions Key Considerations
Scenaric familiar, everyday, agerapppori
fantasy/sci-fi.
A11 numbers appear Scenaric and Equations.
Exactly or 5 equations, requiring at least four steps.
NOT write/solve full word problem.
No examples, explanations, or commentary.
Begin exactly with "Quantitative Building Blocks:" and follow template strictly.
Figure 12: Prompt Writer prompt for GSM8K. This prompt instructs the model to analyze ir- relevant general data and explicitly extract structured information relevant to arithmetic reasoning, including scenarios, equations, symbols, and solution plans for GSMSK.
27

---

MedQA Prompt-Writer"s prompt
You are expert at transforming General Data into olinically useful clinfoal building blocks.
#4 Semeral-to-tliniee Kapping General
Idy.
#t 1. Analyze Seneral Data and every relevant numbers rengas
logical/causal relstionships
temporal progressions
physical prinatples, legal scenarios, or major events.
#4 2, Convert these elements inte clinical building blecks, such pstient demographics, timelines, physiologie/metaboll processes lab values, vital-sign ranges, exposures, accidents, disease progreasions.
3. Output format below:
transformed climics! building blocks
#4 Output Format
clinical Building Blocks:
-emoggappics
Timei ire
Key clinical events
....
Tab imaging patterns
Pathoptyysologie principles
#* Hard Restrictions
options, explanations. Output must begin exactly with "Clinical Building Blocks:" fallowed by bullet
generata questiens,
list
Figure 13: Prompt Writer prompt for MedQA. This prompt guides the model to analyze unrelated
general data and extract structured clinical elements tailored specifically for MedQA.
FinQA Prompt-Writer"s prompt
You expert in trenslating Genezal Data MinnnEinaciaal facts everyday acenarlos) into realistic, decalled, logically consistent tinancial scenario.
#++ Task:
Carefully analyze the provided
Data.
2. Clearly tranaläte the General Data into concise financial scenarto, explicitly aighlighting:
Specific and realistic finencial implications investment
Potential corporate decisions, econonic outcomes, or market
environments logically derived from data.
3. Scenario must be distinct, detailed, clear enough for later use in Stage 2 (atructured financial data and Finga-style quest oos)).
Do NOT produce numerical tables, numeric values, financial questions
5. Avoid ciosely replicating illustrative exempies or specific contexts from prior instructions; generate entirely original scensrio uniquely based provided Genersl Dats.
#+t Output (follaw EXACTLY):
Finacctal Scenario:
<Clearly describe original rinancial scenarlo derived from the provided Generai Data. Include explicit descriptions of financisl events, investment ogportunitias, econemic contexts, corporate implications that realistic and logically consisteat.>
## ded Genarz) nata:
id
Figure 14: Prompt Writer prompt for FinQA. This prompt directs the model to carefully analyze irrelevant general-domain data and extract financial scenario specifically for FinQA.
28

---

CDSL Prompt-Writer's prompt
You are expert at disti ling everyday descript lons into concise String Flow Building Blocks, Your output will feed the next stage, which converts theze blocks into complete String Plow DEL "zecipe" (code) that the intespreter can run.
#t Stage Generel Description * String-Flov suilding Blocks Ganeral Text (raw):
ida;
#+ 1. Define the Initial String 8
Single quoted string,
include at least one separator symbols.
Remove irrelevant dataila,
#4 2,
State the Desired Bnd-State I Iplain words) Describe in one sentence the intended final form of $ (e.g., "reverse the
join then with hyphens"), ## 3. Selact 3-6 Essential Operations
Choose split join reverse_str reverse_tokens replace (* to delete) substr upper lover append prepend.
Briefly Hist each operation with arguments needed (delimiler, replacement text, start/lengtå, etc.).
set (loading S) elways implied, emit from the List.
#+ 4. Draft Operaticn Chain
List operationa in order: (1)
(2)
(3) (3-6 steps).
node: operetion larg).
### 5, Give the Target String T iconcrete)
Write exact resulting string after all ogerations (Stage 2 correctness checking).
#+# Output Format exactly this)
String-Flow Building Blocks:
strings*
Goal <p splain-language description of ded result>
Ops <epl (arg1>, sop2 (arg),, kop3 (arg)>1,
Chain (11
(2)
(3) [+ (4)
(5)
(5) ...l
I *<target strings"
Haтd Ru Tips
Ops and Chain aust match exactly in orde: and content.
unneceasary operations arguments.
substr indices: 0-based [start, length).
upper/lower convert ASCII 2-2/a-2 only.
Everyday, age-appropriate
complex jargon.
to not add examples, explanations, or commentary. your
with "String-Flow Building BLooks:" exactly as above, nothing else.
Figure 15: Prompt Writer prompt for CDSL. This e explicitly instructs the model to analyze
unrelated general-domain data and extract structured elements required by the CDSL benchmark.
29

---

A.4 PANGEA: Projection prompts ($ 3.2)
Using the following prompts. we performed the projection described in 3.2. Here, d,, denotes the doeain-specifie source data instance, and the Synth-Guide Block (r) generated by the Prompt Writer provided as block, facilitating the generation of the final synthetic data.
GSM8K projection prompt
ace expect erafting GSM6K style problems clear, milti-step quantitative reasoning.
require
Quantitative Blocks Transformed Math Queation
Topots
Dtve Building Blocks
e**
[block)
Reference GSMBK Questien (Tor tone length only)
Ids;
#+ Adapt Structure & Inject Realiam
Mirror only tone, langth, complexity "eference; never copy content.
Length: axectly 2 to crtap sentences.
Scenario, Entitiea, Numbers, Required relationships frem Building Blocks inte natural stozy. each
Seamlesaly
value exactly once; repeating numbers.
No intermadiata calculations shown or mentioned.
#4 Compose the New Math Froblem (Projection) Preserve Context
Keep original selLing/chacacters
logical variants).
2. Embed Every Siven Number
numeric
must appear essential.
3. nemand Clear Mplti-Step Challenge
Solver must implicitly perform at least rive dietinct ateps. Avoid Superfluous Dats
No not invent extra numbers; recembine given numbers if needed. Maintain Numerical Censistency
Respect units; conversions must be infersble (e.g., minch, cmmm).
#4 Final Output & Bard Restrictions
Output CNLY the problem: text do answers, hints, explenations
Begin exactly with "Transformed Comain Question:".
to NOT reuse exact wording or aumaric values from reference SSM6x.
Problem must zequire at least
distinct steps Length: exectly 2 to concise sentences.
repeat numbera; never expose rttemmtdi/ reaults.
#4 Output Template transformed Vonain Question: <sentence 1> <sentence <sentence (optional]> <sentence 4 (opticnal and must be explicit quastiont>
Figure 16: Projection prompt for GSMSK. This prompt explicitly directs the model to project the Synth-Guide Block onto GSM8K-specific source data, ensuring the generated synthetic examples.
30

---

MedQA projection prompt
You are expert in crafting complex, CSMIR-sty (NAMR- format] mul mul:ipl welifple--ttte e-chol cened call clinical queationa.
# Inputs
Clinical Building Blocks
[block)
Existing XedgA style clinical question (structure reference only) (dy, a**
Reference versiry fy Existing Question ructteee
Use exiating question for Tormat, difficulty, diagnostic Nlow only; do NOT copy storyline.
Kecreate/diverait clinical context, symрtem progressicn, diagnoses. Add alinically relevant aventa tinjury mechaniams, toxis exposures, clinical suilding slocks.
detsiled history)
Projection MCD Creation (USXLE Blueprint)
1. Develop Complex Clinical Scenario
Concise patient vignette (ege, sex chief-complain: НРI PE/labs),
Insert 22
data (vitals, labs, imaging).
Integrate USMER clues (pathognomentc signa, imaging findings, signature lab patterna, elL-complications drug toxicity).
Trclude dynamic elements (progreasion, comorbiditiea, side-effects). Exposure history details (22: intensity mgin', duration, Irequency, route, protective equipment).
Create Multiple-Choice Question Stem: ask either (a) moat likely diagnosia or (b) next best atep appropriate management (only ons).
Provide exactly answer options (A-D):
One correct enswer (curzent guidelinee).
Three high-quality reslistic distracters (differentials plausible
suboptimal choices).
=1l options USMtR-standard medical concepts/actions.
Final Output
Restriccions
Verify clinical accuracy, originality, data coherence, consistency. Output Restriction: ONLY New Clinical Question answer options. keys, text.
explanations, rationsles,
Output
New Clinical Question:
Questions <elinicel question
Choose one of the following:
A. <Option
<bption
c. coption 3>
<0otion
key Considerations JUSMLE Focus--do output)
Embed guiding numbers (Nat 128 rEg/L, TSll 12 HIU/mL, EF 358).
viagnosis distractors: common Icok-alike éiseases.
Management distrectors: reasonable suboptimal contraindicated
theraptes.
Follow CSMLE/NDME consensus guidelines.
Figure 17: Projection prompt for MedQA. This prompt explicitly instructs the model to transform the Synth-Guide Block into synthetic examples aligned with MedQA.
31

---

FinQA projection prompt
You are expert in creating original, rea istfe, and complex FingA-style Financial reasoning problens based on provided inputé.
#++ Task:
Jsing these inpute:
rinancial Scenaric (Erem General Data)
Existinn FinCA-styla questicn (REFERENCE (KLY)
Generate completely new, high quality financial context and multi step reasoning question explicitly projecting Stage Financial Scenario into financial reporting narrative matching Existing FingA-style Osestion's style, comp.cxity, and detail.
#+ STRICT Requirements:
Narrative Context:
Emulate Existing Queation's narrative complexity with lengthy, complex sentences and detailed explenations.
Uetailed defaniticns/exp aeanttiins//xxllaatiinn 22 financial
acrenyms.
Use formal, passive-voice #inoncial-rpportingg Language (*wos recognized," anticipated
recorded,"
realized").
Include 21 explicit nate-reference clause ("net of tax, *see Note x"),
Describe rinancial eventa quallitativel,, without repeating numbers. (2) Financial Table (Markdown Pormat):
Precisely replicate Existing Question's table dimensions (rows, columns). identictl unit deseripter (*3 milliens,"
billiens").
original yet real istic financial labels/numerroaa data closely
matching Existing Questien's magnitude complexity.
explicit markdown formatting matching Existing Question.
(31 Follow-up up Narrative (Mandatory):
Exactly match Existing Question's length, complexity, style, depth. Provide deteiled qualitative insights without repesting table numbers, Explicitly zddresa profitahility trends, strategic ispacts, or riak management considerations.
control effectiveness,
advanced, Nulti-step Finencial Reascning question:
Require 22 sophisticaten cal calicolaiion steps (ratios, milti-year growth, weighted averages, financial adjustmente). Exactly match Existing Questien's complexity, brevity, and clarity.
StRiCtlY AVOTD:
Copying exact wording, company names, numbers fzom Existing Question. Overly sssplistic or ambiguoua single-step questiona, Repeating illustrative examples; all content must be original. Explanations, solutions, answers, additional commentary. Cmitting generaliring rollos-up Marrative.
OUTPUT FORMAT:
New Pinancial Context and Question:
Context: <Explicitly structured narrative context exactly matching Existing Pings Question's detailed sentence complexity, narrative length, passive voice, financial explanations, note references, and quelitative financisl descriptions.>>
Starkebwn--ormmttee financial table precisely matching dimensions, complexity, sumerical magnitude, digit length, exant unft deacriptor, with artirely original data from stage acenaclo.>
<Detailed Follow-up Narrative explicitly matching Existing Question's length, complexity, style, analytical depth; provide financial insights without repesting table numbers.> Question: <Explicitly stated advanced leve: multi step financial reasoning question exactly matching Existing questien's brevity, complexity, clarity, requiring 22 calculation steps.>.
Provided Inputs:
rinancial Scenario:
(block)
exiating FinDA-style question (REFFRRNCE, ONTY)
Id,,
Figure 18: Projection prompt for FinQA. This prompt instructs the model project the Synth- Guide Block onto FinQA-specific source data to generate high-quality synthetic data for FinQA.
32

---

CDSL projection prompt
You are expert crafting concise, multi-step raniputttiiooottiioon tasks
for benchmark datasets.
### Inputs
String-Flow Huilding Blocks .** (block)
Existing Generic String Task (style and format reference only)
*** {ds;) an*
### Use the Reference ONLY for Format Tone
Match length, phrasing style, and difficulty, but NOT copy storyline or wording.
Recreate transformation using the Building Blocks.
+Style-Mirroring Rules
Begin your sentence with the leading verb parase used in the reference task.
#H# Paraphrase Requirement
Turn bland "Goal" line into vivid instruction (s 60 words). Mention initial string s exactly as given (including quotes, separators, etc.).
Do not reveal internal operation names.
Frefer concrete verbs (strip, recrder, reverse, join).
### Create the New String- Trriig-rranfoormmtion Task
1. Problem Statement
One sentence satisfying Paraphrase requiremen..
2.
Desired Output
Present T in quotes its own line, labeled
"Desirea Output:".
### Output Restrictions
Output only lines shown below--nc explanations, extra text. Begin with exact header "New StrinngTranssoomation Task:".
Freserve line breaks exactly.
### Output Format (copy exactly) Task:
New
<Problem Statement>
Desired Output: "<T>"
Figure 19: Projection prompt for CDSL. This prompt explicitly guides the model in projecting the Synth-Guide Block onto CDSL-specific source data to produce high-quality synthetic data for CDSL.
33

---

A.5 Annotation prompts ($ 3.2)
The annotation prompt is used to collect responses for the generated synthetic data. By specifying a structured output format, this prompt facilitates both the model's tlain-of-thougee reasoning and the parsing of responses, thereby simplifying performance evaluation.
GSM8K annotation prompt
You are an expert in mathematical problem solving. Your task is to provide a detailed and logically scund solution to the given math problem. Each response must be accurate and based on rigorous mathematical reasoning.
IHI Response Rule
Provide clear step-by-step soluticns, including relevant mathematical principles and thecrens.
Explore multiple solution metheds if applicable; compare their efficiency.
Verify final answers through apprepriate validation methods.
Conclude clearly with "The final under the header **Answeree.
is: [Final Answer]"
### Output Format
**nswer**: (Provide detailed mathematical explanation here, inclucing step-by-step derivaticns, logical reasoning, and verification of calculations. Conclude explicitly with "The answer is: [Final Answer] [.]
### Your Task: * Questicn: (Question)
Figure 20: Annotation prompt for GSM8K. Prompt used to collect annotation responses for the GSM8K synthetic data.
MedQA annotation prompt
You are an expert in medical questien answering. Your task is to provide detailed, evidence-based respense to the given miltiple-choice medical question, accurate and aligned with up-to-date medical guidelines.
### Response Rule
Provice comprehensive explanation including relevant clinical reasoning.
Analyze all answer choices, explaining correctness incorrectress. Conclude clcarly with final answer format:
"The answer is: [Answer Letter]. [Answer Option]" under the header **nswwr**.
II Output Format
*<swwr***: [Provide detailed medical oxplanation here, including clinical reasoning, differential diagnosis, and evidence-based references.
Concl ude explicitly with "he answer is: [Answer letter]. [Answer option]".]
### Your Task:
Quuustion** (Question)
Figure 21: Annotation prompt for MedQA. Prompt used to collect annotation responses for the MedQA synthetic data.
34

---

FinQA annotation prompt
You are expert financial question answering. Provide detailed, evidence-based responses to given financial questions. Each response must be accurate, concise, and based on financial principles, accounting standaras, and quantitative analysis.
#+# Response Rule
Provide step-by-step breakdown of calculations, relevant formulas, and financial reasoning.
Clearly explain each step and how data points derive the final answer. Explicitly state and justify necessary assumptions. Conclude with Final Answer clearly boxed: \boxed(()).
### Output Format
:nsswer**:
[Provide detailed financial explanation here, including step-by-step salaulations, financial reasoning, and key insights.]
The answer is: Al Abxedd ( [[iinal Answer])) V
UHF Your Task: Queusticn:*** ** (Question}
Figure 22: Annotation prompt for FinQA. Prompt used to collect annotation responses for the FinQA synthetic data.
35

---

CDSL annotation prompt
You are an expert programming assistant familiar with custom interpreter
called StringElow (alsc known as SousChef). This interpreter uses cocking-thened commands to manipulate strings step-by-step.
Interpreter Command Quick Reference:
'pour' "string"; Set main string (broth) to speci fied string. 'slice "delimiter"'; Split broth into tokens (ingredients) by delimiter.
'stir "delimiter""; Join tokens (ingredients) into broth using
delimiter.
'flip": Reverse broth string.
'toss': Reverse order of tokens (ingredients).
'season "cld" "new"!: Replace eccurrences of substring old with new.
'fillet start length': Extract substring from broth (start, length).
'flambe': Convert broth to uppercase.
"simmer': Convert broth to Lowercase.
"garnish "string"'; Append specified string to broth.
'plate "string" Prepend specified string to broth.
'taste_then "substring" label': If broth contains substring, jump to label.
'move_to label'; Unconditionally jump to specified label.
'label LABEL NAME'; Define jump destination label.
'serve': Print current broth and end execution.
Given cooking-themed instruction, follow these steps: Analyzc and generate final command sequence according to StringglowInterpreeer rules.
1.
Explain step-by-step in detail how final command sequence derived.
2.
3.
Conclude with clearly indicated final generated command sequence:
### output format:
<your step by step explanation
The answer is: u**
<line by line final command sequence> .*.
Important:
NOT include original provided example; only step-by-step explanation clearly followed by final command sequence.
in each command line, Do NOT include comments.
### Your Task:
Questtion:* (Question)
Figure 23: Annotation prompt for CDSL. Prompt used to collect annotation responses for CDSL synthetic data. An additional detailed task description was provided to ensure accurate anno- tation by the model due to the novel and domain-specific nature CDSL tasks.
36

---

B Experimental details
B.1 Models and datasets
Models. The pre-trained weights used in our experiments are summarized below.
Ll a.aam3.. 2-1B: http:///nggiiiiicc..oommmmmmmmmm licensed under Llama3.2 Community License."
Qwen2. aave2.5-1.555 5-1.55. cense."
http://hinggiinggecccoooo/eeeeee licensed under Apache 2.0 Li-
Gemma 2B: http:://hnggginniioooooog//////g licensed under Gemma License."
Llama 3-70B-IIsstuucc
http://hggiiiiiiiccccoottt//m/mo - li-
censed under Llama 3.3 Community License."
Qwen2 S-1tb-Instruc::
http:://ngiiiggeeecc//xeeeeveee2 licensed
under Apache 2.0 License.*
DeepSeek-V3-0322: http://uwggiiiiiocooeeeeeeeceeep licensed un- der MIT License.?
Both our motivation study and the proposed three-stage PANGEA framework employed the I.l - ruct model for generating synthetic data. To ensure fair comparisons, we generated synthetic data for all baseline methods (Naive, Evol-Instruct, and Data'Tune) using the same Li alama3. -0B-IIst model.
Annotations the synthetic data were also performed using Llama 3. 3- Instruct, ex- cept for the CDSL. For CDSL, annotation accuracy with Llama3. -----Innttuut reached only 33.62%, despite detailed task instructions. Thus, we specifically employed the DeepSeek-V3 model annotate synthetic data for the CDSL benchmark.
Due to budget constraints, subsequent experiments utilized smaller pre-trained base models: 1,1 a.a3. 2-1B, 2 5-1 5B, and Gemma-22 22. We conducted supervised fine-tuning (SFT) using the Alpaca template [30], training all models with full-parameter updates.
5-1.56,
Datasets. The datasets used our experiments are summarized below:
COT-Collection [14] consists of 1.84 million Chain-offThhouht (CoT) augmented samples across 1,060 tasks derived from the Flan Collection, designed to induce CoT reasoning capa- bilities into language models.
GSM8K [5] contains 8.792 arithmetic reasoning questions requiring multi-stey logical reasoning and numerical calculations.
MedQA [13] consists of over 12,000 standardized medical exam (USMLE) questions, assessing models' medical knowledge and clinical decision-making abilities.
FinQA [3] includes over 8,000 complex financial reasoning problems extracted from real-world financial reports, testing numerical analysis and financial comprehension.
CDSL is introduced to evaluate models under severe data scarcity and extreme domain nov- elty. It involves transforming input strings into target outputs using a novel interpreter, StringF www.wwteeeee.....mo. with entirely unseen commands and functions. The dataset tains only 100 manually curated training examples and 345 test instances.
Since our experimental setting involves an extreme scenario of data scarcity in highly spe- cialized domains without any related general-domat data available, we constructed our irrel- evant general dataset (Dg) from the CoT-Collection. To ensure strict irrelevance, we man- ually removed samples related to mathematics, medical, finance, and coding tasks by using
3 hhttps:/wwww. http://wwwwwllaaa..mmm http://wwwww.ma....llalaaaaaaaaa com/llama332/liccses/
"https:/www ww dew/gemma//trms
s gottp:://ii.google... hhtps://iigooglleedv/gem//mmnnt
"https://www www.www.ww..mmmm.mmmmm...m.....p com/llama3 3/license/
37

---

3- and 5wenn/0we22.5-728-111 random seeds.
across five different
For the GSM8K, MedQA, and FinQA benchmarks, we randomly selected 100 samples from each dataset's training split to form domain-specific source dataset. In the of the CDSL, which includes 100 manually curated training examples and 343 test samples specifically constructed for evaluating models novel and specialized domain settings, directly utilized the entire of 100 training samples as our domain-specific source dataset (D,).
Our proposed benchmark, Custom Pomain-Speciff Language (CDSL), is specifically designed to evaluate language models' capabilities in highly specialized domains and extreme data scarcity set- tings. The task involves transforming an imput string into desired output by generating executable code conforming to a novel interpreter ietringF" Twwnteerreeee detailed in Algorithm 2). Unlike conventional programming languages commonly used scripting syntax, interpreter features uniquely crafted mooking-themedd commands, enhancing domain specificity and novelty.
This cooking metaphor-based interpreter maintains internal state representations (such broth for strings and ingredients s for tokens) and executes instructions through commands named after culinary actions (e.g.. slice, stir, fl alab))). This distinctive structure not only empha- sizes domain novelty but also ensures the synthetic generation process must capture highly specific instructions and contextually relevant transformations, effectively evaluating the adaptability of syn- thetic data augmentation methodologies in novel enviromments.
Specifically, for annotating the CDSL benchmark, we employed the Deepseek-V3 model. The generated code, enclosed within triple backticks () was extracted and compiled. Instances where the code failed to compile or produced outputs inconsistent with the labels were filtered out.
38

---

Algorithm CDSL: Interpreter
class aaring:lovinterpeeee:::
def
init (sclf)
self-breth
ingrodiencs
5
self, salf-stations 13
labell
self.pe D
selr. cocking True
canning
10
def (rop( rocipe_stx):
pazze ozop
Line recipe aplit Sppiit/Aee ('All: 1ise.atssp(
13
if not disht
14
continue
te.
Flatst shi reeeeee
17
else:
18
acep, vares aele. dice (dish) rel.frecipe.egeene (step, arga))
21
def di de lines:
split Line dice
return findall (AMPAAAAA"N lire!
cook (selEts
run cook
xe1f.pc =
while self.cucking and well 1sn fael c.cccigu)
step, ange recoupe (eel self F-aaate (atepr angst
execute dest
29
self.pe
return se f.broth
def saute (selt, step, argsl:
33
if step -n "pourhs self, brroth wel.. www...eee (arges)
ser
34
step "glicce;;
SP117
37
delin = peel (argat0))
ingredengsss ee.t.broth. split imi
olif stcp "stir":
J018
delim se peell (argstd))
zell F.brthh dellin delim.jonn Is Fningrediente ent
elif step "Elip"s
REVERSK SIR
sel ..both sel brbochl
elif step S01f wol..ingeedeent..oonaa povarsee
49
elié "season":
ACPLACK
new map peclt.pocl, arqo[:213
beothbroth selt. wo...brrcch.repacce.o. new!
53
elif "Eillot":
SUBSZR
stert, length paptint, argel boothbroh selr. a.s.broch/staatttttto leaeth]
57
elif "tianbe":
www..bboch..ppee... UPPER
elif step "simmer"; weit.broch.Lowerr
LONBR
elif step "garnish"; solf broth + self. (args[01)
APPSND
65
st40 "plato*:
BREPEND
solf.brost broth (args seit.broth
69
elif "caste_then": substr, subetr in belt.broth:
IPSTRCONTAING
(arqq arçs[l]
73
selc.pe solf.stations [1abe]]
step "nove_lo"s
solf.pc wowr, [arqs[01] -
77
elif
print "Sorva": Colt.ccoking Falce
5RIN2
ValueEeror Uninnown steps (slep)")
def (sol1 21. alll s.stettaiihh(")) s.embsiith(""))
__arrip_ + poel
39

---

B.2 Main experiments detail & example
In our main paper, we selected Naive and Svol-Instruct baseline methods for comparison, as other approaches such as DataTune [9] showed limited effectiveness in our scenario (e.g.. achieving only 34.95% accuracy on MedQA the 10k scale when using irrelevant general data). Thus, we assessed how effectively these selected baselines could generate diverse and high-quality synthetic data from general-domain data.
We generated synthetic datasets at scales of 10k, 30k, and 120k, subsequently evaluating improve- ments in model performance across various benchmarks. Additionally, we also evaluated our initial proof-of-concept (motivation study) across the same three data scales. All benchmarks were tested under zero-shot setting, where performance was assessed by directly parsing model-generated responses and comparing them to the ground-truth labels.
Table 8: Comparison of synthetic data generation frameworks. Accuracy (%) four bench- marks across synthetic data scales (10k, 30k, 120k). Our PANGEA framework consistently outper- forms baselines, with larger improvements at increased data scales. All evaluations performed zero-shot setting.
Benchmarks
# Synthetic
Method
GSM8K (1)
MedQA (1)
FinQA (1)
CDSL (1)
Avg. (impr.)
Pre-trained
28.91
6.02
10.16
Instruction-tuned
45.03
37.31
26.68
0.57
27.40
Naive
26.91
35.42
24.06
3,20
22.40 (+12.24)
Evol-Instruct
27.36
36.29
26.68
5.22
23.89 (+13.73)
10k
Motivation (ours)
32.14
36.76
32.08
8.89
27.47 (+17.31)
PANGEA (ours)
32.52
37.78
36.44
11.30
29.51 (+19.35)
Naive
34.72
34.24
27.46
5.51
25.48 (+15.32)
Evol-Instruct
32.51
38.09
29.64
9.57
27.45 (+17.29)
30k
Motivation (ours)
35.48
39.12
40.19
10.15
31.24 (+21.08)
PANGEA (ours)
38.36
39.98
41.41
17.68
34.36 (+24.20)
Naive
42.68
38.02
32.43
6.96
30.02 (+19.86)
Evol-Instruct
38.73
42.34
33.74
13.04
31.96 (+21.80)
120k
Motivation (ours)
46.02
42.58
43.07
21.31
38.25 (428.09)
PANGFA (ours)
48.61
44.62
50.22
35.36
44.70 (+34.54)
As demonstrated Table 8, our proposed three-stage PANGEA framework consistently achieves superior performance across all benchmarks and data scales compared to baseline methods, ex- hibiting particularly significant improvements as the data scale increases. Notably, even our initial proof-of-concept (Motivation) outperforms the Evol-Instruct baseline on most benchmarks, indi- cating its robustness and effectiveness in leveraging irrelevant general-domain data. However, the performance gap between Motivation and Evol-Instruce is marginal some cases, highlighting the enhanced capability and stability introduced by our structured three-stage PANGEA framework.
Resource & hyperparameters. For main experiments, utilized four NVIDIA H100 GPUs. All models were trained with Fully Sharded Data Parallel (FSDP) using the Adam W optimizer with learning rate of 10-5 total batch size (batch size gradient accumulation steps) of 64, sequence length of 4096, and a cosine scheduler with a warm-up ratio of 0.03 over three epochs. Inference with Deepseek V3 conducted using its external API, whereas inference for Llama models was performed locally.
40

---

Example. Here, we present examples of synthetic data generated using our proposed three-stage PANGEA framework. Table provides examples of the general-domain data alongside the cor- responding Synth-Guide Blocks generated for each benchmark by the Prompt Writer. Table 10 illustrates how these Synth-Guide Blocks are subsequently projected into synthetic examples.
Table 9: Examples of Synth-Guide Blocks (r). Synth-Guide Blocks generated from general data, tailored specifically to each benchmark using proposed three-stage PANGEA framework.
General Data:
Two space explorers discover two unnamed planets an unknown galaxy and name them after themselves. Sarahn is undergoing major tectonic shifts, while Timon is geologically stable. Question: Which planet's volcanic crust is hotter?
GSM8K
Scenario: Sarahn and Timon are comparing crust temperature of their planets: Sarahn's 's crust is 48 "C, Timon's crust is half as hot (0.5 x), and they note that the temperature difference equals twice (2 x) Timon's 's crust temperature.
Symbols:
= CC -araah''s crust temperature
-B 0.5 "half" factor
-C=2 "ttwice" factor
Equations:
Target: T difference between the D=) (2) C=
plancts' crust temperatures (CC)
Plan: (1) Get A + (2) Find D by multiplying and B + (3) Compute T by doubling D
+ (4) Calculate T= A-D
MedQA
Demographics: Two space explorers, planets named Sarahn and Timon
Timeline: Current large geographic shift on Sarahn, stable period on Timon with no
specified duration
Key events: Continental break apart and movement on Sarahn
Imaging/labs: Continental treak-apart and movement Sarahn
Pathophysiology: Geologic processes such as plate tectonics and volcanic activity, potentially leading to increased heat and volcanic crust temperature on Sarahn compared
to Timon
FinQA
A private space exploration firm, Stellar Frontiers Inc., has discovered two previously unknown planets in distant galaxy, naming them Sarahn and Timon after their lead explorers. Initial scans reveal stark geological differences between the two planets, presenting unique investment opportunities and risks for resource extraction ventures. Sarahn is undergoing signifieant tectonic activity, with continents fracturing and shifting rapidly. This instability suggests high volcanic activity, making the planet a prime candidate for mining rare geothermal energy resources or extracting valuable minerals formed under extreme heat and pressure. However, the volatile crust poses high operational risks, requiring advanced (and costly) drilling technology and insurance against geological disruptions.
Timon contrast, exhibits remarkable geological stability, with no continental movement detected. This makes an attractive site for long-term infrastructure investments, such as establishing permanent research bases or low-risk mining operations for stable mineral deposits. The lack of tectonic activity implies lower energy yields from geothermal sources but offers predictable costs and safer exploration conditions. Market implications: Sarahn: Short-term, high-rewand ventures tied to volatile resources, requiring risk management strategies. Timon: Appeals to conservative investors seeking steady returns.
Corporate decisions: Stellar Frontiers Inc. must decide whether to license exploration rights to third parties (favoring Timon's stability) vertically integrate operations exploit Sarahn's high-risk, high-reward potential. Regulatory bodies may also intervene, imposing environmental safeguards for Sarahn's 's fragile ecosystem. This scenario sets the stage for analyzing risk-adjusted returns, capital allocation strategies, and sector-specific market reactions.
CDSL
S - "Sarahn: hot; Timon: cold"
Goal - Swap temperature descriptors.
Ops spwitpllppp... ("; "), zeverse_toke tokens (), join (", Chain split + (2) reverse tokens (3)
T-TTmmn: cold, Sarahn: hot"
41

---

Table 10: Example of projection output. Domain- sppciiii synthetic data generated by projecting Synth-Guide Blocks onto scarce domain source data using our proposed PANGEA framework.
GSM8K
Sarahn and Timon discovered two planets, where Sarahn's 's crust temperature is 48 "C. and Timon's 's crust half hot due to lower geographic activity. If the temperature dif- ference between their planets twice Timon's 's crust temperature, what is the difference in their crust temperatures?
McdQA
35-year-old space explorer on planet Sarahn presents with a 2-week history wors- ening respiratory symptoms, including cough and shortness of breath, following a re- cent continental break-apart event. The patient's oxygen saturation is 88% on room air, and a chest X-ray shows bilateral infiltrates. Which of the following is the most likely pathophysiologic mechanism contributing to the patient's 's symptoms? Choose one of the following:
A. Increased volcanic crust temperature leading to sulfur dioxide gas exposure
B. Decreased atmospheric pressure due to planetary geographic shift
C. Inhalation of particulate matter from lectonic activity
D. Hypobaric hypoxia resulting from high-altitude terrain formation.
FinQA
Context: Stellar Frontiers Inc., a pioneering private space exploration firm, has cently announced the discovery of two exoplanets, Sarahn and Timon, located in distant galaxy. Initial geological assessments have revealed divergent characteristics that present distinct investment opportunities and operational challenges for resource extraction ventures.
Sarahn, named after lead explorer Dr. Elena Sarahn, exhibits intense tectonic activity, with continental plates undergoing rapid fracturing and displacement. This geologi- cal volatility suggests the presence of rare minerals formed under extreme heat and pressure, such as superconductors and platinum-grouu metals, which are highly sought after in advanced manufacturing, However, the instability of the planet's ' crust necessi- tates the deployment of specialized drilling equipment, which carries significant capi- tal expenditure (CapEx) and operational risk, including potential project delays due to seismic events. These risks are further compounded by the need for comprehensive in- surance coverage against geological disruptions, as outlined in Note 3 to the financial statements.
Timon, named after geologist Marcus Timon, displays remarkable geological stabil- ity, with no detectable lectonic movement. This environment is conducive long-term infrastructure investments, such as modular research facilities low-risk mining oper- ations for stable mineral deposits like iron and silicon. The predictability of Timon's 's conditions reduces both CapEx and operational expenditure (OpEx), making an at- tractive proposition for conservative investors, including pension funds and aerospace conglomerates seeking sustainable returns.
The table below summarizes the projected financial implications of resource extraction ventures on Sarahn and Timon over a five-year period, adjusted for inflation and risk premiums (see Note 4 for detailed risk-adjustment methodology). All figures are pre- sented in S billions, net of tax and exploration licensing fees.
2024
2025
2026
2027
2028
Saruhn Revenue
S1.2
$2.8
$4.5
$6.1
$7.9
Sarahn OpEx
$1.5
$2.3
$3.0
$3.8
$4.6
Timon Revenue
S0.8
$1.6
$2.4
$3.2
$4.0
Timon OpEx
S0.6
$1.0
$1.4
$1.8
$2.2
The financial projections underscore the divergent risk-return profiles of the two exo- planets. Sarahn's 's revenue growth trajectory is steep, reflecting the high-value nature of its volatile resources. but this is offset by escalating operational costs tied to risk mitigation and technological adaptation. Timon, while offering slower revenue growth, demonstrates superior cost efficiency due to its stable environment, resulting in more predictable margins. Notably, Sarahn's 's OpEx sistently high. indicating the ongoing financial burden of geological risk management. In contrast, Timon's 's OpFxx ratio declines steadily, highlighting its suitability for long- term, low-volatility investment strategies. Regulatory considerations, particularly en- vironmental safeguards for Sarahn's 's fragile ecosystem (as discussed in Note 5), may further influence capital allocation decisions.
pereentage of revenue remains per-
Question: What is the cumulative net profit (revenue minus OpEx) for Stellar Frontiers Inc. across both exoplanets from 2024 to 2026, and how does this compare to the net profit in 2027 alone?
CDSL
Given "Saran: hot; Timon: cold", reverse the order of the entrics, swap the separators to commas, and keep the temperature descriptors unchanged. Desired Output: "Timon: cold, Saran: hot"
42

---

B.3 ol score & influence Score
ol score. To quantitatively assess the quality of synthetic data generated by each augmentation method, we conducted an extensive evaluation using OpenAl's ol model. We randomly sampled 5,000 synthetic examples from each method (Naive, Evol-Instruct, and PANGEA) and evaluated each example based on five key criteria: difficulty, formalting, diffcculy.formutinng problem relevance, language quality, and clarity. Each criterion was scored from 20 points, yielding an overall quality score between O and 100 sample. The detailed evaluation rubric tailored specifically for the GSM8K benchmark presented in Fig. 24.
Representative examples from each augmentation method, categorized into three distinct quality lev- els- (95-100), Medium (90-95), and Low (0-90>-llong with their respective ol evaluation scores, are shown in Tables 11, 12, and 13. These examples qualitatively illustrate the variations in synthetic data quality across methods, supporting the quantitative results summarized in Fig. 4.
GSM8K Grading Rubric
Candidate Item
*** (questions]
(Five compenents, equally weighted; 0-20 pts each)
1.
Potentials
Does prompt give every fact required to compute single, unique answer? Prompt must not include worked sclution steps.
2. **Presentation Clarity+*
Is wording clear, unambiguous to average reader? Sections/tablle optional; case understanding key. Prompt must *not* contain answer or derivation.
3. **Mathematical Soundness**
Are numbers, relationships, conditions consi issett and error-iree?
Single-step or multi-step reascning acceptable if correct.
4. *CConiseneess Readability**
Is wording free of unnecessary jargon/filler? Minor formatting quirks not penalized if comprehension intact.
5. **Determinacy Appropriatenesse*
Does problem guarantee exactly one well-defined answer?
Is difficulty suitable for GSMBK (basic arithmetic, percentages, ratios, etc.)?
Scooring:** Sun of five component scores total 0-100.
Figure 24: Evaluation rubric for synthetic data quality (GSMSK). This rubric, utilized by Ope- nAl's ol model, assesses each synthetic data sample across five equally-weighted criteria, with each criterion scored from 0 to 20 points, resulting in a total quality score from 0 to 100.
43

---

Table 11: Naive Method Examples. Representative examples of synthetic data generated by the
Naive method, grouped by ol evaluation scores (High: 95- 95-100, Medium: 90-95, Low: 0-90).
Result
Generated Synthetic Data
Low
If Deacon was years old when released his third album 2009, how old was when he was born in Baltimore, Maryland?
Midium
A freight train rolls 120 mi on flat prairie and then 80 mi through hills. On flat track it normally burns 2 gal of diesel per mile. The trip's s fuel gauge shows 300 gal used in total (this already covers the tiny fuel spent during brief station stops ignore that). About how many gallons per mile does the locomotive use while on the hilly stretch?
High
At the start of 30-day month, Li Beibei receives 60 yuan in pocket money and, spending exactly yuan on lunch each day, how much money will she have left by the end of the month?
Table 12: Evol-Instruct Examples. Representative examples of synthetic data generated by the method, grouped by evaluation scores (High: 95-100, Medium: 90-95, Low:
0 90).
Result
Generated Synthetic Data
Low
Ethan once sorta-berrowed $2,500-ish from Lucas at upproximately 15% interest that may or may not exist. Ethan flung $300 hack sometime while Lucas was napping. It's now both exactly and not exactly one year later. How much random cash should Ethan hurl at I Lucas they can declare the debt mysteriously settled?
Midium
Tom repainting two rooms in his office building: first, rectangular conference floor measures
whose that room he intends to cover
by and whose walls 12 high, only the two longer walls-cach of which contains a single ft ft doorway that will unpainted- and second, a perfectly square meeting room 15 ft on side with walls of the same height, where he plans coat all four walls except for three identical 3 ft 4 ft windows (one window set into each of three different walls while the fourth wall has openings). Ignoring ceilings, floors, trim, and any other fixtures, how many square feet wall surface will actually receive paint?
High
Lily's phone plan costs $45 per month. For the first three months she pays a promotional rate of $25 per month. In the 6th month she incurs an international roaming fee of $18. Assuming all other months are billed at the regular rate and taxes included in the prices, what is her total payment for the first months?
Table 13: PANGEA Examples. Representative examples of synthetic data generated by the PANGEA method, grouped by ol evaluation scores (High: 95-100, Medium: 90-95, Low: 0-90).
Result
Generated Synthetic Data
Low
Carrying the $40 he saved over the summer, a student heads to the campus bookstore and picks out three textbooks the first carries price tag of $18, the second rings at $4 less than that, and the third, sitting the new-releases shelf, is exactly $2 more than the second; with tax waived for the hack-to-school promotion and free bookmark tossed what is the price of the third textbook?
Midium
A runner's GPS watch shows she covered 10 miles in hour 30 minutes; given this, first convert the total time to minutes, then use that time to find her average speed in miles per hour, and finally determine (to the nearest second) her average pace per mile-how many minutes and seconds does each mile take?
High
Alex saved $150 to cover all the costs of getting his driver's license: the state imposes a $50 base license fee and a $30 road-test fee, each which incrcased by 10% sales that is applied once the respective fee, and he must also pay S12 required ID photo, which is not taxed. After paying every fee and the applicable tax, how many dollars remain from Alex's $150 budget?
44

---

The examples provided in Tables 11, 12, and 13 clearly illustrate qualitative differences in the syn- thetic data generated by each method. While the Naive method often generates overly simplistic or poorly structured questions, Evol-Instruct tends to create scenarios that, although slightly im- proved, still frequently include irrelevant details or overly complex phrasing. contrast, PANGEA consistently produces clear, contextually coherent, and precisely structured examples. Specifically, PANGEA -egneraedd problems demonstrate more realistic scenarios, appropriate complexity levels, and more accurate and concise wording, resulting in higher overall evaluation scores by OpenAl's ol model. This qualitative distinction highlights the effectiveness of PANGEA's structured, three- stage approach in generating synthetic data that is both domain-relevan and diverse, addressing key limitations observed in other methods.
Influence score. We measure the usefulness of the synthetic training examples generated by our PANGEA framework via influence scores estimated with the DataInf method of Kwon et al. [16]. We denote the domain-specins source dataset by D. and use it exclusively as the validation set when evaluating how each synthetic example affects model performance. After training our baseline model, we perform additional round of low-rank adaptation (LoRA) fine-tuning on the union of the synthetic set Dyn and D., and the resulting parameters are used only to compute gradients for the influence-score estimator.
Classical oufltence-functionn approaches require the inverse of the Hessian H (0). whose exact com- putation or iterative approximation is prohibitive for modern LLMs, while Datalnf circumvents this bottleneck by approximating the Hessian inverse in closed form using first-order gradients and Sherman-Moriss correction, reducing the cost to O n.d.) time and (mmxx d:) memory while retaining high correlation with exact scores.
Let the synthetic training set be D = Dayn {(x,, D:) Bi) the validation set D Ds = [(ata) v, } and So the LoRA-adapted model; the influence of a synthetic example (ær,, yn) on validation performance is
(ul gn.) (T, 13
Datalnl (Iks y/k)
1/ 91.k
A
n2
A + 91
where Ut Vo, (v,, fo(a;)) and 1
Vo, Au, fo(az)")), and where the damping term A 0.1 nd, Σ I. with di the parameter dimension of layer I.
Naive
Eval-Instrus
PANGEA (ours)
GSMBK
MedQA
FinQA
CDSL
1.08
0.88
1.36
1.44
0.81
0.66
1.02
1.08
Density
0.54
0.44
0.68
0.72
0.27
0.22
0.34
0.36
0.00
0.00
0.00
0.00
-14.5
-13.5
-12.5
-14.5
-13.0
-11.5
-1..5
-13.5
-12.5
-13.5
-12.5
-11.5
Influence score
Influence score
Influence score
Infiuence score
Figure 25: Inffuence scores for data quality analysis. Each histogram shows the distribution of data points for each augmentation method over influence score ranges. A kernel density estimation curve is overlaid solid line for better visualization, and the vertical dashed line denotes the mean.
As shown in [16], the sign of Dastt matches the direction of the validation-loss change; hence Daaatet (xa, yk3 < implies that the sample lowers the validation loss and is therefore beneficial. marge-m negative scores thus pinpoint the most influential, high-quality data.
Figure 25 plots the signed scalsd distribution of the estimated scores, showing that PANGEA- generated synthetic examples yield larger negative influence values than all basclines and therefore lower the validation loss on D, more effectively.
45

---

B.4 Embedding visualization with Cosine similarity
Embedding visualization. To qualitatively analyze the diversity of synthetic data generated by our framework, we visualized embeddings of 30k synthetic data points along with original source data across all benchmarks using the all -ap----p-aasseee clearly illustrating improved diversity and coverage.
model and t-SNE projection (Fig. 26),
GSMBK Naive
GSMBK Eval-instruct
GSMBK PANGEA (ours)
MedQA Naive
MedQA Eval-instrus
MedQA PANGEA (ours)
FinQA Naive
FinQA Evol-instruct
FinQA PANGEA (ours)
CDSL Naive
CDSL Evel-Instruct
COSL PANGEA (ours)
Augmented
Source
Figure 26: Embedding Visualization of Synthetic Data Diversity. t-SNE visualization of embed- dings for 30k synthetic data points generated by Naive, Evol-Instruce and PANGEA methods, along with the original domain-specific source data.
46

---

Table 14: Cosine similarity for measuring synthetic data diversity. We compute pairwise cosine similarities between synthetic samples generated from the same source instance.
Method
GSM8K (1)
MedQA ())
FinQA (4)
CDSL (4)
Naive 0.776x0:003
06.0000
0.872+0.003
0.906+0.003
0.838=0.003
Evol-Instruct
0.847+0.002
0.742+0.002
0.767=00008
PANGEA
0.310=0:000
0.600+0.004
#0.01
0.377=00001
Cosine similarity. Furthermore, to quantitatively assess the diversity of synthetic data, com- puted average pairwise cosine similarities between synthetic samples generated from each of the 100 source instances, with each source instance producing 100 synthetic samples. Table 14 sum- marizes these results, where lower cosine similarity values indicate higher intra-source diversity. PANGEA consistently achieves substantially lower similarities across all benchmarks compared to baseline methods, clearly demonstrating its superior capability to generate numerically diverse and distinctive synthetic samples.
B.5
Data Tune failed example
Expaned Descriations
Plan:
Diriccll scenaris- based multilee-hoice
the provided paragraph exiating
questions intended test medical knowledge, clitical scenaroo clearly include
Identity
characteristics leg. geological changes
Description
denograpiis, conslderations,
diagnesia focused
Rewrite paragraph
ciniccl scenario, interpreting geographic
A y-ar old giri brought the physicion for wal-child oxamination She has no history of prlyteochoride
biological mechanisms, Provide plausible pathogen--eleee
geotogical festures mesaphorizally citiicl symptoms patient characteristics.
medically relevant cholces, with one correct anower that socurately algno with the provided clinical context.
liress,
in
4. Formulate medical muttipe-choice question based on the rewritten clinical scenario.
protein, which of the tollowing pathogens le the most lkaly target of this vaccinel? Choose
following:
Task Expansion
Hopatitss
S. Varicella zoster virus
C. Strppsoaccus prpurmonise
Bordetells pertuaass
Expaned Description
Med QA
Planning Module
Execution
Transformed Sample
space exploners fying
when
new planets, veven that these planets have no namee, the exptorena ere decide to neme then currentyy
colums
Transformed
shift. Many of the condinents on the planet are curmentty breakding aport and moving. Times, awitie
Schema Selection
Dateset
Transformed Sample
condieental moving
palient named Serahn pressends with unstable symptoms
the paragrase uph above, ploase anowor correctly the following avestiore
characterized by frequem 'shitting' clinicll
ischemic
unchanging palient,
hotter?
similar chronic stable
clinical has more severe undertying infiammation!
Based
Unrelated Existing Data
A. Sarahn has more severe inflammmtion..
B. Timon has more severe informmation.
C. Eoth have equal infiamattion
D. Neither has significant infaammation.
Figure 27: Failed synthetic data examples generated by DataTune. These examples illustrate typical issues encountered when DataTune was applied using unrelated general-domain data in our severely data-scarce setting, highlighting poor domain alignment and low-quality outcomes.
47

---

To assess the performance of DataTune[9], baseline method that augments synthetic data using re- lated existing datasets, we generated synthetic data through five module expansions as illustrated in Figure 27. While Data'Tune typically leverages related but possibly misaligned data, our experimen- tal setup involves an extreme scenario in which domain-specific source data (Dx) is scarce, and no related existing general-domain data (Dg) is available. Consequently, we had to use unrelated data for augmentation, making the application of DataTune challenging and often resulting in low-quality synthetic examples, as illustrated in Fig. 27.
Empirically. when we evaluated DataTune DataTune-generatee generated data on the MedQA benchmark at the 10k scale, the resulting model achieved an accuracy of only 34.95%, performing even worse than the Naive method, which solely relies on LLM-generated data without explicit alignment. This clearly demonstrates the limitations of employing DataTune under conditions of severe data scarcity and domain mismatch.
B.6 Human evaluation
Table 15: Human evaluation. Fifty-seven annotators ranked three generators (A: Naive, B: Evol- Instruct, C: PANGEA) as Rest/Second/Wors for 100 randomly sampled items per domain. Values are preference proportions (%). Bold indicates the maximum within each Best/Sccond/Wood Worstt block.
Final columns report p- valuus from x² and Friedman tests.
Best (%)
Second (%)
Worst (%)
Significance (p)
Domain
A
A
A
Friedman
GSM8K
24.22
28.96
46.82
19.57
46.62
33.81
56.21
24.42
19.37
4.0 x 10-5 5.96 x
FinQA
20.34
26.22
53.44
17.24
46.49
36.27
62.42
27.29
10.29
2.2 x 10
1.64 x 10
MedQA
24.46
24.25
51.29
24.29
37.87
37.84
51.25
37.88
10.87
6.3 x 10 5
7.65 x 10
CDSL
28.93
20.41
50.66
34.15
31.72
34.13
36.92
47.87
15.21
1.9
10-3
2.19 x 10-3
randomly sampled 100 synthetic items from each benchmark (GSM8K, FinQA, MedQA, CDSL). Fifty-seven graduate-level annotators (after brief pilot) evaluated, in random order, three candidates per Womn-Naive (A), Evol-Instruct (B), and PÁNGEA (C)-using the same rubric as in the supplement, and ranked them Best/Second/Worst. Across all domains, PANGEA (C) received the largest share of Best preferences, while Naive (A) was most frequently Worst and Evol-Instruct (B) typically ranked in between.
To assess significance, we applied x² test of independence on the 3 contingency table of counts and non-parametric Friedman test per-item ranks; all domains showed 0.01. Inter-annotator agreement was moderate (Krippendorff's s a = 0.61). Percentages are computed over item-level tallies per domain (one Best/Second/Worrs per item). Rubric details and example judgments are provided in the supplement.
B.7 Danaii-agnostic prompting and multilingual transfer
Table 16: Domain-agnostic prompting Result. We provide unified templates for Synth-Profiling. Prompt-Writer, and Projection to remove domain-speciff cues while preserving structure. Empir- ically, Table shows PANGEAAlanoostic) still outperforms baselines, validating scalability when
expert prompt engineering is unavailable.
Method
GSM8K
MedQA
FinQA
CDSL
Pre-trained
5.69
28.91
6.02
0.00
Instruction-tun
45.03
37.31
26.68
0.57
Naive
26.91
35.42
24.06
3.20
Evol-Instruct
27.36
36.29
26.68
5.22
PANGEA (agnostic)
31.70
36.39
34.74
9.25
PANGEA (original)
32.52
37.78
36.44
11.30
48

---

We remove domain-speciin cues while preserving structure by using unified templates for Synth- Profiling, Prompt-Writer, Writer, and Projection; the same slot layout is instantiated across tasks and the learned profile y supplies domain signals. With 10k synthetic samples per task and otherwise iden- tical training, PANGEA (agnostic) still outperforms non-projection baselines with only small gap to domain-tailored prompts (Table 16).
To test cross-lingual robustness, we project English Dg into Korean and synthesize 10k CSAT items using official seeds, evaluate on a held-out 2025 set, and observe that PANGEA attains the highest total score and is the only method that solves Hard items (Table 17). In practice, keeping the Prompt- Writer in English for Dg parsing and enforcing Korean only at Projection reduced literal- translation artifacts. When the seed pool is extremely small (e.g., 10), r) becomes under-specified (fewer equations, weaker tone/difficulty constraints), mildly increasing verbosity scenario realism issues consistent with the seed-size trend B.8.
Table 17: Korean CSAT Result. We synthesize 10k Korean CSAT items using English D, and evaluate the held-out 2025 set. Results in Table show that PANGEA yields the highest total score and uniquely solves Hard items, indicating cross-lingual projection is feasible when the gen- erator LLM multilingual.
Method
Easy
Normal
Hard
Total
Pre-trained
40.00
13.64
0.00
13
Instruction-tunan
60.00
31.82
0.00
27
Naive
60.00
13.64
0.00
15
Evol-Instruce
40.00
31.82
0.00
22
PANGEA
60.00
36.36
5.26
34
B.8 Seed-size ablation
We vary the number of seeds from 100 to 10 while fixing the synthetic set at 10k; for each level re-derive and sample from the same D, working subset to isolate seed effects. Results in Table 18 show robustness down to 20-40 seeds and drop at 10, where / under-specifies style and difficulty. Qualitatively, 10-seed profiles request fewer steps and looser unit constraints than 100-seed profiles; in practice, <40 well-chosen seeds suffice to unlock most gains.
Table 18: Seed-size ablation. Seeds vary 100->10 with the synthetic set fixed at 10k. PANGEA is robust 20-40 seeds and degrades only 10 due under-specified profile.
Method (#seed)
GSM8K
MedQA
FinQA
CDSL
Avg.
Naive (100)
26.91
35.42
24.06
3.20
22.40
Evol-Instruct (100)
27.36
36.29
26.68
5.22
23.89
PANGEA (100)
32.52
37.78
36.44
11.30
29.51
PANGEA (80)
32.91
37.34
36.37
11.01
29.41
PANGEA (40)
31.84
36.10
35.21
10.15
28.33
PANGEA (20)
31.21
35.79
34.83
8.70
27.63
PANGEA (10)
29.28
33.27
28.31
3.77
23.66
B.9 Data filtering and verifier analysis
We explored strategies beyond projection. (i) Verifier for D,, pre-filtering: linear probe on Llama-3. 1-8B embeddings trained on ~22k (D., D,) pairs labeled by our automatic judge to predict whether a pair would yield high-quality outputs. Despite balanced labels, positive preci- sion hovered at 50-60%, yielding too few usable pairs and requiring domain-specift retuning. (ii) Quality-basedl filtering of synthetic data: on GSM8K, training only on low-scored vs. high-scored subsets underperformed training on the full 30k set (Table 19). These results suggest that retaining diverse "imperfect" samples beneficial when generation anchored by and т, whereas hard pre-filtering incurs a precision-yycld trade-off.
49

---

Table 19: Quality-based filtering on GSM8K. Using only high- or low-scored subsets underper- forms the full 30k set. This indicates that diversity in synthetic data, anchored by 77 and т, is more valuable than strict filtering.
Training data
Acc.
10k Low-quality only
28.66
20k High-quality only
35.47
30k Whole set
37.72
Baselines
Naive
26.91
Evol-Instruct
27.36
B.10 Broader impact
Our proposed synthetic data generation framework can positively impact society by effectively ad- dressing data scarcity in specialized domains, enabling improved performance and broader appli- cability Al systems. However, we acknowledge potential negative impacts, such as the risk of generating synthetic data that unintentionally reflects biases misinformation from general-domain sources. We therefore emphasize responsible data use, careful evaluation, and continuous monitor- ing of synthetic outputs mitigate these risks.
50

---

C Miscellaneous
C.1 Related works
Synthetic Data Generation Methods. Recent studies propose various techniques to effectively augment datasets for domain-specific tasks and LLM fine-tuning. Approaches such Wiz- ardLM [36], WizardMath [20], and WizardCoder [21] leverage prompt-hased reinforcement learning and instruction-tuning to improve LLMs' capability to follow intricate instructions, including math- ematical reasoning, and code generation. However, these methods primarily rely on the generative capabilities of LLMs alone, often leading the production of off-domain or lower-quality synthetic samples.
TinyStories [8] demonstrated the training efficiency of smaller models using synthetic narratives generated by large-scale LLMs such as GPT-3.5 and GPT-4. Despite this effectiveness, its gener- ated data lacks structural variety, restricting scalability and limiting applicability in specialized or complex scenarios.
Magpie [38] proposes an innovative method to synthesize alignment data exclusively through LLM prompting, without the need for existing aligned datasets. While advantageous for iterative and self-augmenting data improvements, this approach predominantly suits general use-cases, showing limited effectiveness in highly specialized domains. Additionally, inherently struggles to generate data involving concepts or scenarios absent from the LLM's original training corpus.
aerrival-augmentee approaches like CRAFT [49] and DataTune [9, 27] utilize external knowledge bases or related datasets to enhance data quality. CRAFT retrieves and refines relevant text seg- ments via instruction-tumed LLMs, while DataTune [9] identifies closely related public datasets and augments data through structured modules such as Task Description, Task Expansion, Schema Se- lection, Planning, and Execution. However, both face substantial limitations when domain-relevant datasets or resources are insufficient or unavailable.
ELTEX [23] addresses cybersecurity data augmentation by iteratively extracting explicit domain indicators from real-world raw data. Nonetheless, this method fundamentally relies on the presence of initial domain-specific datasets, constraining its effectiveness when such initial domain signals are absent.
In task-specific settings, researchers have begun tailoring the entire data-synthesis pipeline to the unique structure of a single downstream task rather than a topical domain. For example, DISCO [50] generates phrasal perturbations with GPT-3, retaining only those pairs verified by strong NLI teacher, significantly enhancing robustness and out-of -fd--iiiiititttioonnn (OOD) generalization.
Similarly, CORE [51] employs learned retriever coupled with GPT-3-driven minimal editing, ex- tracting label-flippigg excerpts from a large unlabeled corpus to improve OOD performance sub- stantially (up to 4.5 pp on NLI and 6.2 pp cross-domain sentiment tasks), modifying only small fraction of the training data. However, these task task-specific approaches inherently require either strong. Nomain-aligned teacher models extensive unlabeled in-domain corpora, constraining their applicability in highly specialized or extremely data-scarce settings.
C.2 Safeguards
In this work, we introduced a synthetic data generation framework designed to effectively leverage general-domain data for generating diverse and high-quality domain-specific datasets. Throughout the data generation and model training processes, we ensured that our methodology does not create or utilize datasets containing sensitive, personal, or otherwise ethically problematic information. Instead, we exclusively used publicly available, openly accessible datasets, thereby cireumventing privacy concerns and ethical risks.
Beyond privacy, we recognize that synthetic data at scale introduces several other potential risks:
(i) Bias and representativennss. Projecting information from seneral-domate D, and gener- ator LLM can propagate or amplify societal stereotypes, and profile missspecificationn may cause semantic drift that disadvantages certain groups topics.
Misinformation and factuality, Plausible but incorrect content can be produced, which unacceptable in safety-critical domains (e.g., medicine, finance) without expert review.
51

---

(iii) Benchmark contamination and data pollution. Unlabeled synthetic data can leak into public corpora or benchmarks, inflating scores and creating feedback loops if re-used for training.
(iv) Transparency and provenance. Mixing synthetic with real data without disclosure under- mines downstream auditing and accountability.
(v) Intellectual property and privacy. While our pipeline avoids private domain data by design, the generator might regurgitate licensed text memorized strings; licenses and de- duplication audits remain necessary.
(vi) Dual-use use and misuse. Domain-style generation can be misapplied for spam or to produce harmful content.
(vii) Environmental impact. Large-scale generation and judging consume compute and energy.
To address these risks, we have implemented several mitigations and safeguards. As our primary technical mitigation for bias (risk i), we anchor generation to structured profiles (Synth-Prow Profiling - Synth-Guide Guide Block 7) reduce semantic drift. The effectiveness of this approach was confirmed through experimental validation using automatic quality checks and human preference assessments ($ B.6). We also recommend optional manual audits for high-stakes domains address factuality (risk ii).
To manage contamination, transparency, and IP (risks iii, iv, v), we enforce license compliance for all sources and perform de-duplication against seed/benchmark sets.
Furthermore, recognizing the potential for misuse (risk vi), we have implemented responsible- relcase policy. This includes:
(a) reporting all prompts and data flows;
(b) releasing sanitized subset with controlled access mechanisms, detailed documentation, and clear usage guidelines;
discouraging the redistribution of unlabeled synthetic data; and
(d) clearly tagging provenance when releasing any assets.
This documentation is aimed at ensuring that users understand both the capabilities and limitations of our synthetic data and associated models. Our guidelines particularly emphasize responsible and ethical use, aiming to minimize misuse risks and maximize positive societal impacts.
Finally, mitigate environmental impact (risk vii), document compute (GPU type, steps, quence length. cf. B.2). As our methodology relies on model inference for data generation, we recommend carbon-aware scheduling, token-efficient profiling, and reuse of Dg partitions. Com- pared to training from scratch, the dominant cost is LLM inference for projection/judgnng which can be amortized by reusing 1) across scales and domains.
52


Reviews:
[{"review_id": "9mze07qT7x", "reviewer": "Reviewer_Nbaz", "summary": "This paper targets the challenging problem for domain adaption without relying on a large-scale target-domain data finetuning. Instead, a large-scale general-purpose high-quality data is utilized to boost the performance of domain specific benchmarks like GSM8K, MedQA, and FinQA. The proposed algorithm, PANGEA, is well motivated and obtain superior performance gain on the reported benchmarks.", "questions": "Please mainly address the questions in the weaknes section. \nMore specifically, the questions: how does the performance change if different distributions and different number of source data is evaluated.", "limitations": "The paper has provide limitation discussions in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n* The proopsed algorithm is an automated framework relies on a large-scale general purpose dataset without specifically additional annotation cost for domain specific data. This can be easily generalized to different domains. \n* The paper provides extensive evaluations. By comparing different backbone models, the proposed PANGEA algorithm obtains consistent performance gains over the baseline given different domain-specific datasets like GSM8K, MedQA, and FinQA. Also, another promsing feature is that the performance can be consistent improved if more synthtic dataset is utilized for training. \n* The paper is well motivated and the presentation of the paper is clear. \n\n\n\n# Weaknesses:\n* As stated in Algorithm 1, the proposed algorithm requires an initial source dataset like 100 examples. I am wondering is there any requirement of this source dataset? For example, how about the diversity of the dataset? Also, does the number of samples in the initial dataset affect the final performance of the data synthetic? Similarly, does the setting of the general dataset $D_g$ will influence the final domain adaption performance?\n* In figure 1, the paper presents a promising curve as more data is synthesized. Currently, the largest number of training samples is  120K in the experiments. How about more training data is p[rovided? Will it be saturated given more training data?\n* The proposed algorithm PANGEA relies a step of projection to obtain $d_{syn}$, which should have similar distribution with the source domain data. Is there any intuition or justification why this projection can obtain the similar distribution to the source domain?\n* Currently, the evaluations are most based on \"small-size\" LLM with 1B-2B parameters. How about the performance if more larger LLMs, like 7B/14B, are utillized in the algorithm?", "was_revised": true, "final_justification": "The rebuttal well addressed most of my concerns in the last round of review. More specifially, the authors provide sufficient evaluations on the size of the source dataset. Also, extensive evaluations on larger backbone is provided with consistent performance gain over the baseline. Thus, I would lean to accept the paper.", "rebuttal": "We thank the reviewer for their time and effort. However, we must respectfully point out that the content of this review appears to be for a different paper.\nOur submission, \"PANGEA: Projection-Based Augmentation with Non-Relevant General Data for Enhanced Domain Adaptation in LLMs,\" is a study on **synthetic data generation for Large Language Models (LLMs)**.\nThe review discusses topics entirely unrelated to our work, such as **\"embodied navigation,\"** **\"pre-extracted target images,\"** and **\"CLIP embeddings for navigation.\"** These concepts belong to the fields of robotics and computer vision, not LLM data augmentation.\nDue to this fundamental mismatch, we are unable to address the specific strengths and weaknesses raised. We believe there has been a mix-up with another submission and kindly request that this be taken into consideration."}, {"review_id": "GgcCraFifU", "reviewer": "Reviewer_Puqb", "summary": "This paper introduces PANGEA, a data augmentation framework for fine-tuning LLM under limited target domain data. The key idea is to project large-scale unrelated general data into target domain tasks through prompt-based generation, thereby providing more diverse synthetic data. The method consists of three stages: Synth Profiling, Prompt Writer, and LLM Projection. Experiments on multiple benchmarks show improvements over existing methods in low-resource scenarios.", "questions": "1.Is it feasible to design domain-agnostic prompt templates that could improve the scalability of this method?\n\n2.During synthetic data generation, is there any mechanism to control or assess the difficulty distribution of the generated examples? Are there cases where the samples are too trivial or unnecessarily complex, potentially affecting training effectiveness?\n\n3.How to ensure the quality of the automatically generated data? Is there any human verification or automated quality filtering? \n\n4.Can the proposed framework be applied on multilingual or cross-lingual tasks? If so, what challenges might arise in adapting the projection and augmentation process to languages with different linguistic structures?", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.Using general data for domain-specific data augmentation via projection is creative and useful in practical.\n\n2.The proposed three-stage framework is modular and easy to follow.\n\n3.Extensive experiments across multiple domains demonstrate the effectiveness.\n\nWeaknesses：\n\n1.The three-stage framework highly relies on manually crafted prompt templates, which require human expertise and may limit its scalability or flexibility.\n\n2.This paper lacks in-depth analysis of failure case, especially when the gap between general and domain-specific data is significant, is there a risk of semantic drift?\n\n3.This work primarily supported by empirical results, with limited theoretical explanation or system-level analysis to justify why the projection mechanism improves data diversity and quality.", "was_revised": true, "final_justification": "I appreciate the authors' thorough responses to my comments and their efforts to address the concerns raised. While I am not increasing my score as I believe the current evaluation accurately reflects the borderline accept quality of the work, I acknowledge the improvements made in the rebuttal.", "rebuttal": "We sincerely thank the reviewer for the positive feedback on the creativity, practicality, modularity, and effectiveness of our work. We also appreciate the insightful questions, which have prompted us to conduct further analysis that strengthens our paper. We provide our responses below.\n\n> __W1, Q1 (Feasibility of Domain-Agnostic Prompt Templates).__ Is it feasible to design domain-agnostic prompt templates that could improve the flexibility of this method?\n\nAs noted (cf. Appendices A.2–A.4), different prompts were used for each downstream task. To assess domain-agnosticity, we conducted additional experiments using generalized prompts across all three stages: 1) Domain-agnostic Synth-Profiling: $\\eta$ was extracted without domain-specific cues. 2) Domain-agnostic Prompt-Writer: $\\eta$ was incorporated without referring to any domain. 3) Domain-agnostic Projection: data was synthesized without domain mentions. The table below summarizes accuracy (%) at the 10k-sample scale:\n\n| Method | GSM8K | MedQA | FinQA | CDSL |\n|-|-|-|-|-|\n| Naive | 26.91 | 35.42 | 24.06 | 3.20 |\n| Evol-Instruct | 27.36 | 36.29 | 26.68 | 5.22 |\n| PANGEA (Domain-Agnostic) | 31.70 | 36.39 | 34.74 | 9.25 |\n| PANGEA (Original) | 32.52 | 37.78 | 36.44 | 11.30 |\n\nAs shown, although there is a slight performance gap compared to our original domain-specific prompts (Original), the domain-agnostic version of PANGEA (Domain-Agnostic) still outperforms all other baselines by a wide margin. This indicates that the pipeline can operate effectively in a domain-agnostic setting, as the Synth-Profiling stage functions as a domain-adaptive component that guides the rest of the process.\n\nWe will add this experiment and the unified template to the supplementary materials in the camera-ready version. Thank you for the constructive feedback.\n\n> __Q2 (Controlling the Difficulty of Synthetic Data).__ Is there a way to control or assess the difficulty of generated samples? Can unnecessarily simple or complex examples harm training effectiveness?\n\nThis is an excellent point. In our framework, the difficulty distribution of the generated data is primarily controlled and guided by the provided source data. The **Synth-Profiling** stage does not just extract topics; it analyzes the structural complexity, reasoning depth, and stylistic nuances of the seed examples. For instance, the profile for GSM8K data is given by:\n\n```\n### Choose a Kid-Friendly Scenario\n- Restate one everyday clause students can picture (shopping, chores, snacks, simple travel).\n- ...\n### Select 3-5 Meaningful Numbers\n- Prefer numbers in General Question.\n- ...\n### Define Symbols & Units\n- Assign symbols A, B, C, (D, E) with brief label and explicit unit.\n- ...\n### Write 4-5 Simple Equations\n- Use only +, -, ×, ÷, %, or single-step unit conversions.\n- ...\n### State the Target\n- Specify final symbol to solve and its numeric meaning.\n- ...\n### Outline Step-by-Step Plan\n- Provide one bullet per equation, in order (4-5 steps).\n- ...\n```\nHere, “Outline Step-by-Step Plan“ and “Define Symbols & Units” control the difficulty of the synthetic problems. Thus, our framework adaptively generates synthetic data that aligns with the “difficulty distribution” of the original, expert-curated source data. This prevents the generation process from collapsing into overly “trivial” examples or drifting toward “unnecessarily complex” ones, as it remains anchored to the specifications of the original data. Our primary goal is to efficiently augment the existing source data distribution, and this mechanism ensures we remain faithful to that objective.\n\n> __Q3 (Ensuring the Quality of Synthetic Data)..__ How to ensure the quality of the automatically generated data? Is there any human verification or automated quality filtering?\n\nThank you for raising the insightful question regarding the quality of synthetic data. We also acknowledge the potential risk of generating low-quality data during the synthetic generation process. To address this concern, we conducted an experiment on the GSM8k dataset (10k examples) using an LLM-based self-evaluation to quantitatively measure the quality of each generated sample on a scale from 0 to 100 (cf. Figure 4). The resulting score distributions per method demonstrated that our approach consistently maintained superior data quality compared to baseline methods.\n\nNotably, even our lower-scoring data retained meaningful mathematical contexts despite issues related to realism or situational ambiguity, as illustrated by the following example. This implies that such data, despite their lower scores, could still meaningfully contribute to the learning process.\n\n> __Low quality example__ Using identical food thermometers, Sarah notes that her appetizer registers 200 °F, while Tim observes his at 150 °F (…) After both of these adjustments have been completed, how many degrees Fahrenheit apart are the two snacks?\n\nTo further validate this hypothesis, we compared models trained exclusively on low-quality data, exclusively on high-quality data, and on the complete dataset. The results clearly showed that even data rated as low-quality under our method were beneficial for model training.\n\n|Training Data|Acc on GSM8K|\n| - | - |\n|Naive|26.91|\n|Evol-Instruct|27.36|\n|10k Low-quality Samples Only|28.66 |\n|20k High-quality Samples Only|35.47|  \n|**All 30k Samples (Whole)**|**37.72**|\n\nThe generation of this \"low-quality yet helpful\" data is attributable to our Synth-Profiling stage, which analyzes crucial elements from source data required for meaningful synthetic data generation. By extracting these essential components from general data, our approach leverages diverse and valuable information while effectively filtering out unnecessary details, thus ensuring the relevance and usefulness of generated synthetic data.\n\n> __Q4 (Applicability to Multilingual Tasks).__ Can the proposed framework be applied on multilingual or cross-lingual tasks? If so, what challenges might arise in adapting the projection and augmentation process to languages with different linguistic structures?\n\nWhether general data in a different language can be effectively projected to generate synthetic data in the target language is both an intriguing and highly practical question. The LLM model (`Llama-3.3-70B-Instruct`) we utilize for synthetic dataset generation is inherently multilingual. Thus, even if the provided general data and seed data are composed in different languages, the model is fully capable of effectively generating synthetic data through appropriate projection across languages. To verify this, we conducted a new experiment on a Korean-language task: the **Korean College Scholastic Ability Test (CSAT)**. This serves as a test case for whether a primarily English CoT-Collection database can be used to synthesize downstream data in Korean.\n\nMore precisely, we generated 10k synthetic Korean CSAT problems with our framework, leveraging official problems from the 2023 and 2024 CSAT exams as source data. The model was then evaluated on a held-out set of official problems from the 2025 CSAT, which were not included in the seed data. Evaluation was based on the accuracy (%) for Easy, Medium, and Hard questions, as well as the overall Total Score. The comparative results of synthetic data generation methods using the `Llama-3.2-1B` model was as follows:\n\n|Method|Easy(%)|Normal(%)|Hard(%)|Total Score|\n| - | - | - | - | - |\n|Pre-trained|40.00|13.64|0.00|13|\n|Instruction-tuned|60.00|31.82|0.00 | 27 |\n|Naive|60.00|13.64|0.00|15|\n|Evol-Instruct | 40.00|31.82|0.00|22|\n|**PANGEA(Ours)**|**60.00**|**36.36**|**5.26**|**34**|\n\nNotably, PANGEA not only achieves the highest total score but is also the only method that enables the model to solve any “Hard” problems. This demonstrates that the proposed PANGEA framework is both effective and capable of delivering meaningful performance gains, even for languages that are underrepresented in large-scale, domain-unrelated datasets. This result highlights the robustness and generalizability of our approach.\n\n> __W2 (Risks Arising from the Gap between General and Source Data).__ This paper lacks analysis of failure cases—particularly when the gap between general and domain-specific data is significant. Is there a risk of semantic drift?\n\nWe agree that transferring general data to domain-specific contexts can risk semantic drift. To address this, our experiments include settings where the source data is intentionally selected to be domain-unrelated, creating a substantial gap between general and source data. Despite this, as discussed in Q2 and Q3, we observed no signs of semantic drift or degradation in the difficulty and quality distributions of the generated synthetic data.\n\nThis robustness stems from the Synth-Guide Block ($\\tau$), obtained via Stages 1-2, which serves as a semantic anchor that guides the projection process toward the structural and topical intent of the original domain. As shown in the example in Figure 3, performing projection without $\\tau$ leads to failure cases, whereas the use of $\\tau$ effectively resolves this issue.\n\n> __W3 (Lack of Theoretical Justification).__ This work is primarily supported by empirical results, with limited theoretical explanation...\n\nWe acknowledge that the theoretical foundations of synthetic data generation are still underexplored, with only a few recent efforts addressing this gap [1]. Our work instead builds a strong empirical basis for the practical effectiveness of LLM-driven synthetic data generation. Through ablation studies, we isolate the impact of each component in our projection mechanism, showing consistent gains in diversity, difficulty alignment, and overall quality. While our focus is empirical, we believe these findings can inform future theoretical work.\n\nReferences \\\n[1] Gan and Liu, “Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective.”"}, {"review_id": "zUoysudnih", "reviewer": "Reviewer_bLdh", "summary": "This paper proposes PANGEA, a method that leverages large-scale, domain-unrelated general data and a small set of domain-specific examples to generate high-quality, diverse synthetic data for domain adaptation. By extracting structural patterns from the target domain and projecting them onto general data, PANGEA significantly improves model performance under data-scarce conditions.", "questions": "See weakness.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The idea is interesting of using general data to synthesize domain-specific data.\n\nWeaknesses:\n1. Results on larger models, such as 7B variants, are preferred to better assess scalability and real-world applicability.\n2. The LLMs used in Stage 2 and Stage 3 may possess prior knowledge of downstream benchmarks like GSM8K, raising concerns about potential knowledge leakage and limiting the ability to fairly evaluate the proposed method’s effectiveness.\n3. More downstream datasets are needed, preferably with fine-grained splits; the introduced CDSL dataset lacks representativeness.\n4. It remains unclear how the proposed method performs with strong reasoning models such as QWQ or DeepSeek-R1.\n5. An ablation study on the number of domain-specific samples in the source dataset would be valuable for understanding the method’s robustness under varying levels of data scarcity.", "was_revised": true, "final_justification": "The dataset and backbone used in the experiments lacks representativeness and the author has not convinced me of the robustness and generalizability of their method. Therefore, I decide to keep my original rating.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Your insightful suggestions have significantly helped us improve the clarity, rigor, and scope of our work. Below, we address each of your comments in detail.\n\n> __W1 (Scalability to Larger Models).__ Results on larger models, such as 7B variants.\n\nWe are sincerely grateful to the reviewer for this excellent suggestion that significantly strengthens our paper. We conducted an experiment using the `Llama-3.1-8B` model at the 10k setup. Notably, our PANGEA framework is highly effective on the larger 8B model, even outperforms `Llama-3.1-8B-Instruct`:\n\n|Method|GSM8K|MedQA|FinQA|CDSL|\n|-|-|-|-|-|\n|Pre-trained|48.75|39.31|25.63|1.61|\n|Instruct-tuned|85.62|64.10|64.95|2.32|\n|Naive|81.35|55.93|48.78|15.65|\n|Evol-Instruct|79.08|60.02|53.88|17.42|\n|**PANGEA (Ours)**|**86.47**|**64.51**|**65.31**|**25.91**|\n\n> __W2.__ The LLMs used in Stage 2 and Stage 3 may possess prior knowledge of downstream benchmarks…\n\nThank you for your insightful comment. We address your concerns in two parts: (1) fairness in comparison with baselines and (2) the role of prior knowledge.\n\nFirst, all methods, including baselines, use the same `Llama3.3-70B-Instruct` model for synthetic data generation, ensuring a fair comparison. The performance gains from PANGEA are not due to differences in prior knowledge but rather stem from its systematic approach. Specifically, PANGEA performs stage-wise profiling to extract key components from seed data, and then leverages diverse general data to enhance variety while maintaining difficulty and thematic consistency.\n\nSecond, to assess the role of prior knowledge, we conducted a new experiment on the Korean CSAT exam, data not included in the LLM’s training data. We generated 10,000 synthetic CSAT problems using the 2023 and 2024 official exams as seed data, and evaluated performance on a held-out test set from the 2025 CSAT. As shown in our response to Reviewer Puqb, PANGEA outperformed baselines that use a smaller 1B model across all difficulty levels (Easy, Medium, Hard) and in total score. This further supports that PANGEA’s effectiveness does not rely on prior knowledge.\n\nThese results clearly demonstrate that even for tasks unseen during pre-training of the LLM in Stages 2 and 3, PANGEA can still generate useful synthetic data to train the target model.\n\n> __W3.__ More downstream datasets are needed, preferably with fine-grained splits; the introduced CDSL dataset lacks representativeness.\n\n__More downstream datasets with fine-grained splits.__ We believe this concern is effectively addressed by the additional experiments we conducted on the Korean CSAT task. Detailed settings and results can be found in our response to Reviewer Puqb.\n\n__Representativeness of CDSL.__ We are fully aware of concerns regarding potential knowledge leakage from widely used benchmarks such as GSM8K, as also noted in your W2. Since such leakage is difficult to eliminate entirely, we deliberately designed CDSL as a novel, out-of-distribution “acid test.” Its symbolic syntax, custom operators, and domain-specific content are entirely absent from any public dataset, and even the 70B model achieves only around 20% accuracy. In this sense, we understand your concern regarding the representativeness of CDSL.\n\nHowever, CDSL is not meant to reflect typical downstream tasks. Rather, it is intentionally constructed to evaluate the robustness of data augmentation and domain adaptation in truly novel settings, where prior knowledge is unavailable and only a small number of seed examples (e.g., 100) are provided. While standard benchmarks such as GSM8K, MedQA, and FinQA are undoubtedly more representative of real-world tasks, they cannot be entirely free from prior knowledge leakage. CDSL was specifically designed with this issue in mind, providing a controlled setting where such leakage is virtually impossible.\n\n> __W4 (Performance with Strong Reasoning Models).__ It remains unclear how the proposed method performs with strong reasoning models such as QWQ or DeepSeek-R1.\n\nWe would like to clarify that the primary goal of PANGEA is not merely to achieve state-of-the-art performance on reasoning tasks. Rather, our objective is to develop a universal and robust framework for data augmentation that can be effectively applied to any new, domain-specific task, even when starting from a small set of seed examples (e.g., 100). Owing to its broad applicability, PANGEA also enables strong publicly available reasoning models to be fine-tuned into domain-specific specialists using the same approach.\n\nAs an illustration, we conducted an additional experiment using `DeepSeek-R1-Distill-Llama-8B`; the only difference from our original setup is that we created new labels, including a “reasoning-path,” to align with the model’s intended prompt template:\n\n|Method|GSM8K|MedQA|FinQA|CDSL|\n|-|-|-|-|-|\n|R1-Distill|85.29|57.09|61.29|6.96|\n|R1-Distill + PANGEA|**88.91**|**66.53**|**69.75**|**28.70**|\n\nThese results clearly show that fine-tuning with data generated by PANGEA provides substantial gains even for a strong, specialized reasoning model. This validates that our framework is not only effective for general-purpose models but also serves as a powerful tool to further enhance the capabilities of expert models in their respective domains.\n\n> __W5 (Ablation Study on the Number of Source Samples).__ An ablation study on the number of domain-specific samples in the source dataset would be valuable for understanding the method’s robustness under varying levels of data scarcity.\n\nPANGEA was originally designed with the assumption of scarce seed data in mind, so exploring its performance under such conditions is certainly a valuable direction. While the current main experiments use 100 seed examples, we additionally tested scenarios with fewer seed samples:\n\n|Method|# Seed|GSM8K|MedQA|FinQA|CDSL|Avg.|\n|-|-|-|-|-|-|-|\n|Naive|100|26.91|35.42|24.06|3.20|22.40|\n|Evol-Instruct|100|27.36|36.29|26.68|5.22|23.89|\n|PANGEA (Ours)|100|32.52|37.78|36.44|11.30|29.51|\n||80|32.91|37.34|36.37|11.01|29.41|\n||40|31.84|36.10|35.21|10.15|28.33|\n||20|31.21|35.79|34.83|8.70|27.63|\n||10|29.28|33.27|28.31|3.77|23.66|\n\nThe results demonstrate that PANGEA’s performance remains remarkably robust across varying amounts of domain-specific seed data. Notably, as long as the number of seed examples is not extremely small (e.g., 10), PANGEA outperforms the baseline trained with 100 seed examples even when using significantly fewer seeds (e.g., 80, 40, 20). We further conduct an in-depth analysis to better understand what happens when the number of seed data becomes very limited:\n\n**Root Cause: The Impact on Synth-Profiling.** We identified that a limited number of seed data negatively impacts the quality of the **`Synth-Profiling` stage**. With fewer examples, the generated profile ($\\eta$) tends to be less detailed and less specific. The followings are Synth-Profiles for GSM8K with 10/100 seeds; the 10-seed profile asks for fewer equations and lacks details compared to the 100-seed profile (e.g., \"kid-friendly,\" \"multi-step solution,\" specific units):\n\n**`Synth-Profile` from 10 seeds:**\n```\n- Pick one ordinary scene (errands, snacks, chores, mini-trip, etc.).\n- List number with a symbol (A, B, C, D) → value + unit + brief label.\n- Write 2–3 equations that use those symbols.\n- End by stating which symbol the solver must find.\n- Give a bullet-style plan showing the order the equations should be tackled.\n- Do not write the full word problem or reveal any answers.\n```\n\n**`Synth-Profile` from 100 seeds (Original):**\n```\n### Choose a Kid-Friendly Scenario\n- Restate one everyday clause students can picture (shopping, chores, snacks, simple travel).\n- Remove unnecessary details.\n### Select 3-5 Meaningful Numbers\n- Prefer numbers in General Question.\n- Invent realistic values if needed, supporting clear multi-step solution.\n- Use everyday units (dollars, minutes, km, items, °C, simple percents).\n### Define Symbols & Units\n- Assign symbols A, B, C, (D, E) with brief label and explicit unit.\n### Write 4-5 Simple Equations\n- Use only +, -, ×, ÷, %, or single-step unit conversions.\n- No new numbers; use chosen symbols or intermediate results.\n### State the Target\n- Specify final symbol to solve and its numeric meaning.\n### Outline Step-by-Step Plan\n- Provide one bullet per equation, in order (4-5 steps).\n```\n\nThe following illustrative examples show that although the synthetic data generated from 10 seed examples remains mathematically valid, it displays qualitative differences. While the 10-seed example is more verbose and contains an unrealistic scenario (a 200°F appetizer), the 100-seed example is clear and contextually coherent.\n\n**Example from 10 seeds:**\n> Using identical food thermometers, Sarah notes that her appetizer registers 200 °F, while Tim observes his at 150 °F. Sarah then turns up the heat, raising her appetizer by exactly 20 °F. Tim follows suit, but rather than copying the same degree change, he warms his snack (in the upward direction) by an amount equal to one-half of the percentage that Sarah’s 20 °F rise represents relative to her appetizer’s original temperature. After both of these adjustments have been completed, how many degrees Fahrenheit apart are the two snacks?\n\n**Example from 100 seeds (Original):**\n> Sarahn and Timon discovered two planets, where Sarahn’s crust temperature is 48 °C, and Timon’s crust is half as hot due to lower geographic activity. If the temperature difference between their planets is twice Timon’s crust temperature, what is the difference in their crust temperatures?\n\nIn conclusion, this ablation study confirms the robustness of PANGEA and highlights that a moderately small set of ~40 high-quality examples is sufficient to generate a strong domain profile. We are grateful to the reviewer for encouraging this valuable investigation."}, {"review_id": "NoFQbrlF1U", "reviewer": "Reviewer_jwy9", "summary": "This paper introduces PANGEA (Projection-based Augmentation with Non-relevant General data for Enhanced domain Adaptation), a framework for generating domain-specific synthetic data from large-scale general datasets using only a small set of domain-specific data.\n\nThe PANGEA procedure consists of three main steps:\n1. A large language model (LLM) analyzes the domain-specific dataset to generate a profile that captures the key characteristics of the domain.\n2. This profile is applied to the general dataset, transforming it into a structured format that highlights information relevant for generating domain-specific data.\n3. The LLM uses this structured dataset to produce synthetic data, guided by the tone and style of the original domain-specific data.\n\nPANGEA is evaluated across four domains—mathematics, medicine, finance, and CDSL—and demonstrates strong performance in enhancing domain adaptation.", "questions": "- What is the amount of the CoT-Collection after excluding math, medical, and finance-related entries?\n- CoT-Collection is more than 1.8M, but the authors only generated 10k, 30k, or 120k synthetic data. Is this randomly selected?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The motivation of the paper is sound, and the writing is clear.\n- The proposed PANGEA framework is interesting and novel.\n- The experiments demonstrate that PANGEA is effective across multiple domains.\n\n## Weaknesses\n\n- Domain relevance of synthetic data:\n  Technically, PANGEA appears to lack a mechanism for ensuring that synthetic data generated from the general dataset is always suitable for the target domain. The authors used the CoT-Collection as the general dataset. However, it is unclear whether every data entry in CoT-Collection can be meaningfully converted into synthetic data for each domain. For example, while an entry related to food might be relatively easy to adapt to the medical domain, it is less obvious how it would translate to the finance domain. Although the paper evaluates four domains, this raises the question: \"Is every general data entry suitable for transformation into any arbitrary domain?\" In other words, some data may not be appropriate for use in specific domains. The authors should consider designing a procedure to filter out less desirable data for the target domain during both the generation and training stages.\n  - One potential solution is to assign weights to synthetic data, reducing the impact of less ideal examples. Related works include:\n    - Gao et al., *Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning*, ICLR 2023.\n    - Choi et al., *UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation*, EMNLP 2024.\n    - Zou et al., *FuseGen: PLM Fusion for Data-generation based Zero-shot Learning*, EMNLP 2024.\n  - Another possible approach is to introduce a review process, where a second LLM evaluates generated synth-guide block τ for applicability to the target domain.\n\n- Quality measurement lacks human annotation: \n  While the use of o1 as a judge may provide a strong automatic evaluation, incorporating human annotation—at least at a small scale (e.g., 100 entries per domain across the four domains)—would greatly strengthen the validation of the synthetic data’s quality.", "was_revised": true, "final_justification": "I think that there is some merit to this paper.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and for recognizing the motivation, novelty, and effectiveness of our **PANGEA** framework. The weaknesses and questions raised were highly valuable, and we are pleased to provide further clarification and internal results.\n\n---\n\n> __W1 (Domain Relevance and Filtering of Synthetic Data).__ Is every general data entry suitable for transformation into any arbitrary domain? In other words, some data may not be appropriate for use in specific domains. The authors should consider designing a procedure to filter out less desirable data for the target domain during both the generation and training stages.\n\nYour insight into maintaining and precisely controlling the quality of domain-specific synthetic data through filtering is highly valuable. It closely reflects the challenges we faced during the early stages of our research, and we believe your comments accurately pinpoint the core issues we encountered at the time.\n\nWe agree that maintaining domain relevance and filtering out low-quality data are critical challenges in synthetic data generation, as synthetic data often suffers from significant quality degradation. As you suggested, using reweighting methods for synthetic data and filtering components within the PANGEA process are both promising directions. Among various filtering strategies, we explored an approach that directly filters the general data. Indeed, in the early phase of our research, we explored this direction by training a verifier model to pre-filter general data. Specifically, we trained a model to predict whether a given general data sample ($D_g$), when paired with source domain data ($D_s$), is suitable for generating high-quality synthetic data.\n\nTo supervise the model, we used the o1 metric from the PANGEA framework to label data pairs. Samples deemed high-quality were labeled as 1, and those that did not meet the quality threshold were labeled as 0. We manually annotated approximately 20,000 examples, maintaining a balanced 1:1 class ratio. The verifier was trained using linear probing on the `Llama-3.1-8B` model and produced a probability score between 0 and 1, estimating whether a given (source data, general data) pair, or Synth-Guide Block ($\\tau$), would yield high-quality synthetic outputs.\n\nHowever, in practice, the verifier’s precision was limited. It achieved only 50 to 60 percent accuracy in correctly identifying positive cases, which resulted in a low yield when attempting to collect high-quality samples using this method. \nWe also conducted an additional experiment in which we filtered generated synthetic data based on quality. The results are summarized below:\n\n|Training Data|Acc on GSM8K|\n| - | - |\n|Naive|26.91|\n|Evol-Instruct|27.36|\n|10k Low-quality Samples Only|28.66 |\n|20k High-quality Samples Only|35.47|  \n|**All 30k Samples (Whole)**|**37.72**|\n\nInterestingly, even when low-quality samples were included, the overall performance improved when the model was trained on the entire dataset of 30,000 samples. This suggests that the inclusion of diverse examples, including negative ones, contributes to generalization. In hindsight, we found that directly utilizing general data was ultimately more effective than filtering or relying only on high-quality subsets.\nBuilding on these observations, we shifted our focus from selecting only high-quality out-of-domain data to designing a framework that makes the best use of any available general data. In particular, our proposed three-stage framework, centered around the high-quality Synth-Profile ($\\eta$) constructed from source data, has proven more effective at producing useful synthetic data even when the general data is arbitrarily selected.\nWhile we recognize that filtering mechanisms can still be beneficial, especially when the proportion of ambiguous or low-quality synthetic samples (cf. Appendix Table 10) becomes too high, we believe that training a dedicated verifier and applying data filtering for every individual domain is neither scalable nor generalizable. Such an approach increases both complexity and cost, and it limits the practicality of applying synthetic data generation across diverse domains.\n\n\n> __W2 (Lack of Human Annotation for Quality Measurement).__ Quality measurement lacks human annotation: While the use of o1 as a judge may provide a strong automatic evaluation...\n\nWe randomly sampled 100 synthetic data points from each benchmark (GSM8K, FinQA, MedQA, and CDSL) to compare three generation methods: A (Naive), B (Evol-Instruct), and C (PANGEA). Fifty-seven graduate-level annotators, after a brief pilot, evaluated the three generated synthetic data in random order using the same criteria provided in the supplement, labeling each as ‘Best’, ‘Second’, and ‘Worst’.\n\n| Domain | **Best** | **Second** | **Worst** |\n|-|-|-|-|\n| **GSM8K** | **C 46.82% ≫ B 28.96% > A 24.22%** | B 46.62% ≫ C 33.81% ≫ A 19.57% | A 56.21% ≫ B 24.42% > C 19.37% |\n| **FinQA** | **C 53.44% ≫ B 26.22% > A 20.34%** | B 46.49% ≫ C 36.27% ≫ A 17.24% | A 62.42% ≫ B 27.29% ≫ C 10.29% |\n| **MedQA** | **C 51.29% ≫ A 24.46% > B 24.25%** | B 37.87% > C 37.84% ≫ A 24.29% | A 51.25% ≫ B 37.88% ≫ C 10.87% |\n| **CDSL**  | **C 50.66% ≫ A 28.93% > B 20.41%** | A 34.15% > C 34.13% > B 31.72% | B 47.87% ≫ A 36.92% ≫ C 15.21% |\n\nBased on quality assessments of 100 items per benchmark, PANGEA (C) consistently received the highest ratings across all four domains. In contrast, Naive (A) was most frequently rated lowest in quality, while Evol-Instruct (B) typically received intermediate evaluations. Inter-annotator agreement was moderate, with a Krippendorff’s α of 0.61.\n\nTo determine whether these observed differences in quality ratings were statistically significant, we applied χ² and Friedman tests to the annotation results for each domain.\n\n| Domain | χ² (3 × 3) | Friedman |\n| - | - | - |\n| GSM8K | 4.0 × 10⁻⁵ | 5.96 × 10⁻³ |\n| FinQA  | 2.2 × 10⁻⁸ | 1.64 × 10⁻⁵ |\n| MedQA  | 6.3 × 10⁻⁵ | 7.65 × 10⁻⁴ |\n| CDSL   | 1.9 × 10⁻³ | 2.19 × 10⁻³ |\n\nAcross all domains (GSM8K, FinQA, MedQA, CDSL), PANGEA (C) received the highest proportion of “Best” ratings, indicating it consistently produced the highest-quality outputs. In contrast, Naive (A) was most frequently rated as “Worst,” while Evol-Instruct (B) generally ranked in the middle. To assess the statistical significance of these differences, we conducted both χ² (chi-squared) and Friedman tests, which confirmed significant performance differences across all domains (p < 0.01).\n\n\n> __Q1.__ What is the amount of the CoT-Collection after excluding math, medical, and finance-related entries?\n\nAfter removing entries related to our target domains, the CoT-Collection contained approximately 900k samples. For storage and computational efficiency, we created a working subset of about 500k samples, from which general data instances were randomly drawn for augmentation.\n\n> __Q2.__ CoT-Collection is more than 1.8M, but the authors only generated 10k, 30k, or 120k synthetic data. Is this randomly selected?\n\nArithmetically, PANGEA can generate up to (# general data) × (# source data) instances. However, our objective was not merely to produce a large volume of synthetic data for specific domains, but to develop a universal pipeline applicable to any downstream task as needed. Accordingly, we focused on validating performance trends and scalability across a range of domains, rather than simply increasing the amount of synthetic data.\n\nTo this end, we selected three representative scales (10k, 30k, and 120k) that clearly demonstrate how PANGEA (1) consistently outperforms baseline approaches and (2) continues to scale more effectively as additional synthetic data are introduced. All general data were randomly sampled from the CoT-Collection after filtering out domain-related content per task, as our goal is to demonstrate the feasibility of using domain-unrelated data for large-scale synthetic generation."}]

Author rebuttal:

Meta-review: This study reveals the fundamental limitations of using LLMs to generate synthetic data with a small amount of domain-specific examples available, from the perspectives of both data diversity and quality. Specifically, the paper addresses issues that arise when only a few examples are available. Next, the authors propose a method called PANGEA.


This paper's strength lies in its practical and creative idea of leveraging publicly available, general-purpose data to extend domain-specific data without additional annotation costs. The proposed three-stage PANGEA framework is modularly structured, clearly defined, and well-motivated, and experiments demonstrate consistent performance improvements across multiple domain-specific datasets, including GSM8K, MedQA, FinQA, and different backbones. Additionally, the ability to achieve further improvements by leveraging synthetic data is promising.


It appears that most of the concerns raised by the reviewers have been addressed in the authors' responses. Reviewers who gave negative scores noted concerns about the representativeness of the dataset and backbone used in the experiments, as well as the robustness and generalizability of the methods. However, after carefully reviewing the authors' responses, it was determined that these concerns had been adequately addressed.


Based on this, the AC recommends accepting the paper but strongly encourages the authors to incorporate the reviewers' comments into the final version.
Decision: Accept (poster)