Not All Documents Are What You Need for Extracting Instruction Tuning Data
Authors: Chi Zhang
Huaping Zhong
Hongtao Li
Chengliang Chai
Hongjiawei
Yu-Ping Wang
Yuhao Deng
Jiacheng Wang
Yizhou Yan
Qiu Jiantao
Conghui He
Lei Cao
Venue: ICLR 2026 Poster (2026)
Keywords: data extraction
data efficient
instruction tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Instruction tuning improves the LLMs performance but depends on high-quality training data. Recently, LLMs have been used to synthesize data, enhancing training with seeds like question-answer (QA) pairs. However, this synthesis often results in instruction examples similar to the seeds, lacking diversity and biasing real applications. Thus, we propose to extract instruction tuning data from web corpus with much rich knowledge. The most straightforward strategy is to quickly retrieve domain specific documents from the corpus and then extract all QA pairs of these documents for tuning LLMs, which has two main limitations. (1) Extracting all QA pairs using LLMs is prohibitively expensive; and (2) These extracted pairs are not all beneficial for the downstream applications, and incorporating all of them for tuning may even hurt the model performance. To overcome the limitations,  we introduce $\texttt{EQUAL}$, an $\textbf{E}$ffective and scalable data extraction framework that iteratively interleaves  document selection and extract high-$\textbf{QUAL}$ity QA pairs to optimize instruction tuning. $\texttt{EQUAL}$ first clusters the document set based on the embeddings generated by contrastive learning. Then it leverages the multi-armed bandit based strategy to quickly identify document clusters where can extract high-quality QA pairs for training. This iterative framework significantly reduces computational costs while improving model performance much. Experiments on AutoMathText, KnowledgePile and StackOverflow across 13 downstream tasks demonstrate that $\texttt{EQUAL}$ reduces computational costs by 5–10$\times$ while improving accuracy by 2.5\% on LLaMA-3.1-8B, Qwen2.5-7B and Mistral-7B. Code and data is available at https://anonymous.4open.science/r/EQUAL-DD20.

Published as a conference paper at ICLR 2026
NoT ALL DOCUMENTS ARE WHAT YOU NEED FOR EXTRACTING INSTRUCTION TUNING DATA
Chi Zhang $1.3 Huaping Zhong" Hongtao Li², Chengliang Chai'!, Jiawei Hong", Yuping Wang', Yuhao Deng', Jiacheng Wang', Yizhou Yan", Jiantao Qiu?, Conghui Het3, and Lei Cao
Beijing Institute of Technology
SenseeTime Research
Shanghai Artificial Intelligence Laboratory
*University Arizona
Meta
ze3150bit cdu. en zhoonghuagiinggeccnseiii... com
ABSTRACT
Instruction tuning improves the LLMs performance but depends on high-quality training data. Recently, LLMs have been used synthesize data, enhancing training with seeds like question-answw answer (QA) pairs, However, this synthesis often results instruction examples similar to the seeds, lacking diversity and biasing real applications. Thus, we propose to extract instruction tuning data from web corpus with much rich knowledge. The most straightforward strategy is to quickly retrieve domain specific documents from the corpus and then extract all QA pairs of these documents for tuning LLMs, which has two main limitations.
(1) Extracting all QA pairs using LLMs is prohibitively expensive; and (2) These extracted pairs are not all beneficial for the downstream applications, and incorporating all them for tuning may even hurt the model performance. To overcome the limitations, we introduce EQUAL, an Effective and scalable data extraction framework that iteratively interleaves document selection and extract high-QU QUtttty QA pairs optimize instruction tuning. EQUAL first clusters the document set based on the embeddings generated by contrastive learning. Then leverages the multi-armed bandit based strategy to quickly identify document clusters where can extract high-quas high-quality QA pairs for training. This iterative framework significantly reduces computational costs while improving model performance much. Experiments on Text,, KnowledgePine and StackOverflow across 13 downstream tasks demonstrate that E.UAAL reduces computational costs by 5-10x while improving accuracy by 2.5% on LLaMA-3.1-8PP Qwen2.5-7B and Mistral-7B. Code and data is available at httpt:: www.y/ooooommuummoaaao 4open. www.nceeeeeeeccccc...o... EEUUEEUULLLLE DD20
INTRODUCTION
Previous studies have shown that instruction tuning enables the powerful reasoning capability of Large Language Models (LLMs) (Ouyang al.] [2022] Achiam al. 2023 Dubey et al. (0244). but requires sufficient high-quality traiming data (Ntoutsi et al. [0200] Yu et al. 2023; Shah et al. 2024). However, although the weights of the open I.MMs publicly available, the datasets employed to fine-tune these models are generally private. This lack of data accessibility limits the opportunities to
effectively adapt LLMs targeted domains (Cobbe et al., 2021b Hendrycks al. 2021).
Recently, leveraging LLMs to synthesize instruction tuning data (Li et al. 2024a Yue et al. 2024
Luo et al. 2023 Yu et al. 2023 Li et al. [2224]] [Dig al. 2024) has attracted much attention
as an effective solution to enrich the original training data based on some seeds (e.g., original
"quual Contribution.
*Corresoondence to: [ccl 0bi t edu..ccc heconghui @pjlab. ooo.labborg........ org.en) This work done during the internship Shanghai Artificial Intelligence Laboratory.

---

Published as a conference paper at ICLR 2026
question-answer pairs, knowledge bases, etc.), thanks to the powerful understanding and generative capabilities of LLMs. However, achieving high-quality synthetic instruction data is challenging because LLM-based generation tends to closely imitate seed examples. When those seeds lack diversity, the synthesized data inherits this shortcoming, leading to degraded overall quality (Guo Xu al.
et al. 2024b Li al.
2024c
2024 Ding al. 2024).
Data Extraction from Documents. In reality, there is plenty of high-quality web corpus (c.g.. Common Crawl) which contains rich knowledge and can be leveraged as high-quality instruction data. However, this wealth of knowledge is widely spread within the corpus. Recently, (ue al. 2024) proposed method to retrieve domain-specific documents from large web corpus. followed by employing high-pun segh-performance LLMs to extract QA pairs from these documents and then using the extracted QA pairs to fine-tune an L.M.. However, has the following limitations. Quality Cost
Prohibitive Computational Cost. To extract a high-quality instruction tun-
Soultions
Salution
ing dataset, it uses LLMs repeatedly scan and analyze all the documents to extract question-answer (QA) pairs, each which often requires multiple LLM calls (Gilardi et al., 2023 Yue et al. 2024). Consequently, this pro- cess is prohibitively expensive, espe- cially when there are large number of documents process. Solving this problem requires largely reducing the number candidate documents, e.g.. by discovering the documents most valuable to instruction tuning.
88
QA poir extraction
Select high- quality QA poirs
Solution 2
QA pair extraction
selection
EQUAL
Document selection
QA puir extraction
score computation
Instruction QA Pairs. Even if an or-
Score update
ganization could afford extracting all domain-specific QA pairs from a large number of documents, blindly incorporating all of them to fine-tune an LLM could potentially degrade the model performance due to the presence of significant amount of low quality data. Specifically, a large corpus inevitably includes wild and noisy data with heterogeneous distribution that can degrade model performance on downstream tasks. Therefore, it necessary to judiciously identify high-quality QA pairs for extraction.
Figure 1: Number of training iterations on different models
Intuitive Solutions. There two intuitive solutions to address the above limitations. Solution O: extract all QA pairs first and then select high-quality pairs. Solution @: Select high-quality documents that potentially contain high-quality pairs first and then extract QA pairs from them. Unfortunately, neither of the above two methods can solve both limitations. To be specific, for solution D, although it can achieve good model performance, extracting all pairs beforehand still costly. Solution @ is cost-effective, but it is difficult to accurately discover high-quality documents because documents and pairs have different feature distributions. Note that in this case, the quality of these documents should be measured by the potential contribution of the QA pairs to the target distribution, which is not aligned with the data quality duplication. In other words, even if the embedding of a QA pair is close to that of document, it does not necessarily indicate that the QA pair is close to the pair potentially extracted from the document.
the original documents, c.g., dirty data,
Key Idea. To address both limitations, key idea is to interleave document selection and QA pairs extraction. During this iterative process, the extracted QA pairs help capture the relationship between the document and the pair distribution more and more accurately, and at the same time, the selected documents improve consistently.
Our Proposal. Inspired by the above idea, we propose EQUAL, scalable and effective data extraction framework for constructing QA pairs from documents, aiming to enhance the LLMs instruction tuning. To be specific, EQUAL first clusters over the heterogeneous document considering the feature similarities of QA pairs extracted from these documents. To achieve this, we introduce warm- up step using contrastive learning align the feature space between documents and QA pairs. In this way, EQUAL effectively identifies those high-quality clusters by sampling and extracting QA pairs from them to save cost. Afterwards, we propose a Multi-arm Bandit (MAB) based technique
2

---

Published as a conference paper at ICLR 2026
to iteratively select the clusters. As the reward function, it predicts the benefit of QA pairs that potentially could be extracted from the clusters. More specifically, in each iteration, EQUAL tends to select the cluster where documents can produce QA pairs that are likely to benefit the target model performance. This benefit is measured by the optimal transport (OT) score, where a higher benefit score indicates smaller difference between the distributions of the QA pairs in the cluster and the larget distribution. Then, given the selected cluster, EQUAL samples some documents from it, extracts QA pairs using LLMs, and in turn updates the optimal transport score of this cluster accordingly. In this iterative process, we precisely estimate the distribution of the QA pairs in document cluster without having to conduct extraction over all documents.
Moreover, leveraging the upper confidence bound technique in MAB, EQUAL promotes the potentially low-quality, thus under-sampled clusters. Therefore, improves the diversity of the extraction data. This balance between exploration and exploitation effectively avoids reaching a local optimum.
To summarize, we make the following contributions:
(1) We propose EQUAL, novel framework for data extraction from documents to enhance LLMs instruction tuning with high scalability.
(2) We incorporate an iterative MAB solution to first cluster the documents and extract data from these clusters, achieving a good exploration-exploiaaiio trade-off. We also propose a warm-up strategy to align the features of documents and QA pairs.
(4) Extensive experiments on datasets (AutoMathText, KnowledgePile and StackOverflow) with more than 1 million documents and 13 popular downstream tasks demonstrate that EQUAL significantly outperforms baseline methods by saving 5-10x computation resources consumption while still improving 2.5% in accuracy (train/test Llama-3 3.1BB, Qwen2.5-7B and Mistral-7B model).
PRELIMINARY
We first introduce the necessity of extracting and selecting QA pairs for instruction tuning. followed by our problem definition.
Instruction tuning (a.k.a,, supervised fine-tuning (SFT)) aims to adapt a base LLM to specific domains or user needs, thereby enabling better performance on downstream tasks (Wang al. 2024b 2023a| Zhang et al.] (023). Consequently, many studies on data selection for SFT (Li et al. 2023b Xia et al. 2024 Ni et al. 2024) emphasize selecting Comain-beneficia data from a candidate pool with reference to the target capabilities. This is because fine-tuning LLMs without careful data selection-for instance, using QA pairs extracted indiscriminately from all documents in the candidate set-can impede the model's ability acquire target capabilities. Empirical results from prior works (Kim et al. 20233] Muennighoff et al.) 2002)] Wang et al., (203b)) demonstrate that LLMs fine-tuned on targeted subsets of data outperform those trained on the full SFT dataset, underscoring the importance of relevance-aware data selection.
QA Pair Extraction. To construct SFT data, given a large number of documents, people always leverage advanced LLMs to extract and refine QA pairs. More specifically, LLMs such as Qwen2.5- 72B (Yang al.) 2024a) are utilized to extract QA pairs within documents. By incorporating examples within the prompt, we guide the model to focus on the desired QA pairs while filtering out markups, boilerplate, and other irrelevant content. Although Qwen2.5-72B demonstrates impressive capabilities, the extracted QA pairs still exhibit issues such improper formatting. missing answers, or mismatched responses (Li et al., 2023a| Honovich et al., 2022 Chen et al., 2023). To address these problems, we employ further refinement to these QA pairs using Qwen2.5-72B, which has demonstrated significant enhancement in the quality of the extracted pairs (Xu et al. 2023). The prompts for extraction and refinement are provided in Appendix W
Overall, the entire extraction process requires each entire document as input and repeatedly calling pegforreformanc LLMs, which is rather expensive. Thus, in this paper, focus reducing the number of documents to be extracted to save the cost while keeping high model accuracy. However, how to improve the quality of each extracted pair within each document is orthogonal to this work.
Problem Definition. Formally, we study the problem of data extraction from a candidate document pool DD to extract QA pairs for instruction tuning. Formally, given D. and reference set D, for
3

---

Published as a conference paper at ICLR 2026
4. DS Score Update
1. Clustering
Cluster Sampling
QA-pair Extraction
Add
exploiation
exploration
LLM
Original Documents
2T T.
OT
OT.
Extracted Data Pool
Warmup Phase
near
QA pairs
Distribution Similarity
D
refernece ddstribution:
relevant Doc & QA pair
Doc
inrelevant Doc QA pair
far
embedding model
Figure 2: The Overall Framework EQUAL.
instruction tuning, the problem is to select a subset D, De from which set of QA pairs Q are extracted fine-tune an LLM M, aiming to minimize the loss of the updated model M' D,.
3 PROPOSED APPROACH.
3.1 THE EQUAL FRAMEWORK
Multi-Armed Bandit (MAB) (Vermorel & Mohri 2005) is an effective framework that makes decisions over time under uncertainty. This consists of N possible actions, each known as an arm. Pulling an arm indicates sampling from this arm to capture its reward distribution more accurately. This framework characterizes an agent that iteratively gains new knowledge by pulling arms that are rarely visited (i.c., exploration) while using current knowledge to enhance its decisions by pulling arms already with a high reward (i.e., exploitation). The agent aims balance the exploration and exploitation to maximize their overall reward throughout the given time span.
Bridging EQUAL and MAB. The overall process EQUAL is illustrated in Fig 2 and Algorithm To reduce the computational cost, first cluster all documents in the candidate dataset D) (line such that the QA pairs extracted from each cluster are similar (Step-1 in Fig 2, see $3.22 for details). Thus, can neglect these low-quality clusters to save the cost. To precisely measure the quality of the cluster C; the most straightforward way is to extract all QA pairs and compare their distribution with that of the reference data, measured by the optimal transport score (see $ $3.3 for details), denoted by OT; but it is still very expensive. Hence, we iteratively sample from these clusters to estimate the score. Our key idea inspired by natural connection between EQUAL and MAB.
At a high level, each cluster can be regarded as an arm of MAB. EQUAL iteratively selects a cluster, samples some documents and extract pairs from (i.e., pulling an arm). To be specific, shown in Step-2 of Figure 2 a cluster with high estimated optimal transport score (OT,) tends to be selected. The higher the score, the smaller difference between the distributions of QA pairs from this cluster and the reference data. Moreover, clusters that are rarely visited (denoted by the sampling frequency T(C)) tend be selected as well to explore more diverse documents. Overall, putting OT, and T(C;) together, we use the document sampling (DS) score to measure the quality of a cluster, which can achieve a good exploration -exoraation-ellliiaationn trade-off. Subsequently, as shown Step-3, we obtain these extracted pairs (line 5-7) and update the score OT; of the corresponding cluster (line 8). As more pairs from cluster C, are extracted, the estimated score OT; will become more accurate.
Next, we illustrate the details of the EQUAL framework.
DS Score Computation. Following the upper confidence bound Auer 2002) in the typical MAB framework, define the DS score DS, of the cluster Cj to effectively balance the exploration (i.e.,
4

---

Published as a conference paper at ICLR 2026
data diversity) and exploitation (i.e., data quality) follows.
DS; OT, + a 2 Ecyec T(C))
T(C2)
where T())) denotes the frequency of documents sampled from cluster Cj. Ec T(C)) denotes the total sampling times from all clusters. a set as Σo,er T(CH+ (Hao al. 2019), which provides higher weight to exploration early stages, but in later stages, it provides higher weight to exploitation (line 9-11).
Update the DS Score. In each iteration, subset of documents B; is sampled from the selected cluster C3 with a high DS score, and a set of QA pairs Qi is then extracted from the documents in B, The OT score of C will be updated as follows.
OT, OT(UQ;, D,),
T(C.)+=1
(2)
where UQ, denotes all the extracted QA pairs cluster C, obtained from the beginning and OT(-) denotes the function of computing the OT score. Then we update the DS score of all clusters. Extracted Pairs Collection. As shown in Figure 2 in each iteration, we add the extracted QA pairs Q; to the extracted data pool D.. Finally, use the pairs in the pool to fine-tune the LLMs.
Algorithm 1: EQUAL Algorithm
Input: Candidate data pool D., reference set D,, extracted data ratio 7.
Output: Extracted data pool D..
C =
Cluster(D.):
3.2 WARM-UP FOR CLUSTERING
D.
= 0
Motivation. Considering that the original documents contain much irrelevant content with downstream applications. there exists
while |Del < [D.] do
Select cluster C, with the highest DS Score;
Sample B; documents from C
6 a discrepancy between the feature space of document embeddings and that of the QA pairs extracted from them. Obviously, we hope that similar QA pairs fall into the same cluster, but similar documents do not necessarily indicate similar pairs if we clus- ter purely based on feature embeddings of documents. However, it is rather expensive to extract all pairs and then cluster. There- fore, to improve the clustering quality, we propose to incorporate warm-up step align the two feature spaces using contrastive learning (Khosla et al. 2020).
Extract QA pairs Q from B;;
D. D. U UQi
OT, = OT(UQ, D+), T(C:) +
8
for Cj in C do
2 Lnnce T(C))
DS, = OT, + tap
T(C₁)
end
12 end Return De;
Feature Alignment. In the warm-up stage, we first randomly sample small proportion of the documents from the candidate data pool and use LLMs to extract QA pairs. Then, we fine-tune the model (i.e., BAAI /btee en-vl. 5) used for document embedding to capture the deep connection between the original documents and the extracted QA pairs. Specifically, we treat the sampled documents and these extracted QA pairs as positive training examples (denoted by (d,q*) (+))))))))))))))) for contrastive learning. To generate negative examples q for document d, we conduct negative sampling from all current QA pairs. Then we train with the following loss function:
coim(d.q*)
= =log esiml(gg)) + Σ esim(d.aa)
(3)
where sim denotes the cosine similarity between the embedding of a document d and QA pair In this way, documents containing similar QA pairs tend to be closer the embedding space and are thereby grouped together in the same cluster.
3.3 MEASURING CLUSTER BENEFITS
Heuristic methods (Xia et al. 2024 Xie et al. 2023) estimate the pointwise benefit of each data point (i.e., its contribution to the target capabilities) and simply aggregate these benefits cluster benefits,
5

---

Published as a conference paper at ICLR 2026
implicitly assuming that each data point contributes independently of the others. However, this assumption fails even in simple linear regression tasks, as systematically demonstrated in (Hu et al. 2024). In contrast, EQUAL formulates targeted data selection distribution matching problem. aiming to identify a subset of candidate data that closely matches the target distribution. Specifically, EQUAL computing the distribution similarity of extracted data and reference data utilizing Optimal Transport (OT) (Villani et al.) 2009), which is widely adopted to compute the minimal cost of transforming one distribution into another (more details in Appendix H. Specifically, the lower the transportation cost, the closer the two distributions, indicating that the extracted data is more beneficial for the target distribution.
In our scenario, suppose that the distribution of extraction data and reference set is f and v separately, The transportation cost from j to v can be calculated by OT (j, v):
OT(,, v) def inf =Ell(W.))
(tep.ew)=alllen ev]
where en and e, denote the embedding of extracted QA pairs qu qu from the two distributions, I(,, 2/) denotes the set of all joint distributions π (e Cv) with marginals μlee)) and v(e,). Here, Cxe X x X R is the cost function for moving Cp to Cv. where X denotes the entire c,
embedding space in BQUAL. To be specific, we use 1 [|e,|l e.. as the transportation cost between eu: Ce. which is a popular choice to measure the semantic dissimilarity (Pennington et al. 2014). Then given two distributions f and v, there are numerous possible mappings (i.c., II) between pairs from these distributions. The cost for each mapping can be calculated by various c(E,s, ev) within the mapping, and the OT score represents the minimum cost among all the mappings.
4 EXPERIMENT
In this section, we fine-tune the base models in different domains and conduct sufficient ablation studies to demonstrate the efficiency and effectiveness of EQUAL. More experiments such as enlarging the model size and the diversity of downstream tasks is provided in the Appendix] [[
4.1 EXPERIMENT SETUP
Training Settings. We evaluate EQUAL using two foundational models (i.e., LLAMA-3-SB, Qwen2.5-7B and Mistral-7B) and two training settings, i.e., full fine-tuning (FULL) and Low- Rank Adaption (LoRA). In both training scenarios, the batch size is set 512 and the maximum learning rate is set as 1 x 10 5 with a cosine decay schedule. For the FULL setting we train the extracted data for 2 epochs on 32 H100 GPUs, while for the LoRA setting we train the extracted data for 4 epochs on 16 H100 GPUs. For the warm-up stage, we randomly select 5% documents from D. to extract QA pairs and then use contrastive learning to fine-tune the original embedding model BAAI bage/bgeeeee booge-en-vll en-v1.5 model, and then it is employed to generate document embeddings for subsequent clustering.
Dataset Preparation. In our evaluation, we use Math (Zhng et al. 2024). Knowl- edgePile (Fei et al., (024) (Appendix [ and M and StackOverflow (created by us) datasets as the candidate data pool D. for mathematical, general and coding tasks respectively. Tuto totally contains 4.9M documents, from which we select 1.4M ones by filtering documents with a metadata score of lm_qiqq (2_____reee___ < 0. exclude math-irrelevant content. We use the dataset after filtering as the candidate data pool. StackOverflow is crawled by us from stackover low com which contains 1. 1.2M documents in total. Then we implement an r--ramm filtering (Guo et al.) 2024a) to ensure that our training data is not contaminated by information from the downstream tasks. For the reference (validation) set D,,, we respectively use the training set of GSM8K (Cobbe al.) 2021a) and MBPP (Austin et al.) (02T) for math and code domains, which are both widely used language modeling tasks and often serve as a validation benchmark for instruction tuning. During the clustering process, documents from D. are clustered into 1. 1,000 clusters using the k-means algorithm. The number of clusters is automatically determined by the Elbow (Syakur et al. 2018) method in EQUAL.
fim__i92_scov is metadata attribute for each document in Auto Text ranging from [0, 1]. which quantifies the document's 's relevance, quality, and educational value in the context of mathematical intelligence.

---

Published as a conference paper at ICLR 2026
Table I: Comparison with other algorithms in test accuracy (%) on TuteMat and StarkOverflow
The best results are highlighted. We run each experiment for three times and report the average.
Model
LLAMA-3JB
Mistral-BB
Domain
Math
Code
Methods
GSM8K
MATHI
FLOPPs
HUMANEVAL
PLOPS
FLOP%
HUMANEVAL
MBPP
Bow Model
55.19
23.04
31.1
45.30
14.30
25.2
41.8
Random
LdRA
63.76
10.26
8.05
6.33
25.0
45.0
Jog
LoRA
65.64
30:12
114.70
05.53
45.1
Perplexily
LoRA
63:61
30.94
134.35
31.1
71.07
134.18
25.6
44.6
Infuneaee
I-RA
63:45
240.31
33.5
55.0
130.46
18
133. 90
26.2
41.5
LLM scoring
LoRA
65.11
30.55
271.56
34.3
53.3
181.28
53,02
19:33
260.51
27.7
44.5
137.37
Rewiiing
L-RA
62:21
27.05
18.11
30.7
34.1
13:15
50.19
17.72
17.90
26.3
44.3
13.01
Pemplaxit-MAA
MAB
LoRA
64.52
50,45
18.18
34.6
55.0
13.24
51.78
22.00
17:36
27.4
45.6
12.99
Inflemo-MAR
LoRA
65.75
50.78
23.57
35.3
55.1
19.13
55,12
19.58
18:44
28.7
44.7
18
LoRA
67.32
31,86
17.75
36,0
$5.3
12,99
57,54
23.56
17:57
31.3
46.7
12:64
Random
FULL
68.92
52,46
8.83
42.7
52.3
7.119
51.41
25.76
8.74
31.1
64.2
6:34
70.35
33,15
114.44
37.8
45.1
Poplexity
F.
64-52
33.56
1100-88
148.0%
Infreence
65.20
29.64
256.94
39.6
131.24
22.06
248.97
45.6
LLM-aoorng
68.38
33.19
273.91
46.9
142.73
22.72
34.3
46.1
139.16
Rewniing
F.
50.82
18:71
43.4
13:78
57.21
22.67
18.73
33.3
43.6
13,67
Perplexity MAB
FULL
65:28
32.92
18.96
42.5
49.1
13.76
56.44
26.20
18.44
34.1
42,9
13.42
Infiuence MAD
FULL
67.78
32,86
25.82
46.3
53.5
19.91
60.42
24.44
19.70
34.8
44.0
18.79
EQURZ/Ours)
FULL
73.01
35.10
18.55
49.4
56.3
13.50
67.73
28.28
18.18
39.1
50.6
13.07
Baselines. We compare EQUAL with several baselines. (1) Random. We randomly sample documents to extract QA pairs from De for fine-tuning. (2) All (Mammoth). . We fine-tune our model using the QA pairs extracted from all the documents in candidate data pool DD,. which is the same method used in Mammoth ( Yue et al., 2024). (3) Rewriting Yuu et al. 2023) synthesize new QA pairs based on existing QA pairs (specifically, the reference set D, in our setting) using LLM. We synthesize the same number of pairs as other baselines. (4) Avg sim. We extract QA pairs from all documents in De. Then we select QA pairs with the highest average similarities with the ones in D,., For each QA pair, we compute the embedding similarities between the pair and all pairs in D,,, and compute the average. (5) Perplexite ty (Li al.) 2024b) extract QA pairs from all documents in De. Then select QA pairs with the highest perplexity scores. (6) Inf Luence (Xia et al. (024) extract QA pairs from all documents in D... Then we select extracted QA pairs with the highest influence scores. (7) LLM--MMM scoring (Wettig et al. 2024) employs high-performing LLMs to evaluate the quality score of documents. (8) perplexity--paee t- utilizes perplexity (Li et al.) 2024b) score as the reward utilizes influence (Xia
of MAB to select documents for extracting QA pairs. (9)
et al.
2024) score as the reward to select documents. (10) EQUAL is our full alll-feedgdd solution.
Metric. We evaluate the quality of extracted data by accessing the LLM performance, which has been fine-tuned with these data on several commonly used downstream tasks. (1) Math domain: GSM8K (Cobbe et al. 22021a) and MATH (Hendrycks et al., (221]) are utilized for evaluating math
tasks. (2) Code domain: the fine-tuned LLM is evaluated on JUMANEVAL (Chen et al.) (021) and
MBPP (Austin et al.) [222] datasets. We also report FLOPs to quantify the total GPU cost across the
following three stages: data extraction, data selection, and model training, with details in Appendix F
4.2 RESULT
Overall Performance. In Table we acquire 5% documents (70k from AutoMathText and 60k from StackOverflow) for each baseline. We can observe that EQUAL surpasses all the baseline methods on accuracy across all models and downstream tasks. Specifically, when implementing Full fine-tuning on Llama-3.1-8B EQUAL achieves an accuracy improvement of 4.09% on GSM8k and 2.64% on MATH compared with Inf luence while saving approximately 5x w.r.l. the computational cost. EQUAL surpasses Rewrit tt ing due to the fact that the QA pairs generated by Rewrit t tn are quite similar to those QA pairs in D,,, resulting limited data diversity. Besides, the pairs directly generated by LLMs might be error- error-prone due to the hallucinations. EQUAL outperforms Pe Perple sxleexityy and Perplexi ty-MAB because perplexity score is solely based on the inherent complexity of the QA pairs to select extracted data without considering the downstream tasks. Besides, EQUAL outperforms Inf and Inf 1 cenBe-MAB because influence function is easily affected by the length of the sequence (Xia et al. 2024), often leading to the selection of pairs with fewer tokens (more details in Appendix G). Also, EQUAL outperforms LLM-score and Avg-s im because it selects data based on the similanties between QA pairs, without considering the overall distribution. In terms of the computational cost, we can observe that the FLOPs consumed by the Avg- Influ rafuuence and Pe Perplexi ty extraction method are notably high. This is due their necessity of extracting
7

---

Published as a conference paper at ICLR 2026
Table Comparison with Random different ratios.
LoRA
FULL
Method
Math
Code
Math
Code
GSMBKK
MATH
FL.0%:
HUMANEYAL
MRPP
FL.OP%
GSMMKK
MATH
FL.0Ps
HUMANEVAL
MEPP
FLON
(518)
90.76
67:40
43.7
Kandom (10%)
66.03
30.82
(209))
06.05
31.76
7000
64.1
65.43
32.00
155.6
35.3
70.25
40.02
43.6
56.0
EQUAL (58)
67.32
31.86
73.01
35.10
49.4
EQUAL- (109)
68.10
31.66
23:21
38.8
36.0
74.46
50.1
56.0
(QUALL(209))
68.69
33.43
39.6
35.5
41.40
49.5
56.4
the QA pairs from all the documents in D., which incurs prohibitive cost. Additionally, since the influence score used in Inf uuece requires compute gradients during back propagation, leading to higher FLOPs consumption than EQUAL. In contrast, can be seen that for the influence score used in Influence and the perplexity score used in Perplexity, when combined with the MAB framework, comparable results can be achieved at a lower computational cost, which demonstrates the efficiency of the MAB strategy. The detailed performance of EQUAL Qwen2.5-7B can be found in the Appendix S)
Table presents comparison between EQUAL and Random using various QA pair extraction ratios ranging from 5%. 10%, and 20% up to 100% (i.e., high-cost extraction from all documents). Interestingly, find that the extracting of just 5% of QA pairs for most tasks produces superior results compared to the use of complete D.. This demonstrates the effectiveness of EQUAL. Even for the difficult task MATH, extracting QA pairs from only 20% to the documents in D. can achieve comparable performance to All (Mammoth) across all the training settings. This is because not all the QA pairs extracted from all the documents in De might contribute to the target tasks.
4.3 ABLATION STUDY
In this section, demonstrate the effectiveness of Warm-up phase, Multi-Armed Bandit (MAB) and Document Sampling (DS) score through experiments conducted with no-wa rmup, no-MAB and no-DS settings, which is shown in Table Also, we conduct several ablation studies w.r.f. the number of clusters, different clustering algorithms etc., and the results are illustrated in Figure 3
Effectiveness of Warm-up Phase. As shown in Figure points of the same color represent QA pairs extracted from the documents in the same cluster. Figure [3 shows the clustering results obtained by directly using the existing model to compute embeddings for each document, while Figure 3b presents the results after fine-tuning the embedding model during warm-up phase. Since the original documents contain many contents irrelevant to the extracted QA pairs, leveraging the document embeddings directly to cluster will lead to the inconsistencies of QA pair embeddings within each cluster. can be observed that the warmup phase effectively aligns the embeddings of documents and their corresponding QA pairs. Also, we also conduct the experiment no -armup, which use the original embedding model /AAee en-vl. 5 to generate document embeddings for clustering. Following this, similar EQUAL, both the MAB framework and optimal transport employed to extract QA pairs iteratively based on the clusters. Table 3 illustrates that EQUAL surpasses no-wa rmup across all experimental settings, demonstrating the effectiveness of the warm-up phase.
Effectiveness of MAB. In this sec- tion, we evaluate the performance of MAB by contrasting it with a sim- ple method called no-MAB, which ex- tracts QA pairs directly from several clusters with distributions similar to the reference set without iterative ex- traction like MAB. Specifically, we start by sampling small set of docu- ments from each cluster and then ex- tract QA pairs. Subsequently, opti- mal transport is employed to assess the similarity in distribution between these pairs and the reference set, serv-
Table 3: Effectiveness of Warm-up phase, Multi-Armed Ban- dit, Document Sampling (DS) score in EQUAL.
Domain
Math
Code
Method
GSM8K
MATH
HUMANEVAL
MBPP
no-warmup
LoRA
64.05
30.82
32.8
54.1
ne-MAB
LoRA
66.13
30.55
33,3
53.3
no-DS
LoRA
65.59
31.08
34.4
53.7
EQUAL
LoRA
67.32
31.86
36.0
55.3
по-warmup
69.73
33.51
44,3
53.8
Re-MAB
Full
71.90
33.25
46.9
55.5
ng-nS
Full
70.77
33.40
47.6
54.6
EQUAL
Full
73.01
35.10
49.4
56.3
ing as the score for each cluster. Then, the documents within the top-scoring clusters utilized
8

---

Published as a conference paper at ICLR 2026
73
-40
71
18
MATH
COPMEC
89
32
8
(a) Number of Clusters
Clustering Algeritims
(a)
(c)
Figure 3: (a) shows the clusters based the original embedding model; (b) shows the clusters based on the fine-tuned embedding model; (c) ablation study of cluster numbers and clustering algorithms.
EQUAL outperforms no BB because during the data extraction process, the OT score for each cluster updated dynamically based on the previously extracted QA pairs, thereby enhancing the accuracy of the subsequent selection.
to extract QA pairs and fine-tune the LLM. As illustrated in Table 3
Effectiveness of Document Sampling (DS) score. We evaluate the effectiveness of DS score by comparing EQUAL with no-DS, which utilizes the average similarities with QA pairs in D, as the MAB reward. As shown in Table EQUAL surpasses the no-DS across all the settings. This indicates that DS score provides more precise estimation of the distributional similarity between the extracted data and the reference set, hence better aligning with downstream applications.
Number of Clusters. We use the Elbow (Syakur et al. 2018) method to identify the optimal cluster numbers for AutoMathTess Text and StackOverFlow datasets. In Figure 3 [). we plot the accuracy of EQUAL with different cluster numbers. When the cluster number is around 1000 (the selected optimal number for both datasets), the model consistently performs well. However, a very small number of clusters (i.e.,
50) leads poor accuracy (2.90% and 2.05% lower accuracy on GSM8K and MATH tasks) due to the high variance of QA pairs extracted from the documents in each cluster. Thus, the QA pairs extracted from the sampled documents cannot well represent the cluster. Similarly, when the cluster number is too high (i.e., k 50, 000), there will be many clusters that are in fact contain similar QA pairs, and thus is hard to explore diverse clusters, thereby leading to the performance degradation (2.56% and 1.79% lower accuracy).
As shown in Table we analyzed the impact of the warm-up ratio
Warmup Ratio.
Table 4: Impact of warm-up ratio on model accuracy.
EQUAL, varying it from 0.1%, 1,, 1% to 5% of the candidate document pool D.. Our experiments confirm that using a 1 % warm- up ratio of the corpus achieves comparable performance with 5% under lower computation cost.
Warm-up Ratio 0.1
0.5%
1%
3%
5%
Accuracy
69.66%
71.87%
72.91%
72.96%
73.03%
Clustering Algorithms. Moreover, we evaluate the performance of EQUAL by other typical clustering algorithms including BIRCH (Zhang et al., 1996) and DBSCAN (Ester al. 1996). The details of selecting optimal clustering parameters can be found in the Appendix D As illustrated in Figure EQUAL is robust to clustering algorithms downstream tasks.
4.4 FURTHER EXPERIMENTS
Chain-of-thougle (CoT) Data Gen-
Table 5: Performance comparison with advanced baselines.
eration. In this section, we consider a setting where, for each QA pair, cor-
responding chain-of-thougee (CoT) ra-
Method
AIME24
AIME25
L/veCodeBeuch
CodeElo
FLOPs
tionale is distilled from more capa- ble teacher model. In this CoT distilla- tion scenario, we apply our proposed method EQUAL and directly compare it against the state-of-the-art synthetic data generation meth-
Random
35.1%
26.0%
23.2%
7.29%
50.65
97k
40.7%
32.7%
26.7%
9.30%
197.87
OpenThoughts
40.1%
33.1%
26.1%
9.55%
238.20
ROURL
42.7%
34.2%
27.6%
10.05%
65.17
ods OpenThoupe s and s s1k under the same conditions (details of the experimental settings and baseline implementations are provided in Appendix TT). As shown in Table EQUAL achieves the best performance while incurring the lowest computational cost (excluding the heuristic Random). This advantage stems from performing document-lewe selection before QA pair extraction and CoT generation, demonstrating the generalizability of EQUAL.

---

Published as a conference paper at ICLR 2026
Experiments Qwen Model. More-
Table 6: Comparison under advanced setting.
over, following common practice in re-
conduct experiments on math tasks with Qwen2.5-Math-7B5 Instruct (Yang al., 2024a) and on code
cent studies,
Method
A1ME24
AIME25
LiveCodeeBench
CodeElo
Randon
33.1%
26.0%
23.2%
7.29%
Avg-såm
37.1%
30.7%
24.7%
8.04%
tasks with Qwen2.5- S---oooo------stteee al. 2024) (see Appendix for full exper- imental details). For evaluation, we include
(Hui)
Parplexity
31.9%
24.9%
8.29%
Int-vence
35.3%
29.6%
23.2%
7.04%
LIM scoring
37.6%
31.3%
25.1%
8.54%
Bewriting
39.3%
30.1%
24.2%
7.79%
the AIME24 (Zhang & Math-A1| and
EQUAL
42.7%
34.2%
27.6%
10.05%
AIME25 (Zhang& & Math AI 2025) benchmarks to assess mathematical reasoning, together with LiveCodeBench (Jain et al., 2024) and CodeElo (Quan et al. (025) to measure code generation per-
formance. As shown in 6 EQUAL consistently outperforms all baselines in this setting, further
highlighting its robustness and strong generalization across model families and target capabilities.
RELATED WORK
Data Synthesis using LLMs. In data synthesis, LLMs are commonly used to generate complex and
high-quality data (Zhang & Yang 2023 Yang al. 2024b) training data. For example, (Luo et al. 2023; Yu et al. 2023) construct LLMs pipeline to revise existing training data, thus enhancing the quality and complexity. (Sun et al. [0244] Wang al., 2024a) conduct principle-driven prompting. which inserts some well-crafted principles into prompts to guide the LLMs for synthesis. (Guo et al.) 2024b; Li et al.] 2024c) iteratively synthesize important instruction tuning data and train the model with the newly synthesized data in each round. However, the data generated by these methods often exhibits low diversity, few-shot prompts tend make the newly generated data very similar to the original data (Li et al.) 2024a Ding et al. 2024). To address this, researchers have proposed several techniques to generate diverse data. For instance, (Yu al. 2024a Gupta et al., 2023 use various prompts to synthesize diverse data. (Yoo et al. (2021) integrates different existing training data to generate more diverse data. (Divekar & Durrett 2024) uses retrieval augmentation to enhance data diversity by feeding LLMs different retrieved contents. Closer to our work, (Yue et al. 2024)
synthesizes QA pairs from vast web documents to enhance the diversity of synthetic data.
Data Selection for Instruction Tuning. High-quality data plays critical role in instruction tun- ing (Brown) 2020 Zhou et al. 2024| Xia et al. 2024). Simple approaches such as rule-based
methods (Soldaini et al. 20244
Penedo et al. 2023) and deduplication (Abbas al. 2023) can
improve data quality but the improvement is limited due to simple heuristics. More sophisticated methods like LLMs like GPT-4 assess data based on human-defined metrics but this is rather expen- sive (Wettig et al.] 2024). Perplexity-basee -based methods (Marion et al., 2023 Li et al. 2024b) select data samples that difficult for the model to predict. But all above methods do not consider the target distribution of downstream applications. The influence function can measure the impact of training
data on downstream model performance (Grosse et al.) 2023 Xia et al. 2024), but is computationally intensive and affected by the length of the sequence (Xia et al. 2024), leading to imprecise filtering. (Shao et al., [2224] Yu et al., (2024b) develop a surrogate model trained on available high-quality data to efficiently select data from the candidate dataset; however, its effectiveness is limited by the generalizability of the model.
CONCLUSION
This paper present EQUAL, a scalable and effective method for instruction tuning data extraction. EQUAL first use contrastive learning unify the embedding feature spaces of the original documents and the extracted QA pairs. Based on this, EQUAL clusters all candidate document and regards each cluster as sampling from quality clusters to estimate distribution similarity scores accurately while maintaining diversity.
of MAB framework due uncertain distribution similarity scores. allowing
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Chengliang Chai is supported by the NSFC (U25B2019, 62472031), the National Key Research and Development Program of China (2024YFC3308200), Beijing Nova Program, and Huawei. Yuping Wang is supported by the NSFC (U23A20297, U23A20317). Lei Cao is supported by the NSF (DBI-2327954) and Amazon Research Awards
ETHICS STATEMENT
This work adheres the ICLR Code of Ethics. In this study, no human subjects or animal experimen- tation was involved. All datasets used, were sourced in compliance with relevant usage guidelines, ensuring violation of privacy. We have taken care avoid any biases or discriminatory out- comes in our research process, No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concems. We are committed maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results reported in this paper are fully reproducible. Our experiments conducted with clearly specified datasets, model architectures, and hyperparameters. All data preprocessing steps, training procedures, and evaluation metrics are explicitly described in the main text and supplementary materials. The codebase used for all experiments will be released publicly upon publication, including scripts for training, evaluation, and data preparation. Additionally, we provide pre-trained models and detailed instructions for reproducing all reported results. We also include sufficient ablation studies and analysis to allow independent verification of our claims.
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 23X.x:2333008774 2023.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
P. Auer. Using upper confidence bounds for online learning. In Proceedings 41 Annual Symposium on Foundations of Computer Science, Nov 2002. doi: 10. (099//fcs.2000.922111 URL http:
7/xx org. 10 11..1.199/fffss. 2000. 892116
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 108.077.. 2021.
Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. Knot: knowledge distillation using optimal transport for solving nlp tasks. arXiv preprint arXiv:2110.02432, 2021.
Tom B Brown. Language models few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Jaime Carbonell and Jade Goldstein. The use of mmr, diversity- based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 335-336, 1998.

---

Published as a conference paper at ICLR 2026
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint ar.iv:2307.087700 2023.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. Improving sequence-tor trueeece-o--eqquencee learning via optimal transport. arXiv preprint arXiv: /901 00628,, 2019.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. Cornell University Comell University arXiv, Oct 2021b.
Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. Unleash- ing reasoning capability of Ilms via scalable question synthesis from scratch. arXiv preprint arXiv:2410.18693, 2024.
Abhishek Divekar and Greg Durrett. Synthesizrr: Generating diverse datasets with retrieval augmen- tation. arXiv preprint arXiv:2405.10044 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30088 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04473 04475, 2024.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pp. 226-231, 1996.
Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Hang Yan, Xipeng Qiu, and Dahua Lin. Query of ce: Unearthing large scale domain-specific knowledge from public corpora. arXiv preprint arXiv:2401. 14624 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30)e2300016120, 2023.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arxiv:2308. 03296, 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. deepseek-coder. When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv: 14196, 2024a.
Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. ioaman-instructionn free Ilm self alignmetttt with limited samples. arXiv preprint arXiv:2401.0677 2024b.
12

---

Published as a conference paper at ICLR 2026
Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Ar- jun Sawant, Chitta Baral, and Swaroop Mishra. Targen: Targeted data generation with large language models. arXiv preprint arXtv:231017887,, 2023.
Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. Advances in neural information processing systems, 32, 2019.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09683 2022.
Yuzheng Hu, Pingbang Hu, Han Zhao, and Jiaqi Ma. Most influential subset selection: Challenges, promises, and beyond. Advances in Neural Information Processing Systems, 19:119978-119810 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint 2XXi:2449912288 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.077 079774 2024.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Joongwon Kim, Akari Asai, Gabriel Ilharco, and Hannanch Hajishirzi. Taskweb: Selecting better source tasks for multi-task nlp. arXiv preprint arXiv:2305.13256 2023.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shao- han Huang, Xiaolong Huang. Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint 1xXiv:2402.13060 2024a.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou, Superfiltering: Weak-to-strong data filtering for fast instruction-tuning arXiv preprint 2xXiv:2402.005300. 2024b.
Qintong Li, Jiahui Gao, Sheng Wang, Renjie Pi, Xueliang Zhao, Chuan Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Forewarned is forearmed: Leveraging lims for data synthesis through hallure-indccin exploration. arXiv preprint arXiv:2410. 16736, 2024c.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259. 2023a.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang. Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, et al. One-shot learning instruction data prospector for large language models. arXiv preprint arXiv:2312.10300 2023b.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: constrained text generation challenge for generative commonsense 2019.
reasoning. arXiv preprint
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648, 1-22448. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint wrXv:2230.009583, 2023.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint (rXiv:2309.04564 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Xin Yong, Hailey Schoelkopf. et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022.
Jerzy Neyman. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. In Breakthroughs in statistics: Methodology and distribution, pp. 150-150. Springer, 1992.
Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen, Explor- ing the mystery of influential data for mathematical reasoning. arXiv preprint arXiv:2404.01065 2024.
Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Fsther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. Bias in data-driven artificial intelligence systems-an an introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3):e 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Christopher R Palmer and Christos Faloutsos. Density biased sampling: An improved method for data mining and clustering. Proceedings of the 2000 ACM SIGMÖD international conference on Management of data, pp. 82-92, 2000.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 11100116...0 2023.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao. Yibo Miao, Yunlong Feng, al. Codeelo: Benchmarking competition-levt level code generation of Iims with human numan-comparable elo ratings. arXiv preprint arXiv:2501.01255: 2025.
Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS), (2(3))1-221) 2017.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv: 1704. 04368, 2017.
Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009 2024.
Ketan Rajshekhar Shahapure and Charles Nicholas. Cluster quality analysis using silhouette score. In 2020 IEEE 7th international conference on data science and advanced analytics (DSAA), pp. 747-748. IEFF, 2020.
14

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Mingehuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Muhammad Ali Syakur, B Khusnul Khotimah, EMS Rochman, and Budi Dwi Satoto. Integration k-means clustering method and elbow method for identification of the best customer profile cluster. In IOP conference series: materials science and engineering, volume 336, pp. 012017. IOP Publishing. 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instructioo-following llama model, 2023.
Joannes Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical evaluation. In European conference on machine learning, pp. 437-448. Springer, 2005.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen. Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, et al. Step-on-feet tuning: Scaling self-alignment of Ilms via bootstrapping. arXiv preprint arXiv:2402.0766/1 072007600,, 2024a.
Jiahao Wang, Bolin Zhang, Qianlong Du Jiajun Zhang, and Dianhui Chu. A survey on data selection for Ilm instruction tuning. arXiv preprint arXiv:2402.0512. 2024b.
Yidong Wang. Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint arXiv:2306.0500 2023a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- eeff-generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. Advances in Neural Information Processing Systems, 36:74764-74786. 2023b.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint urXiv:2402.09739 2024.
John Wieting and Kevin Gimpel. Paranmt-50m: Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations. arXiv preprint arXiv: 2017.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In Forty-firt International Conference on Machine Learning, 2024.
Sang Michael Xie. Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 122424444....4, 2023.
15

---

Published as a conference paper at ICLR 2026
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. arXiv preprint arXiv:2406.0 08464, 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self celfiimprovement arXiv preprint TrXiv:2409.12122 2024a.
Zhaorui Yang. Tianyu Pang, Haozhe Feng, Han Wang. Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. arXiv preprint arXiv:2402.1 13669, 2024b.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.08826, 2021.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint PrXiv:2339.12266 2023.
Yue Yu, Yuchen Zhuang, Jieyu Zhang. Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024a.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint arXiv:2406.066 0664666666 2024b.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:24005.m 03358, 2024.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10792, 2023.
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmod record, 25(2):103-114, 1996.
Xuanyu Zhang and Qing Yang. Self-qa: Unsupervised knowledge guided language model alignment. arXiv preprint 2305 /952, 2023.
Yifan Zhang and Team Math-AL. American invitational mathematics examination (aime) 2024, 2024.
Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025.
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Automathtext: Autonomous data selection with language models for mathematical texts. arXiv preprint 202.02240...76255, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu. Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In the preparation of this work, we used LLM as auxiliary tools in limited capacity. Specifically, LLMs assisted in drafting portions of the code and in refining the wording of certain sentences for clarity and readability. All technical content, including the design algorithms, experimental methodology, analysis, and interpretations, of LLMs was confined to language refinement and coding suggestions, and did not influence the scientific contributions or results reported in this paper.
independently developed by the authors. The use
B LIMITATIONS
Although EQUAL incorporates filtering mechanisms to mitigate some biases, the underlying web corpus may still contain culturally skewed content, which can subtly or unpredictably influence the model's behavior. Specifically, given the vast scale of web-crawled data, it is inevitable that some documents may contain content that is inconsistent with human values (c.g., promoting extremist ideologies). Since the focus of our research lies in data selection for targeted capabilities improvement, rather than to detect or filter data with inappropriate or biased content, we did not include strict mechanisms identify remove such content, as this falls beyond the scope of our study. Nevertheless, we fully acknowledge the importance of ethical data usage. We therefore recommend that practitioners applying our method first ensure that their candidate data pool has been screened for harmful or culturally biased content (e.g., using bias-detection models such as bert-hateXplain) order prevent potential misuse of our approach.
C BROADER IMPACT
EQUAL mitigates the bias introduced by over-reliance synthetic domain- data by sourcing instruction tuning data from broad and diverse web corpus. Its scalable and cost-efficient extraction framework democratizes access to high-quality instruction tuning, enabling researchers with limited resources to fine-tune large language models effectively. Moreover, the enhanced generalization of models trained with EQUAL contributes improved performance in socially impactful domains such as education, healthcare, and public services.
D ABLATION STUDY OF CLUSTERING NUMBERS AND ALGGORITHM
[Metric and Criteria.] We use the metric Within-Cluster Sum of Squares (WCSS) to select the best cluster number using the well-known Elbow (Syakur et al. 2018 algorithm. WCSS is the sum of squared distances between each data instance and its cluster center, i.e., WCSS= Exec, || pill- At high level, the criteria should be that within each cluster, data instances are close each other, based on which it is better for different cluster centers to be far away from each other. Based on the criteria, the Elbow algorithm leverages the WCSS as measurement to iteratively select an appropriate cluster number, as follows.
[Specificc hypermarameter selection strategy.] To be specific, Elbow begins with small k, and with increasing, WCSS first decreases rapidly and then slows down. Then, we identify the "elbow point" where the decreasing rate becomes slow as the best k. Thus, within each cluster, data points are sufficiently close to one another. Furthermore, given that & remains modest, different cluster centers tend to maintain a distance from each other.
Clustering algorithms, In terms of the clustering algorithms, we also added experiments to show that EQUAL, is not sensitive to clustering algorithms mainly because different algorithms have their own strategies to select appropriate parameters, which follows the criteria mentioned above.
Specifically, we evaluate the performance of several typical clustering methods including BIRCH (Zhang et al. 1996) and DBSCAN (Ester et al. 1996). Considering that the clustering results are easily affected by the parameters of clustering algorithms, we use different methods to select proper parameters. For DBSCAN, there are 2 key parameters: (1) eps(the radius of neighborhood w.r.t some data points) and (2) minPts (a data point is considered as core point at least min Pts data points within eps of it). They can be using the method in (Schubert et al.
17

---

Published as a conference paper at ICLR 2026
2017). For BIRCH (Zhang et al. 1996).
the Elbow (Syakur al. 2018) algorithm
Sihouette score (Shahapure & Nicholas 2020) to determine the appropriate number of components.
E RESTATING THE IMPORTANCE OF OUR PROBLEM SETTING
In this paper, we focus instruction tuning, also known supervised fine-tuning (SFT). This phase differs fundamentally from pre-training, whose objective to build an unaligned foundation model.
In contrast, instruction tuning aims to adapt base LLM to specific domains user needs, thereby
enabling better performance on specialized downstream tasks (Wang et al. 2024b al. 2023).
2023a
Zhang
Consequently, many studies on data selection for SFT (Li ct al. 2023b Xia et al. 2024 et al. 2024) emphasize selecting Nomain-relevas data from candidate pool with reference to downstream task. This is because fine-tuning LLMs without careful data selection-for instance, using QA pairs extracted indiscriminately from all documents in the candidate set-can impede the model's ability to acquire specialized capabilities. Empirical results from prior works (Kim et al. 20233 Muennighoff 2022 Wang al. 2023b) demonstrate that LLMs fine-tuned on targeted subsets of data outperform those trained on the full SFT dataset, underscoring the importance of relevance-aware data selection.
F FLOPS CALCULATION
FL.PPs is the number floating point operations performed by GPUs. Many state-of-the-art methods [1.2.3] use it measure the consumption of GPU computing resources. In our experiments, FLOPs is collected directly in the data selection process using the Python code: import torch
import torch. .n as nn from torch. profiler import profile, ProfllerActivity
model nn Linnerrr input_date data torch (128, 1024) cuda with
lel (ProfflerActiviity.coomm
profile (activities
trofilerAetivity.cuu.. with_ ftth_flops=True as prof: model (input_data)
print
() table (sort_by="flops" row_limit=10)
G LENGTH OF SELECTED DATA
Table 7: The average length of extracted QA pairs with different methods (i.e., Influence, Perplexity and EQUAL (Curs))
Length
Random
Influeuce
Perplexity
EQUAL(Ours)
Prompt
48.99
37.86
49.58 1235,
58.38
Output
470.05
222.94
438.69
Total 519.04
260.80
1285.48
497.07
For synthesized QA pair data, recent research (Wang et al.) 2022 has shown that having diverse lengths is more beneficial. Short synthesized SFT data may degrade performance due lacking sufficient context (Taori et al. (002)), while excessively long data can introduce irrelevant content, diminishing the learning signal (Arora et al. 2016).
H DETAILED EXPLANATION FOR OPTIMAL TRANSPORT
Optimal transport (OT) is a widely used metric to measure the distribution similarity semantic vectors in NL.P problems (Wang al. 2022 Chen et al. 2019), which is particularly effective when
18

---

Published as a conference paper at ICLR 2026
the support regions of different distributions have relatively large deviations. In scenario, the support regions(i.e., possible values the semantic vector) of extracted QA pairs and the target application often diverge due to varying data sources. Thus, traditional divergences, such as KL divergence, are unsuitable because they tends to approach infinity this situation (Bhardwaj et al. 2021].
Also, in EQUAL, OT computation incurs far less cost than QA pair extraction, which dominates the expense of the entire pipeline. This mainly because extracting QA pairs requires running multiple LLM inferences (e.g., with Qwen2.5-72B) for each document.
Step
FLOPs
Clustering
2.31
Extracting 5% QA pairs for warm-up
6.94
Fine-tuning the contrastive learning model OT calculation
0.41
0.06
Extracting 5% QA pairs
6.94
Fine-tuning the LLM on 5% QA pairs
1.89
Total
18.55
Besides, although computing the OT incurs overhead, it could be negligible compared to the substantial cost savings from avoiding millions of LLM calls. Our experiments (Table 1, Section 4.2) show that EQUAL outperforms baselines while reducing QA extraction documents by 5 to 10 times.
I ENLARGE THE MODEL SIZE
In this section, we enlarged the model sizes from 7B/8B to 20B. As shown in the Table 8 we observe that EQUAL still performs better than other baselines on accuracy because select high-quality data considering the distribution similarity as whole. Moreover, EQUAL L. has good scalability compared with other baselines because we use the MAB framework to quickly identify the data instances that are beneficial to the downstream tasks.
Table 8: Enlarge model size and more downstream tasks
Method
GSMSK
MATH FLOPs HUMANEVAL MBPP FLOPs MMLU
BBH
FLOPs
Base
76.16
25.56
49.6
63.0
67.1
70.3
Random
77.33
27.61
11.23
59.7
63.8
9.67
68.7
71.3
17.65
Avg-sim
77.97
28.33
121.25
61.4
63.0
67.73
67.6
70.4
231.02
Perplexity
76.65
31.16
156.70
62.5
63.8
73.25
68.3
72.0
302.51
Influence
76.38
29.67
511.96
62.2
64.6
373.51
70.3
73.8
976.96
Rewriting
77.51
30.55
21.38
64.7
16.96
69.7
73.3
28.10
Perp.-mab
77.76
31.90
21.35
63.2
63.7
17.01
69.1
72.2
28.23
Infl--mab 76.70
30.73
32.76
64.7
64.1
28.46
70.6
73.7
46.53
EQUAL
80.38
33.78
21.03
67.3
66.7
16.65
73.1
76.3
27.46
J STATISTICAL SIGNIFICANCE
In this section, we show the statistical significance of our EQUAL in Table
K IMPACT AND OPTIMIZATION STRATEGIES OF EMBEDDING MODELS ON QA ALIGNMENT QUALITY
In EQUAL, the quality of alignment between documents and QA pairs could be influenced by the embedding model used. Currently, there have been some pretrained embedding models (e.g.. jinaal/jnaeembbeddinn-y
OpenAl embeddings and large-en-vil 115)) that demonstrate strong performance across multiple domains. Recent studies show that even in specialized fields such as mathematics and programming, the model's 's ability capture
BAAA/bge-large--n--
BAAI/bge-
19

---

Published as a conference paper at ICLR 2026
Table Statistical Significance.
Methods
Exp-1
Exp-2
Exp-3
Avg std
Random
68.76
68.83
69.17
68.92 (0.22)
Avg-sim
70.08
70.24
70.73
70.35 (0.34)
Perplexity
63.98
64.97
64.61
64.52 (0.50)
Influence
65.33
65.25
65.02
65.20 (0.16)
Rewriting
64.38
64.56
64.49
64.47 (0.09)
Perplexit--MAB
65.19
65.35
65.30
65.28 (0.08)
Influence-MAB
68.49
67.68
67.17
67.78 (0,67)
EQUAL (ours)
72.51
72.70
73.78
73.01 (0.69)
general semantics is effective enough to obtain good performance. Furthermore, in EQUAL, to improve the embedding model in specific domains, we fine-tune the embeddings on domain-specific QA pairs through the contrastive learning component. This lightweight fine-tuning can significantly enhance representation quality without introducing substantial computational overhead. Although embedding computation does incur some cost, this step is performed offline in our pipeline, allowing the computational burden to be amortized across the entire dataset. In our experiments, this cost is well justified by the downstream performance gains achieved through improved clustering. Overall, while the embedding model plays a critical role, we find its practical impact on clustering quality to be manageable, and the resulting improvements in QA alignment more than justify the associated computational expense.
L SAMPLING STRATEGY
In this section, we analyze the impact of different document sampling strategies on downstream performance. In the main text, we primarily adopted Random Sampling as the default strategy for extracting documents from the selected cluster. To thoroughly evaluate the robustness of EQUAL, have conducted additional experiments to evaluate the following three more sophisticated sampling strategies:
Within this strategy, the selected cluster is further partitioned into multiple groups based their embeddings. Then, fixed number of samples drawn from each group, which enables the model to learn from a more comprehensive distribution.
Stratified Sampling (Neyman 1992):
Density Sampling (Palmer & Faloutsos) 2000): We prioritize documents located in high-density regions of the cluster, which is achieved by calculating the local density of each document (e.g.. the inverse of k-nearest neighbor distance), with the goal of obtaining the most "typical" documents within the cluster.
Within the selected cluster, we use Maximum Marginal Relevance (MMR) to prioritize documents with the greatest dissimilarity to each other. This strategy aims maximize the internal diversity of single cluster.
Diversity Sampling (Carbonell & Goldstein 1998):
Methods
GSM8K Accuracy HUMANEVAL Accuracy
Random
68.7%
75.2%
Stratified
68.9%
75.5%
Density 69.4%
75.7%
Diversity 69.1%
75.3%
The experimental results show that Stratifies Sampling, Density Sampling and Diversity Sampling achieve slightly higher downstream performance compared to Random Sampling. This suggests that selecting identified by the MAB framework, can marginally improve the efficacy of the extracted data.
representative central documents within high-quality cluster, as
However, the performance gain is relatively modest. We attribute this to the inherent nature of our clustering process. After our contrastive learning warm-up step, documents within the same cluster already highly similar in terms of their potential generate QA pairs aligned with the target
20

---

Published as a conference paper at ICLR 2026
distribution. The clustering step effectively groups documents with homogeneous characteristics, meaning that the variance of QA pair quality within a single cluster is relatively low. This finding reinforces the importance of our tto-stage approach: first, using clustering to create homogeneous groups, and second, using MAB to perform high-level, cluster-wise exploration-explotation trade- off.
M OPEN-ENDED TASKS
To demonstrate the generalizability of EQUAL, we conducted an additional experiment on multiple open-ended tasks.
Specifically, we first sampled 500,000 web documents from the KnowledgePile dataset (a well used dataset for general knowledge), forming heterogeneous and broad candidate data pool De. Then, conduct our experiments on Mistral-7B. To better simulate a multi-task instruction tuning scenario, construct mixed reference set D, for EQUAL'S 's selection, combining several publicly available instruction tuning datasets:
Summarization: Summarization task instructions selected from AlpacaFarm Dubois et al. 2023).
Rewriting: Text rewriting and style transfer task instructions selected from Unnatural Instructions (Honovich et al. 2022).
Brainstorming: Brainstorming and ideation task instructions selected from the "brainstorm- ing" and "ideation" categories of Flan (Longpre al. 2023).
For evaluation, we use A plpacaEval (Dubois et al. 2024), a GPT-4-based benchmark for assessing model's performance open-ended instruction following tasks (e.g., summarization, rewriting, creative writing) and the primary evaluation metric the model's win rate (Dubois et al., 2024)
against the baseline (i.e., the original Mistral-7B base model), as judged by GPT-4 based on response quality.
Methods/Win rate(%)
Summarization
Rewriting
Brainstorming
Average
Random
70.9
70.9
54.7
65.5
Avg-sim
73.5
71.2
56.3
67.0
Ferplexity
75.1
73.3
57.6
68.7
Influence
74.0
73.5
57.4
68.3
LLM-scoring
76.2
74.0
58.3
69.5
Rewriting
75.5
74.2
57.9
69.2
Perplexity-MAB
73.5
73.1
56.5
67.7
Influence-MRB
72.0
71.8
55.7
66.5
EQUAL
78.1
77.1
60.5
71.9
The results clearly indicate that the EQUALL method is particularly effective for open-ended tasks and substantially improves the target model's capability to manage multiple tasks concurrently. Specifi- cally, the model fine-tuned with EQUAL achieved an average win rate of 71.9% on AlpacaEva significantly outperforming the random selection baseline (+6.4%) and the Perplexity-MAD baseline (+4.2%). This provides strong evidence that EQUAL can effectively identify the most valuable content from a large document pool to simultaneously enhance multiple open- open-ended task capabilities.
Specifically, in these experiments, we focus on open-ended and brainstorming, which lack clearly defined ground-truth answers. To address the challenge of evaluating such tasks, we follow the approach proposed in AlpacaEval, which leverages powerful large language models (LLMs) to assess generated responses without relying on reference outputs. Specifically, both our fine-tuned model and a baseline model (i.e., the original base model used in our experiments) are prompted with the same open-ended questions. A high-performing LLM (e.g., GPT-4) is then used to compare their responses and determine which one is better. Following AlpacaEval, we report the win rate of our model over the baseline across all open-ended questions as our primary accuracy metric. Importantly, the default evaluation dataset AlpacaEval does not cover the tasks of summarization, rewriting, brainstorming. Therefore, used (See al. 2017
as summarization, rewriting.
21

---

Published as a conference paper at ICLR 2026
Wieting & Gimpel 2017 Lin al. 2019) the evaluation datasets for these three tasks, respectively, in our experiments.
N DIVERSITY ANALYSIS
In this section, we directly assess diversity further validate the effectiveness of EQUAL. Specifically, we measure the lexical diversity and semantic diversity among the extracted QA pairs.
Lexical Diversity: We compute Type-Tole Token Ratio (TTR) and Measure Textual Lexical Diversity (MTLD) over the QA pairs extracted by each method. As shown in the table below, EQUAL achieves the highest TTR and MTLD scores, indicating richer vocabulary usage the extracted QA pairs.
Methods
TTR
MTLD
Random
55%
62.0%
Perplexity
46%
53.3%
Influence
42%
49.1%
EQUAL
52%
61.7%
2. Semantic Diversity: We compute the average pairwise semantic similarity among embeddings (using BAAI/bo BAAI/bge-en-v1. 5) within the final extracted dataset. Lower average similarity indicates higher semantic diversity. The results in the table below indicate that the QA pairs extracted by EQUAL exhibit greater semantic diversity.
Methods
Cosine Similarity
Random
50%
Perplexity 56%
Influence 59%
EQUAL
51%
These results demonstrate that EQUAL constructs more lexically and semantically diverse instruction dataset.
O COMPARE WITH OTHER DATASETS
To directly evaluate the quality of data constructed by EQUAL, we compare it against four leading instruction-tunie tunnng datasets Mathlnstruct[1], MetaMathQA[21, XwinMath[3), and OpenMathlnstruut((4 using full fine-tuning on LLaMA-3-88 under identical settings. As shown in the table below, EQUAL achieves the highest accuracy on both the GSM8K and MATH benchmarks, while maintaining comparable FLOPs to other methods, demonstrating its effectiveness. Specifically, all the four baselines use LLM rewrite and augment the QA seed data, but limited data diversity leads to their inferior performance compared to EQUAL.
[1] Mammoth: Building math generalist models through hybrid instruction tuning
[2] Metamath: Bootstrap your own mathematical questions for large language models
[3] Common 7b language models already possess strong math capabilities
[4] Openmathinstruct-l:: A 1.8 million math instruction tuning dataset
Methods
GSM8K
MATH
FLOPs
MathInstruee
67.30%
31.33%
18.25
MetaMathOA
69.23%
33.02%
17.76
XwinMath
69.72%
33.79%
17.97
OpenMathInst
70.03%
33.53%
18.31
EQUAL
73.01%
35.10%
18.55
22

---

Published as a conference paper at ICLR 2026
P REFERENCE DATA SIZE
Intuitively, more reference data leads more accurate estimation of the target distribution. However, in this section, would like to clarify that as long the reference data roughly capture the distribution, EQUAL can perform effectively practice. This is consistent with the results reported in other papers this issue (Xia al. 2024: Li al. 2023b] Wang et al. (202b)) all perform well using only dozens of validation examples. To further validate this, we conducted an additional experiment on the math task using full fine-tuning of LLaMA-3-8B by varying the size of the reference set. The results are summarized in the table below:
# Reference data Accuracy
20
50
100
500
1000
1500
73.01% %
73.08%
73.15%
73.21%
73.27%
73.33%
As shown the table, the performance of EQUAL improves consistently with the increase in the number of examples in the reference set. Note that EQUAL achieves good results with just 20 reference examples, reaching an accuracy of 73.01 %-neerly matching the 73.33% obtained with 1500 examples. Furthermore, we also extended our analysis more complex and diverse benchmarks such as MATH. The results show that, for more challenging dataset, the OT-bbaeed distance estimation indeed requires 150 reference samples to reach stable and effective performance. In practice, such reference sets are often readily available (e.g., training set in benchmark tasks), making our method broadly applicable.
#- Reeference data Accuracy
20
50
75
150
500
1000
1500
34.37%
35.10%
35.76%
36.1
11%
36.19%
36.23%
36.40%
Q PERFORMANCE ON MORE DIFFICULT TASKS
In this section, we evaluate the effectiveness of EQUAL on the more challenging math benchmarks AI AMM2202 and OL YMP ABENCHMATH, as well as the code benchmarks HUMANEVAL+ and LCB. The results are presented below:
Methods
AIME2024
OLYMPIABENCHMATHH HUMANEVAL
LCB
8.3% L a.a 1amma-3--- B- Ll ama 3 ---8-------------- 11-88BIInee ruct
1.1%
3.7%
31.1%
3.6%
14.4%
60.4%
9.7%
EQUAL
10.1%
17.3%
61.8%
10.4%
R BASELINE IMPLEMENTATION DETAILS
In our paper, the perplexity scores computed using off-the-shelf language model (i.e., target model) that has not been fine-tuned on the reference set. Here, we include perplexity scores computed on model fine-tuned with the reference we introduced an improved baseline called perplexity-ee the perplexity model on the reference set D,, enabling it to better capture the specific domain characteristics of the downstream task. We then use this finetuned model to compute perplexity scores for QA pairs extracted from the candidate pool to further finetune model.
an informative additional baselines. Specifically,
In this approach, we first finetune
Experimental results show that perplexity--e method. This is because the finetuned model is more aligned with the downstream task, mak- ing the perplexity scores more indicative of the relevance of the data. However, does still not perform better than our proposed BQUAL, as it treats data points independently and fails to capture the underlying relationships between them, which in turn leads to reduced data diversity.
outperforms the standard perp 1 ex i
Besides, our experiments, we used the same method as LESS (Xi et al. 2024) to calculate the influence scores our EQUAL method. The training set consists of QA pairs extracted from all documents, from which select the top 5% with the highest influence function scores calculated against the reference
target model. The reference set and test set are consistent with that used in
23

---

Published as a conference paper at ICLR 2026
Methods
GSMSK 64.52%
MATH
HUMANEVAL
MBPP
Perplexity
33.56% 34.46% 35.10%
44.5%
50.5%
Perplexity-xef
69.10%
47.6%
54.6%
EQUAL
73.01 %
49.4%
56.3%
EXPERIMENTS ON QWEN MODEL
We conducted additional experiments using the Qwen2.5-7B model. The results in the table below
show that our proposed method consistently improves performance on Qwen2.5-7B. demonstrating its strong generalization across different model architectures.
Methods
GSM8K
MATH
FLOPs
HUMANEVAL
MBPP
FLOPs
Random
86. %
59.6%
8.51
66.7%
75.2%
6.73
Avg-sim
87.6%
65.2%
111.73
68.7%
75.9%
64.15
Perplexity
86.1% %
66.7%
146.52
69.1%
76.1%
72.22
Influence
85.6%
61.0%
251.28
66.9%
75.3%
131.65
LLM-scoring
86.9%
67.1%
17.61
69.6%
76.4%
137.96
Rewriting 87.1%
61.3%
19.06
67.2%
75.2%
14.56
Perplexity-MAB
86.4%
67.0%
18.23
69.7%
76.6%
14.19
Influence-MAR 85.9%
61.6%
18.19
67.8%
75.5%
14.72
EQUAL
89.6%
71.3%
17.76
73.3%
78.0%
13.30
T COT DATA GERERATION.
In this section, we further consider an experimental setting where, for each document selected from the candidate pool, extract its QA pair and distill corresponding chain-of-thouge (CoT) for that pair using stronger teacher model. In this CoT distillation scenario, we apply our EQUAL and directly compare it against OponThought and slk as follows:
1) OpenThoughts: We first extract QA pairs from all candidate documents and employ QwQ-32B to generate chain-or Shain-of--though answers for each question, and then use the resulting data train our target model.
2) s1:: We first extract QA pairs from all candidate documents and use QwQ-32B to generate chain-of-thought answers for them. Subsequently, we follow the s1 K methodology to select a small subset of QA pairs by filtering for quality, difficulty, and diversity.
3) EQUAL: We employ mumt bandit strategy to iteratively identify clusters that are most likely to yield high-quality CoT data points. We then restrict expensive QwQ-32B calls to the documents in these high-value clusters only to extract QA pairs and generate chain-of-thougee answers for them.
Methods
AIME24
AIME25
LiveCodeBench CodeBlo
FLOPs
Random 33.1% 1%
26.0%
23.2%
7.29%
50.65
slk 40.7%
32.7%
26.7%
9.30%
197.87
OpenThoughts 40.1%
33.1%
26.1%
9.55%
238.20
EQUAL
42.7%
34.2%
27.6%
10.05%
65.17
As shown in the table above, aside from the heuristic method Random, EQUAL attains the best perfor- mance with the lowest computational cost (i.e., FLOPs), primarily because it performs document-level selection before QA extraction and CoT generation. Specifically, instead of extracting QA pairs from all documents, EQUAL treats document clusters as arms multi-armed bandit and iteratively selects the most promising for QA extraction and CoT generation. This exploration-exploittttio exploitation scheme focuses extraction on clusters that yield larger gains on targeted capabilities, cutting LLM extraction cost while preserving even improving final task performance.
24

---

Published as a conference paper at ICLR 2026
U EXTRACTOR MODEL.
In this part, we evaluate the per- formance of EQUAL using extrac- tor models of varying sizes, as well with closed-source extractor mod- els. Specifically. instead of the Qwen2.5-72B model used in our pa- per, we performed extraction with the weaker-extraction Qwen2.5-7B and the closed-souree Qwen-Flash mod- els. As shown in Figure 4 replac- ing the Qwen2.5-72B extractor with the Qwen2.5-765 model or the closed- source Qwen-Flash model leads to small shifts in absolute performance (a decrease for Qwen2.5-7B and an increase for Qwen-Flash), attributable to differing linguistic and reasoning abilities, the relative improvement trend introduced by FQUAL remains consistent. This indicates that the key strength of our method lies its ability to mine high-quality QA pairs from large-scale web data, rather than in the ability of the extractor itself.
Sandom (EGOR00
(GSMMDI IMATIH
Random (MATU
UNAAT
Accuracy
50
30
Owenz.5-78
Figure 4: Ablation study of different extractor model.
V KEY DIFFERENCE BETWEEN EQUAL AND EXISTING METHODS
In Existing methods, when LLM is prompted with only few seed examples, tends to produce instructions that are locally similar to those seeds (i.e., few-shot examples), which limits diversity. In contrast, all QA pairs generated by EQUAL are extracted from large, heterogeneous web corpus (AutoMathText, KnowledgePile, StackOverflow) that contains rich knowledge beyond the seeds. Thus, whereas existing methods literally "stay close" to the seed examples in input space, EQUAL selects real, naturally occurring QA pairs from an independent corpus and uses the seeds solely as guidance signal.
Furthermore, EQUAL prevents the generated QA pairs from collapsing onto few high-OT clusters by modeling document selection as multi-armed bandit (MAB) strategy with an upper-confidence- bound (UCB) style exploration term. Clusters that have been sampled less often receive larger bonus and are thus more likely to selected. This explicitly promotes under-sampled clusters, encouraging us to explore new regions of the document space instead of repeatedly pulling only the highest-OT cluster, which further increases the diversity the generated QA pairs.
Together, our method avoids generating QA pairs that near-duplicates of the original seed examples by merely rewriting a small set of seed examples. Instead, we selecting naturally generated QA pairs from large, heterogeneous corpora whose distribution matches the target tasks.
W PROMPTS FOR QA PAIR EXTRACTION
In this section, we present the prompts used for extraction tasks across different domains.
W.1 CODING TASK
Code
SYSTEM:
You given set of pre-prreeeee pre-processed documents, each of which may contain natural question-anse wnswerr (Q-A) pairs. Your task is to identify and extract these pairs while ignoring unrelated content such as ads, markup, boilerplate text.
Input:
Each document contains multiple sections of text. Some of these sections may have clear questions followed by answers, while others may be irrelevant (e.g., ads or noise).
Output:
25

---

Published as a conference paper at ICLR 2026
Extract the Q-A pairs found within each document. A valid Q-A pair must consist of clearly defined question and its corresponding answer. If no natural Q-A pair exists in the document, return void for that document. In the document, in order to describe the problem more clearly, the questioner usually attaches some useful information (e.g., code or explaination) to make it easier for others to better understand the problem. You need extract this part of the content that needs be complete as well.
Here are some examples:
# Example 1
Content:
Sorting lines date-wise and time-wist wise using Python from .xxt file. I have just written a Python code to extract data from around 700 text files into one file called out out_data.. txt. The contents of the out_data txt file look something like this:
datetime V_1, 3 V
2013-03-17 18:00:088.372.000 ,884 776 009 .6533, 26553,ffe ffff .987
2013-03-17 0.2::03.22.8866.60
,008 5443 ,879 ,443
2013-05-17
,344,123 23:09:08.171, (98.667788877977000000 (89966 .9887 .098 .0987 2013 35:06:04. 172,264,.7222,.... (9889798776.0 1287 .3998,88888 29877
2013-08-17
There are a total of 5,783,374 lines in the out_data. txt file, and each line (after the header) begins with the datetime value.
However, the problem have that the code wrote extracts the data from each individual file and adds it my out_ pat__dtta.. txt file, but the lines not in the order of date-time as you above. was hoping to get my lines to be in date-time order because need to plot this data. Any help will be highly appreciated!
Here is my current code:
import
Regular expressions
import glob $ File management and reading
name all all_header
main ": Opening for Python
[] $ List declaration
all values [] # List declaration
1 - 0
with open (' l'out_datea netta.xt',, 'w') as of: # Output file
for infile in glob. glob /Users/name wttsar//name/essttop//ooooo Input file
with open (infile)) as febj:
print (E"Processing file (infile)")
for line in fobj:
data = line.
# Split each line into individual tokens
if len (data) 2 and re. re. search (r' (Nd+-\d+-\dH)
da]a Regular expression to identify date and time header ['datetime']
Column name datetime data[1]] # datertime
values [data[0]
as one value
else:
header [d for d in data if data. index (d) $ 2
3 0]
values [d for d in data if data. index (d) $ 2 !=
0]
all iieaderreettee (header)
exl_ (values)
extend
if not header:
if == 0:
of of.write w.joinoonnn (all_heaa)))
i += of of.writ of.write (el.oooaaalllllllaaeeaa
26

---

Published as a conference paper at ICLR 2026
all heacer all_val val ues
of ofwrrite ("An") of.write "',nn (all_values))
EXPECTED RESULT
The expected result from the example data given above would be:
datetime V_1, V_2, V_3 V_4, V_5, 6_ V_7
2013-01-17 35:06:04.172. 267 .977 6687 .9876, 1287 1287,3498, ,33498 ,2987
2013-03-17 18:01:48.3722
,884,776
,6553, ffff .987
2013-03-17 18:02:03.828 08,.08888876,,,8888 ,88,0088 .5423 ,473,443
2013-05-1
20:13:52.483 ,545522288884,,,,00 $944,987.
,344,12 123
23:09:08. 171:0008.11,,67
,09876, ,0987, ,098 ,0987
However, could not figure out how to include the element in the code if there is any other way to achieve this.
SOLUTION USING PANDAS
You can use pandas. A simple example would as follows: import pandas as pd
import glob
df_list []
for infile in glob.globl
httpettop/ra_ddaa>...... xxx"/)
df appen read_esv (infile, harse_datese 'datet ime }) of pd. concat (df_list) sort_val imetetteemm df.to_csv ('ooott_aaaaatee tata. index-False)
SOLUTION USING csv
An alternative method is:
import
with open ("out_data.txt", "r") as f:
reader cav. reader delimiter=", ")
header next (reader) sortedlist corteddreader,, key-lambda x: x(0])
with open ("sorted. "w") f:
writer writer (f, Linetemminato=="la))
writer. writer.writerww//ee (header))
writer.wwiterows (sortedlist)
SOLUTION USING BASH
As alternative, you also Bash:
head -1 out data txt > sorted. txt tail +2 data txt I sort -t, -k1 sorted. txt
Hope this helps.
Q:
I've just written a Python code to extract data from around 700 text files into one file called out aut_d txt. The contents of the out_ txt file look something like this:
datetime V_1, V_3 V V_6.V_7
2013-03-17 18:01:48.33 0,288,372,,11 100 8884,88,,8 888,,78,6009 6553. ffff .987 5423 ,879 443 1344,22
2013-03-17 18:02:03.823 876..888.876622...8
2013-05-17
20:13:52.488.
27

---

Published as a conference paper at ICLR 2026
.09988778880009 ,09870987 2013-01-17 35:06:04.172 267 .987 .9877687778777 017876697.98700 1287 ,3498 ,2987
2013-08-17 23:09:08. 171 .667 .9887, ,9887,9897,7 017777999900777777 ,09876,0... .09876.0987 .0987 ,098 ,0987
There are total of 5.783,374 lines in the out_ dat..ataa.. txt file, and each line (after the header) begins with the datetime value.
However, the problem have is that the code wrote extracts the data from each individual file and adds it to my above. was hoping get my lines be in datet me order because I need plot this data. Any help will be highly appreciated.
txt file, but the lines are not in the order of datet me as you can see
A:
import re # regular expressions import glob # file management and reading
if name = main ": opening for Python
all header [] $ list declaration
values
list declaration
with open ( ('out_data.
'w') as of: # outout file
for infile in glob glob.glob ("/U//r/U/U/eeree haameDDesttop/aw wat_aaaaatttttttt
input files
with open (infile)
febj;
print ("processing file ()" format (infile)) for line fobj:
data = line..plit() # split each line into - individual tokens
if 1en (data) -- 2 and re. search (r' (1d+-\d+-\dH) ' identify date
data
column header l'datetime']
values [data[0] " data[ll]
combine
4 date and time
else:
header [d for d in data if data index (d) $ 2 =
0]
values [d for d in data if data index (d) $ 2 !=
0]
[i__header.extent (header)
all www.waa...eeeeeeeeeeeeeaaa (values)
if not header:
if == 0:
=
of.write ' joiin (all_header)
i += write ("\n")
of of.trite "oiitttt"".jjinnal_laaaaee
all heacer = []
all_val ues []
of write("An")
or jojn (all_values))
EXPECTED RESULT:
From the example data given above, the output should be:
datetime V_1, V 2, V_3, V_4
V_5
V_6 VV7
2013-01-17 35:06:04.172.250
.987
.6897
.2987
2013-03-17 18:01:48.372.10
.884
.776 ,009 .6553. ffff
.987
2013-03-17 18:02:03.828.87
.632
.887 ,008 -5423.879
.443
2013-05-17 20:13:52.488.
.987
.233
.112 .098 .344
.123
2013-08-17 23:09:088.111.67
-9887.9897
.0987
28

---

Published as a conference paper at ICLR 2026
USING PANDAS:
You can use the Pandas library for simplicity. Here is an example:
import pandas as pd
import glob
df_list
for infile in glal.giaac///serr//mm////tttttop http:///essttopeeaaaaaaaaaaa/aaa df list..t... apd, read esv (infile, parse_dates= dates=l "datetime } df pd. concat (df_list), sort_valuse (by-'dattes 'datetime') dE.to_csv ('out_ txt ', index-False)
USING CSV MODULE:
You can also perform an ordinary (dictionary order) sort as follows:
import csv
with open
"r") as f:
reader cav cav. reader (f, delimiter=", ") header next (reader)
sortedlist sorted key-lambda x(0))
with open ("sorted, txt", "w")
wav.writee writer (f, Lineterminator="ha" writer cav
(header) writerrwi (sortedlist)
USING BASHE:
Alternatively, you can use the following Bash commands:
head -1 out_ out_data.txx txt > sorted.txt
tail +2 out_data. txt sort -t, -k1 >> sorted.txt
Hope this helps!
W.2 MATHEMATICAL TASK
Math
System:
You are an excellent assistant who good at constructing question-answer (Q-A) pairs. Your task to construct some math Q-A from the original documents.
Input:
Each document contains multiple sections of text. Some of these sections may contain mathematical content which can be used to construct Q-A pairs.
Output:
Identify valid content and construct Q-A pairs. A valid Q-A pair must consist of a clearly defined question and its corresponding answer. Specially, the questions should be solvable that provide valid and complete pre-conditions, and the answers need to satisfy the Chain of Thought (CoT) format, which instructs the responder solve the question step by step. If the content in the document is not suitable for Q-A construction, return void for that document.
Here an example:
User: As I mentioned certain scientific terms in my previous post, I would like to go in-depth those concepts, beginning with terminal velocity, it being the most fundamental concept in my post. So what is terminal velocity?
Terminal velocity is the velocity of an object when the drag force (dependent on the fluid the object travelling through) acting upon it is equal to the downward force of gravity acting upon it. Simply put, when the air resistance of falling object cancels out the gravitational force which is pulling it downwards and accelerating it.
29

---

Published as a conference paper at ICLR 2026
So how do these forces affect the motion of the object? The forces cancelling each other out make the object remain at constant rate of motion.
You may ask why does the object still move when the forces cancel each other out. This is due the fact that in the beginning the force of gravity still manages to overcome the drag force, allowing the object to gain speed (accelerate) initially. But as the object increases in velocity, the drag force increases. This effect can also be seen in the case of friction (Drag and friction are pretty much the same thing). Let's assume that boy is dragging heavy box, full of files, across distance of 100 meters. Now, will imagine this scenario in two different ways: firstly, in the case whereby the boy is walking slowly, and the second, whereby the boy is running. So in the first case, the boy walks; when he reaches the end, he feels the bottom of the box, where the box and the floor meet, it still feels the same as before. Now the second case, he runs; he once again feels the bottom of the box, this time it feels warmer than before.
So what can we infer from this scenario? Before reveal the answer, would like state few properties of friction:
Friction opposes motion
Friction causes wear and tear
Friction produces heat when kinetic energy converted into thermal energy
So what can we infer? In the second scenario, there was more heat; therefore, we can assume that there was more frictional force produced in the second case.
Now let's go back to what mentioned previously, air resistance increases (Drag Force) as the object's velocity increases. As seen in the example above, Recap:
tell that this statement is true.
Terminal velocity the velocity an object is at when the gravitational force acting upon it is equal to the drag force acting upon it in the opposite direction, therefore cancelling out all forces and resulting a resultant force O.
The drag force acting upon the object increases the object accelerates due the downward force of gravity.
Ok, so let's move on to the math behind terminal velocity and some examples of it. The formula for terminal velocity as follows:
2mg
V = pAC
where:
Mass of falling object
Acceleration of the object due to gravity
p Density of fluid through which the object is travelling
Projected area the object
. C. Drag Coefficient
Example:
Assuming I drop a metal cube which has a mass kg and has a projected area m² on Earth 909 downward, through air temperature of 25 C, what would the terminal velocity of the cube be?
All have do is input all the values into the formula. The acceleration due gravity Earth is 9.81 m/s2. The density air at 25 is 1.1839 kg/m and the drag coefficient cube is 1.05 facing downward.
The result is:
6.881101581 m/s.
That's terminal velocity for you! I would like to thank Mr. Tan Ping Hock and Mr. Yao Zhi Wei Adrian, my current and previous physics teachers respectively, for clearing my doubts about certain concepts within this topic of terminal velocity!
Thanks for reading!
30


Reviews:
[{"review_id": "JnnzbDWuly", "reviewer": "Reviewer_kjNN", "summary": "This paper aims to explore a scalable and cost-effective approach for automatically extracting high-quality instruction data from large-scale document corpora. To this end, the authors propose the EQUAL framework, which integrates contrastive learning–based embedding alignment with a multi-armed bandit–driven adaptive sampling strategy. By iteratively sampling documents and updating distributional similarity based on optimal transport, EQUAL effectively identifies and extracts high-quality instruction data. Experimental results demonstrate that EQUAL significantly reduces computational costs and outperforms multiple strong baselines across various tasks.", "questions": "1. Why did the authors not conduct experiments on the Qwen model and instead choose the relatively outdated Mistral-7B model? Including experiments on a more advanced model could further strengthen the persuasiveness of the paper.\n1. How large is the performance gap between EQUAL and high-cost instruction data extraction methods, such as directly using closed-source model APIs? If this gap is small, the proposed method would represent a highly valuable contribution by maintaining strong performance while significantly reducing data extraction costs.\n1. In the limitations section, the authors briefly note that EQUAL “may still contain imbalanced or culturally skewed content.” However, it remains unclear what proportion of the final dataset consists of such biased content and how it may affect model performance. It is recommended that the authors provide a brief analysis  of these biases.\n1. In the anonymized code repository provided by the authors, the following statement appears: “This repository contains the code for our paper [EQUAL: Efficient Scalable Data Extraction Framework for Instruction Tuning] published at ICML 2025.” Even if the paper was not actually accepted by ICML 2025, such a statement could serve as a unique identifier and potentially reveal the authors’ identities. I am not sure whether this violates the double-blind review policy of ICLR 2026, and I would like the authors to clarify this.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a clear research motivation, and the proposed EQUAL method achieves a favorable trade-off between cost and performance, providing valuable insights for future work.\n2. The paper is well-organized, and the experiments in this paper cover three major datasets and multiple tasks, providing comprehensive evidence for the effectiveness of EQUAL. \n3. The authors conducted comprehensive ablation studies on the proposed framework and provided thorough analyses of the extracted data, further enhancing the persuasiveness of the proposed EQUAL framework.", "weaknesses": "1. In lines 144–148, the authors emphasize that the main goal of this paper is to reduce the number of documents to be extracted in order to lower data construction costs while maintaining high model performance. However, the paper does not clearly show the performance gap between the instruction data extracted under high-cost settings and that obtained using the proposed method. This could be an important issue, as a large gap would call into question the practical significance of the proposed cost optimization.\n2. The main experiments do not report the amount of data used, making it difficult to assess the specific impact of data scale on the experimental results. The superior performance of the proposed method might be partially attributed to training on a larger amount of data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ikFOPVF2fJ", "reviewer": "Reviewer_5USS", "summary": "This paper presents EQUAL, a new framework for efficiently extracting instruction-tuning data (QA pairs) from large web-scale corpora. The core idea is to avoid the \"extract-all\" approach, which is prohibitively expensive. Instead, EQUAL first uses a \"warm-up\" phase (sampling 5% of the data) to train a contrastive learning model that aligns document embeddings with the QA pairs they contain. It then clusters the entire corpus and uses a Multi-Armed Bandit strategy to iteratively sample from the most promising clusters. The \"reward\" for the MAB is based on the Optimal Transport distance to a small, trusted reference set. The authors show this method significantly cuts costs (5-10x) while also improving downstream model performance on math, code, and open-ended tasks.", "questions": "1. Can you run an experiment that proves this isn't just a \"bias amplifier\"? For instance, what happens if you use a D_r for the math task that only contains simple, 1-step arithmetic? Does EQUAL (as I suspect) fail to select clusters with complex, multi-step proofs?\n2. The 5% warm-up cost is a major barrier. How does the entire pipeline's performance (end-to-end) degrade if you only use a 1% or 0.1% warm-up budget? What is the minimum compute needed for this to actually work?\n3. How much of this success is just an artifact of using a giant 72B model for extraction? What happens if I use a smaller, less-capable 7B model? Does EQUAL still find the best data, or does it just find the data that is easiest for the weak extractor to parse?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper applied an innovative application of the Multi-Armed Bandit framework to sample from the most promising clusters iteratively. \n2. The paper's key insight is interesting: don't cluster raw documents, cluster them by the data you can get out of them. \n3. The paper is easy to follow and authors have done quite a lot of experiments including comprehensive downstream benchmarks. My core issues with this paper are that its impressive results seem to rest on a few critical assumptions that don't hold up to scrutiny, making me question its real-world practicality.", "weaknesses": "My core issues with this paper are that its impressive results seem to rest on a few critical assumptions that don't hold up to scrutiny, making me question its real-world practicality.\n1. The entire framework is anchored to the Optimal Transport distance to a reference set, D_r. The paper's claim (in Appendix P) that using just 20 reference samples is nearly as effective as 1500 is not credible to me. Estimating a stable OT distance for high-dimensional embeddings from 20 samples is statistically dubious. More importantly, this setup assumes the reference set is a perfect, unbiased oracle. In reality, a small D_r is likely biased. The current framework doesn't just use this bias; it optimizes for it. \n2. The \"cold start\" cost is still prohibitive. The paper frames this as a low-cost solution, but the \"warm-up\" requires processing 5% of the corpus with a 72B-parameter model. For their 1.4M document dataset, that is 70,000 documents. This is not a \"warm-up\"; it's a massive, expensive inference job in itself. This barrier to entry is extremely high and is not adequately acknowledged.\n3. There might be a risk of cascading bias. Any bias in the reference set D_r gets learned into the contrastive learner, which in turn skews the entire clustering. Since the MAB only explores at the cluster level, it has no way to find high-quality, novel documents if they were already buried in a \"bad\" cluster. The method risks locking onto a local optimum very early in the process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTesLkWWn6", "reviewer": "Reviewer_wiHJ", "summary": "The paper focuses on efficient instruction-tuning data curation from unstructured text in the documents. Specifically, it argues that the existing methods generate many QA pairs from all the documents using LLMs which is computationally prohibitive. In addition, many of them suffer from low-quality data problems. To fix this, the paper proposes an iterative solution, posed as a multi-arm bandit problem, where they first cluster all documents in the embedding space and quantify their quality based on the optimal transport distance between the QA pairs extracted from them and reference datasets. Further, they compute the diversity score of each cluster so that the QA extraction is focused on exploration first and exploitation later. They perform several experiments to show the effectiveness of the method on llama-3.1-8b and mistral models on math and code tasks. Further, they perform several ablations to show the usefulness of diverse design choices.", "questions": "Mentioned above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The limitation pointed out by the work makes sense, and the proposed solution is tailored to fix them. In particular, the algorithm is reasonable which poses document selection and QA pair generation as a multi-bandit problem. \n\n2. The paper also proposes a contrastive learning approach to ensure that the features extracted from the documents and extracted QA pairs are similar in the representation space while they are far apart for original documents and negative QA pairs. \n\n3. The experimental results are quite extensive in terms of baselines and ablation studies. It definitely shows the general usefulness of the approach.", "weaknesses": "1. Line 46-48 argues that the synthetic data generation LLMs suffer from lack of diversity because of their proximity to the seed examples. The proposed method may also suffer from the same problem since the algorithm is optimized to reduce the optimal transport distance between the QA pairs from the documents and SEED datasets (MATH/GSM-8K or MBPP). \n\n2. The choice of models and evaluation datasets is somewhat old. For instance, it is quite common to show some quantitative experiments on Qwen models nowadays. In addition, there are more challenging datasets such as AIME24, 25 for math and LiveCodeBench or CodeElo for Code.\n\n3. There is no comparison against state-of-the-art synthetic data generation methods such as OpenThoughts, s1k etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)