Approximating Language Model Training Data from Weights
Authors: John Xavier Morris
Junjie Oscar Yin
Woojeong Kim
Vitaly Shmatikov
Alexander M Rush
Venue: COLM 2025 (2025)
Keywords: inversion
training data reconstruction
TL;DR: we recover suitable training data given only model weights

Abstract: Modern language models often have open weights but closed training data. We formalize the problem of data recovery from model weights and propose several baselines and metrics. We develop a gradient-based approach that selects the highest-matching data from a large public text corpus and show its effectiveness at recovering data given only weights of the original and finetuned models. The training subset pinpointed by our method in a large corpus can be used to train another model to comparable performance. Even when none of the true training data is available, data selected by our method from publicly available Web documents can be used to train a competent model.

Published as a conference paper at COLM 2025
Approximating Language Model Training Data from Weights
John X. Morris, Junjie Oscar Yin, Woojeong Kim, Vitaly Shmatikov, Alexander M. Rush Department of Computer Science
Cornell University
New York, NY
jxm3@cornel1. edu
Abstract
Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics. We develop gradient-based approach that selects the highest-matching data from a large public text corpus and show its effectiveness at recovering useful data given only weights of the original and finetuned models. Even when none of the true training data is known, our method able to locate a small subset of public Web documents can be used to train model to close to the original model performance given models trained for both classification and supervised- finetuning. On the AG News classification task, our method improves performance from 65% (using randomly selected data) to 80%, approaching the expert benchmark of 88%. When applied to model trained with SFT on MSMARCO web documents, our method reduces perplexity from 3.3 to 2.3, compared to an expert LLAMA model's perplexity of 2.0.
1 Introduction
Modern language models have been scaled up to 1 trillion parameters, which take 2 TB of disk storage store half-bit precision. Yet these models are often finetuned on thousands to millions of samples, the totality of which may take less than GB of disk space. Although models may lack capacity to memorize their entire pretraining data (Morris et al., 2025), they can easily memorize smaller finetuning datasets.
In addition, it is common for language models to be open-weights but not open-data: their creators release the weights of the language model that they can be run on local hardware, but do not release the datasets the models were trained on. For example, recent popular reasoning model DeepSeek-RI (DeepSeek-Al, 2025) trained from DeepSeek-Ban (Liu et al., 2024), both of which are open-weights; however, there has been much public speculation over what data was used to train R1 from Base. Similarly, the instruction-tuned and base versions of the LLAMA-3 models (Meta-AI, 2024) are open-weights, but the instruction- tuning data is unknown.
If the weights are public, they may reveal some information about the training data. Several works in computer vision have approached the problem of reconstructing training data from model weights (Balle et al., 2022; Haim et al., 2022), or distilling data using "trajectory" of multiple model checkpoints (Cazenavette et al., 2022; Guo al., 2024). However, all these approaches require backpropagating gradients from the network output to its input, which is not possible with discrete language data.
Instead of generating data from scratch, we consider whether adversary might select data points from a web-scale corpus to simulate the performance of the true finetuning data. This problem resembles conventional core-set selection, but is much harder: we have access to only a single final model and no validation data. These restrictions prevent us from using conventional approaches such gradient matching (Xia et al., 2024).
We consider the additional fact that many open-weights models are tne-tunes of base models. A modern adversary has access to two model checkpoints: the original ("base")

---

Published as a conference paper at COLM 2025
(1) Label unsupervized datasets
[2] Compute synthetic checkpoints
(3) Compute low dimension gradient
(4) Greedy data selection
Unkrown
Model
Synthwic Trapectory
F/ Dimessoon
oc Cimaroocc
Figure 1: SELECT Method Overview. We approximate training data from initial and final language model checkpoints. Given an unsupervised seed dataset, we first use the final model of to assign label to each instance. To compute gradients efficiently, we leverage random projections under the Johnson-Lindenstrauu lemma, which guarantees that gradient projection preserves inner-product information. Finally, we perform greedy data selection by picking instances with the highest running gradient sum.
and fine-tuned ("final") model. We develop a new method (SELECT) that relies only on the signal from these two models: gradients from the base model and pseudolabels from the final model.
Our algorithm greedily selects datapoints from seed set such that their sum points along the direction between the two models weight space. Gradient-based selection significantly outperforms all baselines across variety of settings, showing that we recovering essential information about the actual finetuning data from the weights of the fine-tuned model. In the case of both text classification and supervised finetuning, even when data selected by our method is completely disjoint from the true finetuning set, our selected data still of sufficiently high quality that (different) model trained on this data achieves high accuracy on the downstream task. On the AG News classification task, our method improves performance from 65% (using randomly selected data) to 80%, approaching the expert benchmark of 88%. When applied to model trained with SFT on MSMARCO web documents, our method reduces perplexity from 3.3 to 2.3, compared to an expert LLAMA model's s perplexity of 2.0.
To measure the extent to which data selected by our method is similar to the true finetun- ing data, we compute metrics based Optimal Transport (OT) of sentence embeddings between the true and selected datasets. scaling analysis shows that with a larger "seed" dataset, our method outperforms the baselines according to this OT-based metric.
To run our method at scale, we propose efficiency improvements that make our gradient- based approach tractable even when searching over set of millions of documents. Our method uses only the final-layer gradient of the model, which is relatively cheap to compute and known be recoverable from any model that provides open-source access (Morris et al., 2023; Finlayson et al., 2024; Carlini et al., 2024). We show our method is effective at recovering useful data from models trained for both classification and supervised fine- tuning.'
2 Related Work
Coreset selection. Coreset selection aims to select a subset of training data such that models achieve performance comparable to training the full dataset (John & Draper, 1975; Borsos et af., 2020). One line of methods develops utility functions to score and select data based on various features including n-grams, distances to k-means clusters, and model losses (Xie et al., 2023; Chen et al., 2012; Feng et al., 2022). Another line is optimization- based, with methods like Mirzasoleiman et al. (2020) selecting coresets whose gradients closely match those computed over the entire dataset. This approach has been extended by
'Our project is fully open-source. All code is available on Github.
2

---

Published as a conference paper at COLM 2025
Killamsetty et al. (2021b;a) to handle noisy and imbalanced datasets by performing gradient matching against held-out validation set.
In language modeling, gradient-based coreset selection has proven valuable, with Pruthi et al. (2020) using estimated gradient information to identify mislabeled data and Xia et al. (2024); Lee et al. (2023) applying gradient matching in supervised fine-tuning to improve data efficiency. In contrast, our work focuses on recovering finetuning data from sparse training checkpoints, operating under the constraint of having access only to two model checkpoints, without any validation data.
Dataset distillation. Data distillation synthesizes a small synthetic dataset rather than selecting from existing data. Wang et al. (2020) first proposed data distillation as bi-level optimization problem: the distilled dataset is treated as the inner optimization variable, while the model weights are treated as a function of this dataset in the outer optimization. Despite its theoretical elegance, this bi-level meta-learning framework incurs substantial computational costs. Follow-up works incorporate gradient matching as a supervised signal, aligning model's gradients trained on original and distilled datasets (Zhao et al., 2021; Lee et al., 2023). Further works have develop techniques like trajectory matching (Cazenavette et al., 2022; Cui et al., 2023; Yin et al., 2024), difficulty alignment (Guo et al., 2024), and kernel methods (Zhou et al., 2022). Our work extends trajectory matching methods and reformulates them for data selection in language modeling.
Training data reconstruction and extraction. A small number of works have attempted to directly recreate data from the weights of feed-forward neural networks (Haim et al., 2022; Buzaglo et al., 2023; Runkel et al., 2024) and convolutional neural networks (Balle et al., 2022). In the realm of language models, research has instead focused on the extraction of single training examples by direct generation from the models (Carlini et al., 2021; Nasr et al., 2023). Maekawa et al. (2024) and Nguyen et al. (2025) recently propose methods for generating synthetic text from model weights, but still require a real dataset for training, and match gradients of real data to synthetic data. Concurrent work (Huang et al., 2025) also infers some information from language model weights, focusing on the problem of property inference. Learning from the difference vector between base and finetuned model has also been recently explored by concurrent work (Lin al., 2025).
3
Approximating Training Data
3.1 Threat Model
We consider the increasingly-coommnn scenario of language models that are open-weights, but not open-dala. In particular, we assume access to not just the model weights but program to run the model as well as optimization details such as the optimization method used and number of training steps. Notably, we also assume access to an initial set of model parameters corresponding to the model state before finetuning, as well as knowledge of the optimizer used (e.g. SGD vs. Adam). This threat model corresponds many popular LLM releases such as LLAMA-3 (Meta-AI, 2024) and DeepSeek-RI (DeepSeek-Aí, 2025).
Given this small amount of prior knowledge, we seek to approximate the training data used for finetuning. Specifically, we aim to find data that is as effective for finetuning as the actual data used to create the released model.
3.2 Background
Formally, we assume access to some training algorithm T that solves an optimization problem:
0 DC = arg min Ex- [C(x,
Given loss function C and a dataset D, the training algorithm produces the set of parame- ters that minimizes the average loss over datapoints x ~ D.
3

---

Published as a conference paper at COLM 2025
3.3 Optimization Difficulties
To use any technique that requires gradients, we must compute the gradient of the model loss; compute the model loss, we need labeled data. This problematic because our threat model assumes access to 6g and L, but not D. Given the finetuned model parameters ff, we can express our problem as the search for dataset that, after training, produces model close the finetuned model:
D* = =arg min | TC,, D)I arg min |10 arg min Ex [x(x, 0]] D
D
8
Solving this optimization problem has number of difficulties. First, its nested nature requires training a model on any candidate dataset to obtain a candidate set of parameters that can be compared to the known f This bi-level optimization is typically expensive but shown to be tractable via brute-force in the dataset distillation literature (Cazenavette et al., 2022).
Second, in case of language models, the dataset D = [x1x)) y())), (x(1), y(t) consists of input samples x' E v that are discrete sequences of length: in vocabulary V. Unfortu- nately, computing the loss (xx, 0) requires non-differentiable lookup operation to convert token sequence to sequence of dense embedding vectors. This non-differentiaabiiiy means that typical dataset distillation approaches are no longer applicable, because we cannot backpropagate directly from L to D.
Method: SELECT
Unlike the differentiable vision case (Wang et al., 2020) we cannot generate data that directly matches the training trajectory of the final model in parameter space. We constrain our problem to data selection instead of data generation: given a large corpus text data, search for small set of datapoints that, after training, produce model close to the final model.
We hypothesize that the initial update in stochastic gradient descent, computed from the initial model 6g should be aligned with the direction leading toward the final model 8y. Solving this problem requires access to the set of initial model parameters, which we term 6-- We can express this goal search for data with gradient that maximizes its projection onto the model diff 0 0g:
arg max ( (0p-
Since we have access fo, we can compute example-level gradients for all x D. A naive solution to this problem might be to take the examples with the top similarity with the parameter difference. However, in practice, this yields highly redundant samples, as it neglects to account for batch-level interactions; when training with stochastic gradient descent, we typically take steps using gradients summed across multiple examples.
In light of this information, instead express our search for the set of points that produces a total gradient pointing the direction of the parameter difference:
Σ ve(x;00) (0, 0o) .EB
arg max BCD
Solving for B exactly requires enumerating all possible subsets of D and is generally intractable to solve in polynomial time. However, the batch search objective is submodular because it exhibits the diminishing returns property: the marginal gain of adding new datapoint decreases as the batch grows. The submodularity is known to have an efficient, close-to-optimal greedy solution (Nemhauser et al., 1978).

---

Published as a conference paper at COLM 2025
4.1 Creating synthetic checkpoints via linear interpolation
Our threat model only allows us access to a single final model checkpoint, as well as the base model initialization d-- State-of-the-art dataset distillation approaches (Cazenavette et al., 2022; Guo al., 2024) achieve more effective distillation with gradients that match trajectories of several final model checkpoints 0j,} E [I. This puts us at significant disadvantage because examples' gradients at the beginning of training may point in different direction later on during the optimization process. To make up for our lack of additional model checkpoints, create synthetic checkpoints by linearly interpolating between the initial and final model:
- (100) -1-0)(11-14 (,0)).
where P is the desired number of synthetic checkpoints. We then search for the batch of examples with a gradient that most aligned, on average, with the direction the synthetic checkpoints:
P Σ
v((x;))) (0,- 0,) -E
Σ
arg max BCD
4.2 Efficient per-example gradient computation
Prior work has demonstrated that the gradient of the last layer of language model can be high-resolution enough for synthetic data generation (Nguyen et al., 2025). Since our approach requires per-example gradients, which are typically computationally expensive (Li et al., 2022), we run backpropagation only for the last layer save memory and reduce overall computation. Specifically, run forward pass with large batch, and then compute batched per-example gradients for only the last layer using torch. func. vmap.?
4.3 Low dimension gradient computation
A key challenge arises from the potentially large size of the dataset D and the high dimen- sionality of the gradient Ve). Storing all gradients in their original dimension requires [ parameters, which can quickly become prohibitive. To address this, we leverage the classic johnson--Lndensteenn
lemma (Johnson & Lindenstrauss, 1984), which guaran- tees that a set of points in IR" can be mapped to lower-dimensional space IR (for k < n) while preserving inner products with high probability.
This approach has been successfully applied in prior work (Engstrom et al., 2024; Xia et al., 2024), and we adopt it here to store and retrieve gradients efficiently. Our final SELECT algorithm, which incorporates these projected gradients for batch construction, is summarized in Algorithm 1.
4.4 Distilling labels from the final model
Our threat model assumes no access to the true training data. When used for classification, our method produces synthetic training dataset by labeling large amount of web text:
y = arg max (xx;
Practically, most texts x; are not relevant to the typical text classification task. We expect that most data will be discarded during selection, and our gradient-based selection will help identify the small subset of points that are actually useful for training.
5

---

Published as a conference paper at COLM 2025
Algorithm 1 SELECT: Greedy Dataset Creation with Autolabeling
Require: X, M,d
1: p Autolabel(X,9)
2: [ t GelyyrhhetiMMMdeecccccppppp....e
3: G - CompottePorxammpleGaaiienttt Y, (
4: G + JohnsonLindennttaus((...
5: 86 < Ø
6: I <-
7: while [Z] do
i f arg max( & 077) I.append(**)
8:
9:
G broadcast(Gr))
11: end while
return
Experiments
Datasets and tasks. We test models on two tasks, classification and token-level supervised finetuning (SFT). For classification, we train final models on AG News (Zhang al., 2016), four-way text classification task, as well as DBPedia (Auer et al., 2007), a fourteen-way classification task, and the 20-Newsgroup dataset (Mitchell, 1997). For SFT, we finetune models generic web documents sampled from MSMARCO (Bajaj et al., 2018).
For all final model tasks we subsample 10K representative points. We truncate sequences to a maximum of 64 tokens. For classification we use typical instruction-tuning setup with an instruction token delimiter and all non-label tokens masked. For methods that require seed data, we evaluate using Wikipedia documents drawn from Natural Questions (Kwiatkowski et al., 2019).
Final models and hyperparameters. For classification tasks we use GPT-2 medium (Rad- ford et al., 2019); for supervised finetuning we choose the 1B and 3B parameter sizes of LLAMA-3.2 (Meta-Al, 2024). We train final models with the Adam optimizer (Kingma & Ba, 2017) for three epochs with a learning rate of 10 4, All results are averaged over three seeds, which each produce different final models and seed set splits. To evaluate the performance of a selected dataset, we retrain the model on the selected dataset with the same hyperparameters for 100 epochs and take the best model according validation set.
Baselines. Since no validation data is available, do not consider traditional coreset selection methods such as herding. Our main baseline is simple random selection, where data are automatically labeled using the final model and selected at random.
Next, we consider selecting data by gradient features without our greedy gradient- replacement strategy (top-k). Since top-k is prone to selecting highly similar data and our problem space is constrained to balanced classification, we further consider a "balanced top-k" approach, where adversary selects data based closest gradient distance but constraints selection to include an equal number of examples from each class.
Although we cannot apply the traditional influence function formulation, we can still compute the perplexity of the seed data and use it as selection mechanism. We consider the highest- and lowest-likelihood data as baselines (P-Min and P-Max), which is known to be surprisingly effective baseline for data selection (Yin & Rush, 2025).
We also evaluate variant of our greedy algorithm, SELECT (batch), which greedily selects examples based on the sum of the gradient within a given batch.
hatpss//yyoorchooo//d/e/////eeee http:://hhheccceee/eee//cotttaaa

---

Published as a conference paper at COLM 2025
Table 1: Classification: Main results comparing SELECT on dataset recovery and model performance, with 1K datapoints selected. Seed sets are Wikipedia passages from Natural Questions automatically labeled with the victim model, which is GPT-2 medium (355M parameters).
classes] Acc
DBFeddia
classss] Acc
classes]
Vocab
OTD
Loss 1
Vocab
OTD
Loes
Vocab
OTD
Acc
Lees
Random
62.31
1.11
65.62
0.91
73.26
1.08
40.15
2.15
75.79
1.16
0.80
Top-K (Bal)
$2.85
1.30
61.15
0.97
73.04
1.02
40.04
2,01
74.41
1.17
57.71
0.69
Top-K
52.61
1:11
17.97
1.78
72.34
1.05
20:18
3.86
71.79
1.17
53.19
2.30
P-Max
35.51
1:12
38.51
2.07
35.38
1.08
35.38
2.70
51.25
1.15
51,25
1.75
SELECT-Botch
81.63
1.06
75.57
0.71
76.53
0.99
56.04
1.54
77.09
1.11
59.91
0.69
SELBCT
55.01
0.70
0.98
39,91
1.54
0.71
Expert
88.32
0:40
88.01
0.12
80.87
0.51
Table 2: SFT: Comparison of selection strategies on dataset recovery and model performance across Llama-3.2 models with 1K datapoints selected.
Llama-3.2 3.2-3B
Llama-3.2-1B
PPL 4
OT
Vocab
PPL
OT
Vocab
P-Min
1.331
85.0
2.63
1.330
85.1
P-Max
36.01
1.330
85.1
46.26
1.331
85.0
Random
3.31
1.331
85.1
4.15
1.331
85.1
Top-K
9.93
1.329
85.1
20.97
1.331
85.2
SELECT (Batch)
2.67
1.330
85.0
3.44
1.330
85.3
SELECT
2.30
1.329
85.0
2.61
1.329
85.2
Expert
2.01
2.15
Metrics (Lexical). To evaluate lexical similarity, compute the 'containment similarity' at the vocabulary level by computing the union of tokens present from each dataset. This metric is defined the maximum ratio between the size of the intersection and the size of either token set, capturing the degree of exact and partial token overlap. Lexical metrics have been extensively employed in data extraction and memorization studies (Carlini et al., 2021; 2022), making them robust candidates for evaluating data recovery.
Metrics (Embedding). To capture semantic similarity, we evaluate dataset correspon- dences in embedding space. Specifically, we compute sentence-level embeddings and solve for the optimal transport (Wasserstein) distance between the resulting embedding distri- butions (Kusner et al., 2015). We employ both the full optimal transport formulation and, too expensive, its efficient entropic regularization variant via the Sinkhorn algorithm (Cuturi, 2013). These embedding- metrics capture semantic correspondences that are otherwise not be captured through lexical measures.
Results
Main results. We first compare our methods with fixed seed set size of 1k selected from 10k points; results are shown in Table 1. The Top-K gradient-based data selection performs the worst overall, with the lowest evaluation accuracy after finetuning and relatively high optimal-transpon distance. We see significant improvement by simply balancing the top-k choices by class, but this method still underperforms random selection on all three datasets. Random selection is by definition near-balanced for any balanced dataset. SELECT, on the other hand, naturally balances the class distribution of selected datapoints through gradient comparison.
In general we come close to the performance of the final model (10K sample dataset) on AG News and lag behind DBPedia and IMDB, which have more classes and may be more difficult to compress into fewer samples (since we are selecting 1000 from 10, 000 datapoints).

---

Published as a conference paper at COLM 2025
0.775
1.00
Evaluation Accuracy
0.7-
0.750
0.725
Accuracy
0.700
0.90
0.4
STLECT
0.650
SELECT (B3tch)
Random
0.625
kyp-E
Top K Balanced
1000
1500
2000
2500
Datapoints Selected
Leaked Data
Figure Data selection performance on AG News while scaling amount of selected data from 100 to 2.5K (selected from seed set of size 10k).
Figure 3: Accuracy (y-axis, left) and OT dis- tance (y-axis, right) vs the percent of seed data that is leaked from the true dataset (x- axis).
In our SFT experiments (Table 2), we are interested in examining whether per-example gradients might be useful for data selection. We find that SELECT is significantly effective for selecting data for SFT on both the 1B and 3B-parameter variants of LLAMA, giving better perplexity than any baselines. All methods perform similarly recovery as measured by OT and Vocab, indicating that the seed set (MSMARCO) is already highly similar to the true finetuning set (Wikipedia documents from Natural Questions).
Scaling selection size In previous experiments we only considered selecting 10K data- points. We then examine the performance of various methods as the number of datapoints selected scales from 100 to 2.5K. We see SELECT achieves similar performance to random sampling with just 100 samples and significantly outperforms it at higher numbers of sam- ples. In all cases, selecting data by simple Top-K gradients performs poorly. We hypothesize that this happens because the datapoints with the highest gradient similarity to the param- eter difference are similar to one another, causing highly similar subset to be selected, which is not useful for finetuning.
Analyzing the affect of leakage. Finally we consider the case where the finetuning data is publicly leaked and available some subset of our public seed dataset. We titrate the number of additional non-finetume con-fiietuning samples from 90K, where the leaked data is only 10% of the available data, down 1K, where the leaked data comprises approximately 90% of the publicly available data.
We see in Figure that our method improves with the percentage of the seed dataset that corresponds to leaked data in both accuracy and distance from the original dataset. These results indicate that an adversary may be able to 'locate' true finetuning data when it is exposed part of a public finetuning dataset.
7 Ablations
Optimal transport metric analysis. We perform ablation by evaluating the sensitivity of our dataset similarity metrics to incremental improvements in data recovery. Specifically, we progressively replace the recovered dataset of 1k datapoints with the ground-truth examples, in 10% increments. For each replacement level, run five trials and report the average. We observe in Figure 4 that both lexical and embedding metrics improve predictably as selected data are replaced by ground ground-truths.
8

---

Published as a conference paper at COLM 2025
Seed Dataset
AG NEWS
DBP
IMDB
NEWS
RT
MSM
NQ
AG NEWS DBPEDIA
81.8
50.5
54.8
69.9
36.3
74.4
79.2
27.5
60.3
17.5
19.4
11.3
48.9
59.4
IMDB
60.9
52.6
75.6
63.7
67.5
67.1
64.8
Table 3: Comparison of datasets when used as seed (horizontal) vs. test (vertical). Base model is GPI-2 Medium; we select 1k with 100k seed documents. Natural Questions is useful seed, achieving close to the performance of the true data on two of three tasks.
Table 4: Optimizer Performance Comparison: optimizer used to train the ground-truth model (horizontal) vs. the expert model, after selection (vertical).
SELECT
Random
Optimizer
Adam
AdamW
SGD
Adam
AdamW
SGD
Adam
2.15
2.44
1.97
3.42
3.42
3.42
AdamW
2.27
2.53
1.99
3.39
3.39
3.39
SGD
2.38
2.73
2.12
3.75
3.75
3.75
For the setedding-based metric (optimal transport distance), the distance decreases to 0.685 as recovered data are fully replaced by ground truth. Importantly, even when one dataset is a strict subset of the other, the OT distance remains nonzero because the larger set's mass is distributed over additional examples, incurring extra transport cost under uniform weighting. contrast, the vocabulary-level containment similarity saturates at 100% once the recovered data are fully replaced by ground truth, showing complete lexical overlap.
Seed data distribution. Most of our experiments include only seed data from Wikipedia passages, which have been shown to be effective enough train news, topic, and sentiment classifiers. We question whether how much results depend on the chosen distribution of seed dataset. We explore variety of lask-seed data distribution pairs. For each pair, we run SELECT to choose 1K representative points from a seed set of 10K, and show all pairwise results in Section
Perhaps unsurprisingly, each dataset is most useful to seed model for itself, but we do see interesting interactions. The Newsgroup dataset proves useful seed dataset for AG News, and Rotten Tomatoes movie reviews are still helpful for IMDB. Our default seed dataset of Natural Questions is surprisingly useful in each category, although lags behind using the same distribution.

---

Published as a conference paper at COLM 2025
1.05
L035
100
Containment Similarity
1.00
Full OT Distance
95
0.95
0.85
85
0.80
0.75
75
0.70
R.685
72.292
09
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage Replaced
Percentage Replaced
Distance
Containment Similarity
Figure 4: Dataset Evaluation Metrics Ablation.
Optimizer comparison. We compare the choice of optimizer used for finetuning the ground-truth model vs. final expert model, to determine whether knowledge of optimizer is necessary assumption of our threat model. Results are shown in Table 4. Values are test perplexity from DELECT/Randoe
in our the pretraining setting with LLAMA-1B. We observe that using Adam W to train the expert model slightly deteriorates performance our method, likely due to the weight-level signal lost to weight decay. SGD, which has second-order optimization, provides even better selection signal than raw Adam. (We plan to add this experiment to the ablation section, space-permittin, in the camera-ready version.)
8 Conclusion
We introduce the problem of approximating finetuning data given only model weights, e.g,, those available from an open-weights model release. We propose an approach that selects a subset of large text corpus, then show that the selected subset is sufficient train a model whose performance is close to the open-weights model. We develop metrics for dataset approximation based on optimal transport of sentence embeddings and show that our method improves over random selection according to these metrics.
These results show that releasing models as open weights, without the corresponding train- ing data, reveals sufficient information to enable selection of effective substitute datasets. Finetuning on these datasets significantly improves performance vs. alternative data se- lection methods and, in some cases, even comes close to the performance of the original open-weights models.
Acknowledgments
Research for this project was supported by NSF Grant DRI--229873, NSF CAREER 2037519, IARPA HIATUS Program 2022-22072200003, and an NSF GRFP.
References
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data, 2007. URL httpp://ling
springer.com//hapteee
10

---

Published as a conference paper at COLM 2025
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL
.9266..
Borja Balle, Giovanní Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries, 2022. URL https://arxxi..rrg/bbb/// 048455.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances neural information processing systems, 33: 14879-14890, 2020.
Gon Buzagle, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin, and Michal Trani. Deconstructing data reconstruction: Multiclass, weight decay and general losses, 2023. URL https: wrgg/bs//23//. /ttt:s///aartt.oorr////////..1 /25
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Hors,rt-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021. URL 07805.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Maithew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of a production language model, 2024. URL https://arxiv..ot /ew.///////...a.../e/// 06634.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories, 2022. URL https: //arxiv http//aawivv...a/aaaaaaaa org/abs/ 2203. 11932.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding, arxiv preprint arXi::1203.3472, 2012.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-lk with constant memory, 2023. URL https://arxix www.iii.og//bbs..... brg/abs/22111 10586.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL htttps://arai..or//aa////55// 12948.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024. URL https: http://awwwi...r/a/////////////e 12926.
Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document
selection for efficient encoder pretraining. arXiv preprint arX1m:22110.10951,222
Matthew Finlayson, Xiang Ren, and Swabha Swayamdipta. Logits of api-protected llms leak proprietary information, 2024. URL
09539.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-alinned trajectory matching, 2024. URL https:
barxiv.oor//bb/2220 057773.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Trani. Reconstructing training data from trained neural networks, 2022. URL https://arxiv.. wtt./r/////b//////eee 07758.
Penguin Huang, Chhavi Yadav, Ruihan Wu, and Kamalika Chaudhuri. Can infer confidential properties of training data from llms?, 2025. URL https://aaxis: https//////xxxoooo//a//////s 2506. 10364.
11

---

Published as a conference paper at COLM 2025
RC St John and Norman R Draper. D-optimality for regression designs: review. Technomet- rics, 17(1):15-23, 1975.
William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a Hilbert space. In Contemporary Mathematics, volume 26, pp. 189-206, American Mathematical Society, 1984. doi: 10.1090/ conm/026/7374000
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh lyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh lyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, 2017. URL httt:://arxiv.orr/bb///// 1atss//11...6000 6980.
nttps://arxivv org/
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp. 957-966. PMLR, 2015.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1162/tacl 2at2//ttallaaa// a a_00276. URL https://aclaattholgg/ org/Q19-1825//
Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W White, and Sujay Kumar Jauhar. Making large language models better data creators. Xi preprint arXit:2310.20111 2023.
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners, 2022. URL http:tts///axxv.oorr/aa///////// 05679.
Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, and Tu Vu. Efficient model development through fine-tuning transfer, 2025. URL https://arxiv.or 2010..
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. ar preprint ar Xix::2412.19437 2024.
Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, and Manabu Okumura. Dilm: Distilling dataset into language model for text-level dataset distillation, 2024. URL https://arxiv. org/abs/2404 002664.
Meta-Al. The llama 3 herd of models, 2024. URL URLnttpps///arrii..g//////////
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33:11465-11477, 2020.
Tom Mitchell. Twenty Newsgroups. UCI Machine Learning Repository, 1997. DOI: http://dooorgg/0.2244/////C
John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. Language model inversion, 2023, URL https://arxivv org/abs//3111 13647.
John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexan- der M. Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize?, 2025. URL htttt://aaaaiii wwtp://////........// 24832.
12

---

Published as a conference paper at COLM 2025
Milad Nast, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023. URL https://arxiv.orr//bb//22 hwtp:oor/////.///.// 17035.
G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular functions Mathematical Programming, 14(1):265-294, 12 1978. doi: 10.1007 /B////FF11589/7 URL httpp::///iiie. wtw.oooooog/..... 1007/
Dang Nguyen, Zeman Li, Mohammadhossein Bateni, Vahab Mirrokni, Meisam Razaviyayn, and Baharan Mirzasoleiman. Synthetic text generation for training large language models via gradient matching, 2025. URL
/ttp///xxii.orr/////////./2222 17607.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Christina Runkel, Kanchana Vaishnavi Gandikota, Jonas Geiping, Carola-Bibiane Schönlieb, and Michael Moeller. Training data reconstruction: Privacy due uncertainty?, 2024. URL https::/arxii.org/aasssse
24444.0044/.1.04444.4.4 08544.
Tongzhou Wang, Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020. URL https://aarxiv.orr/bbb/// 10959.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024. URL https://arxiv.orr abs/2402. 04333.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang, Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201-34222 2023.
Junjie Oscar Yin and Alexander M. Rush. Compute-constrained data selection, 2025. URL http::///aaxxw..o//bbbbsoo/ 16288.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensa- tion imagenet scale from new perspective, 2024. URL 13092.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2016. URL https://arxits https://arxiv.rrg//b////B wttp///////////////// 01626.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, 2021. URL httpps://arset httts:://arxi..rr//bb/////// 05929..
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 35:9813-9827, 2022.
A Ethics
The question recovering training data from the weights of machine learning models is related to protecting the intellectual property of companies that train these models. Model creators often want to release model weights, so that their models might gain users, but not want to release the data, for many reasons.
Our work is the first step towards recovering information about this unknown data, which may already be possible with the model weights that public today. Ours is but the first exploration into this problem space, yet our method is able to find similar data available on the Web (which may or may not be the exact training data).
13

---

Published as a conference paper at COLM 2025
It remains unknown exactly how much information about the training data can be extracted or leaked from the weights of open models. We advise model creators to approach open- weights model releases with caution.
B Ablation: Gradient Store
Evaluation
0.66
0.64
0.62
6x100
30
Projection Dim.
Figure 5: Performance of SELECT (blue) vs. random selection (red, dashed) on selecting 1K points from Natural Questions to train an AG News classifier. We note that our method improves somewhat with larger projection dimension, although improves over random with dimension of 512.
Gradient store dimension. We rerun our base experiment (NQ -> AG News, selection size of 1K, seed size of 10K) with variety of projection dimensions. We compare SELECT across projection dimensions to random selection in Figure 5. We see that gradient-based selection outperforms random even with a dimension as small as 512.3
'In addition to storage size, the total number of FLOPPs grows quadratically with the dimensionality of the projection. Thus, we opt use size of 096 for all experiments.
14


Reviews:
[{"review_id": "k9aVxD15VT", "reviewer": "Reviewer_cimo", "summary": "This paper presents a method for generating approximating fine-tuning data for a text classification tasks. \n- The goal is to find a set of synthetically generated data that can \"simulate\" the training effect -- measured as either close to the actual fine-tuning data (lexically or semantically) or leading to the same training effect (approximating similar accuracies or losses). \n- The key idea of the method is to search for examples that can lead to gradient changes along the direction between the gradient difference between pre-trained model as well as the fine-tuned model. \n- Several techniques are considered to improve the efficiency (using only the last layer gradient, plus mapping the stored gradients to lower dimensions) and effectiveness (considering the interpolated weights between the starting and end point) of the algorithm. \n- The papers empirically show that the algorithm can outperform several simple baselines, and conducted ablations and analysis on the proposed method to show its effectiveness.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extracting the training data of the open-weight language models is an interesting setup. It can help people better understand the training dynamics of the language models, as well as the safety of the models. \n- The method seems to be reasonable and Novel; the empirical results seem to show the strength of the proposed method.", "reasons_to_reject": "- The current setup may be a bit synthetic. \n    1. First only having the task classification task seems to be a bit limited for such language models -- also the training setup doesn't use the typical instruction fine-tuning setup (as it doesn't use a textual prompt but simply requiring the last token to be a class label). \n    2. Furthermore, perhaps what's interesting in this setup is how to avoid some sensitive (exact) training data being recovered; however this method does not have the guarantee for recovering verbatim copies of the training data---all it can do is \"approximating\" the data---unless it can have access to a good set of seed data. In that spirit, it is closer to dataset distillation, which the authors seem to be aware of. In that case, I suggest the authors change the framing of the paper a bit. \n- I think there are several additional experiments for this setup: \n    1. As one important baseline, one can select the data based on the logprobs: for example, we can score the sequences using the fine-tuned model, and higher logprobs can indicate more confidence or closer to the training data. Running a top-k experiment using the logprob seems to be an important baseline and I suspect it can achieve nice results. \n    2. Since the authors only use the last layer gradient, during the training of the final model, one should also consider freeze all the layers expect for the last one. Also the authors can consider a layer selection method -- identifying the best layers to choose. \n    3. An interesting analysis for the experiment in figure 3 is that the authors should report how much of the leaked data is selected using different algorithms.", "questions_to_authors": "- In figure 3, the x axis should be 10, 20.. instead of 0.1, 0.2, .., if I understand it correctly. \n- the reference to table 2/3 is not correct (it writes section 7 in the last paragraph on page 8)", "extra_scores": {}}, {"review_id": "gb1dS0HQTM", "reviewer": "Reviewer_N6vu", "summary": "This paper addresses an interesting problem by proposing a method called SELECT, which identifies samples from a publicly available dataset that closely resemble the original training set of an open-weight model—despite the original training data being unavailable. The method leverages two versions of the same model: a base model and its fine-tuned counterpart. It uses the base model to compute last-layer gradients and the fine-tuned model to generate pseudo-labels for the samples. Overall, the paper presents a modern take on the coreset selection problem in scenarios where no information about the original training set is accessible.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper is very well written and has the core message is well written.\n\n2) The design of the method is very well explained and based on prior works (also make sense).\n\n3) This is a hard problem and the authors have used many approximations regarding gradient computation and solving the original bi-level problem.\n\n4) The results overall looks reasonable (although I have some reservations about baselines).\n\n5) The problem is very interesting and very useful for the community.", "reasons_to_reject": "1) The paper started with absense of training data and availability of model weights but then claims to be a good coreset selection algorithm which is orthogonal to the main story. Refer page 2 para 1 \"disjoint from true training set ...\"\n\n2) Based on my understanding AdamW is widely used for LLM training not sure why the authors are using adam.\n\n3) The baselines are quite weak -- for instance show us first what are the open weight model performance on the test set of the datasets you are using. Then claim that even without having the train set we have achieved a certain level of performance. This method is beating random subset selection baseline so, it has some merit already.\n\n4) Can you use Openwebtext or any pre-train data and use your method and show how much perplexity difference it has with the open weight base model. As all the results are shown in smaller scale data. Openwebtext is also a small size data should fit in academic compute budget. This will help us gauge the method's impact on pre-training.", "questions_to_authors": "Here are few questions to the authors regarding the design choices for the select algorithm.\n\n1) Why computing gradients is neccessary and can't a similar result be shown with just computing loss which is much cheaper. Is it because this method is inspired by Less (by Xia et. al)  and they use gradients instead of loss.\n\n2) For gradients computation the base model is used and for pseudo labelling the finetuned model is used. Can we use the finetuned model for both?", "extra_scores": {}}, {"review_id": "qjcAaWsUVi", "reviewer": "Reviewer_pFEW", "summary": "The authors propose a gradient-based approach for approximating private training data of publicly available model weights. Their method \"SELECT\" relies on having a base and fine-tuned open weight models as well as an unsupervised seed dataset. The authors use the \"final model\" checkpoint to assign labels to data instances, then compute synthetic checkpoints, use the Johnson-Lindenstrauss lemma to compute low-dimensional gradients, and finally perform greedy data selection by picking instances with the highest running gradient sum.\n\nThe authors evaluate their method on GPT-2 Medium and Tiny-Llama, and evaluate their methods on three different classification datasets comparing it against Random selection and Top-K selection. The authors, restrict their method to the last layer of the model in order to save compute. They also perform Ablations using optimal transport metric analysis.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and well written.\n- The topic and experiments seem highly relevant to some of the most important current discussion regarding LLMs and copyrighted data.\n- Even though the models used are small, the authors show the effectiveness of their methods and perform ablations.\n- The authors plan to release the code along with instructions fro reproduction.", "reasons_to_reject": "- While using the last layer of language model can be high-resolution enough for synthetic data generation, I don't immediately see how this translates to your method.\n- At least from the way the method is presented, it seems to rely a lot on the seed selection, so if the proposed method wants to be used to approximate training data, this seems like a major challenge as most of the datasets for current LLMs might not even be publicly available, so your method is not applicable.\n- Even though you say that even in the cases when the training dataset is not available, your method also works for data selection, there is a lack of comparison between your method an more traditional techniques for data selection.\n- There is an important discussion missing in this paper of how the method scales computationally as the model size and dataset sizes increase, this is not evident from the paper as the models and datasets used were rather small compared to modern standards.", "questions_to_authors": "- How does your method scale to larger models and datasets? While this might not be easy to answer with your current compute resources, providing an estimate based on your experiments would be useful.\n- How does your method compared both in performance and computational needs to more traditional methods for data selection? Especially for pre-training.\n- Figure 2 is completely unreadable for a colorblind person. Please use a more accesible palette. Most libraries today have them and around 8% of the male population suffers from colorblindness so this will make your paper more accessible.\n- Please refrain from using notation like $\\theta^T$, this is already standard notation for something else.", "extra_scores": {}}, {"review_id": "GFQ0tVTH3R", "reviewer": "Reviewer_T5hs", "summary": "This paper introduces a gradient-based data selection method designed to approximate the training data of a fine-tuned language model using only the base and final model checkpoints and a large unlabeled text corpus. By selecting data whose gradients align with the weight difference vector, the authors show that it is possible to reconstruct a dataset that achieves similar downstream performance to the original training data.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is technically interesting and makes clever use of weight-space geometry and gradient submodularity.\n2. Good ablations. The authors probe various aspects of their proposed method such as seed set distribution, gradient projection dimension,  etc.", "reasons_to_reject": "1. The paper assumes access to both the base and fine-tuned model weights, the training algorithm, optimizer type, and training steps. In practice, many open models do not release all these details just the weights.\n2. The experiments are restricted to small-scale classification tasks and relatively small models. Unclear if the findings will translate to realistic scenarios such as instruction-tuned LMs.\n3. Optimal transport on sentence embeddings is used to assess recovery, but the authors do not discuss how well this correlates with actual overlap or usefulness of recovered data. It’s possible to achieve good OT scores without semantically meaningful overlap.\n4. The comparison to “random” and “top-k gradient” baselines seems insufficient. More sophisticated baselines from data distillation or data selection literature (e.g., influence functions) are not considered. I understand they are slow, but the experimental setup considered in this paper will work just fine with influence functions.", "questions_to_authors": "What happens when the seed dataset is large and diverse (e.g., Common Crawl)? Does your method still recover task-relevant data?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a gradient-based approach to approximate the data a language model has been fine-tuned on. 

Reviewers all agree that the proposed approahc tackes an importnat problem, is original, and is supported by good experimental evidence. They also note that the manuscript is well written, and that the proposed method should be easy to reproduce. 

Reviewers raised several criticism during discussion period; authors have incorporated nearly all of them, significantly improving the quality of the paper. 2 out of 4 reviewers expressed satisfaction with new experimental results and explanations, raising their score.

**Pros**

- Solid motivation for this work (cimo,N6vu,pFEW)
- Novel method (cimo, N6vu,T5hs)
- Reasonable approach (cimo,N6vu)
- Good experimental results (pFEW,T5hs)
- Should be easy to reproduce (pFEW)
- Well written (N6vu,pFEW)

**Cons**

- the experimental setup is uncommon; instead of instruction-tuning, it considers more narrow classification tasks (cimo, N6vu)
  - fixed in rebuttal
- the proposed method is not extracting data, more similar to distillation (cimo)
- missing several baselines (cimo)
  - fixed in rebuttal
- paper uses adam instead of more common adamw (N6vu)
  - fixed in rebuttal
- Requires access to data similar to training data (pFEW) as well as training algo, optimizer type (T5hs)
- unclear how paper scales (pFEW) 
  - partially fixed in rebuttal
Decision: Accept