Influence-Preserving Proxies for Gradient-Based Data Selection in LLM FineTuning
Authors: Sirui Chen
Yunzhe Qi
Mengting Ai
Yifan Sun
Ruizhong Qiu
Jiaru Zou
Jingrui He
Venue: ICLR 2026 Poster (2026)
Keywords: Data Selection
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce IProX, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model’s influence. Experimental results across diverse LLM families and evaluation tasks show that IProX consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with IProX achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, IProX achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that IProX provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.

Published as a conference paper at ICLR 2026
INFLUENCE-PRESERVIIG PROXIES FOR GRADIENT- BASED DATA SELECTION IN LLM FINE-TUNING
Sirui Chen", Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui He University of Illinois Urbana- Chamaapag (siruie yunzheq2, mail0, yifan50, rq5, jiaruz2, jingrui) (erruieeill jeill aelinol inois.ii........ is. edu
ABSTRACT
Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits model's downstream performance. Gradient-based data selection meth- ods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-pareee parameeer large language models (LLMs). A common alternative is to use off shellf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-bart based in- fluence estimation. To address these challenges, we introduce IPROX, two-stage framework that derives influence-preserving proxies directly from the target model. It first applies low-rank compression stage preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's 's influence. Experimental results across diverse LLM families and evaluation tasks show that IPROX consistently outperforms off-the- shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with IPROX achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, IPROX achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that IPROX provides effective influence-preseer gradient-basse data selection more scalable for LLMs. The code is available at attps: /i thub com/ 1eerr1//PPre
proxies, making
INTRODUCTION
Supervised fine-tuning (SFT) has become the standard approach for adapting Large Language Models (LLMs) to various downstream tasks. However, the effectiveness of SFT hinges critically on the training data. Prior studies (Wang et al., 2023b; 2024a) show that naively combining datasets can even degrade downstream performance. The key challenge, therefore, not the sheer amount of data available but the identification of a curated subset that most effectively enhances model performance.
prominent line of work addressing this challenge is gradient- based data selection, where each sample's importance is estimated through its influence the model performance. For example, TracIn (Pruthi et al., 2020; Xia et al., 2024; Han et al., 2023) estimates the impact of training sample by accumulattin gradient inner products with validation sample across multiple model checkpoints, while Infuence Functions (Koh Liang, 2017; Kwon et al., 2024; Zhang et al., 2024; Wang et al., 2025) approximate the effect of infinitesimally upweighting scaling its gradient with the inverse Hessian to account for the local curvature of the loss landscape. Despite their success, both methods impose substantial computational overhead, requiring either the storage of numerous checkpoints with repeated backpropagation or the computation of costly inverse-Hessian vector products. This overhead scales poorly with model size, making these methods impractical for multi-bille multi-billionn billion--prameter LLMs (Grosse et al., 2023).
downweighting training sample by
While there are some efforts focusing on simplifying the influence computation itself (Kwon et al., 2024; Yu et al., 2024; Xia et al., 2024; Lin et al., 2025c), we pivot to an alternative, orthogonal
"Equal Contribution.

---

Published as a conference paper at ICLR 2026
question: can the expensive influence calculation for target model be effectively offloaded smaller, cost-effective proxy model? The idea of using smaller models to predict the behavior of larger ones is already prevalent, most notably through scaling laws that estimate a target model's performance from its smaller counterparts (Kaplan et al., 2020; Shum et al., 2025; Zeng et al.; Lin al., 2025b; Zou et al., 2025a). Motivated by this, we explore whether this proxy paradigm can also be extended to data selection by leveraging gradient-based influence scores from smaller models as approximations for larger ones, thereby mitigating the prohibitive cost of full-scale computation.
A direct strategy is to use off-the-shelf proxy models (Xia et al., 2024; Yang et al., 2024b), such as applying Llama3-8B to select data for Llama3-70B. These proxies provide strong baselines and useful guidance. but remain suboptimal for three main reasons. First, while their task performance is usually reported, much less is known about their learning dynamics on the data. As a result, choosing an off-the-shelf proxy for gradient-based influence estimation typically relies on prior knowledge (e.g., assuming the larger model always behaves similarly to its smaller counterparts), without a clear understanding of how much benefit is gained by increasing size. Second, the available off- shelf models within each family are restricted to a handful of fixed sizes, which limits flexibility in adjusting proxy capacity to different computational budgets. Third and most importantly, there is no systematic way to better align these proxies with the target model for influence estimation.
To address these challenges, stage framework that constructs a proxy directly from the target model, starting with compression and followed by alignment. The key idea is straightforward: instead of relying on a smaller model with assumed
propose IPROX, a principled two-
When Smaller Beats Bigger
Ang. Pofemance
preferences, we derive smaller model directly from the target so that it inherits the gradient characteristics of the original. This design provides flexibility in controlling computational cost and, more im- portantly, establishes a principled path to preserve the influence of the target model. Concretely, we first employ Influence- PPeeerrviig Singular Value Decomposition (IPSVD), where each weight matrix of the target model is compressed to retain components most relevant for madientt-based influence. Building on this, we then introduce an aligning stage that refines the proxy by matching its gradients to those of the target model within the low-rank space while anchoring its output logits to remain consistent. Together, these stages yield a proxy that is both efficient and tailored for gradient-based data selection.
09 shette
3:58
Figure 1: For Qwen3-4B, 1.5B IPROX outperforms the Qwen3-1.7B off-the- shelf proxy, demonstrating that a smaller influence-pre- serving proxy can achieve better data selection perfor- mance.
Experimental results demonstrate that IPROX achieves consistently better performance than off-the-shelf proxies across diverse tasks and model families, and its advantages hold under different gradient-based influence estimators. A representative example is shown in Fig. 1, where for the Qwen3-4B target model, our 1.5B proxy constructed by IPROX surpasses larger 1.7B off-the-shelf proxy in average performance, high- lighting that smaller IPROX can outperform larger off-the-shelf ones. In addition to stronger performance, IPROX is efficient. In our experiments on Llama3.2, it reduces the computational overhead by more than half relative to the full 3B model, offering a practical and scalable path for efficient gradient-based data selection in LLM fine-tuning.
2
RELATED WORKS
Gradient-Based Data Selection. Traditional data selection is often model-agnostic,, emphasizing dataset structure such as diversity (Liu et al., 2024), redundancy reduction (Wei et al., 2015), or similarity to a target distribution (Kang et al., 2024a). In contrast, gradient-based selection is model-aware, leveraging first- and second-order training dynamics to estimate each example's 's impact on learning and validation performance. A main line builds on influence functions (Kwon et al., 2024; Grosse et al., 2023; Klochkov & Liu, 2024), which approximate the leave-one-out effect of data points with respect upweighting or removing a training example, helping identify dynamics attribution (Bac et al., to target distribution. Another complementary line uses training-dnnmiic 2024), scoring examples by how their gradients align with improvements on held-out targets across training checkpoints. A representative approach is TracIn (Pruthi al., 2020; Han et al., 2023; Xie
2

---

Published as a conference paper at ICLR 2026
et al., 2024), which aggregates such pradient-similariy signals along the optimization trajectory, avoiding expensive second-ordes computations while remaining model-aware.
Efficient Data Selection for LLMs. With the growing size of LLMs, gradient-basee based data selection has become increasingly impractical, motivating more efficient adaptations. Some works reduce the cost of influence estimation by simplifying second-order derivatives (Kwon al., 2024; Grosse et al., 2023; Zhang et al., 2024), while others compute influences on a small subset and extrapolate to the full dataset (Xia al., 2024; Yu et al., 2024; Gu al., 2024; Lin et al., 2025c). Recently, an alternative line of work has explored using smaller off-the-shelff proxy models to guide data selection for larger ones, though these approaches primarily rely on loss signals rather than exploiting gradient information (Yang et al., 2024b; Shum et al., 2025). In the broader context of efficient LLM adaptation, recent studies also leverage fine-tuning dynamics (Zeng et al.) and automated scaling laws (Lin et al., 2025b) to optimize computational allocation.
Decomposition- based compression exploits the low intrinsic rank of weight matrices. Early work showed that singular value decomposition (SVD) can effectively approximate transformer layers (Ganesh et al., 2021). I). Subsequent studies refined this idea: ASVD incorporates neuron activation patterns (Yuan et al., 2025), CALDERA combines low-rank factorization with quantization (Saha et al., 2024), and MoDeGPT applies Nyström approximation to entire transformer blocks (Lin et al., 2025a). SVD-based strategies have also been extended to Additionally, ShortGPT introduces an importance-scoring mechanism identify and retain the most critical layers (Men et al., 2024).
LLM Compression via Decomposition Methods.
models (Ai et al., 2025c; Yang et al., 2024a; Li et al., 2025).
3 PRELIMINARIES AND PROBLEM DEFINITION
We consider a candidate training dataset Danin and target validation dataset D which may either follow the same distribution or a different one. The objective of model-aware data selection is to identify subset D C Drrain with fixed budget le such that fine-tuning model fo on DD' maximizes its downstream performance on D
D = arg max Ee'DD., ((fo( DCD [D|=k
where Ui is a task utility (e.g.. accuracy), ((D) are the model parameters fine-tuned on D, and a' E Daal is validation sample. Directly solving the combinatorial optimization in Eq. 1 is intractable. A widely used strategy is to instead score each training sample E D based on its gradient-based inftuence on Dval and select the top-k samples. This is typically achieved by defining a pairwise influence score [2, z'), which quantifies the utility of training on a sample z for the model's performance on a target sample z'.
A prominent example of this idea is TracIn (Pruthi et al., 2020), which approximates [(z, x²) by accumulating gradient similarities between training and target samples over multiple checkpoints:
T
Tracia (2, 2, n (VVo((z;)) VoL(z'; (.
Σ
where T-(:: -) is the loss function, O2 is the model's parameters at checkpoint / and n) is the averaged learning rate in iteration t. By probing the geometry of the loss landscape throughout training, this method provides a faithful measure of sample's utility. Another seminal method, Influence Functions (Koh & Liang, 2017), estimates the influence of a training sample by modeling how the final model parameters would change if that sample were infinitesimally upweighted. This parameter change is approximated as the inverse Hessian of the loss multiplied by the sample's ' gradient.
However, the computational cost of these gradient-based methods is prohibitive for large-scale models, motivating the use of smaller proxies to approximate influence scores. The central challenge, and the focus of this work, is to design proxy model for that not only approximates the influence scores of the target model fe but also strikes balance between efficiency and selection quality. Ideally, the proxy should be small enough to offer notable computational savings while remaining sufficiently aligned with the target model to guide effective data selection.
3

---

Published as a conference paper at ICLR 2026
Stagel: Influence-Preservvee SVD
Stage2: Gradient Alignment
Torget LLM
Input sequence
Labeled onswers
Hilden
En cligneent Leca m projection
stanes
Snadents
Prany
Weights
Esherna
ottersion
Figure 2: Overview IPROX. the first stage (left), IPSVD leverages hidden states and gradients to build mooond matrices that reweight the model weights for proxy initialization. In the second stage (right), the proxy further aligned with the target LLM through internal gradient alignment in the low-rank space and external logits anchoring for stability.
PROXY CONSTRUCTION VIA INFLUENCE- CELLUENCE-PRESEREE PRESERVING COMPRESSION
We introduce IPROX, summarized in Fig. 2, which consists of stages. The first stage compresses the model with an influence- uufluence-reeserving SVD ($4.1) that uses second- reweighting to retain influence-relevan components. The second stage aligns the proxy with the target LLM ($4.2) by matching gradients in the low-rank space and anchoring the logits distribution for stability.
4.1 STAGE 1: INFLUENCE- PREENNCCI SVD
Limitation of Standard SVD. We begin by describing how the proxy model is initialized. A natural approach is to compress the model via low-rank approximation of its weight matrices. For any weight matrix W E gaxm in the target model fa. where n, m are output and input dimensions, where A E Txx and B E RR The rank r < min(n, m) directly controls the size of the resulting proxy model, with lower ranks corresponding to higher model sparsity. The standard method for such decomposition is Singular Value Decomposition (SVD), which yields the optimal rank-r approximation under the Frobenius reconstruction eтror objective (Eckart Young, 1936; Golub & Van Loan, 2013). However, this objective is misaligned with our goal of data selection, since minimizing reconstruction error provides no guarantee that the proxy model will preserve the gradient-basee influence of the target model.
Retemtion
approximate it as W se AB,
5.2
Compression Sparstty Sparsity Figure 3: Loss and influence (TracIn) retention SVD and our IPSVD under different compression sparsity.
As illustrated in Fig. 3, when a 4-layer MLP is compressed on a synthetic classification task using standard SVD, loss retention (measured the ratio between the original and compressed losses) remains relatively stable when the sparsity is low, while influence retention (measured by Spearman correlation with the oracle influence) deteriorates much more rapidly. These observations highlight the need for a compression method that explicitly preserves influence. To this end, our IPSVD is designed to retain refleance-reeeant components. As previewed in Fig. 3, IPSVD attains markedly higher influence retention than standard SVD while maintaining comparable loss retention across sparsity levels. We now present the technical details.
IPSVD with Reweighting. Our goal is to construct a compressed proxy whose influence scores approximate those of the target model. For clarity, we focus on simplified variant of the TracIn computed from a single checkpoint and denote it as I, omitting the subscript. Without loss of generality, we present the analysis with TracIn, and the results also apply to other gradient-baser methods such as Influence Functions (see Appendix E). Specifically, for weight matrix We at layer its gradient given by the outer product V w. L(2; 0) = he (z)T T, where he-1(2) (2) is the input to layer l and 6z(2) is the upstream gradient from the loss. Then the influence of We is:
[w, x (V ((:;), Vw, L(2: = (5e(z), (e(22) ()=)))) (2), (z-)))
where ( is Frobenius inner product. From this definition, we observe that any sufficiently small perturbation W. We We + F₁ affects the influence only through the resulting changes dz(-). A
4

---

Published as a conference paper at ICLR 2026
first-order Taylor expansion of the loss with respect We in the direction of the perturbation Es yields the scalar (Vw, L(2;0), Et) = 5e(2) E which captures the effect of the perturbation on the sample loss. We therefore define the layer-local directional effect of a perturbation E, sample 2 as:
ex(2) - T Eghe-1(z).
(3)
The following proposition provides a theoretical justification for using the expected squared effect, E. [e as a tractable surrogate for preserving the influence score.
Proposition 4.1. Consider a perturbation to layer & We We We + Eg. Under assumptions of local smoothness, geometric coherence, and a bounded covariate shift condition between the distributions of z and z' (see Appendix D) for details), there exists data-dependent constant CR > O such that the expected change in the influence contribution is bounded by:
E Bg Lw, (2, 2/) Iw,(2, 2¹) S CxvEelee(=))]
(4)
The proof is deferred to Appendix D. Intuitively, the smoothness assumption ensures that perturbations in layer weights translate into proportionally bounded changes in the gradients. The error term ea (x(2) represents the local gradient deviation caused by Er, and its squared expectation thus serves as surrogate for bounding the discrepancy in pairwise influence /w, across samples. Minimizing [e] therefore directly controls the distortion of the influences.
Building on this result, goal is to find the optimal low-rank approximation We that minimizes the expected squared effect, E. (2)) This objective can be expressed as a weighted Frobenius norm between the original and compressed weights, which under the K-FAC approximation (Martens & Grosse, 2015; Grosse & Martens, 2016) takes the following form:
min Eg [ee(z)²] W,
Re min C/ (W, - (W. W,
1/2
F²
where Cn.t A Elbe-1hi- and Cx.e [[[6,,7 are the second moment matrices of the inputs and upstream gradients, respectively. In effect, these matrices form reweighting scheme. They rescale Eg to more heavily penalize errors in directions where inputs are typically large (identified by Ch,e) and where the loss is most sensitive (identified by Cs,z). This ensures that approximation prioritizes preserving the weights most critical to the influences.
This reweighting can be expressed by the data-dependent matrix S C. C.C.W. We then compute the SVD of this matrix, S VV and truncate it to the top re singular values obtain the components Ur., Ve,r, and Ve,-- The optimal low-rank approximation We is then constructed by transforming these truncated components back the original weight space:
W. = -C,,
For implementation, this is directly decomposed into the low-rank matrices Wt A Ar where A 1/2 Ee and B, V.T.V.C,,,,l To ensure numerical stability, we add small damping term AI each second moment matrix. In this low-rank approximation, the weight matrix We E RTT x is approximated with two smaller matrices, A₁ Rm and BN Tee" reducing the parameters layer re + ne). The rank re provides flexible control over the proxy size, enabling a balance between efficiency and approximation quality under given computational budget.
Efficient and Scalable Implementation. Computing the square roots and inverses of the large second moment matrices Ch, and Cx is prohibitively expensive for large models. To avoid forming these matrices, we approximate the second-moment statistics using smallprobe set of N samples. A single forward and backward pass collects the inputs and gradients at each layer l, which are then used to form two matrices:
Hi ( E X and A. =
E R
Instead of building the full second moment matrices (e.g.. Ch. Rs 3/4 H.HH,), we compute the "skinny" SVDs of these tall-and-thin probe matrices directly: H, = UH. Hee and A = UA V This decomposition provides the key to bypassing the expensive computation. The
5

---

Published as a conference paper at ICLR 2026
SVD of the large, re- ee-weihtted matrix St be almost entirely constructed from the SVD of much smaller core matrix, which is built using the components of our skinny SVDs. This reduces the problem to finding the SVD of a matrix whose dimensions are at most N x N, dramatically smaller task. The complexity is then reduced from On + mz) for full eigen-decompositions to O(N' N² + 22N where N ng, m.g. For complete derivation, please see Appendix F.
4.2 STAGE 2: APPROXIMATE GRADIENT ALIGNMENT IN THE WEIGHT SPACE
The initial proxy model for adheres to the theoretical bound established Proposition 4.1. However, approximation errors compound across layers, its alignment in terms of influence preserving with the original model fe should still be refined. To this end, we employ an aligning stage wherein the proxy is trained to directly mimic the gradient responses signals of the target model.
Aligning Internal Gradient via Low-Rank Projection. Our goal is align the gradients of the initialized proxy with those of the target model. A direct comparison of their gradients, V and VgL. is ill-posed due the dimensional mismatch between the models. In practice, this can be addressed by projecting the proxy's 's gradient into the original model's 's high- sigg-dimensional weight _ space. For instance, for any layer é and given batch B { one can reconstruct the proxy gradient V LB; 0*) and minimize its distance to the target gradient V L(B 0). However, this approach has a critical drawback. Once we align the gradients of W2 and W/ in the full parameter space, any subsequent influence calculation would also require reconstructing the proxy's gradient in this gigh-dimessional form. Performing this reconstruction for each sample introduces substantial computational and memory overhead, which undermines the efficiency benefits of a low-rank proxy.
To ensure the proxy remains efficient for downstream tasks, we adopt more practical strategy: we project the original model's gradient down into the low-rank proxy space and perform the alignment there. Since the proxy layer is defined by low-rank matrices Aç and B₃ (where W₃ Ae),). its true gradients are with respect to these matrices, VA, and L. Using the chain rule, we can project the full gradient Vw, L onto Aş and B₁, where VA,L = aL aW,
aW: aAc = Vw, LB, and ØL oW; A Vw,L. This yields loss based on the following alignment objectives: aw; an,
V VBL-=
LGA (B; 0')
A s))V
(6)
C
FEC
where .(...) ) is a distance function and L denotes all decomposed layers in the proxy model. Here sg( sg( L) indicates stop gradient. This objective aligns the gradients entirely within the parameter space of the proxy, eliminating any need for high-dimensional reconstruction during influence calculation and thus preserving its efficiency.
Anchoring External Output Behavior. To stabilize gradient alignment and prevent the proxy from collapsing, we anchor its output distribution to that of the teacher model, inspired by the idea of knowledge distillation. This provides a stable basis for alignment, where we employ the standard forward Kullback-Leine (KL) divergence loss:
LKL LBL(B:O) 0¹)
KL (sfotmaxx(fa((())) (ff) )
(7)
B
where F the distillation temperature and fer for are output logits. Our final objective for the initialized proxy combines the gradient alignment and output anchoring losses:
min (B: + AKL (B: 0*),
(8)
0r
where AKL controls the strength of the anchoring term
Discussion. IPROX shows that low-rank proxies can preserve gradient-based influences, but there are trade-off's to consider. The embedding layer and LM head are essential for model performance and are less suitable for compression (Namburi et al., 2023; Dettmers et al., 2022), which places a natural limit on parameter reduction. Moreover, prior work finds that model quality drops sharply once the rank falls below about 10% of the original size (Wang et al., 2024b; Hsu et al., 2022), meaning proxies cannot be reduced arbitrarily without sacrificing performance or their ability to preserve influence. Even with our aligning stage, fully recovering gradient behavior under such aggressive

---

Published as a conference paper at ICLR 2026
compression remains difficult. These limitations do not diminish the usefulness of our method but highlight the inherent trade-offs between efficiency and proxy quality. Further discussion is provided in Appendix B.
5 EXPERIMENTS
In this section, we provide comprehensive evaluation of IPROX. We first describe the experimental setup 1$5.1), then present results comparing IPROX with off-the-shelf proxies and baselines ($5.2). We follow with analysis ($5.3), covering different influence estimators, efficiency, factors behind its effectiveness, probe set quality, and ablations. Additional results under varying data budgets, the effect of IPSVD, with diverse candidate training datasets, and sensitivity studies on extreme sparsity and the KL coefficient are provided in Appendix C.
5.1 EXPERIMENTAL SETUP
Datasets and Models. We use the DOLLY dataset (Conover et al., 2023) as our candidate training data Duain following (Wang et al., 2023b). It provides a diverse collection instruction-respome pairs designed for aligning large language models with human preferences. We further evaluate alternative candidate training datasets spanning diverse domains in Appendix C. We evaluate models ranging from 3B to 7B parameters across four different model families: Llama3.2-3B (Dubey et al., 2024), Gemma3-4B (Team et al., 2025), Qwen3-4B (Yang et al., 2025), and Qwen2-7B (Team, 2024).
Baselines and Evaluation. To knowledge, this direction is underexplored. so we mainly compare with off-the-shelf proxies within the same model family. In addition, we propose two baselines based on related work: Layer Ex- traction, which selects layers from the original model using heuristics (Men et al., 2024), and Influence Scorer, which trains smaller model
Table 1: Statistics of the evaluation datasets for fine-tuning.
Dataset
Task
Dal
# Shots
Metric
TyD:QA
Muttilinuul
MMLU
Multiple choice
18,721
285
Accuracy
BBH
Reasoning
920
81
Accuracy
to predict influence scores for the dataset (Yu et al., 2024). Following (Xia et al., 2024), we use MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2022), and TyDiQA (Clark et al., 2020) to evaluate the final performance. Table 1 shows some statistics. Appendix A. contains more details.
Data Selection Settings. We implement TracIn-based influence estimation following Xia et al. (2024), adopting the SGD influence variant and omitting the gradient projection component for simplicity. For influence function estimation, we implement it based on the K-FAC method (Grosse & Martens, 2016). The target models are first warmed up on a randomly selected 5% subset Duain for subsequent data selection. Data are then scored according to the computed influence values, and the top 5% are selected. Each model is full fine-tuned on the selected data for 4 epochs. As discussed in Section 4.2, we freeze the embedding and LM head during warm-up to prevent performance degradation and exclude them from influence calculation. Appendix A.2 contains more details.
Implementation Details. IPROX is built from the warmed-up target model. We implement it using 1% of the data source, of which 10% is allocated as probe set, and 90% aligning data. We vary the sparsity level p. the proportion of parameters removed by compression, to examine the trade-off between efficiency and performance. Appendix A.3 contains more details.
5.2 MAIN RESULTS
We first compare IPROX with off-the-shelf proxies, with the results summarized in Table 2. We vary so that proxy sizes range from off-the-shelf scale to near the target model. The key findings are:
IPROX is effective across different models. IPROX consistently outperforms the off-the-shelf proxies across all sparsity levels on BBH and TyDiQA, while also achieving competitive results on MMLU, demonstrating the effectiveness of our approach. Notably, on Qwen3, IPROX even surpasses the larger 1.7B off-the-shelf proxy with a proxy of only 1.5B parameters.
Larger proxies yield better performance. Across all four model families, we observe a clear trend: increasing proxy size leads to improved performance. This highlights that our approach enables a controllable trade-off between computational cost and downstream performance.

---

Published as a conference paper at ICLR 2026
Table 2: IPROX compared with off-the-shelf proxies across four target model families. For each target model, we report results using the full model (shown in gray, provided only as a reference), an off-tthe-sheff proxy from the same family, and IPROX with different sparsity levels p. Bold and underline indicate the best and second-best proxy results, respectively.
Target Model
Proxy Model
#Params
MMLU
BBH
TyDiQA
Avg.
Llama3.2-38
56.28
47.78
43.10
49.05
LJama3.2-1B
IB
55.89
47.31
38.84
47.35
[PROX, p 0.3
2.5B
56.77
49.16
40.98
48.97
Llama3.2-3B
IPROX. p 0.5
1.8B
56.35
47.69
39.77
47.94
IPROX, p
1.3B
56.28
47.31
39.04
47.54
Gemma3-41
4B
$9.67
47.68
28.14
45.16
Gemma3-1B
1B
59.61
47.31
25.43
44.12
IPROX, 0.3
3B
59.36
49.63
32.19
47.06
Gemma3-4B
IPROX, p 0.5
2.3B
59.47
48.70
31.42
46.53
[PROX, p 0.7
1.6B
59.32
48.52
29.12
45.65
Qwen3-4B
4B
69.90
74.62
49.56
64.69
Qwen3-1.7B
1.7B
69.65
74.44
47.35
63.81
IPROX. p
3.1B
70.15
75.18
50.63
65.32
Qwen3-4B
IPROX. p 0.5
2.2B
70.08
74.72
48.45
64,42
[PROX, p= 0.7
1.5B
69.94
74.62
47.98
64.18
Qwen2-7B
7B
70.35
61.85
51.46
61.22
Swen2-1.5BB IPROX, 0 0.3
1.5B
70.18
59.72
47.29
59.06
5.8B
70.36
60.93
53.56
61.62
Qwen2-7B
IPROX, P 0.5
4.4B
70.27
60.74
51.36
60.79
IPROX, p = 0.7
3.3B
70.41
60.28
50.61
60.43
Task type matters. We find that the benefits of IPROX vary across tasks. The performance gains are more pronounced on TyDiQA than on MMLU. We argue that this difference may stem from the nature of the tasks, since TyDiQA and Dolly are both closer open- ween-domiin QA settings, whereas MMLU emphasizes complex reasoning tasks where data selected from Dolly provides only limited improvements. This observation aligns with Eq. 4, which indicates that greater distributional shift between training and validation sets results in a looser error bound.
outperform target models. In some cases, IPROX surpasses the performance obtained with data selected by the target model itself, such as Qwen3-4B with = 0.3 on BBH and Qwen2-7B with p 0.3 TyDiQA. This phenomenon, where smaller models identify more generalizable training data, has also been reported in prior work across pre-training (Xie et al., 2023; Engstrom, 2024), fine-tuning (Xia al., 2024), and in-context learning (Wang al., 2023a). Our experiments reinforce this observation, showing that sometimes proxies can select data for the target model more effectively than the target model itself.
Proxies
Table 3: Comparison of IPROX with two baselines: Layer Extraction and Influence Scorer. For IPROX and Layer Extraction, report the results based on p 0.3. A denotes the performance gain of IPROX over the strongest baseline. Bold indicates the best results.
Llama3.2-3B
Gemma3-4B
Task
Layer Exuraction
Inffuence Scorer
IPROX
A
Layer Extraction
Influence Scorer
IPROX
A
MMLU
56.44
56.42
56.77
0.33
59.30
59.49
59.36
-0.13
BBH
46.85
46.57
49.16
2.31
48.79
47.87
49.63
0.84
TyDiQA
38.18
34.11
40.98
5,80
26.91
26.91
32.19
5.28
Avg. 46.16
45.70
48.97
2.81
45.00
44.76
47.06
1.99
Next, we compare IPROX with two baselines. As shown in Table 3. IPROX achieves overall stronger performance than both baselines, with an average improvement 2.81% 1% on Llama3.2-3B and 1.99% on Gemma3-4B, We observe that while the two baselines obtain comparable or slightly higher results on MMLU, these improvements are less conclusive, since both methods perform notably worse than the off- the-shelf proxy BBH and TyDiQA. We also acknowledge that both baselines computationally cheaper, but they do not preserve gradient information and therefore struggle to identify useful data. Additional results other model families provided in Appendix C.
8

---

Published as a conference paper at ICLR 2026
5.3 ANALYSIS
Table 4: Evaluation results of IPROX on Influence Function. Bold and underline indicate the best and second- second-best results within each target group, respectively.
G4
Tusk
IPROX
IPROX
LJama33 2-38
2-18
p= 0.3
= 0.5
p= 0.7
Gemmu3-448
Gemma3-18
0.3
0.5
| 0.7
MMLU
56.31
56.10
56.09
55.96
56.52
59.50
59.18
59.37
59.57
59.34
BBH
48.43
46.20
48.24
47.96
47.31
49.54
45.09
48,98
48.52
48.15
TyDiQA
44.35
11.57
30.01
Avg. 48.87
46.81
49.56
48.50
47.63
47.17
44.76
47.51
47.34
45.31
Results on Influence Function. To validate the effectiveness
Stgge Stap2
Colsdtion
of IPROX across different gradient-based influence, also evaluate IPROX under the Influence Function. The results are reported in Table 4. We find that IPROX outperforms off-the- shelf proxies on BBH and TyDiQA while remaining competitive on MMLU, Averaged across tasks, IPROX achieves clear gains over the smaller proxies on both Llama3.2- 3.BBB and Gemma3-4B, leading conclusion consistent with Table 2. These results suggest that the improvements brought by IPROX are consistent across different gradient-based influences.
250
200
TFLORs
91.1
120
1406
0.3
03
6.7
Sparsity
Figure 4: TFLOPs breakdown on Llama3.2 2-3B across different spar- sity levels.
Efficiency Analysis. We further analyze the efficiency of IPROX by reporting both theoretical FLOPs and actual GPU hours. Figure 4 shows the FLOPss breakdown on Llama3.2-3) across different sparsity levels. As sparsity increases, the total FLOOPs drop substantially, leading to 0.7 compared to the full 3B model. Moreover, Stage I and Stage
over 140 TFLOPs savings at p 000.7 account for only a small portion of the total FLOPs, and their cost further decreases as sparsity grows.
Table 5 reports the estimated wall-clock com- putation measured on a single GH200 GPU, with IPROX ranging across all sparsity levels from 0.3 to 0.7. Compared to ~99 minutes required for influence calculation with the 3B model and <40 minutes with the 1B off-the- shelf proxy, our method performs influence cal- culation in only ~38-44 minutes. Proxy con- struction (Stage and Stage 2) adds less than 10 minutes of extra cost, bringing the total runtime to about 43-51 minutes. Thus, the ef- ficiency of IPROX mainly comes from the reduced cost of influence calculation, with proxy construction contributing only small computational overhead. Together, these results high- light that IPROX achieves notable efficiency improvements while maintaining strong perfor- mance, making it practical alternative to direct influence calculation with target models.
Table 5: Computation breakdown on Llama3.2- measured in single GH200 GPU hours. Infl. Cale. denotes the time for influence calculation.
Model
Stage
Stage
Infl. Calc.
Llama3.2-3 2-33B
90 mins
Llama3.2-1B
40 mins
IPROX ~2 mins 335 mins 33844 mins
Behind IPROX Effectiveness. To understand why IPROX is effective in data selection, we first examine the similarity between the selected data and the target task using subspace affinity (SA) (Soltanolkotabi et al., 2014). As shown in Table 6, proxies with lower sparsity (e.g., p 0.3) achieve higher SA than the off-the-shelf proxy, most notably on TyDiQA, suggesting that they capture gradient directions more consistent with the target task.
Table 6: Similarity and diversity of selected subsets with the target model Llama3.2-33 2-BB. SA measures subspace aligmment with the target task (higher is better), and 1-NND measures average nearest- neighbor distance within the selected dataset for diversity (higher is better). Bold and underline indicate the best and second- proxy results.
Proxy
MMLU
TyDiQA
SAT
L-NND
SM1
L-NND
SAT L-NND1
Llama3.2-18
29.01
13.91
20.94
13.29
18.61
13.13
Beyond similarity, diversity also plays a key role in boosting downstream performance (Zhang et al., 2024). Therefore, we measure the average nearest-neighhor distance (1-NND) within selected subsets measurement for diversity and find
IPROX, p
14.04
21.78
15.67
IPROX, 0.5 33.14
14.31
21.32
15.45
20.59
16.17
IPROX, p = 0.7 32.19
16.07
21.32
15.63
19.72
15.82

---

Published as a conference paper at ICLR 2026
that proxies with higher sparsity (e.g., P 0.7) yield larger 1-NND values than the 1B proxy. This suggests that even when compressed, IPROX preserve sufficient degree of diversity in their selections. We argue that IPROX steers selection toward task-relevant directions while its sparsity allows variation in less dominant components, which helps maintain diversity in the selected data.
Probe Set Quality. We investigate the impact of the probe set configuration on the perfor- mance of IProX, specifically focusing on the trade-offs regarding probe set size and data di- versity. All experiments this section are con- ducted on Llama3.24 .2-BB with sparsity ratio of == 0.7.
Table 7: Impact of Probe Set Size. Increasing probe size yields diminishing returns while signifi- cantly increasing computational overhead.
Probe Size
MMLU
BBH
TyDiQA
Avg.
0.5x
56.12
46.85
37.71
46.89
Default
56.28
47.31
39.04
47,54
We first analyze the sensitivity of proxy perfor-
3x
56.26
47.41
39.89
47.85
mance to the size of the probe set N. As shown
5x
56.41
47.50
38.76
47.55
in Table 7, there is clear trade-off between marginal performance gains and computational efficiency. While increasing the probe size to 3x the default value yields performance improvements, these gains saturate around 3x. Notably, further increasing the size to 5x results in diminishing returns, From an efficiency perspective. IPSVD benefits from small N enable fast "skinny SVDs" (Appendix F), and the Stage cost scales roughly linearly with N. Overall, the saturation in performance gains and the increasing Stage cost justify our choice of a moderate default probe size.
To assess the role of probe-set diversity, we simulated low-diversity scenarios by re- placing 10%-30% of the probe set samples with SMOTE-based interpolation, while strictly keeping the total size N fixed. The results in Table 8 demonstrate that perfor- mance degrades consistently as diversity decreases. This confirms that IProX ben- efits significantly from the high diversity naturally provided by our random data and uniform token sampling strategy.
Table 8: Impact of Probe Set Diversity. Reducing diver- sity (via redundancy injection) while keeping size fixed leads to performance degradation.
Diversity Setting
MMLU
BBH
TyDiQA
Avg.
Default (Random)
56.28
47.31
39.04
47.54
10% redundancy
56.28
47.22
38.76
47.42
20% redundancy
56.15
46.76
38.40
47.10
30% redundancy
56.12
45.65
37.67
46.48
Ablation Studies. Table presents an ablation study on different components. We observe that removing KL an- choring consistently reduces performance across all three benchmarks, while removing the entire aligning stage leads to even larger drops, particularly on TyDiQA. The degradation more pronounced at higher sparsity levels, suggesting that alignment becomes increasingly important as the proxy is more aggressively compressed. Overall, the results show that KL anchoring and gradient alignment are complementary. KL anchoring stabilizes training by constraining outputs, while gradient alignment preserves influence-relevan directions, and together they maintain the quality of selected data.
Table 9: Ablation study on Llama3. 3.BBBB. Removing KL anchoring or the entire aligning stage leads to consistent drops performance across all tasks.
Model
MMLU
BBH
TyDiQA
IPROX, 0.3
56.77
49.16
40.98
wlo KL anchoring
56.52
48.88
40.85
weo alignment
56.41
48.51
39.33
IPROX, 0.5 wlo KL. anchoring we aligmment
56.35
47.69
39.77
56.19
47.59
39.04
47.03
IPROX, == 0.7 56.24
47.31
39.79
anchoring wo alignment
46.85
37.66
55.99
46.48
35.32
CONCLUSION
We introduced IPROX, principled framework for constructing influence- preeseervieeeeiinngg proxies for efficient data selection in LLM fine-tuning. By compressing the target model with an influence- preserving low-rank approximation and refining it through model gradient and output alignment, IPROX preserves the influence information of the target model while reducing computational cost. Experiments across multiple model families and tasks show consistent gains over off-the-shelf proxies and baselines, together with clear efficiency benefits. These results suggest that reffunnce-peseevvve proxies offer scalable approach to gradient- gradient-based data selection LLM fine-tuning.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by National Science Foundation under Award No. IIS-2117902. The views and conclusions are those of the authors and should not interpreted representing the official policies of the funding agencies or the government. This research used the DeltaAI advanced computing and data resource, which supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAl is joint effort of the University of Illinois its National Center for Supercomputing Applications. This work used the DeltaAl system at the National Center for Supercomputing Applications through allocation CIS250246 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
and
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study focuses on methodological advances in efficient data selection for LLM fine-tuning. All experiments are conducted on publicly available datasets with open-sourced models. We do not involve human subjects, private or sensitive informa- tion, nor do we release new datasets. The proposed method is designed to reduce computational costs for gradient-based data selection and does not introduce foreseeable risks of harm, privacy violation, or discrimination. We have carefully documented implementation details to promote transparency and avoid risks of misuse. Overall, we view our work as having positive impact by encouraging efficiency and responsible use of computational resources.
REPRODUCIBILITY STATEMENT
We make substantial efforts to ensure reproducibility. Theoretical results are presented with complete assumptions and proofs (see Appendix D and Appendix E). Details of the proposed method, including the inffuence-preserving compression and alignment stages, are fully described in Section 4 and Appendix F, with algorithmic formulations provided. Comprehensive experimental setups, datasets, and evaluation metrics are specified in Section 5 and Appendix A.1. All datasets and models employed in this paper are publicly available. The source code will be released via an anonymized link: https:://///. /i thub. com/ cww.://rrrooooo/
REFERENCES
Mengting Ai, Sirui Chen, Maggie Zarekarizi, Kommy Weldemariam, and Jingrui He. OA- SIS: An optimized approach to systematic calibration data selection, 2025a. URL https:
//openreview.c hett:/ormmiidnnnuXYYYV.
Mengting Ai, Tianxin Wei, Sirui Chen, and Jingrui He. Nirvana: Structured pruning reimagined for large language models compression, 2025b. URL https: httpss//aaavvveeee abs abs/2509. 2509. 14230.
Mengting Ai, Tianxin Wei, Yifan Chen, Zhichen Zeng, Ritchie Zhao, Girish Varatkar, Bita Darvish Rouhani, Xianfeng Tang, Hanghang Tong, and Jingrui He. Resmoe: Space-efficient compression of mixture of experts Ilms via residual restoration. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD '25, pp. 1-12. ACM, July 2025c. doi: 10. 10.1145/3660024.37000 3709196,
URL http:htt////e httttp://xxxdooo doi org / 11 11445.666444
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger B Grosse. Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems, 37:66647-66686. 2024.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 454-470, 2020.
11

---

Published as a conference paper at ICLR 2026
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instructiontumed 1lm. 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 Q) 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 30318-30332, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- frika, 133):111-218, 1936.
Logan Engstrom. Dsdm: Model-aware dataset selection with datamodels. In Forty-first International Conference Machine Learning, 2024.
Spencer Frei, Niladri S Chatterji, and Peter I. Bartlett. Random feature amplification: Feature learning and generalization in neural networks. Journal of Machine Learning Research, 24(303):1-49 2023.
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang. Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. Compressing large-scale transformer- models: A case study on bert. Transactions of the Association for Computational Linguistics, 9:1061-1080, 2021. ISSN 2307-387X. doi: 10.1 162/taccc__pp a0411. URL http: /dx dow. _acl_a_0044]
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learming, pp. 573-582. PMLR, 2016.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 230.00396, 2023.
Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. arXiv preprint arXiv:2410.07064 2024.
Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12660-12673, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXh::2207.001 12, 2022.
Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in 1lm.. The Twelfth International Conference on Learning Representations, 2024a.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Automatic prediction of compute-optimal data compositions for training 1lms. 2024b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001 (00000 2020.
12

---

Published as a conference paper at ICLR 2026
Yegor Klochkov and Yang Liu. Revisiting inverse hessian vector products for calculating influence functions. arXiv preprint arXiv:2409. 17357, 2024.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Wei Li, Lujun Li, Hao Gu, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Moe-SVD: Structured mixture-of -f-peessss LLMs compression via singular value de- composition. In Forty-second International Conference on Machine Learning, 2025. URL net/ Eorum? boruumiddaccJvdd111
Chi-Heng Lin, Shanggian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression, 2025a. URL httph:tt///// arxiv. org/ wwtg/ab///a /tab////48 09632.
Haowei Lin, Yacine Jernite, HT Kung, and Andrew Gordon Wilson. Evosld: Automated neural scaling law discovery with large language models. Preprint, 2025b.
Xiaoqiang Lin, Xinyi Xu, See-Kiong Ng. and Bryan Kian Hsiang Low. Efficient top-m data values identification for data selection. The Thirleenth International Conference on Learning Representations, 2025c.
Zifan Liu, Amin Karbasi, and Theodoros Rekatsinas. Tsds: Data selection for task-specific model finetuning, arXiv preprint urXiv:2410.11300 2024.
James Martens and Roger Grosse. Optimizing neural networks with frattcker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https: /atpiv.ivvvv.////ia//...///..avv www.oo/ggaooo.o. abs /a.pbs//4440 240/244333.00 03853.
Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, and Frederic Sala. The cost of compression: Investigating the impact of compression parametrie knowledge in language models. arXiv preprint arXiv:23/2.00966. 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Yunzhe Qi, Sirui Chen, Jiaru Zou, and Jingrui He. Auto-regressive in-context demonstration selection, 2026. URL https:ttt.........t. hwwp//veevvveeeeeeee ew. net 1ttpmmmt99000222090001111
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition, 2024. URL https: // //arxivv org//oorr/ooo abs/ 2405.445..00 18886.
Kashun Shum, Yuzhen Huang, Hongjian Zou, Qi Ding, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts is the data that teaches. arXiv preprint 1rXiv:2503.00808 2025.
Mahdi Soltanolkotabi, Ehsan Elhamifar, and Emmanuel J Candes. Robust subspace clustering. 2014.
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving data efficiency for Im reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. arXiv preprint arXiv:2506.053: 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le. Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain-of-thouuht can solve them. arXiv preprint 10.0022210.09266, 2022.
13

---

Published as a conference paper at ICLR 2026
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19785 2025.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.16671, 2024.
Hieu Tran, Zhichao Yang. Zonghai Yao, and Hong Yu. Bioinstruct: instruction tuning of large language models for biomedical natural language processing. Journal of the American Medical Informatics Association, 31(9):1221-1832 2024.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Andrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae, Sheila Mcllraith, and Roger Grosse. Better training data attribution via better inverse hessian-vector products. arXiv preprint 2eXi:25577.14400 2025.
Renxi Wang, Haonan Li, Minghao Wu, Yuxia Wang, Xudong Han, Chiyu Zhang, and Timothy Baldwin. Demystifying instruction mixing for fine-tuning large language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 4: Student Research Workshop). pp. 68 68-75, 2024a.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation- singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, 2024b.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36: 15614-15638, 2023a.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. Advances in Neural Information Processing Systems, 36:74764-74786 2023b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thouuht prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, pp. 24824-24837, 2022.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pp. 1954-19999 1963. PMLR, 2015.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference Machine Learning, pp. 54104-54132, 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798 668188 18, 2023.
Tong Xie, Haoyu Li, Andrew Bai, and Cho-Jui Hsieh. Data attribution for diffusion models: Timestep- induced bias in influence estimation. arXiv preprint arXiv:2401.0903, 2024.
Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma, Yao Shu, Jingtan Wang, Xinyuan Niu, Zhenfeng He, Jiangwei Chen, Zijian Zhou, et al. Position paper: Data-centric ai in the age of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1 11895- 11913, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint Siv:2505.00938. 2025.
14

---

Published as a conference paper at ICLR 2026
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang. Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i²: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition, 2024a. URL
org/abs/
2411. 2211.00016.
Yu Yang. Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. Smaltolaage (s2l); Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: ppwerevvewweweeeeooe/pppee net / forum? d=k GGGM@pif f.
Zichun Yu, Spandan Das, and Chenyan Xiong. MATES: Model-aware data selection for efficient pretraining with data influence models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https:/ http:::///peeeeview... ntw.forumttttommtteeemmem idm 6gzPSMUAZ2.
Zichun Yu, Fei Peng, Jie Lei, Arnold Overwijk, Wen-tau Yih, and Chenyan Xiong. Group-level data selection for efficient pretraining. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models, 2025. URL https:
wwg/aas////a/.ooooooo/.....aaaaa /2322 05821.
Xinyue Zeng, Haohui Wang. Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Unveiling fine-tuning dynamics for Ilm selection. In Forty-second International Conference on Machine Learning.
Bolin Zhang, Jiahao Wang, Qianlong Du, Jiajun Zhang, Zhiying Tu, and Dianhui Chu. A survey on data selection for IIm instruction tuning. Journal of Artificial Intelligence Research, 83, 2025.
Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, al. Hamesssin diversity for important data selection pretraining large language models. arXiv preprint arXiv:2409. 16986, 2024.
Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, and Jingrui He. Transformer copilot: Learning from the mistake log Ilm fine-tuning, arXiv preprint arXiv:2505.16270 2025a.
Jiaru Zou, Ling Yang, Yunzhe Qi. Sirui Chen, Mengting Ai, Ke Shen, Jingrui He, and Mengdi Wang. Autotool: Dynamic tool selection and integration for agentic reasoning. arXiv preprint arXiv:2512.13227,, 2025b.
Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, et al. Latent collaboration in multi-agent systems. arXiv preprint arXiv:25/ 1.20639, 2025c.
15

---

Published as a conference paper at ICLR 2026
Appendix
APPENDIX CONTENTS
A Further Details on Experiment Setup
17
A.1 Baseline and Evaluation Details
17
A.2 Data Selection Setting Details
17
A.3 Implementation Details
17
B Further Discussion
19
Additional Experiment Results
20
Proof of Proposition 4.1
23
E Inftuence-Preservingg Low-Rank Approximation for Influence Functions
25
Efficient Implementation via Probe-Based Approximation and Core SVD
28
16

---

Published as a conference paper at ICLR 2026
A FURTHER DETAILS ON EXPERIMENT SETUP
A. BASELINE AND EVALUATION DETAILS
Here, we provide additional implementation details of the baseline and detailed evaluation settings.
Layer Extraction: We extract layers from the warmed-up target models. Following Men et al. (2024), each block (i.e., attention MLP) is scored with an influence defined as:
LE = 1 Er h h4+1
h
where h; and hi+1 denote the hidden states before and after the i-th block, respectively. This score captures how much the representation changes across the block, with larger values indicating greater influence. For fair comparison, we set the sparsity to & 0.3 and select the top 70% of blocks ranked by their influence scores /LE. The influence computed using a 1 % random sample from Dusin. and the final results are reported in Table 3.
Prior work (Gu et al., 2024; Yu et al., 2024) formulates this task as a regression problem, where a smaller model is trained to predict influence scores from limited set of annotated data. Concretely, the target model is first used to compute influence a hold-out set, and these values are then used supervise the smaller model. Once trained, the smaller model is applied to generate influence scores for the entire dataset. This approach raises two concerns. First, it still requires influence computation with the original model to produce annotations. Second, the generalizability of the smaller model is uncertain, as data preferences may shift during training, necessitating repeated re-annotation and retraining for accuracy. In our implementation, we adopt the off-the-shelf model from Table as the backbone, attach a regression head, and freeze all other layers during training. For a fair comparison, once. The default learning rate is set to le 5, and we optimize using Adam with a weight decay of 2 for 5 epochs.
Influence Scorer:
use 1% of Dusin the hold-out set and perform training only
We follow Xia al. (2024) to evaluate the performance of the models the target tasks. For MMLU, we evaluate 5-shot accuracy on the test set averaged across 57 subtasks. For TyDiQA, we report 1-shot macro-averaged exact match across 9 languages under the gold-passage setting, where passage containing the reference answer is provided. For BBH, we measure the average 3-shot exact match across all tasks. All models are trained for 4 epochs with default learning rate of 1c 5, and we report the final performance.
A.2 DATA SELECTION SETTING DETAILS
For TracIn influence, the implementation follows Xia et al. (2024) with two key modifications. As the experiments are conducted with full fine-tuning rather than parameter-efficient fine fine-tuning (PEFT), the most consuminn gradient projection step is omitted. The averaged gradient on the validation set is computed and its cosine similarity with the gradient of each training sample is used as the final influence score, rather than Adam moments. In addition, due to computational budget constraints, we warm-up for only one epoch with a default learning rate of 1e and weight decay of 1
For Influence Functions, K-FAC (Grosse & Martens, 2016) is used to compute the inverse Hes- sian-vector product for each layer, and the resulting vectors are concatenated form the final representation. For computational efficiency, Hessian statistics are estimated using 1024 samples rather than the entire dataset.
A.3 IMPLEMENTATION DETAILS
We initialize IPROX using 1% of randomly sampled data from D allocating 10% the first stage and 90% to the second stage.
In the first stage, the number of collected second-moment moment matrices N ranges from 512 to 2048, depending on the model size. Rather than averaging over entire sequences to collect hs and 8s, sample tokens within each sequence, with the sampling budget precomputed ensure uniform
17

---

Published as a conference paper at ICLR 2026
coverage over the entire probe set. This design offers two advantages: (i) random stratified token sampling better captures local geometry across different positions, difficulty levels, and attention patterns; and (ii) it mitigates length bias. Since sequence lengths vary widely, per-sequence averaging tends to compress the internal diversity of long sequences while disproportionately amplifying diminishing short sequences. For numerical stability, add damping term of 10-3 when computing the SVD. To improve hardware efficiency, the rank of each layer is further aligned to multiple of 128, which facilitates optimal tensor core utilization during GPU computation.
For the second stage, we perform a grid search over the following hyperparameters: learning rates of 1e-5, 5e-5, and 1c-4, and λKL values of 0.1, 0.01, and 0.001. We use weight decay of 0.01, align only the decomposed layers while keeping all others (including biases) fixed, and set the batch size We use cos(.,) as the distance metric in Eq 6.
All experiments are conducted on compute nodes with ARM architecture and equipped with NVIDIA GH200.
18

---

Published as a conference paper at ICLR 2026
B FURTHER DISCUSSION
Data-centric Al has received increasing attention in recent years (Xu et al., 2024; Kang et al., b024a;b; Zhang et al., 2025), with data selection emerging central paradigm spanning many areas (Ai et al., 2025a; Zou et al., 2025b; Qi et al., 2026; Sun et al., 2025; Zou et al., 2025c). Among these approaches, gradient-based methods have become one of the dominant families due to their direct connection to training dynamics and downstream performance. Building on the discussion in the main text (e-g., $4.2), we further clarify how IPROX relates to prior work on efficient gradient-baser data selection (Kwon et al., 2024; Yu et al., 2024; Xia et al., 2024; Lin et al., 2025c; Yu et al., 2025).
At high level, these lines of work share the same core signal. They treat gradient geometry, such as gradient norms and gradient similarities, as a surrogate for example utility. However, they optimize different parts of the pipeline. Most efficient gradient based selection methods accelerate the selection procedure given access gradients. Typical strategies reduce the number of scoring steps or checkpoints, compute divendiiensional or partial gradients, amortize scoring across batches, algorithmic approximations to avoid expensive influence computations. Importantly, these approaches still rely on running the original target model to generate the underlying gradients, and mainly reduce the overhead of how gradients computed, represented, aggregated for scoring. In contrast, IPROX targets a complementary bottleneck by reducing the cost of producing gradients. replaces the full target model with an influence preserving proxy whose forward and backward passes are cheaper, while maintaining the influence relevant gradient signal that downstream selectors depend on. This makes IPROX orthogonal to algorithmic speedups in the selection rule and naturally composable with them. In practice, one can run a lightweight selector on top of a IPROX proxy and obtain multiplicative gains by reducing both the per gradient cost and the number or dimensionality of gradient evaluations required by the selector.
Despite these benefits, IPROX also has limitations that mirror the assumptions behind influence-basee approximations. Aggressive compression compound approximation errors across layers (Ai et al., 2025b). While our alignment stage mitigates this effect, there remains an inherent trade-off between proxy efficiency and the truthfulness of gradient signals used for selection. In addition, our proxy construction is intrinsically layer-wise, so in sufficiently deep networks, small layer-level mismatches may accumulate they propagate through the model. This effect may be more noticeable under stochastic fine-tuning dynamics, where noise in optimization can interact with approximation error and lead modest deviations in per example scores selection rankings.
19

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENT RESULTS
Table 10:
Additional evaluation results of IPROX on Influence Function. Bold and underline indicate the best and second-best results within each target group.
Qwen3-4B
Qwen2-7B
Task
Layer Extraction
Influence Sconer
IPROX
A
Layer Extraction
Influence Scorer
TPROX
A
MMLU
69.86
69.60
70.15
0.29
70.31
70.28
70.41
0.10
BBH
74.25
74.90
75.18
0.28
59.63
59.17
60.93
1.30
TyDiQA
46.78
46.52
50.63
3.85
44.15
45.72
53.56
7.84
Avg. 63.63
63.67
65.32
1.47
58.03
58.39
61.63
3.08
Additional Results Compared with Baseline Methods We extend the comparison of IPROX with baselines to two target models, Qwen3-4B and Qwen2-7B, under the TracIn influence. The results summarized in Table 10. We observe that the trends in Table 10 are consistent with those reported in Table 3, confirming that IPROX consistently outperforms the baselines across different target models and tasks. In particular, the gains on TyDiQA are especially notable, with IPROX improving by +3.85 on Qwen3-4B and +7.84 on Qwen2-7B compared to the strongest baseline. These improvements highlight that the prreeerriii design of IPROX is more effective at capturing task-relevant gradients than heuristic or predictive alternatives. Moreover, the consistency of the results across both medium-size and large-size models suggests that the advantages of IPROX generalize beyond single model family, further reinforcing its effectiveness and scalability for gradient-based data selection.
Table 11: Evaluation results of IPROX on different data budgets. Bold and underline indicate the best and second-bos results within each target group.
Data
Tusk
IPRoX
IPROX
Lloma3,2-388
Llama32--18
0=3.3
0.5
0.7
Llama3,2-388
LJamm3.2-18
0.3
0.5
0.7
MMLU
56:43
56.13
56.52
56.22
56.23
55.77
55.15
56.36
55.18
$5.66
BBH
46.85
46.67
48.33
47.31
47.41
47.13
45.83
47.47
46.20
46.11
TYDIQA
35.27
33.32
40.73
Avg.
45.88
45.06
47.21
46.27
45.65
47.88
45.84
47.34
46.29
46.13
Effect of Data Budgets. Table 11 reports the evaluation results of IPROX under two different data budgets, 1 1% and 20%. In both cases, IPROX consistently outperforms the off off-the-shelf 1B proxy, demonstrating its effectiveness regardless of the amount of data used for selection. However, we also find that the 20% budget leads noticeable degradation, particularly TyDiQA. This decline can be attributed to the inclusion of redundant or noisy samples at higher budgets, which dilutes the benefits of high-quality data and increases the risk of overfitting. Similar observations have been reported in prior work (Liu et al., 2024), further underscoring the importance of data selection.
Table 12: Ablation of IPSVD vs. standard SVD on Llama3.2-3B. "w/o IPSVD" replaces IPSVD with standard SVD; numbers parentheses denote the drop relative IPSVD.
Target Model
Proxy Model
MMLU
BBH
TyDiQA
Avg.
IPROX, = 0.3
56.77
49.16
40.98
48,97
w/o IPSVD
56.42 (-0.35)
46.94 (-2.22)
36.53 (-4.45)
46.63 (-2.34)
IPROX, p = 0.5
56.35
47.69
39.77
47.94
Llama3.2-3p
w/o IPSVD
56.11 (-0.24)
46.30 (-1.39)
34.50 (-5.27)
45.64 (-2,30)
IPROX, = 0.7
56.28
47.31
39.04
47.54
w/o IPSVD
55.97 (-0.31)
46.11 (-1.20)
32.73 (-6.31)
44.94 (-2.60)
To isolate the contribution of the second-moment reweighting in IPSVD, conduct ablation study where replace IPSVD with standard SVD while keeping all other components unchanged. As shown in Table 12, replacing IPSVD with standard SVD leads to consistent performance degradation across all sparsity levels and benchmarks. The average score
Ablation of IPSVD.
20

---

Published as a conference paper at ICLR 2026
drops by approximately to 3 points, with the most severe decline observed TyDiQA (up points). These results empirically confirm that standard SVD, which minimizes output reconstruction error, is insufficient for preserving gradient- gradient-based influence, thereby validating the necessity of the reweighting strategy employed in IPSVD.
Table 13: Performance on diverse candidate training data. IProX achieves competitive performance with the full model and outperforms the IB proxy, with optimal results 0.3.
Candidate Training Data
Proxy Model
MMLU
BBH
TyDiQA
Avg.
Llama3.2-33
56.53
48.61
47.90
51.01
Llama3.2-18
56.17
47.31
42.67
48.72
CoT
IProX, 0.3
56.96
48.80
48.72
51.49
IProX, = 0.5
56.48
48.06
46.73
50.42
IProX, = 0.7
56.26
47.60
43.18
49.01
Llama3.2 3-BB
56.61
47.22
38.96
47.60
Llama3.22 3-BBBB.BBBBBBB
55.93
47.04
33.94
45.64
BioInstruct
IProX, 0.3
56.25
48.15
39.17
47.86
IProX, 0.5
56.21
47.41
38.36
47.27
IProX, 0.7
56.09
47.13
36.48
46.56
Diverse Candidate Training Data. To further validate the robustness of IProX across distinct task
formats and domain shifts, we extend our evaluation to two additional training datasets: CoT (Wei al., 2022) and Biolnstruct (Tran al., 2024). For fair comparison, we keep the total size of the candidate training data fixed by randomly sampling the same number of samples.
Table 13 summarizes the performance of Llama3.2-3B proxies constructed via IProX compared to baselines. IProX consistently outperforms the off-the-shelf 1B proxy and remains competitive with the full 3B model on both new datasets. We also observe distinct behaviors arising from domain shifts. Training on BioInstruct leads noticeable degradation TyDiQA), likely due to the distribution shift towards specialized biomedical content. However, the performance drop BBH is mild, consistent with the partial overlap between Biolnstruct and the biomedical subsets within BBH. Conversely, training on CoT tends to improve performance across all benchmarks. Most notably, observe significant gains TyDiQA, suggesting that the reasoning-focusel supervision in CoT data transfers effectively to other complex tasks.
general benchmarks (MMLU,
Table 14: Performance under Extreme Compression (p 0.9). Even 90% sparsity, IProX consistently outperforms the Layer Extraction baseline. Gains shown in parentheses.
Target Model
Method
MMLU
BBH
TyDiQA
Avg.
IProX
56.17 (40.20)
46.57 (40.64)
37.26 (45.25)
46.67 (42.03)
Llama3.2-3B
Layer Extraction
55.97
45.93
32.01
44.64
IProX
70.25 (+0.21)
60.00 (+0.56)
48.67 (46.02)
59.64 (42.26)
Qwen2-7B
Layer Extraction
70.04
59,44
42.65
57.38
Performance under Extreme Compression We investigate the behavior of IProX under extreme compression scenarios (p = 0.9). While SVD-based approximations naturally face limitations in this regime due the significant reduction in rank, we aim determine if IProX retains utility compared to heuristic baselines.
Table 14 compares IProX against Layer Extraction on both Llama3.2 2-B3 and Qwen2-7B 90% sparsity. Although performance naturally degrades compared to lower sparsity settings, IProX consistently outperforms the Layer Extraction baseline across all metrics. The degradation is relatively mild, and the performance gap highlights that even in extreme regimes, method preserves influence information more effectively than simple heuristic alternatives.
21

---

Published as a conference paper at ICLR 2026
Table 15: Sensitivity of KL Coefficient (TKL). moderate coefficient (7KL 0.1) strikes the best balance between influence alignment and output stability. Experiments performed on Alama3.2-3B with p 0.7.
Target Model
Configuration
MMLU
BBH
TyDiQA
Avg.
Llama3.2-3B
IProX, 7KL. 0..
56.02
46.85
37.83
46.90
IProX, KKL 0.1
56.28
47.31
39.04
47.54
IProX, KKL 0.01
56.12
47.04
36.61
46.59
IProX, "KL. 0.001
56.05
46.48
36.54
46.36
The Sensitivity of KL Coefficient We analyze the sensitivity of IProX to the KI. divergence coefficient (?KL)) used the alignment objective. The KI. term provides essential anchoring for stability, preventing the proxy from drifting too far from the target model's s output distribution.
Table 15 presents the results on Llama3.2-3B with a sparsity ratio of 0.7. We observe that performance degrades if 7KI. is set too high (0.5), as the distillation loss begins to overpower the influence alignment objective. Conversely, values that are too low (s 0.01) provide insufficient regularization, leading suboptimal retention of the target model's capabilities. Based on these findings, we adopt a moderate value of "KI." = 0.1 for our main experiments.
22

---

Published as a conference paper at ICLR 2026
D PROOF OF PROPOSITION 4.1
Here, we provide a complete proof for Proposition 4.1. We fix a layer l and perturbation Eg to its weight matrix We, such that the perturbed weight is We = We + Eg. The influence contribution of layer and the layer-local directional effect are defined as:
and ex(z) Eche(z), where (z) and (z) denotes the input and the upstream gradient at the layer We begin by stating the technical assumptions required for our result, which are similar to simplifying assumptions often adopted in theoretical studies of deep neural networks (Virmaux Scammn, 2018; Frei et al.,
Iw: (2, = ()<(=), F (he())) he(z)))
2023).
(A1) (Backward Smoothness). For almost every sample z, the map v -> 5g(2; u) is differentiable in a neighborhood of the unperturbed pre-activation up(z) Weh(z). There exists measur- able function (2) 2 0 such that the Jacobian D, u) satisfies (2; S (2)
uniformly for u along the line segment (ue(=) 7 FFg(2)): : [0, 1]}.
(A2) (Finite Second Moments). The expectations [ [=]]))(())))) 13, (2]), (2))( and [E] (='))( are all finite for an independent copy ='.
(A3) (Coherence of Local Directions). There exists constant 7) E (0, 1] such that for almost This implies the cosine of the angle between d(2) and Egh(z) is bounded away from zero.
every (())(2), Figh(()) 2 [[ll
(A4) (Bounded Covariate Shift). The distributions of and z' are such that there exists constant
k 2 0 satisfying By (e<(==))] S (TE,
With these assumptions in place, we can state the following proposition.
Proposition D.1. Under Assumptions (AI)-(44) for any perturbation Es, there exists a finite, data-dependent constant C₂ > O such that:
E Eg Lw, (2,2') Iwe(z, S C.VBBlle(2)7
Proof. Let Wi(r) = We + TE for r E [0, 1]. Define 2(T: & (2, The input M=)) does not depend on W,, so the dependence on 7 enters only through 0e(2;u)(T, where ue(T,2) We(r)h(z). We can represent the change in influence
W (2, Iw, (2,(),2) = -f, d'(r:2,2') dr.
Differentiating @ with respect to gives:
o' (TiZ,z) = To
d
ue((, he(()),
do(2; (6 2)), aे on(='s he(=))
By the chain rule and assumption (AI), we have:
dr be (((;))((TT))) = (et)
and its norm is bounded as:
d Se(2; 5e(2; < [(<))(Ech((2)))
dr
Using the triangle inequality, Cauchy-Schwarz, and (he()), he(=')) < [|he())]llbb((]]]]| we obtain pointwise bound on (0'(T:2,
1ø' (r; 2') & ((c))]l (1b,
[(x')) (dde)]]) (T))
23

---

Published as a conference paper at ICLR 2026
Taking the expectation over (2,2"), applying Fubini's theorem and Jensen's inequality the integral, and using assumption (A1) to replace the r-dependent norms with their suprema (denoted []6e(2)] for simplicity), we obtain
E E lw. Iwel E. [L E. [166 E.
+ E
By the independence of z and 2' and another application of Cauchy-Schward Schwarz, we introduce the finite constants
Mr E (1)(
Moat =- Ez [1666()))1)1lb((())]}
which are bounded by Assumption (A2). Hence,
Eg,z/ /w. -wwl < Mva E. (())) + Mu (())
(10)
Next, we use the coherence assumption (A3) to relate |[ to the scalar error ee(z) = ((()),
|[ghh())]l - 1 lee(=)] for a.e.
17 [|6e(2)]l'
Substituting this into equation 10 and applying Cauchy-Schyee Schwarz once more yields
Eg,y' Iw. Iwel S C [E + C' (B
where the finite constants C' and C" are given by
Mval
[x))) |]6{())112
E. Mr 7
C
Eg
C'
||o{(=!11
7
Now, we invoke the bounded covariate shift from Assumption (A4), which implies v le(2")) s
V E. [e( This allows us to bound the entire expression in terms of the expectation over
Ez,² Iw -ww, < CC
= (C [=)
By defining C₂ A C + C" VR. which a finite, data-dependent constant, we arrive at the desired result:
Eg,z' Iw, (2, Iw. (2,2 s C.vEEle((3)))
24

---

Published as a conference paper at ICLR 2026
E DFLLUECCE-RRESEVIING LOW-RANK APPROXIMATION FOR INFLUENCE FUNCTIONS
We now extend the analysis from the simplified TracIn score to Influence Functions (IF). IFs refine the influence measure by incorporating the inverse Hessian of the loss, which accounts for the local curvature the optimization landscape. The influence of a training sample 2 on a validation smaple 2' is defined as:
Tr =²) - V=I(=';0) ((0)- (
To analyze the contribution of single weight matrix We at layer É, we consider its vectorized form wy Å (e). The gradient of the loss with respect to these vectorized parameters the outer product of the upstream gradients 6z(2) and the inputs he(2). Using the identity vec(ab b @ this gradient is:
wy [(z 0) (
he(2) @ 6e(z).
Following previous works (Martens Grosse, 2015; Grosse et al., 2023), we make key simplifying assumptions about the Hessian's structure. We assume the full Hessian matrix is block-diagonal with each block corresponding to the parameters of a single layer, and that within each layer l, the inputs hx(2) independent of the upstream gradients (z).
These assumptions allow define tractable surrogate Hessian He for layer & as the expected outer product of its vectorized gradients:
He e E, (()),
Ex [(ee(z) @ (elz)) @ 8e(=))
E. [l<(=)he(z) @ (())((=))
(E.lh(())h(()"]] ())))))))))) & Ch, @ Cs.e-
Here, Ch.t and C&.t are the second moment matrices of the activations and upstream gradients for layer respectively. Leveraging the property that (A B) A @ B-1, the inverse is given by HP Cr, @ Cz.-- The contribution of layer to the influence then defined as:
JI,,,, (2, A - -V I(=')) Hi (V we I(=))
= (he(=') @ de(x')) (cr. 8 Czdl (he(=) @ de(z))
= (hele')Tondhhle)) ordhe(z)) (6e(2')T(7,
where he = O 11/2 he and 6, = Cz,e 0-1/2 are reweighting matrices.
An Objective for Preserving Influence Functions To preserve the influences under low-rank approximation, we penalize the compression error using a norm aligned with the reweighting induced by Cn,t and Cs.e- We assume that Cr, and C& are symmetric positive definite. The objective is to find an error matrix Eg = Wt W; that minimizes the following term:
min
|| C
1/2
1/2 (We We) CT,
1/2
(11)
F
W. st. rank We
We now demonstrate that minimizing this objective effectively controls the expected change in influence score. Our theoretical guarantees rely the following assumptions.
(B1) (Finite moments). (=]]) [[]e(>)]]] and E., (1)) [>e(('))]] are finite.
(B2) (Backward smoothness). Let & denote the upstream gradient under We. There exists a measurable function K(.) 2 0 such that [x)r[ < K(2) [E]les where Ade(z) e d<(z) ōz(=), and (K( [=)(])] (=( are finite.
25

---

Published as a conference paper at ICLR 2026
(B3) (Quadratic remainder). There exists p such that for all We with
||0 (We Wo) C 11 P:
the Taylor remainder R(2,, 2') in the perturbation of /IF,,, satisfies
E (1,2, 2/]1 S Crem 1/2 F, 1/
Proposition E.1. Let We be perturbed to We We - Eg. Under (B1)-(B3), there exists a finite, data-depent deeeadeenndntt constant Cx > 0 such that
Ez,z 11, (z, -1F, (2, & CR | 1/2 Ea 1aCT1/211
(12)
Proof. Recall that the layer-l influence is given by JF.w (z,
(1e(2"),h<(z))) (ōc(z"), (((=)).
Let Ade(z) & (6))) 5e(=)) denote the change in the reweighted upstream gradient. The total change in influence consists of first-order Taylor expansion term, AT " (z, 2'), and remainder term R(z,2²) The first-order term is:
A (2,z))= = (=)) (he()))) (ō<(==), (d)( (fe(="), he(=))
By taking the expectation over 2, 2', applying the triangle and Cauchy-Sohhe Schwarz inequalities, and using the independence of and 2', we can bound the expected first-order change:
Eg - 1 < My E. (1 (1) + Mval [=[[ [=)e(,
(=)
where Mrr = E. (=)]l [[][())]]] and Myal E., (=') [[ae(=)11 are finite by Assumption (BI). Our main task is to bound the expectation (=( [[ae(=)]] in terms of the objective function. Let E. & Cz.
1/2 E,CT,
1/2 We first establish pointwise bound on [x)]( using
Assumption (B2).
| 8, (1 Ad((2))l <
S K(=)IIC7) 1/2
122 |e
Next, we relate |E||| to |E From the definition of E,, we have E, C/, E,C E ||
E. Ir 1 16
Recall that Ch.e and Cs,e are all symmetric and positive definite, combining these inequalities yields a pointwise bound on [ (z)Il terms of [Eel|::
(=)]l)) K(=)]IC7,)/ 112 (IC) 1/2
8. |2 Eel
KK(=) (ICTT 1/2 11 |2 la) || ||
= K())) cCnd((Ca.e) (mnx(Cr,e) [ell
where cond(.) and X-mxx-) denote the condition number and maximum eigenvalue, respectively. Let us define the data-dependent scaling constant S 2 /cond(Cs,e) (Ca, (Cr.e). We now use this result to bound the expectation term:
(=)I < E, K(:)S| 11
S E, E. [K( -1
By Assumption (B2), the expectations Ktr & ( [e(())] and Kval A (=(((())) are finite. The bound on the expected first-order change becomes:
(1) < S (Midival Moalttr) [e]
26

---

Published as a conference paper at ICLR 2026
The total expected change is bounded by the sum of the first-order term and the remainder from Assumption (B3):
E I W,, (2,2²) IIF, wi (z, 2')1 EE [] + E [[]].
Using Assumption (B3), for perturbations satisfying Ée]l S p. we have Eg,e' of [|]] S Crem Eell S Grem pl Eel Combining the terms gives the final result:
E Ez, Ar. S (M + Mvaltitr) + Cramp) || IF.
This proves the proposition with the constant C₂ - S(M Myal + Cremp, which is finite and depends on data properties but not on the specific perturbation Et.
27

---

Published as a conference paper at ICLR 2026
F EFFICIENT IMPLEMENTATION VIA PROBE-BASED APPROXIMATION AND CORE SVD
The theoretical solution presented in the main text requires computing, inverting, and taking the square root of the second moment matrices Ch. E T Xxt and Ca.e E mR For modern neural networks, the dimensions nę and my can be in the thousands for typical transformer layers, and can even reach the millions in domains like high-resolution computer vision or for layers tied to large vocabularies. This makes the direct formation and manipulation of these matrices computationally infeasible due to both memory and time constraints. To overcome this, we employ a memory memory-efficient approximation scheme that avoids forming these large matrices entirely.
The core strategy is to approximate the true second moment matrices using statistics gathered from small, representative batch of N data samples, which we refer to as probe dataset. We perform single forward and backward pass through the model for these N samples to collect the corresponding inputs and upstream gradients for each layer L. These are stacked column- to form two tall-and- aninthin probe matrices:
H₂ (he-1(
RE and A. =
RT
With these probe matrices, we can approximate the full second moment matrices as Cr. & & Hell and Cs.t ~ A Instead of computing these prohibitively large second moment matrices, the key insight directly compute the "skinny" Singular Value Decompositions of the much smaller probe matrices:
H. = UH.
and
A. = UU-
where UH. R ΣHE E RNXN, VH, E RNNN and similarly for the decomposition of A, This decomposition is the key to bypassing the expensive computations, as we can express the regularized square roots of the approximate second moment matrices without ever forming the full matrices. For example, for Cr,e, we have (Ch,e + X1) x Hell, + A1)1/2 HH, + 2/) UH.e( EH + AI) H.e- We then define the small, diagonal
matrices that hold the regularized singular values:
DH. & (
1/2
and Da (H + + +1)
1/2
(13)
The required reweighting transformations are thus efficiently represented as C. = UHHDHH. He and &.L. ~ UUA.D,,UX.P Substituting these efficient representations into the definition of the data-aware matrix S = C/2 - W&C reveals the final computational trick:
Se = Us.
Mom.
This shows that the SVD of the very large matrix S₁ is directly related to the SVD of the small core matrix Morc, e. which has dimensions at most N N. We compute the SVD of this core matrix, Moore, = QT and truncate to the desired rank ra by selecting the top ry columns Pe and Qi (denoted Pt. Qe.r) and the top-left Tę re block of Σg (denoted Σe,r). The optimal low-rank approximation Wi = A BJ is constructed by transforming the truncated SVD of the core matrix back into the original weight space. This yields numerically stable, closed-form expressions for the low-rank matrices A and B without ever forming the full W; matrix:
Aス = Ua Pr. P
1/2 and B = = DT
(14)
All computationally intensive steps now performed on matrices whose dimensions are related to the small probe set size N, not the target model dimensions ny and my. This entire procedure is highly efficient, assuming N < min(ne, me) and r N. For each layer, the complexity is composed of a single forward and backward pass for N samples, two skinny SVDs of the probe matrices with complexity (2) and N2,, the formation of the matrix which costs O(mnneN), an
28

---

Published as a conference paper at ICLR 2026
SVD of the small core matrix with complexity O(N*), and the final factor construction which costs (rr) for A and One) for By. The dominant costs are the core matrix formation and the skinny SVDs, which is dramatic reduction from the O(min(me, n₂)²) complexity required for the SVD of the full second moment matrices.
29


Reviews:
[{"review_id": "BIHNdxyOTf", "reviewer": "Reviewer_fpV6", "summary": "IPROX addresses the computational cost of gradient-based data selection for LLM fine-tuning by constructing smaller proxy models that preserve the target model's influence characteristics. The framework uses two stages: (1) Influence-Preserving SVD (IPSVD) that compresses the target model while retaining influence-relevant components via second-moment reweighting, and (2) alignment that matches gradients in low-rank space and anchors output logits.", "questions": "How does IPROX compare to non-gradient baselines like GLISTER, LESS, or coreset methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. **Flexible efficiency-performance trade-off**: Enables controllable proxy size through sparsity parameter ρ, allowing practitioners to balance computational cost and selection quality.\n\n2. **Practical scalability with minimal overhead**: Achieves >50% cost reduction on Llama3.2 with proxy construction adding only ~5-7 minutes overhead, using efficient probe-based approximation that avoids forming large  matrices.", "weaknesses": "**Missing comparisons to established data selection methods**: The paper lacks empirical comparison with core-set and generalization-based selection methods like GLISTER[1], Coresets, and gradient-based selection frameworks. These are only briefly mentioned in related work but not evaluated. The absence of comparison to methods like LESS[3] and DsDm[2] weakens the argument for IPROX's advantage over the broader data selection literature beyond just gradient-based influence methods.\n\n\n\n\n[1] GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning\n[2] DsDm: Model-Aware Dataset Selection with Datamodels\n[3] Less: Selecting influential data for targeted instruction tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MASkvaYybK", "reviewer": "Reviewer_PeGn", "summary": "This paper addresses the computational inefficiency of gradient-based data selection methods for large language models (LLMs), which are critical for SFT but impractical for multi-billion-parameter models due to high costs. It introduces IPROX, a two-stage framework that constructs influence-preserving proxy models directly from the target LLM: first, an Influence-Preserving SVD stage compresses the target model’s weight matrices to retain gradient-based influence information (unlike standard SVD, which prioritizes reconstruction loss), and second, an alignment stage refines the proxy by matching its gradients to the target model in low-rank space and anchoring output logits via KL divergence.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Targeted solution to an important problem**: The paper directly addresses the limitation of gradient-based data selection, which is the  poor scalability with LLM size. The authors focus on proxy design, which is an orthogonal approach to simplifying influence computation itself. This fills a gap in existing work, which either uses suboptimal off-the-shelf proxies or reduces influence calculation cost without preserving alignment to the target model.\n-  **Theoretically Grounded Proxy Construction**: IPROX is not heuristic: Proposition 4.1 provides a theoretical bound linking the expected squared error of layer perturbations to influence preservation, and IPSVD is designed to minimize this error via second-moment reweighting of inputs and upstream gradients.\n- **Flexibility and Efficiency**: IPROX allows flexible control of proxy size via the rank parameter in IPSVD, enabling trade-offs between computational cost and performance.", "weaknesses": "- **Dependence on Probe Set Quality**: IPROX relies on a small probe set to approximate second-moment matrices for IPSVD. The paper mentions using token sampling to avoid length bias but does not evaluate how probe set size, diversity, or representativeness affects performance.\n- **Narrow Task and Data Scope**: While the paper uses three diverse tasks, all training data is drawn from DOLLY (instruction-response pairs). It does not test IPROX on other data types (e.g., code, scientific text) or tasks with larger distributional shifts from DOLLY (e.g., low-resource language QA). This limits conclusions about IPROX’s performance in more heterogeneous fine-tuning scenarios.\n- **Lack of Ablation for Second-Moment Reweighting**: While IPSVD’s second-moment reweighting is core to its design, the paper does not ablate this component (e.g., comparing IPSVD to unweighted SVD) to isolate its impact on influence preservation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NojFn8k9s0", "reviewer": "Reviewer_npPH", "summary": "This paper introduces IPROX, a two-stage framework for constructing influence-preserving proxy models to enable scalable gradient-based data selection in the fine-tuning of LLMs. The approach first compresses the target LLM using a low-rank, influence-aware SVD (IPSVD) and then further aligns the proxy with the target via internal gradient alignment and output anchoring losses. Empirical results are presented across several LLM families, demonstrating improved performance and efficiency compared to existing off-the-shelf proxy models and two baseline approaches.", "questions": "- Is there empirical evidence on how sensitive the effectiveness of IPROX is to the choice and size of the probe set N? Can an ablation show how influence approximation degrades (or not) as N varies, especially in large model regimes?\n- Regarding the incompressible layers (embedding, LM head): Are there empirical ablation studies quantifying their effect on overall model or proxy performance?\n- Could the authors clarify the mathematical assumptions required for the influence preservation bounds, and discuss scenarios where these may not hold (e.g., high non-linearity, significant distribution shift)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Principled Influence-Preserving Compression**: The paper develops a well-justified influence-preserving low-rank SVD technique for model compression, addressing the misalignment between traditional reconstruction objectives and the needs of influence-based data selection.\n- **Comprehensive Theoretical and Implementation Details**: The paper carefully connects theoretical derivations to practical considerations, including efficient computation using “skinny” SVDs and probe-based approximations.", "weaknesses": "- **Theoretical Limitations and Empirical Edge Cases**: While the influence-retention bound is elegant, its assumptions (local smoothness, geometric coherence, bounded covariate shift) could be restrictive in practice. Furthermore, the treatment of the embedding and LM head as incompressible is not systematically evaluated in ablation.\n- **Performance Gains need more explanation**: The paper acknowledges task-type sensitivity but does not deliver deeper insight into when/why influence preservation matters most versus simple SVD or other alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xbw0w8tF4G", "reviewer": "Reviewer_U9aH", "summary": "This paper proposes IPROX, a framework for building smaller, influence-preserving proxy models to make gradient-based data selection feasible for large language models. Instead of using fixed off-the-shelf proxies, IPROX derives them directly from the target model through a two-stage process: an Influence-Preserving SVD that retains gradient-relevant components via second-moment reweighting, followed by a gradient-alignment stage that matches the proxy’s gradients and logits to the target model for stability. The authors provide theoretical justification that this compression preserves influence consistency and validate the method empirically across multiple model families (Llama3, Gemma3, Qwen2/3) and benchmarks (MMLU, BBH, TyDiQA). Results show that IPROX consistently outperforms larger off-the-shelf proxies while cutting computation by more than half, establishing a principled and efficient approach for scalable data selection in LLM fine-tuning.", "questions": "1. The paper primarily evaluates with TracIn and Influence Functions. How would IPROX behave with more recent estimators such as Fisher-based or curvature-aware influence methods? Is IPSVD theoretically compatible with these variants?\n2. What is the sensitivity of the extra hyperparameter (e.g., λ_KL, τ)?\n3. The experiments cap at 70 % sparsity. What happens under more aggressive compression (e.g., 80–90 %)—does influence preservation collapse gradually or sharply?\n4. The reported efficiency is impressive for 3B–7B targets. Could the authors provide projected or preliminary numbers for larger models, such as 70B, to demonstrate scalability under realistic industrial constraints?\n5. Two highly relevant recent works should be included: (a) LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection and (b) EvoSLD: Automated Neural Scaling Law Discovery With Large Language Models.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Presents a novel, well-motivated framework (IPROX) that bridges model compression and gradient-based influence estimation for LLM data selection.\n2. Offers strong theoretical grounding through Proposition 4.1, connecting layer perturbations to bounded influence deviation.\n3. Demonstrates consistent empirical improvements across multiple LLM families (Llama3, Gemma3, Qwen2/3) and tasks (MMLU, BBH, TyDiQA) and achieves significant computational efficiency, reducing cost by over 50% while maintaining or improving performance.", "weaknesses": "1. The theory relies on strong smoothness and bounded shift assumptions that may not fully hold for deep transformer landscapes.\n2. The alignment stage introduces extra hyperparameters (e.g., λ_KL, τ) that may require dataset-specific tuning.\n3. Limited exploration of extreme compression regimes (beyond 70% sparsity), where influence preservation may degrade.\n4. Experiments focus solely on text-based fine-tuning, leaving open how the method generalizes to multimodal or retrieval-augmented LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)