Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs
Authors: Jonas Hübotter
Sascha Bongni
Ido Hakimi
Andreas Krause
Venue: ICLR 2025 Poster (2025)
Keywords: data selection
retrieval
active learning
transductive active learning
local learning
test-time fine-tuning
transductive learning
language modeling
uncertainty quantification
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop SIFT, an effective data selection method for fine-tuning LLMs. We show that test-time fine-tuning with SIFT can significantly and robustly improve language modeling ability.

Abstract: Recent efforts in fine-tuning language models often rely on automatic data selection, commonly using Nearest Neighbors retrieval from large datasets.
However, we theoretically show that this approach tends to select redundant data, limiting its effectiveness or even hurting performance.
To address this, we introduce SIFT, a data selection algorithm designed to reduce uncertainty about the model's response given a prompt, which unifies ideas from retrieval and active learning.
Whereas Nearest Neighbor retrieval typically fails in the presence of information duplication, SIFT accounts for information duplication and optimizes the overall information gain of the selected examples.
We focus our evaluations on fine-tuning at test-time for prompt-specific language modeling on the Pile dataset, and show that SIFT consistently outperforms Nearest Neighbor retrieval, with minimal computational overhead.
Moreover, we show that our uncertainty estimates can predict the performance gain of test-time fine-tuning, and use this to develop an adaptive algorithm that invests test-time compute proportional to realized performance gains.
We provide the `activeft` (Active Fine-Tuning) library which can be used as a drop-in replacement for Nearest Neighbor retrieval.

Published as a conference paper at ICLR 2025
EFFICIENTLY LEARNING AT TEST-TIME: ACTIVE TUNING OF LLMS
Jonas Hübotter; Sascha Bongni, Ido Hakimi, Andreas Krause ETH Zürich, Switzerland
ABSTRACT
Recent efforts in fine-tuning language models often rely automatic data selection, commonly using Nearest Neighbors retrieval from large datasets. However, we theoretically show that this approach tends to select redundant data, limiting its effectiveness or even hurting performance. To address this, we introduce SIFT, a data selection algorithm designed reduce uncertainty about the model's response given a prompt, which unifies ideas from retrieval and active learning. Whereas Nearest Neighbor retrieval typically fails in the presence of information duplication, SIFT accounts for information duplication and optimizes the overall information gain of the selected examples. We focus our evaluations on fine-tuning at test-time for prompt-specific language modeling on the Pile dataset, and show that SIFT consistently outperforms Nearest Neighbor retrieval, with minimal computational overhead. Moreover, show that our uncertainty estimates can predict the performance gain of test-time fine-tuning, and use this develop adaptive algorithm that invests test-time compute proportional realized performance gains. We provide the act ivei (Active Fine- (unn-Tunigg) library which can be used as a drop-in replacement for Nearest Neighbor retrieval.
INTRODUCTION
The standard paradigm of machine learning separates training and testing. Training aims learn a model by inductively extracting general rules from data, and testing applies this model to new, unseen data. We investigate an alternative transductive paradigm where the model fine-tuned test-time specifically to the given task, Variations of this paradigm have been studied since the inception of machine learning as a field. Early examples local learning (Cleveland, 1979; Cleveland & Devlin, 1988; Atkeson et al., 1997) and local fine-tuning (Bottou & Vapnik, 1992). More recently, with the advent of large pre-trained models which have good representations and strong foundations for fine-tuning, the idea of test-time fine-tuning has re-gained attention (Krause al., 2018: 2019; Sun et al., 2020). Hardt & Sun (2024) show that fine-tuning on data related to the prompt to large language model (LLM) can significantly improve performance. Also, test-time fine-tuning is the central component state-of-the-ard approaches to the ARC chal- lenge (Chollet, 2019; Cole & Osman, 2023; Akyürek et al., 2024), a non-saturated benchmark which is intended to test reasoning capabilities based on "core knowledge" rather than mere memorization.
1.6
Bits per Byte (1 better)
Nearest Neighbor with duplication
1.4
1.2
Nearest Neighbor without duplication
1.0
1
SIFT (ours)
0.8
0
20
40
Test-Time Iterations
Figure 1: Selecting fine-tuning data using SIFT (red) robustly outperforms Nearest Neighbor retrieval (black) and avoids the failure-mode of Nearest Neighbor retrieval where the same data is selected repeatedly, which com- mon result of information duplication.
Active Fine-Tuning: Effective data selection for fine-tuning LLMs Test-time fine-tuning demands automatic data selection since manually selecting data for each test instance infeasible. Moreover, the sample efficiency of test-time fine-tuning is a central bottleneck the number of gradient steps is directly proportional to inference time. Previous works on data selection for fine-tuning LLMs have fundamentally relied on Nearest Neighbor retrieval within some embedding space (Hardt& & Sun, 2024; Xia et al., 2024), We show theoretically and empirically that Nearest
"Correspondence jonas. hwebotter@ www.eerriineeeeet... ethz..ch

---

Published as a conference paper at ICLR 2025
Neighbor retrieval is insufficient for fine-tuning LLMs since it can lead to the selection of redundant data. Notably, recent works using influence functions for data selection such as Xia et al. (2024) have pointed out this limitation. In contrast, large body of work on (inductive) active learning has studied non-redundant data selection (e.g.. Sener & Savarese, 2017; Ash et al., 2020; Yehuda al., 2021; Kirsch al., 2018) that covers the data manifold well (cf. Figure 2). Retrieval and active learning can be seen as two extreme ends of a spectrum: retrieval selects relevant but potentially redundant data, while active learning selects diverse but potentially irrelevant data.
We bridge this gap by unifying ideas from retrieval and active learning SIFT, an algorithm based on emerging literature on transductive active learning (Hübotter et al., 2024) that Selects Informative data for Fine-Tuie Tuning as illustrated in Figure 2. Our results show that SIFT leads substantial improvements in performance and efficiency. Concretely, we show the following:
Prompt
Data Manifold
Selected Data
1. Nearest Neighbor retrieval is insufficient ($2): We prove that selecting the top-N highest scoring points from large dataset according to fixed scoring function leads to the selection of redundant data.
Full Data Space
We consider scenario where we have pre- pre-trained language model capturing a latent manifold (red) in the large sequence space (white). We aim to improve the models performance on a given prompt (blue) by efficiently fine- tuning the model on few relevant and diverse data points (black) at test-time.
Figure 2:
SIFT estimates uncertainty about responses ($3): We develop the notion of uncertainty about the response to the prompt, and derive an anytime high probability bound to the total variation distance between the model's distribution over responses and the ground truth which governed by this uncertainty.
SIFT provably reduces uncertainty ($4): We propose SIFT, an algorithm that selects data which reduces uncertainty about the response to the prompt. We prove statistical rates for the uncertainty reduction ($4.1) and show that SIFT is compute-efficient with minimal overhead compared to Nearest Neighbor retrieval ($4.2).
4. SIFT performs better and is more robust than Nearest Neighbor retrieval ($5): We find that fine-tuning an LLM on data selected by SIFT consistently and robustly improves performance, which is not the case with Nearest Neighbor retrieval. Moreover, results suggest that at test- time, an LLM might be able to learn more effectively through fine-tuning than from its context.
SIFT can invest test-time compute proportionally performance gains ($6): We observe that our uncertainty estimates can accurately predict the performance gain of test-time fine-tuning. Motivated by this, we dynamically adapt compute to the expected performance gain.
TEST-TIME FINE-TUNING
We define test-time fine-tuning of LLMs (Hardt & Sun, 2024) as follows. We consider domain X of token sequences and assume that we have access to a large dataset of examples D C X' which we call the data spuce. further assume that we have access to a pre-trained autoregressive language model that maps token sequences X' to probability distributions over the next token from vocabulary of size V. Our work addresses the central question:
Given a prompt x² X, how can we effectively select fine-tuning data from the large dataset 'D such that the fine-tune model performs well on the prompt?
We then fine-tune the model for a single gradient step on each selected sequence.
Locally adjusting model at test-time has gained popularity with few-shot in-context learning (Brown et al., 2020; Wei et al., 2022b; Bubeck et al., 2023; OpenAI, 2024) and retrieval augmented generation (RAG, Lewis et al., 2019; Guu et al., 2020; Borgeaud et al., 2022). In contrast to this approach, test-tims fine-tuning works by fine-tuning the parameters of a pre-trained model at test-time specifically to each prompt. Notably, test-time fine-tuning takes time linear in the number of tokens whereas in-context learning with a transformer has quadratic complexity (Vaswani et al., 2017). Next to this, Hardt & Sun (2024) and other works have found (test-time) fine-tuning perform
2

---

Published as a conference paper at ICLR 2025
substantially better than in-context learning (Hu et al., 2022; Mosbach et al., 2023). This work further improves the performance test-time fine-tuning, Prior work has also studied how one can explicitly meta-learn the ability to perform test-time fine-tuning (Finn et al., 2017; Sun et al., 2024), though we find this capability to emerge even from models that are not explicitly trained in this way.
The central question studied in this work also arises when fine-tuning LLMs during post-training. For example, in targeted instruction tuning, the goal is to fine-tune a model to obtain desired capabilittes, which are commonly embodied by a set of examples x* (Xia et al., 2024). The extension of our work to such "batched" setting is straightforward.
2.1 NEAREST NEIGHBOR RETRIEVAL IS INSUFFICIENT
Prior work on data selection for fine-tuning has relied on Nearest Neighbor retrieval. The idea of making predictions on x" depending on its nearest neighbors has been around as long as machine learning itself (Fix & Hodges Jr., 1951; Cover & Hart, 1967). Bottou & Vapnik (1992) were the first to apply this idea to the fine-tuning of convolutional neural networks by selecting the nearest neighbors of a test image in pixel-space. More recently, due to advances in representation learning (Devlin et al., 2018; Reimers & Gurevych, 2019) and efficiency (e.g.. Johnson et al., 2019), Nearest Neighbor retrieval has regained attention and been applied to test-time fine-tuning (Hardt & Sun, 2024).
What is the age of Michael Jordan and how many kids does he have?
Prompt:
Nearest Neighbor:
The age Michael Jordan is 61 years.
2. Michael Jordan was born on February 17, 1963.
SIFT (ours):
1. The age of Michael Jordan is 61 years.
Michael Jordan has children.
Figure 3: We retrieve two data points to answer the prompt. Nearest Neighbor selects redundant data, while SIFT yields maximal information (ef. $L).
Xia et al. (2024) use influence functions (Cook, 1977; Koh & Liang, 2017; Pruthi al., 2019) to select data for fine-tuning LLMs. This line of work aims to select data that reduces first-order Taylor approximation
the test loss after fine-tuning, an approach that corresponds to Nearest Neighbor retrieval in certain embedding space. They highlight two main limitations of the use of influence functions and Nearest Neighbor retrieval for data selection:
Nearest Neighbor retrieval leads the selection of redundant data. Figure illustrates this limitation with a qualitative example. We formalize this limitation in Proposition K.1, which we summarize here informally:
Informal Proposition 2.1. Selecting the top-N nearest neighbors from the data space (according cosine similarity Euclidean distance) may not reduce the uncertainty about the response to the prompl beyond fine-tuning on the closest neighbor. Every additional passage may be redundant.
Nearest Neighbor retrieval selects data with high positive cosine similarity to the prompt. Yet, data with high negative cosine similarity can be equally informative data with high positive cosine similarity (Xia et al., 2024, Appendix K.2), but is ignored by standard Nearest Neighbor retrieval.
In this work, we propose SIFT and show that naturally addresses both limitations. SIFT unifies work on retrieval. which finds relevant but redundant data, and active learning (AL), which finds non- redundant but irrelevant data. In $B, we discuss how SIFT relates to prior work in retrieval and AL.
PRELIMINARIES: UNCERTAINTY ESTIMATION FOR TUNE-TUNNG
We suppose the assigned probability that y E [V] is the class label of an input x X' is given by *y(f" (xr), where 8y is the softmax (f) = exp(fy)/(E
exp(fi)) That is, f (2x) denotes the "ground truth" logits for given input æ. In the context of language modeling. V is the number of tokens in the vocabulary, and denotes the index of the next token. We defer all proofs to Appendix K.
We use a surrogate model quantify the informativeness of data, which we define next.
Assumption 3.1 (Surrogate model: Linear model class within a known latent space). We assume f' (=) W ((()) with unknown W E Vxd and where Φ(-) gd denotes known embeddings. The surrogate model uses the latent space induced by the pre-trained model to describe the data manifold. We emphasize that while SIFT relies on this surrogate model for data selection, it still fine-
3

---

Published as a conference paper at ICLR 2025
tunes the full pre-tra pre-trained model, including latent features. Surrogate dense embedding models of this kind have been used extensively for data selection via Nearest Neighbor retrieval (e.g., Lewis et al., 2019; Karpukhin et al., 2020; Borgeaud et al., 2022; Xia et al., 2024), and to understand the training dynamics and generalization of large neural networks (e.g., Jacot et al., 2017; 7; Lee et al., 2018; Malladi al., 2023; Templeton et al., 2024; Park et al., 2024). Furthermore, a surrogate model that assumes linearity in some fixed latent space may be a reasonable approximation for test-time fine-tuning since the latent space of the unfrozen model not expected change substantially by a few gradient steps.
In this work, we explore scenario where we have pre-trained model (xx) - W Φ(x). We let f(x W) = W ((()) and denote by L(W: D) the negative log- likelihood loss of f(; W) on a dataset D of imputs a with corresponding class labels y: L(W; D) = - E(a.)eD log sy [f (x; W)).
Uncertainty Estimation Our first intermediate goal is to estimate the uncertainty about the response to a given prompt x' after having fine-tuned on selected data D,, of size n. To this end, we generalize prior work on confidence sets under categorical feedback (i.e., class feedback, Amani & Thrampoulidis, 2020; Zhang & Sugiyama, 2023) to our fine-tuning setting. We consider the function class WB = ( E R Vxd IIW - W Up I B} where II-Ilp denotes the Frobenius norm and with B a constant such that W E WB. Then given data Dn. we can refine the prior estimate Wpre of W by minimizing the regularized negative log- legllkklihood loss
C D.) = L(W; P..) + 2 IW W
(1)
with regularization coefficient À > O. We write its minimizer as Wn. = arg miggmineet ewa C* Dn). We will further denote the ground truth probability distribution over the response to x by and our approximation after selection of n samples by s,(()) = sff(x; W,)).
s" (a))
We construct confidence sets of the form (x) + 3,, (x) centered around this prediction, and show their uniform anytime validity. The width of these sets is characterized by our central quantity o, (xx which we define next. We consider the inner-product kernel k(x,, (x" = =x(x)² ((a(")
and define for a set of inputs X- = [x11,... 2. C D:
(x) = k(æx, x)) - 1x (x) + -
(2)
where (x(xx) = (k(2), ().(),,)....k,(( x)) E TR. Kx E Raxa is the kernel matrix satisfying (Kx)ss )x k( and K = sup WEWB 1/A (A(æ; W)). Here, A(x; W) E R is the matrix satisfying (A(æ; W))s. = (2; W (1( = j} s5(2;; W)) which is the proper gener- alization of the derivative of the sigmoid function, standard in the analysis of binary feedback (Faury et al., 2020; Pásztor et al., 2024). We write o, = oxx (x) where X,, D X' are the inputs in Dn. With this we are ready to state our first result, namely that for careful choice of B, (6), the confidence sets contain s* (z) simultaneously for all x E X and n 2 1 with probability at least 1 - 8.
Theorem 3.2 (Confidence Sets). Let Assumption 3.1 1 hold and W* E Wn. Let & E (0, 1) and sel
(x( = 2/ V(11 2B) B LV3/2d
log
2 5V
1+
n dx
]
E (llo((n/8))
(3)
X
where L = SuPPEx, WEW, Amax (A(x; W)). Then
P(vn 2 1, E &: drv ( (x)) S (n(0)),, 2 1-8 8
where dry (8, = Σ 114--4 -411 is the total variation distance.
We use on (x) as a proxy to the uncertainty about the response to after having fine-tuned on the selected data D,., since it directly governs the size of the confidence sets around our current estimate of response probabilities. This uncertainty a key quantity not just in classification: In Appendix K.5, we state analogous confidence sets for regression with the standard squared error loss, building on results by Abbasi- Yadkori (2013) and Chowdhury & Gopalan (2017).
The Close Relationship of Regularized Loss Minimization and Test- Time Fine- Mine-TTning Recall that test-time fine-tuning does not solve the regularized objective of Equation (1), but instead takes a single gradient step. So why do we expect the surrogate model f(; W.)) be closely related to the fine-tuned prze? To answer this question, we contrast two alternative models:
WW - arg minw L (W),
(minimizer of regularized loss)
4

---

Published as a conference paper at ICLR 2025
W, W 17 ((WP)) with any step size 1) 0, (single gradient-sted step fine-tuning)
where we keep the dataset D fixed and omit the dependency on D. Our following proposition shows that both models are close if the loss landscape is relatively smooth and for careful choice of X ~ ".
Proposition 3.3. It holds that [| 1/1 W, ||r S VV - V L[(PP |||-
Recent works have also observed Wi/n = W, empirically (Ali et al., 2019; 2020). Intuitively, with a larger step size, W,, is farther away from Wire, and hence corresponds the regularized estimate with less regularization. This connection between regularized loss minimization and test-time fine-tuning is closely linked the tight connection between regularization and early stopping (Morgan & Bourlard, 1989; Yao al., 2007; Li et al., 2020). We will use this connection in the following to derive SIFT in the context iinentuning.
SIFT: EFFICIENTLY REDUCING UNCERTAINTY ABOUT THE RESPONSE
We introduce SIFT, algorithm for selecting data for fine-tuning that effectively reduces the uncer- tainty about the response to the prompt x" E X. Note that we can compute the uncertainty ox (x") about the response to the prompt x' for any selected data X C D in closed-form, since its definition (cf. Equation (2)) depends only on the selected inputs X. SIFT minimizes this uncertainty about wt:
In+1 = arg min ox ((,)( (x*) arg max ls x xnu (a") (xx) (Kx.u(m) + X'In-1) (xxu(m) (x²).
RED
ØED
(SIFT(X))
SIFT selects data that minimizes bound on the approximation error of the surrogate model, and then fine-tunes the full LLM using this data. We discuss the design choices, including the choice of embed- dings, that make SIFT efficient in $4.2. In SC.1, we illustrate with an example of how SIFT balances relevance and diversity, where we also see that the free parameter X' = An controls this trade-off. Larger X' emphasize relevance of selected data, while smaller X' emphasize diversity. Probabilistically, SIFT can be interpreted as maximizing the information gain of the selected data X,, on the response to the prompt x* in a tractable model. We formally introduce this interpretation of SIFT in $G.
4.1 UNCERTAINTY PROVABLY VANISIES
We prove that unlike with Nearest Neighbor retrieval, the uncertainty about the response to the prompt vanishes if SIFT is used select data for fine-tuning. We give an informal overview here, and defer the formal treatment to SC.2. Our theoretical analysis shows that test-time fine-tuning can fully reduce uncertainty only if the data space contains sufficient information to determine the correct response. If the data space does not contain all relevant information, the remaining uncertainty is quantified by the limiting uncertainty after seeing "all data the data space infinitely often", which we call the irreducible uncertainty and denote by ax (xxxx) We provide the formal definition in $C.2, but intuitively, the irreducible uncertainty is the largest quantity satisfying σ (æ*) 2 aoo (a=) for all X C D. We then specialize the result of Hübotter et al. (2024) to show that the uncertainty about the response to the prompt shrinks at the rate ((1/vn) until reaches the irreducible uncertainty:
Informal Theorem 4.1 (Convergence Guarantee). Fix any X' > 0 and ler SIFT(X) select X,, from the data space D. Then for all n 2 and x" E X,
(x 02 (x*) O(Xlogg))
Vn
Naturally, convergence is slower with larger regularization parameter / smaller step size. Notably, the irreducible uncertainty depends on the data space. With a large and diverse data space, the irredueible uncertainty is typically negligible. This statistical guarantee is key property of SIFT. As we show in Proposition K. Nearest Neighbor retrieval fails to satisfy a guarantee of this kind.
4.2 COMPUTE--EFICIEET DATA SELECTION
We have established how to select informative data for fine fine-tuning Next to good statistical efficiency, good computational efficiency is key for selecting data at test-time. In the following, we describe design choices such that SIFT has negligible overhead compared Nearest Neighbor retrieval.
5

---

Published as a conference paper at ICLR 2025
Sequence-Level Selection In the self-supervised paradigm, each sequence of tokens E D corresponds to a dataset of next-token predictions T1:k H 2k+1- Rather than selecting individual next-token predictions from the data space of all sub-sequences 2i:k. we select full sequences from the significantly smaller data space D, then fine-tune for a single gradient step on each sub-sequence within ac. This is a common practice in prior works that use Nearest Neighbor retrieval for data selection (e.g., Xia et al., 2024; Hardt & Sun, 2024).
Surrogate Sequence Embedders We use surrogate sequence embedding model to generate embeddings of the data space and prompts. We use the same embedding model as Hardt & Sun (2024) which is a large Roberta model (Liu, 2019) with 355M parameters that was fine-tuned for one pass on the Pile training set. The embedding dimension is 1024. Unlike Hardt & Sun (2024), we additionally normalize the embeddings to unit length, the reasons for which we discuss in $I.
We obtain decent performance with this surrogate model. Nevertheless, our theoretical results indicate that using embeddings extracted from the LLM to be fine-tuned could further improve the performance of SIFT. Empirical neural tangent embeddings (Wei et al., 2022a; Holzmüller et al., 2023) and influence function embeddings (Xia et al., 2024) can implemented efficiently and offer alternative latent spaces capturing the pre-trained model. We hypothesize that the decent performance of the surrogate model is explained by the similarity of emergent latent spaces of language models that were trained similar data.
Efficient Implementation of SIFT In our experiments, pre-select 200 candidates via Nearest Neighbor retrieval with Faiss (Johnson et al., 2019) and then apply SIFT to select 50 sequences from this smaller data space. On the Pile dataset, we find that performance can be increased further by pre-selecting more candidates (cf. Fig- ure 18 in $H) but the marginal gains diminish. The precise performance benefit of pre-selecting more candidates may differ on other datasets. We describe in $H how SIFT can be solved iteratively without computing the inverse in ev- ery iteration. When matrix of the size of the pre-selected data space fits in GPU memory, we find that SIFT has negligible computational overhead compared to Nearest Neighbor retrieval. We report results with an NVIDIA RTX 4090 GPU in Figure 4.' While our main implemen- tation of SIFT is fast the data space is small, it does not scale linearly with the size of the data space K. In SH, we show that priority queue can be used achieve almost-linear runtime in K.
Compute Overhead
Retrieval Training
1.6x
Retrieval
1.4x
1.2x
1.0x
50
200
1 1000
6000
Size of Pre-Selected Data Space
Figure 4: The (multiplicative) compu- tational overhead of SIFT compared to Nearest Neighbor retrieval is minimal. The compute overhead with 1K data space is less than 1.05x.
RESULTS
We focus on language modeling with causal language models. Following Hardt & Sun (2024), we fine-tune pre-trained LLM for single gradient step each on N = 50 selected data points in the order that they are selected, most to least relevant. We use the Pile dataset (Gao et al., 2020) for evaluation, restricting our use to data which is obtained and used in compliance with the terms of service of the data host. This version of the Pile contains a diverse set of 17 high-quality sub-datasets, ranging from Q&A to code, scientific publications, math, and more. Concretely, we use the Pile training set containing 210M sequences of total size 1.3TB as data space for data selection, and we evaluate on the Pile test set.2 2 We report the hits per byte metric as recommended by Gao et al. (2020), which is proportional to the negative log-likelihood loss normalized by a dataset-specific constant. Error bars correspond 90% confidence intervals computed via bootstrapping with 1 1'000 samples.
Base Models and Baselines We evaluate the GPT-2 model (Radford et al., 2019) with 124M param- eters also evaluated by Hardt & Sun (2024), with the default learning rate of the t trans consformee rs library (Wolf et al., 2020). We obtain analogous results with GPT-2-large (774M parameters) and the state-of-the Phi-3 (3.8B, Abdin et al., 2024).² 3 With Phi-3, we use low-rank adaptation (LoRA, Hu et al., 2022), fine-tuning slightly less than 1% of the model's 's total parameters. We compare SIFT
We use the client-server architecture described by Hardt & Sun (2024) with CPU-only servers. We evaluate on 1% of the test set (0. 1% with Phi-3), corresponding to 1'812 sequences. We detail hyperparameter choices for larger models in SI.

---

Published as a conference paper at ICLR 2025
100
90
Bits per Byte in %
(ours)
NN
85
NN
90
SIFT (ours)
80
70
70
GPT-2 (124M) GPT-2-large (774M)
Phi-3 (3.8B)
10
10
10
Figure 5: Bits per byte (in % relative to the base model, + better) after 50 test-time iterations. Left: Performance gains of SIFT are consistent across models. The failure-mode of Nearest Neighbor consistently performs worse than the base model. Tables 4 and 5 in $F detail our results with GPT-2-large and Phi-3 analogously to Table Right: Most choices of X lead to comparable performance. With X' + co, SIFT(X) repeatedly selects the nearest neighbor.
with X' 0.01 Nearest Neighbor retrieval (NN) and the failure mode of Nearest Neighbor retrieval that repeatedly selects the closest neighbor. The failure mode of Nearest Neighbor retrieval (NN-F) corresponds an extreme case of redundancy in the data space which we suspect be realistic scenario in larger or less curated datasets. Finally, we compare to Uncertainty Sampling (US), which a widely used active learning strategy (Lewis, 1995; Settles, 2009) that selects the data with the highest uncertainty in the model's 's response by selecting according to 1x+1 arg max c of (z). We compare to the heuristic that uses US to choose from the 200 nearest neighbors, in which case US can be understood as finding a diverse cover of this pre-selected data space (see, e.g., Holzmüller et al., 2023; Kirsch et al., 2). contrast, SIFT minimizes the uncertainty in the model's response to the prompt x", leading to "denser" cover close to x and "coarser" cover further away from x*.
Insight 1: SIFT consistently se- lects better data for fine-tuning than Nearest Neighbor retrieval.
US
NN
NN-F
SIFT
A
NIH Grants
(1.1
91.6(16.7) (2.
(3.8
331.1
We show in Figure/ that SIFT out- performs NN and avoids its failure mode where the same data point is selected repeatedly. In Figure 5 (left), we show that the performance gains of SIFT are consistent across models. Table compares the performance of SIFT against NN across all datasets of the Pile, using GPT-2 as base model. Overall, we find that SIFT improves performance both on datasets where NN already performs well, such as GitHub, and on datasets where NN performs poorly, such as NIH Grants. On all datasets of the Pile, SIFT per- forms at least as well as the strongest baseline (within margin of error), sug- gesting that it robust method for data selection. We observe the trend that relative performance gains of SIFT over Nearest Neighbor retrieval increase with model capability. That is, with stronger base models, informativeness of selected data appears become more important.
US Palents
81.3 81.6(1.5)
(6.68
(2.9
117.4
GitHuh
(2.6)
42.1 (2.0)
(3.2
30.0
112.1
Enron Emails
68.6
91.6 61.4
53.1
411.3
Wikipedia
61.5
66.3 (2.0)
62.7 (2..2
43.6
Common Crawt
92.6
(0.5
87.5 (1.5)
42.9
PuhMed Abstr.
(0.3)
(0.4
162.6
84.4
2.8
ArXiv
(1.4)
166.8 (1.0
82.5
42.5
PubMed Central
(2.6)
81.7 (2.6)
(55.6(5.1)
(2.5
22.2
Stack Exchange
(8.7 (8.6
76.7 (1.59
(0.7)
$1.5
Hacker News
(0.5
79.2 (2.)
133.1 (6.3)
78.4
40.8
FreeLaw
63.9 (4.1)
64.1 (4.0)
(2..4(7.1)
64.0
10.1
DeepMind Math
(2.4
69.6 (2.1)
69.7 (3.18
(2.1)
10.3
AR
(0.5)
78.3 (0.5)
73.5 13..3
(0.6)
44.8
Table I: Bits per byte (in % relative to the base model, L) after 50 test-time iterations on individual datasets of the Pile. We only include datasets with at least 10 examples in our test set. Bold numbers denote the best performing selected subset. Numbers in parentheses are standard errors. A denotes the performance gain of SIFT over the strongest baseline.
Insight 2: SIFT is robust to the choice of X' We evaluate SIFT with varying choices of X, and summarize the results in Figure 5 (right). We include extended results in Table 11 of $J, showing that for all evaluated X' between 1e-8 and 10, SIFT performs at least on-par with Nearest Neighbor re- trieval on all datasets of the Pile, often outperforming it. This suggests that SIFT is robust to the choice of X' Nevertheless, there may be an advantage to adaptively tuning (c.g., via cross cross-validatie validation). In partieular, choosing the best X' for each dataset, SIFT outperforms all baselines on every dataset.
7

---

Published as a conference paper at ICLR 2025
Bits per Byte
1.25
Base
Context
Fine Tuning
1.00
0.75
0.50
GPT-2 (124M)
GPT-2-large (774M)
Phi-3 (3.8B)
Phi-3 (14B) Gemma-2 (27B)
Figure 7: Bits per byte (L better), comparing fine-tuning and in-context learning with 50 test-time examples selected by SIFT, We find that fine-tuning systematically outperforms or performs on-par with in-context learning, even when fine-tuning only LoRA adapter with Phi-3. Test-time fine-tuning with Phi-3 (3.8B) surpasses the performance of the more than 3x larger Phi-3 (14B) and the 7x larger Gemma-2 (27B).
Insight 3: SIFT selects data the "right" number of times. Nearest Neighbor retrieval implicitly relies on non-redundancy within the data space to not select duplicate information, as illustrated in the example of Figure 3. This is almost never the case in practice, and in the extreme case of duplicate data, Nearest Neighbor selects the same data point repeatedly. SIFT does not rely on excluding previously selected data points. Instead, SIFT may select the same data point any number of times, adaptively taking more than one gradient step on it, if beneficial. To ensure that the selected data is maximally informative, SIFT takes into account the redundancy of data points explicitly. This makes SIFT robust to information duplication by design.
overall
# 25
We illustrate this in Figure 6 where performance gain of SIFT over Nearest Neighbor and its failure mode. As expected, we find that on all test prompts where SIFT selects many unique points, SIFT outper- forms repeatedly selecting the closest neighbor by a large margin. Interestingly, we also find that on all test prompts where SIFT selects only a single point, SIFT outperforms Nearest Neighbor by large margin. This suggests that in some cases repeatedly taking gradient steps on the closest neighbor is beneficial, and SIFT identifies these cases.
evaluate the
100%
Bits per Byte
50%
over NN
NN.FF
Nearest Neighbor
0%
Figure 6: Bits per byte (in % relative NN NN-F, better) after test- time iterations. Error bars correspond to standard errors. The left bars measure the performance gain The middle and right bars measure the performance gain for all prompts where SIFT selects # unique points.
Insight Test-time fine-tuning can significantly improve language modeling ability. Our results from Figure 7 indicate that test-time fine-tuning improves the performance of the base LLM substantially, surprisingly, even with a state-of-the-aat model such as Phi-3. Our Phi-3 with test-time fine-tuning and SIFT achieves 0.595 bits per byte, outperforming the previous leader in the Pile language modeling benchmark, a 30 larger model. 4 We also evaluate the recent Llama-3.2 family of models (Dubey et al., 2024), and with Lla (3B) as base model we achieve 0.557 bits per byte. a significant improvement upon the previous state-of-the-art. We compare test-time fine-tuning to the common in-context learning, where we include as much of the data as possible into the context window of the test instance, in addition its original context, by concatenating text order of selection. While in-context learning tends to improve the performance of the base model, we find that fine-tuning test-time tends to outperform or perform on-par with in-context learning. Furthermore, the compute cost of in-context learning grows quadratically with the context window size, meaning that including long texts within large context windows is expensive. Remarkably, test-time fine-tuning consistently outperforms in-context learning by more than 25% on math and coding, tasks that require
all of the Pile.
complex reasoning ($F).
Further Insights In SD. we discuss additional findings on active fine-tuning such as that the per- formance gains of SIFT over Nearest Neighbor retrieval grow with dataset size, and that normalizing embeddings is important for the effectiveness of data selection. In $E, we discuss additional findings on test-time fine-tuning, for example, the trend that larger models learn faster test-time.
*We compare to prior work in the Pile language modeling benchmark in Table of $A.
8

---

Published as a conference paper at ICLR 2025
ADAPTIVE SIFT
Bits per Byte
100
1.5
100%
SIPT
1.0
50%
0.5
0%
10-4
0.2
0.4
0.6
20
40
Uncertainty Estimate (x*)
Uneertainty Estimate 5,,(2))
Test-Time Compute
Compute-Performanee Ratio
0.15
0.2
0.25
0.35
0.5
0.75
1.0
2.0
Figure 8: Left: We visualize the empirical density of the uncertainty estimates o, wrt. the bits per byte bpb,,- Brighter colors indicate higher density logarithmic scale. We observe strong linear relationship between uncertainty estimates and bits per byte. Middle: We construct a "reliability diagram" of uncertainty estimates. Notably, since we evaluate with respect bits per byte rather than an accuracy, canonical calibration plots are not applicable. In particular, it is well known that bits per byte do not go to zero for perfect models due irreducible aleatoric uncertainty, which is not captured by our epistemic uncertainty estimates. Nevertheless, we observe that our epistemic uncertainty estimates are predictive of the model's performance. The red line indicates linear fit. Right: We visualize the bits per byte (in % relative to the base model, J. better) of all prompts whose model is fine-tuned at a given iteration. We find that by adaptively stopping with respect to the known uncertainties on, we can spend test-time compute proportional to realized performance gains (see also Figure 26 in ŞJ). Remarks: Results are with GPT-2. In the left and middle plots, we remove the lowest and highest 0.25% of uncertainty estimates (i.e., the outliers) for better visualization. In the left plot, we additionally remove the lowest and highest 0.25% of bits per byte.
COMPUTE-PROPORTIONN TEST-TIME FINE-TUNING
We have shown that test-time fine-tuning can improve language modeling ability and that SIFT is robust method for data selection, outperforming Nearest Neighbor retrieval. However, a key shortcoming of previous approaches to test-time fine-tuning is that they spend a fixed amount of test-time compute, regardless of the nature of the prompt. the available data, the model. This is not computationally scalable in many practical applications, since fixed test-time compute budget leads non-pn con-proportionat performance gains. For example, for the prompt "Hello" chatbot would not like to spend any test-time compute, while for a more complex prompt we would like to spend more compute. In this section, evaluate whether uncertainty estimates can be used to adaptively stop test-time fine-tuning such that the realized performance gain is proportional to the compute used.
The response uncertainty can predict performance gain. We find that a (x=") ) is mono- tonically and linearly correlated at coefficient ze 0.4 with the model eтror after n test-time iterations, i.c., the bits per byte bpb,, (x*) This is remarkable because 0, contains information only from the surrogate embedding model, and is normalized such that ox(x²) 1. To determine the importance of the base model, we also evaluate the denormalized uncertainty estimate o (x*) = o, (x=")-bbb, (a:). which unlike on cannot be evaluated at test-time. We multiply o, by bpb, to ensure that the uncer- tainty measure
Insight 5:
the same units as the performance metric, correcting for the use normalized surrogate embeddings. We find that o, (x") is strongly correlated at coefficient ~ 0.5 with the bits per byte. We summarize correlations in Table 12 of ŞJ and visualize the predictive capability of â, in Figure (left) and Figure 8 (middle). Our findings indicate that approximations of the base model's uncertainty, before test-time fine-tuning, can be beneficial. In future work, we intend to determine whether generating embeddings from the base model can provide such scale-correction.
Recall that SIFT minimizes the response uncertainty on to the given prompt. The predictive ability of uncertainty estimates provides intuitive explanation for the effectiveness of SIFT.
Conpute-Proporttone Performance Gains: Early stopping at the "right" time. Motivated by the predictive power of uncertainty estimates, we evaluate whether they be used adaptively

---

Published as a conference paper at ICLR 2025
stop test-time fine-tuning such that the realized performance gain is proportional to the compute used. In the following, we propose a such a stopping criterion for SIFT. Using the approximation of the error via uncertainty estimates discussed above and that To(2=)) ==
performance gain bpbg(x") o0(x')
(4)
(2b,,
(x*)
(x*)
We would like to stop fine-tuning when further test-time compute does not yield proportional performance gain, i.e., when "performance gain < a (Q.1i with n approximating the compute of n iterations and a constant comparing the units of compute and performance. Plugging in our above approximation of the performance gain, we propose to stop test-time fine-tuning before iteration n if (ADAPTIVE SIFT) Intuitively, this stops fine-tuning the LLM when its progress in crafting a better response stalls. For complex prompts that benefit from fine-tuning, ADAPTIVE SIFT spends more test-time compute, whereas for prompts where the model is already strong where the data space is not informative, ADAPTIVE SIFT spends less test-time compute. Figure 8 (right) shows that the performance gains of this approach are proportional to the compute used.
on (x") > (an)-1
Towards Scaling Laws of Time Fine-Tuning Interestingly, our results bear resemblance to scaling laws of LLM pre-training (Kaplan et al., 2020; Henighan et al., 2020; Hoffmann et al., 2022). These scaling laws express the performance of model as a function of the compute used for pre-training ge. the number of parameters or training tokens). Such scaling laws are crucial for determining how to optimally spend fixed amount of compute. Recently, scaling laws for "test-time inference" have gained attention, where test-time compute is usually spent on search (e.g.. beam search) with a variable number of forward passes of few-shot prompted base LLM (Brown et al., 2024; Snell et al., 2025). Our results suggest that similar scaling laws exist for test-time fine-tuning, expressing the performance of model as a function of the compute used for fine-tuning at test-time. Such scaling laws can be an important tool to determine how to spend test-time compute. There many open questions in this direction, which we do not address in this work. For example, how does model size affect the scaling laws of test-time fine-tuning? Or, can a model be fine-tuned at test-time build reasoning chains? Based on our results and previous evaluations of fine-tuning and in-context learning (c.g., Hu et al., 2022; Mosbach et al., 2023; Hardt & Sun, 2024), we conjecture that test-time fine-tuning may lead more efficient use of compute than repeatedly prompting a base LLM. We believe that these open questions are exciting directions for future work.
7 DISCUSSION AND FUTURE WORK
We propose a data selection algorithm, SIFT, unifying ideas from retrieval and active learning- SIFT estimates the uncertainty about the response to given prompt after having been fine-tuned on some data ($3), and then selects the data that minimizes this uncertainty ($44. This addresses the limitations of Nearest Neighbor retrieval ($2). SIFT can be seen as generalization of Nearest Neighbor retrieval from search method to a learning method, which ensures explicitly that the retrieved data is maximally informative. We show on the Pile dataset that SIFT consistently outperforms Nearest Neighbor retrieval in prompt-specific fine-tuning at test-time and that this kind of local learning can be more effective than locally learning from examples in-context ($5). Finally, we observe that our uncertainty estimates can predict the performance gain of test-time fine-tuning, and use this to develop an adaptive algorithm which achieves compute-proportioall performance gains (56).
Test-time fine-tuning addresses a fundamental limitation of in-context learning, namely that in-context learning is typically limited to fixed and finite context window. In contrast, test-time fine-tuning allows the LLM to dynamically and effectively access a potentially unbounded non-parametric memory. By improving the effectiveness of test-time fine-tuning, this work opens up several exciting directions for future research. Test-time fine-tuning may be used to ground the model on a trusted dataset, mitigate biases against under-represented groups in the training data, or to dynamically include private data depending on user privileges. Particularly interesting would be a broad evaluation on morpperplyx tasks such as code generation or in the life sciences with large-scale medical or protein data. Unlike few-shot in-context learning which is limited in scope to autoregressive models, test-time fine-tuning and SIFT may be extended to other model classes such as diffusion models. Furthermore, SIFT may be used effectively in other settings that require automatic data selection, such as targeted instruction tuning during post- trainita of LLMs. Finally, our results suggest scaling laws for test-time fine-tuning and we outline several exciting open questions ($6).
10

---

Published as a conference paper at ICLR 2025
CONTRIBUTIONS
JH conceived and led the project, being involved in all its components and leading the theory, implementation of the SIFT algorithm, design of experiments, and writing. SB set up and ran the first experiments validating the approach, and contributed to running the final ablation studies. IH ran additional experiments, especially those with larger models, and optimized the code. AK advised.
ACKNOWLEDGEMENTS
We would like to thank Armin Lederer, Vignesh Ram Somnath, Bhavya Sukhija, Scott Sussex, and Lenart Treven for feedback on early versions of the paper. This project was supported in part by the European Research Council (ERC) under the European Union's Horizon 2020 research and Innovation Program Grant agreement no. 815943, and the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545. Ido Hakimi was supported by an ETH AI Center Postdoctoral fellowship.
REFERENCES
Yasin Abbasi Yadkori. Online learning for linearly parametrized control problems. PhD thesis, University of Alberta, 2013.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 24f.:24404.42119 2024.
Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for abstract reasoning. arXiv preprint arXiv:2411.07279, 2024.
Alnur Ali, J Zico Kolter, and Ryan Tibshirani. A continuous-time view of early stopping for least
squares regression. In AISTATS, 2019.
Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient
flow for least squares. In ICML, 2020.
Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. NeurIPS, 2020.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4, 2016.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In ICLR, 2020.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, 1997.
Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. Ann-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems, 87, 2020.
Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of generalist policies, arXiv preprint arXiv:2410.05026. 2024.
Soumya Basu, Ankit Singh Rawat, and Manzil Zaheer. A statistical perspective retrieval-based models. In ICML, 2023.
Aman Bhargava, Cameron Witkowski, Manav Shah, and Matt Thomson. What's the magic word? control theory of IIm prompting. arXiv preprint arXiv:2310.044.4 2023.
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and lims by learning to learn discrete functions. In ICLR, 2024.

---

Published as a conference paper at ICLR 2025
Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-oriented bayesian active learning. In AISTATS, 2023.
Ilija Bogunovic, Jonathan Scarlett, Andreas Krause, and Volkan Cevher. Truncated variance reduction:
A unified approach bayesian optimization and level-set estimation. In NeurIPS, 2015.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4(6), 1992.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ArXiv:2005. 14165, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science, 1995.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.
François Chollet. On the measure of intelligence. arXiv preprint axXf::1911.01547,200
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In ICML, 2017.
William S Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American statistical association, 74(368), 1979.
William S Cleveland and Susan Devlin. Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American statistical association, 83(403). 1988.
Jack Cole and Mohamed Osman. Dataset-induced meta-learning (and other tricks): Improving model efficiency on arc. https: /tts:://laaaa /1/b42/b22.// htt.a//p....oooooooooooooooooeee /
[Accessed 22-08-2024).
Dennis Cook. Detection of influential observation linear regression. Technometrics, 19(1), 1977.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1), 1967.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
Matthijs Douze, Alexandr Guzhva, Chenggi Deng. Jeff Johnson, Gergely Szilvasy,
Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21780 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint 2209 10090 2022.
12

---

Published as a conference paper at ICLR 2025
Louis Faury, Mare Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In ICML, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In ICML, 2017.
Evelyn Fix and Joseph Lawson Hodges Jr. Discriminatory analysis: nonparametric discrimination, consistency properlies, volume 1. USAF school of Aviation Medicine, 1951.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoen- coders. In NeurIPS, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 1.Xiv:2/01.0000 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, et al. framework for few-shot language model evaluation, 2024.
Robert Geirhos, Priyank Jaini, Austin Stone, Sourabh Mcdapati, Xi Yi, George Toderici, Abhijit Ogale, and Jonathon Shlens. Towards flexible perception with visual memory. arXiv preprint arXiv:2408.08/ 07/402 2024.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICML, 2020.
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi, Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2303.08/ 00000.0/144 2023.
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In ICLR, 2024.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010 14701, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint N.Kiv:2203..55335 2022.
David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. A framework and benchmark for deep batch active learning for regression. JMLR, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Transductive active learning: Theory and applications. In NeurIPS, 2024.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data. In ICMI, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. NeurIPS, 2017.
Vidit Jain and Erik Learned-e Miller. Online domain adaptation of a pre-trained cascade classifiers In CVPR, 2011.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-seas similarity search with gpus. IEEE Transactions Big Data, 7(3), 2019.
13

---

Published as a conference paper at ICLR 2025
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.08361, 2020.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020.
Diederik P Kingma and Jimmy L Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In NeurIPS, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards statistical theory of data selection under weak supervision. In ICLR, 2024.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. ICLR, 2024.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning realistic scenarios. In NeurIPS, 2020.
Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A rich class of parameterized submodular information measures for guided data subset selection. In AAAI, 2022.
Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In ICML, 2018.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. arXiv preprint arXiv:1904.0880 2019.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. NeurIPS, 2018.
David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In ACM Sigir Forum, volume 29, 1995.
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, genera-
Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. tion for mhowledge-intensinn nip tasks. In NeurlPS, 2019.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterizee neural networks. AISTATS, 2020.
Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. One sentence one model for neural machine translation. LREC, 2018.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: 1: Technical details and evaluation. Technical report, A121 Labs, 2021.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. 2019.
arXiv preprint
Xuan Luo, Jia depth estimation. ACM Transactions Graphics (ToG), 2020.
Jia-Bin Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video
14

---

Published as a conference paper at ICLR 2025
David JC MacKay. Information- objective functions for active data selection. Neural computa- tion, 4(4), 1992.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In ICML, 2023.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In NAACL, 2013.
Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions. Optimiza- fion Techniques, 7, 1978.
Nelson Morgan and Hervé Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. NeurIPS, 1989.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning in-context learning: A fair comparison and evaluation. In ACL, 2023.
Kevin P Murphy, Probabilistic machine learning: Advanced topics. MIT press, 2023.
Elizbar Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1), 1964.
George Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-2 Mathematical programming. 14, 1978.
OpenAl. Learning reason with Ilms. OpenAl blog, 2024.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In ICML, 2024.
Barna Pásztor, Parnian Kassraie, and Andreas Krause. Bandits with preference feedback: A stackel- berg game perspective. In NeurIPS, 2024.
Jay M. Ponte and W. Bruce Croft. A language modeling approach to information retrieval. In SIGIR. Association for Computing Machinery, 1998.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In NeurIPS, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In IJCNLP, 2019.
Stephen Robertson, Hugo Zaragoza, et The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4), 2009.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2017.
Sambu Seo, Marko Wallat, Thore Graepel, and Klaus Obermayer. Gaussian process regression:
Active data selection and test point rejection. In Mustererkennung. Springer, 2000.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
Jack Sherman and Winifred Morrison. Adjustment of inverse matrix corresponding to change in one element of given matrix. The Annals of Mathematical Statistics, 21(1), 1950.
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. CVPR, 2018
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. In ICLR, 2025.
15

---

Published as a conference paper at ICLR 2025
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1), 1972.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML, 2009.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with superviieo for generalization under distribution shifts. In ICML, 2020.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, al. Learning to (learn test time): Runs with expressive hidden states. arXiv preprint arXiv:2407.0400 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models practical size. arXiv preprint arXiv:2408.00116 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, Anthropic, 2024.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In ICML, 2023.
Chaoqi Wang. Shengyang Sun, and Roger Grosse. Beyond marginal uncertainty: How accurately can bayesian regression models estimate posterior predictive correlations? AISTATS, 2021a.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021b.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In NeurlPS, 2023.
Geoffrey Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A. 1964.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In ICMI, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. In NeurIPS, 2022b.
Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press, 2006.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, et al. Huggingface's transformers: State-of-the-aat natural language processing. arXiv preprint 1910:31910 2020.
Henry P Wynn. The sequential generation of d-optimum experimental designs. The Annals of Mathematical Statisties, 1970.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In ICMI., 2024.
16

---

Published as a conference paper at ICLR 2025
Minjie Xu and Gary Kazantsev. Understanding goal-oriented active learning via influence functions. In NeurlPS Workshop on Machine Learning with Guarantees, 2019.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2). 2007.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In ICML, 2023.
Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. In NeurIPS, 2021.
Kai Yu, Jinbo Bi, and Volker Tresp. Active learning via transductive experimental design. In ICML, 2006.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- 130b: An open bilingual gre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Improved regret and constant computation cost. NeurIPS, 2023.
17

---

Published as a conference paper at ICLR 2025
APPENDICES
CONTENTS
A Comparison to the State-of-the-Art on the Pile Language Modeling Benchmark
20
B Extended Related Work
21
B.1 Learning at Test-Time
B.2 Data Selection
22
B.3 SIFT Unifies Work on Retrieval and Work on Coverage
23
Further Details on SIFT
24
C.I How SIFT Balances Relevance and Diversity
24
C.2 The Uncertainty of SIFT Provably Vanishes
24
Further Insights on Active Fine- Tunnig
26
Further Insights on Test-Time Fine-Tue Fine-Tuning
27
F Extended Results
30
F.1 Active Fine-Tuning
30
F22 Test-Time Fine-Tuning
30
G SIFT Maximizes Information Gain
33
G.I Preliminaries: Information Theory and Gaussian Processes
33
G.2 Probabilistic Observation Model
34
G.3 The Probabilistic Interpretation of SIFT
34
G.4 How SIFT Balances Relevance and Ier.............................
35
G.5 The Perspective of Classification
35
Efficient Computation of SIFT
36
H.I Exact Implementation
36
H.2 Fast (Exact) Implementation
36
H.3
Pre-Selecting Data via Nearest Neighbor Retrieval.
37
H.4 Future Work: Improving GPU Utilization of SIFT-FAST
37
I
Experiment Details
40
L.1 Properties of the Pile Dataset
41
1.2 In-Context Baseline
41
I.3 Inference Cost with Test-Time Fine-Tuning
42
J Ablations
43
18

---

Published as a conference paper at ICLR 2025
K Proofs
47
K.1
Notation
47
K.2
Insufficiency of Nearest Neighbor Retrieval (Informal Proposition 2.1)
47
K.3
The close relationship Regularized Loss Minimization and Test-Time Fine- Tuning (Proposition 3.3)
48
K.4
How SIFT Balances Relevance and Diversity
48
K.5
Confidence Sets for Regression
49
K.6
Confidence Sets for Classification (Theorem 3.2)
50
L Qualitative Examples
52
L.1 Balancing Relevance and Diversity
52
L.2 Examples from the Pile
52
19

---

Published as a conference paper at ICLR 2025
A COMPARISON TO THE STATE-OF-THE-AN ON THE PILE LANGUAGE MODELING BENCHMARK
Table 2 summarizes the state-of-the-art in the Pile language modeling benchmark.
Model
Bits per Byte
Bits per Byte (without Wikipedia)
Jurassic-1 (1788, Lieher 2021)
n/a
0.601
GLM (130B, Zeng et al., 2022)
n/a
0.622*
GPT-2 (124M, Radford 2019)
1.241
GPT-2 (774M, Radford ot al,, 2019)
1.093
Llama-3.2-Instee (IB, Dubey at.,,
0.807
Llama-3.2 Instruct (3B, Dubey al., 2024)
0.737
Gemma-2 (2B,
2024)
0.721
1lama-3.2 (IB, Dubey ol., 2024)
0.697
0.684
Phi-3.5 (3.8,,
2024)
0.690
Phi-3 (3.8B, Abdin al., 2024)
0.679
0.678
Phi-3 (7B. Abdin al., 2024)
0.678
Gemma-2 (98, Team al., 2024)
0.670
GPT-3 (175B, Brown et al., 2020)
0.666
Phi-3.5-MoE (188,.8B, Abdin al., 2024)
0.656
Phi-3 (14B, Abdin et al., 2024)
0.651
Llama-3.2 (3B, Dubey al.,
0.640
0.627
Gemma-2 (27B. Team et 2024)
0.629
Test-Time FT with SIFT + GPT-2 (124M)
0.862
Test-Time FT with SIFT GPT-2 (774M)
0.762
Test-Time FT with SIFT + Llama-3.2 (IB)
0.606
0.607
Test- Cest-Time/aae FT with SIFT Phi-3 (3.8B)
0.595
0,599
Test- Teme-Time/ee FT with SIFT Llama-3.2 (38)
0.557
0.559
Table 2: Evaluation of state-of-the-art models on the Pile language modeling benchmark, without copyrighted datasets. (*): Results with GPT-3 are from Gao et al. (2020); results with Jurassic-1 I and GLM are from Zeng et al. (2022) and do not report on the Wikipedia dataset. For a complete comparison, also evaluate
Phi-3 and Llama-3.2 with test-time fine-tuning when excluding the Wikipedia dataset. Bold numbers denote the best performing model. Underlined numbers denote model that is better than the previous state-of-the-art.
Due our dataset being restricted the non-copyrighted part of the Pile, the data distribution changes slightly. To account for this, we take the reported results of prior work and exclude the datasets that have copyright restrictions from the evaluation. Notably, some prior reported results of state-of-the-art models miss evaluation of the Wikipedia dataset, which we therefore also exclude for a direct comparison. To the best of our knowledge, our results with test-time fine-tuning and SIFT achieve a new state-of-the-an on the Pile benchmark.
20

---

Published as a conference paper at ICLR 2025
B EXTENDED RELATED WORK
B.1 LEARNING AT TEST-TIME
The subject of learning at test-time has a rich history in statistics and machine learning. By "learning test-time" we refer to models that are constructed specifically for a given test instance, differing from the model used for other test instances. The following discussion provides a brief overview with emphasis on the most recent developments.
k-Nearest Neighbors and Kernel Regression (since 1950s) One of the most basic forms of learning at test-time was developed by Fix & Hodges Jr. (1951) and Cover & Hart (1967). Given the supervised data 'D S X' x y with input domain X' C Rd and labels y = {o..... ................................ K}. the K--N algorithm predicts the label of test instance x² E X by taking the majority vote of the nearest neighbors of x" in D according to some distance metric on X' such as Euclidean distance. In the case of regression, Y' = R and the prediction is the average of the labels of the k nearest neighbors. Kernel regression extended upon this idea in the 1960s by weighting neighbors according to their distance to the test instance (Nadaraya, 1964; Watson, 1964). This is a simple and often effective method if the inputs well-structured and low-dimensional, e.g., if X' is learned oow-dimensional manifold (Geirhos et al., 2024). When K is large, as for example when y is the set of all tokens in language modeling task, naive application of k-NNs is difficult, nevertheless they have been shown to be effective when mixed with parametric language models (Khandelwal et al., 2020).
Local Learning (since 1970s) Local learning is the idea of using data "relevant" the test in- stance x" to train a parametric model. Formally, given a test instance x", conventually a model f is used to predict f(x") where f is trained to minimize the average loss over the training data. Instead, local learning trains a model fæ- specifically for x" and predicts fa- (x*). Original works train a lin- car model by weighting data according to their proximity to x (Cleveland, 1979; Cleveland & Devlin, 1988; Atkeson et al., 1997). Here, each test instance trains a model from scratch since the optimal solution of linear regression is independent of initialization. This perspective has regained interest recently in the context of neural networks, with Sun et al. (2020) naming it "test-time training".
Transductive Learning (since 1980s) Vladimir Vapnik developed the general principle of trans- duction which he states in Vapnik (2013) as follows:
Vladimir Vapnik: "When solving a problem of interest, do not solve more general problem as an intermediate step. Try to get the answer that you really need but not more general one."
This is perhaps the most general principle behind learning at test-time, and directly opposed to the principle of induction - extracting the most general rules from data- - which has arguably dominated machine learning research over the last decades. In way, local learning is pushing the principle of transduction to the opposite extreme: Each test instance defines its own learning problem, with the test instance alone being the target of prediction.
Local Tane-Tuning (since 1990s) Bottou & Vapnik (1992) were the first to use local learning in conjunction with pre-trained parametric model. They train (i.e., "fine-tune") the last layer of con- volutional neural network for handwritten digit classification based on the nearest neighbors to the test instance in pixel space. Very recently, Hardt & Sun (2024) applied the same idea to language models, showing that local fine-tuning can significantly improve the performance of large language models on standard benchmarks. Previously, this idea has also been evaluated by Li et al. (2018) and Basu et al. (2023). "Test-time fine-tuning" " (as well as "active inference") has frequently been used to refer to this approach of locally fine-tuning a pre-trained model. Within the last few years, test-time fine-tuning has regained substantial interest in the context of self-superviset learning, where the pre-trained model fine-tuned on the test instance itself. Notable applications of this approach are in vision (Jain & Learned-Miller, 201 1; Shocher et al., 2018; Luo et al., 2020; Sun et al., 2020; Wang et al., 2021 b) and in language modeling (Krause et al., 2018; 2019), where it is called dynamic evaluation. As one would also naively expect, test-time fine-tuning yields the largest improvements when the prompt is not (well- ) represented in the pre-training data, e.g., due to a distribution shift (Gandelsman et al., 2021; Hardt & Sun, 2024). Notably, test-time fine-tuning is the central component of the state-of-ther of-tth--rtt approaches to the ARC challenge (Chollet, 2019; Cole & Osman, 2023), a non-saturated benchmark which is intended test reasoning capabilities based on "core knowledge" rather than mere memorization.
21

---

Published as a conference paper at ICLR 2025
(Few-Shot) In-Context Learning (since 2020s) Very recently. with the advent of large language models (LLMs), learning at test-time has regained interest. Brown et al. (2020) showed that GPT-3 can learn in-context from input-label pairs that are appended to the prompt, emergent phenomenon of LLMs that has been widely studied since ( (Von Oswald et al., 2023; Kossen et al., 2024;
Bhattamishra et al., 2024). In contrast to standard in-weights learning, in-context learning requires no parameter updates. Interestingly. in-context learning adopts the same paradigm as local learning wherein a model is adapted specifically for the test instance x", here by skewing the autoregressive distribution towards the data included in the prompt. This is often combined with the automatic soureing of nearest neighbors to x in an external dataset, which is known as "retrieval augmented generation" (RAG, Lewis et al., 2019 Borgeaud et al., 2022), and is akin to the other methods of test-time learning discussed above. A crucial difference between test-time fine-tuning and in-context learning appears to be that learning from context works by changing the test inslance (Bhargava et al., 2023) whereas in-weights learning works by changing the model. With small datasets, in-context learning is therefore often more computationally efficient than test-time fine-tuning, however this ceases to be the case when the dataset grows since the complexity of transformers grows quadratically in the number of context tokens whereas the complexity of test-time fine-tuning grows linearly.
B.2 DATA SELECTION
Clearly, the choice of data to learn from at test-time is crucial for predictive performance. Selecting uninformative data can increase inference time or even degrade performance (see, e.g., Kolossov et al., 2024). Today, datasets for fine-tuning are often hand-designed, however, this is not possible in test-time setting. Automatic data selection has a rich history in machine learning, studied extensively in search, experimental design (Chaloner & Verdinelli, 1995), and active learning (Settles, 2009). The following attempts to give a brief overview of the most recent developments.
(Document) Retrieval (since 1970s) Retrieval methods aim to search dataset 'D for the most relevant data to a given query/prompt. The most classical methods such as TF-IDF (Sparek Jones, 1972) and BM25 (Robertson et al., 2009) are based on keyword matching, and were developed alongside the first search engines. Due to their reliance on "bags of words", i.e., sets of one-hot- encoded word vectors, they are known as sparse retrievers. An alternative idea is select the data that maximizes the likelihood of the query x" given the data, i.e., arg m D p(xx æ), known as query likelihood retrievers (Ponte & Croft, 1998; Wang et al., 2023). Here, the conditional probability can be a non-parametric term frequency or parametric language model. More recently, due to significant advances in representation learning (Devlin et al., 2018; Reimers Gurevych, 2019), dense retrievers have become popular (e.g.. Lewis et al., 2019; Karpukhin et al., 2020; Borgeaud et al., 2022). A dense retriever embeds dataset and query into a metric vector space, and retrieves the nearest neighbors to the query. Standard vector-based search methods use cosine similarity or (equivalently") Euclidean distance. Recent advances in algorithms and implementation mean that (approximate) nearest neighbor retrieval can be performed efficiently with databases of billions or even trillions of tokens (e.g., Johnson et al., 2019; Aumüller et al., 2020). The most common metric is cosine distance, which coincides with Fuclidean distance when vectors are normalized to unit length. Nearest neighbor retrieval has been the de-facto standard for data selection in RAG and local learning."
Influence Functions (since 1970s) Influence functions measure the change in model's prediction when a single data point is removed from the training data. First proposed by Cook (1977) for linear regression, they have since been used extensively to interprer predictions (Koh & Liang, 2017; Pruthi et al., 2019). Very recently. Xia et al. (2024) applied influence functions to select data that leads to the largest (approximate) reduction in test-loss. Concretely, using first-order Taylor approximation of the loss l and if the model at time is updated via stochastic gradient descent with step size 1)t on data æ, the loss reduction can be approximated as
{(x": 0 l(x" ((x;0,) 0.) = (V ((x; Vo ((x":0));
That is, the data a whose loss gradient is most aligned with the loss gradient of the test instance ax", can be expected to lead to the largest loss reduction." Note that this simply leads to nearest neighhor
Herr we assume that vectors are normalized to unit length, cf. Appendix K.2.
There is substantial literature that investigates selection of "informativv"" data for RAG (e.g., Ye ct al., 2023). "iia et al. (2024) normalize embeddings before computing the inner product (thus, maximizing cosine similarity) account for varying gradient norms depending on sequence lengths.
22

---

Published as a conference paper at ICLR 2025
retrieval in embedding space informed by the model at time t. A major limitation of using influence functions for data selection is that they implicitly assume that the influence of selected data adds linearly (i.e., two equally scored data points expected to doubly improve the model performance, Xu & Kazantsev, 2019, Section 3.2). This assumption does quite obviously not hold in practice seen, e.g., by simply duplicating data. The same limitation applies to the related approach of datamodels (Ilyas et al., 2022). A recent line of work aims to address this limitation by designing simulators that can be probed with datasets to estimate their effect on prediction requiring less compute than training the full model (Guu et al., 2023), yet, this does not address the data selection problem as the space of possible datasets is exponentially large.
Coverage & Inductive Active Learning Next we discuss an orthogonal line of work, which takes into account the interaction between selected data, but not the interaction of that data with respect to test instance. Roughly speaking classical active learning studies how to most effectively select data from a domain X' for learning model over this domain X. Intuitively, this task can be thought of as selecting a subset X C X of fixed size that captures the most "information" about the target function f. As such, this task is of an inductive nature; we aim to extract general rules from the data that can be applied to unseen data later, without concrete specification of the unseen data. Approaches to (inductive) active learning are broadly aiming to select diverse data that covers the data manifold in X' well. Methods include those that maximize the mutual distances between selected data (e.g., CORESET (Sener & Savarese, 2017), BADGE (Ash et al., 2020), and PROBCOVER (Yehuda et al., 2021)) with respect to latent distance metric and those "uncertainty sampling" methods that select data that the model is most uncertain about (e.g.. D-optimal design (Wynn, 1970) and BATCHBALD (Kirsch al., 2018)).* Both families of methods can be seen as determining some decent covering of the data manifold in X. In a probabilistic sense, uncertainty sampling can be seen to minimize the "posterior predictive entropy" in expectation over the observed data. Approaches to inductive active learning have frequently been applied to pre-training models, with image classification as the canonical application (e.g, Holzmüller et al., 2023).
B.3 SIFT UNIFIES WORK ON RETRIEVAL AND WORK ON COVERAGE
Retrieval and inductive active learning fall on two extreme ends of a spectrum: Retrieval methods search for relevant data without ensuring that data is non-redundant. As such, naive application of search methods insufficient for a learning task since those generally do not take "distinctiveness" into account (cf. Section 2.1). In contrast, active learning methods select non-redundant data without ensuring that data relevant. Like SIFT, many active learning methods are based on some measure of "uncertainty", however how this measure is utilized for data selection differs fundamentally in SIFT:
Transductive Active Learning: Unifying retrieval & coverage Transductive active learning is motivated from the central observation that learning and prediction requires synthesizing information that is both relevant and non-redundant. Transductive active learning (Hübotter et al., 2024) bridges this gap by selecting data that is both relevant and non-redundant. In this work, we propose SIFT, an approach to test-time transductive active learning (i.c., transductive active learning with a single prediction target), which extends previously proposed algorithms (MacKay, 1992; Sco et al., 2000; Yu et al., 2006; Hübotter et al., 2024). Similar algorithmic ideas have recently been evaluated empirically in variety of other settings (Kothawade et al., 2020; Wang et al., 2021a; Kothawade et al., 2022; Bickford Smith et al., 2023) such as Bayesian optimization (Hübotter et al., 2024), multi-task reinforcement learning (Bagatella et al., 2024), and the amortized fine-tuning of neural networks (Hübotter et al., 2024). SIFT aims to select data that is both relevant and non-redundant with respect to the already seen data, whereby the hyperparameter X' controls the trade-off between relevance and redundancy. Hübotter et al. (2024) introduce extensions of SIFT to more than one prediction target, i.e., amortizing learning across multiple prompts. They show that if the prediction targets include all of X, then the method reduces to a form of inductive active learning.
"Section 5.2 of Holzmüller al. (2023) provides comprehensive overview.
23

---

Published as a conference paper at ICLR 2025
C FURTHER DETAILS ON SIFT
C.I HOW SIFT BALANCES RELEVANCE AND DIVERSITY
Let us look more closely at the points selected by SIFT. We will assume here for ease of notation that embeddings have unit length." 9 The first point selected by SIFT has the largest (absolute) cosine similarity to the prompt within the latent space:
(0(x*) o(z))²
(<o(xr,,))
T arg min 0f (x*) = arg max RED
arg max RED
(1st point)
1 X
RED
cosine similarity of <b(x") db())
This recovers the standard approach of Nearest Neighbor retrieval with respect to cosine similarity, provided cosine similarities are non-negative. However, we show next that selecting more than one point, SIFT not only considers the relevance with respect to the prompt x", but also the redundancy with respect to the already seen data I1
Lolw" 211) La (20 x0)
1+X
<)(x1, 1+X
[[
- Lo x1) <x(x" " (2nd point)
32 arg min of 2ED
(01.0) (21,,)("*)))
(x*) arg max #ED
(x)
To illustrate how SIFT balances relevance and diversity, we compare the value of observing #1 twice to observing a different æ with cosine similarity <0(11.)) = O. We show in Appendix K.4 that SIFT(X)) prefers æ over æ for selecting 22 if and only if
X
Ls(x" x)²
Lo(x" x1)²
The hyperparameter X' controls the trade-off between relevance and diversity: if X' then even if x has third the relevance of 2, is still preferred. As X + oo, SIFT(X)) performs retrieval by repeatedly selecting the same point; and as X' - O, SIFT(X') aims only to select the most diverse points. We observe the same relationship empirically on the Pile dataset (cf. Figure 9 (left)). Table 3 summarizes the effect of the regularization parameter À and its interpretations.
Parameter regularization À step size noise p (cf. SG)
Relation Div.
A
1/n
t
p²
+
Table 3: The effect of and its other interpretations diversity of selected data (as the parameter is increased).
C.2 THE UNCERTAINTY OF SIFT PROVABLY VANISHES
We now formally prove that unlike with Nearest Neighbor retrieval, the uncertainty of (x=) about the response the prompt vanishes if SIFT is used to select data for fine-tuning. As discussed in 1, this requires that the data space contains sufficient information to determine the correct response. In general, there might be an irreducible error remaining. We will denote basis of the embeddings {Φ(x) æ E DD} within the data space 'D by $ R= with size m and dimension d, and we denote by Hı its orthogonal projection onto the orthogonal complement the span 4. Hübotter et al. (2024) show that for all X C.
xx [0(x*) li
(5)
where |ellA Vo Av denotes the Mahalanobis distance. We call (x²)) ducible uncertainty about x². It can be seen that of (xel) ) for all xll E X with o(x span +. That is, the irreducible uncertainty is zero for points the span of the data space. In contrast, for points _ with Φb ((x+) E (span + the irreducible uncertainty equals the initial uncertainty: o2 (x+) = of (x
(x*)
= 2 the irre-
The irreducible uncertainty of any prompt x* can be computed by simple decomposition of (xxx" into parallel and orthogonal components. Hence, if the data space is large and includes all relevant information to answer the prompt, the irreducible uncertainty is negligible.
We will denote the uncertainty reduction about the prompt x" achieved by fine-tuning Vaa (X) x²*))) - xx(xx (x") and note that SIFT selects 1n+1 = arg max via. (X,, U {x)). Stating the convergence guarantee of SIFT requires straightforward assumption.
X by
24

---

Published as a conference paper at ICLR 2025
Irreducible Uncertainty
0.1
Variance Reduction dhr
1.00
10000
10% of data
10
50% of data
0.97
0.06
100% data
0.95
0.1
0.01
0.93
0.0001
0.02
0.90
20
40
0
20
40
0
20
40
Number of Unique Points
Number of Neighbors
Test-Time Iterations
Figure Left: The parameter X' controls the trade-off between relevance and diversity of the selected data. As X' + oc, SIFT selects the same point repeatedly whereas as X + 0, SIFT selects a diverse set points. Middle: The irreducible uncertainty of test prompts from the Pile given neighbors selected from fractions of the Pile training dataset in the data space. The irreducible uncertainty captures how much information available, and decays quickly. Right: We empirically observe that Vn- is monotone submodular, i.e., its "marginal gains" decrease the number of iterations increases. The shaded region denotes the standard deviation, gray lines are from randomly selected prompts.
Assumption C.I. The uncertainty reduction v/ (X) is submodular.
Intuitively, Assumption C.I states that the marginal uncertainty reduction achieved by adding point the selected data (i.e., the 'marginal gain") decreases as the size of the selected data increases, which is a common assumption in prior work." 10 Formally Assumption C.I is satisfied if, for all D and X'
CD
Dw. x') 2 Az. (z (X)
(6)
where Ag (x X) - Via (XU(z)) the (X) ) is the marginal uncertainty reduction of æ given X.
Though theoretically this assumption may be violated by some instances (Hübotter et al., 2024, Example C.8), we observe that is satisfied practice (cf. Figure (right)). Under this assumption. Ve (Xx) 2 (1 1/e) maxx CD,jXign Va (X) due to the seminal result on monotone submodular function maximization of Nemhauser al. (1978). That is, the iterative scheme of SIFT achieves constant factor approximation of the optimal uncertainty reduction. Moreover, recent work on transductive active learning of Hübotter et al. (2024) which we restate here shows that the uncertainty of SIFT converges to the irreducible uncertainty. We assume w.l.o.g. that (< < for all æ E X.
Theorem C.2 (Convergence Guarantee, formalization Informal Theorem 4.1). Let Assumption C.I hold and Xn, be selected by SIFT(X') from the data space D. Then for all n 2 1 and x" E X, Ag X7
o (x=) S of (x*) d(1 + 2dX >
min log(1 +
Vn
where Amin is the smallest eigenvalue of ΦΦ with & E Rm basis of (Φ(x) D}, and where dn < (n) is the largest eigenvalue of 4b $
Proof. Theorem C.2 follows from Theorem 3.2 of Hübotter et al. (2024) noting that
The SIFT objective is special case of VTL (Variance-based Transductive Active Learning) with "target space" A = {x*}.
Theorem 3.2 of Hübotter et al. (2024) can be extended to finite-dimensional reproducing kernel Hilbert spaces (Hübotter al., 2024, Appendix C.6.4).
The "maximum information gain of iterations", "a. in the statement of Hübotter al. (2024) is bounded as follows (Srinivas et al., 2009, Appendix C.3): Y S doog(( (1 + X,/X).
"See Appendix K.4 for the expressions with non- normnommalzzed emheddings. Simiilr assumptions have heen made by Bogunovic (2015) and Kothawade al. (2020).
25

---

Published as a conference paper at ICLR 2025
D FURTHER INSIGHTS ON ACTIVE FINE-TUNING
We expand the analysis of our results that we summarized in $5. We analyze aspects the two key contributions of our work separately: In the following, we analyze the performance of SIFT in active fine-tuning, and in $E, we analyze the performance of test-time fine-tuning more generally.
Insight 6: SIFT's improvement over NN grows with dataset size. As shown in Figure 10, find that the relative improvement of SIFT over Nearest Neighbor retrieval grows with dataset size. We suspect that going from small-size dataset medium-size dataset, the additional performance stems mainly from the ability of SIFT to adaptively select the same data for multiple gradient steps. Going from a medium-size dataset a large-size dataset, we suspect that the additional performance stems mainly from the ability of SIFT to select more diverse data points.
100.0
Bits per Byte in %
97.5
95.0
3%
33%
92.5
100%
90.0
Test- Test-Timeeae Iterations
Figure 10: Bits per byte (in % relative to the Nearest Neighbor retrieval baseline, & better). We evaluate data selection from 3%, 33%, and 100% of the Pile training dataset. We see a clear trend that SIFT's improvement over Nearest Neighbor retrieval grows with dataset size even from 33% to 100% with the highly curated Pile dataset.
Insight 7: Points with high negative cosine similarity may help. With the Roberta embedding model, we find that there different embeddings such as influence embeddings can give negative cosine similarities (Xia et al., 2024, Appendix K.2). Inspection of those points found by Xia et al. (2024) suggests that they be equally informative as points with high positive cosine similarity. Our derivation of SIFT naturally addresses this by selecting points with large absolute cosine similarity. Geometrically, points with positive or negative cosine similarity are both equally "parallel" the test prompt. Our theoretical results suggest that the informativeness of data point is closely related how parallel its embedding is to the test prompt. We leave further investigation to future work.
negative cosine similarities in the data (cf. Figure 21 in $J). Choosing
Cosine Similarity
Bits per Byte
1.2
Inser Product
1.0
Insight 8: Normalizing embeddings helps. We eval- uate the performance of Nearest Neighbor retrieval and SIFT with without explicitly normalized embeddings in Figure 11. We find that for both selection strategies, normalizing embeddings consistently improves perfor- mance. Previously, Hardt & Sun (2024) minimized the Euclidean distance between unnormalized embeddings, which we find to perform identically to maximizing cosine similarity.
20
40
Test-Time Iterations
Figure 11: Data selection via SIFT (red) and Nearest Neighbor (black) performs best with normalized embeddings.
26

---

Published as a conference paper at ICLR 2025
E FURTHER INSIGHTS ON TEST-TIME FINE-TUNING
Insight 9: Scaling pre-training compute may not be all you need. In Table of $A, we compare state-of-the-art LLMs our test-time fine-tuned models. We show that our Phi-3 with test-time fine-tuning outperforms all evaluated base models, from a wide selection of state-of-the-art LLMs, by large margin. Notably, we see clear advantage of using stronger base models, i.e., better initializations. The leading base model Gemma-2 (27B, Team et al., 2024), which is 7x larger and more recent than Phi-3, achieves 0.629 bits per byte, whereas our test-time fine-tuned Phi-3 achieves 0.595 bits per byte. This indicates that scaling pre-training compute is not all you need to achieve state-of-the- performance, and that test-time fine-tuning can be an effective method for improving the performance of base LLM.
Insight 10: Test-time fine-tuning outperforms in-context learning in "hard" tasks. Interestingly, we observe that across all evaluated models, updating the base model via fine-tuning as opposed to augmenting the models' context leads to large improvements on the DeepMind Math, GitHub, ArXiv, and FreeLaw datasets. We include the per-dataset results in $F.2. These datasets contain school-level math problems, code, scientific papers, and court opinions, which are often colloquially understood tasks that require "undderstadding" and ArXiv, augmenting the models' context does consistently not improve the performance of the base model at all, whereas test-time fine-tuning can lead to significant performance improvements.
"reasoning". In the case of DeepMind Math
Test-time fine-tuning yields largest gains at the boundary of the data distribution. In Figure 12, we plot the improvement of test-time fine-tuning with SIFT over the base model against the weight of dataset in the Pile. We observe the trend that test-time fine-tuning yields largest performance improvements for datasets that have smaller weight in the Pile. We hypothesize that this trend occurs because the weight of a dataset in the Pile corresponds roughly the weight of similar data in the pre- vee-traiiing dataset of GPT-2, in which case the perfor- mance gains would be largest for prompts that are at the "boundary" of the data distribution. Notable is the outlier of the large Gutl dataset where test-time fine-tuning leads to large performance gains. We hypothesize that this because coding relatively dissimilar other data in the Pile, and therefore the GitHub dataset can be seen as "small" relative to the rest of the data.
Insight 11:
100
Bits per Byte in %
75
50
25
10
10
Dataset Weight in Pile
Figure 12: Improvement of 50 test-time iterations over the base model (blue; + better) with SIFT against the percent- age of bytes occupied by the dataset in the Pile. Error bars correspond to stan- dard errors. We observe the trend that test-time fine-tuning benefits prompts the "boundary" of the data distribution most. The "outlier" GitHub dataset is highlighted in red.
We make the observation that if the problem domain large (like general language modeling), almost every sub-task can be seen at the "boundary") / as an "outlier". We see that datasets closest to the center of mass of the data distribution do not benefit as much from test-time fine-tuning as datasets that are further away from the center of mass. Therefore, expect test-time fine-tuni fine-tuning to benefit those models most that are learning a diverse data distribution as opposed to models that are learning very concentrated data distribution.
The order of fine-tuning data does not matter. In Figure 13, we evaluate the performance of test-time fine-tuning with Nearest Neighbor retrieval when taking gradient steps in the order of selected data compared to reversed order. We find that the order of gradient steps does affect the final performance. This indicates that sequentially fine-tuning on selected data is not necessary, and that batched gradient steps can be used to further speed up test-time fine-tuning. We leave detailed exploration of batched updates future work.
Insight 12:
27

---

Published as a conference paper at ICLR 2025
In Order
Reverse Order
0.98
Cosine Similarity
Bits per Byte
1.2
1.1
0.97
1.0
0.9
0.96
20
40
20
40
Test-Time Iterations
Neighbors
Figure 13: Taking gradient steps in order of selected data compared reversed order. Data is selected using Nearest neighbor retrieval. We observe that the order of gradient steps does not affect the final performance.
Insight 13: Test-time fine-tuning works also when fine-tuning only the last linear layer. Motivated by the linear representation hypothesis (cf. Assumption 3.1) which informs SIFT's surrogate model for data selection, we evaluate whether we can fine-tune this surrogate model directly instead fine-tuning the full model. Concretely, fine-tune only the last linear layer of the LLM, keeping its latent space fixed. The gradients for this linear surrogate model can be computed efficiently almost cost. Re- markably, we find in Figure 14 that large gains of test-time fine-tuning already be realized by fine-tuning only the last linear layer. Given these preliminary results with GPT-2 would be interesting to evaluate the performance gains of fine-tuning the linear head of larger base models.
1.4
Linear Head
Bits per Byte
Full Model
1.2
0.8
0
20
40
Test-Time Iterations
Figure 14: Bits per byte (L better) against the number of test-time iterations. We compare fine-tuning only the linear head fine-tuning the full model. We use learning rate 1e-4 and evaluate parameter-efficient fine-tuning. In our experiments 0.1% of the full test set. with Phi-3, we use Low-Rank Adaptation (Lora, LoRA converges slower than fine-tuning the full model, and therefore use the learning rate 5e-4, which is a factor 10 larger than the learning rate used for fine-tuning the full model. In Figure 15, we evaluate the performance of LoRA compared to fine-tuning the full model. On the smaller GPT-2 and GPT-2-large we use a rank of 32. generally observe that fine-tuning with LoRA can recover roughly the same performance fine-tuning the full model. We expect that with more careful tuning of the learning rate, learning curves could be made more similar.
Insight 14: Test-time fine-tuning works also with
al., 2022) with rank of 64. We find that
with LoRA
without LoRA
1.2
Bits per Byte
1.2
1.0
0.8
0.8
0
20
40
20
40
Test- Come-Time&ae Iterations
Figure 15: Bits per byte (L better) against the number of test-time iterations. We compare parameter-
efficient fine-tuning with LoRA and fine-tuning the full model. We use 0.1 1% of the full test set.
28

---

Published as a conference paper at ICLR 2025
Insight 15: Larger models appear to learn faster at test-time. We find that with a larger model (e.g., GPT-2-large GPT-2), smaller X' tends be more beneficial. For example, keeping the learning rate fixed at 5e-5, using SIFT(0.1) is the best choice for GPT-2, but leads to slight overfitting at later iterations for GPT-2-large shown Figure 16. Recall that smaller X leads to more diverse sampling of the data space. Thus, this observed trend indicates that larger models learn faster, and therefore benefit more from less redundant training data. The same trend can also be observed from the behavior of NN-F from Figure 17: GPT-2-large overfits much faster with NN-F than GPT-2. This offers potential explanation why the advantage of SIFT over Nearest Neighbor retrieval grows with larger models (cf. $F.1).
X 0.1
Bits per Byte
X 0.01
1.0
0.8
0
20
40
Test-Time Iterations
Figure 16: Bits per byte (L better) with GPT-22llrre and varying X. larger X can lead to overfitting in later iterations. We use 0. of the full test set.
SIFT (ours)
NN
Failure-mode of NN
GPT-2
GPT-2-large
Phi-3
1.2
0.8
Bits per Byte
1.2
1.0
0.7
1.0
0.8
0.6
0.8
20
40
20
40
0
20
Test-TTmme Iterations
Figure 17: Bits per byte (L better) against the number of test-time iterations with various base models.
29

---

Published as a conference paper at ICLR 2025
F EXTENDED RESULTS
This section includes additional per-dataset results to support our findings on active fine-tuning and test-time fine-tuning,
F.1 ACTIVE FINE-TUNING TUNING
We compare SIFT against the data selection baselines Uncertainty Sampling (US), Nearest Neighbor retrieval (NN), and the mallure-mode of Nearest Neighbor retrieval (with information duplication) that repeatedly retrieves the same point (NN-F). Our results with GPT-2 as base model are summarized in the main text in Table 1.
Table we include the comparison with GPT-2-large.
In Table 5, include the comparison with Phi-3.
We find that our results on GPT-2 are consistent across all models. particular, test-time fine- tuning with SIFT improves the base model on all datasets of the Pile, often significantly. SIFT outperforms Uncertainty Sampling and Nearest Neighbor retrieval consistently. Notably, we find that the improvement of SIFT over Nearest Neighbor retrieval is larger with stronger base models, indicating that informativeness of data becomes more important the stronger the base model.
F.2 TEST-TIME FINE- TUN--TUNING
We compare the in-context baseline against test-time fine- fine-tuning.
In Table include the comparison with GPT-2.
In Table 7, we include the comparison with GPT-2-large.
In Table 8, we include the comparison with Phi-3.
We find that test-time fine-tuning consistently outperforms in in-context learning with GPT-2 and GPT- 2-large. With Phi-3, in-context learning and test-time fine-tuning have roughly matching performance, though test-time fine-tuning is more computationally efficient (cf. Figure 7). Interestingly. we observe that test-time fine-tuning leads to large gains on math ("DeepMind Math") and coding ("GitHub") on all models, two tasks that require more complex reasoning.
US
NN
NN-F
SIFT
A
NIH Grants
91.6
(7.9
107.6
51.9 (93)
26.0
US Patents
86.8 (23)
(2.9
129.1
(3.7
14.22
Enron Emails
73.9 (12.3)
68.6
102.9
55.5 (12.2)
13.1
GitHub
45.2
42.8 (2.2)
(4.5)
31.0 (2.2)
11..
Wikipedia
(2.0
71.5 (2.0)
141.3 (3.5)
64.4 (2.2)
66.6
PubMed Abstr.
94.5 (0.4)
93.7 (0.6)
202.6
87.8 (0.7)
45.9
ArXiv
90.2 91.6
175.8
84.8 (2.1)
45.4
Hacker News
79.4
79.0 (2.9)
138.7 (4.4)
75.6 (3.6)
J3.4
Stack Exchange
84. (0.7)
(65.2 84.6
80.7 (0.9)
J3.4
Common Crawl
93.7 (0.6)
89.9 (0.7)
163.6
87.1 (L.0)
12288
PubMed Central
87.9 (2.7)
87.6 (2.7)
157.8 (4.6)
85.4
[2.2
Freel aw
66.8 (4.2)
67.4 (4.1)
132.0
68.3 (4.2)
1.5
DeepMind Math
71.2
72.2 (2.0)
186.1 (4.1)
74.2 (2.3)
f..0
All
82.6
80.6
(1..3
74.9 (0.)
45.7
Table 4: Results with GPT-2-large. Bits per byte (in % relative to the base model, }) after 50 test-time iterations on individual datasets of the Pile. We only include datasets with at least examples in our test set. Bold numbers denote the best performing selected subset. Numbers in parentheses are standard errors. A denotes the performance gain of SIFT over the strongest baseline.
30

---

Published as a conference paper at ICLR 2025
US
NN
NN-F
SIFT
A
GitHub
80.6
80.8
105.2
46.5
344.1
US Patents
95.4
94.2
274.6
83.7
10.5
Enron Emails
113.6
86.6
319.9
78.7
47.9
Wikipedia
84.6
85.3
263.2
79.2
15.4
PubMed Abstr.
93.5
93.3
301.8
89.5
13.8
NIH Grants
100.4
100.1
327.6
98.6
41.5
ArXiv
95.5
96.5
282.4
94.3
41.2
Common Crawl
95.3
94.9
257.0
93.7
11.2
PubMed Central
80.3
82.1
204.9
79.7
10.6
DeepMind Math
76.4
75.5
221.4
75.3
10.2
Hacker News
95.1
94.8
243.8
95.0
1.22
FreeLaw
66.9
67.8
178.0
67.2
10.3
Stack Exchange
99.7
98.7
309.9
99.4
0.77
All
(L.)
91.6(1.1)
27..6
85.7 (2.0)
1559
Table 5: Results with Phi-3. Bits per byte (in % relative to the base model, 4) after 50 test-time iterations on individual datasets of the Pile. Bold numbers denote the best performing selected subset. Numbers in parentheses are standard errors. A denotes the performance gain of SIFT the strongest baseline.
Context
Fine-Tuning
GitHub
74.5 (2.5)
28.6 (2.2)
L45.9
DeepMind Math
100.4 (0.1)
70.1 (2.1)
L30.3
US Patents
86.8 (2.5)
62.2 (3.6)
124.6
Enron Emails
73.3 (9.8)
52.4 (11.8)
200.9
FreeLaw
85.5 (4.0)
65.5 (4.2)
220.0
Stack Exchange
96.7
77.0 (0.7)
119.7
ArXiv
99.2 (1.4)
81.6
117.6
Wikipedia
77.4 (2.1)
63.7 (2.1)
113.7
PubMed Central
92.8 (3.1)
80.6 (2.7)
11.22
Hacker News
89.0 (3.8)
77.8
111.
NIH Grants
63.7 (9.5)
$2.9.9.0
10.8
Common Crawl
93.4 (0.7)
86.1 (0.9)
47.3
PubMed Abstr.
91.8 (0.6)
84.8 (0.7)
$7.0
All
89.3 (0.5)
73.2 (0.7)
L16.1
Table Comparison between the in-context baseline and test-time fine-tuning with GPT-2. Bits per byte (in % relative to the base model, J) after 50 test-time iterations on individual datasets of the Pile. We only include datasets with at least 10 examples in our test set. Bold numbers denote the best performing selected subset. Numbers in parentheses are standard errors. A denotes the performance gain of test-time fine-tuning over in-context learning.
31

---

Published as a conference paper at ICLR 2025
Context
Fine-Tuning
GitHub
(2.6
31.0 (2.2)
443.
DeepMind Math
100.2 (0.7)
74.2 (2.3)
262
US Patents
87.4 (2.5)
64.7 (3.8)
2.2.77
FreeLaw
87.2 (3.6)
68.3 (4.2)
18.9
Hacker News
(2.6
(3.6)
117.0
Stack Exchange
97.2 (0.4)
80.7 (0.9)
J16.5
NIH Grants
67.7 (9.4)
51.9 (9.3)
11588
Enron Emails
71.9 (10.2)
55.5 (12.2)
15.5
ArXiv
98.8 (1.8)
84.8 (2.1)
14.00
Wikipedia
(2.6
64.4 (2.2)
12.2
PubMed Central
92.3 (3.3)
85.4 (3.1)
4699
Common Crawl
93.5 (0.7)
87.1 (1.0)
J6.4
PubMed Abstr.
91.6 (0.6)
87.8 (0.7)
$3.8
All
89.4 (0.5)
74.9 (0.7)
14.5
Table 7: Comparison between the in-context baseline and test-time fine-tuning with GPT-2-large. Bits per byte (in % relative to the base model, 4) after 50 test-time iterations on individual datasets of the Pile. We only include datasets with at least 10 examples in our test Bold numbers denote the best performing selected subset. Numbers in parentheses are standard errors. A denotes the performance gain of test-time fine-tuning over in-context learning.
Context
Fine-Tuning
DeepMind Math
100.8
75.3
225.5
GitHub
71.3
46.5
24.8
FreeLaw
78.2
67.2
$11.0
ArXiv
101.0
94.3
6644
Enron Emails
81.8
78.7
J3.1
Hacker News
97.6
95.0
22.6
Stack Exchange
100.9
99.4
41.4
PubMed Central
79.9
79.7
10.2
US Patents
83.3
83.7
10.4
Wikipedia
77.1
79.2
2.1
NIH Grants
95.1
98.6
3.5
Common Crawl
89.9
93.7
13.8
PubMed Abstr.
85.7
89.5
13.8
All
87.1 (1.7)
85.7 (2.0)
11.4
Table Comparison between the in-context baseline and test-time fine-tuning with Phi-3. Bits per byte (in % relative to the base model, !) after 50 test-time iterations individual datasets of the Pile. Bold numbers denote the best performing selected subset. Numbers in parentheses are standard error.. A denotes the performance gain of test-time fine-tuning over in-context learning.
32

---

Published as a conference paper at ICLR 2025
G SIFT MAXIMIZES INFORMATION GAIN
We discuss here briefly that SIFT can be interpreted as maximizing the information gain of data Xx, on the response the prompt x".
This probabilistic interpretation takes the perspective that the sequence model predicting the next token is a probabilistic model with a prior belief over its state W which induces an epistemic prior belief over what might be the next token." Our main text describes closed loop where this sequence model interacts with a con-paaamctiic memory (i.c., the data space) to update its epistemic belicf's about W, obtaining posterior beliefs W D conditional on the selected data D. Again, these posterior epistemic beliefs induce an epistemic uncertainty over what might be the next token. We discuss in the following how SIFT can be interpreted probabilisticall;; the model interacting with the non-parametric memory with the goal of reducing its posterior uncertainty about the next token.
Our brief overview will proceed as follows:
We establish fundamentals from information theory and Gaussian processes, which are tractable probabilistic model ($G.1).
We define the prior belief and probabilistic observation model and derive the posterior belief ($G.2).
We show that, in this probabilistic model, SIFT can be interpreted as maximizing the information gain of the data about the response to the prompt x² ($G.3).
We show that balancing relevance and diversity of data is natural consequence of maximiz- ing information gain ($G.4).
SIFT uses relatively simple probabilistic surrogate models that are tractable, and which remarkably lead to strong empirical performance. Hübotter et al. (2024) cover the probabilistic interpretation in greater detail.
G.1 PRELIMINARIES: INFORMATION THEORY AND GAUSSIAN PROCESSES
Information Theory We briefly recap several important concepts from information theory. The (differential) entropy H[f] = Ep(r) log p(f) of random vector f is one possible measure of uncertainty about f. Here, log p(f) is also called the suprisal about event with density p(f). The entropy can be interpreted as the expected suprisal about I upon realization. The conditional entropy H[f y) = Ep(r.w) log y)] is the (expected) posterior uncertainty about f after observing the random vector y. The information gain T(f; y) H[f] H[f y) measures the (expected) reduction in uncertainty about f due to y. Refer to Cover (1999) for more details.
The stochastic process is Gaussian process (GP, Williams & Rasmussen (2006)), denoted f ~~ GP( k). with mean function μe and kernel k if for any finite subset X (2)..... v,, C X. fx ~N(x, Kx) ) is jointly Gaussian with mean vector (μx); = p(x:) and covariance matrix [Kx j.f x();,,;), A Gaussian process can be interpreted as capturing an epistemic functional belief. i.c., belief over functions. Our linear surrogate model from Assumption 3.1 leads to Gaussian process with the linear kernel described in the main text. That is, our surrogate model assumption can be interpreted as the prior belief that the ground truth function predicting the next token is a logit-linear function in latent representation space. This closely linked to the hypothesis that LLMs learn linear representations of high-level concepts, which is widely known as the "linear representation hypothesis" (e.g.. Park et al., 2024; Mikolov et al., 2013; Arora et al., 2016; Elhage et al., 2022). There are two lenses through which to view such linear Gaussian processes: the weight-space view which considers belief about weights W. or the function-space view which directly considers the belief about functions f. Both views are equivalent, and we will focus the function-space view in the following.
Gaussian Processes
1 log det Var(f) and For Gaussian random vectors f and y. the entropy is H[f] = loge) + = the information gain is I(f: y) = 1 (log det Var(f) - log det Var(f y)).
This epistemic uncertainty is distinct from the irreducible aleatoric uncertainty of natural language, such as uncertainty about the continuation of "I love..."
33

---

Published as a conference paper at ICLR 2025
G.2 PROBABILISTIC OBSERVATION MODEL
We will focus in the following on the case of regression, which we introduced in Appendix K.5. We suppose that observations of follow the probabilistic model
y/2 fæ + Ex:
where we make the following assumptions about the prior distribution of f and the noise Ew: Assumption G.1 (Gaussian prior). We assume that f GP(u, k) with known mean function μ and kernel k.
Assumption G.2 (Gaussian noise). We assume that the noise Ea is mutually independent and zero-mean Gaussian with known variance p²
Under Assumptions G.I and G.2, the posterior distribution of f after observing points X with values yx is GP( kn) with
(x(x)) u(xx) (x)
Hxx);
(xx, x²) = k(x,, x') - (x) (xxx + p²1) -xxx==xxxxx))
of (x) = (x, x).
G.3 THE PROBABILISTIC INTERPRETATION OF SIFT
Observe that the above definition of o matches the definition from Equation (2). That is, under the above probabilistic model,
(x) = Var(f(x) Vim)..
As such, SIFT(p²) is minimizing the variance of the response to the prompt x" after observing the data Xn:
En+1 arg min Var(f(x") yl:n, y(x)).
#ED
By simple algebraic manipulation this can be seen to be equivalent to maximizing the information gain of the data on the response to the prompt x":
1 2
log Var(f(#*) Vinn) log Var(f(x*) yim, 8(x)))
En+1
arg max
aED
coust
arg max ((f(x=)); (<x) v1=)).
aED
Discussion
The above offers very intuitive probabilistic interpretation of SIFT(p²). In this prob- abilistic interpretation, the regularization parameter X' of SIFT is equal to the observation noise p². Intuitively, larger observation noise leads slower convergence of the estimate of analogously to our discussion of larger regularization parameter and smaller step size in Proposition 3.3.
The reason why SIFT(p²) can be interpreted both as minimizing the variance and as minimizing the entropy of the response to the prompt x* is that for Gaussians, variance is proportional to the entropy of the response the prompt x². As observed by Hübotter et al. (2024), if learning amortized with respect to multiple prompts (.i...., = A. this ceases to be the case and the two objectives lead to different data selection schemes. It appears to be a special property non-amortized transductive active learning that measures of uncertainty and resulting data selection schemes are interchangeable.
quick remark in order. SIFT does not only maximize the marginal information gain as shown in Equation (7), if Assumption C.I is satisfied, it also maximizes the joint information gain I(f(x*)) V1..). That is, in this case the "entropy reduction" " of data X,, selected by SIFT achieves constant factor approximation of the maximum possible joint information gain max maxxcp,|Xiss Xisw I(f(x)); y (X) due to the seminal result on monotone submodular function maximization of Nemhauser et al. (1978).
Notably, it also he shown that pa is the closed-form solution the regularized loss from Equation (10).
34

---

Published as a conference paper at ICLR 2025
G.4 How SIFT BALANCES RELEVANCE AND DIVERSITY
In 1, we discussed how SIFT chooses data that is both relevant and diverse. The probabilistic inter- pretation offers simple explanation for how this behavior naturally emerges from selecting the most informative data. To this end, observe that the information gain from Equation (7) can be expressed as
(f(("));(( yim) (1(x)));(()) (xf(x)); (1));
(8)
relevance
rodundancy
where I(f;; x; = xx) - [[f; x y) = I[{;=) I(f:y) I[f; y) denotes the multivariate information gain (Murphy. 2023). The multivariate information gain is measure of the redundancy of æ and y in predicting f. and is therefore often called simply "redundancy" (which is the opposite of "synergy"). Equation (8) shows that the balancing of relevance and non-redundancy (i.e., diversity) arises naturally from maximizing the information gain. Note that the tradeoff between relevance and diversity is governed by the noise parameter p² ~ X of the probabilistic model.
G.5 THE PERSPECTIVE OF CLASSIFICATION
The above interpretation takes the perspective of regression. However, the above interpretation can be extended to classification. We will focus here on the case of binary classification for notational convenience, but the same argument can be made for multi-class classification (Williams & Rasmussen, 2006, Section 3.5).
In (binary) Gaussian Process Classification the logit f ~ GP k) is modeled as Gaussian process, and the likelihood follows the model introduced in Section 3: y(z) (ern(s(f(z)) where we have Bernoulli rather than categorical feedback and use the logistic function s(a) = 1/(1+ 1e +e rather than the softmax by virtue of restricting to binary classification.
The standard approach (Williams & Rasmussen, 2006, Section 3.4) is to approximate the poste- rior distribution of the latent function f given observations Vi.. by Gaussian using Laplace's method. This Gaussian can be shown to have covariance (K + W) with W > x where su 1/a(a) and a(a) = s(a) - s(a)) denotes the derivative of the logistic function." 13 is then straightforward to derive that
(x k(x" x²) k (x)) 1W kx, (x") < k(x", x²) )- - kx, (x*) (Kx, + =I,,. n..,,,)1
(..)-1xxa 'kx, (x*)
Thus, SIFT minimizes a tight upper bound to the (approximate) posterior variance of the latent function f at the prompt x². The same relationship to maximizing information gain that was discussed above applies.
In the binary case, this equal the more general from the main
35

---

Published as a conference paper at ICLR 2025
H EFFICIENT COMPUTATION OF SIFT
In the following, we show how to select data via SIFT at low computational cost. Our implementation extends the Faiss library (Johnson et al., 2019; Douze et al., 2024) for Nearest Neighbor retrieval. We open-source the acti ve (Active Fine-Tune (ine-Tunigg) library which can be used as drop-in replacement for Nearest Neighbor retrieval.
In our runtime analysis, we will denote by K the size of the data space D, and by N the number of points be selected. We describe implementations of SIFT:
The first exact implementation has sequential computation cost O(K2N), however, compu- tation can be effectively parallelized on GPU.
2. The second "fast" implementation assumes submodularity (i.e., Assumption C.I) and has computation
O(K + No) where O(+) suppresses log-factors. This cost only marginally above the cost of Nearest Neighbor retrieval.
Both implementations achieve virtually identical performance gains (ef. Figure 19 (right)), which is further evidence that Assumption C.1 satisfied in our language modeling setting.
H.1 EXACT IMPLEMENTATION
The central object of the first implementation is the conditional kernel matrix of the data space given the selected points Xn:
Kn KD - KD. x,, (Kx, + +XIIn)''Kx...
The entries (æ, x') of this matrix can be updated efficiently via the following relation (Chowdhury & Gopalan, 2017, Appendix F) arising from properties of the Schur complement:
(x, x') = 1n-11 x') 1 F.n -nn
99)
1 xx,)) + X'
The implementation is detailed in Algorithm 1. The computation of the objective value line and the kernel matrix update in line 5 can be parallelized on GPU. Thus, the main bottleneck of this implementation is the requirement that the kernel matrix of size x fits onto GPU. In case this is not possible, such as with large data spaces, the following sections detail methods to reduce the computational cost.
Algorithm 1 SIFT(X)
1: Input: prompt x", data space D, (initial) kernel matrix ko(ær, x²) ((xx) (<x))(((²)) x, x' DD number of points to select N
2: Output: set of selected points ( ØNN
3: for from to do
4: En f- arg maX Tn-1 (x,,)))
{Select next point}
5: for each ar ax 'D do
6: Update k, (zx x') +- knn ( x²)
en (00 Men (x)) kn 16 +N
{Update kernel matrix)
7: end for
8: end for
H.2 FAST (EXACT) IMPLEMENTATION
The following "fast" implementation SIFT rests on the assumption that the objective function optimized by SIFT is submodular (cf. Assumption C.I). Recall that this objective function can be expressed 2 arg max Vha (X,, {æ}) where vha (X) (x+) ox (x*) denotes the uncertainty reduction about x upon fine-tuning the model on data X.
The "trick" of the fast implementation is use max-heap (with O(1) lookup and Oloog K)
insertion) to keep track of upper bounds of vha (X,, U (z))) for each x E D. The upper bounds come directly from the submodularity assumption:
V/ (X₂ (m)) 2 tha (X, (x)) Vj 2
36

---

Published as a conference paper at ICLR 2025
At iteration n, evaluate Ve- (Xn-1 U {x}) for in max-heap order. As soon find whose re-computed upper bound is smaller than a previously re-computed upper bound, we stop the evaluation. In the worst case, one might iterate through all K points in each iteration, but in practice, it can sometimes be reasonable to assume that one only needs to consider O(1) points per iteration. This algorithm is known the "lazy greedy algorithm" in submodular function maximization (Minoux, 1978) where it is typically seen to result in large speed-ups.
We summarize the fast implementation in Algorithm 2. The kernel matrix K tracks the conditional kernel matrix of the prompt xt and the previously selected data X A tracks the (regularized) inverse of the kernel matrix of the previously selected data Xn Whenever necessary. the cached kernel matrix and cached inverse are updated. We denote by E (n-1)xd the matrix of embeddings of previously selected points and by o E inxd the same matrix extended by ((x") ) as the first row.
Initializing the max-heap takes time Ö(K) and is analogous to standard Nearest Neighbor retrieval. Additionally, SIFT-FAST performs a data selection loop for N iterations where each operation takes O(N3) time requiring persistent memory of size ( Notably, only the kernel matrix of the prompt and the previously selected data kept in memory.
Algorithm 2 SIFT-FASTCH
1: Input: prompt æ", data space D, number of points to select N 2: Output: set of selected points ( EN
(Initializing max-heap ("Nearest Neighbor retrieval"))
3: for a E D do
4: Ote f (b(=") d()))
[((())] 3+
5: Insert (z, into max-heap
Ozze
6: end for
(Data selection]
7: Initialize K [|1(b(x=)11 )12) and A an empty square matrix
8: for from to N do
9: Initialize lower bound a +- 000
10:
for each popped (ar, ax) in max-heap order do if a. then
12:
x: break end if
{ar maximizes the SIFT(X) objective)
13:
14:
Owz, K' +- RECOMPUTE(2,, K, 16: a' max(a", )=)
15:
|Recompute objective value)
17: Insert (xt, Omx) into max-heap
18: end for
19: K < UPDATESTATE K')
UUpdate cached kernel matrix]
end for
H.3 PRE-SELECTING DATA VIA NEAREST NEIGHBOR RETRIEVAL
The reason for SIFT-FAST being efficient is that it effectively "discards" all points in D that are completely irrelevant to the prompt. Whereas SIFT recomputes the objective value of every point in D at each iteration, SIFT-FAST only reevaluates points that are potentially relevant. An alternative to make SIFT fast is therefore simply to preemptively discard irrelevant points. In our experiments we do so by pre-selecting a subset of size K 200 via Nearest Neighbor retrieval within D (cf. Appendix for more details). This step aims to eliminate all points from the data space that SIFT would not end up picking anyway while retaining a diverse set of relevant points. Figure 18 shows the effect of K on statistical performance and Figure 4 shows the effect computational performance.
H.4 FUTURE WORK: IMPROVING GPU UTILIZATION OF SIFT-FAST
In our experiments on the Pile dataset, we find that SIFT-FAST is less efficient than SIFT (cf. Fig- 19 (left)). We attribute this the fact that for any given prompt, the closest neighbors in the
37

---

Published as a conference paper at ICLR 2025
Algorithm 3 SIFT-FAST(X) RECOMPUTE
1: Input: prompt x², current iteration n, candidate cached kernel matrix K, cached inverse A
2: Output: objective value ax updated cached inverse A, expanded kernel matrix K
[Expand cached kemel matrix K (if required)
3: if has not been selected yet then
4: (Update A with the Saaimma---oorsooooooollll formula (Sherman & Morrison, 1950))
5:
Let denote the size of A
6:
if < n then
7:
A f- 4 i+1m-1
8:
B <- 4 Pi+1m- 4 1-nn11
9:
C
B A AA)-
A + AACA A
-ACC
10:
A
-CA
11:
end if
(Expand kernel matrix K}
12:
A I T
A
13:
k - A
K
k
14:
K
k
|
15: end if
16: Og f
(a(=".a))
{Compute objective value using the relation from Equation (9))
k(x,))X
Algorithm 4 SIFT-FAST(Y): UPDATESTATE
1: Input: selected point Tn, expanded kernel matrix K'
2: Output: conditional kernel matrix K
(Update kernel matrix using the relation from Equation (9))
3: for each æt x' E {x=1 U X,, do
4: Update k(x, x²) -- ('(xxx')
(20.000. W/ (20.. RC K(ann (2t +
5: end for
90
SIFT (ours)
85
NN
Bits per Byte
80
75
70
200
10000
6000
K
Figure 18: We SIFT (X' 1) with various values of K and report the bits per byte (4 better) after 50 test-time iterations. We find that performance on the Pile plateaus after K 1'000. Even at K 50, which equals the number of points selected, SIFT outperforms Nearest Neighbor retrieval due to being able to select the same points multiple times.
38

---

Published as a conference paper at ICLR 2025
Compute Overhead
Training Retrieval
SIFT-FAST
Only Retrieval
Bits per Byte
1.2
SIFT
10'
1.0
10
50
200
1000
000
0
10
20
30
40
50
Size of Pre-Selccted Data Space
Test-Time Iterations
Figure 19: Left: Computational overhead of SIFT-FAST over Nearest Neighbor retrieval. This overhead is larger than the overhead of SIFT over Nearest Neighbor retrieval (cf. Figure 4). Right: SIFT-FAST achieves identical statistical performance SIFT, which is further evidence that Assumption C. is satisfied in our language modeling setting.
data space all relatively similar the prompt (cf. Figure 20), meaning that each iteration of SIFT-FAST has to loop (sequentially) over the entire priority queue. In contrast, SIFT performs this operation in parallel on GPU.
We believe that a promising computational approach is to combine the advantages of the SIFT and SIFT-FAST im- plementations. This could be achieved by keeping a large sub-selected kernel matrix on the GPU (akin to the SIFT implementation) and selectively using the SIFT-FAST im- plementation points on the priority queue that are not in the sub-selected kernel matrix may be selected. This would allow for more efficient use of the GPU mem- ory of SIFT-FAST, which expect to yield comparable computational performance to the SIFT implementation in most cases, while still being able to handle large data spaces.
1.000
Cosine Similarity
0.975
0.950
Pile
0.925
Indiv. Dutasets
0.900
10a
10a
10a
10a
Neighbors
Figure 20: Average cosine similarities of test prompts to closest 1 100 neighbors in the data space of the Pile; with the Roberta embedding model.
39

---

Published as a conference paper at ICLR 2025
I EXPERIMENT DETAILS
We fine-tune the pre-trained model for a single gradient step each on = 50 selected data points. We evaluate the performance on 1% of the test instances of the Pile. We use the Pile training dataset as data space for data selection, which notably does not include data from the validation and test sets.
Evaluation We brary (Gao et al., 2024) for computing the bits per byte. This implementation computes the log-likeldhoo of document using rolling-windse window approach, which ensures that the model's maximum context window is fully utilized.
the standard implementation of the
li-
Truncation Long Sequences Analogously Hardt Sun (2024), to generate embeddings, we naively truncate long sequences to the maximum sequence length of the embedding model, that is, we only consider the prefixes of long sequences for data selection.
Learning Rate and Optimizer Following Hardt & Sun (2024), we use the Adam optimizer (Kingma & Ba, 2014) with e-value 1e-8. We use the default learning rate 5e-5 of the transforme formers rs library (Wolf et al., 2020) unless noted otherwise. Hardt & Sun (2024) used a learning rate of 2e-5 for their experiments. We show Figure 24 that 5e-5 leads to strictly better performance of the Nearest Neighbor baseline. In our ablation study over metrics for Nearest Neighbor retrieval (cf. Figure 11), which was conducted concurrently, we still used learning rate 2c-5 of Hardt & Sun (2024).
Low-Rank Adaptation (LoRA) We use LoRA (Hu et al., 2022) for fine-tuning Phi-3, and also eval- uate the performance of LoRA with GPT-2 and GPT-2-large (cf. $E). We use LoRAs with rank 64, out- put scaling 16, without dropout and bias. When fine-tuning with LoRA, we use the learning rate 5e-4.
gradient checkpointing (Chen et al., 2016) for fine-tuning Phi-3 to reduce memory footprint and allow fine-tuning on our hardware.
Gradient Checkpointing We additionally
Uncopyrighted Pile Dataset We use only those datasets of the Pile where with the terms of service of the data host (Gao et al., 2020). This excludes the Books3, BookCorpus2, OpenSubtitles, YTSubtitles, and OWT2 datasets.
is in compliance
We provide an overview of all hyperparameters of test-time fine-tuning in Table 9.
Model family
GPT-2
Phi-3
Llama-3.2
X
0.01
0.01
0.01
Learning rate
5e-5
5e-4
1e-4
Adam's s e-value
1e-8
1e-8
1e-8
Max. sequence length (in tokens)
1024
4096
4096
LoRA
yes
yes
Gradient checkpointing
no
yes
yes
Table 9: Hyperparameters during test-time fine-tuning, unless noted otherwise.
40

---

Published as a conference paper at ICLR 2025
1.1 PROPERTIES OF THE PILE DATASET
Figure 21 shows the average cosine similarities of test prompts to neighbors in the data space of the Pile. Table 10 shows the weight of each dataset in the Pile.
1.0
Cosine Similarity
0.5
Pilc
Indiv. Datasets
0.0
0.0
0.5
1.0
1.5
Neighbors
x10
Figure 21: Average cosine similarities of test prompts to neighbors in the data space of the Pile; with the Roberta embedding model.
Weight
Common Crawl
24. 14%
PubMed Central
19.19%
ArXiv
11.94%
GitHub
10. 10%
FreeLaw
8.18%
Stack Exchange
6,84%
US Patents
4.87%
PubMed Abstracts
4.09%
Project Gutenberg
2.89%
Wikipedia
2.04%
DeepMind Math
1.65%
Ubuntu IRC
1.17%
EuroParl
0.97%
Hacker News
0.83%
PhilPapers
0.51%
NIH ExPorter Grants
0.40%
Enron Emails
0.19%
Table 10: Overview of datasets the (uncopyrighted) Pile. Weight the percentage bytes in the final dataset occupied by each dataset. Numbers are taken from Gao et al. (2020) and renormalized.
1.2 IN-CONTEXT BASELINE
In our evaluation of in-context learning, use the following format to insert the selected data into the context of the model: We separate all retrieved token sequences with the string "\n)nn" which can be seen a paragraph separator, and additionally add this string between the data string and the prompt.
Notably, our results with in-context learning on GPT-2-large outperform the results previously reported by Hardt & Sun (2024). We suspect that this is due to combination of more reasonable evaluation and using SIFT as opposed to Nearest Neighbor retrieval for data selection.
Evaluation of Inference Cost of In-Context Baseline We estimate the inference cost of in-context learning as follows. We evaluate the time takes compute the rolling log log-liiel of the test instance with context included and subtract the time it takes to compute the rolling likeliieldood of
41

---

Published as a conference paper at ICLR 2025
the test instance without context. This is lower-bound of the inference cost of in-context learning, as unlike autoregressive generation, computing the log- ligelikeihoodd partially parallelized.
To compute the token throughput of the in-context baseline, divide the total compute time by the number tokens added to the context.
1.3 INFERENCE COST WITH TEST-TIME FINE-TUNING
Figure 22 evaluates the inference cost of test-time fine-tuning on all the Pile and the largest datasets.
GPT-2 (124M)
10a
10
GPT-2-large (774M)
109
10-1
Test- Time Compute (seconds)
Phi-3 (3.8B)
100
10
Llama-3.2 (IB)
100
10
Llama-3.2 (3B)
10
10
All
ArXiv
FreeLom
GuHHub
Common Crawi
PubMed Abstracs
PuliMeed Central
Stack Exchange
US Patents
Wikipedia
Figure 22: Cost of taking a single gradient step. Results are with an NVIDIA GH200.
42

---

Published as a conference paper at ICLR 2025
J ABLATIONS
This section summarizes ablations that we conducted to investigate test-time fine-tuning and SIFT.
Hyperparameter X/: Table 11
Learning Curves for Individual Datasets of the Pile: Figure 23
Learning Rate: Figure 24
Uncertainty Estimation:
Summary of correlations (Table 12)
Visualization of o, (Figure 2.5)
Compute-rooportional Performance Gain:
Details on ADAPTIVE SIFT (Figure 26)
43

---

Published as a conference paper at ICLR 2025
Table 11: Percentage of bits per byte after 50 test-time time iterations for varying X' relative to the bits per byte of the base model. We only include datasets with at least examples in our test set. Bold numbers denote the best performing selected subset. Underlined numbers denote better or on- on-par par performance with Nearest Neighbor retrieval. A denotes the performanee gain Sf SIFT with the strongest X per dataser over Nearest Neighbor retrieval. Numbers in partentheses are standard errors. We remark that X' is on a logarithmic scale. For any choice of X' E [le- 8, 10), SIFT always performs at least on-par with Nearest Neighbor retrieval.
432.00
118.1
413.5
412.0
4.4.3
13.8
43.6
2288
42.2
$1.5
11.4
(.0.6)
L.0.6
45.44
(211 91.6
60.3(1.9) 108 (6.6)
42. 42.1 (2.0) 53.2
91.6
148.8
(6..8
121.2(3.5)
162.6(1.3)
155.6(5.1)
141.9
133.1
121.8
122.4
(3.33
NN-F
64.4
90.4
85.0c1.6
66.3(2.0)
87.2
81.7
78.2
79,2
69.6.2.1)
64.1
(8.3
NN
10'000
93.5 93.5 (16.9)
105.4 (6.4)
51.3
89.4 (20.4)
146.2
158.0
(3.71
(1.49
151.6
136.9
131.3
103.5
17..1
12.28 (1.2)
64.5 (10.9)
72.9 (4.2)
59.011344
99.3
94.62.8)
77.5
102.0
98.6(3.))
85.9
86.2
71.9
85.3
74.9 (0:3) (0.)) 85.6
100
36.1
10
54.2
63.2
29.2
(12.2)
88.3
81.2 (1.8) 82.8
(5.6
86.7 (0.7)
83.8
78.1
78.4
(2.1
(4.0
53.3 (9.1)
62.7 (3.7)
28.6 (22)
53.8 (1.2)
87.8 (0.9) (..9)
64.8 (2.2)
86.4 (0.7)
82.0
(200
(4.7
74.3 (8.7)
77.8
78.1
0.1
52.9 (9.0)
62.2 (3.6)
28.6
52.4
(6.1(0.9)
81.6(1.9)
104.2(30) 64.921)) (22) 62.7 (3.7
84.8(0.7)
80.6
77.0
77.8
70.1
65.5
73.2
0.01
(9.0 (6.7 53.8
82.9 70.2 62.9
414 (2.9 30.0
(165) 68.6 (3.1 53.1
91.0(05) 90.7 87.5
(1.) 83.6 (1.5
87.0(04) 87.0 84.4
80.5 80,0277) 79.5
77.6 77.6 (0.7
78.8 (2.9 78.4
69.7
(4.0
76.7 76.7 (0.6) 73.5
69.3 (2.1
102.5 (3.0) 63.5
le-4
le-8
(29 (8.5
le- 12
(6.)
119.9
54.6(3.1)
117.9
(70.
(12.3
(4.9
Smnck Eicchnnge (18.0(17))
(13.9
(6.)
Common Cawd
Pab Abstr:
PubMed Central
DespMind Math
Enron Emails
Hacker News
NIH Grants
US Potents
Wikipala
Freel no
Gidllb
ArXiv
Au
44

---

Published as a conference paper at ICLR 2025
SIFT (ours)
NN
Failure-mode
NIH Grants
US Patents
GitHub
Enron Emails
Bits per Byte
1.0
1.00
1.5
1.5
0.75
1.0
1.0
0.5
0.50
0.5
0.5
25
50
25
50
25
50
0
25
50
Test- Time Iterations
Figure 23: Performance in some of the datasets of the Pile, with GPT-2 as base model. Error bars correspond to standard errors.
le 6
2c
..... 7) le 1
17
5e
11 5e-5
77 5e
1,2
Bits per Byte
1.0
10
20
30
40
50
Test-Tima Iterations
Figure 24: Ablation the learning rate with data selected by Nearest Neighbor retrieval. We find that the default learning rate 5e-5 of the trans formers library (Wolf et al., 2020) works best, and conduct our other experiments with this learning rate unless noted otherwise. Hardt & Sun (2024) had previously used 2c-5 which find to be suboptimal.
10
1.25
Bits per Byte
1.00
0.75
0.50
10-1
Uncertainty Estimate xx(x")
Uncertainty Estimate o (a".
Figure 25; visualize the predictive ability the uncertainty estimates o,, analogously to Figure
45

---

Published as a conference paper at ICLR 2025
Spearman
Pearson
all steps
0.485
0,421
final step
0.496
0.443
all steps 0.722
0.581
on
final step
0.682
0,482
all steps 0.485
0,468
log On
final step
0.496
0,466
all steps 0.722
0.618
log on
final step
0.682
0.526
Table 12: We find strong moderate correlation between the uncertainty estimates o, / a and bits per byte. We further consider the correlation all test-time iterations (from 50) as well as only at the final iteration. We report both the Spearman and Pearson correlation coefficients, measuring monotonic and linear relationships, respectively. Before determining the Pearson correlation, we exclude the 0.25% of the data points with the lowest and highest uncertainty estimates to avoid the influence outliers. The p-value all correlations is below 1e-5 due to the large sample size.
2.0
#
1.0
Bits per Byte
0.75
0.5
10
0.35
0.25
ADAPTIVE SIFT
SIFT
0.15
10
10a
20
Test Time Compute
Test-Time Compute
We evaluate ADAPTIVE SIFT with the same choices of in Figure (right). Left: Bits byte of ADAPTIVE SIFT ( better) against test-time compute. Every marker corresponds to the performance of ADAPTIVE SIFT with given a, where the associated test-time compute the average number of test-time iterations prompts. We compare ADAPTIVE SIFT to SIFT, where spend the same test-time compute on every prompt. We see a slight advantage of ADAPTIVE SIFT the prompt. Our current experiment exhibits
Figure 26:
over SIFT, due to adaptively stopping depending a bias as test-time compute approaches 50, since we force-stop the compute at 50 iterations. This biases ADAPTIVE SIFT to perform similarly to SIFT. We hypothesize that the initial advantage of ADAPTIVE SIFT over SIFT may grow with more test-time compute compute is not force-stopped at 50 iterations. Right: Frequency of stopping at given iteration for given values of
46

---

Published as a conference paper at ICLR 2025
K
PROOFS
This section provides the formal proofs of the results presented in the main text.
S. proves the insufficiency of Nearest Neighbor retrieval (Informal Proposition 2.1).
$K.3 shows the close relationship of regularized loss minimization and test-time fine- tuning (Proposition 3.3).
$K.4 details how SIFT balances relevance and diversity ($C.1).
$K.5 states confidence sets for fine-tuning regression models that are analogous to the confidence sets for classification from the main text.
$K.6 derives the confidence sets from the main text (Theorem 3.2).
K.1 NOTATION
Throughout this work, log denotes the natural logarithm. Unless noted otherwise {...} denotes multiset. We define the feature map 4n, - = (((()),,..,(,)))) E goxd which gives rise to the kernel matrix K,, Kx, $ ob E R and the covariance operator Σn = $ $ Rdxd
K.2 INSUFFICIENCY OF NEAREST NEIGHBOR RETRIEVAL (INFORMAL PROPOSITION 2.1)
We refer $C.2 for the formal definition of the irreducible uncertainty 0 (x=";
We remark that if embeddings are of umit length, the cosine similarity scoring function is equivalent the (negative) Euclidean distance scoring function:
a: (x* Tx=" x) ||00| ||0|| 2x. x = 22cos(x",, (x), x))
We henceforth consider the Euclidean distance scoring functiion
Proposition K.1 (Insufficiency of Nearest Neighbor Retrieval). Suppose w.l.o.g. that ((x) a. Consider the data space D = U D; where D; = {e; E N) with e₁ the i-th basis vector of Let x (4+(d-1) (2, 12... 1) Rd.
Then, for all n 1:
1. If Xn are the nearest neighbors of xx in D, [²(x)) 2 (2t;; > 0.
2. If X is selected by SIFT, oz " (x ;D) = 0.
Proof.
1. Clearly, ||= - e1l2 < || - e
_ for all i >>>> 1. Hence, X,, = {er 1 i [n] C Dj. This
is as the data space was restricted to Dj, and hence (x²)) 2 02 (
2. This follows readily from Theorem C.2 and noting that span D Rd. implying 22 (D)
Discussion The setting examined in Proposition K. 1 is an extreme case (where data exists with exact duplication), yet we deem that it illustrates a realistic scenario. Particularly nowadays that similar information is accessible from many sources in different forms, it is crucial to explicitly select diverse data for fine-tuning. We show here theoretically and in Appendix I.. qualitatively that SIFT does not have this limitation.
We remark that (...) denotes multiset.
47

---

Published as a conference paper at ICLR 2025
K.3 THE CLOSE RELATIONSHIP OF REGULARIZED Loss MINIMIZATION AND TEST-TIME FINE- TUNING (PROPOSITION 3.3)
Proof. We note that the regularized negative log-likelihood loss L from Equation (1),
x w W l
L (W: =-
log sy W (x())) (æ.y)ED
Σ
C(W
strictly convex in W and has unique minimizer WA which satisfies
V (W D) VL(WA; D) X(W, WP = 0.
It follows that WA- = W V WLWW: D).
Finally, recall that W, 7 Wore WVLCwpee D). We obtain
W1/n Wolle = | C(W D) 7) D)lle
= 7/| V L(Were; D) V VL(W1/1) 1/ni D)llr.
K.4 How SIFT BALANCES RELEVANCE AND DIVERSITY
1st point For non-unit length embeddings, the first selected point can be expressed follows: as X 0 as X co.
(Ø(x*) (()))² {{x))²
(kalart. x²)
31 arg min of (x0) arg max
( + + arg max RED
(o)(==))<(()))))))))))))))))))))
RED
RED
2nd point Next, we consider the second selected point. We derive the results in terms of the ( Tx' which such that k(æ, x²) Lx x') for unit length
dot product kernel k(xxx')= = ((x) embeddings. Let æ be such that k( x) O. We have
k((=", x1)
X
k(xr", 2t1) k(æ,, xt1)
th (011
k(xe", x1)
1
[+
kl@* x11) k(x= x11)
1+ X
k( x1) k( x1
(1 X/2 -
-1
+
2Xk(x" xz1)²
(1+
2k(x*, 121)²
2+ X
For æ, we have
kk(#" x11) k(0", x))
1+X
0
Tk(w" x1)
te (101, x() =
0
1+X
kk x))
1
k(ot,an x1)
1+X
[k(2", x1) k(x= x))
(1 + X)2 k(xx", x))
0
1 X
k(x" x1 2 + k(2" x²
1+ X We that is preferred over x= if and only if k(x=, x1 ) + k(xx, xx)² 2k(x* x1
k(x*, x)2
k(x x1 )2
I+X
2+ X
2 x+X x <(X)
As X' + xe, c(X') + 1; whereas as X' + 0, c(X' + 0.
We interpret the expressions extensively in Section 4.
48

---

Published as a conference paper at ICLR 2025
K.5 CONFIDENCE SETS FOR REGRESSION
Before moving on to deriving confidence sets for the setting with categorical feedback, state analogous results for the regression setting under the following standard assumptions. For ease of notation, consider the scalar case.
Assumption K.2 (Linear function in a known latent space). We assume (*) )(x) "* E Rd and where ( E Id denotes known embeddings. We assume that w has bounded norm, i.e, ||w0 uree| la B for some finite B E
T Tw* with
Assumption K.3 (Sub- Sab-Gaussian Noise). We assume that the data follows
Vn (x:) En
where each En from the noise sequence {ex m is conditionally zero-mean p-sub-Gaussian with known constant O. Formally,
X²
Vn 2 1,XER: Biear Da-i] S exp
2
where Dn corresponds to the o-algebra generated by the random variables {a, M and an-
We consider the standard squared loss L(w; D) =
(f(x - y)² where we write
f(;; w) <bw) w. The regularized loss with minimizer w,, is then
> (aw Dr) = L(w; D.) + llf w 2ro
(10)
where A > 0 is the regularization parameter. In the following, we write fn (x) - f(æ; w.)) and fore (a: = f(x; wPee). The closed-form solution to the optimization problem from Equation (10) is well-known (see, e.g., Williams & Rasmussen, 2006, Section 6.2.2) to be
(x) = (x) + kx,, (xx + - - f
where pi is the vector of predictions of frre at X, and yn is the vector of observations Dr--
The below result is almost immediate consequence of the results of Abbasi- Yadkori (2013) and Chowdhury Gopalan (2017).
Theorem K.4 (Confidence Sets for Regression). Pick 5 E (0, and let Assumptions K.2 and K.3 hold. Let
Ba( B PV 2(7 + +1+ logg(11)))
where To max mn 1 log det (1, + --- Then
P(vn 21, x: (x)) [x(x) s (a()))n(x)) 2 8.
Proof. Let us define the residual of the ground truth and pre-trained model as f() (z) fr with corresponding weight vector it. Analogously, let ÿn = f fxx t be the observed error. We have that i² wn wre E Rd with norm ||ur upreel 1| The unbiased estimate of the remaining error is
j. (x, + - Dn.
By Theorem 2 of Chowdhury & Gopalan (2017), for all æ E X' and 2 1, jointly with probability at remains now only to observe that
least 1 5, 1f'(())) - (x)) < (5) (z))
[f*(x) - (+) = (x: - fa(x))
49

---

Published as a conference paper at ICLR 2025
K.6 CONFIDENCE SETS FOR CLASSIFICATION (THEOREM 3.2)
We begin by re-stating Corollary of Amani & Thrampoulidis (2020). Analogous results can be obtained from Theorem of Zhang & Sugiyama (2023). Substantial work has studied the special case of binary feedback, K = Faury et al. (2020); Pásztor et al. (2024).
Let A(;; W) E R be the matrix satisfying (a( W)). = s;(2; w) - sx; W)). Equivalently, A (a;; W) - diag[ s (:; W)} s(;; W {s W)T. Based on this matrix, we define = SuPzex,ye WPee..WEWa WEW Amax (a;;; W)) and x = S WeW, 1/ 1/Amin ( (A(æ; W)).
Lemma K.5 (Corollary of Amani & Thrampoulidis (2020)), Assume W E WB and W = 0.
Let 8 E (0, 1) and set
Bn(d) = VX
((00==X(BB) B
(B(B+
2K3
2
log
1
(11)
2vK
VX
dA
Then,
PP where Vn = En + kALa-
PP vn 2 1,,EE X: (x)( s2()ll1 S 22L x(1+ 2B) [0(x)] v.
21-5.
Our result follows from two auxiliary lemmas.
Lemma K.6. For any 8, s' E RK drv (8,88) s 2K
2 ||s 8'112
Proof. We have
K
VK |
drv (8, = 1| 8'11,
81 1 2 V
( s/
s'1l2
2
t=1
where the inequality follows from Gauchy-Schwarr
The following lemma is standard result in the literature (Srinivas et al., 2009; Chowdhury & Gopalan, 2017; Pásztor et al., 2024), which we include here for completeness. Lemma K.7. Let an be as defined in Equation (2). Then, /=>ll<(=>llv (x) for any x E X.
Proof. Note that (E,, R - TTTTK, KAI,) which implies + kx -b " (K, =AI,)) Further, by definition of kn kn (2x) = (())) which permits writing Emn xxLa) x kn(x) x
(12)
and implies
(xx)
En + RAI 1 kz + =X(E, AAI a) (pxx)
(12) 12 Kn + - kn
+>) +nAa)) ((x)
(13)
We have
k(xr, m) ((x) Tx(())
(13) ( Kn, + kAI,) - RA(E, + kAIa)
-A (xx)
(x) (Kn + kAI,) kx) kXΦ(Z) (En, +RAla) -xX(a)-'o ((xx)
=x) (Kn + kAIn)
(x) + k)Φ(2)) Vo-(o(z)
Reordering this equation, we obtain
ελ || = =AΦ(2) T V, (xx)) = k(xxx) kx) (K, + RAI -
k,(x) = o.
concluding the proof.
50

---

Published as a conference paper at ICLR 2025
We now proceed to prove version of Theorem 3.2 with W 0. Theorem K.8. Assume W E Wg and W O. Ler E (0, and 3x(() as in Equation (3).
Then
P(vn 2 1, & dry((s())) s BL(() on():)) 21 8.
Proof. We have
VK
dIv(sn(z), sx)) -
[s, (z) s" s2(2l
(Lemma K.6)
2
w.h.p
L3, (KK(1 + 2B) |l
(Lemma K.5)
K(1 2B) X
L3x(0))
(2)).
(Lemma K.7)
It remains to note that
K(1 + 2B) (6)
2LK2dv1+ 2B
2
LVKLLIII K(1 +11+220 2B)
B +
log
1 +
X
2vK
ax
LK3/2d
2
2 (1BBB)
B+
log
1+
(0)
dd
With this we are ready to prove Theorem 3.2.
Proof of Theorem 3.2. We will proceed analogously to the proof of Theorem K.4. That is, our objective will be to bound the deviation of our biased model, which refer W. arg min L\WW' D,), to W Let
2(W';D) - Σ
log ((W' + )(()) and ('(W'; ž( D) + W
(x.v)ED
with minimizer W = arg migmiw::wwl (W'; Dz). We further define the residual weights W
W Wpre.
Next, make the following observation: In their proof of Lemma K.5, Amani & Thrampoulidis (2020) bound
[f W s(f(==(((])) W W S const ||vee vec (o,,,,, (WW")llaww ww)
(14)
where const is independent W Wre W' n and the matrix G(W", W is invariant to a change of variables, i.e., G(W", W G(W* W + WPre with G defined with respect to the loss _ and G defined with respect to the loss L*. Theorem K.8 applies to sf W()) and s( f(: W WP since the regularization of E is unbiased and the residual weights satisfy W || = W Wree Ip < B by assumption.
Since W W V (W", - WP'C), the bounds of Equation (14) as well as Theorem K.8 then also apply s fz(;; W + #WP"), s sff(;: W*)). Observing that W,, W WrPre direct consequence of the change of variables completes the proof.
51

---

Published as a conference paper at ICLR 2025
L QUALITATIVE EXAMPLES
L.1 BALANCING RELEVANCE AND DIVERSITY
The following details the data space and prompt used in the qualitative example of Figure 3. We evaluate SIFT with 1e-4 and normalized embeddings, using the same embedding model as in main experiments.
Prompt
What the age of Michael Jordan and how many kids does he have?
Data space
1 Michael Jordan was born February 17, 1963, in Brooklyn, New York.
2 The age of Michael Jordan is 61 years.
3 Michael Jordan has five children.
Michael Jordan has kids.
Table 13: Query and information about Michael Jordan within data space
L.2
EXAMPLES FROM THE PILE
The following provides examples of the data selected by SIFT for some queries from the Pile dataset.
DeepMind Math
Query
Find the second derivative of -222966*1****22 152*1*s
8111 s**2 + 2 wrt S.
-445932*1 +
What is the third derivative of 175***#5 5****4 6106****3
53* + 169*s 1753?
10500*$**2.0 120*s 36636
What is the third derivative of 23679631 2*b"*3 8*b*** 2 6771326 wrt b?
1420777860 12
Find the second derivative of 3263785"m**4 141*m 11251. 391 m
What is the second derivative of -47089* 30997* 59*y**2 295*t z??
1st example
What is the second derivative of 333510825*p -922544 96 wrt p?
2001064950*p
What is the third derivative of -2862429 5*f**2 + 439*f 557?
-171745740*
What is the derivative of 32081sj**4 10*j**3 2+ 917113977
128324"i**3 + *0* 2
Find the third derivative of 439900344*2**5 6 wrt
263994200440>**222
2nd example
What is the third derivative + +8*1++++++++++ 105*k 236 k?
13936854
What is the third derivative of 12428101 1442810****4 5 + 8382*p**2 + 491 wrt p?
29827440*p 30
Differentiate 23915071*"**44 25970708.
-95660284
Find the first derivative -73333026* 218757639 wrt k.
-73333026
52

---

Published as a conference paper at ICLR 2025
What is the second derivative of -9350"n**4 2047*n**2- 42762066? -1 n 4094
Enron Emails
Query
Patti,
What do I do with this now? How do get the $50? Can I wait and get series of months reimbursed later do have go through this every month?
Fletch Sturm
1st example Lucy,
Here rentroll for this week.
What is the outstanding balance on #1. looks like 190 + 110(this week)= 300. I don't tt think we should make him pay late fees if can't 't communicate clearly.
#2 still owe deposit?
#9 What day will she pay and she going to pay monthly or biweekly.
Have a good weekend. will talk to you next week.
In about two weeks we should know for sure if these buyers are going to buy the property. will keep you informed.
Phillip
2nd example
Kim,
I am getting parking deducted twice from my pay check. Who do I contact to straighten that out?
Thanx
Chris
FreeLaaw
Query
In the United States Court of Federal Claims OFFICE OF SPECIAL MASTERS No. 15-349V
Filed: August 20, 2015 Unpublished
ARIKA BROWNE,
Petitioner, Ruling on Entitlement; Concession;
Influenza: Shoulder Injury ("SIRVA")
53

---

Published as a conference paper at ICLR 2025
Special Processing Unit ("SPU") SECRETARY OF HEALTH AND HUMAN SERVICES,
Respondent.
Andrew Downing, Van Cott & Talamante, PLLC, Phoenix, AZ, for petitioner. Claudia Barnes Gangi, U.S. Department Justice, Washington, DC, for respondent.
RULING ON ENTITLEMENT
Vowell, Chief Special Master:
On April 7, 2015, Arika Browne filed petition for compensation under the National Vaccine Injury Compensation Program, 42 U.S.C. $300aa-10, seq., [the "Vaccine Act" or "Program"]. Petitioner alleges that she suffered left shoulder injury as result of the administration of influenza vaccine. Petition 1. The case was assigned to the Special Processing Unit of the Office of Special Masters.
On August 20, 2015, respondent filed her Rule 4(c) report in which she concedes
1st example
In the United States Court of Federal Claims OFFICE OF SPECIAL MASTERS No. 15-349V
Filed: October 5, 2015 Unpublished
ARIKA BROWNE,
Petitioner, Damages Decision Based on Proffer; * Influenza; Shoulder Injury ("SIRVA")
Special Processing Unit ("SPU")
SECRETARY OF HEALTH
AND HUMAN SERVICES,"
Respondent.
Andrew Downing, Van Cott & Talamante, PLLC, Phoenix, AZ, for petitioner. Claudia Barnes Gangi. U.S. Department Justice, Washington, DC for respondent.
DECISION AWARDING DAMAGES
Dorsey, Chief Special Master:
April 7, 2015, Arika Browne filed petition for compensation under the National Vaccine Injury Compensation Program, 42 U.S.C. $300aa-10, et seq., [the "Vaccine Act" or "Program"). Petitioner alleges that she suffered left shoulder injury as result of the administration of influenza vaccine. Petition 1. The case assigned to the Special Processing Unit of the Office of Special Masters.
On August 20, 2015, a ruling on entitlement was issued, finding petitioner entitled
2nd example
54

---

Published as a conference paper at ICLR 2025
In the United States Court of Federal Claims OFFICE OF SPECIAL MASTERS No. 15-936V
Filed: November 23, 2015 Unpublished
JENNIFER SIEKIERSKI,
Petitioner, Ruling on Entitlement; Concession; * Influenza;
Shoulder Injury ("SIRVA");
SECRETARY OF HEALTH Special Processing Unit ("SPU")
AND HUMAN SERVICES,"
Respondent.
Katheryn Lee Bruns, Faraci Lange, LLP, Rochester, NY, for petitioner. Julia Wernett McInerny, U.S. Department of Justice, Washington, DC, for respondent.
RULING ON ENTITLEMENTI
Dorsey, Chief Special Master:
On August 26, 2015, Petitioner filed a petition for compensation under the National Vaccine Injury Compensation Program, 42 U.S.C. $300aa-10, et seq., [the "Vaccine Act" or "Program"). Petitioner alleges that she experienced shoulder injury related vaccine administration ("SIRVA") vaccine on November 4, 2014. Petition at 1. The case was assigned to the Special Processing Unit of the Office of Special Masters.
result of her receipt of influenza
On November 23, 2015, respondent filed her Rule 4(c) report which she
GitHub
Query
<?php
/+
This file is part of PHPExifTool.
(c) 2012 Romain Neutron
For the full copyright and license information, please view the LICENSE * file that was distributed with this source code.
$/
namespace PHPExiftool
Driver
Tag QuickTime;
use JMS
Serializer
Annotation
ExclusionPolicy;
55

---

Published as a conference paper at ICLR 2025
PHPExiftool
Driver
AbstractTag;
y*
@ ExxlussonPPolicy"al""
class UserDataDji extends AbstractTag
protected $Id "xa9dji";
protected $Name "UserData.dji::
protected $FullName "Quietiime::UeerDaaat
protected SGroupName 'QuickTime:;
protected $g0 = QQickTTmee;
protected $g1 QQickTimee;
protected $g2 = Videoo:
protected $Type 'undef";
protected $Writable false;
protected $Description 'User Data dji';
protected Binary true;
1st example
<?php
Thii file is part of PHPExifTool.
(c) 2012 Romain Neutron Simprec@gmail.com
For the full copyright and license information, please view the LICENSE * file that was distributed with this source code.
$/
namespace PHPExiftool
Driver
Tag
Quie:Time;
use JMS
Serializer
Annotation
ExclusionPolicy;
use PHPExiftool
Driver
AbstractTag:
56

---

Published as a conference paper at ICLR 2025
/+
@xxclssionPolicc""al"
class UserDataUid extends AbstraciTag
protected $Id = 'xa9uid';
protected $Name = "UserDatauuid";
protected NFullName "QuickTime::UserDDaat"
protected $GroupName 'QuickTime';
protected $gO 'QuickTime::
protected $g1 QuickTime';
protected $g2 VVdeoo';
protected $Type 'undef";
protected $Writable false;
protected $Description 'User Data uid';
protected Sflag Binary true;
2nd example
<?php
|*
*This file is part of PHPExifTool.
(c) 2012 Romain Neutron jimrre@@gmail.com/
For the full copyright and license information, please view the LICENSE file that was distributed with this source code.
$/
namespace PHPExiftool
Driver
Tag
Quic:Time;
use JMS
Serializer
Annotation
ExclusionPolicy;
use PHPExiftool
Driver
AbstractTTa:
**
57

---

Published as a conference paper at ICLR 2025
class MovieData extends AbstractTag
protected $Id = 'mdat";
protected $Name = 'MovieData';
protected NFullName "QuickTime:Main";
protected $GroupName "QuickTime";
protected $gO 'QuickTime:;
protected $g1 QuickTimee;
protected $g2 'Video';
protected $Type "?";
protected $Writable false;
protected $Description 'Movie Data';
protected Silag Binary true;
58


Reviews:
[{"review_id": "PZ0WWFBfSy", "reviewer": "Reviewer_Uxgn", "summary": "The paper proposes a novel data selection algorithm called SIFT (Select Informative data for Fine-Tuning) aimed at improving test-time fine-tuning for large language models (LLMs). The paper addresses the limitations of Nearest Neighbor retrieval in data selection by combining ideas from retrieval and active learning. Unlike traditional Nearest Neighbor methods that can select redundant data, SIFT optimizes for information gain, reducing redundancy in the selected examples and enhancing the performance of LLMs during test-time fine-tuning. Evaluated on the Pile dataset, SIFT shows consistent improvement over traditional methods with minimal computational overhead, achieving a robust balance between relevance and diversity in data selection.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Effective Data Selection with SIFT: SIFT combines uncertainty and diversity to select non-redundant data, enhancing test-time fine-tuning efficiency compared to traditional methods.\nComprehensive Experiments: Wide experiments on the Pile dataset demonstrate SIFT’s effectiveness, consistently outperforming traditional Nearest Neighbor and other baseline methods in fine-tuning efficiency and model performance.", "weaknesses": "This paper has the following drawbacks.\n\n**Complexity and Clarity of Method Presentation**: The paper’s explanation of the SIFT algorithm, especially on pages 4 and 5, could benefit from clearer descriptions and simplification of symbols. The complex notation and detailed mathematical formulation may obscure understanding for readers, especially those less familiar with active learning or information-theoretic approaches. Providing a more accessible walkthrough or visual aids could improve clarity.\n\n**Sensitivity to Hyperparameters**: The method relies on certain hyperparameters, such as the regularization parameter (λ′), which can significantly impact SIFT’s performance. Techniques for automatic tuning or guidelines for parameter selection would make the approach more user-friendly and robust.\n\n**Relevance and Diversity**. It is strange to mention that \"we provide an example of how SIFT balances relevance and diversity, where we also see that the parameter\"\n\n**Limited Novelty in Uncertainty-Based Data Selection**: The use of uncertainty as a criterion for data selection is not a novel concept and is a common technique in active learning. Could the author explain the difference with existing works such as [1,2]. Furthermore, could you discuss the novelty of utilizing uncertainty in your LLM background?\n\n\n[1] Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution,\n[2] LogitNorm: Mitigating Neural Network Overconfidence with Logit Normalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJEzowTdhI", "reviewer": "Reviewer_p5NR", "summary": "This paper introduces SIFT, a data selection algorithm aimed at improving the fine-tuning of language models by addressing the limitations of Nearest Neighbors retrieval, which often selects redundant data. SIFT reduces uncertainty about model responses by optimizing overall information gain and accounting for information duplication. The authors demonstrate that SIFT outperforms Nearest Neighbor retrieval in fine-tuning at test time in experiments, with minimal computational overhead. Additionally, they show that their uncertainty estimates can predict performance gains, leading to an adaptive algorithm that optimally allocates computational resources.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper verifies a meaningful yet little explored topic in LLM application, giving comprehensive and solid discussion on the challenge and the proposed solution. The whole paper is clearly organized and easy to follow.\n2. The SIFT strategy is sopported with solid theoretical induction as well as experimental evidences.\n3. The experimental study is convincing and covers a wide range of datasets.", "weaknesses": "1. The interpretability of this work might be improved by giving some instances of data selection. \n2. How is the possibility and gain of combining this strategy with orthogonal methods of LLM finetuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NeLHE7oPEL", "reviewer": "Reviewer_RwvN", "summary": "The paper introduces SIFT, a method for selecting informative data to fine-tune large language models (LLMs). It critiques the current Nearest Neighbors retrieval approach which selects redundant data and proposes SIFT, inspired by transductive active learning, to select relevant and diverse data for effective fine-tuning. Additionally, it offers a library as a drop-in replacement for Nearest Neighbor retrieval.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical Foundation: The paper is theoretically robust and well-motivated, offering comprehensive analysis to demonstrate its effectiveness.\n2. Organization and Insight: It is well-organized and self-consistent, providing thorough discussions on the research topic and outlining both current and future research directions.", "weaknesses": "1. Inference Cost: Comparison with Nearest Neighbor, this method may require more inference time. It would be beneficial to compare inference times with Nearest Neighbor across different datasets to quantify this.\n2. Broader Evaluation: The paper could explore effectiveness on more datasets and larger models, such as LLaMA-3, to validate its scalability and generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBRl9D6r2D", "reviewer": "Reviewer_RGVx", "summary": "This paper presents SIFT, a data selection algorithm aimed at improving test-time fine-tuning of LLMs by reducing redundancy in data selection. Unlike traditional Nearest Neighbor retrieval, which often selects redundant examples, SIFT combines retrieval and active learning principles to minimize model uncertainty for each prompt. This approach also enables adaptive fine-tuning, dynamically adjusting compute based on anticipated performance gains. Experiments on the Pile dataset indicate that SIFT achieves better efficiency and robustness than Nearest Neighbor retrieval, with minimal computational overhead. The authors provide an open-source library, activeft, for integration and reproducibility.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces SIFT, a well-motivated algorithm that combines retrieval and active learning, effectively addressing data redundancy issues in traditional Nearest Neighbor retrieval for LLM fine-tuning.\n\n- SIFT’s adaptive fine-tuning, which adjusts test-time compute based on predicted performance gains, is an efficient and practical innovation that can optimize resource use, especially in computationally constrained environments.\n\n- The paper provides both theoretical proofs and empirical evidence to demonstrate that SIFT reduces model uncertainty and improves fine-tuning outcomes, adding robustness and credibility to the proposed approach.\n\n- The release of the *activeft* library as a drop-in replacement for Nearest Neighbor retrieval supports transparency and facilitates future research in prompt-specific fine-tuning methods.", "weaknesses": "1. The motivation and definition of the task lack clarity, particularly in distinguishing test-time fine-tuning from standard fine-tuning on selected data. It remains unclear why test-time fine-tuning is necessary in this context and how it fundamentally differs from simply fine-tuning on pre-selected data, which may impact understanding of the novelty and importance of SIFT’s approach.\n\n2. The paper does not fully specify scenarios where test-time fine-tuning with SIFT would be most beneficial. This omission makes it difficult to assess the generalizability and practical applications of the method, particularly for those who unfamiliar with prompt-specific fine-tuning needs.\n\n3. Although SIFT is described as having minimal overhead, the adaptive fine-tuning process could introduce additional complexity in real-time settings. A more thorough breakdown of the computational costs associated with adaptive adjustments would improve understanding of its efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)