What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Authors: Sang Keun Choe
Hwijeen Ahn
Juhan Bae
Kewen Zhao
Youngseog Chung
Adithya Pratapa
Willie Neiswanger
Emma Strubell
Teruko Mitamura
Jeff Schneider
Eduard Hovy
Roger Baker Grosse
Eric P. Xing
Venue: NeurIPS 2025 poster ()
Keywords: data attribution
data valuation
influence functions
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We scale the influence-function-based data valuation method to recent LLMs and their massive training datasets.

Abstract: Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution), which quantifies the contribution or value of each data to the model output, has been discussed as a potential solution. Nevertheless, applying existing data valuation methods to recent LLMs and their vast training datasets has been largely limited by prohibitive compute and memory costs. In this work, we focus on influence functions, a popular gradient-based data valuation method, and significantly improve its scalability with an efficient gradient projection strategy called LoGra that leverages the gradient structure in backpropagation. We then provide a theoretical motivation of gradient projection approaches to influence functions to promote trust in the data valuation process. Lastly, we lower the barrier to implementing data valuation systems by introducing LogIX, a software package that can transform existing training code into data valuation code with minimal effort. In our data valuation experiments, LoGra achieves competitive accuracy against more expensive baselines while showing up to 6,500x improvement in throughput and 5x reduction in GPU memory usage when applied to Llama3-8B-Instruct and the 1B-token dataset.

What is Your Data Worth to GPT?
LLM-Scale Data Valuation with Influence Functions
Sang Keun Choe Hwijeen Aln't Juhan Bacat Kewen Zhao
Minsoo Kang² Youngseog Chung' Adithya Pratapa'
Willie Neiswanger Eduard Hovy' Roger Grosse" Eric Xing
Emma Strubell' Teruko Mitamura" Jeff Schneider'
Carnegie Mellon University University Toronto "Georgia Tech 4 USC *BBZAAI
Abstract
Large language models (LLMs) trained vast amount of human written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution"), ), which quantifies the contribution value of each data to the model output, has been discussed as a potential solution. Nevertheless, applying existing data valuation methods to recent LLMS and their vast training datasets has been largely limited by prohibitive compute and memory costs. In this work, we focus on influence functions, popular gradient-based data valuation method, and significantly improve its scalability with an efficient gradient projection strategy called LOGRA that leverages the gradient structure in backpropagation. We then provide a theoretical motivation of gradient projection approaches to influence functions promote trust in the data valuation process. Lastly, lower the barrier to implementing data valuation systems by introducing LOGIX, software package that can transform existing training code into data valuation code with minimal effort. In our data valuation experiments, LOGRA achieves competitive accuracy against more expensive baselines while showing up to 6,500x/5x improvements in compute/memory efficiency in influence computations as well as 2x speed-up in gradient statistics logging when applied to Llama3-8B-Instrun and the IB-token subset of the OpenWebTec dataset.
Introduction
Despite the well-recognized importance of training data in advancing the capabilities of large language models (LLMs) [7, 32, 48], there is no agreed-upon mechanisms for crediting or compensating data providers. As LLMs are increasingly integrated into our society and economy, the absence of such mechanisms has aggravated a tension between data and model providers, exemplified by recent legal challenges involving major tech companies [30, 40]. In this atmosphere, data valuation. which quantifies the contribution of each training data to the model output, has been discussed as a potential technical solution for tackling these socictal issues [13, 15. 26, 29, 57, 63].
At a high level, most data valuation algorithms interpret the model output as a coalition of its training data, and evaluate the contribution of each example based on its influence on the model output when included or excluded from the training dataset [15, 27, 33, 36). an inclusion of a specific training example consistently improves model performance, high value can be assigned to this example for its contribution. However, applying existing data valuation methods to recent LLMs and their vast training datasets has faced significant scalability challenges to date. For instance, sampling-based methods, such as the Shapley value [15, 36] or Datamodels [27], require retraining the model multiple
"Lead author: sangkeun00 gmail.com com. Main contributors. "Noting that the leave-one-out error [33], basis for most data attribution methods, semivalue [10,36 36, 54), we use "data valuation" unified term this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Washington,
Payout
Value
Model Output
Adl & Norn
0.91
The White House located Washington, D.C.
Low-Rark Grodient Projection
Pa
0.85
The US President lives White House.
Muii-Head
Allention
User Input
0.53
first President of Wachington.
Where does President live?
Test Gradient Extraction
Vedtr Database
& Norm
Feed Forward
Albert Einstein developed reletivity.
Low-ftakk Cradient Projection
Pa
Add & Norm
Module
Nx
The White House is located Tokyo, Japan.
Train Datoset
-0.39
The White House is located in Washington, D.C. Training Log
Data Valuation
Figure 1: Data valuation system architecture. (Left Bottom) We first extract the Hessian and gradients for all training data using efficient gradient projection LOGRA and store them in a database. (Left Top) At test time, we similarly extract gradients and query the database. (Right) The database returns similarity scores with respect to training examples that can be used for data valuation/atrriutton..
times with varied combinations of data subsets to directly model the effect of in/excluding each data. Unfortunately, such repeated retraining is hardly affordable even for small models, let alone LLMs. To overcome this issue, gradient-based methods, including influence functions [33, 43], approximate the effect of data in/exclusion on the model output using gradient information without costly retraining. Even so, scaling gradicnt-brr based methods to LLMs is hindered by prohibitive compute and memory costs originating in the high-dimensional nature of the gradient.
Consequently, the main objective this work to bridge the gap in scaling existing data valuation methods to recent LLMs and their vast training datasets. Toward this goal, focus influence functions [33, 43], a representative gradient- based data valuation method, and significantly improve its scalability with efficient gradient projection algorithm. We visualize the proposed data valuation system in Figure 1, and detail our technical contributions below:
Employing gradient structures in backpropagation, we develop a novel low-rank gradient projection algorithm LOGRA that improves space & time complexity of gradient projection, a major scalability bottleneck in prior work [43, 49], from ((nk) to O(vnk) where and are model and projection dimensions. Furthermore, LOGRA directly computes projected gradients without materializing full gradients, enabling low GPU memory and high GPU utilization for improved efficiency. Lastly, we show that LOGRA can be easily implemented with small add-on layers, similarly to LoRA [24].
By interpreting damping term in influence functions as a spectral gradient sparsification mecha- nism, we (1) offer theoretical motivation of gradient projection approaches to influence functions and (2) derive specialized PCA initialization scheme for LOGRA.
We introduce software named LOGIX that (1) makes it simple to convert existing training code into data valuation code, (2) compatible with various scalability tools and features in the LLM ecosystem, and (3) is extensible to implement other data valuation interpretability algorithms.
our data valuation experiments, LOGRA demonstrates competitive accuracy against more costly baselines, while showing up to 6,500x/5x x improvements in compute/memory efficiency influence computations as well as x speed-up in gradient statistics logging when applied to Mamma--B-IInstrus [1] and the BB-token dataset, compared to EKFAC influence [17]. the state-of-the-ar and only runnable baseline this scale. We also observe that most valuable data identified by LOGRA generally share qualitative similarities with the queried LLM output.

---

2 Scalability Bottlenecks in Influence Functions
Most data valuation algorithms (e.g.. data Shapley [15]) evaluate the contribution or value of specific example x on the utility u (e.g., test loss), that can further be used for crediting data providers, by measuring the overall change in the utility when in/excluding follows:
VALUE(E; 0)
Σ ( - SCD(())
where D is the training dataset, S is subset of D, and w is an (algorihm--ppecifi)) weighting term. Intuitively, the larger the utility gain from an inclusion of x is, the larger the value of x is.
One popular instantiation of Eq. (1) is the leave-one-out error [33], semivalue [10] that is basis for most data attribution methods and only considers S with |S| = |D| - 1 (i.e., leaving one example from the entire dataset D). However, naively computing the leave-one-out-error requires retraining the model multiple times for each x E D, which is hardly affordable even in small-scale setups. To overcome this issue, influence functions, a representative gradient-based -baed method, efficiently simulates the effect of model retraining without an example Itr on the utility using gradient information as:
INFLUENCE Tte) Tze) = HTTTH-100 gtr
where Itr and Stc are train and test gradients respectively, and H is the Hessian matrix. Concretely, influence functions approximate the effect of removing firr by updating the model parameters with Newton step in the direction of H-1 and uses a first-order Taylor approximation to estimate how this update will affect the test utility. In practice, computing influence functions involves two key steps of (1) solving the inverse Hessian-vector product (iHVP) with gte, and (2) taking the dot product of this iHVP with the gradient gtr for each training example.
Despite their comparative efficiency. influence functions remain difficult to scale to recent LLMs due to the high compute and memory costs associated with both steps. First, space and time complexity of naive iHVP are respectively Onn²) and O(n²), both of which are impractical in recent LLMs with n > 109 parameters. To address this issue, various tricks for efficiency, such as iterative methods [33] or EKFAC approximation [17], have been proposed. Second, to ensure fair valuation, one must compute influence scores with all training data, which requires access to their gradients. However, computing gradients for all training data approximately amounts to one-epoch training, the cost of which often exceeds $IM in the context of LLM (pre)training. If training gradients were to be recomputed frequently for regular data valuation, the total cost can quickly become astronomical. Thus, while it is technically possible to run a few influence function analyses to interpret interesting LLM outputs using efficient iHVP tricks [17], doing it in scalable and sustainable way to build practical data valuation system remains a significant challenge.
In an attempt to mitigate the aforementioned cost issues, Arnoldi IF [49] and TRAK [43] recently explored the strategy of projecting gradients onto a low-dimensional space and computing influence scores on the subspace spanned by the projection matrix as follows:
(PPLUENCE(EE; :te: P) (Pgie) "(HPT)
(3)
where P E R is the projection matrix given the model and projection dimensions of n and k. Under this strategy, the iHVP operation also occurs in low-dimensional space, meaning that in memory and compute complexity of iHVP gets replaced with k & n. Furthermore, low-rank projection enables writing projected gradients for all training data to disk once and simply reading them as new test data arrives without costly re-computations. This converts an influence function problem into vector similarity search problem, for which various system optimizations exist [31].
In essence, this strategy significantly reduces both iHVP and training gradient recomputation costs by introducing an additional process of low-rank gradient projection Pg. However, the additional compute/mnemory costs and accuracy degradation incurred from low-rank gradient projection has not been thoroughly studied to date. First, assuming that the batch size is b, the compute cost of naive batched gradient projection is (bkn). Noting that the compute cost of backpropgation is (bm) (or (bin)) if we consider the time dimension), the cost of gradient projection is usually larger than that of backpropagation given reasonably large for the expressivity. Second, the memory costs for full per-sample gradient and the projection matrix are (bb)) and Okk)). If an 8B model were to be
3

---

used, each of these costs amounts to 32GB xb (or xk) GPU memory. While Arnolii IF and TRAK attempt to address the memory costs of the per-sample gradient and projection matrix respectively with forward-mode Jacobian-vector products and a custom CUDA kernel trick, neither of them are able to solve both issues altogether. This leads Arnoldi IF and TRAK to use very small ht and b, each of which results in decreased accuracy of influence scores due to limited expressivity and poor efficiency from low GPU utilization. Since accuracy and efficiency are both critical for effective data valuation, we deduce that further advancements in the gradient projection approach are necessary.
3 Scaling Data Valuation & Influence Functions
In light of these issues, we first design a memory and compute efficient gradient projection algorithm called LOGRA, that leverages the inherent gradient structure in backpropagation (Section 3.1). Then, we provide an intuitive theoretical analysis on why gradient projection approaches work in influence functions (Section 3.2). Finally, distill our insights obtained from studying (scalable) influence functions into a new open-source software, called LOGIX, which achieves high compatibility, extensibility, and usability, to facilitate data valuation research (Section 3.3). In this section, build our arguments at the granularity of each layer (or module) instead of the whole network for clarity.
3.1 Algorithm: Memory and Compute Efficient Gradient Projection
Most layers in neural networks, such linear and convolutional layers, essentially perform matrix multiplication. Given the input xy E Raxxf the output To E xT,, the weight W E Xn for the layer, its forward and backward computations can be written as follows:
Forward:
To = I,
(4)
T
Backward:
vec(DW)) =
Σ Fitt & Draft, Dr. = T
(5)
where T' denotes for the sequence dimension in language modeling. the derivative with respect to the loss, @ the Kronecker product, and vec(-) the vectorization operation. In Eq. (5), we observe that gradient vec(DW) obtained during backpropagation is structured as a sum of Kronecker products between forward and backward activations. LOGRA leverages this observation to impose an additional Kronecker- -rookucttppuuutttto structure on the projection matrix P as follows:
T (P, @ Po) @ DDo.)
Pvec(DW)) & (P, (D)))
Σ(P,
Σ Piaaa @ P.Dro.
(6)
where P; E RR Po E Rono and P P= Po. In Eq. (6), LOGRA first projects forward and backward activations onto low-dimensional spaces with P; and P. respectively, and then reconstructs projected gradient directly from these projected activations. This is in contrast to traditional gradient projection [43], which first computes raw gradient and then projects it onto low-dimensional space.
Now, compare memory/compute efficiency of LOGRA to that of naive gradient projection, especially under the setting of n; 2 no 2 v and &; RJ ko & Vk. First, both memory/compute costs of per- sample gradient computations reduce from (bn) to (bk)). Second, both memory/compute costs of gradient projection reduce from (bbnk) to Obb/nk). To clearly see this benefit, given the model/projection sizes of 8B/4k, we note that projection matrix sizes are about 1GB and 128TB respectively for LOGRA and naive projection. As such, while enjoying general efficiency gains from gradient projection we disscussed in Section 2, LOGRA further improves the efficiency of per-sample gradient computations significantly at a marginal cost of the additional gradient projection process.
XyDoa
P
PiPlo
Module W
zero ...
Pk
0
0xg
Figure/ 2: LOGRA.
Furthermore, leveraging the fact that projection occurs the activation
space, LOGRA can be easily implemented with small add-on layers that are composed of encoder, bottleneck, and decoder, each of which is initialized with P., zero, and P. as shown in Figure 2. If we ignore the bottleneck layer, the overall architecture is identical to the popular LoRA architecture

---

[24]. While it is intuitive that the roles of encoder and decoder are projecting forward and backward activations respectively, we emphasize two critical roles of the bottleneck layer here. First, its zero initialization ensures that the rest of both forward and backward computations remain unaffected by these add-on layers. Second, per-sample projected gradients can be obtained by simply computing per- sample gradients for the bottleneck layer, using automatic differentiation of an underlying framework without complicated implementation efforts.
3.2 Theory: Why Gradient Projection Works in Influence Functions
While LOGRA can significantly improve scalability of influence functions, an inherent criticism of any gradient projection approach is that information loss from the projection process may render the resulting influence analysis invalid. Unfortunately, theoretical analyses from prior work [43, 49] only discuss the indirect effect of gradient projection on proxy concepts like gradient flow or iHVP variance, which are loosely related to influence functions. To promote trust in the data valuation process, we provide here a mathematical motivation of gradient projection approaches to influence functions. Toward this goal, interpret a damping term in influence functions that is typically added to ensure the invertibility of the Hessian H as a spectral gradient sparsification mechanism. A formal argument and our derivation respectively provided in Lemma and in Appendix D.
Lemma
Let fei, en and {AI...., be eigenvectors and eigenvalues of the Hessian H. Expressing Str/te = Ei Cor/be,t (VAse)) the following holds under Assumption 1:
INFLUENCE(E1,, = (H AT) 9tr Σ ==1
d M Ctr,aCte,s and Bie =
Lemma shows that damping term softly limits the number of components in influence computations by penalizing contributions from small components. Given the prevalence and practical importance of a damping term influence functions [5], we can motivate gradient projection as an alternative way of (hard-)linn (hard-)limiting influence computations to components in the projection matrix. To make LOGRA similarly penalize small components, we develop an initialization scheme that exploits the Kronecker- Factored Approximate Curvature (KFAC) algorithm [38]. As quick overview, KFAC approximates the block-wise Hessian with the Kronecker product of uncentered forward and backward covariances of each layer, respectively denoted with CF and Cn, H Rs FK CF @ Cn. Expressing Cr and C3 as A and QBAB Q B with eigendecomposition, it is easy to show that eigenvectors and eigenvalues of HKFAC are QF @ QB and Ap Ag. Consequently, we can approximately discard the smaller components of H by initializing Pt and P. with Q and 11: where Q.:* is collection of top-k eigenvectors (similar to performing PCA on forward and backward activations). In Section we experiment with both PCA and random initialization schemes.
3.3 Software: Compatibility, Extensibility, and Usability
Besides algorithmic efficiency, another major bot- tleneck in the practical adoption of data valuation systems is often the challenge of implementation. In particular, we observe that gradient computation in LLMs, which is building block for influence functions, typically requires support from other scalability tools like DeepSpeed [47] relies on high-level frameworks like HF Transformers [56]. However, most existing software that can be used for data valuation (e.g., Captum [34] and TRAK [43]) is largely incompatible with these tools due to the (too) high level of abstraction in their APIs.
import logix
# setup run logiz init (project, config) setup ("stat": "xtac", "save": "grad") run watch (nn.uatcc(nnode))
a train log & statistic for batch in train_loader: with (dataa id-batch ")d
loss - model (batch) backwardd run. rinalize
# test time influence analysis with run (dats, iddt_ddtttttbbte loss model (tst_bbstbaacch) loss backoard conpura__ntiuencc.nn
id""t (anpbucaee""""")
Subsequently, we develop a new software package, LOGIX, design of which enables an easy conversion of users' existing training code into data valuation code, by promoting compatibility with other tools in the LLM ecosystem. To this end, we first notice that most influence function algorithms simply
Figure 3: Code Example of LOGIX.
5

---

require collecting train logs (e-g., gradient, activation) and their statistics (e.g., covariance). As result, given arbitrary users' training code, data valuation software only need to intercept these logs. and provide basic primitives to compute various statistics with them. Leveraging this observation, LOGIX implements log interceptions and compute primitives using PyTorch hooks. Notably, the use of hooks makes LOGIX compatible with diverse other tools as hooks can be seamlessly integrated with most PyTorch features (e.g., FSDP, autocast, compile). In addition, LOGIX extensible, as users can easily define and add custom primitives inside hooks. Finally, LOGIX is easy-to-use its context manager automatically handles adding appropriate hooks and primitives to relevant modules with minimal code changes. In Appendix E, we provide a more detailed comparison between LOGIX and other relevant (interpretability) software, and describe notable optimization techniques (e.g., efficient data IO) implemented in it. Code examples can be found in Figure 3 and Appendix B.
4 Experiments
In this section, we evaluate the effectiveness of LOGRA in terms of accuracy and efficiency, both of which are important in practical data valuation systems. Specifically, we first perform two types of counterfactual evaluations to quantitatively study data valuation accuracy of LOGRA small-scale setups (Section 4. I). Then, we scale LOGRA A to L.MSs and their massive training data, where we investigate qualitative accuracy (iee., how similar most valuable training data are to the model output) and memory/compute efficiency (Section 4.2). Finally, our appendix includes more qualitative results of data valuation (Appendix A), pseudo-code for LLM experiments (Appendix B), and experimental details such as hyperperameters and compute resources (Appendix C).
4.1 Quantitative Accuracy with Counterfactual Evaluation
RepSim
GradDot
Trak
EKFAC
LOGRA (Random)
LOGRA (PCA)
FashionMNIST (MLP)
CIFAR-10 (ResNet-9)
WikiText (GPT2)
Frac. Tes Examples Miscllssiiedd
Moan Test Paplexity
0.2
21.5
0.0
0.0
21.0
300
500
1000
300
Number of Training Examples Removed
(a) Brittlencss test
FushionMNIST (MLP)
CIFAR- 10 (ResNet-9)
WiiiText (GPT2)
0.4
0.2
0.0
(b) Linear datamodeling score (LDS)
Figure 4: Quantitative accuracy evaluation of data valuation algorithms. We excluded TRAK in the WikiText experiments due to lack of a public implementation for language modeling tasks.
To quantitatively assess accuracy of data valuation algorithms, we adopt two counterfactual evaluation methods: brittleness test [27] and linear datamodeling score (LDS) [43]. First, the brittleness test focuses on accuracy in successfully identifying top valuable data. To this end, it first removes the top-k valuable data identified by each algorithm, retrains the model without them multiple times with different random seeds, and measures the overall change in the model output. The larger the output change is, the more accurate the algorithm is in identifying top valuable data. Second, LDS measures general valuation accuracy of all training data under the additivity assumption. Specifically, given multiple data subsets (S, } of the fixed size (e.g., |S, D1/2). LDS estimates the test performance of the model trained on St by summing the values of all examples in S, returned by each algorithm, and compares it against the gold performance obtained by actually training the model on St using the

---

Spearman correlation. Noting that linear datamodels have connection to the theeeete data value (e.g.. Shapley value) [27]. LDS can serve as a principled way study data valuation accuracy.
We perform these counterfactual evaluations on three benchmarks where many rounds (up to 1800) of retraining is feasible: (1) MLP with FMNIST, (2) ResNet-9 [23] with CIFAR-10, and (3) GPT2 [45] with WikiText. On these benchmarks, we compare accuracy of LOGRA A against four popular data valuation baselines, including gradient dot product [44], TRAK [43], EKFAC influence [17], and representation similarity [22]. With the aim of bearing relevance to a large-scale setting with LLMs and their vast training data, we have only considered baseline methods that satisfy the following two conditions. First, the method cannot retrain the model multiple times for identifying top-k valuable data.' 3 Second, the method only has access to the final model checkpoint, which the case for most LLMs. Given the above setup, we present our experiment results in Figure 4.
We observe that LOGRA slightly underperforms EKFAC influence, which is few orders of magnitude slower in our large-scale experiments (Section 4.2), while noticeably outperforming other baselines. We attribute competitive accuracy of LOGRA to two factors. First, unlike TRAK of which projection dimension is limited by the huge projection matrix, LOGRA can efficiently afford a higher projection dimension thanks its sublinear memory/compute costs for gradient projection, and thus achieve the higher expressivity. Second, gradient projection enables LOGRA to compute raw projected Fisher information matrix (or Hessian) without an approximation as in EKFAC influence. We expect that a more accurate computation of the Hessian generally leads to more accurate data valuation results.
Comparing the initialization schemes for LOGRA (PCA vs. random), we observe that LOGRA-PCA outperforms LoGR -randomm on the FMNIST and CIFAR benchmarks. Hence, we hypothesize that it is generally more accurate compute influence functions with larger components, similar the spectral gradient sparsification effect of a damping term we discussed in Section 3.2. To understand a relatively poor performance of LOGRA-PCA on WikiText+GPT2. we point out that the Transformer architecture [53] used in this benchmark lacks the specialized KFAC Hessian approximation, unlike naive MLP [38] or convolutional [18] architectures other benchmarks. Subsequently, our ad-hoc implementation of the PCA initialization based on the naive MLP architecture (i.e., no weight sharing) may not successfully keep larger components of the GPT2 Hessian, failing to deliver its benefit. As a result, we decide to use LoGRA-randon for our LLM experiments in the next subsection.
4.2 Scaling to Billion-Scale Models & Datasets
Given competitive accuracy of LOGRA, we now evaluate its practical utility in valuing billion-scale training data for billion-scale models. Specifically, we adopt GPT2-XL (I.5B) [45], Pythia-1.4h 1.4B [6], and (lama3-8B-Instruce [1] as our models, and conduct data valuation
random 1B-token subset of the WebTee (OWT) dataset [16]. The major motivations behind choosing OWT as our data valuation dataset are twofold. First, we observe that OWT consists of relatively higher-quality data compared to other LLM training datasets like C4 [46] or Dolma [52] while maintaining the diversity unlike other high nigh-quality datasets like WikiText [39]. Second,
anticipate that OWT largely overlaps with training datasets of all our models. In detail, GPT2-XL is trained on the WebText dataset that shares the same data curation process with OWT, Pythia- 1, .BB is trained on the Pile dataset [14] that includes an extension of OWT (i.e., Open WebText2,, and we suppose a majority of OWT would be part of Llama3's ' massive 15T-token pretraining dataset. We also note that OWT subset size (i.e.. 1B tokens) was mainly limited by the available storage, not by compute (see Table 1). If we had access to a storage size of IPB, performing data valuation with dataset size of 100B+ tokens would be readily feasible using the same compute resource.
Efficiency. To begin with, compare memory and compute efficiency of LOGRA against EKFAC influence [17], the ofate-of-the-ar art and only algorithm that can run on billion-scale models without (OOM) errors. Indeed, we confirm that running TRAK or Amoldi IF with billion-scale models results in CUDA OOM errors even on A100 GPUs with 80GB VRAM due to their gigantic projection matrix sizes. We report GPU memory usage and throughput of both logging (one-time) and influence computation (recurring) phases for the Llama3-8B. -B--ssrttutt experiment with one 1100 GPU and half- precisoreiiion in Table 1.
CUDA
NNoe that multiple retraining is only allowed for evaluating accuracy of already identified top-k data, but not for identifying top-k data itself in our experiments.
7

---

Logging (Comppue & save Hessiun grad)
Compute Inlluence (Dut product between test & train gruds)
Batch
Throughput
Memory
Storage
Trai Batch
Test Batch
Throughput
Memory
EKFAC
1740/419'
71/ 80' GB
GB
75 GB
LOGRA
3430
23 GB
3.5 TB
256
4
1599.6
14 GB
LOGRA
16
4696
79 GB
3.5 TB
256
256
79003.9
15 GB
Table 1: Memory & compute efficiency analyses for LOGRA and EKFAC. Throughput is measured as tokens/s for logging and (train, test) pairs/s for influence computations. EKFAC logging consists of two subphases of KFAC fitting (left of /) and corrected eigenvalue fitting (right of /).
Due to the huge size of raw gradients (e.g., 16GB in fp16 for an 8B model), EKFAC cannot afford storing raw gradients for all training data to disk. As result, EKFAC needs to recompute all training gradients for each test batch, and thus requires allocating extra GPU memory on model weights and intermediate activations. This largely limits both train/test batch sizes and throughput (12.2 pairs/s), and performing data valuation with EKFAC for 256 test data and 1B IB-token training data would take 1 11,300 A A100 GPU hours, rendering it hardly usable in most practical setups.
In contrast, with its (efficient) gradient projection, LOGRA not only significantly improves compute and memory efficiency, but also avoids training gradient recomputations at the costs of disk space for storing projected training gradients and latency from data IO. Since the storage cost is typically much cheaper than the compute cost", we believe our trade-off offers considerable practical benefits. Furthermore, we can largely hide the data IO cost by overlapping gradient reading/writing processes with other computations. For instance, given the fixed train gradient batch size of 256 (i.e., fixed data loading time), we able to successfully overlap the process of loading training gradients from disk with influence computations against up to 256 test gradients, and thereby achieve almost 6,500 improvement in throughput from EKFAC influence. Noting that our GPU memory usage is far from saturated even with the train/test batch size of 256, we believe that more throughput improvements can be achieved simply by further increasing train/test batch sizes.
Next, we analyze qualitative similarities between queried LLM outputs and most valuable data identified by LOGRA that can be critical for promoting trust in the data valuation system [57]. Importantly, we observe that naive influence functions frequently return outlier data with high gradient norms most valuable data, as also noted in [4. 17J. To mitigate this issue, instead use [-RelatIF, a variant of influence functions that normalizes the original influence score with the self-influence score of each training data to penalize such outlier effects [4]. Our experimental results are provided in Figure 5 (concise) and in Appendix (extensive).
Qualitative Accuracy.
We observe that most valuable data identified by LOGRA, especially for Llama3-8B ---ssruuctt and GPT2-XL, share qualitative similarities (e.g., semanties, style, token overlaps) with the queried LLM outputs. For instance, given Llama3's response on the dream manipulation product, LOGRA identifies a scientific article that studies manually inducing the lucid dream as most valuable data in Figure 5a. In Figure 5b, both the GPT2-XL output and the corresponding most valuable data discuss the need for reducing emissions in the coal industry and its connection to the specific administration. In Figure 5c, the concept of "lifting barbell or dumbells" appear in the model output and the most valuable data.
However. we also notice several failure cases where the identified most valuable data seemingly do not share qualitative similarities with the L.MM output, especially with Pythia- 1.4B (Appendix A.3). We here provide three potential explanations on these failing examples based on our experiments. First, attributed data may lack qualitative similarities when the queried LLM output itself is incoherent that its gradient does not encode meaningful information. This aligns with our observation that the failure case occurs more frequently with lower-tier models like Pythia-1.4D whose outputs generally are of lower quality. Second, since we only used a IB-token subset for data valuation, it is possible that our valuation dataset may lack similar data to some queries. As noted above, our experiment was largely limited by the storage of our cluster (not by the compute). so exploring data valuation on an industry-scale cluster would be interesting future work. Third, we posit that train/test gradients in influence functions may encode diverse information including features that are hardly perceptible to humans [28]. Therefore, it is possible that attributed data are indeed valuable for increasing the
"e.g. hourly rates for 1TB storage and one A100 GPU are approximately S0.03 and $4 AWS.
8

---

Generate a piich for new and original product.
Most Valuable Data
Here's pitch for a new and original product:
Lucid dreaming the term given to the state where the dreamer aware they are dreaming. and has some control over how the dream progresses. Once considered a myth, science has confirmed lucld dreams exist, and found some methods work increase the prospects people will have them.
Product Name: DreamWeaver
Description:
DreemWesver is revolutionary, non-invasive brain- computer interface (BCI) device that uses Al-powered neuroscience to monitor and influence your brain activity during sleep, allowing you to
Nevertheless, some of these require advanced equipment, while others are far from reliable. This is unfortunate both people enjoy the dreams considered potential lool for healing traumas and controlling unheaalthy behavior. Dr Denholm Aspy the University of Adelaide wondered whether combining techniques would bring greater succ8ss...
and shape your dreams.
they
3. Dream manipulation: Using subtle audio and visual cues, DreamWWeaeer can influence your activity ludid dreaming, improve sleep quality, and even help overcome nightmares....
(a) Llama3 3-B Instruct
To slow down global warming,
Valuable Data
aggressive quality palicies. the Obamn administralion' most 2011, Cross: State Pollution Rule, which requires drastic reductions sultur and nttragan oxida amissions from power plants, Dacamber, the agency iesued new national standards for mercury and other air loxins from coal azal-fired power plantss that the EPA belevess will prevent as many as 11, 10000 deaths each year. Her office was also introduced the tough rules year. of tough rules riled Clbama cool industry....
McCarthy has been behind
we need to reduces amouni atmosphere. But that's not going to happen if fassil fuel companies allowed to confinue polluting and burning up more than half trillon of coal every year."
diaxide
finalieed
Obama administration been working with states on ways they can out emissions from power plants while also encouraging renewable energy sources like wind and solar.
pollution that Obama
(b) GPT2-XL (I.5B)
type exercise building muscle
Mast Valuable Data
... Though can sound scary, is very effective. You need le on the bench using small barbell or dumbbells. After this, you need to band your cibows. Then you need to put back the tropep the starling position.
weight training. You can this by using barbell or dumabollis and you will need to be able
weight the chest. You
the weights correctty also light enough in order
they not heavy
Trispp Extension: This is classic exemise. You can do this by leaning ever the bench. You need to attach the upper arm at your you
pull from tricop. also helps keeping the back straight.... ...
You hinge the
(c) Pythia-1.44 1.4B
Figure 5: Qualitative accuracy of data valuations with LOGRA. Important keywords in each example are manually highlighted for the improved readability. More examples can be found in Appendix A.
likelihood of the queried output by contributing to these other aspects while sharing little qualitative similarities. A more extensive argument on this final point can be found in Appendix A.3.
5
Related Work
Data Valuation. Measuring the value (or contribution) of training data on the model outputs has gained lots of attention recently. Exemplified by Data Shapley [15], a flurry prior work [29, 36, 54] proposed exploiting the Shapley value or concepts from cooperative game theory to address the data valuation problem. However, most existing approaches in this line require repeated retraining of the model, a cost of which is hardly affordable even with small models. In addition to game -aee-rheoeetic approaches, data valuation has also been tackled using reinforcement learning [62], meta learning [9], and training-free methods [42, 58). Nevertheless, these works either suffer from high complexity from the need to train other models [9, 62] or high computational costs [42]. We direct readers to Sim et al. [51] for a more extensive survey on diverse data valuation approaches.
Influence Functions. Influence functions, a classic concept from robust statisties [20], estimate the infinitesimal effect of removing or adding training data point without model retraining. They have various applications in machine learning, such as interpreting the model's behavior [21, 43. 17] and curating training datasets [37, 11]. However, when applied to large neural networks, the computation of the iHVP and its dot product with all training examples introduce scalability challenges. Besides gradient projection, past works have explored computing influence functions only on the last (few) layers [33, 49] to mitigate these challenges. However, subsequent works [12, 17] have shown that the influence on only subset of layers is insufficient to capture the overall influence of training data point. To avoid computing the gradient of all training examples, various filtering strategies, such using the similarity in the model's representation space [19] TF-IDF [61, 17]. have also been

---

proposed. While it is possible to adopt these filtering strategies for LOGRA, they may introduce bias the selection of the most influential sequences. For example, filtering candidate training sequences with TF-IDF might miss interesting influential sequences that do not share many tokens but are semantically related. Recently, similarly to LOGRA, Datalnf [35] and LESS [60] proposed using LoRA to efficiently compute influence functions. However, these approaches are only applicable in finetuning settings, whereas LOGRA also supports influence analyses for pretraining. Concurrently, GRASS [25] also explored leveraging the Kronecker-product structure of the linear layer gradient to improve the scalability of influence functions. Unlike LOGRA that achieves gradient projection using the bottleneck architecture, GRASS handles gradient projection by explicitly attaching forward and backward hooks to linear layers.
Conclusion
In this work, we explored scaling data valuations with influence functions billion-scale models and datasets as a potential technical solution to properly credit or compensate data providers for training LLMs. Toward this goal, developed novel gradient projection algorithm that significantly improve the scalability influence functions, and designed a simple and interoperable software. Our experiments showed that LOGRA achieves competitive accuracy to other more expensive baselines on counterfactual evaluations, while efficiently scaling to billion-scale models and datasets, thereby demonstrating the initial potential of the practical data valuation system. Last but not least, discuss broader impacts and limitations of our work in Appendix F.
References
[1] AI@Meta. Llama 3 model card, 2024.
[2] Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse Training data attribution via approximate unrolled differentiation, 2024.
[3] Juhan Bae, Nathan Ng. Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953-1799 17967, 2022.
[4] Elnaz Barshan, explanatory training samples via relative influence. In International Conference Artificial Intelligence and Statistics, pages 1899-1909, PMLR, 2020.
Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying
[5] Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions deep learning are fragile. arXiv preprini arXiv:2006. / 14651, 2020.
[6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'B O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[8] Yixiong Chen, Alan Yuille, and Zongwei Zhou. Which layer is learning faster? systematic ex- ploration of layer-wise convergence rate for deep neural networks. In The Eleventh International Conference on Learning Representations, 2023.
[9] Sang Choe, Sanket Vaibhav Mehta, Hwijeen Ahn, Willie Neiswanger, Pengtao Xie, Emma Strubell, and Eric Xing. Making scalable meta learning practical. Advances in neural informa- tion processing systems, 36, 2024.
[10] Pradeep Dubey, Abraham Neyman, and Robert James Weber. Value theory without efficiency. Mathematics of Operations Research, 6(1):122-128 1981.
[11] Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024.
10

---

[12] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2899 2020.
[13] Raul Castro Fernandez. Data-sharing markets: Model, protocol, and algorithms to incentivize the formation of data- sharing consortia. In Proceedings ACMSIGMOD International Conference on Management of Data. 2023.
[14] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster. Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.0002: 2020.
[15] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242-2251. PMLR, 2019.
[16] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019.
[17] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukošiūte, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023.
[18] Roger Grosse and James Martens. A kronecker-factoree approximate fisher matrix for con- volution layers. In International Conference on Machine Learning, pages 573-582. PMLR. 2016.
[19] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint arXiv:2012.15781, 2020.
[20] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american stutistical association, 69(346):833333, 1974.
[21] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06670 2020.
[22] Kazuaki Hanawa, Sho Yokoi. Satoshi Hara, and Kentaro Inui. Evaluation of similarity-based explanations. arXiv preprint arXiv:2006.04522 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[24] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[25] Pinghang Hu, Joseph Melkonian, Weijing Tang, Han Zhao, and Jiaqi W. Ma. Grass: Scalable influence function with sparse gradient compression. In Advances in Neural Information Processing Systems, 2025.
[26] Jie Huang and Kevin Chen-Chuan Chang. Citation: A key to building responsible and account- able large language models. arXiv preprint 20X 2023.
[27] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry, Datamodels: Predicting predictions from training data, 2022.
[28] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[29] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019.

---

[30] J.L. et al. v. Alphabet Inc. Case 3:23-cv-03416, N.D. Cal., 2023.
[31] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE 2019.
Trunsuctions on Big Data,
[32] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[33] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, puges 1885-1894. PMLR, 2017.
[34] Narine Kokhlikyan. Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generie model interpretability library for pytorch. arXiv preprint arXiv:2009.07896, 2020.
[35] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in loRA-tuned L.MSs and diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[36] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint 1xxi::2110.14045 2021.
[37] Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9274-9283, 2021.
[38] James Martens and Roger Grosse.
Optimizing neural networks with kronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408-2411 PMLR, 2015.
[39] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint urXiv:1609 (684497774, 2016.
[40] Cade Metz. Lawsuit takes aim at the way A.I. is built. New York Times, 2022.
[41] Neel Nanda and Joseph Bloom. Transformerlens. 2022.
https://github. com/
TrannsooreeLLennDrg/Traaaa
[42] Ki Nohyun, Hoyong Choi, and Hye Won Chung. Data valuation without training of model. In The Eleventh International Conference Learning Representations, 2023.
[43] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. TRAK: Attributing model behavior at scale. In International Conference Machine Learning, pages 27074-27113, 213... PMLR, 2023.
[44] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent, Advances in Neural Information Processing Systems, 11119920-19930 2020.
[45] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 67, 2020.
[47] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
12

---

[48] Yasaman Razeghi, IV RobertL.Logan, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Conference on Empirical Methods in Natural Language Processing, 2022.
[49] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 8179-8186, 2022.
[50] Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsifica- tion in distributed deep learning. arXiv preprint arXiv: 2019.
[51] Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning:" ingredients", ; strategies, and open challenges. In IJCAI, pages 5607-5614, 2022.
[52] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:240..00159 2024.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Jiachen T Wang and Ruoxi Jia, Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388-6421. PMLR, 2023.
[55] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Helen Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Newral Information Processing Systems, 2017.
[56] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[57] Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. Unifying corroborative and contributive attributions in large language models. arXiv preprint arXiv:2311.12233, 2023.
[58] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning, pages 24150-24170 24176. PMLR, 2022.
[59] Zhengxuan WU, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D Goodman, Christopher D Manning, and Christopher Potts. pyvene: A library for understanding and improving pytorch models via interventions. arXiv preprint arXiv:2403.07809, 2024.
[60] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 24Xi:22422.0433 2024.
[61] Chih-Kual Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar, First is better than last for language data influence. Advances in Neural Information Processing Systems, 35:32285-3222 -32298, 2022.
[62] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842-10851. PMLR, 2020.
[63] Boxin Zhao, Boxiang Lyu, Raul Castro Fernandez, and Mladen Kolar. Addressing budget allocation and revenue allocation in data market environments using adaptive sampling algorithm. In International Conference on Machine Learning, pages 42081 42097. PMLR, 2023.
13

---

A Qualitative Analysis
In this section, we provide more qualitative analyses on most valuable data identified by LOGRA. In particular, we show top-2~-4 valuable data for each query here, given the possibility that the model utilizes information from multiple training examples for generating its output. We also include several failure cases where identified most valuable data do not share qualitative similarities with the queried LLM output.
A.1 Llama3-8B-Instrun
A.1.1 Example 1
Prompt
To improve data privacy,
Response
improve privacy. consider the following strategies:
Implement robust data encryption: Use end-to-end encryption protect data in transit and at rest. This ensures that even if an unauthorized party gains access to the data, they wan't be able to read or understand it.
2. Use secure protocols: Use secure communication protocols, such as HTTPS, SFTP, and SSH, to protect data during transmission.
Limit data collection: Only collect the minimum amount data necessary achieve your goals. Avoid collecting sensitive data unless it's absolutely necessary.
4. Pseudonymize or anonymize data: Consider pseudonymizing or anonymizing data to protect individual privacy while
still allowing for data analysis and research.
only authorized personnel can sensitive data. This includes multi-factor suthentication, role-based access control, and data masking.
Implement access controls: Establish strict controls
6. Monitor and audit data access: Regularly monitor and audit data access to detect and respond to potential data breaches.
Use data minimization techniques: Implement data minimization techniques, such as data aggregation, reduce amount of sensitive data stored and processed.
8. Conduct regular security assessments: Conduct regular security assessments and penetration
First Most Valuable Data
American public, overseers and stakeholders. (3) Improve civil liberties and privacy protections by supporting research, education and training. (4)Advise NSA leaders on liberties and privacy issues.
Ihis new report, the CLPO uses framework Fair Information Practice Principles (FIPPs) to evaluate how safeguards general civil liberties and privacy. Fair Information Practice Principles FIPP's broken down eight founding parts federal agencies to adhere to. These principles are:
Transparency: organizations should be transparent and notify individuals regarding collection, use, dissemination, and maintenance of personally identifiable information (PII)
organizations should transparent personally identifiable information (PII) Individual participation: (Skip).
notify individuals regarding collection, use, dissemination, and maintenance
organizations should involve the individual in the process of using Pil. (Skip)
organizations should specifically articulate the authority that permits the collection of Pll and specifically articulate the purpose(s) for which will be used Data minimizations: (Skip)
organizations should only collect PIl that is directly relevant and necessary to accomplish the specified purose and only retain PIl for as long as it necessary to fulfill the specified purpose. (Skip)
Organizations should use Pil solely for the purpose specified in the notice. (Skip)
Organizations should protect PIl (in all media) though appropriate (Skip)
Second Most Valuable Data
(Skip) Easy reporting with out-of-the-box compliance reports To achieve GDPR compliance, organizations need to demonstrate that they monitor crilical infrastructure holding personal data of EU cilizens.
Using powerful compliance and forensics reporting engine, Exabeam generates series of built-in, GDPR-specifi reports, that help reduce the time to prove compliance auditors.
Protect employee Personally Identifiable Information (PII) critical GDPR requirement is to protect employee Pll from unwarranted access. Exabeam provides role-based access control (RBAC) that can be used to enforce PIi data masking. (Skip)
Figure 6: Llama3-88 Bllama--B-IInstee IB-Irrtt data valuation result.
14

---

A.1.2 Example
Prompt
How can one become billionaire?
Response
Becoming billionaire requires combination of hard work, smart financial decisions, and bit of luck. Here strategies that can increase your chances of achieving this goal:
Start with great business idea: Identify a market need and create product service that solves prablem or competition.
meets a demand. innovalive stay ahead
Work hard and be persistent: Building successful business takes time, effort, and perseverance. Be willing the long hours and make sacrifices to achieve your goals.
3. Develop strong entrepreneurial mindset: Be willing to take calculated risks, adapt to changing circumstances, and continuously learn improve.
Build strong team: Surround yourself with talented and dedicated individuals who share your vision and values.
5. Make smart financial decisions: Invest wisely, manage your finances effectively. and avoid debt.
6. Diversify your investments: Spread your investments across different asset classes, such as stocks, real estate, and
bonds, to minimize risk.
patient and disciplined: Building wealth takes time, patient disciplined your financial decisions.
Stay informed and educated1: Continuously educate yourself business, finance, and investing to make informed
First Most Valuable Data
(Skip) Make the most money. Demand for WordPress development high and clients willing pay for developers who are the best in their field.
Demand for WordPress development is high and clients are willing pay more for developers who are the best in their field. Get the best clients. When you at the you have the freedom to "No" to the projects you don't want, and "Yes" to the projects you do.
When you are at the top, have the freedom to say "No" to the projects don't want, and "Yes" to the projects do. Have the most influence. Being at the top means you have influence (and responsibility) and the capability to shape the future of WordPress as well as the ecosystem that is built around it.
One Hour of Reading Day
If you're going to make it to the top. then you need to spend at least one hour each workday focused on reading and learning more about WordPress - outside of any development work. (Skip)
Second Most Valuable Data
(Skip) hugely successful game about psychology than merely some lovely design cool coding.
As successful gaming company, you need to:
Take time to understand Human Psychology
Give before you receive
Be patient and persistent, but not offensive, your efforts to sell Apply. Leam. Revise. (epeat.(Skip)
Third Most Valuable Data
(Skip) Unfortunately, many companies attempt become agile by implementing only process part and ignoring the harmony. think we've been there. don't claim be expert area, here some pointers that have helped build harmonious in the past.
fundamental change in people management that
careful when shuffling team members. (Skip)
Control negativity by reframing problem. (Skip)
Praise in public, have difficult conversations in private. (Skip)
4: Rephrase criticisms into request for help. (Skip)
5: Protect your team at all costs. (Skip)
6: Turn problems into action items. (Skip)
Figure 7: Llama3-8B-Instte data valuation result.
15

---

A.1.3 Example
Prompt
Do aliens exist?
Response
The existence of aliens, extraterrestrial life, topic angoing debate and research in the fields of astrobiology, astrophysics, and the search for extraterrestrial intelligence (SETI). While there is currently no definilive evidence of allen life, there are many reasons to believe that the possibility of life existing elsewhere in the universe is quite high.
Here
key arguments for and against the existence of aliens:
Arguments for the existence of aliens:
The vastness the universe: With an estimated 100-400 billion stars the Milky Way galaxy alone, and over 100 billion galaxies in the observable universe, the potential for life-suuporting planets immense.
The discovery of exoplanets: Over 4,000 exoplanets have been discovered so far, and many these planets are believed to be located in the habitable zones of their respective stars, where conditions are suitable for life as we know it. 3. The building blocks life: Many organic compounds, such as amino acids and sugars, have been found in meteorites and interstellar space, suggesting that the raw materials for are widespread in the universe. The existence of extremophiles: Earth, there informed
First Most Valuable Data
(Skip) Prior to Seticon Il in 2012 Tarter addressed concems about alien invasion, like those seen science fiction films, in particular those expressed by Stephen Hawking. Correlated with Hawking's reservations is his bellef that humans should not attempt to broadcast signals in order to contact alien civilizations (Active SETI). Tarter thinks that aliens evolved enough to communicate with and travel to Earth will be long-lived enough to have needed to grow beyond violent tendencies. [25]
[26)
"Otten the of science fiction say more about they do about themselves... While Stephen Hawking warned that alien life might try conquer colonize Earth, respectfully disagree. aliens able visit Earth that would mean they would have technological capabilities sophisticated enough to need slaves, food, or other planets. (Skip)
Second Most Valuable Data
(Skip) ...Or Are The Aliens Just Hiding?
There is another class of possible solutions to the Fermi paradox that boll down the fact that allen civilisations do exist, but simply see evidence of them.
Distance scales: Perhaps civilisations are spread too thinly throughout the Galaxy to effectively communicate with each other? Civilisations may be separated space, and also in time, so two civilisations just don't overlap during the time that they're each active.
Technical problems: Maybe we're not looking in the right place, or in the right way? Or maybe we just haven't been looking for long enough? Perhaps we've not recognised signal that's out there, because the alien civilisation is using technology that we simply cannot comprehend.
Alcamo Marina - Torre Alkamar in zona Battigia Wikimedia Commons Isolationist: Perhaps the aliens are out there, but they're choosing to hide themselves from us? Perhaps everyone is listening, but nabody is transmitting? It may be that other civilisations know we're here, the Earth purposely isolated, as we're some kind of exhibit in a zoo. (Skip)
Third Most Valuable Data
(Skip) About aliens, Li claimed that extraterrestrial aliens are actively intervening in human [ Li claimed that aliens developed and introduced the technology used by humans today.66][69] Li has denounced modern technology part of an alien plot against humanity[[7] Li believes humans are being impersonated by alien agents.[71] <|begin_ of tentt-Cotrrcttr Safedem used 'good practice' but "underestimated the robustness' of the buildings. (Skip)
Figure 8: Llama3-8P -------nnn In-Irutt data valuation result.
16

---

A.1.4 Example 4
Prompt
What is the difference between machine learning and deep learning?
Response
Machine learning and deep learning both subfields of artificial intelligence (Al) that involiee training algorithms perform specific tasks, The key difference between them lies in the complexity and depth of the algorithms used.
Machine Learning (ML):
Machine learning type of that enables computers learn from data without being explicitly programmed. involves training algorithms on labeled data recognize patterns and make predictions decisions. ML algorithms can be categorized into three types:
Supervised Learning: The algorithm learns from labeled data to predict outcomes.
Unsupervised Learning: The algorithm discovers patterns and relationships unlabeled data.
3. Reinforcement Learning: The algorithm learss through trial and error by interacting with environment.
Machine learning algorithms are lypically shallow, meaning they have limited number of layers (usually 1-2) and are designed to solve specific, well-defined problems. Examples of machine learning applications include: Image classification (e.g.. recognizing objects in images)
Natural Language Processing (NLP) for text classification sentiment analysis
Recommendation systems for personalized product suggestions
Deep Learning (DL):
Deep leamming subfield of machine learning that involves the of neural networks with multiple layers
First Valuable Data
(Skip) three lypes mainly differ from one other in the they use the machine. Supervised Learning. the machine trained using certain dalaset. (Skip) Unsupervised Learning, machine certain arguments
trained using initial dataset rather it has only a function which accepts computations, an outcome generated. (Skip)
performing Reinforcement Learning. machine is yet again trained any initial dataset then programme executes, it learns, formats
mistakes and improves (Ski))
Like mentioned before, Artificial Intelligence and Machine learning go hand hand.(Skip)
Second Most Valuable Data
(Skip) In supervised learning, learning itself is what you care about. You've got your cost function. which you want to minimize. In unsupervised leaminn, the goal is always to help some other task, like classification categorization. For example, might ask a computer system passively watch lot of YouTube videos (so unsupervised learning happens here), ask recognize objects great accuracy (that's final supervised learning task). (Skip)
DB: What are some other areas where you see exciting progress?
general direction believe be extremely important are learning models capable of more sequential computations? mentioned how think that deep leamning successful because than previous ("shallow") models. (Skip)
more sequential computations
Third Most Valuable Data
(Skip) Eleni: Can you clarify the difference between AI, machine learning, and deep leaming?
Inga: Al is the umbrella over machine learning. AI, to is our hope for human intelligence exhibited machines. There multiple goals of Al, which all facets human intelligence. We reason, accumulate knowledge, plan for things, manipulate objects, and communicate with each other with language. Leaming, of human.
part of what makes
Machine learning the ability learn without being explicitly programmed. The reason why machine learning has become popular in recent years that teaching machine to leam-or building machine that can lear on its wwn-can achieve all the goals of Al. It can learn how to reason, and understand our language, and perceive and move objects. You give machine leamning algorithm lots and lots of data and that algorithm leams the concepts around this data. It's then able to make a determination or prediction about something in the world.
Deep learning is the newest field of machine leaming, and it has really catapulted us into the renaissance that we're in today. The "deep" in deep learning comes from multiple hidden layers of transformation in data. Examples of things that are enabled by deep learning are self-driving cars or the Google Translate app in which the camera on your phone can immediately translate text that sees into a different language. (Skip)
Figure 9: Llama3-88 Baamaa--B-Inntteee SB-Instruct data valuation result.
17

---

A.1.5
Example
Prompt
Create comic strip Illustrating the life of software engineer.
Response
Here is a comic strip illustrating the a software engineer:
Panel Morning Routine
[Image description: A software engineer, John, sitting at his desk with of coffee and laptop. He's wearing t-shirt and jeans, and has messy hair.]
John: (thought bubble) Ugh, another day, another bug to squash.
Code Review
[Image description: John sitting at his desk, staring his laptop with puzzled expression. His manager. Sarah, is standing behind him, pointing at the screen.]
Sarah: "John, this code is a mess. Can you explain what you're trying to do here?"
John: "Uh... um... was trying to... uh..."
Panel Meeting Mayhem
[Image description: John sitting in conference lead, Mark, is talking onthusiastically]]
with his surrounded whileboards and sticky notes. The team
Mark: "And then, we'll integrate the new API with the existing framework, and-'
John: (thought bubble) Zzz... heard this before...
Panel 4: Lunch Break
[Image description: John walking the cafeteria, carrying sandwich and book.
First Most Valuable Data
(Skip) Pyrrha woke up after dawn. She got up, made quick stop bathroom, then wandered the lounge. of the couches, magazine her
Summer was sitting
steaming cup coffee on table before
"Morning Pyrrha"" "
"Good morning Summer." Pyrrha yawned. She took
couch opposite Summer stretched out.
"Pyrha, everything alright?" Summer asked.
"Yes, why?" Pyrrha asked.
"Did you have bad dream last night?" Summer asked. heard whimpering..."
'Oh...uh...I don't know." Pyrrha lied. "I usually don't remember my dreams." Summer shrugged and went back to her magazine.
After few minutes Pyrrha got and made her to the kitchen. She some bread the toaster and started making some coffee. As the freshly made toast popped Yang walked in with yawn and stretch. "Hey Pyr." Yang said. "Smeils good."
"Good morning Yang." Pyrrha said. "Sleep well?" (Skip)
Second Most Valuable Data
(Skip) Ruby cheered. wish you both
been there."
"This your work Ruby laughed. can persuaded to."
Blake chuckled. don't suppose you commissions?"
(Skip)
As she had after the previous class, Ruby packed up and rushed home once it was She one of the last to leave, working late finish up her sketch Pyrrha Mistrali hoplite. Few students wandered paths this but she not interested meeting with anyone anyway. Ruby just wanted get back her room and rest after long night. Walking fast she turned corner around small maintenance shack and nearly ran into girl standing the middle of the path.
"Warh..sorr!!" Ruby gasped. The girl. mint-haired and olive-skinned, just stared back. "Um..." "Ruby Rose." The girl said flatly. "Uh...eah.." Ruby confirmed. "Who you?" The girl shifted her position, closing Ruby while simultaneously forcing her loward the wall of the shack. Ruby backed up nervously but soon bumped into the wall. The girl leapt forward and grabbed her by the shoulders. "You didn't think you'd get away with that little stunt did you?" The girl hSkip).(Skip)
Figure 10: Maana3-8B-Instruee data valuation result. LOGRR identifies novel literature most valuable data.
18

---

A.1.6
Example 6 (Failure)
Prompt
Compose tweet that addresses the issue of environmental degradation.
Response
"The clock is ticking! Our planet is crying out for help as environmental degradation worsens. Let's take action NOW to reduce plastic waste, conserve water, and protect our natural habitats. The future of our planet depends on it #GoGreen #Sustainability #EnvioomeenallPootcioon
First Most Valuable Data
Learn more about the winner and the contest results (good news: behavior changed for the better!) in our recap blog post and this one-page summary
Download. Drive better. Win.
Up for a little friendly competition and safer streets?
Seattle's Safest Driver is an app-based safe driving competition, complete with cash prizes and bragging rights. In partnership with PEMCO Insurance and Cambridge Mobile Telematics, the 2-month competition seeks change behavior and support Seattle's Vision Zero goal of ending traffic deaths and serious injuries on city streets by 2030.
Step 1 Get the app and get moving.
Please note, app is no longer available.
Download the free app and it'll start tracking your trips - by car, bike, transit (ferry, lool). You get scored for trips when
you're behind wheel. The app measures speed, acceleration, braking, comering, and phone distraction. safer you higher your score, and the more chances you have win variety of prizes (mmm hmm, prizes read about them in Step 3).
Anyone download the app,
have eligibility requirements for prizes. need resident of King. Pierce, Snohomish, Kitsap County, and a portion of your trips must occur within the city of Seattle. This after all, competition for Seattle's safest driver. For the details, check out the contest's terms and conditions.
least 18 years old,
Step Challenge your friends, family, and coworkers. Then beat them.
Settle the age-old debate of who's the better driver - invite your friends and family to compete and see how you rank against each other. Get feedback and tips how to improve, and earn achievement badges. Friendly competition fun, but traffic safety no game. Warking together, can make Seattle's streets safer for everyone.
Step 3 Compete for prizes!
We realize lakes some carrots to incentivize good behavior. Thanks to partnership with PEMCO Insurance company that shares our vision for safe streets eligible participants can compete for nearly $15,000 in prizes, including grand prize of $2.000. Cash. 2nd place gets $1,000, and 3rd place gets $500.
Second Most Valuable Data
(Skip) "I get very hangry if miss meal." The most important thing my food
that actually eat. For many years I didn't hunger pains, Which, recently learned, is quite common. never "wanted" to est. Too nauseous, tired, and pained to cook anything, essentially lived coffee and cookies during the day until about years ago.
These tips assume you're already your chosen type diet, if you haven't found yet there's treasure trove of condition-specfic advice available on the internet.
Tip #1
Remember why you're re doing an elimination type diet in the first place.
Despite having many things wrong with me, never would have agreed that food would make difference. someone said, "You should try eating better." would have thrown shoe at them. I have an imbalance in my brain, it's not food problem! Or, it?
My new relationship with food started out of necessity. After bad with narcolepsy meds, found myself unable to take big pharm meds anymore. Everything gave me terrible side effects, even over the counter meds. wasn't quite sure what to do and stumbled on a TedX talk by Dr. Terry Wahls. She had pushed her MS (multiple scierosis) into remission with just food. She did a lat of research on her own on pub med, in hopes of getting out of the zero gravity wheelchair she'd ended up in, despite the best doctors and treatments. (Skip)
Figure 11: Llama3 --B------B-- In--nnst data valuation result.
19

---

A.2 GPT2-XL
A.2.1 Example
Prompt
measure economy inflation,
Response
the Fed uses price index called CPI. The Bureau of Labor Statistics (BLS) calculates this number by adding all items in its Consumer Price Index and dividing by total employment get estimate for what consumers paying average each month.
The BLS also publishes another statistic that measures how much money being spent per worker: Gross Domestic Product GDP. This figure includes everything from consumer spending at stores like Walmart to government expenditures such as Social Security payments. It's calculated using data collected through surveys conducted with households about their income and expenses.
First Most Valuable Data
(Skip) According to the BLS, the average U.S. worker earned $22.87 hour during the month of March, which is exactly the same number we saw in February. So inflation is going up and wages are staying flat. That means that American family budgets are going to be squeezed even more. In addition, the numbers from the BLS show that it is still incredibly difficult to get a job. In fact, the average length of unemployment in the U.S. is now an all-time record 39 weeks. (Skip)
Second Most Valuable Data
(Skip) The Social Security Administration bases the COLA captures how fast costs rising for workers. Most seniors retirees longer have jobs. Health their biggest expense, and it's one of the fastest rising costs in America. (Skip)
of inflation called CPI-W, statistic that
Third Most Valuable Data
consumer consumption (+2.2%, week, yesterdayAdā, data showed that US consumption fell in Mar. even after the strong gains the 1st 2-months this year (-0.2% -0.11 1%, m/m). Analysts
(Skip) Despite positive spin
gain)
the report suggests payments government transfersAçā, *a,f. ItÄça,a..çs worth noting foreseeable pace deterioration
consumer Áçå, -Eomaa have received a temporary personal income falling
00.2%)
past 7-months, consumption payments supporting growth Jan. (+1.1%) Feb. (+0.4%), an (-3.7% and supplements wages salaries headline. personal savings advanced
expected remain
government
transfers
salaries Mar., along with weaker proprietors' Income (-0.6% mlm), (-0.1%). have all contributed this job and economic uncertainty, default this bound affect consumption. their inflation measure of choice opposed to using CPI, last monthAgā, -a,es defiator eased 40.6%, y/y. further evidence that this the lowest growth rate 50-years. With the yly rate +1 +1.8% the Fed should be concerned about disinflationary pressures! (Skip)
+4 44.2%, the PCE deflator Fed preferning
Fourth Most Valuable Data
(Skip) price inflation, which is rise in the general level of prices of goods and services in an economy over period of time.[65] A consumer price index (CPI) is measure estimating the average price of consumer goods and services purchased by households. The United States Consumer Price Index, published the measure estimating the average price consumer goods services in
Bureau Labor Statistics, United States.[66] It reflects inflation as experienced by consumers their day-to-day living expenses.[67 A graph showing the U.S. CPI relative to 1982-1988 1984 and the annual year-over-year change CPI shown at right. (Skip)
Figure 12: GPT2-XL data valuation result.
20

---

A.2.2 Example
Prompt
Dogs make better pets than cats because
Response
they are more social and have a higher tolerance for stress.
Cats, on the other hand, can very destructive to their environment left alone too long or not exercised enough. They also tend to get into fights with each other over food which why it's important that you keep them area where
much competition between your two dogs!
First Valuable Data
(Skip) 'Social sensitivity appears play important role pet shelter dogs' willingness engage in problem- solving behaviour, which could suggest generalized dependence on, deference human action," she said. WWile an increased proclivity for looking at humans may represent cognitive shift dogs compared with wolves, it does not necessarily suggest cognitive advancement. Dogs heightened social response may interfere with their problem-solving behaviour (Skip)
Second Most Valuable Data
(Skip) Bengals tend fairly high maintenance
Bengals do as rule tend to be demanding and fairly high maintenance cats. They often bond strongly with their families and can be very demanding their time and attention, which is just what many people want-but course, not for everyone, then it may not be the cat for you.
can drive some owners mad! you don't have time to devote Bengal every
Owning a Bengal can many ways more like owning dog than cat, as they house, yell attention want get involved everything that you are doing-and they very much dislike being ignored!
follow you around
They can be noisy
Before you welcome Bengal to your family, you should (Skip)
Third Most Valuable Data
(Skip) Temperament [ edit
Siamese are usually very affectionate and intelligent cats, renowned for their social nature. Many enjoy being with people and are sometimes described as "extroverts" Often they bond strongly to a single person. Some Siamese are extremely vocal, with a loud, low-pitched nicknames(6)-iie that has been compared to the cries a human baby, and persistent in demanding attention. These cats are typically active and playful, even as adults, and are often described as more dog-like behavior than other cats. (atss(14][15) (Skip)
siice-knownn as "Meezer", from which they get one of their
Figure 13: GPT2-XL data valuation result.
21

---

A.2.3 Example
Prompt
Will Al replace humans?
Response
question whether or not machines eventually be able outperform human beings in all areas a hot topic. The answer, according some experts and futurists, could come as soon 2040 too early that prediction have any meaning this point.
First Most Valuable Data
(Skip) Let's not place humanity in the position of those ants. Please encourage your students think not only about how to create Al, but also about how to ensure its beneficial use.
John Leonard:
TL;DR: Not any time soon: There's lot of merit to what [Hawking] saying, but I think the question is: what's the timeline? Is this going to unfold over years, decades or centuries? In the field, there's this feeling of exponential growth. But as roboticist trying to get robots to do things like drive on streets and walk up and down stairs, I can see clearly that there are whole other parts of the problem that remain unsolved. I think this notion of evil A.I., think have much greater societal challenges to worry
happens at all, is many. many years in the future-ard about, in the here-and-now. My view, as roboticist, is that trying to get robots to do things like drive safely in urban traffic, or make left turn,, or make real decisions within the wall of information- 30 years ago, and they're re still hard today. would claim our advances in a more broad A.I. are actually pretty lame; we're re not making as much progress as some people might say.
are problems that were hard
Will Robots Take Our Jobs?
TL;DR: by robots. you mean Amazon machines produce everything we need, the outcome will depend on how things are distributed. Everyone can enjoy a life of luxurious leisure if the machine-produued wealth is shared, or most people can end up miserably poor if the machine-owners successfully lobby against wealth redistribution. So far, the trend seems to be toward the second option, with technology driving ever-increasing inequality. Leonard:
TL;DR: Again, not any time soon (Skip)
Second Most Valuable Data
(Skip) How soon, and should welcome fear it?
big controversy, dividing even leading researchers, involves forecasting what will happen. When, ever, will outperform humans intellectual tasks, and
good thing?
Do you want superintelligence?
Everything love about civilization arguably the product of intelligence, so potentially do better by amplifying human intelligence with machine intelligence. But worry that superintelligent machines would end up controlling us and wonder whether their goals would be aligned with ours. Do you want there be superintelligent AI, i.e., general intelligence far beyond human level?
What Should the Future Look Like?
In his book, Tegmark argues that we shouldn't passively ask "what will happen?" as the future is predetermined, but instead ask what we want to happen and then try to create that future. What sort of future do you want?
If superintelligence arrives, who should be in control?
If you one day get an Al helper, do you want it to be conscious, i.e.. to have subjective experience (as opposed to being like a zombie which can at best pretend to be (onsci)uss?(Skip)
Figure 14: GPT2-XL data valuation result.
22

---

A.2.4 Example 4
Prompt
Interventions that may reduce drug abuse include:
Response
Encouraging parents to talk with their children about drugs and alcohol. Parents should be encouraged not only to discuss first place (e.g., by
the dangers of using illegal substances, keeping close eye what is going into coming of their home). Providing information regarding safe prescription medications such pain relievers and anti-anxiety medication; providing education for health care providers patients suffering from substance dependence. This includes educating physicians about prescribing oploids safely when prescribed appropriately. Educating students at school about the risks associated with illicitly obtained likely than adults engage risky behaviors given access to these products through friends or family members. In addition, many adolescents have idea where obtain heroin because it has been replaced by synthetic versions made without oversight by pharnaceutical companies. The availaailitt of this product increases risk factors including addiction, overdose death, criminal activity, and HIV transmission among injection users. Increasing awareness within communities surrounding the need to address issues related to youth involvement street crime. These efforts must focus specifically addressing the needs of young people involved in gangs rather than focusing solely upon those individuals engaged in other forms of violence.
they can help prevent from happening
narcotics. Students
First Most Valuable Data
(Skip) addictive and have horrific side effects."
The measure was sent to the governor on 35-21 vote. Quinn has said was "open minded" to the proposal after meeting with a veteran who used marijuana to ease pain related to war wounds. On Friday, his office said that he will "carefully review" the legislation when it reaches
desk.
The Democratic govemnr must weigh not only the merits of campaign next year. large extent. Quinn has positioned himself a liberal, signing on major income tax increase, approving civil unions abolishing
but also the politics, given his re-election
death penalty while in office.
Critics of the marijuana legislation argued that while the intentions may good, the bill would encourage gateway drug that could lead users harder substances, destroying families along way. (Skip)
Second Most Valuable Data
(Skip) who years age or (in Ontario], but we do expect that students do not show up to school possession alcohol or having consumed alcohol."
The Ontario government's health and physical education curriculum already teaches students about substance use addiction. The Durham board runs "healthy choices" program, which focuses on risks associated with alcohol and other drugs, whether legal or illegal.
Legalization likely won't "dramatically" change the conversation with students about marijuana, Bowyer said, because the impact and possible consequences of using it remain the same. (Skip)
Third Most Valuable Data
not for human consumption" or "nat for minars." " The clinical effects of most new designer drugs can be described as either hallucinogenic, stimulant, opioid-like. They may have combination these effects due easy accessibiliyy and rapid emergence of new designer drugs have
designer side-chain substitutions. created challenges for health providers when treating patients presenting with acute toxicity from these substances, many of which produce significant and/or life-threatening adverse effects. Moreover, the health care provider has way verify the contents and/or potency of the agent ingested because can vary between packages and distributors. Therefore, thorough knowledge of the available designer drugs, common signs and symptoms of toxicity associated with these agents, and potential effective treatment modalities are essential to appropriately manage these patients. (Skip)
Figure 15: GPT2-XL data valuation result.
23

---

A.2.5 Example 5 (Failure)
Prompt
Can humans fall in love with robots?.
Response
The answer is yes, But it's not romantic relationship, and the robot doesn't have to be human-like or even sentient for that matter - just as we don't need our pots to look like us (or at least act like them). The question whether people will want their own personal Al companions has been debated since before computers were invented; some argue they'll never happen because there are too many other things on Earth worth living for. Others say this isn't much about what can do but rather how well your life could work out if someone else was around all day instead of being left alone by themselves. And then there are those who think having an intelligent companion would make one more fulfilled than ever: "I'm sure I'd feel happier knowing my partner had something better going." says David Levy from University College London. He believes that while he wouldn't necessarily choose his spouse over another person, he might prefer him/her live longer.
First Most Valuable Data
(Skip) "This is of critical importance and perhaps one of the biggest factors eroding campaign finance contribution limits," said Craig Holman with Public Citizen. "Many single candidate super PACs are probably set up by the candidate's is own staff or supporters or family for that matter, so these are means for candidates and political parties to get around the limits, and they can take unlimited amounts of contributions."
Last election, former Florida Gov. Jeb Bush (R) was able to solicit nearly $100 million for the super PAC Right to Rise, because he did so before officially announcing he was running for president, working around laws prohibiting coordination. The earlier it was formed, and the longer he put off his declaration candidacy, the longer the super PAC could work with Bush's team and fill the group's coffers.
We haven't found any such blatant lies among this year's crop of super PACs, there are some familiar names. Main Voters' treasurer Seth Tanner, an alum of the teams Sen. Elizabeth Warren (D-Mass), former Gov. Richardson (D- N.M.). and its custodian records is Amy Pritchard, a political strategist and DNC alum. America First Action, Inc. lists Charles Gantt the custodian of records, was Chief Financial Officer Trump America, Inc. 736's treasurer, Kate Gage, is former Obama policy adviser. (Skip)
Second Most Valuable Data
(Skip) ArbCom should certainly examine how its procedures can contribute to harassment. The length of time involved major problem. An outside advisor would help. Smallbones (talk) 03:25, 21 June 2016 (UTC)
This would have to be handled carefully, but think that an outside advisor would be great idea. Most large and powerful committees should have someone to serve as a separate witness/observer. That person might not have the power to actually stop something, but would beneficial to have someone who could serve this position and raise valid points well being a good person that the committee could turn any questions they might have well. Tokyogir179 (talk) 03:41, June 2016 (UTC)
expert advisor on things like conflict resolution, group dynamics and ethics sounds like a good deal. would say an eighteen-month trial period that automatically expires unless renewed.
wouldn't say "adult supervision," but some
turns out that having this person around prevents the Arbitrators from doing their jobs, off she goes. Darkfrog24 (talk)
17:41, 1, 21 June 2016 (UTC) One problem I've noticed among long-serving admins that they become very immersed in Wikipedia's rules, to the point where they forget how unintuilive they be to newbs and even longstanding editors
who've focused on other areas. An outsider would give much-needed perspective. Darkfrog24 (talk) 03:14, 22 June 2016
(UTC) (Skip)
Third Most Valuable Data
(Skip) BB: OK, you have statistics back up, but it still seems like it's risky proposition. What's for the employer reduce hours?
MB: That's the thing. take lot credit for this, but it's my brother and my mother who actually started the company and started with six-hour workdays. But they worried that for the company, that you wouldn't get much done. But do. And the benefits are that we have happy staff. And we can hire the best staff. People come to us looking for jobs every week. And when we find good staff, we want to keep them because the staff is the most important thing have. Some our employees have been offered other jobs with more salary, and they didn't take the job. BB: So you have advantage over your competitors then. But wouldn't that advantage disappear if all of your competitors adopted a 30-hour work week as well?
MB: Of course. So let's not do that [laughs]. (Skip)
Figure 16: GPT2-XL data valuation result.
24

---

A.3 Pythia-I.4B (with many failure cases)
While a majority of experiments with Llama3-8B- and GPT2-XL returned semantically or stylistically similar texts as most valuable data, we observed that the quality of most valuable data from Pythia-I.4B 1.4B experiments are generally much poorer. Here, we provide one hypothesis behind this observation. Influence functions tend to give high score for the example that contributes most to decreasing (test) loss at the current weight [3]. At the same time, it is also hypothesized that different layers learn different concepts at different stages of training [8]. Combining these two facts, when interpreting influence analysis results, we need think about which features the model most likely learning at the current weight. Here, we specifically discuss two factors: training data quality and training steps. First, if the training data quality is low, then there would be a lot features (e.g., random email address) that are frequent enough in the training dataset to be considered learnable patterns. In other words, even though these features look redundant to humans, they may still be useful for decreasing loss from the model perspective. Second, many LLMs are only pretrained for a single epoch, or under-trained to their pretraining dataset. That being said, redundant features from the first point would likely still remain are captured by influence functions. In sum, we hypothesize that as the model is well-trained on a high-quality dataset, influence functions would capture more similar data to the query LLM output. This hypothesis may also explain the observation from Grosse et al. [17] that most valuable data identified by influence functions on larger models tend to share more semantic similarity with results on smaller models, noting that larger models tend to converge faster to the point where they can only further decrease loss by learning high-level features. With this, we present experiment results with Pythia- 1.4B below. (some of them are not totally bad, but mostly lack specificity to be considered "most" valuable data humans)
sarning-worthy features at the end of training and
25

---

A.3.1
Example
Prompt
Cristiano Ronaldo
Response
the best player in world football. Portuguese superstar has been named FIFA'ss World Player The Year a second time, after winning third d'Or
Sunday night (18 February).
First Most Valuable Data
(Skip) Just ask the U.S. Women's Soccer Team.
This stuff isn't just about statistics and averages. There many concrete and specific examples of the ways which the pay gap is real force women's lives today. (See hospital doctors, Jennifer Lawrence and even Buffer, company obsessed with eliminating the pay gap that took hard look at itself and found...a pay gap.) But while it's certainly disheartening that five members of the U.S. women's national soccer team had to file sult against the U.S. Soccer Federation to push for the same pay and bonuses their male counterparts, it also makes for kind of perfect, high-profile example of the pay gap in action just in time for Equal Pay Day. (Skip)
Second Most Valuable Data
Barcelona finish the league promising them Champions League spot.
Ronaldinho Gaucho Biography stats show that following year at Barceliona would be perhaps the best year Ronaldinhos career winning the league title 2004-05 and being named the FIFA Player of The Year 2004.
Ronaldinho took FIFA World Player The Year again following year, beating Chelsea's Frank Lampard and Barcelona teammate Samuel Eto'o become consecutive time FIFA World Player The Year. (Skip)
Third Most Valuable Data
(Skip) Best Championship Performance: Kawhi Leonard, who led the Spurs past Miami in the NBA Finals, averaging 17.8 paints and 6.4 rebounds on 61 percent shooting. He exploded in the last three games of the series, scoring 71 points on 24-for-35 shooting.
Best Comeback Athlete: Russell Westbrook, who missed two months because of reinjured knee, then returned to the Thunder lineup to help Oklahoma City finish the season with a 59-23 record and the second seed in the Westem Conference.
Best NHL Player: Sidney Crosby, who the NHL with 104 points on the season, including a league-high 68 assists, 17 points ahead his closest competitor.
Driver: Ryan Hunter-Reay, who became first American win Indianapolis 500 since 2006.
Best International Athlete: Cristiano Ronaldo, star forward for Portugal and Real Madrid who scored goals games on way to winning FIFA's Ballon d'Or as World Player the Year.
Best Male College Athlete: Doug McDermott, winner of the Wooden, U.S. Basketball Writers Association and Player of the Year awards who led the country scoring, averaging 26.7 points per game 52.6 percent shooting from the field. (Skip)
Figure 17: Pythia-I. 1.4B data valuation result. LOGRA captures the broad topic soccer but lacks the specificity (except for the third most valuable data, which states that Christiano Ronaldo is the best soccer play who won Ballon d'Or award).
26

---

A.3.2 Example
Prompt
The healthcare system in the United States is
Response
complex web interrelated systems and organizations. The complexity has increased as more people have access health care, but also means there are many different lypes or levels which patients can seen by providers (e... Read More
First Most Valuable Data
(Skip) Advertisement
Shelby has also had stranger ventures, including stint chicken farmer, founding Texas chili festival. once having brand of deodorant named after him, Carroll Shelby's Pit-Stop.
recent years, health has been issue, with Shelby recelving heart transplant 1991. The experience of the transplant powerful Shelby that started the Carroll Shelby Children's Foundation to help children in need of organ transplants. (Skip)
Second Most Valuable Data
T0GGI "imagee
urf""hhtpss//inng22222
iscribdassets. sscribaasets..ommigggdccmmmmme
"small Image wttp/httss//mn/2222222222.
fscribdassees.com// wwtph//cwcnntll12../o//////////s
==1"."mmdiium lttpaia_aaannaaaa"aaaaae"e""_
v=1", "large image uttp:http///mmgg111111111//////0
v= "large embed hmmbaed_aaaeuurtfttttt/////////
"smmll embedd image htt.:///aaa.....aaaaaaaoooooooon
iscribdassets
hid:2210788999
"iitl""""The Dark
_wwer_semeee"SSAAAA "USA TODAY
Comics" "word user wttp://www.....//o..oooooo/oott.
Comics" htt.g//.ww....../////////////a/.
v=1" "small image laaaage_urr"""ttppttsss wutp/ntss/igg/222222..oooooo/eeo
f.scribdassols.com// Cct.ii/e@onnni0ee00000oaaaaaaaa0 v=1", "m"emii image hmmaaaeur"""htpsss///iii" "ff":hhtts:///mmgy
Most Valuable Data
"=e"."mmdium
v= "=""r"""rgeeiim" haarrg__iaag_uuur"httttttts wttp:///hpps////g//////////////
v=1", v=1", "=r""""laree_mmmll"aa embed http:///mmma........ooo/////////
v=1", "=sa," embed image hmtpguuurr////t/ss//////////////
Escrridaasees httpa///wcooaooo0000000000aaaaa0
Figure 18: Pythia-1.4D data valuation result. We suspect that the random url in the model output
dominates the query gradient and affects the data valuation result.
27

---

A.3.3 Example
Prompt
Does artificial intelligence eliminate the need for human intelligence?
Response
The answer is no. Artificial Intelligence (Al) does not replace humans, can help us to do our jabs better and faster by automating some them or even replacing few tasks that are currently done manually many organizations today:
First Most Valuable Data
answer is "yes. "It the only path forward us as a species; we really have choice. This then becomes a matter of when and how. When will we invent the technologies to not only stop the output of greenhouse gasses, but reverse them? This is known as genengineering. and halds the true promise to fix problems we have created. There are proposals have autonomous ships sailing around the ocean that spray salt water, sulfur dioxide, other particles that reflect light planet
the atmosphere absorb some so the are other devices like filtrations systems that can pull carbon dioxide trees do), sequester so that we can reduce the existing levels. A company called CarbonEngineeriig is already daing this
warm.
the air (like
expanding as speak. truly think that this the future for resources fixing the planet and bringing back pristine state. The jobs of the future will likely related restoring the planet, re-seeding the world's biodiversity, restocking resources, fighting for oil water food, we bring humanity together to focus on the big picture: Our continued survival as whole.
future where we spend oceans, and making
breathable
again. When people can stop having
Second Most Valuable Data
(Skip) Now, Artificial Intelligence (Al) has come to rescue, this, as in so many other computation-intensivee tasks. An Indian startup, co-founded by three engineering dropouts, has hamessed AI, to power a tool, that will significantly reduce manual work involved such tasks. It will come as a blessing animation and game developers everywhere.
tool Norah.Al the work Vrushali Prasade, Shubham Mishra Harikrishna Valiyath
coincidentally aged 23, time), who quit BITS Pilani, Goa 2015 ca-founded the startup, Absentia, Bangalore. Norah arguably. India's first game workflow (the process creating game animation from story to product), driven by Neural Networks, Deep Learning and Artificial Intelligence. (Skip)
Figure 19: Pythia-1 1.4B data valuation result.
28

---

A.3.4
Example 4
Prompt
What the and cons of having robots every home?
Response
The Pros: Robots can do a lot more than humans, They have better eyesight, they don't get tired easily or need to sleep much (they're not like us), their hands aren't as clumsy when comes time for them take careaf things around you etc... The Cons You will be paying an arm & leg each month just so that your robot does what its programmed too! also very expensive if one is going buy all these machines from China which has been doing this since forever now. I'm sure there must some people who would rather pay $1000+ per year on top offrabat service but then again we live here... We'll see how long our government lasts with such policies!!
First Most Valuable Data
Former DeKalb County State's Attomny Clay Campbell attended Wednesday's sentencing and said was a sad end to an Attorey December shortly Curl was sorry he could not complete commitment to the Keller family noted absence Wednesday was likely sign
event demanded greater justice. Campbell, who
charged,
they about the agreement.
scratches Cur'ss chest, his multiple accounts of events and his attempt to flee Mexico and Louisiana, Campbell said the evidence and timeline would have held up trial. very likely would have been the longest trial DeKalb County history," he said. "I was 100 percent confident in the evidence."
From bone fragments that tested positive Keller's
important for the state's attorne to pursue justice the maximum extent the law victims it is protect the public.
Campbell said
Keller's 's family and supporters were not the only ones displeased with the agreement. Wednesday's sentencing started with plea from Moria Curl to her brother she shouted for him to turn down the deal
and fight for his freedom.
"They're railroading you!" she shouted to her brother she was escorted from the courtroom.
The reaction came somewhat of surprise to DeKalb County Public Defender Tom McCulloch, who said his client was at peace with his decision Wednesday moming. McCulloch said Curl maintains his innocence in an Alford plea while having a chance at life outside of prison with his scheduled release date to come when he is 71 years old.
In an Alford plea, the defendant maintains innocence but admits the evidence could convince a judge or a jury to find him guilty.
Second Most Valuable Data
(Skip) also been clear, said, enforcement officers aren't fully aware requirements placed on them by state and federal gun laws. Harden's letter seemed like a wamiin, Silvoso said.
"Here's our waming to you that know you're doing and we you stop," said. "And you from this day forward you are going continue this after we've warned you."
After the ATF investigation in the Sacramento area, Assemblssalllyman Roger Dickinson sponsored bill 2012 that would have allowed officers buy off-roster guns reseil them, closing exception the law.
The Sacramento Democrat said the law didn't make sense: guns which the state had decided not safe could be legally purchased law enforcement officers and then be sold anyone essentially putting banned guns into the state, underculting the purpose of the safe gun list.
"It was hard to come with rational justification for it," Dickinson, who left the Assembly in 2014, said Friday, (Skip)
Figure 20: Pythia-1 1.4B data valuation result.
29

---

B Code Examples
We provide simplified code for our language modeling experiment from Section 4.2 demonstrate usability of LOGIX. LOGIX will be open-sourced under Apache 2.0 license.
B.1 Log Extraction
import logix
from atttiatttttttc import Covariance
model, toxenizer, train_loader setup
Initialize LogIK
run - logix. init (project='lln", config"
Register the model
run. eatch type_ (ww.fflterrrmm..nnaaa fame_t"ltorr""al"")
Add LoGra ruc. add_lora()
Setup logging
setup("log'" "grad', "save"; "grad", "statietic"; ("grad": Covarience))
logging
for batch in train_loader:
data_id tokenizer. baaiiizerrbaach_ddee (batch ("input_id=*))
targets batch. patttppppp""aal"""
with (dnddaa_idddata..
mask-batch ["attention_naax"
User's existing trsining code
model zero_grad
in. (odel(*)
shift_logits in_logits :-1, :] contiguous
shift_labels targets I..., contiguouz
loss = =. entross. vieeaeeeeee..eeeeee.eeeeeeeeeeee view(-1),
(-ev shift_logits. (-1)),
shift_labele.
reductioa="sun"
indore__ndex--100
backward
Pinalize logging
logix.finalin finalize
B.2 Influence Computation
import logix
tokenizer, test_loader setup()
logix. (project=*lla", config="contig.yanl
run. vun.vat (ype.filler-(an
fame_filter=("ap"""
Load saved logs (e.g. gradient Hessian)
logiz. Lnitieliie_fron_lle
log_loader buw..lllee...daaaoorrbbbbo.....
logix. "e""" "grad"})
for batch test_loader:
data_id tokenizer tokem/zerr.batch.deeddd
batchh_eccocede (batch ["input_ "))
targets batch. (olabbbele""
with run run(data_id-dsta_t. mask=batch ["atteettion__aa="]]
model.zoro_grad()
im_logita = model(xxbater
shift_logita lm.logits [..... :-1, contiguous
chift_labels targete(..... 1:1. (ontiguuus))
loss - F.cross_entropy (shift__ogiia.. (-1, sbift, (-1)),
shift_labele. viex (-1), reductioa"'sun", ignora_index--100)
1osa. backward
det the (gradient) log for the test batch test_log - run run.get_log()
# Compute influence scores (vith 1 1-RealtIP) influence_scoree run compute_ intluttennninnneeeeeeee ftest_log, leg_ loader, code="cosina")
30

---

C Experiment Details
For EKFAC influence [17] and LOGRA, we the damping term in influence functions as 0.1 X mean(eigennallus)) for all layers following the practice in Grosse et al. [17]. Since most of the model parameters come from linear layer weights, we compute influence scores only for these parameters. This means that biases and Layer Normalization parameters are excluded in the influence score computation.
C.1 Quantitative Counterfactual Experiments
For all our quantitative counterfactual experiments, we project gradients onto low-dimensional space using LOGRA with 6; K. 128. We used the same experimental setup, including the configurations for the baseline data valuation techniques, from Park et al. [43] and Bae et al. [2]. We used one A100 GPU with 80GB VRAM for all our counterfactual evaluation experiments. For model training, we used hyperparameters in Table for each experiment.
FMNIST
CIFAR-10
WikiText
Model
3-layer MLP
ResNet-9
GPT2
Optimizer LR Scheduler
SGD-M
SGD-M
AdamW
None
Cyclic
None
Learning Rate
3e-2
4e-1
3e-5
Weight Decay
le-3
le-3
le-2
Batch Size
64
512
Sequence Length
N/A
N/A
512
Epochs
20
25
Table 2: Hyperparameter used in experiments in Section 4
Brittleness Test. For classification tasks, we first selected 100 correctly classified test examples when the model is trained the full dataset (across all 5 random seeds). Then, for each test example Ite, we identified the top-k influential data points using the data valuation algorithm, removed these training data points, retrained the model, and examined if this removal causes misclassification of zze on average (across random seeds). In Figure 4, we reported the fraction of test examples (out of 100) that get misclassified after removing most training data points. For the language modeling task, we selected the 50 test sequences, obtained the top influential training sequences using the data valuation method, and reported the mean test perplexity after removing the top-k influential sequences and retraining the model.
Linear Datamodeling Score (LDS). We measured LDS by generating 100 data subsets of size |S|| = DD/2. For each data subset, we retrained the model 10 times for FashionMNIST, 20 times for CIFAR-IO, and times for WikiText to construct the ground truth. The LDS results Figure 4 show the mean and standard deviation of LDS obtained from 5 distinctly trained models. A more detailed description of the LDS evaluation can be found in Park al. [43].
C.2 Scaling to Billion-Scale Models and Datasets
We used up to 4 A100 GPUs with 80GB VRAM for these experiments. To save the storage cost, we used k; = k. 64 for gradient projection in this experiment. Unlike counterfactual evaluations, as our LLM experiments do not require any retraining, there are no other noticeable hyperparameters to report. We used tf32 precision in all our LLM experiments prevent gradient quality degradation.
31

---

D Derivation of Lemma
Assumption In this work, make the following two assumptions on train & test gradient distributions and the Hessian H:
1. Given that language modeling falls under the maximum likelihood framework, replace the Hessian with the Fisher Information Matrix (FIM), and further approximate the FIM with the empirical FIM, i.e.,
(p(la)) logpo(ylz)) log log
Σ
[Vl Po (vnl (..)) V Po xn) ?
(Zn.Un)EDtr
Given that test data are directly sampled from the model given the prompts, we assume test gradients Dre and train gradients gor approximately follow the same distribution.
Lemma 1 Let (ei, ,nd and {A,,... An be eigenvectors and eigenvalues of the Hessian H. With Assumption and 9tr/te- Ei Ctr/tos (Viee,) the following holds;
X
(F(tte - + +X) Jtr Σ
A + Ste, and E|c,, x
Proof.
Let Q = |e1, ,,,
en] and = diag( (1,, An))
IF [F Fre) -H - 9tr
=QAT + + gtr
= -OTA( + + gtr
-9TAQ(A + AT) ghr
Σ Cro (VAres) Q(A QT (Iani ctni (V (vrrss)
V :een V Am
A Viii.. Cenn An
X
Ctr,iCte
Since we assume Stc and gtr follow the same distribution, we need to show Ef(?ral ~ 1 for all i.
A Q
Q
HQ
1
Σ
V
(Assumption
N
E
E
Q
T Q QT Σ cenii (VAres) (I Ceni (V (VAres)
E
E [ Ai; Ctrn Am [Ctr.1 /i;... ... Ctrin V Am
Inspecting diagonal terms, get X Elcr Elc2ryl>. Therefore, E.cl.dl R 1.
32

---

E LOGIX Details
In this section, we discuss several key differences between LOGIX and other interpretability tools, and optimizations we implemented in LOGIX.
E.1 Differences with Other Tools
Influence functions have been extensively studied as an interpretable Al method. Accordingly, there have been several tools originating the Al interpretability field that implement influence functions, with most notable examples including Captum [34]. TRAK [43], and Kronfluence [17]. Overall, the software design of these tools aim at easing the from--cratee implementation of influence functions by introducing a lot of abstraction, following the philosophy of high-level frameworks. In fact, such software designs were well-received in the pre-L..M era. Nonetheless, as scaling has become key aspect of AI research, the (LLM) development ecosystem has become complicated and being able to compatibly work with other tools in the ecosystem has become a core aspect in the ML software design. Hence, unlike existing software, the design of LOGIX aims at enabling the easy conversion of users' (already efficient) training codes into data valuation codes. This design is also motivated by the observation that gradient is simply a by product of the training procedure that most of the training code for data valuation without needing to write the gradient computation code from scratch as in other tools.
reuse
Recently, there have been active developments (mechanistic) interpretability software, represented by Transformerl [41] and pyvene [59]. Interestingly, these software also extensively use Py PyTorch hooks, similarly to LOGIX, probably due to its high compatibility with other features such as autocast, distributed data parallelism, fully-sharded data parallelism, and gradient checkpointing. Nevertheless, we point out two major differences between these (mechanistic) interpretability software and LOGIX. First, support for dataset-level statistics computations in LOGIX is largely missing in these tools. In data valuation, we often need to compute several dataset-level statistics such as the Hessian (or Fisher information matrix) for accurate influence computations, and therby supporting these computations seamlessly was an important design principle behind LOGIX. However, analyses in (mechanistic) interpretability research typically focuses each instance and computing dataset-level statistics is typically not supported. Second, support for efficient data I0 in LOGIX is not a priority in other tools. As we propose to convert the data valuation problem into vector similarity search problem with gradient projection, we put efforts into improving efficiency of data 10 (see the next subsection for details), whereas this issue is rarely considered in other interpretability tools. We hope to explore the possibility of supporting both data valuation and other interpretability research in unified way with LOGIX as our future work.
F.2 Optimizations
Efficient Data 10 With LOGRA, we propose to save projected gradients for all training data to disk, and frequently load them as a new test batch arrives. As a result, reducing latency from data IO renders to be critical in realizing efficient data valuation. In particular, as the total size of all training gradients is usually far beyond the limit of CPU memory, we should optimize data transfer between disk and CPU (or GPU). To address this issue, we adopted the memory-mapped files that bypasses the need for intermediate copying between kernel space and user space, reducing the overhead associated files is also motivated by the observation
with data 10 operations. The use of the memory-mapre that, given each query batch, data valuation often requires computing influence scores with all training data. Therefore, we can access training gradients in predefined or sequential order instead of in random order, which can be done efficiently with memory -aaped files (sequential access is faster than random access).
Moreover, we overlap memory wwwppd--ill--aseeeeee base- data IO with computations further enhance data valuation efficiency. In the logging phase, we overlap the process of saving gradients extracted from the current training batch to disk with computations for the next training batch using Python multiprocessing. In the influence computation phase, we overlap the process of loading saved training gradients from disk with computing dot product with the query batch using the pre-fetching feature of PyTorch DateLo
33

---

We also note that more efficient data I0 can be achieved by the use of more advanced techniques like GPU-accelerated vector database, especially in the production setting. While we considered supporting this feature, we decided to focus on the memory-mapped. saap-eddllee-aeedd data 10 in our initial version of LOGIX, as offers more flexibility to explore different algorithms in the research setting.
Memory Optimization When dealing with LLMs, GPU memory is often a major scaling bottleneck. To alleviate this issue, we support CPU offloading dataset-level statistics by utilizing the sequential nature of backpropgation. When this feature is enabled, we by default keep all dataset-level statistics (e.g.. gradient covariance) on CPU, move it GPU when the corresponding module is called during forward/backwan passes, and then move it back to CPU asynchronously as soon as updating statistics for the module done. Depending on the CPU-GPU communication bandwidth, this feature may slow down the logging process.
Communication Optimization If training data are split across multiple processes with distributed training, we need to aggregate dataset-level statistics across processes for consistency. To minimize the communication cost, delay the synchronization process until the training loop (one epoch) is over, and perform synchronization only once at the end. Following the similar logic, users can maximize the efficiency of the logging phase by disabling gradient synchronization (e.g., forch.no_syye
34

---

Broader Impacts & Limitations
F.1
Broader Impacts
The data valuation problem can be socially sensitive topic. As of now, we do not have the agreed- upon social norm for data valuation, and thus we refrained from discussing how exact data values should be determined based on our method. Rather, our work is an initial attempt to tackle the technical challenges enabling LLM-scale data valuation. For equitable data valuation, we believe future research for improving both accuracy and efficiency of data valuation systems along with extensive social discussions are necessary.
F.2 Limitations & Future Work
We generally observed that influence function approaches are susceptible to outlier data with large gradient norms. This outlier issue is particularly severe for language modeling tasks due the fact that the gradient of each sequence is the sum of gradients for all tokens in that sequence. If a few tokens in the sequence have large gradient norms, their gradients may dominate the total gradient for the sequence and hurt data valuation accuracy. While our work tried to reduce the outlier effect with (self-influence) normalization, exploring other filtering heuristics (e.g.. L2/ L1 norm ratio [17]) may be an interesting research direction.
We attempted lay the software foundation for data valuation with LOGIX, but did not implement extensive system support, such as high-performance vector datahase (e-g., Faiss [31]). We expect further system optimizations would enable significantly more efficient data valuation. To reduce the cost of influence functions, our work mostly explored low-rank gradient projection, which compresses the gradient in spectral domain essence. Noting that gradient compression has been extensively studied in the efficient distributed training literature, it is worth exploring (or combining) different gradient compression strategies, e-g., top-k compression [50] low-bit compression [55], further reduce the compute/memorr//soraage costs for influence functions.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims in the abstraction and introduction correctly reflect the contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Justification:
We discuss major limitations of work in Appendix F2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
36

---

We provided the full set of assumptions and a complete proof for Lemma in Appendix D.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide both the code and a set of hyperparameters to reproduce our experiment results in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
37

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release an open-source software package that can reproduce our experi- ments with extensive documentations. We used publicly available data for our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/gun wewggudidcCossuuee cetamiiissioooiiyo...oooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ wwtpppccpp//ccccc.cccccccc..lltl waabbic/guiiess. SodeSSumisssiiieeooooo www.issisiiiccyco.ccccmcmmmsiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all our experiment details including hyperparameters in Ap- pendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For our LDS experiments 4.1, we marked the variance from runs with error bars. For our brittleness test 4, evaluating each method requires 1800 training runs. Reported numbers are the average from such multiple runs, so we expect the variance is very small. For our LLM experiments, note that GPU memory usage and throughput largely deterministic in contrast to accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
38

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide all our experiment details including compute resources in Ap- pendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http://eee.pppeeeeee https//cccclcctcthcceciiiiiddddd
Answer: [Yes]
Justification: Our work conforms to the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in Appendix F.I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
39

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly credited (e.g. citation) the original (owner of) work in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
40

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
As part of the release of our software LOGIX, we provide extensive documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing and/or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing and/or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
We mainly used LLMs to extract gradient and Hessian information for influence functions.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... for what should or should not be described.
42


Reviews:
[{"review_id": "69hJUWaynU", "reviewer": "Reviewer_BWig", "summary": "This paper build on the use of influence functions to determine the relationship between model outputs and a vast set of training data. In particular the paper introduces an efficient method called LoGra that uses a low rank projection that can be efficiently integrated with the training process. The authors consider tradeoffs in the size and selection of the low rank projection (e.g. initialization through a PCA-like process). The method is first validated against competing methods in a small scale test using smaller models on typical datasets like CIFAR-10. The authors then test this method against a 1B token sized training set (a proxy used for access to actual training sets of models) and test realistic model LLM model sizes (i.e. billions of parameters). The authors show throughout where prior influence function methods may fail, often due infeasible computations and enormous systems costs.", "questions": "- Clarify the decisions around the 1B OWT data selection\n- Clarify the usability of the software package\n- Since qualitative comparisons (the example prompts) are part of the results for this method, are there possible experiments that could show that the data marked as valuable is of further use in model training? One version of this could use high value documents as ICL exemplars (or simply in context knowledge) to test the effects on evaluations.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "In general I found this paper to be very strong and convincing, with huge improvements over similar methods that allow this work to scale to both billion parameter data and models. While very dense, it is still clearly written and touches on a broad range of areas to show that their method is both theoretically sound and practically scalable. \n\n## Strengths:\n\nS1: The paper is very well motivated, discussing shortcomings in both \"accuracy and efficiency\" of existing systems (such as difficulties in computing solutions to the inverse hessian products). Continually considers practical limitations.\n\nS2: Considers tweaks to the core method such as the PCA-like initialization of the low rank projections. The explanations here (3.2) motivate this idea and help understanding of the overall method. \n\nS3: Considers systems aspects like maximizing available compute on typical GPU setups, recomputation, and storage costs of projected gradients. \n\nS4: Gives some details about a software implementation of this method through the pytorch hooks interface. If this works as advertised (see W2), it makes this method even more relevant as it could be easily applied in many complex existing training stacks. \n\nS5: Good baselines and comparisons of why some competing methods could not even be evaluated. I appreciate that the authors validate and compared the method on both small scale experiments similar to the literature and larger scales. \n\n## Weaknesses\n\nW1: I am a bit unclear on the selection of the 1B train tokens from OWT. This is meant to be a standin for direct actual access to a sample of the LLM training data, correct? Is it valid or accepted in data influence literature to use a proxy for training in this way? I believe these experiments could have used a subset of the dolma data and the olmo models, as these are both modern (i.e. compared to your issues with the pythia models) and open access. You note that OWT may be higher quality than the overal dolma mixture, but it also exists as high quality filtered subsets. \n\nW2: Since the software package is considered a key contribution of this work (i.e. mentioned in the abstract) it would be good to show a practical end-to-end implementation of the package. Was the same package used to produce the LLM experiments? \n\nW3: Could provide more details about the comment in 255-257 about complications (\"... (i.e., no weight sharing) may not successfully keep larger components of the GPT2 Hessian...) with regards to \"Transformer architecutre ... lacks the specialized KFAC Hessian approximation\", as part of the central premise of the paper is that the method can be applied to billion param scale transformer models.", "was_revised": true, "final_justification": "I will maintain my positive score (5). The authors clarified some points in their response and I do not consider any of the raised weaknesses a reason to avoid acceptance.", "rebuttal": "Thanks for your reviewing effort and positive feedback! Here, we attempt to answer your questions.\n\n---\n\n> Q. W1: I am a bit unclear on the selection of the 1B train tokens from OWT. This is meant to be a standin for direct actual access to a sample of the LLM training data, correct? Is it valid or accepted in data influence literature to use a proxy for training in this way? I believe these experiments could have used a subset of the dolma data and the olmo models, as these are both modern (i.e. compared to your issues with the pythia models) and open access. You note that OWT may be higher quality than the overal dolma mixture, but it also exists as high quality filtered subsets.\n\n\nA. We agree that OLMo + Dolma, similar to Pythia + Pile, is a valid experiment setup! For instance, C4 is a part of Dolma (similar to OWT being a part of Pile) that is frequently used in LLM pretraining, and we could have used C4 instead of OWT in our experiments. In our work, we chose one of a few reasonable data setups (i.e., OWT) and studied data valuation with multiple models (i.e., Pythia, GPT2, Llama3) to demonstrate the generality of the proposed method. While OWT may not be an exact subset of GPT2 and Llama3 pretraining data, we expect that OWT hugely overlaps with their pretraining data as mentioned in the paper, and thus the validity of our experiments largely hold. As for the common practice, we note that our work is one of very few work that studies data valuation at LLM pretraining scale with public models, so we hope that the more standardized experiment setup develops in the future.\n\n---\n\n> Q. Since the software package is considered a key contribution of this work (i.e. mentioned in the abstract) it would be good to show a practical end-to-end implementation of the package. Was the same package used to produce the LLM experiments?\n\n\nA. All our experiments, including ones involving LLMs, are implemented with LogIX, and we provide the skeleton-code for our experiments in Appendix B. Furthermore, we included all our experiment code in the attached supplemental material. As for the implementation of LogIX itself, we described several important design principles and implementation details in Appendix F.\n\n---\n\n> Q. Could provide more details about the comment in 255-257 about complications (\"... (i.e., no weight sharing) may not successfully keep larger components of the GPT2 Hessian...) with regards to \"Transformer architecutre ... lacks the specialized KFAC Hessian approximation\", as part of the central premise of the paper is that the method can be applied to billion param scale transformer models.\n\n\nA. KFAC-based gradient compression is proposed in Section 3.2 as a technique to potentially improve accuracy of data valuation with LoGra. That being said, LoGra enables efficient data valuation with billion parameter-scale transformer models regardless of KFAC-based gradient compression, as empirically demonstrated in Section 4.1. We leave designing more accurate structured gradient compression techniques, especially for transformers, as interesting future work!\n\n---\n\n> Q. Since qualitative comparisons (the example prompts) are part of the results for this method, are there possible experiments that could show that the data marked as valuable is of further use in model training? One version of this could use high value documents as ICL exemplars (or simply in context knowledge) to test the effects on evaluations.\n\n\nA. We would like to note that the brittleness experiment in Section 4.1 is closely connected to practical data curation tasks. For example, if the model shows an undesired behavior (e.g., gender/racial bias), we can trace this behavior back to its training data using influence functions and remove most influential data. Indeed, influence functions have a variety of applications including toxic dataset removal [1] and hallucination tracing [2]. Considering that the major bottleneck in using influence functions in those applications is the scalability, we focused on showcasing the scalability of LoGra in this paper. \n\n---\n\n[1] Coalson, Z., Bae, J., Carlini, N., & Hong, S. (2025). $ IF-GUIDE $: Influence Function-Guided Detoxification of LLMs. arXiv preprint arXiv:2506.01790.\n\n[2] Wuhrmann, A., Kucherenko, A., & Kucharavy, A. (2025). Low-Perplexity LLM-Generated Sequences and Where To Find Them. arXiv preprint arXiv:2507.01844."}, {"review_id": "0V5qVLwChU", "reviewer": "Reviewer_mPa1", "summary": "This paper proposes an efficient data attribution algorithm by leveraging the gradient-projection approximation of the influence function with the special gradient structure for linear layers. Besides the algorithmic advancements, the authors also demonstrate how the algorithm can be efficiently implemented with the existing PyTorch library and provide a library that is capable of large-scale attribution.", "questions": "1. I have played around with the provided library, and I have some questions: In the `LoraLinear` class, why is the initialization of the projection matrix using `nn.init.kaiming_uniform_`? In addition, it seems like the implementation does not handle the biases in the linear layers.\n2. (**Weakness 1**) Can the authors provide a rigorous statement of the Lemma and also the rigorous version of the proof? In particular, is it possible to avoid using $\\approx$ in the statement (and also the proof)?\n3. (**Weakness 3**) While I understand that the authors try to \"propose\" an overall logging/dot-product framework for data attribution, specifically the influence-function-style attribution, I think the key advancement lies in the introduction of the efficient gradient-projection in the logging stage. Hence, it is misleading to claim the speedups during the dot-product stage, especially when the baseline is largely limited by memory constraints. I suggest that the authors state the scenarios in which the speedup is obtained.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The research problem is well-motivated, and the scope is clear.\n2. The overall writing and the illustration are professional and easy to follow.\n3. The contribution is sound: in particular, the algorithmic design by exploiting the linear gradient structure, as well as the backward pass for the sample gradient calculation.\n4. The evaluation is complete and thorough.\n\n**Weaknesses**\n1. The theory is not instructive and not entirely rigorous (in terms of the statement as well as the proof).\n2. After some investigation, the structure of the codebase does not seem to be as extensible as claimed: the logic is scattered throughout the codebase, hard to track the lineage of definition for data/function.\n3. The discussion around throughput gained is overclaimed: it comes from the dot products part, instead of the key technical advancements of this work, i.e., the logging part.", "was_revised": true, "final_justification": "The authors addressed/answered all of my questions. In particular, the most important one being a potential over-claim on the speedup. For that, the authors also acknowledge the potential confusion, and promise to update to clarify this in the next iteration. Overall a technically solid paper that has a high impact on efficient gradient-based data attribution.", "rebuttal": "Thanks for your detailed review on our work and code! Here, we attempt to address your questions:\n\n---\n\n> Q1.1 I have played around with the provided library, and I have some questions: In the LoraLinear class, why is the initialization of the projection matrix using nn.init.kaiming_uniform_? \n\nA. We experimented with both uniform and normal initialization schemes for the projection matrices, but found no significant differences in our quantitative evaluations (regarding brittleness and LDS metrics). If you prefer to use different initialization strategies, you can easily modify it via `config.init_strategy`.\n\n> Q1.2 In addition, it seems like the implementation does not handle the biases in the linear layers.\n\nA. You're correct that we compute influence scores using only the weight parameters. Since weights account for the vast majority of network parameters, calculating influence scores based on weights alone typically provides sufficient accuracy. We'll clarify this experiment detail explicitly in our final manuscript to prevent the confusion for readers. Thanks for pointing it out!\n\n---\n\n> Q2/W1. Can the authors provide a rigorous statement of the Lemma and also the rigorous version of the proof? In particular, is it possible to avoid using $\\approx$ in the statement (and also the proof)?\n\nA. To our knowledge, approximating FIM with empirical FIM is frequently adopted in previous (inference function) literature [1, 2, 3, 4]. We used $\\approx$ to explicitly indicate the approximation process, but if the reviewer thinks it only adds confusion, we can remove it in our equations in the final manuscript.\n\n---\n\n> W2. After some investigation, the structure of the codebase does not seem to be as extensible as claimed: the logic is scattered throughout the codebase, hard to track the lineage of definition for data/function.\n\nA. We acknowledge that the codebase structure and function dependencies can be challenging to navigate. This complexity arose from our efforts to optimize logging performance and ensure compatibility with popular frameworks like Huggingface Transformers. Appendix E provides further details on the design considerations that shaped our implementation. When we claim extensibility, we're specifically referring to the statistics computation components (such as covariance, mean, and corrected eigenvalue calculations), which we anticipate will be the primary focus for practitioners looking to extend the framework. For instance, we expect fewer users will need to modify lower-level features like our memory-mapped file logging system.\n\n---\n\n> Q3/W3. While I understand that the authors try to \"propose\" an overall logging/dot-product framework for data attribution, specifically the influence-function-style attribution, I think the key advancement lies in the introduction of the efficient gradient-projection in the logging stage. Hence, it is misleading to claim the speedups during the dot-product stage, especially when the baseline is largely limited by memory constraints. I suggest that the authors state the scenarios in which the speedup is obtained.\n\nA. We highlight the dot-product stage speedups because, in practical workflows, logging happens once while influence computations may be performed repeatedly across different queries (as noted in L86-90). This means the dot-product phase often becomes the primary bottleneck when practitioners explore multiple queries. Furthermore, we believe that the improvements in both stages are closely connected: the enhanced computational and memory efficiency during logging allows us to store comprehensive gradient information (e.g., gradient covariance and per-sample gradients), which in turn enables rapid influence calculations (dot-products) against all training examples. This end-to-end optimization is what makes our framework particularly effective for iterative data attribution analysis.\n\n---\n\nThanks again for your detailed feedback, and let me know if you have any further questions!\n\n[1] Kwon et al., DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models. ICLR, 2024.\n\n[2] Wen et al., Interplay between optimization and generalization of stochastic gradient descent with covariance noise. AISTATS, 2020.\n\n[3] Chaudhari et al., Entropy-SGD: Biasing gradient descent into wide valleys. ICLR, 2017.\n\n[4] Kim et al., SqueezeLLM: Dense-and-Sparse Quantization. ICLR, 2024."}, {"review_id": "u9M0Xzj9bi", "reviewer": "Reviewer_fJkS", "summary": "This paper proposes a low-rank gradient projection algorithm named LOGRA, aiming to extend influence functions to LLM-scale data valuation tasks without significantly compromising accuracy. The authors also design LOGIX, a modular framework that seamlessly integrates data valuation procedures into standard training pipelines. Experiments demonstrate that this method substantially reduces computational and memory overhead while maintaining reasonable estimation quality.", "questions": "1.LOGRA's performance depends on the initialization of the Kronecker projection matrices (PCA or random). Is the method inherently less effective for architectures with less structured activations (e.g., Transformers with multi-head attention)?\n\n2.Does LOGIX support more complex training pipelines, such as multi-stage pretraining followed by SFT or RLHF? \n\n3.How do you assess whether the “valuable data” identified by LOGRA truly affect model behavior? Have you considered introducing metrics such as embedding coverage or goal-oriented perturbation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n1. Practical problem formulation: Data valuation is a highly relevant issue in the context of LLMs, especially with regard to legal attribution and compensation. The paper addresses the core challenge of extending influence function methods to modern large-scale models.\n\n2. Complete engineering implementation: The LOGIX framework is carefully designed with compatibility in mind, supporting modern PyTorch ecosystems such as FSDP. This adds practical value to the work.\n\n3. Well-executed quantitative experiments: The authors provide solid empirical evidence through standard metrics such as LDS and brittleness tests on small models.\n\n*Weaknesses*\n\n1. Limited generalizability: The effectiveness of LOGRA appears to rely on structural assumptions about the Hessian. For instance, PCA initialization performs poorly in Transformer architectures without weight sharing, raising concerns about broader applicability.\n\n2. Subjective qualitative analysis: The identification of “valuable data” relies heavily on human inspection of semantic similarity between training samples and model outputs. A more systematic and quantitative evaluation metric is lacking.\n\n3. Lack of ethical discussion: Despite the centrality of data ownership and provenance, the paper lacks a thorough discussion on potential misuse, legal concerns, or fairness implications of the proposed system.", "was_revised": true, "final_justification": "Most concerns are resolved.", "rebuttal": "Thanks for your reviewing effort. We attempt to address questions raised by the reviewer here.\n\n---\n\n> Q1/W1.LOGRA's performance depends on the initialization of the Kronecker projection matrices (PCA or random). Is the method inherently less effective for architectures with less structured activations (e.g., Transformers with multi-head attention)?\n\nA.Our results in Section 4.1 demonstrate that LDS scores for WikiText+GPT2 (using both random and PCA projections) actually exceed those obtained for FashionMNIST+MLP and CIFAR10+ResNet-9 experiments. This suggests that LoGra's effectiveness for architectures with less structured activations may not be inherently limited as the question implies. That said, we recognize that understanding the factors affecting data valuation accuracy—including the impact of Kronecker projection matrix initialization—represents a valuable direction for future research.\n\n---\n\n> Q2. Does LOGIX support more complex training pipelines, such as multi-stage pretraining followed by SFT or RLHF?\n\nA. Yes, LogIX supports diverse training pipelines. Thanks to its modular software design, for instance, LogIX integrates seamlessly with Huggingface Trainer, which is commonly used for various post-training stages including SFT and other fine-tuning approaches.\n\n---\n\n> Q3/W2. How do you assess whether the “valuable data” identified by LOGRA truly affect model behavior? Have you considered introducing metrics such as embedding coverage or goal-oriented perturbation?\n\nA. We extensively evaluated how valuable data identified by LoGra affects model behavior through our brittleness and LDS experiments in Section 4.1. Specifically, our brittleness experiments, which is recognized as one of strengths of our work by the reviewer, demonstrate that removing the most valuable data selected by LoGra results in non-trivial increases in loss values for target text examples. I assume this is conceptually close to the goal-oriented perturbation suggested by the reviewer (please correct us if we are wrong). If our writing was not clear, please let us know how we can improve it. \n\n---\n\n> W3. Lack of ethical discussion: Despite the centrality of data ownership and provenance, the paper lacks a thorough discussion on potential misuse, legal concerns, or fairness implications of the proposed system.\n\nA. Given NeurIPS's technical focus, we prioritized addressing the computational challenges of enabling LLM-scale data valuation, as outlined in Appendix F.1. For readers interested in comprehensive discussions of AI data governance, including ethical and legal considerations, we recommend references [1, 2, 3] and will add these citations to Appendix F.1 to guide audiences seeking deeper exploration of these important topics.\n\n---\n\n[1] Pahune et al., The importance of AI Data Governance in Large Language Models, 2024.\n\n[2] Reuel et al., Open Problems in Technical AI Governance, 2024.\n\n[3] Cheng et al., Training Data Attribution (TDA): Examining Its Adoption & Use Cases, 2025."}, {"review_id": "9ZYFTy915k", "reviewer": "Reviewer_ffNn", "summary": "This paper proposes a data valuation framework for large-scale language models by introducing **LoGra**, a low-rank gradient projection method that significantly reduces the computational and memory overhead of influence functions. Alongside, the authors present **Logix**, a practical software toolkit that simplifies the integration of data valuation into existing training pipelines. Compared to existing methods, LoGra achieves orders-of-magnitude improvements in efficiency without sacrificing accuracy.", "questions": "The experiments are limited to models like Llama3-8B-Instruct, GPT2-XL, and Pythia-1.4B. Is it possible to evaluate the approach on larger-scale models (e.g., 30B or 65B) to further validate scalability?\n\nThe analysis of failure cases on Pythia-1.4B is insightful. However, could the authors propose concrete remedies or practical strategies to mitigate such issues in future applications?\n\nIs the proposed acceleration method hardware-agnostic? For example, can similar improvements be expected on TPUs or different GPU architectures?\n\nIf the authors are able to address the above issues, I would be willing to raise my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides numerous concrete examples to demonstrate the effectiveness of the proposed method, making the results more convincing.\n\nIt introduces a practical toolkit (Logix) to lower the barrier of adoption for data valuation in LLMs.\n\nAdditionally, the authors analyze failure cases, offering insightful guidance for future improvements.\n\nWeaknesses:\n\nThe dataset used for large-scale experiments is limited both in variety and size, as only a 1B-token subset of OpenWebText (OWT) is tested. This may affect the generalizability of the results, especially to broader domains such as multilingual or multimodal datasets.", "was_revised": true, "final_justification": "Thank the author for resolving my confusion. This work has a good starting point and relatively significant contributions. However, there are still issues such as the small size of the test dataset and model. Based on the above opinions, the final score is given as follows.", "rebuttal": "Thanks for your reviewing effort! We attempt to address questions raised by the reviewer below:\n\n---\n\n> Q1.1 The experiments are limited to models like Llama3-8B-Instruct, GPT2-XL, and Pythia-1.4B. Is it possible to evaluate the approach on larger-scale models (e.g., 30B or 65B) to further validate scalability?\n\n> Q1.2 The dataset used for large-scale experiments is limited both in variety and size, as only a 1B-token subset of OpenWebText (OWT) is tested. This may affect the generalizability of the results, especially to broader domains such as multilingual or multimodal datasets.\n\nA. While we successfully tested gradient statistics extraction and computation on models up to 13B parameters, we encountered substantial challenges when attempting experiments with larger datasets and models due to computational and storage constraints in our research environment—extracting and computing gradients requires resources comparable to actual model training. Given our focus on LLM pretraining data valuation, we conducted the most extensive experiments our resources permitted. Since computational budget can be approximated as constant x model_size x dataset_size, further scaling either dimension would have required compromising the other, making it practically infeasible within our constraints. We hope these resource limitations do not detract from the evaluation of our contributions.\n\n---\n\n> Q2. The analysis of failure cases on Pythia-1.4B is insightful. However, could the authors propose concrete remedies or practical strategies to mitigate such issues in future applications?\n\nA. In Appendix F.2, we identified outlier tokens as a primary source of failure cases—these tokens produce gradients that dominate the entire sequence gradient and consequently skew influence scores. We propose several mitigation strategies including gradient normalization, clipping techniques, and filtering heuristics to address these issues. Generally, we believe developing influence computation strategies that are robust to outlier tokens would be a promising future research topic.\n\n---\n\n> Q3. Is the proposed acceleration method hardware-agnostic? For example, can similar improvements be expected on TPUs or different GPU architectures?\n\nA. Yes, we believe our method is largely hardware-agnostic — neither LoGra nor the software design principles underlying LogIX contain any hardware-specific dependencies that would prevent compatibility with TPUs or alternative GPU architectures. Our experiments already demonstrate this flexibility to some extent, as we successfully deployed our approach across various GPU types including RTX 2080Ti, 3090, L40, A6000, and A100.\n\nThanks again and please let us know if you have any other questions."}]

Author rebuttal:

Meta-review: The reviewers were strongly positive about this submission, agreeing that it tackles an important and timely problem with both technical depth and practical relevance. The proposed LoGra method makes influence-function–based data valuation feasible at LLM scale, and the accompanying LogIX toolkit lowers the barrier to adoption in real training pipelines. Reviewers emphasized that the work is well-motivated, carefully executed, and convincingly validated through experiments at a scale that prior approaches could not reach. They also noted that the paper is clearly written and likely to attract interest from both researchers and practitioners.

Some concerns were raised around dataset choice, overstatement of speedups, and the lack of a deeper ethical discussion, but the authors addressed these points thoughtfully in the rebuttal. After the exchange, reviewers either maintained or raised their scores, and there was broad agreement that the strengths; scalability, rigor, and practical utility, clearly outweigh the limitations. Overall, this is a technically solid and impactful piece of work, and I recommend acceptance.
Decision: Accept (poster)