A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Authors: Junwei Deng
Weijing Tang
Jiaqi W. Ma
Venue: ICML 2025 poster (2025)
Keywords: influence function
data attribution
Primary area: social_aspects->accountability_transparency_and_interpretability

Abstract: Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution---quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that decompose into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to 1000 times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Junwei Deng'
Weijing Tang²
Jiaqi W. Ma
Abstract
attribution, enabling efficient cifluence-function based attribution across wide range of machine learning paradigms, with broad potential for prac- tical use cases.
Inftuence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution- quantifying how individual training data points affect a model's predictions. However, the com- mon derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent exam- ples known as M-estimators. This restricts the application of influence functions to more com- plex learning objectives, which refer as non-decomposable losses, such contrastive ranking losses, where unit loss term depends on multiple data points and cannot be decom- posed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends be- yond M-estimators. Based on this formulation, we propose novel method, the Versatile Influ- ence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF designed to fully leverage the power of diteedifferetiitoonn hereby eliminating the need for case-specific derivations of each loss func- tion. We demonstrate the effectiveness of VIF across three examples: Cox regression for sur- vival analysis, node embedding for network anal- ysis, and listwise learning-to-ran -o-rknk for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up 100 times faster compute. We believe VIF represents significant advancement in data
1. Introduction
Influence function (IF) is well-established technique orig- inating from robust statisties and has been adapted to the novel application of data altribution in modern machine learning (Koh & Liang, 2017). Data attribution aims to quan- tify the impact of individual training data points on model outputs, which enables a wide range of data-centric appli- cations such as mislabeled data detection (Koh & Liang, 2017), data selection (Xia et al., 2008), and copyright com- pensation (Deng & Ma, 2023).
Despite its broad potential, the application of IFs for data attribution has been largely limited to loss functions that can be decomposed into a sum of individual data point losses-such as those commonly used in supervised learning or maximum likelihood estimation, which are also known M-estimators. This limitation arises from the specific way that IFs are typically derived in the data attribution literature (Koh & Liang, 2017: Grosse et al., 2023), where the derivation involves perturbing the weights of individual data point losses. As a result, this restricts the application of IF-based data attribution methods to more complex machine learning objectives, such as contrastive or ranking losses, where unit loss term depends multiple data points and cannot be further decomposed into individual data point losses. We refer to such loss functions non-decomposable losses.
To address this limitation, we revisit the general formula- tion of IF in statistics literature (Huber Ronchetti, 2009), which can extend beyond M-estimators. Specifically, sta- tistical estimators are viewed as functionals of probability measures, and the IF is derived functional derivative in specific perturbation direction. In principle, this formu- lation applies to any estimator defined as the minimizer of a loss function that depends on an (empirical) probability measure, which corresponds to the learned parameters in the context of machine learning. However, directly applying
UUnivessity of Illinois Chabaaa-haampiin Carnegie Mel- lon University. Correspondence to: Jiaqi W. Ma <ji- aqima @ illlioiis.cuu>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
this general formulation modern machine learning models poses significant challenges. Firstly, deriving the precise IF for a particular loss function often requires complex, case- by-case mathetical derivations, which can be challenging for intricate loss functions and models. Secondly, for non- convex models, the (local) minimizer of the loss function not unique; as a result, the mapping from the probability measure to the learned model parameters not well-defined, making it unclear how the IF should be derived.
2020; Park et al., 2023) estimate the training data influence based the gradient and higher-order gradient information of the original model, avoiding expensive model retraining. In particular, many gradient- based methods (Koh & Liang, 2017; Guo et al., 2020; Barshan et al., 2020; Schioppa et al., 2022; Kwon et al., 2023; Pruthi et al., 2020; Park et al., 2023) can be viewed as variants of IF-based data attribution methods. Therefore, extending IF-based data attribution methods wider domains could lead to a significant impact on data attribution.
To overcome these challenges, we propose the Versa- tile Influence Function (VIF), a novel method that ex- tends IF-based data attribution to models trained with non- decomposable losses. The proposed VIF serves as an ap- proximation of the general formulation of IF but can be efficiently computed using auto-differentiation tools avail- able in modern machine learning libraries. This approach eliminates the need for case-specific derivations of each loss function. Furthermore, like existing IF-based data attribu- tion methods, VIF does not require model retraining and can be generalized to non-convex models using similar heuristic tricks (Koh & Liang, 2017; Grosse et al., 2023).
There are a few studies that adapt influence functions for graph neural networks (Chen et al., 2022; Wu et al., 2023), which can be viewed as special cases of non-decomposable losses. Chen et al. (2022) developed an influence function specifically for Simplified Graph Convolution model, which is linearized graph neural network model. Wu et al. (2023) proposed a machine unlearning method for graph neural networks based on influence function, where the influence function is adapted to consider the graph dependency among samples. In comparison to these methods, our approach has a more general formulation and can be broadly applied to various different non-decomposable losses.
We validate the effectiveness of VIF through both theoret- ical analysis and empirical experiments. In special cases like M-estimation, VIF recovers the classical IF exactly. For Cox regression, we show that VIF closely approximates the classical IF Empirically, we demonstrate the practicality of VIF across several tasks involving non-dccomposable losses: Cox regression for survival analysis, node embed- ding for network analysis, and listwise learning-to-re torrnkkkt for information retrieval. In all cases, VIF closely approximates the influence obtained from the brute-force leave-one-out retraining while significantly reducing computational time- achieving speed-ups of up to 103 times. We also provide case studies demonstrating VIF can help interpret the behav- ior of the models. By extending IF to non-decomposable losses, VIF opens new opportunities for data attribution in modern machine learning models, enabling data-centric applications across wider range of domains.
Influence Function in Statistics. The IF is well- established concept in statistics dating back at least to Ham- pel (1974), though it is typically applied for purposes other than data attribution. Originally introduced in the context of robust statistics, it was used to assess the robustness of statis- tical estimators (Huber & Ronchetti, 2009) and later adapted as a tool for developing asymptotic theories (van der Vaart, 2012). Notably, IFs have been derived for a wide range of estimators beyond M-estimators, including L-estimators, R-estimators, and others (Huber & Ronchetti, 2009; van der Vaart, 2012). Closely related to an example of this study, Reid & Crepeau (1985) developed the IF for the Cox re- gression model. However, the literature in statisties of- ten approaches the derivation of IFs through precise defini- tions specific to particular estimators, requiring case-specific derivations. In contrast, this work proposes approxima- tion for the general IF formulation in statistics, which can be straightforwarrdly applied to a broad family of modern machine learning loss functions for the purpose of data attribution. While this approach involves some degree of approximation, it benefits from being more versatile and computationally efficient, leveraging auto-differentiation ca- pabilities provided by modern machine learning libraries.
2. Related Work
Data Attribution. Data attribution methods can be roughly categorized into two groups: retraining- based and gradient- based methods (Hammoudeh & Lowd, 2024). Retraining- based methods (Ghorbani & Zou, 2019; Jia et al., 2019; Kwon & Zou, 2021; Wang & Jia, 2023; Ilyas et al., 2022) typically estimate the influence of individual training data points by repeatedly retraining models on subsets of the training dataset. While these methods have been shown effective, they are not scalable for large-scale models and applications. In contrast, gradient-based methods (Koh & Liang, 2017; Guo et al., 2020; Barshan al., 2020; Schioppa et al., 2022; Kwon et al., 2023; Ych et al., 2018; Pruthi et al.,
3. The Versatile Influence Function
3.1. Preliminaries: IF-Based Data Attribution for Decomposable Loss
We begin by reviewing the formulation of IF-based data attribution in prior literature (Koh & Liang, 2017; Schioppa
2

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
et al., 2022: Grosse al., 2023). IF-based data attribu- tion aims approximate the effect of leave-one-out (LOO)) retraining-the change of model parameters after removing one training data point and retraining the wwdcc could he used to quantify the influence of this training data point.
3.2. Non-Decomposable Loss
In practice, there are many common loss functions that are not decomposable. Below we list few examples.
Example 1: Cox's Partial Likelihood. The Cox regression model (Cox, 1972) one of the most widely used models in survival analysis, designed to analyze the time until spe- cific events occur (e.g.. patient death customer churn). A unique challenge in survival analysis is handling censored observations, where the exact event time is unknown be- cause the event has either not occurred by the end of the study or the individual is lost to follow-up. These censored data points contain partial information about the event tim- ing and should be properly modeled improve estimation. The Cox regression model is defined through specifying a hazard function over time conditional on the individual feature x:
Formally, suppose we have the following loss function',
7
CD( = Σ E(0: 24),
(1)
where 0 is the model parameters, {2 is the training dataset, and each Z(,; = =,.., 1,...,n, n, corresponds to the loss function of one training data point 24. The IF-based data attribution is derived by first inserting a binary weight wi in front of each ((:24) to represent the inclusion or removal of the individual data points, transforming (f)
h( x)) = (o) exp()'))))))))) x));
to weighted loss
where (() is a baseline hazard function and exp(0 a:) is (2) the relative risk with @ as the model parameters to be esti- mated. Given n data points {(X;, Y,, A:))== where X; represents the features for the i-th data point, Y, denotes the observed time (either the event time or the censoring time), and A; is the binary event indicator (A; 1 if the event has occurred and A; 0 if the observation is censored), the parameters 0 can be learned through minimizing the following negative log partial likelihood
Ln(0,w) = wil(0; 21).
Note that corresponds to the original loss in Eq. (1);
while removing the i-th data point is to set w; 0 or, equivalently,
1-., where 1-; is a vector of all one except for the i-th element being zero. Denote the learned parameters as (w) == arg ming (0,, The LOO effect for data point i is then characterized by (1-1)))- 0D(1).
Lcox
= (o)
Σ
0 0 log Σ exp(O x₁) JER.
However, evaluating (1-,) is computationally expensive as it requires model retraining. Koh & Liang (2017) pro- posed to approximate the LOO effect by relaxing the binary weights in w to the continuous interval [0, 1] and measuring the influence of the training data point 2; on the learned parameters as
111,=1
(4)
where R² := {j : Y2 > Y2} is called the at-risk set for the i-th data point.
In Eq. (4), each data point may appear in multiple loss terms if it belongs the at-risk sets of other data points. Consequently, we can no longer characterize the effect of removing a training data point by simply introducing the loss weight.
alp(w) Ow;
= (3CD(0((1).1)
(1)()))))
(3)
which can be evaluated using only â (1), hence eliminating the need for expensive model retraining.
Example 2: Contrastive Loss. Contrastive losses com- monly seen in unsupervised representation learning across various modalities, such as word embeddings (Mikolov et al., 2013), image representations (Chen et al., 2020), or node embeddings (Perozzi et al., 2014). Gener- ally, contrastive losses rely on set of triplets, D (uu Ui, No) where u; is an anchor data point, i; is positive data point that is relevant u4 while N, is set of negative data points that are irrelevant to u, The contrastive loss is then the summation over such triplets:
However, by construction, this approach critically relies the introduction of the loss weights wy's, and is thus limited to loss functions that are decomposable with respect to the individual training data points, taking the form of Eq. (I).
'The subscript D in LD refers to "decomposable", which is included to differentiate with the later notation.
"hilee this definition is technically valid only under specific assumptions about the loss function (e.g., strict convexity), ), in prac- tice, methods developed based on these assumptions (together with some heuristies tricks) often applicable to more complicated models such as neural networks (Koh & Liang, 2017).
Ccontrast (0) = = E(O: (u:, Us N.)),
(5)
3

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
where the loss I(+) could take many forms. In word2vee (Mikolov al., 2013) for word embeddings or
training, i.e., for i 1, ,....?,,
DeepWalk (Perozzi et al., 2014) for node embeddings, 0 corresponds to the embedding parameters for each word or node, while the loss I(.) could be defined by heirarchical softmax or negative sampling (see Rong (2014) for more details).
b; =
{
1 if the i-th object presents, otherwise.
Suppose the machine learning model parameters are denoted as 0 E Rd, a non-decomposable loss any function
L: R {0, -
Similar to Eq. (4), each single term of the contrastive loss in Eq. (5) involves multiple data points. Moreover, taking node embeddings as an example, the set of triplets D is constructed by running random walks on the network. Re- moving one data point, which is a node in this context, could also affect the proximity of other pairs nodes and hence the construction of D.
that maps given model parameters @ and the object presence vector to a loss value [( b).
arg ming [( b) on any non- decomposable loss L( b), the LOO effect of data point on the learned parameters can still be properly defined by
Denoting ((b)
Example 3: Listwise Learning-to-Rante Learning-to soaminggto-rank to-rank is a core technology underlying information retrieval appli- cations such as search and recommendation. In this con- text, listwise learning-to to-kank methods aim to optimize the ordering of a set of documents or items based on their rel- evance given query. One prominent example of such methods is ListMLE (Xia et al., 2008). Suppose we have annotated results for m queries over items as a dataset N where æy is the query fea- E [n] := {l...., n indicate the top items for query i. Then the ListMLE loss function is
0(1-))-000
However, in this case, we can no longer use the partial derivative with respect to by to approximate the LO0 effect, as 0(b) is only well-defined for binary vectors b. Remark 3.2 ("Non- "NNo--Decopposabee" v.s. "Not Decompos- able"). The class of non non-decomposable loss Definition 3.1 includes the decomposable loss in Eq. (1) special case when [(0 := End,= (0).. Throughout this paper, we will call loss functions that cannot be written in the form of Eq. (1) as "not decomposable". We name the general class of loss functions Definition 3.1 as non-decomposable loss to highlight that they are generally not decomposable.
{arrs (v/² 3/ (2)
(k)
yi
ture, yi 3/; .....
defined as following
(LIR(0)= = Σ>
>1->>nis (f(#;0),
m
Remark 3.3 (Randomness in Losses). Strictly speaking, many contrastive losses are not deterministic functions of training data points as there is randomness in the construc- tion of the triplet set D, due to procedures such as negative sampling or random walk. However, our method derived for the deterministic non-decomposable loss still gives mean- ingful results in practice for losses with randomness.
==1
-log
exp(f(z;
where f(:: is a model parameterized by 9 that takes the query feature as input and outputs n logits for predicting the relevance of the items.
3.3. The Statistical Perspective of Inffuence Function
The Statistical Formulation of IF. To derive IF-based data attribution for non-decomposable losses, revisit general formulation of IF in robust statistics (Huber & Ronchetti, 2009). Let Ω be a sample space, and T(-) is function that maps from a probability measure on Ω2 to a vector in Rd Let P and Q be two probability measures on Q. The IF of T-)) at in the direction Q measures the infinitesimal change towards a specific perturbation direction Q, which defined
In this example, Eq. (6) is decomposable with respect to the queries while not decomposable with respect to the items. The influence of items could also be of interest in informa- tion retrieval applications. For example, in a search engine, we may want to detect webpages with malicious search engine optimization (Invernizzi et al., 2012); in product co- purchasing recommendation (Zhao et al., 2017), both the queries and items are products.
General Loss Formulation. The examples above can be viewed as special cases of the following formal definition of non-decoompoabbee loss.
IF(T()): = lim T((1 - (( T(P)
In the context of machine learning, the learned model pa- rameters, denoted as ((P), can be viewed as function of the data distribution P. Specifically, the parameters of the learned model are typically obtained by minimizing a loss
Definition 3.1 (Non-Decte Decompocooesllee Loss). Given n objects of interest within the training data, let a binary vector E {0, 1) indicate the presence of the individual objects in
can

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
function, i.e., O(P) arg ming [(0, P). Here, [(,, P) is loss function that depends a probability measure P, distinguishing it from the non-decomposable loss b) that depends on the object presence vector b.
loss in Eq. (3) and Koh & Liang (2017) mitigate this prob- lem through heuristic tricks specifically designed for Eq. (3). However, the IF in Eq. (7) is in general more complicated for non-decomposable losses and generalizing it to modern setups like neural networks remains unclear.
Assuming the loss is strictly convex and wwice-differentabhe with respect to the parameters, the learned parameters ((P) are then implicitly determined by the following equation
3.4. VIF as A Finite Difference Approximation
We now derive the proposed VIF method by applying Eq. (7) to the non-decomposable loss while addressing the afore- mentioned challenges through a fnite-differeen approxima- tion.
P) = O.
Moreover, the IF of ((P) with perturbation towards Q is given by²
Definition 3.5 (Finit--Differenc IF), Define the finite- difference IF as following
(O()(P):Q) -- P2((0(P))))] -
IF (.()) = - [HC(((P)) P)] P)]_
VoC(O(P), - s-+
- V)(()(P).P) (7)
lim
(1-)) - VoL(O(P). P)
(8)
The advantage of the IF formulation in Eq. (7) is that it can be applied to more general loss functions by properly specifying P. Q, and L.
which approximates the IF in Eq. (7), (F()))): by re- placing the limit with a finite difference.
Example: Application of Eq. (7) to M-Estimators. As an example, the following Lemma 3.4 states that the IF in Eq. (3) for decomposable loss can be viewed as special case of the formulation in Eq. (7). This is a well-known re- sult for M-estimators in robust statistics (Huber& & Ronchetti, 2009), and the proof of which can be found in Appendix A.2. Intuitively, with the choice of P, Q, and [ in Lemma 3.4, e) =Q= = (1 - P,,, + = leads to the effect of upweighting the loss weight of 2; with a small perturbation, which is essentially how the IF in Eq. (3) is derived.
Observation on M-Estimators. The proposed VIF method for general non decompsooeoelee losses is motivated by the following observation in the special case for M-estimators.
Theorem 3.6 (Finite-Difference IF for M-Fstimators). Un- der the specification of P Pa, = òzc, and L Ez-p [e(0; z)] in Lemma 3.4, the IF is identical to the finite- difference IF with E = 1- i.e.
IF(((P)); = IF (0(PP));02))
Lemma 3.4 (IF for M-Estimators). Eq. (7) reduces to Eq. (3) up to constant when we specify that 1) P is the empirical distribution Pn, = Σm= 5z,/n, where 82, is the
Furthermore, denote Q the empirical distribution
where Pr(z,) = 0 and Pr(2,) = = # i. Then we have
Dirac measure, i.e., Pr(z;)= = 1 and Pr(zz)= = 0,j # i; 2)
Q = dz.jann 3) [(., P)= = Ez-p (((0:))) Specifically,
(1+
n 1 p 1
5z. = =01-)
n-
IF 1 (((P.); (1-1)IF 1
IF (((((P)))...) - [VL (1((0D(1)2))
The first part of Theorem 3.6 suggests that, for M-estimators, the limit in (F(((P))) oz,) can be exactly replaced by finite difference with proper choice of s. The second part of The- orem 3.6 further shows that we can construct another finite- difference IF, IF, (8(P... ()=,) with a different choice of Q Q/-2 and ε = 1, that differs from IF (Ō(P,,. only by a constant factor. For the purpose of data attribution, we typically only care about the relative influence among the training data points, so the constant factor does not matter.
Challenges of Applying Eq. (7) in Modern Machine Learning. While the IF in Eq. (7) is a principled and well- established notion statistics, there are two unique chal- lenges when applying to modern machine learning models for general non-dccomposable losses. Firstly, solving the limit in the right hand side of Eq. (7) requires case-by-case derivation for different loss functions and models, which can be complicated (see an example of IF for the Cox regres- sion (Reid & Crepeau, 1985) in Appendix A.5). Secondly, the mapping 0( P), hence the limit, are not well-defined for non-convex loss functions as the (local) minimizer is not unique. A similar problem exists the IF for decomposable
Generalization to General Nec-Decomposable Losses. The benefit of having the form iF (Ō(P.): x is that it is straightforward to generalize this formula from M- estimators to general non-decomposable losses. Specifically,
"See Appendix A.1 for the derivation.
5

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
noticing that Pn, and Q In-1 are respectively empirical dis- tribution on the full dataset and the dataset without 2,, we can apply this mnieediifferenc IF to any non-eecomposable loss through an appropriate definition of L.
2023). We have included the results of efficient approxi- mate implementations of VIF based on Conjugate Gradient (CG) and LiSSA (Agarwal et al., 2017; Koh & Liang, 2017) in Appendix C. Finally, note that VIF calculates the differ- ence (1), 1) - L(((1). 1- before taking the gradient with respect to the parameters. In some special cases (see, e.g., the decomposable loss case Section 3.5), taking the difference before the gradient significantly simplifies the computation the loss terms not involving the i-th data object will cancel out.
Definition 3.7 (C(0, P) for Non- Non-Decomposable Loss). Let P(n) be the set of uniform distributions supported on sub- sets of fixed points { { Note that both of the empiri- cal distributions P,, and Q!-i belong to the set P(n). For any P E (mn). denote bf E 10. as binary vector such that 6." (2()) 0] 1,...,77 n. The C(0, P) for non-decomposable loss L can be defined as follows:
Attributing a Target Function. In practice, we are often in- terested in attributing certain model outputs or performance. Similar to Koh & Liang (2017), given a target function of interest, [2z 9), that depends on both some data and the model parameter 0, then the influence of a training data point i on this target function can be obtained through the chain rule:
[ P) = C(0,b")
Proposition 3.8
IF on Non-Decompee Decompo---s----
able Loss). Under Definition 3.7, we have
Th (0(P)) ( [V
(11)
Vo
(9)
3.5. Approximation Quality in Special Cases
The Proposed VIF. We propose the following method to approximate the LOO effect for any non-decomposable loss. Definition 3.9 (Versatile Influence Function). The Versatile Influence Function (VIF) that measures the influence of a data object i on the parameters ((1) learned from a non- decomposable loss L is defined as following
To provide insights into how accurately the proposed VIF approximates Eq. (7), we examine the following special cases. Although there is universal guarantee of the ap- proximation quality for all con-decomposable losses, our analysis in these cases suggests that VIF may perform well in many practical applications.
M- M-Estimation (Decomposable Loss). For a decomposable loss, we have VoLp(Ob(1), = Σ (1(())(1) and VeLp (1), 1_;) = -j V+l(00(1) In this case, it is straightforward to see that
VIF(O(1):)) = 1 1
Vo (C(0(1), -((((), (10)
(IF()());)) = -n
The proposed VIF is variant of Eq. (9), as it can be easily
shown that
which indicates that the VIF here is identical to the IF in Lemma 3.4 without approximation error.
VIF(0(1): = = ( Q.E.)
Cox Regression. The close-form of the IF for the Cox re- gression model, obtained by directly solving the limit in Eq. (7) under the Cox regression model, exists in the statis- tics literature (Reid& & Crepeau, 1985), which allows to characterize the approximation error of the VIF in compari- son to the exact solution.
The inclusion of the additional constant factor is motivated by Theorem 3.6 to make it better align with the original IF in Eq. (7). In practice, this definition is also typically more numerically stable the Hessian is normalized by n.
Computational Advantages. The VIF defined in Eq. (10) enjoys a few computational advantages. Firstly, VIF de- pends the parameters only at 0(1 and does not require 0(1-)). Therefore, does not require model retraining. Secondly, compared to Eq. (7), VIF only involves gradients and the Hessian of the loss, which can be easily obtained through auto-differentiation provided in modern machine learning libraries. Thirdly, VIF can be applied to more com- plicated models and accelerated with similar heuristic tricks employed by existing based data attribution methods for decomposable losses (Koh & Liang, 2017; Grosse et al.,
Theorem 3.10 (Approximation Error under Cox Regression; Informal). Denote the exact solution by Reid & Crepeau (1985) as IF IF (0(1;;)) while the application of VIF on Cox regression as VIF (lFcc((((1)))) Their difference bounded as following:
VIF -Fc.
Theorem 3.10 suggests that the approximation error of the VIF vanishes when the training data size is large. A formal
6

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
statement of this result and its proof can be found in Ap- pendix A.5, and empirical verification of the theoretical results can be found in Appendix A.
4.2. Performance
Table I. The Pearson correlation coefficients of VIF and brute- force LOO retraining under different experimental settings. Specif- ically, "Brute-Force" refers to the results of two times of brute- force LOO retraining using different random seeds, which serves as reference upper limit of performance.
4. Experiments
4.1. Experimental setup
Sceuario
Dataset
Method
Pearson Correlation
We conduct experiments on three examples listed in Sec- tion 3.2: Cox Regression, Node Embedding, and Listwise Leaming-to-Rank, In this section, we present the perfor- mance and runtime of VIF compared to brute-force LOO retraining. We also provide two case studies to demonstrate how the influence estimated by VIF can help interpret the behavior of the trained model.
VIF
0.997
METABRIC
Cox Regression
Brute-Force
0.997
SUPPORT
VII
0.943
Force
0.955
VIF
0.407
Node Embodding
Karate
Brute-Foree
0.419
VIF
0.823
Mediamill
Datasets and Models. We evaluate our approach on mul- tiple datasets across different scenarios. For Cox Regres- sion, we use the METABRIC and SUPPORT datasets (Katz- man et al., 2018). For both of the datasets, we train a Cox model using the negative log partial likelihood following Eq. (4). For Node Embedding, we use Zachary's Karate network (Zachary, 1977) and train a DeepWalk model (Per- ozzi et al., 2014). Specifically, we train a two-layer model with one embedding layer and one linear layer optimized via contrastive loss following Eq. (5), where the loss is defined as the negative log softmax. For Listwise Learning-to- to-Rank, we use the Delicious (Tsoumakas et al., 2008) and Medi- amill (Snoek et al., 2006) datasets. We train a linear model using the loss defined in Eq. (6). Please refer to Appendix B for more detailed experiment settings.
Listwise Leamiiggto-Rank
Brute-Foree
0.999
VIF
0,906
Delicious
Brute-Force
0.999
We utilize the Pearson correlation coefficient to quantita- tively evaluate how closely the influence estimated by VIF aligns with the results obtained by brute-force LOO retrain- ing. Furthermore, a reference upper limit of performance, we evaluate the correlation between two brute-force LOO retraining with different random seeds. As noted in Re- mark 3.3, some examples like contrastive losses are not deterministic, which could impact the observed correlations.
Table presents the Pearson correlation coefficients com- paring VIF with brute-force LOO retraining using different random seeds. The performance of VIF matches the brute- force LOO in all experimental settings. Except for the Node Embedding scenario, the Pearson correlation coefficients are close to 1, indicating a strong resemblance between the VIF estimates and the retraining results. In the Node Embed- ding scenario, the correlations are moderately high for both methods due to the inherent randomness in the random walk procedure for constructing the triplet set in the Welp a algorithm. Nevertheless, VIF achieves correlation that is close to the upper limit by brute-force LOO retraining. In Figure 1, we show that the influences of each training sam- ple predicted by VIF align almost perfectly with the exact test loss differences after LOO retraining that respectively removes the individual training samples. Additional experi- ment results on larger datasets and models are presented in Appendix B and C.
Target Functions. We apply VIF to estimate the change of a target function, f=2 f). before and after a specific data object excluded from the model training process. Below are our choice of target functions for difference scenarios.
For Cox Regression, we study how the relative risk func- tion, f(Iteat, 0) = exp(@T Ztest). of a test object, Iteats would change if one training object were removed.
For Node Embedding, we study how the contrastive loss, f((u, v, 6)) = I(0; (u, v. N)), of an arbitrary pair of test nodes, (u, v), would change if a node w E N were removed from the graph.
For Listwise Learning-to-Rant we study how the ListMLE loss of test query, f((Etest, Vhl 0)
4.3. Runtime
E} (f(rtett)
We report the runtime of VIF and brute-force LOO retrain- ing in Tabel 2. The computational advantage of VIF is significant, reducing the runtime by factors up to 1097x. This advantage becomes more pronounced as the dataset size increases. The improvement ratio on the Karate dataset is moderate due to the overhead from the random walk
0))), would change if one item E [n] were removed from the training process.
log Z1E()) (ult) (i
"The code is publicly available at https: http://ggttt////// com/ TRA TS Lab/Versat 1e- w-w.IIflueeeen Function.
7

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
samppe
sample
Tost sample
0.0
108-
B 6:30
Table 3. The top-5 influential training samples 2 test samples in the METABRIC dataset. "Features Similarity" is the cosine similarity between the feature of the influential training sample and the test sample. "Observed Time" and "Event Occurred" are the Y and A of the influential training sample as defined in Eq. (4),
130+
8
g
3
VF
VF Test-somele
VF
T:25 sample 6
Tett samplc: 6
Sepe 1:31
Sape E6
Shoe: 1.0
Feature Similarity
Observed Time
Event Occured
Influence Rank
8
000
Test Sample
1
0.84
322.83
False
4004
1.2
9.00
20M 18. 027 300 SM
-20-0X SA 1E1 Co. co KM
2
-0.34
9.13
True
3
0.77
258.17
True
Figure 1. The influences predicted by VIF versus the exact loss dif- ferences after LO0 retraining on 6 randomly selected test samples. The experiment is done with Cox Regression on the METABRIC dataset. Each sub-figure corresponds one sample. The x- axis indicates the influence of training sample on the test sample, while the y-axis indicates the change of loss on the test sample after LOO retraining by leaving that training sample out.
0.23
131.27
False
0.81
183.43
False
Test Sample
-0.49
16.57
True
2
-0.22
30.97
True
3
-0.39
15.07
True
-0.65
4.43
True
0.72
307.63
False
process and potential optimizations in the implementation. All runtime measurements were recorded using an Intel(R) Xeon(R) Gold 6338 CPU.
influential nodes (on the top right in Figure 2b and 2c) are the hub nodes that lie on the shortest path of the pair of test nodes. For example, the shortest path from node 12 to node 10 passes through node O, while the shortest path from node 15 to node 13 passes through node 33. Conversely, the nodes with the most negative influence (on the bottom left in Figure 2b and 2c) are those that likely "distract" the random walk away from the test node pairs. For instance, node distracts the walk from node 12 to node 10, and node 30 distracts the walk from node 15 to node 13.
Table 2. Runtime comparison of VIF and frute-force LOO retrain-
ing.
Senario
Duisset
Brute Fauue-Frrce
VIF
Improvement Ratio
Cox Regression
METARRIC
min
2.43
593x
SUPPORT
225 min
12.3 ses
1097x
Nerwork Emibedding
Kanae
204 min
100 min
1.87x
Liswise Leaming- Rank
Mediamill
52 min
2.6 min
20x
Delicions
660 min
288 min
236x
4.4. Case Studies
We present two case studies to show how the influence estimated by VIF can help interpret the behavior of the trained model.
5. Conclusion
In this work, we introduced the Versatile Influence Function (VIF), a novel method that extends IF-based data attribu- tion to models trained with non-decomposable losses. The key idea behind VIF is a finite difference approximation of the general IF formulation in the statistics literature, which eliminates the need for case-specific derivations and can be efficiently computed with the auto-differentiation tools pro- vided in modern machine learning libraries. Our theoretical analysis demonstrates that VIF accurately recovers classical influence functions in the case of M-estimators and provides strong approximations for more complex settings such as Cox regression. Empirical evaluations across various tasks show that VIF closely approximates the influence obtained by brute-force leave-one-out retraining while being orders- of oa--iaanitude faster. By broadening the scope of IF-ba data attribution to non-decomposable losses, VIF opens new avenues for data-centric applications in machine learning, empowering practitioners to explore data attribution in more complex and diverse domains.
Case study 1: Cox Regression. In Table 3, we show the top-5 most influential training samples, estimated by VIF, for the relative risk function of two randomly selected test samples. We observe that removing two types of data samples in training will significantly increase the relative risk function of test sample, which leads to a shorter event time: (I) training samples that share similar features with the test sample and have long event times (e.g., training sample ranks 1, 3, 4, 5 for test sample 0 and ranks 5 for test sample 1) and (2) training samples that differ in features from the test sample and have short event times (e.g., training sample ranks 2 for test sample 0 and ranks 1, 2, 3, 4 for test sample 1). These findings align with domain knowledge.
Case study 2: Node Embedding, in Figure 2b and 2c, we show the influence of all nodes to the contrastive loss of 2 pairs of test nodes. The spring layout of the Karate dataset provided in Figure 2a. We observe that the most
8

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Karate Club Graph
those of the authors and do not necessarily reflect the official policies or positions of the supporting agency.
References
50
Agarwal, N., Bullins, B., and Hazan, R. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116):1- 40, 2017. URL /ttttp://jmr... http://mm....ppppa/////s/// http//ppprr///1// 16- 091-49...00 html.
Barshan, E., Brunet, M.-.,, and Dziugaite, G. K. Relatif: Identifying explanatory training samples via relative in- fluence. In International Conference on Artificial Intelli- gence and Statistics, pp. 1899-1909, PMLR, 2020.
(a) Karate Club Graph
Contrastive
Contractive loss
Node 12 and Nodo 10.
Nodo and Node 13 Pearson correlation 0.763
Pearson correaation 0.934
Achurai det Contractee lons
Athura derf Contrection
6.3-
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. Simple Framework for Contrastive Learning of Visual Representations. In Ii,, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference chine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
0.1
Ma-
6.2
2.0
02
CA 0.5
0.2
0.0
Influence esdimated VIF
Influence axtimatedd by
(b) Node 12 and Node 10
(c) Node 15 and Node 13
Chen, Z., Li, P., Liu, H., and Hong, P. Characteriz- ing the influence of graph elements. arXiv preprint arx14:2210.07441, 2022.
Figure 2. VIF is applied to Zachary's 's Karate network estimate the influence of each node on the contrastive loss of a pair of test spring layout of the Karate network. Fig- ure 2b and Figure 2c illustrate the alignment between the influence estimated by VIF (x-axis) and the brute-force LOO retrained loss difference (x-axis).
nodes. Figure
Choe, S. K., Ahn, H., Bac, J., Zhao, K., Kang, M., Chung, Y., Pratapa, A., Neiswanger, W.. Strubell, E., Mitamura, T., et al. What is your data worth to gpt? Ilm-scale data valuation with influence functions. arXiv preprint arXiv: 13954, 2024.
Limitation and Future Work. Similar to early IF-based methods for decomposable loss (Koh Liang, 2017), the formal derivation of VIF assumes convexity of the loss func- tion, which requires practical tricks to adapt the proposed method to large-scale neural network models. While we have explored the application of Conjugate Gradient and LiSSA (Agarwal et al., 2017) for efficient inverse Hessian approximation (see Appendix C), more advanced techniques to stabilize and accelcrate based methods developed for decomposable losses, such as EK-FAC (Grosse et al., 2023), ensemble (Park et al., 2023), or gradient projection (Choe al., 2024), may be adapted to further enhance the practical applicability of VIF on large-scale models.
Cox, D. R. Regression models and life-tables. Jour- nal of the Royal Statistical Society. Series B, Statisti- cal methodology, 14(2):187-202 January 1972. ISSN 1369-7412,1467-9966 doi: 10.1111//2117-6661..77 tb00899.x.
Cox, D. R. Partial likelihood. Biometrika, 62(2):269-276. 1975.
Deng, and Ma, J. Computational Copyright: Towards Royalty Model for Music Generative AI. arXiv [cs.Al], December 2023.
Impact Statement
Ghorbani, A. and Zou, J. Data shapley: Equitable valuation of data for machine learning. In Chaud- huri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pp. 2242-2251. PMLR, 09-15 Jun 2019. URL https: http::pss///ppceedeeseem mlr.press/v97/ ghorbanil9c.hm html.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgements
WT was partially supported by NSF views and conclusions expressed in this paper are solely
The
Grosse, R., Bac, J., Anil, C., Elhage, N., Tamkin, A., Taj- dini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hub-

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
inger, E.,, Lukošiūté, K., Nguyen, K., Joseph, N., McCan- dlish, S., Kaplan, J., and Bowman, S. R. Studying large language model generalization with influence functions. arXiv [cs.LG], August 2023.
Kwon, Y. and Zou, J. Beta shapley: unified and noise- reduced data valuation framework for machine learning. arXiv preprint arXiv:2110. 14049, 2021.
Kwon, Y., Wu, E., Wu, K., and Zou, J. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. arXiv preprint arXiv:2310.00000
Guo, H., Rajani, N. F., Hase, P., Bansal, M., and Xiong, C. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint wiXi:22012.15781, 2020.
2023.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. Neural information process- ing systems, 2013.
Hammoudeh, Z. and Lowd, D. Training data influence analysis and estimation: A survey. Machine Learning,
11365):331122403 2024.
Park, S. M., Georgiev, K., Ilyas, A.. Leclere, G., and Madry, Trak: Atrributing model behavior at scale. arXiv preprint arXiv:2303.14188 2023.
Hampel, F. R. The influence curve and its role in robust es- timation. Journal of the American Statistical Association,
69(346):383-392 June 1974. ISSN 0162-1459, 1537- 274X. doi: 10.1080/0162145) 1974. 19482992.....
Perozzi, B., AI-Rfou, k., and Skiena, DeepWalk: online
learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pp. 701-710, New York, NY, USA, August 2014. ACM. ISBN 9781450329569.
Huber, P. J. and Ronchetti, E. M. Robust Statis- tics. Wiley Series in Probability and Statistics. Wiley- Blackwell, Hoboken, NJ. 2 edition, January 2009. ISBN 97804701 2990009999999997444444000 9780470434697.
doi: 10. 10.1455222339222237333
doi: 10.1002/
Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimat- ing training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Ilyas, A., Park, S. M., Engstrom, L., Leclerc, G., and Madry, A. Datamodels: Predicting predictions from training data. arXiv preprint wwXiv:2202.00622, 2022.
Reid, N. and Crepeau, H. Influence functions for propor- tional hazards regression. Biometrika, 72(1):1, April 1985. ISSN 0006-34444.1444-5510 2336329.
Invemnzzzi L., Comparetti, P. M., Benvenuti, S., Kruegel, C., Cova, M., and Vigna, G. Evilseed: A guided approach to finding malicious web pages. In 2012 IEEE symposium on Security and Privacy, pp. 328-442. IEEE, 2012.
doi: 10.2307/
Rong, X. word2vec Parameter Learning Explained. arXiv [cs. CLJ, November 2014.
Jia, R., Dao, D., Wang, B., Hubis, F. A., Hynes, N., Gürel, N. M., Li, B.. Zhang, C., Song, D., and Spanos, C. J. To- wards efficient data valuation based on the shapley value. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1167-1176. PMLR, 16- 18 Apr 2019. URL https: /proceee/pccccccppooooooo mlr. press v jial9a.
Schioppa, A., Zablotskaia, P., Vilar, D., and Sokolov, A. Scaling up influence functions. In Proc. Conf. AAAI Artif. Intell., volume 36, pp. 8179-8188 Association for the Advancement of Artificial Intelligence (AAAI), June 2022. doi:
Snock, C. G., Worring, M., Van Gemert, J. C., Geusebroek, J.-M., and Smeulders, A. W. The challenge problem for automated detection of 101 semantic concepts in multi- media. In Proceedings of the 14th ACM international conference Multimedia, pp. 421 430, 2006.
Katzman, J. I.., Shaham, U., Cloninger, A., Bates, J., Jiang, T., and Kluger, Y. Deepsurv: personalized treatment rec- ommender system using a cox proportional hazards deep neural network. BMC medical research methodology. 18: 1-12, 2018
Tsoumakas, G., Katakis, I,, and Vlahavas, I. Effective and efficient multilabel classification in domains with large number of labels. In Proc. ECMI/PKDD 2008 Workshop on Mining Multidimensional Data (MMD'08) volume 21, pp. 53-59, 2008
Koh, P. W. and Liang, P. Understanding Black-box Pre- dictions via Influence Functions. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2017.
van der Vaart, A. W. Asymptotic Statistics. Cam- bridge University Press, Cambridge, England, June 2012. ISBN 978051 1802256,98805219960333 doi: 10.1017/ cbo978051 1802256.
PMLR,
10

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Wang, J. T. and Jia, R. Data banzhaf: A robust data valu- ation framework for machine learning. In International Conference Artificial Intelligence and Statistics, pp. PMLR, 2023.
Wu, J., Yang, Y., Qian, Y., Sui, Y., Wang, X., and He, X. Gif: general graph unlearning strategy via influence function. In Proceedings of the ACM Web Conference 2023, pp. 651-661, 2023.
Xia, Liu, T.-Y., Wang. J., Zhang, W., and Li, H. List- wise approach to learning to rank: theory and algo- rithm. In Proceedings of the 25th international confer- ence on Machine learning ICML '08, pp. 1192-1199 New York, New York, USA, 2008. ACM Press. ISBN 9781605582054. doi: 10.1 19.114///301155.33900000
Yeh, C.-K., Kim, J., Yen, I. E.-H., and Ravikumar, P. K. Representer point selection for explaining deep neural networks, Advances in neural information processing systems, 31, 2018.
Zachary, W. W. An information flow model for conflict and fission in small groups. Journal of anthropological research, 33(4):452-473 1977.
Zhao, T., McAuley, J., Li, M., and King, I. Improving recommendation accuracy using networks of substitutable and complementary products. In 2017 International Joint Conference on Neural Networks (LJNN). pp. 3649-3655. IEEE, 2017.

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
A. Omitted Derivations
A.1. Derivation Eq. (7)
Consider an E perturbation towards another distribution Q, i.c., (1-E) + =Q. Note that 0((1 - E) + + solves VoČ(O, (1- e)
=Q) 0. We take derivative with respect to E and evaluate
both side, which leads
8((1 ([1-=))P - ((P) lim 6
VoL(O(P). (1 =) +Q) VoC(((P))) 0.
VaL(((P), P) lim 6-0
&
c
Given the strict convexity, the Hessian is invertible at the global optimal. By plugging the definition of IF, we have
VoL(O(P),P)
IF(((P):Q) P))
lim
A.2. Proof of Lemma 3.4
Proof. Under M-estimation, the objective function becomes the empirical loss, i.e., Ż(0, P) = [Ezrr(((;=2) z)]. where
P= nn, = TM /n is the empirical distribution over the dataset. Note that [(0, 1 CD (0, 1) for any 0, therefore they share the same minimizer, i.e.,
((P) = 61
The gradient and Hessian of L(((P), P) are respectively
VoL(O(P), P) = - = 1
V [o)( =0
and
V/C(((P)) P)
V Σ V2((0(P):z)/n1
n (1)))
The infinitesimal change on the gradient towards the distribution Q = 821 equals to
-
(1- - V)(()()),P)
lim
e00
lim
c-0
(1 Bza V (ol( =)] + +e
lim
c-0
(1-8). 0 E. V (e(((()))
lim
c-0
= Vol()))
= =V((((()), = vgl(0b(1),2).
Plugging the above equations into Eq. (7), becomes
IF(((P.): (VCC0(001)..1 (1)(())1)))))
A.3. Proof of Theorem 3.6
Lemma A.1. Let Pn and Cy (-1) be the empirical distributions respectively on (2g1ÿ_1 and {zg1ī=1 f2ih, while 6z, is the distribution concentrated on 2;. Then
(1 +
Pr
82. ((-1)
71
11
12

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Proof of Lemma A.I. For any j + i,
(1 + 1
1
1 0
(2
Se (z)) (1 +
1
-x))
For 1,
1
1
(1 + (z))
82, (2s) (1
71
-0- (z))
Proof of Theorem 3.6. We first prove the first part of Theorem 3.6, where our goal is show
(i(P.)::2)) n [1 V=((0)(1):2s)
Expanding (O(P::: 82.) by its definition Eq. (8),
(1 (1- 1 5622) V.
IF (P(P.)))) (3C()(P,).
Setting 8 =-
n-T and by Lemma A.I,
VoL(O(P,)) 2 V ((0(PP)),)
(((p)) = V3L(0(P.)..) P..)
(12)
-1/nn-1
E En 9 (o((O((p)) - EEMFE (ol(((PP))))
V3C(O(P.).P.)
-1/(n-1 -1)
Ej Vol(O((.):))) -n) -1) vo((((Pw):zg///n
= V.L(O(P.).P.))
-1/(n-1)
Vel(((P.):zz) + n-1 n
V3Z(((P.).PP))
Ve((O(P)):2)) (13)
n
Noting that ((P..) is the optimizer for P(),
0 V.C(
V (P((
Therefore,
vol(((P.):z)) v
j=1.jii
Plugging the two equations above into Eq. (13), we have
iF 1 (((P.): [V
13

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
From the proof of Lemma 3.4 in Appendix A.2, know
((Pr.) = (I)) V. (1
Therefore,
(0(P.): = ( [
(1)(0
which completes the proof for the first part of Theorem 3.6. For the second part, the first equation has been proved as Lemma A.1. The second equation is straightforward from Eq. (12):
VoL(0(P))) (())) - -VC(((P)).))))
IF T (((P)):
VeL(O(P,).P..)
-1/(n-
-n 1) - (3C(0(P)))P.))
VoL(O(PP)) (CC(((P)).011))) VoL(((P.).P))
(n 1)IF (!1.)
A.4. Proof of Proposition 3.8 Proof. It is easy verify that
b²= 1, - =1 1-.
Hence, based on the definition of E in Proposition 3.8, we have
(,0 = 1), 2(0, (0,0 C(O, 1.)..
Therefore, we also have ((P,)) = ((1). The result in Eq. (9) follows directly by plugging these quantities into the definition of IF (0(P.)) I(-)).
A.5. Formal Statement and Proof of Theorem 3.10
Setup and Notation. For convenience, adopt set of slightly different notations tailored for the Cox regression model.
Consider n i..d. generated right-censoring data {Z, (X,, Y, A,) where Y, min{T, C.) } is the observed time, T; is the time to event of interest, and C is the censoring time. We assume non-informative censoring, i.e., T and C are independent conditional on X, which is a common assumption in the literature. Suppose there are no tied events for simplicity.
well-known estimate for the coefficients 3 under the Cox model is obtained by minimizing the negative log-partial likelihood:
(l) =
Σ
A
T X, - log
Σ exp (0T x,) (oTx,)
i=1
jER,
a Σ
(0Tx,) xi) ))
A
0 X; - log
1(Y, 2 Yi) Yx)
Note that (0) is convex function and the estimate ô equivalently solves the following score equation:
Si 0,
VoL, (0)
X;
S (Y::
(0:20))
14

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
where
(1:0) 1
Σ
T(; 2 exp (0Tx.).
(14)
==1
1
(t:00 = n Σ 2) exp exp (0Tx.) X,) X,
(15)
It has been shown that, under some regularity conditions, i is consistent estimator for 8*. Note that the above score equation is not a simple estimation equation that takes the summation i.idd terms, because S (t: 0) and S (t; 0) depend on all observations.
Analytical Form of Influence Function in Statistics. Reid & Crepeau (1985) derived the influence function by evaluating the limit in (7) with being the underlying data-generating distribution and Q = 8z, (i.e., the Gateaux derivative at fr = (P) in the direction dz To start with, we define the population counterparts of Eq. (14) and Eq. (15):
(0) (t: 0) E (T 2 1) exp (0" X)),
(0)) (t: 0) = (T 2 t) exp (0 x),
and introduce the counting process notation: the counting process associated with i-th data NN(()) = [(,, S A, = 1), the process G, (t) n "E (t), and its population expectation G(t) EtG, Then the influence function for the
observation Z4 (X,, Y1, A,) is given by
A (1) =A; (x,- (Y::0))))))
s(;(();;)*)
- exp Xi) /
I(Y, 2() 2t)
(t:07)
(x.)
dG(t)
((0) 0-)
(10)
where A is the non-singular information matrix. A consistent estimate for A is given by (0) (Cox, 1975). The empirical influence function given n data points is obtained by substituting A, 0", and G(t) by V/C(0)/n 0, and (t) respectively:
(1) -,
where
1
I(Y t)
(011 (t; 0)
Ci(0) exp(0 xx))
Xi
dt,(t)
c(00
(1:0)
S (1; 0)
j=1
( (x,- 0 (Y,:
I(Y; Y,)4,
= exxp(0 Xi)
$ (Y;;
S. (Y;;
The first term is analogous to the standard influence function for M-estimators and the second term captures the influence of the i-th observation in the at-risk set.
The Proposed VIF. Under the Cox regression, the proposed VIF becomes
VIF ;= - - (VV.C.
(V) [Vac, -v
where V is the gradient of the negative log-partial likelihood after excluding the i-th data point at Ô. Given no tied events, we can rewrite voc(-? as
Voc - Σ A, Xj 5(0) (Y;; exp(0 X,)/n )-
(111) (Yz:0) exp(êT X.)
(V;
Σ A, X,
((0) (Y;:
J-Y
15

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Then it follows that
(o) - VIF - (0)mm())
( (Y;:
Sh (Y;; 0) exp(0 x,)X(/n)
=- [V)m((0)/)]
(oln(0: 24) +
Σ A,
Sn (Y;:
Sh (Y;; 0) exp(0 x:)/n
=- (0)mm) (Vo]))))) 'oln((; 2i)
- (0/mm() (xxp(0 X.)
T(Y < V;)A; j=1 Sn (Yg:0) exp(0 (0T x,)/n
n
5 (Y:00) X; -,- S 5 (Yz: 0)
Σ
00)
Approximation Error. Below, we formally bound the difference hetween the analytical form of IF and our proposed approximation. Our result implies that the difference between the analytic expression of the IF and the proposed VIF approximation, i.e., VIF,, nn (i) IF, F, (i), diminishes at rate of 1/n as the sample size grows and is of smaller order than (i). This is because IF, vs (i) (t) + op (1), where (i)) is non-degenerate random variable that doesn't 't converge zero in probability; therefore IF, (i) remains bounded away from zero in probability, denoted as (1)
Theorem A.2 (Approximation Error Bound under Cox Model), Assume that (1) the true parameter 6 is an interior point of a compact set B C R (2) the density of X is bounded below by a constant c > O over its domain X, which is a compact
subset of : (3) there is a truncation time т < oo such that for some constant do, Pr(Y > =|X)= 2 g0 almost surely; (4) the information matrix A nin-singular. Assuming uninformative censoring, the difference between IF, (1) and ,I)
upper bounded by
Diff == VIF, (1) (1) = Opl 2 m
Proof. The difference between IF, (i) and VIF, (i) is given by
(i) (i) (i)
=[V3C(0)/]] exp X,) 1h0 (Yz:0)
1
I(Y, sKiD,, (x-- (Y;0) Sh (Yg:0)
n
I(Y, < Y,)A,
S (Y3; 0)
Xi
S (Yz;0) - exp(0 X,)/n
S (Yz:
j-1
[3c(6)//]] exp(0 Xi)
1
1(Y, < Ki)A,
(xi- ((1) (Y;:0)
n
1 (0) (Y;
5 (Yg:
T(Y, S Y;)
S (Yz: 0)
}
X,
SR (Yz:0) - exp(0 X,)/n
S (Y):0)
j=1
A; Sh (Y:; exp (xx)//
(xi-
S( (V::0)
+
(0)
(0)) (Y:; 0)
[3)(())
exp(20T X,)
I(Y, S Y;)A,
1
X
5h (Y;: 0]
}
n
S (Y);0)
S (Y,: exp (T))//
S: (Y):
+ V3C(0)/
-1 xx)))
X²)
A:
(x--
X,
(Y
n
0 (Y:;) 0) - exp(0 TX,)/n
5
(Y::
Define
(, = T(t$
(1 0)
X,
S (t: 0)
S
(0) (t: 0) -ex(( Xi)/n
S (t:
and
T(
J(t: 0, 7t) = (1:00
(00) (1:0) exp(0 Xi)/n
((0((1:0)
16

---

A
Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Then rewrite DifT, (1) using the empirical process notation:
exp((20 Xi),
(i)
L. Jn(t; Z.)dG
Diff
V2c(0)/m]
n
-1 exp(0 Xi) (32(0)/u]
A
(xi- Sn 5% (Y:: 0)
(x:- 5t (Y;
(16)
a
Sn S1 (Y:: exp((TX.))) Xi)/n
Next, we show that
J Jn 0, Z.) [
I. J(t:0r Zi)d((() + op()).
(17)
To prove Eq. (17), we further decompose it into four terms:
[ O, Z.)dG,,
L'" ((t:0"Z,)dCCr = [- (J)(t;0ZZ)) - J[;;; 2.) ((G(())-(O))
J(;; (()(G)(())G((( + L' (n)t::,))-JJ(:4) dG(()
1.
/ (J(:0, 21) O1;00" (11;00,,,)) :()) dG(().
For the first term /1, by the triangle inequality, we have
sup tE(0,r),0EB
Jt (t: Z:) - [t;; Z,))
T(L$
7(15% Vi) (t:0))
sup rE(0.r].eEB
.X;
X;
1 (t:0) (5 (t; - exp((0 (.)/x)))))
(1)S
1(1 S Yi)
(1;
(2) (1:
+ sup 16 (E
[st0 (t:00) 2 (5h (t: exp(0T (xx((Tx.)/n)
[(=(0)
1
1
+ +02(2) 2 n
sup tE(0.r]
[(s (t; [1:0)]
[(s(0) (t: (t:
+ sup tE(0.r].0EB
(t;
[(s(0) 0)) (t ((:0)
(18)
[s (t: ():
where the second inequality relies on the the boundedness of the support of X,, t, and B. Here, "W, S W₂" denotes that there exists a universal constant C >> 0 such that W, < CW2 Under Conditions (1)-(3,, the function class (f (z, y) =
[y 2 exp(p 2) it E [0, 7]. 0 E B} is Glivenko-Cantell class, i.c. suPrejo.F].0Es [s! (t; 5(0) (t; (1). Similarly, we have suprejo.r],eE5 | (t: (t; (1). By applying the first-order Taylor expansion of the function 1/x² at x s0)) (t: 0), we obtain
[1Sh [ISF)(100]]' [1=00) [t))3 (t:
(t: 0) ((: + (1s (t: (t:0))
17

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Since X, B, and T are bounded, there exists constant C
0 such that infrejo,r) s(0) (1:0)
E (1 2 t) exp x)) 2 C, which implies that the denominators in the expansion are uniformly bounded away from zero. Therefore, the first term in Eq. (18) satisfies
1
1
15h S (t; (t: oy()).
sup EE BEB
sup LE 1,,,,1,00B
((S) (1; ():
[(s(0
(t; 0)]
Similarly, by applying the first-order Taylor expansion to the bi-variate function y/z² at the point (z. (s(") 0), (() 0)), we can show that the third term in Eq. (18) is also o1)).. Combining these results, we obtain the uniform convergence:
sup 2,)(;;,2,)) ((; o))
(19)
kEE
By the empirical process theory, we have vT((G(()) - G()) converges to Gaussian process uniformly. Therefore, follows that
1 = I.'
J(; 2,) - - -(t) -0;
To bound the second term 12, we first show that suPrg(o, [t; Z,) 0(1). By definition, all numerators in [t; 0, Z.) are bounded due to the bounded support of X, the compactness of B, and the bounded т. The denominators of J (t; Z,) involve both s0 (t; 0) and 50 (t: exp (0 X.) /n, which we show are uniformly bounded away from zero. Specifically, since X, B, and are bounded, there exists a constant C > 0 such that infr s(0) (t: 8)
E (T 2 t) exp (ØT x)) C. Moreover, the term d" X. /n O(1/n) is of smaller order, so s00) (t: 0) cxp (0 X.) /n remains bounded below by positive constant for large n. Therefore, [t; 0, Z,) O(1) uniformly over E [0, r| and E B. By plugging in Ô E B and combining the fact that /n(G,, G(t)) converges Gaussian process uniformly,
we have I2- (1/v/n).
For the third term T3, due to uniform convergence in Eq. (19), it follows that 13 (1).
To bound the fourth term 44, we use the Lipschitz continuity of (t; Zz) in 0, which follows from the boundedness of X, B, and r. Combined with the consistency of 0, i.e., 0 = 0" + (1), under the regularity conditions of Theorem A.2 (Cox, 1975), we have suP [| 0, Z.) O(t;0" Z:)ll (1) and thereby it follows that 14 (1). So far, we have completed the proof of Eq. (17). Finally, we plug in Eq. (17) together with known consistency results into Eq. (16): 0 = 0 + (1) and V2 =
A + (1),,, and obtain that
(if [A op(1)) exp T X,) " x,)+
(f'stt:rr J(t:0r Z,)dG(t) + Z.)dG(t)+
n
exp(() T xx)+ + op((1) + [A + (1))
(Y;;0) oz,(1) (xi s(4)(Y;; + o1p(1) s(0)({;;0'') ,,,(1))
A,
exp(28- Xi) xx))
SS O Zt)
= =- [A]
n
exp Xi)
(Y;: + t 1 m7
(Y;:0))
+ [A] [A]-
Xi
(00) (Y;;0'))
n
=0p 1
The second equality holds by the continuous mapping theorem and the third equality holds due to the boundedness of the support of X, B, and T. We used the fact that there exists positive constant C' >> 0 such that inf infeejo.r].0eg (0) (1; (Y 2 t) exp (ØT (0xx)) 2 C. This completes the proof.
18

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Log-log plot of ||VIF IFI| and |||i|| vs
2
0
log(norm)
-2
-3
-4
||IF IF]| (fitted slope -1.08)
-5
|||i|| (fitted slope -0.14)
5
log(n)
Figure 3. Empirical verification of the theoretical results in Theorem 3.10. The x-axis indicates the size of the training dataset n, while the y-axis indicates the average L2 error. The plot a log-log plot.
A.6. Empirical Verification of Theorem 3.10
Here we empirically verify the theoretical results in Theorem 3.10. Specifically,
randomly sampled subsets of
the METABRIC datuset with varying sample size the training datasets, and then plot the || VIF,, (f)ll and ,[ (1)l2, averaged over as function of n. As can be seen from Figure 3, || VIF, IF, (1)l2 decreases roughly in 0(1/)), while || ,I, (1)l22 fluctuates at a constant level for larger n. This aligns with our discussion in the "Approximation Error" paragraph in Appendix A.5.
B. Detailed Experiment Results
Additional Results on larger experiment settings. Table shows that the Pearson correlation coefficients of VIF are high on larger datasets and linear model and non-trivial neural networks.
Table Additional Results larger experiment settings.
Scenario
Dataset
Model
Pearson Correlation
Linear
0.9997
Cox Regression
RR-NL-NHP
Neural Network
0.3619
Listwise Searning-to-Rann
Delicious (full)
Linear
0.8337
Datasets. For Cox regression, both METABRIC and SUPPORT datasets are split into training, validation, and test sets with a 6:2:2 ratio. The training objects and test objects are defined the full training and test sets. For node embedding, the test
19

---

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
objects are all valid pairs of nodes, i.e., 34 x 34 1156 objects, while the training objects are the 34 individual nodes. In the case of listwise learning-to-rank we sample 500 test samples from the pre-defined test set as the test objects. For the Mediamill dataset, we use the full label set as the training objects. For the Delicious dataset, we sample 100 labels from the complete label set for our primary experiments, Additionally, we conduct experiments on the full label set, denoted as "Delicious (full)." The brute-force leave-one-out retraining follows the same training hyperparameters as the full model, with one training object removed at a time.
Table 5. Training objects and test objects in different experiment settings.
Scenario
Dataset
Training obj
Test obj
METABRIC
1217 samples
381 samples
SUPPORT
5677 samples
1775 samples
Cox regression
RR-NL-NHP
16000 samples
5000 samples
Node embedding
Karate
34 nodes
1156 pairs of nodes
Mediamill
101 labels
500 samples
Delicious
100 labels
500 samples
Listwise learning-to-rank
Delicious (full)
983 labels
500 samples
Models. For Cox regression, we train a CoxPH model with a linear function on the features for both the METABRIC and SUPPORT datasets. The model is optimized using the Adam optimizer with learning rate of 0.01. We train the model for 200 epochs the METABRIC dataset and 100 epochs the SUPPORT dataset. For node embedding, walks per node, each with a length of 6, and set the window size to 3. The dimension of the node embedding is set to 2. For listwise learning-to-rank, the model is optimized using the Adam optimizer with a learning rate of 0.001, weight decay of 5e-4, and a batch size of 128 for 100 epochs on both the Mediamill and Delicious datasets. We also use TruncatedSVi to reduce the feature dimension to 8.
sample 1,000
C. Efficient Inverse Hessian Approximation
Existing methods for efficient inverse Hessian approximation used by the conventional IF for decomposable losses can be adapted to accelerate VIF. Specifically, we consider two methods used by Koh & Liang (2017), Conjugate Gradient (CG) and LiSSA (Agarwal et al., 2017). The application of CG to VIF is straightforwar, as it can be directly applied to the original Hessian matrix. LiSSA is originally designed for decomposable losses in the form Σx= (0) and accelcrates the inverse Hessian vector product calculation by sampling the Hessians of individual loss terms, (O), The adaptation of LiSSA to VIF depends on the specific form of the loss function. In many non- non-decomposable losses (e.g., the all three examples in this paper), the total loss can still be written as the summation of simpler loss terms, even though they not decomposable to individual data points. In such cases, LiSSA can still be applied to efficiently estimate the inverse Hessian vector product through sampling the simpler loss terms.
C.1. Experiments
We implement the CG and LiSSA versions of accelerated VIF for the Cox regression model, and experiment them on the METABRIC dataset. In addition to the linear model, we also experiment with a neural network model, where the relative risk function is implemented as a two-layer MLP with ReLU activation. We use VIF (Explicit) to refer to the VIF with explicit inverse Hessian calculation, while using VIF (CG) and VIF (LiSSA) to refer to the accelerated variants.
Performance. As can be seen from Table 6, the accelerated methods VIF (CG) and VIF (LiSSA) achieve similar performance as both the original VIF (Explicit) and the Brute-Force I.0 on both the linear and neural network models. The correlation coefficients of all methods on the neural network model are lower than those on the linear model due to the randomness inherent in the model training.
Runtime. We further report the runtime of different methods on neural network models with varying model sizes. VIF (CG) and VIF (LiSSA) are not only faster than VIF (Explicit), especially as the model size grows, but also much more memory efficient. VIF (Explicit) runs out of memory quickly as the model size grows, while VIF (CG) and VIF (LiSSA) can be
20

---

A
Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Table 6. The Pearson correlation coefficients of methods for Cox regression the METABRIC dataset.
Models
Linear
Neural Network
Methods
VIF (Explicit)
0.997
0.238
VIF (CG)
0.997
0.201
VIF (LiSSA)
0.981
0.197
Brute-Force
0.997
0.219
scaled much larger models.
Table Runtime comparison of methods for Cox regression the METABRIC dataset. The "#Param" refers to the total number of parameters in the neural network model.
#Param
VIF (Explicit)
VIF (CG)
VIF (LiSSa)
Brute-Force
0.04K
9.88s
5.68s
8.85s
5116s
10.3K
116s
27.7s
17.18s
6289s
41.0K
OOM
113s
67.7s
/
81.9K
OOM
171s
79.1s
//
D. Heatmap of Node Embedding
In Figure present the heatmap of the influence estimated by VIF and the actual LOO loss difference on two pairs of nodes. VIF could identify the top and bottom influential nodes accurately, while the estimation of node influence in the middle more noisy. One caveat of these heatmap plots is that there is misalignment between the color maps for VIF and LOO. This reflects the fact that, while VIF is effective at having decent correlation with LOO, the absolute values tend to be misaligned.
Heatmap; Node (12, 10)
Groundtruth Heatmap: Node:
Heatmap: Node (15,
Groundtruth Heatmap: Node:
P
4
(a) VIF on (12.10)
(b) LOO on (12, 10)
(c) VIF on (15.13)
(d) LOO on (15,13)
Figure VIF is applied Zachary's Karate network estimate the influence of each node the contrastive loss of pair of test nodes. Figure 4a and Figure 4b represent the heatmap of influence on the node pair (12,10). Figure 4e and Figure 4d represent the heatmap of influence the node pair (15,13).
21


Reviews:
[{"review_id": "lxUlEwmnkv", "reviewer": "Reviewer_mDSx", "summary": "The paper proposes a method called Versatile Influence Function (VIF) for data attribution in machine learning models. Traditional influence functions require the loss function to be separable (decomposable) into individual data points, limiting their application. The authors extend the influence function to handle more complex loss functions involving multiple data points at once (non-decomposable losses), such as those found in Cox regression, contrastive loss, and learning-to-rank problems. VIF approximates the influence of each data point using auto-differentiation, making it easy to apply without manually deriving influence functions for each loss. Experiments conducted on three different scenarios demonstrate that VIF closely matches the results from leave-one-out retraining, while being significantly faster.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive feedback and further suggestions. \n\nWe will revise our abstract to make it more succinct.\n\nFollowing the reviewer's suggestion, we have also included more experiments on a larger-scale survival analysis dataset, RR-NL-NHP (Kvamme et al. 2019), with 16,000 training samples and 5,000 test samples. We run experiments on both the linear Cox model and the neural-network-based Cox model. The performance of VIF is shown below.\n\n|  Model  | Linear | Neural Network     |  \n|-----------|--------|--------|  \n| Pearson Correlation | 0.9997 | 0.3619 |\n\nAs can be seen from this preliminary result, on the larger dataset, VIF still achieves almost perfect performance on the linear Cox model, and achieves non-trivial performance on the more challenging neural network case (due to the inherent randomness in neural network training ).\n\nWe plan to include a couple of more larger-scale datasets for the other task setups too into our final version of paper.\n\n*References*\n\nKvamme, H., Borgan, Ø., & Scheel, I. (2019). Time-to-event prediction with neural networks and Cox regression. Journal of machine learning research, 20(129), 1-30.", "claims_and_evidence": "The claims in the paper are supported by clear and convincing evidence. The authors claim that the proposed Versatile Influence Function (VIF) closely approximates the leave-one-out retraining results for non-decomposable losses, which is demonstrated clearly through experiments on multiple tasks (Cox regression, node embedding, learning-to-rank). They also claim that VIF is significantly faster, which is supported by the runtime comparisons provided.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria make sense for the problem addressed in this paper. The authors propose VIF to generalize the influence function to handle non-decomposable losses, which fits well with the motivation of extending influence functions to more complex learning scenarios. The chosen benchmark tasks (Cox regression, node embedding, and learning-to-rank) represent practical examples of non-decomposable loss scenarios, effectively demonstrating the usefulness of the proposed method.", "theoretical_claims": "The authors made several theoretical analyses in the paper. While I did not thoroughly check the proof of the theorem in detail, the arguments made by the authors in the theorem proofs make sense. In the case of non-decomposable loss functions, many properties about the loss function can still be computed efficiently using several decomposition techniques.", "experimental_designs_or_analyses": "The authors evaluated their method (VIF) on three tasks: Cox regression, node embedding, and learning-to-rank, which are appropriate examples of non-decomposable loss scenarios. The choice of comparing the VIF results with the leave-one-out retraining method (brute force) also makes sense. However, I would suggest the authors explain more detail about the properties of the datasets. Evaluating on various ranges of datasets (small to large size) is also suggested. The datasets used in the papers are relatively small, it would be good to see the performance difference as the size of the dataset grows.", "relation_to_broader_scientific_literature": "The key contributions clearly build on previous literature, extending traditional influence functions from decomposable losses (Koh & Liang, 2017) to more complex, non-decomposable losses. The authors relate their method (VIF) explicitly to classical results in robust statistics and practical ML problems such as Cox regression, node embedding, and learning-to-rank.", "essential_references_not_discussed": "Nothing as far as I know", "other_strengths_and_weaknesses": "Overall, the authors proposed a solid contribution on making influence functions works on generic non-composable functions. The proposal is baked by both theoretical grounds as well as empirical verifications.", "other_comments_or_suggestions": "The abstract is too long.", "questions_for_authors": "Nothing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCJbAxdHtn", "reviewer": "Reviewer_1uWn", "summary": "## Update after author discussion period\nThanks a lot for all the discussion below, both for correcting my mistake with the asymptotics and for addressing some of my concerns. I've updated my score to vote for an accept, since I think the extra experiments now address the accuracy of VIF more clearly and I think the theory backs up its performance as well. I still have some concerns about the reliance on metrics like coefficient of correlation, which is why I've only gone up to a 3/5 on the score.\n\nI just had a few extra small comments on the theory that seem very fixable to me:\n\n1. On the boundedness of $J$: to assume that $e^{\\hat\\theta^T X_i} / n$ is of smaller order than $C$, I think that $n$ needs to be sufficiently large. I would recommend just adding this statement to the theorem. Also, I'm not sure that this theorem holds deterministically for a large enough $n$. Couldn't we have random data that makes $\\hat\\theta$ huge, thus driving up $e^{\\hat\\theta^T X_i} / n$? I think constraining $\\hat\\theta$ to a compact set would alleviate this. But right now, only $\\theta^*$ is constrained to be within some compact set, which doesn't deterministically guarantee that $\\hat\\theta$ will be bounded.\n2. Also on boundedness of $J$: the fact that $E[ I(Y \\geq t) e^{\\theta^T X} ] \\geq C > 0$ I think only follows because of the truncaction time (assumption (3)) assumed in the theorem. This just took me a minute to sort through, so it could be worth making this explicit.\n3. Sorry I might have missed the assumption that the Cox model was well-specified! I think this should go directly into the theorem statement, since it's an important assumption.\n\nBest,\n1uWn\n\n## Original review summary before author discussion period\nThis paper proposes a new method to approximate leave-one-out (LOO) model retraining. While influence functions (IFs) are widely used to approximate LOO, the IF commonly found in the machine learning literature only immediately applies to models fitted with loss functions that \"decompose\" across training examples (i.e. $\\sum_{i=1}^n \\ell(\\theta, x_i)$). The paper notes that various models do not meet this assumption, such as Cox regression or models fit with contrastive losses. So, the paper proposes the *versatile influence function* (VIF), an approximation to the IF that is exactly equal to the IF when the loss is decomposable across training examples. In one case (Cox regression) where previous authors have derived an exact form of the IF, the authors prove that the error between the VIF and IF is $O(1/n)$, where $n$ is the number of training examples. In experiments, the authors show that the value of the VIF correlates with the value of the IF.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and feedback. We have prepared a thorough response to each comment, but have to omit some of them due to the strict character limit. However, we could further provide the omitted response once the reviewer replies to us.  \n\n# Claims and Evidence\n\n> Theoretical Results ... (Theorem 3.10) suggests that the VIF is a bad approximation to the IF ... That is, $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}| \\approx |VIF_i - \\hat{\\theta}\\_{-i}|$.\n\nWe clarify that the result in Theorem 3.10 is meaningful even if $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}| = O(1/n)$. Note that $VIF_i$ or $IF_i$ themselves are not directly approximating $\\hat{\\theta}\\_{-i}$. Instead, IF (or VIF) provides an approximation of $\\hat{\\theta}\\_{-i}$ through $\\hat{\\theta}\\_{-i}\\approx \\hat{\\theta} - \\frac{1}{n-1} IF_i$ (or $\\hat{\\theta}\\_{-i}\\approx \\hat{\\theta} - \\frac{1}{n-1} VIF_i$). See page 7 of Reid and Crepeau (1985) that shows \"$\\hat{I}_i \\approx (n-1) (\\hat{\\beta} - \\hat{\\beta}\\_{-i})$\", which is equivalent to $IF_i \\approx (n-1)(\\hat{\\theta} - \\hat{\\theta}\\_{-i})$ in our notation.\n\nLet's denote $\\hat{\\theta}\\_{-i}^{IF} \\triangleq \\hat{\\theta} - \\frac{1}{n-1} IF_i$ and $\\hat{\\theta}\\_{-i}^{VIF} \\triangleq \\hat{\\theta} - \\frac{1}{n-1} VIF_i$. **Theorem 3.10 suggests that $|\\hat{\\theta}\\_{-i}^{IF} - \\hat{\\theta}\\_{-i}^{VIF}| = O(1/n^2)$, which is still non-trivial even if $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}| = O(1/n)$.** \n\nIn our paper appendix, we had a relevant discussion regarding the significance of the results of Theorem 3.10 between lines 782 and 786, where we showed that $IF_i = \\Omega(1)$, so the result $|VIF_i - IF_i| = O(1/n)$ is significant. We will further elaborate the implication on the parameter approximation in our revised paper based on our clarification above.\n\nWe further empirically investigated the errors with varying sample size $n$ using subsets of Metabric dataset. In [the first figure](https://imgur.com/a/S8L95vm), we show that $|VIF_i - IF_i|$ decreases roughly in $O(1/n)$, while $|IF_i|$ fluctuates at a constant level for larger $n$. This aligns with our discussion in appendix lines 782 - 786. In [the second figure](https://imgur.com/a/cy0HmDl), we plot $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}|$, $|\\hat{\\theta}\\_{-i}^{VIF} - \\hat{\\theta}\\_{-i}|$, and $|\\hat{\\theta}\\_{-i}^{VIF} - \\hat{\\theta}\\_{-i}^{IF}|$ for varying $n$. While $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}|$ indeed decreases roughly in $O(1/n)$, both $|\\hat{\\theta}\\_{-i}^{VIF} - \\hat{\\theta}\\_{-i}|$ and $|\\hat{\\theta}\\_{-i}^{VIF} - \\hat{\\theta}\\_{-i}^{IF}|$ are much smaller than $|\\hat{\\theta} - \\hat{\\theta}\\_{-i}|$ and decrease roughly in $O(1/n^2)$.\n\n> Secondary concern: it doesn't seem like the error ... is uniform in $i$.\n\nWe acknowledge that our theoretical result is not uniform in $i$ and will make this clear in our revised paper. However, the empirical error aligns pretty well with our theoretical result *on average*, as shown in our response to the previous point.\n\n> Quantitative results: the quantitative results only show the Pearson correlation coefficient between the VIF and exact LOO.\n\nFirst, we clarify that Pearson correlation coefficient with LOO is a commonly used evaluation metric in data attribution literature, including both the seminal paper by Koh and Liang (2017) and more recent paper published in ICLR 2025 (Wang et al., 2025). This is because, for most data attribution applications, the scaling constant and the offset do not matter much. \n\nMoreover, for the Cox model, we empirically observe that the influence predicted by VIF does align almost perfectly with the exact LOO, as can be seen in [this figure](https://imgur.com/a/bGtwktg).\n\n*References*\n\nWang, et al. Capturing the Temporal Dependence of Training Data Influence. ICLR 2025.\n\n> Qualitative results: The paper presents a \"case study\" ... too brief to really help understand at a qualitative level what the VIF was doing.\n\nThanks for the suggestion. We will include more details to make the takeaways clearer. Regarding the case study on Cox model, one takeaway is that, for the model to predict a long survival time for a test data point, the most influential training data points are either 1) data points that are similar to this test data point AND have long event time, or 2) data points that are dissimilar to this test data point AND have short event time AND the event has occurred. By domain knowledge, we meant our knowledge about Cox model.\n\n# Theoretical Claims\n\nWe double checked the correctness of our proof in response to all questions by reviewer. We will provide details in the follow up discussion.\n\n# Essential References Not Discussed\n\nWe implemented LiSSA as a proof-of-concept for VIF extension. We will discuss its limitation. We will also discuss Ghosh et al.\n\n# Other Strengths And Weaknesses\n\nWe will revise our statements per reviewer's suggestions. One note: our M-estimator notion followed the textbook *Asymptotic Statistics* by van der Vaart.", "claims_and_evidence": "I think the central point of the paper, which is that the VIF is a good approximation to LOO retraining, is not currently well supported by the paper. I've broken my concerns into two areas below:\n\n**Theoretical Results**\nThe main theoretical result backing up the accuracy of the VIF comes from Theorem 3.10, which says that for Cox regression under regularity assumptions, the difference between the VIF and IF is $O_p(1/n)$. I have some questions about the proof of this result described below. But, more importantly, I think this result actually suggests that the VIF is a bad approximation to the IF. While $O(1/n)$ sounds small, it's important to understand the overall scale of the problem. In particular, the VIF is intended to be an approximation to IF, which in turn is an approximation to the exact LOO parameters (i.e., the model parameters refit with one datapoint left out). So Theorem 3.10 shows us that VIF's error to the exact LOO parameters is at least $O(1/n)$. But there's typically a much simpler approximation to the LOO parameters that is also $O_p(1/n)$ in error -- the model parameter fit with the full dataset. That is, $\\| \\hat\\theta - \\hat\\theta_{-i} \\| \\approx \\| VIF_i - \\hat\\theta_{-i}\\|$.\n\nI want to emphasize this is *typically* true -- for example, see Wilson et al. (2020) Theorem 14. It's possible that for the specific types of non-decomposable models considered here, the LOO parameters vary more. But I think this needs to be demonstrated in the current paper.\n\nA secondary concern is that it doesn't seem like the error for the $i$th datapoint, $VIF_i - IF_i$ being $O_p(1/n)$ is uniform in $i$. This makes the theory less useful in practice. For example, the VIF could be arbitrarily inaccurate for any particular training point; the theory would just tell us that, if we were to hypothetically gather more data, the accuracy of all existing training samples would improve. I think this should be discussed or clarified in the proof that the error rate really is uniform over $i$. It's not straightforward to prove such uniform convergence, so I don't think this should be majorly held against the paper as long as it's clearly explained.\n\n**Empirical Results**\nThe experiments are broken into two sections: quantitative and then two qualitative case studies. I didn't feel like either fully told the story of why the VIF is a good approximation.\n\n*Quantitative results*: the quantitative results only show the Pearson correlation coefficient ($\\rho$) between the VIF and exact LOO. They show that the correlation is often high ($>0.9$), except for the node embedding model. But why does a high $\\rho$ tell us that the VIF is working well? $\\rho$ will be high if there is a perfect linear relationship between $VIF$ and $LOO$. So, say, $LOO = 823742*VIF + 8176234718942893472834$. But certainly in that case we wouldn't say that VIF is a good approximation to LOO -- the error would be horrible. *One* of the purposes of the IF is to rank datapoints in order of importance. And $\\rho$ being high does suggest that the VIF probably gets this ordering correct. But in that case, the results should just show the error in the actual ordering. Or I think the paper needs to justify why $\\rho$ being high is sufficient for the ordering to be correct. So, overall, I don't think measuring $\\rho$ backs up the idea that the VIF is accurate.\n\n*Qualitative results*: The paper presents a \"case study\" for using the VIF in Cox regression and a node embedding model with a contrastive loss. I found these to be too brief to really help understand at a qualitative level what the VIF was doing. In the (censored) Cox regression example, Table 3 shows, for two different test points, a few pieces of information about the top-5 VIF-scoring training points. The information given is the cosine similarity between the training point and the test point, the amount of time the point was observed for, and whether or not the point was censored. I didn't understand why this tells us anything about whether VIF is doing anything useful. The paper notes that the table shows results that \"align with domain knowledge.\" But there's no additional information about what this domain knowledge is or how well these findings align. I think a lot more detail needs to be given in these case studies.\n\nA. Wilson, M. Kasy, and L. Mackey. Approximate Cross-validation: Guarantees for Model Assessment and Selection. AISTATS. 2020.", "methods_and_evaluation_criteria": "I think the datasets used do make sense. I have some concerns about the evaluation criteria that are detailed in the Claims and Evidence section.", "theoretical_claims": "I only skimmed the derivation of the VIF; I actually thought the derivation didn't need to be as long as it was (see suggestions below).\n\nI did read the proof of Theorem 3.10 carefully, and I have a few comments below:\n\n1. Around line 872 \"by applying Taylor expansion to terms in Eq. (18) and the boundedness\" -- I think this needs much more detail. E.g., what terms? Expansion in which variables? To what order of expansion? Etc.\n2. Line 880: \"note that $J(t, \\hat\\theta, Z_i)$ is bounded\". I don't see why $J$ is bounded. $J$ contains the term $1 / (s^{(0)}(t;\\hat\\theta) - e^{\\theta^T X_i}/n)$. For a given $t$, $s^{(0)}(t;\\hat\\theta)$ is a constant. Unless it's a negative constant, there will exist some $X_i, \\hat\\theta$ such that $e^{\\hat\\theta^T X_i}$ gets arbitrarily close to $s^{(0)}(t; \\hat\\theta)$. This makes it seem like $J$ is not bounded.\n3. Line 881: \"Given the boundedness and the consistency of $\\hat\\theta$\" -- this needs to be made an explicit assumption; the theorem doesn't say anything about $\\hat\\theta$.\n4. Around line 743: \"A consistent estimate for [the information matrix] is given by $\\nabla^2_\\theta L(\\hat\\theta) / n$. This isn't true. This is only correct if the model is well-specified, which hasn't been assumed here.", "experimental_designs_or_analyses": "Yes, the experimental setup seems reasonable.", "relation_to_broader_scientific_literature": "Computing influence functions is a growing area in the machine learning literature. As the authors note, most work focuses on decomposable losses, where there is one term in the loss corresponding to each datapoint. However, non-decomposable losses are not uncommon in machine learning -- I thought the current paper's example of contrastive losses was particularly broadly applicable. I think applying influence functions to such models is an important area to study, so this paper is focused on an important and understudied problem.", "essential_references_not_discussed": "I think the paper is missing two important references:\n\n1. Basu et al. (2021) show that LiSSA (a method proposed in the current paper to approximate the Hessian inverse needed to compute the VIF) works very poorly for large models (i.e., exactly when it is needed). But most new papers on influence functions seem to miss this point because Koh and Liang (2017) used LiSSA. I really think no one should be using LiSSA to approximate influence functions at this point, and I would suggest it's taken out of the current paper completely to avoid influencing future authors in this space.\n2. Ghosh et al. (2020) also tackles the problem of applying influence functions to \"non-decomposable\" losses. Their results are exact, but are focused on models with latent variables.\n\n**References**\n- Samyadeep Basu, Philip Pope, Soheil Feizi. Influence Functions in Deep Learning Are Fragile. ICLR. 2021\n- Soumya Ghosh, William T. Stephenson, Tin D. Nguyen, Sameer K. Deshpande, Tamara Broderick. Approximate Cross-Validation for Structured Models. NeurIPS. 2020.", "other_strengths_and_weaknesses": "**Strengths**\nI think the idea of a finite difference approximation to resolve the issues of applying IFs to non-decomposable losses is an interesting one! I actually thought there was an even more direct approach to deriving the VIF: noting that for decomposable losses, the loss is linear in each of the weights $w_i$: $\\sum_i w_i \\ell(\\theta, x_i)$. So when setting one weight to zero, all of the other terms in the difference in gradients in Eq. 10 cancel out. Then VIF can be thought of assuming the same thing holds for other models. Anyway, I thought this was a cool idea.\n\n**Weaknesses**\n1. I think the early discussion in the paper has misunderstood what M-estimators are. In general, an M-estimator is just an estimator that minimizes some loss. There's no requirement that it decompose among different datapoints, which is what the text implies.\n2. The citation to Huber & Ronchetti (~line 231) should reference a specific result and not an entire book\n3. The introduction implies that the current method overcomes the challenge of non-convexity: \"secondly, for non-convex models... to overcome these challenges, we propose the Versatile Influence Function\". I think this is misleading because the VIF doesn't solve this problem.\n5. What's the compute time required for the given algorithm? It seems like it might be $O(n)$ per datapoint (e.g., looking at the equation near the bottom of p. 14, the gradients for Cox regression seem to each take $O(n)$ to compute), for a total of $O(n^2)$ time across all datapoints. This sounds expensive, so this would be good to discuss in the main text.", "other_comments_or_suggestions": "Nothing not covered above!", "questions_for_authors": "My main questions are:\n1. Why does Theorem 3.10 tell us the VIF is good given the issues raised above?\n2. Why does measuring the Pearson correlation coefficient tell us that the VIF is good?\n3. What are the takeaways from the case studies?\n\nThank you!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JG43Am3pEF", "reviewer": "Reviewer_p9eS", "summary": "handle non-decomposable loss functions, thus broadening their application in machine learning models. Unlike conventional approaches limited to decomposable losses, VIF can be directly applied to any model trained with complex losses like contrastive or ranking losses, without needing retraining. Utilizing automatic differentiation, VIF simplifies the computation process and is applicable to non-convex models. Experiments on various tasks, including Cox regression and network analysis, demonstrate VIF's effectiveness and significant computational efficiency gains, up to 1000 times faster than brute-force methods. Additionally, VIF integrates well with efficient inverse Hessian approximations, further enhancing its scalability and performance in large neural networks. Despite assuming convex loss functions, VIF opens new possibilities for data attribution in complex models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and further suggestions. We address your comments in detail below.\n\n> Although VIF performs excellently in many aspects, its assumption that the loss function is convex may limit its effectiveness in certain practical applications. Therefore, future work might explore how to adapt to more complex model structures to enhance its practicality.\n\nIn our current paper, we have heuristically extended the proposed method to non-convex models and experimented a few non-convex settings, such as the neural-network-based Cox model and the node embedding model. However, we agree that it is an important future work to further explore the adaptation of VIF to more complex model structures.\n\n> This article primarily focuses on the issue of data attribution for non-decomposable losses but does not mention recent advancements in the application of influence functions within the fields of graph neural networks.\n\nThanks for the suggestion. We have further explored the literature and found two relevant papers that aim to adapt influence functions for graph neural networks, which can be viewed as special cases of non-decomposable losses. Chen et al. (2023) developed an influence function specifically for Simplified Graph Convolution (SGC) model, which is a linearized graph neural network model. Wu et al. (2023) proposed a machine unlearning method for graph neural networks based on influence function, where the influence function is adapted to consider the graph dependency among samples. In comparison to these methods, our approach has a more general formulation and can be broadly applied to various different non-decomposable losses.\n\n*References*\n\nChen, Z., Li, P., Liu, H., & Hong, P. (2023). Characterizing the Influence of Graph Elements. In _The Eleventh International Conference on Learning Representations_.\n\nWu, J., Yang, Y., Qian, Y., Sui, Y., Wang, X., & He, X. (2023). Gif: A general graph unlearning strategy via influence function. In _Proceedings of the ACM Web Conference 2023_(pp. 651-661).\n\n> The VIF proposed in this article is a second-order method, which implies a relatively high computational complexity.\n\nThe proposed VIF is indeed a second-order method. However, we have shown that we could leverage computational tricks used in conventional IF, such as CG and LiSSA, to accelerate the computation. In the future work, we will explore the integration of more advanced computational tricks, such as EKFAC (Grosse et al., 2023) or dimension reduction (Schioppa et al., 2022), into the proposed VIF.\n\n*References*\n\nGrosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., ... & Bowman, S. R. (2023). Studying large language model generalization with influence functions. _arXiv preprint arXiv:2308.03296_.\n\nSchioppa, A., Zablotskaia, P., Vilar, D., & Sokolov, A. (2022). Scaling up influence functions. In _Proceedings of the AAAI Conference on Artificial Intelligence_ (Vol. 36, No. 8, pp. 8179-8186).\n\n> Is the VIF proposed in this article effective for any loss function?\n\nThe proposed VIF is in principle applicable to any loss function covered by Definition 3.1, which is a very general form of loss function. However, the practical effectiveness of VIF (in terms of its approximation to the LOO) may vary from case to case, as our theoretical guarantees are limited to certain assumptions.", "claims_and_evidence": "**Claim-1**: VIF provides a general form of influence function that can be directly applied to machine learning models trained with any non-decomposable loss.\n\nEvidence: The article demonstrates how VIF generalizes from M-estimators to general non-decomposable losses through theoretical derivations (such as Theorem 3.6 and proofs in the appendix). Furthermore, experimental results are provided showing that in tasks such as Cox regression, node embeddings, and list-wise learning to rank, the influences calculated by VIF closely match those obtained through brute-force leave-one-out retraining.\n\n**Claim-2**: VIF offers a new avenue for data attribution concerning non-decomposable losses, opening up opportunities for data analysis applications across broader fields.\n\nEvidence: The article showcases the effectiveness of VIF through multiple case studies, including Cox regression for survival analysis, node embeddings in network analysis, and list-wise learning to rank in information retrieval. These case studies demonstrate the broad application potential of VIF.", "methods_and_evaluation_criteria": "The Versatile Influence Function (VIF) method and its evaluation criteria proposed in the article are reasonable for addressing data attribution problems associated with non-decomposable losses and are applicable to current issues and application scenarios. Below is a detailed analysis:\n\n**Proposed Method:**\n- **VIF Approach:** VIF is a general form of influence function designed to overcome the limitation of traditional influence functions that only apply to decomposable loss functions. By integrating automatic differentiation techniques with inverse Hessian matrix approximations, VIF can be directly applied to machine learning models trained with any non-decomposable loss, enabling the estimation of the impact of data points on model parameters without the need for retraining.\n- **Acceleration Techniques:** The article discusses how to use Conjugate Gradient (CG) and LiSSA to speed up VIF calculations, which is particularly important for improving computational efficiency, especially when dealing with large-scale neural network models.\n\n**Evaluation Criteria:**\n- **Benchmark Datasets:** The authors selected the METABRIC dataset for experimental validation of Cox regression models and further tested them on neural network models. These choices reasonably reflect VIF's performance across different levels of model complexity.\n- **Performance Metrics:** The article uses Pearson correlation coefficients to measure the similarity between VIF and other methods (such as brute-force leave-one-out), which is an effective way of assessing consistency and reliability in model outputs.\n- **Runtime Analysis:** In addition to accuracy, the article also focuses on the runtime of each method. Results show that CG and LiSSA accelerated VIF not only perform comparably to original VIF and brute-force leave-one-out in terms of performance but also offer more memory and time savings in large models, further demonstrating its practical value.\n\n**Reasonableness Analysis:**\n- **Applicability:** The VIF method has been proven to effectively apply to various scenarios, including Cox regression in survival analysis, node embeddings in social networks, and list-wise learning to rank in information retrieval, showcasing its broad applicability.\n- **Validation of Effectiveness:** Through theoretical derivation and empirical experiments, the article verifies the effectiveness of VIF. Particularly, in specific cases (e.g., M-estimation), VIF can accurately recover classical influence functions; for more complex non-decomposable losses (such as Cox regression), VIF also shows good approximation performance.\n- **Potential for Improvement:** Although VIF performs excellently in many aspects, its assumption that the loss function is convex may limit its effectiveness in certain practical applications. Therefore, future work might explore how to adapt to more complex model structures to enhance its practicality.", "theoretical_claims": "The theoretical claims and their proofs presented in the article are generally correct, with a logically rigorous connection between each step.", "experimental_designs_or_analyses": "In reviewing this article, I examined the effectiveness of the experimental design and analysis. Below is a detailed analysis:\n\n**Experimental Design and Analysis**\n\n- **Experimental Setup:**\nThe article selects three different application scenarios: Cox regression, node embeddings, and listwise learning-to-rank. For each scenario, appropriate benchmark datasets were chosen for experimental validation, such as the METABRIC and SUPPORT datasets for Cox regression, Zachary's Karate network for node embeddings, and Delicious and Mediamill datasets for listwise learning-to-rank.\n\n- **Performance Evaluation:**\nPerformance was measured by calculating Pearson correlation coefficients to assess the similarity between results obtained by the VIF method and brute-force leave-one-out retraining. The results show that in most cases, the VIF method achieves outcomes close to those of brute-force retraining, especially in Cox regression tasks (with a Pearson correlation coefficient of 0.997 on the METABRIC dataset). In node embedding tasks, although the performance of the VIF method is not as strong as brute-force retraining (Pearson correlation coefficients of 0.407 versus 0.419 on the Karate network), considering its significant improvement in computational efficiency (several orders of magnitude faster), this slight performance drop is acceptable.\n\n- **Application of Acceleration Techniques:**\nThe article explores how to use Conjugate Gradient (CG) and LiSSA to accelerate VIF calculations, with experimental verification conducted on Cox regression models using the METABRIC dataset. These acceleration techniques not only improve calculation speed but also conserve memory and time resources in neural network models.\n\nOverall, the experimental design of this article is reasonable and effective, supporting its theoretical claims.", "relation_to_broader_scientific_literature": "The key contribution of this article is the introduction of a new method called Versatile Influence Function (VIF), which extends the application of traditional influence functions to scenarios involving non-decomposable losses. This contrasts with previous data attribution methods that were limited to decomposable losses, such as M-estimation. By drawing on application ideas from complex models like Cox regression and utilizing automatic differentiation tools in modern machine learning libraries for efficient computation, VIF opens up new research directions and technical approaches in the field of data attribution.", "essential_references_not_discussed": "This article primarily focuses on the issue of data attribution for non-decomposable losses but does not mention recent advancements in the application of influence functions within the fields of graph neural networks.", "other_strengths_and_weaknesses": "The VIF proposed in this article is a second-order method, which implies a relatively high computational complexity.", "other_comments_or_suggestions": "The overall structure of the article is clear and logically sound. However, future work could consider adding experimental validations on different types of non-convex models to enhance the applicability of the VIF method. It is also recommended to further explore and specify the practical techniques needed for applying VIF to large neural networks in real-world scenarios. Additionally, providing a detailed list of typos would help improve the professionalism and readability of the paper, although specific typo examples are not listed in the current version.", "questions_for_authors": "Question: Is the VIF proposed in this article effective for any loss function?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes data attribution method called Versatile Influence Function (VIF) that extends influence functions to non-decomposable loss functions. 
Most natural loss functions are decomposable, so this is an uncommon, but interesting problem since non-decomposable losses appear indeed in several important applications and settings including contrastive and ranking problems. 
The connection of VIF to auto-differentiation is important because it makes it straightforward to apply due to existing infrastructure. 

The authors presented strong results and participated in detailed discussions with the reviewers which were convinced that the paper is theoretically solid and interesting. 

Overall a solid contribution to a somewhat narrow but important problem.
Decision: Accept (poster)