Kernel von Mises Formula of the Influence Function
Authors: Yaroslav Mukhin
Venue: NeurIPS 2025 poster ()
Keywords: Influence function
Fisher-Rao gradient
first variation
tangential interpolation
semiparametric estimation
distributional robustness
kernel PCA
Mercer kernel
Primary area: theory
TL;DR: We derive a regularized representation of the influence function using the spectral theory of positive semidefinite kernels.

Abstract: The influence function (IF) of a statistical functional is the Riesz representer of its derivative, also known as its first variation and Fisher-Rao gradient.  It is a key object for numerical optimization over probability measures, semiparametric efficiency theory, standard constructions of efficient estimators, and an arsenal of inference methods for these estimators.  Yet, deriving the IF analytically is often an obstruction for practitioners.  To automate this task, we develop a novel spectral representation of the IF that lends itself to a low-rank functional estimator in a reproducing kernel Hilbert space (rkHs).  Our estimator (i) does not require analytic derivations by the user, (ii) relies on kernel Principal Component Analysis and numerical pathwise derivatives along these components.  We present the derivation of the representation and prove consistency of the low-rank rkHs estimator.

Kernel von Mises Formula of the Influence Function
Yaroslav Mukhin
Cornell University kif kif@ymx.
Abstract
The influence function (IF) of a statistical functional is the Riesz representer of its derivative, also known as its first variation and Fisher-Rao gradient. It is a key object for numerical optimization over probability measures, semiparametric efficiency theory, standard constructions of efficient estimators, and an arsenal of inference methods for these estimators. Yet, deriving the IF analytically is often obstruction for practitioners. To automate this task, we develop a novel spectral representation of the IF that lends itself to low-rank functional estimator in a reproducing kernel Hilbert space (rkHs). Our estimator (i) does not require analytic derivations by the user, (ii) relies on kernel Principal Component Analysis and numerical pathwise derivatives along these components. We present the derivation of the representation and prove consistency of the low-rank rkHs estimator.
Introduction
The target 0 of a statistical learning procedure often takes the form of a mapping P ((P) of the probability measure P on a sample space X' C IR governing the training data Xim- The difficulty of learning 9 from data depends on the combination of structural properties of all possible P and the local variability of f at the unknown true P. For example, estimating the mean 01 (P) = ʃ x dP is easy even if P is unrestricted beyond having finite moments. By contrast, estimating the density ff (0o) of P at a point Zo is easy in a model for that is smoothly parametrized by a subset of RP, but much harder in nonparametric settings. The rate of convergence of an estimator 6,, (X to the target (P)) as the sample size n increases The rate of estimating 02 with the sample mean is n by the Central Limit Theorem (CLT), whereas the rate of estimating (xo) can range from /n arbitrarily slow, depending on the regularity of P. If fxx) is smooth in neighborhood of for all P. then observations close to zo can be aggregated, as for estimating 0,, resulting in fast rate; if f ((x varies roughly near zin and as perturbed, then aggregation leads to bias and must be limited at the expense of precision.
measure of the statistical difficulty of estimating
We consider scalar functionals ((P) that can be estimated at the parametric /n rate in nonparametric models for P. like the mean 0₁. This includes scalar and vector estimands that depend on averages f ff dPP and estimands of causal inference, but not (zo). For
of nuisance functions, e.g., (p) these parameters the rate is fixed and the difficulty of estimation is characterized in terms of lower bound on the asymptotic variance. The object that determines this bound is the influence function (IF) of at P, denoted tpp X - IR; the bound is the norm f V dP of this function. Setting aside the interpretation of ( until Theorem 2.1, recall when one can expect ((P) to be estimable at the vn rate and to have an IF. This requires that O(P) varies smoothly with P. Specifically, the map P - ((P) must have derivative DOp that maps perturbations to into infinitesimal changes in 0(P) [Mi477; Ste56; KL76; IHB1; Vaa91]. When DOp exists and is a bounded linear map (P)( Riesz's theorem guarantces the existence of a function upp E (PPP) such that for any perturbation & to P. the effect on ((P) is given by D0r[d] f dup dP; we review this further in Theorem 2. If 0 is not smooth in P. so the derivative D0,, doesn't exist or is unbounded and has no IF, then the estimation rate for ((P) is typically slower than Vn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Example 1.1. The IF for 81 (P) is (2) = = 81: the IF for (P( is vu(x) - 2[f(x) - 0o]. so that F(O,) == P(X - 0,) = 9. and its IF is (o(r) [g - I (x v fg)l/ where A is the indicator of event A. To simplify notation, we may write d without the subseript P. but the IF always depends on P. as do the functionals @ and DO.
With q E (0, 1), the q-quantile (P) F (
Consider now an estimator sequence O,, 0, for ((P) constructed with i.i.d. samples X1: from P. It is regular if vn(0, - (()) converges in distribution, and, furthermore, the limit law is invariant to vanishing perturbations in P: counterexample is hard thresholding. An estimator is asymptotically linear if it has the representation analogous to the CLT:
Vn(θn -(P)) = vn (x,), + O1(),, where Up E LP))
(1.1)
meaning S6 cp < +o0 and & dP 0, and the (1( term vanishes in P as n 0C. Note that ppp in (1.1) the IF of 0, also called the IF of the estimator ô. [Vaa9]] shows that ((P) is differentiable in P and has an IF if and only if there exists a regular asymptotically linear estimator. Moreover, [Kla87] shows that a regular asymptotically linear estimator exists if and only if the IF exists and can itself be consistently estimated. For further details of semiparametric efficiency theory we refer to (Vaa00, ch25] and [Bic+93], and now turn to a very brief and selective review of its recent methodological uses in machine learning (ML), statistics and econometrics.
According (L.1), the contribution of a datum X; the fluctuations in the estimator approximately (x,))/n. Consequently, the observations with the most influence on the realized estimate are X; with a large (xx)] value. Returning to Example 1.1, for estimates of 01. the influential data are those far away from the mean, i.e., the 'outliers'; by contrast, for estimates of fg all observations have similar influence; for estimates of On the observations with extreme density values are influential. This interpretation of the IF to study robustness to outliers goes back to [Hub64] and appears recently in e.g., 20 BGM20). In Theorem 2.1, we discuss related interpretation of the IF, but for the functional rather than its estimators, that is an important baseline for our work.
Closely related with robustness are the uses of the IF toward data attribution and interpretability of nontransparent estimators. The idea is to approximate the r; + ((11.)) mapping with x; + vxx;)) to gain insight into the effect of data point or statistic on the estimate; e.g., in large language [Gro- black box MI. [KL17], structural econometric [AGS20a; AGS20b) models. A different question asks for the effect of perturbing the location of r; spatially in the sample space X' as I; + A, rather than its probability weight; this can also be answered with the IF. The Wasserstein gradient vector field V VaVp = (8z, p...... up): : X + Ed of ((P) describes the direction of transporting the mass of P at x in X' C IRd with the greatest influence on the value of ((P). Furthermore, for any transport perturbation v E (22(P)d of P. the effect on ((P) is D0p [v] = f(v, Vz 1. dP, See (vi103, chej for the details of optimal transport theory, and e.g., [Mad+18; SND18] for applications.
A classical use of the IF is to approximate the distribution of an estimator [Chas2: New34; FS19), useful for
constructing confidence intervals and statistical tests. From (1.1), the variance of é,, is approximately
f w dP/n and the distribution is approximately Normal by the CLT. Beyond insights into the distribution of a given estimator, the IF is a key ingredient for the construction of estimators that achieve the efficiency bound, notably in semiparametric problems like causal inference. Several techniques are well known, all start with preliminary estimators á of 9 and dh of thp, which are combined to construct better estimator of 6. The one-step adaptive estimator of Bickel [Bic75] given by 0 := 0 + Σi= v)r(X))/ estimates and removes the bias in 0 with the sample average of dhp. The IF is required to construct the targeted likelihood of van der Laan [VR06: Cho+24), and Neyman orthogonal estimating equations of Chernozuukov [Che+18; Che+22], see also the reviews [Hin+22; Kan24].
Recently, the IF is used extensively to study robustness of ML models [Bre+19; Guo+20; Bae+22; Sch+23] and the sensitivity of econometric estimands 0 to misspecification of structural properties of P and other modeling choices reflected in 6 ((P) [AGS17; Muk18; Muk19; cc23). For example, with censored data, the mean 0, is not identified, but there is a set of values for 01 compatible with the true P and the observed data that can be estimated [HM95; Sem20; Sem255; an MDP model may assume Gumbel payoff shocks [Rusa7] or an ecological model of population dynamics may posit logistic propensity scores [Cat+o0], and the IF can be used to construct bounds for the estimated MDP parameters and population size that are robust to the parametric assumptions [Muk21].

---

How does one estimate the IF? In Example 1.1, estimating 1 amounts to estimating the mean #1; by contrast, estimating Vn requires estimating the density f; estimating 40 requires estimating 0, and )))). ). For a general functional ((P), one first derives the analytic form of the IF. In parametric models, the IF is the normalized derivative of the log-density, i.e., the score. In nonparametric models, with significant mathematical subtlety, the problem can be reduced to the parametric case. For path + Pt, i.e., a parametric submodel with parameter E R and score function g E L)(P), the pathwise derivative d0(Pt) dt is computed. Then one hopes to express it as f 9 &dP to match the representation of Riesz' theorem; the function & is a candidate for t.p. Given the analytic form of s/p, as in Example 1.1, the unknown components can be estimated. Deriving the IF is highly idiosyneratic to the functional at hand and can be challenging, akin to solving a differential equation, and often constitutes a significant contribution. Several techniques are described in the literature e.g., [Cha87: New90; Jor93], see also the reviews [Hin+22; Ken24]. This task can be time-consuming. requires familiarity with functional analysis, and often highly specialized technical knowledge.
Prior works It is widely documented in recent literature that the analytic derivation of the IF poses an obstruction to the adoption of IF-based methods [Fra+15; LCL15; CLV19; Hin+22; Ken24; Jwz22] and that replacing this derivation with automated estimation would be a useful contribution. Automated estimation has been explored in [Fra+15; JWZ22] based on the von Mises representation of the II, explained in Theorem 2.1. Specifically, these works show how to estimate the value of the IF at a fixed point = E X' as ()) = lim d0( // /dt, by constructing a special perturbation P. of P. that depends on the chosen point 2 and bandwidth 6, so that the pathwise derivative of 0 along this perturbation approximates (2). practice, 8 is fixed and the derivative is computed as a finite difference; computation can be ill-conditioned due to the nature of the required perturbations.
Regularization X=1e-6
Regularization x=1e-4
0.5
NF
r=B
r
3=32
r=32
-0.5
8
r=8
323
3=32
-1
-2
2
-2
2
Figure 1: IF 1h of quantile 0g. regularized oracle surrogate 4x (dashed) and estimate 2x,,, (solid).
Contribution We propose estimating the entire IF uh as
{(x) Σ 12/0/
d (())) dt
(x))),
|t=0
where A > 0 is the regularization loading for controlling the nonparametric bias-variance trade-off, and r is the rank of the approximation for controlling the number of eigenvectors of the Gram matrix and pathwise derivatives of ( one has to compute (e.g.. r = 16) to estimate the entire IF. See Figure 1. Furthermore, (e,);21 21 is an orthonormal basis for L3 (P) and (αz))21 is a decreasing to 0 sequence of scalars such that (Voee))s
is orthonormal basis for Hilbert space of smooth approximating perturbations S of the density ʃ of P are in the direction of ej. e.g. S [1 + Lé leading to stable pathwise derivative computation.
functions;
Our estimator is based on a novel regularized representation of thp by the best approximation in a ball of Hilbert space H, Theorem 3.3. We view this representation as our main result, because is not trivial to apply rkHs methods to the IF, and our variational characterization lends itself to this task. We relate estimation of the IF to kernel Principal Component Analysis (PCA) by taking // be the rkHs of positive definite kernel. This allows leveraging extensive learning theory and methods of numerical linear algebra to analyze and compute our estimator. In the proposed implementation. both e; and o, depend on P and are estimated with a low-rank matrix eigenvalue and eigenvector decomposition. We prove consistency and lay the foundation for studying convergence rates, statistical and computational efficiency, and their trade-offs in follow-up work. To our knowledge, this is the first data-driven functional estimator of the IF, the first estimator with convergence guarantees in r?² and rkHs norms, and the first work to show how to apply rkHs methods to the estimation of the IF.
3

---

Paper organization Section 2 reviews the classical von Mises formula, which is the baseline for our spectral representation and proposed estimator vOy. Section 3 presents our variation representation, regularization, and the solution of the resulting approximation problem. Section implements the approximation with an rkHs, applies the Nyström method for integral operators to estimate the regularized surrogate and proves consistency of the estimator. The Appendix contains simulation experiments and deferred technical details.
Notation We write IF for influence function; P is a of probability measures on sample space E X' C R", X1m = (X1..... Xn)) ) is sample; 0 p - R denotes a functional; (P) is the X R with f @ dP oo and PP( the subspace with f Φ dP
space of functions The derivative of with respect P at P is denoted by D0p (P) - R and maps directions of perturbing P into infinitesimal changes in ((P). 1 denotes the indicator of set A; Ldd denotes Lebesgue measure on g
2
von Mises Formula
The following calculation extends those of [Mis47: IN22] and allows evaluation of the IF & at a point E X' via evaluations of the functional (PP) on certain perturbations of P. A useful analogy is to think of computing partial derivatives of multivariate function of IR? or IR to evaluate the gradient vector.
+ THEOREM 2.1 [von Mises formula]. Let 0 : - IR be pathwise differentiable functional on
nonparametric model P with IFs up E (P(P) for E P. Suppose = f. LLdd is a.c. with
continuous density f. Let K be bounded probability density with support in {|ar| S 13 Rd.. Define the dilated kernels by (xx = 8 8- (((-k(8)))) 2)) for 6 > 0. Translate to the location of approximation 2 E R.d and control the likelihood ratio with f via cutoff as K K²,²(x) == cK* x) 11 ( (x) with c = -1rso K$ 1 x) dz. For small 8 and
{ff 0}, consider the family, indexed by the bandwidth of paths " with parameter and density f. (z) = -
Note these paths perturb the measure P toward the point-mas. distribution at E X, regularized via the approximation to the identity KS. Then the following IF formula holds:
pp((2)
lim
for P-almost every z Rd.
(2.1)
5 5-40
1==
Proof. We outline the proof as a way of reviewing pathwise differentiability and provide the details in the Appendix. The score function, i.e., the derivative of log-density, of the path t + p at is
d log log(f(a) f(x) + (())) == (2-2z)))f(a) x)/f(x)-11
(x) = it
The score @5.5 (x) is an (P) function that characterizes the infinitesimal change in the density at x as P is perturbed along the path p. Pathwise differentiability of 0 at P means that the derivative of the functional 0 along the path + p²5. exists and is a bounded linear functional of the score Φs... By Riesz' theorem (SS09, 4.5), [Dud18, 5.5.1] for bounded functionals the Hilbert space L2(P), the derivative DO [088,=] is given by the (P) inner product of the score Φ8,2 with the IF ahp:
0(P..2) = D0p(0s.2] = / (p(x) K - x) dr = (vpp* * K¹) )=) d=
sptp
The assumed properties of the mollification kernels 5 ensure that an approximation to the identity (SS09, 3.2] in the sense that it converges - 0 to the singular point-mass distribution in the integral pairing with a [.,( function. By the Lebesgue differentiation theorem (SS09, 3.3), [Dud18, 7.2] follows that the convolution (wr [²) - vr(22) converges as 0 pointwise at
the Lebesgue points of vhp and therefore for P-almost every Rd
Let's interpret Theorem 2.1. It says that to compute a single value of the IF is sufficient to compute the values of the functional 0 along certain perturbation to P. Specifically, (P(2) is the effect

---

on ((P) of perturbing the P-weight of the outcome & E X. Recall the IF-methods described in Section I and note that these rely this interpretation and the approximation of (PP) by 0. Thus, provided with a device for computing the derivative d0/ dt in (2.1) numerically, one can numerically query v/; indeed, [CLV19; JWZ22] finite differences with similar von Mises representation. Our regularity assumptions for this result are different from those in the literature, by employing Lebesgue differentiation make additional regularity assumption about ppp (g)
In statistical applications one typically requires the entire map = > (=) rather than particular value (2). For example, to find influential data points for an estimate 6, one seeks the global maximum or level sets of vi; for constructing a debiased estimator of 0 one needs to integrate against vi: to find influential data points in the Wasserstein sense, one needs to apply a differential operator to the gradient (=x) and maximize the resulting score function. Therefore, in practice, formula (2.1) is used to evaluate many values of v simultameously. With this in mind, note that (2.1) requires separate computation for each evaluation and that the required perturbations toward point mass have been found numerically challenging [CLV19: JWZ22). Also note that the regularization in (2.1) does not account for properties of the measure P such as concentration or properties of the function ei such as smoothness. Furthermore, downstream tasks e.g., causal inference, require convergence rates in function norms for the estimator of the IF, while it is not known how obtain these with (2.1). These observations suggest that (2.1) may not be numerically and statistically optimal for estimating the entire function vi or even isolated values of tb. To address these, we propose a new representation.
3 Spectral von Mises representation
3.1 Exact representation
We begin by finding a variational representation of the IF in terms of pathwise derivatives of the functional. The following lemma is immediate consequence of Riesz' theorem [Dud18, 5,5.1] and Cauchy-Schward inequality [Dud18, 5.1.4] and records in a suitable form the basic observation: the IF th is the direction of perturbing P with the most rapid variation in the functional ((P) for the Fisher-Ras Rao geometry (with the (P(P) metric tensor) of the model P.
LEMMA 3.1. Lei 0 be pathwise differentiable fanctional on P with derivative D0p and IF up for P E P. The IF is the unique solution to the following dual optimization problems:
up arg min Dor[o) [|0lLZ $1}
SEL
α arg min D0 + @
Ap > 0.
(3.1)
The proportionality constant in (3.1) is 1 if and only if the penalty loading Ap 1/2.
In (3.1) we used the duality between constrained and penalized optimization. For the exact represen- tation of vh, both the constraint and the penalty are in terms of the L²(P) norm i.e., the metric tensor of the Fisher-Rao distance on the tangent spaces of P. In other words, the Fisher-Rao metric gives rise to the geometry where the IF is the gradient perturbation.
3.2 Regularized representation
The variational representation and the geometric interpretation of the IF suggest strategy for constructing a regularized approximation of &b as follows. Suppose there is function space c (P) with norm | that quantifies suitable notion of smoothness of functions Φ H. Suppose we wish to find the best approximation of s in H with given degree of smoothness as measures by ||- For example, H can be a Sobolev space. Then the projection thu of sb the ball
BM = (Ø: | S M) c H C L3(P)
of radius M > is the desired approximation, and M controls the degree of regularization. If H is dense in (P)(P,, then we indeed obtain an approximation of db by the projection vay that improves and converges to vp as M + oc.
5

---

LEMMA 3.2. Let be a pathwise differentiable functional on P with derivative D and IF D for P P. Let (H, II-IlH)) be Hilbert space, densely contained in P)) Then the projection of the IF up on the set Bas is the unique solution to the following dual optimization problems:
VP arg min DDrlo)
|| S and ||ôl| M
6EH
arg min 10pl0l +1221000 + 1
(3.2)
6EH
for some regularization loading (MM) 2 0. Furthermore, V'M + & in Lg(P) as M - 0 and, equivalently, 1/> 1/2 in (P) as X + 0.
In (3.2) again used the duality between constrained and penalized optimization. The regular- ized surrogate of db is obtained by strengthening the metric on P and taking its gradient as the approximation of the IF
3.3 Spectral representation
We obtained a regularized functional representation (3.2) of the IF in terms of the evaluation of the pathwise derivative the parameter 0:
d
[0r
(PP)) where dt 1=0
log o pt= Se- Se
dt
and the path {PP Jossse can be taken be any regular parametric model with parameter t and score function e E H at P. This representation of dhp is rather implicit. But it is the correct representation because emphasizes the geometry of the problem and lends to thinking about v as vector in an inner product space rather than a bag of numbers, one for each x E X. Returning to our analogy of computing the gradient of multivariate function on R.3 we make the main observation this paper:
Main idea: The computationally fruitful way of thinking about purtial derivatives of f in P is not along perturbations toward point mass at each x E X, but rather along the directions of an orthonormal basis on the tangent space (g))
Equipped with this intuition, we solve the optimization problem (3.2) analytically to obtain an infinite Fourier series representation of the regularized IF P Our main result is the following
THEOREM 3.3 (Spectral von Mises formula). Suppose there exists an orthonormal busis of functions {e; X + 2} for (P)) and decreasing (Voje; 121 is an orthonormal basis for the Hilbert space H C (P)
sequence of scalars {a, 0} such that
Let S: PP) + H be the linear regularization operator given in diagonalized form by
S[u[(z) = Σ -oylu ()) u L)(P)
(3.3)
and assume that its adjoint operator S* : H. + L3(P) is the inclusion (*(v) = e, so that the inner products of H and L2)() are related by
(Su, v) = (u, S () = (u, v)LP(P) for all u L2(P), v VE H.
(3.4)
Let 0 be a pathwise differentiable functional on P with derivative D0p and IF up () for
P. E P. Then the following representation holds the norm of H:
d
(P,A(I)) lim
((P)))
ez(2)
(3.5)
I+ 2A/0,
1==
where, for each basis function ej, the path + pd can be any regular perturbation of P with the score function 0, log Pi ej.

---

Let's interpret Theorem 3.3 and compare with Theorem 2.1. Formula (3.5) is similar to formula (2.1)
in that expresses the (regularized) IF ph in terms of pathwise derivatives of the functional ((P)
along certain regular perturbations P. of the measure P. So, in order to compute with formula (3.5),
one requires the same numerical tools as for implementing formula (2.1), The main difference is in the choice of the perturbation directions 0 log P. that are employed by the two representations: (i)
In (2.1), the perturbation depends on the evaluation point = E X. By contrast, in (3.5) the scores ej are fixed, once the approximating space H/ and the data distribution P are fixed; and, once the derivatives { [ are computed, an approximation to all values of t/p are obtained. (ii) The directions of perturbation e, depend the approximating function class H. which allows to adapt to the smoothness of tb. (iii) In our proposed implementation, H is taken to be a reproducing kernel Hilbert space (rkHs) of positive semidefinite kernel K, and the scores ej can be interpreted as nonlinear principal components of the measure P. which allows to adapt to its effective dimension. (iv) Perturbation directions ej of P are ordered by the magnitude of the corresponding multiplier sequence G. which leads to a natural low-rank approximation for ς/y by the first r terms of the formula (3.5). (v) The directions 1=, for perturbing P are well-behaved.
Proof. We outline the main ideas of the proof and provide the details in the Appendix. Let J ()) denote the objective function (3.2) and note that it strictly convex, so that the first order conditions are necessary and sufficient for characterizing the function </ E H at which the unique minimum of attained. For a direction v E H, the Gateau derivative of the objective function evaluated the candidate function @ E H is
(Ø) = (v. (P() + (v. LP(P)) + 22(v, 6)
Applying the adjoint relationship (3.4) to express (P) inner products in terms of H inner products,
obtain ((b)) = (v, St) + (v, St) n + 2X の) n, for all Φ, v E H. follows that the H gradient of J (the Riesz representer in the H inner product) is given by J() and that the solution , of (3.2) is characterized by the first order condition (( = 0 E H.
Stb + St + 2X
Using the spectral resolution (3.3) of the smoothing operator S, note that the first order condition is equivalent to the system of equations
o,(,, e,) (P()) + (v' ()) + 2XWx,, e))((()) O, 2 1.
Solving for the Fourier coefficients (bx, LA()) of the spectral resolution of のi in LP)),, and applying Riesz' representation DO(e;] = (w, ()),, obtain formula (3.5).
Regularization in the rkHs of a Mercer kernel
rkHs notation A linear space H of functions o= X + R with inner product o, 4p (0. ()) is an rkHs if (i) it is complete in the norm ||0| = (ò. Φ) of the inner product, and (ii) for every X, the evaluation functional o ((x) is continuous in the norm of H. Convergence of sequence on - é in the norm of H implies pointwise, and often uniform or stronger, convergence.
According to Riesz' theorem [Dud18, 15.5. 1), [SS09. 15.3), for every x E X, there exists function k, E H such that the evaluation functional has the representation d(x)) = (t) kn) h for all é E H. The function K= X²2 - IR given by K (x. .) == (x)) = (ky kz),1 is known as the reproducing kemnel of and is guaranteed to be a symmetric and positive semideñnite map: the matrix [K (xx,, ..,,,,))3 ) is strictly positive definite for all n > 1 and distinct {2,} C X: we call any such function PSD kernel. We call (-) = K(- x)) the slice of K at z. By linearity, any finite superposition of slices x + ΣØ= aikii, (z),, with {r C and {a,} C R and n E N, is in H. Conversely, according to Moore's 's thcorem IPR16, 12.14), [CS08, ch4), any function & E H is a (possibly infinite) superposition of kernel slices; and, any PSD kernel K generates an rkHs of superpositions for which it is the reproducing kernel. In practice, one picks a PSD kernel function and works with the associated rkHs somewhat implicitly because it's norm and inner product are not immediately obvious.
Widely used examples are the Gaussian kernel K(x,8) y) exp(-11x yl3/03). it produces a space of infinitely smooth functions; and the Laplacian kernel K(x.v) = exp( v/l2/o), it produces the space of Sobolev functions with (d + 1)/2 square integrable derivatives. rkHs spaces are used extensively in numerical analysis [Wen04; FM15] and nonparametric estimation [Wai19; Bac24] due to their analytic and algebraic properties. We use a PSD kernel to form an estimator of the IF via the spectral representation (3.5). We assume the following properties, and call any such function Mercer kernel:
7

---

ASSUMPTION 4.1 [Mercer kernel]. (0) probability measure P is absolutely continuous with compact
support X C. R (i) K : X² + a PSD function: (ii K is continuous, bounded, so that its rkHs H C (P)) and with the bound K. == maxzex K (xx, x) < +00; (ii) for every y E X, .x ky dP = 0
so that H C (P); (iv) H is universal in the sense that it is dense in (P))
In particular, the Gaussian and Laplacian kernels are universal; see [SFL11] [CS08, ch4.6). The normal- ization property (ii) can be imposed on any PSD kernel K via one step of the Cholesky algorithm K (z, = (z,, v) Jx hz dP Sx ky dP/ Sx K dPdP is also PSD (PR16,
4.1 Spectral basis
For Mercer kernel K, consider the integral operator SK [T(P) - H CT3(P) with signature K:
Sx(o](>) - / ((s,)) dP(y)
Φ E L3(P).
(4.1)
Under Assumption 4.1, K E (P2(PP P) and S bounded on (g(P)) by the Cauchy-Schward inequality. Operator S can be thought of continuous superposition of slices By of the kernel, and the range of S is properly contained in H. Note that the range of S is the rkHs of the PSD function J' dyp, in particular, depends on the measure P (PR16, ch11].
THEOREM 4.2 [Mercer]. Let K be a Mercer kernel on a compact sample space X and P be a probability measure supported on X. Then there an orthonormal sequence {ej : X' - [R]221 of continuous LP) eigenfunctions of the integral operator S with signature K defined in (4.1), and a corresponding decreasing to zero sequence of eigenvalues, repeated according to multiplicity, {az > 0)i21 such that: (i) (e) is an orthonormal basis for LP(P); (ii) (voje, is
orthonormal basis for H; (iii) S has the diagonalization (3.3), and the adjoint S H - (2(P) is the inclusion operator S* 6 6, in particular, relationship (3.4) holds between the L P) and H inner products.
See [S809. 4.6), [FM09] for background on integral operators and (SS12; Sun05] for extensions of Mercer's s theorem to noncompact domains. In particular, Theorem 4.2 holds for the Gaussian kernel on Rd and a measure P = p Ldd with p E L (RR)), ), as shown in [Sun05, s4].
Example 4.3. For the Gaussian kernel K K(I, y) exp(-e² Ix- 3) on IR and the centered Gaussian weight distribution p(z) = a exp( 2002==22 2²) / / T, the Mercer basis is given by:
e,(x) = 7 6²-² -5'2 H (a
a
oj = V a² + 82
221,EER
+ + a + 82
+
2
where H, is the Hermite polynomial of degree j. and constants 3 (1 + 22//13) % (1/))-1(()))22, 8² = (3² - 1)/2 are defined in terms of the shape parameters and a. In particular, the eigenvalues o, decay exponentially, so that only the first few terms in (3.5) capture most of the variation when up is smooth. Both K and p can be extended to Rd as tensor products.
4.2 Nyström method for integral operators
In order to estimate the IF with a given Mercer kernel K via the spectral representation (3.5), the leading eigenvalues {ay} and the corresponding eigenfunctions {e>}5=1 of the integral operator Sk are required. If P is known, these can be computed numerically (FM15, 12.2.2). If P unknown and only random sample from P is available, these must be estimated statistically. The Nyström method for approximating the eigendecomposition of the integral operator (4.1) is to discrctize the integral with the empirical sum, reducing to cigendecomposition of the empirical Gram matrix Kn = K Xx) /n] This is essentially the well-studied kernel PCA problem to estimate the main nonlinear features of P (Mik ZBOS; Sha+05; RBD10; SS22b; ss22a]. A closely related problem is the functional PCA [Boso0], and the low-rank Gaussian process approximation [VV.08; BRV19; BRV20; ss20). We find the exposition in [RBD10] particularly lueid and follow it closely.
8

---

LEMMA 4.4 [Hilbert space LLN). Let K be Mercer kernel on (X, P) generating the rkHs H and let XI...., Xn be an i...d. sample from P. Let the integral operators TH Tn H + H be defined by
(x) = -,x ( HyyH
H K(x (2.1) dP(y),
(x)d(()) = Σ t kx,)
& H.
Then Tn + TH as n + oo in the Hilbert- norm in probability, and, for each n 2 1,
[TH Tn IH S KVT
(4.2)
n
with probability at least 1 2T
Note that Tū is the empirical analogue of Th obtained by replacing the continuous integral with respect to P by the discrete integral with respect to the empirical distribution of random sample from P. By appealing to a suitable law of large numbers or concentration inequality, it follows that Tr is consistent for Th. Furthermore, Th related to Sk. whereas T, is related Kn. providing a link between the continuous operator Sk and the matrix K2 that is otherwise not immediately clear.
Recall that 5* is the inclusion of H into LT(P), and note that the operators Tи SS and Tk := S S are essentially the same operator with the same action on all functions Φ E H, differing only in the domain of definition and the embedding space of the range. In particular, the eigenvalues of Tx. S and Th are exactly the same and the eigenfunctions of Tx and Tи are related by the inclusion (resp. regularization) operators S (resp. S), in other words the same functions but viewed as elements of L)(P) and H respectively.
For the empirical analogues, the finite-rank operator Tn and the Gram matrix Kn. are similarly related via the Nyström restriction operator R. H + R" given by Rn = (o(X)) Its adjoint R2 In + H is the Nyström extension operator given by [e = - /xxx for = "y't...,,'" v²) Ih" endowed with the inner product (x, y)n == ΣE=1 D y'/nn With this notation, we have T, R,, Rn and K, R₂ R from which it follows that Tn and K₁, have the same nonzero eigenvalues and the corresponding eigenvectors are related via the restriction (resp. extension) operators R, (resp. RT).
The next result an application of the perturbation bound of [Kat87], see also [RBD10], to infer consistency of the spectrum of K,, for the spectrum of TwK S
LEMMA 4.5 [Consistency of cigenvalues). Let K be a Mercer kernel on (X, P) generating the rkHl H and let Xi...., Xn be an i.idd sample from P. Let the integral operator TK LP)() + (P) and the empirical Gram matrix multiplication operator Kn
IR" + +R" be given by
Tk (x) f K(x.g)o(y) dP(w) Kn1y]=
[] K(X... x,) n
y. LE y ER.
i.j=
Let {o;1521 be the decreasing enumeration of the eigenvalues T', repeated according to the multiplicity, and let {ōzbııı denote the analogous enumeration of the eigenvalues of Kn, extended by zero. Then o, + oj as + oo uniformly in probability, and, for each n 2 1,
K/TT
suplo, - 8 < |TK Kn Knllls S f21
(4.3)
and E, o,)² < T K liis x
شق and 1Σ (o, -,,) = (t((TT))tt(([))) & Vn
KYT
with probability at least -e-
Consistency of the empirical eigenfunctions is framed in terms of projections to accommodate the nonuniqueness of an orthonormal basis corresponding to eigenvalues with multiplicity. For N E N, let r(N) denote the total number of eigenvalues, accounting for the multiplicity, corresponding to the leading N distinct eigenvalues 0,(1) > ... > G((N) and let (NN)+1 be the next largest distinct cigenvalue of Tk. Let Hy = span e...... ( C H C L)(P) be the eigenspace of the N leading distinct eigenvalues, and let
r(N)
r(N)
PN H HN, PNiø] = Σ ( vo
Σ (b, S) ΦE H j=l
J=1

---

be its spectral projection. The following perturbation bound was used by [zeos: KGO0]: if Tk is finite-rank estimate of the operator Tk with precision on the order of the Nth spectral gap with | Tollop I (Ør(N) + /4, then the eigenspaces Hy of T and Hy of the leading r(N) eigenvalues of Tx must also be close with || PAy llop S 2/( +r(N)+1]] Tx lop-
LEMMA 4,6 [Consistency of spectral projections]. In the setting of Lemma 4.5, let yi...., yn denote the orthonormal (for the scalar product (:, +)n = (t, +)R= /n so that || vni eigenvectors of the empirical Gram matrix K.:
n
k Σ oily yiduy.. yER..
(4.4)
Then, with y, = ......., ,. E R" 1
1 n 5; n j=1
ēx()) o;
R (x)))
Σ
((,, i=1....,
(4.5)
denote the eigenfunctions of the empirical integral operator Tw (with normalization ll Ivo and [| s 1):
a (x) Lo
de H.
Let N E N and PN (resp. PHx) denote the spectral projection operator on the eigenspace of the leading r(N) eigenvalues T' (resp. Tw) and La denote the identity on Н. Then, for any vp E H,
r(N)
r(N)
Hj) + PN|v = Σ Loyl ej) PAN | = Σ 8z(ر
as n + 00
1=1
1=1
in H in probability, and, if 2 128 T/ON [N+I]3 then
r(N)
32x²r IPPNVo,ejIia (or(N) Or(N)+1)2n
1I (N +
Σ
+N+++
with probability at least 1 2T_
4.3 Kernel von Mises estimator
Consistency of the estimated basis, via a continuous mapping theorem, implies consistency of the pathwise derivatives of 0 evaluated along the estimated perturbation directions. The combination of the estimated basis and the estimated projection coefficients results in a consistent plug-in in estimate of the regularized surrogate IF. We conclude with the following summary of our results:
THEOREM 4.7 [consistency of the kernel von Mises estimator]. Let K be a Mercer kernel on (X, P) gener- ating the rkHs H and let XI.....X. Xx, be an i.i.d. sample from P. Let 0 be pathwise differentiable functional on P with derivative DOp p and influence function up E L)(P) for P E P. Assume that D0p, equivalently v.p, are continuous in P E P in an appropriate sense and that f is consistent estimator of the density of P a compatible notion of convergence. Let {ē,} and {ōz} be the Nyströrm estimators of the eigenfunctions and eigenvalues T'.. For fixed rank 1 < S n and regularization loading > 2 0, let
D8p e,, 22/0, ex(x) x(()=Ex E1+
d 22/0, dt
(x(
éạ(x) lt=0
(4.6)
denote the rank-r approximation of vp and its plug-in in estimator obtained by replacing the unknown o ej, D0ple,] with their estimates. The pathwise derivative can be computed along, e.g., the linear perturbation ft = 1 + téylf since the scores e, and é; are bonded. Then 1| & + 0 as + oc in probability and there exist sequences r(n) + co increasing and X(n) - 0 decreasing such that | (Pllz?(P) + 0 as + co in probability.
10

---

Limitations and future work Our spectral formula (3.5) and kernel implementation (4.6) aim to enable automation of methods based on asymptotic analysis and first-order techniques. These methods, whether carried out analytically or numerically, require theoretical justification and provide approximations that might be more or less accurate in a partieular problem. For statistical inference, this is similar to the validity of bootstrap [Err2]]. It is hoped that the ideas provided here will allow leveraging efficient computation with PSD kernels |RGR15; Sta+20; Che-25), not considered here, for the applications described in the Introduction. However, further theoretical work is required: Downstream tasks, such as debaised machine learning [Che+18], make assumptions on the rate of convergence, e.g., e.g.,((n 11/1,, of the estimator. While we discuss some of the ingredients to study rates for our estimator, we only show consistency here and will investigate the rates in follow-up work. Furthermore, to unlock automation in earnest, further work is required develop efficient estimates of the pathwise derivatives with automatic differentiation. Two strategies are possible: (i) introducing explicit weights wy 1/n for each observation and perturbing this distribution; (ii) perturbing the spatial location of the observation X; and working with the duality of Wasserstein and Fisher-Rao tangent vectors.
Acknowledgements am grateful to Jan- Christian Hütter, Francesca Molinari, Kengo Kato, Kenji Fukumizu, Kyra Gan and Promit Ghosal for stimulating discussions, their support and interest. I thank the anonymous reviewers for their service, valuable feedback and suggestions to improve the exposition.
References
[AGS17]
I. Andrews, M. Gentzkow, and J. M. Shapiro. "Measuring the Sensitivity of Parameter Estimates to Estimation Moments". The Quarterly Journal of Economics (2017).
[AGS20a]
Andrews, Gentzkow, and M. Shapiro. "On the informativeness descriptive statisties for structural estimates", Econometrica 88.6 (2020), pp. 2231-2258.
[AGS20b]
Andrews, M. Gentzkow, and M. Shapiro. "Transparency in structural research". Journal of Business & Economic Statistics 38.4 (2020), pp. 711-722.
[Bac24]
F. Bach. Learning theory from first principles. MIT press, 2024.
[Bae+22]
Bae et al. influence functions are the answer, then what the question?" Advances in Neural Information Processing Systems 35 (2022), pp. 17953-17967.
[BGM20]
T. Broderick, R. Giordano, and R. Meager. "An automatic finite-sample robustness metric: when can dropping little data make a big difference?" arXiv preprint Skiv:2011.1490 (2020). P. Bickel et al. Efficient and adaptive estimation for semiparametric models. Johns Hopkins University Press Baltimore, 1993.
[Bic+93]
[Bic75]
P. Bickel. "One-step Huber estimates the linear model". Journal of the American Statistical Association 70.350 (1975), pp. 428-434.
[Bos00]
D. Bosq, Linear processes in function spaces: theory and applications. Vol. 149. Springer Science
& Business Media, 2000.
[BR07]
V. I. Bogachev and M. A. S. Ruas. Measure theory. Vol. 1. 1. Springer, 2007.
[Bre+19]
W. Brendel et al. "Accurate, reliable and fast robustness evaluation". Advances in neural informa- tion processing systems 32 (2019).
[BRV19]
D. Burt, C. E. Rasmussen, and M. Van Der Wilk. "Rates of convergence for sparse variational Gaussian process regression". International Conference on Machine Learning. PMLR. 2019, pp. 862-871.
[BRV20]
D. R. Burt, C. F. Rasmussen, and M. Van Der Wilk. "Convergence of sparse variational inference
in Gaussian processes regression", Journal Machine Learning Research 21.131 (2020), pp. 1- 63.
[Cat+00]
E. Catchpole al. "Factors influencing Soay sheep survival". Journal of the Royal Statistical Society Series C: Applied Statistics 49.4 (2000), pp. 453-472.
[CC23]
T. Christensen
Connault. "Counterfactual sensitivity and robustness". Econometrica 91.1 (2023), pp. 263 298.
[Cha87]
G. Chamberlain. "Asymptotic efficiency estimation with conditional moment restrictions".
Journal of Econometrics 34.3 (1987), pp. 305-334.
[Cha92]
G. Chamberlain. "Efficiency bounds for semiparametric regression". Econometrica: Journal of the Econometric Society (1992), pp. 567-596.
[Che+18]
V. Chermozhukov et al. Double/debiased machine learning for treatment and structural parameters.
2018.

---

[Che+22]
V. Chernozhukov et al. "Locally robust semiparametric estimation", Econometrica 90.4 (2022). pp. 1501-1535.
[Che+25]
Y. Chen et al. "Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations". Communications on Pure and Applied Mathematics 78.5 (2025), pp. 995-1041. B. Cho et al. "Kernel debiased plug-in in estimation: Simultaneous, automated debiasing without influence functions for many target parameters", Proceedings of machine learning research 235 (2024), p. 8534.
[Cho+24]
[CLV19]
Carone, A. Luedtke, and Van Der Laan. "Toward computerized efficient estimation infinite-dimensiee dimensional models". Journal of the American Statistical Association (2019).
[CS08]
A. Christmann and Steinwart. "Support vector machines" (2008).
[Dud18]
R. M. Dudley, Real analysis and probability. Chapman and Hall/CRC, 2018.
[Efr92]
B. Efron. "Bootstrap methods: another look the jackknife". Breakthroughs in statistics: Method- ology and distribution. Springer. 1992. pp. 169.599.
[FM09]
Ferreira and V. Menegatto. "Eigenvalues of integral operators defined by smooth positive definite kernels". Integral Equations and Operator Theory 64.1 (2009), pp. 61-81.
[FMI5]
G. E. Fasshauer and M. McCourt. Kernel-bused approximation methods using Matlab. Vol. 19. World Scientific Publishing Company, 2015.
[Fra+15]
C. Frangakis et al. "Deductive derivation and suring-computerization semiparametric efficient estimation". Biometrics 7 71.4 (2015). pp. 867-874.
[FS19]
7. Fang and A. Santos. "Inference on directionally differentiable functions". The Review of Economic Studies 86.1 (2019), pp- 377-412.
[Gro+23]
R. Grosse et al. "Studying large language model generalization with influence functions". arXiv preprint arXiv:2308. 03296 (2023).
[Guo+20]
H. Guo et al. "Fasúi: Scalable influence functions for efficient model interpretation and debugging". arXiv preprint 2012. 1578/ (2020).
[Hin+22]
O. Hines et al. "Demystifying statistical learning based on efficient influence functions". The American Storistician 76.3 (2022), pp. 292-304.
[HM95]
J. L. Horowitz and C. F. Manski. "Identification and robustness with contaminated and corrupted
data". Econometrica: Journal of the Econometric Society (1995), pp. 281-302.
[Hub64]
P. J. Huber. "Robust Estimation of a Location Parameter". The Annals of Mathematical Statistics
35.1 (1964), pp. 73 1311
[Hub72]
P. J. Huber. "The 1972 wald lecture robust statistics: A review". The Annals of Mathematical Statistics 43.4 (1972). pp. 1041-1067.
[Hub92]
P. J. Huber. "Robust estimation of location parameter". Breakthroughs in statistics: Methodology and distribution. Springer, 1992, pp. 492-518.
[[H81]
L A. Ibragimov and R. Z. Has" Minskii. Statistical estimation: asymptotic theory. Vol. 16. 1981.
[IN22]
H. Ichimura and W. K. Newey. "The influence function of semiparametric estimators". Quantita- rive Economics 13.1 (2022), pp. 29-61.
[Jo193]
M. A. Jorgensen. "Influence functions for iteratively defined statistics". Biometrika 80 (1993),
pp. 253-253.
[JWZ22]
M. Jordan, Y. Wang, and A. Zhou. "Empirical gateaux derivatives for causal inference". Advances
in Neural Information Processing Systems 35 (2022), pp. 8512-8525.
[Kat87]
T. Kato. "Variation of discrete spectra". Communications Mathematical Physics 111 (1987),
pp. 501-504.
[Ken24]
E. H. Kennedy. "Semiparametric doubly robust targeted double machine learning: review".
Handbook of Statistical Methods for Precision Medicine (2024), pp. 207 207-236.
[KG00]
V. Koltchinskii and E. Giné. "Random matrix approximation of spectra of integral operators" (2000).
[KL17]
P. W. Koh and P. Liang. "Understanding black-box predictions via influence functions". Interna- tional conference on machine learning. PMLR. 2017, 1885-1894.
[KL76]
Y. A. Koshevnik and B. Y. Levit. "On a Non-Parametric Analogue of the Information Matrix". Theory of Probability & Its Applications 21.4 (1976), pp- 738-753.
[Kla87]
C. A. Klaassen. "Consistent estimation of the influence function of locally asymptotically linear estimators". : The Annals of Statistics 15.4 (1987). pp. 1548-1562.
[LCL15]
A. R. Luedtke, M. Carone, and M. J. van der Laan. "Discussion of "Deductive derivation and turing-compuleriization of semiparametric efficient estimation" by Frangakis et al." Biometrics 71.4 (2015), pp. 875-879.
[Luc97]
D. G. Luenberger. Optimization by vector space methods. John Wiley & Sons, 1997.
12

---

[Mad+18]
A. Madry et al. "Towards Deep Learning Models Resistant to Adversarial Attacks". International Conference on Learning Representations. 2018.
[Mik+98]
S. Mika et al. "Kernel PCA and de-noising in feature spaces". Advances in neural information processing systems 11 (1998).
[Mis47]
R. Mises. "On the asymptotic distribution of differentiable statistical functions". The annals
of mathematical statistics 18.3 (1947), pp. 309-348.
[Muk18]
Y. Mukhin. "Sensitivity of regular estimators".. arXiv preprint arXiv: 18050088888 (2018).
[Muk19]
Y. Mukhin. "Geometric methods in econometrics and statistics". PhD thesis. Massachusetts Institute of Technology, 2019.
[Muk21]
Y. Mukhin. On Robusteess of Counterfactualss in Structural Models. Presented at the NeurIPS Workshop on Robustness and misspecification in probabilistic modeling. 2021.
[New90]
W. K. Newey. "'emiparamettric efficiency bounds".Journal Journal applied econometrics 5.2 (1990). pp. 99-135.
[New94]
W. K. Newey. "The asymptotic variance of semiparametric estimators". Econometrica: Journal of the Econometric Society (1994), pp. 1349-1382.
[Pin12]
I. Pinelis. "Optimum bounds for the distributions of martingales in Banach spaces". arXiv preprint 120. 2200 (2012).
[PR16]
V. I. Paulsen and M. Raghupathi. An introduction to the theory of reproducing kernel Hilbert spaces. Vol. 152. Cambridge university press, 2016.
[Pru+20]
G. Pruthi et al. "Estimating training data influence by tracing gradient descent", Advances in Neural Information Processing Systems 33 (2020), pp. 19920-1999999
[RBDI0]
L. Rosasco, M. Belkin, and De Vito. "On Learning with Integral Operators." Journal of Machine Learning Research 11.2 (2010).
[RCRIS]
A. Rudi, R. Camoriano, and L. Rosasco. "Less is more: Nyström computational regularization". Advances in neural information processing systems 28 (2015).
[Roy10]
H. Royden, Real Analysis. 2010.
[Rus87]
Rust. "Optimal replacement of GMC bus engines: An empirical model Harold Zurcher". Econometrica: Journal of the Econometric Society (1987), pp. 999-1033.
[Sch+233
A. Schioppa et al. "Theoretical and practical perspectives on what influence functions do".
Advances in Neural Information Processing Systems 36 (2023), pp. 27560-27581.
[Sem20]
V. Semenova. "Generalized lee bounds", arXiv preprint arXiv:2008, 12720 (2020).
[Sem25]
V. Semenova. "Debiased Machine Leamning of Aggregated Intersection Bounds and Other Causal Parameters". Available at SSRN 5134514 (2025).
[SFL.11]
B. K. Sriperumhudur, K. Fukumizu, and G. R. Lanckriet. "Universality, Characteristic Kernels and RKHS Embedding of Measures." " Journal of Machine Learning Research 12.7 (2011).
[Sha+05]
Shawe- Taylor al. "On the eigenspectrum the Gram matrix and the generalization error of kernel-PCA JEEE Transactions on Information Theory 51.7 (2005), pp- 2510-2522.
[SND18]
A. Sinha, H. Namkoong, and J. Duchi. "Certifying Some Distributional Robustness with Principled International
Adversarial Inneninng.tteerrattoooaal Conference on Learning Representations. 2018.
[SS09]
E. M. Stein and R. Shakarchi. Real analysis: measure theory, integration, and Hilbert spaces. Princeton University Press, 2009.
[SS12]
I. Steinwart and C. Scovel. "Mercer's theorem on general domains: On the interaction between measures, kernels. and RKHSs". Constructive Approximation 35 (2012), pp. 363-417.
[SS20]
A. Solin and S. Särkkä. "Hilbert space methods for reduced-rank Gaussian process regression". Statistics and Computing 30.2 (2020), pp- 419-446.
[SS22a]
B. K. Sriperumbudur and N. Sterge. " "Approximate kernel PCA: Computational versus statistical
trade-off". The Annals of Statistics 50.5 (2022), pp. 2713 2736.
[SS22b]
N. Sterge and B. K. Sriperumbudur: "Statistical 'iiinaatt and computational efficiency of nystrom kereel pea". Journal of Machine Learning Research 23.337 (2022), pp. 1-32.
[Ste+20]
N. Sterge et al. "Gain with pain: Efficiency of kernel-pca by nyström sampling". International Conference on Artificial Intelligence and Statistics. PMLR. 2020, pp. 3642-3652.
[Ste56]
C. Stein. "Efficient Nonparametric Testing and Estimation". Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probabiliry, Volume J: Contributions to the Theory of Statistics. Berkeley, Calif.: University California Press, 1956, pp. 187-195.
[Sun05]
H. Sun. "Mereer theorem for RKHS on noncompact sets". Journal of Complexity 21.3 (2005).
pp. 337-349.
[Vaa00]
A. W. van der Vaart. Asymptotic statistics. Vol. 3. Cambridge university press, 2000.
[Vaa91]
A. W. van der Vaart. "On differentiable functionals". The Annals of Statistics (1991), pp. 178-204.
[Vi103]
C. Villani. Topics in optimal transportation. Vol. 58. American Mathematical Soc., 2003.
13

---

[VR06]
M. J. Van Der Laan and D. Rubin. "Targeted maximum likelihood learning". The international journal of biostatisticy 2.1 (2006).
[VV+08]
A. W. Van Der Vaart, J. H. Van Zanten, al. "Reproducing kernel Hilbert spaces of Gaussian priors". IMS Collections (2008), pp. 200-222.
[Wai19]
M. J. Wainwright. Hight-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge university press. 2019.
[Wen04]
H. Wendland. Scattered data approximation. Vol. 17. Cambridge university press, 2004.
[ZB05]
L. Zwald and G. Blanchard. "On the convergence of eigenspaces in kernel principal component analysis".. Advances in neural information processing systems 18 (2005).
14

---

A Pathwise derivatives and von Mises formula
Tangent space [KL76], [Bic+93, s3.2], [Vaa00, $25.3] and also |Vi003, 38.1.2] At each P E P we consider perturbations to P in P along one-dimensional parametric submodels t + P. P with parameter t E (0,c) and = Pt=u- These perturbations must be smooth and admit infinitesimal directions of perturbations; if we think of {Pi) what is required is that has a tangent vector at P. Let's assume that measures in our model are absolutely continuous P = p. Ldd with density function p with respect to the Lebesgue measure Ldd on Rdd The direction of perturbation P. can then be identified with the time-derivative of the density along the curve ((p((z) for each point X C Rd The kinds of perturbations that relevant for defining the influence function do not change the support of the distribution P. so P, well-defined, and it turns be more convenient mathematically work with the score function
curve through P in the space of probability measures,
δ (x) = d=jt= log (z);
I I C.R
(A.1)
The score function & 2 the tangent vector to the curve Pts the infinitesimal change in P along the curve. The derivative in (A.1), need not hold pointwise, but rather in the Hellinger norm:
lim / [ (vD vD) 2 (())) (x).
(A.2)
The existence of this limit implies that f 6dP = 0 and f oddP +o0. We denote the space of all such function by (P), indicating with the subscript that is the subspace of (P) of all functions that have P-mean zero.
Pathwise derivative A functional P - R is pathwise differeeniable at P (with respect to a collection of paths) if, (i) for given regular path P. the composition t -> ((P:) is differentiable function from [0, <) to at time t = 0; and (ii) there a bounded linear map DBp [P( + R
such that
lim (0(P)) -PP) p0 1-0
(A.3)
for every score function o E (PP) and every admissible path Pa with score 6. The definitions of the score and pathwise derivative are those Riemannian geometry that extends techniques of calculus nonlinear spaces. Notions of smoothness the penultimate paragraph in [VID0, s3.2.3]
nuanced in this infinite dimensional setting,
Inftuence function By Reisz' representation theorem for Hilbert spaces (SS09, 4.5), [Dud18, 5.5.1), for the bounded linear functional D0p (P) >> IR there exists fixed score tpp E T.3(P) such that the action of the derivative D0p on any score v has the following representation in terms of the inner product:
[0r (Ø, 1P))) for all & (P).
(A.4)
The Riesz representer thp of the derivative functional of parameter ((P) is known as the influence function. It has the useful geometric interpretation of the gradient score for parameter 8, i.e., the direction of perturbation to such that the functional changes post rapidly:
D0p[ø] = (Ø, (P() < [1e@lc=ce & L)P)
(A.S)
where equality holds if and only if ct by the Cauchy-Schwarz incquality. If we restrict the norm of the perturbation [[ < as in Lemma 3.1 and use linearity of D0 it follows that the unique maximum in (A.5) is achieved the score vp bp//|ýp||<*(()) and the norm of the influence function || (((())) is the largest sensitivity of 0 to perturbation at P.
A.1 Proof of Theorem 2.1
We begin with a score calculation for the original von Mises [Mis47] calculation with point-mass perturbation. The example shows that this singular perturbation does not have a score function, hence the need to smooth out the point mass in general applications of this technique in [IN22: CLV19].
15

---

Example A.1 Score for the von Mises calculation. Let P and d₂ be a continuous distribution and a point mass on X. Take the path [0, 1] + P. (1 P) + tő for the calculations of von Mises and Huber (Hub72; Hub92]:
lim t [0(P.) - = / up - -1 vp(x)
a (P))
(A.6)
(-0
which cam be made rigorous if upp is, for instance, continuous at æ. However, this perturbation to P is not smooth in the sense of differentiability in quadratic mean (A.2). We compute the tangent vector to P: at f = 1/3 and = O. Take μ = + 6g to be the dominating measure for f1l [x) (z) is the Radon-Nikodym derivative at time The corresponding embedding of Pt into the space of square roots of measures H2 [BR07, o4] is vF((2) = - x1)/ + (2) Note that the path in no longer linear in the embedding space. Also note that drv Pt, Pa+a) = sup,a Pt+ [4] P.[A]] = 2h is continuous in H2. For t t= 1/3, the density (2) can be differentiated pointwise for each X' to find the score function
the path, that fz(2) = (1 1 x
3 204 (=)TT)
2 1/2 (=) + 113) -
1z (=) and verify differentiability in quadratic
2
mean
| 3 - (3)
17 V3 -t--v (-1))(())) -1/2
Pix \ x]
2
3 + / - - 6(w)|a)
+
((1 +
Repeating the calculation with f O, we note that the right derivative of vt is infinite, there is no score function with finite μ-a.e. values that can satisfy (A.2). Consequently, the path P. is not smooth in the Hellinger norm and does not have tangent vector at t = 0.
To remedy the lack of smoothness and extend the von Mises formula (A.6) to all pathwise differen- tiable functionals, the point-mast perturbations must be mollified.
LEMMA A.2 IApproximation to von Mises perturbation with score]. Suppose K is a bounded probability density function on Ie with support in the unit ball xx| < 1. Then
(*(x)= 5 (6) 8>0 0 an approximation the identity in the sense of [sS09, p109), that
(A.7)
(i) Ige (x) dr 1.
(ii) (K S A8-d for all 8 0.
(ii) { | for all 8 >> O and x E IRd.
Here A is a constant independent of
Suppose Po is a probability measure that is absolutely continuous with respect to the Lebesgue measure Ldd with a continuous density function fo- Let
55.²(x) [F (= x) dx
1 (10>))(1)K* x),
(A.8)
then for z E (Jo= on that is fixed throughout). Furthermore,
> O} we have (3) = K²(z - for all sufficiently small 8 > 0 (which depend
(x)
+ (x)
(A.9)
is curve of probability densities with parameter t in interval around 0, that is differentiable in quadratic mean (A.2) at t = O with the score function
(x) = dt 1=
d
log ph.z (x)
(z)²(z) 1.
(A.10)
1==
fo(x)
16

---

The three properties of an approximation to the identity follow respectively from dilation invariance of Lebesgue integral, boundedness and compact support of the kernel K.
Proof.
Fix a E [o > 0}. By the continuity of fo there is neighborhood. N of z such that fo is bounded away from zero on A.. For all 6 small enough, I + ( z) is supported in N by bounded support and dilation construction, so that (' - x) = K (x)) Therefore for negative and close enough to O, function f1,6,2 is a well-defined probability density and its score functions
d
K fo(z)
6 (x)
log p8. (x)
(A.11)
dt
(x)
are bounded in E X. To check (A.2)
2 08, Vo dr ++
lx
fidds vfo
(A.12)
note that the map & f(,,,(z)) continuously differentiable for each in neighborhood t E (-e,e) of 0 with the derivative (01,8,=()) V (T,n,,())) therefore the problem is to justify the change of order of the limit t + and the integral Sx dr in (A.12). By the fundamental theorem of calculus, we can write the difference quotient as
foohE(( folx)) S/" VSonne(z) dh L'' dh Therefore, by (a b)2 < 2a² 2b and Cauchy-Sehward inequality, have the pointwise bound v (x) VTol.)
d
2 500+he(D) fothe dh.
2 (P)
72
J 2 @he,,.. (z) Fhe,s,, dh
1
1
2
+ 2 S8, +2708 (x) fo(x) f((x)
$ l-" 1
2 She (o)) dh + 08, (x))
By the generalized Lebesgue dominated convergence theorem [Roy10, p89. 119), in order to conclude (A.12), it is sufficient to show that fx So (x) fothe dhdr converges t - O. By Fubini's theorem
1 L. Int,8, dh. ( Jothe dhdr S.'' S -02 (0me.::20)) Jotne dadh = 2 2
1
1
Since the scores (A.11) are bounded, the information matrix I,,,, is continuous in at and the above integral converges to Io.6.z-
Proof of Theorem 2.1. By the pathwise differentiability of functional 0, differeentiability in quadratic mean of the path pd and Riesz' representation we have
dt t=0 ((P.i.) D#p[0s.z1
/ vp(() dn. (x)
Assume that (2)))) = 1 (1=00) (2). Below P is fixed and we drop the subscript P for convenience. Using the score Φs computed in Lemma A.2 and the fact that has zero P-mean, have the expression for the pathwise derivative as the convolution of the influence function with the approximation identity kernels:
d
0(p(.))= (p(()) (bp(a)(<<<==>))) ())(==)/((())))--------))))))<)- ap
d
I br(x) (2)))( dz dz
speP
= (wpp K2)(2).
17

---

It suffices to show that for each > 0 and M >> 0 the set
Ea = E sptP; lim sup (Vp ((z)))(p(z) > 20 6-+0
has zero Lebesgue measure, because then E Uド monotonicity, and the assertion (2.1) of the Theorem holds all points E. Thus, we may assume that thp has compact support and therefore belongs to I.² TTT(RR).
{|2| S j}l has zero measure by
Because K is a bounded probability density function, with support in |o| 8 by the dilation construction (A.7), we can write
1
(1bp Ks)(z) - vb((z)
RM prr( - x)) =)--)(=)) Ks(x) dr
<
1/po (z - x) (z) (rr((=)Kon) Ko(w) de
Rd
/
UPo x) u (z) dx.
82 lolso
Fix and recall that continuous functions of compact support are dense in L (I2d) [ss09, p71], so that for each e > we can choose function g with 1|pp - ( < e. By the triangle inequality can upper bound the expression above with
Nd 11100 prlz
/ - 2)
-0(2= -
c x)) dr + zz 10150
g(2 g/2=-00 x) / -g(=) dx c((g(2) bp(2)).
By the continuity of 9 it follows that
lim Td / 1 10(= x)) -x)- -9()) de = 0, for all z.
We find that
lim sup 8-+0
(pp Ks)(2) -bb(2))b)(()) S sdlpp paprrgg/(( () + ((()(() vr(2)]
where the superscript indicates the Hardy-Littlewood maximal function:
(x) sup mE Ld / ((()) dy,
for f e (RR),, T
(A.13)
we set
Fa = fa sptP ; 1pp - gl) > and Ga = {2 E sptP; [x)(()) g(x)l a}
then E,, C Fo U G. by De Morgan's law since ES C Fş G Furthermore, by Chebyshev's inequality
L S _ lúp. glIz.
and by the
maximal inequality (Ss09,
3d 1/ 9l (
Recall that the function g chosen such that || gllenRRo < e, that
L d 3d c+c c
a
Since >> 0 is arbitrary, conclude that Ldd and consequently PIU E1/5 = 0.
18

---

B Spectral representation
B.1 Calculation for Lemma 3.1
The characterization of the influence function as a constrained optimizer was discussed in Appendix A around equation (A.5).
We verify the equivalence of the constrained problem and the penalized problem via an explicit calculation that is simple and instructive for the calculation of the spectral representation. Define the penalized objective function with penalty loading X 0:
(() = D0plui [u] + >xen >pen||u|l22<c>>> u U L.P).
(B.1)
Observer that J is strictly convex on LP(P) by the Cauchy-Schwer inequality. By the strict convexity, the unique minimum of is attained at the tangent vector un [g(P) where the derivative functional of J' vanishes [Lue97]:
DJ, [v] = for e E L)(P).
(B.2)
To compute the derivative of J at some u, fix a direction v E L)(P) of perturbation and compute the difference quotient
J'uu + cv) ((u)= = [D0plu + cul + Apenllu + coll3 [DAp[u] (]] + Xpen bp
= {(u ev, 2)2, + Xpon (u cv, u + cu)2,,}
{(u, 2)2,p + Apen (u, u/2,pb
= 2)2,P + 2Xpen u)2,p) + O(e)).
(B.3)
We find that the gradient (Riesz representer) of the derivative functional of J' at vector is
VJ'(u) w + 2Apen
(B.4)
Using Riesz' representation, the first order condition (B.2) becomes
0 = DJ ug [u] (u. VJ'(w))
= {v thp + 2Apen for all v E (P)))
and conclude that ug
- /22peen is the minimizer of J.
From this explicit solution to the penalized problem (3.1) we see that the direction of solution is always along the influence function, larger penalty loading Apon leads to solution with smaller (P)) norm, and Aen = 1/2 uniquely identifies the influence function.
B.2
Calculation for Lemma 3.2
We define the projection of the infuence function up on the ball Bas the solution of the constrained optimization program. Define the penalized objective function J on (P) with the regularization loading À 2 0:
(u) = [0r[u] + >ppen|uul@, + Areg mu " E T2)
(B.5)
We observe that the constrained problem has the linear objective D0r(v), that the constraints are given by quadratic functionals and that the problem satisfies Slater's condition and that the strong convex duality holds.
The penalized objective Jm is strictly convex and the first order optimality condition
DJW [u] = O, for all e E L
(B.6)
necessary and sufficient. Compute the difference quotient:
(u + ev) - (u)
(D0r(u ev] + Apen + eull2,p + Areg Aregllu+ evl12
- [DD0plu] + Apen boenllul2,rr + Areg [
=((u ev, 1)2,p Apen (u + et, u + eu)2,P + Areg (u + ev, u + ev) {uu, 122, + Apen (u, 4)2,r + Aeelu, u)n} =c(( 122, + 2Xpen(V. u)2.r 22reg (v, u) + 0(2)).
(B.7)
19

---

Take the limit as e - to obtain:
(u) (v, 0)2, + 2Xpen (v, u/2,p + 2>reg (v, u)
(B.8)
From the first order condition, Areg + 0. the optimal solution uo converges to that of the penalized but unregularized objective function .
B.3 Proof of Theorem 3.3
First we check that the relationship (3.4) between the inner products of (P) and H actually follows from the assumptions about the bases. Suppose {e,} } and { are orthonormal bases (ONB) for (P) and H respectively and the operator S LP) + is defined by (3.3). From the definition of the adjoint 5 H (P)
(Sej, 4) (ej S* St)),
all i
(B.9)
By the assumption,
1= (e, 2 e Vo,e,) all
(B.10)
On the other hand, applying the eigenfunction property (B.10) and using bilinearity of the inner product
(e: 2)2, (o,es ei) (Se: ei)
all i.
(B.11)
Similarly, we check for i +
Vol S 2)2, Vo,
0 0= ( Vo,
Se. Vo
all i *
(B.12)
Since 0./0: f and {e:} is complete, follows that ej is eigenfunction of S and in the view of (B.11), the eigenvalue is 1 so that [']e;] must be equal to ej. In other words, S is the inclusion operator + (P))
Next, we use the adjoint relationship (B.9) and (3.4) in the expression (B.8) for the directional derivative of the objective function J:
(u) = (v. vP)2.P + 2U,an(U, u/2,P + (vrez(U, u)H
(B.13)
= (w. Str) + (1 Su) + (V u)Hr.
(B.14)
It follows that the H gradient (the representer in Riesz' representation for Hilbert spaces) of D.,, is given by:
Jn) = S[ + 22pen S(u + 2
(B.15)
= =o 222, + pon>>nn(u ej),,p + 2rreg (u, @))2..e;
(B.16)
With this expansion of the gradient J(). the first order condition
( = O, よ E H
(B.17)
of the penalized and regularized optimization program (3.2) becomes the follow system of equations:
0 = (1b e)) + 20 Ap ())(((P) + 2Xreg (1/x L)) j j21 1.
(B.18)
Solving for the (P) Fourier coefficients of the optimal solution d/x:
(ux. L)) 20 + 2Areg (v, (() j21 1.
(B.19)
Conclude that the optimal solution of (3.2) has the following Fourier series representation
(.A Σ 2Xpen + 2 [wb,, e)) (x)))
(B.20)
Observe that the sequence of (2(P) coefficients is shrunk toward zero by the eigenvalue sequence {a,) and is in fact valid sequence of coefficients for element in H.
Finally, recall that (vr; ()) [e,(e,) and that the penalty loading should be Apen = 1/2 for the correct scaling of the influence function from Lemma 3.1.
20

---

C Nyström method
Our proofs of Lemmas 4.4, 4.5, 4.6 are modifications of [RBD10, Thm7, Prop10, Thm12).
C.1 Proof of Lemma 4,4
Define the sequence of random operators & H H given by
[ = (Ø, kx.) Hxx, Tolo), O E H, i= I,..., n2
(C.1)
We compute the norm of the continuous operator: for any orthonormal basis {6jb22 of the rkHS H
Thr lis
Σ IITTO
J21
1/
(z(2 dP())) dp(x) 2
12
Σ
65(c)h / os(o))a)P(()) "
12
/
Σ
121 x
Σ [
$21
= J [ ((.)))
J21
/ K(x, 9) } ( = || ((P()PP))
where we exchanged the Bochner integral with the inner product by Bochner integrability, used the reproducing property of the kernel, and the standard Mercer expansion of the kernel in the orthonormal basis that converges uniformly, exchanged the sum with the double integral by Fubini's.
Compute the Hilbert-Schmidt norm of the empirical operator, noting that kx, is the eigenfunction of the rank-1 operator:
Hs |lo(X))xx,lls IHH + |TT ll & |K(X,, X,)I + ||]] S 2k.
This norm is an integrable real-valued random variable and therefore & is Bochner integrable with the expectation
ElG/] = / (.
k.)kk dP(r) - Th =
By the strong law of large numbers a random sequences separable Hilbert space (the space of Hilbert-Schmidt operators H in our case) [Bos00, Thm2.4]
Tr, - THllss || Σ
&
+ 0 a.s.
Furthermore, applying the Hoeffding inequality for bounded (in norm, verified above) random elements of separable Hilbert space (the space Hillbett-Schmid operators on H) [Pin12], obtain
26 2T
HS S
(C.2)
72
Vn
with probability at least -T
21

---

C.2 Proof of Lemma 4.5
Applying [Kat87] the empirical operator Tn and the population counterpart operator Tи defined on the separable rkHs H, for any nonnegative convex function d with 4 = 0:
Σ T - o,) - Σ 2(1)) J21
J21
where f79/22 is an extended by enumeration of the eigenvalues of the random operator
B- - A= - Tи Ee
defined in equation (C.1).
We apply [Kat87] with the choice ((8) = =s||" for p 2 1. In particular, with 2, this becomes
(2x)
E); -0,²- 121
b [T THll I
n
with probability at least - TT from the bound (C.2).
Recalling that the sup norn is the limit of the p-norms:
suplo, o,l lim jzl
Σ(o (o, ,,)p
sup p-00c
sup hyy j21
p000
j21
2n 27 nn
Th alap S |T, THllHs I
with probability at least 1-24 2e- from the bound (C.2).
Given ε > 0, E 2ny 27 and solve for obtain = nE. nE22/((x)) Inverting the above finite vn
sample concentration bound, find
suplo -o 10,26, < 2e- 22e 21
as nl + oc.
For the bound on the difference of the traces, compute the trace of the empirical operator:
Σ -r,=t((T,)) tr (K.)
Ki).
J21
Compute the trace of the population analogue: for any orthonormal basis (0,b21 of the rkHS
tr(TH) Σ
(T Φ5)
321
Σ
(5(x)), dP(x), 0b))
12
Σ
(x)) (5) dP(x)
Σ
(z))
12
x
(2P()
X
J1
/
K(z,z)
22

---

where we interchanged the integral f dP with the inner product by Bochner integrability,, applied the reproducing property of the kernel ky. interchanged the sum with the integral by Fubimi's, and used the standard Mercer expansion of the kernel that has uniform convergence.
Define the centered random variables G ( X²) E ( X) supported the interval [H, ki.. and apply the standard Hoeffding inequality (Wai19, eq2.11J:
Σ の, o3 = tr(((Tn)) tr(TH)) J21
Σ S
with probability at least 2e 22e2 1(xx)
C.3
Proof of Lemma 4.6
From (RBD10, prope), for compact positive operators A, B
if A Blpop $ JaN - aN+11/4 then |Pl Pollop S aN+1
||A - Bllop
(C.3)
where og and 0'N+1 are the Nth and (N + 1)st distinct eigenvalues and P is the projection on the eigenspace of the top N distinct eigenvalues of A, whereas P is the projection on the eigenspace of top eigenvalues of B the same dimension. If, in addition, A, are Hilbert-Schmidet
2
if | Bl|Hs [aN aN+1)/44 then |P
PV [|S S
|| BlIHs- oN ax+1
(C.4)
As [RBD10, thm12] point out, bound on the projection onto the eigenspace of simple (multiplicity 1) eigenvalue implies a bound on the eigenfunctions: let J, & be unit-norm and (Ø Φ) > 0, then
|1 21 2(1 - (6, σ) < 2(1 (8, ))1)) = PP -Poliss.
2k 2T/ /v < JoN + 1//4, then by (C.2) 7T,, - Thilss S saN + 11/4 with probability
least 1 -ee and therefore by (C.4)
22
(2n)22T
2
|P - Pv Py 11As <
To TH <
ON -N++1
n [oN N
This event occurs if 2 (1)2
Next, we work with the population orthonormal basis (Ø, Voje,}5=1 for H and extend the population counterpart (d, == ) In == to orthonormal basis for H. This is possible because there are n independent eigenvectors P-a.s. by our assumptions that P. is continuous and K,, strictly positive definite.
23

---

Using Parseval's identity, and then Parseval's again with the projection operators (I - PN) and Pxt
2)
|| | By
PNIlHS
EIl(PAN
HN
Pr)
Σ
((Pix
PN)) ,5)
M(N)
T(N)
(or: क5) 1
Σ
Σ
Σ
(2
M(N+1
+
Σ
Σ
(-oi, Ōi)H
+ Σ 0
52r(N)+1
1
r(N)
r(N)+
Σ
Σ
Kōr: di)
E
Σ
(0i, 25) m
2
(M)++++
12
Σ
Σ ((1- PN)[o,, , bi)e
Σ
Σ (PN(or) 33) 2
+
12r(N)+1
Σ (os, (1 Pw)l
Σ
Σ (or, PM10)))
+
(2)
((N)
Σ
(1 Pa) +
Σ
Px(Ø5]
2
H
S2r(N)+1
r(N)
Σ
(1 (1-
px)
2 +
Σ
Pxl0il
1=1
j=r(N)+1
Note that the bound we obtain a bound terms of the rkHs norm, which implics a counterpart bound for the (P) norm.
C.4 Proof of Theorem 4.7
Fix 2 and À 2 0, for 1, the ô, - og by Lemma 4.5, Assuming for simplicity that the eigenvalues are distinct, || -jll E 0 by Lemma 4.6. Recall that also éj ej uniformly on the compact set X. Assuming f, ff are continuous and f 7 P-a.s. and f/f is bounded on spt(f))
assuming that 3 + f in L (f); then by dominated convergence
(w ()) f Vjjrfaa = o/eff/far
E / Mrej WW,, e))
Conclude that [| - Kll - 0 in P. For r. + co slow enough, also have || 16 in P. Finally, by the universality H, there exists a sequence An - slowly enough such that [[rr()) ( + 0 in P.
((() || + 0
C.5 Toy Monte Carlo experiment
We check our theoretical results with simple numerical experiment. Let 0(P) = Ep(X] be the mean functional. Then vp(x)) = x O(P). We use the Gaussian PSD kernel from our Example 4.3 and set the shape parameter distribution, corresponding to the shape parameter
1. We simulate Monte Carlo data from the standard Normal
1/V2 of our Example 4.3. This allows us to compute the oracle V using Hermit polynomials that we numerically evaluate using the MATLAB code provided with the textbook (FM15]. We estimate the eigenvalues оy and cigenfunctions e, (X.)
24

---

the using Nyström method via MATLAB's eig function. We estimate the pathwise derivatives as n Di= X,,,(X, note this does not take into account estimation of the density and evaluation of the mean functional on the estimated distribution.
Simulation experiments As toy experiment, we compute the oracle low-rank regularization yx and its estimator v., as well as the distribution of the estimation error || xxXlLZ(P)) for the mean functional 0 = E[X] in the setting of Example 4.3.
Regularization A=1e-4
Regularization A=1e-2
2
IF
1=
f=8
0-
fw16
f=16
h=8
-
1=8
-2
r=16
-2
1=16
-4
-4
-4
-2
0
2
-4
-2
Figure 2: Influence function vb, regularized oracle surrogate Φ (dashed) and estimate K (solid).
Regularization bias
Average error
Error distribution
10
101
n-
n=64
10
258
n=256
10
1024,ro64
10
1-16
10
r:32
90
1=64
10
10
10
10
10
10
10
10"
10
10
0.05
0.1
0.15
0.2
regularization loading A
regularization loading A
error
Figure 3: Surrogate bias || 4 (X1122(p) mean integrated squared error Ellei S (m and
distribution of the eтror || ( based on 10 Monte Carlo experiments.
In these experiments we focus on estimating the spectral basis, leaving the development of numerical pathwise derivatives and their estimates to future work. The setting of Example 4.3 allows working with the exact surrogate. We use Riesz' theorem to compute pathwise derivatives as Dø[g] precise.
[ tb gdP either with Monte Carlo or numerical integration, which is considerably Figure C.5 shows the bias-variance trade-off of regularization via the loading X, and the asymptotic concentration of the distribution of the estimator around the oracle surrogate.
25

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The spectral von Mises formula is presented Theorem 3.3. The rkHs estimator of the influence function based on this representation is developed in Section 4.2, and Theorem 4.7 proves its consisteney.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the last parahraph of Section 4.2. Automation considered here requires theoretical justification, which limits its utility in practice. Our analysis does consider rates convergence or efficient/scalable computation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tes, system might not be used reliably to provide closed captions for online lectures hecause fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If a'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Question:
Answer: [Yes]
The main focus of the paper is on theoretical foundations and a significant effort has been made to provide explicit assumptions, correct and complete arguments and explanations, careful citations.
Justification:
26

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross cross-rfferenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the details of the toy Monte Carlo experiment in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, descrihed supplemental material?
Answer:
[Yes]
Justification:
We provide our code for the toy Monte Carlo experiment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines http://hw..cppllllluuuuuooooaaae for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
27

---

The instructions should contain the exact command and environment needed to run to repro-
duce the results. See the NeurIPS code and data submission guidelines (unpssitaugs cosp CodesuhmissioPPolle))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should relcase anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental sefting/details
Question: Does the paper specify all training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details of the toy Monte Carlo experiment in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the full distributions of our Monte Carlo experiments in Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It OK report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric eиror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Apple M2 mac.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn make into the paper).
9. Code of ethies
Question: Does research conducted the paper conform, every respect, with NeurIPS Code of Ethics hmpsoilteenig..ooohiiii Coaieoe
Answer: [Yes]
Justification:
We have reviewed the guidelines and confirm that our practices conform to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
foundational research not tied to particular
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
29

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., ., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, license, copyright information, and terms of use the package should
be provided. For popular datasets, papensittccdeecom sewewittndd..ommluueettt com/dasusess has curated licenses for some datasets.
Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
30

---

Answer: [NA]
Justification:
Not applicable.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ttp::/wwoncccccoooooeoeeeesss)) for what should or should not be described.
31


Reviews:
[{"review_id": "sjMA7cg9Bq", "reviewer": "Reviewer_PMrS", "summary": "In the work the authors introduce an estimator for the influence function $\\psi_P$, for some statistical $\\theta(P)$ with statistical estimator $\\hat{\\theta}(X_1,\\ldots, X_n)$ and the influence function is equal to $\\sqrt{n}(\\hat{\\theta} - \\theta(P)) = \\frac{1}{\\sqrt{n}} \\phi_P(X_i)$ (according to (1.1) in the paper). The influence function is useful to know for a variety of statistical tasks. The influence function can be evaluated for specific points in a fairly straightforward way, but it is useful to have a full functional representation of it. In this work the authors introduce an estimator for the influence function using a sort of RKDE kernel basis (i.e. it uses a truncated bases) to estimate $\\psi$ from data.", "questions": "* I’d really like to see a concrete example—a specific dataset and task—where your method truly enables something new to be done, or clearly outperforms previous benchmarks.\n* Can you elaborate more on the tasks where the influence function is important? Maybe full, concrete example.", "limitations": "Not address", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "It's possible that the topic of the paper is simply too far outside my expertise for me to understand it totally, but I found this paper very hard to follow. I really think it needs to be significantly reworked for the NeurIPS audience. The paper is pretty idiosyncratic, in my opinion. It has no \"Related Works\" section for example, and the core contribution only occurs on the last page and is framed as a Theorem, where it really should be a technique. Some points about this can be found in weaknesses.\n\n## Strengths\nThis paper introduces a new approach to an established task I statistics and thus is of reasonable significance and usefulness. The derivation of their technique is quite extensive, which really precisely and rigorously grounds their technique. Additionally  the work contains several theoretical results, thereby supporting the technique well mathematically.\n\n## Weaknesses\nI really think the style of the paper doesn't fit well with NeurIPS. It has a rather strange style where the paper immediately begins explaining the technique it's presenting from the first page and only finishes on the last page. One would typically expect sections providing background, examples, consequences, etc, but virtually the whole paper is a derivation of the technique. I personally don't think this is a good fit for NeurIPS. That being said, I don't really feel like I grasped the paper well, perhaps it is a highly technical topic that can't really be presented any other way; maybe the other reviewers can shine some light on this. I get the impression that this work may fit better in a more core theoretical statistics venue.\n\nA lot of the mathematical objects are not quite familiar enough that I think they can be introduced without any elaboration or are really beyond what most of the community will understand. Other parts are elaborated on too much.\n* The introduction to the influence function should be more extensive. I think a couple of examples one trivial (mean) and one less trivial would be good. Is (1.1) really the only definition? Any funciton that satisfies this property is the influence function?\n* Concrete applications of the influence function, fully explained, would also be useful to convey the significance of it.\n* Some of the exposition is just way beyond what I can understand reasonably easily, a good example is lines 87-97. It really feels like one would need to be quite familiar with the topic to really get whats going on here\n* Section 3.1 is unnecessary in my opinion. Much of the community is very familiar with RKHS and this kind of background is unlikely to bring the less familiar to a working familiarity of RKHSs to understand what is going on.\n* For papers like these I think its useful for the main text to be a pretty simplified and for the appendix to contain the full technical underpinnings\n* It would be good to have the full core contribution technique explained together without any of the extra exposition, I feel like I'm often losing the tread reading this.  \n### Small Points\n* Line 39: is $\\psi$ without the $P$ subscript something different?\n* line 34: $L_0^2$ is never defined.\n* Line 52: What is pathwise differentiable?\n* line 168: is the $\\partial$ operator different from the $D$ operator in some significant way?\n* Line 224: what is a signature? Is it oveloading the variable $K$? or is it somehow the same thing?\n* Stylistically it's good to refer to Figures in the main text, not just have them stand alone with no reference. (This may even be in the NeurIPS style guide)\n\n\nAgain I want to highlight that its possible I'm just too far out of my area to expertise to appreciate whats going on in the paper, I will be interested to see what the other reviewers say.", "was_revised": true, "final_justification": "I appreciate the author's new introduction, which I think is helpful. Nonetheless I find the style, topicality, and presentation somewhat problematic for NeurIPS along with the weak experimental evidence. This would be a good fit for COLT I think. But the improvements included by the authors are good enough to increase the score by one point to 4.", "rebuttal": "## Organization of the paper\nWe thank Reviewer and acknowledge that our presentation can be\nsomewhat more streamlined, including a more thorough introduction of the\nbackground on the influence function (IF) and discussion of prior work, and we\nwill do this for the camera-ready version.\nWe provide an expanded introduction and motivation at the end of this rebuttal\nas an example and to help with assessing the novelty and contribution of our\nwork.\n## Level of mathematical detail\nRe Paragraph 2 (P2) and Bullet points 3,5,6 (B3,B5,B6) under Weaknesses.\n\nRegarding the suggestion to simplify the presentation in the main text and defer\nall technical underpinnings to the Appendix, we respectfully disagree with the\nreviewer: it is acknowledged in P2 that our problem requires a high level of mathematical\ndetail, and in B3 it is suggested that more rather than less introduction of\ncore concepts is required to communicate the results of the paper.\nThe goal of our paper is to communicate a mathematically novel approach to\nestimating the IF and the ideas that lead us to this result, rather than a\ndemonstration of the application of the results with all details and ideas\nhidden in the Appendix.\nToward this goal, we chose to present our work in a linear fashion, thoroughly\nintroducing the background concepts and baseline solution first in Section 1,\nthe main theoretical ideas of our proposal in Section 2, and a practical\nimplementation of these ideas in Section 3.\nIn fact, Reviewers NjaD and em1K highlighted that the paper is well written and\nsuccessfully communicates our ideas despite the highly technical nature of the\nproblem, and Reviewer ExzA acknowledges that the technical part of the paper is\nwell readable given a general background in analysis and suggests expanding\nthe explanation rather than simplifying it.\nFurthermore, Reviewer ExzA acknowledges that our work is within scope for\nNeurIPS, as can similarly be inferred from Reviewers NjaD and em1K, and Reviewer\n8gPt acknowledges that our work is of good significance for NeurIPS and suggests\nspending significant effort to clarify the background and novelty of our result.\n\nGiven this feedback, for camera-ready version we will do\nbest to move more technical details to Appendix and will focus on\nexpanding Introduction and discussion of our results while still maintaining\nthe overall structure.\n## Necessity of Section 3.1\nWe agree that rkHs setting and notation are familiar to most NeurIPS readers,\nwe included this section mainly to fix our notation that is needed extensively\nfor the rest of Section 3.\nWe will shorten this section according to the recommendation of the Reviewer.\n## More examples/applications\nInfluence functions were discovered in statistics while studying the hardness of\nthe estimation of a given functional and can be used to find the efficiency\nlower bound and to construct an estimator that achieves this bound.\nAn example from Bickel'88 is\n$\\theta_0(P) = E[p(X)]$  with IF\n$\\psi(x) = 2p(x) - 2\\theta(P)$ and an efficient estimator given by\n$\\hat\\theta=\\int\\hat p^2 + 2\\sum_{i=1}^n[\\hat p(X_i) - \\int\\hat p^2]/n$.\nHere, the IF is needed to both construct and check the efficiency of the estimator.\nAnother example are estimands of causal inference, as Reviewer em1K noted.\nHere, the IF of the target parameter is necessary to construct an efficient\nestimator.\nWe will expand our experiments to cover examples of such tasks as described in\nresponse to Reviewer em1K and include $\\theta_0$ example in Section 1.\n\nIF is also necessary to study the robustness of the estimand to the\nperturbations of the model.\nFor example, in ecological population dynamics models, researchers assume that\nthe propensities for capture/recapture and survival follow a logistic model.\nOne can ask if the results, say the estimated population size, based on such a\nmodel are robust to the logistic assumption.\nThis can be assessed by relaxing the logistic functional form baked into the\npopulation model in a way that reveals the bounds on the estimated population\nseize.\nA concrete data application is to construct robustness bounds for the population\nsize of Soay sheep estimated in Catchpole'00.\n## Small points\nOmitted to save space (ask in discussion)\n## Expanded introduction\n\nA classical result is that MLE is statistically efficient—attains the\nsmallest asymptotic variance among “regular” estimators.\nMLE applies to parametric (finite-dimensional) models, where calculations are a\nsimple exercises in calculus:\nthe derivative of the log-likelihood with respect to the parameter yields the\nscore, its variance is the Fisher information, and the inverse of the\ninformation is the lower bound on the estimator variance.\nHere, the score (up to normalization) is the IF.\n\nHowever, parametric models are unrealistic, and extending efficiency theory to\nnonparametric (infinite-dimensional) models took decades to develop, culminating\nin foundational texts [Bic+93] and [Vaa00], and recent literature on DML [Che+22].\n\nIn nonparametric models, there is no canonical parametrization to carry out the\nanalysis. \nInstead, models and estimands (functionals) vary widely in description and often\nrequire problem-specific techniques for estimation and inference. \nFunctional analysis replaces calculus, and solving for efficiency bounds becomes\na mathematically rich and challenging task.\n\nThe central object in this theory is the IF $\\psi_P$, which\ncharacterizes the local sensitivity of a functional $\\theta(P)$ to perturbations\nin the distribution $P$.\nThe norm of $\\psi_P$ gives the lower bound on the variance of regular\nestimators, and $\\psi_P$ is the key ingredient for constructing debiased and\nefficient estimators of $\\theta(P)$.\nDeriving $\\psi_P$ for a given functional is a fundamental task,\nthat can be compared to solving a differential equation:\neach model requires specific methods, and finding new IFs is a significant\ncontribution.\nAnalytic expressions of the IF, as in parametric models, may not even exist.\n\n*How does one estimate the IF?*\nIn parametric models, this is mechanical: \ndifferentiate the log-likelihood, compute the information, normalize the\nscore, plug in parameter estimates.\nRecent work by R Agrawal et al, NeurIPS'24 has automated it via autodiff and\nprobabilistic programming.\n\nIn nonparametric models, this is much more complicated.\nOne typically begins by deriving the analytic form of the IF \nusing pathwise derivatives along a parametric submodel.\nThat is, for a path $P_t$ with known score $g$, the derivative $d\\theta(P_t)/dt$\nis computed,\nthen one hopes to express it as $E[\\phi \\cdot g]$ to match the\nrepresentation of Riesz' theorem.\nThis result of functional analysis says that any linear map,\nspecifically $g\\mapsto d\\theta(P_t)/dt$, must have such representation with some\nfunction $\\phi$.\nThe function $\\phi$ obtained via this representation matching is a candidate for\n$\\psi_P$.\nThe form of $\\psi_P$ often includes unknown components (e.g.,\ndensities, regression functions) that must then be estimated using appropriate\nstatistical methods.\n\n*How does this paper relate to prior work?*\nSeveral recent works—[CLV19], [Hin+22], [Ken24], [JWZ22, M Jordan et al\nNeurIPS'22]—document the challenge of deriving IFs and propose\nmethods for automating this task.\nSpecifically, the goal is to remove the functional analysis derivations\ndescribed above, replacing them with standard data analysis tasks such as\ndensity estimation.\nThese works focus on estimating pointwise evaluations of $\\psi_P(x)$, using\nthe von Mises formula (Theorem 1.1 of our paper; it dates to [Mis47] but\ntechnical details have been iterated on by many authors).\nHowever, these methods provide only isolated values, not a functional estimator\nof $\\psi_P$, and they lack convergence guarantees in function norms needed for\ndownstream tasks like DML.\nBy contrast, we develop a completely novel representation that treats $\\psi_P$\nas an element of a function space and enables consistent estimation in that\nspace.\n\n*What are the contributions of this paper?*\n(I) Spectral representation:\nWe derive a regularized representation (Theorem 2.3) that generalizes von\nMises formula (Theorem 1.1) by replacing pointwise perturbations with an\nexpansion into a basis of a Hilbert space $H$.\nThis yields a formula for all values of $\\psi_P$ simultaneously, rather than\nrequiring a separate derivative calculation per point $x$.\nCompared to von Mises, which perturbs the measure $P$ sharply near each\nevaluation point, our method perturbs $P$ along smooth basis functions,\nwhich is both more numerically stable and aligned with functional approximation\ntheory and practice.\n\n(II) Estimation via rkHs approximation and kPCA:\nUsing our spectral formula, we construct a data-driven estimator of $\\psi_P$ by\ntaking $H$ to be an rkHs, and estimating the basis functions \nvia kernel PCA—a standard and well-studied (in NeurIPS) data analysis task.\nThis construction yields a fully data-driven functional estimator of the IF.\nWe prove consistency of this estimator (Theorem 3.7), and argue that this\napproach is practical, scalable, and grounded in rkHs theory and numerical\nanalysis.\n\n(III) \nThis combination of our spectral representation with kernel PCA enables the use\nof mature and rich theoretical and numerical tools.\nFor theory, it allows the use of results on operator concentration.\nFor computation, it allows the use of scalable kernel methods (low-rank\napproximations, randomized numerical linear algebra, fast kernel matrix\ncomputation).\n\nWhile our paper focuses on consistency, it lays the groundwork for\nanalysis of convergence rates, Gaussian approximations, scalable\ncomputation, and applications to downstream inferences.\nThese are all directions for follow-up work.\n\nTo our knowledge, no prior work provides a data-driven functional estimator\nof the IF or convergence guarantees in rkHs norms,\nnor has it been shown how to leverage modern kernel-based learning and inference\npipelines to estimate the IF."}, {"review_id": "1Ql1nJlmk3", "reviewer": "Reviewer_NjaD", "summary": "This article introduces kernel-based estimators for the influence function, which is well-known in semiparametric and non-parametric statistics. The authors show that their procedure yields asymptotically consistent estimators and they numerically validate their approach on a toy experiment.", "questions": "Key points:\n\n- How do we know that the RKHS $\\mathcal H$ is dense in the tangent space $L_0^2(P)$? Do you implicitly appeal to Theorem 4.26 in [1] and are the assumptions there satisfied?\n- Similarly, in Assumption 3.1: Which definition of universality do the authors use? The usual ones consider denseness in the space of continuous functions.\n- Does the normalization for obtaining (ii) given in line 222 preserve/imply (iii)?\n- The \"extension operators\" $R_n$ seem to be the similar to the usual sampling operators; see [2] and [Lemma A.7, 3].\n\n[1] Steinwart, I. & Christmann, A. (2008). Support vector machines. Springer.\n\n[2] Smale, S., & Zhou, D. X. (2007). Learning theory estimates via integral operators and their approximations. _Constructive approximation_, _26_(2), 153-172.\n\n[3] Sterge, N., & Sriperumbudur, B. K. (2022). Statistical optimality and computational efficiency of Nystrom kernel PCA. _Journal of Machine Learning Research_, _23_(337), 1-32.\n\n\nFurther questions/remarks:\n\n1. The relationship to [1] could be made explicit, or, at least, the reference could be included as from a quick gScholar search this seems to be one of the few NeurIPS publications also considering the influence function.\n2. Line 21, \"foundational role in theory\" => add a few relevant references.\n3. \"Dilated\"/\"mollified\" seems to be used interchangeably. \n4. In line 76, $L_\\text{loc}^1(\\mathbb R^d)$ is not introduced.\n5. In the paragraph starting from line 87, what is the difference of $\\psi$ and $\\psi_P$ (where the latter was used before)?\n6. In line 108, \"kPCA\" not introduced.\n7. In the paragraph starting from line 198, the notations for $K$ and $\\phi$ clash.\n8. Assumption 3.1: absolutely continuous w.r.t.?\n9. The discussion from line 266 could come before Lemma 3.4 to motivate the result.\n\n[1] Kandasamy, K., Krishnamurthy, A., Poczos, B., & Wasserman, L. (2015). Nonparametric von Mises estimators for entropies, divergences and mutual informations. _Advances in Neural Information Processing Systems_, _28_.\n\nA few notes on typos/grammar:\n- In line 43, add $\\hat \\cdot$ to $\\theta'$ and $\\theta''$.\n- In line 53, \"is an absolutely\".\n- In line 100, check grammar/formulation.\n- In my opinion, \"rkHs\" should be \"RKHS\". The former looks weird.\n- In line 174, add \"RKHS; recalled below in Section 3.1\" or something similar.\n- The section titles in Section 3 have full stops at the end.\n- In line 308, capitalize \"consistency\".\n- In line 312, typo in \"Nyström\".\n- In line 317, typo in \"bounded\".\n- In line 328, \"describe\" => \"described\".\n- In line 328, typo in \"debiased\".", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Very well written article, bridging classical statistical theory and reproducing kernel Hilbert space methods commonly employed in machine learning.\n### Weaknesses\n\n- No finite sample guarantees. As classical theory [1] considers asymptotic results only, are there existing results to obtain rates, as hinted at in line 110?\n- The results apply to compact sample spaces only. While the authors hint at possible generalizations [2], their theory does not take these adjustments into account.\n\n[1] Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge University Press.\n\n[2] Steinwart, I., & Scovel, C. (2012). Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs. _Constructive Approximation_, _35_, 363-417.", "was_revised": true, "final_justification": "As in my original review, I think this is a very well written article, bridging classical statistical theory and reproducing kernel Hilbert space methods commonly employed in machine learning. While I see the concerns of another reviewer w.r.t. the topic being niche at NeurIPS, I also find the article to be a valuable contribution to machine learning theory and NeurIPS has published works on (and methods using) the influence function in the past.", "rebuttal": "We thank the Reviewer for the careful reading and insightful interpretation of\nour work, valuable and encouraging feedback.\nIn particular, we are delighted that the Reviewer found our work to be well\nwritten and connection between classical statistical theory and rkHs methods\ninteresting.\nWe also thank the Review for graciously catching and reporting our typos.\n\n### Weaknesses\n\nIt would be helpful to clarify which finite sample guarantees the Reviewer is\nasking about?\nIt is a basic limitation of all statistical methods that use the influence\nfunction to be approximate and asymptotic, as we explain in Limitations and\nfuture work paragraph.\nHowever, we do provide finite sample concentration bounds for estimation of the\neigenfunctions and eigenvalues, and in principle this analysis can be extended\nto establish finite sample bounds for our estimator.\nPlease let us know if further clarification is necessary.\n\n### Key points\n\nWe require that the rkHs $H$ be dense in $L^2$, it then, indeed, follows readily\nthat the normalization (ii) produces a subspace that is dense in $L^2_0$.\nSince continuous functions are dense in $L^2$, any definition of universality\nshould work for our estimator, but what we require precisely is that any score\nfunction can be approximated as stated on line 219.\n\n### Further questions/remarks\n\nThank you for pointing out reference [1] in NeurIPS on estimating the influence\nfunction, this is indeed a great reference to cite in our paper and discuss in\nour introduction.\n\nline 21: we will be expanding the introduction to be more accessible to a wider\naudience per feedback of multiple reviewers.\n\n\n$L^1_{loc}$ denotes the space of locally integrable functions.\n\n$\\psi$ refers to the influence function $\\psi_P$, we drop the subscript for ease\nof notation, but realize that it is confusing for some of the reviewers and will\nalert the reader to this in the camera-ready version.\n\nline 108: we will replace kPCA with kernel PCA.\n\nline 198: we use $\\phi$ as a generic function of $H$ in this paper, so the\nnotation should be fine, but we realize that $\\phi$ is commonly used to denote\nthe canonical features in rkHs references and may be confusing for that reason.\nWe will rethink our choice of notation.\n\nAssumption 3.1: absolutely continuous with respect to the Lebesgue measure.\n\nline 266: this is a valid point, we will expand the discussion before Lemma 3.4\nto motivate the result for our problem."}, {"review_id": "eNqRfxpdhV", "reviewer": "Reviewer_8gPt", "summary": "This work studies estimating the influence function of functionals that are asymptotically linear. The work introduces an estimator for the influence function by approximating changes in the influence function from the derivatives of a kernel estimator. The authors prove that their estimator is consistent and provide some experimental demonstration of their approach.", "questions": "1.\tHow strong is this assumption that our estimators are asymptotically linear? \n2.\tHow can the estimators error be represented by contributions of individual data points in a way that is independent of the total number of samples? Is this an assumption as well? If so, how is this assumption motivated?\n3.\tCan influence functions in sample distribution space be connected somehow to influence in input space? I.e. instead of looking at perturbations in the distribution of the data, what if you examined the influence of perturbing a sample in a particular direction in its ambient space.", "limitations": "The biggest limitation of this work is that the majority of the NeurIPS audience will likely have difficulty appreciating this result. I recommend spending significant effort to clarify the background and novelty of their results, as well as possibly expanding the experimental section to include comparisons to previous methods. If previous methods don’t exist or are unimplementable for some valid reason, then an explanation of this fact is sufficient.", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I am not familiar with the literature, so I may be incorrect on some of these criticisms and strengths. If the authors can address my confusion, I may be willing to raise my score.\n\nStrengths:\n\n-\tThe problem of estimating the influence function seems fundamental and well-motivated.\n\n-\tTheir approach seems novel and theoretically grounded.\n\n-\tSome experimental validation of their method is provided.\n\nWeaknesses:\n-\tI am having difficult placing the novelty of the result in the context of prior work. How has the influence function been estimated in prior works and how does this approach compare?\n\n-\tThe plots are largely unclear. What are the axes labels? What should the reader take away from these plots? How should the reader interpret the magnitude of the errors in the estimate of the influence function? What about functionals other than the mean?\n\n-\tIt would be good to provide some explanation of what the ‘Von Mises’ formula is for the reader.\n\n-\tIt is not clear to me how to compute the derivatives of your functional. This needs to be clarified, perhaps with an example.", "was_revised": true, "final_justification": "I found it difficult to judge the novelty and significance of the contributions in this work. I think the style, content, and format of the work may be a better fit for other more statistically oriented venues and not the broader NeurIPS audience.", "rebuttal": "Thank you for the careful reading and thoughtful feedback, and for\n acknowledging that our problem is fundamental and well-motivated.\n\n### Novelty and Relation to Prior Work\nWe agree that the Introduction could make the connection to prior work more explicit, and we will revise it accordingly for the camera-ready version. Below we expand on this connection to clarify the novelty and contribution of our work.\n\n\n*What is the influence function (IF) and why is it important?*\n\nIt is a classical result that the MLE is efficient—it attains the smallest possible asymptotic variance among regular estimators. In parametric models, this is a simple exercise in calculus: the score function (the derivative of the log-likelihood) has variance equal to the Fisher information, and its inverse provides the lower bound on estimator variance. The score, up to normalization, is the influence function.\n\nHowever, in nonparametric models, there is no canonical parametrization to carry\nout the analysis, necessitating more advanced theory [Bic+93,Vaa00].\n\nIn nonparametric models, however, there is no canonical parametrization, and one must resort to more advanced theory [Bic+93, Vaa00]. The key object in this theory is the IF $\\psi_P$​, which quantifies the sensitivity of a functional θ(P) to local perturbations of the distribution P. Its L2-norm gives the lower bound on the variance of any regular estimator and is central to constructing efficient estimators. Deriving $\\psi_P$ is a fundamental task, analogous to solving a differential equation—each model requires distinct analytical tools, and obtaining a new influence function is a substantive contribution.\nClosed-form expressions may not even exist.\n\n*How does one estimate the IF?*\n\nIn parametric models, this task is straightforward: differentiate the log-likelihood, compute the information, normalize the score, and plug in parameter estimates. Recent work by R. Agrawal et al. (NeurIPS'24) automates this process using autodiff and probabilistic programming.\n\nIn nonparametric models, this is much more complex.\nOne typically begins by deriving the analytic form of the IF \nusing pathwise derivatives along a parametric submodel.\n\nOne typically begins by computing the derivative $d\\theta(P_t)/dt$\nalong a parametric submodel with known score function $g$ and then attempts to express this derivative as\n$E[\\phi \\cdot g]$ to identify $\\psi_P = \\phi$ via Riesz representation.\nThe form of $\\psi_P$ often includes unknown components (e.g.,\ndensities, regression functions) that must then be estimated.\n\n*How does this paper relate to prior work?*\n\nSeveral recent works—[CLV19], [Hin+22], [Ken24], [JWZ22, M Jordan et al\nNeurIPS'22]—note the challenge of deriving IFs and propose\nmethods for automating this task.\nThese works aim to bypass the functional analytic derivation by replacing it with standard statistical tasks like density estimation.\nHowever, these methods only yield pointwise evaluations $\\psi_P(x)$,\nnot functional estimators, and they do not offer convergence guarantees in function norms—guarantees that are crucial for downstream tasks like debiased machine learning (DML).\nBy contrast, we develop a novel representation that treats $\\psi_P$\nas an element of a Hilbert space and construct a data-driven estimator of the entire function,\nwith convergence guarantees in L2 and rkHs norms.\nWe are not aware of any prior work that achieves this.\n\n*Contributions of This Paper*\n\n(I) Spectral representation:\nWe derive a regularized representation (Theorem 2.3) that generalizes von\nMises formula (Theorem 1.1) by replacing pointwise perturbations with an\nexpansion in a Hilbert space $H$.\nThis yields a formula for all values of $\\psi_P$ simultaneously, avoiding calculation per point $x$.\nCompared to von Mises, which perturbs the measure $P$ sharply near $x$, our method perturbs $P$ along smooth basis functions,\nwhich is numerically stable and well-aligned with functional approximation\ntheory.\n\n(II) Estimation via rkHs approximation and kPCA:\nWe construct a data-driven estimator of $\\psi_P$ by\ntaking $H$ to be an rkHs, and estimating the basis functions \nvia kernel PCA—a well-studied (in NeurIPS) data analysis task.\nThis yields a fully data-driven functional estimator of the IF.\nWe prove consistency (Theorem 3.7), and argue that this\napproach is practical, scalable, and grounded in rkHs theory and numerical\nanalysis.\n\n(III) This combination of our spectral representation with kernel PCA enables the use\nof mature and rich theoretical and numerical tools.\nFor theory, it allows the use of results on operator concentration.\nFor computation, it allows the use of scalable kernel methods (low-rank\napproximations, randomized numerical linear algebra, fast kernel matrix\ncomputation).\n\n\nWhile we focuses on consistency, we also lay the groundwork for\nfurther analysis of convergence rates, Gaussian approximations, scalable computation, and downstream applications.\nThese are all directions for follow-up work.\n\nTo our knowledge, no existing method provides a data-driven functional estimator\nof the IF or convergence guarantees in rkHs norms,\nnor has it been shown how to use kernels to estimate the IF.\n\nAs Reviewers em1K and ExzA note, our approach rethinks the discretization of the IF—shifting the paradigm.\n\n### Plots, briefly\n\nWe will expand the section on experiments in the camera-ready version accordingly. \nIn the following, we provide some more details.\n\nTop row:\nthe IF (straight line), the regularized surrogate (smooth lines)\nand estimates (jagged lines).\n\nThe estimates are consistent for the surrogates rather than the IF and the\ndegree of regularization (via $\\lambda$) controls bias (less/more on the left/right\nplots).\n\nThe estimates based on 8-16 basis functions and are\nvisually indistinguishable from targets.\nThe takeaway is that very few basis functions can be sufficient to form a good\nestimate.\n\nBottom row: the distribution of the error $||\\hat\\psi-\\psi||$\nfor sample sizes of $256$ and $1024$ shows that with more data, the mean\nintegrated squared error (MISE) and the distribution of error both shift toward 0,\nas guaranteed by our result.\nOn the x-axis is the value of the error and on the y-axis is the frequency of\nthat value.\n\nIn order to judge the magnitude of the error, an efficiency theory for this\nfunctional estimation problem is required.\nWe will investigate the best possible rate of convergence in follow-up work.\n\nPer your suggestion (and that of PMrS),\nwe are including an example of a non-trivial IF in Section 1.1\nand the simulations.\nA good choice is the average density functional from Bickel'82,\n$\\theta(P) = E[p(X)^2]$ with $\\psi_P(x) = 2p(x) - 2\\theta(P)$, where $p$ is the\ndensity of $P$.\nThe simulations results are qualitatively the same for this functional (also for\nthe median whose IF contains a jump discontinuity, the entropy and mutual\ninformation, Gini coefficient and Lorenz value B functionals).\n\n\n### Computation of pathwise derivatives\n\nRegarding computation of derivatives $d\\theta(P_t)/dt$,\nwe claim no novelty or contribution here, but agree with the reviewer and will\nclarify how this is done in our and prior work.\nThe most basic approach is to compute the finite difference\n$[\\theta(\\hat f_t) - \\theta(\\hat f)]/t$\nfor a small $t$ and density estimates $\\hat f$ for the data and \n$\\hat f_t = [1+t\\hat e_j]\\hat f$ for the perturbation toward basis function\n$e_j$.\nA better approach would use automatic differentiation, but this is functional-\nand implementation-specific and outside the scope here.\n\nRegarding providing more explanation of the von Mises formula, we would love to\nimprove,\nand kindly ask Reviewer to clarify exactly what would be helpful to explain?\nWe outline how the formula works via the proof sketch to Theorem 1.1 and\nspend lines 79-83 interpreting the formula toward our estimation task.\nMore input about what is not clear or not covered would help us,\nand we hope to iterate on this in the discussion.\n\n### Asymptotic linearity\n\nIt is an assumption on both the functional $\\theta(P)$ and the estimator\n$\\hat\\theta$,\nbut not a restrictive one.\nFor $\\theta$ to have an IF it must vary smoothly when $P$ is perturbed.\nFunctionals that arise through taking averages with respect to $P$ are smooth,\nbut the density $\\theta(P)=p(x_0)$ at a fixed point is not smooth.\nConsequently, averages are easier to estimate than density values.\n[Vaa91] shows that existence of IF for $\\theta(P)$ is equivalent to the existence\nof an estimator $\\hat\\theta$ that converges at the fast parametric root-n rate.\nSo, if it is expected that $\\theta$ can be estimated at the roon-n rate, then it is\nexpected to have an IF and admit regular estimators that satisfy (1.1).\nNot all estimators of smooth functionals are regular, e.g. shrinkage estimators\nare not.\nOne can construct regular estimators of the average of a regression function\nthat is estimated with shrinkage as in DML.\n\nRegarding the error contribution of a single $X_i$, it does depend on $n$ in\n(1.1) and is approximately $\\psi_P(X_i)/n$.\nE.g., for the sample mean, the error of an observation is $(X_i-\\theta)/n$ for\nany $n$, and (1.1) states that for smooth functionals, there exists estimators\nfor which a similar representations holds asymptotically.\nPlease let me know if further clarification is needed about this.\n\n### Transport perturbations\nThe IF describes the change in $\\theta$ when the probability weight of $x$ is\nperturbed.\nThere exists a related object, mentioned on line 90, called the Wasserstein\ngradient vector field (WGVF), that describes the change in $\\theta$ to perturbations of\n$P$ by transporting the particles of mass continuously in the sample space.\nThe WGVF is related to the influence function via\n$\\nabla_W \\theta(x) = D_x \\psi_p(x)$\nand describes the sensitivity of $\\theta$ to perturbations of the location of mass\nat $x$ in the sample.\nIn fact, there is a one-to-one correspondence between the IF and the W gradient."}, {"review_id": "b6tNI11vuA", "reviewer": "Reviewer_em1K", "summary": "The paper proposes a novel way of deriving the influence function of a target parameter via the Riesz representer of its derivative operator. Previous results had focused on point-wise derivations of the influence function (von Mises formula, Theorem 1.1 in the paper), which allow for point-wise approximations of the influence function. In contrast, the ideas presented in this contribution open the door to simultaneous  approximations to all values of the influence function, given that the approximation is conducted via the influence function spectral representation. \n\nThe overall idea builds on an approximated variational representation of the influence function: the influence function can be exactly represented as a functional optimization program; this program can be further regularized with a Sobolev norm to obtain approximate solutions. In order to do so, the spectral representation of the regularized functional is exploited. In particular, if this Sobolev space is a Mercer RKHS, then the solutions are actionable in practice. The consistency of the kernel von Mises estimator is established.", "questions": "My questions have been posed as the two main weaknesses above. I believe they are concise. If they are addressed satisfactorily, I will raise my score.", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The paper is well written overall, reading very well despite being technical in nature. The submission is technically sound, with all the claims being theoretically supported. In this sense, the authors also clearly state the main technical/theoretical limitation of the work: they only prove consistency of the estimator, without studying the convergence rates of it. However, the idea seems original and powerful enough so that consistency of the estimator suffices for a first paper on this idea.  While going from point-wise evaluations of functions to spectral approximations of such functions is not extremely original by itself, the combination of these ideas with the variational representation of the influence function makes the contribution non-trivial and original. \n\n**Weaknesses**: While the originality of this paper is not under question, I think these spectral approximations are not substantially significant until proven practically useful. Based on this comment, I see two main flaws in the current version of the paper. \n\n- First, I would like to see a discussion on how easy/computationally feasible it is to obtain the pathwise derivatives $D\\theta_P$ on the eigenfunctions $\\hat e_j$ in Section 3. Do these need to be numerically approximated themselves? Overall, what is the complexity of the whole procedure?\n- Second, the simulations are only run with $\\theta$ taken to be the mean functional, arguably the easiest possible functional. I would like to see the performance of these approximations in more challenging/real scenarios. For example, given the importance of influence functions in causal inference and their connection to doubly robust estimators, the authors could take $\\theta$ to be the average treatment effect of some causal experiment, and compare the results using the approximate influence function of this paper with the usual plug-in  or inverse probability weighting estimators (as if the doubly robust estimator was not known). I would also encourage the authors to run a simulation on a causal target whose influence function is actually unknown, and compare the results obtained with the approximate influence function obtained using the techniques presented in this paper with some naive estimator of such a causal target.\n\n**Minor comments**: \n- Discuss the choice of $r(N)$, both theoretically (does Lemma 3.6 hold for any definition of $r(N)$?) and practically (are practitioners encouraged to used r(N)-rank approximations in practice?).\n- It almost seems like the authors run out of space. Theorem 3.7 is hanging there without any context or explanation. Please make some space for explaining what Theorem 3.7 is doing there and its connection to previous results (I do understand it but it does not read well). \n- The definition of $D\\theta$ first appears in Section 2.3, but the concept is already used in Equation (2.2). Please make the definition appear before it is used. \n- The plots are too small, especially the legends.", "was_revised": true, "final_justification": "The paper is well written overall and technically sound. The main contribution seems original and powerful enough for the paper to be accepted. However,  and as explained in my review, I think the contributions of this paper are not extremely significant until proven practically useful. Based on the rebuttal, I trust that the authors have run further experiments, but it is hard to judge without seeing the actual results. Thus, I believe a borderline accept is a sensible score on my end.", "rebuttal": "We thank the Reviewer for the careful reading and insightful interpretation of\nour work, valuable and encouraging feedback.\nIn particular, we are delighted that the Reviewer found our paper well-written,\nand our variational characterization, its spectral solution and\nits implementation with rkHs projections to be original and nontrivial results.\n\n### Computational feasibility\n\nAs with the baseline method of [CLV19, JWZ22] that are based on the von Mises\nformula (Theorem 1.1),\nour spectral estimator of the IF requires evaluation and estimation of pathwise\nderivatives $d\\theta(P_t)/dt$.\nWe claim no direct contribution to this subtask.\nPrior works use the finite difference\n$[\\theta(\\hat f^{\\delta,z}_t) - \\theta(\\hat f)]/t$\nwith \n$f^{\\delta,z}_t(x) = (1-t)f(x) +t K^{\\delta,z}(x)$\nfrom line 60 in the paper being a mixture perturbation of the density $f$\ntoward the mollified point mass at $z$.\nWe follow the same basic strategy and approximate $d\\theta(P_t)/dt$ with a\nfinite difference\n$[\\theta(\\hat f^j_t) - \\theta(\\hat f)]/t$\nwhere\n$\\hat f^j_t = [1+t\\hat e_j]\\hat f$\nis the perturbation toward the $j$th eigenfunction.\nAn indirect advantage of our approach to this basic strategy, is that the scores\n$e_j$ for our differences are well-behaved rkHs functions (e.g. Hermite\npolynomials) and our differences are numerically stable.\nBy contrast, the former approach requires a careful tuning of the mollification\nparameter $\\delta$ and leads to finite differences that are ill-conditioned [JWZ22,\ndirect communication with the authors].\n\nSo, the cost here is that of density estimation (complexity and accuracy vary\ndepending on the method), kernel PCA, and evaluation of the functional on the\nperturbed density, say $C$.\nIf we use kernel methods for density estimation, and use Nystr\\\"om\napproximations for both the density and kPCA estimation, then the\ncomputational complexity is $O(nr^2 + rC)$.\n\nA better approach would be a composable numerical implementation of the\nfunctional evaluation,\nthe density estimate,\nand the kernel PCA that would allow automatic differentiation with respect to\n$t$ instead of finite differences.\nIt is out of scope for this paper but would be an interesting topic for\nfollow-up work.\n\n### More challenging/real scenarios\n\nWe acknowledge that expanding our simulation experiments to include more\ninteresting estimation targets would greatly aid the impact of the paper.\nWe have experimented with the following more interesting functionals:\nthe average density functional from Bickel'88,\n$\\theta(P) = E[p(X)^2]$ with $\\psi_P(x) = 2p(x) - 2\\theta(P)$, where $p$ is the\ndensity of $P$;\nthe median/quantile functional whose influence function contains a jump discontinuity;\nthe entropy and mutual information functionals;\nthe Gini coefficient and Lorenz value B functionals.\nThe simulations results are qualitatively the same for these functionals as\nthose for the mean, we obtain visually good estimates with very few spectral\nterms, with the exception of the median whose Fourier series converges slower\ndue to the discontinuity. \n\nWe fully agree with the Reviewer that applying our method to estimation of\naverage treatment effect (ATE) functionals would be a powerful test of the\ntheory and an impactful demonstration of significance of our work.\nA good baseline for such experiment is the Kernel Debiased Plug-in\nEstimator (KDPE) of [Cho+24] which implements the debiased estimators of\ntreatment effect functionals without requiring the influence function as input\nand compares them with the well-known Targeted Maximum Likelihood estimator\n(TMLE) that requires the influence function as input.\nThe paper considers both a straightforward experimental setup as well as a\nlongitudinal design.\nAs [JWZ22] and [Cho+24] point out, working out the influence function and an\nefficient estimator in the longitudinal setting is rather complicated, making\nfor a convincing case for automated estimation of the IF and debiasing.\n\nFor the camera-ready version of the paper we will include experiments that will\ncompare the accuracy in downstream tasks of estimating functionals ATE, RR, OR\nfrom page 8 in [Cho+24]\nusing TMLE with the analytically estimated IF, TMLE with the IF estimated using\nour method, and KDPE that does not require the IF.\n\n### Choice of $r(N)$\nTo fully address the choice of the number of basis functions $r(N)$ and the\nchoice of regularization parameter $\\lambda$, a finer analysis of the rate of\nconvergence of the estimator is required.\nOne can further consider the trade-off between statistical precision and\nnumerical complexity.\nFor Lemma 3.6, the finite-sample concentration bound requires that \n$n> 128\\kappa^2\\tau/[\\sigma_N - \\sigma_{N+1}]^2$ the sample size be larger\nenough relative to the spectral gap.\nIn our simulations we have found that as few as $r=8$ terms are sufficient to\napproximate the overall shape of $\\psi_P$ visually and we expect that the\npractical choice $r$ to be relatively low in applications.\n\n### Context of Theorem 3.7\nWe completely agree and will invest significant effort in improving the\nexposition throughout the paper for the camera-ready version.\nIn particular, we will expand the explanation, context and interpretation of\nTheorem 3.7.\n\n### Definition of $D\\theta$\nThank you for pointing this out, our intention was to introduce $D\\theta_P$ in\nthe outline of the proof to Theorem 1.1, but we realize from your and Reviewer\nExzA feedback that we didn't succeed.\nWe will address this in the camera-ready version.\nAny further feedback would be much appreciated.\n\n### Clarity of plots\nPer your feedback and similar from other reviewers, we will be expanding the\nexperiments section and will include more explanations and update the plots to\nbe legible for the camera-ready version."}, {"review_id": "cJKzExOk8R", "reviewer": "Reviewer_ExzA", "summary": "The paper considers a way to compute the first variation of a statistical estimator in the Fisher-Rao geometry at some probability measure $P$. To this end, the authors first represent the first variation as steepest descent direction among all (normalized) variations. To make the problem tractable, they introduce a regularization of this variational formulation by some Hilbert space norm. In this case the (regularized) first variation can be expanded into an eigenbasis of the Hilbert space.\n\nIn practice, the Hilbert space is chosen as RKHS of a Mercer kernel and the eigenbasis as the principal components of the measure. For this case the authors prove that the computation of the eigenvalues of the kernel matrix still leads to a consistent estimator of the eigenvalues of $P$. As a final result they prove in Thm 3.7 that the combination out of cutting off low eigenvalues and estimating the principal components from data still provides a consistent estimator.", "questions": "see strenghts/weaknesses", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## General comment\n\nI guess, the paper will be very niche for NeurIPS, but I would see it within the scope of NeurIPS such that there is not too much of a problem.\n\n## Strengths\n\n- I find the concept of discretizing the influence function in terms of eigenfunctions instead of using a spatial discretization quite nice.\n\n- The authors provide a theoretical foundation of their regularization and approximation method including convergence (consistency) results. The technical tools (regularization with an RKHS norm, spectral decomposition) are well-established and quite likely to work.\n\n## Weaknesses\n\n- A critique which might be quite common for theory papers: It is not clear whether the proposed framework works well in practice. There is only one very small 1D example in the paper. It would be beneficial to implement the method in a more serious setting and compare the error to the point evaluation method mentioned in the introduction (which will be of course more expansive when evaluating more points, but is it more accurate?). How does it scale with the dimension of the probability measures (which often is a problem for kernelizations)? Given the theoretical nature of the paper, I wouldn't weight this critique to much, but given that the paper claims to propose a faster and more practical method than previous papers, at least some evaluations could be included.\n\n- I think that the authors could invest a bit more efforts in writing and explaining their results. While the technical part of the paper is well readible (given a general background in analysis), the explanations in the introduction and limitations/future work are very sparse and I am not fully clear what the authors mean even after reading the paper.\n\n- Similarly, the paper would benefit from a how-to section in the paper (potentially in the appendix): That is, a section which says, given data points $x_1,...,x_n$, what do I have to do to get the (approximated) influence function.\n\n- The approximation procedure is specific to the choice of $P$. So if I want to compute the first variation of the same estimator for different base distributions this might be restrictive.\n\n- It remains unclear to me how much bias is introduced by the regularization and how this depends on the kernel/RKHS which is chosen.\n\n## Minor\n\n- line 35: please clarify that the tangent space is the tangent space within Fisher-Rao geometry (for other geometries the tangent spaces look different). Also define $L_0^2$ (probably zero-mean $L^2$-functions?).\n\n- it is a bit strange that the differential operator $D\\theta_P$ is introduced in line 142-143 while it is already used before. I would suggest to put the definition directly in Section 2.1.\n\n- Section 1: The $\\psi$ is often the index P (unless this is intentional, in which case the authors should explain the difference between $\\psi$ and $\\psi_P$)\n\n- The simulation experiments at the end of the main paper don't have enough explanations. What do we see in the plots? Please add some description. What does MISE stand for? I would suggest the authors to rather shift this part to the appendix and properly explain it instead of presenting it without explanations in the main part.\n\n- It's \"principal components\" not \"principle components\" (appears in line 102, 106, 175, 176)...", "was_revised": true, "final_justification": "During the discussion phase, the authors addressed several of my comments. In particular, the authors explained their numerical setup and numerically investigated the bias introduced by the regularization. Some loose ends of the paper remain open (dependency on the kernel, theoretical investigation of the bias introduced by the regularization, applications) or cannot be checked during the discussion phase (that the explanations for a broader audience will be improved).\n\nHowever, I appreciate the authors efforts in the discussion and I acknowledge that some of the loose ends can be classified as \"future work\". Presuming that the promised changes will be incorporated in the paper and that the accessibility for a broader audience will be improved (which is in the authors own interest), I raised my rating to 5 (accept).\n\nRegarding the scope, I still think that the paper will be very niche at NeurIPS. But the developed method is based on RKHS theory and could be useful for several applications (e.g. for uncertainty quantification or robustness of statistical estimators) and I would therefore see it within the scope of NeurIPS.", "rebuttal": "We thank the Reviewer for the careful reading and insightful interpretation of\nour work, valuable and encouraging feedback.\nIn particular, we are delighted that the Reviewer found our work within scope\nfor NeurIPS,\nour core idea of discretizing along a basis of eigenfunctions quite nice,\nand our method theoretically well founded,\nthe technical part our paper well readable,\nand numerical feasibility of our method quite likely to work.\n\n### Computational feasibility and comparison to the baseline\n\nWe acknowledge that expanding our simulation experiments to include more\ninteresting and more complex estimation targets would greatly aid the impact of\nthe paper.\nWe have experimented with the following more interesting functionals:\nthe average density functional from Bickel'88,\n$\\theta(P) = E[p(X)^2]$ with $\\psi_P(x) = 2p(x) - 2\\theta(P)$, where $p$ is the\ndensity of $P$;\nthe median/quantile functional whose influence function contains a jump\ndiscontinuity;\nthe entropy and mutual information functionals (a 2d example);\nthe Gini coefficient and Lorenz value B functionals.\nThe simulations results are qualitatively the same for these functionals as\nthose for the mean, we obtain visually good estimates with very few spectral\nterms, with the exception of the median whose Fourier series converges slower\ndue to the discontinuity.\n\nRegarding the computational complexity,\nthe cost of our method is incurred with\ndensity estimation (complexity and accuracy vary depending on the method),\nkernel PCA,\nand evaluation of the functional on the perturbed density, say $C$.\nIf we use kernel methods for density estimation, and use Nystr\\\"om\napproximations for both the density and kPCA estimation, then the\ncomputational complexity is $O(nr^2 + rC)$.\n\nRegarding comparison to the point-wise baseline method, it too incurs the cost\nof estimating the density, which is at best comparable to the cost of kPCA, so\nit would scale as $O(nr^2 + nC)$ with the same numerical accelerations.\nPlease also see our response to Reviewer em1K.\n\nWe agree with the Reviewer that the most convincing demonstration of the\nfeasibility would be to apply our method to a real-world-scale problem and think\nthat this would be a great goal for follow-up work.\nAs for the present theoretical paper, we thank the Reviewer for acknowledging\nthat kernel methods are well-established and numerical methods that scale them\nare extensively and actively researched.\nIndeed, we are excited to explore applications of [Che+25] and other randomized\nnumerical linear algebra methods we recently learned about at the Householder\nSymposium to implementations of our estimator on such problems.\nPlease also see our response to Reviwer em1K describing additional experimental\nwork we will focus on for the camera-ready version to demonstrate applicability\nto relevant statistical settings.\n\n### Improving the exposition\nPer your suggestion (and similar feedback of other reviewers) we will\nexpand the introduction for the camera-ready version to make the background and\nconnection with prior and related works more accessible, invest significant\neffort on explaining our results, clarify our simulations and limitations/future\nwork.\n\n### How-to section\nWe will include detailed steps to implement our method in he Appendix for he\ncamera-ready version.\n\n\n### Dependence on $P$\nThis is a very keen observation, but we view it as follows.\nThe influence function $\\psi_P$ that we want to estimate depends on $P$,\ntherefore, we see the dependence on $P$ of our eigenbasis as a strength rather\nthan a weakness, as it opens the door for adaptation.\nAlso, we cannot really think of a setting in statistics where one would want to\nestimate the exact same functional on many different populations, but it's\npossible that such settings exists in other fields.\n\n\n### Regularization bias and optimal choice of $\\lambda$ and kernel\nThis is a great question.\nWe do not claim to offer any insight into the choice of the regularization\nparameter $\\lambda$ of our estimator in this paper.\nThe optimal choice would strike a balance between the bias (increases in\n$\\lambda$) and the variance (decreases in $\\lambda$) and would depend on the\nsmoothness of the target IF/functional and $P$.\nA finer analysis of the rate of convergence of our estimator is required for\nthis analysis.\nThis is a goal of our follow-up work, but requires a lot more theory and\nreasonably out of scope for the present paper as acknowledged by Reviewer em1K.\n\n### Minor points\nline 35: the tangent space is indeed the one of the Fisher-Rao geometry\ncontaining all zero-mean $L^2$ functions. We will clarify in the paper.\n\nline 142: our intention was to introduce $D\\theta_P$ in the outline of the proof to\nTheorem 1.1, clearly this didn't work, so we will revisit and re-evaluate how we\ncan be more effective at introducing this and related background\nnotation/concepts in Section 1.\nAny further feedback would be much appreciated.\n\nPlots: per your suggestion and similar feedback from other reviewers we will\nrework and expand the simulations section, improve the readability of the plots,\nand provide an more detailed explanation of the experiments and results in the\ncamera-ready version.\nMISE stands for Mean Integrated Squared Error, please also see the response to\nReviewer 8gPt.\n\nThank you very much for pointing out the typos in PCA."}]

Author rebuttal:

Meta-review: The paper proposes an innovative idea to estimate the influence function, and it 
theoretically investigates the derived method. The main concern raised in the reviews
was centering around the presentation. Having read the current version, I see that certain 
parts can still be improved. On the other hand, the current version's presentation is not 
worse than that of some accepted papers I have seen in the past. The experimental section is 
short, but as this is mostly a theoretical paper, this is acceptable. Finally, some more theory
like non-compact domains or finite sample bounds would have been nice, but from my 
perspective, the presented theory is enough for an innovative paper presenting a new idea.

Despite the issues I have discussed above, the paper is a clear accept for me, as the presented
idea is too nice to be rejected and the presented theory is already sold.
Decision: Accept (poster)