Title: Kernel von Mises Formula of the Influence Function
Venue: NeurIPS 2025
Accepted: yes
Keywords: Influence function

Fisher-Rao gradient

first variation

tangential interpolation

semiparametric estimation

distributional robustness

kernel PCA

Mercer kernel
Abstract: The influence function (IF) of a statistical functional is the Riesz representer of its derivative, also known as its first variation and Fisher-Rao gradient.  It is a key object for numerical optimization over probability measures, semiparametric efficiency theory, standard constructions of efficient estimators, and an arsenal of inference methods for these estimators.  Yet, deriving the IF analytically is often an obstruction for practitioners.  To automate this task, we develop a novel spectral representation of the IF that lends itself to a low-rank functional estimator in a reproducing kernel Hilbert space (rkHs).  Our estimator (i) does not require analytic derivations by the user, (ii) relies on kernel Principal Component Analysis and numerical pathwise derivatives along these components.  We present the derivation of the representation and prove consistency of the low-rank rkHs estimator.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a way to compute the first variation of a statistical estimator in the Fisher-Rao geometry at some probability measure $P$. To this end, the authors first represent the first variation as steepest descent direction among all (normalized) variations. To make the problem tractable, they introduce a regularization of this variational formulation by some Hilbert space norm. In this case the (regularized) first variation can be expanded into an eigenbasis of the Hilbert space.\n\nIn practice, the Hilbert space is chosen as RKHS of a Mercer kernel and the eigenbasis as the principal components of the measure. For this case the authors prove that the computation of the eigenvalues of the kernel matrix still leads to a consistent estimator of the eigenvalues of $P$. As a final result they prove in Thm 3.7 that the combination out of cutting off low eigenvalues and estimating the principal components from data still provides a consistent estimator.\n\nSTRENGTHS AND WEAKNESSES:\n## General comment\n\nI guess, the paper will be very niche for NeurIPS, but I would see it within the scope of NeurIPS such that there is not too much of a problem.\n\n## Strengths\n\n- I find the concept of discretizing the influence function in terms of eigenfunctions instead of using a spatial discretization quite nice.\n\n- The authors provide a theoretical foundation of their regularization and approximation method including convergence (consistency) results. The technical tools (regularization with an RKHS norm, spectral decomposition) are well-established and quite likely to work.\n\n## Weaknesses\n\n- A critique which might be quite common for theory papers: It is not clear whether the proposed framework works well in practice. There is only one very small 1D example in the paper. It would be beneficial to implement the method in a more serious setting and compare the error to the point evaluation method mentioned in the introduction (which will be of course more expansive when evaluating more points, but is it more accurate?). How does it scale with the dimension of the probability measures (which often is a problem for kernelizations)? Given the theoretical nature of the paper, I wouldn't weight this critique to much, but given that the paper claims to propose a faster and more practical method than previous papers, at least some evaluations could be included.\n\n- I think that the authors could invest a bit more efforts in writing and explaining their results. While the technical part of the paper is well readible (given a general background in analysis), the explanations in the introduction and limitations/future work are very sparse and I am not fully clear what the authors mean even after reading the paper.\n\n- Similarly, the paper would benefit from a how-to section in the paper (potentially in the appendix): That is, a section which says, given data points $x_1,...,x_n$, what do I have to do to get the (approximated) influence function.\n\n- The approximation procedure is specific to the choice of $P$. So if I want to compute the first variation of the same estimator for different base distributions this might be restrictive.\n\n- It remains unclear to me how much bias is introduced by the regularization and how this depends on the kernel/RKHS which is chosen.\n\n## Minor\n\n- line 35: please clarify that the tangent space is the tangent space within Fisher-Rao geometry (for other geometries the tangent spaces look different). Also define $L_0^2$ (probably zero-mean $L^2$-functions?).\n\n- it is a bit strange that the differential operator $D\\theta_P$ is introduced in line 142-143 while it is already used before. I would suggest to put the definition directly in Section 2.1.\n\n- Section 1: The $\\psi$ is often the index P (unless this is intentional, in which case the authors should explain the difference between $\\psi$ and $\\psi_P$)\n\n- The simulation experiments at the end of the main paper don't have enough explanations. What do we see in the plots? Please add some description. What does MISE stand for? I would suggest the authors to rather shift this part to the appendix and properly explain it instead of presenting it without explanations in the main part.\n\n- It's \"principal components\" not \"principle components\" (appears in line 102, 106, 175, 176)...\n\nQUESTIONS:\nsee strenghts/weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel way of deriving the influence function of a target parameter via the Riesz representer of its derivative operator. Previous results had focused on point-wise derivations of the influence function (von Mises formula, Theorem 1.1 in the paper), which allow for point-wise approximations of the influence function. In contrast, the ideas presented in this contribution open the door to simultaneous  approximations to all values of the influence function, given that the approximation is conducted via the influence function spectral representation. \n\nThe overall idea builds on an approximated variational representation of the influence function: the influence function can be exactly represented as a functional optimization program; this program can be further regularized with a Sobolev norm to obtain approximate solutions. In order to do so, the spectral representation of the regularized functional is exploited. In particular, if this Sobolev space is a Mercer RKHS, then the solutions are actionable in practice. The consistency of the kernel von Mises estimator is established.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The paper is well written overall, reading very well despite being technical in nature. The submission is technically sound, with all the claims being theoretically supported. In this sense, the authors also clearly state the main technical/theoretical limitation of the work: they only prove consistency of the estimator, without studying the convergence rates of it. However, the idea seems original and powerful enough so that consistency of the estimator suffices for a first paper on this idea.  While going from point-wise evaluations of functions to spectral approximations of such functions is not extremely original by itself, the combination of these ideas with the variational representation of the influence function makes the contribution non-trivial and original. \n\n**Weaknesses**: While the originality of this paper is not under question, I think these spectral approximations are not substantially significant until proven practically useful. Based on this comment, I see two main flaws in the current version of the paper. \n\n- First, I would like to see a discussion on how easy/computationally feasible it is to obtain the pathwise derivatives $D\\theta_P$ on the eigenfunctions $\\hat e_j$ in Section 3. Do these need to be numerically approximated themselves? Overall, what is the complexity of the whole procedure?\n- Second, the simulations are only run with $\\theta$ taken to be the mean functional, arguably the easiest possible functional. I would like to see the performance of these approximations in more challenging/real scenarios. For example, given the importance of influence functions in causal inference and their connection to doubly robust estimators, the authors could take $\\theta$ to be the average treatment effect of some causal experiment, and compare the results using the approximate influence function of this paper with the usual plug-in  or inverse probability weighting estimators (as if the doubly robust estimator was not known). I would also encourage the authors to run a simulation on a causal target whose influence function is actually unknown, and compare the results obtained with the approximate influence function obtained using the techniques presented in this paper with some naive estimator of such a causal target.\n\n**Minor comments**: \n- Discuss the choice of $r(N)$, both theoretically (does Lemma 3.6 hold for any definition of $r(N)$?) and practically (are practitioners encouraged to used r(N)-rank approximations in practice?).\n- It almost seems like the authors run out of space. Theorem 3.7 is hanging there without any context or explanation. Please make some space for explaining what Theorem 3.7 is doing there and its connection to previous results (I do understand it but it does not read well). \n- The definition of $D\\theta$ first appears in Section 2.3, but the concept is already used in Equation (2.2). Please make the definition appear before it is used. \n- The plots are too small, especially the legends.\n\nQUESTIONS:\nMy questions have been posed as the two main weaknesses above. I believe they are concise. If they are addressed satisfactorily, I will raise my score.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies estimating the influence function of functionals that are asymptotically linear. The work introduces an estimator for the influence function by approximating changes in the influence function from the derivatives of a kernel estimator. The authors prove that their estimator is consistent and provide some experimental demonstration of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nI am not familiar with the literature, so I may be incorrect on some of these criticisms and strengths. If the authors can address my confusion, I may be willing to raise my score.\n\nStrengths:\n\n-\tThe problem of estimating the influence function seems fundamental and well-motivated.\n\n-\tTheir approach seems novel and theoretically grounded.\n\n-\tSome experimental validation of their method is provided.\n\nWeaknesses:\n-\tI am having difficult placing the novelty of the result in the context of prior work. How has the influence function been estimated in prior works and how does this approach compare?\n\n-\tThe plots are largely unclear. What are the axes labels? What should the reader take away from these plots? How should the reader interpret the magnitude of the errors in the estimate of the influence function? What about functionals other than the mean?\n\n-\tIt would be good to provide some explanation of what the ‘Von Mises’ formula is for the reader.\n\n-\tIt is not clear to me how to compute the derivatives of your functional. This needs to be clarified, perhaps with an example.\n\nQUESTIONS:\n1.\tHow strong is this assumption that our estimators are asymptotically linear? \n2.\tHow can the estimators error be represented by contributions of individual data points in a way that is independent of the total number of samples? Is this an assumption as well? If so, how is this assumption motivated?\n3.\tCan influence functions in sample distribution space be connected somehow to influence in input space? I.e. instead of looking at perturbations in the distribution of the data, what if you examined the influence of perturbing a sample in a particular direction in its ambient space.\n\nLIMITATIONS:\nThe biggest limitation of this work is that the majority of the NeurIPS audience will likely have difficulty appreciating this result. I recommend spending significant effort to clarify the background and novelty of their results, as well as possibly expanding the experimental section to include comparisons to previous methods. If previous methods don’t exist or are unimplementable for some valid reason, then an explanation of this fact is sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces kernel-based estimators for the influence function, which is well-known in semiparametric and non-parametric statistics. The authors show that their procedure yields asymptotically consistent estimators and they numerically validate their approach on a toy experiment.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Very well written article, bridging classical statistical theory and reproducing kernel Hilbert space methods commonly employed in machine learning.\n### Weaknesses\n\n- No finite sample guarantees. As classical theory [1] considers asymptotic results only, are there existing results to obtain rates, as hinted at in line 110?\n- The results apply to compact sample spaces only. While the authors hint at possible generalizations [2], their theory does not take these adjustments into account.\n\n[1] Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge University Press.\n\n[2] Steinwart, I., & Scovel, C. (2012). Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs. _Constructive Approximation_, _35_, 363-417.\n\nQUESTIONS:\nKey points:\n\n- How do we know that the RKHS $\\mathcal H$ is dense in the tangent space $L_0^2(P)$? Do you implicitly appeal to Theorem 4.26 in [1] and are the assumptions there satisfied?\n- Similarly, in Assumption 3.1: Which definition of universality do the authors use? The usual ones consider denseness in the space of continuous functions.\n- Does the normalization for obtaining (ii) given in line 222 preserve/imply (iii)?\n- The \"extension operators\" $R_n$ seem to be the similar to the usual sampling operators; see [2] and [Lemma A.7, 3].\n\n[1] Steinwart, I. & Christmann, A. (2008). Support vector machines. Springer.\n\n[2] Smale, S., & Zhou, D. X. (2007). Learning theory estimates via integral operators and their approximations. _Constructive approximation_, _26_(2), 153-172.\n\n[3] Sterge, N., & Sriperumbudur, B. K. (2022). Statistical optimality and computational efficiency of Nystrom kernel PCA. _Journal of Machine Learning Research_, _23_(337), 1-32.\n\n\nFurther questions/remarks:\n\n1. The relationship to [1] could be made explicit, or, at least, the reference could be included as from a quick gScholar search this seems to be one of the few NeurIPS publications also considering the influence function.\n2. Line 21, \"foundational role in theory\" => add a few relevant references.\n3. \"Dilated\"/\"mollified\" seems to be used interchangeably. \n4. In line 76, $L_\\text{loc}^1(\\mathbb R^d)$ is not introduced.\n5. In the paragraph starting from line 87, what is the difference of $\\psi$ and $\\psi_P$ (where the latter was used before)?\n6. In line 108, \"kPCA\" not introduced.\n7. In the paragraph starting from line 198, the notations for $K$ and $\\phi$ clash.\n8. Assumption 3.1: absolutely continuous w.r.t.?\n9. The discussion from line 266 could come before Lemma 3.4 to motivate the result.\n\n[1] Kandasamy, K., Krishnamurthy, A., Poczos, B., & Wasserman, L. (2015). Nonparametric von Mises estimators for entropies, divergences and mutual informations. _Advances in Neural Information Processing Systems_, _28_.\n\nA few notes on typos/grammar:\n- In line 43, add $\\hat \\cdot$ to $\\theta'$ and $\\theta''$.\n- In line 53, \"is an absolutely\".\n- In line 100, check grammar/formulation.\n- In my opinion, \"rkHs\" should be \"RKHS\". The former looks weird.\n- In line 174, add \"RKHS; recalled below in Section 3.1\" or something similar.\n- The section titles in Section 3 have full stops at the end.\n- In line 308, capitalize \"consistency\".\n- In line 312, typo in \"Nyström\".\n- In line 317, typo in \"bounded\".\n- In line 328, \"describe\" => \"described\".\n- In line 328, typo in \"debiased\".\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the work the authors introduce an estimator for the influence function $\\psi_P$, for some statistical $\\theta(P)$ with statistical estimator $\\hat{\\theta}(X_1,\\ldots, X_n)$ and the influence function is equal to $\\sqrt{n}(\\hat{\\theta} - \\theta(P)) = \\frac{1}{\\sqrt{n}} \\phi_P(X_i)$ (according to (1.1) in the paper). The influence function is useful to know for a variety of statistical tasks. The influence function can be evaluated for specific points in a fairly straightforward way, but it is useful to have a full functional representation of it. In this work the authors introduce an estimator for the influence function using a sort of RKDE kernel basis (i.e. it uses a truncated bases) to estimate $\\psi$ from data.\n\nSTRENGTHS AND WEAKNESSES:\nIt's possible that the topic of the paper is simply too far outside my expertise for me to understand it totally, but I found this paper very hard to follow. I really think it needs to be significantly reworked for the NeurIPS audience. The paper is pretty idiosyncratic, in my opinion. It has no \"Related Works\" section for example, and the core contribution only occurs on the last page and is framed as a Theorem, where it really should be a technique. Some points about this can be found in weaknesses.\n\n## Strengths\nThis paper introduces a new approach to an established task I statistics and thus is of reasonable significance and usefulness. The derivation of their technique is quite extensive, which really precisely and rigorously grounds their technique. Additionally  the work contains several theoretical results, thereby supporting the technique well mathematically.\n\n## Weaknesses\nI really think the style of the paper doesn't fit well with NeurIPS. It has a rather strange style where the paper immediately begins explaining the technique it's presenting from the first page and only finishes on the last page. One would typically expect sections providing background, examples, consequences, etc, but virtually the whole paper is a derivation of the technique. I personally don't think this is a good fit for NeurIPS. That being said, I don't really feel like I grasped the paper well, perhaps it is a highly technical topic that can't really be presented any other way; maybe the other reviewers can shine some light on this. I get the impression that this work may fit better in a more core theoretical statistics venue.\n\nA lot of the mathematical objects are not quite familiar enough that I think they can be introduced without any elaboration or are really beyond what most of the community will understand. Other parts are elaborated on too much.\n* The introduction to the influence function should be more extensive. I think a couple of examples one trivial (mean) and one less trivial would be good. Is (1.1) really the only definition? Any funciton that satisfies this property is the influence function?\n* Concrete applications of the influence function, fully explained, would also be useful to convey the significance of it.\n* Some of the exposition is just way beyond what I can understand reasonably easily, a good example is lines 87-97. It really feels like one would need to be quite familiar with the topic to really get whats going on here\n* Section 3.1 is unnecessary in my opinion. Much of the community is very familiar with RKHS and this kind of background is unlikely to bring the less familiar to a working familiarity of RKHSs to understand what is going on.\n* For papers like these I think its useful for the main text to be a pretty simplified and for the appendix to contain the full technical underpinnings\n* It would be good to have the full core contribution technique explained together without any of the extra exposition, I feel like I'm often losing the tread reading this.  \n### Small Points\n* Line 39: is $\\psi$ without the $P$ subscript something different?\n* line 34: $L_0^2$ is never defined.\n* Line 52: What is pathwise differentiable?\n* line 168: is the $\\partial$ operator different from the $D$ operator in some significant way?\n* Line 224: what is a signature? Is it oveloading the variable $K$? or is it somehow the same thing?\n* Stylistically it's good to refer to Figures in the main text, not just have them stand alone with no reference. (This may even be in the NeurIPS style guide)\n\n\nAgain I want to highlight that its possible I'm just too far out of my area to expertise to appreciate whats going on in the paper, I will be interested to see what the other reviewers say.\n\nQUESTIONS:\n* I’d really like to see a concrete example—a specific dataset and task—where your method truly enables something new to be done, or clearly outperforms previous benchmarks.\n* Can you elaborate more on the tasks where the influence function is important? Maybe full, concrete example.\n\nLIMITATIONS:\nNot address", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}