Empirical Gateaux Derivatives for Causal Inference
Authors: Michael Jordan
Yixin Wang
Angela Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal inference
double robustness
bias-adjustment
influence function
semiparametric
offline reinforcement learning

Abstract: We study a constructive procedure that approximates Gateaux derivatives for statistical functionals by finite-differencing, with attention to causal inference functionals. We focus on the case where probability distributions are not known a priori but need also to be estimated from data, leading to empirical Gateaux derivatives, and study relationships between empirical, numerical, and analytical Gateaux derivatives. Starting with a case study of counterfactual mean estimation, we verify the exact relationship between finite-differences and the analytical Gateaux derivative. We then derive requirements on the rates of numerical approximation in perturbation and smoothing that preserve statistical benefits. We study more complicated functionals such as dynamic treatment regimes and the linear-programming formulation for policy optimization infinite-horizon Markov decision processes. In the case of the latter, this approach can be used to approximate bias adjustments in the presence of arbitrary constraints, illustrating the usefulness of constructive approaches for Gateaux derivatives. We find that, omitting unfavorable dimension dependence of smoothing, although rate-double robustness permits for coarser rates of perturbation size than implied by generic approximation analysis of finite-differences for the case of the counterfactual mean, this is not the case for the infinite-horizon MDP policy value. 


Empirical Gateaux Derivatives for Causal Inference
Michael I. Jordan"
Yixin Wang" Department Statistics University of Michigan y exin@@umm Bum ch edu
Department of EECS and Department Statistics University of California, Berkeley
jordan@cs berkeley. edu
Angela Zhou"
Department of Data Sciences and Operations
University of Southern California zh chouseee wowuuuucceeee
Abstract
We study constructive algorithm that approximates Gateaux derivatives for sta- tistical functionals by finite differencing, with a focus on functionals that arise in causal inference. We study the setting where probability distributions are not known priori but need to be estimated from data. These estimated distributions lead empirical Gateaux derivatives, and we study the relationships between em- pirical, numerical, and analytical Gateaux derivatives. Starting with case study of the interventional mean (average potential outcome), we delineate the relation- ship between the empirical Gateaux derivative (via finite differencing) and the analytical Gateaux derivative. We then derive requirements on the rates of nu- merical approximation in perturbation and smoothing that preserve the statistical benefits of one-step adjustments, such as rate double robustness. further study more complicated functionals such dynamic treatment regimes and the linear- programming formulation for policy optimization infinite-horizon Markov de- cision processes. The ability to approximate bias adjustments in the presence of arbitrary constraints in these more complicated settings illustrates the usefulness of constructive approaches for Gateaux derivatives. We also find that the statisti- cal structure of the functional (rate double robustness) can permit less conservative rates for finite-difference approximation. This property, however, to particular functionals; e.g., occurs for the average potential outcome (hence average treatment effect) but not the infinite-horizon MDP policy value.
be specific
Introduction
Inferential targets in causal machine learning often take the form of statistical functionals of the data distribution. Examples include average treatment effects or average policy values for infinite- horizon offline reinforcement learning. Estimation of these statistical functionals can be vulnerable to the first-stage bias introduced by the estimation of nuisance functions such as outcome regressions transition probabilities. However, by leveraging the causal structure, it is possible to derive bias- adjusted estimators that weaken the need for accurate estimation of these nuisance functions. The general frameworks
celebrated doubly robust estimator is one such bias adjustment, and there for deriving such estimators, including semiparametric efficiency. double robustness, Neyman orthogonality, and "debiased/double" machine learning [7, 12, 13. 35, 52, 53, 56, 62]. These general frameworks can also be applied to more complicated functionals in longitudinal causal
Authors listed in alphabetical order.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

inference and offline reinforcement learning (also known as offline policy evaluation/leariing) [5, 11, 43, 46-49, 67, 68, 71].
A drawback of these general frameworks is that significant analytic effort is often necessary to ob- tain concrete estimators that are appropriate for particular situations. Indeed, practitioner may be interested in novel variants of an estimand, or be working within a constrained class of functionals that are appropriate for a partieular class of problems. For example, in constrained Markov Deci- sion Processes (MDPs) [3], or aptimization-basee estimators more broadly, a practitioner can easily avail themselves of custom convex- cevex-oopimmization solvers that are computationally efficient and easy to deploy to add additional constraints. Unfortunately, such choices may require case-specific re- analysis to establish the desired statistical properties. and deriving the actual estimator that yields bias adjustment may require significant analytical effort. It is therefore important to develop a suite of constructive, numerical or algorithmic methods that yield desired forms of bias adjustment. Such a suite would be complementary to analytic derivations.
To this end, we develop off-the-shelf procedures for estimating the statistical functionals for bias adjustment in causal inference. We do so via numerical approximation of the Gateaux derivatives that underlie the general analytic frameworks, building on prior work in this area. We focus on interventional effects. [57] suggested numerical differentiation to estimate Gateaux derivatives of functionals unavailable in closed form but approximated by computational procedures, such as the solution to system of differential equations; this approach is later used in [34]. Recent pedagogical surveys of [35, 53] also emphasize the usefulness of Gateaux derivatives for deriving influence functions, an approach which is directly applicable for discrete data and generalizes to continuous distributions via a smoothing argument as discussed in [10, 39]. In this spirit, we build on the line of work of Carone et al. [10], Frangakis et al. [24], Ichimura and Newey [39], Newey [57]. which proposes a constructive procedure by numerical approximation of a Gateaux derivative (in the sense of Hampel [32], Huber [38) by finite differences.
Evaluating influence functions by numerical differentiation can be useful because it only requires black-box evaluation of the functional. In other areas such as optimization, numerical differentiation underpins common subroutines that enable analysts to use optimization algorithms without special- ized training in a particular modeling language or paradigm. 2 Although more advanced paradigms compute exact gradients, they can require specialized training to use. In addition, the numerical differentiation viewpoint connects the conceptual interpretation of influence functions as qualitative sensitivity analysis (i.e. sensitivity of functional to perturbations in the distribution) to the use of influence functions in causal inference. Typical presentations of the latter framework are not con- cretely connected to the former notion. We provide a concrete connection, which can be interesting independent of computerized estimation for algorithmic, pedagogical, or conceptual reasons.
In this paper, we build on the previously mentioned line of work on the use of numerical derivatives to approximate influence functions, focusing on the implications for statistical estimation. We pro- vide exact characterizations of empirical Gateaux derivatives, in contrast to numerical derivatives computed based on oracle knowledge of probability distributions. This work is focused on exact characterizations that can inform statistical and computational trade-offs. Although we expect that some of these characterizations may be apparent to experts, we also hope that this level of concrete- ness can be helpful to non-expenr non-experts. These trade-offs could eventually be practically relevant because the finer-scaled discretization required for lower approximation error can incur fundamental issues with floating-point computation that prohibit further gains in accuracy [23, 64).
Our contributions are as follows. We begin with a case study of the average potential outcome. The average treatment effect is the difference of the average potential outcomes, and our analysis applies also to the average treatment effect. We discuss the treated mean for simplicity; by symmetry the same results hold for the control mean. In this setting, we characterize the exact relationship between fiite-differences and the analytical Gateaux derivative. This characterization helps concretize the proposal of empirical Gateaux derivatives by allowing us to study the rates of numerical approxima- tion that can preserve statistical performance. Finally, with these concretizations in hand, we turn to more complicated motivating examples including a finite-horizon dynamic treatment regime and infinite-horizon reinforcement learning. We illustrate with the case of policy learning in finite-state,
For example, many data scientists use optimization through the scipy. , opt imize or r. opt om pack- ages: if they do not provide a gradient function callback, in some configurations, it is by default approximated by numerical differentiation.
2

---

Markov decision processes how this constructive approach to Gateaux derivatives can directly extend also the case of custom constraints.
finite-action
2 Problem Sctup: From Numerical to Empirical Gateaux Derivatives
We begin by introducing our first example, that of the mean potential outcome, and the canonical doubly-robust estimator. We then briefly introduce the key objects in the more general framework of orthogonalizing or debiasing statistical functionals by Gateaux derivative adjustments via the one- step estimator. We also describe the numerical approximation of these adjustments via perturbed black-box evaluations of the statistical functional and plug-in estimation. After introducing these frameworks, clarify what mean by empirical rather than numerical Gateaux derivatives and delineate our specific research questions.
We let O P denote draw of observations following the distribution P belonging to statistical model M. We will focus on the estimation of statistical functionals (PP,, where : : - R2 is pathwise differentiable. Throughout, take given the assumptions that grant causal identifiabil- ity (i.e., overlap/positivityy unconfoundednness/nnrrbbiiii
The mean potential outcome and augmented inverse propensity weighting. We very briefly overview the celebrated doubly robust estimate of the mean potential outcome. The average treat- ment effect is simply the difference of these doubly robust estimators for the treated and control means; for simplicity, we discuss the treated mean only.
Example 1 (Mean potential outcome). The observation O = (X, includes covariate X, treatment A {0, 1}. and outcome Y E R. The statistical functional corresponding to the mean potential outcome is:
(P)) ([) = E|E ==
Plug-in estimation of the conditional expectation is termed regression adjustment. The overall func- tional is also referred to as inverse propensity weighting. (D(P) E[Y = E
y [|==
c(X) where
e(x) = P(A = 1 X = x) is the propensity score. (We omit explicit discussion of assumptions such consistency and ignorability, under which these identification results hold).
Given two identifying functionals for the mean potential outcome, natural question is: can these identification approaches be combined in some way improve estimation? The doubly-robust estimator of Robins et al. [62] achieves this as follows:
(1) E H (Y E[Y A 1, + E|y - 1,x 1,
Appealing properties such as the mixed-bias property and the rate double robustness properoty can be verified readily from its functional form. This provides canonical example of one-step adjust- ment by influence functions that applies more broadly.
Influence function and Gateaux derivative. A functional Vi is Gateaux differentiable at P if V'H;; P) dP(())))) l exists and is linear and continuous in H for all H.3
Hampel [32] and Huber [38] define the influence function o(O; P) as a Gateaux derivative with respect to perturbations H = 6, P. where 6 is degenerate Dirac measure satisfying the identity f [oo (o) = o'. That is, we have:
d (P + c(5. P)) l==0
b( P)
de
Numerical Gateaux derivatives. Although this definition of influence function via perturbation not quite enough for semiparametric inference with continuous distributions, Carone al. [10] and Ichimura and Newey [39] show how to obtain absolute continuity in semiparametric models by taking an additional limit as a smoothed perturbation converges the point mass. They replace the Dirac delta measure d, with smoothed distribution 5% (o), function satisfying f (u) (u)
Another way to state the definition is that v is Gateaux differentiable at P if there exists op(H) such that s0 + cff) (P)) = (f)) + o(t), as t + 0 [73, p.296]. This statement coincides with the previous statement when the functional maps to the real line.
3

---

and dominated by P. A common choice for dy has the form 0% (o) = (2 d'), where Kx is bandwidth-normailn
kernel function and product kernel. We then define perturbation in the direction of a generic observation or as follows: Pe. Pr,x = (1 c) c,,
Carone et al. [10], Frangakis et al. [24] discuss numerical approximation by finite differences: for X, evaluate the numerical derivative δ(0; P) (V (P)). Further, [35, 53] highlight influence function derivations based on the analytic Gateaux derivative.
fixed e, with
Carone et al. [10] establish that the limit of P.A when and À tend to zero is precisely the Gateaux derivative. and they provided generic approximation rates in (c, A) for multi-point finite difference scheme. These rates establish sufficient conditions for approximation error that ensure validity of the numerical approximation, but can he conservative. For a two-point approximation with uniform kernel, the rate from Carone et al. [10, Thm. 5] is (ex-d) + x-2 We later strengthen this result.
Empirical Gateaux derivatives. In the context of statistical estimation, the underlying proba- bility distributions also need to be estimated. We consider plug-in estimation of the statistical functional via density estimation of the collection of joint probability estimates p(o) that com- prise P. For example, plug-in representation of the mean potential outcome is as follows: te (P)) S p(g.A=1.)) P(A=1,2) p(x)
is important to distinguish between the empirical Gateaux derivative computed by numerical approximation with estimated probability densities (c.g., P(Y, A = 1, X)). the numerical Gateaux derivative obtained by finite differencing based on the true probability densities, and the analytic Gateaux derivative:
ē(O,) - (V(P))) W(P) (v())
empirical derivative at smoothed and estimated distributions,
((O.) c- (V(P.) (P)) d ((P.) ((.)) de r=0
numerical derivative at smoothed and true distributions,
analytical Gateaux derivative.
It is the first of these representations that is appropriate for serving as a statistical estimator of influence functions. In particular, the one-step estimator justified by the following asymptotically linear representation [60]:
V v(P) ! ((O,),
where ( (P)).
See Appendix A for further discussion of this expansion. Given these definitions, we now turn to the major mathematical and statistical questions that we address in this paper:
Q1: How exactly does the empirical Gateaux derivative approximate the analytical derivative?
Q2: What are the required rates of numerical approximation in perturbation and smoothing based on (c, A) that preserve the beneficial statistical properties of the constructed estimator?
In Section 4, we study these questions in the simple case of our running example: estimation of the mean potential outcome. In that setting we can provide fine-grain comparison of the finite- difference estimator and the classical doubly robust estimator. Section we study more compli- cated functionals to highlight more general applicability.
Example 2 (Example 1, continued). The influence function [31, 62] for mean potential outcome is:
]A=iI ((O) = -(
(Y - E| A 1I) + B| A = 1, (P)).
Hence the canonical doubly-robus, estimator (augmented inverse propensity weighting, or AIPW) can be viewed as a one-step adjustment.
We conclude our introductory remarks by noting some limitations of our study. We omit discussion of semiparametric efficiency since we focus on computing influence functions without the consid- eration of model restrictions and tangent spaces. That is, we focus on Gateaux derivatives under nonparametric model. Throughout, we assume the functional pathwise differentiable. This as- sumption does warrant caution, because this condition pose fundamental barrier to efforts for

---

"automation"; ; place analytical expertise. See Appendix D.2 for extended discussion on how stronger smoothness conditions admit sufficient conditions for asymptotic linearity.
hence empirical Gateaux derivatives are intended to augment but not supplant or re-
3 Related Work
In this section, we discuss the most closely related directions but do not attempt to provide survey or overview of influence functions or semiparametric statistics. See Bickel et al. [7], Fisher and Kennedy [21], Kennedy [53]. Tsiatis [69] the appendix for broader overview and further de- tails. In Appendix D.I, for additional context, we also overview work on numerical derivatives in optimization and statistical machine learning more broadly.
Recent pedagogical reviews [21, 35, 53] em- phasize the Gateaux derivative and its use a heuristic tool for deriving influence functions. While our point of departure is a finiteddifference approach to the former, we also consider more compli- cated optimization functionals, with focus how dual variables characterize the Gateaux deriva- tives therein. [10] derive high-level sufficient conditions for approximation error via general results for finite-difference approximations. Relative to those general sufficient conditions, we focus on specific improvements in approximation error for various functionals. For the specific case of the mean potential outcome/average treatment effect, Frangakis al. [24] interchange integration and differentiation as a result of analytical insight hence their approach is less "automatic." In general, it may be difficult deduce the form of the nuisance functions without deriving the influence function analytically. The approach in Appendix B of [9] considers the specific case of debiasing moment conditions with respect to first stages but also implicitly leverages this change in order of integration.
Numerical approximation of Gateaux derivatives.
Other work on automating causal inference. Recent work in causal inference aims at "auto- matic" or "computerized" semiparametric inference via variety of methods [4, 14-16, 20, 45, 51]. To summarize briefly the difference with our approach, our approach focuses on the use of numerical derivatives, and requires only zeroth-order computation oracle of perturbed functional evaluations, without additional nuisance estimation. This approach can be particularly helpful for incorporat- ing debiased causal inference in the design and analysis of algorithms. On the other hand, it may not achieve as strong improvements in estimation performance as other approaches that estimate additional nuisance functions. Concurrently with the initial submission of this work, [66] studied automatic debiased machine learning for dynamic treatment effects. Our development is most dis- tinctive in the infinite-horizon setting, which not a generalized regression residual.
We discuss differences to other works in the line of work of "automatic" semiparametric estimation that are less directly related in Appendix A.
4 Empirical Gateaux Derivative of the Mean Potential Outcome
To illustrate the numerical approach to Gateaux derivatives, we work through the example of the mean potential outcome and indicate precisely how the numerical differentiation incurs error relative to the analytical derivative, which here is the doubly robust or augmented IPW (AIPW) estimator.
Relating the empirical and analytical Gateaux derivatives
Setup. We target the estimation of a mean potential outcome as in Example 1. For the smoothed perturbation, specialize to kernel functions K2 Kx(n) from kernel density estimation satisfying 1% K(u)du = (normalization to a probability density) and K(-u) K(u), for all u (symme- try). The mean potential outcome is identified as follows,
(P(P) = (E = EY A = 1, x]] = s yr(y A = 1, x)ddd = f f P(y.A-1.)) p(x) dydd.. p(A=1,))
(2)
We obtain (P.) by plugging in the á probability density estimates. Note this is distinct from the plug-in estimator in empirical process theory, which plugs in the empirical CDF; we plug in statistical estimates. See Algorithm 1 for summary of the procedure.
EExmmples include Gaussian kernel with ((u) (2m) =221,, or uniform with K(u) 1/2l [|u] $1] We generally consider Kx(u)= - K (u/ A), K(x) K (zi,..., za).
5

---

Comparison of empirical Gateaux derivative and augmented inverse propensity weighting. We have argued that, in the context of statistical estimation, approximating Gateaux derivatives re- quires plug-in estimation of the statistical functional with estimated probabilities. Towards resolving QI, we observe that the perturbed probability densities induce the functional form of the nuisance functions in the analytical derivative. This is in contrast to the analytical derivative, which describes the estimand in terms of, c.g,, conditional expectations, and allows the analyst to choose the func- tional form for estimating the nuisance function.
We will denote the P-induced conditional expectation as
Ep Y 1,X = x| = yp(y A = 1,
For example, the density-induced conditional expectation of kernel density estimates is exactly Nadaraya-Wand Watson regression. Given this, we study empirical derivatives with kernel density esti- mation because they provide classic textbook example of density estimates that induce conditional expectation estimators. Other density estimation approaches can be used and their rates of con- vergence would simply be substituted in Lemma 1. In Appendix D we discuss other choices of density estimates. Of course, all such nonparametric estimation approaches suffer from the curse of dimensionality, additional structure is required to satisfy the product-rate conditions.
Our characterization will show exactly how plug-in evaluation with finite differences is nearly equiv- alent to evaluating the doubly-robust estimator with a Nadaraya- Watson regressor as the conditional outcome regression for E A = 1, X] and a kernel density estimate of the propensity score. is equivalent up to an additive bias that arises due to the smoothed perturbation distribution, and what we will term smoothed nuisance evaluation. We discuss what this smoothed nuisance function is, and how we study its approximation error relative to estimation error of the induced nuisance.
Integrating the smoothed perturbation induces smoothed nuisance functions. Integrating with respect to the smoothed perturbation distribution can be interpreted as a smoothed evaluation of the resulting nuisance functions. We define the j² -smoothed conditional expectation which smooths the evaluation point, i.e., smooths evaluation around xn rather than precisely at z0.
Definition 1 (62, smoothed conditional expectation).
Ep A 1, = _o] = Ep|Y = = u/ 8 (u) du.
(3)
A similar definition appears in the literature on nonregular inference or the localization of a global functional [57]. The following lemma, similar van der Laan et al. [72, Lemma 25.1], summarizes convergence rate implications of these smoothed nuisances; these are direct consequences of typical analyses of kernel density estimation.
Lemma 1. Let (X) nuisances induced by plug-in estimation of probability distributions P. For any bounded function g(æ) in a Hölder class of degree B. (J ()))((() du -g(x))² dr = 0()*). Assume that 8* is product kernel in Hölder class of degree B. and f /u18182) du oo and f 8 du f S 3. Assume Y is bounded. Then the perturbed muisances satisfy the following rates:
1 A and (X) Pc(A 1, ((X) denote the
(X) x(X))| = (IA(x) - x(X)]l + O(eA -d22), (X) - e(x) = (|((x) - e(x) + O(e) 4/2).
The perturbed nuisances are asymptotically consistent, so long as P induces asymptotically con- sistent nuisances and we fix e as a sequence vanishing in n at rate appropriate to counteract the growth in Å due to "roughness"; i.e., the variability of 8A, The difference between the function eval- uation and its smoothed nuisance arises from the bias analysis of kernel density estimators and the dimension dimensioon-dependence of ensuring absolute continuity. but it is non-stochastic, entirely determin- istic argument. While the analysis improves mildly the dimension dependence on À compared to the general analysis of Carone et al. [10. Thm. 5]. this still highlights the generally unfavorable dependence of the numerical approach on the dimension due to smoothing.
'For example, van der Laan al. [72] proposes smoothing a nonregular target estimand, such dose- response curve or density evaluated at point Fo. by conducting smoothed evaluation locally around zo. In that work, function evaluated at 2o. ((2)), ) is approximated with a kernel smooth over x in a neighborhood of æn with bandwidth h;; so Vx (xx) = Kx(u u zo (u)d( Jung et al. [44] use a similar smoothing/localization device but for different problem.

---

Algorithm Empirical Gateaux derivatives
1: Inputs: P probability density estimates, data {0:) perturbation and smoothing (c, A) param-
eters, functional y
2: for = I,... do
3: (1 -1-1) (1-c))++ 60.
4:
-(P)))
5: end for
6: Output: one-step adjusted estimate from empirical Gateaux derivatives, V(P) + 1/2 E ((os)
Our first proposition establishes the exact approximation emor induced by finite differencing. Proposition (Gateaux derivative of probability density representation). Consider perturbations in the direction of O; (a, az, Vi). Let p(x), P(A = 1,x),P(U. A = 1, denote kernel density estimates. Let (P) be as in Equation (2):
((P))-((P))
- f
p(x) [a 1] 02, (2:)
{0
y & (o)aa)))
Eply 1= 1, = =x0 21) dx
E
(A = 1,
+ (E A -= 2.1 (P))
We compare the expansion of Proposition to the canonical AIPW estimator in Example 2. We spe- cialize to uniform kernel for 5, (v) so that f 5y, (v)dy = yi. We next observe that the smoothing kernel introduces additive bias relative to evaluating terms in the expression at the observation os.
Corollary 1. When the perturbation observation is the observation datapoint O, = (X;, A,, Yi).
[A (Y; - EE|Y A x,) + (Ep. Y A= - v()) 3(O,) (A=IIX])
0(
Lemma allows us to further simplify and conclude that the decomposition of Proposition is close to the canonical AIPW estimator up the O(2)) bias induced by smoothed evaluation.
Implications for estimation Next, we use our characterization Proposition 1, Lemma and Corollary to study the stutistical properties of our computationnl//uueeee
approximation. Corol- lary arises from deterministic equivalences and allows us to study the relationship to the typical "oracle" AIPW estimator; albeit with induced e-perturbed nuisances. An appealing property of bias-adjusted treatment effect estimation rate double robustness: because we incur the product of convergence rates of the nuisances, we may enjoy parametric n rate convergence of the target functional while nuisances converge at slower rates, for example at n n- see [13, 63). We use the rate double robustness property of the target and Lemma to infer the required rate conditions on (e, A) that retain the beneficial statistical properties of the canonical AIPW estimator. The next result combines our previous characterizations with the standard analysis of AIPW.
Assumption (Regularity conditions). Assume the following regularity conditions hold:
(i) Y bounded.
(ii) ply A=
are in Hölder classes of minimum degree B.
(ii) The estimates ply, a, x),, p(a, z), p(x) belong to a Donsker function class.
(iv) Assume satisfy the product-rate condition: ||ã(X) μ(X)] [[((X) e(X)ll (n ju, é RMSE-consistent with ruy Te. respectively, that rn re 2 2. Theorem (Rate double robustness). Consider the one step-estimator with the empirical Gateaux derivative. Under Assumption 1, when eA -d22 o(n max((r))r))), and X = o(n" =b):
((()) + + vP)) P) = On +).
We now interpret the qualitative estimation implications of the analysis. Theorem states the rate conditions required of numerical approximation in e, À to preserve the statistical property of rate
7

---

16:55
4c-01
6e-03
Se-04
4e-04
0e-04
DM
AIPW
0.2
0.001
4e-01
4e-02
0e
02
4e
02
0.0
5e-01
0e-01
01
3e-01
0e-01
-0.4
1e+00
9e-01
9e:01
9e-01
-0.6
0.1
0.01
0.001
0.0001
10:55
10a
10a
(b)
Figure 1: (a) Epsilon-lambla plot for mean potential outcome: MAE in approximation of one-step
adjustment. (b) Error in estimation of (1) over
double robustness. Assuming that the nuisances induced by the probability density estimates satisfy a product-rate condition that the product their RMSES faster than the parametric Onn rate, the form of Proposition 1 suggests that the rate of RMSE convergence of perturbed nuisances is required to be faster than the rates of the unperturbed nuisance functions, max(T Te), which can be a slower rate than the For a direct comparison, the rate of Carone al. [10, Thm. 5] suggests a rate requirement of (ex-d.)*= + X-2 o(n a).
o(n à) implied by the generic analysis of finite differences.
In general, the dimension dependence on À in Lemma will prevent setting a slower rate for e for nontrivial dimensionality. Note that without the smoothing requirement, the potential improvement in the rate of e could be on the order of generic rate improvements implied by a central difference scheme. That is, we would have attained the approximation benefits of generically approximation- improved finite diffe-iiffeeenc schemes via the specialized statistical structure of the target adjustments.
Applications: Mean Potential Outcome and Dynamic Settings
As empirical illustrations, first show plot of numerical approximation error ranging over (c, A) in the simple case the mean potential outcome. We then apply the tools that we derived in the previous section to analyze more complicated settings.
5.1 Mean potential outcome
We first conduct validation of our decomposition and theoretical characterization. Our data- generating process includes a piecewise linear outcome model (such that kernel regression is mis- specified). conduct plug-in evaluation the mean potential outcome with kernel density esti- mates. In Figure la we include an (e, A)-plot for the simple case of AIPW (see [10] for more discus- sion and examples). linear Y, and smooth propensity scores that logistic in sin(X). We use 500 and fix the bandwidth h = 0.05. Colors denote magnitude of the mean absolute error, included in text on the heatmap. Without loss of generality, study the estimation of mean under missingness, E|Y (1)]- Figure 1b illustrates the estimation eтror of various strategies with the comparable kernel-based es- timates (DM is regression adjustment). Note that in this simple setting AIPW offers benefits and sample efficiency relative to DM. We include further numerical experiments in the Appendix.
consider a one-dimensional case with uniformly distributed X, piecewise-
5.2 Dynamic treatment regimes
We study the estimation properties of the empirical Gateaux derivative for multi-stage DTR from the probability density representation. T-stage dynamic treatment regimes, the causal quantity of interest is the mean potential outcome E (ā)), where ā = (ao, ... ,ay) is the (deterministic) treat- ment strategy. Assuming Y (a) is sequentially ignorable given the treatment and covariate history, (A,, Xz), each time f, this causal quantity can be identified by the g-foruuaa,
E ( f E [Y À a. X = I1TT_ p(1, ă Tt-1) dž,
8

---

where A = - :.).....AA)... = (Xo...... X2). To derive its influence function, we take the empirical Gateaux derivative by considering the perturbation in the direction of o; (ã ais yi), where T; (I:0..... Fitt and a; (d:0, ,iit).
v(P)) (f (1 dy) II
De - Pe (2o) dr. (c(--11
(A= P
Below we characterize how this empirical Gateaux derivative at the smoothed distribution differs from the one the (unsmoothed) estimated distribution. The following result is analogous to Propo- sition I and Corollary 1 but is extended to the dynamic treatment regime. Proposition 2.
(V(PP
- v(())
= (Bp(EIT)) [Bp [Y [ [o] - (P)) + 1/Ama:]
[Y, - BB(x)
Ep (Y []]]]]
(a 2)
ΣT p.(uT - (Br->)
[|A=ū,
E Ep, Y A X]
(5)
-Ep [E [YIAXX] a..rs,, +
The expansion verifies that the requirements for numerical approximation are similar in c as in the case of a single-timestep mean potential outcome, despite the additional complexity of the dynamic treatment regime function as nested expectation. Again, smoothing incurs overall unfavorable dependence in the dimension, vorable dependence on the dimension. The history nuisances imply that the dimension additionally grows in the time horizon.
the extension of Lemma 1 to E. [Y A, X] also incurs unfa-
5.3 Perturbations for (constrained) infinite-horizon off-policy evaluation
We show how this framework can be applied to derive generic bias corrections for infinite-horizon off-policy optimization in offline reinforcement learning, via the canonical linear programming char- acterization [17, 61]. Our application is relevant to recent line of work on off-policy evaluation and learning (OPF/L.) [5, 43, 47-49, 67, 68).
Setup. We recall the classical linear programming formulation of finding an optimal policy a tabular (finite-state, finite-action) Markov decision process, due to De Farias and Van Roy [17]. Put- erman [61]. Recent work has revisited this formulation with interest developing primal-dual algorithms [1, 55, 65, 74]. The dynamics follow an infinite-horizon Markov decision process with discount factor %. The offline dataset is comprised of trajectories of state, action, next state ob- servations: {(5) of....,,',, st....,,$ sf " We derive our bias adjustment under statistical model where the stationary distribution factorizes as p(s, a, s') = d(s, a) P(s 8, a). From the joint state- action-state occupancy distribution p(s, a, 8'), we estimate the marginalization d(s, a), the sta- tionary state-action occupancy distribution, which we use to estimate the transition probability P(s' a, s) p(s.a.e"), Let Ho(s) denote the initial state distribution, estimated from offline data and assumed be equivalent in the original confounded and evaluation setting. Because we focus on the tabular case with discrete distributions, we do not require smoothing.
d(s.a)
The optimal policy linear program. The optimal value function solves the following linear pro- gram in a finite-state and finite-action setting; the objective value at optimality is the policy value. Here, Pa E |||S is transition matrix at action a, with (a marginal occupancy distribution of the initial state. We assume, as is fairly common in theoretical analyses of reinforcement learning, that the reward function is a known function of state and action, r(s, a). Let ra {r(s,a)}<es be the vector of reward values per state, for fixed action a. We have:
P(s' s, a), (5)(s) is the
v (P) min - 7)po TV: (I PPa) V r 2 O, Va E A}
(6)
The dual formulation is well known to parametrize the stationary occupancy probabilities of the optimal policy (at optimality):
V p (P) max E.CAMore
- = ->)pu, Ha 20 Va E A]. (7)

---

We verify the bias correction from the Gateaux derivative de (P.) this functional the perturbed distribution evaluates the linear program with perturbed left-hand side coefficients; i.e., perturbing the constraints of the coefficient matrix as well as the objective (marginalizing over perturbed initial state distribution. We evaluate the Gateaux derivative in the direction of (8, 0, 8) We assume nondegeneracy (unique optimal solution); this can be relaxed. Assumption 2. The optimal solution is unique; there is degeneracy.
c=0 Note that evaluating
Proposition 3. Perturb in the direction of a generic observation (8, a'))
d
(5, d(3.a) (55.0)+vV7 + - (P))
de (P() ==0 (1-7)V
(8)
Sketch of argument. The statement of Proposition is not surprising since we can immediately verify the mean-zero sample version of the Bellman residual, and so one might plausibly derive this form from double robustness. Rather, we provide an alternative argument based on the sensitivity analysis characterization of [25].For small enough, the active basis remains the same. When the perturbation matrix is P + eG for some matrix G. [25] notes that the derivative of the optimal value is d' (c) = -G (where (τ are the dual and primal optimal solutions, respectively) can be obtained from the derivative of a matrix inverse, which yields higher-order expansions.
We next compute the approximation error that arises from evaluating finite differences, so long as small enough to maintain the same active basis, which is empirically verifiable.
Proposition 4 (Error analysis from finite differencing). Perturb in the direction of a generic obser- vation (5, a, 3.) Then:
(3, à) (5, a)
E - (1-7)
(r(3,a) + -
(9)
In this setting, unlike the earlier case of the treatment mean in Corollary and Theorem 1, even in a favorable discrete-state case without smoothing requirements, rate double robustness does not admit weaker numerical requirements on c. In the analysis, eq. (45) shows that V((s) V(s) O(c), so that Proposition 4 holds with the unperturbed nuisances and the same order of approximation error.
Generalization to constraints. We argue the main benefit of this approach is that by computing these finite differences, we can obtain generic bias corrections via a more broadly applicable argu- ment that does not require interpretation of the closed-form solution to an optimization problem. For example, consider a relevant subelass of constrained Markov decision processes comprised of additional constraints on the policy variables: the linear programming formulation is particularly appealing because linear constraints can be directly added [3]. The additional constraint E P re- flects linearly representable constraints on occepaccyy under the optimal policy. For example, relevant constraints could require the occupancy in certain risky states to be bounded. [59] develops a framework with a "caution function," " convex function on the atatin occupancy measures. Example 3 (Constrained policy optimization).
PP) max {EaEA Pit Ta: Encall -PT 1 7))o, Ha 2 O, Va A, pEP)
6
Conclusions
We have presented constructive algorithm that approximates Gateaux derivatives finite differenc- ing, with focus on the statistical functionals used in causal inference. There are several directions which are natural follow-ups to this work. First, the smoothing perturbation, although it restores absolute continuity, introduces technical barriers in the implied approximation rates with regards to dimension dependence. Another useful direction would be to explore methods that are fully adap- tive, not requiring any prior knowledge of beneficial statistical properties. Finally, our work has been mainly theoretical, and it would be useful to conduct further empirical comparisons.
Acknowledgements Angela Zhou gratefully acknowledges support from the Foundations of Data Science Institute and the Simons Institute's 's Program on Causality. Part of this work was done while the author was visiting the Simons Institute for the Theory of Computing. Yixin Wang acknowledges National Science Foundation Grant NSF-CHE-2231 174. Michael Jordan acknowledges support from the Vannevar Bush Faculty Fellowship program under grant number N00014-21-1-2941.
10

---

References
[1] Y. Abbasi- Yadkoi, P. Bartlett, Chen, and Malek. Large-scale Markov decision prob- lems via the linear programming dual. arXiv preprint arX1v:1901.01992, 2019. (Cited on
9)
[2] A. Alaa and M. Van Der Schaar. Validating causal inference models via influence functions. In International Conference on Machine Learning, pages 191-201. PMLR, 2019. (Cited on 16)
[3] E. Altman, Constrained Markov Decision Processes: Stochastic Modeling. Routledge, 1999. (Cited on 10)
[4] R. Bhattacharya, R. Nabi, and I. Shpitser. Semiparametric inference for causal effects in graph- ical models with hidden variables. arXiv preprint arXiv:2003. /2659, 2020. (Cited on 5, 16)
[5] A. Bibaut, Malenica, N. Vlassis, and M. J. van der Laan. More efficient off-policy evalua- tion through regularized targeted learning. arXiv preprint arXiv: 192.02291..00222. 2019,, (Cited on
2, 9)
[6] Bickel and Y. Ritov. Nonparametric estimators which can be "plugged-in", The Annals of Statistics, 3 3144):0033-100 2003. (Cited on 31)
[7] P. J. Bickel, C. A. Klaassen, Y. Ritov, and J. A. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press, 1993. (Cited 5)
[8] F. Bonnans and A. Shapiro. Optimization problems with perturbations: A guided tour. SIAM Review, 40(2):228-264, 1998. (Cited on 31)
[9] Bravo, C. Escanciano, and I. Van Keilegom. Two-step semiparametric empirical likelihood inference. The Annals of Statistics, 48(1):1-26, 2020. (Cited on 5)
[10] M. Carone, A. R. Luedtke, and M. van der Laan. Toward computerized efficient estimation in dimenteddimensionall models. Journal of the American Statistical Association, 2018. (Cited
on 2, 3, 4, 5, 6, 8)
[11] B. Chakraborty and E. Moodie. Statistical Methods for Dynamic Treatment Regimes. Springer. (Cited on
[12] V. Chermozhukov, J. C. Escanciano, H. Ichimura, W. K. Newey, and J. M. Robins. Locally robust semiparametric estimation. arXiv preprint arXiv: 00033, 2016. (Cited on 1)
[13] V. Cheroozuuko,, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018. (Cited on 1,
7)
[14] V. Chermozhukov, W. Newey, R. Singh, and V. Syrgkanis. Adversarial estimation of Riesz representers. arXiv preprint arXiv:2101.0000 2020. (Cited on 5, 16)
[15] V. Chernozhukov, W. K. Newey, V. Quintas-Martinez, and V. Syrgkamis. Automatic de- biased machine learning via neural nets for generalized linear regression. arXiv preprint arXiv:2/04.1 14737, 2021. (Cited on 16)
[16] V. Chernozhukov, W. Newey, V. M. Quintas-Martinez, and V. Syrgkanis. Riesznet and fore- striesz: Automatic debiased machine learning with neural nets and random forests. In In- ternational Conference on Machine Learning, pages 3901-3914, PMLR, 2022. (Cited on 5, 16)
[17] D. De Farias and B. Van Roy. The linear programming approach to approximate dynamic programming. Operations Research, 51(6):850-865, 2003. (Cited on 9)
[18] J. C. Duchi and Ruan. Asymptotic optimality in stochastic optimization. The Annals of Statistics, 49 (9(11:21-43 2021. (Cited on 31)
[19] J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations, IEEE Transactions on Informa- tion Theory, 61(5))2788-2800 2015. (Cited on 30)

---

[20] M. H. Farrell, T. Liang, and S. Misra. Deep learning for individual heterogeneity; an automatic inference framework. 2021. (Cited on 5, 16)
[21] A. Fisher and E. H. Kennedy. Visually communicating and teaching intuition for influence functions. The American Statistician, 75(2):162-172, 2021. (Cited on 5)
[22] A. D. Flaxman, A. T. Kalai, and McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. pages 285-294, 2005. (Cited on 30)
[23] B. Fornberg. Numerical differentiation of analytic functions. ACM Transactions Mathe- matical Software (TOMS), 7(4):512-526, 1981. (Cited on 2)
[24] C. E. Frangakis, T. Qian, Z. Wu, and I. Diaz. Deductive derivation and Turing-compinn compteriammtiiiiiiioonn of semiparametric efficient estimation. Biometrics, 71(4):867-874, 2015. (Cited on 2, 4, 5)
[25] R. M. Freund. Postoptimal analysis of a linear program under simultaneous changes in matrix coefficients. In Mathematical Programming Essays in Honor of George B. Dantzig Part pages 1-13. Springer, 1985. (Cited on 10, 28, 29)
[26] R. Giordano, M. I. Jordan, and T. Broderick. A higher-order swiss army infinitesimal jackknffe. arXiv preprint
1tt.. 2019. (Cited on 16)
[27] R. Giordano, W. Stephenson, R. Liu, M. Jordan, and T. Broderick. A swiss army infinitesimal jackknife. The 22nd International Conference on Artificial Intelligence and Statistics, pages 1139-1147. PMLR, 2019. (Cited on 16)
[28] M. C. Grant and S. P. Boyd. Graph implementations for nonsmooth convex programs. In Recent Advances in Learning and Control, pages 95-110. Springer, 2008 (Cited on 31)
[29] W. Guo, M. I. Jordan, and A. Zhou. response. arXiv preprint urXiv:2202. /2958, 2022. (Cited on 16)
evaluation with policy-dependent optimization
[30] V. Gupta, M. Huang, and P. Rusmevichientong. Debiasing iample policy performance for small-data, large-scale optimization. arXiv preprini arXiv:2107.12443 2021. (Cited 16)
[31] Hahn. On the role of the propensity score efficient semiparametric estimation of average treatment effects. Econometrica, pages 315-331, 1998. (Cited on 4)
[32] R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346):383-39, 1974. (Cited on 3)
[33] L. A. Hannah, D. M. Blei, and W. B. Powell. Dirichlet process mixtures of generalized linear models. Journal of Machine Learning Research, 12(6), 2011. (Cited on 31)
[34] J. A. Hausman and W. K. Newey. Nonparametric estimation of exact consumers surplus and deadweight loss. Econometrica, pages 1445-1476, 1995. (Cited on
[35] O. Hines, O. Dukes, K. Diaz-Ordaz, and S. Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician, pages 1-13, 2022. (Cited on 4. 4,
1,
16)
[36] H. Hong and J. Li. The numerical delta method. Journal of Econometrics, 206(2):379-394 2018. (Cited on 30)
[37] H. Hong, A. Mahajan, and D. Nekipelov. Extremum estimation and numerical derivatives. Journal of Econometrics, 188(1):250-2633 2015. (Cited on 30)
[38] Huber. Robust Statistics, volume 523. John Wiley & Sons, 2004. (Cited on 2, 3)
[39] H. Ichimura and W. K. Newey. The influence function of semiparametric estimators. arXiv 1508.0/378, 2015. (Cited on 2, 3, 16,31) 31)
[40] H. Ichimura and W. K. Newey. The influence function of semiparametric estimators. Quanti- tative Economics, 13(1):29611 2022. (Cited 16)
12

---

[41] S. Ito, A. Yabe, and R. Fujimaki. Unbiased objective estimation in predictive optimization. In International Conference on Machine Learning, pages 2176-2185, PMLR, 2018. (Cited on 16)
[42] A. Jesson, S. Mindermann, Y. Gal, and U. Shalit. Quantifying ignorance in individual- causal-effect estimates under hidden confounding. In International Conference on Machine Learning, pages 4829-4838. PMLR, 2021. (Cited on 31)
[43] N. Jiang and L. Li. Doubly robust off off-po value evaluation for reinforcement learning. Proceedings of the 33rd International Conference Machine Learning, 2016. (Cited on 2, 9)
[44] Y. Jung, J. Tian, and E. Bareinboim. Double machine learning density estimation for local treatment effects with instruments. Advances in Neural Information Processing Systems, 34: 21821-21833, 2021. (Cited on 6)
[45] Y. Jung, J. Tian, and E. Bareinboim. Estimating identifiable causal effects through double machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13-121222, 2021. (Cited on 5)
[46] N. Kallus and M. Uehara. Double reinforcement learning for efficient off-policy evaluation in Markov decision processes. Journal of Machine Learning Research, 21(167):1-63, 2020. (Cited on 2)
[47] N. Kallus and M. Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research, 2022. (Cited 9)
[48] N. Kallus and A. Zhou. Confounding-robute policy evaluation in infinite-horizon reinforce- ment learning. Advances in Neural Information Processing Systems, 33:22293-22304, 2020.
[49] N. Kallus and A. Zhou. Stateful offline contextual policy evaluation and learning, In Interna- tional Conference on Artificial Intelligence and Statistics, pages 1199-111144 PMLR, 2022. (Cited on 9)
[50] N. Kallus, X. Mao, and M. Uehara. Localized debiased machine learning: Efficient inference on quantile treatment effects and beyond. arXiv preprint arXhv:1912.12945 2019. (Cited on 31)
[51] K. Kandasamy, A. Krishnamurthy, B. Poczos, L. Wasserman, et al. Nonparametric von mises estimators for entropies, divergences and mutual informations. Advances in Neural Informa- tion Processing Systems, 28, 2015. (Cited on 5, 16)
[52] E. H. Kennedy. Semiparametric theory and empirical processes causal inference. In Sta- ristical Causal Inferences and their Applications in Public Health Research, pages 141-167. Springer, 2016. (Cited on 1)
[53] E. H. Kennedy. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.0644 2022. (Cited 2, 5, 16)
[54] W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pages 1885-1894. PMLR, 2017. (Cited on
16)
[55] D. Lee and N. He. Stochastic primal-dual Q-learning. arXiv preprint 17X00:18100088298,,00 2018. (Cited on 9)
[56] W. K. Newey. The asymptotic variance of semiparametric estimators. 1349-1382, 1994. (Cited on 1)
Econometrica, 62:
[57] W. K. Newey. Kernel estimation of partial means and general variance estimator. Economet- ric Theory, 10(2):1-21, 1994. (Cited on 2, 6)
[58] D. Oktay, N. McGreivy, J. Aduol, Beatson, and R. P. Adams. Randomized automatic differentiation. arXiv preprint 1011:20007.14422 2020. (Cited on 30)
13

---

[59] S. Paternain, L. Chamon, M. Calvo-Fullans, and A. Ribeiro. Constrained reinforcement learn- ing has zero duality gap. Advances Neural Information Processing Systems, 32, 2019, (Cited on 10)
[60] J. Pfanzagl. Estimation in semiparametric models. In Estimation in Semiparametric Models, pages 17-22. Springer, 1990. (Cited on 4)
[61] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, 2014. (Cited on
[62] M. Robins, A. Rotnitzky, and L. Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427): 846-866, 1994. (Cited on 1, 3, 4)
[63] A. Rotnitzky, E. Smucler, and M. Robins. Characterization of parameters with a mixed bias property. Biometrika, (08(1)231-236) 2021. (Cited on 7)
[64] T. Sauer. Numerical Analysis. Pearson, 2017. (Cited on
[65] J. B. Serrano and G. Neu. Faster saddle-point optimization for solving large-scale Markov decision processes. In Learning for Dynamics and Control, pages 113-423. PMLR, 2020. (Cited on
[66] R. Singh and V. Syrgkanis. Automatic debiased machine learning for dynamic treatment ef- fects. arXiv preprint arXiv:2203. 13887, 2022. (Cited on 5)
[67] 7. Tang, Y. Feng, L.. Li, D. Zhou, and Q. Liu. Doubly robust bias reduction in infinite horizon off off-policy estimation. arXiv preprint 190.07199 2019. (Cited on 2, 9)
[68] Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learn- ing. Journal of Machine Learning Research. 2016. (Cited 2. 9)
[69] A. A. Tsiatis. Semiparametric Theory and Missing Data. Springer, 2006. (Cited 5, 16)
[70] A. Ullah and A. Pagan. Nonparametric Econometrics. Cambridge University Press, 1999. (Cited on 18)
[71] M. J. Van der Laan and J. M. Robins. Unified Methods for Censored Longitudinal Data and Causality. Springer, 2003. (Cited on 2)
[72] M. J. van der Laan, A. Bibaut, and A. R. Luedtke. CV-TMLE for nonpathwise differentiable target parameters. In Targeted Learning in Data Science, pages 455-481. Springer, 2018. (Cited on 6)
[73] A. W. Van der Vaart. Asymptotic Statistics. Cambridge University Press, 2000. (Cited on 3, 19)
[74] M. Wang. Randomized linear programming solves the discounted Markov decision problem in nearly-linear running time. Mathematics of Operations Research, 45:403-795, 2020. (Cited
on 9)
[75] A. Wilson, M. Kasy, and L. Mackey. Approximate cross-validation: Guarantees for model assessment and selection. In International Conference on Artificial Intelligence and Statistics, pages 4530-4540. PMLR, 2020. (Cited on 16)
14


Reviews:
[{"review_id": "raLi3WPAAVk", "reviewer": "Reviewer_54Sg", "summary": "Efficient / doubly-robust estimation in causal inference typically requires constructing estimators based on the (efficient) influence function, which is derived case by case. The paper considers numerical approximation to the influence function with *empirical Gateaux derivatives*, which are obtained through finite differencing and smoothed perturbation. The approximation uses plugin estimate of the probability density. The paper studies conditions on the smoothing and perturbation parameter such that the resulting one-step estimator retains desired statistical properties. The paper first examines the case of estimating the counterfactual mean and then extends to more complex dynamic settings. ", "questions": "1. Gateaux differentiable vs influence function\n\n(1) The definition of the influence function ($\\S$2) in terms of Gateaux derivative comes from the robust statistics literature. How is it related to the influence function used for semiparametric inference?\n\n(2) page 2, line 59-60: in the direction of $H$ or for all $H$?\n\n2. On plugin estimation through density estimation\n\n(1) Given the curse of dimensionality, it is unclear to me why the approximation must be based on a density estimate. I do not parse the argument provided in line 117-121: can the authors elaborate on this? More specifically, for the running example, it seems that one can easily simulate data from the perturbed distribution. Can the approximation be based on an estimator using the simulated perturbed data (e.g., empirical average of an estimated conditional mean, where the distribution is that of the simulated data)?\n\n(2) Suppose density estimation is necessary. Then it is worth discussing when the kernel-based estimators can fulfill the conditions required, e.g., (iv) in Assumption 1. \n\n(3) To really \"bolster confidence in the numerical procedure\", the case study of the running example should be expanded accordingly to show that the proposed method works in a range of settings (e.g., at least when dimension is not high). \n\n3. I have a hard time trying to follow the extension to dynamic settings in $\\S$4. Can the authors summarize the take-away message there?\n\n4. A few points worth clarification. \n\n(1) How is $\\tilde{P}$ used in Algorithm 1?\n\n(2) line 71: it is unclear what \"$\\Psi(P_{\\epsilon,\\lambda}^{\\ast})$ is the projection of $\\Psi(P_{\\epsilon,\\lambda})$ ...\" means. \n\n5. On the relation to previous works in “automatic” or “computerized” semiparametric inference\n\nThe paper features a long list of references. Since the focus here is the influence function, I would suggest adding a paragraph after $\\S$2, which clarifies the relation to previous papers on \"computerizing\" influence-function based estimations, e.g., [9] and [17] in the bibliography. ", "limitations": "I do not foresee a potential negative societal impact of the proposed work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The paper is built upon recent efforts in \"automating / computerizing\" influence-function-based estimation and studies an important question. \n2. The paper provides a clean analysis for the basic case of estimating the counterfactual mean. \n\n### Weaknesses\n1. The approximation is based on a density estimator of the underlying data distribution, which seems to bear serious practical limitations. It is unclear if this choice is necessary, and if so, how to overcome the limitations. \n2. The conditions impose a product-form rate condition on the smoothed estimates; see (iv) of Assumption 1. It is unclear when these conditions can be met in practice. \n3. Other than a simple simulation, the paper does not seem to provide concrete practical recipes for the running example that can flexibly incorporate different underlying distributions, dimensions, smoothness, etc. This weakens the relevance for the results developed for the more complex, dynamic settings. \n4. The clarity of certain sections can be improved. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RWJXtkBWopS", "reviewer": "Reviewer_cYXd", "summary": "Gateaux derivatives arise in a number of statistical estimation problems, and are used to improve asymptotic efficiency in estimation (e.g., in one-step estimators).  For a given problem, the form of the Gateaux derivative is typically derived analytically, and the relevant components then estimated from data (e.g., certain conditional expectations).  Here, the form of the derivative is known, and typical analysis focuses on how different rates of estimation (in the relevant nuisance parameters) lead to appropriate rates of convergence in the original problem.\n\nThis paper considers \"empirical\" Gateaux derivatives, which do not involve such analytic derivations, but instead directly seek to estimate the derivative via finite-differencing.  This introduces an additional source of error in computing the Gateaux derivatives (numerical approximation error), but has the potential to broader the utility of one-step estimators, by not requiring explicit derivations in the case of new statistical functionals.\n\nWhile other work has considered the problem of numerical approximation error, this work analyzes how both numerical and statistical errors interact in the context of several examples: One of the main results, for instance, gives rates on numerical approximation hyperparameters that preserve parametric rates in the one-step estimator for a standard causal inference problem (counterfactual mean estimation).  Two other illustrative estimation problems are considered as well.\n", "questions": "My questions can be found in the \"Strengths and Weaknesses\" section, where every bullet point corresponds to a question.  I collect them (verbatim) below.\n\n* Line 188: \"can be a slower rate than implied by the generic analysis of finite differences\".  What kind of rate would that be, and is there a reference for such results?\n* Lines 190-191: \"potential improvement...could be on the order of generic rate improvements implied by a central difference scheme\".  What is this referring to?\n* Lines 278-280: \"does not appear that rate-double robustness would admit weaker numerical requirements on $\\epsilon$\".  Weaker requirements than what?  The \"generic analysis\" referenced above?\n* Are there \"conservative\" rates on $(\\epsilon, \\lambda)$ that will always preserve $O_P(n^{-1/2})$ rates of estimation, obtainable via some generic analysis?\n* Proposition 2 (DTR) \"verifies that the requirements...are similar in $\\epsilon$ as in the case of a single-timestep\", but there is no $O(\\epsilon^2)$ term in either Proposition 1 or Corollary 1.  Could you clarify what is meant here?\n* Propositions 3 & 4 differ not only in an additive term, but also in the usage of perturbed nuisances, which makes them difficult to compare directly (this also applies to Corollary 1, as noted on line 170).  Is there a reason why a direct comparison (e.g., isolating only an additive difference) is unnecessary here?\n* Are there existing scenarios where the analytic form of the gateaux derivative is non-obvious, but where these conditions (pathwise differentiability, second-order remainder) can nonetheless be verified to hold? Or does verifying these conditions *always* require derivation of the analytic form?\n* More broadly, are there scenarios where we can apply this approach (with appropriately conservative rates on $(\\epsilon, \\lambda)$) and have confidence in achieving $O_P(n^{-1/2})$ rates, **without** deriving the analytic derivative?  E.g., would the constrained MDP with arbitrary linear constraints be such an example?\n* In the display following line 563, it is claimed that the following holds due to Cauchy-Schwarz.  I'm not sure I see the application of CS here: Is there another reason why we would expect the cross term $2 E[(\\tilde{\\mu}_{\\epsilon}(X) - \\tilde{\\mu}(X))(\\tilde{\\mu}(X) - \\mu(X))]$ to be non-positive?\n\n$$ E(\\tilde{\\mu}\\_{\\epsilon(X)} - \\mu(X))^2 \\leq E(\\tilde{\\mu}_{\\epsilon}(X) - \\tilde{\\mu}(X))^2 + E(\\tilde{\\mu}(X) - \\mu(X))^2 $$\n* In the display following line 562, the last inequality seems like a non-trivial jump, would you mind walking through the logic explicitly?\n", "limitations": "I think the authors do a fine job of explaining limitations of the approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[**REBUTTAL UPDATE**: See discussion below.  My main concerns were addressed during the response, and I am convinced that the relevant clarifications are straightforward to incorporate into a camera-ready, if accepted.  My score has been updated as follows:\n* Presentation: 2 -> 3\n* Overall Score: 5 (borderline accept) -> 7 (accept)\n]\n\n### Strengths\nThis paper tackles an interesting and highly relevant problem, and provides novel technical results. Theorem 1 in particular is a clean and informative result, characterizing rates of perturbation and smoothing that preserve $\\sqrt{n}$-consistency of counterfactual mean estimation with empirical Gateaux derivatives.  Overall, one of the main take-away messages is that the structure of the statistical problem can lead to weaker approximation requirements (e.g., rates on $(\\epsilon, \\lambda)$) in some cases, but not in others, which I thought was an interesting point.\n\n### Weaknesses\n\nThe clarity and quality of presentation could be improved, which is a weakness that is hopefully straightforward to fix.  Below, I will detail some points that were particularly unclear to me, in the hope that it is useful for the authors as they craft a response. I put explicit questions in \"bullet points\", with the remaining text as context.  I am willing to increase my score if these points can be sufficiently clarified.\n\n**(1) Unclear significance of analysis of required rates for $(\\epsilon, \\lambda)$**\n\nAs I understand it, the main goal of the analysis is to understand the extra error incurred by using a finite-differencing approach, above and beyond using analytic derivatives:  As in Theorem 1, if this error decays fast enough, then one can use empirical derivatives while preserving $O_p(n^{-1/2})$ rates of estimation.  This seems like a very clear type of result, but for an application where the analytic derivative is already well known.  Meanwhile, the results in the remaining sections do not go \"all the way\" to a result like Theorem 1, but instead stop at giving some side-by-side comparison of the analytic and empirical derivatives.\n\nIt seems that there are two conclusions the reader is supposed to draw from this analysis, in the context of the remainder of the work:  First, that $(\\epsilon, \\lambda)$ can decay slower than we might generically \"expect\", for problems with special structure. Second, that this special structure is present in the dynamic treatment regime (DTR) functional, but not in the policy optimization functional.  This second point is implied to be surprising, because all three problems exhibit some double-robustness structure (see lines 279-280).\n\nI had trouble drawing such conclusions, though I suspect this is mostly an issue of presentation / clarity.\n\n(1a) First, it seems in several places that the reader should have a \"baseline\" result in mind, to contrast with the results presented here, but this baseline was not entirely clear.  A few examples:\n+ Line 188: \"can be a slower rate than implied by the generic analysis of finite differences\".  What kind of rate would that be, and is there a reference for such results?\n+ Lines 190-191: \"potential improvement...could be on the order of generic rate improvements implied by a central difference scheme\".  What is this referring to?\n+ Lines 278-280: \"does not appear that rate-double robustness would admit weaker numerical requirements on $\\epsilon$\".  Weaker requirements than what?  The \"generic analysis\" referenced above?\n+ Are there \"conservative\" rates on $(\\epsilon, \\lambda)$ that will always preserve $O_P(n^{-1/2})$ rates of estimation, obtainable via some generic analysis?\n\nThe first three questions are about contextualizing the results, but the last one this is important for clarifying whether (a) there is always a generic approach to derive rates on $(\\epsilon, \\lambda)$ that preserve $O_P(n^{-1/2})$ convergence, and this is just an improved analysis for specific estimands that shows slower rates are possible, or (b) it is generally necessary to do a \"Theorem 1-style\" analysis to verify $O_P(n^{-1/2})$ convergence.  The latter conclusion seems much more restrictive than the former.\n\n(1b) Second, conclusions are often made by comparing the form of the empirical and analytical derivatives directly, but these were somewhat difficult to follow:\n* Proposition 2 (DTR) \"verifies that the requirements...are similar in $\\epsilon$ as in the case of a single-timestep\", but there is no $O(\\epsilon^2)$ term in either Proposition 1 or Corollary 1.  Could you clarify what is meant here?\n* Propositions 3 & 4 differ not only in an additive term, but also in the usage of perturbed nuisances, which makes them difficult to compare directly (this also applies to Corollary 1, as noted on line 170).  Is there a reason why a direct comparison (e.g., isolating only an additive difference) is unnecessary here?\n\n**(2) Unclear significance of limitations of Empirical Gateaux derivatives**: \n\nAs outlined in the introduction, constructive / algorithmic approaches to bias adjustment are very appealing, particularly for problems where small changes require re-derivation of the analytic derivative. This paper strikes an appropriate note of humility in the conclusion, giving limitations of Empirical Gateaux derivatives as a \"completely general approach\" (lines 329-335), namely the fact that (a) pathwise differentiability and (b) the second-order nature of the remainder must be verified analytically. However, these limitations do seem to undercut the general value of the approach. With that in mind, a few relevant questions\n* Are there existing scenarios where the analytic form of the gateaux derivative is non-obvious, but where these conditions (pathwise differentiability, second-order remainder) can nonetheless be verified to hold? Or does verifying these conditions *always* require derivation of the analytic form?\n* More broadly, are there scenarios where we can apply this approach (with appropriately conservative rates on $(\\epsilon, \\lambda)$) and have confidence in achieving $O_P(n^{-1/2})$ rates, **without** deriving the analytic derivative?  E.g., would the constrained MDP with arbitrary linear constraints be such an example?\n\nIf there exist some space of problems where the answer to these questions is \"yes\", then it would go a long way towards mitigating the impact of these limitations.\n\n**Comments on soundness**: Regarding technical soundness, I have a (hopefully minor) question or two on Lemma 1\n* In the display following line 563, it is claimed that the following holds due to Cauchy-Schwarz.  I'm not sure I see the application of CS here: Is there another reason why we would expect the cross term $2 E[(\\tilde{\\mu}_{\\epsilon}(X) - \\tilde{\\mu}(X))(\\tilde{\\mu}(X) - \\mu(X))]$ to be non-positive?\n\n$$ E(\\tilde{\\mu}\\_{\\epsilon(X)} - \\mu(X))^2 \\leq E(\\tilde{\\mu}_{\\epsilon}(X) - \\tilde{\\mu}(X))^2 + E(\\tilde{\\mu}(X) - \\mu(X))^2 $$\n* In the display following line 562, the last inequality seems like a non-trivial jump, would you mind walking through the logic explicitly?\n\nOtherwise, the proofs seem correct to me.  Note that I only read through the proofs for Section 3 in depth, and only skimmed the proofs of other relevant results (e.g., Propositions 3 and 4).  While I am well-versed in the causal inference literature, I am not otherwise an expert on non-parametric / semi-parametric statistics, so I may have missed something. \n\nAs an aside, it may be helpful to include a citation in the proof for some of the assumed results regarding kernels. [58] is referenced in the main text, referring to kernel smoothing more broadly, but it seems some of the prerequisite results could be cited more precisely (e.g., Lemma 25.1 of [58] appears to be a relevant result)\n\n### Other Minor Feedback\n\nI consider the following points to be minor feedback re: presentation / notation / possible typos, and they did not meaningfully influence my score, and they **do not require an explicit response from the authors** (some are stated as questions only because I am unsure if they are typos).\n\n**Suggestions on clarity**:\n* (Lines 15-19) This and some other sentences are a bit long and difficult to parse, and could perhaps be split into multiple sentences.\n* (Line 71) Is the introduction of projections onto the semi-parametric model necessary, given Remark 1's statement that this work focuses on nonparametric models?  CLvdL (Equation 2.2) seems to refers to Luedtke, Carone, and van der Laan (2015) as a reference for the equation on between lines 71-72 holding generally in a non-parametric model.\n\n**Other typos / inconsistencies**\n* Example 1 uses $\\mathbb{E}_P$ for the outer expectation, but not for the inner expectation.\n* Proposition 1 uses $\\tilde{\\mathbb{E}}_{\\tilde{P}_\\epsilon}$ in one place, perhaps the tilde on $\\mathbb{E}$ was not intended?\n* Line 65, what is the observation $\\tilde{o}$?  This is not referenced anywhere, I assume this is meant to be $o'$.\n* Footnote 1, should the kernel be $\\lambda^{-d} K(u / \\lambda)$ instead of $h^{-d} K(u / \\lambda)$?  There is also a reference to an $O(h^J)$ error term on line 568 that should perhaps be $O(\\lambda^{\\beta})$?\n* Algorithm 1: Should it be $\\tilde{P}$ on lines 3-4?  Should it likewise be $P_{\\epsilon, \\lambda}^i$ instead of $P_{\\epsilon}^i$?\n* Lemma 1: Should $\\tilde{e}\\_{\\epsilon}(X) = \\tilde{p}\\_{\\epsilon}(A = 1, X) / \\tilde{p}(X)$ instead of the current formulation, which uses $\\tilde{p}_{\\epsilon}(A = 1 \\mid X)$ in the numerator?  This would also make it consistent with usage in the proof (see e.g., line 561)\n* Line 154, there seems to be a missing parenthesis\n* Line 149 \"analyses of from kernel density estimation\"\n* Assumption 1 (iv), should the equation refer to $\\tilde{\\mu}_{\\epsilon}$ or just $\\tilde{\\mu}$?  Additionally, should the product-rate condition be $o_p(n^{-1})$ as written or $o_p(n^{-1/2})$?\n* Line 561, says to bound perturbed $e$ one should \"argue similarly\", seemingly in reference to the (later) bound on the perturbed $\\mu$, perhaps the order was swapped.\n* Line 562, following equation, third equality, missing a square on the final $\\tilde{p}(A=1,x)$ term.\n* Line 572, $E[\\Gamma(O; e_{\\epsilon}, \\mu)]$ should be $E[\\Gamma(O; \\tilde{e}_{\\epsilon}, \\tilde{\\mu})]$ \n* Line 637, indicator is missing an $\\epsilon$ on the left-hand side\n* Supplement, Section D.2, the reference is to citation [20], but I believe this should be citation [18]\n\n**Undefined notation**:\n* Line 66, I did not see a definition of the function $g(u)$.\n* I'm unsure if the dimension $d$ was precisely defined prior to usage in Lemma 1, though it is fairly obvious from context.\n* Eq. 7: I'm not sure if $\\mu_a$ is defined anywhere, aside from being the optimization variable, and similar for $\\mu^*(s, a)$ in Equation 8.\n* $\\nu$ is used a few times in the proofs without being defined (end of equation starting on 576, end of equation starting on 562), presumably referring to a strong-overlap constant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6GjZhhls6FG", "reviewer": "Reviewer_egsh", "summary": "This paper discusses the Gateaux derivative for a parameter functional of a probability distribution. The authors propose a method for approximating the Gateaux derivative empirically, with applications to causal inference. The authors also suggest dynamic treatment regimes as an interesting application.", "questions": "Following on from the above question, I have the following minor question.\n- Is it possible to apply the proposed method to robust statistics, such as Koh and Liang (2017)?\n- I think one of the parentheses in Lemma 1 is not closed.\n- What estimator (method) is used to estimate the nuisance parameter in the counterfactual mean estimation in Section 4.1?", "limitations": "See the above comments.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First, I could not fully understand this study. In particular, I would like the authors to elaborate on the motivation more. Therefore, I would like to make a final evaluation after a discussion with the authors.\n\nI think that the authors' proposed method for approximating the Gateaux derivative is interesting. As the authors mention, the Gateaux derivative and the influence function play an important role in statistical inference. For example, in (semiparametric) counterfactual mean estimation, we usually analytically derive the (semiparametric) influence function and use it as a lower bound of the asymptotic variance; then, we propose an estimator that achieves the lower bound. However, I could not understand what it means to approximate the Gateaux derivative empirically, and I would appreciate a careful explanation.\n\nThe influence function is helpful in constructing the estimator, but I do not think it is necessary to be concerned with its estimation as long as it can be analytically derived. For example, Hirano, Imbens, and Ridder (2003) propose an IPW-based estimator that achieves the lower bound derived from Hahn (1998)'s (AIPW-style) influence function. Of course, defining the influence function for a broad class of estimators is an important issue, and several previous studies have attempted it. However, once the Gateaux derivative is defined, it is not clear to what extent its empirical approximation needs to be discussed. At least, it does not seem to be necessary for counterfactual mean estimation and many OPE tasks.\n\nIn summary, while I acknowledge the contribution of the method, I would like to know more about the motivation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept