Efficient Online Estimation of Causal Effects by Deciding What to Observe
Authors: Shantanu Gupta
Zachary Chase Lipton
David Childers
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: online data collection
causal inference
online moment selection

Abstract: Researchers often face data fusion problems, where multiple data sources are available, each capturing a distinct subset of variables. While problem formulations typically take the data as given, in practice, data acquisition can be an ongoing process. In this paper, we introduce the problem of deciding, at each time, which data source to sample from. Our goal is to estimate a given functional of the parameters of a probabilistic model as efficiently as possible. We propose online moment selection (OMS), a framework in which structural assumptions are encoded as moment conditions. The optimal action at each step depends, in part, on the very moments that identify the functional of interest. Our algorithms balance exploration with choosing the best action as suggested by estimated moments. We propose two selection strategies: (1) explore-then-commit (ETC) and (2) explore-then-greedy (ETG), proving that both achieve zero asymptotic regret as assessed by MSE. We instantiate our setup for average treatment effect estimation, where structural assumptions are given by a causal graph and data sources include subsets of mediators, confounders, and instrumental variables.

Efficient Online Estimation of Causal Effects by Deciding What to Observe
Shantanu Gupta, Zachary Lipton, David Childers
Careegie Mellon University 2
Abstract
Researchers often face data fusion problems, where multiple data sources are available, each capturing distinct subset of variables. While problem formulations typically take the data as given, in practice, data acquisition can be an ongoing process. In this paper, we aim estimate any functional of probabilistic model (e.g., causal effect) as efficiently as possible, by deciding, at each time, which data source to query. We propose online moment selection (OMS), a framework in which structural assumptions are encoded as moment conditions. The optimal action each step depends, part, on the very moments that identify the functional of interest. Our algorithms balance exploration with choosing the best action as suggested by current estimates of the moments. We propose two selection strategies: (1) explore-then-comee
(OMS-ETC) proving that both achieve zero asymptotic regret as assessed by MSE. We instantiate our setup for average treatment effect estimation, where structural assumptions are given by a causal graph and data sources may include subsets of mediators, confounders, and instrumental variables.
(OMS-ETC) and (2) explore-then-grees
Introduction
Statistical and causal modeling typically proceed from the assumption that already know which variables are (and are not) observed. However, this perspective fails to address the difficult data collection decisions that precede such modeling efforts. Doctors must select set of tests run. Survey designers must select a slate of questions to ask. Companies must select which datasets purchase. Whether not model these decisions, they pervade the practice of data science, influencing both what questions we can ask and how accurately we can answer them.
One might ask, why not collect everything? The answers are two-fold: First, data acquisition can be expensive. In a medical setting, blood tests can cost anywhere from tens to thousands of dollars. Running every test for every patient is infeasible. Likewise, space surveys is limited, and asking every conceivable question of every respondent infeasible. Second, in many settings, we lack complete control over the set of variables observed. Instead, might have access to multiple data sources, each capturing a different subset of variables. Such data fusion problems pervade economic modeling and public health, and present interesting challenges: (i) efficiently estimating (or even identifying) a population parameter of interest often requires intelligently combining data from multiple sources; (ii) data collection is often iterative, with tentative conclusions at each stage informing choices about what data to collect next.
In this paper, formalize the sequential problem of deciding, at each time, which data source to query (i.e., what to observe) order to efficiently estimate a target parameter. We propose online moment selection (OMS), framework that applies the generalized method of moments (GMM) [7 both to estimate the parameter and to decide which data sources to query. This framework can be
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

applied to estimate any statistical parameter that can be identified by set of moment conditions. For example, OMS can address (i) any (regular) maximum likelihood estimation problem [13] Page 109]; and (ii) estimating average treatment effects (ATEs) using instrumental variables (IVs), backdoor adjustment sets, mediators, and/or other identification strategies.
Our strategy requires only that the agent has sufficient structural knowledge to formulate the set of moment conditions and that each moment can be estimated using the variables returned by at least one of the data sources. Interestingly, the optimal decisions which lead to estimates with the lowest mean squared error (MSE) depend on the (unknown) model parameters. This motivates our adaptive strategy: as we collect more data, we better estimate the underlying parameters, improving our strategy for allocating our remaining budget among the available data sources.
We first address the setting where the cost per instance is equal across data sources (Section 4.. First, show that any fixed policy that differs from the oracle suffers constant asymptotic regret, as assessed by MSE. We then overcome this limitation by proposing two adaptive strategies- then-commit (OMS-ETC) and explore- saellorr-ttengreedy boMt of which choose data sources based on the estimated asymptotic variance of the target parameter.
Under OMS-ETC, we use some fraction of the sample budget explore randomly, using the collected data to estimate the model parameters. We then exploit the current estimated model, collecting the remaining samples according to the fraction expected to minimize our estimator's 's asymptotic variance. In COM-ETTG we continue to update our parameter estimates after every step as we collect new data. We prove that both policies achieve the sample moments, we draw upon martingale theory. To derive zero asymptotic regret, we show uniform concentration of sample moments and sample concentration inequality for the GMM estimator with dependent data. Next, we adapt OMS-ETC and OMS-ETG to handle heterogencous costs over the data sources (Section 5) and prove that they still have zero asymptotic regret.
asymptotic regret. To overcome the non-i.i.d. nature of
Finally, we validate our findings experimentally ! (Section 6). Motivated by ATE estimation in causal models encoded directed acyelic graphs, generate synthetic data from variety of causal graphs and show that the regret of our proposed methods converges to zero. Furthermore, we see that despite being asymptotically equivalent, OMS-ETG outperforms OMS-ETC in finite samples. Finally, we demonstrate the effectiveness of our methods on two semi-synthetic datasets: the Infant Health Development Program (IHDP) dataset [19] and Vietnam draft lottery dataset 2
2 Related Work
Many works attempt to identify and estimate causal effects from multiple datasets. 4 21
study the
problem of combining multiple heterogeneous datasets and propose methods for dealing with various biases. Other works study causal identification when observational and interventional distributions
involving different sets of variables are available (28 40] [12] introduce estimators of the ATE
that efficiently combine two datasets, one where confounders are observed (enabling the backdoor adjustment) and another where mediators are observed (enabling the frontdoor adjustment). [29] derive an estimator for the ATE in linear causal models with multiple confounders, where the confounders are observed different datasets.
Another related line of work addresses finding optimal adjustment sets for covariate adjustment 45 36 While these works take for granted the collection of available datasets, focus on the problem of deciding which data to collect. Our work shares motivation with active learning, where the learner strategically chooses which (unlabeled) samples to label in order to learn most efficiently [6]. [ design algorithms for actively collecting samples in a manner that minimizes the learner's 's variance. In settings where there is a cost associated with collecting each feature, active feature acquisition methods incrementally query feature values to improve predictive model 47 propose an active learning criterion to find the most informative questions to ask each respondent in survey. In the context of causal inference, (37] study active structure learning of causal DAGs by finding cost-optimal interventions, [1] demonstrate that for ATE estimation, actively deconfounding data can improve sample complexity. 4k propose strategies for acquiring missing confounders to efficiently estimate the ATE.
18
33
22 34 20
'The code and data are
available at
https://www http:///ww..tthtu/t github. www. acmi -lab/
onl inn- moment-eeeteeeinn

---

Others have studied moment selection and IV selection from batch data. introduce consistent moment selection procedures for the GMM setting with some incorrect moments. 6 propose an
information-basee lasso method for excluding invalid or redundant moment conditions. 41 propose
a variable selection framework that uses lasso regression to decide which covariates to include. [14
propose four statistical criteria-includin estimation efficiency and non-redundancy--fo selecting among a set of candidate IVs. [9] develop an IV selection criteria based on asymptotic MSE and develop it for the GMM and generalized empirical likelihood estimators. 7 develop conservative confidence intervals for structural parameters when the off-diagonal entries of the covariance matrix of the empirical moments unknown (e.g., when the moments obtained from different datasets). By contrast, we are interested in selecting the data sources for the moments in an online setting.
Previous works address learning from adaptively collected data. [25] propose methods for adaptive experimental design, where at each step, the experimenter must decide the treatment probability propose doubly-robust estimator for off-policy evaluation with dependent samples. 46 provide regret bounds for learning an optimal policy using adaptively collected data, where the probability of selecting an action is function of past data. [48 49] study statistical inference for OLS and M-estimation with non-i.i.d. bandit data. While these settings are different from ours, some of the theoretical tools (e.g., martingale asymptotics and uniform martingale concentration bounds) are similar.
using past data in order to efficiently estimate the ATF. 23 24
Preliminaries
In the GMM framework 17 30), we estimate model parameters by leveraging moment conditions that are satisfied at the true parameters 8", A moment condition is a vector g(X,, 9) such that Elg(X,, X2, 0*]] = 0. We estimate 8* by minimizing the objective Qr:
T
1 0 = arg min (0), where (f) = T Σ (0)
I
W
Σ T
9i(0)
AEO
+ is the parameter space, 9c(0) == g(X,, 0), and W is some (possibly data dependent) positive definite matrix. In this work, we use the two-step GMM estimator, where the one-step estimator 0tos)
is computed with W == (identity) and the two-step estimator with W = (((())) }] where
LT(()) + 10 (0)
Let V be the set of variables of interest and a collection of subsets of V, each corresponding to the specifie variables observable via one of the available data sources. Our methods are applicable whenever the target parameter can be identified by a set of moment conditions such that each moment relies variables simultaneously observable in at least one data source. The selection vector, denoted by 8t E {0, |el is the binary vector indicating the data source selected at time t.
Assumption 1. The agent queries one data source at each step: Σ
|0l Di=1 St,s = 1, i.e. St is one-hot.
We can handle the querying of multiple sources by adding the union of their variables th In our setup, the moment conditions can be written as (x) = m(s,) @ (+) E R where @ is the Hadamard product, {0, |+| {0, 1] is fixed known function such that m(st) determines which moments get selected, and ge (0) are i.i.d. across t. For concreteness, we instantiate our setup with simple example:
Example 1 (Instrumental Variable (IV) graph). Consider a linear IV causal model (Figure 2a with instrument Z, treatment X, outcome Y. and the following data-gen matta-genrratin process:
X aZ + 1) Y := 3X e, c c.4. n, 11 Z, 1 1L Z.
The target parameter is the ATE 3. For 1p (1Z, X}, [Z, YJ), the moment conditions
(0)
St.1
Ze(X, aZ.)
$t,1 (x, aZ.) (1 - 8t,1) 1Y, - a.3iz)
St,2
Z.(Y. a.Zi.)
=m(e))
=3.10)
where 0 [B,a]" and (2,, Yi) are i.i.d.
3

---

For some known function fur O - let := (6m be the target parameter (e.g., the ATE). In
practice, estimate the target parameter by plugging-in the GMM estimate: B = fur(() (0). Let He represent the history or the data collected until time with Ho- = {} and space H4. A data collection
policy consists of a sequence of functions म. H4-1 - {0, 13 vl with &t me (H;-)) Thus 8t can be dependent on data collected until time (t 1) and so the sample moments 9: (0) are not i.i.d.
Definition 1 (Selection ratio). The selection ratio, denoted by m3 encodes the fraction of samples collected from each data source until time t: Ki = - 8t E A | (standard simplex).
We use 0% and 0(0)) to denote the two-step and one-step GMM estimators, respectively, that use the data H4. To reduce clutter, we use Ap == A 1 ctr (A M To. T (center of the
simplex), and might drop the superseript и from Ke, 0r. and B,. ||-I| denotes the spectral and l2 norms for matrices and vectors, respectively, and (0) == {0' ||0 01| S <) (e-ball around f).
4 Adaptive Data Collection
The central challenge in this work is make strategic decisions about which data to observe so that we most efficiently estimate the target functional. In this section, we present three policies: (i) fixed: query the data sources according to a pre-specified ratio; (ii) OMS-ETC: query uniformly for specified exploration period, estimate the optimal ratio based on the inferred parameters and thereafter aim for that ratio; and (iii) OMS-ETG: same as OMS-ETC but continue update parameter (and thus oracle ratio) estimates after the exploration period. For now, we analyze these policies for the case when the cost query is identical across data sources. By T N. we denote the (known) horizon, which can be thought of as the agent's 's data acquisition budget. We defer all proofs to Appendix A
We now present sufficient conditions for consistency and asymptotic normality of the GMM estimator under adaptively collected data. We later use these results to derive the regret of our policies.
Assumption 2. (a) (Identification) 4o + 0", P (lim infT +oo Q(0) > 0) 1, where Q())
of (0) (6) and ()) = ΣT m(s:) & E T
(0) (b) O c R is compact; (c) V0, ğ: (0) is twice continuously differentiable (c.d.): (d) VO, ([] is continuous; and (e) fior is c.d. at 0*.
By Assumption aa) the GMM objective is uniquely minimized 0* Informally, this means that each moment is (asymptotically) collected enough times to allow identification. If M = D (just- identified case), this holds when (i) an asymptotically non-negligible fraction of every moment is collected: Vj E [M,, P (lim inf T Et ms(s)) #0) 1; and (ii) 40 + 0", E (0) # O.
Property (ULLN), Let (x( == a(X;; E R be a continuous function with X, sampled i..dd. We say that a, (0) satisfies the ULLN property (i) VO, E
[0)(0)]] < oo; (ii) a, is dominated by function A A(X,): V0, [a)( S A(X,); and (iii) E[)( < xc.
Proposition 1 (Consistency). Suppose that (i) Assumption 2 holds, (ii) Vj E [M] (0) satisfies Property and (iii) v(i.j) E [M]] [0.(0)..(0) T satisfies Property
1 Then, ( p 8"
T
Proposition (Asymptotic normality). Suppose that (i)
0 Ex 8"; (ii) vii E [M] x
[D),
0ā: 80 (8)
satisfies Property
1 (iii) 36 0 such that E
[11:((*)11 2+0 co, and (iv) (Se-
lection ratio convergence) noy Is for some constant Is E Ag. Then Gr is asymptotically normal:
where E00", .k) is a constant matrix that depends only on 0 and k (see Appendix A.2 for the complete expression). By Assumption 2)e) and the Delta method, Br is asymptotically normal:
VT - B²) $ N (0, VO0", k)), where V(0", ,,,) vo ( Vo)
Proposition P shows that for policy under which the selection ratio Krr converges in probability to a constant, the GMM estimator 0r can be asymptotically normal. The specific order in which the data sources are queried does not affect asymptotic normality long the selection ratio Noy converges.
(x)

---

Input: T E N, E (0,1)
k ctr (Ap); 2 J !
for [1,2,..., J. do
Input: T N, c E (0,1)
[Te];
4
== [Tsj];
2 Collect samples s.t. tin = ctr (An)):
5
Collect [Ts] samples s.t. Ki k;
On = :.))
6
0. GMM(H, W = W Wvand);
4 = arg mig ((
h. = arg argmin (0),
7
5 Collect remaining samples such that
k proj(ks, (
8
KT proj Á);
end
6 8T = HTT;
10
07 = GMM HT, W Womeient);
Output: fur (0r)
Output: fur (B.)
(a) The Sxplore-then-Commi policy.
(b) The Explore-then-Greede policy.
Figure 1: Algorithms for OMS-ETC and OMS-ETG.
A fixed (or non-adaptive) policy, denoted by тRe has KT = k for some constant k E Ag. Here, the collection decisions do not depend on the data and each data source is queried a fixed fraction of the time. By Proposition 0i is asymptotically normal. The oracle policy, denoted by x², the fixed policy with the lowest asymptotic variance. Thus for n", we have x = K", where R" arg min, V(8", k). We call k* the oracle selection ratio. For the oracle policy, we have VT T B) 4 N V(0", consequently the data collection decisions are meaningful.
The following assumption ensures that n² is unique and
Assumption 3. x² uniquely minimizes V(0", ,K): VN S. K", V(O .R)
V(0", k*).
Definition (Asymptotic regret). The asymptotic regret captures how close the scaled asymptotic error of a given policy is to the oracle policy. We define the asymptotic regret of a policy as
Roc (T)) = AMSE
VT (
VVO""
(1)
where AMSE is the asymptotic MSE (i.e., the MSE of the limiting distribution).
(Tw) k for some constant k * k², we have Roc (T&) For any fixed policy T& such that KT [V k) - V(0", バ²) > 0 (by Assumption B) This shows that fixed policy suffers constant regret. This motivates the design of adaptive policies, where the regret asymptotically converges to zero
4.1 Online Moment Selection via Explore-then-Comii (OMS-ETC)
OMS-ETC inspired by the ETC strategy for multi-armed bandits (MABs) 27 Chapter 6]. Under OMS-ETC, we first explore by collecting fixed number of samples for each choice. Then, we use these samples to estimate the oracle selection ratio x*. Finally, we commit to this ratio for the remaining time steps. We denote the OMS-ETC policy by TETC (Figure la).
The policy TEre is characterized by an exploration fraction e (0, 1). We first collect Te samples by querying each data source equally so that KTc = ctr (A We then estimate Ör and obtain the plugin estimate of n" as arg min V(ōre, k). The feasible region for RT is defined as the set of values that KT can take after we have devoted Te samples exploration and given by Á (ENT. + (1 e)x
A } (proof in Appendix C We collect the remaining T(1-e) samples such
that KT is as close to k as possible: ky proj Å), where proj
kk Á) = arg min &' || [k]]-
Remark. The feasible region shrinks as increases because, as increases, the T(1 e) samples that remain after exploration decrease thereby shrinking the possible values that KT can take.
Theorem 1 (Regret of OMS-ETC). Suppose that (i) Conditions (i)-(iüi) of Proposition hold and (ii) Assumption holds. Case (a): For fixed E (0, if K* E A. then the regret converges to zero:
5

---

Re (TETC) = 0. 1f & A. then TETc suffers constant regret: Reo (TETC) = r for some constant r > O. Case (b): If e depends on T such that o(1) and Te + oo as T + 00 (e.g. vチ ). then 48" E we have (TETT) = 0.
The theorem provides sufficient conditions for when the regret converges to zero. Case (a) of the theorem shows that if we explore for fixed fraction of the horizon T, the regret will only converge to zero x* inside the feasible region. Thus the regret will not converge to zero over the entire parameter space θ as there would be certain parameter values for which k" would be outside the feasible region. Case (b) shows that we can achieve zero asymptotic regret for every 0* E ④ by setting such that becomes asymptotically negligible (e o(1)). The main idea in the proof (see Appendix A.3 p to show that KT 3 x* and apply Proposition 2 Case (b), the feasible region A asymptotically covers the entire simplex (Å T+00, A²) and this is sufficient to show that R/T - K²"
4.2 Online Moment Selection via Explore-then-Greed (OMS-ETG)
We extend OMS-ETC by periodically updating our estimate of n* as we collect additional samples instead of committing to value after exploration. The data is collected in batches. OMS-ETG (Figure lb) is characterized by a batch fraction s E (0. 1). The algorithm runs for J = rounds and we collect 6 Tss samples in each round. In the first round, we explore and thus Kb ctr (Aw). After every round) j E [J 1], we estimate 0bj and the oracle selection ratio: Ras n). The feasible region for round j + (the set of values of ki6j++13 can take) is
arg minxea, V( 35+1(Kb)) = jkb,- KKK E Au) (proof in Appendix C In round (j + 1), we (greedily) collect samples such that Ko(j+1) is as close to kj+1 as possible: KbG+1) proj (Ryg. A5+1
Theorem 2 states sufficient conditions for when OMS-ETG has zero regret. We first state finite- sample tail bound for the two-step GMM estimator under adaptively collected (non-i.idd)) data in Lemma I (which might be of independent interest) and use it to prove the theorem.
Property 2 (Concentration). Let (0) E [[((X;;)); 0), A(),,0) = Dā(X, To,
a(X;; 0) R with X; sampled i..dd. (.)
00 u.(n) = supe, (ree..10-'11s (āx(0) - ā,(+)), and u. (n) = E [u:(])) Let L1, No. and Ag be some positive constants. We say that (0) satisfies the Con-
centration property if (i) ä_ (f) is Li-Lipschitz, (ii v0 E O [ā,( 1 ā_(()) is sub- -ab-exponentall, (ii) E (X,,, fI]l < Au < 0o, and (iv) one of the following two conditions hold: (a) Vn E (0, no), (n)(n) u. (n)] is sub- sub-Exponential, or (b) supinee [|A 0)]l is sub- uub-Exponential.
Remark. Property is is used to derive uniform law (see Lemma 4 that is used to prove Lemma ! Property (vv) might be hard to check but (iv)() is satisfied for bounded forrndedfuncttin function classes, i.e., when ||ā: loo A < oo (see Proposition D and (iv)(b) for linear function classes with sub-Ex nub-Exponential data (see Proposition 10). For the linear IV model in Example 1 Property 2 iv) would hold if Z? is sub- Sabb-xxpoential (e.g.. when Z₁ is sub- Gausssaa....
Lemma 1 (GMM concentration inequality). Let Xx, Co. T2, and du be some positive con- stants. Suppose that (i) Assumption 2
holds; (ii) Vj. (0) satisfies Property 2
(iii) The
spectral norm of the GMM weight matrix W is upper bounded with high probability: V5 e
(0, Co), P (IWII 2 1 to exp -- (182)} (see Remark 1y; (iv) (Local strict con- vexity) YO E (0*), P ((]
(0) sh) = (Q(0) is defined in Assumption 20) (v) (Strict minimizution) v0 E (0*). there is a unique minimizer k()) arg min, V(U,)) k) s.f. V(0,, x) - ((0,x((0) S o02 Then, To- = arg min ((0 .k), any policy n,, and 48 E (0.
+ xx0]]] S 8; and (vi) sup, (V, rs) (00 < L| 0'].
P (1 - 0~ > <
82D exp (-0 (To*)) and P ((ITT -o--K" x² >
8A exp ( (-O(1o*)) (To*))
Better rates for Tep are applicable under additional restrictions on 6* (see Lemma
To derive the tail bound for Ōz, we first prove that the minimized empirical GMM objective is close to (0")) with high probability (w.h.p.) (using Conditions (i)-(i)). Then we show that 07 is close to

---

0" w.h.p. (using Condition (iv)). Next, we use the inequality for dr to derive the tail bound for k. For this, we show that V(0", k) is close to V(0", k*) (using Condition (vi)) and then show that le is close to 1²" w.h.p. (using Condition (v)).
Theorem 2 (Regret of OMS-ETG). Suppose that Conditions (i)-(iv) of Proposition 2 hold. Let
(+)) {SKA + (1 - 8)N K E Av). Case (a): For a fixed & E (0, 1), ifthe oracle selection ratio R² E (s), then the regret converges to zero: Rose (TETC) = 0. If K* & (ss) then Roo (oo( (TETG) > 0 (non-zero regret). Case (b): Now also suppose that the conditions for Lemmal hold. If = CT
for some constant C and any E [0, 1), then Y0" E E Roc (TETG) = 0.
Similar to OMS-ETC, Case (a) of the theorem shows that if the batch size is a constant fixed fraction, some values of K" will be outside the feasible region and thus we do not get zero regret over the entire parameter space. Case (b) shows that to get zero asymptotic regret for every 0* E 0, 8 must depend on T and be asymptotically negligible. But unlike OMS-ETC, the estimate of k* is updated after every round. The batch fraction s can be as small as CT allowing the agent to collect a constant number of samples in each batch.
We prove the theorem by showing that fr D K" and applying Proposition 2 To do so for Case (b), we show that the estimated oracle ratio after every round is close to k", i.e., Ve >
O, P [J- Raj E Ne(x*)
1 (by using Lemma 1p. Since we move as close as pos-
T
sible bay after every round, this ensures that Ve > 0, P (KT E (N.)x) - (and thus KT - n²*).
Both OMS-ETC and OMS-ETG are asymptotically equivalent as they can achieve zero regret for every 8* E O. But our experiments show that OMS-ETG outperforms OMS-ETC (in terms of regret) for small sample sizes. This may be because with OMS-ETG, the estimate of x* keeps improving as more samples are collected instead of being fixed after exploration. This suggests that better estimates of x* may lead to higher-order reductions in MSE.
Remark (Weight matrix W). For OMS-ETG, the GMM weight matrix W needs to satisfy Condi- tion (iii) of Lemma 1 rill round (J JJ-11 1). We denote this matrix by Woatid in Figure Ib For the final
step of OMS-ETG we use the standard efficient two-step GMM weight matrix (denoted by W Weffiiinnt in Figure bbl and thus the final estimate of 0 is still asymptotically efficient. Condition (ii) of
Lemma would hold for the efficient two-step GMM weight matrix (i.e., for W := [52
) if (see Lemma for proof). This would hold if the moments Je. (0) are uniformly bounded. Condition (tii) can also be satisfied with regularized weight matrix:
V(j k), I ( (0) [0)] satisfies Property 2
W = (0())) 08) + {w1c for some Xw > 0 as this ensures that [W]] S Aw.
Additional exploration. MAB algorithms usually require additional exploration to perform well (e.g., e-greedy and upper confidence bound strategies [27] Chapter 7)). However, we empirically noticed that additional exploration hurts performance in our setup. This might be because unlike typical bandit setups where pulling one arm does not improve the estimates of another arm, querying any data source can improve the estimates of the model parameter (/ in our setup.
5 Incorporating Cost Structure
In many real-world settings, the agent has budget constraint and must pay different cost to query each data source. We adapt OMS-ETC and OMS-ETG to this setting where cost structure is associated with the data sources in v. We prove that these policies still have zero asymptotic regret for every 6* E ④ Let (1b)) be an indexed family. We denote the (known) budget by B E N and [el >0*
by c E R
a cost vector such that e; is the cost of querying data source 4/;. Due to the cost structure,
the horizon T' is a random variable dependent on π with T = B
Kic The setting in Section 4 is
special case of this formulation when Vi, C = 1 and B = T. We defer proofs to Appendix
For a fixed policy Th. we have 1e7 (TT) = k. for some constant E Ag.
By Proposition as B + oo, we have vB ( 3*) 4, 4N VOO k) c)). Here we scale by / B instead of
7

---

vT to make comparisons across policies meaningful. The oracle selection ratio is now defined as K" arg mi, [V(0*, x)) (x' c)] and the asymptotic regret of policy π now is
Roo AMSE (VB ( - VVO", (((")))
OMS-ETC-CS (OMS-ETC with cost structure) is an adaptation of OMS-ETC for this setting. We use Be budget to explore and estimate x² by 1e arg min [v(rr, R) (x c)] where Tę
and NT. = ctr (A Exploration strategies that utilize the cost structure can also used (e.g.. evenly dividing the budget across data sources while exploring). With the remaining budget, we
collect samples such that Kir = proj Ri -). where Á is the feasibility region (expression given in
Appendix C The following proposition shows that OMS- EMS-ETC-CS can achieve zero regret.
Proposition 3 (Regret of OMS-ETC-CS), Suppose that the conditions of Theoree hold. Ife== o(1) such that Be > do as B + oc, then v0" E e, (TTTTE)) = O.
We propose two ways of adapting OMS-ETG this setting: (i) OMS-ETG-FS (OMS-ETG with fixed samples) where we collect a fixed number of samples in every round and (ii) OMS-ETG-FB (OMS-ETG with fixed budget) where we spend a fixed fraction of the budget every round.
Let Cmax
max G and Cmin = min
C.. In
(Figure 6a we collect
b
Ba
samples in every round except the last one (the last batch can be smaller as we may not
Crooo
have enough budget left for full batch). The number of rounds is random since, in every round, depending on what we collect, we use up a different amount of the budget. Like each round, we estimate x² and greedily collect samples to get as close to it as possible.
after
In OMS-ETG-FB (Figure 6b), we spend Bs budget in each round. Thus the number of rounds J is fixed with J - but the number of samples collected per round now random. Like OMS-ETG, after each round, we estimate k* and collect samples to get as close to the estimate as possible. The next two Propositions show that both OMS-ETG-F and OMS-ETC-FB have zero asymptotic regret.
Proposition 4 (Regret of OMS-ETG-FS), Suppose that the conditions of Theorem 2 hold. If s = B for any E [0, 1). then A0" E θ, Rxe (TerG-rS) = O.
Proposition (Regret of OMS-ETG-FBB. Suppose that the conditions of Theorem 2b hold. If B for any E [0, 1), then v0" E O, Roo (TETG.FR) 0.
6 Experiments
6.1 Synthetic data
We validate our methods on synthetic data generated from known causal graphs (see Appendix D for parameter values and moment conditions used). In our experiments (including the ones in Section 6.2 for OMS-ETG, we use the regularized GMM weight matrix with Aw 0.01 (see Remark I The regret is only minimally affected with an unregularized matrix (Aw := 0) despite theoretical guarantees only holding for the regularized case (the maximum change in regret our conclusions do not change. We first simulate data from a linear IV graph (Figure 2a and compare the performance of different polices (Figure 3ab. We use s/- {[2, X), [Z, Y)} and assume that both sources cost the same. We set parameter values such that ** s [0.36, 0.64] We compare policies based on relative regret (RR) with respect to the oracle policy:
1.98%) and thus
Relative regret- RR(T) (SE MSE (ornle)
x 100%.
The MSE values are computed across 12, 000 runs. The label etc {x} in the plot refers to OMS- ETC with exploration fraction e = x (e.g., etc
etc_0.l 0.I means e = 10%). Similarly, etg [z) refers to OMS-ETG with x. We see that as the horizon increases, the RR of all policies converges to zero. This supports the claim that both OMS-ETC and OMS-ETG have zero asymptotic regret. However, when the horizon is small (T = 300), both etc_0./ and elc_0. 0.2 perform poorly due to insufficient exploration. In contrast, OMS-ETG has close to zero RR even for small horizons which shows that repeatedly improving the parameter estimates can Iead to faster convergence of regret.
8

---

Z2
W
Z1
X
Y
(a) Instrumental variable graph.
(b) Two IVs graph.
(c) Confounder-mediator graph.
Figure 2: Examples causal models treatment X and outcome Y-where the ATE can be identified by different data sources returning different subsets of variables.
Rel atve
Relative
horizon
Relative budget
c0-
60
Relsave regret (Ng
ebc 5.4
etc
6o-
eig 5-1
Restee regret
60
4ig 2:4
Relssve regret
30-
13
e1g 0
et. 0.
20-
cig 1,
-0-
facd c119
-20
420
500
300
2000
2203
205
400
500
500
1000
1200
300 500
732
1000 1:00 1200 1200 Total budget
550
900
Total samples collected (Horizon)
Total sampless collected (Hfoiiono)
(a) IV model.
(h) Two IVs model.
(c) Confoumder-mediatrr model.
Figure 3: Relative regret (RR) across horizons/budgets for different policies (error bars denote 95% X), [Z,Y)) (b) RR for the two IVs model (Figure 2b) with = f(X, Y, Zi [X, Y, Z- (c) RR for the minfounder-mediann model
Cls). (a) RR for the IV model (Figure 2ap with
(Figure 2c with v {[X, Y, W), (X, Y. M)} and a cost structure.
Next, we simulate data from a linear graph with two IVs (Figure 2b
Here
f[x, Y. Z,), (X,Y, } and both choices cost the same. We set the parameters such that K" [0.1] (x² is on the corner of the simplex). We compare the RR across various horizons (Figure We see the OMS-ETC performs worse than OMS-ETG for small horizons. One dif- ference from the previous case (Figure 3a is that etc ot4 performs poorly even for large horizons. This is because after using 40% of the samples for exploration, the feasibility region not large enough to get close to the corner of the simplex. This demonstrates another benefit of OMS-ETG over OMS-ETC: OMS-ETG can achieve close zero regret finite samples when the oracle ratio n" is either on the boundary or in the interior of the simplex.
Finally, simulate data from linear inrfounder-mediatoo graph (Figure 2c Here, both the backdoor (using {X, Y, W') }) and frontdoor (using {X, Y, MM) adjustments are applicable [31 Sec- tion 3.3]. We use v = {{X, Y, W), (X, Y, }} with cost structure c = [1.8, 1] (confounders cost more than the mediators M). We set the parameters such that x* ss [0.15, 0.85] T We see similar conclusions as the previous cases. OMS-ETC with low exploration performs poorly but converges Ciiaants-OM--EGGG.
and OMS-ETG-FEB have close to zero RR for all horizons. We see no significant difference between the regret of OMS-ETG-FS and DMS-ETG-FB. The policy fixed equal is a fixed policy that collects an equal fraction both subsets. Its RR does not converge and is substantially higher than the oracle (ss 20%). This demonstrates that adaptive policies can lead to significant gains in MSE over fixed policies and that our methods remain applicable with associated cost structure the data sources.
large horizons. Both OMS-ETG
6.2 Semi-synthetic data
IHDP. Hill constructed a dataset based on the Infant Health and Development Program (IHDP). The data 110 is from randomized experiment studying the effect of home visits by trained provider on future cognitive test scores of children. Following Hill [9), we create an observational dataset by removing a non-random subset of the data. The treatment X is binary. The dataset contains pre-treatment covariates which are measurements on the mother and the child. For simplicity, we only use two covariates: birth weight (continuous) (Wi) and whether the mother smoked (binary) (W2). For each sample of the generated semi-synthetic data, (X, Wi, W2) are sampled uniformly at random from the real data. The outcome Y (continuous) is simulated: Y = aq W + a2 + B. + Ey.

---

Relativee
Relative dcc.22
50
Relothre regret (%)
eec 3.4
W1
W2
62.00.2
atg 6.2
feed)
s1g-2
Relative
20.
10
10.
X
Y
500
620
700
200
920
1000
1100
1360
0038
30200
100.0
12000
54200
52500
Total budget
Total samples collected (Horizon)
(a) IHDP data causal graph
(b) RR for IHDP data
(c) RR for Earnings data
Figure 4: Relative regret (RR) on semi semi-synthetic data (error bars denote 95% Cls). For both IHDP
(b) and Earnings data (c), adaptive policies converge to zero RR but fixed policies (collect all, fixed equal) suffer constant regret. OMS-ETG outperforms OMS-ETC for small horizons.
where by (0, o on, 022 B E R. and of E R+ (see Figure 4ap. Here on; 02,, and By model parameters with 3 being the ATE (target parameter).
For this experiment, we use & {(x, Y, Wi). YX,Y, W2). (X,Y, WT, W2]] with cost structure c= 3, 3.5] Thus, at each step, the agent can collect either one of the covariates or both of them, and each choice has a distinet cost. Setting model parameters such that x* R [0.59, 0, 0..41] T we compare performance across policies for various total budgets (Figure 4b). The policy collect all is a fixed policy that collects {.X, Y, W1, } at every step. This policy has higher RR (+s 50% higher MSE than the oracle) for all budgets demonstrating that collecting all covariates for every sample can be sub-optimal. The policy etc 0_2 does poorly with a small budget whereas etc 0.4 has close to zero RR. Both OMS-ETG-FB and
have close zero RR for all horizons.
The Vietnam draft and future earnings. Angrist 2 computed the effect of veteran status on future earnings from the Vietnam draft lottery data (3] using IV (Figure 2ap. The IV Z (binary) indicates whether an individual was eligible for the draft based on a random lottery. The treatment X (binary) indicates whether they actually served. The outcome Y (continuous) represents their future earnings. The IV removes bias caused by certain types of men being more likely to serve. In this dataset, {Z, X and Y) were collected using different data sources (thus { (Z, X, Y) are not observed simultaneously). which suits our framework. We construct a semi-synthetis dataset that closely matches the real data so that we know the ground -truh causal effect (needed to compute the MSE). For each instance, we sample Z uniformly at random from the empirical distribution. X is sampled from the Bernoulli distribution P(X|Z) with conditional probabilities given by the empirical distribution (values taken from 2 Table 2)). We generate the outcome as Y := 3X + + c, where <~ (N, of) and c.K X. The parameters B, 7, and o² are such that the distribution of (Z, Y)) is close to the real data (see Appendix D.5) for details). We compare the RR our policies on this dataset (Figure 4cp. Most adaptive policies converge near zero RR the horizon gets large. OMS-ETC does poorly with low exploration while OMS-ETG policies have significantly lower RR for smaller horizons. By contrast, fixed_ _qqualeuullaa (the fixed policy that queries both data sources equally) suffers constant regret and has s 25% higher regret than the oracle even for large horizons. This demonstrates that adaptive policies can lead to substantial MSE gains in real-world setting.
7 Conclusion
This paper takes some initial strides towards endogenizing decisions about which variables to solicit into the modeling process. Addressing the problem of deciding, sequentially, which data sources to query in order to efficiently estimate a parameter, we developed the online moment selection (OMS) framework and two instantiations: OMS-ETC and OMS-ETG. We prove that over the entire parameter space, adaptive data collection with either method can provide substantial MSE gains. While our work focuses on ATE estimation, our framework is more broadly applicable any parameter identified by moment conditions. In future work, we hope to apply our framework to more general prediction problems, addressing practical considerations including high-dimensional data and complex model classes (e.g., neural networks). Moreover, in real-world settings, common assumptions like ignorability rarely hold. We hope to extend our framework to overcome issues such model misspecification, to overcome biases present in some, but not all data sources.
10

---

References
[1] D. W. Andrews. Consistent moment selection procedures for generalized method moments estimation. Econometrica, 1999.
[2] J. D. Angrist. Lifetime earnings and the vietnam era draft lottery: evidence from social security administrative records. The American Economic Review, 1990.
[3] D. Angrist. Replication data for: Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, 2009. URL https://doo. org/ 7910/DVW/PLFOVI
[4] E. Barcinboim and J. Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 2016.
[5] J. Bell. The lindeberg central limit theorem, 2015.
[6] X. Cheng and Z. Liao. Select the valid and relevant moments: An information-based lasso for gmm with many moments. Journal of Econometrics, 2015.
[7] M. D. Cocci and M. Plagborg-Moller Standard errors for calibrated parameters. Princeton University, 2019.
[8] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of artificial intelligence research, 1996.
[9] S. G. Donald, G. W. Imbens, and W. K. Newey. Choosing instrumental variables in conditional moment restriction models. Journal of Econometrics, 2009.
[10] V. Dorie. Non-parametrics for causal inference.
http:///ttbb.coo///orrrrr//////p 2016.
[11] K. Gan, A. A. Li, C. Lipton, and S. Tayur. Causal inference with selectiellyddecorfouneedd data. arXiv preprint arXiv:2002. /096, 2020.
[12] S. Gupta, Z. C. Lipton, and D. Childers. Estimating treatment effects with observed confounders and mediators. arXiv preprint 10X1v:2003.11991, 2020.
[13] A. R. Hall. Generalized method of moments. Oxford university press, 2005.
[14] A. R. Hall and F. P. Peixe. A consistent method for the selection of relevant instruments. Econometric reviews, 2003.
[15] P. Hall and C. C. Heyde. Martingale limit theory and its application. Academic press, 1980.
[16] D. Hamilton. Time series analysis. Princeton university press, 1994.
[17] L. P. Hansen. Large sample properties of generalized method of moments estimators. Econo- metrica: Journal of the Econometric Society, 1982.
[18] L. Henckel, E. Perković, and M. Maathuis. Graphical criteria for efficient total effect estimation via adjustment in causal linear models. arXiv preprint arXiv:1907.02435, 2019.
[19] L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 2011.
[20] S.JJ. Huang, M. Xu, M.-K. Xie, M. Sugiyama, G. Niu, and S. Chen. Active feature acquisition with supervised matrix completion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, 2018
[21] Hünermund and E. Bareinboim. Causal inference and fata-fusion in econometrics. arXiv preprint 1912 2019.
[22] Ji and I.. Carin. Cost-sensitive feature acquisition and classification. Pattern Recognition, 2007.

---

[23] M. Kato. Adaptive doubly robust estimator and paradox concerning logging policy: Off-policy evaluation from dependent samples, 2021.
[24] M. Kato. Adaptive doubly robust estimator from non-stationary logging policy under a conver- gence of average probability, 2021.
[25] M. Kato, T. Ishihara, J. Honda, and Y. Narita. Adaptive experimental design for efficient treatment effect estimation, 2020.
[26] P. Kumar and A. Gupta, Active learning query strategies for classification, regression, and clustering: A survey. Journal of Computer Science and Technology, 2020.
[27] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[28] S. Lee and E. Bareinboim. Causal effect identifiability under partial-observabiiiyyy In Interna- tional Conference on Machine Learning. PMLR, 2020.
[29] H. Li, W. Miao, Z. Cai, X. Liu, T. Zhang, F. Xue, and Z. Geng. Causal data fusion methods using summary-level statistics for continuous outcome. Statistics in Medicine, 2020.
[30] W. K. Newey and D. McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics, 1994.
[31] J. Pearl. Causality. Cambridge university press, 2009.
[32] A. Rakhlin, K. Sridharan, and A. Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability Theory and Related Fields, 2015.
[33] A. Rotnitzky and E. Smucler. Efficient adjustment sets for population average treatment effect estimation in non-parametric causal graphical models. arXiv preprint 1921v:191. 2019.
[34] M. Saar-Tsechansky, Melville, and Provost. Active feature-value acquisition. Management Science, 2009.
[35] B. Settles. Active learning literature survey. Computer Science Technical Report 1648, 2009.
[36] E. Smucler, F. Sapienza, and A. Rotnitzky, Efficient adjustment in causal graphical models with hidden variables. arXiv preprint arXiv:2004. 10521, 2020.
[37] C. Squires, S. Magliacane, K. Greenewald, D. Katz, M. Kocaoglu, and K. Shanmugam. Active structure learning causal dags via directed clique trees. Advances in Neurul Information Processing Systems, 2020.
[38] T. Tao. Topics in random matrix theory. American Mathematical Soc., 2012.
[39] G. Tauchen. Diagnostic testing and evaluation of maximum likelihood models. Journal of Econometrics, 1985.
[40] S. Tikka, A. Hyttinen, and J. Karvanen. Causal effect identification from multiple incomplete data sources: A general search-based approach. arXiv preprint arXiv:1902.0107. 2019.
[41] O. Urminsky, C. Hansen, and V. Chernozhukov. Using double-lasso regression for principled variable selection. Available at SSRN 2733374, 2016.
[42] R. Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge university press, 2018.
[43] M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge University Press, 2019.
[44] S. Wang. S. E. Yi, S. Joshi, and M. Ghassemi. Confounding feature acquisition for causal effect estimation. In Machine Learning for Health. PMLR, 2020.
[45] Witte, L. Henckel, M. Maathuis, and V. Didelez. On efficient adjustment in causal graphs. Journal of Machine Learning Research, 2020.
12

---

[46] R. Zhan, Z. Ren, S. Athey, and Z. Zhou. Policy learning with adaptively collected data. arXiv preprint arXiv:2105.02:: 02344234,, 2021.
[47] C. Zhang, J. J. Taylor, C. Cobb, J. Sekhon, et Active matrix factorization for surveys. Annals of Applied Statistics, 2020.
[48] K. W. Zhang, L. Janson, and S. A. Murphy. Inference for batched bandits, 2021.
[49] K. W. Zhang, L. Janson, and S. A. Murphy, Statistical inference with m-estimators bandit data, 2021.
13


Reviews:
[{"review_id": "wbMQlptkbxB", "reviewer": "Reviewer_vPND", "summary": "Given multiple data sources, each related to a subset of variables, authors study the problem of deciding which data source to sample from at each time step. Then, by using the samples, the goal is to estimate the model parameters as good as possible. The proposed framework is based on the generalized method of moments. Their contributions are as follows:\n\n1. In Section 4, it is assumed that each instance has the same cost across all data sources. Then, authors first show that any fixed policy that differs from the optimal one will attain a constant asymptotic regret. To overcome this issue, two adaptive methods are proposed OMS-ETC and OMS-ETG. Here, OMS-ETC resembles the classical explore-then-commit algorithm in the context of bandits, where there is an exploration step to estimate the model parameters and then the model commits to a data source that minimizes the asymptotic variance of the target parameter. In contrast, OMS-ETG keeps updating the model parameters as more data is gathered. Under certain conditions, authors show that both adaptive methods will attain a zero asymptotic regret.\n\n2. In Section 5, both adaptive methods are modified as to attain zero regret in the case where instances have different costs across data sources.\n\n3. Experiments are provided in Section 6, where authors depict the behavior of the regret for different causal DAGs. Semi-synthetic dataset are also provided to showcase the usability of their method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here I present my review along the axes from the NeurIPS reviewing guidelines:\n\n**Originality**\n\n- Are the tasks or methods new? \n  - Yes. I believe the proposed setting is new as it differs from the data fusion problem.\n\n- Is the work a novel combination of well-known techniques?\n  - Yes. The methods are motivated by similar ideas applied in online settings. \n\n- Is it clear how this work differs from previous contributions? \n  - Yes. The main contrast starts with the setting presented in the paper, as it can be regarded as a \"online data fusion\" problem.\n\n- Is related work adequately cited?\n  - Yes, authors compare their methods against relevant work.\n\n\n**Quality** \n\n- Is the submission technically sound? \n  - Yes, the theoretical analysis relies heavily on martingale theory and leverages existing concentration and asymptotic results from other settings.\n\n- Are claims well supported (e.g., by theoretical analysis or experimental results)? \n  - Yes. Proofs are given in the appendix, and experiments effectively support the theoretical claims.\n\n- Are the methods used appropriate? \n  - Yes, given the non-iid nature of moment samples, martingale is an appropriate tool for the analysis.\n\n- Is this a complete piece of work or work in progress? \n  - This work is complete.\n\n- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n  - Yes, authors state the conditions/assumptions required for their methods.\n\n\n**Clarity** \n\n- Is the submission clearly written? \n  - Yes. I particularly appreciate Example 1, which helped grasp the setting faster.\n\n- Is it well organized? \n  - Yes, I was able to follow the main results without issues.\n\n- Does it adequately inform the reader? \n  - Yes, several details are provided to reproduce the results.\n\n\n**Significance** \n\n- Are the results important? \n  - Yes, the setting along with the algorithms provided are of interest to the machine learning community. In the introduction, authors provide a compelling story to motivate the setting.\n\n- Are others (researchers or practitioners) likely to use the ideas or build on them? \n  - Very likely in my opinion. The setting itself might get the attention of other researchers to propose alternative methods or improve upon the proposed ones.\n\n- Does the submission address a difficult task in a better way than previous work? \n  - As the setting differs to existing ones, there is no direct comparison to other methods.\n\n- Does it advance the state of the art in a demonstrable way? \n  - The methods presented can be regarded as the first set of results for the setting under study.\n\n# Questions / Comments\n\nI believe the paper discusses an interesting setting, where two non-trivial algorithms (motivated by existing methods from the online learning community) are proposed. My questions relate to the assumptions required to carry out the regret analyses of the methods.\n\n1. The identifiability assumption given in Assumption 2 was initially hard to follow to me. Can authors provide more intuition/interpretation on this assumption?\n\n2. Do authors have any comments about the necessary conditions for their methods?\n\n3. I understand that the reason for the lack of comparison against other methods is due to the setting. Still, I wonder if there is a possibility to (even naively) apply existing methods for ATE estimation under this setting. \n\n---\nI thank the authors for their response. I remain positive about the paper and still recommend acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "PwaQ4dOP7rU", "reviewer": "Reviewer_vjWD", "summary": "The paper proposes online moment selection (OMS), a framework in which\nstructural assumptions are encoded as moment conditions. This\nalgorithm allows users to actively acquire data (variables) given the\nvery moments that identify the functional of interest. The paper shows\nthat there is substantial empirical gains in doing so.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes online moment selection (OMS), a framework in which\nstructural assumptions are encoded as moment conditions. This\nalgorithm allows users to actively acquire data (variables) given the\nvery moments that identify the functional of interest. The paper shows\nthat there is substantial empirical gains in doing so.\n\nThe theoretical and empirical results of the paper make sense. I have\na few major comments.\n\nOne comment is on the potential bias due to actively collecting data.\nIn particular, several works have identified that actively acquired\ndata / variables have bias due to the active nature of the procedure.\n[*] Does the proposed algorithm suffer from similar bias too? Why or\nwhy not? Is there any correction procedure that would avoid this\nissue?\n\n[*] Nie, X., Tian, X., Taylor, J., & Zou, J. (2018, March). Why\nadaptively collected data have negative bias and how to correct for\nit. In International Conference on Artificial Intelligence and\nStatistics (pp. 1261-1269). PMLR.\n\nFarquhar, S., Gal, Y., & Rainforth, T. (2020, September). On Statistical Bias In Active Learning: How and When to Fix It. In International Conference on Learning Representations.\n\nThe other comment is that the theoretical results of the paper focus\non zero asymptotic regret, which is a fairly weak guarantee. Loosely,\nit says eventually we will converge to the truth. Do we know how fast\nthe algorithm converge to the truth? Can we bound the finite step\nregret? Does the algorithm admit optimal finite horizon regret? How\ndoes regret grow with the number of time steps?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discuss the limitations in terms of the assumptions it require.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "G6wfMpbRh2", "reviewer": "Reviewer_XDyk", "summary": "This paper proposes online moment selection, an approach for choosing which data sources to draw from in an online setting. The goal is estimation of causal effects, the structural causal model is encoded as moment conditions, and the data sources drawn from are chosen to minimize the variance of the estimate. Explore then commit and explore then greedy approaches are suggested and analyzed. A limitation is that the structural equations need to be parametric and have a clear noise term. Improvement is shown over simple fixed policy baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the problem interesting and the results good. I would have liked to see more discussion of the many assumptions and some exploration of whether they hold in practice and a theoretical discussion of under what types of causal models we can expect them to hold, e.g. the uniqueness assumptions etc. \n\nLine 163: I assume it is the case that this oracle fixed policy is also optimal across adaptive policies, if so this is important to clarify as the regret is being defined for an active policy. \n\nMinor: may be of interest to include related work on active learning of causal models, e.g. (Squires et al). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no social impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "NA"}, "extra_text": {}}, {"review_id": "1bldRAil8u_", "reviewer": "Reviewer_pVCh", "summary": "The asymptotic property of GMM estimator is proposed by some previous papers. In this paper, from the theoretical aspect, the authors generalize that to the setting with multiple domains and apply the results in the online estimation of causal effects. They evaluate the regret under three online policies to select which dataset to observe. ", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After rebuttal: the authors have addressed my quesions. The paper is very interesting and I am happy to see it accepted. I increase my score to 8.\n\n------------------------------------\nAt first, I want to say that I am not familiar with asymptotic theory. Although I read the proof and some related papers [15,27], I do not feel quite qualified to judge at the theoretical level. Hence it is hard for me to give some suggestions from theoretical viewpoint. Hence I mainly focus on the algorithm.\n\nThe main reason that I give a borderline score is that the problem tackled seems not reasonable. This makes me think the method right now is solid, but does not contribute a lot to causality literature, especially in a causal effect estimation task.\n\nIt seems that in reality, the optimal policy is always as [0,0,\\cdots,0,1,0,\\cdots,0] (It implies one dataset is better then the others). The estimator under different datasets have different asymptotic distributions. In this case, the optimal policy is to select the dataset with which the estimator has the least asymptotic variance. Hence it seems the main problem is to estimate the asymptotic variance for each dataset. Although the authors propose a more general method, which could be applied no matter what the optimal policy is like, it seems the contribution does not have a strong connection with causal effect estimation task. Could the authors further provide an example in reality when the optimal policy is not as [0,0,\\cdots,0,1,0,\\cdots,0]?\n\n\nIn addition, there are a few suggestions, questions, or reminders of typos:\n-The authors had better propose the explanation of symbol \\tilde{g_t}(\\theta) in Line 115.\n-Line 181: Appendix D -> Appendix C\n-I do not understand the equation on Line 170. Why is the order of difference constant when the policy is fixed?\n-In reality, it seldom holds that the parameter could be identified by each dataset. As said by the example of doctor on Line 19,  the different datasets are possible from different interventions (tests). In this case, it is hard to guarantee that the parameter is identifiable in each dataset. Hence I think it will be better if the authors could mention that they discard the dataset by which the parameter is unidentifiable.\n\n\n\nI look forward to the authors' rebuttal. And I will be happy to increase my score if the authors could provide some persuasive opinions, no matter by correcting me or responding to my questions above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a methodology for estimating statistical and causal parameters in an online setting in which multiple available sources of data contain information only on a subset of relevant variables. Repeatedly, an agent much choose which data source to interact with in order to as efficiently as possible estimate the desired parameter. Conditions are given under which the proposed estimators (OMS-ETC, OMS-ETG) achieve vanishing regret, both in uniform and non-uniform cost settings. The approach is evaluated on  synthetic and real-world data. 

The reviewers all recognised the novelty of the setting and the value of the proposed algorithm and analysis. There were no major questions or issues remaining after the discussion period, and all four recommended accepting the paper. I believe the unconventional take on the causal estimation problem could be of interest to the wider NeurIPS community. 
Decision: Accept (Spotlight)