Asymptotically Best Causal Effect Identification with Multi-Armed Bandits

Abstract: This paper considers the problem of selecting a formula for identifying a causal quantity of interest among a set of available formulas. We assume an online setting in which the investigator may alter the data collection mechanism in a data-dependent way with the aim of identifying the formula with lowest asymptotic variance in as few samples as possible. We formalize this setting by using the best-arm-identification bandit framework where the standard goal of learning the arm with the lowest loss is replaced with the goal of learning the arm that will produce the best estimate. We introduce new tools for constructing finite-sample confidence bounds on estimates of the asymptotic variance that account for the estimation of potentially complex nuisance functions, and adapt the best-arm-identification algorithms of LUCB and Successive Elimination to use these bounds. We validate our method by providing upper bounds on the sample complexity and an empirical study on artificially generated data.


Reviews and author responses:

This paper proposed a novel BAI framework. In the proposed framework, the goal is to identify an estimator with the lowest asymptotic variance.  ## Concerns:
### Asymptotic optimality
This study insists that "Asymptotically Best Casual Effect Identification," but there is not proof the asymptotic optimality.  
The "Asymptotically Best" may refer to asymptotically efficient estimator for the causal effect.  
However it does not mean the asymptotic optimality of the best arm identification (BAI) algorithm.
For more details, see Gariver and Kaufmann 2016.  
[Optimal Best Arm Identification with Fixed Confidence](http://proceedings.mlr.press/v49/garivier16a.pdf  )

### Inefficient sampling
In the main algorithm, the authors do not use half of samples to use double/debiased machine learning (DML).  
However, I think that this is a bit inefficient.  
Can we use the method of Howard et al. 2019?
That is, by using past observations, we can conduct DML for dependent samples.  
This is also used by van der Laan 2008 and Kato, Yasui, and McAlinn 2020.  
[The Construction and Analysis of Adaptive Group Sequential Designs](https://biostats.bepress.com/ucbbiostat/paper232/)  
[The Adaptive Doubly Robust Estimator for Policy Evaluation in Adaptive Experiments and a Paradox Concerning Logging Policy](http://128.84.4.18/pdf/2010.03792)

### Application
I don't really understand how this paper can be applied.  
I understand that the authors' study are motivated by the study of causal inference, but I could not understand the contribution to the field of bandit and design of experiments until the authors connect it a bit more with the literature on bandit and design of experiments. I do not fully understand where the proposed method can be used.   
I will consider social impact in discussion with the authors.  

 Dear Reviewer-

Thanks for responding quickly. We understand that you are trying to write an honest assessment of the paper. In turn, we are trying to ensure that you have a good understanding of the setting of the paper, because we still think there is a misunderstanding.

We aren’t interested in learning the population variance but, rather, the asymptotic variance of the estimator. Each estimator can have a different asymptotic variance, and this asymptotic variance is  a function of the characteristics of the estimator and the population distribution. We realize that our notation is partially responsible for the confusion since for estimator $k$, we wrote the asymptotic variance as $\sigma_k^2=\mathbb E[ \phi_k(W,\eta_k,\tau)^2]$, which obscures the fact that each estimator uses different covariates and hence $W$ is different between estimators. 

Please bear with us for the following, very explicit, example, which should hopefully make things clearer.

Let us consider the concrete example of choosing a covariate adjustment set, as in [Perkovic, Textor, Kalisch, Maathuis 2018], [Rotnitzky, Smucler 2020], and [Smucler, Sapienza,  Rotnitzky 2021]. In this setting, there are several adjustment sets one could use to obtain a valid formula for the causal effect (determining which sets of covariates are valid relies on knowledge of the causal DAG and the tools of do-calculus). Let us consider the case when $X$ is binary and the response $Y$ is real valued and we want to estimate the average treatment effect (ATE), i.e. $\mathbb E[Y|do(X=1)] - \mathbb E[Y|do(X=0)]$.

Assume we have the causal graph with links
$X\leftarrow Z_1\rightarrow Z_2\rightarrow Y$ and $X\rightarrow Y$. There are three adjustment sets, namely $\\{Z_1\\},\\{Z_2\\},$ and $\\{Z_1, Z_2\\}$, and each adjustment set defines an estimator with covariates equal to that adjustment set. Specifically, consider the augmented inverse propensity weight estimator (which has many nice properties: it is asymptotically efficient, doubly robust, and satisfies Neyman orthogonality). This gives three different estimators to choose from:

1. The first estimator uses $\{Z_1\}$, so it requires observations of the $(X, Y, Z_1)$ tuple. It has nuisance functions $e_{Z_1}(z) = P(X|Z_1=z)$ and $\mu_{x, Z_1}(z) = \mathbb E[Y| X=x, Z_1=z]$. It is calculated by using the first half of the data to fit $\hat e_{Z_1}$ and $\hat \mu_{x, Z_1}$, then using the second half of the data to calculate the empirical expectation over $X, Y, Z_1$ and set

$$\hat\tau := \mathbb E_n \left[\left(\frac{X}{\hat e_{Z_1}(Z_1)}\left(Y-\hat\mu_{1, Z_1}(Z_1)\right) + \hat\mu_{1,Z_1}(Z_1)\right)
-\left(\frac{X-1}{1 - \hat e_{Z_1}(Z_1)}\left(Y-\hat\mu_{0,Z_1}(Z_1)\right) + \hat\mu_{0,Z_1}(Z_1)\right)
\right].$$

This corresponds to the uncentered influence function

$$\psi_{Z_1}(W,\eta) = \left(\frac{X}{e_{Z_1}(Z_1)}\left(Y-\mu_{1,Z_1}(Z_1)\right) + \mu_{1,Z_1}(Z_1)\right)
-\left(\frac{1-X}{1 - e_{Z_1}(Z_1)}\left(Y-\mu_{0,Z_1}(Z_1)\right) + \mu_{0,Z_1}(Z_1)\right). 
$$

where $W=(X,Y,Z_1)$ and $\eta = ( P(X|Z_1=z), \mathbb E[Y| X=x, Z_1=z])$. The asymptotic variance of this estimator is $\mathrm{var}(\psi_{Z_1}(W,\eta))$ for this value of $W$ and $\eta$.

2. The second estimator uses $\{Z_2\}$, so it requires observation of the $(X, Y, Z_2)$ tuple. It has nuisance functions $e_{Z_2}(z) = P(X|Z_2=z)$ and $\mu_{x, Z_2}(z) = \mathbb E[Y| X=x, Z_2=z]$. It is calculated by using the first half of the data to fit $\hat e_{Z_2}$ and $\hat \mu_{x, Z_2}$, then using the second half of the data to calculate the empirical expectation over $X, Y, Z_2$ and set

$$\hat\tau := \mathbb E_n \left[\left(\frac{X}{\hat e_{Z_2}(Z_2)}\left(Y-\hat\mu_{1, Z_2}(Z_2)\right) + \hat\mu_{1,Z_2}(Z_2)\right)
-\left(\frac{X-1}{1 - \hat e_{Z_2}(Z_2)}\left(Y-\hat\mu_{0,Z_2}(Z_2)\right) + \hat\mu_{0,Z_2}(Z_2)\right)
\right].$$

This corresponds to the uncentered influence function

$$\psi_{Z_1,Z_2}(W,\eta) = \left(\frac{X}{e_{Z_2}(Z_2)}\left(Y-\mu_{1,Z_2}(Z_2)\right) + \mu_{1,Z_2}(Z_2)\right)
-\left(\frac{1-X}{1 - e_{Z_2}(Z_2)}\left(Y-\mu_{0,Z_2}(Z_2)\right) + \mu_{0,Z_2}(Z_2)\right). 
$$

where $W=(X,Y,Z_2)$ and $\eta = ( P(X|Z_2=z), \mathbb E[Y| X=x, Z_2=z])$. The asymptotic variance of this estimator is $\mathrm{var}(\psi_{Z_2}(W,\eta))$ for this value of $W$ and $\eta$.


3. The third estimator uses $\\{Z_1, Z_2\\}$, so it requires observations of the $(X, Y, Z_1, Z_2)$ tuple. It has nuisance functions $e_{Z_1, Z_2}(z_1, z_2) = P(X|Z_1=z_1, Z_2=z_2)$ and $\mu_{x, Z_1, Z_2}(z_1, z_2) = \mathbb E[Y| X=x, Z_1=z_1, Z_2=z_2]$. It is calculated by using the first half of the data to fit $\hat e_{Z_1, Z_2}$ and $\hat \mu_{x, Z_1, Z_2}$, then using the second half of the data to calculate the empirical expectation over $X, Y, Z_1, Z_2$ and set

$$\hat\tau := \mathbb E_n \left[\left(\frac{X}{\hat e_{Z_1, Z_2}(Z_1, Z_2)}\left(Y-\hat\mu_{1, Z_1, Z_2}(Z_1, Z_2)\right) + \hat\mu_{1,Z_1, Z_2}(Z_1, Z_2)\right)
-\left(\frac{X-1}{1 - \hat e_{Z_1, Z_2}(Z_1, Z_2)}\left(Y-\hat\mu_{0,Z_1, Z_2}(Z_1, Z_2)\right) + \hat\mu_{0,Z_1, Z_2}(Z_1, Z_2)\right)
\right].$$

This corresponds to the uncentered influence function

$$\psi_{Z_1,Z_2}(W,\eta) = \left(\frac{X}{e_{Z_1, Z_2}(Z_1, Z_2)}\left(Y-\mu_{1,Z_1, Z_2}(Z_1, Z_2)\right) + \mu_{1,Z_1, Z_2}(Z_1, Z_2)\right)
-\left(\frac{1-X}{1 - e_{Z_1, Z_2}(Z_1, Z_2)}\left(Y-\mu_{0,Z_1, Z_2}(Z_1, Z_2)\right) + \mu_{0,Z_1, Z_2}(Z_1, Z_2)\right). 
$$

where $W=(X,Y,Z_1, Z_2)$ and $\eta = ( P(X|Z_1=z_1, Z_2=z_2), \mathbb E[Y| X=x, Z_1=z_1, Z_2=z_2])$. The asymptotic variance of this estimator is $\mathrm{var}(\psi_{Z_1,Z_2}(W,\eta))$ for this value of $W$ and $\eta$.

Thus, we can see that the asymptotic variance of each estimator is different; the expectations are evaluated on different populations (with different covariates), the nuisance functions have different values, and the empirical expectations are over different random variables. In the paper, we left the notation at a more abstract level and didn’t explicitly define $\eta$ and $W$ for each estimator separately to save on notation; we will make this clearer.

__Individual responses__

> The asymptotic variance is a measure of the population, so it is a bit strange that it changes with sample collection.
I think what the authors consider is the variance in a finite sample (finite sample MSE).

Hopefully the above example makes it clear that we are considering the asymptotic variance of the estimator, not the population variance. Learning the estimator with the best asymptotic variance is a well defined task and a common task in the literature.

>I do not agree with this either. Chernozhukov et al., (2018) designed the estimator to be asymptotically insensitive to the nuisance parameters.
Also, Ichimura and Newey discussed a similar topic by defining the semiparametric influence function.
It is true that the value of the variance estimator changes with the estimate of the nuisance parameters, but that does not mean that the value of the asymptotic variance of the population changes.
An estimator of the variance is just a realization of a random variable, and how it is useful depends on the application.


Chernozhukov designed the estimator to be insensitive to the _error_ of the nuisance function, but not the true value of the nuisance function $\eta_0$. As our example above shows, the true value of the nuisance function certainly has an effect on the asymptotic variance. Again, we are not trying to learn the asymptotic variance of the population, but the asymptotic variance of the estimator error.


 Dear authors,

I am very sorry for the confusion.   
I had concerns when I first read the paper, but I forgot about them as I rushed to respond just before the reviewing process deadline.  
I reconsidered the concerns, and the setting still does not seem to be meaningful, at least to me.

> In general, it is not possible to calculate the asymptotic variance with just theory. Consider the simplest example: estimating the mean of the distribution. Given a sample of data, we can calculate the empirical mean and we know that, generally, the empirical mean is a good estimator of the mean of the distribution. However, we cannot calculate the asymptotic variance from just theory. The asymptotic variance is the population variance divided by the sample size, so we need to estimate the population variance (using data) in order to understand the asymptotic variance of the sample mean estimator.

The asymptotic variance is a measure of the population, so it is a bit strange that it changes with sample collection.   
I think what the authors consider is the variance in a finite sample (finite sample MSE).  

> Our setting is only a bit more complicated. We have several different estimators, each with different covariates and nuisance function estimators. The asymptotic variance of an estimator will necessarily depend on the distribution of the covariates and the value of the nuisance function, and neither can be calculated from theory alone. Just like in the empirical mean case, we need to estimate the asymptotic variance using data.

I do not agree with this either. Chernozhukov et al., (2018) designed the estimator to be asymptotically insensitive to the nuisance parameters.   
Also, Ichimura and Newey discussed a similar topic by defining the semiparametric influence function.   
It is true that the value of the variance estimator changes with the estimate of the nuisance parameters, but that does not mean that the value of the asymptotic variance of the population changes.   
An estimator of the variance is just a realization of a random variable, and how it is useful depends on the application.   

I apologize for the last-minute comment, but I still feel that the setting is a bit unreasonable, at least to me.    
I will keep my initial score.

 Thanks for engaging with our work. We respectfully, but strongly, disagree with the conclusion that the setting of the paper doesn’t make sense. We still think there is some confusion about it, which we will try to eliminate now; a new draft of the paper, written in response to the reviews, is already complete and has a better explanation of the setting. If you find the following discussion convincing, we kindly ask you to consider revising the score as soon as possible (please consider that your change in score came less than a week before final decisions are revealed).


__Setting__

The high-level setting is the following. The investigator has some causal effect to measure and a small set of estimators she could use.
1. The investigator collects some small number of samples and selects an estimator (the problem this paper considers)
2. The investigator then collects a large amount of data using the covariates of this estimator, then uses this estimator to calculate the causal effect. The amount of data is large enough to be in the asymptotic regime. 


This paper tackles step 1 by 
1. Recognizing that asymptotic variance is the right measure to select an estimator
2. Adapting best-arm-identification bandit algorithms to identify the arm with the best asymptotic variance. Bandit algorithms will be more sample efficient than simply measuring all the estimators at the same time, as they allow us to focus samples on the most promising estimators.
3. Bandit algorithms require finite-sample confidence intervals on the target quantity, so we develop finite-sample confidence intervals on the asymptotic variance. These bounds replace the finite-sample bounds on the arm means in the usual best-arm-identification setting.

__Measuring the asymptotic variance__

Learning the estimator with the best asymptotic variance is a reasonable goal because we only consider estimators that are 
Asymptotically linear, and 
Have known influence functions.
Property 1. implies that all the estimators are _consistent_, meaning that they approach the true value. This means that the constant estimator (with zero asymptotic variance) cannot be included. When comparing consistent estimators, asymptotic variance is by far the most studied measure of performance in the statistical estimation literature. For example, any statistics textbook will have a chapter on asymptotic efficiency, which is the ratio of a consistent estimator’s asymptotic variance to the best possible.

__Calculating the asymptotic variance from theoretical analysis__

In general, it is not possible to calculate the asymptotic variance with just theory. Consider the simplest example: estimating the mean of the distribution. Given a sample of data, we can calculate the empirical mean and we know that, generally, the empirical mean is a good estimator of the mean of the distribution. However, _we cannot calculate the asymptotic variance_ from just theory. The asymptotic variance is the population variance divided by the sample size, so we need to estimate the population variance (using data) in order to understand the asymptotic variance of the sample mean estimator. 

Our setting is only a bit more complicated. We have several different estimators, each with different covariates and nuisance function estimators. The asymptotic variance of an estimator will necessarily depend on the distribution of the covariates and the value of the nuisance function, and neither can be calculated from theory alone. Just like in the empirical mean case, we need to estimate the asymptotic variance using data.

__Other concerns__

>Moreover, the asymptotic efficiency describes the theoretical guarantee for the iid case, and the same result does not hold for the non-iid case.

This is true; however, the data we collect now will only be used to choose an estimator. In step 2, we will use the chosen estimator for the causal effect on a fresh batch of i.i.d. data. Hence, asymptotic variance is still an appropriate measure of performance.

>Algorithmically, the sample is designed to be iid, but the algorithms are not guaranteed to be the best when the samples is collected in an adaptive way.

Collecting samples in an adaptive way does not prevent us from constructing finite-sample confidence intervals; indeed, the entire stochastic multi-armed bandits literature requires that we can do this. The important restriction is that the samples are i.i.d. given the choice of the algorithm. When the confidence intervals are correct, we can provably recover the estimator with the lowest asymptotic variance, just as a stochastic bandit algorithm can provably recover the arm with the lowest mean.


 Thanks for the authors' replies!  
I apologize for the multiple typos in my initial comment and for the delay in my response.

I could not understand the meaning of "asymptotically" when I first read the paper.   

> The work of Howard et al. 2019 was considered extensively in our work. Specifically, the confidence sequence construction that we used is based on that work, and lemmas from that The error decomposition used to prove Theorem 1 was chosen specifically to allow all the random variables to be bounded. The error decomposition used to prove Theorem 1 was chosen specifically to allow all the random variables to be bounded by confidence sequences rather than confidence intervals.

This is a question out of interest, but I think the result from Howard et al. (2019) is also based on LIL-based confidence interval, so it is a somewhat pessimistic confidence interval.  
When I implemented it, the confidence interval was a bit loose.   
What do you think about this problem? Can we make it tighter?

The reason why I was confused throughout the paper was that the title is "Asymptotically Best Casual Effect Identification with
Multi-Armed Bandits."  
I misunderstood the meaning of "asymptotically best" and misread the content of the paper.   
"Asymptotic" is often used for the algorithm itself, so it might be better to change the title.  

If there had been a rigorous discussion of the asymptotic optimality of the algorithm, I would have accepted the paper.  
However, since there is not, I voted to weak accept. I increased my score because I felt that the setting is interesting.   

Finally, the authors cited an asymptotically linear estimator, but isn't this the one used when the sample size is non-random?   
When the sample size is a random variable, it is not clear whether the lower bound of the non-random variable will work as a lower bound on the estimator when the sample size is a random variable.   
Howard et al. (2019) seem to link the two by combining a LIL-based confidence interval with a Bernstein-based confidence interval, but I don't know how far this can be justified.  
In the setting of this study, we do not know about the distributions, which may be the reason for the tightness (in a minimax sense) of the authors' result. However, I am not sure.

I am sorry that I could not discuss this because of my delay in replying to your question.  

 Dear authors,

Thanks for the reply.  
I realize that I can still reply, so although the discussion period has finished, I respond to the authors' last reply.  

> Asymptotic Linearity: 
We limit our setting to only consider asymptotically linear estimators because: 1) they are common in practice 2) many estimators can be shown to be asymptotically linear thanks to the double machine learning literature (results from Jung and Bareinbiom 2021 and Chernozhukov et al., 2016, 2018) 3) their long-run behavior is captured by the asymptotic variance which can be written down explicitly. This explicit formula is necessary for us to create our finite-sample estimates for the asymptotic variance (Theorem 1) needed in our CS-LUCB and CS-SE algorithms. Our confidence sequences are based on techniques from Howard et al. and hold for all sample sizes simultaneously, so the random stopping time is not an issue. The CLT-based bounds discussed above are applied at non-random sample sizes that are fixed ahead of time, so there is no issue there either.

I think that we cannot justify the comparison of the asymptotic variances.   
The asymptotic variance is a guarantee of performance for infinite sample size, not for a finite number of samples.   
For example, if an algorithm returns constant scores for finite samples, its finite variance (MSE) will be zero, but it will be chosen by the algorithm.

Moreover, the asymptotic efficiency describes the theoretical guarantee for the iid case, and the same result does not hold for the non-iid case.   
Algorithmically, the sample is designed to be iid, but the algorithms are not guaranteed to be the best when the samples is collected in an adaptive way. 

> We will change the title and reword the abstract to make the point clearer. Would “Selecting the Asymptotically Best Causal Effect Estimator with Multi-Armed Bandits” be clearer?    

If the "best" implies the lowest asymptotic variance, it should be determined by theoretical analysis, not by empirical evaluation of the variance, as I explained. Therefore, the meaning of " Asymptotically Best Causal Effect Estimator " is not clear, and the title may not be appropriate.   

I could not read the paper carefully before, so I read it again.  
Then, I still could not get rid of my concerns.  
I apologize for confusing the authors, but I keep my initial score.

 Thanks for taking the time to respond and for the feedback. We appreciate your curiosity and thank you for engaging with our work. 

__Tightness of Confidence Sequences__

Howard et al. 2019 uses Ville’s maximal inequality (also known as Komolgorov’s maximal inequality) applied to certain supermartingales to derive all the confidence sequences, which is a bit more general than the LIL. Whether the confidence sequences are loose or not depends on who you ask; for the normal mixture of sub-Gaussian random variables, you essentially double the width of the confidence intervals vs. the classical, single-sample size confidence interval. This result is either amazing (only a factor of two to get a union bound over all time?) or underwhelming (I have to pay a factor of two?), depending on your expectations. 

In the experiments, we implemented two types of confidence sequences: one that uses the construction of Howard et al. 2019 and another that uses a union bound with a central limit theorem (CLT) approximation. The latter is much tighter (in agreement with your empirical findings) for a few reasons. First, the CLT is only asymptotically valid, so it lacks the finite-sample guarantee that Howard et al. 2019 gives you but can be narrower. Second, we only updated the confidence intervals every 100 datapoints, and so we only needed to apply the union bound every 100 datapoints with size $\alpha = 6/\pi t^2/\delta$, where $t$ is the batch number instead of the number of samples. In practice, we would recommend reducing the confidence interval width by batching the data. 

__Title__

We will change the title and reword the abstract to make the point clearer. Would “Selecting the Asymptotically Best Causal Effect Estimator with Multi-Armed Bandits” be clearer?

__Asymptotic Optimality__

To the best of our knowledge, all the results on asymptotic optimality for best-arm-identification rely on parametric assumptions about the reward distributions (such as belonging in the exponential family), which are too restrictive in our setting.

Instead, we could ask about the minimax optimality in the sense of finding matching upper and lower bounds on the sample complexity. We can import the standard lower bounds for fixed confidence from the literature, which is $\sum_{i\neq i^*} \Delta_i^{-2} \log(\log(\Delta_i^{-2}))$. In the specific case in which the widths of the confidence sequences for every estimator shrink at the same speed, this lower bound matches (up to logarithmic factors) the upper bounds we present in our paper. However, in our setting, we cannot assume that the widths shrink at the same speed, so the lower bounds and upper bounds need to depend on the gaps and the speed of shrinking, which is significantly more complex than the gap-only lower bounds in the literature. Obtaining such lower bounds is an interesting, but difficult, future direction.

Would this discussion be a useful addition to the paper?

__Asymptotic Linearity__

We limit our setting to only consider asymptotically linear estimators because: 1) they are common in practice 2) many estimators can be shown to be asymptotically linear thanks to the double machine learning literature (results from Jung and Bareinbiom 2021 and Chernozhukov et al., 2016, 2018) 3) their long-run behavior is captured by the asymptotic variance which can be written down explicitly. 

This explicit formula is necessary for us to create our finite-sample estimates for the asymptotic variance (Theorem 1) needed in our CS-LUCB and CS-SE algorithms. Our confidence sequences are based on techniques from Howard et al. and hold for all sample sizes simultaneously, so the random stopping time is not an issue. The CLT-based bounds discussed above are applied at non-random sample sizes that are fixed ahead of time, so there is no issue there either. 

 Our discussion and your helpful suggestions will definitely be used to make a better paper.

This work considers the problem of selecting identification formulas to estimate a causal effect. The problem is formed as a best-arm-identification bandits problem, i.e. in each round, the investigator can select a formula and observe some data, and the target is to find out the best formula with as few samples as possible. The return of each arm is weighted asymptotic variance of the estimator. The authors provide finite-sample confidence sequences for asymptotic variances, which is then used in the bandits algorithm. Theoretical guarantees and numerical experiments for a toy problem are provided. This paper is the first work in using observational data to select formulas.  - Novelty: This paper is the first work that frames the problem of formula identification as a best-arm-identification problem. In addition, the authors develop novel finite-sample confidence sequences with nuisance functions for a sample-splitting estimator of the asymptotic variance. They proved that having an uncentered influence function allows for confidence sequences that are insensitive to errors of the nuisance function estimates.

- Clarity: Overall the paper is well-writed and structured while there are some minor issues. This work covers two area, i.e. causal inference and bandits, which makes the exhibition of background knowledge important. It is reasonable to move some background knowledge to the appendix, but it should be mentioned in the main text. In addition, little background knowledge on bandits theory is provided. It would be great if the authors can give an overview on algorithms for best-arm-identification, especially LUCB and SE. 

- Significance: It is interesting that the authors connect the problem of formula identification with bandits and this is not the popular casual bandits/RL. However, this work seems to be lack of technical and algorithmic contribution. The only technical contribution is Theorem 1, which seems to be quite straightforward. The algorithms are also just LUCB and SE with few changes.

I am curious about why the problem is formed as best-arm-identification instead of regret minimization. With the confidence sequence, the UCB-type algorithms for regret minimization can be adapted just as LUCB and SE. Is there any difficulty in doing so? The main limitation is stated in the Main Review, i.e. this work lacks technical and algorithmic contributions. There are also some minor issues.

- Are there typos in Appendix D? For example in (2), $\lambda^\top S_t$ is a scalar while $\psi(\lambda)V_t\in\mathbb{R}^{d\times b}$. In addition, what is $\psi(\lambda,V)$? Also in Lemma 5, what does $sub-\psi(\lambda,V)=\sigma^2\lambda^\top V\lambda/2$ mean?

- The experiments only compare the algorithms with uniform sampling. It would be great if they can be compared with graph-structured-based algorithms.

- In the Discussion, the authors state that when the graph is partially unknown or latent variables exist, their methods still work. Is there any example when the above situation happens, the influence functions are still available?

- As mentioned in the paper, this method relies on asymptotic linearity and influence functions need to be known.

 Your response clearly addressed my concerns. Overall I think this is a good paper and I raise my score to 7.There are some possible improvements as we have discussed and I hope that they can be addressed in the final version.

 Thanks for providing more helpful feedback! 

To clarify, having an uncentered influence function is a sufficient but not necessary condition for a second-order dependence on the nuisance function rate when estimating the variance. This is analogous to the fact that Neyman orthogonality is a sufficient but not necessary condition to obtain second order dependence for estimating $\tau$. 

Neyman orthogonality is not sufficient for estimating $\mathbb E[ \phi^2(W, \eta,\tau)$ at a $O(\Vert\eta - \hat\eta\Vert^2)$ rate and additional assumptions about $\phi$, such as its being uncentered, are necessary. To see this, recall that the error of the asymptotic variance is equal to $\mathbb E[ \phi^2(W, \eta,\tau) -  \phi^2(W, \hat\eta,\hat\tau)]$. Neyman orthogonality plus a Lipschitz property of $\phi$ lets us conclude that $\mathbb E[ \phi^2(W, \hat\eta,\hat\tau)] - \mathbb E[ \phi^2(W, \hat\eta,\tau)] = O(\Vert\eta - \hat\eta\Vert^2 + \sqrt{1/n})$, so applying the triangle inequality, we see that
$$
  \mathbb E[ \phi^2(W, \eta,\tau) -  \phi^2(W, \hat\eta,\hat\tau)] = \mathbb E[ \phi^2(W, \eta,\tau) -  \phi^2(W, \hat\eta,\tau)] + O\left(\Vert\eta - \hat\eta\Vert^2 + \sqrt{1/n}\right).
$$
Taking a Taylor expansion of the first term, we see that, for some $\eta’$ in the convex hull of $\\{\eta,\hat\eta\\}$,
$$\begin{eqnarray*}
\mathbb E[ \phi^2(W, \hat\eta,\tau) -  \phi^2(W, \eta,\tau)]  
&=
\mathbb E[ (\phi(W, \eta,\tau) + (\hat\eta - \eta)^\top \nabla_\eta \phi(W,\eta’,\tau))^2 -  \phi^2(W, \hat\eta,\tau)]  \\\\
&=
\mathbb E[(\hat\eta - \eta)^\top \nabla_\eta \phi(W,\eta’,\tau))\phi(W,\eta’,\tau)  + \phi(W,\eta’,\tau)^2]\\\\
&=
(\hat\eta - \eta)^\top \mathbb E[ (\nabla_\eta \phi(W,\eta’,\tau) + \phi(W,\eta’,\tau) ) \phi(W,\eta’,\tau)].
\end{eqnarray*}
$$
Hence, without some assumptions on $\phi$ beyond Neyman orthogonality, we cannot argue that $\mathbb E[ \phi^2(W, \hat\eta,\tau) -  \phi^2(W, \eta,\tau)]   = O(\Vert\hat\eta-\eta\Vert^2)$.

Additionally, we can show common examples of $\phi$ that are uncentered and will take a first-order dependence.  Partial linear regression, the most common example in the semi-parametrics literature, can be Neyman orthogonalized but not represented by an uncentered influence function (see Equation 2.13 in Chernozhukov et al. 2018, for example).


We discussed the graphical criteria because they represent the most advanced methods in the literature for selecting identification formulas based on the asymptotic variance.  However, they do not apply to the sequential setting that we consider.
There are no other baselines we could discuss given the novelty of our setting.

 Thanks for the detailed response. The clarifications on the influence function and regret minimization make sense to me. I add some comments on your response and maybe some points can help you improve the paper.

### Significance

You state that Theorem 1 represents a solid contribution because it implies that having an uncentered influence function is a sufficient condition for obtaining the second-order dependence on the rate of nuisance function estimation, while this is 'generally' not possible without this condition. It seems that the impossiblity is only a conjecture and it is proved in the paper. The significance of Theorem 1 can be improved if the impossiblity result is strictly proved.

### Other comments

- Typos: In the appendix $\psi(\cdot)$ is a function defined on $\mathbb{R}^d$. Please check the definitions carefully in the revision.
- Experiments: In the Introduction section, it is said that some of the previous selection approaches use the structure of the casual graph. I don't know if these methods are comparable with the method proposed and serve as the baselines.


 Thank you for your encouraging remarks. 

We believe that you have a firm grasp on our contributions. We would like to emphasize the importance of achieving a second-order dependence on the nuisance function estimation rates. We realised that we did not sufficiently highlight and explain its significance in the submission. We attempt to do this below (we will incorporate the explanation also in the paper).

Theorem 1 implies that having an uncentered influence function is a sufficient condition for obtaining the second-order dependence on the rate of nuisance function estimation. Neyman orthogonality, while sufficient for predicting the causal contrast at a parametric rate, is not sufficient for predicting the variance at the parametric rate; a second-order result is generally not possible without an uncentered influence function.

__Weaknesses__

- _Number of formulas_:
An analysis on the effect of the number of formulas could indeed be insightful, thanks for the suggestion. We began by running experiments with 3, 4, 5, and 6  backdoor paths, obtaining 9, 17, 33 and 65 identification formulas respectively. The average sample complexity of the three algorithms, as a function of the number of formulas, are summarized in the table below. We can see that the benefits from using a bandit algorithm increase dramatically with the number of formulas; we also see that CS-LUCB is generally superior. We will add more exhaustive experiments to the paper.

| Number of formulas | 9 | 17 | 33 | 65 |
| --- | --- |--- | --- |--- | 
| CS-SE | 39,250 | 68,250 | 81,750 | 274,000 |
| CS-LUCB | 40,000 | 53,500 | 59,500 | 110,500 |
| Uniform | 118,125 | 320,875 | 313,500 | 4,810,000 |
| CS-SE relative to uniform | 33% | 21% | 26% | 5.6% |
| CS-LUCB relative to uniform | 33% | 17% | 19% | 2.3% |

- _General problem_:
Generally, estimating causal effects from only observational data is the central problem in causal inference. To the best of our knowledge, we are the first to consider the problem of learning a causal effect from observational data as a sequential decision problem when you can change what covariates are observed. While we have made a first step by considering choosing between estimators, the general problem that simply tries to produce the best estimate of the causal effect under adaptive observations is wide open. This problem is significantly more difficult than the one we considered and will require a general theory of how to combine estimates from different formulas and different nuisance functions. We will add a discussion in the paper.

- _Implication of our results for bandits_:
We neglected the impact of our techniques on the bandit literature. As you hinted, our confidence sequences on the variance could be an upgrade to algorithms in the infinite-armed bandit setting which use variance estimation (e.g. UCB-V or Bernstein-SiRI) when nuisance functions are present. Our method does not require each estimator to have the same expected value, so we could use it to solve best-arm-identification problems where “best” refers to some risk-adjusted measure that depends on the variance in addition to the expected reward of an arm: the results from this paper would provide finite-sample confidence bounds on the variance, and we could use a similar decomposition to the one used in Theorem 1, or results from Foster and Syrgkanis 2019, to provide finite-sample confidence bounds on the mean estimate. In general, our work suggests the problem of bandits where the reward estimation requires nuisance functions rather than the almost ubiquitous empirical mean estimate. In particular, non-contextual bandit algorithms in online experimentation typically collect lots of user covariate data, so a theory of confidence intervals for estimators with nuisance parameters would allow the use of variance-reduced estimators using covariate adjustment. We will add this discussion in the paper.

 Thank you for your feedback and questions. We hope that our replies below provide some clarifications.


__Asymptotic optimality__

Our use of the term “asymptotically” has created confusion; indeed, it refers to the formula/estimator that has the best asymptotic performance, but the bandit algorithm itself is not asymptotically optimal. We will make this distinction clear in the paper. On a side-note, we cannot apply track-and-stop style algorithms to our setting, since they rely on parametric assumptions about the reward distributions and all we can guarantee is a frequentist confidence sequence on the reward. Additionally, we could not use  the BAI algorithms known to be minimax optimal, such as lil’UCB or Exponential Gap Elimination, for the following reasons. The stopping condition of lil’UCB is triggered when one arm is sampled at a constant proportion, but the proof of correctness requires that the confidence bound widths shrink at the same rate for every arm, which is not necessarily true in our framework. The same difficulty arises in Exponential Gap Elimination and even Median Elimination; both algorithms also assume a uniform rate of decay of the bound widths across arms. Hence, we were forced to use the simpler algorithms LUCB and SE, which allow us to derive correctness and sample complexity results when the confidence intervals decay at arbitrary speeds. We will add this clarification to the paper.

__Inefficient sampling__

We agree that simple sample splitting can be inefficient, as only half the data is used to fit the nuisance function. As you pointed out, an approach more similar to cross-fitting (as suggested by Chernozhukov et al. 2018, Kato et al. 2020) would instead enable us to use all the data. We used the simple sample-splitting estimator because the bandit algorithm requires finite-sample confidence intervals, which are difficult to obtain for the cross-fitted estimator. All the results we know of from the semi-parametric literature for the cross-fitted estimator (e.g. Chernozhukov et al. 2018) are asymptotic. Additionally, the errors of nuisance function estimates between different folds are dependent, which significantly complicates the analysis.

Unfortunately, we cannot see how the ADR estimator from Kato et al. 2020 will help. While a powerful result, that paper shows that asymptotic normality can hold when the interventions are chosen in a data-dependent way. In contrast, our data are observational, as we never change the treatment assignment and only change which covariates we observe; therefore, our observations remain independent and do not require the techniques in Kato et al. 2020. However, the ADR estimator from Kato et al. 2020 opens up a very interesting question: how can we select the best estimator when we may also intervene?

More generally, your question suggests an important future direction: establishing confidence bounds on cross-fitted estimators. Such results would have many applications beyond our setting. We will add a discussion in the paper.

The work of Howard et al. 2019 was considered extensively in our work. Specifically, the confidence sequence construction that we used is based on that work, and lemmas from that work are essential in deriving our Corollary 1. The error decomposition used to prove Theorem 1 was chosen specifically to allow all the random variables to be bounded by confidence sequences rather than confidence intervals. 

__Application__

Your feedback here is helpful. We should have made it clearer that our contribution was intended to be entirely in the field of causal inference, rather than in the field of bandits. Specifically, our contributions can be summarized as follows.

First, we wanted to overcome the following three current limitations of the causal inference literature on selection of identification formulas based on asymptotic variance:
- Inability to select the best identification formula in some latent variables cases. As a concrete example, the DAG $Z_1\rightarrow Z_2 \leftarrow Z_3\rightarrow Y$, $Z_1\rightarrow A\rightarrow Y$  admits the following adjustment sets: $\emptyset$, $\\{Z_1\\}$, $\\{Z_3\\}$, $\\{Z_1, Z_2\\}$,$\\{Z_1, Z_3\\}$, $\\{Z_2, Z_3\\}$, and $\\{Z_1, Z_2, Z_3\\}$. According to Theorem 5 in the Appendix, the set $\\{Z_3\\}$ has the lowest asymptotic variance. If, however, $Z_3$ is latent, then either $\\{Z_1, Z_2\\}$ or $\emptyset$ has the lowest variance depending on the model distribution, and therefore graphical criteria are unable to return the adjustment set with smallest variance. Our algorithm would be able to resolve this case.
- Applicability to adjustment criteria only.
- Inability to account for the costs of observing covariates.

In addition, we wanted to consider an adaptive covariate measurement (ACM) setting, namely a setting where one can choose which covariates to measure adaptively and wishes to identify the best formula as efficiently as possible. This ACM setting is a sequential decision making problem: modelling this problem as a bandit problem  enables us to use powerful bandit algorithms.

The ACM setting might be relevant to several applications where causal inference can play an important role, such as medical applications or advertising. Consider, for example, the question of which estimator we should use for a large clinical trial. The trial has the option of using several different estimators and, before the trial begins, we wish to identify, using a group of individuals, the estimator with the best long-run behavior, i.e. with the lowest cost-adjusted asymptotic variance. One approach would be to assign an estimator to each individual randomly, collect the covariates required by the estimator, and then estimate the estimator’s asymptotic variance. However, using a bandit algorithm to assign an estimator to the $i$th individual based on the feedback from the individuals $k$ for $j<i$ would have the advantage of requiring a smaller group of individuals.

The problem of selection of identification formulas based on asymptotic variance has only started to receive some attention and is still at its infancy. Before the recently established graphical criteria, the set of parents or the smallest set of covariates was often used as adjustment sets, which is often inefficient. Whilst representing a huge advancement, graphical criteria have some limitations. Our idea of framing this problem as a best-arm-identification bandit problem is simple, yet effectively addresses many of the current issues. We also believe that our work may serve as a first step for solving more general problems.


 Thank you for your thoughtful review. We were pleased that you found our work to be novel and clearly exposed.

__Clarity__

Thanks for pointing out that the paper would benefit from more background knowledge on bandit theory. We will add an overview on algorithms for best-arm-identification as well as more details about the LUCB and SE algorithms. 

__Significance__


We agree that there is no algorithmic contribution, as both bandit algorithms are trivial to extend to the confidence sequence case. However, in our opinion Theorem 1 represents a solid technical contribution. We realized that we did not sufficiently highlight and explain its significance in the submission. We attempt to do this below (we will incorporate the explanation also in the paper).
Theorem 1 implies that having an uncentered influence function is a sufficient condition for obtaining the second-order dependence on the rate of nuisance function estimation. Neyman orthogonality, while sufficient for predicting the causal contrast at a parametric rate, is not sufficient for predicting the variance at the parametric rate. The proof of Theorem 1 is indeed straightforward under the condition of having an uncentered influence function, but a second-order result is generally not possible without this condition; most of the challenge was in identifying it.
Another key observation made during the proof of Theorem 1 was that the error admits the following two decompositions: 
1) A decomposition that would require to control $\sum_{i=1}^n \psi(w_i, \hat\eta_n)$ for all sequences of estimators $\hat\eta_n$ of $\eta$.
2) A decomposition that would only require a confidence sequence for $\sum_{i=1}^n \psi(w_i, \eta)$, i.e. at the true $\eta$, albeit with a few more details in the analysis.

Using the second decomposition allowed us to generalize all our results from confidence intervals to confidence sequences because we only needed to control the deviations at $\eta$.

__Other comments__

- _Typos_: 
Thanks for finding these! 
There is a transpose missing as $\lambda$ is a vector. 
$\psi$ characterizes how quickly the tails of the distribution of the increments decay; for example, if $\psi$ is a quadratic, then the tails of the distribution decay like a Gaussian. 
The notation of Lemma 5 means that equation (2) is satisfied with $\psi(\lambda, V) = \sigma^2 \lambda^\top V \lambda /2$; we’ll make sure to define this.

- _Experiments_:
We were unable to think of better baselines for comparison, and are not sure what graph-structured-based algorithms you are referring to. We would appreciate it if you could point out some references.

- _The influence function_:
The influence function is a property of the estimator and does not explicitly depend on the graph structure or the models for the nuisance function. In particular, the influence function for every adjustment criterion in the Experiments Section (which all use the AIPW estimator) are identical, even though the observed variables, unobserved variables, and nuisance function models may change for each adjustment set. Knowledge of the causal graph is only needed to obtain the adjustment sets (i.e. set of variables that block all non-causal paths and are non descendants of nodes on causal paths from $X$ to $Y$). For a concrete example, consider the DAG in the Experiments Section but now assume that $Z_1$ is latent. If we remove all the adjustment sets containing $Z_1$, we can simply run our bandit algorithms on the remaining estimators and learn the optimal estimator in the presence of a latent variable; we emphasize that every adjustment criterion estimator will still have the same influence function.

- _Regret minimization_:
As you pointed out, it is straightforward to use confidence sequences with regret-minimizing algorithms (e.g. UCB). We opted for best-arm-identification because it seemed more natural for our setting: we want to have a high probability guarantee on finding the best estimator. In contrast, it is difficult to see how a small cumulative regret on sums of estimated asymptotic variances relates to our ability to estimate the causal contrast. The feedback we receive (an updated estimate of the asymptotic variance) from trying an estimator is also not easily interpreted as a loss, as there is a very strong dependence between successive rounds. However, there is a natural cumulative regret question in this setting: can you adaptively select which covariates to observe in a way that allows you to learn the causal contrast almost as well as the best covariate observation distribution in hindsight? This problem is substantially harder, as it requires general methods for combining estimates from different estimators with finite-sample confidence bounds.


This paper aims to select the best causal formula among a set of available formulas based on the asymptotic variance criterion. An online setting with best-arm-identification framework is considered. The paper poses the sequential decision problem on how to choose which observations in a sample efficient way. Framing the problem in multi-armed bandits framework avoids the requirement of structural knowledge of the graph and allows comparing arbitrary formulas. Novel confidence sequences with nuisance functions for a sample-splitting estimator are introduced. Two well-known bandit algorithms, lower-upper confidence bounds (LUCB) and successive elimination (SE), are adapted with these confidence sequences and sample complexity results are established.   This paper attempts to provide a step towards identifying best available formula for the causal effect identification without deriving a solution from graphical criteria. Causal effect identification is posed as a sequential decision problem where the investigator chooses which observations to make and updates the beliefs regarding the available formulas. 

Strengths: 
-	The difference from the previous works is stated clearly. Some of the limitations of the recent works, e.g., practical constraints for selection methods based on graphical criteria, are addressed via estimating the asymptotic variance directly. 
-	The idea of framing this problem as a best-arm-identification bandits problem is simple yet effective. It may serve as a first step for solving more general problems as briefly touched in the discussion section.
-	The main technical result is a novel confidence sequence for the sample-splitting estimator. It shows that having an uncentered influence function leads to a better convergence rate in terms of nuisance estimation. Since most causal estimators require a nuisance estimate, this can be a nice feature.
-	Adaptation of LUCB and SE bandit algorithms are sound.

Weaknesses:
-	In simulations section, the proposed algorithms are shown to require significantly smaller number of samples than uniform sampling at finding the optimal arm (best formula). However, only 9 formulas are considered. It would have been insightful to comment on the effect of the number of formulas, e.g., does scaling behavior bear similarities to the sample complexity results presented in the section 4.
-	The general problem of trying to obtain the best estimate of the causal effect is briefly mentioned in the limitations section. The reviewer understands that this paper is a first step in that direction. Nevertheless, it would have been interesting to talk a bit more about the general problem. For instance, a discussion on the applicability of the methods employed in this paper to infinite-armed bandits setting would shed light on the importance of this work for the future research.
 Limitations of the work are addressed in discussion section. Negative societal impacts are not given, yet the reviewer does not foresee any such impact.

Meta-review: This paper used best-arm identifications to select the best estimator for identifying a causal effect. While the techniques are not new, the reviewers and the AC believe the result in this paper is an important first step. 
The AC also recommends that the authors add discussions on recent theoretical results (e.g., finite-sample regret analysis) of causal bandits.

Decision: accept