Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies off-policy evaluation from data generated from unknown and adaptive logging policies (logging policy can be updated based on past observations). The paper modified the doubly robust estimator in the non-adaptive and known-logging-policy setting to this setting by using an estimated logging policy and an estimated reward model. The estimation models used at each time step are built only on data collected before. They show the asymptotic normality of this estimator under some assumptions and make the step-wise construction and how it achieves asymptotic normality as a general approach. They empirically evaluated the proposed estimator on some simulation dataset and  discussed why the proposed estimator enjoys smaller mean squared error empirically compared to estimators with known logging policy information, even when they all have asymptotic normality. 
main_review: Off-policy evaluation with unknown and adaptive logging policies is pretty realistic setting in real-world applications. 

The paper propose an off-policy evaluation estimator in this setting and characterize the conditions under which it can achieve asymptotic normality, which provides theoretical groundings for using this estimator. 

One question I have is, if we apply adaptive-fitting to DM and EIPW estimators, do we achieve asymptotic normality. And it would be great to also include the experiments comparing with these two baselines as reference. 


Minor issues:

The acronym AIPW is very confusing. I do not know what 'A' stands for and I always thought it was adaptive when I saw it. But it was not. 

line 226 satisfies what? 

line 360 as section 6.3 -> as discussed in Section 6.3

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In the setting studied in this paper, the analyst has access to data consisting of treatment assignments $A_t \\in \\mathcal{A} = \\{1,...,K\\}$, covariates $X_t$ and responses $Y_t = Y_t(A_t)$, where $(Y_t(a))_{a \\in \\mathcal{A}}$ are potential outcomes. The data are collected at time steps $t=1,...,T$ and the data generation occurs through an adaptive experiment. That is, at the $t$-th time step, the analyst can use all information up to time $t-1$, as well as $X_t$ and can decide with what probabilities $\\pi_t(a \\mid x)$ to assign any of the treatments in $\\mathcal{A}$ to the subject arriving at time $t$.   In other words, $A_t \\sim \\pi_t(\\cdot \\mid X_t)$ and then $Y_t = Y(A_t)$ is revealed.

The consequence of the adaptive experimentation is that the tuples $(X_t, A_t, Y_t)$ are dependent across time-steps $t$ (even though $(X_t, (Y_t(a))_{a \\in \\mathcal{A}})$ are iid and therefore it becomes a challenging task to conduct inference for average treatment effects (and other causal contrasts/estimands of interest).


Existing work for inference has focused on the situation wherein the policies $\\pi_t( \\cdot \\mid x)$, that were used during experimentation, are known to the analyst, while the response functions $f^*(a, x) = E[Y_t(a) \\mid X=x]$ are not known. The present paper develops a doubly robust approach to inference that also works when $\\pi_t( \\cdot \\mid x)$ is also not known to the analyst. 

The paper also emphasizes that the new approach can sometimes outperform existing approaches that use additional information (on the true treatment assignment probabilities), without using the additional information (they call it a paradox).


main_review: 
The paper is well-written and the assumptions are clearly stated. Adaptive experimentation is bound to become more popular, and most traditional statistical approaches break down in that setting. Instead new approaches are needed. Hence it is an important area of research, and this paper further contributes to this area.

1) The main aspect that is missing from the paper, in my opinion, is some motivation for the statistical setting. I find it hard to imagine that someone has access to data from an adaptive experiment (and e.g., the temporal succession of treatment assignments $t=1,...,T$ is known), but does not know how the experiment was actually implemented (i.e., how the treatment assignments were sequentially randomized). When would this occur? This is in contrast to e.g., the typical setting for doubly robust estimators for estimating average treatment effects in observational data. There, the treatment assignment is clearly not known and one estimates propensities and potential outcome regression functions to deal with confounding (under weaker conditions compared to both the IPW approach and the outcome modeling approach).

2) I wonder whether the novelty of the "adaptive fitting" procedure introduced by the authors may be overstated. For example, Hadad et al. also fit the outcome models $f^*(a, x)$ based on data up to the previous time point (similar to the adaptive fitting in this paper), so that I am not convinced of the novelty of this methodological contribution. I do agree however with the authors, that it appears that from a technical perspective, also having an estimated $\\hat{\\pi}_t(\\cdot \\mid X_t)$ based on past data (and not just response function) would make the theoretical analysis more challenging.

3) What is the take home message from the "paradox"? Can the authors provide practical recommendations about what to do in case the treatment assignment probabilities are actually known?


Some minor remarks: On Page 6 there is typo "wit" instead of "with". I also wanted to let the authors know that  Zhan et al. (2021) have a follow-up on the paper of Hadad et al. that accounts for context covariates (https://arxiv.org/pdf/2106.02029.pdf). (Of course the reviewed manuscript precedes the preprint I linked above, so I just provided the above reference in case it is useful, the authors do not need to cite it or compare to it.)

**Update:** Based on the author response, one take-away is that the proof techniques and technical results are a key contribution of this paper. I have therefore increased my score from 5 to 6. I am still not entirely convinced however about the practicality of the statistical setting (i.e., I am not convinced that a tech company doing adaptive experimentation would have trouble storing the experimental protocol).

limitations_and_societal_impact: The authors discuss some limitations (but not all, see review above).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose and analyze an estimator for off policy value estimation (OPVE) in a Markovian setting where (X_t,A_t,Y_t(A_t)) ~ p(x)p_t(a|x)p(y_a|x) and the logging policy pi_t(a|x,Omega_{t-1}) is unknown. The estimator involves a familiar doubly robust moment function and a new form of cross fitting called adaptive fitting. The authors prove consistency, asymptotic normality, and semiparametric efficiency. Thorough simulations demonstrate low RMSE and correct coverage.
main_review: Overall assessment

Originality: The innovation is to allow the logging policy to be unknown. For the theory to extend to this case, adaptive fitting is introduced as an elegant solution that appears to be new. Please also compare with Kallus and Uehara’s mixingale sample splitting to shed light on originality.

Quality: The estimator and analysis are interesting. I am concerned about whether the Hadad et al estimator (2019) was given a good faith implementation since its coverage is zero (when it should be 95). 

Clarity: This paper is unusually well written. See comments below about specific points to clarify or assumptions to further explain.

Significance: The contribution is clearly explained and not overstated. It is a natural progression of existing results that is relevant to empirical practice.

Suggestions to improve the paper

5: with non-Donsker

54: Nicely written

57: Nicely written

67: Nicely written

90: Crystal clear. The Markov property is a major simplification relative to dynamic treatment effects.

116: Please give further interpretation of Assumption 2.2. Unlike the other assumptions, it is not obvious. What cases are being ruled out?

145: the MSE to the true R(pi^e)? What does this mean given that R(pi^e) is a scalar for a given pi^e?

153: Impressively thorough

169: What is mixingale based sample splitting? How does it compare?

170: It would be nice to additionally mention the sample splitting of Chiang et al [arXiv:1909.03489]

180: please define MAB

264: What does this sentence mean?

Table 1: I am happy to see RMSE, SD, and CR

303: Please summarize the DGP in a clearer way, perhaps as an offset algorithm

328: It does not seem that the Hadad et al (2019) approach was given a good faith implementation. If their proposed weights do not include covariates, what would be a reasonable extension with covariates? Indeed, in Table 1, the coverage of AW-AIPW is even zero, which suggests this was not a reasonable implementation.
limitations_and_societal_impact: What assumptions could be violated, and how would that impact conclusions and policy recommendations?
review_title: null

Meta-review: The paper studies off policy estimation using data collected by a changing (adaptive) and unknown logging policy. This is an important and under-explored setting, since adaptive experimentation is increasingly common in many application domains (e.g., online platforms). The paper gives an adaptive doubly-robust estimator, establishes semiparametric efficiency of the estimator, and also provides some nice simulation studies that showcase how fitting propensities may be beneficial even when the true ones are known. 

Overall, all of the reviewers were quite positive about the paper, and so we are recommending acceptance. Please incorporate comments from the discussion section into the final version, as this will clear up some confusions and improve the manuscript. 
Decision: Accept (Poster)