Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers evaluation and optimization of a policy that selects a binary treatment for each individual, who then interact via an optimization problem. The challenge, as it usually is with policy evaluation, is imputing the counterfactual outcomes that depend on the treatments assigned by the policy, which may not be the ones observed in the data. They consider the case where the counterfactuals are the coefficients of the optimization objective (or, thanks to duality, the constraints), which may be performed implicitly by a market, or explicitly by an operations system, after observing the outcomes induced by the policy. This differs from other perspectives on this problem, which focus on estimating the per-individual treatment effect after optimization, introducing serious challenges in SUTVA violations. The structure of the optimization problem and the policy evaluation / learning objective circumvents these challenges.
strengths_and_weaknesses: The problem of policy evaluation is a critically important one to the field of machine learning today, and this paper adds a valuable, original, perspective on how to think about policy evaluation and its relationship to treatment effects when individuals interact in a market or other system. The paper gives a convincing demonstration of the significance of the optimization problem structure that they study by giving examples of real, practical problems that can be formulated in their framework.

The issue with this paper is the clarity / soundness of the statistical model for the optimization problem. In Assumption 2.1, the coefficients of the optimization problem depends on the potential outcomes, $c_i(\pi)$. However, in Section 3, the problem changes and the coefficients are (abusing notation to emphasize the similarity) $\mu_\pi(w_i)$. Normally in policy evaluation or causal inference, this difference in irrelevant due to the tower law and the fact that $\mathbb{E}[ c_i(Z_i) \mid w_i, Z_i ] = \mu_{Z_i}(w_i)$. However, in this case, there is no expectation that allows one to use this property of the tower law. So, the objective has simply changed, and it's unclear if the new class of optimization problems is a reasonable / practically significant one to pursue.

One might hope, based on the aesthetics, that Assumption 2.2 wouldn't have this issue, given that the objective has an expectation that would allow the tower property to be used. However, I had trouble conceptually parsing Assumption 2.2, because I did not understand what type of mathematical object $x$ is. At first glance, it seems like a scalar. However, the description says that we should think of this expectation as the limit as we take the number of objects to infinite. So, it seems like x is now a mapping from the sample space of the probability measure for the expectation to $\mathbb{R}$. In this case, $x$ would not be measurable with respect to $w_i$ so that the desired tower property can apply. Additionally, I would wonder about measurability issues, although these are probably resolvable under reasonable assumptions and not a primary concern, just and indication of subtleties that can trip even the careful mathematician up.
questions: The main text says that Appendix B discusses why AIPW cannot be used, but I couldn't find any such discussion. Can you please point me to that?

The paper mentions that Shi et al. [2019] uses the GRDR method for CATE estimation, but the referenced paper seems to focus on ATE estimation. Can you please clarify?

The GRDR method seems potentially challenging for fitting with certain black box ML methods, but it looks aesthetically similar to the TMLE, with $\hat\mu$ fit first, and then $\epsilon_1, \epsilon_0$ fit afterwards. Would this have the same nice properties as the GRDR method that the AIPW lacks? Is it important that $\hat\mu$ is fit at the same time?

If the plug-in estimate with GRDR is equivalent to the AIPW (as quoted from Bang and Robins), and the AIPW cannot directly be used with the downstream optimization problem, why can the GRDR plug-in be used? They don't look quite equivalent to me, but I can't tell what the key property is that GRDR has and AIPW lacks.
limitations: Methodology for studying the potential implications of automated policies in economic and other social settings is the primary topic of the paper, and is discussed. Negative impact is implicitly discussed via discussion of the biases from using this methodology (and attempts to mitigate it), and via discussion of when post-policy optimization affects the potential individual improvements purportedly created by the policy.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studied a new framework for off-policy evaluation, where there is a personalized treatment policy on individual-level outcomes and a policy-dependent optimization response. Unlike previous work focus on the average outcome of individuals, this new framework concerns the output of the downstream decision making or planning problems. Under the new framework, the authors  propose a general algorithm that addresses the challenge of optimization bias for policy evaluation. Empirical synthetic results show the effectiveness of the proposed method under this new framework. 
strengths_and_weaknesses: - **Originality**.   
The authors proposed to study a new settings / framework that focus on the output of downstream decision-making problems, which is quite interesting and difference from prior works. As far as I know, this new framework has not been studied in previous off-policy evaluation / optimization works. 

- **Quality**.   
This paper is technically sound. The authors discussed the estimation and optimization bias under this new framework, and shows how to adjust the optimization bias based on a perturbation method. The authors also validate their method on synthetic data to support their claim. 

- **Clarity**.   
The paper is well organized and well-written. The authors first introduce the problem in Section 2 and the challenge of optimization bias in the proposed setting in Section 3, later the authors discussed how to adjust the optimization bias and conduct empirical experiments in Section 5. The whole story line is well organized.   
A small suggestion is that the authors may add some figures to explain the example 2.3

- **Significance**.   
This paper studied a new framework for off-policy evaluation. Since the focus of this work is quite different from previous where people concerns more about the average outcome, potentially this work may bring many new perspectives for causal policy evaluation and optimizations. 

questions: - The authors mainly focus on performing synthetic dataset experiments to validate the estimation properties. Do the authors think it is possible to conduct more realistic experiments? Though it might be challenging, I feel it would be a strong argument to support your claim that the new framework might be more important in practice. 

- Can you explain the computational cost of the proposed method? What if we have really large dataset? 
limitations: There is no potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considered the off-policy evaluation with the utility measured by the output of a downstream decision-making problem, instead of the classical average of individual causal outcomes across a population. The new target was further referred to as policy-dependent linear optimization responses. The authors constructed an unbiased value estimator based on a perturbation method, which addressed the fundamental challenge of an additional optimization bias. Theoretical results such as asymptotic variance properties for a set of adjusted plug-in estimators were derived, followed by extensive numerical simulations.
strengths_and_weaknesses: **Strengths**

- The authors tackled the off-policy evaluation problem from an interesting and novel angle by handling the policy-dependent linear optimization responses. 
- Related literature is well discussed and noted.
- The theoretical analyses are sound with promising simulation results. 
- This paper is well written and organized.


**Weaknesses**

- The authors may consider improving the presentation of important concepts. For example, are these concepts, *policy-dependent response*, *policy-dependent optimization response*, *downstream
policy-dependent response*, and *downstream decision response* the same or different? An organized introduction of these terms and consistent terminologies may be helpful. Though many examples the authors provided are quite helpful for understanding the policy-dependent response, I would expect an earlier high-level introduction of these important concepts in Section 1 and more detailed explanations of a decision problem $x$ on $m$ units in Section 2. 

- Though the authors show that Proposition 4.3 implies asymptotic unbiasedness of the proposed debiased estimator, I wonder if there is any possibility of getting the convergence rate of the policy evaluator as has been done by other recent OPE methods [Athey et al., 2017, Kitagawa and Tetenov, 2018, Kallus and Zhou, 2018b]?


Minor: 

- Please define and detail terms before using their abbreviation, such as augmented inverse probability weighted for AIPW in line 86, conditional average treatment effect for CATE in line 216, and GMM in line 246?
- Please consider introducing the notation before using it, such as $A$ in Equation (1). 
questions: Please consider answering the questions mentioned in the Weaknesses above.
limitations: I didn't find any limitations discussed. Please correct me if I am missing anything.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed an off-policy evaluation method through the output of a downstream decision making problem, which is interpreted as a policy-dependent optimization response. Optimization bias is addressed by extending an existing perturbation method for bias adjustment in parameter estimation to nonlinear predictions. The parameters of interest are of the outcome and propensity model. In the meantime, policy optimization is achieved by a subgradient method. Experiments conducted on simulated data show contribution from each algorithmic component. No comparisons to any baselines seem to have been shown.
strengths_and_weaknesses: Main strengths:
1 Clear assumptions, algorithm details and experimental setup
2 addressed a meaningful problem of optimization bias in OPE

Main weaknesses:
1 no comparisons to appropriate previous work
2 some variables are left unexplained e.g. what is constraint b, and terminologies unaligned e.g. is causal intervention equivalent to policy?
3 unclear whether the various definitions are self-defined or extensively adopted since there are hardly any references made, and it's hard to tell which parts are the authors' own takeaway message
questions: 1 does it mean you can have a policy trained on one task whilst evaluated on another task?
2 how is Eq. 4 different from a multi-task RL objective
3 from line 177 the estimation bias and optimization bias seem to be two components of the overall error, whereas from line 181 the estimation bias becomes due to optimization. Is there not any contradiction?
limitations: Yes
review_title: null

Meta-review: metareview: 
In the paper, the authors considered a new formulation for the off-policy evaluation and optimiation, in which the utility measured by the output of a downstream decision-making problem, therefore, the policy value is dependent on individual reponse. This difference induces additional optimization bias in the estimand and requires new technique to handle it. The authors constructed estimators by combining perturbation method with IPW estimators, and provided the theoretical guarantees and empirical study. 

Most of the reviewers provide positive feedback on this paper. It will be great if the authors can take the authors suggestions into account to improve the papers:

- Although as the authors explained the finite sample analysis is not new, for completeness, I think it will be great if this can be added into the paper. 

- As reviewer suggeseted, the method is motivated well from practical problems, however, has not justified with real-world applications. The paper will be significantly improved if this part can be added. 


Finally, there is naturally technique, interchangeability principle [1, 2], can be used to bypass the optimization bias, besides the perturbation method, which should be discussed and compared. 

> [1] Dai, Bo, Niao He, Yunpeng Pan, Byron Boots, and Le Song. "Learning from conditional distributions via dual embeddings." In Artificial Intelligence and Statistics, pp. 1458-1467. PMLR, 2017.\
[2] Shapiro, A., and Dentcheva, D. (2014). Lectures on stochastic programming: modeling and theory. SIAM (Vol. 16).
confidence: Certain
Decision: Accept