Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a doubly robust estimator for counterfactual logistic regression with flexible constraints. The authors provide a novel algorithm along with theoretical guarantees and empirical evaluation. 
strengths_and_weaknesses: **Strengths:**
* addresses a common problem in causal inference (robust counterfactual or effect estimation with binary outcome)
* provides theoretical guarantees for both estimation and inference
* the presentation is clear and cohesive
* the empirical study is compelling

**Weaknesses:**
* The title is somewhat misleading as it only tackles a specific type of parametric classification (logistic regression). Does not easily extend to more complex classification methods
* I wonder how practical the main algorithm is. While I appreciate that the constraints can be very general, I wager that most applications will require a $L_1/L_2$ constraint (penalty) on $\beta$. One advantage of the naive/natural estimator in (2) is that it is equivalent to a weighted logistic regression (because $\\hat{\\mu}_a(x)\in[0, 1]$) so it can be used with any black-box logistics regression implementation (e.g. in Python). This is not true for the proposed estimator since some of the weights will be negative. Is there any simplification of the algorithm in the case of $L_1/L_2$ constraints?
questions: See section above.
limitations: The authors have adequately addressed and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to perform counterfactual classification (inferring categorical/binary potential outcomes) by minimizing a cross-entropy objective with constraints. The objective is not directly computable (because of unobserved counterfactual outcomes), so the authors propose to use uncentered efficient influence functions (based on model propensity and model expected outcome functions) to compute the objective. The authors then show double robustness properties of their estimate, i.e. they show that distance between their solution $\hat{\beta}$ and the true solution set $s^*(P)$ is upper bounded by the “estimation quality” $||\hat{\pi}_a-\pi_a||$ and $||\hat{\mu}_a-\mu_a||$ of the propensity & outcome models, and the achieved objective value $\hat{v}$ is similarly close to the optimal $v^*$. Under additional assumptions (LICQ, SC, B2), they show that $\hat{\beta}$ is $\sqrt{n}$-consistent and asymptotically normal. They illustrate the stated theoretical properties with a toy example, and they also evaluate the classification accuracy of their method on the COMPAS recidivism prediction task.
strengths_and_weaknesses: I will first list the weaknesses of the paper, then its strengths, then give minor comments.

$\textbf{Weaknesses:}$

- I think my biggest concern about the paper is that it is difficult to understand why we should care about solving problem (P) (Section 2). Typically, for categorical outcomes Y, people assess quantities like Relative Risk (RR) or (Conditional) Average Treatment Effects (CATE/ATE). They then can make claims about how good their estimates are relative to those “ground-truth” quantities (CATE/RR), and can analyze e.g., consistency & convergence to CATE/RR. But here it’s unclear why $\beta^*$ is an important quantity, and why we should care about consistency & rates of convergence of an estimate $\hat{\beta}$ to the “ground-truth” quantity $\beta^*$.
- Overall, I think the introduction needs some work. Currently, it is too short and does not motivate the proposed solution well. This is related to my first point – ideally, the intro should make it crystal clear why the reader/the causal inference community should care about solving the objective (P). Some concrete comments:

-- L35 “Our approach allows investigators to flexibly incorporate various constraints into the models [...] to accommodate a wide range of practical constraints relevant to their classification tasks.” – would be good to give examples here. Same comment for L54 “possibly with various constraints on our model.” Eventually, you do give some examples of constraints (L66/67) but these should be moved earlier in the text.

-- L37 “This poses both theoretical…” – what is “this” here? Ambiguous.

-- In general, the intro does not help me/the reader understand why your solution is appropriate, and what is wrong with alternative proposals.

--It wouldn’t hurt to further expand on the differences between your work and [31], i.e. expanding on the points in L77-79, and clearly delineating the differences.

$\textbf{Strengths:}$

Now that I’ve said the unpleasant stuff, I’d like to talk about the strengths of the paper. Suppose that the reader is convinced that problem (P) is worth solving, and that $\beta^*$ is indeed a quantity of interest. 

- Sections 2,3,4 are very well written, and it is mostly easy to follow the authors’ reasoning as they guide the reader to their solution.
- The theoretical results stated in section 4 are nice, and the authors use the toy experiment (Sec 5.1) effectively to showcase their theoretical results.

Minor/Misc.:
- Please make the y-axis the same in Figs 1 and 2 (i.e., 1 scale per row) so we can see the effect of X distortion.
- L158: “Here, we propose a more efficient estimator”. Respectfully, you did not propose this. Rather, you can say something like “we use…” and cite a reference for uncentered influence functions. This is a minor point, just a matter of language.
- Any practical guidance for readers on how to pick the basis functions $b(\cdot)$? How does the choice of basis function affect the solution to the problem?
- In problem (P), I would just write $\sigma(\beta^\intercal b(V))$ from the beginning, not $\sigma(\beta, b(V))$, this just makes it more difficult to understand (since you end up just using $\sigma(\beta^\intercal b(V))$ anyway).

questions: - Why do you make a point about runtime confounding if you are setting V==X in the toy experiment? Since you are making a point about how your method can deal with runtime confounding, I would suggest you either (a) showcase this or (b) remove the points about runtime confounding.
- On L110, $\sigma: \mathcal{B}\times \mathcal{X} \rightarrow (0,1)$ – should it be $\mathcal{X}$ here? Since we are using the features b(V) as input?
- How do you justify the distorted X? Can you provide intuition/an explanation for why the plug-in method "fails" here? Would be good to put this in the text too.

limitations: Yes, I think the authors adequately address limitations of their work & concretely outline next steps.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an approximating program to the original so-called counterfactual classification program that can incorporate flexible constraints. The approximating program utilizes doubly robust estimators so that the optimal solution is $\sqrt{n}$-consistent as long as nuisance functions satisfy the stated assumptions. 
strengths_and_weaknesses: Originality: 4. The whole strategy is similar to DR-learner, and the work is close to "Fair double ensemble learning for observable and counterfactual outcomes." But involving a flexible constraint is interesting, and the asymptotic property analysis for beta is solid.

Quality: 3. I think the experiments are not so rich. There are also some typos or unclear statements.

Clarity: 4. The logistics and mathematical proofs are clear. But it would be great if the authors can state the connection between the so-called counterfactual classification and general causal inference problems.

Significance: 3. I think this method does not bring significant improvement compared with simpler methods.
questions: 1. If I am right, I think your "counterfactuals" include both observed and unobserved outcomes. More precisely, I think it should be called "potential outcomes" in your setting. It is inaccurate if regard counterfactuals as potential outcomes.

2. The key is to estimate Y^a more accurately. Why can't we just use mu_a(X) or varphi_a(Z; eta) to estimate Y^a, instead, we should solve beta from P?

3. Assumptions A2 and A3 seem to limit the convergence rate for nuisance parameters. Your claim that "beta can attain sqrt{n} consistent even nuisance parameters have slower rates" seems not consistent with the arguments A2&A3. As I expect, nuisance parameters should converge at a rate of o(n^{-1/4}) if beta is sqrt{n} consistent.
 
4. The quality of beta depends on i) the estimation of nuisance parameters pi_a and mu_a, ii) the plug-in or doubly robust learner. You only consider a case of ii), but I am wondering what the results would be if pi_a is not correctly specified. In this case, will the proposed method be better than the simple plug-in?
limitations: In Section 6, the authors well discuss the limitation and potential extensions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to solve counterfactual classification problems by solving a doubly robust estimator of the cross-entropy loss with smooth nonlinear programming, incorporating constraints. Theoretical analysis is provided in the asymptotic limit, including the rate of convergence of the solution to the set of optimal solutions, and asymptotic normality of the solution under the assumption that there is a unique solution. 
strengths_and_weaknesses: Strengths

This is a well-written paper with sound and clear presentation of the mathematics. It makes clear the problem it is trying to solve, as well as the method it deploys to solve it. 

The cross entropy loss function is widely-used, and the way authors extended it to the counterfactual setting, subject to constraints, seems natural and intuitive to me. The scope of contribution might be a bit limited, as it feels like a rather simple and straightforward marriage of very well-known and widely-used problem and technique, I still think the theoretical analysis and the algorithms are valuable to the community. 

I do not work directly in the field of counterfactual prediction, where the accurate prediction of each counterfactual case is important rather than a comparison between counterfactuals (e.g. CATE), but the authors convince me that it is an important problem. Given this, and given how commonplace and intuitively important classification is, I find it surprising that no attempt has been made so far to tackle this problem. 

Weaknesses

I do not like the fact that the proof of Lemma A.1 was omitted. I had a look through [21], to see if things really carried over seamlessly, but at least to me, things weren't so obvious. In particular, I would like to see worked out how a factor of h(X) makes no other difference than simply multiplying the influence function by the same factor, and why the authors choose to work with the uncentred influence function rather than the centred one. Perhaps it's obvious to people working in the field, but I have not seen such formulations in other papers. 
questions: lines 110-112: sigma is a function from \mathcal{B}\times\mathcal{X}, so b(V) should be in \mathcal{X}, but on line 112, b(V) seems to be in R^k? I see that later, authors restrict attention to the simple case of \sigma(\beta^Tb(V)), which means that \beta and b(V) indeed has to be in the same space. But before this restriction takes place, wouldn't it be better to retain generality and let b(V) take place in \mathcal{X}?
Also, could the authors comment on why they restrict to finite (k-dimensional) parameter space / basis functions? This seems to rule out many useful cases; for example, the authors list the use of kernels as an example, but useful kernels such as the Gaussian kernel seem to be ruled out by this finite-dimensional restriction. 
limitations: Limitations are discussed at the end of the paper, and in my opinion, sufficiently. The work is mostly of theoretical nature, and I do not deem it necessary to discuss potential negative societal impact of the work.
review_title: null

Meta-review: metareview: Reviewers agreed that the paper makes interesting theoretical contributions to the problem of counterfactual classification, with a novel and valuable result. The reviewers pointed out several aspects in which the empirical evaluation could be more thorough, though it is my view that is definitely sufficient for validating the paper's main claims. Finally, the paper is well-written and presents the math crisply. 
confidence: Certain
Decision: Accept