Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a robust decision learning framework with quantile or infimum optimal treatment objective. Their idea is to improve the worst-case scenarios of individuals when decisions, with uncertainty, are required, to be made with sensitive albeit important information missing. 
strengths_and_weaknesses: # Introduction

- First paragraph: I do not think this is convincing. Certainly there are sensitive variables, but a few real-world examples would help solidify your examples. They certainly crop up in the media from time to time, but which organisation is actively pursuing this design philosophy in their models? Please give real examples.
- L31: you say “The main question of interest is whether the learned IDR could yield similar outcomes across all realizations of the sensitive variables.” -  why is this the main question? I do not follow the logic as to why focus is on this.
- Table 1: you’ll need to put some work into re-jigging this table. The expectations are the coloured (and black) entries but you’ve but the expectation column-header over the action-space column. This is confusing. Further, the whole example is somewhat confusing. Attack your example with the assumption that your reader is not an expert in your field and so start from first-principles, be verbose, and explain like you would to a novice.

# Related work

- You use a version of this comment a few times in this section: “See [your listed references] and references therein for a comprehensive review”. That is not helpful. Give us a summary of what those sources say rather than divert your reader to another suite of papers. You are meant to give us a micro-review herein as this is the related work section, that you have chosen to include (you don’t have to). Now you are just giving us a more reading to do. This is a conference paper, not a journal paper, so you are forced to compress and combine. Right now what you are doing is not helpful to our (the reader’s) understanding of the state-of-the-art in this area.

# Robust Decision Learning Framework with Sensitive Variables 

- Equation 2: why is $d^*$ not equal to the right-hand side but lies in the set of optimal IDRs? Does equation (2) yield one optimal IDR or multiple? If it is the former then $\in$ should be an equality instead? If the latter, please explain further how the argmax yields multiple optimal IDRs. Algorithm 1 seems to suggest that one IDR is output by RISE.
- You appear to have dropped $d \in \mathbb{D}$ at the bottom of page four, in the aromas.
- What is the function of $n^{-1}$ in equation 3? Or rather, why the inclusion of a weighting?
- Algorithm 1 would be more elegant if you looped over n rather than saying “for every i”. 

# Numerical studies

- Line 250: it would appear that you’re missing the formatting for the indicator function in your estimator for $V(d)$.
- It would be helpful, at least for the synthetic experiments, to visualise the underlying causal structure and if possible, how it fits in the IDR - the causal diagram is a key-part of causal inference so it would be interesting to see how you graphically include it (if possible).
- Line 309: what on earth is a “causal forest” your reference title, [45], says ‘random forest’ but you must be referring to something else, hence what is a causal forest? Is there a reference?
- Line 328: again causal forest - reference?
- Line 341: explain why “lactate and Sequential Organ Failure Assessment (SOFA) score 4 hours before sepsis onset” are sensitive variables?

# Discussion

- Line 372, typo: “find” —> “finding”
questions: See main review in Strengths and Weaknesses.
limitations: See main review in Strengths and Weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is concerned with identifying the best decision option for individuals in the presence of sensitive variables---variables that are relevant to the decisions but are not available at the time the decision is made---at inference time. Existing decision-making frameworks may simply ignore sensitive variables or seek to optimize the mean utility overall all the individuals. Such solutions might lead to bias decision-making or unfair outcome toward different subgroups in the population. To resolve these issues, the paper takes a robust approach that optimizes with respect to the best worst-case scenarios. It does so by solving a weighted classification problem. Experiments on both synthetic data and three real-world datasets are conducted to demonstrate the performance of the proposed method.
strengths_and_weaknesses: Strengths:
* The paper is concerned with an interesting and important problem in decision making.
* The proposed approach that optimizes with respect to the best worst-case scenarios makes sense, and the paper also offers a conversion of the problem into solving weighted classification problems.
* The paper is clearly written overall with intuitive explanations and examples provided.

Weakness:
* While the paper offers an empirical evaluation of the proposed method over both synthetic and real-world datasets. It is not clear to me whether the competing approaches that the proposed method compared to are competitive, representative, and exhaustive.
* The method also operates under the unconfoundedness assumption, which might not be easy to satisfy in practice. It will be desirable to carry out further synthetic experiments to understand the robustness of the proposed approach when various assumptions of the proposed approach is violated.
questions: * Can the authors please clarity what is the meaning of "individualized"? For example, in equation 2, there is an expectation over $X$, which seems to indicate the average utility rather than the individualized utility?
* Related to this, in line 174-175, the author also mentioned that the decision rule is only related to $X$. Does that mean the decision rule is not related to $S$? If so, it is a bit counter-intuitive that it is still necessary to factor in $S$ in decision-making.
limitations: The paper discussed limitations. It would also be desirable to discuss whether it is possible to further relax some of the assumptions that the paper is based on (e.g. unconfoundedness).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for robust decision-making when there is some important attribute that we have at training time, but not test time. Using a potential outcomes framework, they discuss an robust objective which aims to output good decisions for a range of values of the unavailable attribute. Studies with simulated and real data support that this method improves on "vulnerable" subjects.
strengths_and_weaknesses: Strengths:
- This is a useful idea and not one that I think I've seen directly proposed before
- pretty clear exposition throughout section 3 of the idea
- I appreciate the datasets in section 4.2 - seems like the authors used pretty interesting, challenging data for this evaluation

Weaknesses:
- language around "sensitive" attribute seems a little general. I understand why it's used in some sense, but really it's just any attribute we won't have access to at deployment time - as shown in section 4, this could be a whole range of things which are not really aligned which what we consider "sensitive" information
- similarly, I would say that the "vulnerable" language doesn't really align with what I would consider a vulnerable population. Rather, it's a post-hoc definition of who the standard mean-outcome model doesn't work for
-for this reason, I find the results in Table 4 a little bit hard to interpret. The paper post-hoc defines who is in our vulnerable group as exactly those users that the baseline didn't work for, and now shows that the proposed method improves on them. This would be more convincing if it was a coherent group that we saw improvement on, as we could just as easily construct such a group with the reverse property (that the baseline performs better on). Unless I'm misunderstanding this!
- unclear about how estimation works of the quantile objective when X and S are correlated - what if we don't observe many values of S for   regions of X? Are we relying mostly on the extrapolation ability of our Y-hat model?
-it's a little odd that in Table 4 RISE is best on the whole data, makes me think the results may be fairly noisy
- re: novelty, there is some related work I mention in the "Questions" section which may make the statement "we are among the first to propose a robust-type fairness criterion under causal inference" a little less true - although depends on exactly what those terms mean here

Minor points:
- line 5: it's not really convention in fair ML to ignore sensitive variables anymore
- line 201: why would negativity make (3) hard to solve?
- line 204: clarify you're minimizing over f, and its range
- is the job training data from an RCT?
-line 350: rational -> rationale



questions: - seems fairly close in concept to Makar et al. "Causally Motivated Shortcut Removal Using Auxiliary Labels". Would be good to see some engagement with that work and comparison to it in terms of what each paper is aiming for (at least in the rebuttal)
- language of "sensitive" and "vulnerable" don't align that closely IMO with the algorithm proposed - I'd either target that more specifically in experiments or change the language
- how is the vulnerable group calculated in Table 4, and why is this a useful group to look at for evaluating our algorithms?
- what if we don't observe many values of S for regions of X? Are we relying mostly on the extrapolation ability of our Y-hat model? Could this create problems?
limitations: no suggestions
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes RISE, a robust individualized learning framework that improves the worst-case outcomes of decision when sensitive variables are unknown in test time, thus reducing the variation across subjects. The robustness considers the uncertainty of outcome distribution due to sensitive variables and focuses on the protection to subjects in the lower tail of the outcome distribution. A corresponding learning objective is theoretically defined and is transformed to an equivalent weighted classification problem for optimization. Experiments on real-world examples show good performance of the method and its importance from fairness and safety perspectives.
strengths_and_weaknesses: ### Strengths:
1. This paper proposes a new perspective on measuring individualized decision with fairness or safety concerns, which can be extremely useful in many practical scenarios.
2. The theoretical modeling and derivation are very clear and detailed.
3. The paper is well organized and well written.

### Weaknesses
1. The related work on Robustness in individualized decision rules is insufficient, so it is hard to judge the innovation of this paper in terms of robustness learning, because similar treatment regimes as quantile-optimal have already widely used in various cases.
2. This proposed robust learning objective is essentially a minor modification to change the expectation $E_{s|x}$ into another measure, $G_{s|x}$ , which is rather intuitive with no technical innovations.
3. The derivation of the transformation from the raw problem to a weighted classification problem is not of great significance for practical training. As the optimization object is rather simple, it is reasonable to believe that neural networks can directly handle the original one.
4. The comparison in the experiments is inadequate with only vanilla baselines provided. For a fair comparison, more modern methods should be included with appropriate adjustment on their objective function to fit the proposed learning objective in this paper.

questions: 1. Should the framework of RISE be regarded as a generalized form of individualized decision rules rather than a robust one? The proposed objective considering quantile- or infimum-optimal is not a must for the derivation. In other specific scenarios, for example, the venture capital investment, the highest reward might be the concern, which can also be included in this framework.
2. Is there any error between the simulation results shown in Table 2 and its theoretical results? The qualitative conclusion is clear, while the quantitative results are confusing. For example, does the numeral result 14.4 corresponds to a theoretical value of (30+0+5+27)/4=15.5?
3. Does the framework in this article refer to some works in section 'Robustness in IDRs'? All these methods are summarized with only one sentence in line 109-111, which is not informative enough.
4. The objective of RISE seems to be a protective weight increase of vulnerable groups, just like the false negative punishment in classical medical statistics to avoid serious consequences of undetected diseases. Therefore, this problem is obviously a weighted classification problem with an adaptive weight according to the objective function. Thus, the framework seems to be another paraphrase of a classic problem. Are there any other differences to be considered?
5. Have you tried to optimize the objective function in (2) directly with a neural network? It does not seem to pose a challenge. The transformation to classification problem may only serves for traditional approaches as SVM or decision trees.
6. Are there no other suitable baselines for your experiments? If so, the objective function of some other methods should be modified for your proposed task as comparison.
limitations: The discussion on limitations is sufficient.
review_title: null

Meta-review: metareview: Reviewers are on the whole positive about this paper, after detailed responses from the authors. If this paper is accepted, many readers will be interested, and it does have the potential to be useful for real-world applications in medicine and elsewhere. Therefore, on balance, the paper does reach the standard required for NeurIPS.
confidence: Less certain
Decision: Accept