Venue: UAI 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility, incompletely addressed ethical considerations.
review_content: Q1 Summary and contributions: The paper proposes a method for algorithmic recourse to recommend overturning a machine learning decision. The idea is to minimize the posterior probability odds ratio, which is model-agnostic. The method is extended to a robust version that considers possible perturbations of the data. Compared to the related work, the method does not require a linear approximation step. 
Q3 Main strengths: 1) The paper tackles an interesting problem, and proposes a model-agnostic approach.
2) The approach removes the need for an intermediate linear model.

Q4 Main weakness: While the paper tackles an interesting problem, I have some concerns:
1) I am not sure how the method answers counterfactual questions without a causal model? 
2) The robust bayesian recourse does not cover all possible recourses that would change the decision? I am not sure how it will be used to turn over a decision.
3) The experiments section show good results on synthetic data only, and does not appear to be statistically significant.
Q5 Detailed comments to the authors: 1) Figure 1 does not appear to be referenced in the text.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: The method is not clear to me. In particular, how are counterfactual questions answered without having a causal graphical model? How do we control for spurious correlations? The experiments section would need to be further improved.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility, incompletely addressed ethical considerations.
review_content: Q1 Summary and contributions: A method is proposed for recourse: recommending an alternative, nearby value of the covariates for which a classifier would assign a positive outcome instead of the received negative outcome. The method proposed here is based on Bayesian posterior odds, and is constructed to be robust to changes in the classifier.
Q3 Main strengths: Robust recourse can be expected to be important in applications.
Q4 Main weakness: Section 2 gave me the feeling the paper was trying to sell the method as something it is not.
Q5 Detailed comments to the authors: Definition 2.1 defines a Bayesian recourse $x_\text{Bayes}$ as a point $x$ in some region minimizing $P(\hat{Y}=0 | X=x) / P(\hat{Y}=1 | X=x)$. This is equivalent to minimizing $P(\hat{Y}=0 | X=x)$. The relevance of the links drawn to Bayesian statistics in the next paragraph isn't clear to me because of this, and also because neither of the two variables involved is a model parameter.

In the paragraph below that, "we can argue that $P(\hat{Y}=0 | X=x_\text{Bayes})$ tends to be low" -- actually, you can say more strongly that $x_\text{Bayes}$ minimizes this quantity.

Other comments:
* Abstract: "probability odds" - these terms contradict each other
* The first sentence of the introduction reads like this paper will be about planning. I suggest rewriting this part of the introduction.
* page 2 top right: "our method ... employ" -> "... employs"; "the followings" -> "the following"
* Sampling scheme, bullet 3: min should be argmin

Q7 Justification for your score: Unfortunately, after getting a negative impression early on and not being familiar with other recourse literature, I could not justify spending more time to review this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: The authors proposed a method to provide a model agnostic algorithmic recourse that is robust to distributional shifts. The main contributions are: 1) unlike existing methods of generating recourse, the presented formulation does not depend on a linear surrogate of the original non-linear model. Instead a Bayesian approach is proposed to generate recourse. 2) to improve the robustness of the recourse, a method similar to adversarial perturbation of the empirical conditional distribution is used

Q3 Main strengths: 1. Formulation of recourse as a Bayesian optimization problem.
2. This approach considers the original non-linear classifier in finding robust recourse, unlike prior work that considers linear approximation of the non-linear model.
3. Reformulating the inner maximization as two subproblems that can be solved using projected gradient descent.

Q4 Main weakness: The experimental section requires clarity and improvement, and an analysis of the results is missing.
Q5 Detailed comments to the authors: Although the main paper is well written, it is difficult to follow the section on proofs; it would be good if you could restate the proposition and provide corresponding proof.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: This paper presented a new approach for finding robust recourse and showed its efficiency by good results. Apart from a few minor issues regarding the writing style of the experimental section and proofs, this paper has no significant issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: The paper proposes a Bayesian resource -- a model-agnostic method that aims to reverse algorithmic outcomes under potential model shifts. In other words, the method provides a way to answer the question "what action can be taken in order to receive an alternative algorithmic outcome". Bayesian resource minimizes the odds ratio between the posterior probability of negative and positive predicted outcomes and aims to be robust to changes in the parameters defining the classifier. 
Q3 Main strengths: The paper is very well-written and clearly motivated. The topic is quite specialized, but the problem is introduced in a very friendly way while not lacking in technical details. 
Q4 Main weakness: None that I can name. 
Q5 Detailed comments to the authors: While not an expert on this particular topic, I found the paper easy enough to follow and genuinely enjoyed reading it.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: The paper is very clearly written and very well motivated. The reproducibility of the results is supported by extensive details and submitted code.
review_title: null

Meta-review: Meta Review: The reviews of this paper were mixed. Two of the reviewers gave it solid "accept" scores, while two reviewers gave it a very low score. After the author feedback and discussion, one of those reviewers said that the response seemed reasonable (though did not update his score). The second reviewer did not believe the paper was "Bayesian," even after the feedback; the rest of that review was not very informative. I then checked the paper myself; the statistical underpinnings of the work are posterior (log) odds, so I believe it is, in fact, "Bayesian." The reviewer also had a low confidence score. Thus, I tend to discount that particular review somewhat.

For reference, the most confident reviewer recommended a "weak accept".

Aside from the "Bayesian" considerations, the reviewers agreed that the paper was clearly written, that the problem is generally important, and that avoiding the linear approximation common in similar approaches is an interesting, novel approach. Considering that, I tend to recommend accepting the paper due to its novelty, while acknowledging the writing could likely be improved in a future version.

Pros
* Important problem
* Model agnostic
* Removes linear approximations used in other methods
* Efficient optimization

Cons
* Does not discuss counterfactuals
* Experimental result discussion is very limited
* Proofs hard to follow


Decision: Accept (Poster)