Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: To explain machine learning models, the paper proposes learning counterfactual example rules (regions) rather than generating individual counterfactual examples. This has benefits in stability and avoids the variance of producing individual examples. 
strengths_and_weaknesses: ### Strengths
* Useful formulation to think of counterfactual regions versus examples.
* Technically sound method using random forests that avoids discretization parameters in past work
* Evaluation is reasonable and compares to past work.

### Weaknesses
* The writing can be improved. The idea of using Random Forests and ignoring Sbar features is simple enough. It will be nice if the paper can provide intuitive examples as in Figures 2,3,4 before presenting the technical details.
* possibly similarities to the Anchors paper that has not been discussed https://homes.cs.washington.edu/~marcotcr/aaai18.pdf

The paper solves a well-motivated problem. Counterfactual example methods, by their nature, can be unstable and lack guarantees. Providing rules or sets of feature values instead of individual examples is an important advance. I wish the authors had engaged more with explanations literature outside CFs that does propose such rules (e.g., the Anchors paper linked above). 
Other than that, I think the paper is technically strong and I like the use of Random Forest as an estimation tool given its non-parametric properties. The writing can be improved to make it more intuitive. It will help to describe section 5 in an algorithm or a table to understand the different steps. It looks like there are multiple steps in the optimization and it is unclear how they interact. 

questions: * Can the approach from Anchors be used in a similar way to solve the CF regions problem?
* If I understand correctly, the output policies/regions may not be 100% correct (i.e., may not flip the label). Wouldn't this violate the expectation of a counterfactual explanation?
* Can you provide some examples of the policy learnt by the model, e.g., on Compas? A qualitative analysis of the regions and their interpretability will help.
limitations: The provided policies/regions may not be 100% correct. That is a limitation.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper aims to address multiple problems with counterfactual
explanations that consist of single points, by replacing them by
rectangular regions that switch the class $f(x)$ of an input x with high
probability. These rectangular regions can then be communicated in the
form of rules, which are called Local Counterfactual Rules. There is
also an extension called Regional Counterfactual Rules, which explains
regions instead of single inputs x.

The definitions of the local and regional counterfactual rules rely on a
certain conditional distribution of the inputs, which therefore has to
be estimated very accurately. An important observation of the paper is
that this estimation can be made computationally efficient if $f$ is first
approximated by a random forest $\hat{f}$.

Finally there are experiments that compare the new methods to existing
methods on several data sets.


strengths_and_weaknesses: The structure of the paper is to discuss a series of limitations of
existing methods in the literature, and then to posit new definitions
for Local and Regional Counterfactual Rules. Unfortunately, however,
there is very little motivation for why the new definitions should be
exactly the way they are, and what their strengths and weaknesses are.
And they do strike me as having at least the following significant
weaknesses (I will focus on Local Counterfactual Rules here):

* Definition 3.3 of Local Counterfactual Rules only provides a necessary
  requirement that the rectangles should satisfy, so it is in fact not
  a full definition.
* The rectangles produced by Local Counterfactual Rules may
  actually be *implausible*, i.e. definition 3.3. leaves open the
  possibility that P(C_S(x;y*)) is arbitrarily small, because it is only
  a requirement about a probability conditional on C_S(x;y*).
* The approach critically relies on estimating a conditional probability
  for x. There should be a discussion on when this is reliable. For
  instance, this cannot work well when explaining inputs x from
  low-probability regions for which the exact distribution cannot be
  estimated very well, or if the true distribution is very irregular, or
  if the input dimension is large. The conditional probability is also
  estimated for each possible rectangle C_S(x), which will inflate the
  estimation error.
* It is not clear how a user is supposed to obtain recourse from hearing
  a rectangular region. Definition 3.3 implies that, if the user picks a
  point from the region according to the true distribution of x, then
  they will obtain recourse with high probability. But a) the user does
  not know the true distribution of x, and b) even if they did, it would
  seem unreasonable to expect them to modify their features in
  accordance with this distribution. The introduction (line 98) claims
  that their approach "partly answer[s] the problem of noisy responses
  to 99 prescribed recourses" identified by Pawelczyk et al., 2022, but
  this claim is not backed up.

Presentation:

The presentation is sufficient, but clarity could still be improved
and claims could be substantiated better. For example: 
- There are multiple cases where terms are unclear or become clear only
  later, especially in the introduction. For example, line 51 refers to
  "the best promising directions", but it is not clear by which measure
  'best' is determined.
- In line 34 it is claimed that solutions from prior work are "not
  entirely satisfactory", but it is not specified in which way they are
  not satisfactory.
- Definition 3.2 confuses the distinction between a "Divergent
  Explanation" and a "Minimal Divergent Explanation".

questions: The limitations of the paper seem very fundamental, and would require a
significant revision to address.

limitations: Limitations of the approach have not been sufficiently addressed, as
discussed above. In particular, it is not clear when it can be expected
to work and when not, or how to detect whether it can be applied for a
particular type of data.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose to counterfactual rules instead of counterfactual samples as explanations -- i.e. the explanation is a (set of) rule(s) instead of single sample (counterfactual).
strengths_and_weaknesses: Good paper!

Only minor issues:
I think an illustration inf the form of a 2d data set + decision boundary might help even more to understand Local and Regional Counterfactual Rules.
I am bit concerned about generating rectangles for high-dimensional spaces (i.e. curse of dimensionality) -- maybe the authors could briefly comment on this.
If I understood it correctly, the users always used a LightGBM as a black-box model that is going to be explained. Since the authors propose a model agnostic method, I think evaluating on a single model only is enough -- the authors should consider several highly different models for their empirical evaluation.
questions: None
limitations: None
review_title: null

Meta-review: metareview: This paper highlights a series of limitations of existing methods in the literature on algorithmic recourse (e.g., recourses are not implemented exactly and are often noisy), and posits new definitions for Local and Regional Counterfactual Rules and proposes a novel algorithmic framework to learn them. All the reviewers opine that there is very little motivation for why the new definitions should be exactly the way they are, and what their strengths and weaknesses are. In addition, it is unclear when we can expect the proposed approach to provide a good estimate of the criterion. Furthermore, the problems highlighted in this work have been explored in several recent works (e.g., Dominguez-Olmedo et. al., ICML 2022, On the adversarial robustness of causal algorithmic recourse; Pawelczyk et. al., 2022, Let Users Decide: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse). We encourage the authors to address the aforementioned aspects, discuss related works, and also compare the proposed approach with these works. 

confidence: Certain
Decision: Reject