Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides a framework for recourse (i.e. counterfactual explanations) that is robust to shifts in the model. They formulate the robustified recourse setup as a min-max optimization problem, where the max is over a neighborhood around the distribution over model parameters. The model parameters are drawn from a mixture of K distributions, so that the neighborhood is specified by Gelbrich distance on each component. 

They propose a finite-dimensional version of the robustified optimization problem, which can be optimized using projected gradient descent. They evaluate their approach on the German credit dataset, the Small Business Administration dataset, and the Student performance dataset, each of which demonstrates a different type of data distribution shift. 
main_review: The setting of recourse in the presence of model shifts is well-motivated, especially since models are often updated over time. The idea of formulating the problem as a distributionally robust optimization problem is compelling. 

One weakness of this paper is that the technical solution provided is somewhat limited. In particular, the formulation in (4) relies heavily on the structural properties of the mixture distribution and Gelbrich distance to reformulate the optimization problem, and is not surprising given these assumptions. Since this is the main technical result in the paper, it would have been interesting to see a more general setup that considered richer distance metrics. 

For the empirical section, a weakness is that DiRRAc (the method proposed in this paper) seems to optimize for ell_2 cost, whereas existing approaches seem to optimize for ell_1 cost. Thus, it is not clear how to compare the costs obtained by DiRRAc with the costs obtained in existing approaches. Moreover, it could be interesting to further investigate the tradeoffs between validity and cost for DiRRAc to illustrate the "cost of robustness" incurred by the approach. 

Minor comment:
- Distribut,lly -> "distributionally" p. 4
summary_of_the_review: Weak reject due to limited technical contribution and limited empirical comparison   

-----
Update after author response: I appreciate the additional experiments for l1 cost included in the revision. Moreover, although I appreciate the author's discussion of the motivation for the Gelbrich distance, I still think that the technical results are somewhat limited. Thus, I keep my assessment the same.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of recourse actions (a.k.a. counterfactual explanations) while considering data distribution shifts or model shifts. The proposed Distributionally Robust Recourse Action (DiRRAc) framework has the ability to generate valid recourse actions when model parameters shift over time. DiRRAc adopts the distributionally robust optimization technique and the paper proposes a projected gradient descent method to solve the optimization problem. Experiments are conducted with both synthetic and real world data, and the results have shown that DiRRAc methods can generate recourse actions with higher validity than two existing methods.
main_review: Strengths:
1. Most existing work on recourse actions do not consider model change, so the problem addressed by the paper is relatively new, and it is an important problem since model/data shifts are common in practice.
2. The idea of considering/modelling model shift as a mixture shift of model parameters, and formalizing the problem as a min-max problem.
3. The experiment results demonstrate the superiority of DiRRAc over the methods compared.
4. The paper is well written.

Weaknesses:
1. It is not very clear how challenging it is to adopt the distributionally robust optimization technique for solving the recourse action problem. It would be useful to let readers know clearly that the adoption is non-trivial, which is particularly helpful for readers who are not familiar wi the distributionally robust optimization techinque.
2. From the paper, ROAR is a method for generating counterfactual explanations that are robust to model shifts, but the experiments conducted do not consider ROAR as a baseline.
3. It is not clear how efficient the proposed method is, compared to existing methods.
4. The performance of the proposed method under no model shifts should be evaluated as well.

Other comments: 
1. on page 2, there is a typo: Distribut,lly --> Distributionally
2. The paper does not have a Conclusion section.


summary_of_the_review: The paper tackles a very practical and relatively new problem regarding recourse actions. The overall idea seems reasonable to me and the experiments have demonstrated the effectiveness of the proposed method. The paper is also well written. 

However, the paper has a few weaknesses as described above. The missing comparison with ROAR is a main concern. The novelty of the proposed method (regarding the adoption of distributionally robust optimization) should be clarified too.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the problem of generating recourse actions that are robust to shifts in the parameters of the classifier. The authors present a distributionally robust optimization approach and experimentally show that for linear classifiers and under no actionability constraints, the approach generates recourse actions that have high probability of being valid under shifts to the weights of the linear classifier.
main_review: Strengths: very clear and thorough exposition of the approach proposed. The approach proposed is very sound technically.

Weaknesses: the experiments presented are rather limited and the value of the contributions is unclear. In particular, the problem of generating robust recourse was previously considered by Upadhyay et al., 2021, but the authors do not provide any evidence as to why their approach may be preferable, which is particularly concerning given that the experiments considered by the authors are heavily inspired in those of Upadhyay et al., 2021.

To strengthen the contributions of the paper, the authors should focus on improving the experiments section. The contribution of this paper would be much stronger if the authors compared the performance of their approach to that of Upadhyay et al., 2021, validated the claim that “robust optimization solutions can be overly conservative because it may hedge against a pathological parameter in the uncertainty set” in the context of algorithmic recourse, and showed that their proposed approach overcomes this issue.

Detailed comments on the experiments section:
* As previously mentioned, authors should compare their approach to Upadhyay et al., 2021.
* Why was L1 not used as the cost function similarly to AR and MACE? Is this an inherent limitation of DiRRAc? If possible, it would be best to use L1 as the cost function for all three approaches.
* It would be very valuable to also include experiments for non-linear classifiers (e.g. MLPs).
* It would be very valuable to consider actionability constraints. It seems like in practice many of the features of the real-world data sets would be immutable (e.g. “Recession” in the SBA data set).
* For the real-world data, you use significantly fewer features than Upadhyay et al., 2021, why is this?
* In my opinion, none of the test data for which recourse is generated should be used for estimating theta and Sigma. My suggested approach: split the data into train and test, train 100 classifiers only with the train data (e.g. by subsampling 80% of the train data) to estimate theta and Sigma, and then compute recourse on the test data.
* In the recourse setting, one typically does not assume access to the training data. It would be valuable to discuss how could theta and Sigma be estimated (or rather guessed) given no access to the data, and what would be the implications in terms of recourse validity. This could also be tested experimentally by setting theta to be the weights of the classifier for which recourse is generated and over and under estimating Sigma to various degrees.
* For the real-world data, please include the accuracy of the classifier for which recourse is generated.


Detailed comments on the introduction:
* Contrary to what is stated in  Section 1 Paragraph 2, in my opinion “counterfactual explanations” and “recourse actions” should not be used interchangeably, but rather authors should refer only to “recourse actions”, since the problem of recourse validity under data shifts, at least how it is presented in this work (Section 1 Paragraph 5), is not applicable to counterfactual explanations.
* Section 1, Paragraph 2: “If a specific application can provide the negative outcomes with recourse actions, it can improve the user engagement and boost the interpretability at the same time.” Citations needed for these statements.
* Section 1, Paragraph 3: “we must consider age as an immutable feature” It is not a must to consider age immutable, several works consider it as a non-decreasing feature.
* Section 1, Paragraph 4: “Various solutions has have been proposed...”
* Section 1, Paragraph 5, “Data shifts usually induce corresponding shifts in the machine learning models’ parameters” Organizations usually retrain models periodically in repose to data shifts. The data shift itself does not induces changes to the parameters of the model.
* Section 1, Paragraph 5: “If a recourse action fails to generate a favorable outcome in the future, then the recourse action becomes useless” Not necessarily, see Handling change over time via ex post facto in Venkatasubramanian and Alfano, 2020.
* Section 1, Paragraph 5: “and the trust on the machine learning system is lost.” Citation needed.

Comments regarding section 2, 3 and 4:
* In my opinion Algorithm 1 and Theorem 3.4 should not be in the main text, as these are just general well-known concepts from convex optimization, leaving more room for the experiments sections. Authors could just mention convergence rate of $O(1/\sqrt(T))$.
* Similarly, Sections 4.1 and 4.2 could be moved to the appendix, since they are not core to the main contribution of the paper and are not evaluated in the experiments section. This would also give the authors more space for the experiments section and a conclusion.
* Equation 1: min_x
* Equation 2, 7, 8a, 9a: $inf/min_{x \in X}$ or add $x \in X$ as an explicit constrain similarly to Equation 1.
* Equation 9a, please explain why a margin of 0.5 rather than 1 is used.

summary_of_the_review: While the authors present a very clear and thorough exposition of the approach proposed and the approach is technically sound, the experiments presented are very limited and the overall value of the contribution is unclear. In particular, the authors do not compare their proposed method with previous approaches addressing the problem of generating robust recourse actions.
review_title: null

Meta-review: The paper provides a framework for recourse (i.e. counterfactual explanations) that is robust to model shifts. The setup for the proposed method is a min-max optimization problem, where the max is over a neighborhood around the distribution over model parameters. The model parameters are drawn from a mixture of K distributions, so that the neighborhood is specified by the Gelbrich distance on each component. The authors propose a finite-dimensional version of the robustified optimization problem, which can be optimized using projected gradient descent. They evaluate their approach on the German credit dataset, the Small Business Administration dataset, and the Student performance dataset, each of which demonstrates a different type of data distribution shift.

Strengths:

- Most existing work on recourse actions do not consider model change, so the problem addressed by the paper is relatively new
- The experiment results demonstrate the superiority of the proposed method over baselines.

Weaknesses:

- The solution provided is somewhat limited as it relies heavily on the structural properties of the mixture distribution and Gelbrich distance to reformulate the optimization problem.

Most of the reviewers voted initially for rejection. The paper is borderline, tending to rejection after the rebuttal. The authors have also considerably updated the paper with new results after the initial reviews. It seems therefore that the paper may benefit from another round of reviewing and, because of this, I recommend rejection and the authors to use the reviewers' comments to improve the paper before resubmitting to another venue for another round of reviewing.
Decision: Reject