Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to generate post-hoc recourses which remain valid under model shift, called DiRRAc. The model parameters are considered as a random vector modeled according to a mixture distribution (whose parameters are fit by taking models trained on trainset samples). Future model shifts are modeled by considering all distributions close to the initial distribution, called the ambiguity set. The paper proposes a minimax problem where inner max is over the ambiguity set and min is over possible counterfactuals. By choosing Gelbrich distance, the optimization problem is shown to be solvable, and the paper proposes a Projected GD algorithm. Experiments are conducted on 3 real-world datasets, and they compare against 4 baselines. 
strength_and_weaknesses: + The paper deals with a practical problem; generating robust recourses is necessary for models which are to be deployed in the real world
+ Experimental results are encouraging, as DiRRAc achieves best post-shift validity for all datasets shown, and compares favorably against ROAR [1] in cost of recourses
+ Unlike ROAR [1], DiRRAc doesn’t have to specify the set of possible model parameter perturbations (\Delta), but it has its own parameters to tune/specify. I am not sure if ROAR could have achieved similar M2 validity results with different \Delta, but figure 3 seems to show DiRRAc consistently achieves lower cost recourses


- DiRRAc is not the best performing method (in M2 validity) for non linear models, which are usually the ones deployed, since it relies on the local linear approximation 
- How can the initial distribution parameters be computed for classifiers which already exist? 
- How is \rho_k determined? Ideally, we want to have a larger ball if we expect the model to change significantly. Is there a guideline for this?
- How costly is the projection step? A study of time taken to generate recourses vs ROAR is necessary to determine practicality

Other points
* cite the source for Definition 3.1
* possible typo on page 5, last paragraph, should it be f_k(x) <1 instead of f_k(x)\leq 0 in the reformulation?
* Citations are missing for the 3 datasets used

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and technical contributions are described thoroughly. Code is provided for reproducibility. Some implementation details are not described.
summary_of_the_review: The paper applies distributionally robust optimization techniques to generate robust recourses in a technically sound way. Their way of quantifying model uncertainty, by using ambiguity sets, gets rid of having to specify an explicit set of possible future model parameters as in ROAR. Experimental results are convincing for linear models. DiRRAc may see limited success for highly non-linear models which can change in complicated ways.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed the Distributionally Robust Recourse Action (DiRRAc) framework, which generates a recourse action that has a high probability of being valid under a mixture of model shifts. The authors formulated the robustified recourse setup as a min-max optimization problem, where the max problem is specified by Gelbrich distance over an ambiguity set around the distribution of model parameters.  Numerical experiments with both synthetic and three real-world datasets demonstrate the benefits of our proposed framework over state-of-the-art recourse methods.

strength_and_weaknesses: Strength
1.  proposed a framework of Distributionally Robust Recourse Action (DiRRAc) for designing a recourse action that is robust to mixture shifts of the model parameters
2. Numerical experiments with both synthetic and three real-world datasets demonstrate the benefits 

Weakness
1. it requires LIME for local approximation
2. Lack of comparison with new baselines


clarity,_quality,_novelty_and_reproducibility: see main review
summary_of_the_review: 1 . The convergence result is just a direct application of existing result, which can hardly be counted as a contribution
2 . The analysis is based on linear case which is not really practical, which for nonlinear classifiers, it requires to use LIME to approximate, which could lead to degraded performances. .
3. The authors may want to comment on/compare with the following recent work:
Nguyen, Tuan-Duy H., et al. "Robust Bayesian Recourse." Uncertainty in Artificial Intelligence. PMLR, 2022.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work formulates and targets the setting of distributional robust recourse action to deal with model perturbation and changes in recourse action problems. It proposes the formulation of distributional robust recourse action as a min-max problem with a Gelbrich distance as the distance metric of the uncertainty set. In addition, a corresponding projected gradient descent algorithm is proposed to solve it. Experiments have been conducted to compare with baselines using the metric cost and validity.
strength_and_weaknesses: Strength:
1. This work seems to be the first work that considers the setting of distributional robust recourse action to deal with model perturbation and changes when generating recourse actions.
2. By formulating the problem using Gelbrich distance, it proposes the reformulation of this problem in general and Gaussian framework and also the corresponding algorithms.

Weaknesses:
1. If I didn't understand wrongly, the baseline AR and Wachter has better ( smaller ) $\ell_1$ cost than the proposed DiRRAc or Gaussian DiRRAc? And for M1 validity and M2 validity, it works the same as ROAR or similar to other baselines? So the performance of the proposed methods doesn't show very convincing results with comparisons to baselines.

clarity,_quality,_novelty_and_reproducibility: -The clarity and quality of the presentation and writing is very clear and easy to follow.

-The problem setting targeted in this work is interesting and somewhat novel. The proposed reformulation of the problem and methods are also somewhat novel.

-The experimental details have been included in the presentation so that reproducibility seems also possible.
summary_of_the_review: In summary, this is a work targeting a new setting called distributional robust recourse action to deal with model perturbation and changes, inspired by a distributionally robust optimization framework. It not only provides the formulation of this new problem but also proposes new methods DiRRAc or Gaussian-DiRRAc for it. The only concern is that the experimental results does not show a large gap between the proposed method and other baselines. So I suggest a borderline acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the Distributionally Robust Recourse Action (DiRRAc) framework that generates an optimal recourse action, which under a mixture of model shifts, enjoys the highest probability of being valid. A projected gradient descent algorithm is designed to solve the model and find an optimal recourse action.
strength_and_weaknesses: Strength:
1. The framework is well motivated and the distributionally robust optimization approach is a suitable tool to address the issues raised.
2. The ambiguity set to capture model shifts is effective, leading the framework to a formulation that can be solved with the designed algorithm.
3. The extension to hedge against the misspecification of the mixture weights also supports that the framework is well-designed.
4. The numerical experiments are convincing.

Weakness:
My only comment is that some existing works should be cited:
1. "Distributionally robust multi-item newsvendor problems with multimodal demand distributions" addresses a similar issue of uncertain mixture weights.
2. "Sharing the value-at-risk under distributional ambiguity" also values the probability under distributional ambiguity around a reference Gaussian distribution.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and of high quality. The framework is novel and well-designed. I am not doubted that the numerical experiments can not be reproduced.
summary_of_the_review: I would recommend an acceptance of this paper: the framework is well-motivated and well-designed. The treatment is clear yet noval.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors consider the recourse action problem with the uncertainty of the model, and provide a solution based on the distributional robust optimization framework. The authors then extend the method for nonlinear models with local interpretable linear model. The method is justified empirically on several datasets. 

All the reviewers agree the paper is novel for recourse action problem and the empirical performance is promising. 

It will be better if the authors can discuss the mixture of Gaussian distribution assumption for the parameters, and the practical setup in the optimization. 
justification_for_why_not_higher_score: 
The method relies on the distribution assumption of the parameters as mixture of Gaussian, which might be limited to practice. This should be justified and discussed in details. 
justification_for_why_not_lower_score: All the reviewers agree the paper is novel for recourse action problem and the empirical performance is promising.
Decision: Accept: poster