Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a RecourseNet, which learns to instance-dependently recourse to environments so that the learner gets better samples at training time and the predictor is well informed which instance to recourse with which environments. Towards this goal, they propose a novel learning objective to optimize trigger, and the parameters of predictor and recourse recommender. Due to its non-differentiability, they resort to simple heuristics to approximate the learning objective and develop an efficient learning framework. The experimental results demonstrate the efficacy of RecourseNet over simple baselines, achieving superior recourse accuracy versus fraction of recourse. The results also show that instance-wise recourse is important as well.
strengths_and_weaknesses: # Strengths
- Paper is generally well written
- The motivation is very clear, and the problem setup is very interesting, novel, and practical, which I think of as the biggest contribution of this paper. The assumption that the underlying true process $Z$ is unavailable such that we need to instance-wisely recommend alternative $\beta$ sounds very reasonable.
- The learning objective (1) and (2) seems novel and looks interesting.
- The experimental results demonstrate that the proposed RecourseNet clearly outperforms the baselines (although no confidence intervals are provided and those baselines seem rather weak)
- The authors made a good effort in developing novel datasets, which can be used by other papers.

# Weaknesses

- Doubt about the original learning objective (1) and (2). I understand what (1) and (2) are meant to do, but in my understanding they can have a trivial solution: for instance, with sufficient network capacity and enough training time, the learner may simply find a trivial solution $\pi(x,\beta)=0 \forall (x,\beta)$ and simply maximize $\log f_\theta (y|x)$ as much as it can. I understand that during training and with limited computational capacity of $f_\theta$, sometimes $\pi(\cdot,\cdot)$ will become $1$ because "bad" $x$'s will have high prediction errors, but it is only in terms of training process, not encoded by the learning objective itself. In other words, the learning objective itself cannot distinguish between bad local optima (trivial solutions) and good local optima (meaningful solutions) if those minima can achieve the same loss value. So I guess we need more discussion about whether (1) and (2) are well-posed in the first place. 

- Crude approximation of the learning objective. I understand the difficulty of dealing with the original learning objective (1) and (2) and crude approximations are inevitable. Then, what we need to empirically show is that the proposed approximations are indeed maximizing the original objective (1) and (2) in a stable manner. I would appreciate if the authors could show this in the rebuttal phase.

- Typos spread in the text, especially in the mathematical notations. (See questions and typos below)

- Limited baselines. I understand that there may not exist suitable baselines to compare because the problem setup is novel, but I felt they are too few. As an example, in Figure 2 the authors could have added more baselines that train $f_\theta$ selectively, in order to truly show the validity of the "steepness" of the red curves. (i.e. the different steepnesses may come from the superiority of $f_\theta$?) Honestly I'm not an expert in this area, so I will have discussions about the possible baselines with other reviewers.

- In Figure 4, it would be more informative if the authors could actually show the different $\beta$ realized for each class (either qualitatively or quantitatively), instead of the actual recourse accuracies.

- The experimental results do not provide confidence intervals, so the authors cannot argue statistical significancy.

# Summary
In summary, I found that the problem setup is very meaningful, interesting, novel, and practical. But the overall quality of the paper is a bit low and more experiments seem to be needed. If the authors can address my concerns, I can increase my score.
questions: # Questions
- How did you choose the hyperparameters $b$, $\Delta$ and $\delta$?
- In Figure 4, how did you select the 5 classes?
- In L154, I think $F(\theta^{k+1}(R\cup \{(i,j)\})) \geq F(\theta^{k}(R\cup \{(i,j)\}))$ is wrong (also, $F$ should take two arguments as an input) because we cannot guarantee the predictor performance at the next iteration for many reasons (e.g. non-convexity). What we can guarantee is $F(\theta^k, R\cup \{(i,j)\}) \geq F(\theta^k, R)$. Am I understanding correctly?

# Typo
- In Eq.(2), $\pi(x_{ij})$ should be $\pi(x_{ij},\beta_{ij})$
- L124, 125: $(1-\pi(\cdot,\cdot)f_\theta(\cdot,\cdot))$ should be $(1-\pi(\cdot,\cdot) \log f_\theta(\cdot,\cdot))$
- In Eq.(6), in the first line there should be $\log$ before $f_\theta$ (in Appendix A, the authors say that the second line of Eq.(6) should be argmax, but it sounds weird because argmax returns $\beta$ which is not something that we maximize)
- I wonder whether the second line of Eq.(8) is correct? 

limitations: The authors did not particularly mention the limitations of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose using a recourse mechanism that allows a model to recommend to its human users how they could improve the predictions of the model by capturing another sample of the current instance with somewhat different view hyper-parameters (less zoom, more brightness etc). The model is made of three stages, one for prediction of a target class, one for predicting whether resampling could be useful, and another recommending how the resampling should be made. 

The method requires that the model has access to different views of various instances and a corresponding real value that describes somewhat the transform variation over those views. 

The authors state that this could be useful in settings such as the medical setting. 

They then evaluate their model on a variety of synthetic and non synthetic datasets and show that their method can improve performance successfully on those.
strengths_and_weaknesses: One of my main concerns is around the idea of recourse/resampling. The authors state that this could be useful in medical settings. But is it really? It can be expensive to get additional MRI/CT etc images for the patient and the hospital, and, more than that, for the model to work it would need to be trained on lots of variations of the same instances with certain imaging parameters annotated. Furthermore, it would need to somehow be able to generalize on different humans and hospital machines and how they interpret the recommended changes for the resampling. It simply seems too expensive for what appears to be highly uncertain and unverifiable (at in-the-wild application) predictive improvements. 

Originality:

The work attempts to do something novel and potentially useful. I consider it's novelty at acceptable levels. 

Quality:
The paper is written well, and presented well, however the experimental quality is lacking as the model is only evaluated on datasets that are very precise in how a particular instance is viewed in terms of the view parameters given. In medical settings, for example, this would not be very possible with current datasets. The authors also do not evaluate on a more real world dataset of the types that they mentioned their method could be useful in (such as medical settings).

The authors also do not compare with randomly recoursed/resampled instances to evaluate the effectiveness of their recommender model.

Clarity:

The paper is overall quite clear, and the methods clearly described. 

The paper is also lacking optimizer settings, data augmentations and regularization details for the models that were trained.

Significance:

Assuming the work could be reframed in a way that it requires a more realistic data source with more noisy imaging parameters, the idea of recourse could be a useful one. As it currently stands however, it does not seem to offer real world significance, but does offer fair research novelty and significance
questions: 1. The authors state that this could be useful in medical settings. But is it really? It can be expensive to get additional MRI/CT etc images for the patient and the hospital, and, more than that, for the model to work it would need to be trained on lots of variations of the same instances with certain imaging parameters annotated. Furthermore, it would need to somehow be able to generalize on different humans and hospital machines and how they interpret the recommended changes for the resampling. It simply seems too expensive for what appears to be highly uncertain and unverifiable (at in-the-wild application) predictive improvements. 
2. Have you evaluated the model on randomly recoursed instances? That could help evaluate whether a. the recourse predictor is useful and b. whether the recourse recommender makes useful recommendations.
3. Since samples are being resampled, have you considered comparing your method with existing few-shot learning models? ProtoNets and Matching Nets could be an initial baseline to compare to. As sample size increases, the predictions of those models improves as well. 
limitations: The technical limitations I stated earlier stand, but in terms of negative societal impact, I don't see any obvious problems (other than potentially increased hospital spending from patients and the government alike if implemented poorly).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a mechanism, termed RecourseNet,  to identify and modify input instances so as to achieve a better classification outcome from a classifier. These modifications are performed in the latent space of environment-settings.
RecourseNet consists of three components – a classifier, a recourse-trigger (which selects instances that require modification), and a recourse-recommender (that makes modifications).  A mechanism for training the three components is detailed. Conditions under which the recourse-recommendation leads to improved accuracy are studied. Experiments are performed on ShapeNet, Speech Commands dataset, and a synthetic dataset.

strengths_and_weaknesses: STRENGTHS
==========
+ Offering recourse in the form of alternative environments is interesting, especially for images.
+ The proposed optimization problem is original, and various tricks are proposed to solve it.
+ Various ablations of RecourseNet are studied and the design choices are justified through experimental results.

WEAKNESSES
===========
- It is not clear what the problem RecourseNet actually solves. Is it trying to perform classification when training-data is noisy? 
- L21: During training, aren’t such adverse examples (if labeled correctly) important for generalization? It would be concerning if all training instances were from a fixed set of views/environments. Finding alternative environments makes sense during test time, because the classifier might not have seen such examples during training.  
- Misuse of the concept of offering recourse: In various places (L5,7,37) the problem is defined as “finding an alternative environment setting so that the classifier is more likely to get a correct prediction”. In traditional recourse, a fixed  classifier is assumed, and actionable changes are suggested to a user (input instance) that has been negatively classified. I am not convinced that performing pre-classification data modifications can be termed as recourse. The usage of the concept of recourse needs justification. 
- Lack of a suitable baseline: In experiments, the questions studied are for different settings of the proposed algorithm. For instance, Section 4.1 is an ablation study that measures how important their iterative greedy proposal is for solving eq (5). It would be interesting to see how RecourseNet performs when compared to other methods that solve the same problem.  (Because the underlying problem it solves is not clear to me, it becomes difficult for me to recommend baselines at this time, but I suppose baselines like Invariant Risk Minimization or other methods that attempt to find domain-invariant representations could be baselines.)

questions: * L21: During training, aren’t such adverse examples (if labeled correctly) important for generalization? It would be concerning if all training instances were from a fixed set of views/environments
* It is not clear to me what is meant by an ‘upstream’ classifier? In Fig 1, the classifier seems to be downstream. 
* L110: D is used to index the set of objects. Maybe better to use different notation for training dataset.
* In Eq 1 (training objective), what happens if training data does not contain some objects? How is the summation over i handled?
* L126 typo: is it analytical form of Z?
* In Algorithm 1, does the TRAIN() function solve for \theta using Eq 5? If that is the case, how is Rec constructed? How is x_ir computed?
* L157: It is not clear why this is an approximation. 
* L169: What is the significance of computing \pi? Doesn’t Algorithm 1 already compute \pi? 

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **Summary**: The paper tries to improve the prediction accuracy through a different perspective: recourse on instance environment. Firstly, it points out the fact that a classifier’s performance is also dependent on the data quality, that is, a good classifier is also likely to misclassify an image taken in very bad environment setting. To address this problem, the author proposes the RECOURSENET to first identify whether the data is good for classification or needed to taken from another environment setting, such as another angle. If the image is not qualified for a good classification, the recoursenet will recommend the user an appropriate environment setting. Then the recoursenet will trains a classifier to classify the image of good quality or the images generated by the user. 

**Contribution**: This paper proposes a novel three-level framework for the recourse mechanism without human involvement in the training. 
strengths_and_weaknesses: **Strengths**: 
- The paper provides a novel perspective for enhancing prediction accuracy. Instead of focusing on the classification model for improving robustness, it focuses on the input data quality and introduces a recourse mechanism for improving the data quality and consequently improving the prediction accuracy. This is promising in medical applications. For example, the algorithm can be deployed on some online diagnosis platform and provides hints/tips to the user about how to take a clear image that can make the diagnosis more accurate.
- The proposed method is intuitive and relatively clearly explained.
- The experiment is thorough in terms of the ablation study, namely how every component is beneficial.

**Weaknesses**:
- The experiments are limited to small datasets. 
- There is some inconsistency in terms of notation and writing. For example, the environment setting appears as $B$ or $\mathcal{B}$. And in line 88: it is inconsistent with the algorithm when saying that humans do not participate in (the) prediction task but they only generate new instances under the recommended environments. 
- How these two propositions benefit seems to be unclear to me. 
questions: - It would be better if the author can add a more detailed discussion about the theoretical analysis part, e.g., how does these two propositions benefit the design of the mechanism. It seems the assumption of $||\beta_{i,j}^{\prime} - \beta|| \le \epsilon$ is quite strong.
limitations: The paper's limitations are not discussed by the author.
review_title: null

Meta-review: metareview: The paper proposes a recourse approach that recommends how to improve performance on instances by modifying their environment. The paper is well motivated and provides a novel approach that is empirically demonstrated to be useful, though the empirical evaluation is limited. Reviewers agree that this paper addresses an important question that has more recently started to get attention, and that the contribution is novel, creative and significant. The quality of the write up could be improved, and I encourage the authors to do so for the camera ready version. 
confidence: Certain
Decision: Accept