Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper shows that generalization error on the target task is non-monotonic in the number of OOD samples. The authors empirically evaluate this phenomenon on MNIST, CIFAR-10, PACS, and DomainNet. Finally, the authors show how to ensure that the generalization error does decrease monotonically with the number of OOD samples. 
strength_and_weaknesses: Strengths:
- The paper shows and brings insight into the phenomenon that generalization error doesn’t decrease monotonically with the number of OOD samples. 
- The writing is quite clear overall, and the paper is well-structured.
Weaknesses:
- Not clear why intuitively generalization error should decrease monotonically with the number of OOD samples. In particular, if the out-of-distribution task is far from the target task, it’s not clear why have those samples would improve generalization on the target task. Additionally, if there are a lot more OOD samples than target task samples, increasing the number of OOD samples intuitively may not improve the generalization on the target task. Can the authors provide a cite for the claim that the monotonicity is intuitive? If not, would suggest changing the framing to deemphasize how intuitive this is. 
- In order to mitigate the non-monotonic nature of the generalization error, the paper requires knowing which samples in the dataset are OOD. In addition, the proposed method is just using a weighed objective, which has been proposed before.
Small nits:
- Not sure if there’s a point to stating Theorem 1.
- Italicized sentence at the end of page 7 seems important in making a point but doesn’t make sense. Can the authors clarify?
clarity,_quality,_novelty_and_reproducibility: Clarity: pretty good.
Quality: See weaknesses above.
Novelty: Paper has novel insights although the proposed method is not novel.
Reproducibility: should be easily reproduced.
summary_of_the_review: Please see weaknesses. I think the paper is interesting but not quite ready for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how OOD samples within datasets impact the generalization error on the desired task and observe that the generalization error of the task can be a non-monotonic function of the number of OOD samples. This work also develops an algorithmic procedure to train on the target task that is resilient to OOD data.
strength_and_weaknesses: Strengths:

S1. The main observation that generalization error on the target task is non-monotonic in the number of OOD samples is counter-intuitive and interesting.

S2. The structure of this paper is well-written and easy to follow. The motivation is clear.

S3. The extensive experimental results on numerous datasets look good.

Weaknesses:

W1: It is unclear how to select the threshold of OOD samples, after which the generation error can deteriorate.

W2: How to quantify the OOD samples that can improve generalization.

W3: It would be interesting if the authors can conduct experiments on large vision benchmarks such as ImageNet-O, and ImageNet-R.




clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is clear and easy to follow.
summary_of_the_review: This paper presents interesting observations that are useful for applications. The extensive experiments and ablation studies are good but there are a few weak issues that need to be clarified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper begins by providing a toy example on how OOD samples could reduce generalization error. Then they show that this phenomena can also occur in some real dataset when the training data is extremely small. Finally, under the assumption of knowing the exact target and OOD distribution, they can come up with an optimal weighting to reduce the effect of OOD samples.
strength_and_weaknesses: Strength:
- The writing is clear, the analysis seems rigorous.

Weakness: 
- This phenomenon only seems to appear in extremely low data regime, which makes this phenomenon not really counter-intuitive. In a low data regime when the feature space is large, there are generally not much guarantees, which makes the existence of Theorem 1 not that surprising. Being only able to operate in such low data regime also limits the scope of the paper.
- The choice of n, m and m/n seems quite arbitrary in each figure.
  - Also I think it should be more reasonable to keep m+n fixed instead of keeping n fixed and increase m. In many of their figures, m/n > 1, which is not common to have more OOD samples than in-distribution samples.
- Only generalization error is shown. Without also looking into the training/testing accuracy, it is hard to grasp what exactly is happening. One can have a constant classifier, it would always have a generalization error close to 0. In other words, generalization error itself cannot reflect how well the model is learning.
- The developed algorithmic procedure for reducing the affect of OOD samples is also not very useful as normally we don't even have access to P_o and P_t.
clarity,_quality,_novelty_and_reproducibility: There are no code provided, I cannot say for certain of the reproducibility.

The paper is written well.
summary_of_the_review: The paper does provide interesting examples on how OOD examples might effect the model. However, none of the experiment setup/assumption operates in a condition that commonly seem tasks operates in, which limits the scope of the paper.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: I informed the area chair that I cannot review this paper.
strength_and_weaknesses: I informed the area chair that I cannot review this paper.
clarity,_quality,_novelty_and_reproducibility: I informed the area chair that I cannot review this paper.
summary_of_the_review: I informed the area chair that I cannot review this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies distribution shifts and shows that the generalization performance can be a non-monotonic function of the number of OOD samples.  
The reviews appreciated the non-inutuitive insight the paper brings about the non-monotonic nature of generalization performance w.r.t. the number of OOD samples and that the experimental results were strong to support this conclusion.
However, the reviewers also had concerns about how to appropriately choose various hyperparameters (i.e. regarding how to identify such OOD examples) that limit the practical value of the findings. This gap is the key reason for rejection.
justification_for_why_not_higher_score: The results depend on quantities that are not available in practice, thus causing a serious gap to make the method of practical relevance.
Decision: Reject