Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes a novel framework of invariance learning that enables invariant predictor to be estimated in single domain data by assuming additional data from multiple domains for a higher level of classification task, and proposes two methods of cross-validation for selecting hyperparameters without accessing any samples from unseen target domains. In the given experiments setting and with 2 datasets, the proposed framework shows improvment over other existing methods. In addition, the paper carries out a detailed theoretical analysis.
strengths_and_weaknesses: Strengths:

* Invariance learning in a single domain data is a novel idea that can save cost of annotation and training.
* The paper proposes two methods that can select suitable hyperparameters without accessing the unseen domain data, and gives a sufficient theoretical analysis.
* The theoretical proof of the paper is solid.
* The paper is well-written in general.

Weaknesses:

* Using a higher level of label hierarchy will lose a certain degree of fine-grainedness.
* In the experimental part, only two datasets are used, and more experiments need to be supplemented.
* For the experiments on ImageNet, the target task only predicts three labels, which is not convincing.

questions: 
* How to ensure that data in different environments are in different domains in higher level of classification task？
* Is it possible to validate the method on another out of domain dataset?
* Does the method just do a normal invairant learning by reducing the fine-grainedness of the labels?

limitations: The author discusses a little in the limitation section, and lacks discussion on scalability, efficiency, and future research directions.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces an interesting extension of IRM, which leverages the higher-level classes to learn the invariant classifier, thus it releases the data annotation efforts in many cases when the previous methods require the detailed annotation of classes. 
strengths_and_weaknesses:   - Strengths
     - the paper aims to release the requirements of find-grained class annotations, which is an important aim toward real-world usage of the algorithm. 
     - the paper involves reasonable sound and comprehensive statistical efforts, which might be valuable to the statistics community. 
  - Weakness
     - the link between the statistical work and the real-world implications seems missing (see below questions for details)
     - the empirical evidence of the strength of the method is not convincing enough. 
questions: This is an interesting paper with sound statistical effort contributing to expanding the areas of IRM, also with a realistic aim to remove fine-grained label dependence. The statistical efforts are sufficient, but my main questions are about the real-world implications of these statistical efforts. 

  - multiple theorems rely on several additional assumptions, it will be much more convincing if the real-world implications of these assumptions are discussed and how easy/hard these assumptions will be met in the applications. 
     - e.g., premise (A) at line 177 (although there is an explanation on line 182 but the explanation does not link to the reality well enough) seems to suggest that "there must be enough domains so that the desired function is the only one that is shared with any two domains" what are the corresponds interpretations for premises (I)(ii) for theorem 4 and 5? 
  - related to Theorem 3, the proof in Appendix is presented in a non-conventional way, and line 482 seems to introduce additional assumptions that are not presented in the theorem statement in the main manuscript. 
  - about the proofs of other theorems, there seem no clean proofs but a continued introductions of additional theorems. please clear all the numbering issues if there are. 
  - the empirical work is not very convincing either, there is not one selection method that can stand out from others, thus, in reality, we only know there exists a better method than previous ones, but not sure which one exactly to deploy to the real-world. 
      - is ACC on e1 not considered important for evaluation, if so, then CV-II methods are probably considered good enough? 
  - the imagenet results are a little misleading, while the authors keep using "ImageNet" to refer to the dataset, what's used in fact is a slice of the whole imageNet. 
      - why to use a creation of OOD imagenet where there are multiple OOD imageNet already? (imageNet-A, imageNet-sketch)
  - too many baseline methods along the debiasing literature (OOD robustness) are missing, only to name a couple [1,2]

[1] Learning De-biased Representations with Biased Representations

[2] Toward learning human-aligned cross-domain robust models by countering misaligned features
limitations: there is a devoted paragraph regarding the topic.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes an invariance learning framework by using a training dataset of a single domain and additional data with coarser annotations. Two cross-validation methods are further proposed for the hyperparameter selection. The authors provide theoretical analyses of the proposed methods, and experiments on several datasets show the effectiveness of the methods.
strengths_and_weaknesses: Strength:

1. The idea of adopting coarser labels to learn the invariance among different domains is interesting. This is a good exploration on how to reduce the annotation costs in invariance learning. 

2. This work provides sound proof of why the proposed method will work.


Weakness:

1. The method proposes to adopt coarser labels to reduce the annotation costs. However, the authors may give a more strict explanation on how "coarse" the higher label hierarchy is. The examples provided in this work seem too simple, which makes me confused about how applicable this method can be to other more practical datasets or situations.

2. The authors should conduct experiments on benchmark datasets as those in previous works like CIFAR-10. Now the settings of the datasets adopted in this work seem to be simple, which makes it not easy to understand the effectiveness of the proposed method. Also, it will help if the authors can compare the proposed method with previous works including but not limited to [1,2].

3. The writing of this paper can be further improved. The authors should be more careful and proofread the paper on the typos and grammar errors, especially in the Introduction section. And some notations should be explained to help the readers to better follow the work, e.g. $e^*$ in Line 119.

[1]. Gregory Benton, et al. Learning Invariances in Neural Networks

[2]. Alexander Immer, et al. Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations
questions: Questions:

1. The settings of the datasets adopted in this work seem to be simple. Even though the authors claim that the ImageNet dataset is more practical, it only contains 3 classes with 2 high-level classes. It is better to see the performance on practical datasets with more classes, which can help to show how effective the proposed method is.

2. In Line 154, what does the $n$ mean? Why will there be many different classes even for the addional data? If we want to reduce the annotation costs, the number of coarser labels should be controlled to a relatively small number. So what will be a typical value for the $n$ here to perform Method I?
limitations: The authors have discussed and addressed some potential limitations in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the invariant learning problem for out-of-distribution generalization. The main efforts focus on the new training framework for saving the expensive annotations and hyperparameter selection strategies for searching optimal hyperparameters under appropriate settings. Both the training framework and selection strategies are supported by the theoretical results, where the main results show the proposed method ensure better generalization and optimal hyperparameter under some assumption. Overall, this paper is well-written and generally easy to follow while there are still some concerns regarding clarity, assumption, and rigor.
strengths_and_weaknesses: **Strengths:**
+ This paper takes a further step towards a more applicable out-of-distribution generalization model with appropriate and reliable hyperparameter selection. Thus, the technical novelty is significant.
+ The methodology on meta(coarser)-label and theoretical results do provide a guaranteed way to deal with the dilemma of annotation cost on multiple source domains.
+ The overall organization of this paper is good, and the claims are self-contained.

**Weaknesses:**

  *Major*
  - Some assumptions seem to be strong (Q1).
  - The essential cost (complexity) of annotations seems to be unchanged. (Q2)
  *Minor*
  - Clarity on definition and notation could be improved.
  - The term ‘spurious correlations’ may be misleading.

questions: 1. In lines 129 and 179, an assumption ‘run all the probability density functions’ is made on posterior distribution (i.e., the concept/predictor). As far as I understand this assumption, it also implicitly admits that the training distributions cover all possible distributions/samples. which seems to conflict with the setting of out-of-distribution generalization. More explicitly, if the training distributions cover all possible cases, the target distribution can be a standard test distribution without shift. More justifications for this assumption are appreciated.

2. In the proposed framework, the complexity of manual annotations seems to be unchanged since the meta(coarser)-labels on all samples are still necessary. Specifically, assuming that $|\mathcal{Y}|=k_1$ and $|\mathcal{Z}|=k_2$, it is clear that $k_2\leq k_1$. If the complexity of manually annotations for target task is $\mathcal{O} (k_1 n)$, then the complexity of annotations for propose framework is $\mathcal{O} (k_2 n)$. Though $ k_2 n \leq k_1 n$, the complexity of meta(coarser)-labels is still linear w.r.t. the sample size. I suggest more rigorous discussions on the cost of label hierarchy framework since one of the most important contributions is the reduction of labeling cost.

3. Clarity of definition and notation.
-  3.1) The claim ‘$P$ does not depend on $e$’ in definitions of ‘set of invariances’ could be more rigorous. In my understanding, it can be defined as conditional independence on $e$ by introducing $e$ as a variable of domain, or just defined as ‘$P(Y^{e_i}) = P(Y^{e_j})$ for any $e_i, e_j \in \mathcal{E}$’.
-  3.2) The notation ‘(ii)’ is used without definition. Does it imply (ii) in Thm.2 (i.e., (ii) in line 169)?
-  3.3) It is not straightforward to understand the claim in line 172 (a paragraph). Specifically, the claim ‘composed of all possible distributions of $X_2$’ is stated while the equivalent properties are given w.r.t. the invariant component $X_1$ (i.e., (i) and (ii)). More explanations will be appreciated.
-  3.4) There seems to be no definition for $\Lambda$ in line 197. Besides, what is the purpose of $\Lambda$ in algorithm?
-  3.5) The notation $-$ for complement operation could be confused with other operations on set or analysis. I suggest using $A\backslash B$ instead of $A-B$.

4. The term ‘spurious correlations’ seems to be misleading while the ‘biased correlations’ could be more appropriate. Though the model learns on source domain does may be biased on the target domain, the embedding is still useful on source domain and generally characterize most of the task information. Besides, according to my understanding of the ‘spurious correlations’ with the example provided in line 24, this problem still cannot be guaranteed to be addressed when target domain or multiple domains are available. Thus, revision or justification on this term will be appreciated.

limitations: Overall, this paper is well-written and the merits are significant. Thus, if most of the concerns (i.e., **Question**) are addressed, it can be a good paper towards a more applicable out-of-distribution generalization model.
review_title: null

Meta-review: metareview: This paper targets the invariant learning problem for out-of-distribution generalization. A new framework is proposed, which enables invariant predictors to be learned in single domain data with the help of additional data from multiple domains. Both theoretical analysis and empirical evaluations are proposed to verify the effectiveness of the framework. 

All reviewers are positive about this paper. They highlight (a) the idea is interesting; (b) the theoretical analysis is detailed to support the paper's claims, which can contribute to the research community; (c) the writing and organization are overall great. Major concerns raised in the review process are also addressed. The meta-reviewer is happy to recommend an acceptance and suggests the author carefully merge the rebuttals in the final version.
confidence: Certain
Decision: Accept