Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the out-of-distribution (OOD) generalization problem. It proposes a Kernel heterogeneity & Invariance Learning method, which jointly learns both the latent heterogeneity exploration and invariant learning in kernel space. This paper provides strong theoretical guarantees for the proposed method. We know the proposed method makes clear improvements over related baselines in OOD scenarios from the empirical results.
main_review: The proposed studied in this paper is an important and promising direction. This paper first tries to jointly learn latent heterogeneity environments and invariance learning modules on representation level. It is important for applying invariance learning on unstructured data (e.g., image).
By introducing Neural Tangent Kernel (NTK) into the proposed algorithm, the authors convert non-linear neural networks into linear regression in NTK. Thus, mutual promotion will be well guaranteed. Moreover, a novel invariant gradient descent method is proposed to learn an invariant classifier. Meanwhile, the environment-specific features are captured by the orthogonal heterogeneity-aware kernel to accelerate the heterogeneity exploration. Thus, the proposed method is both novel and theoretically sound.
The paper conducts extensive experiments on synthetic and real-world data to validate the effectiveness of the proposed method. The proposed outperforms baselines with a large margin on OoD datasets.
Questions:
1.	How to select the hyperparameter lambda in Eq.(8)?
2.	Could the proposed framework integrate the deep model (e.g., deep CNN) as the backbone to deal with a more complex image dataset?

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel optimization framework for Out-Of-Distribution (OOD) generalization problems. OOD generalization problem is one of the most important problems in today's machine learning,  and it is highly non-trivial to investigate it at the optimization framework level.
The proposed framework focuses on the integration of latent heterogeneity discovery and invariant learning on representation level, by introducing neural tangent kernel (NTK). Theoretical analysis is also conducted to verify the mutual promotion of these two modules. Experiments show that the proposed method can significantly outperform the commonly used and latest optimization algorithms.
main_review: This paper is well-written overall with a clear positioning with regard to the related research literature. I am convinced that this problem is relevant and critical to the ICML community. Although the overall logic follows the Heterogeneous Risk Minimization proposed in [15], this paper makes an important step forward by weakening the assumption that invariant and variant features can be decomposed on raw feature level. I like in particular how the proposed framework is theoretically justified. The heterogeneity exploration part is analyzed with a rate-distortion perspective, and the orthogonality property of kernels is guaranteed.  Experiment detail is clear in the paper and appendix on dataset details, baseline methods, and how to choose the hyperparameters for the proposed method. The results are satisfactory overall, and some of them are quite impressive.

The weakness of the proposed method would be the requirement of the hyperparameter to control the cluster number. I would like to see how the number of clusters affects the performance. Also, the authors are encouraged to provide some failure case analysis. For example, whether the mutual promotion between these two modules is sensitive to initialization. 

I believe this paper has merits that outweigh flaws and I would like to vote for acceptance.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present an algorithm for invariance learning under distributional shifts which greatly improves OOD generalization. This is an extremely important problem, and the authors give both good analysis and empirical results. 
main_review: Section 2 is a really present read. Unfortunately, section 1 is almost unreadable to readers without this prior knowledge. I would suggest the authors restructure, such that the "related work" section comes after you have defined what you mean by invariant learning etc. 

At line 165 you lose me. What is big theta_j ? Does delta denote Dirac's delta?

How is it determined how many clusters (or sources) there are?

Can the authors expand why KerHIL^s is of importance?

Do you think the results on Colored MNIST are affected by you using an MLP for your predictor? Would the color have as much influence if a CNN was used?

Overall, I think the results are really good and you test out all the relevant hypotheses with a positive outcome. Very nice!

I did not verify mathematical proofs.
limitations_and_societal_impact: I have big no comments here, but I hope the authors will include more limitations in the final version of the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes KerHIL, a new invariant learning method for out-of-distribution (OOD) generalization. KerHIL both explores the latent heterogeneity and learns the invariance in kernel space by iterating between two modules M_c and M_p, respectively. 
main_review: Despite some typos and references errors (in checklist), the paper is well-written. The authors explained and demonstrated the method clearly, and gave theoretical proofs for the theorems. Experiments are also done in a good manner. However, there are still some points I am curious about. 

First of all, the authors stressed the difficulty brought by multi-source training datasets. However, this issue is only discussed empirically using training dataset with only two sources. It would be better if the authors could extend the training sources to three or more. 

Secondly, it would be more intuitive if the authors provide explanations or even guesses about how predicting training environment labels aid the training process. And how accurate the prediction need to be in order to fully help the training.

For the experiments conducted, experiment details are well listed. However, the parameter settings for different baselines and the proposed KerHIL are not demonstrated. For example, the specific value for the regularization term in Equation (8) is not included. What’s more, it would be good if the authors add ablation studies so as to explore more about the properties of KerHIL. This can help to identify what is the bottleneck (which module) for KerHIL and may provide intuitions about how to extend the method to more complicated datasets and deeper neural networks. 

Overall, I think the paper gives a great direction for future research while the current efforts seem somehow preliminary, and need further improvement in both theoretical and experimental parts.
limitations_and_societal_impact: None.
review_title: null

Meta-review: This paper studies the OOD generalization. The paper proposes kernel heterogeneity and the invariance learning method, which enable to learn the latent heterogeneity and invariant learning in kernel space.  The proposed method and theory provide novel and interesting directions for the OOD problem.  The empirical results are also good.   This work deserves to be presented in NeurIPS.  
Decision: Accept (Poster)