Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper:
* solves the domain generalization problem
* presents several observations that diversity helps mitigate serious correlations
* proposes a sampling method that helps train robust models
* conducts experiment on Rotated MNIST and Rotated Fashion MNIST to show the effectiveness of the proposed algorithm
strength_and_weaknesses: The paper is easy to follow and the proposed algorithm is easy to understand. The idea is not totally new but seems promising. Since the work is likely considered as adopting and improving existing method, I believe the experimental results should be strong. However, I am not fully convinced by the current experiments. Here are my complaints:
* The paper can be more professionally written. For example, the definition of set $C$ is very vague. What is precisely a "good" correlation?
* The experiments are very limited. Why not try some large scale experiments mentioned in DomainBed? It is important to show the proposed algorithm on larger and tougher dataset. I am also curious to see how it works on ColoredMNIST.

clarity,_quality,_novelty_and_reproducibility: * Please see above. The paper can be more professionally and well written. The idea is new and promising but needs more support.
* Source code is provided.
summary_of_the_review: I think the paper is not ready for publication at this point. If the authors can show more evidence and support, I am happy to raise my scores.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors looks into Domain Generalization for a large number of domains. They propose to sample (using two stage procedure) both the domains and the data within each domain, in order to help the model to filter out spurious correlation.  They show a motivational example that the assumption that domains usually have shared causal features and differ in spurious correlation features does not always hold. For the first stage sampling (choosing the domains), they  train an inverse DANN (classify domains while not being able to assign the class labels correctly) to get to extract only domain specific features. Then using these features, they construct domain similarity matrix . For the second stage sampling,  DPP samples diverse domains and diverse data points within the domains (using similarity matrix constructed by ERM)

strength_and_weaknesses: Pros:
- good observations (spurious correlations)
- interesting idea to choose diverse domains and data points
- clarity - pretty easy to follow
Cons:
- overly complicated multi-stage method that requires many expensive computations (train inverse DANN, construct matrix of pairwise domain similarity, construct matrix of data points similarities, train ERM etc) 
- no comparison with no sampling whatsoever - so impossible to say if it was needed in the first place
- only image experiments. Is it because you can't find non image DG datasets with such a large number of domains? is it possible to generate some synthetic data? why not to try WILDS and OGB-MolPCBA that you mention? Or DRUG OOD
clarity,_quality,_novelty_and_reproducibility: Clarity: excellent, easy to follow, well explained
Novelty: unsure. While the sampling methods seems to be working well compared to other sampling, it is not the contribution of the paper since they use the existing DPP. 
Reproducibility: good
summary_of_the_review: Update:
Raising my score a bit, since authors included new experiments. TLDR: without subsampling the performance does improve but on some datasets the drop with sampling is small enough considered time saved


Overall my main complains about this paper are
1) multi step procedure with each step step being expensive (train inverse DANN, construct matrix of domain similariities, construct data points similarities matrix etc). Each step will require its own tuning. Unclear how your method compares in terms of time/computation to any methods you labeled as expensive
2) No comparison with no sampling at all for the table 3. Without this, it is impossible to tell whether the sampling is required at all
3) You state that DANN works with pairwise domains (n^2)??? You have an adversarial head that classifies into the number of domains, how is it n^2??? Also it is misleading to state this for  MMD too - it will indeed work on pairs, but only on pairs from the batch, so it is batch_size^2 max

Minor: DPP related work - repulsive (interactions) is a strange word to use here. Spurious? Non intended?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary
The authors consider the problem of domain generalization (DG). In specific, they are addressing the issues of scalability and objective.
 Scalability: The state-of-the-art DG methods involve computational
 complexity of 𝒪 (n2) corresponding to pairwise domain operations with n
domains. In addition, each domain may contain a large number of data points. This is computationally prohibitive.
Objective: The existing methods entirely focus on excluding domain-side impacts, i.e., spurious correlations, and don’t consider the data-side impacts.
The authors address the above two issues by proposing a two-level sampling method called the Diversity boosted two-level sampling framework (DOMI).
The authors argue that diverse domains of data help exclude spurious correlation (observation 1). In regard to this, they propose to sample diverse domains using invDANN as featurizer and DPP as a sampler in the first level.
The authors observe excluding domain-induced correlations is insufficient for learning a robust model (observation 2). To alleviate this, they propose to sample diverse data batches from the selected domains (domains obtained from first-level sampling) using ERM and DPP in the second level.
The authors validate their algorithm with five backbone DG algorithms on two simulated benchmarks (Rotated MNIST and Rotated Fashion MNIST).
strength_and_weaknesses: Strengths
The problem of DG considered and the issues highlighted such as scalability and objective in this work is very relevant.
The authors provide an important counter-example, i.e., cats and lions to draw attention to the issue of data-side impacts. They also present the comparison of various sampling methods such as random sampling, and different diversity-boosted sampling methods to illustrate the importance of their work and their effect on both object-side spurious correlations and domain-side spurious correlations.
       
The authors give reasonable arguments for observation 1 using concepts such as good, casual, and spurious correlations. In addition, observation 2 is well explained using the structural causal model.
The solution proposed, i.e., diversity-boosted two-level sampling to mitigate the issues highlighted seems intuitively well grounded.
The experimental results show the validity of their algorithm. The proposed method outperforms level 0 and level 1 sampling schemes across 5 backbone DG methods and two simulated benchmark datasets.
The authors provide a decent analysis of the gap between test accuracy and maximal accuracy as well as the impact of the choice of the hyper-
parameter δ (proportion of data points) . 

Drawbacks:

No analysis regarding the choice of the hyper-parameter β (proportion of domains) is presented.
The statement “spurious correlations essentially result from imbalance data” is not entirely true. The spurious correlations result from the existence of anti-causal paths. You could have the same data and different causal models, some of which result in spurious correlations.
It would be better if the authors elaborate on causality-related concepts such as causal correlations and unobserved confounders or give proper references.
There are a couple of typos (is composed by Featurizer, Classifier ..., while the test accuracy of level 0 .... that of level 2 centers, and ...)
The experiments on large-scale datasets are missing. 
clarity,_quality,_novelty_and_reproducibility: Please see the comments before. 
summary_of_the_review: The authors consider the problem of domain generalisation (DG). In specific, they are addressing the issues of scalability and objective.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides a new sampling strategy for domain and datapoint sampling in the domain generalization context. The method encourages sampling diverse domains and diverse datapoints within each domain. To achieve domain diversity, the authors train a class-adversarial neural network (the inverse of the DANN method in [1]) and use a Determinantal Point Process (DPP) with the features produced by this network. To achieve datapoint diversity, the authors train standard ERM on the datapoints sampled randomly from the sampled domains, and also use a DPP.
strength_and_weaknesses: ## Strengths

**S1**: The proposed method makes intuitive sense.

**S2**: Empirical results show improvements compared to the baselines with multiple base domain generalization methods.

## Weaknesses

**W1**: I think some of the presentation regarding the "object spurious correlations" vs "domain spurious correlations" is somewhat imprecise

**W2**: Some of the design decisions are not very clearly motivated and not ablated, in particular the use of invDANN

**W3**: The Empirical evaluation is focusing primarily on Rotated MNIST and Rotated FashionMNIST, which are small-scale synthetic datasets, rather than more standard domain generalization datasets.
clarity,_quality,_novelty_and_reproducibility: ## Clarity (W1)

Overall, the paper is written reasonably well. The one issue I found is that he authors appear to define the _domain_ quite narrowly, as basically the backgrounds of the images. I believe what authors call _object_ could also be a part of domain in the general sense. In the toy example that the authors use, the color of the object (cat or lion) can and should also be a part of the domain label. My understanding of the distinction between the object and domain then is that the authors assume that the domain information is incomplete and doesn't cover all of the relevant spurious features.

Formally, I think I can object to the presentation is Section 4.3, where the authors suggest that Proposition 1 in [2] is wrong. However, this proposition is a formal statement that is proved by [2]. I believe the issue is again that the "object-side" spurious feature $\hat x$ distribution is not the same across the training and testing domains in the toy cat-lion example, the authors assume that for the test domain the distribution of the object color (tan) $\hat x$ conditioned on the class label $y$ will be different compared to the training domains. I believe this setup contradicts the assumptions of Proposition 1 in [2].

## Methodology (W2)

Methodologically, it wasn't clear to me why we need to train a domain featurization which is class independent with invDANN? In particular, why couldn't you just apply ERM on the domain labels? What is the role of class invariance?

Similarly, couldn't you completely remove the first step of the procedure and just do the following?
1. Train ERM on all data with no domain information
2. Use a DPP with the ERM featurization to produce a diverse sample of datapoints (and the corresponding domains)

Is there a reason for the added complexity of the two-step procedure?

Further, the authors ablate the need for the second stage (adding diversity within sampled domains) but not the first stage of the procedure. What is the performance if you sample the domains randomly, but then use the ERM-based DPP to sample diverse datapoints?

## Experiments (W3)

The experiments show promising results, but are quite limited currently.

The authors only report performance on the Rotated MNIST and Rotated FashionMNIST datasets. This choice is unclear to me:
- Both datasets are small-scale. Further, these are not the most standard benchmarks in domain generalization to the best of my knowledge.
- The method is motivated by having a large number of domains and a large number of datapoints in each domain. While the number of domains is fairly large (65), it is unclear if the quadratic cost of using all the domains would already be an issue at this scale.
- It is unclear what the second stage of the procedure is supposed to be doing in these datasets. What is the spurious correlation that you are trying to address within each domain? Or maybe I am missing the point of the second stage here.

Generally, I think these datasets would be reasonable _if_ they were a part of a larger evaluation, which also included more realistic datasets where the motivation for the method is more clear. In fact, in the Appendix the authors also consider the iWildCam dataset, although they mention a few issues with the results, e.g. large computational overhead compared to the baselines and only using the FISH base domain generalization method, which performs relatively poorly on this dataset. At the same time, it does seem like the authors achieve some improvement on this dataset with FISH.
summary_of_the_review: This is an interesting paper, with promising results. Above, I highlighted a few questions and concerns about the presentation, methodology and experiments. The limited empirical evaluation is the main issue in my opinion.

## References

[1] [_Domain-Adversarial Training of Neural Networks_](https://arxiv.org/abs/1505.07818);
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky

[2][_Domain Generalization using Causal Matching_](https://arxiv.org/abs/2006.07500);
Divyat Mahajan, Shruti Tople, Amit Sharma
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is well-written and easy to follow. The observation is interesting and inspiring. The major concern is that the proposed method is a bit complicated and involves extra computing overhead. The performance gain achieved by the method is relatively small compared to the computing cost. In addition, the experiments are mainly performed on small datasets. Although the authors claim that the necessity of using sample-based methods for domain generalization is extremely high on large-scale datasets with multiple domains, no experiments are shown on these large-scale datasets, e.g., WILDS and DRUG OOD.
justification_for_why_not_higher_score: The contribution of the work is below the acceptance bar.
Decision: Reject