Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes Mix and Reason, a new domain generalization approach consisting of two components. The first component, category-aware data mixing, basically performs data augmentation by replacing the background of an image with a random patch of another image, aided by binary masks produced by Grad-CAM. The second component, adaptive semantic topology refinement, is built using a graph neural network, which aims to learn semantic features through interactions among source domains. The effectiveness of the approach is demonstrated on a number of commonly-used domain generalization benchmarks.
strengths_and_weaknesses: **Originality**: From the technical point of view, the proposed ideas, including the data mixing strategy and the graph neural network-based module, are novel. It's interesting that Grad-CAM can be used in this way for data augmentation.

**Quality**: Overall the quality is good. The main experiments are comprehensive and solid for justifying the technical contributions. The ablation studies are also sufficient. It's worth mentioning that the datasets cover a wide range of tasks including generic object recognition and medical image analysis. However, the paper misses more in-depth analysis on "how" the graph neural network-based ASTR module improves the performance. The quantitative results are sufficient to justify the effectiveness but a more scientific analysis is needed to help readers understand how the "topology" is built by the neural network and in what way it helps learn structural invariance. In other words, the concept of semantic topology is new and sounds "big" but there is no concrete studies on how this concept works. (Maybe use a theory or visualize what was learned by the model?)

**Clarity**: The paper is well-written.

**Significance**: The paper provides new insights on how data augmentation can be combined with graph neural networks to improve domain generalization. Moreover, the results demonstrate that each of the two components alone could be useful to the community: the data mixing strategy works with some existing methods like MixStyle and EFDM, while the graph neural network can be used for medical image analysis.
questions: - Would it be possible to show some visualization on what the "semantic topology" looks like or what was learned by the ASTR module?
- In Table 7, only ASTR is evaluated. Does CDM also work for the medical datasets?
- Regarding the use of Grad-CAM, does it always give accurate masks? Are there any failure cases? And what happens if Grad-CAM doesn't produce accurate masks?

-------- Post-rebuttal updates --------

The authors' responses have resolved all the reviewer's concerns.

As mentioned in the 1st round review, the reviewer appreciated the contributions and believed the proposed approach and insights will be useful to the community (the reviewer's stance remains the same). The added materials during the rebuttal have significantly strengthened the motivation of the paper and made the story more convincing.

The reviewer has also read other reviewers' comments as well as the authors' responses, and found no critical issues that could lead to rejection.

Overall, the reviewer likes the contributions of the paper and believes they would be of interest to the domain generalization community. Therefore, the reviewer strongly recommends acceptance and stands firmly on this side.
limitations: Currently there is no discussion about the limitations of the approach. Perhaps there are some limitations related to the use of Grad-CAM (see the Questions section).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes Category-aware Data Mixing (CDM) to augment data on the data level and Adaptive Semantic Topology Refinement (ASTR) to maintain an invariant semantic topology of classes across different domains. The method (combination of CDM and ASTR) achieves excellent results on multiple DG datasets. Besides, experiments also show that CDM can bring performance gains to several other DG methods.
strengths_and_weaknesses: Strengths (originality, quality, clarity and significance):
1. As far as I know, the originality is good.
2. The proposed method is technically sound. The experimental results compared with other DG methods is excellent.
3. This paper is generally easy to follow.
4. The idea of maintaining semantic topology of classes is novel.

Weaknesses:
1. The standard deviation (std) over 10 runs is missing. Considering most datasets are small, such as PACS and VLCS, it is more convincing to report the average accuracy with std for comparing model performances.
2. Some opinions in this paper are weak and unconvincing:
- Domain-wise invariance cannot guarantee generalizable representations. The authors argue that such invariance may be susceptible to including some misleading spurious correlations. This situation may occur in simulated data, where some semantically independent properties exist across all source domains. However, spurious correlation is difficult to hold simultaneously across all source domains in real-world cases, so it is reasonable to expect feature extractors to learn more semantic information through domain-invariant representation learning.
- The widely-adopted style-content-separation idea may fail to extract true semantic factors. The authors’ basis is that the activation map induced by domain classification does not focus on the background. This phenomenon is easy to understand because the style of foreground objects is also related to domain classification. However, I think that whether domain classification focuses on the background or not has no direct relationship with whether style-content-separation helps extract semantic information.
3. In Table 1 – Table 6, the authors should report DeepALL results that are implemented by themselves with the same training strategy as the proposed method. It is unfair to compare other methods with the extremely low DeepALL baseline, especially in Table 5. According to my experiments, it is easy for DeepALL to achieve an average accuracy above 82% with the ResNet-18 backbone on the PACS dataset. Experiments on DomainBed also report excellent performances of DeepALL (ERM).
questions: Please see weaknesses for my concerns.
limitations: The authors didn't address the limitations and potential negative societal impact of their work.
- It is better to point out potential limitations, e.g., the proposed method (ASTR) can't work when the domain label is unavailable.
- I didn't see potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a method called MiRe, for Mix and reason over semantic topology, to do domain generalization. The motivation is that existing work typically ignores the semantic information of the inputs and further lack topological structure, which authors claim to be consistent across domains. MiRe consists of two steps: background (other domains) + foreground (current domain) Mixup to generate new samples in different backgrounds, and graph network to mine the neighbor class feature information. Experiments are done in various benchmark datasets and demonstrates the efficacy of MiRe. 

=======

Post rebuttal: I really appreciate authors' efforts in providing detailed rebuttal, which resolves most of my concerns. I hope that the future version can include the modifications w.r.t. all comments. But please understand that everyone has a standard for accepting a paper. To me, my main concern is that the novelty is not that "sparking" and the method is extremely complicated with so many hyperparameters to tune. Thus, I could not give you high scores. However, I will not stand against rejection. I will give you a 4 accordingly. I'm trying to be nice to you even if my papers still do not get any responses...
strengths_and_weaknesses: ### Strength

1. The idea of bringing Mixup operation to domain generalization and further enhance its semantic representation is interesting.
2. The writing is sound, with interesting figures to show the ideas.
3. The experiments are extensive, with different comparison methods.

### Weakness

#### 1. Overclaims or several claims are not evaluated.

There are many claims in this paper and I fail to find supports for some of them. Thus, this paper could have some overclaims:
- In L36, how to evaluate that MiRe does not have the "data-dependent'' spurious correlations? I fear that MiRe is still a data-driven method and it heavily rely on the Mixup operation of the fore-background.
- In L42, authors claimed that previous methods require "some distribution of values for an attribute". I would assume MiRe does not. However, the adoption of Grad-Cam indeed assumes that the input data is image with clear fore and background information. Thus, it is an overclaim.
- In L44, I fail to find the answers to the question: "how many factors of the latent factors".
- In L132, why do existing data augmentation approaches lack diversity? Any support?

#### 2. Methodology.

- Technically, the method is not novel. It is a combination of Grad-Cam, Mixup, and GCN. I admire the application of these existing techniques. But there is no further insight and motivation is not strong. Plus, there are the following issues:
- In figure 3, why only cropping 1/8 of the original image? There lacks motivation.
- Similarly, in Eq. 4, why adopting $\operatorname{cos}(f, c_j) / 2$? How did the term $1/2$ come up? This also goes to Eq. 6.
- The second part of the method is too complicated and computationally expensive for multiple domains. Thus I fear the comparison to other methods is not fair.
- Regarding the reproducibility, this approach has introduced many extra hyperparameters to be tuned, to name a few, the hyperparameters in Grad-Cam, $\operatorname{threshold}$ in Eq. 1, $1/8$ in Figure 3, $1/2$ in Eq. 4 and 6, the Mixup hyperparameter in Figure 3, $\xi$ in Eq. 7, and $\lambda$ in Eq. 8. Given that there are so many hyperparameters, I highly doubt the reproducibility of this approach.
- Furthermore, the robustness of the hyperparameters should be reported. Thus, not only accuracy, but also the variation.

#### 3. Experiments.

- The experiments seem extensive. But after a careful check, I found that there is no reason to compare with different SOTA results in different datasets (e.g., the comparison methods in PACS and Office-Home are different). Plus, the model selection strategy is also different for each dataset. There is no support or explanation in the paper to illustrate why this is done.
- It is unknown where the authors obtain the results for comparison methods for the main paper. Then, if you can use DomainBed, why not just use it as the main benchmark?

#### References

Overall, the references are good. But you can also cite more recent works from the following DG survey articles:

[1] Zhou et al. Domain generalzation in vision: a survey.

[2] Wang et al. Generalizating to unseen domains: a survey on domain generalization.

#### Minor comments

Typos:
- L26, "has attract".
- L137, "we propose develop".
questions: See above comments.
limitations: N/A
review_title: null

Meta-review: metareview: Some of the reviewers had concerns about novelty, and one of the reviewers was worried about the care taken in training a baseline.  however, another reviewer has a strong positive opinion of the work; and I believe the authors have done a good job in rebuttal making an effort to address the concerns about baselines.  I am recommending acceptance; but I expect the authors to release code to allow further scrutiny w.r.t. baselines. 
confidence: Less certain
Decision: Accept