Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization
Authors: Chaoqi Chen
Luyao Tang
Feng Liu
Gangming Zhao
Yue Huang
Yizhou Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain generalization
distribution shift
semantic topology
structural invariance
data mixing

Abstract: Domain generalization (DG) enables generalizing a learning machine from multiple seen source domains to an unseen target one. The general objective of DG methods is to learn semantic representations that are independent of domain labels, which is theoretically sound but empirically challenged due to the complex mixture of common and domain-specific factors. Although disentangling the representations into two disjoint parts has been gaining momentum in DG, the strong presumption over the data limits its efficacy in many real-world scenarios. In this paper, we propose Mix and Reason (MiRe), a new DG framework that learns semantic representations via enforcing the structural invariance of semantic topology. MiRe consists of two key components, namely,  Category-aware Data Mixing (CDM) and Adaptive Semantic Topology Refinement (ASTR). CDM mixes two images from different domains in virtue of activation maps generated by two complementary classification losses, making the classifier focus on the representations of semantic objects. ASTR introduces relation graphs to represent semantic topology, which is progressively refined via the interactions between local feature aggregation and global cross-domain relational reasoning. Experiments on multiple DG benchmarks validate the effectiveness and robustness of the proposed MiRe. 

Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization
Chaoqi Chen' Luyao Tang² Feng Liu Gangming Zhao' Yue Huang? Yizhou Yu
'The University of Hong Kong X University
DDeepwise Al Lab
cqchen19940gmail. com, lytangostu. xmu xmu. edu en, liuf siufengddeepuise.o com
guzhao@connect. hku hk, yhuang2010@ 200mgm0 edu. en, yizhouydacm.cmmmm org
Abstract
Domain generalization (DG) enables generalizing a learning machine from multiple source domains unseen target one. The general objective of DG methods is to learn semantic representations that are independent of domain labels, which is theoretically sound but empirically challenged due to the complex mixture of common and domain-specific factors. Although disentangling the representations into disjoint parts has been gaining momentum in DG, the strong presump- tion over the data limits its efficacy in many real-world scenarios. In this paper, we propose Mix and Reason (MiRe), a new DG framework that learns semantic representations via enforcing the structural invariance of semantic topology. MiRe consists of two key components, namely, Category-awe aware Data Mixing (CDM) and Adaptive Semantic Topology Refinement (ASTR). CDM mixes two images from different domains in virtue of activation maps generated by two complementary classification losses, making the classifier focus on the representations of semantic objects. ASTR introduces relation graphs to represent semantic topology, which is progressively refined via the interactions between local feature aggregation and global cross-domain relational reasoning. Experiments on multiple DG benchmarks validate the effectiveness and robustness of the proposed MiRe.
Introduction
In real-world scenarios, such as autonomous driving and computer-aided diagnosis, deep neural networks expected to be trained data collected from small number of domains, but with the objective of being deployed across novel domains with different data distributions to reduce annotation costs and meet broader range of needs. However, distrii-itiiiioonn data [29] does not satisfy the basic IID assumption of modern deep learning models and emerges as an inevitable challenge, significantly hindering the deployment of source-trained models in target domains. Domain generalization (DG) [47, 6, 23], which addresses this challenge by learning invariant representations across multiple source domains, has attract great deal of attention in the research community.
The objective of DG is to recover latent semantic factors that are independent of domain. Most DG approaches rely the assumption that the latent representations can be divided into two disjoint components, i.e,, common and domain- specifcc components. Inspired by this, mainstream DG paradigms include data/feature augmentation [68, 75], feature disentanglement [53, 40], invariant risk minimization [3, 2), meta-learning [33, 17], to name a few. By facilitating common knowledge extraction and alleviating domain-specife components, these approaches have made tremendous progress in various tasks, such as image classification [6, 84) and semantic segmentation [37, 51].
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Source-1 1
Source-2
Target
lateut feature
commonn specific
Triangle
Disentanglemeen
Circle
common factor: shape domain-specific factor: color
spurious correlation:
hlue triangle and cirde
Data Mixing with Random Background
(a) Ideal Case
(b) Roal-World Case
(a) In ideal case, we assume that the data has two attributes (shape and color). (b) In real-world case, we observe that bird and airplane share more similarity as the design of airplane is motivated by bird, while both of them are dissimilar with bicycle. Thus, the embedding of bird and airplane should be closer and their connecting weight should be enlarged, which are independent of domain and robust to unseen test environments. We call this property as structural invariance.
Figure 1:
Despite the promise, existing DG approaches may still be confined by two critical limitations. (1) Traditional DG methods [47, 36, 6, 34, 17] strive to enforce invariance between domains but ignore the potential spurious correlations between common and specifie components. Take Fig. (a) as an example, the correlation between shape and color may induce the classifier to use background colors as the classification evidence. (2) To statistically avoid the spurious correlations, a body of research [54, 53, 16, 44, 61, 40] proposes to identify the common component through latent factor disentanglement. In this regard, causal graphs [50] provide promising theoretical guarantees. As shown in Fig. 1 (a), previous disentanglement -eaudd methods manifest remarkable success in simulated data, requiring that models have seen some distribution of values for an attribute [71] (e.g.. shape and color in DSPRITES [46]), which is too restrictive to be satisfied. In real-world case (cf. Fig. (b)), we cannot (i) explicitly know the latent feature composed of how many factors, (ii) identify which factors are semantic-related or not, and (iii) how these potential factors interact. Since we have no a priori knowledge or assumptions about the attributes (factors), it is prohibitively difficult to disentangle them in unsupervised manner [42], largely impeding the practical use of these approaches. On the other hand, the internal relations among different classes are neglected, which are crucial for the robustness of learned models to distribution shifts. This finding draws motivation from the human intelligence that when learning new concepts, humans are talented at comparing and reasoning [26, 15, 5]. In that sense, aim to endow the classifier with the ability of perceiving and maintaining structural semantic relations rather than ideally seeking clean semantic' representations.
Grounded on the above discussions, we propose Mix and Reason (MiRe), simple yet effective DG approach, which enables generalization via data-level augmentation and feature-level regularization. The proposed MiRe consists of two key components, Category- aware Data Mixing (CDM) and Adaptive Semantic Topology Refinement (ASTR), which learn semantic representations by enforcing structural invariance of semantic topology across domains in the embedding space. CDM induces the classifier to focus on recognizing foreground objects as their supporting evidence for classification, alleviating the potential influence of domain-specine factors (e.g., scene layout and style). Specifically, foreground region is derived via integrating two complementary network activation maps generated by category and domain classification losses, while background is randomly cropped from images sampled from other domains with Gaussian blur. ASTR introduces the concept of semantic topology, which is instantiated the relation graphs, progressively refine the representations (node) and the topological relations (edge) of semantic anchors. Semantic anchor is denoted by the centroids of category-wiss features in per domain, largely reducing the antra-domain variations. To be specific, each input instance is compared to the set of semantic anchors and dynamically aggregate features according to the global2 semantic topology and local adjacent relations. Then, the global features updated by the aggregated local features. Finally, a bipartite graph network is developed on the top of semantic topology to perform domss- relational reasoning and induce structural invariance.
'The word 'semantic" and 'common' used interchangeably. but 'invariance' does not equal to them. Invariance may contain spurious correlation. which can be alleviated but difficult to be absolutely eliminated. 'The word 'global' denotes that the features are computed from the whole domain (dataset), and the word 'local' denotes that the features computed from mini-batch samples at each iterations.

---

Experiments on four standard DG benchmarks (PACS, VLCS, Office-Homes and DomainNet) reveal that MiRe exceeds the performance of ttate-of-the-art methods. In addition, CDM is applied as a plug-and-play module for ofte--o-thhearr DG methods (such as MixStyle [84] and EFDM [78]) to boost their performance. The robustness of ASTR further evaluated on a challenging Chest X-ray benchmark [44] that has explicit spurious correlation.
2 Related Work
Domain Generalization. Generalizing a well-trained model to novel environments with different
data distributions is a challenging and long-standing machine learning problem [81, 69]. Current state-of-the-art DG approaches can be roughly categorized into four types. (1) Feature Alignment. Bridging the domain gap through statistic matching and adversarial learning [38, 36, 45] provides invariant representations, which are expected to be shared by novel domains. A major defect is that the learned representations is prone to mix common and domain-specific components, potentially resulting in strong bias towards spurious relations. (2) Feature Disentanglement. To address the above issue, disentangling latent feature into two disjoint parts has attract a great surge of interest. Prevailing approaches [54, 53, 16, 44, 61, 40] resort to causal graphs [50] to explicitly identify causal and non-causal factors with theoretical guarantees. If this could be perfectly achieved, learned models will generalize well under any circumstances. However, these theoretical results require strong assumptions, e.g., the degree of diversity between causal and non-causal factors and the presence of an estimated causal graph [62], or a priori knowledge about the combinations of latent factors, e,g. distribution of values for a certain attribute. The complex combinations of many real- world cases (e.g., DomainNet [52]) greatly impede the practical uses. In that sense, principled disentanglement solution is hard to be reached. (3) Data/Feuture Augmentation. A simple yet effective approach is to augment the diversity of data or feature so to avoid overfitting training data [67, 75, 48, 84, 74, 8, 78, 13]. Among them, learning and imposing heuristic style transmission strategies take the dominant positions, while explicitly separating and recombination images remain the boundary to explore. (4) Meta-Learning Some of recent works [33, 39, 34, 17, 41, 14] simulate the distribution shifts between seen and unseen environments by using meta-learning [21], which splits the training data into meta-traie and meta-test domains.
Despite a proliferation of DG approaches, Wiles et al. [71] reveal that the best DG methods are not consistent over different shifts and datasets in which disentangling offers limited improvements in most weal-world scenarios. In addition, Gulrajani and Lopez-Paz [23] cast doubt on the progress that have been made by comparing to a standard empirical risk minimization baseline, showing that conventional domain-invarian methods [60, 22] exhibit robust improvements. In this work, embrace the strengths of cross-domain invariance by considering a more fine- fine-grained and stable property- structural invariance which is evaluated on number of challenging benchmarks.
Data Mixing. Mixing data, such as Mixup [77, 66] and its various variants, has shown compelling results in standard supervised and semi- superviid learning. Mixup operations aim to conduct data interpolation via convex combinations of pairs of images and their labels. Instead, our CDM generates diverse training samples by replacing the background of certain image with a randomly cropped patch from other images but keeps its object label fixed. CutMix [76] replaces a regularly-shaper image region with a patch from another training image and proportionally mix their ground-truth labels. CutPaste [31] cuts an image patch and randomly pastes it at an another image. BackErase [56] uses the object annotations copy and pastes foreground objects on background image. Compared to the aforementioned methods that cut a regularly-shaped image patch [76, 31] or rely on ground-truth object labels [56], the proposed CDM simultaneously leverage the category and domain information to derive an irregularly-shape foreground regions while avoiding fine- -o-ggrrineed annotations.
Cross-Domaiis Invariance. Seeking cross-domain invariance is crucial for out-of-distribttion generalization problems, such as Unsupervised Domain Adaptation (UDA) [49] and DG. Traditional methods typically resort statistical distribution divergence, such as maximum mean discrep- ancy [43] and second-order moment [60], for measuring domain-wise distribution shifts. More recently, a promising approach is to utilize the concept of class prototype for enforcing semantic consistency across domains [73, 12, 10]. This line of research has also been extensively explored in various downstream tasks. However, we argue that no matter domain- or class-wise invariance cannot guarantee the generalizable representations especially when encountering unseen test environments.
3

---

Calegory-awaare Data Mixing (CDM)
Adaptive Semantic Topology Refinement (ASTR)
D2
Di
G
Contraslive "oosistrncy
D2
D2
Graph Conv
SemanticAnchor
Local Instance
Topology
Step
Stop
Step
Figure 2: The pipeline of the proposed MiRe. Assume that we have two domains D1 and D2, CDM augments them to Dj and 2/2 In each iteration of ASTR, the embedding feature of each local instance performs feature aggregation based on the semantic topology in previous iteration and their local adjacent relations (step 1). Then, global anchor features are updated based on the aggregated local features (step 2). Finally, a bipartite graph learning procedure is imposed on the top of semantic topology to perform eetween-domain relational reasoning and induce structural invariance (step 3).
Such invariance may be susceptible to include some misleading spurious correlation, such as wrongly associating an foreground object with specific background. The justification that the topological structure of these prototypes is unexplored, making them being sensitive to the change of environments [44, 9, 79J. By contrast, the proposed MiRe models the graphical structures different semantic anchors by means of feature aggregation and message-passiine
3 Methodology
In the task of DG, given that we have access to data from N (N > 2) source domains D, {D,, D2,,D,,.. D....., Dx that are sampled from different data distributions defined on the joint space X x y. The n-th domain can be defined D, {(I;, M4, d where y: {1, ..., K} } is the class label, d: = {1, ..., N} is the domain label, and Na is the number of samples in this domain. An overview of MiRe is illustrated in Fig. 2. MiRe consists of CDM stage (Section 3.1) and an ASTR stage (Section 3.2), which jointly enable generalization on data-level and feature-level.
3.1 Category-aware Data Mixing
Most of DG works aim to learn predictive models with certain invariant properties that are independent of domain. As aforementioned, this sort of invariance may be brittle due to the presence of spurious correlation (cf. Fig. 1). To build robust classifier, many recent efforts are devoted synthesize novel images/features through various augmentation strategies, such as image transformations [67] and style transformations [84, 48, 78]. They typically manipulate the low-level feature statistics (e.g., color and texture) to generate diverse image styles while maintaining the content unchanged. However, we argue that these operations may still be constrained by the spurious correlation as style-agnostic representation is not sufficient to ensure semantic invariance [44].
Inspired by this, we introduce a novel data mixing strategy, which incorporates the categorical information to directly recombine source images. Considering that the category and domain labels of each sample are two orthogonal yet complementary supervision signals, we directly obtain the activation maps generated by these two classification networks for depicting the foreground regions.
An overview of the CDM is illustrated in Fig. Specifically, for an input sample I;, first leverage the vanilla Grad-CAM [57] to generate activation maps M, (class) and Ma (domain) by using its label yi and d,, respectively. As shown in Fig. the activation map (Ma) induced by domain classification does not focus on the background as our expectation (this phenomenon holds in most DG benchmarks). Thus, we merge Me and M2 extract the foreground objects as follows,
My threshold(M. + Ma).
(I)

---

Cartoon
Element wise product
Elemesl-wist addition
D2
Gaussion Blur
Random
Crop
Mc
1 M
Art
D
Merge
Mixed Images
My
Photo
Ma
D3
Goussian
Random
Blaz
Crop
Figure 3: The workflow of CDM. For image from Domain "Art", (i) extract two types of its foreground CAM (Me and Ma) and merge them (My); (ii) randomly select two images from Domain "Cartoon" and Domain "Photo" to construct two backgrounds; (iii) apply Gaussian blur to the selected image and then randomly crop a patch from the blurred image; (iv) mix the foreground and background images by virtue of the computed M₁-
where threshold()) aims to reduce the outlier points and is set to 0.2 in all experiments to avoid additional tuning. Intuitively, Ma is a coarse-level supervision, which is sensitive to the domain-wise variations, while M. is fine-level supervision, which is sensitive to the intra-domain variations.
Next, simulate the real-world situation that an object may appear in various scenes with distinet backgrounds, we randomly crop small region of an image sampled from other domains. Before cropping, a Gaussian blur is applied to image Iy to reduce image noise and detail, i.e., x' = (1, A small patch is cropped from z'j and resized to the same scale, i.e., x;) resize resize(crop(a)) cx) Finally, we synthesize a new image x/ by substituting the background of 2; with the background of 2' using the merged Grad-CAM of image 2; as a balancing weight, i.e.
crop
(2) where @ is the element-wise product. The above operation applied to every source image, and will obtain (N - 1) |Da| images in all, where |D,| is the number of original source images.
nix = My O x (1-11
My) @ xj
crop
Adaptive Semantic Topology Refinement
Structural Invariance. After the data mixing, we introduce Adaptive Semantic Topology Refinement (ASTR) mechanism adaptively and progressively refine the topological structure of semantic anchors in the feature space, explicitly constraining the structural invariance across domains. The key idea is to endow the semantic anchor with the capability of reasoning over semantic topology globally, where semantic anchor is represented by the prototype features of each class. We argue that semantic topology, which consists of the semantic anchors as well as their interactions, is robust to unseen test environments. Technically, the structural invariance is characterised from two aspects: cross-domain variation and cross-model variation (historical and present models), implying that the semantic topology should be independent of the spatial and temporal changes.
Overview. We introduce an undirected graph G (V,E, C') to represent the semantic topology, where | K is the set of K nodes and & C V x V is the set of edges. The feature matrix is denoted by C (c.....,c cx) E KxD where Ca denotes prototype corresponding to the k-th class, D is the feature dimension, and each element in C' is associated to node E V. The adjacency
5

---

matrix is denoted by A E R where each element Asy denotes the weight of edge eij V. During training, each imput sample first compared to set of semantic anchors and then adaptively aggregates features from its adjacent anchors to update its features. All updated sample features will be further used to update the corresponding anchor feature. Finally, the structural invariance optimized based on bipartite graph learning.
Cg based on the vanilla DeepAll Step 0: ASTR first computes the initial global anchor features (0)
model, which is trained on both original and generated data. For the relation graph, we define its corresponding adjacency matrix G8lobal as follows,
1 expl 1.2 - -cyll
20²
where / indicates the iteration times and x² is the kernel bandwidth.
Step I: At each training iteration, a batch of samples B is projected into the embedding space via the feature extractor G, i.e, fi = G(r)), x, E B. For the sake of simplicity, we omit the subscript i. In the /-th iteration, for image that belongs to class k, we perform feature aggregation as follows,
K
exp((A)) (1-1) 1) kj
+ cos(S, (;))/22
=f + (1
Σ 1=1
my
global
Wi
Σ, exp(( A (1-1) + cos( 22))/2)
kj
where j = {1, 21, K}. cos(..-) denotes cosine distance, and Φ E (0, 1) is a positive constant. cos(f. Cj) locally measures the relations between each instance and all semantic anchors, while A (1-1) provides the global relations among different semantic anchors. By doing so, each instance will be aware of the holistic semantic topology and dynamically conduct aggregation.
Step 2: Then, we update the global anchor features based on B in moving average manner, @
global i(n)
Jocal + +1 Φ) ((T)
global ((1-1))
(5)
docaa stands for the local anchor features computed from all f" Meanwhile, the adjacency where ((l) matrix will be updated accordingly.
Step 3: Finally, optimize the coos--domain and cross-mode consistencies achieve structural invariance. A naive approach is to compare each anchor in one domain/model to all anchors in the other domain/model. However, the structural relations of each anchor within the graph are neglected. On the other hand, the constructed graph may still contain some spurious edges, leading to unreliable inter-graph node correspondence reasoning. Remedying these issues, we introduce a Bipartite Graph Convolutional Network (BGCN) to perform droms- relational reasoning based on a bipartite graph (V Va.r,& & A where s' and s" can be any two source domains. The node affinities between Vor and V."' are simultaneously constrained by cross-domain and cross-model variations. Specifically, for node v; in Var and node vy in Vyr, we have,
global global + cos( cos(c global global cos 1) ((1)
bg
511-13
cos( (g global globa)) ((I) Su)
(6)
By doing so, the influence of those unreliable intra- and inter- giteergrapphhe correspondence were largely reduced. Then, we conduct bipartite graph convolution [28, 11] over go by stacking multiple layers. Mcanwhile, we introduce a contrastive consistency regularization on the top of BGCN,
Loca =
)..))
) +
Σ (maxf0, D( (b,,))))
(7)
1==1
where z is the embedded vector through BGCN, & is the squared Euclidean distance, and & is the margin term and is set to 2 in experiments. To this end, the overall training objective of MiRe is,
min E ( Less + X Loss
(8)
where Cots is the standard cross-entropy loss and À is the trade-off parameter.

---

Table 1: Generalization results (%) on PACS benchmark.
DeepAll
JiGen
CrossGrad
MSAF
ER
Metareg [4]
RSC
MixStyle
SagNet [48]
EFDM [78]
MiRe
Source
Target
[6]
[59]
[17]
[80]
[25]
[84]
(Ours)
C,P,S
Art
77.0:1.3
79.4
79.8
80.3
80.7
83.7
78.9
83.1
83.6
83.9
8..610.55
A.P.S
Cartoon
74.8:105
75.3
76.8
77.2
76.4
77.2
76.9
78.6
77.7
79.4
79.510.4
A,C,S
Photo
95.8+0.11
96.0
96.0
95.0
96.7
95.5
94.1
95.9
95.5
96.8
96.8+0.2
A.C.P
Sketch
70.0:10.6
71.6
70.2
71.7
71.8
70.3
76.8
74.2
76.3
75.0
1..4
Average
79.4
80.5
80.7
81.1
81.4
81.7
81.7
82.9
83.3
83.9
84.8
Table 2: Generalization results (%) on VLCS benchmark.
DeepAll
DBA-DG
MMD-AAE
MLDG
Epi-FCR
JiGen
MASF
MMLD
SFA-A [35]
MiRe
Source
Target
[32]
[36]
[33]
[34]
[6]
[17]
[45]
(Ours)
I,C,S
VOC
71.8+0.6
70.0
67.7
67.7
67.1
70.6
69.1
72.0
70.4
70.3+0.3
V,C,S
LabelMe
61.110.4
63.5
62.6
61.3
64.3
60.9
64.9
58.8
62.0
63.6
V.L.S
Caltech
95.8++0.2
93.6
94.4
94.4
94.1
96.9
94.8
96.7
97.2
96.2=t0.4
V.L.C
Sun
62.5+00.6
61.3
64.4
64.4
65.9
64.3
67.6
68.1
66.2
69.4+0.7
Average
72.8
72.1
72.3
72.3
72.9
73.2
74.1
73,9
74.0
74.9
Table 3: Generalization results (%)
Office-Home benchmark.
DeepAll MMD-AAE
CrossGrad JiGie
DDAIG
DOSN
RSC
SagNet [48)
MixStyle [84]
MiRe
Source Target
[36]
[59]
[6]
[82]
[58]
[25]
(Ours)
C,P.R
Art
59.4:10.4
59.9
58.4
53.0
59.2
59.4
58.4
60.2
58.7
60.2:10.8
A.P.R
Clipart
48.0
47.3
49.4
47.5
52.3
45.7
47.9
45.4
53.4
53.2:10.9
A.C.R
Product
72.7
72.1
73.9
71.5
74.6
71.8
71.6
70.4
74.2
75.1100.6
A.C.P
Real
75.3
74.8
75.8
72.8
76.0
74.7
74.5
73.4
75.9
76.4110.6
Average
63.9
62.7
64.4
61.2
65.5
62.9
63.1
62.4
65.5
66.2
4
Experiments
4.1 Setup
Datasets. We fully verify the generalization performance of MiRe on four standard DG benchmarks:
PACS [32], VLCS [19], Office-Home [65], and DomainNet [52]. (1) PACS is the most-widely used DG benchmark, which has 9,991 images of 7 classes from four kinds of environment: Photo, Art Painting, Cartoon, and Sketch. (2) VLCS contains 10,729 images of 5 classes from four photographic
domains: PASCAL VOC 2007 [18), LabelMe [55], Caltech [20], and Sun [72]. (3) Office-Home composed of 15,500 images of 65 classes from four domains (Artistic, Clipart, Product, and Real World). The images are collected from office and home environments. (4) DomainNet is a more recent large-scale dataset for multi-source domain adaptation and DG. It spans 0.6 million images of 345 classes from six domains (Clipart, Infograph, Quickdraw, Painting, Real and Sketch).
Model Selection. To facilitate fair comparisons, we strictly follow the model selection strategy used by baseline methods. In general, we follow the leave-one-dee one-domain-ou setting, í.e., one domain is chosen as the held out domain and the remaining domains are seen source domains. For PACS, we follow the original train and val splits established by [32]. On VLCS, according to the prior methods [6, 45], we split 30% of the source samples validation datasets. For Office-Home, we employ the train and val splits established in [32, 83, 84]. On DomainNet, we follow the train and val splits used in [7], i.e., randomly divide the train split from [52] in 90-10% proportion to obtain train and val splits. Then, we select the model that shows the best performance in the validation dataset.
Implementation Details. For PACS, Office-Home and DomainNet, we use ResNet-18 18 [24] pre- trained on the ImageNet as the backbone feature extractor. To include more baselines for comparison, we use AlexNet [30] for VLCS. On DomainNet, we also provide results of ResNet-50. The networks are trained using SGD with momentum 0.9 and weight decay 5e-4 for 100 epochs. For PACS, VLCSS and Office-Home, the batch size is set to 16. For DomainNet, the batch size is set to 256. For CDM, the cropped area ratio (background image) is empirically set to 5 For Eq. (4), we set Φ = 0.5. For Eq. (8), we set X 0.1. We evaluate the out-of-domain generalization performance in all object

---

Table 4: Generalization results (%) on DomainNet benchmark.
Method
Clipart
Infograph
Quickdraw
Painting
Real
Sketch
Avg
ResNet-18 18
DeepAll
17.7
43.2
13.9
54.9
39.4
37.7
Multi-Headed [7]
55.4
17.5
40.8
11.2
52.9
38.6
36.1
MetaReg [4]
53.6
21.0
45.2
10.6
58.4
42.3
38.5
DMG [7]
60.0
18.7
44.5
14.1
54.7
41.7
39.0
MiRe
62.5
23.0
46.5
15.7
59.1
44.4
41.9
DeepAll
61.8.10.6
20.2.10.7
45.0.1
1.
57.310.7
44.8.10.6
40.6
ResNct- 50
Multi-Headed [7]
61.7
21.2
46.8
13.8
58.4
45.4
41.2
MetaReg [4]
59.7
25.5
50.1
11.5
64.5
50.0
43.6
DMG [7]
65.2
22.1
50.0
15.6
59.6
49.0
43.6
MiRe
64.7+1.
27.8=1.2
53.120.8
16.4
64.1=0.6
52.310.9
46.4
classes and report the means and standard derivations over 10 runs with different random seeds. We implement all experiments based on the PyTorch framework.
4.2 Baselines
We compare MiRe against state-of-the-art DG methods, which are summarized follows. DeepAll:
Training a classification model through simply merging all source domains. Feature Alignment:
MMD-AA- MAD--AE [36] and EFDM [78]. Feature Disentanglement: DOSN [58], DMG [7] and Sag-
Net [48]. Data Augmentation: JiGen [6] and DDAIG [82]. Feature Augmentation: Cross-
Grad [59], MixStyle [84] and SFA-A [35]. Meta-Learning: MASF [17] and Metareg [4]. Other SOTA: RSC [25] designs a model-agnostic training strategy, and ER [80] ensures the conditional invariance via entropy regularization term. For most of the compared approaches,
cite the classification accuracy reported in their original papers. In PACS, the results of 'RSC' and 'MixStyle' obtained with their official codes by following the leave-one-domain-out setting.
4.3 Results
PACS and VLCS. In Tab. 1 and Tab. 2. we display the quantitative comparisons with baseline methods on PACS and VLCS benchmarks, As can be seen, MiRe outperforms state-of-the arr DG methods by large margins, revealing the effectiveness the proposed structural invariance in real- world data. In particular, MiRe improves on the PACS dataset. For the hard generalization task A, C, + Sketch, our method substantially outperforms the suboptimal result by 1.6%.
the best baseline method [78] (CVPR 2022) by 0.9%
Office Home and DomainNet. Tab. and Tab. report the results on Office-Home and DomainNet benchmarks, which have larger data volume in contrast to PACS and VLCS. The proposed MiRe substantially and consistently exceeds all comparison methods, showing the scalability our MiRe on large-scale datasets with distinct classes.
4.4 Empirical Analyses and Ablations
Ablation Study. We investigate the role of each part of MiRe by conducting in-depth ablation studies on four benchmarks. The results listed in Tab. 5. We highlight two of key observations. (I) CDM w/o (1 - Ma) represents that we replace Ma by (1 -Ma) in Eq. (1). The results testify that the domain classification evidence is beneficial for depicting the foregrounds.
addition, Mc- My. and Gaussian blur operation are reasonably designed, providing high-quality augmentation results. Only with CDM could provides very compettive results. (2) Introducing relation graphs to model semantic topology plays an important role in encouraging the so-called structural invariance especially on the challenging benchmark, such as DomainNet.
The Effects of CDM. Since CDM does not requires any specific model architectures and training procedures, we apply it to three existing methods, i.e., ER [80], MixStyle [84] and EFDM [78], to further enhance their classification performance. The results are shown in Tab. 6, where CDM consistently improves over the baseline methods.

---

Table 5: Ablation of MiRe four DG benchmarks (%).
Method
PACS
VI.CS
Office-Hame
NomainNet
Average
DeepAll
79.4
72.8
63.9
63.5
MiRe w/o CDM
83.5
74.0
65.1
39.3
65.5
CDM w/o Me
82.6
72.8
39.4
65.0
CDM MM
84.4
74.4
65.7
41.0
66.4
CDM w/ Gaussian Blur
83.9
74.2
64.6
40.6
65.8
CDM w/ (1 Ma)
83.0
73.4
64.9
38.8
65.0
MiRe w/o ASTR
82.3
71.8
64.7
38.9
64.4
ASTR w/o Cross-Domain Invariance
84.0
74.5
65.6
40.6
66.2
ASTR w/o Cross-Model Invariance
84.3
74.6
65.7
41.0
66.4
ASTR w/o Graph Structure
82.8
73.3
64.8
40.1
65.3
ASTR w/o Feature Aggregation
84.2
74.1
65.7
40.4
66.1
MiRe
84.8
74.9
66.2
41.9
67.0
Table 6: The effects CDM on PACS benchmark.
Source
Target
DeepAll
CDM
ER (81.4)
+ Mixxttyle (82.9)
EFDM (83.9)
C.P.S
Art
77.0
80.8
80.4
82.9
83.8
A.P,S
Cartoon
74.8
78.5
77.8
78.2
81.3
A,C,S
Photo
95.8
94.8
94.4
96.0
95.5
A.C.P
Sketch
70.0
75.0
77.4
77.6
78.2
Average
79.4
82.3
82.5 (+1.1)
83.7 (40.8)
84.7 (+0.8)
(a) DeepAll
(b) MixStyle
(c) EFDM
(d) MiRe (Ours)
Figure 4: The t-SNE visualization of extracted deep features using different methods on PACS dataset.
The different colors stands for different classes.
Table Results (%) on Chest X-Rays data.
The Effects of ASTR. To testify the robustness of ASTR
Method
RSNA
ChexPert
NIHI
under the presence of spurious correlation, we addition- ally perform experiments on three Chest X-ray datasets:
ERM
55.1
60.9
53,4
IRM
57.0
63.3
54.6
NIH [70], ChexPert [27] and RSNA [1]. We follow the same settings as [44] to create spurious correlation. The empirical comparisons
CSD
58.6
64.4
54.7
MatchDG
58.2
59.0
53.2
shown in Tab. 7, where the results of baseline methods (i.e., ERM [64], IRM [3],
ASTR
63.6
65.0
56.4
CSD [53] and MatchDG [44]) are cited from [44].
Feature Visualization. We use the t-SNE algorithm [63] visualize the deep representations in ResNet- 18 when Photo (PACS benchmark) is regarded as target domain. We compare the proposed MiRe with DlepAll, MixStyle [84], and EFDM [78] Fig. 4. As expected, MiRe exhibits better clustering (class) patterns due to the refinement of semantic topology during training.
Conclusion
In this paper, we proposed the MiRe framework to achieve out-of -i-diitrinttion generalization. The key idea our method endow the predictive models with the capability of reasoning over the semantic topology. MiRe instantiates this objective with the incorporation of two elaborate modules, i.e., CDM and ASTR. CDM augments the original data to reduce potential bias towards spurious correlations, while ASTR builds relation graphs represent semantic topology, which is updated through feature aggregation and message-passing Experiments on standard benchmarks verified the effectiveness of MiRe. Additional empirical analysis reveals the individual effeett of each component.

---

Acknowledgements
This work was supported by Hong Kong Research Grants Council through Research Impact Fund (Grant R-5001-18).
References
[1] Kaggle: Rsna pneumonia detection challenge. 2018.
[2] Kartik Ahuja, Karthikeyan Shanmugam. Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML, pages 145-155, 2020.
[3] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz Paz. Invariant risk minimization. arXiv preprint / 2019.
[4] Yogesh Balaji. Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularizatioo. NeurIPS, pages 998-1008, 2018
[5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 100.vv888000166 2018.
Vinicius Zamhaldi, Mateusz
[6] Fabio M Carlucci, Antonio D' Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, pages 2229-222222222222222222222- 2238,, 2019.
[7] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In ECCV, pages 301-318, 2020.
[8] Chaoqi Chen, Jiongeheng Li, Xiaoguang Han, Xiaoging Liu, and Yizhou Yu. Compound domain general- ization via meta meta-knowledge encoding. In CVPR, pages 2119-712,,200
[9] Chaoqi Chen, Jiongeheng Li, Zebiao Zheng, Yue Huang, Xinghao Ding, and Yizhou Yu. Dual bipartite graph learning: A general approach for domain adaptive object detection. In ICCV, pages 2703 2712, 2021.
[10] Chaoqi Chen, Jiongeheng Li, Hong-Yu Yu Zhou, Xiaoguang Han, Yue Huang, Xinghao Ding, and Yizhou Yu. Relation matters: Foreground-aware graph-based relational reasoning for domain adaptive object detection. TPAMI, 2022.
[11] Chaoqi Chen, Yushuang Wu, Qiyuan Dai, Hong- Yu Zhou, Mutian Xu, Sibei Yang, Xiaoguang Han, and Yizhou Yu. A survey graph neural networks and graph transformers in computer vision: A trientri perspective. arXiv preprint Xrxv:22009 13232, 2022.
[12] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In CVPR, pages 627-636, 2019.
[13] Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In CVPR, pages 187 18710-18719 2022.
[14] Liang Chen, Yong Zhang, Yibing Song, Jue Wang, and Lingqiao Liu. OST: Improving generalization of deepfake detection via one-shot test-time training. In NeurlPS, 2022.
[15] Noam Chomsky. Aspects of the Theory of Syntax, volume 11. MIT press, 2014.
[16] Rune Christiansen. Niklas Pfister. Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework for distribution generalization. IEEE TPAMI, 2021.
[17] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NewrlPS, pages 6447-6458, 2019.
[18] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):03-333, 2010.
[19] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV. pages 1657-1664, 2013.
[20] Li Fei-Fei. Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In CVPR-M W pages 178 178,, 2004.
10

---

[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.
[22] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario Marchand, and Victor Lempitsky. Domain-adversariill training of neural networks. JMLR, 17(1):2096-200 2016.
[23] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[25] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross domainn generalization. ECCV, pages 124 140, 2020.
[26] Wilhelm Humboldt. On language: On the diversity of human language construction and its influence on the mental development the human species. 1999.
[27] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI, volume 33, pages 590-597, 2019.
[28] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[29] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In ICMI, pages 5637-5664, 2021.
[30] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, pages 1097-1105, 2012.
[31] Chun-Liang Li. Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In CVPR, pages 9664-9674, 2021.
[32] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper. broader and artier domain generalization. In ICCV, pages $542-5550 2017.
[33] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018
[34] Da Li, Jianshu Zhang, Yongxin Yang. Cong Liu, Yi-7he Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, pages 1446-1455, 2019.
[35] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV, pages 8886-8895, 2021.
[36] Shuang Li. Shiji Song, Gao Huang, Zhengming Ding, and Cheng Wu. Domain invariant and class discriminative feature learning for visual domain adaptation. IEFE Transactions on Image Processing, 27(9):4260-42) 4273, 2018.
[37] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty modeling for out-of-distribution generalization. In ICLR, 2022.
[38] Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pages 624-639, 2018.
[39] Viying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In ICML, pages 3915-3924, 2019.
[40] Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang. Tao Li, Tao Qin, Wei Chen, and Tie- Yan Liu. Learning causal semantic representation for out-of-distribation prediction. NewrlPS, 34, 2021.
[41] Quande Liu, Qi Dou, and Pheng-Am Ann Heng. Shape-aware meta-lcarning for generalizing prostate mri segmentation to unseen domains. In MICCAI, pages 475-485. Springer, 2020.
[42] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, pages 4114-4124 2019.

---

[43] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, pages 97-105, 2015.
[44] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML, pages 7313-7324, 2021.
[45] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. In AAAI, 2020.
[46] Loie Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. Disentanglement testing sprites dataset. In https:://iitub..oo/eeeeee http://wwww.meeeeed////seeeeddee 2017.
[47] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In ICCV, pages 5715 5725, 2017.
[48] Hyeonseob Nam, HyunJac Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, pages 1690-8699, 2021.
[49] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199 210, 2011.
[50] Judea Pearl. Causality. Cambridge university press, 2009.
[51] Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, and Wen Li. Semantic-aware domain generalized segmentation. In CVPR, 2022.
[52] Xingchao Peng, Qinxun Bai, Xide Xia. Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, pages 1406-1415, 2019.
[53] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In ICML, pages 7728-7738, 2020.
[54] Matco Rojas- Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 1994 2018.
[55] Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. 1JCV, 77(133)157-173, 2008
[56] Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to detect every thing in an open world. In ECCV, 2022.
[57] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cama Visual explanations from deep networks via gradient-based based localization. In ICCV, pages 618 626, 2017.
[58] Sconguk Sco, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In ECCV, pages 68-83, 2020.
[59] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Precthi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross coossggradient training. In ICLR, 2018.
[60] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pages 443 450, 2016.
[61] Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tic- Yan Liu, Recovering latent causal factor for generalization to distributional shifts. NeurlPS, 34, 2021.
[62] Takeshi Teshima and Masashi Sugiyama. Incorporating causal graphical prior knowledge into predictive modeling via simple data augmentation. In UAI, pages 86 96, 2021.
[63] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008
[64] Vladimir Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988-999, 1999.
[65] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 2017.
12

---

[66] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, pages 6438-6447, 2019.
[67] Riccardo Volpi and Vittorio Murino. Addressing model vulnerability to distributional shifts over image transformation sets. In ICCV, pages 7980-7989, 2019.
[68] Riccardo Volpi, Hongseok Namkoong. Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, pages 5334 5334-5344, 2018,
[69] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang. Tao Qin, Wang Lu. Yigiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. TKDE, 2022.
[70] Xiaosong Wang. Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale scalee chest x-ray database and benchmarks on Srakly-superrisee classification and localization of common thorax diseases. In CVPR, pages 2097-2100 2017.
[71] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise- Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al. fine-grained analysis on distribution shift. 2021.
[72] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In CVPR, pages 3485-3492, 2010.
[73] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsuper- vised domain adaptation. In ICML, pages 5419-5428, 2018.
[74] Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yanfeng Wang, and Qi Tian. fourier-based framework for domain generalization. In CVPR, pages 14383-14392, 2021.
[75] Zhenlin Xu, Deyi Liu, Junlin Yang. Colin Raffel, and Mare Niethammer. Robust and generalizahle visual representation learning via random convolutions. In ICLR, 2021.
[76] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pages 6023-6032, 2019.
[77] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR. 2018
[78] Yahin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact feature distribution matching for arbitrary style transfer and domain generalization. In CVPR, 2022.
[79] Gangming Zhao, Quanlong Feng, Chaoqi Chen, Zhen Zhou, and Yizhou Yu. Diagnose like a radiologist: Hybrid neuro-probabiliste reasoning for autribute-based medical image diagnosis. TPAMI, 2021.
[80] Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. NeurlPS, 33: 33:1609666 16066-16177, 2020.
[81] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, TPAMI, 2022.
[82] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI, pages 13025-13032, 2020.
[83] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, pages 561-578, 2020.
[84] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report bars (e.g.. with respect the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [N/A]
(c) Did you include any
either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're 're
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "IoGjQ2WD_Eg", "reviewer": "Reviewer_Rsk7", "summary": "The paper proposes Mix and Reason, a new domain generalization approach consisting of two components. The first component, category-aware data mixing, basically performs data augmentation by replacing the background of an image with a random patch of another image, aided by binary masks produced by Grad-CAM. The second component, adaptive semantic topology refinement, is built using a graph neural network, which aims to learn semantic features through interactions among source domains. The effectiveness of the approach is demonstrated on a number of commonly-used domain generalization benchmarks.", "questions": "- Would it be possible to show some visualization on what the \"semantic topology\" looks like or what was learned by the ASTR module?\n- In Table 7, only ASTR is evaluated. Does CDM also work for the medical datasets?\n- Regarding the use of Grad-CAM, does it always give accurate masks? Are there any failure cases? And what happens if Grad-CAM doesn't produce accurate masks?\n\n-------- Post-rebuttal updates --------\n\nThe authors' responses have resolved all the reviewer's concerns.\n\nAs mentioned in the 1st round review, the reviewer appreciated the contributions and believed the proposed approach and insights will be useful to the community (the reviewer's stance remains the same). The added materials during the rebuttal have significantly strengthened the motivation of the paper and made the story more convincing.\n\nThe reviewer has also read other reviewers' comments as well as the authors' responses, and found no critical issues that could lead to rejection.\n\nOverall, the reviewer likes the contributions of the paper and believes they would be of interest to the domain generalization community. Therefore, the reviewer strongly recommends acceptance and stands firmly on this side.", "limitations": "Currently there is no discussion about the limitations of the approach. Perhaps there are some limitations related to the use of Grad-CAM (see the Questions section).", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: From the technical point of view, the proposed ideas, including the data mixing strategy and the graph neural network-based module, are novel. It's interesting that Grad-CAM can be used in this way for data augmentation.\n\n**Quality**: Overall the quality is good. The main experiments are comprehensive and solid for justifying the technical contributions. The ablation studies are also sufficient. It's worth mentioning that the datasets cover a wide range of tasks including generic object recognition and medical image analysis. However, the paper misses more in-depth analysis on \"how\" the graph neural network-based ASTR module improves the performance. The quantitative results are sufficient to justify the effectiveness but a more scientific analysis is needed to help readers understand how the \"topology\" is built by the neural network and in what way it helps learn structural invariance. In other words, the concept of semantic topology is new and sounds \"big\" but there is no concrete studies on how this concept works. (Maybe use a theory or visualize what was learned by the model?)\n\n**Clarity**: The paper is well-written.\n\n**Significance**: The paper provides new insights on how data augmentation can be combined with graph neural networks to improve domain generalization. Moreover, the results demonstrate that each of the two components alone could be useful to the community: the data mixing strategy works with some existing methods like MixStyle and EFDM, while the graph neural network can be used for medical image analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hpru00A7Uh6", "reviewer": "Reviewer_GJ2F", "summary": "This paper proposes Category-aware Data Mixing (CDM) to augment data on the data level and Adaptive Semantic Topology Refinement (ASTR) to maintain an invariant semantic topology of classes across different domains. The method (combination of CDM and ASTR) achieves excellent results on multiple DG datasets. Besides, experiments also show that CDM can bring performance gains to several other DG methods.", "questions": "Please see weaknesses for my concerns.", "limitations": "The authors didn't address the limitations and potential negative societal impact of their work.\n- It is better to point out potential limitations, e.g., the proposed method (ASTR) can't work when the domain label is unavailable.\n- I didn't see potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths (originality, quality, clarity and significance):\n1. As far as I know, the originality is good.\n2. The proposed method is technically sound. The experimental results compared with other DG methods is excellent.\n3. This paper is generally easy to follow.\n4. The idea of maintaining semantic topology of classes is novel.\n\nWeaknesses:\n1. The standard deviation (std) over 10 runs is missing. Considering most datasets are small, such as PACS and VLCS, it is more convincing to report the average accuracy with std for comparing model performances.\n2. Some opinions in this paper are weak and unconvincing:\n- Domain-wise invariance cannot guarantee generalizable representations. The authors argue that such invariance may be susceptible to including some misleading spurious correlations. This situation may occur in simulated data, where some semantically independent properties exist across all source domains. However, spurious correlation is difficult to hold simultaneously across all source domains in real-world cases, so it is reasonable to expect feature extractors to learn more semantic information through domain-invariant representation learning.\n- The widely-adopted style-content-separation idea may fail to extract true semantic factors. The authors’ basis is that the activation map induced by domain classification does not focus on the background. This phenomenon is easy to understand because the style of foreground objects is also related to domain classification. However, I think that whether domain classification focuses on the background or not has no direct relationship with whether style-content-separation helps extract semantic information.\n3. In Table 1 – Table 6, the authors should report DeepALL results that are implemented by themselves with the same training strategy as the proposed method. It is unfair to compare other methods with the extremely low DeepALL baseline, especially in Table 5. According to my experiments, it is easy for DeepALL to achieve an average accuracy above 82% with the ResNet-18 backbone on the PACS dataset. Experiments on DomainBed also report excellent performances of DeepALL (ERM).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3gj03t8gHcw", "reviewer": "Reviewer_Ywq6", "summary": "This paper introduces a method called MiRe, for Mix and reason over semantic topology, to do domain generalization. The motivation is that existing work typically ignores the semantic information of the inputs and further lack topological structure, which authors claim to be consistent across domains. MiRe consists of two steps: background (other domains) + foreground (current domain) Mixup to generate new samples in different backgrounds, and graph network to mine the neighbor class feature information. Experiments are done in various benchmark datasets and demonstrates the efficacy of MiRe. \n\n=======\n\nPost rebuttal: I really appreciate authors' efforts in providing detailed rebuttal, which resolves most of my concerns. I hope that the future version can include the modifications w.r.t. all comments. But please understand that everyone has a standard for accepting a paper. To me, my main concern is that the novelty is not that \"sparking\" and the method is extremely complicated with so many hyperparameters to tune. Thus, I could not give you high scores. However, I will not stand against rejection. I will give you a 4 accordingly. I'm trying to be nice to you even if my papers still do not get any responses...", "questions": "See above comments.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. The idea of bringing Mixup operation to domain generalization and further enhance its semantic representation is interesting.\n2. The writing is sound, with interesting figures to show the ideas.\n3. The experiments are extensive, with different comparison methods.\n\n### Weakness\n\n#### 1. Overclaims or several claims are not evaluated.\n\nThere are many claims in this paper and I fail to find supports for some of them. Thus, this paper could have some overclaims:\n- In L36, how to evaluate that MiRe does not have the \"data-dependent'' spurious correlations? I fear that MiRe is still a data-driven method and it heavily rely on the Mixup operation of the fore-background.\n- In L42, authors claimed that previous methods require \"some distribution of values for an attribute\". I would assume MiRe does not. However, the adoption of Grad-Cam indeed assumes that the input data is image with clear fore and background information. Thus, it is an overclaim.\n- In L44, I fail to find the answers to the question: \"how many factors of the latent factors\".\n- In L132, why do existing data augmentation approaches lack diversity? Any support?\n\n#### 2. Methodology.\n\n- Technically, the method is not novel. It is a combination of Grad-Cam, Mixup, and GCN. I admire the application of these existing techniques. But there is no further insight and motivation is not strong. Plus, there are the following issues:\n- In figure 3, why only cropping 1/8 of the original image? There lacks motivation.\n- Similarly, in Eq. 4, why adopting $\\operatorname{cos}(f, c_j) / 2$? How did the term $1/2$ come up? This also goes to Eq. 6.\n- The second part of the method is too complicated and computationally expensive for multiple domains. Thus I fear the comparison to other methods is not fair.\n- Regarding the reproducibility, this approach has introduced many extra hyperparameters to be tuned, to name a few, the hyperparameters in Grad-Cam, $\\operatorname{threshold}$ in Eq. 1, $1/8$ in Figure 3, $1/2$ in Eq. 4 and 6, the Mixup hyperparameter in Figure 3, $\\xi$ in Eq. 7, and $\\lambda$ in Eq. 8. Given that there are so many hyperparameters, I highly doubt the reproducibility of this approach.\n- Furthermore, the robustness of the hyperparameters should be reported. Thus, not only accuracy, but also the variation.\n\n#### 3. Experiments.\n\n- The experiments seem extensive. But after a careful check, I found that there is no reason to compare with different SOTA results in different datasets (e.g., the comparison methods in PACS and Office-Home are different). Plus, the model selection strategy is also different for each dataset. There is no support or explanation in the paper to illustrate why this is done.\n- It is unknown where the authors obtain the results for comparison methods for the main paper. Then, if you can use DomainBed, why not just use it as the main benchmark?\n\n#### References\n\nOverall, the references are good. But you can also cite more recent works from the following DG survey articles:\n\n[1] Zhou et al. Domain generalzation in vision: a survey.\n\n[2] Wang et al. Generalizating to unseen domains: a survey on domain generalization.\n\n#### Minor comments\n\nTypos:\n- L26, \"has attract\".\n- L137, \"we propose develop\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept