Adversarial Teacher-Student Representation Learning for Domain Generalization
Authors: Fu-En Yang
Yuan-Chia Cheng
Zu-Yun Shiau
Yu-Chiang Frank Wang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Domain Generalization
Representation Learning
Adversarial Learning

Abstract: Domain generalization (DG) aims to transfer the learning task from a single or multiple source domains to unseen target domains. To extract and leverage the information which exhibits sufficient generalization ability, we propose a simple yet effective approach of Adversarial Teacher-Student Representation Learning, with the goal of deriving the domain generalizable representations via generating and exploring out-of-source data distributions. Our proposed framework advances Teacher-Student learning in an adversarial learning manner, which alternates between knowledge-distillation based representation learning and novel-domain data augmentation. The former progressively updates the teacher network for deriving domain-generalizable representations, while the latter synthesizes data out-of-source yet plausible distributions. Extensive image classification experiments on benchmark datasets in multiple and single source DG settings confirm that, our model exhibits sufficient generalization ability and performs favorably against state-of-the-art DG methods.

Adversarial Teacher-Student Representation Learning for Domain Generalization
Fu-En Yangl.2
Yuan-Chia Cheng'
Zu-Yun Shiau'
Yu-Chiang Frank Wang" 1.2
GGraduate Institute of Communication Engineering, National Taiwan University, Taiwan
ASSUS Intelligent Cloud Services, Taiwan {f07942077, r08942154, r09942069, yewang)
Abstract
Domain generalization (DG) aims transfer the learning task from single or multiple source domains to unseen target domains. To extract and leverage the information which exhibits sufficient generalization ability, we propose simple yet effective approach of Adversarial Seacher-Studeee Representation Learning, with the goal of deriving the domain generalizable representations via generating and exploring out-of-source data distributions. Our proposed framework advances Teacher-Studen learning in an adversarial learning manner, which alternates be- tween knowledge-distilaaiee based representation learning and novel-domain data augmentation. The former progressively updates the teacher network for deriving domain-genenn generaliianaleebee representations, while the latter synthesizes data out-of- source yet plausible distributions. Extensive image classification experiments on benchmark datasets in multiple and single source DG settings confirm that, model exhibits sufficient generalization ability and performs favorably against titte-of-theaale art DG methods.
Introduction
Deep neural networks have achieved promising performance wide variety of tasks. However, these networks assume the training and testing samples fall in the same data distribution. Such strong assumption would limit the applicability of the learned models in real-world scenarios (e.g.. cross-city autonomous driving or multi-cite medical imaging task), which training and testing data are typically observed under different conditions. In other words, the generalizability of the model at unseen target domains might be poor due to unexpected domain shifts. tackle the domain discrepancy, domain generalization (DG) has been proposed and drawn increasing attention recently.
The aim of DG to train models using data observed from single or multiple source domains, while expecting that the model would be generalized to unseen target domains. Most existing DG approaches focus on deriving domain-invariant features [T] among multiple source domains or adopting meta-learning techniques 2 3 5 which would simulate domain shifts during the
meta-training stage. However, the features derived by the above methods are generally guaranteed to be invariant to the seen source domains, not the generalizability of the learned representation to describe unseen domain data. To overcome the limitation, (6] 9 turn to leverage data generation techniques for diversifying the source distributions, and thus avoid overfitting on source domains yet improve the generalization ability of models. Specifically, several works (6] g choose to generate novel-domain images by either perturbing the style of source data to confuse the domain classifier Z7. or transporting the source data to novel styles via optimal transport based objective (8]. [9] adopts Mixup [00 to interpolate the feature statistics between samples from different domains. While the above methods perform well, designing an objective for generating samples with DG guarantees remains challenging and open problem.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recently, self- self--upervised pre-training manifests the potential to derive generalizable representa- tion, which serves as a promising start point for downstream tasks (e.g., image segmentation or object detection). In domain generalization, number of self-supervision techniques have been to improve network transferability by discovering the intrinsic properties adopt jigsaw puzzles as the pretext task, which predicts the relative positions of image patches to constrain the semantic feature learning in a multi-task training fashion. Recently, contrastive learning approaches [14 IS 16] [8] have been proposed and widely applied, which establish the representation learning from multiple views of an image to extract the task-relevant information and discard task isselirteeevant noise. However, the concept of such multiview learning [9 [8] is simply realized by hand-crafted image transformations (e.g.. RandomResizeddCrp. Color Jittering. Gaussian Blur). The effectiveness of these Hand-crafted image transformations for benefiting the generalization to unseen distributions is still not guaranteed.
introduced 11 12
within images. For instance, [ 12
[
In this paper, we propose a unique Adversarial Teacher-Student Representation Learning framework for tackling domain generalized visual classification. Based on the recent success of contrastive learning, we advance the concept of multi-view learning into DG regime for augmenting source instances to out-of-source styles and diversifying training distributions. To be more precise, with the goal of learning representations which are robust to unseen domain shift, we propose to jointly perform Domain Generalized Representation Learning and Novel Domain Augmentation in an adversarial learning manner. Based on Teacher-Studee learning schemes (20 16 21]. our framework utilizes original images as inputs to the teacher network and takes stylized augmentations as input to the student network. To ensure both learning stages produce domain generalized representation, adopt the Teacher-Student co-training scheme, which progressively refines Teacher by the distilled knowledge learned from Student by observing augmented novel-domain data, enabling Teacher be generalizable to data with out-of-source distributions. On the other hand, Adversarial Novel Domain Augmentation aims at augmenting unseen domain data using source-domain training instances. The objective is to maximize the discrepancy between the input and augmented data, derived from the teacher and student modules, respectively. In order to have such augmented data exhibit sufficient domain differences, the above discrepancy will be calculated using features derived from data across different source domains. By iteratively training the above two stages in an adversarial learning fashion, the resulting model (Teacher) would be able to derive domain generalizable representations.
The contributions of this paper are highlighted below:
Different from existing meta-learning based approaches, we propose a teacher-student adversarial learning scheme for addressing domain generalization classification problems.
In the stage of Domain Generalized Representation Learning, the student network observes augmented novel-domain data and distills the information to update the teacher network. allowing derivation of domain generalizable representation.
In the stage of Novel Domain Augmentation, the generator aims at producing unseen yet plausible domain data, which maximizes the discrepancy between augmented and existing domains while the semantic information is preserved.
Evaluations on several benchmark datasets in multiple and single source domain settings verify that our method performs favorably against existing DG approaches and exhibits sufficient domain generalization capability.
2 Related Works
Domain Generalization (DG) Different from domain adaptation (DA), which observes both source
and target-domain training data for performing learning tasks across domains [22 23 24 25 26). DG deals with more realistic yet challenging setting. More precisely, DG aims at generalizing the model trained only on single or multiple source domains to recognize the test instance in unseen but similar target domain. With only source- -omrce-domain data observed during training, a number of features. These methods typically partition source domains into meta-train and meta-test splits to simulate the domain shifts during training. Peature-Critic (27 meta-learns a critic network to evaluate the generalized degree of extracted features for encouraging robust feature derivation. 4 introduces an episodic training that cross-trains domain-specific feature extractors and classifiers to let the learned model invariant to the domain shift. MLDG [2] and MASF [5] both adopt gradient based meta-learning to simulate
works B 2 27 apply meta-learning for learning
2

---

the domain shift, while [3] additionally enforces local and global constraints in meta-training. In addition meta-learning approaches. [T] jointly solves jigsaw puzzle as an auxiliary task with standard classification in multi-task fashion. RSC [28] iteratively discards the dominant features on the training data to improve generalization. Nevertheless, these approaches employ solely limited source domains to derive generalizable features, which still draws a concern about over-fitting on source domains [8 (29] and restricts the generalization ability to unseen domains.
Recent research works consider data generation as an alternative technique for domain generalization, which increase the diversity of training data distribution. To achieve this goal. 6 7 are inspired by adversarial attack [30]. CrossGrad [6] perturbs source data by adding adversarial gradients;
DDAIG 7 learns transformation network that outputs perturbations to confuse the domain classifier. However, such perturbed images do not necessarily exhibit sufficient data domain diversity. In contrast of adding perturbation to images, I.2A-OT g learns conditional generator that transforms images from source distribution to a pseudo-novel distribution by an optimal transport based objective. MixStyle 9 produces image features with mixed feature statistics across source domains. Very recently, PDEN [31] utilizes a progressive learning strategy for single-source domain generalization, which iteratively expands the training data set by adding augmented data. Note that although they adopt contrastive and adversarial learning objectives which are similar to ours, our proposed approach is able to tackle both multi source and single-source DG problems, and also comes with superior memory efficient performance and comparably stable training process.
Self-Supervised Learning (SSL) Self-supervision is a recent paradigm for unsupervised learning. The idea is to design pretext tasks for feature learning to facilitate the downstream task learning. Such auxiliary pretext tasks can be predictions of the image colors (32] relative locations of patches from the same image
16 17 [8] has achieved promising results on network pre-training to learn generalized image features. 19 reveal that the success of contrastive learning typically built on the multi-view perspective. and prove theoretically and experimentally that the compact and robust representations can be learned by deriving the invariance among multiple views of an image. We adapt the above concept of multi-view learning into DG regime. We focus on learning novel-domain data augmentations across source-domain instances in an adversarial training fashion. As detailed and verified later, our proposed learning scheme would produce domain generalizable representation for unseen target-domain data, and performs favorably against state-of-the-art DG approaches.
[T] [2]. and image rotation
Very recently, contrastive learning 14 15
3 Proposed Method
3.1 Problem Formulation and Model Overview
For the sake of completeness, we first define the problem setting and notations used in this paper. We assume that training data are observed from N source domains Der = {D1, D2, ..., Dw J. each of which contains set of image and label pairs D;- = {X,, Y.}. Our goal is to learn model which would exhibit sufficient generalization capability. so that classification of test data in unseen target domains can be performed. In order to derive domaanngenerallzzed feature representations, Adversarial Teacher- Stuuent Representation Learning framework, which is a min-max max deep learning framework alternating between the following stages: domain generalized representation learning (Sec. (.2) and novel domain augmentation (Sec. 3.3). as depicted in Fig. [] For domain generalized representation learning, we learn Nomman-genneallized teacher network (Teacher) Fr with the help from a student network (Student) Fs. which observes synthesized novel-domain augmentation and distills knowledge to Teacher. As for novel-domain augmentation, the novel-domain augmenter G is learned to observe the discrepancy of Seacher-Student encoders, which progressively generates strong novel stylized augmentations to diversify training distributions. Once the learning of the above framework is complete, the teacher network would extract Nomain-generalle network (c.g.. classifier), and thus classification of unseen target-domain data can be performed accordingly. We now detail our proposed learning schemes in the following subsections.
present novel
features for the task
3.2 Teacher-Student Domain Generalized Representation Learning
While techniques based on learning across multiple source domains for DG exist (e.g., using meta- learning techniques like [3 5), it is not clear how the learned model and feature representations
3

---

Domain Generalized Representation Learning
Novel Domain Augmentation
Student
Student
Augmenter
Augmenter
Fs
Leee
G
Fs
Lox
Fixed
Fixed
EMA
backprop.
Minimize
backprop.
Maximize Lib.
Teacher
Teacher
Fi
FT
Fixed
stop grad.
Figure 1: Overview of our Adversarial Teacher-Student Representation Learning scheme, which includes the teacher network Fr, the student network Fs, classifier C, and novel-domain augmenter G. Note that we alternate between the stages of domain generalized representation learning and novel-domain augmentation mutually beneficial manner, resulting in discriminative yet domain generalized representations.
would be generalized to unseen target domains. Instead of directly fitting models across source domains, we propose Domain Generalized Representation Learning based on the Teacher-Student learning scheme, with the goal of extracting domain generalizable feature representations. To ensure our teacher encoder to gain generalizability by observing out-of-source domain information, we deploy Student Fs for exploring novel-domain augmentation synthesized from the novel-domain augmenter G, while distilling the associated knowledge to update Fr.
To address this representation learning task, we first train the teacher module together with single- layer classifier C' using multiple source-domain data. The standard cross-entropy loss Loe is utilized to initialize Fr warm-up. As illustrated in Fig. [ we then input training images sampled from the source domains into the novel-domain augmenter G (detailed in the following sub-section), producing the style (or domain) perturbed augmentation 인 yet preserving its semantic information. While such a domain augmented would fed into the student module resulting in feature 2 = F we also feed the original input into Teacher to derive semantic information as z does, we particularly propose an objective to minimize the discrepancy between ž and z. To be more specific, we define the discrepancy loss Lis to minimize the distance between the normalized features Σ and z:
(zr). To ensure that would contain the same
(x -1
2
Fs( 2
min (2, Fs
2 ||2||2
|| - [Fr(2))]
In addition, calculate the cross-entropy loss on the domain- aamniin-augmented feature i.c., Lee (C(2), y). which further enforces the classification capability of the student module (note that C' indicates the single-layer classifier, and y denotes the corresponding class label). We note that, this representation learning stage, only the student network Fs is updated by the above two objectives C (2, 2) and Lee (C y), and we apply stop-grade gradient strategy forbid Fr and G from being updated by gradients. Thus, optimization of Fs with learning rate 7 can be expressed as follows:
0g - 0s a dis (2, 8) + Lee Lee (C(2),)))
00s
As for the teacher network Fr, we adopt exponential moving average (EMA) 20 16 progressively refine the associated model parameter 0r. That is, the learned knowledge from Student's parameter Os is distilled to update Gy follows,
21 to
0r - 70, +1 (1-7)0s, where [0, 1),
Note that controls the updates on the teacher network. Finally, it is also worth pointing out that, such refinement strategy would avoid the teacher module from directly observing unrealistic domain augmentations, which might degrade its domain generalization capability.

---

3.3 Adversarial Novel Domain Augmentation
To motivate the student network to explore sufficient diversity of domain augmentation, we present an adversarial learning scheme, which would progressively perform novel-domain data augmentation in our proposed framework. Inspired by both adversarial learning strategy [3]] and multiview learning from SSL
formulate novel-domain augmentation stage together with representation learning (Sec. 3.2 into an adversarial learning framework. As depicted in the right hand side of Fig. [T] aim at training the novel-domain augmenter G and freezing both Fr. and Fs, while the discrepancy between 2 and & serves as the adversarial guidance. That is, when the above discrepancy is small (i.e., the outputs of Teacher and Student are similar), it implies that such domain augmentations have been seen by existing source-domain data. To encourage more the augmented data to be sufficiently distinct in terms of domain information, train our novel-domain augmenter by maximizing the discrepancy as follows,
[19 18).
m] = Fr(x) [+)]] [[Fs(G())]]
Fs(G(z))
max LC dia (2 (2, 3) = 1 2 2
- m) (4)
G
where [ min(- 0), and the margin m can either be calculated by the means/centroids of data from each source domain in mini-match, followed by averaging the L2 distances between the above centroid pairs, or simply viewed as a hyperparameter. It is worth pointing out that, this margin serves as regularization observed from the separation between existing source domains. Thus, it reflects the desirable domain gap between the augmented and existing domain data.
To guarantee the produced domain augmentations preserve the original categorical content, we still observe the cross-entropy loss Lee ( (C(3). y) with regard to C()) and the corresponding label y. Thus, optimization of G' can be performed as follows,
O C (2, (2, + Leo (C(E))
0g 8g 0g
(5)
08,
Note that, we only pretrain the classifier using source domain data available, and keep it fixed during the learning of our teacher-student augmentation framework. If we allow the update of this classifier during the training process, it might observe undesirable outputs and affect the learning of both augmenter and teacher/student modules in the early training stage, where either the augmented data or its extracted features is not yet quality.
Once the learning of the proposed framework is complete (i.e., alternative optimization between the two stages), we deploy the derived domain generalized Teacher to extract discriminative and transferable features, that classification of unseen target domain can be performed accordingly. The pseudo code our Adversarial Teacher-Studer Representation Learning is summarized in Algorithm in supplementary material.
Experiments
4.1 Datasets and Evaluation Protocol
is composed of four data domains, Photo, Art painting, Cartoon and Sketch, with diverse image colors and styles. Each domain contains categories, with 9991 images in total. Following the experimental protocol proposed by (35), images from source domains are divided into the training split and the validation split, at ratio of about 9:1. Office-Home [36] comprised of four domains, Art, Clipart, Product and Real world, and exists larger label sets of 65 categories, with about 15500 images in total. The dataset contains images of everyday objects with various styles, backgrounds and camera viewpoints. Images are divided into the training split and the validation split at a ratio of about 9:1. DomainNet [37] is a recently proposed large-scale dataset which consists 0.6 million images of 345 classes distributed across 6 domains, Real, Clipart, Infograph, Painting. Quickdraw and Sketch. We follow the training and testing splits for all the domains released by [37]. Also, for the single source DG experiments, we follow (38) and partition the training split from (37 into the training and validation splits at ratio of 9:1. Due to page limitation, we additionally provide quantitative comparisons on VLCS [39] and DD [7] datasets in the supplementary material.
Datasets We evaluate our method on several public benchmark datasets. PACS 35
5

---

Table I: Comparisons to con--ata-generaaan based methods PACS using ResNet- 18 in leave-one-
domain-out settings. Bold denotes the best result.
Target
(buselinc)
DeepAll
AAE [1]
MMD-
MLDG
[2]
RGen
[11
MenaKeg
[3]
FCR [4]
Epi-
MASF
[3]
EISNat
[12
DMG [38]
Barlino st [44]
DSON [55]
RSC [28
Ours
Photo
95.6
96.0
96.1
95.3
93.9
95.0
95.9
93.4
95.0
55.9
96.0
97.3 1 0.3
painting
83.7
82.7
Cartoon
24.2
72.7
77.2
75.3
77.2
77.0
77.2
76.4
80.4
78.0
77.7
80.3
80.7
= 0.5
Skatch
68.4
64.2
72.3
71.4
70.3
73.0
71.7
74.3
75.2
81.6
82.2
80.9
77.3 4 0.5
78.3
77.0
81.8
81.7
81.5
81.
822
81.5
84.3
85.1
85.2
833
Table 2: Comparisons non-data- generrttinnnn based methods Office- Home using ResNet-18 18 in
leave-one-done cone-domain-ou settings. Bold denotes the best result.
DeepAll
CCSA
MMD-
MLDG
D-SAM
JiGen
Borlino et (44
DSON
RSC
Ours
Target
(baseline)
[46]
AAE
[2]
[47
11
455
22
Art
59.0
59.9
56.5
58.1
58.0
53.0
58.7
59.4
58.4
60.7 + 0.5
Clipart
48.4
49.9
47.3
49.3
44.4
47.5
52.3
45.7
47.9
52.9
+ 0.3
Product
72.5
72.1
72.9
69.2
71.5
73.0
71.6
Real world
75.5
75.7
74.8
74.7
71.5
72.8
75.0
74.7
74.5
77.2 + 0.2
Average
63.9
64.9
62.7
63.8
60.8
61.2
64.8
62.9
63.1
66.7
Evaluation Protocol For fair comparison purposes, we follow the leave-one-domain- protocol for our experiments. That is, one data domain from a dataset is selected as the target unseen domain to be recognized, and the remaining ones as the source domains for training. And, we report the top-1 classification aceuracy (%) for quantitative evaluation.
as considered in о 12
4.2 Implementation Details
For PACS, Office-Home, and DomainNet, input images are resized to 224 x 224 pixels, and we use ResNet-18 18 and ResNet-50 [40] pre-trained on ImageNet [41] as the backbones of our teacher and student networks. Fs is trained with the SGD optimizer, with an initial learning rate of 0.0005, and a batch size of for 60 epochs. The learning rate is decayed by 0.1 after 30 epochs. Fr is updated via EMA with the momentum coefficient of 0.999 by default. Our novel-domain augmenter G is realized by fully convolutional network similar to the generator's architecture in 7 and trained with the SGD optimizer. In the warm-up phase, we train Fr together with the classifier C' using only source data with the SGD optimizer, and then the parameters of C are fixed in the following training process. Note that we also use the official implementation from [1] 6 7 42 8 9 for our comparisons. all our experiments, we implement our model using Py and Dassi. pytoreh 43 toolbox, and conduct training single NVIDIA TESLA V100 GPU with 32 GB memory.
4.3 Quantitative Evaluation
We first perform domain- domain--eneralize generalized visual classification tasks and compare our results with existing non-data- non-data-generation genrrateennnnnnn [ [46] [I 3 4 52 38| 44, 45 28) and
based 66 7 8, methods on two commonly- used public benchmarks, PACS and Office-Home. In our experiments, DeepAll is viewed as a baseline, in which both feature extractor and classifier are trained on data aggregated from all source domains.
Tables [ and 2 summarize the quantitative comparisons with existing con-data-generuan based 3
45 28 on PACS and Office-Home (ResNet-18 18 as the backbone), respectively. Particularly, Epi-FCR and MASF [ are meta-learning approaches which either adopt episodic training scheme that cross-train encoders and classifiers from different domains, or employ gradient-base baased optimization strategy with global and local losses for regularizing the model training. JiGen (IT) and EISNet [12] both consider solving jigsaw puzzles as the auxiliary task for better capturing spatial information. Recent start-of of-the-rrt method RSC (28) iteratively dropouts the most contributing features force models explore the remaining features that correlate with semantic information. As we can observe from Table []] our approach achieved the best performance worth noting that, significant gap in visual appearance can
methods 46 2
4 5
12
38
44
on Photo, Art paining and Cartoon. be seen between Sketch and other image domains, which makes the associated domain generalization more difficult. Nevertheless, our approach still achieved satisfactory results over the state-of-ther -tt methods on Sketch, and reported the highest average accuracy of 85.3% On the other hand, Table 2 demonstrates that our method performed favorably on all the domains (i.e., 60.7% on Art, 52.9% on Clipart, 75,8% on Product, and 77.2% Real world), and thus achieves the highest average

---

Table 3: Comparisons to data-gentn getrraatenrtnnnnn based methods PACS using ResNet in leave-one-domaie out settings. Bold denotes the best result.
ResNer-18
ResNev-S0 DIATO
Targel
DeepAll (baseline)
CrosSCrad
DDAIG
L2A-0T
MinStple
Ours
DeepAll (baseline)
CrossCrrd
MisSyyle
Ours
(7)
[3]
[9
[7]
[
95.6
Art painting
75.1
79.8
84.2
83.3
84.1
85.8
0.6
81.5
87.5
85.4
87.4
90.0
0.3
Cartoon
74.2
76.8
78.1
78.2
78.8
80.7
0.5
78.6
80.7
78.5
83.3
83.5
0.5
Skesch
68.4
70.2
74.7
73.6
75.9
77.3 0.5
69.7
73.9
80.0
78.5
80.0
0.6
Average
78.3
80.7
83.1
828
83.7
85.3
812
85.7
84.
88.1
Table 4: Comparisons to data iatta-geneattion based methods on Office-Home using ResNet in leave-one- domain-out settings, Bold denotes the best result.
Target
DeepAll (haseline)
CrossGrad
DDAIG
L.2A-0T
MixSSyle
Ours
DeepAll (haseline)
CrossGrad 6
DDAIG
MisStyle
Ours
6
[7]
[8]
[
[7]
[
390
Clipart
48.4
49.4
52.3
50.1
53.4
$2.9
58.8
57.7
59.2
58.8
60.1 0.5
Product
72.5
73.9
74.6
74,8
74,2
75.8 1: 0.1
77.9
79.1
77.7
78.3
81.5
1.4
Real world
75.5
75.8
76.0
73.0
75.9
77.2
+
79.0
80.4
76.7
78.7
82.1 +.2
Average
639
64.4
65.5
65.6
70.1
69.7
70.2
73.3
accuracy 66.7%. The above quantitative comparisons verify that, comparing to directly (meta-)learn from existing source domain data, our approach for augmenting diverse, novel, yet semantically practical source-domain training data would be preferable in domain generalization tasks.
With the above observation, we further compare our method with the state-of-the- art data-g nata-generation based models 66 7 18 9 using ResNet- 18 and ResNet-50 as backbones. As shown in Table 3 our approach consistently performed superiorly against the method of (9] by 1.6% and 1.3% on PACS with ResNet-18 and ResNet-50 backbones, respectively. Table 4 presents the results Office- Home, which shows that our method would be preferable among the DG methods considered. Also, the above results demonstrate that our proposed framework able achieve general preferable performances regardless of the backbone choice. is worth noting that, CrossGrad [6] and DDAIG [ add perturbation input images, which might represent the domain variations, and the data generation processes of L2A-OT g do not jointly take the representation learning into consideration. Also, MixStyle can only produce image features with interpolated domain styles. Different from these methods, our approach learns to synthesize out-of distribution augmentations and derive domain generalized representations in mutually beneficial manner, hence exhibiting more robust generalization capability.
4.4 Analysis of Our Method
4.4.1
Ablation Study
We now conduct the ablation study to verify our network design on PACS with ResNet-50 backbone, and we list the results in Table[5] Also, we evaluate the effectiveness of Jigsaw puzzle. Such spatial transformation has been applied in several DG works [2]. In the bottom part of Table 5 we consider different network designs, including Siamese architecture, Student without EMA, and Student with EMA, to be derived for performing on unseen target domains.
Effectiveness of Adversarial Augmenter In the upper part of this table, we first verify the effec- tiveness of our designed novel-domain augmenter G by replacing G with different types of data augmentation strategies Random Augmentation and Jigsaw puzzle. Random Augmentation denotes directly performing hand-crafted image transformations, including RandomResizedCrop. Color Jit- tering, Gaussian Blur, Rand Augment and Color Dropping. From Table 5) it can be observed that our model surpassed other controlled versions and the baseline on all four domains. We notice that replacing our learnable novel-doman augmenter with hand-crafted random augmentations results in significant performance drops, and the performance was just marginally better than that of the baseline (i.e., DeepAll). This verifies that such random image transformations can merely achieve limited improvement on generalization capability. Although the average accuracy of Jigsaw Puzzle was better than that of Random Augmentation by about 2.7%, it was still worse than that of our full version by about 2.6%. This is possibly because that, while Jigsaw Puzzle provides more visual clues about spatial information as stated in [1] 12), there is guarantee that such image transformation
7

---

Table 5: Ablation studies PACS using ResNet-50 as the backbone.
Module
Method
Photo
Art painting
Cartoon
Sketch
Average
DeepAll
94.8
81.5
78.6
69.7
81.2
Augmentation
Random Aug.
96.4
83.2
75.9
75.5
82.8
Jigsaw puzzle
97.1
79.0
80.5
85.5
Stamese archi.
98.3
87.5
82.7
74.5
Representation
Fs w/o EMA
98.2
74.7
84.9
Pg w/ EMA
97.9
88.9
82.0
75.1
86.0
Ours (G + Ff)
98.9
90.0
83.5
80.0
88.1
75-
40-
20-
-50
-0
-75
-50
100
(a) DeepAll
(b) Our method
Figure 2: t-SNE visualization on PACS with Photo as the unseen target domain. (a) Representations
extracted by the baseline approach of DeepAll. (b) Representations derived by our approach.
would contribute to domain invariance. With the above experiments, our learnable novel-domain augmenter exhibits sufficient ability to generate novel-domain augmentations for facilitating the model robustness to unseen domains.
Effectiveness of Domain Generalized Teacher From the results shown in the lower half of Table we see that the performance dropped when we replace the Stuuher-Student scheme by a Siamese Architecture, where the parameters are shared between the teacher and student networks. This is due the fact that the Siamese architecture is prone to output collapsing solutions, hampering the derivation of domain generalized representations. In addition, we examine the performance of applying the trained student network to unseen domains instead of applying Teacher. Studens withour EMA denotes that Teacher is fixed during training, while Student with EMA denotes that Teacher still updated with EMA which benefits the learning of Student. We observe that adopting EMA achieved the better results, but the performance of the above two versions (which apply Student) were still inferior to ours (which applies Teacher). From the above results, we confirm that Teacher updated with EMA would be less likely to be affected by possibly unrealistic domain augmentations during training, avoiding the degradation its domain generalization capability. As verified by the above experiments, all components presented in our learning scheme would contribute to the domain generalization capability.
4.4.2 Visualization
We now qualitatively assess the ability of our approach in deriving domain generalizable features. As shown in Fig. 2 we apply t-SNE to compare the features derived by our teacher network Fr with the features extracted by DeepAll network on PACS. In this figure, while the source image features extracted by DeepAll can be grouped according to their semantic categories, the target-domain features still cannot be properly separated. It can be observed that both source and target-domain features derived by our Teacher sufficiently aligned, and the distances between different class clusters are representation
more evident, indicating that equipped with our proposed adversarial learning, our model is capable of learning more discriminative yet domain generalizable features.
Moreover, Fig. 3 we visually compare the synthesized images by our method and those by the state-of-the-arr data-generation method of DDAIG (7] using PACS as the training dataset. As described Sec. learns to perturb the input images for confusing the domain classifier, with the goal of producing output images to be domain-agnosiin However, from Fig. 3 that images

---

Domain
Input
DDAIG
Ours
Domain
Input
DDAIG
Ours
Photo
Art painting
Cartoon
Sketch
Figure 3: Visual comparisons of augmented novel-domain images produced by DDAIG and ours on PACS dataset.
Table 6: Impact of momentum coefficient on Office- Home using ResNet--50 the backbone.
Art
Clipart
Product
Real world
Average
0.9
66.5
56.2
78.9
80.9
70.6
68.1
56.9
80.1
81.4
71.6
0.999
69.3
60.1
81.5
82.1
73.3
generated by DDAIG 7 tended to exhibit visual perturbation, which might not correspond to domain variations. On the contrary. our approach was capable of producing images the data domains which are visually realistic yet distinct from source domains. We also note that, our model is trained in deterministic manner, and the two augmented outputs are generated from our augmenter learned at different time steps with distinct mini-batch data sampled. This supports that our novel-domain augmentation mechanism is able expand the training distributions.
4.4.3 Impact of the Momentum Coefficient
In exponential moving average (EMA), is momentum coefficient to control the update degree of our teacher network Fr. As shown in Table 6 we conducted ablation studies on Office-Home with ResNet-50 backbone and observed that large momentum coefficient smoothly refining 0r could achieve better performance than by rapid updating. These results indicate that a smooth refinement of Teacher avoids the degradation of generalization capability.
4.5 Generalization from A Single Source Domain
We evaluate our method on a more challenging DG task, single source domain generalization, to further verify the effectiveness of our method. In the single source DG setting, we only observe training data from single source domain during training with the aim of generalizing to multiple unseen domains. To confirm that our approach can be extended the single source DG setting, we conduct experiments on PACS and the large-scale benchmark dataset DomainNet with the ResNet-50 backbone. For PACS, we select Photo as the source domain and the remaining ones (i.e., Art painting, Cartoon, and Sketch) as the target domains. On the other hand, Real domain in DomainNet is chosen as the source domain, while Clipart, Infograph, Painting. Quickdraw, and Sketch domains serve as the target domains. We note that, since only single source domain is observed during training, the margin m in 44 is viewed as a hyperparameter instead calculating from source domain data. Due to page limitation, additional experiments on PACS using Arl painting, Carloon, and Sketch as the single source domains are presented in the supplementary material.
We provide quantitative comparisons with the baseline (DeepAll), JiGen (I]. and other three data- generation based methods 6 7 42
evaluate the generalization capability on this challenging setting. As shown in Table 7 our approach performed favorably against the baseline (DeepAll) worth noting that, compared with data- ceta-generation based methods of [6] 77, 42], our approach was able to achieve superior accuracy on all the target domains of interest. This confirms that, while our method can also be viewed as data-generation based approach, we are able to better augment novel-domain data based on the observation of single source domain data. From the above experiments, the use of our approach for single source domain generalization tasks can be successfully verified.
and the above DG methods on both benchmark datasets.

---

Table 7: Single-souse domain generalization PACS and DomainNet using ResNet-50 as the backbone. Note that Photo PACS and Real of DomainNet are selected as the single source domain for training.
Method
PACS
DomainNet
painting
Average
Cliputt
Infograph
Paintin
Quickdraw
Average
DecpAIl
60.7
23.5
29.0
57.7
34.5
15.7
40.7
3.6
25.9
24.1
JiGien 11
63.6
28.5
30.2
40.8
50.0
19.0
46.3
7.2
35.5
31.6
CrossGrad
DDAIG
64.1
32.5
29.6
42.1
41.4
16.5
40.9
3.2
26.7
25.7
M-ADA
42
64.5
34.6
26.6
41.9
50.3
19.5
48.1
7.1
36.0
32.2
68.2
36.3
3355
46.0
52.2
21.6
0.3
38.3
34.1
Conclusion
In this paper, we proposed Adversarial Mencher-Student Representation Learning for addressing domain generalization classification tasks. By alternating between the training stages of Teacher- Student representation learning and novel-domain augmentation an adversarial manner, our learning scheme allows derivation of domain generalizable representations while semantic information properly preserved. We conduct extensive experiments, including comparisons to state-of-the-art meta-learning and data-generation based DG methods and ablation studies, which quantitatively and qualitatively confirm the effectiveness and robustness of our proposed approach in solving DG classification by training single or multiple source-domain data.
This work is supported in part by the Ministry of Science and Technology of Taiwan under grants MOST 110-2634-F-002--33 and 110-2221-E-02-122
Acknowledgement
References
[1] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5400-5409, 2018.
[2] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning generalize: Meta- learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[3] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-reggularization. Advances in Neural Information Processing Systems, 11:998-1008, 2018
[4] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1446-1455, 2019.
[5] Qi Dou, Daniel C Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning semantic features. arXiv preprint 1rXiv:1910.13580. 2019.
[6] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri. Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- grossggradient training. arXiv preprint arXiv: 10745, 2018.
[7] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang, Deep domain-adversart adversarial image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13025-13032, 2020.
[8] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference on Computer Vision, pages 561-578. Springer, 2020.
[9] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008, 2021.
10

---

[10] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411 2017.
[11] Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2229-2238 2019.
[12] Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In European Conference on Computer Vision, pages 159-176. Springer, 2020.
[13] Silvia Bucci, Antonio D' D'Innocente, Yujun Liao, Fabio Maria Carlucci, Barbara Caputo, and Tatiana Tommasi. Self-supervised learning across domains. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[14] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6707-6717, 202..
[15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 155971607. PMLR, 2020.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[17] Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-super learning. arXiv preprint arXiv:2006.07733, 2020.
[18] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[19] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency, Self- supervised learning from a multi-view perspective. arXiv preprint arXiv:2006. 05576, 2020.
[20] Antti Tarvainen and Harri Valpola. Mcan teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. arXiv preprint 17X1v:1703.01780, 2017.
[21] Yen- Yen-Cheng Liu, Chih- Chin-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi- semi-supervised object detection. arXiv preprint (xXi::2022.0948) 2021.
[22] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation.
In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[23] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. arXiv preprint
2016.
[24] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired saaageto-immge translation using sycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232. 2017.
[25] Judy Hoffman, Eric Tzeng, Taesung Park. Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018.
[26] Yun-Chun Chen, Yen- Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang. Crdoco: Pixel-level domain transfer with cross-domain consistency. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1791-1800, 2019.

---

[27] Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales, Feature-criti networks for heterogeneous domain generalization. arXiv preprint arXiv:1901.11444 2019.
[28] Zeyi Huang, Haohan Wang. Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. arXiv preprint arXiv:2000.02454, 2020.
[29] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization:
A survey, arXiv preprint arXiv:2103.02550, 2021.
[30] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9))2805- 2824, 2019.
[31] Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
224 233, 2021.
[32] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European conference computer vision, pages 577-593, Springer, 2016.
[33] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803. 2018.
[34] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 160.v11406.2661, 2014.
[35] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pages
2017.
[36] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[37] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406-1415, 2019.
[38] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffmmnn. Learning to balance specificity and invariance for in and out of domain generalization. European Conference on Computer Vision, pages 901-318. Springer, 2020.
[39] Chen Fang. Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[40] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[41] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 245-255. leee, 2009.
[42] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 12556-12566 2020.
[43] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. arXiv preprint arXiv:2003.07325, 2020.
12

---

[44] Francesco Cappio Borlino, Antonio D'Innocente, and Tatiana Tommasi. Rethinking domain generalization baselines. In 2020 25th International Conference Pattern Recognition (ICPR), pages 9227-9233. IEEE, 2021.
[45] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to opti- mize domain specific normalization for domain generalization. arXiv preprint 1rXiv:1907.04275, 3(6):7, 2019.
[46] Saeid Motiian, Marco Piccirilli, Donald Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 5715-5725, 2017.
[47] Antonio I'noncente and Barbara Caputo. Domain generalization with domain-speciffc ag- gregation modules. In German Conference on Pattern Recognition, pages 187-198. Springer, 2018.
13


Reviews:
[{"review_id": "vX7nbzWodCE", "reviewer": "Reviewer_5PA5", "summary": "The paper studies the problem of Domain Generalization (DG), where a model trained on single or multiple source distributions / domains is expected to generalize to data from unseen / novel distributions at test-time. The authors propose an adversarial teacher-student representation learning setup consisting of two stages -- (1) synthesizing “plausible” out-of-domain instances as augmented source data (termed Novel Domain Augmentation) and (2) enforcing the representations obtained from augmented and vanilla source domain instances to be similar and relevant to the task at hand (termed Domain Generalized Representation Learning). This is achieved via teacher (fed regular instances) and student (fed augmented instance) encoders and a novel data augmentor unit. In the second stage, the student encoder is incentivized to be good at the downstream task and ensure representations from augmented samples are similar to the vanilla ones (the teacher parameters are kept up to date via an EMA step). To generate novel augmented samples, the augmentor unit is incentivized to maximize the discrepancy between the representations obtained from the teacher and student encoders. Broadly, the proposed approach falls in the category of DG relying on hallucinating reasonable out-of-domain samples (without assuming any access to domain labels) during training. Obtained results indicate that the proposed approach is competitive with existing DG approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will now highlight the strengths and weaknesses of the submission, which will address quality, clarity, originality and significance of the submission.\n\nWith the exception of the points covered under weaknesses, I believe the following are the strengths of the submission in its current form:\n\n- The paper is generally well-written and easy to follow. With the exception of a few points under weaknesses regarding clarity issues, the authors do a decent job of motivating several components of the proposed approach -- the multi-view perspective on Domain Generalization, the Adversarial Teacher Student setup and other design choices.\n\n- From a modelling standpoint, perhaps one of the biggest strengths of the paper lies in the fact that it does not rely on domain-labels -- making it useful for not only the multi-source but also the single-source domain generalization (DG) setup. More importantly, there is value on relying on DG approaches agnostic to domain labels especially for settings where access to domain labels may be a restrictive assumption (due to annotation cost / lack of clarity in terms of what constitutes different domains -- for instance, datasets like CARLA, CityScapes, etc.).\n\n- The proposed approach seems simple, intuitive and results in competitive (on PACS) and improved (on Office-Home) multi-source DG performance. In terms of single-source DG performance, the proposed approach seems to offer improvements on both PACS and DomainNet (under the experimental settings considered; see weaknesses for more details.). In addition to competitive quantitative results, the ablations in section 4.4.1 demonstrate the utility of different components in the proposed approach.\n\n- The simplicity, intuitiveness and consistent improvements offered by the proposed approach are likely going to make the submission useful for future work to build on top of (provided some of the experimental comparisons are exhaustive; see weaknesses). Overall, I believe this, combined with the lack of reliance on domain labels generally makes the approach valuable. \n\nI will now highlight major weaknesses associated with the draft below. These are centered mostly around some clarity issues and lack of some experimental comparisons.\n\n - L178-179 states that the margin m (an important component in the proposed approach) in the objective used to update the novel domain augmentor is “calculated by the L2 distance between the means of the sampled source domains”. It’s unclear to me what this means -- is this the average L2 distance b/w the means across all possible pairs of source domains in the mini-batch? If not, could the authors comment on how exactly this is computed for the multi-source case? Similarly, in the single source generalization setting, m is treated explicitly as a hyper-parameter. However, it’s unclear how the best m is chosen for the single source experiments. Is the best m chosen based on target domain performance? If not, could the authors comment on how this is chosen? \n\n- Right now, there’s only one line in the paper (and supplementary) detailing how the novel-domain augmentor is realized (as an image translation FCN network) and is hard to locate. The paper would benefit from discussing the architecture and pre-training details (if any) of G earlier section 3. Did the authors experiment with other generators as choices for the augmentor design? Including a discussion about the choice of augmentor architecture (image translation networks vs other plausible generators) would further benefit the paper. \n\n- While (a) Table 5 clearly highlights the progressive improvements offered by the novel domain augmentor + the teacher-student (EMA) representation learning scheme over RandAugment and JigSaw and (b) Tables 3 & 4 highlight improvements over other data-generation DG approaches, the paper would benefit from discussing recent work [A] (evaluated solely on Single Domain Generalization, but still applicable here) in related work -- particularly since the broad components of the proposed approach in [A] and the current submission seem similar, including plausible O.O.D. instance generation & ensuring similarity across representations from augmented and non-augmented samples. This will further help highlighting the novelty and benefits of the proposed approach over [A].\n\n- Since the proposed approach does not rely on domain labels, single domain generalization benchmarks are somewhat best suited to assess the utility of the proposed approach against other existing single DG approaches. While the authors conduct single DG experiments on PACS and DomainNet (more exhaustive on PACS, as highlighted in the supplementary) and compare with M-ADA, the paper would benefit from including comparisons with [A,B] (which outperform M-ADA) either on PACS / DomainNet or on the benchmarks used [A,B] -- CIFAR-10C or DIGITS or the semantic segmentation benchmarks.\n\n- [Minor Points] -- It’s unclear what L63-64 is trying to convey in the introduction, revising the same would improve the paper. It’s unclear what L228-229 is trying to convey. Not a major point, but I think the opening line of the abstract should be repurposed to replace “task” with “models trained for a task”, it seemed confusing otherwise. The current draft would benefit from a little proofreading to address such minor issues.\n\n\n[A] - Progressive Domain Expansion Network for Single Domain Generalization\n\n[B] - Uncertainty-guided Model Generalization to Unseen Domains\n\n**Post-Rebuttal Update**\nMy primary concerns about the paper (centered mostly around clarity and experimental comparisons) have been satisfactorily addressed. I also appreciate the detailed responses to the concerns raised by other reviewers. Given this, I am increasing my rating of the paper to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The strengths and weaknesses highlighted in the review form the basis of my rating. I think addressing the points highlighted under weaknesses, especially the clarification and experimental concerns will help improve the paper. I would encourage the authors to do the same.\n\nI don't see any potential negative social impacts of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "VfDG6idxKQW", "reviewer": "Reviewer_3Rjy", "summary": "The article proposes an approach for domain generalization (DG) based on adversarial augmentation and knowledge distillation. In particular, the model contains three components: a teacher, a student, and an augmenter. The student receives input processed by the augmenter while the teacher the original data, and a discrepancy loss measures the difference between student and teacher features for a given sample. Training is held out in two cyclic stages: in the first, the augmenter is fixed and the student parameters are learned by minimizing the semantic and discrepancy loss, with the teacher being an exponential moving average of the student. In the second stage, student and teacher are fixed and the augmenter is updated by minimizing the semantic classification loss and maximizing the discrepancy one. The teacher is used at test time. Experiments on a variety of benchmarks show the efficacy of the proposed approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall:\nThe paper proposes a relatively simple yet effective approach for domain generalization, merging the strength of Mean-Teacher [20] with adversarial data augmentation [8], showing non-marginal improvements over the state of the art in multiple datasets and in both single and multi-source domain generalization. Since both distillation and augmentation strategies are new in the context of domain generalization and I did not spot any major weakness I lean toward the acceptance of the manuscript. Below I detail my comments.\n\nStrengths:\n+ Exploiting the consistency between the teacher and the student predictions to train the data augmentation network is interesting since 1) standard DG approaches for domain generalization use a domain classifier (e.g. [7,8]); 2) does not require any domain information; 3) exploits the same network used to predict the semantic of the input, being computationally less demanding.\n\n+ Not many multi-source DG models show to be effective in the single-source DG task (one example is [28]). The proposed approach outperforms existing generative and non-generative approaches in both settings, sometimes by a margin (e.g. +~2% on the challenging single-source DG in DomainNet, Table 2 and +1.2% on multi-source DG in Office-Home, Table 4).\n\n+ The ablation study (Table 5) thoroughly analyzes the impact of each component and design choice.\n\nWeaknesses:\n\n1. The article does not fully specify some of the methodological components, even if it is easy to understand their role. In particular, the classification loss is indicated in Figure 1 but briefly mentioned in the text (line 183).\n\n2. The organization of the experiments can be improved. For instance, multiple tables are used for the comparison with the state of the art, containing redundant results, with the main difference being on the competitors (e.g. Table 1-3 and Table 2-4). It would be good to keep one table for each tested dataset while specifying which methods are generative and which are not. The saved space can be used to either integrate other analyses from the supplementary (e.g. B3, C) or add more qualitative augmentation results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations are discussed as well as possible negative social impacts. While I deem the latter to not be present, it would have been important to discuss the limits of the work in terms of, e.g. variety of augmentations produced (Figure 3 shows mostly color-based transformations) and future improvements of the single components.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LKtsmFQwXv3", "reviewer": "Reviewer_kqaD", "summary": "This paper proposes Adversarial Teacher-Student Representation Learning to learn domain invariant features for domain generalization. The method learns teacher-student co-training and novel-domain data augmentation in an adversarial manner. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to understand.\n\nAuthors proposes the data augmentation to generate synthetic data for domain generalization problem. Such data augmentation is used in domain adaptation and few shot classification often and it is interesting that authors incorporate this idea with domain generations. \n\nThe formulation of minimizing the data discrepancy when training the teacher-student network and maximizing the data discrepancy when updating the data augmenter. Authors evaluate their methods on multiple domain adaptation dataset collections with different network backbones. For the data augmentation network G, authors proposes to augment the input image which requires a large augmentation network. But augmenting extracted features instead of the input image is widely used in data hallucination [A], [B]. It is unclear of the reason why authors choose to augment the image.\n\nThe main experiments shows the proposed methods achieve better average acc in domain generalization. Ablation studies show the effectiveness of adversarial augmenter and domain generalized teacher. Visualization provides some generated images. It is shown that the data augmenter generates re-styled images based on the input. The re-styling is visually similar to the re-coloring instead of content preserved style transfer [C]. I am not sure with this kind of data augmentation whether the augmenter succeeds in generating out-of-distribution data and benefits the generalization.\n\nSome questions:\n1. Why do you choose to augment the image rather than features?\n2. Instead of re-coloring image, is there any other augmentation generated by the augmentor?\n3. Is there any noise as input the augmentor in order to generate multiple outputs with the same image?\n\n[A] Wang, Yu-Xiong, et al. \"Low-shot learning from imaginary data.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[B] Pahde, Frederik, et al. \"Discriminative hallucination for multi-modal few-shot learning.\" 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018.\n \n[C] Huang, Xun, et al. \"Multimodal unsupervised image-to-image translation.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\nUpdate: Thanks for the authors' response. It addresses my concerns described above and explains very clearly the difference between FSL hallucinations and DG's. I think it is good to include such discussion in the final version, either the main paper or supplementary. I updated the rating. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they mention it in section 2.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4hr"}, "extra_text": {}}, {"review_id": "EhUEJ0fh7W0", "reviewer": "Reviewer_oJ4V", "summary": "This paper proposes a new data augmentation method for domain generalization. The main idea is to train a neural network-based generator to generate novel images that can be correctly classified by a student model and in the meantime, make the feature discrepancy larger between the student model and the EMA teacher model. The student model, which is fed with the generated images, is trained to minimize both the classification error and the feature discrepancy with its teacher. Only the EMA teacher is deployed at test time. This method is evaluated on both multi- and single-source domain generalization settings, outperforming previous SOTAs by a clear margin.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n- Though learning-based data augmentation has been studied in domain generalization, the idea of using the teacher-student paradigm proposed in this work is novel and could add value to this line of research.\n\n- The evaluation is comprehensive and sufficient, which covers both the multi- and single-source settings as well as different network architectures. The results are encouraging (clearly better than previous data augmentation methods).\n\n**Weakness**\n\n- Section 2 lacks discussions on the differences between this work and existing data augmentation methods. In addition, I do not see a strong connection between this work and SSL methods—except those based on multi-view learning (contrastive learning).\n\n- It's unclear why the single-layer classifier is trained and then fixed during the entire learning process. Why not train the classifier along with the student model?\n\n- The loss formulation in Eq.4 is a bit strange. Once the feature discrepancy is smaller than the margin, no gradients will be backpropagated due to the max(x, 0) function. Why not simply maximize Eq.1 for maximizing the feature discrepancy?\n\n- Compared with confusing a domain classifier or maximizing a distribution distance for synthesizing novel-domain images, the design of maximizing the feature discrepancy between a model and its EMA is not that intuitive. More insights should be provided to explain how this design helps the generator produce images that deviate from the source distributions.\n\n- Line217 says that the generator's architecture is similar to that used in CycleGAN. I'd like to confirm if this generator is trained with only the classification loss and the feature discrepancy loss? Did the authors use other losses like the L1 loss to train the generator?\n\n- In Fig.3, why there are two generated images given the same input? Isn't the generator a deterministic model?\n\n- Line218: Why pretrain the EMA teacher model while giving a random initialization to the student model?\n\n- Line290: What is the difference between Student with EMA and the proposed model (G + F_T)?\n\n---\nPost-rebuttal update:\n\nI have gone through the authors' responses, which have addressed my concerns. I have read other reviewers' comments as well.\n\nThe proposed data augmentation approach is novel in DG and well justifed with extensive experiments and ablation studies. My concerns raised during the initial review were mainly related to the presentation and clarity, which have been cleared in the rebuttal. I think this paper is worth sharing to the DG community so I change my rating to clear accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have properly discussed the limitations of this work in Sec.2, but do not talk about potential negative societal impact. Since this work is a data augmentation method based on learning a generator network from data, the learning process might be biased toward specific datasets used.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "AO2bF_R5wVK", "reviewer": "Reviewer_GTRz", "summary": " This paper proposed adversarial teacher-student knowledge distillation for domain generalization. Motivated by recent progress in self-supervised representation learning, the authors proposed two learning stages including domain generalization representation learning and novel domain augmentation. During domain generalized representation learning, the student is updated while the augmenter is fixed and the loss minimizes the discrepancy between the teacher and student. During the Novel domain augmentation learning, the student is fixed and the augmenter is updated. The objective is to maximize the discrepancy and encourage the augmenter to generate more distinct samples. Experiments on several public benchmark datasets very the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\nThe paper is well written and the idea is novel. The authors perform extensive experiments and ablation studies to verify the effectiveness of each model.  The proposed 2 stage representation and augmentation learning dynamically combine cycle GAN and self-supervised learning by knowledge distillation. The insight that the teacher module is avoided from observing the unrealistic domain augmentation is interesting and verified from experiment results. The results are very good and achieve state-of-the-art performance on multiple benchmarks.  \n\n[Weakness]\n\nIn the Siamese archi. (From table 5), the paper shows optimizing both teacher and student through gradient descents still achieve reasonable performance. In self-supervised image representation learning, there will be severe mode collapses without EMA update. I wonder why siamese architecture is working here? Could the author illustrate more. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N.A.", "review_text": "", "extra_scores": {"ethical_concerns": "N.A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was recommended for acceptance by all reviewers. The reviewers appreciated the fact that the paper was well-written, had comprehensive experiments, and presented a simple yet intuitive method. Though the reviewers originally had a few questions, the authors' response sufficiently answered those questions and convinced the reviewers of the merit of this work. The paper addresses the problem of domain generalization and the key aspect the reviewers appreciated the most was the relaxation of the assumption of domain labels and the effective performance shown from the method.
Decision: Accept (Spotlight)