Generalizing to Unseen Domains:
A Survey on Domain Generalization
Jindong Wang1, Cuiling Lan1, Chang Liu1, Yidong Ouyang2, Wenjun Zeng1, Tao Qin1
1 Microsoft Research Asia, Beijing, China

2 Central University of Finance and Economics, Beijing, China

jindong.wang@microsoft.com
https://arxiv.org/abs/2103.03097

https://github.com/jindongwang/transferlearning/tree/master/code/DeepDGBackground Domain adaptation
SketchCartoonTraining set
 𝑃!"#$% 𝑥, 𝑦 ≠ 𝑃!&'( 𝑥, 𝑦 Source: Train, Target: test
Images are copied from PACS dataset [Li et al.’17]

Art painting

PhotoTest set

Domain adaptation
 Basic theory of DA [Ben-David et al’07]

Target risk

Source risk

Complexity of ℋ

Source-target distribution divergence To solve DA, we need to: Reweight instances to select a subset of two domains where their 𝑑(⋅,⋅) is small
 Tradaboost [Dai et al’07], KMM [Sugiyama et al’08], Distant TL [Tan et al’17] Learn domain-invariant feature representations to reduce 𝑑(⋅,⋅)
 TCA [Pan et al’11], DANN [Ganin et al’15], DDC[Tzeng et al’14] and their extensions as of today

 How about testing phase? Model selection?
DA requires direct access to target domain in training!

Domain generalization
 Definition
 Given: 𝑀 training domains 𝒮 = 𝒮$ 𝑖 = 1, ⋯ , 𝑀}, where 𝒮$ =
 Condition:)$ Joint distributions are different, i.e., 𝑃,≠ 𝑃,- , 1 ≤ 𝑖 ≠ 𝑗 ≤ 𝑀

 Test domain cannot be accessed in training
 Goal: Achieve minimum test error on test domain$(&'( (𝑃,≠ 𝑃,)𝑥)$ , 𝑦)$%!
)*+Related area

Theory Domain adaptation error boundTheory Domain adaptation error boundTheory Domain generalization error boundMethodology

Data manipulation
 Data quantity and quality are key factors of generalization
 Increase quality and quantity

Data augmentation
Data generation

Data augmentation
 Typical augmentation
 Rotation, noise, color…

 Domain randomization (DR) Shape, position, texture, viewpoint, lighting condition, noise…

Yue et al. Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data. 2019.

Domain randomization

Sim->Real robot control••

Synthetic images -> Real images

Tobin, et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017.
Tremblay et al. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. CVPR workshop 2018.

Context-aware randomization

Prakash et al. Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data. 2018.

Adversarial data augmentation
 CrossGrad: Adversarially augment data via gradient training Generate data that are with same label 𝑦, but different domain label 𝑑 ADV augmentation
 Learning the worse-case distribution to enable generalization••

Shankar et al. Generalizing across Domains via Cross-Gradient Training. ICLR 2018.
Volpi, et al. Generalizing to Unseen Domains via Adversarial Data Augmentation. NeurIPS 2018.

Data generation
 Directly generate data Learning to generate, instead of randomization / adversarial augmentation (Fixed scheme)

Variational auto-encoder (VAE)•••

Generative adversarial net (GAN)

Mixup

Kingma D P, Welling M. Auto-encoding variational bayes[J]. arXiv preprint arXiv:1312.6114, 2013.
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in neural information processing systems, 2014, 27.
Zhang H, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization[J]. arXiv preprint arXiv:1710.09412, 2017.

Data generation

VAE for generation

Conditional GAN for generation••••

Qiao et al. Learning to Learn Single Domain Generalization. CVPR 2020.
Rahman et al. Multi-component Image Translation for Deep Domain Generalization. 2020.
Zhou et al. Learning to Generate Novel Domains for Domain Generalization. ECCV 2020.
Somavarapu et al. Frustratingly Simple Domain Generalization via Image Stylization. 2020.

Multi-component generation

Image stylization

Mixup

DomainMix•••

Style mixup

Wang et al. DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations. 2020.
Wang et al. Heterogeneous domain generalization via domain mixup. ICASSP 2021.
Zhou et al. Domain generalization with mixstyle. ICLR 2021.

Representation Learning Learning domain-invariant representations
Classifier How to learn representations?
••••

Kernel-based methods
Domain adversarial learning
Explicit feature alignment
Invariant risk minimization

FeatureRegularization

Kernel-based methods
 Using kernel methods to learn domain-invariant features
 DICA: domain-invariant component analysis

 TCA: Transfer Component Analysis

 SCA: Scatter Component Analysis••••

Blanchard et al. Generalizing from Several Related Classification Tasks to a New Unlabeled Sample. NeurIPS 2011.
Muandet et al. Domain Generalization via Invariant Feature Representation. ICML 2013.
Grubinger et al. Domain Generalization Based on Transfer Component Analysis. IWANN 2015.
Ghifary et al. Scatter Component Analysis: A Unified Framework for Domain Adaptation and Domain Generalization. TPAMI 2017.

Domain adversarial learning Adversarial trainingGanin et al. Unsupervised Domain Adaptation by Backpropagation. ICML 2015.

Domain adversarial learning

DLOW

MMD-AAE••Conditional GAN•

Li et al. Domain Generalization with Adversarial FeatureLearning. CVPR 2018.
Gong et al. DLOW: Domain Flow for Adaptation andGeneralization. CVPR 2019.
Li et al. Deep Domain Generalization via Conditional
Invariant Adversarial Networks. ECCV 2018.

Explicit feature alignment
 Distance Maximum mean discrepancy: Correlation alignment:

Jin X, Lan C, Zeng W, et al. Style Normalization and Restitution for Domain Generalization andAdaptation[J]. arXiv preprint arXiv:2101.00588, 2021.

Invariant risk minimization
 IRM
 Do not match distributions; enforce optimal classifier on top of the representation space to be

the same across all domains

Arjovsky et al. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

Feature disentanglement
 Formulation
Multi-component analysis
Generative modeling•• UndoBias
 Structure low-rank DG••

Khosla A, Zhou T, Malisiewicz T, et al. Undoing the damage of dataset bias[C]//European Conference on Computer
Vision. Springer, Berlin, Heidelberg, 2012: 158-171.
Ding Z, Fu Y. Deep domain generalization with structured low-rank constraint[J]. IEEE Transactions on Image Processing,
2017, 27(1): 304-313.

Generative modeling DIVA: domain-invariant variational-autoencoder
 DAL: domain-agnostic learning•

X. Peng, Z. Huang, X. Sun, and K. Saenko, “Domain agnostic learning with disentangled representations,” in ICML, 2019
Ilse M, Tomczak J M, Louizos C, et al. Diva: Domain invariant variational autoencoders[C]//Medical Imaging with Deep Learning.
PMLR, 2020: 322-348.

Learning strategy
 Meta-learning Divide domains into several tasks, then use meta-learning to learn general knowledge

 Ensemble learning Design ensemble models

Meta-learning

Feature Critic training

MLDG

•
•

Li D, Yang Y, Song Y Z, et al. Learning to generalize: Meta-learning for domain
generalization. AAAI 2018.
Li Y, Yang Y, Zhou W, et al. Feature-critic networks for heterogeneous domain
generalization. ICML 2019.

Ensemble learning•••

Mancini M, Bulo S R, Caputo B, et al. Best sources forward: domain
generalization through source-specific nets. ICIP 2018.
Segu M, Tonioni A, Tombari F. Batch normalization embeddings for deep
domain generalization[J]. arXiv preprint arXiv:2011.12672, 2020.
D’Innocente A, Caputo B. Domain generalization with domain-specificaggregation modules[C]//German Conference on Pattern Recognition. Springer,Cham, 2018: 187-198.

Datasets and applications
 Datasets Application
 Image classification / segmentation / detection / ReID
 Reinforcement learning Parkinson’s disease Activity recognition Fault diagnosis

Challenges
 Continuous domain generalization
 Continuous / online learning Generalize to novel categories
 New categories instead of closed set Interpretable domain generalization
 Learning to interpret: why it can generalize?

 Large-scale pre-training / self-learning and DG
 The role of pre-training and self-learning with DG

 Performance evaluation
 Develop more fair and application-driven evaluation standards

Thanks

jindong.wang@microsoft.com
https://arxiv.org/abs/2103.03097

https://github.com/jindongwang/transferlearning/tree/master/code/DeepDG