IJCAI-ECAI 2022 Tutorial on

Domain Generalization

Jindong Wang

Microsoft Research Asia

Haoliang Li

City University of Hong Kong

Tutorial website: https://dgresearch.github.io/

Sinno Jialin Pan

Nanyang Technological University

The breakthrough of AI

https://openai.com/dall-e-2/#demosArtificial intelligenceBackground Computer vision: How do we represent an image?

Artificial intelligence?
 More artificial, more intelligence…

Figure credit: https://cn.nytimes.com/technology/20181126/china-artificial-intelligence-labeling/Background Models do not generalize well to new domains; not like humans!
 Are big data always available?
 It is impossible to consider data in all scenarios.
 Data can be protected under privacy regulation.

• Pan et al. A Survey on Transfer Learning. IEEE TKDE 2010.
•

Wang et al. Generalizing to unseen domains: a survey on domain generalization. IEEE TKDE 2022.

Domain adaptation
 DA: Train on source and adapt to target

ImageNetCIFAR-100

Domain adaptation

Sample Re-weighting[Huang,NIPS’06]

Subspace Learning[Fernando, ICCV’13]

Adversarial Learning [Ganin, ICML’15, Tzeng, CVPR’17]

Domain adaptation: Train on Source and Adapt to Target

Are we able to obtain
ImageNet
unlabeled testing data? CiFAR100

Domain adaptation: Train on Source and Adapt to Target

NO!
ImageNet Real-time deployment
CiFAR100
Data privacy regulation

Domain Generalization
 DG: Build a system for previously unseen datasets given one or

multiple training datasets.

UnseenSeen domains

Model
Train

Test

Wang et al. Generalizing to unseen domains: a survey on domain generalization. IEEE TKDE 2022.

Formal definition of domain generalization
 Definition
 Given: 𝑀 training domains 𝒮 = 𝒮𝑖 𝑖 = 1, ⋯ , 𝑀}, where 𝒮𝑖 =𝑥𝑗𝑖 , 𝑦𝑗𝑖𝑛𝑖𝑗=1

 Condition:𝑗𝑖 Joint distributions are different, i.e., 𝑃𝑋𝑌≠ 𝑃𝑋𝑌 , 1 ≤ 𝑖 ≠ 𝑗 ≤ 𝑀

 Test domain cannot be accessed in training
 Goal: Achieve minimum test error on test domain𝑖𝑡𝑒𝑠𝑡 (𝑃𝑋𝑌≠ 𝑃𝑋𝑌)Wang et al. Generalizing to unseen domains: a survey on domain generalization. IEEE TKDE 2022.

Different DG settings
 Different DG settings

The general setting;
Focus of this tutorial

SettingSituation

Traditional domain generalization

The traditional setting

Single-source domain generalization

Only 1 source domain available for training

Semi-supervised domain generalization Training domains are partially labeled

•
•
•
•
•

Federated domain generalization

Training data cannot be accessed by central server

Open domain generalization

Training and test domains have different label spaces

Unsupervised domain generalization

Training domains are totally unlabeled

Peng X, Qiao F, Zhao L. Out-of-domain Generalization from a Single Source: A Uncertainty Quantification Approach[J]. arXiv preprint
arXiv:2108.02888, 2021.
Lin L, Xie H, Yang Z, et al. Semi-Supervised Domain Generalization in Real World: New Benchmark and Strong Baseline[J]. arXiv preprint
arXiv:2111.10221, 2021.
Zhang L, Lei X, Shi Y, et al. Federated Learning with Domain Generalization[J]. arXiv preprint arXiv:2111.10487, 2021.
Shu Y, Cao Z, Wang C, et al. Open domain generalization with domain-augmented meta-learning. CVPR 2021.
Qi L, Wang L, Shi Y, et al. Unsupervised Domain Generalization for Person Re-identification: A Domain-specific Adaptive Framework[J]. arXiv
preprint arXiv:2111.15077, 2021.

Relation with existing paradigms

DG has close relationship with other paradigms, but also different from them
Wang et al. Generalizing to unseen domains: a survey on domain generalization. IEEE TKDE 2022.

Overview of this tutorial
Introduction of DG & related areas
Methodology
Applications
Datasets, benchmark & evaluation
Theory & challenges

Introduction of DG & related areas
Methodology
Applications
Datasets, benchmark & evaluation
Theory & challenges

Overview of DG methodology

Domain Generalization

Data
manipulation

Data augmentation
Data generation

Domain randomization
Adversarial data augmentation

Kernel methods

Domain-invariant
representation learning

Representation
learning

Explicit feature alignment
Domain-adversarial learning
Invariant risk minimization
Multi-component analysis

Feature disentanglement

Generative modeling

Causality-inspired methods

Meta-learning

Learning
strategyDistributionally robust optimization

Ensemble learning

Self-supervised learning

Gradient operation

OthersWang et al. Generalizing to unseen domains: a survey on domain generalization. IEEE TKDE 2022.

Data manipulation for DG

Data manipulation
 Data quantity and quality are key factors of generalization
 Increase quality and quantity

Data augmentation
Data generation

Data augmentation
 Typical augmentation
 Rotation, noise, color…

 Domain randomization (DR) Randomly draw K real-life categories from ImageNet for stylizing the synthetic images.

Yue et al. Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data. ICCV,2019.

Domain randomization
Domain randomization through graphics software.

Sim->Real robot control••

Synthetic images -> Real images

Tobin, et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017.
Tremblay et al. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. CVPR workshop 2018.

Context-aware randomization

Prakash et al. Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data. 2018.

Adversarial data augmentation
 CrossGrad: Adversarially augment data via gradient training Generate data that are with same label 𝑦, but different domain label 𝑑 ADV augmentation
 Learning the worse-case distribution to enable generalization••

Shankar et al. Generalizing across Domains via Cross-Gradient Training. ICLR 2018.
Volpi, et al. Generalizing to Unseen Domains via Adversarial Data Augmentation. NeurIPS 2018.

Data generation
 Directly generate data Learning to generate, instead of randomization / adversarial augmentation (Fixed scheme)

Variational auto-encoder (VAE)•••

Generative adversarial net (GAN)

Mixup

Kingma D P, Welling M. Auto-encoding variational bayes[J]. arXiv preprint arXiv:1312.6114, 2013.
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in neural information processing systems, 2014, 27.
Zhang H, Cisse M, Dauphin Y N, et al. Mixup: Beyond empirical risk minimization[J]. arXiv preprint arXiv:1710.09412, 2017.

Data generation

VAE for generation

Conditional GAN for generation••••

Qiao et al. Learning to Learn Single Domain Generalization. CVPR 2020.
Rahman et al. Multi-component Image Translation for Deep Domain Generalization. 2020.
Zhou et al. Learning to Generate Novel Domains for Domain Generalization. ECCV 2020.
Somavarapu et al. Frustratingly Simple Domain Generalization via Image Stylization. 2020.

Multi-component generation

Image stylization

Mixup

DomainMix•••

MixAllStyle Mixup

Wang et al. DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations. 2020.
Wang et al. Heterogeneous domain generalization via domain mixup. ICASSP 2021.
Zhou et al. Domain generalization with mixstyle. ICLR 2021.

Data generation for DG
 Is vanilla Mixup enough for DG?

Semantic range

 No.
 Consider semantic range.
 We also need a large margin.

 SDMix: Semantic-Discriminative Mixup.

Discrimination

Lu et al. Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition. ACM IMWUT, 2022.

Summary of data manipulation
 Advantages
 Easy to understand and simple to implement
 General to all kinds of data and networks

 Potential disadvantages
 Lack of theoretical guarantee Restricted by quality of training data

Representation learning for DG

Representation Learning Learning domain-invariant representations
 Learning features which are expected to be better generalized to unseen target domain.

Classifier

FeatureRegularization

Representation learning
 How to learn generalized representations for DG?
••••

Kernel-based methods
Domain adversarial learning
Explicit feature alignment
Invariant risk minimization

Kernel-based methods
 Using kernel methods to learn domain-invariant features
 DICA: domain-invariant component analysis

 TCA: Transfer Component Analysis•••

Blanchard et al. Generalizing from Several Related Classification Tasks to a New Unlabeled Sample. NeurIPS 2011.
Muandet et al. Domain Generalization via Invariant Feature Representation. ICML 2013.
Grubinger et al. Domain Generalization Based on Transfer Component Analysis. IWANN 2015.

Kernel-based methods Marginal distribution adaptation
 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝐷𝑠 , 𝐷𝑡 ≈ 𝑀𝑀𝐷 𝑃𝑠 𝑥 , 𝑃𝑡 𝑥 , 𝑓= sup 𝔼𝑃 (raw version)𝑓∈ℱ= tr(𝐀𝐓 𝐗𝐌𝐗 𝐓 𝐀)

 (kernel version) = tr(𝐊𝐌)

𝐾𝑠,𝑠 , 𝐾𝑠,𝑡𝐾=
𝐾𝑡,𝑠 , 𝐾𝑡,𝑡1 𝑚1
σ𝑖=1 𝑓(𝑥𝑖 ) − σ𝑛𝑗=1 𝑓(𝑦𝑗 )𝑚𝑛

𝐗 = 𝐗 𝑠 , 𝐗 𝑡 ∈ ℝ𝑑×(𝑚+𝑛) , 𝐀 ∈ ℝ 𝑚+𝑛 × 𝑚+𝑛1, 𝑥 , 𝑥 ∈ 𝐷𝑠
𝑚2 𝑖 𝑗1
𝑀𝑖,𝑗 =, 𝑥 , 𝑥 ∈ 𝐷𝑡
𝑛2 𝑖 𝑗−1, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒𝑚𝑛

min tr(𝐊𝐌) − 𝜆 tr(𝐊)[1] Pan et al. Domain adaptation via transfer component analysis. IEEE TNN 2011.

33

Kernel-based methods
 More than just distribution adaptation
 ESRand: Elliptical Summary Randomisation (ESRand)
 comprises of a randomised kernel and elliptical data summarization

 projected each domain into an ellipse to represent the domain information and then used some similarity metric tocompute the distance.

 SCA: scatter component analysis
 Adopted Fisher’s discriminant analysis to minimize the discrepancy of representations from the same class and the same

domain, and maximize the discrepancy of representations from the different classes and different domains••

Erfani S, Baktashmotlagh M, Moshtaghi M, et al. Robust domain generalisation by enforcing distribution invariance. AAAI 2016.
Ghifary et al. Scatter Component Analysis: A Unified Framework for Domain Adaptation and Domain Generalization. TPAMI 2017.

Explicit feature alignment
 Explicit distance: 𝑅(⋅,⋅) ≈ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝐷𝑠 , 𝐷𝑡 )
 Goal: 𝑓 ∗ = arg 𝑚𝑖𝑛𝑓

1 𝑚
σ 𝐿(𝑓 𝑥𝑖 , 𝑦𝑖 ) + 𝜆 ⋅ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝐷𝑠 , 𝐷𝑡 )𝑚 𝑖=1

 Kernel-based distance Maximum mean discrepancy (MMD) [1] KL-divergence Cosine similarity

 Geometrical distance Geodesic flow kernel (GFK) [2] Correlation alignment (CORAL)[3] Riemannian manifold [4][1] Pan et al. Domain adaptation via transfer component analysis. IEEE TNN 2011.
[2] Gong et al. Geodesic flow kernel for unsupervised domain adaptation. CVPR 2012.
[3] Sun et al. Return of frustratingly easy domain adaptation. AAAI 2016.
[4] Baktashmotlagh et al. Domain adaptation on statistical manifold. CVPR 2014.

35

Explicit distance Maximum mean discrepancy (MMD) Given 𝑥 ∼ 𝑃, 𝑦 ∼ 𝑄, 𝑓 is a feature map: 𝑥 → ℋ, where ℋ is reproducing kernel
Hilbert space (RKHS), then
𝑀𝑀𝐷 𝑃, 𝑄, ℱ ≔ sup 𝔼𝑃 𝑓 𝑥 − 𝔼𝑄 𝑓 𝑦
 Empirical estimate

𝑀𝑀𝐷 𝑃, 𝑄, ℱ

𝑓∈ℱ

1 𝑚
1 𝑛
≔ sup 𝔼𝑃 σ𝑖=1 𝑓(𝑥𝑖 ) − σ𝑗=1 𝑓(𝑦𝑗 )𝑚𝑛𝑓∈ℱTheorem: 𝑀𝑀𝐷 𝑃, 𝑄, 𝐹 = 0 𝑖𝑓𝑓 𝑃 = 𝑄, when ℱ = 𝑓 ||𝑓||ℋ ≤ 1} is a unit ball in a
RKHS, provided that ℋ is universal. [1][1] Alexander J. Smola. Maximum mean discrepancy. ICONIP 2016, Hong Kong. http://alex.smola.org/teaching/iconip2006/iconip_3.pdf

36

Explicit feature alignment Learning shareable information across domain
 Maximum mean discrepancy: KL Divergence: Correlation alignment:•••

Ya Li, et al., Deep domain generalization via conditional invariant adversarial networks, ECCV 2018
Haoliang Li, et al., Domain Generalization for Medical Imaging Classification with Linear-Dependency, NeurIPS, 2020
Jin X, Lan C, Zeng W, et al. Style Normalization and Restitution for Domain Generalization and Adaptation, Arxiv, 2021.

Multi-layer Feature Learning Feature disentanglement at deep layer.

 Neuron independence regularization
𝑃 𝐻1 , 𝐻 2 , … , 𝐻 𝑑′= 𝑃 𝐻1 𝑃 𝐻2 … 𝑃(𝐻𝑑 )Total Correlation Minimization
through dimension permutation[Arcones1992][Arcones1992] M. A. Arcones and E. Gine, “On the bootstrap of u and v statistics,” The Annals of Statistics, pp.
655–674, 1992.

Domain-adversarial training Implicit distance: 𝑅(⋅,⋅) ≈ 𝑆𝑒𝑝𝑎𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦(𝐷𝑠 , 𝐷𝑡 )
 Goal: 𝑓 ∗ = arg 𝑚𝑖𝑛𝑓

1 𝑚
σ 𝐿(𝑓 𝑥𝑖 , 𝑦𝑖 ) + 𝜆 ⋅ 𝑆𝑒𝑝𝑎𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦(𝐷𝑠 , 𝐷𝑡 )𝑚 𝑖=1

 How to measure 𝑆𝑒𝑝𝑎𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦(𝐷𝑠 , 𝐷𝑡 )?

 Domain discriminator in generative adversarial nets (GAN) [1] Objective: ℓadv = 𝔼𝑧~𝑃 𝑧 log (1 − 𝐷 𝐺 𝑧 ) + 𝔼𝑥~𝑃img 𝑥 log 𝐷(𝑥) Train: min max ℓadv
𝐺𝐷

 How to use GAN for transfer?

[1] Ganin et al. Unsupervised domain adaptation by backpropagation. ICML 2015.
Figure: https://becominghuman.ai/generative-adversarial-networks-gans-human-creativity-2fc61283f3f6

39

GANs
 Generative adversarial nets
 GAN -> transfer learning -> domain generalization

GAN

GAN-based DA

GAN-based DG

Real faces

Source domain

Domain 1

Random noise

Target domain

Domain 2…

Generator

Generator

Generator

Discriminator

Discriminator

Discriminator

Figure: https://medium.com/sigmoid/a-brief-introduction-to-gans-and-how-to-code-them-2620ee465c30

40

DANN
 Domain adversarial neural network (DANN)[1] Feature extractor: 𝐺𝑓 (⋅; 𝜃𝑓 )
 Label predictor: 𝐺𝑦 (⋅; 𝜃𝑦 )
 Domain classifier: 𝐺𝑑 (⋅; 𝜃𝑑 )Figure: Ganin et al. Unsupervised domain adaptation by backpropagation. ICML 2015.

41

DANN
 Training of DANN Objective: Learning:

Classification loss
 Minimize feature extraction and classification loss

Separation loss

 Maximize domain confusion

 Stochastic gradient descent Problem: 𝜆 is hard to implement in SGD

42

DANN
 Train DANN in SGD
 Gradient reversal layer (GRL) Forward propagation: GRL is an identity map

𝑅𝜆 𝑥 = 𝑥
 Backward propagation: take gradient from subsequent level, and × (−𝜆)

𝑑𝑅𝜆
= −𝜆𝐈
𝑑𝑥
 Transformed objective function:

 GRL can be easily implemented in Pytorch/Tensorflow/Caffe…

+
Code of DANN: http://github.com/jindongwang/transferlearning/code/deep/DANN

43

Domain-adversarial learning for DG

DLOW

MMD-AAE••

Haoliang Li et al. Domain Generalization with Adversarial Feature Learning. CVPR 2018.
Rui Gong et al. DLOW: Domain Flow for Adaptation and Generalization. CVPR 2019.

Domain-adversarial learning for DG
 Is local alignment enough for DG?

Data of walking activity

 No. Ignore some big picture features.
 We also need a global alignment.
 LAG: Local and Global Alignment.

Global feature learning

Lu et al. Local and global alignments for generalizable sensor-based humanactivity recognition. ICASSP 2022.

MixupRepresentation augmentation for DG
 Is vanilla Mixup or simple alignment enough?
 No. Domain-invariant feature Mixup.

Domain-invariant

 FIXED: Domain-invariant Feature MIXup with Enhanced Discrimination.

Ours

Enlarged distribution cover range

Lu et al. FIXED: Frustratingly Easy Domain Generalization with Mixup. Under Review, 2022.

Invariant risk minimization
 IRM
 Do not seek to match the representation distribution of all domains, but to enforce theoptimal classifier on top of the representation space to be the same across all domains
 The intuition is that the ideal representation for prediction is the cause of y, and the causal
mechanism should not be affected by other factors/mechanisms, thus is domain-invariant.
Learn 𝑔Arjovsky M, Bottou L, Gulrajani I, et al. Invariant risk minimization[J]. arXiv preprint arXiv:1907.02893, 2019.

Feature disentanglement
 What is disentanglement
 Learn a function that maps a sample to a feature vector, which contains all the information

about different factors of variation and each dimension (or a subset of dimensions) contains
information about only some factor(s).

 Formulation
Common features

Specific features

Reconstruction error

Feature disentanglement• Multi-component analysis• Generative modelingMulti-component analysis
 UndoBias
 Weights can be disentangled into: common and specific weights

 Structure low-rank DG••

Khosla A, Zhou T, Malisiewicz T, et al. Undoing the damage of dataset bias. ECCV 2012.
Ding Z, Fu Y. Deep domain generalization with structured low-rank constraint. IEEE TIP 2017.

Feature disentanglement
Invariant feature learning + style transfer

Yufei Wang, et al., Variational Disentanglement for Domain Generalization, Arxiv 2021

Multi-layer Feature Learning Deep features eventually transit from general to specific along the network.

 Shallow Layer extracts shareable information while deep layer extracts category specificinformation (with regularization).

Cat…
• Haoliang Li, et.al., “GMFAD: Towards Generalized Visual Recognition via Multi-Layer
Feature Alignment and Disentanglement”, T-PAMI 2020

Domain-Invariant Learning with Uncertainty• Uncertainty should be considered during domain-invariant learning.

Bayesian Neural Network••

Uncertainty modeling through re-parameterization trick

Zehan Xiao, et al., A Bit More Bayesian: Domain-Invariant Learning with Uncertainty , ICML’21
Xiaotong Li, et al., Uncertainty Modeling for Out-of-Distribution Generalization." ICLR’22.

Generative modeling DIVA: domain-invariant variational-autoencoder

Domain-input-label

 CSG: Causal semantic generative model

S: semantic factor
V: variation factor••

Liu et al, Learning Causal Semantic Representation for Out-of-Distribution Prediction. NeurIPS 2021.
Ilse M, Tomczak J M, Louizos C, et al. Diva: Domain invariant variational autoencoders[C]//Medical Imaging with Deep Learning.
PMLR, 2020: 322-348.

Summary of representation learning Advantages
 General and popular Better performance Some theoretical guarantee Potential disadvantages
 Still difficult to remove spurious features
 Data-driven

Learning strategy for DG

Different learning strategy for DG
 Meta-learning Divide domains into several tasks, then use meta-learning to learn general knowledge

 Ensemble learning Design ensemble models

 Gradient operation
 Alter the gradient interaction between domains

 Distributionally robust optimization
 Acquire models that are better for worst-case distribution scenario

 Self-supervised learning Others

Meta-learning Learning to learn, or meta-learn the general knowledge
 Instead of the original tasks, meta-learning wants to acquire knowledge about new tasks

Meta-knowledge acquisition

Meta-knowledge validation

Huisman M, Van Rijn J N, Plaat A. A survey of deep meta-learning[J]. Artificial Intelligence Review, 2021, 54(6): 4483-4541.

Meta-learning for DG
 How to adopt meta-learning for DG?
 Key: Old tasks to new tasks in meta-learning → Old domains to new domains

 MLDG: Meta-learning for DG

 MetaReg: meta-learning for regularization••

Li D, Yang Y, Song Y Z, et al. Learning to generalize: Meta-learning for domain generalization. AAAI 2018.
Balaji Y, Sankaranarayanan S, Chellappa R. Metareg: Towards domain generalization using meta-regularization. NeurIPS 2018.

Meta-learning for DG
 Feature-critic training Learning the regularization terms using

meta-learning Meta-VIB
 Meta variational information bottleneck tomodel uncertainty between domain shifts

Li Y, Yang Y, Zhou W, et al. Feature-critic networks for heterogeneous domain generalization.
ICML 2019.
Du Y, Xu J, Xiong H, et al. Learning to learn with variational information bottleneck for domain
generalization. ECCV 2020.

Meta-learning for DG
 DADG: MLDG with adversarial training•• DGSML: MLDG with semi-supervised

learning

Chen K, Zhuang D, Chang J M. Discriminative adversarial domain generalization with meta-learning based cross-domain
validation. Neurocomputing 2022.
Sharifi-Noghabi H, Asghari H, Mehrasa N, et al. Domain generalization via semi-supervised meta learning[J]. arXiv preprint
arXiv:2009.12658, 2020.

Ensemble learning Is a single model or representation enough for generalization?
Linear
K nearest Decision Neural
Regression Neighbors Tree Network

Acc: 80%

Acc: 85% Acc: 91% Acc: 90%

Bagging

Linear
K nearest Decision Neural
Regression Neighbors Tree Network
Ensemble
Model

Ensemble
ModelsAcc: 98%

Boosting• Ensemble learning allows formore diversities in feature andclassifier learning• The power from the crowd

Stacking

Ensemble learning for DG
 Ensemble-learned DG representations
 Feature weighting•••

Feature combination

Feature attention

Mancini M, Bulo S R, Caputo B, et al. Best sources forward: domain generalization through source-specific nets. ICIP 2018.
Segu M, Tonioni A, Tombari F. Batch normalization embeddings for deep domain generalization[J]. arXiv preprint arXiv:2011.12672, 2020.
D’Innocente A, Caputo B. Domain generalization with domain-specific aggregation modules[C]//German Conference on Pattern Recognition.
Springer, Cham, 2018: 187-198.

Ensemble learning for DG
 Ensemble learning for classifier learning SEDGE: ensemble of pre-trained models for classifier learning
DAEL: domain adaptive ensemble learning

Li Z, Ren K, Jiang X, et al. Domain Generalization using Pretrained Models without
Fine-tuning[J]. arXiv preprint arXiv:2203.04600, 2022.
Zhou K, Yang Y, Qiao Y, et al. Domain adaptive ensemble learning[J]. IEEE TIP
2021.

Ensemble learning for DG
Domain-specific features

 Is ensemble learning enough for DG?
 No. Ensemble → domain-specific knowledge
 We also need a balance with domain-invariant knowledge
 AFFAR: Adaptive Feature Fusion+Domain-invariant features

Qin et al. Domain generalization for activity recognition via adaptive
feature fusion. ACM TIST 2022.

Gradient operation for DG
 Model the interactions between cross-domain gradients
Fish: gradient inner product••

RSC: self-challenging for gradient

Shi Y, Seely J, Torr P H S, et al. Gradient matching for domain generalization. ICLR 2022.
Huang Z, Wang H, Xing E P, et al. Self-challenging improves cross-domain generalization. ECCV 2020.

Self-supervised learning for DG
 Construct pretext tasks for general representation learning
Self-supervised learning

Selfreg: self-supervised contrastive loss

JiGen: Jigsaw puzzle + DG••

Carlucci F M, D'Innocente A, Bucci S, et al.
Domain generalization by solving jigsaw puzzles.
CVPR 2019.
Kim D, Yoo Y, Park S, et al. Selfreg: Selfsupervised contrastive regularization for domain
generalization. ICCV 2021.

Contrastive Learning
Minimizing/Maximizing feature distance among samples from with same/different
category information from different domains••

Motiian, et al., Unified Deep Supervised Domain Adaptation and Generalization, ICCV’17
Dou, et al., Domain Generalization via Model-Agnostic Learning of Semantic Features, NeurIPS’19

Distributionally robust optimization for DG
 Learn a model at worst-case distribution scenario
GroupDRO: Convex hull +Regularization

VRex: Convex hull + Perturbation + Risk
variance•

S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case
generalization,” in ICLR, 2020.

•

D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, “Out-of-distribution generalization via risk extrapolation (rex),” in ICML,
2021, pp. 5815–5826.

Other learning strategy
 Other interesting learning strategy for DG
Shapelet feature: invariant across domains••

SWAD: Smooth training loss

Narayanan M, Rajendran V, Kimia B. Shape-biased domain generalization via shock graph embeddings. ICCV 2021.
Cha J, Chun S, Lee K, et al. Swad: Domain generalization by seeking flat minima. NeurIPS 2021.

Introduction of DG & related areas
Methodology
Applications
Datasets, benchmark & evaluation
Theory & challenges

Applications for DG
 Wide applications across CV, NLP, RL, and others

Figure credit: DG survey by Wang et al. (TKDE’22)

Wide applications of DG
 Computer vision

Action recognition

Image classification
Test

Style transfer
Semantic segmentation

Person ReID

Wide applications of DG
 Natural language processing

Semantic parsing

Sentiment classification
DVD

Book

Kitchen applianceTrain Reinforcement learning
Sim-to-realRobot
control

Electronic device

TestWide applications of DG
 Medical applications
COVID X-ray classification

Parkinson’s disease diagnosis

Tissue segmentationWide applications of DG
 Sensor-based human activity recognition Create a model that learns generalizable representations for different age groups Different people/device locations/activity patterns generate different sensor readings••

Lu et al. Local and global alignments for generalizable sensor-based
human activity recognition. ICASSP 2022.
Lu et al. Semantic-discriminative mixup for Generalizable Sensor-based
Cross-domain Activity Recognition. ACM IMWUT 2022.

Wide applications of DG
 Time series forecasting AdaRNN: adaptive forecasting of time series using DG
ACB

Unseen test𝑡

Raw data

𝑷𝑨

𝑷𝑩

?
𝑷𝑪

Probability
distribution

𝑷𝑻𝒆𝒔𝒕

Temporal Covariate Shift: 𝑷𝑨 ≠ 𝑷𝑩 ≠ 𝑷𝑪 ≠ 𝑷𝑻𝒆𝒔𝒕
Du et al. AdaRNN: adaptive learning and forecasting of time series. CIKM 2021.

Introduction of DG & related areas
Methodology
Applications
Datasets, benchmark & evaluation
Theory & challenges

Benchmarks for DG
 Important consideration for DG benchmarks:dWhich dataset?

Which codebase?

Which metric?

 Popular datasets
 Common benchmarks and codebases

 Evaluation strategy: model selection
Note:• Technically, any application settings that fits in DG scenario can beconsidered as a good test bed.
• There exists no “golden-standard” for benchmarking and evaluation.

Datasets for DG
 Common benchmarks

Benchmark and codebase
 DomainBed A unified benchmark for domain generalization

Interesting results: DomainBed found that there are not significant
improvements for recent DG algorithms. Is it the case?
https://github.com/facebookresearch/DomainBedBenchmark and codebase
 DeepDG
 Built by borrowing the knowledge from DomainBed, but faster, and easier to use• Avoids huge hyperparameter tuning• More friendly interface• Better customization

https://github.com/jindongwang/transferlearning/tree/master/code/DeepDG

Model selection
 Model selection in DomainBed

𝒟1

𝒟2

𝒟3

 Test-domain validation set (oracle)
 Use part of test domain as the validation

 Leave-one-domain-out cross-validation
 One domain as testing domain for validation

 Training-domain validation set (popular)
 Leave some part of the training data as the validation set• Q: is it reasonable to use training-domain validation
for model selection?
• A: no. Since the validation distribution cannot
represent the test distribution.

𝒟4
Train
Validation
Test
Assume 𝒟4 is theunseen test domain

Discussion about the performance of DG
 Performance should be restricted to certain applications
 Cross-dataset human activity recognition[1] Cross-dataset object detection[2]Hint: maybe we should develop application-oriented evaluation benchmarks?
[1] Lu et al. Semantic-discriminative mixup for generalizable cross-domain sensor-based human activity recognition. ACM IMWUT 2022.
[2] Jin X, Lan C, Zeng W, et al. Style normalization and restitution for domain generalization and adaptation. IEEE TMM 2021.

Introduction of DG & related areas
Methodology
Applications
Datasets, benchmark & evaluation
Theory & challenges

Theory Domain adaptation error bound The error on target domain is bounded by:Target risk

Source risk

Complexity of ℋ

Source-target distribution divergenceℋ-divergence:ℋΔℋ-distance:
Discrepancy distance:•••

Ben-David S, Blitzer J, Crammer K, et al. Analysis of representations for domain adaptation. NIPS 2016.
Ben-David S, Blitzer J, Crammer K, et al. A theory of learning from different domains[J]. Machine learning, 2010, 79(1): 151-175.
Mansour Y, Mohri M, Rostamizadeh A. Domain adaptation with multiple sources. NIPS 2009.

Theory for DG
 Assumption 1: convex hull Key: approximate target domain using the convex hull of source distributions

Ideal joint risk (best source vs. target)

Target risk

Weighted source risk

Distance between target and convex hull

Diameter of Λ
Albuquerque I, Monteiro J, Darvishi M, et al. Adversarial target-invariant representation learning for domain generalization[J]. 2020.

Theory for DG
 Assumption 2: classifier variation
 Key: the gap between available environments and all invariants

Variation
Ye H, Xie C, Cai T, et al. Towards a theoretical framework of out-of-distribution generalization. NeurIPS 2021.

Theory of DG
 Assumption 3: subpopulation shift Key: Gaussian mixture model to contain all sub-distributions

Gaussian mixture distribution

Yao H, Wang Y, Li S, et al. Improving Out-of-Distribution Robustness via Selective Augmentation. ICML 2022.

Theory of DG
 Other theory
 Adversarial training and pretrained model is good for DG[1] DG can be bounded under kernel learning conditions[2] Current progress The research on DG theory is still on the go• [1] Yi M, Hou L, Sun J, et al. Improved OOD Generalization via Adversarial Training and Pretraing. ICML 2021.
• [2] Deshmukh A A, Lei Y, Sharma S, et al. A generalization error bound for multi-class domain generalization[J]. arXiv preprint arXiv:1905.10392, 2019.

Challenges
 Continuous domain generalization
 Continuous / online learning Generalize to novel categories
 New categories instead of closed set

 Interpretable domain generalization
 Learning to interpret: why it can generalize?

 Large-scale pre-training / self-learning and DG
 The role of pre-training and self-learning with DG

 Performance evaluation
 Develop more fair and application-driven evaluation standards

Conclusion
General ML

Non-IID

Domain adaptation

Unseen target

Domain generalization

Introduction and background

Relation with existing area: transfer learning, domain adaptation, multi-task learning…

DG Roadmap

Data manipulation: augmentation, or generation
Algorithm

Representation learning: domain-invariant learning, disentanglement
Learning strategy: meta-learning, ensemble learning, gradient, DRO, SSL…

Applications: CV, NLP, RL, medical…
Datasets, benchmark, evaluation
Theory and future challenges

Thanks
Contact: jindong.wang@microsoft.com, haoliang.li@cityu.edu.hk

Tutorial website: https://dgresearch.github.io/
DG survey paper: https://arxiv.org/abs/2103.03097
Codebase: https://github.com/jindongwang/transferlearning/tree/master/code/DeepDG