Invariant Information Bottleneck for Domain Generalization
Bo Li[1] Yifei Shen[2] Yezhen Wang[1] Wenzhen Zhu[3] Colorado Reed[4] Kurt Keutzer[4] Dongsheng Li[1] Han Zhao[5][1]Microsoft Research Asia, China [2]HKUST, China [3]WUSTL, USA [4]UC Berkeley, USA [5]UIUC, USA

Motivation
Domain generalization(DG): learn a model from several training
domains so that it generalizes to unseen test domains.

Invariant Information Bottleneck
We first write IRM’s objective into a mutual information (MI) one asdenoted in red underline part. Then we combine IB’s objective ingreen underline part. IIB’s integrated objective is listed below.

Invariant Risk Minimization

Experiments
Cross Line: we create ten-valued spurious feature by adding cross lines
to images. We set 10 line patterns for 10 classes. For certain class 𝑖,majority (𝑝!! = 0.5) images are added with line pattern 𝑖, minority images(𝑝!" = 0.05) are added with other line pattern 𝑗.

Information Bottleneck

The mutual information form indicates the following properties:• max 𝐼 𝛷 𝑋 , 𝑌 denotes feature Φ X should be informative to!

predict the class label.
• min 𝐼 𝑌, 𝐷 | 𝛷 𝑋 denotes feature Φ X should not change acrossDomainBed: leave one domain out model selection (train-validation
selection is provided in supplementary file)!

(a) Bears in training/test domains

domains for the same class label (e. g. outline and shape should
always be the conditional invariant feature to predict a bear across(b) Constrain pseudo-invariant feature

Invariant risk minimization (IRM): a promising DG method but issusceptible to pseudo-invariant feature. In figure (a), training
domains contain most brown color bears while test domains containbears in other fur color. IRM would probably rely on fur color as
pseudo-invariant feature in training domains.
Question:
How could we eliminate the usage of those features while they arepseudo-invariant in training domains?
Answer:By constraining the information of feature Z and input X. We denotethe concept with figure (b).

domains).
• min 𝐼 𝑋, 𝛷 𝑋!

denotes feature Φ X should contain least

information about input.
Variational Approximation
We denote 𝑍 = 𝛷 𝑋 and formulate the MI into practical losses. Moredetails are in Loss Function Design section.
" & #)• 𝐼 𝑍, 𝑌 − 𝛽𝐼 𝑍, 𝑋 ≥ 𝔼" # ,%,& log 𝑞 𝑦 𝑧) − 𝛽𝔼" #,& log) &•𝐼(𝑌, 𝐷 | 𝑍) = 𝐻(𝑌 𝑍 − 𝐻 𝑌 𝐷, 𝑍)= sup* 𝔼"!,# [log 𝑞 𝑦 𝑧)] − sup+ 𝔼"!,#,$ [log ℎ 𝑦 𝑧, 𝑑)]Training/Inference procedure of IIB, IIB optimizes a model consisting of three parts (1)an invariant predictor. (2) a domain-dependent predictor. (3) an encoder.

MethodPrevious Invariant Risk Minimization objective:

Practical Optimization Objective of IRMConclusion: Cross Line experiment performance suggest IIB can betterovercome pseudo-invariant features than IRM and other methods.
DomainBed experiment performance suggest IIB is applicable to realistic
DG task.

Conclusion
In this paper, we have following conclusions:• To mitigate pseudo-invariant features, inspired by theIB’s principle, we propose to constrain the mutual
information between inputs and features.
• We developed a novel information-theoretic approach
IIB to overcome above issues.
• We further adopt variational approximation to develop
tractable loss functions.
• We analyze IIB’s performance with extensive
experiments on both synthetic and large-scale
benchmarks.