Quantifying and Improving Transferability inDomain Generalization

Guojun Zhang, Han Zhao, Yaoliang Yu and Pascal Poupart

Main result

Algorithms

Theoretical results
Generalization bound (tighter than the one using ℋ-divergence)

We propose an evaluation metric for measuring the transferability in domaingeneralization. Based on this metric, we design algorithms to further
improve out-of-domain generalization over SoTA methods.

ϵ (h) ≤ ϵ (h) +,) + ϵ* − ϵ*, ϵ* = inf ϵ (h)h∈ℋ

If the optimal errors are zero, transfer measure with 0-1 loss is equivalentto total variation when Γ = ℋt includes all binary classi ers

Transfer learning
SourcedomainΓ(Total variation

Guitar!
Classi er hEmbedding gdTV( ,)=∑∫Two domains can still be transferable even if the joint distributions are dissimilar

Classi er hEmbedding g

Colors: intensity of theprobability mass

Piano!

There exists a common
optimal classi er
Transferability is more generalthan matching joint distributions

Transferable:
Every near-optimal source classifier is also near-optimal on the target

Sourcedomains| p (x, y) − p (x, y) | dxyHow do we measure the transferability of feature representations?

Targetdomain

Domain Generalization (DG) learns feature embeddings and classi ers from several
source domains

TargetdomainSuppose we have n source domains:

1, …,

Evaluate transferabilitymaxi,j∈[0,n] ∥h− h* ̂∥≤δmaxh∈ℋ

is (δ , δDe nition:)-transferable toδ-minimal set

Invariance principles (error depends on both the marginal and conditional
distributions)ϵ (h)ifhexc i(h ∘ g) − exc j(h ∘ g)1 nmin
ϵ i(h ∘ g) + max max exc i(h′ ∘ g) − exc j(h′ ∘ g)
∑g,h ni,j∈[1,n] ∥h′−h∥≤δi=1adversarial training

Experiments• H-divergence: invariance of feature distributions p(z) (marginal)• Invariant risk minimization (IRM): invariance of the optimal predictors (conditional)• Generalized label shift: invariance of feature distributions within the same classp(z | y) (conditional)

How to Compute Transfer Measures?
We make the following approximations

SourcedomainClassi er hTargetdomainexc (h) := ϵ (h) − inf ϵ (h)Excess risk

Γ(Transfer measure

Classi er hh) = sup | exc (h) − exc (h) |,h∈Γexc (h) := ϵ (h) − inf ϵ (h)Excess riskh••••We use surrogate loss to approximate 0-1 loss
We use the final loss after training to approximate the optimal loss
We use a neighborhood in the parameter space to approximate the minimal set
The resulting transfer measure is a lower bound, but we can still use it to refutetransferability
Final result as a lower bound oftransfer measure

θ: parameter of classifiersup∥θ− θ* ̂∥≤δϵ (h) − ϵ (h) − ϵ ( h* ̂)

h* ̂ / θ* ̂: the learned classifier after training

If the lower bound is large, then it is not transferable• Many popular algorithms are not learning transferable features: a near-optimal
source classifier is not near-optimal on the target• Our Transfer algorithm learns more transferable features than ERM, DANN, IRM,GroupDRO, etc.
• Best performers: Transfer (ours), CORrelation ALignment (CORAL), Spectral
Decomposition (SD)
References• Arjovsky et al. "Invariant risk minimization." arXiv preprint arXiv:1907.02893 (2019).
• Ben-David et al, “A theory of learning from different domains,” Machine Learning, 2010
• Koltchinskii, “Rademacher complexities and bounding the excess risk in active learning,”
JMLR 2010• Tachet des Combes et al. "Domain adaptation with conditional distribution matching andgeneralized label shift." NeurIPS 2020

𝒮𝒯

𝒮

Full talk: https://www.youtube.com/watch?v=Ce3PyHA54GI
￼𝒟𝒮

𝒟𝒯

𝒟

Code: https://github.com/Gordon-Guojun-Zhang/Transferability-NeurIPS2021
fi

fi

𝒮

𝒯

𝒯

𝒯

𝒟𝒯

𝒯

𝒮
𝒯

𝒮

𝒮

𝒮
fi

𝚃

𝒮

𝒯𝒮

𝒮

𝒮

𝒟𝒮𝒮fififi

fi

𝒯

𝒯

𝒮

𝒯
𝒮

𝒮

fi

𝒯𝒮

𝒮

𝒯

𝚃𝒮

𝒯

𝒯

𝒯

𝒮

𝒯

𝒮

𝒯𝒮￼𝒟𝒮𝒮￼Transferable ≡ Small transfer measure, if Γ = argmin(ϵ , δ) .

𝒮0

• Transferability: invariance of excess risks (joint)

argmin(ϵ , δ ) ⊆ argmin(ϵ , δ )Use Transfer Measures to quantify transferability=Improve transferability

ERMargmin(ϵ , δ) := {h ∈ ℋ, ϵ (h) ≤ inf ϵ (h) + δ}n and a target domain