,,,,,,,,,,

Some Recent Insights on Transfer Learning
P + Q → Q?
Samory Kpotufe
Columbia University, Statistics
Earlier work with G. Martinet, and ongoing work with S. Hanneke

,,,,,,,,,,

Transfer Learning:
Given data {Xi , Yi } ∼i.i.d. P , produce a classifier for (X, Y ) ∼ Q.
Case study: Apple Siri’s voice assistant- Initially trained on data from American English speakers ...
- Could not understand 30M+ nonnative speakers in the US!
Costly Solution ≡ 5+ years acquiring more data and retraining!

A Main Practical Goal:
Cheaply transfer ML software between related populations.

,,,,,,,,,,

Transfer Learning:
Given data {Xi , Yi } ∼i.i.d. P , produce a classifier for (X, Y ) ∼ Q.
Case study: Apple Siri’s voice assistant- Initially trained on data from American English speakers ...
- Could not understand 30M+ nonnative speakers in the US!
Costly Solution ≡ 5+ years acquiring more data and retraining!

A Main Practical Goal:
Cheaply transfer ML software between related populations.

,,,,,,,,,,

Transfer Learning:
Given data {Xi , Yi } ∼i.i.d. P , produce a classifier for (X, Y ) ∼ Q.
Case study: Apple Siri’s voice assistant- Initially trained on data from American English speakers ...
- Could not understand 30M+ nonnative speakers in the US!
Costly Solution ≡ 5+ years acquiring more data and retraining!

A Main Practical Goal:
Cheaply transfer ML software between related populations.

,,,,,,,,,,

Transfer is of general relevance:
AI for Judicial Systems- Source Population: prison inmates- Target Population: everyone arrested
Over 60% inaccurate risk assessments on minorities(2016 Pro-Publica study)

Main Issue: Good Target data is hard or expensive to acquire
AI in medicine, Genomics, Insurance Industry, Smart cities,...

,,,,,,,,,,

Transfer is of general relevance:
AI for Judicial Systems- Source Population: prison inmates- Target Population: everyone arrested
Over 60% inaccurate risk assessments on minorities(2016 Pro-Publica study)

Main Issue: Good Target data is hard or expensive to acquire
AI in medicine, Genomics, Insurance Industry, Smart cities,...

,,,,,,,,,,

Transfer is of general relevance:
AI for Judicial Systems- Source Population: prison inmates- Target Population: everyone arrested
Over 60% inaccurate risk assessments on minorities(2016 Pro-Publica study)

Main Issue: Good Target data is hard or expensive to acquire
AI in medicine, Genomics, Insurance Industry, Smart cities,...

,,,,,,,,,,

Transfer is of general relevance:
AI for Judicial Systems- Source Population: prison inmates- Target Population: everyone arrested
Over 60% inaccurate risk assessments on minorities(2016 Pro-Publica study)

Main Issue: Good Target data is hard or expensive to acquire
AI in medicine, Genomics, Insurance Industry, Smart cities,...

,,,,,,,,,,

Many heuristics ... but theory and principles are still evolving

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,

Basic questions remain largely unanswered:Suppose: ĥ is trained on source data ∼ P , to be transferred to target Q.
• Is there sufficient information in source P about target Q?
• If not, how much new data should be collected?
• Would unlabeled data help?
• What’s the right mix of P and Q data w.r.t. $$ sampling costs?

What’s the relative statistical value of P and Q data?
Depends on how far P is from Q ...

,,,,,,,,,,How do we proceed?
Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?
Which notion of dist(P → Q) captures this error?

,,,,,,,,,,How do we proceed?
Nonparametric work
• (Covariate Shift) [Kpo. and Martinet]• (Posterior Drift) [Scott 19] [Cai and Wei, 19]• (Covariate Shift, Posterior Drift) [Reeve, Cannings, Samworth, 21]

Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?

,,,,,,,,,,How do we proceed?
Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?
Which notion of dist(P → Q) captures this error?

,,,,,,,,,,How do we proceed?
Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?
Which notion of dist(P → Q) captures this error?

,,,,,,,,,,How do we proceed?
Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?
Which notion of dist(P → Q) captures this error?

,,,,,,,,,,How do we proceed?
Formal Setup:Classification X 7→ Y , fixed VC class H
Given: source data {Xi , Yi } ∼ P nP , target data {Xi , Yi } ∼ QnQ .
Goal: ĥ ∈ H with small excess target error
EQ (ĥ) = EQ [ĥ(X) 6= Y ] − inf EQ [h(X) 6= Y ]h∈HBasic Information-theoretic Question:
Which EQ (ĥ) is achievable in terms of sample sizes nP and nQ ?
Which notion of dist(P → Q) captures this error?

,,,,,,,,,,

Many competing notions of dist(P → Q) ...
• Extensions of TV: consider |P (A) − Q(A)| over suitable A(e.g. dA divergence/Y-discrepancy of S. Ben David, M. Mohri, ...)EQ (ĥ) . oP (1) + dist(P → Q)
• Density Ratios: consider ratio dQ/dP over data space(e.g., Sugiyama, Belkin, Jordan, Wainwright, ...)EQ (ĥ) . oP (1) + estimation error(dQ /dP ),,,,,,,,,,

Many competing notions of dist(P → Q) ...

• Extensions of TV: consider |P (A) − Q(A)| over suitable A(e.g. dA divergence/Y-discrepancy of S. Ben David, M. Mohri, ...)EQ (ĥ) . oP (1) + dist(P → Q)
• Density Ratios: consider ratio dQ/dP over data space(e.g., Sugiyama, Belkin, Jordan, Wainwright, ...)EQ (ĥ) . oP (1) + estimation error(dQ /dP ),,,,,,,,,,

Many competing notions of dist(P → Q) ...

• Extensions of TV: consider |P (A) − Q(A)| over suitable A(e.g. dA divergence/Y-discrepancy of S. Ben David, M. Mohri, ...)EQ (ĥ) . oP (1) + dist(P → Q)
• Density Ratios: consider ratio dQ/dP over data space(e.g., Sugiyama, Belkin, Jordan, Wainwright, ...)EQ (ĥ) . oP (1) + estimation error(dQ /dP ),,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q ,,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q ,,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q ,,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q 

Large TV, dA , Y-disc ≈ 1/2

,,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q 

Asymmetry in transfer =⇒ Metrics are inappropriate

,,,,,,,,,,

Many notions: (TV, dA , Y-disc, KL, Renyi, MMD, Wasserstein ...)
They all tend to be over-pessimistic about transfer § Transfer is Hard=⇒
Namely: P far from Q 

Large dQ/dP , KL-div ≈ ∞,,,,,,,,,,

Relating source P to target Q [Hanneke, Kpo. 19]

Intuition: h ∈ H has low error under P =⇒ low error under Q
For now assume h∗P = h∗Q ...
Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h) ≤ c · EP (h)ρ captures a continuum of easy to hard transfer ...

,,,,,,,,,,

Relating source P to target Q [Hanneke, Kpo. 19]

Intuition: h ∈ H has low error under P =⇒ low error under Q
For now assume h∗P = h∗Q ...
Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h) ≤ c · EP (h)ρ captures a continuum of easy to hard transfer ...

,,,,,,,,,,

Relating source P to target Q [Hanneke, Kpo. 19]

Intuition: h ∈ H has low error under P =⇒ low error under Q
For now assume h∗P = h∗Q ...
Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h) ≤ c · EP (h)ρ captures a continuum of easy to hard transfer ...

,,,,,,,,,,

Relating source P to target Q [Hanneke, Kpo. 19]

Intuition: h ∈ H has low error under P =⇒ low error under Q
For now assume h∗P = h∗Q ...
Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h) ≤ c · EP (h)ρ captures a continuum of easy to hard transfer ...

,,,,,,,,,,

Relating source P to target Q [Hanneke, Kpo. 19]

Intuition: h ∈ H has low error under P =⇒ low error under Q
For now assume h∗P = h∗Q ...
Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h) ≤ c · EP (h)ρ captures a continuum of easy to hard transfer ...

,,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ ),,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ )For deterministic Y = h∗ (X) this reduces to:
QX (h 6= h∗ ) ≤ c · PX 1/ρ (h 6= h∗ ),,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ )

ρ = 1 but dA (P, Q) = Y-disc(P, Q) = 1/4

,,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ )

ρ = 1 but KL, Renyi, blow up ...

,,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ )

ρ > 1 ≡ how much P covers decision boundary,,,,,,,,,,

Examples:Transfer exponent ρ > 0:∀h ∈ H,

1/ρ

EQ (h, h∗ ) ≤ c · EP (h, h∗ )0 < ρ < 1 ≡ Super Transfer (P has better coverage of decision boundary)

,,,,,,,,,,

ρ captures performance limits (minimax rates) under transfer ...

,,,,,,,,,,

Performance depends on ρ + hardness of classification:

Easy to hard classification

Easy Classification

Hard Classification

Essential: Noise in Y |X, and X-mass near decision boundary
Bernstein condition:

QX (h 6= h∗ ) . EQ β (h; h∗ ),Similar noise condition on P .

β ∈ [0, 1]

,,,,,,,,,,

Performance depends on ρ + hardness of classification:

Easy to hard classification

Easy Classification

Hard Classification

Essential: Noise in Y |X, and X-mass near decision boundary
Bernstein condition:

QX (h 6= h∗ ) . EQ β (h; h∗ ),Similar noise condition on P .

β ∈ [0, 1]

,,,,,,,,,,

Performance depends on ρ + hardness of classification:

Easy to hard classification

Easy Classification

Hard Classification

Essential: Noise in Y |X, and X-mass near decision boundary
Bernstein condition:

QX (h 6= h∗ ) . EQ β (h; h∗ ),Similar noise condition on P .

β ∈ [0, 1]

,,,,,,,,,,

Performance depends on ρ + hardness of classification:

Easy to hard classification

Easy Classification

Hard Classification

Essential: Noise in Y |X, and X-mass near decision boundary
Bernstein condition:

QX (h 6= h∗ ) . EQ β (h; h∗ ),Similar noise condition on P .

β ∈ [0, 1]

,,,,,,,,,,

Performance depends on ρ + hardness of classification:

Easy to hard classification

Easy Classification

Hard Classification

Essential: Noise in Y |X, and X-mass near decision boundary
Bernstein condition:

QX (h 6= h∗ ) . EQ β (h; h∗ ),Similar noise condition on P .

β ∈ [0, 1]

,,,,,,,,,,

Minimax rates of Transfer: [Hanneke, Kpo. 2019]
Given: labeled source and target data {Xi , Yi } ∼ P nP × QnQ .
Theorem. Let ĥ trained on samples from P + Q:
−1/(2−β)1/ρinf sup EQ (ĥ) ∝ nP + nQĥ (P,Q)

Tight for any H, ρ ≥ 1, β, nP , nQ ...
• Benefits of Unlabeled data: cannot improve the rates ...
• Benefits of Labeled Q data: transition at nQ > n1/ρP
• Adaptive sampling at optimal $$ costs: possible in some regimes

,,,,,,,,,,

Minimax rates of Transfer: [Hanneke, Kpo. 2019]
Given: labeled source and target data {Xi , Yi } ∼ P nP × QnQ .
Theorem. Let ĥ trained on samples from P + Q:
−1/(2−β)1/ρinf sup EQ (ĥ) ∝ nP + nQĥ (P,Q)

Tight for any H, ρ ≥ 1, β, nP , nQ ...
• Benefits of Unlabeled data: cannot improve the rates ...
• Benefits of Labeled Q data: transition at nQ > n1/ρP
• Adaptive sampling at optimal $$ costs: possible in some regimes

,,,,,,,,,,

Minimax rates of Transfer: [Hanneke, Kpo. 2019]
Given: labeled source and target data {Xi , Yi } ∼ P nP × QnQ .
Theorem. Let ĥ trained on samples from P + Q:
−1/(2−β)1/ρinf sup EQ (ĥ) ∝ nP + nQĥ (P,Q)

Tight for any H, ρ ≥ 1, β, nP , nQ ...
• Benefits of Unlabeled data: cannot improve the rates ...
• Benefits of Labeled Q data: transition at nQ > n1/ρP
• Adaptive sampling at optimal $$ costs: possible in some regimes

,,,,,,,,,,

Minimax rates of Transfer: [Hanneke, Kpo. 2019]
Given: labeled source and target data {Xi , Yi } ∼ P nP × QnQ .
Theorem. Let ĥ trained on samples from P + Q:
−1/(2−β)1/ρinf sup EQ (ĥ) ∝ nP + nQĥ (P,Q)

Tight for any H, ρ ≥ 1, β, nP , nQ ...
• Benefits of Unlabeled data: cannot improve the rates ...
• Benefits of Labeled Q data: transition at nQ > n1/ρP
• Adaptive sampling at optimal $$ costs: possible in some regimes

,,,,,,,,,,

Minimax rates of Transfer: [Hanneke, Kpo. 2019]
Given: labeled source and target data {Xi , Yi } ∼ P nP × QnQ .
Theorem. Let ĥ trained on samples from P + Q:
−1/(2−β)1/ρinf sup EQ (ĥ) ∝ nP + nQĥ (P,Q)

Tight for any H, ρ ≥ 1, β, nP , nQ ...
• Benefits of Unlabeled data: cannot improve the rates ...
• Benefits of Labeled Q data: transition at nQ > n1/ρP
• Adaptive sampling at optimal $$ costs: possible in some regimes

,,,,,,,,,,

Lower-Bound Analysisĥ has access to (P, Q) samples, but has to do well on just Q ...

Construction: family {(P, Q)h }, any H, ρ ≥ 1, β:• (P nP × QnQ )h are close in KL-divergence• But far under distance Qh (h0 6= h)
The rest is extensions of Fano (see e.g. Tsybakov, or Barron and Li) ...

,,,,,,,,,,

Lower-Bound Analysisĥ has access to (P, Q) samples, but has to do well on just Q ...

Construction: family {(P, Q)h }, any H, ρ ≥ 1, β:• (P nP × QnQ )h are close in KL-divergence• But far under distance Qh (h0 6= h)
The rest is extensions of Fano (see e.g. Tsybakov, or Barron and Li) ...

,,,,,,,,,,

Lower-Bound Analysisĥ has access to (P, Q) samples, but has to do well on just Q ...

Construction: family {(P, Q)h }, any H, ρ ≥ 1, β:• (P nP × QnQ )h are close in KL-divergence• But far under distance Qh (h0 6= h)
The rest is extensions of Fano (see e.g. Tsybakov, or Barron and Li) ...

,,,,,,,,,,

Upper-bound Analysis:−1/(2−β)1/ρPerformance limits: EQ (ĥ) ∝ nP + nQ

(Optimal Heuristics for unknown ρ)
Low Classification noise (β = 1):
ERM on combined source and target data.
Non i.i.d. Bernstein + usual fixed point argument

Unknown Noise Level (β ∈ [0, 1]):Minimize R̂Q (h) subject to R̂P (h) ≤ minh0 R̂P (h0 ) + ∆nP (h)Lepski-type argument

,,,,,,,,,,

Upper-bound Analysis:−1/(2−β)1/ρPerformance limits: EQ (ĥ) ∝ nP + nQ
We are interested in adaptivity to ρ ...
(Optimal Heuristics for unknown ρ)
Low Classification noise (β = 1):
ERM on combined source and target data.
Non i.i.d. Bernstein + usual fixed point argument

Unknown Noise Level (β ∈ [0, 1]):Minimize R̂Q (h) subject to R̂P (h) ≤ minh0 R̂P (h0 ) + ∆nP (h)Lepski-type argument

,,,,,,,,,,

Upper-bound Analysis:−1/(2−β)1/ρPerformance limits: EQ (ĥ) ∝ nP + nQ

(Optimal Heuristics for unknown ρ)
Low Classification noise (β = 1):
ERM on combined source and target data.
Non i.i.d. Bernstein + usual fixed point argument

Unknown Noise Level (β ∈ [0, 1]):Minimize R̂Q (h) subject to R̂P (h) ≤ minh0 R̂P (h0 ) + ∆nP (h)Lepski-type argument

,,,,,,,,,,

Upper-bound Analysis:−1/(2−β)1/ρPerformance limits: EQ (ĥ) ∝ nP + nQ

(Optimal Heuristics for unknown ρ)
Low Classification noise (β = 1):
ERM on combined source and target data.
Non i.i.d. Bernstein + usual fixed point argument

Unknown Noise Level (β ∈ [0, 1]):Minimize R̂Q (h) subject to R̂P (h) ≤ minh0 R̂P (h0 ) + ∆nP (h)Lepski-type argument

,,,,,,,,,,

Upper-bound Analysis:−1/(2−β)1/ρPerformance limits: EQ (ĥ) ∝ nP + nQ

(Optimal Heuristics for unknown ρ)
Low Classification noise (β = 1):
ERM on combined source and target data.
Non i.i.d. Bernstein + usual fixed point argument

Unknown Noise Level (β ∈ [0, 1]):Minimize R̂Q (h) subject to R̂P (h) ≤ minh0 R̂P (h0 ) + ∆nP (h)Lepski-type argument

,,,,,,,,,,

Quick Summary and some New Directions ...

,,,,,,,,,,

Quick Summary:• ρ captures a more optimistic view of transferability P → Q.
• Reveals general form of optimal heuristics:
Minimize R̂P (h) subject to R̂Q (h) not too large ...
• Cost-sensitive sampling is possible with no knowledge of ρ.
• Results extend to h∗P 6= h∗Q :∃ĥ s.t.
no−1/(2−β)ρ
−1/(2−β)∗EQ (ĥ) . min nP+ EQ (hP ), nQ

,,,,,,,,,,

Quick Summary:• ρ captures a more optimistic view of transferability P → Q.
• Reveals general form of optimal heuristics:
Minimize R̂P (h) subject to R̂Q (h) not too large ...
• Cost-sensitive sampling is possible with no knowledge of ρ.
• Results extend to h∗P 6= h∗Q :∃ĥ s.t.
no−1/(2−β)ρ
−1/(2−β)∗EQ (ĥ) . min nP+ EQ (hP ), nQ

,,,,,,,,,,

Quick Summary:• ρ captures a more optimistic view of transferability P → Q.
• Reveals general form of optimal heuristics:
Minimize R̂P (h) subject to R̂Q (h) not too large ...
• Cost-sensitive sampling is possible with no knowledge of ρ.
• Results extend to h∗P 6= h∗Q :∃ĥ s.t.
on−1/(2−β)ρ
−1/(2−β)∗EQ (ĥ) . min nP+ EQ (hP ), nQ

,,,,,,,,,,

Quick Summary:• ρ captures a more optimistic view of transferability P → Q.
• Reveals general form of optimal heuristics:
Minimize R̂P (h) subject to R̂Q (h) not too large ...
• Cost-sensitive sampling is possible with no knowledge of ρ.
• Results extend to h∗P 6= h∗Q :∃ĥ s.t.
on−1/(2−β)ρ
−1/(2−β)∗EQ (ĥ) . min nP+ EQ (hP ), nQ

,,,,,,,,,,

Quick Summary:• ρ captures a more optimistic view of transferability P → Q.
• Reveals general form of optimal heuristics:
Minimize R̂P (h) subject to R̂Q (h) not too large ...
• Cost-sensitive sampling is possible with no knowledge of ρ.
• Results extend to h∗P 6= h∗Q :∃ĥ s.t.
on−1/(2−β)ρ
−1/(2−β)∗EQ (ĥ) . min nP+ EQ (hP ), nQ

,,,,,,,,,,

Current work:
Performance limits in multi-task (new arXiv with S. Hanneke)

P1 + P2 + · · · + PN + Q → Q?
Prior theory only yields single source rates ...

,,,,,,,,,,

Current work:
Performance limits in multi-task (new arXiv with S. Hanneke)

P1 + P2 + · · · + PN + Q → Q?
Prior theory only yields single source rates ...

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Setup:
ρt
N sources {Pt }N
t=1 7→ Q with Et (h) & EQ (h)Minimax Rate on EQ (ĥ) :mint∈[N +1]tX!−1/(2−β)ρ̄t
n(t)

s=1

Adaptive Strategies (as N → ∞):Low noise (β = 1): ERM on combined data
Information on ranking ρ(1) ≤ ... ≤ ρ(N ) : Greedy strategy ...
No adaptive strategy outside above regimes !!!
inf sup−1/(2−β)sup EQ (ĥ) & nQĥ rankings {Pt }×Q.

,,,,,,,,,,

Somehow we are still just scratching the surface ...

Thanks!

,,,,,,,,,,

Somehow we are still just scratching the surface ...

Thanks!