Few-shot learning with KRR
Prudencio Tossou
Groupe de Recherche en Apprentissage Automatique
Départment d’informatique et de génie logiciel
Université Laval

April 6, 2018

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 20181 / 25

Motivation

Plan

1

Motivation2

Problem statement

3

Our approach: MetaKRR

4Experiments

5

Future works and conclusion

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 20182 / 25

Motivation

The future of ML

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 20183 / 25

Motivation

What few learning is trying to do?
Distribution T

By leveraging past learning
experiences

TFamily F ={T1 , T2 , . . . , Ti . . .}TMeta-train{D1 , D2 , . . . , DM }TkThrough META-LEARNING

Meta-learningalgorithm

TThe past gives a strong priorknowledge

Training
data Dtrain

Meta-modelor learningalgorithm A

Input xModel hIf one use it, things can be done
more efficiently in the present
Output h(x)Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 2018

4 / 25

Motivation

Few-shot regression

Recent works focus on classification and reinforcement learning

Not much experiments with regression datasets

Versus classification: harder to generalize from few examples

Applications:,→ drug discovery –> Drugs at lower costs,→ recommender systems –> Deal with products with few ratings

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 2018

5 / 25

Problem statement

Plan

1

Motivation2

Problem statement

3

Our approach: MetaKRR

4Experiments

5

Future works and conclusion

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 2018

6 / 25

Problem statement

The objective
Distribution TT

Family F ={T1 , T2 , . . . , Ti . . .}TThe optimal meta-modelA∗= argmin
AEEEk (x,y )∼TT ∼T Dtrain ∼T

Meta-train{D1 , D2 , . . . , DM }Tk

L(A(Dtrain )(x), y )Meta-learningalgorithmTFew-shot regression
Quadratic lossL(y , y 0 ) = y − y 0

Training
data Dtrain

Meta-modelor learningalgorithm A

Input xModel h2|Dtrain | ≤ 20
Output h(x)Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 20187 / 25

Problem statement

In practice (1)Distribution TT

Family F ={T1 , T2 , . . . , Ti . . .}TThe meta-datasets
Sample a distribution Ti from F

Meta-train{D1 , D2 , . . . , DM }Tk

For each Ti sample a Di
Split the resulting collection ofdatasets in 3 partitions:,→ Dmeta−train for training,→ Dmeta−valid for
hyper-parameter selection,→ Dmeta−test for unbiased
evaluation of the meta-model

Meta-learningalgorithm

TTrainingdata Dtrain

Meta-modelor learningalgorithm A

Input xModel hOutput h(x)Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 20188 / 25

Problem statement

In practice (2)Distribution T

Episodic training

TFamily F ={T1 , T2 , . . . , Ti . . .}TInitialize Θ
Loop

Meta-train{D1 , D2 , . . . , DM }Sample an Di from Dmeta−train
Sample Dtrain and Dtest of k
examples each from Di

Tk
Meta-learningalgorithm

T

Compute h := A(Dtrain , Θ)Estimate the loss of h on Dtest

Training
data Dtrain

Meta-modelor learningalgorithm A

Input xModel hUpdate ΘAn episodeA pair of Dtrain and Dtest from a Di
Prudencio Tossou (UL)

Few-shot learning with KRR

Output h(x)April 6, 2018

9 / 25Our approach: MetaKRR

Plan

1

Motivation2

Problem statement

3

Our approach: MetaKRR

4Experiments

5

Future works and conclusion

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201810 / 25

Our approach: MetaKRR

The meta-model

Tandem combination
Feature extractor φ : X → K shared by all tasks
Regression Algorithm –> h(x) = w · φ (x), w ∈ K
Feature extractorCould be anything
CNN for images
LSTM for sequences
FC for vectors, etc
Parameters to be found during the episodic training

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201811 / 25

Our approach: MetaKRR

The model (1)The regression algorithm should aim for generalization (SRM) [4]Given Dtrain ,X∗
wD= argmin
trainw(w · φ (x) − y )2 + λ kwk22 ,(x,y )∈Dtrain

The optimal solution is given by KRR[3]w∗ =kX

αi φ (xi ),with α = (α1 , α2 , . . . , αk )T = (K + λI )−1 y,i=1

Kij = φ (xi ) · φ (xj ),Prudencio Tossou (UL)

where i = 1 . . . k, j = 1 . . . k

Few-shot learning with KRR

April 6, 201812 / 25

Our approach: MetaKRR

The model (2)Advantages of KRR
Closed form
Few-shot –> solving the dual system is highly advantageous over theprimal
Drawbacks
Fine tuning regularizer and kernel hyper-parameters
Cross-validation and validation set: costly and need more data

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201813 / 25

Our approach: MetaKRR

Selection of regression hyper-parameters
Option 1: Episode dependant,→ FC network g to predict the right values,→ inputs = sufficient statistics of the training examples of Dtrain,→ statistics = mean, std, max, min of {y1 , y2 , . . . , yk } andφ(x1 ), φ (x2 ), . . . , φ (xk )}{φ,→ For example, for a given Dtrain , the KRR regularizer is given by: a if x < ax if a ≤ x ≤ b
exp (HardTanha,b (g (Dtrain ))), with HardTanha,b (x) =
b if x > bOption 2: Same for all episodes
Associate a parameter to each Hp
Find the right value during back propagation
Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201814 / 25

Our approach: MetaKRR

MetaKRR: the training with option 1
Pseudo-code
Initialize Θ of φ and Λ of gLoop
Sample an Di from Dmeta−train
Sample a Dtrain and Dtest from Di
Transform all inputs with φCompute λtrain with gSolve KRR to find w∗ , thus h∗Compute the quadratic loss of h on Dtest
Back-propagate the loss and update Θ and Λ
Other details
Train for 20K episodes
Use Dmeta−valid to select the best model

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201815 / 25

Experiments

Plan

1

Motivation2

Problem statement

3

Our approach: MetaKRR

4Experiments

5

Future works and conclusion

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201816 / 25

Experiments

Datasets
MHC class II peptides

Binding molecules

Task: predict the binding energyof a peptide to a protein (MHCII complex).

Task: predict the binding affinityof small molecules to a protein

Collection of 14 datasets, one
per protein

Collection of 3741 regression
task each related to a protein
and an organism

Each dataset has from 500 to5K examples

Input = molecule SMILES(string)

Input = peptide (string)

Output = binding affinity

Output=energy to a MHC
protein14 few-shot regression tasks

Meta-train, meta-valid andmeta-test contain 2104, 702 and935 tasks

CNN feature extractor 256 × 3

CNN feature extractor 512 × 4

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201817 / 25Experiments

Results on MHC class II peptides
MetaKRR-g

MetaKRR-u

MAML[1]

MANN[2]

pretrain

0.435
0.512
0.547
0.573
0.643
0.694
0.404
0.509
0.641
0.471
0.640
0.318
0.574
0.660
0.544

0.475
0.501
0.555
0.608
0.652
0.694
0.388
0.535
0.626
0.477
0.629
0.356
0.602
0.624
0.552

0.469
0.405
0.457
0.470
0.531
0.613
0.407
0.389
0.567
0.401
0.623
0.294
0.548
0.559
0.481

0.530
0.522
0.484
0.617
0.676
0.673
0.426
0.565
0.537
0.465
0.644
0.313
0.596
0.669
0.551

0.176
-0.177
0.165
0.105
0.156
0.199
0.125
0.159
-0.169
0.116
0.180
0.071
0.203
0.221
0.109

Test complex
DRB1*0101
DRB1*0301
DRB1*0401
DRB1*0404
DRB1*0405
DRB1*0701
DRB1*0802
DRB1*0901
DRB1*1101
DRB1*1302
DRB1*1501
DRB3*0101
DRB4*0101
DRB5*0101
Average
Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201818 / 25

Experiments

Results on BindingDB

MetaKRR versus MAML

Prudencio Tossou (UL)MetaKRR versus MANN

Few-shot learning with KRR

April 6, 201819 / 25

Experiments

Discussion

MAML

Hard to find the best
initialization point.

Prudencio Tossou (UL)

MANN

Easy to classify with but harderfor regression

Few-shot learning with KRR

April 6, 201820 / 25

Future works and conclusion

Plan

1

Motivation2

Problem statement

3

Our approach: MetaKRR

4Experiments

5

Future works and conclusion

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201821 / 25

Future works and conclusion

Future works

Select the right values with the network g within a grid ofhyper-parameters
Include in our experiments a recommender system dataset : Netflix
challenge

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201822 / 25

Future works and conclusion

Conclusion

We have introduced MetaKRR, a few-shot regression algorithm
State of the art performances
Three key ideas:
Leverage past experiences to find the most appropriate mapping
function
Use the structural risk minimization to enforce generalization
Leverage past experiences to choose adequately the trade-off inside theSRM

Not new ideas but they graciously combine together to give theMetaKRR

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201823 / 25

Future works and conclusion

References[1] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning forfast adaptation of deep networks. arXiv preprint arXiv:1703.03400,
2017.
[2] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap.
Meta-learning with memory-augmented neural networks. In
International conference on machine learning, pages 1842–1850, 2016.
[3] C. Saunders, A. Gammerman, and V. Vovk. Ridge regression learningalgorithm in dual variables. 1998.
[4] V. Vapnik. Principles of risk minimization for learning theory. In
Advances in neural information processing systems, pages 831–838,1992.

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201824 / 25

Questions

Thanks for your attention

Prudencio Tossou (UL)

Few-shot learning with KRR

April 6, 201825 / 25