JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

1

Learning Categories from Few Examples with
Multi Model Knowledge Transfer
Tatiana Tommasi, Francesco Orabona and Barbara Caputo
Abstract—Learning a visual object category from few samples is a compelling and challenging problem. In several real-world
applications collecting many annotated data is costly and not always possible. However a small training set does not allow to cover thehigh intraclass variability typical of visual objects. In this condition, machine learning methods provide very few guarantees. This paper
presents a discriminative model adaptation algorithm able to proficiently learn a target object with few examples by relying on otherpreviously learned source categories. The proposed method autonomously chooses from where and how much to transfer information
by solving a convex optimization problem which ensures to have the minimal leave-one-out error on the available training set. We
analyze several properties of the described approach and perform an extensive experimental comparison with other existing transfer
solutions, consistently showing the value of our algorithm.
Index Terms—Knowledge Transfer, image categorization, discriminative learning

F

1

I NTRODUCTIONA

S human beings, our learning ability develops progressively in time. At the age of six, we recognize around
104 object categories [1] and we go on learning more while
we grow up. All the information acquired through our fivesenses are encoded and stored in memory, with concepts andcategories organized on the basis of their common properties.
This means that any new concept is not learned in isolation,
but considering connections to what is already known, whichmakes the skill of building analogies one of the cores of human
intelligence [2]. Even focusing only on visual tasks, we cangive several examples of this cognitive ability. Have you everseen a guava or an okapi? The guava is a fruit that externally
looks like a lime, while its inner part is similar to an apple. An
okapi is an animal that can be roughly described as a horse,with the legs of a zebra and the head of a giraffe (see Figure1). Once we have seen a single image for each of the twotarget objects, we can easily memorize and recognize themby referring to the source objects mentioned in the provided
description. In psychology this process is known as knowledgetransfer: it encompasses phenomena ranging from simple (e.g.
generalization of conditioned response between familiar andnovel stimuli) to extremely complex (e.g. carrying over asolution from a problem in arithmetic to a novel class ofproblems) behaviors [3], and it makes learning further concepts extremely efficient. This capacity allows us to mine manykinds of recurrent patterns and to make inductive inferences• B. Caputo is with the University of Rome La Sapienza, Dept. of Computer, Control and Management Engineering, Rome, Italy. E-mail: caputo@dis.uniroma1.it• T. Tommasi is with KU Leuven, ESAT-PSI and iMinds, Belgium. Email:tatiana.tommasi@esat.kuleuven.be• F. Orabona is with the Toyota Technological Institute at Chicago, USA
Email: francesco@orabona.com
For T.T and B.C. most of this work was done while at Idiap Research Institute,
Martigny, Switzerland.

Fig. 1. Two examples of using some source knowledge
on fruits and animals while learning the target objects
guava and okapi.
on a new task even with only a small amount of data.
A large part of recent literature on visual object categorization focuses on reaching impressive results on large
and difficult datasets [4], [5]. However, these works rarelyrefer to the effort done in collecting the data. In many realapplications gathering fully annotated images can be extremely
time consuming and might have a significant impact on theoverall cost of the final system. On the other hand, standard
learning techniques do not handle well the case of very small
training sets. Differently from the described cognition mechanism, all the learning approaches consider each task separately
with respect to other possible source of relative information.
Reproducing the knowledge transfer process in this scenario
might consistently boost the learning performance. The basic
intuition is that, if a system has already learned j categories,
learning the (j + 1)-th should be easier even from one or few
training samples [6].
A first practical implementation of the knowledge transfer
idea was presented in [7] following a Bayesian approach. A
generic object model is estimated from some source categories
and it is then used as prior to evaluate the target object
parameter distribution with a maximum-a-posteriori technique.
This work left some open questions discussed in its conclusive
section: (i) All the different known source categories are used

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

2

together to define a single prior; would a more sophisticatedmulti-modal prior be beneficial in learning? (ii) Is there anyother productive point of view beside the generative Bayesian
one that allows to incorporate prior knowledge? (iii) Is it easierto learn new target categories which are similar to some ofthe source categories? Several other works in the computer
vision literature followed this first attempt [8], [9], [10], [11],introducing different methods to increase the categorization
performance with respect to learning from scratch in case of
few available samples. However, due to the small differencesin the chosen settings, the proposed solutions were nevercompared among each other.
In this work we focus on knowledge transfer across visualobject categories and our main contribution is a learningalgorithm that directly addresses the open problems in [7]. We
consider (i) the availability of several separate source models
and we introduce (ii) a discriminative approach based on
Least Square Support Vector Machines (LS-SVM, [12]). Any
new target class is learned through adaptation by imposing
closeness between the target classifier and a linear combinationof the source classifiers already learned on the j object sources.
The weight assigned to each source knowledge is defined bysolving a convex optimization problem which minimizes anupper bound of the leave-one-out error on the training set.
This provides a principled solution for choosing from whereto transfer and how much to rely on each known source.
In practice, the proposed method (iii) autonomously tunes
the transfer process depending on the similarity between thesources and the target tasks. We analyze in detail several
properties of the described approach and perform an extensive
experimental comparison with other existing transfer solutions,consistently showing the value of our algorithm.
The rest of the paper is organized as follows. Section 2
provides a short introduction to the goals, challenges andpossible scenarios of knowledge transfer. Section 3 briefly
reviews the literature. A detailed description of the notationand of the mathematical framework for our method follows
in Section 4. Section 5 contains the formal definition of
our knowledge transfer algorithm. Section 6 introduces anextension to the case of heterogeneous sources. Finally in
Section 7 we present a thorough experimental evaluation,
benchmarking against several other state of the art approaches.
Section 8 concludes the paper with an overall discussion andpointing out possible avenues for future research.

2 K NOWLEDGE
S CENARIOST RANSFER :I SSUESAND

The main assumption in theoretical models of learning, suchas the standard PAC (Probably Approximately Correct [13])model, is that training instances are drawn according to thesame probability distribution as the unseen test examples. Thishypothesis permits to estimate the generalization error and theuniform convergence theory [14] provides basic guarantees on
the correctness of future decisions.
This ideal assumption is not always true in practical problems. It can happen that we have a lot of annotated data on a

Fig. 2. Three ways in which transfer might improve thelearning performance when the number of target training
samples increases. Forcing the target learning process torely on unrelated sources produces the negative transfer
effect. (Figure reproduced and adapted from [16]).

Fig. 3. A scheme of the possible transfer learning conditions in visual object categorization. The number ofsource sets can increase with different possible levelsof relatedness with respect to the target category. The
tasks are heterogeneous (homogeneous) if the samples
are represented with different (the same) descriptors.
The target task can be supervised with an increasing
number of training samples or unsupervised when thetarget samples are not annotated.
source problem and the need to solve a different target problem with few labeled samples, where source and target present
a distribution mismatch. In this case knowledge transfer (a.k.atransfer learning [15]) may decrease the effort of collecting
new samples, while at the same time it may reduce the risk ofoverfitting by leveraging over the existing source knowledge
to solve a target task. It is possible to define three measuresby which transfer may improve the effectiveness of learning(see Figure 2):(1) Higher start: the initial performance achievable on thetarget task is much better compared to that of an ignorantagent [16]. This is true even using only the source transferredknowledge, before any further learning on the target problem.
(2) Higher slope: this indicates a shorter amount of timeneeded to fully learn the target task, given the transferredknowledge, in comparison with learning from scratch [16].
(3) Higher asymptote: in the long run, the final performancelevel achievable over the target task may be higher comparedto the final level without transfer [16].
How to get these advantages and up to which extent thetransfer process can be useful depends on the specific scenario

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

at hand (object categorization, detection, reinforcement learning, etc.) and on the relation between source and target tasks.
Apart from the different levels of semantic similarity, source
and target might be represented with the same or with different
descriptors which give rise respectively to a homogeneous or aheterogeneous transfer process. Moreover, a transfer learning
problem can scale with respect to the number of annotatedtarget samples and of possible source sets (see Figure 3).
Indeed, to fully define any knowledge transfer method it isnecessary to answer to three main questions.
(1) What to transfer? It refers to which knowledge can betransferred and to the form in which it is coded. In generalterms, some knowledge might be specific for a task while someother knowledge might be common and shareable.
(2) How to transfer? This question is about the definition ofa learning algorithm that can properly incorporate the source
knowledge while building on the target samples.
(3) When to transfer? Finally, it is always necessary toevaluate the differences among the source and the target task
and question whether the transfer is worthwhile or not.
In the following section we review the knowledge transfer
literature, referring to how each of the proposed methodaddresses these challenging questions.

3

R ELATED W ORK

The fundamental motivation for knowledge transfer in the fieldof artificial learning was discussed in a NIPS-95 workshop
on learning to learn [17] which focused on the need foropen ended learning systems that retain and reuse previously acquired knowledge. Since then, research on this topic
has attracted more and more attention and several transfer
approaches has been proposed in machine learning, natural
language processing and computer vision.

3

has proven to be extremely useful in the deep learning framework for unsupervised classification tasks [23]. In this settingsome recent work proposed also to represent object categories
indirectly by their attributes [24]. An attribute is a high level
semantic information (e.g. striped, furry) that is shared bymultiple object categories and can be easily transferred as a
descriptor.
Finally, a parameter or model transfer approach assumes
that the source tasks and the target tasks share some parameters
or prior distributions of the models. As already mentioned, FeiFei et al. [7] proposed to transfer information via a Bayesianprior on object class models, using knowledge from knownclasses as a generic reference for newly learned models. Starket al. [10] defined a technique to transfer a shape model acrossobject classes. Yang et al. [25] presented a method to transfer
the source information originally coded into an SVM model.
3.2

A large variety of methods have been studied to integrate
in different ways the source and target information: boosting
approaches [18], [9], KNN [26], Markov logic [27], graphicalmodels [28]. Most of the work has however been done in
the generative probabilistic setting. Given the data, the target
model makes predictions by combining them with the priorsource distribution to produce a posterior distribution. A
strong prior significantly affects these results serving as anatural way for Bayesian learning methods to transfer source
knowledge. Some discriminative (maximum margin) methods
are presented in [21] by learning a distance metric, and in[25] by exploiting a pre-learned SVM model. Also [11], [29]proposed to use a template learned previously for some object
categories to regularize the training of a new target category.
3.3

3.1

What to Transfer

Depending on the problem to solve, the transferred knowledge
can be in the form of instances, feature representation, ormodel parameters [15].
The main idea at the basis of instance transfer approaches
is that, although no all the source data are useful, there arecertain parts of them that can still be selected and considered
together with the few available target labeled samples. In [18]Dai et al. proposed a boosting algorithm that uses both thesource and the target data to solve visual object classification
problems. Lim et al. [19] have shown that it is possible
to borrow and transform examples across different visualobject classes, demonstrating a performance improvement in
detection problems.
Any feature transfer approach consists in learning a good
representation for the target domain encoding in it somerelevant knowledge extracted from the source. Bart and Ullman [20] proposed to perform feature adaptation using asingle example of a novel class and showed a significant
gain in classification performance. An alternative solution is
to consider directly a metric learning approach [21] or more in
general to exploit suitable kernels for the target data in SVMbased methods [22]. Moreover, the feature transfer approach

How to Transfer

When to Transfer

In real learning scenarios, the information acquired in the past
is not always relevant for a new target problem. Rosenstain
et al. [29] empirically showed that if two tasks are dissimilar,
brute force transfer hurts the performance producing the socalled negative transfer (see Figure 2). Ideally, a transfer
method should be beneficial between appropriately related
tasks while avoiding negative transfer when the tasks are nota good match. In practice, these goals are difficult to achieve
simultaneously. Approaches that have safeguards against negative transfer often produce a smaller effect from positivetransfer due to their caution. Conversely, approaches thattransfer aggressively and produce large positive-transfer effects
often have no protection against negative transfer.
It is possible to identify two main strategies to decide whento transfer. One consists in rejecting bad information or at least
making sure that its impact is minimized. This means choosing
always how much to transfer, and disregard completely thesource knowledge if not relevant for the target. A different
strategy can be applied when there are more than one source
task: in this condition the problem becomes choosing thebest source. Transfer methods without much protection against
negative transfer may still be effective in this scenario, as longas the best source task is at least a decent match. Taylor etJOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

4

al. [30] proposed a transfer hierarchy, sorting the tasks bydifficulty. Given a task ordering, it may be possible to locate
the position of the target task in the hierarchy and select themost useful source set. In [31] the authors used conditional
Kolmogorov complexity to measure relatedness between tasks
and transfer the right amount of information.
Our work fits in this context. We propose a discriminativeknowledge transfer method that relies on a set of models
learned on the source categories (what to transfer) which
are then used to regularize the target object model (how totransfer). The relatedness among the tasks is automaticallyevaluated (when to transfer) through a principled optimization
problem without any need of hand tuned parameters, extra
validation samples or a pre-defined ontology.

4

M ATHEMATICAL F RAMEWORK

We introduce here the formal notation and the necessary mathematical tools used in the rest of the paper. In the following wedenote with small and capital bold letters respectively column
vectors and matrices, e.g. a = [a1 , a2 , . . . , aN ]T ∈ RN andA ∈ RM ×N with Aji corresponding to the (j, i) element.
When only one subscripted index is present, it represents thecolumn index, e.g., Ai is the i-th column of the matrix A.
P
1/p
N
p
Moreover we indicate with kakp :=|a|the pii=1
N
norm of a vector a ∈ R .
Let us assume xi ∈ X to be an input vector to a learning
system and yi ∈ Y its associated output. Given a set ofdata D = {xi , yi }N
i=1 drawn from an unknown probability
distribution P , we want to find a function f : X → Y such thatit determines the best corresponding y for any future sample
x. We consider X ⊆ Rd and Y = {−1, 1}.
The described learning process can be formalized as an optimization problem which aims at finding f in the hypothesis
space of functions H, which minimizes the structural risk [14]Ω(f ) + C

N
X`(f (xi ), yi ) .

(1)

i=1

Here Ω(f ) is a regularizer, which encodes some notion ofsmoothness for f , and guarantees good generalization performance avoiding overfitting. In the second term, ` is someconvex non-negative loss function which assesses the qualityof the function f on the instance and label pair {xi , yi }. In
practice it expresses the price we pay by predicting f (xi ) inplace of yi . The predictivity is a trade-off between fitting thetraining data and keeping the complexity of the solution low,
controlled by the parameter C > 0.
4.1

Adaptive Regularization

We set H equal to space of all the linear models of the formf (x) = w> φ(x) + b .

(2)Here φ(x) is a feature mapping that maps the samples intoa high, possible infinite dimensional space, where the dot
product is expressed with a functional form K(x, x0 ) =φ(x)> φ(x0 ) named kernel [32]. We also set the regularizer

to be Ω(f ) = 21 kwk2 , so that, regardless of the specific formof the loss function, the learning problem (1) becomesNX
1
min kwk2 + C`(w> φ(xi ) + b, yi ) .
w 2
i=1

(3)

In this classical scheme for inductive learning, the knowledge
eventually gained on the data D̂ = {x̂i , ŷi }N̂
i=1 extracted froma distribution P̂ , different with respect to the target one P ,is not taken into consideration. However, if N̂  N witha small number of available samples N (∼ 10) and if thetwo distributions P , P̂ are somehow related, the auxiliary
knowledge can be helpful in guiding the learning process.
Let us suppose that the optimal ŵ has been already foundby minimizing (3) for some source problem. When facing a
new target task, we can always ask w to be close to the knownŵ by simply changing the regularization term [33] such thatthe learning problem results
NX
1
2`(w> φ(xi ) + b, yi ) .
min kw − ŵk + C
w 2
i=1

(4)

Thus, the optimization problem aims now at obtaining a vector
w close to the source model ŵ by maximizing the projectionof the first on the second. To properly scale the importanceof this projection in the optimization problem, it is possible
to add a weighting factor β such that the regularizer becomeskw − β ŵk2 .

5

M ULTI M ODEL K NOWLEDGE T RANSFER

Consider the following situation. We want to learn the target
object class okapi from few examples, having already a modelfor the source categories horse, zebra, melon and apple. On
the basis of the visual similarity, we can guess that the
final model for okapi will be close to that of horse andzebra. Thus in the learning process we would like to transfer
information from these two categories. We would expect themodel obtained in this way to produce better recognition
results with respect to (i) just considering horse or zebra as
reference, and (ii) relying over all the source knowledge in aflat way, as melon and apple might induce negative transfer.
This kind of reasoning motivates us to design a knowledgetransfer algorithm able to find autonomously the best subset of
known models from where to transfer, and to weight properly
the relevant information.
Any transfer method, based on the adaptive regularization
described in the previous section, answers the question what totransfer in terms of model parameters, by passing the knownŵ to the new target problem. However, previous work did not
pay too much attention on when and how much to transfer.
The discussed weight factor β in the regularizer is usually
set equal to 1 with the hypothesis that the known models areuseful and related to the target problem [25]. In other cases
β is treated as a learning parameter, and is chosen by crossvalidation assuming the availability of extra target training
samples [11]. Both these choices present some issues: thefirst case does not consider the danger of negative transfer
when only unrelated prior information is available, while inJOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

5

the second, the existence of extra data for cross validation is
incoherent with the small sample scenario of transfer learning.
Here we study instead the case of multiple (J) available
sources. We propose a learning method which relies over allof them and assigns to each a weight βj for j = 1, . . . , J.
These values are automatically tuned on the basis of the few
available target training data. We name our algorithm Multi
Model Knowledge Transfer (Multi-KT) and we present its
basic components in the following subsections.
5.1

Adaptive Least-Square Support Vector Machine

The first step to define our transfer learning algorithm
PJ consistsin combining linearly the source models to have j=1 βj ŵjand using this as reference instead of the single source in (4).
Moreover, we choose the weighted square loss `(f (xi ), yi ) =ζi (f (xi ) − yi )2 [34], where the parameter ζi can be used
to balance the contribution of positive and negative samples,taking into account that their proportion in the training set may
be not representative of the operational class frequency.
The obtained optimization problem is:JX
1
βj ŵj
w−
min
w,b 2
j=1

2

where y and ŷ j are the vectors containing respectively
the label of each training sample and the predictionof the previous model j, i.e. y = [y1 , . . . , yN ]> ,>>ŷ j = [ŵ>=j φ(x1 ), . . . , ŵ j φ(xN )] . Moreover, Z−1 −1−1diag{ζ1 , ζ2 , . . . , ζN } and to balance the contribution ofdifferently labeled samples to the misfit term we set
 N
if yi = +1
2N +
(9)
ζi =
N
if yi = −1 .
−2NHere N + and N − represent the number of positive andnegative examples respectively.
Finally, the model parameters can be calculated simply by
matrix inversion:




PJ
a
y − j=1 βj ŷ j
=P
,
(10)
b
0
where P = M −1 and M is the first matrix on the left in (8).
We underline that the pre-trained models ŵj can be obtainedby any training algorithm, as long as it can be expressed as aweighted sum of kernel functions; the framework is therefore
very general.

N

+

CX 2
ζi ξ
2 i=1 i

5.2

When and How Much to Transfer

Finding the optimal value for the elements of the weight vector
β corresponds to ranking the prior knowledge sources andwhere we have introduced the slack variables ξi which mea- decide from where and how much to transfer. We propose
sure the degree of misclassification on the data xi . Thus we to choose β in order to minimize the leave-one-out error,
obtain a new formulation for Least Square Support Vector which is an almost unbiased estimator of the generalization
Machine (LS-SVM [12]), that uses the adaptive regularizer error [34]. While in general computing the leave-one-out error
is a very expensive procedure, we show that for (5) it can
introduced before. The corresponding Lagrangian L is
be obtained with a closed-formula, using quantities that are
J
N
N
X
X
X
1
already computed for the training.
2 C
2
>
kw−
βj ŵj k +ζi ξi −
ai (w φ(xi )+b+ξi −yi ) .
2
2
Let us denote by ỹi , i = 1, . . . , N , the prediction on
j=1
i=1
i=1
sample i when it is removed from the training set. LS-SVM
Here a ∈ RN is the vector of Lagrange multipliers and the in its original formulation makes it possible to write these
optimality condition with respect to w isleave-one-out predictions in closed form and with a negligible
additional computational cost [34]. We show below that the
J
N
X
X
∂L
= 0 =⇒ w =βj ŵj +ai φ(xi ) .
(6) same property extends to the modified problem in (5).
∂w
Proposition 1: Let [a0> , b0 ]>=PP[y > , 0]> and
j=1
i=1
J
>
00> 00 >>0[aj , bj ] = P [ŷ j , 0] with a = a − j=1 βj a00j . If
Thus, the adapted model is given by the weighted sum of the00
we indicate with A the matrix containing the vector a00>
inj
pre-trained source models ŵj and a linear combination of thethe j-th row, the prediction ỹi , obtained on sample i when ittarget samples. Note that when all the βj are 0 we recover theis removed from the training set, is equal tooriginal LS-SVM formulation without any adaptation. Considering also the derivative of L with respect to ξi and ai , we
β > A00ia0,(11)yi − i +have respectively ai = Cζi ξi and w> φ(xi ) + b + ξi − yi = 0.
Pii
Pii
By combining them with (6) we findwhere β ∈ RJ is a vector containing all the values βj .
J
N
X
X
ai
Proof of Proposition 1: We start fromak φ(xk )> φ(xi ) + b += yi −
βj ŵ>j φ(xi ) . (7)

 

PJ
Cζi
j=1
k=1
a
y − j=1 βj ŷ j
M
=
,
(12)
b
0
Denoting with K the kernel matrix, i.e. K ji = K(xj , xi ) =φ(xj )> φ(xi ), the obtained system of linear equations can be and we decompose M into block representation isolating thewritten more concisely in matrix form as
first row and column as follows:


 

PJ

 

1
K + CZ 1ay − j=1 βj ŷ j
K + C1 Z 1
m11
m>
1
=
,
(8)
=
.
M=
b
1>
0
m1 M(−1)
0
1>
0
s.t.

yi = w> φ(xi ) + b + ξi , ∀i = 1, . . . , N,

(5)

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

6

Let a(−i) and b(−i) represent the model parameters
during the i-th iteration of the leave-one-out cross validation
procedure. In the first iteration, where the first training sample
is excluded we have


J
X
a(−1)
= P(−1) (y (−1) −βj ŷ j(−1) ),
b(−1)
j=1

M−1
(−1)

where P(−1) =, y (−1) = [y2 , . . . , yN , 0]> and>>ŷ j(−1) = [ŵj φ(x2 ), . . . , ŵ>j φ(xN ), 0] . The leave-one-out
prediction for the first training sample is then given by
 X

J
a(−1)
>
ỹ1 = m1+
βj ŵ>j φ(x1 )
b(−1)
j=1


y (−1) −= m>
1 P(−1)JXβj ŷ j(−1)  +

j=1

J
X

βj ŵ>j φ(x1 ) .

j=1

Considering the last N equations in the system
PJ in (12), it isclear that [m1 M(−1) ][a> , b]> = (y (−1) − j=1 βj ŷ j(−1) ) ,and so>ỹ1 = m>1 P(−1) [m1 M(−1) ][a1 , . . . , aN , b] +

J
X

=

J
X

min
βj ŵ>j φ(x1 ) .

j=1

Infirst equation of the system isy1 −PJ (12) the
>
>
>
β
ŵ
φ(x
)
=
m
a
+
m
[a
,
.
.
.
,
a
,
b]
,
and
we
1
11 1
2
N
1
j=1 j j>have ỹ1 = y1 − a1 (m11 − m1 P(−1) m1 ) . Finally, by using
P = M−1 and the block matrix inversion lemma we getµ−1−µ−1 m1 P(−1)P=,−1
P(−1) + µ−1 P(−1) m>
P(−1) m>1 m1 P(−1) −µ1where µ = m11 − m>
1 P(−1) m1 . By noting that the systemof linear equations (12) is insensitive to permutations of theordering of the equations and of the unknowns, we haveaiỹi = yi −
.
Pii
00 >
By defining [a0> , b0 ]> = P [y > , 0]> , [a00>=
j , bj ]
PJ
>
>
0
00
P [ŷ j , 0] , and a = a − j=1 βj aj , we getỹi = yi −where |x|+ = max{0, x}. It is a convex upper bound to theleave-one-out misclassification loss and it favors solutions inwhich ỹi has an absolute value equal or bigger than 1, andthe same sign of yi . The weights ζi are set again according to(9). Finally, the objective function is

βj ŵ>j φ(x1 )

j=1
>
>
m>
1 P(−1) m1 a1 + m1 [a2 , . . . , aN , b] +Algorithm 1 Projected Sub-gradient Descent Algorithm
00
1: Input: Set a0 , a00
j , and A according to Proposition 1
2: Initialize: β ← 0 and t ← 1
3: repeat
PJ
A00
a0
ji
∀ i = 1, . . . , N
4:ỹi ← yi − Piii + j=1 βj Pii
5:di ← 1{yi y˜i > 0} ,∀ i = 1, . . . , N
PN
a00
6:
βj ← βj − √1t i=1 di yi Pji,∀ j = 1, . . . , Jii
7:if kβk2 > 1 then8:β ← β/kβk2
9:
end if10:
βj ← max(βj , 0),∀ j = 1, . . . , J
11:
t←t+1
12: until convergence
Output: β

J
X
A00ji
a0
β > A00i
a0i+
βj= yi − i +,Pii j=1 Pii
Pii
Pii

where β ∈ RJ is a vector containing all the values βj and A00
is the matrix containing the vector a00>
in the j-th row.
j
Notice that in (11) a depends linearly on β, thus it isstraightforward to obtain the learning model once all the βjhave been chosen. The best values for βj are those producing
positive values for yi ỹi , for each i. However, focusing only
on the sign of those quantities would result in a non-convex
formulation with many local minima. We propose instead thefollowing loss function, similar to the hinge loss
a0 − β > A00i`(ỹi , yi ) = ζi |1 − yi ỹi |+ = ζi yi i
Pii

,
+

(13)

β

N
X

`(yi , ỹi ) subject to kβkp ≤ 1 , βj ≥ 0 ,

(14)

i=1

where we added some constraint on the β vector as a formof regularization. They may be helpful to avoid overfitting
problems when the number of known models J is large
compared to the number of training samples N . Dependingon the value of p, how the target learning model leverages
over the source models changes:
p = 2, L2 norm constraint. This is the well known
Euclidean norm indicated by k · k2 or simply k · k. A regularization based on it generally induces numerical stability. The
optimization process can be implemented by using a projected
sub-gradient descent algorithm, where at each iteration β isprojected onto the L2 -sphere kβk ≤ 1, and then on the positive
orthant. The pseudo-code is in Algorithm 1.
p = 1, L1 norm constraint. This is simply the sum of theabsolute values of the vector elements. This constraint induces
a sparse solution, i.e. only some vector elements remain different from zero. Applied on prior knowledge regularization,
the condition kβk1 ≤ 1 can be easily implemented (e.g. byusing the algorithm in [35]), and it gives rise to an automatic
source selection technique.
p = ∞, L∞ norm constraint. This norm is defined as
kxk∞ := max{|x1 |, . . . , |xd |}.

(15)

In practice, by using kβk∞ ≤ 1 as constraint, all the vector
elements will have an absolute value not bigger than one. In
this case the projection consists of a simple truncation.
The second condition in (14) limits the weights of the source
knowledge models to be positive. In fact, in the object category
detection problem, all the considered source and target sets
have the background category as common negative class, thus
it is reasonable to expect that the angle between w and any
ŵj is always acute.

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

5.3

Computational Complexity

From a computational point of view the runtime of the MultiKT algorithm is O(N 3 +JN 2 ), with N the number of training
samples, and J the number of source models. The first termis related to the evaluation of the matrix P , which must
anyway occur while training, while the second term is thecomputational complexity of (11), which results negligible,
if compared to the complexity of training. Thus, we matchthe complexity of a plain SVM, which in the worst case is
known to be O(N 3 ) [36]. The computational complexity ofeach step of the projected sub-gradient descent to optimize(13) is O(JN ), and it results extremely fast (our MATLAB
implementation takes just half a second with N = 12 andJ = 3 on current hardware).

6

H ETEROGOENOUS K NOWLEDGE T RANSFER

7

bination of all the known models results:JX
βj ŵ0j = [β1 ŵ1 , . . . , βJ ŵJ ]>
j=1

= [β1

N̂1
X

αi1 φ1 (xi ), . . . , βJ

i=1

j=1

j=1

where Kj (x, z) is the kernel function in the j-th space.
Now let us consider the transfer learning problem with
j = 1, . . . , J source object classes and suppose to solve thebinary classification object-vs-background for each of themin a specific space, i.e. choosing different feature descriptors,
different kernel functions, and/or different kernel parameters.
The obtained model vectors are
N̂j
X
ŵj =
αij φj (xi ) .
i=1

These solutions can always be mapped in the composed
new space using zero padding. In fact, φj (x) → φ0j (x) =[0, . . . , φj (x), . . . , 0]> , and we haveŵj → ŵ0j = [0, . . . , ŵj , . . . , 0]>= [0, . . . ,N̂jX

αij φj (xi ), . . . , 0]> .

i=1Hence, in the new space, a vector obtained as linear com-

αiJ φJ (xi )]> .

i=1

By supposing that the target problem lives in the new composed space, we can apply our Multi-KT algorithm there.
Hence the original optimization problem in (5) becomes
J
X
1
0
w
−
βj ŵ0jmin
w0 ,b 2
j=1

2

N

+

CX
ζi (yi − w0> φ0 (xi ) − b)2 .
2 i=1

The solving procedure is the same described in Section 5.1
and the optimal solution is:
J
N
X
X
0
0
w =
βj ŵj +ai φ0 (xi ) .
j=1

The proposed Multi-KT transfer method is based on the ideaof pushing the target model
PJ w close to a linear combinationof prior known sources j=1 βj ŵj . However, to impose thiscloseness, all the vectors should live in a single space. Thismeans that the kernel used in learning over all the sourcesand on the new target must be the same. This is quite a strict
condition because it does not give the freedom to build thesource knowledge over heterogeneous feature descriptors, andimposes a unique metric to evaluate the sample similarity. In
this section we show how to overcome this limit by enlarging
the space in which the learning function lies, by a multi-kernel
approach. We call this variant MultiK-KT.
Assume to have j = 1, . . . , J mappings, each to a different
space, where the image of a vector x is φj (x). We can always
compose all of them orthogonally (see Figure 4) obtaining
the mapping to the final space by concatenation: φ0 (x) =[φ1 (x), φ2 (x), . . . , φJ (x)]> . The dot product φ0 (x)> φ0 (z) inthis new space is equal to the kernel K 0 , defined as
J
J
X
X
0
>
K (x, z) =φj (x) φj (z) =Kj (x, z),
(16)

N̂J
X

i=1

When we use it for classification we getw0> φ0 (z) =

=

J
X

0
βj ŵ0>j φ (z) +

N
X

j=1

i=1

J
X

N
X

βj ŵ>j φj (z) +

j=1

i=1

ai φ0 (xi )> φ0 (z)

ai

J
X

!
>

φj (xi ) φj (z),j=1that is exactly the same that would be obtained from (6) using
K 0 (x, z) as kernel. Even the original procedure to choose thebest β can be easily enlarged to the case of linearly combined
orthogonal spaces. The vector ŷ 0j containing the predictionsof the j−th known model is:
0> 00ŷ 0j = [ŵ0>j φ (x1 ), . . . , ŵ j φ (xN ))]>= [ŵ>j φj (x1 ), . . . , ŵ j φj (xN ))] = ŷ j .

This indicates that MultiK-KT is formally equivalent to theoriginal Multi-KT with the kernel chosen as in (16). As aconsequence the computational complexity of MultiK-KT isagain O(N 3 + JN 2 ) (see Section 5.3).

7

E XPERIMENTS

In this section we show empirically the effectiveness of
our transfer algorithm1 on three datasets: Caltech-256 [37],Animals with Attributes (AwA) [24] and IRMA [38].
The Caltech-256 contains images of 256 object classes
plus a clutter category that can be used as negative classin object-vs-background problems. The objects are organized
in a hierarchical ontology that makes it easy to identify the
related and unrelated categories. We downloaded2 the precomputed features of [39] and we selected four different image
descriptors: PHOG Shape Descriptors [40], SIFT Appearance
Descriptors [41], Region Covariance [42], and Local Binary
Patterns [43]. They were all computed in a spatial pyramid[44] and we considered just the first level (i.e. information
extracted from the whole image).
The AwA dataset contains 50 animal classes and it has
been released with several pre-extracted feature sets for each
1. We implemented it in MATLAB, the code is available online http://www.
idiap.ch/∼ttommasi/source code CVPR10.html
2. http://files.is.tue.mpg.de/pgehler/projects/iccv09/

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

8source classes. Finally, we selected the γ value producing on
average the best recognition rate. The value of C is instead
determined as the one producing the best result when learningfrom scratch. There is no guarantee that the obtained C value
is the best for the transfer approach; still in this way wecompare against the best performance that can be obtainedby learning only on the available training samples, without
exploiting the source knowledge. We used this setup for all
the experiments; specific differences are otherwise mentioned.
7.1

Fig. 4. For Multi-KT the source and target models must
live in the same space identified by the kernel K. For
MultiK-KT all the sources can be defined independently
in their own space and the target solution lives in thespace obtained by orthogonal combination. We show also
a geometrical interpretation of the kernel combination.
image3 . From the full set of categories we extracted the six
sea mammals (killer whale, blue whale, humpback whale, seal,
walrus and dolphin) and used them to define the backgroundclass. We used three of the precomputed descriptors for our
experiments: color histogram, PHOG and SIFT.
The IRMA database is a collection of x-ray images presenting a large number of classes defined according to a fouraxis hierarchical code [45]. We decided to work on the 2008
IRMA database version [38], just considering the third axis of
the code which identifies the depicted body part. A total of 23classes with more than 100 images were selected from various
sub-levels of the third axis, 3 of them were used to define
the background class. Following [46], we used as features theglobal pixel-based and local SIFT-based descriptors.
We performed all the experiments in a leave-one-class-out
approach, that is considering in turn each class as target and all
the others as sources. The number of negative training samples
is kept fixed while the number of positive training samples
increases in subsequent steps till reaching the same amountof the negative set. The samples are extracted randomly 10
times for an equal number of experimental runs. Each priorknowledge model is built with classical LS-SVM. We use theGaussian kernel both on the source and on the target for all theexperiments K(x, x0 ) = exp(−γkx−x0 k2 ). To integrate multiple (F) features we calculate one kernel for P
each of them andF
we use the average kernel K(x, x0 ) = 1/F f =1 Kf (x, x0 ).
All the transfer results are benchmarked against no transfer:
this corresponds to learning from scratch with weighted-LSSVM , i.e. solving the optimization problem in (5) with β = 0.
Regarding the parameters, a unique common value for γ waschosen for all the kernels by cross validation on the source
sets. In particular, we trained a model for each class in thesource set and we used it to classify on the remaining J − 1
3. http://attributes.kyb.tuebingen.mpg.de/Setting the Constraints

To fully define the Multi-KT algorithm it is necessary tochoose the p value in the constraint of (14). We evaluate
empirically three cases with p = 1, 2, ∞ and we compare
the obtained results over three groups of data that differ
in the level of relatedness among source and target knowledge. Specifically, we extracted 6 unrelated classes (harp,
microwave, fire-truck, cowboy-hat, snake, bonsai), 6 relatedclasses (all vehicles: bulldozer, fire-truck, motorbikes, schoolbus, snowmobile, car-side) and 10 mixed classes (motorbikes,
dog, cactus, helicopter, fighter, car-side, dolphin, zebra, horse,
goose) from Caltech-256. We refer to a class as the combination of 80 object and 80 background images. For eachclass used as target, we extracted 20 training and 100 testing
samples with half positive and half negative data.
The results in Figure 5 (top line) show the clear gain obtained by Multi-KT with respect to no transfer4 . The advantage
is maximum in case of related classes (the difference between
Multi-KT L2 and no transfer is 39% in recognition rate for 1
positive sample), it is just a little bit smaller for mixed classes(34%) and drops more in case of sources unrelated to the target
task (29%). However, regardless of the relatedness level, thechoice of the constraint on β does not produce significantly
different results, apart for a slightly lower performance of theL1 case with respect to the others. Hence, in the following wewill always use the L2 norm constraint.
7.2

Transfer Weights and Semantic Similarity

The Multi-KT algorithm defines automatically the relevance ofeach source model to the current target task. We analyze here
the β vector obtained as a byproduct of the transfer process,to verify if its elements have a correspondence with the realvisual and semantic relation among the tasks.
We start from the results obtained in the previous section
with the L2 norm constraint and we consider the intermediate
training step with 5 positive samples. We average the β vectors
obtained over the 10 runs defining a matrix of weights with one
row for each class used as target. By simple algebra we cantransform it to a fully symmetric matrix containing measuresof class dissimilarities evaluated as (1−βj ) and apply multidimensional scaling on it [48], with two dimensions. We obtainplots where each point represents a class, and the distanceamong the points is proportional to the input dissimilarities.
4. Using SVM for learning from scratch produces slightly better results thanLS-SVM. However in all our experiments this difference is not significant andit does not change the conclusions on the proposed transfer approach. We use
the sign test [47] for our statistical evaluations.

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

9

6 classes unrelated

6 classes related

10 classes mixed

0.95

0.95

0.9

0.9

0.85

0.85

0.8
0.75
0.7
0.65
no transfer
Multi−KT L2

0.6

Multi−KT L1

0.55

Multi−KT L∞

0.5
0

2

4

6

8

10

0.8
0.75
0.7
0.65
no transfer
Multi−KT L2

0.6

2

4

6

8

0.7
0.65
no transfer
Multi−KT L2
Multi−KT L1

0.55

Multi−KT L∞

0.5
0

0.8
0.75

0.6

Multi−KT L1

0.55# of positive training samples

10

Multi−KT L∞

0.5
0

2

# of positive training samples

6 classes unrelated

4

bulldozer

0.5

cowboy−hat

microwave
00.5

snowmobile

snake

0

10 classes mixed

dolphindog

0fire−truck

−0.5

−1

−1

−0.5

0

0.5

motorbikes

cactushelicopter

1

−1
−1

−0.5

car−side

0.4

5 − fighter−jet
6 − motorbikes

horse
−0.5

−0.5

bonsai0.5

4 − helicopter

fighter−jet

car−side0.6

2 − horse
3 − zebra

school−bus

fire−truck10

1 − dog

goose
harp

8

# of positive training samples

1

1

0.5

6

10 classes mixed

6 classes related

1

Recognition Rate

0.9
0.85

Recognition Rate

Recognition Rate

0.95

motorbikes

0.3

7 − car−side
0.2

zebra

8 − dolphin
9 − goose

0

0.5

1

0.1

10 − cactus

−1
−1

−0.5

0

0.5

1

1

2

3

4

5

6

7

8

9

10

0

Fig. 5. Top line: Performance of the Multi-KT method with various settings for the constraint on the source knowledge
weights. The results correspond to average recognition rate over the categories, considering each class out experimentrepeated ten times. Bottom line: output of the bidimensional scaling applied on the β vector values. For 10 mixed
classes we also show the assigned weigths with a heat map where each row corresponds to a target class and eachcolumn to a source class.
Figure 5 (bottom line) shows the obtained results. It can beseen that in the case of unrelated classes the corresponding
points tend to be far from each other. On the other hand,among the related classes extracted from the general category
motorized-ground-vehicles, the four wheels vehicles form acluster, leaving aside motorbikes (two wheels), snowmobile(skis) and bulldozer (tracks). Finally, among the mixed classes,
helicopter and fighter-jet appear close to each other and todolphin. Probably this is due to the shape appearance of
these object classes and to the common uniformity of the sky
and water background. Moreover, all the four legged animals(zebra, horse and dog) appear on the right side of the plot,
while the vehicles (car-side and motorbikes) are in the left
bottom corner. The heat map of the β weights also showsthat Multi-KT does not leverage over the source models ina flat way, but chooses properly which part of the available
knowledge can be reused.
Globally all the results indicate that the β vectors actuallycontain meaningful values in terms of semantic relation between the object classes.
7.3

Comparison and Evaluation

Here we evaluate our Multi-KT algorithm in comparison withseveral state of the art transfer learning approaches. We briefly
review them before discussing the experimental results.
Single Source. Most of the existing knowledge transfer
methods suppose the availability of a single source knowledge.
Among the approaches listed below, the first two are based ontransferring model parameters as our Multi-KT, while the last
one is an instance transfer technique and exploits directly thesource samples.
Adaptive SVM (A-SVM). This method has been originallypresented in [25] and is based on substituting the usual

regularizer of the SVM formulation with the adaptive version
NX
min kw − β ŵk2 + C`H (w> φ(xi ), yi ) .
(17)
w

i=1

Projective Model Transfer SVM (PMT-SVM). Maximizing
the projection of w onto ŵ corresponds also to minimizing
the projection of w onto the source separating hyperplane(orthogonal to ŵ). Following this idea the objective functionof PMT-SVM [11] isNX
min kwk2 + βkRwk2 + C`H (w> φ(xi ), yi )
w

s. t.

i=1

w> ŵ ≥ 0,

(18)

here R is the projection matrix and kRwk = kwk sin2 θ,
where θ is the angle between w and ŵ.
TrAdaBoost: boosting for Transfer Learning. This extension
to the Adaboost learning framework was proposed in [18]. It isbased on mechanism which, starting from the combination ofsource and target samples, iteratively decreases the weights ofthe source data in order to weaken their impact on the learning
process.
Experiments. We benchmark here our Multi-KT algorithm
against the described A-SVM, PMT-SVM and TrAdaBoost.
Since these baseline methods were defined in the hypothesisof a single available source set, we considered two cases: apair of unrelated and a pair of related classes. Both the pairs
were extracted from Caltech-256 and each of the classes is
considered in turn as target while the other represents thesource task.
We used the MATLAB code of PMT-SVM provided by
its authors, together with their implementation of A-SVM52

5. http://www.robots.ox.ac.uk/∼vgg/software/tabularasa/

2

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

cereal box − mountain bike
0.85

0.75
0.7
0.65

no transfer
Multi−KT
A−SVM
PMT−SVM
TrAdaBoost

0.6
0.55
0

2

4

6

8

0.8

0.7
0.65

no transfer
Multi−KT
A−SVM
PMT−SVM
TrAdaBoost

0.6
0.55
0

10

2

4

6

8

# of positive training samples# of positive training samples

10 classes

20 classes

10

unrelated

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.75
Difference against No Transfer

0.8

Recognition Rate

Recognition Rate

0.85

relatedRecallfire truck − school bus

10

fire
truck

0.2

schoolbus

cerealbox

Muti−KT TPR
Muti−KT TNR
0.2

mountainbikeMuti−KT TPR
Muti−KT TNR

0.2

0.1

0.1

0

5

10

0

# of positive training samples

5

10# of positive training samples

Stability, 20 classes
1

0.75
0.7
no transfer
Multi−KT
Single−KT
A−SVM
MultiSourceTrAdaBoost
TaskTrAdaBoost

0.65
0.6
0.55
0

2

4

6

8

# of positive training samples

10

0.8
0.75
0.7
no transfer
Multi−KT
Single−KT
A−SVM
MultiSourceTrAdaBoost
TaskTrAdaBoost

0.65
0.6
0.55
0

2

4

6

8

# of positive training samples

10

Average Variation

0.8

Multi−KT
Single−KT

0.9

0.85

Recognition Rate

Recognition Rate

0.85

0.8
0.7
0.6
0.5
0.4
0.3
0.2
2−1

3−2

4−3

5−4

6−5

7−6

8−7

9−8

10−9

Step Difference

Fig. 6. Left and middle columns: recognition rate as a function of the number of positive training samples. In eachexperiment we consider in turn one of the classes as target and the others as source, on ten random training sets.
The final results are obtained as average over all the runs. Top Right: (up) the histogram bars represent the recall
produced by the source model (indicated on the x-axis) when used to classify on the target class; (down) we computeseparately the true positive and true negative recognition rate of Multi-KT and we show the absolute value of theirdifference with no transfer. Bottom Right: average norm of the difference between two β vectors obtained for a pair ofsubsequent training steps.
slightly modifying them to introduce the weights ζi for i =1, . . . , N in the corresponding loss function, so to have a faircomparison with our Multi-KT. The original formulation considered the linear kernel, thus we chose K(x, z) = x> z for
all the experiments together with the SIFT feature descriptors.
In [11] the β value is defined by cross validation on extra
validation target samples. Here we decided to simply tune it on
the test set, showing the best result that could be obtained. The
same approach was adopted to choose the number of boosting
iterations for TrAdaBoost.
The results are shown in Figure 6 (top line). In the related(left plot) case all the transfer learning methods show better
performance than learning from scratch with different extent.
The results of our Multi-KT are significantly better than thoseof no transfer and PMT-SVM (p ≤ 0.01). Only for 10
positive training samples PMT-SVM and Multi-KT produce
comparable results. Multi-KT also outperforms TrAdaBoost
for all the training steps (p ≤ 0.01) except the first one,
where they are statistically equivalent. Finally, the difference
between Multi-KT and A-SVM is not significant: since the βparameter for A-SVM is tuned on the test set, this indicatesthat Multi-KT is autonomously able to optimally weight thesource knowledge. The bias of A-SVM towards the best
possible recognition rate is evident in the case of unrelatedclasses (middle plot) where it is the only method to outperform
no transfer along all the steps. The other knowledge transfer
approaches show better results than no transfer only for lessthan three positive training samples (p ≤ 0.05), becoming thenstatistically equivalent to learning from scratch.
The histogram bars on the right in Figure 6 (top right up) show the recall produced by each source model whenused directly to classify on the target task. This indicates
how good is the source in recognizing the target object
without adaptation and it is clearly lower for unrelated than for
related classes. For a deeper understanding of the method wealso calculated separately the true positive and true negative
recognition rate of Multi-KT. We show the absolute value of
their difference with no transfer in Figure 6 (top right - down).
From the plots we can conclude that the main advantage intransferring is in fact due to the relation between the source
and target positive classes rather than to the joint backgroundclass.
Multiple Sources. When more than one source set isavailable, there are three main strategies that a transfer learning
method can consider. Two extreme solutions consist in either
selecting only one source, evaluated as the best for the target
problem, or averaging over all of them supposing that they are
all equally useful. The third strategy considers the intermediatecase where only some of the source sets are helpful for thetarget task and consists in selecting them by assigning to eacha proper weight. To our knowledge, only our Multi-KT method
is based on the third selective technique.
MultiSourceTrAdaBoost: boosting by transferring samples.
An extension to the TrAdaBoost approach in the case of

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

multiple available sources has been presented in [9]. The
method MultiSourceTrAdaBoost considers one source set at thetime, combining it with the target set and defining a candidate
weak classifier. The final classifier is then chosen as the one
producing the smallest training target classification error byautomatically selecting the corresponding best source.
TaskTrAdaBoost: boosting by transferring models. This is aparameter transfer approach consisting of two steps. Phase I
deploys traditional AdaBoost separately on each source task
to get a collection of candidate weak classifiers. Only the mostdiscriminative are stored. Phase II is again an AdaBoost loop
over the target training data where at each iteration the weakclassifier is extracted from the set produced in the previous
phase.
Single KT. Our Multi-KT algorithm chooses the best set ofweights for all the prior knowledge models at once on the basisof the loss function defined in (13). An alternative approach
can be defined adopting a logistic loss function [49]:1.
(19)`(ỹi , yi ) = ζi
1 + exp{−10(ỹi − yi )}If we consider one single source knowledge j at the time,
the correspondingi , yi ) will depend on the difference
 0 loss `j (ỹ

A00
ai
ji
(ỹi − yi ) = Pii − βj Pii for all i = 1, . . . , N . Although
this formulation results in a non convex objective function
with respect to βj , it is always possible to evaluate (19) fora finite set SPof weights6 . We can store for each source thevalue minS { i `j (ỹi , yi )}, and then compare all the results
to identify the best prior knowledge model and its best weight.
We call this variant of our method Single-KT.
Average Prior Knowledge. As already mentioned in the
introduction, the first knowledge transfer approach able toperform one-shot learning on computer vision problems waspresented in [7]. This approach does not make any assumption
on the reliability of the prior knowledge, which is always
considered as an average over all the known classes. The
algorithm structure is strictly related to the part-based model
descriptors and neither the code nor the feature used forthe experiments in [7] have ever been publicly released.
However, by following the proposed main idea, any single
source transfer learning method can be extended to the caseof multiple sources by relying on their average model.
Experiments. Here we show a benchmark evaluation of
our Multi-KT algorithm against its Single-KT version, MultiSourceTrAdaBoost and TaskTrAdaBoost. We also consider
A-SVM as baseline using the average
PJ of all the prior models
as source knowledge, thus ŵ = J1 j=1 ŵj and β = 1.
We adopted the same experimental setting of the previous
section with SIFT features, linear kernel and two randomly
extracted sets of 10 and 20 classes from Caltech-256. In
particular, the second set is obtained by adding an extra
random group of 10 classes to the first one. For the boosting
approaches all the learning parameters where tuned on thetest set and only the best results are presented. From Figure
6 it is clear that in both the experiments that Multi-KT
clearly outperforms Single-KT and the two boosting methods(p ≤ 0.01), besides producing better results than learning from
6. We considered a fine tuning varying β in {0.01, 1} with step of 0.01.

11

scratch (p ≤ 0.01). Moreover, for very few samples, properly
weighting each prior knowledge source with Multi-KT is better(p ≤ 0.05) than averaging over all the known models as doneby A-SVM: the two approaches are equivalent only after fivepositive training samples with 10 classes and respectively threepositive training samples for 20 classes.
The behavior of any method that chooses only one source
model in transferring may vary significantly every time there
is a change in the selected source. This indicates low stability.
Recent work has shown that the more stable is an algorithm,
the better is its generalization ability [50]. The plot on thebottom right in Figure 6 shows the comparison of Multi-KT
with its Single-KT version in terms of stability. The best βjvalue chosen by Single-KT can be considered as an elementof the full β vector where all the remaining elements are zero.
For each pair of subsequent steps in time, corresponding to a
new added positive training sample, we calculate the difference
between the obtained β both for Multi-KT and Single-KT.
From the average norm of these differences it is evident thatchoosing a combination of the prior known models for transfer
learning is more stable than relying on just a single source(lower average variation in the vector β).
7.4

Heterogeneous Knowledge

In this section we consider an heterogeneous setting where
each source knowledge lives in its own feature space and wecompare the performance of MultiK-KT with that of MultiKT applied on a restricted homogeneous condition. We showthat the space enlarging trick at the basis of MultiK-KT,
not only allows to overcome the problem due to the source
variability, but it also produces better results than Multi-KT inthe corresponding single space case.
We ran experiments on the same subset of data used in
section 7.1. Here we considered SIFT as unique descriptor
together with the generalized Gaussian P
kernel: K(x, z) =exp(−γdρ,δ (x, z)), where dρ,δ (x, z) = i |xρi − ziρ |δ . Eachsource knowledge is defined by using the best set {γ, ρ, δ}
obtained by cross validation on the corresponding object
category. We learn on the target class considering the sum overthe source kernels. We name no transfer multiK the baseline
corresponding to learning from scratch in this combined space.
Figure 7 presents the obtained results in comparison with thecase of using a single standard Gaussian kernel, with fixed γ
for sources and target tasks (no transfer and Multi-KT curvesin the plot): MultiK-KT always performs significantly better
than Multi-KT (p ≤ 0.002).
Among the baseline methods considered in the homogeneous experiments, the only one that allows heterogeneous
sources is TaskTrAdaBoost. We compare it with MultiK-KT
over the random set of ten classes already used in the previous
section. For each source we suppose to have already learned an
SVM model with SIFT descriptors and Gaussian kernel wherethe γ parameter is set to the mean of the pairwise distancesamong the samples. This means that each source model lives
in its own specific feature space. TaskTrAdaboost in eachboosting iteration simply chooses one of the source models,
while MultiK-KT learns the target task in the composed

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

12

6 classes unrelated

6 classes related

10 classes mixed

0.95

0.95

0.9

0.9

0.85

0.85

0.8
0.75
0.7
0.65
0.6

0.5
0

2

4

6

8

0.8
0.75
0.7
0.65
0.6

no transfer
no transfer multiK
Multi−KT
MultiK−KT

0.55

Recognition Rate

0.9
0.85

Recognition Rate

Recognition Rate

0.95

10

0.5
0

# of positive training samples

2

4

6

8

0.7
0.65
0.6

no transfer
no transfer multiK
Multi−KT
MultiK−KT

0.55

0.8
0.75

no transfer
no transfer multiK
Multi−KT
MultiK−KT

0.55
0.5
0

10

2

# of positive training samples

4

6

8

10

# of positive training samples

Fig. 7. Performance of the MultiK-KT method in comparison with the single kernel Multi-KT formulation. The curves
identified by no transfer and no transfer multiK corresponds respectively to learning from scratch by using only theGaussian kernel or the combination of generalized Gaussian kernels.
100 classes

10 classes, Heterogeneous sources

0.8
0.75
0.7

0.8

0.8
no transfer
Multi−KT
A−SVM

0.6
5

10

# of positive training samples
0.65

256 classes
0.9

0.6
no transfer multiK
MultiK−KT
TaskTrAdaBoost

0.55
0.5
0

2

4

6

8

10

# of positive training samples

Fig. 8. Recognition rate as a function of the number of
positive training samples. Each source model is definedby using a Gaussian kernel with a different γ parameter.
space defined by all the sources and obtained on the basis of
the sum kernel. Figure 8 shows that multiK-KT outperforms
TaskTrAdaBoost (p ≤ 0.01) besides obtaining better results
than learning from scratch.
7.5 Increasing Number of Sources
For any open-ended learning agent the number of known
object categories is expected to grow in time. An increasing
number of sources may give rise to a scalability problem intransfer learning due to the necessity of checking the reliabilityof each known model for the new task. Specifically, for 102
sources the boosting methods described in Section 7.3 become
extremely expensive in computational terms (indeed in [9] they
considered a maximum of 5 sources).
We performed experiments with 100 and 256 object classes
from Caltech-256 dataset, reporting the result of Multi-KT, notransfer and A-SVM with average prior knowledge in Figure 9.
In both cases, properly choosing the weights to assign to eachsource pays off with respect to average over all the sources
for very few training samples: Multi-KT outperforms A-SVM(p ≤ 0.05) for less than three positive samples. With enoughtraining samples and a rich prior knowledge set, the best choice
is to not neglect any source information.
We can expect that with a growing prior knowledge set,also the probability to find a useful source for the target task
increases. To verify this behavior we focus on the MultiKT results obtained with a single positive image. The oneshot performance for 2 unrelated classes, 2 related classes

Multi−KTno transfer

0.7

0Recognition Rate

Recognition Rate

0.85

0.85

Recognition Rate

Recognition Rate

0.9

0.75

0.7

0.65

0.8
0.7

0.6
no transfer
Multi−KT
A−SVM

0.6
0

5

10

# of positive training samples

0.55

2 unr2 rel 10

20 100 256

Number of Classes

Fig. 9. Multi-KT performance for high number of source
knowledge sets. Right: one shot learning recognition rate
results when varying the number of prior known object
categories.
and increasing sets of 10, 20, 100 classes plus the final
full set of 256 objects are summarized in Figure 9 (right).
Specifically for each group of J classes we show the average
one-shot recogniton result over all the possible source/target(J − 1)classes/(1)class combinations. In this way the numberof evaluations grows with the class group dimension andthis may cause small oscillations in the final average results.
Nevertheless, from the bars it is it is clear that by increasing
the number of available sources the one-shot recognition rate
obtained with Multi-KT grows. After an evident gain obtainedby passing from 100 to 101 classes, the difference becomesless evident from 101 to 102 classes.
7.6

Increasing Number of Samples

Transfer learning has its maximum effectiveness in the small
sample scenario in comparison to learning from scratch. However, it is also interesting to evaluate the performance of aknowledge transfer approach when the number of available
training instances increases, thus checking its asymptotic behavior (see Figure 2).
We repeated the experiments on the full Caltech-256 dataset
considering {1, 5, 10, 30, 50} positive training samples with afixed set of 50 negative training samples. We also run analogous experiments on the AwA and IRMA dataset, consideringrespectively 60 (60) and 70 (70) positive (negative) training

0.8
no transfer
Multi−KT

0.6
1

5

10

30

50

13

AwA, 44 classes
1
0.8
no transfer
Multi−KT

0.6
1

5

10

30

50

60

Recognition Rate

Caltech−256, 256 classes
1

Recognition Rate

Recognition Rate

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

0.6

0.4
5
10
30
# of positive training samples

50

0.6

max βjmax βj

j

max β

0.8
no transfer
Multi−KT

0.6
1

5

10

30

50

70

0.7

0.5

1

IRMA, 20 classes
1

0.4
1

5
10
30
50
# of positive training samples

60

0.6
0.5
1

5
10
30
50
# of positive training samples

70

Fig. 10. Top line: recognition rate as a function of the number of positive training samples. Each experiment is definedby considering in turn one of the classes as target and the others as sources. The final results are obtained as averageover ten runs. Bottom line: Maximum value over the elements of the β vector averaged over the classes and the splits.
samples. For all the datasets the test set contains 60 (30positive and 30 negative) instances.
All the results are reported in Figure 10 (top line). Although
it is clear the gain of Multi-KT with respect to learning fromscratch for limited available data, in general this advantage
disappears when the number of positive training samples
reaches 30. The absence of the asymptotic advantage was to be
expected for Multi-KT and can be justified in theoretical terms.
Any universally consistent classifier will converge to the target
optimal solution for an infinite number of training samples.
As discussed in [51], this is the case for SVM with universal
kernels, thus we expect that both our Multi-KT and the notransfer curve will reach the same asymptotic value when
the amount of data increases. The only possible advantage
in performance can be obtained for a reduced set of target
training samples where the effect of the adaptive regularizationin Multi-KT is relevant and advantageous over learning fromscratch. It does not exists a general rule to establish when thesmall samples regime ends and the large sample regime starts,for our algorithm we showed that the limit appears around 30
target training samples.
As a final remark we underline the overall smooth behaviorof Multi-KT in assigning the relevance weights to the source
knowledge. In case of a single positive target training sample the prediction is strongly supported by the sources, but
their importance progressively decreases when the number oftraining samples grows (see Figure 10, bottom line).

8

C ONCLUSION

A learning system able to exploit prior knowledge whenlearning something new should rely only on the available
target information for choosing from where and how muchto transfer. To be autonomous it should not need an externalteacher providing either information on which is the bestsource to use, or extra target training samples. In this paper we
presented our Multi-KT algorithm, a LS-SVM based transfer
learning approach with a principled technique to rely on source
models and avoid negative transfer. The results of extensive
experiments demonstrated the effectiveness of Multi-KT for
object categorization problems with respect to other existingtransfer learning methods. Moreover the weight assigned tothe source knowledge set proved to be meaningful in terms of

the semantic relation among the considered classes. We also
extended our algorithm to the heterogeneous setting.
Recently the computer vision literature has seen an increasing interest towards high scale (104 ) object problems [5]. Mostof the proposed transfer learning algorithms in this setting havebeen developed for object detection [52] and segmentation[53], while how to scale up the classification problem is still
an open issue. Introducing a structure on the source knowledge
while learning something new might be a promising strategy
to use Multi-KT in this condition. Moreover the associatedscalability problem due to the increasing number of training
examples can be overcome by casting Multi-KT in an online
learning framework [54]. All this clearly indicates possible
directions for future research.

R EFERENCES[1]

I. Biederman, “Recognition-by-components: A theory of human image
understanding,” Psychological Review, vol. 94, pp. 115–147, 1987.
[2] D. R. Hofstadter, Fluid Concepts and Creative Analogies: Computer
Models of the Fundamental Mechanisms of Thought. Basic Books, Inc.,
1996.
[3] N. Intrator and S. Edelman, “Learning to learn,” ch. Making a lowdimensional representation suitable for diverse tasks, pp. 135–157,
Kluwer Academic Publishers, 1996.
[4] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL Visual Object Classes Challenge 2007 (VOC2007)
Results.” http://www.pascal-network.org/challenges/VOC/.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in Conference on
Computer Vision and Pattern Recognition (CVPR), 2009.
[6] S. Thrun, “Is learning the n-th thing any easier than learning the first?,”in Advances in Neural Information Processing Systems (NIPS), 1996.
[7] L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object
categories,” IEEE Trans. Pattern Analysis and Machine Intelligence(PAMI), vol. 28, pp. 594–611, 2006.
[8] A. Quattoni, M. Collins, and T. Darrell, “Transfer learning for image
classification with sparse prototype representations,” in Computer Vision
and Pattern Recognition Conference (CVPR), 2008.
[9] Y. Yao and G. Doretto, “Boosting for transfer learning with multiple
sources.,” in Computer Vision and Pattern Recognition Conference(CVPR), 2010.
[10] M. Stark, M. Goesele, and B. Schiele, “A shape-based object class model
for knowledge transfer,” in International Conference on Computer Vision(ICCV), 2009.
[11] Y. Aytar and A. Zisserman, “Tabula rasa: Model transfer for object
category detection,” in International Conference on Computer Vision(ICCV), 2011.
[12] J. Suykens, T. V. Gestel, J. D. Brabanter, B. D. Moor, and J. Vanderwalle,
Least Squares Support Vector Machines. World Scientific, 2002.
[13] L. G. Valiant, “A theory of the learnable,” Communications ACM,
vol. 27, no. 11, pp. 1134–1142, 1984.

JOURNAL OF LATEX CLASS FILES, VOL. X, NO. Y, MONTH YEAR

[14] V. N. Vapnik, The nature of statistical learning theory. Springer-Verlag
New York, Inc., 1995.
[15] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–
1359, 2010.
[16] L. Torrey and J. Shavlik, “Transfer learning,” in Handbook of Researchon Machine Learning Applications and Trends: Algorithms, Methods,
and Techniques, IGI Global, 2009.
[17] http://socrates.acadiau.ca/courses/comp/dsilver/NIPS95 LTL/transfer.
workshop.1995.html.
[18] W. Dai, Q. Yang, G. Xue, and Y. Yu, “Boosting for transfer learning,”
in International Conference on Machine Learning (ICML), 2007.
[19] J. J. Lim, R. Salakhutdinov, and A. Torralba, “Transfer learning byborrowing examples for multiclass object detection,” in Advances in
Neural Information Processing Systems (NIPS), 2011.
[20] E. Bart and S. Ullman, “Cross-generalization: Learning novel classes
from a single example by feature replacement,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2005.
[21] M. Fink, “Object classification from a single example utilizing class relevance metrics,” in Advances in Neural Information Processing Systems(NIPS), pp. 449–456, 2004.
[22] U. Rückert and S. Kramer, “Kernel-based inductive transfer,” in European conference on Machine Learning and Knowledge Discovery in
Databases (ECML PKDD), 2008.
[23] G. Mesnil, Y. Dauphin, X. Glorot, S. Rifai, Y. Bengio, I. Goodfellow,
E. Lavoie, X. Muller, G. Desjardins, D. Warde-Farley, P. Vincent,A. Courville, and J. Bergstra, “Unsupervised and transfer learning
challenge: a deep learning approach,” in Journal of Machine Learning
Research, vol. 27, pp. 97–110, 2012.
[24] C. H. Lampert, H. Nickisch, and S. Harmeling, “Learning to detect
unseen object classes by between class attribute transfer,” in Computer
Vision and Pattern Recognition Conference (CVPR), 2009.
[25] J. Yang, R. Yan, and A. G. Hauptmann, “Adapting SVM classifiers todata with shifted distributions,” in International Conference on Data
Mining Workshops (ICDM), 2007.
[26] Y. Zhang and D. Yeung, “Transfer metric learning by learning task relationships,” in ACM SIGKDD International Conference on Knowledge
discovery and data mining, 2010.
[27] J. Davis and P. Domingos, “Deep transfer via second-order markov
logic,” in International Conference on Machine Learning (ICML), 2009.
[28] W. Dai, O. Jin, G. Xue, Q. Yang, and Y. Yu, “Eigentransfer: aunified framework for transfer learning,” in International Conference
on Machine Learning (ICML), 2009.
[29] Y. Aytar and A. Zisserman, “Enhancing exemplar svms using part leveltransfer regularization,” in Proc. of British Machine Vision Conference(BMVC), 2012.
[30] M. E. Taylor, G. Kuhlmann, and P. Stone, “Accelerating search withtransferred heuristics,” in ICAPS-07 workshop on AI Planning and
Learning, 2007.
[31] M. M. Mahmud and S. R. Ray, “Transfer learning using Kolmogorov
complexity: Basic theory and empirical evaluations.,” in Advances in
Neural Information Processing Systems (NIPS), 2007.
[32] B. Scholkopf and A. J. Smola, Learning with Kernels: Support Vector
Machines, Regularization, Optimization, and Beyond. Cambridge, MA,
USA: MIT Press, 2001.
[33] H. Daumé III, “Frustratingly easy domain adaptation,” in Association
for Computational Linguistics Conference (ACL), 2007.
[34] G. C. Cawley, “Leave-one-out cross-validation based model selection
criteria for weighted LS-SVMs,” in In Proceedings of the International Joint Conference on Neural Networks (IJCNN), (Vancouver, BC,
Canada), pp. 1661–1668, 2006.
[35] J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra, “Efficient
projections onto the l1-ball for learning in high dimensions,” in International Conference on Machine Learning (ICML), 2008.
[36] D. Hush, P. Kelly, C. Scovel, and I. Steinwart, “QP algorithms withguaranteed accuracy and run time for support vector machines,” Journalof Machine Learning Research, 2006.
[37] G. Griffin, A. Holub, and P. Perona, “Caltech 256 object category
dataset,” Tech. Rep. UCB/CSD-04-1366, California Institue of Technology, 2007.
[38] T. Deselaers and T. Deserno, “Medical image annotation in ImageCLEF
2008,” in working notes CLEF, 2008.
[39] P. Gehler and S. Nowozin, “On feature combination for multiclass object
classification,” in International Conference on Computer Vision (ICCV),
2009.

14

[40] A. Bosch, A. Zisserman, and X. Munoz, “Representing shape with aspatial pyramid kernel,” in ACM international conference on Image andvideo retrieval (CIVR), 2007.
[41] D. Lowe, “Distinctive image features from scale-invariant keypoints,”
International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110,
2004.
[42] O. Tuzel, F. Porikli, and P. Meer, “Human detection via classification
on riemannian manifolds,” in Computer Vision and Pattern Recognition(CVPR), 2007.
[43] T. Ojala, M. Pietikinen, and D. Harwood, “A comparative study of
texture measures with classification based on featured distributions,”
Pattern Recognition, vol. 29, pp. 51–59, Jan. 1996.
[44] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features:
Spatial pyramid matching for recognizing natural scene categories,” inComputer Vision and Pattern Recognition Conference (CVPR), 2006.
[45] T. Lehmann, H.Schubert, D. Keysers, M. Kohnen, and B. Wein, “TheIRMA code for unique classification of medical images,” in International
Society for Optical Engineering (SPIE), 2003.
[46] T. Tommasi, F. Orabona, and B. Caputo, “An SVM confidence-based
approach to medical image annotation,” in Evaluating Systems forMultilingual and Multimodal Information Access – Proceedings of
CLEF, 2008.
[47] J. Gibbons, Nonparametric Statistical Inference. New York: Marcel
Dekker, 1985.
[48] T. F. Cox and M. A. A. Cox, Multidimensional Scaling. Chapman andHall, 2001.
[49] T. Tommasi and B. Caputo, “The more you know, the less you learn:from knowledge transfer to one-shot learning of object categories,” inProc. of British Machine Vision Conference (BMVC), 2009.
[50] O. Bousquet and A. Elisseeff, “Stability and generalization,” Journal of
Machine Learning Research, vol. 2, pp. 499–526, 2002.
[51] I. Steinwart, “Consistency of support vector machines and other regularized kernel classifiers,” IEEE Transactions on Information Theory,
2005.
[52] M. Guillaumin and V. Ferrari, “Large-scale knowledge transfer for object
localization in imagenet,” in Proceedings of the 2012 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), CVPR ’12, 2012.
[53] D. Kuettel, M. Guillaumin, and V. Ferrari, “Segmentation propagation inimagenet,” in European Conference on Computer Vision (ECCV), 2012.
[54] T. Tommasi, F. Orabona, M. Kaboli, and B. Caputo, “Leveraging overprior knowledge for online learning of visual categories,” in British
Machine Vision Conference (BMVC), 2012.
Tatiana Tommasi Is a Research Assistant at KU
Leuven. Her research interests include machinelearning and computer vision with a focus on
knowledge transfer and object categorization using multimodal information. She received her MS
degree in physics (2004) and the Dipl. degree
in medical physics (2008) from the Universityof Rome, La Sapienza, Italy. She completed
her PhD in electrical engineering at the École
Polytechnique Fédérale de Lausanne, in 2013.
Francesco Orabona is a Research AssistantProfessor at the Toyota Technological Institute at
Chicago. His research interests are in the areaof theoretically motivated and efficient learningalgorithms, with emphasis on online learning,
kernel methods, and computer vision. He received the PhD degree in Bioengineering andBioelectronics at the University of Genoa, in2007. He is (co)author of more than 40 peerreviewed papers.
Barbara Caputo is Associate Professor at the
University of Rome La Sapienza since 2013,
where she leads the Visual and Multimodal Applied Learning Laboratory. She received her PhD
in Computer Science from the Royal Instituteof Technology (KTH) in Stockholm, Sweden,in 2004. Her main research interests are incomputer vision, machine learning and robotics,
where Prof. Caputo has been active since 1999.
Prof. Caputo has edited 4 books, and she is(co)author of more than 70 peer-reviewed papers.