Taskonomy: Disentangling Task Transfer Learning
Amir R. Zamir1,2 Alexander Sax1∗ William Shen1∗ Leonidas Guibas1 Jitendra Malik2 Silvio Savarese1
1

Stanford University

2

University of California, Berkeley

http://taskonomy.vision/
Abstract

3D
Edges

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers
these questions positively, implying existence of a structure
among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies acrosstasks, in order to, for instance, seamlessly reuse supervision among related tasks or solve many tasks in one system
without piling up the complexity.
We propose a fully computational approach for modeling the structure of the space of visual tasks. This is done
via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty-six 2D, 2.5D, 3D, andsemantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study
the consequences of this structure, e.g. nontrivial emerged
relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number oflabeled datapoints needed for solving a set of 10 tasks canbe reduced by roughly 23 (compared to training independently) while keeping the performance nearly the same. We
provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ
to devise efficient supervision policies for their use cases.

Point
Matching

Object Class.

Reshading(1000 class)Curvature
3D Edges
Normals

2.5D Segm.

Reshading Z-Depth Distance
TripletPoint
Cam. Pose
Matching

Semantic
Segm.

Normals

2D Segm.
2D Keypoints
Denoising

Autoencoding

Cam. Pose(fixated)

Room
Cam. Pose Layout(non-fixated)

Vanishing Pts

Figure 1: A sample task structure discovered by the computational
task taxonomy (taskonomy). It found that, for instance, by combining thelearned features of a surface normal estimator and occlusion edge detector,
good networks for reshading and point matching can be rapidly trainedwith little labeled data.

The field of computer vision has indeed gone far without
explicitly using these relationships. We have made remarkable progress by developing advanced learning machinery(e.g. ConvNets) capable of finding complex mappings fromX to Y when many pairs of (x, y) s.t. x ∈ X, y ∈ Y aregiven as training data. This is usually referred to as fully supervised learning and often leads to problems being solved
in isolation. Siloing tasks makes training a new task or acomprehensive perception system a Sisyphean challenge,
whereby each task needs to be learned individually fromscratch. Doing so ignores their quantifiably useful relationships leading to a massive labeled data requirement.
Alternatively, a model aware of the relationships among
tasks demands less supervision, uses less computation, andbehaves in more predictable ways. Incorporating sucha structure is the first stepping stone towards develop-

1. Introduction
Object recognition, depth estimation, edge detection,pose estimation, etc are examples of common vision tasks
deemed useful and tackled by the research community.
Some of them have rather clear relationships: we understand that surface normals and depth are related (one is a
derivate of the other), or vanishing points in a room are useful for orientation and layout estimation. Other relationships are less clear: how edge detection and the shading ina room can, together, perform pose estimation.
∗ Equal.

1ing provably efficient comprehensive/universal perception
models [34, 4], i.e. ones that can solve a large set of tasks
before becoming intractable in supervision or computation
demands. However, this task space structure and its effects
are still largely unknown. The relationships are non-trivial,
and finding them is complicated by the fact that we haveimperfect learning models and optimizers. In this paper,
we attempt to shed light on this underlying structure andpresent a framework for mapping the space of visual tasks.
Here what we mean by “structure” is a collection of computationally found relations specifying which tasks supply
useful information to another, and by how much (see Fig. 1).
We employ a fully computational approach for this purpose, with neural networks as the adopted computational
function class. In a feedforward network, each layer successively forms more abstract representations of the input containing the information needed for mapping the input to the
output. These representations, however, can transmit statistics useful for solving other outputs (tasks), presumably if
the tasks are related in some form [83, 19, 58, 46]. This isthe basis of our approach: we compute an affinity matrix
among tasks based on whether the solution for one task canbe sufficiently easily read out of the representation trained
for another task. Such transfers are exhaustively sampled,
and a Binary Integer Programming formulation extracts a
globally efficient transfer policy from them. We show thismodel leads to solving tasks with far less data than learning them independently and the resulting structure holds oncommon datasets (ImageNet [78] and Places [104]).
Being fully computational and representation-based, theproposed approach avoids imposing prior (possibly incorrect) assumptions on the task space. This is crucial because
the priors about task relations are often derived from either
human intuition or analytical knowledge, while neural networks need not operate on the same principles [63, 33, 40,
45, 102, 88]. For instance, although we might expect depth
to transfer to surface normals better (derivatives are easy),
the opposite is found to be the better direction in a computational framework (i.e. suited neural networks better).
An interactive taxonomy solver which uses our model
to suggest data-efficient curricula, a live demo, dataset, andcode are available at http://taskonomy.vision/.

13, 23, 55, 92, 90], homomorphic cryptography [42], lifelong learning [93, 15, 85, 84], functional maps [71], certain
aspects of Bayesian inference and Dirichlet processes [54,91, 90, 89, 37, 39], few-shot learning [81, 25, 24, 70, 86],transfer learning [75, 84, 29, 64, 67, 59], un/semi/selfsupervised learning [22, 8, 17, 103, 19, 83], which are studied across various fields [73, 94, 12]. We review the topics
most pertinent to vision within the constraints of space:
Self-supervised learning methods leverage the inherent
relationships between tasks to learn a desired expensive one(e.g. object detection) via a cheap surrogate (e.g. colorization) [68, 72, 17, 103, 100, 69]. Specifically, they use amanually-entered local part of the structure in the task space(as the surrogate task is manually defined). In contrast, ourapproach models this large space of tasks in a computationalmanner and can discover obscure relationships.
Unsupervised learning is concerned with the redundancies in the input domain and leveraging them for forming
compact representations, which are usually agnostic to thedownstream task [8, 49, 20, 9, 32, 77]. Our approach is not
unsupervised by definition as it is not agnostic to the tasks.
Instead, it models the space tasks belong to and in a way
utilizes the functional redundancies among tasks.
Meta-learning generally seeks performing the learning
at a level higher than where conventional learning occurs,e.g. as employed in reinforcement learning [21, 31, 28],
optimization [2, 82, 48], or certain architectural mechanisms [27, 30, 87, 65]. The motivation behind meta learning has similarities to ours and our outcome can be seen as
a computational meta-structure of the space of tasks.
Multi-task learning targets developing systems that can
provide multiple outputs for an input in one run [50, 18].
Multi-task learning has experienced recent progress and thereported advantages are another support for existence of a
useful structure among tasks [93, 100, 50, 76, 73, 50, 18, 97,
61, 11, 66]. Unlike multi-task learning, we explicitly model
the relations among tasks and extract a meta-structure. The
large number of tasks we consider also makes developing
one multi-task network for all infeasible.
Domain adaption seeks to render a function that is developed on a certain domain applicable to another [44, 99,
5, 80, 52, 26, 36]. It often addresses a shift in the input domain, e.g. webcam images to D-SLR [47], while the task
is kept the same. In contrast, our framework is concernedwith output (task) space, hence can be viewed as task/output
adaptation. We also perform the adaptation in a larger spaceamong many elements, rather than two or a few.
In the context of our approach to modeling transfer learning across tasks:
Learning Theoretic approaches may overlap with anyof the above topics and usually focus on providing generalization guarantees. They vary in their approach: e.g. bymodeling transferability with the transfer family required

2. Related Work
Assertions of existence of a structure among tasks dateback to the early years of modern computer science, e.g.
with Turing arguing for using learning elements [95, 98]
rather than the final outcome or Jean Piaget’s works on
developmental stages using previously learned stages assources [74, 39, 38], and have extended to recent works [76,73, 50, 18, 97, 61, 11, 66]. Here we make an attempt to actually find this structure. We acknowledge that this is related
to a breadth of topics, e.g. compositional modeling [35, 10,
2

(I) Task-specific Modeling
LayoutReshading

Normals(II) Transfer Modeling
LayoutNormals

3D Keypoints

2.5D Segm

...
u
tp
Ou

ts

ce
pa

2D Segm.

3D Keypoints

2.5D Segm(III) Task Affinity
Normalization

Reshading...
2D Segm.

2D Segm.

Inpu

2D Edges
Denoising

Autoencoding

AHP task affinities
Binary Integer
Program)e
ac ionSp ntatkes s
Ta epre(rpts(IV) Compute TaxonomyeacTask-specific

LayoutCam. Pose Vanishing Pts.
2D Keypoints(fix) Cam. Pose
Jigsaw(nonfix)
Distance
Normals In-painting
Z-Depth Reshading
Colorization
Egomotion
Scene Class.
ass.
Occlusion Edges
3D Keypoints
Matching Object Class. (100)Semantic Segm.
Object Class. (1000)
0)m Proj.
2.5D Segm.Random
Curvature

1st Order
2nd Order
3rd Order
FrozenFigure 2: Computational modeling of task relations and creating the taxonomy. From left to right: I. Train task-specific networks. II. Train (firstorder and higher) transfer functions among tasks in a latent space. III. Get normalized transfer affinities using AHP (Analytic Hierarchy Process). IV. Find
global transfer taxonomy using BIP (Binary Integer Program).
Query Image

to map a hypothesis for one task onto a hypothesis for another [7], through information-based approaches [60], or
through modeling inductive bias [6]. For these guarantees,learning theoretic approaches usually rely on intractablecomputations, or avoid such computations by restricting themodel or task. Our method draws inspiration from theoretical approaches but eschews (for now) theoretical guarantees
in order to use modern neural machinery.

3. Method
We define the problem as follows: we want to maximize the collective performance on a set of tasks T ={t1 , ..., tn }, subject to the constraint that we have a limited
supervision budget γ (due to financial, computational, or
time constraints). We define our supervision budget γ to be
the maximum allowable number of tasks that we are willingto train from scratch (i.e. source tasks). The task dictionary
is defined as V=T ∪ S where T is the set of tasks which wewant solved (target), and S is the set of tasks that can betrained (source). Therefore, T − T ∩ S are the tasks that
we want solved but cannot train (“target-only”), T ∩ S are
the tasks that we want solved but could play as source too,and S − T ∩ S are the “source-only” tasks which we may
not directly care about to solve (e.g. jigsaw puzzle) but canbe optionally used if they increase the performance on T .
The task taxonomy (taskonomy) is a computationallyfound directed hypergraph that captures the notion of task
transferability over any given task dictionary. An edge between a group of source tasks and a target task represents a
feasible transfer case and its weight is the prediction of its
performance. We use these edges to estimate the globally
optimal transfer policy to solve T . Taxonomy produces afamily of such graphs, parameterized by the available supervision budget, chosen tasks, transfer orders, and transfer
functions’ expressiveness.
Taxonomy is built using a four step process depicted inFig. 2. In stage I, a task-specific network for each task in S

Surface Normals

Eucl. DistanceObject Class.

Scene Class.

Top 5 prediction:

Top 2 prediction:

sliding door
home theater, home theatrestudio couch, day bed
china cabinet, china closet
entertainment center

living room
television room

Jigsaw puzzle

Colorization

2D Segm.

2.5D Segm.

Semantic Segm.

Vanishing Points

2D Edges

3D Edges

2D Keypoints

3D Keypoints

3D Curvature

Image Reshading

In-painting

Denoising

Autoencoding

Triplet Cam. Pose

Room Layout

Point Matching

Cam. Pose (non-fixated) Cam. Pose (fixated)

Figure 3: Task Dictionary. Outputs of 24 (of 26) task-specific networks
for a query (top left). See results of applying frame-wise on a video here.

is trained. In stage II, all feasible transfers between sources
and targets are trained. We include higher-order transfers
which use multiple inputs task to transfer to one target. In
stage III, the task affinities acquired from transfer function
performances are normalized, and in stage IV, we synthesize a hypergraph which can predict the performance of any
transfer policy and optimize for the optimal one.
A vision task is an abstraction read from a raw image.
We denote a task t more formally as a function ft whichmaps image I to ft (I). Our dataset, D, contains for eachtask t a set of training pairs (I, ft (I)), e.g. (image, depth).
3

Representation Es(I)

FrozenDs→tTransfer Function

Source Task Encoder(e.g., curvature)

Target Task Output(e.g., surface normal)

Input

GroundTruth

Task
Specific

ReshadeLayout2D Segm.

Autoenc.

ScratchInput

GroundTruth

Task
Specific

ReshadeLayout2D Segm.

Autoenc.

ScratchTransfers Results (2k training images)...
3rd ordernd
2 order

2.5D
Surface Normal
Segmentation
Estimation

Es

I

Figure 4: Transfer Function. A small readout function is trained to map
Figure 5: Transfer results to normals (upper) and 2.5D Segmentation(lower) from 5 different source tasks. The spread in transferability among
different sources is apparent, with reshading among top-performing ones inthis case. Task-specific networks were trained on 60x more data. “Scratch”
was trained from scratch without transfer learning.

representations of source task’s frozen encoder to target task’s labels. Iforder> 1, transfer function receives representations from multiple sources.

Task Dictionary: Our mapping of task space is done
via (26) tasks included in the dictionary, so we ensure they
cover common themes in computer vision (2D, 3D, semantics, etc) to elucidate fine-grained structures of task space.
See Fig. 3 for some of the tasks with detailed definition ofeach task provided in the supplementary material. We include tasks with various levels of abstraction, ranging fromsolvable by a simple kernel convolved over the image (e.g.
edge detection) to tasks requiring basic understanding ofscene geometry (e.g. vanishing points) and more abstract
ones involving semantics (e.g. scene classification).
It is critical to note the task dictionary is meant to be asampled set, not an exhaustive list, from a denser space of
all conceivable visual tasks/abstractions. Sampling gives us
a tractable way to sparsely model a dense space, and the hypothesis is that (subject to a proper sampling) the derived
model should generalize to out-of-dictionary tasks. The
more regular / better sampled the space, the better the generalization. We evaluate this in Sec. 4.2 with supportive
results. For evaluation of the robustness of results w.r.t the
choice of dictionary, see the supplementary material.
Dataset: We need a dataset that has annotations for every task on every image. Training all of our tasks on exactly
the same pixels eliminates the possibility that the observed
transferabilities are affected by different input data peculiarities rather than only task intrinsics. There has not beensuch a dataset of scale made of real images, so we createda dataset of 4 million images of indoor scenes from about
600 buildings; every image has an annotation for every task.
The images are registered on and aligned with buildingwide meshes similar to [3, 101, 14] enabling us to programmatically compute the ground truth for many tasks without
human labeling. For the tasks that still require labels (e.g.
scene classes), we generate them using Knowledge Distillation [43] from known methods [104, 57, 56, 78]. See thesupplementary material for full details of the process anda user study on the final quality of labels generated using
Knowledge Distillation (showing < 7% error).

the encoder is large enough to extract powerful representations, and the decoder is large enough to achieve a good
performance but is much smaller than the encoder.

3.2. Step II: Transfer Modeling
Given a source task s and a target task t, where s ∈ S
and t ∈ T , a transfer network learns a small readout function for t given a statistic computed for s (see Fig 4). The
statistic is the representation for image I from the encoderof s: Es (I). The readout function (Ds→t ) is parameterizedby θs→t minimizing the loss Lt :h i
Ds→t := arg min EI∈D Lt Dθ Es (I) , ft (I) , (1)θwhere ft (I) is ground truth of t for image I. Es (I) may or
may not be sufficient for solving t depending on the relation
between t and s (examples in Fig. 5). Thus, the performanceof Ds→t is a useful metric as task affinity. We train transfer
functions for all feasible source-target combinations.
Accessibility: For a transfer to be successful, the latent
representation of the source should both be inclusive of sufficient information for solving the target and have the information accessible, i.e. easily extractable (otherwise, theraw image or its compression based representations would
be optimal). Thus, it is crucial for us to adopt a low-capacity(small) architecture as transfer function trained with a small
amount of data, in order to measure transferability conditioned on being highly accessible. We use a shallow fully
convolutional network and train it with little data (8x to
120x less than task-specific networks).
Higher-Order Transfers: Multiple source tasks cancontain complementary information for solving a target task(see examples in Fig 6). We include higher-order transfers
which are the same as first order but receive multiple representations in the input. Thus, our transfers are functions
D : ℘(S) → T , where ℘ is the powerset operator.
As there is a combinatorial explosion inof the numberthfeasible higher-order transfers (|T | × |S|forkorder),kwe employ a sampling procedure with the goal of filteringout higher-order transfers that are less likely to yield goodresults, without training them. We use a beam search: for3.1. Step I: Task-Specific Modeling
We train a fully supervised task-specific network foreach task in S. Task-specific networks have an encoderdecoder architecture homogeneous across all tasks, where4GT (Normals){Occlusion Edges + Curvature

Fully Supervisednd} = 2 order transfer

Image

GT (Reshade)

Autoencoding
Object Class. (1000)
Scene Class
Curvature
Denoising
2D Edges
Occlusion Edges
Egomotion
Cam. Pose (fix)2D Keypoint
3D Keypoint
Cam. Pose (nonfix)
Matching
Reshading
Z-DepthDistance
Normals
Layout2.5D Segm.
2D Segm.
Semantic Segm.
Vanishing Pts.

Fully Supervisednd{ 3D Keypoints + Surface Normals } = 2 order transfer

Au
toe
n
Cu codinD rvatu g
Oc 2 enois reclu D E ing
s
2Dion Edges
3D Key dges
Ke poinRe ypoin t
sha t
Z-Dding
Dis epthta
E No nce
Va gomrmals
nis otihin on2 gP
2.5D Se ts.
C
Ca am D gm
m. . P Seg .
Po ose m.
seOb(no (fix)jecntC
las Layfix)s. ( ou
M 100 tSe Scenatchin0)
ma e
nti Cla gIn- c Se ss
Pain gm.
C Jig ting
Raoloriz sawn
Ob Tas domation
jec k-S Prot C pe j.
cla
Ob Au ss ( ific
jec toe 100t C ncla od )Ob S ss. (1 ing
jec cen 00
t C e C 0)
Colass ( lasslo 10
Curizati 0)
D rvatuon
Oc 2enois reclu D ing
sio Edgn
Ca Ego Edges
m. mo esPti
In-ose ( onPa fixin )2D Jigting
Ca 3D Key sawm.
poK
Po ey int
se po
(no int
Ra Matcnfix
nd h )o
Re m Pingsha roj.
Z-Dding
Dis epth
No tancerm
2.5 Layoals
D
Se 2D SegutmaS m
Va ntic Segm.
nis eg .
Tas hin m.
k-S g P
pec ts.
ific

Image

Figure 6: Higher-Order Transfers. Representations can contain complementary information. E.g. by transferring simultaneously from 3D
Edges and Curvature individual stairs were brought out. See our publicly
available interactive transfer visualization page for more examples.

Figure 7: First-order task affinity matrix before (left) and after (right)
Analytic Hierarchy Process (AHP) normalization. Lower means bettertransfered. For visualization, we use standard affinity-distance method
dist = e−β·P (where β = 20 and e is element-wise matrix exponential).
See supplementary material for the full matrix with higher-order transfers.

transfers of order k ≤ 5 to a target, we select its 5 bestsources (according to 1st order performances) and include
all of their order-k combination. For k ≥ 5, we use a beamof size 1 and compute the transfer from the top k sources.
Transitive Transfers: We examined if transitive task
transfers (s → t1 → t2 ) could improve the performanceover their direct counterpart (s → t2 ), but found that thetwo had equal performance in almost all cases in both highdata and low-data scenarios. The experiment is provided in
the supplementary material. Therefore, we need not consider the cases where branching would be more than one
level deep when searching for the optimal transfer path.

responding (ith ) component of the principal eigenvector of
Wt0 (normalized to sum to 1). The elements of the principal
eigenvector are a measure of centrality, and are proportional
to the amount of time that an infinite-length random walk on
Wt0 will spend at any given source [62]. We stack the principal eigenvectors of Wt0 for all t ∈ T , to get an affinity
matrix P (‘p’ for performance)—see Fig. 7, right.
This approach is derived from Analytic Hierarchy Process [79], a method widely used in operations research tocreate a total order based on multiple pairwise comparisons.

3.3. Step III: Ordinal Normalization using Analytic
Hierarchy Process (AHP)

3.4. Step IV: Computing the Global Taxonomy
Given the normalized task affinity matrix, we need todevise a global transfer policy which maximizes collective
performance across all tasks, while minimizing the used supervision. This problem can be formulated as subgraph selection where tasks are nodes and transfers are edges. The
optimal subgraph picks the ideal source nodes and the bestedges from these sources to targets while satisfying thatthe number of source nodes does not exceed the supervision budget. We solve this subgraph selection problem using Boolean Integer Programming (BIP), described below,
which can be solved optimally and efficiently [41, 16].
Our transfers (edges), E, are indexed by i with the form({si1 , . . . , simi }, ti ) where {si1 , . . . , simi } ⊂ S and ti ∈ T .
We define operators returning target and sources of an edge:

We want to have an affinity matrix of transferabilities
across tasks. Aggregating the raw losses/evaluations Ls→tfrom transfer functions into a matrix is obviously problematic as they have vastly different scales and live in different
spaces (see Fig. 7-left). Hence, a proper normalization isneeded. A naive solution would be to linearly rescale eachrow of the matrix to the range [0, 1]. This approach fails
when the actual output quality increases at different speeds
w.r.t. the loss. As the loss-quality curve is generally unknown, such approaches to normalization are ineffective.
Instead, we use an ordinal approach in which the output
quality and loss are only assumed to change monotonically.
For each t, we construct Wt a pairwise tournament matrix
between all feasible sources for transferring to t. The element at (i, j) is the percentage of images in a held-out test
set, Dtest , on which si transfered to t better than sj did (i.e.
Dsi →t (I) > Dsj →t (I)).
We clip this intermediate pairwise matrix Wt to be in[0.001, 0.999] as a form of Laplace smoothing. Then we
divide Wt0 = Wt /WtT so that the matrix shows how manytimes better si is compared to sj . The final tournament ratio
0matrix is positive reciprocal with each element wi,jof Wt0 :0wi,j=EI∈Dtest [Dsi →t (I) > Dsj →t (I)].
EI∈Dtest [Dsi →t (I) < Dsj →t (I)]sources{si1 , . . . , simi }, ti

7−−−−−→ {si1 , . . . , simi }{si1 , . . . , simi }, ti

7−−−−→

targetti .

Solving a task t by fully supervising it is denoted as {t}, t .
We also index the targets T with j so that in this section, irepresents edges and j represents targets.
The parameters of the problem are: the supervision budget (γ) and a measure of performance on a target from eachof its transfers (pi ), i.e. the affinities from P . We can also
optionally include additional parameters of: rj specifying
the relative importance of each target task and `i specifying
the relative cost of acquiring labels for each task.

(2)

We quantify the final transferability of si to t as the cor5

Task

The BIP is parameterized by a vector x where each transfer and each task is represented by a binary variable; x indicates which nodes are picked to be source and which transfers are selected. The canonical form for a BIP is:
maximize: cT x ,subject to: Ax  b and x ∈ {0, 1}|E|+|V| .
Each element ci for a transfer is the product of the importance of its target task and its transfer performance:
ci := rtarget(i) · pi .
(3)

avg rand Task

Layout99.6 89.1
2D Edges 100 99.9
Pose (fix) 76.3 79.5
2D Segm. 97.7 95.7
Matching 86.8 84.6
Vanishing 99.5 96.4
Distance 92.4 92.1

avg rand

Scene Class. 97.0 93.4
Occ. Edges 100 95.4
Pose (nonfix) 60.2 61.9
2.5D Segm. 94.2 89.4
Egomotion 67.5 72.3
2D Keypnt. 99.8 99.4
3D Keypnt. 96.0 96.9

Table 1: Task-Specific Networks’ Sanity: Win rates vs. random (Gaussian) network representation readout and statistically informed guess avg.

Hence, the collective performance on all targets is the summation of their individual AHP performance, pi , weighted
by the user specified importance, ri .
Now we add three types of constraints via matrix A toenforce each feasible solution of the BIP instance corresponds to a valid subgraph for our transfer learning problem: Constraint I: if a transfer is included in the subgraph,
all of its source nodes/tasks must be included too, Constraint II: each target task has exactly one transfer in, Constraint III: supervision budget is not exceeded.
Constraint I: For each row ai in A we require ai · x ≤ bi ,
where


|sources(i)| if k = iai,k := −1
if (k − |E|) ∈ sources(i) (4)0otherwisebi = 0.

avg rand Task

Denoising 100 99.9
Autoenc. 100 99.8
Reshading 94.9 95.2
Inpainting 99.9 Curvature 78.7 93.4
Normals 99.4 99.5
Z-Depth 92.3 91.1
Mean
92.4 90.9

lows the user to specify any desired partition.
Network Architectures: We preserved the architecturaland training details across tasks as homogeneously as possible to avoid injecting any bias. The encoder architecture is
identical across all task-specific networks and is a fully convolutional ResNet-50 without pooling. All transfer functions include identical shallow networks with 2 conv layers(concatenated channel-wise if higher-order). The loss (Lt )and decoder’s architecture, though, have to depend on thetask as the output structures of different tasks vary; for all
pixel-to-pixel tasks, e.g. normal estimation, the decoder is a
15-layer fully convolutional network; for low dimensionaltasks, e.g. vanishing points, it consists of 2-3 FC layers.
All networks are trained using the same hyperparameters
regardless of task and on exactly the same input images.
Tasks with more than one input, e.g. relative camera pose,share weights between the encoder towers. Transfer networks are all trained using the same hyperparameters as thetask-specific networks, except that we anneal the learning
rate earlier since they train much faster. Detailed definitionsof architectures, training process, and experiments with different encoders can be found in the supplementary material.
Data Splits: Our dataset includes 4 million images. We
made publicly available the models trained on full dataset,but for the experiments reported in the main paper, weused a subset of the dataset as the extracted structure stabilized and did not change when using more data (explainedin Sec. 4.5). The used subset is partitioned into training(120k), validation (16k), and test (17k) images, each from
non-overlapping sets of buildings. Our task-specific networks are trained on the training set and the transfer networks are trained on a subset of validation set, ranging from
1k images to 16k, in order to model the transfer patterns under different data regimes. In the main paper, we report all
results under the 16k transfer supervision regime (∼10% of
the split) and defer the additional sizes to the supplementarymaterial and website (see Sec. 4.5). Transfer functions areevaluated on the test set.
How good are the trained task-specific networks? Win
rate (%) is the proportion of test set images for which abaseline is beaten. Table 1 provides win rates of the taskspecifc networks vs. two baselines. Visual outputs for a ran-(5)

Constraint II: Via two rows a|E|+j and a|E|+j+1 , we enforce that target j has exactly one transfer:
a|E|+j,i := 1{target(i)=j} ,b|E|+j := +1,(6)
a|E|+j+1,i := −1{target(i)=j} ,b|E|+j+1 := −1.
Constraint III: the solution is enforced to not exceed thebudget. Each transfer i is assigned a label cost `i , so
a|E|+2|V|+1,i := `i ,b|E|+2|V|+1 := γ.
(7)The elements of A not defined above are set to 0. The
problem is now a valid BIP and can be optimally solved ina fraction of a second [41]. The BIP solution x̂ corresponds
to the optimal subgraph, which is our taxonomy.

4. Experiments
With 26 tasks in the dictionary (4 source-only tasks), ourapproach leads to training 26 fully supervised task-specificnetworks,
22 × 25 transfer networks in 1st order, and 22 ×
25thorder, from which we sample according to thek for kprocedure in Sec. 3. The total number of transfer functions
trained for the taxonomy was ∼3,000 which took 47,886
GPU hours on the cloud.
Out of 26 tasks, we use the following 4 as source-only
tasks (described in Sec. 3) in the experiments: colorization,
jigsaw puzzle, in-painting, random projection. However,
the method is applicable to an arbitrary partitioning of thedictionary into T and S. The interactive solver website al6Supervision Budget 2
2D Keypoints

Jiggsaw
w

Transfer Order 1

Supervision Budget 8

2D Segm.

Coloorizaation

Occlusion Edges Normals

Autoencoding

Z-Depth

2D Edges
Semantic Segm.
Cam. Pose (fix)Distance
2D Segm.

3D Keypoints
Curvature

DenoisingNormals
2.5D Segm. Matching

Scene Class.

2D Edges
Scene Class.
Object Class.
(1000)

In-ppainnting

Transfer Order 4

2D Edges
Semantic Segm.
Cam. Pose (fix)2D Segm.

Distance
3D Keypoints
DenoisingCurvature
Normals
2.5D Segm. Matching

LayoutScene Class

Reshading Occlusion Edges
Object Class.
(1000) Randdom Projo.

Egomotion
Cam. Pose (nonfix)
Vanishing Pts.
In-ppainnting

Jiigsaww

Rando
om Projo.

Cam. Pose (nonfix)

Semantic
Segm. Distance

Z-Depth
2.5D Segm.
Curvature
Normals
Layout Vanishing Pts.
Reshading
3D Keypoints
Jigsaaw
Egomotion Occlusion Edges

2D Segm.

Colorrizattion

Z-DepthColoorizaation

Egomotion

ReshadingDistance
Vanishing Pts. 2D Edges
Layout2D Segm.

Matching
2D Edges
Semantic Segm.
Randdom Projo.
Vanishing Pts.
Autoencoding
2.5D Segm.
Scene Class.
Cam. Pose (fix)3D Keypoints
In-ppainttingg Layout
2D Keypoints
DenoisingCam. Pose (nonfix)
Colorizattion Occlusion Edges Object Class. (1000)

2D
Segm. 2D
Edges
DenoisingOcclusion Edges
Distance
2D Keypoints
Normals

2D Edges

Layout2D Keypoints

Egomotion
Curvature

2D Segm.
DenoisingDistance
Normals
Semantic
Segm.

Jiigsaww

Z-DepthScene Class.

In-ppaintting
Reshading

Z-DepthMatching
Cam. Pose (nonfix)
Denoising
Coloorizaation
Vanishing Pts.
Object Class.
Randdom Projo.
Autoencoding(1000)

Jiggsaaw

Autoencoding

Egomotion
Cam. Pose (fix)ReshadingIn-ppainnting

In-painting

ReshadingColorization

Object Class.
In-paintting 2D Segm. (1000)
Cam. Pose (fix)Jiigsaww
LayoutDenoising
Z-Depth
2D Edges
Vanishing Pts.
Egomotion
Distance
Cam. Pose (nonfix)
2D Segm.
Cam. Pose (nonfix)
LayoutLayoutDenoisingCam. Pose Vanishing Pts.
Coloorizattion
2D Keypoints
Matching
Normals(fix) Cam. Pose
Autoencoding
Vanishing Pts.
Scene Class.
Reshadingigsaaw(nonfix)
Autoencoding
Scene Class.
Distance
Z-DepthRanndom Projo.
Normals In-painting
2D Keypoints
2D Edges
Z-Depth Reshading
Normals
2D Keypoints
Coloorizaation
Occlusion Edges
Colorization
Egomotion
Distance
andoom Projo.
Scene Class.
Occlusion
Matching
Semantic Segm.
Edges
OcclusionEdges
3D Keypoints
Semantic Segm.
Curvature
Matching
3D Keypoints
2.5D Segm.
Semantic Segm.
Object Class. (1000)
Curvatureo.
2.5D Segm.Random Proj
3D Keypoints
Object Class.
Curvature
2.5D Segm.
(1000)
2D Edges
DenoisingCam. Pose(nonfix)

Jigssaw

3D Keypoints

Cam. Pose (fix)2D Segm.
.

Cam. Pose(fix) Vanishing Pts.

LayoutCurvature 2.5D Segm.

Object Class. (1000)
In-painting
2D Keypoints
Autoencoding

Autoencoding

DistanceOcclusion Edges
3D Keypoints
Semantic Segm.
Scene Class. Normals Jiigsaaw
Cam. Pose
2.5D Segm.
(fix)
Cam. Pose (nonfix)
Randomm Projo.
Z-DepthColoorizaation

Supervision Budget 8 - Order 4 (zoomed)

Curvature

Z-Depth
Jiggsaw

Matching
In-painting

Matching

Autoencoding

2D Keypoints

Z-DepthNormals

Reshading

Z-DepthDenoising
3D Keypoints
Random Projo.
Object Class. Vanishing
2.5D Segm.
Pts.
(1000)
Collorization
Scene Class.
2D Segm.
2D Edges
2D Keypoints
Curvature
Matching

2D Keypoints

Cam. Pose (fix)

Egomotion
Cam. Pose (nonfix)
Vanishing Pts.
Jiigsaww

2.5D Segm.

LayoutReshading Occlusion Edges
Object Class.
(1000) Randdom Projo.

Distance

3D Keypoints

Denoising
Autoencoding
2D Segm.

Colorrizattion

Egomotion

Autoencoding

2D Segm.

Autoencoding

2D Keypoints

Scene Class.
Matching

Denoising

LayoutIn-ppaintting

Egomotion

Jigsaw
ReshadingiDistance

Cam. Pose(nonfix)

Object Class.
(1000)

Coloorizaation

Jig
gsaw
w

Randoom Projo.

Transfer Order 2Curvature
Egomotion

2D Keypoints

Curvature Object Class. (1000)
LayoutReshading

LayoutRando
om Projo.

Cam. Pose (fix)Scene Class.

Cam. Pose (nonfix)

In-painttingg

2D Edges
Semantic
Autoencoding
Segm.

3D Keypoints

Normals
Semantic Segm.
Cam. Pose (fix)Normals

Supervision Budget 26

Occlusion Edges Cam. Pose (nonfix)

Occlusion Edges

Denoising

2D Edges
Reshading
2.5D Segm.
Semantic Segm.
Vanishing Pts.
Distance
Matching
Egomotion
Z-DepthSupervision Budget 15

Vanishing Pts.
Cam. Pose (fix)In-painttinggEgomotion
Scene Class.
Occlusion Edges

3D Keypoints

Matching
Semantic
Object Class.
Segm.
(1000)
2.5D Random Projo.
Segm.
Curvature

Figure 8: Computed taxonomies for solving 22 tasks given various supervision budgets (x-axes), and maximum allowed transfer orders (y-axes). One
is magnified for better visibility. Nodes with incoming edges are target tasks, and the number of their incoming edges is the order of their chosen transfer
function. Still transferring to some targets when tge budget is 26 (full budget) means certain transfers started performing better than their fully supervised
task-specific counterpart. See the interactive solver website for color coding of the nodes by Gain and Quality metrics. Dimmed nodes are the source-only
tasks, and thus, only participate in the taxonomy if found worthwhile by the BIP optimization to be one of the sources.

Supervision Budget Increase (→)

dom test sample are in Fig. 3. The high win rates in Table 1
and qualitative results show the networks are well trained
and stable and can be relied upon for modeling the task
space. See results of applying the networks on a YouTube
video frame-by-frame here. A live demo for user uploaded
queries is available here.
To get a sense of the quality of our networks vs. state-ofthe-art task-specific methods, we compared our depth estimator vs. released models of [53] which led to outperforming [53] with a win rate of 88% and losses of 0.35 vs. 0.47
(further details in the supplementary material). In general,
we found the task-specific networks to perform on par or
better than state-of-the-art for many of the tasks, though we
do not formally benchmark or claim this.

Budgetmax transfer order=1

max transfer order=4

Gain

max transfer order=1

max transfer order=4

Quality

Figure 9: Evaluation of taxonomy computed for solving the full task

4.1. Evaluation of Computed Taxonomies

dictionary. Gain (left) and Quality (right) values for each task using thepolicy suggested by the computed taxonomy, as the supervision budget
increases(→). Shown for transfer orders 1 and 4.

Fig. 8 shows the computed taxonomies optimized tosolve the full dictionary, i.e. all tasks are placed in T and S(except for 4 source-only tasks that are in S only). This wasdone for various supervision budgets (columns) and maximum allowed order (rows) constraints. Still seeing transfers to some targets when the budget is 26 (full dictionary)means certain transfers became better than their fully supervised task-specific counterpart.
While Fig. 8 shows the structure and connectivity, Fig. 9
quantifies the results of taxonomy recommended transfer
policies by two metrics of Gain and Quality, defined as:

Gain: win rate (%) against a network trained from scratchusing the same training data as transfer networks’. Thatis, the best that could be done if transfer learning was not
utilized. This quantifies the gained value by transferring.
Quality: win rate (%) against a fully supervised network
trained with 120k images (gold standard).
Red (0) and Blue (1) represent outperforming the reference method on none and all of test set images, respectively (so the transition Red→White→Blue is desirable.
White (.5) represents equal performance to reference).
7

Taxonomy

full sup.

Noroozi.[68]Zhang.[103]Zamir.[100]Agrawal.[1]Wang.[96]scratch

Task
Depth

ImageNet[51]Order Increase (→)
Order

Taxonomy Significance Test
9
7

88 88 93 89 88 84 86 43 .03

.04

.04

.03

.04

.03

.03

.02

.02

5

80 52 83 74 74 71 75 15 Scene Cls. 3.30
2.76 3.56 3.15 3.17 3.09 3.19 2.23 2.63
Sem. Segm. 78 79 82 85 76 78 84 21 -3
1

1.74 1.88 1.92 1.80 1.85 1.74 1.71 1.42 1.53

79 54 82 76 75 76 76 34 Object Cls. 4.08
3.57 4.27 3.99 3.98 4.00 3.97 3.26 3.46
Normals 97 98 98 98 98 97 97 6 .22

.30

.34

.28

.28

.23

.24

.12

Supervision Budget.15

Figure 11: Structure Significance. Our taxonomy compared with ran-

80 93 92 89 90 84 87 40 2.5D Segm. .21.34 .34 .26 .29 .22 .24 .16 .17
Occ. Edges 93 96 95 93 94 93 94 42 .16

Curvature

.19

.18

.17

.18

.16

.17

.12

dom transfer policies (random feasible taxonomies that use the maximum
allowable supervision budget). Y-axis shows Quality or Gain, and X-axis
is the supervision budget. Green and gray represent our taxonomy and random connectivities, respectively. Error bars denote 5th –95th percentiles.

.13

88 94 89 85 88 92 88 29 .25

.28

.26

.25

.26

.26

.25

.21

.22

79 78 83 77 76 74 71 59 Egomotion 8.60
8.58 9.26 8.41 8.34 8.15 7.94 7.32 6.85
80 76 85 79 77 78 70 36 Layout
.66

.66

.85

.65

.65

.62

.54

.37

Supervision Budget

predicable and strong structure. For graph visualization of
the all-for-one taxonomy policies please see the supplementary material. The solver website allows generating the taxonomy for arbitrary sets of target-only tasks.

.41

Figure 10: Generalization to Novel Tasks. Each row shows a novel
test task. Left: Gain and Quality values using the devised “all-for-one”
transfer policies for novel tasks for orders 1-4. Right: Win rates (%) of thetransfer policy over various self-supervised methods, ImageNet features,and scratch are shown in the colored rows. Note the large margin of win
by taxonomy. The uncolored rows show corresponding loss values.

4.3. Significance Test of the Structure
The previous evaluations showed good transfer results interms of Quality and Gain, but how crucial is it to use ourtaxonomy to choose smart transfers over just choosing any
transfer? In other words, how significant/strong is the discovered structure of task space? Fig. 11 quantifies this byshowing the performance of our taxonomy versus a large setof taxonomies with random connectivities. Our taxonomy
outperformed all other connectivities by a large margin signifying both existence of a strong structure in the space aswell as a good modeling of it by our approach. Complete
experimental details is available in supplementary material.

Each column in Fig. 9 shows a supervision budget. As
apparent, good results can be achieved even when the supervision budget is notably smaller than the number of solved
tasks, and as the budget increases, results improve (expected). Results are shown for 2 maximum allowed orders.

4.2. Generalization to Novel Tasks
The taxonomies in Sec. 4.1 were optimized for solving
all tasks in the dictionary. In many situations, a practitioner
is interested in a single task which even may not be in the
dictionary. Here we evaluate how taxonomy transfers to anovel out-of-dictionary task with little data.
This is done in an all-for-one scenario where we put one
task in T and all others in S. The task in T is target-onlyand has no task-specific network. Its limited data (16k) isused to train small transfer networks to sources. This basically localizes where the target would be in the taxonomy.
Fig. 10 (left) shows the Gain and Quality of the transfer
policy found by the BIP for each task. Fig. 10 (right) compares the taxonomy suggested policy against some of thebest existing self-supervised methods [96, 103, 68, 100, 1],
ImageNet FC7 features [51], training from scratch, and afully supervised network (gold standard).
The results in Fig. 10 (right) are noteworthy. The large
win margin for taxonomy shows that carefully selecting
transfer policies depending on the target is superior to fixedtransfers, such as the ones employed by self-supervised
methods. ImageNet features which are the most popular
off-the-shelf features in vision are also outperformed bythose policies. Additionally, though the taxonomy transfer
policies lose to fully supervised networks (gold standard) inmost cases, the results often get close with win rates in 40%
range. These observations suggests the space has a rather

4.4. Evaluation on MIT Places & ImageNet
To what extent are our findings dataset dependent, andwould the taxonomy change if done on another dataset? We
examined this by finding the ranking of all tasks for transferring to two target tasks of object classification and scene
classification on our dataset. We then fine tuned our taskspecific networks on other datasets (MIT Places [104] forscene classification, ImageNet [78] for object classification)and evaluated them on their respective test sets and metrics.
Fig. 12 shows how the results correlate with taxonomy’s
ranking from our dataset. The Spearman’s rho between thetaxonomy ranking and the Top-1 ranking is 0.857 on Places
and 0.823 on ImageNet showing a notable correlation. See
supplementary material for complete experimental details.

4.5. Universality of the Structure
We employed a computational approach with various design choices. It is important to investigate how specific tothose the discovered structure is. We did stability tests bycomputing the variance in our output when making changes
in one of the following system choices: I. architecture of8Transferring to ImageNet(Spearman’s correlation = 0.823)

Transferring to MIT Places(Spearman’s correlation = 0.857)
Top-5

Accuracy

Top-1

Top-5

Accuracy

Top-1

Figure 12: Evaluating the discovered structure on other datasets:
ImageNet [78] (left) for object classification and MIT Places [104](right) for scene classification. Y-axis shows accuracy on the externalbenchmark while bars on x-axis are ordered by taxonomy’s predicted performance based on our dataset. A monotonically decreasing plot corresponds to preserving identical orders and perfect generalization.

Figure 13: Task Similarity Tree. Agglomerative clustering of tasks
based on their transferring-out patterns (i.e. using columns of normalized
affinity matrix as task features). 3D, 2D, low dimensional geometric, andsemantic tasks clustered together using a fully computational approach.

scratched the surface in this regard. We also made a numberof assumptions in the framework which should be noted.
Model Dependence: We used a computational approachand adopted neural networks as our function class. Though
we validated the stability of the findings w.r.t various architectures and datasets, it should be noted that the results
are in principle model and data specific. The current model
also does not include a principled mechanism for handling
uncertainty or probabilistic reasoning.
Compositionality: We performed the modeling via a setof common human-defined visual tasks. It is natural to consider a further compositional approach in which such common tasks are viewed as observed samples which are composed of computationally found latent (sub)tasks.
Space Regularity: We performed modeling of a dense
space via a sampled dictionary. Though we showed a goodtolerance w.r.t. to the choice of dictionary and transferring
to out-of-dictionary tasks, this outcome holds upon a proper
sampling of the space as a function of its regularity. Moreformal studies on properties of the computed space is required for this to be provably guaranteed for a general case.
Transferring to Non-visual and Robotic Tasks: Given
the structure of the space of visual tasks and demonstrated
transferabilities to novel tasks, it is worthwhile to question
how this can be employed to develop a perception module
for solving downstream tasks which are not entirely visual,e.g. robotic manipulation, but entail solving a set of (a priori
unknown) visual tasks.
Lifelong Learning: We performed the modeling in one
go. In many cases, e.g. lifelong learning, the system is
evolving and the number of mastered tasks constantly
increase. Such scenarios require augmentation of thestructure with expansion mechanisms based on new beliefs.

task-specific networks, II. architecture of transfer function networks, III. amount of data available for training
transfer networks, IV. datasets, V. data splits, VI. choiceof dictionary. Overall, despite injecting large changes (e.g.
varying the size of training data of transfer functions by 16x,
size and architecture of task-specific networks and transfer
networks by 4x), we found the outputs to be remarkably
stable leading to almost no change in the output taxonomy
computed on top. Detailed results and experimental setup
of each tests are reported in the supplementary material.

4.6. Task Similarity TreeThus far we showed the task space has a structure, measured this structure, and presented its utility for transfer
learning via devising transfer policies. This structure canbe presented in other manners as well, e.g. via a metric ofsimilarity across tasks. Figure 13 shows a similarity tree forthe tasks in our dictionary. This is acquired from agglomerative clustering of the tasks based on their transferring-out
behavior, i.e. using columns of normalized affinity matrix
P as feature vectors for tasks. The tree shows how tasks
would be hierarchically positioned w.r.t. to each other whenmeasured based on providing information for solving othertasks; the closer two tasks, the more similar their role intransferring to other tasks. Notice that the 3D, 2D, low dimensional geometric, and semantic tasks are found to cluster together using a fully computational approach, whichmatches the intuitive expectations from the structure of task
space. The transfer taxonomies devised by BIP are consistent with this tree as BIP picks the sources in a way that allof these modes are quantitatively best covered, subject tothe given budget and desired target set.

5. Limitations and Discussion

Acknowledgement: We acknowledge the support of NSF(DMS-1521608), MURI (1186514-1-TBCJE), ONR MURI
(N00014-14-1-0671), Toyota(1191689-1-UDAWF), ONR
MURI (N00014-13-1-0341), Nvidia, Tencent, a gift by
Amazon Web Services, a Google Focused Research Award.

We presented a method for modeling the space of visualtasks by way of transfer learning and showed its utility in
reducing the need for supervision. The space of tasks is aninteresting object of study in its own right and we have only9

References[19] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. In International conference on machine learning, pages 647–655,
2014. 2[20] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782, 2016. 2[21] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever,and P. Abbeel. Rl2: Fast reinforcement learning via slow
reinforcement learning. arXiv preprint arXiv:1611.02779,
2016. 2[22] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training
help deep learning? Journal of Machine Learning Research, 11(Feb):625–660, 2010. 2[23] A. Faktor and M. Irani. clustering by composition–
unsupervised discovery of image categories. In European
Conference on Computer Vision, pages 474–487. Springer,
2012. 2[24] L. Fe-Fei et al. A bayesian approach to unsupervised oneshot learning of object categories. In Computer Vision,
2003. Proceedings. Ninth IEEE International Conferenceon, pages 1134–1141. IEEE, 2003. 2[25] L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning ofobject categories. IEEE transactions on pattern analysis
and machine intelligence, 28(4):594–611, 2006. 2[26] B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars.
Unsupervised visual domain adaptation using subspacealignment. In Proceedings of the IEEE international conference on computer vision, pages 2960–2967, 2013. 2[27] C. Finn, P. Abbeel, and S. Levine. Model-agnostic metalearning for fast adaptation of deep networks. arXiv
preprint arXiv:1703.03400, 2017. 2[28] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization.
CoRR, abs/1603.00448, 2016. 2[29] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and
P. Abbeel. Deep spatial autoencoders for visuomotor learning. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pages 512–519. IEEE, 2016. 2[30] C. Finn, T. Yu, J. Fu, P. Abbeel, and S. Levine. Generalizing
skills with semi-supervised reinforcement learning. CoRR,
abs/1612.00429, 2016. 2[31] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. Oneshot visual imitation learning via meta-learning. CoRR,
abs/1709.04905, 2017. 2[32] I. K. Fodor. A survey of dimension reduction techniques.
Technical report, Lawrence Livermore National Lab., CA(US), 2002. 2[33] R. M. French. Catastrophic forgetting in connectionist networks: Causes, consequences and solutions. Trends in Cognitive Sciences, 3(4):128–135, 1999. 2[34] R. Ge. Provable algorithms for machine learning problems.
PhD thesis, Princeton University, 2013. 2[35] S. Geman, D. F. Potter, and Z. Chi. Composition systems.
Quarterly of Applied Mathematics, 60(4):707–736, 2002. 2[1] P. Agrawal, J. Carreira, and J. Malik. Learning to see by
moving. In Proceedings of the IEEE International Conference on Computer Vision, pages 37–45, 2015. 8[2] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman,
D. Pfau, T. Schaul, and N. de Freitas. Learning to learn
by gradient descent by gradient descent. In Advances in
Neural Information Processing Systems, pages 3981–3989,
2016. 2[3] I. Armeni, S. Sax, A. R. Zamir, and S. Savarese. Joint
2d-3d-semantic data for indoor scene understanding. arXiv
preprint arXiv:1702.01105, 2017. 4[4] S. Arora, A. Bhaskara, R. Ge, and T. Ma. Provable bounds
for learning some deep representations. In International
Conference on Machine Learning, pages 584–592, 2014. 2[5] Y. Aytar and A. Zisserman. Tabula rasa: Model transfer
for object category detection. In Computer Vision (ICCV),
2011 IEEE International Conference on, pages 2252–2259.
IEEE, 2011. 2[6] J. Baxter. A bayesian/information theoretic model of learning to learn viamultiple task sampling. Mach. Learn.,
28(1):7–39, July 1997. 3[7] S. Ben-David and R. S. Borbely. A notion of task relatedness yielding provable multiple-task learning guarantees.
Machine Learning, 73(3):273–287, Dec 2008. 3[8] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE
transactions on pattern analysis and machine intelligence,35(8):1798–1828, 2013. 2[9] P. Berkhin et al. A survey of clustering data mining techniques. Grouping multidimensional data, 25:71, 2006. 2[10] E. Bienenstock, S. Geman, and D. Potter. Compositionality,
mdl priors, and object recognition. In Advances in neural
information processing systems, pages 838–844, 1997. 2[11] H. Bilen and A. Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in neural
information processing systems, pages 235–243, 2016. 2[12] J. Bingel and A. Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv
preprint arXiv:1702.08303, 2017. 2[13] O. Boiman and M. Irani. Similarity by composition. In
Advances in neural information processing systems, pages
177–184, 2007. 2[14] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Nießner,M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d:
Learning from rgb-d data in indoor environments. arXiv
preprint arXiv:1709.06158, 2017. 4[15] Z. Chen and B. Liu. Lifelong Machine Learning. Morgan& Claypool Publishers, 2016. 2[16] I. I. CPLEX. V12. 1: Users manual for cplex. International
Business Machines Corporation, 46(53):157, 2009. 5[17] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015. 2[18] C. Doersch and A. Zisserman. Multi-task self-supervised
visual learning. arXiv preprint arXiv:1708.07860, 2017. 2

10

[54] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.
Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015. 2[55] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people.
Behavioral and Brain Sciences, pages 1–101, 2016. 2[56] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional instance-aware semantic segmentation. arXiv
preprint arXiv:1611.07709, 2016. 4[57] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 4[58] F. Liu, G. Lin, and C. Shen. CRF learning with CNN
features for image segmentation. CoRR, abs/1503.08263,
2015. 2[59] Z. Luo, Y. Zou, J. Hoffman, and L. F. Fei-Fei. Label efficient learning of transferable representations acrosss domains and tasks. In Advances in Neural Information Processing Systems, pages 164–176, 2017. 2[60] M. M. H. Mahmud. On Universal Transfer Learning, pages
135–149. Springer Berlin Heidelberg, Berlin, Heidelberg,
2007. 3[61] J. Malik, P. Arbeláez, J. Carreira, K. Fragkiadaki, R. Girshick, G. Gkioxari, S. Gupta, B. Hariharan, A. Kar, and
S. Tulsiani. The three rs of computer vision: Recognition,
reconstruction and reorganization. Pattern Recognition Letters, 72:4–14, 2016. 2[62] N. Masuda, M. A. Porter, and R. Lambiotte. Random walks
and diffusion on networks. Physics Reports, 716-717:1 –58, 2017. Random walks and diffusion on networks. 5[63] M. Mccloskey and N. J. Cohen. Catastrophic interference inconnectionist networks: The sequential learning problem.
The Psychology of Learning and Motivation, 24, 1989. 2[64] L. Mihalkova, T. Huynh, and R. J. Mooney. Mapping andrevising markov logic networks for transfer learning. In
AAAI, volume 7, pages 608–614, 2007. 2[65] T. Mikolov, Q. V. Le, and I. Sutskever. Exploiting similarities among languages for machine translation. CoRR,
abs/1309.4168, 2013. 2[66] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Crossstitch networks for multi-task learning. In Proceedingsof the IEEE Conference on Computer Vision and Pattern
Recognition, pages 3994–4003, 2016. 2[67] A. Niculescu-Mizil and R. Caruana. Inductive transfer forbayesian network structure learning. In Artificial Intelligence and Statistics, pages 339–346, 2007. 2[68] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European
Conference on Computer Vision, pages 69–84. Springer,
2016. 2, 8
[69] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning to count. arXiv preprint
arXiv:1708.06734, 2017. 2[70] M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, J. Shlens,A. Frome, G. S. Corrado, and J. Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv
preprint arXiv:1312.5650, 2013. 2[36] R. Gopalan, R. Li, and R. Chellappa. Domain adaptationfor object recognition: An unsupervised approach. In Computer Vision (ICCV), 2011 IEEE International Conferenceon, pages 999–1006. IEEE, 2011. 2[37] A. Gopnik, C. Glymour, D. Sobel, L. Schulz, T. Kushnir,and D. Danks. A theory of causal learning in children:
Causal maps and bayes nets. 111:3–32, 02 2004. 2[38] A. Gopnik, C. Glymour, D. M. Sobel, L. E. Schulz,
T. Kushnir, and D. Danks. A theory of causal learning inchildren: causal maps and bayes nets. Psychological review, 111(1):3, 2004. 2[39] A. Gopnik, A. N. Meltzoff, and P. K. Kuhl. The scientist in
the crib: Minds, brains, and how children learn. William
Morrow & Co, 1999. 2[40] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. 2[41] I. Gurobi Optimization. Gurobi optimizer reference manual, 2016. 5, 6[42] K. Henry. The theory and applications of homomorphic
cryptography. Master’s thesis, University of Waterloo,
2008. 2[43] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,
2015. 4[44] J. Hoffman, T. Darrell, and K. Saenko. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 867–874, 2014. 2[45] Y. Hoshen and S. Peleg. Visual learning of arithmetic operations. CoRR, abs/1506.02264, 2015. 2[46] F. Hu, G.-S. Xia, J. Hu, and L. Zhang. Transferring deep
convolutional neural networks for the scene classification ofhigh-resolution remote sensing imagery. Remote Sensing,7(11):14680–14707, 2015. 2[47] I.-H. Jhuo, D. Liu, D. Lee, and S.-F. Chang. Robust visual
domain adaptation with low-rank reconstruction. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE
Conference on, pages 2168–2175. IEEE, 2012. 2[48] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014. 2[49] D. P. Kingma and M. Welling. Auto-encoding variational
bayes. arXiv preprint arXiv:1312.6114, 2013. 2[50] I. Kokkinos. Ubernet: Training auniversal’convolutional
neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv preprint
arXiv:1609.02132, 2016. 2[51] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
NIPS, pages 1097–1105, 2012. 8[52] B. Kulis, K. Saenko, and T. Darrell. What you saw is notwhat you get: Domain adaptation using asymmetric kernel transforms. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1785–1792.
IEEE, 2011. 2[53] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and
N. Navab. Deeper depth prediction with fully convolutional
residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239–248. IEEE, 2016. 7

11

[88] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan,
I. J. Goodfellow, and R. Fergus. Intriguing properties ofneural networks. CoRR, abs/1312.6199, 2013. 2[89] J. B. Tenenbaum and T. L. Griffiths. Generalization, similarity, and bayesian inference. Behavioral and Brain Sciences, 24(4):629640, 2001. 2[90] J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011. 2[91] J. B. Tenenbaum, C. Kemp, and P. Shafto. Theory-based
bayesian models of inductive learning and reasoning. In
Trends in Cognitive Sciences, pages 309–318, 2006. 2[92] D. G. R. Tervo, J. B. Tenenbaum, and S. J. Gershman. Toward the neural implementation of structure learning. Current opinion in neurobiology, 37:99–105, 2016. 2[93] C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and
S. Mannor. A deep hierarchical approach to lifelong learning in minecraft. In AAAI, pages 1553–1561, 2017. 2[94] S. Thrun and L. Pratt. Learning to learn. Springer Science& Business Media, 2012. 2[95] A. M. Turing. Computing machinery and intelligence.
Mind, 59(236):433–460, 1950. 2[96] X. Wang and A. Gupta. Unsupervised learning of visual
representations using videos. In Proceedings of the IEEE
International Conference on Computer Vision, pages 2794–
2802, 2015. 8[97] X. Wang, K. He, and A. Gupta. Transitive invariancefor self-supervised visual representation learning. arXiv
preprint arXiv:1708.02901, 2017. 2[98] T. Winograd. Thinking machines: Can there be? Are we,volume 200. University of California Press, Berkeley, 1991.
2[99] J. Yang, R. Yan, and A. G. Hauptmann. Adapting svm classifiers to data with shifted distributions. In Data Mining
Workshops, 2007. ICDM Workshops 2007. Seventh IEEE
International Conference on, pages 69–76. IEEE, 2007. 2[100] A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and
S. Savarese. Generic 3d representation via pose estimationand matching. In European Conference on Computer Vision, pages 535–553. Springer, 2016. 2, 8
[101] A. R. Zamir, F. Xia, J. He, A. Sax, J. Malik, and S. Savarese.
Gibson Env: Real-world perception for embodied agents.
In 2018 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). IEEE, 2018. 4[102] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals.
Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2016. 2[103] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision,
pages 649–666. Springer, 2016. 2, 8
[104] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using places
database. In Advances in neural information processing
systems, pages 487–495, 2014. 2, 4, 8, 9[71] M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher,and L. Guibas. Functional maps: a flexible representationof maps between shapes. ACM Transactions on Graphics(TOG), 31(4):30, 2012. 2[72] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A.
Efros. Context encoders: Feature learning by inpainting. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 2536–2544, 2016. 2[73] A. Pentina and C. H. Lampert. Multi-task learning withlabeled and unlabeled tasks. stat, 1050:1, 2017. 2[74] J. Piaget and M. Cook. The origins of intelligence in children, volume 8. International Universities Press New York,1952. 2[75] L. Y. Pratt. Discriminability-based transfer between neuralnetworks. In Advances in neural information processing
systems, pages 204–211, 1993. 2[76] S. R. Richter, Z. Hayder, and V. Koltun. Playing for benchmarks. In International Conference on Computer Vision(ICCV), 2017. 2[77] S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. science,
290(5500):2323–2326, 2000. 2[78] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211–252,
2015. 2, 4, 8, 9[79] R. W. Saaty. The analytic hierarchy process – what it is andhow it is used. Mathematical Modeling, 9(3-5):161–176,
1987. 5[80] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting
visual category models to new domains. Computer Vision–
ECCV 2010, pages 213–226, 2010. 2[81] R. Salakhutdinov, J. Tenenbaum, and A. Torralba. One-shotlearning with a hierarchical nonparametric bayesian model.
In Proceedings of ICML Workshop on Unsupervised andTransfer Learning, pages 195–206, 2012. 2[82] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and
P. Abbeel. Trust region policy optimization. CoRR,
abs/1502.05477, 2015. 2[83] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline
for recognition. In Proceedings of the IEEE conference on
computer vision and pattern recognition workshops, pages
806–813, 2014. 2[84] D. L. Silver and K. P. Bennett. Guest editors introduction:
special issue on inductive transfer learning. Machine Learning, 73(3):215–220, 2008. 2[85] D. L. Silver, Q. Yang, and L. Li. Lifelong machine learning
systems: Beyond learning algorithms. In in AAAI Spring
Symposium Series, 2013. 2[86] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng. Zeroshot learning through cross-modal transfer. In NIPS, pages
935–943, 2013. 2[87] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: A simple way to prevent neuralnetworks from overfitting. Journal of Machine Learning
Research, 15:1929–1958, 2014. 2

12