The Visual Task Adaptation Benchmark
Authors: Xiaohua Zhai
Joan Puigcerver
Alexander Kolesnikov
Pierre Ruyssen
Carlos Riquelme
Mario Lucic
Josip Djolonga
Andre Susano Pinto
Maxim Neumann
Alexey Dosovitskiy
Lucas Beyer
Olivier Bachem
Michael Tschannen
Marcin Michalski
Olivier Bousquet
Sylvain Gelly
Neil Houlsby
Venue:  (2020)
Keywords: representation learning
self-supervised learning
benchmark
large-scale study

Abstract: Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Yet, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one principle: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models competitive? Is self-supervision useful if one already has labels?

Under review as a conference paper at ICLR 2020
THE VISUAL TASK ADAPTATION BENCHMARK
Anonymous authors Paper under double-blind review
ABSTRACT
Representation learning promises to unlock deep learning for the long tail of vi- sion tasks without expansive labelled datasets. Yet, the absence of a unified yard- stick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are ei- ther too constrained (linear classification), limited in scope (ImageNet, CIFAR. Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one princi- ple: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models com- petitive? Is self-supervision useful if one already has labels?
INTRODUCTION
Deep learning has revolutionized computer vision. Distributed representations learned from raw pixels have enabled unprecedented performance on many visual understanding tasks. Hand-crafted features have been replaced with hand-annon anndttndtttted datasets, with thousands to millions of exam- ples (Krizhevsky) 2009; Russakovsky et al., 2015). By contrast, humans learn wide range vision tasks using just few examples per task. A key research challenge is close this gap in sample efficiency, and unlock deep learning for the long tail of problems without many labels.
The goal of improving sample efficiency has been approached from many angles, such as: few-shot learning (Fei-Fei et al. 2006),, transfer learning (Pan & Yang 2009), domain adaptation (Wang &
Deng 2018), and representation learning (Bengio et al. 2013)) Representation learning is studied in
many contexts: supervised pre-training (Sharif Razavian et al. 2014), self-supervised learning (Do-
ersch et al., 2015), semi-supervised learning (Chapelle et al. 2009). generative modeling (Don-
ahue al. 2017), and disentanglement learning (Higgins et al.)
2017). Yet many questions remain unanswered, such as: Do representations trained using ImageNet labels generalize to diverse tasks? Does the great progress on generative modeling translate in better general representations? self- supervision useful when labelled datasets already exist? Are semantic image-level labels useful for tasks that require understanding the structure of the world?
The lack of a common benchmark impedes the understanding of visual representations; each sub- domain has its own evaluation protocol, and it is hard assess different approaches. Benchmarks have been critical in other domains, such as RL (Mnih et al. 2013). natural image classification (Rus-) sakovsky al. 2015), and NLP (Wang et al.) (018). Inspired by these successes, we propose benchmark that follows similar principles: (i) we put minimal constraints to encourage creativity, (ii) focus on practical considerations, and (iii) make it challenging.
We present the Visual Task Adaptation Benchmark (VTAB). VTAB is based on single principle: better algorithm is one that solves a diverse set previously unseen tasks with fewest possible labels. The focus on sample complexity reflects our belief that learning with few labels is the key objec- tive of representation learning. Task diversity is also crucial to assess a representation's 's generality. Therefore, our benchmark extends beyond standard natural classification tasks, and includes those related to sensorimotor control, medical imaging, and visual reasoning.
We treat the benchmark tasks as unseen, but otherwise, learner, or transfer algorithm. Pre-training may be performed on any data (labelled or unlabelled), provided that it does not overlap with the benchmark tasks, which would violate the notion of unseen.
do not constrain the representation

---

Under review as a conference paper at ICLR 2020
To transfer the representations one may freeze, fine-tune, add new layers, apply domain adaptation, meta-learning, or any other technique.
With VTAB, perform an extensive representation study, which reveals: (i) Supervised pre- training on ImageNet yields useful representations for natural images. Surprisingly, these work on distant domains, but to lesser extent. (ii) Self- Supf-supe is effective, although other unsupervised methods, such as generative models appear less so. (iii) Combining self-supervised with supervision outperforms all other studied algorithms, even without additional unlabelled data. (iv) Restricted evaluation using a linear classifier could lead to different conclusions. Data, code, continuously updated results, and models used are made available at [anonymous-url]
2 THE VISUAL TASK ADAPTATION BENCHMARK
We seek algorithms that perform well on wide variety of unseen visual understanding tasks with few labels per task[ We first formal- ize this objective and then specify a practical benchmarking procedure measure progress.
Adspt on D2
Test
A dataset D a set of n instances {(x;, ys)) / with observations æ; E X and labels yı E Y. A prediction function is any mapping F X - Y (e.g. classifier). A (learning) algorithm, A, takes as input a dataset and outputs a prediction function. For example, A may be pre-trained neural network coupled with a training mechanism. An evaluation pro- cedure, Ex, takes F and outputs scalar mea- suring F's performance (e.g. test-set accuracy). We seek the algorithm that maximizes the expected performance over distribution of tasks Pr, where a task T a tuple containing a task-specific dataset distribution Dr and evaluation proce- dure. Given access to only samples per task we aim to maximize:
924
Upstream Data
Figure Overview of the VTAB protocol.
SCORE, (A) = EI-P, &y [A(D7)],
(1)
This general formulation requires a few clarifications. First, the task distribution needs to be appro- priate; desire spectrum of tasks covering those solvable by vision algorithm with human-like capabilities. Second, n may be varied to measure the relative sample complexity of different al- gorithms. In practice we choose it be similar to that of a modest labelling budget (Section B.I). Third, we assume that Pr is known, and the goal is to build an algorithm with the best inductive biases for solving samples from it. We desire an "open world", where evaluation data extremely diverse and can always be freshly sampled. Unfortunately, in practice the benchmark must con- tain a finite number of fixed test sets, therefore we must ensure that the algorithms have not been pre-exposed to specific evaluation samples, as described below.
2.1 A PRACTICAL BENCHMARK
We now describe the Visual Task Adaptation Benchmark (VTAB), designed to be the best possible proxy for Eq. (.. Fig. contains an overview diagram. We then discuss a suite of methods that may be used to optimize VTAB: deep representation learning.
Eq. is very difficult to compute in practice. One must define representative Pr and compute the desired expectation. If sampling the distribution, one must avoid overfitting to the sample. Finally, limiting computational complexity becomes paramount as the number of tasks grows.
Distribution over tasks We aim for universal visual understanding, so we informally define Pr as "Tasks that human can solve, from visual input alone.". See Appendix [ where validate this. Intuitively, such tasks should benefit from visual representations learned by observing and interact- ing with the natural world. The second clause eliminates tasks that require external knowledge that
In some practical settings additional unlabelled data may be available for unseen tasks in addition to the lahelled data. We omit this setting, leaving to future work.
2

---

Under review as a conference paper at ICLR 2020
is (currently) unreasonable for vision algorithm to acquire an extreme example would classifying objects grouped by a function of their spelling in a natural language. Section 2.2 details the samples.
Expectation over tasks We approximate the expectation over tasks by empirical average over a number of hand-picked samples. Ideally, we would sample a new task for each evaluation, as is possible in procedural environments, e.g. (Finn et al. 2017). However, real-world vision tasks are expensive to collect, so we define a fixed, representative set of samples from Py. While fixing the set reduces variance, it introduces risk of meta-overfitting.
Mitigating meta-overfitting To do this, we treat the evaluation tasks like a regular test set, and thus consider them unseen. This implies that algorithms that use pre-training must not pre-train on any of the evaluation tasks (even their unlabelled images). Upstream training should provide useful inductive biases for any draw from Pr, so should not use test samples. Fortunately, despite numerous test re-evaluations on popular ML benchmarks, such as ImageNet, progress seems to transfer to new data (Recht et al. 2018 2019 Kornblith al. 2019).
Unified implementation We use many diverse tasks, however, this could render usage impractical. As discussed, the algorithms must have no prior knowledge of the downstream tasks. Therefore, while it is permitted to run a hyperparameter search on each task, the search space cannot be task- dependent. To get meaningful results, we need to define architectures and hyperparameter searches that work well across the benchmark.
For this, we convert the tasks into classification problems. With a homogeneous task interface, we may control for possible confounding factors. For example, we may use the same architecture and hyperparameter sweep throughout. Of course, not all tasks can be efficiently modeled as classifi- cation with global labels. such as those requiring per-pixel predictions. Nonetheless, tasks such that success on VTAB requires learning of diverse set of visual features: object identifi- cation, scene classification, pathology detection, counting, localization, 3D geometry, and others.
design the
2.2 TASKS
VTAB contains 19 tasks which cover broad spectrum of domains and semantics. Appendix con- tains details. These are grouped into three sets: NATURAL, SPECIALIZED, and STRUCTURED.
The NATURAL group represents classical vision problems. These tasks contain natural images cap- tured using standard cameras. The classes may represent generic, fine-grained, or abstract objects. The group includes: Caltech101, CIFAR-100, DTD, Flowers102, Pets, Sun397, and SVHN.
The SPECIALIZED group also contains images of the world, but captured through specialist equip- ment. These images contain invariances from those in the NATURAL tasks. Nonetheless, humans recognize the structures therein, thus generic visual representation should also capture the visual concepts in these images. We have two sub-groups: remote sensing, and medical. Remote sensing includes Resisc45 and EuroSAT: aerial images of the earth captured using satellites or aerial pho- tography. Medical includes Patch Camelyon, metastases detection from microscopy images, and Diabetic Retinopathy, retinopathy classification from fundus images.
The STRUCTURED group assesses comprehension of the structure of a scene, for example, object counting, or 3D depth prediction. Most of these tasks are generated from simulated environments, whose structure is easy for human to determine, but whose domain differs greatly to datasets like ImageNet. These tasks are intended as a step towards useful representations for perceptual control. We include: Clevr: Simple shapes rendered in a 3D scene, we include two tasks: counting and depth prediction. dSpritee: Simple black/white shapes rendered in 2D, we include two tasks: location and orientation prediction. SmallNORB: Artificial objects viewed under varying conditions, we include two tasks: abject-azimuth and camera-elevation prediction. DMLab: Frames from rendered 3D maze. The task involves predicting the time for a pre-trained RL agent to navigate to an object. KITT7: frames captured from car driver's perspective. We predict the depth of the nearest vehicle.
2.3 REPRESENTATION AND TRANSFER LEARNING
Success on VTAB, and ultimately optimizing Eq. (T. requires some knowledge of Py. For example, algorithms that use CNN architectures have a useful inductive bias for vision. While these biases usually manually designed (e.g. through choice of an architecture), they also be learned.
3

---

Under review as a conference paper at ICLR 2020
For example: (i) Zoph & Le (2017) learn performant architectures, (ii) Bello al. (2017) learn optimization algorithms, (iii) Wong et al. (2018) learn hyperparameter search strategies, (iv) Cubuk et al. (2019) learn data augmentation functions, (v) Raghu et al. (2019) learn layer-wise initialization distributions, and (vi) pre-trained data representations can be learned.
While all of the above merit investigation for VTAB, we focus representation learning. Human visual perception is refined through years of observation and interaction with the world, resulting in system that solves new tasks with few in-domain labels. Likewise, we aim pre-train network that extracts useful features, or representations, from raw data.
Transfer strategy When representations are not pre-trained on the evaluation tasks themselves (which is explicitly forbidden in VTAB), they must be adapted to solve the new task (using limited in-domain data). State-of-the-art representation learning algorithms typically train deep neural network on some task(s), and then adapt the network to the new task. The simplest adaptation strategy is to freeze the network's weights and train another usually smaller model on top. When the upstream and downstream datasets differ significantly, fineantllyyyleuiiiiii fine-tuning the original weights is more effective (Yosinski et al.) 2014; Kornblith et al. 2019). VTAB does not constrain the transfer strategy; here we benchmark methods using fine-tuning since it tends to perform best.
Upstream training The representation learning literature often focuses on unsupervised learning as it may be applied to any data without human annotation. However, supervised data, where avail- able, may also produce good representations. Indeed, the most popular models used in practice are pre-trained on ImageNet labels (Huh et al., 2016) and refs therein). VTAB does not constrain the rype of data (supervised or unsupervised) used to pre-train the representations.
EXPERIMENTS
We use VTAB to evaluate popular representation learning algorithms. We ask four primary ques- tions: (i) How effective are these algorithms, such as ImageNet pre-training, across broad spectrum of tasks? (ii) How do different algorithm classes compare? (iii) Which method achieves the best overall performance on VTAB? (iv) How does VTAB compare to other benchmarking strategies?
3.1 SETUP
Data Section 2.2 describes the tasks. VTAB aims to assess adaptation with limited data, so we eval- uate primarily on 1,000 training examples per task, drawn i...d. from each training set. This simula- tes performance under a small per-task labelling budget which is the main goal of this benchmark. For completeness, also the full datasets. This allows learning as in-domain data increases, and to check performance against prior art.
assess the value of representation
Representation learning algorithms We evaluate the following representation learning algo- rithms: supervised, semi-supervised, self-supervised, and generative models. As a baseline we train a "from-scratch" model, starting from random weights. We evaluate a total of 16 algorithms.
Throughout, the models are pre-trained ImageNet 2
The supervised model is trained on 100% of the ImageNet labels (0up)100). For self-supervised learning. we include both image-based and patch-base models. The image-based models include ROTATION (Gidaris et al. 2018) and EXEMPLAR (Dosoviskkiy et al., 2014). The patch-based models include RELATIVE PATCH LOCATION (Doersch et 2015) and JIGSAW (Noroozi & Favaro 2016). We use the public implementations of Kolesnikov et al. (201)). The patch-based models are con- verted into image-level classifiers by averaging the representations from 3 x 3 grid of patches (see Appendix C.. The semi-supervised models are trained using 10% of labelled ImageNet data with an auxiliary loss on all of the data, see (Zhai et al. 2019). We use either rotation -OU-ROOTT 10%) or Exemplar SUP-EXEMPLAR- 10%) auxiliary losses. These models can also use all of the labels, denoted -OTA--RTTTOO---OO use GANs and VAFS. The most common way to extract representations from a GAN is from the discriminator, replacing its final linear layer. We use both the label-conditional and unconditional BigGAN discriminators (Brock et al.) 2019) (COND-BIGG) AN and UNCOND-BIGGAND using the public implementations of Chen et al. (2019) Lucic al. 2019 For autoencoders, the output of
and HET--EXMMPLAR-OOOE
For generative models,
ImageNet is VTAB evaluation task by design permit upstream training on this dataset.
4

---

Under review as a conference paper at ICLR 2020
the encoder can be used as the representation. We test VAEs (Kingma & Welling 2014). and WAEs with three distribution matching losses: GAN, MMD (Tolstikhin et al. 2018). and UKL Rubensteei et al. 2019).
Transfer algorithm VTAB permits any transfer algorithm. We use fine-tuning of the entire net- work here, since that performs best (Komblith al. 2019). Adding linear model to frozen net- works is also popular, so we compare to this approach in Section B.4
Hyperparameters Both the upstream pre-training of representations and transfer involve many hy- perparameters. One major choice is architecture. Bigger architectures almost always perform better (see Appendix and Kolesnikov et al. (2019). Therefore, for comparability, we ResNet00- or similar, for all methods. For the supervised and semi-supervised methods we use the architecture in He al. (2016). For GANs and auto-encoders we also use deep residual networks, but with ap- propriate modifications to train them successfully on their generative pre-training task. Appendix contains details.
The hyperparameters used for transfer also influence performance, and different datasets may require different settings. We run VTAB two modes: lightweight and heavyweight.
The lightweight mode performs a restricted hyperparameter search on each tasks, permitting fair comparison with few resources. Many hyperparameters are fixed, including optimizer, batch size, pre-processing, and weight decay. For each task, lightweight mode sweeps four hyperparameters: 2 initial learning rates, and 2 learning rate schedules. See Appendix [ for details. We choose short schedules to limit cost, but show in Section 3.5 that these yield near-optimal performance.
In heavyweight mode we perform a large hyperparameter search, limited only by resource availabil- ity. Here, we perform random search over learning rates, schedules, optimizers, batch size, train pre pre-processing functions, evaluation pre-processing functions, and weight decay. We include longer training schedules and higher resolution images. This mode is used establish the best attainable VTAB score with the algorithms studied. Details in Appendix
Due to the high cost of heavyweight mode, we perform Section 3.3 we show that although extensive tuning improves all methods, the relative performances mostly unaffected. For future study using VTAB, we recommend a similar lightweight setup that facilitates fair comparison without high cost. If large computation resources are available, the heavyweight mode may be used to push the state-of-the-art.
main study in lightweight mode. In
Protocol We perform model selection for each task on a validation set. We then re- re-train the best model on the training and validation (or 1000-example subset thereof), and evaluate on the test set. We run the test set evaluation three times and report the median score by default.
Metrics We evaluate all methods with top-1 accuracy. We consider other reported metrics in Ap- pendix[D To aggregate scores across methods, we compute the mean top-1 accuracy. We investigate more complex aggregation strategies in Section 3.5 but the relative performances remain the same, so we report mean accuracy due to its simplicity.
3.2 VTAB RESULTS
We run all 16 methods on 19 tasks with 1000 training examples and the full datasets using lightweight mode. We assess the value of representation learning over from-scratch training, and compare the performance of the different methods across the dataset groups. Fig. 4 shows two de- tailed per-task comparisons: (i) SUP-100% (ImageNet) versus FROM-SCRATCH which shows the value of standard ImageNet representations. (ii) SUP-ROTATION- 100% versus SUP-100% which shows the best performing method versus ImageNet. Appendix G Table 5 contains the complete results table.
Generative models These perform worst, often more poorly than training from scratch. This is perhaps unsurprising because scaling these models to complex datasets known to be challeng- ing. Nonetheless, despite SOTA generative quality on ImageNet, the GAN's discriminator does not provide useful representations. On the 1000-example subsets the UNCOND-BICE BIGGAN outper- forms from-scratch, but all generative models lose substantially on the full datasets. Interestingly, on the full dataset the GANs perform better than autoencoders on the NATURAL tasks, but worse on STRUCTURED. This indicates that they may be more strongly fit the ImageNet domain.
5

---

Under review as a conference paper at ICLR 2020
Natural
Specialized
Structured
Average Top-1 Accurocy
80-
60-
Samples
1000
Generativve
From Scratch
Self Supervised
Sami Supervised
Supervited
Generarive
From Scrateh
Self Supervised
Seri Supervised
Supervised
Generative
From: Scratch
Self Supervised
Semi Supervised
Supervised
Figure 2: The methods are divided into five groups: Supervised with 100% labels, semi-supervised with 10% labels, self-supervised without labels, generative models, and training from-scratch. The plots show results on NATURAL, SPECIALIZED, and STRUCTURED datasets, respectively. The bar height indicates the mean top-1 accuracy across all methods and 3 test repeats within each group. The error bars indicate the variance across methods and runs.
1000-Example Datasets
Full Datasets
90
90
Accuracy
80
Top-1 Accuracy
80
70
50
50
Natural
Natural
40
Specialized
40
Specializedd
Structured
Structured
20
20
10
10
WAE GAN
WAE-UKL
VAE
WAE-MMO
Cond-Big
From Scratch
Uncond BigGAN
Jigsal
Rel Pal. Loc
Exemplar
Rotation
Semii Rotation- 1.%%
Semi-Exemple Exemppaa 10%
Sup 100%
Sup Exemplar ar- 100%
Sup-Retation-1000 100%
WAE-UKL
WAE- GAN
Cond BigGAN
WAE-MMD
Uncond BigGAN VAE
From-Scratch
j/gsaw
Rel.Pal PalPaa. Loc
Exemplar
Rotation
Semi-R Semi-Rotatinn Rotation 10%
Seml-Exem wxi.-ppleaaa----. 1.0%
Sup 100%
Sup eep-xxmplla ar- 100%
Sup-Rotaa -ooo--Rtttionn1111 100%
Figure 3: Average top-1 accuracy across the tasks in each group. The x-axis indexes the methods.
The methods are ordered according to their average accuracy across all datasets (dashed curve).
Similarly, Ravuri & Vinyals (2019) show that models with high sample quality fail generate data from which an accurate classifier can be trained. Explicit consideration of representation quality, in addition to generative scores, may lead further progress in generative modeling research.
Self-supervised All self-supervised representations outperform from-scratch training. The best, ROTATION attains 60.4% mean top-1 accuracy, while FROM-SCRATCH attains 43.1% (1000- example datasets). Methods applied to the entire image (ROTATION, EXEMPLAR) outperform patch- based methods (JIGSAW, REL.PAT. Loc.). However, the patch based methods perform slightly better on DTD (Describable Textures) and Retinopathy (Fundus images), see Appendix G Table These tasks require sensitivity local textures, which is well represented by patch- based methods. Fig- ureßishows that self-supervised perform worse than supervised methods on the NATURAL datasets, but similarly on SPECIALIZED, and even better on some STRUCTURED tasks. This indicates that ImageNet labels may encode sub-optimal invariances for non-natural tasks.
(Semi-)Supervised Supervised models perform best. The benefits are most pronounced on the NATURAL tasks, whose domain and task-semantics arguably most similar ImageNet classi- fication. The combination of self-supervision with 10% labels (semi-supervisee performs only marginally worse (1 1.5%) than methods with 100% labels. Interestingly, there is benefit of com-

---

Under review as a conference paper at ICLR 2020
80
Sup-Rotation-22 -0-0-0
Top Accuracy Deta
70
Top-1 Accuracy Delta
60
50
Sup- 100%
40
2-
30
20
10
Sup- Sup-100s 100%
SHORD tiee
Clour Des
Ceur-Coun:
Camelyon
dSpr-Drintt
eSpr
RUmBAT-
SNORS SNOns-aain-
SVIN
KITTHDXS:
Sun301
RusiscA--
CIFAR 100
DTD-
Howes 102-
Caneehtol.
Pets
From-Scratch
CIPAR-100
Retiropathy
SMORI--1 SOORI-FIe
Pets
Sun Sun307
DKL
EuroSAT-
Catech101-
Reslrcos-
Clewr Dot
SYHN-
Flowor102.
d5pri Lac-
Camelyon
DTO
SKORB-Adm
dipe dip--rient-
Clevs
KITT-Dk:
Figure Absolute difference in top-1 accuracy between method pairs for each dataset. The bar colour indicates the task group: NATURAL, o SPECIALIZED, and STRUCTURED. Left: supervised pre-trane pre-training yields large improvement on the
SUP-100% versus FROM-SCRATCH NATURAL datasets and some others. Right: SUP-ROTATION- 100% versus SUP-100% the additional self-supervised loss on top of supervised loss yields improvements, especially on the
STrUCTURED tasks. Note that the y-scales differ; the left-hand side's differences are much larger.
Caltech101
CIFAR- 100
DTD
Rowess 102
Pots
SVHN
Sun397
Camelyon
EbroSAT
Rosise45
RRtinopathy
Clev-Count
CloveDist
DMLab
KITT--ist
dSprLLoc
dSpr-Ori
SNORD- Azim
SNORD Bll
Mean
From From--cratch Scratch
60.2
24.5
51.1
73.4
43.1
88.3
14.9
82.1
93.3
63.9
75.4
53.0
57.3
36.6
55.7
88.6
67.6
52.2
81.6
61.2
Sup-100%
89.7
52.7
65.4
94.2
90.1
91.0
35.5
87.3
96.0
84.0
78.0
59.0
63.0
44.4
80.1
89.3
70.7
57.5
54.7
72.8
Sup-Rot-100%
88.0
48.9
68.1
94.7
89.0
91.8
33.7
86.2
96.6
85.2
78.5
66.5
61.5
45.5
77.2
88.8
71.8
59.4
52.0
72.8
Sup-Ex- hop-EE--100e
100%
90.4
50.4
66.1
94.3
90.6
91.3
35.2
86.7
95.8
83.3
78.4
73.
62.4
44.1
79.7
87.5
71.1
62.8
54.8
73.6
From Sera-hhhhcchhhh
74.5
77.8
67.1
85.8
70.9
97.0
70.1
91.2
98.8
94.3
82.8
99.8
96.7
76.6
68.4
100.0
96.7
99.9
94.0
86.4
Sup-100%
93.0
83.4
73.7
91.3
92.7
97.5
75.6
87.3
98.8
96.1
83.4
100.0
97.0
78.8
81.0
100,0
96.8
100.0
98.5
91.1
Sup Rot- 100%
93.5
84.0
76.8
97.4
92.6
97.4
75.6
86.5
99.1
96.3
83.7
100.0
96.8
79.6
82.4
100.0
96.8
100.0
96.7
91.3
Sup-ES, E---Ex- 100%
93.8
83.1
76.5
97.8
92,9
97.5
75.3
86.5
99.0
96.3
83.7
99.9
96.8
79.3
82.8
100.0
96.7
100.0
99.1
91.4
Table 1: Top-1 accuracy of the top performing models and Resnet50 from-scratch.
bining self-supervised losses with 100% labelled ImtgeNet, i.e. without additional unlabelled data, see Fig. SUP-ROTATION-1009S attains 66.4%, whereas SUP-ROTATION-106 100% attains 68.0%.
3.3 HEAVYWEIGHTT HYPERPARAMETER SWEEPS
We compare FROM-SCRATCH and the three best models: SUP-100% SUP-ROTATION- 1 100%, and SUP-EXEMPLLAR-O0 100% using the heavyweight hyperparameter search. Table the results. The hyperparameters selected are shown in Appendix [ As expected, all methods improve signif- icantly, c.g. FROM-SCRATCH improves from 43.1% to 61.2% (1000-examples). Prior work (He) et al. 2018) shows that with sufficient data and training steps, from-scratch training is competitive for detection. We observe, with the heavy sweep, that although the gap to SUP-100% is smaller, it is still substantial: SUP-1005 attains 72.8%. Again, addition of self-supervision helps, but the margin smaller. SUP-EXEMPLAR- 00% performs best with a VTAB (1000-example) score of 73.6.
We check our results against those recently reported in the literautre (Appendix Table B) Our results are comparable: behind on highly popular tasks which more complex architectures have been applied (e.g. CIFAR), but ahead in others. In Appendix L we show that simply increasing the architecture size improves VTAB performance.
In summary, pre-trained representations still offer substantial value with heavy hyperparameter tun- ing and long training schedules on VTAB. The overall ranking of methods remains the same, with the exception that SUP-EXEMPLAR- 100% now outperforms ROTARION 100%, although these methods' performances always similar.
7

---

Under review as a conference paper at ICLR 2020
3.4 FROZEN FEATURES EXTRACTORS
Representations are often evaluated by training a linear layer on a frozen model (Kolennkovv al., 2019). ImageNet is often used for linear evaluation, but this is meaningless for the mod- els we consider because many use ImageNet labels for pre-training. However, linear evalu- ation is also used in transfer setting (Goyal er al.] (0019)), so we apply this protocol to the VTAB tasks, and contrast to fine-tuning used above. The full protocol resembles the lightweight fine-tuning sweep, see Appendix[K]
1.0
1000 Example Datasets
Kendall Rank Correlation
0.8
0.6
0.4
0.2-
0.0
-0.2
Sumt-
Pes-
Catcohill
Howw
Sun399-
arah 100-
DTD-
Recingentyy
Canelonn
turestt.
Rerocit
covt Dis-
ClonnCom
DMLab-
dpporren
KITION-
Fig. 5) shows the per-task correlation between linear and fine-tuning. Appendix K. Table y contains the full table of results. We first note that linear evaluation significantly lowers per-
Figure 5: Kendall rank correlation coefficient be- tween fine-tuning and linear evaluation on each dataset.
formance, even when downstream data is lim- ited to 1000 examples. For example, SUP-1009; attains 66.4% with fine-tuning (lightweight sweep), but 57.3% with linear. Linear transfer would not by used in practice unless infrastructural con- straints required it. Second, Fig. 5 shows that on many datasets, particularly SPECIALIZED and STRUCTURED, the correlation is low. Linear evaluation may lead to different conclusions, for ex- ample, BOOND-BIGGAN attains 43.3% (1000-examples) on linear, outperforming both both patch- based self-supervised methods (REL. PxT.LOC and JIGSAW). Yet when fine-tuned these patch-based Another discrepancy is between semi-supervised and supervised. REMI-ROTATION- 10% and SOMI-EXEMPLAR-- 10% perform 1 1.5% worse than SUP- 100% with fine-tuming but 4 - 5% worse with linear evaluation. The self-supervised meth- ods extract useful representations, just without linear separability. Previous works (Kornblith et al.] 2019 Kolesnikov et al. 2019) claim that linear evaluation results are sensitive additional factors that we do not vary, such as ResNet version or pre-training regularization parameters. We conclude that linear evaluation poor proxy for overall reduced sample complexity.
methods attain around 53%, but the GAN, 41 44%% 2
3.5 ANALYSIS
Per-Task Metric In previous works, some datasets are reported using alternative metrics to top- 1 accuracy, namely: mean-per-class accuracy (for Caltech101) or Cohen's quadratic kappa (for Retinopathy). We study whether these metrics reveal different results to top-1 accuracy. We find that, although there are some minor ranking differences, the overall picture remains unchanged. Kendall's ranking correlation scores, with respect to top-1 accuracy, are: 1.0 (mean per-class accuracy), 0.97 (quadratic kappa). Appendix shows the full rankings according to each metric.
Metric Aggregation Across Tasks Mean top-1 accuracy across tasks creates an implicit weighting. First, some tasks use the same input data (c.g Clevr-Count and Clevr-Dist), thus upweighting those domains. Second, the task groups differ in size. Third, the tasks exhibit different performance ranges across methods. Therefore, we compare seven different ranking aggregation strategies: (i) Macro- average across datasets (grouping tasks with the same input data). (ii) Macro-average across groups (merging tasks in the same group, NATURAL, etc.). (iii) Geometric mean. (iv) Average rank. (v) Average rank after small perturbation of the scores with noise. (vi) Robust (binned) average rank. (vii) Elimination rank- - equivalent to an "Exhaustive Ballot".
All strategies have high Kendall's correlation with the vanilla mean across tasks is high (above 0.9). The most dissimilar strategy on the 1000-example datasets is the perturbed average rank, with correlation 0.91. Therefore, we mean top-1 accuracy because it is simple, interpretable, and can be computed for each method independently, unlike rank-based aggregation. See Appendix for details.
'Surprisingly fine-tuning the conditional GAN is slightly worse than linear evaluation. Fine-tuning also involves training linear head from-scratch. It seems that while doing the representations in the rest of the model are degraded slightly.
8

---

the results. The top-5 subsets tend to span different domains, Under review as a conference paper at ICLR 2020
Representative Subset For rapid- pnpitpprototypingg representative subset of VTAB may be useful. We compute the rank correlation between the mean scores produced by each (25 subsets of five tasks, and the full suite, Empendix[F]conaaae but differ to each other. Although using subset might be useful for screening models, they were computed using our particular set of models, and may correlate less well in other experiments. Using subsets also increases the risk of meta-overfittnng which VTAB aims avoid by having many tasks.
Limiting Evaluation Cost The cost near linear in the schedule length. To determine a minimal, yet meaningful, schedule, sweep over schedules ranging from 40 to 40,000 steps, with batch size 512 on all datasets. Fig. [ summarizes the results, details Figs. [9 and (0) Appendix M Most runs reach their optimal performance within 1,000 steps (updates), and do not improve significantly when trained for longer.
0.9
We fine-tune the models using 16 Google Cloud TPU-v3 hardware accelerators. We conduct an additional experiment assess whether our re- sults can be reproduced with more basic hard- ware setup. We evaluate SUP-100% on all
0.8
0.7
Accuracy
0.6
VTAB tasks (1,000 examples) using single NVidia P100 GPU with 1,000 steps, 64 images per mini-batch, and a learning rate of 0.01 In this experiment we match the mean perfor- mance of the same model reported in Table
0.5
Sup-100% (Full)
Sup-100% (1000)
0.4
Rotation (Full)
0.3
Rotation (1000)
30
100
1000
10000
50
Total training steps
RELATED WORK
Figure 6: Performance of two methods under var- ious evaluation time constraints.
Vision Benchmarks The Visual Decathlon (Rebuffi al.. 2017 contains classification tasks: Omniglot, and nine using natural images. Four overlap with VTAB. but VTAB includes several other domains and tasks. Importantly, these benchmarks have opposite evaluation protocols: The Visual Decathlon allows direct joint optimization on the tasks, but forbids external data. By contrast, VTAB forbids multi-task learning, but permits transfer from any arbitrary dataset- this simulates one-off training of representations which are then applied to novel tasks. Further, VTAB considers a low- sample regime (1000-examples), which reflects performance under a reasonable labelling budget. We conduct experiments showing that the methods ranked according to VTAB are more likely to transfer to new tasks, than those ranked according to the Visual Decathlon (Appendix N
The Facebook AI SSL challenge (Goyal al. 2019b) was proposed to evaluate self-supervised models. It contains two datasets of natural images, with four tasks, including classification, detce- tion, and low-label classification. The original paper (Goyal al., (019a) also includes navigation and surface normal estimation. In contrast, VTAB uses classification tasks only to admit task- independent implementationss. VTAB attains diversity with many tasks from alternative domains and with different semantics (localization, counting, etc.). VTAB experiments include self-supervised methods, but outperform supervised ImageNet models. Most importantly, Facebook SSL requires transfer via shallow network stacked on fixed CNN, whereas VTAB permits any transfer, focussing only on performance with few samples.
restricted them. Indeed, a challenge posed by the VTAB benchmark is to
Meta-Dataset (Triantafillou et al. 2019) contains natural image classification tasks. The protocol differs to VTAB: most of Meta- Dita--Dattt'ss s evaluation datasets are also in the training set. The train/test split is instead made across classes. Meta-Dataset is designed for few-shot learning, algorithms that use considerably fewer than 1000 samples. This regime requires different solutions to VTAB.
Other ML Benchmarks Benchmark suites have driven progress in other areas of machine learn- ing, such the GLUE (Wang et al.) 2018) and the NL Decathlon (McCann et al. (218)) in NLP, or the Atari suite (Mnih et 2013 and Mujoco environments (Todorov et al. (012)) in RL. NLP, the GLUE Benchmark (Wang et al. 2018) has been central benchmark used to measure progress of BERT (Devlin et al. 2018) and variants.
Representation Learning Evaluation
Popular evaluations for representation learning are li- near/MLP and semi- semi-supervised. In linear/MLP PaeearrMLeevllluttioo widely used in for self-supervised repre- sentations (Doersch et al. 2015: Zhang al. 2016 Noroozi & Favaro! 2016 Doersch & Zisserman

---

Under review as a conference paper at ICLR 2020
2017). the weights of a pre-trained network are frozen, and a linear layer/MLP is trained top to predict the labels. Evaluation is often performed on the same dataset that was used to train the repre- sentations, but using all labels, defeating the goal of sample efficiency. A semi-supervised protocol is more realistic and performs better (Zhai et al.; 2019). This evaluation is performed on a single dataset, by contrast VTAB concerns transfer unseen tasks. Linear evaluation is sensitive small upstream training details, such as the ResNet version or regularization, that make little difference when training end-to-end (Kornblith al. 2019; Kolesnikov et al. 2019). Other intrinsic scores have been proposed to measure the disentanglement of representations (Locatello et al.) 2019), or the mutual information between the input and representations (Hjelm
2018). Both were shown to be weakly correlated with representation utility (Locatello et al. 2019 Tschannen et al.) 2019).
Generative Models Evaluation Generative models are usually evaluated by generative qual- ity, difficult goal with numerous proxy metrics, such as reconstruction error, Frechet Inception Distance (Heusel et al., 2017), Inception score (Salimans et al., 2016), precision-recall (Sajjadi
ct al. 2018 log likelihood (for auto-regressive models), or the evidence lower bound (for VAEs).
Sometimes generative models are also evaluated using linear classification (Radford et al.
2016
Donahue al., 2017: Dumoulin al., 2017). or semi-supervised learning (Kingma et al.
2014
Warayanaswamyy et al. 2017 Tschannen et al. (018))
DISCUSSION
Our results illuminate the current state of representation learning. First, ImageNet labels provide a strong baseline, especially for natural datasets. However, on specialized datasets e.g. medical, distant domains, or tasks that require structured understanding, their value is reduced. Research on how to train on more diverse data sources, such as video, may be the key to moving beyond the "ImageNet domain" towards general, more human-like, visual representations.
Second, generative models - at least the common ones appear not to be competitive representa- tion learners Generation may be useful endeavour in of itself, but if generative losses are used as a means towards representing data, then more explicit representation evaluation may assist progress.
Third, self self- supervised learning appears promising. being the best performifo unsupervised method. When combined with few labels (10%), these methods perform almost as well as fully supervised ImageNet pre-training. Self-supervised even provides some value on top of ImageNet labels on the same data. We observe modest, yet significant, improvements over pure supervised learning, especially on tasks that require structured understanding. ImageNet labels encode lots of useful information, but also encode invariances that universal representation should instead be sensitive to. For example, the distance to an object.
Representation learning is inherently tied to transfer learning; without a transfer strategy represen- tations cannot be used for new tasks. Here, we elect fine-tuning since it performs better than the popular linaarmoder alternative, and is known to be less brittle. However, it would be interesting to research alternative strategies from domain adaptation (Wang & Deng 2018). residual adapters (Re- buffi et al. 2017). transfusion (Raghu et al. 2019), and other fields.
VTAB may be used to optimize many design choices involved in learning generalizable representa- tions. Here, we study upstream training loss. We therefore control other factors such as hyperparam- eter sweep size, architecture, transfer algorithm, downstream preprocessing, and upstream training data. Varying these factors, and others, to improve VTAB valuable future research. However, the effect of each factor should be isolated from other confounders, such as hyperparameter sweep size. The only approach that is out-of-bounds to condition the algorithm explicitly on the test tasks, since this would compromise representation generalization.
Code and data to run the Visual Task Adaptation Benchmark is made available, and progress is pub- All pre-trained models presented are also released. We
licly monitored at [anonymous-ur/]] hope that VTAB drives representation learning towards the goal of making deep learning accessible to the many applications without vast labelling budgets.
'Concurrent work, BigBiGAN DDnnhuee & Simonyan 2019 claims better representations, this would be interesting evaluate in future work.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. In Ad- vances Neural Information Processing Systems, 2018.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
Irwan Bello, Barret Zoph. Vijay Vasudevan, and Quoc Le. Neural optimizer search with rein- forcement learning. In International Conference on Machine Learning, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE Trans- actions on Neural Networks, 2009.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data. In IEEE Conference on Computer Vision and Pattern Recognition. 2019.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In IEEE Interna- tional Conference on Computer Vision, 2017.
Carl Doersch, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In IEEE International Conference on Computer Vision, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning, arXiv preprint arXiv:1907.0255 2019.
Jeff Donahue, Philipp Krähenbülhl, and Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations, 2017.
Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems, 2014.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. In International Conference on Learning Representations, 2017.
Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In International Conference World Wide Web, 2001.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 2006.

---

Under review as a conference paper at ICLR 2020
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research, 2013.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. arXiv preprint arXiv:1905.0/2335 2019a.
Priya Goyal, Dhruv Mahajan, and Ishan Misra. Facebook AI Self-Supervision Challenge, 2019b. URL https://sites httpoo//......o.ooooooo//aaaaaaa
home
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision. Springer, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking ImageNet pre- pre-train mee-trinning. arXiv preprint arXiv: 1811.08881.88 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. GANs trained by two time-scale update rule converge Nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, volume 3, 2017.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2018.
Karan Grewal, Phil Bachman, Adam
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, and Zhifeng Chen. Gpipe: Efficient training giant neural networks using pipeline parallelism. arXiv preprint arXiv: 11.111111..00 1.06665, 2018.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learn- ing? arXiv preprint arXiv:1608 08614, 2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Kaggle and EyePacs. Kaggle diabetic retinopathy detection, July 2015. URL https://wwel //ww. data
reteeticcreetttpaaee kaggle le. hct.:////ddaabbeei-eeeeeerreeeeo
Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 1945.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, International Conference on Learning Representations, 2014.
Diederik Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
12

---

Under review as a conference paper at ICLR 2020
Simon Korblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. IEEE Conference on Computer Vision and Pattern Recognition, 2004.
Fei-Fei Li, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE Transac- fions on Pattern Analysis and Machine Intelligence, 2006.
Francesco Locatello, Stefan Bauer, Mario Lucic. Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. International Conference on Machine Learning, 2019.
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. fighhfiieeltty image generation with fewer labels. In International Conference on Machine Learn- ing. 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1806v 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop 2013, 2013.
Siddharth Narayanaswamy, T Brooks Paige, Ban-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011.
M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes.
In Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, 2016.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems. 2018.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 2009.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Repre- sentations, 2016.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning with applications to medical imaging. arXiv preprint 1xXiv:1902.07208, 2019.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models.
arXiv preprint arxiv: 1905 10887, 2019.
13

---

Under review as a conference paper at ICLR 2020
S. Rebuffi, A. Vedaldi, and H. Bilen. Efficient parametrization of multi-domain deep neural net- works. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems. 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:/806.004: 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning, 2019.
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transac- tions on Pattern Analysis and Machine Intelligence. 2018.
Paul K Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya Tolstikhin. Prac- tical and consistent estimation of f-divergences. In Advances in Neural Information Processing Systems, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. In Advances in Neural Information Processing Systems, 2018
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
Siamak F Shahandashti. Electoral systems used around the world. Real-world electronic voting: Design, analysis and deployment, 2016.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off- the-shelf: An astounding baseline for recognition. In CVPR Workshops, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, 2019.
Eu Wern Teh and Graham W Taylor. Metric learning for patch classification in digital pathology. In Medical Imaging with Deep Learning, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. International Conference on Learning Representations, 2018.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Pierre-Antoing Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning learn from few examples. arXiv preprint 153::1903300099 2019.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint 1rXiv:1812.05063 2018.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 1907. 13625, 2019.
14

---

Under review as a conference paper at ICLR 2020
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations, 2018.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputiing 2018.
Zhe Wang. Yanxin Yin, Jianping Shi, Wei Fang. Hongsheng Li, and Xiaogang Wang. Zoom-in- net: Deep mining lesions for diabetic retinopathy detection. In Medical Image Computing and Computer Assisted Intervention, 2017.
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neural automl. In Advances Neural Information Processing Systems. 2018.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In JEEE Conference Computer Vision and Pattern Recognition, 2010.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems. 2014.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. SÅL: SuperSuuee Semi- Supervised Learning. IEEE International Conference on Computer Vision, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Con- ference on Computer Vision, 2016.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. Interna- fional Conference on Learning Representations, 2017.
15

---

Under review as a conference paper at ICLR 2020
A
TASKS
Category
Dataset
Train size
Classes
Reference
Natural
Caltech101
3,060
102
Li et al.
2006
Natural
CIFAR-100
60,000
100
Krizhesky 2009)
Natural
DTD
3,760
47
Cimpoi et al.
(2014)
Natural
Flowers102
2,040
102
Nilsback
Zisseraan
2008
Natural
Pets
3,680
37
Parkhi et al.
(2012)
Natural
Sun397
87,003
397
Xıao et al.
(0TT0)
Natural
SVHN
73,257
10
Netzer et al.
201
Specialized
EuroSAT
21,600
10
Helber al.
(2019)
Specialized
Resisc45
25,200
45
Cheng ct al. 2017
Specialized
Patch Camelyon
294,912
2
Veeling al.
(2018
Specialized
Retinopathy
46,032
5
Kaggle & EyePacs
2015
Structured
Clevr/count
70,000
8
Johnson et al.
20017
Structured
Clevr/distance
70,000
6
Johnson et al.
2017
Structured
dSprites/location
663,552
16
Matthey al.
2017
Structured
dSprites/orientttion
663,552
Matthey ct
(2017
Structured
SmalINOR
36,450
18
LeCun al.
(2004)
Structured
SmallNORRBeelevtiinn
36,450
9
LeCun et al.
(2004)
Structured
DMLab
88,178
6
Beattie al.
(2116
Structured
KITTI/distance
5,711
4
Geiger et al.
2013)
Table Description of the datasets used for the tasks in VTAB.
Table 2
provides statistics and references for all of the tasks in VTAB.
We now provide brief description of each task.
1altech101 Li al. 2006) The task consists in classifying pictures of objects (101 classes plus background clutter class), including animals, airplanes, chairs, or scissors. The image size varies, but it typically ranges from 200-300 pixels per edge.
CIFAR-100 (Krizhevsky 2009) The task consists in classifying natural images (100 classes, with 600 training images each). Some examples include apples, bottles, dinosaurs, and bicycles. The image size 32x32.
DTD (Cimpoi et al. 2014) The task consists in classifying images of textural patterns (47 classes, with 120 training images each). Some of the textures are banded, bubbly, meshed, lined, or porous. The image size ranges between 300x300 and 640x640 pixels.
Flowers102 (Nilsback & Zisserman, 2008) The task consists in classifying images of flowers present in the UK (102 classes, with between 40 and 248 training images per class). Aza- lea, Californian Poppy, Sunflower, or Petunia are some examples. Each image dimension has at least 500 pixels.
Pets PPakki et al.) 2012) The task consists in classifying pictures cat and dog breeds (37 classes with around 200 images each), including Persian cat, Chihuahua dog, English Setter dog, or Bengal cat. Images dimensions are typically 200 pixels or larger.
Sun397
(Xiao al. 2010) The Sun397 task is a scenery benchmark with 397 classes and, least, 100 images per class. Classes have hierarchy structure, and include cathedral, staircase, shelter, river, or archipelago. The images are (colour) 200x200 pixels or larger.
SVHN Netzer al. 2011) This task consists in classifying images of Google's street-view house numbers (10 classes, with more than 1000 training images each). The image size is 32x32 pixels.
EuroSAT (Helber et al. 2019) The task consists in classifying Sentinel-2 satellite images into 10 different types of land use (Residential, Industrial, River, Highway, etc). The spatial reso- lution corresponds to 10 meters per pixel, and the image size is 64x64 pixels.
16

---

Under review as a conference paper at ICLR 2020
Resise45
(Cheng al. 2017) The Remote Sensing Image Scene Classification (RESISC) dataset is a scene classification task from remote sensing images. There are 45 classes, containing 700 images each, including tennis court, ship. island, lake, parking lot, sparse residential, or stadium. The image size is RGB 256x256 pixels.
Patch Camelyon (Veeling et al. 2018) The Patch Camelyon dataset contains 327,680 images of histopathologic scans of lymph node sections. The classification task consists in predicting the presence metastatic tissue in given image (i.c., two classes). All images are 96x96 pixels.
Retinopathy
(Kaggle & HyePacs 2015) The Diabetic Retinopathy dataset consists of image-label pairs with high-resolution retina images, and labels that indicate the presence of Diabetic Retinopahy (DR) in a 0-4 scale (No DR, Mild, Moderate, Severe, Proliferative DR).
Clevr/count (Johnson et al. 2017) CLEVR is visual question and answer dataset designed to evaluate algorithmic visual reasoning. We use just the images from this dataset, and create a synthetic task by setting the label equal to the number of objects in the images.
Clevr/distance (Johnson et al., (017) Another synthetic task we create from CLEVR consists of predicting the depth of the closest object in the image from the camera. The depths bucketed into size bins.
dSprites/location
(Matthey et al. 2017) The dSprites dataset was originally designed to asses dis- entanglement properties unsupervised learning algorithms. In particular, each image is a 2D shape where six factors are controlled: color, shape, scale, rotation, and (x.y) center coordinates. Images have 64x64 black-and-white pixels. This task consists in predicting the (horizontal) coordinate of the object. The locations are bucketed into 16 bins.
dSprites/orientation
(Matthey et al. 2017) We create another task from dSprites consists in pre- dicting the orientation of each object, bucketed into 16 bins.
SmallNORRB/zimuth (LeCun al.) (004) The Small NORB dataset contains images of 3D-toys from 50 classes, including animals, human figures, airplanes, trucks, and cars. The image size is 640x480 pixels. In this case, we define labels depending on the azimuth (angle of horizontal deviation), in intervals of 20 degrees (18 classes).
SmallNORBBellvation (LeCun et al., 2004) Another synthetic task we create from Small NORB consists in predicting the elevation in the image. There 9 classes, corresponding to degrees.
different elevations ranging from 30 to 70 degrees, in intervals
DMLab
(Beattie et al. 2016) The DMLab (DeepMind Lab) suite set of control environments focused on 3D navigation and puzzle-solving tasks. We collected a classification dataset by running some episodes using pre-defined policy that collects objects by moving to them. The objects in the environment are grouped into two classes. We record the frame (agent's view) at each state, and the time (number of steps) until the agent reaches the next object. The label space is the cross prosuct of this object's 's group and the time taken to reach the next it (bucketed into three groups). This task amounts to predicting the identity and distance of an object in 3D environment.
KITTI-Dist (Geiger et al.) 2013) The KITTI task consists in predicting the (binned) depth to the vehicle (car, van, or truck) in the image. There are bins classes.
17

---

Under review as a conference paper at ICLR 2020
B HUMAN EVALUATION
We define P(T) informally as "All tasks that a human can solve using visual input alone". In- tuitively, the tasks in Appendix A satisfy this property because the task semantics involve simple visual concepts (for human), and the images contain recognisable objects either in natural or artificial environment. However, we also check empirically that humans can solve the types of tasks using in VTAB from examples alone. We therefore evaluate human raters on a representative subset of the tasks used VTAB: Pets (natural images, fine-graines object labels), DTD (natural, textures), Camelyon (specialized, medical images), EuroSAT (specialized, aerial images), DMLLa (structured, distance prediction), and Clevr-count (structured, object counting).
For each human-evaluacee dataset, raters are given 20 random examples for each class, taken from the training split. Raters are asked to classify between 50 and 100 images each, for a total of IK images (except for DMLab: 534 images), randomly taken from the test split, based on the provided training examples. Beside the examples, no hints nor explanations are given on the nature of the tasks. The raters have to deduce which properties of each image should be used to classify those, for example: the breed of the animal, the distance between the camera and the objects, or the number of objects. The raters asked not exchange the tasks, each rater produces independent work.
All datasets rated by humans are the same as the one rated by the models, except for DMLab, where we only assess the distance prediction aspect and not object type. This is because there are many object types for the raters to learn the two groups of objects from 20 examples per class. Therefore, asses cestance-predictti contains many object types, and differ only in object distance.
alone. The human DMLab task contains only 3 classes, each of which
Note that this evaluation is not meant quantify the relative performances of humans and machines, since protocol differences e.g. number of training examples, render the performance incomparable. Instead, is meant as a verification that the kinds of domains (natural images, aerial imagery, etc.) and semantics (object type classification, localization, counting, etc.) are possible to learn from examples alone using human-level visual representations.
Random Guess
Human
Pets
2.7%
63.1%
DTD
2.1%
64.0%
Camelyon
50%
68.0%
EuroSAT
10%
86.5%
DMLab
33.3%
49.0%
Clevr-count 12.5%
99.0%
Table 3: Human evaluation scores, measured using mean-per-class accuracy.
We measure human performance using mean-per-class accuracy. This is because the human training sets are class-balanced, so the raters cannot learn the class priors which algorithms that see an i.i.d sample of the training set could. Table 3 shows the results. The results indicate that some tasks are harder due more subtle distinctions (Camelyon) or noise (DMLab), and some very easy (Clevr- count). However, in all cases the raters perform significantly better than random guessing. This demonstrates that the types tasks used VTAB are ones for which human-like visual representa- tions useful to solve them, using few labels and visual-ing input alone.
18

---

Under review as a conference paper at ICLR 2020
C NETWORK ARCHITECTURES
All of the methods (except for BigGAN models) we evaluate in the paper use the standard ResNet50- v2 (He al. 2016) architecture.
This architecture consists of 50 convolutional layers. The representation produced by this model has 2048 dimensions. Exact details of this architecture be found in (He et al. 2016).
In the Cond-BigGAN and NAN models we use publicly available implementation by Lucic et al.) (2199) and (Chen al.] 2019) of the custom ResNet-like architecture proposed and described in (Brock et al. 2019). It has 1536 dimensions in the final representation layer.
We specialized procedure for evaluating patch-based models (Relative Patch Location (Doer- sch et al. 2015) and Jigsaw (Noroozi & Favaro (016)). These models use ResNet50 model with the overall stride reduced from 64 16 (by substituting the first and the last strided convolution of the standard ResNet50 model by convolution with stride one). During the adaptation phase, we apply ResNet50 with reduced stride the following way (assuming that input image size 224 224):
Perform central crop of size 192 192.
Cut image into 3 3 patches of size 64 x 64.
Apply the ResNet50 model independently to every patch.
Output the final representation and dlement-wise average of 9 individual patch representa- tions.
arahttecbures/rennn_..igggggg
19

---

Under review as a conference paper at ICLR 2020
D ALTERNATIVE METRICS
100
Top:1 Accuracy,
Megn-per-caas Accuracy,
Quadratic Kappa
90
70
50
40
WAE-UKC
WAE WAE-GAN GAN
Cond-BigGA0
WAE MMD
Uncond BigGAN
From-Scratch Scra-chhhhh- Jigsaw
Re RelPat.t OC
Exem ar
Rotation
Semi-Rtttttion 10%
10%
Sue- 100%
Sup Sup Exemp ar 100%
100%
WAE-UKL
WAE WAE-GAN GAN-GAN-
Cond-Big Aidd--iGAAN
VAE
Uncond -Bi---AAANN
From-Scratoo Jigsaw
Exemp ar
SemilEx Exempla 10%
Sup- 100%
Exemplam 100%
Sup-Rotati on- 100%
Cond Bidd-BiGAN WAE-UK-
WAE WAE-GAN-
WAE MMD.
VAE
Uncond BigGAN
From-Scratch Scr--Srrtch-
Jigsaw
Rel Lal.att PattLoc oc
Exemp ar
Rotation
Pottttion-n-e 10%
Exemplor 10%
Sue- 100%
Sup Sup-Exen Exempla ar 100%
Sup-Rotations 100%
Rel. Pal.Pot..c
Rote
Semi-RRtta Rami-Rotationn
Semi-Rotation
Figure 7: Ranking of the methods using the average scores across datasets (validation split) using three different metrics: top-1 accuracy (left), mean-per-class accuracy (center), Cohen's quadratic kappa (right). The methods on the x-axis are sorted according the highest scores according to each metric. Although there are some minor changes in the ranking between top-1 and Cohen's 's quadratic kappa, the overall performance of groups of methods remains unchanged.
Figure 7 shows different test metric scores (top-1 accuracy, mean-per-class accuracy, and Cohen's quadratic kappa), averaged across all datasets in the benchmark, for the different methods studied. The methods were ranked in the x-axis according to their score in each metric. Observe that top- and mean- meen-per-elass accuracy give exactly the same ranking. There are subtle differences when using Cohen's quadratic kappa, but the overall picture remains unchanged: supervised methods outperform semi-supervised methods by a small margin, followed by self-supervised methods, etc. Kendall's ranking correlation scores, with respect top-1 accuracy, are: 1.0 for mean per-class accuracy, and 0,97 for Cohen's quadratic kappa. This confirms that our conclusions hold even if we not using the standard metrics for few datasets.
20

---

Under review as a conference paper at ICLR 2020
E ALTERNATIVE WEIGHTING AND RANKING SCHEMES FOR MODELS
Throughout, we use (unweighted) mean top-1 accuracy across all tasks to rank the different models. This assumes that the samples represent our desired task distribution Pr in an unbiased manner. However, there may be other sensible weighting schemes that are not uniform. Here we explore three alternative weighting schemes (i) assigning equal weight to every dataset there are three data sets that are used in two tasks, and the others are used in one task, (ii) assigning equal weight to each task group: NATURAL. SPECIALIZED, and STRUCTURED, and (iii) the weighting scheme introduced in Balduzzi et al. 2018).
Another issue inherent in (weighted or unweighted) mean accuracy that the mean accuracies of individual tasks vary significantly depending on the task's 's difficulty. Since maximum accuracy is bounded, this may limit the range performances, implicitly upweighting tasks with more "head- room". Therefore, we explore few simple alternative ranking strategies (see Dwork et al. (2001) for an introduction raking aggregation methods):
(i) Ranking according to the geometric mean.
(ii) The average rank obtained by ranking the models for each task according to accuracy and then computing the average rank across tasks.
(iii) A noise-perturbed version of (i) in which the accuracies are perturbed by Gaussian noise with variance 1.0 prior to ranking.
(iv) A robust variant of the average rank, where, prior to averaging ranks across tasks, the accuracy binned into buckets of size 1% and all models in the same bucket obtain the same rank.
(v) An elimination ranking scheme equivalent to the "Exhaustive Ballot" voting system, see e.g. (Shahandashtil 2016).
We measure the agreement between these ranking strategies using the Kendall rank correlation efficient (Kendall 1945). To account for the training/evaluation stochasticity in VTAB, we sample (independently for every pair of model and task) one out of the three test-set repetitions, and compute the rank correlation between ranking according to the mean accuracy and each alternative ranking. We average over 100 such samples. The results are in Table 4 The mean rank correlation between the ranking according to the mean across tasks correlates very well with alternative ranking schemes. In particular, weighted means for 1000 samples and the full dataset as well as the geometrie mean have a rank correlation with the ranking according to the mean that exceeds 0.93. The agreement with different types of average rank somewhat lower for 1000 samples, but still considerable. We hence conclude that the mean accuracy across tasks is a fairly robust metric to rank models in VTAB.
Ranking Method
1000 Samples
Full Dataset
Reweighted data sets
0.979 + 0.017
0.977 + 0.018
Reweighted groups
0.988 + 0.013
0.992 + 0.011
Balduzzi al. (018)) weighting
0.925 = 0.022
0.945 + 0.023
Geometric mean
0.963 + 0.022
0.969 + 0.020
Average rank
0.922 = 0.022
0.951 + 0.017
Average rank (perturbed)
0.912 + 0.028
0.931 0.031
Average rank (robust)
0.926 + 0.021
0.971 + 0.020
Elimination rank
0.913 + 0.029
0.935 0.020
Table 4: Kendall rank correlation coefficient (Kendall 1945) (with standard deviation) measuring
the agreement of the ranking of models according to the mean accuracy across tasks with different types of weighted means, the geometric mean, and different types of average ranks. Agreement of ranking according to mean accuracy with alternative ranking schemes is high.
21

---

Under review as a conference paper at ICLR 2020
F REPRESENTATIVE SUBSET OF TASKS
We explore another direction related to ranking of models: Which subset of tasks is most represen- tative of the performance of the entire benchmark? Such a subset allows for cheap iteration which is beneficial during early development of new models.
We search the most representative tasks from the full of 20 tasks by performing exhaustive search over all (2)) subsets. For each subset compute the mean accuracy and compute the Kendall rank correlation coefficient between the resulting ranking of models and the ranking ac- cording the mean over all tasks (averaged over 10 trials sampled described in Appendix E.. Table 5 shows the subsets which produce the highest rank correlation. There are many subsets which lead an excellent agreement with the ranking according to the full mean. These subsets can be quite different provided that they are sufficiently diverse.
To assess how well we can expect this approach generalize unseen models, we perform the fol- lowing experiment. For each pair of models, we perform subset selection based on the accuracies of the remaining models, and check whether the ranking of the left out pair based on the mean across all data sets agrees with the ranking of the pair according to the mean over the subset. The mean outcome of this binary test across all pairs is an estimate of the probability for correctly ranking an unseen pair of models: 0.940 for the full data sets and 0.972 for 1000 examples. The subset selection hence generalizes to an unseen pair of models with high probability.
The representative subsets in Table 5 may be used for rapid prototyping of new methods before running the full VTAB. However, a caveat to the above analyses is that we evaluate the ranking of very diverse set of models, from those whose performance is worse than from-scratch training performance combinations of self-supervision and supervision. Therefore, the ranking is reasonably robust. To make fine-grained distinctions between models of similar class, the representative subsets above may be less reliable. Repeated iteration on just a few tasks exposes one more to the risk of meta-overfitting
Dataset
Rank Correlation
Subset
0.983 0.017
Caltech101, EuroSAT, Pets. Camelyon, Resisc45
0.980 + 0.012
DTD, EUroSAT, Flowers 102, Pets, SVHN Caltech101, DMLLab, EurSSAT, Flowers 102, Camelyon Caltech101, EuroSAT, Flowers 102, Pets, Camelyon 100, Clevr-Count, dSpr-Orient, Flowers Pets
1000
0.978 0.013
0.978 + 0.011
0.975 = 0.013
0.987 0.012
Clevr-Dist, DTD, Camelyon, Sun397, SVHN
0.987 + 0.012
100, DMLab, dSpr-Loc, Flowers 102, SVHN Clevr-Dist, Clevr-Count, Retinopathy, DTD, Flowers 102 Retinopathy, dSpr-Loc, Flowers sNORB-Azin SVHN Clevr-Dist, DTD, EUroS Camelyon, Sun397
full
0.983 + 0.020
0.983 0.017
0.982 + 0.016
Table 5: Task subsets of size 5 that lead to the largest rank correlation with the mean accuracy across all tasks. There are many different subsets that lead to an high rank correlation with the full mean.
22

---

Under review as a conference paper at ICLR 2020
G LIGHTWEIGHT EXPERIMENTS
Caltech1ol
CIFAR 100
Flowest02
Camelyon
BBrSAT
Rosise45
Retinopathy
Clev--Count
Cleve-isst
DM-Lab
KITT--ist
dSpr.Lo
dSpr-On
SNORB-AAAE Azimm
SNORD Ble
DTD
Pets
SVHN
Stm.97
Mean
WAE-GAN
29.9
7.8
6.5
12.0
7.5
42.4
3.2
69.0
59.1
14.4
73.4
32.4
55.1
25.0
54.7
70.1
17.8
27.0
25.4
33.3
WAE-UKL
29.3
8.2
7.2
13.1
7.8
36.6
3.3
69.8
61.2
16.4
73.6
32
55.2
22.4
50.9
72.9
23.0
27.5
28.1
33.6
VAE
34.3
9.9
10.3
18.2
94
49.3
4.5
72.6
73.1
23.5
67.7
36.8
61.2
26.7
42.9
89.9
23.0
35.6
37.5
38.2
WAE-MMD
35.5
10.7
8.0
18.4
9.3
58.7
4.9
73.8
73.1
22.4
73.2
34.4
58.1
26.3
55.4
86.9
26.3
31.6
28.1
38.7
Cond BigGAN
30.6
73.6
From-Scratch
20.6
5.0
55.2
37.6
Un Un.C-BigGAN
58.7
15.2
34,1
18.7
65.4
9.4
76.3
74.2
35.7
66.1
44.7
55.8
28.7
37.2
76.8
48.4
33,9
40.3
45.8
1080
Jigsaw
66.7
18.9
51.4
66.1
37.5
55.1
12.1
76.0
91.5
66.2
72.4
42.8
55.9
30.5
68.2
69.5
35.0
44.9
36.3
52.5
Rol.Pat.Loc.
68.5 19.1
52.2
69.0
41.3
60.9
11.1
77.5
92.6
65.4
70.7
43.5
59.6
33.6
68.2
70.7
29.3
47.2
35.2
53.5
Exemplar
69.6 11.0
48.9
69.9
42.3
87.1
13.7
79.1
94.7
67.3
72.5
54.8
60.9
36.7
75.5
91.8
44.3
42.1
40.3
58.0
Rotation
77.6
27.3
51.4
68.8
48.3
89.4
12.9
80.4
93.5
69.4
70.9
45.6
60.9
40.4
75.9
89.6
49.5
52.2
44.3
60.4
Semi-Rot- 10%
88.1 48.1
64.3
87.6
84.6
87.2
27.0
81.4
95.6
79.5
73.0
40.9
55.3
37.9
57.4
87.5
53.5
$1.9
35.7
65.
Semi Ex 10%
88.6
53.2
60.8
86.8
85.3
88.0
29.0
83.2
95.2
77.3
71.7
42.3
57.4
36.7
71.4
74,9
53.9
52.7
32.3
65.3
Sup-100%
91.0 57.0
66.0
88.6
89.9
87.3
34.4
80.6
95.3
80.8
73.2
41.0
56.1
36.3
70.6
85.7
46.0
45.7
35.4
66.4
Sup-Ex-100P
77.7
Sup-Rot-100%
91.7
69.5
90.8
88.1
88.5
32.8
83.4
96.0
82.0
71.1
47.3
57.2
36.6
77.1
88.3
52.1
51.6
33.7
68.0
78.1
73.6
55.1
WAE-GAN
42.0
62.3
49.9
52.2
Cond-RigGAN
0.1 56.3
44.8
68.8
31.6
91.4
44,9
81.3
94.5
76.5
75.3
12.4
24.5
51.4
49.7
6.2
7.4
80.6
79.2
51.4
WAE-MMD
50.8
38.8
11.0
20.8
16.2
90.9
31.6
80.6
94.1
64.8
73.8
98.1
89.3
52.6
61.6
100.0
90.2
96.3
72.4
64.9
VAE
48.4
44.2
16.0
18.4
14,0
93.1
29.3
81.3
92.5
65.0
74.2
98.4
90.1
59.7
57.0
100.0
94,7
97.9
95.6
66.8
Un.C. BigGAN
73.6 58.1
44.9
63.5
30.9
93.0
46.9
82.2
89.8
75.4
75.9
47.6
54.9
54.8
57.4
86.1
95.9
88.1
76.6
68.2
From-Scratch
55.9 64.4
31.3
50.6
23.8
96.3
52.7
81.2
96.2
86.8
76.8
99.7
89.4
71.5
68.4
100.0
96.3
99.9
91.7
75.4
Jigsaw
79.1 65.3
63.9
77.9
65.4
93.9
59.2
83.0
97.9
92.0
80.1
99.6
88.6
72.0
74.7
100.0
90.3
99.9
93.6
83.0
R.P.L
79.9 65.7
65.2
78.8
66.8
93.7
58.0
85.3
97.8
91.5
79.8
99.5
87.7
71.5
75.0
100.0
90.4
99.7
92.6
Exemplar
70.7
79.3
79.0
99.8
100.0
99.9
97.4
Rotation
73.6
Scmi-Rat-10%
88.1
82.4
72.4
87.9
66.7
98.7
81.0
96.5
97.5
88.6
Semi 100
85.3
82.7
70.5
92.2
89.0
97.0
67.4
86.0
98.6
94.7
78.8
99.8
93.1
76.8
81.5
100.0
96.5
100.0
97.8
88.8
Sup-100%
94.1
83.8
74.0
93.2
91.9
97.0
70.7
83.9
98.8
95.3
79.3
99.8
92.1
76.4
80.7
100.0
96.4
99.8
97.7
89.7
Sup-Ex-100%
94.4
84.1
74.5
93.4
91.8
97.1
69.4
86.7
98.6
95.
79.5
99.8
92.7
76.8
84.0
100.0
96,4
99.8
98.0
90.1
Sup. Rot. 100%
94.6
84.8
75.9
94.7
91.5
97.0
70.2
85.9
98.8
94.9
79.5
99.8
92.5
76.5
82.3
100.0
96.5 100.0
98.4
90.2
Table 6: Top-1 accuracy of all the models evaluated on VTAB in lightweight mode. Each entry represents the median score of three runs with different random seeds evaluated on the test set. Within each dataset-size group (1000-example and full), the methods are sorted from best to worst according to their mean accuracy.
23

---

Under review as a conference paper at ICLR 2020
H HEAVYWEIGHT EXPERIMENTS
Table 7 shows the full heavyweight results. Table shows recently reported results for each dataset in the literature (where available). The literature results may use substantially more complex chitectures, or additional task-specific logic. For example, the Retinopathy result uses three neural networks combined with decision trees, and uses additional information, such as combining the images from two eyes of the same patient.
SNORD Azim
Callech101
CIFAR-11 100
DTD
Flowers12
Pets
SVHN
Sun397
Camelyoo
EaroSAT
Resiso45
Retinopathy
Cleve-Cou Count
Clev-Dist
DMLab
KITTI-Dist
dSpr-On
.NORD Elee
Mean
From--Sratch
60.2
24.5
51.1
73.4
43.1
88.3
14.9
82.1
93.3
63.9
75.4
53.0
57.3
36.6
55.7
88.6
67.6
52.2
81.6
61.2
Sup-100%
89.7
52.7
65.4
94.2
90.1
91.0
35.5
87.3
96.0
84.0
78.0
59.0
63.0
44.4
80.1
89.3
70.7
57.5
54.7
72.8
Sup-RRtt Rot. 100%
100%
88.0
48.9
68.1
94.7
89.0
91.8
33.7
86.2
96.6
85.2
78.5
66.5
61.5
45.5
77.2
88.8
71.8
59.4
52.0
72.8
Sop-Ex-100sS
90.4
50.4
66.1
94.3
90.6
91.3
35.2
86.7
95.8
83.3
78.4
73.8
62.4
44.1
79.7
87.5
71.1
62.8
54.8
73.6
From Scrut-Scratch
74.5
77.8
67.1
85.8
70.9
97.0
70.1
91.2
98.8
94.3
82.8
99.8
96.7
76.6
68.4
100.0
96.7
99.9
94.0
86.4
Sup-100%
93.0
83.4
73.7
91.3
92.7
97.5
75.6
87.3
98.8
96.1
83.4
100.0
97.0
78.8
81.0
100.0
96.8
100.0
98.5
91.1
Sup Rot 100%
93.5
84.0
76.8
97.4
92.6
97.4
75.6
86.5
99.1
96.3
83.7
100.0
96.8
79.6
82.4
100.0
96.8
100.0
96.7
91.3
Sup-Es Exp-Ex- 100%
93.8
83.1
76.5
97.8
92.9
97.5
75.3
86.5
99.0
96.3
83.7
99.9
96.8
79.3
82.8
100.0
96.7
100.0
99.1
91.4
Table 7: Top-1 accuracy of all the models evaluated on VTAB in heavyweight mode.
SUP-EX.-1009
Result
Reference
Caltech 101
90.4 / 95.1*
86.9 / 95.1*
Cubuk et al.
2019)
Kornblith et al. 2019
CIFAR-100 100
83.1
91.7
Tan Le
2019
DTD
76.5
78.1
Kornblith et al.
2019
Flowers
97.8
98.8
Tan & Le (20T
Pets
92.9
95.9
Huang et al.
201
SVHN
97.5
99.0
Cubuk et al.
2019
Sun397
75.3
72.0
Wang et al.
(2017)
Camelyon
86.5
90.6
Ich & Taylor
(201
AT
99.0
96.4
Helber al.
2019
Resisc45
96.3
90.4
Cheng et al.
2017
Retinopathy
74.71
85.41
Wang et al.
(2017)
Mean per-class accuracy
Quadratic Kappa
Table 8: Comparison of the best method the full datasets using heavyweight sweep (SUP- EXEMPLAR- ( to results published the literature (where available). For some datasets prior work does not use top-1 accuracy, therefore, we present our the performance of SUP-EXEMPLAR- 100% using the same metric.
24

---

Under review as a conference paper at ICLR 2020
I HYPERPARAMETER SWEEPS
Lightweight sweep The lightweight mode performs a restricted hyperparameter search for all tasks, permitting fair comparison with few resources. It uses fixed values for most hyperparame- ters. We set the batch size to 512 and use SGD with momentum of 0.9. When fine-tuning, we do use weight decay, and when training from scratch we set 0.001. We resize all images to 224 x 224, except for generative models, where we resize to 128 x 128 since training at higher resolution is currently very challenging.
Lightweight mode sweeps four hyperparameters:
Learning rate: {0.1, 0.013
Traning schedule: In all cases we decay the learning rate by factor of 10 after and 3 of the training time, and one more time shortly before the end. We train for (2500, 000) training steps (i.e. model updates).
Note that when training on 1000 examples, we perform model selection using the regular validation set for each task. This setting is somewhat unrealistic, since in practice would train the larger validation set. However, we use this setting, which is common in prior art (Oliver et al. (0188), because significantly reduces evaluation noise. Further, Zhai et (2019) show that using small validation yields the same conclusions.
Heavyweight sweep We define a relatively large search-space over relevant hyperparameters for the fine-tuning adaptation phase and perform an individual random search for each downstream task's 's validation that space. The random search consists of 100 independent trials. Specifically,
we search:
Batch-size: {128,
Training schedule: In all cases we decay the learning rate by a factor of 10 after 3/ and & of the training time, and one more time shortly before the end. We train for either
-any of {100, 200, 300, 400, 500} epochs over dataset, or
- -any of {2 500, 5 000, 000, 20 000, 40 000} training steps (i.e. model updates).
Preprocessing: during training, optionally include random horizontal flipping.
Preprocessing: during training, optionally include random color distortions.
Preprocessing: during training, use any of the following techniques for resizing:
res. : resize the image to 224;
res. crop: resize the image to 256 and take a random crop of size 224;
res. sma crop: resize the image keeping its aspect ratio such that the smaller side is 256, then take random crop of size 224;
inn. crop: "inception crop" from [Szeedy et al. 2015):
-cif crop: resize the image to 224, zero-pad by 28 on each side, then take random crop of size 224.
Preprocessing: for evaluation, either
resize the image one of (224, 320, 384), or
resize it such that the smaller side is of size (256, 448) and then take a central crop of size {224, 320, 384).
Weight decay: we sample log-uniform randomly [10-5, 5,10-1).
Optimizer: we randomly choose between
SGD with learning-rate chosen log-uning log-uniform randomly in [0.001, 0.0],
Adam with learning-rate chosen log-uniform randomly in [10-5, 10-2).
We then choose the set of hyperparameters with the best performance at the end of training for each task individually.
For both training from scratch, and from a pre-trained supervised ImageNet training, we present the best set of hyperparameters for each task in Figs. g to 10 as red dot. The green background
25

---

Under review as a conference paper at ICLR 2020
shows the distribution of hyperparameters which performed within 2% of the best one for each task, providing sense for the importance of tuning that hyperparameter.
In Figs. 12 and 13 show the achieved performance for the full distribution of hyperparameters as violin plots for the CIFAR-100 and DMLab downstream tasks, both for the small and full variants. The supervised pre-training not only achieves better accuracy than training from scratch, but also displays much smaller variance across hyperparameter values.
Preprocessing
Weight Learningrate Decay
Batch
Schedule
FlipCol. Train
Eval
SGD Adam
Caltech101
Cifar100
DTD
Flowers102
Pets
SVHN
Sun397
Camelyon
EuroSAT
Resisc45
Retinopathy
Clevr-Count
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc
dSpr-Orient
SNORB-AZim
sNORB-Elev
Sum
128
256
512
100 epochs
200 epochs
300 epochs
400 epochs
500 epochs
2500 steps
5000 steps
10000 steps
20000 steps
steps
none
raw horizontal
distort
res. res [crop res. 1i-smma ccrop- Inc. crop
cif.crop
res 225.42224
res @ 920
384
res @ 384
cen www.crop@222
cen 0oop0000
320
cen ssn.crop@3ee
1e-5
1e-4
1e:3
le-2
1e-1
0.001
0.01
0.1
1.0
1e:5
1e-4
1e-3
1e-2
40000
Figure 8: Training from scratch with only 1000 examples. in red and those within 2% in green to show sensitivity.
Best hyperparameter values are marked
Preprocessing Train
Weight Learningrate Decay
Batch
Schedule
FlipCol.
Eval
SGD
Adam
Caltech101
Cifar100
DTD
Flowers102
Pets
SVHN
Sun397
Camelyon
EuroSAT
Resisc45
Retinopathy
Clevr-Count
Clevr-Dist
DMLab
KITTI-Dist
d5pr-Loc
dSpr-Orient
SNORB-Azim
SNORB-Elev
Sum
128
256
512
100 spochs
300 epochs
400 epechs
epochs-
2500 steps
10000 steps
20000 steps
40000 steps
raw horizontal
distort-
res.
res. crop
res sma crop-
Inc. crop
cif.c crop
res -2224
res -30
re5.0-384
cen crop@22p2244
cen eopprpp000
cen. 384
1e-5
1e-3
1e-2
0.001
01
0.1
1.0
le-5
1e-4
1e-3
1e-2
5000
200
Figure 9: Training from scratch on the full datasets. Best hyperparameter values are marked in red
and those within 2% in green to show sensitivity.
26

---

Under review as a conference paper at ICLR 2020
Preprocessing Train
Weight Decay
Learningrate
Batch
Schedule
FlipCol.
Eval
SGD Adam
Caltech101
Cifar100
DTD
Flowers102
SVHN
Sun397
Camelyon
EuroSAT
Retinopathy
Clevr-Count-
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc-
dSpr-Orient-
SNORB-AZim
SNORB-Elev
Sum
128
256
512
100 epochs
200 epochs
300 epoc
400 epochs
epochs
2500 steps
10000 steps
20000 steps
40000 steps
none
raw horizontal
distort
res, res.
res. res.smalcron crop Inc. crop
crop
res T25.0222
res @2000320
res 0s5.0388
320
cen Crocrre
cen cen.crop@:322
1e-5
1e-3
1e-2
le-1
0.001
0.01
0.1
1.0
1e-5
1e-4
1e-3
1e-2
5000
Figure 10: ImageNet supervised representation fine-tuned on only 1000 examples. Best hyperpa-
rameter values are marked red and those within 2% in green to show sensitivity.
Preprocessing Train
Weight Learningrate Decay
Batch
Schedule
FlipCol.
Eval
SGD Adam
Caltech101
Cifar100
DTD
Flowers102
SVHN
Sun397
Camelyon
EuroSAT
Resisc45
Retinopathy
Clevr-Coume
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc
dSpr-Orient
SNORB-AzIm
SNORB-Elev
Sum
128
256
512
100 epochs-
200 epochs
300 epochs
400 epochs
epochs
2500 steps
10000 steps-
20000 steps
40000 steps-
horizontal
raw
distort
res.
res crop
s.ssmaccrop
Inc. crop
cif.crop
res 724
re3.0320
res.0384
cen @on.crop:::22
320
cen.cro Croneep
384
1e-5
1e-3
1e-2
1e-1
0.001
0.01
0.1
1.0
le-5
1e-4
1e-3
5000
Figure 11: ImageNet supervised representation fine-tum iue-tuned on the full datasets. Best hyperparameter
values marked in red and those within 2% in green show sensitivity.
27

---

Under review as a conference paper at ICLR 2020
ScrotchSCIFAR-10
1.0
1k
6.8-
f.I1
full
full
6.8
aparsay
0.0
THC Srop
69244
can cropprant
con crop cropiptco choptpate
128
25b
512
crop Brain
3.0
eycuumey
ercuurscyy
0.4-
6.2
0.0
00.501. 8.11
10.00 0.11
(8.1 18)
(tests, comn
10,9901. 0.001)
(ea01, det)
(lees, esot!
8e.001.. 0.001
eeee1.0 dati
(0.02 9.11
200
70
500
4060
schedule stopo
Adam
Sup 100% 100
1.0
ful
full
0.24
0.0
yes
TIE Strop
smeicmo
Crop
9V24
rnd 8K200
Bules oen crongga34
den croug 32D
cen cropp 1814
130
359
632
distort train
emp
0.8
0.6-
0.4
18 00001.a0 0.01
dees 0.6001
0.6011
(8.011 0.01)
De-es, 66001
te 0d01, o 2.001
cel. 0.011
(8 0001. 0.1
400
500
5005
es..00
10.0011
schedulee spacts
schedule stage
Figure 12: Heavy parameter sweep on CIFAR-100 dataset. Top 2 rows show the results of from scratch algorithm and bottom 2 rows show the results of supervised 100% algorithm. Each plot shows the violin plot of the results with respect to given parameter. Here we show the results evaluated on full datasets and 1k datasets.
28

---

Under review as a conference paper at ICLR 2020
Scrotch@OMLab
1.0
10
6.0-
ful
ful
fut
6.8
0.0
1RC Jrop
inc
6924
can cropgrant
cen cropirtco choptpate
250
512
clop Brain
3.0
eccurcoy
0.2
0.0
(0.601. 8011
10.0 0.11
(8.1 10)
(tesss 00001
10,9901, 0.001)
(ea01, neti
(laes, es0t)
te.001. 0.01
neee1. dati
(0.0x 0.1
200
300
500
2500
5000
Ado00
schedule stops
Adam
Sup L00%
1.0
full
tull
0.2
0.0
yes
res.
mis jtrop
smelcmo
Grop
ra 9224
nec 46220
Bule
crongga3d
den cropi 320
Oen cropi 184
120
359
532
distort train
emp
0.8
0.6.
spturber
0.4-
6.-
i8 18.00..0 0.01)
ess.an
deds 0.0001
100011 0.001)
(8.011 0.01)
Oe-es, 600011'
te a. 2.001
Io cel. 0.01
(8 1001. 0.1
700
y00
500
5000
10000
200000
sctedule spocts
achedule staps
Figure 13: Heavy parameter sweep on DMLab dataset. Top rows show the results of from scratch algorithm and bottom 2 rows show the results of supervised 100% algorithm. Each box shows the violin plot of the results with respect given parameter. We show the results evaluated on full datasets and 1k datasets.
29

---

Under review as a conference paper at ICLR 2020
J LIGHTWEIGHT VERSUS HEAVYWEIGHT SEARCHES
45
From-Scratch
Sup-100%
Top-1 Accuracy Dela
40-
Top-1 Accuracy Delta
20
35
30
15.
Heavy
25
10
20
Clevn-Dis
Ratinopshhy
Camelyon-
EureSAT
Sun397-
DMlaa-
dspr
CIFAR 1-00-1100-0
KITTI Dist
Clevr-Caunt
Resio455
Catbech1o1
SNOnB-Azin
5WNN
eSpr-Ortent
DTD-
Pets
Rowers10x
SNORB-tlew
CIFAR-1 100-
Caltech001
DTD
Pats-
Sun307
Redioc45
d5pp Loc
SWMN
Ratinopatly
Floeers 202-
Camalyon-
Dlevr Dleve-Dic--
DR wwLbbb
KITTI Dist
sNORB-Azim.
SNORE Elev-
dSpr-Drient
Light
20
Sup-Rotation-100
40
Sup Exemplar-1009
Top-1 Accuragy Deta
Top-1 Accuracy Delta
15-
30
Heavy
10
20
10-
CIFAR 200
Calaeh1101
d5pri.nc
EuroSAT-
Pets
Sun397
Camelyon-
Resioc55
5VHN
Rlwwrr1002
Clevn
Ratimpathy-
SHURB-ASm Aoom-
DNLLb
Cleur Cler-Coount
diar Ortent-
-10
CIFAR-100-
DTD
DureSnT-
Centect001
Sun397-
Pats-
KIT--ist
SVAN-
Besicc4-
Cemelyon-
dspn
Flwers2222
Retinopattyy
DMLSb
Clev--Ditt
SHORB Arii-
SNOWB- Ble
(Spn-Orent
Clevr Count-
Light
Figure 14: Per-dataset, absolute difference in top-1 accuracy two hyper parameter sweep strate-
gies. Top left:
FROM-SCRATCH. Top right: SUP-100% Bottom left: StP-ROTATION-
Bottom right: SUP-EXEMPLAR- -000%% The bar colour denotes the task group as usual.
30

---

Under review as a conference paper at ICLR 2020
Sup- -or---ottiio---- 100%
Top-1 Accuracy Dell
Accurscy
50-
Sup- 100%
30
Supp- 100%
-10
Rechopertyy
CevnCount-
Camebon
dSprO/enn-
DeLL
EUrOSAT-
SUHR
KITT BIT
CFAR-100
Flowes
Caleea
From Scretch
Redimppatty
shone Elo
Surd57-
OHLab
Caltcchot
Rosiols
SWIN-
Rowers102
Camiiyo--
DTD
eip--rrant
Clev--uunt Count-
OTTI
Sup- Rotalton 00%
Accuracy Detta
Sup- 100%
Top-1 Accuracy Della
20
10
-10
From Scretch
Sup- 100%
SNORD Elev
Retengathy
SWhN-
EUrOSAT-
ston--luim-
Clean Det-
DilLa:
OTO
Howns10
KITTADIt-
Calwent PeS
Dis-
KITT Dis
soblle Bix-
Cattechll1-
Clbar Olz--
Pets
dier Lcc
floerst
EUroSAT
Resscdt
DTD
CeentCoutt
Figure 15: Absolute difference in top-1 accuracy between pairs of methods for each dataset. The bar colour denotes the task group as usual. Top: Lightweight hyperparameter sweep. Bottom: Heavyweight hyperparameter sweep. Left: SUP- 100% versus FROM-SCRATCII supervised pre- training yields substantial improvement on the NATURAL datasets and some others. Right: SUP- ROTATION- 100% versus SUP- 1 100% the additional supervueee loss yields better representa- tions for the STRUCTURED tasks.
31

---

Under review as a conference paper at ICLR 2020
K LINEAR EVALUATION
Here we describe the setup for linear evaluation. We follow (Kolesnikov et al. 2019) and evaluate the frozen representation by training a linear logistic regression model. We use exactly the same hyperparameters as described in lightweight sweep from Section I The only difference here is that only the linear layer is trained instead of fine tuning the whole network.
1.0
1000 Example Datasets
1.0
Full Datasets
Kandall Carrelation
Kendall Rank Correlation
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2-
0.0
0.0
-0.2
Roves
Sunk7-
tatropalyy
E
SKORB Ain
-0.2
GRRKTT 10
Dro-
Sind77-
Ber-
Caneyyn
Basacd-
dip Oren-
der Caunt
kimos
DNL20
dip
cie
Figure 16: Kendall rank correlation coefficient between finetuning and linear evaluation on each dataset.
SNORB- Azim
Caltech101
CIFAR- 100
Flowers102
Camelyon
-umoSAT
Rosise45
Retinopathy
Clevr-Cont
Clev--iitt
DM-Lab
KITT--ist
dSpr.Loc
dSpr-Ori
N Elev
DTD
Pets
SVHN
Sum.99
Mean
VAE
34.9
8.0
7.8
10.1
7.3
23.8
3.4
61.8
38.4
13.2
73.6
26.9
52.3
22.6
38.3
60.7
11
19.3
20.8
28.2
WAE-UKL
34.7
7.9
7.1
10.7
6.5
23.1
3.0
62.7
39.6
11.7
73.6
26.6
52.8
22.8
45.1
61.8
10.5
20.3
21.0
28.5
WAE-GAN
32.8
7.5
6.2
11.3
6.9
25.5
3.2
65.1
43.0
11.0
73.6
28.0
53.4
23.9
45.4
60.0
10.2
19.6
22.7
28.9
MMD
37.1
9.2
7.4
14.6
7.5
23.9
3.8
65.3
44.6
13.4
73.6
27.8
53.
23.8
43.0
66.3
10.1
22.9
22.2
30.0
Un.C. Un.C-BiGGAN
51.8
14.3
29.5
38.6
11.7
37.9
8.8
78.2
68.7
29.7
73.4
34.6
48.5
25.4
30.1
30.1
24.0
26.6
28.5
36.3
Jigsaw
58.6
14.4
39.0
85.9
73.5
28.4
Lot.
62.9
15.4
43.5
20.6
41.6
12.7
74.9
86.0
54.2
73.4
39.4
49.1
28.0
53.0
29.8
19.4
26,9
28.8
42.6
1000
Cond BigGAN
63.3
20.8
36.2
61.2
23.0
47.1
12.6
73.5
75.9
43.1
63.5
40.9
50.2
27.9
37.8
40.5
42.8
30,2
32.0
43.3
Exemplar
57.8 20.4
40.7
56.6
27.3
42.8
13.1
78.6
88.3
57.6
74.1
45.2
50.9
28.1
59.3
55.1
28.7
25.6
33.1
46.5
Rotation
66.0 22.8
44.1
48.4
17.4
53.9
14.4
78.3
87.2
50.3
73.7
42.0
53.5
30.9
61.9
59.7
30.3
26.5
37.8
47.3
Semi Ex 10%
82.3
40.4
55.3
75.1
82.8
48.8
27.5
77.9
89.0
64.7
68.1
35.6
39.6
28.9
65.8
29.6
38.2
26.5
26.5
52.8
Semi-Rot- 10%
81.8
38.7
56.8
76.8
81.6
40.5
27.2
79.9
91.1
70.1
66.9
39.2
39.4
28.1
62.7
45.0
31.6
21.8
27.6
53.0
Sup-Rot-
Ro0% 100%
86.8
50.8
65.4
86.3
89.5
42.6
35.7
79.3
92.3
76.9
68.9
40.5
37.4
32.1
68.5
39.7
34.0
25.5
26.7
56.8
Sup- 100%
87.7
50.9
64.7
83.5
89.3
50.7
35.7
81.5
91.6
72.9
70.5
41.0
40.0
29.0
65.0
42.6
38.4
27.1
26.8
57.3
Sup-Ex-1008
87.8 50.9
64.0
84.5
89.1
53.3
35.4
78.4
92.0
73.3
68.5
39.6
40.0
31.1
72.0
42.1
39.5
27.7
27.6
57.7
VAE
42.9 16.7
8.4
12.3
9.6
24.6
7.8
69.4
45.9
20.1
73.6
30.7
57.3
30.3
41.0
79.7
13.6
20.1
25.9
33.1
17.3
7.8
9.1
67.5
49.7
73.6
31.0
59.5
13.9
21.0
26.2
WAE-GAN
8.4
73.6
13.0
29.4
WAE-MMD
43.3 19.4
8.6
17.4
9.3
26.2
9.5
71.0
55.0 23.6
73.6
33.0
62.1 31.9
50.6
85.5
13.7
27.7
36.0
Un. C. BigGAN
60.5 35.7
38.0
49.1
17.1
55,3
26.7
79.4
77.1 50.9
74,1
40.4
55.6
37.3
30.9
57.4
40.4
32.5
39.2
47.2
Jigsaw
68.7 29.9
49.3
53.4
27.7
50.6
29.2
78.2
90.6
70.2
73.6
49.7
57.2
41.7
55.2
40.4
23.0
31.2
36.6
50.3
Ret.Pat.L.o.
73.2 29.0
55.5
57.5
29.7
50.2
27.8
77.4
90.7
70.2
74,5
49.7
57.7
42.8
53.0
38.5
20.7
32.7
36.3
50.9
Exemplar
68.2 49.2
52.2
67.4
38.3
59.3
43.7
81.5
94.2
79.9
74.8
56.3
58.8
41.4
62.8
70.6
34.2
32,3
45.0
58.4
Cond BigGAN
73.6 47.9
43.5
70.1
29.4
67.7
39.4
76.3
84.6
66.9
69.7
52.1
60.4
39.7
43.7
75.5
64.9
67.3
52.7
59.2
Rotation
77.5
48.4
56.2
60.2
30.6
71.6
42.1
82.0
93.4
74.7
74.7
56.6
64.8
46.0
62.0
75.1
36.1
38.3
52.8
60.2
Semi-Rot-109g
86.7 65.6
65.5
83.6
84.6
62.2
58.8
84.0
96.0
87.6
74.6
56.8
53.1
45.9
64.9
69.7
45.7
41.8
44.1
66.9
Semi 10%
80.7
85.3
57.7
94.5
82.8
74.3
54.6
46.8
74.1
65.2
41.7
69.1
Sup-Rot-100%%
91.1
68.2
96.8
74.8
51.9
49.4
71.4
44.7
72.3
Sup- 100%
91.6
74.9
72.6
88.7
91.0
68.0
84.4
96.5
88.6
76.0
61.7
55.1
50.6
74.4
75.4
62.2
72.0
44.1
73.6
Sup-Ex 100%
92.5
74.1
72.1
89.2
91.6
73.6
67.6
84.0
96.0
87.9
75.0
62.2
57.9
51.4
74.1
75.6
62.0
73.2
46.5
74.0
Table 9: Top-1 accuracy of all the models with linear evaluation on VTAB.
32

---

Under review as a conference paper at ICLR 2020
L SCALING UP THE ARCHITECTURE
In this section, we study the problem "is scaling up the architectures helpful?". Figure 17 shows top-1 accuracy ImageNet public validation set when scaling up the architectures. Widening factor is the multiplier on the network width, where x stands for the standard ResNet architecture. Depth stands for the number of architecture layers. As expected, the model accuracy goes up with either wider or deeper architectures. Figure 18 shows the results on VTAB benchmark, where the 2x wider ResNet152 52 architecture performs consistently better than the standard ResNet50 model.
79.0
78.5
78.0
27.5-
77.0
76.5
101
152
75.5
75.0
Widening Factor
Figure 17: Top-1 accuracy on ImageNet public validation when scaling up the architectures. The accuracy goes up with either wider or deeper architecture.
8152X
81 52x2
VTA8 Accuracy
68.0
Acouract
810
R50x
B101x4
101x2
g101x1
R101x2
85.6
R50x2 R101x1
67.0
B152x1
85.4
R152x1
B50k2
R50x4
85.2
R50x1
85.0
A50x1
66.0
84.8
75.0
75.5
76.0
76.5
77.0
77.5
76.0
78.5
75.0
75.5
76.0
76.5
77.0 77.5
78.0
78.5
ImageNet Accurasy
ImageNet Accuracy
Figure 18: VTAB performance on the scaled up architectures. Here the architecture is represented by different color and width is represented by different circle size. The performance increases when switching from the standard ResNet50 architecture to ResNet152 2x wider architecture.
33

---

Under review as a conference paper at ICLR 2020
M BUDGET ANALYSIS
Figures 19 and 20 show the top-1 accuracy attained by SUP-100% and ROTATION using different
schedule lengths when using 1000 samples or the full dataset, respectively.
Natural Datesets
Specialized Datasets
Structured Datasets
500-100%
Clevn-Dist
CIFAR-100
ClevrCouutt
Flowers102
dSpr-brtent
0.4
Sviik
EureSAT
AZORD-Azm
Caltech101
Resisc45
sNOR8-Elev
DTD
Ratinapathy
DMLab
Sun397
Came'yon
KITT-Dist
1.0
Cleur-Dist
0.8
Sppebrisnt
Aspr-Loc
Roletton
SNOGB Azim
CIFAR,IñO
SNORB-Elev
Flowers102
Pets
EuroSAT
0.2
Collgch101
Resisc45
Retinopathy
Sun397
Camalyon
0.0
10a
20
schedule steps
schedulc stops
steps
Figure 19: Top-1 accuracy for each task with respect to the number of fine-tuning steps the e000-ex datasets. More steps usually better, performance stable after 1000 steps.
Natural Datasets
Specialized Datasets
Structured Datasets
0.8
100%
Clevn Dist
CIFAR-100
Clevn-Count
Flowers102
dSpr-Orient
Pets
dSpr-Loc
SNORD-Azim
0.2
Calterh101
Resisc45
SNORR.Flev
Retinapathy
Sun397
Camelyon
KITTI-DE Dist
0.0
Rototion
0.6
CIFAR-100
Clevn-Count
flowers102
JSpr-Orient
0.4
Pets
dSpr-Loc
SVHN
EuroSAT
SNORR-Azim
DTD
Retinopathy
DMLab
Sun397
Camelyon
KITT-Dist
10'
schedule_stene stepp
schedulo,
steps
schedule, steps
Figure 20: Top-1 accuracy for each task with respect to the number fine-tuning steps on the full datasets. More steps usually better, but performance is stable after 1000 steps.
34

---

Under review as a conference paper at ICLR 2020
N COMPARISON TO VISUAL DECATHLON
0.12
0.10
0.08
0.06
0.04
0.02
0.00
-0.02
0..0
-0.6
Flowers102
SNORB-Elen
Caltech101
Pets
UCF101
dSpr-Orient
Aircraft
DPed
Oglot
dSpr-Loc
DMLab
SNORB-Azim
Clevr-Dist-
Resisc45
Sun397
EuroSAT
Camelyon
GTSR
DTD
SVHN
ImNet:
Retinopathy
KiTTI-Dist Clevr-Cound
CIFAR-100
Figure 21: Absolute differences in Kendall's rank correlation score between the "gold" ranking in each dataset, and the ranking obtained with either VTAB or Visual Decathlon. Bar colors indicate the category as usual, and yellow indicates datasets only present in Visual Decathlon. Positive values indicate that VTAB ranking is closer than Visual Decathlon ranking, when compared against the "gold" ranking of a particular dataset. The average ranking correlation for VTAB is 0.76, and 0.70 for Visual Decathlon.
In Section we describe the differences between VTAB and Visual Decathlon protocols. Here to compare the adaptation
answer more practical question: which benchmark should abilities of a set of methods to unseen tasks? We will show that the rank correlation with an unseen task, is expected be higher for the ranking obtained using VTAB, than using Visual Decathlon.
First, we fine-tuned each of our 16 baseline models in each of the Visual Decathlon datasets (using the lightweight hyperparameter search described in Section 3.1 which we downloaded directly from the competition's 's website. All datasets in the Visual Decathlon are provided so that the shorter size of the image is 72 pixels, while our baseline models were trained much larger resolutions. To overcome this difference in resolution, we resize the Visual Decathlon images to the resolution required by each model. We checked that our baseline models obtained reasonable results on the Visual Decathlon benchmark. In fact, our best model, SUP-ROTATION- 100%, reports test average accuracy of 78.22%, and a decathlon score of 3580, which are both slightly better than the best results reported in Rebuffi et al. (2018) (78. 78.08% and 3412. respectively), and other works (e.g. Rebuffi et al. (2017): Rosenfeld Tsotsos (018)). ). However, notice that comparing these numbers delicate, since did not use any data augmentation during training and all our models based on the Resnet50 architecture, while these works use heavy data augmentation (that depends on the dataset), and Resnet26-like architectures.
Then, for each task T' in the union of VTAB and Visual Decathlon, we rank the 16 baseline methods according their accuracy on task T. If we consider T the unseen dataset, this is the "gold" ranking of the studied methods. Now, we obtain two alternative rankings: one based on the mean accuracy in VTAB and another on Visual Decathlon. excluding task T in both cases avoid any bias. We can then compute, for each "unseen" dataset, the Kendall's 's ranking correlation between the "gold" ranking and each of the alternative rankings. Figure 21 shows the absolute differences in the rank correlation score between the VTAB-based ranking and Visual Decathlon- based ranking.
For most datasets, the difference positive, which means that the ranking according VTAB corre- lates better with the "gold", than the raking obtained using Visual Decathlon. Notice that even tasks that were not part of VTAB (colored gold), are better represented by VTAB's ranking than that of Visual Decathlon. These results not surprising, since VTAB contains more representative set of tasks than Visual Decathlon. The average ranking correlation for VTAB is 0.76, and 0.70 for Visual Decathlon.
35


Reviews:
[{"review_id": "rkxC3VqNqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a Visual Task Adaptation Benchmark (VTAB) as a diverse and\nchallenging benchmark to evaluate the learned representations. The VTAB judges whether the learned representation is good or not by adapting it to unseen tasks which have few examples. This paper conducts popular algorithms on a large amount of VTAB studies by answering questions: (1) how effective are ImageNet representation on non-standard datasets? (2) are generative models competitive? (3) is self-supervision useful if one already has labels?\n\nWhat is the size of the training dataset in the source domain?\n\nThe authors need to compare the conclusion obtained with other works. It seems that there is no new founding in this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkeMSAznFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n###  Summary\n1. The paper presents a new benchmark (VTAB) to evaluate different representation learning methods. VTAB mostly consists of a combination of existing classification datasets. (The classification tasks based on Clevr, SmallNorb, and DMLab are introduced by this paper.)\n2. It evaluates and compares the existing representation learning method on the proposed benchmark with an extensive hyper-parameter search. \n\n### Decision with reasons\nI vote for accepting this paper (weak accept). \n\n1. The experimental evaluation in this paper is outstanding. Different methods are compared fairly with extensive hyper-parameter tuning (Heavyweight tuning). \n\n2. Even though I find the experiment work to be illuminating, I'm not fully convinced that we need more standardized benchmarks. I'm also not convinced by some of the choices made in designing the benchmark (Such as the models can be pre-trained on any data as long as there is no overlap with the evaluation data).  \n\nI find the experimental evaluation to be useful enough for considering accepting this paper, but I will not strongly advocate for acceptance (Unless the authors can better explain why we need yet another benchmark.) \n\n### Supporting arguments for the reasons for the decision.\n\n1. The paper compares existing representation learning methods in a very convincing way. Even though it misses some baselines (Meta-learning based representation learning), it compares supervised, self-supervised, semi-supervised, and generative methods comprehensively. Moreover, the hyper-parameter selection process for the benchmark is fully defined in a dataset agnostic way (It does not depend on the evaluation dataset). This is more important than it might seem -- fully specifying algorithms is important and currently rarely done in the research literature. \n\nI'm not aware of any existing work that compares various representation learning methods on a diverse benchmark fairly and comprehensively, and I'm sure I'll be using the results in this paper to support my claims in the future. As a result, I think this work is worth accepting just for its thorough evaluation of existing methods. \n\n2. However, the paper does not add much to discourse apart from the thorough evaluation. Benchmarks are already prevalent in representation learning, and VTAB doesn't focus on a single well-defined problem with-in representation learning. \n\nFor example, dSprites is a benchmark to evaluate how well algorithms can disentangle the underlying data generation factors. An algorithm that significantly improves results on dSprites would, as a result, be better at learning disentangled representation. VTAB, on the other hand, is not trying to highlight any specific problem in representation. As a result, it's not clear what it means if an algorithm does better on VTAB (Apart from the vague notion that it is learning 'better' representations.) \n\nThis problem is especially exacerbated by the fact that VTAB does not restrict representation learning to a fixed dataset. This means that a trivial way of improving performance is to just get more representative data (Which is not a bad way of improving performance, but not the way that requires scientific study). \n\n### Questions for Authors \n\n1. The paper makes a binary distinction between fine-tuning and linear evaluation when in-fact this is more nuanced. I suspect that the best performance would be achieved by feezing layers up to a point (And not just not freezing any layers or freezing everything and training a linear classifier). Did the authors explore other choices, and if not, why do they think the current binary distinction is enough. (From my perspective, it makes sense to freeze the initial layers which might be extracting more local/general features and adapt the later layers. The boundary point should be a hyper-parameter)\n\n2. Currently, VTAB puts no restrictions on the representation learning dataset as long as the representation learning dataset does not over-lap with the evaluation dataset. This is problematic since an easier way of improving results on VTAB is to just collect more data/ data that is more representative of evaluation tasks. One could argue that the point of VTAB is to not use details of evaluation datasets in making any decisions for representation learning, however, it is hard to enforce this. For example, if I augment Imagenet with a medical dataset different from the ones in VTAB, is that fine or does that count as cheating?\n\nI feel that a much better way of defining the benchmark would be to restrict representation learning to Imagenet. \n\n### Other minor comments \nI don't have many minor comments since the paper is very clear and well written. The author could consider augmenting their graphs with patterns in addition to color to make paper friendly for grey-scale printing and use a more colorblind-friendly color combination.  \n\n\n### Post discussion update ###\n\nOne of my primary concerns was the benchmark didn't restrict representation learning to a fixed dataset. That made it trivial to improve performance: just train on a more representative dataset. However, as authors clarified, any dataset collected conditioned on the test set is against the spirit of VTAB. Moreover, discovering what kind of data improves representations is also a research problem, and by restricting to a fixed pre-training dataset, VTAB would not be suitable for such research directions. \n\nI agree that solving representation learning might not just be about finding a better algorithm, but a better algorithm coupled with the right data. In that sense, VTAB's decision to allow any data for pretraining makes sense. However, I'm also concerned that the benchmark might push people to (directly or indirectly) start training on data more representative of VTAB evaluation tasks, and the performance gains in the benchmark will not translate to better algorithms. However, I don't see how this can be avoided for an open-ended benchmark. Since the authors have agreed to include guidelines on how to use the benchmark without abusing it, letting the community police itself given the guidelines is not a bad idea. \n\nI can see VTAB as a benchmark for answering interesting questions (mentioned by authors in the response) such as:\n\n1- How to incorporate large amounts of unlabelled data for representation learning. For example, can we learn better representations by self-supervised learning on a large corpus of data (Such as unlabelled youtube videos)\n2- Can we generate data using simulators that can help us learn better representations? \n\nFinally, the authors agreed to add meta-data in their leaderboards (which they plan to release after the review period). The meta-data would allow us to compare methods at a more granular level, such as comparing different unsupervised learning methods. Overall, I think the benchmark and the leaderboards would act as a good source for comparing different unsupervised, semi-supervised, and supervised learning methods (and their combination) fairly. \n\nI'm also updating my score from a weak-acceptance to an acceptance. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lno2my5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper essentially addresses the difficult problem of visual representation evaluation. It attempts to find a universal way of assessing the quality of a model's representation. The authors define a good representation as one that can adapt to unseen tasks with few examples. With this in mind, they propose Visual Task Adaptation Benchmark (VTAB), a benchmark that is focused on sample complexity and task diversity. \n\n- Very clear, well written and well structured. (Although not fully self contained in the main body of the paper - 20 pages of supplementary material!)\n- The benchmark tasks are constrained to unseen tasks, which seems obvious but is often violated when evaluating representations\n- It does a good attempt at covering a large spectrum of realistic domains (19 tasks!) to assess generality. \n- Extensive study is conducted, covering the published state of the art methods in each domain. \n- The study leads to interesting finding, such as promising results on self-supervision and negative results on generation.\n\nOverall, I believe the paper is an important contribution as it provides some interesting analysis of the current state of the art for visual representation learning.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a new benchmark for evaluating a plethora of models on a variety of tasks. In terms of scores, the paper received a borderline rating, with two reviews being rather superficial unfortunately. The last reviewer was positive. The reviewers generally agreed that the benchmark is interesting and carries value, and the AC agrees. Authors certainly invested a significant effort in designing the benchmark and performing a detailed analysis over several tasks and methods. However, the effort seems more engineering in nature and insights are somewhat lacking. For an experimental paper, presenting the results is interesting yet not sufficient. A much more in-depth analysis and discuss would warrant a deeper understanding of the results and open directions for future work. This part is currently underwhelming. 
Decision: Reject