Title: THE VISUAL TASK ADAPTATION BENCHMARK

Abstract: Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Yet, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one principle: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models competitive? Is self-supervision useful if one already has labels?

1. INTRODUCTION:

Deep learning has revolutionized computer vision. Distributed representations learned from raw pixels have enabled unprecedented performance on many visual understanding tasks. Hand-crafted features have been replaced with hand-annotated datasets, with thousands to millions of examples (Krizhevsky, 2009; Russakovsky et al., 2015) . By contrast, humans learn a wide range vision tasks using just a few examples per task. A key research challenge is to close this gap in sample efficiency, and unlock deep learning for the long tail of problems without many labels.

The goal of improving sample efficiency has been approached from many angles, such as: few-shot learning (Fei-Fei et al., 2006) , transfer learning (Pan & Yang, 2009) , domain adaptation (Wang & Deng, 2018) , and representation learning (Bengio et al., 2013) . Representation learning is studied in many contexts: supervised pre-training (Sharif Razavian et al., 2014) , self-supervised learning (Doersch et al., 2015) , semi-supervised learning (Chapelle et al., 2009) , generative modeling (Donahue et al., 2017) , and disentanglement learning (Higgins et al., 2017) . Yet many questions remain unanswered, such as: Do representations trained using ImageNet labels generalize to diverse tasks? Does the great progress on generative modeling translate in better general representations? Is selfsupervision useful when labelled datasets already exist? Are semantic image-level labels useful for tasks that require understanding the structure of the world?

The lack of a common benchmark impedes the understanding of visual representations; each subdomain has its own evaluation protocol, and it is hard to assess different approaches. Benchmarks have been critical in other domains, such as RL (Mnih et al., 2013) , natural image classification (Russakovsky et al., 2015) , and NLP (Wang et al., 2018) . Inspired by these successes, we propose a benchmark that follows similar principles: (i) we put minimal constraints to encourage creativity, (ii) focus on practical considerations, and (iii) make it challenging.

We present the Visual Task Adaptation Benchmark (VTAB). VTAB is based on a single principle: a better algorithm is one that solves a diverse set previously unseen tasks with fewest possible labels. The focus on sample complexity reflects our belief that learning with few labels is the key objective of representation learning. Task diversity is also crucial to assess a representation's generality. Therefore, our benchmark extends beyond standard natural classification tasks, and includes those related to sensorimotor control, medical imaging, and visual reasoning.

We treat the benchmark tasks as unseen, but otherwise, we do not constrain the representation learner, or transfer algorithm. Pre-training may be performed on any data (labelled or unlabelled), provided that it does not overlap with the benchmark tasks, which would violate the notion of unseen.

To transfer the representations one may freeze, fine-tune, add new layers, apply domain adaptation, meta-learning, or any other technique.

With VTAB, we perform an extensive representation study, which reveals: (i) Supervised pretraining on ImageNet yields useful representations for natural images. Surprisingly, these work on distant domains, but to a lesser extent. (ii) Self-supervision is effective, although other unsupervised methods, such as generative models appear less so. (iii) Combining self-supervised with supervision outperforms all other studied algorithms, even without additional unlabelled data. (iv) Restricted evaluation using a linear classifier could lead to different conclusions. Data, code, continuously updated results, and models used are made available at [anonymous-url] . We seek algorithms that perform well on a wide variety of unseen visual understanding tasks with few labels per task. 1 We first formalize this objective and then specify a practical benchmarking procedure to measure progress.

2. THE VISUAL TASK ADAPTATION BENCHMARK:

A dataset D n is a set of n instances {(x i , y i )} n i=1 with observations x i ∈ X and labels y i ∈ Y . A prediction function is any mapping F : X → Y (e.g. a classifier). A (learning) algorithm, A, takes as input a dataset and outputs a prediction function. For example, A may be a pre-trained neural network coupled with a training mechanism. An evaluation procedure, E T , takes F and outputs a scalar measuring F 's performance (e.g. test-set accuracy). We seek the algorithm that maximizes the expected performance over a distribution of tasks P T , where a task T is a tuple containing a task-specific dataset distribution D T and evaluation procedure. Given access to only n samples per task we aim to maximize:

SCORE n (A) = E T ∼P T E T [A(D n T )] ,

(1) This general formulation requires a few clarifications. First, the task distribution needs to be appropriate; we desire a spectrum of tasks covering those solvable by a vision algorithm with human-like capabilities. Second, n may be varied to measure the relative sample complexity of different algorithms. In practice we choose it to be similar to that of a modest labelling budget (Section 3.1). Third, we assume that P T is known, and the goal is to build an algorithm with the best inductive biases for solving samples from it. We desire an "open world", where evaluation data is extremely diverse and can always be freshly sampled. Unfortunately, in practice the benchmark must contain a finite number of fixed test sets, therefore we must ensure that the algorithms have not been pre-exposed to specific evaluation samples, as described below.

2.1. A PRACTICAL BENCHMARK:

We now describe the Visual Task Adaptation Benchmark (VTAB), designed to be the best possible proxy for Eq. (1). Fig. 1 contains an overview diagram. We then discuss a suite of methods that may be used to optimize VTAB: deep representation learning.

Eq. ( 1) is very difficult to compute in practice. One must define a representative P T and compute the desired expectation. If sampling the distribution, one must avoid overfitting to the sample. Finally, limiting computational complexity becomes paramount as the number of tasks grows.

Distribution over tasks:

We aim for universal visual understanding, so we informally define P T as "Tasks that a human can solve, from visual input alone.". See Appendix B where we validate this. Intuitively, such tasks should benefit from visual representations learned by observing and interacting with the natural world. The second clause eliminates tasks that require external knowledge that is (currently) unreasonable for a vision algorithm to acquire -an extreme example would classifying objects grouped by a function of their spelling in a natural language. Section 2.2 details the samples.

Expectation over tasks We approximate the expectation over tasks by an empirical average over a number of hand-picked samples. Ideally, we would sample a new task for each evaluation, as is possible in procedural environments, e.g. (Finn et al., 2017) . However, real-world vision tasks are expensive to collect, so we define a fixed, representative set of samples from P T . While fixing the set reduces variance, it introduces a risk of meta-overfitting.

Mitigating meta-overfitting To do this, we treat the evaluation tasks like a regular test set, and thus consider them unseen. This implies that algorithms that use pre-training must not pre-train on any of the evaluation tasks (even their unlabelled images). Upstream training should provide useful inductive biases for any draw from P T , so should not use test samples. Fortunately, despite numerous test re-evaluations on popular ML benchmarks, such as ImageNet, progress seems to transfer to new data (Recht et al., 2018; 2019; Kornblith et al., 2019) .

Unified implementation We use many diverse tasks, however, this could render usage impractical. As discussed, the algorithms must have no prior knowledge of the downstream tasks. Therefore, while it is permitted to run a hyperparameter search on each task, the search space cannot be taskdependent. To get meaningful results, we need to define architectures and hyperparameter searches that work well across the benchmark.

For this, we convert the tasks into classification problems. With a homogeneous task interface, we may control for possible confounding factors. For example, we may use the same architecture and hyperparameter sweep throughout. Of course, not all tasks can be efficiently modeled as classification with global labels, such as those requiring per-pixel predictions. Nonetheless, we design the tasks such that success on VTAB requires learning of diverse set of visual features: object identification, scene classification, pathology detection, counting, localization, 3D geometry, and others.

2.2. TASKS:

VTAB contains 19 tasks which cover a broad spectrum of domains and semantics. Appendix A contains details. These are grouped into three sets: NATURAL, SPECIALIZED, and STRUCTURED.

The NATURAL group represents classical vision problems. These tasks contain natural images captured using standard cameras. The classes may represent generic, fine-grained, or abstract objects. The group includes: Caltech101, CIFAR-100, DTD, Flowers102, Pets, Sun397, and SVHN.

The SPECIALIZED group also contains images of the world, but captured through specialist equipment. These images contain invariances from those in the NATURAL tasks. Nonetheless, humans recognize the structures therein, thus generic visual representation should also capture the visual concepts in these images. We have two sub-groups: remote sensing, and medical. Remote sensing includes Resisc45 and EuroSAT: aerial images of the earth captured using satellites or aerial photography. Medical includes Patch Camelyon, metastases detection from microscopy images, and Diabetic Retinopathy, retinopathy classification from fundus images.

The STRUCTURED group assesses comprehension of the structure of a scene, for example, object counting, or 3D depth prediction. Most of these tasks are generated from simulated environments, whose structure is easy for a human to determine, but whose domain differs greatly to datasets like ImageNet. These tasks are intended as a step towards useful representations for perceptual control. We include: Clevr: Simple shapes rendered in a 3D scene, we include two tasks: counting and depth prediction. dSprites: Simple black/white shapes rendered in 2D, we include two tasks: location and orientation prediction. SmallNORB: Artificial objects viewed under varying conditions, we include two tasks: object-azimuth and camera-elevation prediction. DMLab: Frames from a rendered 3D maze. The task involves predicting the time for a pre-trained RL agent to navigate to an object. KITTI: frames captured from a car driver's perspective. We predict the depth of the nearest vehicle.

2.3. REPRESENTATION AND TRANSFER LEARNING:

Success on VTAB, and ultimately optimizing Eq. ( 1), requires some knowledge of P T . For example, algorithms that use CNN architectures have a useful inductive bias for vision. While these biases are usually manually designed (e.g. through choice of an architecture), they can also be learned.

For example: (i) Zoph & Le (2017) learn performant architectures, (ii) Bello et al. (2017) learn optimization algorithms, (iii) Wong et al. (2018) learn hyperparameter search strategies, (iv) Cubuk et al. (2019) learn data augmentation functions, (v) Raghu et al. (2019) learn layer-wise initialization distributions, and (vi) pre-trained data representations can be learned. While all of the above merit investigation for VTAB, we focus on representation learning. Human visual perception is refined through years of observation and interaction with the world, resulting in a system that solves new tasks with few in-domain labels. Likewise, we aim to pre-train a network that extracts useful features, or representations, from raw data.

Transfer strategy When representations are not pre-trained on the evaluation tasks themselves (which is explicitly forbidden in VTAB), they must be adapted to solve the new task (using limited in-domain data). State-of-the-art representation learning algorithms typically train a deep neural network on some task(s), and then adapt the network to the new task. The simplest adaptation strategy is to freeze the network's weights and train another -usually smaller -model on top. When the upstream and downstream datasets differ significantly, fine-tuning the original weights is more effective (Yosinski et al., 2014; Kornblith et al., 2019) . VTAB does not constrain the transfer strategy; here we benchmark methods using fine-tuning since it tends to perform best.

Upstream training:

The representation learning literature often focuses on unsupervised learning as it may be applied to any data without human annotation. However, supervised data, where available, may also produce good representations. Indeed, the most popular models used in practice are pre-trained on ImageNet labels (Huh et al., 2016, and refs therein) . VTAB does not constrain the type of data (supervised or unsupervised) used to pre-train the representations.

3. EXPERIMENTS:

We use VTAB to evaluate popular representation learning algorithms. We ask four primary questions: (i) How effective are these algorithms, such as ImageNet pre-training, across a broad spectrum of tasks? (ii) How do different algorithm classes compare? (iii) Which method achieves the best overall performance on VTAB? (iv) How does VTAB compare to other benchmarking strategies?

3.1 SETUP Data Section 2.2 describes the tasks. VTAB aims to assess adaptation with limited data, so we evaluate primarily on 1,000 training examples per task, drawn i.i.d. from each training set. This simulates performance under a small per-task labelling budget which is the main goal of this benchmark. For completeness, we also use the full datasets. This allows us to assess the value of representation learning as in-domain data increases, and to check performance against prior art.

Representation learning algorithms We evaluate the following representation learning algorithms: supervised, semi-supervised, self-supervised, and generative models. As a baseline we train a "from-scratch" model, starting from random weights. We evaluate a total of 16 algorithms. Throughout, the models are pre-trained ImageNet. 2The supervised model is trained on 100% of the ImageNet labels (SUP-100%). For self-supervised learning, we include both image-based and patch-based models. The image-based models include ROTATION (Gidaris et al., 2018) and EXEMPLAR (Dosovitskiy et al., 2014) . The patch-based models include RELATIVE PATCH LOCATION (Doersch et al., 2015) and JIGSAW (Noroozi & Favaro, 2016) . We use the public implementations of Kolesnikov et al. (2019) . The patch-based models are converted into image-level classifiers by averaging the representations from a 3 × 3 grid of patches (see Appendix C). The semi-supervised models are trained using 10% of labelled ImageNet data with an auxiliary loss on all of the data, see (Zhai et al., 2019) . We use either rotation (SUP-ROTATION-10%) or Exemplar (SUP-EXEMPLAR-10%) auxiliary losses. These models can also use all of the labels, denoted SUP-ROTATION-100% and SUP-EXEMPLAR-100%. For generative models, we use GANs and VAEs. The most common way to extract representations from a GAN is from the discriminator, replacing its final linear layer. We use both the label-conditional and unconditional BigGAN discriminators (Brock et al., 2019) (COND-BIGGAN and UNCOND-BIGGAN) using the public implementations of Chen et al. (2019) ; Lucic et al. (2019) . For autoencoders, the output of the encoder can be used as the representation. We test VAEs (Kingma & Welling, 2014), and WAEs with three distribution matching losses: GAN, MMD (Tolstikhin et al., 2018), and UKL (Rubenstein et al., 2019) .

Transfer algorithm VTAB permits any transfer algorithm. We use fine-tuning of the entire network here, since that performs best (Kornblith et al., 2019) . Adding a linear model to frozen networks is also popular, so we compare to this approach in Section 3.4.

Hyperparameters Both the upstream pre-training of representations and transfer involve many hyperparameters. One major choice is architecture. Bigger architectures almost always perform better (see Appendix L and Kolesnikov et al. (2019) ). Therefore, for comparability, we use ResNet50-v2, or similar, for all methods. For the supervised and semi-supervised methods we use the architecture in He et al. (2016) . For GANs and auto-encoders we also use deep residual networks, but with appropriate modifications to train them successfully on their generative pre-training task. Appendix C contains details.

The hyperparameters used for transfer also influence performance, and different datasets may require different settings. We run VTAB in two modes: lightweight and heavyweight.

The lightweight mode performs a restricted hyperparameter search on each tasks, permitting fair comparison with few resources. Many hyperparameters are fixed, including optimizer, batch size, pre-processing, and weight decay. For each task, lightweight mode sweeps four hyperparameters: 2 initial learning rates, and 2 learning rate schedules. See Appendix I for details. We choose short schedules to limit cost, but show in Section 3.5 that these yield near-optimal performance.

In heavyweight mode we perform a large hyperparameter search, limited only by resource availability. Here, we perform a random search over learning rates, schedules, optimizers, batch size, train pre-processing functions, evaluation pre-processing functions, and weight decay. We include longer training schedules and higher resolution images. This mode is used to establish the best attainable VTAB score with the algorithms studied. Details in Appendix I.

Due to the high cost of heavyweight mode, we perform our main study in lightweight mode. In Section 3.3 we show that although extensive tuning improves all methods, the relative performances are mostly unaffected. For future study using VTAB, we recommend a similar lightweight setup that facilitates fair comparison without high cost. If large computation resources are available, the heavyweight mode may be used to push the state-of-the-art.

Protocol We perform model selection for each task on a validation set. We then re-train the best model on the training and validation set (or a 1000-example subset thereof), and evaluate on the test set. We run the test set evaluation three times and report the median score by default.

Metrics We evaluate all methods with top-1 accuracy. We consider other reported metrics in Appendix D. To aggregate scores across methods, we compute the mean top-1 accuracy. We investigate more complex aggregation strategies in Section 3.5, but the relative performances remain the same, so we report mean accuracy due to its simplicity.

3.2. VTAB RESULTS:

We run all 16 methods on 19 tasks with 1000 training examples and the full datasets using lightweight mode. We assess the value of representation learning over from-scratch training, and compare the performance of the different methods across the dataset groups. Fig. 4 shows two detailed per-task comparisons: (i) SUP-100% (ImageNet) versus FROM-SCRATCH which shows the value of standard ImageNet representations. (ii) SUP-ROTATION-100% versus SUP-100% which shows the best performing method versus ImageNet. Appendix G, Table 6 contains the complete results table.

Generative models These perform worst, often more poorly than training from scratch. This is perhaps unsurprising because scaling these models to complex datasets is known to be challenging. Nonetheless, despite SOTA generative quality on ImageNet, the GAN's discriminator does not provide useful representations. On the 1000-example subsets the UNCOND-BIGGAN outperforms from-scratch, but all generative models lose substantially on the full datasets. Interestingly, on the full dataset the GANs perform better than autoencoders on the NATURAL tasks, but worse on STRUCTURED. This indicates that they may be more strongly fit to the ImageNet domain. The methods are ordered according to their average accuracy across all datasets (dashed curve).

Similarly, Ravuri & Vinyals (2019) show that models with high sample quality fail to generate data from which an accurate classifier can be trained. Explicit consideration of representation quality, in addition to generative scores, may lead to further progress in generative modeling research.

Self-supervised All self-supervised representations outperform from-scratch training. The best, ROTATION attains 60.4% mean top-1 accuracy, while FROM-SCRATCH attains 43.1% (1000example datasets). Methods applied to the entire image (ROTATION, EXEMPLAR) outperform patchbased methods (JIGSAW, REL.PAT.LOC.). However, the patch based methods perform slightly better on DTD (Describable Textures) and Retinopathy (Fundus images), see Appendix G, Table 6 . These tasks require sensitivity to local textures, which is well represented by patch-based methods. Figure 2 shows that self-supervised perform worse than supervised methods on the NATURAL datasets, but similarly on SPECIALIZED, and even better on some STRUCTURED tasks. This indicates that ImageNet labels may encode sub-optimal invariances for non-natural tasks.

(Semi-)Supervised Supervised models perform best. The benefits are most pronounced on the NATURAL tasks, whose domain and task-semantics are arguably most similar ImageNet classification. The combination of self-supervision with 10% labels (semi-supervised) performs only marginally worse (1 -1.5%) than methods with 100% labels. Interestingly, there is benefit of com- bining self-supervised losses with 100% labelled ImageNet, i.e. without additional unlabelled data, see Fig. 4 . SUP-ROTATION-100% attains 66.4%, whereas SUP-ROTATION-100% attains 68.0%.

3.3. HEAVYWEIGHT HYPERPARAMETER SWEEPS:

We compare FROM-SCRATCH and the three best models: SUP-100%, SUP-ROTATION-100%, and SUP-EXEMPLAR-100% using the heavyweight hyperparameter search. Table 1 contains the results. The hyperparameters selected are shown in Appendix I. As expected, all methods improve significantly, e.g. FROM-SCRATCH improves from 43.1% to 61.2% (1000-examples). Prior work (He et al., 2018) shows that with sufficient data and training steps, from-scratch training is competitive for detection. We observe, with the heavy sweep, that although the gap to SUP-100% is smaller, it is still substantial: SUP-100% attains 72.8%. Again, addition of self-supervision helps, but the margin is smaller. SUP-EXEMPLAR-100% performs best with a VTAB (1000-example) score of 73.6.

We check our results against those recently reported in the literautre (Appendix H, Table 8 ). Our results are comparable; behind on highly popular tasks on which more complex architectures have been applied (e.g. CIFAR), but ahead in others. In Appendix L we show that simply increasing the architecture size improves VTAB performance.

In summary, pre-trained representations still offer substantial value with heavy hyperparameter tuning and long training schedules on VTAB. The overall ranking of methods remains the same, with the exception that SUP-EXEMPLAR-100% now outperforms SUP-ROTATION-100%, although these methods' performances are always similar.

Under review as a conference paper at ICLR 2020 Representations are often evaluated by training a linear layer on a frozen model (Kolesnikov et al., 2019) . ImageNet is often used for linear evaluation, but this is meaningless for the models we consider because many use ImageNet labels for pre-training. However, linear evaluation is also used in a transfer setting (Goyal et al., 2019a) , so we apply this protocol to the VTAB tasks, and contrast to fine-tuning used above. The full protocol resembles the lightweight fine-tuning sweep, see Appendix K.

3.4. FROZEN FEATURES EXTRACTORS:

Fig. 5 shows the per-task correlation between linear and fine-tuning. Appendix K, Table 9 contains the full table of results. We first note that linear evaluation significantly lowers performance, even when downstream data is limited to 1000 examples. For example, SUP-100% attains 66.4% with fine-tuning (lightweight sweep), but 57.3% with linear. Linear transfer would not by used in practice unless infrastructural constraints required it. Second, Fig. 5 shows that on many datasets, particularly SPECIALIZED and STRUCTURED, the correlation is low. Linear evaluation may lead to different conclusions, for example, COND-BIGGAN attains 43.3% (1000-examples) on linear, outperforming both both patchbased self-supervised methods (REL.PAT.LOC and JIGSAW). Yet when fine-tuned these patch-based methods attain around 53%, but the GAN, 41.4%.3 Another discrepancy is between semi-supervised and supervised. SEMI-ROTATION-10% and SEMI-EXEMPLAR-10% perform 1 -1.5% worse than SUP-100% with fine-tuning, but 4 -5% worse with linear evaluation. The self-supervised methods extract useful representations, just without linear separability. Previous works (Kornblith et al., 2019; Kolesnikov et al., 2019) claim that linear evaluation results are sensitive additional factors that we do not vary, such as ResNet version or pre-training regularization parameters. We conclude that linear evaluation is a poor proxy for overall reduced sample complexity.

3.5. ANALYSIS:

Per-Task Metric In previous works, some datasets are reported using alternative metrics to top-1 accuracy, namely: mean-per-class accuracy (for Caltech101) or Cohen's quadratic kappa (for Retinopathy). We study whether these metrics reveal different results to top-1 accuracy. We find that, although there are some minor ranking differences, the overall picture remains unchanged. Kendall's ranking correlation scores, with respect to top-1 accuracy, are: 1.0 (mean per-class accuracy), 0.97 (quadratic kappa). Appendix D shows the full rankings according to each metric.

Metric Aggregation Across Tasks Mean top-1 accuracy across tasks creates an implicit weighting. First, some tasks use the same input data (e.g Clevr-Count and Clevr-Dist), thus upweighting those domains. Second, the task groups differ in size. Third, the tasks exhibit different performance ranges across methods. Therefore, we compare seven different ranking aggregation strategies: (i) Macroaverage across datasets (grouping tasks with the same input data). (ii) Macro-average across groups (merging tasks in the same group, NATURAL, etc.). (iii) Geometric mean. (iv) Average rank. (v) Average rank after small perturbation of the scores with noise. (vi) Robust (binned) average rank. (vii) Elimination rank -equivalent to an "Exhaustive Ballot".

All strategies have a high Kendall's correlation with the vanilla mean across tasks is high (above 0.9). The most dissimilar strategy on the 1000-example datasets is the perturbed average rank, with correlation 0.91. Therefore, we use mean top-1 accuracy because it is simple, interpretable, and can be computed for each method independently, unlike rank-based aggregation. See Appendix E for details.

Under review as a conference paper at ICLR 2020 Representative Subset For rapid-prototyping, a representative subset of VTAB may be useful. We compute the rank correlation between the mean scores produced by each 20 5 subsets of five tasks, and the full suite. Appendix F contains the results. The top-5 subsets tend to span different domains, but differ to each other. Although using a subset might be useful for screening models, they were computed using our particular set of models, and may correlate less well in other experiments. Using subsets also increases the risk of meta-overfitting, which VTAB aims to avoid by having many tasks.

Limiting Evaluation Cost:

The cost is near linear in the schedule length. To determine a minimal, yet meaningful, schedule, we sweep over schedules ranging from 40 to 40,000 steps, with batch size 512 on all datasets. Fig. 6 summarizes the results, details Figs. 19 and 20, Appendix M. Most runs reach their optimal performance within 1,000 steps (updates), and do not improve significantly when trained for longer. We fine-tune the models using 16 Google Cloud TPU-v3 hardware accelerators. We conduct an additional experiment to assess whether our results can be reproduced with a more basic hardware setup. We evaluate SUP-100% on all VTAB tasks (1,000 examples) using a single NVidia P100 GPU with 1,000 steps, 64 images per mini-batch, and a learning rate of 0.01 . In this experiment we match the mean performance of the same model reported in Table 1 .

4. RELATED WORK:

Vision Benchmarks The Visual Decathlon (Rebuffi et al., 2017) contains ten classification tasks: Omniglot, and nine using natural images. Four overlap with VTAB, but VTAB includes several other domains and tasks. Importantly, these benchmarks have opposite evaluation protocols: The Visual Decathlon allows direct joint optimization on the tasks, but forbids external data. By contrast, VTAB forbids multi-task learning, but permits transfer from any arbitrary dataset -this simulates one-off training of representations which are then applied to novel tasks. Further, VTAB considers a lowsample regime (1000-examples), which reflects performance under a reasonable labelling budget. We conduct experiments showing that the methods ranked according to VTAB are more likely to transfer to new tasks, than those ranked according to the Visual Decathlon (Appendix N).

The Facebook AI SSL challenge (Goyal et al., 2019b) was proposed to evaluate self-supervised models. It contains two datasets of natural images, with four tasks, including classification, detection, and low-label classification. The original paper (Goyal et al., 2019a ) also includes navigation and surface normal estimation. In contrast, VTAB uses classification tasks only to admit taskindependent implementations. VTAB attains diversity with many tasks from alternative domains and with different semantics (localization, counting, etc.). VTAB experiments include self-supervised methods, but are not restricted to them. Indeed, a challenge posed by the VTAB benchmark is to outperform supervised ImageNet models. Most importantly, Facebook SSL requires transfer via a shallow network stacked on a fixed CNN, whereas VTAB permits any transfer, focussing only on performance with few samples. Meta-Dataset (Triantafillou et al., 2019) contains natural image classification tasks. The protocol differs to VTAB: most of Meta-Dataset's evaluation datasets are also in the training set. The train/test split is instead made across classes. Meta-Dataset is designed for few-shot learning, algorithms that use considerably fewer than 1000 samples. This regime requires different solutions to VTAB.

Other ML Benchmarks Benchmark suites have driven progress in other areas of machine learning, such as the GLUE (Wang et al., 2018) and the NL Decathlon (McCann et al., 2018) in NLP, or the Atari suite (Mnih et al., 2013) and Mujoco environments (Todorov et al., 2012) in RL. In NLP, the GLUE Benchmark (Wang et al., 2018 ) has been a central benchmark used to measure progress of BERT (Devlin et al., 2018) and variants.

Representation Learning Evaluation Popular evaluations for representation learning are linear/MLP and semi-supervised. In linear/MLP evaluation, widely used in for self-supervised representations (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Doersch & Zisserman, 2017) , the weights of a pre-trained network are frozen, and a linear layer/MLP is trained on top to predict the labels. Evaluation is often performed on the same dataset that was used to train the representations, but using all labels, defeating the goal of sample efficiency. A semi-supervised protocol is more realistic and performs better (Zhai et al., 2019) . This evaluation is performed on a single dataset, by contrast VTAB concerns transfer to unseen tasks. Linear evaluation is sensitive to small upstream training details, such as the ResNet version or regularization, that make little difference when training end-to-end (Kornblith et al., 2019; Kolesnikov et al., 2019) . Other intrinsic scores have been proposed to measure the disentanglement of representations (Locatello et al., 2019) , or the mutual information between the input and representations (Hjelm et al., 2018) . Both were shown to be weakly correlated with representation utility (Locatello et al., 2019; Tschannen et al., 2019) .

Generative Models Evaluation Generative models are usually evaluated by generative quality, a difficult goal with numerous proxy metrics, such as reconstruction error, Frechet Inception Distance (Heusel et al., 2017) , Inception score (Salimans et al., 2016 ), precision-recall (Sajjadi et al., 2018) . log likelihood (for auto-regressive models), or the evidence lower bound (for VAEs). Sometimes generative models are also evaluated using linear classification (Radford et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017) , or semi-supervised learning (Kingma et al., 2014; Narayanaswamy et al., 2017; Tschannen et al., 2018) .

5. DISCUSSION:

Our results illuminate the current state of representation learning. First, ImageNet labels provide a strong baseline, especially for natural datasets. However, on specialized datasets e.g. medical, distant domains, or tasks that require structured understanding, their value is reduced. Research on how to train on more diverse data sources, such as video, may be the key to moving beyond the "ImageNet domain" towards general, more human-like, visual representations.

Second, generative models -at least the common ones -appear not to be competitive representation learners. 4 Generation may be a useful endeavour in of itself, but if generative losses are used as a means towards representing data, then more explicit representation evaluation may assist progress.

Third, self-supervised learning appears promising, being the best-performing unsupervised method. When combined with a few labels (10%), these methods perform almost as well as fully supervised ImageNet pre-training. Self-supervised even provides some value on top of ImageNet labels on the same data. We observe modest, yet significant, improvements over pure supervised learning, especially on tasks that require structured understanding. ImageNet labels encode lots of useful information, but also encode invariances that a universal representation should instead be sensitive to. For example, the distance to an object.

Representation learning is inherently tied to transfer learning; without a transfer strategy representations cannot be used for new tasks. Here, we elect fine-tuning since it performs better than the popular linear-model alternative, and is known to be less brittle. However, it would be interesting to research alternative strategies from domain adaptation (Wang & Deng, 2018) , residual adapters (Rebuffi et al., 2017), transfusion (Raghu et al., 2019) , and other fields.

VTAB may be used to optimize many design choices involved in learning generalizable representations. Here, we study upstream training loss. We therefore control other factors such as hyperparameter sweep size, architecture, transfer algorithm, downstream preprocessing, and upstream training data. Varying these factors, and others, to improve VTAB is valuable future research. However, the effect of each factor should be isolated from other confounders, such as hyperparameter sweep size. The only approach that is out-of-bounds is to condition the algorithm explicitly on the test tasks, since this would compromise representation generalization.

Code and data to run the Visual Task Adaptation Benchmark is made available, and progress is publicly monitored at [anonymous-url]. All pre-trained models presented are also released. We hope that VTAB drives representation learning towards the goal of making deep learning accessible to the many applications without vast labelling budgets. Table 2 provides statistics and references for all of the tasks in VTAB.

A TASKS:

We now provide a brief description of each task.

Caltech101 (Li et al., 2006) The task consists in classifying pictures of objects (101 classes plus a background clutter class), including animals, airplanes, chairs, or scissors. The image size varies, but it typically ranges from 200-300 pixels per edge. CIFAR-100 (Krizhevsky, 2009) The task consists in classifying natural images (100 classes, with 600 training images each). Some examples include apples, bottles, dinosaurs, and bicycles. The image size is 32x32. DTD (Cimpoi et al., 2014) The task consists in classifying images of textural patterns (47 classes, with 120 training images each). Some of the textures are banded, bubbly, meshed, lined, or porous. The image size ranges between 300x300 and 640x640 pixels. Flowers102 (Nilsback & Zisserman, 2008) The task consists in classifying images of flowers present in the UK (102 classes, with between 40 and 248 training images per class). Azalea, Californian Poppy, Sunflower, or Petunia are some examples. Each image dimension has at least 500 pixels. Pets (Parkhi et al., 2012) The task consists in classifying pictures of cat and dog breeds (37 classes with around 200 images each), including Persian cat, Chihuahua dog, English Setter dog, or Bengal cat. Images dimensions are typically 200 pixels or larger. Sun397 (Xiao et al., 2010) The Sun397 task is a scenery benchmark with 397 classes and, at least, 100 images per class. Classes have a hierarchy structure, and include cathedral, staircase, shelter, river, or archipelago. The images are (colour) 200x200 pixels or larger. SVHN (Netzer et al., 2011) This task consists in classifying images of Google's street-view house numbers (10 classes, with more than 1000 training images each). The image size is 32x32 pixels. EuroSAT (Helber et al., 2019) The task consists in classifying Sentinel-2 satellite images into 10 different types of land use (Residential, Industrial, River, Highway, etc) . The spatial resolution corresponds to 10 meters per pixel, and the image size is 64x64 pixels.

Resisc45 (Cheng et al., 2017) The Remote Sensing Image Scene Classification (RESISC) dataset is a scene classification task from remote sensing images. There are 45 classes, containing 700 images each, including tennis court, ship, island, lake, parking lot, sparse residential, or stadium. The image size is RGB 256x256 pixels. Patch Camelyon (Veeling et al., 2018) The Patch Camelyon dataset contains 327,680 images of histopathologic scans of lymph node sections. The classification task consists in predicting the presence of metastatic tissue in given image (i.e., two classes). All images are 96x96 pixels. Retinopathy (Kaggle & EyePacs, 2015) The Diabetic Retinopathy dataset consists of image-label pairs with high-resolution retina images, and labels that indicate the presence of Diabetic Retinopahy (DR) in a 0-4 scale (No DR, Mild, Moderate, Severe, or Proliferative DR). Clevr/count (Johnson et al., 2017) CLEVR is a visual question and answer dataset designed to evaluate algorithmic visual reasoning. We use just the images from this dataset, and create a synthetic task by setting the label equal to the number of objects in the images. Clevr/distance (Johnson et al., 2017) Another synthetic task we create from CLEVR consists of predicting the depth of the closest object in the image from the camera. The depths are bucketed into size bins. dSprites/location (Matthey et al., 2017) The dSprites dataset was originally designed to asses disentanglement properties of unsupervised learning algorithms. In particular, each image is a 2D shape where six factors are controlled: color, shape, scale, rotation, and (x,y) center coordinates. Images have 64x64 black-and-white pixels. This task consists in predicting the x (horizontal) coordinate of the object. The locations are bucketed into 16 bins. dSprites/orientation (Matthey et al., 2017) We create another task from dSprites consists in predicting the orientation of each object, bucketed into 16 bins. SmallNORB/azimuth (LeCun et al., 2004) The Small NORB dataset contains images of 3D-toys from 50 classes, including animals, human figures, airplanes, trucks, and cars. The image size is 640x480 pixels. In this case, we define labels depending on the azimuth (angle of horizontal deviation), in intervals of 20 degrees (18 classes). SmallNORB/elevation (LeCun et al., 2004) Another synthetic task we create from Small NORB consists in predicting the elevation in the image. There are 9 classes, corresponding to 9 different elevations ranging from 30 to 70 degrees, in intervals of 5 degrees. DMLab (Beattie et al., 2016) The DMLab (DeepMind Lab) suite is a set of control environments focused on 3D navigation and puzzle-solving tasks. We collected a classification dataset by running some episodes using a pre-defined policy that collects objects by moving to them. The objects in the environment are grouped into two classes. We record the frame (agent's view) at each state, and the time (number of steps) until the agent reaches the next object. The label space is the cross-product of this object's group and the time taken to reach the next it (bucketed into three groups). This task amounts to predicting the identity and distance of an object in a 3D environment. KITTI-Dist (Geiger et al., 2013) The KITTI task consists in predicting the (binned) depth to the vehicle (car, van, or truck) in the image. There are 4 bins / classes.

B HUMAN EVALUATION:

We define P (T ) informally as "All tasks that a human can solve using visual input alone". Intuitively, the tasks in Appendix A satisfy this property because the task semantics involve simple visual concepts (for a human), and the images contain recognisable objects -either in a natural or artificial environment. However, we also check empirically that humans can solve the types of tasks using in VTAB from examples alone. We therefore evaluate human raters on a representative subset of the tasks used in VTAB: Pets (natural images, fine-grained object labels), DTD (natural, textures), Camelyon (specialized, medical images), EuroSAT (specialized, aerial images), DMLab (structured, distance prediction), and Clevr-count (structured, object counting).

For each human-evaluated dataset, raters are given 20 random examples for each class, taken from the training split. Raters are asked to classify between 50 and 100 images each, for a total of 1K images (except for DMLab: 534 images), randomly taken from the test split, based on the provided training examples. Beside the examples, no hints nor explanations are given on the nature of the tasks. The raters have to deduce which properties of each image should be used to classify those, for example: the breed of the animal, the distance between the camera and the objects, or the number of objects. The raters are asked not to exchange on the tasks, so each rater produces independent work.

All datasets rated by humans are the same as the one rated by the models, except for DMLab, where we only assess the distance prediction aspect and not object type. This is because there are too many object types for the raters to learn the two groups of objects from 20 examples per class. Therefore, we asses distance-prediction alone. The human DMLab task contains only 3 classes, each of which contains many object types, and differ only in object distance.

Note that this evaluation is not meant to quantify the relative performances of humans and machines, since protocol differences e. Table 3 : Human evaluation scores, measured using mean-per-class accuracy.

We measure human performance using mean-per-class accuracy. This is because the human training sets are class-balanced, so the raters cannot learn the class priors which algorithms that see an i.i.d sample of the training set could. Table 3 shows the results. The results indicate that some tasks are harder due to more subtle distinctions (Camelyon) or noise (DMLab), and some very easy (Clevrcount). However, in all cases the raters perform significantly better than random guessing. This demonstrates that the types of tasks used in VTAB are ones for which human-like visual representations are useful to solve them, using few labels and visual-input alone.

C NETWORK ARCHITECTURES:

All of the methods (except for BigGAN models) we evaluate in the paper use the standard ResNet50-v2 (He et al., 2016) architecture.

This architecture consists of 50 convolutional layers. The representation produced by this model has 2048 dimensions. Exact details of this architecture can be found in (He et al., 2016) .

In the Cond-BigGAN and Uncond-BigGAN models we use publicly available5 implementation by Lucic et al. (2019) and (Chen et al., 2019) of the custom ResNet-like architecture proposed and described in (Brock et al., 2019) . It has 1536 dimensions in the final representation layer.

We use a specialized procedure for evaluating patch-based models (Relative Patch Location (Doersch et al., 2015) and Jigsaw (Noroozi & Favaro, 2016) ). These models use ResNet50 model with the overall stride reduced from 64 to 16 (by substituting the first and the last strided convolution of the standard ResNet50 model by a convolution with stride one). During the adaptation phase, we apply ResNet50 with reduced stride in the following way (assuming that input image size is 224 × 224):

• Perform central crop of size 192 × 192.

• Cut image into 3 × 3 patches of size 64 × 64.

• Apply the ResNet50 model independently to every patch.

• Output the final representation and element-wise average of 9 individual patch representations.

E ALTERNATIVE WEIGHTING AND RANKING SCHEMES FOR MODELS:

Throughout, we use (unweighted) mean top-1 accuracy across all tasks to rank the different models. This assumes that the samples represent our desired task distribution P T in an unbiased manner.

However, there may be other sensible weighting schemes that are not uniform. Here we explore three alternative weighting schemes (i) assigning equal weight to every dataset -there are three data sets that are used in two tasks, and the others are used in one task, (ii) assigning equal weight to each task group: NATURAL, SPECIALIZED, and STRUCTURED, and (iii) the weighting scheme introduced in Balduzzi et al. (2018) .

Another issue inherent in (weighted or unweighted) mean accuracy is that the mean accuracies of individual tasks vary significantly depending on the task's difficulty. Since maximum accuracy is bounded, this may limit the range of performances, implicitly upweighting tasks with more "headroom". Therefore, we explore a few simple alternative ranking strategies (see Dwork et al. (2001) for an introduction raking aggregation methods):

(i) Ranking according to the geometric mean.

(ii) The average rank obtained by ranking the models for each task according to accuracy and then computing the average rank across tasks.

(iii) A noise-perturbed version of (i) in which the accuracies are perturbed by Gaussian noise with variance 1.0 prior to ranking.

(iv) A robust variant of the average rank, where, prior to averaging ranks across tasks, the accuracy is binned into buckets of size 1% and all models in the same bucket obtain the same rank.

(v) An elimination ranking scheme equivalent to the "Exhaustive Ballot" voting system, see e.g. (Shahandashti, 2016) .

We measure the agreement between these ranking strategies using the Kendall rank correlation coefficient (Kendall, 1945) . To account for the training/evaluation stochasticity in VTAB, we sample (independently for every pair of model and task) one out of the three test-set repetitions, and compute the rank correlation between ranking according to the mean accuracy and each alternative ranking. We average over 100 such samples. The results are in Table 4 . The mean rank correlation between the ranking according to the mean across tasks correlates very well with alternative ranking schemes.

In particular, weighted means for 1000 samples and the full dataset as well as the geometric mean have a rank correlation with the ranking according to the mean that exceeds 0.93. The agreement with different types of average rank somewhat lower for 1000 samples, but still considerable. We hence conclude that the mean accuracy across tasks is a fairly robust metric to rank models in VTAB. (Kendall, 1945) (with standard deviation) measuring the agreement of the ranking of models according to the mean accuracy across tasks with different types of weighted means, the geometric mean, and different types of average ranks. Agreement of ranking according to mean accuracy with alternative ranking schemes is high.

Ranking:



F REPRESENTATIVE SUBSET OF TASKS:

We explore another direction related to ranking of models: Which subset of tasks is most representative of the performance of the entire benchmark? Such a subset allows for cheap iteration which is beneficial during early development of new models.

We search the most representative 5 tasks from the full set of 20 tasks by performing exhaustive search over all 20 5 subsets. For each subset we compute the mean accuracy and compute the Kendall rank correlation coefficient between the resulting ranking of models and the ranking according to the mean over all tasks (averaged over 10 trials sampled as described in Appendix E). Table 5 shows the subsets which produce the highest rank correlation. There are many subsets which lead to an excellent agreement with the ranking according to the full mean. These subsets can be quite different provided that they are sufficiently diverse.

To assess how well we can expect this approach generalize to unseen models, we perform the following experiment. For each pair of models, we perform subset selection based on the accuracies of the remaining models, and check whether the ranking of the left out pair based on the mean across all data sets agrees with the ranking of the pair according to the mean over the subset. The mean outcome of this binary test across all pairs is an estimate of the probability for correctly ranking an unseen pair of models: 0.940 for the full data sets and 0.972 for 1000 examples. The subset selection hence generalizes to an unseen pair of models with high probability.

The representative subsets in Table 5 may be used for rapid prototyping of new methods before running the full VTAB. However, a caveat to the above analyses is that we evaluate the ranking of a very diverse set of models, from those whose performance is worse than from-scratch training to performance combinations of self-supervision and supervision. Therefore, the ranking is reasonably robust. To make fine-grained distinctions between models of a similar class, the representative subsets above may be less reliable. Repeated iteration on just a few tasks exposes one more to the risk of meta-overfitting. Table 5 : Task subsets of size 5 that lead to the largest rank correlation with the mean accuracy across all tasks. There are many different subsets that lead to an high rank correlation with the full mean.

H HEAVYWEIGHT EXPERIMENTS:

Table 7 shows the full heavyweight results. Table 8 shows recently reported results for each dataset in the literature (where available). The literature results may use substantially more complex architectures, or additional task-specific logic. For example, the Retinopathy result uses three neural networks combined with decision trees, and uses additional information, such as combining the images from two eyes of the same patient. 

I HYPERPARAMETER SWEEPS:

Lightweight sweep The lightweight mode performs a restricted hyperparameter search for all tasks, permitting fair comparison with few resources. It uses fixed values for most hyperparameters. We set the batch size to 512 and use SGD with momentum of 0.9. When fine-tuning, we do not use weight decay, and when training from scratch we set it to 0.001. We resize all images to 224 × 224, except for generative models, where we resize to 128 × 128 since training at a higher resolution is currently very challenging.

Lightweight mode sweeps four hyperparameters:

• Learning rate: {0.1, 0.01}

• Traning schedule: In all cases we decay the learning rate by a factor of 10 after 1 3 and 2 3 of the training time, and one more time shortly before the end. We train for {2 500, 10 000} training steps (i.e. model updates).

Note that when training on 1000 examples, we perform model selection using the regular validation set for each task. This setting is somewhat unrealistic, since in practice one would train on the larger validation set. However, we use this setting, which is common in prior art (Oliver et al., 2018) , because it significantly reduces evaluation noise. Further, Zhai et al. (2019) show that using a small validation yields the same conclusions.

Heavyweight sweep:

We define a relatively large search-space over relevant hyperparameters for the fine-tuning adaptation phase and perform an individual random search for each downstream task's validation set in that space. The random search consists of 100 independent trials. Specifically, we search:

• Batch-size: {128, 256, 512}

• Training schedule: In all cases we decay the learning rate by a factor of 10 after 1 3 and 2 3 of the training time, and one more time shortly before the end. We train for either any of {100, 200, 300, 400, 500} epochs over the dataset, or any of {2 500, 5 000, 10 000, 20 000, 40 000} training steps (i.e. model updates). We then choose the set of hyperparameters with the best performance at the end of training for each task individually.

For both training from scratch, and from a pre-trained supervised ImageNet training, we present the best set of hyperparameters for each task in Figs. 8 to 10 as a red dot. The green background shows the distribution of hyperparameters which performed within 2% of the best one for each task, providing a sense for the importance of tuning that hyperparameter.

In Figs. 12 and 13, we show the achieved performance for the full distribution of hyperparameters as violin plots for the CIFAR-100 and DMLab downstream tasks, both for the small and full variants. Figure 20: Top-1 accuracy for each task with respect to the number of fine-tuning steps on the full datasets. More steps is usually better, but performance is stable after 1000 steps.

:

In some practical settings additional unlabelled data may be available for unseen tasks in addition to the labelled data. We omit this setting, leaving it to future work.

ImageNet is not a VTAB evaluation task by design to permit upstream training on this dataset.

Surprisingly, fine-tuning the conditional GAN is slightly worse than linear evaluation. Fine-tuning also involves training a linear head from-scratch. It seems that while doing so, the representations in the rest of the model are degraded slightly.

Concurrent work, BigBiGAN (Donahue & Simonyan, 2019) claims better representations, this would be interesting to evaluate in future work.

https://github.com/google/compare_gan/blob/master/compare_gan/ architectures/resnet_biggan.py

Figures and Tables:

Figure 1: Overview of the VTAB protocol.

Figure 2: The methods are divided into five groups: Supervised with 100% labels, semi-supervised with 10% labels, self-supervised without labels, generative models, and training from-scratch. The plots show results on NATURAL, SPECIALIZED, and STRUCTURED datasets, respectively. The bar height indicates the mean top-1 accuracy across all methods and 3 test repeats within each group. The error bars indicate the variance across methods and runs.

Figure 5: Kendall rank correlation coefficient between fine-tuning and linear evaluation on each dataset.

Figure 6: Performance of two methods under various evaluation time constraints.

Figure 8: Training from scratch with only 1000 examples. Best hyperparameter values are marked in red and those within 2% in green to show sensitivity.

Figure 9: Training from scratch on the full datasets. Best hyperparameter values are marked in red and those within 2% in green to show sensitivity.

Figure 10: ImageNet supervised representation fine-tuned on only 1000 examples. Best hyperparameter values are marked in red and those within 2% in green to show sensitivity.

Figure 11: ImageNet supervised representation fine-tuned on the full datasets. Best hyperparameter values are marked in red and those within 2% in green to show sensitivity.

Figure 12: Heavy parameter sweep on CIFAR-100 dataset. Top 2 rows show the results of from scratch algorithm and bottom 2 rows show the results of supervised 100% algorithm. Each plot shows the violin plot of the results with respect to a given parameter. Here we show the results evaluated on full datasets and 1k datasets.

Figure 13: Heavy parameter sweep on DMLab dataset. Top 2 rows show the results of from scratch algorithm and bottom 2 rows show the results of supervised 100% algorithm. Each box shows the violin plot of the results with respect to a given parameter. We show the results evaluated on full datasets and 1k datasets.

Figures 19 and 20 show the top-1 accuracy attained by SUP-100% and ROTATION using different schedule lengths when using 1000 samples or the full dataset, respectively.

Absolute difference in top-1 accuracy between method pairs for each dataset. The bar colour indicates the task group: NATURAL, SPECIALIZED, and STRUCTURED. Left: SUP-100% versus FROM-SCRATCH -supervised pre-training yields a large improvement on the NATURAL datasets and some others. Right: SUP-ROTATION-100% versus SUP-100% -the additional self-supervised loss on top of supervised loss yields improvements, especially on the Sup-Rot-100% 88.0 48.9 68.1 94.7 89.0 91.8 33.7 86.2 96.6 85.2 78.5 66.5 61.5 45.5 77.2 88.8 71.8 59.4 52.0 72.8 Sup-Ex-100% 90.4 50.4 66.1 94.3 90.6 91.3 35.2 86.7 95.8 83.3 78.4 73.8 62.4 44.1 79.7 87.5 71.1 62.8 54.8 73.6 Top-1 accuracy of the top performing models and Resnet50 from-scratch.

Table content: <table><tr><td/><td>Top-1 Accuracy Delta</td><td>-10 0 10 20 30 40 50 60 70 80</td><td>sNORB-Elev</td><td>Retinopathy</td><td>Clevr-Dist</td><td>Clevr-Count</td><td>Camelyon</td><td>dSpr-Orient</td><td>dSpr-Loc</td><td>DMLab</td><td>EuroSAT</td><td>sNORB-Azim</td><td>SVHN</td><td>KITTI-Dist</td><td>Sun397</td><td>Resisc45</td><td>CIFAR-100</td><td>DTD</td><td>Flowers102</td><td>Caltech101</td><td>Pets</td><td>Sup-100% From-Scratch</td><td/><td>Top-1 Accuracy Delta</td><td>-2 0 2 4 6 8 -4</td><td>CIFAR-100</td><td>Retinopathy</td><td>sNORB-Elev</td><td>Pets</td><td>Sun397</td><td>DMLab</td><td>EuroSAT</td><td>Caltech101</td><td>Resisc45</td><td>Clevr-Dist</td><td>SVHN</td><td>Flowers102</td><td>dSpr-Loc</td><td>Camelyon</td><td>DTD</td><td>sNORB-Azim</td><td>dSpr-Orient</td><td>Clevr-Count</td><td>KITTI-Dist</td><td>Sup-Rotation-100% Sup-100%</td></tr><tr><td colspan="8">Figure 4: Caltech101</td><td/><td colspan="2">CIFAR-100</td><td/><td>DTD</td><td/><td>Flowers102</td><td/><td>Pets</td><td/><td colspan="2">SVHN</td><td/><td>Sun397</td><td>Camelyon</td><td>EuroSAT</td><td colspan="2">Resisc45</td><td>Retinopathy</td><td/><td colspan="2">Clevr-Count</td><td/><td>Clevr-Dist</td><td/><td>DMLab</td><td/><td>KITTI-Dist</td><td/><td colspan="2">dSpr-Loc</td><td colspan="2">dSpr-Ori</td><td/><td>sNORB-Azim</td><td/><td>sNORB-Elev</td></tr><tr><td/><td colspan="44">From-Scratch 74.5 77.8 67.1 85.8 70.9 97.0 70.1 91.2 98.8 94.3 82.8 99.8 96.7 76.6 68.4 100.0 96.7 99.9 94.0 86.4</td></tr><tr><td>Full</td><td colspan="44">Sup-100% Sup-Rot-100% 93.5 84.0 76.8 97.4 92.6 97.4 75.6 86.5 99.1 96.3 83.7 100.0 96.8 79.6 82.4 100.0 96.8 100.0 96.7 91.3 93.0 83.4 73.7 97.3 92.7 97.5 75.6 87.3 98.8 96.1 83.4 100.0 97.0 78.8 81.0 100.0 96.8 100.0 98.5 91.1</td></tr><tr><td/><td colspan="44">Sup-Ex-100% 93.8 83.1 76.5 97.8 92.9 97.5 75.3 86.5 99.0 96.3 83.7 99.9 96.8 79.3 82.8 100.0 96.7 100.0 99.1 91.4</td></tr></table>

Description of the datasets used for the tasks in VTAB.

Table content: <table><tr><td>Category</td><td>Dataset</td><td colspan="2">Train size Classes Reference</td></tr><tr><td>Natural</td><td>Caltech101</td><td>3,060</td><td>102 Li et al. (2006)</td></tr><tr><td>Natural</td><td>CIFAR-100</td><td>60,000</td><td>100 Krizhevsky (2009)</td></tr><tr><td>Natural</td><td>DTD</td><td>3,760</td><td>47 Cimpoi et al. (2014)</td></tr><tr><td>Natural</td><td>Flowers102</td><td>2,040</td><td>102 Nilsback &amp; Zisserman (2008)</td></tr><tr><td>Natural</td><td>Pets</td><td>3,680</td><td>37 Parkhi et al. (2012)</td></tr><tr><td>Natural</td><td>Sun397</td><td>87,003</td><td>397 Xiao et al. (2010)</td></tr><tr><td>Natural</td><td>SVHN</td><td>73,257</td><td>10 Netzer et al. (2011)</td></tr><tr><td colspan="2">Specialized EuroSAT</td><td>21,600</td><td>10 Helber et al. (2019)</td></tr><tr><td colspan="2">Specialized Resisc45</td><td>25,200</td><td>45 Cheng et al. (2017)</td></tr><tr><td colspan="2">Specialized Patch Camelyon</td><td>294,912</td><td>2 Veeling et al. (2018)</td></tr><tr><td colspan="2">Specialized Retinopathy</td><td>46,032</td><td>5 Kaggle &amp; EyePacs (2015)</td></tr><tr><td>Structured</td><td>Clevr/count</td><td>70,000</td><td>8 Johnson et al. (2017)</td></tr><tr><td>Structured</td><td>Clevr/distance</td><td>70,000</td><td>6 Johnson et al. (2017)</td></tr><tr><td>Structured</td><td>dSprites/location</td><td>663,552</td><td>16 Matthey et al. (2017)</td></tr><tr><td>Structured</td><td>dSprites/orientation</td><td>663,552</td><td>16 Matthey et al. (2017)</td></tr><tr><td>Structured</td><td>SmallNORB/azimuth</td><td>36,450</td><td>18 LeCun et al. (2004)</td></tr><tr><td>Structured</td><td>SmallNORB/elevation</td><td>36,450</td><td>9 LeCun et al. (2004)</td></tr><tr><td>Structured</td><td>DMLab</td><td>88,178</td><td>6 Beattie et al. (2016)</td></tr><tr><td>Structured</td><td>KITTI/distance</td><td>5,711</td><td>4 Geiger et al. (2013)</td></tr></table>

Kendall rank correlation coefficient

Table content: <table><tr><td>Method</td><td>1000 Samples</td><td>Full Dataset</td></tr><tr><td>Reweighted data sets</td><td colspan="2">0.979 ± 0.017 0.977 ± 0.018</td></tr><tr><td>Reweighted groups</td><td colspan="2">0.988 ± 0.013 0.992 ± 0.011</td></tr><tr><td colspan="3">Balduzzi et al. (2018) weighting 0.925 ± 0.022 0.945 ± 0.023</td></tr><tr><td>Geometric mean</td><td colspan="2">0.963 ± 0.022 0.969 ± 0.020</td></tr><tr><td>Average rank</td><td colspan="2">0.922 ± 0.022 0.951 ± 0.017</td></tr><tr><td>Average rank (perturbed)</td><td colspan="2">0.912 ± 0.028 0.931 ± 0.031</td></tr><tr><td>Average rank (robust)</td><td colspan="2">0.926 ± 0.021 0.971 ± 0.020</td></tr><tr><td>Elimination rank</td><td colspan="2">0.913 ± 0.029 0.935 ± 0.020</td></tr></table>

Top-1 accuracy of all the models evaluated on VTAB in heavyweight mode.

Table content: <table><tr><td/><td>Caltech101</td><td>CIFAR-100</td><td>DTD</td><td>Flowers102</td><td>Pets</td><td>SVHN</td><td>Sun397</td><td>Camelyon</td><td>EuroSAT</td><td>Resisc45</td><td>Retinopathy</td><td>Clevr-Count</td><td>Clevr-Dist</td><td>DMLab</td><td>KITTI-Dist</td><td>dSpr-Loc</td><td>dSpr-Ori</td><td>sNORB-Azim</td><td>sNORB-Elev</td><td>Mean</td></tr><tr><td/><td colspan="20">From-Scratch 60.2 24.5 51.1 73.4 43.1 88.3 14.9 82.1 93.3 63.9 75.4 53.0 57.3 36.6 55.7 88.6 67.6 52.2 81.6 61.2</td></tr><tr><td>1000</td><td colspan="20">Sup-100% Sup-Rot-100% 88.0 48.9 68.1 94.7 89.0 91.8 33.7 86.2 96.6 85.2 78.5 66.5 61.5 45.5 77.2 88.8 71.8 59.4 52.0 72.8 89.7 52.7 65.4 94.2 90.1 91.0 35.5 87.3 96.0 84.0 78.0 59.0 63.0 44.4 80.1 89.3 70.7 57.5 54.7 72.8</td></tr><tr><td/><td colspan="20">Sup-Ex-100% 90.4 50.4 66.1 94.3 90.6 91.3 35.2 86.7 95.8 83.3 78.4 73.8 62.4 44.1 79.7 87.5 71.1 62.8 54.8 73.6</td></tr><tr><td/><td colspan="20">From-Scratch 74.5 77.8 67.1 85.8 70.9 97.0 70.1 91.2 98.8 94.3 82.8 99.8 96.7 76.6 68.4 100.0 96.7 99.9 94.0 86.4</td></tr><tr><td>Full</td><td colspan="20">Sup-100% Sup-Rot-100% 93.5 84.0 76.8 97.4 92.6 97.4 75.6 86.5 99.1 96.3 83.7 100.0 96.8 79.6 82.4 100.0 96.8 100.0 96.7 91.3 93.0 83.4 73.7 97.3 92.7 97.5 75.6 87.3 98.8 96.1 83.4 100.0 97.0 78.8 81.0 100.0 96.8 100.0 98.5 91.1</td></tr><tr><td/><td colspan="20">Sup-Ex-100% 93.8 83.1 76.5 97.8 92.9 97.5 75.3 86.5 99.0 96.3 83.7 99.9 96.8 79.3 82.8 100.0 96.7 100.0 99.1 91.4</td></tr><tr><td/><td/><td colspan="4">SUP-EX.-100%</td><td/><td colspan="2">Result</td><td/><td colspan="3">Reference</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>Caltech101</td><td colspan="4">90.4 / 95.1 *</td><td colspan="15">86.9 / 95.1 * Cubuk et al. (2019) / Kornblith et al. (2019)</td></tr><tr><td/><td>CIFAR-100</td><td/><td colspan="2">83.1</td><td/><td/><td colspan="2">91.7</td><td/><td colspan="4">Tan &amp; Le (2019)</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>DTD</td><td/><td colspan="2">76.5</td><td/><td/><td colspan="2">78.1</td><td/><td colspan="6">Kornblith et al. (2019)</td><td/><td/><td/><td/></tr><tr><td/><td>Flowers102</td><td/><td colspan="2">97.8</td><td/><td/><td colspan="2">98.8</td><td/><td colspan="4">Tan &amp; Le (2019)</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>Pets</td><td/><td colspan="2">92.9</td><td/><td/><td colspan="2">95.9</td><td/><td colspan="5">Huang et al. (2018)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>SVHN</td><td/><td colspan="2">97.5</td><td/><td/><td colspan="2">99.0</td><td/><td colspan="5">Cubuk et al. (2019)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>Sun397</td><td/><td colspan="2">75.3</td><td/><td/><td colspan="2">72.0</td><td/><td colspan="5">Wang et al. (2017)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>Camelyon</td><td/><td colspan="2">86.5</td><td/><td/><td colspan="2">90.6</td><td/><td colspan="5">Teh &amp; Taylor (2019)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>EuroSAT</td><td/><td colspan="2">99.0</td><td/><td/><td colspan="2">96.4</td><td/><td colspan="5">Helber et al. (2019)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>Resisc45</td><td/><td colspan="2">96.3</td><td/><td/><td colspan="2">90.4</td><td/><td colspan="5">Cheng et al. (2017)</td><td/><td/><td/><td/><td/></tr><tr><td/><td>Retinopathy</td><td/><td colspan="2">74.7  †</td><td/><td/><td colspan="2">85.4  †</td><td/><td colspan="5">Wang et al. (2017)</td><td/><td/><td/><td/><td/></tr></table>

Comparison of the best method on the full datasets using heavyweight sweep (SUP-EXEMPLAR-100%) to results published in the literature (where available). For some datasets prior work does not use top-1 accuracy, therefore, we present our the performance of SUP-EXEMPLAR-100% using the same metric.

Table content: <table/>

Preprocessing: during training, optionally include random horizontal flipping. • Preprocessing: during training, optionally include random color distortions. • Preprocessing: during training, use any of the following techniques for resizing:res.: resize the image to 224; res.|crop: resize the image to 256 and take a random crop of size 224; res.sma|crop: resize the image keeping its aspect ratio such that the smaller side is 256, then take a random crop of size 224; inc.crop: "inception crop" fromSzegedy et al. (2015); cif.crop: resize the image to 224, zero-pad it by 28 on each side, then take a random crop of size 224. • Preprocessing: for evaluation, either resize the image to one of {224, 320, 384}, or resize it such that the smaller side is of size {256, 352, 448} and then take a central crop of size {224, 320, 384}. • Weight decay: we sample log-uniform randomly in [10 -5 , 10 -1 ].

Table content: <table><tr><td>• Optimizer: we randomly choose between</td></tr><tr><td>-SGD with learning-rate chosen log-uniform randomly in [0.001, 0.0], or</td></tr></table>

The supervised pre-training not only achieves better accuracy than training from scratch, but it also displays much smaller variance across hyperparameter values.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="9">Preprocessing</td><td/><td/><td/><td/><td colspan="5">Weight</td><td colspan="8">Learningrate</td></tr><tr><td colspan="3">Batch</td><td/><td/><td colspan="7">Schedule</td><td/><td colspan="9">FlipCol. Train</td><td/><td colspan="4">Eval</td><td/><td colspan="13">Decay SGD Adam</td></tr><tr><td>Caltech101</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Cifar100</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>DTD</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Flowers102</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Pets</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>SVHN</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Sun397</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Camelyon</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>EuroSAT</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Resisc45</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Retinopathy</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Clevr-Count</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Clevr-Dist</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>DMLab</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>KITTI-Dist</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>dSpr-Loc</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>dSpr-Orient</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>sNORB-Azim</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>sNORB-Elev</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Sum</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>128</td><td>256</td><td>512</td><td>100 epochs</td><td>200 epochs</td><td>300 epochs</td><td>400 epochs</td><td>500 epochs</td><td>2500 steps</td><td>5000 steps</td><td>10000 steps</td><td>20000 steps</td><td>40000 steps</td><td>none</td><td>horizontal</td><td>raw</td><td>distort</td><td>res.</td><td>res.|crop</td><td>res.sma|crop</td><td>inc.crop</td><td>cif.crop</td><td>res.@224</td><td>res.@320</td><td>res.@384</td><td>cen.crop@224</td><td>cen.crop@320</td><td>cen.crop@384</td><td>1e-5</td><td>1e-4</td><td>1e-3</td><td>1e-2</td><td>1e-1</td><td>0.001</td><td>0.01</td><td>0.1</td><td>1.0</td><td>1e-5</td><td>1e-4</td><td>1e-3</td><td>1e-2</td></tr></table>

Peer review (iclr2020):
### Summary 1. The paper presents a new benchmark (VTAB) to evaluate different representation learning methods. VTAB mostly consists of a combination of existing classification datasets. (The classification tasks based on Clevr, SmallNorb, and DMLab are introduced by this paper.) 2. It evaluates and compares the existing representation learning method on the proposed benchmark with an extensive hyper-parameter search. ### Decision with reasons I vote for accepting this paper (weak accept). 1. The experimental evaluation in this paper is outstanding. Different methods are compared fairly with extensive hyper-parameter tuning (Heavyweight tuning). 2. Even though I find the experiment work to be illuminating, I*m not fully convinced that we need more standardized benchmarks. I*m also not convinced by some of the choices made in designing the benchmark (Such as the models can be pre-trained on any data as long as there is no overlap with the evaluation data). I find the experimental evaluation to be useful enough for considering accepting this paper, but I will not strongly advocate for acceptance (Unless the authors can better explain why we need yet another benchmark.) ### Supporting arguments for the reasons for the decision. 1. The paper compares existing representation learning methods in a very convincing way. Even though it misses some baselines (Meta-learning based representation learning), it compares supervised, self-supervised, semi-supervised, and generative methods comprehensively. Moreover, the hyper-parameter selection process for the benchmark is fully defined in a dataset agnostic way (It does not depend on the evaluation dataset). This is more important than it might seem -- fully specifying algorithms is important and currently rarely done in the research literature. I*m not aware of any existing work that compares various representation learning methods on a diverse benchmark fairly and comprehensively, and I*m sure I*ll be using the results in this paper to support my claims in the future. As a result, I think this work is worth accepting just for its thorough evaluation of existing methods. 2. However, the paper does not add much to discourse apart from the thorough evaluation. Benchmarks are already prevalent in representation learning, and VTAB doesn*t focus on a single well-defined problem with-in representation learning. For example, dSprites is a benchmark to evaluate how well algorithms can disentangle the underlying data generation factors. An algorithm that significantly improves results on dSprites would, as a result, be better at learning disentangled representation. VTAB, on the other hand, is not trying to highlight any specific problem in representation. As a result, it*s not clear what it means if an algorithm does better on VTAB (Apart from the vague notion that it is learning *better* representations.) This problem is especially exacerbated by the fact that VTAB does not restrict representation learning to a fixed dataset. This means that a trivial way of improving performance is to just get more representative data (Which is not a bad way of improving performance, but not the way that requires scientific study). ### Questions for Authors 1. The paper makes a binary distinction between fine-tuning and linear evaluation when in-fact this is more nuanced. I suspect that the best performance would be achieved by feezing layers up to a point (And not just not freezing any layers or freezing everything and training a linear classifier). Did the authors explore other choices, and if not, why do they think the current binary distinction is enough. (From my perspective, it makes sense to freeze the initial layers which might be extracting more local/general features and adapt the later layers. The boundary point should be a hyper-parameter) 2. Currently, VTAB puts no restrictions on the representation learning dataset as long as the representation learning dataset does not over-lap with the evaluation dataset. This is problematic since an easier way of improving results on VTAB is to just collect more data/ data that is more representative of evaluation tasks. One could argue that the point of VTAB is to not use details of evaluation datasets in making any decisions for representation learning, however, it is hard to enforce this. For example, if I augment Imagenet with a medical dataset different from the ones in VTAB, is that fine or does that count as cheating? I feel that a much better way of defining the benchmark would be to restrict representation learning to Imagenet. ### Other minor comments I don*t have many minor comments since the paper is very clear and well written. The author could consider augmenting their graphs with patterns in addition to color to make paper friendly for grey-scale printing and use a more colorblind-friendly color combination. ### Post discussion update ### One of my primary concerns was the benchmark didn*t restrict representation learning to a fixed dataset. That made it trivial to improve performance: just train on a more representative dataset. However, as authors clarified, any dataset collected conditioned on the test set is against the spirit of VTAB. Moreover, discovering what kind of data improves representations is also a research problem, and by restricting to a fixed pre-training dataset, VTAB would not be suitable for such research directions. I agree that solving representation learning might not just be about finding a better algorithm, but a better algorithm coupled with the right data. In that sense, VTAB*s decision to allow any data for pretraining makes sense. However, I*m also concerned that the benchmark might push people to (directly or indirectly) start training on data more representative of VTAB evaluation tasks, and the performance gains in the benchmark will not translate to better algorithms. However, I don*t see how this can be avoided for an open-ended benchmark. Since the authors have agreed to include guidelines on how to use the benchmark without abusing it, letting the community police itself given the guidelines is not a bad idea. I can see VTAB as a benchmark for answering interesting questions (mentioned by authors in the response) such as: 1- How to incorporate large amounts of unlabelled data for representation learning. For example, can we learn better representations by self-supervised learning on a large corpus of data (Such as unlabelled youtube videos) 2- Can we generate data using simulators that can help us learn better representations? Finally, the authors agreed to add meta-data in their leaderboards (which they plan to release after the review period). The meta-data would allow us to compare methods at a more granular level, such as comparing different unsupervised learning methods. Overall, I think the benchmark and the leaderboards would act as a good source for comparing different unsupervised, semi-supervised, and supervised learning methods (and their combination) fairly. I*m also updating my score from a weak-acceptance to an acceptance.