The Visual Task Adaptation Benchmark

1 INTRODUCTION . Deep learning has revolutionized computer vision . Distributed representations learned from raw pixels have enabled unprecedented performance on many visual understanding tasks . Hand-crafted features have been replaced with hand-annotated datasets , with thousands to millions of examples ( Krizhevsky , 2009 ; Russakovsky et al. , 2015 ) . By contrast , humans learn a wide range vision tasks using just a few examples per task . A key research challenge is to close this gap in sample efficiency , and unlock deep learning for the long tail of problems without many labels . The goal of improving sample efficiency has been approached from many angles , such as : few-shot learning ( Fei-Fei et al. , 2006 ) , transfer learning ( Pan & Yang , 2009 ) , domain adaptation ( Wang & Deng , 2018 ) , and representation learning ( Bengio et al. , 2013 ) . Representation learning is studied in many contexts : supervised pre-training ( Sharif Razavian et al. , 2014 ) , self-supervised learning ( Doersch et al. , 2015 ) , semi-supervised learning ( Chapelle et al. , 2009 ) , generative modeling ( Donahue et al. , 2017 ) , and disentanglement learning ( Higgins et al. , 2017 ) . Yet many questions remain unanswered , such as : Do representations trained using ImageNet labels generalize to diverse tasks ? Does the great progress on generative modeling translate in better general representations ? Is selfsupervision useful when labelled datasets already exist ? Are semantic image-level labels useful for tasks that require understanding the structure of the world ? The lack of a common benchmark impedes the understanding of visual representations ; each subdomain has its own evaluation protocol , and it is hard to assess different approaches . Benchmarks have been critical in other domains , such as RL ( Mnih et al. , 2013 ) , natural image classification ( Russakovsky et al. , 2015 ) , and NLP ( Wang et al. , 2018 ) . Inspired by these successes , we propose a benchmark that follows similar principles : ( i ) we put minimal constraints to encourage creativity , ( ii ) focus on practical considerations , and ( iii ) make it challenging . We present the Visual Task Adaptation Benchmark ( VTAB ) . VTAB is based on a single principle : a better algorithm is one that solves a diverse set previously unseen tasks with fewest possible labels . The focus on sample complexity reflects our belief that learning with few labels is the key objective of representation learning . Task diversity is also crucial to assess a representation ’ s generality . Therefore , our benchmark extends beyond standard natural classification tasks , and includes those related to sensorimotor control , medical imaging , and visual reasoning . We treat the benchmark tasks as unseen , but otherwise , we do not constrain the representation learner , or transfer algorithm . Pre-training may be performed on any data ( labelled or unlabelled ) , provided that it does not overlap with the benchmark tasks , which would violate the notion of unseen . To transfer the representations one may freeze , fine-tune , add new layers , apply domain adaptation , meta-learning , or any other technique . With VTAB , we perform an extensive representation study , which reveals : ( i ) Supervised pretraining on ImageNet yields useful representations for natural images . Surprisingly , these work on distant domains , but to a lesser extent . ( ii ) Self-supervision is effective , although other unsupervised methods , such as generative models appear less so . ( iii ) Combining self-supervised with supervision outperforms all other studied algorithms , even without additional unlabelled data . ( iv ) Restricted evaluation using a linear classifier could lead to different conclusions . Data , code , continuously updated results , and models used are made available at [ anonymous-url ] . 2 THE VISUAL TASK ADAPTATION BENCHMARK . We seek algorithms that perform well on a wide variety of unseen visual understanding tasks with few labels per task.1 We first formalize this objective and then specify a practical benchmarking procedure to measure progress . A dataset Dn is a set of n instances { ( xi , yi ) } ni=1 with observations xi ∈ X and labels yi ∈ Y . A prediction function is any mapping F : X → Y ( e.g . a classifier ) . A ( learning ) algorithm , A , takes as input a dataset and outputs a prediction function . For example , A may be a pre-trained neural network coupled with a training mechanism . An evaluation procedure , ET , takes F and outputs a scalar measuring F ’ s performance ( e.g . test-set accuracy ) . We seek the algorithm that maximizes the expected performance over a distribution of tasks PT , where a task T is a tuple containing a task-specific dataset distribution DT and evaluation procedure . Given access to only n samples per task we aim to maximize : SCOREn ( A ) = ET∼PT ET [ A ( DnT ) ] , ( 1 ) This general formulation requires a few clarifications . First , the task distribution needs to be appropriate ; we desire a spectrum of tasks covering those solvable by a vision algorithm with human-like capabilities . Second , n may be varied to measure the relative sample complexity of different algorithms . In practice we choose it to be similar to that of a modest labelling budget ( Section 3.1 ) . Third , we assume that PT is known , and the goal is to build an algorithm with the best inductive biases for solving samples from it . We desire an “ open world ” , where evaluation data is extremely diverse and can always be freshly sampled . Unfortunately , in practice the benchmark must contain a finite number of fixed test sets , therefore we must ensure that the algorithms have not been pre-exposed to specific evaluation samples , as described below . 2.1 A PRACTICAL BENCHMARK . We now describe the Visual Task Adaptation Benchmark ( VTAB ) , designed to be the best possible proxy for Eq . ( 1 ) . Fig . 1 contains an overview diagram . We then discuss a suite of methods that may be used to optimize VTAB : deep representation learning . Eq . ( 1 ) is very difficult to compute in practice . One must define a representative PT and compute the desired expectation . If sampling the distribution , one must avoid overfitting to the sample . Finally , limiting computational complexity becomes paramount as the number of tasks grows . Distribution over tasks We aim for universal visual understanding , so we informally define PT as “ Tasks that a human can solve , from visual input alone. ” . See Appendix B where we validate this . Intuitively , such tasks should benefit from visual representations learned by observing and interacting with the natural world . The second clause eliminates tasks that require external knowledge that 1In some practical settings additional unlabelled data may be available for unseen tasks in addition to the labelled data . We omit this setting , leaving it to future work . is ( currently ) unreasonable for a vision algorithm to acquire – an extreme example would classifying objects grouped by a function of their spelling in a natural language . Section 2.2 details the samples . Expectation over tasks We approximate the expectation over tasks by an empirical average over a number of hand-picked samples . Ideally , we would sample a new task for each evaluation , as is possible in procedural environments , e.g . ( Finn et al. , 2017 ) . However , real-world vision tasks are expensive to collect , so we define a fixed , representative set of samples from PT . While fixing the set reduces variance , it introduces a risk of meta-overfitting . Mitigating meta-overfitting To do this , we treat the evaluation tasks like a regular test set , and thus consider them unseen . This implies that algorithms that use pre-training must not pre-train on any of the evaluation tasks ( even their unlabelled images ) . Upstream training should provide useful inductive biases for any draw from PT , so should not use test samples . Fortunately , despite numerous test re-evaluations on popular ML benchmarks , such as ImageNet , progress seems to transfer to new data ( Recht et al. , 2018 ; 2019 ; Kornblith et al. , 2019 ) . Unified implementation We use many diverse tasks , however , this could render usage impractical . As discussed , the algorithms must have no prior knowledge of the downstream tasks . Therefore , while it is permitted to run a hyperparameter search on each task , the search space can not be taskdependent . To get meaningful results , we need to define architectures and hyperparameter searches that work well across the benchmark . For this , we convert the tasks into classification problems . With a homogeneous task interface , we may control for possible confounding factors . For example , we may use the same architecture and hyperparameter sweep throughout . Of course , not all tasks can be efficiently modeled as classification with global labels , such as those requiring per-pixel predictions . Nonetheless , we design the tasks such that success on VTAB requires learning of diverse set of visual features : object identification , scene classification , pathology detection , counting , localization , 3D geometry , and others . 2.2 TASKS . VTAB contains 19 tasks which cover a broad spectrum of domains and semantics . Appendix A contains details . These are grouped into three sets : NATURAL , SPECIALIZED , and STRUCTURED . The NATURAL group represents classical vision problems . These tasks contain natural images captured using standard cameras . The classes may represent generic , fine-grained , or abstract objects . The group includes : Caltech101 , CIFAR-100 , DTD , Flowers102 , Pets , Sun397 , and SVHN . The SPECIALIZED group also contains images of the world , but captured through specialist equipment . These images contain invariances from those in the NATURAL tasks . Nonetheless , humans recognize the structures therein , thus generic visual representation should also capture the visual concepts in these images . We have two sub-groups : remote sensing , and medical . Remote sensing includes Resisc45 and EuroSAT : aerial images of the earth captured using satellites or aerial photography . Medical includes Patch Camelyon , metastases detection from microscopy images , and Diabetic Retinopathy , retinopathy classification from fundus images . The STRUCTURED group assesses comprehension of the structure of a scene , for example , object counting , or 3D depth prediction . Most of these tasks are generated from simulated environments , whose structure is easy for a human to determine , but whose domain differs greatly to datasets like ImageNet . These tasks are intended as a step towards useful representations for perceptual control . We include : Clevr : Simple shapes rendered in a 3D scene , we include two tasks : counting and depth prediction . dSprites : Simple black/white shapes rendered in 2D , we include two tasks : location and orientation prediction . SmallNORB : Artificial objects viewed under varying conditions , we include two tasks : object-azimuth and camera-elevation prediction . DMLab : Frames from a rendered 3D maze . The task involves predicting the time for a pre-trained RL agent to navigate to an object . KITTI : frames captured from a car driver ’ s perspective . We predict the depth of the nearest vehicle .

Summary: The paper essentially addresses the difficult problem of visual representation evaluation. It attempts to find a universal way of assessing the quality of a model's representation. The authors define a good representation as one that can adapt to unseen tasks with few examples. With this in mind, they propose Visual Task Adaptation Benchmark (VTAB), a benchmark that is focused on sample complexity and task diversity.