Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a Visual Task Adaptation Benchmark (VTAB) as a diverse and
challenging benchmark to evaluate the learned representations. The VTAB judges whether the learned representation is good or not by adapting it to unseen tasks which have few examples. This paper conducts popular algorithms on a large amount of VTAB studies by answering questions: (1) how effective are ImageNet representation on non-standard datasets? (2) are generative models competitive? (3) is self-supervision useful if one already has labels?

What is the size of the training dataset in the source domain?

The authors need to compare the conclusion obtained with other works. It seems that there is no new founding in this paper.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
The paper essentially addresses the difficult problem of visual representation evaluation. It attempts to find a universal way of assessing the quality of a model's representation. The authors define a good representation as one that can adapt to unseen tasks with few examples. With this in mind, they propose Visual Task Adaptation Benchmark (VTAB), a benchmark that is focused on sample complexity and task diversity. 

- Very clear, well written and well structured. (Although not fully self contained in the main body of the paper - 20 pages of supplementary material!)
- The benchmark tasks are constrained to unseen tasks, which seems obvious but is often violated when evaluating representations
- It does a good attempt at covering a large spectrum of realistic domains (19 tasks!) to assess generality. 
- Extensive study is conducted, covering the published state of the art methods in each domain. 
- The study leads to interesting finding, such as promising results on self-supervision and negative results on generation.

Overall, I believe the paper is an important contribution as it provides some interesting analysis of the current state of the art for visual representation learning.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
###  Summary
1. The paper presents a new benchmark (VTAB) to evaluate different representation learning methods. VTAB mostly consists of a combination of existing classification datasets. (The classification tasks based on Clevr, SmallNorb, and DMLab are introduced by this paper.)
2. It evaluates and compares the existing representation learning method on the proposed benchmark with an extensive hyper-parameter search. 

### Decision with reasons
I vote for accepting this paper (weak accept). 

1. The experimental evaluation in this paper is outstanding. Different methods are compared fairly with extensive hyper-parameter tuning (Heavyweight tuning). 

2. Even though I find the experiment work to be illuminating, I'm not fully convinced that we need more standardized benchmarks. I'm also not convinced by some of the choices made in designing the benchmark (Such as the models can be pre-trained on any data as long as there is no overlap with the evaluation data).  

I find the experimental evaluation to be useful enough for considering accepting this paper, but I will not strongly advocate for acceptance (Unless the authors can better explain why we need yet another benchmark.) 

### Supporting arguments for the reasons for the decision.

1. The paper compares existing representation learning methods in a very convincing way. Even though it misses some baselines (Meta-learning based representation learning), it compares supervised, self-supervised, semi-supervised, and generative methods comprehensively. Moreover, the hyper-parameter selection process for the benchmark is fully defined in a dataset agnostic way (It does not depend on the evaluation dataset). This is more important than it might seem -- fully specifying algorithms is important and currently rarely done in the research literature. 

I'm not aware of any existing work that compares various representation learning methods on a diverse benchmark fairly and comprehensively, and I'm sure I'll be using the results in this paper to support my claims in the future. As a result, I think this work is worth accepting just for its thorough evaluation of existing methods. 

2. However, the paper does not add much to discourse apart from the thorough evaluation. Benchmarks are already prevalent in representation learning, and VTAB doesn't focus on a single well-defined problem with-in representation learning. 

For example, dSprites is a benchmark to evaluate how well algorithms can disentangle the underlying data generation factors. An algorithm that significantly improves results on dSprites would, as a result, be better at learning disentangled representation. VTAB, on the other hand, is not trying to highlight any specific problem in representation. As a result, it's not clear what it means if an algorithm does better on VTAB (Apart from the vague notion that it is learning 'better' representations.) 

This problem is especially exacerbated by the fact that VTAB does not restrict representation learning to a fixed dataset. This means that a trivial way of improving performance is to just get more representative data (Which is not a bad way of improving performance, but not the way that requires scientific study). 

### Questions for Authors 

1. The paper makes a binary distinction between fine-tuning and linear evaluation when in-fact this is more nuanced. I suspect that the best performance would be achieved by feezing layers up to a point (And not just not freezing any layers or freezing everything and training a linear classifier). Did the authors explore other choices, and if not, why do they think the current binary distinction is enough. (From my perspective, it makes sense to freeze the initial layers which might be extracting more local/general features and adapt the later layers. The boundary point should be a hyper-parameter)

2. Currently, VTAB puts no restrictions on the representation learning dataset as long as the representation learning dataset does not over-lap with the evaluation dataset. This is problematic since an easier way of improving results on VTAB is to just collect more data/ data that is more representative of evaluation tasks. One could argue that the point of VTAB is to not use details of evaluation datasets in making any decisions for representation learning, however, it is hard to enforce this. For example, if I augment Imagenet with a medical dataset different from the ones in VTAB, is that fine or does that count as cheating?

I feel that a much better way of defining the benchmark would be to restrict representation learning to Imagenet. 

### Other minor comments 
I don't have many minor comments since the paper is very clear and well written. The author could consider augmenting their graphs with patterns in addition to color to make paper friendly for grey-scale printing and use a more colorblind-friendly color combination.  


### Post discussion update ###

One of my primary concerns was the benchmark didn't restrict representation learning to a fixed dataset. That made it trivial to improve performance: just train on a more representative dataset. However, as authors clarified, any dataset collected conditioned on the test set is against the spirit of VTAB. Moreover, discovering what kind of data improves representations is also a research problem, and by restricting to a fixed pre-training dataset, VTAB would not be suitable for such research directions. 

I agree that solving representation learning might not just be about finding a better algorithm, but a better algorithm coupled with the right data. In that sense, VTAB's decision to allow any data for pretraining makes sense. However, I'm also concerned that the benchmark might push people to (directly or indirectly) start training on data more representative of VTAB evaluation tasks, and the performance gains in the benchmark will not translate to better algorithms. However, I don't see how this can be avoided for an open-ended benchmark. Since the authors have agreed to include guidelines on how to use the benchmark without abusing it, letting the community police itself given the guidelines is not a bad idea. 

I can see VTAB as a benchmark for answering interesting questions (mentioned by authors in the response) such as:

1- How to incorporate large amounts of unlabelled data for representation learning. For example, can we learn better representations by self-supervised learning on a large corpus of data (Such as unlabelled youtube videos)
2- Can we generate data using simulators that can help us learn better representations? 

Finally, the authors agreed to add meta-data in their leaderboards (which they plan to release after the review period). The meta-data would allow us to compare methods at a more granular level, such as comparing different unsupervised learning methods. Overall, I think the benchmark and the leaderboards would act as a good source for comparing different unsupervised, semi-supervised, and supervised learning methods (and their combination) fairly. 

I'm also updating my score from a weak-acceptance to an acceptance. 
review_title: Official Blind Review #3

Meta-review: The authors present a new benchmark for evaluating a plethora of models on a variety of tasks. In terms of scores, the paper received a borderline rating, with two reviews being rather superficial unfortunately. The last reviewer was positive. The reviewers generally agreed that the benchmark is interesting and carries value, and the AC agrees. Authors certainly invested a significant effort in designing the benchmark and performing a detailed analysis over several tasks and methods. However, the effort seems more engineering in nature and insights are somewhat lacking. For an experimental paper, presenting the results is interesting yet not sufficient. A much more in-depth analysis and discuss would warrant a deeper understanding of the results and open directions for future work. This part is currently underwhelming. 
Decision: Reject