A Universal Representation Transformer Layer for Few-Shot Image Classification
Authors: Lu Liu
William L. Hamilton
Guodong Long
Jing Jiang
Hugo Larochelle
Venue: ICLR 2021 Poster (2021)

Abstract: Few-shot classification aims to recognize unseen classes when presented with only a small number of samples. We consider the problem of multi-domain few-shot image classification, where unseen classes and examples come from diverse data sources. This problem has seen growing interest and has inspired the development of benchmarks such as Meta-Dataset. A key challenge in this multi-domain setting is to effectively integrate the feature representations from the diverse set of training domains. Here, we propose a Universal Representation Transformer (URT) layer, that meta-learns to leverage universal features for few-shot classification by dynamically re-weighting and composing the most appropriate domain-specific representations. In experiments, we show that URT sets a new state-of-the-art result on Meta-Dataset. Specifically, it achieves top-performance on the highest number of data sources compared to competing methods. We analyze variants of URT and present a visualization of the attention score heatmaps that sheds light on how the model performs cross-domain generalization.

Published as a conference paper at ICLR 2021
A UNIVERSAL REPRESENTATION TRANSFORMER LAYER FOR FEW-SHOT IMAGE CLASSIFICATION
Lu Liu 1.2, Liu! William Hamilton' 1,3, Guodong Long", Jing Jiang,, Hugo Larochelle 1.4
1 Mila, 2 Australian Al Institute. UTS, McGill University, 4 Google Research, Brain Team Correspondence lu. 1iu. cs@icloud. com.
ABSTRACT
Few-shot classification aims to recognize unseen classes when presented with only a small number of samples. We consider the problem of multi-domain few-shot image classification, where unseen classes and examples come from diverse data sources. This problem has seen growing interest and has inspired the development of benchmarks such is to effectively integrate the feature representations from the diverse set of train- ing domains. Here, we propose Universal Representation Transformer (URT) layer, that meta-learns to leverage universal features for few-shot classification by dynamically re-weighting and composing the most appropriate domain-specific representations. In experiments, we show that URT sets a new state-of-the-ar result on Meta-Dataset. Specifically, it achieves top-performance on the highest number of data sources compared to competing methods. We analyze variants of URT and present a visualization of the attention score heatmaps that sheds light how the model performs cross-domain generalization. Our code is available at hutps://githu..com//ii
A key challenge in this multi-domain setting
12060//// 12601/URT
INTRODUCTION
Learning tasks from small data remains challenge for machine learning systems, which show no- ticeable gap compared to the ability of humans to understand new concepts from few examples. A promising direction to address this challenge is developing methods that are capable of performing transfer learning across the collective data of many tasks. Since machine learning systems generally improve with the availability of more data, natural assumption is that few-shot learning systems should benefit from leveraging data across many different tasks and domains-even if each individ- ual task has limited training data available.
This research direction is well captured by the problem of multi-domain few-shot classification. In this setting, training and test data spans number of different domains, each represented by different source dataset. A successful approach in this multi-domain setting must not only address the regular challenge of few-shot classification-iie the challenge of having only handful of examples per class. It must also discover how to leverage (or ignore) what is learned from different domains, achieving generalization and avoiding cross-domain interference.
Recently, Triantafillou et al. (2020) proposed a benchmark for multi-domain few-shot classifica- tion, Meta-Dataset and highlighted some of the challenges that current methods face when training data is heterogeneous. Crucially, they found that methods which trained on all available domains would normally obtain improved performance on some domains at the expense of others. Follow- ing on their work, progress has been made, which includes the design of adapted hyper-parameter optimization strategies (Saikia et al. 2020) and more flexible meta-learning algorithms (Requeimal et al. 2019) Most notable is SUR (Selecting Universal Representation) (Dvornik al. 2020)), method that relies on a so-called universal representation, extracting from a collection of pre-trained and domain-specific neural network backbones. SUR prescribes hand-crafted feature-sclection procedure to infer how to weight each backbone for each task at hand, and produces an adapted rep- resentation for each task. This was shown lead to some of the best performances on Meta- Sata-Datsset.
"This work was done while Lu Liu was a research intern with Mila.
1 Canada CIFAR AT Chair

---

Published as a conference paper at ICLR 2021
In SUR, the classification procedure for each task is fixed and not learned. Thus, except for the underlying universal representation, there is no transfer learning performed with regards to how classification rules inferred across tasks and domains. Yet, cross-domain generalization might be beneficial in that area as well, in particular when tasks have only few examples per class.
Present work. To explore this question, propose Universal Representation Transformer (URT) layer, which can effectively learn to transform a universal representation into task-adapted represen- tations. The URT layer is inspired from Transformer (Vaswani et al. 2017) and uses an attention mechanism to learn to retrieve or blend the appropriate backbones to use for each task. By training this layer across few-shot tasks from many domains, can support transfer across these tasks.
We show that our URT layer on top of universal representation's pre-trained backbones sets new ofate-of-the-ar performance on Matta-Datsee.. succeeds at outperforming SUR on 4 dataset sources without impairing accuracy on the others. This leads to top performance on 7 dataset sources when comparing to a set of competing methods. To interpret the strategy that URT learns to weigh the backbones from different domains, we visualize the attention scores for both seen and unseen domains and find that our model generates meaningful weights for the pre-trained domains. A com- prehensive analysis on variants and ablations of the URT layer is provided to show the importance of various components of URT, notably the number of attention heads.
2 FEW-SHOT CLASSIFICATION
2.1 PROBLEM SETTING
In this section, we will introduce the problem setting for few-shot classification and the formulation of meta-leamning for few-shot classification. Few-shot classification aims classify samples where only few examples are available for each class. We describe a few-shot learning classification task as the pair of examples, comprising of support set S to define the classification task and the query set Q of samples to be classified.
Meta-learning a technique that aims model the problem of few-shot classification learning to learn from instances of few-shot classification tasks. The most popular way to train a meta-learning model is with episodic training. Here, tasks T (Q. S) are sampled from a larger dataset by taking subsets of the dataset to build support set S and a query set Q for the task. A common approach is to sample N-way- K- -soo tasks, each time selecting random subset of N classes from the original dataset and choosing only K examples for each class to add to the support set S.
The meta-learning problem can then be formulated by the following optimization:
min (().))=((T) [C(S, (1S,Q.0)
((S,Q.9)
1
Σ - logr(ylz, - ( (1)
Q
(I..)~~
where p(T) is the distribution of tasks, θ are the parameters of the model and p(v x S; ()) is the probability assigned by the model label y of query example æ (given the support set S), and ((()) is an optional regularization term on the model parameters with factor X.
Conventional few-shot classification targets the setting of N-way-K-shot, where the number of classes and examples are fixed in each episode. Popular benchmarks following this approach include Omniglot (Lake et al. 2015) or benchmarks made of subsets of ImageNet, such as minilmageNet (Vinyals et al.] (0166) and lieredImageNet (Ren et al., 2018). In such benchmarks, the tasks for training cover set of classes that is disjoint from the classes in the test set of tasks. However, with the training and test sets tasks coming from single dataset/domain, the distribution of tasks found in either sets is similar and lacks variability, which may be unrealistic in practice.
It is in this context that Triantafillou et al. 2020) proposed Meta-Dataset as a further step towards large-scale, multi-domain few shot classification. Meta-Dataan -ataset includes ten datasets (domains), with eight of them available for training. Additionally, each task sampled in the benchmark varies in the number of classes N, with each class also varying in the number of shots K. As in all few-shot learning benchmarks, the classes used for training and testing do not overlap.
2

---

Published as a conference paper at ICLR 2021
2.2 BACKGROUND AND RELATED WORK
Meta-Learning A promising approach for few-shot classification is to use meta-learning to more directly train model learn perform few-shot classification, in an end-to-end way. The two most popular methods are Prototypical Networks (Snell et al. (017)) and Model Agnostic Meta-Learning (MAML) (Finn al. 2017). Triantafillou al. (2020) showed that prototypical networks and MAML could be combined by leveraging prototypes for the initialization of the output weights value in the inner loop. [Requeima et al. (2019) also proposed Conditional Neural Adaptive Processes (CNAPs) for few-shot classification, which can be seen as extending prototyp- ical networks with a more sophisticated architecture that allows for improved task adaptation. This architecture was later improved further by Bateni et al. (2020) with Simple CNAPS, leading to one of the current best methods on Meta-Dataset. Another line of work which leverages the idea of "transfer by fine-tuning" can be found in Appendix
Universal Representations In contrast, our work instead builds on that of Dvornik et al. (2020) and their method SUR (Selecting from Universal Representations), Bilen & Vedaldi (2017) intro- duced the term universal representation to refer to a representation that supports good performance in multiple domains. One proposal towards such representation is to train different neural net- works backbones separately on the data of each available domain, then simply to concatenate the representation learned by each. Another is to introduce some parameter sharing between the back- bones, by having single network conditioned on the domain of the provenance of each batch of training data (Rebuffi al., 2018). c.g. using Feature-wise Linear Modulate (FiLM) (Perez et al.) 2018). SUR proposes to leverage universal representation in few-shot learning tasks with a feature selection procedure that assigns different weights to each of the domain -ppeciicee subvectors of the universal representation. The objective is to assign high weights only to the domain-specifer repre- sentations that are specifically useful for each few-shot task at hand. The weights are inferred by optimizing a loss the support set that encourages high accuracy of a nearest-centroid classifier. As such, the method does not involve any meta-learning. a choice motivated by the concern that meta-learning may struggle in generalizing to domains that are dissimilar the training domains. SUR achieved some of the best performances on Meta-Dataset. However, a contribution of our work is to provide evidence that meta-leariing can actually be used replace SUR's hand-designed inference procedure and improve performance further.
Task Adaptive Representations Another line of work tries to retrieve task adaptive representations for each task. Task specific representations can be conditioned on a representation of the current task (Oreshkin et al. 2018: Wang et al., 2019). projected to another space (Yoon et al. 2019), or masked based on inter-class commonality and inter-class uniqueness (Li et al. 2019). While the rep- set of pretrained backbones and can be applied to more complicated multi-domain scenarios. [Wang & Hebert (2016) proposed to improve a CNN by adding extra layers and train it using unsupervised data while our contribution mainly lies in composing representations instead of improved CNN. Alet al. (2018) introduced a modular meta-learning method, which learns a repertoire of modules that serves as nodes to con- struct tree structure to solve a new robotic-related task. Comparatively, URT is one-for-all layer which doesn't 't need to construct different module structures for each task.
resentation extracted from URT is also task adaptive, it is adaptive
Transformer Networks Our meta-learning approach to leverage universal representations is in- spired directly from Transformer networks (Vaswani et al. 2017). Our model structure is inspired by the structure of the dot-product self-attention in the Transformer, which we adapted here multi- domain few-shot learning by designing appropriate parametrizations for queries, keys and values. Self-attention was explored in the single-domain training regime by YYe al. (2020); Liu et al. (2019b)a a 2020). however for different purpose, where each representation of individual examples in task support set is influenced by all other examples. Rather than using self-attention between individual examples in the support set, our model uses self-attention to select between different domain-specific backbones.
UNIVERSAL REPRESENTATION TRANSFORMER LAYER
In this section, we describe our proposed URT layer, which uses meta-learning episodic training to learn how to combine the -specff- backbones of a universal representation for any given few- shot learning classification task. URT layer can be built on top of any of pretrained backbones
3

---

Published as a conference paper at ICLR 2021
Uiniersal Rupressntattrr
Scaled Dot-procuct Attention
Samp represontii centtsotattioon transfomed URT
Query (5
Key (i
Fr)
Inferring adapted representation for task
Baskbonis
inprosentaaiion
slot
[r]
Scaled
Average
Dot product
Aftent on scones (Waxcapi
Attention scores
Dot-product
Ig)
<(S))
for task
Buckbonse
[n]
Scaled Dot product
Attertion scores (Ply agand
Averaee
Lnd
Attention score
Figure Illustaation of how single-head URT layer uses universal representation produce a task-specific representation. This example assumes the use of four backbones, with each color illustrating their Nomainn-specife sub-vector representation in the universal representation.
without further costly fine-tuning of the backbones. More details on how to train multiple domain- specific backbones can be found in Appendix C
Conceptually, the proposed model views the support set task as providing information on how to query and retrieve from the set {r,} of pre-trained backbones the most appropriate backbone to build an adapted representation o for the task.
We would like the model to support variety of strategies how to retrieve backbones. For ex- ample, it might be beneficial for the model to retrieve a single backbone from the set, especially the domain of the given task matches perfectly that of domain found in the training set. Alterna- tively, if some of the training domains benefit from much more training data than others, a better strategy might be attempt some cross-domain generalization towards the few-shot learning task by blending many backbones together, even if none matches the domain of the task perfectly.
This motivates works (Vaswani et al., 2017). For this reason, we refer to our model as Universal Repre- sentation Transformer (URT) layer. Additionally, since each class of the support set might re- quire a different strategy, we perform attention separately for each class and their support set S. = {z](x, y) and y = c).
use dot-product self-attention, inspired by layers of Transformer net-
3.1 SINGLE-HEAD URT LAYER
We start by describing an URT layer consisting of single attention head. An illustration of a single- head URT layer is shown in Figure Let (x) be the output vector of the backbone for domain We then write the universal representation
r(x) concat (r1 (x), ... T' (x)).
(2)
This representation provides natural starting point to obtain representation of a support set class. Specifically, we will note
r(S.)
Σ r())
15a
#ES-
as the representation for the set S... From this, we can describe the URT layer by defining the queries]] keys, the attention mechanism and output of the layer:
obtain a query through qe W + ba where we have learnable query linear transformation represented by matrix W and bias bg,
Queries q: For each class
Keys c For each domain i and class we define keys as ki.- W*+::SS)+bb* using learnable linear transformation W and b and where (S()) S/ Exes, ra(1)), using a similar notation as for r(S)).
Unable to avoid the unfortunate double usage of the term "query" due to conflicting conventions, we high- light the difference between the query Q of few-shot tasks and the queries qe of attention mechanism.
4

---

Published as a conference paper at ICLR 2021
Algorithm Training of URT layer
Input: Number of tasks Ttotals m pre-trained backbones;
1: for {l,..., Ttotal do
2: Sample few-shot task 7 with support set S and query set Q:
3: # Infer adapted representation for task from S
4: For each class, obtain representation using m pre-trained backbones as in Eq.
(3);
5:
Obtain attention scores using Eq. (45) for each head using support set S;
6: # Use adapted representation to predict labels in from support set S
7: Compute adapted representation of examples in S and Q as in Eq. (
8: Compute probabilities of label of examples in Q using Prototypical Network as in Eq. (
9: Compute loss as in Eq. (8) and perform gradient descent step on URT parameters θ;
end for
Attention scores as for regular Transformer layers, we use scaled dot-produce attention
cxp(B1,c) 0,scc Σx (x
9. ki,
Br,c
(4)
vi
where is the dimensionality of the keys and queries. Then, these per-class scores aggregated to obtain scores for the full support set by averaging
Ee Oz.c
(5)
N
Equipped with these attention scores, the URT layer can now produce adapted representation for the task (for the support and query set examples) by computing
((x) = Σ (x):(x).
(6)
As see, this approach has the flexibility of either selecting single domain-specinfe backbone (by assigning a; = for single domain) or blending different domains together (by having &; >> 0 for multiple backbones).
3.2 MULTI-HEAD URT LAYER
The URT layer described so far can only learn to retrieve a single backbone (or blending of back- bones). Yet, it might beneficial retrieve multiple different (blended) backbones, especially for few-shot task that would include many classes of varying complexity.
Thus, achieve such diversity in the adapted representation, we also consider URT layers with multiple heads, i.e. where each head corresponds to the calculation of Equation [aand oland each head has its own set of parameters (W b4, W b*). Denoting each head now as Φb, multi-head URT layer then produces as its output the concatenation of all of its heads:
((x) concat (Ø (x)...., (x)))
Empirically we found that the randomness in the initialization of head weights alone did not lead to uniqueness and being complimentary between the heads, inspired by Lin al. (2017), we add regularizer to avoid duplication of the attention scores:
?(@) |[AAT T 1)
2
(
where || || is the Frobenius norm of a matrix and A Raxm the matrix for attention scores, with A being the vector of all scores for head h. The identity matrix I regularizes each set of attention scores to be more focused so that multiple heads can attend different domain-specinf backbones.
3.3 TRAINING STRATEGY
We train representations produced by the URT layer by following the approach of Prototypical Networks (Snell et al. 2017), where the probability of label y for query example given the
5

---

Published as a conference paper at ICLR 2021
Test accuracy (mean=CC&%95) over 600 few-shot tasks. URT and the most recent methods, which are listed in the first column, are compared on Meta- Meta-Dataset (Triantafilluu et al. (2020), which are listed in the first row. The numbers bold have intersecting confidence intervals with the most accurate method.
Table 1:
ILVVR
Onnagglot
Aircraft
Textures
QuickDraw
VGGFlower
TrafficSigns
MSCOCO
avg. rank
Birds
Fungi
MAMI.
3.
1.
76420.7
12.41
64.1
19.71
13.51111
79.9
1.
29.4
ProtoNet
44.5:11.1
79.6.1.1
71.1109
67.01.0
65.2.0888
64.9099
40.3111
86.910.7
46.51.10
39.91111
5.7
ProtoMAMI.
1..
12
75.240.8
60
68.3408
42.0412
88.740.7
12441.1
41.74:111
CNAPs
50.8+1.11
91.7+05
83,7=06
73.6
59.5
74.7#08
50.2
88.900
16.5+11
39.4+.11
3,6
SUR
56.1::1.1
93.1+005
84.6=0.7
10.6+10
710-00
813306
642210
828=007
53,4100
50.1
SimpleCNAPS
56.51:111
91910.6
83.8007
76.11088
70.0108
78.31077
49.1112
91.3:066
59.2110
42.41:111
URT (Ours)
55.7.1
94.410.4
8..
0.6
76.3
71..10.7
8255
63.5.10
88.2.0.
51.111
52.2111
support set of task is modeled as:
ply = S:0) = exp(-d( P.))
>x exp( exp(-d(<(x) por))
corresponds to the centroid of class c, where d is a distance metric and pr: = 1/|Sa Ea ags, o(x)
referred to its prototype. We (negative) cosine similarity as the distance. The full training algorithm is presented in Algorithm
4 EXPERIMENTS
In this section, we seek to answer three key experimental questions:
Q1 How does URT compare with previous state-of-the-art on Meta-Dataset for multi-domain few- shot classification?
Q2 Do the URT attention heads generate interpretable and meaningful attention scores?
Q3 Does the URT layer provide consistent benefits, even when pre-trained backbones trained in different ways?
In addition, we investigate architectural choices made, such as our models for keys/queries and their regularization, and study their contribution achieving strong performance with URT.
DATASETS AND SETUP
We test methods the large-scale few-shot learning benchmark Meta-Dataset (Triantafillou ctt al., (020). consists of ten datasets with various data distributions across different domains, including natural images (Birds, Fungi, VGG Flower), hand-written characters (Omniglot, Quick Draw), and human created objects (Traffic Signs, Aireraft). Among the ten datasets, eight provide data that can be used during either training, validation and testing (with each class assigned to only one of those sets), while two datasets are solely used for testing. Following Bateni et al. 22000; Requeima al. (2019), we also report results MNIST (LeCun al. 1998). CIFARTI and CIFARIO0 (Krizhevsky et al. 2009) as additional unseen test datasets. Following Triantafillou al. (2020), few-shot tasks are sampled with varying number of classes N, varying number of shots K and class imbalance. The performance is reported as the average accuracy over 600 sampled tasks.
More details of Meta-Dataset can be found in Triantafillou et al. (202)).
The domain-specific backbones are pre-trained following the setup in (Dvornik et al. (020). Then, we freeze the backbone and train the URT layer for 10,000 episodes, with an initial learning rate of 0.01 and cosine learning rate scheduler. Following Chen et al. (2020), the training episodes have 50% probability coming from the ImageNet data source. Since different pre-trae pre-trained backbones may produce representations with different vector norms, normalize the outputs of the backbones in Dvornik et al. (2020). URT is trained with parameter weight decay of le-5 and with a regularization factor À 0.1. The number of heads (H in Equation 7),, is set to 2 and the dimension of the keys and

---

Published as a conference paper at ICLR 2021
Source domain pretrained bacibone
Source domain for pretranedd backbone
ILSVRC
Omniglet
Aironaft
Bids
Testuen
Qrack Draw
Fungl
KGerlowwr
ILSVRC
Omnigiat
Alrrntt
Blrds
Tecture
Crack
Fungl
0.1
Orniglo:
-.00 0.0 0.0 0.1 0.0
0.0
0.0
Omnigiat
0.0 0.0 0.0 0.0 0.1
0.88
0.1
0.0
0.0
0.8
0.0
0.0
0.0
0.0
attention score heed
Arcratt
0.0 0.0
1.0
0.0
0.0
0.0
0.0
0.0
attertion head
Tost Toot took domain
Birds
0.9 0.0
0.0
0.0
0.0
0.0
0.0
0.0
1066
Test book domain
Birds 0.0
0.0
0.0
0.9
0.0
0.0
0.0
0.0
Textures
0.8
0.0
0.1
0.0
0.0
0.1
Textures 0.1 0.0
0.0
0.0
0.8
0.0
0.1
QuickDraw
0.0
0.4
QrickDraw
0.0
0.4
Fungi
Fungi
0.0
VGGFlower
0.9
0.0
0.0
0.0
0.0
0.0
0.0
VGGFlower
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.2
TraffiSSigne 0.1 0.0
0.0
0.2
0.0
0.3
0.0
TrafliSSggn 0.5 0.1
0.0
0.2
0.1
0.1
0.0
0.1
MSCOCO - 0.3
0.0
0.0
0.4
0.0
0.1
0.2
0.0
MSCDCO 0.6 0.0
0.1
0.2
0.1
0.0
0.0
0.0
Figure 2: Average attention scores generated by URT with two heads. Rows correspond to the domain of the test tasks and the columns correspond to the pre-trained backbones r2( trained on the eight training domains.
queries (1 in Equation 4 is set to 1024. We choose the of the validation set. Details of the blyper-parameee selection and how the performance is influenced by them are outlined Section 4.5 We find that the bottleneck of training URT is extracting features from CNN. Since freeze the CNN when training the URT, find dumping the extracted feature episodes can significantly speed up the training procedure from days to around 2 hours.
based on the performance
4.2 COMPARISON WITH PREVIOUS APPROACHES
Table I presents a comparison of URT with SUR, as well as other baselines based on transfer learn- ing by fine-tuning (Saikia al., 2020 meta-leaming (Prototypical Networks (Snell al.) (017))
first-order MAML (Finn et al. 2017), ProtoMAML (Triantafillou et al. 2020), CNAPs (Requeima
al. 2019g) and Simple CNaeeiBateni et al. 2020).
We observe in Table T that URT establishes a new state-of-the-art on Meta-Dataset, by achieving the of the 10 dataset sources. When comparing to its predecessor, URT out- performs SUR on 4 datasets without compromising performance on others, which is challenging to achieve in the multi-domain setting. Of note, the average inference time for URT is 0.04 second per task, compared to 0.43 for SUR, on single V100. Thus, getting rid of the optimization procedure for every episode with our meta-trained URT layer also significantly increases the latency, by more than 10x. More results on additional datasets can be found in Appendix A
top performance on
4.3 INTERPRETING AND VISUALIZING ATTENTION BY URT
To better understand how the URT model of Section uses its two heads to build adapted repre- sentations, we visualize the attention scores produced on the test tasks of Meta-Dataset in Figure Ø
The blue (first head) and orange (second head) heatmaps summarize the values of the attention scores (Equation 55). averaged across several tasks for each test domain. Specifically, the element on row and column i is the averaged attention scores computed on test set domain t for the backbone from domain i. Note that the last two rows are the two unseen domain datasets. We found that for datasets from the seen domains, i.e. the first eight rows, one head (right, orange) consistently puts most of its weight on the backbone pre-trained on the same domain, while the other head (left, blue) learns relatively smoother weight distributions that blend other related domains. For unseen datasets, the right head puts half of its weight on ImageNet and the left head learned to blend the representations from four backbones.
7

---

Published as a conference paper at ICLR 2021
Table 2: Test accuracy (mean=CCI%95) over 600 few-shot tasks. All methods use parametric net- work family (pf) backbones.
SUR-pf
VS. $5.5
URT-pf
ILSRCC
56.0 + 1.1
Omniglot
90.0
90.2
Aircraft
7.87
7.7
Birds
7.99
7..5
Textures
73.5 7275 7.77
Quick Draw
7.78
Fungi
49.8
48.1
VGG Flower
9..
91,9 0.5
Traffic Signs
5.8
52.0 1.4
MSCOCO
1.1
52.1 1.111.0
MNIST
9.4
93.9
CIFARIO
6.84
6..1
CIFARI00
57.1
57.3 1.0
4.4 URT USING FILM MODULATED BACKBONES
As additional evidence of the benefit of URT on universal representations, we also present exper- iments based on different set of backbone architectures. Following SUR (Dvornik et al. 2020). we consider the backbones from parametric network family, obtained by training a base backbone one dataset (ILSVRC) and then learning separate FiLM layers (Perez al.) (2018) for each other dataset, to modulate the backbone so it is adapted to the other domains. These backbones collec- tively have only 0.5% more parameters than single backbone. More details of the backbones can be found in Appendix C
A comparison between SUR and URT using these backbones (referred SUR-pf and URT-pf) is presented in Table 2 Once again, URT can improve the performance on VGG Flower without sacrificing performance others.
4.5 HYPER- PARAMETER AND ABLATION STUDIES
We analyze the importance of the various components of URT's attention mechanism structure and training strategy in Table 3 First we analyze the importance of using the support set to model queries and/or keys. To this end, consider setting the matrices W W of the query key linear transformation to O, which only leaves the bias term. We found that the support set representation is most crucial for building the keys (row w /o W in the table) and has minor benefits for queries (row w/o WT) in the table. This observation is possibly related to the success of nitention-based models with learnable constant queries (Liu et al. 2016 Lin et al. 2017). We also found that adding a reg- ularizer ?(+) in Equation 8 is important for some datasets, specifically VGG Flower and Birds.
Table 3: Meta-Dataset performance variation on ablations of elements of the URT layer.
ILSVRC Omniglot Aireraft Birds Textures Draw Fungi Flower Signs MSCOCO
w
w/o Wa
+0.2
-0.2
-0.6
-0.1
-0.3
-0.2
0.0
-0.2
-0.8
-0.1
W
W
-14.2
-2.8
-10.7
-18.1
-7.6
-9.3
-22.4
-3.6
-0.26
-10.9
w
w/o rS.))
-14.2
-2.8
-10.7
-18.1
-7.6
-9,2
-22,4
-3,6
-0.26
-10.9
w w/o ((O)
0.0
-0.9
-0.4
-3.3
-1.2
-0.2
+0.3
-9.0
-2.0
0.0
An important hyper-parameter in URT is the number of heads H. We chose this harem-- based on the performance on validation set of tasks in Meta-Dataset. In Table 4 we show the validation performance of URT for varying number of heads. As suggested by Triantafillou et al. (2020), we considered looking at the rank the performance achieved by each choice of H for each validation domains, and taking the average across domains validation metric. However,
8

---

Published as a conference paper at ICLR 2021
since the performances when using two to four heads are similar and yield the same average rank, we instead simply consider the average accuracy as the selection criteria.
Table 4: Validation performance on Meta-Datasee using different number of heads
1
3
5
8
Average Accuracy
74.605
77.145
76.943
76.984
76.602
75.906
75.454
74.473
Average Rank
2.875
1.000
1.000
1.000
2.250
2.250
2.25
2.50
In general, we observe large jump in performance when using multiple heads instead of just one. However, since the number of heads controls the capacity, predictably we also observe that having too many heads leads to overfitting.
CONCLUSION
We proposed the URT layer to effectively integrate representations from multiple domains and demonstrated improved performance in multi-domain few-shot classification. Notably, our URT approach was able its predecessor (SUR) while also being 10x more efficient at inference. This work suggests that combining meta-learning with pre-trained universal representations a promising direction for new few-shot learning methods. Specifically, we hope that future work can investigate the design of richer forms of universal representations that go beyond simply pre-training a single backbone for each domain, and developing meta-learners adapted to those settings.
new state-of-the-art Meta-Dataset, and never performs worse than
REFERENCES
Ferran Alet, Tomás Lozano-Péree and Leslie P Kaelbling. Modular meta-learning. In Conference on Robot Learning (CoRL), 2018.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv: 2017.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Liu, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations (ICLR), 2019.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning, arXiv preprint T0t::20003.043900 2020.
Guneet Singh Dhillon, Pratik Chaudhari. Avinash Ravichandran, and Stefano Soatto. baseline for few-shot image classification. In International Conference on Learning Representations (ICLR), 2020.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a universal representation for few-shot classification. arXiv preprint arXiv:2003.09337 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In The International Conference Machine Learning (ICML), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

---

Published as a conference paper at ICLR 2021
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-leved concept learning through probabilistic program induction. Science, 350(6266):1332-133333 1338, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) 86(11):2278-2324, 1998.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2019.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint 1rXiv:1703.03130, 2017.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype propaga- tion networks (PPN) for weakly-supervised few-shot learning on category graph. In International Joint Conferences on Artificial Intelligence (IJCAI), 2019a
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Learning to propagate for graph meta-learning. In Neural Information Processing Systems (NeurIPS), 2019b.
Lu Liu, Tianyi Zhou, Guodong Long. Jing Jiang, and Chengqi Zhang. Attribute propagation network for graph zero-shot learning. In AAAI Conference on Arlificial Intelligence (AAAT), 2020.
Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. Learning natural language inference using bidirectional Istm model and inner-attention. arXiv preprint arXiv: 09090, 2016.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In The Conference Neural Information Processing Systems (NeurIPS), 2018.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. AAAI Conference on Artificial Intelligence (AAAT), 2018.
Sylvestre-Alvee Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In International Conference on Learning Representations (ICLR), 2018
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In The Conference on Neural Information Processing Systems (NeurIPS), pp. 7957-7968, 2019.
Tonmoy Saikia, Thomas Brox, and Cordelia Schmid. Optimized generic feature learning for few- shot classification across domains. arXiv preprint arXiv:2001.07926. 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In The Conference on Neural Information Processing Systems (NeurIPS), 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Aitrre-Antoine Manzagol, al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations (ICLR), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In The Conference Neural Information Processing Systems (NeurIPS), 2017.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In The Conference Neural Information Processing Systems (NeurIPS). 2016.
10

---

Published as a conference paper at ICLR 2021
Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, and Joseph E Gonzalez. Tafe-net: Task-aware feature embeddings for low shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yu-Xiong Wang and Martial Hebert. Learning from small sample sets by combining unsuper- vised meta-training with cnns. In The Conference on Neural Information Processing Systems (NeurIPS), 2016.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. The International Conference on Machine Learning (ICML), 2019.
A EXPERIMENTS ON MORE DATASETS
We also report performances on the MNIST, CIFAR-10 and CIFAR-100 dataset sources in Table and compare with the subset of methods that have reported these datasets. There, URT neither improves nor gets worse performance than SUR, yeilding top performance on the MNIST domain but not on the CIFAR-IOCIIFA--I0 domain, on which Simple CNAPS has the best performance.
Table 5: Test performance (mean+C1%95) over 600 few-shot tasks on additional datasets.
MNIST
CIFARIO
CIFAR100 avg. rank
CNAPs
92.7 0.4 61.5
50.1 + 1.0
4.7
TaskNorm
92.3 - 0.4 69.3 0.8
54.6
1.1
3.3
SUR
94.3 0.4 0.9
56.6
2.3
SimpleCNAPS
93.9 = 0.4 74.3
60.5 1.0
1.7
URT (Ours)
94.8
0.4 67.3
0.8
56.9 + 1.0
2.0
B MORE RELATED WORKS
Transfer by fine-tuning simple and effective method for few-shot classification is to perform transfer learning by first learning neural network classifier on all data available for training and using its representation to initialize and then fine-tune neural networks on the few-shot classification tasks found at test time (Chen et al., 2019 Triantafillou et al. 2020; Dhillon et al.) [2200] Saikia al.) 2020). Specifically, Saikia et al. (2020) have shown that competitive performance can be reached us- ing strong hyper-pane paremmrrreetter optimization method applied carefully designed validation metric appropriate for few-shot learning.
C STRUCTURES AND TRAINING STRATEGY OF BACKBONES
Our URT layer can be built on top of a set of pretrained backbones. The structures of the backbones follow the approach in Dvornik et al. (022)). For Table D we use the ResNet18 architecture (He et al. 2016) for the backbones, where a separate backbone is pretrained on each domain separately using the corresponding training data in meta-dataset. The training domains include ImageNet, Omniglot, Aircraft, CU-Birds, Textures, Quick Draw, Fungi and VGG-Flower. For the parametric network family in Table 2, a base ResNet18 is first trained on ImageNet. Then small number of modulating parameters data, while the rest of the base backbone's 's weights stay fixed. Specifically, FiLM feature modu- lation (Perez et al. 2018) is used. This type of parametric network family thus allows for much reduced number of learnable parameters, by reusing the weights from the base network. In exper- iments, the pretrained backbones released by Dvornik et al. 2020 for both cases, without
trained the other domains, using their domain-specific training

---

Published as a conference paper at ICLR 2021
any further finetuning. End-to-end training of set of backbones and the URT layers requires unaf- fordable computational cost, so we fix the pretrained backbones and only train the URT layer.
Implementation details for training backbones The training details of the backbones come from Dvornik et al. (220). For optimization, SGD with momentum was used, using cosine learning rate annealing. Since the datasets come from different domains, the starting learning rate, the max- imum number of training iterations and annealing frequency are set individually for each dataset. Data augmentation applied and constant weight decay of 7 10 search over batch size in [8, 16, 32, 64] was run and the one that maximizes accuracy on the valida- tion set was picked. For the parametric network family, the base ResNet18 trained on ImageNet is the same. For other backbones, cosine annealing as learning rate policy is also used, weight decay and data augmentation employed as above. Please refer to Table and Table in the original SUR paper (Dvornik et al. (020) for the specific values of the hyperparameters for the individual feature networks and the parametric network family, respectively.
(0-4pp is set. For each dataset, grid
D RESULTS ON TRAFFIC SIGNS
The shuffle buffer bug described in meta-dataset issue #54 https://gitubb.oom/ggoo vescarcb/meta-dattseeiissssss
has been propagated to previous works such CNAPs (Requeima) at. 2019). The results for the Traffic Signs dataset are considerably worse after fixing this bug. For instance, URT degrades from 69.4 69.4 to 1..1 Please visit the official meta-dataset GitHub repo for more details. The main paper shows the corrected results (for URT and competing ap- proaches). For completeness, we also provide the tables they in the initial version of this paper for your reference in this Appendix. Both sets of results overall support the advantageous performance of URT over previous work.
Table 6: Test accuracy (Iean=CC1%95)) over 600 few-shot tasks. URT and the most recent methods, which are listed in the first column, are compared on Meta-Dataset (Triantafilluu et al. 2020), which listed in the first row. The numbers bold have intersecting confidence intervals with the most accurate method.
ILVVRC
Ohnaggot
Aircraft
Textures
QuickDraw
VGGFlower
TraffteSigns
MSCOCO
agg rank
Birds
Fungi
MAMI.
37.8:1100
1.
76410.7
62.41.1
64.1108
19.7
33.51111
79.9
1.
29.4
8.0
ProtoNet
1..5
79.61:11
71.1
67.0
65.2108
649409
40.3
86.9-0.7
46.54:00
39.9
7,3
ProtoMAML.
46.5.111
82.7110
75.2008
60.91.0
68.3108
42.0.1
88.710.7
52411.
CNAPs
52.3+100
08.4
80.5206
72.2
58.3
72.54088
47.410
86.0=0.5
60.2+0.9
42.641.11
5.1
BOHB-E
55.4::1.1
77.51111
60.9009
73.6408
7284007
61.2409
44.5+111
90.6=0.6
57.5+10
51.9+110
TaskNorm
50.6::1.1
90.7::0.6
83.8006
74.6208
62.11:077
7488:077
48.7+:10
89.6
67.0
43.44:10
3,8
SUR
56.3111.1
93.1100.5
8..41077
714110
715108
813106
63.1:110
82.8
704108
52.41.1
2.5
SimpleCNAPS
1.
91.71066
82.410.7
74.900
67.81088
77.710.7
4..91..0
9..7
73.5
46.2411.1
URT (Ours)
55.7+1.0
94.4+0.4
858:2066 76.3:10.8 7188:0.7 82.5+066
63.5+110
88.2 88.2=0.6
69.40088
52.2+1111
1.6
Table 7: Test accuracy (Iean::C1%95)) over 600 few-shot tasks. All methods use parametric net- work family (pf) backbones.
SUR-pf
URT--pf
ILSVRC
56.4
55.5 1.1
Omniglot
8.8
90.2 0.6
Aircruft
0.8
0.7
Birds
77.5 7..4
Textures
7..1
73.5 0.7
Quick Draw
7.77
0.7
Fungi
4.9
48.1 1.91
VGG Flower
0.5
91.9 0.5
Traffic Signs
65.1
67.5
0.8
MSCOCO
52.1 1.0
1.0
MNIST
9.
93,9
CIFARIO
66.4
66.1 0.8
CIFARI00
57.1 2 1.0
1.0
12


Reviews:
[{"review_id": "vChIuirDQAs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a meta-trained Universal Representation Transformer (URT) layer, which learn to dynamically re-weight domain-specific representation for classifying given target images. The evaluation on Meta-Dataset shows proposed method achieved competitive performance against compared baselines.\n\nStrengths:\n+ The proposed URT, inspired by self-attention and Transformer network, learned to dynamically  re0weight domain-specific representation for classifying images on an unseen target domain.\n+ The proposed URT  can be used as a single-head URT layer  or a multi-head URT layer, where a regularizer (i.e., eqn (8)) is added to avoid duplicate attention scores in different URT layer.\n\nWeakness: \n- The idea of mixing pre-trained representation to a universal representation was first proposed in SUR. Compare to SUR, this work meta-trained a attention module for the mixing process (as compared to handcraft approach in SUR). While  this work shows good improvement over SUR with a learnable URT, I argue this work does not have sufficient theoretical or algorithmic contributions. \n- The evaluation is only conducted on Meta-Dataset. Has the method evaluated on other widely-used domain generalization benchmarks, such as PACS and Office-Home.\n\nMinor comments: \n- Line 1 of the abstract \"Few-shot classification aims to recognize unseen classes when presented with only a small number of samples\" is incorrect. The statement is missing key information and misleading. There is an established literature on few-shot classification that learn a classifier with less samples (e.g, 5 sample per class) and the target is the learned classes. The author is probably refer to the few-shot domain generalization where the target class are from an unseen domain. But \"unseen classes\" is a fairly big statement to make in this case. \n- Figure 1 shows the attention scores generate by URT.  For test domain ILSVRC, it assign high scores to Birds, followed by Fungi then ILSVRC.  Then, for test domain Birds and Textures, both assign high score to ILSVRC. I am curious why textures domain required higher attention to ILSVRC, but ILSVRC does not assign any score (or too low) to Textures. Birds' score seems to be more consistent. In other words, while URT attention heads generate interpretable score, how can we reason on the score and generate more insights on why a particular source domain is selected. \n\n------------------\nPost Rebuttal:\nThe authors' response has addressed my concerns. Based on the response and other reviewers' comment, I have updated my rating for this work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X5fspbD-lNp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The review is brief because of time pressure. However, I have gone through the paper carefully.\nMotivation\nThe paper is well motivated. It is keenly aware of previous work in the field and establishes its advancement of the state of the art clearly. It reviews past work in meta-learning as well as universal representations and transformers. I do have a suggestion for improvement, which is to consider the lifelong learning literature where reinforcement learning based methods have been developed for learning tasks over a lifetime. While reinforcement learning is a qualitatively different approach, lifelong learning requires the kind of adaptation to changes in tasks that the authors are addressing in their paper. It might behoove them to look at that literature and make a critical assessment with respect to their work. I don't see this as a weakness of the paper at all.\n\nApproach\nThe approach is clearly described and is technically sound. It essentially sets up an optimization across multiple domain specific backbones to solve the multi-task problem. Such an approach has the advantage of modular design although I am curious to know if the authors have any opinions on how to introduce a new backbone into their system without having to retrain the entire system end to end. Or just in general how they would introduce a new domain specific backbone.\nThe optimization is clearly described and convincing. \nResults\nThe results are convincing. They are at par or better than the state of the art. They are carried out on datasets well accepted by the community.\nQuality, Clarity, Originality and Significance\nClarity - The paper is extremely well written. There are typos for example Representation is misspelled (misspelt). Those can be easily removed with a single editing pass. The paper motivates its approach well and describes the approach systematically. The results are presented convincingly and clearly. I would say the clarity of the paper is high.\nQuality, Originality and Significance - The idea presented here is certainly novel in its details. The overall idea of using multiple domains to compensate for data-scarcity in certain domains is not new, but realizing that in a mostly better than the state of the art manner is a challenge that the authors address successfully. The overall proposal is a small but good idea that leads to good results. I would therefore say that the paper has good quality, significance and originality.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MP7SSSxtuEr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper presents a method for tackling multi-domain few-shot image classification problem where it obtains a task-adapted representation by weighing representations from pretrained domain-specific backbones according to the support set at hand. The desirable property of this framework is that the model can leverage information from other domains to make predictions. The effectiveness of Universal Representations have been discussed in the past work - SUR [1], and this work builds on top of it and introduces a learnable component (self-attention), and showed the improvement both quantitatively and qualitatively.\n\n\nStrengths\n- The paper is well-written\n- The hypothesis is clearly conveyed, tested and is interpretable as seen from the attention weights\n- The model improves over the results of the past works that were based on conditioning backbones using FiLM layers - CNAPs [2], Simple CNAPS [3]. While these past works have used additional modules such as a small CNN set encoder to encode task-representation, FiLM layers for conditioning; the simplicity and effectiveness of this model is appealing\n\n\n\nWeaknesses\n\nI have a high-level comment.\n\n- Domain mixing during training:\n    - If I recall correctly, the way sampling works in Meta-dataset is that a dataset domain is picked and then a task is sampled. Is there a way to try mixing domains in a task? I guess then the class-specific attention scores would vary a lot among classes (because some classes would prefer a different backbone that the other classes). So task-adapted representations would change to class-specific representations. And the only change in eq 9 and the expression of $p_c$ would be to replace $\\phi(x)$ for a query image $x$ to $\\phi_c(x)$\n    - I don’t know if the above makes sense, but this will allow you to generalize to any real-world setting, and will also allow similar classes in different datasets to share information. Right now, the model is good at figuring out what domain does the task come from and find an appropriate mix of backbone for that task, however, what if it’s geared to do that for classes?\n\n\nMinor concerns (suggestions, typos, etc.)\n- Section 3.1\n    - Mention dimensionality of weights and representations\n\n\nPreliminary Rating and its justification\n- I don’t see any glaring faults in this paper so I recommend accept.\n\n\n[1] Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a universal representation for few-shot classification. arXiv preprint arXiv:2003.09338, 2020\n\n[2] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In The Conference on Neural Information Processing Systems (NeurIPS), pp. 7957–7968, 2019\n\n[3] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "INP0MSRrOsx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper addresses the problem of multi-domain few-shot image classification (where unseen classes and examples come from diverse data sources), and proposes a Universal Representation Transformer (URT) layer, which learns to transform a universal representation into task-adapted representations. The method proposed builds on top of SUR [Dvornik et al 2020], where a universal representation is extracted from the outputs of a collection of pre-trained and domain-specific backbones and a selection procedure infers how to weight each backbone for a given task at hand. While SUR inferred those weights by optimising a loss on the support set (the few examples provided in a task), the authors in this paper introduce an attention-based layer (inspired by Vaswani et al Transformer) that learns to weight the appropriate backbones for each task. This layer has the main advantage that it can be learned across few-shot tasks from many domains so it can support transfer across these tasks.\n\n\n## Strengths\n \n- The method and contributions are very well motivated and introduced. The paper is also very well written and very well presented. I also think that this new proposed URT layer is a very interesting contribution, and acknowledged its novelty for this specific task.\n\n- The experimental section is good, which includes comparison with other state-of-the-art methods and an ablation study that analyses the contribution of the different components of the proposed approach. I find especially interesting section 4.3, where the attention scores produced by the network are visualised on the test tasks, which gives a better understanding of how this URT layer works.\n\n\n## Weaknesses\n\n- Architecturally, URT and SUR are pretty much identical, the only difference and novelty being the way the weights for the different backbones  are computed. This might affect the paper’s novelty impact.\n\n- It would have been interesting to see how does SUR compare to URT with a single head, specially since the performance gap is quite significative from 1 to 2 layers as shown in Table 4. First, because it would give a deeper insight about the contribution of the different components of URT (attention layer vs multi-head). Second, because to me that’d be a bit more fair comparison between SUR and URT given that SUR only uses a single representation head: two heads means double dimensionality of the representation (from Eq 7, where representations are concatenated), and multi-head could also be applied to SUR using a similar approach (Eq 8).\n\n\nMinor comments:\n\n- It is not clear to me the claim done by the authors that SUR follows “hand-crafted feature-selection procedure” and that this procedure “is fixed and not learned”. If I understood correctly, SUR infers those weights by optimising a loss on the support set. While URT has a clear advantage since it doesn’t need to optimise on the support set, at the end of the day both infer those weights from the data, so how is it that SUR is hand-crafted? Apologies if I’m missing something, but I would like the authors if they could elaborate on this.\n\n\n## Recommendation\n\nEven though the method proposed doesn't differ too much from SUR, since the main difference is the way these weights are inferred, I still think that the new URT layer is an interesting contribution, and that paper brings enough novelty. For this reason, I’m initially leaning towards accepting the paper. However, I would like the authors to address my last two comments in the weakness section.\n\n## After rebuttal\n\nThe authors have addressed my main concerns and I've decided to raise my score from 6 to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4PLoNmDS5Mc", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nFew-shot learning on meta-dataset is challenging due to the domain gap between train and validation. In order to bridge this gap, the authors present a model that learns to combine domain-specific representations to generalize to new domains. This combination is done with a transformer model that pays attention to the features extracted from domain-specific backbones. The authors demonstrate empirically that their model attains comparable performance to previous state-of-the-art at higher efficiency and include ablation results to test their model components.\n\nOverall Review\n=============\nThe proposed method is sound and relevant for the research community. Although it lies towards the application side (i.e. a transformer on top of pre-trained features) and it has some weak points (see weaknesses), I still think that its simplicity will make it impactful. Thus, once the weaknesses are addressed I will happily raise my score.\n\nStrengths\n========\n* The proposed method is simple and works well.\n* The authors provide code and ablation experiments.\n* The text is well-written and easy to follow.\n\nWeaknesses\n==========\n* In understand that the model is more efficient because it does not need gradient descent at test time. Is that the case? if it is, could you include this information in the paper for completeness?\n* Given that the proposed model is an efficient version of SUR, there are some questions that naturally come to the reader that are not answered in the current submission. For instance, how do the attention coefficients of URT compare with the coefficients learned by URT? Why does URT perform better on the held-out data? What is the difference in training time?\n\n-------------------------------------\nAfter Rebuttal\n============\nMy main concerns were about the similarity between SUR and URT and the lack of detail in their comparison. I also asked for a clarification on the efficiency of the method.\n\nOn the first concern, they partially address it with the Coefficient characteristics, I say partially because I would have liked a more in-depth comparison of the characteristics, but technically they have addressed my question. For the second one, they now provide the training time, and the testing time could be found in Section 4.2.\n\nOverall, even though I still think that this work lies in the application side, it is interesting enough to be published at ICLR, so I have accordingly raised my score.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of multi-domain few-shot image classification and proposes a Universal Representation Transformer (URT) layer, which leverages universal features by dynamically re-weighting and composing the most appropriate domain-specific representations in a meta-learning way. The paper extends the prior work of SUR [Dvornik et al 2020] by using meta-learning and avoiding additional training during test phase. The experimental results show improvements over SUR in both accuracy (not always significant on some datasets though) and inference efficiency. Overall, the paper is well written with sufficient contributions. After the author's rebuttal and revision, reviewers generally agree the paper can be accepted. I recommend to Accept (Poster). 
Decision: Accept (Poster)