Improving Few-Shot Visual Classification with Unlabelled Examples
Authors: Peyman Bateni
Jarred Barber
Jan-Willem van de Meent
Frank Wood
Venue:  (2021)
Keywords: Meta-learning
Few-shot image classification
Transductive few-shot learning

Abstract: We propose a transductive meta-learning method that uses unlabelled instances to improve few-shot image classification performance. Our approach combines a regularized Mahalanobis-distance-based soft k-means clustering procedure with a modified state of the art neural adaptive feature extractor to achieve improved test-time classification accuracy using unlabelled data. We evaluate our method on transductive few-shot learning tasks, in which the goal is to jointly predict labels for query (test) examples given a set of support (training) examples. We achieve new state of the art performance on the Meta-Dataset and the mini-ImageNet and tiered-ImageNet benchmarks.

Under review as a conference paper at ICLR 2021
IMPROVING FEW-SHOT VISUAL CLASSIFICATION WITH UNLABELLED EXAMPLES
Anonymous authors Paper under double-bling review
ABSTRACT
We propose a transductive meta-learning method that uses unlabelled instances to improve few-shot image classification performance. Our approach combines a regularized
balalaobbb-diittnnee distance-based soft k-means clustering procedure with a modified state of the art neural adaptive feature extractor to achieve improved test-time classification accuracy using unlabelled data. We evaluate our method on transductive few-shot learning tasks, which the goal is to jointly predict labels for query (test) examples given set of support (training) examples. We achieve new state of the art performance on the tatt--Dtaset and the mini-ImageNoc and ImageNet benchmarks.
INTRODUCTION
Deep learning has revolutionized visual classification, enabled in part by the development of large and diverse sets of curated training data (Szegedy al. 2014 He ct al. 2015 Krizhevsky et al. 2017
Simonyan & Zisserman 2014 [oonnm et al. 2017). However, in many image classification settings, millions of labelled examples are not available; therefore, techniques that can achieve sufficient classification performance with few labels are required. This has motivated research on few-shot learning (Feyjie al. 2020 Wang & Yao 2019 Wang et al., 2019 Bellet et al. 2013), which seeks to develop methods for developing classifiers with much smaller datasets. Given few labelled "support" images per class, few-shot image classifier is expected to produce labels for given set of unlabelled "query" images. Typical approaches to few-shot learning adapt a base classifier network to a new support set through various means, such as learning new class embeddings (Snell et al. 2017] Vinyals et al.) 2016 [Sung et al. 2018), amortized (Requeima et al. 2019 Oreshkin et al. 2018) or
iterative (Yosinski et al. 2014) partial adaptation of the feature extractor, and complete fine-tuning of the entire network end-to-end (Ravi & Larochelle 2017 Finn et al. 2017).
In addition to the standard fully supervised setting, techniques have been developed to exploit additional unlabeled support data (semi-supervision) (Ren et al. (018)) well information present in the query set (transduction) (Liu et al. 2018; Kim et al. 2019). In our work, we focus on the transductive paradigm, where the entire query set is labeled at the same time. This allows exploit the additional unlabeled data, with the hopes of improving classification performance. Existing transductive sew-shot classifiers rely on label propagation from labelled to unlabelled examples in the feature space through either k-means clustering with Euclidean distance (Ren et al. 2018) or
message passing in graph convolutional networks (Liu al. 2018 Kim et al. 2019).
Since few-shot learning requires handling a varying number of classes, an important architectural choice is the final feature class mapping. Previous methods have used the Euclidean distance (Ren et al., 2018), the absolute difference (Koch et al., 2215).. cosine similarity (Vinyals et al.
2016), linear classification (Finn et al. 2017 Requeima et al. 2019) or additional neural network layers (Kim et al. 2019, Sung et al. 2018). 8). Bateni et al. (2020) improved these results by using class-adaptive Mahalanobis metric. Their method, Simple CNAPS, uses a conditional neural-adaptive feature extractor, along with regularized
classifier. This modification to CNAPS (Requeima et al. 2019) achieves improved performance on the Meta-Date Dataset benchmark (Triantafillou et | 2019), only recently surpassed by SUR (Dvomik et al. 2020) and URT (Liu et al.) 2020). However, performance suffers in the regime where there are five or fewer support examples available per class.

---

Under review as a conference paper at ICLR 2021
Inttialization
SoftK Assignment
Cluster Updates
Soft k-means Mahalanobis-distanee based clustering method used in Transductive CNAPS. First, cluster parameters are initialized using the support examples. Then, during cluster update iterations, query examples are assigned class probabilities as soft labels and subsequently, both soft- labelled query examples and labelled support examples are used to estimate new cluster parameters.
Figure 1:
Motivated by these observations, explore the use of unlabelled examples through transductive learning within the same framework as Simple CNAPS. Our contributions are as follows. (1) We propose transductive few-shot learner, namely Transductive CNAPS, that extends Simple CNAPS with transductive two-step task encoder, as well as an iterative soft k-means procedure for refining class parameter estimates (mean and covariance) using both labelled and unlabelled examples. (2) We demonstrate the efficacy of our approach by achieving new state of the art performance on Meta-Dataset (Triantafillou et al. 2 (3) When deployed with feature extractor trained on their respective training sets, Transductive CNAPS achieves state of the art performance on 4 out of 8 settings on Aani-Im (Snell et al. 2017) and tiered-Imagenet (Ren al.) 2018), while matching state of the art on another 2. (4) When additional non-overlapping classes from ImageNet (Russakovsky et al., 2013) are used to train the feature extractor, Transductive CNAPS is able to leverage this example-rich feature extractor to achieve state of the art across the board on mini- ImageNet and tiered-Image ImageNet.
2 RELATED WORK
2.1 FEW-SHOT LEARNING USING LABELLED DATA
Early work on few-shot visual classification has focused improving classification accuracy through the use of better classification metrics with a meta-learned non-adaptive feature extractor. Matching networks (Vinyals al. 2016) use cosine similarities over feature vectors produced by independently learned feature extractors Siamese networks (Koch et al. 2013) classify query images based on the nearest support example in feature space, under the L metric. Relation networks (Sung et al. 2018) and variants (Kim et al., 2019; Satorras & Estrach| 2018) learn their own similarity metric. parameterised through Multi-Layer Perceptron. More recently, Prototypical Networks (Snell et al., 2017) learn shared feature extractor that is used to produce class means in feature space where the Euclidean distance is used for classification.
Other work has focused on adapting the feature extractor for new tasks. Transfer learning by fine- tuning pretrained visual classifiers (Yosinski et al.) 2014) was an early approach that proved limited in success due to issues arising from over-fitting MAML (Finn et al. 2017) and its variants (Mishra ct al.
2017: Nichol et al. 2018; Ravi & Larochelle 2017) learn meta parameersss that allow fast task-adaptation with only few gradient updates. Work has also been done on partial adaptation
of feature extractors using conditional neural adaptive processes (Oreshkin et af.) 2018 Garnelo
2019 Bateni et al. 2020). These methods rely on channel-wise adaptation of pretrained convolutional layers by adjusting parameters of FiLM layers (Perez et al. 2018) inserted throughout the network. Our work builds on the most recent of these neural adaptive approaches, specifically Simple CNAPS (Bateni et al. 2020). SUR (Dvorik et al.) 2020) and URT (Liu et al. 2020) are two very recent methods that employ universal representations stemming from multiple
al. 2018 Requeima al.
feature extraction heads. URT (Liu et al. (2020). which was developed and released publicly parallel this work, achieves state of the art performance by using a universal transformation layer.
2.2 FEW-SHOT LEARNING USING UNLABELLED DATA
Several approaches Kim et al. 2019 Liu et al. 2018 Ren et al. 2018) have also explored the use of unlabelled instances for few-shot visual classification. EGNN (Km et al. 2019) employs
2

---

Under review as a conference paper at ICLR 2021
Layer
Layer
Layer
Layer 4
S
Pro
Block
Block
Block
Block2
Blook
Blook
Block
Block 2
Post
SB
Layers (rr
Layar
ayars (ro.
Layers (ruoBe)
Encoder
do
gy(S) ot go(S, Q)
r/8
Figure 2: Overview of the neural adaptive feature extraction process used in Transductive/Siimle CNAPS, Figure was adapted from Bateni et al. (020)).
graph convolutional edge-labelling network for iterative propagation of labels from support to query instances. Similarly, TPN (Liu et al., 2018) learns graph construction module for neural propagation of soft labels between elements of the query set. These methods rely on a neural parameterization of distance within the feature space. TEAM (Qiao et al. 2019) uses episodic-wise transductive adaptable metric for performing inference on query examples using a task-specific metric. [Song et al. (2020) use cross attention network combined with transductive iterative approach for augmenting the support set using the query examples. The closest method to our work is |Ren et al. (018). Their approach extends prototypical networks by performing a single additional soft-label weighted estimation of class prototypes. Our work, on the other hand, is different in three major ways. First, we produce labelled estimates of both class mean and covariance. Second, we use an iterative algorithm with a data-driven convergence criterion allowing for a dynamic number of soft-label updates, depending on the task at hand. Lastly, we employ neural adaptive procedure for feature extraction that is conditioned a two-step learned transductive task representation, as opposed to a fixed feature-cxtractor As we discuss in Section 4.2 this novel represetrtnnnniionnn encoder is responsible for substantial performance gains out-of-domain tasks.
3 METHOD
3.1 PROBLEM DEFINITION
Following (Snell ot al. 2017 Bateni et al. 2020 Requeima et al. 2019 Finn et al. 2017). ). we focus
on few-shot classification setting where a distribution D over image classification tasks (8, Q) is provided for training. Each task (S, 2) D consists of a support set S = {(x,, y: J of labelled images and query set Q fx; 1 of unlabelled images; the goal is predict labels for these query examples, given the (typically small) support set. Each query image x E Q has a corresponding ground truth label yf available training time. A model will be trained by minimizing, over some parameters 0 (which are shared across tasks), the expected query set classification loss log po xx S, 2)]; the inclusion of the dependence on all of Q here allows for the model to be transductive. At test time, a separate distribution tasks generated from previously unseen images and classes is used to evaluate performance. We also define shot as the number of support examples per class, and way as the number of classes within the task.
over tasks: ((s, D>x=>
3.2 SIMPLE CNAPS
Our method extends the Simple CNAPS (Bateni al. 2020) architecture for few-shot visual classifi- cation. Simple CNAPS performs few-shot classification in two steps. First, it computes task-adapted features for every support and query example. This part of the architecture is the same that in CNAPS (Requeima et al. 2019), and is based on the FiLM meta-learning framework (Perez et al. 2018). Second, it uses the support set to estimate a per-elass Mahalanobis metric, which is used to assign query examples to classes. The architecture uses ResNet18 (He et al. 2015) feature extractor. Within each residual block, Feature-wise Linear Modulation (FiLM) layers compute a scale factor y and shift 3 for each output channel, using block-specific adaptation networks dh that are conditioned on task encoding. The task encoding go (S) consists of the mean-pous soaand feature vectors of support examples produced by dg, separate but end-to-end learned Convolution Neural Network (CNN). This produces an adapted feature extractor fo (which implicitly depends on the support set S) that maps support/query images onto the corresponding adapted feature space. We will denote by Sy, Qo versions of the support/query sets where each image is mapped into its feature representation fo(x). Simple CNAPS then computes Mahalanobis distance relative each
3

---

Under review as a conference paper at ICLR 2021
dg
Simple CNAPS
Transductive
CNAPS
LSTM
L8TM
gv18,
(a) Transductive Set-Encoder Overview
(b) Transductive CNAPS Simple CNAPS
Figure 3: a) Overview the transductive task-encoding procedure, (S, Q), used Transductive CNAPS. b) Transductive CNAPS (right) extends the Mahalanobis-distann based classifier in Simple CNAPS (left) through transductive soft k-means clustering of the visual space.
class by estimating a mean μl and regularized covariance Q₁ in the adapted feature space, using the support instances:
n1k Σ
Tl k) Z4:
Q dE Eg
Aα) E BI,
A
T4 +
Here Iyy k] is the indicator function and n& class in the support set S. The ratio D balances a coss-condiiional sample covariance E and class-conditional sample covariance Σx:
Σ, I[u k] is the number of examples with
- - > - (Z)=-))
Σg = Thk ΣE
Tve = (z, - (2: -2x)²
(2)
where μa - is the task-level mean. When few support examples are available for particular class, 0 small, and the estimate regularized towards the task-level covariance E. number of support examples for the class increases, the estimate tends towards the class-conditional covariance Σx. Additionally, a regularizer 31 (we set invertibility. Given the class means and covariances, Simple CNAPS computes class probabilities for each query feature vector z through a softmax over the squared Mahalanobis distances with respect to each class:
in our experiments) is added to ensure
ply" = lz) x exp ( (-(z- -1x))" Q=1(z p4x)))
3.3 TRANSDUCTIVE CNAPS
Transductive CNAPS extends Simple CNAPS by taking advantage of the query set, both in the feature adaptation step and the classification step. First, the task encoder is extended to incorporate both support-set embedding e, and a query-set embedding e, such that,
e, =
I Σ K
Thu Ad] k) (x;);
Σ dx())
(4)
1hg
where dg is learned CNN. The support embedding e, is formed by an average (encoded) support examples, with weighting inversely proportional to their class counts to prevent bias from class imbalance. The query embedding ey uses simple mean-pooling; both e, and eg are invariant to permutations of the respective support/query instances. We then process 6, and eg through two steps of Long Short Term Memory (LSTM) network in the same order to generate the final transductive task-embedding gS, Q) used for adaptation. This process is visualized in Figure 3
4

---

Under review as a conference paper at ICLR 2021
Algorithm Iterative Refinement in Transductive-CNAPS
1: procedure COMPUTE QUERY_LABBER Qo Nuer)
if (z), v/f) E Sg and Bj k
2: For j ranging over support and query sets, Wijk
otherwise
The first iteration is equivalent to Simple CNAPS: Compute class parameters μı,; Qs according to update equations equation eequatinn 5: Compute class weights using class parameters according to equation 5
3:
for iter= 0... Nite do
4:
6: break the most probable class for each query example hasn't 't changed
7: end for
8: return class probabilities Wikk for corresponding to Qo
9:
end procedure
(a) In-Domain
(b) Out-of-Domain
(c) Overall
Figure 4: Class recall (otherwise noted in-class query accuracy) averaged between classes across all tasks and (a: In-Domain, b: Out-of-domain, c: all) Meta-Datased datasets. Class recalls have been grouped together, averaged and plotted according to the class shot in (a), (b), and (c).
Second, we can interpret Simple CNAPS as a form of supervised clustering in feature space; each cluster (corresponding to class k) is parameterized with centroid μ4 and metric Q and interpret equation 3 as class assignment probabilities based on the distance to each centroid. With this viewpoint in mind, natural extension to consider is probabilities on unlabelled data to refine the class parameters μıs- Q2 in a soft k-means framework based on per-cluster Mahalanobis distances (Melnykov & Melnykov 2014). In this framework, as we alternate between computing updated assignment probabilities using equation
the estimates of the class assignment
shown in Figure M on the query set and using those assignment probabilities to compute updated class parameters.
We will define Rş Sş Li Qe as the disjoint union of the support set and the query set. For each element of RR, which index by j. we define responsibilities wyR in terms of their class predictions when it is part of the query set and in terms of the label when it is part of the support set,
ply] = k z 1// k]]
zij Qo.
Wisk
1/) E Sg.
(5)
Using these responsibilities we can incorporate unlabelled samples from the support set by defining weighted estimates / and Q';
1
Σ Wisk z T
+ -
(6)
where n/c = E, Wij defines X/ =n///mit 1), and the covariance estimates E' and Σ
E' Σn m/
Σ
Winh (x's -1' (1) - Σ
1 1 Σ Win (2l, - (4) (z)- n//
jk
with 4e = 14)
Eje Wjzz'z being the task-level mean.
These update equations are simply weighted versions of the original Simple CNAPS estimators from Section 5 and reduce them exactly in the case of empty query set.
Algorithm summarrzes the soft k-means procedure based on these updates. We initialize our weights using only the labelled support set. We use those weights to compute class parameters, then compute
5

---

Under review as a conference paper at ICLR 2021
updated weights using both the support and query sets. At this point, the weights associated with the query set Q are the same class probabilities as estimated by Simple CNAPS. However, we continue this procedure iteratively until we reach either reach a maximum number of iterations, until class assignments a Wijk stop changing.
Unlike the transductive task-encoder this second extension, namely the soft k-mean iterative estima- tion of class parameters, is used at test time only. During training, a single estimation is produced for both mean and covariance using only the support examples. This, as we discuss more in Section 4.2
was shown to empirically perform better. See Figure 3+b for a high-level visual comparison of classification in Simple CNAPS vs. Transductive CNAPS.
4 EXPERIMENTS
4.1 BENCHMARKS
Meta- DatasDt (Tranntaiillou et al. 2019) is a few-shot image classification benchmark that consists
of 10 widely used datasets: ILSVRC-201 2 (ImageNet) (Russakovsky et al. 2015), Omniglot (Lake)
et al. 2015), FGVC-Aircraft (Aireraft) (Maji et al. 2013). CUB-200-201 (Birds) Wah et al. (011).
2016), FGVCX Fungi Describable Textures (DTD) (Cippoi al. 2014), QuickDraw (Jongejan et al.)
(Fungi) (Schroeder & Cui 2018). VGG Flower (Flower) (Nilsback & Zisserman 2008), Traffic Signs
(Signs) (Houben al.] 2013) and MSCOCO (Lin et al. 2014). Consistent with past work (Requeimal ct al. |019 Bateni et al. 2020). we train our model on the official training splits of the first 8 datasets and use the test splits evaluate in-domain performance. We use the remaining two dataset as well as three external benchmarks, namely MNIST (LeCun & Cortes 2010). CIFAR10 10 (Krizhevsky) 2009) and CIFAR 100 (Krizhevsky) (009), for out-of-domain evaluation. Task generation in Meta- Dataset follows complex procedure where tasks can be of different ways and individual classes can be of varying shots even within the same task. Specifically, for each task, the task way is first sampled uniformly between 5 and 50 and way classes are selected at random from the corresponding class/dataset split. Then, for each class, 10 instances are sampled at random and used as query examples for the class, while of the remaining images for the class, shot is sampled uniformly from [1, 100] and shot number of images are selected at random as support examples with total support set size of 500. Additional dataset-specific constraints are enforced, as discussed in Section 3.2 of (Triantafillou et al. 2019), and since some datasets have fewer than 50 classes and fewer than 100 images per class, the overall way and shot distributions resemble Poisson distributions where most tasks have fewer than 10 classes and most classes have fewer than 10 support examples (see Appendix [A.]. Following Bateni et al. (2020) and Reequeiaa et al. (2019), we first train our ResNet18 feature extractor on the
defined training split of ImageNet following the procedure in Appendix A.3 The ResNet18 parameters are then kept fixed while we train the adaptation network
for a total of sampled 1 10K tasks using Episodic Training (Snell et al. 2017 Finn et al. 2017) (see Appendix A.3 for details).
mini/tiered-ImageNe (Vinyals al.] 2016; Ren et al. 2018) are two benchmarks for few-shot learning. Both datasets employ subsets of ImageNet (Russakovsky et al. 2013) with a total of 100 classes and 60K images in mini ImageNe and 608 classes and 779K images in tiered- ImtgeNe. Unlike
tasks across these datasets have pre-defined shots/ways that are uniform across
every task generated in the specified setting. Following (Nichol et al. 2018 Liu al. 2018 Snell et al. 2017). we report performance on the 1/5-shot 5/10-way settings across both datasets with 10 query examples per class. We first train the ResNet18 on the training set of the corresponding benchmark at hand following the procedure noted in Appendix [ipendix-A.4] We also consider a more feature- rich ResNet18 trained on the larger ImageNet dataset. However, we exclude classes and examples from test sets of mini/tiered ImageNet to address potential class/example overlap issues, resulting in 825 classes and 1,055,495 494 images remaining. Then, with the ResNet18 parameters fixed, we train episodically for 20K tasks (see Appendix A.2 for details).
minitiereedImageNet
4.2 RESULTS
Evaluation on Meta-Dataset: In-domain, out-of out-of-domain and overall rankings shown in Table ! Following [ateen et al. (2020) and Requeima et al. (2019), we pretrain our ResNet feature extractor the training split of the ImageNet subset of Mata-Dataset. As demonstrated,

---

Under review as a conference paper at ICLR 2021
In Domain Accuracy (%)
Out of Domain Accuracy (%)
Avg Rank
Insgoker
Orangle sld
Aironat
DTD
Drec
Fingl
Rower
Sign
MSCOCO
MNSST
CIFARIO
CIVARIO
Model
RelationNet
30,9+0-9
86668088
69.7
14.+
56.6etl.7
61.5e10
52.621l1
76.14088
37.560.9
27.4
NA
NA
11.0
10.6
MatchingNet
16.1
3+1.0
69.2:1.0
4+
8e0.
80
33.7610
81.9
55.6e11
NA
NA
10.1
9.8
MAMI.
37,8+1.0
83.9+1.0
76.4+0.7
62.
4+1.1
64.1+0.8
59.7-1.1
53.5+1.1
79.940.
42.0+13
29.4+1.1
NA
NA
NA
9.2
10.5
9.5
ProcoNer
44.5+1.1
79.6+1.1
71.1
67.0e1
0
65.2+l..8
6499=09
40.3+1.1
86.900.7
46.5+10 39 39,9+1.1
NA
NA
NA
8.2
9.5
8.5
Prot
46.5+1.1
$2.7a1.0
88.740.7
52.4e1.1 41.7s1.1
NA
CNAPS
52.3+1.0
58.4+0.7
80.540.
72.2+09
18.3
7
47.4+10
86.00.5
60.2+0.0
42.6+1.1
92.7+0.4
61.5.0.7
10.1+00
BOIIB-E E
S5
77.5x1.1
60.910.9
73.6.08
72.8.0.7
61.2:09
44.5a.1
90,610
57.5
51,91100
NA
NA
NA
6.4
4.0 5.9
TaskNorm
50.6+1.1
90.7
83.840.6
74.6
62.
74.8=0.
48.
48.7=1.0
89.610.5
020.0007 43.4+1.0
92.3+0.4
69.3e0.8
646.111
4.7
4,8
Simple CNAPS
5.
91.7+066
82,4+0.7 74.9008
77.7-0.7
46.-1.
90.740.5
73.5H0.7 46.2+1.1
93.0H0.
74.5+0.7
60 5-1
SUR URT
56. 36.311.1
90. 1 85,410.7
71. 0 71.5408
61.1.06
63.1110 82.810.7 70.4 52,411.1 94.34044
66..
56 6.
3.1
2.9
55.7e1.0 94.4e0.4 85.880.6 76. 360.8 71 71,8 82.5e0.6
63.5 88.2+0.6 60 4+0.8
52.2e1.1 94.800.4
67.5+0.8
56 9-
1.7
28
2.2
Our Method
5888111 93.9004 64.120.5 76.820.8 69 19.0 78.620.7 48.881.1 91,610.4 76.1.0.7 48.721.0 95.7 75.7 62.9110
2.1 1.6 1,9
Table 1: Few-shot classification Meta-Dataset, MNIST, and CIFAR10/100) Error intervals showcase 95% confidence interval, and bold values indicate statistically significant state of the art performance. Average rank is obtained by ranking methods on each dataset and averaging the ranks.
mini Nett Accuracy (%)
tiered ImageNett Accuracy (%)
S-way
10-way
S-wwy
10-way
Model
Transductive
1-shor
5-shot
I-sher
S-shot
1-shor
S-shor
5-shor
Finn al 2017
48.72188
31.341.1
51.711.8
70.311.7
34,411.2
$3.121.1
MAML- +3 40115
Yes
50.8:1 8
66 8
318840
48.221..
S3.211 8
80
34.811.2
S1.7a1
Reptile Nochor 6.)))
No
47 1:0.3
63. 4
31.1.0.3
44.7:0.3
49.0e0.2
66 5:0
33.7.0.3
48.0e0.3
Reptile-BX (t.h 2018
BN
49 9.0.3
66.010.6
32.020.3
47.610.3
52.410.2
71 71.0002
35.310.3
52.040.
ProtoNet
46.12088
32.90.55
49.310.4
48.610.9
37.310.6
$7.8005
RelatioNNr
BN
41.0008
67.0.0.7
34.000.5
47.9.04
54,500.9
71 3+0.
36.3+0.6
6
TPN Liu 2016)
Yes S1.4.0.8
67.1.0.7
34.900.5
47.910.1
59.910.9
73.3
41.810.5
59.1
AttWrighLem Kmbrrs & Kommddaki 2013
BN
56.2.0.9
73.010.6
NA
KA
NA
NA
NA
NA
TADAM Oresficn dt 6013
58.520.3
76.720.3
NA
NA
NA
NA
Simple CRAPS (53tmm stal. A00
BN
53.2:009
70.810.7
37.120.5
56,710.5
63.011.0
$0,0008
18.1
70.200.6
LBO Raso el al 2013
BN
61 8:00.1
77.610.1
NA
NA
66.310.1
41.
NA
NA
Transoucove
55 6.069
73.1
42840.7
59.620.5
65.911.0
81880.7
54.610.8
7255206
CNAPS CNAPS (Rateni al.
2007
FETI
BN
77 8
90.340.4
63.5+046
$3.1+044
71,4+1.0
86.0006
57.1+0.7
78.5+05
Transductive CNAPS
79.9:0.8
91.550.4
68.540.6
85.9:0.3
73.8+1.0
87.75066
65.120.8
506605
Table 2: Few-shot visual classification results on 1/5-shot 5/10-way few-shot on mini/iereddImageNl For CNAA-based model, "FETT" indicates that the feature extractor used has been trained on ImageNet Russakovsky et al. (213)) exluding classes within the test splits of mini/tiered mini/iereedImageNer (for more details see (ppendix-TTBDD). "BN" indicates implicit transductive conditioning on the query set through the use of batch normalization. Error intervals showcase 95% confidence interval.
Transductive CNAPS sets new state of the art accuracy on 2 out of the 8 in-domain datasets, while matching other methods on 2 of the remaining domains. On out-of-domain tasks, it performs better with new state of the art performance on 4 out of the 5 out-of-domain datasets, Overall, it produces an average rank of 1.9 among all datasets, the best among the methods, with an average rank of 2.1 on in-domain tasks, only second to URT which was developed parallel to Transductive CNAPS, and 1.6 on out-of-domain tasks, the best among even the most recent methods.
Evaluation on Imageeeet We consider two feature extractor training settings these benchmarks. First, we employ the feature extractor trained on the corresponding training split of the mini/tiered-ImageeNe As shown in Table [ hered-ImageNee Transductive CNAPS achieves state of art performance on both 10-way settings while matching state of the art accuracy of LEO (usu 2018) on the 5-way settings. On mini-ImageNet, Transductive CNAPS out-performs other al.
methods on 10-way settings while coming second to LEO (Rusu et al. 2018) and TADAM (Requeima
al. 2019 Oreshkin et al. 2018) on 5-way. We attribute this difference in performance between mini- MegeNNet and ticred- ImageNet to the fact that mini-ImageNNe only provides 38,400 training This results in a lower performing ResNet-18 18 feature extractor (which is trained in traditional supervised manner). This hypothesis is further supported by the results provided in our second model (denoted by "FETI", for "Feature Extractor Trained with ImageNet", in Table 2. In this model, we train the feature extractor with a much larger subset of ImageNet, which has been carefully selected to prevent any possible overlap (in examples or classes) with the test sets of mini/tiered ImageNet. In this case, Transductive CNAPS is able to take advantage of the more example-rich feature extractor, establishing state of the art performance across the board. Additionally, even when compared to Simple CNAPS using the
examples, compared to 448,695 examples provided by hered-Imagenh
7

---

Under review as a conference paper at ICLR 2021
In Domain Accuracy (%)
Out-of Domzin Accuracy (%)
Avg Apc.
Josgelke
Ormngld
Airoraft
Bints
QhickDrav
Pangl
Flower
Signs
MSCOCO
MAIST
CIPARIO
CHFARD0
CNAPS Model
All
GMM-EM-
53.3+1.0
91.8 31.840.6
81 2.022..6
75.860.7
71.8:116
72.920.
42,8609
91 4.0004 66. 16.
40.311 94 2 69 99.80.7
51. 1:0.
64.2 69.3
GMM
45.3+1.0
88.0.0.9
8088:08
8
61 1=0.7
70.7.0.8
42.9=1.0
88.
140.6
68.9.0.7
37.2
4
64.5
46 64..0
61.7 65.0
FROT GMM
52.6+1.1 89.6+0.7
84.0+0.6
76.2-08
66.
56.5-0.8
73.4+0.8
45.7+1.0
89.8+0.6
74.4+0.7
44.2+100
95.1-0.4
71.1+0.8
56. 56.9-1.
72.2
67.9 70.6
COT GMM
48.7+1.0
92.3+0..
90.0-007
72.4m0.7
59.880
59.8e00.
71.1
41.4
87.7+00.5
63.6608
90.2=088
89.8
66 66.9H0.7
50,5
09.2 62.0 66.4
GMM-RM
$2.3e1.0
92.0s0.5
8
64.3e0.7
72.610.8
4..611.0
71.460.7
44.7e0.9
93.000
Transductivee
53.321.1
92.340.5
S1.2
75.0=0.
72.0=0.7
74.8208
45.1a1.0
92.140.4
71.0408
44.0g1.1
95949.3
71.120.7
53,3611
73.2 67.9 71,2
Simple
S86e11. 91.7=0.6
82,460.7
74.9.0.
67.8+0.8
77.7+0.7
46.9+10
90.740.5
73.5+077
46221111
93.904
74.360.7
69.7 72.2
PEOT
57.9+1.1
90.540.7
52.92977
74.3200
67.320.8
70.32338
47.721.0
90.540..5
75.8+0.7
47.1=1.1
94.9204
74.350.8
61.221.0
73.4
70.7 72.4
COT
58.811.1
95.2+0.3
$4.02006
76.450.7
68.5:0.8
77.8
49.7&10
92.720.4
70.840.7
47.3
94
94.2
75.2 61.2s1. 75.4
69.7 73.2
Transdoctive
58.8+1.1 93.940.4 $4.190.6 76.8:0.8 69.0e0.8 78.620.7
48.8:11
91.660.4 76.150.7 48.7=1.0
95.750.3 75.750.7 62.9s1.0
71.8 73,9
Table 3: Performance of various ablations of Tranductive and Simple CNAPS on Meta-Dataset. Error
intervals showcase 95% confidence interval, and bold values indicate statistically significant state of the art performance.
same example-rich feature extractor, it outperforms the baseline with strong margins; this comparison demonstrates the gains we get from leveraging the additional query set information.
Performance vs. Class Shot: In Figure 4 we examine the relationship between class recall (i.e. accuracy among query examples belonging to the class itself) and the number of support examples in the class (shot). As shown, Transductive CNAPS very effective when class shot is below 10, showing large average recall improvements, especially at the 1-shot level. However, the class shot increases beyond 10, performance drops compared to Simple CNAPS. This suggests that soft k-means learning of cluster parameters can be effective when very few support examples are available. Conversely, in high-shot classes, transductive updates can act as distractors.
Training with Classification-Time Soft K-means Clustering: In our work, iterative updates of means and covariance at test-time only. It's natural to consider training the feature adaptation network end-to-end through the soft k-means transduction procedure. We provide this comparison in the bottom-half of Table 3 with "Transductive+ CNAPS" denoting this variation. Iterative updates during training result in an average accuracy decrease of 2.5%, which we conjecture to be due to training instabilities caused by applying this iterative algorithm early in training on noisy features.
soft k-means
Transductive Feature Extraction vs. Classification: Our approach extends Simple CNAPS in two ways: improved adaptation of the feature extractor using transductive musk-encoding, and the soft k-means iterative estimation of class means and covariances. We perform two ablations, "Feature Extraction Only Transductive" (FEOT) and "Classification Only Transductive" (COT), to independently assess the impact of these extensions. The results are presented in Table B As shown, both extensions outperform Simple CNAPS. The transductive task-encoding is especially effective on out-of-domain tasks whereas the soft k-mean lcarning of class parameters boosts accuracy on in-domain tasks. Transductive CNAPS is able to leverage the best of both worlds, allowing it to achieve statistically significant gains over Simple CNAPS overall.
The Mahalanobis disttancdddtteeeee based class probabilities produced by Equation Saattionncoosee closely resembles the cluster posterior probabilities (responsibilities) inferred by a Gaussian Mixture Model (GMM). The only changes required to make this correspondence exact is introduce a class prior distribution n, and to change the class probability model equation to the Gaussian likelihood:
Comparison to Gaussian Mixture Models:
p'y" k: = x aly" = exp (-b m TQ (z (xx)
1 2
exp
log Qu
(8)
With these modifications, Transductive CNAPS would exactly correspond to inference in GMM, with cluster parameters learned through semi semi-supervised expectation maximization (EM), Given this observation, we consider five GMM-based ablations of our method where the logedeterminant is introduced (a uniform class prior is used). These ablations are presented in Table and correspond to their soft k-means counterparts in the same order shown. The GMM-based variations of our method and Simple CNAPS result in a notable 4-8% loss in overall accuracy. It's also surprising to observe that the FBOT variation matches the performance of the full GMM-EM model.
8

---

Under review as a conference paper at ICLR 2021
Maximum and Minimum Number of Refinements: In our experiments, we use a minimum number of 2 refine- ment steps of class parameters, with the maximum set to 4 on the Meta-Dataset and 10 on the mini/tiered- min/tteered-mageNeet benchmarks. We explore the impact of these hyperparam- eters on the performance on Transductive CNAPS on the Meta-Dataset in Figure 5 As shown, requiring the same number of refinement steps for every task results in sub- optimal performance. This is demonstrated by the fact
74-
73.6
Mesa-Dattsstt Accuracy (N)
138
that the peak performance for each minimum number of steps is achieved with larger number of maximum steps,
showcasing the importance of allowing different numbers of refinement steps depending on the task. In addition, observe that as the number of minimum refinement steps increases, the performance improves up to two steps while declining after. This suggests that, unlike Ren al. 2018)) where only a single refinement step leads to the best performance, our pahalanobis-basee approach can leverage extra steps to further refining the class parame- ters. We do see a decline in performance with a higher number of steps; this suggests that while our refinement criteria can be effective at performance different number of steps depending on the task, it can potentially lead to over-fitting, justifying the need for an accurately chosen maximum number of steps.
72.5
Maximum # of K K-mean Clustering Updates
-Nin Steps
Steps
-iin Stops
-in Steps
-in Stops
- Min Stops
-Nin Stops
Min 7 Stops
Min 8 Stopo -iin Stope
-Min Stops
-in 11 Stops
Evaluating Transductive CNAPS on Meta-Dataset with differ- ent minimum and maximum number of steps. As shown, performance is im- proved with 2 minimum refinement steps required, with the best results observed at maximum of 4 refinement steps. Performance is, however, degraded with more refinement steps required mini- mum as that may lead to over-fitting.
Figure 5:
5 DISCUSSION
In this paper, we have presented a few-shot visual classi- fication method that achieves new state of the art perfor- mance via a transductive clustering procedure for refining class parameters derived from previous neural adaptive CNAPS, is more effective at producing useful estimates of class mean and covariance especially in low-shot settings, when used at test time. Even though we demonstrate the efficacy of our approach in the transductive domain where query examples themselves used unlabelled data, soft k-means clustering procedure can naturally extend to use other sources of unlabelled examples in a semi-supervised fashion.
caahalanobi--distanne
based approach. The resulting architecture, Transductive
Transductive CNAPS superficially resembles a transductive GMM model stacked on top of learned feature representation; however, when we try make this connection exact (by including the log- determinant of the class covariances), we suffer substantial performance hits. Explaining why this happens will be the subject of future work.
REFERENCES
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Aurelien Bellet, Amaury Habrard, and Mare Sebban. A survey on metric learning for feature vectors and structured data. arXiv preprint 1rXiv:/306.6709, 2013.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Nikita Dvorikk Cordelia Schmid, and Julien Mairal. Selecting relevant features from a multi-domain representation for shot classification, 2020.

---

Under review as a conference paper at ICLR 2021
Abdur R Feyjie, Reza Azad, Marco Pedersoli, Claude Kauffman, Ismail Ben Ayed, and Jose Dolz. Semi-supervised few-shot learning for medical image segmentation, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning-Vont -Volume 70, pp. 1336-1135. JMLR. org. 2017.
Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and M. Ali Eslami. Conditional neural processes. CoRR, abs/1807.01613. 2018. URL http://aaxiiii org/ abs//801/00 01613
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. CoRR, abs/1804. 09458, 2018 URL http: http:/ttaa.iiia.aaa///aaaaa org/ ab / 1804.0880000 09458
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR,
2015. URL htwp:tt::// /ttp:p///axivv../ orq/ habs///b///// 1512.03388
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In The 2013 international joint conference neural networks (IJCNN) pp. 1-8. IEEE, 2013.
Jonas Jongejan, Henry Rowley. Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment.(2016) 2016.
Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D. Yoo. Edge-laheling graph neural network for few-shot learning. CoRR, abs/1905.01436 2019. [RRp::///aaxii /at//ab////// 1905
01436
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun. ACM, 60(6):84-90, May 2017. ISSN 0001-0782. doi: 10.1 10.1145/3065386 WtRLhhtt:///dot acm org/ 10. 1ew./115////0 3065583
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133022666:333-133333 2015.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL /aann
lecun. mrom//exbb/mnie
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. universal representa- tion transformer layer for few-shot image classification, 2020.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, and Yi Yang. Transductive propaatooiveeoooaaaatnoo network for few-shot learning. CoRR, abs/1805. 10002, 2018. URL http:tpp/aaaaav..rr//b////////// 10002
Subhransu Maji, Esa Rahtu, Juho Kannala, Mattew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Igor Melnykov and Volodymyr Melnykov. On k-means algorithm with the use of mahalanobis distances. Statistics & Probability Letters, 84:88 95, 2014.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. Meta-leamnin with temporal convolutions. CoRR, abs/1 2017. http:///.p./aaaa.....r///////./0
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-nann algorithms CoRR, Abs/1803.0200 2018. URL hatt:://arxiin http:////aawww..ooaa////////////
www.or/////bsa.r 02999
10

---

Under review as a conference paper at ICLR 2021
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEEE, 2008
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam:
Task dependent adaptive metric for improved few-shot learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31. pp. 721-731. Curran Associates, Inc., 2018. URL BRL.htpp:/ htatt::/ppppess.oo http://ppprrs..ss.cccoc wwtp:icccc.cc.r.r.....o.pppp
N.
7352- hat.-ttaam-tttt task-taak--aeee dep.addeteennnnttt-em adapt mwt.ieetieeem for- mproved- fer- shot
learning.
pdf
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang, Tiejun Huang, and Yonghong Tian. Transductive episodic-wise adaptive metric for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https::/ EJY0-Ke11
?id=
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-leaming for semi-supervised few-shot classification. CoRR, abs//800,.nm 8888000066666 2018.
00676
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. arXiv preprint arXiv:1906.0777 07697, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. CoRR, abs/ 1877 005600 2018. /tR:: h/tp:v/aaaxiiee/ea org. abs 1807. C5960
Victor Garcia Satorras and Joan Bruna Estrach.
Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018. URL
net.
B. Schroeder and Cui. Fgvex fungi classification challenge 2018. https:://ithub.. com/ visipedia/fqya fungi comp 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv 1409. 1556, 09 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems. pp. 4077-4087, 2017.
Xibin Song, Yuchao Dai, Dingfu Zhou, Liu Liu, Wei Li, Hongdong Li. and Ruigang Yang. Channel attention based iterative residual learning for depth map super-resolution. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
M. Somam,, K. Muthusubash, and V. Vanitha. A survey image classification and activity recognition using deep convolutional neural network architecture. In 2017 Ninth International Conference on Advanced Computing (ICoAC), pp. 121-126, Dec 2017. doi: 10. 0009/CCCA..20..8441111.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. URL
abs//40ss///// 4842
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning learn from few examples. arXiv preprint arXiv:7903.03096 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Wei Wang, Vincent W. Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Set- tings, methods, and applications. ACM Trans. Intell. Syst. Technol, 11-3333: January 2019. ISSN 2157-6904. doi: 10. 145/3293933338 URL 3293318
Yaqing Wang and Quanming Yao. Few-shot learning: A survey. CoRR, ab/11944.00066 2019. URL http: /ttt::///rxi...ot/// http:/ggg//bbaaooe 05046
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? CoRR, abs/1411. 2014. URL http::::/aaarii..rrraaa////////a
192
12


Reviews:
[{"review_id": "u_jfklGi4Nb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a transductive few-shot learning method, Transductive CNAPS, by using the unlabeled examples. Experimental results on the Meta-Dataset and mini/tiered-ImageNet benchmarks are reported. \n\nThe proposed approach is a simple extension of the simple CNAPS method. It extends the simple CNAPS by using the unlabeled query instances to update the class centroids \\mu_k and variance Q_k with the predicted class probabilities. The level of technical contribution and novelty is very incremental and low.\n\nThe experiments are not very convincing. First, for both experiments on Meta-Dataset and min/tiered-ImageNet, the feature extractor is pretrained on some ImageNet subset. This is not appropriate as it borrows significant information from the pretraining dataset. The authors also mentioned this issue. \nSecond, the transductive few-shot learning methods need to be compared with on the Meta-Dataset in Table 1.  Even with the current comparison methods, the performance gains of the proposed approach are not very notable. \n\nSome additional questions: \n1.\tIn Table 1, are all the results obtained using exactly the same sets of training/testing tasks for all comparision methods? What are the source domains for each of the out-of-domain task?\n2.\tDo all the comparision methods perform the same pre-training? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kdgRWoJTnoV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n1) The paper is well-motivated, clearly written and neatly organized. I like the writing style of this paper: explaining very clearly what the improvements are and how they perform with respect to the counterpart methods. \n\n2) The ablation study validates the effectiveness of the improved adaptation of the feature extractor using transductive task-encoding, and that of the soft k-means iterative estimation of means and covariances for classification.\n\n3) Experiments are solid and extensive. The results on Meta-Dataset, mini/tiered-ImageNet are competitive, as opposed to the state-of-the-art methods.\n\nCons:\n\n1) The originality of the proposed method is incremental. Compared to simple CNAPS (Betni et al, CVPR 2020), the task encoder is extended to incorporate both a support-set embedding and a query-set embedding through Long-Short Term memory (LSTM) network, and the classifier is extended to include the unlabeled examples in the query set as well. As far as I know, encoding both support examples and query ones using LSTM has been used in Matching Network (Vinyals et al., 2016) while the idea of using query examples to refine class porotypes via soft k-means clustering were proposed by Ren et al. ,2018.\n\n2)  Regarding the soft-kmean iterative estimation, why can it be only used at test time while the performance decreases significantly if used during training as well? How many iterations are performed? Will various number of iterations matter? In particular, I am not satisfied with the limited explanation on this method provided in Section 5.2 (last paragraph on page 7), and I wish the authors give more analysis and insight on this problem.\n\n3)  Some minor points:\nIn Table 2, what does \"BN” mean?\nHow to set the value of $\\beta$ in Equation (6)?\nSome typos: In Section 3.2, \"metalearning framework \"  --> \"meta learning framework \"; in Section 4.2, \"ou-of-domain\" --> \"out-of-domain\".\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bXCFs9RNEmW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to extend the simple CNAPS few-shot learning method to the transductive setting. Specifically, the proposed method introduces transductive task encoding and soft k-means iterative estimation to improve CNAPS. The proposed method is able to achieve SOTA results on the transductive benchmarks.\n\n+ The proposed method can effectively improve the CNAPS as it achieves SOTA or comparable results on Meta-dataset, mini-ImageNet and tiered-ImageNet. \n+ The writing is clear and easy to follow.\n\n- The proposed method lacks enough novelty. The proposed method is actually a transductive extension of CNAPS. The authors introduce two novel components: transductive task encoding and soft k-means clustering. The former is a natural and straight-forward solution while the latter is also similar to previous work [1]. This weakens the paper's overall contribution. \n- The experiments could be improved. As stated in the experiment section, some results are not well-prepared before the submission, e.g. Overlapping-excluded ImageNet results.\n- According to Fig.4, the authors could further analyze why transductive updates can not remain improvements as the shot increases. Is there a way to design an adaptive scheme of deciding whether to use or not?\n[1] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum,\nHugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8GLtI-3hOQV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In order to improve few shot visual classification, the authors propose a transductive meta-learning method using unlabelled examples. The authors have introduced a two-step transductive encoder as well as soft k-means clustering procedure on the existing simple CNAPS architecture.\n\nPros:\nThe paper is well written.\nThe authors have improved upon the existing simple CNAPS model by making two changes:\ni) Simple CNAPS extracted feature vector by passing support example through a CNN to get support task representation $g_\\theta(S)$ which is given as an input to FiLM layers to learn scale and shift parameters. This produces an adapted feature extractor to map both support/query onto adapted feature space. While in transductive SNAPS the support task representation is used to compute support-set embedding ($e_s$) and query-set embedding ($e_q$). Both $e_s$ and $e_q$ are processed by a two-step LSTM to generate the final transductive task-embedding used for adaptation.\nii) Introduce an iterative algorithm to assign soft-label assignment of query image which in turn performs iterative estimation of class means and covariances. \n\nCons:\ni) The performance on Meta-Dataset in Table 1 seems to show that the average rank is better only in the case for the Out-of-Domain test case.\nii) Table 2 shows when Transductive CNAPS compared with other transductive approaches the performance is not that great. However, when it is compared with other SNAPS architecture the performance is good. The missing experiment of excluding the overlapping classes would provide better insight into it.\niii) As already mentioned by the authors, this work resembles Gaussian Mixture Models.\n\nOverall the model's novelty over the existing Simple SNAPS architecture is limited. Even the quantitate scores reported are not superior enough; for example in the case of Meta-Dataset the model suffers in In-Domain accuracy and does not achieve SOTA in case of mini/tiered-ImageNet. \n\nFinal review\nThe authors have addressed some of my concerns so I am updating the rating. I still feel that the mode's novelty is limited and thus my highest rating will be 6. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a transductive few-shot classification method on the basis of the simple Conditional Neural Adaptive Processes (CNAPS) introduced by Bateni et al. The paper received two borderline accept and two borderline reject reviews, indicating that the paper may not be yet ready for a publication. The meta reviewer recommends rejection based on the observations below.

All reviewers indicated that the paper is well-motivated, clearly written and neatly organized. However, all four reviewers agree that the novelty of the paper compared to the CNAPS paper is limited. The main novelty of the method being transductive CNAPS extends the task encoder of CNAPS to incorporate both a support-set embedding and a query-set embedding through Long-Short Term memory (LSTM) network. Similarly, the classifier in CNAPS has been modified to operate in the transduction setting, i.e. it is extended to include the unlabeled examples in the query set.  The reviewers indicate that extension of the task encoder via LSTM may not be enough technical novelty for such a competitive venue. Additionally, in terms of experimental evaluation, although R1 found the experimental evaluation adequate, R3 indicated some concerns about the unexpected behaviour of the method and R4&R2 found the benchmark evaluations limited.

Decision: Reject