Transductive Decoupled Variational Inference for Few-Shot Classification
Authors: Anuj Rajeeva Singh
Hadi Jamali-Rad
Venue: Accepted by TMLR ()

Abstract: The versatility to learn from a handful of samples is the hallmark of human intelligence. Few-shot learning is an endeavour to transcend this capability down to machines. Inspired by the promise and power of probabilistic deep learning, we propose a novel variational inference network for few-shot classification (coined as TRIDENT) to decouple the representation of an image into semantic and label latent variables, and simultaneously infer them in an intertwined fashion. To induce task-awareness, as part of the inference mechanics of TRIDENT, we exploit information across both query and support images of a few-shot task using a novel built-in attention-based transductive feature extraction module (we call AttFEX). Our extensive experimental results corroborate the efficacy of TRIDENT and demonstrate that, using the simplest of backbones, it sets a new state-of-the-art in the most commonly adopted datasets miniImageNet and tieredImageNet (offering up to 4% and 5% improvements, respectively), as well as for the recent challenging cross-domain miniImagenet --> CUB scenario offering a significant margin (up to 20% improvement) beyond the best existing baselines.

Published in Transactions on Machine Learning Research (02/2023)
Transductive Decoupled Variational Inference for Few-Shot Classification
Anuj Singh 1.2, Hadi Jamali-Rad 1.2
fa.r.singh, h.jamalirad Hatnnliiddttudlffftt. @tudelft.n.. nl, fanaj.singhh,, jadi wwweel...sshhooooommmmooo Delftt University of Technology. The Netherlands
Shell Globel Solutions International B. V., Amsterdam, The Netherlands
Reviewed on OpenReview: https: // openreview. net/ per/formm? id= bomdTc9Hyl
Abstract
The versatility to learn from a handful of samples is the hallmark of human intelligence. Few- shot learning is an endcavour to transcend this capability down to machines. Inspired by the promise and power of probabilistic deep learning, propose novel variational inference network for few-shot classification (coined as TRIDENT) to decouple the representation of an image into contert and label latent variables, and simultaneously infer them in an intertwined fashion. To induce task-awareness, as part of the inference mechanics TRIDENT, we exploit information across both query and support images of few-shot task using a novel built- in attention-based transductive feature extraction module (we call AttFEX). Our extensive experimental results corroborate the efficacy of TRIDENT and demonstrate that, using the simplest of backbones and meta-learning strategy, it sets new seate-of-the-ate in the most commonly adopted datasets miniTmageNet and tieredImageNet (offering up to 4% and 5% improvements, respectively), as well as for the recent challenging cross-domain miniimmagene - CUB scenario offering a significant margin (up to 20% improvement) beyond the best existing baselines!]
1 Introduction
Deep learning algorithms are usually data hungry and require massive amounts of training data to reach a satisfactory level of performance on any task. To tackle this limitation, few-shot classification aims to learn to classify images from various unseen tasks data-deficient setting. In this exciting space, metric learning proposes learn a shared feature extractor to embed the samples into metric space of aggregated class embeddings (Sung et al. 2018 Vinyals al. 2016 Snell al. 2017 Wang al.] 2019: Liu et al.) 2020). Due to limited data per class, these embeddings suffer from sample-biar and fail to elficienlly represent class characteristics. Furthermore, sharing a feature extractor across tasks implies that the discriminative infor- mation learnt from the seen classes are equally effective on any arbitrary unseen classes, which is not true in most cases. Transductive task-aware fow-shot learning approaches (Batoni et al., 2022 Ye ct al., 2020 Cui & Guo 2021) address these limitations by exploiting information hidden in the unlabeled data. As a resull, the model learns task-specific embeddings by aligning the features of the labelled and unlabelled task instances for optimal distance metric based label assignment. Since the alignment of these embeddings is still subject to the relevance of the characteristies captured by the shared feature extractors, task-aware methods some- times fail to extract meaningful representations particularly relevant to classification. Probabilistic hods address sample-bias by relaxing the need to find point estimates to approximate data-dependent distribu- tions of either high-dimensional model weights (Nguyen et al.] 2019 Ravi & Beatson] 2019 Gordon et al.
2019 Hu al. 2020) or lower-dimensional class prototypes (Sun al. 2021 /Zhang et al., 2019). However, inferring a high-dimensional postcrior of model parameters is inefficient in low-data regimes and estimating
distributions of class prototypes involves using hand-crafted non-parametric aggregation techniques which may not be well suited for every unseen task.
Codebasee available https::/ http:e/////ttttt.eee/ http:/tttt..cc///mmjjeet/tttttit

---

Published in Transactions on Machine Learning Research (02/2023)
AttFEX
Convival di Wel
Gontil Wo
P
Ø1
P4
0
ConvEnc
fd(N) Su(N) fo(N)
143
P2
F
Attention
Z
02
ConvEnc
X
Hy 12
Ze
Figure High-level process flow TRIDENT. Inferred label latent variable z; contains class-characterizing informa- tion, as is reflected by better separation of the distributions when compared to their context latent counterparts ze. AttFEX module generates task- aware feature maps by exploiting information from both support and query images, which compensates for the lack of label vectors Y in inferring
Although fit for purpose, all these approaches seem to overlook an important perspective. An image is composed of different attributes such style, design, backdrop and setting which not necessarily relevant discriminative characteristics for classification. Here, refer to these attributes contertual information. On the other hand, other class-characteriziin attribules (such wings of bird, trunk of an elephant, hump on camel's back) are critical for classification, irrespective of context. We refer to such attributes as label information. Typically, contextual information is majorly governed by context attributes, whereas the label characteristios are subtly embedded throughout an image. In other words, contextual information can be predominantly present across an image, whereas allending to subtle label information determines how effective classification algorithm would be. Thus, we argue that attention label-specific information should be ingrained into the mechanics of the classifier, decoupling it from contextual information. This becomes even more important in few-shot selting where the network has to quickly learn from little data. Building upon this idea, we propose transductive variational inference of decoupled latent variables (coined as TRIDENT), to simultaneously infer decoupled label and context information using two intertwined variational networks. To induce task-awareness while constructing the variational inference mechanics of TRIDENT, we introduce novel attention-based transductive feature extraction module (we call AttFEX) which further enhances the discriminative power of the inferred label attributes. This way TRIDENT infers distributions instead of point estimates and injects handcrafted inductivebiis into the network to guide the classification process. Our main contributions can be summarized as:
We propose TRIDENT, a variational inference network to simultaneously infer two salient decoupled attributes of an image (label and context), by inferring these two using two intertwined variational sub-networks (Fig.
2. We introduce attention-based transductive feature extraction module, AttFEX, to enable TRIDENT see through and compare all images within task, inducing transductive task-cognizance in the inference of label information.
We perform extensive ovaluations demonstrate that TRIDENT sets a new state-of-the-art by outperforming all existing baselines the most commonly adopted datasets mindImagenet and tieredImagenet (up to 4% and 5%), well for the challenging cross-domain scenario of minilmagenet + CUB (up to 20% improvement).
2 Related Work
Metric-based learning. This body of work involves mapping input samples into a lower-dimensional embedding space and then classifying the unlabelled samples based distance similarity metric. By
2

---

Published in Transactions on Machine Learning Research (02/2023)
parameterizing these mappings with neural networks and using differentiable similarity metrics for classi- fication, these networks can be trained in an episodic manner (Vinyals et al. 2016) to perform few-shot classification. Prototypical Nets (Snell et al.
2017), Simple Shot (Wang et al. 2019], FRN (Wertheimer
ct al. 2021,, Relation Networks (Sung et al.
2018). Matching Networks (Vinyals et al. 2016) variants of
Graph Neural Nets (Satorras & Estrach 2018
Yang el al. 2020), are a few examples of seminal ideas here.
Transductive Feature-Extraction and Inference. Transductive feature extraction or transductive task- aware learning is variant of metric-learning with an adaptation mechanism that aligns support and query feature vectors in the embedding space for better representation of task-specific discriminative information. This not only improves the discriminative ability of classifiers across tasks, but also alleviates the problem of overfitting on limited support set since information from the query set is also used for extracting features of images in a task. CNAPS (Requeima et al. 2019). Transductive- CAnnsS (Bateni et al.) 2022 FEAT (Ye et al., 2020 Assoc-Align (Afrasiyabi et al.) 2020), TPMN (Wu et al., 2021 and CTM (Li et al. 2019) are prime examples of such methods. Next to transduction for task-aware feature extraction, there are methods that use transduclive inference to classify all the query samples at once by jointly assigning them labels, opposed to their inductive counterparts where prediction is done the samples one at a time. This is either done by iteratively propagating labels from the support to the query samples or by fine-tuning a pre-trained backbone using an additional entropy loss on all query samples, which encourages confident
class predictions at query samples. TPN (Liu et al. 2019 Ent-Min (Dhillon et al. 2020). TIM (Boudiaf let al. 2020), Transductive CNAPS (Bateni et al. 2022), LaplacianShot (Ziko et al., 2020 DPGN (Yang 2020 and ReRank (SHEN et al., 2021) ) are a few notable examples in this space that usually report state-of-the-art results in certain few-shot classification settings (Liu et al. 2019). That being said, TRIDENT can be regarded as transductive feature-cxtraction method, owing to AttFEX's unique ability to sce through and compare all images within task.
ct al.
Optimization- based meta-learning, These methods optimize for model parameters that are sensitive to task objective functions for fast gradient-based adaptation to new tasks. MAML (Finn et al. 2017) and its variants [Rajeswaran el al. 2019 Nichol et al. 2018b), (Oh et al. 2021) are a few prominent examples while LEO (Rusu et al. 2019) efficiently meta-updates its parameters in lower dimensional latent space. Meta- 2017b) uses a separate meta-learner model to learn the exact optimization algorithm used to train another 'learner' neural network classifier.
learner LSTM Raai & Larochelle
Probabilistic learning. The estimated parameters of typical gradient-based meta-learning methods dis- cussed earlier FFnn et al. 2017 Rusu et al. 2019 Mishra et al.) 2018 Nichol et al. 2018b Rajeswaran et al., 2019), have high variance due to the small task sample size. To deal with this, a natural extension is to model the uncertainty by treating these parameters as latent variables a Bayesian framework as proposed in Neural Statistician (Edwards & Storkey 2017), PLATIPUS (Finn et al. 2018), VAMPIRE (Nguyen et al. 2019), ABML (Ravi & Beatson 2019), , VERSA Gordon et al. 2019). SIB (Hu et al. 2020), SAMOVAR
Iakovleva et al.) 2020). Methods like ABPML Sun et al. 2021 and VariationalF'SL (hang et al. 2019
infer latent variables of class prototypes to perform classification and avoid inferring high-dimennsionl model parameters. ABPML (Sun et al., 2021) and VariationalFSLL (Zhang et al. 2019) are the closest to our ap- proach. In contrast to these two methods, we avoid hand-crafting class-level aggregations. Additionally, we enhance variational inference by incorporating a cosifiication-relvean inductive bias through decoupling of label and context information.
3 Problem Definition
Consider a labelled dataset D - (x,, E ['] of images x, and class labels y/- This dataset D is divided into three disjoint subsets: D = (Dir U Deal U Dieat), respectively, referring to the training, validation, and test subsets. The validation dataset Deal is used for model selection and the testing dataset Drest for final evaluation. Following standard few-shot classification scttings, as proposod in Vinyals et al. (2066); /ung et al. (2018); [Snell et al.] (2017), we use episodic training on a set of tasks T, p(T). The tasks are constructed by drawing K random samples from N different classes, which we denote an (N- way, K-shot) task. Concretely, each task T, is composed of support and query set. The support set s [(x] v l K],Kn n E N]} contains K samples per class and the query set Q {(xb. v l E
3

---

Published in Transactions on Machine Learning Research (02/2023)
y
Zi
Ze
X
~ pT])
Figure 2: Generative Model of TRIDENT. Dotted lines indicate variational inference and solid lines refer to generative processes. The inference and generative parameters are color coded correspond their respective architectures indicated in Fig 1 and Fig
11 Q. n E [1, N]} contains Q samples per class. For a given task, the NQ query and NK support images are disjoint to assess the generalization performance.
4 The Proposed Method: TRIDENT
Let us start with the high-level idea. The proposed approach is devised to learn meaningful representations that capture two pivotal characteristies of an image by modelling them as separate latent variables: (i) reprosenting contert, and (ii) 2 embodying class labels. Inferring these two latent variables simultancously allows 2, to learn meaningful distributions of clas-ddscriminattngg characteristics decoupled from context features represented by ze. We argue that learning 2 as the sole latent variable for classification results in capturing a mixture of true label and other context information. This in turn can lead to sub-optimal classification performance, especially in a few-shot setting where the information per class is scarce and the network has adapt and generalize quickly. By inferring decoupled label and context latent variables, inject handcrafted inductive-bias bias that incorporates only relevant characteristics, and thus, ameliorates the network's s classification performance.
4.1 Generative Process
The directed graphical model in Fig. 2 illustrates the common underlying generative process such that P. = =(xx,, ys Zi,, 2.). For the sake of brevity, in the following we drop the sample index as we always refer to terms associated with single data sample. We work on the logical premise that the label latent variable z₁ is responsible for generating class label as well as for image reconstruction, whereas the context latent variable z, is only responsible for image reconstruction (solid lines in the figure). Formally, the data is explained by the generative processes: Pa; (y 2)) = Cat(y (2) and Pe, (x 2,) 9ea (x; 24, ze), where Cat(.) refers to multinomial distribution and 9o, (x; Zı, 2.) is a suitable likelihood function such as a Gaussian or Bernoulli distribution. The likelihoods of both these generative processes are parameterized using deep neural networks and the priors of the latent variables are chosen to be standard multivariate Gaussian distributions (Kingma & Welling 2014; Kingma et al. 2014): p(z.) = =N(c 00, I) and p(2:) = -N 0,
4.2 Variational Inference of Decoupled Z₁ and Z,
Computing exact posterior distributions is intractable due high dimensionality and non-linearity of the deep neural network parameter space. Following Kingma & Welling (2014); Kingma et al. (2014). we instead construct approximate posterior over the latent variables by introducing fixed-form distribution g(z,, Ze xxy) y) parameterized by Φ. By using (.). as an inference network, the inference is rendered tractable, scalable and amortized since Φ now acts as the global variational parameter. We assurne has a factorized form 9o (ze, 2i xx y) = 9o, (z) x, 2e) 4d. (zp x), where qo, (.), (.) are assumed to be multivariate Gaussian distributions. As also depicted Fig. 2 2 we use 2, as input to gor (.) infer z because of their conditional dependence given x. This way we forge a path to allow necessory context latent information flow through the label inference network. On the other hand, the opposite direction (using 2 to infer 2.) is unnecessary,
4

---

Published in Transactions on Machine Learning Research (02/2023)
because label information does not directly contribute to the extraction of context features. We will further reflect on this design choice in the next subsection. Neural networks are then used to parameterize both inference networks as:
9b (ze x) = N (zel 1ea (x), diagl (x))),
(1)
1b (7) x, kc) = N (2) Mai (x, 2c), ding (x, kc)))
To find the optimal approzimate posterior, we derive the evidence lower bound (ELBO) on the marginal likelihood of the data to form our objective function:
p(x, v) = If p(x,y y Zor zi) p(z,, zi) dz dzi,
1x) p(x zi, ze ky) Z) (1(p)(z (p(())))((())
((21, 2c x) In (1xx(xn)) v) Beg(e. In p(x
21, Ze ) 21 (1)p((z))((2))
(z 21 x) Z 1))pp )p(z)))))))
p(x Zo: 2:)p(y
-Beoz
Engr
In
(Zz x) x, ke)
Denoting V = =001,0 Φ1, 022,, the negative ELBO can be given by
L(V)
Egga Eye, [In po, (x Ze, zi) + In po, (y 21) Bys, [Dxr. (gon (z x, 2x)) p(z))))
(2)
DIC (Yos (2e x) p(z.)),
where the second line follows the graphical model in Fig 2 and E(.) and In(.) denote the expectation operator and the natural logarithm, respectively. We avoid computing biased gradients by following the re-paramcterization trick from Kingma & Welling (2014). Note that equation | we deliberately choose to exclude the label information y as input qá (-) be able exploit the associated generative network pe, (y zi) classifier. The consequence and the proposed solution to accommodate this design choice are discussed in the next subsoction.
4.3 AttFEX for Transductive Feature Extraction
Our design choice to omit label information y when inferring 2; (as discussed for equation I can be information bottleneck and counter-productive to the discriminative power 2, holds. However, this allows us to employ z for classification and not reconstruction of the label. To compensate for this bottleneck, we introduce an attention-based transductive feature extractor (AttFEx) module that allows the network do (z: x, 2e) see through and compare images across all classes within each task (irrespective being from the query or support sets), thus, induces task-cognizance in the inference network. We first extract the feature maps of all images in the task using convolutional block F ConvEnc(X) where X E B F ((K+))x xH''H' The feature map tensor F is then transposed into F' (C* xH' and led into two consecutive 1 x 1 convolution blocks. This helps the network utilize information across corresponding pixels of all images in a task T,, which can be considered as a parametric comparison of classes. We leverage the fact that ConvEnc already extracts local pixel information by using larger kernels, and thus, use parameter-light
1 convolutions subsequently to focus ouly on individual pixels. Let F denote the 7th channel (or feature map layer) out of total of C' available and ReLU denote the rectified linear unit activation. The x convolution block (Convix1) is formulated as follows:
M, ReLU(Convisi (F, War).. Vi e [1,C"]
N; ReLU(Convixi (M,, W Wn)), Vj [1,
where N C2 32" and WM M [X*xN(+()xXII WN E 32x64x denote the learnable weights. Next, we want to blend information across feature maps for which self-ttention mechanism (Vaswani

---

Published in Transactions on Machine Learning Research (02/2023)
Input
ConvEnc
Support Set Feature Meps
Query Set Festure Maps
F
F
(NN.)
G(N
Conv W Co W
Ni
%%
Attention
AttFEX
G
G2
G3
Figure 3: AttFEX module depicting colors as images and shades as feature maps. We illustrate only 3 image feature maps and 3 channels instead 32 for N, for the sake of simplicity.
et 2017 across Nj.Yj [1.32]. To do so, we feed N to query, key and value extraction networks JoGi W We), (a WR), (o Wy) which are also designed to be convolutions
Q, ReLU (Conv xx (N,, W WQ)), Ei [1, C"):
K, ReLU (Convixi (N,, W )), vi [1]
V, ReLU (Convix (N,, Wr)), Ei [1,C]:
where Wo. K, Wv E 1x322x1x1 are the learnable weights and Q K, V RC "0"x"xxe xIxW"xH' are the query, key and value tensors. Next, cach fcature map N, is mapped to its output tensor G, by computing a weighted of the values, where each weight (within parentheses in equation 5) measures the compatibility (or similarity) between the query and its corresponding key tensor using mnnerepoodue::
exp Q2 K,) dg. De exp (Q, K2)
G j=1
V
(5)
where d; = W' x H', and G, e xI'''xW'X'' Vi. Finally, we transform the original feature maps F by applying Hadamard product between the feature mask G and F, thus, rendering the required feature maps transductive:
8 = G
Here, S and Fº represent the feature maps corresponding to the support and query images, respectively. As a result of operating on this channel-pixel distribution across images in a task, F$ and F are rendcred transductive. Unlike other attention-based few-shot learning methods (Ye et al. 2020 Vinyals et al. 2016)) we do not compute an attention- praset transform on the flattened support and query vectors, but rather the outputs of the Convix (4; Wx) to effectively fuse information from multiple class-pixel comparisons. Note that the query tensor Q must not be confused with the query set Q of task.
4.4 TRIDENT's Transductive ELBO
AttFEX's transductive feature extraction process int rnduces task-level dependencies in the variational for- mulation of ga,- To incorporate this dependency in equation 2 we now revise the derivation of our negative ELBO to be defined in terms of the entire task and not individual data points. Let = x denote the tensor containing all images sampled in task, Y y U e denote all the labels corresponding to the

---

Published in Transactions on Machine Learning Research (02/2023)
images in the task and N' NK + NQ be the total mumber of samples in task. Considering all samples to be independently and identically distributed (T..D.), the likelihood of the entire task can be written as:
N II i=1
/]
p Y)
p(x,, Di Zoc Zis) p(Zei 21s) dzes dzys.
(6)
Since the generative networks Pe, (x Zo, Zi) and pe, (3/ zi) remain inductive, while the approximate inference network qs, (z) x, ze) becomes transductive (via AttFEx), the log- bielliill now becomes:
p(x;
Zeii Zie pp(y))()))))))))) Zis ) (((())) ]
In Inp(X, Y) Σ Bgez
(),
Eges
In
g(Zci
x;)g X, Zes)
Finally, the overall negative ELBO for the entire task can be given by
N' C(V) = Ego Bass Slapos(e (xx; Zeis Zii) + In po. (Mii 2()) +
1=1
(8)
Bugs [DKL (gon (Zie X, Zez ne:iil pl p(zi)]] +
DEL (goz (Zoi xx) p(ze))
Assuming Gaussian distributions for the priors as well the variational distributions allows to compute
the KL Divergences of and ze (last two terms in equation g) analytically (Kingma & Welling 2014]. By considering multivariate Gaussian distribution and multinomial distribution as the likelihood functions for po, [x] Ze, 2)) and Po; (y 4t), respectively, the negative log-likelihood of x becomes the mean squared error (MSE) between the reconstructed images x and the ground-truth images x while the negat ive log- likelihood of y becomes the cross-entropy between the actual labels y and the predicted labels ÿ. After working equation S out, we arrive our overall objective function LR + Lc, where:
N'
CR
01
x - KL(Heei des):
1m
(9)
N
Le - 012
Σ
[yd In (0; n n121)= (4) K(un,,ou)
where KI(H,0) = E (1 (1+ 2ln(an)) (10)2- (od))), [ye]n denotes the n-th dimension of the i-th one- hot encoded ground-truth vector y, D denotes the dimension of the latent space, N is the total number of classes in (N-way, K-shot) task, 01, a2 are constant scaling factors, Me and of denote the mean and variance vectors of context latent distribution, and hi and of denote the mean and variance vectors of label latent distribution. The since the reconstruction loss is in practice three orders of magnitude greater than the cross-ent loss. Moreover, these scaling factors can be understood as gradient- scaling parameters which help improve training in heterogencous likelihoods (Gaussian and Categorical in our case) (Javaloy et al. 2022.
01,02 only scale the evidence lower-bound appropriately,
4.5 Algorithmic Overview and Training Strategy
Overview of TRIDENT. The complete architecture of TRIDENT is illustrated in Fig. 4 The ConvEnc feature extractor and the linear layers Po, (.), 02 (.) constitute the inference network go, of the context latent variable (bottom row of Fig. 4h. The AttFEX module, another ConvEnc, and lincar layers Po, (.) and o3 (.) make up the inference network yo, of the label latent variable (lop row of Fig. 4p_ The proposed approach, TRIDENT, is described in Algorithm[] Note that TRIDENT trained in a MAML Finn et al., 2017) fashion, where depending on the inner or outer loop, the support or query set (g (S, 2)) will be the reference, respectively. First, the lower ConvEnc block extracts feature maps X ConvEnc(X*)) X 's's are then flattened and passed onto (-), o² (.), which respectively output the mean and variance vectors of the
7

---

Published in Transactions on Machine Learning Research (02/2023)
A/0.1)
4
p01
X
ConvEnc
AttFEX
F.
Z
Mo.
Z
Ya
Hos!
+ (2%
X
ConvEnc
Ze
Z --.+
X
gahz
P0a
Figure TRIDENT comprised two intertwined variational networks. Ze is concatenated with output of AttFEX, and used for inferring Zi, where {S, Q}. Next, both Z and Z. used to reconstruct images X while z is used to extract y8.
Algorithm 1: TRIDENT
Require: X X Ys, X where g E (S, 2} Sample: ~00 [Ze Mos (X),, diag( 02 (X
Compute lask-cognizand embeddings: [F
AttFEX(Convvace X)): X x uxe Concatenate Z. and F into [F9, Zo and sample: Z ~ 4s (Z, pss ([F" Z.")), diag(o ((F), Z.")))
Reconstruct X using X = Po, [X| ZJ, Z.")
Extract class-conditional probabilities using: p("" [2) = softmax (Y"12?))
Compute L C C% using equation
Return: L
contert latent distribution, as discussed in equation I This done either for the entire support or the query images X where g E {S, Q} for a given task T, We then sample a set vectors Z (subscript c for contezt) from their corresponding Gaussian distributions using the re- re-parameterization trick (line 1, Algorithm I through the upper ConvEnc, the AttFEX module of 9á, comes into play to create
Upon passing X = x task-cognizant feature maps F" for oither S or Q (line 2). Z together with F are passed onto the lincar layers Mb (-), o (.) generate the mean and variance vectors of the label latent Gaussian distributions (line 3). After sampling the set vectors Z (subscript 1 for label) from their corresponding distributions, we use ZP and Ze to reconstruct images X using the generative network pe; (line 4). Next, ZÍ's are input to the classifier network Po, to generate the class logits, which are normalized using sof tmax (.), resulting in class-conditional probabilities p(Ý' |Z)) (line 5). Finally (in line 6), using the outputs of all the components discussed earlier, we calculate the loss L as formulated in equation S
Training strategy. An important aspect of the training procedure of TRIDENT that its set of parameters V (01,0s, Φ1, Φ2) are meta-learnt by back-propagating through the adaptation procedure the support set, proposed in MAML (Finn al., 2017) and illustrated here in Algorithm 2 This increases the sensitivity of the parameters V towards the loss function for fast adaptation to unseen tasks and reduces gencralization crrors on the query set Q, as discussed from a dynamical systems standpoint in Finn et al. (2017). First, we randomly initialize the parameters V (line 1, Algorithu 2 to compute the objective function over the support set _ (V) using equation 9 and perform a number of gradient descent steps on the parameters & adapt them the support set (lines 5 to 9). This is called the inmer-update and done separately for all the support sets corresponding their B different tasks (line 3). Once the muer-update is computed for each of the B parameter sets, the loss is evaluated on the query set L (2.(1) (V)) (line 12), following
8

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 2: End to End Meta-nraining of TRIDENT
Require: Drr, O; 3, B Randomly initialise = (01, 02.
while not converged do
Sample tasks T. S Q from D for each task Ti do for number of adaptation sleps do Compute (V:(V) (RIDENT(T: ())
6
7
Evaluate V (V)
8
V v<w v aV (V)
9
end
10
(V')
) = w
11
end
12
Compute (2(( TRIDENT () [1
13
Mela-update on Q: v + v vv Σ (V))
end
which a mela-updale conducted over all the corresponding query sets, which involves computing a gradient through gradient procedure as described in Finn et al. (2017) (line 13).
5 Experimental Evaluation
The goal of this section is address the following four questions: (i) How well does TRIDENT perform when compared against the state-of-the-art methods for few-shot classification? (ii) How reliable is TRIDENT in terms of the confidence and uncertainty metries? (iii) How well does TRIDENT perform in cross-domain setting where there is domain shift between the training and testing datasets? (iv) Does TRIDENT actually decouple latent variables?
Benchmark Datasets. We evaluate TRIDENT on the three most commonly adopted datasets: ménilmagenet Ravi & Larochelle (017a)), tieredImagenet. (Ren et al.) 2018) and CUB (Welinder al.) (0010) minélmagenet (Vinyals al.
a subset of ImageNet (Dng al.) 2009 for few-shot classifi- cation. It contains 100 classes with 600 samples each. We follow the predominantly adopted settings of Ravi & Tarochelle (2017s) 7al; Chen et al.
2016
2019 where we split the entire dataset into 64 classes for training, 16 for
validation and 20 for testing. tiered is a larger subset of ImageNet with 608 classes and 779, 165 total images, which are grouped into 34 highor-level nodes the ImageNet human-curated hicrarchy. This set of nodes is partitioned into 20, 6, and disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets. CUB (Welinder al. 2010) dataset has a total of 200 classes, split into training, validation and test sets following Chen et al. (2019). We use this dataset to simulate the effect of a domain shift where the model is first trained
(5-way, 1 or 5-shot) configuration
of miniTmagenee and then tested on the test classes of CUB, used in Chen et al. 2019: Boudiaf et al. 2020
Ziko et al. (020)) Long et al. (2018).
Implementational Details. We use PyTorch Paszke et al. 2019 and learn2learn Arnold et al. 2020 for all our implementations. We use a commonly adopted Conv4 architecture (Ravi & Larochelle, Patacchiola et al., 2020 Afrasiyabi al. 2020 Wang et al. 2019 Boudiat
2017a: Finn et al.] 2017
el al. 2020) ConvEnc to obtain the generic feature maps. Following the standard setting in the lit- Ravi & Larochelle, 2017a]. the Conv4 has four convolutional blocks where each block has a 3 x 3 convolution layer with 32 feature maps, followed by batch normalization (BN) [Ifff & Szegedy 2015) layer, a 2 x 2 max-pooling layer and LeakyReLU(O 2) activation. The gen- erative network Po, for 21 is a classifier with two linear layers and LeakyReLU 2) activation in be- tween, while pe, for Zo consists of four blocks of a 2-D upsampling layer, followed by a 3 convo- lution and LeakyReLU(O.2) activation. Both latent variables z; and z have a dimensionality of 64.
erature (Finn et al.) 2017

---

Published in Transactions on Machine Learning Research (02/2023)
Following Nichol et al. (2018a): Liu et al. (2019):
Vaswani et al. (017),, images are resized to 84 x 84 for all configurations and we train and report test accuracy of (5-way, and 5-shot) settings with 10 query images per class for all datasets. The hyper- parameter values (H.P.) used for training TRIDENT on mönilmagenet and tieredlmagenet are shown in Table ! We apply the same hyperparameters for the cross-domain testing scenario of minilmagenet + CUB used for training TRIDENT on minilmagenet, for the given (N-way, K-shot) configuration. Hyperparameters are kept. fixed throughout training, valida- tion and testing for a given configuration. Adam (Kingma & Baj 2015) optimizer is used for inner and meta-updates. Finally, the query, key and value extraction networks fgl:; WQ), fa(:; Wx), fo(:: Wv) of the AttFEX module only use Convixi(.) and not the LeakyReLU 2) activation function for (5-way, I-shot) tasks, irrespective of the dataset. We observed that utilizing BatchNorm (loffe & Szegedy 2015) in the decoder of ze (p62) to train TRIDENT on (5-way, 5-shot) tasks of ménilmmgeenet and on (5-way, 1-shot) tasks of tieredImagenet leads to better scores and improved stability during training. We used the ReLU activa- tion function instead of LeakyReLU(O.22 2) to carry out training on (5-way, 1-shot) tasks of lieredImagenel. Meta-learning objectives can lead to unstable optimization processes in practice, especially when coupled with stochastic sampling in latent spaces, as also previously observed in Antreas Antoniou et al.] (2019); ; Rusu et al.] (2019). For ease of experimentation, we clip the meta- gratt-grrdient norm at an absolute value of 1. Since AttFEX operates on all samples available in task, scaling to a larger number of ways and shots per task requires more computational resources. TRIDENT converges in 82, 000 and 22, 500 epochs for (5-way, 1-shot) and (5-way, 5-shot) tasks of minilmagenet,, respectively and takes 500 and 48, 000 epochs for convergence on (5-way, 1-shot) and (5-way, 5-shot) tasks of tieredImagenet, respectively. This translates to an average training time of 110 hours on an 11GB NVIDIA 1080Ti GPU. Note that we did not employ any data augmentation, feature averaging or any other data apart from the corresponding training subset Dir during training.
Table 1: H.P. values when training TRIDENT.
miniImagonet
tieoodmmageeet
H.P.
5-way, 1-shot
5-way, 5-shot
5-way, 1-shot
5-way, 5-shot
lc-2
le2
1o-2
le:2
100
100
150
150
a le-3
le-3
1.5e-3
1.7e-3
le-1
le-1
1.5e-1
1.7c-1
B
20
20
20
20
5
5
5
5.1 Evaluation Results
We report test accuracies indicating 95% confidence intervals over 600 tasks for mineImagenet, and 2000
tasks for both tieredImagenet and CUB, is customary across the literature (Chen et al.,
2019
Dhillon
el al.
2020: [aateni et al. 2022). We compare our performance against a wide variely of stale-of-the-at
few-shot classification methods such as: (i) metric-learning (Wang et al.) 2019 Bateni et al. 2020 A Afrasiyabi
ct al.
2020 Yang et al.
) (ii) transductive feature-cxttaction based (Oreshkin ct al. 2018
Ye ct al.
2020
Li et al. 2019 Xu el al. 2021), (ili) optinization--ased (Finn et al.] 2017 Mishra et al.
2018 Oh
et al.
2021
Lee et al.
2019 Rusu et al.
2019), (iv) transductive inference- (Bateni et al. 2022: Boudiaf
et al.
2020
Ziko et al. 2020 Liu et al.
2019). and (v) Bayesian (lakovleva et al.) 2020 Zhang et al. 2019
Hu et
al.
2020 Patacchiola et al. 2020 Ravi & Beatson 2019) approaches. Previous works such Liu
2019g, and Hou et al. (2019) have demonstrated the superiority of transductive inference mods over their inductive counterparts. In this light, we compare against a larger number of transductive (18 baselines) rather than inductive (7 baselines) methods for a fair comparison. It is important to note that TRIDENT is only a transductive feature- extraction based method as we utilize the query set images to extract task-aware feature embeddings; is not a transductive inference based method since we perform inference of class-labels over the entire domain of definition and not just for the selected query samples (Vapnik 2006 Gammerman et al. 1998). The results minilmagenet and tieredImagenet for both (5-way, 1 and 5-shot) settings summarized in Table 2 We accentuate the fact that also compare against Transd- CNAPS+FETI (Bateni et al. 2022). where the authors pre-train the ResNet-18 backbone on the entire train split of Imagenet. We, however, avoid training on additional datasets, in favor of fair comparison with the rest of literature. Regardless of the choice of backbone (simplest in our case), TRIDENT sets a new state-of-the-art on mandImagenet and tieredImagenet for both (5-way, 1 and 5-shot) scttings, offering up to 5% gain over the prior art. Recently, a more challenging cross-domain setting has been proposed for few-shot classification to assess its generalization capabilities to unseen datasets. The commonly adopted setting is
et al.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Table 2: Accuracies in (% std). The predominant methodology of the baselines: Ind.: inductive inference, TF: transductive feature extraction methods, TI: transductive inference methods. Conv: convolutional blocks, RN: ResNet backbone, t: extra data. Style: best and second best. TRIDENT employs a transductive feature extraction module (TF), and the simplest of backbones (Conv4).
miniImaggnet
HeredInagenet
mmn--UUB
Mothods
Backbone
Approach
D-way
1-shot
G-way G-shot
G-way I-shot
E-way G-shot
G-way
1-4hot
D-way
5-shor
MAML
(Fion of 2017
Convé
Tud.
48-70
0.92
51,67
1.81
70.30 0.08
34.01
1.25
48.83
ABML
ToRE Boowee
2010
Comes
Ind.
40.88
0.25
58.19 4 0.17
31.51
0.32
47.80
0.51
OVE
Paaae Booa
Convt
Ind.
45.00
0.24
67.14 1 0.23
37.49
0.11
57.23 0.31
+
16.64
00.22
BOIL Ob
au 2021
Convt
Ind.
49.61
4: 0.16
48.58
4: 0.27
96.45 1: 0.37
89.37
0.12
LFWT Exiiy et m: 2020
3210
TF-2I
06.32
0.50
81.98
0.05
0.75 54.11
47.47
06.96
0.68
FRN Marthrmmr
71:16
0.22
86.01
0.15
77.00
DPGN 1 K024
2012
IF-12
67.77
84.0
72.45
67.24
PAL Ma de me KIP4
3212
TF+CI
00:37 # 0.64
84.10 +: 0.44
T2:25 + 0.72
80.90
0.47
Compiitiini Zilang
3021
TP-TT
0.54
0.50
87.42
0.57
TPMN Wa et
2024
2012
IP-11
67.64 0.63
83.44
0.43
72.24
0.70
86.55
0.65
LIF-EMD [ar
3212
TF+II
66.54 0.28
80.07 #: 0.50
T3.75 = 0.32
87.83
0.50
Truned CNAYS Battee
55.0 0.9
73.1
63.9
Boseline CChen et al
2015
2518
TF
61.87 # 0.77
76.08 + 0.65
42.86
0.00
02.04 = 0.77
PEAT Ye of 2T ATA1)
2218
66.78
82.05
70.80
84.79
50.67
0.78
71.08 0.73
SimpleSmut Warkece 2019
Ind.
03.32
80,28
85,
48.3
Assoe-Alin Airmsyya mi
20120
VE3
1F
69,92 # 0.00
22.5 # 0.55
74.00 = 0.88
86.61 0.50
0.76 47.25
72:37 = 0.89
RoRank
SH nt 2012
VRN
TP+7T
72.4+0.5
80.2+0.4
70.5+0.6
64.8-0.4
TIM-GD
77.8
62.1
89.6
LaplacianSint (233o oy mi
20120
V231
TI
74.9
84.07
80.22
87.49
55.46
06.33
S2M2 Mangla it XUK
61.93
0.18 67.80
83.18 1 0.11
13.71 1 0.22
88,59
0.14
18.21
0.84
70.41 0.75
MetaQIIA (Zhang d
20111b
$1.28
74.33 + 0.85
0.79 99.58
0.72 68.70
71.84
RAVARDAH:: [Hi < 11 21720
VRN
82.7
89.5
83.5
80.0
EASY BBoylou is of MM3
TP451
61.01 1: 0:23
89.14 1: 0.11
84.29 1: 0.24
89.76 0.14
PT+MXP B.
IF-71
0.13
85.67 0.25
90.45 0.14
01.40
0.32
70.51
PRM, E-BNi5
[iu ol m. 20/226
VRN
TP+71
83.35 + 0.25
80.53 + 0.13
96007 + 0.25
91.00 = 0.14
63.90
0.31
79:16 - 0.18
Trand- CNAPSFFETT
2022
IM116!
TP421
70.9
0.8
21.50
1.:00:100 0.4
73.8 2: 0.1
67.7 0.6
(Our)
Comrd
86.11 0.69
95.95
0.28
86.97 = 0.30
96.57 0.17
84.61 0.33
90.74 0.35
where one trains mindlmagenet and tests on CUB (Chen et al 2019). The results of this experiment are also presented in Table 2 We compare against any eristing baselines for which this cross-domain experiment has been conducted. As can be seen, and to the best of our knowledge, TRIDENT again sets a new state- of-the-art by a significant. margin of 20% for (5-way, 1-shot) setting, and 1.5% for (5-way, 5-shot.) setting. Computational Complexity. Most of the re- portod baselines in Table 2 use stronger backbones such as ResNet12, ResNet18 and WRN which contain 11.5, 12.4 and 36.4 millions of parameters respec- tively. On the other hand, we use three Conv4s along with two fully connected layers and an AttFEX mod- ule which accounts for 410,958 and 412,238 param- eters in the (5-way, 1-shot) and (5-way, 5-shot) sce- narios, respectively. This is sunmarized in details in Table 3 Even though we are more parameter heavy than approaches that use single Conv4 as fcature extractor, TRIDENT's total parameters still lies in the same order of magnitude as these approaches. In summary, when it comes to complexity in param- eter space, we are considerably more efficient than the vast majority of the cited competitors.
Table 3: Parameter count of TRIDENT against competitors.
Cocart
2ix
ARSFEX
TRIDENT
Coart
B518
vB3
R Rei
25006
61244
51264
6984
2500
01344
412,238
190, 410
12.4M
36.482M
Bay
22t5
+ 13200
Table 4: Calibration errors of TRIDENT. Style: best and second best.
Metries
MAMEL
PLATIPUS
ABPML
ABML
BMAMML
|AMPIIE
TRIDEN
Beway. L-shot
ECE
0.946
0.022
0.000
0.005
0.0030
MCE
0.03
0.08
0.012
0.068
0.002
0.08
0.029
ECE
00030
1007
0.0015
6-ahoo
MCE
0.444
00300
0.018
Table 5: Style: best and second best.
Methods
ECE
MCE
Brier
Feature Transfer Chen et al.
2019
0.275
0.646
0.772
Baseline Chen et al. 2019
0.315
0.537
0.716
Reliability Metrics. A complementary set of met- rics are typically used in probabilistic settings to measure the uncertainty and reliability of predic- tions. More specifically, expected calibration er- ror (ECE) and maximum calibration error (MCE) respectively measure the expected and maximum binned difference between confidence and accuracy (uuo et al., 2017). This is illustrated in Table where TRIDENT offers superior calibration on miniImaeeeet (5-way, and 5-shot) as compared to other
Proto Nets Sncll et nl. 2017
0.009
0.025
0.604
DoT+C Patacrhinla el al.
2020
0.236
0.426
0.670
RMAML++Chane Chaser Yoom et al.
2018
0.066
0.260
0.639
LogSoftGPPML. (allyFajoon el al 2020
0.220
0.513
0.709
CogSoftCP (PL Caly-Fatoo et ab.
2020
0.022
0.042
0.564
OVE(ML)) Snell & Zemel
Snell
2021
0.049
0.066
0.576
OVE(PI.) Snell & Zeme!
0.020
0.032
0.556
TOuDsNT(OUrs)
0.009
0.02
0.276
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Ablation study for mindmmageent (5-way, 1-shot) tasks. Accuracies in (% + std.).
(5, 3)
(5, 5)
(10, 3)
(10, 5)
(20, 3)
(20, 5)
(B, m))
67.43 + 0.75
09.21 + 0.66
71.6 = 0.81
80.82 + 0.68
86.11 0.59
(dim((x)
(32, 82)
(32, 64)
(32, 128)
(64, 32)
(64, 64)
| (64, 128)
(128, 32)
(128, 64)
(128, 128)
dim(x.))
76.29 1. 0.72
75.44 0.81
79.1 1. 0.57
82.93 1. 0.8
86.11
0.59
85.62
a 0.52
81.49 1. 0.65
82.80
0.48
84.42 0.59
(dim(W v).
(32, 92)
(32, 64)
(32, 128)
(64, 32)
(64, 64)
(64, 128)
(128, 32)
(128, 64)
(128, 128)
dim(Wx))
78.4 0.23
77.80 0.30
79.55 0.87
86.11
0.59
84.87 0.45
82.11 0.35
84.67 0.7
85.8
0.58
83.92 0.63
2017). To further examine the reliability and calibration probabilistic approaches, and MAML (Finn et al. of our method, we assess the ECE, MCE (Guo el al. 2017) and Brier scores (BRIER, 1950) of TRIDENT on the challenging cross-domain scenario of minimageeet + CUB for (5-way, 5-shot) tasks. When compared against other baselines that report these metrics on the aforementioned scenario, TRIDENT proves to be the most calibrated with the best reliability scores. This is shown in Table
5.2 Decoupling Analysis
As a qualitative demonstration, we visualize the la- bel and contert latent means (I and μe) of query images for a randomly selected (5-way, 5-shot) task from the test split of mánilmagenet, before and af- ter the MAML, meta-update procedure. The UMAP (Melnnes et al., 2018) plots in Fig. 5 illustrate sig- nificant improvement in cansscondiiional separation of query samples for label latent space upon meta- update, whereas negligible improvement is visible on the context latent space. This is qualitative evi- dence that Zμ captures more dlass-iisrimminttin in- formation as compared to Z, To substantiate this quantitatively, the clustering capacity of these latent spaces is also measured by the
Z. [Before)
DBCS.79
Zi (Bdame)
Ze (Atte))
DRE44.354
2, (After)
DDB
34-
score (DBI) (Daiees& & Bouldin 1979), where, the lower the DBI score, the better both the inter-cluster sep- aration and intra-cluster "tightness". Fig. 5 shows that the DBI score drops significantly more after meta-update in the case of Zi as compared to Ze, indicating better clustering of features in the former than the latter. This aligns with the proposed de- coupling strategy of TRIDENT and corroborates the validity of our proposition to put an emphasis on la- bel latent information for the downstream few-shot tasks.
vox
Figure 5: Better class separation upon meta-update is confirmed by lower DBI scores. Different colors/ markers indicate classes.
5.3 Ablation Study
We analyze the classification performance of TRIDENT across various paramaters and hyper-parameters, as is summarized in Table We use minilmagenet (5-way, 1-shot) setting to carry out ablation study experiments. To cover different design perspectives, batch size B and number of inner adaption steps n, (ii) latent space dinensionality: 2, and ze to assess the impact of their size, (iii) AttFEX features: number of features extracted by Wa, Wr, W.- Looking al the results, TRIDENT's performance is directly proportional to the number of tasks and inner- adartadaat steps, as is previously demonstrated in Antreas Antoniou et al. (2019); Finn et al. (2017) for MAMI. based training. Regarding latent space dimensions, a correlation between a higher dimension of z and z and a better performance can be observed. Even though, the results show that increasing both dimensions beyond 64
carry out ablation on: (i) MAML-style training parameters: meta-
12

---

Published in Transactions on Machine Learning Research (02/2023)
leads to performance degradation. As such, (64,64) seems to be the sweet spot. Finally, feature space dimensions of AttFEX, the performance improves when WM > W N: and the best performance is achieved when the parameters are set to (64, 32). Notably, the exact set of parameters return the best performance for (5-way, 5-shot) setting. To sum up, (B, n, dim(z;), (zz), dim (Wn) M), dim(WN))) (Wn) = (20, 5, 64, 64, 64, 32) turns out to be the best setting for (5-way, 1-shot), consistently the same for (5-way, 5-shot).
5.4 Impact of AttFEX and the Decoupled Inference Strategy
In order to study the impact of the transductive feature extractor AttFEX, we exclude it dur- ing training and train the remaining architecture. Training proceeds exactly as mentioned in Al- gorithm 2 As can be seen in Table 7 the exclusion of AttFEX from TRIDENT (AttFEX OFF) results in a substantial drop in classification performance across both datasets and task settings. Empirrcallly, this further substantiates the importance of AttFEX's ability to render the feature maps baassductive/takk-aaare. twsk- As explained earlier in section [.3] the derivation of TRIDENT's ELBO implies that y should be included as an input to Q due its dependence on Z1- However, in order to utilize TRIDENT as classifi-
cation and not label reconstruction network, we
choose not to input y to go,(), (-), but rather do so
Table 7: Impact AttFEX on classification accuracies.
indirectly by inducing a semblance of label charac- teristics in the features extracted from the images in task. Thus, it is important to realize that this ability of AttFEX to render feature maps transduc- tive is not just adhoc performance enhancer, but rather an essential part of TRIDENT. To further un- derstand the impact of AttFEX on TRIDENT, we train TRIDENT with a transductive feature extraction module different from AttFEX. The three modules that we replace AttFEX with are:
mandInageonnt
(S-way, 1-shot]
(G-way, 6-shot)
(S-way, 1-shot)
(G-wag. G-aloo)
AcEPEE OPF
67:08 a: 0.55
78.33 a 0.21
69.32 4: 0.75
2033 = 0.76
TRIDENT (EP)
89.84 4: 0.5
30.15 a 0,67
73.29 4: 0.00
82.17 a 0.65
TuLDOr (FEAT)
0.18
82.50 #t 0:45
88.88 # 0.39
TRIDENT (1STM)
75:01 0.40
53.20 0.45
79.79 0.52
86.90 + 0.92
62.35
64.66
TRIDEST(OUrS)
0.50
0:28
0.50
(i) Embedding propagation module (EP): This has been adapted from Embedding Propagation Networks (Rodriguex et al. 2020)- Here, non-parametric graph-based propagation matrix helps smoothen the em- bedding manifold to remove undesirable noise from the support and query feature vectors;
(ii) Attention-bbasd feature adaption module (FEAT): This has been adapted from FEAT (Ye et al. 2020 A self-attention module is used to transform the support and query set by computing a weighted average of all the fcature vectors in a task. The weights are calculated using a dot-product between each pair of feature vectors;
(iii) L-ae- feature adaption module (LSTM): We introduce the LSTM-based transductive task- encoding procedure from Transductive CNAPS (Bateni et all 2022) in place of AttFEX and carry out the same training procedure. The results for each of these experiments, when trained with TRIDENT on minilmagenet and tieredlmagenet, are shown in Table 7
TRIDENT's superior results corroborate the importance of our design choices in AttFEX. Furthermore, to empirically verify the contribution of the decoupled variational inference vs AttFEx, we trained a simplified network ConvFEX = Conv4 + AttFEX as the inference network q(z x) to generate class labels y using an MLP p④y 1)). ConvFEX embodies the inference and generative mechanics of z; while omitting the second latent variable ze, thus dropping the decoupled inference strategy. As shown in Table 7 the classification accuracios across both datascts and task settings for ConvFEX corroborate that when label-specific and context information are coupled, we observe a significant performance degradation as compared to TRIDENT, thus reaffirming the importance of our decoupled variational inference strategy.
5.5 Impact of End-to-End Meta- Meta-Learning
To understand the importance of end-to-end meta-training of the entire network architecture, we train parts of TRIDENT in different steps. More specifically, we pro-train a ConvEnc on the train- ing split of ménilmagenet to perform 64-way classificatinn. Note that during this pre-training phase, training proceeds by sampling random batches from the entire training split without. defin-
13

---

Published in Transactions on Machine Learning Research (02/2023)
ing support query sets. We use the pre-trained feature extractors in TRIDENT's inference net- works to1 and q6 for fine-tuning. We then conduct three different. experiments for fine-tuning the network: (i) freeze both the ConvEnc's and fine-tune episodically without any MAML-style meta- learning; (ii) fine-tune the entire architecture episodically without any MAML-style meta-learning: (iii) freeze both the ConvEnc's and fine-tune using MAML-style meta-learning- Fine-tuning proceeds by sampling (N-way, K-shot) tasks from the train- ing split of manilmagenet. Notably, in (i) and (fi), we do not have separate updates for the support and query sets following simple episodic training. There- fore, employing an MLP for classification is a sub- optimal utilization of the labelled samples. To ad- dress this, we use a prototypical classification frame- work proposed in Prototypical Networks (Snell
Table 8: Impact of meta-learning accuracies.
mandlmagennt
(5-way, I-shot)
(f-way, 5-shot)
Frozen ConvEnc (Episodic)
67.68 0.55
78.53 # 0.21
Fine-tune ConvEnc (Episodie)
69.84 + 0.5
80.15 + 0.67
Frozen ConvEnc (Meta-Learu)
80.11 0.43
87,61 = 0.12
TRIDENT((Orr)
86.11 + 0.59
95.95 +: 0.28
et al. 2017). The results of all the experimentation is illustrated in Table [
It can be observed that episodic fine-tuning is not effective meta-learning the entire network architecture. This can be attributed to the ability of MAML-style meta-learning to render the network's weights sensitive to the loss function, thus enabling quicker generalization to unseen tasks (Finn et al. 2017).
6 Concluding Remarks
We introduce a novel variational inference network (coined as TRIDENT) that simultaneously infers decoupled latent variables representing context and label information of an image. The proposed network is comprised of two intertwined variational sub-nctworks responsible for inferring the context and label information sepa- rately, the latter being enhanced using an attention-baser transductive feature extraction module (AttFEX). Our extensive experimental results corroborate the efficacy of this transductive decoupling strategy on variety of few-shot classification settings demonstrating superior performance and setting new state-of-the- art for the most commonly adopted datasets mini and tieredImagenet as well as for the recent challenging cross-domain scenario of miniimagenet. - CUB. As future work, we plan to demonstrate the applicability of TRIDENT in semi-supervised and unsupervised settings by including the likelihood of unlabelled samples derived from the graphical model. This would render TRIDENT as an all-inclusive holistic approach towards solving few-shot classification.
References
Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In European Conference on Computer Vision, pp. 18-35. Springer, 2020.
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your maml. In ICLR (Poster). OpenReview.nei, 2019.
Sébastien M R Arnold, Praateck Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learm: A library for Meta-Learning research. August 2020. URL
2008.
12284
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogration (CVPR), Junc 2020.
Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, and Frank Wood. Enhancing few-shot image clas- sification with unlabelled examples. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 2796-2805, January 2022.
Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdcloup, Stéphane Pateux, and Vincent Gripon. Easy: Ensemble augmented-shon y-shaped learning: the---rf few-shot classification with simple ingredients, 2022. URL http://aarra...rr/////////...//
14

---

www.:eeuurrccppppeeeeee Published in Transactions on Machine Learning Research (02/2023)
Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, Jose Dolz, Pablo Piantanida, and Ismail Ben Ayed. Infor- mation maximization for few-shot learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2445 2457. Curran Associates, Inc., 2020. URL
GLENN W. BRIER. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78 2.0.CO,, URL httts://journall 1950 1000008880000 10100000000000001001110010011110
(1):1 3, 1950. doi:
ametsoc. amgtssc.orgg/viee//orrraaaaa
1001_1ofeit__ttee vofeit 2_0 ___0______000 xm1
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Wentao Cui and Yuhong Guo. Parameterless transductive feature re-representation for few-shot learning, In Marina Meila and Tong Zhang (eds.), Proceedings of the 98th International Conference on Machine Learning, volume 139 of Proceedings of Machine Leorning Research, pp. 2212-2221. PMLR, 18-24 Jul
2021. URL hhttps://ppooeeeeeeeeeee wtt.eiiinnn.......sss.eeeeeeeeee
mlr wwwpppresssvsie vttp//131//ccuu http://ci2aaaaaa..a
David L. Davics and Donald W. Bouldin. A cluster separation mcasure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2):221 227, 1979. doi: 10.109/TPAMM.19947766000
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pallern recognition, pp. 248 248-255. Ieee, 2009.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference Learning Representations, 2020.
Harrison Edwards and Amos J. Storkey. Towards neural statistician. ArXiv, abs/1606.02185. 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 94th International Conference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pp. 1126-1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In S. Bengio, Н. Wallach, Н. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceddngss http://oo...15500032255255330000 Paperr.pppp.pppppp .ddf.
neurips.cc/papeee 1tt/////////e//e///
Théo Galy-Fajou, Florian Wenzel, Christian Donner, and Manfred Opper. Multi-class gaussian pro- cess classification made conjugate: Efficient inference via data augmentation. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, vol- ume 115 of Proceedings of Machine Learning Research, pp. 755 765. PMLR, 22 25 Jul 2020. URL attps://proceedings mlr. press//11//aaly-aaajooooo
html
A Gammerman, V Vovk, and V Vapnik. Learning by transduction, vol uai 98, 1998.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations, 2019. URL https://openreviiet net/forum? hite:rmmmiiiiiitttS555..
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th Internalional Conference on Machine Learning, volume 70 of Proceedings of PMLR, 06-11 Aug 2017.
Machine Learning Research, pp.
Ruibing Hou, Hong Chang, Bingpeng MA, Shiguang Shan, and Xilin Chen. Cross attention net- work for few-shot classification. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (ods.), Advances Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceedings hwtp./ot....p.p.ooo/////aa.. 11s9aEr002aaaaacceeee770llaaaaa3
15

---

Published in Transactions on Machine Learning Research (02/2023)
Shell Xu Hu, Pablo Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil Lawrence, and Andreas Dami- anou. Empirical bayes transductive meta-learning with synthetic gradients. Tn International Conference on Learning Representations (ICLR), 2020. URL
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution transfer-basod few-shot learning. In International Conference on Artificial Neural Networks, pp. 487 499. Springer, 2021.
Yuqing Hu, Stéphane Pateux, and Vincent Gripon. Squeezing backbone feature distributions to the max for efficient few-shot learning. Algorithmss, 15(5):147, 2022a.
Yuqing Hu, Stéphane Pateux, and Vincent Gripon. Adaptive dimension reduction and variational inference for transductive few-shot classification, 2022b. URL htttp:://axxv..org// httpgg/bb//2005 08527
Ekaterina lakovleva, Jakob Verbeek, and Karteek Alahari. Meta-learning with shared amortized variational inference. In Proceedings of the 37th International Conference on Machine Learning, Procoedings of Machine Learning Research. PMLR, 13 18 Jul 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reduc- ing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456. PMLR, 07 09 Jul 2015.
Adrian Javaloy, Maryam Meghdadi, and Isabel Valera. Mitigating modality collapse in multimodal VAFs via impartial optimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9938 9964. PMLR, 17-23 Jul 2022. URL https://prrceeeddng mlr. press/v162. http/////eejaaaly22aa.a
javaloy22a. httml html
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR (Poster), 2015.
URL http://arxiv. http:///axvvvrrg//////// /a////b///111 0ats/1112.69988 6980.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimencz Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, volume 27, 2014.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differen- tiable convex optimization. In CVPR, 2019.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding Task- Relevant Features for Few-Shot Learning by Category Traversal. In CVPR, 2019.
Junjie Li, Zilei Wang, and Xiaoming Hu. Learning intact features by erasing-inpainting for few-shot classifi- cation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9))8401-8409, May 2021. URL
https://ojaa http://ojsaaaaa.c.ooomm www.orrgggnnn..oeeoooeeeeeeee.et wwwpai.ppp//AA//aataa.a.eeeeeea. phtp/AAAT/aatiicle//eeee///0
Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In Computer Vision -CCVV 2020 16th European Conference, Glasgow, UK, August 29-28, 2020, Proceedings, Part I, 2020.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In International Conference on Learning Representations, 2019.
Mingsheng Long, ZHANGJE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Tne., 2018.
Jiawei Ma, Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, and Wacl Abd-Almageed. Partner-assisted learning for few-shot image classification. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2021.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Puncet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot. learning. In Proceedings of the LEEE/CVE winter conference on applications of computer vision, pp. 2218 2227, 2020.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint ar Xiv: 03426, 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. simple neural attentive meta-learner. In International Conference on Learning Represenlations, 2018.
Cuong C. Nguyen, Thanh-Toan Do, and Gustavo Carnciro. Uncertainty in model-agnostic mota-lcarning using variational inference. CoRR, 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. abs/1803.02999) 02280, 2018a.
CoRR
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018b.
Jachoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. Boil: Towards representation change
for few-shot learning. In International Conference on Learning Representations, 2021. URL https:
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, volume 31, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gegh-performnce deep learning library. In Advonces in Newral Information Processing Systems 32. 2019.
Massimiliano Patacchiola, Jack Turner, Elliot Crowley, Michael F. P. O'Boyle, and Amos J. Storkey, Bayesian meta-learning for the few-shot setting via deep kernels. In NewrIPS, 2020. URL https://
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gra- dients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. AAlch--Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https:
//proceedings. wttpp//www..pppp...........////a
pdf
Sachin Ravi and Alex Beatson. Amortized bayesia meta-learning. In International Conference on Learning Representations, 2019. URL
Sachin Ravi and H. Larochelle, Optimization as model for few-shot learning. In ICLR, 2017a.
Sachin Ravi and Hugo Tarochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017b. URL
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In In- ternational Conference Learning Representations, 2018. URL sttp::://pearevvii...ttttttoooo
HJcSzz-CZ CZ
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Advances in Neurol Infor- mation Processing Systems, volume 32, 2019.
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embodding propagation: Smoother manifold for few-shot classification. In European Conference on Computer Vision, pp. 121- 138. Springer, 2020.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascamu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. Tn International Conference on Learning Representations, 2019. URL
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In In- ternational Conference Learning Representations, 2018. URL https:///ppareview.nnes wot./rouooo.tt// BJJ6qGbRW
Xi SHEN, Yang Xiao, Shell Xu Hu, Othman Shai, and Mathieu Aubry. Re-ranking for image retrieval and transductive few-shot classification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vanghan
(eds.), Advances in Neural Information Processing Systems, 2021. URL http:tpp///ppmmeeeeeeeeee//////t
1d Sde
Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. In International Conference on Learning Representations, 2021. URL https://
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in
Newral Information Processing Systems, volume 30, 2017.
Zhuo Sun, Jijie Wu, Xiaoxu Li, Wenming Yang, and Jing-Hao Xue. Amortized bayesian prototype meta- learning: A new probabilistic meta-learning approach to few-shot image classification. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics, Proceedings of Machine Learning Rescarch, 2021.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pullern Recognition (CVPR), June 2018.
Hung-Yu Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsual Yang. Cross-domain few-shot classification via learned satture-iise transformation. arXiv preprint arXiv:2001.087 2020.
Vladimir Naumovich Vapnik. Estimation of dependences based empirical data. Estimation of Dependences Based on Empirical Data, 2006.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukeuoglu, and Daan Wierstra. Matching net- works for one shot learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighhor classification for few-shot learning, 2019.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Tochnical Report CNS-TR-201044
California Institute of Technology, 2010.
Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map re- construction networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8012 8021, June 2021.
Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Task-aware part mining network for fho-shot learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8433 8442, 2021.
Woijian Xu, yifan xu, Huaijin Wang, and Zhuowen Tu. Attentional constellation nots for few-shot learning. In International Conference on Learning Representations, 2021. URL http:::///paarevieeene/t//oo/o
id=vujTf 18Kmc
18

---

Published in Transactions on Machine Learning Research (02/2023)
Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distribution propagation graph network for few-shot learning. 2020 IERE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13387 1997-13396, 2020.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with sct-to-set functions. In IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR),
pp. 8808 8817, 2020.
Jacsik Yoon, Tacsup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, volume 31, 2018.
Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. Protolype completion with primitive knowledge for few-shot learning, In CVPR, pp. 3754-3762, 2021a.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 1685-1694 2019. doi: 10.1109/ICCH C00.0//ICCVV.21.00177. 2019.00117.
Xucting Zhang, Debin Meng, Henry Gouk, and Timothy M Hospedales. Shallow bayesian meta learning for real-world few-shot recoguition. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pp. 651-660, 2021b.
Imtiaz Masud Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In TCML, pp. 11660-11670, 2020.
19


Reviews:
[{"review_id": "JPPmUWMVqc", "reviewer": "Reviewer_sHKr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Interesting idea\n- relevant topic\n- paper is clearly written and easy to follow\n- SOTA results\n\nWeaknesses\n\n- The effectiveness of the key ingredient of the idea, i.e. jointly learning representations of label and semantics, is not supported with empirical evidence\n- No qualitative examples demonstrating the proposed intuition in action\n- a few unsubstantiated claims\n- ablation studies are vacuous\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a technique for few-shot learning by jointly learning semantic and label latent representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I do not understand what decoupled stands for in the title and throughout the paper. How comes $z_s$ and $z_l$ are decoupled if $z_l$ clearly depends on $z_s$ in Figure 2 and equation (1)? Please fix this.\n- Equation deriving ELBO uses factorization of $p(z_s, z_l)$ assuming $z_s$ and $z_l$ are independent. First, what is the justification for this assumption? Is it really possible that label is independent of semantic information? It is hardly believable that this independence is actually viable in practice. Since this is foundational to the theory of the paper, I believe it is very important to motivate theoretically and empirically that this assumption holds in practice or otherwise modify the theory accordingly so that the algorithm does not need to rely on this assumption. Second, Figure 1 does show the arrow conditioning $z_l$ on $z_s$, which again contradicts the theoretical foundation of the paper.\n- Related work fails to demonstrate where the current work falls in the literature and which research gaps it helps to bridge. Please rewrite accordingly.\n- Relationship and novelty w.r.t. matching networks and other few-shot approaches using attention must be more clearly articulated\n- \"This helps the network utilize information across corresponding pixels of all images in a task $T_i$ which acts as a parametric comparison of classes\". Can you provide empirical support of this claim in the manuscript?\n- \"This increases the sensitivity of the parameters Ψ towards the loss function for fast adaptation to unseen tasks and reduces generalization errors on the query set $Q$.\" Can you provide empirical support of this claim in the manuscript?\n- The main idea and claim of the paper is \"Thus, we argue that attention to label-specific information should be\ningrained into the mechanics of the classifier, decoupling it from semantic information. This becomes even\nmore important in a few-shot setting where the network has to quickly learn from little data.\" I do not see a proof of this concept in the empirical results. Where is the proof that when label-specific information and semantic information are coupled, then performance is worse and when what the authors propose is implemented, performance improves. There is no ablation study like this in the paper and moreover, the section \"Ablation study\" is doing hyperparameter search instead of doing the ablation study. In ablation study we would like to see the effects of adding and removing key components of the proposed architecture, as well observing any interaction effects between them. Please fix this.\n- Another claim in the paper is the $z_s$ learns semantic representation of an image. Please provide empirical evidence supporting this claim. At the moment it is not at all clear what exactly learned in $z_s$ and why it helps (if indeed it does) to learn a better $z_l$.\n- Please prove that $z_s$ helps to learn better $z_l$. What if we do not supervise $z_s$  at all, may be we still get the same performance level?\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A0IChg8tAV", "reviewer": "Reviewer_MEaT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n========\n- the idea of disentangling latent factors in terms of those relating to class labels and those that don’t is a good one, as indeed utilizing only the former set for classification can intuitively be a better fit for few-shot classification problems due to focusing on a more ‘robust’ set of features, which is important in the face of limited task-specific data.\n\n- the proposed method performs very strongly on the problems it is tested on, which is particularly impressive given that it uses a much smaller architecture compared to other works they compare against.\n\nWeaknesses:\n===========\n- the paper has several clarity and writing issues\n\n- the proposed architecture is more complicated than some of the previous work compared against, which makes it harder to understand and deconstruct the improved performance over previous work. While the authors perform some ablations which are helpful, they are not sufficient. For instance, they ablate entirely removing the AttFEX module, but don’t consider simpler architectural designs for that module.\n\n- in terms of experimental setup, the paper focuses on simple benchmarks that the community is largely moving away from, in favor of more diverse few-shot learning benchmarks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose TRIDENT, a transductive few-shot learning method that has a generative model that disentangles label-relevant from label-irrelevant information in the latent space. To this end, they utilize a variational objective has a reconstruction term, which is based on MSE between the generated (via both latent semantic and label information) images and the original images, as well as a classification term, where the class label prediction is made from the label latent variables.\n\nThey also propose the AttFEX module, which is a transductive one that utilizes unlabeled information from both the support and query sets, in order to obtain a set of ‘task-cognizant’ features, which are used for the branch of the inference network that predicts the label-related latent variables. For this, they extract features from all images in the task using a convolutional feature extractor, and then utilize 1x1 convolutions for dimensionality reduction over (individual pixels of) all of the task’s images, thus blending information across classes. Next, they blend information across feature maps via a self-attention layer, the result of which is multiplied with the original non-transductive features (with a Hadamard product) to yield the transductive features. This is done independently for each of the support and query sets of the task.\n\nOverall, they utilize a meta-learning framework to train the parameters in the convolutional feature extractors, AttFEX, and inference networks. Their meta-learning objective is a gradient-based one, where a few steps of gradient descent are performed on the support set to update all parameters (‘inner loop’), followed by an update on the query set (‘outer loop’). In both inner and outer loop, the transductive branch sees both the support and the query set (though without labels), whereas the non-transductive branch uses only the corresponding set (support set in inner loop, query set in outer loop).\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarity and correctness \n==================\nMetric learning is described as learning a “shared feature extractor to embed the samples into a metric space of class prototypes”. This is not correct, as building class prototypes is not a necessary part of all metric learning approaches. For example, out of the ones cited, there are several that don’t create prototypes, like Matching Networks which instead have a nonparameteric flavor. Further, the next sentence talks about how sharing a feature extractor across tasks isn’t necessarily ideal, but not all cited methods in this context share the feature extractor. E.g. Bateni et al (2020) conditions it using an amortized meta-learned model to output parameters for each new task, so this citation for instance should be part of the “task-aware” category instead. \n\nAlso, the definition of “task-aware” in the paper is an unconventional one that involves exploiting unlabeled data. This isn’t necessarily true - there is a vast literature of task-aware methods (also referred to as task-conditioning methods) where task-awareness is built based on the (labeled) support set of the task, like TADAM, CNAPs, Simple CNAPs etc. Perhaps a more appropriate term for what the authors are talking about here is “transductive task-aware”, since they are talking about a specific way of achieving task-awareness that makes additional assumptions (the availability of (additional) unlabeled data) over other task-aware methods.\n\nIn related work, SNAIL is incorrectly classified into the optimization-based meta-learning category. SNAIL is actually a black-box meta-learner that does not apply gradient descent at inference time to solve each new task, therefore it’s not optimization-based. Overall, from reading the intro and related work, it felt on several occasions like the authors haven’t read the papers they cited.\n\nPlease correct the citation style used throughout the paper which is incorrectly used and can be distracting. Use citep to refer to a paper and citet to refer to the authors of the paper. For instance, “Bateni et al. (2022) proposes a method for…” should use the latter, but “A method for … was proposed in (Bateni et al. 2022)” should use the former. \n\nStyle, design and context don’t sound like “semantic” information to me. The term semantics is typically used to refer to information relating to e.g. class information, so I find this terminology odd. A more appropriate term would be perhaps nuisance, style or context variables.\n\nProblem definition: it should be noted that the splits into train/valid/test is done at a *class* level (assuming the authors are following the standard few-shot classification setup).\n\n“The NQ query and NK support images are mutually exclusive” - disjoint is a more appropriate word than mutually exclusive, since we’re referring to sets.\n\nSection 3 presents a framework for variational inference for a graphical model that assumes that images are generated from latent factors corresponding to both “semantics” and “label” information, whereas the latter is responsible for generating the label. However, it is unclear in this section how the loss function presented here fits into the episodic framework. The authors say that the loss is applied separately to both the support and query sets which makes me wonder then, why is it useful to have a support/query separation? The role of the support and query becomes clear later but at this stage causes confusion. I would recommend not mentioning support/query in this section and only describing the variational inference model there.\n\nDo the two feature extractors shown in figure 4 have shared parameters with each other? The use of different colors in this figure made me guess that they don’t, but that would be unintuitive and perhaps wasteful. If they don’t have shared parameters, it would be good to include an empirical comparison to a version that does share them.\n\nI did not understand the difference between transductive feature extraction and transductive inference. It would be good to explain this in more detail.\n\nIn Figure 5 for the visualization, it should be clarified whether the query images used to create this were from the meta-training or meta-validation/meta-testing set. Also, in the paragraph describing this figure (section 5.2), the authors introduce the term ‘meta-adaptation’, while the caption of Fig. 5 calls this ‘meta-update’, neither of which has been defined. I assume this means task-adaptation using the support set, but this needs to be clarified. Note that, if my assumption is correct, the terminology used here is confusing since in meta-learning, ‘meta-update’ refers to the update to the meta-parameters, which is based on the query set loss. Either way, meta-adaptation and meta-update are not synonyms.\n\nMinor: in equation 4, maybe use a symbol other than N to denote the result of the 1x1 convolution, since N denotes the number of classes in each task.\n\n\nExperimental setup\n===============\nIt would be useful to also have ablations with no meta-learning, or less meta-learning, as these would better aid in understanding the success of the proposed approach. This is important as the proposed system is quite complex and involves several components, and it’s not clear whether learning all of them need to be end-to-end meta-learned, or what is the effect of that design choice. For instance, can one pre-train the convolutional feature extractor and inference network for the variational objective to reconstruct images, using batches of examples from the meta-training set, without separate support/query sets? Then this could be kept frozen during the meta-learning phase which can learn e.g. the AttFEX parameters. Pushing further along this direction, could one use a pre-trained convolutional feature extractor (e.g. trained on ImageNet) and integrate that (frozen) into this architecture? Some experiments in this direction would strengthen the paper.\n\nIt would also be great to have other ablations for the transductive component, aside from the current ablation that removes it entirely. What are the architectures used in the cited related works to handle incorporating unlabeled information from the entire task? Can one of these be subbed in in the place of AttFEX?\n\nThe experimental setting is focused on older benchmarks (and thus also precludes comparisons against newer methods). The community has largely moved away from simpler benchmarks like mini- and tiered- imagenet that are relatively homogeneous and evidently don’t require adaptation to the feature extractor for good performance on new few-shot tasks. While the cross-domain scenario explored here is useful, it is quite limited. It would be much more informative to instead use more recent benchmarks for few-shot learning like Meta-Dataset [1], Meta-Dataset + VTAB [2], Orbit [3], MetaAlbum [4], etc.\n\nReferences\n=========\n[1] Meta-Dataset: A dataset of datasets for learning to learn from few examples. Triantafillou et al. ICLR 2020.\n\n[2] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin et al. NeurIPS 2021.\n\n[3] ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition. Massiceti et al. ICCV 2021.\n\n[4] Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification. NeurIPS 2022.\n", "broader_impact_concerns": "I have no broader impact concerns about this paper.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wiYxkxiyid", "reviewer": "Reviewer_Y8UK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts:**\n- The paper is clearly written and easy to follow for the most part. Figures help a lot to understand the proposed approach. \n- The empirical results are strong and the qualitative results are convincing. \n- Literature review is properly conducted, and the paper is mostly self-contained.\n- The idea is intuitive, and all proposed changes are reasonable and well-argued.\n\n**Weaknesses:**\n- The definition of the probabilistic model has some issues that can be misleading (see in next section), as well as some claims in regards to the probabilstic methods are arguable.\n- Limitations of the method are not properly discussed. For example, the model should not scale really well, as it needs to process the entire dataset at once.\n- To my understanding, all results from other models are taken from the original papers. I would've appreciate to reproduce at least one of them with the actual codebase of this paper to make sure that the setups are equivalent.\n- The last experiment (ConvFEX) in section 5.4 is unclear and (personally) unconvincing. Unclear in that I do not understand whether everything was exactly as in the proposed method (e.g., trained using meta-learning), as well as why to use a simplified model. Unconvincing as I would require a deeper discussion or stronger results to show a counter-intuitive result: if the only goal is classification, why wouldn't the model drop all irrelevant information? is it because reconstructing helps aligning the features of query and support variables? \n- As far as I could see, there are no details regarding hyperparameter optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a novel approach to few-shot learning for classification. In layman's terms, the proposed method is a hierarchical VAE in which the first latent variable $z_s$ provides semantic information (that information not important for classification), and the second latent variable $z_l$ embodies the information useful for classification. The authors make a number of modifications to make the model work in their settings, being the most important that:\n1. The inference model does not depend on the label (so that it can be used in the during testing).\n2. To cope with the loss of information, the authors shift to the transductive setting, using attention on query and support data *at once* to extract the relevant features (making it \"task-aware\").\n3. Finally, the model is trained using meta-learning in the same fashion as MAML models.\n\nThese key changes enable a model which convincingly outperform existing methods using far less parameters than the average model, as empirically demonstrasted in their experiments.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes to secure my recommendation:\n\n1. The probabilistic model introduced in section 4.2 is misleading, in the sense that it is defined per sample $(x, y)$, but the proposed AttFEX introduces dataset-dependent dependences that are not contemplated anywhere. To my eyes, the probabilistic model should be defined in terms of the entire dataset $\\mathcal{D}$, rather than individual points. While the generative network is inductive $p(\\mathcal{D}|\\mathcal{Z}) = \\prod_i p(x_i | z_i)$, the inference network is transductive like $q(\\mathcal{Z}|\\mathcal{D}) = \\prod_i q(z_i | \\mathcal{D})$. This is a *huge* difference which is not reflected in the model. To help rewriting the probabilistic model, I would suggest looking at [Neural Processes](https://arxiv.org/abs/1807.01622), which also perform variational inference over the entire dataset.\n2. Equation 2 is wrong, as the first KL should be inside an expectation w.r.t. $q_{\\phi_2}(z_s|x)$. Also, that equation is the *negative* ELBO.\n3. While I can understand $\\alpha_1$ as the normalizing constant of a fixed variance Gaussian likelihood, $\\alpha_2$ does not make sense in probabilistic terms and there is no such a thing as \"being considered variational inference by consensus.\" Scaling factors such as those proposed can be understood instead as gradient-scaling parameters *during training*, see [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://arxiv.org/abs/2206.04496). This same paper also showed that properly tuning these parameters can improve the training in heterogeneous likelihoods (such as Gaussian + Categorical).\n4. In section 5.4, the phrase \"it is imperative to include y in the input to $q_{\\phi_1}(.)$ for mathematical correctness of the variational inference formulation\" is wrong. The choice of variational family is up to the user and will always be correct (another matter is wether it is feasible or sensible).\n\nChanges that would strengthen the paper:\n\n5. Formatting could be improved: properly using `\\citep` and `\\citet`, moving line numbers of algorithms outside the margins, remove the short-line at the beginning of page 11, correcting typos (e.g. remove doubled `equation equation` within the text), etc. \n6. I'd make sure that the $86.11 \\pm 0.59$ in Table 6 is correct, as it appears twice in the table.\n7. I would mention meta-learning somewhere in the abstract, and definitely earlier than where it appears right now.\n8. In problem definition, letter $K$ is used twice for difference things. Something similar goes for $\\alpha$, $\\alpha_1$ and $\\alpha_2$. Indexes of $G_i$ after Equation 6 are wrong $1 \\times C'$ instead of $C' \\times 1$.\n9. It is a bit odd to pass $X_{CE}^g$ as input in Algorithm 1, rather than computing it within the algorithm.", "broader_impact_concerns": "I don't have any concerns regarding the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model for few-shot classification that makes use of two latent representations of an image, each with a different semantic content (one for context information, one for label information). The paper also introduces an attention-based network to extract features that exploits information from both query and support images of the few-shot task. The effectiveness of the proposed model and network, trained with variational inference, is demonstrated through a thorough set of experiments.

In the initial version of the paper, the reviewers raised some valid concerns, including: clarity/writing issues (including some mistakes in the equation), missing ablation studies, choice of baselines, and incomplete citations. This was mostly fixed by the authors through an extensive revision, in line with the reviewers' suggestions.

After the authors' rebuttal, Reviewers MEaT and Y8UK recognized that the paper had been significantly improved and that their concerns had been addressed. Reviewer sHKr, however, still has some concerns and leans towards rejection. Specifically:
+ The concerns about "decoupling" and "inference" seem to be caused by a mere nomenclature misunderstanding. Here, "decoupling" refers to being able to recover two latent representations, each with different semantic content. Perhaps a less confusing way of stating that would be dropping the word "decouple" and simply states that the model considers both types of latent variables, but I think it is also fine to leave as-is. The term "inference" refers to the process of learning the latent variables through approximate posterior inference (as opposed to "at test time"). In this sense, the model and inference are allowed to be different, and any choice of the variational model will lead to a valid ELBO - this should also address the point about the "theoretical justification".
+ Regarding the *missing citations*, the authors have now added citations to original work in Section 4.5, as well as a connection to matching networks in Section 4.3.
+ The *ablation analysis* in Table 7 shows the importance of the proposed architecture, while Section 5.4 (last paragraph) also analyzes the importance of having two decoupled latent variables.
+ *Figure 5* shows empirically how $z_l$ captures label information whereas $z_c$ doesn't. As a **suggestion to the authors**, another way to quantify that would be to compute some mutual information between $Z_l$ and the labels (and between $Z_c$ and the labels); it may be nice to include that in the text as well.

I thus recommend acceptance. In the discussion period, the reviewers also mentioned the paper can be improved as there are some remianing writing issues. I encourage the authors to take a full pass over the text and also to take care of any of the suggestions from the reviewers that haven't been incorporated yet.

Finally, a couple of minor comments:
+ Please replace "Fig. 5.2" with "Fig. 5".
+ Please add a cross-link in the last paragraph of Section 5.4.
Decision: Accept (poster)