Robust Cross-Modal Semi-supervised Few Shot Learning
Authors: Xu Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
noisy labels
variational inference
cross-modality
uncertainty
robustness

Abstract: Semi-supervised learning has been successfully applied to few-shot
learning (FSL) due to its capability of leveraging the information
of limited labeled data and massive unlabeled data. However, in many
realistic applications, the query and support sets provided for FSL
are potentially noisy or unreadable where the noise exists in both
corrupted labels and outliers. Motivated by that, we propose to
employ a robust cross-modal semi-supervised few-shot learning
(RCFSL) based on Bayesian deep learning. By placing the uncertainty
prior on top of the parameters of infinite Gaussian mixture model
for noisy input, multi-modality information from image and text data
are integrated into a robust heterogenous variational autoencoder.
Subsequently, a robust divergence measure is employed to further
enhance the robustness, where a novel variational lower bound is
derived and optimized to infer the network parameters. Finally, a robust semi-supervised
generative adversarial network is employed to generate robust
features to compensate data sparsity in few shot learning and a
joint optimization is applied for training and inference. Our
approach is more parameter-efficient, scalable and adaptable
compared to previous approaches. Superior performances over the
state-of-the-art on multiple benchmark multi-modal dataset are
demonstrated given the complicated noise for semi-supervised
few-shot learning.

Under review as a conference paper at ICLR 2022
ROBUST CROSS-MODAL SEMI- SHOT LEARNING
FEW
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-supervised learning has been successfully applied few-shot learning (FSL) due to its capability of leveraging the information of limited labeled data and massive unlabeled data. However, in many realistic applications, the query and support sets provided for FSL are potentially noisy unreadable where the noise exists in both corrupted labels and outliers. Motivated by that, we pro- pose to employ robust image-text multi-modal semi-supervised few-shot learn- ing (RCFSL) based on Bayesian deep learning. By placing the uncertainty prior on top of the parameters of infinite Gaussian mixture model for noisy input, multi- modality information from image and text data are integrated into a robust het- erogenous variational autoencoder. Subsequently, robust divergence measure is employed to further enhance the robustness, where a novel variational lower bound is derived and optimized to infer the network parameters. Finally, robust semi-supervised generative adversarial network is employed to generate robust features to compensate data sparsity in few shot learning and joint optimization is applied for training and inference. Our approach is more parameter-efficien scalable and adaptable compared to previous approaches. Superior performances over the tate-of-theear approaches on multiple benchmark multi-modal dataset are demonstrated given complicated noise for semi-supervised few-shot learning.
INTRODUCTION
Despite the impressive success of deep learning models, frequeenly requires massive amount of training data to fully demonstrate the potential of the model. In contrast, human is capable of learn- ing new concepts given limited data. Consequently, few-shot learning gathers extensive research interest due to the capabilities of learning new concepts from limited training data. Nevertheless, the success of few-shot learning requires careful handling to robustness and generalization as is extremely susceptible to noisy labels, outliers well as adversarial attack Lu et al. (2020a). For instance, in order to automatically recognize several kinds of uncommon animals, only a few anno- tated images for them are available due their rarity. Moreover, the images could potentially be corrupted due to an uncontrollable shooting environment or an instrumental malfunction. To miti- gate this, common approach to few shot learning is meta-learning Ren al. (2018), where the goal is to learn a classifier to distinguish between previously unseen classes, given labeled classes and larger pool of unseen examples, some of which may belong the classes of interest, namely semi-supervised few shot learning (SFSL).
Despite the impressive capabilities equipped the ability leverage unlabeled examples for SFSL, the challenge of lacking novel samples remains to be a bottleneck. Besides visual information, textual data frequently contains rich information and more descriptive concepts for learning. Incorporating image-text mudti-modal learning into the framework by training on image-text pairs provides an ef-
ficient tool to inject the diversity to the generation process Pahde1 et al. (2021 Pahdel et (2018).
The work in Pahdel et al. (2018) provides benchmark for multimodal few-shot learning re relying on a class-discimminaive text conditional generative adversarial network. Later on, on, Pahdel et al. (2021)
tackles the multimodal few shot learning problem by employing cross-moda feature generation network to infer the class membership of unseen samples with simple nearest neighbor approach. Despite of the success of these methods with clean features and perfect labels, the important that features and labels are contaminated due to out-of-distribution samples, adversarial attack and human fatigue is rarely studied. In parallel, Bayesian deep learning (BDL) has served powerful

---

Under review as a conference paper at ICLR 2022
The overall blockdiagram of the proposed robust cross-modal semi-supervised few shot learning (RCFSL) framework to leverage both the advantage of robust GAN as high quality gen- erative model and a robust heterogenous semi-supervised VAE as a posterior distribution learner on multi-modaling data. Robust multimodal prototypes are then calculated from the robust embed- ding using the last hidden layer the discriminator of RCFSL, where three novel components are highlighted in the orange bounding boxes.
Figure 1:
Uncertainty
prior for mean
Part
Robust diverpence measurre
muitimadai protolype
Infinite Gaussian mixture models
Heterogenous Encoder q(z[x)
p(j (oe))
Latent Space
Decoder
pix|
Images
D/C
Word embedding
Part
p 0 (2)) P M171(=
with large black beak and black eve ring.
Integration of robust hetegenous VAE with robust GAN
this largc pira nas a distinctive curvea Dill, white markings on it's eyebrow and lace, and a dark gray body...
Part
tool in terms of transforming the problem of posterior inference of a BDL model into the optimiza- tion of objective function based latent variables. Now the question then is: how design Bayesian deep learning which counters the noisy labels and outliers jointly in the multimodal semi- supervised few-shot learning. Accordingly, this paper tackles this challenging problem in robust cross-modal few-shot learning by integrating a deep generative heterogenous model that general- izes well to multi-modality (e.g. image-text modeling) in order to counter noisy labels and outliers. Specifically, a robust heterogeneous variational auto-encoder is first proposed to encode the noisy visual features and labels order to jointly learn the information from both modalities by placing the uncertainty prior the top of the infinite Gaussian mixture models. Subsequently, robust variational lower bound based on 3-divergence derived to infer the network parameters. Finally, robust semi-supervise supervisd GAN is integrated with the heterogenous variational auto-encoder by col- lapsing the generator and the decoder into one to further boost the learning capabilities. RCFSI is built top of Bayesian deep learning by fusing cross-modal information via the approximation of the joint posterior distributions. In contrast to modality-alignmeet al. (2017) for robust few-shot learning, the robustness of RCFSL achieved by accurate model- ing of the complicated joint distribution of mudti-modality data and robust variational inference by the derived lower bound. Distinct from the work in Xing et al. (2019)) calculating linear combina- tions in the prototypical representation space, our fusion of multimodality features in the probability distribution is completely data-driven, yielding more robust classification performance in few-shot learning. Major contributions of this paper are: (1) RCFSI harnesses three levels of denoising to ensure the robustness of cross-modal semi-supervised few-shot learning: Firstly, motivated by Hou & Galata 2008), it places the uncertainty prior of the parameters of an infinite Gaussian mixture distribution of image data to avoid mixture components collapsing onto a point or a hyperplane due to outliers. (2) Subsequently, the robust (-diveegeenc is employed to replace Kullback-Leible di- vergence used for data fitting to infer the network parameters and novel evidence lower bound for semi-supervised few shot learning is derived. (3) Noise-transition layers are applied to both the heterogenous variational encoder and the robust discriminator in semi-supervised learning with an end-to-end training. The performance of RCFSL further boosted with robust feature generation yielding 7% to 10% absolute accuracy improvement over STOA approaches.
methods Xing et al. (2019a) Tsai
Related Work Previous work in multimodal few shot learning frequently tends to first learn text to image mapping to generate additional visual features and then calculate the joint prototype us- ing a weighted average from two representations. Two recent approaches have attracted signif- icant attention in the few-shot learning domain: Matching Networks Vinyals et al.] (2016), and Prototypical Networks [nell et al.] (2017) where the sample set and the query set are embedded with a neural network, and nearest neighbor classification is exploited relying embedded space. In Oreshkin et al. (2018), metric scaling and metric task conditioning are uti- lized to improve the performance of few-shot learning algorithms. Kim et al. (008)) and Finn
metric in the
2

---

Under review as a conference paper at ICLR 2022
et al. (2018) employ a probabilistic extension model-agnostic meta-learning (MAML) framework trained with variational approximation so that the model can generalize well to a new task with few fine-tuning updates. In Zhang et al. (2020), a bidirectional joint image-text modeling was proposed and VHEE-raster-scan-GiMM was applied. RCFSI, advances the work from /Zhang et al.) (2020) by improving the robustness of the multi-modal heterogenous encoder and extend the lution to semi-supervised few shot learning. In Tseng et al.] (2020). feature-wise transformation layers are utilized for augmenting the image features relying on affine transforms simulate vari- ous feature distributions under different domains for few-shot learning. Different from [Tseng et al. (2020), RCFSL augments the robust feature generation from BDL perspective relying on robust semi-supervised GAN. Moreover, our method advances from other robust few shot learning such as Rapnets Lu et al. (2020b) and Adversarial Query Goldblum et al. (2020) by providing mathmatically rigoriously denoising schemes via uncertainty priors and robust divergence in variational inference.
OUR METHOD
Our approach is to focus first constructing a semi-supervised robust heterogeneous varia- tional autoencoder leveraging mixture model to encode both image and text data few shot learning insensitive both noisy labels and outliers. Subsequently, a novel robust variational lower bound is derived facili- tate the inference of network parameters rely- ing on 3 3-divergence for both labeled and un- labeled data. Finally, a robust generative adver- sairial network is integrated with denoising lay- ers to strengthen the denoising performance to- gehter with the end-to-end optimization to gen- erate additional visual features to alleviate the sparsity in the semi-supervised few shot learn- ing. Let Ω denote image space, T denote text space and C = (1...., .....,, R) be the discrete la- bel space. Further let x; Ω the i-th input image observation, t; E T as its correspond- ing textual description and I/ E C' as its la- bel. Denote Cbuse as base classes where we have both labeled and unlabeled samples and denote Crovet novel classes, which are under- represented in the data. Inspired by the fact that the student-t distribution is more robust to the outliers than Gaussian distribution by con- straining the shapes of the mixture components from collapsing Hou & Galata (008), we pro- pose place uncertainty priors (e.g. Gaussian
x
The probabilistic graph model of the proposed RCFSL framework, where x denotes the observations of image data and represents the observations of embedded feature vectors for text, ÿ and refer noisy labels and corrected labels respectively, and stand for the latent variable and auxiliary variable for VAE respectively, serv- ing bridge fuse two modalities. Te and Ck represent the mean and the covariance for the Gaussian component of image data. λ represents the parameter in the Poisson distribution for the /th feature. Each observation pair (I;, t,) depends on their cluster assignment and distribution pa- rameters and each cluster assignment depends on the stick-breaking procedure Beta((K - 1)a, x).
Figure 2:
priors) on parameters of infinite Gaussian mixture models to characterize the influence of outliers on image data and constrain the shape of the components prevent them from collapsing.
The Heterogeneous Mixture Model: Variational autoencoder Diederik & Welling (2013) has been recently proposed as a powerful solution for semi-supervised learning. To the best of our knowledge, this is the first time that robust heterogenous VAE model has been applied which naturally inte- grates noisy images and text together which cohesively fuse continuous and discrete multi-mode modality features. Variational inference are applied to fit the heterogenous model on both of image and text features v from base classes Chase. where the embedding is obtained from the last dense layer right before the softmax output in the discriminator. Once good mapping V based on heteroge- nous image and text data is learnt, given test sample, the class membership is given by assigning the class label of the closest prototype to unseen test sample. In particular, the heterogenous features are first fed into the Dirichlet process mixture and clustered based on their similarity mea- sures, where here image features are modeled infinite Gaussian mixture distribution Allen et al.
3

---

Under review as a conference paper at ICLR 2022
2019) to characterize the samples from the unseen classes by computing the posterior distribu- tion for unrepresented clusters more accurately and inferring the number of classes automatically and text features are characterized by Poisson distribution. Specifically, 2n is noisy measure- ment of its true position and is a draw from the Gaussian mixture model, where the mean of each Gaussian component Tk is unknown and the variance Ck is known. In order to characterize the uncertainty and the outliers from the input, the Gaussian prior is placed on the top of the mean for each Gaussian component. Namely, Th satisfies the normal distribution with the mean Pa and the precision matrix Ag. Wi is the latent variable for the ith data point specifying which Gaussian it came from and the mixing weight for the Gaussian mixture model. Specifically, a Normal- Wishart prior MMupphy (007)) is placed on the mean and precision of the Gaussian components: A) I N (ML be a very small value. W(AjW,,) v) is a Wishart distribution with scale matrix W and v degrees of freedom. For text features after word embeddings, ej and Çi are the prior shape and rate parameters respectively in the Gamma distribution that generates the average rate parameter for the 1-th Pois- son feature. In particular, the cluster assignments for each observation are drawn from multinomial distributions where the prior parameters represent the mixing weights of the corresponding clusters. Specifically, the truncated stick- breckbrg process is employed to construct mixing weights. By con- struction, single parameter a controls the portion of the K - 1 major clusters, and separately controls for the portion of the remainder via stick-breaking procedure Beta((K' - 1)), x) Blei & Jordan 2017). Namely, where w [w],..... -wi,...,KKK _wk] belongs to the (K - 1)- simplex and is generated from Dirichlet prior with two parameters p(w) Dir())irr.......
(R, mo, (SoAA))))((AA)
My) where mo is set to be zero and Bg is set to
a, x). Typically, given the image data æ and the textual features f, variational inference with deep learning from powerful probabilistic models constructed by inference neural network q=2[22, t) and a generative neu- ral network p(r, t/2). The generative model and the inference model are parameterized by 0 and ø respectively. Subsequently, the network parameters estimated by optimizing the evidence lower bound (ELBO) in the variational inference. For unsupervised learning of multimodality data, the vanilla VAEs are optimized by maximizing ELBO:
(I) The Generative and Inference Model: We shall now present the proposed generative and in- ference model for few-shot learning. Given a semi-supervised learning setting, the labels y are either unknown (for unlabeled data) noisy (for labeled data). The generative model is then defined as: po(x, tho C)po (TIZ, nt A, A)po(alz, y, x. t) (z, t/ū, ())((())()))p(()((()) pa as the deep neural network with parameters and y as the ground truth of class label. For unlabeled data, y is considered as latent variable. Further denote Cat(.) as multinomial distribution and in this paper we reasonably assume that the labels are of categorical distribu- tion and the proposed model applies to other distributions for the latent variable y. In order to fit more complicated posteriors for the marginal distribution p(z|æ; t) and bridge the gap be- tween two modalities, motivated by Maaloe et al. (2017), we extend the variational distribution with auxiliary variables a, so that the generative model is invariant to marginalization over a p(z, z, a, w, T, C, μ, A) po (, C)pp' ence with the noisy labels, further denote ÿ as the corrupted labels and @ as the corrected label after denoising layer. Define K x K noise transition matrix M to associate with ğ and estimation of M has been addressed in previous methods Sukhhaatar et al. (2015) Patrini et al. (2017). In particular, M = (Mr,) E [0, 1) [x, Mij 1). The proposed generative model can then be expressed as: p(z) = (= I). p(O) = Cat(ūl)) po(alz, ÿ. x. t) f(a; 2, v. t), Pa (x, ()) = [xx t: 8. P(Telun A&) = N(4auu,, A (.')).( P(p = 1lū = j) = Ma (2)
ELBO
(L.)) L(x) := E=~0(=1221) [In p(t[=)] Dxz t)/ D//p(=)]
Define
u, A)pp (a) x x))p(1, z). To attenuate the influ-
p(z, N/,,t/T, X) = N(x. (te. Cx) II N (t Cr.,) if the jth cluster is represented
Atexp
k!
teexp= R
p(z, t/T, C, X) = p(z,)tx, Cx)p(tal Awlp(u, A Ax)duudA if the jthh cluster is unrepresented
The inference model can be represented as:
2, A, G, A/rr, t) = g(2 a, J. t)a(a/a, 20, H, U,A, A, A/2r, (0(2/a, ÿ. x, = N(>luo(a U. x. t), diag(o?)), go (ūla, x, = (ol I, t)), (Me,, As) g( x Gamma (Niles + +, Phatu, &r Σ, Wik))
(3)
4

---

Under review as a conference paper at ICLR 2022
where (t) characterizes posterior variational density of Poisson parameters given the discrete embedded feature vectors t converted from text data. Vix stands for the probability of the ith obser- vation belongs to the cluster k. represents the 1-th Poisson feature of the ith observation. Denote N-n.j as the number of data points, excluding En which belongs to the mixture component j. To compute q(T, 14, A, A/zr, t), mean-field approximation is applied Bishop (2006) factorize all the latent variables and parameters for the represented and unrepresented 5th cluster respectively:
(T, p4, A, N-1+0
N-m (I [t,, Ck Ih N(t., It.. Cg.)a(Avl))
(4) The above equation indicates that given the multimodality data, the unlabeled samples has a certain probability of being classified unrepresented mixtures (e.g. unseen new classes), which facilitates the learning capabilities of the proposed robust remi-supervised few shot learning. Within an iter- ation, some unlabeled samples are associated with unrepresented mixtures, new represented mix- tures will emerge which successfully addressed the challenges in semi semi-supervised few shot learning when the unlabeled data contains unseen classes which doesnot exist in labeled data.
(T. x P(z.llt Cx)p(ts/pu Ax) AA Aa)dpddAee(AA()))
Robust Variational Lower Bound: To further alleviate the impact from outliers, the robust di- vergence is employed to infer network parameters more accurately. The theoretical foundation of 3-divergence has been initially defined at Basu et al. (998), where the 3-divergence between two functions y and f are defined as
Do(0 fl 3 ʃ ((x)) dx / f(x)) B+1 / (zx))(x))dd
(5) When B + O, the 3-divergence converges to KL-divergence, limg- Dola [] = DEL s. As described in Futami et al. (2018), minimizing the 3-divergence from the empirical distribution p(x) to p(a; 8) arg ming D (f(x) Il 6)). it is easy to show >>=1p(r;; p(x;; B)@ o p(x,)))
80
Ep(x:07 [p(z;; 8)8 BIn pfar1]] As the probability densities of outliers are usually much smaller than those of inliers, the first term of the above equation is the likelihood weighted according to the power of the probability density for each sample, which effectively suppress the likelihood of outliers. This estimator is also called as M M-sstimator| Huberr & Ronchetti (2011), which provides provably superior performance in various machine applications Li & Gal (2017). The variational lower bound for the proposed RCFSL model for labeled data can be represented as
20
log p(x, 8. fo J. Jr JA Jxx fo (2, h, a, 2, T, μ. A, X, dpdd:ddTddddddd N E[log(po(a, 2, T, A, A, w. x, 7. t))] - T A.
= Ellog(po 2, u, 2, 1)))
-E] El Elgoc=la.g....))
Elqa(r n. Elgo(an)] E[ga(woo) E[qacvol
The above inequality can be rewritten as
logp(x, J. > Eggta. M.s.A. Dog Pol..a.T.n.AAA......
Roco A.
[I, t, gla 2, T, μt A, w)) + ( 2, T, n. A, A/z, t. w///p/////////p// (,, (6)
To mitigate the influence of outliers, let H fa, T,W, μ. A, ^} represent the set of all the latent variables and leverage the technique from Futami et al. (2018), we can replace KL-divergence with 3- 3-Divergence and cast the 3-ELBO for labeled data Li
N
T,g (H||,, 5)
3 + 1
Σ
t,) + N M0 1) + D tD)/ (7)
B
For unlabeled data, by introducing the variational distribution for ÿ as go x. tly),, the variational lower bound for the proposed RCFSI can be represented as
logp(x, t) = Ja f fr JA Ig (og( 5. a, 2, T, ,t, (dadzdTdppdd 2 E.s.a...s..T...l..
[log Pela.s.T.... T. = E Eilog(po (a, 2, μ, A, w, X, x. t, 7 E 1( E(as((la..))
-Eleccsaa.g.r.))
El p...a) E( E E[gs(w m Elge(Ano)
Similarly, replacing the KL-divergence with 3 3-Divergence and augmenting the latent variable H., (a, z, y. T, w. A, A}. the 3-ELBO for unlabeled data in RCFSL is:
U3 = ((Hul),,)) t)
3+1 +1 B
Σ
N
p(a,, t Ha) + +f 2H. dx) ) + Dxulg(HH,lp, (//ppHHHH.
5

---

Under review as a conference paper at ICLR 2022
Practically. Lg and Ug are calculated via Monte Carlo sampling. The robustness of pro- posed ELBO can be guaranteed leveraging the influence function Futamii et al. (2018 Huber & Ronchetti 20IT). As IF is widely used to analyze how much contamination affects estimated statis- tics, is straightforward to show that given the perturbation on the empirical cumulative distribution caused by outliers, it is straightforward to show that the IF of our posterior distribution is bounded. The objective function for robust rross-modal variational autoencoder is: LCC AB Lg + >1U3 where A represents the weight control the trade-off between the labeled data and unlabeled data. Robust Feature Generation: The construction of the proposed uncertainty priors and the robust divergence measure in our framework aims at better approximation of the posterior distribution un- der noisy labels and outliers. This is also related to employing generative adversarial learning of and x t defined by the data, the encoder, the prior and decoder. Different from VAES that assumes parametric data distribution and perform posterior inference, GANs in general utilize implicit data distribution and do not provide meaningful latent representations. By learning both a generator G and discriminator D, min-max objective is optimized:
min max (ax (Erttrrsactt (1..) [In (z) t))] + Ex-p(=) [In(1 D(G(=))
(8)
G
In Kaneko et al. 2019) noise transition model is incorporated to learn clean label conditional generative distribution. But their model only considered noisy labels without outliers and is lim- ited to supervised learning. Recent work semi-supervised GAN with k classes Kumar et al.. (2017 Salimans et al. (2016) modify the discriminator with & - 1 outputs on the discriminator by considering the fake images as the k + 1th class. Hence, the loss for the training can be computed with a supervised loss and an unsupervised loss respectively:
=
= Lsup + umsup = - -E,, (pp...p log [ul 1, S k)
Feg ( log(ps(v = k- + ||, t)) E (=...) log(1 - py = k + xx, t)) where L Laup represents the negative log probability of the label given the data is real. De- note D(r, t) 1 - 1-pryy = k + 1|x; t), the loss for semi-supervised GAN on multimodal-
(9)
ity data LAGAN can be written as: L = (x.,)) log D(ar,t) Ez-noise (1 D(G(=)))
Fm..sppalatt.. log (o t, S. k)- More recently, the label noise robust GAN (rGAN) Kaneko jet al.] (2019) has achieved promising performance in classifying images with noisy labels by in- corporating a noise transition model to learn clean label conditional generative distribution under noisy labels. Thus, given the noise samples "") ~ Pa(a, D), to construct label-noise robust conditional generator and discriminator, the objective function of the robust semi-supervised GAN in the proposed robust pross-modes remi-supervised few-shot learning is expressed as: LRSGAN = (z..)) log(1 [1 - Č(ū = + x t)) - Exxnnosee (1 (1-D ((-(())())))-)) -E -.(x)_x((....)) log C(ū I, S k) -E,,--patz. log(1 - Mg . Cgp k + 1|x, f)) - Ez~noise (1 D(G(2))
E(ar ((7 log Eg- Mg . Cli = 8" |r. 1 < k). Without modeling the data distribution explicitly and representing the latent space in meaningful manner, does not provide the function- ality to counter the outliers in the generator and discriminator. Motivated by the thought of bridging the gap between robust VAE and robust GAN, we feed the robust variational posterior p(z[z, t) in- stead of the random noise p(2) into the label noise robust semi-supervised GAN as the source of randomness as both of the decoder and the generator of RSGAN share the mapping from 2 to x and t. The full optimization function for the proposed RCFSL framework can be represented as:
minG max F. F (x, (T(x, L t) == In D(r, 1) + DRL. (g H ar, 1, 0) /// (H)) +
X₂ De 1)//p(H)))))))))))))))))))(())) + E (( [In(1 D(GGRGANN()) (- 8+1 N H( 11 dy] + A1 Er 2+1 E. p(z,, (H,)* + A1 p(z, ((,,) dx)] (10)
# MN p(ū 21; t;)
where A1 is set to be the ratio of unlabled samples verus labeled samples and the discrimina- tor loss is characterized by ElIn D(x, = ----------- (E log(1 - Mg -5..s k + 1|,, t)) Eaaa 1.07))(u(2. log Eg- Ma .7 C(j = Ú" S k) This new architecture is expected to better predict the class labels under the compound noise. We then train RCFSL including the robust heterogenous encoder, the generator and the robust discriminator in an end-to-end manner using adaptive moment estimation (Adam) Kingma & Ba (2015). Each multimodal embedding prototype Pe (of category c) is computed by averaging the embeddings of all support samples of class Once the robust embedding is obtained, the distance between the embedding of the query q: and the multi- exp(-d(f((...p)) modal prototype P. is calculated by p(y = clp.) Σn exp(-d(((n)) distance and the query is classified as the class with the minimum distance.
samples
where d refers to Euclidean

---

Under review as a conference paper at ICLR 2022
Table 1: Semi-supervised few-shot classification accuracy on test split of minilmageNet. The top
row of uni-modality results are only applied to visual features. The middle row reports the classi- fication performance on the methods of cross modality alignment based methods extended few shot learning framework and the last row demonstrates the results with robust few shot learning.
Dataset
m5N 3K 1C)
mIN(5N 5K 2C)
mIN(5N 10K 3C)
Uni-modality few shot learning methods
MAML Finn et al. (2017)
43.351 0.81%
51.641 0.47%
Matching Network Vinyals et al.
2016
40.23 0.75%
49.68 0.63%
Prototypical NetworkSnell et al.
(2017)
46.72: 0.59%
63.49 0.36%
0.59% 71.27 = 0.59%
Discriminative k-sho et al.
2017
51,64 0.37%
68.36 0.42%
72.83 = 0.25%
SNAII Mishra et al. (2018)
53.22 0.28%
69.16-t 0.48%
73.45 + 0.13%
CAMI Jiang al. 2019)
57.87 0.62%
70.67+ 0.39%
Cross modality modeling extended to metric- based FSL framework
CBPL-FSLXing et al. (2019a)
56,391 0.27%
73.11 0.45%
ReViSE-FSI sai al. (2017)
41.69# 0.69%
64.25 0.34%
72.35 0.47%
f-CLSWGAN ----S-XXaann ei al. (2018
11.63:10.66%
53.62 0.26%
73.71 0.59%
CADA SAAA-VEEFFE FSL Schönteld et (2019)
56.33-10.45%
71.97=0.27%
75.42 0.60%
AM3+TADAM Xing et al. (20 9b)
64.31 0.37%
74.03+
77.60+ 0.49%
VHE-raster--can- SwwpestterscannGGAAAA GAN Zhang et (2020)
78.65 0.42%
79.23+0.75%
78.52+ 0.35%
Robust few shot learning methods
AQGoldblum et al. (2020)
73,42:1: 0.63%
76.26: 0.58%
79.05 0.42%
RÁPNeiLu et al. (2020b)
75.72 %
79.87 0.47%
81.24 0.64%
RCFSL((urs)
84.63 0.35%
85.35 0.39%
87.21= 0.39%
EXPERIMENTS
Dataset and Competing Methods: We extensively evaluate the proposed robust cross-modal semi- supervised few shot learning (RCFSL) algorithm multiple benchmark datasets including mini- ImageNet(mIN), tieredImageNet (tIN), Sewshot-CIFAR 100 (FC100) and Caltech-UCCS Birds 200- 2011 (CUB). The minilmageNNet dataset is a subset of ILSCRS-12 12 which includes 100 classes where each class contains 600 RGB images with size 84 by 84. We split the datasets by using 64 classes for training, 16 classes for validation and 20 classes for testing where 16 classes itor the model's generalization performance. The CUB Bird dataset contains 1 11788 images of 200 different bird species, where the data split equally training and testing data. Namely, for each category, 30 training image and 30 testing images are included, where in the training dataset, 10% images are labeled images and the rest of the images are unlabeled. 10 short textual descriptions per image are provided by Reed et al. (2016). Following the work Zhang et al. (2017). the text encoder pretrained by Reed al. (2016) 6 employed and the data is split such as CBase! = 150, [Crovell 50. Specifically, n 21,2, 5, 10, 15, 20} images are selected from novel classes in order to perform few shot learning. We evaluate the two types of label noise including symmetric noise and asymmetric noise. For symmetric noise, we inject the label noise by randomly flipping the labels of the labeled data into a different label in the novel classes. For asymmetric noise, the corrupted labels are generated by replacing the correct labels with their most similar classes (using nearest neighbor measurement). Outliers are mimicked by including samples from data dis- tributions far away from the training data distributions, namely out-of-distributtin (OOD) samples. Glo /Penninggoon et al. (2014) is utilized to extract the word embeddings for the category labels, where the embeddings are trained with large unsupervised text corpora. RCFSL is compared with multiple benchmark methods and state-of-the-art approaches. Typically. similarly as Xing et al. (2019)), three families of methods are evaluated: (1) uni-modality few-shot learning methods such as MAMLFiin et al.] (2017). LECRusu et al.] (2019) and CAMLIJang et al.) (2019). In particular, latent embedding optimization (LBO) Rusu et al. (2019) addresses MAML problem by relying on a few updates low data regime to train models in high dimensional parameter space. (2) cross modality few shot learning methods VeVi Tsai et al. (2017), CCDDA-VAE Schönfeld et al. (2019) et al. (2020) and Feature transform Tseng et al. (2020). Among them, ReViSE Tsai et al. (2017) minimizes maximum mean discrepancy (MMD) of the distributions from representation spaces for better alignment. CADA-VAR Schönfeld et al.
utilized mon-
Xing et al. (2019b), VHE-raster-see www.rraeseennnn--aee -aan-GANZhaann
7

---

Under review as a conference paper at ICLR 2022
Table 2: Semi-supervised few-shot classification accuracy on test split of tiered (tIN). The top row of uni- uni-modality results are only applied to visual features. The middle row reports the classification performance on the methods cross modality alignment based methods extended to few shot learning and the last row demonstrates the results with robust few shot learning.
Dataset
UN 3K 1C)
UN(5N 5K 2C)
UN((N 10K 3C)
Uni-modality few shot learning methods
MAMLFiin et al. (2017)
47.654 0.66%
52.631 0.71%
Prototypical Network et al.
(017))
49.65 0.56%
68.13 0.57%
73.57 + 0.67%
Discriminative k- k-sho Bauer et al.
(2017)
51.64 0.37%
69.27 0.42%
74.06 + 0.25%
SNAIL al. (2018)
55.63# 0.31%
70,23 0.48%
75.69 + 0.43%
CAMIJiang et al.
2019)
58.76-+- 0.45%
71,35+ 0.32%
Cross modality modeling extended to metric-based
FSL framework
CBPL-FSLXing et al. (2019a)
58.574: 0.45%
74.35 = 0.67%
ReViSE-FSI sai et al. (2017)
44.53: 0.81%
67.61 0.85%
73.65 0.56%
Feature transform I I'seng al. (2020)
59.78=00.5%
73.21 0.42%
75.23 0.63%
CADA-VAE-FF SAAE-FLSSchönee ct al. (2019)
57.64=0.59%
71.97+0.27%
77.31 0.57%
AMM++ADAMXIng et al. 201 9b))
65.92 0.53%
74.03=0.555%
78.34 0.45%
VHIB-raster-scan-GGANN Zhang et al. (2020)
73.41+ 0.63%
76.35
79.03 0.32%
Robust few shot learning methods
AQGoldblum et al. (2020)
75.31 0.56%
78.35 0.62%
81.33 0.48%
RAPNeLLu et al. (20206)
76.48 0.51% %
80.11 0.53%
82.61 0.39%
RCFSL(OUrS)
85.65 0.47 %
87.63 0.34%
88.57 0.42%
2019) leverages two VAEs to embed information for both modalities and align the distribution of the two latent spaces. (3) Robust few shot learning methods including robust attentive profile networks (Rapnets) Lu et al. (2020b) and Adversarial Querying (AQ) Goldblum et al. (2020). Rapnet{Lu cet al. (2020b) consists of an embedding module, correlation module, and an attentive module to suppress the outliers in few-shot learning. Adversarial Querying Goldblum et al. (2020) produces adversarially robust meta-learners and achieves efficient classification for few-shot learning tasks. Implementation Details: For minilmagenet. tiredImagenet and FC100 datasets, we parameterize the heterogeneous VAE with three sets of 5 by 5 fully convolutional, ReLU and pooling layers fol- lowed by two fully connected hidden layers where each pair of layers contains the hidden units as dim(h) = 1000. We further set the dimension of the auxiliary variable and the latent variable be 200 respectively. For the CUB dataset, we leverage Resnet-18 backbone and incorporate the uncertainty prior for the input, the noise transition model along with the auxiliary variables as our encoder. The network is trained with SGD using a batch size of 128. A momentum of 0.9 is set with a weight decay of 0.0005. The network is trained for 30000 iterations. The initial learning rate set as 0,1, and reduce by factor of 10 after iterations 15,000, 17,500 and 19,000. For the configurations of discriminator in GAN, we follow the same configuration of DC-GAN and fol- low the similar discriminator Mirza & Osindero (2014) for DC-GAN. 3 is varied from 0. to 0.4 where the best performance is reported. The similar setup in Xing al. (2019b) is utilized relying on (CC(N, K, C') to evaluate the performance of classification accuracy in N-way and K-shot SFSL settings, where C' represents the total number of outliers and noisy labels in each class. We use 10 textual descriptions per image to generate multimodal prototypes. The semi-supervised few- shot classification accuracy test split of minilmageNet is shown in Table 1. As shown in Table 1. RCFSL achieves the best classification accuracy compared to all of the competing methods in different cases. The large performance margin suggests the proposed mechanism not only fuses two modalities of features cohesively and but also effectively suppresses the noisy labels and outliers in superiuee few-shot learning. Compared to the adaptive modality mixture mechanism (AM3) with TADAM as backbone Xing et al.] (2019b), RCFSL has the advantage of fusing multimodal distributions the heterogeneous encoder via the latent space to fully utilize the multimodal feature representation instead of computing multimodal prototypes with convex combination afterwards. In contrast to robust FSL methods like Rapnets Lu et al. (020b) and AQ Goldblum et al. (2020), the performance gain be mainly attributed to the fact that RCFSL employs mathematically rigorous variational inference insensitive to noisy labels and outliers to optimize network parame- ters, yielding more accurate classification performance. The margin in performance is particularly
8

---

Under review as a conference paper at ICLR 2022
Table 3: Semi-supervised few-shot learning results (%) using our method with different combi- nations of denoising strategies training with different levels of label noise and outliers. The bold number in each column of sub-boxes represents the best result. Here UP stands for uncertainty priors, RD means robust divergence and CM represents cross multimodal data (images and text).
Dataset
Denoising Strategies
mlN
|
uN
FC100
CUB-200
UP
RD
SM
No
No
No
78.79
80.55
61.35
70.77
Yes
No
No
79.25
83.07
62.27
72.34
5N 5K 2C
Yes
Yes
No
80.36
83.38
64.13
72.66
Yes
Yes
Yes
85.35
87.63
68.33
76.93
outliers
outllers good labels
0.85
CUB200
0.823
62
Accureey
RCEEL Pronetus
3.775
RCESL Briontoun
wrong labels
(a) RepNETS
RCFSL (ours)
shitss
(d) shots
Figure 3: Comparison of the t-SNE visualization with two dimensional multimodal prototype space
for the FewShot CIFAR-100 dataset using RapNetsLu et al. (2020b) and our RCFSI. (8=0.2) 10- 8-shot semi-supervised learning (Fig (.)a) and (b)), where each class includes 2 noisy labels and 2 outliers. The index number represents superelasses cates on the median position of the corresponding vectors and the outliers are marked with squares. The embeddings from 10 distinct clusters using our method corresponds to true class labels, which validates the robustness of our method to label noise and outliers. Fig 3(c)- (d) illustrate the classi- fication accuracy vs the number of shots for FC100 and CUB-200 datasets respectively.
in CIFAR-100. Each number lo-
remarkable in the fewer shots scenario, supporting the robustness of RCFSL. Similar performance evaluations are conducted on tieredImageNet dataset in Table 2 and RCFSL again is the top per- former under various noise settings. Fig.3 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR- 100 dataset using Rap- NetsLu et al. (2000) and our RCFSL (f=0.2) on a 10-Way 8-shot semi-supervised learning (Fig 3.(a)- (b)), where it clearly shows our method has better capability in terms of separating noisy labels and outliers in semi-supervised few-shot learning. We also investigate the classification ac- curacy of different methods versus the number of shots in Fig 3.(c) (d) for FC100 and CUB200 datasets respectively with 25% noisy labels and outliers respectively, RCFSL yields considerable performance gains over competing methods especially with smaller number of shots.
Ablation Study: Described in the ablation study in Table 3, RCFSL w/o uncertainty prior excludes the uncertainty prior from the model. Therefore, the performance degradation suggests the impor- tance of the proposed hierarchical structure for variational inference by placing the uncertainty prior on the infinite GMMs to counter the detrimental effects of outliers. Secondly, our method w/o robust divergence replaces 3-divergence which places small weights noisy labels and outliers with the regular KL-divergence. Moreover, RCFSL w/o cross modality relies on only image data for SFSL. where multi multi-modality further improves the classification accuracy by at least 4%.
4 CONCLUSION
By integrating uncertainty prior of an infinite Gaussian mixture model into the heterogenous encoder and the robust lower bound based 3-divergence for variational inference, our RCFSL is capable of tackling the outliers and noisy labels simultaneously in semi-supervised few shot learning. More- over, by integrating the robust semi-supervised GAN for feature generation, experimental evalua- tions on multiple benchmark datasets have demonstrate the superiority of the proposed method with large margins using multimodality data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
K. Allen, E. Shelhamer, Н. Shin, and J. Tenenbaum. Infinite Mixture Prototypes for Few-shot Learning. International Conference on Machine Learning, 2019.
A. Basu, I. Harris, N. Hjort, and M. Jones. Robust and efficient estimation by minimising density power divergence. Biometrika, 1998.
M. Bauer, M. R. Carulla, J. B. Swikatkowski, B. Scholkopf, and R. E. Turner. Discriminative k-shot learning using probabilistic models. Neural Information Processing Systems, 2017.
C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
D.M. Blei and M.I. Jordan. Variational inference for dirichlet process mixtures. Bayesian Analysis, 2017.
Y. Burda, R. Grosse, and R. Salakhutdinov. Importance Weighted Autoencoders. arXiv preprint- 1r0iv:1509.005/1 2015.
Diederik and M. Welling. AutoEncoding Variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
C. Finn, Abbeel, and Levine. Model-agnostie meta-learning for fast adaptation of deep net- works. International Conference Machine Learning (ICML), 2017.
C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning, Neurips, 2018
Futami, I. Sato, and M. Sugiyama. Variational Inference based on Robust Divergences. Twenty- First International Conference Artificial Intelligence and Statistics, 2018
M. Goldblum, L. Fowl, and T. Goldstein. Adversarially Robust Few-Shot Learning: A Meta- Learning Approach. Neural Information Processing Systems, 2020.
Hou and Galata. Robust Estimation of Gaussian Mixtures from Noisy Input Data. Computer Vision and Pattern Recognition, 2008
Huber and E. Ronchetti. Robust Statistics. Wiley Series in Probability and Statistics, 2011.
X. Jiang, M. Havaei, Varno, G. Chartrand, N. Chapados, and S. Matwin. Learning to learn with conditional class dependencies. International Conference Learming Representation, 2019.
Kaneko, Y. Ushiku, and T. Harada. Label-Noise Robust Generative Adversarial Network. IEEE Conference Computer Vision and Pattern Recognition, 2019.
Kim, Yoon, Dia, Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. Neurips, 2018.
D.P. Kingma and J.L. Ba. Adam: Method for Stochastic Optimization. International Conference on Learning Representation, 2015.
A. Kumar, P. Sattigeri, and P. T. Fletcher. Semi-supervised Learning with GANs: Manifold Invari- ance with Improved Inference. Neural Information Processing Systems, 2017.
Y. Li and Y. Gal. Dropout inference in Bayesian neural networks with alpha-divergences. Interna- tional Conference on Machine Learning, 2017.
J. Lu, S. Jin, J. Liang, and C. Zhang. Robust Few-Shot Learning for User-Provided Data. IEEE Transactions on Neural Networks and Learning Systems, 2020a.
J. Lu, S. Jin, J. Liang, and C. Zhang. Robust Few-Shot Learning for User-Provided Data. 1EEE Transactions on Neural Networks and Learning Systems, 2020b.
L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther. Auxiliary Deep Generative Models. International Conference Machine Learning, 2017.
10

---

Under review as a conference paper at ICLR 2022
M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. ICLR, 2018.
T. Miyato, S.-i.and M. Koyama Maeda, K. Nakae, and S. Ishii. Distributional Smoothing with Virtual Adversarial Training. arXiv preprint arXiv: 2015.
K. Murphy, Conjugate Bayesian analysis of the Gaussian distribution. Lecture notes on UC Berkeley, 2007.
B. Oreshkin, P. Rodriguez, and A. Lacoste. TADAM: Task dependent adaptive metric for improved few-shot learning. Neurips, 2018.
Pahde1, M. Nabi1, and T. Klein. Multi-Modal Few-Shot Learning: A Benchmark. Workshop at International Conference Learning Representations, 2018.
Pahdel, M. Puscas, Klein, and M. Nabi. Multimodal Prototypical Networks for Few-shot Learning. IEEE Winter Conference Computer Vision (WACV), 2021.
G. Pandey and Dukkipati. Variational methods for conditional multimodal deep learning. Inter- national Joint Conference Neural Network, 2017.
G. Patrini, A. Rozza, to label noise: A loss correction approach. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Menon, R. Nock, and L. Qu. Making deep neural networks robust
J. Pennington, R. Socher, and C. Manning, Glove: Global vectors for word representation. EMNLP, 2014.
A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. Neural Information Processing Systems, 2015.
S. Reed, 7. Akata, H. Lee, and B. Schiele. Learning deep representations of finegrained visual descriptions. International Conference on Computer Vision, 2016.
M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, R. S. Zemel, X. Li, and Y. Guo. META-LEARNING FOR SEMI-SUPERVISED FEW-SHOT CLASSIFICA- TION. IEEE Conference on Learning Representations, 2018.
A. A Rusu, D. Rao, J. Sygnowski. O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta- learning with latent embedding optimization. International Conference on Learning Representa- fions, 2019.
T. Salimans, I. Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. Neural Information Processing Systems, 2016.
E. Schönfeld, S. Ebrahimi, S. Sinha, Darrell, and Z.. Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. IEEE Conference on Computer Vision and Pat- tern Recognition, 2019.
Y. Shi, N. Siddharth, B. Paige, and P.H.S. Torr. Variational Mixture-of-Fxperts Autoencoders for Multi-Modal Deep Generative Models. Neurips, 2019.
C. Sonderby, Т. Raiko, L. Maaløe, S. Sonderby, and O. Winther. Ladder variational autoen-coders. arXiv preprint 1XX::1662.00282.. 2016.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Neural Information Processing System, 2017.
Sukhbaatar, Bruna, M. Paluri, L. Bourdey, and R. Fergus. Training convolutional networks with noisy labels. ICLR workshop, 2015.

---

Under review as a conference paper at ICLR 2022
M. Suzuki, K. Nakayama, and Y. Matsuo. Joint multimodal learning with deep generative models. International Conference on Learning Representations Workshop, 2017.
D. Tran, R. Ranganath, and D. Blei. Variational Gaussian process. arXiv preprinuuXxx:/1111066499 2015.
Y. Tsai, L. Huang, and R. Salakhutdinov. Learning robust visual semantic embeddings. 1EEE Conference on Computer Vision and Pattern Recognition, 2017.
H. Tseng, H. Lee, J. Huang, and M. Yang. Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation. International Conference on Learning Representations, 2020.
O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one shot learning. Neural Information Processing Systems, 2016.
Y. Xian, Lorenz, B. Schiele, and Z. Akata. Feature generating networks for zero-shot learning- CVPR, 2018.
C. Xing, N. Rostamzadeh, B. N. Oreshkin, and O. Pinheiro. Zero and few shot learning with semantic feature synthesis. Neural Information Processing Systems, 2019a.
C. Xing, N. Rostamzadeh, B. N. Oreshkin, and O. Pinheiro. Adaptive Cross-Modal Few-shot Learning. Neural Information Processing Systems, 2019b.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas. Stackgan: Text to photo- realistic image synthesis with stacked generative adversarial networks. International Conference Computer Vision, 2017.
H. Zhang, B. Chen, L. Tian, Z. Wang, and M. Zhou. Variational hetero-encoder randomized gans for joint image-text modeling. International Conference Learning Representation, 2020.
A DERIVATION OF INFLUENCE FUNCTION FOR LOWER BOUND OF RCFSL
Define G(x) as empirical cumulative distribution given by {x; h and denote the perturbed ver- sion of = as (x)) = (1 -1-ej e)(((>> + Az(x), where e the contamination portion and A is the point mass at Given a statistic T. the influence function (IF) is defined as Futami et al. 2018
(F(2,T,G) (T() le=0
(11)
de
Thus, the of the heterogenous variational autoencoder for LRCV AF is given by
8 LECVAR OH2
aH.
+N( 1+ p(z H 3 J p(a, (191
LaCy AB y-1 3H Fe((H) kL. (a( H (/t.tt///////////////// t)//p(HH) 0H
+N(
8+1 p x, H p(ū x. 1+
(12)
It is straightforward to show that the above result is always bounded, namely RCFSL robust the compound noise (the outliers on the data and the labels y).
B VISUAL COMPARISON WITH ROBUST DIVERGENCE
Fig.1 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-101 100 dataset using the same setting as our method ex- cept replacing robust divergence with KL-divergence and our RCFSL (8=0.2) 10-Way 8-shot semi-supervised learning, where it clearly shows the efficacy of the robust divergence by offering a stronger capability in separating noisy labels and outliers in semi-supervised few-shot learning.
12

---

Under review as a conference paper at ICLR 2022
outliers
outliers
O
good labels
good labels
wrong labels
wrong labels
(a) KL-divergence
(b) robust divergence
Figure 4: Comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-100 dataset using RCFSL with KL-divergence in variational inference and RCFSL. with robust divergence (8=0.2) on 10-Way 8-shot semi-supervised learning, where each class includes noisy labels and outliers. The index number represents superclasses CIFAR-100, Each number locates on the median position of the corresponding vectors and the outliers are marked with squares. The embeddings from 10 distinct clusters using our method sponds to true class labels, which validates the robustness of our method to label noise and outliers.
9 in
Table 4: Comparison of the training time (hours) of RCFSL on Fewshot CIFAR-100 with several state-of-the-art approaches evaluated on single Nvidia V100 GPU.
CBPL-FSI Xing et (019a)
ReViSE-FSI Tsai et al. (017
Feature transforn
Tseng al. 2020
6.3h
7.6h
5.9h
CADA-VAB-FE Schönfeld al. (2119)
AM3+TADAMXXng et al.
2019b)
VHEraster-scan-GAN GaAZaan al. 2020
7.2h
6.3h
8.3h
AGGoldblum et al.
2020
RAPNe et al.
2020b)
RCFSL
4.7h
6.1h
5.2h
C DETAILS ON NETWORK STRUCTURE, TRAINING AND INFERENCE TIME COMPARISON
The discriminator of RCFSL following deep convolutional GAN is structured as follows: the first convolutional layer is of size 32 32 x 64, the second convolutional layer is of dimension 16 16 x 128, the third layer is 8 8 x 256 and the last convolutional layer is of the dimension 4 x x12 with the convolutional filter size layer and then a ReLU layer.
3. Each convolutional layer is followed by batch normalization
We analyze the training time (hours) and inference time (seconds) of our RCFSL demonstrate its computational efficiency in Table 4 and Table 5. The training and the inference time of RCFSL reported on Newshot-CIFARU
approaches, evaluated on a single Nvidia V100 GPU. RCFSL is slower than AQGGoldblum et al. (2020), however is faster than the rest of competing methods for both training and inference time due to the efficient probabilistic graphical model and the effective novel lower bound and compact network architecture.
with several state-of-the--e
D DISCUSSION ON DIFFERENCES FROM OTHER MULTIMODAL VAES
Prior approaches Pandey & Dukkipati (017) on multimodal VAEs mostly focus on cross-moda generation. In particular, given two modalities xι and 22(e.g. image and text), these approaches
13

---

Under review as a conference paper at ICLR 2022
Table 5: Comparison of the inference time (seconds) of RCFSL Fewshot CIFAR-100 with several state-of-the-art approaches evaluated on single Nvidia V100 GPU.
CBPI.-FSI Finnn et (019a)
ReViSE-FSI Tsai et al. (017
Feature transforn Tseng et al. (020
43s
52s
36s
CADA-VARE FSL Schönfeld et al.
2019
AM3+TADAM Xing et al. (019b)
VHEraster- Laann-GANNZaaa et al. 2020)
47s
39s
57s
AGGoldblum et al.
2020
RAPNe et al. 20200b)
RCFSL
17s
37s
23s
learn the conditional generative model (22), where the conditioning modality 22 and the gen- eration modality 21 are usually not interchangebale. Compared to Panaeyy & Dukkipati (2017). our heterogeneous robust VAE targets joint image-text modeling by explicitly modelling the joint dis- tribution over latent variables from both image and textual data. Moreover, our multimodal VAEs integrates the uncertainty priors, denoising layers and robust variational lower bound to ensure the robustness in the presence of noisy labels and outliers. The work (uuuuii et al. (2117) introduced joint multimodal VAE (JMVAE) which aims at learning shared representation with joint encoder qu (2(2221, 222) Distinct from Suzuki al. (2017) which does not consider the robust modeling to tackle with outliers, our approach leverages infinite Gaussian mixture models equipped with uncer- tainty prior which not only characterizes the distributions for image-text modeling more accurately but also facilitates the learning of new unseen classes by inferring the number of classes automati- cally in few-shot learning. Recently, Zhang et al. (2020) utilized a VAE that encodes an image into a deterministic-upe spprminitic--pward--ae wwwar-sssohsstii-dooeee dowwward ladder-strue structurud latent representation, and then is ap- plied to decode the corresponding text. While our work is relying on joint encoding and decoding schemes on image-text modelling and focusing on tackling the noisy-labels and outliers by enhanc- ing the robustness of multimodal modeling using the various proposed techniques. More recently, Shi et al. (2019) leverages mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, indicating the suitability of mixture models for learning from multi modal data. Our work (RCFSL) advances MMVAE Shi et al. (019)) in three aspects: (1) We places uncertainty prior on the parameters of mixture models to avoid the collapsing of models in the presence of outliers and derived novel inference solution given the uncertainty prior (shown in our novel generative and inference models in Eqs (2)-(4)); (2) instead of using importance weighted autoencoder (IWAE) estimator Burda et al. (2015 Shi et al. (2019), RCFSL takes advan- tage of robust divergence to derive the robust evidence lower bound for unlabeled data and labeled data respectively which are unique and mathematically rigorously (e.g. Eq(7)), yielding superior performance for classification of noisy dataset in semi-supervised few-shot learning. (3) RCFSL integrates the denoising layers in both multimodal VAE and robust GAN to mitigate the influence of noisy labels, which further enhances the robustness of the model.
E EVALUATION ON SEMANTIC LABEL NOISE
In order to evaluate our algorithm under real-world noise (semantic label noise), RCFSL is further evaluated on Clothing 1M dataset with the network architecture Resnet-18. For Clothing IM dataset, it includes million training images with 14 classes obtained from online shopping websites and la- bels are generated from surrounding texts. Since the data is collected from multiple online shopping websites and include many mislabelled samples, therefore the labels are contaminated by real-world noise (semantic label noise). The data is split such as |CBasel = 10, |CNovet = 4. Table demonstrates the comparison of classification accuracy using different learning algorithms on the Clothing IM datasets (real-world noise with 10% and 20% outliers respectively). where n = 5 im- ages are selected from novel classes in order to perform few shot learning. As it can be seen from Table 6, our RCFSL continues to serve as the best performing method in the presence of seman- tic label noise compared to other competing methods under different noise statistics, which further confirms the outstanding learning capability of the proposed approach in robust semi- semi-supervised few-shot learning. Moreover, Figure illustrates exemplary images with compound noise detected
14

---

Under review as a conference paper at ICLR 2022
Table 6: Comparison of classification accuracy using different learning algorithms the Cloth- ingIM IM datasets (real-world noise with 10% and 20% outliers respectively) for 5 way 5 shot semi- supervised few shot learning on 1000 test episodes.
Dataset
Clothing M (10%)
Clothing MM (20%)
CBPL-FSII Xing et al. 2019a
59.3
56.8
ReViSE-FSII I et al. 20177)
70.2
68.4
Feature transform l'snng al. (2020)
71.6
69.3
AMM++ADAMXIng et al. 92019b)
71,8
70.5
VHE-raster-scan- GANZhang
(2020)
72.9
70.7
AGGoldblum et al. 2020
72,6
70.1
RGAN Kaneko et al.
2019
73,9
71.5
RAPNeeLu Lu et al. 20206
74.3
71.8
RCFSI.
82.6
77.1
jumpsuit
romper
sweater
culottes
sweater
SWeater
dress short
romper blouse
blouse
short
hoodie
blouse
romper
skirts
jumpsuit
blouse
Exemplary images with compound noise detected by RCFSL from Clothing IM dataset, where the false labels are noted in orange and the true labels are noted in light blue.
Figure 5:
by RCFSL from Clothing 1M dataset, where the false labels are noted in orange and the labels are noted in light blue.
F EVALUATION OMNIGLOT DATASET
We further evaluate the proposed algorithm on Omniglot dataset. The Omniglot dataset includes 1623 different handwritten characters from 50 different alphabets where each of the 1623 charac- ters was drawn online via Amazon's Mechanical Turk by 20 different people and each image is paired with stroke data, a sequences of spatial and temporal coordinates with in milliseconds. Ta- ble 7 demonstrates the comparison of classification accuracy using different learning algorithms the Omniglot dataset (20% symmetric label noise with 10% outliers) for semi semi-supervised few shot learning 1000 test episodes, where it can be seen from Table 7 that our RCFSL provides the best performance against other competing methods the presence noisy labels and outliers in semi-supervised few-shot learning.
15

---

Under review as a conference paper at ICLR 2022
Table Comparison of classification accuracy using different learning algorithms the Omniglot dataset (20% symmetric label noise with 10% outliers) for semi-supervised few shot learning on 1000 test episodes.
Dataset
5 way 5 shot
way 10 shot
CBPL-FSI Xing et al. 2019a)
95.2
97.8
T ReViSE-FSIJ sai et al. (017))
95.6
98.1
Feature transform I'snng al. (2020)
96.3
98.3
AMM++TADAMXIng et al. 92019b)
96.2
98.2
VHE-raster-scan- GAae--GNNNNA Zhang
(2020)
96.5
98.6
AGGoldblum et al. 2020
96.8
98.7
RGANKancko et al.
2019
97.1
98.9
RAPNe(Lu et al. 20206
97.3
99.1
RCFSI.
97.9
99.7
Table 8: Unsupervised test log-likelihood using different learning algorithms on the permutation invariant MNIST dataset (with 20% outliers) the normalizing flows VAE (VAE+NF). importance weighted auto-encoder (IWAE), variational Gaussian pro-cess VAE (VAE+VGP), Ladder VAE (LVAE) with FT denoting the finetuning procedure Sonneeby et al. (2016) and auxiliary deep gen- erative models Maaløe et al. (2017) and our method (3=0.2), where 7. represents the number of stochastic latent layers 21,...; Zr. and /W characterizes the importance weighted samples during training.
Method
log p(x)
Miy+oo et al.
2015
L=1
-89.35
IWAE, I.=1, [W=
Burda et
2015
-90.26
IWAE, L=1, 5W=50
Burda et al. (0115
-88.36
IWAE, L=22, IW=1
Burda et
-89.71
IWAE, L=2, 5W=50 Burda et al.
(2015
-86.43
VAE+VGP, L=2
ran cl al. 2013
-85.79
LVAE, L=5, IW=1 Sönderby al.
(2016)
-85.08
ADGM, L=1, IW=T Maalde et al.
2017
-84.67
ADGM, I.=1, IW=2
Maaløe et
2017)
-84.34
RCFSL+ KL divergence, L=1, IW=2
-83.76
RCFSL (robust divergence), L=1, IW=2
-81.35
G COMPARISON OF LOG-LIKELIHOOD WITH ROBUST DIVERGENCE
We report the roust lower bound for the unlabeled data with 5000 importance weighted samples where the similar setting as Rasmus et al. (015) with warm up. batch normalization and Monte Carlo and IW sample for training. The percentage of outliers is set be 20%. Table 8 demonstrates the log-likelihood scores for the permutation invariant MNIST dataset. The results shown in Table 8 indicates the the proposed method has strong expressive power by performing better than other methods in terms of ligeliold due to the utilization of the robust divergence in the inference, especially by comparing with the same RCFSI. method but with KI--divergence.
H ANALYSIS FOR DIFFERENT TERMS IN THE OPTIMIZATION FUNCTION
In the above optimization cost function of RCFSL in equation (10), the first term represents the supervised and unsupervised loss from the discriminator, the second and the third terms tend to min- imize the approximation error by "regularizing" q xx. t. Ø) and (HH Ja., t) to the prior p(H) and p(H.) for labeled data and unlabeled data respectively. The fourth term characterizes the generator loss from the robust cemi-supervised GAN (RSGAN), The fifth to the eighth terms are employed to enhance the robustness in the presence of data outliers with 3-divergence for the ELBO.. on the
16

---

Under review as a conference paper at ICLR 2022
labeled data and unlabeled data respectively, which is shown to be insensitive to the small contami- nation of the data.
17


Reviews:
[{"review_id": "zwjdf-RHib3", "reviewer": "Reviewer_SEhY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides enough details of their method, and shows performance improvement over some existing methods. However, I have major concerns about the problem formulation, the motivation of the method, and the experiments. \n\nThis paper aims to address four problems at the same time: (1) few-shot learning, (2) multimodal FS learning, (3) semi-supervised FS learning, and (4) FS learning under data noise. It is not clear what are the unique challenges posed by each problem, and why this paper combines all the problems together to create a new and complex problem. There need to be more intuitive justifications on the importance of the problem studied. \n\nThe method is also quite complex and not well-motivated. It is unclear what challenge doe each component address. It is also difficult for me to understand what value does the proposed mixture model offer that is specific to the problem studied.\n\nThe experiment setting is confusing to me. Could the author explain in more details about the (1) multimodal data, (2) semi-supervised setting, and (3) dataset noise. It seems to me that all the baseline results are reproduced on the new setting (please correct me if I'm wrong), because previous methods are not targeted to this problem. In such cases, the improvements over baselines are expected. The authors either need to adapt the baseline, or provide more justifications on why the baselines are strong enough. ", "summary_of_the_paper": "This paper proposes to address the complex problem of cross-modal semi-supervised few-shot learning with noisy data. A rather complex approach is proposed with combines Bayesian mixture mode, VAE, GAN, and prototypical learning. ", "summary_of_the_review": "This paper has limitations in the problem formulation, the motivation of the method, and the experiments. In my opinion, it needs major improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eh4PMQYWHvs", "reviewer": "Reviewer_JD5c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n1. The idea of introducing Bayesian deep learning to the semi-supervised few-shot learning problem sounds reasonable.\n2. The provided experimental results show the effectiveness of the proposed method to some extent.\n\n# Weaknesses\n1. Why could the introduced uncertainty prior mitigate the impact of corrupted labels and outliers?  How to ensure the relationship between the uncertainty prior and the noisy data? It is suggested to give a more intuitive explanation. \n2. The authors claim that their robust variational lower bound can infer network parameters more accurately. However, it is hard to see the advantages of the robust variational lower bound intuitively from the paper.\n3. It seems that there are multiple loss terms in the proposed objective function, such as Eq. (10). How does the author determine their respective weights during optimization? Do these weight parameters have much influence on the results?\n4. It is suggested to give more implementation details, such as the network structure, experimental configuration, training and inference time.\n5. The writing of this paper needs to be further polished. For example, \"the parameters of infinite Gaussian mixture model\" -> \"the parameters of an infinite Gaussian mixture model\"? \"the performance gains shows\" -> \"the performance gains show\"?\n", "summary_of_the_paper": "This paper focuses on semi-supervised few-shot learning with multi-modality data. The authors introduce an uncertainty prior of an infinite Gaussian model, integrating multi-modality information from image and text data into a heterogenous variational autoencoder. Meanwhile, a new variational lower bound is derived for the inference of parameters. In addition, a GAN is developed for the data sparsity in few shot learning. Experimental results demonstrate the effectiveness of the proposed method to some extent.", "summary_of_the_review": "Overall, this paper provides a new perspective on the cross-modal semi-supervised few-shot learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xt263puBf0H", "reviewer": "Reviewer_RfHj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths:\n\n+ The proposed approach relies on existing methodologies that are combined in a new way. \n\n+ The problem is clearly described and the paper is well-written.\n\n+ The approach delivers good performance in comparison to the related approaches.\n\nPaper weaknesses: \n\n- The paper focuses on image-text multi-modal learning. In the introduction, this is can be only implicitly inferred after reading the introduction. A clear problem statement would be necessary.\n\n- Novelty: It is not clear in which sense is the variational autoencoder (VAE) heterogeneous. The term is actually not defined in the context of the problem. Moreover, relying on text and images to train the VAE is not unusual. It should be clarified what is heterogeneous and why it is novel.\n\n- Although the paper is well-written, it relies on several approaches which have to be described. The current structure of the paper does not help to understand the complete approach. For instance, the explanation of the uncertainty prior is not clear. Moreover, the method section lacks an overall description of the approach prior to describing the details. Overall, the method section could be improved to help the reader easier follow the paper's idea.\n\n- Symmetric and asymmetric noise are not often encountered in real-world problems. It would be more interesting to see semantic label noise being considered. \n\n\nImprovements:\n\n- Omniglot could be considered in the evaluations too. It is also a standard benchmark. ", "summary_of_the_paper": "The paper presents a cross-modal semi-supervised few-shot learning approach for image classification. The idea is to train variational auto-encoder (VAE) with both image and text data for learning a feature representation. Then features are extracted from a test sample and assigned to the class of the closest prototype train sample. In addition, a generative adversarial objective is employed for learning the latent code of the VAE. Since the proposed idea is meant for learning from noisy labels, there is an uncertainty prior in the image features as part of the infinite Gaussian mixture distribution. The approach is evaluated on standard few-shot learning benchmarks which are modified to account for the noise labels. The results are promising compared to the prior work. ", "summary_of_the_review": "Overall, the paper addresses the problem of noisy labels next to cross-modal semi-supervised few-shot learning. This problem formulation makes the proposed work different from the existing approaches. Moreover, the proposed formulation could be considered novel regardless of combining existing approaches. The main issue of the paper is the difficulty to follow the proposed approach. One needs to read several times the paper for understanding the structure of the proposed method. This is a major point that needs improvement. Finally, considering semantic label noise would be the whole idea more plausible for real-world setups.\n\nPost-rebuttal comment:\n\nThe rebuttal has helped to improve the paper. My concerns have been addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to address the problem of cross-modal semi-supervised few-shot learning with noisy data, and proposed a robust cross-modal semi-supervised few-shot learning (RCFSL) based on Bayesian deep learning. The approach combines several existing techniques for tackling a new problem in a non-trivial approach. Empirical results demonstrate the effectiveness of the proposed method to some extent. 

While the proposed integrated complex approach seems to be novel in the proposed unique setting, there are some major concerns from the reviewers. One concern is about the lack of clear justification on technical contributions for the proposed methodology in the complex settings. In particular, it lacks of comprehensive ablation studies for analyzing and understanding the source of gains by the proposed complex method, and the baselines in the experiments also do not look strong enough. In addition, many aspects of the paper writing and presentation are not satisfied (e.g., the math formulation in Section 2 is densely presented making it difficult to follow). 

Overall, this is a borderline case, where the paper did contribute a new method for the interesting cross-modal semi-supervised few-shot learning task, but some major concerns on the weaknesses remain at its current form. Therefore, it cannot be recommended for acceptance. Nonetheless, I hope authors can improve the paper by fully addressing these issues and hope to see it accepted in the near future.
Decision: Reject