Year: 2021

Abstract:
Semi-supervised learning has been successfully applied to few-shot learning (FSL) due to its capability of leveraging the information of limited labeled data and massive unlabeled data. However, in many realistic applications, the query and support sets provided for FSL are potentially noisy or unreadable where the noise exists in both corrupted labels and outliers. Motivated by that, we propose to employ a robust image-text multi-modal semi-supervised few-shot learning (RCFSL) based on Bayesian deep learning. By placing the uncertainty prior on top of the parameters of infinite Gaussian mixture model for noisy input, multimodality information from image and text data are integrated into a robust heterogenous variational autoencoder. Subsequently, a robust divergence measure is employed to further enhance the robustness, where a novel variational lower bound is derived and optimized to infer the network parameters. Finally, a robust semi-supervised generative adversarial network is employed to generate robust features to compensate data sparsity in few shot learning and a joint optimization is applied for training and inference. Our approach is more parameter-efficient, scalable and adaptable compared to previous approaches. Superior performances over the state-of-the-art approaches on multiple benchmark multi-modal dataset are demonstrated given complicated noise for semi-supervised few-shot learning.

heading: 1 INTRODUCTION
text: Despite the impressive success of deep learning models, frequently it requires massive amount of training data to fully demonstrate the potential of the model. In contrast, human is capable of learning new concepts given limited data. Consequently, few-shot learning gathers extensive research interest due to the capabilities of learning new concepts from limited training data. Nevertheless, the success of few-shot learning requires careful handling to robustness and generalization as it is extremely susceptible to noisy labels, outliers as well as adversarial attack Lu et al. (2020a). For instance, in order to automatically recognize several kinds of uncommon animals, only a few annotated images for them are available due to their rarity. Moreover, the images could potentially be corrupted due to an uncontrollable shooting environment or an instrumental malfunction. To mitigate this, one common approach to few shot learning is meta-learning Ren et al. (2018), where the goal is to learn a classifier to distinguish between previously unseen classes, given labeled classes and a larger pool of unseen examples, some of which may belong to the classes of interest, namely semi-supervised few shot learning (SFSL).
Despite the impressive capabilities equipped the ability to leverage unlabeled examples for SFSL, the challenge of lacking novel samples remains to be a bottleneck. Besides visual information, textual data frequently contains rich information and more descriptive concepts for learning. Incorporating image-text multi-modal learning into the framework by training on image-text pairs provides an efficient tool to inject the diversity to the generation process Pahde1 et al. (2021)Pahde1 et al. (2018). The work in Pahde1 et al. (2018) provides a benchmark for multimodal few-shot learning relying on a class-discriminative text conditional generative adversarial network. Later on, Pahde1 et al. (2021) tackles the multimodal few shot learning problem by employing a cross-modal feature generation network to infer the class membership of unseen samples with a simple nearest neighbor approach. Despite of the success of these methods with clean features and perfect labels, the important case that features and labels are contaminated due to out-of-distribution samples, adversarial attack and human fatigue is rarely studied. In parallel, Bayesian deep learning (BDL) has served as a powerful
tool in terms of transforming the problem of posterior inference of a BDL model into the optimization of an objective function based on latent variables. Now the question then is: how to design a Bayesian deep learning which counters the noisy labels and outliers jointly in the multimodal semisupervised few-shot learning. Accordingly, this paper tackles this challenging problem in robust cross-modal few-shot learning by integrating a deep generative heterogenous model that generalizes well to multi-modality (e.g. image-text modeling) in order to counter noisy labels and outliers. Specifically, a robust heterogeneous variational auto-encoder is first proposed to encode the noisy visual features and labels in order to jointly learn the information from both modalities by placing the uncertainty prior on the top of the infinite Gaussian mixture models. Subsequently, a robust variational lower bound based on β-divergence is derived to infer the network parameters. Finally, a robust semi-supervised GAN is integrated with the heterogenous variational auto-encoder by collapsing the generator and the decoder into one to further boost the learning capabilities. RCFSL is built on top of Bayesian deep learning by fusing cross-modal information via the approximation of the joint posterior distributions. In contrast to modality-alignment methods Xing et al. (2019a)Tsai et al. (2017) for robust few-shot learning, the robustness of RCFSL is achieved by accurate modeling of the complicated joint distribution of multi-modality data and robust variational inference by the derived lower bound. Distinct from the work in Xing et al. (2019b) calculating linear combinations in the prototypical representation space, our fusion of multimodality features in the probability distribution is completely data-driven, yielding more robust classification performance in few-shot learning. Major contributions of this paper are: (1) RCFSL harnesses three levels of denoising to ensure the robustness of cross-modal semi-supervised few-shot learning: Firstly, motivated by Hou & Galata (2008), it places the uncertainty prior of the parameters of an infinite Gaussian mixture distribution of image data to avoid mixture components collapsing onto a point or a hyperplane due to outliers. (2) Subsequently, the robust β-divergence is employed to replace Kullback-Leibler divergence used for data fitting to infer the network parameters and a novel evidence lower bound for semi-supervised few shot learning is derived. (3) Noise-transition layers are applied to both the heterogenous variational encoder and the robust discriminator in semi-supervised learning with an end-to-end training. The performance of RCFSL is further boosted with robust feature generation yielding 7% to 10% absolute accuracy improvement over STOA approaches. Related Work Previous work in multimodal few shot learning frequently tends to first learn text to image mapping to generate additional visual features and then calculate the joint prototype using a weighted average from two representations. Two recent approaches have attracted significant attention in the few-shot learning domain: Matching Networks Vinyals et al. (2016), and Prototypical Networks Snell et al. (2017) where the sample set and the query set are embedded with a neural network, and nearest neighbor classification is exploited relying on a metric in the embedded space. In Oreshkin et al. (2018), metric scaling and metric task conditioning are utilized to improve the performance of few-shot learning algorithms. Kim et al. (2018) and Finn
et al. (2018) employ a probabilistic extension to model-agnostic meta-learning (MAML) framework trained with variational approximation so that the model can generalize well to a new task with a few fine-tuning updates. In Zhang et al. (2020), a bidirectional joint image-text modeling was proposed and VHE-raster-scan-GAN was applied. RCFSL advances the work from Zhang et al. (2020) by improving the robustness of the multi-modal heterogenous encoder and extend the solution to semi-supervised few shot learning. In Tseng et al. (2020), feature-wise transformation layers are utilized for augmenting the image features relying on affine transforms to simulate various feature distributions under different domains for few-shot learning. Different from Tseng et al. (2020), RCFSL augments the robust feature generation from BDL perspective relying on robust semi-supervised GAN. Moreover, our method advances from other robust few shot learning such as Rapnets Lu et al. (2020b) and Adversarial Query Goldblum et al. (2020) by providing mathmatically rigoriously denoising schemes via uncertainty priors and robust divergence in variational inference.

heading: 2 OUR METHOD
text: Our approach is to focus on first constructing a semi-supervised robust heterogeneous variational autoencoder leveraging a mixture model to encode both image and text data in few shot learning insensitive to both noisy labels and outliers. Subsequently, a novel robust variational lower bound is derived to facilitate the inference of network parameters relying on β-divergence for both labeled and unlabeled data. Finally, a robust generative adversairial network is integrated with denoising layers to strengthen the denoising performance togehter with the end-to-end optimization to generate additional visual features to alleviate the sparsity in the semi-supervised few shot learning. Let Ω denote image space, Υ denote text space and C = {1, . . . , R} be the discrete label space. Further let xi ∈ Ω as the i-th input image observation, ti ∈ Υ as its corresponding textual description and yi ∈ C as its label. Denote Cbase as base classes where we have both labeled and unlabeled samples and denote Cnovel novel classes, which are underrepresented in the data. Inspired by the fact that the student-t distribution is more robust to the outliers than Gaussian distribution by constraining the shapes of the mixture components from collapsing Hou & Galata (2008), we propose to place uncertainty priors (e.g. Gaussian
priors) on parameters of infinite Gaussian mixture models to characterize the influence of outliers on image data and constrain the shape of the components to prevent them from collapsing. The Heterogeneous Mixture Model:Variational autoencoder Diederik & Welling (2013) has been recently proposed as a powerful solution for semi-supervised learning. To the best of our knowledge, this is the first time that a robust heterogenous VAE model has been applied which naturally integrates noisy images and text together which cohesively fuse continuous and discrete multi-modality features. Variational inference are applied to fit the heterogenous model on both of image and text features ΨI from base classes Cbase, where the embedding is obtained from the last dense layer right before the softmax output in the discriminator. Once a good mapping ΨI based on heterogenous image and text data is learnt, given a test sample, the class membership is given by assigning the class label of the closest prototype to an unseen test sample. In particular, the heterogenous features are first fed into the Dirichlet process mixture and clustered based on their similarity measures, where here image features are modeled as infinite Gaussian mixture distribution Allen et al.
(2019) to characterize the samples from the unseen classes by computing the posterior distribution for unrepresented clusters more accurately and inferring the number of classes automatically and text features are characterized by Poisson distribution. Specifically, xn is a noisy measurement of its true position and is a draw from the Gaussian mixture model, where the mean of each Gaussian component Tk is unknown and the variance Ck is known. In order to characterize the uncertainty and the outliers from the input, the Gaussian prior is placed on the top of the mean for each Gaussian component. Namely, Tk satisfies the normal distribution with the mean µk and the precision matrix Λk. ωi is the latent variable for the ith data point specifying which Gaussian it came from and π is the mixing weight for the Gaussian mixture model. Specifically, a NormalWishart prior Murphy (2007) is placed on the mean and precision of the Gaussian components: p(µ,Λ) = ∏K k=1N(µk|m0, (β0Λk)−1)W (Λk|W0, ν0) , where m0 is set to be zero and β0 is set to be a very small value. W (Λ|W, ν) is a Wishart distribution with scale matrix W and ν degrees of freedom. For text features after word embeddings, l and ζl are the prior shape and rate parameters respectively in the Gamma distribution that generates the average rate parameter for the l-th Poisson feature. In particular, the cluster assignments for each observation are drawn from multinomial distributions where the prior parameters represent the mixing weights of the corresponding clusters. Specifically, the truncated stick-breaking process is employed to construct mixing weights. By construction, a single parameter α controls the portion of the K − 1 major clusters, and χ separately controls for the portion of the remainder via a stick-breaking procedure Beta((K − 1)α, χ) Blei & Jordan (2017). Namely, where ω = [ω1, . . . , ωK ] belongs to the (K − 1)-dimensional simplex and is generated from a Dirichlet prior with two parameters p(ω)D̃ir(α, . . . , α, χ). Typically, given the image data x and the textual features t, variational inference with deep learning from powerful probabilistic models are constructed by an inference neural network q(z|x, t) and a generative neural network p(x, t|z). The generative model and the inference model are parameterized by θ and φ respectively. Subsequently, the network parameters are estimated by optimizing the evidence lower bound (ELBO) in the variational inference. For unsupervised learning of multimodality data, the vanilla VAEs are optimized by maximizing ELBO:
ELBO = Epdata(t,x)[L(t, x)], L(x) := Ez∼q(z|x,t)[ln p(t|z)]−DKL[q(z|x, t)//p(z)] (1) The Generative and Inference Model: We shall now present the proposed generative and inference model for few-shot learning. Given a semi-supervised learning setting, the labels y are either unknown (for unlabeled data) or noisy (for labeled data). The generative model is then defined as: pθ(x, t|ω, T,C)pθ(T |Z, µ,Λ, λ)pθ(a|z, ỹ, x, t)pθ(x, t|ỹ, z)p(z)p(ỹ)p(ω)p(t|λ). Define pθ as the deep neural network with parameters θ and y as the ground truth of class label. For unlabeled data, y is considered as a latent variable. Further denote Cat(.) as a multinomial distribution and in this paper we reasonably assume that the labels are of categorical distribution and the proposed model applies to other distributions for the latent variable y. In order to fit more complicated posteriors for the marginal distribution p(z|x, t) and bridge the gap between two modalities, motivated by Maaløe et al. (2017), we extend the variational distribution with auxiliary variables a, so that the generative model is invariant to marginalization over a p(x, z, a, ω, T, C, µ,Λ) = pθ(x|ω, T,C)pθ(T |Z, µ,Λ)pθ(a|z, x)pθ(x, z). To attenuate the influence with the noisy labels, further denote ỹ as the corrupted labels and ŷ as the corrected label after denoising layer. Define K×K noise transition matrix M to associate ỹ with ŷ and estimation of M has been addressed in previous methods Sukhbaatar et al. (2015)Patrini et al. (2017). In particular, M = (Mi,j) ∈ [0, 1]c×c( ∑ iMi,j = 1). The proposed generative model can then be expressed as:
p(z) = N(z|0, I) , p(ỹ) = Cat(ỹ|η) , pθ(a|z, ỹ, x, t) = f(a; z, ỹ, x, t) , pθ(x, t|z, ỹ) = f(x, t; z, ỹ, θ) , p(Tk|µk,Λk) = N(tk|µk,Λ−1k ), p(ŷ = i|ỹ = j) = Mij , (2) p(x, t|T,C, λ) = N(xn|tk, Ck) ∏Nl n=1N(tn|tỹn , Cỹn) λkexp−λ k! if the jth cluster is represented p(x, t|T,C, λ) = ∫ p(xn|tk, Ck)p(tk|µk,Λk)p(µk,Λk)dµkdΛk λ kexp−λ k!
if the jth cluster is unrepresented
The inference model can be represented as:
qφ(a, z, µ,Λ, T, ỹ, λ|x, t) = q(z|a, ỹ, x, t)q(a|x, t)q(ỹ|a, x, t)q(T, µ,Λ, λ|x, t) qφ(z|a, ỹ, x, t) = N(z|µφ(a, ỹ, x, t), diag(σ2)), qφ(ỹ|a, x, t) = Cat(ỹ|ηφ(a, x, t)), qφ(µk,Λk) = q(µk|Λk)q(Λk), q(λl|t) ∝ Gamma(λl| l + ∑ i ψiktil, ζl + ∑ i ψik) , (3)
where q(λl|t) characterizes posterior variational density of Poisson parameters given the discrete embedded feature vectors t converted from text data. ψik stands for the probability of the ith observation belongs to the cluster k. til represents the l-th Poisson feature of the ith observation. Denote N−n,j as the number of data points, excluding xn which belongs to the mixture component j. To compute q(T, µ,Λ, λ|x, t), mean-field approximation is applied Bishop (2006) to factorize all the latent variables and parameters for the represented and unrepresented jth cluster respectively:
q(T, µ,Λ, λ|x, t) = N−n,jN−1+αN(xn|tk, Ck) ∏Nl n=1N(tn|tỹn , Cỹn)q(λl|t)
q(T, µ,Λ, λ|x, t) = αN−1+α ∫ p(xn|tk, Ck)p(tk|µk,Λk)p(µk,Λk)dµkdΛkq(λl|t) (4)
The above equation indicates that given the multimodality data, the unlabeled samples has a certain probability of being classified as unrepresented mixtures (e.g. unseen new classes), which facilitates the learning capabilities of the proposed robust semi-supervised few shot learning. Within an iteration, some unlabeled samples are associated with unrepresented mixtures, a new represented mixtures will emerge which successfully addressed the challenges in semi-supervised few shot learning when the unlabeled data contains unseen new classes which doesnot exist in labeled data. Robust Variational Lower Bound: To further alleviate the impact from outliers, the robust divergence is employed to infer network parameters more accurately. The theoretical foundation of β-divergence has been initially defined at Basu et al. (1998), where the β-divergence between two functions g and f are defined as
Dβ(g ‖ f) = 1
β
∫ g(x)1+βdx+ ∫ f(x)1+βdx− β + 1
β
∫ g(x)f(x)βdx (5)
When β → 0, the β-divergence converges to KL-divergence, limβ→0Dβ(g ‖ f) = DKL(g ‖ f). As described in Futami et al. (2018), minimizing the β-divergence from the empirical distribution p̂(x) to p(x; θ) arg minθDβ(p̂(x) ‖ p(x; θ)), it is easy to show 1N ∑N i=1 p(xi; θ) β ∂ ln p(xi;θ) ∂θ − Ep(x;θ)[p(x; θ) β ∂ ln p(x;θ)
∂θ ] . As the probability densities of outliers are usually much smaller than those of inliers, the first term of the above equation is the likelihood weighted according to the power of the probability density for each sample, which effectively suppress the likelihood of outliers. This estimator is also called asM -estimator Huber & Ronchetti (2011), which provides provably superior performance in various machine applications Li & Gal (2017). The variational lower bound for the proposed RCFSL model for labeled data can be represented as
log p(x, ỹ, t) = ∫ a ∫ z ∫ T ∫ µ ∫ Λ ∫ λ ∫ ω
log(x, ỹ, a, z, T, µ,Λ, λ, t)dadzdTdµdΛdλdω ≥ E[log(pθ(a, z, T, µ,Λ, λ, ω, x, ỹ, t))]− E[qφ(a,z,T,µ,Λ,ω|x,t,ỹ)] = E[log(pθ(a, z, T, µ,Λ, ω, x, t, ỹ))]
−E[qφ(a|x,t)]− E[qφ(z|a,ỹ,x,t)]− E[qφ(T |µ,Λ,x)]− E[qφ(µ,Λ)]− E[qφ(ω|α)]− E[qφ(λ|t)] The above inequality can be rewritten as
log p(x, ỹ, t) ≥ Eqφ(a,z,T,µ,Λ,λ,ω|x,y,t) [log pθ(a,z,T,µ,Λ,λ,x,ỹ,t) qφ(a,z,T,µ,Λ,λ,ω|x,ỹ,t) ] = Eqφ(a,z,T,µ,Λ,λ,ω|x,t,ỹ)
[log(pθ(x, t, ỹ|a, z, T, µ,Λ, λ, ω))] + DKL[q(a, z, T, µ,Λ, λ|x, t, ỹ)//p(a, z, T, µ,Λ, λ)] (6) To mitigate the influence of outliers, let H = {a, z, T, ω, µ,Λ, λ} represent the set of all the latent variables and leverage the technique from Futami et al. (2018), we can replace KL-divergence with β-Divergence and cast the β-ELBO for labeled data Lβ as:
Lβ =
∫ q(H|x, t, ỹ)(−β + 1
β N∑ i=1 p(ỹi|H;xi, ti)β +N ∫ p(ỹ|H;x, t)1+βdỹ) + DKL[q(H|x, t, ỹ)//p(H)] (7)
For unlabeled data, by introducing the variational distribution for ỹ as qφ(a, x, t|ỹ), the variational lower bound for the proposed RCFSL can be represented as
log p(x, t) = ∫ a ∫ z ∫ T ∫ µ ∫ Λ ∫ ỹ log(x, ỹ, a, z, T, µ,Λ)dadzdTdµΛdỹ ≥ Eqφ(a,ỹ,z,T,µ,Λ|x,t)
[log pθ(a,z,T,µ,Λ,x,t,λ,ỹ)qφ(a,z,T,µ,Λ,λ,ỹ|x,t) ] = E[log(pθ(a, z, T, µ,Λ, ω, λ, x, t, ỹ))]− E[qφ(a|x,t)]− E[qφ(ỹ|a,x,t)] −E[qφ(z|a,ỹ,x,t)]− E[qφ(T |µ,Λ,x)]− E[qφ(µ,Λ)]− E[qφ(ω|π)]− E[qφ(λ|t)]
Similarly, replacing the KL-divergence with β-Divergence and augmenting the latent variableHu = {a, z, y, T, ω, µ,Λ, λ}, the β-ELBO for unlabeled data in RCFSL is:
Uβ = ∫ q(Hu|x, t)(− β + 1
β N∑ i=1 p(xi, ti|Hu)β + ∫ p(x, t|Hu)1+βdx) + DKL[q(Hu|x, t)//p(Hu)] ,
Practically, Lβ and Uβ are calculated via Monte Carlo sampling. The robustness of our proposed ELBO can be guaranteed leveraging the influence function (IF) Futami et al. (2018)Huber & Ronchetti (2011). As IF is widely used to analyze how much contamination affects estimated statistics, it is straightforward to show that given the perturbation on the empirical cumulative distribution caused by outliers, it is straightforward to show that the IF of our posterior distribution is bounded. The objective function for robust cross-modal variational autoencoder is: LRCV AE = Lβ + λ1Uβ , where λ1 represents the weight to control the trade-off between the labeled data and unlabeled data. Robust Feature Generation: The construction of the proposed uncertainty priors and the robust divergence measure in our framework aims at better approximation of the posterior distribution under noisy labels and outliers. This is also related to employing generative adversarial learning of z and x, t defined by the data, the encoder, the prior and decoder. Different from VAEs that assumes parametric data distribution and perform posterior inference, GANs in general utilize implicit data distribution and do not provide meaningful latent representations. By learning both a generator G and a discriminator D, a min-max objective is optimized:
min G max D {Ex,t∼pdata(x,t)[ln(D(x, t))] + Ez∼p(z)[ln(1−D(G(z))]} (8)
In Kaneko et al. (2019), a noise transition model is incorporated to learn a clean label conditional generative distribution. But their model only considered noisy labels without outliers and is limited to supervised learning. Recent work on semi-supervised GAN with k classes Kumar et al. (2017)Salimans et al. (2016) modify the discriminator with k + 1 outputs on the discriminator by considering the fake images as the k + 1th class. Hence, the loss for the training can be computed with a supervised loss and an unsupervised loss respectively:
L = Lsup + Lunsup = −Ex,t,y∼pd(x,t,y) log pf (y|x, t, y ≤ k)− Ex,t∼pg(x,t) log(pf (y = k + 1|x, t))− Ex,t∼pd(x,t) log(1− pf (y = k + 1|x, t)) , (9)
where Lsup represents the negative log probability of the label given the data is real. Denote D(x, t) = 1 − pf (y = k + 1|x, t), the loss for semi-supervised GAN on multimodality data LsGAN can be written as: L = −Ex,t∼pd(x,t) logD(x, t) − Ez∼noise(1 − D(G(z))) − Ex,t,y∼pd(x,t,y) log pf (y|x, t, y ≤ k) . More recently, the label noise robust GAN (rGAN) Kaneko et al. (2019) has achieved promising performance in classifying images with noisy labels by incorporating a noise transition model to learn a clean label conditional generative distribution under noisy labels. Thus, given the noise samples (x̃r, ỹr) ∼ p̃d(x, ỹ), to construct a label-noise robust conditional generator and discriminator, the objective function of the robust semi-supervised GAN in the proposed robust cross-modal semi-supervised few-shot learning is expressed as: LRSGAN = −Ex,t∼pd(x,t) log(1− C̃(ỹ = k+1|x, t))−Ez∼noise(1−D(G(z)))−E(x̃r,t,ỹr)∼p̃d(x,t,ỹ) log C̃(ỹ = ỹr|x, t, ỹ ≤ k) = −Ex,t∼pd(x,t) log(1 −Mŷr,ỹr Ĉ(ŷ = k + 1|x, t)) − Ez∼noise(1 − D(G(z))) − E(x̃r,t,ỹr)∼p̃d(x,t,ỹ) log ∑ ŷr Mŷr,ỹr Ĉ(ŷ = ŷ
r|x, t, ŷ ≤ k) . Without modeling the data distribution explicitly and representing the latent space in a meaningful manner, it does not provide the functionality to counter the outliers in the generator and discriminator. Motivated by the thought of bridging the gap between robust VAE and robust GAN, we feed the robust variational posterior p(z|x, t) instead of the random noise p(z) into the label noise robust semi-supervised GAN as the source of randomness as both of the decoder and the generator of RSGAN share the mapping from z to x and t. The full optimization function for the proposed RCFSL framework can be represented as:
minGRCFSL maxD Epdata(x,t)[L(x, t)]L(x, t) := lnD(x, t) + DKL[q(H|x, t, ỹ)//p(H)] + λ1DKL[q(Hu|x, t)//p(Hu)] + Ez∼q(H|x,t,ỹ)[ln(1−D(GRSGAN (z)) + (−β+1β ∑N i=1 p(ỹi|H;xi, ti)β +
N ∫ p(ỹ|H;x, t)1+βdỹ] + λ1Ez∼q(Hu|x,t)[(− β+1 β ∑N i=1 p(xi, ti|Hu)β + λ1N ∫ p(x, t|Hu)1+βdx)] , (10)
where λ1 is set to be the ratio of unlabled samples verus labeled samples and the discriminator loss is characterized by E[lnD(x, t)] = −Ex,t∼pd(x,t) log(1 − Mŷr,ỹr Ĉ(ŷ = k + 1|x, t)) − E(x̃r,t,ỹr)∼p̃d(x,t,ỹ) log ∑ ŷr Mŷr,ỹr Ĉ(ŷ = ŷ
r|x, t, ŷ ≤ k) . This new architecture is expected to better predict the class labels under the compound noise. We then train RCFSL including the robust heterogenous encoder, the generator and the robust discriminator in an end-to-end manner using adaptive moment estimation (Adam) Kingma & Ba (2015). Each multimodal embedding prototype pc (of category c) is computed by averaging the embeddings of all support samples of class c. Once the robust embedding is obtained, the distance between the embedding of the query qt and the multimodal prototype pc is calculated by p(y = c|pc) = exp(−d(f(qt),pc))∑ k exp(−d(f(qt),pk))
where d refers to Euclidean distance and the query is classified as the class with the minimum distance.

heading: 3 EXPERIMENTS
text: Dataset and Competing Methods: We extensively evaluate the proposed robust cross-modal semisupervised few shot learning (RCFSL) algorithm on multiple benchmark datasets including miniImageNet(mIN), tieredImageNet (tIN), Fewshot-CIFAR100 (FC100) and Caltech-UCSD Birds 200- 2011 (CUB). The miniImageNet dataset is a subset of ILSCRS-12 which includes 100 classes where each class contains 600 RGB images with size 84 by 84. We split the datasets by using 64 classes for training, 16 classes for validation and 20 classes for testing where 16 classes are utilized to monitor the model’s generalization performance. The CUB Bird dataset contains 11788 images of 200 different bird species, where the data is split equally in training and testing data. Namely, for each category, 30 training image and 30 testing images are included, where in the training dataset, 10% images are labeled images and the rest of the images are unlabeled. 10 short textual descriptions per image are provided by Reed et al. (2016). Following the work Zhang et al. (2017), the text encoder pretrained by Reed et al. (2016) is employed and the data is split such as |CBase| = 150, |CNovel| = 50. Specifically, n = {1, 2, 5, 10, 15, 20} images are selected from novel classes in order to perform few shot learning. We evaluate the two types of label noise including symmetric noise and asymmetric noise. For symmetric noise, we inject the label noise by randomly flipping the labels of the labeled data into a different label in the novel classes. For asymmetric noise, the corrupted labels are generated by replacing the correct labels with their most similar classes (using nearest neighbor measurement). Outliers are mimicked by including samples from data distributions far away from the training data distributions, namely out-of-distribution (OOD) samples. GloVe Pennington et al. (2014) is utilized to extract the word embeddings for the category labels, where the embeddings are trained with large unsupervised text corpora. RCFSL is compared with multiple benchmark methods and state-of-the-art approaches. Typically, similarly as Xing et al. (2019b), three families of methods are evaluated: (1) uni-modality few-shot learning methods such as MAMLFinn et al. (2017), LEORusu et al. (2019) and CAMLJiang et al. (2019). In particular, latent embedding optimization (LEO) Rusu et al. (2019) addresses MAML problem by relying on a few updates on a low data regime to train models in a high dimensional parameter space. (2) cross modality few shot learning methods ReViSE Tsai et al. (2017), CADA-VAE Schönfeld et al. (2019) Xing et al. (2019b), VHE-raster-scan-GANZhang et al. (2020) and Feature transformTseng et al. (2020). Among them, ReViSE Tsai et al. (2017) minimizes maximum mean discrepancy (MMD) of the distributions from two representation spaces for better alignment. CADA-VAE Schönfeld et al.
(2019) leverages two VAEs to embed information for both modalities and align the distribution of the two latent spaces. (3) Robust few shot learning methods including robust attentive profile networks (Rapnets) Lu et al. (2020b) and Adversarial Querying (AQ) Goldblum et al. (2020). RapnetsLu et al. (2020b) consists of an embedding module, a correlation module, and an attentive module to suppress the outliers in few-shot learning. Adversarial Querying Goldblum et al. (2020) produces adversarially robust meta-learners and achieves efficient classification for few-shot learning tasks. Implementation Details: For miniImagenet, tiredImagenet and FC100 datasets, we parameterize the heterogeneous VAE with three sets of 5 by 5 fully convolutional, ReLU and pooling layers followed by two fully connected hidden layers where each pair of layers contains the hidden units as dim(h) = 1000. We further set the dimension of the auxiliary variable a and the latent variable z to be 200 respectively. For the CUB dataset, we leverage Resnet-18 as backbone and incorporate the uncertainty prior for the input, the noise transition model along with the auxiliary variables as our encoder. The network is trained with SGD using a batch size of 128. A momentum of 0.9 is set with a weight decay of 0.0005. The network is trained for 30000 iterations. The initial learning rate is set as 0.1, and reduce it by a factor of 10 after iterations 15,000, 17,500 and 19,000. For the configurations of discriminator in GAN, we follow the same configuration of DC-GAN and follow the similar discriminator Mirza & Osindero (2014) for DC-GAN. β is varied from 0.1 to 0.4 where the best performance is reported. The similar setup in Xing et al. (2019b) is utilized relying on ACC(N,K,C) to evaluate the performance of classification accuracy in a N -way and K-shot SFSL settings, where C represents the total number of outliers and noisy labels in each class. We use 10 textual descriptions per image to generate multimodal prototypes. The semi-supervised fewshot classification accuracy on test split of miniImageNet is shown in Table 1. As shown in Table 1, RCFSL achieves the best classification accuracy compared to all of the competing methods in different cases. The large performance margin suggests the proposed mechanism not only fuses two modalities of features cohesively and but also effectively suppresses the noisy labels and outliers in semi-supervised few-shot learning. Compared to the adaptive modality mixture mechanism (AM3) with TADAM as backbone Xing et al. (2019b), RCFSL has the advantage of fusing multimodal distributions in the heterogeneous encoder via the latent space to fully utilize the multimodal feature representation instead of computing multimodal prototypes with a convex combination afterwards. In contrast to robust FSL methods like Rapnets Lu et al. (2020b) and AQ Goldblum et al. (2020), the performance gain can be mainly attributed to the fact that RCFSL employs a mathematically rigorous variational inference insensitive to noisy labels and outliers to optimize network parameters, yielding more accurate classification performance. The margin in performance is particularly
remarkable in the fewer shots scenario, supporting the robustness of RCFSL. Similar performance evaluations are conducted on tieredImageNet dataset in Table 2 and RCFSL again is the top performer under various noise settings. Fig.3 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-100 dataset using RapNetsLu et al. (2020b) and our RCFSL (β=0.2) on a 10-Way 8-shot semi-supervised learning (Fig 3.(a)- (b)), where it clearly shows our method has a better capability in terms of separating noisy labels and outliers in semi-supervised few-shot learning. We also investigate the classification accuracy of different methods versus the number of shots in Fig 3.(c)- (d) for FC100 and CUB200 datasets respectively with 25% noisy labels and outliers respectively, RCFSL yields considerable performance gains over competing methods especially with smaller number of shots. Ablation Study: Described in the ablation study in Table 3, RCFSL w/o uncertainty prior excludes the uncertainty prior from the model. Therefore, the performance degradation suggests the importance of the proposed hierarchical structure for variational inference by placing the uncertainty prior on the infinite GMMs to counter the detrimental effects of outliers. Secondly, our method w/o robust divergence replaces β-divergence which places small weights on noisy labels and outliers with the regular KL-divergence. Moreover, RCFSL w/o cross modality relies on only image data for SFSL where multi-modality further improves the classification accuracy by at least 4%.

heading: 4 CONCLUSION
text: By integrating uncertainty prior of an infinite Gaussian mixture model into the heterogenous encoder and the robust lower bound based on β-divergence for variational inference, our RCFSL is capable of tackling the outliers and noisy labels simultaneously in semi-supervised few shot learning. Moreover, by integrating the robust semi-supervised GAN for feature generation, experimental evaluations on multiple benchmark datasets have demonstrate the superiority of the proposed method with large margins using multimodality data.

heading: A DERIVATION OF INFLUENCE FUNCTION FOR LOWER BOUND OF RCFSL
text: Define G(x) as a empirical cumulative distribution given by {xi}ni=1 and denote the perturbed version of G at z as Gε,z(x) = (1− ε)G(x) + ∆z(x), where ε is the contamination portion and ∆z(x) is the point mass at x. Given a statistic T . the influence function (IF) is defined as Futami et al. (2018)
IF (z, T,G) = ∂T (Gz,ε(x))
∂ε |ε=0 (11)
Thus, the IF of the heterogenous variational autoencoder for LRCV AE is given by
(∂ 2LRCVAE ∂H2u )−1 ∂∂HuEq(Hu)[DKL[q(Hu|x, t)//p(Hu)]
+N(β+1β p(z|Hu) β − ∫ p(x, t|Hu)1+βdx]
+(∂ 2LRCVAE ∂H2 ) −1 ∂ ∂HEq(H)[DKL[q(H|x, t)//p(H)]
+N(β+1β p(ỹ|x, t, H) β −
∫ p(ỹ|x, t, H)1+βdỹ] , (12)
It is straightforward to show that the above result is always bounded, namely RCFSL is robust to the compound noise (the outliers on the data x and the labels y).
B VISUAL COMPARISON WITH ROBUST DIVERGENCE
Fig.1 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-100 dataset using the same setting as our method except replacing robust divergence with KL-divergence and our RCFSL (β=0.2) on a 10-Way 8-shot semi-supervised learning, where it clearly shows the efficacy of the robust divergence by offering a stronger capability in separating noisy labels and outliers in semi-supervised few-shot learning.

heading: C DETAILS ON NETWORK STRUCTURE, TRAINING AND INFERENCE TIME COMPARISON
text: The discriminator of RCFSL following deep convolutional GAN is structured as follows: the first convolutional layer is of size 32×32×64, the second convolutional layer is of dimension 16×16× 128, the third layer is 8×8×256 and the last convolutional layer is of the dimension 4×4×512 with the convolutional filter size 5×5×3. Each convolutional layer is followed by a batch normalization layer and then a ReLU layer.
We analyze the training time (hours) and inference time (seconds) of our RCFSL to demonstrate its computational efficiency in Table 4 and Table 5. The training and the inference time of RCFSL is reported on Fewshot-CIFAR100 with several state-of-the-art approaches, evaluated on a single Nvidia V100 GPU. RCFSL is slower than AQ Goldblum et al. (2020), however is faster than the rest of competing methods for both training and inference time due to the efficient probabilistic graphical model and the effective novel lower bound and compact network architecture.

heading: D DISCUSSION ON DIFFERENCES FROM OTHER MULTIMODAL VAES
text: Prior approaches Pandey & Dukkipati (2017) on multimodal VAEs mostly focus on cross-modal generation. In particular, given two modalities x1 and x2(e.g. image and text), these approaches
learn the conditional generative model p(x1|x2), where the conditioning modality x2 and the generation modality x1 are usually not interchangebale. Compared to Pandey & Dukkipati (2017), our heterogeneous robust VAE targets joint image-text modeling by explicitly modelling the joint distribution over latent variables from both image and textual data. Moreover, our multimodal VAEs integrates the uncertainty priors, denoising layers and robust variational lower bound to ensure the robustness in the presence of noisy labels and outliers. The work Suzuki et al. (2017) introduced joint multimodal VAE (JMVAE) which aims at learning shared representation with joint encoder qψ(z|x1, x2). Distinct from Suzuki et al. (2017) which does not consider the robust modeling to tackle with outliers, our approach leverages infinite Gaussian mixture models equipped with uncertainty prior which not only characterizes the distributions for image-text modeling more accurately but also facilitates the learning of new unseen classes by inferring the number of classes automatically in few-shot learning. Recently, Zhang et al. (2020) utilized a VAE that encodes an image into a deterministic-upward–stochastic-downward ladder-structured latent representation, and then is applied to decode the corresponding text. While our work is relying on joint encoding and decoding schemes on image-text modelling and focusing on tackling the noisy-labels and outliers by enhancing the robustness of multimodal modeling using the various proposed techniques. More recently, Shi et al. (2019) leverages a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, indicating the suitability of mixture models for learning from multi-modal data. Our work (RCFSL) advances MMVAE Shi et al. (2019) in three aspects: (1) We places uncertainty prior on the parameters of mixture models to avoid the collapsing of models in the presence of outliers and derived novel inference solution given the uncertainty prior (shown in our novel generative and inference models in Eqs (2)-(4)); (2) instead of using importance weighted autoencoder (IWAE) estimator Burda et al. (2015)Shi et al. (2019), RCFSL takes advantage of robust divergence to derive the robust evidence lower bound for unlabeled data and labeled data respectively which are unique and mathematically rigorously (e.g. Eq(7)), yielding superior performance for classification of noisy dataset in semi-supervised few-shot learning. (3) RCFSL integrates the denoising layers in both multimodal VAE and robust GAN to mitigate the influence of noisy labels, which further enhances the robustness of the model.

heading: E EVALUATION ON SEMANTIC LABEL NOISE
text: In order to evaluate our algorithm under real-world noise (semantic label noise), RCFSL is further evaluated on Clothing 1M dataset with the network architecture Resnet-18. For Clothing1M dataset, it includes 1 million training images with 14 classes obtained from online shopping websites and labels are generated from surrounding texts. Since the data is collected from multiple online shopping websites and include many mislabelled samples, therefore the labels are contaminated by real-world noise (semantic label noise). The data is split such as |CBase| = 10, |CNovel| = 4. Table 6 demonstrates the comparison of classification accuracy using different learning algorithms on the Clothing1M datasets (real-world noise with 10% and 20% outliers respectively), where n = 5 images are selected from novel classes in order to perform few shot learning. As it can be seen from Table 6, our RCFSL continues to serve as the best performing method in the presence of semantic label noise compared to other competing methods under different noise statistics, which further confirms the outstanding learning capability of the proposed approach in robust semi-supervised few-shot learning. Moreover, Figure 5 illustrates exemplary images with compound noise detected
by RCFSL from Clothing1M dataset, where the false labels are noted in orange and the true labels are noted in light blue.

heading: F EVALUATION ON OMNIGLOT DATASET
text: We further evaluate the proposed algorithm on Omniglot dataset. The Omniglot dataset includes 1623 different handwritten characters from 50 different alphabets where each of the 1623 characters was drawn online via Amazon’s Mechanical Turk by 20 different people and each image is paired with stroke data, a sequences of spatial and temporal coordinates with in milliseconds. Table 7 demonstrates the comparison of classification accuracy using different learning algorithms on the Omniglot dataset (20% symmetric label noise with 10% outliers) for semi-supervised few shot learning on 1000 test episodes, where it can be seen from Table 7 that our RCFSL provides the best performance against other competing methods in the presence of noisy labels and outliers in semi-supervised few-shot learning.

heading: G COMPARISON OF LOG-LIKELIHOOD WITH ROBUST DIVERGENCE
text: We report the roust lower bound for the unlabeled data with 5000 importance weighted samples where the similar setting as Rasmus et al. (2015) with warm up, batch normalization and 1 Monte Carlo and IW sample for training. The percentage of outliers is set to be 20%. Table 8 demonstrates the log-likelihood scores for the permutation invariant MNIST dataset. The results shown in Table 8 indicates the the proposed method has strong expressive power by performing better than other methods in terms of log-likelihood due to the utilization of the robust divergence in the inference, especially by comparing with the same RCFSL method but with KL-divergence.

heading: H ANALYSIS FOR DIFFERENT TERMS IN THE OPTIMIZATION FUNCTION
text: In the above optimization cost function of RCFSL in the equation (10), the first term represents the supervised and unsupervised loss from the discriminator, the second and the third terms tend to minimize the approximation error by ”regularizing” q(H|x, t, ỹ) and q(Hu|x, t) to the prior p(H) and p(Hu) for labeled data and unlabeled data respectively. The fourth term characterizes the generator loss from the robust semi-supervised GAN (RSGAN). The fifth to the eighth terms are employed to enhance the robustness in the presence of data outliers with β-divergence for the ELBOs on the
labeled data and unlabeled data respectively, which is shown to be insensitive to the small contamination of the data.

Review summaries:
The paper presents a cross-modal semi-supervised few-shot learning approach for image classification. The idea is to train variational auto-encoder (VAE) with both image and text data for learning a feature representation. Then features are extracted from a test sample and assigned to the class of the closest prototype train sample. In addition, a generative adversarial objective is employed for learning the latent code of the VAE. Since the proposed idea is meant for learning from noisy labels, there is an uncertainty prior in the image features as part of the infinite Gaussian mixture distribution. The approach is evaluated on standard few-shot learning benchmarks which are modified to account for the noise labels. The results are promising compared to the prior work. 

This paper focuses on semi-supervised few-shot learning with multi-modality data. The authors introduce an uncertainty prior of an infinite Gaussian model, integrating multi-modality information from image and text data into a heterogenous variational autoencoder. Meanwhile, a new variational lower bound is derived for the inference of parameters. In addition, a GAN is developed for the data sparsity in few shot learning. Experimental results demonstrate the effectiveness of the proposed method to some extent.