Robust Cross-Modal Semi-supervised Few Shot Learning

1 INTRODUCTION . Despite the impressive success of deep learning models , frequently it requires massive amount of training data to fully demonstrate the potential of the model . In contrast , human is capable of learning new concepts given limited data . Consequently , few-shot learning gathers extensive research interest due to the capabilities of learning new concepts from limited training data . Nevertheless , the success of few-shot learning requires careful handling to robustness and generalization as it is extremely susceptible to noisy labels , outliers as well as adversarial attack Lu et al . ( 2020a ) . For instance , in order to automatically recognize several kinds of uncommon animals , only a few annotated images for them are available due to their rarity . Moreover , the images could potentially be corrupted due to an uncontrollable shooting environment or an instrumental malfunction . To mitigate this , one common approach to few shot learning is meta-learning Ren et al . ( 2018 ) , where the goal is to learn a classifier to distinguish between previously unseen classes , given labeled classes and a larger pool of unseen examples , some of which may belong to the classes of interest , namely semi-supervised few shot learning ( SFSL ) . Despite the impressive capabilities equipped the ability to leverage unlabeled examples for SFSL , the challenge of lacking novel samples remains to be a bottleneck . Besides visual information , textual data frequently contains rich information and more descriptive concepts for learning . Incorporating image-text multi-modal learning into the framework by training on image-text pairs provides an efficient tool to inject the diversity to the generation process Pahde1 et al . ( 2021 ) Pahde1 et al . ( 2018 ) . The work in Pahde1 et al . ( 2018 ) provides a benchmark for multimodal few-shot learning relying on a class-discriminative text conditional generative adversarial network . Later on , Pahde1 et al . ( 2021 ) tackles the multimodal few shot learning problem by employing a cross-modal feature generation network to infer the class membership of unseen samples with a simple nearest neighbor approach . Despite of the success of these methods with clean features and perfect labels , the important case that features and labels are contaminated due to out-of-distribution samples , adversarial attack and human fatigue is rarely studied . In parallel , Bayesian deep learning ( BDL ) has served as a powerful tool in terms of transforming the problem of posterior inference of a BDL model into the optimization of an objective function based on latent variables . Now the question then is : how to design a Bayesian deep learning which counters the noisy labels and outliers jointly in the multimodal semisupervised few-shot learning . Accordingly , this paper tackles this challenging problem in robust cross-modal few-shot learning by integrating a deep generative heterogenous model that generalizes well to multi-modality ( e.g . image-text modeling ) in order to counter noisy labels and outliers . Specifically , a robust heterogeneous variational auto-encoder is first proposed to encode the noisy visual features and labels in order to jointly learn the information from both modalities by placing the uncertainty prior on the top of the infinite Gaussian mixture models . Subsequently , a robust variational lower bound based on β-divergence is derived to infer the network parameters . Finally , a robust semi-supervised GAN is integrated with the heterogenous variational auto-encoder by collapsing the generator and the decoder into one to further boost the learning capabilities . RCFSL is built on top of Bayesian deep learning by fusing cross-modal information via the approximation of the joint posterior distributions . In contrast to modality-alignment methods Xing et al . ( 2019a ) Tsai et al . ( 2017 ) for robust few-shot learning , the robustness of RCFSL is achieved by accurate modeling of the complicated joint distribution of multi-modality data and robust variational inference by the derived lower bound . Distinct from the work in Xing et al . ( 2019b ) calculating linear combinations in the prototypical representation space , our fusion of multimodality features in the probability distribution is completely data-driven , yielding more robust classification performance in few-shot learning . Major contributions of this paper are : ( 1 ) RCFSL harnesses three levels of denoising to ensure the robustness of cross-modal semi-supervised few-shot learning : Firstly , motivated by Hou & Galata ( 2008 ) , it places the uncertainty prior of the parameters of an infinite Gaussian mixture distribution of image data to avoid mixture components collapsing onto a point or a hyperplane due to outliers . ( 2 ) Subsequently , the robust β-divergence is employed to replace Kullback-Leibler divergence used for data fitting to infer the network parameters and a novel evidence lower bound for semi-supervised few shot learning is derived . ( 3 ) Noise-transition layers are applied to both the heterogenous variational encoder and the robust discriminator in semi-supervised learning with an end-to-end training . The performance of RCFSL is further boosted with robust feature generation yielding 7 % to 10 % absolute accuracy improvement over STOA approaches . Related Work Previous work in multimodal few shot learning frequently tends to first learn text to image mapping to generate additional visual features and then calculate the joint prototype using a weighted average from two representations . Two recent approaches have attracted significant attention in the few-shot learning domain : Matching Networks Vinyals et al . ( 2016 ) , and Prototypical Networks Snell et al . ( 2017 ) where the sample set and the query set are embedded with a neural network , and nearest neighbor classification is exploited relying on a metric in the embedded space . In Oreshkin et al . ( 2018 ) , metric scaling and metric task conditioning are utilized to improve the performance of few-shot learning algorithms . Kim et al . ( 2018 ) and Finn et al . ( 2018 ) employ a probabilistic extension to model-agnostic meta-learning ( MAML ) framework trained with variational approximation so that the model can generalize well to a new task with a few fine-tuning updates . In Zhang et al . ( 2020 ) , a bidirectional joint image-text modeling was proposed and VHE-raster-scan-GAN was applied . RCFSL advances the work from Zhang et al . ( 2020 ) by improving the robustness of the multi-modal heterogenous encoder and extend the solution to semi-supervised few shot learning . In Tseng et al . ( 2020 ) , feature-wise transformation layers are utilized for augmenting the image features relying on affine transforms to simulate various feature distributions under different domains for few-shot learning . Different from Tseng et al . ( 2020 ) , RCFSL augments the robust feature generation from BDL perspective relying on robust semi-supervised GAN . Moreover , our method advances from other robust few shot learning such as Rapnets Lu et al . ( 2020b ) and Adversarial Query Goldblum et al . ( 2020 ) by providing mathmatically rigoriously denoising schemes via uncertainty priors and robust divergence in variational inference . 2 OUR METHOD . Our approach is to focus on first constructing a semi-supervised robust heterogeneous variational autoencoder leveraging a mixture model to encode both image and text data in few shot learning insensitive to both noisy labels and outliers . Subsequently , a novel robust variational lower bound is derived to facilitate the inference of network parameters relying on β-divergence for both labeled and unlabeled data . Finally , a robust generative adversairial network is integrated with denoising layers to strengthen the denoising performance togehter with the end-to-end optimization to generate additional visual features to alleviate the sparsity in the semi-supervised few shot learning . Let Ω denote image space , Υ denote text space and C = { 1 , . . . , R } be the discrete label space . Further let xi ∈ Ω as the i-th input image observation , ti ∈ Υ as its corresponding textual description and yi ∈ C as its label . Denote Cbase as base classes where we have both labeled and unlabeled samples and denote Cnovel novel classes , which are underrepresented in the data . Inspired by the fact that the student-t distribution is more robust to the outliers than Gaussian distribution by constraining the shapes of the mixture components from collapsing Hou & Galata ( 2008 ) , we propose to place uncertainty priors ( e.g . Gaussian priors ) on parameters of infinite Gaussian mixture models to characterize the influence of outliers on image data and constrain the shape of the components to prevent them from collapsing . The Heterogeneous Mixture Model : Variational autoencoder Diederik & Welling ( 2013 ) has been recently proposed as a powerful solution for semi-supervised learning . To the best of our knowledge , this is the first time that a robust heterogenous VAE model has been applied which naturally integrates noisy images and text together which cohesively fuse continuous and discrete multi-modality features . Variational inference are applied to fit the heterogenous model on both of image and text features ΨI from base classes Cbase , where the embedding is obtained from the last dense layer right before the softmax output in the discriminator . Once a good mapping ΨI based on heterogenous image and text data is learnt , given a test sample , the class membership is given by assigning the class label of the closest prototype to an unseen test sample . In particular , the heterogenous features are first fed into the Dirichlet process mixture and clustered based on their similarity measures , where here image features are modeled as infinite Gaussian mixture distribution Allen et al . ( 2019 ) to characterize the samples from the unseen classes by computing the posterior distribution for unrepresented clusters more accurately and inferring the number of classes automatically and text features are characterized by Poisson distribution . Specifically , xn is a noisy measurement of its true position and is a draw from the Gaussian mixture model , where the mean of each Gaussian component Tk is unknown and the variance Ck is known . In order to characterize the uncertainty and the outliers from the input , the Gaussian prior is placed on the top of the mean for each Gaussian component . Namely , Tk satisfies the normal distribution with the mean µk and the precision matrix Λk . ωi is the latent variable for the ith data point specifying which Gaussian it came from and π is the mixing weight for the Gaussian mixture model . Specifically , a NormalWishart prior Murphy ( 2007 ) is placed on the mean and precision of the Gaussian components : p ( µ , Λ ) = ∏K k=1N ( µk|m0 , ( β0Λk ) −1 ) W ( Λk|W0 , ν0 ) , where m0 is set to be zero and β0 is set to be a very small value . W ( Λ|W , ν ) is a Wishart distribution with scale matrix W and ν degrees of freedom . For text features after word embeddings , l and ζl are the prior shape and rate parameters respectively in the Gamma distribution that generates the average rate parameter for the l-th Poisson feature . In particular , the cluster assignments for each observation are drawn from multinomial distributions where the prior parameters represent the mixing weights of the corresponding clusters . Specifically , the truncated stick-breaking process is employed to construct mixing weights . By construction , a single parameter α controls the portion of the K − 1 major clusters , and χ separately controls for the portion of the remainder via a stick-breaking procedure Beta ( ( K − 1 ) α , χ ) Blei & Jordan ( 2017 ) . Namely , where ω = [ ω1 , . . . , ωK ] belongs to the ( K − 1 ) -dimensional simplex and is generated from a Dirichlet prior with two parameters p ( ω ) D̃ir ( α , . . . , α , χ ) . Typically , given the image data x and the textual features t , variational inference with deep learning from powerful probabilistic models are constructed by an inference neural network q ( z|x , t ) and a generative neural network p ( x , t|z ) . The generative model and the inference model are parameterized by θ and φ respectively . Subsequently , the network parameters are estimated by optimizing the evidence lower bound ( ELBO ) in the variational inference . For unsupervised learning of multimodality data , the vanilla VAEs are optimized by maximizing ELBO : ELBO = Epdata ( t , x ) [ L ( t , x ) ] , L ( x ) : = Ez∼q ( z|x , t ) [ ln p ( t|z ) ] −DKL [ q ( z|x , t ) //p ( z ) ] ( 1 ) The Generative and Inference Model : We shall now present the proposed generative and inference model for few-shot learning . Given a semi-supervised learning setting , the labels y are either unknown ( for unlabeled data ) or noisy ( for labeled data ) . The generative model is then defined as : pθ ( x , t|ω , T , C ) pθ ( T |Z , µ , Λ , λ ) pθ ( a|z , ỹ , x , t ) pθ ( x , t|ỹ , z ) p ( z ) p ( ỹ ) p ( ω ) p ( t|λ ) . Define pθ as the deep neural network with parameters θ and y as the ground truth of class label . For unlabeled data , y is considered as a latent variable . Further denote Cat ( . ) as a multinomial distribution and in this paper we reasonably assume that the labels are of categorical distribution and the proposed model applies to other distributions for the latent variable y . In order to fit more complicated posteriors for the marginal distribution p ( z|x , t ) and bridge the gap between two modalities , motivated by Maaløe et al . ( 2017 ) , we extend the variational distribution with auxiliary variables a , so that the generative model is invariant to marginalization over a p ( x , z , a , ω , T , C , µ , Λ ) = pθ ( x|ω , T , C ) pθ ( T |Z , µ , Λ ) pθ ( a|z , x ) pθ ( x , z ) . To attenuate the influence with the noisy labels , further denote ỹ as the corrupted labels and ŷ as the corrected label after denoising layer . Define K×K noise transition matrix M to associate ỹ with ŷ and estimation of M has been addressed in previous methods Sukhbaatar et al . ( 2015 ) Patrini et al . ( 2017 ) . In particular , M = ( Mi , j ) ∈ [ 0 , 1 ] c×c ( ∑ iMi , j = 1 ) . The proposed generative model can then be expressed as : p ( z ) = N ( z|0 , I ) , p ( ỹ ) = Cat ( ỹ|η ) , pθ ( a|z , ỹ , x , t ) = f ( a ; z , ỹ , x , t ) , pθ ( x , t|z , ỹ ) = f ( x , t ; z , ỹ , θ ) , p ( Tk|µk , Λk ) = N ( tk|µk , Λ−1k ) , p ( ŷ = i|ỹ = j ) = Mij , ( 2 ) p ( x , t|T , C , λ ) = N ( xn|tk , Ck ) ∏Nl n=1N ( tn|tỹn , Cỹn ) λkexp−λ k ! if the jth cluster is represented p ( x , t|T , C , λ ) = ∫ p ( xn|tk , Ck ) p ( tk|µk , Λk ) p ( µk , Λk ) dµkdΛk λ kexp−λ k ! if the jth cluster is unrepresented The inference model can be represented as : qφ ( a , z , µ , Λ , T , ỹ , λ|x , t ) = q ( z|a , ỹ , x , t ) q ( a|x , t ) q ( ỹ|a , x , t ) q ( T , µ , Λ , λ|x , t ) qφ ( z|a , ỹ , x , t ) = N ( z|µφ ( a , ỹ , x , t ) , diag ( σ2 ) ) , qφ ( ỹ|a , x , t ) = Cat ( ỹ|ηφ ( a , x , t ) ) , qφ ( µk , Λk ) = q ( µk|Λk ) q ( Λk ) , q ( λl|t ) ∝ Gamma ( λl| l + ∑ i ψiktil , ζl + ∑ i ψik ) , ( 3 ) where q ( λl|t ) characterizes posterior variational density of Poisson parameters given the discrete embedded feature vectors t converted from text data . ψik stands for the probability of the ith observation belongs to the cluster k. til represents the l-th Poisson feature of the ith observation . Denote N−n , j as the number of data points , excluding xn which belongs to the mixture component j . To compute q ( T , µ , Λ , λ|x , t ) , mean-field approximation is applied Bishop ( 2006 ) to factorize all the latent variables and parameters for the represented and unrepresented jth cluster respectively : q ( T , µ , Λ , λ|x , t ) = N−n , jN−1+αN ( xn|tk , Ck ) ∏Nl n=1N ( tn|tỹn , Cỹn ) q ( λl|t ) q ( T , µ , Λ , λ|x , t ) = αN−1+α ∫ p ( xn|tk , Ck ) p ( tk|µk , Λk ) p ( µk , Λk ) dµkdΛkq ( λl|t ) ( 4 ) The above equation indicates that given the multimodality data , the unlabeled samples has a certain probability of being classified as unrepresented mixtures ( e.g . unseen new classes ) , which facilitates the learning capabilities of the proposed robust semi-supervised few shot learning . Within an iteration , some unlabeled samples are associated with unrepresented mixtures , a new represented mixtures will emerge which successfully addressed the challenges in semi-supervised few shot learning when the unlabeled data contains unseen new classes which doesnot exist in labeled data . Robust Variational Lower Bound : To further alleviate the impact from outliers , the robust divergence is employed to infer network parameters more accurately . The theoretical foundation of β-divergence has been initially defined at Basu et al . ( 1998 ) , where the β-divergence between two functions g and f are defined as Dβ ( g ‖ f ) = 1 β ∫ g ( x ) 1+βdx+ ∫ f ( x ) 1+βdx− β + 1 β ∫ g ( x ) f ( x ) βdx ( 5 ) When β → 0 , the β-divergence converges to KL-divergence , limβ→0Dβ ( g ‖ f ) = DKL ( g ‖ f ) . As described in Futami et al . ( 2018 ) , minimizing the β-divergence from the empirical distribution p̂ ( x ) to p ( x ; θ ) arg minθDβ ( p̂ ( x ) ‖ p ( x ; θ ) ) , it is easy to show 1N ∑N i=1 p ( xi ; θ ) β ∂ ln p ( xi ; θ ) ∂θ − Ep ( x ; θ ) [ p ( x ; θ ) β ∂ ln p ( x ; θ ) ∂θ ] . As the probability densities of outliers are usually much smaller than those of inliers , the first term of the above equation is the likelihood weighted according to the power of the probability density for each sample , which effectively suppress the likelihood of outliers . This estimator is also called asM -estimator Huber & Ronchetti ( 2011 ) , which provides provably superior performance in various machine applications Li & Gal ( 2017 ) . The variational lower bound for the proposed RCFSL model for labeled data can be represented as log p ( x , ỹ , t ) = ∫ a ∫ z ∫ T ∫ µ ∫ Λ ∫ λ ∫ ω log ( x , ỹ , a , z , T , µ , Λ , λ , t ) dadzdTdµdΛdλdω ≥ E [ log ( pθ ( a , z , T , µ , Λ , λ , ω , x , ỹ , t ) ) ] − E [ qφ ( a , z , T , µ , Λ , ω|x , t , ỹ ) ] = E [ log ( pθ ( a , z , T , µ , Λ , ω , x , t , ỹ ) ) ] −E [ qφ ( a|x , t ) ] − E [ qφ ( z|a , ỹ , x , t ) ] − E [ qφ ( T |µ , Λ , x ) ] − E [ qφ ( µ , Λ ) ] − E [ qφ ( ω|α ) ] − E [ qφ ( λ|t ) ] The above inequality can be rewritten as log p ( x , ỹ , t ) ≥ Eqφ ( a , z , T , µ , Λ , λ , ω|x , y , t ) [ log pθ ( a , z , T , µ , Λ , λ , x , ỹ , t ) qφ ( a , z , T , µ , Λ , λ , ω|x , ỹ , t ) ] = Eqφ ( a , z , T , µ , Λ , λ , ω|x , t , ỹ ) [ log ( pθ ( x , t , ỹ|a , z , T , µ , Λ , λ , ω ) ) ] + DKL [ q ( a , z , T , µ , Λ , λ|x , t , ỹ ) //p ( a , z , T , µ , Λ , λ ) ] ( 6 ) To mitigate the influence of outliers , let H = { a , z , T , ω , µ , Λ , λ } represent the set of all the latent variables and leverage the technique from Futami et al . ( 2018 ) , we can replace KL-divergence with β-Divergence and cast the β-ELBO for labeled data Lβ as : Lβ = ∫ q ( H|x , t , ỹ ) ( −β + 1 β N∑ i=1 p ( ỹi|H ; xi , ti ) β +N ∫ p ( ỹ|H ; x , t ) 1+βdỹ ) + DKL [ q ( H|x , t , ỹ ) //p ( H ) ] ( 7 ) For unlabeled data , by introducing the variational distribution for ỹ as qφ ( a , x , t|ỹ ) , the variational lower bound for the proposed RCFSL can be represented as log p ( x , t ) = ∫ a ∫ z ∫ T ∫ µ ∫ Λ ∫ ỹ log ( x , ỹ , a , z , T , µ , Λ ) dadzdTdµΛdỹ ≥ Eqφ ( a , ỹ , z , T , µ , Λ|x , t ) [ log pθ ( a , z , T , µ , Λ , x , t , λ , ỹ ) qφ ( a , z , T , µ , Λ , λ , ỹ|x , t ) ] = E [ log ( pθ ( a , z , T , µ , Λ , ω , λ , x , t , ỹ ) ) ] − E [ qφ ( a|x , t ) ] − E [ qφ ( ỹ|a , x , t ) ] −E [ qφ ( z|a , ỹ , x , t ) ] − E [ qφ ( T |µ , Λ , x ) ] − E [ qφ ( µ , Λ ) ] − E [ qφ ( ω|π ) ] − E [ qφ ( λ|t ) ] Similarly , replacing the KL-divergence with β-Divergence and augmenting the latent variableHu = { a , z , y , T , ω , µ , Λ , λ } , the β-ELBO for unlabeled data in RCFSL is : Uβ = ∫ q ( Hu|x , t ) ( − β + 1 β N∑ i=1 p ( xi , ti|Hu ) β + ∫ p ( x , t|Hu ) 1+βdx ) + DKL [ q ( Hu|x , t ) //p ( Hu ) ] , Practically , Lβ and Uβ are calculated via Monte Carlo sampling . The robustness of our proposed ELBO can be guaranteed leveraging the influence function ( IF ) Futami et al . ( 2018 ) Huber & Ronchetti ( 2011 ) . As IF is widely used to analyze how much contamination affects estimated statistics , it is straightforward to show that given the perturbation on the empirical cumulative distribution caused by outliers , it is straightforward to show that the IF of our posterior distribution is bounded . The objective function for robust cross-modal variational autoencoder is : LRCV AE = Lβ + λ1Uβ , where λ1 represents the weight to control the trade-off between the labeled data and unlabeled data . Robust Feature Generation : The construction of the proposed uncertainty priors and the robust divergence measure in our framework aims at better approximation of the posterior distribution under noisy labels and outliers . This is also related to employing generative adversarial learning of z and x , t defined by the data , the encoder , the prior and decoder . Different from VAEs that assumes parametric data distribution and perform posterior inference , GANs in general utilize implicit data distribution and do not provide meaningful latent representations . By learning both a generator G and a discriminator D , a min-max objective is optimized : min G max D { Ex , t∼pdata ( x , t ) [ ln ( D ( x , t ) ) ] + Ez∼p ( z ) [ ln ( 1−D ( G ( z ) ) ] } ( 8 ) In Kaneko et al . ( 2019 ) , a noise transition model is incorporated to learn a clean label conditional generative distribution . But their model only considered noisy labels without outliers and is limited to supervised learning . Recent work on semi-supervised GAN with k classes Kumar et al . ( 2017 ) Salimans et al . ( 2016 ) modify the discriminator with k + 1 outputs on the discriminator by considering the fake images as the k + 1th class . Hence , the loss for the training can be computed with a supervised loss and an unsupervised loss respectively : L = Lsup + Lunsup = −Ex , t , y∼pd ( x , t , y ) log pf ( y|x , t , y ≤ k ) − Ex , t∼pg ( x , t ) log ( pf ( y = k + 1|x , t ) ) − Ex , t∼pd ( x , t ) log ( 1− pf ( y = k + 1|x , t ) ) , ( 9 ) where Lsup represents the negative log probability of the label given the data is real . Denote D ( x , t ) = 1 − pf ( y = k + 1|x , t ) , the loss for semi-supervised GAN on multimodality data LsGAN can be written as : L = −Ex , t∼pd ( x , t ) logD ( x , t ) − Ez∼noise ( 1 − D ( G ( z ) ) ) − Ex , t , y∼pd ( x , t , y ) log pf ( y|x , t , y ≤ k ) . More recently , the label noise robust GAN ( rGAN ) Kaneko et al . ( 2019 ) has achieved promising performance in classifying images with noisy labels by incorporating a noise transition model to learn a clean label conditional generative distribution under noisy labels . Thus , given the noise samples ( x̃r , ỹr ) ∼ p̃d ( x , ỹ ) , to construct a label-noise robust conditional generator and discriminator , the objective function of the robust semi-supervised GAN in the proposed robust cross-modal semi-supervised few-shot learning is expressed as : LRSGAN = −Ex , t∼pd ( x , t ) log ( 1− C̃ ( ỹ = k+1|x , t ) ) −Ez∼noise ( 1−D ( G ( z ) ) ) −E ( x̃r , t , ỹr ) ∼p̃d ( x , t , ỹ ) log C̃ ( ỹ = ỹr|x , t , ỹ ≤ k ) = −Ex , t∼pd ( x , t ) log ( 1 −Mŷr , ỹr Ĉ ( ŷ = k + 1|x , t ) ) − Ez∼noise ( 1 − D ( G ( z ) ) ) − E ( x̃r , t , ỹr ) ∼p̃d ( x , t , ỹ ) log ∑ ŷr Mŷr , ỹr Ĉ ( ŷ = ŷ r|x , t , ŷ ≤ k ) . Without modeling the data distribution explicitly and representing the latent space in a meaningful manner , it does not provide the functionality to counter the outliers in the generator and discriminator . Motivated by the thought of bridging the gap between robust VAE and robust GAN , we feed the robust variational posterior p ( z|x , t ) instead of the random noise p ( z ) into the label noise robust semi-supervised GAN as the source of randomness as both of the decoder and the generator of RSGAN share the mapping from z to x and t. The full optimization function for the proposed RCFSL framework can be represented as : minGRCFSL maxD Epdata ( x , t ) [ L ( x , t ) ] L ( x , t ) : = lnD ( x , t ) + DKL [ q ( H|x , t , ỹ ) //p ( H ) ] + λ1DKL [ q ( Hu|x , t ) //p ( Hu ) ] + Ez∼q ( H|x , t , ỹ ) [ ln ( 1−D ( GRSGAN ( z ) ) + ( −β+1β ∑N i=1 p ( ỹi|H ; xi , ti ) β + N ∫ p ( ỹ|H ; x , t ) 1+βdỹ ] + λ1Ez∼q ( Hu|x , t ) [ ( − β+1 β ∑N i=1 p ( xi , ti|Hu ) β + λ1N ∫ p ( x , t|Hu ) 1+βdx ) ] , ( 10 ) where λ1 is set to be the ratio of unlabled samples verus labeled samples and the discriminator loss is characterized by E [ lnD ( x , t ) ] = −Ex , t∼pd ( x , t ) log ( 1 − Mŷr , ỹr Ĉ ( ŷ = k + 1|x , t ) ) − E ( x̃r , t , ỹr ) ∼p̃d ( x , t , ỹ ) log ∑ ŷr Mŷr , ỹr Ĉ ( ŷ = ŷ r|x , t , ŷ ≤ k ) . This new architecture is expected to better predict the class labels under the compound noise . We then train RCFSL including the robust heterogenous encoder , the generator and the robust discriminator in an end-to-end manner using adaptive moment estimation ( Adam ) Kingma & Ba ( 2015 ) . Each multimodal embedding prototype pc ( of category c ) is computed by averaging the embeddings of all support samples of class c. Once the robust embedding is obtained , the distance between the embedding of the query qt and the multimodal prototype pc is calculated by p ( y = c|pc ) = exp ( −d ( f ( qt ) , pc ) ) ∑ k exp ( −d ( f ( qt ) , pk ) ) where d refers to Euclidean distance and the query is classified as the class with the minimum distance .

Summary: This paper focuses on semi-supervised few-shot learning with multi-modality data. The authors introduce an uncertainty prior of an infinite Gaussian model, integrating multi-modality information from image and text data into a heterogenous variational autoencoder. Meanwhile, a new variational lower bound is derived for the inference of parameters. In addition, a GAN is developed for the data sparsity in few shot learning. Experimental results demonstrate the effectiveness of the proposed method to some extent.