Closed-form Sample Probing for Learning Generative Models in Zero-shot Learning

Generative model based approaches have led to significant advances in zero-shot learning ( ZSL ) over the past few-years . These approaches typically aim to learn a conditional generator that synthesizes training samples of classes conditioned on class definitions . The final zero-shot learning model is then obtained by training a supervised classification model over the real and/or synthesized training samples of seen and unseen classes , combined . Therefore , naturally , the generative model needs to produce not only relevant samples , but also those that are sufficiently rich for classifier training purposes , which is handled by various heuristics in existing works . In this paper , we introduce a principled approach for training generative models directly for training data generation purposes . Our main observation is that the use of closed-form models opens doors to end-to-end training thanks to differentiability of the solvers . In our approach , at each generative model update step , we fit a task-specific closed-form ZSL model from generated samples , and measure its loss on novel samples all within the compute graph , a procedure that we refer to as sample probing . In this manner , the generator receives feedback directly based on the value of its samples for model training purposes . Our experimental results show that the proposed sample probing approach improves the ZSL results even when integrated into state-of-the-art generative models . 1 INTRODUCTION . Zero-shot Learning ( ZSL ) has recently received a great interest for being one of the promising paradigms towards building very large vocabulary ( visual ) understanding models with limited training data . The problem of ZSL can be summarized as the task of transferring information across classes such that the instances of unseen classes , with no training examples , can be recognized at test time , based on the training samples of seen classes . Similarly , the term Generalized Zero-Shot Learning ( GZSL ) ( Chao et al. , 2016 ; Xian et al. , 2018a ) is used to refer to a practically more valuable variant of ZSL where both seen and unseen classes may appear at test time . GZSL brings in additional challenges since GZSL models need to produce confidence scores that are comparable across all classes . Recent work shows that hallucinating unseen class samples through statistical generative models can be an effective strategy , e.g . ( Mishra et al. , 2017 ; Xian et al. , 2018b ; 2019 ; Sariyildiz & Cinbis , 2019 ; Zhu et al. , 2018 ; Schonfeld et al. , 2019 ; Arora et al. , 2018 ; Narayan et al. , 2020 ) . These approaches rely on generative models conditioned on class embeddings , obtained from auxiliary semantic knowledge , such as visual attributes ( Xian et al. , 2019 ) , class name word embeddings ( Schonfeld et al. , 2019 ) , or textual descriptions ( Zhu et al. , 2018 ) . The resulting synthetic examples , typically in combination with existing real examples , are used for training a supervised classifier . In generative GZSL approaches , the quality of class-conditional samples is crucial for building accurate recognition models . It is not straight-forward to formally define the criteria of good training samples . Arguably , however , samples need to be ( i ) realistic ( e.g . free from unwanted artifacts ) , ( ii ) relevant ( i.e . belong to the desired class distribution ) and ( iii ) informative ( i.e . contain examples defining class boundaries ) to train an accurate classifier . Clearly , a primary factor affecting the quality of generated samples is the loss driving the conditional generative model training process . In this work , we aim to address the problem of training data generating models via an end-to-end mechanism that we call sample probing.1 Our main goal is to directly evaluate the ability of a generative model in synthesizing training examples . To this end , we observe that we can leverage classification models with closed-form solvers to efficiently measure the quality of training samples , in an end-to-end manner . More specifically , we formulate a simple yet powerful meta-learning approach : at each training iteration , ( i ) take a set of samples from the generative model for a randomly selected subset of classes , ( ii ) train a zero-shot probing model using only the synthesized samples , and ( iii ) evaluate the probing model on real samples from the training set . We then use the loss value as an end-to-end training signal for updating the generative model parameters . Since we specifically focus on probing models with exact closed-form solutions , the probing model optimization simplifies into a differentiable linear algebraic expression and takes part as a differentiable unit within the compute graph . A graphical summary of the proposed training scheme is given in Figure 1 . In the rest of the paper , we first provide an overview of related work , and explain our approach in detail . We then present a through experimental evaluation on GZSL benchmarks in Section 4 , where the results show that sample probing yields improvements when introduced into state-of-theart baselines . We conclude with final remarks in Section 5 . 2 RELATED WORK . The generalized zero-shot learning problem has been introduced by Xian et al . ( 2017 ) and Chao et al . ( 2016 ) . The extensive study in Xian et al . ( 2018a ) has shown that the success of methods can greatly vary across zero-shot and generalized zero-shot learning problems . The additional challenge in the generalized case is the need for deciding whether an input test sample belongs to a seen or unseen class . Discriminative training of ZSL models , such as those based on bilinear compatibility functions ( Weston et al. , 2011 ; Frome et al. , 2013 ) , are likely to yield higher confidence scores for seen classes . To alleviate this problem , a few recent works have proposed ways to regularize discriminative models towards producing comparable confidence scores across all classes and avoid over-fitting , e.g . Jiang et al . ( 2019 ) ; Liu et al . ( 2018 ) ; Chou et al . ( 2021b ) . Generative approaches to zero-shot learning naturally address the confidence score calibration problem . However , in general , generative modeling corresponds to a more sophisticated task than learning discriminant functions only ( Vapnik & Vapnik , 1998 ) . In this context , the problem is further complicated by the need of predicting zero-shot class distributions . To tackle this challenging task , a variety of techniques have been proposed ( Mishra et al. , 2017 ; Xian et al. , 2018b ; 2019 ; Sariyildiz & Cinbis , 2019 ; Zhu et al. , 2018 ; Schonfeld et al. , 2019 ; Arora et al. , 2018 ; Elhoseiny & Elfeki , 2019 ; Narayan et al. , 2020 ) by adapting generative models , such as VAEs ( Kingma & Welling , 1Our use of the sample probing term is not closely related to the natural language model analysis technique known as probing ( Belinkov et al. , 2017 ; Peters et al. , 2018 ; Hewitt & Liang , 2019 ) . 2014 ; Rezende et al. , 2014 ) and GANs ( Goodfellow et al. , 2014 ) . To enforce class conditioning , the state-of-the-art approaches use a mixture of heuristics : Xian et al . ( 2018b ) uses the loss of a pretrained classifier over the generated samples during training . Narayan et al . ( 2020 ) additionally uses the loss of a sample-to-attribute regressor , in combination with a feedback mechanism motivated from feedback networks ( Zamir et al. , 2017 ) . Sariyildiz & Cinbis ( 2019 ) uses projection discriminator ( Miyato & Koyama , 2018 ) and gradient matching loss as a gradient based similarity measure for comparing real versus synthetic data distributions . None of these generative ZSL approaches , however , directly measure the value of the generated samples for training classification models . The main difficulty lies in the need for back-propagating over long compute chains , which is both inefficient and prone to gradient vanishing problems . To the best of our knowledge , we introduce the first end-to-end solution to this problem by the idea of using probing models with closed-form solvers to monitor the sample quality for model training purposes . Our approach is effectively a meta-learning ( Snell et al. , 2017 ) scheme . Meta-learning is a prominent idea in few-shot learning , where the goal is learning to build predictive models from a limited number of samples . The main motivation is the idea that general-purpose classification models may behave suboptimally when only few training samples are provided . A variety of meta-learning driven few-shot learning models have been proposed , such as meta-models that transform few samples to classifiers ( Gidaris & Komodakis , 2018 ; Snell et al. , 2017 ) , set-to-set transformations for improving sample representations ( Ye et al. , 2020 ; Bronskill et al. , 2020 ) , fast adaptation networks for few examples ( Finn et al. , 2017 ; Rusu et al. , 2019 ; Nichol et al. , 2018 ) . In contrast to such mainstream learning to classify and learning to adapt approaches , we aim to address the problem of learning to generate training examples for GZSL . There are only few and recent studies that aim to tackle generative ZSL via meta-learning principles . Verma et al . ( 2020 ; 2021 ) embrace the learning-to-adapt framework of MAML ( Finn et al. , 2017 ) , which originally aims to learn the optimal initialization for few-shot adaptation . The MAML steps are incorporated by iteratively applying a single-step update to the generative model using the generative model ( VAE/GAN ) loss terms , and then back-propagating over the re-computed generative loss terms on new samples from a disjoint subset of classes using the single-step updated model . Our approach differs fundamentally , as we propose to use the discriminative guidance of ZSL models fully-trained directly from generated sample batches . In another work , Yu et al . ( 2020 ) proposes an episodic training-like approach with periodically altered training sets and losses during training , to learn non-stochastic mappings between the class-embeddings and class-centers . Chou et al . ( 2021a ) , similarly inspired from meta-learning and mixup regularization ( Zhang et al. , 2018 ) , proposes to train a novel discriminative ZSL model over episodically defined virtual training classes obtained by linearly mixing classes . Neither of these approaches learn sample generating models , therefore , they have no direct relation to our work focusing on the problem of measuring the sample quality for ZSL model training purposes , with end-to-end discriminative guidance . The use of recognition models with closed-form solvers has attracted prior interest in various contexts . Notably , Romera-Paredes & Torr ( 2015 ) proposes the ESZSL model as a simple and effective ZSL formulation . We leverage the closed-form solvability of the ESZSL model as part of our approach . Bertinetto et al . ( 2019 ) utilizes ridge-regression based task-specific few-shot learners within a discriminative meta-learning framework . In a similar fashion , Bhat et al . ( 2020 ) ; Liu et al . ( 2020 ) tackle the problem of video object segmentation ( VOS ) and use ridge regression based task-specific segmentation models within a meta-learning framework . None of these approaches aim to use recognition models with closed form solvers to form guidance for generative model training . Another related research topic is generative few-shot learning ( FSL ) , where the goal is learning to synthesize additional in-class samples from few examples , e.g . Hariharan & Girshick ( 2016 ) ; Wang et al . ( 2018 ) ; Gao et al . ( 2018 ) ; Schwartz et al . ( 2018 ) ; Lazarou et al . ( 2021 ) . Among these , Wang et al . ( 2018 ) is particularly related for following a similar motivation of learning to generate good training examples . This is realized by feeding generated samples to a meta-learning model to obtain a classifier , apply it to real query samples , and use its query loss to update the generative model . Apart from the main difference in the problem definition ( GZSL vs FSL ) , our work differs mainly by fully-training a closed-form solvable ZSL model from scratch at each training step , instead of a few-shot meta-learners that is jointly trained progressively with the generative model . Finally , we note that meta-learning based approaches have also been proposed for a variety of different problems , such as learning-to-optimize ( Li & Malik , 2016 ; Chen et al. , 2017 ) and long-tail classification ( Liu et al. , 2019 ; Ren et al. , 2020 ) .

Summary: This paper proposes sample probing, a meta-learning scheme for zero-shot learning (ZSL), to measure the quality of the synthetic samples provided by certain generative models. Specifically, an existing closed-form ZSL solver is plugged into an existing generative ZSL framework. Owing to the differentiability of the solver, the whole pipeline is end-to-end trainable. Experiments were conducted on four standard benchmarks, where we can observe the state-of-the-art performance achieved by the proposed sample probing based approach.