Closed-form Sample Probing for Learning Generative Models in Zero-shot Learning

Generative model based approaches have led to significant advances in zero-shot learning ( ZSL ) over the past few-years . These approaches typically aim to learn a conditional generator that synthesizes training samples of classes conditioned on class definitions . The final zero-shot learning model is then obtained by training a supervised classification model over the real and/or synthesized training samples of seen and unseen classes , combined . Therefore , naturally , the generative model needs to produce not only relevant samples , but also those that are sufficiently rich for classifier training purposes , which is handled by various heuristics in existing works . In this paper , we introduce a principled approach for training generative models directly for training data generation purposes . Our main observation is that the use of closed-form models opens doors to end-to-end training thanks to differentiability of the solvers . In our approach , at each generative model update step , we fit a task-specific closed-form ZSL model from generated samples , and measure its loss on novel samples all within the compute graph , a procedure that we refer to as sample probing . In this manner , the generator receives feedback directly based on the value of its samples for model training purposes . Our experimental results show that the proposed sample probing approach improves the ZSL results even when integrated into state-of-the-art generative models . 1 INTRODUCTION . Zero-shot Learning ( ZSL ) has recently received a great interest for being one of the promising paradigms towards building very large vocabulary ( visual ) understanding models with limited training data . The problem of ZSL can be summarized as the task of transferring information across classes such that the instances of unseen classes , with no training examples , can be recognized at test time , based on the training samples of seen classes . Similarly , the term Generalized Zero-Shot Learning ( GZSL ) ( Chao et al. , 2016 ; Xian et al. , 2018a ) is used to refer to a practically more valuable variant of ZSL where both seen and unseen classes may appear at test time . GZSL brings in additional challenges since GZSL models need to produce confidence scores that are comparable across all classes . Recent work shows that hallucinating unseen class samples through statistical generative models can be an effective strategy , e.g . ( Mishra et al. , 2017 ; Xian et al. , 2018b ; 2019 ; Sariyildiz & Cinbis , 2019 ; Zhu et al. , 2018 ; Schonfeld et al. , 2019 ; Arora et al. , 2018 ; Narayan et al. , 2020 ) . These approaches rely on generative models conditioned on class embeddings , obtained from auxiliary semantic knowledge , such as visual attributes ( Xian et al. , 2019 ) , class name word embeddings ( Schonfeld et al. , 2019 ) , or textual descriptions ( Zhu et al. , 2018 ) . The resulting synthetic examples , typically in combination with existing real examples , are used for training a supervised classifier . In generative GZSL approaches , the quality of class-conditional samples is crucial for building accurate recognition models . It is not straight-forward to formally define the criteria of good training samples . Arguably , however , samples need to be ( i ) realistic ( e.g . free from unwanted artifacts ) , ( ii ) relevant ( i.e . belong to the desired class distribution ) and ( iii ) informative ( i.e . contain examples defining class boundaries ) to train an accurate classifier . Clearly , a primary factor affecting the quality of generated samples is the loss driving the conditional generative model training process . In this work , we aim to address the problem of training data generating models via an end-to-end mechanism that we call sample probing.1 Our main goal is to directly evaluate the ability of a generative model in synthesizing training examples . To this end , we observe that we can leverage classification models with closed-form solvers to efficiently measure the quality of training samples , in an end-to-end manner . More specifically , we formulate a simple yet powerful meta-learning approach : at each training iteration , ( i ) take a set of samples from the generative model for a randomly selected subset of classes , ( ii ) train a zero-shot probing model using only the synthesized samples , and ( iii ) evaluate the probing model on real samples from the training set . We then use the loss value as an end-to-end training signal for updating the generative model parameters . Since we specifically focus on probing models with exact closed-form solutions , the probing model optimization simplifies into a differentiable linear algebraic expression and takes part as a differentiable unit within the compute graph . A graphical summary of the proposed training scheme is given in Figure 1 . In the rest of the paper , we first provide an overview of related work , and explain our approach in detail . We then present a through experimental evaluation on GZSL benchmarks in Section 4 , where the results show that sample probing yields improvements when introduced into state-of-theart baselines . We conclude with final remarks in Section 5 . 2 RELATED WORK . The generalized zero-shot learning problem has been introduced by Xian et al . ( 2017 ) and Chao et al . ( 2016 ) . The extensive study in Xian et al . ( 2018a ) has shown that the success of methods can greatly vary across zero-shot and generalized zero-shot learning problems . The additional challenge in the generalized case is the need for deciding whether an input test sample belongs to a seen or unseen class . Discriminative training of ZSL models , such as those based on bilinear compatibility functions ( Weston et al. , 2011 ; Frome et al. , 2013 ) , are likely to yield higher confidence scores for seen classes . To alleviate this problem , a few recent works have proposed ways to regularize discriminative models towards producing comparable confidence scores across all classes and avoid over-fitting , e.g . Jiang et al . ( 2019 ) ; Liu et al . ( 2018 ) ; Chou et al . ( 2021b ) . Generative approaches to zero-shot learning naturally address the confidence score calibration problem . However , in general , generative modeling corresponds to a more sophisticated task than learning discriminant functions only ( Vapnik & Vapnik , 1998 ) . In this context , the problem is further complicated by the need of predicting zero-shot class distributions . To tackle this challenging task , a variety of techniques have been proposed ( Mishra et al. , 2017 ; Xian et al. , 2018b ; 2019 ; Sariyildiz & Cinbis , 2019 ; Zhu et al. , 2018 ; Schonfeld et al. , 2019 ; Arora et al. , 2018 ; Elhoseiny & Elfeki , 2019 ; Narayan et al. , 2020 ) by adapting generative models , such as VAEs ( Kingma & Welling , 1Our use of the sample probing term is not closely related to the natural language model analysis technique known as probing ( Belinkov et al. , 2017 ; Peters et al. , 2018 ; Hewitt & Liang , 2019 ) . 2014 ; Rezende et al. , 2014 ) and GANs ( Goodfellow et al. , 2014 ) . To enforce class conditioning , the state-of-the-art approaches use a mixture of heuristics : Xian et al . ( 2018b ) uses the loss of a pretrained classifier over the generated samples during training . Narayan et al . ( 2020 ) additionally uses the loss of a sample-to-attribute regressor , in combination with a feedback mechanism motivated from feedback networks ( Zamir et al. , 2017 ) . Sariyildiz & Cinbis ( 2019 ) uses projection discriminator ( Miyato & Koyama , 2018 ) and gradient matching loss as a gradient based similarity measure for comparing real versus synthetic data distributions . None of these generative ZSL approaches , however , directly measure the value of the generated samples for training classification models . The main difficulty lies in the need for back-propagating over long compute chains , which is both inefficient and prone to gradient vanishing problems . To the best of our knowledge , we introduce the first end-to-end solution to this problem by the idea of using probing models with closed-form solvers to monitor the sample quality for model training purposes . Our approach is effectively a meta-learning ( Snell et al. , 2017 ) scheme . Meta-learning is a prominent idea in few-shot learning , where the goal is learning to build predictive models from a limited number of samples . The main motivation is the idea that general-purpose classification models may behave suboptimally when only few training samples are provided . A variety of meta-learning driven few-shot learning models have been proposed , such as meta-models that transform few samples to classifiers ( Gidaris & Komodakis , 2018 ; Snell et al. , 2017 ) , set-to-set transformations for improving sample representations ( Ye et al. , 2020 ; Bronskill et al. , 2020 ) , fast adaptation networks for few examples ( Finn et al. , 2017 ; Rusu et al. , 2019 ; Nichol et al. , 2018 ) . In contrast to such mainstream learning to classify and learning to adapt approaches , we aim to address the problem of learning to generate training examples for GZSL . There are only few and recent studies that aim to tackle generative ZSL via meta-learning principles . Verma et al . ( 2020 ; 2021 ) embrace the learning-to-adapt framework of MAML ( Finn et al. , 2017 ) , which originally aims to learn the optimal initialization for few-shot adaptation . The MAML steps are incorporated by iteratively applying a single-step update to the generative model using the generative model ( VAE/GAN ) loss terms , and then back-propagating over the re-computed generative loss terms on new samples from a disjoint subset of classes using the single-step updated model . Our approach differs fundamentally , as we propose to use the discriminative guidance of ZSL models fully-trained directly from generated sample batches . In another work , Yu et al . ( 2020 ) proposes an episodic training-like approach with periodically altered training sets and losses during training , to learn non-stochastic mappings between the class-embeddings and class-centers . Chou et al . ( 2021a ) , similarly inspired from meta-learning and mixup regularization ( Zhang et al. , 2018 ) , proposes to train a novel discriminative ZSL model over episodically defined virtual training classes obtained by linearly mixing classes . Neither of these approaches learn sample generating models , therefore , they have no direct relation to our work focusing on the problem of measuring the sample quality for ZSL model training purposes , with end-to-end discriminative guidance . The use of recognition models with closed-form solvers has attracted prior interest in various contexts . Notably , Romera-Paredes & Torr ( 2015 ) proposes the ESZSL model as a simple and effective ZSL formulation . We leverage the closed-form solvability of the ESZSL model as part of our approach . Bertinetto et al . ( 2019 ) utilizes ridge-regression based task-specific few-shot learners within a discriminative meta-learning framework . In a similar fashion , Bhat et al . ( 2020 ) ; Liu et al . ( 2020 ) tackle the problem of video object segmentation ( VOS ) and use ridge regression based task-specific segmentation models within a meta-learning framework . None of these approaches aim to use recognition models with closed form solvers to form guidance for generative model training . Another related research topic is generative few-shot learning ( FSL ) , where the goal is learning to synthesize additional in-class samples from few examples , e.g . Hariharan & Girshick ( 2016 ) ; Wang et al . ( 2018 ) ; Gao et al . ( 2018 ) ; Schwartz et al . ( 2018 ) ; Lazarou et al . ( 2021 ) . Among these , Wang et al . ( 2018 ) is particularly related for following a similar motivation of learning to generate good training examples . This is realized by feeding generated samples to a meta-learning model to obtain a classifier , apply it to real query samples , and use its query loss to update the generative model . Apart from the main difference in the problem definition ( GZSL vs FSL ) , our work differs mainly by fully-training a closed-form solvable ZSL model from scratch at each training step , instead of a few-shot meta-learners that is jointly trained progressively with the generative model . Finally , we note that meta-learning based approaches have also been proposed for a variety of different problems , such as learning-to-optimize ( Li & Malik , 2016 ; Chen et al. , 2017 ) and long-tail classification ( Liu et al. , 2019 ; Ren et al. , 2020 ) .

Summary: The paper outlines a method for improving generative zero-shot learning (ZSL) approaches. Rather than simply training the generative model with the goal of reproducing some real data distribution, the work proposes to train it with an additional goal of synthesizing samples that directly improve the performance of the downstream classification model.  The authors propose to do so through a novel sample-probing loss in which generated samples are used to train a closed-form ZSL model with a differentiable solution. The ZSL model is then evaluated directly on the classification task - and gradients are back-propagated to the generative model's parameters. By applying this approach to an existing generative ZSL model, the authors demonstrate improved sample quality for their synthetic data and increased classification performance across multiple datasets.  In summary, the paper's contributions are: 1) A major contribution: An approach to improving existing generative ZSL methods with a loss that maximizes their performance on the downstream task. 2) A minor contribution: A more detailed and rigorous reporting of the methodologies used to fine-tune model hyperparameters, aimed at increasing reproducibility.