Year: 2021

Abstract:
Generative model based approaches have led to significant advances in zero-shot learning (ZSL) over the past few-years. These approaches typically aim to learn a conditional generator that synthesizes training samples of classes conditioned on class definitions. The final zero-shot learning model is then obtained by training a supervised classification model over the real and/or synthesized training samples of seen and unseen classes, combined. Therefore, naturally, the generative model needs to produce not only relevant samples, but also those that are sufficiently rich for classifier training purposes, which is handled by various heuristics in existing works. In this paper, we introduce a principled approach for training generative models directly for training data generation purposes. Our main observation is that the use of closed-form models opens doors to end-to-end training thanks to differentiability of the solvers. In our approach, at each generative model update step, we fit a task-specific closed-form ZSL model from generated samples, and measure its loss on novel samples all within the compute graph, a procedure that we refer to as sample probing. In this manner, the generator receives feedback directly based on the value of its samples for model training purposes. Our experimental results show that the proposed sample probing approach improves the ZSL results even when integrated into state-of-the-art generative models.

text: Generative model based approaches have led to significant advances in zero-shot learning (ZSL) over the past few-years. These approaches typically aim to learn a conditional generator that synthesizes training samples of classes conditioned on class definitions. The final zero-shot learning model is then obtained by training a supervised classification model over the real and/or synthesized training samples of seen and unseen classes, combined. Therefore, naturally, the generative model needs to produce not only relevant samples, but also those that are sufficiently rich for classifier training purposes, which is handled by various heuristics in existing works. In this paper, we introduce a principled approach for training generative models directly for training data generation purposes. Our main observation is that the use of closed-form models opens doors to end-to-end training thanks to differentiability of the solvers. In our approach, at each generative model update step, we fit a task-specific closed-form ZSL model from generated samples, and measure its loss on novel samples all within the compute graph, a procedure that we refer to as sample probing. In this manner, the generator receives feedback directly based on the value of its samples for model training purposes. Our experimental results show that the proposed sample probing approach improves the ZSL results even when integrated into state-of-the-art generative models.

heading: 1 INTRODUCTION
text: Zero-shot Learning (ZSL) has recently received a great interest for being one of the promising paradigms towards building very large vocabulary (visual) understanding models with limited training data. The problem of ZSL can be summarized as the task of transferring information across classes such that the instances of unseen classes, with no training examples, can be recognized at test time, based on the training samples of seen classes. Similarly, the term Generalized Zero-Shot Learning (GZSL) (Chao et al., 2016; Xian et al., 2018a) is used to refer to a practically more valuable variant of ZSL where both seen and unseen classes may appear at test time. GZSL brings in additional challenges since GZSL models need to produce confidence scores that are comparable across all classes.
Recent work shows that hallucinating unseen class samples through statistical generative models can be an effective strategy, e.g. (Mishra et al., 2017; Xian et al., 2018b; 2019; Sariyildiz & Cinbis, 2019; Zhu et al., 2018; Schonfeld et al., 2019; Arora et al., 2018; Narayan et al., 2020). These approaches rely on generative models conditioned on class embeddings, obtained from auxiliary semantic knowledge, such as visual attributes (Xian et al., 2019), class name word embeddings (Schonfeld et al., 2019), or textual descriptions (Zhu et al., 2018). The resulting synthetic examples, typically in combination with existing real examples, are used for training a supervised classifier.
In generative GZSL approaches, the quality of class-conditional samples is crucial for building accurate recognition models. It is not straight-forward to formally define the criteria of good training samples. Arguably, however, samples need to be (i) realistic (e.g. free from unwanted artifacts), (ii) relevant (i.e. belong to the desired class distribution) and (iii) informative (i.e. contain examples defining class boundaries) to train an accurate classifier. Clearly, a primary factor affecting the quality of generated samples is the loss driving the conditional generative model training process.
In this work, we aim to address the problem of training data generating models via an end-to-end mechanism that we call sample probing.1 Our main goal is to directly evaluate the ability of a generative model in synthesizing training examples. To this end, we observe that we can leverage classification models with closed-form solvers to efficiently measure the quality of training samples, in an end-to-end manner. More specifically, we formulate a simple yet powerful meta-learning approach: at each training iteration, (i) take a set of samples from the generative model for a randomly selected subset of classes, (ii) train a zero-shot probing model using only the synthesized samples, and (iii) evaluate the probing model on real samples from the training set. We then use the loss value as an end-to-end training signal for updating the generative model parameters. Since we specifically focus on probing models with exact closed-form solutions, the probing model optimization simplifies into a differentiable linear algebraic expression and takes part as a differentiable unit within the compute graph. A graphical summary of the proposed training scheme is given in Figure 1.
In the rest of the paper, we first provide an overview of related work, and explain our approach in detail. We then present a through experimental evaluation on GZSL benchmarks in Section 4, where the results show that sample probing yields improvements when introduced into state-of-theart baselines. We conclude with final remarks in Section 5.

heading: 2 RELATED WORK
text: The generalized zero-shot learning problem has been introduced by Xian et al. (2017) and Chao et al. (2016). The extensive study in Xian et al. (2018a) has shown that the success of methods can greatly vary across zero-shot and generalized zero-shot learning problems. The additional challenge in the generalized case is the need for deciding whether an input test sample belongs to a seen or unseen class. Discriminative training of ZSL models, such as those based on bilinear compatibility functions (Weston et al., 2011; Frome et al., 2013), are likely to yield higher confidence scores for seen classes. To alleviate this problem, a few recent works have proposed ways to regularize discriminative models towards producing comparable confidence scores across all classes and avoid over-fitting, e.g. Jiang et al. (2019); Liu et al. (2018); Chou et al. (2021b).
Generative approaches to zero-shot learning naturally address the confidence score calibration problem. However, in general, generative modeling corresponds to a more sophisticated task than learning discriminant functions only (Vapnik & Vapnik, 1998). In this context, the problem is further complicated by the need of predicting zero-shot class distributions. To tackle this challenging task, a variety of techniques have been proposed (Mishra et al., 2017; Xian et al., 2018b; 2019; Sariyildiz & Cinbis, 2019; Zhu et al., 2018; Schonfeld et al., 2019; Arora et al., 2018; Elhoseiny & Elfeki, 2019; Narayan et al., 2020) by adapting generative models, such as VAEs (Kingma & Welling,
1Our use of the sample probing term is not closely related to the natural language model analysis technique known as probing (Belinkov et al., 2017; Peters et al., 2018; Hewitt & Liang, 2019).
2014; Rezende et al., 2014) and GANs (Goodfellow et al., 2014). To enforce class conditioning, the state-of-the-art approaches use a mixture of heuristics: Xian et al. (2018b) uses the loss of a pretrained classifier over the generated samples during training. Narayan et al. (2020) additionally uses the loss of a sample-to-attribute regressor, in combination with a feedback mechanism motivated from feedback networks (Zamir et al., 2017). Sariyildiz & Cinbis (2019) uses projection discriminator (Miyato & Koyama, 2018) and gradient matching loss as a gradient based similarity measure for comparing real versus synthetic data distributions. None of these generative ZSL approaches, however, directly measure the value of the generated samples for training classification models. The main difficulty lies in the need for back-propagating over long compute chains, which is both inefficient and prone to gradient vanishing problems. To the best of our knowledge, we introduce the first end-to-end solution to this problem by the idea of using probing models with closed-form solvers to monitor the sample quality for model training purposes.
Our approach is effectively a meta-learning (Snell et al., 2017) scheme. Meta-learning is a prominent idea in few-shot learning, where the goal is learning to build predictive models from a limited number of samples. The main motivation is the idea that general-purpose classification models may behave suboptimally when only few training samples are provided. A variety of meta-learning driven few-shot learning models have been proposed, such as meta-models that transform few samples to classifiers (Gidaris & Komodakis, 2018; Snell et al., 2017), set-to-set transformations for improving sample representations (Ye et al., 2020; Bronskill et al., 2020), fast adaptation networks for few examples (Finn et al., 2017; Rusu et al., 2019; Nichol et al., 2018). In contrast to such mainstream learning to classify and learning to adapt approaches, we aim to address the problem of learning to generate training examples for GZSL.
There are only few and recent studies that aim to tackle generative ZSL via meta-learning principles. Verma et al. (2020; 2021) embrace the learning-to-adapt framework of MAML (Finn et al., 2017), which originally aims to learn the optimal initialization for few-shot adaptation. The MAML steps are incorporated by iteratively applying a single-step update to the generative model using the generative model (VAE/GAN) loss terms, and then back-propagating over the re-computed generative loss terms on new samples from a disjoint subset of classes using the single-step updated model. Our approach differs fundamentally, as we propose to use the discriminative guidance of ZSL models fully-trained directly from generated sample batches. In another work, Yu et al. (2020) proposes an episodic training-like approach with periodically altered training sets and losses during training, to learn non-stochastic mappings between the class-embeddings and class-centers. Chou et al. (2021a), similarly inspired from meta-learning and mixup regularization (Zhang et al., 2018), proposes to train a novel discriminative ZSL model over episodically defined virtual training classes obtained by linearly mixing classes. Neither of these approaches learn sample generating models, therefore, they have no direct relation to our work focusing on the problem of measuring the sample quality for ZSL model training purposes, with end-to-end discriminative guidance.
The use of recognition models with closed-form solvers has attracted prior interest in various contexts. Notably, Romera-Paredes & Torr (2015) proposes the ESZSL model as a simple and effective ZSL formulation. We leverage the closed-form solvability of the ESZSL model as part of our approach. Bertinetto et al. (2019) utilizes ridge-regression based task-specific few-shot learners within a discriminative meta-learning framework. In a similar fashion, Bhat et al. (2020); Liu et al. (2020) tackle the problem of video object segmentation (VOS) and use ridge regression based task-specific segmentation models within a meta-learning framework. None of these approaches aim to use recognition models with closed form solvers to form guidance for generative model training.
Another related research topic is generative few-shot learning (FSL), where the goal is learning to synthesize additional in-class samples from few examples, e.g. Hariharan & Girshick (2016); Wang et al. (2018); Gao et al. (2018); Schwartz et al. (2018); Lazarou et al. (2021). Among these, Wang et al. (2018) is particularly related for following a similar motivation of learning to generate good training examples. This is realized by feeding generated samples to a meta-learning model to obtain a classifier, apply it to real query samples, and use its query loss to update the generative model. Apart from the main difference in the problem definition (GZSL vs FSL), our work differs mainly by fully-training a closed-form solvable ZSL model from scratch at each training step, instead of a few-shot meta-learners that is jointly trained progressively with the generative model.
Finally, we note that meta-learning based approaches have also been proposed for a variety of different problems, such as learning-to-optimize (Li & Malik, 2016; Chen et al., 2017) and long-tail classification (Liu et al., 2019; Ren et al., 2020).

heading: 3 METHOD
text: In this section, we first formally define the generalized zero-shot learning problem and then define a mathematical framework to summarize the core training dynamics of mainstream generative GZSL approaches. We then express our approach in the context of this mathematical framework.

heading: 3.1 PROBLEM DEFINITION
text: In zero-shot learning, the goal is to learn a classification model that can recognize the test instances of unseen classes Yu, which has no training examples, based on the model learned over the training examples provided for the disjoint set of seen classes Ys. We refer to the class-limited training set by Dtr, which consists of sample and class label pairs (x ∈ X , y ∈ Ys). In our work, we focus on ZSL models where X is the space of image representations extracted using a pre-trained ConvNet. In generalized zero-shot learning, the goal is to build the classification model using the training data set Dtr, such that the model can recognize both seen and unseen class samples at test time. For simplicity, we restrict our discussion to the GZSL problem setting below.
In order to enable the recognition of unseen class instances, it is necessary to have visually-relevant prior knowledge about classes so that classes can visually be related to each other. Such prior knowledge is delivered by the mapping ψ : Y → A, where A expresses the prior knowledge space. In most cases, the prior knowledge is provided as dψ-dimensional vector-space embeddings of classes, obtained using visual attributes, taxonomies, class names combined with word embedding models, the textual descriptions of classes combined with language models, see e.g. Akata et al. (2015). Following the common terminology, we refer to ψ as the class embedding function.
Generative GZSL. In our work, we focus on generative approaches to GZSL. The main goal is to learn a conditional generative model G : A × Z → X , which takes some class embedding a ∈ A and stochasticity-inducing noise input z, and yields a synthetic sample x ∈ X . Once such a generative model is learned, synthetic training examples for all classes can simply be sampled from the G-induced distribution PG, and the final classifier over Y can be obtained using any standard supervised classification model. We refer to the trainable parameters of the model G by θG.
As summarized in Section 2, existing approaches vary greatly in terms of their generative model details. For the purposes of our presentation, most of the GZSL works (if not all) can be summarized as the iterative minimization of some loss function that acts on the outputs of the generative model:
LG = E (x,a)∼Dtr [`G (G(a, zx), a)] (1)
where zx refers to the noise input associated with the training sample (x, y) and `G is the generative model learning loss. (x, a) ∼ Dtr is shorthand notation for (x, ψ(y)) ∼ Dtr At each iteration the goal is to reduce LG approximated over a mini-batch of real samples and their class embeddings.
In our notation, we deliberately keep certain details simple. Noticeably, zx greatly varies across models. For example, in the case of a conditional GAN model, zx ∼ p(z) can simply be a sample from a simple prior distribution p(z), e.g. as in (Xian et al., 2018b; Sariyildiz & Cinbis, 2019; Elhoseiny & Elfeki, 2019). In contrast, in variational training, zx is the latent code sampled from a variational posterior, i.e. zx ∼ q(z|x), where the variational posterior q(z|x) is given by a variational encoder trained jointly with G, e.g. as in (Narayan et al., 2020; Xian et al., 2019).
Another important simplification that we intentionally make in Eq. (1) is the fact that we define the generative model learning loss `G as a function of generator output and class embedding, to emphasize its sample-realisticity and class-relevance estimation goals. However, the exact domain of `G heavily depends on its details, which typically consists of multiple terms and/or (adversarially) trained models. In most of the state-of-the-art approaches, this term is a combination of VAE reconstruction loss (Xian et al., 2019), conditional or unconditional adversarial discriminator network (Xian et al., 2018b; Narayan et al., 2020; Sariyildiz & Cinbis, 2019; Xian et al.,
2019), a sample-to-class classifier for measuring class relevance (Xian et al., 2018b) and sample-toembedding mappings (Narayan et al., 2020). The loss LG may also incorporate additional regularization terms, such as `2 regularization or a gradient penalty term (Sariyildiz & Cinbis, 2019).
In the following, we explain our sample probing approach as a loss term that can, in principle, be used in conjunction with virtually any of the mainstream generative zero-shot learning formulations.

heading: 3.2 SAMPLE PROBING AS GENERATIVE MODEL GUIDANCE
text: The problem that we aim to address is enforcement of G to learn to produce samples maximally beneficial for zero-shot model training purposes. We approach this problem through a learning to generate training samples perspective, where we aim to monitor the quality of the generative model through the synthetic class samples it provides.
In construction of our approach, at each training iteration t, we first randomly select a subset of Y tpb-tr ⊂ Y of seen classes. We refer to these classes as probe-train classes. This subset defines the set of classes that are used for training the iteration-specific probing model over the synthetic samples. More specifically, we first take samples from the model G with the parameters θtG for these classes, and fully train a temporary ZSL model over them using regularized loss minimization:
Γt = arg min Γ E x=G(z,a∼Atpb-tr) [`pb (fpb(x, a), a)] (2)
where fpb is the scoring function of the temporary probing model parameterized by Γ and `pb is its training loss. Atpb-tr is the set of class embeddings of classes in Y t pb-tr. Regularization term over Γ is not shown explicitly for brevity.
The result of Eq. (2), gives us a purely synthetic sample driven model Γt, which we leverage as a way to estimate the success of the generator in synthesizing training examples. For this purpose, we sample real examples from the training set Dtr as validation examples for the probe model. Since we use a (G)ZSL model as the probing model, we can evaluate the model on examples of the classes not used for training the model. Therefore, we sample these probe-validation examples from the remaining classes Ypb-val = Ys \ Y tpb-tr, i.e. the classes with real training examples but unused for probe model training, and use softmax cross-entropy loss over these samples as the probing loss:
Lpb = − E (x,y)∼Dpb-val
[ log p(y|x; Γt) ] (3)
where Dpb-val ⊂ Dtr is the data subset of classes Ypb-val. p(y|x; Γt) is the target class likelihood obtained by applying softmax to fpb(x, ψ(y); Γt) scores over the set of target class set. Here, as target class set, one can use only the classes in Ypb-val (ZSL probing) or those in both Ypb-tr and Ypb-val (GZSL probing). We treat this decision as a hyper-parameter and tune on the validation set.
We use a weighted combination of Lpb and LG, as our final loss function. Therefore, the gradients ∇θGLpb act effectively as the training signal for guiding G towards yielding training examples that results in (G)ZSL probing models with minimal empirical loss.

heading: 3.3 CLOSED-FORM PROBE MODEL
text: A critical part of construction is the need for a probing model where minimization of Eq. (2) is both efficient and differentiable, so that the solver itself can be a part of the compute graph. Probing models that require iterative gradient descent based optimization are unlikely to be suitable as one would need to make a large number of probing model updates for each single G update step, which is both inefficient and prone to gradient vanishing problems. We address this problem through the use of a ZSL model that can be efficiently fit using a closed-form solution.
For this purpose, we opt to use the ESZSL (Romera-Paredes & Torr, 2015) as the main closed-form probe model in our experiments. The model is formalized by the following minimization problem:
min Γ ‖XTΓA− Y ‖2Fro + Ω (Γ) (4)
whereX ∈ Rdx×m andA ∈ Rdψ×k represent the feature and class embeddings corresponding tom input training examples and k classes, Y is the {0, 1}m×k matrix of groundtruth labels, Γ ∈ Rdx×dψ
is the compatibility model matrix, and Ω (Γ) is the regularization function, defined as:
Ω (Γ) = λx‖ΓA‖2Fro + λa‖XTΓ‖2Fro + λn‖Γ‖2Fro (5) where λx, λa, λn correspond to term weights. When the regularization term weights are set such that λn = λxλa, the optimal solution to Eq. (4) can be computed in a closed-form:
Γ? = (XXT + λxI) −1XY AT(AAT + λaI) −1 (6)
This approach was originally proposed as a standalone label-embedding based ZSL model in Romera-Paredes & Torr (2015), with the practical advantage of having an efficient solver. Here, we re-purpose this approach as a probing model in our framework, where the fact that the model is solvable in closed-form is critically important, enabling the idea of end-to-end sample probing. For this purpose, we utilize the solver given by Eq. (6) as the implementation of Eq. (2), which takes a set of synthetic training samples and estimates the corresponding probing model parameters.
Alternative probe models. While we utilize ESZSL in our main experiments, we demonstrate the possibility of using the proposed approach with different probe models using two additional alternatives. The first one, which we call Vis2Sem, is the regression model from visual features to their corresponding class embeddings, defined as follows (using the same notation as in ESZSL):
min Γ ‖ΓTX −AY T‖2Fro + λn‖Γ‖2Fro. (7)
A discussion of the Vis2Sem model can be found in Kodirov et al. (2017). The second one, which we call Sem2Vis, is the class embeddings to visual features regression model of Shigeto et al. (2015):
min Γ ‖X − ΓAY T‖2Fro + λn‖Γ‖2Fro. (8)
Both models, just like ESZSL, are originally defined as non-generative ZSL models, and we repurpose them to define our data-dependent generative model training losses. Unlike the bi-linear compatibility model of ESZSL, however, these models rely on distance based classification, and do not directly yield class probability estimates. While one can still obtain a probability distribution over classes, e.g. by applying softmax to negative `2 distances, for simplicity, we directly use the Sem2Vis and Vis2Sem based distance predictions between the visual features of probe-validation samples and their corresponding class embeddings to compute Lpb as a replacement of Eq. (3).
Summary. A summary of the final approach from a compute graph point of view, is given in Figure 2. The proposed approach aims to realize the goal of learning to generate good training samples by evaluating the synthesis quality through the lens of a closed-form trainable probe model, the prediction loss of which is used as a loss for the G updates. Therefore, G is expected to be progressively guided towards producing realistic, relevant and informative samples, through the reinforcement of which may vary depending on the inherent nature of the chosen probe model.

heading: 4 EXPERIMENTS
text: Datasets. We use the four mainstream GZSL benchmark datasets: Caltech-UCSD-Birds (CUB) (Wah et al., 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), Animals with Attributes 2 (AWA2, more simply AWA) (Xian et al., 2018a) and Oxford Flowers (FLO) (Nilsback & Zisserman, 2008). CUB is a medium scale fine-grained dataset consisting of 200 bird species classes, 11.758 images and 312 dimensional attribute annotations. SUN is a fine-grained dataset with 14.340 images and 102 dimensional attributes. AWA is a coarse-grained dataset with 30.475 images of 50 animal classes with 85-dimensional attributes. FLO is a medium-scale fine-grained dataset with 102 classes and 1024-dimensional attributes. Following the state-of-the-art, we use the class embeddings and the version-2 splits defined by Xian et al. (2018a). We evaluate the results in terms of GZSL-u, GZSL-s and h-score (H) values (Xian et al., 2018a). The h-score, i.e. the harmonic mean of GZSL-u and GZSL-s scores, aims to measure how well a model recognizes seen and unseen classes collectively.
In our experiments, we use the image features extracted from ResNet-101 backbone pretrained on ImageNet 1K. In the experiments based on fine-tuned representations, we use the backbone finetuned with the training images of seen classes, as in Xian et al. (2019); Narayan et al. (2020).
Hyper-parameter tuning policy. In our preliminary studies, we observe that final GZSL performance, especially in terms of h-score, of most models strongly depends on the selection of the hyperparameters. We also observe that there is no widely-accepted policy on how the hyper-parameters of GZSL models shall be tuned. It is a rather common practice in the GZSL literature to either directly report the hyper-parameters used in experiments without an explanation on the tuning strategy or simply refer to tuning on validation set, which we find a vaguely-defined policy as (i) Xian et al. (2018a) defines an unseen-class only validation split, which does not allow monitoring the h-score, and, (ii) it is unclear which metric one should use for GZSL model selection purposes.
Therefore, to obtain comparable results within our experiments, we use the following policy to tune the hyper-parameters of our approach and our baselines: we first leave-out 20% of train class samples as val-seen samples. We periodically train a supervised classifier by taking synthetic samples from the generative model, and evaluate it on the validation set, consisting of the aforementioned val-seen samples plus the val-unseen samples with respect to the benchmark splits. We choose the hyper-parameter combination with the highest h-score on the validation set. We obtain final models by re-training the generative model from scratch on the training and validation examples combined.

heading: 4.1 MAIN RESULTS
text: In this subsection, we discuss our main experimental results. As we observe that the results are heavily influenced from hyper-parameter tuning strategy, our main goal throughout our experiments is the validation of the proposed sample probing idea by integrating it into strong generative GZSL baselines, and then comparing results using the same tuning methodology. Using this principle, we present two main types of analysis: (i) the evaluation of the proposed approach using ESZSL as the probe model in combination with a number of generative GZSL models, and (ii) the evaluation of alternative closed-form probe models within our framework.
Generative GZSL models with sample probing. To evaluate the sample probing approach as a general technique to improve generative model training, we integrate it to four recent generative GZSL approaches: conditional Wasserstein GAN (cWGAN) (Arjovsky et al., 2017; Miyato & Koyama, 2018), LisGAN (Li et al., 2019), TF-VAEGAN (Narayan et al., 2020) and FREE (Chen et al., 2021). We additionally report results for the variant of TF-VAEGAN with the fine-tuned representations (TF-VAEGAN-FT), as it is the only one among them with reported fine-tuning results. For the cWGAN, we follow the implementation details described in Sariyildiz & Cinbis (2019), and tune hyper-parameters using our policy. For LisGAN, TF-VAEGAN and FREE models, we use the official repositories shared by their respective authors. We use the version of TF-VAEGAN without feedback loop (Narayan et al., 2020), for simplicity, as the model yields excellent performance with and without feedback loop. In all models (except cWGAN), we only re-tune the number of training iterations of the original models using our hyper-parameter tuning policy, to make the re-
sults comparable, as it is unclear how the original values were obtained.2 We keep all remaining hyper-parameters unchanged to remain as close as possible to the original implementations.
The results over the four benchmark datasets are presented in Table 1. In terms of the h-scores, we observe improvements in 17 out of 19 cases, at varying degrees (up to 4.6 points ). Only in two cases we observe a slight degradation (maximum of 0.2 points) in performance. Overall, these improvements over already strong and state-of-the-art (or competitive) baselines validate the effectiveness of the proposed sample probing approach, suggesting that it is a valid method towards end-to-end learning of generative GZSL models directly optimized for synthetic train data generation purposes.
Sample probing with alternative closed-form models. We now evaluate our approach with different closed-form probe models, specifically ESZSL, Sem2Vis (Shigeto et al., 2015) and Vis2Sem (Kodirov et al., 2017), as described in Section 3.3. For these experiments, we use the TF-VAEGAN as the base generative model.
The results with four configurations over four benchmark datasets are presented in Table 2. First of all, in terms of h-scores, we observe considerable performance variations across the probe models and datasets: Sem2Vis performs the best on CUB (+0.9 over the baseline), ESZSL provides a clear gain on FLO (+4.6) and a relative improvement on AWA (+0.3), and Vis2Sem improves the most on SUN (+1.8). These results suggest that sample probe alternatives have their own advantages and disadvantages, and their performances can be data dependent. Therefore, in a practical application, probe model options can be incorporated into the model selection process. More in-depth understanding of closed-form model characteristics for sample probing purposes, and the formulation and evaluation of other probe models can be important future work directions. Overall, the fact that we observe equivalent (2) or better (9) h-scores in 11 out of 12 sample probing experiments indicates the versatility of the approach in terms of compatibility with various closed-form probe models.
Comparison to other generative GZSL approaches. Performance comparisons across independent experiment results can be misleading due to differences in formulation-agnostic implementation and model selection details. Nevertheless, an overall comparison can be found in Appendix A.
2We also tune LisGAN for AWA2 as the original paper reports AWA1 results instead.

heading: 4.2 ANALYSIS
text: ZSL vs GZSL loss in sample probing. In Table 3, we present a comparison of our approach, using TF-VAEGAN as the generative model and ESZSL as the probe model, when two different types of losses (zsl-loss and gzsl-loss) are used as Lpb in Eq. (3). They differ from each other in terms of classes among which the real examples of probe-validation classes are classified, during the evaluation of sample probing ZSL model. zsl-loss and gzsl-loss indicate that the examples of probe-validation classes are classified among only probe-validation classes, and both probe-train and probe-validation classes, respectively. We observe that using either one during the evaluation of sample probing ZSL model, brings its own characteristic results (Table 3). On all datasets, using zsl-loss increases the seen accuracy while using gzsl-loss increases the unseen accuracy. We choose among these two options using our same hyper-parameter tuning policy, on the validation set.
Effect of sample probing loss weight. Figure 3 shows the validation and test set h-score values as a function of sample probing loss weight. In the test set results, we observe an overall increasing performance trend with larger loss weights, up to the weight 6, highlighting the contribution of sample probing. The optimal weight with respect to the validation and the test sets, however, differs. This observation is an example for the difficulty of tuning the ZSL model based on validation set. We set the loss weight to 5 following our hyper-parameter tuning policy, which yields almost 0.5 lower than the maximum test-set score observed for this single hyper-parameter.
Quantitative analysis of sample quality. We quantitatively evaluate the sample quality using Fréchet (Wasserstein-2) distance, which is also used in the FID metric for evaluating GAN models. In Table 4, we provide a comparison for TF-VAEGAN and our approach for respective mean per-class Fréchet distances between real and synthetic samples (200 synthetic samples per class) of unseen classes on three datasets. The results show that sample probing helps generator to generate more realistic samples compared to TF-VAEGAN. We further investigate the qualitative analysis of the sample quality via t-SNE visualizations in Appendix B.

heading: 5 CONCLUSIONS
text: We propose a principled GZSL approach, which makes use of closed-form ZSL models in generative model training to provide a sample-driven and end-to-end feedback to the generator. Extensive experiments over four benchmark datasets show that the proposed sample probing scheme consistently improves the GZSL results and the sample quality, can easily be integrated to existing generative GZSL approach and can be utilized with various closed-form probe models.
Reproducibility statement. We discuss the problem of model selection in GZSL, and the difficulty of making fair comparison across GZSL models in Section 4. We believe that these are open problems in GZSL research. We also clearly explain our hyper-parameter tuning policy, which we use in our experiments. We provide our source code implementing the main model, as a supplementary material to provide reproducibility. You can find how we process the data in meta dset.py and util.py, the ESZSL implementation in eszsl.py, final classifier in classifier.py, main training code in train tfvaegan inductive sample probing.py, and the models in model.py. README.txt includes a brief explanation on how to run an experiment. We are committed to releasing a complete repository upon acceptance.

heading: A COMPARISON TO THE STATE-OF-THE-ART
text: In Table 5, we compare our results with the state-of-the-art generative approaches for GZSL. During training, we select our best model based on the validation results and report test results on models that give best validation scores. For consistency and to keep the baseline comparable to our results, we again report our own results for LisGAN, TF-VAEGAN (without feedback loop) and FREE using our hyper-parameter tuning policy, but do acknowledge that the original papers typically report higher results. The upper part of the table contains results with the original image representations, and the lower part contains those based on fine-tuned representations.
From the results without fine-tuning, we observe that the proposed sample probing based generative model yields state-of-the-art h-scores in all CUB, FLO, SUN and AWA datasets. We also observe competitive results in terms of individual unseen and seen class accuracy values. When compared against results using fine-tuned representations, we again observe state-of-the-art h-scores on CUB and FLO datasets, with a close second on SUN. On AWA, we observe that f-VAEGAN achieves the highest results with a significant margin over our TF-VAEGAN based baseline, where the sample probing improves the baseline yet still achieves a score below that of f-VAEGAN. Overall, while it is hardly fair to compare models with significant implementation details, these results suggest the overall competitiveness of the obtained data generating models with sample probing.
B VISUALIZATION OF UNSEEN DATA
To provide additional qualitative insight into the improvements that can be gained using sample probing, we present t-SNE visualizations of synthetic class samples in Figure 4. In the figure, each plot corresponds to an unseen class on the FLO dataset, and the points correspond to the t-SNE embeddings of real samples (× points), generated samples using TF-VAEGAN with sample probing ( points) and those using the baseline TF-VAEGAN model without sample probing (N points).
From the plots, we can observe that the generative model trained with sample probing tends to yield samples much more aligned with the corresponding true class distributions, compared to those of the baseline model. Overall, these plots demonstrate how sample probing can improve the overall sample quality of a generative model, and possibly lead to superior recognition models when the generated samples are used for classifier training.

Review summaries:
This paper aims to address the problem of generalized zero-shot classification based on generative models. The main contribution is that it considers training and evaluating a generative model in synthesizing training examples that are helpful to improve classification performance. To this end, it leverages the zero-shot learning model of ESZSL that can be efficiently fit using a closed-form solution. This then serves as a sample probing mechanism, so that the generator receives training signal directly based on the value of its samples for classifier training and thus enables end-to-end training. The corresponding sample probing loss function is added into the standard generative model training loss. The final training procedure is performed in a way that is similar to meta-training. The approach is tested on standard generalized zero-shot classification setups, including CUB, SUN, AWA2, and FLO, and compared with state-of-the-art results.

The paper outlines a method for improving generative zero-shot learning (ZSL) approaches. Rather than simply training the generative model with the goal of reproducing some real data distribution, the work proposes to train it with an additional goal of synthesizing samples that directly improve the performance of the downstream classification model.  The authors propose to do so through a novel sample-probing loss in which generated samples are used to train a closed-form ZSL model with a differentiable solution. The ZSL model is then evaluated directly on the classification task - and gradients are back-propagated to the generative model's parameters. By applying this approach to an existing generative ZSL model, the authors demonstrate improved sample quality for their synthetic data and increased classification performance across multiple datasets.  In summary, the paper's contributions are: 1) A major contribution: An approach to improving existing generative ZSL methods with a loss that maximizes their performance on the downstream task. 2) A minor contribution: A more detailed and rigorous reporting of the methodologies used to fine-tune model hyperparameters, aimed at increasing reproducibility.

 The paper addresses the task of GZSL – more specifically, they provide a way to improve the quality of the generative samples in generative GZSL. A closed-form probe model is introduced to provide an efficient and differentiable solution in compute graph. In this manner, the generator receives feedback directly based on the value of its samples for model training purposes. It shows the results on two different settings, with fine-tuning features and without fine-tuning the features.  

This paper proposes sample probing, a meta-learning scheme for zero-shot learning (ZSL), to measure the quality of the synthetic samples provided by certain generative models. Specifically, an existing closed-form ZSL solver is plugged into an existing generative ZSL framework. Owing to the differentiability of the solver, the whole pipeline is end-to-end trainable. Experiments were conducted on four standard benchmarks, where we can observe the state-of-the-art performance achieved by the proposed sample probing based approach.