Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The paper addresses the task of GZSL – more specifically, they provide a way to improve the quality of the generative samples in generative GZSL. A closed-form probe model is introduced to provide an efficient and differentiable solution in compute graph. In this manner, the generator receives feedback directly based on the value of its samples for model training purposes. It shows the results on two different settings, with fine-tuning features and without fine-tuning the features.


main_review: Strong Points:  1- The presentation is clear, and the research problem is well motivated. 
2- The proposed method is evaluated on four relatively comprehensive benchmark datasets.


Weaknesses: 1- The paper could not highlight its novelty well. The idea to improve model generalization ability with cross-validation is not new. The proposed method seems to be an integration of generative models and the existing closed-form solution.  The closed-form probe model is borrowed from [a],[b],[c].

2- To prove the better efficacy of the proposed model, it should be trained using a few examples (5/10 samples) per seen class.

3- The experiments for a large dataset (ImageNet) should be included for better evaluation.


[a]- Meta-Learning for Generalized Zero-Shot Learning, AAAI 2020.
[b]- Episode-Based Prototype Generating Network for Zero-Shot Learning, CVPR 2020.
[c]- Towards Zero-Shot Learning With Fewer Seen Class Examples, WACV 2021.

summary_of_the_review: 
Due to lack of novelty and the performance of the proposed model is marginally above the existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the problem of generalized zero-shot classification based on generative models. The main contribution is that it considers training and evaluating a generative model in synthesizing training examples that are helpful to improve classification performance. To this end, it leverages the zero-shot learning model of ESZSL that can be efficiently fit using a closed-form solution. This then serves as a sample probing mechanism, so that the generator receives training signal directly based on the value of its samples for classifier training and thus enables end-to-end training. The corresponding sample probing loss function is added into the standard generative model training loss. The final training procedure is performed in a way that is similar to meta-training. The approach is tested on standard generalized zero-shot classification setups, including CUB, SUN, AWA2, and FLO, and compared with state-of-the-art results.
main_review: Paper Strengths：

The authors tackle an important and challenging problem of generative modeling based generalized zero-shot learning. The proposed approach is simple. Experimental evaluations demonstrate the effect by introducing the sample probing method and end-to-end training.

Paper Weaknesses：

1) In the related work section, the authors discussed meta-learning and few-shot classification. However, there was a lack of discussion on using generative models to perform data augmentation for meta-learning and few-shot classification, such as [Low-shot visual recognition by shrinking and hallucinating features, ICCV, 2017] [MetaGAN: An adversarial approach to few-shot learning, NeurIPS, 2018] [Low-shot learning from imaginary data, CVPR, 2018] [Delta-encoder: an effective sample synthesis method for few-shot object recognition, NeurIPS, 2018] [Image deformation meta-networks for one-shot learning, CVPR, 2019]. While zero-shot and few-shot learning are different, they are related. In particular, I feel that the generative model framework in this paper is similar to [Low-shot learning from imaginary data, CVPR, 2018], where the classification loss is used to train the generator in an end-to-end fashion and the generator is integrated into the meta-training framework. An in-depth discussion on this is needed.

2) The proposed approach relies on a closed-form zero-shot model, which is also the main technical contribution of this paper. For this purpose, the ESZSL model [Romera-Paredes & Torr, 2015] is used. However, it seems that such requirement makes the method restrictive to this zero-shot model, and thus is not general and cannot be combined with more state-of-the-art zero-shot models. Even for ESZSL [Romera-Paredes & Torr, 2015], its closed-form solution only applies for some particular parameter setting.

3) From the results, like in Table 1 and Table 3, the improvements of the proposed approach seem quite marginal, especially when the features are fine-tuned. Also, the proposed approach is not consistently better than existing methods. For example, in Table 3, for the fine-tuning setting, on 2 out of 4 benchmarks, the proposed approach is worse than the state of the art.

4) It would be interesting to see the ablation study regarding the different loss components.

5) Somehow it is a little vague – is the paper synthesizing raw images or feature vectors in the pre-trained feature space? This is only explicitly mentioned until it talks about the implementation details. It would be better to make this explicit starting in the introduction.

6) Following the previous comment, it would be interesting to visualize the synthesized features, like using t-SNE visualizations, and analyze why they are helpful.

7) There are a bunch of hyper-parameters involved in the proposed approach. The authors also mentioned the difficulty in consistently setting up these hyper-parameters. Moreover, how is the hyper-parameter sensitivity?

8) There are several grammar issues and typos in the paper. Please proofread.
summary_of_the_review: The paper introduces sample probing to train a generative model that synthesizes data in feature space for generalized zero-shot learning. The proposed approach is restrictive to certain zero-shot learning models. Some ablation studies and analysis are missing.

=================

post rebuttal:

I thank the authors for the extensive experiments and clarification made in the rebuttal. The rebuttal has addressed most of my concerns, especially showing the generalizability of the proposed approach (different types of generative models and different types of closed-form ZSL models), so I raised my score. However, I am still a little concerned about the novelty of the proposed approach and its marginal improvements.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a generalized zero-shot learning approach by providing feedback for generator with the evaluation of real samples of unseen classes.
main_review: Strengths: 

1- The paper is well written.

2- This approach tries to generate realistic, relevant, and informative features train an accurate classifier using a sample probing mechanism.

3- It also uses feature fine-tune to improve the performance further.

Weaknesses:

1.  The novelty of the proposed approach is limited. The main contribution of this paper is the sample probing technique to generate features. However, prior works ([1],[2],[3]) have proposed a very similar approach. Therefore, given [1],[2],[3] methods, the contribution of this paper is not novel.

2. Considering prior works ([3], and other results from Table 1 of [3]), the accuracy improvements are either not significant or are lower.


[1] Episode-Based Prototype Generating Network for Zero-Shot Learning

[2] Meta-Learning for Generalized Zero-Shot Learning

[3] Meta-Learned Attribute Self-Gating for Continual Generalized Zero-Shot Learning
summary_of_the_review: Considering the weaknesses of the paper, I would give this paper marginally below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper outlines a method for improving generative zero-shot learning (ZSL) approaches. Rather than simply training the generative model with the goal of reproducing some real data distribution, the work proposes to train it with an additional goal of synthesizing samples that directly improve the performance of the downstream classification model.

The authors propose to do so through a novel sample-probing loss in which generated samples are used to train a closed-form ZSL model with a differentiable solution. The ZSL model is then evaluated directly on the classification task - and gradients are back-propagated to the generative model's parameters. By applying this approach to an existing generative ZSL model, the authors demonstrate improved sample quality for their synthetic data and increased classification performance across multiple datasets.

In summary, the paper's contributions are:
1) A major contribution: An approach to improving existing generative ZSL methods with a loss that maximizes their performance on the downstream task.
2) A minor contribution: A more detailed and rigorous reporting of the methodologies used to fine-tune model hyperparameters, aimed at increasing reproducibility.
main_review: Paper strengths:

1) The paper is well written and easy to follow and understand. 
2) The motivations for the work are clear, as is the method itself.
3) Providing greater detail on hyperparameter choices is great, and as the authors demonstrate - it is also crucial.
4) The proposed method is also general in the sense that it can be readily applied to future work. In my opinion, this is a major selling point of the work.
5) Results appear to indicate an improvement over the state of the art. 

Note, however, see point (1) in the weaknesses section in regards to points 4 and 5.

Paper weaknesses:

While the method is promising and the suggested approach makes intuitive sense, I feel like the experimental results do not currently support it well enough.

In particular (listed in order of importance for my evaluation):

1) 

As the author's note in their closing remarks: "Our method works in an end-to-end manner and it can
be easily integrated into any mainstream generative zero-shot learning framework.". This is, in my opinion, one of the most significant selling points of the paper. Alas, it is not investigated. The method is applied only to a single baseline model*, where it shows improved performance only when the baseline is trained with a different set of hyper-parameters than in it's original implementation.

*There is an additional experiment on a single dataset (out of 4) with a different, very basic baseline.

1.1) What were your results when you used the same hyper-parameters as the baseline model? Why do you think your model outperforms the baseline with a specific number of iterations, but underperforms with another? Perhaps your model is simply more efficient and converges faster, but introduces other problems which mitigate the advantage in the long-run?

1.2) If things are highly hyper-parameter sensitive, why did you optimize only their number of iterations? What about the other parameters?

1.3) Can your method be applied to other generative ZSL models? I would have more confidence in the general applicability of the method if it demonstrated improvements when integrated into multiple existing works.

In this context, Table 3 (where most of the results reside) is largely irrelevant. Most of these numbers are simply an indication that TF-VAEGAN is better than its competition. (I am not advocating the table's removal - but applying your method to the models listed there would be a considerable improvement).

2) 

Do you have any intuition as to why the choice of ZSL / GZSL in your loss (i.e. table 4) is so crucial that a wrong choice may make your model perform equally to, or worse than the baseline on some datasets? Is this merely a function of how well the original model performs on seen vs unseen classes? (this is something that could be seen if we had more adapted baselines to compare with!)

3) 

How much of an effect does your method have on training times? Does the closed-form solution of EZSL have a noticeable impact on the time required per training iteration?

4) 

A natural alternative to your approach may be the use of meta-learned ZSL model in place of the closed-form solution. This would allow updating the model with a single training iteration which may produce sufficiently strong gradients for your generator. My knowledge of recent works in the field is limited, are you aware of any works doing something similar? If so, comparisons to them could strengthen your work.

5) 

Given the aim of increasing reproducibility by reporting hyper-parameter tuning methodologies, I would add answers to the following questions (at the very least to the released code):

5.1) Are you using the same 20% validation split for all experiments on a given dataset? If so, can this split be released?

5.2) Did you use a single split, or cross-validate?

6) 

Sample quality metrics - Fréchet distance is typically sensitive to the number of generated samples used in the comparison. These should be reported to facilitate future comparisons.

summary_of_the_review: Overall, I enjoyed the paper and would have liked to recommend acceptance. The approach makes intuitive sense, and can no doubt be extended with multiple future works, offering the community a parallel line of investigation into improving GZSL results.

However, I think the flaws outlined in point (1) of the weakness section are fundamental enough that I am worried that any future works which try to build on this one might simply be wasting their time. I do not believe this to be the case, but I would like to see additional experimental results that would convince me otherwise.


*I have marked my confidence as 3 due to limited familiarity with related work. As an extension to that, I may have missed prior art which already suggested similar ideas.

******************************
Post rebuttal update:

The authors conducted an extensive set of experiments and addressed my primary concerns (the method's generalizability to additional baselines). I am still concerned about some aspects of the evaluation (considerably worse results for all baselines compared to their originally reported values). However, since the paper suggests a method for improving other models, the relative improvements are what matters most. As such, I am willing to accept the current demonstration of a (fairly) consistent improvement when using the same hyper-parameter selection approach across the board. 

Even if accepted, I urge the authors to better highlight and explain the difference between their experimental results and the original baseline values.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes sample probing, a meta-learning scheme for zero-shot learning (ZSL), to measure the quality of the synthetic samples provided by certain generative models. Specifically, an existing closed-form ZSL solver is plugged into an existing generative ZSL framework. Owing to the differentiability of the solver, the whole pipeline is end-to-end trainable. Experiments were conducted on four standard benchmarks, where we can observe the state-of-the-art performance achieved by the proposed sample probing based approach.
main_review: This work attempts to address a major concern in current generative ZSL models, i.e., the quality of the synthesized training examples used to train the final GZSL classifier, as the final GZSL performance highly depends on those generated samples. By measuring the quality of those samples during the training process, more informative samples can be obtained, leading to improved performance. Overall, the paper is well-written and easy to follow, with adequate technical details for re-implementation. Below please find the detailed suggestions and questions:

1) In the last paragraph on Page 1, the authors claimed that samples need to be realistic, relevant and informative. However, in the method section, there are no detailed discussions regarding how the proposed solution endows the synthetic samples with these three properties. More clarifications on this point need to be added.

2) Another concern is the performance gain over TF-VAEGAN. From the tables, we can only observe less than 1% increase w.r.t. the harmonic mean in most cases on the benchmarks. One could have expected to see more gains as measuring the sample quality provides important additional information for the generative model.

3) Does the overall framework highly depend on the solvers it adopts? Is it possible to adopt other solvers like the one in SAE [R1]? If employing this solver, can we obtain even more improved GZSL performance as SAE shows the advantage over ESZSL?

4) From Fig. 3, it is interesting to see that the validation set and the test one exhibits the almost opposite trend w.r.t. the harmonic mean. Does this indicate that the hyper-parameter tuning policy is not suitable for the GZSL task?

5) Table 2 only depicts the results on the CUB dataset - how about the performance on the other three datasets?

[R1] Kodirov et al., Semantic Autoencoder for Zero-Shot Learning, CVPR 2017.
summary_of_the_review: This paper addresses an important issue existing in current generative ZSL models. The paper is well-motivated and the solution seems to be effective. The major concern lies in the performance gain and some details regarding the method need further clarifications.
review_title: null

Meta-review: The paper proposes to improve (generalized) zero-shot learning, by training a generator jointly with the classification task, such that it generates samples that reduce the classification loss.  To achieve this, they use a zero shot model that has a (differentiable) closed form solution (ESZSL), so the full model can be optimized end-to-end. The approach is evaluated on the standard benchmarks of GZSL. 

Reviewers had some concerns regarding novelty compared with previous work and quality of experiments and evaluations. The authors answered most of these concerns in their rebuttal including discussion with previous work and additional evaluations.  As a result, the paper would be interesting for the ICLR audience.
Decision: Accept (Poster)