Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a dual progressive prototype network (DPPN) for generalized zero-shot learning.  The attribute prototypes are learned to yield visual representations with attribute localizations, while the resulting features can be used to improve the category prototypes. The two mechanisms are coupled to progressively carry out and collaboratively achieve good GZSL performance. 
main_review: The main idea of the proposed method is reasonable and partly motivated by the related work, APN [47]. The proposed DPPN comprises two main modules. The first module (Progressive Attribute Localization, PAL) includes progressively trainable attribute prototypes, which are used to yield the visual representation for inference. The second is the Progressive Category Classification (PCC) module, which can be used to enhance the representation of the category prototypes. My detailed comments are as follows.

• In comparison with APN [47], the proposed method enriches the technique with the PCC module and the generalization of the progressive mechanism of K iterations. From the experimental results and the brief discussion in line 255, the classification performance would degrade for K>3. Could the authors provide more comprehensive explanations to this phenomenon?

• Suppose DPPN is trained with K=3. Then, (from line 156) it is expected to use f^3(X) as the visual representation in inference.  Does this strategy always give better classification performance than simply using f^1(X) as the representation in inference since the network parameters have already been updated in the “progressive” training process of PAL and PCC?

• Some of the related work is not included. For example, the following paper from ICLR 2021, “Adaptive and Generative Zero-Shot Learning,” which should be included in the experiment comparisons.
  
• In line 154, it says that the semantic projection function g is implemented by FC for “redundancy reduction.” What are the justifications behind this statement? In particular, g is a mapping from R^{N_a} to R^{N_v}, where N_v > N_a and N_v = D \times N_a. 

• Minor mistakes: line 148, the subscript should start from 1 (not 0). In Table 3, the results for the SUN dataset by TF-VAEGAN [34] are incorrect (those are based on transductive).

limitations_and_societal_impact: In the revision, the authors are suggested to concisely describe the limitations and potential negative social impacts, which may arise from the particular work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a dual prototype network for generalized zero-shot learning (GZSL). The basic idea is to extend the existing framework, i.e., APN, to update the attribute localization function progressively. Beside, an additional progressive category classification (PCC) module is developed to achieve more discriminative classification results by repelling visual representations of different categories. Experimental results on four popular benchmarks demonstrate the effectiveness of the proposed modules.
main_review: Overall, the idea/contribution of this paper is a bit incremental as it originates from a previous work that was the first to propose the concept of attribute prototypes. However, the extension to/the difference from [47] has been clearly stated and adequately justified, which can be found in the extensive ablation study. The presentation is also very clear and easy to follow.

Some detailed comments are as follows:

1. L154: g(.) is a semantic projection function for redundancy reduction. Does this function influence the performance significantly? What if we discard it? How will the final performance change?

2. It seems that the proposed method is not specifically designed for GZSL and can also be used for the conventional zero-shot learning setting (CZSL). Have you tried to evaluate the method in terms of CZSL?

3. L156-157: only the final K-th representation is used for inference. Have you tried to use the summation or concatenation of all the K representations as the final one?

4. Since the PAL module learns/updates attribute prototypes based on each image, will the order of the images fed into the network affect the final performance? How will it affect or why won't it affect?

5. The performance on aPY is exceptionally higher than the current state of the art. How to explain this?

Typo:
L148: $p_0^{k}$ should be $p_1^{k}$.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a dual progressive prototype network (DPPN) for the generalized zero shot learning task, and they argue that the network can produce the attribute-based visual representation with category-specific features. The DPPN has two different modules. The Progressive Attribute Localization module was trained iteratively, and its final K-th output is mainly applied as adjusted visual representation. The Progressive Category Classification module is used to enhance the feature maps through contrastive learning.

main_review: The method is novel compared to previous works, and achieves sufficient improvements on four classification datasets and SOTA on three of four datasets.  The paper is clearly structured and well written in language. The experiments and ablation studies are clear, but some of them can be more complete and comparable.

1. Compared to the most related APN method, the novelty of DPPN is located in the iterative learning (dynamic update for inference) of attribute prototypes, regarding the main difference between Eq.(2) and Eq.(7), also between Eq.(3) and Eq.(11).
2. The descriptions of the update process of the L_sa and L_cl loss in L155 and L182 are not clear enough, how to integrate the loss of k-th iteration from K, since the progressive processing is claimed as a major contribution. 
3. The discussion in Sec.3.5 is not sufficient, since attribute prototypes in PAL are also defined as shared prototypes, though adjusted by K times iterations for specific input images. Or explain more about the contribution of â€œdynamically adjust attribute prototypes according to different imagesâ€. The training process in DPPN is declared as more compact than in APN, but PAL and PCC require multi-stage training (mentioned in L182). The end-to-end training of DPPN (in L60) is not consistent with the two-stage training (L213).
4. Only the lambda settings of the CUB and APY datasets are discussed between L278-L282. Whether the default lambda values of AWA2 and SUN are set as 0.1, which was mentioned in L216 before, but 0.1 is excluded in Fig.5(a)(b). Or why are the other two datasets excluded from this discussion? The same question for the N_v discussion.
5. The abbreviation definition of v2s is not mentioned in the text. There is less discussion (only ResNet 101 was mentioned in Sec.4.1 ) of the backbone in the CNN encoder. How about applying other backbones?
6. All four datasets are from the classification task, how about an experiment on a dense prediction task, such as semantic segmentation or instance segmentation. 
7. The comparison of generative methods is insufficient, since the improvement on SUN is not exactly the same as the other three datasets in Table 3. It would be better to test how the features from DPPN can be applied in a generative method, and what results. For example, the setting of APN in VAEGAN or other GAN-based methods. The comparison can be more comprehensive, if the evaluation of DPPN in Zero-shot Learning Metrics is conducted.
8. Adam as the optimizer mentioned in L214 should be cited as well.
9. Visualization of visual features shown in Fig.1 of Supplementary has only unseen categories, what about the seen categories? How about the visualization comparison with previous works.

- 
We have carefully read the rebuttal by the authors and the reviews from the other reviewers.

In general, the author made clear responses to the unclear issues of the paper. In the discussion on the two hyperparameters (lambda and N_v), the test results of the other two benchmarks have been added, making the discussion relatively more comprehensive. However, two questions are unclear. The first one is about the training process, where the whole ResNet101 is retained between multiple stages, and how it will affect the batch size regarding various K settings in the same machine. Another problem is that the discussion about different backbones is insufficient. The results of adding other recent backbones are expected instead of a ResNet50.

Overall, we will thus keep our initial rating.


limitations_and_societal_impact: Limitation is not included, and we suggest the authors can address the limitation and potential improvement of this work, or some future work for the next step.
The broader impact was mentioned in the supplementary.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a Dual Progressive Prototype Network,  which constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. It aims to search attribute-related local areas, and updates attribute prototypes to explore accurate attribute-region correspondence.
main_review: The method does appear to bring some significant novelty with respect to the literature.
The classification experiments prove the efficiency of the proposed method.
The ablation study is sufficient.
The code is provided.
limitations_and_societal_impact: Evaluation:
Fig.2 is not quite clarifying for the method put forward in the paper.
How to set the hyper-parameters? What are the values of the parameters in section 4.3?
In Fig. 4,   the proposed method obtained the best results when k was set as 3 in some datasets. Is there any analysis of it? Why not the results get better with K increasing?
Are the K in PAC and PCC the same?
How do you get the comparison results in Table 3? Did you run the code by yourself or just record them from the papers? 

Presentation:
The presentation of the proposed method requires significant improvements. Also, very long sentences should be avoided.
The description of the method is incomplete and hard to understand. If the available space is not sufficient for a complete presentation of the model, it is better to move some existing subsections to a supplementary material (e.g. ablation study part).
Also, the proposed additions to existing APN should be better motivated (what problem they solve? what is the insight?) and clearly described. 

Minor corrections (non-exhaustive list):
Line 42 “dynamically adjust” should be “dynamically adjusts”
Line85 “complementary” should be “complimentary”
review_title: null

Meta-review: This paper has four reviewers: two for borderline accept, two for borderline reject. Despite mixed scores, the reviewers have very thorough discussion and recommendation to further improve the papers, particularly, including,  futher clarifying the training process, experimental settings, and experimental analysis.  Some more ablation experiments are all essential to validate the contributions. 

The authours make very hard jobs and try to clarify the comments from reviewers. This is much appreciated. Unfortunately, there are still several things unclear, and the reviewers think they are very important for the acceptance. Specifically, (1) The traing stage is unclear. As ResNet-101 is retained between multiple stages; how it will affect the batch size regarding various K settings in the same machine. Additionally, note that ResNet101 is pre-trained on ImageNet, which may have overlapped images with the benchmark datasets, and potentially an issue of breaking the ZSL setting. I personally guess this is one reason why the improvement on aPY is exceptionally higher as commented by the reviewer; (2) Discussion about different backbones is insufficient; adding more recent backbones are expected. This can better validate the contributions claimed in this paper. Due to these reasons, we would expect the authours to integrate these comments, and further improve this paper for the future top  conference.
Decision: Accept (Poster)