Title: TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generalized Zero-shot Learning

Vision-Language Models

Contrastive Learning
Abstract: Pre-trained vision-language models (VLMs) such as CLIP have shown excellent performance for zero-shot classification. Based on CLIP, recent methods design various learnable prompts to evaluate the zero-shot generalization capability on a base-to-novel setting. This setting assumes test samples are already divided into either base or novel classes, limiting its application to realistic scenarios. In this paper, we focus on a more challenging and practical setting: generalized zero-shot learning (GZSL), i.e., testing with no information about the base/novel division. To address this challenging zero-shot problem, we introduce two unique designs that enable us to classify an image without the need of knowing whether it comes from seen or unseen classes. Firstly, most existing methods only adopt a single latent space to align visual and linguistic features, which has a limited ability to represent complex visual-linguistic patterns, especially for fine-grained tasks. Instead, we propose a dual-space feature alignment module that effectively augments the latent space with a novel attribute space induced by a well-devised attribute reservoir. In particular, the attribute reservoir consists of a static vocabulary and learnable tokens complementing each other for flexible control over feature granularity. Secondly, finetuning CLIP models (e.g., prompt learning) on seen base classes usually sacrifices the model's original generalization capability on unseen novel classes. To mitigate this issue, we present a new topology-preserving objective that can enforce feature topology structures of the combined base and novel classes to resemble the topology of CLIP. In this manner, our model will inherit the generalization ability of CLIP through maintaining the pairwise class angles in the attribute space. Extensive experiments on twelve object recognition datasets demonstrate that our model, termed Topology-Preserving Reservoir (TPR), outperforms strong baselines including both prompt learning and conventional generative-based zero-shot methods.


Peer reviews:
comments: SUMMARY:
This paper proposes a new task, "generalized zero-shot learning (GZSL)," in which both seen and unseen objects should be recognized for vision-language tasks. It also proposes a new method based on CLIP that uses the loss in the "attribute space" to perform better in both seen and unseen classes. This method is evaluated on various kinds of data sets and evaluated by the harmonic mean of the accuracies of seen and unseen classes.

STRENGTHS:
The proposed approach using the attribute space seems novel enough, and its effectiveness was verified by a detailed comparison of the other methods and the well-designed ablation studies.

WEAKNESSES:
1) It is unclear what is learned in "learnable attribute tokens." It is not so beneficial for unseen classes. It is unclear what information is represented as tokens for seen classes. It may be better to analyze the acquired tokens in more detail.

2) It is difficult to think about the case that we have never seen an object, but we know its attributes quite well. In such a sense, 
I believe this method is more appropriate for few-shot learning.

QUESTIONS:
Strangely, the accuracy increases as the number of learnable tokes increases in Fig 4. (a) AwA2. It would be appreciated if you could provide any insights into this phenomenon.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-01

comments: SUMMARY:
In this paper the author proposed a dual-space feature alignment module to keep the semantic consistency between visual and attribute. In addition, the authors proposed Topology-Preserving Reservoir (TPR) to tackle the issue into the generalized zero shot learning (GZSL) setting, which utilized the Pearson correlation coefficient to define a topology-preserving loss, which effectively prevents overfitting of the seen and unseen classes. Sufficient experiment demonstrate the effectiveness of the proposed method.

STRENGTHS:
(1)The Paper is well-written, meanwhile, the method is intuitive and easy to understand.
(2)The proposed method focused on Generalized Zero-Shot Learning (GZSL) to present Topology-Preserving Reservoir to finetune the pre-trained CLIP for better fit the distribution of seen and unseen classes, which seems reasonable.
(3)Sufficient and significant experiments demonstrate the effectiveness of the proposed method.

WEAKNESSES:
(1)The Dual-Space Feature Alignment proposed by the author, which uses a Cross Attention mechanism for cross-modal alignment, lacks innovation.
(2)The author mentions "attribute reservoir" in the article, but essentially it is just a fully connected layer that generates different feature representations through various loss constraints. Additionally, in Figure 2, the attribute reservoir is shown in two states: frozen and trained. I am unsure about when these two states should transition between each other.
(3)The idea proposed by the author to fine-tune feature distribution using spatial topological structures is intriguing. However, relying solely on the Pearson correlation coefficient to define a topology-preserving loss seems somewhat simplistic.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed approach targets the generalized zero-shot learning (GZSL) problem for the vision language model (VLM). It is observed that a strong VLM model shows promising results for novel class generalization. Fine-tuning these models for seen classes leads to a loss in generalization capability and poor results for unseen classes. Additionally, a single latent space demonstrates limited ability to adapt to complex visual-linguistic patterns in fine-grained datasets. The paper proposes dual-space alignment, augmenting the latent space with static and learnable tokens. To address the generalization problem post fine-tuning, the paper introduces a Topology-Preserving Reservoir (TPR), which helps preserve the model's generalization ability for unseen classes. The authors conducted extensive experiments across several standard ZSL datasets and explored the impact of various components through ablation studies.\n\nSTRENGTHS:\n[1] Generalization of unseen classes in VLM is a critical problem. The strong pretrained model also loses its generalization ability, which the author explores, and the proposed model shows a significant impact.\n\n[2] The idea and intuition behind the static and learnable attribute reservoir are interesting. Additionally, TPR helps improve generalization.\n\n[3] The wide-ranging experiments conducted across various ZSL datasets and the ablation studies are satisfactory.\n\nWEAKNESSES:\n[1] The standard ZSL model assumes that there is a description per class rather than per sample, which is more intuitive since a single description for each class suffices for the model to understand the class, making it cost-efficient. Standard annotation-based attributes often yield better results for ZSL/GZSL settings. For example, [a] demonstrates impressive results for the CUB dataset compared to the proposed complex static, learnable, and description-based model. This issue is particularly observed in fine-grained datasets. Why is this the case?\n\n[2] It is unclear how the base attribute vocabulary is created. At a high level, the author collected a few attributes and obtained LLM embeddings. This description may not be sufficient for reproducibility since the code and data are not provided.\n\n[3] There are multiple variants of TPR (Table-3) in various scenarios where different methods work, making it difficult to apply and choose the best one. What does the author conclude here?\n\n[4] In Table-1 for the SUN dataset, the model shows inferior performance. While we do not expect the model to outperform in all scenarios, a clear description and author observations are required: why is this the case?\n\n[a] Meta-Learned Attribute Self-Interaction Network for Continual and Generalized Zero-Shot Learning, WACV-24\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nAuthors has not discussed the limitation in the paper, while in the Checklist they said \"Yes\" i.e. they had discussed. This is a bad practice, I don't know what to do.\n\nDear AC please look into it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a new study that introduces the Generalized Zero-Shot Learning (GZSL) framework within VLMs, aiming to classify both known and novel classes without class partitioning. Key innovations include a dual-space feature alignment module, enhancing latent representations with an attribute reservoir for nuanced visual-linguistic patterns. Additionally, a topology-preserving objective ensures that model adaptations preserve the semantic structure learned by CLIP, thus maintaining generalization across all classes. Extensive experiments across diverse datasets validate the proposed Topology-Preserving Reservoir (TPR) model, demonstrating superior performance over conventional methods in recognizing both seen and unseen classes, underlining its potential for practical applications in complex visual recognition tasks.\n\nSTRENGTHS:\n1. This paper introcuces a novel research aspect for VLMs: generalized zero-shot learning, which requires the model to identify both seen and unseen concepts at the same time. From my perspective, this proposal could be a great contribution to VLM community.\n2. This paper is well-organized and well-written, which makes it easy to follow. \n3. Extensive experiments，ablation study and visualization results demonstrate the effectiveness and rationality of TPR.\n\nWEAKNESSES:\nNone in particular.\n\nQUESTIONS:\nGZSL is a long standing problem in ML/AI community with many classic solutions, like stacking[1] .etc. Since the authors only compare with VLM-based methods, my concern is about the perfromance of classic methods in VLM-based GZSL tasks, can they achieve surprising performance with well-trained features?\n[1] Chao W L, Changpinyo S, Gong B, et al. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild[C]//Computer Vision–ECCV 2016\n\nLIMITATIONS:\nNone in particular.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}