TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning
Authors: Hui Chen
Yanbin Liu
Yongqiang Ma
Nanning Zheng
Xin Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Generalized Zero-shot Learning
Vision-Language Models
Contrastive Learning
Primary area: machine_vision
TL;DR: a model incorporating dual-space alignment and topology-preserving strategies for GZSL

Abstract: Pre-trained vision-language models (VLMs) such as CLIP have shown excellent performance for zero-shot classification. Based on CLIP, recent methods design various learnable prompts to evaluate the zero-shot generalization capability on a base-to-novel setting. This setting assumes test samples are already divided into either base or novel classes, limiting its application to realistic scenarios. In this paper, we focus on a more challenging and practical setting: generalized zero-shot learning (GZSL), i.e., testing with no information about the base/novel division. To address this challenging zero-shot problem, we introduce two unique designs that enable us to classify an image without the need of knowing whether it comes from seen or unseen classes. Firstly, most existing methods only adopt a single latent space to align visual and linguistic features, which has a limited ability to represent complex visual-linguistic patterns, especially for fine-grained tasks. Instead, we propose a dual-space feature alignment module that effectively augments the latent space with a novel attribute space induced by a well-devised attribute reservoir. In particular, the attribute reservoir consists of a static vocabulary and learnable tokens complementing each other for flexible control over feature granularity. Secondly, finetuning CLIP models (e.g., prompt learning) on seen base classes usually sacrifices the model's original generalization capability on unseen novel classes. To mitigate this issue, we present a new topology-preserving objective that can enforce feature topology structures of the combined base and novel classes to resemble the topology of CLIP. In this manner, our model will inherit the generalization ability of CLIP through maintaining the pairwise class angles in the attribute space. Extensive experiments on twelve object recognition datasets demonstrate that our model, termed Topology-Preserving Reservoir (TPR), outperforms strong baselines including both prompt learning and conventional generative-based zero-shot methods.

TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning
Hui Chen', Yanbin Liu², Yongqiang Ma Nanning Zheng'; Xin Yu² National Key Laboratory of Sluman--Mchhnn Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applications, and Institute of Artificial Intelligence and Roboties, Xi'an Jiaotong University, Auckland University of Technology,
The University of Queensland
Abstract
Pre-trained vision-language models (VLMs) such as CLIP have shown excellent performance for sero-shot classification. Based CLIP, recent methods design various learnable prompts to evaluate the zero-shot generalization capability on base-to-novel setting. This setting assumes test samples already divided into either base or novel classes, limiting its application to realistic scenarios. In this paper, focus zero-shot learning (GZSL), i.e., testing with no information about the base/novel division. To address this challenging zero-shot problem, we introduce two unique designs that enable us to classify an image without the need of knowing whether it comes from seen or unseen classes. Firstly, most existing methods only adopt a single latent space to align visual and linguistic features, which has a limited ability represent complex visual-linguistic patterns, especially for fine- ine-grained tasks. Instead, we propose dual-space feature alignment module that effectively augments the latent space with novel attribute space induced by well devisde attribute reservoir. In particular, the attribute reservoir consists of static vocabulary and learnable tokens complementing each other for flexible control over feature granularity. Secondly, finetuning CLIP models (e.g. prompt learning) on seen base classes usually sacrifices the model's original generalization capability on unseen novel classes. To mitigate this issue, we present new objective that can enforce feature topology structures the combined base and novel classes resemble the topology of CLIP. In this manner, model will inherit the generalization ability of CLIP through maintaining the pairwise class angles in the attribute space. Extensive experiments twelve object recognition datasets demonstrate that our model, termed Hopology-Preservingg Reservoir (TPR), outperforms strong baselines including both prompt learning and conventional generative-ban zero-shot methods.
more challenging and practical setting: generalized
1 Introduction
Young children often exhibit remarkable capacity to identify novel visual objects only based on verbal descriptions provided by their caregivers. This phenomenon has spurred considerable interest in developing learning models with similar feats, known as zero-shot learning (ZSL). Early ZSL works 2 trained model on the seen base classes and evaluated the generalization performance on novel unseen classes. This can be done by aligning the visual features and textual descriptors
Work done while visiting The University of Queensland. i Corresponding author: nazheng @ maa..xjtu.edu.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MAAZ
AwA2
seen features
OctandPah
CLB
unseen features
Fied101
FLO
FoutOO
(b) CLIP features
UC3101
SUN DCTII
SEX
115
PGVC-Airrrat
FCVC. Cinch-innuf
(c) Finetuned features
Country
StanfeeCars
SeateedCar
C₁
CUP
Padiidd
ZLAP
Onsi
(a) Performance comparison seen (left) and unseen (right) classes
(d) Our features
Figure 1: (a) In the challenging and realistic generalized zero-shot learning (GZSL) setting, our method significantly outperforms the state-of-the-arr methods on both seen and unseen classes. (b-d) Finetuning CLIP will lead to the weak generalization problem (8] unseen classes. We tackle this problem by inheriting the topology of CLIP feature space by maintaining the pairwise angles.
into a shared latent space, where the intrinsic visual-linguistic relationship extracted for zero-shot classification at test time. However, this conventional ZSL setting assumes test examples only come from novel classes, limiting its application to realistic scenarios where both base and novel classes need to be classified. Therefore, more realistic setting generalized zero-shot learning (GZSL) B 4 has been proposed to recognize both base and novel classes without knowing the base/novel division. Recent GZSL methods such as CE (5), LSA 6 and ZLAP 7 design various generative models, which generate the missing visual features for unseen novel classes and then conduct joint base and novel classification.
The emergence of large vision-language models (VLMs) such as CLIP have demonstrated good potential for ZSL. For example, CLIP is trained on 400 million image-text pairs, which can effectively capture the lisuul-linguisti links essential for ZSL. After training. the model can be applied for zero- shot classification using hand-crafted prompt, e.g., 'a photo of <class>' Starting from CLIP, recent works design diverse learnable prompts (e.g., conditional prompt S Multi-modal prompt [T0], and self-regulating prompt (11) improve CLIPP's performance for downstream zero-shot tasks. We find that these prompt methods adopt a base-to-novel setting to evaluate their zero-shot generalization capability, similar to the conventional 7SI. setting. In particular, they classify the base and novel classes separately, with the strong assumption that test samples have already been divided into either base or novel classes. Driven by the practicality of zero-shot learning, this paper focuses on more realistic setting of generalized zero-shot learning (GZSL) under the VLM context.
To tackle the challenging GZSL problem, introduce Topology-Preserviee Preserving Reservoir (TPR) model to effectively unleash the generalization potential of VLMs for the simultaneous classification of base and novel categories. To achieve the goal, proposed TPR embraces two core novel designs: dual-space feature alignment module and feature semantic topology preserving objective. Considering that most previous methods establish the visual-linguistic relations by employing single shared latent space 7 g 6) the complex and fine-grained patterns cannot be effectively captured. To mitigate this key issue, present dual-space feature alignment module by enhancing the latent space with representative attribute space, which is constructed from a well-devised attribute reservoir. The reservoir is designed to contain both static and learnable vocabulary tokens. In this fashion, both prior knowledge and task-specific information can be extracted, enriching the feature representations and avoiding overfitting single task.
Moreover, recent works g identify the weak generalizability problem of prompt learning on VLMs, i.e., the learned prompts
classes do not often generalize well to unseen classes (e.g., .. in Fig. (a) (right), ProGrad [22 underperforms CLIP on unseen classes). To address this problem, propose
objective (Fig. [ (b-d)) to maintain the semantic topology structure of the combined seen and unseen") classes by referring to the original CLIP embeddings.
For unseen, only the class names without accessing any additional data.

---

Specifically, we adopt the Pearson correlation coefficient to constrain the variation of angles between pairwise categories before and after CLIP finetuning. As a result, after finetuning. our model still inherits the good generalization ability of CLIP, without suffering from the weak generalization problem. This can be shown by the superior performance on the unseen classes in Fig. D (right).
Extensive experiments conducted on twelve object recognition datasets, including both traditional GZSL benchmarks and prompt learning benchmarks, demonstrate that our proposed method signifi- cantly improves the recognition performance on unseen classes over our baselines, and outperforms the state-of-the-aa ahe on eleven datasets. Our contributions are summarized as follows:
Different from the traditional base-to-nove setting, our work emphasizes a challenging yet practical generalized zero-shot learning problem for VLMs, without knowing the division of the base and novel categories. In this scenario, our proposed model significantly improves the recognition performance compared prior arts.
Reservoir
We introduce dual-space feature alignment module. It enhances the latent space, which is shared by visual and textual features, with representative attribute space constructed from an attribute reservoir. In this manner, we essentially improve the representativeness of the latent space, leading to better fine-grained alignment between visual and textual features.
We introduce new feature semantic topology-preserving objective that is designed to maintain the semantic topology structure of finetuned features. In this fashion, finetuned features will not severely overfit seen classes. Thus, we can effectively preserve the generalization capability of VLMs on unseen categories.
2 Related Works
Zero-shot Learning. Zero-shot Learning (ZSL)
aims to recognize unseen objects by leveraging auxiliary knowledge such as tags, attributes, or textual descriptions to bridge the gap between seen and unseen classes. Generalized Zero-shot Learning (GZSL) 16 18
1 14 15
extends the scope of ZSL by considering a more realistic scenario where both seen and unseen classes are 23
seek to learn latent space where visual and linguistic data are aligned and the inference is performed by searching which class has the highest similarity score. Besides, generative models 25 26 27
present during testing. Several methods [99 20 21 22
16
like GANs have been explored for ZSL/GZSL due to their superior performance. These models generate synthetic samples for unseen classes based on their semantic descriptions. In the Large Language Model (LLM) era, many works [24 29 30) focus on generating improved iisual-linguisti features for better alignment. While most methods default to utilizing attributes as semantic embeddings, attribute annotation poses scalability challenges for large-scale datasets. Moreover, attribute annotation exhibits subjectivity 31, leading to perceptible discrepancies among different annotators.
28
Vision- Language Models. Vision-Language Models (VLMs) aim to bridge the semantic gap between visual and textual modalities, enabling tasks such as image captioning, visual question answering. and image-text retrieval. Inspired by the success of self-supervised learning [32], vision-langual
pretraining has emerged as a powerful paradigm for learning rich representations of images and
text. Recent VLMs, such as CLIP 19, ALIGN (3 BLIP [44 VLMO [35]. CoCa [66 and FLIP 37. learn powerful joint representations by using contrastive learning on large amounts of paired Vision-language data. After pretraining on 400M pairs of data, CLIP constructs image classifiers using the class names of the target dataset in a zero-shot manner and achieves superior performance. Despite the direct applicability of VL.Ms to zero-shot recognition, empirical observations show suboptimal performance on fine-grann ane-grained tasks (1 In this work, we propose to align multimodal representations in dual-space improve the fine- fine-grained representation ability of VLMs for GZSL.
Prompt Learning in VLMs. Prompt learning has gained traction natural language processing as a powerful approach for adapting pre- treitrained language models to new tasks with minimal supervision B8 39. In prompt learning, task- speciicc prompts or templates are designed to guide the language model to generate outputs tailored to particular task, enabling effective adaptation to diverse downstream applications. Recent research [40] g has extended the concept of prompt learning to VLMs. By providing task-specific prompts that incorporate both visual and textual cues, VLMSs can seamlessly integrate knowledge from multiple modalities and generalize to unseen tasks with limited labeled data [10 12) For example, CoOp [40] finetunes the pretrained CLIP model by
3

---

input image
Image Projector
Latent space
X
cosine similarity
Po
Image
Encoder
K
s(x, 2)
V
attribute reservoir
PV
A
fi
brsaated chat merthenn Inlnat
gadwall
N1
N2
P.
Attribute space
V
gv
class topology
input description
A
X
HPV
Biight
Ghrest
comnestng
yetlon
Text Encodler
Pt
9t
Text Projector
shift
Pu PE, Pf:: learnable projection matrices
CA: cross attention
SA: self-attention
3: dot product
Overview of our TPR framework. The latent space directly aligns visual and linguistic features extracted from frozen VLMs. To augment latent space for fine fine-grained disual-textual pattern mining, we devise novel attribute reservoir for constructing new attribute space. The reservoir consists of both static and learnable vocabulary tokens, enabling flexible exploration and control of feature granularity for the GZSL task. Furthermore, we propose copology-preserving objective to keep the generalization capability VLMs, mitigating the weak generalization problem g
Figure 2:
inserting learnable context vectors into a fixed textual template. However, recent work (8) identified a weak generalizability problem of prompt learning: the learned prompt is not generalizable to wider unseen classes. In this work, we devise a novel copoloyy-preservngg objective to tackle this problem.
3 Methodology
Reservoir (TPR) framework (Fig. 2 to unleash the generalization capability of VLMs for GZSL. Specifically, TPR has two unique designs top of VLMs: (1) Dual- space feature alignment module to strengthen feature discriminability by aligning visual and linguistic features in both latent space and attribute space (constructed by the well-devised attribute reservoir); objective to maintain the generalization capability of VL.Ms through preserving both seen and unseen class topology before and after fine-tuning.
We propose a
(2) Feature semantic
Problem Formulation. Different from conventional ZSL or nase-to-novel setting, GZSL needs to address the challenge of recognizing both seen and unseen classes without knowing the scen/unseen division at test time, During training, only samples from the seen classes are available. Formally, given dataset S {xx,, uf = [=x E X E )² | E 2*), consist of n samples, where x y/. =; denote the visual feature, class label, and textual description feature of the i-th seen image, respectively. Mcanwhile, another dataset U {(2 V 2"" |a=" X" y E )" 2" E 1"1 contains ne samples, where 2y yj, 2 represent the visual feature, class label, and textual description feature of the j-th unseen image, respectively. The seen and unseen label sets are disjoint: ) n n = Following common practice in GZSL, the seen dataset S is split into training set D; and test set Di,, while the unseen dataset UI constitutes the test set D Then, the model is trained on D;, and evaluated on the union set D U D.. In the following, and will be omitted for simplicity.
3.1 Dual-Space Feature Alignment
Attribute Reservoir Construction. We devise the attribute reservoir to construct new attribute space, which can exploit the meticulous features overlooked by simple latent space. This further facilitates the effective mining of complex visual- lingul-inggiitic patterns for better GZSL. The reservoir design takes into account both the generalization capability and task-specific adaptation. Initially, we curate an extensive array of attribute terms sourced from diverse literature repositories, including CUB 42). MIT-States 4 MAD (1), VAW 44 and L.SA (45), which collectively delineate the shape, color, motion, material, texture, and part of an object. After climinating redundancies,

---

obtain a base attribute vocabulary of size N1, including attributes such as washing up, on stick, and pinstriped. Subsequently, we employ a pre-trained LLM [46] to extract features of this base attribute vocabulary for G,SS,, obtaining A E RNixd. The attribute vocabulary covers extensive attribute repositories to facilitate the generalization to unseen classes, but it is impossible to exhaustively include all conceivable attributes lying in image and textual data. Therefore, we introduce the flexible learnable attribute tokens, denoted as Ag E gNaxda, to augment the base attribute vocabulary. These tokens have two functions: (1) they learn complementary attribute knowledge absent in the base attribute vocabulary in a data-driven manner, and (2) they incorporate the sse--sccific information into the reservoir for better downstream task adaptation. Eventually, the two components of reservoir are concatenated together to form our attribute reservoir E R xd., where N = N1 + N2-
Multi-Modaliity Encoding. Given an input image, we initially utilize the pre-trained CLIP image encoder to extract its visual feature x E Rlxdd Subsequently, we project the visual feature into both a latent space and an attribute space (i.e., dual-space). To achieve this, we transform the visual feature and attribute reservoir into the same dimension with two linear layers (Pv. P.):
I f P. R Rld A = AP, E RN
The visual feature is then encoded into the dual-space:
x' Attn(z, Ap, A.) x" = Attn(r', xz, x²). = x" + x 2 R lxd
.0 f. IRIN,
(2)
where -t) is the attention function 47 Attn(Q.
softmax(QK") /VXV/VVVV fu denotes the visual feature encoded in the latent space, and 9u represents the visual feature encoded in the attribute space. On the text side, given the textual description corresponding to the image, utilize the CLIP text encoder to extract the linguistic feature: & E R xd Subsequently, we transform and the attribute reservoir to d- d-dimension using two additional linear layers (P,, P!t):
+ =P? IR A AP E Nxd
The linguistic feature is then encoded into the dual-space using cross-attention
Attn(2, A1, Ji = + = E 91 = f1 E IRIXN,
It is notable that only cross-attention is employed encode the linguistic feature, as overly intricate operations on the textual side may lead to overfitting.
Multi-Modality Alignment. We use the contrastive loss [48] to align the disual-linguisee features within the dual-space. Specifically, the contrastive loss in the latent space is defined as:
exp(Se /7 La fi) = ---log log exex(fo -(g/T) log exp(si fe /T) exexp(h /T//T)'
(5)
where foj and Jú, are the (2-normalized features of the j-th input image and text, respectively. - is temperature hyperparameter. The contrastive loss in the attribute space is formulated as follows:
Loa (gv: 91) -loog explg. of /T) expl(y. g[g/T)
exp(g 9: /T) E,explar g'g/T)
log
(6)
where Io.j and 9t, denote the 2-normalizee visual and linguistic attribute representations of the j-th input image and text, respectively.
3.2 Feature Semantic Topology Preservation
Compared with conventional ZSL, the GZSL problem is more challenging: the model needs to decide between both seen and unseen classes without knowing whether the test example comes from seen or unseen classes. Previous GZSL methods (5] 49 observed the domain bias problem: trained model is seriously biased towards seen classes in the testing phase. The bias arises because the model only trained on seen classes and, therefore, learns features and patterns specific to those classes. This phenomenon often leads to inferior performance on unseen classes in the GZSL evaluation setting. For VLM methods such as prompt learning & 10 [1 although the model weights are frozen, another problem called weak generalizability problem is observed [ the learned prompl is not generalizable to unseen classes within the same dataset. A reasonable explanation the learned task task-specific prompt overfits the seen base classes when finetuing VLMs. CoCoOP 8 try to alleviate
5

---

this problem by using setting rather than the more challenging GZSL setting To our knowledge, in the VLM context, no prior work has evaluated the challenging G7SL setting, let alone a principled solution to tackle the generalization problem under this setting.
prompt, but the evaluation is only in the Base-to-novel
In this paper, we tackle the weak generalizability problem in the feature space, particularly our newly introduced attribute space. Intuitively, the design of base attribute vocabulary has the effect of combating overfitting to some extent. But here, we are looking for more effective and principled solution. straightforward idea is to avoid overfitting to the seen classes by regularizing the features during the model finetuning process. For example, we can constrain the rank of image features to be no-decreasing, which boils down to maximizing the nuclear norm of the feature matrix [50]. However, this method empirically proves less effective. Then, we observe that the pre- pre-trained VLMs such as CLIP perform equally well on both seen and unseen classes (Fig. [ and also [8] 10 11).
This motivates us to inherit the generalization capability of CLIP to prevent overfitting to seen classes during the finetuning process, as shown in Fig. [b-d). Specifically, we want to maintain the class topology of CLIP embedding space topology is composed of all pairwise class angles, which can be calculated by cosine similarities (CLIP embedding is f-2 normalized), Denote the textual description features of all (seen+unseen) classes in CLIP's 's embedding space as Z = ===² 2.....?.....
way for generalization inheritage. The class
........................ 2 E Roxd The corresponding features in the attribute space are denoted G {gl .6 ..... 9 R Finally, in order to preserve the class topology in attribute space to be similar to that in CLIP embedding space, we adopt the Pearson correlation coefficient to define topology-preserving loss as follows:
(wn 62 Eig Wiy (Wls 12 Eig wiy))
(xp(
VEA (wi 1o2 Eig ) V V (wiy
A/2 Eij m
02-01 i = 1,2, c. Note that Eq. 7 maintains the pairwise Iaab feature angles between categories before and after finetuning. Hence, both seen and unseen classes can leverage the class topology of CLIP for better GZSL classification. More importantly, with
where Way = || [|= Wyy
Lip(Z, Gr), the finetuned model is less likely to be seriously biased towards seen classes, thus improving the generalization to unseen classes.
3.3 Learning Objective and Inference
The overall learning objective is:
L( Let + TLLa + BL
(8)
where 1) and 3 are loss weights, and 9 denotes model's trainable parameters. During inference, given a testing sample, we first extract its visual feature fo and perform the nearest neighbor search from all seen and unseen classes:
arg max Su Tly yy
where fuy is the linguistic feature of class y (all vectors are (2-normalized).
4 Experiments
Datasets and Metrics. We validate the effectiveness of TPR on four widely-used datasets in GZSL: AwA2 [5I] CUB 42. FLO [52], and SUN [53]. We follow the commonly-used dataset split 26
but use the generated textual descriptions instead of attribute annotation. To further evaluate the generalization ability, we conduct experiments
eight other object recognition datasets: FGVC- Aircraft [54], Country (9]. StanfordCars (55), EuroSAT [56), DTD (57), UCF101 [58]. Food101 [59]
and OxfordPets (60 These datasets are divided into seen and unseen classes in a similar way. Note that under the GZSL setting, the model can only be trained on seen classes and evaluated on both seen and unseen classes evaluate its generalization ability. We report the average per-class top-1 accuracy on seen classes (S) and unseen classes (U). respectively. To balance the two metrics, we also report the harmonic mean (H) of the seen and unseen accuracy: H SxU
S+U
In Table we evaluate CoCoOP under GZSL setting, but the performance unsatisfying on unseen classes.

---

Table 1: Comparison with state-of-the-art methods in generalized zero-shot learning (G7SL) setting. The proposed TPR obtains the best harmonic mean (IT) in 11 out of 12 datasets. Among them, TPR performs the best on all fine-grainer datasets (marked with *), and best (bolded) or second best (underlined) on almost all other datasets. Besides CLIP. TPR also works well with other VLMs: 1 CCoCa [36] (ViT-B/32) and EEVA-02 [62 (VIT-B/16). Equipped with the latter. we can fürther push forward existing SOTA by large margin.
AwrA2
SUN
Airenit
Countr U
Model
11
S
CUP
77.66
79.62
39.68
10.22
$1.91
$2.50
46
18.25
11:15
13.16
12:15
74.92
20.03
56.27
49.85
17.13
12.10
9.33
CoCu0p
76 10
21.51
60.21
54.26
18.81
13.60
13.59
8.08
10,09
21.25
21.75
PrmpiRCC
47.83
73.91
22.90
52.71
19.00
13.99
CE
76.69
67.80
71.97
19.01
23.50
62.02
51.86
44.11
45.54
28.63
25.25
12.80
8.07
9:90
LSA [3
71.07
25.66
13.51
29.44
27.85
12.21
7.31
9.30
ZLAP
74.24
75.54
32.41
25.51
29.55
08:22
69.76
48.18
47.29
47.73
29.35
27.10
29.19
12.64
10:42
TPK
87.10
81.63
32.53
77.56
70.45
50.47
45,40
47.80
29.65
18.75
16.05
17:26
TPR'
75
31.25
33.85
15.68
TPR
95.60
75.81
86.39
53,10
32.55
40,36
83,75
72.97
56.29
52.08
55.01
43.50
31.30
27.82
23.31
25.37
StanfosCaas**
EunSAT
DTD
UCFI01
Fowd101
OxfondPass
Model
0
.4
8
12
| 15
1
14
0
/
CLIP [9]
46.65
37.78
41.75
21.13
11.25
14.68
36.39
41.39
38.75
53.72
64.92
58.79
67.74
73.05
70.20
82.67
65:85
73.29
Colpp [00
49.36
38.47
43:43
29.89
12.27
17.40
44.34
36.56
40.07
62.13
47.41
53.77
71.82
64.64
68.04
73.47
57.66
64.61
CoCoOp B
$1.93
37.84
45:7%
52.64
18.34
27.21
42.
12.19
35.94
38.82
58.29
00.62
59.43
72.55
00.42
63.93
72.53
58.97
65.05
MaPLe 10
55.29
35.67
45:36
30.72
19.52
23.87
42.25
39:72
40.95
55.37
02.51
58.73
72.16
71.47
71.82
75.87
56.00
61.45
PromiiSC
11
55.36
39.85
46.41
28.71
14,40
19:18
51.30
41.56
35.52
61.92
39.39
00.39
77.06
56.31
55.07
78.60
52.99
63.30
ProGrd [2]
52:20
35:36
42.16
59.14
17.12
26.55
54.62
39.78
45.03
60.10
38.76
39.62
73.48
64.25
69.56
72.53
59.95
65.64
CF
40.94
47.52
61.61
43.58
74.79
29.61
35.65
54.78
42.46
00:48
53.83
21.07
1.A [g]
$9.19
41.41
48.73
55.10
34.20
45.
27.73
34.49
51.76
37.22
43.30
60:10
53.82
60.51
73.75
59:27
65.71
ZLAP
45.22
46.19
74.90
23.23
35.41
46.94
30.94
37.30
56.39
42.65
48.75
70.20
60.85
65.20
72.93
59.89
65.77
TPR
69.48
46.33
$5.99
52.75
45.73
58.91
55.47
43.06
47.84
69.14
66.88
67.99
93:67
85.41
90.60
66.39
76.63
TPK"
88.09
70.84
78.53
75.00
61,43
67.78
62.00
45.99
52.66
76.20
61.81
67.44
62.37
74.
84.07
21.32
77.17
TPR
87:25
73.57
79.83
76.07
36.41
64.75
68.95
46.39
$5.46
75.01
74.57
74.79
58.57
79.97
84.15
92.27
70.17
79.72
Textual Description Generation. We propose employing ChatGPT 61 to autonomously generate a descriptive paragraph for each class, leveraging its extensive knowledge base learned from diverse sources in the Internet. To facilitate this process for each dataset, we advocate employing the following prompt structure: "I have never seen images of <lype>. The following is good description of <class name>, so I can easily recognize <class name>." Here, the placeholder <type> corresponds to categories such as animals, birds, flowers, etc., while <class name> denotes the specific name of each class. As an illustration, consider the antelope class from the Aw dataset, where the generated description reads as follows: "Antelopes are herbivorous mammals known for their slender bodies and long, curved horns. They are often found in grasslands and open savannas, where they graze on vegetation and use their speed and agility to escape predators."
Training Details. We employ pretrained ViT-B/32 CLIP as the feature extractor, which outputs 5 512-dimensional visual and linguistic features. In total, we collect N1 = 5, 996 attribute words and we use pretrained Bert 46 to extract attribute features of dimension d, = 768. For all datasets, TPR trained for 200 epochs with batch-size of 512 via Adam optimizer on a single NVIDIA RTX4090 GPU. Overall, optimal hyperparameters of TPR are chosen from the following ranges: learning rate [le-5, 3c-5, 5e-5, 7c-5, lc-4). 17 E [0.2, 0.5, 1.0, 2.0), 3 E [le-4, 5c-4, 1e-3), e (0.03, 0.05, 0.07, 0.10J, and N₂ E {100, 200, 300, 4003, which are tuned on the validation set via grid search.
4.1 Comparison with the SOTA
We compare with the state-of-the-aa methods including prompt learning and generative- GZSL. We adapt them to our description- GZSL setting for a fair comparison. (1) Prompt learning methods, including CoOp [40), CoCoOp [8] MaPLe [10] PromptSRC [II] and ProGrad [12], fine- tune the pretrained CLIP through integrating trainable prompt tokens within a predefined template. To adapt these methods for GZSL, we input (image, label) pairs while omitting the utilization of textual descriptions. Notably, our setting differs from the base-to-novel setting (40) typically used in these methods, which originally predicts the seen and unseen classes separately. Instead, we conduct the comparison in the GZSL setting [26] where the prediction space includes both the seen and unseen classes, thereby posing a more challenging lask than the base-to-novel setting. (2) Generative- GZSL methods, including CE [5) LSA 6 and ZLAP 77, are adapted by substituting the input semantic features derived from attribute annotations with those obtained from textual descriptions. For fair comparison, all methods adopt the same backbone and dataset split.
7

---

AwA2
CUB
(c)
90
80
necurncy (%)
(%)
40
75
80
S
uecuracy
70
8
U
30
65
H
20
60
3
5
2
4
3
6
attribute vocabulary size (x10*)
attribute vocabulary size (x10")
attribute vocabulary size (x10P)
Figure 3: Impact size the attribute vocabulary (i.e., N1) on model performance.
(a) AwA2
CUB
(c) FLO
90
80
necurccy (%)
(5))
40
moqurncy (%)
75
70
30
70
65
20
60
200
500
300
500
100
200
400
#lcarnalle attribute tokens
#learnalle attribute tokens
#leamable attribute tokens
Figure 4: Impact of the number of learnable attribute tokens (i.e. N2) on model performance.
The performance analysis presented in Table reveals that TPR outperforms other methods in terms of H metric in 11 out of 12 datasets. Compared with prompt learning methods, TPR obtains an average absolute performance improvement of 12.46%, 9.46% and 11.94% on S, U and metrics, G7SL methods, TPR demonstrates
respectively. Similarly, in comparison with generative-basee an average absolute performance gain of 11.24%, 9.29%, and 10.68% on S, U and И metries, respectively. Remarkably, TPR achieves the best performance all ine-ggrined datasets (marked with *), underscoring its exceptional fine-grained data perception capability. Furthermore, while the recognition performance of many baselines, such CoCoOp, deteriorates for unseen classes compared to zero-shot CLIP, TPR continues to achieve outstanding results. This indicates TPR's proficiency effectively preserve CLIP's generalization capability to unseen classes. To verify the applicability of TPR to different VLMs, we utilize two VLMs, CoCa (36 and EVA-02[62],, backbone networks for feature extraction from input images and text descriptions. The results, presented at the bottom of Table demonstrate TPR's 's robust generalization across various VLMs.
4.2 Ablation Study
Loss Functions. We study the efficacy of two novel module designs Table 2 By augmenting the latent space with the attribute space (Let + Lei + Lea), the accuracy of unseen classes is improved by 4.26%, 2.78%, and 3.81% % for the three datasets. This underscores the attribute space's 's adeptness in capturing attribute features overlooked by the latent space, thus promoting the perception of unseen classes. Meanwhile, the inclusion of accuracy improvement for unseen classes, manifesting objective (Lat + Ltp) results a notable increase of 6.36%, 2.91 19,, and 1.8 % across the three datasets. Finally, incorporating both objectives (Lea and Ltp) into our full model, the accuracy outperforms each single objective almost all metrics (seen, unseen and harmonic) except for seen on CUB, demonstrating the complementary effect of two devised modules: dual-space feature alignment (Tea) and eopology-preservvv
objective (Iep).
Attribute Reservoir. We conduct an ablation study across various configurations of the attribute reservoir, presented in Table 2 Notably, competitive performance is attained when solely relying on the static attribute vocabulary. Conversely, utilizing solely Icarnable attribute tokens exhibits improved performance on seen classes, albeit with decrease in accuracy for recognizing unseen classes. Optimal H performance achieved when both types of attribute knowledge are combined.
Static Vocabulary Size N1. In general, larger N₁ entails the inclusion of a greater number of attribute words, facilitating finer-grained representation of objects and potentially enhancing recognition performance. The findings depicted in Fig. substantiate this notion. It is evident that as N; increases, the model performance improves significantly, especially for unseen classes. Overall, the harmonic mean H tends to saturate beyond 4,000 attribute words the coarse grained AwA2

---

classes Table Ablation different regularizers and reservoir components.
AwA2
CUB
FL.0
Setting
5
H
5
U
H
5
H
full model
87.10
76.81
81.63
41.22
26.87
32.53
77.58
64.52
70.45
Lit
85.70
68.14
75.92
41.55
23.24
29.81
74.01
60.45
66.55
objoctive
Lot
84.69
72.40
78.06
40.77
26.02
31.77
74.41
64.26
68.96
Lap
84.44
74.50
79.16
40.72
26.15
31.85
76.72
62.26
68.74
static vocabulary
86.71
75.23
80.56
41.11
26.12 12
31.94
74.62
64.11
68.97
reservoir
learnable tokens
87.69
64,58
74.38
41.27
20.85
27.70
82.72
60.71
70.03
Table 3:
Ablation on the topology preserving loss Ltp.
Setting
AwA2
CUB
FLO
S
H
U
H
U
w/o Lip
84.69
72,40
78.06
40.77
26.02
31.77
74.41
64.26
68.96
nuclear norm
87.88
72.48
79.44
41.09
25.36
31.36
74.72
64.47
69,22
orthogonality
87.15
74.87
80.55
40.36
26.13
31.72
73.68
64.95
69.04
Lift
86.93
77.00
81.67
40.61
26.00
31.70
74.65
64.34
69.11
Lup 87.10
76.81
81.63
41.22
26.87
32.53
77.58
64.52
70.45
Table 4: Comparison with the state-of-the-ar prompt learning methods under base-to-novel setting
EuroSAT Novel
DTD
Food101
Method
HM
Buse
Novel
Novel
CLJP 9
56.48
64.05
60.03
53.24
59.90
56.37
90.10
91.22
90.66
CoOp 40
92.19
54.74
68.69
79.44
41.18
54.24
88.33
82.26
85.
CoCoOp
71.21
77.01
56.00
64.85
90.70
MaPLe (10
94.07
73.23
82.35
80.36
59.18
68.16
90.71
92.05
91.38
PrompiSRC [1
82.32
83.37
62.97
90.67
TPR
95.12
76.66
84,90
84.80
59.39
69.86
94.03
91.15
92.57
dataset, while H continues to increase on the fine-grained CUB and FLO datasets. This again verifies the effectiveness of our model in capturing the fine-gres ane-ggrined complex patterns.
Learnable Tokens Quantity N2. As shown in Fig. 4 the accuracy unseen classes on the AwA2 dataset exhibits improvement with increasing N2. However, for the CUB and FLO datasets, the accuracy of unseen classes tends to plateau around 200 tokens, diverging from the pattern on Aw We conjecture that excessive number of learnable tokens may lead overfitting for fine-grained datasets, thereby limiting further enhancement.
Constraint. We ablate on various choices of the appoloyypreeseringg loss. These variants encompass: (1) muclear norm: maximizing the nuclear norm to ensure that visual features span the entire space; (2) orthogonality: enforcing orthogonality among class text features the attribute space; (3) aopoloyy-presevattie
Hopology-Preservee
in latent space (Lig'); maintaining pairwise angles between class features (Eq. ) in the latent space; (4) our topology preservation in attribute space (Lzg)). From Table 3, we observe that both nuclear norm and orthogonality cannot consistently improve the performance, possibly due the lack of reference CLIP. As the applogy--rreeeving objective, enforcing it on the novel attribute space yields consistently superior performance, which again reveals the better expressivity of the proposed attribute space.
Base-to-Novel Evaluation. As the prompt learning methods originally evaluate on the base-to-non novel setting, compare with them on standard prompt learning benchmarks for completeness and better understanding of the GZSL setting. Comparing T and Table 4 we notice that the performance setting surpasses the corresponding GZSL results, corroborating that
of all methods in GZSL is more challenging task. Moreover, our method outperforms or at least on on-par-witt the prompt learning methods in the base-to-novel setting.
5 Conclusion and Future Work
In this paper, we tackle the challenging GZSL problem by designing Topology-Preserving Reservoir (TPR) model. Specifically, TPR embraces two unique designs VL.ss: dual-space feature alignment module and feature semantic topology preserving objective. First, reservoir containing

---

both static and learnable vocabulary tokens is devised to construct a representative attribute space to enhance the latent space, which facilitates the exploitation of complex and fine-graing ane-grained visual- linguistic patterns. Second, we propose hopology-presenvve generalization ability of CLIP to mitigate the weak generalization problem of prompt learning methods. In particular, vopology-preserviie pairwise categories before and after CLIP finetuning. Comprehensive experiments are conducted on twelve object recognition datasets, validating the superior performance of our method in the challenging and more practical GZSL setting. the future, we want to investigate TPR on other applications such as few-shot learning and other modalities such video.
objective, which inherits the good
objective constrains the variations of angles between
Limitations. Text alone may not fully capture the nuances of fine- -eee-grrind datasets like CUB, while attribute annotations, though more accurate, are costly. Thus, a more desirable solution would be combining the knowledge from expert-provided attribute annotations with L.-M-generatd text to enhance performance. Additionally, our method may face challenges in aligning visual features of generic scenes with description features the attribute space, especially when descriptions are not sufficiently specific. This could be alleviated by providing more distinct and human-refined descriptions.
Acknowledgments
We thank all the anonymous reviewers and ACS for their valuable comments. This work was supported by the National Science Foundation of China (No. 62088102).
References
[1] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute- -ased classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):453-465, 2013.
[2] Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write classifier: Zero-shot learning using purely textual descriptions. Proceedings of the IEEE International Conference on Computer Vision, pages 2584-2591, 2013.
[3] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):1757-177, 2012.
[4] Yongqin Xian, Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 2265, 2018.
[5] Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive embedding for generalized zero-shot learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2371 2371-2381, 2021.
[6] Celina Hanouti and Hervé Le Borgne. Learning semantic ambiguities for zero-shot learning. Multimedia Tools and Applications, pages 1-15, 2023.
[7] Dubing Chen, Yuming Shen, Haofeng Zhang, and Philip H.S. Torr. Zero-shot logit adjustment. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 813-819. International Joint Conferences on Artificial Intelligence Organization, 2022. Main Track.
[8] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[9] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
10

---

[10] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[11] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan, Self-regulaiing prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15190-15200, 2023.
[12] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15669,15669, 2023.
[13] Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE international conference on computer vision, pages 4247-4255, 2015.
[14] Wii-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 52-68. Springer, 2016.
[15] Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5327-5336, 2016.
[16] Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Generalized zero-shot learning with deep calibration network. Advances in neural information processing systems, 31, 2018.
[17] Man Liu, Feng Li, Chunjie Zhang, Yunchao Wei, Huihui Bai, and Yao Zhao Progressive semantic-visual mutual adaption for generalized zero-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15337-15346, 2023.
(18) Vinay Verma, Nikhil Mehta, Kevin Liang, Aakansha Mishra, and Lawrence Carin. Meta- learned attribute self-interaction network for continual and generalized zero-shot learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2721-2731, 2024.
[19] Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region graph embedding network for zero-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 562- 580. Springer, 2020.
[20] Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gradient matching generative networks for zero-shot learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2168-2178. 2019.
[21] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. Advances in Neural Information Processing Systems, 33:21969- 21980. 2020.
[22] Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6857-6866, 2018.
[23] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov, Devise: deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[24] Muhammad Ferjad Naeem, Yongqin Xian, Luc V Gool, and Federico Tombari. 12dformer: Learning image to document attention for zero-shot image classification. Advances Neural Information Processing Systems, 12294 2022.

---

Chang-Dong [25] He Huang, Changhu Wang, Philip S Yu, and Chang- Dong Wang. Generative dual adversarial network for generalized zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 801-810, 2019.
[26] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5542-5551, 2018
[27] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for sho- learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10275- 10284, 2019.
[28] Junhan Kim, Kyuhong Shim, and Byonghyo Shim. Semantic feature extraction for generalized zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1166-1111111 1173, 2022.
[29] Muhammad Ferjad Nacem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Didier Stricker, Luc Van Gool, and Federico Tombari. 12mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55169-151177 2023.
[30] Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few- shot adaptation for vision-language models. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 15534-15546, 2023.
[31] Hui Chen, Jingjing Jiang, and Nanning Zheng. Learning infer unseen single-/multi-attribne object compositions with graph networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[32] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[33] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages PMLR, 2021.
[34] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified dision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
[35] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified lision-language pre-training with couuur-of-mddlltyyeeee 32912, 2022.
Advances in Neural Information Processing Systems, 35:32897-
[36] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205001917, 2022.
[37] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23390-23400, 2023.
[38] Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Mian-Guang Lou. Input-tuning: Adapting unfamiliar inputs to frozen pretrained models. arXiv preprint arXiv:2203.08 031331 2022.
[39] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
12

---

[40] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision, 13009))3337-2348, 2022.
[41] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans-
formers: An occam's s razor for domain incremental learning, Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
[42] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongic. The caltech-uesd
dataset, 2011.
[43] Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2015.
[44] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13018-13028, 2021.
[45] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Improving closed and open-vocabulary attribute prediction using transformers. In European Conference on Computer Vision, pages 201-219. Springer. 2022.
[46] Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1818.000000555. 2018.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 13:18661-18673, 2020.
[49] Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi- view zero-shot learning. IEEE transactions on pattern analysis and machine intelligence, 37(11):2332-2345 2015.
[50] David F Gleich and Lek-heng Lim. Rank aggregation via nuclear norm minimization. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 60 00-688,77 2011.
[51] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on puttern analysis and machine intelligence, 36(3):453 2013.
[52] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-72-777 722-729. IEEE, 2008
[53] Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and rec- ognizing scene attributes. In 2012 IEEE conference on computer vision and pattern recognition, pages 2751-2758. IEEE, 2012.
[54] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5757 2013.
[55] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pages 551, 2013.
[56) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
13

---

[57] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and paltern recognition, pages 3606-3613, 2014.
[58]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[59] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 10i-mnnng discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[60] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498 3505. IEEE, 2012.
[61] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[62] Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva-02:
A visual representation for neon genesis. arXiv preprint arXiv:2303. 1/331, 2023.
[63] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[64] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang. Jingho Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 3090-3100, 2023.
[65] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19187-19197, 2023.
[66] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
[67] Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint 1901v:1901.1 2019.
[68] Yi Li, Hualiang Wang. Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653. 2023.
[69] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 011225788-5299 2022.
14

---

A Appendix
A.1 Dataset Statisties
We present statistical information on the 12 datasets used in the experimental evaluations, depicted in Table The first four rows correspond to benchmark datasets widely employed GZSL, with dataset splits provided in previous works 4 The remaining eight datasets are commonly used for object recognition and encompass natural objects, textures, actions, scenes, etc. We adhere to a similar data split methodology in [4] dividing these eight datasets into seen and unseen classes.
Table 5: Detailed dataset statistics.
Training & validation classes
Training & validation size
Datasets
Classes
Seen classes
Unseen classes
Testing size
AwA2
50
40
27+13
23,527
13,795
CUB
200
150
100+50
7,057
4,731
FL.
102
82
20
62+20
4,401
3,788
SUN
717
645
580+65
10,320
4,020
100
55+25
Country
211
1.1
101+50
33,220
30,080
StanfordCars
196
136
60
95+41
7,891
8,294
EuroSAT
10
6
10,200
16,800
DTD
47
32
21+11
2,304
3,336
UCF101
101
70
31
48+22
5,973
7,347
Food101
101
72
29
49+23
50,400
50,600
OxfondPets
1748
A.2 Textual Description Exemplars
Exemplars of textual descriptions of 12 classes in the AwA2 dataset generated by ChatGPT are shown in Table Notably, the generated descriptions almost cover the visual characteristics of the corresponding class.
A.3 Additional Experiments
Multiple Textual Description Evaluation. Prior work [29 has demonstrated that using multiple textual descriptions for each class can capture its semantic characteristics more comprehensively; because these descriptions may complement one another. Therefore, we undertake an investigation into the influence of multiple descriptions on model performance on CUB. this endeavor, we employ ChatGPT to generate five distinct descriptions for each class 29 Subsequently, we randomly select between and descriptions, denoted as n, to form multiple descriptions for each class. To conduct experiments in the multiple text setting, adapt the loss functions by integrating multi-view contrastive learning techniques (63), As depicted in Fig. we present the performance for each n by repeating the sampling process 10 times. Notably, when is small, the recognition performance exhibits more significant fluctuations, attributed to substantial differences among each sampled text. As increases, the model's performance stability improves, and all three accuracy metries gradually enhance, particularly the accuracy for recognizing unseen classes (U). The results underscore the advantages of leveraging multiple descriptions in improving overall model performance, with notable impact on recognizing unseen classes.
Textual Description Evaluation. Although ChatGPT demonstrates significant generation capabilities, a comprehensive evaluation of the quality of the generated text descriptions has not yet been conducted. In this study, approximately evaluate the quality of these text descriptions by calculating the cosine similarity between the description features and the corresponding attribute features (if available) for each class. As illustrated in Fig. 6 empirical analyses are performed on the AwA2 and CUB datasets, both of which provide ground-trutt -tuun attribute annotations. The findings indicate a high similarity between the description features and the corresponding ground-truth attribute features, thereby suggesting a high quality of the generated descriptions.
Correlation between Attributes. We curate an extensive attribute vocabulary to comprehensively encapsulate various characteristies. described the main paper, TPR can still achieve relatively good results even if most of the items are removed from the attribute vocabulary. Here, present
15

---

Classes
Images
Textual deseriptions
antelope
Antelopes herbivorous mammals known their slender bodies and long, curved horns. They are often found in grasslands and open savannas, where they graze on vegetation speed and ugility escupe predators.
grizzly bear
Grizzly bears large, ommiverouu mammals distinctive hump of their shoulders. They have shugey fur that can range from blond to dark brown and are known strength and ability fish from rivers.
killer whale
Killer whales, known as orcas, marine mammals known for their striking black and white coloration. They highly intelligent and social, often traveling pods. Orcas are skilled predators that feed on lish, seals, and even other whales.
beaver
Beavers are semi-aquati rodents known for building dams and lodges. They have webhed feet and flat, scaly tnil, which they use for swimming and steering, Beavers are herbivores and primarily consume tree bark and aquatic plants.
dalmatian
Dalmatians are breed of dog known for their distinctive black spots on white cost. They are energetic often associated firehouses. Dalmatians their friendly demeanoe and strong guarding instincts.
persian
Persian cars are known for their long, luxurious fur and distinctive flar faces. They are calm and gentle broed, often found as inclor pets. Persian cars roquire regular grooming due their thick coats.
horse
Horses large, hoofed mammals often used tiding, racing. and work. They come in various breeds and colors, with distinctive features such as flowing mune and tail. Horses have played crucial mle in human history for mansportation and agriculture.
german shepherd
German Shepherds breed of dog known for their intelligence, loyalty, and versatility. They have a distinctive appearance with a strong, muscular body and a double German Shepherds are often used working dogs like police work and search and rescue.
blue whale
wholes largest animals Farth, a long and streamlined hody that is peedominantly blue color. They are filter feeders, using baleen plates to capture krill other small organisms.
siamese eat
Stamese cats are knowa for their striking blue almond-shaped eyes and celor-pointed fur, with pale body and dark face, paws, and tail. They are vocal and social cats with strong band their owners.
skunk
Skunks are small mammals known for their distinctive black and white coloring and the ability spray foul-smelling liquid defense mechanism. They have bushy tails and are omniverous, feeding on insocts, small animals, and plants.
mole
Moles are burrowing mammals with velvety for, small eyes, and powerful forelimbs equipped for digging. They primarily feed intricate tunnel sy systems underground.
earthworms create
Table 6: Textual description demonstrations for 12 classes in AwA2, which are generated by ChatGPT. Best viewed in color.
16

---

U Uaceurscy (%)
H accurney (%)
Saceurccy
41
51
32
39
37
4
2
5
2
5
text number /
text number n
text number v
Figure 5: Effects of multiple textual descriptions on CUB. For each class, we randomly sample n textual descriptions 10 times.
(a) cosine similarity distribution AwA2
(b) cosine similarity distribution CUB
1.0
0.8
cosine similarity
0.8
cosine similarity
0.6
0.6
0.4
0.4
0.2
0.2
10
15
25
30
35
15
30
id
Figure 6: The cosine similarity between textual description features and corresponding attribute annotation features AwA2 (left) and CUB (right).
Correlation matrix between fixed attribute words
Correlation matrix between learnable attribute tokens
1.0
1000
50-
0.5
0.5
2000
100
Attribute ID
Attribute
3000
0.0
150
0.0
2500
170
200
0.5
5000
2400
250
150
2100
2200
100
120
1.0
300
1.0
1000
2000 3000
4000
5000
50
100
150
200
250
300
Attribute
Attribute ID
Figure 7: Correlation matrix between the features of attribute vocabulary items (left) and correlation matrix between the features of learnable attribute tokens. Best viewed color.
the correlation matrix for the attribute vocabulary items and the correlation matrix for the learnable attribute tokens, as illustrated in Fig. 7 The figure demonstrates that many items in the attribute vocabulary are highly correlated, indicating redundancy within vocabulary. Consequently, re- moving highly similar items does not significantly affect the final performance. In contrast, the correlation between learnable attribute tokens much lower, requiring only few hundred tokens to achieve performance gains. While it may seem possible to achieve the same recognition accuracy with significantly smaller attribute vocabulary [64] 655, this paper primarily focuses on enhancing the fine-graines discrimination ability of features. We will address the issue of obtaining more condensed attribute vocabulary in our subsequent work, potentially utilizing principal component analysis to achieve this objective.
Visual Features Visualization. In Fig. S we present the distribution visual representations the attribute space for the AwA2 and CUB datasets. For each dataset, the features of 500 samples are visualized with the t-SNE technique. The visualization reveals that samples from the same class are well clustered in the attribute space, while those from different classes separated. In contrast,
17

---

(A) CLIP seen image features on NwAZ
(b) CLIP unsecs image deatures on AnAAWAZ
CLIP seen inage leatres CUB
(c) CLIP urseen image features CUB
0.
0,5
0,
0.3
0.2
0.2
0.3
(6) 1PK seen inage Testures on NwAA
10 TPR unseen image features on AwAZ
(3) TPR scen image Teatures on CUB
(1) TPK usseed image features on CUB
065
06
0,5
10
0.2
0.
0.0
0.
0.00
0.25
0.25
0.50
0.75
1.00
0.50
0.75
Figure 8: t-SNE visualization visual representations on AwA2 and CUB. Instances of the same class are marked with the same color. Best viewed in color.
78
88
U accuracy (%)
87
76
accuracy
|| accuracy
86
74
85
78
72
83
70
76
0.1
0.3
0.4
0.0
0.3
0.0
0.2
0.5
Pa
Figure 9: Impact of noisy textual descriptions on model performance on AwA2, and pr. denotes the intensity of the imposed noise.
CLIP image features do not present good clustering characteristics. This observation aligns with findings the literature (66), which indicate that CLIP image features tend be more scattered and cluttered due to CLIP's objective of learning the association between images and their corresponding text, rather than developing compact image features. The visualization result demonstrates that TPR effectively aligns visual and textual features in semantically structured attribute feature space.
Robustness Noisy Textual Deseriptions. To examine the impact of noisy text deseriptions on TPR's performance, we introduce perturbations into the generated text descriptions. Specifically, we utilize the natural language perturbation tool proposed by EDA 677, which produces noisy text through four operations: synonym replacement, random insertion, random swap, and random deletion. Let p., denote the probability of applying these operations. These noisy descriptions are then used the textual input for TPR, while the rest of the network remains consistent with the configuration described in the main paper. The experimental results, shown in Fig. g indicate that TPR maintains robustness against textual perturbations. As the noise intensity pr. increases, the recognition performance unseen classes significantly degrades, whereas the performance seen classes gradually decreases. Notably, even under 50% noise intensity, TPR consistently achieves performance exceeding 78% on the / metric.
Visualization of Multimodal Features in the Attribute Space. We present the distribution of visual features (gv) and corresponding textual features (gi) in the attribute space. For clarity, we display the feature values for 14 attributes across 10 categories in AwA2, as illustrated in Fig. 10 The visual feature values are averaged for each class. The distributions of visual and textual features for each class exhibit substantial similarity, thereby facilitating zero-shot recognition. Furthermore, these representation values align with the actual characteristics of each class, such as their visual appearance. For example, the persian cat class prominently features the attribute with brown eyes.
18

---

(a) Visual peojection value
Testual projection vulue
9012
002
001
bonver
9.00
003
parkon CR
0001
NERROc cot
-002
skank
-010
#10 SeooSaaand Saseano
go
Figure 10: Projection visualization of multimodal features, with the left figure illustrating visual
projection and the right figure illustrating textual projection. Best viewed in color.
change Let loss on AwA2
83
change Lap loss on AwA2
33
(c) change Lu loss on CUB
33.0
()) change Lap loss on CUB
accurnyy (%)
accurnc (%)
accuracc (%)
122
11 acqurncy
81
31
32.0
79
30
31.5
78
79
29
31.0
0.8
1.2
1.6
2.0
10
0.4
0.8
1.2
1.6
10
7
x10
Figure 11: Impacts of the loss weights y and
original image
back
rump
tail
upperwing
CLIP
TPR
CLIP
TPR
Figure 12: Textual respondence visualization. We present the response distribution to specific text within an image by using CLIP surgery (68) The top row denotes the query text, with each subsequent row illustrating the heatmap distribution of responses from CLIP and our proposed method, respectively. Best viewed in color.
Hyperparameter Sensitivity. We investigate the impact of hyperparameter settings, specifically the loss weights 7) and 3, model performance the AwA2 and CUB datasets. As illustrated in Fig. II H initially improves and subsequently declines as both and / increase, with the maximum fluctuation being approximately 3%. TPR exhibits robustness to variations in and within reasonable range.
Textual Respondence Visualization. In Fig. 12 we present a heatmap of textual descriptions in images using CLIP Surgery (68). The color intensity represents the response value of the region to the respective text, with redder colors indicating higher response values. As demonstrated, TPR
19

---

Table Ablation backbones and pretraining data scale.
Pretraining data
AwA2
CUB
FLO
Backbone
U
H
S
U
H
S
U
H
RN50
81.64
67.86
74.11
33.22
23.41
27.47
62.76
58.69
60.66
RN101
87.20
71.73
78.71
36.78
29.15
32.52
66.29
63.68
64.96
400M
VIT-B/16
89.34
83.15
86.13
46.21
35.71
40.29
80,04
65.47
72.02
VIT-B/32
87.10
76.81
81.63
41.22
26.87
32.53
77.58
64.52
70.45
ViT-B/16
90.77
81.86
86.08
58.56
48.44
53.02
82.02
65.51
72.84
2B
ViT-B/32
86.80
78.41
82.39
47.05
37.02
41.44
81.62
64.78
72.24
Table 8: Results of TPR with text descriptions and TPR with ground-truth attributes. The experiments are conducted on both textual descriptions and attribute annotations to validate TPR's compatibility.
AwA2
CUB
SUN
Method
5
U
altribute
76.73
63.50
69.49
24.39
22.74
23.54
42.98
41.60
42.28
text
87.10
76.81
81.63
41.22
26.87
32.53
50.47
45.40
47.80
localizes objects of interest, which helps to minimize background interference. contrast, the heatmap generated by CLIP is more dispersed and tends capture cluttered objects. However, TPR may exhibit insufficient focus; for instance, when the query is "back", regions outside the back still show large response values. We defer addressing this issue to future work, which will explore the potential use of attention mechanisms to refine focus on specific regions.
Backbone Networks and Pretraining Data Scale. In Table 7 we delve into the comparison of four distinct variants the CLIP visual backbone, each characterized by considerable number of parameters. Specifically, the ResNet50 model comprises 102M parameters, the ResNet101 model comprises 120M parameters, the ViT-B/16 model comprises 150M parameters, and the ViT-B/32 model consists of 15 parameters. It is noteworthy that the number of model parameters increases, there a substantial improvement all three accuracy metrics across all three datasets. These results underscore the robust generalizability of TPR across different backbone networks. Subsequently, examine the impact of the amount of pretraining data on model performance by extracting visual and linguistic features using Laion-CLIIP (2B pretraining data) [69] As depicted in Table Laion-CI.I outperforms CLIP when employing both ViT-B/16 and ViT-B/32 backbones. These findings suggest that larger pretraining datasets generally result in improved model performance.
Attribute-bssed GZSL. In addition to utilizing pure text-based descriptions for GZSL, TPR can also leverage attribute annotations (if available). Specifically, we utilize the CLIP text encoder to extract attribute-annotated features for each category. Simultaneously, the image encoder is utilized to extract visual features for each image. The CLIP text and image encoders kept frozen. The structure of TPR, except for the modified semantic inputs, remains consistent with the description provided in the main paper. The results presented in Table underscore the compatibility of TPR with atriibue--annotaioo
GZSL.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clarify the research contributions and scope of the study in the abstract and introduction of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the Conclusion and Future Work section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detail the model network structure and hyperparameter configurations needed to reproduce the results in the Methodology and Experiments sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to the code the Anonymous GitHub link https///annnmmoooos.oooosseeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details in the Methodology and Experiments sections.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in the appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in Fig.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments can be run on a single NVIDIA RTX 4090 GPU with 24GB
RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Running the experiment consumes GPU computational resources, resulting in carbon dioxide emissions.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We restrict the release of models to academic use only to avoid misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the creators or original owners of assets in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We detail the network architecture and training/testing implementations the paper.
Justification:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "viVsE6h4wt", "reviewer": "Reviewer_BqNV", "summary": "This paper is a new study that introduces the Generalized Zero-Shot Learning (GZSL) framework within VLMs, aiming to classify both known and novel classes without class partitioning. Key innovations include a dual-space feature alignment module, enhancing latent representations with an attribute reservoir for nuanced visual-linguistic patterns. Additionally, a topology-preserving objective ensures that model adaptations preserve the semantic structure learned by CLIP, thus maintaining generalization across all classes. Extensive experiments across diverse datasets validate the proposed Topology-Preserving Reservoir (TPR) model, demonstrating superior performance over conventional methods in recognizing both seen and unseen classes, underlining its potential for practical applications in complex visual recognition tasks.", "questions": "GZSL is a long standing problem in ML/AI community with many classic solutions, like stacking[1] .etc. Since the authors only compare with VLM-based methods, my concern is about the perfromance of classic methods in VLM-based GZSL tasks, can they achieve surprising performance with well-trained features?\n[1] Chao W L, Changpinyo S, Gong B, et al. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild[C]//Computer Vision–ECCV 2016", "limitations": "None in particular.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 1, "strengths": "1. This paper introcuces a novel research aspect for VLMs: generalized zero-shot learning, which requires the model to identify both seen and unseen concepts at the same time. From my perspective, this proposal could be a great contribution to VLM community.\n2. This paper is well-organized and well-written, which makes it easy to follow. \n3. Extensive experiments，ablation study and visualization results demonstrate the effectiveness and rationality of TPR.", "weaknesses": "None in particular.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer BqNV\n\nWe sincerely thank the reviewer BqNV for the very positive and helpful comments. \nThank you for acknowledging *the novelty of our idea*, *our contribution to the VLM community*, *the writing and organization of our paper*, and *the extensive experiments & ablation study*. \nWe address your questions as follows: \n\n**Q1: Can classic methods [1] achieve superior performance with well-trained features in VLM-based GZSL tasks?**\n\n**A1:** Thank you for the comment. In Table R1 below, we compare with the two variants of the classic method [1] using pretrained VLM-based features (*i.e.*, ViT-B/32 CLIP). The classic method SynC [1, 2] achieves competitive results on seen classes, but degrades significantly on unseen classes. Specifically, SynC [1, 2] constructed classifiers for unseen classes by linearly combining the phantom classifiers trained on seen class data. Therefore, it exhibits a significant bias towards seen classes, resulting in inadequate generalization to unseen classes. This poor generalization phenomenon is also observed in Ref [3]. \n\nIn contrast, our TPR method obtains promising results on both seen and unseen classes, demonstrating the generalization capability. We will include the results and discussion with Ref [1-3] in the revision. \n\n**Table R1. Performance comparison between the classic methods [1] and ours in the VLM-based framework.**\n|                     |         AwA2         |         CUB          |\n|---------------------|----------------------|----------------------|\n|        Model        | $S$ / $U$ / $H$ |  $S$ / $U$ / $H$ |\n| a) CLIP             |  81.7 / 77.7 / 79.6  |  29.9 / 29.6 / 29.7  | \n| b) SynC$^\\text{o-vs-o}$ [1] |  86.0 / 15.1 / 25.7  |  31.9 / 10.5 / 15.8  | \n| c) SynC$^\\text{struct}$ [1] |  88.3 / 16.7 / 28.1  |  33.8 / 11.4 / 17.0  | \n| d) TPR (Ours)       |  87.1 / 76.8 / **81.6**  |  41.2 / 26.9 / **32.5**  | \n\n\n[1] Wei-Lun Chao, et al. *An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild*. ECCV, 2016.\n\n[2] Soravit Changpinyo, et al. *Synthesized Classifiers for Zero-Shot Learning.* CVPR, 2016.\n\n[3] Yongqin Xian, et al. *Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly.* IEEE T-PAMI, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpl2xOty9T", "reviewer": "Reviewer_TcJo", "summary": "The proposed approach targets the generalized zero-shot learning (GZSL) problem for the vision language model (VLM). It is observed that a strong VLM model shows promising results for novel class generalization. Fine-tuning these models for seen classes leads to a loss in generalization capability and poor results for unseen classes. Additionally, a single latent space demonstrates limited ability to adapt to complex visual-linguistic patterns in fine-grained datasets. The paper proposes dual-space alignment, augmenting the latent space with static and learnable tokens. To address the generalization problem post fine-tuning, the paper introduces a Topology-Preserving Reservoir (TPR), which helps preserve the model's generalization ability for unseen classes. The authors conducted extensive experiments across several standard ZSL datasets and explored the impact of various components through ablation studies.", "questions": "Please refer to the weakness section.", "limitations": "Authors has not discussed the limitation in the paper, while in the Checklist they said \"Yes\" i.e. they had discussed. This is a bad practice, I don't know what to do.\n\nDear AC please look into it.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] Generalization of unseen classes in VLM is a critical problem. The strong pretrained model also loses its generalization ability, which the author explores, and the proposed model shows a significant impact.\n\n[2] The idea and intuition behind the static and learnable attribute reservoir are interesting. Additionally, TPR helps improve generalization.\n\n[3] The wide-ranging experiments conducted across various ZSL datasets and the ablation studies are satisfactory.", "weaknesses": "[1] The standard ZSL model assumes that there is a description per class rather than per sample, which is more intuitive since a single description for each class suffices for the model to understand the class, making it cost-efficient. Standard annotation-based attributes often yield better results for ZSL/GZSL settings. For example, [a] demonstrates impressive results for the CUB dataset compared to the proposed complex static, learnable, and description-based model. This issue is particularly observed in fine-grained datasets. Why is this the case?\n\n[2] It is unclear how the base attribute vocabulary is created. At a high level, the author collected a few attributes and obtained LLM embeddings. This description may not be sufficient for reproducibility since the code and data are not provided.\n\n[3] There are multiple variants of TPR (Table-3) in various scenarios where different methods work, making it difficult to apply and choose the best one. What does the author conclude here?\n\n[4] In Table-1 for the SUN dataset, the model shows inferior performance. While we do not expect the model to outperform in all scenarios, a clear description and author observations are required: why is this the case?\n\n[a] Meta-Learned Attribute Self-Interaction Network for Continual and Generalized Zero-Shot Learning, WACV-24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer TcJo\n### Response Q1-Q6\n\nThank you for the valuable comments with many kind words to our work: *a critical problem, significant impact, interesting, improve generalization, wide-ranging experiments, satisfactory ablations studies*. \nBelow, we address the raised questions:\n\n**Q1: Whether a description is provided to each class or each sample?** \n\n**A1:** We would like to clarify that our method utilizes the **Class-Level** textual descriptions (a description per class). Therefore, our method is cost-efficient with the class-level design.\n\n**Q2: Standard annotation-based attributes often yield better results for GZSL settings. For example, [a] demonstrates impressive results on CUB.** \n\n**A2:** Thank you for your constructive comments.\nFirstly, we agree with Reviewer TcJo that sometimes standard annotation-based attributes yield good results for ZSL/GZSL. However, they require *extensive expert knowledge and detailed class-level attribute annotation for both seen and unseen* classes. In contrast, our method relies *only on attribute names and LLM-generated descriptions*, which are easier to obtain and significantly reduce the need for human efforts. \n\nSecondly, we speculate that the impressive results of [a] on CUB are due to the usage of meticulously annotated attributes. Specifically, the CUB dataset provides 312-dimensional densely labeled attributes, which capture the characteristics of bird body parts, such as the beak, head, trunk, tail, and claws. Compared with the well-annotated attributes, LLM-generated descriptions may overlook some key attributes characterizing the bird classes. Thus, learning with detailed attribute annotations [a] leads to recognition improvements. \n\nWe will **cite [a] and include the above discussions** in the revision. \n\n[a] Vinay Verma, et al. *Meta-Learned Attribute Self-Interaction Network for Continual and Generalized Zero-Shot Learning.* WACV, 2024.\n\n**Q3: How is the base attribute vocabulary created ?**\n\n**A3:** We detail its construction as follows:\n- First, we collected the readily available attribute words from diverse attribute recognition related repositories, such as MAD, VAW, and LSA. For example, MAD dataset has 158 attributes, which are incorporated into our base vocabulary. Note that we just used the attribute names without involving any annotations.\n- Then, we eliminated duplicate attribute words to form a base vocabulary of 5,996 attribute words. A pre-trained LLM was adopted to extract features. \n\nIn Appendix Line 751, we provided the anonymous code link. We will make all code and data public upon acceptance.\n\n**Q4: Choose of multiple variants of TPR and the conclusion.**\n\n**A4:** Thank you for the comment. In GZSL, Harmonic Mean ($H$) is the most comprehensive metric to evaluate model performance, which takes into account both seen and unseen classes. Therefore, we choose the best model according to the results of $H$. \nFor ease of comparisons, in Table R2 below, we average the performance *w.r.t.* three datasets and observe that our proposed topology-preserving loss (row e: $L_{tp}$) obtains the best results. \n\nOur conclusions are two-fold: 1) all regularization variants exhibit a degree of performance improvement compared to the baseline, indicating the importance of constraining spatial structure of CLIP; 2) among these, our topology-preserving loss $L_{tp}$ shows better performance than the other variants, highlighting its effectiveness. \n\n**Table R2. Average performance of different variants of TPR on three datasets: AwA2, CUB and FLO. The proposed topology-preserving loss $L_{tp}$ obtains the best results.** \n|        Model        |   $S$ / $U$ / $H$    |\n|---------------------|----------------------|\n| a) w/o $L_{tp}$     |  66.6 / 54.2 / 59.6  |\n| b) nuclear norm     |  67.9 / 54.2 / 60.0  |\n| c) orthogonality    |  67.1 / 55.3 / 60.4  |\n| d) $L_{tp}^{lat}$   |  67.4 / 55.8 / 60.8  |\n| e) $L_{tp}$         |  68.6 / 56.1 / **61.5**  |\n\n**Q5: Explanations for the inferior performance on the SUN dataset.**\n\n**A5:** Unlike object-specific classes, SUN is a **generic scene** dataset containing 717 categories, such as \"amphitheater\" and \"amusement park\". These broad scene categories exhibit extensive variations in their components and features, making it challenging for our TPR method to align images with descriptions. For instance, the category \"amusement park\" may include diverse elements such as \"crowds\", and \"buildings\", which vary significantly across different instances. This diversity complicates the projection of both multimodal features into the attribute space, where consistency is harder to achieve.\n\nMoreover, we observe that several class descriptions in the SUN dataset exhibit high cosine similarity scores. Specifically, about 5% of the textual descriptions have similarities higher than 0.8, with an average similarity of **0.69**. In contrast, the average similarity for the FLO dataset is **0.52**. The high similarity across descriptions of different categories adversely impacts the model's performance, especially for unseen classes. \n\nFurthermore, we would like to emphasize that although TPR is 3.2% lower than ProGrad [12] on the SUN dataset, while on other datasets TPR is on average about 10% higher than the state-of-the-art.\n\n**Q6: Summary of limitations.**\n\n**A6:** We summarize the limitations of TPR as follows:   \nText alone may not fully capture the nuances of fine-grained datasets like CUB, while attribute annotations, though more accurate, are costly. Thus, a more desirable solution would be combining the knowledge from expert-provided attribute annotations with LLM-generated text to enhance performance. Additionally, our method may face challenges in aligning visual features of generic scenes with description features in the attribute space, especially when descriptions are not sufficiently specific. This could be alleviated by providing more distinct and human-refined descriptions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9qxaQicbeD", "reviewer": "Reviewer_k9s8", "summary": "In this paper the author proposed a dual-space feature alignment module to keep the semantic consistency between visual and attribute. In addition, the authors proposed Topology-Preserving Reservoir (TPR) to tackle the issue into the generalized zero shot learning (GZSL) setting, which utilized the Pearson correlation coefficient to define a topology-preserving loss, which effectively prevents overfitting of the seen and unseen classes. Sufficient experiment demonstrate the effectiveness of the proposed method.", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)The Paper is well-written, meanwhile, the method is intuitive and easy to understand.\n(2)The proposed method focused on Generalized Zero-Shot Learning (GZSL) to present Topology-Preserving Reservoir to finetune the pre-trained CLIP for better fit the distribution of seen and unseen classes, which seems reasonable.\n(3)Sufficient and significant experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "(1)The Dual-Space Feature Alignment proposed by the author, which uses a Cross Attention mechanism for cross-modal alignment, lacks innovation.\n(2)The author mentions \"attribute reservoir\" in the article, but essentially it is just a fully connected layer that generates different feature representations through various loss constraints. Additionally, in Figure 2, the attribute reservoir is shown in two states: frozen and trained. I am unsure about when these two states should transition between each other.\n(3)The idea proposed by the author to fine-tune feature distribution using spatial topological structures is intriguing. However, relying solely on the Pearson correlation coefficient to define a topology-preserving loss seems somewhat simplistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer k9s8\n### Response Q1-Q3\n\nThank you for the valuable comments and recognizing the various strengths of our paper: \n\"*well-written\", \"intuitive and easy to understand\", \"the method better fits the seen and unseen classes\", \"reasonable*\", and \"*sufficient and significant experiments\"*. \nWe address the raised concerns by the reviewer as follows. \n\n**Q1: Lack of innovation for the dual-space feature alignment module.**\n\n**A1:** We appreciate the opportunity to further clarify innovation for our dual-space feature alignment module. To the best of our knowledge, *there is no existing related work that employs a latent-attribute dual-space representation to address GZSL*. Most previous methods rely solely on a single latent space for multimodal feature alignment. We observe that these methods fail to effectively capture complex and fine-grained patterns, leading to suboptimal performance, especially for unseen classes (see Table 1 of the main paper). \n\nTo overcome these limitations, we propose to align multimodal features in a dual-space consisting of a latent space and an attribute space. The latent space provides a general representation of the input multimodal features, while the attribute space, induced by a predefined base attribute vocabulary and learnable attribute tokens, offers a more structured and interpretable representation. By aligning multimodal features within these two complementary spaces, our method not only captures both prior knowledge and task-specific information more effectively but also reduces the risk of overfitting to seen classes. This dual-space alignment design leads to superior performance compared to previous works.\n\nFinally, we would like to emphasize that cross-attention is a common practice for cross-modal alignment. How to *effectively align image and text modalities* is the key contribution of our work. Thus, we design a novel dual-space to align the multimodal image-text representations instead of projecting them into a single latent space.\n\n**Q2: Explanations for the attribute reservoir.**\n\n**A2:** Thank you for the comment. To the best of our knowledge, *our work is the first attempt to leverage an attribute reservoir to solve the description-based GZSL task*. By utilizing this reservoir, we are able to significantly improve performance through capturing complex and fine-grained patterns and facilitating alignment between image and description features.\n\nMore specifically, the attribute reservoir consists of two complementary components: the base attribute vocabulary and the learnable attribute tokens, which is *conceptually and technically distinctive from simply applying a fully-connected layer* to project features. \n- The **base attribute vocabulary** contains 5,996 extensive attributes collected from the existing literature related to attribute recognition. These attributes are universally shared across all datasets, which provide prior knowledge and effectively improve the generalization of the model to unseen classes.\n- The **learnable attribute tokens** are introduced to compensate for the cases where base attribute vocabulary cannot cover sufficient attributes to perform effective GZSL. Concretely, they have two functions: 1) learning complementary attribute knowledge that is missing in the base vocabulary; 2) integrate task-specific information into the attribute reservoir so as to better align the image and description features.\n\nExperimentally, the two components of the attribute reservoir are complementary to each other (Table 2 of the main paper) and together they achieve superior performance over the state-of-the-art. This demonstrates the effectiveness of our attribute reservoir. \n\n\n**Q3: The idea of using spatial topological structures is intriguing. Relying solely on the Pearson correlation coefficient to define a topology-preserving loss seems somewhat simplistic.**\n\n**A3:** Thank you for the helpful comment. In this work, our primary goal is to maintain the class topology structure of the CLIP embedding space, thereby improving the generalization ability of unseen classes. As shown in Table 3 of the main paper, we conducted various forms of topology-preserving losses, but we found that using the Pearson correlation coefficient to define topology-preserving loss is simple yet effective. Further empirical results show that it indeed improves the generalization ability of the model significantly (Table 1 and Fig. 1(a)).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yJGpWFdUIy", "reviewer": "Reviewer_8m4c", "summary": "This paper proposes a new task, \"generalized zero-shot learning (GZSL),\" in which both seen and unseen objects should be recognized for vision-language tasks. It also proposes a new method based on CLIP that uses the loss in the \"attribute space\" to perform better in both seen and unseen classes. This method is evaluated on various kinds of data sets and evaluated by the harmonic mean of the accuracies of seen and unseen classes.", "questions": "Strangely, the accuracy increases as the number of learnable tokes increases in Fig 4. (a) AwA2. It would be appreciated if you could provide any insights into this phenomenon.", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach using the attribute space seems novel enough, and its effectiveness was verified by a detailed comparison of the other methods and the well-designed ablation studies.", "weaknesses": "1) It is unclear what is learned in \"learnable attribute tokens.\" It is not so beneficial for unseen classes. It is unclear what information is represented as tokens for seen classes. It may be better to analyze the acquired tokens in more detail.\n\n2) It is difficult to think about the case that we have never seen an object, but we know its attributes quite well. In such a sense, \nI believe this method is more appropriate for few-shot learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 8m4c\n### Response Q1-Q3\n\nThank you for the positive and insightful comments. The reviewer appreciated *the novelty of our approach, the creation of the attribute space, the effectiveness of our method*, and *the well-designed ablation studies*. We address the mentioned concerns below.\n \n**Q1: Analysis of the learned attribute tokens in more detail.**\n\n**A1:** Thanks for the insightful comment. \nTo better analyze the learned attribute tokens, we attach a **PDF in the top-level comment** with two figures (Fig. R1 and Fig. R2). Our analysis is two-fold: \n- In Fig. R1 of the attached top-level PDF file, we visualized the correlation matrix of the base attribute vocabulary and the learned attribute tokens, respectively. We observe that the correlation of learned tokens is indeed different from that of base vocabulary. Specifically, many items in the base vocabulary are highly correlated, with an average value of **0.858**, indicating redundancy within the base vocabulary. In contrast, the correlation of learned tokens is much smaller, with an average value of only **0.003**, indicating that separable and independent tokens are learned. \n- In Fig. R2 of the attached top-level PDF file, we visualize the distribution of the base attribute vocabulary and the learnable attribute tokens using **t-SNE**. It can be seen that the learnable tokens (marked in orange) can **fill in part of the gaps** in the base vocabulary (marked in blue), indicating that the two represent different semantic meanings and are therefore complementary to each other. \n\n**Q2: It is difficult to think about the case that we have never seen an object, but we know its attributes quite well. In such a sense, I believe this method is more appropriate for few-shot learning.**\n\n**A2:** Thank you for the comment. \nIn the ZSL/GZSL setting, a general assumption is that for unseen classes, their prior semantic information such as attributes is known in advance, although there are no images from unseen classes for training. The overlapping attributes between seen and unseen classes can facilitate the knowledge learned on the seen classes to be transferred to the unseen for zero-shot recognition. \n\nAs per your suggestion, our framework can be readily extended to few-shot learning. It is important to note that few-shot learning differs from ZSL/GZSL in that it leverages a small number of labeled samples from unseen/novel classes. To accommodate this difference, we can align the multimodal representations of both the seen/base and unseen/novel classes within the dual-space. The model can then be trained using our multi-modal alignment loss (Eq. 5 and 6) plus the topology-preserving loss (Eq. 7) proposed in this work. In this setting, our model still inherits the good generalization ability of the CLIP model for effective few-shot learning. \n\n**Q3: Explanations for the accuracy increase in Fig.4(a).**\n\n**A3:** On the AwA2 dataset (Fig.4(a)), the performance of the unseen classes (i.e., $U$) improves as the number of learnable attribute tokens (i.e., $N_2$) increases. \n\nWe attribute this to the following reason: The AwA2 dataset is a coarse-grained dataset encompassing a wide range of animal categories. The base vocabulary may not comprehensively cover all essential attributes, leaving some important features unrepresented. This gap necessitates the addition of more learnable attribute tokens, which contribute to the observed increase in accuracy as $N_2$ grows.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Rebuttal 
We thank all reviewers for their insightful and positive feedback. We are encouraged that the reviewers acknowledge our paper:
- **Novel and impactful**. Reviewer BqNV -- "*introduces a novel research aspect*", "*a great contribution to VLM community*"; Reviewer TcJo -- "*the proposed model shows a significant impact*", "*idea and intuition are interesting*", "*TPR helps improve generalization*"; Reviewer k9s8 -- "*the method is intuitive*"; Reviewer 8m4c -- "seems novel enough".
- **Sufficient and significant experiments demonstrating the effectiveness of our method**. Reviewers reckon our experiments are *extensive* (BqNV), *wide-ranging* (TcJo), *sufficient and significant* (k9s8), and with *detailed comparison and well-designed ablation studies* (8m4c).
- **Well-organized, well-written, and easy to follow**
Reviewer BqNV -- "*This paper is well-organized and well-written, which makes it easy to follow.*" Reviewer k9s8 -- "*The paper is well-written, meanwhile, the method is intuitive and easy to understand.*" 

We thank all the reviewers' suggestions and comments that significantly improve the quality of our work, and we have done our best to address the reviewers' concerns.

Below, we summarize the key changes we have made for rebuttal:
- Perfromance of classic methods in VLM-based GZSL tasks (reviewer BqNV)
- Clarification of Class-Level description, comparison with standard annotation-based attributes methods, chosen of model variants, explanation of the performance on SUN dataset (reviewer TcJo)
- Clarification of key innovations regarding dual-space design, attribute reservoir, and topology-preserving loss (reviewer k9s8)
- Explanation of attribute reservoir (reviewer TcJo and k9s8)
- Provision of a summary of limitations (TcJo) 
- Visualization and Analysis of the learned attribute tokens (PDF attached, reviewer 8m4c)

Please find individual responses to your questions below.

Meta-review: This paper received an overall positive review score. The reviewers pointed out that the work has merits in motivation, performance, and presentation, and the point of contention is about the novelty of this paper. After reading the paper, reviews, and rebuttal, the AC agrees that the proposed dual-space alignment learning exhibits differences from the prior approaches and leads to superior performance. Due to the above considerations, the AC gives a recommendation for acceptance.
Decision: Accept (poster)