Learning Clustering-based Prototypes for Compositional Zero-Shot Learning
Authors: Hongyu Qu
Jianan Wei
Xiangbo Shu
Wenguan Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Compositional Zero-Shot Learning
Prototype Learning
Representation Disentanglement
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a clustering-based prototype mining framework for compositional zero-shot learning, which defines conceptual boundaries of primitives through a set of diversified prototypes, and automatically discovers these prototypes via clustering.

Abstract: Learning primitive (i.e., attribute and object) concepts from seen compositions is the primary challenge of Compositional Zero-Shot Learning (CZSL). Existing CZSL solutions typically rely on oversimplified data assumptions, e.g., modeling each primitive with a single centroid primitive presentation, ignoring the natural diversities of the attribute (resp. object) when coupled with different objects (resp. attribute). In this work, we develop ClusPro, a robust clustering-based prototype mining framework for CZSL that defines the conceptual boundaries of primitives through a set of diversified prototypes. Specifically, ClusPro conducts within-primitive clustering on the embedding space for automatically discovering and dynamically updating prototypes. To learn high-quality embeddings for discriminative prototype construction, ClusPro repaints a well-structured and independent primitive embedding space, ensuring intra-primitive separation and inter-primitive decorrelation through prototype-based contrastive learning and decorrelation learning. Moreover, ClusPro effectively performs prototype clustering in a non-parametric fashion without the introduction of additional learnable parameters or computational budget during testing. Experiments on three benchmarks demonstrate ClusPro outperforms various top-leading CZSL solutions under both closed-world and open-world settings. Our code is available at CLUSPRO.

Published as a conference paper at ICLR 2025
LEARNING CLUSTERING-BASED PROTOTYPES FOR COMPOSITIONAL ZERO-SHOT LEARNING
Hongyu Qu Jianan Wei Xiangbo Shu Wenguan Wang
Nanjing University of Science and Technology "Zhejinng University NNational Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi' an Jiaotong University
ABSTRACT
Learning primitive (i.e., attribute and object) concepts from seen compositions is the primary challenge of Compositional Zero-Shot Learning (CZSL). Existing CZSL solutions typically rely on oversimplified data assumptions, e-g., model- ing each primitive with a single centroid primitive representation, ignoring the natural diversities of the attribute (resp. object) when coupled with different ob- jects (resp. attribute). In this work, we develop CLUSPRO, a robust clustering- based prototype mining framework for CZSL that defines the conceptual bound- aries of primitives through set of diversified prototypes. Specifically, CLUSPRO conducts within trthin-primitive clustering on the embedding space for automatically discovering and dynamically updating prototypes. These representative proto- types are subsequently used to repaint a well-structured and independent primitive embedding space, ensuring intra-primitive separation and inter-primitive decor- relation through prototype-based contrastive learning and decorrelation learn- ing. Moreover, CLUSPRO efficiently performs prototype clustering parametric fashion without the introduction of additional learnable parameters or computational budget during testing. Experiments on three benchmarks demon- strate CLUSPRO outperforms various top-leading CZSL solutions under both closed-world and open-world settings. Our code is available CLUSPRO.
non-
INTRODUCTION
Humans possess the unique ability to recognize potentially infinite number of novel combi- nations by associating known components [1], i.., make "infinite use of finite means" [2]; for instance, despite never having seen one, people easily imagine unicorn by bining their concept of a horse with the idea of single horn. Inspired by such composi- tional generalization ability of human intelli- gence [3. 4], Compositional Zero-Shot Learn- ing (CZSL) [5, 6, 7, 8, 9] is proposed, aiming to recognize unseen attribute-object compositions based on learned knowledge from seen ones.
Tiger
cat
old cat
wet
(a)
(b)
O0 features
prototypes
+ primitives
Figure I: (a) Previous CZSL methods model all ples each primitive concept with only one centroid primitive presentation. neglecting feature divergence within each primitive when involved in different com- positions. (b) Our method represents each primitive as a of prototypes capture primitive diversities.
Existing CZSL solutions [10, 11, 12, 13] typi- cally achieve compositional learning by align- ing the visual representation from pre-trained image encoder backbone with the attribute-object textual representation derived from pre-trained word embeddings, Rather than learning to align visual and textual representation from scratch, re- cent approaches [14, 15, 16, 17, 18] have pivoted towards leveraging large-scale pre-trained vision-
"Equal contribution "Corresonndigg author

---

Published as a conference paper at ICLR 2025
language models (e.g., CLIP [19]) by treating compositional labels learnable tokens in pre- defined prompt like "a photo of [attribute] [object]". Though impressive, these methods exhibit two limitations: First, they struggle to learn visual concepts by modeling an "ideal" primitive (i.e., at- tribute and object), but ignore an essential issue: each visual concept (i.e., attribute-object pairs) can be semantically similar but visually different. For example, the attribute "broken" combined with "cord" typically signifies disconnection, but conveys the notion of rugged landscape when applied to "valley". Thus, we argue that a single centroid primitive representation exhibits limited tolerance to intra- intra-priminn primitive variance (Fig. la), and it is essential to incorporate more exemplars to capture the natural diversities of primitive. Second, they endeavor to explore more effective disentangling strategies (e.g., contrastive learning [20], knowledge distillation [21] or graph representation learn- ing (22]) to achieve independent primitive modeling a multi-branch manner, but typically present representation disentanglement from a local view (i.e., a few images within a batch [20, 23] or small training subset [24D, thus failing to perceive underlying data distribution in the entire dataset.
In light of the above, we present CLUSPRO, clustering-based prototype mining framework for CZSL (Fig. lb). Specifically, we propose to describe each primitive by abstracting it through a set of representative prototypes, which are automatically discovered by performing within-primitivy clus- tering the visual representation. Based on these prototypes established across the entire dataset, we introduce two complementary self-supervised learning strategies to repaint the attribute and ob- ject embedding spaces, prompting intra-primitive separation and inter-primitivy decorrelation.
Specifically, CLUSPRO employs two disentangling adapters to project visual representation, ex- tracted from pre-trained image encoder, into separate attribute and object embedding spaces. Then, each primitive is described by clustering K centroids on primitive-wist features. This process (i.e., Local-aware Prototype Assignment) involves assigning the visual feature of each primitive one of a set of prototypes that share the same attribute or object category, while considering the in- trinsic coherence of the feature distribution. For computational efficiency, we opt for Generalized Conditional Gradient (GCG) algorithm [25] to enable fast prototype assignment. Additionally, to keep non-learnable prototypes up-to-date, we employ a dynamic Prototype Updating mechanism, which recomputes prototypes over the entire dataset in each iteration. The attribute embedding and object embedding, derived from the same visual representation with compositional semantics, in- herently exhibit entanglement, which is toxic for prototype construction within primitive. To learn well-structured and independent attribute/object embedding space, we propose two complementary metric learning mechanisms: i) Prototype- based Contrastive Learning aims to encourage each prim- itive feature to be similar to its assigned prototype and dissimilar with all other prototypes from the attribute and object branch. In this way, our model can not only capture intra-primitive discrimina- tiveness within the group of attributes or objects, but also promote a clear distinction between at- tributes and objects. ii) Prototype-based Decorrelation Learning is devised to shape an independent primitive embedding space (i.e., object representation should be invariant to attribute alterations, relations between attributes and objects.
and vice versa) by exploring conditional-indeeeene
CLUSPRO has several appealing merits: First, comprehensive modeling of data distribution: By conducting within- primitive clustering on the visual embedding space across the entire dataset, CLUSPRO can automatically mine the global data distribution of each primitive from a holistic view. Second, explicit supervision of representation disentanglement: The clustering- based prototypes enable CLUSPRO to directly shape well-structured yet independent attribute and object embedding spaces via prototype-anchored contrastive learning and decorrelation learning. Such improved prim- itive embedding spaces, in turn, enable typical pithin-primitive variation pattern mining. Third, high efficiency: CLUSPRO perform prototype clustering in a non- non-parametric fashion without any modification of network architecture or additional computational budget during testing.
To effectively assess our method, we conduct extensive experiments on three gold-standan CZSL datasets (i.e., MIT-States [26], UT-Zappos [27], and C-GQA [28]). Experimental results demon- strate that CLUSPRO significantly exceeds existing -tate-of-the-arts in both Close-world (CW) and Open-world (OW) settings ($4.3). Concretely, the CW setting, CLUSPRO achieves +11.8% % and +20.2% % AUC gains on UT-Zappos and C-GQA, respectively. On the OW settings, it also yields solid improvements of + +1..7% % AUC UT-Zappos and +11.1 % AUC on C-GQA. In $4.4, a set of ablative studies confirms the power of our idea and the efficacy of core model designs.
GGiven that CLIP might be exposed to certain unseen compositions during pre-training, provide detailed data overlap discussion in SG of Appendix.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Compositional Zero-shot Learning (CZSL). The goal of CZSL is to recognize unseen attribute- object compositions by combining learned concept knowledge from seen pairs. Early CZSL solu- tions can be summarized into two paradigms: the first paradigm [29, 28, 5, 30, 31, 32, 33] directly compose attributes and objects with a transformation function and learn classifier for recognition; the second paradigm [23, 34, 20, 22, 35, 36, 21] mainly decomposes attribute and object in the composition space by well-designed disentangling strategies, e.g., contrastive learning [20]. knowl- edge distillation [21] or graph representation learning [22], and employ two separate classifiers to identify attributes and objects individually. Recent breakthroughs in Vision-Language Models (VLM) [37, 38, 19] make it a promising direction to hamess knowledge from pre-trained VLM (e.g., CLIP [19]) for zero-shot and open-vocabulary tasks. Pioneer works [17, 18, 16, 39] build learnable soft prompts with a combined attribute and object vector representation. To capture the contextual nuances in the composition space, recent works [14, 15, 24] jointly model the attribute, object, and composition through llnnonel alignments in multiple identification branches.
Despite these advancements, they generally focus on learning one single representative prototype to model each primitive. This limits their ability to interpret the complex and subtle meanings that arise from the combination of various visual concepts. Besides, these methods primarily focus on disentangling attributes and objects with a restricted set of samples, neglecting the potential of incorporating global information to reshape a well-structured and independent embedding space.
Prototype Learning. Studies cognitive psychology evidence that people often explore proto- typical knowledge foundation for learning and problem-solving across various domains, such as natural language understanding and visual scene understanding [40, 41]. Unlike Softmax-base methods [42, 43, 44], prototype-basee classifiers [45, 46, 47, 48] make decisions by computing the distance between new observations and prototype representations of each class. The prototypes typically refer to the centroids of all samples belonging the same category [49]. For its exemplar- driven nature, spectrum of recent works attempts to combine deep learning techniques and the idea of prototype learning, boosting great potential various learning paradigms, including supervised learning (50, 51, 52, 53, 54, 55], few-shot learning [56, 57], and (compositional) zero-shot learning [58, 22, 10]. These (compositional) zero-shot learning works [59, 60, 61, 58] extensively explore prototype learning to enhance feature representation. However, they typically model each class with only one prototype, and their prototypes are often learnable parameters.
Building upon these successes, we aim to advance CZSL by developing learning scheme. Different from previous works [20, 22], which employ one single learnable proto- type for each primitive, CLUSPRO explicitly derives prototypes via clustering primitive features over the entire dataset, which are subsequently used to repaint attribute and object embedding spaces.
cluster-based prototype
Self-supervised Representation Learning. Self-supervised representation learning (SSRL) meth- ods [62, 63, 64] aim to construct a well-structured embedding space without requiring extensively annotated datasets. Recently, metric learning [65] has emerged prominent technique in SSRL, which learns a distance function to reflect the relationships between data points based on their labels. This approach results in more compact, interpretable, and versatile feature representations, which could benefit subsequent tasks, e.g., classification [66, 67, 68] or clustering [69, 70]. It aligns well with CLUSPRO that seeks automatically discover prototypes of primitive concepts by clustering features associated with coarse grainne labels. Inspired by this, we raise a disentangled representa- tion learning strategy that integrates two complementary self-supervised learning strategies to shape a primitive embedding space with intra- primitveee separation and inter- teiim-iii decorrelation.
METHODOLOGGY
3.1
PROBLEM STATEMENT
Given the attribute set A {ai, .2....,aMf and the object set O = (01, (0.,02....,.00 ,N). the com- positional label set C can be defined as the Cartesian product between A and O, i.e., C A x O. Subsequently, C is divided into two disjoint subsets: the seen composition set C* and the unseen com- position set C", where C* n C = Ø. During training, the model can only access images paired with labels from the seen composition set C* i.e,, the training set is defined as T {(xx c)J e X, c C*).
3

---

Published as a conference paper at ICLR 2025
(b) features (JM) M
clusters
protolypos (P2) K
hg Image Engoder Phis ho
Ey (7)
Eg (9)
trannin samples
local-aware protatype assignment
protobype updating
protobpe- based
prototype-based decorrelation leariing
object
$o
contrast learning Erer
Text Encoder (Pici
pair
5
Ert
Bq. (11)
ASIC
attribute
5a
Figure 2: The overview of CLUSPRO. (a) CLUSPRO is built upon three- path paradigm to jointly recognize attribute, object, and attribute-object composition ($3.2). (b) To capture the diversity within each primitive, ClusPro describes each primitive with set prototypes, and conducts withi-prriittiv- clustering across training data for prototype assignment and updating ($3.3). (c) CLUSPRO imposes two constraints based on these constructed prototypes to promote intra-primitive separation and inter-primitive decorrelation ($3.4).
where X' is the visual space. In the Closed- Worldd (CW) setting, the composition space for testing is defined as C* = C CC", where only the known composition space is required. For the Open-World (OW) setting, the composition space considers all potential attribute-object pairs, i.e., C C.
3.2 BASELINE ARCHITECTURE
Encoding Visual Representations, Our framework is built upon a three three-path paradigm [14,2] 23, 34], which jointly recognizes three kinds of semantic components, i.e., attribute, object, and attribute- object composition. Given an input image X <RxW>x we adopt a visual encoder & of CLIP [14] to obtain visual representation f TR.D. We consider image representation f composition visual representation f and adopt attribute adapter it and object adapter ht" [71, 72], each implemented as separate MLP, to project f into the discriminative attribute and object spaces respectively:
S 1 (f), f = (f), f" f.
(1)
where f and f are visual features extracted for attribute and object, respectively.
Encoding Prompt Representations. Following existing CZSL [18, 14], we construct prompt repre- sentation via soft learnable prompt strategy for all candidate compositions, attributes, and objects. Specifically, for each attribute-object composition Ct,j 0j). we create three prompts for each branch, i.e., attribute prompt S. = [sf. v object prompt Sj [ 8f, vf]. and com- position prompt S [ ... sf, vft Dg], where si. si. and i learnable pretix contexts initialized by "a photo of Additionally, of and of are trainable vocabulary tokens for the attribute a, and object 0j. respectively. These prompts are then fed into frozen text encoder GM of CLIP to obtain corresponding prompt features, formulated as:
t = (S")))
ty (8)",, = (S
Three-path Learning Objective. Given visual and prompt representations from three branches, we compute the probabilities for attribute, object, and composition classes, denoted (a.)f.) p( ((,I5n) p(e fn), respectively. To recognize primitive concepts and their compositions each
branch, three cross-entropy loss functions are employed:
exp(S² (T/TT)
L
N En logp(alf.., p(a)ff))
(3)
El exp( +t/T)² exp(Fi -t5/T)
L" 1 Σn logr(olf.) p(o,(fn)=
(4)
N
El exp (R tg/T)
c 1 Σw log.( p(c.)f))) N
exp(Fi ///
(5)
En exp(fi +(/T)
where E IR is pre-defined temperature parameter CLIP. For simplicity, all the features are 4 normalized by default. Then, the three-path classification loss can be formulated as:
BAS Xx L +S
(6)
where X", No, X are all set to 1, following [14].
4

---

Published as a conference paper at ICLR 2025
Our Main Idea. Though impressive, this three-branch paradigm only achieves implicit feature disentanglement to a limited extent by using one single image, failing to perceive the potential structures of the whole dataset. Moreover, it only considers an isolated centroid for each primi- tive, ignoring rich and diverse intra-primitive patterns. To address this limitation, we propose a clustering-based prototype mining framework (i.e., CLUSPPO)), as shown in Fig. 2. Our model not only learns primitive recognition with pre-given semantic labels, but also automatically discovers diverse and Sine-grained sub-primitive patterns across the entire dataset. For training, our algorithm alternates between two steps: i) perform primitive-wist online clustering to discover sub-primitiv prototypes ($3.3); ii) impose two prototype-anchored constraints to explicitly shape well-structured and independent attribute/object feature spaces($3.4) The improved features, in turn, facilitate more reliable primitive-wise clustering, and eventually boost composition predictions.
3.3 CLUSTERING-BASEE BASED PROTOTYPE MINING
To model the natural diversities of primitives, we exploit rich dataset-level context knowledge to automatically identify informative prototypes within each attribute or object, facilitating primitive concept representation learning. Specifically, we first assign each attribute (resp. object) visual feature to the prototypes belonging to the same attribute (resp. object) (i.e., Local-aware Prototype Assignment), and then continuously update prototypes online according to the assignments (i.e., Prototype Updating) with batch training. Such an online clustering strategy forces the model to mine putra-primitiee discriminativeness. Notably, we present the online primitive-wise clustering process within both the attribute and object embedding spaces, diverse patterns within each primitive. For clarity, we only explain the prototype construction in the attribute branch, while the object branch follows the same process.
to well represent rich and
Local-aware Prototype Assignment. For each attribute a E A. we leverage K prototypes (pg))-))
K to represent its diverse semantic patterns, where Pt is k-th prototypes of attribute To get informative yet hidden prototypes, we perform clustering within each attribute on the attribute embedding space. More specifically, for given set of attribute features F { y IR D*N" associated with attribute a, where f% is n-th attribute features of attribute a and is the number of attribute features, our goal is to assign these attribute features to the K prototypes p (p=(p]) k= DD=K The mapping matrix from F to Pa can be denoted as L [4"] E (0, KxN" where I e {0,1)K is an one-hot assignment vector of n-th attribute features over K prototypes. Let S RA"""""" denote cosine similarity among these attribute features Fa e RR embedding space. Thus, the clustering within each attribute can be achieved by the optimization of the assignment matrix L", i.e., maximizing the similarity Q" between attribute features F" and the prototypes pa (i.e., Q" Softmax (acinaxxxPTTaaa (aa FT) E KRN"), ) while considering intrinsic coherence
the attribute
structure of features:
min (L" - log Q") R(((L"), t
where (.) is the Frobenius dot-product. Note that M(L (S" (L' ('''''((('''")) local coherent regularized term [73], and x > 0 is the strength of the regularization, where @ de- notes element-wise multiplication. Different from the classical formulation in [50, 74], i.e., Optimal Transport with entropic constraints, our local-aware prototype assignment can produce superior as- signments by fully considering the intrinsic coherence structure of attribute feature distribution, iee. intra-distribution coherence. Specifically, this term promotes assigning higher weights to L and L if the i-th and j-th attribute feature are highly similar (indicated by a high value of 5 and both exhibit strong similarity, as measured by Q and Qi to the k-th prototype of attribute a.
Q*)) is
As in 169, 75, 76), we relax L to an element of transportation polytopes, i.e., L R RK*N" Un- like offline clustering [77, 78] requiring multiple passes over the entire dataset, we cast prototype assignment as an optimal transport problem, so as to scale our algorithm to massive data by online clustering:
-= (L" E K*N" IL = L 1N1
No
1x),
(8)
K
where 1x denotes the vector of all ones in dimension K. L ensuring each attribute feature is assigned to exactly one prototype, and L K 1, is the cquipartition constraint, guarantecing that, on average, each prototype is selected an equal number of
is the assignment constraint Nº
Forr clarity, we slightly reuse and to define certain attribute and object concept, respectively.
5

---

Published as a conference paper at ICLR 2025
times in the batch. With differentiable regularized term and soft assignment relaxation, the solution of Prob. (8) can be given by efficient GCG algorithm [25], which relies on a few matrix-vector multiplications via iterative Dykstra algorithm [79].
Prototype Updating. During iterative network training, primitive representations evolve continu- ously, necessitating offline clustering to recompute sub-primitive prototypes over the entire dataset after each batch, which incurs substantial computational costs. To address this, we propose an online clustering approach with momentum updates, where prototypes are dynamically updated using the embeddings within the current mini-batch. In particular, after each training iteration, each prototype Pž of the attribute a is updated as:
PE HPE + (1-
where uee [0. 1] is a momentum coefficient, and f R.D is the mean vector of the attribute fea- tures assigned to the prototype P by clustering. As such, the prototype updating scheme (Eq. 9) iteratively refines the prototype values in response to the evolving primitive feature representations, thereby facilitating smoother training process. This online clustering strategy enables our model to effectively discover rich brbbiiiiiiive patterns over massive training data.
3.4 PROTOTYPE-ANCHOREE PRIMITIVE REPRESENTATION LEARNING
By performing online within-primitive clustering separately in the attribute and object embedding spaces, we construct a set of prototypes for each attribute, i.e., {p() k-1 and each object, i.e., [p&]E= to represent diverse sub-primitive patterns. Therefore, the following question naturally arises: what should a well-structured and independent embedding with discriminative prototypes be like? To answer this, we enhance the three-path classification loss (Eq. 6) by incorporating two com- plementary loss constraints based on these constructed prototypes: Prototype-based Contrastive Learning and Prototype-based Decorrelation Learning, which fully exploits the relationships between primitive features and sub-primitive centers in the embedding space.
Prototype-based Contrastive Learning. Our prototype-based contrastive learning strategy con- trasts the similarities between each primitive feature, i.e., fn, E F" u Fo, where fr is 1-th primitive features, i.e., Pa u P This strategy encourages each primitive feature fn to be similar to its as- signed prototype p. and dissimilar all other K(M + +N)- irrelevant prototypes P_. Different from only using K (M/+ + N-1) irrelevant prototypes from other primitives as negative samples, our strategy not only ensures inter-primitive separation to some extent, but also guarantee intra-primitive separation. The corresponding training objective for each features fn. is defined as:
exp(fn PP+/T)
LrC = log exp(Fn p-/7)+ +//T))>Epeppp Ep_ep_exp(fi pp-/7)
(10)
where >> 0 is temperature hypee-paramettr. Notably, we treat both attributes and objects equally as primitives to increase the scale and diversity of negative samples.
Our prototype-based contrastive learning exhibits two primary advantages: Traditional contrastive learning approaches often rely on sophisticated negative sampling strategies to form contrasting pairs, but inevitably yield negative pairs that share similar semantic meaning and should be closer in the embedding space. In contrast, CLUSPRO avoids this long-standing challenge by construct- ing positive and negative pairs using clustering-based representative prototypes. thus effectively shaping the embedding space by leveraging dataset-level contextual knowledge. @ Unlike previous contrastive learning-basee
CZSL models [20, 80], which necessitate the extra processing for posi- tive and negative feature extraction, our model leverages already constructed prototypes for contrast computation, without incurring extra computational and storage budget.
Prototype-based Decorrelation Learning. By leveraging prototype-based contrastive learning, CLUSPRO can effectively distinguish primitive prototypes by maximizing their distance to enhance the independence of linear relationships. But, in CZSL, it is also essential to maintain independence between the embeddings of attributes and objects, iee, inter-primitive decorrelation; for instance, apple should be distinguished from specific attributes, no matter whether apple is red or green. decorrelation learning to enforce a distinct separation between attribute and object prototypes, ensuring promising disentanglement results. Based on constructed primitive prototypes, the conditional iiattiional-iddeeedenneee relations can be captured by the following prop- erties: i) f 1l p" and ii) fo p", where denotes the independence between samples. Here, f and f are disentangled attribute and object features, respectively, with pd and p" representing the corresponding sub-primitive prototypes entangled with these features.
Thus, we propose a prototype-

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative res.lts(34.3) MIT-States [26]. UT-Zappos [27] and C-GQA [28] within CW setting.
Gloond
Backbome
MIT--State
UT-Zuppos
C-QQA
Method
Secnt
Unscent
HM
AUC:
Sccut
Unsecnt
HM1
AUC1
Secnl
Unseenf
HMT
AUCT
VIT-I
30.2
46.0
11.0
49.1
25.0
CoOp [82 pccarmn
VIT-
34,4
47.6
29.8
13.5
52.1
49.3
34.6
18.8
20.5
26.8
17.1
4,4
PCVL [39 Aocd202)
VIT-L
48.5
47.2
35.3
18.3
64.4
64.0
46.1
32.2
CSP
VIT-L
46.6
36.3
19.4
64.2
66.2
33.0
28.8
26.8
20.5
DFSPG20 [18] cVPSS19
VIT-L
47.4
5244
37.2
20.7
64.2
66.4
45.1
32.1
35.6
29.3
24.3
DFSP [15] cVP2222
VIT-L
47.1
$2.8
37.7
20.8
63.3
69.2
47.1
33.5
36.5
32.0
26.2
9.9
DFSP(20) [18
VIT-L
46.9
52.0
37.3
20.6
66.7
71.7
47.2
36.0
38.2
32.0
27.1
10.5
GIPCOL [8] soCVIEN
VIT-L
48.5
49.6
19,9
65.0
68.3
48.8
36.2
31.9
28.4
22.5
CDS-CZSL [15 cV112204
VIT-L
50.3
52.9
39.2
22.4
63.9
74,8
$2.7
39.5
38.3
34.2
28,1
11.1
Troika [14] c110a
VIT-L
49.0
53.0
39.3
22.1
73.8
54.6
41.7
41.0
35.7
29.4
12.4
PLJD (evvenn
WIF-I
49.7
22.1
67.3
68.8
38.7
38.8
33.0
27,9
11.0
CLUSPRO (Ours)
VIT-L
52.1.0s
54.0.03
40.7.83
23.8.a2
70.7.10
76.0A12
58.5
46.6.0
44.3
37.8.3
328.02
1499A1
Specifically, we minimize the correlation between attribute and object embedding spaces by using the Hilhert-Schmidd Independence Criterion (HSIC) [81]. HSIC is a non-parametric, kernel-based statistical measure that evaluates the independence between two continuous random variables, yield- ing a value of zero if and only if the two variables statistically independent in the unfinit-ssaplle limit. Thus, our prototype- based decorrelation learning strategy can be achieved by minimizing:
-TH HS .p") HSIC(
(11)
A similar approach also adopted by [36]: however, our decorrelation strategy benefits from the use of prototypes, which capture the intrinsic characteristics of primitives, thus more effectively disentangling attribute features and object features in the compositional space.
Overall Training Objective. The final learning target of CLUSPRO combines the three-path clas- sification loss CBSS (Eq. 6) with prototype- loss constraints CPCL (Eq. 10) and LPDL (Eq. 11):
C= + PDL
(12)
where the coefficients and 3 are empirically set: a= 0.2, 300.5.
Inference for CSZL. During testing, the test image is fed into CLUSPRO obtain prediction scores for attribute p( ((al.)), object p(o,(ar), and composition prediction y (z). Then the final composition class can be predicted by incorporating three branch prediction results:
è argmax p(cs,l==) + p(a:(z) p(oglz)).
(13)
4 EXPERIMENT
4.1 EXPERIMENTAL. SETUP
Datasets. We conduct experiments on three widely-used CZSL benchmarks: MIT-States [26], UT- Zappos [27], and C-GQA [28]. MIT-States consists of 53, 753 natural images in total, with 115 states and 245 objects. UT-Zappos contains 50, 025 fine- fine-grain shoe images with 16 states, 12 objects and 116 state-object compositions. C-GQA is the most extensive CZSL dataset, containing 453 states and 870 objects for 39, 298 images in total and over 9, 500 state-object compositions. More details provided in Table (cf. $A in Appendix).
Evaluation Metric. Following the official evaluation protocol [28, 30, 15, 14], four metrics are adopted for evaluation, i.e., best-Seen accuracy (Seen), best-Unseen accuracy (Unseen), best Har- monic Mean (HM), and Area Under the Curve (AUC). Among them, AUC is the priority as it evaluates the model comprehensively. Please see [84, 30] for full details about metries.
4.2 IMPLEMENTATION DETAILS
Network Architecture. CLUSPRO adopts pre-trained CLIP VIT-L/14 model [19], serving as the image and text encoder. The adapters of attribute ha and object h² are implemented by two indi- vidual MLPs. We group the features of each primitive into K prototypes to describe intra-primitive diversity. The number of prototypes K and the momentum coefficient μ in Eq. 9 are empirically set to and 0.99, respectively (ablation study in Table 4a and 4b). We follow [73] set k= in Eq. 7.
Training. CLUSPRO is trained end-to-end for 15 epochs with Adam optimizer [85]. To manage the learning rate, we initialize it at 1e-4 for all datasets and set weight decay to 5e-5. The coefficients

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results($4.3) MIT-States [26], UT-Zappos [27] and C-GQA [28] within OW setting
Open-Woondd
Metbod
Hackhone
Seen
Unseent
MIT-States
HM
AUC
Seent
Unseen!
UT-Zappos
1MM1
AUC1
Seent
Unseen!
C-GQA
HM1
AUC1
CLIP
ViT-I.
36.1
20.6
11.2
2.2
0.3
CoOp
[82] UCY23221
VII-L
34.6
9.3
12.3
2.8
52.1
31.5
28.9
13.2
21.0
4.6
5.5
0.7
PCVL [39] koodson
VIT-L
48.5
16.0
17.7
6.1
64.6
44.0
37.1
21.6
CSP
VIT-L
46.3
15.7
17.
5.7
64.1
44.1
38.9
22.7
DESPG20
[18) CVPIDNC
VII-L
47.2
18.2
19.1
6.7
64.3
41.2
26.4
35.6
6.5
9.0
2,0
DFSPYBIF) [16
VIT-L
47.1
18.1
19.2
6.7
63.5
$7.2
42.7
27.6
36.4
7.6
10.6
244
DFSP
VIT-L
47.5
18.5
6.8
44.0
30.3
38.3
7.2
10.4
GIPCOL [83 MACYIEM
VII-L
48.5
16.0
17.9
40.1
23.5
CDS-CZSL [15
EVE3DGE
VIT-L
49.4
21,8
22.1
8.5
61.7
61.3
48.2
32.3
37.6
8.2
11.6
2.7
Troaka [14]
VIT-L
48.8
18.7
20.1
7.2
61.2
47.8
33.0
40.8
7.9
10.9
2.7
PI.I12 (16eemn
VIT-I
49.1
18.7
20.0
67.6
55.5
30.8
7.5
CLUSPRO (Ours)
VITL
512,00
22.1
23.0m0
9,3
710.0
66.2.
$4.1
39.5.l
41.6a3
8.30a
IL6.30 3.0.ar
and 3 in overall training objective (Eq. 12) are empirically set to 0.2 and 0.5, respectively (related experiments in $D of Appendix). Eq. 10, the temperature parameter T is maintained at 0.1. CLUSPRO is implemented in PyTorch and trained on one NVIDIA RTX 4090 GPU.
Testing, Following previous works [17, 83]. we apply the post-training calibration to filter out in- feasible compositions in the open-world setting during testing. Note that, during model deployment, there is no any network architectural modification or extra inference cost introduced to the base model. The primitive prototypes, Pa and po, are directly discarded after network training.
4.3 COMPARISON STATE-OF-THE-ARIS
In this section, we compare our method CLUSPRO with top-leading CZSL solutions on three dataset (i.e., MIT-Staates [26], UT-Zappos [27], and C-GQA [28]) in CW and OW settings.
Performance on CW Setting, As summarized in Table 1, our approach CLUSPRO outperforms recent state-of-the-art (SOTA) CZSL algorithms across all datasets [26, 27, 28] on CW setting. Concretely, in terms of AUC which is the priority metric for evaluating the model comprehensively, CLUSPRO yields +1.4, +4.9, and +2.5 AUC score gains compared with SOTA methods on MIT- States, Zappop,, and C-GQA, respectively. Besides, CLUSPRO boosts HM to 40.7 (+3.6%) on MIT-States, 58.5 (+7.1% %) on UT-Zappos, and 32.8 (+11.6%) on C-GQA. Moreover, CLUSPRO earns consistent best Seen Accuracy (Seen) and Unseen Accuracy (Unseen) improvement. These consistency improvements are attributed to the fact that our algorithm captures diverse sub- sub-primitive patterns, i.., intra- primi-primitive variations, which improves generalization unseen compositions.
Performance on OW Setting. Table 2 reports comparison results on OW setting. As seen, most CZSL methods suffer substantial performance drop due to vast search space in OW setting. In contrast, our method CLUSPRO still surpasses all published competitors across three datasets [26, 27, 28]. In particular, CLUSPRO attains the highest AUC scores: 9.3 (+9.4%) on MIT-States, 39.5 (+19.7%) on UT-Zappos, and 3.0 (+11.1% %) on C-GQA. In terms of HM, best Seen Accuracy (Seen) and Unseen Accuracy (Unseen), CLUSPRO still achieves the best results. This reinforces our belief that learning group of discriminative prototypes for each primitive helps our model to recognize unseen compositions, even within challenging OW setting.
4.4 DIAGNOSTIC EXPERIMENT
To evaluate our algorithm designs and gain further insights, we conduct ablation studies on UT- Zappos [26] and C-GQA [28] in CW settings.
Key Component Analysis. We first investigate the effectiveness of our core idea, i.e., clustering- based prototype learning. make use of discovered rich sub-primitive prototypes to shape attribute and object embedding spaces, two key training objectives are proposed, i.e., Prototype-basee Con- trast CPCL (Eq. 10) and Decorrelation CPDL (Eq. 11). As shown in Table 3, we build BASELINE that trains in the three-branch paradigm, without primiin-rimitive prototype clustering (i.e., prototype assignment and updating). We can find that, adding CCL or CPDL individually leads to a substamtial performance gain, e.g., +4.9/+2.8 AUC UT-Zappos [26], and +2.4/+1.9 AUC C-GQA [28]. This verifies the efficacy of explicitly promoting inter-primitive prototype separation and attribute- object independence. Last, by combining PCLL and PPII. full model yields the best results.
8

---

Published as a conference paper at ICLR 2025
Table 3: Analysis core components ($4.4) on UT-Zappos [27] and C-GQA [28] within CW setting.
Lia.
Zenn
UT-Zappos
C-GQA
(Eq. 10)
(Eq. (1)
Seent
Unseent
HM
AUC1
Scent
Unseent
HM*
AUC1
BASELINE (w/o Clustering)
66.2
74.6
54.1
41.0
40.5
33.4
29.7
11.8
Prototype-bsaeu Contrust
69.9
74.7
57.1
45.9
43.6
36.7
32.1
14.2
Prototype-bsaed Decorcelation
67.6
75.2
56.5
43.8
43.1
36.1
3144
13.7
Contrust Decorrelation
70.7
76.0
58.5
46.6
44.3
37.8
32.8
14,9
Table 4: A set of ablation studies on UT-Zappos [27] ($4.4).
UT-Zappos
U'I-Zappos
Prototype
Socnt
Unseent
HMr
AUC1
Coefficient μ
Scent
Unseenf
HMt
AUC1
68.9
73.6
55.2
42.8
67.2
74.1
54.9
42.6
K
70.6
74.6
56.9
45.1
= 0.5
69.8
75.1
56.0
43.3
K
70.7
76.0
46.6
==
70.5
74.9
58.1
46.0
K 10
69.9
75.4
58.3
46.0
0.99
70.7
76.0
58.5
46.6
K=
70.1
75.2
58.0
45.9
74.7
57.0
45.1
(a) Per- Der-primitive Prototype Number
(b) Prototype Updating Coefficient μ
Clustering Branch Object
UT
UTpZaapps
UT-Zaapps
Clustering Strategy
Attrbbte
Soent
Unseent
HMt
AUC1
Soent
Unseeat
HMf
AUC1
66.2
71.6
51.1
11.0
None
66.2
74.6
54.1
41.0
69.7
74.3
44.3
Cosine Similarity
74.7
57.7
69.5
73.6
56.4
41.1
Classical OT
70.1
75.2
58.2
45.8
70.7
76.0
58.5
46.6
Ours
70.7
76.0
58.5
46.6
(c) Clustering Branch
(d) Clustering Strategy
Prototype Number Per Primitive K. We next investigate the impact of the prototype number per primitive. The results are reported in Table 4a. Note that for K = 1, each primitive is directly represented by the mean embedding of primitive features in the current batch without prototype clustering. As shown in Table 4a, this baseline yields the HM score of 55.2 and AUC score of 42.8 on UT-Zappos [26], respectively. When representing one primitive concept with a group of prototypes, we observe that our model CLUSPRO gains stable improvements (i.e., HM: 55.2--8.55, AUC: 42.8-+4.6) as the number of prototypes grows (i.e., K = 5). This supports our hypothesis that leveraging set of diversified prototypes to describe primitive concept can capture diverse intra- primitive patterns. However, too many prototypes above K results negative gains. This may be because CLUSPRO suffers from insignificant sub primiitiiiee patterns produced by over-clustering.
Momentum Coefficient μr. Table 4b probes the impact of momentum coefficient μ (Eq.9), which controls the speed of primitive prototype updating. We performs better with a relatively large coefficient (i.e., μ = 0.99), verifying that slow updating is beneficial, but not too slow (i.e., μ 0.999). When μ is too small, the performance decreases. In particular, our algorithm encounters a large decrease at the extreme of no momentum (i.e., 0).
clearly observe that, algorithm
Multi-branch Clustering. In Table 4c, we study the impact of attribute and object clustering branches by removing one or more specific branches. Removing one branch clustering means that, discard primitive-wise clustering to mine sub-primitive patterns in the corresponding branch, and remove the corresponding training objectives from Eq. 10 and 11. As shown in Table 4c, us- ing attribute or object clustering branches individually only yields limited performance gains, e-g., +3.3/+3.1 AUC score. By unifying the two clustering branches together, our full model achieves the best performance across four metries, confirming their complementarity.
Clustering Strategy. We examine the impact of our proposed local-aware clustering strategy (ef. Eq.7) by contrasting it with the model without primitive-wise clustering, the cosine similarity updating [86], and classical Optional Transport (OT) [87, 88]. As shown in Table 4d, our local- aware clustering strategy proves to be more effective: it outperforms the model without clustering, the cosine similarity, and classical OT across all metrics, e.g., +4.7, +1.6, and +0.8 AUC scores UT-Zappos, respectively. This study confirms that considering the intrinsic coherence structure of attribute/object feature distribution is beneficial for superior prototype assignment.
4.5 QUALITY ANALYSIS
Success Cases. The first four columns Fig. present success cases of our method CLUSPRO for both seen and unseen compositions UT-Zappos [27] and C-GQA [28]. As seen, compared with the base model without primitive-wise prototype clustering, CLUSPRO works much better. Even for the complex C-GQA dataset, CLUSPRO still correctly predicts labels. For example, CLUSPRO can

---

Published as a conference paper at ICLR 2025
Success casts
Failure cases
Ul-Zappos
Baseline
Shoes Loafers
Suede Boots, Knee High Suada Boots, Mid-Ca Stade Boots Mid-Calf
Leather Satin Sandals Sata Sandals
Suade Boots, Laather Shoes, Heels Shoes, Heels Satmn Sandals
Loafers
Ours
Leather Stappers
Sheepskin Bools.Ankle Sheepskin Bools.Ankle
Wool Slippers Leather Sipppers
Ground Truth
Leather Slippers
CGQA
Baseline
Eaten Pizza
Large Sheep
Yellow Umbeslla
Square Floor
Metal Spooo
Green Plant
Ours
Sliced Pizza
While Sheep
Orange Umbeslla
Tiled. Floor
Salver Utensil
Green Vegetable
Trith
Sleed Pizza
hite Shoep
Orange
Minioture Vegetable
Figure 3: Case study UT-Zappos [27] and C-GQA [28]. We compare CLUSPRO with baseline without
primitive- wise prototype clustering. Correct and incorrect predictions are marked in green and red, respectively.
Laather
Leather
HaicCalf
Satn
Anide Sandials
Slipper
Mid-Calf Shoes Oxfords Shoes Heels
Baseline
Ours
Daseline
Ours
(a)
Attribute
(b) Object
Figure 4; Visualization attribute and object features learned by baseline and CLUSPRO on UTT-Zappos [27].
calibrate Suede to Leather (materials) and Yellow to Orange (colors). This demonstrates CLUSPRO can capture fine- mne-grained primitive patterns (e.g., various materials and colors) by representing each primitive set of prototypes. Moreover, benefiting from prototype clustering across the whole dataset, CLUSPRO automatically mines the global data distribution of each primitive, leading to generalizing well to unseen compositions. More success cases provided in $E of Appendix.
Failure Cases and Limitations. The last two columns of Fig. 3 show failure cases, where the attribute and object of images highly entangled and visually confusing. However, CLUSPRO still identifies the part of attribute-ohject compositions. In addition, though making mistakes on attribute predictions (e.g., Green Vegetable in column 6, row 2). such wrong predictions interpret another attribute (i.e., the color) of Miniature Vegetable. Thus we will make use of large language models to generate informative descriptions for each composition in the future, so as emphasize primary primitives. More failure cases are provided in ŠE of Appendix.
Feature Distributions of Attribute and Object. We visualize learned features of attribute and object by LBAS (Eq. 6) and L. (Eq. 12) in Fig. 4. We observe that, after considering clustering-basee prototype mining, learned attribute and object features become more compact and better separated. This demonstrates that CLUSPRO shape well-structured attributc/object embedding spaces by clustering- based analysis across the whole dataset, hence ensuring better visual disentanglement.
CONCLUSION
In this work, we present CLUSPRO, a clustering-baser prototype mining framework for Composi- tional Zero-Shot Learning. This framework aims to learn a well-structured and independent embed- ding space with multiple discriminative prototypes for each primitive, which alternates between two steps: 1) within primmitvee online clustering for automatically discovering and dynamically updat- ing prototypes; 2) prototype-bet basedd primitive representation learning for promoting primiiiie separation and inter- ttiii-rriittve decorrelation. Experimental results on three gold-standard datasets demonstrate the superiority of our clustering-based scheme against existing methods.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (No. 62222207, 62332010, 62427808, and 62372405), the Fundamental Research Funds for the Central Universities 226-2024-00058, the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHAI-202400 Bytedance Doubao Fund, and Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
REFERENCES
[1] Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris Baker. Revealing the multi- dimensional mental representations of natural objects underlying human similarity judgements.
Nature human behaviour, 4(11):1173-1111 2020.
[2] Noam Chomsky, Aspects of the Theory of Syntax. MIT press, 2014. 1
[3] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[4] Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, and Gal Chechik, Learning generalize to new compositions image understanding, arXiv preprint wrXiv:1608.0763, 2016.
[5] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. CVPR, pages 1792-1801, 2017. 1, 3, 18
[6] Zhe Liu, Yun Li, Lina Yao, Xiaojun Chang, Wei Fang, Xiaojun Wu, and Abdulmotaleb El Sad- dik. Simple primitives with feasibility- and dettextulltty-deeeeeeee for open-world compo- sitional zero-shot learning. IEEE TPAMI, 2023.
[7] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-ojeect compositions. In CVPR, pages 11316-111 11325, 2020. 1, 18
[8] Hanjae Kim, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn. Hierarchical visual primi- tive experts for compositional zero-shot learning. In ICCV, pages 5675-5681 2023. 1
[9] Massimiliano Mancini, Muhammad Ferjad Nacem, Yonggin Xian, and Zeynep Akata. Open
world compositional zero-shot learning. In CVPR, pages 5222-5230, 2021. 1, 18, 20
[10] Xiaoming Hu and Zilei Wang. Leveraging sub-class discimination for compositional zero-shot learning. In AAAI, volume 37, pages 890-898, 2023. 1,3
[11] Chenyi Jiang and Haofeng Zhang. Revealing the proximate long-tail distribution in composi- 2024.
tional zero-shot learning. In AAAI, volume 38, pages
[12] Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang L Liang, Peng Wang, and Chunhua Shen. Learning conditional attributes for compositional zero-shot learning. CVPR, pages 1119--11206 2023. 1, 18, 19
[13] Tian Zhang. Kongming Liang, Ruoyi Du, Xian Sun, Zhanyu Ma, and Jun Guo. Learning invariant visual representations for compositional zero-shot learning. In ECCV, pages 339- 355, 2022. 1
[14] Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, and Donglin Wang. Troika: Multi-path cross-modal traction for compositional zero-shot learning. In CVPR, pages 24005- 24014, 2024. 1, 4, 7, 16, 17, 18, 19
[15] Yun Li, Zhe Liu, Hang Chen, and Lina Yao. Context-based and diversity-driven specificity in compositional zero-shot learning. In CVPR, pages 17037-17046, 2024. 1, 3, 7, 8. 17, 18, 20
[16] Wentao Bao, Lichang Chen, Heng Huang, and Yu Kong. Prompting language-informed distri- bution for compositional gero-shot learning. arXiv preprint arXiv:2305..1442. 2023. 1, 3, 7,
8, 18
11

---

Published as a conference paper at ICLR 2025
[17] Nihal V Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for composi- tional zero-shot learning. In ICLR, 2023. 3, 7, 8, 17, 18, 20
[18] Xiaocheng Lu, Song Guo, Ziming Liu, and Jingcai Guo. Decomposed soft prompt guided fusion enhancing for compositional zero-shot learning. CVPR, pages 23560-23569, 2023. 1, 4, 7, 8, 17, 18, 20
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 7763, 2021. 2, 3, 7,
8, 16, 17, 18, 20
[20] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding network for compositional zero-shot learning. In CVPR, pages 9326-9335, 2022. 2, 3, 6, 18, 19
[21] Yun Li, Zhe Liu, Saurav Jha, and Lina Yao. Distilled reverse attention network for open-world
compositional zero-shot learning. In ICCV, pages 1782- 1791, 2023. 2,
[22] Frank Ruis, Gertjan Burghouts, and Doina Bucur. Independent prototype propagation for zero- shot compositionality. In NeurIPS, volume 34, pages 10641-10653, 2021. 2,
[23] Shaozhe Hao, Kai Han, and Kwan- Yee K Wong. Learning attention as disentangler for com- positional zero-shot learning. In CVPR, pages 15315-15324, 2023. 2, 3, 4, 18, 19, 20
[24] Chenchen Jing, Yukun Li, Hao Chen, and Chunhua Shen. Retrieval-augmented primitive rep- resentations for compositional zero-shot learning. In AAAI, volume 38, pages 2652-2660, 2024. 2, 3, 17, 20
[25] Alain Rakotomamonjy, Rémi Flamary, and Nicolas Courty. Generalized conditional gradient: analysis of convergence and applications. arXiv preprint arXiv:1510.06550 2015. 2, 6
[26] Phillip Isola, Joseph J Lim, and Edward H Adelson. Discovering states and transformations in
image collections. In CVPR, pages 1383-1391, 2015. 2, 7, 8, 9, 16, 18, 19, 21
[27] Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In CVPR, pages 192-199, 2014. 2, 7, 8, 9, 10, 16, 17, 18, 19, 21
[28) Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In CVPR, pages 953-962, 2021. 2, 3, 7, 8, 9, 10, 16, 18, 19, 20, 21
[29] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In ECCV, pages 169-185, 2018. 3, 18, 19
[30] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In ICCV, pages 3593-3602, 2019. 3, 7, 18
[31] Muhammad Umer Anwaar, Zhihui Pan, and Martin Kleinsteuber. On leveraging variational graph embeddings for open world compositional zero-shot learning. InACM MM, pages 4645- 4654, 2022. 3, 18, 19
[32] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata, Learn- ing graph embeddings for open world compositional zero-shot learning. IEEE TPAMI, 1545)11545-1160, 2022. 3, 18
[33] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Alain Pagani, Di- dier Stricker, and Muhammad Zeshan Afzal. Learning attention propagation for compositional zero-shot learning. In WACV, pages 3828-3837, 2023. 3. 18, 19
[34] Nirat Saini, Khoi Pham, and Abhinav Shrivastava. Disentangling visual embeddings for at- tributes and objects. In CVPR, pages 13658-13667, 2022. 4, 19, 20
12

---

Published as a conference paper at ICLR 2025
[35] Muli Yang, Cheng Deng, Junchi Yan, Xianglong Liu, and Dacheng Tao. Learning unseen concepts via hierarchical decomposition and composition. CVPR, pages
2020. 3
[36] Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. causal view of compositional zero-shot recognition. In NeurIPS, volume 33, pages 1462-1473, 2020. 3,
[37] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag pre-training for unified vasion-language understanding and generation. In ICML, pages 12888- 12900, 2022. 3
[38] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision Maaguug representation learning with noisy text supervision. In ICML, pages 4904-4916, 2021.
[39] Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. Prompting large pre-trained vision- language models for compositional concept learning. arXiv preprint arXiv:2211.05077, 2022. 3, 7.
[40] Agnar Aamodt and Enric Plaza. Case-based reasoning: Foundational issues, methodological variations, and system approaches. Al communications, 7(1):39 59, 1994. 3
[41] Yi Yang. Yueting Zhuang, and Yunhe Pan. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic Engineering, 22(12):1551-1558 2021.
[42] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2016. 3,
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021. 3
[44] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 3
[45] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. JEEE TIT, 13(1):21-27, 1967. 3
[46] Salvador Garcia, Joaquin Derrac, Jose Cano, and Francisco Herrera. Prototype selection for nearest neighbor classification: Taxonomy and empirical study. IEEE TPAMI, 34(3)4417435,
2012. 3
[47] Jacob Goldberger, Geoffrey Hinton, Sam Roweis, and Russ Salakhutdinov. Neighbour- hood components analysis. In NeurlPS, volume 17, 2004. 3
[48] Xiaofei He, Deng Cai, Shuicheng Yan, and Hong-Jiang Zhang. Neighborhood preserving embedding, In ICCV, volume 2, pages 1208-1213, 2005. 3
[49] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, volume 30, 2017. 3
[50] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmentation: A prototype view. In CVPR, pages 2582-2593, 2022. 3,
[51] Tuo Feng, Wenguan Wang. Xiaohan Wang, Yi Yang, and Qinghua Zheng. Clustering based point cloud representation learning for 3d analysis. ICCV, pages 8283-8294, 2023. 3
[52] Zheyun Qin, Cheng Han, Qifan Wang, Xiushan Nie, Yilong Yin, and Lu Xiankai. Unified 3d segmenter as prototypical classifiers. In NeulPS, volume 36, pages 46419-46432, 2023.
[53] Yuhang Ding, Liulei Li, Wenguan Wang, and Yi Yang. Clustering propagation for universal medical image segmentation. In CVPR, pages 3357-3369, 2024. 3
13

---

Published as a conference paper at ICLR 2025
[54] James Chenhao Liang, Tianfei Zhou, Dongfang Liu, and Wenguan Wang. Clustseg: Clustering for universal segmentation. In ICML, pages 20787-20809, 2023. 3
[55] Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. arXiv preprint TrXiv:2404.04308., 2024. 3
[56] Mingcheng Hou and Issei Sato. A closer look prototype classifier for few-shot image clas- sification. In NewrIPS, volume 35, pages 25767-25778, 2022. 3
[57] Hao Zhu and Piotr Koniusz. Transductive few-shot learning with prototype-based label prop- agation by iterative graph refinement. In CVPR, pages 23996-24006, 2023. 3
[58] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. In NeurlPS, volume 33, pages 21969-21980, 2020.
[59] Chaoqun Wang, Shaobo Min, Xuejin Chen, Xiaoyan Sun, and Houqiang Li. Dual progressive prototype network for generalized zero-shot learning. In NeurIPS, volume 34, pages 2936- 2948, 2021.
[60] Wenjin Hou, Shiming Chen, Shuhwang Chen, Ziming Hong. Yan Wang, Xuetao Feng, Salman Khan, Fahad Shahbaz Khan, and Xinge You. Visual-augmented dynamic semantic prototype for generative zero-shot learning. In CVPR, pages 23627-23637, 2024. 3
[61] Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Shaoqiu Zheng, Ying Tan, and Erjin Zhou. Protoclip: Prototypical contrastive language image pretraining. IEEE TNNLS, 2023. 3
[62] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural net- works: A survey. IEEE TPAMI, 43(11):4037-4458, 2020. 3
[63] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Mag-
azine, 39(3):42 62, 2022. 3
[64] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmseg: Gaussian mixture based generative semantic segmentation models. In NeurIPS, volume 35, pages 31360-31375, 2022.
3
[65] Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey, Symmetry, 11(9)):066, 2019.
[66] Andrew Zhai and Hao-Yu Wu. Classification is a strong baseline for deep metric learning. arXiv preprint 1811. 12649, 2018. 3
[67] Guikun Chen, Xia Li, Yi Yang, and Wenguan Wang. Neural clustering based visual represen- tation learning. In CVPR, pages 57 5714-5725, 2024. 3
[68] James Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clus- terfomer: clustering as a universal visual learner. In NeurIPS, volume 36, 2024. 3
[69] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 3.5
[70] Ruijie Quan, Wenguan Wang, Fan Ma, Hehe Fan, and Yi Yang. Clustering for protein repre- sentation learning. In CVPR, pages 319-329, 2024. 3
[71] Neil Houlsby, Andrei Ciurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efffcient transfer learning for nlp. In ICML, pages 2790-2799, 2019. 4, 17
[72] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. In ICLR, 2022. 4, 17
14

---

Published as a conference paper at ICLR 2025
[73] Wanxing Chang. Ye Shi, and Jingya Wang. Csot: Curriculum and structure-awar optimal transport for learning with noisy labels. In NeurIPS, volume 36, pages 8528 8841,851,111 2023. 5, 7, 20
[74] Guangyi Chen, Weiran Yao, Xiangchen Song. Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022. 5
[75] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. ICLR, 2023.
[76] Tianfei Zhou and Wenguan Wang. Prototype-based semantic segmentation. IEEE TPAMI, 2024.
[77] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, pages 332-149, 2018. 5
[78] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, volume 33, pages 9912-9924, 2020. 5, 18, 19
[79] Richard L Dykstra. An algorithm for restricted least squares regression. Journal of the Ameri- can Statistical Association, 78(384):837 842, 1983.
[80] Yanhua Yang, Rui Pan, Xiangyu Li, Xu Yang, and Cheng Deng. Dual-stream contrastive learning for compositional zero-shot recognition. JEEE TMM, 26: 1909-19199 2023. 6
[81] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola.
A kernel statistical test of independence. In NeurlPS, volume 20, 2007. 7
[82] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lision-language models. IJCV, 130(9))2337-2348 2022. 7,8, 7,8,1 18
[83] Guangyue Xu, Joyce Chai, and Parisa Kordjamshidi. Gipcol: Graph-injected soft prompting for compositional zero-shot learning. In WACV, pages 5774-5783, 2024. 7, 8, 18
[84] Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In ECCV, pages 52-68, 2016. 7
[85] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[86] Guiyang Chan, Pengcheng Zhang, Hai Dong, Shunhui Ji, and Bainian Chen. Scribble- supervised semantic segmentation with prototype-han feature augmentation. In ICMI., 2024. 9
[87] Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 133(4), 2006. 9
[88] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurlPS, volume 26, 2013. 9
[89] Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning. In CVPR, pages 9336- 9345, 2022. 18
[90] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, pages 248-255, 2009. 18
[91] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NeurIPS, volume 26, 2013. 18,
19
[92] Meitar Ronen, Shahaf E Finder, and Oren Freifeld. Deepdpm: Deep clustering with an un- known number of clusters. In CVPR, pages 9861-9870, 2022. 20
15

---

Published as a conference paper at ICLR 2025
This appendix provides additional details for the ICLR 2025 submission, titled "Learning Clustering-based based Prototypes for Compositional Zero-shot Learning". The appendix is organized follows:
A Detailed data split statistics.
$B Algorithm overview.
SC More details about baseline model.
SD More quantitative results.
SE More qualitative visualization.
$F The pseudo-code of prototype assigmment and updating.
$G More discussions.
A DETAILED DATA SPLIT STATISTICS
We conduct experiments on three widely-used CZSL benchmarks: MIT-States [26]. UT- Zappos [27], and C-GQA [28]. MIT-States consists of 53, 753 natural images in total, with 115 states and 245 objects. Following conventional procedures, 1, 962 available compositions in the dataset split into 1, 262 seen and 300/400 unseen compositions for trai n/val 1 d spectively. UT-Zappos contains 50, 025 fine-grain shoe images with 16 states, 12 objects and 116 state-object compositions. Following standard practices, the compositions are split into 83 seen compositions, 15 seen and 15 unseen compositions, 18 seen and 18 unseen compositions for tra peiin/vaiidad onnttssn/sssssee splits. C-GQA the most extensive CZSL dataset, containing 453 states and 870 objects for 39, 298 images in total and over 9, 500 state-object compositions. The dataset divided into 592 seen compositions for train, 252 seen and 1, 040 unseen com- positions for va idation, and 888 and 923 unseen compositions for test. The detailed data split statistics is provided in Table 5. Here |C*| and [C4| indicate the number seen and unseen compositions, respectively. |X| represents the number of images.
Table detailed data split statistics ($A) on MIT-States [26]. UT-Zappos [27] C-GQA [28].
val validation on
Dataset
[A]
0
1C*
X
|C*|
[C*
[X]
|C*
|C*|
X
MIT-States [26]
115
1262
30k
300
300
400
400
13k
UT-Zappos [27]
16
83
23k
15
15
3k
18
18
3k
C-GQA [28]
413
674
5592
27k
1252
1040
7k
888
923
5k
B ALGORITHM OVERVIEW
Fig. presents the architecture our CLUSPRO. It takes a batch images and all the semantic labels (i.e., attributes, objects, and compositions) input. CLUSPRO first utilizes the visual en- coder of CLIP [19] along with attribute and object adapters to obtain attribute features F", object features F", and composition features F (Bq. 1). Besides, CLUSPRO constructs attribute, object, and composition prompt representations (Eq. 2) via a soft learnable prompt strategy [14] based on pre-given semantic labels. Based on these visual and prompt representations from three branches, the three-path classification loss (i.e., CBAS) in Eq. 6 are employed to recognize primitive concepts and their compositions. Meanwhile, based on the obtained attribute and object visual feature rep- resentation (i.e., Fa and FO), CLUSPRO learn prototypes by within-primitive clustering ($3.3) and then propose two complementary metric learning mechanisms (i.e., CPCL and CPDL)) based on these prototypes,
to explicitly shape well-structured and independent primitive embedding space ($3.4). Finally, we assemble the three-path classification loss CASS and our proposed prototype- based loss constraints (i.e., CPLL and LDDL) as final learning objective. Our algorithm not only learns primitive recognition with pre-given semantic labels, but also automatically discovers diverse and fine-grained intra-prin intra-prinitive patterns via of prototypes across the entire dataset.
16

---

Published as a conference paper at ICLR 2025
C MORE DETAILS ABOUT BASELINE MODEL
Visual Feature Extraction. Following [14, 15, 24], we adopt the visual encoder ti of CLIP [19] to splits the input image #RRR* into N₂ HW/P patches, where P is the resolution of each patch. Note that we following [14, 24] to tune the image encoder of CLIP with LoRA [71, 72],
lightweight parameter efficient fine-tuning (PEFT) strategy. The encoder te projects these patches into patch tokens along with [cls] token, and then updates these tokens via Transformer blocks. Finally, the [cls] token serves as the image representation fa. We adopt attribute adapter ht and object adapter h [71, 72], each implemented as separate MIP,, to project f into the discriminative attribute feature f and object feature f", respectively.
Prompt Feature Extraction. We follow existing CZSL [18, 14] to employ an independent prompt prefix for each branch. Specifically, for each attribute-object composition Cijj (ai, we create : 8f, vgJ. object prompt Sj [s!....,.i si, v, and composition prompt S = i of, u.. v where si $i., and s are learnable pretix contexts initialized by "a photo of Then these prompts are then fed into the frozen text encoder of CLIP [19] to obtain prompt features.
three prompts for each branch, iee, attribute prompt S. [8f :
Training Loss C Following previous CZSL approaches [14, 15, 24], the parameters 0 of the baseline model are learned by minimizing the three-path classification loss (Eq. 6) on the training dataset. Note that we have omitted the weight decay in Eq. 6 for simplicity. We just follow [14] set the weight decay as 5 for our experiments.
Feasibility Calibration for Open- Setting, Following [17. 14, 24]. we adopt post-training feasibility calibration to filter out infeasible compositions that might be present in the open- world setting. The calibration relies the assumption that similar objects tend to share similar attributes, while dissimilar objects are unlikely to exhibit shared attributes. Therefore, given candidate pair
e = (a, o), We calculate the feasibility compositions by computing the relationships between the objects and the attributes. First, we compute the similarities between the objects:
max 6(0) (())
(14) where Ô is the other objects paired with the attribute in seen compositions, and o(.) is an em- bedding function that maps the primitive to pre-trained embedding. We calculate the similari- ties Pa o) between attributes as same. Next, we combine the two similarities (i.e., Pa o) and Pa(a,o) )) with a pooling function to obtain p(a,0).
po(a,o)
6e
(())
Finally, we filter out infeasible compositions by only considering compositions above a threshold p(a,o) > T on the validation set to make the prediction:
arg max Pugectot pla,,,))
p(c.sl)) + p(as/)) p(uslz).
(15)
D MORE QUANTITATIVE RESULTS
Loss Coefficients and B. We further study the effect of loss coefficients and 3 for loss functions
CrCL (ef. 10) and LYDL (cf. Eq. 11) on UT-Zappos [27]. In Table 6a, after fixing the loss coefficient 8, CLUSPRO achieves the best performance when is set to 0.2. Additionally, in Table 6b, we fix a and set 3 with different values to test the impact of LPDL. We observe that setting 3 as 0.5 leads to the best results across all metrics. Accordingly, we set 0.2 and B 305 in the training stage.
Table 6: The impact of loss coefficients and 3 ($D).
UT-ZZapos
UT--Zappos
Coefficient a
Coefficient 3
Scent
Uneent
HMt
AUCt
Seent
Uneent
HMM
AUCt
a 0.1
70.5
74.2
57.0
45.1
B= 0
67.2
74.1
54.9
42.6
a 0.2
70.7
76.0
58.5
46.6
3 0.5
70.7
76.0
58.5
46.6
a 0.3
70.6
75.3
58.6
46.3
B B=
70.7
75.0
58.0
45.9
a 0.4
70.4
75.2
58.2
46.2
8=5
70.3
74.9
56.7
44.6
a= 0.5
70.5
74.7
58.2
45.9
=
67.6
74.0
54.8
41.7
(a) Loss Coefficient
(b) Loss Coefficient 3
17

---

Published as a conference paper at ICLR 2025
methods, further compare our algorithm CLUSPRO with existing CZSL methods [29, 30, 7, 9, 32, 28, 20, 12, 31, 5] with pre-trained ResNet18 [42] backbone across three datasets [26, 27, 28J. Table and Ta- ble 8 report additional comparison results within CW and OW settings, respectively. As can be seen, CLIP-based methods significantly outperform traditional vision-based methods. This evidences that CLIP-based CZSL methods have stronger compositionality for zero-shot generalization. Notably, CLUSPRO surpasses all other methods and achieves state-of-the-ar performance.
More Comparison Results with Existing CZSL Methods. Apart from
Table 7: More comparison results( DD) on MIT-States [26], UT-Zappos [27] and C-GQA [28] within CW setting.
Josed-World
MiT-St
U UT-Zappos
C-GQA
Method
Seenf methous
Unseent
HM
AUC
Seent
Unseen?
HM
AUCT
Seent
Unseen"
HM
AUCT
Traditional
AnP [29]
14.3
17.4
9.9
1.6
59.8
54.2
40.8
25.9
17.0
5.6
5.9
0.7
LE+
15.0
20.1
10.7
53.0
61.9
41.0
25.7
TMN [30]
20.2
20.1
13.0
2.9
58.7
60.0
45.0
29.3
23.1
6.5
7.5
1.1
SymNet [7]
24.2
25.2
16.1
49.8
57.4
40.4
23.4
10.3
11.0
2.1
CompCos
25.3
24.6
16.4
4.5
59.8
62.5
43.1
28.1
28.1
11.2
12.4
2.6
CGB (28)
28.7
25.3
17.2
5,1
56.8
63.6
41.2
26.4
28.1
10.1
11.4
2.3
CGB
27.8
2.8
SCEN [20]
29.9
25.2
18.4
5.3
63.5
63.1
47.8
32.0
28.9
12.1
12.4
2.9
CVGAB [31]
28.5
25.5
18.2
65.0
62.4
49.8
34.6
11.9
13.9
2.8
CANet [12]
29.0
26.2
17.9
5.4
61.0
66.3
47.3
33.1
30.0
13.2
14.5
3,3
CAPE [33]
30.5
26.2
19.1
60.4
67.4
45.5
31.3
32.9
15.6
16.3
4.2
CLIP-bused methody
CLIP (19)
30.2
46.0
26.1
11.0
15.8
49.1
15.6
5.0
7.5
25.0
8.6
1.4
CoOp [82]
34.4
47.6
29.8
13.5
52.1
49.3
34.6
18.8
20.5
26.8
17.1
4.4
PCVL [39]
48.5
47.2
35.3
18.3
64.4
64.0
46.1
32.2
CSP [17]
46.6
49.9
36.3
19.4
64.2
66.2
46.6
33.0
28.8
26.8
20.5
6.2
DFSP
47.4
52.4
37.2
20.7
64.2
66.4
45.1
32.1
29.3
24.3
8.7
DFSP(BIP) [18]
47.1
52.8
37.7
20.8
63.3
69.2
47.1
33.5
36.5
32.0
26.2
9.9
DFSP(20)
46.9
52.0
37.3
36.0
GIPCOL [83]
48.5
49.6
36.6
19.9
65.0
68.5
48.8
36.2
31.9
28.4
22.5
7.1
CDS-CZSL [15]
50.3
52,9
39.2
22.4
63.9
74,8
52.7
39.5
38.3
34.2
28.1
11.1
Thiik [14]
49.0
53.0
39.3
22.1
66.8
73.8
54.6
41.7
41.0
35.7
29.4
12.4
PLID [16]
49.7
52.4
39.0
22
67.3
68.8
$2.4
38.7
38.8
33.0
27.9
11.0
CLUSPRO (Ours)
52.1.36
40.7
23.8
58.5,0
32,8
Table 8: More comparison results($D) on MIT--ttate [26], UT-Zappos [27] and C-GQA [28] within OW
setting.
Open-World
Mil-States Unseent
UT-Zappos Unseeat
C-GQA
Method
Scent
HMt
AUC1
Sccat
HM
AUC1
Seent
Unseen?
HM
AUC1
Traditional vision based methous
AoP [29]
16.6
5.7
4.7
0.7
50.9
34.2
29.4
13.7
LE+ [5]
14.2
2.5
2.7
0.3
60.4
36.5
30.5
16.3
19.2
0.7
1.0
0.1
[30]
SymNet [7]
21.4
7.0
5.8
0.8
53.3
44.6
34.5
18.5
26.7
2.2
3,3
0.4
CompCos [9]
25.4
10.0
8.9
1.6
59.3
46.8
36.9
21.3
28.4
1.8
2.8
CGE
0.7
58.8
38.0
21.5
28.3
1.3
Co-CGE [32]
26.4
10.4
10.1
2.0
60.1
44.3
38.1
21.3
28.7
1.6
2.6
0.4
KG-SP [89]
28.4
7.4
1.3
61.8
52.1
42.3
26.5
31.5
2.9
4.7
CVGAE [31]
27.3
9,9
10.0
1.8
58.6
48.4
41.7
22.2
26.6
2.9
6.4
0.7
CLJP bored method's
CLIP [19]
30.1
12.8
3.0
15.7
11.2
2.2
7.5
4.6
CoOp [82]
34.6
9.3
12.3
2.8
$2.1
31.5
28.9
13.2
21.0
4.6
5.5
0.7
PCVL [39]
48.5
16.0
17.7
6.1
64.6
44.0
37.1
21.6
[17]
17.4
28.7
5.2
DESP(220 [18]
47.2
18.2
19.1
6.7
64.3
$3.8
41.2
26.4
35.6
6.5
9.0
2,0
DESP(BF) [18]
47.1
18.1
19.2
6.7
63.5
57.2
42.7
27.6
36.4
7.6
10.6
DFSP((2)) [18]
47.5
18.5
19.3
6.8
66.8
60.0
44.0
30.3
38.3
7.2
10.4
GIPCOL [83]
48.5
16.0
17.9
6.3
65.0
45.0
40.1
23.5
31.6
5.5
7.3
1.3
[15L[115]
Troika [14]
48.8
18.7
20.1
7.2
66.4
61.2
47.8
33.0
40.8
7.9
10.9
2.7
PLID [16]
49.1
18.7
20.0
7.3
67.6
$5.5
46.6
30.8
39.1
7.5
10.6
CLUSPRO (Ours)
51.2.04
22.1.m2
23.0.l
9.3,02
71.0.1
66.2.s
54.1.0
39.5.0a
41.6.03
8.3.2
11.6,03
3.0.a1
Evaluation Results for Models Pre-trained on Datasets with No Overlap. To further highlight the robustness and superiority of our approach, additionally present results under CW setting that utilize ViT-B backbone pre- trained with DINO [78] on ImageNet [90] in a supervsed manner as ADE [23] instead of CLIP model [19]. Besides, encode text representation with word2vec [91]
18

---

Published as a conference paper at ICLR 2025
[23, 12, 28] instead of the text encoder of CLIP. Table 9 reports the comparison results UT- Zappos [27] and CGQA [28]. As seen, our algorithm also demonstrates better performance than the baseline and SOTA non CnnC methods [20, 31, 33, 23].
Table More comparison results (D) UT-Zappos [27] and C-GQA [28] within CW setting. Our algorithm utilizess ViT-B backbone pre-trained with DINO [78] as the visual encoder and word2vec [91] the text encoder for a fair comparison with non-CLIL methods.
UT-ZZappss
C-GQA
Method
Seen"
Unseen
HM1
AUC1
Seen
Unseen"
HM
AUC1
AoP [29]
59.8
54.2
40.8
25.9
17.0
5.6
5.9
0.7
SCEN [20]
63.5
63.1
47.8
32.0
28.9
12.1
12.4
2.9
CVGAE [31]
65.0
62.4
49.8
34.6
28.2
11.9
13.9
2.8
CANet [12]
61.0
33.1
30.0
13.2
14.5
CAPE [33]
60.4
67.4
45.5
31.3
32.9
15.6
16.3
4.2
ADE [23]
63.0
64.3
51.1
35.1
35.0
17.7
18.0
5.2
CGE [28]
38.0
17.1
18.5
5.4
OADis [34]
38.3
19.8
20.1
7.0
Baseline
61.0
62.9
45.1
31.9
15.9
16.6
CLUSPRO (Ours)
65.1
68.0
52.3
39.3
23.0
22.3
7.6
Efficiency Analysis. The efficiency comparison results with the state-of-the-ar Trokia [14] and our baseline are reported in Table 10. Note that, CLUSPRO conducts within seriiin-rimmiiee prototype clustering in a nonparametric manner and discards these learned sub-primitive prototypes during the testing phase. Thus, as shown in Table 10, CLUSPRO neither requires additional trainable parameters nor causes any inference delay during testing compared to the base model. Though efficient in terms of parameters and inference speed, our online clustering algorithm brings slight training delay ( 11.5% on U'T-Zappos [27]). Moreover, the effective clustering algorithm allows CLUSPRO to outperform the state-of-the-arr Trokia in terms of classification accuracy, trainable parameters, and inference speed.
Table 10: Efficiency comparison UT-Zappos [27]. Here, report trainable parameters, training time per epoch, and inference speed for each model. See in $D for more details.
Method
Params
Memory.
Training timel
Inference Speed,
AUC1
Troika [14]
21.7M
19.9G
4.1min
14.9ms
41.9
Baseline
8.7M
18.2G
4.0min
14.6ms
41.0
CLUSPRO (ours)
8.7M
18.5G
4.6min
14.6ms
46.6
Number Prototypes K. Table 11, we conduct the experiment by setting the number K of prototypes based on the proportion of training samples for each primitive. In UT-zappos [27] dataset, training samples per primitive range from 0.2% to over 20%. Thus, we assign K=1 to the primitive with 0.2 5% training samples, K = to the primitive with 5~ 10% training samples, ==3 to the primitive with 10~ 15% training samples, K to the primitive with 15 ~ 20% training samples, and K 5 to the primitive with over 20% training samples. As seen, this approach results in slightly better performance than setting a fixed value for all the primitives.
Table 11: Ablative experiments regarding varying K on UT-Zappos [27]. See &D for more details.
UT-Zappos Uneent
K range
Seent
HMf
AUC1
unique value
70.7
76.0
58.5
46.6
[1.5]
71.0
76.0
58.6
46.8
E MORE QUALITATIVE VISUALIZATION
More Case Study. We provide additional success and failure cases of our method CLUSPRO across three CZSL. benchmarks, iee., MrT-States [26] in Fig. 5, UT-Zappos [27] in Fig. 6 and C-GQA [28]
19

---

Published as a conference paper at ICLR 2025
in Fig. 7. We also compare approach CLUSPRO with baseline without within-primitive cluster- ing. As seen, by mining rich sub- suli-rrmitivee patterns via within-primitive clustering, CLUSPRO can produce more accurate composition predictions, even recognizing fine-grained primitives, such as various materials and colors. For failure cases, where the attribute and object of images are highly entangled, CLUSPRO still identifies the part of the attribute-object composition.
F PSEUDO CODE OF PROTOTYPE ASSIGNMENT AND UPDATING
Algorithm provides the pseudo-code of "Local-aware Prototype Assignment" and "Prototype Up- dating". To guarantee reprodueibility, our code is available at CLUSPRO.
G DISCUSSION
Data Overlap Analysis. Given that CLIP [19] is trained on millions of text-image pairs sourced from the web, it is hard to know whether CLIP has been exposed to certain unseen compositions during its pre-training, which violates the zero-shot learning setting factually. Most current re- searches [18, 17, 15, 24] in CZSL, including our work, report the performance in the Generalized Zero-shot Learning [9] for both CW and OW settings, where test samples include both seen and unseen compositions. Hence, it naturally brings up the question: whether CLIP meets the definition of Generalized Zero-shot Learning. Based on the data overlap analysis on 35 datasets reported in [19], there is median overlap of 2.2% and an average overlap of 3.2%. Due to this small amount of overlap, the overall accuracy shift is less than 0.1% with the largest shift 0.6%. As such, CLIP is only exposed to a very small number of unseen compositions during pre-training, and the impact on the performance limited. However, the potential composition leaking in the pre-training of CLIP indeed leads to an unfair comparison with other non-CLE non-CLIP methods [23, 28, 34]. Thus, we argue that it is important to emphasize the comparisons with other CLIP-based methods that share the same pre-training (comparison results in Table 1 and 2). Moreover, where possible, it is also advisable to report performance metrics for non-CI.IP variants to ensure a comprehensive evaluation.
Limitation. One limitation of our algorithm is that it needs extra authin-primitive prototype clus- tering from the perspective of optimal transport after each training iteration, leading to increasing time complexity. However, in practice, our clustering algorithm only brings slight training delay attributed to efficient GCG algorithm [73] for solving such clustering problem. Additionally, our mined sub-primitive prototypes are subject to the data distribution of the training dataset. Thus rare primitive concepts in the dataset (i.e., long-tail -til distribution), like many previous pose significant challenges for primitive-wise clustering discover diverse sub-primitive patterns, thus resulting in poor performance on unseen compositions about these primitives. Also, the number of prototypes for each primitive currently is set a fixed value, which may not be optimal given that intra-pring tirm--rimitive variability varies across primitives. Thus is interesting to find ways to automatically determine K [92] for different primitives, which may further boost performance.
Border Impact. This work introduces CLUSPRO, a powerful clustering- framework for Com- positional Zero-Shot Learning via exploring dataset-level context, which overcomes the limitations of previous solutions relying on single or paired images for visual disentanglement. This model pro- vides a feasible way to discover diverse sub-primitive patterns in massive training data, and directly shape well-structured embedding space based on these mined patterns. On the positive side, CLUS- PRO pushes the boundary of CZSL algorithms, and can benefit a number of potential real-world applications, e.g., autonomous driving and robotics. For the potential negative societal impacts, our CLUSPRO struggles in handling very rare primitives in the dataset, which is a common issue of current CZSL algorithms, thus leading to inaccurate decisions planning of systems. To avoid this potential problem, it is crucial to develop a security protocol in case our approach fails to perform expected in real-world scenarios.
20

---

Published as a conference paper at ICLR 2025
Success cases
Failure cases
MIT-SSutes
Baseline
Weathered Clock
Empty Highway
Polded Kniffe
Stright Swond Blunt Sword Blant Sword
Knife
Narrow Cabline
Ours
Ancient Clock
Barnn Road Barme Road
Bent Blade
Chipped Kaiće
Unpainted Furniture
Grund Truth
Ancient Clock
Bent Blade
Chipped Ceramic
Brokan Purniture
MIT-States
Basdlie
Bmiken Handle
Pressed Cookie
Shiny Bronse
Caramcizzed Chocolate
Pressed Sandwich
Prissue Wool
Ows
Broken Borde
Browned Butter
Brushed Copper
Carsmelized Broad
Mollen Cheese
Brushed Wool
Ground Truth
Brkke Bottle
Browned Butter
Brushed Copper
Cammellzzd Bread
Browzed Cheese
Brushed Velvet
Figure 5: More studies MIT-States [26]. We compare CLUSPRO with baseline without primitive-
prototype clustering. Correct and incorrect predictions are marked in green and red, respectively.
Success
Fallure cases
UT-Zaggas
Resdine
Lotther Roniss Mid-CDif
Suidk Shows Fieds
Teuther Rrots MiS-
Sualio Shoes colors Stede Stepas Shypers
Tanther Ponks Sraen High
Suadlo Shors cnfiees
Okes
Stade Books Mtdi taar
Hainthit Shoes Hinds
Stade Hoots
Leather Boots Mid-hir
Canras Shoes Loafers
Mid-Cadl
UT-7aggos
Reseline
Sundils
Tasther Rocks,
Stack High
Prlcnt. wother Shoes
Synthitie Rhotss Syntheti
Osss
Satio Sandiis
Santhetic RootsAride
Senthetic Boots Mid-Caif
Leathhe Shoos Fotort oother Shoos Plats
Hublorr Bootss.uala
Shoes Hooks
Omand Tlath
Scat Sandils
Synthrtic BootsAnkie
Synthetic Books Mid-Cit
Leather Aoods
Stato Stndall
Figure 6: More studies UT-Zappos [27]. We compare CLUSPRO with haseline without primitive-wise prototype clustering. Correct and incorrect predictions are marked in green red, respectively.
Succees cases
Fallure cases
CGQA
Bandime
Cokefuul Jocket
Wood Honae Beewn
White Time Hare Tree Bare Time
Brtan Counter Cnyy Gray Countrr
Purple Krifi Gnay Spoon Silver Spoon
Pink Shirt Oringe Dress
Doss
Stirt Calorfil Shirt
Grund Trab
Browm Hons:
Busdine
Shoggy
Calte WhiteCar
Carpet
Funy
Breed
Grord Trath
Crem Rush
RhaeCappet
Broem Cake
Statin
Figure 7: More case studies C-GQA [28]. We compare CLUSPRO with baseline without primitive
prototype clustering, Correct and incorrect predictions are marked in green and red, respectively.
21

---

Published as a conference paper at ICLR 2025
Algorithm Pseudo-code of prototype assignment and updating in PyTorch-like style.
P: primitive protocypes x & x
4 Ft primitive feature embeddings (N x
y: labels for primitive features F (N)
C: number of altribule or ject primilives
4 Kt number of prototypes for each primitive
N: batch size
nomentum coefficient (Bq.8)
local-aware prololype assignment def prototype assignment (F, label):
protstype assigament for each primilive TeaLure (Bq.7) P)
torch.einsum cinsum nnd ckd->nke', forch.einsum (' ad, ad-2nn', F, F) # grimilive sell-similarily matzix
for с in range (C):
init_g 0[...,c]
init_] = secaloonline.coo clustessssstrrnnnnnttn (init__, 8) one-hot matrix
$ prototype assignments for features in primitive 1_c Qlaabel
f_c FFlabl = c, ...]
# find features that are assigned to each sub-primitive prototype __c_tile wwrsume tile-K)
1_q = init_] Le_tile
4 rind Tealures wilh primilive e that are correctly classified f_c_tile repeat (l_, tile-f_c.shape (-1])
__e_ __ feellile
new cluster features for grimilive e torch..mml_g....... tta.soggggttsssees.s_e (),
momentum updating for each primitive prototype (Rq.8)
prototype.udaatingg (1_g, a, c)
pratotype updating def prototype.updiee updating c):
assigmms gnments each prototype primitive c torch, sum (1_s, dim=C)
normalize
prototype updating
if torch. surchhsummpee (n) > 0:
P_e Plc, n !- 0,:] ma ain 0, mu)
P[c, 1= 0, :] P_c
22


Reviews:
[{"review_id": "j1QFoHJGKK", "reviewer": "Reviewer_siAz", "summary": "This paper presents a clustering-based prototype mining framework namely CLUSPRO, for Compositional Zero-Shot Learning. \nSpecifically, it considers within-primitive online clustering for automatically discovering and dynamically\nupdating prototypes. Besides, it models prototype-based primitive representation learning for promoting intra-primitive separation and inter-primitive decorrelation. Experimental results on three datasets demonstrate the superiority of the proposed method against existing methods.", "questions": "see the weaknesses above please.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well written and easy to follow.\nThe motivation behind addressing CZSL is clear and convincing.\nThe technical contributions are solid and novel.\nThe algorithm details are well provided and it should be possible to reproduce the results reported in the paper.", "weaknesses": "Figure 2 shows  the clustering-based prototype mining framework, while it is still unclear about the whole pipeline for CZSL. It is encouraged to introduce the complete network architecture in the main paper or supplementary material. It will be help to understand why the proposed method can achieve new state-of-the-art performance.\n\nIn the experiments, it is concerned about why all the compared methods including CLUSPRO employ the same ViT backbone,\nto avoid the unfair comparisons between ViT-B and ViT-L. \n\nIt is interesting to know more details about the baseline model, including network structure and training loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihRI8IvMHM", "reviewer": "Reviewer_1rZT", "summary": "This paper reveals that exsiting compositional zero-shot learning (CZSL) methods only consider an isolated centroid for each primitive, ignoring rich and diverse intra-primitive patterns. To address this, the authors propose a clustering-based method CLUSPRO to learn a well-structured and independent embedding space with multiple discriminative prototypes for each primitive, thus improving CZSL. Specifically, CLUSPRO alternates between two steps: 1) performing within-primitive online clustering to automatically discover and dynamically update prototypes, and 2) using prototype-based primitive representation learning to encourage intraprimitive separation and inter-primitive decorrelation.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-motivated, highlighting that a single centroid primitive representation demonstrates limited tolerance to intraprimitive variance.\n- The proposed method CLUSPRO is reasonable and has a good performance across three benchmarks under both closed-world and open-world settings.\n- The authors present a comprehensive analysis of their method CLUSPRO.", "weaknesses": "- My major concern is the number of prototype. In the experiments, the number of prototypes $K$ is empirically set to 5, which may not ne a good strategy. Additionally, using the same number of prototypes for all attributes seems unreasonable. Are there any automatic methods for discovering prototypes, thus mitigating the need for manually setting this number?\n- Does performing clustering on the entire dataset will face resource limitations when dealing with very large datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWzyjA9DfM", "reviewer": "Reviewer_Aocv", "summary": "This paper proposes clustering-based method for compositional zero-shot learning.\nSpecifically,", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Clear presentation", "weaknesses": "Major problem:\n\nProblem1. The definition of Compositional Zero-shot Learning (CZSL) is not correctly used.\nThe authors mentioned CZSL requires the model to recognize unseen compositions without additional training data, but in your method, you use pretrained CLIP trained by large-scale dataset: CLIP is trained from millions text-image pairs from web.\nIn other word, all your claimed unseen compositions are seen by CLIP actually.\nThus, a key question is emerged: if your visual&text encoders never meet your test compositions, can the seen and unseen compositions be seperated?\nPlease provide the results that using other visual&text encoders or using vision-language models whose pretrained training datasets have no overlap with the evaluted datasets.\n\nProblem2. Besides, in the original paper of CLIP, it has clearly claimed they changed the definition of ZSL from class-level to dataset-level. \"In computer vision, zero-shot learning usually refers to the study of generalizing to unseen object categories in image classification (Lampert et al., 2009). We instead use the term in a broader sense and study generalization to unseen datasets.\" ﻿ This excerpt is from Section 3.2 of the CLIP paper. There is a significant gap between CLIP’s definition of zero-shot learning and the CZSL task you are addressing, which still focuses on classifying unseen composition categories. This difference is substantial and cannot be overlooked.\n\nIn a short, I suggest the proposed method is a transductive classification method insteat of ZSL method actually since the pre-trained CLIP have been exposed to numerous unseen composition samples.\n\nProblem3. \nYou only compare your method to CLIP-based CZSL methods, but there are many methods that do not use CLIP.\nI know those non-CLIP methods perform worsely than CLIP-based methods.\nBut, as I said in Problem1&2, since CLIP have been exposed to numerous unseen composition samples, the experiments are extramely unfair for other non-CLIP SOTAs.\n\nProblem4. Why only consider the diversities of the attribute, but do not for the objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uvqv2ny9fC", "reviewer": "Reviewer_Q4uv", "summary": "This paper proposes a clustering-based prototype mining framework for CZSL which conducts within-primitive clustering on the embedding space for automatically discovering and dynamically updating prototypes. These representative prototypes are subsequently used to repaint a well-structured and independent primitive embedding space, ensuring intra-primitive separation and inter-primitive decorrelation through prototype-based contrastive learning and decorrelation learning. Experiments on three benchmarks demonstrate CLUSPRO outperforms top-leading CZSL solutions under both closed-world and open-world settings.", "questions": "In the Prototype-based Contrastive Learning, the strategy encourages each primitive feature $f_n$ to\nbe similar to its assigned prototype $P+$ and dissimilar to all other $K(M+N)−1$ irrelevant prototypes\n$P-$. It means that the $K-1$ prototypes belonging to the same primitive but with different assignments are treated as negative samples in the same way as prototypes  $K(M+N-1)$ of other primitives. \nDoes this have a  impact on representation learning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Overall this paper is well-written and is easy to understand.\n- The motivation is reasonable and the proposed framework is a little novel for CZSL.\n- The CLUSPRO outperforms top-leading CZSL solutions under both closed-world and open-world settings on three benchmarks.", "weaknesses": "- There is a lack of discussion and citation of some related works. In ZSL/CZSL fields, prototype learning is wide adopted in the works [A][B]. The Local-aware Prototype Assignment module is also similar to the previous work [C], but the necessary discussion is absent.\n\n- The overall novelty is limited. The Local-aware Prototype Assignment technique is adopted in the work [C], and the Prototype-based Contrastive Learning have been used in previouse works (e.g., [A]).\n\n- Writing of Chapter 3.3 could be improved. The authors do not make it clear how the K prototypes are obtained. With reference to the context, is it means k-means clustering results on visual fetaures?  In addition, when the variable notations appear for the first time, the authors should give an obvious comment (e.g., $n$ in $f_{n}$) for reducing ambiguity.\n\nI will consider updating the score if the authors can address my concerns well.\n\n[A] ProtoCLIP: Prototypical Contrastive Language Image Pretraining \n\n[B] Dual progressive prototype network for generalized zero-shot learning\n\n[C] Rethinking semantic segmentation: A prototype view", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)