Conditional Representation Learning for Customized Tasks
Authors: Honglin Liu
Chao Sun
Peng Hu
Yunfan Li
Xi Peng
Venue: NeurIPS 2025 spotlight ()
Keywords: Unsupervised Learning
Representation Learning
Conditional Similarity
Primary area: deep_learning

Abstract: Conventional representation learning methods learn a universal representation that primarily captures dominant semantics, which may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics, leading to suboptimal results. As a solution, existing approaches resort to supervised fine-tuning, which however incurs high computational and annotation costs. In this paper, we propose Conditional Representation Learning (CRL), aiming to extract representations tailored to arbitrary user-specified criteria. Specifically, we reveal that the semantics of a space are determined by its basis, thereby enabling a set of descriptive words to approximate the basis for a customized feature space. Building upon this insight, given a user-specified criterion, CRL first employs a large language model (LLM) to generate descriptive texts to construct the semantic basis, then projects the image representation into this conditional feature space leveraging a vision-language model (VLM). The conditional representation better captures semantics for the specific criterion, which could be utilized for multiple customized tasks. Extensive experiments on classification and retrieval tasks demonstrate the superiority and generality of the proposed CRL. The code is available at https://github.com/XLearning-SCU/2025-NeurIPS-CRL.

Conditional Representation Learning for Customized Tasks
Honglin Liu', Chao Sun?, Peng Hu', Yunfan Li Xi Peng
College of Computer Science, Sichuan University, Chengdu, China 2 Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China National Key Laboratory of Fundamental Algorithms and Models for Engineering Numerical Simulation, Sichuan University, Chengdu, China (TristanLiuHl, penghu. ml, yunfanli. qm, pengx. gm) gmail com (sunchao) Caircas ac. cn
Abstract
Conventional representation learning methods learn a universal representation that primarily captures dominant semantics, which may not always align with tomized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categor- ical semantics, leading to suboptimal results. As a solution, existing approaches resort to supervised fine-tuning, which however incurs high computational and annotation costs. In this paper, we propose Conditional Representation Learning (CRL), aiming extract representations tailored arbitrary user- speciiie criteria. Specifically, we reveal that the semantics of space are determined by its basis, thereby enabling a set of descriptive words to approximate the basis for a cus- tomized feature space. Building upon this insight, given user-specified criterion, CRL first employs large language model (LL.) to generate descriptive texts construct the semantic basis, then projects the image representation into this condi- tional feature space leveraging a vision-language model (VLM). The conditional representation better captures semantics for the specific criterion, which could be utilized for multiple customized tasks. Extensive experiments on classification and retrieval tasks demonstrate the superiority and generality of the proposed CRL. The code is available at SccammngSSCU2225-NeelISSS--
Introduction
Representation learning aims at extracting meaningful patterns from raw data to create representations that are easier to understand and process. Its impact spans wide range of downstream tasks, such as classification and retrieval. In classification, representation learning enhances the discrimination and linear separability of features, significantly improving performance across diverse data modalities, including images [29], text [41], and video [59]. Similarly, in retrieval tasks, representation learning underpins efficient and accurate query-to-itent matching, evidenced by developments in image retrieval [1]] and cross-modal retrieval [50]. In recent years, driven by self-supervision techniques such as contrastive learning 6
and mask prediction 9 22 7k] 61], representation learn-
ing methods have undergone rapid advancements, leading to substantial performance improvements
across various fields, including graph 42 point-cloud [64), and skeleton 655
Though remarkable progress has been made, a crucial yet often overlooked question remains: What underlying criterion governs the learned representation? In fact, most existing representation learning methods inherently impose an implicit criterion. Previous research [56] has demonstrated
*Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

that representations learned by existing approaches exhibit a strong bias toward a single dominant aspect, typically "shape" or "category" - as these are the most salient features in many datasets. This inherent bias causes models to prioritize specific attributes while disregarding other potentially informative features, such as "texture" and "color". Consequently, the resulting universal embed- dings predominantly capture a single prominent criterion, leading to sub-optimal performance in downstream tasks that rely on alternative perspectives. As illustrated in Fig. | existing methods primarily identify the elephant "category". which is insufficient for customized tasks like population monitoring or habitat analysis. In comparison, our CRL could adaptively capture "count" and "scene" semantics, demonstrating broader generality. This narrow focus ultimately constrains the generaliza- tion capability of representation learning methods, underscoring the need for more adaptable and criterion- aware approaches.
To transform the image representation to align with specific criteria, a straightforward approach would be supervised fine-tuning [16] [35), where models are retrained using labeled data that ad- here to the given criterion. However, such a paradigm is not always practical due to the sub-
Existing Conventiona Represenatiinn Learning
Downstream Tasks
Population Monitoring
Animal Ident ficetion
Habitat Analysis
Limited
Generality Semantics
Elephant
Criterion
Universal (No Criterton)
stantial annotation effort required. In the un-
supervised scenario, where only images and a user-specified criterion are provided, a fea-
sible solution is to query visual question answer-
Proposed Conditional Repr nscaaaaion Learning
to extract rele- vant attributes from each image. However, this approach is computationally expensive and re- quires additional representation learning steps for the generated textual responses. With these considerations, an efficient way of learning the Anterion-oriente image representation is highly expected.
ing (VQA) models [52 69 31
Criterion
Count
Catagory
Scene
+
Semantics
Two
Elephant
Savanna
Broad
Generality
Downstream Tasks
Population Monitoring
Animal Identification
Habitat
Analysis
Figure 1: Existing conventional representation learning learns a universal representation that pri- oritizes the dominant semantics while overlooking other meaningful features, limiting their adaptabil- ity to customized tasks. In contrast, our proposed conditional representation learning (CRL) extracts representations conditioned on specific criteria, en- hancing its applicability.
In recent years, rescarchers have also been ex- ploring computationally efficient approaches Goal- target learning rep-
to learning useful representations.
conditioned works 46 [
resentations that meet the required outcomes or goal states. An area that is more closely related to our work is task-conditioned works (70] (2).
which aim to learns representations that reveal the underlying correlations among different tasks. For example, taskonomy 770 computes the optimal
transfer learning paths among tasks (point matching. reshading, etc.) to minimize the amount of required annotation. While there are certain commonalities between these works and ours, they haven't investigated the relationship between criteria and representations.
In this paper, we introduce Conditional Representation Learning (CRL), a novel approach that adapts the image representation to any user-specified criterion. Unlike conventional representation learning methods, which primarily focus on general-purpose feature extraction, CRL constructs customized feature space by leveraging the concept of basis transformation. The key insight behind CRL is that the semantics of a feature space are determined by its basis. For example, in a dimensional Cartesian coordinate system, the x, y, and unit vectors define the space, allowing for the decomposition of any vector. Similarly. in color theory. red, green, and blue serve as the basis for the trichromatic color space, enabling the synthesis of all perceivable hues. Extending this idea to high-dimensional semantic representations, well-chosen set of descriptive words can form basis for a customized feature space, which captures specific semantic properties aligned with user-defined criterion. Building on this perspective, CRL formulates conditional representation learning as a basis transformation process. Given a user-specified criterion, we first employ large language model (LLM) to generate a set of descriptive texts that serve as a semantic basis, spanning the relevant feature space. We then utilize a vision-language model (VLM) to encode both the generated texts and the images, obtaining their representations respectively. Finally, we project the image representation into the conditional feature space with the textual representation acting as a

---

basis. The transformed conditional representation would be more expressive under the specified criterion, which could be utilized for downstream tasks that require customized semantics.
The major contributions of this paper could be summarized as follows:
Different from conventional representation learning that primarily captures a single dominant semantics, we propose conditional representation learning (CRL), which enables learning representations tailored to arbitrary uper-spe criteria.
We formulate CRL as a basis transformation process, offering a computationally efficient and highly generalizable solution. It eliminates the reliance on supervised fine-tuning while substantially improving the applicability and interpretability of the learned representation.
Extensive experiments validate the effectiveness and generality of CRL in customized classification and retrieval, showcasing its superiority in seamlessly adapting to varying criteria and tasks.
2 Related Work
2.1 Representaion Learning
Representation learning aims to extract informative features from raw data, facilitating downstream tasks like classification and retrieval. As a classic method, autoencoder [24] learns compact repre- sentations through unsupervised reconstruction. Building upon it, denoising autoencoders [58] and variational autoencoders [77 have been proposed to enhance the robustness and structure of the learned latent representations. In the past few years, the field has further evolved with self-supervised learning techniques, which encourage models to learn semantical features by addressing pretext tasks such as patch and rotation prediction [10] 17), solving jigsaw puzzles 44 and colorization [72] A notable advancement in this direction is contrastive learning, exemplified by methods like SimCLR [6] and MoCo [23], which leverage instance discrimination to learn discriminative representations. More recently, the emergence of large language models (LLMs) such as GPT [5] and vision-language models (VLMs) like CLIP [48] has introduced more interpretable approach for representation learning. A series of works 774 45 38 21] have then researched using CLIP to improve zero-shot or few-shot image classification performance. By analyzing the Vision Transformer [[33 architecture of CLIP, studies such as Text-Span 14] have shed light on the underlying semantics captured by individual attention heads. Leveraging the strengths of L.LMs and VLMs, approaches like VCD (40) LaBo [66] and LM4CV 633 have demonstrated that interpretable representation learning can achieve performance on par with black-box methods in downstream image classification.
Despite significant progress, most existing representation learning approaches remain centered on a single criterion, typically "caaegory" or "shape", while overlooking other meaningful semantic di- mensions. This narrow focus limits the generalizability of learned representations, often necessitating extensive supervised fine-tuning when adapting to tasks that depend on alternative semantic cues. To address this limitation, we advocate for a paradigm shift from universal to conditional representation learning, underexplored yet promising direction. Specifically, our approach first constructs semantic basis composed of descriptive texts aligned with a user-specified criterion. Leveraging this customized basis, transform the image representation enable conditional adaptation, enhancing the flexibility and applicability of learned features without additional laborious fine-tuning.
2.2 Conditional Similarity
Conditional similarity refers to the similarity between samples based on specific criteria. This concept was first formalized by CSN (57). which learns multiple feature spaces to enable customized fashion item retrieval under different criteria. With the advent of representation learning, a series of tailored fashion retrieval approaches have been developed [39 [T] 12), significantly improving the retrieval performance. Recently, the idea of conditional similarity has gained traction the clustering domain [37] Driven by the powerful language processing capabilities of large-scale pre-trained models, ICITC (28 pioneers the concept of customized clustering by directly querying VLMs and LLMs to obtain clustering results based on specific criteria. However, this approach incurs high computational costs. To address this limitation, Multi-Map (68) introduces a more cost-efficient alternative, injecting customized semantics from VLM and the LLM to guide the clustering process.
3

---

Despite the success of existing methods, they are all delicately designed for specific tasks, limiting their generalization ability other domains. In contrast, we propose CRL, a simple yet effective method for learning general conditional representation, which could seamlessly adapt to diverse customized tasks.
3
Method
This section details the proposed Conditional Representation Learning (CRL) framework, which consists of basis construction and representation transformation. As depicted in Fig. [ given user-specified criterion, CRL first constructs a customized basis by querying an LLM about descriptive words. Subsequently, CRL computes the conditional image representation through basis transformation operation.
3.1 Basis Construction
Mathematically, basis refers to a set of linearly independent vector that span the entire space. For
example, in the three-dimensional Cartesian coordinate system, vectors (1, O, 0). (0, 1, 0), and (0, 0, 1), which denote the x, y. and axes, form a basis since any vector in the space can be expressed as linear combination of these three vectors. Analogously, "green", and "blue" form basis as they could compose all possible hues. From broader view, set of descriptive words related the user-specified criterion, that spans the customized feature space, intrinsically acts as the basis as well.
the trichromatic color space, "red",
To construct the basis under the specific criterion C, query LLM generate the related descriptive texts W via
W = LLM(M,
(I)
where denotes the LLM prompt template. As general solution, we use the following prompt for all customized tasks:
Generate common expressions to describe the C, as many as possible.
where C' is replaced with the user-specified criterion words such "color", "shape", "texture". etc. Notably, we incorporate additional instructions to encourage the LLM to produce formatted, comprehensive texts and avoid repetitions, which are detailed the Appendix.
Given the prompted query, the LLM would generate texts W semantically correlated with the user-specified criterion, transforming the abstract criterion into concrete textual basis. Once the descriptive texts W are obtained, we feed them into a VLM text encoder VLM to compute their normalized representation T via
VLM P2,, C, W),
where P2 denotes the VLM prompt constructed as follows:
Objects with the C of W.
It is worth noting that, when prior knowledge about the dataset is available, the word "Objects" could be replaced by more specific descriptions. The complete prompts used for all customized tasks in this paper, well as the LLM responses, attached in the Appendix.
As previously discussed, the text representation T could act as the basis spanning the customized feature space. Remarkably, compared with the basis of the classic universal feature space, the constructed basis T enjoys superior interpretability where each dimension has an explicit physical meaning.
3.2 Representation Transformation
After acquiring the text basis, we leverage it to transform the universal representation into the conditional representation, by projecting data into the constructed customized feature space.
In this paper, relax the linear independence requirement and allow redundancy in the constructed basis.

---

Object Color
Encoder
Criterion
LLM
VLM
Text Basis
Dimienaton 2
Image
Blue
Dimenslon Universal
Image
VLM
Universal Representation
Conditional
Conditional Feature Space
Feature Space
Representation
Figure 2: The overall framework of the proposed CRL. Given images and user-specified criterion (e.g., "color"), CRL first queries an LLM to generate descriptive texts semantically related to the criterion (e-g., "red". "green" and "blue"). Then, CRL encodes the generated texts and original images through a VLM. Subsequently, CRL projects the original image representation (e.g., dominated by "shape") into the conditional feature space spanned by the textual representation. The transformed conditional representation would be more expressive under the specified criterion and enjoy superior interpretability, facilitating customized downstream tasks.
To be specific, we first feed the images X into the VLM image encoder VI. image to obtain their normalized representation via
VLMimage (X).
Subsequently, we transform the image representation by projecting it to the customized space spanned by text basis T, namely,
R IT
where R denotes the transformed conditional representation. The validity of this transformation exploits the alignment between image and text modalities in the VLM's feature space. The conditional representation R emphasizes the attributes related to the user-specified criterion, and thus is more favorable customized tasks.
The complete process of our CRL is outlined in A To deliver a more intuitive understanding of CRL's working mechanism and underlying rationale, we provide an example about learning color-conditioned representation illustrated in Fig.
Consider the customized clustering task, which aims at grouping images based on their colors. The original image representation is dominated by the most significant shape information, which suboptimal for color-based grouping. To build customized feature space focusing on colors. we first query an LLM about the common colors. Supposing the LLM outputs descriptive texts W ("red", "green", "blue"), we calculate the text basis as
[ t 1
(5)
where (t1, t2, t3} denote the rows of T. corresponding to the representations of "red", "green", and "blue".
Then project the k-th original image representation i conditional representation T'k via Th
i T lix t1, in t2 in ta).
(6)
As shown in Eq. (6), the transformed conditional representation of the k-th image refers to the projection of its original representation the text basis T. Consequently, the three elements of r correspond to its degree of "red". "green", and "blue", respectively. In other words, тą is more expressive than & under the "color" criterion, leading superior performance on the customized clustering task.
4 Experiments
To assess the conditional representation learning performance of the proposed CRL, we apply it to two classic downstream tasks, including classification and retrieval. Notably, different from standard
5

---

Algorithm Conditional Representation Learning (CRL)
Input: Criterion C, LLM Prompt Pt VLM Prompt P2, Images X
Output: Transformed Conditional Representation R.
1: Query an LLM to generate the descriptive texts W related to the user-specified criterion C' via Eq.
2: Compute the text basis T via Eq.
3: Compute the original universal image representation I via Eq.
4: Transform into conditional representation R. via Eq. which could be then utilized for various customized tasks.
Spade
Diamond
Criterion
Jack
King
Number
Figure 3: A customized classification example of classifying poker cards based on the criteria of "suit" and "number", respectively.
representation learning, CRL focuses on learning conditional representation, and thus the downstream classification and retrieval based on various customized criteria. After that, parameter analysis is conducted to investigate the robustness of CRL.
4.1 Customized Classification
As shown in Fig. 3. customized classification aims to classify samples into different semantic categories under the specific criterion, which includes subtasks, i.e., supervised few-shot learning and unsupervised clustering.
4.1.1 Customized Few-shot Learning
Dataset. For this task, we utilize Clevr4-10k [56] and Cards (67 as benchmark datasets. Clevr4-10k a synthetic dataset consisting of 10, 531 samples and 4 distinct data partition criteria, categorized by "shape", "texture", "color", and "count", respectively. Cards is poker card dataset comprising 8, 029 samples, organized according to 2 criteria, i.e., "number" and "suit".
Setup. For fair comparisons, we adopt the logistic regression function from the scikit-learn pack- age 45 to perform few-shot learning, under the number of shots 5, 10 per class, respectively. To alleviate the influence of randomness, we stochastically select the training data 20 times for each shot and report the mean result. As for the backbone, we adopt ViT-B/32 pre-trained on CLIP, keeping the same with Section 4.2
Metric. For the task of customized few-shot learning, we adopt accuracy (ACC) as the evaluation metric.
Baseline. We conduct comparisons between proposed CRL and image representations of CLIP 48 ALIGN 25 and MetaCLIP 62 across six semantic criteria.
Performance. As illustrated in Table. [] CRL achieves a noticeable improvement over CLIP, ALIGN and MetaCLIP across most experimental settings, with mean accuracy gain nearly 10%. Particularly, CRI gains significant improvements when the target criterion differs substantially from the originally dominant one, such "color" (nearly +40% at I-shot). The consistent performance advantage indicates that CRL's representation exhibits better generality under multiple criteria.

---

Table 1: Performance the task of customized few-shot learning.
Clev44-10k
Method
Texture
Shape
Color
Mean
10
5
10
5
10
CLIP 48
17,46
29.39
58.16
83.17
89.47
26.85
57.33
70.00
52.01
ALIGN
25
18.80
34.35
45.22
73.40
91.82
95.02
20.08
41.89
56.45
53.00
MetaCLIP 62
17.68
30.96
39.03
70.13
91.69
95.47
22.37
46.71
61.74
52.86
BLIP2 30
15,93
25.23
32.58
72.91
95.18
97.88
28.96
60.53
73.25
55.83
CLIP+CRL
18.76
35.54
45.54
58.67
86.61
92.29
65.28
88.89
93.08
64.96
ALIGN+CRL
92.74
60.26
87.
92.56
MetaCLIP-CRI
18.14
34.89
44.69
66.36
92.01
95.50 50
62.41
88. 45
92.50
66.11
BLIP2+CRL
16.35
34.67
47.28
73.22
95.12
97.90
63.75
86.16
92.13
67.40
Clevr4-10K
Cards
Method
Count
Number
Suits
Mean
10
5
10
5
CLIP 48
17.50
23.43
25.45
20.63
33,73
41.84
37.65
56.36
65.98
35.84
ALIGN
25
14.64
21.63
25.16
16.97
24.70
29.15
34.67
52.75
61.78
31.27
MetaCLIP
16.61
22.64
24.92
37.47
55.03
65.16
20.71
35.16
42.97
35.63
BLIP2
30
16.92
25.63
29.38
27.21
45.54
55.94
44.61
70.14
78.16
43.73
CLIP+CRL
23.38
29.59
32.40
17.66
44.52
$1.09
37.10
67.16
72.64
41.73
ALIGN+CRL
18.16
32.62
36.80
17.39
30.61
35.93
42.13
76.36
80.11
41.12
MetaCLIP-CRL
17.36
26.29
29.93
42.32
71.88
77.32
25,30
50.53
56,90
44.20
BLIP2+CRI.
23.06
34.86
39.07
23.47
61.19
70.05
49.57
80.44
84.06
51.75
Table Performance on the task of customized clustering.
Clevr4-10k
Method
Texture
Shape
Color
Mean
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
CC 32
0.16
11.34
94.66
96.89
93.90
16.54
11.42
0.07
36.11
SCAN 54
0.41
11.97
0.86
90.99
89.10
84.03
0.20
11.51
0.01
32.12
Multi-Map
68
3.77
17.25
1.81
67.48
66.01
57.40
56.83
56.46
45.73
41.42
CLIP 48
13.09
74.22
73.19
64.15
0.83
12.23
0.27
26.61
ALIGN 25
1.36
13.30
0.41
89.35
86.77
83.37
0.47
11.79
0.10
31.88
MetaCLIP
71.58
BLIP2 [30
0.79
12.32
0.28
86.98
85.68
81.17
0.99
11.92
0.24
31,15
CLIP&CRI
10.74
25.11
6.35
78.69
83.05
72.42
88.67
88.05
82.30
59.49
ALIGN+CRL
15.08
26.08
9.18
88.27
87.63
81.83
85.07
76.15
72.69
60.22
MetaCLIP+CRI
12.74
25.89
7.28
87.32
88.15
82.98
88.35
86.27
81.08
62.23
BLIP2+RLL
84.52
84.67
81.97
74.85
Clevz4- 10k
Cards
Method
Count
Number
Suits
Mean
ACC
ARI
NMI
ACC
ARI
NM1
ACC
ARI
CC 32
26.34
12.30
24.94
39.21
SCAN 154
3.42
14.29
1.23
11.11
18.21
17.60
15.01
32.02
9.48
13.60
Multi-Map
11.38
20.13
16.32
20.61
7.95
14.02
46.65
11.08
17.31
CLIP 48
9.50
19.02
5.70
16.84
18.91
8.44
16.52
43.74
12.93
16.84
ALIGN 25
0.63
12.50
0.19
14.86
17.51
6.47
3.49
31.72
2.31
9.96
MetaCLIP 62
7.62
17.27
3.97
17.39
19.78
9.04
15.48
38.72
13.11
15.82
BLIP2 30
6.11
16.36
3.13
24.34
25.25
13.08
31.26
47.04
22.25
20.98
25.57
12.14
39.71
67.15
ALIGN+CRL
22.78
26.59
12.18
20.12
25.32
10.24
42,94
50.79
34.47
27.27
MetaCLIP+CRI
12.22
20.80
6.18
39.07
41.63
24.37
45.19
58.71
36.97
31.68
BLIP2+CRL
28.55
30.92
16.28
46.55
48.35
32.31
60.86
76.07
55.94
43.98
4.1.2 Customized Clustering
Dataset. We continue perform experiments on Clevr4-10k and Cards datasets for the task of customized clustering.
Setup. We directly conduct k-means on the representations obtained by CRL get the clustering. Keeping the same as the customized few-shot learning, we also perform k-means 20 times and report the average clustering result. As for the backbone, we follow the previous method (68), adopting ViT-B/32 pre-trained on CLIP.
Metric. Three widely used clustering metrics, namely Normalized Mutual Information (NMI), Accuracy (ACC), and Adjusted Rand Index (ART), are used for evaluation. Higher scores indicate better clustering results.

---

Gent Preee
Gare Soond Ofject
Wong Sceee
Query Image
Sare Soor
Seete Sette dioect
Witno Soare
Toe
Focus on an object
(b) Change an object
Figure 4: Two settings of the customized similarity retrieval task.
Table 3: Performance on the task customized similarity retrieval. The symbol means using the fine-tuned CLIP model weights.
Focus
Change
Method
Mean
R@1
R@2
R@
R@1
R@2
R@3
CLIP
9.4
17.0
25.4
7.6
17.1
25.5
17.0
CLIIP
7.4
14.0
23.0
8.1
16.4
24.7
15.6
CLIPanarsten
11.5
20.1
29.2
9.8
20.0
28.9
19.9
Pie2Word 449
19.3
27.4
18.2
26.1
SEARLE
10.8
18.2
27.9
15.6
25.8
17.8
LinCIR 20
10.1
19.1
28.1
7,9
16.3
25.7
17.9
CIG [60
10.6
19.2
27.4
7.9
16.9
25.4
17.9
CLIP++CI
15.4
26.7
35.8
17.0
27.8
37.8
26.8
Combiner" [55]
16.6
27.7
37.2
18.0
32.2
41.6
28.9
CLIP+CRL"
19.7
32.7
41.3
21.0
35.9
44.8
32.6
Baseline. We first compare CRL with two traditional clustering methods, CC [32 and SCAN 54
Furthermore, incorporate Multi-Map [68]. customized clustering approach that leverages the CLIP model, into the comparison. Additionally, we report the performance of k-means clustering applied to the image representation of CLIP, ALIGN and MetaCLIP, to provide an intuitive bascline analysis.
Performance.As shown in Table. CRL gains consistent performance improvement compared with the original CLIP, ALIGN and MetaCLIP. In particular, CRL obtains an ACC boost of CLIP over 75% on the color criterion. This improvement can be better visualized by T-SNE [53], shown in the Appendix. Though traditional clustering methods exhibit some superiority on the "shape" criterion, CRI. achieves consistently better results on other criteria. This implies that traditional clustering methods have strong bias towards a single criterion, yet lack the flexibility and capability to cluster data based other meaningful criteria.
4.2 Customized Retrieval
For customized retrieval, we also conduct experiments on its two subtasks, namely, customized similarity retrieval and customized fashion retrieval. Given a query image and a condition object, customized similarity retrieval aims to retrieve the most conditionally similar image from candidates, illustrated in Fig. 4 As shown in Fig. S customized fashion retrieval searches all candidate images of fashion items, which share the same value as the query image under the specific criterion.
4.2.1 Customized Similarity Retrieval
Dataset. We adopt GeneCIS (55) as the benchmark for this task, which comprises two settings. As shown in Fig. (a) "Focus" setting aims retrieve the candidate that contains both the same scene (e.g., living room) and the condition object (e.g., table) as the query image.
(b) In contrast, the "Change" setting requires the target image to maintain the same scene (e.g., railway) as the query image while including the condition object (e.g., tree) that is absent in the query image.
Setup. This benchmark involves two factors, namely, object (text condition) and scene (query image). To employ CRL, we ask the LLM for the common scenes, obtaining the conditional representation of the query and candidate images. Then calculate and sum the similarities of these two factors for retrieval. This operation is detailed in the Appendix. Additionally, we use ViT-B/16 pre-trained on CLIP the backbone, following the previous work (55)
Metric. The recall rates R@ 1, R @22, and @@3 serve as the evaluation metrics for this task, Higher recall rates imply better retrieval results.

---

Quary Image
Images
Sty
Figure 5: An example of the customized fashion retrieval task. Given a criterion, it searches all the candidate images that share the same value as the query image.
Table 4: Performance on the task of customized fashion retrieval. The symbol signifies that no training conducted.
Method
Texture
Fabric
Shape
Part
Style
Mcan
Random
6.69
2.69
3.23
2.55
1.97
3.38
Triplet
13.26
6.28
9.49
4.43
3.33
7.36
CSN
14.09
6.39
11.07
3.49
8.01
ASEN 39
7.11
12.39
3.56
8.74
ASEN++ [1]]
15.60
7.67
14.31
6.60
4.07
9.64
RPF 12
15.62
8.30
15.02
7.38
4.77
10.22
CLIP 48
9.14
4.68
7.86
4.26
4.48
6.08
CLIP+CRL
11.03
6.76
11.80
5.56
4.42
7.93
CLIP+CRL
16.88
9.31
16.98
7.54
5.95
11.33
we first provide three simple CLIP-only baselines, namely CLIP image CLIP and CLIP maggettxt. detailed in the Appendix. In addition, we include five retrieval baselines Pic2Word 44), SEARLE 4, LinCIR (20). CIG 66)] and Combiner [55] for benchmarking. Notably, Combiner leverages the external dataset CC3M [51] to fine-tune the CLIP model. Thus evaluate the performance of CRL under two scenarios: using the original CLIP weights and using the weights
Baseline. Following 55
fine-tuned by Combiner.
CRL demonstrates substantial improvements over the original CLIP baselines, achieving notable gain of 6.9% in the mean recall. When leveraging fine-tuned CLIP weights, CRL further extends its advantage, surpassing Combiner by 3. 3.7% in mean recall, simultaneously maintaining consistent performance gains across all metrics.
Performance. can be observed from Table, 3
4.2.2 Customized Fashion Retrieval
Dataset. Following previous works [39], DeepFashion (36 as the evaluation dataset for this task, which consists of 221k 27k 27k images for training/ / validating testing. This benchmark has 5 criteria, namely, "texture", "fabric", "shape", "part" and "style", with 156, 218, 180, 216, and 230 values, detailed in the Appendix.
the category and attribute prediction benchmark of
Setup. We first obtain the embeddings by CRL in training-fri free manner. After that, seamlessly append a two-layer MLP to the embeddings, subsequently training this MLP and the backbone. The training process is detailed in the Appendix. In addition, following previous works, ViT-B/16 is adopted as the backbone for this task.
Metric. Following existing works, use the Mean Average Precision (MAP) the evaluation metric for the customized fashion retrieval task. Higher MAP values indicate better retrieval results.
Baseline, We first add a Random baseline, which randomly sorts all the candidate images. Moreover, we provide Triplet baseline, which uses the standard triplet ranking loss [57] to train joint embedding space. Further, we compare CRL with 5 state-of-the-an fashion retrieval methods, including Triplet [57], CSN [57], ASEN (3)] ASEN+ [ and RPF [22]. Besides, we also provide CLIP baseline that embeds all the images with the image encoder.
Performance. As shown in Table CRL achieves notable improvements over the CLIP baseline in training- free manner, with a relative mean MAP gain of 30%.
Once the training is completed, CRL establishes new state-of-the-ar performance, surpassing the best competitive method RPF by 10% relativelty mean MAP. These results further validate CRL's effectiveness in customized tasks and its compatibility with model fine-tuning strategies.

---

50-
Moa 40
Simillaity Ressooal
Figure 6: Performance with different numbers of texts.
4.3 Analysis on Textual Basis
To prove the robustness of CRL, examine CRL's performance based the CLIP model for the above-mentioned four customized tasks under varying levels of textual basis. To be specific, we explicitly control the number of generated descriptive texts and report the mean value of each task here, while the complete results can be seen in the Appendix. As Fig. @ishows, CRL achieves stable performance for different numbers of texts except when the number CRL is a robust method for various tasks, as long as there a reasonable number of descriptive texts establish the semantical basis for the customized space. More ablation studies can be found in the Appendix.
small. In other words,
Limitation
Based observations and experiments, found that method suffers from two main limitations. Firstly, despite its generalizability across different criteria, it may not outperform clustering methods like CC and SCAN under the universal criterion "shape". This is likely because these methods employ specially targeted designs for clustering under this criterion. Anyway, we acknowledge that CRL is not optimal on the universal criterion. Secondly, our method only roughly approximates the basis. We've tried various strategies to filter the texts generated by the LLM, but none have proven be effective across all criteria. Nevertheless, are confident that better strategies could be devised to acquire the text basis.
Conclusion
In this paper, identify fundamental limitation of existing representation learning methods: they predominantly derive universal embeddings that capture the most salient semantic features, making them suboptimal for customized tasks that prioritize non-dominant semanties. To address this, propose CRL, simple yet effective conditional representation learning method that adapts the univer- sal representation to specific criteria through basis transformation process. In brief, CRL utilizes large language model (L.LM) and a vision-language model (VLM) to generate textual descriptors that are semantically aligned with the user-specified criterion. These descriptors form an interpretable text basis, guiding the transformation of the image representation to enhance its expressiveness under the given criterion. Extensive experiments validate the effectiveness and generality of CRL across diverse tasks and criteria. By shifting the focus toward conditional representation learning, an underexplored yet promising paradigm, we hope this work could spark new insights and foster further research in this direction.
Acknowledgements
This work supported in part by NSFC under Grant 62176171, U21B2040, 623B2075, 62472295; in part by China National Postdoctoral Program for Innovative Talents under Grant BX20250392; in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303; and in part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Gpt-4 technical report. arXiv preprini wrXiv:2333.08774 2023.
[2] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6430-6439, 2019.
[3] Anthropic. Claude opus claude sonnet 4 system card. Technical report, Authropic, May 2025.
[4] Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338-15347, 2023.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances neural information processing systems, 33:1 187--190,, 2020.
[6] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758 2021.
[8] Gheorghe Comanici, Eric Bieher, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, ct al. Gemini 2.5: Pushing the frontier with ad- vaneed reasoning. multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
[9] Jacob Devlin. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint Xiv:: 2018.
[10] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Pruceedings of the IEEE international conference computer vision, pages 1422-1430. 2015.
[11] Jianseng Dong, Zhe Ma, Xiaofeng Mao, Xun Yang, Yuan He, Richang Hong, and Shouling Ji. Fine-grainee fashion similarity prediction by attribute-specific embedding learning. JEEE Transactions on Image Processing, 30:8410-8425, 2021.
[12] Jianfeng Dong. Xiaoman Peng. Zhe Ma. Daizong Liu, Xiaoye Qu Xun Yang, Jixiang Zhu, and Baolong gagrruund-bckkrruuddd contrastive Icarning for fine-grained
Liu. From region to patch: Attribute-aware fashion retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1273-1282, 2023.
[13] Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1rXiv:2010.1192, 2020.
[14] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint 2xXiv:2310005911,22222
[15] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng and Qiao. Clip-adapter: Better vision-languee models with feature adapters. International Journal of Computer Vision, 132(2):581-55)) 595, 2024.
[16] Xiwen Geng, Suyun Zhao, Yixin Yu, Borui Peng, Pan Du Hong Chen, Cuiping Li, and Mengdie Wang. Personalized clustering via targeted representation learning. arXiv preprint arXi::2412.13690, 2024.
[17] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint (rXiv:1803.07777 2018
[18] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part VI 14, pages 241-257. Springer, 2016.
11

---

[19] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond. Elena Buchatskaya. Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems. 33:21271- -22284, 2020.
[20] Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang. and Sangdoo Yun. Language-only training of zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13225 13234, 2024.
[21] Yuncheng Guo and Xiaodong Gu. Mmrl: Multi-modal representation learning for vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25015-25025, 2025.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 9729-9738, 2020.
[24] Geoffrey Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, (004786):50455000 2006.
[25] Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Lc, Yun- Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[26] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
[27] Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint 1t..iv:1112.6//11 2013.
[28] Sehyun Kwon, Jaeseung Park, Minkyu Kim, Jaewoong Cho, Ernest K Ryu, and Kangwook Lee. Image clustering conditioned on text criteria. arXiv preprint arXiv: 282.18877, 2023.
[29] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 577-593. Springer, 2016.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- Bootstrapping language -magge pre pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742, PMLR, 2023.
[31] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, Oscar: Object-semantics aligned pre-training for lasionnlanguage tasks. In Computer Vision- -EconnECCV22 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[32] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Jocy Tianyi Zhou, and Xi Peng. Contrastive clustering, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8547-8555, 2021.
[33] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Vision-ECC ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[34] Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint Xr1iv:2412. 191.7 2024.
[35] Honglin Liu, Peng Hu, Changqing Zhang, Yunfan Li, and Xi Peng. Interactive deep clustcring via value mining, Advances in Neural Information Processing Systems, 37:42369 -46877, 2025.
[36] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages (096-1104, 2016.
12

---

[37] Yiding Lu. Haobin Li, Yunfan Li, Yijie Lin, and Xi Peng. A survey deep clustering: from the prior perspective. Vicinagearth, 1(1):,, 2024.
[38] Xiaosong Ma, Jie Zhang. Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems, 36:65252-652666 2023.
[39] Zhe Ma, Jianfeng Dong, Zhongzi Long, Yao Zhang, Yuan Hc, Hui Xue, and Shouling Ji. Fine-grained fashion similarity learning by atrribue-specific embedding network. In Proceedings of the AAAI Conference on artificial intelligence, volume 34, pages 1 11741-1 1148,, 2020.
[40] Sachit Menon and Carl Vondrick. Visual classification via description from large language models, arXiv preprint 1rX00:2210.07/83, 2022.
[41] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[42] Yujie Mo. Liang Peng, Jie Xu, Xiaoshuang Shi, and Xiaofeng Zhu, Simple unsupervised graph repre- sentation learning. In Proceedings of the AAAI conference on artificial intelligece, volume 36, pages 7797-7805, 2022.
[43] Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE international conference on robotic computing (IRC), pages 590-595, IEEE, 2019.
[44] Mehdi Noroozi and Paolo Favaro. Unsupcrvised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69 84, Springer, 2016.
[45] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn Machine leaming in python. the Journal of machine Learning research, 12:2825-2834) 2011.
[46] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville, Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[47] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15691-15701, 2023.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR, 2021.
[49] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li. Chen-Yt Yu Lee, Kate Saenko, and Tomas Plister. Pie2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305-19314, 2023.
[50] Nikolaos Sarafianos, Xiang Xu, and loannis A Kakadiaris. Adversarial representation learning for text-to- image matching. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5814-5824, 2019.
[51] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmmed image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volune I: Long Papers), pages 2556-2565, 2018.
[52] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[53] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008
[54] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European conference on compuder vision, pages 168-285. Springer, 2020.
13

---

[55] Sagar Vaze. Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. pages 6862-6872, 2023.
[56] Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. Advances in Neural Information Processing Systems, 36, 2024.
[57] Andreas Veit, Serge Belongic, and Thcofanis Karaletsos. Conditional similarity networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 830-838, 2017.
[58] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol, Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1 103, 2008.
[59] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 504-521. Springer, 2020.
[60] Lan Wang, Wei Ao, Vishnu Naresh Boddeti, and Ser-Nam Lim. Generative zero-shot composed image retrieval.
[61] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuaa Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for sulf-supe visual pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14668-14678, 2022.
[62] Hu Xu, Saining Xie, Xiaoging Ellen Tan, Po- Yao Huang. Russell Howes, Vasu Sharma, Shang-War Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
[63] An Yan, Yu Wang. Yiwu Zhong, Chengyu Dong. Zexue He, Yujie Lu, William Yang Wang. Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3090-3100, 2023.
[64] Siming Yan. Zhenpei Yang, Haoxiang Li, Chen Song. Li Guan, Hao Kang, Gang Hua, and Qixing Huang. Implicit autoencoder for point-clos coint-cloud self supervide representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14530-14542, 2023.
[65] Di Yang, Yaohui Wang. Antitza Dantcheva, Lorenzo Garattoni. Gianpiero Francesca, and François Brémond. View-invariant skeleton action representation learning via motion retargeting. International Journal of Computer Vision, (1177))2351-2366 2024.
[66] Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19187- 19197, 2023.
[67] Jiawei Yao, Enbei Liu, Maham Rashid, and Juhua Hu. Augdme: Data augmentation guided deep multiple clustering. Procedia Computer Science, 222:571-580, 2023.
[68] Jiawei Yao, Qi Qian, and Juhua Hu. Multi-modal proxy learning towards personalized visual multiple clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 40666-40075, 2024.
[69] Zhou Yu. Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281 16200, 2019.
[70] Amir R Zamir, Alexander Sax. William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018.
[71] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised lcarning via redundancy reduction. In International conference on machine learning, pages 12310-12320 PMLR, 2021.
[72] Richard Zhang, Phillip Isola, and Alexii A Efros. Colorful image colorization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 649.666, Springer. 2016
14

---

[73] Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-supervised visual representations learning by contrastive mask prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10160-10169, 2021.
[74] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models, International Journal of Computer Vision, 130(9):2337-2344 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Experiment section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: The data used the paper is available to everyone. We are now organizing our code and will release it soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
publ ic /eteddeee/eeededeeee
ss Loooolic cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ ctt.ps.ccppubbic
CooeeoodeSSSaa Submisbbbsss ss cessoooPllcyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would be too computationally expensive for us since extensive experiments were conducted and we don't have enough computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments were conducted on single Nvidia RTX 3090 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper citations are provided throughout the document and the licenses will be included with the code when it is released.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We provide the complete prompts and the usage of the LLMs for the all customized tasks in the Appendix.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttprr////errs.....e c/tp////sssscc///offfeeeeee///// httpsssccccccoeecccecc////////ee LLM) for what should should not described.
2025/
22

---

Appendix
In the Appendix, we provide supplementary information on the four customized tasks that are briefly introduced in the main paper. The Appendix is organized according these four tasks, with each section dedicated to elaborating on one specific task. In the end, we provide ablation experiments for different LLMs, temperatures, and prompts of the LLM and VLM.
A Customized Few-shot Learning
A.1 Dataset Description
We adopt Clevr-4 [56] and Cards 67 as benchmark datasets for this task. Based on the CLEVR dataset [26], Clevr-4 is synthetic benchmark that introduces four distinct yet equally valid groupings of the data, namely, "texture", "shape". "color" and "count". It employs computer graphics tools to generate images featuring multiple objects positioned within fixed scenes, shown in Fig. 7 As for Cards, it contains 8,029 images of poker cards, categorized along two independent dimensions: card number (such as Ace, King, Queen) and suit type (clubs, diamonds, hearts, spades).
Texture: zigtag
Texture: zigtag
Texture: metal Shape: monkey
Shape: cylinder
Shape: sphere
Color: yellow
Color: purple
Color: cyan
Count:
Count: 4
Count:
Figure 7: Three examples of clevr4 dataset.
A.2
Prompt and Response
For the task of customized few-shot learning, take the criterion of "Color" example.
A.2.1 LLM Prompt
Please generate common expressions ons to describe the color, as many as possible, formatted as: ["...", "...", ", "..."]. knsure all items are unique and written in single line, without any nested lists or additional formatting. You may describe the same color in different ways,
such as "red", "crimson", or "scarlet". Only generate the list, and do not include any additional information.
where the underlined part corresponds to some example synonyms to encourage diversity in the generated descriptive texts.
In addition, control the number of the generated texts by the GPT-4, we modify the prompt as follows:
Please generate 100 expressions to describe the color, formatted as: ["...", "...", "....]. Ensure all items are unique and written in a single line, without any nested lists additional formatting. You may describe the same color in different ways, such as "zed", "crimson", or informaticn.
"scarlet". Only generate the list, and do not include any additional
Due to the output length limitation of the LLM, we need to repeat the inquiry several times until obtaining the specific number of generated texts. Since this modification is very slight, we only present here once for clarity.
23

---

A.2.2 LLM Response
M"red", "erimson", "scarlet", "ruby", "chezry", "cose", "buzgundy", "cardinal", "wine", "Liretruck red", "Terrari red", "carmine", "sangris", "apple red", "poppy", "raspbersy", "pink", "Elush", "fuchsia", "magents", "hot pink", "baby pink", "ccral", "peaca", "apricot", "salmon", "Light pink", "ponder pink", "lavender", "violet", "indigo", "puzple", "amethyst", "grape", "orchid", "eggplant", "plum", "lilac", "mauve", "periwinkle", "blue", "navy", "azuze", "aky blue*, "cobalt", "cerulean", "sapphire", *eleatric blue", *royal blue", "baby blue", "powder blus", "tesl", "turqucize", "aqua", "seafocn", "green", "emarald", *forest green", "mint", "lime", "olive", "kelly grean", "sage", "pea green", "jade", "chartreuse", "yalles", "gold", "lemon", "butter", "canary", "mustard", "sunflover", "cream", "ivozy", "beige", "tar", "coffee", "chocolate", "brown", "copper", "rust", "maloogay"", "walnu:", "espreaso", "gray", "slate", "charcoal", "peuter", "silves", "platinum", "black", "coal", "ebony", "jet black", "onyx", "anow", "ivory", "unite", "pearl", "bone", "egeshell", *vanilla"]
A.2.3 Ground Truth Label
(Tyaay", *ced", "blue", "green", "brown", "purple", "cyan", "yellow", "pink", "orange"]
A.2.4
VLM Prompt
Objects with the color of red.
objects with the color of green.
Objects with the color of blue.
A.3 Experimental Setting
After multiplying the same text basis, the gap between images shrinks. To accelerate the few-shot learning process, we normalize the transformed conditional representation to have zero mean and unit variance.
A.4 Performance
As shown in Table 5 CRL achieves stable few-shot learning results under different numbers of L.M--generaee descriptive texts, except when the text number is too small.
B Customized Clustering
B.1
Dataset, Prompt and Response
For the task of customized clustering, the same datasets, prompts and LLM responses the customized few-shot learning task. Therefore, we omit the repeated descriptions here.
B.2 Improvement Visualization
CRL achieves the representation projection from the original feature space (which is often dominated by the "shape" criterion) to the conditional feature space, making it more expressive under the specified criterion. Fig. X shows the T-SNE visualizations of the original CLIP representation and CRL representation, from which one can clearly observe the improvement.
B.3 Performance
CRL maintains consistent clustering performance under different quantities of --g--rreertted texts, presented in Table 6 with drop only when the number of texts is very limited.

---

Table 5: Customized few-shot learning performance under different numbers of texts.
Clevr4-10k
Text-num
Texture
Shape
Color
Mcan
5
10
16.98
25.79
29.68
52.86
71.87
78.59
47.09
70.19
75.75
$2.09
20
18.02
36.25
53.51
75.78
82.79
55.02
83.05
88.02
58.05
50
18.58
34.67
44.01
56.82
82.19
88.57
61.81
86.70
91.57
62.77
100
19.19
36.73
47.11
57.47
84,64
91.13
61.86
87.12
92.20
64.16
200
20.49
39.23
50.08
54,96
84.38
91.48
66.82
89.60
93.68
65,64
55.04
84.62
91.34
93.28
65.53
400
20,54
39.94
51.14
55.39
84,95
91.57
65.16
88.51
93.07
65.59
500
20.36
39.83
50.98
55.09
84,86
91.59
64.44
88.11
92.83
65,34
Clevr4- 10k
Cards
Text-num
Count
Number
Suits
Meun
10
5
10
10
10
23.94
30.94
33.31
14.35
31.67
36.61
33.79
50.44
55.25
34.48
20
22.68
29.49
31.68
16.01
39.40
46.85
37.14
56. 70
61.75
37.97
50
22.34
29.05
32.01
16.08
40.75
48.95
37.52
62.43
69.25
39.82
100
21.92
28.64
31.58
15.43
39.53
48.99
37.80
63.69
71.04
39.85
200
22.06
31.16
41.81
51.63
37.26
73.89
41.02
300
21.11
27.56
30.71
16.75
42.30
52.95
36.41
66.33
73.86
40.89
400
21.53
28.04
31.01
16.82
42.50
53.08
35.89
65.90
73.36
40.97
500
21.48
28.04
31.06
16.90
43.22
54.24
35.73
65.82
73.56
41.12
(a) CLIP
(b) CRL
Figure 8: T-SNE visualizations of the representations obtained by CLIP and CRL, under the "color"
criterion the Clevr4-10k dataset.
C Customized Similarity Retrieval
For this task, the criterion is "Scene." Below, we present both the prompt used and the corresponding results generated. It's worth noting that there are ground truth labels for this task.
C.1 Dataset Description
In both settings, the candidate images are required to share the same scene as the query image and satisfy the given object condition. The difference lies in the presence of the object condition the query image. In the "Focus on an object" setting, the query image contains the object condition, while the 'Change an object' setting, the query image doesn't. In other words, the "Focus" setting retrieves positive target, while the "Change" setting searches for negative target. Both settings consist 960 query images sourced from the classical CoCo dataset, with each query image corresponding to 10-15 candidate images in the gallery.
25

---

Table 6: Customized clustering performance under different numbers of texts.
Clevr4-10k
Text-num
Texture
Shape
Color
Mcan
NM1
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
10
12.79
26.37
7.32
67.27
66.68
54.59
55.50
58.23
40.50
43.25
20
13.57
8.04
67.92
68.86
57.00
78.74
67.79
$1.67
50
11.94
24.25
6.76
74.89
78.6
67.08
85.90
86.11
78.88
57.16
100
10.62
23.29
5.88
77.72
80,61
70.41
85.20
82.92
76.27
56,99
200
13.16
26.49
7.97
75.71
78.78
67.34
88.73
86.68
81.40
58.47
300
12.58
74.78
76.28
86
57.52
400
11.90
24.91
7.12
74.79
78.58
67.19
87.92
85.07
80.33
57.53
500
11.13
24.44
6.64
74.13
77.62
66.58
87.66
86.61
80.51
57.26
Clevr4-10k
Cards
Text-num
Count
Number
Suits
Meun
NMI
ACC.
ART
NMI
ACC
ARI
NMI
ACC
ARI
10
27.14
32.08
15.30
22.50
26.72
10.54
33.72
4.18
19.64
20
25.03
27.83
12.62
18.35
22.88
9.02
24.21
50.01
19.44
23,27
50
21.33
26.13
11.57
17.86
23.03
9.47
27.14
56.68
23.78
24.11
100
21.90
25.64
11.31
16.51
21.61
8.25
27.67
57.03
24.64
23.84
200
20.41
24.51
10.19
22.72
8.90
29.84
53.96
26.86
23,90
300
15.79
22.59
7.80
16.95
22.00
8.23
28.48
53.62
25.60
22.34
400
17.58
23.71
8.66
15.79
20.47
7.57
28.35
53.17
25.26
22.28
500
17.33
23.38
8.58
15.89
20.65
7.61
28.69
53.62
25.65
22.38
Query Image
Object
Same Scone condition Object
Same Scane No Condition Object
Wroon Scone Condition Object
Pizza
Focus object
Image
Object
Condition ofject
Same Scone
No Condition object
Scone
Condition Object
Scene
Mountain
(b) Change object
Figure 9: Two settings of the customized similarity retrieval task.
C.2 Prompt and Response
C.2.1 LLM Prompt
Please generate common expressions to describe the scene, as many as possible, formatted as: ["...", " "...", " "...", "..."I, Ensure all items are unique and written in a single line, without any nested lists additional formatting. You may describe the sare scene different ways, such as "a cozy living room", "a snug Lounge", warm and inviting sitting area". Only generate the 1ist, and do not incluce any additional information.
C.2.2 LLM Response
(*a spacious living room", "s cozy lounge", "s snug sitting area", "a well-lit kitchen", "a modern dining room", t atylish becroon", "an airy home office", * rustic cabin interior", minima studio apartmens", * vibrant indoor garden", home library", ta charming ba lonny"" "a serene patio", " Iush backyard", "a sunlit gardent, "a peaceful park*, * sceric mountein viex", bustling city street", quiet suburban street", sunny beach",
26

---

serene lakeside view", "a dense forest", "a tranquil meadow", "a wild desert landscape", "a foggy morning in the woods", "a snowy winter scene", "s lively cuzdoer market", "a cozy backyard barbecus", "a sun-drenched terrace", *a secluded countryside lane", *a colorful street art acene", * tropicsl rainforest", "a modern rooftop tarrace", *an open-air café", "a bright sunny veranda", *a quiet rivarbank", "a sunny playground", "a rugged coastlire", "a pictuzesque vineyard", "a lively park", "a quiet pond", "a bustling urban square", "a scenic hillside view", charming street cozner", "an expansive field of flowera", "a peaceful plenic spot*, *a lively street festival", "a serece walking path", "a bus:ling Lown square", "a remote mountain pass", *a tranquil beachfront cabin", "a sunny picnic area", "a dense jungle trail", "a charming country road", "a misty hilltep view*, "a windy coastal cliff", "a lush trepical beach", "an open farm field", quiet coastal town", "a vibrant city plaza", "a bust Hnn shopping district", "a colorful flower garden", *a charming riverside walk", "a large public park", "a peaceful countryside landacaps", "a warm outdoor te"race", *an outdoor wodding venue", "a lively anusement park", "a remote fishing village", ta scenic coastal highway", "a green city park", "a vibrant batenical garden", "a sondy dune landscape", *a coay cabin porch", " charming old town square", "s quiet beachaide café", "a aunny urben Lerzace", "a tranquil garden couzlyard", "a vibrant farmers' market", "a lively caraival scene", "a remote mountain village", "a rustic barnyard*, ta calm rural road", "a dranatis eliffside view", "z lively beachfrent promenade", "a peaceful lakeshore cabin", "a quiet sural pathway", "a lush botanical park", "a scenic hiking trail", "s lively town market", "a secluded desert casis", "a colorful coastal town", "a quiet fores: cleasing", "a scenis boat dack*, "s vast open field", "a serene oliffaide walk", *a lively open-air concert", "a quiet hillside retreat", "a bright tropical beach", "a calm sandy shore", "a warn outdoor patio", "a charming outdcor café", "a vibrant city park*, *a pascaful desert sunset", "a lush green terrace", "a rustic lakeside cabin", "a bright gardan path", "a misty river valloy", "a bustling port Lown", " quiet mountain retieat", "a Lranquil city couctyard", "a picturesque Lown hacbos", * lively street market", "a scenic desert plateau", "a quiet neighborhood street", * charning sesside village", calm beachside retreat", "a misty forest trail", "s sunny farm field", "a lively city park", "a charning cobblestone street", "a peaceful urban coursyard", "s tranquil village square', "a vibrant mountain town", "a lively marina", "a quiet waterfront view", "a peareful countryside lane", "a cozy lakeside cabin", "a bright tropical garden", "a peaceful fishing spot", *a scentc ley view", *a 1 lively park bench", "a quist country lame", "a rustic vinoyard", *a tropical outdoor pool", "a peaceful city park", "a lively cutdoor music venue", "a calm lakeside dock", *a vibrant vaterfrent café", "a quiet gardan path", "a sunny riversida park", "a scenic mountain road", "a charming forest cabin", "a tranquil rural vista", " lively outdooz fair", "a calm and quiet pord", "a bustling pedestrian atreet", *a serene lakeside trail", "a Lively street corner", *a bright and sunny laws", t ruatic country garden", " quaint village street", "a peaceful nature reserve", "a vibrant open-sir market", *s scenic rural soad", "s quiet forest retreat", "s sunny city rooftop", "a cozy open-air restaurant", "a secluded beach cove", "s lively herbor viex", " scenic lakeside trail", t charming countryside path", "a lively town aquare", *a aunny country road", *a tranquil riverside path", "a busy urbar park", "a rustic hillaide cabin", "s scenie beach hoardwalk", *a quiet rural farm", "a pescoful coastal village", "a lively urban park", "a tranquil mountain valley", "a vibrant strect fair*, "a charming cceanfront pathi, "a quiet street corner", lush tropical garden", "a scenic hilltop vlew", "a quiet lakeside retrest", "a busy shopping district", "a calm and quiet garden", "a Lively mountain town aquare", peacetul coastal blutr", vibrant outdoor market aquare", "a quiet nature trail", "a scentc mountain cabin", "a sunny desert trail", "a peaceful urban garden", "a vibrans outdoor community center", "a caim lakeshore viex", "a tranquil city park", " quie= riverside retreat", " bustling urban plaza", "a sereze oceenfrent view", "a quiet hillsop vista", "a lively carnival parade", "a vibrent beach fastival", ta peaceful orcharé", "a sunny green park", "a charming beach house", "a scenis ocean drive", "a peaceful rural countzyside", "a vibrant plaza scene", "a lively downtown street", "a quiet city park heneh*' " colorful street festival", "a tranquil nature apot", "a sunny village aquare", *a buatling beachside promenade", "a rustic watezfront cabin", "a auay shopping mail entrance", "a charming lakeaide promenade", t scenic eliffaide", t quiet street park*, t colortul beach scene", "a Lively beach party", "a quiet garden cate", "a calm sandy shore*, t vibrant rooftop garden", "a serere lakeside dock*, * peacetul open rield" "a quiet scenic treil", t lively street performes", t rustic fores: retreat", "a scenic
denae evergreen foreat",
27

---

city skyline view", "a peaceful retreat", *a lively gathering", "a busy seaside boardwalk", "s scenic countryside village"]
C.2.3 VLM Prompt
A photo with
spacious living
A photo with scene of a cozy lounge.
A photo with scene of a snug sitting area.
C.3 Experimental Setting
This benchmark involves an object factor (text condition) and scene factor (query image). For the object factor, directly compute the similarity between the CLIP representations of the text condition and candidate images. For the scene factor, we first ask the LLM for the common scenes. Leveraging these scene texts the text basis, Eq. (4) in the main paper. Then, we calculate the similarity S2 between the conditional representations of the query image and the candidate images. Finally, select the candidate with the maximum combined similarity value S = S1 + a * S2, where the weighting parameter o is set to 10.
obtain all images' conditional representations by
C.4 Baseline
CLIP imgg baseline employs the CLIP image encoder generate embeddings for both query and gallery images, subsequently retrieving the most similar gallery image to the query. CLIP text adopts cross-modal approach, where the textual condition is encoded by the CLIP text encoder while gallery images are processed through the image encoder, enabling retrieval based on text-image alignment. CLIP imagettxx computes the average of query image embeddings and condition text embeddings, which is then used for retrieval from the gallery space.
C.5 Performance
Table / demonstrates that CRL performs robustly across a wide range of descriptive text quantities, with performance degradation observed only when the number of texts is insufficient.
Table 7: Customized similarity retrieval performance under different numbers of texts.
Focus
Change
Fext-num
Mean
R@1
R@2
R@3
R@D
R@2
R@3
10
17.7
29.1
38.0
18.7
29.9
38,5
28.6
20
18.8
32.0
40.4
20.4
32.5
41.8
31.0
50
18.4
31.7
20.1
33.2
42.2
30.9
100
18.5
31.0
41.6
20.7
34.1
43.4
31.5
200
20.1
33.0
21.4
35.1
43.8
32.4
300
19.2
32.6
21.3
34.5
32.0
400
19.4
31.9
20.8
34.1
31.6
500
19.0
31.7
41.0
20.4
33.7
43.3
31.5
D Customized Fashion Retrieval
For the task of customized fashion retrieval, we take the criterion of "Texture" as an example. We provide the prompts, responses and ground truth labels.
D.1 Dataset Description
a large-scale clothing dataset that provides four benchmarks, each tailored to specific task. Following previous work [39], we the category and attribute prediction split the
DeepFashion 36
28

---

Please generate common expressicns to describe the texture, many as possible, formatted as: ["...", "...", "..."]. Ensure all items are unique and written in single line, without any nested lists or benchmark. We provide summary of the criteria for this dataset in Table. 8 listing some examples for each criterion. As shown in Fig. [0] given the criterion "style". this benchmark requires retrieving the candidate that shares the same value ("Mickey") as the query image.
Query Image
Criterion
Candidate Images
Style
Figure 10: An example of the customized fashion retrieval task.
Table 8: A summary of the criteria of the customized fashion retrieval benchmark.
Criterion
Value
Count
Texture
abstract, animal, baroque, camo, circle,
Fabric
acid, bead, canvas, cloud, feather,
218
Shape
a-line, boxy, cropped, flared, shirt,
180
Part
bell, bow, buttoned, cowl, flat, ...
Style
baseball, bike, eagle, fox, mickey...
230
D.2
Prompt and Response
D.2.1
LLM Prompt
additional formatting. You may describe the same texture in different ways, such as "baroque", "ornate", or "iuxurious". Only generate the list, ane do not include any additional information.
D.2.2 LLM Response
("smooth", "rough", "silky", "velvety", "gzainy", "bumpy", "Euzzy", "soft", "coerse", "texturea", "polished", "matte", "glosay", "shiny", "satin-lika", "woven", 'embe!lished",
"ambroidored", "fluffy", "sheer", "lacsy", "ribbad", "pleated", "crinkled", "frosted", "distrassed", "weathered", "sparkling", "glittery", "glimering"' "luxurious", "elegant", "ornate", "ornamental", "baroque", "rustic", "vintage", "gritty", "delicaze", "intricate",
tomooth-totthe-touue teetturedto-pprrteet.ot "matte-tinish", "hign-gloss", "tusay-finia"",
"heavy-duty", "saft-touch", "pebbled", "canvas-1ike", "embroldered", "fringed", "knitted", "grocheted", "tight-knit", "lcose-knit", "structured", *fluid", "eloud-like", "s'iox" "furzy", *cozy", "saug", "plush", "velvety-smocth", "sandpaper-like", "suede", "nubuck", "grippy", "twilled", *crinkled", "slubbed", "grainy texture", "soft soft-ggip", "ecuffed", "westhered leether", "textured leathes", "crinkly", "pleated finish", "wazerproof", "thick-threadod", "goassmer", "cranclucene", "soven-texture", "froyad", "sightly-woven", "icose-woven', "threadbors", "matted", "dense-weave", "open-weave", "honeycomb", "cut-out", "quilted", "pleated-taxtura", "smocthrieather" "grain-lesther", "burnished")
D.2.3 Ground Truth Label
('abstractt, fabstract chevron', 'abatract chevron. print', 'abstract diamond', rabstract floral,, 'abstract florsl print', 'abstract geo', rabat geo prins', fabatract pataley', abbttract pattera', 'abstract print', 'abstract printed', fabstrect stripe', 'animal',
29

---

anniml print', 'bandana', 'bandens print', 'baroque', 'baroque print', 'bird', 'bird print', "ooaaniaa', "botenical print', 'boxy striped', "breton', 'breton scripe', 'brushstroke', brushstroke print', "butterfly', (butterfly print', 'camo', 'camourlage', 'checked', checkered', 'cheetah', 'chevron', 'chevron print', 'chiffon floral', 'airala', 'alashiat', classic striped', 'colorblock', 'colarblocked', 'erachet floral', 'daisy', 'dalay print', diamond', 'diamond priat', 'ditsy', 'ditsy floral', 'ditay flozal peint', 'dot', 'dots', "ootedd 'embroidered floral', 'flozal', 'flozal fluttes', 'floral paisley", 'floral paltera', flozal print', 'floral Lextuzed', "floral "Fooral-embzoide embzoide:ed', 'flower', 'Ioil', "Tolk', 'folk print', "gea", 'geo pattern', 'geo print', 'geo stripe', 'giraffe', 'giraffa print', 'graphic', "ridd,, 'grid print', 'heart', 'heart print', 'heathered stripe', 'houndstocth', "ikat', 'ikat print', "kaleidoscope', 'kaleidosccpe print', 'knit stripe', 'knit striped', 'leaf', 'leaf printt, "leave', 'leopard', " 'leppard print", '1inen', "I 'Hinen-bland', 'mandala', 'mandala print', 'marble', "mixed', 'mixed print', 'mixed stripe', 'mosaic', 'moscia print', 'milti-stripe', 'nautical', 'nautical stripe', 'nautical stripad', 'ombre', 'ornate', 'ornate paialay', 'ornate print', "paint', 'paint eplatter', 'painted', 'paisley', 'paleley print', 'pain", 'palm print', palm eprings', 'paln Lzee', 'pattera', "paltezned', 'pinstripe', 'pinstriped", 'polka dot', pon', 'print', 'print shirt', 'print woven', 'printed', 'ribbed stripe', 'ringer', rugby stripe', 'rugby striped', 'sophisticated', 'southwestem'' 'southwesterr-innpiiee soothwesstrrrpptteeeedd 'scuthwessen-ppinntt 'speckled', 'splatter', 'spetzed', 'stripe', "striped', "stripes', 'structured', 'tenal', 'tribal', sribal-inspired' 'txo-tone', 'vars! ty-striped', 'watercolor', 'zig', 12igzag")
print', 'ma-led', 'mar'e ed stripa', 'medallion', 'medal" fon print',
D.2.4 VLM Prompt
fashion with a texture of smooth.
A fashion with texture of rough.
fashion with texture of silky.
D.3 Experimental Setting
Following previous works, we exploit the triplet ranking loss to train this MLP and the backbone by 100k triplets, which are derived from the training split of the DeepFashion dataset. The training process consists of two stages. In the first stage, we only train the MLP and freeze the CLIP model for 1000 epochs, with an initial learning rate of le-4. In the second stage, we freeze the MLP and slightly fine-tune the CLIP model for 100 epochs, with a smaller initial learning rate of 1e-6. The optimizer, the decaying rate, the decaying step size and the triplet margin are Adam, 0.9, 3 and 0.3, respectively.
D.4 Performance
As illustrated in Table CRL exhibits consistent f'ashion retrieval performance across varying numbers of LLM -egeeeerrdteed descriptive texts, except in cases where the number of texts is too small.
Table 9: Customized fashion retrieval performance under different numbers of texts.
Text-num
Texture
Fabric
Shape
Part
Style
Mean
10
15.80
8.15
14.40
6.49
4.80
9.93
20
16.21
8.93
15.18
7.07
5.06
10.52
50
16.64
9.02
16.48
7.10
5.66
10.98
100
17.01
9.24
16.58
7.55
6.17
11.30
200
17.14
9.25
17.20
7.42
6.05
11.40
300
17.28
9.42
17.32
7.64
6.10
11.54
400
17.05
9.38
17.06
7.58
6.07
11.42
500
16.68
9.40
17.14
7.67
6.13
11.40
30

---

E
Ablation Studies
E.1
LLM
As for the selection of LLMs, we use the same prompt to query four mainstream LLMs: GPT-4o,
Deepscek-v3, Gemini 2.5, and Claude 4. As can be seen in Table our method does not particularly rely on any specific LLM.
Table 10: Customized classification performance under different LLMs.
Task
GPT-4o I
Deepseek-v3
34
Gemini 2.5 g
Claude 4 A
Std
Clustering
44.96 + 0.52
43.40 + 0,50
43.80 + 0.55
43.75 + 0.58
0.59 0.23
Few-shot Learning
53.34 # 01.4
52.94 # 0.40
52.93 + 0.41
53.43 + 0,45
E.2
Temperature
As for the LLM temperature t, we set é to 0, 0.5, 1.5 to obtain the text basis, respectively. The temperature ranges from 0 to 2, with higher values introducing more variability and randomness in the LLM's output. When the temperature approaches 2, the generated content becomes almost entirely random, so we did not include this setting in our experiments. The experimental results in Table 1I validate the robustness of our method the temperature parameter.
Table Il: Customized classification performance under different temperatures.
Task
(=0
100.
1=1.5
Std
Clustering
43.33 0.73
43.74 + 0.66
44.96 0.52
43.27
0.39
0.68
Few-shot Learning
53.07 # 0.49
53.27 # 0.48
53.34 + 0.44
52.50 + 0.41
0.33
E.3 LLM Prompt
As for the LLM prompt, we require include the [criterion]. We devise below 5 different templates:
I) Generate common expressions to describe the [criterion].
2) List a wide variety of typical phrases used to characterize the [criterion].
3) Enumerate familiar terms or expressions people often use when referring to the [criterion].
4) Identify and list expressions frequently used to convey the concept of the [eriterion].
5) How do people usually talk about the [criterion]?
One can observe from Table 12 that different LLM prompts can yield close performance improve- ments, indicating that our method is robust against the LLM prompt.
Table 12: Customized classification performance under different LLM prompts.
Task
Prompt
Prompt
Prompt 3
Prompt 4
Prompt 5
Std
Clustering
44.96 0.52
42.45 + 0.44
42.87 0.31
44.75 0.65
42.60 0.55
1.10
Few-shot Learning
53.34 0.44
52.95 0.39
53.42 0.37
52.82 a 0.48
52.40 0.44
0.37
E.4 VLM Prompt
As for the VLM prompt, require it to contain the [criterion] and the generated [text] by the LLM. We also devise below different templates:
1) objects with the [criterion] of [text]
2) photo with the [criterion] of [text]
31

---

3) itap with the [criterion] of [text]
4) art with the [criterion] of [text]
5) a cartoon with the [criterion] of [text]
As suggested in the Table 13 our method remains stable across different VLM prompts.
Table 13: Customized classification performance under different VLM prompts.
Task
Prompt
Prompt
Prompt 3
Prompt 4
Prompt 5
Std
Clustering
44.96 = 0.52
43.72 = 0.44
44.78 = 0.46
43.07 # 0.53 53.48
42.33 0.49
1.00
Few-shot Learning
53.34 0.44
52.28 0.38
$2.56 0.42
$2.58 0.23
0.47
32


Reviews:
[{"review_id": "eDt1OsgqTR", "reviewer": "Reviewer_iGLB", "summary": "The paper proposes conditional representation learning, a two stage framework for tailoring visual embeddings to a user-specified criterion. In the first step, the authors propose using an LLM to generate texts corresponding to the criteria which is then used as a basis for projecting the VLM embeddings into the same space. The primary motivation is semantic alignment of the embeddings to a task specific domain without fine-tuning. Experimental results are provided on multiple criterion tasks setup on four datasets - Clevr4-10k and Cards for image tasks, and GeneCIS and DeepFashion for retrieval tasks.", "questions": "Please refer to the weaknesses section above.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths \n\n1. The key idea proposed by the authors is fairly simple, and the methodology seems straightforward and interpretable. \n\n\n2. The paper is generally well written and easy to follow. \n\n\n3. Since the method involves inference on LLMs followed by simple projection operations, it seems to be efficient compute wise. \n\n\n## Weaknesses \n\n1. There is a large body of work in the domain of semantic alignment in projection spaces, and even domain adaptation/alignment which is missing from either from the related work section, or in use as baselines or even both. TextSpan [1], ICITC [2] are mentioned in the related work, however other relevant works like MMRL [3], SwapPrompt [4], and more are not mentioned. All of them are missing from the use as baselines altogether, hence there is not enough justification of placing their work in the context of related work. In fact, the methodology of TextSpan and ICITC is almost identical to the one proposed by the authors, however there is no discussion comparing and contrasting their work. \n\n\n2. While the results in experimental section demonstrate strong gains over the unaligned and non-finetuned baselines, the experimental setup is not very convincing. The datasets are on the smaller scale size for both vision and retrieval domains as compared to the real world large scale task datasets. All of the main results in tables 1, 2, 3 and 4 are also missing confidence intervals for statistical significance. \n\n3. Details about the selection of LLM and decision choices about prompts etc are missing. The main paper does not even mention what LLM is used for the generation task, what is the rationale behind the selection, the dependence on GPT-4 and impact of using other LLMs etc. There are also no ablation studies in the paper (or the appendix) to give more intuition into the method and it’s robustness to factors like the prompt, temperature, redundancy in generation and so forth. \n\n\n4. The advantage that general purpose embeddings offer is the usability across tasks, and hence a comparison with the base models is not a well-formed setup. Plus the use of a naive projection operator can cause issues in correlation across basis vectors, and also cause some degradation in performance. Some of the cases where the performance is degraded, e.g. in Table 1 for clevr4-10k in the “count” task, the failures are not analysed. \n\n\n\n[1] INTERPRETING CLIP’S IMAGE REPRESENTATION VIA TEXT-BASED DECOMPOSITION, Gandelsman et. al, ICLR ‘24\n\n[2] IMAGE CLUSTERING CONDITIONED ON TEXT CRITERIA, Kwon et. al, ICLR ‘24\n\n[3] MMRL: Multi-Modal Representation Learning for Vision-Language Models, Guo et. al, CVPR ‘25\n\n[4] SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models, Ma et. al, NeurIPS ‘23", "was_revised": true, "final_justification": "The authors address some of my experimental concerns by providing additional ablations, confidence intervals for results, etc, which partly addresses my original concerns and hence I raised my score. However I still have concerns about the task setup, and missing comparisons with some baseline methods that the authors did not provide during the rebuttal phase, with the primary comparisons being with the base models. Hence, I will do not believe that the submission is strong and convincing enough for a clear accept decision.", "rebuttal": "We sincerely thank you for the time and effort spent reviewing our paper. Nevertheless, we respectfully believe that some of the concerns raised may be due to misunderstandings, and we would like to clarify these points in detail below.\n\n**Weakness 1.1: Related work**\n\nFirst of all, we must point out that the core of our work, CRL (conditional representation learning), aims to obtain conditional representations tailored to the user-specified criterion without any supervision signal. \n\nAccording to this, we choose \"representation learning\" and \"conditional similarity\" as our related work. The former refers to traditional methods that aim to learn general feature representations, while the latter focuses on newly emerging areas that learn conditional similarity conditioned on specific criteria. \n\nThe relevant works you mentioned, MMRL (fine-tuning the model structure for image classification) and SwapPrompt (adapting the VLM prompt for image classification), are two works that aim to learn representations under the general criterion (shape or category).  We'll add them to the \"representation learning\" section in related works. \n\n**Weakness 1.2: Methodology novelty**\n\nWe disagree with the comment \"the methodology of TextSpan and IC|TC is almost identical to the one proposed by the authors\". Below is an intuitive comparison among these three works:\n\n1\\) Different tasks:  CRL aims to obtain conditional representations under specified criteria for multiple tasks. TextSpan seeks to analyze the role of each head in the CLIP model. IC|TC is designed for the clustering task.\n\n2\\) Different techniques: CRL utilizes the descriptive texts as the semantic basis, and then projects images into this conditional feature space to obtain conditional representations. TextSpan iteratively selects a set of texts that maximizes the variance of the head's output features, using these texts to interpret the corresponding head. IC|TC queries the LLM to get the cluster affiliation with three steps.\n\nIt's clear to see that CRL is substantially different from TextSpan and IC|TC.\n\n**Weakness 2.1:  Dataset scale**\n\nTo the best of our knowledge, previous multi-criteria works typically focused on a single downstream task. So in this paper, we integrate existing multi-criteria datasets from multiple domains.  We acknowledge that, as an emerging field, this area currently lacks abundant high-quality datasets. However, we would like to point out that not all dataset scales are small, as the DeepFashion dataset contains over 270k images (in the 4.2.2 section).\n\nWe hope our work can inspire broader interest in this promising research direction. Moving forward, we also intend to develop larger-scale datasets to support continued progress.\n\n**Weakness 2.2: Missing confidence intervals**\n\nWe present the confidence intervals (at the confidence level of 0.95) for both the customized clustering and linear probe tasks across 20 trials. The experiments in the rebuttal are all conducted under the same setting.\n\n|     Task     |     CLIP     |    CLIP + CRL(Ours)    |\n| :----------: | :----------: | :--------------------: |\n|  Clustering  | 21.73 ± 0.28 | 44.96 ± 0.52 (↑23.23%) |\n| Linear Probe | 43.93 ± 0.43 | 53.34 ± 0.44 (↑9.42%)  |\n\nThe table below reveals that the random seed has minimal impact on the performance of our method.\n\nWe'll complement tables 1, 2, 3, and 4 in the main text for the next version.\n\n**Weakness 3.1: LLM ablation**\n\nThanks for the suggestion, and we provide thorough ablation studies below. \n\nAs for the selection of LLMs, we would like to clarify that our method does not particularly rely on any specific LLM. To be specific, we use the same prompt to query four mainstream LLMs, GPT-4o, Deepseek-v3, Gemini 2.5, and Claude 4.\n\n|     Task     |    GPT-4o    | Deepseek-v3  |  Gemini 2.5  |   Claude 4   | Std  |\n| :----------: | :----------: | :----------: | :----------: | :----------: | :--: |\n|  Clustering  | 44.96 ± 0.52 | 43.40 ± 0.50 | 43.80 ± 0.55 | 43.75 ± 0.58 | 0.59 |\n| Linear Probe | 53.34 ± 0.44 | 52.94 ± 0.40 | 52.93 ± 0.41 | 53.43 ± 0.45 | 0.23 |\n\nThe result demonstrates that all four LLMs achieve comparable and consistent performance.\n\n**Weakness 3.2: LLM prompt ablation**\n\nAs for the LLM prompt, we require it to include the [criterion]. We devise below 5 different templates:\n\n1\\) Generate common expressions to describe the [criterion].\n\n2\\) List a wide variety of typical phrases used to characterize the [criterion].\n\n3\\) Enumerate familiar terms or expressions people often use when referring to the [criterion].\n\n4\\) Identify and list expressions frequently used to convey the concept of the [criterion].\n\n5\\) How do people usually talk about the [criterion]?\n\n|     Task     |   Prompt 1   |   Prompt 2   |   Prompt 3   |   Prompt 4   |   Prompt 5   | Std  |\n| :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :--: |\n|  Clustering  | 44.96 ± 0.52 | 42.45 ± 0.44 | 42.87 ± 0.31 | 44.75 ± 0.65 | 42.60 ± 0.55 | 1.10 |\n| Linear Probe | 53.34 ± 0.44 | 52.95 ± 0.39 | 53.42 ± 0.37 | 52.82 ± 0.48 | 52.40 ± 0.44 | 0.37 |\n\nThe table above shows that different LLM prompts can yield close performance improvement, indicating that our method is robust against the LLM prompt.\n\n**Weakness 3.3: VLM prompt ablation**\n\nAs for the VLM prompt, we require it to contain the [criterion] and the generated [text] by the LLM. We also devise below 5 different templates:\n\n1\\) objects with the [criterion] of [text]\n\n2\\) a photo with the [criterion] of [text]\n\n3\\) itap with the [criterion] of [text]\n\n4\\) art with the [criterion] of [text]\n\n5\\) a cartoon with the [criterion] of [text]\n\n|     Task     |   Prompt 1   |   Prompt 2   |   Prompt 3   |   Prompt 4   |   Prompt 5   | Std  |\n| :----------: | :----------: | :----------: | :----------: | :----------: | :----------: | :--: |\n|  Clustering  | 44.96 ± 0.52 | 43.72 ± 0.44 | 44.78 ± 0.46 | 43.07 ± 0.53 | 42.33 ± 0.49 | 1.00 |\n| Linear Probe | 53.34 ± 0.44 | 52.28 ± 0.38 | 52.56 ± 0.42 | 53.48 ± 0.38 | 52.58 ± 0.23 | 0.47 |\n\nAs suggested in the above table, our method remains stable across different VLM prompts.\n\n**Weakness 3.4: Temperature ablation**\n\nAs for the LLM temperature t, we set t to 0, 0.5, 1 and 1.5 to obtain the text basis, respectively. The temperature ranges from 0 to 2, with higher values introducing more variability and randomness in the LLM's output (When the temperature approaches 2, the generated content becomes almost entirely random, so we did not include this setting in our experiments).\n\n|     Task     |     t=0      |    t=0.5     |     t=1      |    t=1.5     | Std  |\n| :----------: | :----------: | :----------: | :----------: | :----------: | :--: |\n|  Clustering  | 43.33 ± 0.73 | 43.74 ± 0.66 | 44.96 ± 0.52 | 43.27 ± 0.39 | 0.68 |\n| Linear Probe | 53.07 ± 0.49 | 53.27 ± 0.48 | 53.34 ± 0.44 | 52.50 ± 0.41 | 0.33 |\n\nThe experimental results also validate the robustness of our method to the temperature parameter.\n\n**Weakness 3.5: Redundancy ablation**\n\nAs for the redundancy, we analyze the number of generated words in Figure 6 in the main text. The result shows that our method maintains stable performance, indicating robustness to redundancy.\n\n**Weakness 4.1:  Baseline**\n\nIt appears that the scope of our baseline comparisons may have been overlooked. Our proposed method is built upon the representations of base models. Consequently, it is intuitive to observe the effect of representation transformation when compared directly to these representations. \n\nHowever, we don't limit our comparisons to the base models. In fact, except for the linear probe task (which is a direct evaluation of the transformation), the other three tasks are also compared against methods specifically designed for those tasks. \n\nTo be specific, for the clustering task (Section 4.1.2), the baselines include three specifically designed methods: CC, SCAN, and Multi-Map. For the customized similarity retrieval task (Section 4.2.1), the baselines include five representative methods: Pic2Word, SEARLE, LinCIR, CIG, and Combiner. For the fashion retrieval task (Section 4.2.2), we compare against five competitive baselines: Triplet, CSN, ASEN, ASEN++, and RPF.\n\n**Weakness 4.2 Performance**\n\nWe think this comment may be overly strict. In fact, our method, CRL,  shows significant improvements over both the base models and existing baselines across all four tasks. Specifically, for the linear probe task, CRL obtains a performance boost of the original CLIP representation of over 21% relative. For the clustering task, CRL obtains a performance boost of the SOTA method (Multi-Map) of over 53%. For the customized similarity retrieval task, CRL achieves a gain of 13% compared to the SOTA method (Combiner). For the fashion retrieval task, CRL surpasses the best competitive method (RPF) by 10%.In other words, the cases of performance degradation are isolated exceptions rather than indicative of overall trends. Moreover, we would like to clarify that for the “count” criterion on the clevr4-10k dataset in Table 1, there is no actual performance degradation.\n\n\n\nWe will add the above results and discussions in the next version. \n\nFinally, we sincerely hope that our response will be carefully considered and lead to a more favorable assessment. Thank you for the time spent reviewing our paper again."}, {"review_id": "ca786RA5Ty", "reviewer": "Reviewer_xrw3", "summary": "This manuscript tackles to learn multiple representation that can capture various semantics on given conditions. For a given criterion like color, it generate descriptive texts semantically related to criterion by using LLM. Then it encodes generated texts and images through VLM like CLIP to generate conditional representation by projecting image representation into representation space of generated texts. Experiments on Clevr4-10k and Card benchmarks demonstrate the effectiveness of the proposed method.", "questions": "See the above Strengths And Weaknesses section.", "limitations": "No. I can not find explicit limitations and social impact sections in this manuscript. Although the authors mentioned that they discuss the limitations in the Experiment section, I can not find them in their section 4.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Learning conditional representation is interesting and important.\n- The proposed method is quite simple and technical novelty is limited. The equation 4 is equivalent conventional logits for zero-shot classification of CLIP. The proposed method uses LLM to class categories and produces zero-shot logits on given classes, and then employ the logits as the conditional representation. Empirical results in Table 1 also show that the original CLIP often outperforms the proposed method CLIP + CRL on various metrics like Shape, Number, and Suits.\n- This manuscript only performs the evaluation on a few benchmarks. As I think conditional representation is closely related to VQA problem, I suggest to include evaluations on common vqa benchmarks like COCO, VQAv2, and so on.\n- Only CLIP based baselines are compared to this work. For example, BLIP2 Qformer [1] could also produce image representation conditioned on given texts. More comparisons and discussions to explain how can the proposed simple method work better than the existing vision-language approaches.\n\n[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,", "was_revised": true, "final_justification": "Some of my concerns regarding additional baselines, such as BLIP2, have been addressed. However, as reported in Table 2 of the manuscript, the proposed method exhibits substantial performance gaps in certain tasks, such as the color task. This raises questions about whether the method can consistently improve performance across all tasks.\n\nWith regard to technical novelty, I am not convinced by the provided rebuttal. The proposed approach seems largely similar to the original CLIP method. This is particularly evident in the previously mentioned observation that the original CLIP often outperforms the proposed + CRL method on several metrics, including Shape, Number, and Suits. So, the key contribution that generating predefined classes for zero-shot classification using an LLM (instead of humans) sounds incremental to me.\n\nAdditionally, this manuscript does not provide new benchmark results like VQA yet.\n\nDuring the discussion period, the authors addressed the remaining concerns further. I raised my score to 4, but it is still close to borderline.", "rebuttal": "We appreciate your careful comments. However, we believe that the judgment may overlook key aspects of our contribution, which we would like to clarify below.\n\n**Weakness 1.1: Technical novelty**\n\nThe concept of conditional representation learning we propose is of high potential and research value. Existing representation learning works primarily focus on learning a representation under a universal criterion (\"shape\" or \"category\"). However, this may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics.  Moreover, in the fashion retrieval scenario (Section 4.2.2), users often search for items based on different criteria, such as color, material, or occasion. A user might look for \"red dresses\" today, \"formal outfits\" tomorrow, and \"lace tops\" the next day. While the input image remains the same, the relevant features for retrieval vary depending on the user's intent. Conditional representation learning enables the model to generate image representations that are tailored to the given criterion conveniently and quickly, improving retrieval efficiency and user satisfaction.\n\nWe propose conditional representation learning and offer a computationally efficient and highly generalizable solution. This is exactly where our novelty lies.\n\nBesides, our proposed method is a simple, plug-and-play approach. Nevertheless, we believe this is not a weakness but rather a strength. Furthermore, we provide a basis transformation perspective to justify the technical soundness of our method.\n\n**Weakness 1.2: Comparison to zero-shot classification**\n\nWhile equation 4 is similar in computational form to zero-shot classification, our method is fundamentally different in the following two aspects:\n\n1\\) Different tasks: Zero-shot classification directly outputs a predicted class label, solving the classification task. By comparison, our method serves as a plug-and-play module for the conditional representation learning task.\n\n2\\) Different priors: Zero-shot classification relies on the availability of class names as prior knowledge, while our method does not require them. Obtaining such priors is impractical in domains featuring a large number of classes. For example, in the fashion retrieval task(Section 4.2.2), the DeepFashion dataset we adopt consists of 1000 fine-grained classes. What's more, in real-world applications, it's even harder to make class names as priors since they are dynamic.\n\n**Weakness 1.3: Performance**\n\nWe respectfully note that the performance degradation does not occur as \"often\" as the comment suggested. \n\nIn fact, across all four tasks, our method consistently outperforms the base model and existing baselines. Specifically, for the linear probe task (Table 1), CRL obtains a performance boost of the original CLIP representation of over 21% relative. For the clustering task (Table 2), CRL obtains a performance boost of the SOTA method (Multi-Map) of over 53%. For the customized similarity retrieval task (Table 3), CRL achieves a gain of 13% compared to the SOTA method (Combiner). For the fashion retrieval task (Table 4), CRL surpasses the best competitive method (RPF) by 10%. The few cases of degradation mentioned in the comment are isolated exceptions rather than the norm.\n\n**Weakness 2: Benchmark**\n\nOur work falls under the scope of representation learning. Therefore, we selected downstream tasks that are closely tied to this area, specifically, classification and retrieval. We included four such sub-tasks, which we believe are sufficient to validate the effectiveness of our approach.\n\nNevertheless, we truly value the suggestion of applying our method to the VQA problem. Since we are not familiar with this field and previous representation learning works seldom select the VQA problem as the downstream task, we wonder how to apply our method to it.\n\nBy the way, we believe this is feasible, as the dataset used in the customized similarity retrieval task (Section 4.2.1) is derived from the COCO dataset. We'll leave this as a promising direction for future exploration.\n\n**Weakness 3: Baseline**\n\nThanks for the insightful suggestion. Before the rebuttal, we indeed only considered applying our method based on the CLIP-like architecture. Following your suggestion, we apply our method to BLIP2 and find that it even outperforms CLIP, as the following table shows. This further demonstrates the generalizability of our work, and we will continue exploring broader applicability in future research.\n\n|     Task     |     CLIP     |       CLIP + CRL       |   BLIP2    |     BLIP2 + CRL      |\n| :----------: | :----------: | :--------------------: | :--------: | :------------------: |\n|  Clustering  | 21.73 ± 0.28 | 44.96 ± 0.52 (↑23.23%) | 26.07±0.79 | 51.64±0.64 (↑25.57%) |\n| Linear Probe | 43.93 ± 0.43 | 53.34 ± 0.44 (↑9.42%)  | 49.78±0.61 | 59.58±0.35 (↑9.80%)  |\n\nMoreover, except for the linear probe task, the other three tasks are also compared against methods specifically designed for those tasks. To be specific, for the clustering task (Section 4.1.2), the baselines include three specifically designed methods: CC, SCAN, and Multi-Map. For the customized similarity retrieval task (Section 4.2.1), the baselines include five representative methods: Pic2Word, SEARLE, LinCIR, CIG, and Combiner. For the fashion retrieval task (Section 4.2.2), we compare against five competitive baselines: Triplet, CSN, ASEN, ASEN++, and RPF.\n\n**Weakness 4: Discussion on limitations**\n\nDespite CRL's generalizability across different criteria, it may not outperform clustering methods like CC and SCAN under the universal criterion \"shape\". This is likely because these methods employ specially targeted designs for clustering under this criterion.\n\n\n\nWe will add the above results and discussions in the next version. \n\nFinally, we would like to thank you for your time and effort once again, and sincerely hope our response could address your concerns."}, {"review_id": "YKDcwZjazq", "reviewer": "Reviewer_YAxP", "summary": "The authors introduce conditional representation learning, which aims to learn a representation conditioned on user-specified criteria. They frame this as a transformation of basis, using LLM-generated descriptors generated from the user's text input passed through a VLM as the basis. They pass the query images through the VLM and project those representations onto the text vector basis, essentially obtaining a vector of similarities to each of the text descriptors, which they use as their initial representation. They train MLPs on top of these. They suggest these representations are more suited to the downstream tasks related to the user's query.  They evaluate on various 'customized' variants of tasks such as classification, clustering, and retrieval.", "questions": "- This approach applies to methods like CLIP which produce image embeddings and text embeddings separately, with fusion only at the final embedding similarity. How does this compare to methods that create image representations conditioned on text already? What are the pros and cons? \n- What differentiates the CRL task from previous related tasks?\n- Are there any existing tasks that could be framed in this setting for broader comparison?\n- With how many examples do the improvements level off?\n- How robust is the approach to variations in the user-specified input? What is that input for the evaluated tasks?\n- How does it compare to learned visual prompt approaches?", "limitations": "The authors say the limitations are in the experiments section, but I didn't catch it if so.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea is simple to implement.\n- It seems effective for the presented tasks.\n- It provides a nice way to condition image representations on the text for the task with models where these representations are made separately. \n\nWeaknesses:\n- One of the stated main contributions is the conditional representation learning task, but there is a substantial body of work on task-conditioned and goal-conditioned representations; this specific instantiation has new elements, but it seems overstated to me, and these other works are not really discussed. Taskonomy, Task2Vec, many works in robotics/RL, FiLM, come to mind.\n- It's unclear how realistic the self-made tasks are relative to established similar tasks.\n- Missing discussion of other work that uses LLMs to generate descriptors for further use, such as https://arxiv.org/abs/2209.03320 / https://arxiv.org/abs/2210.07183 / subsequent works building on these\n\n\nI could be convinced to increase my score with more focused positioning and a compelling reason this setting is worth unique exploration/differentiated from other previously established settings.", "was_revised": true, "final_justification": "The authors provide a convincing justification for why this task customization setting is of interest, and answer many other of the original remaining questions, so I increase my score to borderline accept. Given the issues of task realism -- none of the evaluations are established and still seem a bit artificial to me -- I'm still not inclined higher than borderline and could see it going either way.", "rebuttal": "We are grateful for your constructive and kind comments. Below, we provide the point-by-point response to your concerns.\n\n**Weakness 1 & Question 2: Our contribution**\n\nWe clarify that our proposed conditional representation learning is remarkably distinct from works on task-conditioned and goal-conditioned representations. In essence, our method operates at a more fundamental and general level compared to task- or goal-conditioned approaches..\n\nWe first make a summary of our work and task/goal-conditioned works as follows:\n\n1\\) Task-conditioned works aim to learns representations that reveal the underlying correlations among different tasks. For example, taskonomy computes the optimal transfer learning paths among tasks (point matching, reshading, etc.) to minimize the amount of required annotation.\n\n2\\) Goal-conditioned works aim to learns representations that meet the required outcomes or goal states. For example, Robotics/RL works typically involve using the goal state as a conditional input to learn policies to instruct the agent from its current state to the goal state.\n\n3 \\) Our work aims to learn representations that adapt to arbitrary user-specified criteria (\"color\", \"texture\", etc.), extending the conventional representation learning that focuses on a single dominant aspect (typically “category”). \n\nFrom the above comparison, we highlight the key difference:\n\nTask-conditioned works focus on generalization across tasks and are commonly used in supervised multi-task learning. Goal-conditioned works, on the other hand, emphasize generalization across goal supervision and are typically applied in reinforcement learning. In contrast, our work targets generalization across user-specified criteria, which is situated in the context of unsupervised representation learning, a more fundamental and upstream field. \n\nWe will clarify this positioning more clearly in the introduction section.\n\n**Weakness 2 & Question 3: Task realism (Framing existing tasks under this setting)**\n\nThe proposed concept of conditional representation learning holds significant promise and research merit. Existing representation learning works primarily focus on learning a representation under a universal criterion (\"shape\" or \"category\"). However, this may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics.  Moreover, in the fashion retrieval scenario, users often search for items based on different criteria, such as color, material, or occasion. A user might look for \"red dresses\" today, \"formal outfits\" tomorrow, and \"lace tops\" the next day. While the input image remains the same, the relevant features for retrieval vary depending on the user's intent. Conditional representation learning enables generating representations that are tailored to the given criterion conveniently and quickly, improving user satisfaction.\n\nWe propose conditional representation learning and offer an efficient and generalizable solution. We utilize it as a plug-and-play module to apply to four downstream tasks: linear probe  (Section 4.1.1), clustering (Section 4.1.2), customized similarity retrieval (Section 4.2.1), and fashion retrieval (Section 4.2.2). In fact, they are not self-made tasks. To be specific, these four tasks are derived from classification and retrieval, the most fundamental tasks to evaluate representation learning.\n\n**Weakness 3: Discussions on LLM-based descriptor generation works**\n\nThe first paper you mentioned (paper 1) belongs to prompt learning, and we discuss these works in Question 6.\n\nThe second paper (paper 2) in the comment is similar to LaBo and LM4CV in our Related Work section (line 109), belonging to interpretable classification. \n\nCompared to these two papers, CRL differs in the following two aspects:\n\n1\\) Generated content: They use LLM to generate content to describe each class name, implicitly related to universal \"category\" criterion. For example, paper 1 asks the LLM to generate \"two legs\", \"a small head\", etc., for the class \"hen\". In comparison, our work uses LLM to generate content to describe the criterion, which is a higher-level concept. The generated results can be understood as class names, which can be found in our appendix.\n\n2\\) Purpose: Paper 1 aims to improve classification accuracy and interpretability under the universal criterion. Paper 2 aims to learn the optimal prompt to improve zero-shot image classification accuracy. In contrast, our work focuses on obtaining conditional representations across different criteria.\n\n**Question 1: Comparison to text-conditioned representation methods**\n\nThe baselines (Combiner, LinCIR, etc.)  in Section 4.2.1 are exactly the methods you mentioned. These methods obtain image representations conditioned on text already, and then use the new representations to retrieve the most similar images from the gallery.\n\nCompared to these methods, CRL does not require task-specific design and can be used in a plug-and-play manner. Nevertheless, CRL needs to be constructed on models where image and text representations are made separately, thus it’s hard to apply CRL to backbones that are not inherently designed for this type of architecture.\n\nBy the way, we apply CRL to BLIP2 and achieve consistently ideal performance improvement like CLIP, which further demonstrates the generalizability of CRL.\n\n| Task |CLIP|CLIP+CRL|BLIP2|BLIP2+CRL|\n| :----------: | :--------: | :--------: | :--------: | :--------: |\n|  Clustering  | 21.73±0.28 | 44.96±0.52 | 26.07±0.79 | 51.64±0.64 |\n| Linear Probe | 43.93±0.43 | 53.34±0.44 | 49.78±0.61 | 59.58±0.35 |\n\n**Question 4: With how many examples do the improvements level off**\n\nFirstly, we would like to clarify that CRL itself is unsupervised and does not rely on additional labeled examples. When applying to downstream tasks, the training data we use is the same as the corresponding baselines. \n\nWe understand that you might be referring to the number of texts generated by the LLM. We have explored this in Section 4.3. As Figure 6 indicates, when the number of generated texts increases to around 100, the performance improvements level off across all four tasks.\n\n**Question 5.1: Input for the evaluated tasks**\n\nThe input for the evaluated tasks includes the criterion and the LLM/VLM prompt. \n\n**Question 5.2: Method Robustness**\n\nWe now present an analysis of CRL's robustness for these factors.\n\n**Robustness against different LLMs**\n\nAs for the selection of LLMs, we would like to clarify that our method does not particularly rely on any specific LLM. To be specific, we use the same prompt to query four mainstream LLMs, GPT-4o, Deepseek-v3, Gemini 2.5, and Claude 4. \n\n|     Task     |   GPT-4o   | Deepseek-v3 |  Gemini 2.5  |   Claude 4   | Std  |\n| :----------: | :--------: | :---------: | :----------: | :----------: | :--: |\n|  Clustering  | 44.96±0.52 | 43.40±0.50  | 43.80 ± 0.55 | 43.75 ± 0.58 | 0.59 |\n| Linear Probe | 53.34±0.44 | 52.94±0.40  | 52.93 ± 0.41 | 53.43 ± 0.45 | 0.23 |\n\nThe result demonstrates that CRL remains stable when adopting different LLMs to generate the texts.\n\n**Robustness against different LLM prompts**\n\nAs for the LLM prompt, we devise 5 different templates:\n\n1\\) Generate common expressions to describe the [criterion].\n\n2\\) List a wide variety of typical phrases used to characterize the [criterion].\n\n3\\) Enumerate familiar terms or expressions people often use when referring to the [criterion].\n\n4\\) Identify and list expressions frequently used to convey the concept of the [criterion].\n\n5\\) How do people usually talk about the [criterion]?\n\n|Task|1|2|3|4|5|Std|\n| :----------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--: |\n|Clustering| 44.96±0.52 | 42.45±0.44 | 42.87±0.31 | 44.75±0.65 | 42.60±0.55 | 1.10 |\n|Linear Probe | 53.34±0.44 | 52.95±0.39 | 53.42±0.37 | 52.82±0.48 | 52.40±0.44 | 0.37 |\n\nThe table above suggests that CRL is robust against the LLM prompt.\n\n**Robustness against different VLM prompts**\n\nAs for the VLM prompt, we also devise 5 different templates:\n\n1\\) objects with the [criterion] of [text]\n\n2\\) a photo with the [criterion] of [text]\n\n3\\) itap with the [criterion] of [text]\n\n4\\) art with the [criterion] of [text]\n\n5\\) a cartoon with the [criterion] of [text]\n\n|Task|1|2|3|4|5|Std|\n| :----------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--: |\n|Clustering|44.96±0.52|43.72±0.44|44.78±0.46|43.07±0.53|42.33±0.49|1.00|\n|Linear Probe|53.34±0.44|52.28±0.38|52.56±0.42|53.48±0.38|52.58±0.23|0.47|\n\nAs suggested in the above table, CRL achieves stable performance across different VLM prompts.\n\n**Question 6: Comparison to visual prompt learning methods**\n\nWe summarize the differences between our work and prompt-learning methods as follows:\n\n1\\) Tasks: prompt learning methods aim to learn an optimal prompt to solve the classification task. By comparison, our method serves as a plug-and-play module for the conditional representation learning task.\n\n2\\) Priors: prompt learning methods rely on the prior of class name, while CRL does not require it. Obtaining such priors is impractical in domains featuring massive classes. For example, in the fashion retrieval task, the DeepFashion dataset consists of 1000 classes. In real-world applications, it's even harder to make class names as priors since they are dynamic.\n\n**Limitations Discussion**\n\nDespite CRL's generalizability across different criteria, it may not outperform clustering methods like CC and SCAN under the universal criterion \"shape\". This is likely because these methods employ specially targeted designs for clustering under this criterion.\n\nWe will add the above results and discussions in the next version. \n\nFinally, we deeply appreciate your time and constructive feedback, and we earnestly hope that our response meets your expectations and resolves the issues raised."}, {"review_id": "JRaeJE4Xim", "reviewer": "Reviewer_GS6R", "summary": "This work proposes a conditional representation learning method, which transforms the universal image embeddings into customized ones catering to the criteria of downstream tasks. Experiments on both supervised/unsupervised classification and retrieval demonstrate the effectiveness and superiority of the proposed method.", "questions": "I expect the authors to address my concerns in the weaknesses section. Besides, what are the differences between the task of \"Customized Similarity Retrieval\" in the paper and the task of \"compositional retrieval\"?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed concept of conditional representation learning is very interesting. As the authors pointed out, most existing representation learning methods learn universal embeddings, which could be suboptimal for customized tasks that prioritize non-dominant semantics. Instead of resorting to laborious fine-tuning, this work reveals that a simple but ingenious basis transformation operation could uncover semantics with respect to a specific criterion.\n2. The proposed method is intuitively and technically correct. Its simple but effective design makes it a plug-and-play module for various customized downstream tasks, as validated in the experiments.\n3. The paper is well written in general, making it easy to follow the key idea.\n\nWeaknesses\n1. Currently, the nouns used for constructing the textual basis are simply generated based on the given criteria. However, intuitively, the best set of textual basis could differ for different input images, even under the same criterion. I encourage the authors to take that into consideration and at least discuss such a potential limitation of this method.\n2. I would like to see the quality of the generated texts. Since the prompt is simply \"generate as many as possible\", would the generated words be replicated and redundant? How does that influence the final performance? For example, I expect experimental results on a de-duplicated text basis.\n3. For the current classification evaluation, it is more accurate to highlight \"few-shot learning\" instead of \"supervised linear probing\".", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of the problem this work aims to address, as well as our simple yet effective idea. We sincerely appreciate your positive feedback. Below are the point-by-point responses to your concerns mentioned in the weaknesses and questions sections. \n\n**Weakness 1: Best set of textual basis**\n\nWe appreciate the insightful comment. We will try to adopt a unified selection strategy in this work (e.g., variance, distance, PCA, maximal connected subgraph, etc.). We hold the same view that this is a valuable direction, and we will consider it in future explorations.\n\n**Weakness 2: Quality of the generated texts**\n\nOur method only requires that the generated texts roughly describe the given criterion, without the need for precise or exhaustive phrasing. \n\nTake the \"color\" criterion as an example, the generated texts are:\n\n[\"red\", \"crimson\", \"scarlet\", \"ruby\", \"cherry\", \"rose\", \"burgundy\", \"cardinal\", \"wine\", \"firetruck red\", \"ferrari red\", \"carmine\", \"sangria\", \"apple red\", \"poppy\", \"raspberry\",  \"red\", \"pink\", \"blush\", \"fuchsia\", \"magenta\", \"hot pink\", \"baby pink\",  \"pink\", \"coral\", \"peach\", \"apricot\", \"salmon\", \"light pink\", \"powder pink\", \"lavender\", \"violet\", \"indigo\", \"purple\", \"amethyst\", \"grape\", \"orchid\", \"eggplant\", \"plum\", \"lilac\", \"mauve\", \"periwinkle\", \"blue\", \"navy\", \"azure\", \"sky blue\", \"cobalt\", \"cerulean\", \"sapphire\", \"electric blue\", \"royal blue\", \"baby blue\", \"powder blue\", \"blue\", \"teal\", \"turquoise\", \"aqua\", \"seafoam\", \"green\", \"emerald\", \"forest green\", \"mint\", \"lime\", \"olive\", \"kelly green\", \"sage\", \"pea green\", \"jade\", \"chartreuse\", \"green\", \"yellow\", \"gold\", \"lemon\", \"butter\", \"canary\", \"mustard\", \"sunflower\", \"cream\", \"ivory\", \"beige\", \"tan\", \"coffee\", \"chocolate\", \"yellow\", \"brown\", \"copper\", \"rust\", \"mahogany\", \"walnut\", \"espresso\", \"gray\", \"slate\", \"charcoal\", \"pewter\", \"silver\", \"platinum\", \"black\", \"coal\", \"ebony\", \"jet black\", \"onyx\", \"snow\", \"ivory\", \"white\", \"pearl\", \"bone\", \"eggshell\", \"vanilla\"]\n\nSome generated texts are duplicated (\"red\", \"blue\", etc.), and we'll filter out these redundant texts. As the table below shows, this brings a slight improvement in clustering performance.\n\n| Metric | Original Texts | Texts without replication |\n| :----: | :------------: | :-----------------------: |\n|  NMI   |     88.18      |           88.67           |\n|  ACC   |     87.42      |           88.05           |\n|  ARI   |     81.27      |           82.30           |\n\nAs for the redundancy, we explore the number of generated words in Figure 6 in the main text. As depicted, our method achieves stable performance, demonstrating its robustness to the redundancy of the generated words. \n\n**Weakness 3: More appropriate title**\n\nThank you for pointing this out — we agree that framing the classification evaluation as “few-shot learning” is more precise than “supervised linear probing”. We will modify it in the next version.\n\n**Question 1: Comparison to compositional retrieval**\n\nThe customized similarity retrieval task is more closely associated with directed representation transformation, where the criterion is fixed.\n\nIn comparison, compositional retrieval does not involve such a targeted transformation or toward a specific criterion. Instead, it typically involves retrieving images from the gallery based on the combination of multiple texts and the given image.\n\nIn future work, we also plan to explore how our method can be extended or adapted to compositional retrieval scenarios.\n\n\n\nWe will add the above discussions in the next version. \n\nFinally, we sincerely thank you for your positive feedback and insightful comments again, which are encouraging and helpful in guiding the future development of our work."}]

Author rebuttal:

Meta-review: This paper deals with a drawback of the existing class of vision-language models, where such models strive for a universal representation of everything, leading to limited performance in fine-grained and specialized tasks. This paper introduces conditional representation learning, in which representation are extracted to match any user-specified criterion. After the review and rebuttal stage, the reviews unanimously voted above the acceptance threshold (5, 4, 4, 4). The reviewers are clear and short regarding the strengths: the idea is simple yet effective. The AC agrees and finds that the simplicity of the approach is not a valid argument to argue for lack of technical novelty. The reviewers do point out valid concerns regarding comparisons and further experiments. The authors have amongst other included BLIP-2 comparisons and more linear probing evaluations, strengthening the experiments. Based on these outcomes, the AC follows the review recommendations and votes for acceptance.
Decision: Accept (spotlight)