Bridging the gap to real-world language-grounded visual concept learning
Authors: Whie Jung
Semin Kim
Junee Kim
Seunghoon Hong
Venue: NeurIPS 2025 poster ()
Keywords: visual concept learning
vision-language model
image generation
Primary area: general_machine_learning
TL;DR: We propose language-grounded visual concept learning in real-world scenes

Abstract: Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. 
However, existing approaches to language-grounded visual concept learning are limited to a few predefined primitive axes, such as color and shape, and are typically explored in synthetic datasets.
In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. 
Leveraging a pretrained vision-language model and our universal prompting strategy, our framework identifies a diverse image-related axes without any prior knowledge.
Our universal concept encoder adaptively binds visual features to the discovered axes without introducing additional model parameters for each concept.
To ground visual concepts along the discovered axes, we optimize a compositional anchoring objective, which ensures that each axis can be independently manipulated without affecting others.
We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ, and AFHQ, showcasing superior editing capabilities across diverse real-world concepts that are too varied to be manually predefined. 
Our method also exhibits strong compositional generalization, outperforming existing visual concept learning and text-based editing methods. The code is available at https://github.com/whieya/Language-grounded-VCL.

Bridging the Gap to Real-World Language-Grounded Visual Concept Learning
Whie Jung
Semin Kim
Junee Kim
Seunghoon Hong
School of Computing, KAIST (whieya, seminkim, kje0312, seunghoon hong) @kaist. moansskaiit.,...mmmmmmmm acciisttt...k. kc.
Abstract
Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. However, existing approaches to language-grounds visual concept learning are limited to a few predefined primitive axes, such as color and shape, and are typically explored in synthetic datasets. In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Leveraging a pretrained vision-language model and our universal prompting strategy, our framework identifies diverse image-related axes without any prior knowledge. Our universal concept encoder adaptively binds visual features to the discovered axes without introducing additional model parameters for each concept. To ground visual concepts along the discovered axes, we optimize a compositional anchoring objective, which ensures that each axis can be independently manipulated without affecting others. We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ. and AFHQ. showcasing superior editing capabilities across diverse real-world concepts that are too varied to be manually predefined. Our method also exhibits strong compositional generalization, outperforming existing visual concept learning and text-based editing methods. The code is available at https: thub. com http//hiiieyaaaeee SahieyyaLLanguge.... www.anaaggggggundeede---r grwunded-
Introduction
Perceiving the world through visual concepts such as color, shape, and texture, as human intelligence does, has long been a goal in computer vision. Representing an image as a composition of these concepts not only improves compositional generalization 8 [5 36 37, but also offers interpretable
explanations [14] and enhances visual reasoning tasks 9 38 Early work primarily used discrete language descriptors, ranging from object labels in classification and detection 7 sentence-level captions
15
20
31 to
recent method [16] shows that continuous concept embeddings, grounded along language-informee axes, capture subtle visual nuances, e.g., slight color variations, beyond the reach of purely text-based approaches. Thanks to the visual nuances embedded in continuous representations, this method enables the transfer of subtle, image-dependent details in downstream tasks such as image-editing tasks, where discrete text descriptor- approaches 24 often struggle due to limited linguistic expressiveness.
[
Despite this promise, extending the recent approach [16] learn diverse visual concepts in real-world scenes remains underexplored. A central challenge is the reliance predefined concept axes, such as color or shape, for visual grounding, which fails to capture the rich diversity of real images and limits extension to datasets where relevant factors are unknown in advance. Moreover, since each image consists a wide variety of concept axes, relying on a specialized concept encoder for every axis quickly becomes infeasible, substantially increasing model complexity. Constraining each concept embedding to contain information relevant only to a specific concept axis presents another significant challenge. Although directly matching concept embeddings to textual aeteriptors-alreale a disentangled term in nature offers simple remedy for disentanglement [16], it compromises instance-specid speecifc details, as textual descriptors amage-agnostic.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we take step toward a scalable approach for visual concept learning in real-world scenes. We leverage a pretrained vision-language model (VLM) to adaptively identify image-related axes, replacing fixed predefined ones. Using universal prompt design, we guide the VLM to identify diverse image-related axes without relying on prior knowledge. Our universal concept encoder then binds visual features to these discovered axes within single unified architecture. To ensure that the discovered axes remain disentangled while preserving image-specific details, we introduce a compositional anchoring objective that constrains changes within each axis that they only affect the corresponding axis in the generated images. We demonstrate that our scalable framework can capture diverse real-world concepts and enable novel compositions of visual concepts.
In summary, our contributions are as follows:
1. We introduce a scalable framework that grounds visual concepts along diverse, language- specified axes in real- sal-world images.
2. We propose adaptively identifying image-related axes with a pretrained VLM and designing a universal concept encoder that binds visual features to these axes.
3. We design a novel objective for disentangling discovered concept axes in real-world scenes.
4. We evaluate framework real-world concept editing tasks, showing superior editing capabilities and compositional generalization compared to language-informa visual concept learning methods and text-based editing methods.
2 Problem Setup
Our goal is to develop a scalable framework for extracting visual concepts grounded along image- related linguistic axes in real-world images. To this end, we first outline general formulation of language- unguagg-grounded visual concept learning and identify the key challenges in scaling to real- world scenarios. Given an input image I XWCC the objective to extract set of concept representations Z = {zi...., k ). where E Rº encodes visual concepts relevant to concept axis y/s To define interpretable axes among infinitely many concept axes in real-world images, we define each concept axis Y; with natural languages, e.g., age, gender, and expression. Then the goal to learn set of concept encoders Eg, mapping x to visual concepts z; corresponding to each concept axis y/- A typical approach to train such encoders is training jointly with decoder D with an auto-encoding objective. The decoder D is often replaced by a frozen pre-trained text-to-image (T2I) generative model [16] due to training efficiency and remarkable generation capabilities. Formally, the encoders are optimized with the denoising objective:
L ((0)) E.,, [I] t, (Fo, (x)) - c
where e ~ (0, and / ~ (0, 1) denote noise and timestep, respectively.
Since Equation does not guarantee the disentanglement of visual concepts along the concept axes, prior work [16] introduces additional regularization to ground each visual concept z; to the text embeddings V.. which are obtained by querying the pretrained VLM [18] with predefined templates, e.g., "what is the color of the object". We denote by e; e.g., red or blue, the textual descriptions for each axis, and define V; T(::) as their embeddings, where T is pretrained text encoder.
2.1 Challenges and Desiderata
While prior work [16] demonstrated the extraction of primitive visual concepts, e.g., color, shape, and style. primarily simple synthetic datasets, extending this method to complex real- horl-world scenes poses three key challenges. We briefly outline these challenges and desiderata in this section, and discuss how they are addressed in Section 3
Adaptive Concept Axes Concept axes for visual grounding should be determined adaptively for each image, since real-world images exhibit a vast diversity of attributes that cannot be covered by fixed set of predefined axes. Rather than relying on predefined primitive axes, i.e., color or shape, an adaptive mechanism required to automatically identify relevant concept axes for each image.
Scalable Encoder Architecture To support adaptive concept axes, the encoder architecture should be scalable. Implementing Es with set of specialized concept encoders for each concept axis y: would incur a prohibitive number of model parameters, considering infinitely many potential concept axes in real-world scenes.

---

Universal Concept
Adaptive Concapt Axes Discovery
Compositional Anchoring Loss
Input Image
Encoder
Composite Ripresantatiinss
gonder
Ege
"gge "hair_ fecturr
Encoder
fun_-eecuue Fexhura 'breed expresssion
Decoder
VLM
Encoder
Composite Image
imoir yyung wayy
speciess
comp
fur fr
Encoder
[General Deseristion)
(Dutput Examplej
baohground"
Universal Prompt
Aveed
Repressetiations
MLP projection
Figure 1: Overview of our method. Our framework first identifies image-related concept axes by leveraging VLM. We design an universal prompt that guides the VLM to find concept axes across different datasets. Given discovered axes for each image, our universal concept encoder binds visual features to those axes without introducing any specialized concept encoder for each axis. Finally, the encoded concept representations are regularized with a compositional anchoring loss to promote disentanglement between concept axes. Specifically, we randomly swap concept representation with the one in an identical concept axis extracted from different images, and constrain composite images, rendered from randomly swapped representations, to be aligned with composite text descriptions.
Concept Disentanglement Given adaptive concept axes, each representation z; should capture only the semantics of its corresponding axis ye. while preserving image-specific details. A straightforward solution is to align z; with the text embedding v, [16], since texts are already disentangled along concept axes in nature. However, since v; does not encode any instance-specin information,
this aligmment often leads to a suboptimal trade-off in z, between encoding visual nuisance and disentanglement of the concepts.
3 Approach
Based on the desiderata outlined in Section 2.1 we present scalable framework for language- grounded visual concept learning (Figure D. To extract concept axes adaptive to given images, we propose leverage a pretrained VLM with our simple yet effective prompting strategy (Section [.1] Given adaptive concept axes for each image, our universal concept encoder maps the image features to their corresponding visual concept embeddings (Section 3.2). We then train this encoder to disentangle visual concepts along the discovered axes by maximizing compositional anchoring of the representations (Section 3.3) Instead of directly aligning z; with the image-agnone text embedding T(u)), compositional anchoring ensures that changes in z; affect only its corresponding concept axes y: in the generated image space DZZ). Below, we describe each component in detail.
3.1 Adaptive Concept Axes Discovery
Given an image x, we query a pretrained VLM with a prompt P to extract image-dependent concept axes Y {V1,.... yv) and their corresponding textual descriptions V = {UI, VK). Note that K varies per image, and the extracted descriptions V will be used for visual grounding in Section 33. The prompt P should be universal, generalizing across arbitrary images and properly guiding the VLM to capture rich image-related concepts. To this end, we design a universal prompt with two key components: a general task description and an output exemplar. The general task description instructs the VLM to enumerate all visually relevant concept axes presented in a given image. On the other hand, the output exemplar demonstrates the desired granularity of axes by providing a specific instance. By specifying axes in the exemplar, VLM can be steered to find more detailed axes, e.g., hair color, hair texture, and avoid overly coarse categories, egg., color, texture. Remarkably, single exemplar is sufficient to steer the VLM to identify diverse image-related concepts beyond those provided in the exemplar and to generalize to new domains. For example, given an instance of human face that includes the axis 'hair color', the VLM discovers unspecified attributes such as eye color' or 'lip color' for different human faces, and identifies analogous axes for animal images, e.g., fur color. See Appendix A.4 for more details.
3

---

We also instruct the VLM to structure the output as dictionary mapping each concept axis to corresponding textual description, e.g. l'age': 'young'. "gender': 'male',.. ,...). While prior work [T6] employs the VLM to gather textual descriptions for few predefined axes, they query the model separately for each predefined concept axis (e.g.. "What the color of the image?"). In contrast, our prompting strategy extracts all concept axes and corresponding textual descriptions in a single query, greatly enhancing efficiency and covering a broader range of potential axes beyond typical predefined categories. The complete prompt and outputs are provided in Appendix [A.4] We find this universal prompt effectively captures diverse image-related concepts across multiple datasets, including novel concepts, e.g., breed, eye color, and nose color, which were not present in the exemplar.
3.2 Universal Concept Encoder
In our framework, the concept encoder Eg requires to encode visual concepts adaptive image- related concept axes Y. Rather than defining specialized concept encoders for each concept. we construct Eg to encode all concept representations Z conditioned on set of concept axes Y. i.e., 2 = Fx Y). The architecture for Eg should support adaptive binding of visual features to given axes Y and produce distinct concept representations within single parameterized model. To this end, we adapt the Querying Transformer (Q-Former) [18], which was designed to extract visual features from a frozen vision encoder and align them with pretrained text embeddings. The Q-Former consists of a lightweight transformer with learnable queries to interact with visual features via a cross-attention module. In our adaptation, we replace the learnable queries with the text embeddings T(y:)) }) of each axis encoded from a pretrained text encoder T. Initial queries T(v;) are then updated in subsequent transformer layers by interacting with visual features through cross attention layers. This way, visual features can dynamically bind to arbitrary concept axes within a single architecture.
3.3 Disentanglement with Compositionality
To constrain z, to encode only the information relevant to its axis yi, we introduce compositional anchoring objective that ensures modifying a concept along one axis alters the generated output only in that axis, leaving other attributes unchanged. We implement such variations by randomly swapping a subset of concept representations Z' C with those drawn from the same axis of different images, producing composite representations Z As discovered axes vary across images, we first search for candidate images within each batch that share the same axis yi, and then randomly swap their corresponding 2; among these candidates. When each representation z is disentangled along y/. the composite image x DZ7)) should change only the swapped attributes, leaving others unchanged. Since ground-trute truth images for such composition are generally unavailable, instead measure alignment between the composite image x" and composed textual descriptions set VS, constructed by taking the corresponding descriptions from each swapped axis.
We quantify this alignment using lightweight regression network Io that predicts the textual descriptions of a given image. Instead of constructing Ie with an additional image encoder, we reuse Eg to encode x" back into concept representations, and a lightweight regression network Jo predicts each attribute on top of the representations. Note that do is shared across the axes. Formally, let 2- {2] )= Eg (x",, Y) be re-encoded concept representations from composite image za, where Eg is a fixed copy of Eg. Then, the compositional anchoring objective is defined as:
K
LComp (0) = Σ d go vi). i=1
where d(.,,) is a cosine distance and vद is a text embedding for axis yi in Vc, Note that this objective only updates / by propagating the gradient through x and prevents updating 9s and Egg to avoid samples of D()z). For Jø. we simply train it by predicting the
corruption from text embeddings v; from non-swapped concept representations z::
K
LRey (0, +)
Σ
d (2))) v.).
It is worth noting that our objectives do not force 2, = v, which compromises instance-specific details in z, Instead, disentanglement encouraged by verifying that each axis remains independent in the generated output. As result, our objective ensures concept disentanglement while retaining instance-dependent information, particularly crucial in complex real-world scenarios.

---

3,4 Learning objectives
In this section, summarize our complete framework and learning objectives. Given image the VLM extracts set of image-related axes Y. The universal concept encoder Eg is then trained with an autoencoding objective, while the pretrained decoder D remains fixed. To encourage disentanglement among axes, we randomly swap each concept representation z, with another from the same axis in the batch, and measure the alignment of composite image x D(7)) with its corresponding text embeddings v through lightweight regression network Jø. The overall objective is:
Total O, Φ) = Lpir + AComp ()) + AReg Crog (0, (0,
(4)
where Acopp and AReg are hyper-parameters controlling the importance of each term.
4 Related Work
Visual Concept Learning As language offers a human-interpretable interface, grounding visual concepts in natural language has long been central goal in computer vision. Early efforts primarily aligned images with word-level annotations or object labels, supporting classification and detection tasks [15 20 31. Extensions to neuro-symbolic frameworks [19) integrating with visual concept learning, further advanced visual reasoning. Such taguaage-bssed grounding not only enhanced interpretability 14, but also improved downstream performance on vision tasks 17 18]. However, discrete text descriptors inherently limit the representational capacity to fixed vocabulary. To address this, LIVCL [I6] followed Textual Inversion- based approaches [TO] by optimizing concept encoders with pretrained T2I model to reconstruct the given images. While promising, the scope of the work was limited a few predefined primitive concept axes.
Representation Learning with Compositionality Another line of research explores object-centric learning to uncover generative factors. Recent methods [12 compose latent representations from multiple images similar to our framework, but under more restrictive assumptions. For instance, L2C 12 randomly mixes object representations to produce composite images and maximizes the likelihood of these composites to learn object-centric representations. Wiedemer et al. [3 provides theoretical analysis for compositional generalization and measures compositional consistency through cyclic distance between latent representations and their reconstructions. However, this formulation relies on architectural constraints such as additive decoders, making them effective mainly on synthetic or low-complexity data. Without additive decoders, it can lead to trivial solution where a single latent encodes all. In contrast, our approach employs a pretrained T21 model without imposing additional constraints, addressing real-world scenes with diverse concept axes. Instead of focusing on isolated objects, our compositional consistency objective promotes disentanglement among discovered concept axes and does not require specialized decoder structure.
5 Experiment
5.1 Experiment Setup
Implementation Details We leverage Intern VL [4] for an open-sourced VLM. To handle complex real-world images, employ DINO-v2 (26) to encode the image into visual features followed by our concept encoder, and employ Stable Diffusion-based T21 decoder (30 finetuned at 256x 256 res- olution. When generating composite images with the T2I decoder,
iteratively decode for 10 steps using DDIM [33) Since propagating gradients through all these decoding steps is computationally expensive, we follow 66 22] and truncate gradients at the last few decoding iterations. Lastly, we employ 2-layer MLPs for ga. See Appendix A..3 for additional implementation details.
Dataset We validate our framework on complex and unstructured real-world data, where each image contains a diverse set of conceptual axes that is infcasible to manually predefine these axes to cover all possible variations within the data. To this end, we first conduct experiments on subset of the ImageNet dataset. We randomly sampled 20 classes from ImageNet (referred to as ImageNet-S20), covering categories such animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), and scenes (e.g., boathouse, water tower), yielding approximately 28k training images 1~<1.4k images per class). This dataset presents a challenging scenario as each class contains diverse, image-specific visual concepts that are often not shared by other classes. Given the infeasibility of manually defining all the concept axes in ImageNet-S20 for prior visual concept learning methods [16), we additionally compare our
5

---

Table I: Comparisons visual concept editing task. Our method consistently outperforms recent
text-based editing methods B
23
24 and language-infonen informed visual concept learning [16]
EmageNee-520
HO
AFHi--Dog
APRQ
Method
CL) (t)
BLIP (f)
(T)
(L)
(L)
DLIP
CLIP
BLIP
SDbdit
24
0.155
0.381
0.200
0.447
0.250
0.493
0.257
10474
0.198
0.383
0.230
0.467
Towersion
0.189
0.341
DDPM-Inwersingg
0.243
0,467
0.220
0.483
0.266
0.516
0.266
0.494
LIVCI,
L7
0.236
0.469
0.270
0.518
0.268
0.48
0.251
0.474
Target Concept
Instruct Pix2Pix
NaliText Inv.
Source
Target
Ours
SDEdit
LIVCL
Inv.
Species
Cap Color
Not Applicable
Vehicle Type
Stem Color
Flower color
Age
Accessory
Fur Color
Eye Color
Ear shape
Breed
Figure 2: Qualitative results on ImageNet-S20 CelebA-HQ and AFHQ datasets. Our framework grounds visual concepts to diverse concept axes in real-world images. Note that red concepts are not provided by our prompt but rather adaptively discovered by VLM. Since it's infeasible to predefine all axes covering the whole dataset like ImageNet-S20, LIVCL was not applicable for ImageNet-S20.
approach with [6] using relatively controlled datasets with diverse concept axes, such CelebA- HQ [3). AFHQ-Dog, and AFHQ-Cat [5]. We collect the frequently observed axes discovered by method per dataset and them to train the baseline [16] All images resized to 256 for our experiments. For training and validation, we use the following splits: 28k/0.6k images for ImageNet-S20, 27k/3k for CelebA-HQ, and around 5k/0.5k for AFHQ-Dog and AFHQ-Cat.

---

Evaluation Protocol To evaluate whether the concept representations faithfully capture their associated semantics and are disentangled from other axes, we perform visual concept editing task. In this task, we select source and target images and identify the concept axes to be edited. The objective is to transfer visual concept from the source to the target image without affecting other attributes. For evaluation, we use the top-50 and top-10 10 most frequently discovered axes per dataset for ImageNet-S20 and the remaining datasets, respectively, excluding axes that remain constant across the dataset, such the subject type in CelebA-HQ, which is always human. For quantitative evaluation, we measure the CLIP-Score [29] and BLIP-Score [7 between the edited images and their corresponding swapped text descriptions V Specifically, we construct text prompts V such as "a photo of a cat with brown, fluffy, striped fur, against a black background," and evaluate the alignment with the images using CLIP and BLIP. Additionally, we conduct human evaluation, collecting responses from 10 participants per dataset via Prolific [28]. following the procedure in Lee al. [6]. Details on the human evaluation setup are provided in Appendix A.5
Baselines We compare our method to LIVCL [6]. a recent visual concept learning approach that extracts concept representations along predefined primitive axes such as color, category, and style. As LIVCL explored in low resolution images, e.g. 64 x 64 pixels. we replace its T2I decoder and pretrained image encoder with Stable Diffusion 30] and DINO- v2 (26]. respectively. Since LIVCL requires predefined axes for training. we used the t0p-50/10 most frequent axes discovered by our method for ImageNet-S20 and the others. respectively. We also compare our method to four recent text-based image editing methods- SDEdit [23], InstructPix2Pix []. Null-text Inversion 224, and JDPM--Invesion [.. Although these baselines lack mechanisms for extracting visual concepts from source images, we instead edit the image with GT text descriptions given by the VLM. For each method, we used a prompt including target attributes to be changed, such as "a photo of dog with brown fur" for editing. We used default hyper- pprammerreeters for text-based editing methods.
5.2 Main Results
Quantitative Results We report quantitative compar- ison of our method to the baselines in Table [ Our methods consistently outperform all baselines on all of the datasets by a clear margin. High CLIP and BLIP scores demonstrate the effecttieness of our method in capturing image-relate related visual concepts. A human eval- uation in Table 2 provides more direct assessment of reflecting subtle visual nuances. Since text-based editing methods are inherently independent of source images and LIVCL struggles to encode image-dependent details due to its training objective, the performance gap becomes even more pronounced in the human evaluation. These results validate the effectiveness of our framework in visual grounding with diverse axes in real-world scenes.
Table 2: Human evaluation results.
Method
[elebAAHHQ AFtIQ-Dog AFHI
SDEdit
0.448
0.486
0.464
InstrudPix2Pix
0.465
0.385
0.416
NuliText Inversion
0.414
0.514
0.442
DDPM Inversion
0.528
0.548
0.584
LIVCL 0.465
0.478
Ours
0.636
0.589
0.623
Qualitative Comparison Figure presents the qualitative results on visual concept editing. Our method identified diverse set of image-related axes and discovered novel concepts such species, cap color, vehicle type, eye color, and breed, which were not specified in the prompt. It demonstrates that our universal prompt can generalize to unscen domains. Within the discovered axes, our method accurately alters each concept without affecting others. In contrast, LIVCL often fails to encode image-specific details, such as generating different glasses in the seventh row, last column, or disentangling from other axes like changing fur color and texture in the eighth row, last column. We attribute this to the inherent trade-off in LIVCL's objective between concept disentanglement and capturing degeedennnddntt details. Thanks to our compositional anchoring objective, our method achieves both disentanglement along each axis and the preservation of image-specific details, e.g., transferring similar glasses in the seventh row of third column. Text-based approaches also struggle with concept-wise manipulation, often modifying the global color of images (InstructPix2Pix and Null Inversion)))) or leaving them unchanged (SDEdit and @DPM-Inversion), Even when they transfer the correct attribute, they fail to capture the visual nuances of source attributes. For further visual inspection, please refer to additional qualitative results in Appendix A.6
Compositional Generalization Interestingly, our method demonstrates superior compositional generalization to unseen combinations of concepts compared to the baselines, shown in Figure In the figure, our method successfully generates novel compositions, such as large frog with a panda's fur pattern, pandas with red eyes, or scooters floating on water, which do not exist in the real world. In
7

---

Instruct Pix2Pix
NuilText Inv.
DDPM Inv.
Target
SDEdit
LIVEL
Species
Not Applicable
Vehicle Type
Eye Color
Ear Shape
Nose Color
Figure 3: Compositional generalization to unseen concept combination. Given OOD combination of concepts such as frog with panda's 's fur pattern, only our method generates plausible results.
contrast, the baselines either alter multiple attributes simultaneously change nothing at all, and fail to generate plausible generalizations. For instance, when modifying species or eye color, body colors are also changed as seen in the fifth and sixth columns of the first row, and the fifth column of the third row. Furthermore, while all baselines struggle with manipulating car shapes or nose colors, which are strongly correlated with dog breed, our method shapes the ears of a Labrador into a triangle (third column of fourth row) and renders a dog's s nose in pink (third column of fifth row). We conjecture that such compositional generalization arises from our compositional anchoring objective, which explicitly promotes random composite images to exhibit corresponding compositions of attributes.
Composition From Multiple Images To further analyze the quality of extracted visual concepts, we consider the more challenging multi-image composition task. For each target image, we select N source images and randomly sample unique concept axes from each source, i.e., N different axes. We then edit the target image along those axes to produce composite images. We conduct this task only on CelebA-HQ and AFHQ, as the high image diversity within ImageNet classes (e.g., partial views, different viewpoints, or varying light conditions) often leads to cases where the concept axes are not consistently shared among the same class images, resulting in noisy evaluations. In contrast, CelebA-HQ and AFHQ have more controlled structures, making them better suited for this task.
Table 3 presents CLIP and BLIP scores for editing up to four axes. Our method again consistently outperforms all baselines. Moreover, all baselines suffer a clear drop in both metrics as N increases, whereas our method shows only a marginal decrease. This robustness indicates that our concept representations are well disentangled and faithfully capture the correct semantics of the input images. Figure 4 shows qualitative results for N = 3. Composite images from our method are faithfully modified to reflect all of the source images' concepts. In contrast, the baseline models often omit or distort certain attributes. For example, all baselines fail to render the short hairstyle and blue earrings (first row), and some either drop the facial expression (InstructPix2Pix, DDPM Inversion) or misapply the hair color (LIVCL. SDEdit) in the composite outputs (second row).
Visual Nuance Transfer In contrast to text-based editing methods, visual concept learning methods can capture visual nuances the continuous representation space. Since LIVCL is not applicable to ImageNet-S20, we compare our methods to LIVCL on the CelebA-HQ and AFHQ datasets. Figure 5 highlights visual nuances captured in the concept representation of our method. In the figure, our method transfers subtle visual details such as detailed fur patterns, subtle differences in smiles, or hair color tones. In contrast, LIVCL struggles to correctly transfer these visual details, e.g., the resulting image always exhibits the same expressions in Figure ((.)). It even fails to reconstruct the original images Figure5(e), This implies the suboptimal trade-off between concept disentanglement and image-dependent encoding induced by the objective in LIVCL. While pushing concept representations z, closer to text embeddings v;, iee., 2; = v,, can achieve disentanglement, it sacrifices visual information. In contrast, compositional anchoring objective bypasses such
8

---

Table 3: Comparisons of visual concept editing. Our method outperforms recent text-based editing and language-in informed visual concept learning [6].
methods S
23
24
CelebA-HQ
AFHO-Dog
AFHQ-Cat
method
CLIP
BLIP
CLIP
BLIP
CLIP
BLIP
N=2
N=3
Ne4
N=2
No3
Ne4
N=2
No3
Ne4
N=2
No3
N=4
N=2
No3
No4
NN2
No3
Nisd
SDEdit
0.203
0.202
0.204
0.443
0,440
0.439
0.251
0.254
0.255
0.493
0.496
0.494
0.255
0.254
0.254
0.466
0,456
0.443
2nstructPix2Pix
0.201
0.199
0.197
0.416
0.413
0.411
0.225
0.224
0.221
0.456
0.458
0:433
0.248
0.250
0.249
0.462
0.463
0.450
Nulffest Inv.
0.206
0.199
0.194
0.428
0.420
0.417
0.247
0.250
0.250
0.479
0.482
0.480
0.256
0.257
0.258
0.471
0.469 0.468
DDPM Inv.
0.213
0.207
0.203
0.463
0.449
0.437
0.262
0.260
0.257
0.506
0.501
0.493
0.261
0.258
0.253
0.473
0.458 0.440
LIVEL
0.260
Ours
0.238
0.236
0.236
0.492
0.490
0.491
0.269
0.266
0.262
0,528
0.528
0.523
0.271
0.269
0.268
0.516
0.513 0.512
Concepts
Instruct Pix2Pix
NuliText
DDPM
Target
Ours
LIVCL.
SDEdit
Age
Lengih
Accessory
Inv.
Iny.
Expresonne
Gender
Nove Color
Bar Shupe
For Color
Far Coker
Ejje Coior
Buckground
Figure 4: Composition of visual concepts from multiple images. Only method accurately reflects all of the attributes of source images.
Targer
Ours
LIVEL
Far Color
Expression
Hair Color
Figure 5: Examples of visual nuance transfer. Even when transferring the same attributes, e.g., black and white fur blonde hair, the outputs reflect subtle details of source images.
trade-off and thereby achieves both disentanglement and rich image- degeedeepnddntt details within representations. More qualitative results on visual nuance transfer are provided in Appendix A.6
5.3 Ablation Study
In this section, we conduct ablation study VLM choices, architectural design choices, and objective functions to examine the robustness and effectiveness of our choices. All of the experiments evaluated the visual concept editing task in the CelebA-HQ dataset.
Since the discovery of concept axes in our method is directly affected by the quality of VLM outputs, we examine the robustness of our framework on two additional popular open- sourced VLMs (Qwen2.5-VL 2 Ovis2 which have ranked highly reasoning benchmarks.
VLM choices

---

Table 4: Ablation study on VLM choices.
Table 5: Ablation study on Architectural choices.
VLM choices
CLIP
BLIP
Archisctural choices
Metries
Qwen2.S-VL 2
23.72
48.64
Decoder
Vision Encoder Concept Encoder
CLIP
BLIP
Ovis2 21
23.48
48.35
InternVL225 (Ours)
23.88
49.58
Frozen T21
Dinov2
UCE
23,88
49.58
Intern VL:55 10% drop
23.52
48.69
LoRA finetuned T21
Dinuv2
UCE
23.67
49.62
Inter 5 20% drop
23.65
48.61
Frozen T21
Frozen T21
Dinov2
CLIP
Shared MLP
UCE
21.63
22.03
46.21
46.8
Moreover, as is difficult to directly control or quantify VLM performance, we instead control output quality by dropping partial axes (e.g.. 10% and 20%) from the VLM outputs. It is a practical scenario as VLMS cannot always capture the complete axes for a given scene. Table 4 shows that our method is robust to both VLM choices and missing axes. We hypothesize that this is because even though some image-related axes can be missed in each example, those axes will eventually be repeatedly exposed across the dataset. Additionally, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization, which improves adaptation with fewer samples. In fact, our method capable of generating OOD samples (Figure (3). The robustness of our framework regarding the performance of VLMs suggests that it can scale to more complex real-world datasets, as VLMs do not always need to capture complete axes for every scene.
Architectural choices Table 5 presents the ablation studies architectural choices follows: (1) Frozen T2I decoder versus PoRA-finetuned decoder. (2) choice of vision encoder (Dinov2 versus CLIP), and (3) universal concept encoder (UCE in the T versus shared MLP architectures. First, finetuning the decoder with LoRA does not affect overall performance. Large-scale pretrained T21 models have already learned expressive data priors on natural images, facilitating faster training of the generation model. Therefore, the frozen T21 model does not bottleneck our framework. Replacing the Dinov2 encoder with the CLIP encoder causes a significant performance drop, as CLIP is trained for text-alignment, making its discriminative properties inferior to those of recent self-supervised methods like Dinov2. Lastly, replacing our universal concept encoder with shared MLP architecture, which is a naive version of an axis-agnostic encoder, also results in a severe drop. Specifically, the visual feature mean- prean-pooled into a vector, concatenated with axis embeddings, and passed through shared MLP layers to encode concept representations. To make this encoder generally work for diverse concept axes, we shared this MIP layer for all of the axes. This approach likely fails because the shared MI. treats each concept independently, blocking complex interactions between concepts. clearly highlights the effectiveness of our universal concept encoder.
Component-wiss Contribution We conduct an ablation study on each component in our objective for concept disentan- glement, i.e., 9s and LComp. and report the results in Table 6 Without employing 9o and instead directly regressing each con- we observe
Table Ablation study on our method. Both Lcomp and Je con- tribute to concept disentanglement.
cept representation z; to V; in Equation 2 and 3 significant drops in both CLIP-Score and BL.reScore. This result indicates the importance of 96 in preventing direct trade- off between disentanglement and encoding details. Furthermore, removing LComp also leads to suboptimal CLIP- and BLIP-Score. This is because minimizing Equation Jonly guarantees z; to have information of v; but does not prevent it from encoding entangled information related to other concept axes.
Ccomp
CLIP
BLIP
x
21.1
44.72
22,89
47.47
23,88
49.58
6 Conclusion
In this study, we present a scalable framework for grounding visual concepts along adaptive concept axes in real-world scenes. Our framework leverages a pretrained VLM and universal prompt design to adaptively identify diverse, image-related concept axes. A single, unified concept encoder then binds visual features these axes, eliminating the need for separate per-concept encoders. To ensure each axis remains disentangled while preserving instance- detail, we introduce a compositional anchoring loss. We randomly swap concept representations across images and regularize the resulting composite outputs to match their corresponding text descriptions. In the visual concept editing task on real-world datasets, our method consistently outperforms prior approaches in language-informed visual concept learning and recent text-based editing methods, demonstrating the effectiveness of our framework in learning adaptive visual concepts in real-world datasets. Also, our approach demonstrates successful transfer of subtle visual nuances and stronger compositional generalization.
10

---

Acknowledgment This work was in part supported by the National Research Foundation of Korea RSS-2024-00351212 and RS-2024-004361655 and the Institute of Information & communica- tions Technology Planning & Evaluation (ITTP) (RS- IRS-2022-1122026, RS-2022-11220559, RS-2024- 00509279, and (1S20099-11100755 funded by the Korea government (MSIT).
References
[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jiangiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13990 3323, 2025.
[3] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18392- 15392-18402, 18402, 2023.
[4] Zhe Chen, Jianman Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic disual-linguise tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24185 24185-24198, 2024.
[5] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18888197, 2020.
[6] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint drXiv:2309.17400 2023.
[7] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
[8] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE conference on computer vision and pattern recognition, pages 1778-1785, IEEE, 2009.
[9] Yu Gai, Paras Jain, Wendi Zhang, Joseph Gonzalez, Dawn Song, and Ion Stoica. Grounded graph decoding improves compositional generalization in question answering. In Kranc-Franccne Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Associa- tion for Computational Linguistics: EMNLP 202/, pages 1829-188 1838, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 11653/v1/20211 findings-emnlp.Ir 157. URL htttps://aclannhollggeoooooo org 2021. findingsnnn-iii -mnlemmmmmpp...m 157/
[10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611 2022.
[11] Inbar Huberman- Spiegelllas,sss Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469-12478, 2024.
[12] Whie Jung, Jaehoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving object centric learning by injecting compositionalityy arXiv preprint arXiv:2405.0r 006406 2024.
[13] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. 2017.
11

---

[14] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning. pages 13388-344... PMLR, 2020.
[15] Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[16] Sharon Lee, Yunzhi Zhang, Shangzhe Wu, and Jiajun Wu. Language-informed visual concept learning. ICLR, 2024.
[17] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on muchine learning, pages 12888-12900. PMLR, 2022.
[18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[19] Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Zhu. A competence-aware curriculum for visual concepts learning via question answering. In European Conference on Computer Vision, pages 141-157, Springer, 2020.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740-755. Springer, 2014.
[21] Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024.
[22] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: I 1904. 12584, 2019.
[23] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[24] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6038 1188-604,,10 2023.
[25] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 169-1885 2018.
[26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[27] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak. and Katerina Fragkiadaki. Aligning text-to-
image diffusion models with reward backpropagation, 2023.
[28] Prolific. Prolific (version march 2025) wee platform). 2014. First released in 2014. Copyright
2024. London, UK. Available at
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
12

---

[31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 15:211-252, 2015.
[32] Donald G Saari. Selecting voting method: the case for the borda count. Constitutional Political Economy, 34(3)3557-366 2023.
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[34] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[35] Thaddäus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. arXiv preprint arXiv:23 2023.
[36] Yuan Zang, Tian Yun, Hao Tan, Trung Bui, and Chen Sun. Pre-trained vision- Asioonllnguage models learn discoverable visual concepts. arXiv preprint 12255 2024.
[37] Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, and Fei Sha. Visually grounded concept composition. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguisties: EMNLP 2021, pages 201-215, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/20// htnp://////2.....nnnnnnn-oinn findings-emmllp.20 URL http:s:///ddaaatioooogy.oo 2021 wealffindinggs emnlp.m.mm 20/
[38] Yiwu Zhong. Zi- Yuan Hu, Michael R Lyu, and Liwei Wang. Beyond embeddings: The promise of visual table in multi-modal models. arXiv preprint arXiv:2403. /8252, 2024.
13

---

A Appendix
A.1 Limitations and Future Work
In our work, as in most previous approaches, we cannot guarantee recovery of every ground factor of variation. Some subile or rare attributes may simply fall outside the axes we discover. In fact, perfectly capturing all underlying factors in a complex, real-world dataset is generally intractable. Nevertheless, our method still identifies diverse, meaningful concepts, and extending coverage to additional or more fine-grained factors remains an important direction for our future work. Moreover, our framework depends on the quality and scope of the pretrained vision language model (VLM), so it can only discover concepts the VLM recognizes. Fortunately, as VLMs are improving rapidly and our method is not restricted by specific VLM, we can adopt stronger models as they become available.
A.2 Broader Impact
Our approach can extract diverse visual concepts from images and reuse them to synthesize new content, which could pose privacy issues such as deepfake generation or unauthorized duplication of digital content.
A.3 Additional Implementation Details
Table 7 summarizes hyper-parameters for model architectures and training used in our experiments. For baselines, we follow the default hyper-parameters recommended by the official codes. All baselines used DDIM inversion with guidance of 7.5 and 50 inference steps.
Table 7: Hyperparameters used in our experiments.
General
Ratch Size
Training Steps
100k 0.00003
Learning Rate
Concept Encoder
Layers
Hidden Dim Number of Heads
768
Rogression Network
Layers
768
Input Dimension
768
Tlidden Dimensio
768
Activation Function
ReI.U

---

A.4 Prompt for Concept Axes Extraction
We provide the complete prompt and examples the discovered concept axes per image Figure and Figure / respectively. As shown in Figure 7 our prompt successfully steers the VLM to identity diverse concept axes across different datasets, even when using only single output exemplar of human face.
<image>in You are given image containing any subject (e.g. person, animal. furniture, vehicle, etc.). Your lask identify all relevant visual concept for the subject in broad yet comprehensive way, Recause the subject can vary widely, ensure your appnach applies to humans, animals, vehicles, objects, and more. These axes should include all essential visable attributes of the subject with enough detail to plausibly reconstruct image. Each should fine grained enough for distinct, specific attributes (for example, "hair length" mather than "appearance", "fur color" rather than "fur").
General Task Deseription
Each chosen axis (key) should map to short, descriptive value.
Use concise yet precise terms. Avoid vague or overly brief descriptors like "medium"" by replacing them more specific language
"long, past shoulders" or "short, tied back.") Avoid repeating sume detail across multiple axes; keep attributes distinct prevent overlap. Return your final answer single ISON dictionary.
For example, subject a human:
"subject type": "human", "age": "young adult", "gender": "female" "hair color": "black" "hair longth": "short, above cars", "hair texture": "struight" "expression", "smiling", "background". "white". accessory" "gold, dangling earrings")
Outpat Exemplar
Output only the final JSON.
Figure 6: Our complete prompt consists of general task description and output exemplar.
"wrjjer typr "dag".
pender" "temake.
Tall, coke "Moaae"" lai: lagel" "ancian, aatokeale kouure" "ward'
"ia Jalare" "daee".
"he "hown black
"aar dhpe" "eepy"" tsr aolu "bosss" especeaess "apps""
"estjct tope" "axiral" "spcied" "Naf"
'qeigla".
"ecabul".
"tepleppeilaa" "elber Naair
"tockgrend" fhllge
"aakkgeedd" "gay""
bacckaossal: "gmaa"
babject byse": "sassar'
"agnn "yoong
"ertject bysr "uvinaa"
"spdic"" May..
"sijjotttp"" "axteal"
Jogght?
loatue" "wmaggh""
shoulles
"ur usture?
"wlik".
"he ouee' "arono and back" "ayy oober "gres"
"Beganee "oriped"
"espresiio"" "eeatal".
backemeend: "sbck".
"car "ported".
'ss caker "sask"
"ogpeesau"" "searcoive".
Tip poloe" "ckihing" "jivk
Tome 2ooe" Teckarouad" fosadocs,
"hucyrond "panenoes"
Figure 7: Examples of outputs from the VLM. Concept axes colored in red are unseen from the given exemplar.

---

A.5 Human Evaluation
For human evaluation, we randomly select 10 pairs of images for each attribute. Then, we replace an attribute of one image with another one in each pair using each of the methods. We ensure that randomly selected attributes in each pair are different from each other so that the edited image is always recognizable. We collect 10 participants for each dataset (a total of 30) Prolific [28] and
provide general guideline as Figure X for the task. Our questionnaire (Figure 2 asks participants to rank the images that most closely adhere to the criteria provided in our guideline. Following [16]
we used Borda score metrics [32] to differentiate the scores according each ranking, and final scores are normalized to 0-1 scale.
Concept Swapping Visual Inspection
survey, you be presented pairs images alongside a target concopt edited image should modify. Your task is to evaluate and rank the edited images that most closely adheres to the following crnada:
Concept Adherence:
Given target concept, the edited Image should accurately reffect the same concept details the source image. For example, target concept "heir color", hair oolor the edited image should be seme as that the image.
Preservation Other Attributes:
Only the details the given target concept should change, while other attributes must remain unchanged. The other concepts remain unchanged specified for each task.
You given questions total, questions each the concepts.
Adoittional Guidelines:
is bust whiil is the warst,
Take time to each carefully before making your selection. none of the answers seem accurata, answer with your best guess. tles.
Please ZOOM IN your images for better inspection.
Figure 8: General guidelines used in our human evaluation.
1. Which image best represents (C), reflecting the fur_ fur_color' from (B) while keeping all other
details from (A)?
Rank the answers: 1st (Best) 66th (Worst)
Attributes (C):
for
expression:
background. (A)
(A) Target image
(B) Source image
(C)
(b)
(c)
(d)
(e)
10
13
0
10
10
Figure 9: Questionnaires used in human evaluation.

---

A.6
Additional Qualitative Results
A.6.1
Additional Qualitative Comparisons on Visual Concept Editing
Figures 19 present additional qualitative results along diverse concept axes discovered in ImageNet-S20,, CelebA-HQ. and AFHQ datasets. Across all axes, our method consistently out- performs the baselines. Whereas the baselines often fail accurately capture and transfer the specified visual attributes, our approach reliably extracts the visual concept from the source and transfers the target image. Since LIVCL trains set of separate encoders only for the top-10 frequent axes, it was unable to evaluate "lip color" in Figure and and "collar" in Figure [88 which are not among the top-10 most frequent concepts the dataset, and we therefore mark those entries as N/A.
Target Concept
Instruct
NulfText
DDPM
Source
Target
Ours
SDEdit
Pix2Pix
Inv.
Inv.
Species
Cap Color
Sky Condition (e.g., clear, cloudy)
Figure 10: Additional qualitative comparison to baselines in ImageNet-S20

---

Target Concept
Instruct Pix2Pix
NullText Inv.
DDPM
Source
Target
Ours
SDEdit
Eye Color
Vehicle Type
Wing Color
Figure 11: Additional qualitative comparison to baselines in ImageNet-S20
5

---

Target Concept
Instruct Pix2Pix
NulfText Inv.
DDPM Inv.
Source
Target
Ours
SDEdit
Stem Color
Flower Color
Age
Figure 12: Additional qualitative comparison to baselines in ImageNet-S20

---

Target Concept
Instruet Pix2Píx
NuliText Inv.
DDPM
Source
Target
Ours
LIVCI.
SDEdit
Inv.
Age
Gender
Hair length
Figure 13: Additional qualitative comparison to baselines CelebA-HQ

---

Target Concept
Instruet Pix2Píx
NullText Inv.
DDPM Inv.
Source
Target
Ours
LIVCI.
SDEdit
Accessory
Hair color
Clothing
Figure 14: Additional qualitative comparison to baselines CelebA-HQ

---

Target Concept
Instruet Pix2Píx
NullText Inv.
DDPM
Source
Target
Ours
LIVCI
SDEdit
Inv.
Hair Texture
Expression
N/A
B
Lip Color
N/
m
N/A
Figure 15: Additional qualitative comparison to baselines CelebA-HQ

---

Target Concept
Instruct Pix2Píx
NuliText Inv.
DDPM Inv.
Source
Target
Ours
LIVCI
SDEdit
Breed
Fur color
Ear shape
Figure 16: Additional qualitative comparison to baselines in AFHQ-Dog
10

---

Target Concept
Instruct Pix2Pix
NullText Inv.
DDPM
Target
Source
Ours
LIVCL
SDEdit
Inv.
color
Background
Expression
Figure 17: Additional qualitative comparison to baselines in AFHQ-Dog

---

Target Concept
Instruct Pix2Pix
NullText Inv.
DDPM
Source
Target
Ours
LIVCL
SDEdit
Inv.
Fur Texture
Eye Color
N/A
Collar
N/A
N/A
Figure 18: Additional qualitative comparison to baselines in AFHQ-Dog
12

---

Target Concept
Instruet Pix2Píx
NuliText Inv.
DDPM Inv.
Source
Target
Ours
LIVCI
SDEdit
Fur color
Background
Eye color
Figure 19: Additional qualitative comparison to basclines in AFHQ-Cat
13

---

Target Concept
Instruct Pix2Pix
NullText Inv.
DDPM
Source
Target
Ours
LIVCL
SDEdit
Inv.
Expression
Ni
Fur Pattern
Figure 20: Additional qualitative comparison baselines in AFHQ-Cat
14

---

A.6.2 More Qualitative Results on Compositions from Multiple Images
We provide more qualitative results on the composition of visual concepts from multiple images in Figure 27 We extract N distinct visual concepts from N different images and replace the corresponding visual concepts of the target images with them. Our method successfully transfers multiple visual concepts to target images, which implies that each visual concept extracted from source images is disentangled along other axes.
Concepts
NullText
Target
Ours
LIVCL
SDEdit
Agee
Hair Color
Cloching
Pix2Pix
Inv.
Inv.
Gender
Hair
Rackerowd
Expressiou
Shape
Broed
Nose Color
Eye Color
Fur Textuve
Background
Exprezsion
For Color
Expression
Barkgroomd
Fw Patton
For Coltr
Figure 21: Compositions of visual concepts from multiple images (N 3).
15

---

Concepts from Source
Instruct Pix2Pix
NuliText Iny.
DDPM
Target
Ours
LIVCL
SDEdit
Expeesiio
Hair Langth
Accesoory
Gender
Color
Clocking
Baokground
Exprezzion
Fur
Backgrowmi
Nous Color
Fur
Expression
Backrround
Figure 22: Compositions of visual concepts from multiple images (N 4).
16

---

A.6.3 More Qualitative Results on Visual Nuance Transfer
We provide more qualitative results on transferring visual nuance from source to target images in Figure 23 27
Souree
Ours
LIVCL
(x) Expression (to sarprising)
(b) Expression (to smiling)
(c) Expression (to smiling)
Figure 23: Transferring Visual Nuances from source to target images
Target
Source
Ours
LIVCL
(a) Hair color (tn blonde)
(h) Hair color (to light brown)
Hair color (to browa)
Figure 24: Transferring Visual Nuances from source to target images.
Target
Source
Ours
LIVCL
(4) Fur color (to black)
(b) whitokprey)
(6) orange)
Figure 25: Transferring Visual Nuances from source to target images.
17

---

Source
LIvCL
(4) (to light brown)
color brown)
Rackground (te green grass)
Figure 26: Transferring Visual Nuances from source to target images.
Target
(n) Background (to outdoor, blurred)
(b) Background (to green foliage)
(c) Fur color (to brown)
Figure 27: Transferring Visual Nuances from source to target images.
A.7 Computing Resources
All of our experiments are conducted on GPU Server that consists of an Intel Xeon Gold 6230 CPU, 256GB RAM, and 8 NVIDIA RTX 6000 GPUs (with 48GB VRAM). takes about 48 GPU hours for each dataset.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We states our motivation, contributions, scope of our work in abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
We provide limitation of our work in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not claim for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the information needed to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No].
Justification:
Our code is not cleaned and prepared enough for sharing.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http://oeeubiiiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the details for experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since our method requires costly GPU cost and time in training the diffusion model real images, were not affordable to conduct and provide repetitive experiments. We will add it in future.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information of computing resources used for the experiments in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our paper possess no risk
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all the codes, data, paper, and pretrained model our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer: [Yes]
Justification; We provide detailed instructions of our human evaluation and we provide proper rewards participants through Prolific website.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our method and human evaluation possess no risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We used VLM for automatic extraction of visual concepts and provide detailed information in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non ntn-standard components.
Please refer to LLM policy httppp////eerrppeccc///oeeeerees for what should or should not be described.
25


Reviews:
[{"review_id": "IV8VnByjIb", "reviewer": "Reviewer_vNjW", "summary": "The paper presents an approach for visual concept learning that automatically discovers an open-ended set of semantically meaningful axes along with images in a dataset can vary.  It leverages VLMs to both propose the axes of variation and ground these concepts.   The discovered axes permit more flexible and precise image editing, as well as compositional generalization of multiple image analogies at the same time.", "questions": "Any further information the authors could give that addresses the weaknesses above would be appreciated!", "limitations": "More details on limitations, linked up to analyses of specific kinds of errors that the system makes, would be helpful.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper's motivation is clear and reasonable.  The results show both qualitative and quantitative improvements over baselines. I am not an expert in the paper's area and so I can't really judge the novelty or level of contribution. \n\nWeaknesses: The paper doesn't offer a fundamentally new approach or a fundamental, principles-driven or theory driven approach.  It's all about prompting and tuning the VLM.  It's hard to see what I learn from the work of general or broader interest. \n\nThe evaluations are based on relatively limited datasets, drawn from very limited domains (faces and animals). On both datasets, there are already well-defined axes during data collection (and VLMs/LLMs have seen those examples). It’s unclear how the approach could generalize to more unstructured datasets.  \n\nAlso, on both faces and animals, images can be aligned to a common 2D reference frame which might make concept discovery and independent editing easier – and unrepresentative of what we would expect to see in a larger and more diverse dataset. \n\nAnother potential limitation is that the method relies on VLMs proposing the semantic meaningful axes and attributes for individual images (starting with asking VLMs to categorize all attributes in the input images). There seems to no robustness analysis: what will happen if the VLM misses an axis, or misses an attribute, or how often this happens or under what circumstances.", "was_revised": true, "final_justification": "I appreciate the extra simulations and results but I don't feel they strongly show a major advantage over other work. They also don't negatively impact my impression. So I will keep my score the same.", "rebuttal": "We appreciate the valuable comments. Below we respond to individual concerns,\n\n\n>Q1. The paper doesn't offer a fundamentally new approach or a fundamental, principles-driven or theory driven approach.\n\nWe would like to first clarify the contribution of our work. It is generally difficult for machine learning algorithms to capture underlying visual concepts in real-world scenarios, since each image consists of very diverse, image-specific axes. While it is infeasible to disentangle all of those factors without supervision, a recent breakthrough has been made by incorporating a vision language model to ground each concept axis with textual description [1]. However, this method still requires manual definition of concept axes and thereby limits its application to complex real-world scenes having infinitely many axes.\n\nIn this work, we address such challenges by designing our framework to adaptively perceive image-related axes aided by VLMs. We believe the problem setting and overall formulation itself is an important contribution to the field as it provides a first step towards visual concept learning in a general, real-world scenes. Moreover, we also propose novel objective (compositional anchoring loss) to disentangle each representation along image-specific axes, and propose a simple yet effective architecture (Universal Concept Encoder) to encode a varying number of concept representations. Our ablation study (Table 3 in our main paper) supports that novel objectives is essential component of our framework. Moreover, we provide additional experiments on more diverse and complex natural image datasets, which highlights the effectiveness and scalability of our framework (Please see our response to Q2.) \n\n\n>Q2. Experiments on more diverse and unstructured datasets\n\nTo validate our framework on larger, diverse, and unstructured data, \nwe randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), and scenes (e.g., boathouse, water tower) yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images. \n\n\nAcross this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length. \n\nFor quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (We provide top-50 frequent axes at the end of the response) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs. \n\nBelow Table demonstrates superior performance of our method over text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add extensive qualitative results in our main paper. \n\n=== [Table 1] ===\n|  | Pix2Pix | NullText Inversion | SDEdit | DDPM Inversion | Ours |\n|---|:---:|:---:|:---:|:---:|:---:|\n| CLIP Score ($\\uparrow$) | 19.83 | 18.93 | 19.54 | 24.32. | **25.10** |\n| BLIP Score ($\\uparrow$) | 38.34 | 34.10 | 38.10 | 46.70 | **47.44** |\n\n\n> Q3. Robustness analysis on on VLMs\n\n\n\n\n\n To analyze the robustness of our framework to VLM, we conduct two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes from total axes provided by the VLMs. \n\n\nFor the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and Ovis2) that have shown high ranks in the leaderboard on reasoning benchmarks. \nFor the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the performance differences when we explicitly drop partial axes (10% and 20%) from the axes provided by VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset. \n\nThe below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.) \n\n|  | Ours \\+ InternVL2-5 | Ours \\+ Qwen2.5-VL | Ours\\+Ovis2 | Ours +InternVL2-5  \\+ 10% drop  | Ours +InternVL2-5  \\+ 20% drop  |\n|---|---|---|---|---|---|\n| CLIP Score ($\\uparrow$) | 23.88 | 23.72 | 23.48 | 23.52 | 23.65 |\n| BLIP Score ($\\uparrow$) | 49.58 | 48.64 | 48.35 | 48.69 | 48.61 |\n\n> Q4. More details on limitations, linked up to analyses of specific kinds of errors that the system makes, would be helpful.\n\n\nAs the reviewer pointed, dependency on VLM can cause some errors in practice. For we found that VLM often wrongly annotates attributes of the small objects, or often confuses the attribute of close objects, which possibly affects the overall performance. We will add these specific failure cases in the Appendix. \n\n\n**Reference**\n\n[1], Lee et al., Language-informed visual concept learning, in ICLR 24. \n\n\n**Top-50 axes discovered by our framework**: \n\n[subject_type, background, color, material, shape, species, size, type, vehicle_type, environment, position, cap_color, gender, age, object_type, hair_color, hair_length, lighting, accessory, location, water_body, expression, clothing, activity, sky_condition, eye_color, body_color, windows, wing_shape, fur_texture, stem_color, surroundings, sail_color, seat_color, wing_color, wing_pattern, fruit_type, color_pattern, surface_texture, limbs, flower_color, style, design, wall_color, texture, brand, container_count, flooring, cap_texture, boat_color]"}, {"review_id": "JktS0hTflR", "reviewer": "Reviewer_4ztc", "summary": "The authors propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Specifically, they propose to adaptively identify image-related axes with a pretrained VLM, and design a universal concept encoder that adaptively binds visual features to these axes.", "questions": "According to the weaknesses I previously highlighted, addressing the following points could potentially change my opinion:\n1. Provide a thorough analysis and discussion regarding the reliance on VLMs.\n2. Try to modify the current architecture or clarify that it is not merely a building-block work.\n3. Conduct comprehensive experiments on more complex datasets and compare the results with SOTA approaches.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors propose visual concept learning from the perspective of disentangled representation learning is a sensible approach.\n\n2. The results indicate that the model possesses a certain degree of attribute editing and disentanglement capability.\n\n**Weaknesses**\n\n1. The heavy reliance on Visual Language Models (VLM) diminishes the contribution of this work. Prompting VLM to define attribute axes and values is straightforward, and it limits the model to the upper-bound of the VLM. This approach is based on the assumption that VLMs or MLLMs can accurately perceive the attribute axes for all scenes. While pre-trained VLMs may perform well with simple objects, they may struggle in more complex scenes.\n\n2. Following the first weakness, I expect the paper to provide empirical evidence in more complex scenes; however, it appears that only visual results from facial datasets (CelebA and AFHQ) are presented. The authors should include additional results from challenging scenes, such as cars, buildings, and animals, to demonstrate the generalizability of the model.\n\n3. As a concept/representation learning approach, the combination of VLM prompting, Q-former encoder, and Compositional Consistency Loss is too shallow and simplistic. I believe the authors should delve deeper into how VLM perceives attributes rather than merely using it as a tool. Additionally, I expect a more specially designed autoencoder architecture for concept learning to meet the standards of top-tier venues.", "was_revised": true, "final_justification": "According to the responses, the authors have addressed my primary concerns regarding the necessity of applying the framework to more complex scenes and datasets. Even I maintain that the significant reliance on VLM may impose limitations on the model's upper bound, I am inclined to raise my score to 4, reflecting my appreciation for the authors' current stage of work. I look forward to the inclusion of additional visual results and a more in-depth discussion of VLM's perceptual capabilities, as promised by the authors.", "rebuttal": "We appreciate the valuable comments. Below we respond to individual concerns,\n\n> Q1. Reliance on Visual Language Models (VLM)\n\nWe agree with the reviewer that VLM cannot always perceive a complete set of concept axes. Nevertheless, we believe that the use of VLM provides a significant advantage in complex scenarios, where it is almost infeasible to manually annotate diverse image-specific concepts. Although VLM would not be perfect, it can still capture meaningful and diverse visual concepts ( Please refer to our response to Q2 below.) \n\nIn addition, we conduct additional experiments to investigate the robustness of our framework to VLMs. We perform two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes from total axes provided by the VLMs. For the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and OVis2) that have shown high ranks in the leaderboard on reasoning benchmarks. For the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the performance differences when we explicitly drop partial axes (10% and 20%) from the axes provided by VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10% and 20% of random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset. \n\n\nThe below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes. \n\n=== [Table 1] ===\n\n|  | Ours \\+ InternVL2-5 | Ours \\+ Qwen2.5-VL | Ours\\+Ovis2 | Ours +InternVL2-5  \\+ 10% drop  | Ours +InternVL2-5  \\+ 20% drop  |\n|---|---|---|---|---|---|\n| CLIP Score ($\\uparrow$) | 23.88 | 23.72 | 23.48 | 23.52 | 23.65 |\n| BLIP Score ($\\uparrow$) | 49.58 | 48.64 | 48.35 | 48.69 | 48.61 |\n\n> Q2. Experiments on more complex dataset. \n\nFollowing the suggestion of the reviewer, we validate our framework on larger, diverse, and unstructured data. We randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), and scenes (e.g., boathouse, water tower) yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images. \n\nAcross this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length. \n\nFor quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (We provide top-50 frequent axes at the end of the response) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs. \n\nBelow Table demonstrates superior performance of our method compared to text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add qualitative results in our main paper. \n\n\n|  | Pix2Pix | NullText Inversion | SDEdit | DDPM Inversion | Ours |\n|---|:---:|:---:|:---:|:---:|:---:|\n| CLIP Score ($\\uparrow$) | 19.83 | 18.93 | 19.54 | 24.32. | **25.10** |\n| BLIP Score ($\\uparrow$) | 38.34 | 34.10 | 38.10 | 46.70 | **47.44** |\n\n> Q3. Try to modify the current architecture or clarify that it is not merely a building-block work.\n\nWe would like to first clarify that our main scope is not designing a specific model architecture to improve the performance. Instead, our main contribution is to build a framework that can adaptively capture image-specific visual concepts in real-world scenes. \nTo this end, we focus on designing our framework to address such challenges, while leaving other components general for further applicability. \n\n\nAdditionally, to justify the choice of each component, we conduct ablation studies as follows: (1) Frozen T2I decoder versus LoRA-finetuned decoder, (1) Choice of vision encoder (Dinov2 versus CLIP), (3) universal concept decoder versus shared MLP architectures. \n\n\nFirst, finetuning the decoder with LoRA does not affect the overall performance. Large scale pretrained T2I models are known to be already learned expressive data priors on natural images, which helps faster training of the generation model. Thus the frozen T2I model does not bottleneck our framework. Replacing Dinov2 encoder with CLIP encoder leads to significant performance drop. This is because CLIP is only trained for text-alignment and thereby it’s discriminative property of representation is likely to be inferior to recent self-supervised learning methods (Dinov2). \n\n\nLastly, we replace our universal concept encoder with shared MLP architecture, which is naive version of axis-agnostic encoder. Specifically, the visual feature is first represented as vector through mean pooling, and it is concatenated with axis embeddings followed by MLPs to encode concept representations. To make this encoder generally work for diverse concept axes, we shared this MLP layer for all of the axes. However, this modification causes severe performance drop. We conjecture that this is because shared MLP encodes each concept independently, possibly blocking complex interaction between visual concepts. It clearly highlights the effectiveness of our universal concept encoders. \n\n=== [Table 2] ===\n|  | CLIP | BLIP |\n|---|---|---|\n| Frozen T2I  + Dinov2 + UCE (Ours) | 23.88 | 49.58 |\n| LoRA-finetuned T2I + Dinov2 + UCE | 23.67 | 49.62 |\n| Frozen T2I + CLIP + UCE | 22.03 | 46.21 |\n| Frozen T2I  + Dinov2 + Shared MLP | 21.63 | 46.86 |\n\n\n\n\n**Reference** \n\n[1] language-informed visual concept learning, in ICLR 24. \n\n**Top-50 axes discovered by our framework** : \n\n[subject_type, background, color, material, shape, species, size, type, vehicle_type, environment, position, cap_color, gender, age, object_type, hair_color, hair_length, lighting, accessory, location, water_body, expression, clothing, activity, sky_condition, eye_color, body_color, windows, wing_shape, fur_texture, stem_color, surroundings, sail_color, seat_color, wing_color, wing_pattern, fruit_type, color_pattern, surface_texture, limbs, flower_color, style, design, wall_color, texture, brand, container_count, flooring, cap_texture, boat_color]"}, {"review_id": "Dzam63OnSF", "reviewer": "Reviewer_xMmD", "summary": "This paper proposes a unified framework for compositional visual concept learning from real-world images along user-defined semantic axes. A universal concept encoder is introduced to extract axis-wise latent representations from images, guided by text embeddings. These representations are used to condition a frozen text-to-image (T2I) diffusion model, enabling compositional image editing and generation. The method is trained using a combination of diffusion reconstruction loss, compositional consistency loss (via representation swapping), and text regression loss.", "questions": "Please refer to Weaknesses.", "limitations": "Please refer to Weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. **Clear motivation and task definition**: The authors clearly articulate the goal of axis-guided concept decomposition and recombination, addressing the need for interpretable and composable visual representations.\n    \n2. **Model modularity and flexibility**: The universal concept encoder is compatible with arbitrary concept axes defined in natural language, allowing potential extensibility.\n    \n3. **Loss design for disentanglement**: The proposed compositional consistency loss is novel and aligns well with the goal of achieving axis-specific representations.\n    \n4. **Qualitative and visual interpretability**: The model demonstrates controllable image generation through partial axis manipulation, which is intuitive and informative.\n    \n\n### Weaknesses and Concerns\n\n**1.Dependence on a single VLM for axis generation**:  \nThe semantic axes in this framework are derived using a single vision-language model. It inherently limits the diversity and robustness of the generated axes. The process also depends on the quality and coverage of the VLM’s representations. Incorporating multiple VLMs or leveraging fixed reference taxonomies (e.g., attribute banks or external ontologies) could improve the generality and semantic coverage of the axis discovery process.\n\n**2.Assumption of axis independence**: The model enforces disentanglement by treating each axis as independent, without modeling correlations or dependencies between semantic axes, which are common in natural images.\n\n**3.Evaluation scope is narrow**: The paper primarily evaluates on image editing and attribute recomposition tasks. It does not compare against related methods in compositionality, zero-shot learning, or attribute-conditioned generation, making it difficult to assess general effectiveness.\n\n**4.Frozen T2I model as a bottleneck**: The use of a fixed text-to-image diffusion model constrains the upper bound of generation quality and may obscure the actual representational quality of the learned embeddings.", "was_revised": true, "final_justification": "As my concern has been mostly resolved, I have decided to raise my score.", "rebuttal": "We appreciate the valuable comments. Below we respond to individual concerns,\n\n> Q1. Dependence on a single VLM for axis generation.\n\nWe appreciate the valuable feedback and suggestions. We agree with the reviewer that the performance of VLM would largely affect our framework. While incorporating multiple VLMs or employing fixed reference taxonomies would be an interesting and effective idea, we found that it is not trivial for extend and formulate the method during the rebuttal period. Thereby we leave it for our important future work. \n\nInstead, we provide a clarification that our method is quite robust to the performance of VLMs. We perform two additional experiments: We measure performance on (1) different choices of VLMs, and (2) randomly drop partial axes during training. For the first experiment, we consider two additional opensourced VLMs (Qwen2.5-VL and OVis2) that have shown high ranks in the leaderboard on reasoning benchmarks. For the second experiment, as it is difficult to control and quantify how much of the axes are missed by VLM, we instead investigate the quality of axes provided by VLMs. We explicitly drop partial axes (10% and 20%) from the axes provided by VLMs to simulate imperfect prediction of VLMs. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset. \n\n\nThe below Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes. \n\n\n|  | Ours \\+ InternVL2-5 | Ours \\+ Qwen2.5-VL | Ours\\+Ovis2 | Ours +InternVL2-5  \\+ 10% drop  | Ours +InternVL2-5  \\+ 20% drop  |\n|---|---|---|---|---|---|\n| CLIP Score (\\uparrow) | 23.88 | 23.72 | 23.48 | 23.52 | 23.65 |\n| BLIP Score (\\uparrow) | 49.58 | 48.64 | 48.35 | 48.69 | 48.61 |\n\n\n\n\n> Q2. Assumption of axis independence\n\n We would like to clarify that we do not impose statistical independence between the axes. \nThe only constraint imposed by our framework is compositionality of visual concept axes. Therefore, in our experimental result, our method often disentangle highly correlated factors such as breed of the dog and folding of the ear shape and generates OOD samples (Figure 5 of our main paper). However, we agree with the reviewer that handling the correlation between semantic axes is indeed an important problem and leave it to important direction for future work. \n\n> Q3. Evaluation scope is narrow\n\n We believe our evaluation protocol on image editing is already implicitly measuring the compositionality of the learned representations. As we compose two or more sets of concept representations and measure alignment between resulting composite and those composite attributes, it can be interpreted as measuring the compositionality of the representation. We appreciate the valuable comment from the reviewer and will find the proper evaluation protocol for measuring compositionality and update it in our main paper. \n\n\n> Q4. Frozen T2I model as a bottleneck. \n\nWe agree that fine-tuning the T2I backbone, e.g. via LoRA, would boost overall image fidelity, but we would like to emphasize that even with a fully frozen diffusion model, our learned embeddings already achieve high-quality, semantically accurate edits. This implies that our framework does not require to train huge generative models every time or store task-specific parameters (e.g., LoRA) for T2I model, which eases the training process of our framework. To support our claim, we finetune our T2I model with LoRA, and report CLIP/BLIP Score for visual concept editing tasks. We find that the effect of finetuning LoRA seems to be negligible in terms of concept learning. \n\n\nNevertheless, we agree with the reviewer that joint training of the T2I model would have clear benefits in image generation or faster adaptation to unseen samples and tasks. We appreciate the valuable comment and leave it for the future work. \n\n\n=== [Table 1] === \n\n|  | CLIP | BLIP |\n|---|---|---|\n| ours + frozen T2I | **23.88** | 49.58 |\n| ours + LoRA-tuned T2I | 23.67 | **49.62** |"}, {"review_id": "QQzJ4BgH3s", "reviewer": "Reviewer_5Dzc", "summary": "This paper focuses on editing capabilities for generative models by learning a concept encoder. The authors start their method by prompting a VLM with the task of finding a set of concepts that are visible in the images. Then, they give their universal concept encoder model those concepts as input, which outputs an associated visual representation (based on DINOv2 features) that is used for conditioning a decoder (in this case, a Stable Diffusion decoder). The universal concept encoder is trained by seeking whether the generated images contain the concepts that are in the original images. The authors ran their experiments on the CelebA-HQ and AFHQ datasets and compared their methods to related works such as LIVCL, SDEdit, Instruct Pix2Pix, NullText Inv., and DDPM Inv. The evaluation is done by swapping the concepts learned between two images and observing the results. We clearly saw that the output images are more faithful to the original image while maintaining the target concept.", "questions": "Do you see any challenges (outside compute) for scaling your method to larger and more diverse natural images dataset? I am curious about the point you made in the appendix on identifying all the concepts being intractable. There is a limited number of words in the human languages, so this concepts are defined by mapping to English words, then this should be tractable, isn't it?", "limitations": "Addressed in the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow. The authors compare their methods with most of the related methods. The ablation study on the loss terms is also a great addition to the paper.\n- The authors provide a lot of visualizations that are great to see the improvements that are led by this new method.\n- Really appreciate the human study which is really important to have when comparing quality of generation across different models.\n\nWeakness\n- Lack of ablations over the prompt. Would be curious to know how the prompt influences the results.\n- It would have been better to have some ablations over the VLM choice.\n- The method relies on a lot of different pieces, a VLM, a vision encoder, a diffusion based decoder and the universal concept decoder that is introduced by this work. The interplay between all of those elements is not really studied in depth.\n- The dataset that has been chosen for evaluation has limited diversity. For example, in the CelebA dataset there is a limited number of axis concepts. It is not clear how much this method could be scaled to the entire natural image realm.", "was_revised": true, "final_justification": "After reading the other reviews and the author rebuttal, I am satisfied with the discussion that has taken place. This is an interesting work that will be of interest to the community.", "rebuttal": "We appreciate the valuable comments. Below we respond to individual concerns,\n\n> Q1. Ablations over the prompt\n\nWe find that the design of prompt mainly affects the granularity of the axes. For example, in CelebA-HQ dataset, simple task description without providing examples would lead to finding relatively coarse concepts such as hair and color instead of more fine-grained attributes such as hair-color, hair-length, eye color. As discovered axes are significantly different depending on the prompt, we cannot directly compare them in quantitative measures. We agree with the reviewer that investigation on prompt design might improve overall performance of our work and we leave it for future work. \n\n> Q2. Ablations on VLM choices \n\nFollowing the reviewer’s suggestion, we conduct two studies on the choice of VLM:  \n\n\n(1) We conduct ablation study on different choices of VLMs. We consider two popular opensourced VLMs (Qwen2.5-VL and Ovfi2) that have shown high ranks in the leaderboard on reasoning benchmarks. (2) We also investigate the effect of partial observation of concept axes. While (2) is not a direct ablation study on VLMs, it can provide a controlled evaluation between the performance of VLM’s performance and representation learning quality of our framework. Since it is difficult to directly control and quantify the performance of VLMs, we instead control the quality of the output from VLMs by explicitly dropping partial axes (e.g., 10%) among the axes given by the VLMs. It is a practical scenario as it is impossible for VLM to always capture the complete axes for given scenes. To this end, we first extract concept axes per image for entire images and only 80% or 90% of axes (i.e., 10%/20% random drop) are provided to our model in the training stage. We evaluate our method on the visual concept editing task in CelebA-HQ dataset. \n\nBelow Table reveals that our method is hugely robust to both the choice of VLMs and missing axes. We conjecture that this is because even though some image-related axes are missed in each example, those axes will be eventually, repeatedly exposed across the dataset. Moreover, since our compositional anchoring loss encourages the compositionality of the concept representations, our framework might be internally trained for better compositional generalization, which improves adaptation with relatively few samples. In fact, we observed that our method is capable of generating OOD samples (Figure 5 in our paper.). These results highlight the robustness of our framework regarding the performance of VLMs, and implies that our method can scale to more complex scenarios, since VLMs do not have to always capture complete axes for given scenes. \n\n=== [Table 1] ===\n|  | Ours \\+ InternVL2-5 | Ours \\+ Qwen2.5-VL | Ours\\+Ovis2 | Ours +InternVL2-5  \\+ 10% drop  | Ours +InternVL2-5  \\+ 20% drop  |\n|---|---|---|---|---|---|\n| CLIP Score (\\uparrow) | 23.88 | 23.72 | 23.48 | 23.52 | 23.65 |\n| BLIP Score (\\uparrow) | 49.58 | 48.64 | 48.35 | 48.69 | 48.61 |\n\n\n> Q3. Interplay between components\n\nWe conduct ablation studies on each of the components as follows: (1) Frozen T2I decoder versus LoRA-finetuned decoder, (1) Choice of vision encoder (Dinov2 versus CLIP), (3) universal concept decoder versus shared MLP architectures. \n\n\nFirst, finetuning the decoder with LoRA does not affect the overall performance. Large scale pretrained T2I models are known to be already learned expressive data priors on natural images, which helps faster training of the generation model. Thus the frozen T2I model does not bottleneck our framework. Replacing Dinov2 encoder with CLIP encoder leads to significant performance drop. This is because CLIP is only trained for text-alignment and thereby it’s discriminative property of representation is likely to be inferior to recent self-supervised learning methods (Dinov2). \n\n\nLastly, we replace our universal concept encoder with shared MLP architecture, which is naive version of axis-agnostic encoder. Specifically, visual feature is first represented as vector through mean pooling, and it is concatenated with axis embeddings followed by MLPs to encode concept representations. To make this encoder generally work for diverse concept axes, we shared this MLP layer for all of the axes. However, this modification causes severe performance drop. We conjecture that this is because shared MLP encodes each concept independently, possibly blocking complex interaction between visual concepts. It clearly highlights the effectiveness of our universal concept encoders. \n\n\n=== [Table 2] ===\n\n|  | CLIP | BLIP |\n|---|---|---|\n| Frozen T2I  + Dinov2 + UCE (Ours) | 23.88 | 49.58 |\n| LoRA-finetuned T2I + Dinov2 + UCE | 23.67 | 49.62 |\n| Frozen T2I + CLIP + UCE | 22.03 | 46.21 |\n| Frozen T2I  + Dinov2 + Shared MLP | 21.63 | 46.86 |\n\n\n\n> Q4. Experiments on diverse dataset\n\nTo validate our framework on larger, diverse, and unstructured data, we randomly sampled 20 ImageNet classes, spanning animals (e.g., tree frog, American black bear, sulphur butterfly, giant panda), everyday objects (e.g., padlock, grand piano, motor scooter), scenes (e.g., boathouse, water tower), and etc,  yielding around 28k training images (~1.4k images per class). This is a significantly more challenging scenario as it contains diverse image-specific visual concepts and unstructured images. Across this dataset, our method successfully discovers both coarse and fine-grained across different categories. Spanning from primitive concepts like color, material, shape, it also discovers fine-grained axes such as wing color/shape, body color of butterflies and sail color, boat color, sky condition for boat images. It also captures categorical attributes, such as species, vehicle type, fruit type. Although our dataset does not include human-centered images as in CelebA-HQ dataset, our framework still robustly captures human-specific concepts such as age, gender, hair color/length. \nFor quantitative assessment, we conduct visual concept editing on the 50 most frequent axes (please refer to response to Q2. of Reviewer 4ztc  for top-50 frequent axes) and report CLIP/BLIP Scores against text-conditioned editing baselines. Manually predefined-axis based approach (LICVL [1]) becomes infeasible here, since a fixed set of axes fails to cover a wide range of concept space. Even within the same category, diverse image-specific concepts would appear, for example, humans, animals and plants may appear in motor_scooter class images, making it difficult to predefine a fixed set of concept axes representing the entire images. It highlights the effectiveness of our method, as our framework can adaptively decompose the given scene into image-related axes specified by VLMs. \nBelow Table demonstrates superior performance of our method compared to text-conditioned editing baselines. As the discovered visual concepts become more detailed such as body color of butterflies or stem color of flowers, most of the text-conditioned editing baselines fail to perform faithful editing. Also, we find that our framework is capable of generating OOD samples such as green bears/pandas or a scooter floating above the sea. We will add qualitative results in our main paper. \n\n\n|  | Pix2Pix | NullText Inversion | SDEdit | DDPM Inversion | Ours |\n|---|:---:|:---:|:---:|:---:|:---:|\n| CLIP Score ($\\uparrow$) | 19.83 | 18.93 | 19.54 | 24.32. | **25.10** |\n| BLIP Score ($\\uparrow$) | 38.34 | 34.10 | 38.10 | 46.70 | **47.44** |\n\n\n> Q5. Challenges (outside compute) for scaling your method to larger and more diverse natural images dataset\n\nYes, mapping visual concepts to verbal words is an important motivation of our work. As there could be infinitely many concept axes in real-worlds, aligning with our perception and those semantic representations are crucial and effective way to describe the complex world with a feasible number of concepts. We found that rare and too fine-grained concepts such as the exact brand name of the calculator are hard to be trained, as our encoder rarely meets those concepts and hardly get signals from compositional anchoring loss. \n\n\n**Reference**\n\n[1] Lee et al., language-informed visual concept learning, in ICLR 24."}]

Author rebuttal:

Meta-review: The submission introduces a visual concept discovery framework for real-world images, which can be applied to compositional image editing and generation. The submission received two accept, one borderline accept, and one borderline reject ratings after rebuttal. Reviewer vNjW found the submission to not yet offer a fundamentally new approach guided by principles or theories, which the AC agrees. Overall, the AC believes the submission passes the acceptance bar for NeurIPS.
Decision: Accept (poster)