Language-Informed Visual Concept Learning
Authors: Sharon Lee
Yunzhi Zhang
Shangzhe Wu
Jiajun Wu
Venue: ICLR 2024 poster (2024)
Keywords: Image generation; vision language models
Primary area: generative models

Abstract: Our understanding of the visual world is centered around various concept axes, characterizing different aspects of visual entities. While different concept axes can be easily specified by language, e.g., color, the exact visual nuances along each axis often exceed the limitations of linguistic articulations, e.g., a particular style of painting. In this work, our goal is to learn a language-informed visual concept representation, by simply distilling large pre-trained vision-language models. Specifically, we train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes, with an objective of reproducing the input image through a pre-trained Text-to-Image (T2I) model. To encourage better disentanglement of different concept encoders, we anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts. With a lightweight test-time finetuning procedure, it can also generalize to novel concepts unseen at training.
Project page at https://cs.stanford.edu/~yzzhang/projects/concept-axes.

Published as a conference paper at ICLR 2024
LANGUAGE-INFORME VISUAL CONCEPT LEARNING
Sharon Lee" Stanford University
Yunzhi Zhang"
Shangzhe Wu
Jiajun Wu
ABSTRACT
Our understanding of the visual world centered around various concept axes, characterizing different aspects of visual entities. While different concept axes can be easily specified by language, e.g., color, the exact visual nuances along each axis often exceed the limitations of linguistic articulations, e.g., a particular style of painting. In this work, our goal is to learn a language-informed visual con- cept representation, by simply distilling large pre-trained vision-language models. Specifically, we train a set of concept encoders to encode the information pertinent concept axes, with an objective of reproducing the
set of language-informen input image through pre-preeaee are-trained Text-to- Text-to-Image (T2I) model. To encourage better disentanglement of different concept encoders, we anchor the concept embeddings to a set of text embeddings obtained from pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts. With a lightweight test-time finetuning procedure, it can also generalize to novel concepts unseen at training. Project page htttts:://cs.... wtw..ord....anodd.......... hwwpaaattrr...te/yyaaa///////ooo "yzzhang/pzo conceptt axes
INTRODUCTION
In order to make sense of the myriad visual entities in the world, humans develop an abstracted generative model of them and organize the underlying sources of variation into visual concepts, such as different colors or different types of objects. Designing systems that can recognize visual concepts within images as humans do has been longstanding goal in the fields computer vision and artificial intelligence (Russakovsky et al.) 2015 Krizhevsky et al. 2012. Girshick et al. 2014).
To facilitate efficient reasoning and communication of these concepts, humans created symbolic depictions that have evolved into natural language. Such natural language grounding of visual data has been instrumental in the recent proliferation of powerful large vision-language models that are capable of semantically identifying objects in images (Radford al. [221]] Kirillov et al. (022) or generating photo-realistic images from arbitrary text prompts (Ramesh et al. 2021 Rombach et al.
2022 Saharia et al. 2022 Yu et al. 2022). While different concept axes can be easily specified by words, such as category and style, it is much less intuitive to delineate the subtleties of low-level visual nuances along each axis using language, such one particular style of a painting.
In this work, our goal is to distill from large pre-trained vision-languass models a function that extracts visual concepts along a set of language-specified concept axes from images. As illustrated in Figure once these concepts are extracted, we can recompose them across different image instances inference time produce new images with novel concept combinations. To learn this function, rather than collecting a large-scale dataset of human annotations for each specific visual concept, we design a language-informen visual concept representation, and simply distill from a pre-trained Text-to- Image (T2I) generation model. There are three fundamental properties we seek in this visual concept representation.
First, unlike T2I generation, which relies on generic words visual concept descriptors, we would like to capture fine-grained visual nuances using continuous concept embeddings. One common technique is to invert the text-to-image generation process by optimizing embedding with the objective of reproducing a given input image using a pre-trained T2I model, often referred to as Textual Inversion (Gal al. 2022). However, most existing Textual Inversion methods (Gal et al.
"Equal contribution; alphahetically ordered.

---

Published as a conference paper at ICLR 2024
Input
<cotegery> <color>
scategoryo
composed
input
<colors
ccotegorys
cimaterats
<cotegory>
Training Data
Inference
Concept Recomposition
Figure 1: Language-Informed Visual Concept Learning. Our goal is learn visual concept representation grounded on a set language-inforne concept axes, e.g.,cate e.g.,categrr: tegory, color, and material, by simply distilling from pre-trained text-to-image generation models without manual annotations. After training, the concept encoders extract disentangled axis-specific embeddings from an image, which can remixed to generate new images with novel concept compositions.
photo
<calegory>
<oolor>
"a photo of <color> ccategorya"
Baseline
Ours
Vanilia text prompt
Source
Recomposition
Figure 2: Learned Disentangled Concept Embeddings Improve Compositionality. Left: Vanilla text-to-image model may fail adhere to text prompts of uncommon combinations of concepts even with prompt engineering, e.g. "red banana".. Right: With the same backbone T21 generator, our learned disentangled concept embeddings greatly enhance concept compositionality.
2022) optimize embeddings for individual image instances independently, overlooking the shared nature of visual concepts across instances. For instance, the concept of "red" is shared between "red apple" and a "red dress". Moreover, the concepts of "red" and "yellow" also are instances of the property of color.
Hence, the second desired property of the visual concept representation is to preserve such common concept structures among various visual instances. Instead of optimizing on individual image in- stances independently. we design set of concept encoders, where each encoder learns to encode the visual characteristics of an input image pertaining one concept axis specified by language. This ensures that the inverted concept embeddings can be shared across different instances and remixed to generate new images.
The third crucial aspect of this representation is to ascertain that different concept axes are disentan- gled, allowing for changes to be made specifically on single concept axis without modifying other axes. To do we reuse the disentangled nature of linguistic concepts and ground the predictions to a set of discrete text anchors in the concept embeddings space, which can be obtained by querying a pre-trained generic Visual Question Answering (VQA) model, e,g., BLIP-2 (Li et al. (023b)) This soft anchoring constraint significantly improves the disentanglement of concept embeddings across different axes while still retaining sufficient leeway to capture nuanced visual variations that BLIP-2 struggles to discern, e.g., the style of an art piece in Figure 6
Putting these ideas together, design generic framework for learning disentangled and composi- tional visual concepts grounded to linguistic structures by exploiting pre-trained text-to-image gen- eration and visual question answering models. We show that these concept encoders be trained purely on synthetic images generated by pre-trained T2I model, and extract concept embeddings from real images at time, which capture the fine-grained visual nuances.
Our contributions can be summarized as follows:
1. We propose generic framework for learning language-informed visual concepts by simply distilling pretrained liniun-language models.
At inference time, the trained concept encoders extract concept embeddings from a test image, which can be remixed to generate images with novel compositions of concepts.
Using a wiight test-time finetuning procedure, these encoders can also be quickly adapted extract novel concepts unseen during training.
2

---

Published as a conference paper at ICLR 2024
4. Experiments show that this visual concept representation achieves better disentanglement and compositionality, compared to text-based prompting basclines, as shown in Figures 2 and 6
RELATED WORK
2.1 VISUAL CONCEPT LEARNING
Designing learning-basee systems to discover various visual concepts in natural images has been a long-standing goal in machine perception and intelligence. Early attempts typically rely on exten- sive semantic annotations done by humans, such as object classification (Barnard et al. 2003 Fei- Fei et al. |2006; Fergus et al. 2005), which were later epitomized by the effort of ImageNet (Rus- sakovsky et al. 2015). Visual concepts are intrinsically linked to concepts in language, and such end-to-end supervised learning paradigms can be as learning direct mapping between visual concepts and discrete linguistic concepts. Other approaches attempt to better exploit this inherent structure in language by constructing a structured representation of visual concepts such as scene graphs (Zhong et al. 2021) and symbolic programs (Mao et al.) 2019 Han et al. 2019).
More recently, the success of natural language modeling (Devlin al. 2018 Brown et al. 2020 Raffel et al.] (020) has paved the way for grounding visual concepts to open vocabularies, unlike category labels or fixed symbolic programs, by training large Vision-Language Models (VLMs) on massive image captioning datasets (Schuhmann et al. (0022). This has powered recent Text-to-Imant Imaee (T2I) generation models to turn linguistic concepts from free-form text prompts into photo-realistic
images (Rombach et al. 2022: Saharia et al. (022). These T21 models have been leveraged by Personalization methods for extracting individual visual concepts from one or a few images. This
done by either by optimizing token embeddings (Gal et al. 2022; Vinker et al. [023 Avrahami et al.)
2023 Chefer et al., 2023b Liu et al. 2023), finetuning the backbone denoiser (Ruiz et al. 2023), or training additional encoders for amortized optimization (Gal et al. 2023 Arar et al. 2023 Li al.)
2023a). We also distill visual concepts from a pre-trained T2I model, but unlike existing works, train encoders to adhere to set of sanguage-specifeen concept axes, preserving the disentangled and compositional nature of language. |Ranasinghe & Ryoo (2023) also explores language-defined concepts but focuses video action recognition tasks while we focus on image generation.
A separate line of work focuses on unsupervised visual concept disentanglement without explicitly leveraging language, typically by simply imposing information constraints in the latent space of generative model, like VAEs and GANs (Higgins et al. [0177] Chen et al. 2016 Hsu et al. 2023 Here, we are interested in learning visual concepts that are explicitly grounded language.
2.2 CONTROLLABLE IMAGE GENERATION
The success of GAN-based image generation (Goodfellow et al., 2014 Brock et al. 2018 Karras let al., 2019) has spawned series of works that discover controllable directions in the GAN la- tent space (Voynov & Babenko) 2020 Härkönen et al.] 2020). More recently, the advancements of diffusion-based T2I models have unlocked new possibilities for controllable image generation, where photo-realistic images can be generated from free-form text prompts. Recent works pro- posed to further improve the alignment of image samples and input text conditions by manipulating attention maps within T2I models (Chefer et al.] 2023a) Epstein et al. (023). Another form of controllable image generation compositional generation. Liu et al. (2022) proposes to improve the quality of T21 diffusion models for composing multiple pre-given concepts, specified via text prompts, by modifying the inference procedure. In this work, instead of assuming that concepts are given and are in text format, we tackle the task of identifying disentangled concepts which can be used for composition.
Image generation can also be controlled with image analogies (šubrtová et al. 2023 Hertzmann et al., (00T), a form of visual prompting. These works typically do not explicitly extracts visual concepts from inputs unlike ours. In this work, we amalgamate both visual prompts and text queries, employing them the editing interface.
3 METHOD
Fig. gives an overview of our proposed learning framework. Our goal in this work is to ex- tract visual concepts from images along number of concept axes specified by language, such as
3

---

Published as a conference paper at ICLR 2024
Concept Encoders
11>
T21
Category Encoder f
Color Encoder f
Material Encoder fa
(category)
e > (color)
a
111
in
e color &
material
e3 > (material)
T21
Input Image
Lanchor
panchor
canchor
Pretrained Embedding Space
Text-to- Image (T21)
A: "Chair"
A: "Yellow"
A: "Wood"
"What object?"
"What color?" Q: "What material?"
Visual Question Answering (VQA)
grecon
Reproduced
Image
During training, an input image is processed by a set of concept encoders that predict concept embeddings specific to given concept axes. These embeddings trained to (1) retain information in order to reproduce visual inputs via pre-trained Text-to-lmage model given an axis-informed text template, and (2) ensure disentanglement across different axes by anchoring to text embeddings obtained from a pre-trained Visual Question Answering model.
Figure 3: Training Pipeline.
category, color, and material, so as to enable the flexible composition of concepts into high-quality image generations.
To achieve this, we train a set of visual concept encoders by distilling concept guidance from pre- trained vision-language models. Specifically, the encoders trained to extract concept embeddings from an image in order to fulfill two objectives. First, they should be recomposed to explain the input image through pretrained text-to-image (T2I) generation model, given text prompt. Second, these visual concept embeddings should be anchored to the corresponding text embeddings obtained from a pre-trained visual question answering (VQA) model, further exploiting the disentangled nature of linguistic concepts for better disentanglement of visual concepts.
sencceptaxxi-inforree
3.1 VISUAL. CONCEPT ENCODING BY INVERTING TEXT-TO-IMAN GENERATION
Our understanding the visual world is centered around various concept axes, to which we have of- ten assigned words due their significance in communication and reasoning. This grounding has fueled recent explosion of text-to-inn text-to-image generation models (Rombach et al. [022] Saharia al. 2022 Ramesh al., 2022). allowing them generate photo-realistic images with various combinations of concepts defined by words.
Here, interested in the reverse direction of text-to-image generation, where the goal is to extract language-grounded visual concepts present in natural images. Specifically, given K concept of interest defined by language, we would like learn K concept encoders (ʃ& (-)) each of which extracts a concept representation e₃ = fi (x) along a concept axis from an input image x.
In order train these concept encoders {fx(-)), instead of relying on extensive human labeling, we opt to exploit the vision-language grounding embedded within large pre-trained T2I generation models. Using the technique of Textual Inversion (Gal et al. 2022). one can optimize token embedding <*> to capture visual entity in given image, through the objective of regenerating the image with the T2I model from a text template, such as photo of <+>". Here, we adopt similar objective, but instead of inverting specific embedding capturing the overall "identity" of an individual image instance, we would like to predict embeddings eş that are grounded to a number of meaningful concept axes, using axis-informed text template, such "a photo of <ej with <e>> color and <e>> material". This allows the extracted concept embeddings to be shared across different images, encapsulating the common visual characteristics pertinent
concept axis.
Specifically, given an image x, the concept encoders {f)(-)) extract a set concept embeddings (er "). which have the same dimension D the text embeddings so that they be directly inserted into the text embeddings of the axis-informed text template. To simplify the notations, let (.).) denote the function that takes in the image and produces the final sequence of embeddings

---

Published as a conference paper at ICLR 2024
of the template and the predicted concept embeddings, and "y be the parameters of all the encoders which will be optimized during training. Let Cy be the part of the T2I model's text encoder that takes in sequence of text embeddings and outputs conditioning vector for the T2I model's denoising network ég, where denotes network parameters. We use DeepFloyd (Stability /aharia et al., 2022) as the backbone T2I model, which utilizes a pre-trained '15 model (Raffel et al. (0200) the text encoder, and keep the parameters 0 frozen in all experiments. To train the encoders, we reuse the training objective for the backbone diffusion model:
Creeon (x:7)= (e--N(00.,,,--((((11)) [1l,x(x,4, co =
(1)
where the noise E is sampled from a standardmultivariaate Gaussian distribution and the timestep t is sampled from a uniform distribution in [0, 1]. Minimizing L amounts to finding concept embeddings within the space of the pre-trained T2I model that can best reproduce the input image x, resembling a "reconstrucion" objective.
Compared to per-instance token optimization in vanilla Textual Inversion, the advantages of training these concept encoders are two-fold. First, the concept embedding space is naturally shared across different image instances, encapsulating the common understanding of the corresponding concept axes. Second, makes training more efficient by amortizing the optimization across all instances, and more crucially, it allows for test-time inference in a feed-forward pass.
3.2 CONCEPT DISENTANGLEMENT USING TEXT ANCHORS
The objective of Creem ensures that the extracted concept embeddings can sufficiently reconstruct the concept of given image through pre-trained text-to-image generation model. However, with this loss alone, there is little guarantee that each embedding encodes only the information pertinent to a particular concept axis. In practice, we found that this baseline results poor disentanglement of different concept axes when remixing the concept embeddings to generate new images, potentially due to the imprecise vision-language grounding in the pre-trained T2I model. For instance, as shown in Figure 88] the extracted category embedding <e₁> > for "red berries" cannot be remixed with various co or embeddings <e2> egg., "orange", as <e,> is highly entangled with the concept of a "red" color due to the bias in natural images.
To encourage better disentanglement of different concept axes, we further incorporate a sparse set of text anchors into the concept embedding space. Along each concept axis like color, we have often named some prominent modes, such as "red" or "yellow", and these text labels entail clearly disentangled concepts. Therefore, we would like to reuse this disentangled nature of linguistic concepts to improve the disentanglement of visual concepts. To this end, we make use of the text predictions from pre-trale pre-trained Visual Question Answering (VQA) model, BLIP-2 (Li et al. 2023b), as pseudo ground-truth anchors for the concept embeddings.
Specifically, for each training image and for each concept axis of interest (e.g., color) indexed by h; we query the BLIP-2 model with the image x and a question Ok in natural language that specific to this concept axis, e.g., "what is the color of the object in the image". Denote the answer from BLIP-2, also in the form of natural language, as Vx(x, qk). We encode this answer with the pre-trained text encoder co obtain a text embedding é₁ co (x).). The prediction of our concept encoders fu.y is encouraged to stay close to this anchor text embedding:
C Canchor (x; 7) fl (x) 12.
where ē cx
is crucial to highlight that we use these BLIP-2 predictions only as anchors by assigning a small weight to this anchor loss fanchor during training. Otherwise, the embeddings predicted by the concept encoders could easily collapse to a set of discrete text embeddings and fail to capture the visual nuances in images.
3.3 TRAINING AND INFERENCE
Training. Given collection of training images D containing various combinations of concepts along each axis, the final objective to train the concept encoders consists of the two parts:
K
Cotal (7) = ExnD rreoon (x;
+
Canchor (x: 7))
(3)
5

---

Published as a conference paper at ICLR 2024
Inference. At inference time, given test image, the concept encoders extract embeddings {ex} capturing its characteristics along each concept axis of interest. These embeddings can be remixed across different images, be replaced by embeddings converted from explicit words, to produce images with new compositions visual concepts through the backbone T2I generator.
Generalization to Unseen Concepts via Test-Time Finetuning. While the encoders can pre- cisely extract an axis-specific concept that has been seen during training from a new test image, they tend to be less robust to concepts unseen at training. However, with lightweight test-time optimiza- tion procedure, where we use only the reconstruction objective f to update the parameters for all encoders, "/ these encoders can generalize to novel concepts unseen during training. Note that ganchor is omitted here in order to capture the visual nuances without over-commitiig to the coarse text anchors. After training, the encoders have learned to generate outputs within a relatively narrow region of the embedding space, which allows the model to adapt to the test images shown in Figure(3) within around 600 iterations while maintaining disentanglement and compositional capability.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Training Data Generation. We train the concept encoders only using synthetic images generated by DeepFloyd from 5 different domains, including fruits, figurines, furniture, art, and clothing- More details of our dataset can be found in A.22, For each dataset, we consider 2-3 concept axes, such as category, color, material, style, and season. For example, considering category ry and colox or for the fruits dataset, we generate training images by prompting DeepFloyd with text prompts describing varying combinations of categories and colors, e.g. "a photo of an apple which is red in color". Note that these text prompts are used only for data generation and not for training, as they may not be reliable (Figure 2 On average, we obtain 669 training images for each dataset.
Implementation Details. Inspired by Gal et al. (2023), J we leverage a pre-trained CLIP VITAL- 14 model for image encoding (Radford et al. [2021; Dosovitskiy et al. 2021), which was trained with a contrastive objective aligning image and text features, and hence well-suited for our task. We extract image features from CLIP ViT and train K separate concept encoders fa on top of the features, which share the same architecture but maintains separate weights. Specifically, we take the [CLS] tokens from each CLIP layer and process each token with distinct linear layer. This is different from Gal et al. (2023), which extracts [CLS] ] tokens from every other layer and uses a single shared linear layer for all token features. The transformed features then aggregated with average pooling followed by a LeakyReL.U (Xu et al.) (013)) activation, and passed into another linear layer that produces the final predicted concept embeddings.
To ground the concept embeddings to the concept axes, we adapt the text templates from CLIP (Rad- Ford et al. 2021), which were originally used to assemble captions with class categories from Im- ageNet (Russakovsky et al. 2015). For training, we use Adam W (Loshchilov & Hutter 2017)
optimizer with learning rate 0.02, and randomly flip the images horizontally. For test-time finetun- ing, we use the Adam optimizer with learning rate 0.001. We set X 0.0001 (Equation (3) for the category axis and = 0.001 for others. We use IF-I-XI from DeepFloyd as the backbone model. with training resolution 64 x 64. Training on one dataset takes approximately 12 hours on one NVIDIA GeForce RTX 3090 GPU. Generated images are upsampled 256 x 256 using IF-II-L for visualization purpose.
4.2 QUALITATIVE RESULTS
Visual Concept Extraction, Recomposition and Extrapolation. Once trained, the concept en- coders can extract disentangled concept embeddings specific to each concept axis from different test images, which recomposed to generate new images with various concept compositions. As shown in Figure 4 across various datasets, our method is able to recompose axis-specific visual concepts from different images and consistently generate new images depicting the recomposed concepts. More examples can be found in Appendix A.I where images generated from individual decomposed concept embeddings also presented.
This disentangled concept representation also allows us to extrapolate along a particular concept axis for visual exploration, as shown in Figure 7 For instance, we can ask BIIP-2 "what is the style

---

Published as a conference paper at ICLR 2024
Source <category>
Source <category>
Source <color>
Source <style>
(a) Fruits
(b) Paintings
Source <category>
Source <category>
Source <material>
Source <material>
Source <color>
Source <color>
(c) Furniture
Figure 4: Concept Recomposition. At test time, model extracts visual concepts along various axes from different images and recompose them to generate new images. We show recomposition results across different pairs of concept axes in 3 datasets: (a) Fruits, (b) Paintings, and (c) Furniture.
Train <cotegory>
<cotegory's
Train Images
<cntegoryo
<category's
Test Image
<category'>>
<cotegory/> ccolors
Figure 5: Generalization to Unseen Concepts via Finetuning. After test-time fine-tuning on single test-time image, encoders can adapt to novel concept. Visual details from the input images are preserved can be seen from images visualizing embedding predictions. Importantly, these embeddings do not overfit to the input images and maintain a good disentanglement, such that they be freely recomposed to create
concepts. More results can be found in Figures 9 to 13
More real-world results can be found in Figures 20 to 24
of the painting?" in an image, and prompt GPT-4 (OpenAl 2023 to name a few alternatives. We can then recompose the text embeddings of these alternative styles with our concept embeddings and generate images to visualize these variants. Representing concepts continuous embeddings further enables concept interpolations. Details and results are shown Appendix A.6
Generalization Unseen Concepts via Test-Time Finetuning. Although the encoders have only seen limited range of concepts during training due to the small size of the training dataset, it can be quickly adapted to unseen concepts with the lightweight test-time finetuning procedure in Section 333 shown in Figure For instance, after 600 finetuning iterations, the model can extract the specific style of the dog painting unseen at training and compose it with the content from other images. It can also capture nuanced colors e.g. yellow-ish-orange and transfer them to other objects.

---

Published as a conference paper at ICLR 2024
Source Image
<color>
<colors Barana
<style> "apple""
Change to
"pinspppe"" strawberrj"
"barana"
"pppls"
"tree"
"Fres"
Null-text Inversion
Instruct- Pix2Pix
Baseline w/ BLIP-2
Ours
Figure 6: Baselines Comparisons. Compared the text-based image editing methods, our method achieves significantly better compositionality due to the disentangled concept representation, and captures fine-grained color variations, which the baseline struggles to encode with language.
CLIP-Score
Human Evaluation 1
Edit Category
Edit Color
Edit Cat.
Edit Clr.
&
Cat.
Clr.
Clr.&CIr.
Cat.
Clr.
Score
Score
Null-text Inversion
0.258
0.249
0.249
0.259
0.265
0.223
0.287
0.316
InstructPix2Pix
0.267
0.277
0.226
0.270
0.245
0.268
0.233
0.648
Baseline w/ BI.IP-2
0.313
0.294
0.248
0.287
0.271
0.237
0.448
0.379
Ours
0.308
0.297
0,238
0.302
0.287
0.236
0.968
0,840
w/o CE
0.268
0.276
0.219
0.263
0.257
0.236
w/o Encoder & Caccher
0.288
0.300
0.214
0.242
0.213
0.265
Table 1: Quantitative Comparisons Visual Concept Editing. Compared existing image
editing baselines (Brooks et al.; 2023 Mokady et al. 2022). our method achieves better overall CLIP score when editing either axis, and is particularly effective retaining category-related concepts as reflected in human evaluation. "Cat' denotes Category and 'Clr' denotes Color.
4.3 COMPARISON WITH PRIOR WORKS
Qualitative Comparisons While this task of image generation with disentangled visual concepts new, we identified prior work that capable of text-based image editing and generation, and establish side-by byde-by-side comparison image x, for example, of teal-colored apple in Figure the task is to generate a new image ReL-ee; with concept axis & (e.g. category) modified by text prompt, from <eg> <et?, while preserving other axes {ili 7 k} from the input.
the task of visual concept editing. Specifically, given an
We compare to two existing methods. Null-text Inversion (Mokady et al., 2022) performs concept editing by first inverting the input image to token embeddings of Stable- Diffusion (Saharia al., 2022] and then applying Prompt-to-Promp (Hertz et al. (022). which modifies cross attention maps for editing, a process that leads to pixel-alignee editing results. InstructPix2Pix (Brooks al. 2023) is a conditional diffusion model for based image editing. Since is trained on data curated with Prompt-to-Prome for supervision, also tends to generate pixel-aligned results. We also design naive baseline for this task, where we query BLIP-2 for text description of the attributes to be retained from the test image, and combine the answer with target concept to generate the final recomposed image. As shown in Figure 6 our method achieves better recomposition results, whereas baseline methods fail to disentangle the desired axis-specific concepts from input images, and therefore struggle to faithfully reproduce the desired concepts from the image and the text.
Quantitative Comparisons. We also conduct quantitative evaluations on the task of text-based visual concept editing and compare with prior work. Specifically, record the ground-truth text
8

---

Published as a conference paper at ICLR 2024
Source Image
Input
Extrapolations
<category>
charceall
"Aoryli""
"erreoollor
ccategory
<categary>
<calors "raped"
<colors
Change to
biverr
"orange"
Ours
<category>
"reuf"
"parple"
"Blue"
w/o 4
ganchor
w/o Encoder ganchor
<category>
"spring" "
"autsmun" "
& w/o LK
Baseline w/ BLIP-2
Figure 7: Visual Concept Extrapolation. Given input image, along a concept axis by querying BLIP-2 and GPT-4 to name few alternatives to the concept in the input. Our model can then generate new variants of the input for visual exploration, by mixing them with the extracted concept embeddings.
extrapolate
Figure 8: Ablations. Both the text anchor loss and the encoder design reduce overfitting to in- put images and improve concept disentanglement. Removing these components significantly deterio- rates the visual concept editing results. Addition- ally, we observe that the BLIP-2 baseline has diffi- culties transferring unnatural colors.
prompts y that we used to generate each training image and manually filter out the inconsistent pairs. For each concept axis we randomly select target label <e from a set of candidates that different from the one in the original prompt <e&> We then measure the alignment score between the modified image X +el with the modified text prompt ye, using CLIP (Radford et al. 2021), following Gal et al. (2022). We also break down the score to each individual axis, by computing the alignment of the image to a prompt specifying only one axis, e.g. "a photo <<,>"" All results summarized in T Our method captures the characteristics of category particularly well and outperforms others in changing category or preserving category while changing color, which highlights its ability extract disentangled concepts allowing for flexible compositions. Instruct- Pix2Pix is more effective in changing color, but tends to do poorly in preserving the category, as seen in Figure 6 More details our quantitative results can be found in A.4
We further conduct a human evaluation. Each participant is presented the results from all three methods in a random order together with the editing instruction, and asked rank them considering both realism and faithfulness to the instruction (see Appendix A.3 for details). We aggregate the responses from 20 people and report the average average score normalized 0-1 Table More details of our setup can be found in A.3
4.4 ABLATIONS
We conduct ablation study to understand the effects of the proposed encoder (as opposed per- instance optimization) and the text anchoring loss Lanchor We use the same evaluation dataset and the CLIP- clig-alignment metric described in Section 4.3 As shown in Fig. 8 and Table removing C Canchor and the encoders deteriorates disentanglement of the different concept axes due to severe overfitting, resulting in poor recomposition results. More results are included in the Appendix.
5 CONCLUSION
In this paper, we have presented framework for learning language-informed visual concepts from images, by simply distilling from pre-trained vision-language models. After training, the concept en- coders extract disentangled concept embeddings along various concept axes specified by language, which be remixed edited to generate images with novel concept compositions. We conduct thorough evaluations both quantitatively and qualitatively, and demonstrate that our approach yields superior results in visual concept editing compared to prior work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Kyle Hsu, Joy Hsu, and Stephen Tian for their detailed comments and feedback. This work in part supported by NSF RI #221 1258 and #2338203, ONR MURI N00014-22-1-2740, ONR N00014-23-1-23555 AFOSR YIP FA9550-23-1-011 the Stanford Institute for Human-Centers AI (HAD, Amazon, J.P. Morgan, and Samsung.
REFERENCES
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano. Domain Domain-agnostic tuning-encoder for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06925, 2023.
Omri Avrahami, Kfrr Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from single image. SIGGRAPHI Asia, 2023.
Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M. Blei, and Michael I. Jordan. Matching words and pictures. JMLR, 2003.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096. 2018.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023a.
Hila Chefer, Oran Lang, Mor Geva, Volodymyr Polosukhin, Assaf Shocher, Michal Irani, In- bar Mosseri, and Lior Wolf. The hidden language of diffusion models. arXiv preprint arXiv:2306.00966 00966, 2023b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. NewrlPS, 29, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. NeurlPS, 2023.
Li Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE TPAMI, 2006.
R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning object categories from google's 's image search. In ICCV, 2005.
Bruce Fraser, Fred Bunting, and Chris Murphy. Real world color management, 2004.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
10

---

Published as a conference paper at ICLR 2024
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or Encoder-based domain tuning for fast personalization of text-to-image models. ACM Trans. Graph. 2023.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for aceu- rate object detection and semantic segmentation. In CVPR, 2014.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. Visual concept-metaconeee learning. In NeurlPS, 2019.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. NeurIPS, 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 01666, 2022.
Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David Salesin. Image analogies. In SIGGRAPII, 2001.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Kyle Hsu, Will Dorrell, James C. R. Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. NeurIPS, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan Yen Lo, et al. Segment anything. arXiv preprint wt.Xii:2334.0266444 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing, arXiv preprint arXiv:2305, 14t.20005...20,, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
Nan Liu, Yilun Du, Shuang Li, Joshua B Tenenbaum, and Antonio Torralba. Unsupervised compo- sitional concepts discovery with text-to-image generative models. ICCV, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17105 2017.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.0974. 099994, 2022.
OpenAI. Gpt-4 technical report, 2023.

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061125, 2022.
Kanchana Ranasinghe and Michael Ryoo. Language-based action concept spaces improve video self-supervised learning. NeurlPS, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. I/CV, 2015.
Donald G Saari. Selecting a voting method: the case for the borda count. Constitutional Political Economy, 34(3):357-366, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurlPS, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurlPS, 2022.
StabilityAI, DeepFloyd
floy/IFF
Adéla Šubrtová, Michal Lukáč, Jan Čech David Futschik, Eli Shechtman, and Daniel Sýkora. Dif- fusion image analogies. In SIGGRAPH, pp. 1-10, 2023.
Yael Vinker, Andrey Voynov, Daniel Cohen-Or, and Ariel Shamir. Concept decomposition for visual exploration and inspiration. arXiv preprint arXiv:2305. 18203, 2023.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In /CML, 2020.
Bing Xu, Naiyan Wang. Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.0083:
2015.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10200660070 10789, 2022.
Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate scene graph from natural language supervision. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 MORE QUALITATIVE RESULTS
More qualitative results for concept extraction and recomposition are shown in Figures to Across various datasets, our method achieves superior disentanglement and recomposition results.
13
<category>
<style>
composed
<category>
<style>
composed
wertn
<style>
<category>
<style>
<category>
Figure 9: Visual Concept Extraction and Recomposition Results on Art dataset.
<category>
<color>
composed
<category>
<color>
composed
<color>
<category>
<color>
<category>
<category>
<color>
composed
<category>
<color>
composed
<color>
<category>
<color>
<category>
<category>
<color>
composed
<category>
<color>
composed
<color>
<category>
<color>
<category>
Figure 10: Visual Concept Extraction and Recomposition Results on Fruits dataset.
A.2 DATASETS
We use training datasets for the experiments. The concept axes interest, together with ground truth words used in the text prompt for DeepFloyd to generate the training data, are listed below.
Fruit Dataset.
Category: Cherry, Apple, Banana, Mango, Strawberry, Pineapple, Lemon, Raspberry
Color: Red, Blue, Green, Purple, Black, Yellow, Orange
Art Dataset.
Category: Beach, Tree, Apple, Cat, Dog, Face, City, Hill, Sky, Car, Bike, House, Castle, Chair
Style: Charcoal, Oil, Paint, Acrylic, Crayon, Pastel
Figurine Dataset.
Category: Ball, Block. Statue
Color: Red, Blue, Green, Yellow, White, Cream, Purple, Pink
Material: Plastic, Fur, Wood, Metal, Leather
13

---

Published as a conference paper at ICLR 2024
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category>
<material>
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
<color>
<category>
<material>
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
Figure 11: Visual Concept Extraction and Recomposition Results on Figurine dataset.
Clothing Dataset.
Shirt, Pants, Shoes, Dress, Cap Color: Red, Yellow, Green, Purple, White, Cream
Category:
Season: Spring, Summer, Fall, Winter
Furniture Dataset.
Category: Chair, Table, Bench
14

---

Published as a conference paper at ICLR 2024
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color
<category>
<material>
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
+
<color>
<category>
<material>
+
+
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
Figure 12: Visual Concept Extraction and Recomposition Results on Objects dataset.
Color: Red, Orange, Yellow, Green, Blue, Purple, Black, White Material: Wood, Metal, Plastic
A.3 HUMAN EVALUATION
In the user study, we create a questionnaire where users are asked to choose the image that matches our test setting: given an image prompt and instruction, users rank images that best represents both the image prompt and text instruction. Here, we randomly sample 12 data points from the evaluation dataset for color editing and for category editing. We collected the questionnaire responses from 20 users. Our score follows the Borda point metric (Saari 2023) where 1st, 2nd, and 3rd would receive 2, 1, and point(s) respectively, allowing us to differentiate rankings of 3, 2, and 2, 1, 1. We then calculate the average scores over both the number of questions and participants, and subsequently normalize these values to 0 scale. Results are shown in
A set of instructions Figure 14 are presented to the respondents before the study. Then, they are presented with the 24 questions color and category editing tasks. Example questions are shown in Figures 15 and 16
15

---

Published as a conference paper at ICLR 2024
<category>
<color>
<season>
composed
<season>
<color>
<category>
<color>
<category>
<season>
<category>
<color>
<season>
composed
<season>
<color>
<category>
<color>
<category>
<season>
Figure 13: Visual Concept Extraction and Recomposition Results on Clothing dataset.
this survey, you will be presented with pairs images alongside a specific caption that describes what the images should depict. task evaluate and select the image that most closely adheres to the following criteria:
Fidelity: The image should appear realistic, with convincing colors, textures, and details.
Caption Adherence: The image should accurately represent the content described in the caption. For exomple, if the caption states "a red apple," the fruit should both "red" and represent the shape/ texture an "apple". Your ranking should reflect the specific shade of the sample image's color, but not the specific instance of the sample image's fruit. is the best, while 3 the worst.
none of the answers seem accurate, answer with your best guess.
Additional Guidelines: Take your time to inspect each image carefully before making your selection. Images can be tied e.g. we can have an order of
1,1, 2
2,
2,
Figure 14: Questionnaire Instructions.
A.4 QUANTITATIVE ANALYSIS.
Figure 17
analyzes the mean and standard deviation of CLIP scores introduced in Section 4.3 Our
method achieves the best score with a low variance for both the category (left) and the overall (right) metric, and stays competitive the color (metric).
16

---

Published as a conference paper at ICLR 2024
What best depicts (the type fruit)
which black in color?
(3est)
(werst)
O
O
Figure 15: Questionnaire Attribute Modification.
What best depicts berry with the (color} in
1 (Sest)
(worst)
Figure 16: Questionnaire Category Modification.
A.5 DATASET LABELLING WITH BLIP
In approach, employ BLIP in order to annotate each image with its respective category and attributes. To achieve this, we query BLIP for each image, with one question for each attribute or category. For example, for image of red wooden table, we would ask BLIP what is the name of the item, what material it is made of, and what color it is.
During the pre- pre-training stage, items corresponding identical prompts and therefore the same place- holder tokens are aggregated by taking the most commonly occurring BLIP answer for each category or attribute. For example, if we have images red wooden tables, and one of them is misclassified as red plastic table, we would still label it with 'red', 'wood', and 'table" for color, material, and category respectively.
A.6 INTERPOLATION OF CONCEPT EMBEDDINGS
We further demonstrate concept interpolation results. By interpolating between two concept em- beddings, our model can generate meaningful images depicting gradual changes from one concept to another, such the hybrid fruit of cherries and bananas shown in Figure [8] To interpolate between two input images, first, we extract CLIP features for both images, giving us two vectors of size 12 1024. We interpolate the vectors using Spherical interpolation (SLERP). Specif-
17

---

Published as a conference paper at ICLR 2024
Instruct-Pix2Fix
Prompt-to-Prompt
Encoder
CLI Scores Using Category Metric
CLIP Scores Using Color Metric
CLIP Scores Using Category and Color Metric
0.34
0.34
0.34
Modifying Color
Modifying Color
Moditying Color
0.30
0.30
0.30
0.26
0.26
0.26
0.18
0.22
0.26
0.30
0.34
0.18
0.22
0.26
0.30
0.34
0.10
0.22
0.26
0.30
0.34
Modifying Category
Modifying Category
Modifying Category
Figure 17: Quantitative Baselines.
ically, given two normalized vectors Anorm and Buorm of dimensions 12 1024, we compute the dot product to find the cosine the angle between them as cos(0) Anorm Bnorm- For 12 interpolation points, each point is calculated using Og = 1I and the interpolated vector slerp(A, B, a;) sin((1-a.10) Anorm sin(a,0) sin(9)
sin(9) Bnorm- This ensures the resultant vectors maintain
constant magnitude.
With a set of trained encoders, each encoder takes in the interpolated CLIP features, and their outputs are combined with a training time textual template to generate the interpolation results shown in Figures and I9
Input
Input
Images
Images Interpolation
Interpolation
Points
Points
<aategory>
<cotegory>
<color>
+ <color>
<ateegor>>
<ategorr>
<color>
<color>
Figure 18: Interpolation on Fruit dataset.
Input
Input Images Interpolation Points
Images
Interpolation
Points
<subject>
<subject> +
<style>
<style>
<subject>
<subject>
<style>
<style>
Figure 19: Interpolation on Art dataset.
18

---

Published as a conference paper at ICLR 2024
B INFERENCE ON REAL-WORLD IMAGES
Despite being trained images generated by diffusion-based models, the concept encoders gen- eralize well to diverse, complex real-world images, including unseen types of object cat category, material, and color from casual photo captures (Figures 20 23) and unseen types of artwork style (Figures 5 and 24).
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
<category>
<color>
<material>
composed
<material>
<color>
<category>
+
<color>
<category> <material> <
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
+
+
<color>
<category> <material>
+
+
Figure 20: Visual Concept Extraction and Recomposition Results on Real-world Images of various objects.
19

---

Published as a conference paper at ICLR 2024
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
+
+
<color>
<category>
<material>
+
<category>
<color>
<material>
composed
<material>
<color>
<category>
+
+
<color>
<category> <material>
+
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
+
+
Figure 21: Additional Visual Concept Extraction and Recomposition Results on Real-world Images of various objects.
20

---

Published as a conference paper at ICLR 2024
<category>
<color>
composed
+
<color>
<category>
+
<category>
<color>
composed
<color>
<category>
+
<category>
<color>
composed
<color>
<category>
+
<category>
<color>
composed
+
<color>
<category>
+
<category>
<color>
composed
+
<color>
<category>
+
Figure 22: Visual Concept Extraction and Recomposition Results on Real-worf soal-wolld Images of various fruits.
21

---

Published as a conference paper at ICLR 2024
<category>
<color>
<material>
composed
<material>
<color>
<category>
A
<color>
<category>
<material>
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category> <material>
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category>
<material>
+
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
<color>
<category> <material>
+
+
Figure 23: Visual Concept Extraction and Recomposition Results on Real-world Images of various pieces of furniture.
22

---

Published as a conference paper at ICLR 2024
<category>
<style>
composed
<style>
<category>
+
<category>
<style>
composed
<style>
<category>
+
<category>
<style>
composed
<style>
<category>
+
<category>
<style>
composed
<style>
<category>
+
Figure 24: Visual Concept Extraction and Recomposition Results deal-world Images of artwork.
23

---

Published as a conference paper at ICLR 2024
C
THE EFFECT OF THE ANCHOR LOss
During training time, the anchor loss (Equation (2)) encourages the encoder predictions to converge meaningful subspace within the word embedding space Gal et al. (2023). This ensures that these embeddings can be readily visualized by pre-trained text-to-image generation model, and improves the compositionality across different concept axes, as shown in Figure
Empirically, we find that simply setting small weight on this loss can effectively achieve this objective, allowing the model capture nuances along each axis, without collapsing to the word embeddings. In Figure 25 we empirically show such examples, where we compare the concept embeddings predicted by color encoder to the text embedding
training-time BLIP-2 label, e.g. "blue" from Figure 25 and the former preserves the specific color of the input image while the latter does not.
"blue"
Input
<color>
"white"
Input
<color>
T21
Our Concept Encoder
T2I
Our Concept Encoder
"yellow" "
Input
<color>
"purple"
Input
<color>
T21
Our Concept Encoder
T21
Our Concept Encoder
Figure 25: The concept embeddings extracted by our concept encoders capture various shades of colors instead of generic 'blue', 'white' directly generated by the T2I model DeepFloyd.
24

---

Published as a conference paper at ICLR 2024
D EXTENSION WITH NEW CONCEPT AXES
We show that we are able to extend to additional concept axes by training new concept encoders without retraining the existing ones. the experiments shown in Figures [26 and [ given two trained encoders for category and color or and the corresponding training dataset, we train the material encoder while keeping the other two frozen. We show that such procedural training maintains the disentanglement of the frozen encoders while allowing the framework to extend to new concept axis.
Frozen
Added 'material' axis
Source <category>
Source <category>
Source <material>
Source <material>
Source <color
Source <oolon
Figure 26: Concept Recomposition after adding a new material axis to the model trained with category and color axes.
25

---

Published as a conference paper at ICLR 2024
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category><materiil
+
+
<category>
<color>
<material>
composed
+
<material>
<color>
<category>
+
<color>
<category><material
+
<category>
<color>
<material>
composed
<material>
<color>
<category>
<color>
<category><materiil
<category>
<color>
<material>
composed
<material>
<color>
<category>
I
<color
<aategory><mattriill
Figure 27: Test-Time Generalization Results on images of furniture, where the material encoder
is additionally trained on top of the frozen category and co for encoders.
26

---

Published as a conference paper at ICLR 2024
E ADDITIONAL BASELINE COMPARISONS
In Figure 28 we provide additional qualitative examples accompanying the experiments in Table | From these visual examples, we observed that the color nuances captured by ours are more accurate compared to the BLIP-2 baseline. However, since the CLIP-based metric specified in Section 4.3 cannot fully capture the minute differences, the BLIP-2 baseline still achieves comparable scores to method despite this evident gap in visual results.
To quantify such visual differences in colors, we compare the color 16 x 16 centered patch from the input image and the image generated by the method being evaluated, both of resolution 64 x 64, and report the L2 error of the mean color of the two patches. We report the average of metric across all examples in Figure 28 in Table 2 Results suggest that method captures colors more accurately compared to the baselines.
Source Image
<category>
<cotegory> "uxite"
<color>
<category>
<category>
<color> + "berry"
+
Change
"rangee
"peack"
"melon
"oldd"
melon
silver"
Ours
w/o 4
ganchor
w/o Encoder w/o L panchor
Baseline w/ BLIP-2
Figure 28: More qualitative results for ablations.
CIELAB AB +
Cherry
Mango
Strawberry
Apple
Ours w/o Canchor 101.34
35.50
4.76
16.12
15.64
86.34
85.31
122.01
w/o Encoder & Canchor
85.86
82.46
82.07
127.80
Baseline BLIP-2
79.90
25.20
47.30
73.62
Table 2: Quantitative Comparisons on Color Editing. To quantify the differences in color seen
in Figure 28 we use CIELAB AE*, the color-distance metrie recommended by the International Commision on Illumination (Fraser et al. 2004).
27

---

Published as a conference paper at ICLR 2024
F DISCUSSION ON TEXTUAL INVERSION
As discussed in Section 3.1, compared to directly using text as inputs for image generation, us- ing techniques like Textual Inversion (Gal et al., 2022), our model is able capture more nuanced visual details of particular image with continuous embeddings, instead of discrete words. This can be illustrated in the empirical results in Figures 6, and 28 which show that our method pre- serves the nuances from input images more accurately than the BLIP-2 baseline which uses texts for conditional generation.
F.1 LIMITATIONS
Given a few concept axes specified by language, the current model is able to learn disentangled concept embeddings for various concept remixing and editing tasks. However, this requires that the concept axes are given a priori to the model, which limits the generality of the concept space that can be captured by the model. Moreover, train separate encoders for each concept axis, which does not make full use the inherent hierarchical structure of these concept axes.
28


Reviews:
[{"review_id": "zWJ8Ddf3TE", "reviewer": "Reviewer_qJGK", "summary": "The authors use multiple visual encoders to disentangle various visual concepts from images. These visual concepts are defined as vector axes based on natural language description. The proposed framework performs a simple training on a synthetic dataset that learns disentangled vectors for each concept. These disentangled vectors can be combined with language (similar to textual inversion paper) to generate images containing combined concepts. They show how their method disentangles and generates new images with variably joined concepts better than existing work.", "questions": "* Immediate question - why don't concept axis vectors collapse to same as text embeddings? Explain this more. \n* DeepFloyd - please cite appropriately \n* Consider more discussion on Textual Inversion (as related work), maybe in supplementary at least. Highlight cases where this is better than directly using text. \n* The work in [1] explores language defined concept axes in video domain - maybe an interesting comparison to discuss in related work  \n* Please include BLIP-based baseline results also in Table 1\n* Can you add more CLIP-score based (or a different new metric based) evaluations for other task (like concept extrapolation)? More quantitative evaluation could really strengthen the paper\n\n[1] Ranasinghe, K., & Ryoo, M., Language-based Action Concept Spaces Improve Video Self-Supervised Learning, NeurIPS 2023", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper explores an interesting idea of separating concepts in images in the visual domain in a novel manner\n2. Interesting use of a VQA model to augment their training setup\n3. Good use of diagrams to explain idea \n4. Clearly showcase qualitative improvements for selected cases", "weaknesses": "1. The generality of method on real world images (i.e. where visual concepts are not that easily disentangled) is unclear\n2. Limited evaluation (only one set of quantitative numbers)\n3. Some missing details (refer questions below)\n\n* Table 1: Are you reporting CLIP score and human evaluation on same table?? \nPlease point out CLEARLY that these are two different metrics in the Table caption. Or please separate into two Tables. This is highly confusing.\n\n--- \nThe authors sufficiently respond to raised concerns in rebuttal. \n1. The generality of the method (on real world images) is verified with multiple qualitative and quantitative evaluated added to appendix during rebuttal. \n2. Additional quantitative evaluation with comparison to more baselines are presented. \n3. The requested missing details are provided adequately. \n\nDue to these reasons, I vote to accept this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoZNR8UakC", "reviewer": "Reviewer_dyyk", "summary": "This paper proposed a new framework for Visual Concept Learning. By introducing a set of concept encoders, concept embeddings could be extracted from the input image, which could be recomposed later to produce desired image. The experiments showed that these concept embeddings could capture the visual nuances and they are disentangled with each other. Besides, this framework can learn the shared concepts across instances(images), in other words, it is more efficient than previous methods.\n\nThe paper proposes a new framework for VCL task that avoids massive human annotation, and could boost the research in related fields. It also drew the attention to the research direction of using continuous embeddings (instead of relying on generic words) as the visual concept descriptors. The concept is commendable, although the depiction falls short of perfection and would greatly benefit from additional elaboration and intricate explanations.", "questions": "a. On page 5, in sentence “text encoder c_{theta} of the T2I diffusion model…”, it should be “part of the text encoder c_{theta}”. Because a text encoder should take “text” as input rather than “text embeddings”. The original sentence might be confusing.\n \nb. On page 5, in formula (1), there is no explanation about the N and U notation. I assume they represent “multivariate normal distribution” and “uniform distribution” respectively. It would be more clear to annotate.\n \nc. On page 5, in sentence “so that they can directly inserted into the text embeddings”, it seems that a “be” is missed.\n \nd. The paper should mention the backbone T2I model earlier. It is first mentioned on page 5, it would be better to do it earlier.\n \ne. It would be better if the choice of using “12 CLIP layers” over “6 out of 12 tokens like Gal et al. 2023” is explained more in detailed.\n \nf. More details could be added about the test-time lightweight finetuning process.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "a. Proposed the idea to use BLIP-2 generated embeddings as “anchor” (pseudo ground-truth embeddings) to “separate” the entangled concept embeddings.\n \nb. This work proposed a framework to tackle Visual Concept Learning without human annotation, which is much more efficient than the previous works.\n \nc. Unlike most of the Textual Inversion techniques, this work could capture the concept appearing in different images. Therefore, it does not require a retraining on each image. Also, the learning efficiency is higher because it can learn from a larger pool of images.", "weaknesses": "a. The datasets used for the experiment were small and simple. It is not guaranteed that the claimed conclusions could be maintained when this framework is applied on more complex datasets (with much more concepts). The idea of using anchors to ensure that the embeddings are disentangle is great, however more experiments on larger datasets should be done to prove it. Given that there are only 2 to 3 concepts in each domain, the sparsity of concepts might be one of the reasons why the embeddings are disentangle.\n \nb. The effectiveness of L^{anchor} is not fully explained. The L^{anchor} is omitted during the test-time optimization procedure to avoid “over-committing to the coarse text anchors”. However, in the ablation experiment, the paper claims “disabling L^{anchor} deteriorates the performance”. It seems kind of contradictory, the paper should explain more about why “disabling L^{anchor}” is desired during one phase but it leads to unsatisfactory results in general evaluation.\n \nc. The ablation test is not fully explained. In “Editing Category” column, the results of “w/o Encoder & L^{anchor}_k” is actually higher than the results of “w/o L^{anchor}_k” in two metrics. This does not fully conform to the conclusion, quote, “and further removing the encoder decreases the overall decomposition performance”.\n \nd. It is hard for this framework to generalize to new “concept”. From what I understood, this framework could effectively generalize to new “mode” of seen “concept” (like new style or new color), but not to new “concept”. When applied to new concepts, e.g. “size” or “shape”, the corresponding concept encoders need to be trained. Also, from my perspective, we can’t only train the concept encoder of the new concepts. Because the sentence template “a photo of <e1> with <e2> color and <e3>  and <e4>...” needs to cover at least the majority of the concepts appeared to generate an image close enough to the original input image. Based on this understanding, when this framework is extended to new concepts, the trained concept encoders (of the seen concepts) need to be retrained together with the new ones. This setting is not more efficient than previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U0L4F5k725", "reviewer": "Reviewer_Bcg9", "summary": "The authors claim that their proposed model can learn a language-informed visual concept representation, by simply distilling large pre-trained vision-language models.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors claim that their proposed model can learn a language-informed visual concept representation, by simply distilling large pre-trained vision-language models.", "weaknesses": "1. What is concept representation learning? Is concept learning just the mutual translation of text and images?\n\n2. In the experiments, the authors primarily focus on conducting investigations using synthetic datasets. However, it raises concerns about the generalizability of the conclusions/findings obtained from synthetic datasets to real-world datasets.\n\n3. The concept learning should focus more on the understanding of concepts, especially at different granularities of the same concept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZAmziNmCU6", "reviewer": "Reviewer_jNDi", "summary": "The paper presents a method for learning concepts by distilling knowledge in a text-to-image generative model. The method assumes concept axes, in each of which specific information of an input image is encoded (like colors, materials, and object categories). The method learns a designated encoder for each concept axis with generated images. For training the encoders, the method uses an anchor loss for each axis based on a VQA model to further disentangle the axes and a reconstruction loss. The method is evaluated qualitatively and quantitatively (CLIPScore and human evaluation).", "questions": "I would like to have some responses for (1)-(3) in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is well-written. I can easily follow what is done in the method.\n\n(2) The method is simple and can be trained in 12 hours only with less than 1000 generated images, yet outperforming the similar existing methods. \n\n(3) The image generation results are really nice compared to the existing approaches.", "weaknesses": "(1) The performance of the method is mostly shown in qualitative evaluations. The quantitative evaluation only shows the performance of image generation by modifying some concept axes (and human evaluation). I think the paper would be better if it came with an objective quantitative evaluation of the obtained concepts themselves in some ways (though I didn’t come up with any good approaches for this). \n\n(2) Related to (1), I’m not sure if CLIPScore is really sensitive to arbitrary combinations of concepts. Some references or experimental results may help understand the experiment. \n\n(3) The paper’s purpose is not sufficiently clear. Is it to learn concepts for image generation? Or is it for some other downstream tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)