Learning Vision and Language Concepts for Controllable Image Generation
Authors: Shaoan Xie
Lingjing Kong
Yujia Zheng
Zeyu Tang
Eric P. Xing
Guangyi Chen
Kun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Concept; Identifiability; Controllable text-to-image
Primary area: general_machine_learning->representation_learning
TL;DR: A concept-learning framework  to identify atomic visual and textual concepts

Abstract: Concept learning seeks to extract semantic and interpretable representations of atomic concepts from high-dimensional data such as images and text, which can be instrumental to a variety of downstream tasks (e.g., image generation/editing). Despite its importance, the theoretical foundations for learning atomic concepts and their interactions, especially from multimodal distributions, remain underexplored.
In this work, we establish fundamental conditions for learning atomic multimodal concepts and their underlying interactions With identfiability guarantees. We formulate concept learning as a latent variable identification problem, representing atomic concepts in each modality as latent variables, with a graphical model to specify their interactions across modalities. Our theoretical contribution is to provide component-wise identifiability of atomic concepts under flexible, nonparametric conditions that accommodate both continuous and discrete modalities.  Building on these theoretical insights, we demonstrate the practical utility of our theory in a downstream task text-to-image (T2I) generation. We develop a principled T2I model that explicitly learns atomic textual and visual concepts with sparse connections between them, allowing us to achieve image generation and editing at the atomic concept level. Empirical evaluations show that our model outperforms existing methods in T2I generation tasks, offering superior controllability and interpretability.

essential for wide Learning Vision and Language Concepts for Controllable Image Generation
Shaoan Xie
1
Lingjing Kong" *1
Yujia Zheng'
Zeyu Tang'
Eric P. Xing 12 Guangyi Chen 12 Kun Zhang
Abstract
range of machine learning tasks, including visual under- standing and generation, transfer learning, and explainable decision making (Gal et al., 2022a; Jahanian et al., 2019;
Concept learning seeks to extract semantic and interpretable representations of atomic concepts from high-dimensional data such as images and text, which can be instrumental to a variety of downstream tasks (e.g., image generation/ecditng). Despite its importance, the theoretical founda- tions for learning atomic concepts and their inter- actions, especially from multimodal distributions, remain underexplored. In this work, estab- lish fundamental conditions for learning atomic multimodal concepts and their underlying inter- actions With identfiability guarantees. We formu- late concept learning as a latent variable identifi- cation problem, representing atomic concepts in each modality latent variables, with graphical model to specify their interactions across modal- ities. Our theoretical contribution is to provide component-wis wise identifiability of atomic concepts under flexible, nonparametric conditions that ac- commodate both continuous and discrete modal- ities. Building on these theoretical insights, we demonstrate the practical utility of our theory in a downstream task text-to-image (T2I) generation. We develop principled T2I model that explic- itly learns atomic textual and visual concepts with sparse connections between them, allowing us to achieve image generation and editing at the atomic concept level. Empirical evaluations show that our model outperforms existing methods in T2I generation tasks, offering superior controlla- bility and interpretability.
Härkönen et al., 2020; Shen et al., 2020; Wu et al., 2021;
Ruiz et al., 2023; Burgess et al., 2019; Locatello et al., 2020; Du et al., 2022a;b; Liu et al., 2023). For example, learning atomic concepts within both visual and textual modalities, along with their patterns of interaction, facilitates seamless alignment between these modalities. This alignment can significantly enhance downstream tasks such controllable text-to- ext-to-image (T2I) generation and editing.
However, achieving this alignment without proper guid- ing principles is highly challenging. In practice, current T2I methods often suffer from controllability issues, where modifications specified in the text input lead to unintended variations in the generated images (Saharia et al., 2022; Ding et al., 2022; Esser et al., 2024; Liu et al., 2022: Lee et al., 2023; Liu et al.. 2024; Betker et al., 2023). These issues stem from the entanglement of multimodal concepts and the resulting misalignment. These empirical challenges underscore the need for principled approaches to concept learning. Unfortunately, the theoretical foundations in this have been lacking. Only recently, Kong et al. (2024) have formalized concept learning within the framework of latent- identification, proposing identification condi- tions for learning discrete hierarchical models. Nevertheless, their theoretical formulation is limited to single-modal dis- tributions, restricting its applicability for aligning concepts across different modalities as required in T2I applications.
In this work, seek to establish fundamental conditions for learning atomic concepts and their interactions from multimodal distributions. We approach this challenge by framing it as a latent variable identification problem, where atomic concepts and their interactions are represented as latent variables and graphical model among them. Under this framework, our goal can be formalized as:
1 Introduction
Concept learning seeks to extract semantic and interpretable representations from high-dimensional data, such as images and text. These representations
Under what conditions can we identify multimodal latent variables and the underlying graph?
"quual contribution 'Carnegie Mellon University MMohaned bin Zayed University of Artificial Intelligence.
Closely related to our problem is the literature on mul- timodal causal representation learning (Yao et al., 2024;
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Morioka & Hyvarinen, 2023; 2024: Daunhawer et al., 2023; Sturma al., 2023; Gresele et al., 2020; Sun et al., 2024;

---

Learning Vision and Language Concepts for Controllable Image Generation
von Kügelgen et al., 2021). In particular, Daunhawer al. (2023); von Kügelgen et al. (2021); Yao al. (2024); Gre- sele et al. (2020) have demonstrated that a representation block can be identifiable if it is shared across different modalities. However, these block-wise identifiability re- sults are limited to the shared structure blocks, which may still encompass mixtures of multiple atomic concepts. This limitation hinders their applicability for tasks requiring pre- cise control over individual concepts, such as controllable text-to-image (T2I) editing, a challenge our work sets out to address. While Morioka & Hyvarinen (2024) and Morioka & Hyvarinen (2023) have achieved component-wise identi- fiability, their approaches rely on semi- parameerrttticc assump- tions about the latent distributions, including the use of the exponential family and assumptions of additive causal influ- ences. These conditions may be too restrictive for modeling the complex, sigh--imensional image-text distributions countered in real-world applications. Furthermore, Sun et al. (2024) offer component-we wiip identifiability for multimodal representations by leveraging sparse connections between modalities without specific parametric assumptions. How- ever, their method depends on particular sparsity conditions involving Jacobian matrices of functions among continuous variables, making their theoretical framework incompatible with modalities that include discrete variables, such as text.
disentangled text and vision representations with sparse connections. In our empirical evaluations, ConceptAligner outperforms state-of-the-art text-to-image models in control- lable generation tasks. Moreover, thanks to the identifiabil- ity theory, ConceptAllineer offers interpretability, enabling more precise manipulation.
2 Related Work
Concept learning. A flux of recent work focuses on learn- ing interpretable concepts from images. The concept bottle- neck model (Koh et al., 2020) makes predictions on human- annotated concepts and then applies these concepts to down- stream tasks. This approach has spurred a significant amount of follow-up work (Zarlenga et al.. 2022; Yuksekgonul et al., 2023; Kim et al., 2023; Havasi et al., 2022; Shang et al., 2024; Chauhan et al., 2023). A separate line of research aims to achieve unsupervised concept discovery from vision data by proposing novel neural network architectures and training objectives (Burgess et al., 2019; Locatello et al., 2020; Du et al., 2022a;b; Liu et al., 2023). In contrast, our work considers the theoretical perspective of concept learning and provides reasonable identification conditions with empirical implementations. Rajendran et al. (2024) formulate concepts as affine subspaces of latent variables and provide identifiability guarantees for these subspaces. In contrast, we directly identify each latent variable, which enables us to directly control atomic aspects. Similar to our work, Kong et al. (2024) formulate concept learning as the identification of a discrete hierarchical model and offer theoretical guarantees. However, this work assumes fully discrete latent variables and a single modality, which fails to capture the multimodal alignment problem we consider.
In this work, we establish component-wise identifiability for atomic concepts under flexible nonparametric conditions that accommodate both continuous and discrete modalities. Table 1 highlights the key distinctions between our approach and existing theoretical frameworks. Specifically, we as- sume that the paired text t and image i are transformations of underlying atomic textual concepts zTT := [=>])>> and visual concepts z² := [>|]s respectively. The abstract discrete concepts z" generate detailed visual concepts z via a sparse graphical model (Figure 1). At a high level, our identification theory consists of two stages. First, we lever- age the distribution changes in the vision modality p(i[t) induced by variations in the text t to identify atomic vision concepts =/1- Intuitively, if the visual observation p(iltt) varies sufficiently, one can discern fundamental visual vari- ations. For example, an infant might learn atomic visual concepts like "fur" and "wings" by observing and compar- ing grouped images of "dogs" and "birds". As the second step, we show that once visual concepts =! are identified, they can facilitate the identification of atomic text concepts =T. provided that the interactions between visual and textual atomic concepts are not overly complex. Returning to the running example, after recognizing atomic concepts "wings" and "fur", the infant can ground these two visual concepts with their corresponding, more abstract textual concepts.
Causal representation learning. Causal representation learning seeks to infer high-level causal variables from raw, low-dimensional observations (Schölkopf et al., 2021) with identifiability guarantees. Unfortunately, it has been shown that identifying latent variables in general nonlin- ear causal models is impossible without additional condi- tions (Hyvärinen & Pajunen, 1999; Locatello et al., 2019). Existing identifiability conditions include (1) constraints on the generating function (e.g., sparsity) (Xu et al., 2024a; Zheng et al., 2022; Zheng & Zhang, 2023; Buchholz et al., 2022), (2) multiple distributions arising from the same causal model (Hyvarinen et al., 2019; Khemakhem et al., 2020a; Zhang et al., 2024; Kong et al., 2022; von Kügelgen et al., 2024; Jiang & Aragam, 2023: Brehmer et al., 2022; Xie al., 2023); 3) temporal transitions among latent vari- ables (Yao et al., 2022; 2021; Klindt et al., 2021; Hyvarinen & Morioka, 2017: Lachapelle et al., 2024) and (4) paired multimodal data (c.g., text-image pairs) (Yao et al., 2024; Morioka & Hyvarinen, 2023; 2024: Daunhawer et al., 2023; Sturma et al., 2023; Gresele et al., 2020). Our work aligns most closely with the multimodal category. To contextual-
Our theoretical insights lead to principled T21 genera- tive model ConceptAligner in which we explicitly learn
2

---

Learning Vision and Language Concepts for Controllable Image Generation
ize our contributions, Table compares prior multimodal identification theories with ours and we provide detailed discussion in Section 4.
Notations. We denote the dimensionality of multidi- mensional variable with d(-). We denote the integer set [l....,n} with [n]. We refer to a specific coordinate or dimension of a random vector (variable) as "component", indicated by a subscript.
Table 1: Related work on multimodal causal represen- tation learning. This table considers whether the latent- variable distribution is nonparametric, whether the identi- fiability is component-wisse and whether the framework accommodates discrete latent variables.
Data-generating processes. We illustrate the data- generating process in Figure 1 and define it as follows.
z ~ p(z)), z' ~ p(z'l""))
(1)
t ク i i: o
Related work
Nonparumetric Price Component wise Iden. Discrete Latents
von Kägelge at al. (2021)
Duunhwwer al. (2023)
We denote the observed text as a discrete variable t E T C N and the observed image as a continuous variable i E I C
Moriska Hysarina (2024)
(2004)
al. (2004)
Rd(1 The textual representation is a multidimensional discrete variable zT E ZT C. CN"("") which generates the text observation through a mapping n² 2 T. The components =M encode individual atomic textual concepts with potential statistical dependence. Likewise, the visual representation is denoted as z! 2 C Rd(a'), which generates the image through gو z - I. Its components zy represent atomic visual concepts in the distribution.
Ours
Controllable text-to-image generation. ControlGAN (Li et al., 2019) introduces word-level generator and discrim- inator to disentangle different visual attributes. Since it is challenging to control the output with a text prompt, many methods resort to using additional supervision, such as canny edges and depth maps (Zhang et al., 2023; Zhao et al., 2024; Mou et al., 2024: Voynov et al., 2023). Some methods also try to inject CLIP image representation to allow variations in the input images (Ye et al., 2023).
I
Image editing, Image editing requires the model to fol- low semantic instructions to modify an image. One ap- proach to explore the latent space of the image by the inversion of generative models (Abdal et al., 2019; 2020; Alaluf et al., 2022: Epstein et al., 2022; Xia et al., 2022: Zhu et al., 2020) by learning
22
4
22
23
image encoder (Chai et al., 2021; Richardson et al., 2021; Tov et al., 2021), and then edit images through latent vector manipulation. With the advent of CLIP (Radford et al., 2021), which bridges the latent space between images and text, numerous meth- ods (Crowson et al., 2022; Gal et al., 2022b; Kim et al., 2022: Kwon & Ye, 2022; Patashnik et al., 2021; Abdal et al., 2022) have leveraged its capabilities to conduct textual in- structions on images. More recently, pre-trained image-text diffusion models, such as Stable Diffusion (Rombach et al., 2022), have further facilitated image editing by providing a robust link between textual instructions and image modifi- cations (Avrahami et al., 2022; Brooks et al., 2023; Ramesh et al.. 2022; Hertz et al., 2022; Meng et al., 2021; Kawar et al., 2023). Xu et al. (2024b) focus on designing attention maps to replace the target attention map with the source map. In contrast, we develop superior conditioning representation. These two approaches are complementary.
Figure 1: The data-generating process. The text-image pair (t. originates from its corresponding representation pair (zT,z)) z'). The textual concepts in zT are high-level and govern the generation of relatively low-level visual concepts z'. The dashed curves indicate potential statistical dependence among the components.
Atomic visual/textual concepts and their interactions. We note that "atomic" is defined w.r.t. the distribution. For instance, component z!!' in a distribution rich with textual and visual features could encode the concept "dog eyes" whereas in a less diverse distribution, it might encode a coarser concept like "dog faces". The textual representa- tion ZT serves as an upstream causal variable relative to the visual representation z' because text typically contains high-level abstract concepts, while the visual representation includes lower-level details that reflect these concepts. For instance, the visual concept of "dogs" encompasses not only the general idea of a "dog" from the text but also visual details like textures and subtle contours. Importantly, due
3 Problem Formulation
In this section, we formalize the tata-generating process that underlies the interaction between the textual and visual modalities as the foundation for theoretical analysis.
Note that any multidimensional discrete variable can be con- verted into one-dimensional variable via an invertible mapping, so we represent the observed text t in this way.
3

---

Learning Vision and Language Concepts for Controllable Image Generation
to its low-level nature, each visual concept may be influ- enced by multiple textual concepts. For example, the visual concept "hat" may be affected not only by the textual con- cept "hat" but also by other textual concepts like "color". These complex relationships are encoded in the graphical model Gt Moreover, statistical dependence may exist among the atomic concepts - for instance, dog species may induce dependence between "eyes" and "noses". Our frame- work captures these potential dependencies among textual concepts in z²" (dashed curves in Figure 1).
fundamental indeterminacy in the ordering of latent vari- ables (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Kivva al., 2021).
To achieve our goal, we introduce Condition 4.2 and Con- dition 4.3, which facilitate the identification of the visual concepts z,, and textual concepts iM respectively. Condition 4.2 (Visual Concept Identification).
[Invertibility & Smoothness]: Generating functions g² and g are invertible and g is smooth.
Concept identification and alignment for multimodal representations. Our primary objective is to learn the atomic concepts and their interactions within multimodal data, specifically from textual and visual modalities. By identifying these underlying concepts and modeling their in- teractions, we explicitly characterize how high-level textual concepts influence detailed visual concepts. This founda- tional characterization enables downstream tasks, such as precise control and manipulation of visual content through textual input. In our formulation, this entails: 1) identify- ing the atomic textual concepts =!! present in the text t; 2) identifying the atomic visual concepts z/ in the image and 3) learning the interactions between z!e and z/z through the causal graph G By learning these concepts and their interactions, we align the textual and visual representations at an atomic level. This alignment facilitates downstream applications. For example, when editing an image from "a dog" to "a dog with glasses," our model leverages the learned interactions to add the textual concept "glasses," which in turn affects only the corresponding visual concept without inadvertently altering other aspects.
ii [Smooth and Positive Density]: The probability density function of z' is smooth and positive, i.e., p(z /z (z") 0 is smooth over 2 x z
ii [Conditional Independence]: Components 2 are in- dependent given z²: p(zzlz') = p(=
iv [Sufficient Variabilityl: For any z E 2 there ex- ist 2d(z²) + +1 values of ztz, i.e., [(n) with n 0, 1..... (1....,2d(z) + 1, such that the 2d(z!) vectors w(z (/))) (izz,z))) are linearly independent, where vector vectorw(z), 2
) is defined as follows:
w(z,, = (llog (2/12")
aogg
Ed(at) zTT
Ozl
Plogg (z11221)
02 log [2] (= 0(2 )(2)(2))) )2
(0z])²
Interpretation & discussion. Condition 4.2 ensures that each visual concept 2!, can be disentangled from the oth- ers. The key idea is that each 2/ should exhibit sufficiently distinet behavior be distinguishable. Specifically, Condi- tion 4.2-iv requires the conditional distributions p(= Iz vary sufficiently over different zT. Intuitively, when we change the text from "cats" to "dogs", the visual concepts "cyes" and "noses" exhibit different patterns of change, al- lowing us to recognize them as separate concepts. In gen- eral, we expect this condition hold standard text-image datasets (e.g.. LAION (Schuhmann et al., 2021)) contain millions of captions, far exceeding the number of possi- ble visual concepts. Additionally, we may follow existing methods Cc.g..Wagg et al. (2023); Chen et al. (2024); Betker et al. (2023)) to employ vision-language models to gen- erate higher-quality captions. Oftentimes, other natural properties can also greatly weaken this condition (e.g.. spar- sity (Li et al., 2025)). Such sparsity is often encouraged implicitly or explicitly in generative models (e.g., sparse attention patterns). Condition 4.2-i ensures the observed variables preserve all latent variables' information. Other- wise, it would be impossible to recover them from observed variables. These conditions are commonly assumed in the independent component analysis literature (Hyvarinen &
In Section 4, we provide theoretical guarantees and discus-
sions regarding these objectives.
Identification Theory
In this section, we present the identification conditions and theoretical guarantees that underpin our objectives.
First, we define component-wiss identifiability, which serves formal definition for disentanglement.
Definition& 4.1 (Component-wind wise Identifiability). Let Z and & E 2 be variables under two model specifications and 0 respectively. We say that z and à are identified component-wise if there exists a permutation и such that for each i E [d(z)), 2 hz(2*()) where h4 is invertible.
In our context, the two specifications are given by 0 := (p(z', z²),9', ""'""""))) and = (p(z), " "T",,"' (gT), where we consider 0 as the true model and 0 as its estimate. Under the component-we wise identifiability. our estimate & captures full information of (n( and no information from 2, such that j # (1). This property provides a formal guaran- tee of disentanglement. The permutation accounts for the
4

---

Learning Vision and Language Concepts for Controllable Image Generation
Morioka, 2016; Hyvarinen et al., 2019; Khemakhem al., 2020a;b; Kong al., 2022; Zhang al., 2024). Condition 4.3 (Textual Concept Identification).
visual concepts p(=/.t) over different textual descriptions (since gF is invertible, varying is equivalent varying z ) to identify visual concepts z Subsequently, treating the identified visual concepts 2,, as observed variables, we exploit the sparse graphical structure between = and z (as ensured by Condition 4.3-iv) to disentangle textual concepts =T and recover the causal graph G
i [Non-degeneracyy; IP p|z = -k]. for all k. E Ω2; for all components zt, P[z)Pa(=))) = ki] + P P[=l|Pa(z)) -ka), i # hzz-
Theoretical contribution. Our work has advanced the the- oretical understanding of identifiability in multimodal rep- resentation learning. Previous studies (Daunhawer et al., 2023; von Kügelgen et al., 2021; Yao et al., 2024) have established identifiability under shared representation as- sumptions across modalities. However, their guarantees are limited to identifiability up to subspaces (groups of latent components) determined by the sharing patterns, which is insufficient for tasks requiring precise control over indi- vidual concepts. In contrast, we achieve component-wise identifiability of atomic concepts.
ii [No-twins]: Distinct components 2 have distinct neighbors: ne(zm) & ne(z,) ) for m + n.
ii [Maximality]: There is no DAG ğr resulting from splitting a latent variable z into (21, in G such that the resultant distribution is Markov w.r.t. G and G satisfies it.
iv (Non-Subset Observed Children]: For any pair of and of with i # j. one's 's observed children are not the subset of the other's, Chig-1(22)) & Choe-n
Some recent works, such as Morioka & Hyvarinen (2024), also aim for component-win -iie identifiability but rely on semi- parametric assumptions like the exponential family and ad- ditive causal influences. Our fully nonparametric framework offers greater flexibility for modeling complex real-world distributions. While Sun et al. (2024) also leverage sparse connectivity and assume nonparametric models to achieve component-wise identifiability, their approach depends on specific sparsity conditions involving Jacobian matrices of continuous functions, which preclude discrete variables a capability desired for our problem.
Interpretation & discussion. Condition 4.3 facilitates the identification of discrete textual concepts 2T. The criti- cal aspect is the sparse connectivity between the visual concepts = and the textual concepts 2ne as specified in Condition 4.3-iv. Since the visual concepts 2/, have already been identified (thanks to Condition 4.2), we can treat them as observed variables. Condition 4.3-iv ensures that each textual concept =m exerts distinguishable influences on the different subsets of visual concepts 2/m making them iden- tifiable. This condition is reasonable because the textual concepts zm are defined to be atomic and thus should not have heavily overlapping effects on the visual concepts I! Consider concepts like "fur" and "ears" when describing a cat. These concepts should affect partially distinct visual features. If every visual feature triggered by "ears" was also triggered by "fur," these concepts aren't 't genuinely atomic and should be restructured. Similar conditions have been adopted in prior work (Kivva et al., 2021; 2022; Kong et al., 2024). Condition 4.3-i,ii, jii are a set of necessary conditions for discrete latent variable identification as extensively dis- cussed in prior work (Kivva et al., 2021; 2022). Intuitively, Condition 4.3-i ensures that each discrete latent variable has detectable influence through the observed variables. Condition 4.3-i and iñ eliminate the indeterminacy arising from arbitrary merging and splitting over latent variables, making identification tractable.
Practical implications. As motivated in Section 3, Theo- rem 4.4 provides the identification guarantee for disentan- gled atomic textual and visual concepts (z and 2!)) and their interactions. This foundation is crucial for aligning these two modalities and downstream tasks including T2I generation. In Section 5, we implement the date-generating process and key conditions (c.g., sparse gt in Condi- tion 4.3-iv) evaluate our framework in Section 6.
5 ConceptAligner: Controllable
Text-to-Image with Learning Concepts
Consider common use case in text-to-image generation: a user provides a text prompt to generate an image, then wishes to make minor edits, such as changing only the color of the clothing. Controllable generation enables the user to modify the prompt accordingly, prompting the model to adjust the specified feature while preserving all other aspects of the image. This ability to make targeted changes without unintended alterations underscores the importance of controllable text-to-image generation.
Theorem 4.4 (Atomic Concept Identification). We assume the generating process in (1). Under Condition 4.2 and Condition 4.3, we altain component-wise identifiability of concepts z and z! (Definition 4.1) and the bipartite graph gi-i up to permutation of component indices.
Proof sketch. As we have outlined in discussions on Condi- tion& 4.2 and Condition 4.3, we first utilize the variability of
In this section, we present our empirical approach to control- lable text-to-image generation-guide by the identifiability
5

---

Learning Vision and Language Concepts for Controllable Image Generation
cat holding a sign that says "Concept Aligner"
i
diff
Text Embeddings
Image Encoder R!
Diffusion Model Ve
m
Alyui
spa
Lkl
z?
z!
Learnable Query
Text Encoder RD
Concept Network R
Figure 2: Diagram of ConceptAlignerr We use text network R2 to recover the textual concepts z and image network R? to recover the visual exogenous information We transform them with our concept network R and obtain visual concepts z!. Finally, we feed the visual concepts z! into a diffusion transformer le as the conditioning input.
results established in Section a-and detail the model archi- tecture, implementation specifics, and training objectives.
of learnable queries. The final concept z² are then obtained by conditioning on the text embedding f. as:
5.1 Model Design
z = (t).
(3)
Overall structure. Controllable image generation requires the generative model to faithfully capture the interaction be- tween textual and visual concepts (i.e., ZT and z²), aligning the two modalities and enabling precise control from text to image i. This alignment allows for precise manipulation of image attributes based on textual conditions. To construct our theoretical framework, our model ConceptAllgner con- sists of four major modules:
Image encoder R.. Visual concepts z' are sampled from distribution conditioned on textual concepts z i.c. p(z)|zT). We recover the exogenous information in- volved in the sampling (which can be written as z! == (z e)) by following the variational autoencoder frame- work (Kingma, 2013):
n,, (1), (, +=
(4)
1. Text network RT that extracts textual concepts 2 from the text t;
where R! is the image encoder. We adopt the reparameteri- zation trick and sample from the prior NN I).
2. Image network R! that maps the image back to its ex- ogenous information associated with the conditional distribution p(z) (2T) (i.e., z := I2' (z , .));
Concept network R. After obtaining the exogenous vari- able e and the textual concepts z in (4) and (3), we imple- ment a concept network RC to transform them into visual concepts z (i.e.. z p(z (z"):
3. Concept network R that produces the visual concepts z' given the textual concepts zT and the exogenous information c, i.e., the sampling step z p(z 22T;;
= R (z T @To,)))
where m is a learnable mask whose components take values in [0. to control the sparsity of the connectivity from zT to z! (i.e., gi my = 0 indicates no influence from the textual concepts 27 to the visual concepts =/--
4. Conditional generation model vo that renders out the visual representation z! to image i.
We present our model ConceptAlignes in Fig. 2 and intro- duce each module in detail as follows.
We utilize a diffusion trans- former 0g for our conditional generation. It takes as input a noisy image and certain conditioning to perform denois- ing. In the conventional T2I model, the conditioning the text embedding from a pre-trained text encoder (e.g., CLIP (Radford et al., 2021)). In our method, the visual representation z as the conditioning.
Conditional generation
Text encoder RT. We first use pre-trained text embeddings obtain the embedding t of the text input These embed- dings retain sequential dependencies and do not represent concepts. To extract meaningful and disentangled concepts from these embeddings, we introduce a perceive-resampler text network RT (Alayrac et al., 2022), which maps text embeddings into a structured concept space (i.c., zT). We predefine the number of textual concepts and initialize a set
5.2 Loss Functions
As motivated before, our empirical goals include
6

---

Learning Vision and Language Concepts for Controllable Image Generation
Smile
Angry
Surprised
Blue
Red
Golden
Diamond
Flower
SD3.5-L
2025
2025
2025
FIUX-D
2025
2025
2025
S
SANA
2025
2025
2025
DALLE3
2025
2025
2025
Ours
2025
2025
2025
Figure 3: Controllable text-to-image generation comparisons. Thanks to the identifiability of visual and textual concepts, method can make only necessary changes without affecting other attributes. For example, when modify the prompt from "golden crown" to "flower crown" for the dog. SD-3.5-L changes the color of the sign from black to white background whereas our model precisely implements the requested change.
1. T2I generation: accurately generating images based on the textual description
Kullback-Leibler divergence loss. We apply an image encoder to recover the exogenous information e, which is sampled from prior distribution (0, I). To enforce this, we match the marginal distribution of with the prior distribution as
2. Text-based editing:
applying changes specified in textual instruction t previously generated im- age i, without introducing unnecessary modifications.
Lai log(o) + 0² + 112
1
2 2
Having introduced the architecture our model, we now present the training losses to achieve the above goals.
Sparsity regularization. As indicated in Condition 4.3-iv, the connections from z to z' should be sparse enough to recover the textual concepts 2T. In our implementation, we use a learnable mask m to modulate the connectivity by applying a sparsity regularization:
Diffusion loss. As mentioned previously, condition the diffusion transformer model with visual concepts z!. and draw random
We set the initial diffusion sample noises to corrupt the image in. We adopt the denoising loss (Liu et al., 2022; Lipman et al., 2022; Xie et al., 2024; Esser et al., 2024),
Capa
|m|]
(8)
i, = (1 - io an. Laitt
(6)
Full objective. Combining the above loss functions, we arrive our final objective:
ar 2²z (n- io)
L == Laiff + Aspa + Aua
(9)
where a E (0, 1) is sampled from a predefined distribution, e.g., uniform distribution (4(0,, 1].
where Aspa and Asa balance the three loss terms.

---

Learning Vision and Language Concepts for Controllable Image Generation
Method
CLIP-I
LPIPS 1 CLIP-T
DINO 1
"Huk riding
*Hulk [Jatman thapel riding
Balmanis horse."
SD3.5-M
0.862
0.428
0.321
0.719
SD3.5-L
0.864
0.456
0.318
0.700
FLUX-S
0.868
0.463
0.318
0.740
FLUX-D
0.872
0.452
0.310
0.721
SANA
0.870
0.438
0.313
0.741
CANA-Finetunn
0.865
0.457
0.320
0.750
ConceptAlligeer
0.903
0.357
0.314
0.835
sparsity
0.863
0.388
0.308
0.751
w.o diffusion
0.999
0.000
0.171
0.999
w.o KI. 0.909
0.246
0.206
0.826
Table 2: Comparisons on Emu-Edit dataset. We generate pairs of images with the source and target prompt in Emu- edit dataset (Sheynin et al., 2024). Our method achieves the best or competitive performance across various metrics.
"A chicken is hoiding ice creem
shape) holding
peacock is holding ice oream
Figure 4: Visualization of the learned concepts. From the text embeddings of "Batman" and "peacock", we isolate distinct visual elements specifically, the flowing cape char- acteristic of Batman and the distinctive shape associated with peacocks. Our approach enables selective modifica- tion of individual features while preserving all other aspects. This is demonstrated in the middle columns, where we can substitute one learned feature from the left column with corresponding feature from the right column, resulting in coherent images that differ only in that specific aspect.
6 Experiments
In this section, we first describe the experimental setup, including implementation details, datasets, baselines, and evaluation metrics. We then present the results, covering comparisons with baselines, visualizations of learned con- cepts, disentanglement analysis, and ablation studies. More results and analyses can be located in Appendix B.
T21 generation. Specifically, we compare with stable- aeffuuiion3.5mmedine (SD3.5-M) (Esser et al., 2024), stable- diffusion3..5 large (SD3.5-L) (Esser al., 2024), Flux.I-dew (FLUX-D) (Labs, 2024), Flux. 10oox.1SSceell (FLUX-S) (Labs, 2024) and SANA (Xie et al., 2024). For a fair comparison, we also finetune SANA on our training data and refer to the resulting model as SANA-Finetune. We fix the random seed for each pair of generations.
6.1 Setup
Implementation. We implement our method based on SANA (Xie et al., 2024). Firstly, we employ a 6-block perceiver resampler (Alayrac et al., 2022) to transform the entangled text embedding into our textual concept z²". We define the number of textual tokens as 64. Then we use transformer block transform Siglip (Zhai et al., 2023) image embedding into the mean and variance of the latent c. Then we feed the re- re-parametrized latent into 6-block perceiver resampler with masking m @ ZT. Finally, we obtain the image representation z! and replace the original text embedding with this representation. We use LoRA on the diffusion transformer with rank 256. All the parameters trained with batch size 768 and learning rate 5 10-5,
Metrics. To evaluate the similarity between the generated pairs of images, we calculate the CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) embedding similarities, along with the LPIPS (Zhang et al., 2018) score. Additionally, we assess the average CLIP image-text similarity to determine how well the generated image aligns with the given prompts.
6.2 Results
Datasets. We use FLUX-S (Labs, 2024) to generate 2 mil- lion images using prompts sourced from the LAION dataset. Subsequently, we employ QWEN2-VL (Wang et al., 2024b) to produce accurate textual descriptions.
Comparisons with baselines. We present our sox-tto-mmae generation results in Table 2. As shown, Concept AonneptAligner outperforms other methods in terms of CLIP-I, DINO sim- ilarity, and LPIPS scores. Figure 3 further illustrates that our model can apply the intended changes while preserving unrelated attributes- for example, maintaining the color of the sign or the pose of the dog.
To evaluate the controllability of our generative model, we need to generate pairs of images that reflect specific target changes. For this purpose, we utilize the EMU-Edit dataset (Sheynin et al., 2024), which includes 3,589 paired prompts spanning seven image editing categories. Using the pro- vided source and target prompts, we generate corresponding image pairs for analysis.
Analysis of the concepts. To analyze the learned concepts, we interpolate between them to generate new images and compare these with the originals. As illustrated in Fig. 4, ConceptAlliner effectively disentangles complex text em-
Baselines. We compare with state-of-the-ee methods for
8

---

Learning Vision and Language Concepts for Controllable Image Generation
concepts zf is crucial, which motivates the need for sparsity regularization. To assess this, train model without the masking module, resulting in fully connected z" + z! mapping. A qualitative comparison of outputs from both models is shown in Fig. 5. We observe that the model lacking sparsity introduces more irrelevant changes when the prompt is modified from horse to zebra, underscoring the importance of promoting sparse zT + z connections, consistent with our identifiability theory.
Withott sponce trtiraction
With sparao intercttion
Additionally, removing the diffusion loss results in nearly identical image pairs, indicating that the model fails to lever- age the paired prompts. Eliminating the KL divergence loss causes the exogenous variable e to carry excessive informa- tion about the clean image, allowing the model to ignore the text condition for denoising, and ultimately fail to produce prompt-consistent outputs.
astronsut is ricing
"An retroonnd redingg 200rd"
"An astrooaud is unicoos
Forsal
Figure 5: Comparisons with the model trained without sparsity regularization. Without applying sparsity regu- larization, the model makes unnecessary changes when we alter the text imput from "horse" to "zebra".
Method
Subject f
Prompt f
7 Conclusion
SANA (Xie et al., 2024)
0.913
0.615
In this work, we develop theoretical foundations for learn- ing atomic concepts and their interactions from multimodal data distributions. In comparison with prior thcoretical work Yao et al. (2024); von Kügelgen et al. (2021); Daun- hawer et al. (2023); Gresele et al. (2020); Sun et al. (2024); Morioka & Hyvarinen (2024; 2023), our theory provides component-wise identifiability for each atomic concept with- out resorting to semi- cami-parmmetri assumptions, while admit- ting modalities consisting of continuous and discrete latent variables. As consequence of our theory, we design prin- cipled T21 generative model. Under thorough evaluation, our model outperforms baselines in T2I generation tasks and demonstrates superior interpretability and controllabil- ity. Limitations: order to identify visual atomic concepts an Condition 4.2-iv demands the text t to impose sufficient influences over the visual concept distribution p(z'|t). Prac- tically, this requires captions in the dataset to be sufficiently informative and diverse, which may be violated on datasets with poor caption qualities.
CANA-Finetune (Xie et al., 2024)
0.934
0.731
ConceptAligneer
0.946
0.814
Table 3: Disentanglement comparisons. We modify the subject's 's action in the source prompt and evaluate whether the generative models can preserve the subject's identity, given that only the action in the text input is altered. Our method achieves the highest scores in both subject consis- tency and prompt consistency.
beddings into atomic textual concepts. For instance, in the second row, we recover the shape concept of a peacock and successfully generate chicken with a peacock shape. These results demonstrate that our method learns to decompose text embeddings into interpretable, atomic concepts.
Disentanglement analysis. If the model learns disentan- gled representations, it should be able to preserve unrelated concepts when only specific parts of the text are modified. To evaluate this, we prompt ChatGPT to randomly generate 10 animal names and 10 actions. For each animal, we fix its identity and modify only the action, resulting in 100 origi- nal-edited image pairs. This process is repeated across 10 random seeds, yielding a total of 1,000 pairs.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
to assess two criteria: subject consistency, measuring whether the animal identity preserved, and prompt consistency, evalu- ating whether the intended action change accurately re- flected. ConceptA ligner achieves the highest scores on both metrics compared to SANA and finetuned SANA, demon- strating superior disentanglement capabilities.
For evaluation, we use
A Aknnoleedmennts
We appreciate the anonymous reviewers and the area chair for their insightful feedback. We would also like acknowl- edge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Insti- tutes of Health (NIH) under Contract 501HL159805 and grants from Quris AI, Florin Court Capital, and MBZUAI- WIS Joint Program. The work of L. Kong is supported in part by NSF DMS-21 34080 through an award to Y. Chi.
Ablation study. We present the quantitative ablation results in Table 2. As outlined in our theoretical framework, achiev- ing sparse connections from textual concepts T to visual

---

Learning Vision and Language Concepts for Controllable Image Generation
References
Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1xXxv:9901111900 2019.
Abdal, R., Qin, Y., and Wonka, P. Image2stylegan: How to embed images into the stylegan latent space? Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 4432 4441, 2019.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Abdal, R., Qin, Y., and Wonka, P. Image2stylegan+++ How to edit the embedded images? In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 8296-8305, 2020.
Chai, L., Wulff, J., and Isola, P. Using latent space regres- sion to analyze and leverage compositionality in gans. arXiv preprint arXiv:2103. 10426, 2021.
Abdal, R., Zhu, P. Femiani, J., Mitra, N., and Wonka, P. Clip2stylegan: Unsupervised extraction of stylegan edit directions. In ACM SIGGRAPH 2022 conference pro- ceedings. pp. 1-9, 2022.
Chauhan, K., Tiwari, R., Freyberg, J., Shenoy, P., and Dvi- jotham, K. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 5948-544899 14488-555,44 2023.
Alaluf, Y., Tov, O., Mokady, R., Gal, R., and Bermano, A. Hyperstyle: Stylegan inversion with hypernetorks for real image editing. In Proceedings of the IEEE/CVF conference computer Vision and pattern recognition, pp. 1851 1-18521, 2022.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large modal models with better captions. European Conference on Computer Vision, pp. 370-387. Springer, 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems,
Crowson, K., Biderman, S., Kornis, D., Stander, D., Halla- han, E., Castricato, I.., and Raff, Vqgan-clip: Open domain image generation and editing with natural lan- guage guidance. In European Conference on Computer Vision, pp. 88-105. Springer, 2022.
2022.
Avrahami, O., Lischinski, D., and Fried, O. Blended diffu- sion for text-driven editing natural images. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18208 18218, 2022.
Daunhawer, I., Bizeul, A., Palumbo, E., Marx, A., and Vogt, J. E. Identifiability results for multimodal con- trastive learning, The Eleventh International Confer- ence Learning Representations, 2023. URL https: / opppereeeeww net/ ffwpumidddooduuueeeeeeeooeee
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang. J., Lee, J.. Guo, Y., et al. Improving image generation with better captions. Computer Sci- ence. https://edn. openai. com/papers/dallle-3. pdf, 2(3): 8, 2023.
Deng, Y., He, X., Mei, C., Wang, P., and Tang, F. Fireflow: Fast inversion of rectified flow for image semantic editing. arXiv preprint
Brehmer, J., De Haan, P., Lippe, P., and Cohen, T. S. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319- 38331, 2022.
2024.
Ding, M., Zheng, W., Hong, W., and Tang. J. Cogview2:
Faster and better text-to-image generation via hierarchical transformers. Advances Neural Information Processing Systems, 35:16890-16902, 2022.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Du, Y., Li, S., Sharma, Y., Tenenbaum, J. B., and Mordatch, I. Unsupervised learning of compositional energy con- cepts. In Proceedings of the International Conference on Learning Representations (ICLR 2022), 2022a.
Buchholz, S., Besserve, M., and Schölkopf, B. Function classes for identifiable nonlinear independent component analysis. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Pro- cessing Systems, 2022. URL https: / htttps://opeereeeeee openreview. net/ ditpummmdddppaaaaaee
Du, Y., Smith, K., Ulman, T., Tenenbaum, J., and Wu, J. Un- supervised discovery of 3d physical objects from video. Conference on Computer Vision and Pattern Recogni- tion (CVPR 2022), 2022b.
10

---

Learning Vision and Language Concepts for Controllable Image Generation
Epstein, D., Park, T., Zhang, R., Shechtman, E., and Efros, A. A. Blobgan: Spatially disentangled scene representa- tions. In European Conference Computer Vision, pp. 616-635. Springer, 2022.
Hyvarinen, A., Sasaki, H., and Turner, R. Nonlinear ica us- ing auxiliary variables and generalized contrastive learn- ing. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 559-868. PMLR, 2019.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Jahanian, A., Chai, L.,, and Isola, P. On the" steerability" of generative adversarial networks. In International Confer- ence on Learning Representations, 2019.
Jiang, Y. and Aragam, B. Learning nonparametric latent causal graphs with unknown interventions, 2023.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Confer- ence on Learning Representations, 2022a.
Ju, X., Zeng, A., Bian, Y., Liu, S., and Xu, Q. Pap inversion: Boosting diffusion-based editing with 3 lines of code. In The Twelfth International Conference on Learning Representations, 2024.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani. M. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., and Cohen-Or, D. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), -13) 2022b.
Gresele, L., Rubenstein, P. K., Mehrjou, A., Locatello, F., and Schölkopf, B. The incomplete rosetta stone problem: Identifiability results for vie nonlinear ica. Un- certainty in Artificial Intelligence, pp. 217-227. PMLR, 2020.
Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020a.
Khemakhem, I,, Monti, R., Kingma, D., and Hyvarinen, A. Ice-beem: Identifiable conditional energy- deep models based nonlinear ica. Advances in Neural Infor- mation Processing Systems, 33: 13:12768-12777 2020b.
Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S.
Ganspace: Discovering interpretable gan controls. Ad- vances in neural information processing systems, 33: 9841-9850, 2020.
Kim, E.,, Jung, D.,, Park, S., Kim, S., and Yoon, S. Proba- bilistic concept bottleneck models. In International Con- ference Machine Learning, pp. 16521-16540. PMLR, 2023.
Havasi, M., Parbhoo, S., and Velei- F. Address- ing leakage in concept bottleneck models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL htthhhts/////eeeeee http///eeereeeeeeeeeeee/eeeeee.e net/ forum? id-tglniD_fn9.
Kim, G., Kwon, T., and Ye, J. C. Diffusionclip: Text- guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2235,, 2022.
Hertz, A., Mokady, R. Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-promee im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Kivva, B., Rajendran, G., Ravikumar, P., and Aragam, B. Learning latent causal graphs via mixture oracles. Ad- vances in Neural Information Processing Systems, 34, 2021.
Hyvarinen, A. and Morioka, H. Unsupervised feature traction by time-contrastive learning and nonlinear ica, 2016.
Kivva, B., Rajendran, G., Ravikumar, P., and Aragam, B. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701, 2022.
Hyvarinen, A. and Morioka, H. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Hyvärinen, A. and Pajunen, P. Nonlinear independent com- ponent analysis: Existence and uniqueness results. Neural networks, 12(3):429-433 1999.
Klindt, D. A., Schott, L., Sharma, Y., Ustyuzhaninov, I., Brendel, W., Bethge, M., and Paiton, D. Towards nonlin- ear disentanglement in natural data with temporal sparse

---

Learning Vision and Language Concepts for Controllable Image Generation
coding. In International Conference on Learning Repre- sentations, 2021.
Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, L., Shrirao, S., Lambert, C., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-to-image alignment with deep-fusion large language models. arXiv preprint arXiv:2409. 10695, 2024.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference on machine learning, pp. 5338- 5348. PMLR, 2020.
Liu, N., Du, Y., Li, S., Tenenbaum, J. B., and Torralba, A. Unsupervised compositional concepts discovery with text-to-image generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2085-2095, 2023.
Kong, I.., Xie, S., Yao, W., Zheng, Y., Chen, G., Stojanov, P., Akinwande, V., and Zhang, K. Partial disentanglement for domain adaptation. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11455-11422 PMLR, 17-23 Jul 2022. URLhttps: / http///oceedddee ngs ml. www.mmr....r///..mmmmmmmmmmmm. v1 62 a html.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Kong, L., Chen, G., Huang, B., Xing, E. P., Chi, Y., and Zhang, K. Learning discrete concepts in latent hierarchical models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http///pprrrviieeeeeee nwtp forum? id-bo5bUxviHém.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
Kwon, G. and Ye, J. C. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18062-18071, 2022.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. arXiv preprint arXiv:2108.0107. 2021.
Labs, B. F. Flux. https://gitiiei com/ ffuu, 2024.
Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. L.,, Lacoste, A., and Lacoste-Julien, Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint ar axXiv:2401.04890 2024.
Morioka, H. and Hyvarinen, A.
learning: Combining causal discovery and representation learning for multimodal data. In International conference on artificial intelligence and statistics, pp- 3399-3426 PMLR, 2023.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abheel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Morioka, H. and Hyvarinen, A. Causal representation learn- ing made identifiable by grouping of observational vari- ables. In Forty-first International Conference on Machine Learning, 2024. URL https: openreview net/ Forum?id=s1.6v5271
Li, B., Qi, X., Lukasiewicz, T., and Torr, P. Controllable text-to-image generation. Advances in neural information processing systems, 32, 2019.
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 4296 4304, 2024.
Li, Z., Fan, S., Zheng, Y., Ng, I., Xic, S., Chen, G., Dong, X., Cai, R., and Zhang, K. Synergy between sufficient changes and sparse mixing procedure for disentangled representation learning. In The Thirteenth International Conference on Learning Representations, 2025.
Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., and Zhu, J.-Y. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 11, 2023.
Lipman, Y., Chen, R. T., Ben- Nem-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint 2xxiv:2210.02744 2022.
12

---

Learning Vision and Language Concepts for Controllable Image Generation
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2085-2094, 2021.
Shang. C., Zhou, S., Zhang. H., Ni, X., Yang, Y., and Wang. Y. Incremental residual concept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030-11040, 2024.
Shen, Y., Gu, J., Tang, X., and Zhou, B. Interpreting the latent space of gans for semantic face editing. Proceed- ings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 9243-9252, 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Sheynin. S., Polyak. A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., and Taigman, Y. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871-8879, 2024.
Rajendran, G., Buchholz, S., Aragam, B., Schölkopf, B., and Ravikumar, P. Learning interpretable concepts: Unifying causal representation learning and foundation models. arXiv preprint arXivv2402.092230 2024.
Sturma, N., Squires, C., Drton, M., and Uhler, C. Unpaired multi-domain causal representation learning, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612) 06125, 1(2):3, 2022.
Sun, Y., Kong, L., Chen, G., Li, L., Luo, G., Li, Z., Zhang, Y., Zheng, Y., Yang, M., Stojanov, P., et al. Causal repre- sentation learning from multimodal biological observa- tions. arXiv preprint arXiv:2411.06511 2024.
Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., and Cohen-Or, D. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2287-2296, 2021.
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., and Cohen-On D. Designing an encoder for stylegan image manipulation. ACM Transactions Graphics (TOG), 40(4): 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. nighhresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp- 10684-10695, 2022.
Tsaban, L. and Passos, A. Ledits: Real image editing with ddpm inversion and semantic guidance. arXiv preprint arXiv:2307.0052. 2023.
andpattern
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 1921-1930, 2023.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image dif- fusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
von Kügelgen, J., Sharma, Y., Gresele, L.., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F Self-supervised learning with data augmentations prov- ably isolates content from style. arXiv preprint 2rXiv:2106.04619, 2021.
Saharia, C., Chan, W., Saxena, S., Li, I.., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36494, 2022.
von Kügelgen, J., Besserve, M., Wendong, L., Gresele, L., Kekić, A., Bareinboim, E., Blei, D., and Schölkopf, B. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Schölkopf. B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612-634, 2021.
Voynov, A., Aberman, K., and Cohen-Or, D. Sketch- text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk,
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2/ 10.02/ 14, 2021.
Wang, J.., Pu, J., Qi, Z., Guo, J., Ma, Y., Huang, N., Chen, Y., Li, X., and Shan, Y. Taming rectified flow for inversion and editing. arXiv preprint arXiv:24 1.04766, 2024a.
13

---

Learning Vision and Language Concepts for Controllable Image Generation
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint 1rXXi::2409.12191, 2024b.
Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc con- cept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
Zarlenga, M. E., Barbiero, P., Ciravegna, G., Marra, G., Gi- annini, F., Diligenti, M., Precioso, F., Melacci, S., Weller, A., Lio, P., al. Concept embedding models. In NeurlPS 2022-36th Conference Neural Information Processing Systems, 2022.
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume I: Long Papers), pp. 893-911, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference Com- puter Vision, pp. 11975-11986, 2023.
Wu, Z., Lischinski, D., and Shechtman, E. Stylespace analy- sis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12863-12872, 2021.
Zhang, K., Xie, S., Ng. I., and Zheng, Y. Causal repre- sentation learning from multiple distributions: A general setting. arXiv preprint arXiv:2402.05052, 2024.
Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., and Yang, M.-H. Gan inversion: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(3):3121- 3138, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer 2023.
Xie, E., Chen, J., Chen, J., Cai, H., Tang. H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., et al. Sana: Efficient high- resolution image synthesis with linear diffusion trans- formers. arXiv preprint arXiv:2410. 10629, 2024.
Vision, pp.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Xie, S., Kong, L., Gong, M., and Zhang, K. Multi-domain image generation and translation with identifiability guar- antees. In The Eleventh International Conference on Learning Representations, 2023.
Zhao, S., Chen, D., Chen, Y.C.., Bao, J., Hao, S., Yuan, L., and Wong, K.-Y. K. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xu, D., Yao, D., Lachapelle, S., Taslakian, P., von Kügelgen, J., Locatello, F., and Magliacane, S. A sparsity principle for partially observable causal representation learning. arXiv preprini arXiv:2403.088 083335, 2024a.
Zheng, Y. and Zhang, K. Generalizing nonlinear ica be- yond structural sparsity. Advances in Neural Information Processing Systems, 36: 36:13326-1 13355, 2023.
Xu, S., Huang, Y., Pan, J., Ma, Z., and Chai, J. Inversion- free image editing with natural language. Conference on Computer Vision and Pattern Recognition 2024, 2024b.
Zheng, Y., Ng, I., and Zhang, K. On the identifiability of nonlinear ica: Sparsity and beyond. arXiv preprint 22Xx::2206.07511, 2022.
Yao, D., Xu, D., Lachapelle, S., Magliacane. S., Taslakian, P., Martius, G., von Kügelgen, J., and Locatello, Multi -iet causal representation learning with partial observability. In The Twelfih International Conference on Learning Representations, 2024. URL https:// wtw.:rrwwwwn..oo.oo.oo.....m.mm.
Zhu, J., Shen, Y., Zhao, D., and Zhou, B. In-domain gan inversion for real image editing. In European conference on computer vision, pp. 592-608. Springer, 2020.
openreview.
Yao, W. Sun, Y., Ho, A., Sun, C., and Zhang, K. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv: 10.05428, 2021.
Yao, W., Chen, G., and Zhang. K. Learning latent causal dynamics. arXiv preprint arXiv:2202.04433 2022.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672,, 2023.
14

---

Learning Vision and Language Concepts for Controllable Image Generation
A Proof for Theorem 4.4
We present the proof for Theorem 4.4. We duplicate the theorem here for readability.
Condition 4.2 (Visual Concept Identification).
i (Invertibility & Smoothness]: Generating functions g" and g are invertible and g is smooth.
[Smooth and Posilive Densityl: The probability density function of ot is smooth and positive, i.e, p(zl(=²) > 0 is smooth over Z x T
ii [Conditional Independence]: Components z,, are independent given z": p(zz|zz) = II, (=)11=²))
iv [Sufficient Variability]: For any z! 2", there exist 2d(z') + 1 values z² ie., (im) with n 0, 1,... 2d(z)) + such that the 2d(z²) vectors w(z', TT (m)) w(z', z(oo)) are linearly independent, where vector w(z', z²²) is defined as follows:
w(z,, 2²)) = 10log (z/|z")
log p 2 =d(x))12")) 12t)
0² log p (3 = )2
8 log p =daa))l")))
(2)
Oz
0z d(x)
0 )(2)(2)))
Condition 4.3 (Textual Concept Identification).
i [Non-degeneracyy: P l = for all k Ωt for all components =1, IP [= ki] #P { -k2),
k1 * k2-
[No-twins]: Distinct components TT have distinct neighbors: ne(=T) + ne(z?T) ) for m + n.
ii [Maximality]: ]: There is DAG ğ1 resulting from splitring latent variable a into (21, 22) in Gi such that the resultant distribution is Markov w.r.f. Gi and G satisfies ii.
iv (Non-Subset Observed Children]: For any pair 2! and 2f} with + j. one's observed children are not the subset of the other's, ss, Chig-((z))) & Chig- ((/g)
Theorem 4.4 (Atomic Concept Identification.. We assume the generating process in (1). Under Condition 4.2 and Condition 4.3, allain component-wist identifiability of concepts z and zt (Definition 4.1) and the bipartite graph G-+ii up to permutation of component indices.
Proof. The proof consists of two main steps. In Step 1, we take advantage of the observed discrete variable t and its influence over the visual latent variable z' (Condition 4.2-iv) to establish the component-wise identifiability of z This proof technique is adopted in the causal representation learning and nonlinear independent component analysis community (Hyvarinen et al., 2019; 9; Khemakhem et al., 2020a; Kong et al., 2022).
In Step 2, we leverage the identified visual latent components z! to further identify the textual latent variables zT. together with the graph Gt that connects the textual representation (atomic concepts) to the visual representation (atomic concepts). The crux of this step is to treat the z components as observed variables since they can be identified from the image data i, as shown in Step 1, and also utilize the sparsity constraint on the graph G (Condition 4.3-iv).
Step 1. We first introduce Lemma A.1, which we adapt from Kong et al. (2022) by omitting their invariant latent subspace.
Lemma A.1 (Adapted from Kong et al. (2022)). We follow the following data Cata-generattin process
~ prpp(z))) x == :z))
(10)
and make the following assumptions:
Al (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e., Pzlu is smooth and Pa|u 0 over Z and U.
A2 (Conditional independence): Conditioned on u, each 2 is independent of any other 2, for i,j E [n]) i # j. ie.
log (z)u(Z|u) Σx q:(22, u) where qi is the log density of the conditional distribution, i.e., Os = log Pr.uu-
15

---

Learning Vision and Language Concepts for Controllable Image Generation
A3 (Linear independence): For any z 2, there exist (d) values of u, i.e. u, with 0, 1....... ..., 2d(z), such that the 2n vectors w w(z, 11) w(z, Un) are linearly independent, where vector w (z, u) is defined as follows:
① logg (21
Ologp p (Zdiz) u) 8² logp (21 u)
8 log (Ed(e)lu)
w w(z,, u) =
(I))
θ21
Ozn
dz
02
We can achieve component-wise identifiability of the latent variable z.
Lemma A.1 leverages the conditional distribution index for identification. Since have access text which is equivalent to the textual concept variable z" up an invertible map, can simplify the data-generating process in (I) by omitting Rh:
t ~ p(t), 2 ~' p(z i g2(2).
(12)
We observe that this data- mata-generating process is special of Lemma A.1 where we treat the text variable t as the distribution index in Lemma A. As consequence, under Condition 4.2, we can show visual atomic concepts z!, are identifiable, thanks to the distribution variability (i.e., the changes of the conditional distribution p(z'|z' (z²))) resulting from the discrete variable t.
Step Given the identified visual atomic concepts 2/, we can simplify our data- getta-genrrating process as:
T p(z"), zt p(z'|z"))
(13)
where we treat the identified visual concepts 2 as observed variables and view textual concepts zm latent discrete variables with potential statistical dependence. These two sets of variables are connected via a bipartite graph gr which we assume to be sparse (Condition 4.3-iv).
In the following, we present lemma adapted from Kivva et al. (2021) that utilizes the sparse graphical structure of gt to component- wise identify the textual atomic concepts =TT
Definition A.2 (Mixture Oracles (Kivva et al., 2021)). Let x! be set of observed variables and t E Ω C N be the discrete variable that is an invertible function of the latent discrete variable zt E rdd("") The mixture model is defined as [2]] = Σxer IP ft = k] P Tz =
mixture oracle MixOracle(z)) takes [z!] as input and returns the number of components [5)]. the weights IP it = k] and the component P (z's = k] for e Ω.
Lemma A.3 (adapted from Kivva et al. (2021)). We assume the generating process (13). Under Condition 4.3, one can reconstruct the bipartite graph Gi between at and zF, the invertible map g" z + t, and the joint distribution
2 = , =d(zT) -kd(ET) from IP [z]] and MixOracle(z)).
We note that since have access the joint distribution i) and have identified z! component-wise, we derive the MixOracle(z') from p(t, 2²) by marginalizing out components in zf accordingly. Given this equivalent formulation, invoke Lemma A.3 to establish the identifiability of the map gT. from which we can component identify 2m from the observed variable t.
B More Empirical Results
Generation results on PIE-BENCH dataset Ju al. (2024). We also evaluate our method on the PIE-BENCH dataset (Ju al., 2024). As shown in Table our method achieves superior or competitive performance across various metries. In order test the model's capability when given long captions, apply QWEN2.5- PIE-BENCH. The average caption length increases from 8 to 68. As we can see from Table 5, our method achieves better results than the strong baselines SANA and SANA-Fintune.
expand the prompts in
Generation samples for EMU-Edit dataset (Sheynin et al., 2024). the main paper, we present quantitative results on the EMU-Edit dataset. As shown in Fig. 7, our method produces more consistent paired images compared to the baselines FLUX-D and SANA.
16

---

Learning Vision and Language Concepts for Controllable Image Generation
Sampling efficiency. Our method builds upon SANA (Xie al., 2024) and introduces three lightweight networks during training. However, only the textual and concept networks are used during inference for text-to-image generation. One potential concern is the additional computational cost introduced by these new components. To address this, we present comparison of sampling efficiency in Table 6. Despite the added networks, our method is the fastest in generating an image. This efficiency is primarily due to our use of a more compact representation, whereas SANA relies on 300 text tokens during inference.
Ablation samples. In the main paper, we present quantitative comparisons by systematically removing each proposed module. Fig. shows qualitative examples from different model variants. When sparsity regularization is omitted, the generated images exhibit greater inconsistency, indicating that dense connections between textual and visual concepts hinder controllability. Removing the diffusion loss, which serves as the primary objective for training the diffusion model, leads to outputs that lack meaningful structure. Additionally, excluding the KL loss, which regularizes the information contained in the exogenous variable c, results in c encoding excessive details about the input clean image. Consequently. the model ignores the text prompt for denoising, producing images that fail to reflect the input prompt accurately.
Multiple-concepts editing. is also interesting to investigate whether our model supports multi-concept editing, To explore this, we present examples in Fig. 8, showing results where 2, 3, 4 concepts are edited simultaneously. As observed, the baseline SANA often alters the subject's identity or the overall image layout, even when such changes not reflected in the modified prompt. In contrast, our method produces paired images that accurately reflect the intended edits while preserving unrelated attributes, demonstrating stronger control and disentanglement.
Disentanglement samples. In Fig. 9, we present paired samples generated for disentanglement analysis, where only the action or background in the text prompt modified. We observe that both SANA and SANA-Finetune frequently alter the subject's identity, despite changes in the text inputs being limited action or background. In contrast, our method more effectively preserves the animal's identity, demonstrating stronger disentanglement of concepts.
Real-world image editing. While the primary focus of our work is controllable text-to-image generation, it is also important to evaluate the model's ss ability to handle real-world image editing tasks. To this end, we conduct experiments on a subset of the PIE-Bench dataset. The results, shown in Table 7, demonstrate that our method achieves the best or competitive performance across various metrics. Furthermore, as illustrated Fig. 10, our method applies precise and necessary edits to the input image, avoiding excessive or unintended modifications that can lead to visual distortions.
Method
CLIP-I
LPIPS L
CLIP-TY
DINO 1
SD3.5-M
0.822
0.533
0.339
0.565
SD3.5-L
0.824
0.544
0.340
0.516
FLUX-S
0.837
0.536
0.331
0.603
FLUX-D
0.823
0.566
0.328
0.505
SANA
0.839
0.573
0.331
0.593
SANA-Finetues
0.840
0.574
0.334
0.603
ConceptAligner 0.879
0.488
0.331
0.651
Table Evaluation results on controllable text-to-image generation on PIE-Bench dataset. We add SANA-Fintum as an additional baseline method where we apply LoRA to fine-tune SANA our training dataset for comparison. We also include DINO similarity as metric. We can observe that ConceptAligner consistently outperforms or is comparable to the baselines across all metrics. We bold the best performances.
Method SANA
CLIP-I1
LPIPS +
CLIP-T 1
DINO 1
0.788
0.680
0.328
0.406
SANA-Finetur
0.786
0.707
0.329
0.403
Concept.Aligner 0.826
0.596
0.328
0.549
Table 5: Evaluation results on long captions. We use QWEN2.5-Instruertte to expand the prompts in PIE-BENCH. The average caption length increases from to 68. We bold the best performances. ConceptAligner outperforms SANA and SANA-Finetun
17

---

Learning Vision and Language Concepts for Controllable Image Generation
A cat A cat with gold chain and star on its head
A dog lying the grass A dog lying on the grass with ball surrounded by colorful flowers'
Ours
Ours
w.o comprehensive captions
w.o comprehensive captions
w.o sparsity
w.o sparsity
w.o diffusion loss
w.o diffusion loss
W.O KL loss
W.O
Figure 6: Qualitative ablation results. The five rows correspond to ConceptAAligeer ConceptAligner without compre- hensive captions, ConceptAlignee without sparsity, ConceptAlignes without diffusion loss, and ConceptAligner without KI. loss. We observe that ConceptAligner is robust the caption quality degradation. Sparsity effectively minimizes unnecessary changes in each pair. The model cannot follow text prompts without diffusion losses and KL losses.
18

---

Learning Vision and Language Concepts for Controllable Image Generation
Method
Sampling Speed (seconds/image)
SD3.5-M
1.56
SD3.5-L
2.29
FLUX-S
0.63
FLUX-D
4.72
SANA
0.58
ConceptAligner
0.48
Table 6: Sampling efficiency comparison on H100 GPU. Based on SANA, we employ three lightweight networks during training and only textual and concept networks used for rext-to-image generation. Each model contains 6 lightweight attention blocks. To further reduce the training and inference cost, use 64 textual tokens for sampling. Compared to SANA, which employs 300 tokens (0.58 seconds/per sample), our method is the fastest with 0.48 seconds to sample an image.
Setting
Method
CLIP-I f
LPIPS
CLIP-T1
DINO 1
BlendedDiiffusion (Avrahami et al., 2022)
0.837
0.377
0.303
0.562
Pix2pix-zero (Parmar et al., 2023)
0.777
0.434
0.282
0.472
Unpaired Images
Plug-and-Play (Tumanyan et al., 2023)
0.887
0.324
0.301
0.705
PnpInv (Ju al., 2024)
0.892
0.313
0.301
0.720
LEDITS-SDXL (Tsaban & Passos, 2023)
0.878
0.343
0.299
0.701
RF-Inversion (Wang et al., 2024a)
0.906
0,427
0.285
0.737
Fireflow-FLUX (Deng et al., 2024)
0.891
0.316
0.295
0.725
Paired Images
Instruct-Pix2pix (Brooks et al., 2023)
0.878
0.356
0.287
0.717
Unpaired Images
ConceptAlligeer
0.917
0.314
0.288
0.782
Table 7: Evaluation results real-world image editing dataset. We use the source images, source and target prompts in PIE-BENCH dataset. Given source image and source prompt, apply standard diffusion inversion to obtain the initial noise of the diffusion process. We input the noise and the target prompt to produce the edited image. Across all metrics, CinceptAllgner superior or comparable to baseline methods, including those trained on expensive paired editing data. We bold the best performances.
19

---

Learning Vision and Language Concepts for Controllable Image Generation
Some palm trees and other plants are sitting on a highway overpass on a cloudy day. Some palm trees and other plants are sitting on a highway overpass on a cloudy day with a double rainbow in the sky. Concept-Aligner
FLUX-D
SANA
A red and yellow double decker bus on small narrow street.
A Russian icon painting of a red and yellow double decker bus on a small narrow street.
FLUX-D
SANA
ConceptAligner
A woman a black and white striped shirt has her arm reaching out.
A woman in red and white striped shirt has her arm reaching out.
FLUX-D
SANA
ConceptAllinerr
A giraffe walking in front of a lake.
A giraffe walking in front of a lake with a tree next to it.
FLUX-D
SANA
ConceptAlliner
A couple of birds sitting on branches outside.
A couple of birds sitting on branches a cage.
FLUX-D
SANA
ConceptA Aligner
Figure 7: Examples of controllable text-to-image generation comparisons.
20

---

Learning Vision and Language Concepts for Controllable Image Generation
A cat. A cat, wearing crown, smiling.
SANA
ConceptAllinner
A dog, sketch, A dog, watercolor, mouth opening.
SANA
ConceptAllineer
A cartoon panda eating bamboo.
A cartoon panda eating bamboo, flowers around, butterflies around. ConceptAllinerr
SANA
A beach.
A beach, tree, dog, a boat in the ocean, vangogh style.
SANA
ConceptAligner
Figure 8: Multiple concepts editing comparisons. We prompts that differ by multiple concepts to generate paired images. Baseline SANA creates unnecessary changes, whereas ConceptAligner can preserve all desirable information, e.g.. the dog's shape in the second row, the panda's 's pose in the third row, and the beach's 's layout in the last row. ConceptAligner can support simultaneously editing four concepts.
21

---

Learning Vision and Language Concepts for Controllable Image Generation
SANA
SANA-Finetune
ConceptAligner
A dog is jumping.
A dog is sleeping.
A cat is swimming. A cat is jumping.
An octopus is in frozen tunddra. An octopus is in moonlit beach.
Figure 9: Disentanglement evaluation examples. Please find protocol details in Table 3. Baseline methods demonstrate inadequate preservation of subject identity during action or background modifications, our approach successfully maintains subject consistency throughout these transformations.
22

---

Learning Vision and Language Concepts for Controllable Image Generation
Input
Plug-and-Plae
PnpInv
InstructPix2pix
Concept-Alinner
Figure 10: Real-world image editing comparisons. Target prompts: photo of horse and cat standing on rocks near the ocean, red bird standing a branch, orange with flowers top Christmas illustration of a Santa', 's angry face, lion puppy wrapped in blue towel. The experimental details are provided in Table 7. Concept Aligner correctly follows the target prompt while preserving the subject identity.
23


Reviews:
[{"review_id": "vYsmGdsyfL", "reviewer": "Reviewer_Eyu1", "summary": "- This paper addresses the problem of learning atomic multimodal concepts by proving under certain nonparametric assumptions, it is possible to component-wise identify each textual concept and each visual concept. \n\n- Guided by this theory, they propose ConceptAligner, a T2I model that explicitly learns discrete textual concepts and continuous visual concepts with a sparse bipartite graph between them. \n\n- Empirically, the paper shows that ConceptAligner outperforms existing T2I methods on controllability and visual quality metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below and the uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.\n\n**1. Experiments on various domains.** \n\nThank you for your constructive comment. In light of your comment, we have included experiments on the EMU-edit testset [1] in our manuscript and uploaded Table 2.  The EMU-edit testset contains 3589 paired prompts and covers 7 common editing types, including background alteration (background), comprehensive image changes (global), style alteration (style), object removal (remove), object addition (add), localized modifications (local), and color/texture alterations (texture). We also provide generated samples in the uploaded Figure 2.  In addition to the controllable text-to-image generation task,  we also evaluate our method on real-world image editing tasks against image editing baselines in uploaded Table 3 and Figure 3. Across all metrics, our method is superior or comparable to the baselines, showcasing the effectiveness of our framework.\n \n\n**2. “Real-world text-image datasets might violate the `sufficient variability’ condition. Add experiments to show how robust the method is.”**\n\nThanks to your suggestion, we have included in our manuscript and the uploaded Table 4 experiments on short, coarse captions to demonstrate the robustness of our approach.\nIn our main experiments in the submission, we follow our baseline SANA’s [2] protocol to generate the training data: we first randomly sample 2 million short prompts from DiffusionDB [3] to generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL to generate detailed captions. Here, to assess our model’s robustness to the text quality, we replace the detailed captions with original short prompts for model training. We can observe that across all metrics, the text-quality degradation makes negligible impacts on our method, demonstrating its robustness to text quality changes. \n\nTo further address your concern, we have included the following discussion in our revision.  `` In the paper, we give precise sufficient conditions – given adequately diverse data,  we can achieve desirable component-wise identifiability. In general, we would expect this condition to be satisfied – standard text-image datasets (e.g., LAION) contain millions of captions, far exceeding the number of possible visual concepts. Additionally, we may follow existing methods (e.g., [3,4,5]) to employ vision-language models to generate higher-quality captions.\nEven if this condition is not met, oftentimes other natural properties can be leveraged to greatly weaken this condition. For instance, if the generating function $g ^{I}$ is simple or sparse, less variability would be needed to guarantee the identifiability (e.g., [6]). Such sparsity is often encouraged implicitly or explicitly in generative models (e.g., sparse attention patterns). A thorough theoretical investigation into combining these properties is an interesting problem, which we leave as future work.''\n\nPlease let us know if there are any further concerns, and we are more than happy to address them in the following stage.\n\n**References**\n\n[1] Emu Edit: Precise Image Editing via Recognition and Generation Tasks. Sheynin et al. CVPR 2024. \\\n[2] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers. Xie et al. ICLR 2025. \\\n[3] DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-image Generative Models.\" Wang et al. arXiv. \\\n[4] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. Chen et al. ECCV 2024. \\\n[5] Improving Image Generation with Better Captions. Betker et al. Technical report. \\\n[6] Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning. Li et al. ICLR 2025.", "claims_and_evidence": "1. **Claim:** Atomic multimodal concepts can be learned with component-wise identifiability under nonparametric assumptions. **Evidence:** The paper offers a formal theoretical framework (Section 4) culminating in Theorem 4.4\n\n2. **Claim:** ConceptAligner outperforms standard text-to-image baselines in controllable generation. **Evidence:** The paper compares ConceptAligner with Stable Diffusion, FLUX, and SANA, showing better performance quantitively and qualitatively  .", "methods_and_evaluation_criteria": "1. The paper proposes a text network, an image network, a concept network, and a conditional diffusion model rendering out the\nvisual representation to image.\n\n2. The proposed ConceptAligner and other SOTA methods are evaluated with CLIP scores, LPIPS scores, and some qualitative examples.\n\n3. Overall, the proposed methods and evaluation criteria are fairly standard in controllable generation and make sense.", "theoretical_claims": "1. The paper claims under certain invertibility, smoothness, conditional independence, non-degeneracy, and sparsity assumptions, both discrete textual concepts and continuous visual concepts can be identified component-wise from text–image pairs in Theorem 4.4.\n\n2. I found no obvious flaws in the claim and derivations.", "experimental_designs_or_analyses": "1. The paper primarily uses a paired prompt scenario from u et al. (2024) and CLIP/LPIPS scores to measure how well the model modifies certain image attributes and keeps other attributes.\n\n2. The paper also presents a ablation study showing sparse text-to-visual coupling is needed for robust control.\n\n3. However, the paper only show results from one dataset, lacking extensive experiments on various domains.", "relation_to_broader_scientific_literature": "- The paper situates itself at the intersection of multimodal representation learning, causal representation learning, and conditional generation.", "essential_references_not_discussed": "No essential references missed.", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper shows a nonparametric approach that can handle mixed discrete and continuous latent variables.\n2. The generation results is impressive compared to other SOTA models.\n\n**Weaknesses**\n1. The paper only show results from one dataset, lacking extensive experiments on various domains.\n2. Real-world text–image datas might sometimes violate the “sufficient variability” condition. The paper will be more sound if it shows how robust the method is if the textual descriptions are not comprehensive or are repetitive.", "other_comments_or_suggestions": "See previous sections", "questions_for_authors": "See previous sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWoT8nSVQa", "reviewer": "Reviewer_T6TG", "summary": "This paper introduces an Identification Theory for identifying atomic multimodal concepts. Leveraging this theory, the authors apply the method to controllable text-to-image generation. Both qualitative and quantitative evaluations have been conducted to assess the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time and efforts dedicated to reviewing our work. Please find our responses below and the uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.\n\n**1. “It is recommended to incorporate additional metrics that specifically assess disentanglement to provide a more thorough evaluation.”**\n\nThank you for this valuable suggestion. Unfortunately, most existing disentanglement metrics require access to ground truth latent variables (e.g.,  Mutual Information Gap [1]), which makes them not applicable to the text-to-image task where ground-truth latent variables are unavailable. Based on the definition of disentanglement, we design the following evaluation protocol to assess whether interventions on one latent factor would lead to unintended changes in other factors: We prompt ChatGPT to randomly name 10 animals, 10 actions, and 10 backgrounds. For each animal, we fix the animal's identity and edit its action and background, resulting in 200 original & edited pairs. We repeat over 10 random seeds and end up with 2000 pairs in total. For evaluation, we employ QWEN2.5-VL-Instruct7B to examine whether the editing retains the animal identity (subject consistency) and whether the targeted modification is achieved (prompt consistency). As you can see from uploaded Table 5 and Figure 9, our method achieves the highest subject consistency and prompt consistency scores simultaneously against SANA and finetuned SANA, demonstrating its superior disentanglement capability.\n\n\n**2. Details on the training dataset, dataset sizes, domains, and whether SANA is tuned with other components.**\n \nThank you for pointing this out. As you suggest, we’ve included a dedicated section in the appendix to cover these details.\n“SANA employs around 30 million text-to-image paired data to train the model. Unfortunately, the data is not publicly available. Therefore, we follow SANA’s protocol to first generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL for re-captioning. In total, we use 2M text-to-image data for finetuning.”\n\nIn our model, the SANA backbone is fine-tuned via LoRAs. Thanks to your suggestion, we have included results on finetuning SANA via LoRAs on our data. As shown in Table 4, our model maintains the leading margin against SANA-Finetune, showcasing our approach’s effectiveness.\n\n**3. ``The token number for $z ^{T} $ is set to 64. Is this sufficient to faithfully represent text information, especially for long text?’’**\n\nThank you for the insightful question. In light of your question, we have added comparative experiments on long captions. Specifically, we expand the short prompts in PIE-BENCH [2] to longer captions with QWEN2.5-Instruct-32B.  We can see from Table 6 that our method outperforms SANA and SANA-Finetune (which uses a token number of 300) while enjoying faster training and inference, thanks to our compact representation (a token number of 64). \n\n**4. ``Details on the evaluation dataset.’’**\n\nThank you for raising this point. We have included the following details in the appendix. \n\n“We use the benchmark PIE-BENCH [2], which contains 700 editing instructions and corresponding input and output captions. We utilize the input and output captions to generate paired images and measure the similarity of each pair.”\n\n**5. ``More results.’’**\n\nThanks to your feedback, we have included two new sets of evaluation results: 1) In Table 2 and Figure 2, we present additional experiments on the EMU-edit test set [3], which consists of 3,589 paired prompts and encompasses seven common editing types: background alteration (background), comprehensive image changes (global), style modification (style), object removal (remove), object addition (add), localized modifications (local), and color/texture changes (texture). We can observe that our approach either outperforms or is comparable with baseline methods across all metrics, demonstrating its effectiveness. 2) We also added real-image editing results in Table 3 and Figure 3. Our method shows superior performance against image editing baselines.\n\n\n**6. Typos.**\n\nWe have corrected all these typos in our manuscript – thank you so much for the helpful feedback!\n\n\nWe were wondering if we have addressed all your concerns. Please let us know if there is anything we could further discuss – your further feedback would be greatly appreciated.\n\n\n**References**\n\n[1] Isolating sources of disentanglement in variational autoencoders. Chen et al. NeurIPS 2018. \\\n[2] PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code. Ju et al. ICLR 2024. \\\n[3] Emu Edit: Precise Image Editing via Recognition and Generation Tasks. Sheynin et al. CVPR 2024.", "claims_and_evidence": "The details of how the proposed theory is applied to controllable text-to-image generation are inadequately explained, making it difficult to fully assess the validity of the claims. More details are listed in **Experimental Designs Or Analyses**.", "methods_and_evaluation_criteria": "The proposed method aims to learn atomic concepts, but the metrics used do not evaluate the disentanglement among the learned concepts. It is recommended to incorporate additional metrics that specifically assess disentanglement to provide a more thorough evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments related to controllable text-to-image generation are not convincing for several reasons:\n\n1. Essential details such as the dataset used for training, dataset size, and domain are not provided. Additionally, it is unclear whether the SANA is tuned alongside other components.\n2. The token number for $z^T$ is set to 64. Is this sufficient to faithfully represent text information, especially for long text?\n3. Details of the evaluation dataset are not provided.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n1. Identifying atomic concepts for text-to-image models is both interesting and essential, as it enhances user control over the generation process.\n2. The presented results show improved controllability in the text-to-image generation process.\n\nWeaknesses\n\n1. As discussed in the **Experimental Designs and Analyses** section, the experiments lack sufficient detail, undermining the effectiveness of the proposed theory.\n2. Providing more results and metrics would be beneficial.", "other_comments_or_suggestions": "There are several typos and minor issues:\n1. In line 253, Our -> our \n2. An extra space is present in the caption of Figure 2.\n3. The first sentence may be repetitive with the second sentence in line 266.", "questions_for_authors": "My main concerns have been listed in the weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QB4YVBMcPI", "reviewer": "Reviewer_Vot5", "summary": "This paper explores concept learning by extracting interpretable \"atomic concepts\" from multimodal data (images and text) to support tasks like text-to-image (T2I) generation. It frames concept learning as a latent variable identification problem within a graphical model, establishing conditions for component-wise identifiability of atomic concepts under a flexible, nonparametric framework that handles both continuous and discrete modalities. Unlike prior work limited to block-wise identifiability or parametric constraints, the authors introduce ConceptAligner, a T2I model that learns disentangled textual and visual concepts with sparse connections.\n\n## update after rebuttal\nThank you for your response. After reading the rebuttal, I keep my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thorough assessment. Please find the responses below and our uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.\n\n**1. Detailed ablation analyses.**\n\nThank you for your constructive feedback. We have added evaluations of the loss terms in the uploaded Table 4 and Figure 1. The sparsity regularization improves our model across almost all metrics. Without KL regularization, the exogenous variable $\\epsilon$ contains excessive information about the input image, and the model ignores the text information as shown in Figure 1. The diffusion loss is the primary objective for the diffusion model, without which the model experiences negligible updates (Figure 1). \n\n**2. DINO similarities between original and edited images.**\n\nGreat suggestion! We have included the DINO similarity in Table1, 2,3,4,6, and our revision. Our method consistently obtains the highest DINO similarity score across benchmarks.\n\n**3. Additional justifications for conditions.**\n\nThank you for the thoughtful feedback. We have included the following discussion in our revision.\n\nCondition 4.2-i: ``The invertibility ensures the observed variables preserve all latent variables’ information. Otherwise, it would be impossible to recover these latent variables from observed variables.\nPractically, the high dimensionality of images often offers sufficient capacity to hold all information of $z ^{I}$, and human language is often a verbose articulation for concise, abstract textual concepts $ z ^{T} $, which makes this condition feasible.\nThe smoothness allows us to use partial derivatives to establish identifiability, following prior work (Khemakhem et al., 2020a;b).\n\nCondition 4.3-4: “Condition 4.3-4 calls for sparse connections from the textual to the visual concepts. Consider concepts like \"fur\" and \"ears\" when describing a cat. These concepts should affect partially distinct visual features. If every visual feature triggered by \"ears\" was also triggered by \"fur,\" these concepts aren't genuinely atomic and should be restructured.\nTheoretically, Condition 4.3-4 has been adopted in recent work (Kivva et al. 2021) and offers greater flexibility compared to alternatives in prior literature. For instance, prior work [1,2] assumes each $z ^{T} _{i}$ has at least one unique child $z ^{I} _{j}$, which is strictly stronger.\n\nWe acknowledge that we only provide sufficient conditions that show the possibility of learning concepts. We completely agree that some conditions can be weakened. Nevertheless, developing necessary conditions for general problems is much more challenging, and we hope our work provides a foundation that can be iteratively refined by the research community.\n\n**4. Image-editing baselines.**\n\nThank you for the helpful feedback. We have included image-editing baselines and datasets in the uploaded Table 3 and Figure 3. Our approach outperforms baseline approaches, including those trained on expensive paired editing data.\n\n**5. Computation efficiency analysis.**\n\nWe finetune our model (SANA backbone with LoRA) for 10000 steps, which takes around 12 hours on 8H100 GPUs. We provide the inference speed comparison in the uploaded Table 7. Thanks to our compact textual representation (64 vs. 300 tokens in SANA), our method is the fastest inference method (0.48s/image).\n\n**6. Key related works.**\n\nThank you for these valuable references. We have added the following discussion.\n\n“Xu et al. focus on designing attention maps where they replace the target attention map with the source map for a particular word. In contrast, we concentrate on developing superior conditioning representation. These two approaches are complementary and we leave investigating this synergy as future work.’’\n\n``Rajendran et al. formulate concepts as affine subspaces of latent variables and provide identifiability guarantees for these subspaces. In contrast, we directly identify each latent variable, which enables us to directly control atomic aspects.’’\n\n**7. Figure optimization.**\n\nThank you for the helpful feedback. We have re-designed the framework diagram (Figure 2) and annotated Figure 4 and Figure 5 – see uploaded Figure 4, 5, 6.\n\n**8. Additional downstream task comparisons and parameter analyses.**\n\nWe have included additional ablation results as indicated in responses 1, 2, and 4. These results further highlight the advantages of our framework and validate our theoretical insights – thank you for the constructive suggestions!\n\n**9. Dataset Selection rationale.**\n\nWe select PIE-BENCH (Ju et al. 2024) because it covers ten editing types and evenly distributed styles over four categories. This broad coverage can provide a thorough evaluation.\n\nPlease let us know if you have any questions. We would be happy to discuss further!\n\n**References**\n\n[1] A Practical Algorithm for Topic Modeling with Provable Guarantees. Arora et al. ICML 2013. \\\n[2] Identifiable Variational Autoencoders via Sparse Decoding. Moran et al. TMLR.", "claims_and_evidence": "The claims in the paper are supported by quantitative results from ablation studies, such as Table 2 showing performance degradation without sparsity regularization. However, the paper provides only limited qualitative ablation results and lacks broader ablation experiment support. Furthermore, it does not conduct detailed analyses of the impact of individual loss functions (e.g., diffusion loss, KL divergence loss, and sparsity regularization loss), which restricts the comprehensiveness and persuasiveness of the evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the controllable text-to-image (T2I) generation problem:\n\n*Methods*: ConceptAligner’s architecture integrates a text encoder, image network, concept network, and diffusion transformer, aligning with the theoretical framework. The use of sparsity regularization and diffusion loss to enhance identifiability and generation quality is a reasonable design.\n\n*Evaluation Criteria*: CLIP-I, LPIPS, and CLIP-T are standard metrics in T2I research, and testing on the paired prompt dataset from Ju et al. [1] aligns with the task goals. However, the evaluation metrics are relatively limited. For a study focused on image editing, emphasizing visual changes introduced by the method is crucial. For example, using DINO [2] to compute similarity between original and edited images could assess foreground and background consistency, providing a more comprehensive measure of transformation precision and quality. Adding such metrics would significantly enhance the evaluation’s rigor.\n\n[1] Ju, X., Zeng, A., Bian, Y., Liu, S., and Xu, Q. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. In The Twelfth International Conference on Learning Representations, 2024.\n\n[2] Oquab, M., Darcet, T., Moutakanni, T., et al. \"DINOv2: Learning Robust Visual Features without Supervision.\" Transactions on Machine Learning Research (TMLR), 2024.", "theoretical_claims": "I reviewed the proof of Theorem 4.4 (Appendix A). The proof reasons correctly under the assumption that Conditions 4.2 and 4.3 hold, and its logic is sound. However, the paper does not sufficiently justify the validity of each assumption or prove that these conditions hold generally. For instance, Condition 4.2-i (invertibility and smoothness of generating functions) is assumed true without explaining how it is verified in practice; Condition 4.3-4 (non-subset observed children) relies on sparsity but does not demonstrate its necessity. The lack of derivation or empirical validation of these assumptions undermines the credibility of the theoretical claims. I suggest the authors provide additional justification to improve understanding.", "experimental_designs_or_analyses": "I assessed the experimental design in Section 6:\n\n*Design*: The experiments build on SANA [3], comparing against strong baselines like SD3.5-M/L and Flux.1-D/S, with ablation studies validating sparsity’s role.\n\n*Issues*: \n\n1) Lack of Downstream Task Comparisons: If the goal is to support downstream applications (e.g., controllable T2I generation or image editing), benchmarks against existing methods are essential. However, the paper lacks such comparisons, e.g., with recent image editing approaches. Adding relevant benchmark tests would significantly enhance the method’s practical validation.\n\n2) Insufficient Dataset Description: The paper does not detail how the test dataset was processed, how many samples were selected, or how they were applied, limiting the reproducibility and generalizability assessment.\n\n[3] Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., et al. Sana: Efficient highresolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024.", "relation_to_broader_scientific_literature": "The paper relates to prior work in the following areas:\n\n*Concept Learning*: Extends Kong et al.’s [4] single-modality work to multimodal scenarios.\n\n*Causal Representation Learning*: Advances beyond block-wise identifiability by Yao et al. [5] and von Kügelgen et al. [6], aligning with Morioka & Hyvarinen’s [7][8] component-wise approach while relaxing parametric assumptions.\n\n*T2I Generation*: Improves diffusion models by Rombach et al. [9] and ControlGAN by Li et al. [10], enhancing controllability. Its core contribution—component-wise identifiability with sparse multimodal connections—integrates theoretical identifiability from Khemakhem et al. [11] with T2I applications.\n\n[4] Kong, L., Chen, G., Huang, B., et al. \"Learning Discrete Concepts in Latent Hierarchical Models.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.\n\n[5] Yao, D., Xu, D., Lachapelle, S., et al. \"Multi-view Causal Representation Learning with Partial Observability.\" The Twelfth International Conference on Learning Representations, 2024.\n\n[6] von Kügelgen, J., Sharma, Y., Gresele, L., et al. \"Self-supervised Learning with Data Augmentations Provably Isolates Content from Style.\" arXiv preprint arXiv:2106.04619, 2021.\n\n[7] Morioka, H. and Hyvarinen, A. \"Connectivity-contrastive Learning: Combining Causal Discovery and Representation Learning for Multimodal Data.\" International Conference on Artificial Intelligence and Statistics, pp. 3399-3426. PMLR, 2023.\n\n[8] Morioka, H. and Hyvarinen, A. \"Causal Representation Learning Made Identifiable by Grouping of Observational Variables.\" Forty-first International Conference on Machine Learning, 2024.\n\n[9] Rombach, R., Blattmann, A., Lorenz, D., et al. \"High-Resolution Image Synthesis with Latent Diffusion Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.\n\n[10] Li, B., Qi, X., Lukasiewicz, T., and Torr, P. \"Controllable Text-to-Image Generation.\" Advances in Neural Information Processing Systems, 32, 2019.\n\n[11] Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. \"Variational Autoencoders and Nonlinear ICA: A Unifying Framework.\" International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020a.", "essential_references_not_discussed": "The paper omits key related works:\n\n*Image Editing*: Xu et al.’s [12] InfEdit proposes inversion-free natural language image editing, highly relevant to this paper’s controllable generation goals, but is not cited.\n\n*Causal Representation Learning*: Rajendran et al.’s [13] FCRL proposes a shift from causal to concept-based representation learning, directly related to this paper’s theoretical framework, but is not mentioned. Including these would offer a more comprehensive context for the contributions.\n\n[12] Xu, S., Huang, Y., Pan, J., Ma, Z., and Chai, J. \"Inversion-free Image Editing with Natural Language.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.\n\n[13] Rajendran, G., Buchholz, S., Aragam, B., et al. \"From Causal to Concept-Based Representation Learning.\" Advances in Neural Information Processing Systems, 37:101250-101296, 2024.", "other_strengths_and_weaknesses": "The framework diagram (Figure 2) is poorly designed, failing to clearly illustrate the training or inference pipeline. The operation of the concept network (R^C)—how it transforms external information and textual concepts into visual concepts—lacks intuitive explanation, with inputs and outputs not clearly labeled. I suggest optimizing the diagram to improve readability.", "other_comments_or_suggestions": "1. The number of experimental results is insufficient, lacking additional comparative experiments.\n\n2. Figures are difficult to interpret; for instance, the qualitative ablation in Figure 5 lacks labels distinguishing the original image from the one with sparsity regularization. I suggest adding annotations to improve readability.\n\n3. All figures should be optimized for better comprehension of experimental outcomes.", "questions_for_authors": "*Article Structure and Experimental Completeness*: The overall structure feels rushed and incomplete, missing downstream task comparisons and parameter analyses. How would adding these affect the method’s evaluation? \n\n*Dataset Selection Rationale*: Why were the current datasets chosen for testing, and what criteria justify this? The paper lacks explanation.\n\n*Figures*: The figures (e.g., Figures 2 and 5) are hard to understand. How do you plan to improve them for clarity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QlXiwz9DIE", "reviewer": "Reviewer_pEXE", "summary": "This paper explores the theoretical foundations of concept learning for aligning atomic vision and language concepts, with applications in controllable text-to-image (T2I) generation. The authors formulate concept learning as a latent variable identification problem and propose a novel theoretical framework that guarantees component-wise identifiability under nonparametric conditions. The proposed model, ConceptAligner, explicitly disentangles atomic textual and visual concepts and ensures sparse connections between them. The authors demonstrate the effectiveness of ConceptAligner in controllable image generation tasks, showing improved interpretability and controllability compared to state-of-the-art methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. Please see our responses below and our uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.\n\n**1. Human evaluations.**\n\nThank you for the nice suggestion. We have added the human evaluation results to the uploaded Figure 7. The human scores favor our method across all benchmarks.\n\n**2. Typos in Equation (1).**\n\nWe have corrected the typo – thank you!\n\n**3. Capturing statistical dependence and spurious correlation.**\n\nThank you for this thoughtful question. The statistical dependence among textual concepts $z ^{T}$ actually strengthens our framework rather than contradicts our causal goals. To clarify:\n\nRegardless of these dependencies, Theorem 4.4 guarantees we can disentangle individual textual concepts. If $z ^{T} _1$ represents \"beach\" and $z ^{T} _{2}$ represents \"palm tree,\" we can identify them and intervene on \"beach\" without automatically changing \"palm tree\".\n\nAt the same time, concepts may naturally exhibit statistical dependencies (like \"beach\" and \"palm tree\"). Our framework accommodates these dependencies, making it more general than models that impose independence constraints.\n\n**4. Sufficient variability in real-world datasets.**\n\nGreat question! We have included experiments on training our model on short, coarse captions in uploaded Table 4. The negligible performance variation indicates our method’s robustness to the caption quality.\n\nFurther, we have included the following discussion in our revision.\n``In the paper, we give precise sufficient conditions – given adequately diverse data, we can achieve desirable component-wise identifiability. In general, we would expect this condition to be satisfied – standard text-image datasets (e.g., LAION) contain millions of captions, far exceeding the number of possible visual concepts. Additionally, we may follow existing methods (e.g., [1]) to employ vision-language models to generate higher-quality captions.’’\n\n**5. Implementation details.**\n\nThank you for the thoughtful feedback. We have included implementation details in our revision.\n\nTraining data: “We follow baseline SANA’s protocol to first generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL for re-captioning. In total, we use 2M text-to-image data for finetuning.”\n\nEvaluation: “We use PIE-BENCH (Ju et al. 2024) for evaluation, which contains 700 editing instructions and corresponding input and output captions. We utilize the input and output captions to generate paired images and measure the similarity of each pair.”\n\nOriginal prompts: please see examples in uploaded Figure 2.\n\n**6. Comparative experiments & Is “controllable” consistent with prior work?**\n\nPrior work on “controllable T2I” often employs side information (e.g., edges) to control generation [2], whereas we use only text prompts to control atomic concepts. Thus, we didn’t compare with them. Thanks to your feedback, we have made this distinction explicit in our revision. We have included comparative experiments with recent image-editing methods in the uploaded Table 3. Our approach outperforms baseline approaches, including those trained on expensive paired editing data.\n\n**7. Images with multiple concepts modified & scalability of the learned concepts.**\n\nThank you for the insightful question. We’ve included results on simultaneously modifying multiple concepts in the uploaded Figure 8. Our method manages to simultaneously edit up to four concepts, while the baseline quickly loses the subject identity. \n\n**8. Training and inference cost analysis.**\n\nWe finetune our model (SANA backbone with LoRA) for 10000 steps, which takes around 12 hours on 8H100 GPUs. We provide the inference speed comparison in the uploaded Table 7. Thanks to our compact representation (64 vs. 300 tokens in SANA), our method is the fastest inference method (0.48s/image).\n\n**9. “What if the input images come directly from real-world sources?”**\n\nGiven real-world images, we apply diffusion inversion to obtain the initial noise value, as in standard inversion-based models. We then feed the noise and the target prompt into our model for editing. The uploaded Table 3 and Figure 3 demonstrate that on real-world sources, our method is superior to or comparable with the baselines across all metrics.\n\n**10. Online updates for new concepts.**\n\nGreat question – in that case, our model is flexible to incorporate a continuous learning strategy (e.g., dynamic memory expansion [3]) to absorb new concepts while retaining learned ones – thank you for suggesting this interesting direction.\n\nPlease let us know whether your concerns are addressed. Thank you in advance!\n\n**References**\n\n[1] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. Chen et al. ECCV 2024. \\\n[2] Adding Conditional Control to Text-to-Image Diffusion Models. Zhang et al. ICCV 2023. \\\n[3] Online Task-Free Continual Generative and Discriminative Learning via Dynamic Cluster Memory. Ye et al. CVPR 2024.", "claims_and_evidence": "Partially, the experiments may be less convincing to support the claims, please see the following parts.", "methods_and_evaluation_criteria": "Partially, please see the following parts.", "theoretical_claims": "I check the proof and it seems to be correct.", "experimental_designs_or_analyses": "1. More human evaluations could further validate the improvements in interpretability and user control.\n\n2. The paper states that learned text and visual concept interactions are implemented based on the causal graph $G^{t2i}$. However, in P3, line 157, the authors mention that the proposed framework can \"capture statistical dependence.\" This contradicts the goal of causal graph discovery, as statistical dependence may introduce spurious correlations, which are undesirable in causality learning.\n\n3. The model assumes that text descriptions provide sufficient variability for concept identification, which may not always hold in real-world datasets with ambiguous or incomplete captions. Moreover, the paper lacks details of implementation and experiments, including but not limited to the training dataset, comparison of text-based editing results,  the original prompts used for image generation, etc.\n\n4. The comparative experiments may be unfair and less convincing. Specifically, the authors only compare the proposed model against standard text-to-image generation models but not against existing controllable text-to-image generation and image editing models. Moreover, is the controllable defined in this paper consistent with prior work on controllable T2I models?\n\n5. The generated or edited images are not presented when modifying multiple concepts.\n\n6.The paper lacks a detailed illustration of the learned concepts. While it shows text-to-image generation results after modifying a single concept, the atomicity among any pairwise concept is neither well-demonstrated nor validated. Additionally, how scalable is the learned concept? Can the model support online updates when encountering new concepts?", "relation_to_broader_scientific_literature": "This paper aims to disentangle atomic concepts in text-to-image generation for enhanced interpretability and controllability.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "More human evaluations could further validate the improvements in interpretability and user control.\n\nThe causal relationship in Equation (1) appears inconsistent with the description in Figure 1. Specifically, it seems that the correct formulation should be i = g^I(z^I) and vice versa.\n\nThe paper states that learned text and visual concept interactions are implemented based on the causal graph G^{t2i}. However, in P3, line 157, the authors mention that the proposed framework can \"capture statistical dependence.\" This contradicts the goal of causal graph discovery, as statistical dependence may introduce spurious correlations, which are undesirable in causality learning.\n\nThe model assumes that text descriptions provide sufficient variability for concept identification, which may not always hold in real-world datasets with ambiguous or incomplete captions. Moreover, the paper lacks details of implementation and experiments, including but not limited to the training dataset, comparison of text-based editing results,  the original prompts used for image generation, etc.\n\nThe comparative experiments may be unfair and less convincing. Specifically, the authors only compare the proposed model against standard text-to-image generation models but not against existing controllable text-to-image generation and image editing models. Moreover, is the controllable defined in this paper consistent with prior work on controllable T2I models?\n\nThe generated or edited images are not presented when modifying multiple concepts.\n\nConceptAligner introduces additional computational complexity compared to conventional T2I models. The authors should report the training and inference costs of the proposed framework.\n\nIn Section 5.1, the authors state: \"For text-based editing, we simply reuse the exogenous information ϵ of the previously generated image i.\" However, what if the input images come directly from real-world sources rather than being generated by the proposed model?\n\nThe paper lacks a detailed illustration of the learned concepts. While it shows text-to-image generation results after modifying a single concept, the atomicity among any pairwise concept is neither well-demonstrated nor validated. Additionally, how scalable is the learned concepts? Can the model support online updates when encountering new concepts?", "other_comments_or_suggestions": "Please see the above.", "questions_for_authors": "Please see the above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of learning atomic multimodal concepts. The final recommendations are 1 "Weak reject" and 3 "Weak accept". The reviewers generally agree that
- the problem is interesting and important,
- the proof of the proposed theory for identifying atomic multimodal concepts sounds,
- the ConceptAligner outperforms exisiting T2I methods on controllability

Their major concerns include
- limited evaluation and limted evalution metrics,
- insufficient description of the dataset and experiments,
- poor illurations and explanations,
- lack of a fair comparison with controllable image generation methods,
- validity of each assumption has not been sufficiently justified,
- robustness unclear when assumptions are not met

In their rebuttal, the authors have made great effort to address the reviewers concerns by providing additional experimental results (in terms of more baselines, dataset, and evaluation metrics) and improved over their initial submission. I recommend "Weak accept".
Decision: Accept (poster)