Title: Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion model; Emergence; Emergent capabilities; Science of deep learning; Mechanistic interpretability
Abstract: Modern generative models exhibit unprecedented capabilities to generate extremely realistic data. However, given the inherent compositionality of the real world, reliable use of these models in practical applications requires that they exhibit the capability to compose a novel set of concepts to generate outputs not seen in the training data set. Prior work demonstrates that recent diffusion models do exhibit intriguing compositional generalization abilities, but also fail unpredictably. Motivated by this, we perform a controlled study for understanding compositional generalization in conditional diffusion models in a synthetic setting, varying different attributes of the training data and measuring the model's ability to generate samples out-of-distribution. Our results show: (i) the order in which the ability to generate samples from a concept and compose them emerges is governed by the structure of the underlying data-generating process; (ii) performance on compositional tasks exhibits a sudden "emergence" due to multiplicative reliance on the performance of constituent tasks, partially explaining emergent phenomena seen in generative models; and (iii) composing concepts with lower frequency in the training data to generate out-of-distribution samples requires considerably more optimization steps compared to generating in-distribution samples. Overall, our study lays a foundation for understanding emergent capabilities and compositionality in generative models from a data-centric perspective.

Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task

Abstract

Modern generative models exhibit unprecedented capabilities to generate extremely realistic data. However, given the inherent compositionality of the real world, reliable use of these models in practical applications requires that they exhibit the capability to compose a novel set of concepts to generate outputs not seen in the training data set. Prior work demonstrates that recent diffusion models do exhibit intriguing compositional generalization abilities, but also fail unpredictably. Motivated by this, we perform a controlled study for understanding compositional generalization in conditional diffusion models in a synthetic setting, varying different attributes of the training data and measuring the model's ability to generate samples out-of-distribution. Our results show: (i) the order in which the ability to generate samples from a concept and compose them emerges is governed by the structure of the underlying data-generating process; (ii) performance on compositional tasks exhibits a sudden 'emergence' due to multiplicative reliance on the performance of constituent tasks, partially explaining emergent phenomena seen in generative models; and (iii) composing concepts with lower frequency in the training data to generate out-of-distribution samples requires considerably more optimization steps compared to generating in-distribution samples. Overall, our study lays a foundation for understanding emergent capabilities and compositionality in generative models from a data-centric perspective.

1 Introduction

The scaling of data, models, and computation has unleashed powerful capabilities in generative models, enabling controllable synthesis of realistic images [1-11], 3D scenes [12-16], videos [1725], accurate image-editing [26-31], and semantically coherent text generation [32-46]. With increased interest in incorporating these models in day-to-day applications [47-50], e.g., to improve robotic systems via better planning and grounding [51-59], analyzing and improving their reliability has become crucial.

With the motivation above, we study compositional generalization abilities of conditional diffusion models, i.e., diffusion models that are conditioned on auxiliary inputs to control their generated images (e.g., text-conditioned diffusion models [6, 26]). Given the inherent compositionality of the real world, it is difficult to create a training dataset that exposes a model to all combinations of different concepts underlying the data-generating process. We therefore argue that compositional generalization is central to model reliability in out-of-distribution scenarios, i.e., when the model has to compose a novel set of concepts to generate outputs not seen in the training data set [60, 61]. As shown by several prior works investigating the compositional generalization capabilities of off-theshelf text-conditioned diffusion models [62-72], modern diffusion models often compose complicated

"White-colored lizard""Green-colored lizard"

"White-colored goldfish""Green-colored goldfish"

"White-colored panda""Green-colored panda"

"Blue-colored lizard""Magenta-colored lizard"

"Blue-colored panda""Magenta-colored panda"

concepts, producing entirely non-existent objects, but also unpredictably fail to compose apparently similarly complicated concepts (see Fig. 1). It remains unclear why models produce novel samples by composing some concepts but not others. Indeed, what concepts does a model learn to compose? What differentiates these concepts from the ones the model is unable to compose?

'Model Experimental Systems' Approach with Interpretable Synthetic Data. To address the above questions, we seek to take a systematic approach to understanding complex systems, such as generative models, by dividing the process into hypothesis generation and testing. Specifically, drawing inspiration from the natural sciences, we adopt a model-experimental systems approach , which involves studying simplified, controllable, and steerable experimental systems to develop mechanistic hypotheses applicable to more complex systems. For instance, in neuroscience research, model organisms like mice or fruit flies are used to study neural mechanisms and develop hypotheses that can be tested on the larger-scale human brain [74-76]. Similarly, in our work, we design a synthetic experimental setup that pursues simplicity and controllability while preserving the essence of the phenomenon of interest, i.e., compositional generalization. Specifically, our data-generating process tries to mimic the structure of the training data used in text-conditioned diffusion models by developing pairs of images representing geometric objects and tuples that denote which concepts are involved in the formation of a given image (see Fig. 2). We train diffusion models on synthetic datasets sampled from this data-generating process,

Test

(1,

a

)

(2,

a

)

(1,

b

)

Image

Tuple

?

(2,

b

)

(1,

a

)

(2,

a

)

(1,

b

)

: circle,

: triangle

1

2

Shape

: red,

: blue

a

b

Color

(a)

(c)

(b)

conditioning the model on tuples denoting which concepts an object in the image should possess, while systematically controlling the frequencies of concepts in the dataset. Thereafter, we study the model's ability to generate samples corresponding to novel combinations of concepts by conditioning the denoising process on a correspondingly novel tuple, thus assessing the model's ability to compositionally generalize. This approach allows us to systematically investigate the key properties of a dataset enabling compositional generalization in an interpretable and controlled manner in conditioned diffusion models. Our main contributions follow.

Concept graphs: A simple, synthetic framework for studying conditional diffusion models. We develop a minimalistic abstraction for training conditional diffusion models by introducing a framework called 'concept graphs'. The framework forms the foundation of our systematic

study of how diffusion models learn to generate samples from a set of concepts and compose these concepts to generate out-of-distribution samples. To our knowledge, our results are the first demonstration of perfect compositional generalization on a (synthetic) vision problem.

Multiplicative emergence of compositional abilities. We use our model system's interpretability to monitor learning dynamics of a diffusion model's capabilities and the ability to compose them to produce out-of-distribution data. As we show, a diffusion model first memorizes the training dataset and then sequentially generalizes to concepts that are at a greater 'distance' from the training distribution. Since progress in learning each capability multiplicatively affects the model's performance in compositional generalization, we find a sudden ability to compose and produce samples out-of-distribution 'emerges'. We thus hypothesize compositionality may partially explain emergent phenomena seen in modern generative models [77-81].

Investigating challenges for compositional generalization. We systematically examine the challenges arising from decreased frequency of specific concepts necessary for learning a corresponding capability. We further evaluate the effectiveness of using fine-tuning to address misgeneralizations in adversarial settings, and find that it is generally insufficient to enable the learning of new capabilities.

Before proceeding, we emphasize the trade-offs and limitations of our model-experimental systems approach. Just as neural mechanisms identified in model animals cannot be directly applied to human medical applications, our observations should not be considered definitive conclusions that can be directly transferred to modern large generative models. Instead, our study aims to establish conceptual frameworks, identify data-centric control variables, and formulate mechanistic hypotheses, paving the way for further theoretical and empirical explorations of larger models.

2 Related Work

Diffusion models. Diffusion models are the state-of-the-art method for generating realistic visual data [1-11]. They are often trained using image-text pairs, where the text is processed using a large language model to produce semantically rich embeddings that allow controlled generation of novel images and editing of existing images [26-31]. Such conditional diffusion models are easy to test for compositional generalization, as one can directly give a text description that requires composition of concepts the model is likely to have learned (e.g., avocado and chair) to produce images that were not encountered in the model's training data (e.g., avocado chair; see [1, 6]). Such results demonstrate the model's ability to compose and generalize out-of-distribution. However, we emphasize that text-conditioned models may fail to generalize compositionally if the text model is unable to sufficiently disentangle relevant concepts in the text-embedding space. To avoid this pitfall, we use ordered tuples that denote precisely which concepts are involved in an image's composition.

Compositional generalization. Compositionality is an inherent property of the real world [82], wherein some primitive such as color can be composed with another primitive such as shape to develop or reason about entirely novel concepts that were not previously witnessed [60]. Notably, compositionality is hypothesized to play an integral role in human cognition, enabling humans to operate in novel scenarios [83-87]. Inspired by this argument, several works have focused on developing [67, 68, 88-93] and benchmarking [64, 66, 90, 94-102] machine learning models to improve and analyze compositional generalization abilities. We especially highlight the works of Lewis et al. [96], who use a similar model experimental system approach as ours to evaluate factors influencing compositionality in contrastive language-image pretraining (CLIP) [34]. Our focus on generative models, instead of representation learning, distinguishes our results however. Relatedly, we note the notion of compositionality has been discussed in several recent works in the fields of disentangled and causal representation learning [101, 103-107]. Our framework for systematic understanding compositionality in generative models was heavily influenced by the synthetic dataset design considerations promoted in these papers. We also emphasize that a thorough formalization of compositionality in modern machine learning is generally lacking, but highlight the notable exceptions by Hupkes et al. [108] and Wiedemer et al. [106]. To avoid ambiguity, we provide a precise formalization that instantiates our intended meaning of compositionality.

3 Concept Graph: A Minimalistic Framework for Compositionality

In this section, we present the concept graph framework, as illustrated in Fig. 3, which enables us to visually depict compositional structure of our synthetic data and forms the basis for generating

hypotheses and designing experiments in our work. We begin by defining the essential building blocks of our framework: concept variables and concept values. In the following, we call a specific output of our data-generating process an 'object' (see Fig. 3).

Definition 1. (Concept Variables.) A concept variable v i uniquely characterizes a specific property of an object. V = { v 1 , v 2 , ..., v n } denotes a set of n concept variables, such that, given an object x , v i ( x ) returns the value of the i th concept variable in that object.

For instance, for geometric objects shown in Fig. 3, concept variables may include shape , color , size , location , and angle . These variables take on values from a pre-specified range, called concept values, as described below.

Definition 2. (Concept Values.) For each concept variable v i ∈ V , C i = { c i 1 , c i 2 , ..., c ik i } denotes the set of k i possible values that v i can take. Each element of C i is called a concept value.

For example, in Fig. 3, the concept values for the shape variable are in the set { circle , triangle , square } , while for the color values are in the set { red , blue , green } . Given n concept variables, with each variable v i having k i concept values, there can be ∏ n i =1 k i distinct combinations that yield a valid object. We use a unique combination of a subset of these concept variables, v 1 , . . . , v p ( p < n ), to define the notion of a 'concept class'.

Definition 3. (Concept Class.) A concept class C is an ordered tuple ( v 1 = c 1 , v 2 = c 2 , ..., v p = c p ) , where each c i ∈ C i is a concept value corresponding to the concept variable v i . If an object x belongs to concept class C , then v i ( x ) = c i ∀ i ∈ 1 , . . . , p .

Note that the remaining n -p concept variables are free in the above definition. When these free variables are specified, we get a specific object from our data-generating process. That is, a concept class represents a family of objects by specifying the values of a pre-defined subset of concept variables. For example, in Fig. 1, different colored lizards instantiate images (objects) from the species (concept class) of lizards; here, color of the lizard serves as a free variable. Similarly, in the geometric objects scenario of Fig. 3, a 'small red circle' would be a concept class for which the shape , color , and size variables have been assigned specific values. Objects are images designated by associating precise values with the remaining concept variables of location and angle . Next, we introduce the notion of concept distance, which serves as a proxy to succinctly describe the dissimilarity between two concept classes.

̸

Definition 4. (Concept Distance.) Given two concept classes C (1) = ( c (1) 1 , c (1) 2 , ..., c (1) n ) and C (2) = ( c (2) 1 , c (2) 2 , ..., c (2) n ) , the concept distance d ( C (1) , C (2) ) is defined as the number of elements that differ between the two concept classes: d ( C (1) , C (2) ) = ∑ n i =1 I ( c (1) i , c (2) i ) , where I ( c 1 i , c 2 i ) = 1 if c 1 i = c 2 i and I ( c 1 i , c 2 i ) = 0 otherwise.

ConceptVariables

ConceptClass

Objects

ConceptGraph

shape=0:circle

011

111

size=O:small

shape

color=1:blue

001

101

color

size

location

angle

010

110

shape=1:triangle

size=0:large

000

100

color=0:red

The concept distance quantifies the dissimilarity between two concept classes by counting the number of differing concept values. It is important to note that this distance serves only as a null model because each axis represents distinct concept variables and each of the variables can assume various possible concept values. We are now ready to define the notion of a concept graph (see Fig. 3), which provides a visual representation of the relationships among different concept classes.

Definition 5. (Concept Graph.) A concept graph G = ( N,E ) consists of nodes and edges, where each node n ∈ N corresponds to a concept class, and an edge e ∈ E connects two nodes n 1 and n 2 representing concept classes C (1) and C (2) , respectively, if the concept distance between the two concept classes is 1, i.e., d ( C (1) , C (2) ) = 1 .

A concept graph organizes different concept classes as nodes in the graph, while edges denote pairs of concept classes that differ by a single concept value. An ideal conditional diffusion model, when trained on a subset of the nodes from this graph, should learn capabilities that allow it to produce objects from other concept classes. We formalize this as follows.

Definition 6. (Capability and Compositionality.) Consider a diffusion model trained to generate samples from concept classes ˆ C = { C 1 , C 2 , . . . , C T } . We define a capability as the ability to alter the value of a concept variable v i to a desired value c i . We say the model compositionally generalizes if it can generate samples from a class ˜ C such that d ( ˜ C,C i ) ≥ 1 ∀ C i ∈ ˆ C .

The ideas above are best explained via Fig. 4. Specifically, assume we train a diffusion model on data from a subset of concept classes, i.e., a subset of nodes in the concept graph. To fit the training data, the model might just memorize the training data or it may learn the relevant capabilities to alter specific concept variables; e.g., given samples from classes 0000 , 0010 , and 0001 in Fig. 4 (a), the model may just memorize the data or learn how to alter the third and fourth concept variables. Models that just memorize the training data lack the capability to generate samples from out-of-distribution classes (e.g., 1100 ). In contrast, the ability to compose concepts would enable the model to produce out-of-distribution samples starting from classes seen in the training data.

In summary, our concept graph framework provides a systematic approach to representing and understanding a minimalistic compositional structure, allowing for an analysis and comparison of different learning algorithms' abilities to generalize across various concept classes.

3.1 Experimental and Evaluation Setup

With the concept graph framework in place, we now have the tools to systematically investigate the impact of different data-centric properties on learning and composing capabilities in conditional diffusion models. Before proceeding further, we briefly define our experimental setup and evaluation protocol. This allows us to interleave our results with our hypotheses, explaining them in context.

Experimental Setup. The detailed setup is presented in Appendix A. In brief, we train conditional diffusion models that follow the U-Net pipeline proposed by Dhariwal and Nichol [1]. Our dataset involves concept classes defined using three concept variables, each with two values; specifically, shape = {circle, triangle}, color = {red, blue}, and size = {large, small}. Tuples with binary elements are used to condition the diffusion model's training; they serve as an abstraction of text conditioning. For example, the tuple 000 implies a large, red circle is present in the image. To sample images from this process, we simply map the size and color axes to the range [0 , 1] and uniformly sample to develop a training dataset of 5000 samples; the precise samples depend on which concept classes are allowed in the data-generating process. In this setup, the minimal required set for learning capabilities to alter concepts is four pairs of tuples

and images, each drawn from one of the following four concept classes: { 000 , ( circle , red , large ) } , { 100 , ( triangle , red , large ) } , { 010 , ( circle , blue , large ) } , { 001 , ( circle , blue , small ) } . Consider the following two tuples: { 000 , ( circle , red , large ) } and { 100 , ( triangle , red , large ) } . In this case, we see that the shape concept is encoded in the first elements. Here, 0 represents a circle and 1 represents a triangle. The remaining concepts are similarly encoded in later elements.

Evaluation Metric. Evaluating whether a generated image corresponds to the desired concept class might require a human in the loop. To eliminate this requirement, we follow methods from literature on disentanglement and train classifier probes to predict whether a generated image possesses some property of interest [103, 109-114]. Specifically, we use the diffusion model training data to train three linear probes, which respectively infer the following three concept variables: shape , color , and size . We define a model's accuracy for generating images of a given concept class as the product of the probabilities outputted by the three probes that each concept variable matches the value for the concept class. Note that a random classifier for a concept variable will have an accuracy of 0.5. We indicate this random baseline with dotted, gray lines in our plots when useful.

4 Multiplicative Emergence of Compositional Abilities

We first investigate the order in which a model learn capabilities to produce sample from a concept class and how the learning of such capabilities affects the ability to compositionally generalize.

Learning dynamics respect the structure of the concept graph (Fig. 5). We find the ability to compositionally generalize, i.e., produce samples from out-of-distribution concept classes, emerges at a rate which is inversely related to a class's concept distance with respect to classes seen in training. Specifically, in Fig. 5 (a) we show the learning dynamics of the model, where lightblue nodes denote concept classes within the training dataset, while pink and darkpink nodes respectively denote classes at concept distances of 1 and 2 from classes in the training dataset. As the model learns to fit its training data (lightblue nodes), it learns capabilities that can be composed to produce samples from out-of-distribution concept classes (pink / darkpink nodes). Fig. 5 (b) further shows that the learning dynamics of compositional generalization are influenced by the concept distance from the training set: the model first learns the concept classes in the training dataset (lightblue lines) and then generalizes to concept classes with a concept distance of 1 from the training dataset (pink lines). Thereafter, the model suddenly acquires the capability to compositionally generalize to a concept class with a concept distance of 2 from the training dataset (darkpink line). Fig. 5 (c) shows the images generated by the model over time. We observe that rough shapes and sizes are learned relatively early in training, by the 4th epoch, while the color is highly biased to be red, the majority color in the training dataset, up to the 10th epoch. Then, around the 20th epoch, the model learns to generate the underrepresented

(a)

(0,1,1)

(1,1,1)

(b)

(c)

(0,0,1)

(1,0,1)

100 1

80

60

Color

(0,1,0)

0

001

40

010

(1,1,0)

100

011

Size

20

101

110▲

111 ▲

(0,0,0) Shape

(1,0,0)

102

OptimizationSteps

103

104

0

4

Number of epochs

6

10

20

40

60

color (blue) for concept classes with a concept distance of 1. Finally, around the 40th epoch, the model learns to generate the underrepresented color (blue) for the class at distance 2, showing a sudden emergence of capability to generate samples from that class. The observations above generalize to higherdimensional concept graphs as well (see Fig. 16). Specifically, we now introduce a fourth concept variable backgroundcolor ={white, black} and again find that compositional generalization occurs in the order governed by the distance from the training set: the accuracy begins to rise for concept distance of 1, continues to increase for distance 2, and peaks for 3.

Multiplicative influence of capabilities drives the sudden emergence of compositional generalization (Fig. 6). The interpretability of our experimental setup illustrates that the multiplicative reliance on underlying capabilities is the critical structure behind the sudden emergence of compositional generalization in our setup. For example, in Fig. 6 (a) we show the dynamics of an additive vs. multiplicative accuracy measure for generating concept class {111, (triangle, small, blue)}, which has a concept distance of 2 from the training data. We observe a sudden improvement of the multiplicative measure. To better understand the above, in Fig. 6 (b), we plot the accuracy of probes used for predicting each concept variable (shape, size, color). From the plot, we notice

(a)100

Multiplicative

Additive

50

111

102

103

104

00(q)

Size

Shape

Color

50

0

102

103

104

Optimization Steps

that the model fails to acquire strong color transformation capabilities until the final stage of training, effectively bottlenecking compositional generalization of 111 class. This empirical observation leads us to the following hypothesis on emergence of compositional abilities in generative models .

Hypothesis. (Compositional Abilities Emerge Multiplicatively.) We hypothesize the nonlinear increase in capability observed in large neural networks as size and computational power scale up is partially driven by the task's compositionality. Models must learn all required concepts, but compositional generalization is hindered by the multiplicative, not additive, impact of learning progress on each concept. This results in a rather sudden emergence of capabilities to produce or reason about out-of-distribution data.

The core aspect worth noting in our hypothesis is that compositional tasks require the concurrent acquisition of all involved 'atomic' capabilities, akin to an AND logical condition. Correspondingly, in generative models, the learning of individual atomic abilities leads to a manifestation of emergent capabilities 1 . We argue this requirement leads to a non-linear increase in a model's capabilities (see also concurrent work making similar claims [78, 117]). We analyze this further via a simple toy model below.

Toy analysis. We consider a toy compositional problem which demonstrates how complex capabilities can rapidly develop ('emerge') from a set of atomic abilities. Assume there are n atomic abilities, each with a probability p of being learned in a given time step, i.e., the dynamics of learning an ability can be modeled as a Bernoulli coin flip: once the coin turns from 0 to 1 , the model learns the said ability. The probability that the ability will be learned in t steps is thus given by 1 -(1 -p ) t . An implicit assumption in this model is that the learning dynamics of

(a)

n

=1

n

=2

n

=3

n

=20

P *

=0.95

(b)

atomic abilities are independent (hence the name atomic) and that once an atomic ability is learned, it will not be forgotten. Now, our goal is to characterize the dynamics of learning a capability that involves composing these atomic abilities. The multiplicative emergence hypothesis argues that the learning such a capability by time t requires that the model have learned the n atomic abilities by that time. We then get the probability that the compositional capability has been learned by time t is P ( n ) = (1 -(1 -p ) t ) n . These dynamics are plotted in Fig. 7 (a) and look strikingly similar to the rapid learning (emergence) we found in our diffusion model setup (see Fig. 5 b)): for increasing degree of compositionality, we see much sharper learning dynamics (i.e., emergence) that is controlled by the concept distance w.r.t. atomic abilities. Further, assuming a given threshold probability P ∗ at which one claims a capability has been learned, for a degree of composition n , we can solve for

∗

∗

⌈

log(1

-

(

P

log(1

the critical time

t

at which a compositional capability is learned as follows:

t

=

∗

-

)

p

1

)

.

Plotting this in Fig. 7 (b), we see the progress on increasingly more compositional tasks is logarithmic w.r.t. the number of atomic concepts being composed (aka concept distance). Importantly, this implies if we allow the model to train infinitely and it learns several atomic abilities, it will have an explosion of capabilities due to the inherent compositionality of the data generating process-we further investigate this in a follow up work [118].

4.1 Challenges for Compositional Generalization

We have shown that, given a well-structured synthetic training dataset, a conditional diffusion model can learn to compose its capabilities to generate novel inputs not encountered in the training set. Next, we analyze an adversarial setups under which the model fails to learn relevant capabilities to compose concepts and if simple interventions can help mitigate these failures.

Critical frequency for learning capabilities (Fig. 8). We first probe the effect of changing the frequencies of samples from different concept classes in the training data and examine how this affects the model's ability to learn and compose capabilities involving that concept class. Results are shown in Fig. 8 and demonstrate how the frequency of color and size concepts in the training data impacts the generalization capabilities of the diffusion model. Specifically, we change the number of samples in the training data from 0 to 300 for concept class 001 (Fig. 8 (a)) and class 100 (Fig. 8 (b)). As can be seen, low frequencies of certain concept values degrade the accuracy of the model in both settings. Notably, training for out-of-distribution concept classes (pink lines) requires more samples than that for in-distribution ones (lightblue lines). This suggests that as the sample size grows, memorization occurs first, and generalization is achieved superlinearly as a function of data frequency. More importantly, we observe a critical number of samples are required before we can see the onset of capabilities to alter a concept. Specifically, in Fig. 8 (a), we can see that the model rapidly learns the color concept after being provided with 10 samples for a concept of large blue circle, 001 . In contrast, in Fig. 8 (b), the model learns the shape concept only after reaching a certain threshold in the number of samples with a concept of large red triangle, 100 .

We believe the results above are especially interesting because an often used strategy to prevent a generative model from learning harmful capabilities, such as the ability to generate images involving sensitive concepts, involves cleaning the dataset to remove images corresponding to such concepts, hopefully hindering the model's ability to generate samples containing the concept [33, 119, 120].

(a)

100↓

(b)

100

80

80

001

011

101

111

40

60

20

100

010

001

011

101

000

20

6

40

000

010

100

001

011

101

010

110

111

110

0

111

110

0

100

200

300

0

100

200

300

000

100

Number of samples for class 001

Numberofsamplesforclass1oo

/n

)

⌉

Fine-tuning + node '101'

Adversarial Task

(a)

(b)

000

100

001

101

011

111

110

010

+ node '010'

+ node '011'

Such filtering is generally expensive and, arguably, statistically impossible to achieve perfectly in a large dataset, i.e., a few samples corresponding to the sensitive concept are likely to remain in the data. Our findings suggest that one might better consider the relationship between the probability of learning a concept and its frequency of occurence in the data to determine the degree of filtering required: if their presence in the training data is below a critical threshold, that can be sufficient to deter the model from learning a capability to generate samples related to that concept, obviating the need for a perfect filtering of the dataset.

Fine-tuning is not a remedy for misgeneralization in an adversarial task. (Fig. 9). We next evaluate a setting where concept classes present in the training data are not neighbors, i.e., their concept distances are greater than 1, but nonetheless represent all concept variables to allow a model to learn relevant capabilities. Specifically, as shown in Fig. 9 (a), we use only the following four pairs of tuples and images for training: { 000 , ( circle , red , large ) } , { 100 , ( triangle , red , large ) } , { 001 , ( circle , blue , small ) } , { 111 , ( triangle , blue , small ) } . Arguably, capabilities corresponding to shape change ( 000 to 100 ) and color change ( 000 to 001 ) can still be learned as the setup for these is similar to our prior experiments. However, our results show this yields an interesting failure mode; specifically, in Fig. 9 (a) we find the model fails to dissociate a second element value of 1 with the shape of a small triangle, and this bias causes the model to generate small triangles for both 011 and 010 , when it should have produced small circles.

We further analyze if the failure above can be fixed via mere downstream fine-tuning (see Fig. 9 (b)). Specifically, we fine-tune the trained model on a dataset that includes the concept class of 101 (left), 010 (middle), and 011 (right). We find the model still generates small triangles for 011 and 010 even after fine-tuning. When the concept classes 010 (middle) and 011 (right) are added, the newly introduced concepts 'overwrite' all existing concepts, causing the previously learned concepts (e.g., the color and shape concept for 101 ) to be forgotten. These results present a further challenge in addressing learned bias.

4.2 Additional Experimental Results with Real Data

To validate our hypotheses in more realistic settings, we conducted experiments using the CelebA dataset. We selected three attributes as the concepts: Gender (categorized as Female and "Male"), Smiling ("Smiling" and Not Smiling), and Hair Color ("Black Hair" and "Blonde Hair"). The concept graph corresponding to these attributes in the CelebA dataset is illustrated in Fig. 10 (a). For each individual concept, we trained a 3-layer CNN and adopted the product of accuracies from each concept as the evaluation metric. Although not all nodes achieved an accuracy of 100% due to limited training time, we find that our observations (partially) extend: (i) learning dynamics following the structure of the concept and (ii) delayed generalization of underrepresented attribute (gender), hold true, even at larger, more realistic scales. Fig. 10 (b) illustrates the learning dynamics using the real CelebA dataset. We again observe the sequential pattern in generalization. The accuracy begins to rise at concept distance 1 (denoted by pink lines), followed by concept distance 2 (red line). Notably, the node labeled 011 -even though it is at concept distance 1-precedes the memorization stage (represented by blue lines). This exception can be attributed to the gender bias present in the training data. In Fig. 10 (c), we plotted the accuracy for the individual concept of gender (i.e., Female and Male). The Female concept class's training curve (red lines) reaches convergence faster

(a)

(b)

100

(c)

100

80

80

60

60

000

000

40

010

100

40

010

001

001

100

20

011

101

20

011

101

110

110

111

111

Training data

U

100

101

0

100

101

OptimizationSteps

OptimizationSteps

compared to the Male concept class (blue lines). This disparity stems from the Female class being more predominant in the dataset, thereby making the Male class underrepresented. This observation offers useful insights for practitioners: to mitigate biased outcomes against underrepresented groups, we need to further train the diffusion model beyond its initial convergence on the training dataset.

5 Conclusion

In this work, we introduced a concept graph framework and used controlled interventions with a synthetic dataset to formulate and characterize four novel failure modes in the compositional generalization of diffusion models. We first hypothesized and verified that compositional generalization to a concept class far from the training data, based on concept distance, can fail, as it requires more optimization steps due to the sequential nature of generalization (Fig. 5). We show compositionality emerges in a sequence that respects the geometric structure of the concept graph, eliciting a multiplicative emergence effect that manifests as sudden increases in the model's performance to produce out-of-distribution data well after it has learned to produce samples from its training distribution. This behavior is reminiscent of the recently observed phenomenon of grokking in language modeling-like objectives [80, 81, 121]. Furthermore, when a particular concept value is underrepresented in the training dataset (e.g., an underrepresented color or gender), avoiding misgeneralization of the concept value, and thus achieving compositional generalization, requires far more additional optimization steps, even after achieving perfect performance on the training set. We also discovered a critical threshold for the inclusion of certain concepts in the training data, which is useful for deliberately making the model fail at learning harmful concepts (Fig. 8). Finally, we first analyze misgeneralizations in an adversarial task and find that fine-tuning does not provide a solution (Fig. 9). Overall, our synthetic data approach allowed us to apply controlled interventions that shed light on four different mechanisms behind the failure of diffusion models in compositional generalization.

Authors' Contributions

ESL initiated the research by defining the problem. ESL and HT formulated the theoretical framework, the concept graph, and collaborated with MO on the experimental design. MO conducted most experiments and, along with ESL and HT, produced the visualizations and figures. ESL and HT authored the introduction. ESL was responsible for the literature review and related work. MO, ESL, and HT jointly worked on the results and discussion sections. Throughout the study, MO, ESL, and HT collaborated on analyzing experimental results and refining the manuscript. HT and RPD provided supervisory guidance and feedback.

References

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Midjourney AI. Midjourney , 2023. https://docs.midjourney.com .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Zhihong Pan, Xin Zhou, and Hao Tian. Arbitrary style guidance for enhanced diffusion-based text-to-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4461-4471, 2023.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2022.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440 , 2022.

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 , 2023.

Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400 , 2023.

Ian Huang, Panos Achlioptas, Tianyi Zhang, Sergey Tulyakov, Minhyuk Sung, and Leonidas Guibas. Ladis: Language disentanglement for 3d shape editing. arXiv preprint arXiv:2212.05011 , 2022.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.08818 , 2023.

Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motion-conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.14404 , 2023.

Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2performer: Text-driven human video generation. arXiv preprint arXiv:2304.08483 , 2023.

Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. arXiv preprint arXiv:2302.07685 , 2023.

Kangfu Mei and Vishal M Patel. Vidm: Video implicit diffusion models. arXiv preprint arXiv:2212.00235 , 2022.

Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. arXiv preprint arXiv:2303.12688 , 2023.

Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single video editing with object-aware consistency. arXiv preprint arXiv:2303.07945 , 2023.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models. arXiv preprint arXiv:2303.17546 , 2023.

Hareesh Ravi, Sachin Kelkar, Midhun Harikumar, and Ajinkya Kale. Preditor: Text guided image editing with diffusion prior. arXiv preprint arXiv:2302.07979 , 2023.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427 , 2022.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 , 2023.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering , 2023.

Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. 2023 , 2023.

Boston Globe. A Boston Dynamics robot can now be run using ChatGPT. What could go wrong? , 2023. https://www.bostonglobe.com/2023/05/03/business/ robots-can-now-be-run-using-chatgpt-what-could-go-wrong/ .

Kevin Riera, Anne-Laure Rousseau, and Clement Baudelaire. Doctor GPT-3: hype or reality? , 2020. https://www.nabla.com/blog/gpt-3/ .

Kevin Roose. A Conversation With Bing's Chatbot Left Me Deeply Unsettled , 2023. https: //www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt. html .

Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991 , 2022.

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

Johann Brehmer, Joey Bose, Pim De Haan, and Taco Cohen. Edgi: Equivariant diffusion for planning with embodied agents. arXiv preprint arXiv:2303.12410 , 2023.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

Anthony Z Liu, Lajanugen Logeswaran, Sungryull Sohn, and Honglak Lee. A picture is worth a thousand words: Language models plan from pixels. arXiv preprint arXiv:2303.09031 , 2023.

Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517 , 2021.

Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-distribution generalization? In International Conference on Machine Learning , pages 12356-12367. PMLR, 2021.

Jivat Neet Kaur, Emre Kiciman, and Amit Sharma. Modeling the data-generating process is necessary for out-of-distribution generalization. arXiv preprint. arXiv:2206.07837 , 2022.

Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204.13807 , 2022.

Evelina Leivada, Elliot Murphy, and Gary Marcus. Dall-e 2 fails to reliably capture common syntactic processes. arXiv preprint arXiv:2210.12889 , 2022.

Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208.00005 , 2022.

Colin Conwell and Tomer Ullman. A comprehensive benchmark of human-like relational reasoning for text-to-image foundation models. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023.

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015 , 2022.

Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552 , 2023.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 423-439. Springer, 2022.

Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. arXiv preprint arXiv:2110.11405 , 2021.

Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 is seeing double: flaws in word-toconcept mapping in text2image models. arXiv preprint arXiv:2210.10606 , 2022.

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032 , 2022.

Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. Underspecification in scene description-to-depiction tasks. arXiv preprint arXiv:2210.05815 , 2022.

Stability AI. Stable Diffusion v2.1 , 2023. https://beta.dreamstudio.ai/generate .

NYTimes. Emily Anthes. Why Scientists Have Spent Years Mapping This Creature's Brain , 2021. https://www.nytimes.com/2021/10/26/science/ drosophila-fly-brain-connectome.html .

IEEE Spectrum. David Adler. Reverse Engineering the Brain , 2008. https://spectrum. ieee.org/reverse-engineering-the-brain .

Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Adv. in Neural Information Processing Systems (NeurIPS) , 2019.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936 , 2023.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint. arXiv:2108.07258 , 2021.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 , 2022.

Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 , 2023.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Noah D Goodman, Joshua B Tenenbaum, Thomas L Griffiths, and Jacob Feldman. Compositionality in rational analysis: Grammar-based induction for concept learning. The probabilistic mind: Prospects for Bayesian cognitive science , 2008.

Steven Phillips and William H Wilson. Categorial compositionality: A category theory explanation for the systematicity of human cognition. PLoS computational biology , 6(7): e1000858, 2010.

Steven M Frankland and Joshua D Greene. Concepts and compositionality: in search of the brain's language of thought. Annual review of psychology , 71:273-303, 2020.

Carlo Reverberi, Kai Görgen, and John-Dylan Haynes. Compositionality of rule representations in human prefrontal cortex. Cerebral cortex , 22(6):1237-1246, 2012.

Nicholas T Franklin and Michael J Frank. Compositional clustering in task structure learning. PLoS computational biology , 14(4):e1006116, 2018.

Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems , 34:15608-15620, 2021.

Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. Prompting large pre-trained visionlanguage models for compositional concept learning. arXiv preprint arXiv:2211.05077 , 2022.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bag-of-words models, and what to do about it? arXiv preprint arXiv:2210.01936 , 2022.

Emanuele Bugliarello and Desmond Elliott. The role of syntactic planning in compositional image captioning. arXiv preprint arXiv:2101.11911 , 2021.

Sam Spilsbury and Alexander Ilin. Compositional generalization in grounded language learning via induced model sparsity. arXiv preprint arXiv:2207.02518 , 2022.

Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. arXiv preprint arXiv:2303.13516 , 2023.

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5238-5248, 2022.

Jacob Andreas. Measuring compositionality in representation learning. arXiv preprint arXiv:1902.07181 , 2019.

Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537 , 2022.

Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning , pages 2873-2882. PMLR, 2018.

Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision-language pretrained models learn primitive concepts? arXiv preprint arXiv:2203.17271 , 2022.

Michael A Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural compositionality in neural networks. arXiv preprint arXiv:2301.10884 , 2023.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Lukas Schott, Julius Von Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2107.08221 , 2021.

Josef Valvoda, Naomi Saphra, Jonathan Rawski, Adina Williams, and Ryan Cotterell. Benchmarking compositionality with formal languages. In Proceedings of the 29th International Conference on Computational Linguistics , pages 6007-6018, 2022.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Proc. Int. Conf. on Learning Representations (ICLR) , 2017.

Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Adv. in Neural Information Processing Systems (NeurIPS) , 2021.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprint. arXiv:2102.11107 , 2021.

Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles. arXiv preprint arXiv:2307.05596 , 2023.

Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. Advances in Neural Information Processing Systems , 31, 2018.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795, 2020.

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning , pages 2649-2658. PMLR, 2018.

Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations , 2018.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 , 2017.

Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? Adv. in Neural Information Processing Systems (NeurIPS) , 2019.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proc. int. conf. on machine learning (ICML) , 2019.

Mark A Bedau. Is weak emergence just in the mind? Minds and Machines , 18:443-459, 2008.

David G Green. Emergence in complex networks of simple agents. Journal of Economic Interaction and Coordination , pages 1-44, 2023.

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004 , 2023.

Rahul Ramesh, Mikail Khona, Robert P Dick, Hidenori Tanaka, and Ekdeep Singh Lubana. How capable can a transformer become? a study on synthetic, interpretable tasks. arXiv preprint arXiv:2311.12997 , 2023.

Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated imagetext data created by the people, for the people. arXiv preprint arXiv:2111.11431 , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:18771901, 2020.

Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems , 35:21750-21764, 2022.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. ICML , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Supplementary Information

A Experimental details

A.1 Synthetic dataset

The dataset consists of 5,000 rendered images of 2D geometric shapes, along with corresponding concept classes. These synthetic images are generated using Blender 2 by creating a scene graph and rendering it. Each scene contains a single object placed on a blank background of size 28 × 28. The objects have three types of attributes: size, color, and shape. There are two shapes (circle and triangle), three colors (red, blue), and two sizes (large, small), resulting in up to eight different combinations of attributes. Each image is annotated with the corresponding object attributes, which can be utilized as conditional features for image generation. This enables us to directly evaluate the text-to-image generation capability of the diffusion model against ground truth images. Fig. 11 depicts example images with the corresponding concept classes.

Color: (0.81,0.16,0.24)

(0.75,0.16,0.19)

1.46

(0.81,0.23,0.19)

2.40

(0.22,0.20,0.80)

2.57

(0.16,0.24,0.85)

1.28

(0.23,0.16,0.79)

2.55

(0.78,0.23,0.16)

1.26

(0.21,0.19,0.79)

1.62

Color: (0.78,0.20,0.20)

(0.76,0.18,0.22)

0.93

(0.84,0.21,0.21)

2.92

(0.22,0.20,0.82)

2.49

(0.20,0.16,0.82)

0.92

(0.23,0.19,0.80)

2.80

(0.80,0.20,0.23)

1.07

(0.15,0.16,0.80)

1.67

Color: (0.85,0.19,0.17)

(0.83,0.25,0.25)

1.33

(0.81,0.20,0.20)

2.37

(0.19,0.22,0.80)

2.29

(0.18,0.21,0.82)

1.31

(0.17,0.18,0.79)

2.34

(0.75,0.25,0.22)

1.18

(0.24,0.22,0.83)

1.00

Color: (0.83,0.23,0.24)

(0.77,0.22,0.16)

0.84

(0.80,0.25,0.25)

3.09

(0.17,0.19,0.76)

2.72

(0.19,0.21,0.84)

0.87

(0.25,0.15,0.76)

2.33

(0.79,0.19,0.16)

1.29

(0.18,0.25,0.81)

1.67

000

001

100

010

011

110

101

111

A.2 Loss function

Diffusion models convert Gaussian noise into samples from a data distribution through an iterative denoising process. The sampling process starts with a noisy input x T , and denoised samples are generated through gradual iteration, x T -1 , x T -2 , until the original input x 0 is obtained. Conditional diffusion models [10, 122] allow for a denoising process conditioned on texts or class labels. In all the experiments, we used the conditional diffusion model with the form p ( x | V ) , where x denotes an image and V = { v 1 , v 2 , ..., v n } denotes a set of n concept variables. To predict the noise ϵ at each timestep t ∈ [0 , T ] , we follow the approach proposed in [3] by training a neural network. Specifically, we construct a neural network ϵ θ ( x t , t, a ) and minimize the mean squared error (MSE) between the predicted Gaussian noise and the true noise:

where q ( x 0 ) denotes the distribution of input image x 0 , and N ( 0 , I ) denotes the standard Gaussian distribution.

A.3 Architecture

We use the conditional U-Net architecture [1], as in [3], for our neural network ϵ θ ( · ) . Our architecture comprises two down-sampling and up-sampling blocks, with each block consisting of 3 × 3 convolutional layers, GELU activation, the global attention, and pooling layers. The conditional information V are fed through an embedding layer and concatenated with the image feature maps at each stage of the up-sampling blocks. We illustrate our network architecture in Fig. 12.

Conditional U-Net

Skip connection

x(T- 1)

Textencoding

XT-1

[28x28x512]

[28x28x512]

[14x14x512]

[14x14x512]

[7x7x256]

[95]

[7x7x256]

Size:[28x28x3]

[28x28x3]

[28x28x3]

Shape

Color

+→

×

Element-wiseaddition

Size

Element-wisemultiplication

Timestep

A.4 Optimizer

We implemented the diffusion model using PyTorch and trained it on four Nvidia A100 GPUs. We performed a hyperparameter search based on a validation set. We tested batch sizes ranging from 32 to 256, the number of channels in each layer from 64 to 512, leaning rate from 10 -4 and 10 -3 , the number of steps in the diffusion process from 100 to 400. We employed the Adam optimizer [123] with β 1 = 0 . 9 , β 2 = 0 . 99 , and weight decay of 10 -5 .

A.5 Evaluation metric

For the evaluation, we follow a probing protocol popularly used in several prior works on learning disentangled representations [103, 109-114]. To evaluate the attributes of the images generated by our conditional diffusion model, we trained linear classifiers on these images for three specific attributes: shape, color, and size. For each attribute, we developed a dedicated classifier: f 0 (ˆ x 0 ) for shape, f 1 (ˆ x 0 ) for color, and f 2 (ˆ x 0 ) for size. Here ˆ x 0 denotes the image generated by the conditional diffusion model. We utilized a cross-entropy loss function to train these classifiers. The output of each classifier fell into one of two categories: for shape, the categories were circle or triangle; for color, blue or red; and for size, large or small. We then calculated the accuracy for each attribute using the corresponding classifier outputs. To quantitatively assess the accuracy of predicted attributes aligning with their corresponding ground-truth concept classes, we utilize a multiplicative measure. This measure gauses the accuracy of all attributes, and defined by the product of individual accuracies for each attribute as follows:

where 1 ( · ) is the indicator function, n is the index of the test samples, and N t is the total number of samples used for evaluation. For our experiments, we generated N t = 50 images for each input of concept classes. v ( n ) 0 , v ( n ) 1 , and v ( n ) 2 denote the actual (ground truth) concepts classes for shape, color, and size, respectively. We trained them over 50 epochs using the training dataset comprising 5,000 pairs of concept classes and images. The trained linear classifiers achieved an accuracy rate of 100% on the test set drawn from the original synthetic dataset.

Ground truth

0.98

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

0.22

0.5

0.6

0.32

0.78

0.62

0.86

1.0

0.94

Ground truth

0.34

0.4

0.64

0.5

0.64

0.72

0.96

0.98

0.98

Ground truth

0.16

0.0

0.2

0.66

1.0

1.0

1.0

1.0

1.0

Ground truth

0.08

0.06

0.0

0.7

1.0

1.0

1.0

1.0

1.0

Ground truth

0.96

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

0.06

0.0

0.0

0.0

0.26

0.68

0.84

0.0

0.2

Ground truth

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

0.98

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

Ground truth

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

A.6 Experiment with real data

We sourced our data from the CelebA dataset 3 . Our training dataset consists of 15,000 facial images, each labeled with its respective concept class. We delineated the concept classes based on these attributes: gender (classified as "Male" or otherwise), facial expression ("Smiling" or not), and hair color (either "Black Hair" or "Brown Hair"). We adjusted the images to a resolution of 48 × 48 pixels.

For our evaluation, we designed three classifiers for each concept using a three-layer CNN with 64 units. The learning rate was established at 10 -3 , trained the model for 10 epochs. We employed the ReLU activation function for the hidden layers and the sigmoid function for the output layer. We divided the image dataset into training and test sets at a ratio of 80% to 20%. The trained classifier exhibited a 96% accuracy for gender, 86% for facial expression, and 95% for hair color. All additional experimental configurations conformed to our synthetic data experiment procedures.

B Additional experimental results

In Fig. 16, we generalize the findings in Fig. 5 to a larger concept graph, i.e., one with four concept variables. Specifically, we now introduce a fourth concept variable backgroundcolor ={white, black} and again find that compositional generalization occurs in the order governed by the distance from the training set: the accuracy begins to rise at a concept distance of 1, continues to increase at a distance of 2, and peaks at 3.

In Fig. 17, we investigate the bottlenecks that affect compositional generalization by examining the prediction accuracy of concept variables such as shape, color, and size, as well as the overall classification accuracy. To eval-

uate this, we generated 50 samples in each epoch of training and classified them using a linear probe. Our findings reveal that various concept bottlenecks impede the learning process for different compositional objects. For instance, the generalization of blue-colored objects ( 011 , 101 , 111 ) is hindered by the color concept, indicating that the model struggles to correctly identify objects based on their color when it comes to these particular compositions. Similarly, the generalization to small red triangles ( 111 ) is bottlenecked by the shape concept, implying that the model faces difficulties in accurately recognizing the shape of these specific compositions. These observations highlight the challenges associated with compositional generalization and shed light on the specific concept bottlenecks that hinder the learning process. Understanding and addressing these bottlenecks can contribute to the development of more robust models capable of achieving better compositional generalization in various contexts.

In Fig. 18, 19, and 20 we present the experimental results for the diffusion model without global attention. Notably, we observe that these results exhibit a similar pattern to the ones obtained from the diffusion model with global attention, which is shown in Fig. 5 and 6. The experimental results consistently support our findings. Firstly, the order of emergence of compositional capabilities aligns with the compositional structure of the data-generating process, as evident in the results shown in Fig. 18. Secondly, we observe a notable delay in the emergence of the ability to generate underrepresented colors as the concept distance within a concept class increases. This emphasizes the importance of continued training beyond the point of achieving in-distribution generalization, as demonstrated in Fig. 19. Specifically, we observe that the model is capable of generating the majority color (red) much earlier than the underrepresented color (blue). Importantly, the training

Shape

Color

Size

000

001

010

100

011

101

110

111

(a)

(0,1,1)

(1,1,1)

(b)

(c)

(0,0,1)

(1,0,1)

100 1

80

60

Color

(0,1,0)

000

40

001

010

(1,1,0)

100

011

Size

20

101

110

111

(0,0.0)Shape

0

(1,0,0)

102

OptimizationSteps

103

104

0

4

6

1020

40

60

Numberofepochs

time to achieve generalization based on a concept increases with concept distance. This observation provides important insights for training models with fairness in mind: even once generalization for in-distribution concept classes is achieved, stopping the training of a model will likely lead to a failure in generating underrepresented concepts, particularly for compositional generalization. Thirdly, our findings reveal a multiplicative impact of learning individual concepts on performance in compositional tasks, offering an explanation for the sudden emergence of capabilities, as depicted in Fig. 20.

In Fig. 21, we conducted experiments with different configurations by using varied sets of nodes in the training data. In Fig. 21(a) and Fig. 21(b), we inverted the axes of the concept graph from Fig. 9(a): (a) from bottom to top and (b) from left to right. In Fig. 21(c) and Fig. 21(d), we inverted the axes of the concept graph from Fig. 5(a): (c) from bottom to top and (d) from left to right. In Fig. 21(b) and Fig. 21(c), we observe that the model incorrectly generates triangles for both 011 and 010 , even though circles are expected. In the other cases, the model outputs correct images for all the nodes. Our observations indicate that the model struggles to to dissociate a second element value of 1 with the shape of a small triangle, exhibiting a bias to generate triangles rather than circles for small objects.

100

80

000

60

001

010

40

100

011

101

20

110

111

102

103

104

OptimizationSteps

(a)

100

Multiplicative

Additive

50

111

0

102

103

104

(b)

OptimizationSteps

100

Color

Shape

50

Size

102

103

104

OptimizationSteps

(a)

(b)

(C)

(d)

111

001

101

010

110

000

100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies compositional generation abilities of conditional diffusion models. The main contributions of the paper are the concept graph framework, which is used to examine compositional abilities in a simplified setting, and insights on the learning dynamics of diffusion models.\n\nSTRENGTHS:\nThe paper is very well written, motivated, and easy to follow. The problem the authors are investigating is relevant to know the limitations of compositional generation of the current diffusion models.\n\nWEAKNESSES:\nEven though the paper provides a very pleasing framework to investigate compositional capabilities of diffusion models through simple and highly controllable interventions, I think the submission is incomplete. This is mainly because the paper considers only a single architecture and does not provide guidelines how insights from concept graph framework could be further studied or provide validation that they hold in a large scale setting. Additionally, I would like to see a summary answering the questions from the abstract: “What are the reasons underlying this behavior? Which concepts does the model generally find difficult to compose to form novel data?”\n\nQUESTIONS:\n1. When does the sudden emergence of compositionality occur during training, can this be somehow predicted? Does it happen at the same time for different architectures and training configurations (noise schedules, augmentations etc.)? Providing answers to these questions would be valuable to researchers designing inductive biases to models.\n\n2. How is the critical threshold for learning new capabilities in Fig. 8 determined? Is there a practical way of estimating critical thresholds for learning harmful concepts based only on the given training dataset?\n\n3. Why Fig. 8 results could not be interpreted as if you have even a slight amount of harmful concepts in the dataset, then the model can eventually learn these given sufficient amount of training/capacity? From this point of view even a small amount of harmful samples leads to unwanted behavior and filtering training data is very important.\n\nMinor points on improving the clarity:\n\n4. Potential typo at line 188, “0010” corresponds to a sample in the training set.\n\n5. Fig. 5 duplicate sentence. For clarity Fig. 5 b) and c) would benefit having the same units in the x-axis (either optimization steps or epochs).\n\n6. I would remove the bullet point: “if so, under what circumstances does it fail?” from the beginning of Sec. 4 to Sec. 5 because that is where this question is assessed.\n\n7.  Figure 8 lattices are potentially flipped. The lattice of (b) should be the lattice of (a) because in (a) the number of samples from “001” is modified, and vice versa. The lattice (b) is also redundant, I suggest refactoring the figure to contain three subfigures (a) showing the lattice and (b), (c) showing the interventions. Additionally, the x-axis shows the number of samples and the text discusses frequencies. I would switch the x-axis to show frequencies instead of the number of images to improve interpretability in a more general case.\n\n8. Fig. 9 says “In this example, large objects are red, and small objects are blue.”, however in (a) there are large objects in both blue and red.\nIn general, figures would benefit if they were converted to vector graphics format (pdf, svg etc.) because they need to be zoomed in order to view.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

comments: SUMMARY:
This paper proposes to empirically studied how compositional structure emerges in diffusion model. The paper proposes the abstraction of concept graphs, and illustrates how diffusion models first learn to fit the training data before compositionally generalizing. The paper illustrates how diffusion models have difficulty disentangling data and modeling low data concepts

STRENGTHS:
- The paper is clearly written and the color coding through the text and figures greatly helped comprehension of the paper
- I enjoyed the introduction of concept graphs and the illustration of compositional distance with respect to graph
- The analysis in the paper is quite enlightening, illustrating how diffusion models first fit in distribution before generalizing out of distribution, as well as its struggles on finding confounding factors between variable and low data variables.

WEAKNESSES:
- The evaluation setting is rather simple and focuses on a single synthetic dataset with 3 classes of attributes
- It would be good to study generalization both to more complex images (for instance photorealistic synthetic scenes rendered using something like Kubric) as well as to more factors larger than 3
- In practice, a lot of concepts in the world follow the Zipf distribution -- it would be interesting to analyze that
- It would also be interesting to see how syntatic form would effect compositional generalization
- Some theoretical analysis on compositional generalization would also be interesting

QUESTIONS:
See above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors try to understand the compositionality aspects of generative models by training a conditional diffusion model in a toy setting on synthetic data. They show that the models indeed learn to be compositional if we train longer. They also hypothesize that the sudden emergence of compositionality in the later parts of training is due to multiplicity of how the model learns the underlying factors. They also show that it is hard to disentangle and generalize to new concepts if the underlying factors are highly correlated.

STRENGTHS:
The paper is easy to follow. The study of compositionality in diffusion models is a crucial problem to understand. The authors studied it in a controlled setting which is novel.

WEAKNESSES:
The authors chose a very simple synthetic dataset to study this problem. The dataset has only 3 attributes. Even though some of the observations make sense, we have no guarantees that any of the results will extend to the real world. 

The figures are quite confusing. Especially the colors. Blue (albeit a different shade), was used as an attribute and to denote the training data. 

In Section 4, the authors used a bit of sensationalist language, such as “sudden emergence” etc, while such terms are not defined nor appropriately cited. They can improve the writing in this section.

QUESTIONS:
1.	Did you try or have results with synthetic data with > 3 attributes?
2.	Why did you not use synthetic data with real objects instead of colors and shapes? Verifying your claims on even a small real-object dataset would make the paper much more robust and useful. 
3.	I might have missed this detail, what is the size of the test set, did you look at only the “pink” examples in the figures?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a framework for studying the compositional generalisation abilities of diffusion models (or generative models more broadly). To that end it introduces the notion of a concept graph which the authors use to manipulate simple synthetic datasets. These concept graphs arrange different combinations of concept values and variables in a way that perseveres a notion of distance between different combinations (based on bit flips). By studying how performance changes as models are required to reconstruct concept value combinations that are further apart from the training data, they can draw inferences regarding the emergence of generalisation capabilities in diffusion models.\n\nSTRENGTHS:\n1. The article is very well written and easy to follow. The idea of the concept graph and the manipulation of the dataset is clearly explained and well motivated.\n2. The use of a hypothesis driven research as opposed to the standard benchmarking approach is welcomed change compared to most of the research in AI. That the way the authors present it makes it seem as particularly novel is more a reflection on the field than this particular work (but see later on some suggestions regarding this point).\n3. The point correlations between concept value make learning difficult for the models is important since it highlights how much model still rely on correlations in spite of the fact of their superior generative capabilities.\n\nWEAKNESSES:\nThere are no substantial weaknesses, but there are some errors and omissions when discussing the literature which I will discuss below.\n\nQUESTIONS:\n1. Is Multiplicative Emergence unsurprising when the authors are forcing accuracy to be 0 if at least one concept value is incorrect during out-of-distribution generation?\n2. I find it weird that colour takes longer to learn than the other properties. Since the reconstruction error must be very high even if the shape and scale are correct, I would expect cooler to be learned quicker. Do the authors have some insight into why this happens?\n3. Is the relation to grokking really that strong? In grokking, models still exhibit high validation error even if the test error is high. But here\n\nLIMITATIONS:\n1. It is incorrect to characterise some of the work discussed in lines 120 and 121 as merely performing benchmarking. Indeed at least [94], [95] and [99] perform manipulations of the training and dataset in much the same way as this present work. Thus, while I definitely like that the authors took a more hypothesis driven approach to their work, it is not by any means the case that this is has not been done before. Other works that do this include [1] and [2]. While these may not be based on generative models, they are still relevant examples of the kind of study that the authors are advocating for.\n\n2. Relatedly, the authors should discuss the relation of their work with [3] and [4]. Unlike the previous references, these do involve compositional generalisation capabilities of generative models. The former also provides one characterisation of compositional generalisation in generative models that is similar to the one used in this work. Thus it is indeed not the case that there have been no previous proposals and that [101] is the only noteworthy attempt (which by its own admission, just summarises previous definitions anyway).\n\n3. The latter uses an approach based on transforming input images which allows it to identify the concepts that vary between images in a way that resembles the example presented in Figure 2, panel b. Their results show a similar pattern as the one in this work where combinations that are close to the training data are easier for the models. I believe that both this work and the previously discussed ones deserve a slightly deeper discussion.\n\n4. Lines 291 and 292 are very close to each other which makes the caption of Figure 7 and the main text hard to distinguish.\n\n\n\n[1] Hermann, K., Chen, T., & Kornblith, S. (2020). The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33, 19000-19015. \n\n[2] Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665-673.\n\n[3] Montero, M. L., Ludwig, C. J., Costa, R. P., Malhotra, G., & Bowers, J. (2020, October). The role of disentanglement in generalisation. In International Conference on Learning Representations.\n\n[4] Montero, M., Bowers, J., Ponte Costa, R., Ludwig, C., & Malhotra, G. (2022). Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation. Advances in Neural Information Processing Systems, 35, 10136-10149.\n\n** note that I used the same citation numbers the authors used for the other references i.e. [95] above corresponds to reference [95] in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}