SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maya Okawa\*1,2, Ekdeep Singh Lubana\*1,3, Robert P. Dick3, Hidenori Tanaka\*1,2  
<sup>1</sup>Center for Brain Science, Harvard University, Cambridge, MA, USA  
<sup>2</sup>Physics & Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA, USA  
<sup>3</sup>EECS Department, University of Michigan, Ann Arbor, MI, USA
Modern generative models exhibit unprecedented capabilities to generate extremely realistic data. However, given the inherent compositionality of the real world, reliable use of these models in practical applications requires that they exhibit the capability to compose a novel set of concepts to generate outputs not seen in the training data set. Prior work demonstrates that recent diffusion models do exhibit intriguing compositional generalization abilities, but also fail unpredictably. Motivated by this, we perform a controlled study for understanding compositional generalization in conditional diffusion models in a synthetic setting, varying different attributes of the training data and measuring the model's ability to generate samples out-of-distribution. Our results show: (i) the order in which the ability to generate samples from a concept and compose them emerges is governed by the structure of the underlying data-generating process; (ii) performance on compositional tasks exhibits a sudden "emergence" due to multiplicative reliance on the performance of constituent tasks, partially explaining emergent phenomena seen in generative models; and (iii) composing concepts with lower frequency in the training data to generate out-of-distribution samples requires considerably more optimization steps compared to generating in-distribution samples. Overall, our study lays a foundation for understanding emergent capabilities and compositionality in generative models from a data-centric perspective.
The scaling of data, models, and computation has unleashed powerful capabilities in generative models, enabling controllable synthesis of realistic images [1-11], 3D scenes [12-16], videos [17-25], accurate image-editing [26-31], and semantically coherent text generation [32-46]. With increased interest in incorporating these models in day-to-day applications [47-50], e.g., to improve robotic systems via better planning and grounding [51-59], analyzing and improving their reliability has become crucial.
With the motivation above, we study compositional generalization abilities of conditional diffusion models, i.e., diffusion models that are conditioned on auxiliary inputs to control their generated images (e.g., text-conditioned diffusion models [6, 26]). Given the inherent compositionality of the real world, it is difficult to create a training dataset that exposes a model to all combinations of different concepts underlying the data-generating process. We therefore argue that compositional generalization is central to model reliability in out-of-distribution scenarios, i.e., when the model has to compose a novel set of concepts to generate outputs not seen in the training data set [60, 61]. As shown by several prior works investigating the compositional generalization capabilities of off-the-shelf text-conditioned diffusion models [62-72], modern diffusion models often compose complicated
*Equal contribution. Email: {mayaokawa, ekdeeplubana, hidenori_tanaka}@fas.harvard.edu.
"White-colored lizard"
"Green-colored lizard"
"White-colored goldfish"
"Green-colored goldfish"
"White-colored panda"
"Green-colored panda"
"Blue-colored lizard"
Figure 1: (Lack of) Compositionality in text-conditioned diffusion models. Images generated using Stable Diffusion v2.1 [73]. We see diffusion models conditioned on text descriptions of the concepts in an image often allow generation of novel concepts that are absent from the training data, indicating an ability to compose learned concepts and generalize out-of-distribution (lizard and goldfish panels). However, the model sometimes unpredictably fails to compose concepts it has learned, i.e., compositional capabilities depend on the specific concepts being composed. For ex., generating a panda in the above figure is difficult for the model, likely because it has not been exposed to different colors of pandas. The model instead alters the background or lighting to change the color.
"Magenta-colored lizard"
"Blue-colored goldfish"
"Magenta-colored goldfish"
"Blue-colored panda"
"Magenta-colored panda"
concepts, producing entirely non-existent objects, but also unpredictably fail to compose apparently similarly complicated concepts (see Fig. 1). It remains unclear why models produce novel samples by composing some concepts but not others. Indeed, what concepts does a model learn to compose? What differentiates these concepts from the ones the model is unable to compose?
"Model Experimental Systems" Approach with Interpretable Synthetic Data. To address the above questions, we seek to take a systematic approach to understanding complex systems, such as generative models, by dividing the process into hypothesis generation and testing. Specifically, drawing inspiration from the natural sciences, we adopt a model-experimental systems approach, which involves studying simplified, controllable, and steerable experimental systems to develop mechanistic hypotheses applicable to more complex systems. For instance, in neuroscience research, model organisms like mice or fruit flies are used to study neural mechanisms and develop hypotheses that can be tested on the larger-scale human brain [74-76]. Similarly, in our work, we design a synthetic experimental setup that pursues simplicity and controllability while preserving the essence of the phenomenon of interest, i.e., compositional generalization. Specifically, our data-generating process tries to mimic the structure of the training data used in text-conditioned diffusion models by developing pairs of images representing geometric objects and tuples that denote which concepts are involved in the formation of a given image (see Fig. 2). We train diffusion models on synthetic datasets sampled from this data-generating process,
conditioning the model on tuples denoting which concepts an object in the image should possess, while systematically controlling the frequencies of concepts in the dataset. Thereafter, we study the model's ability to generate samples corresponding to novel combinations of concepts by conditioning the denoising process on a correspondingly novel tuple, thus assessing the model's ability to compositionally generalize. This approach allows us to systematically investigate the key properties of a dataset enabling compositional generalization in an interpretable and controlled manner in conditioned diffusion models. Our main contributions follow.
- Concept graphs: A simple, synthetic framework for studying conditional diffusion models. We develop a minimalistic abstraction for training conditional diffusion models by introducing a framework called "concept graphs". The framework forms the foundation of our systematic
(a)
Train Set
Tuple
$\mathbf{\Psi},a)$
Image
#
(b)
Shape
1: circle, 2: triangle
Figure 2: Compositionality in a minimalistic conditional generation task. (a) We train diffusion models on pairs of images and tuples, where the tuples denote the values of the concepts represented in the image (e.g., values of color and shape in the figure). (b) Samples between which only a single tuple element differs simplify the learning of a capability to recognize and alter the concept distinguishing the images. (c) To test the existence of such capabilities of a trained model, we ask the model to generate images corresponding to novel tuples that are out-of-distribution, hence requiring compositional generalization.
(2,a)
(1,b)
#
$a$  :red,b:blue
(c) Test
(2,b)
?
Color
study of how diffusion models learn to generate samples from a set of concepts and compose these concepts to generate out-of-distribution samples. To our knowledge, our results are the first demonstration of perfect compositional generalization on a (synthetic) vision problem.
- Multiplicative emergence of compositional abilities. We use our model system's interpretability to monitor learning dynamics of a diffusion model's capabilities and the ability to compose them to produce out-of-distribution data. As we show, a diffusion model first memorizes the training dataset and then sequentially generalizes to concepts that are at a greater "distance" from the training distribution. Since progress in learning each capability multiplicatively affects the model's performance in compositional generalization, we find a sudden ability to compose and produce samples out-of-distribution "emerges". We thus hypothesize compositionality may partially explain emergent phenomena seen in modern generative models [77-81].
- Investigating challenges for compositional generalization. We systematically examine the challenges arising from decreased frequency of specific concepts necessary for learning a corresponding capability. We further evaluate the effectiveness of using fine-tuning to address misgeneralizations in adversarial settings, and find that it is generally insufficient to enable the learning of new capabilities.
Before proceeding, we emphasize the trade-offs and limitations of our model-experimental systems approach. Just as neural mechanisms identified in model animals cannot be directly applied to human medical applications, our observations should not be considered definitive conclusions that can be directly transferred to modern large generative models. Instead, our study aims to establish conceptual frameworks, identify data-centric control variables, and formulate mechanistic hypotheses, paving the way for further theoretical and empirical explorations of larger models.
Diffusion models. Diffusion models are the state-of-the-art method for generating realistic visual data [1-11]. They are often trained using image-text pairs, where the text is processed using a large language model to produce semantically rich embeddings that allow controlled generation of novel images and editing of existing images [26-31]. Such conditional diffusion models are easy to test for compositional generalization, as one can directly give a text description that requires composition of concepts the model is likely to have learned (e.g., avocado and chair) to produce images that were not encountered in the model's training data (e.g., avocado chair; see [1, 6]). Such results demonstrate the model's ability to compose and generalize out-of-distribution. However, we emphasize that text-conditioned models may fail to generalize compositionally if the text model is unable to sufficiently disentangle relevant concepts in the text-embedding space. To avoid this pitfall, we use ordered tuples that denote precisely which concepts are involved in an image's composition.
Compositional generalization. Compositionality is an inherent property of the real world [82], wherein some primitive such as color can be composed with another primitive such as shape to develop or reason about entirely novel concepts that were not previously witnessed [60]. Notably, compositionality is hypothesized to play an integral role in human cognition, enabling humans to operate in novel scenarios [83-87]. Inspired by this argument, several works have focused on developing [67, 68, 88-93] and benchmarking [64, 66, 90, 94-102] machine learning models to improve and analyze compositional generalization abilities. We especially highlight the works of Lewis et al. [96], who use a similar model experimental system approach as ours to evaluate factors influencing compositionality in contrastive language-image pretraining (CLIP) [34]. Our focus on generative models, instead of representation learning, distinguishes our results however. Relatedly, we note the notion of compositionality has been discussed in several recent works in the fields of disentangled and causal representation learning [101, 103-107]. Our framework for systematic understanding compositionality in generative models was heavily influenced by the synthetic dataset design considerations promoted in these papers. We also emphasize that a thorough formalization of compositionality in modern machine learning is generally lacking, but highlight the notable exceptions by Hupkes et al. [108] and Wiedemer et al. [106]. To avoid ambiguity, we provide a precise formalization that instantiates our intended meaning of compositionality.
3 Concept Graph: A Minimalistic Framework for Compositionality
In this section, we present the concept graph framework, as illustrated in Fig. 3, which enables us to visually depict compositional structure of our synthetic data and forms the basis for generating
hypotheses and designing experiments in our work. We begin by defining the essential building blocks of our framework: concept variables and concept values. In the following, we call a specific output of our data-generating process an "object" (see Fig. 3).
Definition 1. (Concept Variables.) A concept variable  $v_{i}$  uniquely characterizes a specific property of an object.  $V = \{v_{1}, v_{2}, \dots, v_{n}\}$  denotes a set of  $n$  concept variables, such that, given an object  $x$ ,  $v_{i}(x)$  returns the value of the  $i^{th}$  concept variable in that object.
For instance, for geometric objects shown in Fig. 3, concept variables may include shape, color, size, location, and angle. These variables take on values from a pre-specified range, called concept values, as described below.
Definition 2. (Concept Values.) For each concept variable  $v_{i} \in V$ ,  $C_i = \{c_{i1}, c_{i2}, \dots, c_{ik_i}\}$  denotes the set of  $k_{i}$  possible values that  $v_{i}$  can take. Each element of  $C_i$  is called a concept value.
For example, in Fig. 3, the concept values for the shape variable are in the set  $\{\text{circle}, \text{triangle}, \text{square}\}$ , while for the color values are in the set  $\{\text{red}, \text{blue}, \text{green}\}$ . Given  $n$  concept variables, with each variable  $v_i$  having  $k_i$  concept values, there can be  $\prod_{i=1}^{n} k_i$  distinct combinations that yield a valid object. We use a unique combination of a subset of these concept variables,  $v_1, \ldots, v_p (p < n)$ , to define the notion of a "concept class".
Definition 3. (Concept Class.) A concept class  $C$  is an ordered tuple  $(v_{1} = c_{1}, v_{2} = c_{2}, \dots, v_{p} = c_{p})$ , where each  $c_{i} \in C_{i}$  is a concept value corresponding to the concept variable  $v_{i}$ . If an object  $x$  belongs to concept class  $C$ , then  $v_{i}(x) = c_{i} \forall i \in 1, \dots, p$ .
Note that the remaining  $n - p$  concept variables are free in the above definition. When these free variables are specified, we get a specific object from our data-generating process. That is, a concept class represents a family of objects by specifying the values of a pre-defined subset of concept variables. For example, in Fig. 1, different colored lizards instantiate images (objects) from the species (concept class) of lizards; here, color of the lizard serves as a free variable. Similarly, in the geometric objects scenario of Fig. 3, a "small red circle" would be a concept class for which the shape, color, and size variables have been assigned specific values. Objects are images designated by associating precise values with the remaining concept variables of location and angle. Next, we introduce the notion of concept distance, which serves as a proxy to succinctly describe the dissimilarity between two concept classes.
Definition 4. (Concept Distance.) Given two concept classes  $C^{(1)} = (c_1^{(1)}, c_2^{(1)}, \ldots, c_n^{(1)})$  and  $C^{(2)} = (c_1^{(2)}, c_2^{(2)}, \ldots, c_n^{(2)})$ , the concept distance  $d(C^{(1)}, C^{(2)})$  is defined as the number of elements that differ between the two concept classes:  $d(C^{(1)}, C^{(2)}) = \sum_{i=1}^{n} I(c_i^{(1)}, c_i^{(2)})$ , where  $I(c_{1i}, c_{2i}) = 1$  if  $c_{1i} \neq c_{2i}$  and  $I(c_{1i}, c_{2i}) = 0$  otherwise.
Figure 3: Concept graphs. We organize our study using a simple but expressive framework called concept graphs. The basis of a concept graph is a set of primitives called concept variables (e.g., shape, color, etc.). A subset of these variables is instantiated with specific values to yield a concept class, e.g.,  $\{\text{shape} = 0, \text{size} = 0, \text{color} = 1\}$  implies a small, blue circle. This is akin to defining a broad set of objects that share some common properties, e.g., all lizards in Fig. 1 belong to the same species, despite their diverse colors. A specific object in this class is instantiated by fixing the remaining variables, e.g., small, blue circles at different locations. Each concept class corresponds to a graph node, where nodes are connected if their concept classes differ by a concept distance of 1.
The concept distance quantifies the dissimilarity between two concept classes by counting the number of differing concept values. It is important to note that this distance serves only as a null model because each axis represents distinct concept variables and each of the variables can assume various possible concept values. We are now ready to define the notion of a concept graph (see Fig. 3), which provides a visual representation of the relationships among different concept classes.
Definition 5. (Concept Graph.) A concept graph  $G = (N, E)$  consists of nodes and edges, where each node  $n \in N$  corresponds to a concept class, and an edge  $e \in E$  connects two nodes  $n_1$  and  $n_2$  representing concept classes  $C^{(1)}$  and  $C^{(2)}$ , respectively, if the concept distance between the two concept classes is 1, i.e.,  $d(C^{(1)}, C^{(2)}) = 1$ .
A concept graph organizes different concept classes as nodes in the graph, while edges denote pairs of concept classes that differ by a single concept value. An ideal conditional diffusion model, when trained on a subset of the nodes from this graph, should learn capabilities that allow it to produce objects from other concept classes. We formalize this as follows.
Definition 6. (Capability and Compositionality.) Consider a diffusion model trained to generate samples from concept classes  $\hat{C} = \{C_1, C_2, \ldots, C_T\}$ . We define a capability as the ability to alter the value of a concept variable  $v_i$  to a desired value  $c_i$ . We say the model compositionally generalizes if it can generate samples from a class  $\tilde{C}$  such that  $d(\widetilde{C}, C_i) \geq 1 \forall C_i \in \hat{C}$ .
The ideas above are best explained via Fig. 4. Specifically, assume we train a diffusion model on data from a subset of concept classes, i.e., a subset of nodes in the concept graph. To fit the training data, the model might just memorize the training data or it may learn the relevant capabilities to alter specific concept variables; e.g., given samples from classes 0000, 0010, and 0001 in Fig. 4 (a), the model may just memorize the data or learn how to alter the third and fourth concept variables. Models that just memorize the training data lack the capability to generate samples from out-of-distribution classes (e.g., 1100). In contrast, the ability to compose concepts would enable the model to produce out-of-distribution samples starting from classes seen in the training data.
In summary, our concept graph framework provides a systematic approach to representing and understanding a minimalistic compositional structure, allowing for an analysis and comparison of different learning algorithms' abilities to generalize across various concept classes.
Figure 4: Capabilities and compositionality in a concept graph. Consider a lattice representation of a concept graph corresponding to four concept variables. Blue nodes denote classes represented in the training data; a model can either memorize data from these classes or learn capabilities to transform samples from one class to another. If it learns capabilities, it can compose them with data-generating process of samples from a concept class in the training data and produce samples that are entirely out-of-distribution, denoted as pink and green nodes.
3.1 Experimental and Evaluation Setup
With the concept graph framework in place, we now
have the tools to systematically investigate the impact of different data-centric properties on learning and composing capabilities in conditional diffusion models. Before proceeding further, we briefly define our experimental setup and evaluation protocol. This allows us to interleave our results with our hypotheses, explaining them in context.
**Experimental Setup.** The detailed setup is presented in Appendix A. In brief, we train conditional diffusion models that follow the U-Net pipeline proposed by Dhariwal and Nichol [1]. Our dataset involves concept classes defined using three concept variables, each with two values; specifically, shape = {circle, triangle}, color = {red, blue}, and size = {large, small}. Tuples with binary elements are used to condition the diffusion model's training; they serve as an abstraction of text conditioning. For example, the tuple 000 implies a large, red circle is present in the image. To sample images from this process, we simply map the size and color axes to the range [0, 1] and uniformly sample to develop a training dataset of 5000 samples; the precise samples depend on which concept classes are allowed in the data-generating process. In this setup, the minimal required set for learning capabilities to alter concepts is four pairs of tuples
and images, each drawn from one of the following four concept classes:  $\{000$ , (circle, red, large)\},  $\{100$ , (triangle, red, large)\},  $\{010$ , (circle, blue, large)\},  $\{001$ , (circle, blue, small)\}. Consider the following two tuples:  $\{000$ , (circle, red, large)\} and  $\{100$ , (triangle, red, large)\}. In this case, we see that the shape concept is encoded in the first elements. Here, 0 represents a circle and 1 represents a triangle. The remaining concepts are similarly encoded in later elements.
Evaluation Metric. Evaluating whether a generated image corresponds to the desired concept class might require a human in the loop. To eliminate this requirement, we follow methods from literature on disentanglement and train classifier probes to predict whether a generated image possesses some property of interest [103, 109-114]. Specifically, we use the diffusion model training data to train three linear probes, which respectively infer the following three concept variables: shape, color, and size. We define a model's accuracy for generating images of a given concept class as the product of the probabilities outputted by the three probes that each concept variable matches the value for the concept class. Note that a random classifier for a concept variable will have an accuracy of 0.5. We indicate this random baseline with dotted, gray lines in our plots when useful.
4 Multiplicative Emergence of Compositional Abilities
We first investigate the order in which a model learn capabilities to produce sample from a concept class and how the learning of such capabilities affects the ability to compositionally generalize.