Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space
Authors: Core Francisco Park
Maya Okawa
Andrew Lee
Ekdeep Singh Lubana
Hidenori Tanaka
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Learning Dynamics
Compositional Generalization
Emergent Abilities
Diffusion Models
Mechanistic Interpretability
Primary area: interpretability_and_explainability
TL;DR: We find that compositional generalization abilities of diffusion models emerge suddenly and robustly, while models might not actively exhibit this ability.

Abstract: Modern generative models demonstrate impressive capabilities, likely stemming from an ability to identify and manipulate abstract concepts underlying their training data. However, fundamental questions remain: what determines the concepts a model learns, the order in which it learns them, and its ability to manipulate those concepts? To address these questions, we propose analyzing a model’s learning dynamics via a framework we call the concept space, where each axis represents an independent concept underlying the data generating process. By characterizing learning dynamics in this space, we identify how the speed at which a concept is learned, and hence the order of concept learning, is controlled by properties of the data we term concept signal. Further, we observe moments of sudden turns in the direction of a model’s learning dynamics in concept space. Surprisingly, these points precisely correspond to the emergence of hidden capabilities, i.e., where latent interventions show the model possesses the capability to manipulate a concept, but these capabilities cannot yet be elicited via naive input prompting. While our results focus on synthetically defined toy datasets, we hypothesize a general claim on emergence of hidden capabilities may hold: generative models possess latent capabilities that emerge suddenly and consistently during training, though a model might not exhibit these capabilities under naive input prompting.

Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space
Core Francisco 1... Maya Okawa*133 Andrew Lee
BCXiv:2406.19370M [cs.LG] 11 Dec 2024
Hidenori Tanaka Ekdeep Singh Lubana f1.3
CBS-NTT Program in Physics of Intelligence, Harvard University *Deparmentt of Physics, Harvard University 3 Physics& & Informatics Laboratories, NTT Research, Inc.
EECS Department, University of Michigan, Ann Arbor
Abstract
Modern generative models demonstrate impressive capabilities, likely stemming from an ability to identify and manipulate abstract concepts underlying their train- ing data. However, fundamental questions remain: what determines the concepts model learns, the order in which it learns them, and its ability to manipulate those concepts? To address these questions, we propose analyzing model's learning dynamics via a framework we call the concept space, where each axis represents independent concept underlying the data generating process. By characterizing learning dynamics in this space, we identify how the speed at which a concept is learned, and hence the order of concept learning, is controlled by properties of the data we term concept signal. Further, we observe moments of sudden turns in the direction of model's 's learning dynamics in concept space. Surprisingly, these points precisely correspond to the emergence of hidden capabilities, i.e., where latent interventions show the model possesses the capability to manipulate a concept, but these capabilities cannot yet be elicited via naive input prompting. While our results focus on synthetically defined toy datasets, we hypothesize a general claim on emergence of hidden capabilities may hold: generative models possess latent capabilities that emerge suddenly and consistently during training, though model might not exhibit these capabilities under naive input prompting.
Introduction
Modern generative models, such as text-conditioned diffusion models, show unprecedented capabili- ties [1-8]. These abilities have led to use of such models in applications as valuable as training control policies for robotics [9-11] and models for weather forecasting [12],
drastic as campaigning in democratic elections [13, 14]. Similar claims can be made for generative models of other modalities, e.g., large language models (LLMs) [15-18], speech and audio models [14, 19], even systems designed for enabling scientific applications such as drug discovery [20, 21]. Arguably, acquiring such general capabilities requires for models internalize the data gentrattnn process and disentangle the concepts (aka latent variables or factors of variation) underlying it [22, 23]. Flexibly manipulating these concepts can then enable generation of novel samples that are entirely out-of-distribution (OOD) with respect to the ones used for training [24-28].
As shown in prior work, modern generative models do exhibit signs of disentangling concepts underlying the data generating process and learning of corresponding capabilities to manipulate said
cqual contribution, equal advising (sce detailed list). Email: corepraaciscoparkkgg harvard. edu, ajyl@umich @umich ch. edu, (mayaokawa, ekdeeplubana, hidenori tanaka) as harvard.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(c)
011
313
000
80
001
Buckground color
60
Stze
110
001
20
Object color
Model Optimization Scale
000
100
Number gradient steps.
20000 40000 60000
Figure 1: Concept Learning Geometry underlies emergence. (a) Top: A multimodal model Iearns to generate the concepts in the order of "astronaut", "horse", and finally "riding" as it scales up (adapted from Yu et al.[45]). Middle: "blue square apple" is generated in the order of "apple", "blue", and "square"" (adapted from Li et al.[46]). Bottom: Despite its simplicity, our model trained on synthetic data shows concept learning dynamics where it first learns "shape" and then "color". (b) Concept space is an abstract coordinate space where individual axes correspond to different concepts and given point corresponds (e.g., large blue circles on bottom left corner). Traversal along axes of the concept space yield change in a specific property of the sample going from large blue circle to large red circle along object color axis). Trajectories show model's dynamics in concept space for learning to generate classes shown in-distribution (blue nodes) versus out of distribution (pink / red nodes). As we show, dynamics in concept space are highly interpretable, enabling precise comments on which concepts the model learns first, why, and what order it follows. (c) Measuring how accurately model generates samples from given concept class, showing an order of concept learning: first background color is learned, then size, and then object color.
"concept class", i.e., predefined collection of concepts
concepts [29-37]. At the same time, contemporary work has argued that models' capabilities can be unreliable, arbitrarily failing for a given input and succeeding on another [38 338-44]. Thus, critical questions remain on how generative models acquire their capabilities (see Fig. 1): what determines whether the model will disentangle a concept and learn the capability to manipulate it; are all concepts and corresponding capabilities learnd at the same time; and is there structure to the order of concept learning such that, given insufficient time, the model learns capabilities to manipulate only a subset of concepts but not others?
This work. To address the questions above, we propose to amalyze model's learning dynamics at the granularity of concepts. Since performing such analysis realistic, off-the-shelf datasets can be challenging, we develop synthetic toy datasets of 2D objects with different concepts (e.g., shape, size, color) that give us thorough control over the data-generating process and allow for an exhaustive characterization of the model's learning dynamics (see Fig. 1). Our contributions are as follows.
Introducing Concept Space. We propose to evaluate a model's learning dynamics in the concept space- an abstract coordinate system whose axes correspond to specific concepts underlying the generatieerrgggggee process. We instantiate a notion of underspecification concept space and establish its effects on model's (in)ability to disentangle concepts.
Concept Signal Dictates Speed of Learning. We find the speed which a model disentangles a concept and learns the capability to manipulate it is dictated by the sensitivity of the data- generating process to changes in values of said concept-a quantity we call concept signal. We show concept signals shape the geometry of learning trajectory and hence control the overall order of concept learning.
Sudden Transitions in Concept Learning, We use analytical curves to explain the phenomenol- ogy of learning dynamics in concept space, showing the dynamics can be divided into two broad phases: (P1) learning of a hidden cupability, whereby even if the model does not produce the desired output for a given input, there exist systematic latent interventions that lead the model to generate the desired output, and (P2) learning to generate the desired output from the input space.
Overall, while our results focus on a toy synthetic task and text-to-image diffusion models, we hypothesize a broader claim on hidden emergence of latent capabilities holds true: generative models possess latent capabilities that are learned suddenly and consistently during training, but these capabilities are not immediately apparent since prompting the model via the input space may not elicit them, hence hiding how "competent" the model actually is [47, 48]. Empirically, signs of "hidden capabilities" have already been shown in generative models at scale [45, 49-53), and our results help provide formal framework to partially ground such results.

---

2 Related Work
Concept learning. The term concepts used in this work is broadly equivalent to the notion of factors of variation from prior work on disentangled representation learning [23, 30, 54-61], and is motivated by use of the term in a similar sense in cognitive science [62-67]. The focus of literature on disentanglement has been to prove identifiability results, e.g., when will a generative model trained dataset learn invert the data-generating process, hence identifying the latent variables, i.e., concepts, that underlie it. Given the success of modern generative models, we argue, despite impossibility results from prior work, these models are in fact learning to disentangle some, if not all, concepts. Precisely what guides which concepts are disentangled however requires studying the learning dynamics the target of our work.
Interpretability. A growing line of papers demonstrate highly interpretable representations of intuitive concepts exist in generative models, especially LLMs (68-74], Similar work in this vein in image diffusion models has found semantic representations in various components of the model [37, 52, 75-77]: e.g., existence of linear representations for concepts such as 3D depth or object versus background distinctions [78, 79]. These papers further bolster our argument, modern generative models truly inverting the data-generating process and identifying the concepts underlying it.
Competence vs. performance. In cognitive science, a system's 's competence on a task is often contrasted with its performance [47, 48, 80-83]: competence is the system's 's possession of a capability (e.g., to converse in a language) and performance is system's 's use of that capability in concrete situations [80]. For example, a bilingual person may generally converse in their primary language L, despite possessing knowledge of another language L', unless it is crucial to use the latter clearly, they are competent in both languages, but gauging their performance on L' requires appropriately "prompting" them to use it. One can analogize this distinction with remarks on a neural network possessing capability versus us being able to elicit it on predefined benchmarks and measure their performance [84-87]. For example, on the BigBench benchmark [88], LLMs were shown to have perform poorly on several tasks, but follow up work [89] showed mere chain-of-thought prompting [49, 50J leads to huge boosts on all tasks. This indicates the evaluated models were in fact "competent", but inappropriate prompting led to undermining how "performant" they are.
3 Concept Space: A Framework for Analyzing Concept Learning Dynamies
We first formally introduce our framework of concept spaces. See Fig. 12 for schematic. Motivated by a rich body of literature on disentangled representation learning [23, 30, 54-60), this framework allows us to systematically analyze how different concepts underlying the data generating process and corresponding capabilities to manipulate them are learned during training. We highlight that since our primary empirical focus in this paper will be an abstraction of text-to-image generative models, parts of the framework will be specifically instantiated with text-to-image generation tasks in mind. To this end, we note our model class of interest is generative model F that is trained using conditioning information h to produce images y. For example, F may be instantiated using a diffusion model that uses embeddings h of textual descriptions of scene produce images y. We next define a concept space.
Definition 1. (Concept Space.) Consider an invertible data- gatta-genrrttig process G : 2 - X that samples vectors 2 ~ P(Z) a vector space Z C Rd and maps them to the observation space X E R We assume the sampling prior is factorizable, i.e., P(z E Z) = II P(zs). and individual dimensions of 2 correspond to semantically meaningful concepts. Then, concept space S is defined as the multidimensional space composed of all possible concept vectors 2, i.e., S = {z 2 ~ P(2))
As an example, consider a concept space defined using three concepts, say, 21 = shape, 22 = size, and 23 color that maps to a dataset of images with objects of different combinations of shapes, sizes, and colors (see Fig. 2). The assumption that concepts are independently distributed implics one can intervene from the dataset above, the concept color in an image can be altered by changing the value of the relevant latent variable and mapping it to the image space via the data-generating process. Now, using a mixing function M that yields conditioning information h := M(z), we can train a conditional generative model and define notion of capabilities relevant to our work as follows.
given concept without affecting the other ones. For example, given sample
3

---

Definition 2. (Capabilltty)) A concept class C denotes the set of concept vectors 2c such that a subset of dimensions of these vectors are fixed to predefined values. Classes C and C' are said to differ the 1nth concept vf E zc, there exists z' E =c with 2[k]] & z' z'[k] and z|i] = 2 for i & k. We say model possesses the "capability to alter the t concept" if for any class whose samples were seen during training, the model can produce samples from class C' that differs from C in the t concept.
Intuitively, the definition above comments whether the model can flexibly manipulate concepts of classes seen during training to produce samples from classes that were not seen, i.e., classes that are entirely out-of-distribution. As an example, consider concept space with shape, color, and size as concepts. If shape and color are fixed to circle and blue respectively, we get the class of blue circles; i..,, V2 Ablue circlee. the first and second dimension respectively take on values that correspond to the shape circle and color blue. Then, given a conditional diffusion model that was shown blue circles during training, will say this model possesses the capability alter the concept color if it can produce samples from concept classes with the same shape as circles, but different colors (e.g., red or green circles). Analyzing learning dynamics in the concept space will thus provide a direct lens into the model's capabilities as they are acquired.
We also note that the definition above is not dependent on the precise manner via which the model is prompted to elicit an output, i.., it need not be the case that the conditioning information h that is used for training the model is used to evaluate the model capability. In fact, in our experiments, try alternative protocols such as intervening on the latent representations to show that substantially before the model can be prompted using h to generate samples from an OOD concept class, it can generate samples from said class via such latent interventions. To this end, we also define a measure that assesses how much learning signal the data provides towards disentanglement of concept, and hence learning of a capability to manipulate it.
will
Definition 3. (Concept Signal.) The concept signal oz concept z; measures the sensitivity of the generatnnn process to change in the value of a concept variable, i.e., o; == (00())/oz.)
Intuitively, if the training objective is factorized at the granularity of concepts, concept signal indicates how much the model would benefit from learning about concept. For example, consider diffusion model trained using the MSE loss with conditioning h == 2 to predict the noise added to an image ((2). de is then merely the component of the loss representing how much change in conditioning yields changes in concept 2,, as it is represented in the image. Concept signal will thus serve as crucial knob in our experiments to analyze learning dynamics in concept space and the order in which concepts are learned.
3.1 Experimental and Evaluation Setup
Before proceeding further, we discuss our experimen- tal setup that concretely instantiates the formalization above. Our data-generating process is motivated by prior work disentanglement [54-60] and involves concept classes defined by three concepts, each with two values: color fred, blue), size (large, small), and shape=(circle, triangle}. In Sec. 4.1 and Sec. 4.4, use two attributes: color (with red labeled as and blue as 1) and size (large labeled as 0 and small as 1). We generate a total of 2048 images for each class, with objects randomly positioned within each image. We train models on classes 00 (large red circles), 01 (large blue circles), and 10 (small red circles), shown as blue nodes in Fig. 2. and test using class 11 (small blue circles), shown as pink nodes, to evaluate model's ability to manipulate con- cepts and generalize OOD (see App. B.1 for further details). In See. 5, we will restrict to two attributes, shape=(circle, triangle) and color = (red, blue), and study the effect of noisy conditioning, i.e., what happens when concepts are correlated in the conditioning information h due to some non-linear mixing
01
11
Color
Color
1
00
10
Concept spaces with differ- ent concept values see different concept signal. Consider a concept space com- prised of concepts s size and color. (Left) The color separation between the classes is stronger than the size separation, re- sulting stronger concept signal in the color dimension. (right) The size sepa- ration between the classes stronger, thus resulting in a stronger signal for size.
Figure 2:

---

function. For all experiments, we use a variational diffusion model [90] to generate 3 x 32 x 32 images conditioned on vectors (see App. B.2 for further training details).
Evaluation Metric. To assess whether a generated image matches the desired concept class without
human intervention, we follow literature on disentangled representation learning [23, 30, 54, 55, 91- 94] and train classifier probes for individual concepts using the diffusion model's training data. The probe architecture involves a U-Net [95] followed by an average pooling layer and n MLP classification heads for the concept variables. See App. B.2 for further details.
4 Learning Dynamics in Concept Space
4.1 Concept Signal Determines Learning Speed
We first demonstrate the utility of concept signal as a tool to gauge at what rate the model learns a concept and the capability to manipulate it. To this end, develop controlled variants of our data getterenngg process by changing the level of concept signal of each concept and train diffusion models conditioned with the latent concept vector z on them. We primar- ily focus on concepts color and size, altering their concept signal by respectively adjusting the RGB con- trast between red and blue and the size difference between large and small objects (see App. B.1 for details). We define the speed of learning each concept as inverse of the number of gradient steps required to reach 80% accuracy for class 11, i.c., the OOD class that requires learning the capability to manipulate concepts as seen during training, Results comparing different concept signals are shown in Fig. 3. For both color and size, observe that concept sig- nal dictates the speed at which individual concepts learned. We also find that when different concepts have varying strengths of concept signals, this leads to differences in the learning speed for each concept.
Spoed of leariing concest
Speed of learning concest
0.034
0.034
0.032
0.032
0.030
0.030
0.028
0.028
0.024
Color distance
Size distance
0.05 0.61 ab15 Pixel distance
0.003 0.004 Pixal distance
Concept signal determines learn- ing speed. The speed of concept learning as an inverse of the time in gradient steps when the separation in color (left) and size (right) between different classes increases. Concept learning is faster when pixel differ- ences among concept class and hence con- cepts are larger.
Figure 3:
4.2 Concept Signal Governs Generalization Dynamics
We next examine the model's learning dynamics in concept space under various levels of concept signal for the concepts color and size. For completeness, we evaluate a model's 's ability to generalize
(a)
generalization for class 00
(b)
OOD generalization
11
01
G-th quadient slep gradient stwp
Color concept signal
1.8
16
Size
Size
Leam
12
Leams donamics for C-th gradken: ahep
20000 gradiont stoy
D0
Color
10
00
Color
10
Figure 4: Concept signal governs generalization dynamics. (a) Learning dynamics in the concept space for in-distribution concept class 00 (bottom left). (b) Learning dynamics for out-of-distribution (OOD) concept class 11 (top right). We plot the accuracy for color on the x-axis and size on the y-axis. The 10,1) normalized color concept signal level color coded. Two trajectories for 01 and 10 are shown to illustrate concept memorization. See App. D.3 for uncertainties.
5

---

both in-distribution (ID) and OOD, but only the latter is deemed learning of a capability to manipulate a concept. Results are shown in Fig. 4, with panel (a) showing dynamies for learning to generate samples from II) class 00 and panel (b) showing dynamics for OOD class 11. Results on OOD generalization show concept memorization, which we define as the phenomenon where the model's generations on an OOD conditioning h are biased towards the training class that helps define the strongest concept signal. For example, for the unseen conditioning 11, the generations are more alike 01 when the concept signal is stronger for size (e.g., blue curve in Fig. 4(b)) and alike 10 when the signal is stronger for color (e.g., red curve). Interestingly, we observe that for settings with high imbalance in concept signals, e.g., the blue curve in Fig. 4 (b), the endpoint of concept memorization is very biased towards one training class, here 01, delaying its out-off-istribun (OOD) generalization. For the learning trajectories of all classes, see Fig. 15 in App. D.2.
Broadly, our results imply that an early stopped text-to-image model can witness concept memoriza- tion and hence simply associate an unseen conditioning to the nearest concept class when asked to generate OOD samples (see Kang et al. [96] for a similar result in L.MS). However, given sufficient time, the model will disentangle concepts underlying the data-generating process and learn to generate entirely novel, OOD samples. For futher evidence in this vein, we also confirm our results across more general scenarios, including with the real-worle CelebA dataset (App. D.4) and using three concept variables: color, background color, and size (App. D.5).
4.3 Towards a Landscape Theory of Learning Dynamics
(b)
Fig. 4 indicates models undergo phases of un- derstanding of concepts at different stages of training. In fact, an intriguing property of tra- jectories shown in Fig. 4 (b) is that there is sudden turn in the learning dynamics from con- cept memorization to OOD generalization (e.g.. see the top-left quadrant in Fig. 4 (b)). To in- vestigate this further, we propose a minimal toy model that captures the geometry of model's learning trajectories shown in that figure. Specif- ically, we use the following dynamics equation, d(t) = 2 + (a- 2) I+t- where is the target point we want to get to and E is the ini- tial, "biased" target. For example, consider the case with color and size concepts in Fig. 4 (b). The model's s generated samples are more alike class 01 and biased towards (ž1, x2) (0, 1)
Color 10 00
Lid or
10
Figure 5: A Phenomenological Model of Learn- ing Dynamics in Concept Space. Using Eq. 1, we simulate the learning trajectory for concept class 00 in panel (a) and the OOD class 11 in panel (b). Initially, target values are set at (0, 1) (1.0) based on the concept signal strengths for color or size, respectively. As the model progressively learns each concept, the target values gradually shift towards 11, 1). This simple toy model accu- rately reproduces the observed curves in Fig. 3(c), which arise from concept memorization.
when the size concept signal 02 is stronger than 01 and to 10, (21, 22) (1,0) when the color concept signal 01 is stronger than 02. We define the learning to head towards (e.g.. 11 for OOD generalization). Based on this framework, we can derive the following energy function.
(21, 822 as the target values or directions we want
dz
2a (d(1 (1) -21 =1)² ==)²+
22)² if > 02 (1 )2 + + (d(1-12) 22) otherwise.
+ ( =2
di = VZL, (2)1 =
(I)
Here, is determined by the difference |0] 02ه and & and iz denote the times when the model learns concepts 21 and 22, respectively. Fig. 5 illustrates the simulated trajectories for classes 00 and 11. based on Eq. 1. Pancls (a) and (b) correspond to classes 00 and 11, respectively. We define the actual target points (2), 82) as (0, 0) for class 00 and (1, 1) for class 11. For the initial targets (ă1, 22). we set both values to (0, 0) for class 00. For class 11, the targets are set to (1, when 01 02 and to (0, 1) when 01 < 02. We find our toy model effectively captures the actual learning dynamics for both in-distriuttinn (Fig. 3(b)) and nut-of-distribmn
(OOD) concept classes (Fig. 3(c)). Notably, our simulation accurately replicates the two types of curves: clockwise (blue trajectory in Fig. 3(b)) and counterclockwise (red trajectory).
An important conclusion that follows from the results above is that the network's learning dynamics can be precisely decomposed into two stages, hence yielding the sudden turns seen in trajectories in

---

(a)
(b)
(c)
1.00
1.00
1,00
0.75
0.75
Accuracy
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
5000
10000 15000 20000
0.00
10000 15000 20000
0.00
5000
10000 15000 20000
Number of gredient steps
Number gradient steps
Number gradient steps
Figure 6: Emergence of hidden capabilities. We plot accuracy function of gradient steps for five different runs, using three different protocols for prompting the model to generate outputs for OOD concept classes. (a) The baseline, naive prompting protocol; (b) linear latent intervention, applied in the activation space; and (c) overprompting, akin to an intervention on the input space.
Fig. 4. We hypothesize there is phase change underlying this decomposition and the model acquires the capability to alter concepts at this point of phase change. We investigate this next.
4.4 Sudden Transitions in Concept Learning Dynamics
The concept space visualization of learning dynamics observed in Fig. 4 (b) and our toy models analysis in Fig. 5 indicate that there exists a phase in which the model departs from concept mem- orization and disentangles each of the concepts, but still produces incorrect images. We claim that at the point of departure, the model has in fact already disentangled concepts underlying the data-gen generatnng process and acquired the relevant capabilities to manipulate them, hence yielding change of direction in its learning trajectory. However, naive input prompting is insufficient to elicit these capabilities and generate samples from OOD classes, giving the impression the model is not yet "competent". This then leads to the second phase in the learning dynamics, wherein an alignment between the input space and underlying concept representations is learned. We take the model corresponding to the second from left curve (the green curve) in Fig. 4 (b) to investigate this claim detail. Specifically, given intermittent checkpoints along the model's learning trajectory, we use the following two protocols for prompting the model produce images corresponding to the class 11 (blue, small). See App. C for further details.
Activation Space: Linear Latent Intervention. Given conditioning vectors h, during inference we add subtract components that correspond to specific concepts (e.g., haaue) Overprompting. We simply enhance the color conditioning to values of higher magnitude, e.g. (r, g, b) (0.4, 0.4, 0.6) to (0.3, 0.3, 0.7).
2. Input Space:
Fig. 6 shows the accuracy for five independent runs under: (a) naive input prompting, (b) linear latent interventions, and (c) overprompting. In Fig. 6 (a), we observe that some runs can produce samples from the target concept class (blue, small) with 100% accuracy after around 8,000 gradient steps, while other runs fail to do so. However, in Fig. 6 (b, c), we find alternative protocols for prompting the model can consistently elicit the desired outputs much earlier than input prompting, e.g., at around as early as 6,000 gradient steps. This indicates the model does possess the capubility to alter concepts and generalize OOD! Furthermore, we note that different protocols enable elicitation the capability at approximately the same number of gradient sleps, irrespective of the seed, and that this is precisely the point of sudden turn in the learning dynamics in Fig. 4! Interestingly, experiments with Classifier Free Guidance (CFG) [97] show that CFG only becomes effective after this transition (Fig. 21).
We further explore the second phase learning in Appendix D.7 by patching the embedding module used for processing the conditioning information from the final checkpoint to an intermediate one. Our results show that when the final checkpoint does enable use of naive input prompting for eliciting a capability, the embedding module can be patched to an intermediate checkpoint and we can retrieve the desired output at approximately the same time that alternative prompting protocols start to work well. This suggests the second phase of learning primarily involves aligning the input space to intermediate representations that enable eliciting the model capabilities. Overall, our results above yield the following hypothesis.
7

---

(a)
(b)
Latent liner intervention
Generated w/
Tea (000) G0t Cxpability
Nalve prompting (baselina)
Finer Intervoction
0.8
With
0.4
Nakee promptinng
0.0
Number of gradient steps 1e6
Thaning
Figure 7: Validating our Findings on CelebA. (a) Concept space dynamics of generated images concepts Gender and With Hat. We find the generalization class (Female, With Hat) ends up (Female, No Hat). (b) Compositional Generalization Accuracy when performing latent interventions naive prompting. This rise of accuracy near x 10 gradient steps clearly demonstrates that the model is capable generalizing out-of-distribution, but is not performant when evaluated via naive prompting.
Hypothesis 1. (Emergence of Hidden Capabilities.) Generative models possess hidden capabilities that are learned suddenly and consistently during training, but naive input prompting may not elicit these capabilities, hence hiding how "competent" the model actually is.
4.5 Additional Results on Realistic Data
To provide further support for our hypothesis, we provide results in more realistic and natural- istic data. Specifically, we use the CelebA dataset [98]. which contains fine-grained attributes corresponding to concepts like Gender, With Hat, and Smil Smiling, and analyze two settings.
Using concepts Gender and Smiling, Results are in Fig. 18, where find that concept learning dynamics discovered in the relatively simple setting in prior sections generalizes. We see that there is first phase of concept memorization, wherein the model learns generate images of (Female, ong). However, as training proceeds, there sudden turn in the learning dynamics and the model learns generalize out-of-distribution.
Using concepts Gender and With Hat. In this setting, given a compute budget of 2M gradient steps, we find that the model seem never learn generalize outst--disttibuttinn (see Fig. 7). However, when we perform latent interventions on the model's representations, we are able to force the model to produce images from the class (Female, With Hat). These results reiterate that the model is indeed capable of generalizing out-of tatto--distrbbttinn but is not performant when naively prompted.
5 Effect of Underspecification on Learning Dynamics in Concept Space
In the results above, we use conditioning information that per-
to tearning concopt
fectly specifies concepts underlying the tane-gnnn process, i.e., h z. In practice, however, instructions are underspec- ified and one can thus expect correlations between concepts in the conditioning information extracted from those instruc- tions [99-103]. For example, images of strawberry are often correlated with the color red (see Fig. 9()). Corre- spondingly, unless a text-to-image model is shown explicit data stating "red strawberry" images of non-red strawberries, the model's ability to disentangle the concept color from the concept strawberry may be impeded (see generations for "yellow strawberry" in Fig. 9). Motivated by this, we next investigate the effects of using underspecified conditioning in- formation on a model's 's ability to learn concepts and capabilities manipulate them.
1000
800
800
20
Percentage masked
Figure 8: Underspecification de- lays out-of-distribation (OOD) generalization. The number of gradient steps required to reach ac- curacy above 0.8, as the percent- age of masked prompts increases. A higher proportion of masked prompts slows down the speed of concept learning.
Experimental setup. The data generation and evaluation process follows the protocol described in Sec. 3.1. We select

---

(a)
rad strwebary"
"yellw strawburyy
(b)
Level onder-pecfiiation
0%% masked
40% masked
70% marked
100% masked
Shope
"odd applee
"yellow apple"
00
10
00
10
Figure 9: Underspecification and Concept Learning, (a) The sfate-of-the-arr generative mod- els [104] erroneously produces a red strawberry (top right corner) for the prompt "yellow strawberry". (b) Without underspecification in the training data, a model F' accurately learns the concepts of shape and color, successfully generalizes to the unseen node blue triangle (leftmost). As masks are applied to the word red for the prompt red triangle, concept signal for triangle increasingly starts to correlate with the concept red. This causes the output images to change from blue to purple as the level of masking increases (panels left to right). Eventually, the color dimension for triangle collapses, biasing the model towards generating solely red triangles (rightmost).
(a) Maskedt: 100% 80% 60% 50% 40% 20% 0%
(b)
Percentage of mask g
Shape
Shape
200d6tth goudect day
00
Color
Color
Figure 10: Underspecification hinders -utto---dittiiuutin (OOD) generalization. (a) The learning dynamics with varying levels of prompt masking, from 0% to 100% and the generated images. At 0% masking (top right image), the model correctly produces an image of blue triangle from the prompt "blue triangle." " As the masking increases (from right to left), the images gradually shift towards the incorrect color, red. (b) The simulation of the learning dynamics under underspecification in concept space based on Eq. 2. Our toy model replicates a trained network's 's learning dynamics.
color (red and blue) and shape (circle and triangle) as the concepts, drawing an analogy to the "yellow strawberry" example. To simulate underspecification, we randomly select training sam- ples that have specific combination of shape and color (e.g.. "red triangle"). We then mask the token representing the color ("red") and train the model on three concept classes (00, 01, 10), represented by blue nodes in Fig. 9, with some prompts masked. We test using class 11 (pink node) with no prompts masked to see if the model can generalize OOD.
Fig, 8 shows how underspecificatioon (masked prompts) affects the speed of concept learning. We see that as the percentage of masked prompts increases, the speed of learning concept decreases, suggesting that underspecification leads to slower learning of concepts. Further, Fig. 10 (a) shows models' learning dynamics in concept space at varying levels of underspecification. With 0% masking, the model accurately produces an image of blue triangle (see Fig. (b)). However, as the percentage of masking increases, the color of generated images shifts from blue to purple (middle), and finally red. This demonstrates that when prompts are masked, the model's understanding of shape triangle becomes intertwined with color red; even when blue is specified in the prompt, the dynamics are biased towards red.
Underspecification Delays and Hinders OOD generalization.
Toy Model of Learning Dynamies with Underspeciffcation. When prompts are masked (i.e., underspecification occurs), target values for the concept variables are shifted: e.g., in our setup, with no mask applied, the target directions for the class is (1, 1). When the word "red" in "red tri triangle" is fully masked, the target shifts to (0. 1). Assuming this shift is linear with respect to the percentage of masked prompts a, we can derive the following energy function.
dz dt = V.L, L(21,22) = ((1 - sa) 21)² + (1 - ==2²²
(2)
9

---

In the above. parameter represents the impact of underspecification. Fig. 10 (b) shows the simulation of model behavior in the concept space according to Eq. (s 0.01). As the masking level increases, the target directions shift from 22 = in the top right corner to =2 = 0 in the top left corner. Our simulated dynamics thus match well with the model's 's learning dynamics shown in Fig. 10 (a).
Influence of Underspecification on Emergence of Hidden Capabilities. Following Sec. 4.4, we also explore whether it is possible to elicit the desired out- puts from a model trained with underspecified data. Fig. 11 shows the accuracy results over five runs (a) without using any prompting method, and (b) using over- ever-prompting In both scenarios, the percentage of masking is set at 50%. Results clearly demonstrate that with crommmppp,,ng the model achieves 100% accu- racy after approximately 1,000 gradient steps, whereas without over- vee-promptinng it fails in three out of five runs even after 2,000 gradient steps. These findings con- firm that capability can develop prior observable behavior, even in cases of underspecification.
(a] 1.5
(b) 1.0
Accurey
0.5
0.6
1000
2000
1000 2000
Mumser of grodione Stons
Nummber gnddertstpp
Figure 11: Underspecification and Hidden Capabilities. We use the verrprommping protocol from Sec. 4.4 to assess if the ca- pabilities to enable OOD generalization are learned before naive input prompting. (a) Ac- curacy for OOD generalization across five different seed runs under naive prompting; (b) Accuracy under over over-promnn seee-prpppinng with a fixed masking percentage of 50%.
6 Discussion
Why study the concept space? One might ask why concept space could be useful beyond synthetic datasets. In Fig. 14, we show the loss, accuracy, and training trajectory in concept space of a model from Fig. 6. The moment during training when the model acquires the capability to manipulate size and color independently is not evident from either the loss or accuracy curve. However, in concept space, one can directly see the divergence of the trajectory from concept memorization. Benchmarking generative models is challenging task, still often involving humans in the loop [105, 106). Our concept space framework suggests that benchmarking core generalization capabilities can potentially be reduced to monitoring the learning trajectory in concept space. Moreover, this information can be used to develop better training schemes as discussed in Ren et al. [107].
Concept learning vs. grokking We make distinction between the potentially delayed elicitation of capabilities our model versus grokking [108]. The phenomenology of delayed increase in performance on the test set, seen in Fig. 14, is shared. However, we deal with out-of-distribution evaluations, which is different from setups like modular arithmetic or polynomial regression in which grokking is usually studied [108-112] Secondly, research grokking using hidden progress measures indicates that the model gradually builds representations toward an ideal one [111, 1 113]; however, our results find that even at the level of latent representations, there is sudden emergence whereby the model learns the capability to manipulate concepts underlying the data-generating process and generalize OOD.
Is concept learning a phase transition? In Sec. 4.4, we have demonstrated that before the capability manipulate a concept is learned, the desired outputs cannot be generated regardless of the protocol used to prompt the model. Moreover, we showed that different protocols yield the desired outputs at the exact same time, which is also highly independent of model initialization (Fig. 6, Fig. 17). We thus hypothesize that: Concept learning is a well-controlled phase transition in model capability. However, the ability elicit this capability through predefined, single-input prompting can be delayed arbitrarily, depending on factors like the strength of the concept signal.
Limitations One limitation of our work is that the compositional setup used in this study is rather simple. Real world concepts are often hierarchical [114, 115] or relational [116]. We also make assumptions that concepts are linearly embedded in the vector space Z in Sec. 3. However, this assumption seems to be at least partially justified as seen in Sec. D.8. Another limitation is that the findings are mainly drawn from synthetic data. While we show that our findings generalize to realistic data to some extent, as seen in Sec 4.5, Fig. 18 and Fig. 7, further studies are needed to investigate how our findings apply to real data in general.
10

---

Acknowledgments and Disclosure of Funding
CFP and HT gratefully acknowledges the support of Aravinthan D.T. Samuel. CFP, MO, ESL and HT supported by NTT Research under the CBS-NTT Physics of Intelligence program. ESL's time University of Michigan was partially supported by the National Science Foundation and CNS-221 1509). The computations in this paper were run the FASRC cluster supported by the FAS Division of Science Research Computing Group at Harvard University. The authors thanks Zechen Zhang, Helena Casademunt, Carolina Cuesta-I Lazaro, Shivam Raval, Yongyi Yang and Itamar Pres for useful discussions.
151
Contributions
Motivated by prior work from MO, ESL and HT [30], CFP and HT started investigating the order in which concepts are learned by diffusion model, leading to CFP identifying the notion of concept signal and HT proposing to study the learning dynamics in concept space. This kickstarted the project. ESL and HT defined the formal setup (Sec. 3). ESL hypothesized the point of sudden turn marks the emergence of hidden abilities, hence defining the project narrative around competence versus performance. AL and CFP led the experimental verification of this hypothesis. AL led identification of linear representation of concepts and performed the latent and MLP intervention, while CFP contributed to the input space one. MO and HT led formulation of the landscape theory (Sec. 4.3), with inputs from CFP. MO led the underspecification picture of learning dynamies, with inputs from ESL and HT (Sec. 5). Paper writing was led by MO and ESL, with inputs from all authors. Visualizations were mainly led by MO, with inputs from HT and CFP. Experiments on CelebA learning dynamics were conducted by CFP in discussions with MO and corresponding latent intervention were led by AL.
References
[1] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276. 2022.
[2] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openee com/research/ video piteooggneration. models- aass-as-woo as-world- simulators.
[3] Dan Kondratyuk, Lijun Yu, Xiuye Gu. José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, al. Vidcopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312. 14/25, 2023.
[4] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300
2022.
[5] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[6] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using /4988, 229.22299914988. 2022.
2d diffusion. arXiv preprint arXiv:2209,
[7] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin Brrtinn Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405. 10334 2024.
[8] Ziyang Chen, Daniel Geng, and Andrew Owens. Images that sound: Composing images and sounds single canvas. arXiv preprint arXiv:2405.12221, 2024.

---

[9] Yilun Du, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints, pages arXiv-2302, 2023.
[10] Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua Tenenbaum, et al. Video language planning. arXiv preprini arXiv:2310. 10625, 2023.
[1]] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald. Chris Apps, al. Genie: Generative interactive environments. arXiv preprint arXiv:2402.15334 2024.
[12] Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David Lobell, and Stefano Ermon. Diffusionsat: A generative foundation model for satellite imagery. arXiv preprint arXiv:2312.03600 2023.
[13] Yan Zhuang (New York Times). Imran Khan's 'Victory Speech' From Jail Shows A.I.s Peril and Promise, 2024. https: http::::///w..ttmmme...... imran- aeaan-artfficial herrraiiiialanneellleeeeeeeeee haw.aaannnneeeleeeeeaeaiiiieee html.
[14] Mark Sullivan (FastCompany). tion season begins, 2024.
Al deepfakes get very real as 2024 elec-
https://ww..fasccomaaay cwtc /00000177777
ai Li-depffakes-tee tareesstttloors-wwii sorll-o--iit--o----bboo-o biden- 2022- election.
[15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXiv:2303..2771 2023.
[16] Gemini Team. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context. Technical report, Google DeepMind, 2024. https://storn: htt.:/ttrrag.ggoglaappsss.s ieweiiidddedii/ggmmninii
googleapis com/
v1_ 5_rporrt....... pdf.
[17] OpenAl. GPT-4 System Card. Technical report, OpenAl, 2023. https://cdn. openai com/ hopers/gpt-4ee httpe//gppt--ssttteeeerreee card, pdf.
[18] Claude team. Introducing the next generation Claude. Technical report, Anthropic AI, 2024. www/nwws/ httews/clause 3-
https://www. wwt.:www.thopi....omm
[19] Generative Media Team (Google Deepmind). Generating audio for video, 2024. https:
htwp://n...ooooo/eiiooooeeeo htw.a/ooorrrbbboooggeeraiiinngn wtw.gerrttiiignnio-oooooo---- modio-for-vided..
[20] Shengchao Liu, Yanjing Li, Zhuoxinran Li, Anthony Gitter, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Arvind Ramanathan, Chaowei Xiao, et al. A text-guided protein design framework. arXiv preprint arXiv:2302.04611 2023.
[21] MIT News.
Speeding generative models, 2024.
discovery with diffussion https://news wwt.mmite.u. edu/203/ speeding-drug diiug-ddccvvry-eee wotthh-fffsion -nnerratvee-mdell-ddiii oww.:raffftttreeeeeteiiieeeaddod
up
drug
[22] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[23] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proc. int. conf. machine learning (ICML), 2019.
[24] Jivat Neet Kaur, Emre Kiciman, and Amit Sharma. Modeling the data-generating process is necessary for out-of-distribution generalization. arXiv preprint. 2rXin:22006.78837, 2022.
[25] Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, al. Can machines learn morality? the delphi experiment. arXiv e-prints, pages arXiv-2110, 2021.
12

---

[26] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprins. arXiv:2102. 10000.//1100 1107, 2021.
[27] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[28] Jean Kaddour, Aengus Lynch, Qi Liu, Matt Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206. 15475, 2022.
[29] Adítya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint (xXiv:204400/255, 2022.
[30] Maya Okawa, Ekdeep Singh Lubana, Robert P. Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicativel:: Exploring diffusion models on a synthetic task, 2024.
[31] Hu Yu, Hao Luo, Fan Wang, and Feng Zhao, Uncovering the text embedding in text-to-image diffusion models. arXiv preprint ar 4rX1v:2404.01154, 2024.
[32] Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P Xing. Yuejie Chi, and Kun Zhang. Learning discrete concepts in latent hierarchical models. arXiv preprint arXiv:2406.005 0091,, 2024.
[33] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2023.
[34] Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311.12092. 2023.
[35] Nan Liu, Yilun Du Shuang Li, Joshua B Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2095, 2023.
[36] Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. arXiv preprint. arXiv:1812.032253, 2018.
[37] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. Advances in Neural Information Processing Systems, 36, 2024.
[38] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint SKKiv:2404.041255 2024.
[39] Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208 00005, 2022.
[40] Colin Conwell and Tomer Ullman. A comprehensive benchmark of human-like relational reasoning for text-to-image foundation models. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
[41] Evelina Leivada, Elliot Murphy, and Gary Marcus. Dall-e 2 fails to reliably capture common syntactic processes. arXiv preprint arXiv:2210.12880 2022.
[42] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vinect, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015 2022.
[43] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. arXiv preprint arXiv:2110.1105,, 2021.
13

---

[44] Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 is seeing double: flaws in word-to- concept mapping in text2image models. arXiv preprint arXiv:2210.10600 2022.
[45] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint (rXX::2200.1088, 2022.
[46] Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R Manmatha, Ashwin Swaminathan, Zhuowen Tu, Stefano Ermon, and Stefano Soatto. On the scalability of diffusion- based text-to-image generation. arXiv preprint arXiv:2404.0288. 2024.
[47] Raphaël Millière and Cameron Buckner. A philosophical introduction to language models part i: Continuity with classic debates, 2024.
[48] Raphaël Millière and Cameron Buckner. A philosophical introduction to language models-part ii: The way forward. arXiv preprint arXiv:2405.03207, 2024.
[49] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, 2022.
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint arXiv:2201.1 1903, 2022.
[51] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive Ilms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024.
[52] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. In The Eleventh International Conference on Learning Representations, 2023. URL https:// https://openrevieer http///ppereeee...omme net. fent:/form?iin ditpum/dddddddPPuBdeee
[53] Sébastien Bubeck. Sparks of artificial general intelligence: Early experiments with GPT-4 (Talk), 2023. URL https:ttp///// http:///wwwwooubbeee/ee com/ http://tttt/===11/JPPPBB
[54] Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International Conference on Machine Learning, pages 2649-2658. PMLR, 2018.
[55] Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? Adv. in Neural Information Processing Systems (NeurIPS), 2019.
[56] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), 2017.
[57] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Adv. in Neural Information Processing Systems (NeurIPS), 2016.
[58] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd Int. Conf. on Artificial Intelligence and Statistics (AISTATS). 2019.
[59] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Adv. in Neural Information Processing Systems (NeurIPS), 2021.
[60] Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Adv. Neural Information Processing Systems (NeurIPS), 2021.
14

---

[61] Yi Ren, Samuel Lavoie. Mikhail Galkin, Danica J. Sutherland, and Aaron Courville. Improving compositional generalization using iterated learning and simplicial embeddings. 2023. URL https://arxie: www..orgggaa.... wwrggabs/233100 231./11.171
[62] Susan Carey. The origin of concepts. Journal of Cognition and Development, 1(1):37-41, 2000.
[63] Susan Carey, Knowledge acquisition: Enrichment or conceptual change. The epigenesis of mind: Essays on biology and cognition, pages 257-291, 1991.
[64] Susan Carey. The origin and evolution of everyday concepts. Cognitive models of science, 15: 89-128, 1992.
[65] Susan Carey and Elizabeth Spelke. Domain-specific knowledge and conceptual change. Mapping the mind: Domain specificity in cognition and culture, 169:200, 1994.
[66] Susan Carey. Bootstrapping & the origin of concepts. Daedalus, 133(1):59 68, 2004.
[67] Susan Carey, Where our number concepts come from. The Journal of philosophy, 106(4):220,
2009,
[68] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06224, 2023.
[69] Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
[70] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01616, 2023.
[71] Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.022077 2023.
[72] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406. 11717, 2024.
[73] Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Under- standing how language models process false demonstrations. arXiv preprint arXiv:2307.09446, 2023.
[74] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models, 2024. URL https::/ www./oo/////aa.eeee 03658.
[75] René Haas, Inbar Huberman-Spiegelghhe Rotem Mulayoff, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. arXiv preprint arXiv:2303.1 1003, 2023.
[76] Yong-Hyun Park, Mingi Kwon, Junghyo Jo, and Youngjung Uh. Unsupervised discovery of semantic latent directions in diffusion models. arXiv preprint 2rXiv:2302.12469, 2023.
[77] Samyadeep Basu, Keivan Rezaci, Ryan Rossi, Cherry Zhao, Vlad Morariu, Varun Manjunatha, and Soheil Feizi. On mechanistic knowledge localization in text-to-image generative models. arXiv preprint arXiv:2405.0100 2024.
[78] Yida Chen, Fernanda Viégas, and Martin Wattenberg. Beyond surface statistics: Scene representations latent diffusion model. arXiv preprint arXiv:2306.05720, 2023.
[79] Mohamed El Banani, Amit Raj, Kevis- Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 21795-21800 2024.
15

---

[80] Noam Chomsky, Aspects of the Theory of Syntax. Number 11. MIT press, 2014.
[81] John Collins. Linguistic competence without knowledge of language. Philosophy Compass, (6):880-895, 2007.
[82] Jill G De Villiers and Peter De Villiers. Competence and performance child language: Are children really competent judge? Journal of Child Language, 1(1):11-22, 1974.
[83] Gillian Brown, Kirsten Malmkjær, and John Williams. Performance and competence in second
language acquisition. Cambridge university press, 1996.
[84] METR. Guidelines for Capabilitiies Elicitation, 2024. https://metr https:://metr.ithhi/ io/
[85] Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability elicitation with password-locked models. arXiv preprint arXiv:2405. 2024.
[86] Tom Davidson, Jean-Stanislas Denain, Pablo Villalobos, and Guillem Bas. Ai capabilities can be significantly improved without expensive retraining. arXiv preprint arXiv:23/2.07413,
2023.
[87] Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black- box access is insufficient for rigorous ai audits. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2254-2273 2024.
[88] Aarohi Srivastava, Abhinav Rastogi, Abhishck Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Marria--Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0466 2022.
[89] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sehastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint 100.9.22 2022.
[90] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023.
[91] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Proc. Int. Conf. on Learning Representations (ICLR), 2017.
[92] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[93] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[94] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv: 1711.00848. 2017.
[95] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[96] Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar fine- tuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612 2024.
[97] Jonathan and Tim Salimans. Classifier-free diffusion guidance. 17X1v:2207.12598, 2022.
arXiv preprint
16

---

[98] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[99] Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. Underspecification in scene descrippion-to-depiiiioo tasks, 2022.
[100] Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, al. Un- derspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226):1-61. 2022.
[101] Sandro Pezzelle. Dealing with semantic underspecification in multimodal nip. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
[102] Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and Ilker Yildirim. When are lemons purple? the concept association bias of vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[103] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: /tstttss:/ppenee http:///ppeeeeeeeeeeeeeeeee wwwperveiieeeeeeeeeeeeeeeeeeeeee wtt..e/eooooomad-AAAAAAaaaa Ad=AOKK4nRwwW..
[104) Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
[105] Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S. Bernstein. Hype: A benchmark for human eye perceptual evaluation of gen- erative models, 2019.
[106] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding, 2022.
[107] Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths, 2022. URL https: //arxiv...ii/////aa . org/ abs/ /200//220...22 02485.
[108] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
[109] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
[110] Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference Learning Representations, 2022.
[111] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. 2023.
[112] Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy rich training dynamies. arXiv preprint arXiv:23 2023.
[113] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit, Advances Neural Information Processing Systems, 35:21750-2177 2022.
[114] Blaž Zupan, Marko Bohanec, Janez. Demšar, and Ivan Bratko. Learning by discovering concept hierarchies. Artificial Intelligence, 109(1-2)211-242 1999.
17

---

[115] Yangqing Jia, Joshua T Abbott, Joseph L Austerweil, Tom Griffiths, and Trevor Darrell. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. Advances in Neural Information Processing Systems, 26, 2013.
[116] Martin Wattenberg and Fernanda B Viégas. Relational composition in neural networks: A survey and call action. arXiv preprint arXiv:2407.14662, 2024.
[117] Core Francisco Park, Victoria Ono, Nayantara Mudur, Yueying Ni, and Carolina Probabilistic reconstruction of dark matter fields from biased tracers using diffusion models, 2023.
[118] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[119] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization, 2016.
[120] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.
[121] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
[122] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[123] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR.
2015.
[124] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library, 2019.
[125] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[126] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models, 2022.
18

---

A Schematic for Concept Space
Fig. 12 illustrates the concept space framework discussed in Sec. 3.
X
S
size shape color location
g
M
Sample
, x
h x
big.
blue.
circle
<irole
bgrbeige,
10006-0.1,0.1)
The concept space framework. 9 is an invertible data-generating that maps process maps sampled vectors z ~ P(Z) (where Z C. IRd) to an observation x IR" (an image in this work). The concept space, S == {=|= P(Z)), is defined generated from a compositional prior P(Z). The mixing function M masks some concept variables, the masked concept variables are thus underspecified.
Figure 12:
space of all possible concept vectors
B Experimental Details
B.1 Synthetic data
We design a data generation process (DGP) compatible with the concept space framework intro- duced Sec. Our full DGP has six concepts: shape=(circle, triangle), x coordinated R, coordinate IR, color=(red, blue), size= small), and background color=(bright, dark) We generally explore only a subset of these at a given time in our experiments.
In Sec. 4, we fix shape=circle and coordinate, masked out. Thus the conditioning vector h only specifies color=(red, blue) and size=(big, small}. We sample both of these concept variables from a mixture of two uniform distri- butions, one component for each class (big, small) and (red, blue). Each value each dimension is sampled from (m² m + s), where m' is the class dependent mean. For example, color-red could be sampled from (0.077 0.9) x (0011, 0.3) x (0.11, 0.3), where m. (0.8, 0.2, 0.2) For brevity, we name the four resulting classes "00", "01". "10", and "11", where the class "1 1" is kept as the unseen target to evaluate out-of-distribution (OOD) generalization. For each training run, DGP was initialized with set random seed and 2048 images were generated in each class.
coordinate, background color
In Sec. 5, we used two concept variables, shape and size. The training dataset included the classes "00", "01", and "10". For evaluation, we used the class "11". For each class, we generated a total of 1,000 images, each featuring objects of varying positions and sizes ensure variability in our dataset.
In App. D.5, we add the concept variable background colorE R3 to have a three dimensional concept space of (color, size, background color). In this the classes ("000", "001", "010", "100") and the test classes are ("011", "101", "1 "110", "111")
training data is composed of
Our DGP is resolution agnostic, but we work with square images of 32x32 for fast experiments.
Adjusting concept signal To adjust concept signals, we vary the class dependent mean of the two components of the mixture distributions. For example a strong color concept signal is achieved by drawing color=red from the mean red (0.9, 0.1, 0.1) and color=blue from the mean m blue (0.1, 0.1, 0.9) whereas a weak color concept signal can be drawn using red (0.6. 0.4, 0.4) and m blue (0.4, 0.4, 0.6). We scale the standard deviation of each component by the same ratio the
19

---

01
Sicc
Color
Color
00 10
Figure 13: Different distributions in concept values result in different concept signal. (Left) The color separation between the classes is stronger than the size separations resulting in a stronger concept signal in the color dimension. (right) The size separation between the classes is stronger, thus resulting in a stronger concept signal in size
difference between the class means has been scaled. This latter is important to avoid the model's natural interpolation or extrapolation abilities from confounding with the generalization dynamics.
Fig. 13 illustrates two datasets with varying concept signals. Fig, 13 (left) illustrates case where the color concept signal stronger than the size concept signal, while Fig. 13 (right) illustrates the inverse case where the size concept signal is stronger.
B.2 Model Details
Model Architecture We used network architecture similar to the U-Net [95] used in Variational Diffusion Models [90]. We used the implementation publicly available in [117]. The architecture has ResNet [1 [18]] blocks before each downsampling layer and has self-attention layer in the bottleneck layer of the network. The U-Net has 64, 128, 256 channels at each resolution levels, an has a LayerNorm [1 19] and a GELU [120] activation. We used a sinusoidal timestep embedding [121] of 64 dimensions followed by 2-layer MLP with hidden dimension 256 and conditioning vector embedding using a 2-yayer MLP with hidden dimensions 256 independent in every residual block.
Optimizer We use the AdamW W [122] optimizer with learning rate 0.001 and weight decay 0.01 to optimize the parameters of our network. We train our networks for 20K gradient steps. We use the default values for the decay rates: 31 = 0.9, 32 0.999.
Classifier Free Guidance At training time drop the conditioning with probability 0.2. This unconventional choice was made instead of the dropping it to the null vector since the null vector the conditioning. The Wefg parameter is used to estimate the noise in the diffusion process by: (0) + We/g (fo( f=(x))))). where denotes parameters of the network f.
- -1.0 filled null vector
near the interpolation limit of our model in the color subspace of
é
Diffusion Process Hyperparameters We used the continuous time variational diffusion model [90] in order (i) keep the sampling step parameter T an inference time hyperparameter and (ii) allow the model to the adjust its optimal noise schedule for these images, especially since our synthetic images are expected to be different in SNR from natural images. In particular,
initialize
network with %/2 = 5.00 and 7/ = 10.0 (see [90] for the definition of ?) and use a learned linear schedule of (t). We use reconstruction loss corresponding to negative log likelihood from standard normal with a = 10-3 centered at the data for the first step of the diffusion process.
Evaluation probe Details. We used a U-Net [95] backbone with 64 output channels followed by max pooling layer and nx 1-layer MLP classifier for each the n classes to estimate each concept of an image independently. We sample from the same data distribution but with maximal data diversity, ie.. with s; values App. B. maximized within the range allowing perfect classification (no overlap of between classes). We sample 4096 images per class from the DGP and train the classifier for 10K gradient steps with Adam [122] and achieve a 100% accuracy on the held out test set. At evaluation phase, average the classifier softmax output over data generation model initialization seeds and 32 inference samples to construct the concept space representation of the generations.
Hyperparameter Search We conducted hyperparameter search, testing batch sizes from 32 to 256. number of channels per layer from 64 to 512, learning rates between 10-4 and 10-3 the number of steps in the diffusion process from 100 to 400, weight decay between 3 x 10 and 5 x 10
20

---

and model weight initialization scale between (0, 0.003) and (0 1). We also tried the Adam optimizer [123] with 31 0.9, $z 0.99, and weight decay of 10 5 No qualitative change were found in our results.
Computational Details. We implement our models in PyTorch [124]. The diffusion model was trained on four Nvidia A100 GPUs and 64 CPUs for the data generating process, A standard model run (e.g., in Sec. 4.2) took 20 minutes on single NVIDIA A 100 40GB GPU. The CelebA runs took ~ 24 hours on the same GPU. The full research required roughly 500 GPU hours, while the experiments in the paper would require roughly 100 GPU hours.
B.3 Code Availability
Our code is available at http:///ghtttbboooocccaaaa//o/oe
C Details on Alternative Protocols for Eliciting Model Capabilities
Input Space: Overprompting. Our model is trained on distribution of conditioning vectors centered around a class dependent mean: (2,) = U(m) $,, mÿ. + 8,) for each class i. We prompt the model with conditioning vectors extrapolated in the direction mz -ig- For instance, assuming the red conditioning was (0.6, 0.4, 0.4) and the blue conditioning was (0.4, 0.6), we "prompt" the model with (0.2, 0.2, 0.8) for blue. practice, we use 5 conditionings (0.4, 0.4, 0.6), (0.35, 0.35, 0.65), (0.25, 0.25, 0.75), (0.15, 0.15, 0.85), (0.05, 0.05, 0.95), and report the maximum joint accuracy.
Activation Space: Linear Latent Intervention. We demonstrate the ability to compose capabilities by manipulating the conditional vectors h. Namely, we create a condition vector h2 for a specific concept by specifying concept of interest (e.g., Praue = M (Zsiue. During the forward pass, given h, we can compute the component of each concept in h by projecting onto a specific concept-condition vector (hauwe). We can then enhance or reduce the component of each concept by scaling each of these projected components. practice, we perform the following operation: hí' + albbuue - BBharge where a, 3 are hyperparameters. We sweep over [0.1, 1, 4] for a and [0.1, 0.25, 0.5, for B.
Tuuuue is constructed by first deriving a blue direction in condition embedding space (h5, (/5.5.95) by embedding a concept vector (25,,95) where its RGB components are set to (0.05, 0.05, 0.95): T55.5.05 = M (55,5.055. We then project hi onto this direction to derive hisuue Fraagee is generated similarly using 0. The model then generates an image conditioned on h!.
D Additional Results
D.1 Loss and Accuracy versus Concept Space (Fig. 14)
In Fig. 14, we plot loss, accuracy, and OOD generalization trajectory in concept space of the models for a training run in Fig. 6. The time at which the model acquires capability manipulate the concept color is not evident from the loss or accuracy curves; however, in concept space, one can identify when the trajectory deviates from concept memorization and thus when the compositional ability has emerged. Benchmarking generative models is challenging task, still often involving humans in the loop [105, 106). Our concept space framework suggests that benchmarking out-of- distribution (OOD) generalization can potentially be reduced to monitoring the learning trajectory in concept space.
D.2 Additional trajectories of learning dynamics (Fig. 15)
Here we show all concept space trajectories for the experiments mentioned in Fig. 4 (a.b), for all classes and color concept signal levels. We find asymmetric behavior for the 01 class and the 10 class when adjusting the color concept signal level. The dynamics of the generations in the training set matches our intuitions of concept signal as discussed in the main text. At low color concept
21

---

Valisorton Loss
Test Accuracy
10a
ELBO ELBO
10a
0.6
Size
0.4
101
10a
10 10a
10a
Calor
Figure 14: Loss vs. accuracy vs. concept space. (left) The training and validation loss on ID classes during training. (center) The accuracy on the training classes and the test (OOD generalization) class. (right) Concept space trajectory of the generalization class. Loss and accuracy do not always intuitively reflect what capabiliities the model has acquired during training. However, one can see the rightmost panel, the point of sudden turn in concept space corresponds to when the capability has emerged, i.e., the moment when well defined prompting protocols can elicit the desired output from the model, which we indicate using the pink star (50% capability).
signal, we observe that the dynamics fit size first both 00 and 10, and afterwards find their correct colors.
ID generalization for dlass 00
ID generalization for class D1
ID generalization for class I0 out pudiee sto
COD generalization
praaee
Siee
Liartigg ny yylanics wo 20000
Coler
Color
Color
Figure 15: Concept space dynamics for all classes (00, 01, 10, 11). The experiment is identical in Fig. 4. The (0,1) normalized color concept signal is color coded in every trajectory. Two training data trajectories are shown in gray in the last panel to illustrate concept memorization.
D.3 Error Quantification (Fig. 16, Fig. 17)
We show the standard еттоr of the mean (s.c.m.) of concept space trajectories of Fig. 4 and Fig. 15 in Fig. 16 (a). Fig. 16 (b) illustrates the 50% capability point across different model initialization seeds. Fig. 16 (c) illustrates the 50% capability point across different color concept signal level. Overall, our results are consistent over random seeds and concept signal levels.
(a)
(b)
(c)
Color concapt signal
Siz
0.32
0.30
3.6
Capabilty
Size
0.4
0.26
25% 75% Capability
0.24
425% 375% Capability
50% Capability
Color
3.0
Color
Color
Color
0.20
Figure 16: Error quantification on concept space trajectories. (a) Concept space trajectories and their standard error of the mean (n=5) for different concept signal levels. (b) Concept space trajectories for 4 different seeds. The pink stars denote the point of 50% capability. (c) Concept space trajectories for different color concept signal level. The stars again denote the 50% capability point and the 25% 75% capability regions are in solid lines.
22

---

We show the standard deviation of compositional generalization accuracy for different prompting methods in Fig. 17. We find that the standard deviation is large for the naive prompting method, while linear latent interpolation and overprompting extract the hidden capability robustly across seeds, resulting in small standard deviation.
(a)
(b)
(c)
Accuracy
Accurecy
Accuracy
0.5
20000
10000
20000
20000
Number gradient steps
Number of gradient steps
Number gradient steps
Figure 17: Standard deviation of compositional generalization accuracy. We show the compo- sitional generalization accuracy and shade the l region for (a) Naive prompting (b) Linear latent intervention (c) Overprompting
D.4 Additional Experiments with real data, CelebA (Fig. 18)
To assess our findings on a more realistic dataset, we ran experiments on the CelebA [98] dataset. The experiments in Fig. 7 and Fig. 18, share the same experiment hyperparameters, described below.
We selected the Female and Smiling features the concept dimensions explore This choice was motivated by the relatively balanced number of samples in all four classes (00=(Male, Not Smiling/ 01=(Male, Smiling), 10=(Female,, Not Smiling). 11=(Female, Smiling)), from which we randomly sampled 30K images in each class. To construct the concept space, we trained a fully convolutional network with average pooling layer followed by a classification MLP head to classify the two attributes with independent cross entropy losses. (See B.2). We trained the classifier with the Adam W [122] optimizer with learning rate 10-3 3 and weight decay 10 10-5 for 10K gradient steps. The classifier achieved final accuracy of respectively 95% and 97% on the held out validation set, which was 10% of the entire dataset.
We trained the same diffusion model (See App. B.2) from our synthetic experiments on 64x64 resized images from the classes (00, 01, 10) and assessed the out-of-distribution (OOD) generalization to 11. We used a color jitter of 0.1 for brightness, contrast and saturation and randomly flipped the images horizontally. As the class attributes are categorical, they were one hot encoded and concatenated to an input conditioning vector of 4 dimensions. The diffusion model was trained for 2 x 10 gradient steps with a batch size of 64 with the same optimizer as in the main experiments.
The concept space dynamics of generations from the in distribution conditioning and OOD con- ditioning shown in Fig. 18. We find similar observations from the concept space trajectories as in our synthetic experiments. Initially, the images corresponding to the class 11 follows the concept space trajectory of 10, optimizing Female, which we intuitively expect have stronger concept signal, although it is intractable to compute since the DGP is unknown. Similar to our synthetic experiments, we see transition in the concept space trajectory of the compositional class corresponding to the model disentangling the concepts. After this transition, concept learning begins but observe a substantial bias towards the training distribution. We see a trade-off of moving in the right direction towards one concept degrades the other one similarly to the observations in Fig. (b) for high color concept signal levels. The visual generations in Fig. 18 confirm that our findings are not merely result of an ill-calibrated classifier model. However, in this case, we do not observe full -uttr--dist (OOD) generalization at 2M gradient steps. We expect this task of generating (Female, Smil ling) is inherently harder than our synthetic setup. We note that the goal of this experiment not to show good compositional generalization but it is more on confirming that qualitative findings generalize to real data without touching the model or training method.
23

---

Femall Not Smiling
Fernale Smilingg
Mals Not Smiling
Male Smill ng
Duiliuu
Gender
Figure 18: Concept space dynamics with CelebA. We train on the classes 00=(Male, NOT Smiling).
(Mle,, Smiling), 10=(Female, NOT Smiling) (plotted in blue) and test the class 11=(Female, Smiling) (plotted in pink). We find similar observations as in Fig. 4.2, concept memo- rization and the resulting bias.
D.5 Experiments with concept space (Fig. 19, Fig. 20)
To further verify findings in Sec. 4, explore a setup where the concept conditioning specifies three concepts: color, size and background color. Fig. 19 illustrates two example scenarios in which the concept signal in color and background color are varied, respectively resulting in cases of success and failure of OOD generalization by naive prompting. The length of edges of the cuboids represent their concept signal magnitude. In Fig. 19 (a), we see that the object color has strong concept signal and this is reflected in the concept space trajectories as this direction being split first. In this case we see, similarly to the blue generalization curve Fig. 4 (b) and Fig. 15 (rightmost panel), a slow generalization process for the compositional class 111. Again similarly to our findings in Sec. 4.2, we observe that the 011 class initially undergoes concept memorization for class 010, which shares the two stronger concept signals color and background color, and shows a transition where it suddenly curves out from this trajectory. Fig. 19 (b), we see case where out-of-distributio (OOD) generalization did not succeed within the given 80K gradient steps. In this case, we see two classes, 011 and 111, which didn't reach the right compositional class via naive prompting. However, the concept space trajectories suggest that the capability to compositionally generalize background color has already emerged as visible from the sharp turns.
Fig. 20 quantifies the accuracy of each concept separately. Fig. 20 (a) shows that stronger concept signal accelerates the corresponding concept learning. Fig. 20 (b) shows that other concepts' learning speed can vary, albeit smoothly, when changing single concept signal while the corresponding concept the most affected.
D.6 Experiments with classifier free guidance (Fig. 21)
An implication of our conclusions in Sec. 4.4 is that before the model has passed the transition point where the model has the capability compose concepts, the model should not be able to generate small blue circles no matter how well "prompted". Here, instead of prompting, we explore Classifier Free Guidance (CFG) [97] to see if our findings apply to conditional diffusion model trained with CFG. In Fig. 21, we see that even the models with CFG show this transition from concept memorization to -ddtt-idiitiittiion (OOD) generalization. In a scenario where there isn't a sharp acquisition of the capability to compositionally generalize, we would expect the sharp transition to disappear with CFG scale. However, as we observe Fig. 21, sharp turn concept space still exists with CFG, perhaps more pronounced than baseline experiments. These results suggest
24

---

(a)
100
011
80
Acoursy
100
011
40
101
000
20000
40000
63000
gradient
(b)
111
100
80
001
010
60
100
310
40
110
001
20
133
101
000
100
0
Number of gradient staps
20000 40000 60000
Figure 19: Generalization dynamics in 3 concept setting. (a) Generalization dynamics when color carries the strongest concept signal. (b) Generalization dynamics when size carries the strongest concept signal. We observe sharp turns corresponding to learning background color, however, naive prompting cannot fully elicit compositonal generalization.
(b)
Colo Acuract
6.8
Sastgroung Coler Accueac
0.8
Augrsga Accursiy
0.0
Maarage Accuraity
3.9
0.6
0.6
0.8
0.4
0.4
Sip D.6
D.A
Seawr 0.3
Calor
Siee
0.1
Bac-ground Color
Background Coder
Number gradertt
Numoer gradint stapps
Color Concept Siraa
Barkgmund Ooler Concegt Sigw
Figure 20: Accuracy individual concepts in 3 concept setting. We show the dynamics of color and background color accuracy during model training, depending on different concept signal levels b) We show the average accuracy until the end of training as metric of learning speed for different concepts, as we change single concept signal level.
that the acquisition of the compositional generalization ability is required for CFG to enhance OOD generalization.
generalization daass
generalization dass
generalization class
OOD generalization
Sue
Size
Color
Coltor
Coler
Color
Figure 21: Learning dynamies in Concept space for classifier-free Guidance We show the learning dynamics the concept space for all the classes (00, 01, 10, 11). The generalization task (rightmost) shows a sharp transition from concept memorization OOD generalization independently of the classifier-free guidance scale.
D.7 Patching the embedding module (Fig. 22)
We test additional elicitation method where we swap the conditioning vector (h)'s embedding module with that of the last checkpoint. An interpretation of this method that the embedding module disentangles the concepts, i.c., generates a representation for each concept, while the U-Net [95] then
25

---

utilizes such representations. representations early during training, while many more gradient steps are needed for the concepts to be disentangled when naively prompted.
This would imply that the U-Net [95] learns how to utilize concept
We test approach 5 random initialization seeds. Results shown in Fig. 22. We find that for some seeds, we are able to elicit the target behavior at around the same time in which overprompting and linear interventions also elicit the target behavior; for other seeds, this is not the case. These results demonstrates that the role of the embedding module and the U-Net might be separated during training for some runs without an explicit term enhancing this separation of roles.
(a)
1.00
(b) 1.00
Accurecy
0.75
Accurecy
0.75
0.50
0.50
0.25
0.25
0.00
5000
10000
20000
0.00
10000 20003
Number gradiont stops
gradient staps
Figure 22: Embedding patching. We patch the embedding module MIP) used for transforming the conditioning information into an embedding that the model processes from the last checkpoint to intermediate checkpoints. Panel (a) shows baseline accuracy for -istt--iistiibution (OOD) generalization across five different seed runs, while panel (b) shows accuracy achieved when the patched embedding module is used.
D.8 Proof of Concept Experiments with Frontier Models (Fig. 23)
We show simple proof of concept experiments on frontier models in Fig. 23. In Fig. 23 (a), we show that CLIP [125] already embeds text/images with compositional concepts in its vector space as a cube respecting the Hamming graph of concepts. In Fig. 23 (b), we show that simple over-prompting can enhance compositional generalization ability in Stable Diffusion v1.4 [126]. These experiments show that the assumptions we made to construct the concept space framework in Sec. 3 and the elicitation method we used work least some extent in frontier models.
(a)
red banana Chsqund
(b)
Naive prompting poommting
Case
Case
30 20 Yo 20 PCA 2 (object color) PCA 3 slackground color)
nrdttrrwerrryy w/
yellow banapd w/ white background
white Babkground
redl banana w/ background
10
red straw/erry w/ 10
black background
Over- Over- prompptott
yellow straw
yellow banaina w/ black hacgound
w/ white barkground
vollow houry backeround
400
PCA 1 (object)
Prompt: "Triangular credit card"
Figure 23: Proof of concept experiments on frontier models (a) CLIP [125] embeds compositional text/images concept eube in its vector space. (b) Simple over-prompting can enhance composi- tional generalization in Stable Diffusion vl.4 [126]
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4.2, Section 4.4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section B.1, Section B.2, Section C, Section B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: Section B.3.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wwtppcccccc/ccccccccccccccccccct httpa///ggieee/eeSuuiiiieieeee /Co////ooiissssseeeeee wwimissssiippcyyyy. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section B. Section B.2
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While report some error bars and show multiple seeds when applicable, did not test for statistical significance as most conclusions were qualitative.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics tttps::/neeuipss http:p:///rrrpppp.ppptttt////e/e csGuidel ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "WaNVOVgY83", "reviewer": "Reviewer_cQgr", "summary": "The paper introduces \"concept space\", a framework for analyzing the learning dynamics of generative models, focused especially on compositional generalization. The key contributions are:\n\n- Introducing the concept of \"concept signal\" that governs the rate of concept learning (and in particular determine learning speed), and is the key driver of the generalization dynamics\n- Analyzing learning dynamics using the concept space framework on a synthetic dataset of 2D objects.\n- Claims about how concept signals shape the geometry of learning trajectories.\n- a proposal on interventional protocols to uncover hidden model capabilities.\n\nThe paper uses a conditional diffusion model trained on synthetic data to study how concepts like shape, color, and size are learned and composed together. They show that concept signal levels determine learning speed and generalization dynamics, and that model capabilities often emerge before observable behaviors. The paper also explores how underspecification hinders compositional generalization.", "questions": "- Would we expect overprompting to still work when concept manifestations are complex? \n- Doesn't the embedder patching technique assume that the final checkpoint has disentangled concepts? Why should we expect to be the case? \n- The paper briefly mentions experiments with the CelebA dataset in the appendix. Could the authors elaborate on how well the findings from synthetic data translate to this more complex dataset?", "limitations": "- The paper should've compared the concept space framework to other existing methods for representations in learning dynamics (e.g., basic linear probing techniques, representational similarity analysis) as baselines for our understanding to see how much \"concept space\" improves it. \n- The models are fairly simple and small, and the study should further look into whether any of the techniques (e.g. overprompting, linear latent intervention etc.) still continue to hold in setting that more closely resemble real-world conditional diffusion models.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "Originality: \n\n- Concept space and concept signals are a novel lens for analyzing the learning dynamics for diffusion models.  \n- The claim that \"capability consistently emerges before behavior\" is interesting and well-backed, and is thus a novel insight into how these models behave over the course of the training. \n\nQuality: \n\n- Operating on the activation space and embedder patching seem like appropriate tests for the claim of testing . \n- It shows the intuitive result that model's generations for unseen combinations initially gravitate towards the training class with the strongest concept value. \n\nClarity: \n\n- The claims are clearly presented with an explanation supported with figures. \n\nSignificance: \n\n- The results suggest that the more distinguishable or salient a concept is in the training data (i.e., the stronger its concept signal), the faster the model will learn to recognize and generate that concept. If true for a broader class of generative models, the result could be widely applicable in data design choice for models.", "weaknesses": "Significance: \n\n- Overall the setup is too simplistic to have the claims be transferable to a broader set of generative models or even diffusion models. All the experiments use a simplified synthetic dataset with 2D objects and binary concept variables; while good for control, it limits the complexity of the kind of concepts real-world models learn by a massive amount. Real world models are learning more complex, hierarchical, and interdependent concepts, not simply characterizable by things like size or color, and use continuous concept variables rather than binary.  \n- The concept signal relies on knowing G, the data generating process. In real-world models, the true data generating process is often unknown, concepts might not have clear, differentiable manifestations in the input space. \n- The toy model for learning dynamics is based on a simple energy function with sigmoid activation; it's not clear why \"concept signals\" should be able to capture any non-linear interactions between concepts. \n\nPresentation of the claims: \n\n- The authors use the word \"capability\" when they really mean the model's internal ability to understand and compose concepts. The wording is quite unclear and should be clarified. \n- The graphs are pixelated, low-quality, poorly labeled and hard to understand. In particular, you should explain what the colors and the axis clearly mean in Figure 3, have multiple random seeds analyzed with error bars for overprompting and linear latent intervention in Figure 4, and make Figure 5 (b) higher quality. \n- Linear latent intervention assumes linear separability of concepts in the latent space, which breaks down in more complex settings than the synthetic data setting. \n- There should've been a higher focus on results on CelebA in the main sections of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer cQgr,\n\nWe thank the reviewer for their detailed feedback. We are excited that you found our work provides a novel lens for analyzing learning dynamics of diffusion models, yielding claims that are interesting, clear, and well-backed. In response, we have added:\\\n(i) A major experiment on CelebA generalizing our findings to more realistic data, Fig. R1\\\n(ii) A test on Stable Diffusion v2.1(SD), Fig R2b\\\n(iii) Experiments on a more complex 2x2x2 setup, Fig. R3\n\n---\n* **1. Simplicity of the setup:** Please note that our goal in this work was to understand how a diffusion model learns various concepts underlying the data-generating process and learns to compose them. The simple setup allowed us to perform a more precise analysis of the model’s learning dynamics and develop novel hypotheses on concept learning. However, our results on CelebA (R1) shows that this simplicity did not bottleneck our claims: our results do transfer to more realistic settings! Orthogonally, prior work on compositional generalization (CG) often uses similar settings (e.g., toy shapes, colors), making our chosen setup a natural starting point.\n\n* **2. Concept Signal relies on G:** Computing precise concept signal values is indeed non-trivial in real data. One approach would be to approximate the derivative of G by training a VAE to compute $dG/dz_i$. However, our primary contribution is a step towards understanding what controls CG when G is known. At the same time, our results show that we can retrospectively establish orders of concept signals (CS) for real data: e.g. we can infer CS for gender is stronger than hat from the CelebA experiment (R1).\n\n* **3. The model for learning dynamics:** We clarify that Fig. 2 shows accuracies over training time, where x and y represents accuracy for color and size. The key insight is that concept signals determine the learning dynamics' trajectory. Fig. 2(c) shows that curves bend toward specific directions reflecting the relative strengths of the concept signals for size and color. To model this behavior, we have defined conditions in Eq. 1: $\\sigma_1 > \\sigma_2$ and $\\sigma_2 < \\sigma_1$​. We will provide further clarification on these points in the revised version.\n\n* **4. The definition of capability:** Thank you for the suggestion! You correctly inferred our intended meaning: by capability, we mean “the model's internal ability to understand and compose concepts”. We have now formalized this in Sec. 3 of the paper. \n\n* **5. Figures:** We apologize for the pixelated plots. We updated all figures with higher quality versions. We also labeled all axes and added generated images to aid understanding, showing how the model improves during training (e.g., see R4). As per your suggestions, we have clearly clarified the axes in the caption and labeled the colorbar in Fig. 3. We updated Fig. 4 so that the curves for each seed are clearly visible. We updated Fig. 5b and labeled the color axis as $a$. We thank the reviewer again, as we really value high quality plots especially for a paper focused on sending a qualitative message.\n\n* **6. Linear Latent Intervention(LLI) Breaking Down**\\\nThe focus of our interventions is mainly to gain understanding of how capabilities are acquired in a model rather than to present a method to improve CG in practical settings. This is also why we presented 3 methods, so that we do not overfit to pitfalls of one specific method. Thus, our methods are *sufficient* to show a model has CG capabilities, but not *necessary* or efficient. Nevertheless, for LLI, we show in R1b that it can generalize to more realistic CelebA data.\n\n* **7. Discussion of Celeb A in main text**\\\nThank you for the suggestion! We moved some of the CelebA results to the main text.\n\n---\nQuestions:\n* **On overprompting(OP):** The short answer is “sometimes\" and just as LLI above, this intervention method was *sufficient* to show capability. While there will be cases where OP will not elicit CG, in R2b, we show that OP elicits CG even in SD, demonstrating its generalizability. In fact, SD’s prompt encoder takes brackets [] around words which need to be enhanced, and internally scales the vectors corresponding to these tokens, implementing OP.\n\n* **Assumptions on Embedder Patching:** Yes, and also an even stronger one: weights in MLP and CNNs should not drift after convergence of training. Restating the point made in 6 above, our intervention methods are sufficient but not necessary to show a model’s capability.\n\n* **CelebA Results:** Certainly! We reproduce both concept memorization(R1a) and hidden emergence(R1b) in CelebA. The former checks out the inverse concept signal learning point made above and the latter shows that this synthetic setup captures realistic concept learning well.\n\n---\nLimitations:\n* **Comparisons:** To our knowledge, the listed tools (e.g., probing) are not standard protocols studying learning dynamics in gen. models. The closest work to ours is Okawa et al. 2023, where the authors used probing to model learning dynamics, but could not identify what determines the order of concept learning, that there is concept memorization, and that there is a hidden emergence of capabilities. Nonetheless, we will add an expanded discussion of these tools in the paper. \n\n* **Generalizability:** We have added 2x2x2 (R3) and CelebA (R1) results to demonstrate that our result, at least, extends to systems which are a little bit more complex.\n\n---\n* **Summary:** We thank the reviewer again for detailed feedback. The reviewer’s comments motivated us to investigate further into the more realistic CelebA setup, and we are happy to show that our major finding on hidden emergence is well reproduced. We hope our changes and new experiments address the reviewer’s concerns adequately, and would be glad if our paper can be recommended more strongly.\n\n---\n[1] arxiv.org/abs/2311.03658", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "541bnP2ui4", "reviewer": "Reviewer_G81k", "summary": "This manuscript investigates the distinction between capabilities and behaviors in diffusion models. Using a toy task, the authors analyze at what point during training the model begins generating objects with correct specific features (in particular color and shape). They find that increasing the salience of certain attributes (“concept signal”) makes learning of those attributes faster and further causes the model's compositional generalization to collapse onto the nearest training point after intermediate training times. They replicate qualitative features of these learning curves with a mathematical toy model. They then use different interventions that enable the model to generalize compositionally much earlier than they would otherwise --- the fact that these interventions all yield generalization at the same time suggests a specific time point where the model has learned the underlying capabilities without this necessarily manifesting in behavior. Finally, they use this toy setup to analyze the impact of underspecification on compositional generalization.", "questions": "1. Could you clarify how sections 4.1-4.3 and section 4.4 are related (see weaknesses, point 1)?\n2. Could you clarify how you determined equations (1) and (2) and what we can learn from them (see weaknesses, point 2)?\n3. As noted, I think the findings in section 4.4 are really intriguing, so I want to make sure I understand exactly what is going on there. a) Since you're using a binary classifier to assess performance, are the overprompted colors/sizes really identical to the original colors/sizes or is it possible that e.g. the color produced from overprompting is different from the ground-truth color and just more clearly on the correct red/blue side of the hyperplane? It would be useful to see a few examples of generated images here. b) It seems that you're only using one model seed in Figs. 4(c) and (d). For 4(d) you explained that you only used the one with \"full capability\" (I assume that's the one with an accuracy close to 1.0?). Why did you only use one model for Fig. 4(c) (or are these multiple lines that are just strongly overlapping?)? c) I think the fact that there's only one model that reaches full compositional generalization accuracy qualifies these findings a bit, in particular as it means that the latter methods only present a single sample. Would the other models improve in their performance as well if they were trained for longer? d) I would suggest providing an additional figure where you plot the different curves for each model seed on top of each other, as it is currently a bit difficult to judge how precisely the transition times overlap.\n4. I'm not very familiar with the literature on underspecification --- have other papers previously noted its effect on compositional generalization (i.e. the \"strawberry\"/\"yellow strawberry\" effect)? If not, I think it'd be worth emphasizing that a bit more --- if yes, it would good to emphasize that as well.\n\n**A couple of minor notes**\n\nL. 29: “pre-training on such models” -> “pre-training of such models”\n\nL. 32: What is the “model experimental systems approach”?\n\nL. 69: Space before citations missing.\n\nL. 96: Is $S$ just the support of the probability distribution?\n\nL. 99: Is $F$ a stochastic function?\n\nL. 117-118: Isn’t $G^{-1}(Y)$ in the concept space, i.e. it should match $z$, not $h$?\n\nFigure 2: Why do the two trajectories for 01 and 10 illustrate concept memorization? I would have thought that this was illustrated by the trajectories for 11 that end up near 01.\n\nL. 247: What does it mean to mask the token? Set it to zero?", "limitations": "As the authors acknowledge, they focus on toy synthetic data here. Furthermore, their analysis is largely empirical in nature, leaving unclear the exact reasons why the models generalize or don't generalize. I think the authors have adequately communicated the limitations of their work overall.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The topic of compositional generalization and concept composition in generative models is important and generally still poorly understood. The authors did a good job motivating this and its connection to the distinction between capability and behavior/competence and performance.\n2. The findings in section 4.4 are surprising and notable and the authors replicated this behavior across different model seeds and methodologies (though I have a couple of questions, see weaknesses, point 3). \n3. The toy model for underspecification provided helpful intuition and appears to be a particularly simple task that gives rise to such underspecification.\n4. The manuscript is generally well-written and the figures are generally easy to follow.\n5. Concept memorization is a useful phenomenological finding.", "weaknesses": "As noted above, I think there are several interesting findings in this manuscript. However, in its current stage, I believe that it falls short of its stated goals. More specifically:\n\n1. As far as I can tell, sections 4.1-4.3 and section 4.4 get at different forms of \"capability.\" Specifically, the concept space studied in sections 4.1-4.3 distinguishes whether the model is able to generate certain properties of the image (e.g. color and size). In contrast, section 4.4 demonstrates that certain interventions can improve the models accuracy on the compositional task substantially. Since those interventions don't fundamentally change the model's capabilities, this suggests that the model has already learned to do the right thing and these interventions simply surface that capability. It's unclear to me whether that is necessarily apparent from the concept space behavior. Put differently, the model, in principle, could still be performing extremely badly according to the concept space but have already exhibited this transition. (You seem to be getting at a related point in section 6, \"Why concept space?\" and the supplementary figure, but it's unclear to me what the time of concept of acquisition is. To the extent that it relates the findings in section 4.4 to the findings in the previous sections, I think you'd have to show this across different models rather than using a single example of a model.)\n2. I did not understand the role played by equation (1). First, the defined energy landscape always has its minimum (for high t) at $c_1,c_2=1$, so why can these curves tend to different corners of the concept space? Second, I did not understand how this differential equation is grounded in model behavior, as it does not appear to be a simplified learning model. Rather, the goal seems to be to recapitulate the (rough) trajectories in concept space, so I'm not sure what insights are gained from that. In particular, the fact that the model first tends towards the correlated training point (e.g. the small red circle) during learning is built in by the definition of the sigmoidal function.\n3. I think the related work section on compositional generalization should provide a better overview of existing insights into the questions you're asking. Right now, you're only citing a number of papers investigating these questions, but I think it would be important to actually give an overview of what these papers are presenting and investigating and how it relates to your own findings. In particular, it's unclear to me what was previously known about the impact of underspecification on compositional generalization.\n4. I think it would be important to report standard errors or some sense of deviation across different model seeds. It is still important in my opinion to understand the reliability of these qualitative findings --- e.g. how similar are the concept space curves you're presenting across different initializations?\n\nAll in all, I think the paper presents several interesting findings, but, in its current state, leaves unclear how these findings fit together. On the one hand, the concept memorization finding is interesting and works well together with the underspecification finding. I think for both of those findings it would be important to more thoroughly evaluate how reliably the model actually ends up generalizing compositionally (e.g. across different seeds of initialization). In addition, it would also be helpful to give additional mathematical insights (or give an intuition in a different way) into why the observed behaviors emerge, as I don't see the current mathematical model as helpful on this end. On the other hand, the finding in section 4.4 is also interesting, but it remains unclear how it is related to the concept space framework and, if it can be explained in terms of learning both of these capabilities, why the presented intervention mechanism can help the model generalize compositionally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer G81k,\n\nWe thank the reviewer for their very detailed feedback. We are happy you found our findings in 4.4 “surprising and notable”, just the way we felt, and 5. and 4.2 “helpful” and “useful”. In response to your comments and concerns, we have added:\\\n(i) A clear definition and clarification of ``Capability” in Sec. 3.\\\n(ii) A demonstration that turns in concept space corresponds to emergence of capabilities, yielding Fig. R5 b,c\\\n(iii) A substantially more thorough related work section.\\\n(iv) A better quantification of uncertainties, yielding Fig. R4 and R5\n\n---\n* **1. Clarification of Capability in 4.1-4.3 vs 4.4:** The model’s capability is only discussed in 4.4. We do not mention capability in 4.1\\~4.3 as we are simply probing its behavior/execution of the task. Nevertheless, we now see that this can be confusing so we clearly defined capability and made this distinction in Sec. 3.\\\nYour comment also suggested that the relationship between concept space and capability should be strengthened. We have now added Fig. R5 b,c, which respectively demonstrates that 1) sudden turns in concept space does correspond to emergence of capability to compositionally generalize (CG) and 2) this correspondence is robust across different concept signal levels. We hope this clarifies the relation of section 4.1\\~4.3(concept space) and 4.4(hidden emergence).\n\n* **2. Motivation of the phenomenological model was to postulate concept learning dynamics hypothesis:** Eq. (1) is a phenomenological model designed to qualitatively reproduce learning dynamics, requiring different values of c1 and c2 to specify the model generalization point. The initial tendency towards the correlated training point is a result of the sigmoidal function definition, illustrating how different concepts evolve over time. Although the model is not derived from a specific learning process, it introduces the idea of defining \"concept variables\" and describing their evolution with differential equations. We are currently working on a theoretical follow-up to rigorously derive this equation and further substantiate our hypothesis.\n\n* **3. A more thorough related work section:** Thank you for the feedback! We have significantly expanded the related work section now, including discussion of prior work on CG, concept learning, use of interpretability tools to understand learning dynamics, and distinction between capabilities and behaviors. Related to CG, we briefly note that prior work has primarily focused on impossibility results, i.e., showing whether neural networks can express compositional solutions (e.g. [1]). To our knowledge, there is only one work focusing on learning dynamics of CG in a generative model [2], and does not include any intervention experiments or underspecified setups. Underspecification's influence on CG has been partially explored by a few papers on disentangled representation learning [3], but again these papers focus on possibility results and not learning dynamics---the target of our study. A position paper by Hutchinson et al. [4] mentions challenges in CG due to underspecification, but has a different goal compared to our work.\n\n* **4. Errors quantified and visualized:** Please see Fig. R4,5 for different quantification of errors. In R4, we show that the std. of behavior across seeds is high while it remains very low for probes of capability. In R5a, we visualized the standard error of mean on trajectories in Fig. 2c.\n\n* **5. Clarifications:** The intervention methods are more meant to investigate the model's internal capabilities and show that they emerge consistently before behavior. Although they do improve compositional generalization, that is more of a by product rather than a practical method we introduce.\n\n---\nQuestions:\n* **1, 2, 4:** We hope the replies above address these questions.\n* **3:** a) We have now added example images on top of Figure 4. The generated colors depends on the exact intervention we apply, as we would expect for a steerable model.\\\nb,c) We have now repeated the Linear Latent Intervention and Embedder Patching for all 5 seeds. We found LLI to work perfectly on all seeds and EP on 2 seeds. We don't think these affect our findings as one intervention method is *sufficient* to demonstrate model capabilities. Embedder Patching is a more subtle technique (higher bar) since it requires the model to have minimal Embedder-CNN compensating weight changes late in training.\\\nd) This is indeed how we have plotted Figs. 4 a, b! If the question was directed towards Figs. 4 c, d, we note we have now added all individual seeds to the plots, as mentioned above.\n\n---\nMinor Notes\n\n* **L.29,69,117-118:** Thank you for these catches.\n* **L.32:** By “model experimental systems approach”, we mean a small synthetic setup in which we can control the data distribution and probe the resulting model's behavior, allowing one to study questions that are not feasible to do so in real data.\n* **L.96,99:** Yes\n* **Figure 2:** The two gray trajectories are taken from the training data curves to help visualize concept memorization.\n* **L. 247:** What does it mean to mask the token? Set it to zero? Yes, the corresponding element is set to zero.\n\n---\n* **Summary:** Enormous thanks to the reviewer for pointing out very important points to make our submission more persuasive. We believe our work became more clear and robust thanks to the suggestions. Your comments motivated us to substantially strengthen our theory and related work section. It has also motivated us to quantify uncertainties on our experiments yielding Fig. R4, R5, which are essential to join concept space and hidden emergence, the two big pillars of our work. We believe our manuscript greatly improved in quality, and would be very happy if you could recommend our work more strongly.\n\n---\n[1] https://arxiv.org/abs/2310.05327\n\n[2] https://arxiv.org/abs/2310.09336\n\n[3] https://arxiv.org/abs/2006.07886\n\n[4] https://arxiv.org/abs/2210.05815", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sE0JrRt200", "reviewer": "Reviewer_uzj6", "summary": "This paper introduces a new framework, \"Concept Space,\" designed to study the learning dynamics of capability and behavior in generative models. It uses the concept signal to measure the rate of concept learning. They trained a diffusion model on a simplified synthetic dataset to validate their hypotheses that transitions in model capability occur much faster than the corresponding changes in behavior. This work provides empirical insights into understanding and controlling the development of AI models' capability and behavior.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-structured and clearly articulated.\n 2. This paper introduces the Concept Space framework and verifies several interesting combinatorial optimization results on synthetic datasets.\n3. The experiment setting is meaningful and interesting.", "weaknesses": "1. The rationale for modifying the differences in attribute values to represent different levels of concept signals is unclear. How does this relate to the formula $σ_i=|\\partial G(\\vec{z})/(\\partial z_i )|$\nThe derivation of equation (1) is also not clear.\n\n 2. Uncertainty in generalization: Despite the introduction of the Concept Space framework, there is still uncertainty about the model's ability to generalize to real-world data. While concept learning is easily validated on synthetic datasets, it is unclear how to effectively validate this approach on large-scale datasets and large models. Can you provide some insights?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer uzj6,\n\nWe thank you for your detailed feedback! We are glad you found our concept space framework and the robust emergence of capability insightful and interesting. We are especially happy to hear that our work provides “insights into understanding and controlling the development of AI models' capability and behavior”, as this was one of the main goals of our study. In response to your comments, we have added:\\\n(i) An improved description of our framework in Section 3.\\\n(ii) experiments on frontier multimodal embedding (CLIP) and generative (Stable Diffusion v2.1) models yielding Fig. R2.\n\n---\n* **Concept Signal can be altered by color differences:** The concept signal $\\sigma_i$ is the absolute change of the data generated when the corresponding concept variable is altered. Here, we are changing the raw RGB difference between the color concept blue and red. Thus ,a rescaling of this difference by $s$ is equivalent to the change of the data generating process $\\mathcal{G}\\to\\mathcal{G}'$, where $|\\frac{\\partial\\mathcal{G}'}{\\partial\\texttt{color}}|=s|\\frac{\\partial\\mathcal{G}}{\\partial\\texttt{color}}|$, and thus results in an enhancement of the color concept signal by a factor of $s$. For a visual explanation, please see Fig.7, where the distances are not schematic, but calculated from the dataset as we have access to the data generating process. We have updated Section 3, where we introduce our framework, so that this relation is more clear. We have also moved Fig.7 to the main text. Thank you for prompting this important clarification!\n\n* **Derivation of Equation 1:** Thank you for your question. Equation (1) is a phenomenological model designed to qualitatively reproduce concept learning dynamics, where our goal was to suggest a novel potential space for future theoretical study. The sigmoid function models sudden capability acquisition, while quadratic terms represent the energy landscape guiding the model towards correct concept values. Although not derived from a specific learning process, it introduces a new perspective of defining \"concept variables\" and describing their evolution via differential equations. We are currently working on a theoretical follow-up to rigorously derive this equation from a concrete model that further substantiates our hypothesis. We will clarify our motivation and limitations in an updated draft.\n\n* **Your feedbacks have inspired us to validate how many of our observations generalize to real-world data scenarios!:** Thank you so much for your insightful suggestions to further explore whether our observations validated in our synthetic setup generalize to real-world data. While, we've made the \"simplicity-interpretability\" trade-off to derive a set of hypothesis, you are right that it's very important to then go back to realistic scenarios to further validate those hypotheses.\n1) Latent linear intervention elicits hidden capabilities also in real world facial attribute dataset (CelebA). [Fig R1] In Fig R1a, we first show how diffusion model struggles to compose unseen combination of \"female\" and \"with hat\" concepts with naive prompting. Then, in Fig R1b, we demonstrate how applying latent linear intervention elicits this capability as was seen earlier in synthetic dataset!\n2) Square/cubic structure of concept graphs can be found in CLIP embeddings. We show that the concept space framework is useful to understand the failures and success of CG. *In some cases*, the concept space can be constructed for real diffusion models since it only requires a feature detector (e.g. CLIP), which is much simpler compared to training a generative model. In Fig R2a, we show that CLIP is ready to serve as a feature detector to construct an orthogonal concept space.\n3) One specific insight from overprompting and latent linear intervention is that one could use model interventions to evaluate model capabilities. In Fig. R2b, we show an explicit example where overprompting results in elicitation of a CG capability. We *speculate* that this could be scaled, as we show in Fig. R2a that CLIP can already serve as an orthogonal feature detector, and the lack of an expected feature can be used to provide feedback onto a method guiding generation by intervention.\n4) Our work surfaces that controlling concept learning might be more difficult than expected. Our results indicate that *even* In a synthetic setup where the acquisition of capabilities is robust, the model's apparent ability to CG might appear limited, and can show high variance across run seeds. Our contribution can be understood as attributing the high uncertainties observed in CG in real models to the variance of behavior seen in models with equivalent capabilities.\n---\n* **Summary:** Thank you again for your review. Your comments have motivated us to enhance the introduction of our framework in the manuscript and  to demonstrate scalability to large models and practical setups. As a result, we have managed to show that our intervention strategies could be valid in Stable Diffusion v2.1 and that CLIP embeddings already encompass orthogonal concepts graphs, suggesting a promising direction for scalability. We hope your concerns have been addressed with these clarifications and experiments, and we would be very glad if you could recommend our paper more strongly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qo6HO9VMI2", "reviewer": "Reviewer_o8es", "summary": "The paper introduces a novel framework for understanding concept learning in text-conditioned generative models. The experiments are carefully designed (although most of them look a bit toyish) and support the analyses well. The three key findings, i.e., concept signal levels determine the learning dynamics; capability consistently emerges before observable behaviors; under-specification harms the system, are quite novel and could provide many insights to the community. The proposed theoretical formulation also has the potential to be further developed. I enjoy reading this paper and suggest an acceptance.", "questions": "- The legends in Figure 2-bc (so as other learning dynamics figures) are a bit hard to read.\n- In Figures 2 and 3, the learning curves exhibit a “zigzag” pattern, i.e., first pointing to the training example sharing the strongest concept value, and then converging to the memorization one or comp-gen one. A similar phenomenon is also observed in a general classification problem discussed in [1]. Is there any similar reason behind these two findings?\n- Both the concept signal and sigmoid function are denoted by $\\sigma$, which is confusing.\n- What are (0.4, 0.4, 0.6) to (0.3, 0.3, 0.7) in line 216 means, normalized RGB values?\n- The color bar of Figure 6 might have the wrong legend (it is the percentage of masked $\\alpha$, but the bar ranges from 1 to 3.5).\n- The paper concludes that  (in line 300) concept learning is a well-controlled phase transition, but the observed behavior can be arbitrarily delayed. Are there any methods that can reduce this delay?\n- [2] studies a very similar compositional generalization problem (but in a representation learning setting). It would be helpful to also discuss that in the related work. The discussion of simplicity bias and Kolmogorov complexity in [2] might also be a potential explanation for the observations presented in this paper.\n\n[1] Ren, Yi, Shangmin Guo, and Danica J. Sutherland. \"Better supervisory signals by observing learning paths.\" ICLR 2022\n\n[2] Ren, Yi, et al. \"Improving compositional generalization using iterated learning and simplicial embeddings.\" NeurIPS 2023", "limitations": "The experimental setting is kind of simple. But I think it is sufficient for a phenomena explanation (and theory) paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Good presentation, easy to follow.\n- The discussions about the learning dynamics of concept memorization and compositional generalization are cool.\n- The three probing methods that verify the claim that the model might learn the concept before generating the correct image are insightful.\n- The under-specification discussion might bring practical suggestions to real systems.", "weaknesses": "- In Section 3, it is a little hard to understand the relationships between different variables and functions. A diagram like Figure 1 in [2] might be helpful.\n- Although the experiments and analysis of the paper are very insightful and persuasive, it is only verified in a relatively simple setting. There are experiments on CelebA, but the number of attributes and their possible values are still kind of small. One or two experiments considering more complex scenarios will strengthen the paper a lot.\n- The paper claims that the model usually has the capability to compose the learned concepts earlier than this behavior shows up. How will this finding influence our understanding of (or provide insights to improve) the practical systems? Does this correlate to early stopping (the switching point from underfitting to overfitting)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer o8es,\n\nThank you so much for thoroughly understanding, and positively evaluating our work. We are glad to hear that you enjoyed our work, and found all of our three main findings to be “quite novel and could provide many insights to the community”. We thank you also for your very insightful and actionable suggestions to robustify our claims. In response to your thoughtful suggestions, we addressed your concerns by adding:\\\n(i) a new schematic diagram inspired by [2] Ren, Yi et al.\\\n(ii) further experiments in a more complex scenario with three concept variables yielding Fig. R1 and R3\\\n(iii) a better grounding into the learning dynamics of compositional generalization(CG) literature [1,2]\\\n(iv) a general improvement of figure qualities.\n\n---\n* **1. Added New Diagram for the framework:** Inspired by [2] Ren, Yi et al., we have updated our framework section of the paper to be easier to understand. We have added a schematic Figure describing the relation of $\\mathcal{S}$, $\\mathcal{G}$, $\\mathcal{M}$, $\\mathcal{X}$ in appendix as well. We haven’t been able to fit this new figure in the attached pdf due to space constraints, but this new figure will be included in the final version of this submission.\n\n* **2. Concept space findings generalize to more complex scenarios:** We have now extended our results to more complex scenarios. In Fig. R1, we show that our *findings on hidden emergence and also our probing methodology generalizes to a compositional setup on CelebA*. In this experiment, the model did not show the CG behavior, while latent linear intervention clearly demonstrated the emergence of the capability. We have also generalized our findings on concept signals in Section 4.1 to a 2x2x2 setup with 3 concept variables. We again find that *concept signal controls the speed of learning in scenarios involving more than 2 concept variables*.\n\n* **3. Practical Implications of Early Capabilities:** One implication of the hidden emergence of capabilities is that standard evaluation pipelines on a fixed dataset might be probing behavior, which we show is very seed dependent. Additionally, this motivates studies on enhancing current models by understanding their internals as it suggests that *models can have capabilities which are not elicited unless intervened appropriately*. With your specific point regarding early stopping, we believe that these models do not explicitly over-fit and nor do frontier diffusion models (see [3]), as they are in the under-parametrized regime and usually trained in an online setup. We also do not have evidence of any capabilities getting lost as we train far beyond fitting the training set.\n\n---\nQuestions:\n\n* **Figure 2-bc legend:** Thank you for this feedback! We have made significant revisions to explain the axes better in the captions and to improve the overall quality of the figure.\n\n* **Zigzag Pattern:** Thank you for this reference [1]! Indeed, our current understanding is similar to [1], where higher difficulty corresponds to lower concept signal. In our case a concept is harder when its concept signal is less important to reduce the diffusion loss. We will make sure to cite [1] and discuss the similarity of the mechanisms underlying the shared trend in learning dynamics.\n\n* **Usage of $\\sigma$:** Thank you for this suggestion! We updated the sigmoid function to $g$ to avoid confusion.\n\n* **Clarification of line 216:** Yes! We will clarify this in our updated manuscript.\n\n* **Figure 6 Legend** Thank you for this catch, we will edit this.\n\n* **Controlling the phase transition:** We did find out that many hyperparameters control the transition, such as weight decay, weight initialization, optimizer, etc.. However, we did not yet find a clear identifiable relationship which allows any predictive hyperparameter transfer. Testing the generalization of these observations to more complex text-to-image models is a great future direction. Currently, we are interested in following up with a theoretical model based on the concept signal values.\n\n* **Relation to reference [2]:** Thank you for pointing out this work. We will definitely cite and discuss it in the final version! The simplicity bias explanation is very interesting, but it seems non-trivial to define it in our setup at the current stage. We briefly looked into weight decay to add an explicit “simplicity” enforcing term, however this did not qualitatively change our results. \n\n---\n**Summary**: Thank you again for carefully going through our manuscript and giving us such a concrete and insightful list for improvements! As clarified above, your suggestions inspired us to not only create new schematic figures, but also to run 8 more experiments resulting in 2 new experimental plots. With the above, we hope that we have fully addressed your concerns and now you could strongly recommend acceptance of this paper.\n\n---\n[3] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representation.\" arXiv preprint arXiv:2310.02557 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to thank the reviewers for their thoughtful feedback and for recognizing the value of our work. We are pleased that all reviewers found our main contributions, particularly the introduction of the “concept space” framework and the studies of hidden capabilities to be "a novel lens for analyzing the learning dynamics for diffusion models" [R cQgr], "surprising and notable" [R G81k], “meaningful and interesting” [R uzj6] and "quite novel and could provide many insights to the community" [R o8es]. We are trying to understand diffusion models using controlled synthetic experiments and many reviewers pointed out that it is crucial to show how much of the findings generalize to real data/models. This feedback was very helpful and we believe our submission became much more persuasive in this direction with the additional experiments we conducted.

We ran:
* 1 major experiment on CelebA in Fig R1
* 2 experiments with frontier multimodal models in Fig R2
* 27 new experiments with the synthetic setup distributed in Fig. R3, 5
* 8 new probing experiments for Fig. R4
Below are the description of the new experiments and the major changes made to our work. Please find the figures in the attached PDF.

Abbreviations: Compositional Generalization (CG)

---
**New experiments**:
* **[Figure R1] Hidden emergence of capabilities reproduced on CelebA:** Many reviewers asked about the generalizability of our results to a more *realistic* scenario. We trained a conditional diffusion model from scratch on CelebA using the compositional concepts With Hat and Female. Fig R1a shows the concept space dynamics evaluated on CelebA. In this case, CG was harder than the experiment in Fig 9 in the paper, and the model was not able to generate (Female, With Hat), and it was *concept memorizing* on (Female, No Hat). However, Fig R1b shows that latent linear intervention was able to elicit this capability and the model can in fact compose the two concepts it has internalized. Fig R1b clearly shows hidden emergence of capability on a realistic dataset with arbitrarily delayed behavior.

* **[Figure R2] CLIP and Stable Diffusion v2.1 suggests potential scalability of our work:** Reviewers also asked if the *assumptions made and the methods used* are only expected to work in simplistic settings. While we accept that there will be more subtleties in generalizing these to real data, we show promising results. In Fig. R2a we show that CLIP already embeds compositional concepts as a cube with roughly orthogonal axes, addressing the concern that concepts might be non-linearly embedded in realistic models unlike our assumption in the synthetic setup. This also suggests that CLIP can be used as a feature critique to construct a real life concept space. In Fig. R2b, we show that overprompting can be used to elicit CG in Stable Diffusion v2.1, demonstrating that our method does generalize to some extent to frontier models.

* **[Figure R3] Concept signal controls CG on 2x2x2 concept graph:** Reviewers also asked if the results would generalize to more *complex* scenarios than two binary concepts. On this end, we ran experiments with 3 concept variables and reproduced the findings in Fig. 2a, showing that concept signal controls CG timings by affecting concept learning speeds. Fig. R3a shows the concept accuracy versus training time and Fig. R3b shows that the speed of concept learning (defined by the average accuracy up to the final checkpoint) clearly depends on concept signal.

* **[Figure R4] Robust Capability Learning across random seeds:** We confirmed that overprompting and latent linear interventions shows consistent acquisition of CG capabilities across model seeds while the behavior(execution) has high variance. We show that the standard deviation of capability learning curves is negligible compared to behavior. We confirm this behavior on multiple seeds as it is the main result of our paper.

* **[Figure R5] Quantifying uncertainties across random seeds and data distributions:** Our findings on hidden emergence are robust across random seeds and data distributions. We ran multiple seeds of the experiments with different data distributions and confirmed that our results are robust. Fig R4a visualizes the standard error of the mean of the concept space learning dynamics of the models and demonstrates the level of uncertainty in the concept space trajectories. Fig R4b shows that the sudden turn in concept space corresponds to the emergence of a capability across different seeds. Fig R4c shows that this finding is also consistent across different data distributions, varying the level of concept signal.

---
**Major Edits:**
* **Figure Revisions:** We edited most of our figures' qualities and labeling. We made sure all figures had clearly labeled axes and non-pixelated graphs. In particular, for Fig 4, the most important figure of our work, we added many seeds (c,d) and example images (a,b,c,d).
* **Related Works:** We substantially increased the related works section to better ground our paper in the intersection of concept learning, interpretability and cognitive science(competence vs behavior).
* **Framework:** We reformatted the framework section(Sec. 3) so that the term capability is well defined. We explicitly distinguish capability and behavior(execution).
* **CelebA:** We added a section in the main text describing CelebA as many reviewers pointed out the importance of showing generalization of the findings to realistic data.
* **Appendix:** We added all our additional experiments to appendix.

Meta-review: This work introduces the idea of "concept space" in the context of conditional generative models and considers the learning process as one that moves from "concept memorization" to generalization. In experiments, the authors demonstrate that, in many cases "capability precedes behavior": models are able to compose concepts under forceful intervention even though they do not successfully generate under normal conditions.

Reviewers found the idea of concept space promising as a lens through which to analyze the dynamics of learning. In particular, the evidence presented that some capability is present in models before it is expressed was deemed highly interesting. Reviewers had some concerns about the limited range of experiments in the model, but the addition of new experiments allayed many of these concerns. I recommend acceptance.
Decision: Accept (spotlight)