Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space

Authors: francisco  core
  park
maya  okawa
andrew  lee
ekdeep  singh lubana
hidenori  tanaka
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:36:55

Abstract: Modern generative models demonstrate impressive capabilities, likely stemming from an ability to identify and manipulate abstract concepts underlying their training data. However, fundamental questions remain: what determines the concepts a model learns, the order in which it learns them, and its ability to manipulate those concepts? To address these questions, we propose analyzing a model’s learning dynamics via a framework we call the concept space, where each axis represents an independent concept underlying the data generating process. By characterizing learning dynamics in this space, we identify how the speed at which a concept is learned, and hence the order of concept learning, is controlled by properties of the data we term concept signal. Further, we observe moments of sudden turns in the direction of a model’s learning dynamics in concept space. Surprisingly, these points precisely correspond to the emergence of hidden capabilities, i.e., where latent interventions show the model possesses the capability to manipulate a concept, but these capabilities cannot yet be elicited via naive input prompting. While our results focus on synthetically defined toy datasets, we hypothesize a general claim on emergence of hidden capabilities may hold: generative models possess latent capabilities that emerge suddenly and consistently during training, though a model might not exhibit these capabilities under naive input prompting.

Main paper: 
 introduction: 
 Modern generative models, such as text-conditioned diffusion models, show unprecedented capabilities [1] [2] [3] [4] [5] [6] [7] [8] . These abilities have led to use of such models in applications as valuable as training control policies for robotics [9] [10] [11] and models for weather forecasting [12] , to as drastic as campaigning in democratic elections [13, 14] . Similar claims can be made for generative models of other modalities, e.g., large language models (LLMs) [15] [16] [17] [18] , speech and audio models [14, 19] , or even systems designed for enabling scientific applications such as drug discovery [20, 21] . Arguably, acquiring such general capabilities requires for models to internalize the data-generating process and disentangle the concepts (aka latent variables or factors of variation) underlying it [22, 23] . Flexibly manipulating these concepts can then enable generation of novel samples that are entirely out-of-distribution (OOD) with respect to the ones used for training [24] [25] [26] [27] [28] .As shown in prior work, modern generative models do exhibit signs of disentangling concepts underlying the data generating process and learning of corresponding capabilities to manipulate said Figure 1 : Concept Learning Geometry underlies emergence. (a) Top: A multimodal model learns to generate the concepts in the order of "astronaut", "horse", and finally "riding" as it scales up (adapted from Yu et al. [45] ). Middle: "blue square apple" is generated in the order of "apple", "blue", and "square"" (adapted from Li et al. [46] ). Bottom: Despite its simplicity, our model trained on synthetic data shows concept learning dynamics where it first learns "shape" and then "color". (b) Concept space is an abstract coordinate space where individual axes correspond to different concepts and a given point corresponds to a "concept class", i.e., a predefined collection of concepts (e.g., large blue circles on bottom left corner). Traversal along axes of the concept space yield change in a specific property of the sample (e.g., going from large blue circle to large red circle along object color axis). Trajectories show a model's dynamics in concept space for learning to generate classes shown in-distribution (blue nodes) versus out of distribution (pink / red nodes). As we show, dynamics in concept space are highly interpretable, enabling precise comments on which concepts the model learns first, why, and what order it follows. (c) Measuring how accurately a model generates samples from a given concept class, showing an order of concept learning: first background color is learned, then size, and then object color. concepts [29] [30] [31] [32] [33] [34] [35] [36] [37] . At the same time, contemporary work has argued that models' capabilities can be unreliable, arbitrarily failing for a given input and succeeding on another [38] [39] [40] [41] [42] [43] [44] . Thus, critical questions remain on how generative models acquire their capabilities (see Fig. 1 ): what determines whether the model will disentangle a concept and learn the capability to manipulate it; are all concepts and corresponding capabilities learned at the same time; and is there a structure to the order of concept learning such that, given insufficient time, the model learns capabilities to manipulate only a subset of concepts but not others? This work. To address the questions above, we propose to analyze a model's learning dynamics at the granularity of concepts. Since performing such an analysis on realistic, off-the-shelf datasets can be challenging, we develop synthetic toy datasets of 2D objects with different concepts (e.g., shape, size, color) that give us thorough control over the data-generating process and allow for an exhaustive characterization of the model's learning dynamics (see Fig. 1 ). Our contributions are as follows.• Introducing Concept Space. We propose to evaluate a model's learning dynamics in the concept space-an abstract coordinate system whose axes correspond to specific concepts underlying the data-generating process. We instantiate a notion of underspecification in concept space and establish its effects on a model's (in)ability to disentangle concepts. • Concept Signal Dictates Speed of Learning. We find the speed at which a model disentangles a concept and learns the capability to manipulate it is dictated by the sensitivity of the datagenerating process to changes in values of said concept-a quantity we call concept signal. We show concept signals shape the geometry of a learning trajectory and hence control the overall order of concept learning. • Sudden Transitions in Concept Learning. We use analytical curves to explain the phenomenology of learning dynamics in concept space, showing the dynamics can be divided into two broad phases: (P1) learning of a hidden capability, whereby even if the model does not produce the desired output for a given input, there exist systematic latent interventions that lead the model to generate the desired output, and (P2) learning to generate the desired output from the input space.Overall, while our results focus on a toy synthetic task and text-to-image diffusion models, we hypothesize a broader claim on hidden emergence of latent capabilities holds true: generative models possess latent capabilities that are learned suddenly and consistently during training, but these capabilities are not immediately apparent since prompting the model via the input space may not elicit them, hence hiding how "competent" the model actually is [47, 48] . Empirically, signs of "hidden capabilities" have already been shown in generative models at scale [45, [49] [50] [51] [52] [53] , and our results help provide a formal framework to partially ground such results. 
related work: 
 Concept learning. The term concepts as used in this work is broadly equivalent to the notion of factors of variation from prior work on disentangled representation learning [23, 30, [54] [55] [56] [57] [58] [59] [60] [61] , and is motivated by use of the term in a similar sense in cognitive science [62] [63] [64] [65] [66] [67] . The focus of literature on disentanglement has been to prove identifiability results, e.g., when will a generative model trained on a dataset learn to invert the data-generating process, hence identifying the latent variables, i.e., concepts, that underlie it. Given the success of modern generative models, we argue, despite impossibility results from prior work, these models are in fact learning to disentangle some, if not all, concepts. Precisely what guides which concepts are disentangled however requires studying the learning dynamics-the target of our work.Interpretability. A growing line of papers demonstrate highly interpretable representations of intuitive concepts exist in generative models, especially LLMs [68] [69] [70] [71] [72] [73] [74] . Similar work in this vein in image diffusion models has found semantic representations in various components of the model [37, 52, [75] [76] [77] : e.g., existence of linear representations for concepts such as 3D depth or object versus background distinctions [78, 79] . These papers further bolster our argument, modern generative models are truly inverting the data-generating process and identifying the concepts underlying it.Competence vs. performance. In cognitive science, a system's competence on a task is often contrasted with its performance [47, 48, [80] [81] [82] [83] : competence is the system's possession of a capability (e.g., to converse in a language) and performance is system's use of that capability in concrete situations [80] . For example, a bilingual person may generally converse in their primary language L, despite possessing knowledge of another language L ′ , unless it is crucial to use the latter-clearly, they are competent in both languages, but gauging their performance on L ′ requires appropriately "prompting" them to use it. One can analogize this distinction with remarks on a neural network possessing a capability versus us being able to elicit it on predefined benchmarks and measure their performance [84] [85] [86] [87] . For example, on the BigBench benchmark [88] , LLMs were shown to have perform poorly on several tasks, but follow up work [89] showed mere chain-of-thought prompting [49, 50] leads to huge boosts on all tasks. This indicates the evaluated models were in fact "competent", but inappropriate prompting led to undermining how "performant" they are.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 6Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
concept space: a framework for analyzing concept learning dynamics: 
 We first formally introduce our framework of concept spaces. See Fig. 12 for a schematic. Motivated by a rich body of literature on disentangled representation learning [23, 30, [54] [55] [56] [57] [58] [59] [60] , this framework allows us to systematically analyze how different concepts underlying the data generating process and corresponding capabilities to manipulate them are learned during training. We highlight that since our primary empirical focus in this paper will be an abstraction of text-to-image generative models, parts of the framework will be specifically instantiated with text-to-image generation tasks in mind. To this end, we note our model class of interest is a generative model F that is trained using conditioning information h to produce images y. For example, F may be instantiated using a diffusion model that uses embeddings h of textual descriptions of a scene to produce images y. We next define a concept space.Definition 1. (Concept Space.) Consider an invertible data-generating process G : Z → X that samples vectors z ∼ P (Z) from a vector space Z ⊂ R d and maps them to the observation space X ∈ R n . We assume the sampling prior is factorizable, i.e., P (z ∈ Z) = Π d i=1 P (z i ), and individual dimensions of Z correspond to semantically meaningful concepts. Then, a concept space S is defined as the multidimensional space composed of all possible concept vectors z, i.e., S := {z | z ∼ P (Z)}As an example, consider a concept space defined using three concepts, say, z 1 = shape, z 2 = size, and z 3 = color that maps to a dataset of images with objects of different combinations of shapes, sizes, and colors (see Fig. 2 ). The assumption that concepts are independently distributed implies one can intervene on a given concept without affecting the other ones. For example, given a sample from the dataset above, the concept color in an image can be altered by changing the value of the relevant latent variable and mapping it to the image space via the data-generating process. Now, using a mixing function M that yields conditioning information h := M(z), we can train a conditional generative model and define a notion of capabilities relevant to our work as follows.Definition 2. (Capability.) A concept class C denotes the set of concept vectors z C such that a subset of dimensions of these vectors are fixed to predefined values. Classes C and C ′ are said to differ in thek th concept if ∀z ∈ z C , there exists z ′ ∈ z C ′ with z[k] ̸ = z ′ [k] and z[i] = z ′ [i] for i ̸ = k.We say a model possesses the "capability to alter the k th concept" if for any class C whose samples were seen during training, the model can produce samples from class C ′ that differs from C in the k th concept.Intuitively, the definition above comments on whether the model can flexibly manipulate concepts of classes seen during training to produce samples from classes that were not seen, i.e., classes that are entirely out-of-distribution. As an example, consider a concept space with shape, color, and size as concepts. If shape and color are fixed to circle and blue respectively, we get the class of blue circles; i.e., ∀z ∈ z blue circles , the first and second dimension respectively take on values that correspond to the shape circle and color blue. Then, given a conditional diffusion model that was shown blue circles during training, we will say this model possesses the capability to alter the concept color if it can produce samples from concept classes with the same shape as circles, but different colors (e.g., red or green circles). Analyzing learning dynamics in the concept space will thus provide a direct lens into the model's capabilities as they are acquired.We also note that the definition above is not dependent on the precise manner via which the model is prompted to elicit an output, i.e., it need not be the case that the conditioning information h that is used for training the model is used to evaluate the model capability. In fact, in our experiments, we will try alternative protocols such as intervening on the latent representations to show that substantially before the model can be prompted using h to generate samples from an OOD concept class, it can generate samples from said class via such latent interventions. To this end, we also define a measure that assesses how much learning signal the data provides towards disentanglement of a concept, and hence learning of a capability to manipulate it. Definition 3. (Concept Signal.) The concept signal σ i for a concept z i measures the sensitivity of the data-generating process to change in the value of a concept variable, i.e., σ i := | ∂G(z) /∂zi|.Intuitively, if the training objective is factorized at the granularity of concepts, concept signal indicates how much the model would benefit from learning about a concept. For example, consider a diffusion model trained using the MSE loss with conditioning h := z to predict the noise added to an image G(z). σ i is then merely the component of the loss representing how much change in conditioning h yields changes in concept z i , as it is represented in the image. Concept signal will thus serve as a crucial knob in our experiments to analyze learning dynamics in concept space and the order in which concepts are learned. The color separation between the classes is stronger than the size separation, resulting in a stronger concept signal in the color dimension. (right) The size separation between the classes is stronger, thus resulting in a stronger signal for size.Before proceeding further, we discuss our experimental setup that concretely instantiates the formalization above. Our data-generating process is motivated by prior work on disentanglement [54] [55] [56] [57] [58] [59] [60] and involves concept classes defined by three concepts, each with two values: color = {red, blue}, size = {large, small}, and shape={circle, triangle}. In Sec. 4.1 and Sec. 4.4, we use two attributes: color (with red labeled as 0 and blue as 1) and size (large labeled as 0 and small as 1). We generate a total of 2048 images for each class, with objects randomly positioned within each image. We train models on classes 00 (large red circles), 01 (large blue circles), and 10 (small red circles), shown as blue nodes in Fig. We first demonstrate the utility of concept signal as a tool to gauge at what rate the model learns a concept and the capability to manipulate it. To this end, we develop controlled variants of our data-generating process by changing the level of concept signal of each concept and train diffusion models conditioned with the latent concept vector z on them. We primarily focus on concepts color and size, altering their concept signal by respectively adjusting the RGB contrast between red and blue and the size difference between large and small objects (see App. B.1 for details). We define the speed of learning each concept as inverse of the number of gradient steps required to reach 80% accuracy for class 11, i.e., the OOD class that requires learning the capability to manipulate concepts as seen during training. Results comparing different concept signals are shown in Fig. 3 . For both color and size, we observe that concept signal dictates the speed at which individual concepts are learned. We also find that when different concepts have varying strengths of concept signals, this leads to differences in the learning speed for each concept.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
concept signal governs generalization dynamics: 
 We next examine the model's learning dynamics in concept space under various levels of concept signal for the concepts color and size. For completeness, we evaluate a model's ability to generalizeColor concept signal both in-distribution (ID) and OOD, but only the latter is deemed learning of a capability to manipulate a concept. Results are shown in Fig. 4 , with panel (a) showing dynamics for learning to generate samples from ID class 00 and panel (b) showing dynamics for OOD class 11. Results on OOD generalization show concept memorization, which we define as the phenomenon where the model's generations on an OOD conditioning h are biased towards the training class that helps define the strongest concept signal. For example, for the unseen conditioning 11, the generations are more alike 01 when the concept signal is stronger for size (e.g., blue curve in Fig. 4(b) ) and alike 10 when the signal is stronger for color (e.g., red curve). Interestingly, we observe that for settings with high imbalance in concept signals, e.g., the blue curve in Fig. 4 (b), the endpoint of concept memorization is very biased towards one training class, here 01, delaying its out-of-distribution (OOD) generalization. For the learning trajectories of all classes, see Fig. 15 in App. D.2.Broadly, our results imply that an early stopped text-to-image model can witness concept memorization and hence simply associate an unseen conditioning to the nearest concept class when asked to generate OOD samples (see Kang et al. [96] for a similar result in LLMs). However, given sufficient time, the model will disentangle concepts underlying the data-generating process and learn to generate entirely novel, OOD samples. For futher evidence in this vein, we also confirm our results across more general scenarios, including with the real-world CelebA dataset (App. D.4) and using three concept variables: color, background color, and size (App. D.5). = z + (ẑ -z) • 1 1+e -(t-t), where ẑ is the target point we want to get to and z is the initial, "biased" target. For example, consider the case with color and size concepts in Fig. 4 (b) . The model's generated samples are more alike class 01 and biased towards (z 1 , z2 ) = (0, 1) when the size concept signal σ 2 is stronger than σ 1 ; and to 10, (z 1 , z2 ) = (1, 0) when the color concept signal σ 1 is stronger than σ 2 . We define (ẑ 1 , ẑ2 ) as the target values or directions we want the learning to head towards (e.g., (1, 1) for OOD generalization). Based on this framework, we can derive the following energy function.EQUATIONHere, a is determined by the difference |σ 1 -σ 2 | and t1 and t2 denote the times when the model learns concepts z 1 and z 2 , respectively. Fig. 5 illustrates the simulated trajectories for classes 00 and 11, based on Eq. 1. Panels (a) and (b) correspond to classes 00 and 11, respectively. We define the actual target points (ẑ 1 , ẑ2 ) as (0, 0) for class 00 and (1, 1) for class 11. For the initial targets (z 1 , z2 ), we set both values to (0, 0) for class 00. For class 11, the targets are set to (1, 0) when σ 1 > σ 2 and to (0, 1) when σ 1 < σ 2 . We find our toy model effectively captures the actual learning dynamics for both in-distribution (Fig. 3(b) ) and out-of-distribution (OOD) concept classes (Fig. 3(c) ). Notably, our simulation accurately replicates the two types of curves: clockwise (blue trajectory in Fig. 3(b) ) and counterclockwise (red trajectory).An important conclusion that follows from the results above is that the network's learning dynamics can be precisely decomposed into two stages, hence yielding the sudden turns seen in trajectories in The concept space visualization of learning dynamics observed in Fig. 4 (b) and our toy models analysis in Fig. 5 indicate that there exists a phase in which the model departs from concept memorization and disentangles each of the concepts, but still produces incorrect images. We claim that at the point of departure, the model has in fact already disentangled concepts underlying the data-generating process and acquired the relevant capabilities to manipulate them, hence yielding a change of direction in its learning trajectory. However, naive input prompting is insufficient to elicit these capabilities and generate samples from OOD classes, giving the impression the model is not yet "competent". This then leads to the second phase in the learning dynamics, wherein an alignment between the input space and underlying concept representations is learned. We take the model corresponding to the second from left curve (the green curve) in Fig. 4 (b) to investigate this claim in detail. Specifically, given intermittent checkpoints along the model's learning trajectory, we use the following two protocols for prompting the model to produce images corresponding to the class 11 (blue, small). See App. C for further details.1. Activation Space: Linear Latent Intervention. Given conditioning vectors h, during inference we add or subtract components that correspond to specific concepts (e.g., h blue ). 2. Input Space: Overprompting. We simply enhance the color conditioning to values of higher magnitude, e.g. (r, g, b) = (0.4, 0.4, 0.6) to (0.3, 0.3, 0.7).Fig. 6 shows the accuracy for five independent runs under: (a) naive input prompting, (b) linear latent interventions, and (c) overprompting. In Fig. 6 (a), we observe that some runs can produce samples from the target concept class (blue, small) with ∼ 100% accuracy after around 8,000 gradient steps, while other runs fail to do so. However, in Fig. 6 (b, c ), we find alternative protocols for prompting the model can consistently elicit the desired outputs much earlier than input prompting, e.g., at around as early as 6,000 gradient steps. This indicates the model does possess the capability to alter concepts and generalize OOD! Furthermore, we note that different protocols enable elicitation of the capability at approximately the same number of gradient steps, irrespective of the seed, and that this is precisely the point of sudden turn in the learning dynamics in Fig. 4 ! Interestingly, experiments with Classifier Free Guidance (CFG) [97] show that CFG only becomes effective after this transition (Fig. 21 ).We further explore the second phase of learning in Appendix D.7 by patching the embedding module used for processing the conditioning information from the final checkpoint to an intermediate one.Our results show that when the final checkpoint does enable use of naive input prompting for eliciting a capability, the embedding module can be patched to an intermediate checkpoint and we can retrieve the desired output at approximately the same time that alternative prompting protocols start to work well. This suggests the second phase of learning primarily involves aligning the input space to intermediate representations that enable eliciting the model capabilities. Overall, our results above yield the following hypothesis. Hypothesis 1. (Emergence of Hidden Capabilities.) Generative models possess hidden capabilities that are learned suddenly and consistently during training, but naive input prompting may not elicit these capabilities, hence hiding how "competent" the model actually is.To provide further support for our hypothesis, we provide results in a more realistic and naturalistic data. Specifically, we use the CelebA dataset [98] , which contains fine-grained attributes corresponding to concepts like Gender, With Hat, and Smiling, and analyze two settings.• Using concepts Gender and Smiling. Results are in Fig. 18 , where we find that concept learning dynamics discovered in the relatively simple setting in prior sections generalizes. We see that there is first a phase of concept memorization, wherein the model learns to generate images of (Female, Smiling). However, as training proceeds, there is a sudden turn in the learning dynamics and the model learns to generalize out-of-distribution. • Using concepts Gender and With Hat. In this setting, given a compute budget of 2M gradient steps, we find that the model seem to never learn to generalize out-of-distribution (see Fig. 7 ). However, when we perform latent interventions on the model's representations, we are able to force the model to produce images from the class (Female, With Hat). These results reiterate that the model is indeed capable of generalizing out-of-distribution, but is not performant when naively prompted. In the results above, we use conditioning information that perfectly specifies concepts underlying the data-generating process, i.e., h = z. In practice, however, instructions are underspecified and one can thus expect correlations between concepts in the conditioning information extracted from those instructions [99] [100] [101] [102] [103] . For example, images of a strawberry are often correlated with the color red (see Fig. 9 (a)). Correspondingly, unless a text-to-image model is shown explicit data stating "red strawberry" or images of non-red strawberries, the model's ability to disentangle the concept color from the concept strawberry may be impeded (see generations for "yellow strawberry" in Fig. 9 ). Motivated by this, we next investigate the effects of using underspecified conditioning information on a model's ability to learn concepts and capabilities to manipulate them. 
effect of underspecification on learning dynamics in concept space: 
 Experimental setup. The data generation and evaluation process follows the protocol described in Sec. 3.1. We select color (red and blue) and shape (circle and triangle) as the concepts, drawing an analogy to the "yellow strawberry" example. To simulate underspecification, we randomly select training samples that have a specific combination of shape and color (e.g., "red triangle"). We then mask the token representing the color ("red") and train the model on three concept classes {00, 01, 10}, represented by blue nodes in Fig. 9 , with some prompts masked. We test using class 11 (pink node) with no prompts masked to see if the model can generalize OOD.Underspecification Delays and Hinders OOD generalization. Fig. 8 shows how underspecification (masked prompts) affects the speed of concept learning. We see that as the percentage of masked prompts increases, the speed of learning a concept decreases, suggesting that underspecification leads to slower learning of concepts. Further, Fig. 10 (a) shows models' learning dynamics in concept space at varying levels of underspecification. With 0% masking, the model accurately produces an image of blue triangle (see Fig. 9 (b)). However, as the percentage of masking increases, the color of generated images shifts from blue to purple (middle), and finally red. This demonstrates that when prompts are masked, the model's understanding of shape triangle becomes intertwined with color red; even when blue is specified in the prompt, the dynamics are biased towards red.Toy Model of Learning Dynamics with Underspecification. When prompts are masked (i.e., underspecification occurs), target values for the concept variables are shifted: e.g., in our setup, with no mask applied, the target directions for the class 11 is (1, 1). When the word "red" in "red triangle" is fully masked, the target shifts to (0, 1). Assuming this shift is linear with respect to the percentage of masked prompts α, we can derive the following energy function.EQUATION)In the above, parameter s represents the impact of underspecification. Influence of Underspecification on Emergence of Hidden Capabilities. Following Sec. 4.4, we also explore whether it is possible to elicit the desired outputs from a model trained with underspecified data. Fig. 11 shows the accuracy results over five runs (a) without using any prompting method, and (b) using over-prompting. In both scenarios, the percentage of masking is set at 50%. Results clearly demonstrate that with over-prompting, the model achieves 100% accuracy after approximately 1,000 gradient steps, whereas without over-prompting, it fails in three out of five runs even after 2,000 gradient steps. These findings confirm that capability can develop prior to observable behavior, even in cases of underspecification. 
discussion: 
 Why study the concept space? One might ask why concept space could be useful beyond synthetic datasets. In Fig. 14 , we show the loss, accuracy, and training trajectory in concept space of a model from Fig. 6 . The moment during training when the model acquires the capability to manipulate size and color independently is not evident from either the loss or accuracy curve. However, in concept space, one can directly see the divergence of the trajectory from concept memorization. Benchmarking generative models is a challenging task, still often involving humans in the loop [105, 106] . Our concept space framework suggests that benchmarking core generalization capabilities can potentially be reduced to monitoring the learning trajectory in concept space. Moreover, this information can be used to develop better training schemes as discussed in Ren et al. [107] .We make a distinction between the potentially delayed elicitation of capabilities in our model versus grokking [108] . The phenomenology of delayed increase in performance on the test set, as seen in Fig. 14 , is shared. However, we deal with out-of-distribution evaluations, which is different from setups like modular arithmetic or polynomial regression in which grokking is usually studied [108] [109] [110] [111] [112] . Secondly, research on grokking using hidden progress measures indicates that the model gradually builds representations toward an ideal one [111, 113] ; however, our results find that even at the level of latent representations, there is a sudden emergence whereby the model learns the capability to manipulate concepts underlying the data-generating process and generalize OOD.Is concept learning a phase transition? In Sec. 4.4, we have demonstrated that before the capability to manipulate a concept is learned, the desired outputs cannot be generated regardless of the protocol used to prompt the model. Moreover, we showed that different protocols yield the desired outputs at the exact same time, which is also highly independent of model initialization (Fig. 6 , Fig. 17 ). We thus hypothesize that: Concept learning is a well-controlled phase transition in model capability. However, the ability to elicit this capability through predefined, single-input prompting can be delayed arbitrarily, depending on factors like the strength of the concept signal.Limitations One limitation of our work is that the compositional setup used in this study is rather simple. Real world concepts are often hierarchical [114, 115] or relational [116] . We also make assumptions that concepts are linearly embedded in the vector space Z in Sec. 3. However, this assumption seems to be at least partially justified as seen in Sec. D.8. Another limitation is that the findings are mainly drawn from synthetic data. While we show that our findings generalize to realistic data to some extent, as seen in Sec 4.5, Fig. 18 and Fig. 7 , further studies are needed to investigate how our findings apply to real data in general.Fig. 12 illustrates the concept space framework discussed in Sec. 3.Figure 12 : The Concept Space Framework. G is an invertible data-generating that maps process maps sampled vectors z ∼ P (Z) (where Z ⊂ R d ) to an observation x ∈ R n (an image in this work). The concept space, S := {z|z ∼ P (Z)}, is defined as a space of all possible concept vectors generated from a compositional prior P (Z). The mixing function M masks some concept variables, the masked concept variables are thus underspecified.We design a data generation process (DGP) compatible with the concept space framework introduced in Sec. 3. Our full DGP has six concepts: shape={circle, triangle}, x coordinate∈ R, y coordinate∈ R, color={red, blue}, size={big, small}, and background color={bright, dark}. We generally explore only a subset of these at a given time in our experiments.In Sec. 4, we fix shape=circle and x coordinate, y coordinate, background color are masked out. Thus the conditioning vector h only specifies color={red, blue} and size={big, small}. We sample both of these concept variables from a mixture of two uniform distributions, one component for each class (big, small) and (red, blue). Each value in each dimension is sampled from U(m i -s, m i + s), where m i is the class dependent mean. For example, color=red could be sampled from U(0.7, 0.9) × U(0.1, 0.3) × U(0.1, 0.3), where m 0 = (0.8, 0.2, 0.2) For brevity, we name the four resulting classes "00", "01", "10", and "11", where the class "11" is kept as the unseen target to evaluate out-of-distribution (OOD) generalization. For each training run, the DGP was initialized with a set random seed and 2048 images were generated in each class.In Sec. 5, we used two concept variables, shape and size. The training dataset included the classes "00", "01", and "10". For evaluation, we used the class "11". For each class, we generated a total of 1,000 images, each featuring objects of varying positions and sizes to ensure variability in our dataset.In App. D.5, we add the concept variable background color∈ R 3 to have a three dimensional concept space of (color, size, background color). In this case our training data is composed of the classes ("000", "001", "010", "100") and the test classes are ("011", "101", "110", "111")Our DGP is resolution agnostic, but we work with square images of 32x32 for fast experiments.Adjusting concept signal To adjust concept signals, we vary the class dependent mean of the two components of the mixture distributions. For example a strong color concept signal is achieved by drawing color=red from the mean m red = (0.9, 0.1, 0.1) and color=blue from the mean m blue = (0.1, 0.1, 0.9) whereas a weak color concept signal can be drawn using m red = (0.6, 0.4, 0.4) and m blue = (0.4, 0.4, 0.6). We scale the standard deviation of each component by the same ratio the Model Architecture We used a network architecture similar to the U-Net [95] used in Variational Diffusion Models [90] . We used the implementation publicly available in [117] . The architecture has 2 ResNet [118] blocks before each downsampling layer and a has self-attention layer in the bottleneck layer of the network. The U-Net has 64, 128, 256 channels at each resolution levels, an has a LayerNorm [119] and a GELU [120] activation. We used a sinusoidal timestep embedding [121] of 64 dimensions followed by a 2-layer MLP with hidden dimension 256 and a conditioning vector embedding using a 2-layer MLP with hidden dimensions 256 independent in every residual block.Optimizer We use the AdamW [122] optimizer with learning rate 0.001 and weight decay 0.01 to optimize the parameters of our network. We train our networks for 20K gradient steps. We use the default values for the decay rates: β 1 = 0.9, β 2 = 0.999.At training time we drop the conditioning to a -1.0 filled null vector ϕ with probability 0.2. This choice was made as opposed to the conventional choice of dropping it to the null vector since the null vector was near the interpolation limit of our model in the color subspace of the conditioning. The w cf g parameter is used to estimate the noise in the diffusion process by: εt-1 = f θ (x t |ϕ) + w cf g * (f θ (x t |h) -f θ (x t |ϕ)), where θ denotes parameters of the network f .Diffusion Process Hyperparameters We used the continuous time variational diffusion model [90] framework to (i) keep the sampling step parameter T an inference time hyperparameter and (ii) to allow the model to the adjust its optimal noise schedule for these images, especially since our synthetic images are expected to be different in SNR from natural images. In particular, we initialize our network with γ i = -5.0 and γ f = 10.0 (see [90] for the definition of γ) and use a learned linear schedule of γ(t). We use a reconstruction loss corresponding to a negative log likelihood from a standard normal with σ = 10 -3 centered at the data for the first step of the diffusion process.Evaluation probe Details. We used a U-Net [95] backbone with 64 output channels followed by a max pooling layer and n 1-layer MLP classifier for each of the n classes to estimate each concept of an image independently. We sample from the same data distribution but with maximal data diversity, i.e., with s i values in App. B.1 maximized within the range allowing perfect classification (no overlap of z between classes). We sample 4096 images per class from the DGP and train the classifier for 10K gradient steps with AdamW [122] and achieve a 100% accuracy on the held out test set. At evaluation phase, we average the classifier softmax output over 5 data generation / model initialization seeds and 32 inference samples to construct the concept space representation of the generations.Hyperparameter Search We conducted a hyperparameter search, testing batch sizes from 32 to 256, number of channels per layer from 64 to 512, learning rates between 10 -4 and 10 -3 , the number of steps in the diffusion process from 100 to 400, weight decay between 3 × 10 -3 and 5 × 10 -2 , and model weight initialization scale between N (0, 0.003) and N (0, 1). We also tried the Adam optimizer [123] with β 1 = 0.9, β 2 = 0.99, and weight decay of 10 -5 . No qualitative change were found in our results.Computational Details. We implement our models in PyTorch [124] . The diffusion model was trained on four Nvidia A100 GPUs and 64 CPUs for the data generating process. A standard model run (e.g., in Sec. 4.2) took ∼ 20 minutes on a single NVIDIA A100 40GB GPU. The CelebA runs took ∼ 24 hours on the same GPU. The full research required roughly 500 GPU hours, while the experiments in the paper would require roughly 100 GPU hours.Our code is available at https://github.com/cfpark00/concept-learning.Input Space: Overprompting. Our model is trained on a distribution of conditioning centered around a class dependent mean: p i (z j ) = U(m i j -s j , m i j + s j ) for each class i. We prompt the model with conditioning vectors extrapolated in the direction ⃗ m 1 -⃗ m 0 . For instance, assuming the red conditioning was (0.6, 0.4, 0.4) and the blue conditioning was (0.4, 0.4, 0.6), we "prompt" the model with (0.2, 0.2, 0.8). In practice, we use 5 conditionings (0.4, 0.4, 0.6), (0.35, 0.35, 0.65), (0.25, 0.25, 0.75), (0.15, 0.15, 0.85), (0.05, 0.05, 0.95), and report the maximum joint accuracy.Activation Space: Linear Latent Intervention. We demonstrate the ability to compose capabilities by manipulating the conditional vectors ⃗ h. Namely, we create a condition vector ⃗ h i for a specific concept i by specifying a concept of interest (e.g., ⃗ h blue = M (z blue ). During the forward pass, given ⃗ h, we can compute the component of each concept in ⃗ h by projecting onto a specific concept-condition vector ( ⃗ h blue ). We can then enhance or reduce the component of each concept by scaling each of these projected components. In practice, we perform the following operation: ⃗ h ′ = ⃗ h + α ⃗ h blue -β ⃗ h large , where α, β are hyperparameters. We sweep over [0.1, 1, 2, 4] for α and [0.1, 0.25, 0.5, 1] for β. ⃗ h blue is constructed by first deriving a blue direction in condition embedding space ( ⃗ h 5,5,95 ) by embedding a concept vector (⃗ z 5,5,95 ) where its RGB components are set to (0.05, 0.05, 0.95): ⃗ h 5,5,95 = M (⃗ z 5,5,95 ). We then project ⃗ h onto this direction to derive ⃗ h blue . ⃗ h large is generated similarly using ⃗ z size=0.7 . The model then generates an image conditioned on ⃗ h ′ .We plot loss, accuracy, and training trajectory in concept space of the models from Fig. 6 . The time at which the model acquires a capability to manipulate the concept color is not evident from the loss or accuracy curves; however, in concept space, it is evident that the concept is learned together with the departure from concept memorization. Benchmarking generative models is a challenging task, still often involving humans in the loop [105, 106] . Our concept space framework suggests that benchmarking out-of-distribution (OOD) generalization can potentially be reduced to monitoring the learning trajectory in concept space. D.2 Additional trajectories of learning dynamics (Fig. 15 )Here we show all concept space trajectories for the experiments mentioned in Fig. 4 (a, b ), for all classes and color concept signal levels. We find asymmetric behavior for the 01 class and the 10 class when adjusting the color concept signal level. The dynamics of the generations in the training set matches our intuitions of concept signal as discussed in the main text. At low color concept signal, we observe that the dynamics fit size first for both 00 and 10, and afterwards fine their correct colors. We show the standard deviation of compositional generalization accuracy for different prompting methods in Fig. 17 . We find that the standard deviation is large for the naive prompting method, while linear latent interpolation and overprompting extract the hidden capability robustly, resulting in a small standard deviation. To assess our findings on a more realistic dataset, we ran experiments on the CelebA [98] dataset. We selected the Female and Smiling features as the two concept dimensions to explore. This choice was motivated by the relatively balanced number of samples in all four classes(00=(Male, Not Smiling), 01=(Male, Smiling), 10=(Female, Not Smiling), 11=(Female, Smiling)), from which we randomly sampled 30K images in each class. To construct the concept space, we trained a fully convolutional network with a average pooling layer followed by a classification MLP head to classify the two attributes with independent cross entropy losses. (See B.2). We trained the classifier with the AdamW [122] optimizer with learning rate 10 -3 and weight decay 10 -5 for 10K gradient steps. The classifier achieved a final accuracy of respectively 95% and 97% on the held out validation set, which was 10% of the entire dataset.We trained the same diffusion model (See App. B.2) from our synthetic experiments on 64x64 resized images from the classes (00, 01, 10) and assessed the out-of-distribution (OOD) generalization to 11.We used a color jitter of 0.1 for brightness, contrast and saturation and randomly flipped the images horizontally. As the class attributes are categorical, they were one hot encoded and concatenated to an input conditioning vector of 4 dimensions. The diffusion model was trained for 2 × 10 6 gradient steps with a batch size of 64 with the same optimizer as in the main experiments.The concept space dynamics of generations from the in distribution conditioning and OOD conditioning are shown in Fig. 18 . We find similar observations from the concept space trajectories as in our synthetic experiments. Initially, the images corresponding to the class 11 follows the concept space trajectory of 10, optimizing Female, which we intuitively expect to have a stronger concept signal, although it is intractable to compute since the DGP is unknown. Similar to our synthetic experiments, we see a transition in the concept space trajectory of the compositional class corresponding to the model disentangling the concepts. After this transition, concept learning begins but we see biases, where we see a trade-off of moving in the right direction towards one concept degrades the other one as seen in Fig. 4 (b) . The visual generations in Fig. 18 confirm that our findings are not merely a result of an ill-calibrated classifier model. However, in this case, we do not observe full out-of-distribution (OOD) generalization at 2M gradient steps. We expect this task of generating (Female, Smiling) is inherently harder than our synthetic setup. We note that the goal of this experiment is not to show good compositional generalization but it is more on confirming that our qualitative findings generalize to real data without touching the model or training method.D.5 Experiments with 3D concept space (Fig. 19 )To further verify our findings in Sec. 4, we explore a setup where the concept conditioning specifies three concepts: color, size and background color. Fig. 19 illustrates two example scenarios in which the concept signal in color and background color are varied, respectively resulting in cases of success and failure of OOD generalization. The length of edges of the cuboids represent their concept signal magnitude. In Fig. 19 (a), we see that the object color has a strong concept signal and this is reflected in the concept space trajectories as this direction being split first. In this case we see, similar to the blue generalization curve in Fig. 4 observe that the 011 class initially undergoes concept memorization for class 010, which shares the two stronger concept signals color and background color, and shows a transition where it leaves this phase. In Fig. 19 (b), we see a case where out-of-distribution (OOD) generalization did not succeed within the given 80K gradient steps. In this case, we see two classes, 011 and 111, which are not correctly learned, and thus suspect that the model did not learn to generalize the background color. However, the concept space trajectories suggest that the capability is present. An implication of our conclusions in Sec. 4.4 is that before the model has passed the transition point where the model has the capability to compose concepts, the model should not be able to generate small blue circles no matter how well "prompted". Here, instead of prompting, we explore Classifier Free Guidance (CFG) [97] to see if our findings apply to a conditional diffusion model trained with CFG. In Fig. 21 , we see that even the models with CFG show this transition from concept memorization to out-of-distribution (OOD) generalization. In a scenario where there isn't a sharp acquisition of the capability to compositionally generalize, we would expect the sharp transition to disappear with CFG scale. Assuming a model does not have the capability to generate images from an OOD concept class, we simply swap the embedding module for processing the conditioning vector h with that of the last checkpoint. One possible interpretation for this method is that the embedding module disentangles the concepts, i.e., generates a representation for each concept, while the U-Net [95] then utilizes such representations. This would imply that the U-Net [95] already learns how to utilize concept representations early during training, while further gradient steps lead to more robust concept representations.We test our approach on 5 random initialization seeds. Results are shown in Fig. 22 and we find that for some seeds, we are able to elicit the target behavior at around the same time in which overprompting and linear interventions also elicit the target behavior; for other seeds, this is not the case. Interestingly, we find that our method works well for seeds with "stable" learning dynamics, in that once the model reaches ∼ 100% accuracy, it converges and stays at such accuracy, while some seeds have "unstable" learning dynamics in that their accuracies oscillate.We show simple proof of concept experiments on frontier models in Fig. 23 . In Fig. 23 (a), we show that CLIP [125] already embeds text/images with compositional concepts in its vector space as a Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: While we report some error bars and show multiple seeds when applicable, we did not test for statistical significance as most conclusions were qualitative.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This work introduces the idea of "concept space" in the context of conditional generative models and considers the learning process as one that moves from "concept memorization" to generalization. In experiments, the authors demonstrate that, in many cases "capability precedes behavior": models are able to compose concepts under forceful intervention even though they do not successfully generate under normal conditions.

Reviewers found the idea of concept space promising as a lens through which to analyze the dynamics of learning. In particular, the evidence presented that some capability is present in models before it is expressed was deemed highly interesting. Reviewers had some concerns about the limited range of experiments in the model, but the addition of new experiments allayed many of these concerns. I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces "concept space", a framework for analyzing the learning dynamics of generative models, focused especially on compositional generalization. The key contributions are:

- Introducing the concept of "concept signal" that governs the rate of concept learning (and in particular determine learning speed), and is the key driver of the generalization dynamics
- Analyzing learning dynamics using the concept space framework on a synthetic dataset of 2D objects.
- Claims about how concept signals shape the geometry of learning trajectories.
- a proposal on interventional protocols to uncover hidden model capabilities.

The paper uses a conditional diffusion model trained on synthetic data to study how concepts like shape, color, and size are learned and composed together. They show that concept signal levels determine learning speed and generalization dynamics, and that model capabilities often emerge before observable behaviors. The paper also explores how underspecification hinders compositional generalization.
strength_weakness: strength_weakness: Significance: 

- Overall the setup is too simplistic to have the claims be transferable to a broader set of generative models or even diffusion models. All the experiments use a simplified synthetic dataset with 2D objects and binary concept variables; while good for control, it limits the complexity of the kind of concepts real-world models learn by a massive amount. Real world models are learning more complex, hierarchical, and interdependent concepts, not simply characterizable by things like size or color, and use continuous concept variables rather than binary.  
- The concept signal relies on knowing G, the data generating process. In real-world models, the true data generating process is often unknown, concepts might not have clear, differentiable manifestations in the input space. 
- The toy model for learning dynamics is based on a simple energy function with sigmoid activation; it's not clear why "concept signals" should be able to capture any non-linear interactions between concepts. 

Presentation of the claims: 

- The authors use the word "capability" when they really mean the model's internal ability to understand and compose concepts. The wording is quite unclear and should be clarified. 
- The graphs are pixelated, low-quality, poorly labeled and hard to understand. In particular, you should explain what the colors and the axis clearly mean in Figure 3, have multiple random seeds analyzed with error bars for overprompting and linear latent intervention in Figure 4, and make Figure 5 (b) higher quality. 
- Linear latent intervention assumes linear separability of concepts in the latent space, which breaks down in more complex settings than the synthetic data setting. 
- There should've been a higher focus on results on CelebA in the main sections of the paper.
questions: questions: - Would we expect overprompting to still work when concept manifestations are complex? 
- Doesn't the embedder patching technique assume that the final checkpoint has disentangled concepts? Why should we expect to be the case? 
- The paper briefly mentions experiments with the CelebA dataset in the appendix. Could the authors elaborate on how well the findings from synthetic data translate to this more complex dataset?
limitations: limitations: - The paper should've compared the concept space framework to other existing methods for representations in learning dynamics (e.g., basic linear probing techniques, representational similarity analysis) as baselines for our understanding to see how much "concept space" improves it. 
- The models are fairly simple and small, and the study should further look into whether any of the techniques (e.g. overprompting, linear latent intervention etc.) still continue to hold in setting that more closely resemble real-world conditional diffusion models.
paper_summary: paper_summary: This manuscript investigates the distinction between capabilities and behaviors in diffusion models. Using a toy task, the authors analyze at what point during training the model begins generating objects with correct specific features (in particular color and shape). They find that increasing the salience of certain attributes (“concept signal”) makes learning of those attributes faster and further causes the model's compositional generalization to collapse onto the nearest training point after intermediate training times. They replicate qualitative features of these learning curves with a mathematical toy model. They then use different interventions that enable the model to generalize compositionally much earlier than they would otherwise --- the fact that these interventions all yield generalization at the same time suggests a specific time point where the model has learned the underlying capabilities without this necessarily manifesting in behavior. Finally, they use this toy setup to analyze the impact of underspecification on compositional generalization.
strength_weakness: strength_weakness: As noted above, I think there are several interesting findings in this manuscript. However, in its current stage, I believe that it falls short of its stated goals. More specifically:

1. As far as I can tell, sections 4.1-4.3 and section 4.4 get at different forms of "capability." Specifically, the concept space studied in sections 4.1-4.3 distinguishes whether the model is able to generate certain properties of the image (e.g. color and size). In contrast, section 4.4 demonstrates that certain interventions can improve the models accuracy on the compositional task substantially. Since those interventions don't fundamentally change the model's capabilities, this suggests that the model has already learned to do the right thing and these interventions simply surface that capability. It's unclear to me whether that is necessarily apparent from the concept space behavior. Put differently, the model, in principle, could still be performing extremely badly according to the concept space but have already exhibited this transition. (You seem to be getting at a related point in section 6, "Why concept space?" and the supplementary figure, but it's unclear to me what the time of concept of acquisition is. To the extent that it relates the findings in section 4.4 to the findings in the previous sections, I think you'd have to show this across different models rather than using a single example of a model.)
2. I did not understand the role played by equation (1). First, the defined energy landscape always has its minimum (for high t) at $c_1,c_2=1$, so why can these curves tend to different corners of the concept space? Second, I did not understand how this differential equation is grounded in model behavior, as it does not appear to be a simplified learning model. Rather, the goal seems to be to recapitulate the (rough) trajectories in concept space, so I'm not sure what insights are gained from that. In particular, the fact that the model first tends towards the correlated training point (e.g. the small red circle) during learning is built in by the definition of the sigmoidal function.
3. I think the related work section on compositional generalization should provide a better overview of existing insights into the questions you're asking. Right now, you're only citing a number of papers investigating these questions, but I think it would be important to actually give an overview of what these papers are presenting and investigating and how it relates to your own findings. In particular, it's unclear to me what was previously known about the impact of underspecification on compositional generalization.
4. I think it would be important to report standard errors or some sense of deviation across different model seeds. It is still important in my opinion to understand the reliability of these qualitative findings --- e.g. how similar are the concept space curves you're presenting across different initializations?

All in all, I think the paper presents several interesting findings, but, in its current state, leaves unclear how these findings fit together. On the one hand, the concept memorization finding is interesting and works well together with the underspecification finding. I think for both of those findings it would be important to more thoroughly evaluate how reliably the model actually ends up generalizing compositionally (e.g. across different seeds of initialization). In addition, it would also be helpful to give additional mathematical insights (or give an intuition in a different way) into why the observed behaviors emerge, as I don't see the current mathematical model as helpful on this end. On the other hand, the finding in section 4.4 is also interesting, but it remains unclear how it is related to the concept space framework and, if it can be explained in terms of learning both of these capabilities, why the presented intervention mechanism can help the model generalize compositionally.
questions: questions: 1. Could you clarify how sections 4.1-4.3 and section 4.4 are related (see weaknesses, point 1)?
2. Could you clarify how you determined equations (1) and (2) and what we can learn from them (see weaknesses, point 2)?
3. As noted, I think the findings in section 4.4 are really intriguing, so I want to make sure I understand exactly what is going on there. a) Since you're using a binary classifier to assess performance, are the overprompted colors/sizes really identical to the original colors/sizes or is it possible that e.g. the color produced from overprompting is different from the ground-truth color and just more clearly on the correct red/blue side of the hyperplane? It would be useful to see a few examples of generated images here. b) It seems that you're only using one model seed in Figs. 4(c) and (d). For 4(d) you explained that you only used the one with "full capability" (I assume that's the one with an accuracy close to 1.0?). Why did you only use one model for Fig. 4(c) (or are these multiple lines that are just strongly overlapping?)? c) I think the fact that there's only one model that reaches full compositional generalization accuracy qualifies these findings a bit, in particular as it means that the latter methods only present a single sample. Would the other models improve in their performance as well if they were trained for longer? d) I would suggest providing an additional figure where you plot the different curves for each model seed on top of each other, as it is currently a bit difficult to judge how precisely the transition times overlap.
4. I'm not very familiar with the literature on underspecification --- have other papers previously noted its effect on compositional generalization (i.e. the "strawberry"/"yellow strawberry" effect)? If not, I think it'd be worth emphasizing that a bit more --- if yes, it would good to emphasize that as well.

**A couple of minor notes**

L. 29: “pre-training on such models” -> “pre-training of such models”

L. 32: What is the “model experimental systems approach”?

L. 69: Space before citations missing.

L. 96: Is $S$ just the support of the probability distribution?

L. 99: Is $F$ a stochastic function?

L. 117-118: Isn’t $G^{-1}(Y)$ in the concept space, i.e. it should match $z$, not $h$?

Figure 2: Why do the two trajectories for 01 and 10 illustrate concept memorization? I would have thought that this was illustrated by the trajectories for 11 that end up near 01.

L. 247: What does it mean to mask the token? Set it to zero?
limitations: limitations: As the authors acknowledge, they focus on toy synthetic data here. Furthermore, their analysis is largely empirical in nature, leaving unclear the exact reasons why the models generalize or don't generalize. I think the authors have adequately communicated the limitations of their work overall.
paper_summary: paper_summary: This paper introduces a new framework, "Concept Space," designed to study the learning dynamics of capability and behavior in generative models. It uses the concept signal to measure the rate of concept learning. They trained a diffusion model on a simplified synthetic dataset to validate their hypotheses that transitions in model capability occur much faster than the corresponding changes in behavior. This work provides empirical insights into understanding and controlling the development of AI models' capability and behavior.
strength_weakness: strength_weakness: 1. The rationale for modifying the differences in attribute values to represent different levels of concept signals is unclear. How does this relate to the formula $σ_i=|\partial G(\vec{z})/(\partial z_i )|$
The derivation of equation (1) is also not clear.

 2. Uncertainty in generalization: Despite the introduction of the Concept Space framework, there is still uncertainty about the model's ability to generalize to real-world data. While concept learning is easily validated on synthetic datasets, it is unclear how to effectively validate this approach on large-scale datasets and large models. Can you provide some insights?
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces a novel framework for understanding concept learning in text-conditioned generative models. The experiments are carefully designed (although most of them look a bit toyish) and support the analyses well. The three key findings, i.e., concept signal levels determine the learning dynamics; capability consistently emerges before observable behaviors; under-specification harms the system, are quite novel and could provide many insights to the community. The proposed theoretical formulation also has the potential to be further developed. I enjoy reading this paper and suggest an acceptance.
strength_weakness: strength_weakness: - In Section 3, it is a little hard to understand the relationships between different variables and functions. A diagram like Figure 1 in [2] might be helpful.
- Although the experiments and analysis of the paper are very insightful and persuasive, it is only verified in a relatively simple setting. There are experiments on CelebA, but the number of attributes and their possible values are still kind of small. One or two experiments considering more complex scenarios will strengthen the paper a lot.
- The paper claims that the model usually has the capability to compose the learned concepts earlier than this behavior shows up. How will this finding influence our understanding of (or provide insights to improve) the practical systems? Does this correlate to early stopping (the switching point from underfitting to overfitting)?
questions: questions: - The legends in Figure 2-bc (so as other learning dynamics figures) are a bit hard to read.
- In Figures 2 and 3, the learning curves exhibit a “zigzag” pattern, i.e., first pointing to the training example sharing the strongest concept value, and then converging to the memorization one or comp-gen one. A similar phenomenon is also observed in a general classification problem discussed in [1]. Is there any similar reason behind these two findings?
- Both the concept signal and sigmoid function are denoted by $\sigma$, which is confusing.
- What are (0.4, 0.4, 0.6) to (0.3, 0.3, 0.7) in line 216 means, normalized RGB values?
- The color bar of Figure 6 might have the wrong legend (it is the percentage of masked $\alpha$, but the bar ranges from 1 to 3.5).
- The paper concludes that  (in line 300) concept learning is a well-controlled phase transition, but the observed behavior can be arbitrarily delayed. Are there any methods that can reduce this delay?
- [2] studies a very similar compositional generalization problem (but in a representation learning setting). It would be helpful to also discuss that in the related work. The discussion of simplicity bias and Kolmogorov complexity in [2] might also be a potential explanation for the observations presented in this paper.

[1] Ren, Yi, Shangmin Guo, and Danica J. Sutherland. "Better supervisory signals by observing learning paths." ICLR 2022

[2] Ren, Yi, et al. "Improving compositional generalization using iterated learning and simplicial embeddings." NeurIPS 2023
limitations: limitations: The experimental setting is kind of simple. But I think it is sufficient for a phenomena explanation (and theory) paper.

Comments: comment: Thank you for the detailed response. I really appreciate the idea and motivation behind this paper. The rebuttal has significantly improved the quality of the paper, and I now feel more confident in recommending its acceptance. As a result, I have raised both my score and confidence level.
Thanks very much for the author's hard work. The new results and discussions indeed strengthen the paper. I would like to increase my evaluation to 8. I am looking forward to seeing the final version of the paper.
Many thanks to the authors for their careful rebuttal, which has addressed many of my concerns. In particular, I appreciate the authors adding a quantification of uncertainty to their figures and I will increase my score to 5.