Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Authors: Yujin Han
Andi Han
Wei Huang
Chaochao Lu
Difan Zou
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Deep Generative Model
Primary area: theory->deep_learning

Abstract: Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e.g., the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.

Can Diffusion Models Learn Hidden Inter- Rules Behind Images?
Yujin Han "
Andi Han 22 Wei Huang? Chaochao Lu Difan Zou
Abstract
& Nichol, 2021; Karras et al., 2022; Tian et al., 2024b), videos (Ho et al., 2022: Yu et al., 2024; Yuan et al., 2024b), and audio (Liu et al., 2023a; Yang et al., 2024b; Lemercier et al.,, 2024), they still encounter specific failures in synthesis quality, such as anatomically incorrect human poses (Borji, 2023; Zhang et al., 2024; Huang et al., 2024) and misalign- ment between generated content and prompts (Feng et al., 2022; Borji, 2023; Chefer et al., 2023; Liu al., 2023b; Lim & Shim, 2024), which could harm the reliability and applicability of DMs in real-world scenarios.
Despite the remarkable success of diffusion mod- els (DMs) in data generation, they exhibit spe- cific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image fea- tures. Specifically, for image data with dependent features (x) and (y) (e.g.. the height of the sun (x) and the length of the shadow (y)), we inves- tigate whether DMs can accurately capture the inter-feature rule (p(y) x)). Empirical evaluations on mainstream DMs (e.g.. Stable Diffusion 3.5) reveal consistent failures, such inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these find- ings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learnng abilities. Extensive experiments show that while DMs can identify coarse- -rrained rules, they strug- gle with fine-grained ones. Our theoretical analy- sis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique- incorporat- ing additional classifier guidance during sampling. which achieves (limited) improvements. Our anal- ysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.
We focus on a specific type of failure with limited attention: the failure of DMs in learning hidden inter-feature rules behind images. Specifically, consider image data containing dependent feature pairs (x.y), such as the height of the sun (x) affecting the length of a pole's shadow (y). Our investigation centers whether DMs targeting the joint distribution p(x, y) can accurately capture the underlying relationships between x and y, effectively recovering the conditional distribution p(y|x). Theoretically, a diffusion model that perfectly estimates the joint distribution should naturally capture the conditional distribution, thereby learn- ing the latent rules between features. However, in practice, numerous factors, such non-negligible score function es- timation errors, can cause the sampled joint distribution to deviate significantly from the true distribution (Chen et al., 2022; 2023; Benton et al., 2024). How do these deviations propagate to inter-feature rule learning? This gap between theory and practice remains largely unexplored.
Although existing studies have explored whether DMs can learn specific rules, they primarily focus independent features, such as DMs compositional capabilities (Okawa et al., 2024: Deschenaux et al., 2024; Wiedemer et al., 2024), Some works have investigated inter-feature dependencies in DMs, but the varying complexity of rules has led to contra- dictory findings. For example, DDPM has been reported to fail in generating images satisfying numerical equality con- straints (Anonymous, 2025), while succeeding in reasoning about shape patterns in RAVEN task (Wang et al., 2024a). These inconsistencies highlight the need for a unified ex- perimental setting that allows for adjustable rule difficulty, enabling an accurate evaluation of DMs' rule-learning capa- bilities. Moreover, existing studies rely heavily on empirical observations, lacking theoretical analysis elucidate the limitations of DMs in rule conformity.
1. Introduction
Despite the remarkable capabilities demonstrated by dif- fusion models (DMs) in generating realistic images (Ho et al., 2020; Song et al., 2020; Vahdat et al., 2021; Dhariwal
'Equal contribution 'The University of Hong Kong "The University of Sydney 'RIKEN AIP "Shanghai AI Laboratory. Correspondence to: Yujin Han <yujin- han @ connect.hku.hk.. Andi Han <andi.han @andi..an@sydeey.c.oo adu. Difan Zou <dzou @cs hk..kk.kk>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Can Diffusson Models Learn Hidden Inter-Feature Rules Behind Images?
Generated Images
Synthetic Tasks
Coorse 3mm Skotion
B
Coorse N1 2 1/
3D-3.5 Large
Fine 1,4g = lgh
Fine 4/4 = 1,42
Spatial
Flux. 1
Dev
Rule Type:
(a) Light shadow
(b) Reeflection/Refrattiio
(c) Semantics
Sun Pole Shadow
Distance Pole Length
Coarse
r # ns
D
Coarse: r < /2
SpntSaatial-
SD-3.5 Large
Fine. f2 v2n
Fine 1.5%
Flux.1 Dev
Rule Type: (d) Size -Texture
(e) Size/Region- Color (f) Color Color
Big Small Circle Size
Region Square Size
Figure 1. Synthetic Tasks Inspired by Real-World Insights. Based on whether fettrrfeature rules involve spatial dependencies, we
categorize the failure cases into spatial and non-spatial rules. Spatial rules include: (a) Light-shadow, where evaluated DMs generate unreasonable multiple shadows or incorrect shadow flips; (b) Reflection/Refractiion showing incorrect mirror rules or missing refraction effects below water surface; (c) Semantics, such inconsistencies between sunflower orientation and sun position, brush and canvas colors. Non-spatial rules involve: (d) Size-Texture, like mismatches between tree diameter and growth rings; (c) Size/Region-Colee where evaluated models fail to capture hurning candle's 's color variations and size-color relationships (e.g., red giants and white dwarf); (f) Color-Color as in Eclectus parrots' body-beak color correlations that DMs fail to maintain. Appendix C provides detailed explanations for each case. These failures mainstream DMs handling real-world inter-feature rules inspire our design of four synthetic tasks.
Our investigation into inter-feature rules begins with observ- ing the limited ability of mainstream DMs (e.g., SD-3.3 La rgge F1 1 Dev) to capture real-world inter-feature rules, as illustrated in Figure 1, even though these mod- els perform well metrics such as FID Their errors in inter-feature relationships evident in various scenarios, such as inconsistent relationships between sun positions and building shadows, mismatched reflections of toys in mirrors, and sunflowers failing to face the sun. Then, we carefully design four synthetic tasks to reflect real- rorl-world rule failures, ensuring the practical relevance of our findings. The rule of each task features two difficulty levels: coarse-grained rules (e.g., the and polc's shadow should be opposite sides) and fine-grained rules (e.g., the shadow's length precise function of the sun's 's height). This hierarchical, controllable framework enables a comprehensive evaluation of DMs' rule-learning capabilities. Next, through extensive experiments considering various factors including model architectures. training data size, and image resolution, we reach consistent conclusion: DMs effectively learn coarse- grained rules but siruggle with fine-grained ones.
specified in terms of norm. We prove a constant error lower bound on learning the hidden rule via optimizing the DSM objective (Ho et al., 2020) with a two-layer network. This demonstrates the incompatibility between learning joint dis- tributions and identifying specific inter-feature rules.
Recognizing DMs' difficulty in learning inter-feature rules, we mitigate this issue by constructing contrastive pairs that satisfy either fine-grained or soarse-grained rules and then using them to train a classifier as additional guidance. While this strategy enhances rule-compliant sample generation, fur- ther improvements are still achievable. The in in-depth anal- ysis identifies that fine-grained rules exhibit weak signals, making accurate classifier training particularly challenging. We summarize our key contributions as follows:
Empirically, inspired by mainstream DMs' struggles with real-world inter-feature rules, we innovatively create syn- thetic tasks with coarse/fine -tars//fine-rraieedd rules to systematically assess DMs' rule learning ability in Section 3.1.1. Extensive experiments in Section 3.3 show that while DMs can learn coarse rules, their ability to grasp precise rules is limited.
Furthermore, we develop rigorous theoretical analysis using a multi-patch data model with an inter-feature rule
Theoretically, we rigorously analyze DMs on a synthetic multi-patch data distribution with a hidden norm depen- dency Section 4. We prove that the unconditional DDPM cannot learn the precise rule of norm constraint, which ex- hibits at least constant error in approximating the desired
Appendix A lists Mixture Gaussian as an example to demon- strate that low FID and incorrect inter-feature relationships in DMs generations not contradictory.
2

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
score function. This identifies the limitation of the current DMs training paradigm and necessitates further improve- ments for learning hidden rules behind images.
conclusions on DDPM's rule-learning ability. For example, DDPM struggles with numerical addition rule (Anonymous, 2025) but maintains shape consistency rule in RAVEN task (Wang et al., 2024a). Inconsistent rule complexity leads to ambiguous evaluation conclusions, and the lack of theo- retical analysis leaves the underlying factors behind DMs performance rule learning poorly understood. Through controlled experiments with adjustable rule complexity, we provide unified assessment of DMs' rule-learning abili- ties and offer a theoretical explanation of their fundamental limitations, as result of their training paradigm.
Methodologically, we mitigate DMs' inability to learn fine-grained rules by introducing guided diffusion with a contrastive-trained classifier in Section 5. However, the chal- lenges of accurately classifying fine-gre aee-grrined rules identify room for improvement in our strategy. This problem, dis- tinet from traditional classification tasks, involves detecting subtle distinctions between fine-grained and coarse- parsee-raaied rules, highlighting valuable insights for future exploration.
3. Exploring Hidden Inter-feature Rule Learning via Synthetic Tasks
2. Related Work
We summarize prior studies on the ability of DMs learn specific rules, and discuss the relations inter-feature rules.
In real-world image generation tasks, rules between features are often complex and difficult to define or quantify pre- cisely. To systematically investigate DMs' ability in rule learning, as previous work (Okawa et al., 2024; Deschenaux et al., 2024; Anonymous, 2025; Wang et al., 2024a), we de- sign simplified and controllable synthetic tasks in Figure 1. These synthetic tasks not only provide explicitly defined inter-feature rules but also abstract essential feature rules present in real-world data, thereby making our conclusions practically relevant. For example, Synthetic Task A in Fig- ure I simulates the Light-Shadow relationship, while Task B simplifies the physical rules of Reflection//efraction..
Factual Knowledge Rules. The violation of factual rules in DMs refers to generated images failing to accurately reflect factual information and common sense, often characterized as hallucinations in existing work (Aithal et al., 2024; Lim & Shim, 2024; Anonymous, 2025). Typical examples include violating common sense, such as extra, missing, or distorted fingers (Aithal et al., 2024; Pelykh et al., 2024; Ye al., 2023), unreadable text (Gong et al., 2022; Tang al., 2023; Xu et al., 2024) and snowy deserts (Lim & Shim, 2024). Additionally, inconsistencies between generated images and given textual prompts (Liu et al., 2023b; Fu & Cheng, 2024; Mahajan et al., 2024; Li et al., 2024b) can be regarded as violations of prompt- based knowledge. Unlike inter-feature rules, factual knowledge rules do not involve relationships between multiple features and are typically attributed to imbalanced training data distribution (Samuel et al., 2024) or mode interpolation caused by inappropriate smoothing of training data (Aithal et al., 2024).
3.1. Synthetic Tasks Inspired by Real-World Insights
1. REAL-WORLD HIDDEN INTER-FEATURE RULES
Inspired by Borji (2023), we investigate several common scenarios where inter-feature rules exist, as illustrated in Fig- ure 1. Specially, we categorize these hidden rules into two types, spatial rules and non-spatial rules, based on whether the inter-feature relationships exist in the form of spatial arrangements or feature attributes themselves.
Independent Features Rules. Prior work has investigated DMs' ability to combine independent features, i.c., composi- tionality. Through controlled studies with independent con- cepts (e.g., color, shape, size), Okawa et al. (2024) observe that DDPM can successfully compose different independent features. Similar findings are reported in (Deschenaux et al., 2024), where interpolation between portraits without and with clear smiles resulted in generations with mild smiles. However, numerous studies highlight DMs limitations in complex compositional tasks (Liu et al., 2022: Gokhale et al., 2022; Feng et al., 2022; Marioriyad et al., 2024), po- tentially due to insufficient training data for reconstructing each individual feature (Wiedemer et al., 2024). These stud- ies primarily examine compositional tasks with independent features, in contrast to our focus on feature dependencies.
Spatial Rules are defined as constraints on the relative posi- tions and layouts between features, such as the correlation between the sun's height and the shadow's 's length. In Fig- ure I, scenario Light-shadon demonstrates how the position of a light source should precisely determine the placement of building shadows. However, both 8-billion Multimodal (Rombach et al., 2022) and 12-billion
SD-3. 3.5 model Flux.1 1 Dev² Dev²(Labs, (Labs, 2023), fail to generate proper shadows, either producing incorrect directions or merely cre- ating symmetrical duplicates of the actual buildings. Sim- ilarly, in scenario Reflection/Refractionn, while objects in front of mirrors should dictate the layout of their reflections, we observe completely unreasonable generations from both
A bstracct (Dependent Feature) Rules. This type closely aligns with our work, which studies feature relationships like shape consistency in generations. Prior studies give mixed
large
'htpps//ialaiimmmdellllaaaaa
3

---

Can Difflsson Models Learn Hidden Inter-Feature Rules Behind Images?
models. Furthermore, semantic consistency in Semantics scenario is violated, shown by sunflowers not facing the sun and mismatched paint colors between brush and canvas.
Drigis Emage
(b) Calon-band Madk
Festure Extraction
1
2
3
L
L
Non-Spatial Rules are defined as correlations between in- trinsic feature attributes, such as the relationship between an object's size and its color. For instance, in type Size -Txxture, tree trunk features should exhibit precise corre- lations between the diameter and annual ring count, and candle flames in type Size/Region- Color should show con- strained relationships between different flame zones and their colors. However, these fine-grained inter-feature con- straints are ignored by both SD-3.5 Large and Flux 1 Dev. More detailed discussion and additional experiments for more advanced DMs are deferred to Appendix C.
Figure 2. Pipeline for extracting features. Given an image, we first apply color bolor-baeed mask by using predefined colors, then count whether the number of masks meets expectatioon expectations, and finally extract fcatures of interest by marking the key points within masks.
requires that the upper square's 's side length l be smaller than the lower square's side length 12, i.e. 1 < l2, while the fine- grained rule specifically requires 12 = 1.5/,
3.2. Experimental and Evaluation Setup
3.1.2. SYNTHETIC TASKS
After designing synthetic tasks with well-defined inter- feature rules, we can systematically investigate the capa- bility of DMs learn these underlying relationships.
Inspired by real-world rules in Section 3.1.1, we design four synthetic tasks (A-D), each with two levels of rule granularity (coarse and fine), as shown in Figure 1. We provide a brief overview of synthetic tasks here, with more details presented in Appendix D. Specially,
Experimental Setup. In subsequent experiments, train an unconditional DDPM (Ho et al., 2020) on four synthetic tasks. Unlike latent-space DMs (c.g.. SD-3.5 Large), pixel-space DDPM makes the conformity of inter- feature relationships potentially simpler, as no additional compression-induced information loss occurs (Rombach et al., 2022; Yao & Wang, 2025). Following the training setting (Aithal et al., 2024), we fix the total timesteps at T 1000 and employ the widely-used U-Net architecture (Ronneberger et al., 2015) as the denoiser. 4000, 2000, 2000, and 2000 samples are generated for synthetic task A, B, C and D, respectively, with an image size of 32 x 32. Additionally, in Appendix E.3, we explore more advanced architectures such as DiT (Peebles & Xie, 2023) and SiT (Ma et al., 2024), alongside larger synthetic datasets of 20000 and 40000 samples and higher image resolutions of 64 x 64. These factors enhance the training of DMs, thus leading to better alignment between generated and real data distributions (Chen et al., 2022; Benton et al., 2024; Chen et al., 2023) and enabling more effective learning of hidden rules. More experimental details are in Appendix E.
Task A is inspired by the spatial rules behind the Light- shadow case, simulating the physical law between the sun and pole shadows. In Task A of Figure I, the coarse-grained rule requires the sun and shadow to be on opposite sides of the pole, while the tieeggained rule requires sun's 's center, pole top, and shadow endpoint align linearly, i.e., satisfying 21 1ghn (see notations in Task A, Figure 1).
Task abstracts the spatial rule from the Reflec- tion/Refraction case, where object's reflection size de- pends on its size and distance from the mirror. Task B uses two rectangles with lengths hn and h2 (notations shown in Task B, Figure 1) to simulate this perspective rule, where size diminishes with distance. Assuming the viewpoint is at the leftmost edge, the coarse-grained rule requires the left rectangle (closer to the viewpoint) to be longer than the right one (farther from the viewpoint), i.e., hi > h2, while the tite-grained rule dictates rectangle lengths be proportional to their distances from the viewpoint, i.e., I h2 = l2h1,
Evaluation Method. To evaluate whether generated images follow the inter-feature rules, Figure 2 designs three-step feature extraction pipeline: (1) Color-based Mask: Segment element masks (e.g.. sun, pole, shadow in Task A) based on predefined color (HSV) ranges when synthesizing training data; (2) Elements Count: Apply contour detection based on masks to verify the presence of essential elements, mark- ing images as Invalid li if any are missing; (3) Feature Extraction: Extract key feature points (e.g., sun center, pole top/center and shadow endpoint in Figure 2) and compute geometric features of interest, such as horizontal sun-to-pole distance I, vertical sun-to -on-to-pole-top distance h1, pole height h2, shadow length Iz. All features are scaled to [o, 1] by dividing them by the image size to eliminate scale effects.
Task C consists of two tangent circles of different radii, aim- ing to capture the relationship between shape/outlook and size as illustrated in non-spatie non-spatial rule. The coarse- coarse-grained rule simply requires distinct radii for the two circles, i.e., rn # r2, while the fine-graime rule specifies a precise ratio between the radii, requiring r2- = V2r
Task D simplifies the non-spatial rule from scenario Size/Region- Color in Figure 1, where, in candle flame gen- crations, colors transition from blue near the wick to yellow at the outer regions. We construct two such squares, with smaller squares positioned in the upper half and larger ones in the lower half of the image. The coarse- coarse-grained rule
4

---

Can Diffussin Models Learn Hidden Inter-Feature Rules Behind Images?
Groad Tratt
Groad Truffc y
Greadd Trui 2x
Groudd y-
Elivalions y-1
O R
Estreuties 500 -0 00,
R
02-
Emverte LAIs+ -0.0 8
1.10
Esi 50lelley--
5tx do, R7
00
0.31
022
622
000
0.5
12 625
61 0.0 030
(a) Task A
(b) Task B
(c) Task C
(d) Task D
Figure 3. Synthetic training data satisfies fine-grained rules. To validate the evaluation method, extract relevant features from the synthetic training data and check if they meet expectations, focusing on generations within the interval [2.5%, 97.5%] for stability. The closely matching Estimation and Ground Truth lines, along with It value near 1, demonstrate effectiveness of the evaluation method.
Seaud Truby y Est rellele you Kic OL 230
Greand Trafiyy
Greadd Tryth 2Z Caripotirsy I.Ex- 67-
Brcand Tasley
cm
Ectrutton Vou SS.
Edtrgghass y 12hx RIS7, RI U22
010
011
0.00
61/5
430
206
2.5 1.00
6200
E00
0008
0305
1.00 0120 1300 800 200
204 0.0
03 220 09
0.0
0.0 012
1,/12
6,/12
(a) Task A
(b) Task B
(c) Task C
(d) Task D
Figure 4. Generated data does not satisfy fine-grained rules. Considering generated samples within the [2.5%, 97.5%) range, we
extract focused features and check if they meet fine-graincd rules. The Estimation line, far from the Ground Truth line, and an R value less than reveal DMs' failure in learning fine-grained rules. Appendix E.2 shows generated images that violate the fine fine-grained rules.
With these features, can verify whether generated im- ages satisfy predefined rules. For example, in Task A, we examine: (1) Coarse Coarse-grained rule: the sun and shadow are on opposite sides of the pole by comparing the relative posi- tions of the sun center, pole center, and shadow endpoint; (2) Fine- Fine-grained rule: validate the precise geometric relation- ship h2 = 1gh1- We extend the same feature extraction approach in Figure 2 to validate inter-feature rules in Tasks B. C, and D. We apply the evaluation method to synthetic training data to validate our approach's 's effectiveness, as shown in Figure 3, which demonstrates a close alignment between the estimation and ground truth across all tasks.
Table 1. DMs satisfy coarse rules. Table shows the invalid ratio is around 20%-10%. And DMs can learn coarse rules with one exception in Task A, which is visualized in Appendix E.
Task
Invalid (%)
Coarse-Grained Violations
A
30.15
B
40.45
C
41.75
0
D
24.90
the sun and shadow on opposite sides of the pole; violating this rule would require generating never-seen mode with both elements the same side.
3.3. Experimental Results
DMs' Failure on Fine-Grained Rules. While following coarse- gaarse-grained rules only requires DMs to avoid unreason- able OOD generations, fine-grained rules are much harder, demanding accurate learning of the in-distribution training data. Figure demonstrates the models' performance across four synthetic tasks, where deviations from the ground truth in linear fitting and the coefficient of determination R² be- low indicate that DMs fail to fully capture the predefined fine-grained rules. Additionally, we observe that DMs strug- gle more with learning non-spatial rules, such as Task C, compared to spatial rules, such as Task A, as evidenced by worse linear fitting and smaller R², This discrepancy likely
For each synthetic task, we generate 2000 samples and report the evaluated results as follows:
DMs' Success on Coarse-Grained Rules. Table demon- strates that DMs rarely generate samples that violate the coarse-grained rules across all tasks. This observation aligns with expectations: generating samples that violate coarse- grained rules requires DMs to generate out of the (training) distribution (OOD) an extrapolation challenge for DMs observed in prior work (Okawa et al., 2024; Kang et al., 2024). Task A, for example, all training samples place
5

---

Can Diffusson Models Learn Hidden Inter-Feature Rules Behind Images?
dataset, which presents objects and their reflections, where rules connect features such as color, size, and shape. We find that DDPM captures coarse rules (e.g., matching col- ors between objects and reflections) but struggles with fine ones, often producing mismatched shapes. We also con- struct the Cifar-MNIST dataset, which pairs specific CIFAR and MNIST classes (e.g.. Cats/Dogs with 0/1). We observe that DDPM satisfies coarse rules (e.g., consistently gen- erating two digits and two objects), but only 20% of the generations adhere to fine-grained rules that require specific class pairings. These observations on real-world datasets validate the conclusions drawn from synthetic tasks: DM fail to capture fine- tne-grained relationships between features. Detailed experimental results are provided in Appendix E.4,
Memorized Urmonarized
Generation Namber
Number
b
6 3
Task
Threshold
(a) Rule-conforming genera- (b) Memorization with differ- tions acTross four tasks.
ent thresholds in Task A.
Figure 5. DMs generate rule-conforming samples. Define Rule- conforming generations have ratios (e.g.- 15kg +0.01 of true ratio (1 in Task A). Figure 5(a) shows DDPM's ability to generate rule-conforming samples across tasks. Fig- ure 5(b) indicates that nearest neighbor distances between 10 rule- conforming samples in Task A and training data are large (> 0.3), suggesting novel generation rather than memorization.
1zh1 in Task A) within
4. DMs' Failure from a Theoretical Perspective
This section provides theoretical explanations for our ob- served phenomenon DMs inability to effectively learn precise rules. Our analysis reveals that without prior knowl- edge on the hidden rules, DMs trained by minimizing the DDPM loss (Ho al., 2020) exhibit constant error in rule conformity, indicating that they cannot accurately learn the ground-truth rule.
arises from the fact that non-spatial rules are more implicit and lack explicit cues, such as object positions and lengths, which are readily available in spatial relationships. More experiments for various settings (c.g., other backbone mod- els) are deferred to Appendix E.3, which shows consistent empirical observations that DMs can capture coarse-graine graiied rules but struggle to master fine-grained ones.
We consider the following multi-pated data setup, which has been widely employed for theoretical analysis of classi- fication (Allen-Zhu & Li, 2020; Cao et al., 2022; Zou et al., 2023; Lu et al., 2024), and recently for diffusion models (Han et al., 2024a).
Despite Instabilities, DMs Can Generate Fine-Grained Samples. While fine-grained rule experiments show DMs generally struggle to exactly satisfy underlying rules, we observe that they can occasionally generate rule-conforming samples in Figure 5(a), albeit with instability. For exam- ple, in Task A, there are 10 generated samples that (almost) satisfy the fine-gran ine-grained rule, i.e., (ah1 determine whether these 10 ideal samples originate from DDPM's generation or are merely training data replicas (Somepalli et al., 2023a;b; Wang et al., 2024b), we ana- lyze their memorization behaviors. For Task A, we repre- sent each sample with a 13D vector capturing key features (11, 12, f1, h2) and encoding RGB colors of sun, pole, and shadow. We then compute Euclidean distances to their nearest neighbors, considering samples as replicas if the distance is below given threshold. Figure 5(b) shows rule- conforming generations are not mere duplicates, achieving 100% memorization at a large threshold (0.3). Appendix E.2 shows 10 ideal samples and their nearest neighbors, high- lighting differences. This suggests that, although unstable, DMs can generate rule-conforming samples. Inspired by this, Section 5 presents a mitigation strategy with additional guidance to improve generation consistency.
Definition 4.1 (Data distribution with Inter-Feature Rules). Let u E Ø2 be two orthogonal feature vectors with unit norm, i.e., ||u|| = |v| = and (u, v) = 0. Let Ç be a random variable with its distribution Dç supporting on a bounded domain [cç ae] for some constants 0 < Cg < cc < oo. Each image data consists of multiple patches
ti E [0.99, 1.01]. To
x x=xx ()T (2) (2T :..., !PITT
where x()) = (u. (()xx(2)) (L-))v,
and x(),, x(2) are independent with the remaining patches.
Definition 4.1 specifies inter-feature rule on the first two patches of the data, requiring that the projection of the first two feature patches along respective feature vectors u, v sum up to one, i.e., (x(1), u) + (x(2), v) = 1. Further- more, we show such a rule will further lead to structural constraint the score function. Specifically, let xo [ -)v x(3) x.., (P)T represent an input im- age. For arbitrary noise scedules {a., 3.). x, 1xxo+ represents the noised image at timestep t. We derive the score function along the diffusion path as follows.
More Experiments on Real-world Data. Beyond syn- thetic tasks, we alsp conduct additional experiments on real- world datasets to further demonstrate that DMs can learn coarse rules but struggle with fine-grained ones. Specif- ically, we considers the SynMirror (Dhiman et al., 2024)
Theorem 4.2. The score function is V Vlogpt(x)) =
6

---

Can Difflssin Models Learn Hidden Inter-Feature Rules Behind Images?
Ground Truth: y= - 2.48
R
Ground Truth: y 1122
Ground Truth y 0.79
Ground Truth: y
X 0.56
15-
10
10
10
10
5-
0-
0
-5-
-10
ReIUU
Linear
Quad
Cubic
ReLU
Lincar
Qued
Cubic
ReLU
Linear
Quad
Cabic
Rel.
Lihea
Quid
Gubic
(a) = 0.2
(b) 1
0.4
(c) L == 0.6
(d) == 0.8
Figure 6. Diffusion model exhibits non-vanishing error on synthetic multi-pateh data with norm constraint. We observe for variety of timestep and activation functions (ReLU, linear, quadratic and cubic). a (two-layer) diffusion model cannot learn precisely the hidden norm constraint in Definition 4.1, with both bias and variance error.
V (x. (1) (2)) )T, V lo Ps (x (3) .... X TT, where
(p) It has been shown in (Chen et al., 2022) that minimizing (2) is equivalent to minimizing
(p) + B,E," where x = axx)
V Pl )
the score matching loss, i.e., Ep, || - log |r|. 1|2
1
C2
ED (S):
We next define the rule-conforming error to measure the learning outcome of the hidden rule (*).
-
36
Xx
+
Bi
E Ne (1 5)lv
Definition 4.3 (Rule-conforming error). For the score net-
where Te(5, Xt) = Epg (C.
Nix.; (C),
(4(C) =
work sw of diffusion model with weights w(f let
(n)"
[aaCuT, .a((1 - TT.
1 x 3
(x, = (s (x.)
, u) + (5 (1 (xx) +
B
is clearly noted that the ground truth score (restricted to the first two patches) exhibits the following identity:
at time L for Xx. We say the diffusion model conforms to rule (+) if (x.) = holds for any x,. We define the rule-conforming error as:
be the coefficient along directions
ED, (5((S +
- -()] =
= 1.
(*)
E ExEx,
[( ((0((()))
04 3.
2
Then, we aim investigate whether a score network, trained via DSM objective, accurately conform to such a hidden rule (*). Specifically, we follow (Han et al., 2024a) and consider the following two-layer neural network model: (1)
Then, we consider training 8w by gradient descent over (2) starting from initialization {w (p).0) ing theorem derives a lower bound on the rule-conforming error for the trained score network model.
]rejm] The follow-
Sw (x,)
- (xx)
(P)) (x:) with
Theorem 4.4. Let with r [m]] be stationary point of the DDPM loss (2). Then we can lower bound
m
" (xx)
(xt)
x
(p) T. x (P) (p)
37
[Var]s.e]) 15.e (7 E {[11)]]]
E E E 5.e
where each patch is processed with a separate set of m neu- rons, and o(-) is an (non-constant) polynomial activation function. Such a network mimics the structure of U-Net (Ronneberger et al., 2015) with shared encoder and decoder weights. The network also contains residual connection that aligns with the score function (Theorem 4.2). Similar network design has been considered in (Shah et al., 2023; Han et al., 2024a). We train the score network by minimiz- ing the DSM loss (Ho et al., 2020; Shah et al., 2023) with expectation on the diffusion noise and the input:
+ E s.c (0) Var Varis.e." ((v,) G. c(1i)
where we decompose c.0) = -,,
c,, + c, with (p) be- ing the projection of E onto span(w
wit (p).0 .... w Wnt
(p).0)
Var(ia)() = Var(-IA) is the conditional variance and (()) is a polynomial with coefficients depending on (W u), (2)x v).
Theorem 4.4 immediately suggests non-vanishing rule- conforming error, as long as the polynomial ă is non- constant and dimension d is sufficiently larger than net- work width m to ensure variability in the random noise EI, which independent of and v.
(p)
L(W)) ) = Ec,,x
Σ p=1
((())
(xc
||
(2)
31

---

Can Diffusson Models Learn Hidden Inter-Feature Rules Behind Images?
Remark 4.5 (Extension to network with patch dependence). Although neural network setup is separable in the patches, the analysis can be extended to network handling dependent patches. For example, consider <0-2) (x,) =
rules.
5. Mitigation Strategy with Guided Diffusion
+x,²)(x))
-x, (122 / + (o(WT x x(1.2)), where x! (1.2) denotes the concatenation of two patches, a()) is polynomial activation function and W E R2dxx__ We can still follow Theorem 4.4 to show that (1) the learned W consists only the direc- tions u, v and its initialization, and (2) the network output becomes a polynomial function of (u, ) (v, ) and their cross terms (which do not appear in the patch- case). As x varies, the function output changes accordingly, which results in a non-vanishing rule conforming error that depends the variation of E,.
Motivated by our finding that DMs can produce rule- conforming samples but instability, we mitigate this by a common technique, Guided DDPM, which introduces ad- ditional classifier guidance (Dhariwal & Nichol, 2021) dur- ing sampling. Specifically, train the classifier fe(x, t) through contrastive learning with constructed contrasting data pairs, where positive samples follow fine-grained rules while negative samples violate fine rules while maintaining coarse- grrsse-granedddee compliance. The training objective is
Liotal Lelassitication
A Leontrastive
(3)
We now show that when simplifying the model to linear activation (x) x and single neuron (w!"),, ) the rule- conforming error can be computed as the sum of bias and variance errors, both of them are lower bounded by some constants. Specifically, we decompose
where À is weight parameter, Lelnsification is Cross-Entropy loss and Ceontrastvve is NT-Xent loss (Sohn, 2016). More de- tails on NT-Xent loss are in Appendix H.1. Then, following Dhariwal & Nichol (2021), gradients from fo t) are used to guide sampling toward fine-grained rule compliance.
E = Ex, [x))( - al
Var je'e (xxe)
Additionally, based on constructed contrastive data, we di- rectly train classifier in raw images to determine whether generation satisfies fine ineigrained rules. We filter samples predicted as non-rule-conforming to ensure generation qual- ity. This approach, called Filtered DDPM, which directly provides guidance based on the noise-free pixel space, can be seen as the upper bound for guided diffusion strategies.
B?
Evariance
The following theorem suggests there exist a constant bias and variance error for any stationary point w/.
1 and consider I such that or,, 3r = O(1). We train the network with the gradient descent on DDPM loss (2) from small Gaussian initialization, i.e., W (p).0 NO, oīLa), on = 0(211/2 and d = (11)). Let w (p)* be any stationary poins. Then (2)+
Theorem 4.6. Suppose (Ix) x.
5.1. Experiment Results
Setup. We use a U-Net classifier fe t) with guidance weight 1. Details of the data construction and training process are provided in Appendix H.I.
(w(")r, u), (w/"), .v)="vv)= (1).
There exists constants Co, Ci > 0 (depending on E[{], [[["), on, Be) such that Epias = Co, Evariance = C1.
Results. In addition to R2, inspired by the theorical analysis in Section 4, we introduce Error, metric capturing how well DMs learn hidden rules from variance and bias. Given the Ground Truth line = B1 and the Estimation line ÿ = 81 + 3o in Figure 3 and 4, Error is defined as:
Theorem 4.6 shows that (1) all data features u and v can be discovered, which is consistent with the results in Han et al. (2024a) and verifies the ability of DMs to conform to coarse rules in the data, i.e., the existence of the key features. (2) It also verifies that DMs fail learn the fine-grad fine-grained hidden rule when constraint or guidance is imposed over the training of DMs. Both of these two results are consistent with our empirical findings in Section 3.
Error := 1 - 1-131-11 + 1801
/ Var(ū y
(4)
Bias Frror
Variance Error
We measure the bias error [E[y 9]] with the deviation in
the estimated coefficients 31, So- The variance error in (4)
Empirical verification. We further train score networks based on the theoretical setup and evaluate the rule- conforming error in Figure 6, where we consider four differ- ent activation functions (see Appendix G for details). We calculate the error of DMs in learning the hidden rule (*) and plot the distribution of thx (xx) over 5000 sampled x,. It is clear that for all activation functions, the rule-conforming error is significant, verifying our theoretical results and sug- gesting the inability of DMs to precisely learn the hidden
corresponds to the square root of Evariance in Section 4.
Table 2 presents results, Error and R², before (DDPM) and after applying classifier guidance (Guided DDPM), along with DDPM filtered by pixel-space classifier (Filtered DDPM). Both Guided DDPM and Filtered DDPM outper- form the baseline DDPM across all tasks, showing reduced Error and improved R², with Filtered DDPM achieving the best performance on most tasks.
8

---

Can Difflusin Models Learn Hidden Inter-Feature Rules Behind Images?
rules remains an open question. Additionally, incorporating inter-feature rules into the tokenizer- e.g., through masking strategies (He et al., 2021; Assran et al., 2023; Deng et al., 2025) may help DMs better capture semantic information related to rules. We leave these explorations to future work.
Table Comparison between DDPM, Guided DDPM (Guid- ance), and Filtered DDPM (Filtering): Additional guidance and filtering improve generation with lower Error and higher R².
Brorr :
nP1
Task
DDPM
Guidance
Filtering
DDPM
Guidance
Filtering
A
0,25
0.21
0.17
0.85
0.90
0.90
B
0.11
0.10
0.05
0.83
0.85
0.86
Acknowledgment
C
0.41
0.26
0.25
0.57
0.67
0.64
D
0.46
0.43
0.39
0.79
0.84
0.85
The authors would like to thank the anonymous review- ers and area chair for their helpful comments. Difan Zou and Yujin Han acknowledge the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A1515012444, and the central fund from HKU.
5.2. Discussions on the Limitation of Guided Diffusion
While guided and filtered diffusion provides some mitiga- tion for rule learning, we acknowledge that this improve- ment is limited. The limited improvement stems from the inherent nature of our problem: unlike conventional clas- sification tasks, the fine-gres ine-grained rules that differentiate our contrastive samples exhibit subtle signals, making effective classifier training particularly challenging. In Appendix H. provide additional experimental evidence that, even such simple synthetic tasks, the classification accuracy the test set remains between 60% and 80%, supporting the difficulty of precise classification in contrastive data.
Impact Statement
This paper aims to advance the fundamental understand- ing of machine learning, with a focus diffusion basedd image generation. We aim to enhance the interpretability of these models, shedding light on how they manage inter- feature relationships during generation. By deepening our understanding, we hope to drive community attention to- ward developing image generative methods that align better with real-world needs, such as adherence physical laws. Acknowledging the societal impact of image synthesis, in- cluding its creative potential and risks like image forgery, we strive to promote principled methods and responsible use through our study of these popular diffusion techniques.
Additionally, the effectiveness of this strategy relies on prior knowledge of fine- fine-grained rules. In real-world scenarios, fine-grained rules are often difficult to accurately define and detect, making the construction of contrastive data im- possible. We leave the solution to DMs' inability to learn fine-grained rules in real-world scenarios for future work.
References
6. Conclusion and Future Work
Aithal, S. K., Maini, P., Lipton, Z. C., and Kolter, J. Z. Understanding hallucinations in diffusion models through mode interpolation. arXiv preprint arXiv:2406.09335 09358, 2024.
This study evaluates DMs from the perspective of inter- feature rule learning, revealing through carefully designed synthetic experiments that DMs can capture coarse rules but struggle with fine-grained ones. Theoretical analysis attributes this limitation to fundamental inconsistency in DMs' training objective with the goal of rule alignment. We further explore some common techniques, such as guided diffusion, to enhance fine- -raiire rule learning, but ob- serve limited success. Our in-depth findings underscore the inherent difficulty of capturing subtle fine-gran fine-grained rules, providing valuable insights for future advancements.
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self oiff-istilllatinn in deep learning. arXiv preprint 20.0:20/2.0088/6, 2020.
Anonymous. Towards understanding text hallucination of diffusion models via local generation bias. In The Thir- teenth International Conference Learning Represen- tations, 2025. URL https: openreview nnt// forum?id=skK10x..
One potential direction to address the limitation of DMs in learning fine-grained rules is to introduce stronger reward signals, such as those from human feedback (Ouyang et al., 2022; Xu et al., 2023) or specialized reward models (Fan et al., 2023; Wu et al., 2024), to better guide DMs during generation. However, unlike typical test- time aligmment tasks, such as focusing on changing image style or aligning CooptssWWallace et al., 2023; Yang et al., 2024c;a; Yuan et al., 2024a; Kim et al., 2025), fine-grained rule learning is more challenging due to its subtle signal. How to obtain a powerful reward model that accurately reflects various
Assran, M., Duval, Q.. Misra, I., Bojanowski. P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with a joint-embedding predic- tive architecture, 2023. URL abs / 2201 08243.
Benton, J., Bortoli, V., Doucet, A., and Deligiannidis, G. Nearly d-linear convergence bounds for diffusion models via stochastic localization. 2024.

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Betker, J., Goh, G., Jing, L.,, Brooks, T., Wang, Li, L., Ouyang, I.,, Zhuang, J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Sci- ence. https://cdn. openai. con/papers/dall-e33 pdf, 2(3): 8, 2023.
Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models, 2023, URL htt.htss://aaatt org/ abs/ 2200 16381.
Feng, W. He. X., Fu, T.-J., Jampani, V., Akula, A., Narayana. P., Basu, S., Wang, X. E., and Wang, W. Y. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022.
Borji, A. Qualitative failures of image generation models and their application detecting deepfakes. Image and Vision Computing, 137:10477) 2023.
Cao, Y., Chen, Z., Belkin, M., and Gu, Q. Benign overfitting in two-layer convolutional neural networks. Advances in Neural Information Processing Systems, 35:25237- 25250, 2022.
Fu, H. and Cheng, G. Enhancing semantic mapping in text- to-image diffusion via gather-and-bind Computers & Gruphics, 125: 255104118 2024.
Gokhale, T., Palangi, H., Nushi, B., Vineet, V., Horvitz, E., Kamar, E., Baral, C., and Yang, Y. Benchmarking spatial relationships -xx-tto-mmage generation. arXiv preprint arXiv: 22. 2022.
Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L.,, and Cohen-Or, D. Attend-and-excite Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4):1-10, 2023.
Gong, S., Li, M., Feng. J., Wu, Z., and Kong, L. Diffuseq: Sequence sequence text generation with diffusion mod- els. arXiv preprint arXiv:2210.08933, 2022.
Chen, H., Lee. H., and Lu, J. Improved analysis of score- based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Con- ference on Machine Learning, pp. 4735 4763. PMLR, 2023.
Han, A., Huang, W. Cao, Y., and Zou, D. On the feature learning in diffusion models. arXiv preprint arXiv: 24Xiv:2412.01021, 2024a.
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209. /215, 2022.
Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng. B., and Liu, X. Infinity: Scaling bitwise autoregres- sive modeling for high-resolution image synthesis, 2024b. URL https:tpps............. / a //axxiivvv./ www..iiv.o...ooooo.oooooo.o woroorrg/ao abs////////// / 04431.
Chen, Y., Wang, O., Zhang, R., Shechtman, E., Wang, X., and Gharbi, M. Image neural field diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8007-8017, 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners, 2021. URL https: //axxiii//rxivv org/ aba 2h1b. 113777711..677... 06377.
Deng, C., Feng, B. Y., Garraffo, C., Garbarz, A., Walters, R., Freeman, W. T., Guibas, L., and He, K. Denoising hamiltonian network for physical reasoning, 2025. URL https: wrxii.ivv..///vv www.wwwwwwwwwawaa....e....ooeeee 22t.//553..0075
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 2022.
Deschenaux, J., Krawezuk, I., Chrysos, G., and Cevher, V. Going beyond compositions, ddpms can produce zero- shot interpolations. In Forty-first International Confer- ence on Machine Learning, 2024.
Huang, X., Shao, R., Zhang, Q., Zhang, H., Feng, Y., Liu, Y., and Wang, Q. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 4568-4577, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Dhiman, A., Shah, M., Parihar, R., Bhalgat, Y., Boregowda, L. R., and Babu, R. V. Reflecting reality: Enabling diffu- sion models to produce faithful mirror reflections. arXiv preprint ar 2409 14677, 2024.
Kang. B., Yue, Y., Lu,, R., Lin, Z., Zhao, Y., Wang, K., Huang, G., and Feng, How far is video generation from world model: A physical law perspective. arXiv preprint A.Vv:24/1.02340 2024.
Fan, Y. Watkins, O., Du, Y. Liu, H., Ryu, M., Boutilier, C., Abbeel, P. Ghavamzadeh, M., Lee, K., and Lee, K.
10

---

Can Diffusson Models Learn Hidden Inter-Feature Rules Behind Images?
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 2022.
Mahajan, S., Rahman, T., Yi, M., and Sigal, L.. Prompting hard or hardly prompting: Prompt inversion for text-to- image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6808-6817, 2024.
Kim, S., Kim, M., and Park, D. Test-time alignment of diffusion models without reward over-optimizationn 2025.
Marioriyad, A., Rezaei, P., Baghshah, M. S., and Rohban, M. H. Diffusion beats autoregressive: An evaluation of compositional generation in text-to-image models. arXiv preprint arXiv:2410.22775 2024.
URL wwtptttpp///... / //aaaxixvv.rr.e wwwp.w.org/ab///511.00
Labs, B. F. Flux. https://github. com/ black-forest- www.-o-rrrtttseeeee flux, 2023.
McInnes, L.., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.0000 2018.
Lemercier, J.-M., Richter, J., Welker, S., Moliner, E., Välimäki, V., and Gerkmann, T. Diffusion models for au- dio restoration. arXiv preprint 2wxiv:2402.09922 2024.
Niu, A., Pham, T. X., Zhang, K., Sun, J., Zhu, Y., Yan, Q. Kweon, I. S., and Zhang, Y. Acdmsr: Accelerated condi- tional diffusion models for single image super-resolution. IEEE Transactions on Broadcasting, 2024.
Li, H., Zou, Y., Wang, Y., Majumder, O., Xie, Y., Manmatha, R., Swaminathan, A., Tu, Z., Ermon, S., and Soatto, S. On the scalability of diffusion based text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9400- 9409, 2024a.
Okawa, M., Lubana, E. S., Dick, R., and Tanaka, H. Com- positional abilities emerge multiplicatively: Exploring diffusion models a synthetic task. Advances in Neural Information Processing Systems, 36, 2024.
Li, X., Lu, J., Han, K., and Prisacariu, V. A. Sd4match: Learning to prompt stable diffusion model for semantic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27558- 27568, 2024b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022. URL https: //arxiv. abs 220022203222 02155.
Lim, Y. and Shim, H. Addressing image hallucination in text-to-image generation through factual image retrieval. arXiv preprint arXiv:2407.10683, 2024.
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint 1.Xiv:2301.122000 2023a.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. +119-4205 2023.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable dif- fusion models. In European Conference on Computer Vision, pp. 423 439. Springer, 2022.
Pelykh, A., Sincan, O. M., and Bowden, R. Giving a hand to diffusion models: two-stage approach to improv- ing conditional human image generation. arXiv preprint arXiv:2403.Im 10731, 2024.
Liu, Q.. Kortylewsk,, A., Bai, Y., Bai, S., and Yuille, A. Dis- covering failure modes of text- eext-guided diffusion models via adversarial search. arXiv preprint arXiv:2306.00977 00974, 2023b.
Podell, D.,. English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 1005 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Lu, M., Wu. B., Yang, X., and Zou, D. Benign oscillation of stochastic gradient descent with large learning rate. In The Twelfth International Conference on Learning Representations, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N.M. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint orXiv:2401. 008400, 2024.

---

Can Difflssin Models Learn Hidden Inter-Feature Rules Behind Images?
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted Intervention-MICCA. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
N. Diffusion model alignment using direct preference op- timization, 2023. URL htt.ht:ss://axxive
abs/
Wang. B. An analytical theory of power law spectral bias in the learning dynamics of diffusion models. arXiv preprint arXiv:2503.0322 2025.
Samuel, D., Ben-Ari, R., Raviv, S., Darshan, N., and Chechik, G. Generating images of rare concepts us- ing pre-trained diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4695-4703, 2024.
Wang. B.. Shang, J., and Sompolinsky, H. Do diffusion models generalize on abstract rules for reasoning? 2024a.
Wang. H., Han, Y., and Zou, D. On the discrepancy and connection between memorization and generation in dif- fusion models. In ICML 2024 Workshop on Foundation Models in the Wild, 2024b.
Shah, K., Chen, S., and Klivans, A. Learning mixtures of gaussians using the ddpm objective. Advances in Neural Information Processing Systems, 11119966199444 2023.
Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024.
Sohn, K. Improved deep metric learning with multi-clase n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and digital forgery? investigat-
Wu, X., Hao, Y., Zhang. M., Sun, K., Huang, Z., Song, G., Liu, Y., and Li, H. Deep reward supervisions for tuning text-to-image diffusion models, 2024. URL https: / arxiv. org. abs 2405. 00760.
Goldstein, T. Diffusion ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023a.
Xu, J. Liu, X., Wu, Y., Tong, Y., Li, Q. Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-imaae generation, 2023. URL https: /ttp//aavvvv/////////////.//a.aa org/abs/ 2304. 05977.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Understanding and mitigating copying in diffusion models. Advances in Neural Information Pro- cessing Systems, 36:47783-47803 2023b.
Xu, M., Geffner, T., Kreis, K., Nie, W., Xu, Y., Leskovec, J., Ermon, S., and Vahdat, A. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21357, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. arXiv preprini arXiv:2011.13445 2020.
Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q. Shen, W., Zhu, X., and Li, X. Using human feedback to fine-tune diffusion models without any reward model, 2024a. URL. https: //t///axvvvv/// oog.oogg///aooo..ooo abs / 1311113311.1....1111111111111111
Tang, 7.,, Wang, P., Zhou, K., Li, J., Cao, Z.,, and Zhang, M. Can diffusion model achieve better performance in text generation? bridging the gap between training and inference! arXiv preprint wrXiv:2305.04460 2023.
Yang, M., Zhang. C.,, Xu, Y., Xu, Z., Wang. H., Raj, B., and Yu, D. Usee: Unified speech enhancement and editing with conditional diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7125-7129. IEEE, 2024b.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024a. URL https: //rxivv. org/abs/ 2404 02905.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 2024b.
Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-to-image diffusion with preference, 2024c. URL https: / /tl:://aaxiiie arxiv. orgg hbss//////// 2402. 08265.
Vahdat, A., Kreis, K., and Kautz, Score-based generative modeling latent space. Advances in neural information processing systems, 34:11287-11302 2021.
Yao, J. and Wang, X. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.0144 2025.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik,
Ye, Y., Hebbar, P., Gupta, A., and Tulsiani, S. Diffusion- guided reconstruction everyday hand-object interaction
12

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
clips. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 19717-19728, 2023.
Yu, S., Nie, W., Huang, D.-A., B., Shin, J., and Anand- kumar, A. Efficient video diffusion models via content- frame motion-latent decomposition. arXiv preprint arXiv:2403. 14148, 2024.
Yuan, H., Chen, Z., Ji, K., and Gu, Q. Self-play fine- tuning of diffusion models for text-to-image generation, 2024a.
URL https: /wwpavaaiii/./////a//ee.eawwta org/ 245/2 10210.
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanie, S., and Ni, D. Instructvideo: instructing video diffusion models with human feedback. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 6463-6474, 2024b.
Zhang, Y., Wang, Z., Li, X., Yuan, Y., Zhang, C., Sun, X., Zhong, Z., and Wang, J. Diffbody: Human body restoration by imagining with generative diffusion prior. arXiv preprint 2444:2240.03664,, 2024.
Zou, D., Cao, Y., Li, Y., and Gu, Q. The benefits of mixup for feature learning. In International Conference on Ma- chine Learning, pp. 43423-43479, PMLR, 2023.
13

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
A. Low FID and Worse Inter-Feature Learning: A Gaussian Mixture Case
In this section, we provide toy example based on the Gaussian Mixture Distribution to explain how low FID and incorrect inter-feature relationships can coexist. This supports the point that even though DMs may perform excellenty on classical metries such as FID, this does not necessarily mean they can perfectly learn the hidden inter-feature rules.
Consider 2-dimensional population, i.e., the true distribution p(r, y). which is Gaussian Mixture Model (GMM) with two components as:
.V (H- :o 9) + - N (E)
: 9)
p(z,9) = F(u,, Ex) - -p)=
(5)
where we can have
+ [ = 0
M = 2
and the covaraince matirx
2
D
Σ
Wi
[ + (14: - 11y - -
i=1
(l.
1 + [ : + 0.5
- - -
1 2
0.5
We assume the estimated data distribution learned by DMs is joint Gaussian distribution:
g(t, 9) (H,, Eg) = - (
2 2)
(6)
With means and covariance matrices of true distributon p and estimated distribution q are identical, that is M1 14g [0,0]' and Σp
2 1]
Eg
we easily have the FID between p(x, y) and p(.: f)) computed as:
1 2
FID = ||,, -Myl1 + Tr
Σ, +
Eg - 2(,,,,))
1/2)
2 1
1
= |10 012 Tr
+
2
1
2
2
0
Although the FID is small (i.e., 0), the inter-feature relationships between x and in true and estimated distribution are fundamentally different, In the true distribution, x and y are independent within each Gaussian component but exhibit dependence in the overall distribution due the mixture of components. the estimated distribution q(t ÿ), & and ğ are dependent with Cov(a: (2. y) 1. Therefore, low FID does not imply a correct recovery of the inter-feature rules.
B. On the Rule Conforming Error with linear settings
2-
We remark a recent work (Wang, 2025) analyzes the case when rule- conforming data lies on low-dimensional linear subspace and demonstrates a linear score network can precisely learn the distribution asymptotically. This can be also reflected in our Theorem 4.4, where the polynomial functions (1) (.), ((2) (.) reduce to constant functions, due the polynomial degree I as we consider linear model. This ultimately leads to zero lower bound for the rule-conforming error.
-1-
Ground Truth: y = 0.56
Linear
2-MLP
2-MLP 3-MLD (quad)
(relu)
(relu)
Nevertheless, our theoretical argument mainly focuses on the general case, where we consider the settings that the underlying rule unknown to the learner, and the model class does not align with the true structure of the
Figure 8. Linear network exhibits lower rule- conforming error than more complex networks under the linear rule.
14

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Table Real-World Inter-Feature Rules. For each scenario containing inter-feature rules, Table provides detailed prompts and annotates the existing inter-feature relationships. By comparing the genuine inter-feature relationships with those in generated images, we can evaluate DMs' ability to learn inter-feature relationships.
[Spatial Rule] (a) Light shadow:
Prompt 1: A desert scene with ma jestic palace under a bright sun. Sun position affects palace's shadow direction.
Inter-Feature Rule 1:
Prompt 2: The moonl i ght casts a clear shadow of tall tree onto the ground.
Inter-Feature Rule 2: Moon position affects tree's shadow direction
ISpatial Rule] (b) Reffection/Refraction
Prompt 1: A plush lion toy in front of the mirror. Its front side is facing the camera. There is its reflection in the mirror.
inter-Feature Rule 1: The lion toy's orientation relative to the mirror determines reflection's orientation. transparent. glass bottle partially submerged in calm, clear pool of water. The upper half the bottle extends above the water's surface and the lower half of the bottle is submerged.
Prompt
Inter-Feature Rule 2: The water surface's position dictates the bottle's shape distortion.
ISpatial Rule] (c) Semantics
Prompt 1: A field sunflowers under a clear blue sky with the sun shining
brightly above.
Inter-Feature Rule I: Sun direction dictates sunflower orientation.
Prompt 2: A paintbrush fully loaded with paint, making stroke blank white canvas.
Inter-Feature Rule 2: Brush tip color matches canvas paint.
[Non-Spatial Rule] (d) Size TTettue
Prompt I: The cross- -eooss-ecctio of sturdy tree, covered with annual rings. Inter-Feature Rule 1: The diameter of tree is related to its growth rings.
Prompt 2: A nautilus fossil, showing its intricate spiral shell structure with visible growth chambers.
Inter-Feature Rule 2: Nautilus fossil size correlates with spiral patterns.
[Non-Spatial Rule] (e) Size/Region- Color
Prompt I: An artistic representation showing the expanded star phase and cooling star phase of the same star.
Inter-Feature Rule 1: Celestial body size and color should align, exemplified by red giants and white dwarfs. Prompt 2:
A burning red candle a dark with the flame, which is vibrant, dynamic, and glowing intensely against the darkness. Inter-Feature Rule 2: Candle flame color varies with distance from the wick.
[Non-Spatial Rule] (f) Color Color
Prompt I: Two Eclectus parrots, showcasing the striking sexual dimorphism of the species.
Inter-Feature Rule 1: Eclectus parrots' body and beak colors match green and yellow for males, red and black for females.
A male Poecilia reticulata and female Poecilia reticulata are swimming gracefully in a clear, freshwater aquarium, showcasing the striking sexual dimorphiss sm of the species Inter-Feature Rule 2: Guppies' body and tail colors match-male are equally colorful in both.
Prompt 2:
15

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Spatial
Non-Spatial
SDXL
Flux.1.1 Ultra
DALL-
0.000
Infinity
Rule Type:
Light shadow (b) Reflection/Refraction (c) Semontics
(d) Size -Texture
(e) Size/Region- Color
(f) Color Color
Figure 7. Evaluating More Mainstream DMs Real-Worle Inter-Feature Rules. We evaluate mainstream DMs scenarios
with interrfeature rules, with 5 random generations and manual selection of unrcasonable samples. Despite their success in metries like
FID, none of these DMs achieve complete correctness in cases involving inter-feature relationships.
rule. In such scenarios, although the network may be able to recover the data distribution, while the hidden rule may still not be well captured. This will still lead non-zero rule-conforming error.
To further support this claim empirically, conduct an experiment comparing rule-conforming errors across different model classes in synthetic linear data setup, where two patches are x(1) Çu, x(2) -Vv, with Ç ~ NN0,, 1). As shown in the Figure 8, linear model achieves significantly lower rule-conforming error compared to more complex, nonlinear models (2-layer, 3-layer MI.ss with ReLU or quadratic activation, operated on all patches jointly). This aligns with our findings that without exact structural alignment between the model/objective and the rule, the small rule-conforming error cannot be guaranteed.
C. Details and More Examples on Real-Wold Hidden Inter-Feature Rules
Table 3 provides a detailed description of the prompts for each case in Figure 1 and Figure 7, including scenarios with inter-feature rules and the corresponding rules themselves. We also consider more DMs such as SDXL (Podell et al., 2023), Flux. 1.1 Ultra (Labs, 2023), DALL-E 36 (Betker et al., 2023), and VAR-based (Tian et al., 2024a) text-to-image model Infinity" (Han et al., 2024b) the evaluation. By comparing these rules, we observe that most mainstream DMs fail in some or all scenarios. For instance, in the Reflection/Refraction scenario, none of the DMs successfully generate plausible images: the reflected toy in the mirror faces the camera just like the real one, and the submerged bottle shows no refraction. Our evaluation covers both classic latent diffusion models (e.g., SD-3.5 Large) and the latest scale prediction based diffusion models (e.g., Infinity Surprisingly, none of them can perfectly handle these inter-feature relationships, highlighting the widespread limitation of DMs in this regard.
D. Details and More Example on Synthetic Tasks
This section presents supplementary details and examples regarding our synthetic datasets.
Task A generates synthetic images featuring simple outdoor composed of vertical pole, sun, and their corre- sponding shadow. The height of the pole is randomly selected within the range of [6.4, 12.8) pixels, which corresponds to (20%, 40%] of the total image size (32 x 32 pixels). The sun's horizontal position is sampled from two predefined distance intervals: far distances (0 pixels or 26 32 pixels) and near distances (10 - 16 pixels or 16 22 pixels), ensuring a
"htpss//gtthhh.com//// http://wonnatttiiiiioooiiiiiiiia
16

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
le 0. Au a Ratie 100
1 oa hus 223 Ranio 1.00
A: 02 by Rarta: 100
t: 8.38 hy: 0.34 Rohro: 1.00
6:0010 0.30 Rahio 1000
8.40 by 8 Ratie:
1., Pario: 1000
As a 6:00.6 Ratre: L00
Task A
Ii 6.06 hu 0.42. t 04 0.0s. Reniec.92
0:3055 nz 9.3 6. 48,8 Ny 0.19 Rohee1004
I: 0.7, no 941, 0. 03, hus 6:16, Ro
0 80, hy: 0.25 b. 057, hy 6.09, Butio 1.9
ii 0/6 hu 0.48 L. 0.44, iy- 019 Butio 1.08
to 0.0777 nu 0,47 tu 0.72, ny. o05. Ratio
N: 01001,, ny 0.27 to 0.5554, fy 006,
1: 0:998, 0.5. 1. 0.61, hy 0.2,
Rahiil
Rohee
Task B
12
h2
h2
h1
h2
h2
h1
h1
0:000, 03. Rchicc,48
ny C.R 0.R,
n: 0, ry: 0.20 Rutio 2.39
ny 6.r,, 0.R. Aatieil.33
0.12, ry: D1B, Rehiss1.4
itt 0.09 ey 0.13,
ny 0.12 11..ty: 2.1.
hy 0.0.7 8.00
Budoocl.66
Runkor142
Rutio:143
Task C
rt
ny 20. ru 3,35, E.0
0.1.9,9 0.88.
0.77 n::04) Berlot 49
n: 921 0.31, Raho
0.16 2z 0.23 Ratho 250
0.83, 0.30 RahiilSpp 11
in QJI C2T, Borlo
0.8. 0.27 Rahe
Rahai150
Task D
12
Figure 9. Synthetic Data Examples. We present synthetic samples in four synthetic tasks, with annotations of features of interest and Ratio calculations. The target Ratios for Tasks A, B. C, and D are 1, 1, v2, and 1.5, respectively.
varied distribution of locations. The shadow length is computed using the formula:
SeadowJJengte pole height x |sun distance]
(8)
sun_height pole_height
where the sun height is determined as twice the pole height, clipped within [9.6, 25.6] pixels (30%-80% of the image size). Colors for the sun, pole, and shadow are randomly selected from predefined HSV Satur-atttttion Vaauu)) ranges: Sun color (yellowish tones) has hue in [0, 30). saturation in [100, 255]. and value in [200, 255). Pole color (blue-green tones) has hue in [90, 150], saturation in [100, 255]. and value in [100, 255]. Shadow color (dark tones like black, brown, gray) has hue in [0, 180], saturation in [0, 50), and value in [50, 150).
Task B generates synthetic images containing two rectangular objects placed within a 32 x 32 pixel space. The first rectangle's position and size are determined as follows: its leftmost position is chosen randomly from the range [0, 9.6] pixels (30% the image width), and its height l2 chosen randomly from [6.4, 19.2] pixels (20% to 60% the image height). The color of the first rectangle is randomly selected from a yellowish hue range with hue [0, 30), saturation (100, 255), and value [200, 255) in HSV space. The second rectangle's position is determined by h1, which is chosen randomly within range dependent
41. Specifically, h1 is sampled from the range [41 + 6.4, 25.6] pixels (ensuring h > 4). The height of the second rectangle h2 is calculated based on the first rectangle's 's height l2, ensuring the relation 4h1 = h222. The color of the second rectangle is chosen randomly from a bluc-green hue range with hue [90, 150]. saturation [100, 255]. and value [100, 255] in HSV space.
Task C generaats images containing two circles: one large and one small. The large circle's diameter randomly chosen between 10% and 30% of the image size, and the small circle's diameter is determined be v2 times the diameter of the large circle. The colors of the circles randomly selected from predefined color ranges in the HSV color space. Specifically, the large circle is assigned a color from the blue-green hue range, with hue values between 90 and 150, saturation between 100 and 255, and brightness between 100 and 255. The small circle is assigned color from the yellowish hue range, with hue values between 0 and 30, saturation between 100 and 255, and brightness between 200 and 255. The circles are randomly positioned such that they are adjacent to each aneereetherr on the left, right, top, or bottom of the large circle.
Task D generates images containing two squares: one smaller and one larger. The small square's size randomly chosen to
17

---

Can Difflssin Models Learn Hidden Inter-Feature Rules Behind Images?
be between 30% and 70% of the top half of the image's size. The larger square's 's size is then set to be 1.5 times the size of the small square. The color of the small square is randomly selected from yellowish hue range, with hue values between O and 30, saturation between 100 and 255, and brightness between 200 and 255. The color of the large square is randomly chosen from a blue-gres green hue range, with hue values between 90 and 150, saturation between 100 and 255, and brightness between 100 and 255. The position of the squares determined within specific regions of the image. The top half and the bottom half of the image are divided into distinct regions, with the small square being placed in the top half and the large square in the bottom half. The exact position of each square randomly chosen within its respective region, while ensuring that the squares do not exceed the image's boundaries. Both squares are positioned such that they do not overlap with each other and remain entirely within the image frame.
E. More Synthetic Tasks Setup and Results
E.1. More Details of Experimental Setup
We use the U-Net architecture as the denoising network, consisting of several down-sampling and up-sampling blocks, each with two convolutional layers followed by ReLU activation. Each down-sampling block incorporates a Self- Self-Attention mechanism and skip connections to preserve fine details. Pooling layers are used to reduce spatial dimensions and capture abstract features. A final 1 convolution layer produces the denoised output image. We use AdamW (Loshchilov, 2017) as the optimizer with learning rate of 3e 4. The noisy steps set to T 1000, with linear noise schedule ranging from le 4 to 2e- 2. For Tasks A, B, C, and D, the sample sizes are 4000, 2000, 2000, and 2000, respectively, and the input data size is (3, 32, 32). The training is performed on a single NVIDIA A800 GPU for 400, 800, 1600, and 1000 epochs, respectively.
E.2. More Results of Synthetic Tasks
This section provides additional details to complement the experimental results in Section 3.3. Notably, to ensure more accurate quality assessment of generated images, we upscale the 32 x 32 images to 128 x 128 during evaluation. This allows the training data to precisely exhibit the expected rule patterns, thereby enabling more reliable evaluation of the generated samples.
Generations that Violate Coarse-Grained Rules. Table 1 illustrates the DDPM's ability learn coarse-grained rules. We observe that in all four synthetic tasks, the number of samples violating the coarse-grained rules is almost zero, except for Task A, where one generated sample, shown in Figure 10, has the sun and shadow on the same side of the pole.
Generations that Violate Fine-Grained Rules.
We then proceed to show the samples
generated by DDPMs that do not satisfy the fine rules Figure 1 11, and highlight the features of interest using the evaluation method developed in Section 3.2.
Generations that Satisfy Fine-Grained Rules. Here, we use two coordinate systems: a 4D representation capturing key features (4, 12, h, h2) and a 13D representation that additionally encodes the RGB colors of the sun, pole, and shadow. This dual-coordinate analysis allows us to distinguish whether differences between generated and training samples arise from structural variations or merely from different color combinations within similar structures (Okawa et al., 2024). We then compute the Euclidean distances between each generated sample and its nearest neighbor in both 4D and 13D spaces. As supplement to the DDPM memory experiment in Section 3.3, Figure 12 presents the three nearest. neighbors in the training data for high-quality generated samples (with ratios in [0.99, 1.01)) in both 4-dimensional and 13-dimensional coordinates. We observe that the 4-dimensional coordinates effectively capture the spatial structure of the nearest neighbors in the training data, while the 13-dimensional coordinates provide a more comprehensive understanding of the similarity of the generated samples, accounting for both color and structure.
Figure 10. For Task A, while all training samples have the sun and shadow on opposite sides, DDPM generates one sample violating this coarse-grained rule where the sun and shadow appear on the same side.
18

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
by by: 0.25 bg: Rortor 0.92
6::0 ne, Aot 922 fo: fizt Runtor L28
Ay 09, by: o M. 1 ho: Ratiel
6: 19, Au: 222 6: Butlo 1.3
f: o0, hu: o 37, Iy 0.00,
Ic 45, Ry: 10, 0g:0
by M4 by: 31, At 16, Rutior 0.68
6:10 40, 0 hy Rabie
Task A
J: 0.066 by: 0,54, 1) 0.41, 0.03, Ratiuid
Ii 10000, no 056, & 0.44 Ry 906, Remte 7a
0 25, hy 0.50 1i 666. o.9. Reh e40.4
I: 0.12 ny c.FP 1) 0.50 by; 0.16. Rehic.cc
A 0.09 by: 0.47, 072 Ay 0.16 Ramodize
Ii 0.D, Au 0.50 1i 0.69, ny: 0.36, Remisd
0.25 h: 0.2 An 0.50 by 0.31, Ratto
0 by 0.53 G 666 nz 016,
Task B
h2
n2
h1
h2
h1
h2
h1
h2
h1
hf
h2
o2 Rorlec.77
0.08, 0.M, Rertec no
0.1.
0.16
0.05,
Botho 1669
Rotto 171
Re
Retha..0
Task C
no 0208 ny 844, 26t
ny- ts 0447
ny: 0.303 ry: D444,
ry 25, ry: 0.38, BorieL50
Ty: 0.8 0.4, RutieiL.66
ny: B. co 0.25 Ratio
0.20 ns 0.44 Retiell55
no 925, ni 0.44 Rahie170
Botis1000
Task D
11
12
12
12
12
Figure 11. Examples of Rule-violating Generations. We present samples generated by DDPM that violate fine-grained rules in four synthetic tasks, with annotations of features of interest and Ratio calculations. The target Ratios for Tasks A, B, C, and D are 1, 1, v2, and 1.5, respectively.
E.3. More Settings of Synthetic Tasks
In this section, we consider additional factors, such as more powerful model architectures and larger training datasets, to evaluate the diffusion model's ability to learn precise rules in Task A. Furthermore, detailed experimental results not included in Section 3.3, such as samples that violate coarse rules, will presented in this section.
More Training Epochs. Taking Task A as an example, Figure 13 shows the impact of more training epochs on learning fine-gu mne-grained rules. We observe that as the number of training epochs increases, the DDPM's s ability to learn fine- ane-grained rules improves significantly from 200 to 400 epochs, with R² increasing from 0.19 to 0.85. This indicates that the relationship between h2 and 12/1 is better described by the linear model. However, there is no noticeable improvement in the model's ability to learn the fine-grained rules, as reflected by the slight changes in the fitted line coefficients and R² remaining around 0.85.
the training continues up to 4000 epochs,
More Model Architectures. Then, we consider the factor modle architectures and use more powerful backbones, DiT (Peebles & Xie, 2023) and SiT (Ma et al., 2024), to replace U-Net as the denoising network. Specifically, we consider two sizes of DiT and SiT: DiT Small with 33M parameters and patch size (DiT-S/2), DiT Base with 130M parameters and patch size (DiT-B/2), SiT Small with 33M parameters and patch size (SiT-S/2), and SiT Base with 130M parameters and patch size (SiT-B/2). Keeping the number of training epochs, noise time steps, and other hyperparameters consistent, we find that, compared the 14M parameter U-Net, the parameter count of SiT and DiT has increased by 2 to 10 times. However, as revealed in Figure 14, although all models follow coarse rules, the deficiency in DDPM's ability to learn fine-grained rules does not significantly improve with the increase in parameter count, and there is even slight decrease in performance with D/T-S/2.
More Training Data. Next, we consider the impact of training data size. For Task A, we gradually increase the sample size from 4000 to 20000 to 40000 and observe whether increasing the sample size improves the DMs' ability to learn rules. Figure 15(a) and Figure 15(b) show that the increase in sample size does not enable DMs to learn fine-grained rules better, as evidenced by the almost unchanged R² and the fitted linear model. Similarly, we do not observe DMs violating coarse
19

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Genene miy
Doge
Wewiie bege
M
Oeeeree Douge
Figure Generations Violate Fine-Grained Rules. Taking Task an example, we show 10 high-quality generated samples and their Top-3 nearest neighbors from the training data. The first column visualizes the generated samples, while columns 2-4 display the Top-3 nearest neighbors from training data in coordinates, where similarity mainly reflects spatial structure. Columns 5-7 show the top-3 nearest neighbors in 13D coordinates, where similarity primarily reflects object colors.
20

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Eoech 200
Fpoch 400
Spzch 600
Epech 800
0.33
031
0.0
3.30
017
0.00
012-
0:00
300
012
0.2
2.30
2.0
012
0.5
0.5
0.5
2.5
6.7
1/Pg
A/b
1/fo
Fpach 1200
Epech 1600
Easch
Epach
0.7
029
025
Grand y-v
Groad
Groud
Scieatern yor wi
Echael Babce 292
011-
011-
0.00
3.0
015
0.5
022
025
0.0
622
058
330
211
0.50
0.00
5.0
500
0.00
012
015
031
000
020
211
Epech
Epoch
Epech
Epach
Arvand Trats
6rard Iuall
Broud Iacty
4rand Tradl
bvinetes
1.00
3.75
0175
0.75
015-
0.59
0.8
3.0
3.59
0.2
020
0.1
0.00
0.29
0.50
2.5
0.00
0.08
0122
0.00
025
0.2
0.30 3.5
Figure 13. The learning capability of DDPM for fine-grained rules across training epochs. We observe that as epochs increase from 200 to 4000, DDPM's ability to learn fine- fineigraieed rules shows no significant improvement, evidenced by the stable Estimation line and R²2 This suggests that increasing training iterations does not alleviate DMs difficulty learning fine-grained rules. The visualized generated samples fall within the interval [2.5%, 97.5%).
635
A31
620
628
Graand Trutcy's
Guaddtatyzz
Srund Thath you
ya
600
M0
1pho
11122
6,h2
fyho
(a) 33M, DIT-S/2
(b) 130M, DIT-B/2
(c) 33M, S/T-S/2
(d) 130M, DIT-B/2
Figure 14. DDPM's capability in learning fine-grained rules with more powerful backbones. Even with larger and more advanced denoising networks, DDPM still cannot avoid generating samples that violate fine-graines rules. This indicates that DDPM's inability to learn -ranned rules is decoupled from model architecture. The visualized generated samples fall within the interval [2.5%,97.55 97.5%).
21

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Ground Truth:
Ground Truth:
Ground y-u
Estimatron: y 78x 0 00, R² 0.84
Estimation: y 76x 0 2.03 R2 0.36
Estimattonn y 82x 0 01. R² 0.85
6.00
0.00
0.29
0.23
12hx
1ph
6.0
0.05
0.35
0.2
0.05
2.30
3.35
0.22
0.25
0.33
0.35
0.30
5.35
0.28
0.25
0.2
0.35
0.30
0.35
3.20
0.5
12h2
12h2
12h2
(a) Training Data Size 20000
(b) Training Data Size 40000
(c) Image Size 64 x 64
Figure 15. DDPM's capability in learning fine-grained rules with increased training samples and larger image sizes. We observe that increasing training samples and image sizes does not significantly improve DDPM's ability to learn fine-grained rules, as evidenced by the stahle Estimation line and R². This suggests that neither expanding the training dataset increasing image resolution alleviates DMs difficulty in learning fine-grained rules. The visualized generated samples fall within the interval [2.5%, 97.5%).
rules with large samples.
More Image Size Choice. Our final consideration image size. In the main text, the images are only 32 x 32. Existing studies suggest that low-resolution images may lead the loss of details in diffusion models' generation (Chen et al., 2024; Niu al., 2024; Li et al., 2024a). Therefore, we consider larger input resolutions of (3, 64, 64), shown in Figure 15(c). We observe almost no improvement in the DMs' ability to learn underlying rules with generated samples that do not violate coarse rules. Due to computational constraints, we were unable to explore even higher resolutions. But it is clear that for relatively simple task like Task A, which does not contain rich semantics, DMs are unable to recover the underlying feature relationships even at the 64 x 64 resolution. This itself highlights the difficulty DMs face in learning hidden features.
E.4. More Experiments on Real-world Data
In this section, we train DDPM on real-world datasets to support the observation that diffusion models (DMs) can learn coarse rules but have limited ability to capture fine-grained ones. Specifically, we first consider SynMirror (Dhiman et al., 2024). a mirror dataset where the rules are defined by strict physical relationships between objects and their reflections. Figure/ 16 shows image generations from DDPM trained on the SynMirror dataset for 400 epochs. We observe that DDPM can learn coarse inter-feature rules-for example, the color and general shape of the objects and their reflections are consistent. However, it fails to capture fine-grained rules, producing errors in the precise position, angle, and detailed contours of the reflections. We further design a synthetic dataset based on CIFAR and MNIST to enable precise quantification. We synthesize 5 new categories by combining specific CIFAR and MNIST classes: (dog, cat, 0, 1), (automobile, truck. 2. 3), (deer, horse, 4, 5), (airplane, bird, 6, 7), and (frog, ship, 8, 9). The inter-feature rule is defined such that only specific categories from CIFAR and MNIST can be combined. We similarly train a DDPM with a U-Net on the CIFAR-MNIST dataset for 1000 epochs. To evaluate whether the generated images comply with the predefined combination rules, we use pretrained classifiers independently identify the CIFAR and MNIST components within each generated image. As shown in Figure 17, only about 20% of the generated samples satisfy the predefined rules, while over 90% of the training data adhere to them.
F. Proofs of Section 4
, x; (2)) (x, (..,,(()) x(P)). We derive pe(x pr(x," (1) (2) follows. First, we notice that x ~ N(aaCu, and xi 12)( N(aa(1 B,1) Then we
Proof of Theorem 4.2. Given the independence, we can write px(x;) = pe(x (1)
obtain
(1) xx (2) = ED, ((,()) (x
where we denote (S) |arcuT, (1-(1-1)) TT.
Thus the score can be computed as V (ogg(((x)
= [xog)((x",xx0) (..,,""),,,,,)))))"))) where
22

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
(a) Training SynMirror Data
(b) Generated SynMirror Data
Figure 16. Experimental results on real-world (SynMirror) data. (a) In SynMirror, the inter-feature rule is defined as the ohject constraining its reflection in mirror terms shape, size, color. train a DDPM with a U-Net on SynMirror for 400 epochs. (b) The trained DDPM can capture coarse rules, such as consistent colors between the object and its reflection, but struggles with fine rules, such as unrealistic contours, angles and positions. This highlights DMs limitation learning fine- inc-grained rules.
(1) x (T) E 2d and can be derived as v Po (x
Vpr(x xxx)
(2)
ED. (VVV (e
V llog (x( (1) x(2))=
(x(xx (1) xx
ED. WE 88
Ep. -Nix M4 (S), 120) (x+ ((())
ED [N (S), 88 I2a)
-,,7xx + 13 RDD (Te (()
ED, (c)
-3,3x + a..37
ED, xx)
where with slight abuse of notation, we let = [x] (1) TT) and denote
((x Ex)
xe)
(XxG E.).
Proof of Theorem 4.4. According to the decomposition of the rule-respecting error in terms of bias and variance, we have
Emse
C2 + Evariances where we compute
m
m
Evias
Σ
E 2(o((w
((])))]] (w.") u) Σ
E ( ,
[2)1)]] (w v) (2) a((w!?) w. x,
Evariance
Var
Σ al
(1) ( u)
where we use the law of total variance and denote Vari = Var(-I()-
Given the gradient direction only consists of u, w and v, w respectively for the two patches, we can decompose the
(1).0
(2).0
weights (1) w (2) r
into
w(1)) dr.w., + 7.
(2) Praw., + u
23

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
3 2
4
6
8
Pro-tniined CNN on CIFAR
Prediction
5
7
9
Split
(a) Training CIFAR-MNIST Data.
(Cat, Dog. 9, 1)
Prodiction
Pre-trained MNIST
4 888
(test acc: 99.45%)
Training Generation
(b) Generated
Data.
(c) Label Data via Pertained CNNs.
(d) Evaluation Results.
Figure 17. Experimental results real-world (-IFRT) data. (a) We synthesize 5 new categories by combining specific CIFAR and MNIST classes: (dog. cat, 0, 1). (automobile, truck, 2, 3). (deer, horse, 4, 5), (airplane, bird. 6, 7), and (frog, ship, 8, 9). The inter-feature rule defined such only specific categories from CIFAR MNIST can be combined. DDPM with U-Net trained on CIFAR-MNIST for 1000 epochs. (b) The trained DDPM can capture coarse rules, such as ensuring each image includes 2 digits and non-digit objects, struggles with fine rules, such as preventing duplicate digits, mismatched digits or mismatched CIFAR- -INASTMNIST categories. (c) To quantify DDPM's ability to learn fine-grained rules, use CNNs pre-trained MNIST and CIFAR to predict labels from sub-images of the generated data. (d) Following (c), we evaluate training and generated data by checking if predicted labels fall within the predefined categories. Over 90% of the training data meet this criterion, while only about 20% of the generated data do.
for [m]. In addition, decompose for each 1,2
c = c.-- + c,1 = -p()) c, + + )
where plr) denotes the projection onto the span of {w (p).0
(p).0) m²t ). Then we can write for the first patch
.... w
σ ((w!) r.t Xi (1))
(w. (1) u)
Cldnew + B +
( (o.,
+ Tr,u, u)
Σe (orsau(wW ( u) + tr we + uu e.2 ) + (u, (()) (ora u) trss))
= ((u, (
where (1) (.) is a polynomial with coefficients depending on Tr.s Pr.i 04s 9c, S. Similarly, we can write for second patch that
Σ
a((w
(2) (w v) = ((v, r.
where (-) is polynomial of the same form (.) except that Φrei Tr.es is respectively replaced with Pr.ts Sr,ts -Ç. Then we can lower bound the variance by
m
m
Evariance 2 E5 Vari E
o ((w (1) pie X (( u))
Vari Σ
ol(w x (2)
(w))
2 c. (0 Var (.
(1 ((,,())))) Var k [2) (2)
where we use law of total variance.
24

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Lemma F.1 ((Cao et al., 2022)). If !f (N( of), we have with probability at least -
agd(1 - O(d-1/3)) & w 0fd(1+(0-4)))
Kw/ u) logg/(8/10 8/8)00:
[w/. v) S (8lgg8/6)00
Proof of Theorem 4.6. Let ((P) (W,) = E 1 x |1
( (x(
(p)) + c
/Bll... Then the loss can be written as
P L(W,)- Σ LO (W.)
P
= Σ
E
(
(P) (x
+
B
Σ
B
(w/
(p)
w
1 B
(p)
(p)
We first simplify the loss follows, where we omit the superseript and consider single patch due to that each patch is independent and weights are separated.
Re,.
(Wr, xx,)) Wt
B7 xx,
61,1
3r
1
1 et,il
-ec..
(W,, X1,i) We
+ Been
B3
Xt,i
3e
Ct
2 Fee. [We,, Xe,)
32xx,
33
Xt,i
Be
12
1s
where we can compute each term following (Han et al., 2024a) as
1 Ee,, ((ee, xx,i) xI well (a² (We, xo,i)² 2 + B.llwell B3]]well3 |/w.l
a I
|xoxill 11
3
13 at Bees (wr,, xxs) (W.: Xn.) B
at
(We, Xo.1)
This suggests
B7 Xt,i a,, ||=
20 (o (w. xo.i)² + f. /r/l2
E Fe,, X1,)
(We. xo,i) + T2
1
where Iz is a constant independent of w.. Then we obtain the loss for the first two patches as
(w (1) [a (1) u)² 33 w! 11) (1)
2a
E
E
(w,
(w, (1)
+ Iz .1)2+1,
3
L (w (2) = (af - (w, (2)
v)? + 30
(2) ||3) || w (2)
20
E[(1 - (w, (2) v)2 + Iz- 3
We next analyze the training dynamics of the gradient descent the first patch. The second patch follows from similar analysis. For notation clarity, we omit the superscript.
The gradient for the first patch can be computed as
VL (2) (w.) = ||w. ww..l2(220)) (2a (w,, uju u23+228 w.) +
(w,,
22() u)² + Blllwall?) we
2 E(S (we, u.u Bi
It is noticed that the gradient only consists of directions of w and u. It suffices to track the gradient descent dynamics projected to the two directions u and wi. where wi = wo (w. u)u, i.e.,
(w k+1 u) = (wf u) 77 L(W.) u)
= (1 22 2 - -
20/ u)²)) (wi, u)
(w/+1, W| (wfi w: w 7/ ( 1 (WW.))))....)).)). 9P)1
(1 + n 438 w.s.l/w:'122 2 (w,, "
25

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Class 0
Class 1
Class 2
Task A
Task B
Task C
Task D
Figure 18. Constructed contrastive data includes three classes, which differ in fine-grainee rules.
It is clear that gradients become zero only when @1 + (w u = (()) This suggests that before conver- gence, |/ = o(1) given the initialization is small, i.e., du = O(d-1/2) We can then verify that (V())((;)) u) 2 ((w), w + for some constant C.
In addition, suppose we decompose w of W Te we can see
0 = (w.. w w|
ne = (w u)
which then implies
|| 11 (w 2 + (w u)²
This combined with the fact that ((W))((w), u) 2 (( w + suggests that (w k+1
w w cannot
increase to (11) without (wf. u) reaching ((1). Thus at stationary point, we must have both (w u). [[w +
Next, analyze the stationary point. Given the gradient only consists of directions w and u, have for any stationary point w,, it satisfies
(V (WE), w.) = 1|w. (we, u)² 2,3 w. 2 u)² + B:llwella) w.
20
[E( (().((.. u/2 =0
333
(V:()(w:)).u) u} = (/v.) E[< (we, u) + 2,3 u\) + +2(a/Ek33)w, u)² +11llwe.l?) (w., W)
2a?
[ u)= =0 82
We solve the stationary equalities
|w -,,, - / (680488 + 00033,"E(3" + 16
2,9
(w(")) uu.2 8?
+1]l|w
a?E({?
26

---

Can Diffusson Models Learn Hidden Inter-Feature Rules Behind Images?
Similarly, we compute and solve the stationary point for the second patch where E[ç"] is replaced with B[(1 c)
We then compute the bias error
Ebias Bei xo (w: (2) xx( (ww (2) u) + (wl (2) (2) (w( (2)
v)
B
= [. (2) u/² a,, (w(w)) (2) v/2
3
CC((E(), a., B.)
It can be easily verified that there exists constant bias Co that depends E(ç], B( 04, Be. In addition, we compute the variance
(a ( u + W 13) (w:") (1) u)² (afE((1 <) v) 1 w/ (2)))))))) (pp)))((') (w!?), v)²
Evariance
+ +2afBis(1 ()( (1) (w. (2) v)² (a.E(</(()) (1) u)² o (w: (2) v))
= varV (1) u)* + a Var(1 - (w, (2) -2a Cov(S. (w // (w? (2) .)?
+ B 11 (I) + [, ww (2) (w v)2
= aa (C(w!") u/ - -1- -()(w)))) v) + B/ ( (w /, .)
= CE(() E(S"), an, a1,3)) >
where we see B[4º]- - E[A]² (A)(A) 2 0 for arbitrary random variable A.
G. Experiment Details on Synthetic Data with Two-layer Diffusion Model in Section 4
In order verify the theoretical claims on DMs failing precisely recover the inter-feature rule (*) (in Section 4), conduct numerical experiments
two-layer diffusion model two-patch data distribution.
Rd v= [0, 1, 0, -.. 0] Rd with d = 100. The score network follows the structure in (1) where we consider (-)) to be ReLU, linear, quadratic and cubic activation functions. We set network width m = 20. To simulate the DDPM loss in expectation, for each epoch, we sample n 1000 input data X0, i E [rx] and for each data we sample The = 1000 standard Gaussian noise i.j E [1000]. and consider minimizing the empirical loss
Specifically we set x = xx()) where x(1) Çu, x(2)T_ (1 ()v. Here we set = [1,0.-..
R2 2
L(W.) = nn
S (xC
(p)
t, 1l2
p
where X(, x 3. p 1,2. We use gradient descent to train the score network for 5000 epochs. We consider Or = exp(-t) and B, VI exp(-20) where we set / = 0.2, 0.4, 0.6, 0.8. We then check whether learned diffusion models Iearn the ground-tre truth rule (+) by plotting the distribution of thy (xx) against as /8.. The distribution (x,)) is estimated with 5000 samples X,-
(p) (p)
H. Details of Mitigation Strategies
H.1. Details of Guided Diffusion
Guided Diffusion is common strategy that trains an additional classifier to guide DDPM generation towards desired samples during the sampling process.
Training Details and Results. This section includes the details of training classifiers with contrastive learning as guidance. Figure 18 visualizes the constructed contrastive data, where each dataset includes three sample types that differ only in fine-grained rules and appear nearly identical at a glance. Figure 19 visualizes the contrastive datasets constructed for each of the four synthetic tasks. The classifier training for each task is treated as a three-class classification problem with 2000
27

---

Can Diffusion Models Learn Hidden Inter-Featur Rules Behind Images?
Srand Teahi
Ground Trush biye 2ellle0e22,... 27-0 alddg
Caro y05t0-002 p1-0.0
alxd33
Geet ydtkke afsa
Oar2 nheay
yet
046
0.31
031
1//2
1
(a) Task A
(b) Task B
(c) Task C
(d) Task D
Figure 19. Construction of contrastive training data. For each task, we build three-class dataset where Class represents samples satisfying ame-grained rules, while Classes and represent samples that only satisfy coarse-grained rules. Based on these constructed
contrastive datascts, we train classifiers as additional guidance to improve DDPM's generation.
Conc 0
Cluss
Conc
Cless Chocc
Cure
Dimentsion
Dimsension 2
Dimenstoo
Dimersion
Dimension
Dimension
Dimension
Dimension 1
(a) Task A
(b) Task B
(c) Task C
(d) Task D
Figure 20. CLIP representation of contrastive training data. For each task. use the CLIP model to extract its representations and apply UMAP for dimensionality reduction. observe that the contrastive data nearly inseparable, which presents a challenge for training the classifier.
positive samples (class 1) and 2000 samples per negative class (classes 0 and 2). We use U-Net as the classifier architecture, trained for 20000 iterations with a learning rate of 3e 4, and a contrastive learning weight À Beyond standard guided diffusion, dynamically adjust guidance weights (gradient scales) with a piecewise strategy where guidance is activated only in the final 20 denoising steps. The weight linearly increases from 0 to predefined gradient scale factors (7 for Tasks A/C, 10 for Tasks B/D). Through comparation of constant versus piecewise weighting, we report optimal strategies: Task A,B,D for standard sweighting method and Task C employs the piecewise weighting method. As noted in Section 5.2, training high-accuracy classifiers is not easy in our problem. as evidenced by the accuracy of the training data for Tasks A, B, C, and being 0.57, 0.51, 0.55, and 0.63, respectively.
NT-Xent Loss. NT-Xent Loss (Normalized Temperature-scal Cross Entropy Loss) (Sohn, 2016) is commonly used in contrastive learning to measure the similarity between positive pairs (similar samples) and distinguish them from negative pairs (dissimilar samples).
exp(sim(Z, 2,)/T)
CNT:Xom((,..) = log
2N k= 1 [1>s]] exp (z;, zz /x)
Where z; and z, are the embeddings of the i-th and j-th samples, sim(z,, z,) = 1 ||||
2, is the cosine similarity and the temperature parameter that scales the similarity which we set T 0.5 in our experiments.
H.2. Details of Filtered DDPM
Filtered DDPM is a more straightforward strategy that uses classifier trained on raw images to filter DDPM generations, keeping only samples predicted to satisfy fine-grained rules.
Training Details and Results. Based on the contrastive data constructed in Figure 19, we split the training and test data in
28

---

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
80:20 ratio and directly train three-way classifier in the raw image space, using MLP, ResNet-8, and U-Net architectures. These models trained for 100 epochs with learning rate of 3e 4. As shown in Figure 21, the classifiers achieve accuracy between 60% and 80%. While they outperform classifiers trained for guided diffusion due to the noise-free setting, they still fail to achieve 100% accuracy, even for these simple synthesis tasks. Additionally, Figure 20 shows the representations extracted by CLIP (Radford et al., 2021) for each synthetic task, followed by dimensionality reduction using UMAP (MeInnes al.. 2018). We observe that the data from different categories in the contrastive data is difficult distinguish, which presents challenge for training the classifier. Based test accuracy, we use the trained MLP model to filter DDPM generations for Tasks A and B, keeping only samples predicted as Class (satisfying fine-grained rules). For Tasks C and D, we use the U-Net model for filtering.
Resher
U-Net
Accuracy (%)
Task
Teak
Taak
Task
Figure 21. Test accuracy differ- ent models on contrastive datasets.
29


Reviews:
[{"review_id": "C3BgOmc4Uo", "reviewer": "Reviewer_67RS", "summary": "This paper is motivated by some prevalent real world failure case of diffusion model learning rules between spatial parts and features. They developed a few synthetic tasks to test the learning of diffusion model on inter object rules (spatial or non spatial). Though the overall layout is correct and rough scene rule is obeyed, more precise (linear) spatial relation was kind of obeyed, but not accurate (in the sense of R2\\neq1). Then the authors developed a theoretical set up to explain why diffusion training do not lead to rule learning, and proved that with certain theoretical setting (patch data, inter patch rule and separable score function approximator per patch) then the network cannot learn the rule, with a constant error bound on the rule. \n\nFinally the authors developed a few simple yet effective way to mitigate rule confirming problems, e.g. via guided sampling or posthoc rejective sampling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time reviewing this paper. We now address your questions as follows.\n\n---\n> Q1: Theory\n\nThanks for your good points.  First, Gaussian setup in [W2025] is a **special case** of our theory. Particularly, as ``Theorem 4.2``, the score can be written as \n$$\\nabla \\log p_t(x_t^{(1)}, x_t^{(2)}) = - \\frac{1}{\\beta_t^2} x_t + \\frac{\\alpha_t}{\\beta_t^2} \\begin{bmatrix} \\gamma(x_t) u , \\\\ (1- \\gamma(x_t)) v \\end{bmatrix}$$\nwhere $\\gamma(x_t) = E_{\\zeta} [\\pi_t (\\zeta, x_t) \\zeta]$. When $\\zeta$ follows Gaussian distribution, the score can be simplified into a linear function over $x_t$, which can be learned by a linear model. However, **we aim to cover more general setup where the above score is *non-linear* in $x_t$**, i.e., $\\zeta$ can be any bounded distribution. In such setting, the score function can be generally formulated as a combination of a fixed linear term $-1/\\beta_t^2 x_t$ and an additional non-linear term, which motivates us to apply the two-layer network with the residual connection as the score network.\n\nMoreover, our current patch-separated configuration follows prior work (Han et al. 2024a), while similar results can also be extended to network handling dependent patches. Particularly, we can consider\n$$s^{(1,2)}_w(x_t) = - \\frac{1}{\\beta_t^2} x_t^{(1,2)} + W \\sigma(W^\\top x_t^{(1,2)})$$\nfor some polynomial activation function $\\sigma(\\cdot)$ and $W \\in \\mathbb R^{2d \\times m}$. Then, it can be seen that only when $\\langle W \\sigma(W^\\top x_t^{(1,2)}), [u; v] \\rangle = \\alpha_t/\\beta_t^2$ for any $x_t$ which concludes the network learns the rule ``(*)``. However, with the new network, we can still follow ``Theorem 4.4`` to show: (1) parameters $W$ will only be the function of $u$, $v$ and the initialization and (2) the network function will be basically a polynomial function of $\\langle u, \\epsilon_t\\rangle$ and $\\langle v,\\epsilon_t\\rangle$ and their cross terms (not appeared in patch-separated network), where $\\epsilon_t$ denotes the added diffusion noise in $x_t$. As $x_t$ varies, the function output also varies, which results in a non-vanishing rule conforming error that depends on the variation of $\\epsilon_t$. Then similar results in ``Theorem 4.4`` can be obtained and our main theoretical arguments still hold. \n\n> Q2: Interpretation of failure / threshold of Figure 5A / Raven’s matrices\n\n**Interpretation of failure:**\n- Compared to the training data with $R^2$ close to 1 (``Figure 3``), synthetic tasks yield significant lower $R^2$ with 0.6–0.8, indicating weaker rule learning.  \n- $R^2$  measures linear fitting quality, not rule accuracy. Differences in coefficients are also important, e.g., Task A’s estimation line $\\beta_1 = 0.82$ is smaller than ground truth $\\beta_1 = 1$. And ``Error`` metric in ``Table 2`` which combines coefficient deviation and MSE further quantifies rule learning limitation of DMs.\n\n**Figure 5A:** we adopt a strict threshold to show DMs can generate high-quality samples even under such strict conditions. Since DDPM perform well under strict settings, they naturally perform well under more relaxed thresholds.\n\n**Raven Matrices:**  fine-grained rules can also be measured in discrete state space. Specifically, we divide features within [0,1] into 20 intervals. A generation is rule-conforming if measured features fall within the interval satisfies predefined rules. [Discrete Results](https://anonymous.4open.science/api/repo/Rebuttal-8656/file/discretemetric_ID15070.pdf?v=1f32cb35) show 95% of training data satisfy rules, while only 50–80% of generations do, highlighting DMs' limitation in learning rules.\n\n> Q3: line 299 is not correct.\n\nWe will modify ``line 299`` to sum of projection.\n\n>Q4: Coarse/fine grained rules.\n\nFine-grained rules impose stricter requirements than coarse ones. As shown in ``Section 4``, coarse rules only require the network to discover key features $\\mathbf{u}$ and $\\mathbf{v}$, while fine rules additionally require satisfying constraints between them—e.g., their projections summing to a constant (``Definition 4.3``). Thus, learning coarse rules doesn't guarantee fine-grained rule learning.\n\n>Q5: Spatial/Non-spatial rule.\n\n- Spatial rules involve spatial arrangements like positions and layouts (e.g., Light-shadow in ``Figure 1``), while non-spatial rules relate to features independent of space, such as size and texture in ``Figure 1``. ``Section 3.3`` explains non-spatial rules are harder to learn, possibly due to the lack of explicit cues like positions and lengths in spatial rule, which may explain why Task C is more hard than others.\n- Our theory focuses on spatial rules with clearly defined patch-level dependencies but also can extend to non-spatial rules. For example, by introducing proper tokenizer, inseparable non-spatial rules in pixel space can be separable in latent space, allowing our theory to be applied.\n\n---\nWe hope this addresses your questions. Please let us know if you have any further concerns.", "claims_and_evidence": "This paper provided ample evidence for most of its claims, and it’s a well completed paper. I totally agree there are rule learning issues in the practical and synthetic setups. However, I have some issue with the conclusion from theory.\n\n- **Main issue:** I get the theorems in the paper, they are correct. But the overall claim / conclusion made in the abstract that “*Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity.*” has some issue.\n    - This claim is based on a specific theoretical set up where two patches are strongly correlated. We can say the data is supported on a effectively 1d manifold like $\\zeta[u; -v]$, with certain offset.\n    - However the authors also designed a patch-wise neural network model. Where the output only depends on corresponding part of input! This design made it basically impossible to approximate the true score of the data manifold.\n    - Consider an analytically solvable case where $\\zeta \\sim \\mathcal N(0,1)$ then $[x]\\sim \\mathcal N([0;v],[u;-v][u;-v]^T)$ is distributed as a degenerate Gaussian, with only one nonzero eigenvector in covariance $[u;-v]$. Then its score at any given moment is tractable (Eq 5 [WV2024]).  For Gaussian $\\mathcal N(\\mu,\\sigma^2I+\\Sigma)$, the score is a linear function, and it looks like $(\\sigma^2I+\\Sigma)^{-1}(\\mu-x)=\\sum_i\\frac{1}{\\lambda_i+\\sigma^2}\\nu_i\\nu_i^T(\\mu-x)$. Basically it’s a full matrix. with a major component spanned by the PC of data $[u;-v][u;-v]^T$. So the score $s^{(1)}$ part depends on both the state $x^{(1)},x^{(2)}$. However your network design prohibit it from depending on the $x^{(2)}$ which **definitely** causes it to be unable to approximate the true score or learn the true data manifold. In the linear case as in Theorem 4.5, the effective weight matrix in your network is block diagonal and each block is rank 1. But the true score for Gaussian requires a full matrix. and the off diagonal blocks cannot be zero.\n    - On a higher level, basically you designed a separable network, and the loss is also separable, so basically each patch network learns its own distribution, and it cannot learn the correlation between the two patches. in the end it can only learned a factorized distribution.\n    - During rebuttal, the author could address this by modifying the theoretical set up, adding certain setup where dependency is not local to a patch. The authors could also edit their overall claim, and not to attribute the failure of rule learning to the diffusion training on Denoising score matching (DDPM) loss, but to their model design. Currently it seems with a full dependency score model, even being linear function, it will converge to the correct supporting manifold [W2025] (Proposition 5.1). (Though linear network will not learn the correct distribution on manifold, it can only learn Gaussian like things)\n- **Minor issue:**\nFor the empirical experiment on synthetic tasks, whether or not to call it a failure is quite arbitrary. I feel it’s quite successful on ABD. In Figure 5A, the threshold of +-0.01 is quite stringent.\n\n[WV2024] Wang, B., & Vastola, J. J. (2024). The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. TMLR\n\n[W2025] Wang, B. (2025). An Analytical Theory of Power Law Spectral Bias in the Learning Dynamics of Diffusion Models. *arXiv:2503.03206*.", "methods_and_evaluation_criteria": "- I agree that based on current results, one way to enforce better rule conforming is via classifier guidance and post hoc rejection. It's nice that the authors tried this and showed some improvement.", "theoretical_claims": "I checked Theorem 4.2\n\n- At line 299 the statement “*data, requiring that the norm of the first two feature patches sum up to one, i.e., ∥x(1)∥ + ∥x(2)∥ = 1.” **is not correct** in some case**. I think it should be the sum of projection on u an v**  sum to 1.* $\\lang u, x^{(1)}\\rang+\\lang v, x^{(2)}\\rang=1$\n- See **Claims And Evidence* for a conceptual issue I have with the theoretical treatment in Section 4.**", "experimental_designs_or_analyses": "I applaud the authors for the synthetic data design in this paper, which were well based on actual image diffusion model and their failure cases. \n\n- **Minor issue: interpretation of failure**\nFor Figure 4, it’s like a half full half empty scenario… whether we call it a success or failure is kind of subjective. To me it’s already quite successful, given A,B,D tasks all have R2~0.80. If you measure correlation you should get sth like 0.90. “…*, where deviations from the ground truth in linear fitting and the coefficient of determination R2 below 1 indicate that DMs fail to fully capture the predefined fine-grained rules*” I feel this is too high a bar to ask for empirical results…\nIn contrast, for cases in previous works like in Raven’s progression matrices, since the state space is discrete, the evaluation of rule following seems to be more accurate.", "relation_to_broader_scientific_literature": "The results in Figure 5 B. is quite related to observations in [WSS2024] (Fig.3), for diffusion trained on Raven’s dataset, where the overall rule conforming samples were novel and far from the training set. While local parts of it could be quite similar to some local parts of the dataset. Seems like recombining local parts to create new “scenes”, potentially due to some mechanism as proposed in [KG2024]. \n\n[WSS2024] Wang, B., Shang, J., & Sompolinsky, H. (2024). Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules. NeurIPS Workshop *arXiv:2411.07873*.\n\n[KG2024] Kamb, M., & Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. *arXiv preprint arXiv:2412.20292*.", "essential_references_not_discussed": "One relevant concurrent work that shares the basic theoretical set up with Theorem 4.5 is in **Prop 5.1** in [W2025], i.e. linear symmetric score / denoiser, small or aligned initialization, with slightly more general requirements. Basically their results also confirm that since the data covariance has such low dimensional structure, the weights will automatically discover such low-dimensional structure (feature dimension) by gradient training. But as I pointed out before, in their case the network is overall linear, without adding patch constraint, so their network will recover the correct 1d data manifold as in your case, and will “learn the rule”, or point towards the 1d manifold. \n\n[W2025] Wang, B. (2025). An Analytical Theory of Power Law Spectral Bias in the Learning Dynamics of Diffusion Models. **", "other_strengths_and_weaknesses": "- Overall this is a quite complete paper, showing practical relevance, well motivated set up and theory explaining it, and ways to mitigate the issue. I applaud the authors on such a work!\n- The visualizations were well done, and the empirical and theoretical results were stated clearly.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- I think the most crucial conceptual question I have is about the distinction about coarse level rule and file grained rules. I can see the distinction in your example, but more generally and theoretically / conceptually, what distinguish coarse level rules from fine grained ones? Why some can be learned and some cannot?\n- Similarly can the authors comment on what is spatial rule and what is non spatial rule, or why the task C is not learned as well as A,B,D.,? Does the theory give you any new insights?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NNXVEEXRc6", "reviewer": "Reviewer_KnJN", "summary": "This paper evaluates diffusion models from both experimental and theoretical perspectives on inter-feature rule learning, indicating that while they can capture coarse rules, they struggle with fine-grained ones. The authors also provide a preliminary method to mitigate this shortcoming in learning fine-grained rules.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's efforts on reviewing this paper. We now address the questions raised as follows.\n\n---\n\n>Q1:The proposed approach to facilitating fine-grained rule learning appears to have no direct connection with the theoretical analysis. /The proposed method is unrelated to the main analysis of this paper\n\nThank you for the question. Our **theoretical analysis** in main text highlights that the failure of classical DDPMs in learning inter-feature rules is mainly due to the objective of denoising, which does not explicitly capture the hidden inter-feature rules. Therefore, classical DDPMs, when trained solely with the standard objective, lack the inductive bias necessary to learn fine-grained inter-feature rules. **This naturally inspires us to introduce additional guidance to steer the sampling process**, encouraging DDPM to generate rule-conforming samples.\n\nAdditionally, ``Figure 5`` in **experimental analysis** in main text shows that DMs can generate high-quality samples that meet fine-grained rules, but the process is unstable and prone to rule violations. Therefore, our proposed method introduce additional information to help DDPM stably sample from high-quality regions (more discussion in ``Lines 300–322``).\n\nWe will add more discussions in our revised version.\n\n>Q2: The proposed method to facilitate fine-grained rule learning is a bit straightforward/ The proposed approach to facilitating fine-grained rule learning d achieves only limited improvements. \n\nThank you for your question. The **main focus of our work** is to identify the limitations of DMs in learning fine-grained rules through experiments and theoretical analysis, rather than to propose a complete solution. This limitation has been overlooked (as discussed in ``Section 2``) and represents 'a relatively underexplored area' (as noted by Reviewer 9KzS). Our work aims to 'extend previous findings on compositionality and factual consistency in diffusion models' (as noted by Reviewer gKDf).\n\nAdditionally, the proposed method is an initial attempt to enhance rule learning. Importantly, we identify a **key bottleneck**: the signal of fine-grained rules is too weak for the classifier to capture, a phenomenon not been highlighted in traditional DDPMs, such as those targeting ImageNet tasks with classifier guidance (see ``Section 5.2``). We hope that these early attempts and bottleneck analysis can provide valuable insights for future exploration.\n\n[1] Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review.\n\n-- ----\n\nWe hope the above response resolved the questions and if there is further concern please let us know.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The motivation for evaluating the ability of diffusion models to learn fine-grained rules is well-justified and necessary, as it aims to address a major concern that limits the quality of generated outputs in recent large diffusion models.", "theoretical_claims": "Yes, the proof is logically clear and correct.", "experimental_designs_or_analyses": "Yes, the experimental designs are reasonable, evaluating the ability of diffusion models to learn physical rules using four carefully designed tasks.", "relation_to_broader_scientific_literature": "This paper is closely related to text-to-image generation and highlights a key limitation of recent large diffusion models: their difficulty in learning fine-grained inter-feature rules.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "### Strengths  \n- This paper provides a comprehensive evaluation of whether diffusion models can learn fine-grained inter-feature rules through both experimental and theoretical analysis.  \n\n### Weaknesses  \n- The proposed approach to facilitating fine-grained rule learning appears to have no direct connection with the theoretical analysis and achieves only limited improvements.", "other_comments_or_suggestions": "None", "questions_for_authors": "I think the proposed method to facilitate fine-grained rule learning is a bit straightforward and unrelated to the main analysis of this paper. I hope the authors can clarify this point, and I will adjust my score accordingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Gv2CUR7F4", "reviewer": "Reviewer_9KzS", "summary": "This paper investigates whether diffusion models can learn hidden inter-feature rules in images, focusing on the distinction between coarse-grained and fine-grained relationships. Through carefully designed synthetic tasks inspired by real-world phenomena—such as the spatial relationship between the sun and its shadow or the connection between object size and texture—the authors demonstrate that while models like Stable Diffusion 3.5 can reliably capture broad, coarse-grained rules, they consistently struggle with learning precise, fine-grained dependencies. The paper also presents a theoretical analysis showing that the denoising score matching objective inherently leads to a constant error in rule conformity, thereby limiting the models' ability to accurately recover the conditional distributions underlying these subtle rules.\nTo mitigate these shortcomings, the authors propose incorporating additional classifier guidance and filtering strategies during sampling, which yield moderate improvements in enforcing fine-grained rule adherence. Despite these enhancements, the experiments reveal that even advanced guidance techniques are insufficient for completely bridging the gap, as the nuanced signals of fine-grained rules remain challenging to capture. Overall, this work provides significant insights into the limitations of current diffusion models and offers a compelling direction for future research to improve rule learning in generative image models.\n\n## update after rebuttal\nThe authors address most of my concerns. So I keep my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and efforts reviewing our paper. We now address raised questions as follows.\n\n---\n\n>Q1: Further ablation studies across different architectures and training configurations.\n\n ``Section D.3 (Lines 964-1032)`` considers different architectures (U-Net, SiT, DiT) and training configurations including training epochs, training data size and image size. Experimental results show DMs still have limitations in learning fine-grained rules with different settings.\n\n>Q2: The proposed mitigation strategies do not fully resolve the issue.\n\nThanks for your comments. Our strategy is an initial effort to solve this issue. Importantly, we identify a **potential bottleneck**: the signal of fine rules is too weak to be captured by the classifier, a problem have not been highlighted in classical and Imagenet task-based conditional DDPMs (see ``Section 5.2``). We hope bottleneck analysis can provide valuable insights for future exploration.\n\nAdditionally, **the goal of our work** is to reveal the limitations of rule learning in DMs through experiments and theory. Fully addressing this challenge requires further work like rule-specific datasets and metrics, which are currently lacking. We leave the complete resolution to future work.\n\n>Q3:Theoretical proofs are based on simplified models.\n\nThanks for your question. ``Theorem 4.4`` highlights for non-linear two-layer neural networks, there is constant errors due to the variance of diffusion noise. ``Theorem 4.5`` explicitly derives the constant error. We believe the conclusions from ``Theorem 4.4`` and ``4.5`` extend beyond the simplified setups. Intuitively, the score function is required to satisfy a low-dimensional constraint that holds for every noised input. However, without explicitly embedding this constraint into the model or the training objective, learning it from data becomes inherently difficult. Since the constraint must hold globally, neural networks lack the inductive bias needed to recover such structure from finite samples.\n\n>Q4:Real-world images.\n\nThank you for your good points. We conduct additional experiments on real-world datasets to further demonstrate DMs can learn coarse rules but struggle with fine ones.\n\n- **SynMirror** [1] presents objects and their reflections, where rules link their features like color, size, and shape. **We find** DDPM captures coarse rules (e.g., matching colors between objects and reflections) but struggles with fine ones, showing shape mismatches.\n- **Cifar-MNIST** pairs specific CIFAR and MNIST classes (e.g., Cats/Dogs with 0/1). **We find** DDPM satisfies coarse rules (e.g., always generating two digits and two objects), but only 20% of generations follow fine-grained rules requiring specific class pairings.\n\nSee [Real-world Data](https://anonymous.4open.science/api/repo/Rebuttal-8656/file/exp_ID15070.pdf?v=e6e553d4) for more details. We will add these into revised manuscript.\n\n>Q5:The sensitivity of feature extraction process to hyperparameters (e.g., predefined HSV ranges).\n\nSorry for the confusion. **There is no hyperparameters in the feature extraction process**. The HSV values used are predefined during training data construction. For example, in Task A, the sun’s HSV is set to yellow with hue [0, 30], saturation [100, 255], and value [200, 255]. The same HSV range is used during feature extraction (see ``Lines 747–806`` for details).\n\n>Q6: Potential future directions. \n\nInspired by [2,3], one potential direction is optimizing the sampling process during inference. We can introduce additional reward signals from human feedback or reward models to guide DMs during sampling. Additionally, improving the tokenizer to better learn semantic information related to rules could also enhance rule learning. We will add this discussion to revised manuscript.\n\n>Q7: Can you provide more insights into why these approaches yield limited gains? \n\nThank you for mentioning this question. ``Section 5.2 (Lines 408-424)`` shows limited improvements is due to weak signals of the fine-grained rule which make the guidance from the classifier isn't strong enough to completely correct the sampling. Specially,\n- ``Figure 16``  shows inseparable CLIP representations of contrastive data, making classifier training challenging.\n- ``Figure 17`` demonstrates that training on simple contrastive data results in test accuracy below 90%, highlighting the difficulty in distinguishing subtle differences.\n- [Visualization](https://anonymous.4open.science/api/repo/Rebuttal-8656/file/vis_ID15070.pdf?v=4c7afd58) demonstrates the weak differences between different classes in raw contrastive data.\n\n[1] Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections.\n\n[2] Aligning Text-to-Image Models Using Human Feedback.\n\n[3] Human preference score: Better aligning text-to-image models with human preference. \n\n-----\nWe hope above response can address your concern and we are open to discuss more if any question still hold.", "claims_and_evidence": "The submission’s claims are generally well-supported by both experimental and theoretical evidence. The authors convincingly demonstrate that diffusion models can reliably learn coarse-grained inter-feature rules while consistently failing to capture fine-grained dependencies, as evidenced by low $R^2$ values and significant error metrics in their synthetic task evaluations. Their theoretical analysis further strengthens this claim by showing that the denoising score matching objective inherently induces a constant error, which limits the models’ ability to precisely learn the hidden rules.\nHowever, some claims could benefit from additional clarification. For example, the assertion that the observed constant error is solely a consequence of the DSM objective might be problematic without further ablation studies across different architectures and training configurations. Additionally, while the proposed mitigation strategies (classifier guidance and filtering) show moderate improvements, they do not fully resolve the issue of fine-grained rule learning, suggesting that further empirical validation is needed to conclusively support their effectiveness.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in the paper are well-suited for the problem at hand. The authors design controlled synthetic tasks that specifically target both coarse-grained and fine-grained inter-feature rules, which effectively isolates the aspects of rule learning from the broader complexities found in natural images. This targeted approach, using tasks inspired by real-world phenomena such as light-shadow interactions and object reflections, allows for a systematic assessment of diffusion models' abilities in capturing these dependencies.\n\nMoreover, the evaluation framework, comprising feature extraction via color-based masking, geometric measurements, and the use of metrics such as R² and a combined error metric that accounts for both bias and variance, provides clear, quantitative insights into how well the generated samples adhere to the predefined rules. While these synthetic benchmarks may not capture all nuances of real-world data, they offer a rigorous and interpretable means to evaluate and compare model performance, making the methods and criteria both reasonable and effective for the study's objectives.", "theoretical_claims": "I reviewed the theoretical proofs presented in the paper, particularly Theorem 4.2, which characterizes the score function for the multi-patch data model, and Theorems 4.4 and 4.5, which establish lower bounds on the rule-conforming error (including bias and variance components). Under the stated assumptions, the proofs appear to be mathematically sound and consistent, effectively linking the denoising score matching objective to the inherent constant error in learning fine-grained inter-feature rules.\n\nThat said, some aspects rely on idealized conditions (e.g., the use of a simplified two-layer network with linear activation in Theorem 4.5), which may not fully capture the complexities of practical diffusion models. While these simplifications are acceptable for isolating the core theoretical insights, further discussion or empirical validation would help clarify how these bounds translate to more complex architectures encountered in real-world applications.", "experimental_designs_or_analyses": "The experimental design is generally sound and well-justified. The authors construct synthetic tasks with clearly defined inter-feature rules, both coarse-grained and fine-grained to isolate the specific challenges of rule learning. The evaluation pipeline, which involves a three-step process of color-based masking, element counting, and keypoint extraction, is a clever way to quantitatively measure how closely generated images conform to the underlying rules. Metrics such as R² and the combined error metric (encompassing both bias and variance) are appropriately used to assess performance differences across various tasks and diffusion model configurations.\nHowever, some potential issues warrant further discussion. First, while synthetic tasks offer control and interpretability, they might not capture the full complexity of real-world images, possibly limiting the generalizability of the findings. Second, the sensitivity of the feature extraction process to hyperparameters (e.g., predefined HSV ranges) is not fully explored, and minor variations could affect the evaluation outcomes.", "relation_to_broader_scientific_literature": "The paper’s key contributions extend the existing body of work on diffusion models by focusing on the subtle, hidden inter-feature rules that standard generative models have largely overlooked. While prior studies (e.g., Ho et al., 2020; Dhariwal & Nichol, 2021) have demonstrated the high fidelity and compositional capabilities of diffusion models, they mainly address independent features and broad factual consistency. In contrast, this work delves into how these models handle nuanced dependencies both spatial (such as light-shadow relationships) and non-spatial (like size-color correlations), thus highlighting a gap in the literature regarding fine-grained rule learning.", "essential_references_not_discussed": "Overall, the set of references provided in the paper is largely sufficient.", "other_strengths_and_weaknesses": "The paper presents an original and comprehensive investigation into the ability of diffusion models to learn inter-feature rules, a relatively underexplored area. The introduction of synthetic tasks with clearly defined coarse- and fine-grained rules is innovative, providing a controlled environment to isolate and analyze model behavior. Moreover, the integration of theoretical analysis with empirical evidence， especially the derivation of constant error bounds due to the denoising score matching objective, adds significant depth and rigor to the work.\nHowever, the reliance on synthetic data may limit the direct applicability of the findings to complex real-world images. Additionally, some theoretical proofs are based on simplified models, such as two-layer networks with linear activation functions, which might not capture the nuances of more advanced architectures.", "other_comments_or_suggestions": "Consider providing additional details about the hyperparameters used in the feature extraction and evaluation process, as well as discussing how sensitive the results are to these settings.\nClarify the assumptions underlying the theoretical proofs, particularly in Theorem 4.5, to help readers better understand the limitations of applying these results to more complex architectures.\nA brief discussion on potential future directions to address the limitations imposed by the denoising score matching objective would be valuable.\nLastly, a careful proofreading to catch any minor typographical errors or inconsistencies would help improve the overall presentation.", "questions_for_authors": "In Theorem 4.5, your analysis is based on a simplified two-layer network with a linear activation function. How sensitive are the derived constant error bounds to these assumptions? Would similar limitations be expected in deeper or more complex networks with non-linear activations?  \nYour evaluation is based on synthetic tasks designed to isolate coarse- and fine-grained rules. Could you elaborate on how well these tasks correlate with real-world image generation?  \nRegarding the classifier guidance and filtering strategies, you show only moderate improvements in enforcing fine-grained rules. Can you provide more insights into why these approaches yield limited gains?  \nYour evaluation pipeline relies on specific hyperparameters (e.g., HSV thresholds for feature extraction). How robust are your experimental results to variations in these parameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ucy9YOYT0R", "reviewer": "Reviewer_gKDf", "summary": "The paper investigates whether diffusion models can learn hidden inter-feature rules in images by designing synthetic tasks that simulate real-world relationships (e.g., the connection between the sun’s height and the length of its shadow). The study finds that while these models can capture coarse-grained rules effectively, they struggle with fine-grained, precise dependencies—a limitation attributed to inherent constant errors in the denoising score matching objective. Additionally, the authors propose mitigation strategies, such as incorporating classifier guidance during sampling and using pixel-space filtering, which yield some improvements but do not fully overcome the challenge, thereby offering both theoretical insights and empirical evidence on the current limitations of diffusion models in rule learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's efforts on reviewing this paper. We now address the questions raised as follows.\n\n---\n\n>Q1: Broader datasets / Real-world data. \n\nThanks for your good question. To supprot that DMs can learn coarse rules but hard to learn fine-grained rules, we conduct additional experiments on two real-world datasets, SynMirror and Cifar-MNIST.\n\n- **SynMirror** [1] displays objects and their reflections, where inter-feature rules manifest as constraints between objects and their reflections in terms of color, size, and shape. **The results show** that the generation by DDPM can capture some coarse rules, such as objects and their reflections share the same colors, but it hard to learn fine-grained rules, such as there are siginificant differences in the shapes and contours.\n\n- **Cifar-MNIST** combines specific classes from CIFAR and MNIST, such as pairing Cats and Dogs from CIFAR with 0 and 1 from MNIST. **The results show** that generations by DDPM satisfy coarse rules, such as ensuring that each generated image contains two digits (MNIST) and two non-digit objects (CIFAR). But only 20% of the generations satisfy predefiend fine-grained rules, where only specific categories from CIFAR and MNIST are allowed to pair.\n\n[Real-world Data Results](https://anonymous.4open.science/api/repo/Rebuttal-8656/file/exp_ID15070.pdf?v=e6e553d4) provide more details and visualizations. We will add these results into the revised manuscript.\n\n>Q2: Mitigation strategy is not novel ... / any alternative strategies that might address these limitations more effectively?\n\nThanks for your good points.\n\n- **Novel Method**. The **main goal of our paper** is to clearly identify the shortcomings of DMs in rule learning from both experimental and theoretical perspectives. At the end of the paper, we make initial attempts to address this issue. Importantly, we highlight that a **potential bottleneck** is the signals of fine-grained rules are too weak to be captured (see ``Section 5.2``). This issue has not been reported in traditional DDPMs, such as those targeting ImageNet tasks with classifier guidance. We hope our initial strategies and bottleneck analysis provide valuable insights for further exploration.\n\n- **Alternative Strategies.** Inspired by existing work [2,3], for further exploration, we can introduce additional reward signals from human feedback or powerful reward models to better guide DMs during sampling. Additionally, improving the tokenizer to better learn semantic information related to rules could also enhance rule learning. We will include the discussion in the revised manuscript.\n\n>Q3: Some of the definitions, like the rule-conforming error, lack intuitive explanations / It would be beneficial to include more detailed explanations ... such as the rule-conforming error. \n\nThank you for the question. The accuracy of score learning is inherently tied to the generation quality of diffusion models [4]. As we have shown in ``Theorem 4.2``, in order to sample from the data distribution (with rule conformity), the score function should satisfy a constraint $\\langle \\nabla \\log p_t(x_t) + x_t/\\beta_t^2, [u; v] \\rangle = \\alpha_t/\\beta_t^2$. By designing the network function in eq. (1) (in the main text), the constraint is equivalent to $\\psi_t(x_t) = \\alpha_t/\\beta_t^2$ that holds for any $x_t$ (as in ``Definition 4.3``). The rule-conforming error, defined as the mean squared deviation from this value, exactly measures how closely the learned score aligns with the ideal constraint (as $x_t$ varies). In practice, this relates to generating images that respect structural properties such as fixed object size. The smaller the rule-conforming error, the smaller the estimation error of the score function and thus the more likely the samples generated by diffusion models adhere to the constraint.\n\nWe will revise the text to clarify this connection.\n\n[1]  Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections.\n\n[2] Aligning Text-to-Image Models Using Human Feedback.\n\n[3] Human preference score: Better aligning text-to-image models with human preference. \n\n[4] Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.\n\n-- ---\nWe hope the above response can resolve your concern and if there is further concern please let us know.", "claims_and_evidence": "The submission’s claims are largely supported by both extensive empirical results and rigorous theoretical analysis. The authors substantiate their main claim—that diffusion models can reliably capture coarse-grained rules but struggle with fine-grained ones—through well-designed synthetic tasks and clear evaluation metrics (e.g., R² values and error metrics), which convincingly demonstrate the performance gap. Additionally, the theoretical framework based on denoising score matching offers solid mathematical backing for the observed constant error in learning fine-grained rules. While the proposed mitigation strategies (guided diffusion and filtering) show some improvement, the evidence also clearly indicates their limited effectiveness. One concern is that the reliance on synthetic tasks may not fully capture the complexity of real-world images, leaving some room for further evidence on broader datasets.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited to the problem at hand, as the synthetic tasks and detailed feature extraction pipelines provide a controlled setting to assess the diffusion models' ability to learn both coarse-grained and fine-grained rules. The evaluation metrics, such as R² values and error measurements, effectively quantify the performance gap and highlight the models' limitations. However, while the mitigation strategy using classifier guidance during diffusion sampling does yield some improvements, it is worth noting that this approach is not novel and does not introduce fresh techniques for enhancing the handling of fine-grained rules. This reliance on an established method may limit the paper's overall innovation in terms of proposing new solutions to the identified challenges.", "theoretical_claims": "I reviewed the theoretical claims, focusing on Theorem 4.2, which derives the score function for the multi-patch data setup, and Theorems 4.4 and 4.5, which provide lower bounds on the rule-conforming error by decomposing it into bias and variance components. The derivations appear mathematically sound under the stated assumptions, such as linear activations and a two-layer network, and they convincingly support the empirical observation that diffusion models incur a constant error when learning fine-grained rules. However, some of the definitions, like the rule-conforming error, lack intuitive explanations; for instance, while the error is defined as the deviation of the score’s projected coefficient from an ideal value reflecting the hidden norm constraint, the paper does not clearly explain why this quantity should intuitively indicate correct rule learning. Providing more intuition behind such definitions would help readers better understand the connection between the theoretical quantities and the practical notion of rule conformity.", "experimental_designs_or_analyses": "I reviewed the experimental design, including the synthetic tasks (A–D) and the associated feature extraction and evaluation metrics (e.g., R², Error metrics), and found that the setup is generally sound and well-motivated for assessing the diffusion models’ ability to capture inter-feature rules. The controlled synthetic environment allows clear differentiation between coarse-grained and fine-grained rule learning, and the quantitative analysis convincingly highlights the performance gap. However, one potential concern is that the experiments are limited to synthetic tasks, which may not fully capture the complexities of real-world data. Additionally, while the use of classifier guidance to improve fine-grained rule learning is effective to some extent, it is a well-known technique rather than a novel contribution.", "relation_to_broader_scientific_literature": "The paper’s key contributions are well situated within the broader literature on diffusion models and image generation. It extends previous findings on compositionality and factual consistency in diffusion models—where prior studies (e.g., DDPM, score-based generative models, and works on hallucinations) primarily addressed independent feature composition and common failure modes—by focusing on hidden inter-feature rules that capture subtle dependencies between image features. Its theoretical analysis, which builds on denoising score matching frameworks, aligns with recent efforts to understand the limitations of diffusion objectives and complements studies on mode interpolation and memorization in generative models. Additionally, while the use of classifier guidance is not new, the paper integrates it into a framework specifically designed to address fine-grained rule learning, thereby contributing a fresh perspective that bridges empirical observations with theoretical insights in the context of inter-feature relationships.", "essential_references_not_discussed": "Overall, the paper sufficiently covers the essential related literature. The authors have cited key works on diffusion models, denoising score matching, and guidance strategies that underpin their theoretical and empirical contributions. The references discussed in the paper provide a comprehensive context for understanding the challenges associated with fine-grained rule learning and the limitations of current diffusion models, and no critical works appear to be missing.", "other_strengths_and_weaknesses": "The paper is commendable for its thorough analysis, combining rigorous theoretical derivations with well-designed synthetic experiments to investigate the limitations of diffusion models in capturing fine-grained inter-feature rules. Its originality lies in framing the rule-learning challenge in terms of hidden dependencies and providing both empirical and theoretical evidence of inherent constant errors, which is a valuable contribution to understanding diffusion model behavior. \n\nHowever, the paper's reliance on synthetic tasks might limit its immediate applicability to real-world scenarios, and while the use of classifier guidance for improvement is well-motivated, it does not introduce novel techniques. Additionally, some definitions, such as the rule-conforming error, would benefit from further intuitive explanation to enhance clarity.", "other_comments_or_suggestions": "Some additional suggestions: It would be beneficial to include more detailed explanations for some of the theoretical definitions, particularly providing intuitive insights behind concepts such as the rule-conforming error. Expanding on how these definitions relate to practical aspects of image generation could enhance clarity. Moreover, while the synthetic tasks are well-designed for controlled evaluation, including experiments on real-world datasets or more complex scenarios would strengthen the applicability of the findings. Finally, a careful proofreading to fix minor typos and improve the overall flow of the text is recommended.", "questions_for_authors": "The definition of the rule-conforming error is not very intuitive. Could you explain in simple terms why this quantity effectively measures the model's ability to learn the hidden rules?\n\nYour experiments are based on synthetic tasks. Do you have any results or insights on how your approach might work on real-world datasets?\n\nYou use classifier guidance to improve fine-grained rule learning, which is an established method. Have you considered any alternative strategies that might address these limitations more effectively?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Motivated by real-world failures in how diffusion models learn inter-object rules, this paper introduces synthetic tasks to evaluate spatial and non-spatial reasoning. While models capture rough layouts, precise spatial relations remain inaccurate. The authors develop a theoretical framework showing that, under certain conditions (e.g., patch-based data and separable score approximators), diffusion models are provably unable to learn such rules, incurring a constant error bound.

All reviewers find the result interesting and solid theoretical studies. Please incorporate the reviewers' feedback for revision in camera ready.
Decision: Accept (poster)