State Combinatorial Generalization In Decision Making With Conditional Diffusion Models
Authors: Xintong Duan
Yutong He
Fahim Tajwar
Wentse Chen
Ruslan Salakhutdinov
Jeff Schneider
Venue: Accepted by TMLR ()

Abstract: Many real-world decision-making problems are combinatorial in nature, where states (e.g., surrounding traffic of a self-driving car) can be seen as a combination of basic elements (e.g., pedestrians, trees, and other cars). Due to combinatorial complexity, observing all combinations of basic elements in the training set is infeasible, which leads to an essential yet understudied problem of zero-shot generalization to states that are unseen combinations of previously seen elements. In this work, we first formalize this problem and then demonstrate how existing value-based reinforcement learning (RL) algorithms struggle due to unreliable value predictions in unseen states. We argue that this problem cannot be addressed with exploration alone, but requires more expressive and generalizable models. We demonstrate that behavior cloning with a conditioned diffusion model trained on successful trajectory generalizes better to states formed by new combinations of seen elements than traditional RL methods. Through experiments in maze, driving, and multiagent environments, we show that conditioned diffusion models outperform traditional RL techniques and highlight the broad applicability of our problem formulation.

Published in Transactions on Machine Learning Research (12/2025)
State Combinatorial Generalization In Decision Making With Conditional Diffusion Models
Xintong Duan, Yutong He, Fahim Tajwar, Wentse Chen, Ruslan Salakhutdinov, Jeff Schneider Carnegie Mellon University
(ainlongd, (rintongd.gutonghee, Matoongeeefajwaaa wew.://aaaarrrreeeeeeeooo www.ssee,sallkkuuukk wtt.:///ee...oooc.cm.......
Reviewed on OpenReview: https: openrevieww net/ farum? id= XB1dd01022
Abstract
Many real-world decision-making problems combinatorial in nature, where states (e.g., surrounding traflic of self-driving car) can be seen a combination of basic elements (e.g., pedestrians, trees, and other cars). Due to combinatorial complexity, observing all combinations of basic elements in the training set is infeasible, which leads to an essential yet understudied problem of zero-shot generalization to states that are unseen combinations of previously seen elements. In this work, we first. formalize this problem and then demonstrate how existing value-based reinforcement learning (RL) algorithms struggle due to unreliable value predictions in unscen states. We argue that this problem cannot be addressed with exploration alone, but requires more expressive and generalizable models. We demonstrate that behavior cloning with a conditioned diffusion model trained on successful trajectory generalizes better to states formed by new combinations of seen elements than traditional RL methods. Through experiments in maze, driving, and multiagent environments, we show that conditioned diffusion models outperform traditional RI. techniques and highlight the broad applicability of our problem formulation.
1 Introduction
In many real-world decision makinngaannnggn tasks, environments can be broken down into combinations of funda- mental elements. For instance, in self-driving tasks, the surrounding environment consists of elements like bicyclos, pedestrians, and cars. Due to the exponential growth of possible element combinations, is im- practical to encounter and learn from every possible configuration during training. Rather than learning how to act in each unique combination, humans instead learn to interact with individual elements such following
or avoiding pedestrians and then extrapolate this knowledge to unscen combinations of elements. Therefore, it is important to study the generalization to unseen combinations of known elements, hereafter referred to as the out-of-combination (OOC) generalization, and to develop algorithms that can effectively handle these unseen scenarios.
Despite the success reinforcement learning (RL) in decision- makiing tasks, many existing RL algorithms, particularly offline settings, struggle perform adeptly under state distribution shifts between training and testing, which typically occur when the learned policy visits states that differ from the data collection policy at test time (Levine al. 2020 Kakade & Langford 2002 Lyu et al.) 2022 Schulman, 2015). While there have been works studying this problem, most of them either (1) focus distribution shifts where the training and testing sets share the same support but different probability densitics, without accounting for the presence of entirely new and unseen element combinations (Finn et al. 2017 Ghosh et al. 2022), or (2) allow unseen elements in test combinations, which makes the problem ill-posed without introducing other potentially unrcalistic assumptions (Song et al. 2024 Zhao et al. (2022.. As result, these works have failed recognize and address the critical challenge of generalization to unseen combinations of seen elements and therefore fail to capture and compose existing knowledge for these fundamental elements.

---

Published in Transactions on Machine Learning Research (12/2025)
In Support
Out of Support
Sample Space
Distribution Shift
Out of Combination
Unseen Base Object
Probability Mass
P
leat
P main
P
Training samples
lest
Test
Samee Support
Out of Combination
Overlápping Support
Non-overlapping Support
Sample Space
Unseen Base Object
Figure 1: Different forms of out-of-distributio states. are seen base elements and unseen base element. Theii combination forms the sample space. Classic distribution shift assumes states to have the same support but different probability density. We study generalization for out-of-combinationn states in this work, where test time state distribution has different and possibly non-overlapping support compared to training statos,
In this work, we directly tackle the problem of state combinatorial generalization in decision-making tasks, where testing states consist of unseen combinations of elements encountered during training. As illustrated in Figure our task differs conceptually from traditional distribution shift problems. Unlike simple distribution shifts, where the testing set remains within the support of the training set, our proposed task requires algorithms to handle out-of-support states that are never seen during training. This makes our problem both more challenging and more representative of real-world scenarios. At the same time, our OOC setting is better defined than the unconstrained out-of-support (OOS) setting, where testing states may include completely arbitrary unseen elements and therefore is inadequately formulated and intractable without other potentially impractical assumptions such as the existence of state distance metries (Song et al. 2024 isomorphic Markov decision processes (MDPs) (Zhao et al. 2022) By focusing new combinations of known elements, our setling strikes balance between real-world applicability and traclability, making it more suitable for standardized evaluation and formal analysis.
To facilitate this study, we first provide formal definitions of state combination and OOC generalization. Wo then demonstrate the challenge of this task by showing how traditional RL algorithms struggle to generalize in this setting due unreliable value prediction, and the need for more expressive policy. On the hunt for suitable solution, draw inspiration from the lincar manifold hypothesis in diffusion models (Chung et al. 2023 He et al. 2024b) and recent advances in combinatorial image generation (Okawa et al. 2024 and present conditional diffusion models as promising direction by showing how they can naturally account for the combinatorial structure of states into the diffusion process, enabling better generalization in OOC settings.
Experimentally, we evaluate the models on three distinct different RL. environments: maze, driving, and multiagent games. All three settings are easily adaptable the OOC generalization problem using existing RL frameworks, demonstrating the broad applicability of the combinatorial state setup. We demonstrate behavior cloning (BC) with conditioned diffusion model outperforms not only vanilla BC and offline RL methods like CQL (Kumar et al. 2020) but also online RL methods like PPO (Schulman et al., 2017) in zero-shot OOC generalization. To explore factors contributing to its generalization, we visualize the states predicted by the conditioned diffusion model. Our results demonstrate that the model effectively captures the core attributes of each base element and accurately composes future states by integrating these fundamental attributes. We demonstrate that, while exploration is commonly used enhance model generalization, 00C generalization relies instead the use of more expressive policy.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Related Work
2.1 Generalization in RL
Meta RL Meta RL is often seen the problem of "learning to learn", where agents are trained on several environments sampled from task distribution during meta-training and tested environments sampled from the same distribution during meta-testing (Yu et al. 2020 Finn et al. 20170. 0.. In the K-shot meta-RL. setting, the model can interact with the testing environment K times during meta-testing time to update the model using reward (Finn et al. 2017 Mitchell et al., 2021 Li et al.) 2020 Rakelly et al. 2019.. Our setting
is different from Meta-RL as the training and testing environments are sampled from different distributions and conditioning provided while restricting K to zero.
Unsupervised RL Unsupervised RL aims to acquire broadly useful skills without access to task-spocific rewards and then leverage these skills to accelerate or improve performance on downstream tasks with extrinsic rewards, often via an initial unsupervised "exploration" phase followed by reward-supervised fino-tuning phase (Eysenbach et al. 2018 Laskin et al., 2022: Jaderborg et al. 2016). While our sctting also assumes reuse of previously learned information and locuses generalization, it differs in that we operate under a fully offline, reward-agnostic setup, without access to or reliance on any implicit reward signals during training.
Zero-shot domain transfer The problem of zero-shot domain transfer assumess that the model is trained and tested on different domains that might have some similarities but are sampled from different underlying distributions (Kirk et al. 2023). The most widely used technique domain randomization, approaching this problem by producing a wide range of contexts in simulation (Kirk et al., 2023 Mehta et al., 2020). However, it commonly assumes that information about the testing environment not accessible (Mehta al., 2020)
and focuses more on sim2real problems (Kirk et al., 2023), whereas we assume test time information given through conditioning but restricting the training set to have narrow coverage.
Subtask and Hierarchical RL These two settings focus on learning reusable skills that be sequenced to complete long horizon
Druunn Training Trajectory
Subtask 1: Follew while avoid bike
Subtask Walt for pedestrian while avoid bike
d
tasks (Parr & Russel]
1997 Lin et al. 2022
Subtask Avaid tree while avoid bike
1:
Dietterich 2000 Nachum et al.) 2018 Joth-
Training Trajectory 2
Follow car while avoid a bike
imurugan et al. 2023 Bakirizis et al. 2024
The concept of compositionally
also a key component in subtask learning, where differ- ent sub-trajectories or intermediate goals are composed together to better perform long horizon task (Jothimurugan et al. 2023 Lin
Sublask 3: Avoid la tree while avoid
dF
Subtaak
Subtask leamnng
Walt for padastrion while avoid bika
D
rest
Avoid padestrian
State combinatorial genersizatiion
ct al. 2022 Bakirtzis et al.) 2024 Mendez
et al. 2022).
We would like to note the dif- ference between compositionally in trajectory stitching and our definition of state composi- tion, where subtasks in trajectory stitching are often data supported as shown in Figure
Visualization of states in trajectories for training, subtask learning, and state combinatorial generalization. Subtask learning in- volves stitching together subtask in the training trajectory 2 with subtask 2 in trajectory 1. Combinatorial generalization involves si- multaneously avoiding tree and waiting for a pedestrian. Fach of those two elements appeared in the training states but had never been combined.
Figure 2:
2.2 Combinatorial Generalization
In Computer Vision The closest line of work to ours is combinatorial generalization for image generation where the model needs to learn new combinations of a discrete set of basic concepts like color and shapes and generalize to unseen combinations (Wiedemer et al. 2024 Okawa et al. 2024 Schott et al. 2021 Hwang cet al. )023). This problem is often approached with disentangled representation learning (Liu al. 2023
Schott al. 2021 with models like VAE but little evidence shows they can fully exhibit generalizatio
ability (Schott et. al. 2021 Montero et al. 2020). Okawa al. (022)) studied the capabilities of conditioned
3

---

Published in Transactions on Machine Learning Research (12/2025)
diffusion models on synthetic shape generation task and showed that their composition ability emerges with enough training, first to closer concepts, then to farther ones. However, we would like to emphasize the difference between image generation and decision-making tasks, where pocision-making in the classic RL setting includes ovaluating candidate actions, interacting with unpredictable environments, and handling future stale distribut ion shifts caused by previous action choices and enviromment. (Levine el al., 2020 different from image generation where is one step and does not interact with the environment.
In RL Song et al. (0024) addresses the problem of generalization to unsupported states by decomposing it into the closest state in the training set and their difference, which requires the existence of distance function to map the unseen state back to data supported region to ensure conservatism. However, we do not assume there exists a distance function between states and we do not explicitly encourage the model to be conservative. Zhao el al. 2022) uses an object oriented enviromment to study compositional generalization by learning the world model under the assumption that different combinations have isomorphic MDPs and objects are replaccable with cach other. However, we do not assume our MDPs be isomorphic, as each object in our setup possesses unique attributes that are non-transferable, leading to the emergence of complex underlying modalities. To the best of our knowledge, we are the first to investigate the problem of generalization to unsupported states with novel combinations of basic elements, without relying on mapping unseen states back to data-supported regions.
2.3 Diffusion Model for Decision making
Diffusion models emerged as a popular architecture for decision-making tasks and demonstrated superior performance compared to traditional RL algorithms, esseciial on long-horizon planning tasks (Janner al.
2022 Wang al., 2022 Liang et al.;
2023). Some following work further studied 2023a Mishra et al.
conditioned diffusion models (Chi et al. 2023 Ajay et al. 2022 ei al. 2023 and demonstrated their ability
to stitch trajectories with different skills constraints logether. Application in multi-task environment (He)
et al.) 2024a Liang et al.] (023b) and meta-learning setting (Ni et al.) 2023 Zhang et al. 2024) further demonstrate their ability to capture multi-modality information in the offline datasct.
3 Problem Formulation
In this section, we formally define the problem of state combinatorial generalization by providing definitions for state combination and identify out-of-commination generalization problem for generalization to different supports in the same sample space.
3.1 States Formed by Element Combinations
Following Wiedemer et al. 2024h. we first denote e E E to be a base element for an environment. base clement is defined to be the most elementary and identifiable element that relevant to the decision making task of interest. For example, in a traffic environment, the set E can be the set of vehicles that can occur in the euvironment such as {car, bike]; and in 2D maze environment, the set E can be the set of possible locations labeled by the æ, y-axis coordinate of the agent, i.c. R Suppose there are basc clements in environment. Since these elements are the fundamental components relevant to the decision making (ask, we can form a latent vector z = (#1, 22, 2n) combination of all rudimentary components appearing in this environment related to the decision making task.
2n) E Z = E", where 2 E E Vi E (1.....n) that represents the
Each element can also be associated with collection of attributes such as the color of the vehicle and the velocity of the agent. Attributes are components that are necessary for rendering the states but not essential for the decision making task. The rendering function f(z, (ri, P2......... . rx)) then maps the latent and the attributes to state S. In the traffic environment example, is equivalent to reconstructing the cars and the bikes given their colors and positions, etc. All reconstructed basc clements collectively determinc a state 8. Concretely, we provide the following definition:
4

---

Published in Transactions on Machine Learning Research (12/2025)
Definition 3.1 (States and latent vectors). For any state with base elements in state space S and rendering funclion J, we have 8 f(z, (r1, r2..... rn)) where the corresponding latent vector in latent space Z = for is 2 = (z1, 22....... zn) where 2, E E for L,,,,
With our definition of base elements and states, the combinalorial properly of states naturally follows the composition of different base elements in the latent space.
Notice that in practico, for the same environment, one can define different base clement sets depending on the desired granularity of the task. In addition, since we usually can only obtain observations of the stales, in practice we can only extract the empirical latent. vector ž from the observation.
3.2 Generalization on Probability Space Support
Since we have identified the fundamental elements of the state in the target decision making task, we formulate the distribution of states with the probability spaces of latent vectors. When our base element set is discrete, finite, and countable, the probability mass function (PMF) p can directly ascribe a probability to sample in Z. Then we can define the corresponding probability space as
Definition 3.2 (Probability space for discrete latents). Define the sample space Z as the set of all possible 2. a-algebra Σ = 22 is the power set of Z. Z + [0, such that Exezp(z) 1 is the PMF. Then the probability space over the latent vector z can be defined as P (Z.
When Z is continuous space, we can also have the correspnding definitions.
Definition 3.3 (Probability space for continuous latents). Define the sample space Z the set of all possible a-algebra Σ = B(Z) is the Borel set of Z. Z + [0, such that Seez Jzezp(z)dzz p(=)dz = is the probabilily dense function (PDF). Then the probability space over the latent vector can be defined as P= E))
The support of P E,) can then be defined supp P
Z Z: 0}.
State combinatorial generalization, or 00C generalization, is then defined as generalizing to latent prob- ability space with different support. Denote the latent probability space of training states as Ptroin = (Z, Etrein, Etrain.Pirain) and testing states as Ptest = (Z, Etest: Ptest), then combinatorial generalization assumes supp (Prrai)) * (tes() That is say, combinatorial generalization in state space requires generalizing distribution of latent vectors with different, and possibly non-overlapping support (Wiedemer et al. 2024). Whereas traditional distribution shift in RL normally assumes different PMF PDF (Ptrain + Prest); shown in Figure
3.3
Constraint for OOC Generalization
One erucial assumption made by OOC generalization is that all base elements are seen at training time. Recall the latent vector z =
zn) where z; E for i= I,.., n represent the appearing base elements for a given state. This indicates that the marginal distribution p(2,) > for all z; al training time, or equivalently the training probability space has full support over the marginals. For discrete latent spaces, this also implies that every base element that appeared in the training state space would appear least once in one latent feature z. To ensure full support of base elements, the union of marginal supports al test. time should be subset of that at training time. Finally, to test generalizability, we assume supp(Ptrases & Z, i.e. the training probability space doesn't have full support on the entire latent space.
Constraint 3.4 (Combinatorial support). Given probability spaces P = (Z, Σp.p) and Q - (Z, 4),4) over latent vector (21, 22, 22,,... .... 2n) E Z where E E for i 1,.., P has full combinatorial support for Q if: o E E q(2,) > 0} C o E p(z,) > 0}.
4 Why traditional RL fails
Most RL algorithms include estimating the expected cumulative reward of choosing specific action given the current state (Schulman et al. 2017 Kumar al.] 2020 Haarnoja al. 2018 We demonstrate the
5

---

Published in Transactions on Machine Learning Research (12/2025)
estimation of value functions is problematic given unsupported states and this not be solved by more exploration or more training data in this section.
4.1 RL and expected reward estimation
Most deep RL algorithms rely on learning a Q or Value function, which takes the current state as network imput and predicts the expected future reward (Schulman et al.) 2017, Haarnoja et al.) 2018). Since states with unseen composition are unsupported and fall within the undertrained regions of the neural network, the value prediction is highly unreliable. This affects both value-based methods that directly choose the maximum action with erroneous Q value and policy-based methods that update the actor with an erroncous value prediction. We plot the expected Q-values learned by CQL alongside the actual return-to-go in both lailed and success scenarios in Roundabout environment. (Leurent 2018) (Section 7.1 when presented with OOC states in Figure 3 The grey dashed line the expected Q-values the model predicts for in-distribution (ID) states. When comparing the Q-value predicted on in-distribution and OOC states:
One observation about inaccuracies of value prediction be made: Q function shows signs of memorizing, which assigns simälar Q values for both ID and 00C stales.
Unsupported States
0.00)
10
10
The problem of distribution shift is even more challeng- ing for offiine RL, where suboptimal action prediction can cause future states deviate away from the train- ing set and lead to compounding errors (Levine al. 2020). This problem can often be mitigated with wider training state distribution under the assumption that test time states are sampled from a distribution with different probability density but same support. However, since new states with different object combinations are oul of support of the training environment, neither using a more exploratory policy collecting more training trajectories for offline RL reach these O0C states, and thus the issue canmot be fundumentally resolved. We need a pol- icy with better generalization to unsupported states to achiove zero-shot generalization in this problem.
8-
8
6-
6
4-
4
2
2
0
0
Success
Crash
Expected Q
RTG
Figure 3: Expected Q value predicted by CQL and in
OOC and ID states actual returu-to-go (RTGG) 00C states in Roundabout environment.
5 Why Diffusion Models Generalize Better
In this section, we first introduce diffusion model notations and then provide a proof sketch of why diffusion models can generalize to 00C states.
5.1 Diffusion Models
Diffusion models are among the most popular methods for density estimation. Ho et al. (2020) proposed DDPM to model the data generation process with a forward and reverse process. In the forward process, noise is added to corrupt data æ, iteratively for T timesteps towards a standard Gaussian distribution. The target of diffusion modeling to learn the reverse process pe( (z,) == (2 4 t). (x t)). This way, sample from the data distribution by first obtaining Gaussian noise æ, and then iteratively sampling from po(z)-1/z;). With reparametrization trick, we train model cg predict the noise at each timestep f, and gradually denoise using update rule 2 vai x vT-a,-o? co( +orGe where 4
1-ax
(), with variance schedulers ag, at. Given the same pretained diffusion model, one can also perform x # -I-VI-areal /oi
DDIM sampling (Song et al. 2020al xt-1 = /ac-1
+ VI a=-1c( t) + orEt to enable
fast sampling.

---

Published in Transactions on Machine Learning Research (12/2025)
Song et al. (0200) formally established the connection between diffusion models and score-based stochastic differential equations (SDE). Interestingly, they discovered that each diffusson process has corresponding probability flow ODE that shares the same intermediate marginal distributions p(æ,, t) for all t. The trans- formation between probability flow ODE and SDE can be casily achieved by adjusting the random noise hyperparameter a DDIM sampling.
5.2 OOC Generalization in Diffusion Models
We now give a theoretical justification for OOC generalization by showing that well-trained diffusion model concentrates probability mass in-combination states (seen and 00C), with quantifiable density lower bound.
Since states are formed by combinations of base elements (Definition 3.1 with a well-constructed and encoded Z, we assume that the set of in-combination states (both supported and OOC states) concentrates on a lower-dimensional manifold M embedded the high dimensional ambient state space. As a result, we first adopt lincar manifold assumption (Assumption B.1), where M the lincar manifold that the in-combination states lie along.
To connect this formulation to the diffusion sampling process, assuine have access to a well-trained diffusion model (Assumption B.2) that: (i) contract the components that are orthogonal to the manifold and pull the samples towards the manifold (block-wise bi-Lipschitz with contraction), and (ii) the learned reserve mean should not mix the manifold orthogonal and on-manifold components (block preserving de- noising). With above assumptions, we present the following theorem that quantifies the lower bound of the probability density assigned a state:
Theorem 5.1. Under the linear manifold assumption (Assumption B.1 and let pa be a DDPM diffusion using paramctrization from Equation that is well trained (Assumption B.2), with some constant C, for all S
(- (-1WM((s - po)ll
| 188
po(8) 2 C Cexp
(1)
M)
2( )2
where AM S B.M < 00, S < are the block-wise bi-Lápschâtz constants for the on-manifold component and the orthogonal component respectively, WM and W1 are projec- tion operators onto the manifold and its orthogonal complement respectively, af and on the ef- fective variances along and orthogonal to the manifold, 140 is the learned noise-free center, and C = (2T) d/2 (IT (BMM/M)) /At )m=) )
Proof sketch: The proof proceeds by induction over diffusion timesteps ..... We begin with the base case at t T where ST (00, I), and establish the initial Gaussian density lower bound that factorizes along the manifold and its orthogonal complement via the projectors (WM, Wi). For the inductive step from timestep & to 1, we decompose the DDPM reverse process Sk-1 po k) +4 into two operations: (1) applying the learned demoising mean (- k), and (2) adding Gaussian noise σge. For operation (1), we prove with Lemma B.3 that the bi-Lipschitz property ensures that applying μ4 transforms the density bound with updated means and scales the variances. Moreover, the block-preserving property ensures that on-manifold and orthogonal components do not mix, maintaining the factorization. For operation (2), we can observe that adding the aforementioned Gaussian noise simply inflates along and orthogonal manifold components' variances by of. Combining (1) and (2) closes the induction and gives, at 0, block Gaussian lower bound centered at 14 (Equation D. We provide the full statement and the formal proof in Appendix B
Intuitively, this theorem suggests that, the lower bound of the density assigned at certain point by a well- trained diffusion model depends on how far is along the manifold from the model's noise-free frise-free center and how far it off the manifold. Bocause the contraction towards the manifold, off-manifold components incurs harsh penalty under the this well-train model, with likelihood dropping rapidly as the off-manifold component grows.

---

Published in Transactions on Machine Learning Research (12/2025)
6 Conditioned Planning with Diffusion
The theoretical analysis in the previous section provides promising signals for OOC generalization in diffusion models. However, also suggests that reliable generalization requires the linear manifold assumption and a well-trained diffusion model that can capture the characteristics of the manifold. While in environments like 2D maze, the lincar manifold assumption are linear subspaces), for more complex environments, this assumption does not necessarily hold true with- out powerful encoder that maps the element set to a proper embedding space. Moreover, the block-wise bi-Lipchitz with contraction and block preserving denoising assumption indicates that the diffusion model needs to be manifold (i.e., element combination) aware, which is also not guaranteed with vanilla uncon- ditional diffusion models. As a result, careful model architectural designs are required to ensure empirical performance.
be naturally satisfied (as 2D planes through the origin
We take inspiration from recent computer vision research, which provides evidence of the combinatorial generalization capabilities of conditioned diffusion models in more complicated data spaces: Aithal et al. (2024) identifies the phenomena where diffusion models generate samples out of the support of training distribution through interpolating different complex modes on a data manifold. Kadkhodaie et al. (2023) demonstrate generalization to unsupported data by showing two diffusion models trained on large enough non-overlapping data converge to the same denoising function. Okawa et al. (2024) showed that given different concepts like shape, color, and size in synthetic shape generation, conditional diffusion models demonstrate multiplicative emergence of combinatorial abilities where it will first learn how generalize to concepts closer to the training samples (i.e. only change one of color, shape, and size) and eventually adopt full compositional generalization ability with enough training.
Therefore, we propose to use a conditioned diffusion model with learnable element embeddings to tackle the OOC state problem for decision-men maaoong tasks. Following the trajectory-basel planning formulation of Janner et al. (2022). we train a conditional diffusion model denoise (predict) future state-action pairs conditioned on the current observation. Importantly, this training is carried out without reward guidance; instead, it is trained via behavior cloning on successful demonstrations collected by PPO (Schulman et al. 2017) model in the in-distribution environment, capturing the relationship between states, successful actions, and future states. Conditioning is applied in two complementary ways. First, we enforce consistency of the generated trajectory with the current observation by replacing the corresponding component of the predicted trajectory with the actual environment observation at every denoising step. Second, to enhance this modeling, we incorporate a poss-attention layer after each residual block in the Diffusion UÑet, allowing the predicted trajectory directly attend to the latent vector z contained in the observation of the current timestep. Consequently, when conditioned on novel combinations of base elements, the generated trajectorics remain dynamically consistent with the underlying dynamics (MDP), facilitating robust generalization to novel scenarios. For deployment, the first action in the generated trajectory is then used to step the environment (Appendix C.4). Detailed architecture of the model can be found in Appendix C.3 and the planning process is described in Algorithm
Note that in practice, our conditioned diffusion model and the element embeddings are trained end-to-end, and thus the resulting learned model does not necessarily fit the theoretical assumptions above. However, we will show in the next section, empirically this model demonstrates successful 00C generalization in various RL. environments.
7 Experiments
The primary goal of our experiments to answer the following questions: (1) (Wide applicability) Does the state-space of different existing RL environments exhibit a compositional nature? (2) (Advantages) What are some interesting features conditional diffusion models have that contribute their performance when generalizing to O0C states? (3) (Conditioning) Does conditioning help with 00C generalization?
8

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 1 Planning with Attention-based Composition Conditioned Diffusion Model
Input: Diffusion model cø, compositional elements extractor r, learnable embedding function h, classifier- free guidance scale A, state dimentionality ds, initial observation o, environment simulator env while not done do Initialize 8, ~ 10)
c ct r())
Extract observed compositional information Obtain element embedding
< h(c)
for t + ..... do
8:1: ds, 0) to
(Cond state) Replace the first denoised state with observed
4 = (1 + ))co( 2,t) - tc)
b (Cond 2) Classifier free guidance
#t-1 = var
-
8: - L-a Et) + toEt, where Et ~ 10,,1)
VI-a.
end for
zo [ds :, 0] 0 - env.step(a)
Extract action
end while
7.1 Single-agent Environment
HighwayEnv [Leurent 2018 is a self-driving environment where the agent needs to control a vehicle to navigate between traffic controlled by predefined rules. We specifically look at the Roundabout environment with two types of traffic: cars and bicycles (Visualization in Appendix C.
Environment
State this environment is a composition of four environment vehicles that are either cars or bicycles and the ego agent, which is always a car. Environment observation contains observability, the locations and speed of the ego and surrounding agent, and whether this agent. is car or likee (Condilioning). During training time, the environment will only generate traffic of all cars or all bicycles with equal probability. During test time, environments will generate mixture of cars and bicycles (detailed setup in Appendix C.ren Cars and bicycles have different sizes, max speeds, and accelerations, leading to different behavior patlerns. This is an instance of generalizing to OOC states with non-overlapping support.
Results As shown in Figure 4 the conditional dif- fusion model has almost half the number of crashes and higher reward when zero-shot generalizing to states with mixture traffic. Results of models with different sizes are shown in Table to eliminate the concern of performance gain due to larger model sizes. Since we train the diffusion model exclusively on successful PPO trajectories, the training state distribution for diffusion much narrower compared that of other online mothods. This is particularly interesting since it is widely acknowledged that on- line models have better generalization compared to offline models (Levine et al. 2020
Roundabout PPO3D
10
50
8
40
6
30
4
20
10
-2
PPO ID
0
0
Crashes (4)
Reward ( (1)
Diffusion (Ours)
PPO
BC
CQL
Takeaway 1: Conditional diffusion models, trained offline dataset with narrow state distribution with full combinatorial generalization support, have better zero-shot generalization performance to OOC states compared to line RL trained in the same environment.
Total number of crashes and average reward for BC(MPP), PPO, CQL, and diffusion model in the testing envi- ronment.
Figure 4:
9

---

Published in Transactions on Machine Learning Research (12/2025)
SMACv2 3v3
SMACV2 5V5
20
20
improvement (%)
Improvement (%)
0
0
-2
- 20
-40
400
-60
60
-80
880
Simple
Hard
Simple
Hard
Random
BCQ
BC
CQL
Diffusion (Ours)
Figure 5: Relative improvement. % compared MAPPO SMACv2 scenarios: 3v3 and 5v5. Conditional Diffusion shows large and consistent improvements over the MAPPO and other BC and RL baselines, especially in the hard scenario, where train teams with the unit type only but test random team compositions.
7.2 Multi-agent Environment
The StarCraft Multi-Agent Challenge (SMAC/SMACv2) (Sanvelyan et al. 2019 Ellis et al. 2022 is multi-agent collaborative game that takes several learning agents, each controlling single army unit, to defeat. the enemy units controlled by the built-in heuristic AI. This benchmark is particularly challenging for its diverse army unit behaviors and complex team combinations, which enable diverse strate- gics like focus fire and kiting enemies to emorge (Ellis et al. 2024). Each agent's observation includes health, shield, position, unit type (Conditioning) of its own, visible teammates, and enemies.
Environment
We treat one agent the ego agent, and consider its teammates and enemies part of the environment. Then states can be naturally seen as compositions of the unit types in a particular playthrough. We expect the ego agent to generate different policies when playing with against different types of units, and we oim to test 00C generalization by changing the unit composition in the environment. Since use MAPPO YYu et al. 2022) for data collection, we report the performance gain/loss compared to MAPPO as shown in Figure
To treat the leammates and enemies of one particular agent as environment and change their combination, we control one unit with a conditional diffusion model and let MAPPO control the rest of its teammates.
Setup The unit types in this experiment are Protoss.Stalker, Protoss. Zealot, and Protoss. Coloosss,,ssu, referred to as a, b, e respectively. We evaluate on two OOC scenarios: (1) (Simple: Different but overlapping support): Train the model on randomly generated combinations (ABC) all units and test it where all the units on the team have same type (AAA), (2) (Hard:: Non-overlapping support): the opposite scenario, where we train teams with only unit type (AAA), but during test-time units (ABC). More information about setup could be found in Appendix C.8
any composition of these three
Results As shown in Table and Table 10 MAPPO performance drastically dropped in the hard 0OC scenario by 55.2% for 5v5 and 33.3% for 3v3. If we substitute one agent generated by MAPPO with conditional diffusion, the success rate can be improved by 16.7% for 3v3 and 23.1% for 5v5 in hard 00C scenario shown in Figure]
Takeaway Multi-agent RL, viewed from the perspective single ego agent, naturally requires combinatorial general- ization to vollaborate//ooupeeue with different agent types.
7.3 How Do Conditional Diffusion Models Generalize to 00C States?
To see how diffusion models generalize OOC states, we render the states predicted by the diffusion models given different conditionings with the same current. state, as shown below in Figure 6
10

---

Published in Transactions on Machine Learning Research (12/2025)
Green: Ego player
Blue: Teammate
Red: Enemy
Zealot
Time
Colossus
o
Stalker
Dark: Shield
Light: Health
Time
Figure 6: Rendering of future states predicted by the diffusion model given different conditionings. 'he grey box the current state. Blue backgrounds conditional Squares (long attack range) and pink backgrounds are conditioned all circles (short attack range). Smaller sizes represent less shield and health. More examples shown Appendix C.8.6
We can see that player lype conditionings determine the wnil lype of agent predicted by the diffusion model and also their behavior pettern. Whereas the current state determines other attributes like initial location and health. Different player type conditionings will lead to different strategies. The circle unit has attack range and the square unit has attack range 6. For units with short attack ranges, the optimal strategy is to approach their enemies before initiating an attack. Conversely, agents with large attack ranges are advised to attack their enemies from a distance ensure their own safety. S that if condition on all circles, the diffusion model thinks players will form cluster and if condition the players to attack each other from distance, aligning well with the optimal policy. This demonstrates conditioned diffusion models' ability to implicitly decompose states to learn underlying compositions and capture multimodality of different unit behavior in the training data. It also demonstrates its ability to perform state stitching to accurately predict the underlying MDP for an unseen combination.
all squares, it will predict
Takeaway 3: Conditioned diffusion models show significant promise by effectively decomposing and capturing modes of individual base elements and performing state stitching, which helps them generalize to 00C scenarios.
8 Ablations
In this section, we ablate over our design choices to (1) show the necessity of using the inductive bias of the latent vector as conditioning, (2) different model architectures to incorporate conditioning information
8.1 Necessity of Combinatorial Inductive Bias
We compare trajectories generated by the conditioned and unconditioned diffusion models in this section to demonstrate the importance of using the latent vector z that contains the combinatorial latent information as conditioning. In Maze2D (Fu et al., 2020), we formulate the navigation problem a one-step generation process where the diffusion model learns how to generate entire valid trajectory without rolling out the current action and replanning. Since there is only planning step in this process, the generated trajectory can be seen as the "state" in this sotting, where unseen trajectories correspond to unseen states instcad of time-horizon trajectory stitching. The inductive bias we use is every training trajectory will pass through three waypoints that equally slice the trajectory. In this case, the set of all waypoints forms the base element set and their combination the latent vector that determines the shape of a generated maze trajectory. During training, we extract three points that equally slice the trajectory and use them as conditioning. During test time, specify new combination of three waypoints want the generated trajectory to pass.
11

---

Published in Transactions on Machine Learning Research (12/2025)
(b) ID (Unconditioned)
(c) OOC (Unconditioned)
(d) OOC (Conditioned)
(a) train traj
Figure 7: Trajectories generated Maze2D for large (a) Samples from the training set. (b) Trajectories generated by the unconditioned diffusion model given in distribution start and end positions. (c) Trajectories generated the unconditioned diffusion model on (black dots) as conditioning. For results in the medium please refer Appendix C.5.1
and end positions. (d) Trajectoiie generated conditioned diffusion model using waypoints
We see that the unconditioned diffusion model successfully generated a trajectory the start and end positions are in the training (Figure 7bp, but failed to do so given unseen start and end positions (Figure [cl. We compare the trajectories generated by the unconditioned and conditioned diffusion model given unseen start and end positions in Figure 7c and 7d The trajectory generated by the conditioned diffusion model accurately follows the given OOC waypoints and is drastically different compared to the one without conditioning. This highlights the conditioned diffusion model's ability to generalize to OOC conditioning and its strong accuracy in adhering to the provided conditioning.
8.2 Model Architecture: Attention vs Concatenation
We also ablate over different model architectures: (1) concatenating the latent vector z with diffusion's 's time embedding,
(2) performing cross attention between z and output of each Unet residual block (Architecture in Figure g Figure shows our result: in general conditioned diffusion models outperform unconditioned ones and attention outperforms concatenation in of cases.
SMACV2 3v3
SMACv2 5v5
improvemen (%)
20
Improvement (%)
10
10
-10
-10
-20
20
-30
30
Simple
Hard
Simple
Hard
No conditioning
Concat
Attn
Figure 8: Improvement percentage over MAPPO for different types of conditioning in SMACv2.
Ablation Takeaway: Conditioned diffusion models, provided with information about the new composition of state, are able generalize out-of-combination conditioning and accurately adhere the provided conditioning. Also, cross-attention with the condition outperforms simply concatenating with the time-embedding
9 Conclusions
Despite the success of traditional RL models decision-making tasks, they still struggle generalize unseen state inputs. Most existing work focuses on RL generalization under the assumption that general- ization to different probability density function with the same support. However, take it further and
12

---

Published in Transactions on Machine Learning Research (12/2025)
study the problem of generalization to out-of-support states, out of combination in particular, hoping the model can exploit the compositional nature of our world. We showed how this task is challenging for value- based RL and also how conditioned diffusion models can generalize to unsupported samples. We compare the models in different environments with detailed ablation and analysis, demonstrating how cach these classic environments can be formulated as a state combinatorial problem.
However, one limitation of our setup is we model combinatorial generalization in state space as a combination of base elements, which is valid for many real-world applications but does not cover all cases. Additionally, the model has difficulty with zero-shot generalization to unseen base objects. Another constraint is efficiency, planning with diffusion models in stochastic environments requires denoising trajectory at cach planning step, which can be computationally intensive. Various approaches have been proposed to accelerate the sampling process in diffusion models via advanced ODE solvers or knowledge distillation techniques (Song ct al. 2020a; Karras et al. 2022 Kim et al.) 2023 Song al. 2023). Incorporating these works into diffusion planning would be a promising future direction to address rollout inefficiency.
Broader Impact Statement
Developing data-driven decision- decision-making models carries the risk of generating inappropriate or harmful ac- tions. This work presents a conditioned model that can be manipulated through carcfully forged conditioning, potentially leading to malicious actions. Our experiments were conducted in simulated environments, and testing this model in the real world without safety restrictions could be dangerous.
Acknowledgments
This work was supported in part by the U.S. Army Research Office and the U.S. Army Futures Command under Contract No. W519TC-23 -3---00,, and also by DSTA. This work was also supported in part of the ONR grant N000142312368.
References
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bernhard Schölkopf, Manuel Wüthrich, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. arXio preprint ar Xiv:20000 Xix:22000002296, 2020.
Sumukh K Aithal, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. arXiv preprint arXiv:2406. 09358, 2024.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-making? ar preprint arXiv:2211, 1...2211.110 1..15577, 2022.
Georgios Bakirtzis, Michail Savvas, Ruihan Zhao, Sandcop Chinchali, and Ufuk Topeu. Reduce, reuse, recycle: Categories for compositional reinforcement learning. arXiv preprint arXiv:2408.138 13376, 2024.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303. 04187, 2023.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint 0tf::233300575.. 2023.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International conference on machine learning, pp. 1282 1289. PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048 2056. PMLR, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 04113099-33060 2020.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-300 2000.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Amuj Mahajan, Jakob N Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforce- ment learning. arXiv preprint ar 0r:::22212.074553 2022.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Fo- erster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Benjamin Eysenbach, Abhishek Gupla, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv.1802.06077, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135, PMLR, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Dibya Ghosh, Anurag Ajay, Pulkit. Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference Machine Learning, pp. 7513 7530. PMLR, 2022.
Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arX:::2010.006 06740, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11 11956, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision trans- formers under data augmentation. Advances in neural information processing systems, 34:3680-3693, 2021.
Haoran Hc, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuclong Li. Diffusion model is an ellective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36, 2024a.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference on Learning Representations, 2024b. URL net peet/foruumidd0oBB0Loooo
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
Shiyu Huang, Wentse Chen, Yiwen Sun, Fuqing Bie, and Wei-Wei Tu. Openrl: A unified reinforcement learning framework. arXiv preprint arXiv:2312.16188 2023.
Geonho Hwang, Jaewoong Choi, Hyunsoo Cho, and Myungjoo Kang. Maganet: Achieving combinatorial
generalization by modeling group action. In International Conference on Machine Learning, pp. 14237 14248. PMLR, 2023.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnocki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 2016.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robolics and Aulomalion Letters, 5(2): 2020.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:220.. 099911 2022.
Kishor Jothimurugan, Steve Hsu, Osbert Bastani, and Rajeev Alur. Robust subtask learning for composi- tional generalization. In International Conference on Machine Learning, pp. 15371 15387. PMLR, 2023.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. arXiv preprint arXiv:190..00 01978, 2019.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models ariscs from gcometry-adaptive harmonic representation. arXiv preprint Xrxv:22300 02557,, 2023.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceed- ings of the Nineleenth International Conference on Machine Learning, pp. 267 267-274, 274, 2002.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.cm 022799. 2023.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76:201-2660 264, 2023.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Aduances Newral Information Processing Systems, 33:1179-1191) 2020.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Railcamu, Marco Selvatici, Edward Grefen- stette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Pro- cessing Systems, 33:7671-7684 2020.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, MA NYU, Aravind Rajeswaran, and Pieter Abbeel. Contrastive intrinsic control for unsupervised reinforcement learning. In Proceedings of the 36th Interna- tional Conference on Neural Information Processing Systems, pp. 34478 34491, 2022.
Edouard Leurent. An environment for autonomous driving decision-making. eleurent/highway-enn 2018.
https://github /
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.014 016443, 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi- task batch reinforcement learning with metric learning. Advances in Neural Information Processing Sys- lems, 33:6197 6210, 2020.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hicrarchical diffusion for offinne decision making. In International Conference on Machine Learning, pp. 20035 20064. PMLR, 2023.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXio:2302.01877, 2023a.
Zhixuan Liang, Yao MU, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldiffuser: Interpretable hicrarchical planning via skill abstractions in diffusion- based task execution. arXiv preprint arXi::2312.1133 2023b.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Yixin Lin, Austin S Wang, Eric Undersander, and Akshara Rai. Efficient and interpretable robot manipu- lation with graph neural networks. IEEE Robolics and Aulomalion Letters, ((2))2740- 2747, 2022.
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplct: An open challenge for intervention-centric causal representation learning. In Conference on Causal Learning and Reasoning, pp. 553 573. PMLR, 2023.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offinee reinforcement learning. Advances in Neural Information. Processing Systems, 35:1711-1724, 2022.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowl- ing. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523 562, 2018.
Oier Mees, Lukas Hermann, Erick Gosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for lori-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7334, 2022.
Bhairav Mehla, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain random- ization. In Conference Robot Learning, pp. 1162- 1176. PMLR, 2020.
Jorge A Mendez, Harm van Seijen, and Eric Faton. Modular lifelong reinforcement learning via neural composition. erXiv preprint arXiv:2207.00429 2022.
Utkarsh Aashu Mishra, Shangjic Xue, Yongxin Chen, and Danfci Xu. Generative skill chaining: Long- horizon skill planning with diffusion models. In Conference on Robot Learning, pp. 2905 2925. PMLR, 2023.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta-reinforcement learning with advantage weighting. In International Conference Machine Learning, pp. 7780 7791. PMLR, 2021.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jelfrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2020.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advonces in neural information processing systems, 31, 2018
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang, Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pp. 26087-26105. PMLR, 2023.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804 087202200, 2018
Maya Okawa, Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka. Compositional abilities emerge mul- tiplicatively: Exploring diffusion models on a synthetic task. Advances in Neural Information Processing Systems, 36, 2024.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXie preprint 1810. 12282, 2018.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. Advonces in neural information processing systems, 10, 1997.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stablo-basclines.. Reliable reinforcement learning implementattons. Journal of Machine Learning Re- search, 22(268):1-8, 2021. URL httpp:/jjmlr..... http:/rrrrr.pppp////////---111 html
16

---

Published in Transactions on Machine Learning Research (12/2025)
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards generalization and simplicity in continuous control. Advances in neural information processing systems, 30, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-po meta- reinforcement learning via probabilistic context variables. In International conference on machine learning. pp. 5331-5340. PMLR, 2019.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The star- craft mulli-agent challenge. arxiv preprinl or 15::1902.044 2019.
Lukas Scholt, Julius Von Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bern- hard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not gen- cralize strongly within the same domain. arXw preprint arXiv:207.08821, 2021.
John Schulman. Trust region policy optimization. arXiv preprint arXi::1502.0544 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arv:: 70077007076333 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.024 025002, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXio preprinl arXiv:2011.18456, 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiw:2303001400, 2023.
Yeda Song, Dongwook Lee, and Gunhee Kim. Compositional conservatism: A transductive approach in offline reinforcement learning. arXiv preprint arXiv:2404. 04662, 2024.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint Stt::2008001990 2022.
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Newral Information Processing Systems, 11:24611-2462.4 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsoa Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection overfitting and generalization continuous reinforcement learning. Xix preprinl ar arxiv:1800 07937, 2018
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the A AAAT Conference on Artificial Intelligence, volume 38, pp. 16687-16695 2024.
Linfeng Zhao, Lingzhi Kong, Robin Walters, and Lawson LS Wong. Toward compositional generalization in object-oriented world modeling. In International Conference on Machine Learning, pp. 26841-26864. PMLR, 2022.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A Related Work
We introduced related problem sctup and generalization methods in Section of RL environments for generalization.
Attached below is a review
A.1 Gemeralization in RL
Environments Most RL environments that Lest. model generalization can be grouped into different reward
functions [Raeeswaran et al.
2017 Zhang et al. 2018
Rakelly et al.
2019
Finn et al. 2017 or transition
functions (Dennis ct al.)
2020
Machado et al., 2018
Packer et al.
2018
Zhang et al. 2018). goals or
tasks Finn et al.) 2017;
Yu et al., 2020), states (Nichol et al. 2018
Cobbe et al.
2019 Juliani et al.
2019 Küttler et al. 2020
Grigsby & Qi 2020; Hansen et al. 2021 Mees et al. 2022 Cobbe et al. (020).
For environments with different state distributions, randomization (Grigsby & Qi 2020) and procedural
generation (Nichol et al. 2018 Küttler al. 2020 Cobbe et al. 2020) widely used to generate new
states. Some rissed environments (Juliani et al. 2019 Hansen et al. 2021 Mees et al. 2022) also
different rendering themes or layouts to generate unseen obsorvations, more targeting sim2rcal problems. For robotics benchmarks like Metaworld (Yu et al. is shared between tasks like open a door and open drawer is ambiguous (Ahmed et al.) 2020). Also, benchmarks like Franka Kitchen (Gupta al.
2020 and RLbench (James et al. 2020), how much structure
2019) focus on composing tasks time horizons, requiring
the model to concatenate trajectorics corresponding different subtasks. However, despite the large volume of generalization benchmarks, there is benchmark designed for stale combinatorial generalization to best knowledge.
18

---

Published in Transactions on Machine Learning Research (12/2025)
B Proof of Theorem 5.1
In this section, we provide the full statement and proofs of Theorem 5.1. First, establish the following assumplions.
Practically, when the element set is discrete, an embedding ø E" 7 Rm is applied to map a discrete reprosentation of latent elements to a continuous embedding space.
Assumption B.1 (Linear manifold assumption). Assume the states lie along a linear manifold M in the ambient slate space $ C TRd and the encoded latent space ((Z) C =R with m < is well constructed so that (Z) is (affine) isomorphic to M.
Here we can interpret the linear manifold M as the subspace containing all "in-combination" (i.e. in support or out-oo-commination)) data points. Recall we can parametrize the DDPM learned reverse process of a diffusion model as
po po( =
0,
(2)
where is pre-fixed noise schedule. Given Assumption B.1 we denote the projection operation onto the manifold M as WM and the projector onto its orthogonal complement M as Wi. We then further assume the diffusion model is well-trained, in particular, it should satisfy the following assumptions:
Asummption B.2 (Well-trained diffusion model). We assume access to well-trained DDPM diffusson model that satisfies:
Block-wiss bipsihcc with contraction: For all æ, S and all timesteps t, there exists constants
A. S B.M < xc,
0< A S B <1
(3)
such that
AMI]WM b)ll [W)(Ho(x noly t))l - B, W WM y)ll W+ (x - y)ll S [ (1)( u 1))ll S B (x y)ll
(4)
2. Block preserving denoising: For all timesteps t, (o((,t) preserves the block decomposition. L.e.
WMM = W (W =0
(5)
for all x E S.
The first assumption suggests that well-trained diffusion model will contract the components that are orthogonal to the manifold and pull the samples towards the manifold (i.e. towards "in-combination" sam- ples). The second assumption indicates that the learned reverse mean should not mix the two orthogonal components.
Based on these two assumptions, we can now prove the following lemmas that will be helpful for our proof to Theorem 5.1.
Lemma B.3. If Rdd has density lower bound
W (m - pi]1 [ (a: - 11)1
px(x) 2 d/2 (12 ( exp
(6)
M(0
2(0-)2
Let g be a function that satisfies block- wise bi Lipschitz with contraction and block reservation from Assump- lion B. with Lipschilz conslants A., BM A-, B+ and Y = xX)), then for conslant a we can have
[WM(y - g(1))11 [ - 1(1)))
py 2 c(2 d/2 M - B-ym- dexppppppppppp exp
(7)
AMM
2
19

---

Published in Transactions on Machine Learning Research (12/2025)
Proof. Because is block-wise bi-Lipschitz, 9 is also injective and differentiable almost everywhere by Rademacher's theorem. Then using the exact density transform, for almost everywhere y, we can have x' =g y) and
Px Px(x* (x*)
Py (y) = det (z(
Equation entails that in order to obtain a lower bound for pr(y), we should obtain a lower bound for px (x)) and an upper bound for det v g(x))1
From Equation t we can directly obtain
WM - 1)ll [ (x*
1))1
px(x*) 2 c(2m) 4/2 (/MooM ym exp
exp
(9)
2(o
2(a-)2
By the Liptschtiz condition we also know that
AMIWM(E'' w1] - E W - 9(())) [[ -11 [WM g(n))
AM
(10)
1 -x)1 ~ | (y g(H))
A-IWW --)1 - W W (g(x") I(x)i1
W - (V((y-
I s(4))1
px(x*) 2 4/2 ( (a cxp
(11)
AMM2
2( [oolA1 1
For det D)(x")), by block presorvation, we can have
Dg(x") W WMDg(=')WM [[Mi[g(> + +Av) ((x"))) s B-MII BMIA
0 W Dg(a") W
(12)
0
Now let M be unit vector and A be scalar, then by Lipschtiz condition, have
(13)
With A + we can have [WMDD((')el )u|l - BM Since this is true for all unit vector u e M, denoting XM as the singular values of W WyDDg(r WMM then for all [1.2...., m.)
BMg z sup [g
W mell 2 8,
(14)
|o| =1
Similarly, for singular values X of W x D',, we can have
B > sup [WiDge"") 2 8
25
(15)
||||| =1
for all jE [1,2,...., m].
Using Equation 12 15
we obtain
d-m M
det Dg(=*)l = det W WgD="">") det W, Dg
X
II f=1
(B
(16)
Putting everything together, we have
[[ 0(1))112 2(oMAMM)
W (y ((m))113
py 2 c(27))=-l) 4/2 ( BBM) BMM (a B 1 exp
(17)
A-12
20

---

Published in Transactions on Machine Learning Research (12/2025)
To further facilitate the proof of Theorem 5.1, we further define the recursive mean
pT O, Mt (o((e+1,tt
(18)
and the block variance scales as
M = o = or 1, Tf
(o²-)²² = (AM))
= (A!)
(19)
Now we are ready to prove our Theorem 5.1. Before that, we first provide the formal statement of Theorem 5.1.
Theorem B.4 (Formal statement of Theorem 5.1). Under Assumption B.1 and let Pe be DDPM diffusson using paramecrizalion from Equalion
2 that salisfies Assumplion B. with some consland C, for all 8 E S
(-
(s po)ll
|
ps(s)= Cexp
(20)
2( )2
Proof. Denoting po (ss t) as the density at time t for noisy sample st. we prove the statement by induction.
1. Base Case: At T, sy ~NOO,) I), therefore
[|
|W_sTll
pO(ST, T) (2T)) 4/2 exp
(21)
2
2
2. Inductive Hypothosis: Assume that for timestep k,
B. A
B. A
(-Ilmnss -M+)l1
[Wi(s - mx)l1
po(sk, k))
(2m)-d/2 d/2
II
mm
ym-d
(ot) exp
2(0
2( )2
M=b+1
(22)
3. Induction Step: For timestep k - 1, recall 8k-1 po k) + T&E for ~ _N(0, I), we can break this down into two operations: applying po(:; k) and adding Gaussian white noise.
Since μs satisfies Assumption
B.2 let y po(skk k), we directly apply Lemma
B.3 and obtain
(y) 2 c(22x) 4/2 (a BM m (a B ym m- exp (- [I ne (),,
| (y no k))
M M
2 A
(23)
where
11=b+11
T
B,M 1/
)
mm Bi jm-d Notice that this equivalent
A
[I -
I na pa k))
(y) 2 c(2iT)
( B. (oM4--m(a0)) (o B - ) exp
2( M A/1
2 A
T
B.M
B+
= (2T)) 8/2
-m
mm
(o -m )
A.M
A
1=
[[
14k-
1)
W (g l1l 1)
exp
2(
M A
M22
2 A
T
B.M
m B
(2m)
d/2
A 1 -m (o A ) md
A.M
A
t=
|| (WWM(y 111-1)11
| (g p4 1 1)
exp
2(0 M A,M2
2 A
21

---

Published in Transactions on Machine Learning Research (12/2025)
Adding zero-mean Gaussian white noise to y with variance of produces the desirable Sk-1, which increase the variance in the original Gaussian bound by of Therefore, we can obtain
pa( pa(sk-1, 1) (2 4/2
II 1=
B A
B A
m-d
(o (AK ( m/ (o)
(At) )3)
(24)
W (8- - PR-1) 1|
I| (s Mk- 1)1
exp
2 + (Ap (o)
2(0 + (A )2
(25)
BBM )m
Br
(2x) 4/2 4d/2
m-d
(ak-1) ( )m (a )
(26)
A.M
Ar
1=
[ 1k-1)11
[[ 1k 1)13 2 )2
exp
(27)
2 )2
Observe that since pe E M and so is block preserving, ir E M for all t. we can simplify the expression to be
B
B
I[ 1k-1))12
|W1s|
ko(8k-1,k-1) 2 (2x) 4/2
II
m m-
(ok-))-m((
(0 ( exp
A
A
2
2
(28)
Hence we conclude the induction.
Let C (2x) 4/2 (IT-=
B ym
-m0)) we arrive at
a.
WWm(s - Mo)ll | 18
ps( 2 Cexp
(29)
M) 2(
From this theorem above, we can make two observations: Firstly, the density assigned at a certain point depends on two factors: (1) how far it is along the manifold from the model's noise-free center, and (2) how far it is off the manifold. Secondly, since
A B, < 1, off-manifold components incurs much harsher penalty under the DDPM model, with likelihood dropping rapidly the off-manifold component grows. In other words, well-trained DDPM model concentrates in-combination states, including the OOC ones. How much density it assigns to each in-combination state is determined by how far away the state is from the model learned noisc-frec center.
22

---

Published in Transactions on Machine Learning Research (12/2025)
C Experiment Details
C.1 Hardware and Platform
Experiments are run a single NVIDIA RTX A6000 GPUs, with all code implemented in PyTorch.
C.2 Statistics
All mean value is obtained by running with three different seeds and calculated with numpy.mean(). All error bar is obtained by mumpy.std().
C.3 Model Architecture
The backbone for Unet is based on Janner et al. (2022). We add cross-attention blocks after each residual block, except for the bottleneck layers. Inputs to the cross-attention blocks are the conditioning embedding and output of the residual block. To ensure local consistency of trajectory, we used 1D convolution along the horizon dimension. To keep the number of parameters for cross attention and the original Unet relatively balanced, we also used ID convolution as the mapping from input to key, query, and value. Detailed model architecture is shown below in Figure
Residual block
Cross attention
Noisy trajectory
Noise prediction
Time embedding
Conditioning embedding
Figure 9: Model architecture.
The number of down-sampling// haapiinngguussappinngge. and feature channel sizes is different for each experiment. De- tailed parameters can be found in the section for each experiment.
23

---

Published in Transactions on Machine Learning Research (12/2025)
C.4
Trajectory formulation
The trajectory т Rd is represented by concalenating the state E Rds and the action au E Rdd at planning time step u and then horizontally stacking them for all time steps. For example, a trajectory with
82
planning horizon can be written as T
a1
a2
Oh
C.5
Maze2D
C.5.1
Extra Results
8
(b) ID
(c) OOC (Unconditioned)
(d) OOC (Conditioned)
(a) train traj
(Unconditioned)
Figure 10: Trajectories generated in Maze2D for medium maze. (a) are samples from the training set. (b) are trajectories generated by the unconditioned diffusion model given in distribution start and end positions. (c) are generated by the unconditioned diffusion model on unseen start and end positions. (d) are generated by a conditionod diffusion model using waypoints (black dots) as conditioning with classificr-froe guidance (efg) weight 1.3.
C.5.2 Experiment Details
We followed the setup used in Janner et al. 2022 The hyperparameters shared for large and medium mazes are shown below in Table I Large maze use a planning horizon of 384 and medium maze use a planning horizon of 256. Conditioning is passed through a positional embedding layer first to map each dimension of the waypoint (z, y, Ue, 0y) to a higher dimension of and concatenate them to form a vector size (1, 4). Three waypoints are then stacked together form a matrix of size (3, 4) and passed into the cross-allention layer. In our experiment, directly using the waypoints as conditioning was unsuccessful.
Parameter
Value
number of diffusion steps
256
action weight
dimension multipliers
(1, 4, 8)
classifier free guidance drop conditioning probability
0.1
steps per epoch
10000
loss type train steps
12
2e6
batch size
32
learning rate
2e-4
gradient accumulate every
2
ema decay
0.995
Table Training parameter for diffusion model Maze2D
24

---

Published in Transactions on Machine Learning Research (12/2025)
C.6 Roundabout
C.6.1
Environment
The training environment consists of all cars all bicycles and the testing environment is mixture of traffic (Figure
(a) train env (cars)
(b) train env 2 (bikes)
(c) Lest env 1
(d) Lest env
Training and testing environments for Roundabout. The green vehicle is the ego agent and the blue ones are controlled by the environment. The large blue box represents car and the small blue box represents a bicycle.
Figure 11:
C.6.2 Environment Parameters
We changed the parameters create different type of traffic in the roundabout as shown below Table
Also, since bicycles have slower speeds, we change the initialization position so that each environment vehicle interact with the ego vehicle.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Parameter
Car
Bicycle
length
5.0
2.0
width
2.0
1.0
speed
[23,
25]
4
max acceleration
6.0
2.0
comfort max acceleration
3.0
1.0
Table Parameters for car and bicycles in Roundabout environment
C.6.3 Dataset
In order to collect expert trajectorics, we train two PPO models separately on the environment with all cars and all bicycles. We then collect 320000 successful trajectories in the training environment. All trajectories have a unified length of 12.
C.7 Setup for Roundabout Environment
Here we describe how the Roundabout task in this paper conforms to our problem description. In this setting our base object set is E = {car, bicycle, null] where null means an object is non-visible. Since the maximum number of objects the roundabout is five and we fix the ego agent to be car, support for the training observation is {(car (ego agent), car, car, car, car), (car (ego agent), bicycle, bicycle, bicycle, bicycle) and for the testing observation is {carr (ego agent), bicycle, bicycle, car, car)) assuming no ordering and when the state is fully observable. Since the supports for training and testing non-overlapping under full observability, they will remain non-overlapping even when some traffic objects are out of sight, unless the ego agent is the only object present in the environment.
C.7.1 Experiment Details
We use stable bascline3 Raffin et al. (2021 as the implementation for PPO. The parameter the dofault parameter used in the Highway enviroument Leurent (2018). We increased total timesteps because the environment now has two modalities (all cars and all bicycles) and we observed that PPO takes longer converge. Detailed parameters for PPO and diffusion are shown below in Table and Table
Parameter
Value
policy
MIpPolicy
batch size
64
__steps
768
n epochs
learning rate
5e-4
gamma 0.8
total timesteps
2c5
Table 3: Training parameter for PPO
26

---

Published in Transactions on Machine Learning Research (12/2025)
Parameter
Value
planning horizon number of diffusion steps
8
80
action weight
10
dimension multipliers conditioning embedding size
(1, 4, 8)
20
classifier free guidance drop conditioning probability
0.1
classifier free guidance weight
1.0
steps per epoch
10000
loss type train steps
12
1e4
batch size
32
learning rate
2e-4
gradient accumulate every ema docay
0.995
Table 4: Training parameter for diffusion model in Roundabout
C.7.2 Model Size
We include the model size for different algorithms below in Table 5 To climinate the concern for performance gain due model size, we include the performance of large BC model that has roughly the same number of parameters the conditioned diffusion model.
BC
PPO
Diffusion
Large BC
Model size
0.30 MB
0.60 MB
54.19 MB
55.65 MB
Number of parameters
75013
148998
13546370
13912325
OOD reward
7.50 (0.03)
8.19 (0.16)
8.81 (0.2)
7.71 (0.3)
OOD crashes
37.7 (0.5)
36.0 (5.7)
20.0 (2.5)
37.3 (4.0)
Table 5: Model size, number of parameters, and performance for different models.
27

---

Published in Transactions on Machine Learning Research (12/2025)
C.8 StarCraft
C.8.1 Experiment Details
We use the codebase OpenRL Huang (2023) for the implementation of MAPPO. Detailed parameters for MAPPO can be found in Table 6
Parameter
Value
learning rale actor
5e-4
learning rate critic
1e-3
data chunk length
mum
8
episode length
400
PPO epoch
5
actor train interval step
use recurrent policy
True
use adv normalize value active masks
True
False
use linear LR decay
True
Table 6: MAPPO hyper-parammttes used for SMACv2. We utilize the typeraarametess used in SMACv2 Ellis
et al. 2022).
Detailed parameters for training a conditioned diffusion model for 5v5 are shown below in Table 7 and 3v3
in Table
Parameter
Value
planning horizon number of diffusion steps
40
256
action weight
1
dimension multipliers conditioning embedding size
(1, 4, 8)
40
classifier free guidance drop conditioning probability classifier free guidance weight
0.1
[0.7, 1.0, 1.3, 1.5]
steps per epoch loss type train steps
10000
2c6
batch size
32
learning rate
2e-4
gradient accumulate every ema decay
2
0.995
Table 7: Training parameter for diffusion model in StarCraft for 5v5
28

---

Published in Transactions on Machine Learning Research (12/2025)
Parameter
Value
planning horizon number of diffusion steps
32
256
action weight
1
dimension multipliers conditioning embedding size
(1, 4, 8)
40
classifier free guidance drop conditioning probability
0.1
classifier free guidance weight steps per epoch
[0.7, 1.0, 1.3, 1.5]
10000
loss type train steps
12
2e6
batch size
32
learning rate
2c-4
gradient accumulate every
2
ema docay
0.995
Table 8: Training parameter for diffusion model in StarCraft for 3v3
C.8.2 Dataset Initial State Distribution
The probability of generating cach unit type in SMACv2 is imbalanced. Specifically, the probability for Stalker, Zealot, and Colossus is 0.45, 0.45, and 0.1 respectively. The initial state distribution of training trajectories collected by MAPPO for random combination is shown below in Figure 12a and 12b Since we only keep the successful trajectories and use them expert data, the distribution depends the generation probability and MAPPO success rate for different team combinations. A total nunber of 240000 trajectories were used to train the diffusion model. Since diffusion is trained local observations and actions of all MАPPO actors, the total number of training samples is 24240000 for 5v5 and 3*240000 for 3v3.
- 25.39
16.00 % bba
ccbaa-12.54 12.54 %
cdbba 13.58 %
ccb 10.09 %
cocba 7.97 %
9.55 %
cbasa 7.73 %
ccbbb
eea 2..99 %
cocbo 4..1
crans 4.00
3.59
b0bba 2.42
cosaa 1.79 %
COcCa 1.51 9
bobbb 0.46
Distribution of initial state for 3v3 simple scenario
(b) Distribution of initial state for 5v5 simple scenario
29

---

Published in Transactions on Machine Learning Research (12/2025)
C.8.3 Detailed Results on SMACv2
Table and show the detailed performance of different algorithms in the 3v3 and 5v5 scenarios, respec- tively.
RL
Imitation Learning
Env: 3v3
2 PPO + Rand
3 PPO
BC
2 PPO + Diffusion
ABC - ABC (ID)
0.18 (0.01)
0.58 (0.02)
0.58 (0.07)
0.59 (0.04)
ABC + AAA (Simple)
0.07 (0.03)
0.52 (0.03)
0.56 (0.02)
0.59 (0.02)
AAA + AAA (ID)
0.09 (0.04)
0.63 (0.02)
0.6 (0.02)
0.61 (0.05)
AAA - ABC (Hard)
0.11 (0.02)
0.42 (0.02)
0.4 (0.06)
0.49(0.02)
Table 9: Success rate of each agent in 100 rounds. The first two rows correspond to the simple setting of generalization to states with different support and the last two rows correspond to non-overlapping support. Numbers in the parenthesis represent the standard error over 3 seeds. The best performing method is labeled with blue color. The 2 PPO + Rand columı shows the effect of replacing one PPO trained agent with a random agent as a baseline for comparison against the PPO Diffusion case.
RL
Imitation Learning
Env: 5x5
4 PPO + I Rand
5 PPO
BC
4 PPO + Diffusion
ABC ABC (ID)
0.22 (0.04)
0.64 (0.05)
0.56 (0.05)
0.66 (0.01)
ABC - AAA (Simple)
0.11 (0.03)
0.54 (0.04)
0.52 (0.05)
0.56 (0.02)
AAA AAA (ID)
0.14 (0.02)
0.58 (0.04)
0.54 (0.04)
0.55 (0.03)
AAA - ABC (Ilard)
0.11 (0.02)
0.26 (0.05)
0.28 (0.04)
0.32 (0.04)
Table 10: Success rate of each agent in 100 rounds. The first two rows correspond the simple setting of generalization to states with different support and the last two rows correspond non-overlapping support. Numbers in the parenthesis represent the standard error over 3 sceds. The best performing method is labeled with blue color. The 4 PPO + Rand column shows the effect of replacing one PPO trained agent with a random agent as a baseline for comparison against the 4 PPO + 1 Diffusion case.
C.8.4 Detailed Results for Ablation
The ablation result for 3v3 and 5v5 scenarios are shown below in Table [1 and Table 12 The first column is the success rate without conditioning (No Cond). The second column represents concatenating the con- ditioning with time embedding (Coneat). The last column represents passing conditioning as another input beside the trajectory to the cross-attention block (Attn).
Table 11: Ablation for Diffusion on 3v3
2 PPO + Diffusion
Env 3v3
No Cond
Concat
Attn
ABC + ABC (ID)
0.55++0.03
0.59
0.59+00.05
ABC + AAA (Simple)
0.5:0.066
0.59
0.59
AAA - AAA (ID)
0.4+-0.03
0.64+00.03
0.61+-0.05
AAA + ABC (Hard)
0.28++0.03
0.44
0.49+0.02
30

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: Ablation for Diffusion on 5v5
4PPO - 1 Diffusion
Env 5v5
No Cond
Concat
Attn
ABC + ABC (ID)
0.33+0.04
0.5940.03
0.66+0001
ABC - AAA (Simple)
0.50++0.03
0.50++0001
0.56++0.02
AAA AAA (ID)
0.47+00.08
0.55+0003
0.58+0004
AAA + ABC (Hard)
0.27::0.03
0.32
0.30+10.04
C.8.5 2v2
The success rates for StarCraft 2v2 are shown below in Table 13 We can see that out-of-combination cases did not cause the performance to drop drastically for MAPPO. This is because the number of combinations in 2v2 is very limited (e.g. aa, b6, ab), and if one agent dies, MAPPO has encountered scenarios playing with cach unit type individually, therefore falling back exists for 5v5 and 3v3 but only at the end of each game when only one agent is left.
distribution state again. This sccnario also
Table SMAC success rate for 2v2
BC
MAPPO
Diffusion
Env
BC
1 PPO + Rand 5 PPO
No Cond
Concal
Alin
ABC > ABC (ID)
0.54
0.06
0.62.10.55
0.43.10.04
0.00
0.57 10.06
ABC + AAA (Simple)
0.47=0.02
0.02+0.01
0.57+0002
0.44+00.2
0.55+0.06
0.49++0.06
AAA AAA (ID)
0.57
0.0110.01
0,64
0.38100.02
0.63
0.63110.02
AAA + ABC (Hard)
0.4+0.03
0.04+0.02
0.44+-0.02
0.29+-0.02
0.43-0.08
0.41++0.06
C.8.6 More Rendering of States Predicted by the Diffusion model
More rendering of the future states predicted by the diffusion model is shown in Figure 13
Green: Ego player
Blue: Teammate
Red: Enemy
Time
Zealot
Colossus
Stalker
Dark: Shield
Light: Health
Time
Figure 13: Rendering of future states predicted by the diffusion model given different conditionings. The grey box the initial state. Yellow boxcs are conditioned on the type of unit in the initial state. Green boxes are conditioned on all Triangles. Smaller sizes represent less shield or health.
31

---

Published in Transactions on Machine Learning Research (12/2025)
C.9
Model Runtime and GPU Memory
We include the training time and GPU memory used for the conditioned diffusion model below in Table 14 and 15
Training Time Roundabout
SMACv2 2v2
SMACv2 3v3
SMACv2 5v5
PPO
0.5h
9h
9h
9h
Diffusion
1h
48h
70h
98h
Table 14: Training time for PPO and conditioned diffusion model in different environments.
GPU Memory Roundabout Diffusion
SMACv2 2v2
SMACv2 3v3
SMACv2 5v5
542 MiB
1004 MiB
2892 MiB
4096 MiB
Table 15: GPU Memory for training conditioned diffusion model in different environments.
32

---

Published in Transactions on Machine Learning Research (12/2025)
C.9.1 Substituting More MAPPO Agents with Diffusion Agents
We would like to ask the question what about replacing more than one MAPPO agent with diffusion model. Figure 14 shows that the number of diffusion models does not have positive correlation with the success rate. This is because MAPPO can learn collaborative policy between actors and simply adding more ego-centric diffusion models will break the coordination between actions. Also, since the diffusion model is trained to play with all PPOs teammates, replacing other PPO actions with actions generated by diffusion models will cause distribution shift that is hard to quantify.
0.325
Succoss rate
0.275
0.200
Number drfusion agent
Success rate vs number of agents in SMACv2 5v5 hard scenario that are replaced with diffusion agents. Replacing more than one MAPPO agent with diffusion agents hurts performance.
Figure 14:
33


Reviews:
[{"review_id": "wa3HqeVYLv", "reviewer": "Reviewer_qK9o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies out of combination (OOC) generalisation, i.e. generalisation in RL to states that contain new combinations of elements not seen during training. This is not so much a problem of distributional shift, but rather of the support of the distribution shifting towards new combinations. The paper shows that Q-learning struggles here because OOC states are not encountered during training, meaning that more training will not help. On the contrary, the paper shows that diffusionn models do have a non-zero probability of sampling OOC states. Planning with a diffusion model should therefore work better. Experimentally this is confirmed.", "claims_and_evidence_yn": "No", "claims_explanation": "I think the empirical side of this paper is generally too weak. The paper aims to show that diffusion models generalize better on out of combination states than classical behavioral cloning. But this is done on only two environments, with only one approach to behavioral cloning. Though its message is an interesting one, for this message to carry weight, evaluation would have to consider more environments and more baselines.", "audience_yn": "Yes", "audience_explanation": "Out-of-support generalization, as a complement to the many existing works on distributional shifts, is an important problem to study, and without a doubt of interest to RL researchers and ML researchers more generally. A comparison of the generalisation abilities within this setting of various RL and behavioural cloning methods is a valuable contribution.", "requested_changes": "Critical to ensuring my recommendation would be first and foremost to extend the empirical evaluation to cover a wider range of environments. I would also like to see a better characterization of when diffusion models generalize well to OOC states and when they do not. Theoretically, we know that these states will be samples with non-zero probability, but as you say in the paper, the sampling probability can still be very near zero, meaning that practically generalisation is not guaranteed.", "broader_impact_concerns": "No special concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NK1FftgyOB", "reviewer": "Reviewer_fsvP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study 'out-of-combination' (OOC) generalisation, an OOD setting where new combinations of concepts are introduced. All concepts are however individually shown during training. Then the authors study this in an RL setting (sort of), and argue that diffusion models generalise better to this OOC setting. They experiment with several settings and show that conditioning on latent concepts improves performance in diffusion models.\n\nStrengths:\n- Important and timely topic\n- Writing is quite good\n\nWeaknesses:\n- Not really explored in a decision-making context\n- Evaluation is on pretty artificial datasets\n- Theoretical arguments for diffusion are unconvincing", "claims_and_evidence_yn": "No", "claims_explanation": "- The theoretical argument in Corollary 5.1 neither seems relevant nor correct\n- The setup in the experiments is confusing and complicated to me. It assumes behaviour cloning of an existing PPO-trained agent. Then, I think it uses a hard-coded concept extractor to extract the concepts from the image. This is usually not available in real-world scenarios.\n- The paper should be clearer about its experimental setup and methodology, including all the steps to get to the final model(s). \n- The use of diffusion to improve combinatorial generalisation is not well supported. In particular, there does not seem to be any experiments with conditioned non-diffusion models.", "audience_yn": "Yes", "audience_explanation": "The topic of the paper certainly is of interest to a fairly broad subset.", "requested_changes": "Critical\n- I fail to follow the point of Corollary 5.1, which is the main theoretical motivation for 'OOC Generalization in Diffusion Models'. The authors should clarify and improve the theoretical backing for diffusion models.\n    - In my understanding, the authors argue that a diffusion model will sample some state $s'$ lying in a linear manifold with non-zero probability under a diffusion model. \n    - This is incorrect: The probability of sampling a particular state within a continuous manifold is almost surely 0. I suppose the authors meant the density is nonzero.\n    - If so: This doesn't say much. Any Gaussian will have the same result, it's just saying $s'$ is in the support.\n    - Furthermore, this statement has nothing to do with $s'$ being OOC or not.\n- The authors should compare to a BC method without diffusion models but with conditioning on $z$. This seems to be an important factor contributing to the methods success, but is not properly ablated.\n- The authors should highlight that they assume the latent concept $z$ is given to the learner. This is not usually assumed in RL. \n- The conditioning method is not clear to me. Some points mention just conditioning the model, but then for the maze experiment, classifier-free guidance is used.\n- It was not clear how the diffusion model is trained to deal with $z$. Also, is the PPO agent trained with knowledge of $z$?\n\nEnhancements\n- Probably it's clear to most, but I would emphasise more that OOC is a specific type of OOD generalisation\n- Relevant related work:\n    - Combinatorial generalisation is also frequently studied within NLP, eg [1, 2], although this is possibly harder to consider as the formal OOC setup of this paper.\n    - The setup in Section 3 is closely related to that studied in Neurosymbolic Learning [3], both formally and informally. Some settings in that paper are examples of OOC (eg Mnist Even-Odd). On such datasets, it was also recently shown that diffusion can help generalisation [4]. One difference is that this line of work does not assume the latent vector of concepts z are given, but are extracted with neural networks. It is also related to concept-based models where concepts z are only given at training time [5]. \n- 5.1: Diffusion is not used for 'density estimation', as it does not have tractable density computation (one can only sample from it)\n\n\n\n1. Hupkes, Dieuwke, et al. \"Compositionality decomposed: How do neural networks generalise?.\" Journal of Artificial Intelligence Research 67 (2020): 757-795.\n2. Press, Ofir, et al. \"Measuring and narrowing the compositionality gap in language models.\" arXiv preprint arXiv:2210.03350 (2022).\n3. Marconato, Emanuele, et al. \"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.\" Advances in Neural Information Processing Systems 36 (2023): 72507-72539.\n4. van Krieken, Emile, et al. \"Neurosymbolic Diffusion Models.\" arXiv preprint arXiv:2505.13138 (2025).\n5. Kim, Been, et al. \"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).\" International conference on machine learning. PMLR, 2018.", "broader_impact_concerns": "No", "additional_comments": "- In algorithm 1, why is the state o noised instead of just conditioning directly on the observation o?\n- Is it correct the setting in Figure 1 is not used in any experiments\n- I did not understand the experiment of 8.1. How is the OOC setting differently constructed? And how is z represented?", "extra_scores": {}, "extra_text": {}}, {"review_id": "cWDzcO9910", "reviewer": "Reviewer_1rA2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates decision-making problems in reinforcement learning where states are composed of basic elements. Building on this perspective, the authors introduce the state combinatorial generalization problem, where agents must handle novel combinations of known elements at test time. They show that traditional RL approaches struggle with such out-of-support generalization due to unreliable value predictions. To address this, the authors propose using conditional diffusion models, leveraging their ability to capture compositional structure. Specifically, they define a structured vector Z representing element combinations and use conditioning on Z to guide trajectory generation. The diffusion model is trained offline on expert demonstrations but deployed online for decision-making. Empirical evaluations across diverse environments, including driving (Roundabout), maze navigation (Maze2D), and multi-agent games (SMACv2), demonstrate that the proposed method consistently outperforms both offline and online RL baselines in zero-shot generalization tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Extensive experiments are conducted to cover a diverse range of environments. Detailed ablation studies and visualizations are also conducted to help clarify why the diffusion model generalizes effectively.", "audience_yn": "Yes", "audience_explanation": "The paper provides a strong motivation for studying generalization in decision-making tasks, particularly focusing on handling unseen combinations of known elements. The concept of state combinatorial generalization and the out-of-combination (OOC) are clearly defined and distinguished from traditional out-of-distribution (OOD) generalization problems.\n\nThe proposed approach demonstrates impressive results, showing that a diffusion model trained offline via behavior cloning can outperform strong online RL baselines such as PPO in zero-shot OOC generalization tasks.", "requested_changes": "Major issue: \n\nTo explain why diffusion models generalize better, the paper introduces Corollary 5.1, which shows that a well-trained diffusion model can, in principle, sample out-of-combination (OOC) instances with non-zero probability. Later, in Section 6, the authors argue that the generated trajectories will remain dynamically consistent when conditioned on novel combinations of base elements. While both points are interesting and seem to be loosely connected, the connection between them is not sufficiently clear. Specifically, it is unclear how the existence result in Corollary 5.1 translates into the practical ability to handle unseen conditioning vectors in a reliable way. For example, if the probability of sampling an OOC instance is technically non-zero but extremely small, how does conditioning ensure that such states are generated with meaningful likelihood? A more rigorous theoretical link between the corollary and the conditioning-based generalization claim would significantly strengthen the argument.\n\nMinor issue: \n\nWhile Section 6 provides a description of the training and testing procedure for the proposed diffusion model, the explanation could be made clearer and more structured. A step-by-step outline or schematic would help readers better understand how the model is trained offline and then deployed online for decision-making.\n\nThe terminology around Z is potentially confusing. Referring to it as a latent vector may be misleading, since it is explicitly defined to represent the combination of base elements and is therefore not latent in the conventional sense. A more precise term, such as compositional encoder (similar to positional encoder in transformers), might avoid confusion.\n\nThe proposed method generates full trajectories during inference, but only the first action is executed in the environment. This raises a potential computational efficiency concern, as much of the generated trajectory is discarded at each step. A discussion of potential strategies to mitigate it would be useful.", "broader_impact_concerns": "No significant concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)