Title: Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: offline reinforcement learning

diffusion models

structural information principles
Abstract: Diffusion-based generative methods have shown promising potential for modeling trajectories from offline reinforcement learning (RL) datasets, and hierarchical diffusion has been introduced to mitigate variance accumulation and computational challenges in long-horizon planning tasks. However, existing approaches typically assume a fixed two-layer diffusion hierarchy with a single predefined temporal scale, which limits adaptability to diverse downstream tasks and reduces flexibility in decision making. In this work, we propose SIHD, a novel Structural Information-based Hierarchical Diffusion framework for effective and stable offline policy learning in long-horizon environments with sparse rewards. Specifically, we analyze structural information embedded in offline trajectories to construct the diffusion hierarchy adaptively, enabling flexible trajectory modeling across multiple temporal scales. Rather than relying on reward predictions from localized sub-trajectories, we quantify the structural information gain of each state community and use it as a conditioning signal within the corresponding diffusion layer. To reduce overreliance on offline datasets, we introduce a structural entropy regularizer that encourages exploration of underrepresented states while avoiding extrapolation errors from distributional shifts. Extensive evaluations show that SIHD significantly outperforms state-of-the-art baselines in decision-making performance and demonstrates superior generalization across diverse scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the classification of states in trajectories from the perspective of the undirected graph, and propose a multi-scale diffusion hierarchy model that simultaneously introduces a structural entropy regularizer to encourage exploration of underrepresented states while avoiding extrapolation errors caused by distributional shifts.\n\nSTRENGTHS AND WEAKNESSES:\nPros: The theoretical framework is well-developed, the paper is well-organized, and the experiments are fairly comprehensive.\nCons: However, some parts of the paper are expressed ambiguously, making it difficult for readers to fully understand the intended meaning.\n\nQUESTIONS:\n1) The authors did not explain the meaning of $\\alpha^{-}$ in Section 3.3.\n2) In line 182, could the authors clarify how k is determined? Additionally, in line 184, since the HCSE optimization algorithm can be used to determine the optimal $\\mathcal{K}$, it would be helpful to understand why $\\mathcal{K}$ is directly specified in the experimental section.\n3) From equation (6), it seems that this partitioning will result in state-action sequences of varying lengths within the trajectory. When h = 1, could the authors clarify whether the corresponding $\\tau$ refers to a state-action pair or a sequence of state-action pairs? Additionally, since sequences of fixed length are required for training the diffusion model, how do the authors handle the issue of varying sequence lengths across different layers in the hierarchy?\n4) A simple example is helpful to help readers understand this paper:\n\nSuppose that a trajectory: [1, ..., 1000]\n\nh=1: [1, 2, 3, ..., 1000]\n\nh=2: [1,2], [3, 4, 5], [6, 7, 8, 9, 10], ...\n\nh=3: [1,2, 3, 4, 5], [6, 7, 8, 9, 10, 11, 12, 13], [14, ...]\n\n....\n\nThen, 1) whether the separation shown above is right? 2) What are the corresponding goals of each layer ?\n\n5) In Figure 3 (c), should k-1 be replaced with h? Additionally, $l_g^{K-1}$ appears in (c) but is not introduced prior to line 197. Would it be possible for the authors to provide clarification? And in line 202, the authors also do not introduce the meaning of $l_g^{h,i}$.\n6) Based on the authors' description, the paper utilizes classifier-free guidance. Could the authors clarify how the conditions of the diffusion model are pre-specified during inference?\n7) Could the authors clarify the meaning of \"shared\" mentioned in line 226? My understanding is that each layer represents a new diffusion model.\n8) In line 230, I suggest important references about class-free guidance, such as \n\n  [1] Classifier-Free Diffusion Guidance.\n\n  [2] More Control for Free! Image Synthesis with Semantic Diffusion Guidance.\n\n9) While I can understand that diffusion models are capable of modeling complex distributions of sequences, I am unclear about how the transition probability is computed. It would be helpful if the authors could provide a more detailed explanation.\n10) Would constructing an additional weighted state graph $\\mathcal{G}_s^\\prime$ incur significant time overhead? How is Shannon entropy calculated? I suggest that the authors provide a more detailed explanation to help readers better understand. Why do the authors only apply entropy regularization term on h=1?\n11) I notice that the paper does not provide a table of important hyperparameters. I suggest that the authors include the hyperparameter settings related to the experiments and methods.\n12) Could the authors clarify how much computational resource is required for state community partitioning on the D4RL dataset?\n13) In Table 4, intuitively, increasing the number of layers in the hierarchical diffusion model should directly increase the time for one generation. However, SIHD takes less time than HD. Why is this the case?\n14) I noticed that the authors mention online decision-making in Figure 2, but the experimental results do not include online RL. Therefore, I suggest that the authors either add relevant online RL experiments or revise Figure 2.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a Structural Information-based Hierarchical Diffusion framework, namely SIHD is proposed, where structural information embedded in offline trajectories is utilized to construct the diffusion hierarchy. Furthermore, a structural entropy regularizer is presented to encourage exploration of underrepresented states. Extensive experiments were conducted on D4RL to show the effectiveness and generalization of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe presented idea that using structural information existed in the historical trajectories for offline policy learning is novel and extensive experiments are conducted to show the effectiveness of the proposed framework. Moreover, theoretical guarantees are provided to enable the decomposition of the conditional generation problem for offline trajectories into a hierarchical diffusion process.\n\nWeaknesses:\nThe main concerns are the scalability and generalization of the proposed framework. Besides D4RL, it is unclear whether the required structure information for the deriving of the tree-structured partitioning of state communities exists in other offline datasets. Also, how can we guarantee the accuracy of the partitioning of the state communities?  Does it have a clear impact on the final performance of the presented method? As is shown in the experiments, the community partitioning is time-consuming and computation-expensive, it may be hardly to be extended to large-scale systems, experiments on some real-world applications may be necessary to verify the method’s scalability. Furthermore, only experimental results are provided to support the performance of the structural entropy regularizer while theoretical guarantees are lacked.\n\nQUESTIONS:\nPlease refer to the Weaknesses part of the review.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SIHD, a novel hierarchical diffusion method for offline RL. The main idea is to build a multi-scale hierarchy by first partitioning the state space, and thus trajectories in the data, into tree-based structures, and then train diffusion models at different levels of this hierarchy to enable long-horizon inference. The method is tested on 3 domains from the D4RL benchmark and compares favorable to both non-hierarchical and hierarchical state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper considers a relevant and well-motivated problem (long-horizon generalization in offline RL)\n2. The paper tackles and important limitation of existing hierarchical methods (i.e., the hierarchy is mostly restricted to 2 layers), which limits their scalability to large horizons\n3. The proposed method performs well empirically in comparison to both non-hierarchical and hierarchical state-of-the-art baselines\n4. Experiments to ablate the main algorithmic design choices are provided, showing that each of them is indeed relevant\n5. Existing literature seems to be properly discussed (though I am not an expert of hierarchical methods for offline RL)\n\nWeaknesses:\n1. I found the paper very hard to read, especially Section 4 given the convoluted notation with many sub/superscripts. After reading it multiple times, I only got a high-level understanding of the proposed method, while I am not sure I fully grasp all the details to properly judge its significance. Both Figure 2 and 3 didn't help as they only list the main algorithmic components labeled by hard-to-read notation and without explaining their functionalities. I think a figure that more directly shows how a trajectory may be segmented into different hierarchies, which part goes into which diffuser, and what's the conditional information used for those would greatly help. \n2. Also, in was not clear what's the conditional information used at the different levels of the hierarchy. Equation 8 seems to imply that the subgoal g_i is used as conditioning, but then the right-hand side doesn't depend on g_i (unless that dependence is hidden in R?). Equation 9 seems to imply that the conditioning is the exponential of the reward for the last level of the hierarchy, while intermediate layers use subgoal information. Finally, Equation 12 seems to imply that intermediate layers actually use the structural information gain\n3. One limitation of the proposed method is that it needs to train one different diffusion model for each level of the hierarchy. While it is true this is in the attempt to overcome the limit of 2 layers in existing methods, this brings its own issues: (1) it may be computationally much more expensive to train and use at test-time, (2) it prevents apple-to-apple comparison with existing methods since the proposed one has more capacity/parameters. There is no discussion about this in the paper\n4. The motivation and benefits of the structural entropy-based exploration regularizer are not clear. What does it even mean to \"explore\" in offline RL? Why would we want to go out of the data distribution, when the majority of offline RL works actually try to stay within data support to prevent extrapolation errors?\n5. While the whole approach is motivated by enabling long-horizon predictions, the experiments are still done on relatively short-horizon tasks and with only 3-4 hierarchy layers. There is no scalability analysis (in terms of compute/data/etc) for these variables\n6. A more qualitative empirical evaluation showing, eg, what the diffusers learn at different hierarchies is missing\n7. It was not clear how the proposed method can be used at test time (eg for planning). I think this is relegated to an appendix, but it seems a very important aspect to have in the main paper\n\nQUESTIONS:\nPlease address limitations above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SIHD, a hierarchical diffusion framework for offline reinforcement learning that adaptively builds multi-scale policy hierarchies based on structural information from offline trajectories. By quantifying information gain across state communities and incorporating it into conditional diffusion, SIHD enables flexible and effective long-horizon planning. A structural entropy regularizer further promotes exploration of underrepresented states while mitigating extrapolation errors. Experiments on D4RL benchmarks show that SIHD outperforms prior methods in both decision quality and generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of this paper is novel and solid, improving from the 2-layer structure to the multi-layer structure is promising.\n2. The method proposed in this paper is interesting, especially about the threorical part. The partition algorithm is great, while the authors need to clarify the intuition of the partition.\n3. The empirical result is very strong, the authors show a impressive improvement compared with previous sota algorithms.\n\nWeakness:\n1. The writing of this paper needs to be polished. The idea and intuition of each algorithm component is unclear in the paper. Some of the notations is confusing, e.g. sec 4.1.\n2.(minor). The figure 2 is hard to read, it would be better if the authors could change the lines in the figure to make them easier to distinguish.\n3. Is the improvement of this work compared with previous work incremental?\n4. The experiments are limited to D4RL, the authors could add more experiments on different benchmarks like meta-world.\n\nQUESTIONS:\n1. How does the result benefit from the hierarchical diffusion structure compared with simple 2-layer diffusion structure? Can the authors show the difference with experiments like a case study?\n2. What is the intuiation of using such a encoding tree to establish the hierarchical partition?\n\nLIMITATIONS:\nThe authors addressed the limitations while they mentioned this work doesn't have any potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}