Title: Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

diffusion language models

diffusion chain of lateral thought
Abstract: We introduce the Diffusion Chain of Lateral Thought (DCoLT), a reasoning framework for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent "thinking" action and optimizes the entire reasoning trajectory to maximize the reward on the correctness of the final answer with outcome-based Reinforcement Learning (RL). Unlike traditional Chain-of-Thought (CoT) methods that follow a causal, linear thinking process, DCoLT allows bidirectional, non-linear reasoning with no strict rule on grammatical correctness amid its intermediate steps of thought. We implement DCoLT on two representative Diffusion Language Models (DLMs). First, we choose SEDD as a representative continuous-time discrete diffusion model, where its concrete score derives a probabilistic policy to maximize the RL reward over the entire sequence of intermediate diffusion steps.  We further consider the discrete-time masked diffusion language model -- LLaDA, and find that the order to predict and unmask tokens plays an essential role to optimize its RL action resulting from the ranking-based Unmasking Policy Module (UPM) defined by the Plackett-Luce model. Experiments on both math and code generation tasks show that using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs outperform other DLMs trained by SFT or RL or even both. Notably, DCoLT-reinforced LLaDA boosts its reasoning accuracy by +9.8%, +5.7%, +11.4%, +19.5% on GSM8K, MATH, MBPP, and HumanEval.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DCoLT, a new framework for reasoning in Diffusion Language Models (DLMs) that follows lateral thinking instead of traditional step-by-step reasoning. It uses reinforcement learning to train all intermediate steps in the reverse diffusion process, with an Unmasking Policy Module (UPM) using a Plackett-Luce model.\n\nSTRENGTHS AND WEAKNESSES:\nI'll provide a comprehensive review of this paper on \"Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models.\"\n\n## Summary\n\nThis paper introduces DCoLT (Diffusion Chain of Lateral Thought), a novel reasoning framework for diffusion language models that treats intermediate diffusion steps as \"thinking\" actions and optimizes the entire reasoning trajectory using outcome-based reinforcement learning. The authors implement DCoLT on two diffusion language models (SEDD and LLaDA) and demonstrate improvements on math and code generation tasks.\n\n## Strengths\n\n### Quality\n- **Strong experimental validation**: The paper provides comprehensive experiments across multiple tasks (GSM8K, MATH, MBPP, HumanEval) and compares against relevant baselines including both diffusion models and autoregressive models.\n- **Solid technical implementation**: The authors properly adapt RL training to two different types of diffusion models (continuous-time SEDD and discrete-time LLaDA), showing technical depth.\n- **Thorough analysis**: The paper includes helpful visualizations of the generation process and ablation studies examining key components like the Unmasking Policy Module.\n\n### Clarity\n- **Well-structured presentation**: The paper is generally well-written with clear motivation and methodology sections.\n- **Good visual aids**: Figure 1 effectively illustrates the key differences between CoT and DCoLT, and the appendix provides helpful examples of the generation process.\n- **Comprehensive details**: The extensive appendix provides implementation details necessary for reproduction.\n\n### Significance\n- **Novel perspective on reasoning**: The concept of \"lateral thinking\" in diffusion models is interesting and provides a fresh angle on reasoning in language models.\n- **Competitive results**: DCoLT achieves strong performance, particularly on LLaDA where it shows substantial improvements (+9.8% on GSM8K, +5.7% on MATH).\n- **Data efficiency**: The approach demonstrates good data efficiency compared to methods requiring much larger proprietary datasets.\n\n## Weaknesses\n\n### Technical Concerns\n1. **Limited theoretical justification**: The paper lacks theoretical analysis of why treating diffusion steps as reasoning should work better than traditional approaches. The \"lateral thinking\" metaphor, while intuitive, needs stronger theoretical grounding.\n\n2. **Unclear advantage over autoregressive models**: While the results are competitive, it's not entirely clear why the added complexity of diffusion models is justified when autoregressive models with CoT can achieve similar or better performance with much simpler training.\n\n3. **Reward design limitations**: The approach is limited to tasks with verifiable rewards (math/code), which significantly constrains its applicability. The authors acknowledge this but don't provide a clear path forward.\n\n### Experimental Limitations\n1. **Limited model scales**: Experiments are conducted on relatively small models (400M SEDD, 8B LLaDA). It's unclear how the approach scales to larger models where autoregressive methods have shown dramatic improvements.\n\n2. **Computational efficiency concerns**: The paper doesn't thoroughly analyze the computational overhead of the multi-step diffusion process compared to autoregressive generation, which is crucial for practical adoption.\n\n3. **Limited baseline comparisons**: While the paper compares to some diffusion models, it would benefit from more comprehensive comparisons with recent strong autoregressive reasoning methods.\n\n### Methodological Issues\n1. **Unclear generalization**: The \"lateral thinking\" concept is demonstrated mainly through examples, but it's unclear how consistently this emerges across different problems and whether it's truly beneficial or just a different path to the same solution.\n\n2. **Evaluation metrics**: The paper focuses primarily on accuracy metrics but doesn't analyze other important aspects like reasoning quality, interpretability, or failure modes.\n\nQUESTIONS:\n1. Have authors considered integrating auxiliary rewards alongside outcome-based RL?\n\n2. Provide concrete examples of failure cases\n\n3. Can you provide theoretical analysis of when and why DCoLT should outperform traditional approaches?\n\n4. How does the computational cost compare to autoregressive methods with similar performance?\n\n5. Can you demonstrate that the \"lateral thinking\" behavior is consistent and beneficial across a broader range of problems?\n\n6. How does the approach scale to larger models and longer sequences?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper introduces the Diffusion Chain of Lateral Thought, a reasoning framework for diffusion language models that treats reverse diffusion steps as thinking actions and optimizes them via outcome-based RL. Unlike linear CoT methods, DCoLT enables bidirectional, non-linear reasoning without strict grammatical constraints on intermediate steps. Experiments show that DCoLT-reinforced models outperform SFT and RL baselines on math and code generation tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. DCoLT leverages reverse diffusion to model lateral thinking, allowing non-linear, format-free reasoning that aligns with human cognitive patterns, which is not supported by causal CoT.
2. Using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs achieve state-of-the-art results, outperforming models trained with proprietary data.
3. Supporting both continuous-time and discrete-time diffusion language models, DCoLT demonstrates versatility across different diffusion paradigms.

Weaknesses:
1. Performance could be improved with more data and compute, and prior works with proprietary data show better results.
2. DCoLT is validated only on tasks with clear reward functions e.g., math and coding, lacking evaluation on open-ended natural language tasks.

QUESTIONS:
1. How does DCoLT's RL approach (e.g., GRPO) compare to other RL algorithms (e.g., PPO) in terms of training stability and performance?

2. The paper shows longer generations improve accuracy, but what is the optimal length for tasks with different degrees of difficulty, and how does it balance computation and model performance?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an online reinforcement learning algorithm called DCoLT for finetuning masked diffusion models on reasoning and code. The method uses group statistics to estimate the advantage function without using a explicit value function. The RL objective is similar to GRPO but it incorporates all the steps from the diffusion trajectory (unlike other works like d1 that only use a single step estimate). The paper also introduces an additional “unmasking” head for selecting the tokens to unmask, and show that this in itself provides significant gains in performance even if the rest of the model is frozen. The results are quite strong showing improvement over existing SFT and other RL baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Clarity**: The paper is well written and easy to follow. There is a good amount of implementation-related details in the paper and the appendix, suggesting that the results should be easily reproducible.\n\n**Quality**: The baselines are a bit weak. Specifically, the only Diffusion+RL-based baseline is d1-LLaDA. But it is unclear if d1 can be compared directly with the proposed approach (see questions 1, and 2) below.\n\n**Significance**: The work could be quite useful for the community, as it demonstrates that full-trajectory based RL performs reasonably well for masked diffusion models.\n\n**Originality**: The work is original because it is the first to successfully apply trajectory-level RL for the masked diffusion model in the text domain.\n\nQUESTIONS:\n1. Based on Appendix C.3, it appears that d1 trains separate models for GSM8K and MATH, each using its own corresponding training set. However, DCoLT seems to combine both datasets during training. This may not be a fair comparison. Can you please clarify whether the same training data is used for both models?\n    \n2. I may be misunderstanding, but it seems like d1 only finetunes LoRA adapters, whereas DCoLT does full parameter updates, making the results difficult to compare.\n    \n3. The term “nonlinear generation” is ambiguous. For instance, in line 131, I don’t quite understand what it means. If the intent for it is to mean “non-left-to-right,” then it might be better to write non-left-to-right or something similar.\n    \n4. Could you comment on the coherence of the complete answer sequence when the answer is correct? I realize that a final-answer-based 0/1 reward may not encourage, or perhaps even discourage, coherent text sequence in the middle of the answer. It will be nice to present some concrete analysis on this front.\n    \n5. Please provide more details regarding the final answer-based reward for the question-answering datasets. Specifically, what is the filter used to extract the final answer for GSM8K-Aug, GSM8K, and MATH datasets for reward generation?\n    \n6. Do you train the model to predict <eos> tokens? In Figure 8, in step 64, I see that there are <eos> tokens. Are these predicted by the model, or are they inserted automatically after the model predicts the first 
?\n    \n7. Have you tried DCoLT on Dream 7B? It would make the paper stronger if the method shows similar gains across the two models.\n\n8. Since RL training is generally unstable, it would be good if you could provide average reward vs training progress curves.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method, diffusion chain of lateral thought, which uses reinforcement learning with final answer only rewards to enhance reasoning ability of  diffusion language models. Authors develop methods for both continuous and discrete time diffusion models. They claim this could have bidirectional and format free reasoning abilities. Authors also implement experiments with SEDD and LLaDA with DCoLT on math, coding and sudoku benchmarks, and present competitive or better performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* It tries to simulate lateral, human-like thinking, where key concepts emerge early and are refined over time. This is a clear advancement over traditional Chain-of-Thought in autoregression models, because autoregressive models cannot modify generated tokens.\n* Authors thoughtfully implement DCoLT on both continuous time and discrete time diffusion language models. This highlights the adaptability across different model architectures.\n* This method leverages final answer only rewards, which is a practical strength, as high quality CoT annotations are expensive and limited.\n* The method achieves better or competitive results on math, coding and sudoku benchmarks. It performs even better than its autoregressive counterpart in some tasks.\n\nWeaknesses:\n* Some claims in this paper are problematic, e.g. for the generation length, I don’t think it plays an important role here. The performances with length from 256-512 are very close. \n* I’m not sure whether the method really optimizes the entire reverse process jointly (line 173-174), because in Algorithm1, the optimization is processed from 1 to n, which means getting loss for each denoising step separately.\n\nQUESTIONS:\nThe method only trained the model using final-answer reward signals, with no supervision or constraint on the intermediate diffusion steps. This setup has been shown to cause unreadable intermediate steps in autoregressive models (e.g. DeepSeek-R1), and this is a reason why they have an additional SFT process. Despite this, attached examples in this paper show semantically meaningful and readable CoT like reasoning. I’m curious why the case is different with autoregressive models and outcome-based RL on diffusion language models that lead to interpretable intermediate reasoning.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}