ICML 2025

Maximum Entropy Reinforcement Learning with Diffusion Policy
Xiaoyi Dong 1 2 Jian Cheng 1 3 4 Xi Sheryl Zhang 1 4
Abstract
The Soft Actor-Critic (SAC) algorithm with a
Gaussian policy has become a mainstream im-
plementation for realizing the Maximum Entropy
Reinforcement Learning (MaxEnt RL) objective,
which incorporates entropy maximization to en-
courage exploration and enhance policy robust-
ness. While the Gaussian policy performs well
on simpler tasks, its exploration capacity and po-
tential performance in complex multi-goal RL
environments are limited by its inherent uni-
modality. In this paper, we employ the diffusion
model, a powerful generative model capable of
capturing complex multimodal distributions, as
the policy representation to fulfill the MaxEnt
RL objective, developing a method named Max-
Ent RL with Diffusion Policy (MaxEntDP). Our
method enables efficient exploration and brings
the policy closer to the optimal MaxEnt pol-
icy. Experimental results on Mujoco benchmarks
show that MaxEntDP outperforms the Gaussian
policy and other generative models within the
MaxEnt RL framework, and performs compara-
bly to other state-of-the-art diffusion-based on-
line RL algorithms.
Our code is available at
https://github.com/diffusionyes/MaxEntDP.
1. Introduction
Reinforcement Learning (RL) has emerged as a powerful
paradigm for training intelligent agents to make decisions in
complex control tasks (Silver et al., 2016; Mnih et al., 2015;
Kaufmann et al., 2023; Kiran et al., 2021; Ibarz et al., 2021).
Traditionally, RL focuses on maximizing the expected cu-
mulative reward, where the agent selects actions that yield
the highest return in each state (Sutton & Barto, 1999). How-
1C2DL, Institute of Automation, Chinese Academy of Sciences
2School of Artificial Intelligence, University of Chinese Academy
of Sciences 3School of Future Technology, University of Chi-
nese Academy of Sciences 4AiRiA. Correspondence to: Xi Sheryl
Zhang <sheryl.zhangxi@gmail.com>.
Pr...