Title: RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and Behavior Regularization
Primary area: reinforcement learning
Keywords: offline reinforcement learning, risk-averse RL, CVaR, distributional RL, generative policies, diffusion model, flow matching, behavior cloning, multimodal actions, out-of-distribution (OOD)

Abstract: In safety-critical domains  where online data collection is infeasible, offline reinforcement learning (RL) offers an attractive alternative but only if policies deliver high returns without incurring catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of value conservatism and restricted policy classes, whereas expressive policies are only used in risk-neutral settings. Here, we address this gap by introducing the **Risk-Aware Multimodal Actor-Critic (RAMAC)** framework, which couples an *expressive generative actor* with a distributional critic. The RAMAC differentiates composite objective combining distributional risk and BC loss through the generative path, achieving risk-sensitive learning in complex multimodal scenarios. We instantiate RAMAC with diffusion and flow-matching actors and observe consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns on most Stochastic-D4RL tasks.

TL;DR: We guide expressive diffusion/flow policies with CVaR from a distributional critic, achieving safer offline RL (better lower tails) while keeping mean returns high and reducing OOD visitation.