Title: SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
Primary area: reinforcement learning
Keywords: Safe Exploration, Sharpness Aware Minimization, Epistemic Uncertainty

Abstract: Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor’s sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy
update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor’s epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the
influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.

TL;DR: SHAPO makes policy updates pessimistic under epistemic uncertainty, reweighting gradients toward rare unsafe actions to enable safer exploration and improved safety–performance trade-offs