NeurIPS 2025

COLA: Towards Efficient Multi-Objective
Reinforcement Learning with Conflict Objective
Regularization in Latent Space
Pengyi Li*, Hongyao Tang†, Yifu Yuan, Jianye Hao†, Zibin Dong, Yan Zheng
College of Intelligence and Computing, Tianjin University
Abstract
Many real-world control problems require continual policy adjustments to bal-
ance multiple objectives, which requires the acquisition of high-quality policies
to cover diverse preferences. Multi-Objective Reinforcement Learning (MORL)
provides a general framework to solve such problems. However, current MORL
methods suffer from high sample complexity, primarily due to the neglect of effi-
cient knowledge sharing and conflicts in optimization with different preferences.
To this end, this paper introduces a novel framework, Conflict Objective Regular-
ization in Latent Space (COLA). To enable efficient knowledge sharing, COLA
establishes a shared latent representation space for common knowledge, which
can avoid redundant learning under different preferences. Besides, COLA in-
troduces a regularization term for the value function to mitigate the negative ef-
fects of conflicting preferences on the value function approximation, thereby im-
proving the accuracy of value estimation. The experimental results across vari-
ous multi-objective continuous control tasks demonstrate the significant superi-
ority of COLA over the state-of-the-art MORL baselines. Code is available at
https://github.com/yeshenpy/COLA.
1
Introduction
Real-world problems often involve multiple performance metrics, where conflicts among these met-
rics make it infeasible to maximize all metrics simultaneously with a single policy. This necessitates
a trade-off among different metrics, requiring the learning of multiple high-quality policies to meet
diverse preferences [1–3]. For instance, in robot control problems, performance metrics may include
both forward speed and energy consumption. In such cases, two reward signals are returned, and
multipl...