Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots
Authors: Wei Hung
Bo Kai Huang
Ping-Chun Hsieh
Xi Liu
Venue: ICLR 2023 poster (2023)
Keywords: Multi-objective reinforcement learning
sample efficiency

Abstract: Many real-world continuous control problems are in the dilemma of weighing the pros and cons, multi-objective reinforcement learning (MORL) serves as a generic framework of learning control policies for different preferences over objectives. However, the existing MORL methods either rely on multiple passes of explicit search for finding the Pareto front and therefore are not sample-efficient, or utilizes a shared policy network for coarse knowledge sharing among policies. To boost the sample efficiency of MORL, we propose $Q$-Pensieve, a policy improvement scheme that stores a collection of $Q$-snapshots to jointly determine the policy update direction and thereby enables data sharing at the policy level. We show that $Q$-Pensieve can be naturally integrated with soft policy iteration with convergence guarantee. To substantiate this concept, we propose the technique of $Q$ replay buffer, which stores the learned $Q$-networks from the past iterations, and arrive at a practical actor-critic implementation. Through extensive experiments and an ablation study, we demonstrate that with much fewer samples, the proposed algorithm can outperform the benchmark MORL methods on a variety of MORL benchmark tasks.

Published as a conference paper at ICLR 2023
Q- :--EESIEVE: BOOSTING SAMPLE EFFICIENCY OF MULTI-OBJECTIV RL THROUGH MEMORY SHARING OF Q-SNAPSHOTS
Wei Hungl Bo-Kai Huang'; Ping-Chun Hsieh', Xi Liu'
DDppartment of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan RReseacch Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan 3 AApllid Machine Learning, Meta AI, Menlo Park, CA, USA (hwe Twei1048577 088576 csca, wwwpppiiinineeeeeeeeeeeeee
tw, xiliu. tamuu com
ABSTRACT
Many real-world continuous control problems are in the dilemma of weighing the pros and cons, multi-objective reinforcement learning (MORL) serves as a generic framework of learning control policies for different preferences over objectives. However, the existing MORL methods either rely on multiple passes of explicit search for finding the Pareto front and therefore are not sample-efficient or utilizes shared policy network for coarse knowledge sharing among policies. To boost the sample efficiency of MORL, we propose Q-Pensieve, a policy improvement scheme that stores collection of Q-snapshots to jointly determine the policy update direction and thereby enables data sharing at the policy level. We show that Q- Pensieve can be naturally integrated with soft policy iteration with convergence guarantee. To substantiate this concept, we propose the technique of Q replay buffer, which stores the learned Q-networks from the past iterations, and arrive at a practical actor-critic implementation. Through extensive experiments and an ablation study, we demonstrate that with much fewer samples, the proposed algorithm can outperform the benchmark MORL methods on a variety of MORL. benchmark tasks.
INTRODUCTION
Many real-world sequential decision-making problems involve the joint optimization of multiple objectives, while some of them may be in conflict. For example, in robot control, it is expected that the robot can run fast while consuming as little energy as possible; nevertheless, we inevitably need to use more energy make the robot run fast, regardless of how energy -ffficieet the robot motion is. Moreover, various other real-world continuous control problems are also multi-objective tasks by nature, such congestion control in communication networks (Ma et al. 2022) and diversified portfolios (Abdolmaleki et al. 2020). Moreover, the relative importance of these objectives could vary over time (Roijers and Whiteson 2017). For example, the preference over energy and speed in robot locomotion could change with the energy budget; network service providers need to continuously switch service among various networking applications (e-g., on-demand video streaming versus real-time conferencing). each of which could have preferences over latency and throughput.
To address the above practical challenges, multi-objective reinforcement learning (MORL) serves as one classic and popular formulation for learning optimal control strategies from vector- vector-valued reward signal and achieve favorable trade-off among the objectives. In the MORL. framework, the goal is to learn collection of policies, under which the attained return vectors recover much of the Pareto front as possible. One popular approach to addressing MORL is to explicitly search for the Pareto front with an aim maximize the hypervolume associated with the reward vectors, such as evolutionary search (Xu et al. 2020) and search by first-order stationarity (Kyriakis et al.) (002)). While being effective, explicit search algorithms known to be rather hample-ineffiene data sharing among different passes of explicit search is rather limited. As result, it is typically
as the
Equul contribution.

---

Published as a conference paper at ICLR 2023
difficult to maintain a sufficiently diverse set of optimal policies for different preferences within reasonable number of training samples. Another way to address MORL is to implicitly search for non-dominated policies through linear scalarization, ie., convert the vector- valued reward signal to single scalar with the help of a linear preference and thereafter apply a conventional single-objective RI. algorithm for iteratively improving the policies (e.g., (Abels et al. 2019 Yang al.; 2 To enable implicit search for diverse preferences simultaneously. a single network is typically used to express whole collection of policies. As result, some level of data sharing among policies of different preferences is done implicitly through the shared network parameters. However, such sharing is clearly not guaranteed to achieve policy improvement for all preferences. Therefore, there remains one critical open research question to be answered: How to boost the sample efficiency of MORL through better policy-level knowledge sharing?
To answer this question, we revisit MORI from the perspective of memory sharing among the policies learned across different training iterations and propose Q-Pensieve where "Pensieve", as illustrated in the novel Harry Potter, is a magical device used to store pieces of personal memories, which can later be shared with someone else. By drawing an analogy between the memory sharing among humans and the knowledge sharing among policies, we propose to construct Q-Pensieve, which stores snapshots of the Q- Q-funciioss of the policies learned in the past iterations. Upon improving the policy for specific preference, we expect that these Q-snapshots could help jointly determine the policy update direction. In this way, we explicitly enforce knowledge sharing on the policy level and thereby enhance the sample use in learning optimal policies for various preferences. To substantiate this idea, we start by considering Q-Pensieve memory sharing in the tabular planning setting and MDPs. Inspired by (Yang)
integrate Q-Pensieve with the soft policy iteration for entropy-regularie et al. 2019), we leverage the envelope operation and propose the Q Q-Pensieve policy iteration for MORL, which we show would preserve the similar convergence guarantee as the standard single- objective soft policy iteration. Based on this result, we propose practical implementation that consists of two major components: (i) We introduce the technique of Q replay buffer. Similar to the standard replay buffer of state transitions, Q replay buffer is meant to achieve sample reuse and improve sample efficiency, but notably at the policy level. Through the use of Q replay buffer, we can directly obtain large collection of Q functions, each of which corresponds to policy in a prior training iteration, without any additional efforts or computation in forming the Q- Q-Pensieve. (ii) We convert the Q-Pensieve policy iteration into an actor-critic off- off-policy MORL algorithm by adapting the soft actor critic to the multi-objective setting and using it as the base of our implementation.
The main contributions of this paper can be summarized as:
We identify the critical sample inefficiency issue in MORL and address this issue by proposing Q-Pensieve, which is policy improvement scheme for enhancing knowledge sharing on the policy level. We then present Q-Pensieve policy iteration and establish its convergence property.
We substantiate the concept of Q-Pensieve policy iteration by proposing the technique of Q replay buffer and arrive at a practical actor-critic type practical implementation.
We evaluate the proposed algorithm in various benchmark MORL environments, including Deep Sea Treasure and MuJoCo. Through extensive experiments and ablation study, we demonstrate the the proposed Q-Pensieve can indeed achieve significantly better empirical sample efficiency than the popular benchmark MORL algorithms, in terms of multiple common MORL performance metrics, including hypervolume and utility.
2 PRELIMINARIES
Multi-Objective Markov Decision Processes (MOMDPs). We consider the formulation of MOMDP defined by the tuple (S, A, Pr D, 6x, A), where S denotes the state space, A is the action space, P S x A S > [0, 1] is the transition kernel of the environment, r :: S x AA - [- -Tmax: T maax] id is the vector- vector-valued reward function with d as the number of objectives, E (0, 1) is the discount factor, 'D is the initial state distribution, Gx : Rdd
R.. is the scalarization function (under some preference vector X E ( and A denotes the set of all preference vectors. In this paper, we focus on the linear reward scalarization setting, i.c., Gx(r) = > r(s, a). as commonly adopted in the MORL literature (Abels et al. 2019 Yang al. 2019 Kyriakis et al.) 2022). Without loss of generality, we let A be the unit simplex. d 1, an MOMDP would degenerate to a standard MDP, and we simply use r(s, a) denote the scalar reward. At each time step E N (O), the learner receives the
2

---

Published as a conference paper at ICLR 2023
observation 8t, takes action ae, and receives reward vector ry. We use π S ((A) to denote a stationary randomized policy, where ((4) denotes the set of all probability distributions over the action space. Let II be the set of all such policies.
Single-Objective
RL. the standard framework of single-objective entropy- regularized RL (Haarnoja et al. 2017 2018 Geist et al. 2019), the goal is to learn optimal policy for an entropy-regularizes MDP, where an entropy regularization term is augmented to the original reward function. For policy क II, the regularized value functions V S and IR can be characterized through the regularized Bellman equations as
Q" S x
Q" a) = r(s, a) + +7 (V" VT = (-1))
(1)
log T (a]s)]]
(2)
where a a temperature parameter that specifies the relative importance of the entropy regularization term. In this setting, the goal is to learn optimal policy л* such that Q" (s, Q* a), for all (s, a). for all π E IT. An optimal policy can be obtained through soft policy iteration, which alternates between soft policy evaluation and soft policy improvement: (i) Soft policy evaluation: For a policy
π, the soft Q-function of π can be obtained by iteratively applying the corresponding soft Bellman backup operator T defined as
T (x r +7B. P( [V (s')),
where (s') E ((( [Q', a') ) a log (क (a' s')) (ii) Soft policy improvement: In each iteration k the policy is updated towards an gvergy-based policy induced by the soft Q-function, i.e.,
exp (10TH (s.
k++1 arg min DKL e'ell
n (-1s)
(4)
n
where II is the set of parameterized policies of interest and = is the normalization term.
Multi-Objective
RL. We extend the standard single-ohjective RI. with entropy regularization to the mujti-objective setting. For each policy и E IT, we define the multi-objective regularized value functions via the following multi-objective version of entropy- Bellman
equations as follows:
Q" ("(sa) r(s, + tB s.a)
(V
(5)
V= (8) (- (x, a log 7 (a
(6)
where 1, denotes d-dimensional vector all ones.
In this paper, our goal is to learn preference- policy <-1;; A) such that for any preference A, AT (T=((-AA)) (s, A) 2 A QT' (s,a), for all (s, a), for all x' IT. For ease of notation, let V (s: A) V X) and Q ( (s, A) (" a; A) in the sequel.
ALGORITHMS
In this section, propose our Q Q-Pensieve learning algorithm for boosting the sample efficiency of multi-objective RL. We first describe the idea of Q-Pensieve in the tabular planning setting by introducing Q-Pensieve soft policy iteration. We then extend the idea develop practical deep reinforcement learning algorithm.
3.1 NAIVE MULTI-OBJECTIVE SOFT POLICY ITERATION
To solve MORL the entropy-regularized setting, one straightforward approach to leverage the single-objective soft policy improvement with the help of linear scalarization. That is, in each iteration k. the policy can be updated by
T641(;; A) arg min DKLL (
exp (2 (2ATO) QT (s, (s,
Z (s)
r'ell
3

---

Published as a conference paper at ICLR 2023
While serves as reasonable approach, designing learning algorithm based on the update scheme in 7 could suffer from sample inefficiency due to the lack of policy-level knowledge sharing: In (7). the policy for each preference À is updated completely separately based solely on the Q-function under X. Moreover, as the update (7) relies on an accurate estimate of the Q-function, the critic learning for the policy of each individual preference would typically require least moderate number of samples. These issues could be particularly critical for a large preference set in practice. While the use of a conditioned policy network (e.g., (Abels et al. 2019)), commonly-used network architecture in the MORL literature, could somewhat mitigate this issue, it remains unclear whether the knowledge sharing induced by the conditioned network can indeed achieve policy improvement across various preferences. As a result, systematic approach is needed for boosting the sample efficiency in MORL..
3.2
Q- Q-PENSIEVE SOFT POLICY ITERATION
To boost the sample efficiency of MORL, we propose enhance the policy-level knowledge sharing by constructing Q Q-Pensieve for memory sharing across iterations. Specifically, Q-Pensieve is a collection of Q-snapshots obtained from the past iterations, and it formed to boost the policy improvement update with respect to the Q-function in the current iteration as these Q-snapshots could offer potentially better policy improvement directions under linear scalarization. Moreover, one major computational benefit of Q-Pensieve is that these Q-snapshots are obtained without the need for any updates or additional samples from the environment (and hence are for free) as they already exist during training. We substantiate this idea by first introducing the Q-Pensieve soft policy iteration in the tabular setting (i.e., |S] and |A|| are finite) as follows:
Q-Pensieve Policy Improvement. In the policy improvement step of the k-th iteration, for each specific X, we update the policy as
exp ( (SIPNew,(A).c'ea Y ('(((.)))1))
Tk+1 (1 A) arg min DKL (=:(1ss;)) r'ell
a
(8)
Ze, (s)
where Zo, is again the normalization term, (A) C A is a set of preference vectors, and Qx is set of Q-snapshots. The two sets W₂ (A) and 2g are to be selected follows:
For (A), the only requirement is that X (A) for all k. The preference sets can be different in different iterations.
Similarly, for Qk. the only requirement is that QT Qk. for all k. The set of Q-snapshots can also be different in different iterations. Hence, the choice of Q is rather flexible.
When choosing W2 (A)= = {}} and Q+ = (Q**), one would recover the update
Policy Evaluation. In the policy evaluation step, we evaluate the policy that corresponds to each preference À by iteratively applying the multi-objectivv softmax Bellman backup operator 7 Mo as
(T Mo Q)(s, X) a() + (f (Qss (''''';;)) log T (()la) (9)
Remark The Q-Pensieve update in (8) is inspired by the envelope Q-learning (EQL) technique (Yang et al., 2019), where in each iteration k, the Q-learning update takes into account the envelope formed by the Q-functions of the current policy т for different preferences. The fundamental difference between Q-Pensieve and EQL that Q-Pensieve further achieves memory sharing across training iterations through the use of Q Q-snapshots from the past iterations, and EQL focuses mainly on the use of the Q-funciion of the current iteration.
Convergence of Q-Pensieve Soft Policy Iteration. Another nice feature of the Q-Pensieve policy improvement step is that it preserves the similar convergence result as the standard singlc-objective soft policy iteration, as stated below. The proof of Theorem 3.1 is provided in Appendix
Theorem 3.1 Under the Q-Pensieve soft policy iteration given by (8) and 9. the sequence of depeedent policies { converges to a policy πO such that X Q (s,a; 2
A Q² (s, a) for all a E II, for all (s,a) E S x A and for all X A.
4

---

Published as a conference paper at ICLR 2023
3.3 PRACTICAL IMPLEMENTATION OF Q-PENSIEVE
In this section, we present the implementation of proposed Q- Q-Pensieve algorithm for learning policies with function approximation for the general state and action spaces.
Q Replay Buffer. Based on 8). we know that the policy update of Q-Pensieve would involve both the current Q-function and the Q-naapshots from the past iterations. To implement this, we introduce Q replay buffer, which could store multiple Q-networks in a predetermined manner (c.g., first-in first-out). Notably, unlike the conventional experience replay buffer (Mnih al.) 2013) of state transitions, Q replay buffer stores the learned Q-networks in past iterations as candidates for forming the Q-Pensieve. On the other hand, while each Q-network would require a moderate amount of memory usage, we found that in practice rather small Q replay buffer is already effective enough for boosting the sample efficiency. We further illustrate this observation through experimental results in Section 4
Next, we convert the Q-Pensieve soft policy iteration into an actor-critic off-policy MORL algorithm. Specifically, we adapt the idea of soft actor critic to Q-Pensieve by minimizing the residual of the multi-objective soft Q Q-function: Let and @ be the parameters of the policy network and the critic network, respectively. Then, the critic network is updated by minimizing the following loss
(Ø A) (x.a)=
(Q (s, A) (r (s.a) a) >Bs s.a) VV (d'))))
(10)
where o is the parameter of the target network and ls is the sampling distribution the state-action pairs (e.g.. distribution induced by replay buffer of state transitions). On the other hand, based on 8 the policy network is updated by minimizing the following objective
C0; A)
Banu
Eawse
sup (A)
{a log (ne (a 8; A)) ATQ' (s,
(II)
The overall architecture of Q-Pensieve is provided in Figure The pseudo code of the Q-Pensieve algorithm described in Algorithm [ in Appendix. The code of our experiments available Notably, in Section we show that empirically relatively small Q buffer size (e.g., 4 in our experiments) can already offer a significant performance improvement.
frelerence adticomminn
G Replay Buller
preference from A
Policies
Snispehot
environment
Added Butter form O-Penereva eve
Repley Butter
Dot comples
Environment
Dritic
Figure 1: The architecture of P--Pesiiee.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of Q-Pensieve on various benchmark RL tasks and discuss how Q-Pensieve boosts the sample efficiency through an extensive ablation study.
5

---

Published as a conference paper at ICLR 2023
4.1 EXPERIMENTAL CONFIGURATION
Popular Benchmark Methods. We compare the proposed algorithm against various popular benchmark methods, including the Conditioned Network with Diverse Experience Replay (CN-DER) in (Abels et al. 2019), the Prediction-Guide
Multi-Objective RL (PGMORL) in (Xu et al. 2020). the Pareto Following Algorithm (PFA) in (Parisi et al. 2014). and SAC (Haarnoj et al. 2018). For CN-DER, as the original EN-DER is built on deep Q-networks (DQN) for discrete actions, we modify the source code of Abels et al. 2019) for continuous control by implementing CN-DER on top of DDPG. Moreover, we follow the same DER technique, which uses diverse replay buffer and gives priority according to how much the samples increase the overall diversity the buffer. For PGMORL and PFA. we use the open-source implementation of (Xu et al. 2020 for the experiments. As these explicit search methods typically require more samples before reaching a comparable performance level, we evaluate the performance PGMORL and PFA under both 1 times and / times (B > 1) of the number of samples used by Q-Pensieve to demonstrate the sample efficiency of Q-Pensieve. For SAC, as the MORL problem reduces single-objective multiple models using single-ohjective SAC (one model for each fixed preference) as a performance reference for other MORL methods.
under fixed preference, train
Performance Metries. In the evaluation, we consider the following three commonly-used perfor- mance metrics for MORL:
HyperVolume (HV): Let R be set of return vectors attained and To R be reference point. Then, we define the HyperVolume as HV == [H(R) 1( E HR))ddz, where (R) ==
{ Rd Ir E R, ro < z - ( and is the indicator function.
Utility (UT): To further evaluate the performance under linear scalarization, define the utility metric UT = Ex
Σ A ATre where the preference À is sampled uniformly from A.
Episodic Dominance (ED): To compare the performance of a pair of algorithms, we define 1 > Ee X rrr"), where r!. rř are the return vectors, and 2 are the episode lengths of algorithm I and 2, respectively. ED serves as a useful metric for pairwise comparison in those problems where the return vectors under different preferences can differ by a lot in magnitude (in this case, HV and UT could be dominated by the return vectors of few preferences).
Episodic Dominance 1D1.
Evaluation Domains. We evaluate the algorithms in the following domains: (i) Continuous Deep Sea Treasure (DST): a two-objective continuous control task modified from the original DST environment. (ii) Multi-Objective Continuous LunarLLnnerr four-objective task modified from the classic control task in the OpenAl gym. (iii) Multi-Objeectvve MuJoCo: modified benchmark locomotion tasks with either two three objectives.
Configuration of Q-Pensieve. For Q-Pensieve, at each policy update, we set the size of the preference set (X) to be 5 (including À and another four preferences drawn randomly) and set the size of the Q replay buffer to be 4, unless stated otherwise.
4.2 EXPERIMENTAL RESULTS
Does Q-Pensieve achieve better sample elliciency than the MORL benchmark methods? Table I shows the performance of Q-Pensieve and the benchmark methods in terms of the three metrics. For each algorithm, we report the mean and the standard deviation over five random seeds. We can observe that Q-Pensieve consistently enjoys higher HV, UT, and ED in almost all the domains. More importantly, Q Q-Pensieve indeed exhibits superior sample efficiency as still outperforms the explicit search methods (i.e., PFA and PGMORL) even if these methods are given 10 times of the number of samples used by Q-Pensieve. Moreover, we can observe that the explicit search methods (i.c., PFA and PGMORL) often have larger HV than the implicit search method (such as CN-DER), while implicit search methods tend to have larger UT. This manifests the design principles and the characteristics of the two families of approaches, where explicit search is designed mainly for achieving large HV and implicit search typically aims for larger scalarized return.
How much improvement in sample efficiency can Q-Pensieve achieve compared to train- ing multiple single-objective SAC models separately? To answer this question, we conduct experiments on 2-objective MuloCo tasks and consider whole range of 19 preference vectors

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of Q-Pensieve and other benchmark algorithms in terms of the three metrics
across ten domains. We report the mean and standard deviation over five random seeds. The ED calculated through comparing each algorithm
multi-objective version of SAC (equivalent to Q Q-Pensieve with the size of the preference set equal to I and without Q replay buffer). We set 3 = 10 for HalfCheetah22, Ant2d, Ant3d, and Hopper3d, set 33 for LunarLanderdd, LunarLander5d, and Hopper5d, and set 3 = 3 for DST2d, Hopper2d, and Walker2d.
Environments
Metrics
PFA (1.x steps)
PGMORL (1.SM steps)
PGMORL (1.5 PM seps)
CN-DER
Q-Fossieve
(1.5M steps)
(1.5M
(1.5M stepe)
HVLx10")
7,4343688
8.67=149
8.10
8.13
5,364471
10.21
DS12d
URXX00")
+.227+6.03
4.904-044
5.02
15.7
ED
0.1310.11
0.10
0.25
0,28
0,21 1:0.7
0,54
HV(x 10*)
0.32 +0.11
0.38=0.11
1,50::000
2.10-:0.10
LumarLanderrd
UTX110")
-0.26 +0.27
1.10=050
3.60+:290
5.10
ED
0.04
0.21 1:012
0.49 10.05
HV(x 10"1)
1.81 20
18870042
8.6420.15
1.48
LunarLanderrd
UT(x 10')
22.77=0.68
4 1.38
0,56:::442
1.07:0.24
1:0.02
0,05
0.49 1:000
0.52:10.02
HVCX 10")
0,734:019
1.3
0.53 17
0.28
2.08:0.54
3.82:20.27
HalfChoctah2d
UT )
0.31
1.0200400
0.288:0.44
0.09
5.09
5..1
ED
0.08:10.10
0.10
0.01 1:000
0.11
0.02 1:000
0.54
HVLx 10")
0.49:1046
2.02
0.63
1.3
0.56:20.16 16
1.33
Hopper2d
UT"
2.89
1.50
1.442246
3.70
1.42:1000
1.0
ED
0.31
0,41
0.31
0.31
0.04 1:000
0,43
HVCx 10")
0,29 37
0.91
3 3.
9.56:20.00
Hopper3d
UT(X10')
0.19110.1
0.31
0.72:10.16
1.39
ED HVLx 10*5)
0.02-4:.02
0.03
0.63 11
0.43
9:42::091
7.24:10.31
Hoppor5d
UT(110")
1,48 28
I 1.6322021
1,76:10043
3.37:10.65
0.18
0.14:20.05
0.21 4:0.5
0.52:1.0.0
HVCx 10")
0.17
0.77=0.53
04
0.13
5.03
1..0
Ant2d
UT(x10 10
00.060001
0.1420.14
-0.21=0.15
-0.18.10.8
3.68112.
3..04:1303
ED
0.22
0.2200002
0.21 4:000
0,21+0003
0.21 4:008
0.60-100.07
HVCx 10 UT(110")
0.41
0.68
13.00:2..11
21.87
Ant3d
0.18:10055
0.25
0.49:10:33
1.14:10.22
ED
0,02
0.03+40.33
0,28
0.56-10.07
HV 10")
0.52 10:00
1.0520.4
0.83
1,28
0.42:1:009
1,12:20.36
Walker2d
UTx110)
0.23
0.95
0.38:1:024
I 1,20:20.67
3.17:10.53
6.37
0.324-006
0.37
0.30++0.10
0.34-40.12
(0.00,, 0.95], (0.1, 0.85).............. [0.95, 0.05)). We train 19 models by using single-objective SAC, one model for each individual preference. Each model is trained for 1.5M steps (and hence the total number of steps under SAC is 28.5M steps). By contrast, Q-Pensieve only uses 1.5M steps in total in learning policies for all the preferences. Figure shows the return vectors attained by Q-Pensieve and the collection of 19 SAC models. Q-Pensieve can achieve comparable or better returns than the collection of SAC models with only 1/19 of the samples. This further demonstrate the sample efficiency of Q P-Pensieve.
1500
15000
28
24
Fonward reward
1200
Forward reward
12000
Treasure reward
20
900
9000
16
600
6000
12
300
Q-Pensieve
SAC
3000
SAC
O-Pensieve
8
4
Q Q-Pesseeee
SAC
-1000
7700
400
100
200
500
100000
-7000
4000
-1000
2000
5000
Control cost
Control cost
Time penalty
(a) Hopper2d
(b) HalfCheetah2d
(c) DST24
Figure 2: Return vectors attained by Q-Pensieve and the collection of single-objective SAC models
under 19 preferences.
7

---

Published as a conference paper at ICLR 2023
Why can Q-Pensieve outperform single-objective SAC in some cases? From Figures a) and (c), we see that Q-Pensieve can attain some return vectors that are strictly better than those of the single-objective SAC models. The reasons behind this phenomenon single-objective SAC, despite that we train one model for each individual preference, it could still occur that single-objective SAC gets stuck sub-optimal policy under some preferences. (ii) By contrast, Q- Q-Pensieve has better chance of escaping from these sub-optimal policies with the help of the Q Q-snapshots the Q replay buffer.
minaly two-fold: (i) Under
To verify the above argument, we design a hybrid SAC algorithm as follows: (a) For the first 10s time steps, this algorithm simply follows the single-objective SAC. (b) At time step 10 it switches to the update rule of Q Q-Pensieve based on the Q Q-snapshots stored in the Q replay buffer of another model trained under Q Q-Pensieve algorithm in parallel. Figure 3 shows the performance of this hybrid algorithm in DST and HalfCheetah. Clearly, the Q Q-Pensieve update could help the SAC model escape from the sub-optimal policies, under various preferences.
25
20
6000
6000
Total return
Total return
10
Total return
4000
Total raturn
4000
10-
2000
2000
SAC
3: BAC
12 SAC
O Hybrid SAC
Hybrid SAC
SAC
-2000
2000
2
10
10
Time Steps
100
Time Steps
Time Steps
x10
Time Stepe x
(a) DST2d
(b) DST2d
(c) HalfCheetah2d
(d) HalfCheetah2d
A [0.9, 0.1]
A = [0.8, 0.2]
À = [0.5, 0.5]
À = [0.6, 0.4]
Figure 3: Comparison of standard single-objective SAC and the hybrid SAC assisted by another Q-Pensieve model trained in parallel.
An ablation study on Q replay buffer. To verify the effectiveness of the technique of Q replay buffer, we compare the performance of Q-Pensieve with buffer size equal to and that without using Q replay buffer (termed "Vanilla" in Figures and HV of both methods. We can see that Q replay buffer indeed leads to a better policy improvement behavior, in terms of both HV and the scalarized returns. However, these figures may sometimes oscillate a lot in the end period. It is because our algorithm finds solutions from another Q-vector, and their inner product Q and preference may be quite close. We can check the points are in the same contour.
and the attained return vectors
1500
4000
2500
2000
Forward roward
1000
Forward reward
5000
1500
Forward
1000
500
1000
Vanilla
Vanilla
Vanila
Q.Pansiave
O-Pensieve
O-Pensieve
.500
500
250
.200
150
-100
.50
-2000
-1000
.000
600
.400
200
800
600
400
.200
Control cost
Control cost
Control cost
(a) Hopper2d
(b) Ant2d
Walker2d
Figure 4: Return vectors attained under preference X [0.5, 0.5] at different training stages (We also plot return vectors under others preference in Figure 7 and Figure ( Appendix). A number x on the red or blue marker indicates that the model is obtained 100 thousand steps.
8

---

Published as a conference paper at ICLR 2023
10 x
10
10
Vanilla Q-Pensieve
10
Vanilla
HyperVoluma
HyparVolume
Q-Pensieve
5
0
2
3
2
3
4
5
Time Steps
x
x10
Time Steps
x10
(a) Ant2d
(b) Walker2d
Figure 5: A comparison in HV between Q-Pensieve with buffer size equal to 4 and that without using Q replay buffer at different training stages.
5 RELATED WORK
The multi-objective RL problems have been extensively studied from two major perspectives:
Explicit Search. A plethora of prior works on MORL updates policy or set of policies by explicitly searching for the Pareto front of the reward space. To learn policies under time-varying preferences,
(Natarajan and Tadepalli 2005) presented to store set of policies, which are to be used in searching for proper policy for new preference without learning from scratch. (Lizotte et al., (0122) leveraged linear value function approximation to search for optimal policies. (Van Moffaert and Nowe] (014))
proposed Pareto Q-learning, which stores the immediate rewards and the non-dominated future return vectors separately and leverage the Pareto dominance for selecting the actions in Q-learning. (Parisi itt al. 2014) presented policy gradient approach to search for non-dominated policies. (Mossalam et al. 2016) solves MORL via scalarized Q-learning along with the concept of prioritizing the corner weights for selecting the preference of the scalarized problem. (Xu et al. 2020) proposed an evolutionary approach to search for the Pareto set of policies, with the help of prediction model for determining the search direction. (Kyriakis et al.) 2022) presented a policy gradient method by approximating the Pareto front via first-order necessary condition. However, the above explicit search algorithms are known to be rather sample-inefficient as the knowledge sharing among different passes of search is limited.
Implicit Search. Another class of algorithms are designed to improve policies for multiple pref- erences through implicit search. For example. (Abels al. 2019) presents Conditioned Network, which extends the standard single-objective DQN to learning preference-dependent multi-objective Q-functions. To achieve invariant MORL, (Abdolmaleki al., 2020) proposed to first learn the Q-functions for different objectives and encode the preference through constraints. Recently, (Yang) ct al. 2019)) proposes envelope Q-functions to encourage knowledge sharing among the Q functions of different the current multi-objective Q-values that any policy can benefit from other preferences' experiences, that make training more efficiently, and (Zhou et al.) (2000) proposed model based envelope value iteration base envelope P-ffuctions, provides an efficient way to get optimal multi-objective Q- Q-functions. Despite that our method is inspired by (Yang et al. 2019). the main difference between our work and theirs is that we boost the sample efficiency of MORL via explicit memory sharing among policies learned during training.
CONCLUSION
This paper proposes Q-Pensieve, which significantly enhances the policy-level data sharing through in order to boost the sample efficiency of MORL problems. We substantiate the idea by presenting Q- Pensieve soft policy iteration in the tabular setting and show that it preserves the global convergence property. Then, to implement the Q- Q-Penssieve policy improvement step, we introduce the Q replay buffer technique, which offers simple yet effective way to maintain Q Q-naapsho.. Our experiments demonstrate that Q- QePessieve is a promising approach in that can outperform the state-of oftte--f-the-rtt MORL methods with much fewer samples in a variety of MORL benchmark tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This material is based upon work partially supported by the National Science and Technology Council (NSTC), Taiwan under Contract No. 110-2628-E-A49-014 and Contract No. 111-2628-E-A49-000 and based upon work partially supported by the Higher Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education (MOE), Taiwan.
REFERENCES
Yiqing Ma, Han Tian, Xudong Liao, Junxue Zhang, Weiyan Wang, Kai Chen, and Xin Jin. Multi-objective congestion control. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 218-235, 2022.
Abbas Abdolmalcki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pages 22. PMLR, 13-18 Jul 2020. URL httpps:/prrcceddiii .ww.ingssmm...mm..nm.mmmmmm.m... ml ww....eeeevvlll. latess//v1//abolll/e lek 120 html
Diederik M Roijers and Shimon Whiteson. Multi-objective decision making. Synthesis Lectures on Artificial Intelligence and Machine Learning, 11(1):1-12, 2017.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction- guided multi-objective reinforcement learning for continuous robot control. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10607-10616. PMLR, 13-18 18 Jul 2020. URL https
//proceedings wng.eeeediggg..... mlr, wwwppprsssvviee wvtss//vl////////////.../ xu20h. html
Panagiotis Kyriakis, Jyotirmoy Deshmukh, Paul Bogdan, Pareto policy adaptation.
International
Conference on Learning Representations, 2022.
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International Conference on Machine Learning, pages 11 11-20. PMLR, 2019.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objectivee reinforce- ment learning and policy adaptation. Advances in Neural Information Processing Systems, 32, 2019.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy- based policies. In International Conference on Machine Learning, pages 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
Matthicu Gcist, Bruno Scherrer, and Olivier Pietquin. A thcory of regularized markov decision processes. In International Conference on Machine Learning, pages 2160-2169, 2019.
Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 13/2.55/ 2013.
Simone Parisi, Mattco Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli. Policy gradient approaches for multi-objective sequential decision making. In International Joint Conference Neural Networks (IJCNN), pages 2323-2330. 2014.
Sriraam Natarajan and Prasad Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In International Conference on Machine Learning, pages 601-608, 2005.
Daniel J Lizotte, Michael Bowling, and Susan A Murphy. Linear fitted-q iteration with multiple reward functions.
Journal of Machine Learning Research, 13(1):325333255, 2012.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforeement learning using sets of pareto dominating policies. Journal Machine Learning Research, 15(1)334833351, 2014.
Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning, 2016. UTR.itttss///axi..orr////////
02707
10

---

Published as a conference paper at ICLR 2023
Dongruo Zhou, Jiahao Chen, and Quanquan Gu. Provable multi-objective reinforcement learning with generative 2020.
models. arXiv preprint
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. Proceedingy of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10. 1609/aai. v3291.11798.
Yunhao Tang and Shipra Agrawal. Discretizing contimuous action space for on-policy optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 3404))59811158888 Apr. 2020. doi: 10.1609/aai,, v34i04.
6059, bttps:/s:///]s aaaal http://innxx....php.p/m oatpa//ccg...naaoapa//////aaaaaa

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOF OF THEOREM 3.1
Before proving Theorem 3.1 first present two supporting lemmas follows. To begin with, establish the policy improvement property the Q-Pensieve update. Recall that the Q-Pensieve policy update is that for each preference À E A,
exp (SUP)EW((A))QEE... a Q(ss,;;1))
T++1(11;;)) = arg min DKL
7' (r((11;; (+]8;2 A)
r'Ell
Zo(s)
=:L(x(A)
(12)
Lemma 1 (Q-Pensieve Policy Improvement) Under the Q- policy improvement update, we have Q (s, a; A) S X NTTH++(s,a A), for all state-action pairs (8,a) S x A, for all preference vectors X E A, and for all iteration k E N (0).
Proof (Lemma 1 By the update rule in (12), we know that T++1 is a minimizer of L(x'; A) and hence [ X) < L(Ta; A). This implies that for each state 8, we have
sup la log (als;; A) a ('
X X + log log Ze. (s)
<Banz((-1))
1 sup A log (als; a (A)
XTQ"(s,a + log Zo
(8) (
(13)
Since Ze. only depends on the state, the inequality 13 reduces to
E BawTH+((11))
A T La log 1
A ('((ss;;)'i) (E
sup
1a log (a)s:
sup QEE
A QQ'(s.
(14)
Next, we proceed to consider the multi-objective soft Bellman equation as follows: A TTh (so, ao: A) AT T ao)
(15)
> (0 (1)1 [QT A) a- A lo A))]
(16)
sup < Paan- NEWL(N),'EE
(T(((s1,1;)))) - a A A log(TA (an
(s1; A))
(17)
sup < (1 QE
A X ('((s),11;')) a > 11 (ag A))
(18)
$ -- - ( a A (ag
+ P(aaaa
T (((111)) - P 11 log(T A))
sup E Plan Q² (52, 사|1] (19)
(A)
< ( ( a A la
+ BB P( (191))
A (()
log( (110)aa A))]
sup Esg Peg. [ Q' (2(88, 02; A)
(A)
(20)
12

---

Published as a conference paper at ICLR 2023
(21)
B-7 Ploo0 (a A (ogg(m+)1((()):22))
+ Ep
Σ 7B'BB (s))
(22)
yB Plo (1041)) [ad la log( (10 2)) 8c. 0rl
(23)
-X = do: A) A (ss0 uo),
(24)
where (16) follows from the multi-objective soft Bellman equation, 17 holds by the sup operation and the fact that QT 2k.
(88) follows from (14) 19 holds by applying the multi-objective soft
Bellman equation to (s1, d1: X) (20 again follows from the inequality in (4). 23 is obtained by
unrolling the whole trajectory, and 24 holds by the definition of Q"
Lemma OMjlti-Objective Soft Policy Evaluation) Under the multi-objectivee soft Bellman backup operator MO with respect to a policy π and some Q(D) : SSx - R the sequence of intermediate Q-functions { during policy evaluation is given by Q Q = T 1,), for all
? NU [0). Then, Q converges to the soft Q 2-function of π., as + oc.
Proof (Lemma 2 This can be directly obtained from the standard convergence property of iterative policy evaluation (Sutton and Barto, 2018) in two steps: (i) Define the entropy- entropy-augmented reward as rs, a; TT) := r(s. a) PPBla- a aa(a)(())) (cJa)(a (a log (a]s)]d)) which is a bounded function. (ii)
Then, rewrite the policy evaluation update as
Q (s,ssa) t r(s, a)) + P 3) (s",
(25)
This completes the proof.
Now are ready to prove Theorem 3.1
Proof (Theorem 3.1 Note that by Lemma the sequence A TTte is monotonically increasing. As each element in QF is bounded above for all π E given the boundedness of both the reward and the entropy term, the sequence of policies shall converge to some policy π². The remaining thing
exp (SsPAAeeass q' (A::AA)
show that n² is optimal: (i) Define LT) :- DRL. <(-
2a
(ii) Upon convergence, we shall have T.g- (1- < Tig (T(+1s)) for all a II. Using the same iterative argument in the proof of Lemma [ get A Q (s, A NTQT(s, a; X) for all (8, a) E A and all À e
B DETAILED CONFIGURATION OF EXPERIMENTS
B.1 DETAILS ON THE EVALUATION DOMAINS
Continuous Deep Sea Treasure (DST): DST is classical multi-objective reinforcement learning environment. We control the agent to find the treasure, while the further the treasure is, the higher its value. In other words, the agent needs spend more resources (-1 penalty for each action) to get the more precious treasure. To extend DST to continuous space, we modify the simple four direction movement to the movement in a circle, set the 3 of DST 3.
Multi-Objectio Continuous LunarLander: We modify LunarLander to the multi-objective version by dismantling the reward to main engine cost, side engine cost, shaping reward, and result reward. conducted in environments with 2 or objectives, created
Since the past MORL methods an environment with 4 and 5 objectives to show our method can be used in high dimension objectives environments, we the of LunarLander 5.
MuloCo: We divide the scalar reward in MuloCo environments into vector rewards. What's more, we amplify the weight of the control cost to make the magnitude of each reward element similar.
HalfCheetah2d: 2 objectives as forward speed, control cost (S C R 17,AC C Rº), 1000 times for control cost, and 10.
13

---

Published as a conference paper at ICLR 2023
Hopper2d: 2 objectives: forward speed, control cost (S RR C RR), 1500 times for control cost, and 3 = 3.
-Hopper3d: 3 objectives: forward speed, jump reward, control cost (S C R A R²), 1500 times for control cost. The jump reward is times of the difference between current height and initial height, and 3 = 10.
Hopper5d: 5 objectives; forward speed, control cost of each of the 3 joints, and healthy reward (S _ IR , S R²), 1500 times for control cost, and /
Ant2d: 2 objectives: forward speed, control cost (S C R A C R*), times for control
cost, and | = 10.
Ant3d: objectives: forward speed, control cost, healthy reward (S C R A C R*),
times for control cost, times for healthy reward, and / 10.
Walker2d: objectives: forward speed, control cost (S C R A C R6), 1000 times for control cost, and 3 =
B.2 HYPERPARAMETERS
B.2.1
HYPERPARAMETERS OF Q-PENSIEVE
We conduct all experiment on baselines with following hyperparameters.
Table 2: Hyperparameters of Q-Pensieve.
Parameter
Value
Optimizer
Adam
Learning Rate
0.0003
Discount Factor
0.99
Replay Buffer Size
1000000
Depth of Hidden Layers
2
Number of Hidden Units per Layer
256
Number of Samples per Minibatch Nonlinearity
256
ReLU
Target Smoothing Coefficient
0.005
Target Update Interval
Gradient Steps
B.2.2 HYPERPARAMETERS OF PGMORL AND PFA
For PGMORL and PFA, we use the hyperparameters as provided in Table 3
n: the number of reinforcement learning tasks.
total steps: the total number of environment training steps.
mw: the number of iterations in warm-up stages.
my the number of iterations in evolutionary stages.
Paumi the number of performance buffers.
Psze: the size each performance buffer.
nweigh the number of sampled weights for each policy.
sparsity: the weight of sparsity metric.
C
PSEUDO CODE OF Q-PENSIEVE
We provide the pseudo code in Algorithm as follows.
14

---

Published as a conference paper at ICLR 2023
Table Hyperparameters of PGMORL and PFA.
Environments
total_steps
m/
m4
P
num
P
size
Twwiight
sparsity
DST2d
1.5 x x10
80
20
100
2
LunarLander4d
35
7.5 x10
40
10
400
2
10
LunarLander5d
35
7.5 x10
40
10
400
2
7
-106
HalfCheetah2d
1.5 x10"
80
20
100
2
7
-1
Hopper2d
4.5 x10
200
40
100
2
-1
Hopper3d
15
1.5 x10"
200
40
200
2
7
-10
Hopper5d
35
7.5 x10
200
40
400
2
7
- -106
Ant2d
1.5 x10"
200
40
100
2
Ant3d
15
1.5 X10"
200
40
200
2
7
-106
Walker2d
5
4.5
x10
80
20
100
2
7
-1
Algorithm 1: Q-Pensieve
Input :01, Φ2, 0, preference sampling distribution Px, number preference vectors Nx, the soft update coefficient т, actor learning rates 7mm critic learning rates n7Q
6 - 01.02 12 + Ø2:
M 0
Initialize replay buffer;
B + 0;
for each iteration do
sample weight X from A according Px
for each environment step do
ae To x-(-1s;; X);
8:41~ P(-ss, a;);
M M {(st, de, rSt,, 8t+1));
10
for each gradient step do
11
sample NN preferences and add them to set W; W + W W{A}:
Φi + Oi no Vo.
(Ø;; A), BU[d; )} for e [1,2]:
compute Vo with eq. 11 with W;
14
i レ 0 nm V Lx A);
Φi -- TW: (1 T) for i 11,
5000
Forward reward
4000
Forward reward
1000
3000
2000
500
Without Q- Pensieve
Without O-Pensieve
1000
Buffer size =
Buffer 2
Butter
Buffer size 4
3000
-500
2000
10000
-500
0
500
Control cost
Control cost
(a) Ant2d
(b) Hopper2d
Figure 6: Return vectors attained under different Q replay buffer sizes of Q-Pessieve
15

---

Published as a conference paper at ICLR 2023
2000
4000
3000
2500
Forward reward
Forward reward
Forward roward
2000
1500
1000
2000
1000
500
1000
500
Vanilla
Vanilla O-Pensiave
Vanilla
O-Pensieve
O-Pensieve
800
600
400
2000
2500
2000
1500
-1000
-00
2000
15000
1000
.500
Control cost
Control cost
Control cost
(a) Hopper2d
(b) Ant2d
(c) Walker2d
Figure 7: Return vectors attained under preference À 00.9, 0.1] at different training stages. A number z on the red or blue marker indicates that the model is obtained at 10 z million steps.
120
400
200
Forward reward
100
Fonward reward
300
Forward roward
150
80
200
50
60
100
Vanila
Vanilaa O Ponsiave
-50
Vanilla
G-Pensieve
Pensieve
-10
-8
-2
-400
-300
-200
-100
-100
-80
-50
-40
-20
Control cost
Control cost
Control cost
(a) Hopper2d
(b) Ant2d
(c) Walker2d
Figure 8: Return vectors attained under preference À 10.1, 00.1 at different training stages. A
number x on the red or blue marker indicates that the model is obtained at 10 x million steps.
D COMPARISON OF LEARNING CURVES
We demonstrate the learning curves of Q- P--Pesiieee and the benchmark methods. In Figures 15 we can find that Q Q-Pensieve enjoys the fastest learning progress in almost all tasks and preferences. Notably, as PGMORL is evolutionary method and does explicit search for policies for only small set of preferences vectors in each generation, the typical learning curve (in terms of expected total reward) under given preference is not very informative about the overall learning progress. Therefore, regarding the learning curves, we compare Q-Pensieve with CN-DER and PFA. Moreover, PFA cannot handle tasks with more than two objectives (this fact has also been mentioned in (Xu et al. 2020),, PFA is evaluated only in tasks with two objectives.
PFA
B O Pensieve ON-ERR PFA
PFA
Average return
C.Peninec GNDER
Average return
300
-B- Pensteee ONEER
1500
200
Average
1000
500
-50
08
-100
.500
10
15
10
10
Time Steps
Time Steps
10
Time Steps
(a) À 0..1
(b)
[0.5, 0.5]
(c) (0.9. 0.1]
Figure 9: Average return in Hopper2d over 5 random seeds (average return is the inner product of the reward vectors and the corresponding preference).
16

---

Published as a conference paper at ICLR 2023
200
2000
4000
B8
6-8
OO-CNDER
Ra
Average retum
00
Avarage ratur
PFA
*A PFA
1000
BgC
-200
Average
1000
-400
G-Pansieve
09
-600
Time Steps
110
Time Steps
x10
Time Steps
x10
(a) À = 0.9
(b)
[0.5,
(c)
[0.9, 0.1]
Figure 10: Average return in Ant2d over random seeds (average return is the inner product of the reward vectors and the corresponding preference).
6000
-B-G G-Persseeve DER PFA
OuParsieve CN-DER PFA
B8Ba
100000
Averoge return
-500
Averag return
4000
8000
-1000
2000
-1500
4000
DePensiece CI-DER PPA
2000
2000
-250
-2000
15
Time Steps
100
Time Steps
x100P
Time Steps
x10
(a) A 0.91
(b) A [0.5, 0.5]
(c) A [0.9 0.1]
Figure 11: Average return in HalfCheetah2d over 5 random seeds (average return is the inner product of the reward vectors and the corresponding preference).
1500
2500
-B
Pens
Average return
Average retum
CN-
Wer
2000
PFA
DER
Ra AM y
1000
-100
1000
-200
-B- Pensiove PFA
C---DEER
-300
5000
6
10
15
0
5
10
15
10
Steps
Steps
x1088
x10
(a) A
0.11
(b)
00.5, 0.5)
(c)
(0.9. 0.1]
Figure 12: Average return in Walker2d over random seed (average return is the inner product of the reward vectors and the corresponding preference).
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, we compare Q-Pensieve with the baseline methods, discuss how the performance of Q- Q-Pensieve can be further improved through hyperparameter tuning, and then demonstrate the model generalization of Q-Pensieve.
E. COMPARISON WITH THE ENVELOPE Q Q-LERRNING ALGORITHM
The Envelope Q-Learning algorithm and its neural version Envelope DQN Yang et al. 2019 presume that the action space is discrete. To adapt Envelope DQN the continuous control tasks
17

---

Published as a conference paper at ICLR 2023
-S-Oh-cE
100
DN-Det
Awrage
Awergge
000000000000
1000
100
200
+0
Steps
(a) A 00.33, 0.33, 0.33]
(b) A
00.1 0.8]
(c) A (0.1,0.8,0.1
(d) X
00.1,0.1,0.8
Figure 13: Average return in Ant3d over random seeds (average return is the inner product of the reward vectors and the corresponding preference).
---0 Pursann
3000
3.
-N
1850
Aeeern age remum
Av mage retur
2000
RORB
Avernge
1000
100
1000
200
110
500
-200
10
16
Stepe
Tire Stepss
101
Time Stape
110
Steps
10
(a) 0..33, 0.33, 0.33]
(b) 0.8
(c) A 0.11 0.1]
(d) À (0.8, 0.1, 0.1]
Figure 14: Average return in Hopper3d over 5 random seeds (average return is the inner product of the reward vectors and the corresponding preference).
Avergg neturn
Bee Bees
Averge
Average
-10
-10
-50
Pensieve ONLERR
-20
e G-Ponsleve
+ ON 3-O-Pureer
4 C--DEE
-30
20
0
5
10
10
15
10
15
x10
10
Time
x10
A
[0.25, 0.25, 0.25, 0.25]
(b) A [0.05, 0.05, 0.05, 0.85]
(c) A [0.05, 0.05, 0.85, 0.05]
150
200
100
Averege retum
Average relum
100
50
PePnsleeee
5
15
10
15
Time Sleps
10
Time Staps
x10
(d) [0.05, 0.85, 0.05, 0.05]
(e) A
[0.85, 0.05, 0.05, 0.05)
Figure 15: Average return in Lunarlander4d over random seeds (average return is the inner product of the reward vectors and the corresponding preference).
considered our paper (including MuJoCo and continuous Deep Sea Treasure), we take the open- source implementation in (Yang al. 2019) and apply action discretization, which has been shown to be also quite effective in MuJoCo control tasks (Tavakoli et al. 2018 Tang and Agrawal 2020). We compare Envelope DQN with Q Q-Pensieve in both Hopper3d and DST2d environments. For Envelope DQN, we set the number of bins for each action dimension to be 11 and for DST2d (actions are 2-dimensional) and Hopper3d (actions are 3-dimensional) respectively, based the suggestions
18

---

Published as a conference paper at ICLR 2023
provided by Tavakoli al., 2018). Table 4 shows the performance of Q-Pensieve and Envelope DQN in terms the two metrics. We can observe that Q Q-Pensieve outperforms Envelope DQN by large margin in the above two popular multi-objectivve tasks.
Table 4: Comparison of Q-Pensieve and Envelope DQN in terms of the two metrics across two domains. We report the mean and standard deviation over five random seeds.
Environments
Metrics
Envelope DQN Q-Pensieve
HV(x 10*)
7.02+024
10.21
DST2d
UT(x 10")
4.61
7.31 1:911
HV(x10*)
0.43
13.31
Hopper3d
UT(X109)
-0.394026
4.08
E.2 EMPIRICAL STUDY OF Q-PENSIEVE
Q Replay Buffer Size: One could expect that a larger Q buffer size could help provide a more diverse collection of Q snapshots and thereby better boost the policy improvement update in each iteration. On the other hand, in practice, the required memory usage also scales with the Q buffer size. evaluate Q-Pensieve under buffer sizes = 2, 4, 6 and compare it to that without using a Q replay buffer in Ant2d. Table show that empirically relatively small Q buffer size already offers significant performance improvement.
Q Replay Buffer Update Interval: To ensure that the Q snapshots in the Q replay buffer are rather diverse, would suggest that the update interval shall not be too small (otherwise the Q snapshots in the buffer would be fairly similar). Moreover, as general this update interval can be viewed hyperparameter to be tuned (similar to the update interval of the target networks in many RL algorithms). We further do an empirical study on the performance of Q-Pensieve under different update intervals. We Q-Pensieve with different update intervals in Ant2d for 1500k steps. Table show that the hypervolume is not sensitive to the update interval, and the performance in UT can potentially be further improved through hy hyperparameter tuning.
E.3 MODEL GENERALIZATION OF Q-PENSIEVE
To demonstrate that the critic model with the preference vector can generalize well, we define metric for the critic as
Ceiiii = Es~S, (s, a; A) Queo (s, a; A) 12,
(26)
where Q is the action-value function learned by critic network and Qine is the true Q function calculated by Monte Carlo method. Table 7 and Table shoow the Cenue under various preferences
different training stages in Hopper2d and HalfCheetah2d respectively. Note that the true Q values are typically in the range of 1000 to few thousands. Therefore, we can see that Leniti under various preferences is indeed pretty low, which indicates that the critic model can generalize well across preferences.
Table Comparison of Q-Pensieve with different Q replay buffer size terms of the two metrics in Ant2d over five random seeds.
Metrics
Without Q Buffer
Size =
Size = 4
Size =
HV(X 10")
0.93
0.99
1.00
1.27
UT(x10*)
8.17
2.75
14.04130
15.31
19

---

Published as a conference paper at ICLR 2023
Table 6: Comparison of Q-Pensieve with different replay buffer update interval in terms of the two metries in Ant2d over five random seeds.
Metries
Interval = 500
Interval 1000
Interval 1500
Interval 2000
HV(x 10")
8.30=060
9.90=231
8.63
1.53
UT(x103)
8.94
12.45
11.83
13.33
Table 7: Leme Hopper2d over five random seeds.
Preferences
100K steps
500K steps
1000K steps
1500K steps
1=
23.74
43.59
38.96
41.47
[=
22.21
14.82
15.10
14.65
{=
27.59
14.45
24.91
11.59
{=
47.54
10.25
15.17
11.52
[=00900.11
87.01
46.95
84.76
31.96
Table 8: Lenie in HalfCheetah2d over five random seeds.
Preferences
100K steps
500K steps
1000K steps
1500K steps
[=
62,92
139.39
118.13 135
92.35
[000.00.77
134.1
174.81
132,305
131.71
{=
192.47
113.20
98.18
90.07
169.70
87.87
83.21
74.69
A= [0.9.0.1]
146.71
66.92
65.09
63.85
20


Reviews:
[{"review_id": "ODH92tZ4Up", "reviewer": "Reviewer_ecaS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for improving sample efficiency in multi-objective reinforcement learning (MORL). Like universal value function approximators, a state-action value function represents a large number of value functions for any preferences. The most important contribution is the policy improvement step utilizing a set of previously learned Q-functions. The proposed method is evaluated on the MuJoCo benchmark tasks, and the experimental results show that the proposed method performs better in several metrics. Additionally, the proposed method outperforms a single-objective SAC because the proposed method can escape from sub-optimal policies. ", "summary_of_the_review": "Overall, this paper is well-written, and the experimental results are promising. However, the proposed method becomes more appealing if it is evaluated on problems with many objectives. ", "strength_and_weaknesses": "Strength\n1. The authors propose a novel multi-objective actor-critic algorithm inspired by Yang et al. (2019). \n2. To improve sample efficiency, a Q-replay buffer is proposed, and it is efficiently integrated with the proposed algorithm. \n3. The experimental results show that the proposed method performs better than the baselines with respect to several metrics. \n\nWeakness\n1. The sampling distribution is not discussed explicitly. If the preference is uniformly sample, applying the proposed method to problems with many objectives is difficult. \n2. The scalability of the proposed method is not discussed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LNfwn1hXY5", "reviewer": "Reviewer_VKcR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method, dubbed Q-Pensieve, for reusing critic knowledge across training iterations in a multi-objective reinforcement learning (MORL) setting. Specifically it proposes a modification of Soft Actor Critic (SAC) in the multi-objective setting that uses snapshots of previous Q functions in the learning update for which a convergence proof is provided. Finally the proposed algorithm is evaluated on a number of continuous control domains against other benchmarks and across 3 metrics and some ablations are considered. ", "summary_of_the_review": "Overall I think this paper is generally well presented and meets the acceptance bar. I am happy to increase my score once my clarification questions and results are presented. ", "strength_and_weaknesses": "The ideas presented in this work are generally well presented and the various sections do a good job of introducing the various background concepts. The method itself is quite easy to understand although I have not looked into the proof in the Appendix closely. \n\nWhile I generally think the paper is well written, there are some questions that I had regarding the implementation and results. One point in particular that could be improved is that the authors should mention earlier in the text the expected size of the replay buffer being used for the Q snapshots. It was not until the experiments that I saw the replay size of 4 was mentioned. On first reading it was not clear how big the replay size was and since this can have a large memory footprint the size matters quite a bit.\n\nIn the experimental section episode dominance is defined as a metric that directly compares a pair of algorithms. In Table 1 this metric is presented for all the baseline methods - how is it computed here? My understanding is that this metric defines on average for how many preferences does algorithm 1 out-perform algorithm 2. But it isn’t clear what the two algorithms chosen in practice are.\n\nThe idea of using different Q networks for learning has also been studied in single objective RL. For example, TD-3 often outperforms SAC in single-objective RL using 2 Q networks in the update rule. There are differences in the two approaches but I am curious how well a TD-3 like update (extended to the multi-objective setting) compares to the method being proposed in the paper? Even with a Q buffer size of 4, the memory footprint of the proposed approach is quite large so it would be compelling if a simpler method worked as well.\n\nAdditionally the mechanism of information sharing through the Q replay is not clear to me. How is the replay buffer implemented? With such a small size is it effectively a queue containing the last N snapshots of learning? What happens when the replay is updated every N/2nd iteration and only stores N/2 snapshots instead say? As things stand I think the idea has merit but I think more can be done to understand exactly where the benefits are coming from which could perhaps simplify the method.\n\nEDIT (Post author discussion period):\n\nI would like to thank the authors for responding to my feedback and updating the paper with my comments. After looking through the discussion between the authors and the other reviewers and the latest draft of the paper, I think this is in good shape to be accepted. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KR73njOqz32", "reviewer": "Reviewer_g5HG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper study the problem of  multiple objective reinforcement learning. As previous methods that search for Pareto front are not sample efficient. The paper propose the Q-Pensieve method, which updates the policy with learned Q-networks from past iterations. The paper provide some theoretical analysis. Finally, experiments on several multi-objectives environments validate its effectiveness over stoa baselines.", "summary_of_the_review": "Overall, the paper is an interesting. A new MORL method is proposed to exploit the learned critic in previous rounds. The performance improvement is significant. But some detailed analysis and the discussion of the limitation are missing.", "strength_and_weaknesses": "Strength:\n\n1.The paper study an important problem and the paper is well written.\n\n2.The performance improvement is significant.\n\n3.Theoretical analysis is interesting.\n\nWeakness:\n\n1.The idea that uses replay buffer is not quite novel. It is common for rl methods trained with replay buffer. I think it is novel to use the learned critic in previous rounds, can this technique helps for single RL algorithms?\n\n2.The performance curve of each algorithm during the training is missing.\n\n3.For the experimental details, how is the preference vector decided in practice?\n\n4.The critic function is related with the preference vector, can does the model generalize well on this variable?\n\n5.How the performance change when the number of objectives increases?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-bQeZhZHh6", "reviewer": "Reviewer_jdUh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents Q-Pensieve, a method using versions of past Q functions to update a new Q function. This method enables information sharing across policies. The authors instantiate the idea in a soft actor-critic algorithm. Experiment results on DST, LunarLander, and several MuJoCo environments are provided where Q-Pensieve outperforms other baseline methods. ", "summary_of_the_review": "The proposed Q-Pensieve algorithm appears to be simple and effective. However, an important baseline is missing.\n\n==========================\nI want to thank the authors for writing a detailed response. My main concern, the comparison with envelope Q-learning, is adequately addressed. I would encourage the authors to expand the comparisons to include all the environments in a later revision.\n\nOn the Q replay buffer size study, it seems the performance is monotonically increasing with the buffer size. It would be good to add a discussion on the trade-off of the performance with memory overhead.\n\nOverall, I decided to raise my score from 5 to 6.\n\n", "strength_and_weaknesses": "Strength:\nThe Q-Pensieve idea is well-motivated as one can interpret this as an extension of the envelope Q-learning algorithm by Yang et. al. It is good that the Q-Pensieve retains the same convergence policy iteration results. \n\nThe empirical results are good as well, with Q-Pensieve outperforming other baselines in three different performance metrics. My understanding is that this is a fairly straightforward method to plug in so I appreciate its effectiveness.\n\nWeakness:\nThe major weakness is the absence of the envelope Q-learning algorithm by Yang et. al. As the Q-Pensieve algorithm is closely related, the authors should provide a comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster