Solving single-objective tasks by preference multi-objective reinforcement learning
Authors: Jinsheng Ren
Shangqi Guo
Feng Chen
Venue:  (2020)
Keywords: reinforcement learning
single-objective tasks
multi-objectivization

Abstract: There ubiquitously exist many single-objective tasks in the real world that are inevitably related to some other objectives and influenced by them. We call such task as the objective-constrained task, which is inherently a multi-objective problem. Due to the conflict among different objectives, a trade-off is needed. A common compromise is to design a scalar reward function through clarifying the relationship among these objectives using the prior knowledge of experts. However, reward engineering is extremely cumbersome. This will result in behaviors that optimize our reward function without actually satisfying our preferences. In this paper, we explicitly cast the objective-constrained task as preference multi-objective reinforcement learning, with the overall goal of finding a Pareto optimal policy. Combined with Trajectory Preference Domination we propose, a weight vector that reflects the agent's preference for each objective can be learned. We analyzed the feasibility of our algorithm in theory, and further proved in experiments its better performance compared to those that design the reward function by experts.

Under review as a conference paper at ICLR 2020
SOLVING SINGLE-OBIECTIVE TASKS BY PREFERENCE MULTI-OBJECTIVE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
There ubiquitously exist many single-objective tasks in the real world that are in- evitably related to some other objectives and influenced by them. We call such task as the abjective-constraine
task, which is inherently a multi-objective prob- lem. Due to the conflict among different objectives, trade-off is needed. A common compromise is to design scalar reward function through clarifying the relationship among these objectives using the prior knowledge of experts. How- ever, reward engincering is extremely cumbersome. This will result in behaviors that optimize our reward function without actually satisfying our preferences. In this paper, we explicitly cast the objective-constraine objective reinforcement learning, with the overall goal of finding Pareto optimal policy. Combined with Trajectory Preference Domination we propose, a weight vector that reflects the agent's preference for each objective can be learned. We analyzed the feasibility of our algorithm in theory, and further proved in experi- ments its better performance compared to those that design the reward function by experts.
task as preference multi-
INTRODUCTION
In recent years, Reinforcement Learning (RL) has achieved great success in many complex tasks, which commonly has a well-defined reward function (Mnih et al. 2015 2016 Silver et al. 2016). However, there ubiquitously exists a type of task, which we call objective-constrane
task, that is quite important but has not yet been well settled, As for the objective-constraine task, though only a single objective (denoted the primary objective in the following context) needs to be optimized, its difference from most RL scenarios lies that there are some additional objectives in the envi- ronment. On one hand, in order to solve the primary objective better, it is necessary to optimize the additional objectives simultaneously. On the other hand, the additional objectives affect the pri- mary objective more or less, whereas it is usually not clear how these additional objectives affect the primary objective. Take DOOM (Kempka et al. 2016) as an example, the primary objective of the agent is to kill many enemies as possible. Mcanwhile, there additional objectives: picking up bullets and medicines, which may help kill more enemies in general, but their relationship with killing is still complex and ambiguous every specific moment.
There have been two kinds of ways to solve the objective-constrainee on the primary objective exclusively, hoping that the agent could learn more flexible policies. Take DOOM as an example, the environment gives back a reward whenever an enemy is killed by the agent (see Figure T(a). The deficiencies of this setting include: 1) Due delayed reward (Arjona- Medina et al. 2018), it is difficult to find the relationship among the primary objective (killing enemies) and the additional objectives (picking up bullets and medicines).
task. The first way focuses
Even if the agent finds that the additional objectives can help solve the primary objective better, the direction of policy updates may not be biased toward this behavior. The reason is that many timesteps are spent on these additional objectives without reward. Therefore, even if the cumulative return is greater, the cumulative discount return is even smaller. The phenomenon called myopic policy (Bertsekas & Tsitsiklis 1996), where too low discount leads to highly sub-optimal policies. The second way is trying to clarify the relationship among these objectives using the prior knowledge of experts (see Figure l(b).. One of the popular methods is to design scalar reward that properly weighs the importance of each objective. However, this will often result in behaviors that optimize our reward function without actually satisfying our preferences (Russell 2016 Everitt al. 2017).

---

Under review as a conference paper at ICLR 2020
5/100
10/100
Kill Eaanis
Ball Sacine
*
Frad Medkine
Find Ammol And Mathens
Pind Arpmoo
Kill Emanio
(a) Focusing on the primary ob- jective only
(b)
Clarifying the trade-offs
(c) Learning the relationship among objectives
among objectives by hand
Figure 1: Comparison performance in DOOM between our algorithm and the other two mainstream works. (a) Due to the reward signal from killing only, it is difficult for the agent to find other objectives, even if they are useful for the task. (b) Trying to design the reward function using the prior knowledge of experts. This usually leads to the agent tending to get easy rewards rather than satisfying our preference. (c) Instead applying excessive prior knowledge, a learnable weight vector is introduced to weigh the reward function of each objective and let the agent understand the purpose of the task.
(Knowles et al. 2001) the process of transforming single objective prob- lem into a multi nulti-objecttiv problem. Recent research (Bleuler et al.) 2001 Knowles et al. 200T) indicates that the methods from Pareto-based multi-objective optimization (MOO) may be helpful to solve single-objective optimization problems. In this paper, we reveal that transforming a single- objective reinforcement learning (SORL) to multi-objective reinforcement learning (MORL) is beneficial for solving the primary problem. Rather than analyzing trade-off's of these objectives by hand, one of the keys of multi-objectivvzzaion is letting the agent learn the relationship among these objectives and understand the purpose of the task see Figure I(c) It means that, instead of apply- ing excessive prior knowledge, a learnable weight vector will introduced to weigh the reward function of each objective. Different from obtaining a set of policies that approximate the Pareto front in MORL (Liu et al.) (207)). we are merely interested in a Pareto optimal policy which fully reflects our preference information. Therefore, the known method (Brys al. (017) is not suit- able for solving this task. The problem which only obtains some Pareto optimal policies in MORL called preference MORL. The core idea underlying preference MORL is to modify the Pareto dominance relationship, so as to enhance the selection pressure of the algorithms and guide the algorithm to converge quickly the preference region. Consequently, ment to estimate the agent's s performance, which is called Trajectory Preference Domination. This can not only improve the efficiency of the algorithms solve the optimization problems, but also reduce the computational complexity. In detail, instead of using a scalar return that combines all objectives, we adopt new method of comparing agent's behavior where more than one measure is provided. Combined with the Trajectory Preference Domination, weight vector that reflects the agent's s preference for each objective can be learned. this way, the learned reward function better shaped. The weight vector assigns suitable preference to additional objectives and reduces the return sub-optimal trajectories, so as to make the objective function maximizing the cu- mulative undiscounted return consistent with the optimal policy. Through theoretical analysis, our algorithm effectively overcomes the problems of delayed reward and myopic policy in the objective- constrained task.
Multi-objecttvizien
propose
measure-
In summary, we solve the objective-constrained tasks from a completely new perspective of multi- objectivization. Our contributions are proposing novel method solve the problems and proving its feasibility in theory. Further, in order to identify the quality of our algorithm, we design a new benchmark problem: Efficient Delivery, which contains two additional objectives besides the primary objective. The experimental results show that our algorithm can learn the trade-off among these three objectives effectively and outperform those RL algorithms with the reward function designed by experts.
2 RELATED WORK
The constrained Markov Decision Process (Achiam et al., 2017,) Horie et al., 2019) is a formulation for RL with constraints, where constraints expectations of auxiliary costs must be satisfied. How-
2

---

Under review as a conference paper at ICLR 2020
ever, the relationship among these additional constraints and the primary objective is assumed to be known. The research this field mainly focuses on acquiring policies that satisfy fixed constraints. Although there have been attempts to learn the reward function through inverse reinforcement learn- ing (Abheel & Ng. 2004 Ziebart et al. 2008), it usually difficult to provide good demonstrations in complex tasks.
The idea of multi-objeectvvizatin has mainly been studied in the evolutionary computation works. There mainly exist two approaches for the multt-objectivizztion either by decomposing the sin- gle objective (Handl et al. 2008 Jähne et al. 22009), or by adding extra objectives [Jensen] 2004] Brockhoff et al. 2007). oicctiibjetti through adding objectives typically involves the corporation of some heuristic information or expert knowledge on the problem. The approach we propose in this paper falls this category. [Jnssen (2004) is one of the pioneers to use what he calls helper objectives next to the primary one. He investigated the job-shop scheduling and trav- eling salesman problems and found that additional objectives based on time and distance-related intuitions respectively help solve these problems faster.
3 PRELIMINARY
RL (Sutton & Barto 1998) is paradigm that allows agent to optimize its policy by interacting with a given environment. The agent is rewarded or punished for its behavior, and the aim to maximize the accumulated reward over time. More formally, the environment is modeled as Markov Decision Process (MDP) < S. A,T, >. S = {s1,82,.-. } is the state space a finite set of states, and A fa1, ..,... is the action space of finite set of actions. When the environment is in state 8t. executing action a makes the agent turn to state 8841 with probability T(s-1 St, (e), and reward signal r(st, ae) is provided. Finally, % the discount factor, defines how important future rewards are. The goal of the agent is to find a policy that maximizes the expected cumulative discounted return J".
--=E"
Σ t=
rs R(s,, at)
MORL (Liu et al. 2017 generalization of standard SORI., with the environment formulated as MOMDP 8, A, T, 7, R >, The difference between MORL and SORL lies the reward function. In MORL, each objective has its own associated reward function, so the reward is not scalar value but a vector:
A(s a) a). Rm Rm (B,(())).,....,(((s.))
In MORL, policies evaluated by their expected vector returns J":
J
E
Ri ae)
7 Rm (8t, an)
(2)
Usually, these objectives often conflict with each other, so any policy can only maximize one of the objectives, or realize trade-off among these conflicting objectives. In this case, it is difficult to order the candidate policies completely, and the concept Pareto optimum is usually used: policy Ti strictly Pareto dominates policy л2, only if æ performs strictly better than л2 at least on one objective and performs as well as 22 on the other objectives. The set of non-dominated policies referred to the Pareto optimum set Pareto front. The focus of MORL so far has been developing new algorithms capable of finding a set of policies that approximate the Pareto front.
The most common approach (Moffaert al. 2013 Vamplew et al. 2011 to deriving policy is to optimize linear scalarization reward function w((saa) -r(s a) based on a weight vector Ø. The weight vector expresses which trade-off solutions the decision- makerrs prefer. However, it is often non-intuitive (Das & Dennis, 1997) for preference elicitation through setting the weight vector, and often requiring significant amounts of parameter tuning processes.
3

---

Under review as a conference paper at ICLR 2020
4 THEOREM OF PREFERENCE MORL
To perform multi-objectivization,
must add some helper-objectives to the primary single- objective problem. More precisely, the reward function not scalar value T'p (5t u+) but a vee- tor r. ["p(,; (r); (s): (e)...... ....,... Th, (8t, dy)]. Generally speaking, provided when the agent accomplishes the ith objective, otherwise r; = 0. Moreover, we should ensure that the optimal policy of the primary SORL is of the policies in Pareto front of the corresponding MORI..
(i = p.i1.....tt,) is
v II T' =
where n² is optimal policy to the SORL problem, and II is the set of Pareto optimal policies.
Typically, the helper-objectives are in conflict with the primary objective, at least for some parts of the search space. Therefore, is necessary design scalar reward that properly weights the importance of the primary objective and helper-objectives so that the intention of the task is re- flected. In our work, kind of measures allowed provide feedback on our agent's 's behavior and use this feedback to learn weight vector in order to reflect the intention of the primary problem. Specifically, new domination criterion designed to compare the agent's 's behavior.
Definition 1. (Trajectory
Preference Domination):
A full trajectory 71
preference dominates another full trajectory denoted as 7 >pf 72, if and only if one of the sollowing two ((s,. ag) ...., (6
((s), ab), (s (s)-1.9t-1))
72
conditions is satisfied.
(1) rl Pareto dominates 72:
Vi, Σ
ri ab)
Σ a((s)
Bi, Enis Eri(st
(2) Ee at) > E. [rrp(s}.at))
In this way. Trajectory Preference Domination can be used to evaluate the agent's behavior quantita- tively. Obviously, if full trajectory r! is better than another full trajectory 72 in the primary SORL problem, then using this preference domination criterion in the corresponding MORL problem can still ensure that 7 is better than 2 So optimizing the reward function produced by Trajectory Pref- erence Domination can guide the agent to quickly converge to the preference multi-objective region, which is also preferred by the primary problem.
More precisely, the process of learning reward function can be modeled in the following way:
((sb: ab)..... (s) xs ((s
then
r(sh, ab) + r(s) ((sk > r(s .... . r
where r(s,, at) = E; (8t o:) rr(st, at), and without losing its generality wi(se, mx) E [0. 1]. The weight vector determines that which objective the agent should achieve at specific state.
Definition 2. < s,a >> is the preference state sate-action pair Pre(s, a) if (s, = 1, w, a) > 0. < s, a is the non non-preference state-action pair n Pre(s, s, a) ifvi, if rr = 1hthen w. (8, a) = 0.
And, < a,a > is trivial state-action pair Tri(s, a) r.
Obviously, from the expression, can see that the Pre(s, a) is the state-action pair that should be contained in the optimal trajectories. Any trajectory containing n nPre(s, a) cannot complete the tasks. Trí(s, a) is the state-action pair that does not achieve any objectives.
Definition 3.
The optimal trajectory cluster T is set of trajectories containing all preference state- action pairs but no non preference state-action pairs.
Lemma Starting from a state s, then the accumulated reward along T is far greater than the accumulated reward along r', where E T' T

---

Under review as a conference paper at ICLR 2020
Proof by contradiction:
Vr E E ((5): ab), (s)..))......6))..0)))) at),
..... (0.)...,(6/111111 (b
3r $ T ((+). ab), at) at)................... (s) ........ (8) a))).
Suppose that Lemma is false, then:
t-1
r(s) +
Σ
r(s and) S <r() +
Σ af)
mmi+l
nmitl
k-1
Σ r( am (
Σ
r(s am)
S Σ
r( trs
Σ
r an), (3)
m=U
m=i+1
n=U
n=i+1
However, oof T', then:
k
Σ r(s a +r(s; a+)+ m=0
Σ m=1+1
r am)
Σ r(s a a++
Σ r az) (4)
n=i+1
Equation is in conflict with equation therefore, Lemmal is true.
Theorem 1. Suppose that the reward funetion has been learned, then the policy that aims to maxi- mize the accumulated discounted reward over time can also maximize the accumulated reward. And the quality the policy is independent of discount factor y,
Proof:
Considering the computational form of the Q-function, we prove it from the terminate state.
VT T T. take the last preference state-action pair < $t-1. a from T, and know from the Lemma[
(( > r(si-1, aL) r(), +...
r(8-11, r(st-1,
((), ai +...,0 <<<<
max Q (s
= -0(==-Q0--00 Q(s--1, a7-1)
Take the next-to-last preference state-action pair< st 0T, >, 1 from r. According to the Lemma [ starting from 8t-is we know that the accumulated reward along 7 greater than the accumulated reward along r'.
is possible that
St-i: a7 >E 7', then:
r(s--,, af_) 2 (( if af
According to Definition 2 the reward value of those trivial state-action pairs and non-preference state-action pairs is Add those trivial state-action pairs between 8t-; and 8;-1, so as to complete the reward over trajectory.
8;-1 E -', then:
r(s--: +0 > r (1--), +0 +.. -0 + r(s,, +
When choosing a trajectory that the time-steps from s._i to st-1 is shortest, then:
T(5--1, a. + 0+
- + th r(51-1. af-))
- a( + m 0+ th . r(st-1, 07 m+1 r(s), +...
5

---

Under review as a conference paper at ICLR 2020
max ((5 Q(8--;
(5)
Similarly, if 8t-1 7' or < St-1, oi 24 T', obviously equation holds
For the same reason, we can prove that the Q-value of any preference state-action pairs is larger than the Q-value with other actions. And, the Q-value reaches the maximum when the corresponding actions make the shortest timesteps from any trivial state to the next preference state. Therefore, the policy that aims to maximize the accumulated discounted reward over time can also maximize the accumulated reward. Moreover, we do not make any assumptions about discount factor y in the proof, so the quality of the policy independent of y in theory exploration tends to infinite sufficiency. In addition, the weight vector assigns preference helper-objectives that are typically helpful for exploring better solutions in the primary objective.
5 ALGORITHM OF PREFERENCE MORL
After converting the single-objective problem preference multi-objective problem, a policy and weight estimate W x - úī are parametrized by the neural network, respectively.
The two networks are updated iteratively by three processes:
1. The policy æ interacts with the environment with an exploration rate to produce a set of full trajectories {r {r'...., T")). Each trajectory I' needs to evaluate its cumulative undis- counted return for each objective and then puts the trajectory with its evaluation ef into trajectory buffer Drrj-
2. Randomly select pairs of trajectories (r', 7²) from the buffer Dtrj. and compare their eval- uation using Trajectory Preference Domination, then the parameters of the mapping W are optimized via supervised learning to fit the comparisons.
3.
The parameters of ग are updated by a traditional reinforcement learning algorithm, order to maximize the discount sum of predicted rewards r(st, a,) = E wi(St, az) -r,(St, oz).
5.1 SETTING TRAJECTORY BUFFER
It is obvious that there must be lots of different trajectories for learning true weight vector. An important problem is that what kind of buffer is more suitable for learning reward function. Making the replay buffer larger that early trajectories obtained through random exploration are still present is impractical for two reasons; (1) Unless the buffer is infinite, older trajectories will eventually be still present, learning from an increasing number of trajectories remains hard, because the learning speed of each iteration will be slower and slower. Therefore, instead of retaining all past trajectories, the following two steps are taken to ensure effective learning of weight vector: I) retaining top K preference optimal trajectories in the current buffer; 2) replacing the remaining M current buffer with new trajectories sampled by the current round. The purpose of the first step is to improve the utilization rate of preference trajectories and push reward learning towards the preferred direction. The second step can enable the reward function to be effectively learned in the case of the limited buffer. In addition, when there a greater diversity of trajectories, it is more helpful for the reward learning and larger exploration rate be realized.
erased before acquiring the true reward function. (2) Even if all past trajectories
trajectories in the
5.2 OPTIMIZING THE WEIGHT VECTOR AND THE POLICY
We can interpret the weight estimation W as preference predictor if we view w as latent factor explaining which objective the agent should prefer at specific time and state. Assume that the probability of preferring trajectory 7í depends exponentially on the value of the latent reward summed over the length of the trajectory:
exp EE ws(s!. a.) r((s).a!)
P|r xp -= exp ΣΣ, (a) at) (1(s) (s6,000 at)
(6)
+ eap ΣΣ, (0,(5) (57, (,(

---

Under review as a conference paper at ICLR 2020
We optimize to minimize the cross-entropy loss:
loss(w)= --
Σ
[[(e' dop e²) loggii! Ans -9]
+ I(e² Aps xl) logPP(2 -pg
where T(e pf (2) (
if E
otherwise.
Equation 6 and 7 are very similar to preference learning proposed by Christiano et al. (2017) The main difference lies in the comparison between trajectories, the latter used a large amount of human prior knowledge, which is usually related to tasks. Instead, Trajectory Preference Domination we proposed is agnostic to tasks. After learning weights w, the reward function r(8t, ae) = E; ws,St, at) (s( ae) can be used to optimize the policy æ. We can train this policy using any RL algorithm that is appropriate for the domain. In this paper, Deep Q-learning (DQN)
(Mnih et al., 2013) adopted.
EXPERIMENTS AND RESULTS
6.1 EFFICIENT DELIVERY
To identify the quality of our algorithms, we design new benchmark problem (Figure 2 Efficient Delivery. In this game, the purpose of the agent to control an unmanned aerial vehicle (UAV) to deliver as many packages as possible in the city (represented by 10 10 grid world). When the current delivery location reached, the next delivery location appears at some location of the city. However, because of its limited battery capacity, the UAV cannot fly for too long. Therefore, charging station is set in the upper left corner of the grid world. In this case, the agent needs to weigh the importance of delivery and charging every moment as deliver as many packages possible in fixed period of time (100 timesteps). Moreover, in order to make the task more interesting and more challenging, an accelerator appears at the same time as the current delivery location appears, but their locations are different. Once the agent gets an accelerator, its flight speed will double in short time (8 timesteps) and the accelerator disappears until the next delivery location appears. In detail, when the current delivery and accelerator locations are relatively close, even though it is longer distance for the agent to obtain the accelerator firstly and then fly to the current delivery location than to reach the current delivery location directly, the continuous acceleration may make the agent reach the next delivery location faster. However, when the current accelerator location is far away from the delivery location, it is not worth the effort to fly through such a long distance to find the accelerator for short-term acceleration. In summary, in order to deliver as many packages as possible in fixed period of time, it is necessary for the agent to weigh the importance of the three objectives at every moment: delivery, charging and acceleration.
6.2 RESULTS
We test the performance of our algorithm in two different settings. The first is that there are only two objectives: delivery and charging, whose importance needs to be clearly weighed. The second that preferences for all three objectives need be optimized. Except for the number of objectives, all hyper-parameter are the same in two settings. The configuration of these prper-parameters is as follows: 7 0.9, the size of trajectory buffer M 300. The baseline is DQN optimized by the reward function which is provided only from the primary dellvyy
Algorithm Effectiveness Testing. To solve the delivery task, the agent must learn to weigh the importance of the three objectives at every moment. However, the existing methods require the task-specific prior knowledge such as a well-defined reward function (Mnih et al. 2015 2016) or demonstration samples (Abbeel & Ng. 2004). To our knowledge, our work is the first one solve such tasks through a learning method. As shown in Figure ((a) our algorithm can make full use of the objectives of charging and acceleration to boost up the performance of the delivery task. The number of delivery is obviously the most in the situation with all three objectives, compared to the situation with two objectives with the reward of delivery only. This result not only shows the
7

---

Under review as a conference paper at ICLR 2020
Charging
Ineffcient
Delivery
Effiiient
Speed
Three objectives
(b) Efficient delivery
(c) Inefficient delivery
Figure 2: A sketch of Efficient Delivery. The purpose of the agent control the UAV to delivery as many packages as possible. When the current delivery location is reached, the next delivery location appears. order to accomplish the delivery task more efficiently, it is necessary for the agent to weigh the importance of delivery, charging and acceleration at every moment.
three objectives
the number delivery
the number of delivery
ard
only the primary objective two objectives
gamma 0.7
threa objectives
gamma 0.9
0.0d0
1.000"
2,0x10
3.0x10'
4.0110
5.0610
6,5630
0.Dx10
0.211
0-4x0"
0.8x10
3.0410f
1.2x2p
timestep
timestep
(b)
Figure 3: Performance comparison on Efficient Delivery games. (a) We compare algorithm ing all three objectives (delivery, charging and acceleration) with that using two objectives (delivery and charging), as well as DQN, which uses the reward function from delivery only. (b) We compare the performance of our algorithm under different discount factor
importance of acceleration to the task, such as helping the agent charge and deliver faster, but also identifies that our algorithm can effectively learn the trade-off among multiple objectives. No matter using two or three objectives, our algorithm performs significantly better than DQN, which is simply optimized through reward from delivery only. There mainly two reasons. One reason is the delayed reward in DQN. Although the agent charged, it does not receive any rewards. Therefore, it is difficult to find the relationship among these objectives, which is essential understanding the task. The other reason is the myopic policy in DQN. Because of misalignment between the behavior and the cumulative discount return, the agent focuses mainly on high but short-term return. However, through multi-objectivization, the learned reward function is better shaped. The learning procedure of weight vector assigns preferences additional objectives that are typically helpful for more delivery and corrects the relationships between the behavior and discount return.
Algorithm Robustness Testing, Some evidence (Prokhorov & Wunsch 1997 Bertsekas Tsitsik- lis 1996) show that the discount factor has great influence on the performance of RLs. Smaller may lead to faster convergence, but poorer sub-optimal policy. However, in our framework, as The- orem proves, different rys induce the common optimal policy. Figure (3(b shows the performance of our algorithm under different discount factors 7. The performances under three different settings are proximately the same, even though there is subtle fluctuation. The change of by 20% results in the change of the algorithm performance by ~ 15%. Compared to the work Xf et al. (018)), where the change of by only 5% results in large change of performance by 30%. algorithm is highly robust against y. Therefore, the difficulty myopic policy can be effectively alleviated in our framework.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 22-31. JMLR. org, 2017.
Jose A Arjona-Medina, Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint arXiv: 078806000777 2018.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming, volume Athena Scien- tific Belmont, MA, 1996.
Stefan Bleuler, Martin Brack, Lothar Thiele, and Eckart Zitzler. Multiobjective genetic program- ming: Reducing bloat using spea2. In Congress on Evolutionary Computation Cec, 2001.
Dimo Brockhoff, Tobias Friedrich, Nils Hebbinghaus, Christian Klein, Frank Neumann, and Eckart Zitzler. Do additional objectives make problem harder? In Proceedings of the 9th annual conference on Genetic and evolutionary computation, pp. 765-772. ACM, 2007.
Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, and Matthew E Taylor. Multi- objectivization and ensembles of shapings in reinforcement learning. Neurocomputiing 263:48- 59, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299-4307, 2017.
Indraneel Das and John Dennis. closer look at drawbacks of minimizing weighted sums of ob- jectives for pareto generation in multicriteria optimization problems. Structural optimization, 14(1):63-69, 1997.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint veXiv:1705.080 17, 2017.
Julia Handl, Simon C Lovell, and Joshua Knowles. Multiobjectivization by decomposition of scalar cost functions. In International Conference on Parallel Problem Solving from Nature, pp. 31-40.
Springer, 2008.
Naoto Horie, Tohgoroh Matsui, Koichi Moriyama, Atsuko Mutoh, and Nobuhiro Inuzuka. Multi- objective safe reinforcement learning: the relationship between multi-objective reinforcement learning and safe reinforcement learning. Artificial Life and Robotics, pp. 1-8, 2019.
Martin Jähne, Xiaodong Li, and Jürgen Branke. Evolutionary algorithms and multi-objectivizzaion for the travelling salesman problem. In Proceedings of the 11th Annual conference Genetic and evolutionary computation, pp. 595-602. ACM, 2009.
Mikkel T Jensen. Helper-ohjectives: Using multi-objective evolutionary algorithms for single- objective optimisation. Journal of Mathematical Modelling and Algorithms, 3(4):323-347, 2004.
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaskowski. Viz- doom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEFF, 2016.
Joshua D Knowles, Richard A Watson, and David W Come. Reducing local optima in single- objective problems by multi-objectivizatinn. In International conference on evolutionary multi- criterion optimization, pp. 269-283. Springer, 2001.
Chunming Liu, Xin Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2017.

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In IEEE Ssci, 2013.
Danil V Prokhorov and Donald Wunsch. Adaptive critic designs. IEEE transactions on Neural Networks, 8(5):997-1007, 1997.
Stwart Russell. Should we fear supersmart robots? Scientific American, 314(6):58 2016.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 95))105541054, 1998.
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning, 84 (1-2):51-80, 2011.
Zhongwen Xu, Hado Van Hasselt, and David Silver. Meta-gradient reinforcement learning. 2018.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforeement learning. 2008
10


Reviews:
[{"review_id": "rygBjki6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After Responses:\nI understand the differences that authors pointed to the relevant literature. However, it is still lacking comparisons to these relevant methods. The proposed method has not been compared with any of the existing literature. Hence, we do not have any idea how does it stand against the existing approaches. Hence, I believe the empirical study is still significantly lacking. I will stick to my decision. Main reason is as follows; I believe the idea is interesting but it needs a significant empirical work to be published. I recommend authors to improve empirical study and re-submit.\n-------\nThe submission is proposing a method for multi-objective RL such that the preference of tasks learned on the fly with the policy learning. The main idea is converting the multi-objective problem into single objective by scalar weighting. The weights are learned in a structured learning fashion by enforcing them to approximate the Pareto dominance relations.\n\nThe submission is interesting; however, its novelty is not even clear since authors did not discuss majority of the existing related work. \n\nAuthors can consult the AAMAS 2018 tutorial \"Multi-Objective Planning and Reinforcement Learning\" by Whiteson&Roijers for relevant papers. It is also important to note that there are other methods which learn weighting. Optimistic linear support is one of such methods. Hence, this is not the first of such approaches. Beyond RL, it is also studied extensively in supervised learning. For example, authors can see \"Multi-Task Learning as Multi-Objective Optimization\" from NeurIPS 2018.\n\nThe manuscript is also very hard to parse and understand. For example, Definition 2 uses but not define \"p\" in condition (2). Similarly, Lemma 1 states sth is \"far greater\" than something else. However, \"far greater\" is not really defined. I am also puzzled to understand the relevance of Theorem 1. It is beyond the scope of the manuscript, and also not really new.\n\nAuthors suggest a method to solve multi-objective optimization. However, there is no correctness proof. We do not know would the algorithm result in Pareto optimal solution even asymptotically. Arbitrary weights do not result in Pareto optimality.\n\nProposing a new toy problem is well-received. However, not providing any experiment beyond the proposed problem is problematic. Authors motivate their method using DOOM example. Why not provide experimental results on a challenging problem like DOOM?\n\nIn summary, I definitely appreciate the idea. However, it needs better literature search. Authors should position their paper properly with respect to existing literature. The theory should be revised and extended with convergence to Pareto optimality. Finally, more extensive experiments on existing problems comparing with existing baselines is needed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg4aCzE5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank the authors for the response. I agree with R2 that the paper lacks comparisons with previous works. I will stick to my previous decision.\n----------------------------------------\nSummary\nThis paper presents a new approach for single-objective reinforcement learning by preferencing multi-objective reinforcement learning. The general idea is to first figure out a few important objectives, add some helper-objectives to the original problem, and learn the weights for each individual objective by trying to keep the same order as Pareto dominance. This paper has potential, but I lean to vote for rejecting this paper now, since it is still not ready. I might change my score based on the reviews from other reviewers.\nStrengths\n- The idea is novel. Learning weights for each objective by keeping the order as Pareto dominance is an interesting idea to me.\nWeaknesses\n- The lack of experiments. The authors tested their method in only one scenario, which makes me feel unsafe. Only testing on one simple scenario does not demonstrate the effectiveness. The authors are supposed to test their method on more (complex) scenarios to show the effectiveness of their method.\nPossible Improvements\nAs mentioned before, the proposed method can be tested on more scenarios (e.g., Deep Sea Treasure, SuperMario, etc.).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers planning through the lenses both of a single and multiple objectives. The paper then discusses the pareto frontiers of this optimization. While this is an interesting direction, the reviewers feel a more careful comparison to related work is needed.
Decision: Reject