Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

In the multi-objective reinforcement learning ( MORL ) paradigm , the relative importance of each environment objective is often unknown prior to training , so agents must learn to specialize their behavior to optimize different combinations of environment objectives that are specified post-training . These are typically linear combinations , so the agent is effectively parameterized by a weight vector that describes how to balance competing environment objectives . However , many real world behaviors require non-linear combinations of objectives . Additionally , the conversion between desired behavior and weightings is often unclear . In this work , we explore the use of a language based on propositional logic with quantitative semantics–in place of weight vectors–for specifying non-linear behaviors in an interpretable way . We use a recurrent encoder to encode logical combinations of objectives , and train a MORL agent to generalize over these encodings . We test our agent in several grid worlds with various objectives and show that our agent can generalize to many never-before-seen specifications with performance comparable to single policy baseline agents . We also demonstrate our agent ’ s ability to generate meaningful policies when presented with novel specifications and quickly specialize to novel specifications . 1 INTRODUCTION . Reinforcement Learning ( RL ) is a method for learning behavior policies by maximizing expected reward through interactions with an environment . RL has grown in popularity as RL agents have excelled at increasingly complex tasks , including board games ( Silver et al. , 2016 ) , video games ( Mnih et al. , 2015 ) , robotic control ( Haarnoja et al. , 2018 ) , and other high dimensional , complex tasks . RL continues to be a valued area of research as algorithms become more generalizable and sample efficient , making them more feasible for deployment in real world scenarios . Many RL tasks can be imagined in which multiple possibly conflicting objectives exist . The relative importance of each objective may not be known by the system designer prior to training , and–when it comes to real world deployment–it may be difficult or impossible to retrain agents as the priorities of objectives change over time . Rather than retrain agents for each prioritization , multiple objective RL ( MORL ) ( Roijers et al. , 2013 ) seeks to learn a set of potential policies so that importance of objectives can be specified after training , thus creating more flexible , adaptable agents . For example , a cleaning agent in a house environment may have several objectives such as dusting , sweeping , and vacuuming . The agent can learn to complete each of these objectives , but certain objectives may take priority over others . A user may specify that dusting is twice as important as sweeping and that vacuuming is not important at all . These priorities may change , and MORL allows agents to learn policies that can satisfy any prioritization of objectives . As part of MORL , a scalarization function is chosen to convert a multiple objective reward vector into a scalar . The most common scalarization function for MORL is a linear combination in the form of a weight vector . However , many real world scalarization functions are non-linear . Additionally , weight vectors are not ideal for specifying desired behavior . A user may need to experiment in order to find the weights that result in a desired behavior . Thus scalarization functions that are interpretable and allow for non-linearities are preferred . Returning to the cleaning agent example , a user may specify that the cleaning agent should either sweep or vacuum , but it is not necessary to do both . Perhaps the decision should be determined by whichever is more likely to be done well in consideration of other prioritized objectives . Or , it may be useful to specify that the house should always be kept 75 % dusted and that the rest of the agent ’ s time should be spent vacuuming . These specifications become difficult or impossible to encode with simple linear weightings . Multi-task RL ( MTRL ) ( Caruana , 1997 ) is a generalization of MORL in which an agent learns to complete multiple tasks simultaneously , often with the ultimate goal of completing some more complex task . Recently , Universal Value Function Approximators ( UVFA ) ( Schaul et al. , 2015 ) were developed for MTRL to learn state or q-values over multiple goals . UVFAs require a goal parameterization as input usually defined as an element of the state ( Andrychowicz et al. , 2017 ) . However , the environment state is not always expressive enough to parameterize all of the goals we may want to learn . Multiple objectives may provide a better way to define additional goals that are not part of the environment state . The contributions of this work can be defined in two parts . First , we propose a simple language based on propositional logic to specify logical combinations of multiple objectives . This language is equipped with quantitative semantics that are used to define scalarization functions for MORL . The resulting scalarization functions can express non-linear combinations of objectives and are more interpretable than traditional weight vectors . The language also acts as a way of specifying goals for multi-task learning that will be parameterized to use UVFAs . Second , we develop a MORL agent for use with this language . The agent prepends a recurrent encoder onto a UVFA architecture to parameterize goals specified in our language . The agent also follows a learning curriculum over specifications to improve training speed and performance . We demonstrate that our agent generalizes to never-before-seen specifications defined in our language by providing a test set of novel specifications ; this can be seen as a form of zero-shot RL . Performance on test sets is compared to baseline agents trained on single specifications , and we show that our agent performs comparably to baselines despite having never been trained on the test specifications . We demonstrate this over multiple grid worlds with various objectives . Finally , we demonstrate our agent ’ s ability to quickly specialize to novel specifications post-training . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . Traditional RL problems are often modeled by a Markov Decision Process ( MDP ) defined by the tuple ( S , S0 , A , p ) where S is the set of states , S0 is the set of initial states , A is the set of actions , and p ( r , s′|s , a ) defines transition probabilities for the environment . Here s′ ∈ S is the subsequent state and r ∈ R is the reward . An agent ’ s objective is to maximize expected return : U = E s∼S [ Rπ ( s ) ] ( 1 ) Rπ ( s ) = E r , s′∼p [ r + γRπ ( s′ ) ] ( 2 ) where Rπ is the return under a policy π : S → A mapping states to actions , and γ is a discount factor . Q-learning is an approach to solving reinforcement learning problems . The Deep Q-Learning ( DQN ) algorithm ( Mnih et al. , 2015 ) utilizes a neural network Q to estimate the Q-value of a state-action pair . The return under a policy Rπ can be expressed by the Q-value when the policy is defined by the network Q : Rπ ( s ) = max a∼A Q ( s , a ) ( 3 ) π ( s ) = argmax a∼A Q ( s , a ) . ( 4 ) We utilize the DQN algorithm in this paper to build our MORL agent , although our language and the way we construct our agent are not limited to the DQN algorithm . 2.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING . We alter the traditional RL formulation as an MDP by using a vector of rewards rather than a scalar reward , resulting in a Multi-Objective Markov Decision Process ( MOMDP ) ( Roijers et al. , 2013 ) . We represent a MOMDP as a tuple ( S , S0 , A , p ) where p ( r , s′|s , a ) maps to the reward vector r ∈ Rn rather than a scalar reward . Each of the n elements in r represents the reward for a particular objective . Next , we modify the DQN algorithm for MORL . Equation 2 requires a scalar reward to sum with the discounted future rewards . Since the new transition function p provides a vector reward r we need a scalarization function . Typically , the scalarization function is defined as a linear combination of objectives in the form of a weight vector ( Abels et al. , 2018 ) . In this work , we explore the use of non-linear scalarizations functions defined by logical combinations of objectives . In offline MORL , the scalarization function is unknown prior to training , so an agent ’ s goal is to learn a coverage set ( Roijers et al. , 2013 ) of policies in which there is at least one optimal policy for any scalarization function . Previously this has been done by finding a convex coverage set of policies ( Mossalam et al. , 2016 ) . However , this method does not work for non-linear scalarization functions . Abels et al . ( 2018 ) explored the use of UVFAs to train MORL agents in an online setting . In an online MORL setting , an agent must learn on the fly and is not given the opportunity to learn a coverage set prior to testing . Instead the agent must adapt quickly to new scalarization functions without forgetting policies for previously seen scalarization functions . Online MORL is outside of the scope of this paper , but we also utilize UVFAs to instead learn a dense set of policies , trained offline , to be used post training with arbitrary behavior specifications . 2.3 UNIVERSAL VALUE FUNCTION APPROXIMATORS . Universal Value Function Approximators ( UVFA ) ( Schaul et al. , 2015 ) learn value functions over multiple goals , taking as input to a neural network a goal parameterization along with environment state or a state action pair . This dense set of value functions induces an equally dense set of policies which can be applied to their respective goals . Empirically , UFVAs are shown to be able to generalize to novel goals . We utilize this method to parameterize goals specified by logical expressions in our language and show that these statements can be generalized across as well . Many similar techniques are applied in multi-task learning scenarios . Although , the purpose of multi-task learning is often to improve the learning speed of an agent rather than improve its ability to generalize to new tasks . The latter is our objective . 3 MULTI-OBJECTIVE RL WITH LOGICAL COMBINATIONS OF OBJECTIVES . Rather than learn a set of individual policies to approximate a coverage set over scalarization functions , we train a single agent to generalize over encoded representations of scalarization functions . We define scalarization functions in the semantics of a custom language . Behavior specifications are represented as logic strings , passed through an encoder , and then given to a Q-network for learning . In this section we lay out our language ’ s syntax and semantics , the agent architecture , the encoding method , and the learning curriculum for our agent . 3.1 LANGUAGE SYNTAX AND SEMANTICS . We define a language based on propositional logic to specify combinations of objectives . The logic operates over objective rewards between [ 0 , 1 ] inclusive , and is defined as follows .

Summary: The authors propose to tackle the problem of multi-objective reinforcement learning (MORL) by considering a logical function as reward signal. In their proposed solution, the logical function is also encoded and concatenated with the state. They argue, via simulation and toy examples, that the proposed model is able to generalize to logical formulas of the multidimensional rewards that has not observed during training.