Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

In the multi-objective reinforcement learning ( MORL ) paradigm , the relative importance of each environment objective is often unknown prior to training , so agents must learn to specialize their behavior to optimize different combinations of environment objectives that are specified post-training . These are typically linear combinations , so the agent is effectively parameterized by a weight vector that describes how to balance competing environment objectives . However , many real world behaviors require non-linear combinations of objectives . Additionally , the conversion between desired behavior and weightings is often unclear . In this work , we explore the use of a language based on propositional logic with quantitative semantics–in place of weight vectors–for specifying non-linear behaviors in an interpretable way . We use a recurrent encoder to encode logical combinations of objectives , and train a MORL agent to generalize over these encodings . We test our agent in several grid worlds with various objectives and show that our agent can generalize to many never-before-seen specifications with performance comparable to single policy baseline agents . We also demonstrate our agent ’ s ability to generate meaningful policies when presented with novel specifications and quickly specialize to novel specifications . 1 INTRODUCTION . Reinforcement Learning ( RL ) is a method for learning behavior policies by maximizing expected reward through interactions with an environment . RL has grown in popularity as RL agents have excelled at increasingly complex tasks , including board games ( Silver et al. , 2016 ) , video games ( Mnih et al. , 2015 ) , robotic control ( Haarnoja et al. , 2018 ) , and other high dimensional , complex tasks . RL continues to be a valued area of research as algorithms become more generalizable and sample efficient , making them more feasible for deployment in real world scenarios . Many RL tasks can be imagined in which multiple possibly conflicting objectives exist . The relative importance of each objective may not be known by the system designer prior to training , and–when it comes to real world deployment–it may be difficult or impossible to retrain agents as the priorities of objectives change over time . Rather than retrain agents for each prioritization , multiple objective RL ( MORL ) ( Roijers et al. , 2013 ) seeks to learn a set of potential policies so that importance of objectives can be specified after training , thus creating more flexible , adaptable agents . For example , a cleaning agent in a house environment may have several objectives such as dusting , sweeping , and vacuuming . The agent can learn to complete each of these objectives , but certain objectives may take priority over others . A user may specify that dusting is twice as important as sweeping and that vacuuming is not important at all . These priorities may change , and MORL allows agents to learn policies that can satisfy any prioritization of objectives . As part of MORL , a scalarization function is chosen to convert a multiple objective reward vector into a scalar . The most common scalarization function for MORL is a linear combination in the form of a weight vector . However , many real world scalarization functions are non-linear . Additionally , weight vectors are not ideal for specifying desired behavior . A user may need to experiment in order to find the weights that result in a desired behavior . Thus scalarization functions that are interpretable and allow for non-linearities are preferred . Returning to the cleaning agent example , a user may specify that the cleaning agent should either sweep or vacuum , but it is not necessary to do both . Perhaps the decision should be determined by whichever is more likely to be done well in consideration of other prioritized objectives . Or , it may be useful to specify that the house should always be kept 75 % dusted and that the rest of the agent ’ s time should be spent vacuuming . These specifications become difficult or impossible to encode with simple linear weightings . Multi-task RL ( MTRL ) ( Caruana , 1997 ) is a generalization of MORL in which an agent learns to complete multiple tasks simultaneously , often with the ultimate goal of completing some more complex task . Recently , Universal Value Function Approximators ( UVFA ) ( Schaul et al. , 2015 ) were developed for MTRL to learn state or q-values over multiple goals . UVFAs require a goal parameterization as input usually defined as an element of the state ( Andrychowicz et al. , 2017 ) . However , the environment state is not always expressive enough to parameterize all of the goals we may want to learn . Multiple objectives may provide a better way to define additional goals that are not part of the environment state . The contributions of this work can be defined in two parts . First , we propose a simple language based on propositional logic to specify logical combinations of multiple objectives . This language is equipped with quantitative semantics that are used to define scalarization functions for MORL . The resulting scalarization functions can express non-linear combinations of objectives and are more interpretable than traditional weight vectors . The language also acts as a way of specifying goals for multi-task learning that will be parameterized to use UVFAs . Second , we develop a MORL agent for use with this language . The agent prepends a recurrent encoder onto a UVFA architecture to parameterize goals specified in our language . The agent also follows a learning curriculum over specifications to improve training speed and performance . We demonstrate that our agent generalizes to never-before-seen specifications defined in our language by providing a test set of novel specifications ; this can be seen as a form of zero-shot RL . Performance on test sets is compared to baseline agents trained on single specifications , and we show that our agent performs comparably to baselines despite having never been trained on the test specifications . We demonstrate this over multiple grid worlds with various objectives . Finally , we demonstrate our agent ’ s ability to quickly specialize to novel specifications post-training . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . Traditional RL problems are often modeled by a Markov Decision Process ( MDP ) defined by the tuple ( S , S0 , A , p ) where S is the set of states , S0 is the set of initial states , A is the set of actions , and p ( r , s′|s , a ) defines transition probabilities for the environment . Here s′ ∈ S is the subsequent state and r ∈ R is the reward . An agent ’ s objective is to maximize expected return : U = E s∼S [ Rπ ( s ) ] ( 1 ) Rπ ( s ) = E r , s′∼p [ r + γRπ ( s′ ) ] ( 2 ) where Rπ is the return under a policy π : S → A mapping states to actions , and γ is a discount factor . Q-learning is an approach to solving reinforcement learning problems . The Deep Q-Learning ( DQN ) algorithm ( Mnih et al. , 2015 ) utilizes a neural network Q to estimate the Q-value of a state-action pair . The return under a policy Rπ can be expressed by the Q-value when the policy is defined by the network Q : Rπ ( s ) = max a∼A Q ( s , a ) ( 3 ) π ( s ) = argmax a∼A Q ( s , a ) . ( 4 ) We utilize the DQN algorithm in this paper to build our MORL agent , although our language and the way we construct our agent are not limited to the DQN algorithm . 2.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING . We alter the traditional RL formulation as an MDP by using a vector of rewards rather than a scalar reward , resulting in a Multi-Objective Markov Decision Process ( MOMDP ) ( Roijers et al. , 2013 ) . We represent a MOMDP as a tuple ( S , S0 , A , p ) where p ( r , s′|s , a ) maps to the reward vector r ∈ Rn rather than a scalar reward . Each of the n elements in r represents the reward for a particular objective . Next , we modify the DQN algorithm for MORL . Equation 2 requires a scalar reward to sum with the discounted future rewards . Since the new transition function p provides a vector reward r we need a scalarization function . Typically , the scalarization function is defined as a linear combination of objectives in the form of a weight vector ( Abels et al. , 2018 ) . In this work , we explore the use of non-linear scalarizations functions defined by logical combinations of objectives . In offline MORL , the scalarization function is unknown prior to training , so an agent ’ s goal is to learn a coverage set ( Roijers et al. , 2013 ) of policies in which there is at least one optimal policy for any scalarization function . Previously this has been done by finding a convex coverage set of policies ( Mossalam et al. , 2016 ) . However , this method does not work for non-linear scalarization functions . Abels et al . ( 2018 ) explored the use of UVFAs to train MORL agents in an online setting . In an online MORL setting , an agent must learn on the fly and is not given the opportunity to learn a coverage set prior to testing . Instead the agent must adapt quickly to new scalarization functions without forgetting policies for previously seen scalarization functions . Online MORL is outside of the scope of this paper , but we also utilize UVFAs to instead learn a dense set of policies , trained offline , to be used post training with arbitrary behavior specifications . 2.3 UNIVERSAL VALUE FUNCTION APPROXIMATORS . Universal Value Function Approximators ( UVFA ) ( Schaul et al. , 2015 ) learn value functions over multiple goals , taking as input to a neural network a goal parameterization along with environment state or a state action pair . This dense set of value functions induces an equally dense set of policies which can be applied to their respective goals . Empirically , UFVAs are shown to be able to generalize to novel goals . We utilize this method to parameterize goals specified by logical expressions in our language and show that these statements can be generalized across as well . Many similar techniques are applied in multi-task learning scenarios . Although , the purpose of multi-task learning is often to improve the learning speed of an agent rather than improve its ability to generalize to new tasks . The latter is our objective . 3 MULTI-OBJECTIVE RL WITH LOGICAL COMBINATIONS OF OBJECTIVES . Rather than learn a set of individual policies to approximate a coverage set over scalarization functions , we train a single agent to generalize over encoded representations of scalarization functions . We define scalarization functions in the semantics of a custom language . Behavior specifications are represented as logic strings , passed through an encoder , and then given to a Q-network for learning . In this section we lay out our language ’ s syntax and semantics , the agent architecture , the encoding method , and the learning curriculum for our agent . 3.1 LANGUAGE SYNTAX AND SEMANTICS . We define a language based on propositional logic to specify combinations of objectives . The logic operates over objective rewards between [ 0 , 1 ] inclusive , and is defined as follows .

Summary: This paper proposes using logical specifications to facilitate Q-learning in multi-objective reinforcement learning (MORL). Empirically the proposed method can generalize to unseen reward specifications with performance competitive to agents being fully trained in the new environment. The proposed setting employs a more expressive objectives space induced by propositional logic. The proposed method uses a recurrent encoder to embed specifications into vectors and uses them to parametrize the Q-function.