A Boolean Task Algebra for Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved recent success in a number of difficult , high-dimensional environments ( Mnih et al. , 2015 ; Levine et al. , 2016 ; Lillicrap et al. , 2016 ; Silver et al. , 2017 ) . However , these methods generally require millions of samples from the environment to learn optimal behaviours , limiting their real-world applicability . A major challenge is thus in designing sample-efficient agents that can transfer their existing knowledge to solve new tasks quickly . This is particularly important for agents in a multitask or lifelong setting , since learning to solve complex tasks from scratch is typically infeasible . One approach to transfer is composition ( Todorov , 2009 ) , which allows an agent to leverage existing skills to build complex , novel behaviours . These newly-formed skills can then be used to solve or speed up learning in a new task . In this work , we focus on concurrent composition , where existing base skills are combined to produce new skills ( Todorov , 2009 ; Saxe et al. , 2017 ; Haarnoja et al. , 2018 ; Van Niekerk et al. , 2019 ; Hunt et al. , 2019 ; Peng et al. , 2019 ) . This differs from other forms of composition , such as options ( Sutton et al. , 1999 ) and hierarchical RL ( Bacon et al. , 2017 ) , where actions and skills are chained in a temporal sequence . In this work , we define a Boolean algebra over the space of tasks and optimal value functions . This extends previous composition results to encompass all Boolean operators : conjunction , disjunction , and negation . We then prove that there exists a homomorphism between the task and value function algebras . Given a set of base tasks that have been previously solved by the agent , any new task written as a Boolean expression can immediately be solved without further learning , resulting in a zero-shot super-exponential explosion in the agent ’ s abilities . We illustrate our approach in a simple domain , where an agent first learns to reach a number of rooms , after which it can then optimally solve any task expressible in the Boolean algebra . We then demonstrate composition in high-dimensional video game environments , where an agent first learns to collect different objects , and then compose these abilities to solve complex tasks immediately . Our results show that , even when function approximation is required , an agent can leverage its existing skills to solve new tasks without further learning . 2 PRELIMINARIES . We consider tasks modelled by Markov Decision Processes ( MDPs ) . An MDP is defined by the tuple ( S , A , ρ , r ) , where ( i ) S is the state space , ( ii ) A is the action space , ( iii ) ρ is a Markov transition kernel ( s , a ) 7→ ρ ( s , a ) from S × A to S , and ( iv ) r is the real-valued reward function bounded by [ rMIN , rMAX ] . In this work , we focus on stochastic shortest path problems ( Bertsekas & Tsitsiklis , 1991 ) , which model tasks in which an agent must reach some goal . We therefore consider the class of undiscounted MDPs with an absorbing set G ⊆ S. The goal of the agent is to compute a Markov policy π from S to A that optimally solves a given task . A given policy π is characterised by a value function V π ( s ) = Eπ [ ∑∞ t=0 r ( st , at ) ] , specifying the expected return obtained under π starting from state s.1 The optimal policy π∗ is the policy that obtains the greatest expected return at each state : V π ∗ ( s ) = V ∗ ( s ) = maxπ V π ( s ) for all s in S. A related quantity is the Q-value function , Qπ ( s , a ) , which defines the expected return obtained by executing a from s , and thereafter following π . Similarly , the optimal Q-value function is given by Q∗ ( s , a ) = maxπ Qπ ( s , a ) for all s in S and a in A . Finally , we denote a proper policy to be a policy that is guaranteed to eventually reach the absorbing set G ( James & Collins , 2006 ; Van Niekerk et al. , 2019 ) . We assume the value functions for improper policies—those that never reach absorbing states—are unbounded below . 3 BOOLEAN ALGEBRAS FOR TASKS AND VALUE FUNCTIONS . In this section , we develop the notion of a Boolean task algebra , allowing us to perform logical operations—conjunction ( ∧ ) , disjunction ( ∨ ) and negation ( ¬ ) —over the space of tasks . We then show that , having solved a series of base tasks , an agent can use its knowledge to solve tasks expressible as a Boolean expression over those tasks , without any further learning . We consider a family of related MDPsM restricted by the following assumptions : Assumption 1 . For all tasks in a set of tasks M , ( i ) the tasks share the same state space , action space and transition dynamics , ( ii ) the transition dynamics are deterministic , ( iii ) reward functions between tasks differ only on the absorbing set G , and ( iv ) the set of possible terminal rewards consists of only two values . That is , for all ( g , a ) in G × A , we have that r ( g , a ) ∈ { r∅ , rU } ⊂ R with r∅ ≤ rU . For all non-terminal states , we denote the reward rs , a to emphasise that it is constant across tasks . Assumption 2 . For all tasks in a set of tasksM which adhere to Assumption 1 , the set of possible terminal rewards consists of only two values . That is , for all ( g , a ) in G ×A , we have that r ( g , a ) ∈ { r∅ , rU } ⊂ R with r∅ ≤ rU . For all non-terminal states , we denote the reward rs , a to emphasise that it is constant across tasks . Assumption 1 is similar to that of Todorov ( 2007 ) and identical to Van Niekerk et al . ( 2019 ) , and imply that each task can be uniquely specified by its reward function . Furthermore , we note that Assumption 2 is only necessary to formally define the Boolean algebra . Although we have placed restrictions on the reward functions , the above formulation still allows for a large number of tasks to be represented . Importantly , sparse rewards can be formulated under these restrictions . 3.1 A BOOLEAN ALGEBRA FOR TASKS . An abstract Boolean algebra is a set B equipped with operators ¬ , ∨ , ∧ that satisfy the Boolean axioms of ( i ) idempotence , ( ii ) commutativity , ( iii ) associativity , ( iv ) absorption , ( v ) distributivity , ( vi ) identity , and ( vii ) complements.2 Given the above definitions and the restrictions placed on the set of tasks we consider , we can now define a Boolean algebra over a set of tasks . Theorem 1 . LetM be a set of tasks . DefineMU , M∅ ∈M to be tasks with the respective reward functions rMU : S ×A → R ( s , a ) 7→ { rU , if s ∈ G rs , a , otherwise . rM∅ : S ×A → R ( s , a ) 7→ { r∅ , if s ∈ G rs , a , otherwise . 1Since we consider undiscounted MDPs , we can ensure the value function is bounded by augmenting the state space with a virtual state ω such that ρ ( s , a ) ( ω ) = 1 for all ( s , a ) in G ×A , and r = 0 after reaching ω . 2We provide a description of these axioms in the Appendix . Then M forms a Boolean algebra with universal bounds M∅ and MU when equipped with the following operators : ¬ : M→M M 7→ ( S , A , ρ , r¬M ) , where r¬M : S ×A → R ( s , a ) 7→ ( rMU ( s , a ) + rM∅ ( s , a ) ) − rM ( s , a ) ∨ : M×M→M ( M1 , M2 ) 7→ ( S , A , ρ , rM1∨M2 ) , where rM1∨M2 : S ×A → R ( s , a ) 7→ max { rM1 ( s , a ) , rM2 ( s , a ) } ∧ : M×M→M ( M1 , M2 ) 7→ ( S , A , ρ , rM1∧M2 ) , where rM1∧M2 : S ×A → R ( s , a ) 7→ min { rM1 ( s , a ) , rM2 ( s , a ) } Proof . See Appendix . Theorem 1 allows us to compose existing tasks together to create new tasks in a principled way . Figure 1 illustrates the semantics for each of the Boolean operators in a simple environment . 3.2 EXTENDED VALUE FUNCTIONS . The reward and value functions described in Section 2 are insufficient to solve tasks specified by the Boolean algebra above . We therefore extend these to define goal-oriented versions of the reward and value function , given by the following two definitions : Definition 1 . The extended reward function r̄ : S × G ×A → R is given by the mapping ( s , g , a ) 7→ { N if g 6= s ∈ G r ( s , a ) otherwise , ( 1 ) where N ≤ min { rMIN , ( rMIN − rMAX ) D } , and D is the diameter of the MDP ( Jaksch et al. , 2010 ) .3 To understand why standard value functions are insufficient , consider two tasks that have multiple different goals , but at least one common goal . Clearly , there is a meaningful conjunction between them—namely , achieving the common goal . Now consider an agent that learns standard value functions for both tasks , and which is then required to solve their conjunction without further learning . Note that this is impossible in general , since the regular value function for each task only represents the value of each state with respect to the nearest goal . That is , for all states where the nearest goal for each task is not the common goal , the agent has no information about that common goal . Conversely , by learning extended value functions , the agent is able to learn the value of achieving all goals , and not simply the nearest one . Because we require that tasks share the same transition dynamics , we also require that the absorbing set of states is shared . Thus the extended reward function adds the extra constraint that , if the agent enters a terminal state for a different task , it should receive the largest penalty possible . In practice , we can simply set N to be the lowest finite value representable by the data type used for rewards . Definition 2 . The extended Q-value function Q̄ : S × G ×A → R is given by the mapping ( s , g , a ) 7→ r̄ ( s , g , a ) + ∫ S V̄ π̄ ( s′ , g ) ρ ( s , a ) ( ds ′ ) , ( 2 ) where V̄ π̄ ( s , g ) = Eπ̄ [ ∑∞ t=0 r̄ ( st , g , at ) ] . The extended Q-value function is similar to universal value function approximators ( UVFAs ) ( Schaul et al. , 2015 ) , but differs in that it uses the extended reward function definition . It is also similar to DG functions ( Kaelbling , 1993 ) , except here we use task-dependent reward functions , as opposed to measuring distance between states . The standard reward functions and value functions can be recovered from their extended versions through the following lemma . 3The diameter is defined as D = maxs 6=s′∈S minπ E [ T ( s′|π , s ) ] , where T is the number of timesteps required to first reach s′ from s under π. Lemma 1 . Let rM , r̄M , Q∗M , Q̄∗M be the reward function , extended reward function , optimal Qvalue function , and optimal extended Q-value function for a task M in M. Then for all ( s , a ) in S ×A , we have ( i ) rM ( s , a ) = max g∈G r̄M ( s , g , a ) , and ( ii ) Q∗M ( s , a ) = max g∈G Q̄∗M ( s , g , a ) .

Summary: The paper proposes a method of combining value functions for a certain class of tasks, including shortest path problems, to solve composed tasks. By expressing tasks as a Boolean algebra, they can be combined using the negation, conjunction and disjunction operations. Analogous operations are available for the optimal value functions of the tasks, which allows the agent to have immediate access to the optimal policy of these composed tasks after solving the base tasks. The theoretical composition properties are confirmed empirically on the four rooms environment and with function approximation on a more complex domain.