Know Your Action Set: Learning Action Relations for Reinforcement Learning

Intelligent agents can solve tasks in a variety of ways depending on the action set at their disposal . For instance , while using a toolkit for repair , the choice of tool ( the action ) closely depends on what other tools are available . Yet , such dependence on other available actions is ignored in conventional reinforcement learning ( RL ) since it assumes a fixed action set . In this work , we posit that learning the interdependence between actions is crucial for RL agents acting under a varying action set . To this end , we propose a novel policy architecture that consists of an input graph composed of available actions and a graph attention network to learn the action interdependence . We demonstrate that our architecture makes action decisions by correctly attending to the relevant actions in both value-based and policy-based RL . Consequently , it consistently outperforms non-relational architectures on applications where the action space can vary , such as recommender systems and physical reasoning with tools and skills . 1 INTRODUCTION . Imagine you want to hang a picture on the wall . You may start by placing a nail on the wall and then use a hammer to nail it in . However , if you do not have access to a hammer , you would not use the nail . Instead , you would try alternative approaches such as using a hook and adhesive-strips , or a screw and a drill ( Figure 1 ) . In general , we solve tasks by choosing actions that interact with each other to achieve a desired outcome in the environment . Therefore , the best action decision depends not only on the environment , but also on what other actions are available to use . In this work , we address the setting of varying action space in sequential decision-making . Typically reinforcement learning ( RL ) assumes a fixed action space , but recent work has explored variations in action space when adding or removing actions ( Boutilier et al. , 2018 ; Chandak et al. , 2020a ; b ) or for generalization to unseen actions ( Jain et al. , 2020 ) . These assume that the given actions can be treated independently in decision-making . But this assumption often does not hold . As the picture hanging example illustrates , the optimality of choosing the nail is dependent on the availability of a hammer . Therefore , our goal is to address this problem of learning the interdependence of actions . Addressing this problem is important for many varying action space applications . These include recommender systems where articles to recommend vary every day and physical reasoning where decision-making must be robust to any given set of tools , objects , or skills . In this paper , we benchmark three such scenarios where learning action interdependence is crucial for solving the task optimally . These are ( i ) shortcut-actions in grid navigation , whose presence shortens the path to the goal , ( ii ) co-dependent actions in tool reasoning , where tools need other tools to be useful , and ( iii ) list-actions or slate-actions ( Sunehag et al. , 2015 ) in simulated and real-data recommender systems , where the user satisfaction is a consequence of the collective effect of the recommended list . There are three key challenges in learning action interdependence for RL with varying action space . First , since the overall actions are not known in advance , the policy framework must be flexible to work with action representations . Second , the action space is now an additional variable component just like state observations in RL . Therefore , the policy framework must incorporate a variably sized set of action representations as part of the input . Finally , an agent ’ s decision for each action ’ s utility ( e.g . Q-value or probability ) should explicitly model its relationship with other available actions . Action Relations → Action Choice . To address these challenges , we propose a novel policy architecture : AGILE , Action Graph for Interdependence Learning . AGILE builds on the utility network proposed in Jain et al . ( 2020 ) to incorporate action representations . Its key component is a graph attention network ( GAT ) ( Veličković et al. , 2017 ) over a fully connected graph of actions . This serves two objectives : summarizing the action set input and computing the action utility with relational information of other actions . Our primary contribution is introducing the problem of learning action interdependence for RL with variable action space . We demonstrate our proposed policy architecture , AGILE , learns meaningful action relations . This enables it to improve optimality in varying action space tasks such as simulated and real-world recommender systems , and reasoning with skills and tools . 2 RELATED WORK . 2.1 STOCHASTIC ACTION SETS . In recent work , Boutilier et al . ( 2018 ) provide the theoretical foundations of MDPs with Stochastic Action Sets ( SAS-MDPs ) , where actions are sampled from a known base set of actions . They propose a solution with Q-learning and Chandak et al . ( 2020a ) extend it to policy gradients . An instance of SAS-MDPs is when certain actions become invalid , e.g . in games , they are masked out from the output action probability distribution ( Huang & Ontañón , 2020 ; Ye et al. , 2020 ; Kanervisto et al. , 2020 ) . However , the assumption of knowing the finite base action set limits the practical applicability of SAS-MDPs . For example , recommender agents often receive unseen items to recommend and a robotic agent does not know beforehand what tools it would encounter in the future . We work with action representations to alleviate this limitation . Furthermore , in SAS-MDPs the action set can vary at any timestep of an episode . Thus , the learned policy will only be optimal on average over all the possible action sets ( Qin et al. , 2020 ) . Whereas in our setting , the action sets only change at the beginning of a task instance and stays constant over the episode . This is a more practical setup and raises the challenging problem of solving a task optimally with any given action space . 2.2 ACTION REPRESENTATIONS . In discrete action RL , action representations have enabled learning in large action spaces ( DulacArnold et al. , 2015 ; Chandak et al. , 2019 ) , transfer learning to a different action space ( Chen et al. , 2019b ) , and efficient exploration by exploiting the shared structure among actions ( He et al. , 2015 ; Tennenholtz & Mannor , 2019 ; Kim et al. , 2019 ) . Recently , Chandak et al . ( 2020b ) use them to accelerate adaptation when new actions are added to an existing action set . In contrast , our setting requires learning in a constantly varying action space where actions can be added , removed , or completely replaced in an episode . Closely related to our work , Jain et al . ( 2020 ) assume a similar setting of varying action space while training to generalize to unseen actions . Following their motivation , we use action representations to avoid making assumptions on the base action set . However , their policy treats each action independently , which we demonstrate leads to a suboptimal performance . 2.3 LIST-WISE ACTION SPACE . In listwise RL ( or slate RL ) , the action space is combinatorial in list size . Commonly it has applications in recommendation systems ( Sunehag et al. , 2015 ; Zhao et al. , 2017 ; 2018 ; Ie et al. , 2019b ; Gong et al. , 2019 ; Liu et al. , 2021 ; Jiang et al. , 2018 ; Song et al. , 2020 ) . In recent work , Chen et al . ( 2019a ) proposed Cascaded DQNs ( CDQN ) framework which learns a Q-network for every index in the list and train them all with a shared reward . For our experiments on listwise RL , we utilize CDQN as the algorithm and show how it can be used with AGILE as the policy framework . 2.4 RELATIONAL REINFORCEMENT LEARNING . Graph neural networks ( Battaglia et al. , 2018 ) have been explored in RL tasks with a rich relational structure , such as morphological control ( Wang et al. , 2018 ; Sanchez-Gonzalez et al. , 2018 ; Pathak et al. , 2019 ) , multi-agent RL ( Tacchetti et al. , 2019 ) , physical construction ( Hamrick et al. , 2018 ) , and structured perception in games like StarCraft ( Zambaldi et al. , 2018 ) . In this paper , we propose that the set of actions possess a relational structure which enables the actions to interact and solve tasks in the environment . Therefore , we leverage graph attention networks ( Veličković et al. , 2017 ) to learn these action relations and show that it can represent meaningful action interactions . 3 PROBLEM FORMULATION . A hallmark of intelligence is the ability to be robust in an ever changing environment . To this end , we consider the setting of RL with a varying action space , where an agent receives a different action set in every task instance . Our key problem is to learn the interdependence of actions so the agent can act optimally with any given action set . Figure 1 illustrates that for a robot with the task of hanging a picture on the wall , starting with a nail is optimal only if it can access a hammer subsequently . 3.1 REINFORCEMENT LEARNING WITH VARYING ACTION SPACE . We consider episodic Markov Decision Processes ( MDPs ) with discrete action spaces , supplemented with action representations . The MDP is defined by a tuple { S , A , T , R , γ } of states , actions , transition probability , reward function , and discount factor , respectively . The base set of actions A can be countably infinite . To support infinite base actions , we use D-dimensional action representations ca ∈ RD to denote an action a ∈ A . These can be image or text features of a recommendable item , or behavior characteristics of a tool , or simply be one-hot vectors for a known and finite action set . In each instance of the MDP , a subset of actions A ⊂ A is given to the agent , with associated representations C. Thereafter , at each time step t in the episode , the agent receives a state observation st ∈ S from the environment and acts with at ∈ A . This results in a state transition to st+1 and a reward rt . The objective of the agent is to learn a policy π ( a|s , A ) that maximizes the expected discounted reward over evaluation episodes with potentially unseen actions , EA⊂A [ ∑ γt−1rt ] . 3.2 CHALLENGES OF VARYING ACTION SPACE . 1 . Using Action Representations : The policy framework should be flexible to take a set of action representations C as input , and output corresponding Q-values or probability distribution for RL . 2 . Action Set as part of State : When action set varies , the original state space S is not anymore Markovian . For example , the state of a robot hanging the picture is under-specified without knowing if its toolbox contains a hammer or not . The MDP can be preserved by reformulating the state space as S′ = { s ◦ CA : s ∈ S , A ⊂ A } to include the representations CA associated with the available actions A ( Boutilier et al. , 2018 ) . Thus , the policy framework must support the input of s ◦ CA , where A is a variably sized action set . 3 . Interdependence of Actions : The optimal choice of an action at ∈ A is dependent on the action choices that would be available in the future steps of the episode , at′ ∈ A. Recalling Figure 1 , a nail should only be picked initially from the toolbox if a hammer is accessible later . Thus , we must explicitly model the relationships between the characteristics of the current action cat and the possible future actions cai∀ai ∈ A .

Summary: This paper tackles an RL problem setting in which the actions available to an agent vary from episode to episode, and the optimal action in some states depends on the other actions that are available. The authors’ approach to this setting is to use a graph neural network to process all available actions, both to summarise the available set and to produce a relationally-informed representation for each action. These are then fed, with the state, into a utility function to give an action value or logit. Experiments in a number of benchmarks show the value of including information about the available actions.