Learning to Represent Action Values as a Hypergraph on the Action Vertices

1 INTRODUCTION . Representation learning methods have helped shape recent progress in RL by enabling a capacity for learning good representations of state . This is in spite of the fact that , traditionally , representation learning was less often explored in the RL context . As such , the de facto representation learning techniques which are widely used in RL were developed under other machine learning paradigms ( Bengio et al. , 2013 ) . Nevertheless , RL brings some unique problems to the topic of representation learning , with exciting headway being made in identifying and exploring such topics . Action-value estimation is a critical component of the RL paradigm ( Sutton & Barto , 2018 ) . Hence , how to effectively learn estimators for action value from training samples is one of the major problems studied in RL . We set out to study this problem through the lens of representation learning , focusing particularly on learning representations of action in multi-dimensional discrete action spaces . While action values are conditioned on both state and action and as such good representations of both would be beneficial , there has been comparatively little research on learning action representations . We frame this problem as learning a decomposition of the action-value function that is structured in such a way to leverage the combinatorial structure of multi-dimensional discrete action spaces . This structure is an inductive bias which we incorporate in the form of architectural assumptions . We present this approach as a framework to flexibly build architectures for learning representations of multi-dimensional discrete actions by leveraging various orders of their underlying sub-action combinations . Our architectures can be combined in succession with any other architecture for learning state representations and trained end-to-end using backpropagation , without imposing any change to the RL algorithm . We remark that designing representation learning methods by incorporating some form of structural inductive biases is highly common in deep learning , resulting in highly-publicised architectures such as convolutional , recurrent , and graph networks ( Battaglia et al. , 2018 ) . We first demonstrate the effectiveness of our approach in illustrative , structured prediction problems . Then , we argue for the ubiquity of similar structures and test our approach in standard RL problems . ∗Correspondence to : Arash Tavakoli < a.tavakoli @ imperial.ac.uk > . Our results advocate for the general usefulness of leveraging the combinatorial structure of multidimensional discrete action spaces , especially in problems with larger action spaces . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . We consider the RL problem in which the interaction of an agent and the environment is modelled as a Markov decision process ( MDP ) ( S , A , P , R , S0 ) , where S denotes the state space , A the action space , P the state-transition distribution , R the reward distribution , and S0 the initial-state distribution ( Sutton & Barto , 2018 ) . At each step t the agent observes a state st ∈S and produces an action at∈A drawn from its policy π ( . | st ) . The agent then transitions to and observes the next state st+1∈S , drawn from P ( . | st , at ) , and receives a reward rt+1 , drawn from R ( . | st , at , st+1 ) . The standard MDP formulation generally abstracts away the combination of sub-actions that are activated when an action at is chosen . That is , if a problem has an Nv-dimensional action space , each action at maps onto an Nv-tuple ( a1t , a 2 t , . . . , a Nv t ) , where each a i t is a sub-action from the ith sub-action space . Therefore , the action space could have an underlying combinatorial structure where the set of actions is formed as a Cartesian product of the sub-action spaces . To make this explicit , we express the action space as A .= A1 ×A2 × · · · ×ANv , where each Ai is a finite set of sub-actions . Furthermore , we amend our notation for the actions at into at ( in bold ) to reflect that actions are generally combinations of several sub-actions . Within our framework , we refer to each sub-action spaceAi as an action vertex . As such , the cardinality of the set of action vertices is equal to the number of action dimensions Nv . Given a policy π that maps states onto distributions over the actions , the discounted sum of future rewards under π is denoted by the random variable Zπ ( s , a ) = ∑∞ t=0 γ trt+1 , where s0 = s , a0 = a , st+1∼P ( . | st , at ) , rt+1∼R ( . | st , at , st+1 ) , at∼π ( . | st ) , and 0≤ γ≤ 1 is a discount factor . The action-value function is defined as Qπ ( s , a ) = E [ Zπ ( s , a ) ] . Evaluating the action-value function Qπ of a policy π is referred to as a prediction problem . In a control problem the objective is to find an optimal policy π∗ which maximises the action-value function . The thesis of this paper applies to any method for prediction or control provided that they involve estimating an action-value function . A canonical example of such a method for control is Q-learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) which iteratively improves an estimate Q of the optimal action-value function Q∗ via Q ( st , at ) ← Q ( st , at ) + α ( rt+1 + γmax a′ Q ( st+1 , a ′ ) −Q ( st , at ) ) , ( 1 ) where 0 ≤ α ≤ 1 is a learning rate . The action-value function is typically approximated using a parameterised function Qθ , where θ is a vector of parameters , and trained by minimising a sequence of squared temporal-difference errors δ2t . = ( rt+1 + γmax a′ Qθ ( st+1 , a ′ ) −Qθ ( st , at ) ) 2 ( 2 ) over samples ( st , at , rt+1 , st+1 ) . Deep Q-networks ( DQN ) ( Mnih et al. , 2015 ) combine Q-learning with deep neural networks to achieve human-level performance in Atari 2600 . 2.2 DEFINITION OF HYPERGRAPH . A hypergraph ( Berge , 1989 ) is a generalisation of a graph in which an edge , also known as a hyperedge , can join any number of vertices . Let V = { A1 , A2 , . . . , ANv } be a finite set representing the set of action vertices Ai . A hypergraph on V is a family of subsets or hyperedges H = { E1 , E2 , . . . , ENe } such that Ej 6= ∅ ( j = 1 , 2 , . . . , Ne ) , ( 3 ) ∪N e j=1 Ej = V. ( 4 ) According to Eq . ( 3 ) , each hyperedge Ej is a member of E = P ( V ) \∅ , where P ( V ) , called the powerset of V , is the set of possible subsets on V . The rank r of a hypergraph is defined as the maximum cardinality of any of its hyperedges . We define a c-hyperedge , where c∈ { 1 , 2 , . . . , Nv } , as a hyperedge with cardinality or order c. The number of possible c-hyperedges on V is given by the binomial coefficient ( Nv c ) . We define a c-uniform hypergraph as one with only c-hyperedges . As a special case , a c-complete hypergraph , denoted Kc , is one with all possible c-hyperedges . 3 ACTION HYPERGRAPH NETWORKS FRAMEWORK . We now describe our framework using the example in Fig . 1 . Consider the sample physical system of Fig . 1a with six action vertices ( solid circles ) . A sample hypergraph is depicted for this system with four hyperedges ( dashed shapes ) , featuring a 1-hyperedge , two 2-hyperedges , and a 3-hyperedge . We remark that this set of hyperedges constitutes a hypergraph as there are no empty hyperedges ( Eq . ( 3 ) ) and the union of hyperedges spans the set of action vertices ( Eq . ( 4 ) ) . We wish to enable learning a representation of each hyperedge in an arbitrary hypergraph . To achieve this , we create a parameterised function UEj ( e.g . a neural network ) for each hyperedge Ej which receives a state representation ψ ( s ) as input and returns as many values as the possible combinations of the sub-actions for the action vertices enclosed by its respective hyperedge . In other words , UEj has as many outputs as the cardinality of a Cartesian product of the action vertices in Ej . Each such hyperedge-specific function UEj is a building block of our action hypergraph networks framework . 1 Figure 1b depicts the block corresponding to the 3-hyperedge from the hypergraph of Fig . 1a . We remark that for any action a = ( a1 , a2 , . . . , aN v ) from the action space A , each block has only one output that corresponds to a and , thus , contributes exactly one value as a representation of its respective hyperedge at the given action . This output UEj ( ψ ( s ) , aEj ) is identified by aEj which denotes the combination of sub-actions in a that correspond to the action vertices enclosed by Ej . We can realise an architecture within our framework by composing several such building blocks , one for each hyperedge in a hypergraph of our choosing . Figure 1c shows an instance architecture corresponding to the sample hypergraph of Fig . 1a . The forward view through this architecture is as follows . A shared representation of the input state s is fed into multiple blocks , each of which features a unique hyperedge . Then , a representation vector of size Ne is obtained for each action a , where Ne is the number of hyperedges or blocks . These action-specific representations are then mixed ( on an action-by-action basis ) using a function f ( e.g . a fixed non-parametric function or a neural network ) . The output of this mixing function is our estimator for action value at the stateaction pair ( s , a ) . Concretely , Q ( s , a ) . = f ( UE1 ( ψ ( s ) , aE1 ) , UE2 ( ψ ( s ) , aE2 ) , . . . , UENe ( ψ ( s ) , aENe ) ) . ( 5 ) While only a single output from each block is relevant for an action , the reverse is not the case . That is , an output from a block generally contributes to more than a single action ’ s value estimate . In fact , the lower the cardinality of a hyperedge , the larger the number of actions ’ value estimates to which a block output contributes . This can be thought of as a form of combinatorial generalisation . 1Throughout this paper we assume linear units for the block outputs . However , other activation functions could be more useful depending on the choice of mixing function or the task . Particularly , action value can be estimated for an insufficiently-explored or unexplored action by mixing the action ’ s corresponding representations which have been trained as parts of other actions . Moreover , this structure enables a capacity for learning faster on lower-order hyperedges by receiving more updates and slower on higher-order ones by receiving less updates . This is a desirable intrinsic property as , ideally , we would wish to learn representations that exhaust their capacity for representing action values using lower-order hyperedges before they resort to higher-order ones .

Summary: This work focuses on learning action representations for problems involving high-dimensional action spaces. The aim is to build a flexible and general methodology for learning representations of multidimensional actions that can be combined with existing architectures (which mostly focus on learning state representations). The ideas initially suggested in Sharma et al (2017) have been generalized and refined in this work leading to impressive empirical results.