Learning to Represent Action Values as a Hypergraph on the Action Vertices

1 INTRODUCTION . Representation learning methods have helped shape recent progress in RL by enabling a capacity for learning good representations of state . This is in spite of the fact that , traditionally , representation learning was less often explored in the RL context . As such , the de facto representation learning techniques which are widely used in RL were developed under other machine learning paradigms ( Bengio et al. , 2013 ) . Nevertheless , RL brings some unique problems to the topic of representation learning , with exciting headway being made in identifying and exploring such topics . Action-value estimation is a critical component of the RL paradigm ( Sutton & Barto , 2018 ) . Hence , how to effectively learn estimators for action value from training samples is one of the major problems studied in RL . We set out to study this problem through the lens of representation learning , focusing particularly on learning representations of action in multi-dimensional discrete action spaces . While action values are conditioned on both state and action and as such good representations of both would be beneficial , there has been comparatively little research on learning action representations . We frame this problem as learning a decomposition of the action-value function that is structured in such a way to leverage the combinatorial structure of multi-dimensional discrete action spaces . This structure is an inductive bias which we incorporate in the form of architectural assumptions . We present this approach as a framework to flexibly build architectures for learning representations of multi-dimensional discrete actions by leveraging various orders of their underlying sub-action combinations . Our architectures can be combined in succession with any other architecture for learning state representations and trained end-to-end using backpropagation , without imposing any change to the RL algorithm . We remark that designing representation learning methods by incorporating some form of structural inductive biases is highly common in deep learning , resulting in highly-publicised architectures such as convolutional , recurrent , and graph networks ( Battaglia et al. , 2018 ) . We first demonstrate the effectiveness of our approach in illustrative , structured prediction problems . Then , we argue for the ubiquity of similar structures and test our approach in standard RL problems . ∗Correspondence to : Arash Tavakoli < a.tavakoli @ imperial.ac.uk > . Our results advocate for the general usefulness of leveraging the combinatorial structure of multidimensional discrete action spaces , especially in problems with larger action spaces . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . We consider the RL problem in which the interaction of an agent and the environment is modelled as a Markov decision process ( MDP ) ( S , A , P , R , S0 ) , where S denotes the state space , A the action space , P the state-transition distribution , R the reward distribution , and S0 the initial-state distribution ( Sutton & Barto , 2018 ) . At each step t the agent observes a state st ∈S and produces an action at∈A drawn from its policy π ( . | st ) . The agent then transitions to and observes the next state st+1∈S , drawn from P ( . | st , at ) , and receives a reward rt+1 , drawn from R ( . | st , at , st+1 ) . The standard MDP formulation generally abstracts away the combination of sub-actions that are activated when an action at is chosen . That is , if a problem has an Nv-dimensional action space , each action at maps onto an Nv-tuple ( a1t , a 2 t , . . . , a Nv t ) , where each a i t is a sub-action from the ith sub-action space . Therefore , the action space could have an underlying combinatorial structure where the set of actions is formed as a Cartesian product of the sub-action spaces . To make this explicit , we express the action space as A .= A1 ×A2 × · · · ×ANv , where each Ai is a finite set of sub-actions . Furthermore , we amend our notation for the actions at into at ( in bold ) to reflect that actions are generally combinations of several sub-actions . Within our framework , we refer to each sub-action spaceAi as an action vertex . As such , the cardinality of the set of action vertices is equal to the number of action dimensions Nv . Given a policy π that maps states onto distributions over the actions , the discounted sum of future rewards under π is denoted by the random variable Zπ ( s , a ) = ∑∞ t=0 γ trt+1 , where s0 = s , a0 = a , st+1∼P ( . | st , at ) , rt+1∼R ( . | st , at , st+1 ) , at∼π ( . | st ) , and 0≤ γ≤ 1 is a discount factor . The action-value function is defined as Qπ ( s , a ) = E [ Zπ ( s , a ) ] . Evaluating the action-value function Qπ of a policy π is referred to as a prediction problem . In a control problem the objective is to find an optimal policy π∗ which maximises the action-value function . The thesis of this paper applies to any method for prediction or control provided that they involve estimating an action-value function . A canonical example of such a method for control is Q-learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) which iteratively improves an estimate Q of the optimal action-value function Q∗ via Q ( st , at ) ← Q ( st , at ) + α ( rt+1 + γmax a′ Q ( st+1 , a ′ ) −Q ( st , at ) ) , ( 1 ) where 0 ≤ α ≤ 1 is a learning rate . The action-value function is typically approximated using a parameterised function Qθ , where θ is a vector of parameters , and trained by minimising a sequence of squared temporal-difference errors δ2t . = ( rt+1 + γmax a′ Qθ ( st+1 , a ′ ) −Qθ ( st , at ) ) 2 ( 2 ) over samples ( st , at , rt+1 , st+1 ) . Deep Q-networks ( DQN ) ( Mnih et al. , 2015 ) combine Q-learning with deep neural networks to achieve human-level performance in Atari 2600 . 2.2 DEFINITION OF HYPERGRAPH . A hypergraph ( Berge , 1989 ) is a generalisation of a graph in which an edge , also known as a hyperedge , can join any number of vertices . Let V = { A1 , A2 , . . . , ANv } be a finite set representing the set of action vertices Ai . A hypergraph on V is a family of subsets or hyperedges H = { E1 , E2 , . . . , ENe } such that Ej 6= ∅ ( j = 1 , 2 , . . . , Ne ) , ( 3 ) ∪N e j=1 Ej = V. ( 4 ) According to Eq . ( 3 ) , each hyperedge Ej is a member of E = P ( V ) \∅ , where P ( V ) , called the powerset of V , is the set of possible subsets on V . The rank r of a hypergraph is defined as the maximum cardinality of any of its hyperedges . We define a c-hyperedge , where c∈ { 1 , 2 , . . . , Nv } , as a hyperedge with cardinality or order c. The number of possible c-hyperedges on V is given by the binomial coefficient ( Nv c ) . We define a c-uniform hypergraph as one with only c-hyperedges . As a special case , a c-complete hypergraph , denoted Kc , is one with all possible c-hyperedges . 3 ACTION HYPERGRAPH NETWORKS FRAMEWORK . We now describe our framework using the example in Fig . 1 . Consider the sample physical system of Fig . 1a with six action vertices ( solid circles ) . A sample hypergraph is depicted for this system with four hyperedges ( dashed shapes ) , featuring a 1-hyperedge , two 2-hyperedges , and a 3-hyperedge . We remark that this set of hyperedges constitutes a hypergraph as there are no empty hyperedges ( Eq . ( 3 ) ) and the union of hyperedges spans the set of action vertices ( Eq . ( 4 ) ) . We wish to enable learning a representation of each hyperedge in an arbitrary hypergraph . To achieve this , we create a parameterised function UEj ( e.g . a neural network ) for each hyperedge Ej which receives a state representation ψ ( s ) as input and returns as many values as the possible combinations of the sub-actions for the action vertices enclosed by its respective hyperedge . In other words , UEj has as many outputs as the cardinality of a Cartesian product of the action vertices in Ej . Each such hyperedge-specific function UEj is a building block of our action hypergraph networks framework . 1 Figure 1b depicts the block corresponding to the 3-hyperedge from the hypergraph of Fig . 1a . We remark that for any action a = ( a1 , a2 , . . . , aN v ) from the action space A , each block has only one output that corresponds to a and , thus , contributes exactly one value as a representation of its respective hyperedge at the given action . This output UEj ( ψ ( s ) , aEj ) is identified by aEj which denotes the combination of sub-actions in a that correspond to the action vertices enclosed by Ej . We can realise an architecture within our framework by composing several such building blocks , one for each hyperedge in a hypergraph of our choosing . Figure 1c shows an instance architecture corresponding to the sample hypergraph of Fig . 1a . The forward view through this architecture is as follows . A shared representation of the input state s is fed into multiple blocks , each of which features a unique hyperedge . Then , a representation vector of size Ne is obtained for each action a , where Ne is the number of hyperedges or blocks . These action-specific representations are then mixed ( on an action-by-action basis ) using a function f ( e.g . a fixed non-parametric function or a neural network ) . The output of this mixing function is our estimator for action value at the stateaction pair ( s , a ) . Concretely , Q ( s , a ) . = f ( UE1 ( ψ ( s ) , aE1 ) , UE2 ( ψ ( s ) , aE2 ) , . . . , UENe ( ψ ( s ) , aENe ) ) . ( 5 ) While only a single output from each block is relevant for an action , the reverse is not the case . That is , an output from a block generally contributes to more than a single action ’ s value estimate . In fact , the lower the cardinality of a hyperedge , the larger the number of actions ’ value estimates to which a block output contributes . This can be thought of as a form of combinatorial generalisation . 1Throughout this paper we assume linear units for the block outputs . However , other activation functions could be more useful depending on the choice of mixing function or the task . Particularly , action value can be estimated for an insufficiently-explored or unexplored action by mixing the action ’ s corresponding representations which have been trained as parts of other actions . Moreover , this structure enables a capacity for learning faster on lower-order hyperedges by receiving more updates and slower on higher-order ones by receiving less updates . This is a desirable intrinsic property as , ideally , we would wish to learn representations that exhaust their capacity for representing action values using lower-order hyperedges before they resort to higher-order ones .

Summary: This paper incorporates a concept called hypergraph network into reinforcement learning. The idea of hypergraph is to extend edge to hyperedge where a set of vertices can be considered at the same time. This seems natural for scenarios like continuous action control with multi-dimension action space. From experimental results this proposed action hypergraph networks outperform several existing baselines.