Value Propagation Networks

1 Introduction . Planning is a key component for artificial agents in a variety of domains . However , a limit of classical planning algorithms is that one needs to know how to search for an optimal – or at least reasonable – solution for each instantiation of every possible type of plan . As the environment dynamics and states complexity increase , this makes writing planners difficult , cumbersome , or simply entirely impractical . This is among the reasons why “ learning to plan ” has been an active research area to address these shortcomings ( Russell et al. , 1995 ; Kaelbling et al. , 1996 ) . To be useful in practice we propose that methods that enable to learn planners should have at least two properties : they should be traces free , i.e . not require traces from an optimal planner , and they should generalize , i.e . learn planners that are able to function on plans of the same type but of unseen instance and/or planning horizons . In Reinforcement Learning ( RL ) , learning to plan can be framed as the problem of finding a policy that maximises the expected return from the environment , where such policy is a greedy function that selects actions that will visit states with a higher value for the agent . This in turns shifts the problem to the one of obtaining good estimates of state values . One of the most commonly used algorithms to solve this problem is Value Iteration ( VI ) , which estimates the state values by collecting and propagating the observed rewards until a fixed point is reached . A policy – or a plan – can then be constructed by rolling out the obtained value function on the desired state-action pairs . When the environment can be represented as an occupancy map , a 2D grid , it is possible to approximate this planning algorithm using a deep convolutional neural network ( CNN ) to propagate the rewards on the grid cells . This enables one to differentiate directly through the planner steps and perform end-to-end learning of the value function . Tamar et al . ( 2016 ) train such models – Value Iteration Networks ( VIN ) – with a supervised loss on the trace from a search/planning algorithm , with the goal to find the parameters that can solve the shortest path task in such environments by iteratively learning the value function using the ∗Work was partly done while at Facebook AI Research . †Work done while at Microsoft Research . convnet . However , this baseline requires good target value estimates , violating our wished trace free property , and limiting its usage in interactive , dynamic settings . Furthermore , it doesn ’ t take advantage of the model structure to generalise to harder instances of the task . In this work we extend the formalization used in VIN to more accurately represent the structure of grid-world-like scenarios , enabling Value Iteration network modules to be naturally used within the reinforcement learning framework beyond the scope of the initial work , while also removing some of the limitations and underlying assumptions constraining the original architecture . We show that our models can not only learn to plan and navigate in dynamic environments , but that their hierarchical structure provides a way to generalize to navigation tasks where the required planning horizon and the size of the map are much larger than the ones seen at training time . Our main contributions include : ( 1 ) introducing VProp and MVProp , network planning modules which successfully learn to solve pathfinding tasks via reinforcement learning using minimal parametrization , ( 2 ) demonstrating the ability to generalize to large unseen maps when training exclusively on much smaller ones , and ( 3 ) showing that our modules can learn to plan in environments with more complex dynamics than a static grid world , both in terms of transition function and observation complexity . 1.1 Related work . Model-based planning with end-to-end architectures has recently shown promising results on a variety of tasks and environments , often using Deep Reinforcement Learning as the algorithmic framework ( Silver et al. , 2016 ; Oh et al. , 2017 ; Weber et al. , 2017 ; Groshev et al. , 2017 ; Farquhar et al. , 2017 ; Serban et al. , 2018 ; Srinivas et al. , 2018 ) . 3D and 2D navigation tasks have also been tackled within the RL framework ( Mirowski et al. , 2016 ) , with methods in some cases building and conditioning on 2D occupancy maps to aid the process of localization and feature grounding ( Bhatti et al. , 2016 ; Zhang et al. , 2017 ; Banino et al. , 2018 ) . Other work has furthermore explored the usage of VIN-like architectures for navigation problems : Niu et al . ( 2017 ) present a generalization of VIN able to learn modules on more generic graph structures by employing a graph convolutional operator to convolve through each node of the graph . Rehder et al . ( 2017 ) demonstrate a method for multi-agent planning in a cooperative setting by training multiple VI modules and composing them into one network , while also adding an orientation state channel to simulate non-holonomic constraints often found in mobile robotics . Gupta et al . ( 2017 ) and Khan et al . ( 2017 ) propose to tackle partially observable settings by constructing hierarchical planners that use VI modules in a multi-scale fashion to generate plans and condition the model ’ s belief state . 2 Background . We consider the control of an agent in a “ grid world ” environment , in which entities can interact with each other . The entities have some set of attributes , including a uniquely defined type , which describes how they interact with each other , the immediate rewards of such interactions , and how such interactions affect the next state of the world . The goal is to learn to plan through reinforcement learning , that is learning a policy trained on various configurations of the environment that can generalize to arbitrary other configurations of the environment , including larger environments , and ones with a larger number of entities . In the case of a standard navigation task , this boils down to learning a policy which , given an observation of the world , will output actions that take the agent to the goal as quickly as possible . An agent may observe such environments as 2D images of size dx × dy , with dpix input panes , which are then potentially passed through an embedding function Φ ( such as a 2D convnet ) to extract the entities and generates some local embedding based on their positions and features . 2.1 Reinforcement Learning . The problem of reinforcement learning is typically formulated in terms of computing optimal policies for a Markov Decision Problem ( MDP ) ( Sutton and Barto , 1998 ) . An MDP is defined by the tuple ( S , A , T , R , γ ) , where S is a finite set of states , A is the set of actions a that the agent can take , T : s→ a→ s′ is a function describing the state-transition matrix , R is a reward function , and γ is a discount factor . In this setting , an optimal policy π∗ is a distribution over the state-action space that maximises in expectation the discounted sum of rewards ∑ k γ krk , where rk is the single-step reward . A standard method to find the optimal policy π : s → a is to iteratively compute the value function , Qπ ( s , a ) , updating it based on rewards received from the environment ( Watkins and Dayan , 1992 ) . Using this framework , we can view learning to plan as a structured prediction of rewards with the planning algorithm Value Iteration ( Bertsekas , 2012 ) as inference procedure . Beside value-based algorithms , there exist other types which are able to find optimal policies , such as policy gradient methods ( Sutton et al. , 1999 ) , which directly regress to the policy function π instead of approximating the value function . These methods however suffer from high variance estimates in environments that require many steps . Finally , a third type is represented by actor-critic algorithms , which combine the policy gradient methods ’ advantage of being able to compute the policy directly , with the low-variance performance estimation of value-based RL used as a more accurate feedback signal to the policy estimator ( Konda and Tsitsiklis , 2000 ) . 2.2 Value Iteration Module . Tamar et al . ( 2016 ) motivate the Value Iteration module by observing that for problems such as navigation , and more generally pathfinding , value iteration can be unrolled as a graph convolutional network , where nodes are possible positions of the agent and edges represent possible transitions given the agent ’ s actions . In the simpler case of 2D grids , the graph structure corresponds to a neighborhood in the 2D space , and the convolution structure is similar to a convolutional network that takes the entire 2D environment as input . More precisely , let us denote by s the current observation of the environment ( e.g. , a bird ’ seye view of the 2D grid ) , q0 the zero tensor of dimensions ( A , dx , dy ) , where dx , dy are the dimension of the 2D grid , and A the number of actions of the agent . Then , the Value Iteration module is defined by an embedding function of the state Φ ( s ) ∈ Rdrew×dx×dy ( where drew depends on the model ) , a transition function h , and performs the following computations for steps k = 1 ... K where K is the depth of the VI module : ∀ ( i , j ) ∈ [ [ dx ] ] × [ [ dy ] ] , vkij = max a=1 .. A qka , i , j , qk = h ( Φ ( s ) , vk−1 ) . Given the agent ’ s position ( x0 , y0 ) and the current observation of the environment s , the control policy π is then defined by π ( s , ( x0 , y0 ) ) = argmaxa=1 .. A q K a , x0 , y0 . We can rewrite the transition function h as a convolutional layer : ( a ) r̄i , j = Φ ( s ) i , j , ( b ) v0i , j = 0 , q k a , i , j = ∑ ( i′ , j′ ) ∈N ( i , j ) p ( v ) a , i′−i , j′−j ∗ v k−1 i′ , j′ + p ( r ) a , i′−i , j′−j ∗ r̄i′ , j′ , v k i , j = max a qk−1a , i , j , where N ( i , j ) is the set of neighbors of the cell ( i , j ) and the cell itself . In practice , Φ ( s ) has several output channels drew and a varying number of parameters when used within a VI module on different tasks . p ( v ) a , i′−i , j′−j ∈ R and p ( r ) a , i′−i , j′−j ∈ R instead purely represent the parameters of the VI module . This model is appealing as a neural network architecture because the step to compute each qka , i , j is a convolutional layer , thus enabling computing q K a , i , j with a convolutional network of depth K with shared weights and as many output channels as actions . The computation of vki , j are then the non-linearities of the network , which correspond to a max-pooling in the dimension of the output channels . To clarify the relationship with the original value iteration algorithm for grid worlds , let us denote by Ra , i , j , i′ , j′ the immediate reward when taking action a at position ( i , j ) with arrival position ( i′ , j′ ) , Pa , i , j , i′ , j′ the corresponding transition probabilities , and γ the discount factor . Then value iteration in a 2D grid can be generally written as : ∀ ( i , j ) ∈ [ [ dx ] ] × [ [ dy ] ] , V kij = max a=1 .. A Qka , i , j , ∀ ( a , i , j ) ∈ [ [ A ] ] × [ [ dx ] ] × [ [ dy ] ] , Qka , i , j = ∑ ( i′ , j′ ) ∈N ( i , j ) Pa , i , j , i′ , j′ ( Ra , i , j , i′ , j′ + γV k−1 i , j ) . where N ( i , j ) is prior knowledge about the states that are accessible from ( i , j ) . The implementation of the value iteration module described above thus corresponds to a specific parameterization of the reward Ra , i , j , i′ , j′ as functions of the starting and arrival states . More importantly , the implementation of the value iteration module with a convolutional network means that the transition probabilities , represented by the parameters p ( v ) a , i′−i , j′−j , are translation-invariant .

Summary: The paper presents an extension of the original value iteration networks (VIN) by considering state-dependent transition function, which alleviates the limitation of VIN to translation-invariant transition functions and further constraining the reward function parametrization to improve sample efficiency of learning to plan algorithms. The first problem is addressed  by interpreting transition probabilities as state-dependent discount factors, given by a sigmoid function that takes as input state features. The second problem is addressed by defining the reward function as the difference between an input reward and an output cost. Obstacle states are given a high cost. The proposed method is evaluated on random grids of different sizes, of the same type as the grids considered in the VIN paper. Comparaisons with VIN show that the proposed MVProp approach outperforms VIN by several orders of magnitude and can learn optimal plans in less than a thousand episodes, compared to VIN that doesn't seem here to learn much even after 30 thousands episodes.