Q-map: a Convolutional Approach for Goal-Oriented Reinforcement Learning

1 INTRODUCTION . In the reinforcement learning ( RL ) setting , an agent attempts to discover and learn to solve a particular task in the environment . The task is generally specified in the form of a reward signal that the agent strives to maximize . While such reward signal is sufficient to describe any task in the environment , it can be advantageous to provide the agent with the task description in form of a specific goal to reach . These goals could be provided by a designed curriculum , or generated by a different agent and have shown to be advantageous for exploration in sparse-reward tasks . General value functions ( GVFs ) formalize the concept of goal-oriented RL by creating a specific value function per goal ( Sutton et al. , 2011 ) , which can be combined in a fixed array ( also called Horde ) of independent daemons , each simultaneously trained off-policy . Universal value function approximators ( UVFAs ) extend GVFs by utilizing a single value function that is provided with the goal specification in input , enabling it to generalize between goals . One of the main challenges in such goal-parameterized policies is to make the best use of the data collected so far , to generalize to goals that were not necessarily selected when the data were collected . Training off-policy , using random goals and bootstrapping from the estimated value of next states allows in theory to learn to reach all goals simultaneously . With the Horde framework , this requires to update separately each of the daemons using a single transition while with UVFA , each goal needs to be passed in input to the same policy independently . Because the number of goals can be very large , possibly infinite , both of these approaches quickly become intractable if one wishes to update simultaneously towards all goals using a single transition . While goal specification can be very complex , for many RL environments , simple spacial coordinates are sufficient to describe many tasks of interest . ATARI or NES games are a great example of such environments , where the agent ’ s location is often easily identifiable , and thus the goal can be defined in terms of on-screen coordinates . These goals are likely to be highly correlated both to the neighbouring goals and to the visual surroundings . Our first contribution is an agent , called Q-map , that uses a convolutional autoencoder-like architecture for goal-oriented RL , that can be used to efficiently and simultaneously ( in a single forward pass ) produce value estimates for an entire range of possible goals in compatible environments . We describe how to efficiently train such an agent and show that such approach is able to generalize to unseen environments to a good degree . As a demonstration how Q-maps can be used , our second contribution is an exploration algorithm that exploits the capacity to extract at the same time an estimated distance towards all goals and a policy to reach one selected randomly at a desired distance . We demonstrate how the ε-greedy exploration mechanism in DQN ( Mnih et al. , 2015 ) can be replaced with random goals reaching , expanding the exploration boundaries much further , especially in environments where actions can cancel each other or lead to terminal states . The source code and several videos can be found at https : //sites.google.com/view/q-map-rl . 2 Q-MAP . 2.1 BACKGROUND . We consider the standard RL framework ( Sutton and Barto , 1998 ) , in which an agent interacts sequentially with its environment , formalized as a Markov Decision Process ( MDP ) with state space S , action space A , reward function r ( s , a , s′ ) and state-transition function p ( s′|s , a ) . At each time step t , the agent provides an action at based on the current state st and sampled from its policy π ( at|st ) . The environment responds by providing a new state st+1 and a reward rt+1 . Some states can be terminal , meaning that no more interaction is possible after reaching them , which can be simply considered as a deadlock state that only transition to itself , providing no reward . The action-value function of the policyQπ ( s , a ) = Es′∼p ( s′|s , a ) , a′∼π ( a′|s′ ) [ r ( s , a , s′ ) +γQπ ( s′ , a′ ) ] indicates the quality ( Q-value ) of each possible immediate action when following the policy afterwards . In the Q-learning algorithm ( Watkins and Dayan , 1992 ) , the action-value function of the optimal policy π∗ is iteratively approximated by updating the estimated Q-values Qt+1 ( s , a ) ← ( 1 − α ) Qt ( s , a ) + α ( r + γmaxa′ Qt ( s′ , a′ ) ) using previously experienced transitions ( s , a , s′ , r ) and a learning rate α . At each time step , this learned action-value function can be used to take greedy actions a = argmaxaQ ( s , a ) or random actions uniformly a ∼ U ( A ) with probability ε . This basic exploration method is called ε-greedy and using the estimate of the value at the next step to improve the estimate at the current step is known as bootstrapping . Finally , the fact that the target r + γmaxa′ Qt ( s′ , a′ ) does not rely on the policy used to generate the data , allows Q-learning to learn off-policy , efficiently re-using previous transitions stored in a replay buffer or generated by a another mechanism . While an action-value function is usually specific to the rewards defining the task , the GVFs Qπg ( s , a ) are trained with pseudo-reward functions rg ( s , a , s′ ) that are specific to each goal g. The Horde architecture combines a large number of independent GVFs trained to predict the effect of actions on sensor measurements and can be simultaneously trained off-policy . UVFAs extend the concept of GVFs by adopting a unique action-value function Qπ ( s , a , g ) parameterized by goals and states together , enabling interpolation and extrapolation between goals . It is possible to extract a notion of the distance γk−1 between the current state and the desired goal in terms of the number of actions k by training a GVF- or UVFA-based agent with a pseudo-reward of 1 and termination in case the goal is reached , or otherwise rewarding 0 . While the agent could in theory learn to reach any goal by attempting to reach them one at a time , the process can become very slow with increasing number of possible goals . However it is theoretically possible to use a single transition to update off-policy the Q-value estimates for the entire range of available goals . In hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) , the goals used for the updates are chosen in the episode that gave a transition ( st , at , st+1 ) , after the step t while in Schaul et al . ( 2015a ) , the goals are simply chosen randomly , as updating for all becomes computationally difficult , requiring two forward passes through the network for every goal update . 2.2 THE PROPOSED AGENT . The core concept of the proposed agent called Q-map is to use a convolutional auto-encoder-like neural network to simultaneously represent the GVFs of all possible on-screen locations for all possible actions from the raw screen pixels . This allows to efficiently share weights and to identify correlations between visual patterns such as walls , enemies or ladders and patterns in the distance towards on-screen locations . This means , for example , that learning how to navigate ladders can be transferred to other ladders later in a game . While a stack of greyscale frames are given in input , a stack of 2D Q-frames are produced in output where a single voxel at column x , row y and depth a represents the expected distance to the on-screen coordinates ( x , y ) given action a . A great advantage of the Q-map architecture is that it allows one to query the estimated distance towards all goals in a single pass through the network and to very efficiently create a learning target from these estimates with just an additional pass using a 3D version of the Q-learning algorithm . Rewards are matrices comprising of zeros with only the value at the next coordinates set at 1 and Q-values are the 3D tensors consisting of a stack of 2D Q-frames with the maximization of Q-values performed on the depth axis and no bootstrapping at the new location . Because the Q-values are bounded to [ 0 , 1 ] , clipping of the estimated Q-frames at the next state can be performed to speedup and stabilize the learning . Creating the target for an update thus consists of five steps : 1. perform a forward pass through the network with the next observation 2. clip and discount the values 3. maximize through the depth axis 4. replace the value at the next coordinate with 1 or in the case of a terminal state , use a zero matrix with the 1 at the new location 5. offset the frame by the required number of pixels if the observations involve a sliding window that moved during the transition . It is worth noting that a similar convolutional structure was used by Jaderberg et al . ( 2016 ) to represent Q-values for the pixel-control auxiliary task of the UNREAL agent . However the architecture was not considered for a goal-oriented framework , with a reward function based on the change in pixel values , without terminal transition at every step . Q-maps are particularly suited for environments in which it is possible to locate the agent ’ s position in screen coordinates , which could either be provided by the environment ( e.g . from the RAM in video games ) , or a classifier trained to localize the agent on the screen . While these coordinates are used during the training to create the targets for the Q-learning , it does not however preclude one from only using raw frames as input for the Q-map agent . In some games , such as Super Mario Bros. ( used later in the experiments ) , the screen scrolls while the player ’ s avatar moves , and thus only a portion of an entire level is shown at once . In the proposed Q-map implementation , we chose to use the coordinates available on the screen as possible goals and not coordinates over an entire level , thus the map is local to the area around the agent . Finally , while the distance to the goals could more directly be represented by the expected number of steps , the decay factor γ forces the values to be bounded to [ 0 , 1 ] , allowing the value of unreachable coordinates to naturally decay to 0 and the coordinates one-step away to have a value of 1 . 2.3 EXPERIMENTS . In order to measure the accuracy of the Q-map ’ s representation of the minimum number of steps towards goals and to test its capacity to generalize to a variety of states , we created a simple grid world environment with three different arrangement of obstacles . The environment consists of a simple pixel that can be moved in cardinal directions in a 28× 400 terrain with surrounding walls and non-traversable obstacles of various shapes . When an action would result in colliding with a wall or obstacle , the pixel is kept fixed . The observations consist of a a stack of 28 × 32 RGB frames corresponding to the view in a sliding window that moves horizontally with the pixel to keep it in the central column . Walls and obstacles are represented in blue while the pixel is represented in yellow . The first and second rooms share the same obstacle shapes with different locations while the third room consists of drastically different shapes of obstacles . The Q-map is trained with transitions generated by randomly starting from any free locations in the first room and taking a random action . As a baseline , we used a multi-layer perceptron ( MLP ) consisting of three fully-connected layers with hidden layers of 1024 units and layer normalization ( Ba et al. , 2016 ) and two output branches consisting of one fully-connected layer for each . The two outputs are aggregated in accordance to the dueling network architecture to represent the final Q-values ( Wang et al. , 2015 ) , reshaped to 28 × 28 × 4 to create the Q-frames . The proposed convolutional autoencoder-like architecture consists of three layers of convolutions , with dimensions 32 , 32 and 64 , kernel sizes 8 , 6 and 4 , and strides 2 , 2 , and 1 respectively . The convolutions are followed by two fully-connected layers with 1024 hidden units and layer normalization , followed by two dueling branches consisting of three transposed convolutions ( also known as deconvolutions ) , with dimensions 64 , 32 and 32 , kernel sizes 4 , 6 and 4 , and strides 1 , 2 , 2 respectively . Both networks use exponential linear activation functions . To evaluate the generalization of the Q-frames we limit the training experience of the agent only to the first room , while evaluating the output against a generated ground-truth Q-frames , generated with path-finding , on all three rooms separately . As can be seen in Figure 2 the Q-frames generated by the convolutional network are almost identical to the ground truth . The agent generalizes extremely well on the second room , which consists of familiar shapes in different arrangements . To our own surprise , the Q-map was able to handle unfamiliar shapes used in the third room to a good degree . Overall , the convolutional network has demonstrated significantly faster learning and better generalization properties when compared to the MLP baseline , validating the hypothesis that the proposed convolutional architecture is well suited for the studied scenario . While the gird world environment is useful for the evaluation of the Q-map performance versus the ground-truth , it does not offer a demonstration of the true capacity of the Q-map agent to handle much more complex visual input . We thus trained the Q-map on Montezuma ’ s Revenge and Super Mario All-Stars games , with examples of the learned Q-frames shown in Figures 4 and 5 .

Summary: Authors propose to overcome the sparse reward problem using an exploration strategy that incentivizes the agent to visit different parts of the game screen. This is done by building Q-maps, a 3D tensor that measures the value of the agent's current state (defined as the position of the agent) and action in reaching other (x, y) locations in the map. Each 2D slice of the Q-map measures the value at different (x, y) locations for one action. Such 2D slices (i.e. channels) are stacked together to form the Q-map. Taking the max across the channels, thus, provides the Q-value for the optimal action.