The Laplacian in RL: Learning Representations with Efficient Approximations

The smallest eigenvectors of the graph Laplacian are well-known to provide a succinct representation of the geometry of a weighted graph . In reinforcement learning ( RL ) , where the weighted graph may be interpreted as the state transition process induced by a behavior policy acting on the environment , approximating the eigenvectors of the Laplacian provides a promising approach to state representation learning . However , existing methods for performing this approximation are ill-suited in general RL settings for two main reasons : First , they are computationally expensive , often requiring operations on large matrices . Second , these methods lack adequate justification beyond simple , tabular , finite-state settings . In this paper , we present a fully general and scalable method for approximating the eigenvectors of the Laplacian in a model-free RL context . We systematically evaluate our approach and empirically show that it generalizes beyond the tabular , finite-state setting . Even in tabular , finite-state settings , its ability to approximate the eigenvectors outperforms previous proposals . Finally , we show the potential benefits of using a Laplacian representation learned using our method in goalachieving RL tasks , providing evidence that our technique can be used to significantly improve the performance of an RL agent . 1 INTRODUCTION . The performance of machine learning methods generally depends on the choice of data representation ( Bengio et al. , 2013 ) . In reinforcement learning ( RL ) , the choice of state representation may affect generalization ( Rafols et al. , 2005 ) , exploration ( Tang et al. , 2017 ; Pathak et al. , 2017 ) , and speed of learning ( Dubey et al. , 2018 ) . As a motivating example , consider goal-achieving tasks , a class of RL tasks which has recently received significant attention ( Andrychowicz et al. , 2017 ; Pong et al. , 2018 ) . In such tasks , the agent ’ s task is to achieve a certain configuration in state space ; e.g . in Figure 1 the environment is a two-room gridworld and the agent ’ s task is to reach the red cell . A natural reward choice is the negative Euclidean ( L2 ) distance from the goal ( e.g. , as used in Nachum et al . ( 2018 ) ) . The ability of an RL agent to quickly and successfully solve the task is thus heavily dependent on the representation of the states used to compute the L2 distance . Computing the distance on one-hot ( i.e . tabular ) representations of the states ( equivalent to a sparse reward ) is most closely aligned with the task ’ s directive . However , such a representation can be disadvantageous for learning speed , as the agent receives the same reward signal for all non-goal cells . One may instead choose to compute the L2 distance on ( x , y ) representations of the grid cells . This allows the agent to receive a clear signal which encourages it to move to cells closer to the goal . Unfortunately , this representation is agnostic to the environment dynamics , and in cases where the agent ’ s movement is obstructed ( e.g . by a wall as in Figure 1 ) , this choice of reward is likely to cause premature convergence to sub-optimal policies unless sophisticated exploration strategies are used . The ideal reward structure would be defined on state representations whose distances roughly correspond to the ability of the agent to reach one state from another . Although there are many suitable such representations , in this paper , we focus on a specific approach based on the graph Laplacian , which is notable for this and several other desirable properties . ∗Work performed while an intern at Google Brain . For a symmetric weighted graph , the Laplacian is a symmetric matrix with a row and column for each vertex . The d smallest eigenvectors of the Laplacian provide an embedding of each vertex in Rd which has been found to be especially useful in a variety of applications , such as graph visualization ( Koren , 2003 ) , clustering ( Ng et al. , 2002 ) , and more ( Chung & Graham , 1997 ) . Naturally , the use of the Laplacian in RL has also attracted attention . In an RL setting , the vertices of the graph are given by the states of the environment . For a specific behavior policy , edges between states are weighted by the probability of transitioning from one state to the other ( and vice-versa ) . Several previous works have proposed that approximating the eigenvectors of the graph Laplacian can be useful in RL . For example , Mahadevan ( 2005 ) shows that using the eigenvectors as basis functions can accelerate learning with policy iteration . Machado et al . ( 2017a ; b ) show that the eigenvectors can be used to construct options with exploratory behavior . The Laplacian eigenvectors are also a natural solution to the aforementioned reward-shaping problem . If we use a uniformly random behavior policy , the Laplacian state representations will be appropriately aware of the walls present in the gridworld and will induce an L2 distance as shown in Figure 1 ( right ) . This choice of representation accurately reflects the geometry of the problem , not only providing a strong learning signal at every state , but also avoiding spurious local optima . While the potential benefits of using Laplacian-based representations in RL are clear , current techniques for approximating or learning the representations are ill-suited for model-free RL . For one , current methods mostly require an eigendecomposition of a matrix . When this matrix is the actual Laplacian ( Mahadevan , 2005 ) , the eigendecomposition can easily become prohibitively expensive . Even for methods which perform the eigendecomposition on a reduced matrix ( Machado et al. , 2017a ; b ) , the eigendecomposition step may be computationally expensive , and furthermore precludes the applicability of the method to stochastic or online settings , which are common in RL . Perhaps more crucially , the justification for many of these methods is made in the tabular setting . The applicability of these methods to more general settings is unclear . To resolve these limitations , we propose a computationally efficient approach to approximate the eigenvectors of the Laplacian with function approximation based on the spectral graph drawing objective , an objective whose optimum yields the desired eigenvector representations . We present the objective in a fully general RL setting and show how it may be stochastically optimized over minibatches of sampled experience . We empirically show that our method provides a better approximation to the Laplacian eigenvectors than previous proposals , especially when the raw representation is not tabular . We then apply our representation learning procedure to reward shaping in goal-achieving tasks , and show that our approach outperforms both sparse rewards and rewards based on L2 distance in the raw feature space . Results are shown under a set of gridworld maze environments and difficult continuous control navigation environments . 2 BACKGROUND . We present the eigendecomposition framework in terms of general Hilbert spaces . By working with Hilbert spaces , we provide a unified treatment of the Laplacian and our method for approximating its eigenvectors ( Cayley , 1858 ) – eigenfunctions in Hilbert spaces ( Riesz , 1910 ) – regardless of the underlying space ( discrete or continuous ) . To simplify the exposition , the reader may substitute the following simplified definitions : • The state space S is a finite enumerated set { 1 , . . . , |S| } . • The probability measure ρ is a probability distribution over S. • The Hilbert spaceH is R|S| , for which elements f ∈ H are |S| dimensional vectors repre- senting functions f : S → R. • The inner product 〈f , g〉H of two elements f , g ∈ H is a weighted dot product of the corresponding vectors , with weighting given by ρ ; i.e . 〈f , g〉H = ∑|S| u=1 f ( u ) g ( u ) ρ ( u ) . • A linear operator is a mapping A : H → H corresponding to a weighted matrix multiplication ; i.e . Af ( u ) = ∑|S| v=1 f ( v ) A ( u , v ) ρ ( v ) . • A self-adjoint linear operator A is one for which 〈f , Ag〉H = 〈Af , g〉H for all f , g ∈ H. This corresponds to A being a symmetric matrix . 2.1 A SPACE AND A MEASURE . We now present the more general form of these definitions . Let S be a set , Σ be a σ-algebra , and ρ be a measure such that ( S , Σ , ρ ) constitutes a measure space . Consider the set of square-integrable real-valued functions L2 ( S , Σ , ρ ) = { f : S → R s.t . ∫ S |f ( u ) |2 dρ ( u ) < ∞ } . When associated with the inner-product , 〈f , g〉H = ∫ S f ( u ) g ( u ) dρ ( u ) , this set of functions forms a complete inner product Hilbert space ( Hilbert , 1906 ; Riesz , 1910 ) . The inner product gives rise to a notion of orthogonality : Functions f , g are orthogonal if 〈f , g〉H = 0 . It also induces a norm on the space : ||f ||2 = 〈f , f〉H . We denoteH = L2 ( S , Σ , ρ ) and additionally restrict ρ to be a probability measure , i.e . ∫ S 1 dρ ( u ) = 1 . 2.2 THE LAPLACIAN . To construct the graph Laplacian in this general setting , we consider linear operators D which are Hilbert-Schmidt integral operators ( Bump , 1998 ) , expressable as , Df ( u ) = ∫ S f ( v ) D ( u , v ) dρ ( v ) , where with a slight abuse of notation we also use D : S × S 7→ R+ to denote the kernel function . We assume that ( i ) the kernel function D satisfies D ( u , v ) = D ( v , u ) for all u , v ∈ S so that the operator D is self-adjoint ; ( ii ) for each u ∈ S , D ( u , v ) is the Radon-Nikodym derivative ( density function ) from some probability measure to ρ , i.e . ∫ S D ( u , v ) dρ ( v ) = 1 for all u . With these assumptions , D is a compact , self-adjoint linear operator , and hence many of the spectral properties associated with standard symmetric matrices extend to D. The Laplacian L of D is defined as the linear operator onH given by , Lf ( u ) = f ( u ) − ∫ S f ( v ) D ( u , v ) dρ ( v ) = f ( u ) −Df ( u ) . ( 1 ) The Laplacian may also be written as the linear operator I − D , where I is the identity operator . Any eigenfunction with associated eigenvalue λ of the Laplacian is an eigenfunction with eigenvalue 1− λ for D , and vice-versa . Our goal is to find the first d eigenfunctions f1 , ... , fd associated with the smallest d eigenvalues of L ( subject to rotation of the basis ) .1 The mapping φ : S 7→ Rd defined by φ ( u ) = [ f1 ( u ) , ... , fd ( u ) ] then defines an embedding or representation of the space S .

Summary: This paper proposes a method to learn a state representation for RL using the Laplacian. The proposed method aims to generalize previous work, which has only been shown in finite state spaces, to continuous and large state spaces. It goes to approximate the eigenvectors of the Laplacian which is constructed using a uniformly random policy to collect training data. One use-case of the learnt state representation is for reward-shaping that is said to accelerate the training of standard goal-driven RL algorithms.