The Laplacian in RL: Learning Representations with Efficient Approximations

The smallest eigenvectors of the graph Laplacian are well-known to provide a succinct representation of the geometry of a weighted graph . In reinforcement learning ( RL ) , where the weighted graph may be interpreted as the state transition process induced by a behavior policy acting on the environment , approximating the eigenvectors of the Laplacian provides a promising approach to state representation learning . However , existing methods for performing this approximation are ill-suited in general RL settings for two main reasons : First , they are computationally expensive , often requiring operations on large matrices . Second , these methods lack adequate justification beyond simple , tabular , finite-state settings . In this paper , we present a fully general and scalable method for approximating the eigenvectors of the Laplacian in a model-free RL context . We systematically evaluate our approach and empirically show that it generalizes beyond the tabular , finite-state setting . Even in tabular , finite-state settings , its ability to approximate the eigenvectors outperforms previous proposals . Finally , we show the potential benefits of using a Laplacian representation learned using our method in goalachieving RL tasks , providing evidence that our technique can be used to significantly improve the performance of an RL agent . 1 INTRODUCTION . The performance of machine learning methods generally depends on the choice of data representation ( Bengio et al. , 2013 ) . In reinforcement learning ( RL ) , the choice of state representation may affect generalization ( Rafols et al. , 2005 ) , exploration ( Tang et al. , 2017 ; Pathak et al. , 2017 ) , and speed of learning ( Dubey et al. , 2018 ) . As a motivating example , consider goal-achieving tasks , a class of RL tasks which has recently received significant attention ( Andrychowicz et al. , 2017 ; Pong et al. , 2018 ) . In such tasks , the agent ’ s task is to achieve a certain configuration in state space ; e.g . in Figure 1 the environment is a two-room gridworld and the agent ’ s task is to reach the red cell . A natural reward choice is the negative Euclidean ( L2 ) distance from the goal ( e.g. , as used in Nachum et al . ( 2018 ) ) . The ability of an RL agent to quickly and successfully solve the task is thus heavily dependent on the representation of the states used to compute the L2 distance . Computing the distance on one-hot ( i.e . tabular ) representations of the states ( equivalent to a sparse reward ) is most closely aligned with the task ’ s directive . However , such a representation can be disadvantageous for learning speed , as the agent receives the same reward signal for all non-goal cells . One may instead choose to compute the L2 distance on ( x , y ) representations of the grid cells . This allows the agent to receive a clear signal which encourages it to move to cells closer to the goal . Unfortunately , this representation is agnostic to the environment dynamics , and in cases where the agent ’ s movement is obstructed ( e.g . by a wall as in Figure 1 ) , this choice of reward is likely to cause premature convergence to sub-optimal policies unless sophisticated exploration strategies are used . The ideal reward structure would be defined on state representations whose distances roughly correspond to the ability of the agent to reach one state from another . Although there are many suitable such representations , in this paper , we focus on a specific approach based on the graph Laplacian , which is notable for this and several other desirable properties . ∗Work performed while an intern at Google Brain . For a symmetric weighted graph , the Laplacian is a symmetric matrix with a row and column for each vertex . The d smallest eigenvectors of the Laplacian provide an embedding of each vertex in Rd which has been found to be especially useful in a variety of applications , such as graph visualization ( Koren , 2003 ) , clustering ( Ng et al. , 2002 ) , and more ( Chung & Graham , 1997 ) . Naturally , the use of the Laplacian in RL has also attracted attention . In an RL setting , the vertices of the graph are given by the states of the environment . For a specific behavior policy , edges between states are weighted by the probability of transitioning from one state to the other ( and vice-versa ) . Several previous works have proposed that approximating the eigenvectors of the graph Laplacian can be useful in RL . For example , Mahadevan ( 2005 ) shows that using the eigenvectors as basis functions can accelerate learning with policy iteration . Machado et al . ( 2017a ; b ) show that the eigenvectors can be used to construct options with exploratory behavior . The Laplacian eigenvectors are also a natural solution to the aforementioned reward-shaping problem . If we use a uniformly random behavior policy , the Laplacian state representations will be appropriately aware of the walls present in the gridworld and will induce an L2 distance as shown in Figure 1 ( right ) . This choice of representation accurately reflects the geometry of the problem , not only providing a strong learning signal at every state , but also avoiding spurious local optima . While the potential benefits of using Laplacian-based representations in RL are clear , current techniques for approximating or learning the representations are ill-suited for model-free RL . For one , current methods mostly require an eigendecomposition of a matrix . When this matrix is the actual Laplacian ( Mahadevan , 2005 ) , the eigendecomposition can easily become prohibitively expensive . Even for methods which perform the eigendecomposition on a reduced matrix ( Machado et al. , 2017a ; b ) , the eigendecomposition step may be computationally expensive , and furthermore precludes the applicability of the method to stochastic or online settings , which are common in RL . Perhaps more crucially , the justification for many of these methods is made in the tabular setting . The applicability of these methods to more general settings is unclear . To resolve these limitations , we propose a computationally efficient approach to approximate the eigenvectors of the Laplacian with function approximation based on the spectral graph drawing objective , an objective whose optimum yields the desired eigenvector representations . We present the objective in a fully general RL setting and show how it may be stochastically optimized over minibatches of sampled experience . We empirically show that our method provides a better approximation to the Laplacian eigenvectors than previous proposals , especially when the raw representation is not tabular . We then apply our representation learning procedure to reward shaping in goal-achieving tasks , and show that our approach outperforms both sparse rewards and rewards based on L2 distance in the raw feature space . Results are shown under a set of gridworld maze environments and difficult continuous control navigation environments . 2 BACKGROUND . We present the eigendecomposition framework in terms of general Hilbert spaces . By working with Hilbert spaces , we provide a unified treatment of the Laplacian and our method for approximating its eigenvectors ( Cayley , 1858 ) – eigenfunctions in Hilbert spaces ( Riesz , 1910 ) – regardless of the underlying space ( discrete or continuous ) . To simplify the exposition , the reader may substitute the following simplified definitions : • The state space S is a finite enumerated set { 1 , . . . , |S| } . • The probability measure ρ is a probability distribution over S. • The Hilbert spaceH is R|S| , for which elements f ∈ H are |S| dimensional vectors repre- senting functions f : S → R. • The inner product 〈f , g〉H of two elements f , g ∈ H is a weighted dot product of the corresponding vectors , with weighting given by ρ ; i.e . 〈f , g〉H = ∑|S| u=1 f ( u ) g ( u ) ρ ( u ) . • A linear operator is a mapping A : H → H corresponding to a weighted matrix multiplication ; i.e . Af ( u ) = ∑|S| v=1 f ( v ) A ( u , v ) ρ ( v ) . • A self-adjoint linear operator A is one for which 〈f , Ag〉H = 〈Af , g〉H for all f , g ∈ H. This corresponds to A being a symmetric matrix . 2.1 A SPACE AND A MEASURE . We now present the more general form of these definitions . Let S be a set , Σ be a σ-algebra , and ρ be a measure such that ( S , Σ , ρ ) constitutes a measure space . Consider the set of square-integrable real-valued functions L2 ( S , Σ , ρ ) = { f : S → R s.t . ∫ S |f ( u ) |2 dρ ( u ) < ∞ } . When associated with the inner-product , 〈f , g〉H = ∫ S f ( u ) g ( u ) dρ ( u ) , this set of functions forms a complete inner product Hilbert space ( Hilbert , 1906 ; Riesz , 1910 ) . The inner product gives rise to a notion of orthogonality : Functions f , g are orthogonal if 〈f , g〉H = 0 . It also induces a norm on the space : ||f ||2 = 〈f , f〉H . We denoteH = L2 ( S , Σ , ρ ) and additionally restrict ρ to be a probability measure , i.e . ∫ S 1 dρ ( u ) = 1 . 2.2 THE LAPLACIAN . To construct the graph Laplacian in this general setting , we consider linear operators D which are Hilbert-Schmidt integral operators ( Bump , 1998 ) , expressable as , Df ( u ) = ∫ S f ( v ) D ( u , v ) dρ ( v ) , where with a slight abuse of notation we also use D : S × S 7→ R+ to denote the kernel function . We assume that ( i ) the kernel function D satisfies D ( u , v ) = D ( v , u ) for all u , v ∈ S so that the operator D is self-adjoint ; ( ii ) for each u ∈ S , D ( u , v ) is the Radon-Nikodym derivative ( density function ) from some probability measure to ρ , i.e . ∫ S D ( u , v ) dρ ( v ) = 1 for all u . With these assumptions , D is a compact , self-adjoint linear operator , and hence many of the spectral properties associated with standard symmetric matrices extend to D. The Laplacian L of D is defined as the linear operator onH given by , Lf ( u ) = f ( u ) − ∫ S f ( v ) D ( u , v ) dρ ( v ) = f ( u ) −Df ( u ) . ( 1 ) The Laplacian may also be written as the linear operator I − D , where I is the identity operator . Any eigenfunction with associated eigenvalue λ of the Laplacian is an eigenfunction with eigenvalue 1− λ for D , and vice-versa . Our goal is to find the first d eigenfunctions f1 , ... , fd associated with the smallest d eigenvalues of L ( subject to rotation of the basis ) .1 The mapping φ : S 7→ Rd defined by φ ( u ) = [ f1 ( u ) , ... , fd ( u ) ] then defines an embedding or representation of the space S .

Summary: The authors propose a Laplacian in the context of reinforcement learning, together with learning the representations. Overall the authors make a nice contribution. The insight of defining rho to be the stationary distribution of the Markov chain P^pi and connecting this to eq (1) is interesting. Also the definition of the reward function on p.7 in terms of the distance between phi(s_{t+1}) and phi(z_g) looks original. The method is also well illustrated and compared with other methods, showing the efficiency of the proposed method.