Policy Generalization In Capacity-Limited Reinforcement Learning

1 INTRODUCTION . Generalization of learning is fundamental to intelligent behavior . For example , if a bird eats a poisonous butterfly , it must learn to avoid preying on that species again , even when no two butterflies look exactly alike . Similarly , in the game of chess a player must generalize intelligently from past experience as the same board position may never be encountered twice . The field of reinforcement learning ( Sutton & Barto , 2018 ) provides a principled framework for the problem of learning , via trial and error , the long-term consequences of actions . However , by itself , RL lacks a formal theory of the generalization of learning . In practice , generalization is achieved through the combination of reinforcement learning algorithms with function approximation techniques , such as deep neural networks ( DNNs ) used to represent states , policies , and/or value functions . A separate thread of research has focused on learning hierarchical representations of policies , or learning extended action sequences ( ‘ options ’ ) that may enable stronger generalization and faster learning in complex tasks ( Solway et al. , 2014 ; Bacon et al. , 2017 ) . While these approaches have yielded notable success in numerous large-scale learning domains ( e.g. , Tesauro , 1995 ; Mnih et al. , 2015 ; Silver et al. , 2016 ) , there is no extant body of theory that fully explains how and why such approaches can work well , and when they may fail in practice . In the current paper , we instead seek to develop a principled mathematical basis for understanding generalization in the RL setting , building on theoretical cognitive science research on generalization in biological intelligence . In particular , within cognitive science the well-known ‘ Universal Law of Generalization ’ ( Shepard , 1987 ) describes the strength of generalization from one stimulus to another , and has been demonstrated in multiple animal species and across a wide range of perceptual modalities . Recently , it has been shown that the Universal Law of Generalization emerges from any system that minimizes the expected cost of failing to discriminate between situations , subject to constraints on the ability to transmit or represent information ( Sims , 2018 ) . This theoretical result was obtained via the mathematical field of rate-distortion ( RD ) theory , which concerns the optimal solution to the problem of optimal , but ‘ lossy ’ data compression . According to this framework , biological organisms generalize because they must : they lack the information capacity to preserve all state or environmental distinctions , and the resulting pattern of generalization depends , in a rational manner , on the expected cost of failing to discriminate . Hence , RL and RD theory are complementary in their strengths and limitations . RL provides a theory of learning the long-term utility of actions via experience , but lacks a theory of generalization of learning . RD theory provides an elegant mathematical account of rational generalization in terms of information processing constraints , but assumes the long-term costs of equivocation between states or actions are known . While many readers may be familiar with RL or RD in isolation , few are likely to have more than passing familiarity with both . Hence , an important goal of the paper is a an accessible introduction of rate-distortion theory in the context of RL . Combining these two ideas , we consider the problem of capacity-limited reinforcement learning , in which the agent ’ s learned mapping from states to actions is treated as a capacity-limited information channel that lacks the ability to represent behavioral policies with perfect fidelity . Such agents must learn , at best , a lossy compressed representation of an optimal behavioral policy . RD theory provides the theoretical bounds on what is achievable in this setting . In pursuit of this idea , we develop a new online reinforcement learning algorithm , Capacity-Limited Actor-Critic ( CL-AC ) that learns a policy that optimizes a tradeoff between the expected utility of behavior , and the information rate of the agent ’ s policy representation . Exploring the behavior of this algorithm in a 2D gridworld environment , we show that the introduction of capacity limits imparts two useful forms of generalization to RL agents . First , learned policies generalize across states during the course of learning in a principled manner . Consequently , at high information rates ( high channel capacity ) , CL-AC achieves faster learning and higher accumulated reward compared to a standard tabular actor-critic ( AC ) algorithm . Second , we demonstrate that agents with capacitylimited policy representations exhibit superior generalization to modified environments , compared to policies learned by agents with unlimited information processing resources . Our work is an instance of the general framework of computational rationality ( Gershman et al. , 2015 ) , an emerging paradigm for understanding biological and artificial intelligence as efficient computing given information processing constraints . While information-theoretic channel capacity has long been understood as a relevant constraint on sensory coding in biological systems ( Attneave , 1954 ; Barlow , 1961 ) , as well as a model of economic decision-making in boundedly-rational humans ( Sims , 2003 ) it has only more recently been explored as a constraint in RL ( Grau-Moya et al. , 2016 ; Ortega & Braun , 2011 ; Still & Precup , 2012 ; Rubin et al. , 2012 ; Botvinick et al. , 2015 ) . 2 REINFORCEMENT LEARNING AND RATE-DISTORTION THEORY . In this section we provide brief overviews of both computational RL and RD theory . RL refers to a large family of different algorithms that address the problem of learning through experience how to select actions so as to maximize long-term expected reward . Here we briefly summarize the aspects relevant to the current work ; a thorough introduction is given in ( Sutton & Barto , 2018 ) . The basic model for the interaction of an agent ( artificial or biological ) with a stochastic environment is a Markov decision problem ( MDP ) , comprising a set of states ( S ) that describe an environment , a set of candidate actions ( A ) available to the agent in each state , and a dynamics function p ( s′ , r | s , a ) specifying the joint distribution over subsequent state s′ and reward r , given a starting state s and taking action a . In the RL setting , the agent ’ s goal is to learn ( or approximate ) an optimal policy function π ( a | s ) . In the present work , it will be useful to consider this policy function as an information channel that takes as input a current state , and produces as output a ( stochastic ) action to be followed in that state ( e.g. , S → A ) . A fundamental construct in RL is an optimal value function , V ? ( s ) , that indicates the expected long-term reward associated with being in a particular state s , and subsequently following an optimal behavioral policy . As shown below , an optimal value function is central to defining optimal performance for a capacity-limited policy channel . For a physical agent with finite computational resources , it may be necessary or desirable to limit the amount of information stored about the policy or environment . Under such circumstances , what constitutes optimal performance ? This question has an answer in the field of RD theory ( Berger , 1971 ) , which concerns the problem of optimal , but ‘ lossy ’ data compression . Within RD theory , optimal performance for a communication channel is defined according to a given cost or loss function , L ( x , y ) that quantifies the cost of a signal x ∈ X being transmitted as the value y ∈ Y . More formally , the input to the channel is a sample from a source distribution p ( x ) , and the channel itself can be modeled as a conditional probability distribution p ( y | x ) . The goal for the channel is to minimize the cost of communication error specified by the loss function , L ( x , y ) . For a given source and channel distribution , information theory provides a measure of the amount of information communicated ( on average ) by the channel in terms of its mutual information : I ( X , Y ) = ∑ y ∑ x p ( y | x ) p ( x ) log ( p ( y | x ) p ( x ) p ( y ) p ( x ) ) . ( 1 ) Any physical communication channel is necessarily limited to transmitting information at a finite rate , and hence I ( X , Y ) ≤ C for some finite value of C. Consequently , an optimally efficient communication channel is one that minimizes channel distortion ( expected cost ) subject to a constraint on information rate : Goal : Minimize E [ L ( x , y ) ] w.r.t . p ( y | x ) subject to I ( X , Y ) ≤ C , ( 2 ) where the distortion E [ L ( x , y ) ] = ∑ x , y L ( x , y ) p ( y | x ) p ( x ) . A major topic within RD theory is finding efficient means of solving this constrained optimization problem . Blahut ( 1972 ) developed an efficient iterative algorithm for the case of channels with discrete input and output alphabets . This algorithm amounts to a minimization of the Lagrangian I ( X , Y ) + β E [ L ( x , y ) ] with respect to p ( y | x ) . The parameter β > 0 controls the tradeoff between the distortion and information rate of the channel , where higher values of β correspond to channels with higher information rate and lower channel distortion . Rather then being concerned with the mapping of abstract communication signals x and y , we are instead interested in a channel that maps from states to actions , S → A . In this case , a critical component in the application of RD theory to RL is the specification of an appropriate loss function for a capacity limited policy channel . To that end , we introduce the Bellman loss function , L ? ( s , a ) , which defines the expected loss to an agent , relative to the optimal policy , associated with a policy channel producing as output an action a in a given state s : L ? ( s , a ) = V ? ( s ) − E [ r ( s , a ) + γV ∗ ( s′ ) ] = V ? ( s ) − ∑ s′ , r p ( s′ , r | s , a ) ( r + γV ? ( s′ ) ) . ( 3 ) Note that this is merely the difference in the optimal value for the starting state s , and the value of the immediate reinforcement plus the ( possibly discounted ) value of the resulting state s′ , associated with the channel transmitting action a . Hence for all states and actions , L ? ( s , a ) ≥ 0 , with equality achieved for the optimal policy . Intuitively , the Bellman loss function states that when there is little or no difference between the cost of actions in a given state , there is no need to precisely represent a policy that distinguishes between them . With a normative loss function defined , Algorithm 1 ( Appendix ) gives the complete Blahut algorithm adapted for computing an optimal , but capacity-limited policy channel . Note that this algorithm takes as input an optimal value function V ? for the MDP . In Section 3.2 we consider the problem of simultaneously learning an approximate value function , and using it to update an approximate Bellman loss function . However , we first assume a known optimal value function in order to examine in isolation the consequences of adopting a capacity-limited policy channel .

Summary: The authors identify a synergy between the rate distortion (RD) and reinforcement learning (RL) literature. RD work shows how to optimise resources when capacity is limited and the authors transfer this idea to RL and posit a novel algorithm based on the Actor critic algorithm. In experiments this is shown to learn more quickly and transfer between similar tasks more easiliy than the conventional AC algorithm.