CAPACITY-LIMITED REINFORCEMENT LEARNING: APPLICATIONS IN DEEP ACTOR-CRITIC METHODS FOR CONTINUOUS CONTROL
Authors: Tyler James Malloy
Matthew Riemer
Miao Liu
Tim Klinger
Gerald Tesauro
Chris R. Sims
Venue:  (2020)
Keywords: Reinforcement Learning
Generalization
Information Theory
Rate-Distortion Theory

Abstract: Biological and artificial agents must learn to act optimally in spite of a limited capacity for processing, storing, and attending to information. We formalize this type of bounded rationality in terms of an information-theoretic constraint on the complexity of policies that agents seek to learn. We present the Capacity-Limited Reinforcement Learning (CLRL) objective which defines an optimal policy subject to an information capacity constraint. This objective is optimized by drawing from methods used in rate distortion theory and information theory, and applied to the reinforcement learning setting. Using this objective we implement a novel Capacity-Limited Actor-Critic (CLAC) algorithm and situate it within a broader family of RL algorithms such as the Soft Actor Critic (SAC) and discuss their similarities and differences. Our experiments show that compared to alternative approaches, CLAC offers improvements in generalization between training and modified test environments. This is achieved in the CLAC model while displaying high sample efficiency and minimal requirements for hyper-parameter tuning.

CAPACITY-LIMITED REINFORCEMENT LEARNING: APPLICATIONS IN DEEP ACTOR-CRITIC METHODS FOR CONTINUOUS CONTROL
Anonymous authors Paper under double-biee blind review
ABSTRACT
Biological and artificial agents must learn act optimally in spite of limited capacity for processing, storing, and attending to information. We formalize this type of bounded rationality in terms of an cntommtiion-thoreeeee constraint on the complexity of policies that agents seek to learn. We present the Capacity-Limitee Reinforcement Learning (CLRL.) objective which defines an optimal policy sub- ject to an information capacity constraint. This objective is optimized by drawing from methods used in rate distortion theory and information theory, and applied to the reinforcement learning setting. Using this objective we implement novel Capacity- Limettt Actor-Critic (CLAC) algorithm and situate within a broader family of RL algorithms such as the Soft Actor Critic (SAC) and discuss their similarities and differences. Our experiments show that compared to alternative approaches, CLAC offers improvements in generalization between training and modified test environments. This is achieved in the CLAC model while display- ing high sample efficiency and minimal requirements for syper-parameter tuning.
INTRODUCTION
The success of Deep RL has led to significant milestones in Al research (e.g Tesauro 1994). Mnih eet al. (2013), Silver et al. (2016). However, at the same time, some have called into question the extent which these approaches are able demonstrate nontrivial generalization of learning, as compared to simply memorizing state-action sequences (Packer et al. (2018), Cobbe et al. (2018) ) In this paper, we look to improve sample efficient generalization in RL by making a connection to the idea of capacity limits in the field of Information Theory. As we will demonstrate, this idea has strong theoretical ties to recently popular approaches for Maximum Entropy Reinforcement Learn- ing (MERL. (Haarnoja et al., (0188) and KL-regularized RI. (Tirumala et al.) 2019). Furthermore, our new framework for capacity limited RL motivates new algorithm called Capacity Limited Actor Critic (CLAC) that displays superior out of distribution generalization properties without sac- rificing sample efficiency or online learning performance.
RL generally consists of Markov Decision Process (MDP) defined by state transition function P(s"|s, a) that determines the probability of the next state, and a reward function as).a). In this setting the primary objective is typically to learn policy, which is defined probability distribution over actions conditioned on states, x (a]s). In fact, we would like to learn policy that optimizes the expected return over horizon T following the state distribution of the policy Pa: (TT) ((,,,,)).. EET r(s,, ).).
action a,
While MDPs define the setting for an idcalized learning agent, any physical communication system (biological or artificial) is necessarily limited to transmitting information finite rate. In informa- tion theory, the information rate of a channel with input x and output y is quantified by the so-called mutual information, I(X, Y). Hence, for any physical information processing system, I(X, Y) < for some finite value of C. At the same time, agents with limited information processing capacities should seek to produce behavior that maximizes expected utility. Hence, for physical agent, opti- mal behavior is the result of solving constrained optimization problem (maximizing the utility of behavior, subject to constraints on available information capacity). Recent work in cognitive science has shown that this constrained optimization perspective information processing can account for

---

In the present work, we apply this perspective to the reinforcement learning framework. In particular, we consider an agent's policy as an informa- tion channel, that maps from the current state of the environment, to probability distribution over actions, and define optimal behavior subject to a constraint on this information capacity of this chan- nel. More formally, an optimally efficient communication channel is one that minimizes expected loss in utility, [x, y)]. to this constraint:
generalization in biological perception (Sims 2018
Goal: Minimize [Z, y)1 w.r.t p(zly). subject to I(X, () sC
This objective is well-studied within the field of Rate Distortion theory (Tretiak] 1974). sub-field of information theory. In our application of RD theory onto RL, we consider the policy function (a|s) learned by an RI. agent be communication channel that maps from the state onto a probability of performing an action a. This allows us to define the optimality condition of an RL. agent with constraint on information representation as
T
Σ r(st, aa) s.t (anaaa)>)_[((((aaaaaa & C] At t=0
max T 3
(1)
Where [[m(s]))) is the mutual information of the policy function when taken to be the information channel mapping states onto actions. The introduction of this policy mutual information term in this constraint connects the objective of a RL agent onto the desire of control over the amount of information used to represent agent behaviour. There are multiple ways of calculating this value which will be discussed in the next section. This allows us to define C, the desired maximum channel capacity, and optimize performance in the environment in relation to this capacity. This optimization can be used to define learning objective that better reflects the reality of information constraints on physical agents, and as we will see, allow for better control over the trade off of immediate performance and generalization.
2
CAPACITY LIMITED RL (CLRL)
2.1
CAPACITY-LIMITED LEARNING OBJECTIVE
Insight from an information theoretic perspective inspires the goal of maximizing reward obtained subject to some constraints on policy complexity (as measured by its mutual information). In prac- tice, the way we impose limit on the amount of information that the agent uses to represent its policy is done by applying a penalty to the reward based on this value. This allows us to define a learning objective that regularizes the observed reward:
(Tx) = Σ Irs az) - -I((T((3))]
The key difference with the standard RL objective the added penalty to the reward observed based on the amount of information that would be required to represent the policy. Policies with higher mutual information values have a greater complexity, weighted value is used to discourage policies that would require high information capacity channel. Thus, this learning objective will directly encourage the development of policies that are simple (use low information represent) but have high utility. Additionally, if there are multiple policies that achieve the same performance, this objective will naturally favor the simplest among them. Higher values of B skew the learning objective to prefer policies with less required information.
an Eeformation-theoreee
sense, and this
2.2 CALCULATING MUTUAL INFORMATION
The mutual information in the CLRL learning objective can be defined in different ways depending on the application, and the features of the learning environment. Because CLRL describes broad learning objective that can be applied to different existing RL methods, these different methods may be best suited by different approximations of mutual information. For discrete state and action spaces in tabular learning conditions the space is small enough that it can be defined in terms of the probability mass functions as follows:
(s,
[(T(a|)))
Σ
ΣΣN
EE ((s.a) a)gggaaogg log
(2)
(a..a), ps((s)
a #ES

---

However, for more complicated problems, especially environments involving continuous state/action spaces, we will need a different approach. One possibility is to use the definition of mutual informa- tion in terms of the probability density functions for continuous distributions. However this method requires the integration over both the state and action spaces, as well as the approximation of both the marginal action Pa(a) and marginal state probabilities (s(s). An alternative that avoids these complications is to break up the policy mutual information into the components of its constituent entropies. This calculation of mutual information also avoids the approximation of both the marginal state distribution and marginal action distribution, with only one approximation required, being cal- culated by either of:
I(T(als)) = H()a(a)) -((((((s)) =s(m((s)) H((T(s)a)
(3)
2.3 CONNECTIONS TO MAXIMUM ENTROPY REINFORCEMENT LEARNING (MERL)
Maximum Entropy RL (Haarnoja al. 2018a is popular RL framework that also makes use of regularized learning objective to maximize
((T) (s (r) (a(())))))((((())))))))
T
Σ ())
+ aH(((a)))
This learning objective alters the traditional method by augmenting the reward maximization ob- jective with an additional weighted value based on the entropy of the policy (=)ss))) This encourages policies that are closer to the uniform distribution and thus more random, since these action distributions will have a higher entropy. The entropy coefficient a controls the weight bal- ancing the reward and entropy of the policy. As this value mcreases infinity, the model will learn to always prefer policies that are completely uniform. When this value is O, the model will only consider the reward returned by the environment and is equivalent to the traditional RL learning objective.
As there is close theoretical tie between the frameworks, we leverage the successful Maximum Entropy method Seft-Actor-Crttie (SAC) as the main point of comparison for our approach. SAC is an off-policy RL method that displays high sample efficiency and resilience to changes in hy- perparameters even in difficult continuous control robot simulation environments (Haarnoja al. 20186). To make the comparison easier with SAC, we utilize the top equation for the calculation of I(T(a|s) s)) in equation 3 This is because it utilizes the entropy of the policy H(T((as)), which is calculated existing MERL methods, instead of H(T (8 (a)). which represents the difficult to define entropy of the probability of being in state given that the agent has preformed action. One ben- efit to this approach is that any MERL method will naturally need to compute the value H(T(als)) and can be altered with relative ease to include the additional marginal action term, as done in this implementation. In the next section on the Deep CLAC algorithm. we apply this alteration to the existing Soft-Actor Critic method that uses the MERL objective to improve exploration. This is done to show that the Capacity-Linne Limited objective can be used to alter existing methods with relative case, especially ones that already compute the policy entropy H(T (s)s), such as MERL methods.
2.4 CONNECTIONS TO KL-REGULARIZED REINFORCEMENT LEARNING
The Kl.-Regularized expected reward objective maximizes
T
((T)
0=0
KL(a))=)) = log T(ce(s))
Tu(a, sst)
referring to the full history up to time t as x, [sp, .o.t. or]. This method is used in the hierarchical learning domain which separates broad high-level task focused decisions and specific

---

low-level task agnostic actions Tirumala et al. 2019). In KL-RL, this is done by training task specific policy [[ass) as well as a default policy (o(a))) and using the KL divergence between these two policies to regularize the reward observed by the agent. In this way the agent learns to generalize across tasks by increasing the reward observed when reusing learned behaviour across tasks. The default and task specific policies do not converge the same policy they have different level of access to state information, encouraging тo to learn a task independent policy. This difference in state information access can come in the form of additional latent state features available to the task specific policy
The structure of CLAC is not hierarchical and thus we do not compare the performance of CLAC to KL-RL method, however these two methods do share mathematical relation that mirrors the connection between KL.-RI. and SAC. As noted in Tirumala et al. (2019) the maximum entropy objective is special case of the Kl-rregularized objective where the default policy is taken to be uniform distribution. This intuitively it seeks to increase exploration by encouraging random behavior where possible. A similar relation can be made between KL-RL and the CLRL defined here, where the 'default policy' is taken to be the product of the marginal action and state distributions. Because of the motivation of discouraging high information capacity policies, a negative sign is applied to the regularization coefficient giving -a. This derivation relating KL-RL to CLRL is given by first taking the definition of the KI-RI. objective:
line with the motivation behind the MERL, objective, as
T
(TT)
Σ
B (ss (r(st, az) + (KK((a,(=z))
1=0
T
r(s), log π (apls)))
[(T)
E (spa
To To
10
We can write the denominator of this logarithm term generally function of state and action such as p(a, s). This allows us to define KI-RI. to be the case where Po (a, 8) = (a (s)). this same manner SAC be defined as the uniform distribution p(a,s) = 1/l| Finally, CLAC can be defined as p(a,s) s) = Ta (a)TT(()) the product of the marginal action and state probabilities.
J(T)
Σ
T
(r (((e).a)) - [1108 log (a(a)))T.
π (acess))
1=
( = Σ
Ba ( -I((T(a(s:))
1=0
As with the relation between SAC and KL-RL, this relation in line with the motivation of these learning objectives, because this product of marginal state and action functions can be though of as a generic inte--indeeendent policy, and we encourage policies that are more similar to this generic policy because they require less information to represent. A clear conceptual example of this is the fact that the information capacity required to represent a policy minimized by both completely uniform policy and one that preforms the same action in all states.
3 DEEP CAPACITY LIMITED ACTOR CRITIC LEARNING
There are many RL approaches that could be applied in optimizing the capacity- limited learning objective. Here, we implement the CLAC algorithm based on a baseline implementation (Hill al. 2018) of the Soft Actor-Critic as described in (Haarnoja et al. 2018a). The reason for doing this is to allow for a comparison a CLRL and MERL objective based method, while keeping other factors such as network structure static. Specifically, this network structure uses policy iteration consisting of a value network with target network updating. q q-function networks used to reduce the overestimation bias, and policy network used determine the action selection. The derivation of SAC relies on the so-called 'soft value function' V(s,) = Ear~z (Q (t. a,) + (H((T((, Here the analogous sapacity-limited value function V(st) Pai-~ [Q (8,, a+) -3I(m ( [s)))

---

Because we choose to define the mutual information value in terms of its constituent entropy's slight alteration to SAC method is carried forward into defining the gradients of the value function network. q-function networks, and policy network. This allows to define the gradients used to update these three networks as follows (a full derivation is provided in the appendix),
xxJv(x)= = v (xVx(s))( - Qs(s,, ar) (T (at) log(To(sE, a)l));
VxJ((() = V.Qa (z) ae) r(t,, az) (+((+)))),
[o)( = V log nt (a(ar + aV (log nx (ax) log =s(arl)s)) - Q(a,, st)) fo (m:: st):
where L (mt 8e) defines the policy under the same reparameterization trick a, = Lo 8t), used in SAC and KL-RL, where Ttt is noise vector sampled from spherical Gaussian distribution Haarnoja et al. 2018a)). Together these give the capacity-limited actor-critic algorithm:
Where Ty (az) D for E D indicates the approximation of the marginal policy function based on estimation obtained from all ac- tions in the current training mini-batch. This is done by using the batch of actions create multi-variate Gaussian distribution of the same order as the action space, and determining the probability of each action given that distribu- tion. Using a different approach estimating the marginal action probability would be possi- ble, such as training an additional DNN to es- timate this marginal and updating it based on observed actions. This would replace the batch approximation step with a gradient update of the marginal action distribution approximation network: μ t H Am JaJ (p).
Algorithm 1: Capacity-Limiter Actor-Critic
Initialize: parameter vectors Ø, M1, K2,
Initialize: Memory D for each iteration do for each environment step do
Gr ~ To (arls)) 8141 ~ p(8:41 ft, ae) {st, at, rst,, de), 8141}
for each gradient step do
(a(a)) D for a2 ED
x X \v Vv Jy
R4 - K; -0--0p -xQ AgVnn Jo(ks) for K2 E 1,2
Φ イ Ø- Av
x xx + T)X
4 LEARNING ENVIRONMENTS
4.1 CONTINUOUS N-CHAIN
We explore a continuous action space version of the n-chain environment as described by [ttress 2000). This environment consists of N states with agents starting in the S₁ state and SN as the terminal state. Agents act by selecting a continuous value from [0.1], and the probability of them moving to the next state, p( |d,, 8t), is proportional to the difference between their action and the hidden value H depending on the state they are in, Hs,- These hidden values are sampled from Beta distribution Beta(a, b). The reward is 1 for all states and 0 for the final state. For all tests shown here the number of states N 10, and beta distribution shape parameters a = 50, b = 50. In Figure 5 of the appendix we show and graph the state transition function and diagram for the state and hidden values sampled from the beta distribution.
In Figure I we compare the policies learned by CLAC and SAC. For both algorithms, after training we sample 100,000 actions from the policy for a particular state, and plot the resulting probabil- ity density function over actions. In this example, the optimal action in this state is 0.61. The regularization coefficients for CLAC and SAC (the mutual information term, and the entropy term, respectively) are chosen so that the resulting policies have the same standard deviation of the re- sulting policy. As shown, the mean of the CLAC action distribution shifts away from the hidden state value towards the across-state mean of the beta distribution. This demonstrates that the CLAC algorithm essentially learns useful "prior" over its policy representation. Meanwhile, this shift in mean action distribution is not present the SAC model. This is intuitively related to the dif- ference between these regularization methods as the mean of the distribution relates to the mutual information but not the entropy. We argue that in many natural tasks, policies are are not statistically independent across states, and hence the learning objective of CLAC may better capture useful sta- tistical regularities of the environment, and consequently allow for better learning as well as better generalization

---

Continuous State Action Distributions by Std
CLAC
SAC
Beta
Hidden Value
0.50
0.55
0.60
0.65
0.70 0.75
0.80
0.50
0.55
0.60
0.65
0.70 0.75
0.80
Policy Action Std 0.9
Policy Action Std 0.8
Figure 1: Normal distribution representations of CLAC (bluc) and SAC (orange) model policies with standard deviations 0.8 and 0.9, Light blue displays the value of hidden state feature defining the state transition probability. Beta distribution used to gereate state features red.
4.2 GENERALIZATION CONTINUOUS N-CIAIN
To test the generalizability of the policies learned by CLAC and SAC, these models are first trained on hidden state features generated from one beta distribution, and then placed in an environment with a shifted beta distribution used to generate hidden state features. To ensure that the agents learn the underlying beta distribution and not just single set of random values, these parameters are sampled at the end of each trial. A randomization trial is used where the parameter of the distribution Beta(a, b) is shifted by 10% from its original. An additional extreme randomization trial is used where the value of in Bela(a,) b) is randomly chosen to be either +200 or -00 from the training value. Figure 2 displays these results with the left panel representing the learning environment, the middle representing the randomized environment, and the right panel representing the extreme randomization environment.
Continuous N-Chain Generalization Results
Training Distribution SAC: Entropy Coefficinet
Testing Distribution Extreme Testing Distribution
SAC: Entropy Coefficinet
SAC: Entropy Coefficinet
-2
CLAC
-30
SAC
-10
50
500
-70
-90
0.05
0.10
0.15
0.20
0.05
0.10
0.15
0.20
0.05
0.10
0.15 0.20
CLAC: Mutual Information Coefficient
CLAC: Mutual Information Coefficient
CLAC: Mutual Information Coefficient
Figure 2: Results from tests in the generalizability of learned policies in SAC and CLAC models, varied by mutual information cocfficient and entropy cocfficient. Again the number of states N=10, and the number of agents trained is S. During training, hidden state features resampled after each trial from the same Beta distribution.
These results show that, for the CLAC model, increasing the value of the mutual information co- efficient increases the performance in these perturbed environments. This contrasts the effect of the entropy coefficient in the SAC model, which displays clear correlation between varying the entropy coefficient and generalized performance. Again this connects to the different motivations behind these models, as the CLAC model enables the shifting of mean actions in policy it lates to the policy mutual information, while the mean does not impact the policy entropy and not captured by the entropy coefficient magnitude.

---

4.3 CONTINUOUS CONTROL
In this section we compare the sample efficiencv of CLAC against suite of DRL baselines from the stable baseline implementation (Hill al. (0188). The models tested are the Soft Actor Critic et al. (a)) Deep Deterministic Policy Gradient (DDPG: [iiliccrp et al. (2155) ) Proximal Policy Optimization (PPO1; Schulman et al. (2017)) and Advantage Actor Critic (A2C; Mnih et al. (2016)). These specific models were chosen to allow for fair comparison of the current Actor-Critic, as opposed to newer methods like PPO2, A3C, and TD3 which leverage improvements such asynchronous training, parallel workers, or delayed policy update, which are not used in the current CLAC implementation. Although these additional features are not used in the present implementation, they potential areas of future development in the CLAC method.
implementation of the Capacity-Limiter
As noted, CLAC and SAC implemented in similar fashion, and for this reason compare these models in a series of environments similar to those used to showcase the high sample effi ciency and low reliance on hyperparameter optimization of SAC. The learning environments used are the Roboschool environments (Brockman et al. (2016)), an open source version of the same en- vironments used to compare the Soft Actor- Critic method against other models. Figure 6 displays visual representations from these learning environments, ranging from the relatively simple Inverse Pendulum task (top-right) which consists of 1 action dimension and observation dimensions, to the humanoid environment consisting of 17 action dimensions and 44 observation dimensions.
Roboschool Learning Results Walker 2D
Humanoid
Pendulum
100
800
W
2000
000
Epiocle
200
230
Ant
Half Cheetah
Double Pendulum
1400
Epkoda Reward
1200
8000
CLAC
2000
1.000
800
400
2000
200
200
0
100
200
300
400
500
100
200
300
400
50
100
150
200
Timesteps (Thousand)
Figure 3: For all tests 5 separate agents are trained. Humanoid and Ant are trained for 0.5M steps, Walker and Cheetah for 0.4 M, and the pendulums for 200K. CLAC mutual information coefficients are chosen to be the optimal values sampled from the range [0,0.3] in 0.025 increments.
The results in Figure indicate that CLAC shares the high sample efficiency of the SAC method, while working with different learning objective. Additionally, both SAC and CLAC require little hyperparameter optimization. The only parameter that is tuned in the CLAC model is the mutual information coefficient, with the used values for each model included in the Appendix. The SAC model uses automatic adjustment of the entropy coefficient, that is discussed in the next sec- tion. Our CLAC model is implemented using the same policy, state value, and state-action value networks, and these results show that it has retained some key features of SAC while using a dif- ferent learning objective. This provides support for considering the capacity limited reinforcement learning objective general approach that can be applied to different existing RI. approaches.
All citations for model implementations taken directly from those referenced by the stable baseline repository http://staallbaaeeinee.ccm readthedonss.oocc/mmaaa ho/en//aster/////////////a
NNote that the Roboschool environments have different reward method from MuJoCu Todorov et al. (0022), the environment originally used to test SAC. The Roboschool environment has higher penalties for certain suh-optimal actions, which explains the negative performance in for some models in environments.

---

4.4 GENERALIZATION CONTINUOUS CONTROL
To test generalization in continuous control environments, we utilize the standard OpenAl Gym Pendulum environment, a similar task as the Roboschool Pendulum environment, which will allow us to vary environment features like the mass and length of the pendulum. We adapt the specific method of generating randomized environment features from Packer al, (2018), by varying both the mass and length of the pole in the Pendulum task from given range after each episode of testing. In the training parameter case, the mass and length is kept the same as was used during training, equal I. In the testing parameter the disjoint set, [0.5, 0.7] U [1.3, 1.5]. In the extreme testing parameter case, the features are sampled from disjoint set further from the training values: [0.3, 0.5) U [1.5, 1.6].
the pole mass and length are both sampled randomly from
Gym Pendulum Generalization Results Testing Parameters
Training Parameters
Extreme Testing Parameters
-100
-200
Episode Reward
-75
-400
120
-600
550
-130
-800
800
-175
Figure 4: Results from tests in the generalizability of learned policies in SAC and CLAC models with agents each. Static parameters represent the same environment features used to test agents. Testing parameters rep- resent sampling from disjoint around the training parameters. Extreme testing parameters represent disjoint further from training than the random parameters. Error bars represent 95% confidence interval.
These results show the improved generalization of the policies learned by the CLAC model com- pared to Soft-Actor Critic in a continuous control task. Because of the complexity of the task compared the more simple continuous n-chain environment, this improvement in generalization comes at cost to training performance. SAC and CLAC models were trained for the same num- ber of time steps before testing their generalization capability. This experiment did not compare the performance to other models like DDPG, PPO, and A2C because their lower sample efficiency meant that their policies at the same time in training were too poor to preform well in even the static parameter case.
AUTOMATIC MUTUAL INFORMATION COEFFICIENT ADJUSTMENT
In defining the algorithm for encouraging policies that required less information to represent, we took the practical approach of applying penalty the reward observed by agent based the complexity of the agents policy. However, there is a way that we can regain our original objective of maximizing reward subject given policy channel capacity. This is done by defining target mutual information for our policy, and automatically updating our coefficient to approach that de- sired target while training. This is done much the same way as the automatic adjustment of the entropy coefficient used in SAC that is described in |Haamoja et al. (2018b). A full derivation of this method provided in the Appendix. This derivation provides us the following gradient used to update the value of B:
(( E,,,x, [8(log (a (s)) - log 5e (a (s)) + 3C
where C is the desired maximum policy channel capacity. Thus can define the automatic mutual information coefficient adjustment version of Algorithm 1, as including the update:
- 3 a

---

As we have seen, the mutual information coefficient controls the extent to which the CLRL algorithm prioritizes some degree of policy generalizability over training reward maximization. However, as with the entropy coefficient in SAC, this mutual information coefficient is sensitive to the scale of the reward of the learning environment, because the penalty - - a (als)) ) is applied to the reward observed in the environment. Automatically adjusting the entropy coefficient is partly motivated by invariance to reward scale, and the same benefit is gained by automatically adjusting the mutual information coefficient in CLAC.
However with CLAC, this automatic adjustment has the additional connection to the original moti- vation for applying a capacity- relative to it. Another benefit of this approach is mitigating some of the potential issues caused by negative transfer, a difficult open question in the area of generalization where attempting to transfer learning can negatively impact performance (Taylor & Stone (2009)). These negative effects could be mitigated by having the model update the 33 coefficient throughout learning and determine which value of the coefficient best reflects the degree to which knowledge of policy in one state can be ap- plied to other states, Only tight constraints on information capacity should be disastrously impacted by negative transfer, weaker constraints do not force agents to reuse policies in states where they are not associated with rewards.
allow us define a channel capacity and maximize reward
DISCUSSION
In this work we present a formalization the Capacity-Limice Reinforcement Learning (CLRL) objective and relate it to existing methods in Deep Reinforcement Learning. We argue that CLRL defines a broad approach that can be applied onto existing RL methods to provide better of control over some aspects of generalization. To support this position we present the the Capacity-Limited Actor-Critic (CLAC), an application of CLRL onto a deep off-policy actor-critic model. We use a continuous N-chain environment to clarify the impact on performance and generalization afforded by altering parameters of the CLAC model, specifically the mutual information coefficient.
Empirical results from robot simulation tasks show that the CLAC model can achieve similar sample efficiency and performance as state-of -oe--offthheaaee art methods in complex tasks while using the CLRL ob- jective. This is a key result as this objective differs from that of the similar Maximum Entropy learn- ing objective. MERL based methods use an entropy coefficient to encourage exploration through more random behaviour where possible. Meanwhile, CLRL based methods use the mutual informa- tion coefficient to balance one aspect of generalization, and is motivated by the natural capacity for storing and processing information that exists in biological agents. This difference present in CLAC enabled it to outperform existing methods in environments with perturbed features, displaying the improvement generalization afforded by a capacity -imited learning objective.

---

REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. arXiv preprint 18Xiv:18/2.02334, 2018. !
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXin:/801.01199, 2018a.
5
11
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05900 2018b.
13
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://githun com/hill-a/ 2018.
stable-baselimg
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrxiv:/50.002990 2015. 7
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: /3/2:530 2013. 0
Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühh,, Vladlen Koltun, and Dawn Song. 2018.
Assessing generalization in deep reinforcement learning. arXiv preprint
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 7
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016. [
Chris Sims. Efficient coding explains the universal law of generalization in human perception. Science, 360(6389):652-656 2018. 2
Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pp. 943-950, 2000.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey.
Journal of Machine Learning Research, 1033u):1633-1665, 2009. g
Gerald Tesauro. Td-gammon, self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219, 1994.
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint arXiv:1903.07438, 2019. 4
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
J Tretiak. Rate distortion theory: A mathematical basis for data compression, toby berger, prentice- hall, urbana, (1971), 1974.

---

7 APPENDIX
7.1 GRADIENTS DERIVATION
Because the mathematical connection to the maximum entropy learning objective present in the capacity-limited learning objective, we are able to derive the gradients for training the three DNNs employed by our agent much in the same manner as is used by the original SAC paper (Haarnoja et al. (2018a) Sa)). This alteration is particularly straightforward due to our use of the entropy based def- inition for mutual information. To achieve this derivation, we replace the use of soft q-function and v- v-function in SAC with the capacity- -imited q functionn and v-function, and carry forward this alteration into the remainder of the derivation.
Taking the capacity-limited limited value function
(st) Burwn [Q(st, ar) B(oog n. (ar) log (ar
With state independent marginal action function nH (a.). We train the agent to minimize the squared residual error between supacity-limitee value funetion and (s..s,,)))
(st) - Earwn [Qx az) + B log (at)) - log [a) ]
Jv(x) = BenD 2
which can be optimized via stochastic gradients with:
vv (xVx(s))(Vx( Qu(st, ((log(T)(a.) -og
Taking the capacity-liminn -imited state-value function:
(st, at) r(st, a.) + BR (V (s++1)
where (xe) is the capacity-limited value function, trained to minimize the Bellman residual:
1 (Qx((se, Qu (st,
(k(x) a.) 2
again optimized via stochastic gradient descent.
(M) v. r(8,, 0z)
V
~
We begin with the same policy updating method in SAC, and then replace the soft functions with the capacity- limited functions. This results in guarantee on improved performance in terms of the where Il represents the spherical Gaussian
capacity-limice value objective. We constrain family. At each step we update the policy according to
= arg min D ( (-((+1))) exp(Q (St. ))
ZTa (st)
n'EnT
After employing the policy reparameterization trick used SAC and KL-RL we can rewrite this objective
Ja (0) = Eg,-D En, ([log=, (1g([))((ss))] Qu 8t]]
which gives the gradient as:
V voJ((0) Volog (a(a))))) + (Va, n. (ar) log (ogro((r(ss)) V., Pla,, Volo

---

7.2
CONTINUOUS N-CHAIN ENVIRONMENT
S1
S2
SN
H [0.64234,
0.73254,
0.89344) ~ Beta(a, B)
at, p at, St) (1 at Hs 1) a E [0, 1]
p 8 Oes 8t))
Figure 5: Top: Diagram of the Continuous N-Chain Learning environment. Middle: Example of of hidden state values. Right: Graph the probability of moving to the next state given the absolute distance from the hidden state value and action preformed. Bottom: Function describing this state transition probability.
8 ROBOSCHOOL ENVIRONMENTS
Figure 6: Images of the six roboschool environments used compare performance CLAC against existing Deep methods. Top-left: humanoid. Top-middle Walker 2D. Top-right Inverted Pendulum. Bottom-left: Half-Chhotah. Bottom-mine midddm- Ant Walker. Bottom-right: Inverted Double Pendulum.
8.1 ROBOSCHOOL MUTUAL INFORMATION COEFFICIENTS
Table 1: Model Hyper-paramettrss
Environment
Mutual Information Coefficient
Humanoid
0.075
Walker 2D
0.025
Half Cheetah
0.075
Ant
0.15
Inverted Pendulum
0.1
Double Inverted Pendulum
0.3

---

8.2
AUTOMATIC COEFFICIENT UPDATE DERIVATION
To derive this approach, Haarnoja et al. as:
2018b) define the MERL constrained optimizziion problem
max ST.T Ep, Σ ((,,, an) s.t (accui)oe_--oorrte 8))) - H| 14
(4)
Where H the desired minimum target entropy. Using a dynamic programming approach and the recursive definition the MERL soft-Q function, this constrained optimization problem produces the optimal dual variable o;
az arg min E.,, (-or, ( o:)) ax.H
which is calculated in the practical application through gradient descent according the objective:
(a) (-ow,,-aal log (aelss) aH)
(6)
We begin the derivation of the similar automatic adjustment for the mutual information coefficient by using the entropy definition of the mutual information [(T((as)) H(=(a)) H((T(s)a) applied
to Eq. T Tss follows:
Σ (t) s.t (1s..aa)=-..l(((((((() (og( s)) S CJ Vt
max Fnr Epp
Because our capacity- and we are using the entropy based definition the policy mutual information, we can define optimal dual variable and use gradient descent method to update this coefficient during learning, giving:
value function is recursively defined in the same manner as in SAC,
( Eu, [8(log Te (r(a()s)) log Fe (a, (s)) + 3C]
where C' is the desired maximum policy channel capacity. Thus we can define the automatic mutual information coefficient adjustment version of Algorithm 1, as including the update:
B イ 3 xJ)
at each gradient step. In the results section we compare the performance of CLAC against SAC using automatic coefficient adjustment as well as constant coefficient using the static coefficient with the best performance.
8.3 MODEL HYPER-PARAMETER
Parameter
Value
All Models
Optimizer
Adam
Learning Rate
3e-4
Discount(?)
0.99
Replay Buffer Size
256
Number of Hidden Layers
2
Number of Nodes per Layer
256
Minibatch Size
256
CLAC and SAC Nonlinearity
ReLu
Other Models
Nonlincarity
Tanh


Reviews:
[{"review_id": "rylS12pUYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a reinforcement learning method that regularizes the objective using the mutual information term.\nThe idea is simple and the paper is easy to follow. \n\nHowever, the novelty is limited since the difference between the proposed method and Soft Actor Critic (SAC) is just adding the entropy term of \\pi(a) to the objective function if I understand the method correctly. In addition, the intuition of adding the entropy term of \\pi(a) to the objective is not clearly described.\n\nThe proposed method is evaluated on continuous control tasks.The results shown in the paper is mixed, and I cannot conclude that the proposed method outperforms SAC. Thus, the benefit of the proposed method is not clearly supported by the experimental results.\n\nFor the current form of the paper, I give \"weak reject\" due to the weak support of the experimental results and the unclear motivation of the method.\n\nOne of my concerns is that the way of estimating the \\pi(a) which is the marginal distribution of the action.\nFrom the current manuscript, I did not fully understand how it is estimated in the proposed method.\nI think that the accuracy of the estimation of \\pi(a) is crucial in the proposed method since it is the difference from SAC.\n\nA comment on the paper structure is that the connection to the \"capacity-limited\" objective should be described more explicitly in Section 2.1.\nAlthough the \"capacity-limited\" reminds me of the objective something like \\mathcal{L} + | C - I(X;Y)  | as in [Dupont, 2018],\nthe objective in the proposed method shown in page 2 is \\mathcal{L} + \\beta I(X;Y).\nI did not understand why the proposed method is \"capacity-limited\" until Section 5.\nI think authors should explicitly mention in Section 2.1 that \\beta is adjusted so as to limit the information capacity.\n\nTo improve the manuscript, I request authors the following things:\n\n- The proposed method can be interpreted as adding the penalizing the entropy of \\pi(a) to the entropy-regularized RL.\nI do not fully understand the intuition of penalizing the entropy of \\pi(a) in the context of RL. Please explain it.\n\n- I think some tasks should be performed with longer training. \nFor example, agent should be trained for 1-2 millions steps on Humanoid and Ant tasks.\nIn addition, evaluation with 10 trials are preferable.\n\n- Please cite papers that estimate the marginal distribution \\pi(a) in the same manner. \nIf there is no previous work, please explain the details of estimating \\pi(a) and how \\pi(a) is approximated from samples.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1eDVHYJ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose capacity-limited reinforcement learning and apply an actor-critic method (CLAC) in some continuous control domains. The authors claim that CLAC gives improvements in generalization from training to modified test environments, and that it shows high sample efficiency and requires minimal hyper-parameter tuning.\n\nThe introduction started off making me think about this area in a new way, but as the paper continued I started to find some issues. To begin with, I think the motivation in the introduction could be improved. Why would I choose to limit capacity? This is not sufficiently motivated. I suspect that the author(s) want to argue that it *should* give better generalization, but this argument is not made very clearly in the introduction. Perhaps this is because it would be difficult to make this argument formally, and so it is merely suggested at?\n\nAre there connections between this and things like variational intrinsic control (VIC, Gregor et al. 2016) and diversity is all you need (DIAYN, Eysenbach et al., 2019)? These works aim to maximize the mutual information between latent variable policies and states/trajectories, whereas this work is really doing the opposite. I would be interested in understanding the author’s take on how the two are related conceptually.\n\nMoving to the connections with past work, this paper seriously abuses notation in a way that actually hinders comprehension. Some of the parts that really bothered me, and should be fixed to be correct:\n\nMutual information is a function of two random variables, whereas it is repeatedly expressed as a function of the policy. Being explicit about the random variables / distribution here is pretty important.\n\nIn Equation 2 (and subsequent paragraph) the marginal distributions p_a(a) and p_s(s) are not well defined, marginalizing over what, what are these distributions? I might guess that p_s(s) is the steady state distribution under a policy pi, and that p_a(a) is marginalizing over the same distribution, essentially capturing the prior probability of each action under the policy. But these sort of things need to be said explicitly.\n\nIn KL-RL section there is a sentence with “This allows us to define KL-RL to be the case where p_0(a, s) = \\pi_0(a_t | s_t).” What does this actually mean? One of these is a joint probability for state and action, and one is an action probability conditional on a state. \n\nWhat does \\pi_\\mu(a_t) \\sim \\mathcal{D} mean? \n\nIn the block just before Algorithm 1, many of these symbols are never defined. This needs a significant amount of care (by the authors) and right now relies on the reader to simply make a best guess at what the authors probably intend.\n\nOverall in the first three sections the message I would like the authors to understand is that, in striving for a concise explanation they have significantly overshot. These sections require some significant work to be considered publishable.\n\nThe experiment in section 4.1 is intended to give a clean intuitive understanding of the method, but falls a bit short here. It is clean, but I needed more explanation to really drive the intuition home. I see that CLAC finds a solution more sensitive to the beta distribution, but help me understand why this is the right solution in this particular case.\n\nI really disagree with the conclusions around the experiments in section 4.2. I do not think these results show that for the CLAC model increasing the mutual information coefficient increases performance on the perturbed environments. First, the obvious, how many seeds and where are the standard deviations? Second, the trend is extremely small and the gap between CLAC and SAC is just as minor. Finally, CLAC has better performance on the training distribution which means that it actually lost *more* performance than SAC when transferring to the testing and extreme testing distributions.\n\nThe results for section 4.3 are just not significant enough to draw any real conclusions. The massive temporal variability makes me very suspicious of those super tight error bands, but even without that question, the gap is just not very large.\n\nFinally, in section 4.4 we see the first somewhat convincing experimental results. These look reasonable, but even here I have a fairly pointed question: compared with the results in Packer et al (2018) the amount of regression from training to testing is extremely large (whereas they found vanilla algorithms transfer surprisingly well). Can you explain why there is such a big discrepancy between those results and these? But again, this section’s results are in my opinion the most convincing that something interesting is happening here.\n\nLastly, in section 8.1 the range of hyper-parameters for the mutual information coefficient is very broad, which really makes it hard to buy the claim of requiring minimal hyper-parameter tuning.\n\nAll in all there is something truly interesting in this work, but in the present state I am unable to recommend acceptance, and the amount of work required along with questions raised lead me to be fairly confident in this assessment.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxHhbyRFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThe paper replaces the entropy term in the objective of SAC with a KL(\\pi(a|s) || p(a)) where p(a) is the marginal distribution p(a) = \\int \\pi(a|s) \\mu(s) ds. It derives the gradient of this modified objective and then proceeds to evaluate the resulting \"SAC with mutual information\" to demonstrate (i) better sample efficiency and (ii) generalization to environment change.\n\n\n# Decision\nThe paper proposes an interesting approach but a more thorough evaluation is needed before it can be recommended for publication.\n\n\n# Comments\nThe theoretical contribution is quite small since the connection to RD theory is well established by now, e.g., see papers by Daniel Alexander Braun from Ulm and his students. Therefore, the main contribution is incorporation of the mutual information into SAC. Then the question is what properties/advantages/disadvantages such approach brings. And on this front, the paper is quite weak.\n\nThe chain environment is quite toy-ish and may only serve to indicate that the implementation is correct. The continuous control environments are more interesting, but the learning curves look very unreliable. Just a quick Google search for SAC results on Roboschool reveals much smoother and higher learning curves (e.g., https://medium.com/@kengz/soft-actor-critic-for-continuous-and-discrete-actions-eeff6f651954). It would be paramount to make sure that the baselines are fairly represented in the evaluations.\n\nThe generalization in continuous control is only evaluated on the pendulum by varying length and mass. It is insufficient to make a decisive judgement. Moreover, length and mass are coupled, such that different combinations of length and mass may yield similar dynamics.\n\n=> If better sample efficiency and better generalization are claimed, then a more thorough evaluation is required.\n\nIn general, writing can be improved, figure made nicer and smaller, introduction and connections section made shorter, and the main derivation moved from the Appendix into the main body and explained better.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Capacity-Limited Reinforcement Learning (CLRL) which builds on methods in soft RL to enable learning in agents with limited capacity.

The reviewers raised issues that were largely around three areas: there is a lack of clear motivation for the work, and many of the insights given lack intuition; many connections to related literature are missing; and the experimental results remain unconvincing.

Although the ideas presented in the paper are interesting, more work is required for this to be accepted. Therefore at this point, this is unfortunately a rejection.
Decision: Reject