Title: Reinforcement Learning with Simple Sequence Priors
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Reinforcement Learning

Compression

Sequence learning

Information bottleneck

Mutual information
Abstract: In reinforcement learning (RL), simplicity is typically quantified on an action-by-action basis -- but this timescale ignores temporal regularities, like repetitions, often present in sequential strategies. We therefore propose an RL algorithm that learns to solve tasks with sequences of actions that are compressible. We explore two possible sources of simple action sequences: Sequences that can be learned by autoregressive models, and sequences that are compressible with off-the-shelf data compression algorithms. Distilling these preferences into sequence priors, we derive a novel information-theoretic objective that incentivizes agents to learn policies that maximize rewards while conforming to these priors. We show that the resulting RL algorithm leads to faster learning, and attains higher returns than state-of-the-art model-free approaches in a series of continuous control tasks from the DeepMind Control Suite. These priors also produce a powerful information-regularized agent that is robust to noisy observations and can perform open-loop control.

Reinforcement Learning with Simple Sequence Priors

Abstract

In reinforcement learning (RL), simplicity is typically quantified on an actionby-action basis - but this timescale ignores temporal regularities, like repetitions, often present in sequential strategies. We therefore propose an RL algorithm that learns to solve tasks with sequences of actions that are compressible. We explore two possible sources of simple action sequences: Sequences that can be learned by autoregressive models, and sequences that are compressible with off-the-shelf data compression algorithms. Distilling these preferences into sequence priors, we derive a novel information-theoretic objective that incentivizes agents to learn policies that maximize rewards while conforming to these priors. We show that the resulting RL algorithm leads to faster learning, and attains higher returns than state-of-the-art model-free approaches in a series of continuous control tasks from the DeepMind Control Suite. These priors also produce a powerful informationregularized agent that is robust to noisy observations and can perform open-loop control.

1 Introduction

walkerrun

Actuator1beforetraining

return

800

700

1.0

600

0.5

500

Action

Episode

400

0.0

000

200

LZ-SAC (ours)

0.5

100

0

SAC

1.0

0:0

0.2

0.4

0.6

0.8

1.0

0

20

40

60

80

100

complexity

1e6

Actuator1aftertraining

1.0

8600

0.5

Action

8500

Sequence

0.0

8400

0.5

8300

1.0

S

0.0

0.2

0.4

0.6

0.8

1.0

0

20

40

60

80

100

1e6

Step

Step

Simplicity is a powerful inductive bias [1-3]. In science, we strive to build parsimonious theories and algorithms that involve repetitions of the same basic steps. Simplicity is also important in the context of reinforcement learning (RL). Policies that are simple are often easier to execute, and practical to implement even with limited computational resources [4, 5]. Many control problems have solutions that are compressible: Motor behaviors like running and walking involve moving our legs in a periodic, alternating fashion (Fig. 1). Here it is the sequence of actions selected that is compressible. Sequences with repetitive, periodic elements are easier to predict and can be compressed more than sequences that lack such structure. In the current work, we augment RL agents with a prior that their action sequences should be simple: If solutions to control problems are generally compressible, one should consider only the set of simple solutions to a problem rather than the set of all solutions. In a series of experiments, we show that RL with simple sequence priors produces policies that perform better and more robustly than state-of-the-art approaches without such priors.

Fixed

Adaptive

LZ-SAC

SPAC

SAC

MIRACLE

prior

prior

Individual

SAC

MIRACLE

actions

Action

LZ-SACSPAC

sequences

Simple

人

人

sequence priors

No sequence

priors

Though there are methods for regularizing policies with respect to the individual actions they produce [6, 7], we present a method that explicitly regularizes the sequences of actions used to solve a task. Our regularization incentivizes the agent to use action sequences that can be compressed with a sequence prior. If an action sequence is likely under the prior, one needs fewer bits of information to represent it [8]. We explore two types of sequence priors: i ) Priors in the form of an autoregressive sequence model [9, 10] that learns to predict future actions based on actions that were performed in the past and ii ) priors distilled from a pre-programmed, lossless compression algorithm. Building on the Soft Actor-Critic algorithm (SAC) [7], we introduce Lempel-Ziv Soft Actor-Critic (LZ-SAC), using an off-the-shelf compression algorithm as its prior, and Soft Predictable Actor-Critic (SPAC), using a learned sequence prior (Fig. 2).

The contributions of this paper are the following: We introduce a model-free RL algorithm for maximizing rewards with simple action sequences. In a series of continuous control tasks, we evaluate the utility of such simple sequence priors. First, we investigate whether simple sequence priors speed up policy search: In our experiments, agents with simple sequence priors consistently outperform state-of-the-art model-free RL algorithms in terms of reward maximization. This holds both in terms of learning speed and often in the final performance. Our second result is that our regularization produces an information-efficient RL agent, using fewer bits of information to solve control problems. Information-regularized models are more robust and better at generalizing [11, 4, 12]. Lastly, we demonstrate the agents' advantages in environments with noisy and missing observations. 1

2 Related work

The idea of simplicity has received significant attention in previous work. Maximum entropy RL, for instance, augments the reward function with an entropy maximization term, effectively encouraging the agent to stay close to a simple uniform prior policy over actions [13, 14]. Though uniform priors can lead to discontinuous and unpredictable behaviors, maximum entropy methods are considered simple in that they try to minimize the use of information about the state to select actions [4, 15, 16]. Many current approaches to deep RL - such as SAC [7] - rely on this principle. This concept has been further extended by models like Mutual Information Regularized Actor-Critic Learning (MIRACLE)

[6] and others [17, 18], which use a learnable state-independent prior policy instead of the uniform prior assumed by SAC. SAC and MIRACLE both induce simplicity at the level of individual actions. In contrast, our proposed approach works on the level of action sequences.

It is not only possible to encode preferences for simplicity at the action level. Instead, simplicity can also be imposed by encouraging the agent to maintain simple internal representations - the core idea behind the information bottleneck principle [19]. Deep RL agents that rely on this principle have many appealing properties, such as improved robustness to noise, better generalization, and more efficient exploration characteristics [20-22]. Related to compression is predictability: Berseth et al. [23] learn a density model over states, and then learn a policy that seeks out states that are predictable, leading to self sustaining behaviors in unstable environments. On the opposite end there are methods that seek out unpredictable states [24, 25], or states that the agent cannot compress, to improve exploration. Recently, [4] demonstrated how to construct RL agents that learn policies that use few bits of information by not only compressing individual observations but entire sequences of observations. In some sense, our approach can be seen as a variant of the algorithm from [4]. However, we compress sequences of actions , rather than sequences of observations. Thus, our regularization does not target the complexity of the sequence of internal representations, but instead the complexity of the agent's behavior, manifested in the sequence of actions selected to solve a task.

Finally, simplicity is also an important feature of natural intelligence, where it has been repeatedly argued that simplicity is a unifying principle of human cognition [2]. For instance, [26] showed that people rely on compressed policies, ultimately leading to behavioral effects such as preservation or chunking [27, 28]. Likewise, [29] demonstrated that human exploration behavior can be described by RL algorithms with limited description length, while [30] showed that compression captures human behavior in a visual search task.

3 Control with simple sequences

In this section, we demonstrate how to construct RL agents that solve tasks using simple action sequences. We start by outlining the general problem formulation. We assume that the task can be posed as a Markov Decision Process (MDP). The MDP consists of a state space s ∈ S , an action space a ∈ A , and environment dynamics p ( s 0 ) and p ( s t +1 | s t , a t ) . The dynamics determine the probability of an episode starting in a particular state and the probability of the next state given the previous state and action, respectively. Lastly, there is the discount factor γ and a reward function r ( s t , a t ) that maps state-action pairs to a scalar reward term. The agent learns a policy π θ ( a t | s t ) parameterized by θ that maps states to actions in a way that maximizes the sum of discounted rewards E π θ [ ∑ T t =1 γ t r ( s t , a t ) ] .

Though we want our RL agent to maximize rewards, we encourage it to do so with policies that produce simple action sequences. Inspired by previous approaches, we achieve this by augmenting the agent's objective [7, 4, 14], and search for a set of policy parameters θ that maximize reward while minimizing the complexity of the policy C ( a t -τ : t , s t , θ ) :

where the hyper-parameter α controls the trade-off between complexity and discounted rewards.

We can recover various previous approaches using this formulation. If we, for instance, set C ( a t -τ : t , s t , θ ) = log π θ ( a t | s t ) , we obtain maximum entropy RL algorithms such as SAC. SAC implicitly assumes a uniform prior over individual actions. An alternative to using the uniform prior in maximum entropy RL is to learn a parameterized prior over actions p θ ( a ) based on the empirical distribution of actions the agent selects when solving the task [18, 6]. Setting C ( a t -τ : t , s t , θ ) = log π θ ( a t | s t ) -log p θ ( a t ) , we obtain MIRACLE.

3.1 Simplicity with learned priors

While both SAC and MIRACLE compress sums of individual actions, they do not account for the structure that is present in whole action sequences. To close this gap, we present two methods for regularizing policies on the level of action sequences. For the first, we train a prior distribution ϕ θ ( a t | a t -τ : t -1 ) to predict the agent's future actions from actions it performed in the past. We parameterize the prior as a neural sequence model. We use a causal transformer model [9, 31] to parameterize ϕ θ . Though any type of sequence model could be used in principle, Transformers are arguably better suited for learning complex sequence data with long-range dependencies. We can augment the reward function to incorporate the preference for predictable action sequences as follows:

where a t -τ : t -1 is a sequence of the last τ actions. Optimizing this objective, the agent will get rewarded for performing behaviors that the sequence model can predict better. The sequence model can learn to predict action sequences more easily if they contain structure and regularity. This has two interesting implications. i ) The agent is incentivized to visit states where its actions will be predictable, for instance by oscillating between states in a periodic manner. ii ) To perform actions that make it easier for the sequence model to predict future actions, for instance by performing behaviors that signal to the sequence model how it will behave in the future. We refer to this agent as the Soft Predictable Actor-Critic agent, or SPAC.

3.2 Simplicity with compression algorithms

Since the sequence model and the policy are adapting their behavior and prior towards each other, the augmented reward function will change throughout training. This plasticity can make it challenging to search for viable policies. Moreover, training a sequence model on top of the RL agent creates additional computational overhead. We, therefore, explore the possibility of instilling a simplicity preference without the use of a sequence prior that necessarily adapts over episodes.

This second method for distilling simple sequence priors relies on off-the-shelf data compression algorithms [32]. Lossless data compression algorithms like LZ4 , bzip2 and zlib encode data into sequences of symbols from which the original data can be reconstructed or decompressed exactly. If there are repetitions, regularities, or periodicity in the data, the length of the encoded sequence can be significantly shorter than the original size of the data (Fig. 3). Relying on pre-programmed rules for data compression, this simplicity prior will not change over the course of training. Since compression

Normalized LZ4 complexity

1.0

0.8

0.6

0.4

0.2

0.0

Random

Periodic

Constant

Bang-Bang

algorithms like LZ4 are fast, the sequence prior can be implemented with little computational overhead.

In this setting, we compute C using the extra number of bits needed to encode a t given that we have already encoded a t -τ : t -1 :

where g ( · ) is our compression function and len ( · ) returns the length of a sequence. We use the LZ4 compression algorithm to compute the augmented rewards and refer to this agent as the LZ-SAC agent.

3.3 Implementational details

We implement all agents as extensions of the SAC algorithm. SAC is an off-policy actor-critic algorithm that performs maximum entropy RL. We train critics to learn the augmented Q -value function ˜ Q ( s t , a t -τ : t ) = E [ ∑ N t =1 γ t ˜ r ( s t , a t -τ : t )] with temporal-difference learning [33]. The actors and sequence models are trained to minimize the same loss:

where D is a replay buffer and a t ∼ π θ ( · | s t ) . The LZ-SAC actor minimizes the same loss except that -log ϕ θ ( a t | a t -τ : t -1 ) is replaced with the term in Eq. 3. In practice, we take the minimum of two target Q -networks to train the actor and critic. Learning is achieved by sampling experiences from a replay buffer. To calculate the augmented rewards, we simply sample action sequences a t -τ : t -1 that led to the ( s t , a t , s t +1 , r t ) tuple used for training (see Appendix A for full implementational details).

4 Simple sequence priors guide policy search

hopper stand

hopper hop

quadruped walk

walker run

800

return

800

LZ-SAC

100

600

SPAC

SAC

75

750

600

MIRACLE

Episode

400

RPC

50

500

400

MN

M

200

V

25

250

WW

200

0

0

0

0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

1e6

1e6

1e6

1e6

cheetah run

fish swim

acrobot swingup

reacher hard

return

800

800

1000

150

750

600

600

m

Episode

400

400

100

500

200

50

250

200

M

0

0

0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Step

1e6

Step

1e6

Step

1e6

Step

1e6

We evaluated the agents described in Section 3 on eight continuous control tasks from the DeepMind Control Suite [34]. As an additional baseline we included Robust Predictable Control (RPC) from [4], which compresses sequences of states rather than actions. Many of the tasks in the DeepMind Control Suite promote behaviors with periodic action sequences, such as running and walking. While specialized architectures exist for such tasks [35], we expect compressibility to be a useful inductive bias for learning these behaviors. We trained agents for 1 million environment steps across five seeds and evaluated their abilities at regular intervals with a deterministic policy, as in [7, 36]. We tuned α for each agent and found an α = 0 . 1 to give the best performance in almost all tasks. We found lower information costs α worked better for RPC. Tasks and hyperparameter fitting is described in Appendix B.

Actuator1

Actuator3

Actuator6

1.0

1.0

1.0

1.0

return

Action

0.5

0.5

0.5

0.0

0.0

0.0

0.8

0.5

0.5

0.5

Mean normalized

-1.0

1.0

1.0

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

1.0

1.0

1.0

0.6

Action

0.5

0.5

0.5

0.0

0.0

0.0

0.5

0.5

0.5

0.4

1.0

1.0

1.0

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

1.0

1.0

1.0

0.2

Action

0.5

0.5

0.5

0.0

0.0

0.0

0.5

0.5

0.5

0.0

Discretization resolution

0

1

2

5

10

1.0

0

25

Step

50

75

100

1.0

0

25

Step

50

75

100

1.0

0

25

Step

50

75

100

In a majority of the tasks, the LZ-SAC agent outperformed the SAC, RPC and MIRACLE agents in learning speed and often final performance (Fig. 4). At worst, the LZ-SAC agent matched the learning curves of SAC. This suggests that learning policies with simple sequence priors is indeed fruitful for policy search. We investigated whether this performance difference could simply be attributed to LZ-SAC acting more deterministically than SAC. Lowering the incentive of acting randomly for SAC did not close the performance gap, and often led to worse returns (see Appendix B.1).

Weconducted an additional ablation experiment to make sure the performance gain could be attributed to the compressibility incentive: We varied the resolution with which we discretized the action sequences used as input to the compression algorithm when LZ-SAC was trained. Across three DeepMind Control tasks ( cheetah run , acrobot swingup and walker walk ), we observe that both too low and too high resolutions remove the performance gain of LZ-SAC: When the resolution is 0, the compression bonus no longer conveys a signal about the simplicity of the policy. Conversely, if the resolution is too high, every action sequence is equally incompressible due to the continuous nature of the action space. We find that rounding to two decimal places gave the best performance on average across the tasks (see Fig 5, left). Though we do not expect LZ-SAC to always outperform SAC in more generic control settings, we see an improvement in many tasks with periodic elements, like walking and running. We evaluate LZ-SAC in non-periodic tasks in Section 8.

In two tasks, acrobot swingup and fish swim , the SPAC agent showed a competitive advantage over the other models. However, the SPAC agent lagged behind both the LZ-SAC agent and SAC agent in tasks from the hopper , cheetah , and walker domains. Here the policy that the SPAC agent learned achieves roughly 75% of the return of the LZ-SAC agent.

The policies learned by the SPAC agent shine in a different setting: The agent has discovered solutions to these tasks that essentially use fewer action dimensions than the competitors (Fig. 5, right): For certain actuators a i , the agent outputs a constant value throughout the episodes. For other actuators, the agent alternates between two extreme values, like a soft bang-bang controller [37, 38]. Essentially, the SPAC agent figures out which degrees of freedom it can eliminate without jettisoning rewards. Having fewer degrees of freedom makes it easier to predict the action sequences produced by the policy. This suggests that policy compression using adaptive sequence priors is better suited in tasks with low-dimensional action spaces. Lastly, the difficulties of learning a policy and a sequence prior jointly can be mitigated by using a pre-trained sequence model as a prior. We pre-trained Transformer models to predict action sequences produced by the converged LZ-SAC agents in all eight control tasks. Using the pre-trained Transformers with frozen weights sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Appendix G).

1.0

LZ-SAC

per l

1.0

I return

SPAC

Normalized return

0.8

0.8

SAC

MIRACLE

Normalized

RPC

0.6

0.6

0.4

0.4

0.2

0.2

0.0

un

0.0

0.1

0.2

0.3

0.4

0.5

fish swim

walker run

cheetah

hopper hop

hopper stand

reacher hard

acrobot swingup

XIem padnspenb

5 Simple sequence priors for information-regularized RL

The expected difference in log-likelihood of the agents' actions under the policy versus the prior is an upper bound on the mutual information between states and actions [4, 11, 19]. Encouraging this difference to be low acts as an information-regularizer, the prior p ( a t | a t -τ : t -1 ) being the information bottleneck. We tested the information-efficiency of learned policies; that is, how much reward the agents could collect relative to the information they used to make decisions. For the experiments, we again tested the deterministic versions of the agents. Simulating 25 episodes, we computed how much reward the agents were able to collect divided by the entropy of the distribution of actions used to solve

the policies were deterministic, this entropy term approximated the mutual information between states and actions I ( s ; a ) (see Appendix D). In the left panel of Fig. 6, we show the normalized episodic return per bit. This quantity represents how much reward the agent attains per bit of information it uses on average to make a decision over the course of the episode.

The SPAC agent attained a superior return per bit ratio in five out of eight tasks. LZ-SAC attained the highest return per bit ratio in two tasks, and SAC in one. This indicates that action sequence compression is a powerful information-regularizer, allowing agents to find policies that use significantly fewer bits of information to collect reward than both policy compression models (SAC and MIRACLE), and state sequence compression models (RPC).

6 Robustness to noise

Information-regularized policies tend to show stronger robustness to noisy observations [39, 4]: The less an agent's actions vary systematically with the state, the less will a perturbation to the agent's observation affect its actions. We assessed how observation noise affected the agents' ability to collect rewards. In the following experiments, we added Gaussian noise to the observations the policies were conditioned on, s t ← s t + ϵ t where ϵ t ∼ N ( 0 , diag ( σ )) . We tested the agents on a series of noise levels σ j ∈ [0 . 01 , 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 , 0 . 3 , 0 . 5] . The effect of noise was probed in all tasks except the hopper hop task, since here only the LZ-SAC agent reliably learned a policy that was better than random. Each agent was evaluated using 50 episodes for each noise level.

We evaluated the agents based on how much reward they collected given various levels of observation noise. Averaged over all tasks, the LZ-SAC agent showed the best ability to collect rewards when observations were perturbed with Gaussian noise (see the right panel in Fig. 6). The agents that were better at maximizing rewards showed greater sensitivity to noise: compared to the noise-free setting, LZ-SAC and SAC dropped to 20% and 17% of their average performance, respectively. While the

LZ-SAC agent suffered greater percentage drops in return than the MIRACLE and SPAC agents, it still retained the highest performance for all noise levels. In the highest noise settings, SAC is comparable to the MIRACLE and SPAC agents, despite its generally stronger performance in the noise-free setting. This indicates that the LZ-SAC agent performed better in the noisy setting not only because the policy it learned was generally better at maximizing rewards, but also because of robustness properties afforded by the sequence prior.

7 Open-loop control

If simple action sequences are pervasive in policies learned with RL, these priors could provide a good starting point for policy search. To further test this claim, we evaluated how well tasks from the DeepMind Control Suite could be solved by autoregressively generated action sequences from the sequence priors themselves. We omitted RPC from this analysis since it has the same prior policy as SAC. In our experiments, all agents produced the first 15 actions of an episode in a closed-loop manner. We then conditioned the sequence priors with these first 15 actions and sampled actions autoregressively for the remainder of the episode. The priors of the SAC and MIRACLE agents have no autoregressive component, and generated action sequences in a memory-less manner. We approximated samples from the LZ4 prior by discretizing the action space and sampling the next action proportionally to how low its encoding cost is, given the previous actions.

cheetahrun

walkerrun

return

1.75

Cumulative rewards

80

50

1.50

一

40

60

Normalized

1.25

30

1.00

40

20

三

LZ-SAC

0.75

0.50

20

10

二

SPAC

SAC

0.25

0

0

MIRACLE

0.00

0

50

100

Step

150

200

250

0

100

200

Step

300

400

500

cheetah run

fish swim

hopper hop

hopper stand

reacher hard

walker run

acrobot swingup

The adaptive prior implemented as a transformer generally performs the best in the open-loop setting (Fig. 7, left). This is expected, as it was trained to predict behaviors that solve the tasks. In the fish swim task a uniform prior collects more rewards in the open-loop phase than the sequences generated by the transformer. However, increasing the number of closed-loop actions used to prompt the transformer to 25 made it surpass the performance of the uniform prior (Appendix E). This points to the importance of providing the sequence models with sufficient context to allow them to accurately predict behavior.

More interesting is the performance of the prior obtained from the LZ4 algorithm. Not only does it perform better than chance, but even comes close to the performance of the learned sequence prior in tasks like cheetah run and walker run . By conditioning on only a few actions from the policy, autoregressively approximating samples from LZ4 's prior produced behaviors outperforming the non-sequential priors used by SAC and MIRACLE (Fig. 7, right). This vindicates the compressibility prior as a starting point for policy search.

8 Non-periodic and high-dimensional environments

Many DeepMind Control Suite tasks have solutions that are composed of repeating sub-sequences. Can simple sequence priors be beneficial in tasks without prevalent periodic aspects, or in more complex tasks with high-dimensional state spaces? We first evaluated LZ-SAC against SAC on

button-press-wall-v2

drawer-close-v2

reach-v2

DMC100k

1.0

1.0

LZ-SAC

600

0.8

0.8

0.6

SAC

eturn

500

Success

0.6

0.6

0.4

ret

400

0.4

0.4

=

300

0.2

0.2

0.2

M

?

200

0.0

0.0

0.0

100

0

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

Step

1e5

Step

1e5

Step

1e5

0

LZ-SAC

CURL

SAC

SAC+AE

three tasks from the Metaworld benchmark [40]. The Metaworld benchmark consists of robotic manipulation tasks where periodic action sequences are less prevalent. Despite this, we found that the simple sequence priors allowed agents to learn policies with higher success rates faster (see Fig. 8, left). In fact, the solutions LZ-SAC developed for the robotics tasks often consisted of single, smooth movements with the Sawyer arm. SAC on the other hand relied on more convoluted movements to manipulate the environment, which were more prone to failure.

Next, we benchmarked LZ-SAC in pixel-based versions of six DeepMind Control Suite tasks. We trained LZ-SAC and a SAC baseline on 100k environment steps, and modified both algorithms with a convolutional neural network encoder and performed a random shift augmentation to images before training the actor and critic. With this simple modification and our compression bonus, LZ-SAC outperformed state-of-the-art off-policy algorithms like CURL [41] and SAC+AE [36] on average over the six tasks (see Fig. 8, right). Across five seeds, LZ-SAC outperformed the three baseline models on three out of the six tasks. See Appendix C.1 for the full scores and implementation details.

9 Discussion

We have argued that simplicity is a powerful principle to guide policy search in RL tasks. Because control problems are often solved with sequences of actions that contain repeating temporal patterns, we proposed to use simple sequence priors to create effective and robust RL agents. To provide agents with a notion of compressibility, we proposed two models: One where the strategy used for compression was fixed throughout training (LZ-SAC), and one where the strategy itself could change with experience (SPAC). While the LZ-SAC agents either outperformed or matched the performance of state-of-the-art methods like SAC, the SPAC agents learned more compressible strategies, attaining more rewards while using fewer bits of information to make a decision. Furthermore, agents trained with the LZ-SAC algorithm proved to be the most robust to observation noise. Lastly, both the trained transformer model and the prior distilled from the LZ4 algorithm could autoregressively generate rewarding behaviors in continuous control tasks.

While SPAC showed a better ability to maximize rewards than MIRACLE, returns were lower than SAC and our alternative regularization technique. This is not unexpected. The transformer always required some amount of learning to be able to predict a particular action sequence. The LZ4 algorithm, on the other hand, could immediately provide feedback about the compressibility of the agent's action sequences without any learning. For SPAC, having to learn a sequence prior induced a stronger bottleneck, resulting in more compressed policies. This is consistent with results reported by Eysenbach et al. [4], where a learned dynamics model was used to compress sequences of states: Here compression with a learned prior led to lower returns, but a higher return per bit rate. Our results suggest that sequence compression based on off-the-shelf compression algorithms like LZ4 are better for policy search since there is no need for learning a sequence prior from scratch.

Limitations: Action sequence compression requires either an adaptive prior, a neural sequence model, or a pre-programmed compression algorithm. The particular algorithm used for compression adds computational overhead and determines the types of action sequences that will be favored by the agent [32]. Future work should address the ways in which different compression algorithms or sequence priors affect policy regularization. Furthermore, a sufficiently sophisticated sequence model could in principle learn to predict complex action sequences. A possible extension of our work could be to further penalize the description length of the weights of the sequence model, or the compression algorithm, itself [42]. Finally, while we evaluated our algorithm on a large and diverse set of control tasks within the DeepMind Control Suite and Metaworld, the utility of simple sequence priors could be tested on other benchmarks. In discrete action settings, Atari games [43] would be an appropriate benchmark.

Future Directions: A central feature of simple action sequences is that they are predictable. Being able to predict one's future behavior from past behavior could allow agents to simplify and compress their representations of the state of the world [11, 4]: If the point of observing the state is to determine what action to choose, one could discard information about the state of the world simply by considering the actions that were performed previously. This suggests that simple sequence priors could be beneficial for compressing policies and internal representations jointly.

Finally, humans show a preference for simplicity and compressibility in various domains [30, 2]: We not only produce art and music full of patterns and regularity [44], but also explore novel environments using compressible trajectories [45], and rely on simple rules to explain and generalize about complex stimuli relationships [46-48]. Recently, dopamine activity in the tail of the mouse striatum was argued to encode an action prediction error signal [49]. Such a signal also features in our augmented reward function to compress the policy. In the end, our algorithms could therefore serve as models of how biological agents learn compressible sequential strategies from reinforcement.

References

Carl Rasmussen and Zoubin Ghahramani. Occam's razor. Advances in neural information processing systems , 13, 2000.

Nick Chater and Paul Vitányi. Simplicity: a unifying principle in cognitive science? Trends in cognitive sciences , 7(1):19-22, 2003.

Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control , 7(1): 1-22, 1964.

Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems , 34:27813-27825, 2021.

Pedro A Ortega and Daniel A Braun. Thermodynamics as a theory of decision-making with information-processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 469(2153):20120683, 2013.

Felix Leibfried and Jordi Grau-Moya. Mutual-information regularization in markov decision processes and actor-critic learning. In Conference on Robot Learning , pages 360-373. PMLR, 2020.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation , 4(2):234-242, 1992.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 , 2020.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018.

Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory , pages 25-55. PMLR, 2018.

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw) , pages 1-5. IEEE, 2015.

Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perceptionaction cycle: Models, architectures, and hardware , pages 601-636. Springer, 2010.

Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-information regularization. In International conference on learning representations , 2018.

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv:1901.10902 , 2019.

Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in neural information processing systems , 32, 2019.

Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. arXiv preprint arXiv:2008.00614 , 2020.

Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. arXiv preprint arXiv:1912.05510 , 2019.

Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732 , 2017.

Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355 , 2018.

Lucy Lai and Samuel J Gershman. Policy compression: An information bottleneck in action selection. In Psychology of Learning and Motivation , volume 74, pages 195-232. Elsevier, 2021.

Shuchen Wu, Noémi Élteto, Ishita Dasgupta, and Eric Schulz. Learning structure from the ground up-hierarchical representation learning by chunking. Advances in Neural Information Processing Systems , 35:36706-36721, 2022.

Noémi Éltet˝ o, Dezs˝ o Nemeth, Karolina Janacsek, and Peter Dayan. Tracking human skill learning with a hierarchical bayesian sequence model. PLoS Computational Biology , 18(11): e1009866, 2022.

Marcel Binz and Eric Schulz. Modeling human exploration through resource-rational reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Sreejan Kumar, Carlos G Correa, Ishita Dasgupta, Raja Marjieh, Michael Y Hu, Robert Hawkins, Jonathan D Cohen, Karthik Narasimhan, Tom Griffiths, et al. Using natural language and program abstractions to instill human inductive biases in machines. Advances in Neural Information Processing Systems , 35:167-180, 2022.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Khalid Sayood. Introduction to data compression . Morgan Kaufmann, 2017.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Arjun Sharma and Kris Kitani. Phase-parametric policies for reinforcement learning in cyclic environments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10674-10681, 2021.

Richard Bellman, Irving Glicksberg, and Oliver Gross. On the 'bang-bang' control problem. Quarterly of Applied Mathematics , 14(1):11-18, 1956.

Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems , 34: 27209-27221, 2021.

Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257 , 2021.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pages 5639-5650. PMLR, 2020.

Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems , 24, 2011.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development , 2(3):230-247, 2010.

Mona M Garvert, Tankred Saanum, Eric Schulz, Nicolas W Schuck, and Christian F Doeller. Hippocampal spatio-predictive cognitive maps adaptively guide reward generalization. Nature Neuroscience , 26(4):615-626, 2023.

Can Demircan, Tankred Saanum, Leonardo Pettini, Marcel Binz, Blazej M Baczkowski, Paula Kaanders, Christian F Doeller, Mona M Garvert, and Eric Schulz. Language aligned visual representations predict human behavior in naturalistic learning tasks. arXiv preprint arXiv:2306.09377 , 2023.

Neil R Bramley, Peter Dayan, Thomas L Griffiths, and David A Lagnado. Formalizing neurath's ship: Approximate algorithms for online causal learning. Psychological review , 124(3):301, 2017.

Bonan Zhao, Christopher G Lucas, and Neil R Bramley. How do people generalize causal relations over objects? a non-parametric bayesian account. Computational Brain & Behavior , 5 (1):22-44, 2022.

Francesca Greenstreet, Hernando Martinez Vergara, Sthitapranjya Pati, Laura Schwarz, Matthew Wisdom, Fred Marbach, Yvonne Johansson, Lars Rollik, Theodore Moskovitz, Claudia Clopath, et al. Action prediction error: a value-free dopaminergic teaching signal that drives stable learning. BiorXiv , pages 2022-09, 2022.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019.

Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml , 2010.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 , 2021.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

A Implementation details

Our algorithm is an extension of the Soft Actor-Critic algorithm [7, 50], implemented in PyTorch. Like [36, 51], we initialize agents' replay buffer with a 1000 seed observations collected with a uniform random policy. We update the Q-network pairs to predict the augmented Q value function at every interaction step, and the actor network to maximize the augmented Q value function every second interaction step. The augmented Q targets take the following form:

where a t +1 ∼ π θ ( ·| s t +1 ) . To train the networks we sampled B tuples of state, actions, next state, reward and terminal flags, as well as the τ actions that led to them. To allow the agent to train on observations early in episodes, we sampled τ from a uniform distribution of integers between 5 and τ max (see tables 1, 2) for every mini-batch sample used for training. We update the adaptive priors used in SPAC and MIRACLE together with the actor network. All actor and critic networks consisted of two hidden layers with 256 ReLU units [52] each. The action prior used in MIRACLE was implemented as a multivariate isotropic Gaussian with learnable mean and standard deviation.

Since actions are bounded between -1 and 1, we transform actions sampled from the policy using the tanh transform a t = tanh( u t ) , u t ∼ π θ . We transformed the log-likelihood of an action under a Gaussian policy π θ or action prior ϕ θ using the following formula [7, 50]:

where log ψ θ ( u t | a t -τ : t -1 ) is the log likelihood of the untransformed action u t under the untransformed sequence prior ψ θ .

A.1 Quantifying compressibility

We used the LZ4 algorithm to quantify the compressibility of action sequences. The following code snippet describes how we computed the sequence complexity term in Eq. 3

Since the actions were continuous vectors, we quantized all action sequences with the following function:

Here N determines the granularity of the quantization, with lower N producing more coarse-grained sequences. We set N = 100 for our experiments.

A.2 Pseudo-code for lZ4 algorithm

The lZ4 algorithm compresses sequences by replacing repeating sub-sequences in the data with references to an earlier occurring copy of the sub-sequence. These copies are maintained in a sliding

window. Repeating sub-sequences are encoded as length-distance pairs ( l, d ) , specifying that a set of l symbols have a match d symbols back in the uncompressed sequence. The following pseudo-code sketches compression implemented by LZ4 [32]:

B Hyperparameters

B.1 Increasing reward scale

cheetah run

walkerrun

hopper stand

800

800

800

eturn

700

600

600

600

500

e

400

LZ-SAC

400

400

SAC

300

S

200

SAC

α

200

200

E

2

100

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

Step

1e6

Step

1e6

We investigated whether LZ-SAC's performance improvement could simply be attributed to the incentive to act more deterministically. We tested whether we could attain the same level of performance with SAC just by lowering the incentive of acting randomly. Doubling the scale of extrinsic reward relative to the intrinsic reward of acting randomly did not close the gap between the algorithms (Fig. 9). Instead, we see a decline in performance when we set the incentive of randomness lower than α = 0 . 1 (or α = 0 . 02 in walker run ). This indicates that there is value in having a preference for simplicity on the sequence level that goes beyond simply being predictable at the level of individual actions.

B.2 Algorithm hyperparameters

We implement all algorithms using hyperparameters from [36], with slight deviations depending on the task. Since the computational overhead of using lz4 as a compressor is small compared to the

transformer, we train the agents using larger action sequences. All networks were trained with the Adam optimizer [53]. A full list of hyperparameters is given below:

B.3 Transformer

The SPAC agent uses a causal transformer [9] to learn a prior over action sequences. Our transformer was implemented with the following hyperparameters:

C Task specification

We evaluated agents on tasks from the DeepMind Control Suite. Though dynamics are otherwise deterministic, the starting state of an episode is sampled from a distribution p ( s 0 ) . All episodes consist of 1000 environment steps. However, in practice the episode length is reduced to a number of interaction steps , that is smaller than 1000. This is due to an action repeat hyperparameter which determines how many times an action a t is repeated after it is selected. An action repeat value of 4 thus reduces the number of time steps where the agent needs to act to 250 interaction steps. The action repeat hyperparameter makes it more practical to train agents in the DeepMind Control Suite [34]. We adopt conventional action repeat settings from the literature [36]. In the walker and hopper domains we fitted the action repeat value for all agents among [2 , 4 , 8] and chose the value that produced the best performance. Table 3 shows the action repeat values used in our experiments:

C.1 Pixel-based control

Our LZ-SAC and SAC implementations in the visual control domain differed little from the state-based implementations. We equipped the agents with the convolutional neural network architecture and image augmentation transformation from [54]. All MLPs had two hidden

layers and 512 ReLU units. We optimized the α hyperparameter both for the LZ-SAC and SAC agents for all tasks with a grid search. For tasks with higher dimensional action spaces a lower α of 0.01 worked best. In the end, the best performing α for both algorithms was the same across task. Since LZ4 encoding lengths are not necessarily on the same scale as the log likelihoods of the actions given the state in Equation 4, we experimented with scaling the LZ4 encoding cost by 0 . 5 , which slightly improved performance. The full scores of the models are in Table 4.

D Mutual information approximation

The mutual information I ( X ; Y ) between variables X and Y is a measure of how much they depend on each other. In our case we are interested in the mutual information between states and actions I ( s ; a ) . The mutual information here quantifies how many bits of information knowing the outcome of the random variable s t provides about the other random variable a t , in other words, how much the state reveals about what action will be selected. The more an agent's actions vary as a function of the state, the more bits of information the state reveals about the action that the agent will select.

The mutual information is defined as the following quantity

Normalized return per bit

1.0

0.8

0.6

0.4

0.2

0.0

run

fish swim

walker run

cheetah i

hopper hop

hopper stand

reacher hard

acrobot swingup

quadruped walk

To compute the mutual information we need to know the entropy of the distribution of actions used to solve the task, and the conditional entropy of a | s . Since we make the policies deterministic, we know that H [ a | s ] = 0 . This way the mutual information reduces to the entropy over actions H [ a ] . Given a sample of actions produced by the agent solving the task, we approximate H [ a ] the following way: We first quantized each selected action into 100 ×|A| bins, where A is the action space. We then calculated a categorical distribution over actions based on the frequencies of the quantized actions, the entropy of which we used as our approximation for H [ a ] . The categorical distribution was calculated based on actions selected over 50 episodes.

D.1 Return per bit for stochastic policies

We evaluated the information efficiency of the stochastic variants of the policies learned by LZ-SAC, SPAC, SAC and MIRACLE. We approximated the entropy of the distribution of actions in the same way described above, sampling actions over 50 episodes. To compute the conditional entropy of actions given the state H [ a | s ] , we sampled 1000 actions from the policy at every state s t . We then calculated a categorical distribution (described in previous section) based on this sample, the entropy of which we used as our approximation of the conditional entropy H [ a | s ] . In this setting too, the SPAC algorithm tends to produce the most information efficient agents (Fig. 10).

E Open-loop control

Increasing the number of closed-loop actions used to prompt the transformer makes it generate more rewarding action sequences. This shows the importance of providing the sequence models with enough context, to be able to predict rewarding behaviors (Fig. 11).

F Partial observability

The augmented reward function that induces the preference for simple action sequences depends on the actions the agent selected in the past. This makes the reward function partially observable for a purely state-conditioned policy. Our agents learn to maximize this reward function despite this partial observability. We tested whether augmenting the state to contain information about actions selected in the past produced substantial differences in the learned policies. We equipped the LZ-SAC agents with a recurrent neural network (a Gated Recurrent Unit [55]) whose inputs were sequences of actions. We trained this network along with a single readout layer to produce embeddings e t of action sequences with which we defined the augmented state a t ∼ π θ ( ·| ˜ s t ) where

fish swim

S

200

M

150

e

lative

100

二

LZ-SAC

50

SPAC

SAC

MIRACLE

0

0

50

100

150

200

250

Step

cheetahrun

hopper hop

hopper stand

Episode return

800

80

70

800

LZ-SAC

LZ-SAC RNN

600

60

600

50

400

40

400

30

200

20

200

10

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

1e6

1e6

1e6

Sequence complexity

4360

1500

4340

1495-

1500

1490-

1450

4320

1485-

1400

4300

1480-

1350

4280

1475

1300

1470

4260

1465

1250

0.0

0.2

0.40.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

Step

1e6

Step

1e6

˜ s t = Concatenate ( s t , e t ) . In three tasks we observed only minor differences in the policies learned (Fig 12).

G Pretraining the prior

We trained Transformer models to perform next-action prediction from action sequences produced by the converged LZ-SAC agent for all tasks. Using the pretrained transformers (with frozen weights) rather than a randomly initialized one whose weights were updated with stochastic gradient descent sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Fig. 13). This showcases an interesting possible connection between our sequence compression framework and behavioral cloning.

return

hopper stand

hopper hop

quadruped walk

walker run

600

SPAC

75

750

750

SPAC-pretrained

400

50

500

W

500

e

d

200

25

250

250

S

d

0L

0

0

0

E

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

1e6

1e6

1e6

1e6

return

cheetah run

fish swim

acrobot swingup

reacher hard

1000

750

750

e

500

500

100

500

iso

250

250

0

0

0

E

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Step

1e6

Step

1e6

Step

1e6

Step

1e6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to use action-based complexity cost. This work argues that this action-based cost can be formulated either as compression over the experienced so far actions or using a transformer that predicts the next action given the past. This approach is then claimed to provide access to simplified / predictable action sequences. The work is evaluated against a number of dmcontrol tasks and is shown to do better than considered alternatives, both in terms of performance but also in terms of learning policies that are easier to learn. Overall, the work proposes an interesting set of evaluation and presents an interesting idea. However, I found the work fairly difficult to follow at times, whereby paper clarity can be improved. I am also unsure that the proposed solution is necessarily going to be useful, given the presence of theoretically stable alternatives, such as [4]. Finally, I think this work has potential but should be evaluated on more complex tasks - such as tasks from the field of robotics and / or tasks where exploration / initially bootstrapping policy learning is important. As a result, I cannot recommend this work for publication yet.\n\nSTRENGTHS:\n- A novel action sequence - driven reward formulation\n\n- Interesting and informative experimentation\n\n- Good literature review coverage\n\nWEAKNESSES:\n- No complex experiments that can further strengthen the claims for policy search and robustness\n\n- Insufficient comparisons to prior works, e.g. such as [4] but also works like HER\n\n- Some unclear prose, specifically the abstract, introduction and methods section can benefit from less grandous wording\n\n- Additional justification of the choice of transformer (as opposed to something simpler) and also some argumentation as to why this cannot be pre-trained ahead of time and then fixed during training\n\nQUESTIONS:\nNothing explicit.\n\nLIMITATIONS:\nThe limitations section is a bit lacking, it is unclear how this would work in the context of complex sequential / long-horizon tasks. Also, it is unclear how would this approach work on policies learnt from pixels.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the idea of using simplicity as a prior when learning control policies using RL. Simplicity (or complexity) here is defined as the cost of predicting action $a_t$ given past actions $(a_\\{t - \\tau:t-1})$ and current state ($s_t$). Two different approaches are proposed to learn this complexity value. First, using a learned neural network that learns to predict current action given past actions trajectory $(\\phi(a_t | a_{t-\\tau:t-1}$. This is used as additional reward in SAC. \n\nSecond approach, aims to use lossless data compression algorithms (lz4) and the motivation is to compute complexity as the additional number of bits required to encode $a_t$ given that we have already encoded  $a_{t-\\tau:t-1}$.\n\nSTRENGTHS:\nOverall, the paper is well written and accessible. The idea of simplicity as a prior is also interesting.\n\nWEAKNESSES:\nI think the paper is very vague in terms of using concepts such as simplicity. In the Related works section, the paper simply connects simplicity with maximum entropy RL and suggests that a uniform policy is simple (”stay close to simple uniform prior policy”). I don’t see this connection, infact I a uniform random policy leads to random walks and infact maybe highly non-simple (see e.g. brownian motion and discontinuous behavior). \n\nOn the approach side the first approach to predict actions from past is very strange since it results in trying to model non-stationary behavior and thus makes little sense. This is sort of elluded in the paper as well, where the paper suggests that learning a policy with this reward can be challenging.  From the results we can further see that this approach really doesn’t work. \n\nThe idea of using LZ4 would try to reduce new actions thus sort of minimizing surprise. However, interestingly, while this approach performs well it doesn’t perform unifromly well across all tasks e.g. on 5/8 tasks (Figure 5) it performs quite poorly. \n\nI also find some baseline SAC  numbers to be strange e.g. hopper-hop, quadraped-walk, walker-run have all been shown to be solved using SAC. However, the numbers reported in this paper are very low. I don’t think these baseline numbers are fully accurate or in case there were some other differences maybe the authors can comment.  \n\nRelated Works: There are many other approaches in the literature which focus on simplicity by compressing action sequences into latent variables, or using surprise minimization  as additional part of the objective. None of these approaches are referred to or compared in the text. I would recommend the authors to cite them, and compare against them as well. \n\nAchiam et al. Surprise-based intrinsic motivation for deep reinforcement learning\n\nBerseth et al. SMIRL: Surprise Minimizing Reinforcement Learning in Unstable Environments\n\nQUESTIONS:\nDo the authors know why the baseline numbers (SAC) for these simple environments are very low?\n\nLIMITATIONS:\nlimitations are discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two approaches to regularizing RL policies based on sequential action priors, rather than on single action priors. One approach is SPAC, which trains an autoregressive open-loop policy prior and uses it as a reference regularizer. Another approach uses the LZ4 compression algorithm to estimate how compressible an action sequence is, and uses that as the regularizer. It tests both approaches along with two baselines, SAC and MIRACLE across a few DM Control tasks, and show that the learned policies end up being simpler (more compressible), as well as more performant.\n\nSTRENGTHS:\nThe main strength of the paper lies in the novelty of using LZ4 for a prior and its results. A priori, it is not clear at all whether relying on a dictionary-based compression method that is essentially doing substring matching would translate well to an RL domain, and the results are novel and interesting to see. Including SPAC as well helps to round out the results, showing that a more generic approach to sequence priors is possible. The method and results are mostly quite clear.\n\nWEAKNESSES:\nThe ablations for noise and open-loop policies are maybe not the most pertinent ablations to have in the main paper.\n\nPerhaps there should be ablation for the size and architecture of the transformer prior for SPAC, as a potential way of controlling the complexity/simplicity of the policy prior. It would’ve shed more light on the tradeoff between simplicity vs. performance. \n\nSimilarly, an ablation for how the actions are compressed with LZ4 (there really should be a short explanation in the main paper about this) such as changing the resolution for discretization would’ve also been very enlightening, to again show the tradeoff between simplicity and performance.\n\nFinally it would’ve been very useful to include a different kind of domain, perhaps gridworld-based domains (MinAtar/Atari) to see whether we can see similar trends.\n\n---- After Author Rebuttal ----\nI appreciate the additional ablations for the transformer architecture and action discretization. Therefore I have increased my score.\n\nQUESTIONS:\nDid all four agents use the same SAC that was augmented with multiple past actions? Or was the augmented SAC only used for LZ-SAC and SPAC? If this is not the case, then the same augmented SAC should be used for all four agents to be consistent.\n\nRobustness to noise ablation: Were the agents retrained with the gaussian noise, or was the noise added to the already-trained agents? The plot shows all agents performance decaying relatively similarly, so it doesn’t seem like LZ-SAC or SPAC were particularly more robust to noise. Perhaps the details of this ablation is better put in the appendix so a different ablation can be in the main paper, such as over the transformer architecture or action discretization for LZ4.\n\nLIMITATIONS:\nThe paper already discusses many of the limitations/weaknesses brought up earlier.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a reinforcement learning method that produces simplified action sequences. Simplicity is defined as the predictability of the next action taken by the RL agent and is measured using the number of bits required to encode the action sequence. The authors propose two methods to introduce an information bottleneck for biasing/regularizing the RL agent toward simplicity. The first method SPAC, uses a learned sequence model that predicts the next action and the RL agent is incentivized to produce predictable actions. A second method LZ-SAC directly uses a compression algorithm to compress the action sequence and penalizes the number of bits required for storage. The authors show that this regularization can have several benefits and can lead to better overall performance as compared to other action regularization techniques.\n\nSTRENGTHS:\nThe paper is well-written and presented. The motivation provided for the simplicity of action sequences is reasonable and the overall idea is interesting for the community. Several different types of experiments have been carried out, comparing performance, performance per bit of information and the ability to use open-loop control. If validated with more experiments, the proposed LZ-SAC method could be a good way to regularize RL policies. The future ability of RL agents to discard information about the state of the world and simply use prior actions instead is promising.\n\nWEAKNESSES:\nThe major weakness of the work would be the lack of further baselines. Since the approach is very general, it needs to be validated by comparison with more SOTA off-policy RL baselines to ground the results. It would also be interesting to compare against other information bottleneck methods such as [4].\nIt would also be beneficial if the authors try to categorize which kinds of environments benefit from compression. It could be that environments without periodicity are worse off with compression. The authors claim that the method never performs worse than SAC but this claim needs to be validated by testing on many more non-periodic tasks.\n\nQUESTIONS:\n- What is the methodology for choosing the compression method LZ4 over others?\n- Why doesn't any agent do well in the hopper-hop task?\nIn section 7, how much does the performance drop in open-loop control as compared to closed-loop control? It is not clear from the axis in Figure 7. Are the differences in the cumulative rewards between the 4 methods statistically significant?\n- Figure 10: Why does the LZ-SAC method work the best in this case with stochastic policies, while SPAC performs much worse? Why does stochasticity have this effect?\n\nLIMITATIONS:\nYes they are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}