Reinforcement Learning with Simple Sequence Priors
Authors: Tankred Saanum
Noemi Elteto
Peter Dayan
Marcel Binz
Eric Schulz
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Reinforcement Learning
Compression
Sequence learning
Information bottleneck
Mutual information

Abstract: In reinforcement learning (RL), simplicity is typically quantified on an action-by-action basis -- but this timescale ignores temporal regularities, like repetitions, often present in sequential strategies. We therefore propose an RL algorithm that learns to solve tasks with sequences of actions that are compressible. We explore two possible sources of simple action sequences: Sequences that can be learned by autoregressive models, and sequences that are compressible with off-the-shelf data compression algorithms. Distilling these preferences into sequence priors, we derive a novel information-theoretic objective that incentivizes agents to learn policies that maximize rewards while conforming to these priors. We show that the resulting RL algorithm leads to faster learning, and attains higher returns than state-of-the-art model-free approaches in a series of continuous control tasks from the DeepMind Control Suite. These priors also produce a powerful information-regularized agent that is robust to noisy observations and can perform open-loop control.

Reinforcement Learning with Simple Sequence Priors
Tankred Saanum
Noémi Éltetö'
Peter Dayan 1.2
Marcel Binz'
Eric Schulz'
Max Planck Institute for Biological Cybernetics, University of Tübingen
ttankred. saanumê wow.uumeetebiiignnn.mmmm mpg. de
Abstract
In reinforcement learning (RL), simplicity is typically quantified on an action- by-action basis but this timescale ignores temporal regularities, like repetitions, often present in sequential strategies. We therefore propose an RL algorithm that learns to solve tasks with sequences of actions that are compressible. We explore two possible sources of simple action sequences: Sequences that can be learned by autoregressive models, and sequences that are compressible with off -hellf data compression algorithms. Distilling these preferences into sequence priors, we derive a novel cetrmatiionthhhoreeee objective that incentivizes agents learn policies that maximize rewards while conforming to these priors. We show that the resulting RL algorithm leads to faster learning, and attains higher returns than kate-of-the-aal model-free approaches in series of continuous control tasks from the DeepMind Control Suite. These priors also produce a powerful information- regularized agent that is robust to noisy observations and can perform open-loop control.
Introduction
Simplicity is a powerful inductive bias [1-3]. In science, we strive build parsimonious theo- ries and algorithms that involve repetitions of the same basic steps. Simplicity is also important in the context of reinforcement learning (RL). Poli- cies that are simple are often easier to execute, and practical to implement even with limited computational resources [4, 5]. Many control problems have solutions that are compressible: Motor behaviors like running and walking in- volve moving our legs in a periodic, alternating fashion (Fig. 1). Here tions selected that is compressible. Sequences with repetitive, periodic elements are easier to predict and can be compressed more than quences that lack such structure. In the current work, we augment RL agents with prior that their action sequences should be simple: If so- lutions to control problems are generally com- pressible, one should consider only the set of simple solutions of all solutions. In a series of experiments, we show that RL with simple sequence priors produces policies that perform better and more robustly than state-of the-art approaches without such priors.
walkee run
Actuator before training
800
1.0
100
000
000
Action
Episode
(ours)
SAC
-1.0
0.0
0.2
0.6
0.8
20
GO
100
Sequence complexty
Actuator after training
1.0
8600
0.5
a3c0
Action
0.0
8400
the sequence of ac-
8300
0.
0.4
0.8
0
20
60
100
Step
Step
Figure 1: Action sequences produced by bipedal walker become more compressible with learning. Our algorithm learns policies that solve tasks with simple action sequences, leading to decreased com- plexity and higher returns.
problem rather than the set
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fixed prior
Adaptive prior
LZ-SAC
SPAC
SAC
MIRACLE
Individual actions
SAC
MIRACLE
Action
LZ-SAC
SPAC
sequences
Simple
sequence priors
No sequence priors
Top left: Policy regularization either incentivizes sequences or individual actions to be close to the prior. Priors may be distinct in that they stay fixed over training change from episode to episode with learning. Top right: Agents need to navigate to a goal location, where the shortest path requires fine control, following a repeating pattern. After learning, SAC randomly diffuses among multiple paths. MIRACLE prefers simple path that only goes up and then to the right. Since the optimal path compressible (repeating UP and RIGHT in a periodic fashion), the agents with the simple sequence priors prefer this path. Bottom: An agent with simple sequence priors, in this case SPAC, learns simple strategies for walking, using mostly the left leg push itself forward in repetitive fashion.
Figure 2:
Though there are methods for regularizing policies with respect to the individual actions they produce [6, 7]. we present method that explicitly regularizes the sequences of actions used to solve task. Our regularization incentivizes the agent to use action sequences that can be compressed with a sequence prior. If an action sequence likely under the prior, needs fewer bits of information to represent it [8]. We explore two types of sequence priors: i) Priors the form of an autoregressive sequence model [9, 10] that learns to predict future actions based actions that were performed in the past and ii) priors distilled from a pre-programmed, lossless compression algorithm. Building on the Soft Actor-Critic algorithm (SAC) [7], introduce Lempel-Ziv Soft Actor-Critic (L.Z-SAC), using an off-the-shelf compression algorithm as its prior, and Soft Predictable Actor-Critic (SPAC), using a learned sequence prior (Fig. 2).
The contributions of this paper are the following: We introduce a model-free RL algorithm for maximizing rewards with simple action sequences. In series of continuous control tasks, evaluate the utility of such simple sequence priors. First, we investigate whether simple sequence priors speed up policy search: In experiments, agents with simple sequence priors consistently outperform state-of-the-art model-free RI. algorithms in terms of reward maximization. This holds both in terms of learning speed and often in the final performance. Our second result is that our regularization produces an information-efficient RL agent, using fewer bits information to solve control problems. ieformation-regularizze models are more robust and better at generalizing [11, 4, 12]. Lastly, demonstrate the agents' advantages in environments with noisy and missing observations.
2 Related work
The idea of simplicity has received significant attention in previous work. Maximum entropy RL, for instance, augments the reward function with an entropy maximization term, effectively encouraging the agent to stay close to simple uniform prior policy over actions [13, 14]. Though uniform priors can lead discontinuous and unpredictable behaviors, maximum entropy methods are considered simple in that they try to minimize the use of information about the state to select actions [4, 15, 16]. Many current approaches to deep RI. such as SAC [7]- rely on this principle. This concept has been further extended by models like Mutual Information Regularized actor-Critic Learning (MIRACLE)
For videos showing behaviors leamed with our algorithm, see our project website: dettps:/seqqeeccess
gi github.io io Code: https: / /ttp//gggtttttttee/e com/ soackkreddsaanuun

---

[6] and others [17, 18), which use learnable inteeeendden prior policy instead of the uniform prior assumed by SAC. SAC and MIRACLE both induce simplicity the level of individual actions. In contrast, our proposed approach works on the level of action sequences.
It is not only possible encode preferences for simplicity at the action level. Instead, simplicity can also be imposed by encouraging the agent to maintain simple internal representations the core idea behind the information bottleneck principle [19]. Deep RL agents that rely on this principle have many appealing properties, such as improved robustness to noise, better generalization, and more efficient exploration characteristics [20-22]. Related compression predictability: Berseth et al. [23] learn a density model over states, and then learn policy that seeks out states that are predictable, leading to self sustaining behaviors in unstable environments. On the opposite end there are methods that seek out unpredictable states [24, 25], or states that the agent cannot compress, to improve exploration. Recently, [4] demonstrated how to construct RI. agents that learn policies that use few bits of information by not only compressing individual observations but entire sequences of observations. some sense, our approach can be seen as a variant of the algorithm from [4]. However, we compress sequences of actions, rather than sequences of observations. Thus, our regularization does not target the complexity of the sequence of internal representations, but instead the complexity of the agent's behavior, manifested in the sequence of actions selected to solve task.
Finally, simplicity is also an important feature of natural intelligence, where it has been repeatedly argued that simplicity is a unifying principle of human cogmition [2]. For instance, [26] showed that people rely on compressed policies, ultimately leading to behavioral effects such as preservation or chunking [27, 28]. Likewise, [29] demonstrated that human exploration behavior can be described by RL algorithms with limited description length, while [30] showed that compression captures human behavior in visual search task.
3 Control with simple sequences
In this section, we demonstrate how to construct RL agents that solve tasks using simple action sequences. We start by outlining the general problem formulation. We assume that the task can be posed as Markov Decision Process (MDP). The MDP consists of a state space E S, an action space a E A. and environment dynamics p(se) and p(s(+1 St. a.). The dynamics determine the probability of episode starting particular state and the probability of the next state given the previous state and action, respectively. Lastly, there is the discount factor y and a reward function r(s,, az) that maps state-action pairs to scalar reward term. The agent learns policy xo (a, (s)) parameterized by 0 that maps states to actions in a way that maximizes the sum of discounted rewards EE=1 7r((s, at)
Though we want our RL agent to maximize rewards, we encourage it to do so with policies that produce simple action sequences. Inspired by previous approaches, we achieve this by augmenting the agent's 's objective [7, 4, 14], and search for a set of policy parameters 0 that maximize reward while minimizing the complexity of the policy ((,
max E.Z
(r(s,, -a C(a. ((tttrrtt Sc, 0))
Complexity cost
where the hyper-parameter controls the trade-off between complexity and discounted rewards.
We can recover various previous approaches using this formulation. If we, for instance, set C(a 0) log (a st), we obtain maximum entropy RL algorithms such as SAC. SAC implicitly assumes a uniform prior over individual actions. An alternative to using the uni- form prior in maximum entropy RI. is to learn a parameterized prior over actions po(a) based on the empirical distribution of actions the agent selects when solving the task [18, 6]. Setting log pe (ax), we obtain MIRACLE.
St, 0) log To (a st))
3

---

3.1 Simplicity with learned priors
While both SAC and MIRACLE compress sums of individual actions, they do not account for the structure that is present in whole action sequences. To close this gap, we present two methods for regularizing policies on the level of action sequences. For the first, we train a prior distribution (a at-r:t-) to predict the agent's 's future actions from actions it performed in the past. We parameterize the prior as a neural sequence model. We use a causal transformer model [9, 31] to parameterize Φg. Though any type of sequence model could be used in principle, Transformers are arguably better suited for learning complex sequence data with long-range dependencies. We can augment the reward function incorporate the preference for predictable action sequences as follows:
F(s), at-r:)) = r8t, as) allog (a ss)) log Φ [a--rt-1))
where ag a--ro-1 is sequence of the last T actions. Optimizing this objective, the agent will get rewarded for performing behaviors that the sequence model can predict better. The sequence model can learn to predict action sequences more easily if they contain structure and regularity. This has two interesting implications. i) The agent is incentivized to visit states where its actions will be predictable, for instance by oscillating between states in a periodic manner. ii) To perform actions that make it easier for the sequence model to predict future actions, for instance by performing behaviors that signal to the sequence model how it will behave in the future. We refer to this agent as the Soft Predictable Actor-Critic agent, or SPAC.
3.2 Simplicity with compression algorithms
Since the sequence model and the policy are adapting their behavior and prior towards each other, the augmented reward function will change throughout training. This plasticity can make it challenging to search for viable policies. Moreover, training sequence model on top of the RL agent creates additional computational overhead. We, therefore, explore the possibility of instilling a simplicity preference without the use of a sequence prior that necessarily adapts over episodes.
Normalized L74 complexity
1.0
0.8
0.6
0.4
0.2
This second method for distilling simple se- quence priors relies on off-the-shelf data com- pression algorithms [32]. Lossless data compres- sion algorithms like L24, bzip2 and zlib encode data into sequences of symbols from which the original data can be reconstructed or decompressed exactly. If there are repetitions, regularities, or periodicity in the data, the length of the encoded sequence can be significantly shorter than the original size of the data (Fig. 3). Relying on pre- pre-programmed rules for data com- pression, this simplicity prior will not change over the course of training. Since compression algorithms like LZ 4 are fast, the sequence prior can be implemented with little computational overhead.
0.0
Random
Periodic
Bang Bang
Constant
Figure 3: Some sequences are more compressible than others. A sequence of randomly generated numbers is less compressible than sequences with periodicity, sequences that only contain two types of values (also known Bang- Bang-Bang control), or constant sequences that only contain a single num- ber.
In this setting, we compute C' using the extra number bits needed to encode a, given that we have already encoded a--ra-1:
5, len(g (annggarrrtt-11 (en(g(ar-ra))
(3)
((s a ar))) = r(s,, a,) alog (e (s:) 5z)
(4)

---

where g(-) is our compression function and len(.) returns the length of sequence. We use the L24 compression algorithm to compute the augmented rewards and refer this agent as the LZ-SAC agent.
3.3 Implementational details
We implement all agents as extensions of the SAC algorithm. SAC is off-policy actor-critic algorithm that performs maximum entropy RL. We train critics to learn the augmented Q-value function (s,, A(-ra) = E N 7'r(s,, at-r:)) with temporal-different learning [33]. The actors and sequence models are trained to minimize the same loss:
-= Ba .. (a (s)) log Φ a Q(s,, a
(5)
where D is replay buffer and (T st). The LZ-SAC actor minimizes the same loss except that - log de (a, at-7:-1) replaced with the term in Eq. 3. In practice, we take the minimum of two target -entworks to train the actor and critic. Learning is achieved by sampling experiences from a replay buffer. To calculate the augmented rewards, we simply sample action sequences at-r:-1 that led the (s,, a,, St-1: tuple used for training (sce Appendix A for full implementational details).
Simple sequence priors guide policy search
hopper stand
hopper hop
quadruped walk
walker run
800
Episode return
800
LZ-SAC
100
600
SMC
SPAC
MIRACI
75
750
600
400
RPC
50
500
400
200
25
200
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
cheetah run
fish swim
acrobot swingup
reacher hard
Episode return
800
800
1000
750
600
400
400
100
500
200
50
250
200
0-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Step
1e6
Step
1e6
Step
1e6
Step
1e6
Figure 4: Learning curves of agents in the DeepMind Control suite. Overall, LZ-SAC shows the best learning speed and final performance. Lines the average episodic returns collected in 20 test episodes with a deterministic policy, averaged over five agents trained with different seeds. Shaded regions represent 20-80 performance percentiles.
We evaluated the agents described in Section 3 on eight continuous control tasks from the DeepMind Control Suite [34]. As an additional baseline we included Robust Predictable Control (RPC) from [4], which compresses sequences of states rather than actions. Many of the tasks in the DeepMind Control Suite promote behaviors with periodic action sequences, such as running and walking. While specialized architectures exist for such tasks [35], we expect compressibility to useful inductive bias for learning these behaviors. We trained agents for 1 million environment steps across five seeds and evaluated their abilities at regular intervals with a deterministic policy, as [7, 36). We tuned for each agent and found an = 0.1 to give the best performance in almost all tasks. We found lower information costs a worked better for RPC. Tasks and hyperparameter fitting is described in Appendix B.
5

---

Actuator 1
Actuator 3
Actuator 6
1.0
Mean normalized return
Action
0.8
106
0.6
Action
0.0
-0.5
-0.5
0.4
-3.0
0.2
Action
-65
0.0
1
2
10
-10
Discretization resolution
Step
Step
Step
Figure 5: Left: LZ-SAC works best when the discretization resolution allows the agent distinguish between compressible and incompressible sequences. Bars show mean return and SEM from 20 evaluation episodes over three seeds and three environments after 300k steps. Right: Time series of actions produced by the LZZSAC, SPAC and SAC agent in the walker run task. Action time series of the SPAC agent exhibits a simpler periodic pattern, even outputting a constant value for its third actuator. Actuators were chosen to show qualitatively different behaviors.
In majority of the tasks, the LZ-SAC agent outperformed the SAC, RPC and MIRACLE agents in learning speed and often final performance (Fig. 4). At worst, the LZ-SAC agent matched the learning curves of SAC. This suggests that learning policies with simple sequence priors is indeed fruitful for policy search. We investigated whether this performance difference could simply attributed to LZ-SAC acting more deterministically than SAC. Lowering the incentive of acting randomly for SAC did not close the performance gap, and often led to worse returns (see Appendix B.
We conducted an additional ablation experiment to make sure the performance gain could be attributed to the compressibility incentive: We varied the resolution with which we discretized the action sequences used as input to the compression algorithm when LZ-SAC was trained. Across three Deep Control tasks (cheetah run, acrobot swingup and walker walk), we observe that both too low and too high resolutions remove the performance gain of LZ-SAC: When the resolution is O, the compression bonus no longer conveys a signal about the simplicity the policy, Conversely, if the resolution is too high, every action sequence is equally incompressible due to the continuous nature of the action space. We find that rounding to two decimal places gave the best performance average across the tasks (see Fig 5, left). Though always outperform SAC in more generic control settings, we see an improvement in many tasks with periodic elements, like walking and running. We evaluate LZ-SAC in non-periodic tasks in Section 8.
do not expect LZ-SAC to
In two tasks, acrobot swingup and fish swim, the SPAC agent showed competitive advan- tage over the other models. However, the SPAC agent lagged behind both the LZ-SAC agent and SAC agent in tasks from the hopper, cheetah, and walkes domains. Here the policy that the SPAC agent learned achieves roughly 75% of the return of the L.-SACC agent.
The policies learned by the SPAC agent shine in different setting: The agent has discovered solutions to these tasks that essentially use fewer action dimensions than the competitors (Fig. 5, right): For certain actuators a, the agent outputs constant value throughout the episodes. For other actuators, the agent alternates between two extreme values, like soft bang- controller [37, 38]. Essentially, the SPAC agent figures out which degrees of freedom can eliminate without jettisoning rewards. Having fewer degrees of freedom makes it easier to predict the action sequences produced by the policy. This suggests that policy compression using adaptive sequence priors is better suited in tasks with low-dimensional action spaces. Lastly, the difficulties of learning policy and sequence prior jointly can be mitigated by using a pre-trained sequence model as a prior. We pre-trained Transformer models to predict action sequences produced by the converged L.Z-ACC agents in all eight control tasks. Using the pre-trained Transformers with frozen weights sped up learning significantly and allowed the SPAC agent learn more rewarding behaviors (see Appendix G).

---

Nomallzed return per bit
1.0
1.0
Normalized return
SPAC
SAC
MIRACLE
0.6
0.6
0.2
acrabot swingup
choerah run
Fish swim
hopper hop
hopper stand
quadruped walk
reacher Rard
walker run
0.0
0.1 0.2
0 0.4
0.3
0.5
Figure Left: Normalized return per bit attained by the agents in the eight tasks. Agents with simple
sequence priors achieve better return per bit ratios. Error bars represent the standard error of the mean (SEM). Right: Normalized return averaged over all tasks function of noise scale. Error bands represent the SEM.
Simple sequence priors for information-regulliiie RL
The expected difference in log-likeling log-likelihood of the agents' actions under the policy versus the prior is
an upper bound on the mutual information between states and actions [4, 11, 19]. Encouraging this difference to be low as information-regulariii bottleneck. We tested the information eefiinienfffcyyy of learned policies; that is. how much reward the agents could collect relative to the information they used make decisions. For the experiments, we again tested the deterministic versions of the agents. Simulating 25 episodes, we computed how much reward the agents were able to collect divided by the entropy of the distribution of actions used to solve ET1
the prior p(aa/arrr:tt being the information
the task E
(see Appendix D.I for details and experiments with stochastic policies). Since H[a
the policies were deterministic, this entropy term approximated the mutual information between states and actions [(s: a) (see Appendix D). In the left panel of Fig. 6, we show the normalized episodic return per bit. This quantity represents how much reward the agent attains per bit of information uses average make decision over the course of the episode.
The SPAC agent attained superior return per bit ratio in five out of eight tasks. LZ-SAC attained the highest return per bit ratio in two tasks, and SAC in one. This indicates that action sequence compres- sion a powerful information-rest reguuariee,,, allowing agents to find policies that use significantly fewer bits of information to collect reward than both policy compression models (SAC and MIRACLE), and state sequence compression models (RPC).
Robustness noise
Information-regullriin policies tend to show stronger robustness to noisy observations [39, 4]: The less an agent's actions vary systematically with the state, less will perturbation to the agent's observation affect its actions. We assessed how observation noise affected the agents' ability to collect rewards. the following experiments, we added Gaussian noise to the observations the policies were conditioned St 5; + & where E (O, doag(a)). We tested the agents on series of noise levels G, E [0.01, 0.05, 0.1, 0. 15, 0.2, 0.3, 0.5]. The effect of noise was probed all tasks except the hopper hop task, since here only the LZ-SAC agent reliably learned policy that was better than random. Each agent was evaluated using 50 episodes for each noise level.
We evaluated the agents based on how much reward they collected given various levels of observation noise. Averaged over all tasks, L.Z-AAC agent showed the best ability collect rewards when observations were perturbed with Gaussian noise (see the right panel in Fig. 6). The agents that were better at maximizing rewards showed greater sensitivity to noise: compared to the noise-free setting, LZ-SAC and SAC dropped to 20% and 17% of their average performance, respectively. While the
7

---

LZ-SAC agent suffered greater percentage drops in return than the MIRACLE and SPAC agents, still retained the highest performance all noise levels. the highest noise settings, SAC is comparable to the MIRACIE and SPAC agents, despite its generally stronger performance in the noise-free setting. This indicates that the I.Z-SAC. agent performed better the noisy setting not only because the policy it learned was generally better at maximizing rewards, but also because of robustness properties afforded by the sequence prior.
7 Open-loop control
simple action sequences are pervasive in policies learned with RL, these priors could provide a good starting point for policy search. To further test this claim, evaluated how well tasks from the DeepMind Control Suite could be solved by autoregressively generated action sequences from the sequence priors themselves. We omitted RPC from this analysis since it has the same prior policy as SAC. In our experiments, all agents produced the first 15 actions of an episode in a closed-loop manner. We then conditioned the sequence priors with these first 15 actions and sampled actions autoregressively for the remainder of the episode. The priors of the SAC and MIRACLE agents have no autoregressive component, and generated action sequences approximated samples from the L 4 prior by discretizing the action space and sampling the next action proportionally how low its encoding cost is, given the previous actions.
memory-less manner. We
cheetah run
walker run
Normalized return
Cumulative rewards
90
375-
83
3.99
2:2--
3.00
SPAC
10
SAC
MIRACLE
8.00
30
100
130
250
100
200
300
600
500
podio
Step
Step
Figure 7: Left: Bars represent return attained in the open-loop phase exclusively. Error bars represent the SEM. The sequence prior learned by the transformer generally performs the best. Notably, the L&4 prior performs well in tasks solved with periodic action sequences, like cheetah and walker. Right: Average cumulative reward obtained by agents the cheetah and wal ker tasks. Dashed lines indicate where the open-loop controls start.
The adaptive prior implemented as a transformer generally performs the best in the open-loop setting (Fig. 7, left). This expected, as was trained to predict behaviors that solve the tasks. In the fish swim task uniform prior collects more rewards in the open-loop phase than the sequences generated by the transformer. However, increasing the number of closed-loop actions used to prompt the transformer to 25 made it surpass the performance of the uniform prior (Appendix E). This points the importance providing the sequence models with sufficient context to allow them accurately predict behavior.
More interesting is the performance of the prior obtained from the Lz4 algorithm. Not only does it perform hetter than chance, but even comes close to the performance of the learned sequence prior in tasks like cheetah run and walker run. By conditioning on only a few actions from the policy, autoregressively approximating samples from LZ4 4 prior produced behaviors outperforming the non-sequential priors used by SAC and MIRACLE (Fig. right). This vindicates the compressibility prior as starting point for policy search.
8 Non-periodic and high-dimensional environments
Many DeepMind Control Suite tasks have solutions that are composed of repeating sub-sequences. Can simple sequence priors be beneficial in tasks without prevalent periodic aspects, or more complex tasks with high-dimensional state spaces? We first evaluated LZ-SAC against SAC on
8

---

nutton-press-wallly2
drawer-close-v2
reach-v2
100k
LZ-SAC
Success rake
0.0
SAC
Mean retur
500
600
04
0.4
200
6.7
0.0
0.5
0.0
Step
145 Step
165
Step
165
12-5AC
CURL
9AC+AE
Learning curves for three robotic manipulation tasks in the Metaworld benchmark. Lines represent the success rate across 20 test episodes with a deterministic policy, averaged over five agents trained with different seeds. Right: Average return attained in the pixel-based version of the Deepmind Control Suite tasks across 20 test episodes after 100k environment steps, averaged over tasks and five seeds. Error bars reflect standard deviation. LZ-SAC with image augmentation outperforms the SAC baseline and two state-of-the-ar off-policy methods that combine SAC with representation learning: one using contrastive learning (CURL) and one using autoencoder (SAC+AE).
Figure 8: Left:
three tasks from the Metaworld benchmark [40]. The Metaworld benchmark consists of robotic manipulation tasks where periodic action sequences are less prevalent. Despite this, we found that the simple sequence priors allowed agents to learn policies with higher success rates faster (see Fig. 8, left). In fact, the solutions LZ-SAC developed for the robotics tasks often consisted of single, smooth movements with the Sawyer arm. SAC on the other hand relied on more convoluted movements to manipulate the environment, which were more prone to failure.
Next, we benchmarked LZ-SAC in pixel-based versions of six DeepMind Control Suite tasks. We trained LZ-SAC and SAC baseline on 100k environment steps, and modified both algorithms with convolutional neural network encoder and performed a random shift augmentation to images before training the actor and critic. With this simple modification and our compression bonus, LZ-SAC outperformed state-of-the-ar art off off-cylicy algorithms like CURL [41] and SAC+AE [36] on average over the six tasks (see Fig. 8, right). Across five seeds, LZ-SAC outperformed the three baseline models on three out of the six tasks. See Appendix C.1 for the full scores and implementation details.
9 Discussion
We have argued that simplicity is a powerful principle to guide policy search in RL tasks, Because control problems are often solved with sequences actions that contain repeating temporal patterns, we proposed to use simple sequence priors to create effective and robust RL agents. To provide agents with a notion of compressibility, we proposed compression was fixed throughout training (LZ-SAC), and one where the strategy itself could change with experience (SPAC). While the LZ-SAC agents either outperformed or matched the performance of sfute-ofthhe-e art methods like SAC, the SPAC agents learned more compressible strategies, attaining more rewards while using fewer bits of information make a decision. Furthermore, agents trained with the LZ-SAC algorithm proved to be the most robust to observation noise. Lastly, both the trained transformer model and the prior distilled from the LZ4 algorithm could autoregressively generate rewarding behaviors in continuous control tasks.
models: One where the strategy used for
While SPAC showed a better ability to maximize rewards than MIRACLE, returns were lower than SAC and our alternative regularization technique. This is not unexpected. The transformer always required some amount of learning to be able predict a particular action sequence. The L24 algorithm, on the other hand, could immediately provide feedback about the compressibility of the agent's action sequences without any learning. For SPAC, having learn a sequence prior induced stronger bottleneck, resulting in more compressed policies. This is consistent with results reported by Eysenbach et al. [4], where a learned dynamics model was used to compress sequences of states: Here compression with a learned prior led to lower returns, but higher return per bit rate. Our results suggest that sequence compression based on off off-the-shelf compression algorithms like LZ 4 are better for policy search since there is no need for learning sequence prior from scratch.

---

Limitations: Action sequence compression requires either an adaptive prior, neural sequence model, or a pre-programmed compression algorithm. The particular algorithm used for compression adds computational overhead and determines the types of action sequences that will be favored by the agent [32]. Future work should address the ways in which different compression algorithms or sequence priors affect policy regularization. Furthermore, sufficiently sophisticated sequence model could in principle learn to predict complex action sequences. possible extension of our work could be to further penalize the description length of the weights of the sequence model, or the compression algorithm, itself [42]. Finally, while we evaluated our algorithm on large and diverse set of control tasks within the DeepMind Control Suite and Metaworld, the utility of simple sequence priors could be tested on other benchmarks. In discrete action settings, Atari games [43] would be an appropriate benchmark.
Future Directions: A central feature of simple action sequences is that they are predictable. Being able to predict one's future behavior from past behavior could allow agents to simplify and compress their representations of the state of the world [11, 4]: If the point of observing the state to determine what action to choose, one could discard information about the state of the world simply by considering the actions that were performed previously. This suggests that simple sequence priors could be beneficial for compressing policies and internal representations jointly.
Finally, humans show preference for simplicity and compressibility in various domains [30, 2]: We not only produce art and music full of patterns and regularity [44], but also explore novel environments using compressible trajectories [45]. and rely on simple rules to explain and generalize about complex stimuli relationships [46-48]. Recently, dopamine activity in the tail of the mouse striatum was argued to encode an action prediction error signal [49]. Such signal also features in our augmented reward function to compress the policy. In the end, our algorithms could therefore serve as models of how biological agents learn compressible sequential strategies from reinforcement.
Acknowledgements
We thank the four anonymous reviewers for the instructive and helpful feedback during the review process, as well as the members of the Computational Principles Intelligence Lab for feedback provided throughout the project. We also thank Can Demircan for providing comments on an earlier draft. This work was supported by the Max Planck Society, the German Federal Ministry of Educa- tion and Research (BMBF): Tübingen AI Center, FKZ: 011$18039A, and funded by the Deutsche Forschungsgemneisscaatt (DFG, German Research Foundation) under Germany's 's Excellence Strat- egy-EXC26641-390077655
10

---

References
[1] Carl Rasmussen and Zoubin Ghahramani. Occam's razor. Advances in neural information processing systems, 13, 2000.
[2] Nick Chater and Paul Vitányi. Simplicity: a unifying principle cognitive science? Trends in cognitive sciences, 7(1):19-25 22, 2003.
[3] Ray Solomonoff. A formal theory inductive inference, part Information and control, 7(1): 1-22, 1964.
[4] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Ad- vances in Neural Information Processing Systems, 34:27813-27822 2021.
[5] Pedro A Ortega and Daniel A Braun. Thermodynamies theory of decision-making with information processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences,
copomattoo-processinng
2013.
[6] Felix Leibfried and Jordi Grau-Moya. processes and actor-critic learning. Conference on Robot Learning, pages 360-373. PMLR, 2020.
regularization in markov decision
[7] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vīkash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 2018.
[8] Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression, Neural Computation, 4(2):234-244 1992.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[10] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):
1735-1780, 1997.
[11] Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:16/2.004100, 2016.
[12] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.1074 2020.
[13] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008
[14] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00000 00909, 2018.
[15] Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff.. Learners that little information. In Algorithmic Learning Theory, pages 25-55. PMLR, 2018.
[16] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IEEE, 2015.
[17] Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perception- action cycle: Models, architectures, and hardware, pages 601-636 Springer, 2010.
[18] Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with regularization. In International conference on learning representations, 2018.
[19] Naftali Tishby, Fernando C Pereira, and William Bialek, The information bottleneck method. arXiv preprint physics/0004057, 2000.

---

[20] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv: 199011991100,,,. 2019.
[21] Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in neural information processing systems, 32, 2019.
[22] Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via informa- tion bottleneck in deep reinforcement learning. arXiv preprint arXiv:2008. 006/4, 2020.
[23] Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement lenanin in unstable environments. arXiv preprint /9/p:/9922055000 2019.
[24] Joshua Achiam and Shankar Sastry. Surprise-bart based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv: (rXiv:170..01332 2017.
[25] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 1808. 04355, 2018.
[26] Lucy Lai and Samuel J Gershman. Policy compression: information bottleneck in action selection. In Psychology of Learning and Motivation, volume 74, pages 195-232. Elsevier, 2021.
[27] Shuchen Wu, Noémi Élteto, Ishita Dasgupta, and Eric Schulz. Learning structure from the ground up hierarchical representation learning by chunking. Advances Neural Information Processing Systems, 3672,,,66-36,,, 2022.
[28] Noémi Éltetō, Dezsö Nemeth, Karolina Janacsek, and Peter Dayan. Tracking human skill learning with hierarchical bayesian sequence model. PLoS Computational Biology, 18(11) 1): e1009866, 2022.
[29] Marcel Binz and Eric Schulz. Modeling human exploration through resource- rationall reinforce- ment learning. In Advances in Neural Information Processing Systems, 2022.
[30] Sreejan Kumar, Carlos G Correa, Ishita Dasgupta, Raja Marjieh, Michael Y Hu, Robert Hawkins, Jonathan D Cohen, Karthik Narasimban, Tom Griffiths, et al. Using natural language and program abstractions instill human inductive biases in machines. Advances in Neural Information Processing Systems, 35:167-180 80, 2022.
[31] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 1 15084-15097, 2021.
[32] Khalid Sayood. Introduction to data compression. Morgan Kaufmann, 2017.
[33] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[34] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprini 1001.00000000000000 2018.
[35] Arjun Sharma and Kris Kitani. Phase-parametric policies for reinforcement learning cyclic environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[36] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency model-free reinforcement learning from images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 10674-10681, 2021.
12

---

[37] Richard Bellman, Irving Glicksberg, and Oliver Gross. On the "bang- bang-bang" control problem. Quarterly of Applied Mathematics, 14(1):11-18, 1956.
[38] Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 34: 27209-27221, 2021.
[39] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust problems. arXiv preprint arXiv:2103.06257 2021.
[40] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. 1100. PMLR, 2020.
[41] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference Machine Learning, pages 5639-5650. PMLR, 2020.
[42] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[43] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 279, 2013.
[44] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):230-22) 247, 2010.
[45] Mona M Garvert, Tankred Saanum, Eric Schulz, Nicolas W Schuck, and Christian F Doeller. Hippocampal ppatio-predictive cognitive maps adaptively guide reward generalization. Nature Neuroscience, 26(4):615-62) 2023.
[46] Can Demircan, Tankred Saanum, Leonardo Pettini, Marcel Binz, Blazej M Baczkowski, Paula Kaanders, Christian F Doeller, Mona M Garvert, and Eric Schulz. Language aligned vi- sual representations predict human behavior in naturalistic learning tasks. arXiv preprint arXiv:2306.099 09377, 2023.
[47] Neil R Bramley, Peter Dayan, Thomas L Griffiths, and David A Lagnado. Formalizing neurath's 's ship: Approximate algorithms for online causal learning. Psychological review, 124(3):301, 2017.
[48] Bonan Zhao, Christopher G Lucas, and Neil Bramley. How do people generalize causal relations over objects? con-paraettric bayesian account. Computational Brain & Behavior, 5 (1):22 44, 2022.
[49] Francesca Greenstreet, Hernando Martinez Vergara, Sthitapranjya Pati, Laura Schwarz, Matthew Wisdom, Fred Marbach, Yvonne Johansson, Lars Rollik, Theodore Moskovitz, Claudia Clopath, et al. Action prediction error: a value-free dopaminergie teaching signal that drives stable learning. BiorXiv, pages 2022-09, 2022.
[50] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rXiv:18/2.05000 2018
[51] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint UrXiv:192..0/60 2019.
[52] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.141..6980, 2014.
13

---

[54] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[55] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 3412.355, 2014.
14

---

A Implementation details
Our algorithm is extension of the Soft Actor-Critic algorithm [7. 50], implemented in PyTorch. Like [36, 51], we initialize agents' replay buffer with 1000 seed observations collected with uniform random policy. We update the Q-network pairs predict the augmented Q value function at every interaction step, and the actor network to maximize the augmented Q value function every second interaction step. The augmented Q targets take the following form:
(6) where ar+11 (-) To train the networks we sampled В tuples of state, actions, next state, reward and terminal flags, as well as the 7 actions that led to them. To allow the agent to train on observations early in episodes. we sampled т from a uniform distribution of integers between and max (see tables 1, 2) for every mini-batch sample used for training. We update the adaptive priors used in SPAC and MIRACLE together with the actor network. All actor and critic networks consisted of two hidden layers with 256 ReLU units [52] each. The action prior used in MIRACLE was implemented as a multivariate isotropic Gaussian with learnable mean and standard deviation.
== rst, ax) (S+++ a++1) (log To (at+1 (8++1) log do (ar+1 [at-r:))]
Since actions are bounded between -1 and 1, we transform actions sampled from the policy using the tanh transform a, tanh(u:), (u,), u, Te. We transformed the log-likelihood of an action under Gaussian policy me or action prior dh using the following formula [7, 50]:
D
log Ta(a)) = log u(
Σ log(1 - tanh?(u:)
(7)
log (o,aal = (u
Σ log(1 tanh?(u:))
(8)
where log vhe (ur(uala the log likelihood of the untransformed action u, under the untrans-
formed sequence prior thp.
A.1 Quantifying compressibility
We used the L&4 algorithm quantify the compressibility of action sequences. The following code snippet describes how we computed the sequence complexity term in Eq. 3
sequence_i i action sequence. numpy() .ravel()
get length of compressed sequence at length_tt t1 len len action next = (compression waapeessiionalrrriitttoeeee alggorathhhh compress (sequence_i)
get next on-policy action sequence_ np. conccteeenttee ((seeeenattesseueeeeeeeee (sequence i, action action_next)) axis=0) get length of compressed sequence at t+1 length t2 le (compression delta = length_t t - length_ta t2 delta is the difference
numpy () rrael( (ramee(())))(())))))
compress (sequence_])
Since the actions were continuous vectors, we quantized all action sequences with the following function:
def quantize sequence, N=100): return (action seciion__equencet]]] fllor()
Here N determines the granularity of the quantization, with lower producing more coarse-grained sequences. We set N 100 for experiments.
A.2 Pseudo-code for 124 algorithm
The 1z4 algorithm compresses sequences by replacing repeating sub-sequences in the data with references an earlier occurring copy of the sub-sequence. These copies maintained in sliding
15

---

window. Repeating sub-sequences are encoded as length-distance pairs (1. d), specifying that a set of 1 symbols have match symbols back the uncompressed sequence. The following pseudo-code sketches compression implemented by L [32]:
Algorithm LZ4 pseudo-code
Require: Buffer size b, window size w, sequence
=
window + ()
while < len(k) do
match <- longest repeated occurrence in window found in k
match exists then
d - distance to start of match
[ length of match
c < symbol at k[+1
else
d - 0 I + 0 c < 0
end if output (d, (d,
start t- max(t w + 0,
end f 11
window + Kstarteend
+ ++1+1
end while
B Hyperparameters
B.1
Increasing reward scale
cheetah run
walker run
hopper stand
800
Episode return
600
LZ-SAC
400
400
SAC
SAC
2
0.0
0.2
0.4
0.6
0a8
1.0
0.0
0.2
0.4
0.6
0.8
3.0
0.0
0.2
0.4
0.6
0.8
1,0
Step
Step
Step
Figure 9: Halving the incentive of acting randomly does close the performance gap between L7-SAC and SAC.
We investigated whether LZ-SAC's performance improvement could simply be attributed to the incen- tive to act more deterministically. We tested whether we could attain the same level of performance with SAC just by lowering the incentive of acting randomly. Doubling the scale of extrinsic reward relative to the intrinsic reward of acting randomly did not close the gap between 9). Instead, we see decline in performance when set the incentive of randomness lower than 0.1 (or a 0.02 in walker run). This indicates that there value in having a preference for simplicity on the sequence level that goes beyond simply being predictable at the level of individual actions.
algorithms (Fig.
B.2 Algorithm hyperparameters
We implement algorithms using hyperparameters from [36], with slight deviations depending on the task. Since the computational overhead of using 1z4 compressor is small compared to the
16

---

transformer, we train the agents using larger action sequences. All networks were trained with the Adam optimizer [53]. full list of hyperparameters is given below:
Table 1: Hyperparameters used for SAC, MIRACLE, LZ-SAC, and SPAC
Hyperparameter
Value
Complexity cost (walker run, hopper hop, quadruped walk)
0.02
Complexity cost (all other environments) a
0.1
Discount
0.99
Critic update frequency
1
Actor update frequency
Action prior update frequency
2
Soft update p Batch size
0.01
128
Learning rate actor
10-3
Learning rate critic
10-3
Optimizer
Adam
Max context length LZ-SAC (Tmax) Max context length LZ-SAC (Tmaxi walker run)
Interaction steps in episode 0.4
Interaction steps in episode x 0.25
B.3 Transformer
The SPAC agent causal transformer [9] learn prior action sequences. Our transformer was implemented with the following hyperparameters:
Table 2: Transformer hyperparameters.
Hyperparameter
Value
Attention heads
5
Embedding dimensions
30
Learning rate decay
Linear
Warmup tokens
10000
Max context length (Tmax)
20
Number of layers
2
Learning rate
3 10
Dropout
0.1
Optimizer
Adam
C Task specification
We evaluated agents on tasks from the DeepMind Control Suite. Though dynamics are otherwise deterministic, the starting state of episode is sampled from distribution p(so). All episodes consist 1000 environment steps. However, in practice the episode length is reduced to a number of interaction steps, that is smaller than 1000. This is due to action repeat hyperparameter which determines how many times an action a is repeated after it selected. An action repeat value of 4 thus reduces the number of time steps where the agent needs to act to 250 interaction steps. The action repeat hyperparameter makes more practical to train agents in the DeepMind Control Suite [34]. We adopt conventional action repeat settings from the literature [36]. In the walker and hopper domains we fitted the action repeat value for all agents among [2, 4, 8) and chose the value that produced the best performance. Table 3 shows the action repeat values used in our experiments:
C.1 Pixel-based control
Our L.Z-ACC and SAC implementations in the visual control domain differed little from the state-based implementations. We equipped the agents with the convolutional neural network architecture and image augmentation transformation from [54]. All MLPs had two hidden
17

---

Table 3: Action repeat values.
Task acrobot swingup cheetah run
Action repeat
8
fish swim
4
hopper hop hopper stand quadruped walk reacher hard walker run walker run (SPAC)
8
4
4
Table 4: All final average model scores in the DeepMind Control 100k benchmark with pixel observations. Scores are averaged over 10 runs after 100k steps for five seeds. Scores of the other baselines are the ones reported in the respective papers [41] [36].
DMC 100k
LZ-SAC
SAC
CURL
SAC+AE
Finger, Spin
814
738
767
740
Cartpole, Swingup
683
609
582
311
Walker, Walk
635
609
403
394
Ball In Cup, Catch
653
499
769
391
Cheetah, Run
307
396
299
274
Reacher, Easy
513
427
538
274
layers and 512 ReLU units. We optimized the hyperparameter both for the LZ-SAC and SAC agents for all tasks with a grid search. For tasks with higher dimensional action spaces lower of 0.01 worked best. In the end, the best performing for both algorithms was the same across task. Since LZ encoding lengths are not necessarily on the same scale as the log likelihoods of the actions given the state in Equation 4, we experimented with scaling the LZ4 en- coding cost by 0.5, which slightly improved performance. The full scores the models are Table 4.
Table 5: Action repeat values and complexity cost for both LZ-SAC and SAC agents in the pixel-based tasks.
Task
Action repeat
a
finger spin cartpole swingup walker walk ball in cup catch cheetah run reacher easy
2
0.01
8
0.1
2
0.01
4
0.01
4
0.01
2
0.1
D Mutual information approximation
The mutual information I(X; Y) between variables X and Y is a measure of how much they depend on each other. In our case
interested in the mutual information between states and actions [(s; a). The mutual information here quantifies how many bits of information knowing the outcome of the random variable s, provides about the other random variable a, in other words, how much the state reveals about what action will be selected. The more an agent's actions vary as a function of the state, the more bits of information the state reveals about the action that the agent will select.
The mutual information is defined as the following quantity
1 ((a;s) H[al H H[als]
(9)
18

---

Normalized return per bit
10-
0.6
0.2
0.0
acrobot swingup
cheetah run
fish swim
hopper hop
hopper stand
quadruped walk
reacher hard
walker run
Figure 10: Return per bit for the stochastic policies. LZ-SAC attains the highest return per bit ratio in most tasks.
To compute the mutual information we need to know the entropy of the distribution of actions used to solve the task, and the conditional entropy of a/s. Since we make the policies deterministic, we know that M[[ass] = 0. This way the mutual information reduces to the entropy over actions H(a). Given sample of actions produced by the agent solving the task. we approximate H[a] the following way: We first quantized each selected action into 100 x |.A| bins, where A is the action space. We then calculated categorical distribution over actions based on the frequencies of the quantized actions, the entropy of which we used as our approximation for H(a) The categorical distribution was calculated based on actions selected over 50 episodes.
D.1 Return per bit for stochastic policies
We evaluated the information efficiency of the stochastic variants of the policies learned by L.Z-SAC, SPAC, SAC and MIRACLE. We approximated the entropy of the distribution of actions in the same way described above, sampling actions over 50 episodes. To compute the conditional entropy of actions given the state H [[aass) we sampled 1000 actions from the policy at every state St. We then calculated a categorical distribution (described in previous section) based on this sample, the entropy of which we used as our approximation of the conditional entropy H H[als]. In this setting too, the SPAC algorithm tends to produce the most information efficient agents (Fig. 10).
E
Open-loop control
Increasing the number of closed-loop actions used prompt the transformer makes generate more rewarding action sequences. This shows the importance of providing the sequence models with enough context, to be able predict rewarding behaviors (Fig. 11).
Partial observability
The augmented reward function that induces the preference for simple action sequences depends on the actions the agent selected in the past. This makes the reward function partially observable for a purely conditoondi policy. Our agents leam maximize this reward function despite this partial observability. We tested whether augmenting the state to contain information about actions selected the past produced substantial differences in learned policies. We equipped the I LZ-SAC agents with a recurrent neural network (a Gated Recurrent Unit [55]) whose inputs were sequences of actions. We trained this network along with single readout layer to produce embeddings e; of action sequences with which we defined the augmented state a, me(-)s:) where
19

---

fish swim
Cumulative rewards
200
150
100
LAC
50
SPAC
SAC
MIRACLE
50
100
150
200
250
Step
Figure 11: Cumulative rewards attained by sequence priors in the fish swim environment, with prompt length of 25.
cheetah run
hopper hop
hopper stand
Episode return
800
70
800
LZ-SAC
LZ-SAC RNN
600
800
50
400
40
30
200
20
200
0
1.0
0.0
0.6
Sequence complexity
4360
1500
4340
1495
1500
1450
4320
1400
4300
4280
1300
1470
1465
0.0
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Step
Step
Step
Figure 12: Returns and compressibility of action sequences when the reward function is partially observable and fully observable.
8, Comeatenate(ss, ex). In three tasks we observed only minor differences in the policies learned (Fig 12).
G Pretraining the prior
We trained Transformer models to perform next-action prediction from action sequences produced by converged LZ-SAC agent for all tasks. Using the pretrained transformers (with frozen weights) rather than randomly initialized one whose weights were updated with stochastic gradient descent sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Fig. 13). This showcases interesting possible connection between sequence compression framework and behavioral cloning.
20

---

Episode return Episode return
hopper stand
hopper hop
quadruped walk
walker run
600
SPAC
750
SPAC pretreineeddd M
500
200
250
cheetah run
le5
fish swim
le6
acrobot swingup
reacher hard
1e6
750
500
500
500
250
250
0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Step
1e6
Step
1e6
Step
Step
Figure 13: Learning curves for SPAC using a pretrained transformer with frozen weights and randomly initialized transformer. Using a pretrained transformer allows SPAC to solve the more challenging tasks in the benchmark. Learning curves are averaged over three seeds. Shaded region represents 20-80 percentile.
21


Reviews:
[{"review_id": "X9pDWCT5sM", "reviewer": "Reviewer_teDV", "summary": "In this paper, the authors propose a reinforcement learning method that produces simplified action sequences. Simplicity is defined as the predictability of the next action taken by the RL agent and is measured using the number of bits required to encode the action sequence. The authors propose two methods to introduce an information bottleneck for biasing/regularizing the RL agent toward simplicity. The first method SPAC, uses a learned sequence model that predicts the next action and the RL agent is incentivized to produce predictable actions. A second method LZ-SAC directly uses a compression algorithm to compress the action sequence and penalizes the number of bits required for storage. The authors show that this regularization can have several benefits and can lead to better overall performance as compared to other action regularization techniques.\n", "questions": "- What is the methodology for choosing the compression method LZ4 over others?\n- Why doesn't any agent do well in the hopper-hop task?\nIn section 7, how much does the performance drop in open-loop control as compared to closed-loop control? It is not clear from the axis in Figure 7. Are the differences in the cumulative rewards between the 4 methods statistically significant?\n- Figure 10: Why does the LZ-SAC method work the best in this case with stochastic policies, while SPAC performs much worse? Why does stochasticity have this effect?", "limitations": "Yes they are addressed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and presented. The motivation provided for the simplicity of action sequences is reasonable and the overall idea is interesting for the community. Several different types of experiments have been carried out, comparing performance, performance per bit of information and the ability to use open-loop control. If validated with more experiments, the proposed LZ-SAC method could be a good way to regularize RL policies. The future ability of RL agents to discard information about the state of the world and simply use prior actions instead is promising.", "weaknesses": "The major weakness of the work would be the lack of further baselines. Since the approach is very general, it needs to be validated by comparison with more SOTA off-policy RL baselines to ground the results. It would also be interesting to compare against other information bottleneck methods such as [4].\nIt would also be beneficial if the authors try to categorize which kinds of environments benefit from compression. It could be that environments without periodicity are worse off with compression. The authors claim that the method never performs worse than SAC but this claim needs to be validated by testing on many more non-periodic tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and feedback. We are happy that the reviewer found our idea of using action sequence compression interesting for the RL community. Moreover, it is encouraging that the reviewer acknowledges the possible impact our method could have for regularizing RL policies, if it is tested with more experiments. We conducted a series of new experiments in order to address three weaknesses that were outlined by the reviewer: First, we tested the models on tasks that are not periodic in nature (metaworld). Secondly, we tested LZ-SAC on pixel-based versions of tasks from the Deepmind Control Suite. Lastly, we implemented Robust Predictable Control (RPC) and compared it to our methods in all Deepmind Control tasks from the original submission. In total, we introduce tasks from two new benchmarks and three new baseline models  (RPC, CURL, and SAC+AE). We believe this has made our contribution more valuable, and we thank the reviewer for the suggestions. We discuss the experiments in more detail below:\n\n>The major weakness of the work would be the lack of further baselines. [...] It would also be interesting to compare against other information bottleneck methods such as [4].\n\nSince several reviewers suggested it, we implemented Robust Predictable Control (RPC) (from [4]) and tested it on all DeepMind Control suite tasks from the original submission. We had to increase the information budget of RPC to make it performant in our task, and even then it is outperformed both by LZ-SAC and SPAC on average (see Fig 1.A and 1.B in the one-page pdf). Our method is thus a viable alternative to existing baselines.\n\n>It would also be beneficial if the authors try to categorize which kinds of environments benefit from compression. It could be that environments without periodicity are worse off with compression.\n\nTo make a stronger case for our method, we tested LZ-SAC on more tasks. First, we trained our algorithm on pixel-based versions of tasks from the Deepmind Control Suite. We tested LZ-SAC in the sample efficiency domain, where the agent experiences 100k environment steps. We modified LZ-SAC with a convolutional neural network encoder and performed a random shift augmentation to images before training the actor and critic. With this simple modification, LZ-SAC outperforms not only SAC with the same modification, but also SOTA off-policy models like CURL and SAC+AE on average (see Figure 2.B and Table 1 in the one-page pdf). Simple action sequence priors can therefore be useful for solving high-dimensional visual control tasks.\n\nSecond, we trained LZ-SAC and SAC on three robotic manipulation tasks from the metaworld benchmark, where we expected periodic sequences to play far less of a role. LZ-SAC not only learns to solve the tasks faster than SAC, but more consistently too (Fig 2.A in the one-page pdf). Moreover, LZ-SAC does not solve the tasks with periodic action sequences, but rather with smooth and predictable ones (see Fig 2.C in the one-page pdf).\n\n>The authors claim that the method never performs worse than SAC but this claim needs to be validated by testing on many more non-periodic tasks.\n\nWe do not wish to claim that LZ-SAC never performs worse than SAC (just that it was equal or better in our experiments), and we have added the following sentence to the results section:\n\n**\"Though we do not expect LZ-SAC to always outperform SAC in more generic control settings, we see an improvement in many tasks with periodic elements, like walking and running.\"**\n\nWhat is the methodology for choosing the compression method LZ4 over others?\n\n>We used LZ4 due to its fast compression speed. This information is now added in the methods section in the paper (see below). We justified this further by testing LZ-SAC with two other lossless compression algorithms (bzip and zlib) on the Cheetah - Run task, and found no significant difference in performance (see Fig 3.C in the one-page pdf).\n\n**\"We chose the LZ4 algorithm due to its state-of-the-art compression speed (see Appendix G for experiments with other compression algorithms)\"**.\n\n>Why doesn't any agent do well in the hopper-hop task?\t\n\nWe thank the reviewer for pointing this out. We repeated the experiments for Hopper-Hop with larger network sizes and lower information costs, and see that the SAC scores are consistent with other papers [1, 2]. To make the comparison fair we make the same changes to LZ-SAC and see that it remains superior to SAC. We have added the new scores to the paper (Fig 1.A in the one-page pdf).\n\n>In section 7, how much does the performance drop in open-loop control as compared to closed-loop control? [...] Are the differences in the cumulative rewards between the 4 methods statistically significant?\n\nThe performance drops vary across tasks. In tasks like Cheetah-Run and Walker-Run, SPAC and LZ-SAC achieve roughly 10% of the closed-loop performance, only observing the first 5% of the states in the episode.  Conducting t-tests, we find that SPAC scores (averaged over tasks) for open-loop control are significantly higher than both LZ-SAC (p=0.006), SAC (p=0.004) and MIRACLE scores (p=0.001). We have added these statistics to the paper.\n\n>Figure 10: Why does the LZ-SAC method work the best in this case with stochastic policies, while SPAC performs much worse? Why does stochasticity have this effect?\n\nThis decrease in the return per bit ratio does not reflect a decrease in returns for SPAC, but rather that our approximation of the mutual information between states and actions is lower for the stochastic LZ-SAC policy. Since we do not have analytic expressions for the entropy of the action marginal or policy, we approximate this through sampling (more information in Appendix D).\n\n[1] Eberhardt et al. 2023 ICLR. Pink noise is all you need: Colored noise exploration in Deep Reinforcement Learning\n\n[2] Hansen et al. 2022 ICML. Temporal Difference Learning for Model Predictive Control"}, {"review_id": "sNRwP5vov4", "reviewer": "Reviewer_6AwJ", "summary": "This paper proposes two approaches to regularizing RL policies based on sequential action priors, rather than on single action priors. One approach is SPAC, which trains an autoregressive open-loop policy prior and uses it as a reference regularizer. Another approach uses the LZ4 compression algorithm to estimate how compressible an action sequence is, and uses that as the regularizer. It tests both approaches along with two baselines, SAC and MIRACLE across a few DM Control tasks, and show that the learned policies end up being simpler (more compressible), as well as more performant.", "questions": "Did all four agents use the same SAC that was augmented with multiple past actions? Or was the augmented SAC only used for LZ-SAC and SPAC? If this is not the case, then the same augmented SAC should be used for all four agents to be consistent.\n\nRobustness to noise ablation: Were the agents retrained with the gaussian noise, or was the noise added to the already-trained agents? The plot shows all agents performance decaying relatively similarly, so it doesn’t seem like LZ-SAC or SPAC were particularly more robust to noise. Perhaps the details of this ablation is better put in the appendix so a different ablation can be in the main paper, such as over the transformer architecture or action discretization for LZ4.\n", "limitations": "The paper already discusses many of the limitations/weaknesses brought up earlier.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper lies in the novelty of using LZ4 for a prior and its results. A priori, it is not clear at all whether relying on a dictionary-based compression method that is essentially doing substring matching would translate well to an RL domain, and the results are novel and interesting to see. Including SPAC as well helps to round out the results, showing that a more generic approach to sequence priors is possible. The method and results are mostly quite clear.", "weaknesses": "The ablations for noise and open-loop policies are maybe not the most pertinent ablations to have in the main paper.\n\nPerhaps there should be ablation for the size and architecture of the transformer prior for SPAC, as a potential way of controlling the complexity/simplicity of the policy prior. It would’ve shed more light on the tradeoff between simplicity vs. performance. \n\nSimilarly, an ablation for how the actions are compressed with LZ4 (there really should be a short explanation in the main paper about this) such as changing the resolution for discretization would’ve also been very enlightening, to again show the tradeoff between simplicity and performance.\n\nFinally it would’ve been very useful to include a different kind of domain, perhaps gridworld-based domains (MinAtar/Atari) to see whether we can see similar trends.\n\n---- After Author Rebuttal ----\nI appreciate the additional ablations for the transformer architecture and action discretization. Therefore I have increased my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the helpful feedback and the encouraging review. We particularly appreciate that the reviewer found our method and results novel and interesting - both our use of dictionary-based compression and generic compression with sequence models. We have focused on running the ablation experiments the reviewer suggested, as well as adding more evaluations of our method.\n\n>Perhaps there should be ablation for the size and architecture of the transformer prior for SPAC, as a potential way of controlling the complexity/simplicity of the policy prior.\n\nWe ran an ablation experiment where we varied the number of heads, number of layers, and embedding dimensions of the transformer SPAC used to predict its own actions. We tested the ablated models on three DM Control domains (Cheetah, Walker and Acrobot). We find that smaller transformer architectures work well for policy regularization (see Fig 3.B in the one-page pdf), outperforming the bigger architectures. Otherwise we do not see a consistent benefit of larger architectures. Instead, it is possible that the extra number of parameters of the bigger architectures makes it more challenging to learn both a sequence prior and a policy concurrently. We have added the ablation experiments to the Appendix.\n\n>Similarly, an ablation for how the actions are compressed with LZ4 (there really should be a short explanation in the main paper about this) such as changing the resolution for discretization would’ve also been very enlightening\n\nWe thank the reviewer for this interesting suggestion. We varied the resolution with which we discretized the action sequences used as input to the compression algorithm. Across three DeepMind Control tasks, we observe that both too low and too high resolutions remove the performance gain of LZ–SAC: When the resolution is 0, the compression bonus no longer conveys a signal about the simplicity of the policy. Conversely, if the resolution is too high, every action sequence is equally incompressible due to the continuous nature of the action space. We find that rounding to two decimal places gave the best performance on average across the tasks (Fig 3.A in the one-page pdf). We also added an extra paragraph explaining how LZ4 works in the methods section (see below), and refer to pseudo-code for LZ4 which we have in the appendix A2. \n\n**\"The LZ4 algorithm compresses sequences by replacing repeating sub-sequences in the data with references to an earlier occurring copy of the sub-sequence. These copies are maintained in a sliding window. Repeating sub-sequences are encoded as *length-distance* pairs $(l, d)$, specifying that a set of $l$ symbols have a match $d$ symbols back in the uncompressed sequence. This allows the sequence to be encoded with fewer bits, should it contain such repeating sub-sequences. See Appendix A2 for pseudocode.\"**\n\n>Finally it would’ve been very useful to include a different kind of domain, perhaps gridworld-based domains (MinAtar/Atari) to see whether we can see similar trends.\n\nFollowing the suggestion of the reviewer, we have included more tasks and baselines in our evaluation of the LZ-SAC algorithm. We now include evaluations in pixel-based versions of the DeepMind Control Suite (using the sample efficient 100k benchmark), as well as from the Metaworld benchmark, containing robotic manipulation tasks. We have added results from both benchmarks to our paper.\n\nIn the pixel-domain, we implement LZ-SAC and SAC like in the state-based tasks, but with a convolutional neural network encoder and applying a random shift augmentation to the images before training the actor and critic. Using this simple modification with our compression objective, we see an improvement not only over SAC, but also over recent approaches specialized for solving visual control tasks, like CURL and SAC+AE: Across six visual control tasks from the DeepMind Control Suite, LZ-SAC attains the highest average score (see Fig 2.B and Table 1 in the one-page pdf for model comparisons), showcasing the viability of using LZ-SAC in the pixel domain.\n\nLastly we compared LZ-SAC and SAC in three metaworld tasks. Across these robotic manipulation tasks, we see that LZ-SAC learns a successful policy faster and more reliably than SAC (Fig 2.A in the one-page pdf). Furthermore, the LZ-SAC policy produced smoother, more predictable action sequences than the policy SAC learned (Fig 2.C in the one-page pdf). This suggests that using dictionary-based compressibility as a prior could be useful in more generic continuous control settings, not just in periodic motor control tasks.\n\n>Did all four agents use the same SAC that was augmented with multiple past actions?\n \nWe did not find it necessary to augment the state with the past actions of the agent in the DMC experiments (see confirmatory experiments in appendix F). As such, all agents had the same input. We have added a sentence to clarify this in the paper (see below).\n\n**\"We also did not find it necessary to augment the state representations of our methods with the past actions to solve the DeepMind Control Suite tasks (see Appendix F).\"**\n\n\n>Were the agents retrained with the gaussian noise, or was the noise added to the already-trained agents [...] The plot shows all agents performance decaying relatively similarly, so it doesn’t seem like LZ-SAC or SPAC were particularly more robust to noise. Perhaps the details of this ablation is better put in the appendix so a different ablation can be in the main paper\n\n\nIn the noise ablation, the noise was added to the already trained agents. We found the reviewer’s proposed discretization resolution ablation very interesting and we have decided to replace the noise ablation with the discretization ablation in the paper.\n"}, {"review_id": "UzAbgxFtLF", "reviewer": "Reviewer_GvQF", "summary": "This paper proposes the idea of using simplicity as a prior when learning control policies using RL. Simplicity (or complexity) here is defined as the cost of predicting action $a_t$ given past actions $(a_\\{t - \\tau:t-1})$ and current state ($s_t$). Two different approaches are proposed to learn this complexity value. First, using a learned neural network that learns to predict current action given past actions trajectory $(\\phi(a_t | a_{t-\\tau:t-1}$. This is used as additional reward in SAC. \n\nSecond approach, aims to use lossless data compression algorithms (lz4) and the motivation is to compute complexity as the additional number of bits required to encode $a_t$ given that we have already encoded  $a_{t-\\tau:t-1}$.", "questions": "Do the authors know why the baseline numbers (SAC) for these simple environments are very low?", "limitations": "limitations are discussed in the main paper.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Overall, the paper is well written and accessible. The idea of simplicity as a prior is also interesting. ", "weaknesses": "I think the paper is very vague in terms of using concepts such as simplicity. In the Related works section, the paper simply connects simplicity with maximum entropy RL and suggests that a uniform policy is simple (”stay close to simple uniform prior policy”). I don’t see this connection, infact I a uniform random policy leads to random walks and infact maybe highly non-simple (see e.g. brownian motion and discontinuous behavior). \n\nOn the approach side the first approach to predict actions from past is very strange since it results in trying to model non-stationary behavior and thus makes little sense. This is sort of elluded in the paper as well, where the paper suggests that learning a policy with this reward can be challenging.  From the results we can further see that this approach really doesn’t work. \n\nThe idea of using LZ4 would try to reduce new actions thus sort of minimizing surprise. However, interestingly, while this approach performs well it doesn’t perform unifromly well across all tasks e.g. on 5/8 tasks (Figure 5) it performs quite poorly. \n\nI also find some baseline SAC  numbers to be strange e.g. hopper-hop, quadraped-walk, walker-run have all been shown to be solved using SAC. However, the numbers reported in this paper are very low. I don’t think these baseline numbers are fully accurate or in case there were some other differences maybe the authors can comment.  \n\nRelated Works: There are many other approaches in the literature which focus on simplicity by compressing action sequences into latent variables, or using surprise minimization  as additional part of the objective. None of these approaches are referred to or compared in the text. I would recommend the authors to cite them, and compare against them as well. \n\nAchiam et al. Surprise-based intrinsic motivation for deep reinforcement learning\n\nBerseth et al. SMIRL: Surprise Minimizing Reinforcement Learning in Unstable Environments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thorough feedback, and we are glad that the reviewer found our idea of using sequence simplicity as a policy prior in RL interesting. In the following we address the issues raised by the reviewer: We have clarified what we mean by simplicity, as well as the strengths and weaknesses of SPAC. We have also fine-tuned SAC to be more performant on Hopper Hop and Quadruped Walk. Lastly, we conducted an experiment where we pretrain the SPAC transformer on converged policies, which improves performance. We thank the reviewer for raising these issues. We think our paper’s claims are better supported as a result of the reviewer’s feedback.\n\n>On the approach side the first approach to predict actions from past is very strange since it results in trying to model non-stationary behavior\n\nThe reviewer is correct to point out the challenges of training the transformer towards non-stationary targets. However, the training is intended to encourage the agent to solve tasks with predictable and simple action sequences. We implemented another baseline using an information bottleneck, Robust Predictable Control (RPC), that uses a learnable prior over states to compress state-representations. In our experiments SPAC outperforms RPC on average over the 8 DeepMind control tasks (see Fig 1.A and Fig 1.B in the one-page pdf). This shows that, for a model with a strong information bottleneck, SPAC is quite performant. \n\nTo address the issue of non-stationarity further, we pretrained the transformer model to predict the action sequences of converged policies, and trained the SPAC agent using this pretrained transformer, instead of a randomly initialized one, on all tasks from the original submission. This agent learns to solve the tasks more effectively, attaining scores comparable to SAC in challenging domains like walker and quadruped (see Fig. 4 in the one-page pdf). \n\n>I also find some baseline SAC numbers to be strange e.g. hopper-hop, quadraped-walk, walker-run have all been shown to be solved using SAC.\n\nWe thank the reviewer for pointing this out. We repeated the experiments for Hopper Hop and Quadruped Walk with larger network sizes and lower information costs, and now obtain scores comparable to those reported in other published papers [1, 2]. We have updated the scores we report in the paper to reflect this, see Fig 1.A in the one-page pdf for the updated scores. The SAC agent does indeed solve the walker run task, and the score was always consistent with those reported elsewhere [1, 2]. Our LZ-SAC algorithm still outcompetes SAC after fine-tuning.\n\n>while this approach performs well it doesn’t perform unifromly well across all tasks e.g. on 5/8 tasks (Figure 5) it performs quite poorly.\n\nThe experiment presented in Figure 5 is not the main result of our paper, rather it shows that the models we evaluated solve the tasks using various amounts of bits of information. We find that SPAC achieves the highest return per bit of information it uses to solve the tasks. While LZ-SAC is better at maximizing reward, it uses more information to do so. SPAC therefore, we argue, is a strong algorithm for performing information-constrained control.\n\n>There are many other approaches in the literature which focus on simplicity by compressing action sequences into latent variables, or using surprise minimization as additional part of the objective. \n\nWe appreciate the reviewer’s pointers to related work. We updated the related work section to discuss similarities to our approach (see below). While Berseth et al. learn a policy that maximizes the predictability of the next state given the current state, we maximize the predictability/compressability of the next action, given previous actions. Moreover, we evaluated our algorithms in the DeepMind Control Suite, which does not satisfy the criterion of being an unstable environment as described in Berseth et al. In these environments, the agent could easily maximize predictability by not moving, remaining in the same state throughout the episode.\n\n**\"Related to compression is predictability: Berseth et al. [21 ] learn a density model over states, and then learn a policy that seeks out states that are predictable, leading to self-sustaining behaviors in unstable environments. On the opposite end there are methods that seek out unpredictable states [22, 23], or states that the agent cannot compress, to improve exploration.\"**\n\n>the paper simply connects simplicity with maximum entropy RL and suggests that a uniform policy is simple (”stay close to simple uniform prior policy”). I don’t see this connection, infact I a uniform random policy leads to random walks and infact maybe highly non-simple\n\nThe reviewer raises an interesting point that maximum entropy RL can produce random behavior that is not predictable. However, as is common in the RL literature  [3, 4], we understand the simplicity of an input-output relation as the mutual information between the states and the actions. Since a uniform random policy does not use information about the state to select actions, it is considered simple. Through the lens of mutual information minimization, there is therefore a clear link between our surprise minimization and maximum entropy RL. We have rewritten parts of the exposition to make this connection clearer (see below).\n\n**\"Though uniform priors can lead to discontinuous and unpredictable behaviors, maximum entropy methods are considered simple in that they try to minimize the use of information about the state to select actions [4, 15, 16].\"**\n\n[1] Eberhardt et al. 2023 ICLR. Pink noise is all you need: Colored noise exploration in Deep Reinforcement Learning\n\n[2] Hansen et al. 2022 ICML. Temporal Difference Learning for Model Predictive Control\n\n[3] Eysenbach et al. 2021 NeurIPS. Robust Predictable Control\n\n[4] Bassily et al. 2018 Algorithmic Learning Theory. Learners that Use Little Information"}, {"review_id": "BA134LQdS2", "reviewer": "Reviewer_7uiK", "summary": "This work proposes to use action-based complexity cost. This work argues that this action-based cost can be formulated either as compression over the experienced so far actions or using a transformer that predicts the next action given the past. This approach is then claimed to provide access to simplified / predictable action sequences. The work is evaluated against a number of dmcontrol tasks and is shown to do better than considered alternatives, both in terms of performance but also in terms of learning policies that are easier to learn. Overall, the work proposes an interesting set of evaluation and presents an interesting idea. However, I found the work fairly difficult to follow at times, whereby paper clarity can be improved. I am also unsure that the proposed solution is necessarily going to be useful, given the presence of theoretically stable alternatives, such as [4]. Finally, I think this work has potential but should be evaluated on more complex tasks - such as tasks from the field of robotics and / or tasks where exploration / initially bootstrapping policy learning is important. As a result, I cannot recommend this work for publication yet.", "questions": "Nothing explicit.", "limitations": "The limitations section is a bit lacking, it is unclear how this would work in the context of complex sequential / long-horizon tasks. Also, it is unclear how would this approach work on policies learnt from pixels.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel action sequence - driven reward formulation\n\n- Interesting and informative experimentation\n\n- Good literature review coverage", "weaknesses": "- No complex experiments that can further strengthen the claims for policy search and robustness\n\n- Insufficient comparisons to prior works, e.g. such as [4] but also works like HER\n\n- Some unclear prose, specifically the abstract, introduction and methods section can benefit from less grandous wording\n\n- Additional justification of the choice of transformer (as opposed to something simpler) and also some argumentation as to why this cannot be pre-trained ahead of time and then fixed during training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and thoughtful feedback. We are glad that the reviewer found our work interesting, and we are encouraged by the fact that the reviewer thinks it has potential. The reviewer points out four main limitations of the submitted paper. We have addressed all four weaknesses pointed out by the reviewer, and it has made our paper a lot stronger. We address the weaknesses in detail below:\n\n>No complex experiments that can further strengthen the claims for policy search and robustness. [...] Also, it is unclear how would this approach work on policies learnt from pixels.\n\nThe reviewer raised the point that we do not address how our algorithm might work in pixel-based environments. We have since added experiments showing that LZ-SAC can be adapted to pixel-based tasks too. We trained LZ-SAC and SAC to perform control from pixels in the DeepMind Control Suite 100k benchmark (DMC 100k), where the agents interact with the task through 100k environment steps. To adapt LZ-SAC to the visual control domain, we equipped it with a convolutional neural network encoder and performed a random shift augmentation on batches of images when we trained the actor and critic. We adapted SAC the same way to make the methods comparable. We see that, with our simple modification and compression bonus, LZ-SAC can beat recent approaches like CURL and SAC+AE, as well as our modified SAC implementation, on average across the six visual control tasks from DMC100k (see Fig 2.B and Table 1. in the one-page pdf).\n\n>Finally, I think this work has potential but should be evaluated on more complex tasks - such as tasks from the field of robotics and / or tasks where exploration / initially bootstrapping policy learning is important.\n\nTo increase the breadth of tasks that we use for evaluating LZ-SAC, we included three robotic manipulation tasks from the metaworld benchmark. Across the three tasks we see that LZ-SAC learns to solve the manipulation problems faster and more consistently than SAC (Fig 2.A in the one-page pdf). Moreover, the action sequences LZ-SAC uses to solve these tasks do not exhibit periodic properties, but are still smooth and predictable (Fig 2.C in the one-page pdf). This showcases the possibility of using LZ-SAC not only in tasks with strong periodic patterns, but also in more generic continuous control settings.\n\n>Insufficient comparisons to prior works, e.g. such as [4] but also works like HER\n\nWe implemented Robust Predictable Control (RPC) (e.g. ref [4]) and evaluated it on all the DeepMind Control Suite tasks from the original submission. We found that RPC could not solve the tasks with the same information budget as our other models, and increased the budget to make RPC more performant. Still with this fine-tuning RPC was outperformed by both LZ-SAC or SPAC. Both of our methods are therefore strong alternatives to existing baselines. We note that prior works like HER are orthogonal to the methods proposed in this paper, and can therefore be combined with our approach.\n\n>Some unclear prose, specifically the abstract, introduction and methods section can benefit from less grandous wording\n\nWe have cleaned up the prose at various places throughout the manuscript, including the abstract and introduction. See our changes below. Strikes indicate that the text is removed.\n\n**\"~~Everything else being equal, simpler models should be preferred over more complex ones.~~ In reinforcement learning (RL), simplicity is typically quantified on an action-by-action basis [...]\"**\n\n**\"Simplicity is an ~~powerful~~ important inductive bias. In science, we strive to build parsimonious simple theories and algorithms that involve repetitions of the same basic steps.\"**\n\n**\"~~Though we want our RL agent to maximize rewards, we encourage it to do so with policies that produce simple action sequences.~~ We train our agents to learn policies that maximize reward as well as the compressibility of the action sequences they produce.\"**\n\n>Additional justification of the choice of transformer (as opposed to something simpler) and also some argumentation as to why this cannot be pre-trained ahead of time and then fixed during training\n\nWe would like to thank the reviewer for suggesting pretraining the transformer ahead of time. We trained transformer models to perform next-action prediction from action sequences produced by the converged LZ-SAC agent for all tasks. Using the pretrained transformers (with frozen weights) rather than a randomly initialized one sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Fig 4 in the one-page pdf). This showcases an interesting possible connection between our sequence compression framework and behavioral cloning.\n\nLastly, we added a justification of our use of the transformer rather than a simpler model. While it is perfectly possible to combine our method with a simpler sequence prior, transformers are the SOTA models for learning complex sequence data, and are better suited for modeling long-range dependencies. We have added the following justification to our paper:\n\n**\"We use transformers rather than simpler architectures since they are better suited for  learning complex sequence data with long-range dependencies, as we expect to see in an RL setting.\"**"}]

Author rebuttal:
We would like to thank all of the reviewers for the time and effort put into providing thoughtful feedback on our paper.

* Reviewer teDV found the paper “well-written and presented” and our idea “interesting for the community.”
* Reviewer 6AwJ praised our paper for its “novelty of using LZ4 for a prior and its results.”
* Reviewer GvQF said that the “paper is well written and accessible” and that the “idea of simplicity as a prior is also interesting.”
* Reviewer 7uiK mentioned that our “work proposes an interesting set of evaluations and presents an interesting idea.”

However, at the same time, reviewers also made important suggestions, especially regarding additional baselines and benchmarks. We believe that we were able to incorporate all of these suggestions and believe that doing so has improved our paper significantly. To summarize, we have made the following additions:

* We evaluated our methods on nine additional benchmarks (requested by reviewers teDV, 6AwJ, 7uiK):
- Six pixel-based tasks from the DeepMind Control Suite, highlighting that our approach scales and remains competitive in high-dimensional, visual environments.
- Three robotic manipulation tasks from the Meta-World benchmark, demonstrating applicability to domains that rely less on periodicity.
* We implemented three additional baselines – Robust Predictable Control (RPC), Contrastive Unsupervised Reinforcement Learning (CURL), and Soft Actor Critic + Auto-Encoder (SAC+AE) – and found that LZ-SAC and SPAC remain superior to all of these baselines (reviewers teDV, 7uiK).
* We improved the performance of the SAC baseline by using larger network sizes and lower information costs (reviewers teDV, GvQF). Even with these changes, LZ-SAC remains superior to SAC.
* We ran additional ablations for both SPAC and LZ-SAC (reviewer 6AwJ, GvQF):
- an ablation experiment where we varied the number of heads, number of layers, and embedding dimensions of the transformer SPAC used to predict its actions.
- an ablation experiment where we varied the resolution with which we discretized the action sequences used as input to the compression algorithm.
- we pretrained the transformer model to predict the action sequences of converged policies and trained the SPAC agent using this pretrained transformer which improved performance even further.
* Incorporated references requested by the reviewers (reviewer GvQF).

Each of these changes is outlined in detail in our responses to the individual reviews below. We again want to thank the reviewers for their time and for actively taking part in the review process.


Meta-review: The paper proposes a simple modification to the reward in RL that penalizes the complexity of action sequences, measured either by the log likelihood of the next action prediction (using a transformer sequence model), or by the number of bits to compress the next action using standard compression algorithms. This is shown to perform well and produce smooth/structured action sequences in various RL domains. 

Following the rebuttal, which added more experimental results, all reviewers agreed that the paper should be accepted.
Decision: Accept (poster)