SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tankred Saanum $^{1\dagger}$
Noémi Éltetó¹
Peter Dayan $^{1,2}$
Marcel Binz
Eric Schulz
$^{1}$ Max Planck Institute for Biological Cybernetics,  $^{2}$ University of Tübingen†tankred.saanum@tuebingen.mpg.de
In reinforcement learning (RL), simplicity is typically quantified on an action-by-action basis – but this timescale ignores temporal regularities, like repetitions, often present in sequential strategies. We therefore propose an RL algorithm that learns to solve tasks with sequences of actions that are compressible. We explore two possible sources of simple action sequences: Sequences that can be learned by autoregressive models, and sequences that are compressible with off-the-shelf data compression algorithms. Distilling these preferences into sequence priors, we derive a novel information-theoretic objective that incentivizes agents to learn policies that maximize rewards while conforming to these priors. We show that the resulting RL algorithm leads to faster learning, and attains higher returns than state-of-the-art model-free approaches in a series of continuous control tasks from the DeepMind Control Suite. These priors also produce a powerful information-regularized agent that is robust to noisy observations and can perform open-loop control.
Figure 1: Action sequences produced by a bipedal walker become more compressible with learning. Our algorithm learns policies that solve tasks with simple action sequences, leading to decreased complexity and higher returns.
Simplicity is a powerful inductive bias [1-3]. In science, we strive to build parsimonious theories and algorithms that involve repetitions of the same basic steps. Simplicity is also important in the context of reinforcement learning (RL). Policies that are simple are often easier to execute, and practical to implement even with limited computational resources [4, 5]. Many control problems have solutions that are compressible: Motor behaviors like running and walking involve moving our legs in a periodic, alternating fashion (Fig. 1). Here it is the sequence of actions selected that is compressible. Sequences with repetitive, periodic elements are easier to predict and can be compressed more than sequences that lack such structure. In the current work, we augment RL agents with a prior that their action sequences should be simple: If solutions to control problems are generally compressible, one should consider only the set of simple solutions to a problem rather than the set
of all solutions. In a series of experiments, we show that RL with simple sequence priors produces policies that perform better and more robustly than state-of-the-art approaches without such priors.
Figure 2: Top left: Policy regularization either incentivizes sequences or individual actions to be close to the prior. Priors may be distinct in that they stay fixed over training or change from episode to episode with learning. Top right: Agents need to navigate to a goal location, where the shortest path requires fine control, following a repeating pattern. After learning, SAC randomly diffuses among multiple paths. MIRACLE prefers a simple path that only goes up and then to the right. Since the optimal path is compressible (repeating UP and RIGHT in a periodic fashion), the agents with the simple sequence priors prefer this path. Bottom: An agent with simple sequence priors, in this case SPAC, learns simple strategies for walking, using mostly the left leg to push itself forward in a repetitive fashion.
Though there are methods for regularizing policies with respect to the individual actions they produce [6, 7], we present a method that explicitly regularizes the sequences of actions used to solve a task. Our regularization incentivizes the agent to use action sequences that can be compressed with a sequence prior. If an action sequence is likely under the prior, one needs fewer bits of information to represent it [8]. We explore two types of sequence priors:  $i$ ) Priors in the form of an autoregressive sequence model [9, 10] that learns to predict future actions based on actions that were performed in the past and  $ii$ ) priors distilled from a pre-programmed, lossless compression algorithm. Building on the Soft Actor-Critic algorithm (SAC) [7], we introduce Lempel-Ziv Soft Actor-Critic (LZ-SAC), using an off-the-shelf compression algorithm as its prior, and Soft Predictable Actor-Critic (SPAC), using a learned sequence prior (Fig. 2).
The contributions of this paper are the following: We introduce a model-free RL algorithm for maximizing rewards with simple action sequences. In a series of continuous control tasks, we evaluate the utility of such simple sequence priors. First, we investigate whether simple sequence priors speed up policy search: In our experiments, agents with simple sequence priors consistently outperform state-of-the-art model-free RL algorithms in terms of reward maximization. This holds both in terms of learning speed and often in the final performance. Our second result is that our regularization produces an information-efficient RL agent, using fewer bits of information to solve control problems. Information-regularized models are more robust and better at generalizing [11, 4, 12]. Lastly, we demonstrate the agents' advantages in environments with noisy and missing observations.<sup>1</sup>
The idea of simplicity has received significant attention in previous work. Maximum entropy RL, for instance, augments the reward function with an entropy maximization term, effectively encouraging the agent to stay close to a simple uniform prior policy over actions [13, 14]. Though uniform priors can lead to discontinuous and unpredictable behaviors, maximum entropy methods are considered simple in that they try to minimize the use of information about the state to select actions [4, 15, 16]. Many current approaches to deep RL – such as SAC [7] – rely on this principle. This concept has been further extended by models like Mutual Information Regularized Actor-Critic Learning (MIRACLE)
For videos showing behaviors learned with our algorithm, see our project website:
https://sequenceprioris.github.io
Code: https://github.com/tankred-saanum/simple_priors
[6] and others [17, 18], which use a learnable state-independent prior policy instead of the uniform prior assumed by SAC. SAC and MIRACLE both induce simplicity at the level of individual actions. In contrast, our proposed approach works on the level of action sequences.
It is not only possible to encode preferences for simplicity at the action level. Instead, simplicity can also be imposed by encouraging the agent to maintain simple internal representations – the core idea behind the information bottleneck principle [19]. Deep RL agents that rely on this principle have many appealing properties, such as improved robustness to noise, better generalization, and more efficient exploration characteristics [20–22]. Related to compression is predictability: Berseth et al. [23] learn a density model over states, and then learn a policy that seeks out states that are predictable, leading to self-sustaining behaviors in unstable environments. On the opposite end there are methods that seek out unpredictable states [24, 25], or states that the agent cannot compress, to improve exploration. Recently, [4] demonstrated how to construct RL agents that learn policies that use few bits of information by not only compressing individual observations but entire sequences of observations. In some sense, our approach can be seen as a variant of the algorithm from [4]. However, we compress sequences of actions, rather than sequences of observations. Thus, our regularization does not target the complexity of the sequence of internal representations, but instead the complexity of the agent's behavior, manifested in the sequence of actions selected to solve a task.
Finally, simplicity is also an important feature of natural intelligence, where it has been repeatedly argued that simplicity is a unifying principle of human cognition [2]. For instance, [26] showed that people rely on compressed policies, ultimately leading to behavioral effects such as preservation or chunking [27, 28]. Likewise, [29] demonstrated that human exploration behavior can be described by RL algorithms with limited description length, while [30] showed that compression captures human behavior in a visual search task.
3 Control with simple sequences
In this section, we demonstrate how to construct RL agents that solve tasks using simple action sequences. We start by outlining the general problem formulation. We assume that the task can be posed as a Markov Decision Process (MDP). The MDP consists of a state space  $\mathbf{s} \in S$ , an action space  $\mathbf{a} \in \mathcal{A}$ , and environment dynamics  $p(\mathbf{s}_0)$  and  $p(\mathbf{s}_{t+1} \mid \mathbf{s}_t, \mathbf{a}_t)$ . The dynamics determine the probability of an episode starting in a particular state and the probability of the next state given the previous state and action, respectively. Lastly, there is the discount factor  $\gamma$  and a reward function  $r(\mathbf{s}_t, \mathbf{a}_t)$  that maps state-action pairs to a scalar reward term. The agent learns a policy  $\pi_\theta(\mathbf{a}_t | \mathbf{s}_t)$  parameterized by  $\theta$  that maps states to actions in a way that maximizes the sum of discounted rewards  $\mathbb{E}_{\pi_\theta}\left[\sum_{t=1}^T \gamma^t r(\mathbf{s}_t, \mathbf{a}_t)\right]$ .
Though we want our RL agent to maximize rewards, we encourage it to do so with policies that produce simple action sequences. Inspired by previous approaches, we achieve this by augmenting the agent's objective [7, 4, 14], and search for a set of policy parameters  $\theta$  that maximize reward while minimizing the complexity of the policy  $C(\mathbf{a}_{t - \tau :t},\mathbf{s}_t,\theta)$ :
$$
\max  _ {\theta} \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 1} ^ {T} \gamma^ {t} \left(r \left(\mathbf {s} _ {t}, \mathbf {a} _ {t}\right) - \alpha \underbrace {C \left(\mathbf {a} _ {t - \tau : t} , \mathbf {s} _ {t} , \theta\right)} _ {\text {C o m p l e x i t y c o s t}}\right) \right] \tag {1}
$$
where the hyper-parameter  $\alpha$  controls the trade-off between complexity and discounted rewards.
We can recover various previous approaches using this formulation. If we, for instance, set  $C(\mathbf{a}_{t - \tau :t},\mathbf{s}_t,\theta) = \log \pi_\theta (\mathbf{a}_t\mid \mathbf{s}_t)$ , we obtain maximum entropy RL algorithms such as SAC. SAC implicitly assumes a uniform prior over individual actions. An alternative to using the uniform prior in maximum entropy RL is to learn a parameterized prior over actions  $p_{\theta}(\mathbf{a})$  based on the empirical distribution of actions the agent selects when solving the task [18, 6]. Setting  $C(\mathbf{a}_{t - \tau :t},\mathbf{s}_t,\theta) = \log \pi_\theta (\mathbf{a}_t\mid \mathbf{s}_t) - \log p_\theta (\mathbf{a}_t)$ , we obtain MIRACLE.
3.1 Simplicity with learned priors
While both SAC and MIRACLE compress sums of individual actions, they do not account for the structure that is present in whole action sequences. To close this gap, we present two methods for regularizing policies on the level of action sequences. For the first, we train a prior distribution  $\phi_{\theta}(\mathbf{a}_t \mid \mathbf{a}_{t - \tau : t - 1})$  to predict the agent's future actions from actions it performed in the past. We parameterize the prior as a neural sequence model. We use a causal transformer model [9, 31] to parameterize  $\phi_{\theta}$ . Though any type of sequence model could be used in principle, Transformers are arguably better suited for learning complex sequence data with long-range dependencies. We can augment the reward function to incorporate the preference for predictable action sequences as follows:
$$
\tilde {r} \left(\mathbf {s} _ {t}, \mathbf {a} _ {t - \tau : t}\right) = r \left(\mathbf {s} _ {t}, \mathbf {a} _ {t}\right) - \alpha \left(\log \pi_ {\theta} \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right) - \log \phi_ {\theta} \left(\mathbf {a} _ {t} \mid \mathbf {a} _ {t - \tau : t - 1}\right)\right) \tag {2}
$$
where  $\mathbf{a}_{t - \tau ;t - 1}$  is a sequence of the last  $\tau$  actions. Optimizing this objective, the agent will get rewarded for performing behaviors that the sequence model can predict better. The sequence model can learn to predict action sequences more easily if they contain structure and regularity. This has two interesting implications. i) The agent is incentivized to visit states where its actions will be predictable, for instance by oscillating between states in a periodic manner. ii) To perform actions that make it easier for the sequence model to predict future actions, for instance by performing behaviors that signal to the sequence model how it will behave in the future. We refer to this agent as the Soft Predictable Actor-Critic agent, or SPAC.
3.2 Simplicity with compression algorithms
Since the sequence model and the policy are adapting their behavior and prior towards each other, the augmented reward function will change throughout training. This plasticity can make it challenging to search for viable policies. Moreover, training a sequence model on top of the RL agent creates additional computational overhead. We, therefore, explore the possibility of instilling a simplicity preference without the use of a sequence prior that necessarily adapts over episodes.
This second method for distilling simple sequence priors relies on off-the-shelf data compression algorithms [32]. Lossless data compression algorithms like LZ4, bzip2 and zlib encode data into sequences of symbols from which the original data can be reconstructed or decompressed exactly. If there are repetitions, regularities, or periodicity in the data, the length of the encoded sequence can be significantly shorter than the original size of the data (Fig. 3). Relying on pre-programmed rules for data compression, this simplicity prior will not change over the course of training. Since compression
algorithms like LZ4 are fast, the sequence prior can be implemented with little computational overhead.
In this setting, we compute  $C$  using the extra number of bits needed to encode  $\mathbf{a}_t$  given that we have already encoded  $\mathbf{a}_{t - \tau :t - 1}$ :
$$
\delta_ {t} = \operatorname {l e n} \left(g \left(\mathbf {a} _ {t - \tau : t - 1}\right)\right) - \operatorname {l e n} \left(g \left(\mathbf {a} _ {t - \tau : t}\right)\right) \tag {3}
$$
$$
\tilde {r} \left(\mathbf {s} _ {t}, \mathbf {a} _ {t - \tau : t}\right) = r \left(\mathbf {s} _ {t}, \mathbf {a} _ {t}\right) - \alpha \left(\log \pi_ {\theta} \left(\mathbf {a} _ {t} \mid \mathbf {s} _ {t}\right) - \delta_ {t}\right) \tag {4}
$$
Figure 3: Some sequences are more compressible than others. A sequence of randomly generated numbers is less compressible than sequences with periodicity, sequences that only contain two types of values (also known as Bang-Bang control), or constant sequences that only contain a single number.
or can be implemented with little computational
where  $g(\cdot)$  is our compression function and  $\mathrm{len}(\cdot)$  returns the length of a sequence. We use the LZ4 compression algorithm to compute the augmented rewards and refer to this agent as the LZ-SAC agent.
3.3 Implementational details
We implement all agents as extensions of the SAC algorithm. SAC is an off-policy actor-critic algorithm that performs maximum entropy RL. We train critics to learn the augmented  $Q$ -value function  $\tilde{Q}(\mathbf{s}_t, \mathbf{a}_{t - \tau:t}) = \mathbb{E}\left[\sum_{t=1}^N \gamma^t \tilde{r}(\mathbf{s}_t, \mathbf{a}_{t - \tau:t})\right]$  with temporal-difference learning [33]. The actors and sequence models are trained to minimize the same loss:
$$
\mathcal {L} = \mathbb {E} _ {\mathbf {s} _ {t}, \mathbf {a} _ {t - \tau : t} \sim \mathcal {D}} [ \alpha (\log \pi_ {\theta} (\mathbf {a} _ {t} \mid \mathbf {s} _ {t}) - \log \phi_ {\theta} (\mathbf {a} _ {t} \mid \mathbf {a} _ {t - \tau : t - 1})) - \tilde {Q} (\mathbf {s} _ {t}, \mathbf {a} _ {t - \tau : t}) ] \tag {5}
$$
where  $\mathcal{D}$  is a replay buffer and  $\mathbf{a}_t\sim \pi_\theta (\cdot \mid \mathbf{s}_t)$ . The LZ-SAC actor minimizes the same loss except that  $-\log \phi_{\theta}(\mathbf{a}_{t}\mid \mathbf{a}_{t - \tau ;t - 1})$  is replaced with the term in Eq. 3. In practice, we take the minimum of two target  $Q$ -networks to train the actor and critic. Learning is achieved by sampling experiences from a replay buffer. To calculate the augmented rewards, we simply sample action sequences  $\mathbf{a}_{t - \tau ;t - 1}$  that led to the  $(\mathbf{s}_t,\mathbf{a}_t,\mathbf{s}_{t + 1},r_t)$  tuple used for training (see Appendix A for full implementation details).
4 Simple sequence priors guide policy search
Figure 4: Learning curves of agents in the DeepMind Control suite. Overall, LZ-SAC shows the best learning speed and final performance. Lines are the average episodic returns collected in 20 test episodes with a deterministic policy, averaged over five agents trained with different seeds. Shaded regions represent 20-80 performance percentiles.
We evaluated the agents described in Section 3 on eight continuous control tasks from the DeepMind Control Suite [34]. As an additional baseline we included Robust Predictable Control (RPC) from [4], which compresses sequences of states rather than actions. Many of the tasks in the DeepMind Control Suite promote behaviors with periodic action sequences, such as running and walking. While specialized architectures exist for such tasks [35], we expect compressibility to be a useful inductive bias for learning these behaviors. We trained agents for 1 million environment steps across five seeds and evaluated their abilities at regular intervals with a deterministic policy, as in [7, 36]. We tuned  $\alpha$  for each agent and found an  $\alpha = 0.1$  to give the best performance in almost all tasks. We found lower information costs  $\alpha$  worked better for RPC. Tasks and hyperparameter fitting is described in Appendix B.
Figure 5: Left: LZ-SAC works best when the discretization resolution allows the agent to distinguish between compressible and incompressible sequences. Bars show mean return and SEM from 20 evaluation episodes over three seeds and three environments after 300k steps. Right: Time series of actions produced by the LZ-SAC, SPAC and SAC agent in the walker run task. Action time series of the SPAC agent exhibits a simpler periodic pattern, even outputting a constant value for its third actuator. Actuators were chosen to show qualitatively different behaviors.
In a majority of the tasks, the LZ-SAC agent outperformed the SAC, RPC and MIRACLE agents in learning speed and often final performance (Fig. 4). At worst, the LZ-SAC agent matched the learning curves of SAC. This suggests that learning policies with simple sequence priors is indeed fruitful for policy search. We investigated whether this performance difference could simply be attributed to LZ-SAC acting more deterministically than SAC. Lowering the incentive of acting randomly for SAC did not close the performance gap, and often led to worse returns (see Appendix B.1).
We conducted an additional ablation experiment to make sure the performance gain could be attributed to the compressibility incentive: We varied the resolution with which we discretized the action sequences used as input to the compression algorithm when LZ-SAC was trained. Across three DeepMind Control tasks (cheetah run,acrobot swingup and walker walk), we observe that both too low and too high resolutions remove the performance gain of LZ-SAC: When the resolution is 0, the compression bonus no longer conveys a signal about the simplicity of the policy. Conversely, if the resolution is too high, every action sequence is equally incompressible due to the continuous nature of the action space. We find that rounding to two decimal places gave the best performance on average across the tasks (see Fig 5, left). Though we do not expect LZ-SAC to always outperform SAC in more generic control settings, we see an improvement in many tasks with periodic elements, like walking and running. We evaluate LZ-SAC in non-periodic tasks in Section 8.
In two tasks, acrobot swingup and fish swim, the SPAC agent showed a competitive advantage over the other models. However, the SPAC agent lagged behind both the LZ-SAC agent and SAC agent in tasks from the hopper, cheetah, and walker domains. Here the policy that the SPAC agent learned achieves roughly  $75\%$  of the return of the LZ-SAC agent.
The policies learned by the SPAC agent shine in a different setting: The agent has discovered solutions to these tasks that essentially use fewer action dimensions than the competitors (Fig. 5, right): For certain actuators  $a_{i}$ , the agent outputs a constant value throughout the episodes. For other actuators, the agent alternates between two extreme values, like a soft bang-bang controller [37, 38]. Essentially, the SPAC agent figures out which degrees of freedom it can eliminate without jettisoning rewards. Having fewer degrees of freedom makes it easier to predict the action sequences produced by the policy. This suggests that policy compression using adaptive sequence priors is better suited in tasks with low-dimensional action spaces. Lastly, the difficulties of learning a policy and a sequence prior jointly can be mitigated by using a pre-trained sequence model as a prior. We pre-trained Transformer models to predict action sequences produced by the converged LZ-SAC agents in all eight control tasks. Using the pre-trained Transformers with frozen weights sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Appendix G).
Figure 6: Left: Normalized return per bit attained by the agents in the eight tasks. Agents with simple sequence priors achieve better return per bit ratios. Error bars represent the standard error of the mean (SEM). Right: Normalized return averaged over all tasks as a function of noise scale. Error bands represent the SEM.
5 Simple sequence priors for information-regularized RL
The expected difference in log-likelihood of the agents' actions under the policy versus the prior is an upper bound on the mutual information between states and actions [4, 11, 19]. Encouraging this difference to be low acts as an information-regularizer, the prior  $p(\mathbf{a}_t|\mathbf{a}_{t - \tau ;t - 1})$  being the information bottleneck. We tested the information-efficiency of learned policies; that is, how much reward the agents could collect relative to the information they used to make decisions. For the experiments, we again tested the deterministic versions of the agents. Simulating 25 episodes, we computed how much reward the agents were able to collect divided by the entropy of the distribution of actions used to solve the task  $\mathbb{E}\left[\frac{\sum_{t = 1}^{T}r_t}{\mathcal{H}[\mathbf{a}]}\right]$  (see Appendix D.1 for details and experiments with stochastic policies). Since the policies were deterministic, this entropy term approximated the mutual information between states and actions  $I(\mathbf{s};\mathbf{a})$  (see Appendix D). In the left panel of Fig. 6, we show the normalized episodic return per bit. This quantity represents how much reward the agent attains per bit of information it uses on average to make a decision over the course of the episode.
The SPAC agent attained a superior return per bit ratio in five out of eight tasks. LZ-SAC attained the highest return per bit ratio in two tasks, and SAC in one. This indicates that action sequence compression is a powerful information-regularizer, allowing agents to find policies that use significantly fewer bits of information to collect reward than both policy compression models (SAC and MIRACLE), and state sequence compression models (RPC).
6 Robustness to noise