SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Information-regularized policies tend to show stronger robustness to noisy observations [39, 4]: The less an agent's actions vary systematically with the state, the less will a perturbation to the agent's observation affect its actions. We assessed how observation noise affected the agents' ability to collect rewards. In the following experiments, we added Gaussian noise to the observations the policies were conditioned on,  $\mathbf{s}_t\gets \mathbf{s}_t + \epsilon_t$  where  $\epsilon_{t}\sim \mathcal{N}(\mathbf{0},diag(\pmb {\sigma}))$ . We tested the agents on a series of noise levels  $\sigma_{j}\in [0.01,0.05,0.1,0.15,0.2,0.3,0.5]$ . The effect of noise was probed in all tasks except the hopper hop task, since here only the LZ-SAC agent reliably learned a policy that was better than random. Each agent was evaluated using 50 episodes for each noise level.
We evaluated the agents based on how much reward they collected given various levels of observation noise. Averaged over all tasks, the LZ-SAC agent showed the best ability to collect rewards when observations were perturbed with Gaussian noise (see the right panel in Fig. 6). The agents that were better at maximizing rewards showed greater sensitivity to noise: compared to the noise-free setting, LZ-SAC and SAC dropped to  $20\%$  and  $17\%$  of their average performance, respectively. While the
LZ-SAC agent suffered greater percentage drops in return than the MIRACLE and SPAC agents, it still retained the highest performance for all noise levels. In the highest noise settings, SAC is comparable to the MIRACLE and SPAC agents, despite its generally stronger performance in the noise-free setting. This indicates that the LZ-SAC agent performed better in the noisy setting not only because the policy it learned was generally better at maximizing rewards, but also because of robustness properties afforded by the sequence prior.
7 Open-loop control
If simple action sequences are pervasive in policies learned with RL, these priors could provide a good starting point for policy search. To further test this claim, we evaluated how well tasks from the DeepMind Control Suite could be solved by autoregressively generated action sequences from the sequence priors themselves. We omitted RPC from this analysis since it has the same prior policy as SAC. In our experiments, all agents produced the first 15 actions of an episode in a closed-loop manner. We then conditioned the sequence priors with these first 15 actions and sampled actions autoregressively for the remainder of the episode. The priors of the SAC and MIRACLE agents have no autoregressive component, and generated action sequences in a memory-less manner. We approximated samples from the LZ4 prior by discretizing the action space and sampling the next action proportionally to how low its encoding cost is, given the previous actions.
Figure 7: Left: Bars represent return attained in the open-loop phase exclusively. Error bars represent the SEM. The sequence prior learned by the transformer generally performs the best. Notably, the LZ4 prior performs well in tasks solved with periodic action sequences, like cheetah and walker. Right: Average cumulative reward obtained by agents in the cheetah and walker tasks. Dashed lines indicate where the open-loop controls start.
The adaptive prior implemented as a transformer generally performs the best in the open-loop setting (Fig. 7, left). This is expected, as it was trained to predict behaviors that solve the tasks. In the fish swim task a uniform prior collects more rewards in the open-loop phase than the sequences generated by the transformer. However, increasing the number of closed-loop actions used to prompt the transformer to 25 made it surpass the performance of the uniform prior (Appendix E). This points to the importance of providing the sequence models with sufficient context to allow them to accurately predict behavior.
More interesting is the performance of the prior obtained from the LZ4 algorithm. Not only does it perform better than chance, but even comes close to the performance of the learned sequence prior in tasks like cheetah run and walker run. By conditioning on only a few actions from the policy, autoregressively approximating samples from LZ4's prior produced behaviors outperforming the non-sequential priors used by SAC and MIRACLE (Fig. 7, right). This vindicates the compressibility prior as a starting point for policy search.
8 Non-periodic and high-dimensional environments
Many DeepMind Control Suite tasks have solutions that are composed of repeating sub-sequences. Can simple sequence priors be beneficial in tasks without prevalent periodic aspects, or in more complex tasks with high-dimensional state spaces? We first evaluated LZ-SAC against SAC on
Figure 8: Left: Learning curves for three robotic manipulation tasks in the Metaworld benchmark. Lines represent the success rate across 20 test episodes with a deterministic policy, averaged over five agents trained with different seeds. Right: Average return attained in the pixel-based version of the Deepmind Control Suite tasks across 20 test episodes after  $100\mathrm{k}$  environment steps, averaged over six tasks and five seeds. Error bars reflect standard deviation. LZ-SAC with image augmentation outperforms the SAC baseline and two state-of-the-art off-policy methods that combine SAC with representation learning: one using contrastive learning (CURL) and one using an autoencoder (SAC+AE).
three tasks from the Metaworld benchmark [40]. The Metaworld benchmark consists of robotic manipulation tasks where periodic action sequences are less prevalent. Despite this, we found that the simple sequence priors allowed agents to learn policies with higher success rates faster (see Fig. 8, left). In fact, the solutions LZ-SAC developed for the robotics tasks often consisted of single, smooth movements with the Sawyer arm. SAC on the other hand relied on more convoluted movements to manipulate the environment, which were more prone to failure.
Next, we benchmarked LZ-SAC in pixel-based versions of six DeepMind Control Suite tasks. We trained LZ-SAC and a SAC baseline on 100k environment steps, and modified both algorithms with a convolutional neural network encoder and performed a random shift augmentation to images before training the actor and critic. With this simple modification and our compression bonus, LZ-SAC outperformed state-of-the-art off-policy algorithms like CURL [41] and SAC+AE [36] on average over the six tasks (see Fig. 8, right). Across five seeds, LZ-SAC outperformed the three baseline models on three out of the six tasks. See Appendix C.1 for the full scores and implementation details.
We have argued that simplicity is a powerful principle to guide policy search in RL tasks. Because control problems are often solved with sequences of actions that contain repeating temporal patterns, we proposed to use simple sequence priors to create effective and robust RL agents. To provide agents with a notion of compressibility, we proposed two models: One where the strategy used for compression was fixed throughout training (LZ-SAC), and one where the strategy itself could change with experience (SPAC). While the LZ-SAC agents either outperformed or matched the performance of state-of-the-art methods like SAC, the SPAC agents learned more compressible strategies, attaining more rewards while using fewer bits of information to make a decision. Furthermore, agents trained with the LZ-SAC algorithm proved to be the most robust to observation noise. Lastly, both the trained transformer model and the prior distilled from the LZ4 algorithm could autoregressively generate rewarding behaviors in continuous control tasks.
While SPAC showed a better ability to maximize rewards than MIRACLE, returns were lower than SAC and our alternative regularization technique. This is not unexpected. The transformer always required some amount of learning to be able to predict a particular action sequence. The LZ4 algorithm, on the other hand, could immediately provide feedback about the compressibility of the agent's action sequences without any learning. For SPAC, having to learn a sequence prior induced a stronger bottleneck, resulting in more compressed policies. This is consistent with results reported by Eysenbach et al. [4], where a learned dynamics model was used to compress sequences of states: Here compression with a learned prior led to lower returns, but a higher return per bit rate. Our results suggest that sequence compression based on off-the-shelf compression algorithms like LZ4 are better for policy search since there is no need for learning a sequence prior from scratch.
Limitations: Action sequence compression requires either an adaptive prior, a neural sequence model, or a pre-programmed compression algorithm. The particular algorithm used for compression adds computational overhead and determines the types of action sequences that will be favored by the agent [32]. Future work should address the ways in which different compression algorithms or sequence priors affect policy regularization. Furthermore, a sufficiently sophisticated sequence model could in principle learn to predict complex action sequences. A possible extension of our work could be to further penalize the description length of the weights of the sequence model, or the compression algorithm, itself [42]. Finally, while we evaluated our algorithm on a large and diverse set of control tasks within the DeepMind Control Suite and Metaworld, the utility of simple sequence priors could be tested on other benchmarks. In discrete action settings, Atari games [43] would be an appropriate benchmark.
Future Directions: A central feature of simple action sequences is that they are predictable. Being able to predict one's future behavior from past behavior could allow agents to simplify and compress their representations of the state of the world [11, 4]: If the point of observing the state is to determine what action to choose, one could discard information about the state of the world simply by considering the actions that were performed previously. This suggests that simple sequence priors could be beneficial for compressing policies and internal representations jointly.
Finally, humans show a preference for simplicity and compressibility in various domains [30, 2]: We not only produce art and music full of patterns and regularity [44], but also explore novel environments using compressible trajectories [45], and rely on simple rules to explain and generalize about complex stimuli relationships [46-48]. Recently, dopamine activity in the tail of the mouse striatum was argued to encode an action prediction error signal [49]. Such a signal also features in our augmented reward function to compress the policy. In the end, our algorithms could therefore serve as models of how biological agents learn compressible sequential strategies from reinforcement.
We thank the four anonymous reviewers for the instructive and helpful feedback during the review process, as well as the members of the Computational Principles of Intelligence Lab for feedback provided throughout the project. We also thank Can Demircan for providing comments on an earlier draft. This work was supported by the Max Planck Society, the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, and funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy-EXC2064/1-390727645.
[1] Carl Rasmussen and Zoubin Ghahramani. Occam's razor. Advances in neural information processing systems, 13, 2000.
[2] Nick Chater and Paul Vitányi. Simplicity: a unifying principle in cognitive science? Trends in cognitive sciences, 7(1):19-22, 2003.
[3] Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7(1): 1-22, 1964.
[4] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems, 34:27813-27825, 2021.
[5] Pedro A Ortega and Daniel A Braun. Thermodynamics as a theory of decision-making with information-processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469(2153):20120683, 2013.
[6] Felix Leibfried and Jordi Grau-Moya. Mutual-information regularization in markov decision processes and actor-critic learning. In Conference on Robot Learning, pages 360-373. PMLR, 2020.
[7] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
[8] Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[10] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[11] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
[12] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020.
[13] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaii, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[14] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.
[15] Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory, pages 25-55. PMLR, 2018.
[16] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (itw), pages 1-5. IEEE, 2015.
[17] Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perception-action cycle: Models, architectures, and hardware, pages 601-636. Springer, 2010.
[18] Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-information regularization. In International conference on learning representations, 2018.
[19] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[54] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[55] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Our algorithm is an extension of the Soft Actor-Critic algorithm [7, 50], implemented in PyTorch. Like [36, 51], we initialize agents' replay buffer with a 1000 seed observations collected with a uniform random policy. We update the Q-network pairs to predict the augmented  $Q$  value function at every interaction step, and the actor network to maximize the augmented  $Q$  value function every second interaction step. The augmented  $Q$  targets take the following form:
$$
y = r \left(\mathbf {s} _ {t}, \mathbf {a} _ {t}\right) + \gamma \left[ Q \left(\mathbf {s} _ {t + 1}, \mathbf {a} _ {t + 1}\right) - \alpha \left(\log \pi_ {\theta} \left(\mathbf {a} _ {t + 1} \mid \mathbf {s} _ {t + 1}\right) - \log \phi_ {\theta} \left(\mathbf {a} _ {t + 1} \mid \mathbf {a} _ {t - \tau ; t}\right)\right) \right] \tag {6}
$$
where  $\mathbf{a}_{t + 1} \sim \pi_{\theta}(\cdot | \mathbf{s}_{t + 1})$ . To train the networks we sampled  $B$  tuples of state, actions, next state, reward and terminal flags, as well as the  $\tau$  actions that led to them. To allow the agent to train on observations early in episodes, we sampled  $\tau$  from a uniform distribution of integers between 5 and  $\tau_{\mathrm{max}}$  (see tables 1, 2) for every mini-batch sample used for training. We update the adaptive priors used in SPAC and MIRACLE together with the actor network. All actor and critic networks consisted of two hidden layers with 256 ReLU units [52] each. The action prior used in MIRACLE was implemented as a multivariate isotropic Gaussian with learnable mean and standard deviation.
Since actions are bounded between -1 and 1, we transform actions sampled from the policy using the tanh transform  $\mathbf{a}_t = \tanh (\mathbf{u}_t)$ ,  $\mathbf{u}_t\sim \pi_\theta$ . We transformed the log-likelihood of an action under a Gaussian policy  $\pi_{\theta}$  or action prior  $\phi_{\theta}$  using the following formula [7, 50]:
$$
\log \pi_ {\theta} (\mathbf {a} _ {t} | \mathbf {s} _ {t}) = \log \mu (\mathbf {u} _ {t} | \mathbf {s} _ {t}) - \sum_ {i = 1} ^ {D} \log \left(1 - \tanh  ^ {2} \left(u _ {i}\right)\right) \tag {7}
$$
$$
\log \phi_ {\theta} \left(\mathbf {a} _ {t} \mid \mathbf {a} _ {t - \tau : t - 1}\right) = \log \psi_ {\theta} \left(\mathbf {u} _ {t} \mid \mathbf {a} _ {t - \tau : t - 1}\right) - \sum_ {i = 1} ^ {D} \log \left(1 - \tanh  ^ {2} \left(u _ {i}\right)\right) \tag {8}
$$
where  $\log \psi_{\theta}(\mathbf{u}_t|\mathbf{a}_{t - \tau ;t - 1})$  is the log likelihood of the untransformed action  $\mathbf{u}_t$  under the untransformed sequence prior  $\psi_{\theta}$ .
A.1 Quantifying compressibility
We used the L24 algorithm to quantify the compressibility of action sequences. The following code snippet describes how we computed the sequence complexity term in Eq. 3
sequence_i = action_sequence.numpy().ravel()
# get length of compressed sequence at t
length_t1 = len(compression_algorithm.compress(sequence_i))
action_next = policy(state).numpy().ravel() # get next on-policy action
sequence_j = np.concatenate((sequence_i, action_next), axis=0)
# get length of compressed sequence at t+1
length_t2 = len(compression_algorithm.compress(sequence_j))
delta = length_t - length_t2 # delta is the difference
Since the actions were continuous vectors, we quantized all action sequences with the following function:
def quantify(action_sequence, N=100):
    return (action_sequence*N).floor()
Here  $N$  determines the granularity of the quantization, with lower  $N$  producing more coarse-grained sequences. We set  $N = 100$  for our experiments.
A.2 Pseudo-code for 1Z4 algorithm
The 1Z4 algorithm compresses sequences by replacing repeating sub-sequences in the data with references to an earlier occurring copy of the sub-sequence. These copies are maintained in a sliding
window. Repeating sub-sequences are encoded as length-distance pairs  $(l, d)$ , specifying that a set of  $l$  symbols have a match  $d$  symbols back in the uncompressed sequence. The following pseudo-code sketches compression implemented by LZ4 [32]:
Algorithm 1 LZ4 pseudo-code
Require: Buffer size  $b$  window size  $w$  , sequence k   
 $t = 0$    
window  $\leftarrow \langle \right\rangle$    
while t < len(k) do match  $\leftarrow$  longest repeated occurrence in window found in  $\mathbf{k}_{t:t + b}$  if match exists then  $d\gets$  distance to start of match  $l\gets$  length of match  $c\gets$  symbol at  $\mathbf{k}_{t + l}$  else  $d\gets 0$ $l\gets 0$ $c\gets 0$    
end if   
output  $(d,l,c)$    
start  $\leftarrow$  max  $(t - w + l,0)$    
end  $t\gets t + l$    
window  $\leftarrow \mathbf{k}_{\mathrm{start:end}}$ $t\gets t + l + 1$    
end while
B.1 Increasing reward scale
Figure 9: Halving the incentive of acting randomly does not close the performance gap between LZ-SAC and SAC.
We investigated whether LZ-SAC's performance improvement could simply be attributed to the incentive to act more deterministically. We tested whether we could attain the same level of performance with SAC just by lowering the incentive of acting randomly. Doubling the scale of extrinsic reward relative to the intrinsic reward of acting randomly did not close the gap between the algorithms (Fig. 9). Instead, we see a decline in performance when we set the incentive of randomness lower than  $\alpha = 0.1$  (or  $\alpha = 0.02$  in walker run). This indicates that there is value in having a preference for simplicity on the sequence level that goes beyond simply being predictable at the level of individual actions.
B.2 Algorithm hyperparameters
We implement all algorithms using hyperparameters from [36], with slight deviations depending on the task. Since the computational overhead of using 1z4 as a compressor is small compared to the
transformer, we train the agents using larger action sequences. All networks were trained with the Adam optimizer [53]. A full list of hyperparameters is given below:
Table 1: Hyperparameters used for SAC, MIRACLE, LZ-SAC, and SPAC
Hyperparameter	Value
Complexity cost α (walker run, hopper hop, quadruped walk)	0.02
Complexity cost (all other environments) α	0.1
Discount γ	0.99
Critic update frequency	1
Actor update frequency	2
Action prior update frequency	2
Soft update ρ	0.01
Batch size	128
Learning rate actor	10-3
Learning rate critic	10-3
Optimizer	Adam
Max context length LZ-SAC (τmax)	Interaction steps in episode × 0.4
Max context length LZ-SAC (τmax; walker run)	Interaction steps in episode × 0.25
B.3 Transformer
The SPAC agent uses a causal transformer [9] to learn a prior over action sequences. Our transformer was implemented with the following hyperparameters:
Table 2: Transformer hyperparameters.
Hyperparameter	Value
Attention heads	5
Embedding dimensions	30
Learning rate decay	Linear
Warmup tokens	10000
Max context length (τmax)	20
Number of layers	2
Learning rate	3 × 10-4
Dropout	0.1
Optimizer	Adam
C Task specification
We evaluated agents on tasks from the DeepMind Control Suite. Though dynamics are otherwise deterministic, the starting state of an episode is sampled from a distribution  $p(\mathbf{s}_0)$ . All episodes consist of 1000 environment steps. However, in practice the episode length is reduced to a number of interaction steps, that is smaller than 1000. This is due to an action repeat hyperparameter which determines how many times an action  $\mathbf{a}_t$  is repeated after it is selected. An action repeat value of 4 thus reduces the number of time steps where the agent needs to act to 250 interaction steps. The action repeat hyperparameter makes it more practical to train agents in the DeepMind Control Suite [34]. We adopt conventional action repeat settings from the literature [36]. In the walker and hopper domains we fitted the action repeat value for all agents among [2, 4, 8] and chose the value that produced the best performance. Table 3 shows the action repeat values used in our experiments:
C.1 Pixel-based control
Our LZ-SAC and SAC implementations in the visual control domain differed little from the state-based implementations. We equipped the agents with the convolutional neural network architecture and image augmentation transformation from [54]. All MLPs had two hidden
Table 3: Action repeat values.
Task	Action repeat
acrobot swingup	8
cheetah run	4
fish swim	4
hopper hop	8
hopper stand	8
quadruped walk	4
reacher hard	4
walker run	2
walker run (SPAC)	4
Table 4: All final average model scores in the DeepMind Control 100k benchmark with pixel observations. Scores are averaged over 10 runs after 100k steps for five seeds. Scores of the other baselines are the ones reported in the respective papers [41] [36].
DMC 100k	LZ-SAC	SAC	CURL	SAC+AE
Finger, Spin	814	738	767	740
Cartpole, Swingup	683	609	582	311
Walker, Walk	635	609	403	394
Ball In Cup, Catch	653	499	769	391
Cheetah, Run	307	396	299	274
Reacher, Easy	513	427	538	274