SlowGuess/NeurIPS_2023_OCR huggingface dataset:
layers and 512 ReLU units. We optimized the  $\alpha$  hyperparameter both for the LZ-SAC and SAC agents for all tasks with a grid search. For tasks with higher dimensional action spaces a lower  $\alpha$  of 0.01 worked best. In the end, the best performing  $\alpha$  for both algorithms was the same across task. Since LZ4 encoding lengths are not necessarily on the same scale as the log likelihoods of the actions given the state in Equation 4, we experimented with scaling the LZ4 encoding cost by 0.5, which slightly improved performance. The full scores of the models are in Table 4.
Table 5: Action repeat values and complexity cost for both LZ-SAC and SAC agents in the pixel-based tasks.
Task	Action repeat	α
finger spin	2	0.01
cartpole swingup	8	0.1
walker walk	2	0.01
ball in cup catch	4	0.01
cheetah run	4	0.01
reacher easy	2	0.1
D Mutual information approximation
The mutual information  $I(X;Y)$  between variables  $X$  and  $Y$  is a measure of how much they depend on each other. In our case we are interested in the mutual information between states and actions  $I(\mathbf{s};\mathbf{a})$ . The mutual information here quantifies how many bits of information knowing the outcome of the random variable  $\mathbf{s}_t$  provides about the other random variable  $\mathbf{a}_t$ , in other words, how much the state reveals about what action will be selected. The more an agent's actions vary as a function of the state, the more bits of information the state reveals about the action that the agent will select.
The mutual information is defined as the following quantity
$$
I (\mathbf {a}; \mathbf {s}) = \mathcal {H} [ \mathbf {a} ] - \mathcal {H} [ \mathbf {a} | \mathbf {s} ] \tag {9}
$$
Figure 10: Return per bit for the stochastic policies. LZ-SAC attains the highest return per bit ratio in most tasks.
To compute the mutual information we need to know the entropy of the distribution of actions used to solve the task, and the conditional entropy of  $\mathbf{a}|\mathbf{s}$ . Since we make the policies deterministic, we know that  $\mathcal{H}[\mathbf{a}|\mathbf{s}] = 0$ . This way the mutual information reduces to the entropy over actions  $\mathcal{H}[\mathbf{a}]$ . Given a sample of actions produced by the agent solving the task, we approximate  $\mathcal{H}[\mathbf{a}]$  the following way: We first quantized each selected action into  $100 \times |\mathcal{A}|$  bins, where  $\mathcal{A}$  is the action space. We then calculated a categorical distribution over actions based on the frequencies of the quantized actions, the entropy of which we used as our approximation for  $\mathcal{H}[\mathbf{a}]$ . The categorical distribution was calculated based on actions selected over 50 episodes.
D.1 Return per bit for stochastic policies
We evaluated the information efficiency of the stochastic variants of the policies learned by LZ-SAC, SPAC, SAC and MIRACLE. We approximated the entropy of the distribution of actions in the same way described above, sampling actions over 50 episodes. To compute the conditional entropy of actions given the state  $\mathcal{H}[\mathbf{a}|\mathbf{s}]$ , we sampled 1000 actions from the policy at every state  $\mathbf{s}_t$ . We then calculated a categorical distribution (described in previous section) based on this sample, the entropy of which we used as our approximation of the conditional entropy  $\mathcal{H}[\mathbf{a}|\mathbf{s}]$ . In this setting too, the SPAC algorithm tends to produce the most information efficient agents (Fig. 10).
E Open-loop control
Increasing the number of closed-loop actions used to prompt the transformer makes it generate more rewarding action sequences. This shows the importance of providing the sequence models with enough context, to be able to predict rewarding behaviors (Fig. 11).
F Partial observability
The augmented reward function that induces the preference for simple action sequences depends on the actions the agent selected in the past. This makes the reward function partially observable for a purely state-conditioned policy. Our agents learn to maximize this reward function despite this partial observability. We tested whether augmenting the state to contain information about actions selected in the past produced substantial differences in the learned policies. We equipped the LZ-SAC agents with a recurrent neural network (a Gated Recurrent Unit [55]) whose inputs were sequences of actions. We trained this network along with a single readout layer to produce embeddings  $\mathbf{e}_t$  of action sequences with which we defined the augmented state  $\mathbf{a}_t \sim \pi_\theta(\cdot|\tilde{\mathbf{s}}_t)$  where
Figure 11: Cumulative rewards attained by sequence priors in the fish swim environment, with a prompt length of 25.
Figure 12: Returns and compressibility of action sequences when the reward function is partially observable and fully observable.
$\tilde{\mathbf{s}}_t = \text{Concatenate}(\mathbf{s}_t, \mathbf{e}_t)$ . In three tasks we observed only minor differences in the policies learned (Fig 12).
G Pretraining the prior
We trained Transformer models to perform next-action prediction from action sequences produced by the converged LZ-SAC agent for all tasks. Using the pretrained transformers (with frozen weights) rather than a randomly initialized one whose weights were updated with stochastic gradient descent sped up learning significantly and allowed the SPAC agent to learn more rewarding behaviors (see Fig. 13). This showcases an interesting possible connection between our sequence compression framework and behavioral cloning.
Figure 13: Learning curves for SPAC using a pretrained transformer with frozen weights and a randomly initialized transformer. Using a pretrained transformer allows SPAC to solve the more challenging tasks in the benchmark. Learning curves are averaged over three seeds. Shaded region represents 20-80 percentile.