SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {C o v} \left[ \left(\boldsymbol {\epsilon} _ {\mathbf {x}}\right) _ {i}, \left(\boldsymbol {\epsilon} _ {\mathbf {x}}\right) _ {j} \right] = \operatorname {C o v} \left[ \alpha \sum_ {k = 1} ^ {N _ {\mathbf {x}}} \left(P _ {\mathbf {x}}\right) _ {i, k} \mathbf {x} _ {k}, \alpha \sum_ {h = 1} ^ {N _ {\mathbf {x}}} \left(P _ {\mathbf {x}}\right) _ {j, h} \mathbf {x} _ {h} \right] \tag {15} \\ = \alpha^ {2} \sum_ {k = 1} ^ {N _ {\mathbf {x}}} \sum_ {h = 1} ^ {N _ {\mathbf {x}}} \mathbf {x} _ {k} \mathbf {x} _ {h} \operatorname {C o v} \left[ \left(P _ {\mathbf {x}}\right) _ {i, k}, \left(P _ {\mathbf {x}}\right) _ {j, h} \right] = 0 \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {C o v} \left[ \left(\boldsymbol {\epsilon} _ {\mathbf {a}}\right) _ {i}, \left(\boldsymbol {\epsilon} _ {\mathbf {a}}\right) _ {j} \right] = \operatorname {C o v} \left[ \sum_ {k = 1} ^ {N _ {\mathbf {x}}} \left(P _ {\mathbf {a}}\right) _ {i, k} \mathbf {x} _ {k}, \sum_ {h = 1} ^ {N _ {\mathbf {x}}} \left(P _ {\mathbf {a}}\right) _ {j, h} \mathbf {x} _ {h} \right] \tag {16} \\ = \sum_ {k = 1} ^ {N _ {\mathbf {x}}} \sum_ {h = 1} ^ {N _ {\mathbf {x}}} \mathbf {x} _ {k} \mathbf {x} _ {h} \operatorname {C o v} \left[ \left(P _ {\mathbf {a}}\right) _ {i, k}, \left(P _ {\mathbf {a}}\right) _ {j, h} \right] = 0 \\ \end{array}
$$
where we used that different elements of  $P_{\mathbf{x}}$  and  $P_{\mathbf{a}}$  come from independent Gaussian distributions. Therefore, the joint distribution of the noise vectors  $\epsilon_{\mathbf{x}}$  and  $\epsilon_{\mathbf{a}}$  is Gaussian, with a diagonal covariance matrix:
$$
\epsilon_ {\mathbf {x}} \sim \mathcal {N} \left(\mathbf {0}, \alpha^ {2} \operatorname {D i a g} \left(S _ {\mathbf {x}} ^ {2} \mathbf {x} ^ {2}\right)\right) \quad \text {a n d} \quad \epsilon_ {\mathbf {a}} \sim \mathcal {N} \left(\mathbf {0}, \operatorname {D i a g} \left(S _ {\mathbf {a}} ^ {2} \mathbf {x} ^ {2}\right)\right) \tag {17}
$$
where the squares are to be intended element-wise. The distribution of a can be directly computed from the formula of the linear transformations of multivariate Gaussian distributions from Eq. (7):
$$
\pi (\mathbf {a} | \mathbf {s}) = \mathcal {N} \left(W \mathbf {x} (\mathbf {s}), \operatorname {D i a g} \left(S _ {\mathbf {a}} ^ {2} \mathbf {x} ^ {2}\right) + \alpha^ {2} W \operatorname {D i a g} \left(S _ {\mathbf {x}} ^ {2} \mathbf {x} ^ {2}\right) W ^ {\top}\right) \tag {18}
$$
where we have used the independence of  $P_{\mathbf{a}}$  and  $P_{\mathbf{x}}$  to find the covariance matrix of  $\epsilon_{\mathbf{a}} + W\epsilon_{\mathbf{x}}$ .
A.3 Noise rescaling across networks of different size
Here, we will show that the variance of Lattice's generative noise model (Eq. (18)) depends the dimension of the latent state  $x$ . We thus, rescale  $S_{\mathbf{x}}$  and  $S_{\mathbf{a}}$  to be invariant to the nework size.
Derivation: We first observe that the standard deviation of each element of the noise vector  $\epsilon_{\mathbf{x}}$  scales with the size of  $\mathbf{x}$ . Indeed:
$$
\begin{array}{l} \mathbb {V} \left[ \left(\boldsymbol {\epsilon} _ {\mathbf {x}}\right) _ {i} \right] = \mathbb {V} \left[ \alpha \sum_ {j = 1} ^ {N _ {\mathbf {x}}} \left(P _ {\mathbf {x}}\right) _ {i, j} \mathbf {x} _ {j} \right] = \alpha^ {2} \sum_ {j = 1} ^ {N _ {\mathbf {x}}} \mathbf {x} _ {j} ^ {2} \mathbb {V} \left[ \left(P _ {\mathbf {x}}\right) _ {i, j} \right] \tag {19} \\ = \alpha^ {2} \langle \mathbb {V} [ (P _ {\mathbf {x}}) _ {i, j} ] \rangle \sum_ {j = 1} ^ {N _ {\mathbf {x}}} \mathbf {x} _ {j} ^ {2} = \alpha^ {2} N _ {\mathbf {x}} \langle (S _ {\mathbf {x}}) _ {i, j} ^ {2} \rangle \langle \mathbf {x} _ {j} ^ {2} \rangle . \\ \end{array}
$$
Therefore, if we consider the average value of  $(S_{\mathbf{x}})_{i,j}^{2}$  and of  $\mathbf{x}_j^2$  to be independent of the size of the latent state (at initialization time, it is the case, e.g., with the common Xavier initialization for the network parameters [58]), then we have that  $\mathbb{V}[\epsilon_i]$  scales with  $N_{\mathbf{x}}$ . By applying the correction
$$
\log S _ {\mathbf {x}} = \log \tilde {S} _ {\mathbf {x}} - 0. 5 \log \left(N _ {\mathbf {x}}\right) \tag {20}
$$
$$
(S _ {\mathbf {x}}) _ {i, j} ^ {2} = \exp \left(2 \left(\log \tilde {S} _ {\mathbf {x}} - 0. 5 \log \left(N _ {\mathbf {x}}\right)\right)\right) = \frac {1}{N _ {\mathbf {x}}} (\tilde {S} _ {\mathbf {x}}) _ {i, j} ^ {2} \tag {21}
$$
In this way the initialization of  $\log \tilde{S}_{\mathbf{x}}$  can be kept the same across networks with different latent state size. An identical argument is valid for  $S_{\mathrm{a}}$ , too.
A.4 Conditions on the covariance matrix of the action distribution
First we show that the covariance matrix of the action distribution, defined as  $W\Sigma_{\mathbf{x}}W^{\top} + \Sigma_{\mathbf{a}}$ , is positive semidefinite by construction. We start by observing that the matrix  $\Sigma_{\mathbf{x}} = \alpha^{2}\mathrm{Diag}(S_{\mathbf{x}}^{2}\mathbf{x}^{2})$  and the matrix  $\Sigma_{\mathbf{a}} = \mathrm{Diag}(S_{\mathbf{a}}^{2}\mathbf{x}^{2})$  are square diagonal matrices, whose elements are larger or equal than 0. We can therefore write  $\Sigma_{\mathbf{x}} = \Sigma_{\mathbf{x}}^{\frac{1}{2}}\Sigma_{\mathbf{x}}^{\frac{1}{2}}$  and  $\Sigma_{\mathbf{a}} = \Sigma_{\mathbf{a}}^{\frac{1}{2}}\Sigma_{\mathbf{a}}^{\frac{1}{2}}$ , where the elevation to  $\frac{1}{2}$  has to be applied to each element of the matrices. For any vector  $\mathbf{y} \in \mathbb{R}^{N_{\mathbf{x}}}$ , we have that
$$
\mathbf {y} ^ {\top} \left(W \Sigma_ {\mathbf {x}} W ^ {\top} + \Sigma_ {\mathbf {a}}\right) \mathbf {y} = \mathbf {y} ^ {\top} \left(W \Sigma_ {\mathbf {x}} ^ {\frac {1}{2}} \Sigma_ {\mathbf {x}} ^ {\frac {1}{2}} W ^ {\top} + \Sigma_ {\mathbf {a}} ^ {\frac {1}{2}} \Sigma_ {\mathbf {a}} ^ {\frac {1}{2}}\right) \mathbf {y} = \left| \left| \Sigma_ {\mathbf {x}} ^ {\frac {1}{2}} W ^ {\top} \mathbf {y} \right| \right| ^ {2} + \left| \left| \Sigma_ {\mathbf {a}} ^ {\frac {1}{2}} \mathbf {y} \right| \right| ^ {2} \geq 0 \tag {22}
$$
The regularization term we apply in Lattice consists in a multiple of the identity matrix by the coefficient  $\gamma$ , so that the minimum eigenvalue of the covariance matrix can never be lower than  $\gamma$  itself.
Without a regularization term, the covariance matrix  $W \Sigma_{\mathbf{x}} W^{\top} + \Sigma_{\mathbf{a}}$  might have 0 eigenvalues. In particular, this is common when using an activation function which promotes sparse latent representations, such as ReLU. In the limit case where  $\mathbf{x}$  is the null vector, the covariance matrix is the null matrix.
Table T1: Task and reward parameters of Elbow pose, Finger pose, Finger reach and Hand pose.
Task	Elbow pose
Parameter	Value	Finger pose
Parameter	Value	Finger reach
Parameter	Value	Hand pose
Parameter	Value
Max steps	100	Max steps	100	Max steps	100	Max steps	100
Pose threshold	0.175	Pose threshold	0.35	Pose threshold	0.8
Target distance	1	Target distance	1	Target distance	0.5
Reward	Pose	1	Pose	1	Reach	1	Pose	1
Bonus	0	Bonus	0	Bonus	4	Bonus	0
Penalty	1	Penalty	1	Penalty	50	Penalty	1
Action reg.	0	Action reg.	0	Action reg.	0	Action reg.	0
Solved	1	Solved	1	Solved	0	Solved	1
Done	0	Done	0	Done	0	Done	0
Sparse	0	Sparse	0	Sparse	0	Sparse	0
Table T2: Task and reward parameters of Hand reach, Baoding, Reorient and Pen.
Task	Hand reach Parameter	Value	Baoding Parameter	Value	Reorient Parameter	Value	Pen Parameter	Value
Max steps	100	Max steps	200	Max steps	150	Max steps	100
Goal range x	(0.25, 0.25)	Goal pos.	(0, 0)	Goal orient. range	(-1, 1)
Goal range y	(0.28, 0.28)	Goal rot.	(-0.785, 0.785)
Reward	Reach	1	Pos. dist. 1	1	Pos.dist.	1	Pos. align	0
Bonus	4	Pos. dist. 2	1	Rot. dist.	0.2	Rot. align	0
Penalty	50	Alive	1	Alive	1	Alive	1
Action reg.	0	Action reg.	0	Action reg.	0
Solved	5	Solved	2	Solved	1
Done	0	Done	0	Done	0
Sparse	0	Sparse	0	Sparse	0
Pos. dist. diff.	100	Pos. align diff.	100
Rot. dist. diff.	10	Rot. align diff.	100
A.5 Empirical covariance of the action
The covariance of the action components of a deterministic policy is given by:
$$
\begin{array}{l} \operatorname {C o v} \left(a _ {i}, a _ {j}\right) = \operatorname {C o v} \left(\sum_ {k} w _ {i, k} x _ {k}, \sum_ {h} w _ {j, h} x _ {h}\right) \\ = \sum_ {k} \sum_ {h} w _ {i, k} w _ {j, h} \operatorname {C o v} \left(x _ {k}, x _ {h}\right) \tag {23} \\ = \left(W \operatorname {C o v} (\mathbf {x}) W ^ {\top}\right) _ {i, j} \\ \end{array}
$$
A.6 Parameters of the PyBullet and MyoSuite environments
We used all the default parameters for the PyBullet environments [20], including the default reward function and episode length. The MyoSuite environments, instead, come with a defined metric for the success (the solved value), while the reward components are adjustable [18]. We keep the same reward components across all the algorithms, with values which consent achieving good solved values. In Table T1 and T2 we detail the environment and reward parameters we choose for our trainings.
We often add a new reward component, called alive, which is equal to 1 when the episode is not finished. It can be used to promote policies that do not trigger an early termination, e.g., by dropping the object. To be noted that we reduced the range of possible target poses in Hand pose, because no algorithm could solve the environment with the full range (without a curriculum), making the environment unsuitable to test the difference between standard and latent exploration.
A.7 Hyperparameters of SAC, PPO, gSDE and Lattice
Our implementation was based on the library Stable Baselines 3 [45], and the code for Lattice will be shared in an open-source way.
Here, we summarize the parameters of SAC, gSDE-SAC and Lattice-SAC for the PyBullet locomotion tasks (Table T3) and the parameters of PPO, gSDE-PPO and Lattice-PPO in the MyoSuite muscle control tasks (Table T4 and T5). We also assessed the dependency of the final reward depending
Table T3: Parameters of SAC, gSDE-SAC and Lattice-SAC in the PyBullet locomotion tasks
Task
Algorithms	Parameters	Ant	Half Cheetah	Walker	Hopper	Humanoid
SAC	Action normalization	Yes	Yes	Yes	Yes	Yes
Buffer size	300 000	300 000	300 000	300 000	300 000
Learning rate	0.0003	0.0003	0.0003	0.0003	0.0003
Warmup steps	10 000	10 000	10 000	10 000	10 000
Minibatch size	256	256	256	256	256
Discount factor γ	0.98	0.98	0.98	0.98	0.98
Soft update coeff. τ	0.02	0.02	0.02	0.02	0.02
Train frequency (steps)	8	8	8	8	8
Num gradient steps	8	8	8	8	8
Target update interval	1	1	1	1	1
Entropy coefficient	auto	auto	auto	auto	auto
Target entropy	auto	auto	auto	auto	auto
Policy hiddens	[400, 300]	[400, 300]	[400, 300]	[400, 300]	[400, 300]
Q hiddens	[400, 300]	[400, 300]	[400, 300]	[400, 300]	[400, 300]
Activation	GELU	GELU	GELU	GELU	GELU
gSDE	Init log std	-3	-3	-3	-3	-3
Full std matrix	Yes	Yes	Yes	Yes	Yes
Lattice	Init log std	0	1	1	1	0
Full std matrix	Yes	Yes	Yes	Yes	Yes
Std clip	(0.001, 1)	(0.001, 10)	(0.001, 10)	(0.001, 10)	(0.001, 1)
Std regularization	0.001	0.001	0.001	0.001	0.001
α	1	1	1	1	1
on the initial value of  $\log \sigma$  in for Hand Pose and Reorient. The default value of 0 seems to perform optimally (Fig. F1).
Figure F1: Final reward as a function of the initial value of the exploration standard deviation of Lattice, with period  $T = 1$ .
A.8 Detailed reward and energy results of all experiments
We list the performance (energy and reward) of SAC, gSDE-SAC and Lattice-SAC in the PyBullet locomotion tasks (Table T6) and the performance of PPO, gSDE-PPO and Lattice-PPO in the MyoSuite tasks (Tables T7,T8, T9 and T10).
All results are averaged over 5 seeds and we report mean and standard error of mean.
A.9 Evolution of the dimensionality of the policy during training
In the main text we analyzed how Lattice explores the environment (Section 6). As discussed in the main text, here we present the results across learning.
We studied how the dimensionality of the policy changes during training when the policy is trained with Lattice, compared to standard SAC and PPO. While the number of relevant components is similar at the beginning of the training, the policies trained with Lattice converge to actions that lie
Table T4: Parameters of PPO, gSDE-PPO and Lattice-PPO in Elbow pose, Finger pose, Hand pose and Finger reach.
Task
Algorithms	Parameters	Elbow pose	Finger pose	Hand pose	Finger reach
PPO	Action normalization	Yes	Yes	Yes	Yes
Learning rate	0.000025	0.000025	0.000025	0.000025
Batch size	32	32	32	32
Gradient steps	128	128	128	128
Num epochs	10	10	10	10
Discount factor γ	0.99	0.99	0.99	0.99
Entropy coefficient	0.0000036	0.0000036	0.0000036	0.0000036
Value function coefficient	0.84	0.84	0.84	0.84
GAE λ	0.9	0.9	0.9	0.9
Clip parameter	0.3	0.3	0.3	0.3
Max gradient norm	0.7	0.7	0.7	0.7
Policy hiddens	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Critic hiddens	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Policy LSTM hiddens	256	256	256	256
Critic LSTM hiddens	256	256	256	256
Activation	ReLU	ReLU	ReLU	ReLU
gSDE	Init log std	-2	-2	-2	-2
Full std matrix	No	No	No	No
Lattice	Init log std	1	0	1	0
Full std matrix	No	No	No	No
Std clip	(0.001, 10)	(0.001, 10)	(0.001, 10)	(0.001, 10)
Std regularization	0	0	0	0
α	1	1	1	1
on a lower dimensional manifold (Fig. F2). We hypothesize that this is the main reason why Lattice leads to more energy-efficient policies. Indeed, the energy consumption is similar for both Lattice and SAC/PPO at initialization and diverges during training with Lattice achieving more energy-efficient policies; this result is also consistent across seeds(Fig. F3). Overall, the final policies trained with lattice have lower dimensionality, consistently across environments and random seeds (Fig. F4).
Table T5: Parameters of PPO, gSDE-PPO and Lattice-PPO in Hand reach, Baoding, Reorient and Pen.
Task
Algorithms	Parameters	Hand reach	Baoding	Reorient	Pen
PPO	Action normalization	Yes	Yes	Yes	Yes
Learning rate	0.000025	0.000025	0.000025	0.000025
Batch size	32	32	32	32
Gradient steps	128	128	128	128
Num epochs	10	10	10	10
Discount factor γ	0.99	0.99	0.99	0.99
Entropy coefficient	0.0000036	0.0000036	0.0000036	0.0000036
Value function coefficient	0.84	0.84	0.84	0.84
GAE λ	0.9	0.9	0.9	0.9
Clip parameter	0.3	0.3	0.3	0.3
Max gradient norm	0.7	0.7	0.7	0.7
Policy hiddens	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Critic hiddens	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Policy LSTM hiddens	256	256	256	256
Critic LSTM hiddens	256	256	256	256
Activation	ReLU	ReLU	ReLU	ReLU
gSDE	Init log std	-2	-2	-2	-2
Full std matrix	No	No	No	No
Lattice	Init log std	0	0	0	0
Full std matrix	No	No	No	No
Std clip	(0.001, 10)	(0.001, 10)	(0.001, 10)	(0.001, 10)
Std regularization	0	0	0	0
α	1	1	1	1
Table T6: Detailed results in the PyBullet locomotion environments. Results are averaged over  $\mathrm{N} = 5$  seeds.
Ant Energy	Reward	Hopper Energy	Reward	Walker Energy	Reward	Half cheetah Energy	Reward	Humanoid Energy	Reward
SAC	0.23 ± 0.01	3381 ± 30	0.26 ± 0.01	2417 ± 106	0.27 ± 0.0	2741 ± 81	0.23 ± 0.0	2934 ± 27	0.12 ± 0.0	2122 ± 169
SAC-gSDE period 4	0.23 ± 0.0	3978 ± 14	0.23 ± 0.01	2356 ± 17	0.26 ± 0.0	2728 ± 74	0.23 ± 0.0	3191 ± 125	0.12 ± 0.0	2114 ± 126
SAC-gSDE period 8	0.23 ± 0.0	3962 ± 7	0.25 ± 0.01	2234 ± 86	0.25 ± 0.0	2746 ± 39	0.28 ± 0.01	2752 ± 23	0.12 ± 0.0	1927 ± 120
SAC-gSDE episode	0.24 ± 0.01	3796 ± 48	0.24 ± 0.01	2472 ± 64	0.26 ± 0.0	2822 ± 32	0.24 ± 0.0	3081 ± 93	0.11 ± 0.0	2460 ± 160
SAC-Lattice (ours)	0.25 ± 0.0	3544 ± 212	0.23 ± 0.01	2610 ± 78	0.29 ± 0.0	2718 ± 92	0.27 ± 0.01	2900 ± 67	0.11 ± 0.0	2742 ± 77
SAC-Lattice period 4 (ours)	0.25 ± 0.0	3926 ± 78	0.23 ± 0.01	2446 ± 133	0.29 ± 0.0	2541 ± 129	0.26 ± 0.0	2964 ± 44	0.11 ± 0.0	2798 ± 197
SAC-Lattice period 8 (ours)	0.24 ± 0.0	3686 ± 100	0.24 ± 0.01	2621 ± 86	0.3 ± 0.0	2641 ± 121	0.26 ± 0.0	3031 ± 23	0.11 ± 0.0	2358 ± 77
SAC-Lattice episode (ours)	0.25 ± 0.0	3845 ± 86	0.22 ± 0.01	2586 ± 23	0.28 ± 0.0	2661 ± 130	0.25 ± 0.0	2948 ± 24	0.11 ± 0.0	2901 ± 211
Table T7: Detailed results in MyoSuite environments: Elbow Pose and Finger Pose. Results are averaged over  $\mathrm{N} = 5$  seeds.
Elbow pose Energy	Reward	Solved	Finger pose Energy	Reward	Solved
PPO	0.21 ± 0.01	88.29 ± 0.33	0.95 ± 0.0	0.05 ± 0.04	81.61 ± 3.08	0.96 ± 0.01
PPO-gSDE period 4	0.18 ± 0.0	71.15 ± 3.84	0.84 ± 0.03	0.02 ± 0.01	47.68 ± 11.88	0.78 ± 0.07
PPO-Lattice (ours)	0.19 ± 0.01	85.55 ± 0.67	0.94 ± 0.0	0.04 ± 0.0	74.35 ± 4.51	0.91 ± 0.03
PPO-Lattice period 4 (ours)	0.27 ± 0.01	62.57 ± 1.35	0.77 ± 0.01	0.05 ± 0.02	52.57 ± 9.0	0.78 ± 0.05
Table T8: Detailed results in MyoSuite environments: Finger reach and Hand pose. Results are averaged over  $\mathrm{N} = 5$  seeds.
Finger reach	Hand pose
Energy	Reward	Solved	Energy	Reward	Solved
PPO	0.2 ± 0.02	242.72 ± 15.53	0.2 ± 0.02	0.04 ± 0.0	-25.85 ± 3.05	0.54 ± 0.03
PPO-gSDE period 4	0.07 ± 0.02	257.97 ± 23.81	0.22 ± 0.03	0.04 ± 0.0	-74.13 ± 7.81	0.24 ± 0.05
PPO-Lattice (ours)	0.04 ± 0.01	337.07 ± 15.62	0.33 ± 0.02	0.04 ± 0.0	-44.8 ± 8.75	0.42 ± 0.06
PPO-Lattice period 4 (ours)	0.05 ± 0.0	206.46 ± 52.45	0.18 ± 0.05	0.03 ± 0.0	-99.59 ± 9.48	0.11 ± 0.04
Table T9: Detailed results in MyoSuite environments: Hand reach and Baoding. Results are averaged over  $\mathrm{N} = 5$  seeds.
Hand reach	Baoding
Energy	Reward	Solved	Energy	Reward	Solved
PPO	0.09 ± 0.0	581.4 ± 30.37	0.52 ± 0.08	0.08 ± 0.0	573.46 ± 73.24	0.4 ± 0.07
PPO-gSDE period 4	0.07 ± 0.0	462.72 ± 25.97	0.21 ± 0.06	0.07 ± 0.01	437.12 ± 91.91	0.26 ± 0.09
PPO-Lattice (ours)	0.04 ± 0.0	586.9 ± 19.19	0.54 ± 0.05	0.04 ± 0.0	627.31 ± 69.01	0.44 ± 0.07
PPO-Lattice period 4 (ours)	0.07 ± 0.0	556.1 ± 11.14	0.44 ± 0.03	0.07 ± 0.01	471.16 ± 99.66	0.29 ± 0.1
Table T10: Detailed results in MyoSuite environments: Reorient and Pen. Results are averaged over N=5 seeds.
Reorient Energy	Reward	Solved	Pen Energy	Reward	Solved
PPO	0.04 ± 0.01	233.74 ± 16.83	0.33 ± 0.05	0.07 ± 0.01	140.83 ± 9.77	0.16 ± 0.03
PPO-gSDE period 4	0.04 ± 0.01	187.99 ± 27.04	0.2 ± 0.08	0.07 ± 0.0	193.85 ± 13.42	0.42 ± 0.06
PPO-Lattice (ours)	0.01 ± 0.0	283.07 ± 14.99	0.48 ± 0.05	0.04 ± 0.0	169.72 ± 15.98	0.29 ± 0.06
PPO-Lattice period 4 (ours)	0.03 ± 0.01	250.13 ± 13.6	0.38 ± 0.04	0.06 ± 0.0	191.8 ± 15.75	0.39 ± 0.08
Table T11: SAC trained on a subset of the MyoSuite environments: Finger Pose, Hand Pose, Reorient and Pen. Results are averaged over  $\mathrm{N} = 3$  seeds.
Finger pose SAC	Hand pose SAC	Reorient SAC	Pen SAC
Energy	Solved	Energy	Solved	Energy	Solved	Energy	Solved
SAC	0.04 ± 0.0	0.96 ± 0.0	0.04 ± 0.0	0.84 ± 0.03	0.06 ± 0.0	0.55 ± 0.03	0.07 ± 0.0	0.39 ± 0.14
SAC-gSDE episode	0.04 ± 0.0	0.96 ± 0.0	0.04 ± 0.0	0.84 ± 0.0	0.07 ± 0.0	0.5 ± 0.03	0.07 ± 0.0	0.17 ± 0.02
SAC-Lattice episode (ours)	0.02 ± 0.01	0.92 ± 0.01	0.04 ± 0.0	0.73 ± 0.02	0.06 ± 0.0	0.67 ± 0.03	0.08 ± 0.0	0.59 ± 0.03
Figure F2: A Both in the Humanoid (top) and in the Reorient (bottom) motor control tasks, policies trained with Lattice require fewer principal components to explain the variance of the actions. The graphs, generated by testing policies at different stages of the training, highlight how the number of components decreases almost uniformly throughout the training, both with Lattice and with independent action noise. However, the effect is much stronger with Lattice. Result for one seed. B Number of principal components that explain at least  $90\%$  of the variance with respect to the training step for different seeds  $(N = 5)$ . For both Humanoid (top) and in the Reorient (bottom) tasks, Lattice reaches lower principal components during training compared to SAC/PPO.
Figure F3: Distribution of the energy consumption across training episodes for different seeds (column) in the Humanoid (top) and Reorient (bottom) task. For both, the energy consumption is similar at the beginning and (relatively quickly diverges during training. Once the policy is trained, LATTICE shows a more energy-efficient policy.
MyoSuite environments
PyBullet environments
Figure F4: Analysis of the effective dimensionality of the policies obtained with Lattice-SAC and Lattice-PPO versus standard SAC and PPO in the PyBullet (left) and Myosuite (right) environments. The dimensionality is estimated by running 100 test episodes per environment (5 random seeds), computing the principal component analysis of the actions counting how many principal components are necessary to reach  $90\%$  cumulative explained variance, rescaled by the number of action components. We can see that, consistently across most environments and random seeds (dotted lines in the plot), Lattice leads to lower dimensional policies.