Latent exploration for Reinforcement Learning
Authors: Alberto Silvio Chiappa
Alessandro Marin Vargas
Ann Huang
Alexander Mathis
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement learning
efficient exploration
curse of dimensionality
motor control
musculoskeletal control
TL;DR: We propose a monolithic framework for implicitly exploring applicable to both off- and on-policy algorithms that learns more energy efficient policies.

Abstract: In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and off-policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18\% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60\%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks. The code is available at: https://github.com/amathislab/lattice.

Latent Exploration for Reinforcement Learning
Alberto Silvio Chiappa
Alessandro Marin Vargas EPFL
École Polytechnique Fédérale de Lausanne (EPFL) alberto. www.soccccaapppp..ccc.mmm
al alessandro haw.aaddro..aiiiaaaaaaeeee....
Ann Zixiang Huang
Alexander Mathis EPFL alexander www.imseeth......
Mila, EPFL
zixiang. haanngg.aangmmmmi... laanngmmail.mmggll.c..
Abstract
In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstruc- tured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's 's effect, or modifying the behavior a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent Exploration (Lattice), method to inject temporally-comee correlaee noise into the latent state of the policy network, which can be seamlessly integrated with on- and off- policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's s activations can be modeled as a multivariate Gaussian distribution with full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAl achieves state of the art results, and reaches 18% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control envi- ronments of MyoSuite, Lattice-PPe achieves higher reward in most reaching and object manipulation tasks, while also finding more mergy-efficient policies with reductions of 20-60%. Overall, we demonstrate the effectiveness of structured action noise time and actuator space for complex motor control tasks. The code is available at: https://gthubb.commmmmaahhssaba
Introduction
Effectively exploring the environment while learning the policy is key to the success of Reinforcement Learning (RL) algorithms. Typically, exploration is attained by using a non-deterministic policy to collect experience, that the random component of the action selection process allows the agent visit states that would have not been reached with deterministic policy. In on-policy algorithms, such A3C I and PPO (2). the policy network parametrizes probability distribution, usually independent multivariate Gaussian, from which actions are sampled. In off-policy algorithms,, such as DQN B. DDPG (4) and SAC (5), the policy used collect experience can be different from the one learned to maximize the cumulative reward, which leaves more freedom in the selection of an exploration strategy. The standard approach in off-policy RL consists perturbing the policy by introducing randomness (e.g., c-greedy exploration in environments with a discrete action space and Gaussian noise environments with a continuous action space). the past years, many notable
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

LATTICE LATent Time- Correlated Exploration
Latent noise
Perturbation matrices
x
Na
Pa
(Pa)i, ~ ~ (0, (Sa)..)
W
Nx
Nx
Px
(Px)is (O,
S
a
Nx
Nx
Latent size
N
Action size
S
Learned perturbation std
x;
B
C correlaornl
LATTICE a = (W + Pa WP
Time Action
T 5
p(0)
T 2 p(O)
p(1)
gSDE a
(W + Pa)x
lime
T=
Default
Wx +
P(0)
P(1)
((2)
p(3)
p(4)
Figure I: A Lattice introduces state-dependent perturbations both to the action space and to the latent space. B Compared to the default action noise, Generalized State- Dependent Exploration (gSDE) [8] introduces a state-dependent perturbation the action. Lattice generalizes gSDE, including a second, policy-dependent perturbation, which induces correlation between action components. C Similarly to gSDE, the perturbation matrices Px and Pa are sampled periodically, providing noise with a temporal structure.
successes of RL are based on independent noise 22 77. However, previous work has challenged the inefficiency of unstructured exploration by focusing on introducing temporal correlations 8 4 9
11]. Indeed, two opposite perturbations of subsequent actions might cancel out the deviation from the maximum probability trajectory defined by the policy, effectively hiding potentially better actions if but
followed by a more coherent policy. Here, we argue that not only correlation in time also correlation across actuators, improve noise-driven exploration.
In motor control, precise coordination between actuators is crucial for the execution of complex behaviors [52 Perturbing each actuator independently can disrupt such coordination, limiting the probability of discovering improvements to the current policy. In particular, musculoskeletal systems [6 typically feature larger number of actuators (muscles) than degrees of freedom (joints). Here, we demonstrate that such systems the exploration achieved by uncorrelated actuator noise is suboptimal. To tackle this problem, we introduce exploration strategy, named LATent TIme-Correlated Exploration (Lattice). Lattice takes advantage of the synergies between actuators learned by the policy network perturb the different action components in structured way. This is achieved by applying independent noise to the latent state of the policy network (Fig- A). With extensive experiments, show that Lattice can replace standard unstructured exploration 2 5 and time-only-correlateed exploration (gSDE) [ in off-policy (SAC) and on-policy (PPO) RL algorithms. and improve performance in complex motor control tasks. Importantly, we demonstrate that Lattice-SAC is competitive in standard benchmarks for continuous control, such as the locomotion environments of PyBullet [20]. In the Humanoid locomotion task. Lattice improves training efficiency. final performance and energy consumption. In the complex object manipulation tasks of the MyoSuite library [8]. involving the control of a realistic musculoskeletal model of the human hand, Lattice-PPo PPO achieves higher reward, while also finding more energy -nffyyeffficient policies than standard PPO. To the best of our knowledge, our work is the first to showcase the potential of modeling the action distribution of a policy network with a multivariate Gaussian distribution with full covariance matrix induced by the policy network weights, rather than diagonal covariance matrix.

---

B
Action noise Latent maise ((+++))
a Wi+
Action
Action noise
11
15 1
0.030
22
10 e2
C3
C
2
angle noise
fi f2
-0 fi
0.015
-5 f2
0.010
fa
fa
e1 63 fa
e 63 f3
p.m 8.2-
10
Figure 2: A Representation of musculoskeletal model of a human arm. The elbow joint is actuated by flexor-extensor muscle groups. Figure adapted from 18 31. B Covariance matrix of the actions (left) and noise of the actions (right) when latent state learned policy is perturbed for flexors fi-fa. and extensors 61- C Distribution of the joint angle perturbation across episodes when noise is applied to the action or the latent space. The action noise was tuned that each action component has the same variance as with latent noise. The difference in distribution, due to the extra-diagonal terms of the action covariance, statistically significant (p- 3.2 x 10-17, Wilcoxon signed-rant test).
Related work
Exploration based on the learned policy. In on-policy RL, the target policy must be stochastic, so that the experience collected includes enough variability for the agent to discover policy improvements. Algorithms such as A3C (1) PPO (2). and TRPO (2 parametrize each action component with an independent Gaussian distribution (continuous actions) or a categorical distribution (discrete actions). In off-policy RL, most exploration strategies are based on perturbations of the policy. For example, in discrete action spaces an exploratory policy can be derived from the target policy by adding 6- greedy exploration, more sophisticated perturbations such the upper confidence bound 22 23 Thompson sampling (24), Boltzmann exploration
[25] and Information- Cormaation-iirie Directed Sampling (IDS) 26
In continuous action space, the perturbation of the actions provided by independent random noise, with (e.g.. SAC (5I) or without (c.g., DDPG [4) learned noise parameters. Unstructured exploration has several drawbacks [9] which have been tackled by ensuring temporally coherent noise through colored noise [1]. an aute-regressive process [27] or Markov chain [28) Another strategy to induce time correlation consists in keeping the perturbation parameters constant for multiple steps. This concept has been applied to perturbations of the network parameters [29] and to state state-dependen
action perturbations g 30). While these exploration strategies provide smoother noise in time, Lattice aims to pair time-correlated state-dependent exploration with correlation in the action space.
Other exploration strategies. In on-policy RL, boset- exploration [32]. curiosity-drine -drivnn exploration 33 and random network distillation (34] are techniques to encourage visiting rare states, In off-policy RL, bootstrapped DQN (35) employs multiple Q-networks guide exploration and improve performance. Additionally, unsupervised methods like Diversity Is All You Need [36) can achieve more diverse exploration by maximizing mutual information between goal-conditioned policy and a set of skill labels. Starting to explore only after returning to recently visited state allows solve complex sequential-dece
problem (37.. Lattice does not aim replace curriculum learning (88 40 15 or exploratory policies, but proposes exploring implicitly via the policy network monolithic way and can be combined with other approaches.
Exploration over-actuated system. Overactuated systems, such as musculoskeletal systems, pose a challenge for exploration. Recent NeurIPS challenges have addressed the problem of learning control policies for locomotion and dexterous hand movements with musculoskeletal models [41] The winners these challenges have typically utilized complex training curricula, such as static-to- to--mmimiic stabilization in the Baoding task, where a hand needs to accurately rotate two balls [15), or feature engineering and expert demonstration for locomotion [41. Exploration can also be encouraged by learning waussle-coordinaatee network [42] or the state-dependent joint torque limits and energy function 43 Similarly, Schumacher et al. proposed a self-organizing controller that outperforms previous approaches by inducing state-space covering exploration [19]. Integrating exploratory policies into RI framework requires ad-hoe solutions, whereas Lattice can be trained end-to-end with both on-policy and off off-policy algorithms.
14 [3).
3

---

3 Motivating example: The case of a flexor-extensor, single joint arm
To motivate Lattice, consider a simple flexor-extensor system, such as the elbow joint in the human upper arm (Fig. A The activation a E [0, 1] of the flexor muscle (biceps) produces negative angular acceleration at the elbow, while the activation de E [o. 1J of the extensor muscle (triceps) produces positive angular acceleration. Using a first order approximation of the system dynamics, write the equation of the angular position of the elbow as 0 a(a. a,), where 0 is the angle between the upper arm and the lower arm and a [rad 8 3 is a parameter of the system which converts the muscle activations into angular acceleration. Given this system, we consider simple control problem, where an agent needs to reach a target angular position of the elbow by activating the flexor and/or the extensor musele. At every step, the agent observes the difference A fu- 0, between the target angle 8 and the angle 0, at time t. For small values of A we assume that the control policy learned by the agent can be approximated with the linear functions a, 0.5 + A0 and as = 0.5 AO. We now want to compare the effects on the angular acceleration produced by noise applied to the actions (ag and ( to the latent state (A0) for various cases:
Case 1: action space noise. If we inject independent noise Ey and ee following a normal distribution (0, )²) to the muscle activations, we have aj (N((a)) o²) and a. ~ NN(aa),, o²). It can be shown that 0 ~ (N(a(((a)) (af)); 20?a?) (Appendix A.1
Case 2: latent space noise. If we instead inject Gaussian noise to the latent state A0, while the marginal distribution of a, and ag is the same as with action space noise, the fact that the two distributions are correlated has an effect on the distribution of It can be shown that Ö ~ ((((((.)) (ar)), 402o²) (Appendix A.I.
This toy example illustrates how perturbing the latent state, instead of the actions, leads to higher variance in the behavior space. In fact, when randomly perturbing two opposing muscles, half of the time the perturbations will be in opposite directions, with a reduced impact on the observed kinematics.
To corroborate this finding about the advantage of latent versus action perturbations, we consid- ered a policy network trained to control a realistic musculoskeletal arm model, implemented in MyoSuite [18) featuring three flexor muscles and three extensor muscles. neural network trained with PPO and Lattice exploration can learn to accurately reach target angle in every episode, and while doing so learns to alternately activate the extensor or the flexor muscle group (Fig. B, left). The training procedure is detailed in Appendix A We collected dataset of 100 episodes where the latent state of the policy network was perturbed with Gaussian noise. While applying independent random noise to the muscle activations produces diagonal covariance matrix, perturbing the latent state of the network leads to a full covariance matrix (Fig. 2 B, right). Thus, as hypothesized, the last layer of the policy network learned to distinguish between agonist and antagonist muscles, namely, the covariance shows positive correlation among muscles of the same group. We then tested whether sampling action noise with this covariance matrix leads higher variance in the task space (joint angles) than noise sampled independently for each muscle. We collected 100 episodes where, at each step, computed action with action space perturbation and one action with latent state perturbation. For consistency of evaluation, the perturbation in the action space was sampled with the same variance magnitude induced by the latent perturbation. We executed each pair of actions in two cloned environments and compared the variance of the joint angles. The latent state perturbation introduces higher variance in the joint angles (Fig. 2 C), thus driving more diverse kinematics and wider exploration. Overall, we conclude that latent exploration might be advantageous when learning to control actuaact systems, and thus developed and empirically tested Lattice on different benchmarks.
4 Methods
4.1 LATent Tlme-Correlated Exploration (Lattice)
In continuous control, the policy update of on-policy algorithms (e.g., REINFORCE [44], A3C D. TRPO [21] PPO 2 and of some off-policy algorithms (e.g., SAC (5) requires the computation of log π (a. (x),, which is the logarithm of the probability of choosing certain action a given the current state St, according to the current policy æ. In such cases, the policy gradient needs to be

---

Algorithm 1 Standard (e.g., SAC)
Algorithm Lattice
Require:
Require:
Policy n, env. dynamics p. std array gl*), (Ea); ~ ~N(,
Policy env. dynamics p.
period T, matrix S(a), matrix s(x), {0.1)
(Pa),, (0, (()) ~
1: Initialize state Sy;
2: step_count 0:
1: Initialize state Sy;
3: while not done do
2: step_coun + 0:
4: X₁ (Se):
3: while not done do
5: 6n sample (or());
4: if stepp_ mod T then
6: a, - Wxi
5:
Pa t- sample (s())):
7:
S rr, done + p(s,, ar):
6:
Px - sample (5(x));
8:
step_co count + step_ stunt +1;
7:
end if
9:
end while
8:
X f (S:)::
9:
a (W Pa + aW aWPx) X₁;
10: S1+1. done + p(st. az);
11: step count step_sotpeoeoppese count +1;
12:
end while
Algorithms 1-2: Experience collection algorithm with independent Gaussian noise (left) and with time- and actuator-correlated noise (right). green, we highlight the parameters defining the independent noise. In brown, the elements inducing time-correlation (gSDE). In blue, those implementing actuator correlation through the perturbation the latent state (Lattice). Note that for a = 0, Lattice is equivalent to gSDE.
backpropagated through the density function of the action distribution. This is typically accomplished via the reparametrization trick, in which the policy network outputs the parameters of a differentiable probability density function, whose analytical expression is known, so that gradients can flow through the probability estimation. In the standard case, when we apply independent Gaussian noise the action components, the action probability distribution can be parametrized as a multivariate Gaussian with diagonal covariance matrix (Algorithm 1). We propose that the weights of the policy network can be naturally used to parameterize the amount of exploration noise (Algorithm 2).
While generic perturbation of the policy network's activations would lead to action distribution with an unknown probability density funetion, Lattice we limit the latent perturbation to the last layer's latent state, which is linearly transformed into the action (Fig. A). Consider the output of the last layer the policy network x R.V. and the matrix W R.N.NN, mapping the embedding state to action according to Wx. We have indicated the size of the latent space with N₂ and the size of the action space with N₂- If we assume a perturbation of the latent state x with independent Gaussian noise then the perturbed latent state X is distributed as X N(x, Ex). Thus, the action distribution (WW,, W WEWWT + E.), where with Σ. we have indicated the independent component of the action noise, can be derived as linear transformation of multivariate Gaussian distribution (details in Appendix A.2). This formula provides an analytical expression for log π (at/s:), as function of the policy network weights and the covariance matrix of the latent state:
(0, Σx). where Ex is positive-definite diagonal matrix,
Na 2
1 2
log (als) =
log(2x) log WW T + Eal (Wx- a) T + EW> a).
(1)
The probability of an action depends on the network weights both through its mean value Wx and through the perturbation variance matrices Ex and Σ, Depending on the RL algorithm, might be convenient not to propagate the gradients of the policy network through the variance component of the loss. In this way, the expected action trains the policy network, while the loss due to the standard deviation can regulate its magnitude by updating the parameters Ex and E..
5

---

g5DE cpisode
Lattice-SAC
95DE-SAC T
SAC-pink
Lattic-SAC T-8
95DE-SAC T-8
Lattice-SAC T-1
Lattice-AAG episode
Episode reward mean
Ant
Hopper
Walker
Half cheetah
Humanoid
4K
3K
2K
2K
2K
2K
2K
1K
1K
0
10M
5M
10M
15M
5M 10M
15M
10M 20M
Number of steps
Figure 3: Learning curves in five locomotion tasks of PyBullet, representing the cumulative reward of each training episode (mean + s.e.m. across random seeds). Our baseline results improve over the benchmark rewards of the pre-trained networks of RI. 7oo [46), possibly because we used 16 vectorized environments to collect transitions and therefore a higher number of steps. The period refers to the time correlation of the exploration noise. Lattice, gSDE and Pink Noise perform comparably in all environments apart from the Humanoid [16], where Lattice is more efficient and achieves higher average reward.
4.2 Lattice generalizes time-correlated noise
Lattice can be thought of as an extension of Generalized State- Sate--ependdet Exploration (gSDE) 8 as it inherits all its properties, while extending with the possibility of perturbing the latent state of the policy, besides the actions (Fig. B). In Lattice, the agent learns two sets of parameters, Sx and Sa. representing the standard deviation of N₂ X (Nxx + NN.) Gaussian distributions (Nx and Na being the size of the latent space and of the action space, respectively). These parameters are used sample, every steps, two noise matrices Px and Pa (Fig. C). These matrices are used to determine the latent noise Ex aPxx and the action noise can be used to turn on off the correlated action noise. For a = 0, Lattice equivalent to gSDE. The value of T regulates the amount of time-correlation of the noise. If T is large, Px and Pa are sampled infrequently, causing the noise applied to the latent state and the action be strongly time-correlated (Algorithm 2).
Pax. The parameter E {0. 1]
In short, the action distribution depends on the latent state x, the last linear layer parameters W, the Lattice noise parameters Sx and the gSDE noise parameters S. in the following way:
क (als) N/ (Wx, Diag (iag(S3x²) a W (5x²)
(2)
4.3 Implementation details of Lattice
Code to study the implementation details is available at
Noise magnitude. The magnitude of the variance of individual noise components either be learned or kept fixed, and can or can not be made dependent on the current state. We extend the implementation of gSDE [8] (45), which makes the noise magnitude a learnable parameter independent of the policy network.
Controlling the variance. We introduce two clipping parameters to limit the minimum and the maximum value of the variance of each latent noise component, to avoid excessive perturbations and convergence to deterministic policy. This clipping does not modify the analytical expression of the action distribution, because it does not affect the sampled values, but rather the variance of the distribution.
Layer-dependent noise rescaling. As the noise vectors are given by 6x = aP&x and 6, = Pax, assuming the components of x have similar magnitude, the noise scales with the size of the latent state. To remove this effect, we apply correction to the learned matrices log Sx and log S, before sampling Px and Pa: log Sx log Sx 0.5 log( (og(Nx) and log S, log S, - 0.5 log (N)) It can be proven that this correction removes the dependence of the noise from the size the latent state (Appendix A.3) and thus from the network architecture.

---

PPO
9SDE-PPO T-4
Lattice-PrO T-1
Lattice-PP0 T-4
Elbow pose
Finger pose
Hand pose
Finger reach
1.0
0.3
0.6
0.3
0.5
0.5
0.4
0.2
0.2
0.1
Solved fraction
0.0
0.0
0.0
200K
400K
600K
2M
4M
6M
5M
10M
15M
20M
5M
10M
Hand reach
Reorient
Baoding
Pen
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10M
20M
10M
20M
10M
Number of steps
Figure 4: Learning curves in the MyoSuite environments, showing the adjusted solved fraction throughout the training (ratio between the number of steps in which the target is reached and the maximum length of the episode, mean s.e.m. across random seeds). In the pose environments, where the target prescribes the angle of all joints, standard exploration can be better than Lattice. In the other environments, where the exploration can leverage the learned muscle synergies, Lattice on par with or better than independent noise. We omit the learning curves with time corre latronation above 4, as they were always worse or much worse than and PPO, in accordance with previous findings for locomotion g
Preventing singularity of the covariance matrix. The covariance matrix of the action distribution must be positive definite. It has the expression W W + Mn (Eq. 2). with Vx a (Sāgg(()*) and Σ. = Diag(S((x") where the square operations are to be intended dlement-wise. By construction, the covariance matrix is Dositive-semidefine
However, it could be singular, e.g.. when x is the null vector (Appendix AA.4for for further details). To prevent the singularity the covariance matrix, we add small positive regularization value to its diagonal terms.
5 Experiments
50 in PyBullet (20, as well musculoskeletal control tasks of MyoSuite [18] built in MuJoCo 31] Both libraries include continuous control tasks of varying complexity. While in PyBullet the actuators apply a torque to each individual joint, MyoSuite the agent controls its body through musele activations. We give complete description of each task in Appendix A.6 We implemented Lattice as extension of gSDE the RL library Stable Baselines (45). which we used for all our experiments; the hyperparameters of the algorithms are detailed in Appendix|] [A.] All the results are averaged across 5 random seeds. The training was run on GPU cluster, for total of approximately 10,000 GPU-hours.
We benchmarked Lattice on standard locomotion tasks 47 6 16
5.1 Pybullet locomotion environments
Lattice can be paired with off-policy RI. algorithms, such as SAC [5]. We tested this combination in the locomotion environments of PyBullet, where gSDE-SAC achieves state-of-the-ee performance [8] 46). We also included Pink Noise exploration [11) as an additional baseline, which is the state- of-the-art colored noise process for exploration with off-policy RL algorithms. We used the same network architecture and hyperparameters for SAC specified in [46] for all the environments (see Appendix (.7)). Preliminary experiments on the parameters of Lattice showed that environments with smaller action space benefit from a higher initial standard deviation of the exploration matrix, so we set all the elements of log Sx and log S, for Ant and Humanoid and to for the other environments. We found that Lattice achieves similar performance to gSDE-SAC and SAC-pink for lower-dimensional morphologies (Ant, Hopper, Walker and Half Cheetah), while outperforming them substantially for the Humanoid (Fig. 3 and Appendix A.8) This suggests that Lattice achieve
7

---

higher performance when controlling larger- digen--ammenioonal morphologies. Furthermore, we later show that the policy learnt with Lattice-SAC for the Humanoid is more energy efficient than that learnt with SAC (Section 6
5.2
Musculoskcletal control: MyoSuite environments
For musculoskeletal control, tested several tasks from the MyoSuite 18
Three pose tasks (Elbow Pose, Finger Pose and Hand Pose). each session, a target angular position is sampled independently for each joint. This means that the policy has to learn how to control each degree of freedom independently.
Two reach tasks (Finger Reach and Hand Reach). In each session, target position for each finger tip (one for Finger Reach and five for Hand Reach) is sampled independently.
Three object manipulation tasks (Reorient, Pend and Baoding). In each session, the target is fixed or moving target for an object (a die in Reorient, pen in Pen and two Baoding balls in Baoding).
We tested Lattice with a LSTM network and PPO. This choice is motivated by (our) winning solution of the Boading ball task in the 2022 NeurlPS MyoChallenge [15,, which used PPO together with LSTM network [5]]. For experiments
the same PPO hyperparameters and network architecture, which we keep identical across methods (see Appendix A.77)
We set the initial value of all the elements of log Sx and log Sa to 0 in every task, except in Elbow pose and Hand pose, where we set it 1.
In the reach and challenging object ma- nipulation tasks (Reorient, Baoding and Pen), where the policy network has to control complex hand model with 39 muscles and receives an observation with more than 100 dimensions, Lattice consistently outperforms the baselines (Fig. and Appendix A In contrast, in the pose tasks PPO performs compa- rably or better than Lattice-PPO (Fig. and Appendix A.8). Those tasks define dense target states and, perhaps, explo-
30%
Lattlce-PPO better
20%
Reward gain (relative)
Finger reach
per
Reorient
Baoding Hand reach
0%
Elbow pase
Finger pose
-10%
PPO better
ration and coordination is less important
Hand pose
(see below).
-30%
-60%
-40%
-20%
20%
40%
60%
Interestingly, focusing action noise in the task-relevant space allows Lattice to avoid activating muscles unnecessarily, leading to conspicuous energy saving in the manipulation and reaching tasks (Fig. 5). In reach and object manipula- tion, Lattice-PP PPO achieves better reward lower energy cost. We speculate that injecting correlated noise across muscle activations improves exploration in the space of task-relevant body poses and facilitates the discovery of efficient, coordinated movements. We tested this hypothesis next.
Energy saving (relative)
Figure 5: Energy saving versus reward gain for the muscle control tasks. Lattice learns policies 20% 60% more energy efficient than independent exploration in the reach- ing and object manipulation tasks, while also achieving higher reward. In the pose tasks, independent exploration performs better, with similar energy consumption.
How does Lattice explore?
In most of the considered muskuloskeletal control environments, Lattice finds a more energy efficient control strategy, without compromising the performance (Fig. and Appendix A.8). Especially in Reorient, where Lattice outperforms standard exploration in cumulative reward, it does at

---

Humanoid
Reorient
1rodono of releett)
vartance
Fractoon of notaeno
Explained vertence
100
0.72
20
of eppordes
0.73
9.50
0.25
0.00
Lattice-SAC
0.00
Lattio
FL
UL
MA LA ponertt
0.30
6.15
0.8
10
0:225
0.050
29
Erergy
components
Alcion componi
Brergy
components
Action
Noise
D
Action
Noise
W
W/
RL
92
SAC
80
000
13
-2
-32
RA
RA
R
LA
-075
LA
RA LA
1.0
H
RL
LL
RA LA
W
M
W
M
101
C75
H
Lattice-SAe
Lattice-Pa
CD LL
RA
RA
0050
LA
LA
-0.9
H
AL
LL RA LA
H RL
LL
RA LA
M
R
W
M
Hpp Right Leg LL Left RA Right
LA Left Arm
W Wrist
Thumb
1- Index
Midde R
Ring
Litlle
Figure A Left: Graph of the fraction of noise allocated each group of action components by stochastic policy trained in the Humanoid environment with SAC and Lattice-SAC. In the Humanoid, Lattice tends focus most of the variance on the task-relevant actuators (legs). Middle: Distribution of the energy consumption. Each bar represents the number of test episodes falling in the corresponding energy consumption interval. Right: Cumulative explained variance of the actions' principal components, computed from dataset of 100 test episodes per seed. Shaded area represents 95% confidence interval across training seeds. More principal components are required to explain the same fraction of variance in SAC and PPO versus Lattice-SAC and Lattice-PPO. B Heatmap the correlation matrix for the action space (left column) and noise of the action (right column) for SAC (top row) and Lattice-SAC (bottom row). C-D Same as A-B but analyzing policy trained on the MyoSuite Reorient task with PPO and Lattice-PPe Lattice tends to induce sharper correlation between action components. Local patterns of correlation between action components be recognized in the noise covariance matrix.
fraction of the energy cost. We hypothesize that this is a direct consequence of biasing the noise with the same correlation as the actuators, preventing mmeryy-consumey needs to compensate for, despite the reward not being affected by them.
co-activations, which the agent
We assessed whether coordination emerged by analyzing policies for the Humanoid locomotion task and the Reorient task. By looking the relative allocation of action noise across actuators (Fig. 6 A, C), we can see that in the Humanoid, compared to SAC, Lattice-SAC re-directs more exploration noise towards the task-relevant leg motors (SAC: 40% Legs 45% Arms and 15% Body; Lattice-SAC: 50% Legs, 32% Arms, 18% Body). In Reorient, where all the muscles contribute to controlling the hand, the difference is less evident. Furthermore, both in the Humanoid and Reorient task, the intrinsic dimensionality of the actions output by a policy trained with Lattice is lower. Indeed, consistently across random seeds, fewer principal components explain higher fraction of the action variance in Lattice than in standard exploration (Fig. 66 A, C). Lattice consistently promotes policies with actions lying on liwerr-dmmesiioall manifold, and this effect extends beyond Humanoid and Reorient, to almost all the considered tasks (Appendix A.9. We speculate that this property of Lattice determines when it be preferred uncorrelated noise, i.e., when there exists Low-dimensions solution to the task which allows the agent to take advantage of motor synergies. In environments where the dimensionality of the task intrinsically low (reach and object manipulation)) Lattice achieves strong performance. Further experiments in the pose and in the object manipulation tasks, performed with Lattice-SAC, confirm the result that Lattice-SAC better than SAC for learning to manipulate objects (Appendix(A.8) A.8).

---

The intrinsic dimensionality of the actions decreases as an effect of training (Appendix A.9 While training tends to reduce the dimensionality the policy regardless of the action noise, the decrease more marked with Lattice. The low dimensionality of the actions is explained by the action correlation matrices, which show increased cross-actuator coordination with Lattice (Fig. 6 B, D). We speculate that this is driven by the correlation structure of the noise. While the off-diagonal elements are close to 0 with uncorrelated noise, those of Lattice present a structure that resembles that of the action correlation matrix F D), also consistent with the motivating example (Section 3), Indeed, if we consider the off-diagonal elements at position (i,J), we have that Cov(a;, aj) = CC (x)( while V (a(s))], = a² (W "
(further details in Appendix A.5). In the case where a = 1, Sx is the identity and the components of x are uncorrelated, then the two matrices are identical. While throughout the training the parameters of Sx can adapt, can empirically see that the covariance matrix retains elements of its original structure.
7 Discussion and Limitations
We proposed Lattice, a method that leverages the synergies between action components encoded in the weights of the policy network to drive exploration based on the latent state. Latent exploration has proven effective in discovering high-reward and energy efficient policies in complex locomotion and muscle control tasks. Remarkably. Lattice outperforms random exploration in the reach and object manipulation tasks, with a large reduction in energy consumption. We showed that perturbing the last layer of the policy network introduces enough bias into the action noise. Importantly, this specific perturbation allowed us to find an analytical expression for the action distribution, which makes Lattice straightforward enhancement for any on-policy off- off-policy continuous control algorithm.
This sets Lattice apart from other forms of exploration using an auxiliary policy to collect experience, as they see their application limited to off-policy RL. Furthermore, knowing the action distribution fundamental for those off-policy algorithms learning a stochastic policy, such as SAC. On the other hand, modeling the action distribution with multivariate Gaussian with full covariance matrix comes at additional computational cost. In fact, Lattice introduces a training overhead over gSDE and PPO/SAC (approx. 20%-30%, depending matrix multiplications required to estimate the action probabilitics (Eq. [ This limitation opens an interesting research direction, e.g., introducing sparsity constraints in the distribution matrices. For example, Sa and Sx could be forced to be diagonal or low-rank, without preventing motor synergies to be included into the noise distribution. We leave this investigation to future work.
the hardware). This is due to the additional
Biological motor learning finds efficient and robust solutions [12 13 52 Lattice also discovers low-energy solutions and it will be interesting future question, if the brain is also performing some form of latent-driven exploration. Beyond this specific question, we are enthusiastic that advances in musculoskeletal simulators [18] as well as reinforcement learning are opening up exciting avenues for Neuroscience (56).
Our research aims to facilitate the training of motor control policies for complex tasks. Lattice might be employed as a tool in Artificial Intelligence, Robotics and Neuroscience, contributing to the creation of powerful autonomous agents. The energy efficiency of the learned policies might be of great interest for energy-sensitiv robotics applications for reducing the carbon footprint of humans. While the advancement of the research in this field is a great opportunity, the related concerns should be carefully addressed [57].
Acknowledgments
We grateful to Adriana Rotondo, Stéphane D' Ascoli and other members of the Mathis Group for comments on earlier versions of this mamuscript. Our work was funded by Swiss SNF grant (310030_ 20l_2112116 and EPFL. A.M.V.: Swiss Government Excellence Scholarship. A.Z.H.: EPFL School of Life Sciences Summer Research Program. We thank the Neuro-X institute at EPFL for supporting the travel expenses (A.S.C. and A.M.V).
10

---

References
[1] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 99281937.. PMLR, 2016.
[2] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.5213, 2013.
[4] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint
2015.
[5] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Internationaa conference on machine learning, pages 1861-1870, 2018.
[6] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference machine learning.
pages 1329-1338. PMLR, 2016.
[7] Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon. Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint crXiv:1707.02286 2017.
[8] Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning, pages 1634-1644. PMLR, 2022.
[9] Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient methods. In Machine Learning and Knowledge Discovery in Databases: Euro- pean Conference, ECML PKDD 2008, Antwerp. Belgium, September 15-19, 2008, Proceedings, Part II 19, pages 234-249. Springer, 2008
[10] Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhu- ber. Exploring parameter space in reinforcement learning. Paladyn, 1:14-24, 2010.
[11] Onno Eherhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[12] Stephen H Scott. Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience, 5(7):532-545, 2004.
[13] Emanuel Todorov. Optimality principles in sensorimotor control. Nature neuroscience, 7(9):
907-915, 2004.
[14] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Seungmoon Song, Yuval Tassa, Mas- simo Sartori, and Vikash Kumar. Myochallenge: Learning contact-rich manipulation using musculoskcletal hand. htttps://sitet google. com/ wwwm///iwww/yoccalleegnnggggeego 2022.
[15] Vittorio Caggiano, Guillaume Durandau, Huwawei Wang, Alberto Chiappa, Alexander Mathis, Pablo Tano, Nishect Patel, Alexandre Pouget, Pierre Schumacher, Georg Martius, et al. My- ochallenge 2022: Learning contact-rich manipulation using musculoskeletal hand. In NeurIPS 2022 Competition Track, pages 233-250. PMLR, 2022.
[16] Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behav- iors through online trajectory optimization. In 2012 JEFE/RSJ International Conference on Intelligent Robots and Systems, pages 4906-4913. IEEE, 2012.

---

[17] Thomas K Uchida and Scott L Delp. Biomechanics of movement: the science of sports, robotics, and rehabilitation. Mit Press, 2021.
[18] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite- a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205. 2022.
[19] Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems. arXiv preprint arXiv:2206.004844 2022.
[20] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[21] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[22] Peter Auer. Using confidence bounds for exploiatiion-exploraiio trade-offs. Journaa of Machine Learning Research, 3(Nov):397-422, 2002.
[23] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256 2002.
[24] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling, Advances in neural information processing systems, 24, 2011.
[25] Nicolò Desa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in neural information processing systems, 30, 2017.
[26] Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. arXiv preprint arXiv: 1812v2/811.07544, 2018.
[27] Dmytro Korenkevych, A Rupam Mahmood, Gautham Vasan, and James Bergstra. Au- toregressive policies for continuous control deep reinforcement learning. arXiv preprint urXiv:/905: / /524, 2019.
[28] Herke van Hoof, Daniel Tanneberg, and Jan Peters. Generalized exploration in policy search. Machine Learning, 106(9-10)1705--1224 2017.
[29] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: arxiiv:706.079900 770.01995600 2017.
[30] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, al. Noisy networks for exploration. arXiv preprint arXiv: 170..10255 2017.
[31] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEF/RSJ international conference on intelligent robots and systems, pages 0026-5033. IEFF, 2012.
[32] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[33] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[34] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 18Xiv:1810.12888 2018
[35] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances in neural information processing systems, 29, 2016.
12

---

[36] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.00677,, 2018.
[37] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-588 2021.
[38] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48,
2009.
[39] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Au- tomatic curriculum learning for deep rl: A short survey. arXiv:2003. 04664 [cs, stal), May 2020.
[40] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth Stanley, Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 1e00:090111011 2019.
[41] Eukasz Kidziński, Carmichael Ong. Sharada Prasanna Mohanty, Jennifer Hicks, Sean Carroll, Bo Zhou, Hongsheng Zeng, Fan Wang, Rongzhong Lian, Hao Tian, et al. Artificial intelligence for prosthetics: Challenge solutions. In The NeurlPS' 18 Competition: From Machine Learning Intelligent Conversations, pages 69-128. Springer, 2020.
[42] Shuzhen Luo, Ghaith Androwis, Sergei Adamovich, Erick Nunez, Hao Su, and Xianlian Zhou. Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. Journal of NeuroEngineering and Rehabilitation, 20(1):1-19 2023.
[43] Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and Karen Liu. Synthesis of biologically realistic human motion using joint torque actuation. ACM Transactions On Graphics (TOG), 38 (4):1-12, 2019.
[44] Ronald J Williams. Simple statistical following algorithms for connectionist reinforce- ment learning. Reinforcement learning, pages 5-32, 1992.
[45] Antonin Rafin,, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emestus,, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations, Journal of Machine Learning Research, 22(268):1-8, 2021. URL http::/jmlr org wwtph//rr.r/ppppp////// p22ers/v22/
20-1364.hrr
[46]
Antonin Raffin.
RI baselines3 700, v1.8.3.
https://github.coo com
rl bww.binesssaae...............sss 200 2020.
[47] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1506v 2015.
[48] Seshashayee S Murthy and Marc H Raibert. 3d balance in legged locomotion: modeling and simulation for the one-legged case. ACM SIGGRAPH Computer Graphics, 18(1):27-27, 1984.
[49] Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learming, pages 1-9. PMLR, 2013.
[50] Alberto Silvio Chiappa, Alessandro Marin Vargas, and Alexander Mathis. Dmap: a distributed morphological attention policy for learning locomote with changing body. Advances in Neural Information Processing Systems, 35:37214-37277, 2022.
[51] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1735-1780, 1997.
[52] Yacl Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3): 139-154, 2009.
[53] Michale Fee and Jesse Goldberg. hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience, 198:152-170, 2011.
13

---

[54] Dagmar Sternad. It's not (only) the mean that matters: variability, noise and exploration in skill learning. Current opinion in behavioral sciences, 20:183-195, 2018.
[55] Adam Lowet, Qiao Zheng, Sara Matias, Jan Drugowitsch, and Naoshige Uchida. Distributional reinforcement learning in the brain. Trends in neurosciences, 43(12):980-997, 2020.
[56] Sébastien B Hausmann, Alessandro Marin Vargas, Alexander Mathis, and Mackenzie W Mathis. Measuring and modeling the motor system with machine learning. Current opinion in neurobiology, 70:11 2021.
[57] Maartje MA de Graaf. An ethical evaluation human-robot relationships. International journal of social robotics, 8:589-598, 2016.
[58] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249 256...6.20 JMLR Workshop and Conference Proceedings, 2010.
14

---

A
Appendix
Contents
A.1
Singe-joint arm: detailed calculations
15
A.2
Lattice's s Action Distribution P'rameterizzttonn
16
A.3
Noise rescaling across networks of different size
17
Conditions on the covariance matrix of the action distribution
17
A.5
Empirical covariance of the action
18
A.6
Parameters of the PyBullet and MyoSuite environments
18
A.7
Hyperparameters of SAC, PPO, gSDE and Lattice
18
A.8
Detailed reward and energy results of all experiments
19
A.9
Evolution of the dimensionality of the policy during training
19
A.1
Singe-joint detailed calculations
Case 1: action space noise. By combining the equation of the angular acceleration Ø a(ae - a,) with the distributions of the muscle activations a, (V(((a), o²) and ae ~N(((e), ,²) we can compute expected value and variance of the angular acceleration:
E
E [a(a, a.)] a [ac] - E [[a)) a((as) (ar))
(3)
V [a(a= - a,)] = a² (V [ae] + [ayl) = 2a20"
(4)
In the second derivation we have used the fact that a, and a, are independent random variables to compute their variances separately, and that V [--a V [ay].
Case 2: latent space noise. this case we apply noise to the latent state, so that A ~ (()),, o²).
While it still holds that a (N((a)), a²) and a. (aa)),) a²), the random variables longer independent. This fact does not change the computation of the expectation:
E
[a< a,)] a (EE [a.] - E [arl) a((ae) (ar))
(5)
For the variance, we need to make the dependence on 0 explicit, using the formulas a, = 0.5 + A0 and a 0.5 A0:
V (a(a, a,)] A2 4a (V (A0) 202
(6)
We observe that the variance is double in the case of latent space noise.
To test whether these results can be observed also in simulation, we considered the Elbow pose task of MyoSuite, where trained policy with PPO and Lattice exploration (with the hyperparameters specified Table (T... The policy reaches a solved value 0.95, meaning that the angle between upper and the lower arm is in the immediate neighborhood of the target angle 95% of the simulation time. We assessed the effect latent and action perturbations on the angular position by carrying out two simultaneous simulations of the environment per noise type. At every step, simulation received a noisy action, while the other would receive a deterministic, noise-free action. We then registered the difference in elbow angle between the noisy and the deterministic simulation. To avoid having the results obfuscated by the cumulative perturbation over longer time periods, we synchronized the simulators after every step at the state achieved through the stochastic simulation.
To compute comparable results with latent and action noise, we had to make sure the variance of each action component would be equivalent in both cases. We accomplished this by first running the simulation with latent noise, with the same variance for each latent state component. We then measured the variance that such perturbation would cause to each action component, and used
15

---

this value to generate the action perturbation noise. This ensures that the difference in the variability of the elbow angle when applying latent or action noise not due difference in scale, but only to the presence of off-diagonal elements in the noise covariance matrix.
A.2 Lattice's Action Distribution Parameterization
In Lattice, the action vector is linear transformation of the perturbed latent vector X + Ex. further perturbed with independent action noise Ea:
a WX + ca = (xx+ + ca
Furthermore, the latent noise 6x and the action noise ca are defined as follows:
6x
Pxx
and
Ca Pax
(8)
The elements of Px and Pa are distributed as independent Gaussians:
(Px)s ~N (0, (Sx)
and (Pa) (D.(5))
(9)
where ((x) and ((a) are learnt parameter matrices whose elements represent the standard deviation of each element of the perturbation matrices. We can therefore compute the distribution of each element of the noise vectors. They are defined as:
Nx (c))= = a > (P)ceyyy
Nx and (ca): = Σ (Pai,sx
(10)
meaning that they are the sum of independent Gaussian random variables. We can compute their mean and variance as follows:
N₂ =a
E
[Ex) = E
P
[Pxx, =0
(11)
j=1
==1
No
Nx
[(((x)
V
(Px P2)a,px
a²
V (Px) )all
= a² Σ x3 (Sx)
(12)
No Σ (Palis
Nx
E [(()]]]] = E
Σ
X (Pa)as))
(13)
F=1
N₂ Σ (Pa)i.x)
N₂ (Paa = Σ x (Sa)
N₂
V((a)) = v
=> X3
=
(14)
j=1
The covariance of noise elements at different indices is O. Indeed, for # j:
Nx (Px)..aK, a >
Nx
Cov [(xx);; (Exx)g = Cov
a
Σ (Px)s.88
k=1
h=1
(15)
No.
a
Cov
(Px)e (Px)x,)= =0
mm
Aum
N Σ
No
Cov ((E)) Cov
(Pa)i Σ (PP)j..AAA
A=1
hmll
(16)
Nx
Nx
Xx Cov [(aai, (Pa) =
16

---

where we used that different elements of Px and Pa come from independent Gaussian distributions. Therefore, the joint distribution of the noise vectors Ex and en is Gaussian, with a diagonal covariance matrix:
Ex ~ N'' (0, a Diag (5xx*)
and E, ~ (0, Diag (5xx²)
(17)
where the squares are to be intended element-wise. The distribution of a can be directly computed from the formula of the linear transformations of multivariate Gaussian distributions from Eq. 7
ग (als) = N (Wx(s). Diag Sxx²) + a aDDag (
(18)
where we have used the independence of Pa and Px to find the covariance matrix of 6a - W Ex
A.3 Noise rescaling across networks of different size
Here, we will show that the variance of Lattice's generative noise model (Eq. (8)) depends the dimension of the latent state æ. We thus, rescale Sx and S, to be invariant to the nework size.
Derivation: We first observe that the standard deviation of each element of the noise vector 6x scales with the size of x. Indeed:
N₂ (xxisxy
Nx Σ
V (((E)) = V
= a²
V
[(P)
j=1
(19)
N₂
=
Σ x
-
N ((S (x))
Therefore, if we consider the average value of (Sx) and of x² be independent of the size of the latent state (at initialization time, is the case, e.g., with the common Xavier initialization for the network parameters (58)), then we have that V [e,] scales with Nx. By applying the correction
log Sx = log Sx 0.5log(Ne (Nx)
(20)
we have that
(8x) exp (2(1og Sx 0.5 logg))
1
(21)
Nx
In this way the initialization of log Sx can be kept the same across networks with different latent state size. An identical argument is valid for Sa, too.
A.4 Conditions on the covariance matrix of the action distribution
First we show that the covariance matrix of the action distribution, defined as W Σ, WT positive semidefinite by construction. We start by observing that the matrix Σx a ( and the matrix Σ. Diag(S") xxx²) are square diagonal matrices, whose elements are larger or equal than O. We can therefore write Σx Σ and Σa E
Eg is
_ where the elevation to has to be applied to each element of the matrices. For any vector y E Nx, we have that
y (WE.WT + +Ea) y = >
x y E WT y11
+ 2 (22)
The regularization term we apply in Lattice consists in multiple of the identity matrix by the coefficient 7, so that the minimum eigenvalue of the covariance matrix can never be lower than itself.
Without a regularization term, the covariance matrix WE WT + Σ, might have 0 eigenvalues. In particular, this is common when using an activation function which promotes sparse latent representations, such as ReL.U. In the limit case where x the null vector, the covariance matrix the null matrix.
17

---

Table T1: Task and reward parameters of Elbow pose, Finger pose, Finger reach and Hand pose.
Task
Elbow pose Parameter
Finger pose
Finger reach
Hand pose
Value
Parameter
Value
Parameter
Value
Parameter
Value
Max sleps
100
Max sleps
100
Max sleps
100
Max steps
100
Pose threshold
0.175
Pose threshold
0.35
Pose threshold
0.8
Target distance
Target distance
Target distance
0.5
Reward
Pose Bonus Penalty
Pose
Reach
Pose
Bonus
0
Bonus
Bonus
0
Penalty
Penalty
50
Penalty
Action reg.
Action reg.
0
Action reg.
Action reg.
0
Solved
Solved
Solved
Solved
Done
Done
0
Done
Done
Sparse
Sparse
0
Sparse
Sparse
Table T2: Task and reward parameters of Hand reach, Baoding, Reorient and Pen.
Task
Hand reach
Baoding
Reorient
Pen Purameter
Purameter
Value
Purameter
Value
Parumeter
Value
Value
Max steps
100
Max steps
200
Max steps
steps
100
Goal range
(0.25, 0.25)
Goal pos.
(0, 0)
Goal orient. range
(-1,
Goal range
(0.28, 0.28)
Goal rot.
(-0.785, 0.785)
Reward Reach
Pos. dist. 1
Pos.dist.
Pos. align
Bonus
Pos. dist. 2
Rot. dist.
0.2
Rot. align
Penalty
50
Alive
Alive
Alive
Action
Action reg.
0
Action reg.
(
Solved
Solved
Solved
Done
Done
Done
Sparse
0
Sparse
O
Sparse
Pos. dist, diff.
100
Pos. align diff.
100
aligh diff.
100
A.5
Empirical covariance of the action
The covariance of the action components deterministic policy is given by:
Cov(ag, aj) Cov Σ WirK-KKK Σ
Winh
h
(23)
W Cov (I))
=(WCov()WWT)
A.6 Parameters the PyBullet and MyoSuite environments
We used all the default parameters for the PyBullet environments 200 including the default reward function and episode length. The MyoSuite environments, instead, come with defined metric for the success (the solved value), while the reward components are adjustable (18). We keep the same reward components across all the algorithms, with values which consent achieving good solved values. In Table T and 12 we detail the environment and reward parameters we choose for our trainings.
We often add a new reward component, called alive, which is equal to when the episode is not finished. It can be used to promote policies that do not trigger an early termination, e.g., by dropping the object. To be noted that we reduced the range of possible target poses in Hand pose, because algorithm could solve the environment with the full range (without curriculum), making the environment unsuitable to test the difference between standard and latent exploration.
A.7 Hyperparameters of SAC, PPO, gSDE and Lattice
Our implementation was based on the library Stable Baselines 451 and the code for Lattice will be shared in open-source way.
18

---

Here, we summarize the parameters of SAC, gSDE-SAC and Lattice- Suttice-SAClie for the PyBullet locomotion tasks (Table T3) and the parameters of PPO, gSDE-PPO and Lattice-PPO in the MyoSuite muscle control tasks (Table T4 and T5
We also assessed the dependency of the final reward depending
Table T3: Parameters of SAC, gSDE-SAC and Lattice-SAC in the PyBullet locomotion tasks
Task
Ant
Half Cheetah
Walker
Hopper
Humanoid
Algorithms
Parameters
SAC
Action normalization
Yes
Yes
Yes
Yes
Yes
Buffer size Learning rate
300 000
300 000
300 000
300 000
300 000
0.0003
0.0003
0.0003
0.0003
0.0003
Warmup steps
10 000
10 000
000
000
10 000
Minibatch size
256
256
256
256
256
Discount factor y
0.98
0.98
0.98
0.98
Soft update coeff. r
0.02
0.02
0.02
0.02
0.02
Train frequency (steps)
8
8
8
8
8
Num gradient steps
8
8
8
8
8
Target update interval
1
Entropy coefficient
auto
auto auto [400, 3001
auto
auto
auto
Target entropy
auto
auto
auto
Policy hiddens
[400, 3001
|400, 3001
[400, 3001
[400, 3001
Q hiddens Activation
[400, 300]
[400, 300]
[400,
300]
[400. 300]
[400,
300]
GELU
GELU
GELU
GELU
GELU
gSDE
Init log std Full std matrix
-3
-3
-3
-3
-3
Yes
Yes
Yes
Yes
Yes
Lattice
Init log std Full std matrix
1
Yes
Yes (0.001, 10)
Yes
Yes
Yes
Std clip
(0.001, I)
(0.001, 10)
(0.001, 10)
(0.001, 1)
Std regularization
0.001
0.001
0.001
0.001
0.001
a
1
on the initial value of log in for Hand Pose and Reorient. The default value of seems to perform optimally (Fig.
0.4
Hand Pose
0.2
Reorient
0.0
-3
-2
Initial value of log(a)
Figure F1: Final reward as function the initial value the exploration standard deviation of Lattice, with period T 1.
A.8 Detailed reward and energy results of all experiments
We list the performance (energy and reward) SAC, gSDE-SAC and Lattice-SACe in the PyBullet locomotion tasks (Table T6 and the performance of PPO. gSDE-PPO and Lattice-PPO in the MyoSuite tasks (Tables T7T8 and TIO).
All results are averaged over 5 seeds and we report mean and standard error of mean.
A.9 Evolution of the dimensionality of the policy during training
In the main text analyzed how Lattice explores the environment (Section 6). As discussed in the main text, here we present the results across learning.
We studied how the dimensionality of the policy changes during training when the policy is trained with Lattice compared standard SAC and PPO. While the number relevant components similar at the beginning of the training, the policies trained with Lattice converge to actions that lic
19

---

Table T4: Parameters of PPO, gSDE-PPO and Lattice-PPO in Elbow pose, Finger pose, Hand pose
and Finger reach.
Task
Elbow pose
Finger pose
Hand pose
Finger reach
Algorithms
Parameters
PPO
Action normalization
Yes
Yes
Yes
Yes
Learning rale
0.000025
0.000025
0.000025
0,000025
Batch size
32
32
32
32
Gradient steps
128
128
128
128
Num epochs
10
10
Discount factor y
0.99
0.99
0.99
0.99
Entropy coefficient
0.0000036
0.0000036
0.0000036
0.0000036
Value function coefficient
0.84
0.84
0.84
0.84
GAE A
0.9
0.9
0.9
0.9
Clip parameter
0.3
0.3
0.3
0.3
Max gradient norm
0.7
0.7
0.7
0.7
Policy hiddens
[256, 256]
[256, 256]
[256, 256]
[256.
256]
Critic hiddens
[256, 256]
[256, 256]
[256, 256]
[256, 256]
Policy LSTM hiddens
256
256
256
256
Critic LSTM hiddens
256
256
256
256
Activation
ReLU
ReI.U
ReLUU
ReLU
gSDE
Init log std Full std matrix
-2
-2
-2
-2
No
No
No
No
Lattice
Init log std Full matrix
1
0
0
No
No
No
Std clip
(0.001,
10)
(0.001,
10)
(0.001,
10)
(0.001,
10)
Std regularization
0
0
0
a
1
lower dimensional manifold (Fig, F2 We hypothesize that this is the main reason why Lattice leads to more energy-efficient policies. Indeed, the energy consumption is similar for both Lattice and SAC/PPO initialization and diverges during training with Lattice achieving more energy-efficient policies; this result is also consistent across seeds(Fig. F3 Overall, the final policies trained with lattice have lower dimensionality, consistently across environments and random seeds (Fig. F4]
20

---

Table T5: Parameters of PPO, gSDE-PPO and Lattice-PPO in Hand reach, Baoding, Reorient and
Pen.
Task
Hand reach
Baoding
Reorient
Pen
Algorithms
Parameters
PPO
Action normalization
Yes
Yes
Yes
Yes
Learning rate
0.000025
0.000025
0.000025
0.000025
Batch size
32
32
32
32
Gradient steps
128
128
128
128
Num epochs
10
10
10
10
Discount factor y
0.99
0.99
0.99
0.99
Entropy coefficient
0.0000036
0.0000036
0.0000036
0.0000036
Value function coefficient
0.84
0.84
0.84
0.84
GAE À
0.9
0.9
0.9
0.9
Clip parameter
0.3
0.3
0.3
0.3
Max gradient norm
0.7
0.7
0.7
0.7
Policy hiddens
[256, 256]
[256,
256]
[256,
256]
[256, 256]
Critic hiddens
[256, 256]
[256, 256]
[256. 256]
[256, 256]
Policy LSTM hiddens
256
256
256
256
Critic LSTM hiddens
256
256
256
256
Activation
ReLU
ReLU
ReLU
ReLU
gSDE
Init log std Full std matrix
-2
-2
-2
No
No
No
No
Lattice
Init log std
0
0
0
Full std matrix
No
No
No
No
Std clip
(0.001, 10)
(0.001, 10)
(0.001,
10)
(0.001,
10)
Std regularization
0
0
1
Table T6: Detailed results in the PyBullet locomotion environments, Results are averaged over N=5 seeds.
Hopper Energy
Walker Frergy
Half cheetsh Energy
Humanuid Fremgy
Energy
Resant
Reword
Resenl
Reward
Revenl
SAC
0.25 0.01
3381 30
0.26 0.01
2417 106
0.27 11O
2741
0.25 2.0
2
0.12 s ILD
2122 s 169
SAC-gSDE period
0.23 0.0
3978 14
2356 17
0.26
2738 74
0.23 0.0
3191 125
0.12 1.0
2114 à 126
SAC-ESDF period
- 0.0
3963
0
2234 96
0.25 0.0
2746
0.26 + 0.01
2752
0.12.+0..
1917 + 120
SAC-gSDF cpionde
0.34 0.01
3706 48
0.00
2472 64
0.26 0.0
2822 32
0.24 0.24=00.
3061 =:
0.11 +..
2460 160
SAC Lattice (ouns)
0:2
3544
0.00
2610 78
0.29:000
2718
0.27 0.27
2500
0.11 1:00
2742
SAC- Latiic period cours)
0.25=00.
3926 78
0.20
2446 133
0.29::000
2541
129
0.26 0:0
2966 4:44
0.11 1:00
2798 à
SAC-L Litice peried (ours)
0.34 0.0
3686
0.00
2621 86
0.0
2641
121
0.26 0.0
3031 a
0.11 0:0
2358 77
SAC Lutike
96
0.22
2
0.28 0.0
130
0.25
2948
Table T7: Detailed results in MyoSuite environments: Elbow Pose and Finger Pose. Results
averaged over N=5 seeds.
Elbow pose Energy
Finger pose Energy
Reward
Solved
Reward
Solved
PPO
0,21 + 0.01
88.29
0.33
0.95 = 0.0
0.05 0.04
81.61 # 3.08
0.96 = 0.01
PPO-gSDE period 4
0.18 + 0.0
71.15
+ 3.84
0.84 a 0.03
0.02 0.01
47.68 + 11.88
0.78 # 0.07
PPO-Lattice (ours)
0.19 0.01
85.55 = 0.67
0.94 4 0.0
0.04 0.0
74.35 # 4,51
0.91 = 0.03
PPO-Lattice period 4 (ours)
0.27 + 0.01
62.57 = 1.35
0.77 : 0.01
0.05 0.02
52.57 0 9.0
0.78 = 0.05
Table T8: Detailed results in MyoSuite environments: Finger reach and Hand pose. Results are averaged over N=5 seeds.
Finger reach Energy
Hand pose Energy
Reward
Solved
Reward
Solved
PPO
0.2 + 0.02
242.72 + 15.53
0.2 0.02
0.04 0.0
-25.85 + 3.05
0.54 = 0.03
PPO-gSDE period 4 PPO-Lattice (ours)
0.07 0.02
257.97 = 23.81
0.22
0.04
-74.13
0.24 = 0.05
0.04 : 0.011
337.07 = 15.62
0.33
0.04
8.75
0.42 = 0.06
PPO-Lattice period 4 (ours)
0.05 + 0.0
206.46 + 52.45
0.18
0.03 0.0
.99,59 4
0.11 + 0.04
21

---

Table T9: Detailed results in MyoSuite environments: Hand reach and Baoding. Results are averaged over N=5 seeds.
Hand reach Energy
Baoding Energy
Reward
Solved
Reward
Solved
PPO PPO-gSDE period 4 PPO-Lattice (ours)
0.09 0.0
581.4 + 30.37
0.52 0.08
0.08 + 0.0
573.46 73.24
0.4 0.07
0.07 * 0.0
462.72 + 25.97
0.21 # 0.06
0.07 + 0.01
4
0.26 0.09
0.04 + 0.0
586.9 + 19,19
0.54 + 0.05
0.04 # 0.0
627.31 + 69.001
0.44 + 0.07
PPO-Lattice period 4 (ours)
0.07 = 0.0
556.1 # 11.14
+.43
0.07 0.01
471.16
0.29 0.1
Table T10: Detailed results in MyoSuite environments: Reorient and Pen. Results are averaged over N=5 seeds.
Reorient Energy
Pen Energy
Reward
Sulved
Reward
Solved
PPO PPO-gSDE period
0.04 0.01
233.74 # 16.83
0.33 = 0.05
0.07 0.01
140.83 1 9.77
0.16 = 0.03
0.04 0.01
187.99 27.04
0.2 0.08
0.07 0.0
193.85 13.42
0.42 0.06
PPO-Lattice (ours)
0.01 0.0
283.07 # 14.99
0.48 = 0.05
0.04 + 0.0
169.72 15.98
0.29 0.06
PPO-Lattice period (ours)
0.03 0.01
250.13 # 13.6
0.38 0.04
0.06 a 0.0
191.8 + 15.75
0.39 # 0.08
Table T11: SAC trained subset of the MyoSuite environments: Finger Pose, Hand Pose, Reorient and Pen. Results are averaged over N=3 seeds.
Finger pose Energy
Hand puse
SAC
Solved
Energy
Solved
Energy
Solved
Energy
Solved
0.04
0.96
0.04
0.84
0.03
SAC-gSDE episode
0.04 0.0
0.96 = 0.0
0.04 0.0
0.84 a: 0.0
0.07 a 0.0
0.5 a 0:03
0.07 0.0
0.17 a 0.02
SAC-Lattiee episode (ours)
0.92
0.73 0.02
0.06
0.03
A
Traliing Latioe
1.0
Lillos Sec
Explained varlance
0.8
0.8
N. of components
14:
Seedt
Seedt
M
0.6
12
Seed
Seedd 5
Explannd
0.4
0.4
20
0.2
1a
15
16
12 5.2 15
N² of compunents
Nº of components
Training step [M]
Explained variance
1.0
Tralking
o.n
1.0-
N. of components
30
Lattee
Seed
Seed 5
5:2
MM
0.6
20
Smert
Soed
0.4
M2
3.8
0.4
20
0.2
0.0
N" of components
20 40
N" of components
20
40
0 0.4 2 5.2 8812.415.6 20
Training [M]
Figure F2: A Both in the Humanoid (top) and in the Reorient (bottom) motor control tasks, policies trained with Lattice require fewer principal components to explain the variance the actions. The graphs, generated by testing policies at different stages of the training, highlight how the number of components decreases almost uniformly throughout the training, both with Lattice and with independent action noise. However, the effect is much stronger with Lattice. Result for seed. B Number of principal components that explain at least 90% of the variance with respect to the training step for different seeds (N=5). For both Humanoid (top) and in the Reorient (bottom) tasks, Lattice reaches lower principal components during training compared to SAC/PPO.
22

---

HUMANDI
MG
020
Ban 10
REORIENT
Figure F3: Distribution of the energy consumption across training episodes for different seeds (columm) in the Humanoid (top) and Reorient (bottom) task. For both, the energy consumption is similar at the beginning and (relatively quickly diverges during training. Once the policy is trained, LATTICE shows a more anergy-efficient policy.
MyoSuite environments
PyBullet environments
Ratio [eff dim/DoF]
Ratio [eff dim/DoF]
0.8
Lattice-SAC SAC
1.0
Lattice-PPO
PPO
0.6
0.8
0.4
0.6
0.2
0.4
FingerPose
FingerReach
ElbowPose
HandPose
HandReach
Pen
Reorient
Baoding
HalfCheetah
Hopper
Walker
Ant
Humanoid
Figure F4: Analysis of the effective dimensionality of the policies obtained with Lattice-SAC and Lattice-PPO versus standard SAC and PPO in the PyBullet (left) and Myosuite (right) environments.
The dimensionality estimated by running 100 test episodes per environment random seeds), com- puting the principal component analysis of the actions counting how many principal components necessary to reach 90% cumulative explained variance, rescaled by the number action components. We can see that, consistently across most environments and random seeds (dotted lines in the plot), Lattice leads to lower dimensional policies.
23


Reviews:
[{"review_id": "EyQhGPHR4p", "reviewer": "Reviewer_9Q2U", "summary": "The paper presents an extension to the previously introduced RL exploration method gSDE called Lattice. The main idea is to add noise to the full covariance matrix (instead of only onto the diagonal) of the multivariate Gaussian used for exploration. In particular, the authors multiply state dependent noise with the weights of the last (linear) noise layer of the policy network (so that the resulting action probability distribution is known) and add this term to the weights and action dependent noise (gSDE). In this manner, the authors achieve correlation between action components. Both, the action and state dependent noise terms are sampled periodically, where the period determines the time-correlation in the noise terms.", "questions": "Please address the points in the weakness section.", "limitations": "The authors appropriately addressed the limitiations of their work. ", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- thorough experimental validation on classical RL benchmarks and high dimensional muscle RL benchmarks (myosuite)\n- detailed implementation details and discussion about how Lattice works\n- paper is easy to read and understand", "weaknesses": "- no comparison to other exploration methods such as [11], [18] or [30]\n- contribution seems rather empirical (no theoretical contribution found)\n- Lattice results for classical RL benchmarks (ant, hopper, walker, half cheetah, humanoid) not very convincing even in comparison to previous method (gSDE).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the precise summary and for praising the clarity of our exposition. We would like to address their concerns:\n\n*no comparison to other exploration methods such as [11], [18] or [30]*\n\nIn our manuscript we have compared Lattice to SAC/PPO with two forms of action noise, standard “white” gaussian noise and state-dependent, time-correlated action noise (gSDE), because Lattice is proposed as an improvement over them. In fact, any Reinforcement Learning algorithm using white noise or gSDE can simply swap this action noise with Lattice, keeping all the other implementation details unmodified. \nMethods such as DEP-RL [18] and curiosity-driven exploration [30], which we cite in the related work section, make use of a separate exploratory policy [18] or curiosity as an intrinsic reward [30] to promote exploration. However, both these methods are paired with some form of action noise (specifically: white noise), which we argue makes them a complementary approach to Lattice, rather than a baseline. A combination of such exploration methods with Lattice action noise is an interesting direction for future research. \nThe very recent paper “Pink noise is all you need” [11] (ICLR 2023), instead, proposes an alternative action noise. Indeed, [11] shows that pink noise is often better than Ornstein-Uhlenbeck (OU) noise (which, however, was already outperformed by gSDE [1]). Pink noise, as formulated by the authors of [11], is limited to off-policy RL. Therefore, we compared its performance with that of Lattice-SAC and not with Lattice-PPO. Here we report the results we have obtained with SAC + pink noise in the 5 PyBullet environments considered in the paper (Ant, Walker, Hopper, Half Cheetah and Humanoid, 5 seeds per environment):\n\n|                            | Ant        | Hopper     | Walker     | Half cheetah   | Humanoid   |\n|:---------------------------|:-----------|:-----------|:-----------|:---------------|:-----------|\n| SAC                        | 3381 ± 30  | 2417 ± 106 | 2741 ± 81  | 2934 ± 27      | 2122 ± 169 |\n| SAC-gSDE episode           | 3796 ± 48  | 2472 ± 64  | 2822 ± 32  | **3081 ± 93**      | 2460 ± 160 |\n| SAC-pink                   | 3648 ± 53  | 2438 ± 73  | 2580 ± 164 | 2999 ± 71      | 2116 ± 176 |\n| SAC-Lattice (ours)         | 3544 ± 212 | **2610 ± 78**  | 2718 ± 92  | 2900 ± 67      | **2742 ± 77**  |\n| SAC-Lattice episode (ours) | 3845 ± 86  | **2586 ± 23**  | 2661 ± 130 | 2948 ± 24      | **2901 ± 211** |\n\nWe highlighted the results when the confidence intervals of a Lattice experiment and the non-Lattice ones are disjoint (and vice-versa). The results and the learning curves are also included in the additional 1-page pdf. Overall, we find that pink noise is a good alternative to white noise, as it produces strong results. However, as it is also a method to introduce time correlation, its performance does not improve over gSDE. We believe that these experiments corroborate our claim that correlation across action components, the distinguishing trait of Lattice, can provide a boost in performance in high-dimensional environments, improving exploration in a different way from existing techniques to add correlated noise.\n\n*contribution seems rather empirical (no theoretical contribution found)*\n\nWe partially agree with the reviewer’s assessment, the results of this paper are mostly experimental. However, we believe the mathematical formulation of non-diagonal, state-dependent action noise for on-policy and off-policy RL algorithms is also of interest for the RL community, as the analytical derivation of the probability distribution (Appendix A1), of the noise rescaling factor (Appendix A2) and on the conditions to prevent the singularity of the covariance matrix (Appendix A3) were important to justify our implementation.  \nWe also want to point the reviewer to the analytical derivation of the effective variance for the reduced arm model, which is of course just a toy model (Appendix A1).\n\n*Lattice results for classical RL benchmarks (ant, hopper, walker, half cheetah, humanoid) not very convincing even in comparison to previous method (gSDE).*\n\nWe agree with the reviewer’s assessment that the performance of gSDE is as strong as that of Lattice in most PyBullet environments, with the notable exception of the Humanoid. In the manuscript, we argue that these results should not surprise the reader, as especially in the four simpler, somewhat low-dimensional locomotion environments (Ant, Hopper, Walker and Half Cheetah) we do not see a motivation why correlated action noise should be preferable. These simpler locomotion environments do not have a large observation/action space and all the motors can contribute to the forward motion, so that the redundancy is limited. We included all environments to prove that Lattice provides strong and reliable results also in environments where action correlation does not improve performance. In the Humanoid environment, where the motor redundancy is high, Lattice substantially outperforms white noise, pink noise and gSDE. We believe that it is state of the art in SB3. \nImportantly, we demonstrate that for high-dimensional, redundant tasks Lattice achieves excellent performance (e.g. > 25 degrees of freedom). Specifically, for that we focussed on the benchmarking tasks of the MyoSuite, where Lattice reaches strong results. \n\n"}, {"review_id": "RXDXyCPcUS", "reviewer": "Reviewer_mQQQ", "summary": "The paper proposes adding time- and actuator-correlated noise instead of simple gaussian noise to improve exploration in RL. \n", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and easy to read.\n2. The approach can be easily applied to any RL algorithm. And can be thought of as a generalization of earlier work in noise for exploration (gSDE).\n3. The improvements on 4 out of 8 tasks in MyoSuite is encouraging.\n", "weaknesses": "1. Is there any intuition as to which tasks are suitable for your method? It appears completely random that PPO outperforms everyone on handpose which is (I expect), an implicit subgoal in tasks like Baoding and Pen. Hence it is unclear why lattice underperforms on hand pose, elbow pose.\n2. A nitpick: “ought” is an ambiguous word to use in the abstract and should be replaced.\n3. Another nitpick: the arrows in the energy saving vs reward gain plot is not very clear. What do they indicate?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and score. \n\n*Is there any intuition as to which tasks are suitable for your method? It appears completely random that PPO outperforms everyone on handpose which is (I expect), an implicit subgoal in tasks like Baoding and Pen. Hence it is unclear why lattice underperforms on hand pose, elbow pose.*\n\nWe believe there is an underlying logic, why Lattice performs strongly on the Humanoid and the object manipulation tasks. Lattice takes advantage of learnt correlations across actuators to direct the exploration towards task-relevant dimensions. Compared to Hand Reach, Baoding, Pen and Reorient, which use the same hand model (and therefore action space) as Hand Pose, we argue that Hand Pose allows fewer possibilities of exploiting “motor redundancies”. In fact, the Hand Pose task demands the agent to reach (many) target poses and thus, for each target pose many target angles with all the hand joints (therefore: 23 target values). This task does not allow learning movements involving a limited number of muscles moving always in the same way, like, e.g., baoding, where the balls always need to perform a rotation, without compromising the reward. For this reason, Lattice is not suitable for this task, as it is also not better than standard noise in Elbow Pose and Finger Pose. An empirical rule to decide whether to use Lattice vs uncorrelated noise is to compare the size of the action space with the size of the expected “action dimensionality” (Additional pdf: Figure 2).\n\n*A nitpick: “ought” is an ambiguous word to use in the abstract and should be replaced.*\n\nWe have rephrased the sentence as follows:\nWhile this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems.\n\n*Another nitpick: the arrows in the energy saving vs reward gain plot is not very clear. What do they indicate?*\n\nFor clarity of presentation, we indicate that the 1st quadrant of the cartesian graph indicates an improvement in reward and energy saving of Lattice-PPO over PPO, while the 3rd quadrant a deterioration. We believed that it would not be clear at first glance that “higher is better” in a metric related to energy, hence we added the two arrows with the sole purpose of making the interpretation of the graph more obvious.\n"}, {"review_id": "kSlfHhmxkp", "reviewer": "Reviewer_p1VW", "summary": "This paper proposes a new method to conduct action exploration based on the latent space representation during the reinforcement learning process. The disturbance noise in this case is based on gSDE, and the further \"interval-hold\" mechanism makes the noise time-correlated. The authors conduct experiments on both classic pybullet benchmarks and musculoskeletal control environments. They show in most cases, the new exploration method can give faster reward learning compared to other baselines (SAC, gSDE-SAC, PPO, gSDE-PPO).", "questions": "1. In Half cheetah experiment, the standard deviation for the reward curves is much smaller compared to the rest benchmarks. What could be the potential reason for it?\n2. It seems like the proposed algorithm tends to introduce more uncertainty in the reward learning (an observation from Fig. 3-4) in most cases. Does that mean it is hard to get a very stable training from the proposed latent exploration?\n3. A bit curious about the 10,000 hours of training. How many GPUs do you use? Do you think there is a way to reduce the workload to make it more friendly for normal AI labs in universities to conduct this kind of research?", "limitations": "The authors have well discussed the limitations in section 7. For some of the limitations (training overhead), the authors put some thought and suggest new promising work directions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to read.\n2. The idea of injecting time-correlated noise is simple, novel, yet effective.\n3. Extensive results comparison on a wide range of control tasks.", "weaknesses": "1. Lack of exploration algorithms comparison: the authors mention many exploring techniques [20-37] in the related work section, but have not compared to any of those, nor add on top of those techniques to boost the performance.\n2. Lack of RL algorithms implemented: I will be eager to see how the proposed latent exploration idea can work on other RL algorithms (TRPO, TD3, etc).\n3. Lack of ablation studies to provide rationale about: (1) why learn noise magnitude rather than a fixed noise magnitude, (2) the combination of the clipping parameters\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and score.\n\n*Lack of exploration algorithms comparison: the authors mention many exploring techniques [20-37] in the related work section, but have not compared to any of those, nor add on top of those techniques to boost the performance.*\n\nWe compared Lattice with two forms of action noise, standard “white” gaussian noise and state-dependent, time-correlated action noise (gSDE), because Lattice is proposed as an improvement over them. In fact, any Reinforcement Learning algorithm using white noise or gSDE can swap it with Lattice, keeping all the other implementation details unmodified. Methods such as DEP-RL [18] and curiosity-driven exploration [30], which we cite in the literature review, make use of a separate exploratory policy [18] or modifying the reward function [30, 31, 33] to promote exploration. However, both these methods still use action noise (specifically: white noise), which we argue makes them a complementary approach to Lattice, rather than a baseline. The same consideration is valid for curriculum learning [34, 35, 36, 37], where individual tasks necessitate local exploration perturbing the policy. A combination of such exploration methods with Lattice action noise is an interesting direction for future research. The paper “Pink noise is all you need” [11] (ICLR 2023), instead, proposes an alternative action noise, which is often better than Ornstein-Uhlenbeck (OU) noise. In the global answer and in the additional pdf (Figure 1), we report the results we have obtained with SAC + pink noise in the 5 PyBullet environments considered in the paper. The results corroborate our claim that correlation across action components, the distinguishing trait of Lattice, can provide a boost in performance, improving exploration in a different way from existing techniques to add correlated noise.\n\n*Lack of RL algorithms implemented: I will be eager to see how the proposed latent exploration idea can work on other RL algorithms (TRPO, TD3, etc).*\n\nWe agree with the reviewer that it would be interesting to test Lattice with other RL algorithms, such as TRPO and TD3, since Lattice can be implemented with all of them. Because of the high computational load of the experiments presented in this paper, we were forced to make some decisions regarding which experiments to run.\nFor the on-policy algorithm, we chose PPO because of its popularity. For off-policy algorithms, TD3 is a valid alternative to SAC. In the considered PyBullet environments, TD3 tends to perform slightly worse than SAC [1], and was found to generally perform worse than SAC with time-correlated noise [2].\n\n*Lack of ablation studies to provide rationale about: (1) why learn noise magnitude rather than a fixed noise magnitude, (2) the combination of the clipping parameters*\n\nThe choice of only considering a learnt noise magnitude is related to the previously outlined experimental setup, involving only PPO and SAC. Both PPO and SAC learn a stochastic policy and, in popular implementations of these two algorithms, the noise magnitude is learnt [1]. We did not modify this characteristic of the implementation. \nThe clipping from below was added to prevent the singularity of the covariance matrix (see Appendix A4). The upper limit to prevent the divergence of the covariance matrix. In the experiments we performed, the upper limit was not reached, as the variance decreases over the training. We still decided to include it in the implementation, in case future work requires limiting the magnitude of the variance.\n\n*In Half cheetah experiment, the standard deviation for the reward curves is much smaller compared to the rest benchmarks. What could be the potential reason for it?*\n\nHalf-cheetah, differently from the other locomotion environments of PyBullet, is statically stable. This means that episode length is mostly 1000 (the maximum) since the beginning of the training, and episodes rarely last fewer steps. In the other environments, a large fraction of the reward variability is caused by the variable episode length in those environments where the agent needs to actively stabilize. \n\n*It seems like the proposed algorithm tends to introduce more uncertainty in the reward learning (an observation from Fig. 3-4) in most cases. Does that mean it is hard to get a very stable training from the proposed latent exploration?*\n\nWe believe that the uncertainty is caused by the higher variance in behavior induced by Lattice (as showcased in the motivating example), which in some cases makes the learning curve non-monotonically increasing. We would like to remark that we have not found a significantly different variability in the final performance across random seeds.\n\n*A bit curious about the 10,000 hours of training. How many GPUs do you use? Do you think there is a way to reduce the workload to make it more friendly for normal AI labs in universities to conduct this kind of research?*\n\nThe experiments were performed over several months. We dedicated 3 computers to the experiments over the development of the project. For the final experiments, we scaled the simulations to a GPU cluster. We recognize that the computation employed for this project is considerable. Most computational resources were used for the musculoskeletal simulations (Approx. 90% MyoSuite vs 10% PyBullet). One limitation of the environments used in this project is that they do not take advantage of the GPU for acceleration. Therefore, the CPU was used for the physics simulation and the GPU for the policy updates. The CPU was the bottleneck. Future RL research might greatly benefit from environments such as IsaacGym, which can exploit GPU acceleration and reduce the simulation time. \n\n[1] Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., & Dormann, N. (2019). Stable baselines3\n\n[2] Eberhard, Onno, et al. \"Pink noise is all you need: Colored noise exploration in deep reinforcement learning.\" ICLR 2023\n\n"}, {"review_id": "gZG7LZgZq7", "reviewer": "Reviewer_Q9SL", "summary": "This paper proposes an exploration method that adds noise to the last linear layer (latent) of the policy network to incorporate time-dependent noise. The method is tested on locomotion tasks and evaluated on the metrics of performance and energy consumption. Evaluation is done on PPO and SAC base algorithm and an existing time-correlated baseline, gSDE.", "questions": "It is unclear why the proposed method enabled energy-saving policies compared to another time-correlated baseline. The comparison is conducted between PPO/SAC and not with gSDE (time-correlated). It is unclear how much the injected noise in the latent space contributes compared to the time-correlated component.\n\nIn Figure 4, the results are mixed. It is important to understand why time correlation sometimes worsens performance. It might be interesting to consider whether the proposed task requires independent or time-correlated noise.\n\nThe description of Figure 6 is currently difficult to understand. Simplifying the description would make it easier to comprehend the implications of the energy comparison.\n\nThe main contribution of the proposed method lies in the addition of noise to the latent space. Therefore, it is crucial to comprehend how this component is implemented. In terms of empirical return, the performance of the proposed method is mostly similar to the baseline. However, it is important to investigate how the introduction of noise in the latent space contributes to the observed energy consumption improvements. This information would provide a better understanding of the implications of the proposed method.\n", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Tackles an important problem of exploration in tasks with continuous action.\n- The proposed method is rather simple and presented well.\n- Thorough experiments with necessary details and hyperparameters are presented.\n- Performance is evaluated on return and energy conservation which is important for practical implications.\n", "weaknesses": "- Unclear why the proposed method enabled energy-saving policy compared to another time-correlated baseline. \n- Energy comparison with gSDE is missing, which makes it difficult to evaluate how the noise injected in the latent layer is helping.\n- Few cases, the proposed method outperforms the existing method in return performance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and strong evaluation (soundess, presentation and contribution) and excellent questions. We hope to be able to provide sufficient motivation to reconsider the assigned score.\n\n*It is unclear why the proposed method enabled energy-saving policies compared to another time-correlated baseline. The comparison is conducted between PPO/SAC and not with gSDE (time-correlated). It is unclear how much the injected noise in the latent space contributes compared to the time-correlated component.*\n\nWe would like to remark that the comparison presented in figure 5 represents Lattice-PPO without time correlation (namely, with the exploration matrices resampled at every step). We made this choice because the time correlation might also have an impact on the energy consumption (as the reviewer suggests), and we aimed to isolate the effect of the action correlation introduced by Lattice. Furthermore, time correlation has proven quite ineffective in combination with PPO (as it was previously found in [1]). Therefore, because of the often low performance of time-correlated exploration in these locomotion environments, we believe the most meaningful comparison is between vanilla PPO and Lattice-PPO with no time correlation and we included it in the main section of the manuscript. The details about the performance and energy consumption of all policies are included in table T6-T10 of the appendix. \nIn short, in the Humanoid environment and in the MyoSuite tasks, the best performing policies using Lattice are always more efficient than the best policies using white noise or gSDE exploration. With gSDE, when time correlation improves energy efficiency, it does so at the cost of a large fraction of the reward. For example, gSDE period 4 finds energy-efficient policies in Elbow Pose and Hand Pose, but the reward is much lower than without time correlation. Importantly, Lattice can find the most energy-efficient policy across many environments without compromising the reward (Humanoid, Finger Reach, Baoding, Hand Reach, Reorient, Pen)\n\n*In Figure 4, the results are mixed. It is important to understand why time correlation sometimes worsens performance. It might be interesting to consider whether the proposed task requires independent or time-correlated noise.*\n\nWe empirically find that PPO does not benefit from time-correlated exploration as much as SAC, confirming the findings of [1]. We believe that this is at least in part due to the on-policy nature of PPO. Previous results on time-correlated noise (gSDE [1], Ornstein-Uhlenbeck noise [2] and pink noise [3]) focus on off-policy RL algorithms. Lattice, instead, in its non-time-correlated version (period 1), is an effective addition also to on-policy PPO, as showcased in the MyoSuite environments.\n\n*The description of Figure 6 is currently difficult to understand. Simplifying the description would make it easier to comprehend the implications of the energy comparison.*\n\nWe have rephrased and extended the caption in the following way:\n“Figure 6: A Left: Graph of the fraction of noise allocated to each group of action components by a stochastic policy trained in the Humanoid environment with SAC and Lattice-SAC. In the Humanoid, Lattice tends to focus most of the variance on the task-relevant actuators (legs). Middle: Distribution of the energy consumption. Each bar represents the number of test episodes falling in the corresponding energy consumption interval. Right: Cumulative explained variance of the actions’ principal components, computed from a dataset of 100 test episodes per seed. Shaded area represents 95% confidence interval across training seeds. More principal components are required to explain the same fraction of variance in SAC and PPO versus Lattice-SAC and Lattice-PPO. B Heatmap of the correlation matrix for the action space (left column) and noise of the action (right column) for SAC (top row) and Lattice-SAC (bottom row). C-D Same as A-B but analyzing a policy trained on the MyoSuite Reorient task with PPO and Lattice-PPO. Lattice tends to induce sharper correlation between action components. Local patterns of correlation between action components can be recognized in the noise covariance matrix.\n“\n\n*The main contribution of the proposed method lies in the addition of noise to the latent space. Therefore, it is crucial to comprehend how this component is implemented. (...)*\n\nWe hope our previous answers clarified the differences between the policies trained with Lattice versus uncorrelated action noise and how they influence energy consumption. We remark that these effects are not a consequence of the time correlation, but rather of the action component correlation, making them a distinguishing property of Lattice. The motivating example provides a high-level intuition as to why Lattice might cause an energy saving, while improving exploration. In highly redundant motor systems, different policies can produce the same observed behavior. For example, in a flexor-extensor system, the angular acceleration is (in first approximation) proportional to the difference between the activation of the two muscles. If both muscles are activated together, the resulting acceleration is 0, but the body spends energy to maintain this active equilibrium. When the environment is explored by randomly activating all muscles independently, it is more likely to obtain policies with non-zero base activation of the muscles. If the muscles activations are perturbed according to their synergy, it will be more likely to find policies with low base activation of the muscle, improving efficiency.\n\n[1] Raffin, Antonin, et al. \"Smooth exploration for robotic reinforcement learning.\" CORL 2022.\n\n[2] Lillicrap, Timothy P., et al. \"Continuous control with deep reinforcement learning.\" arXiv 2015.\n\n[3] Eberhard, Onno, et al. \"Pink noise is all you need: Colored noise exploration in deep reinforcement learning.\" ICLR 2023."}, {"review_id": "4qXI39n91o", "reviewer": "Reviewer_dZAS", "summary": "This paper explores the exploration problem in RL algorithms by considering the action-space correlation. The approach is motivated by the fact that over-actuated systems with high-dof action space require correlated control actions. Such action-space correlation is ignored by action-space random perturbation in a standard RL algorithm which results in inefficient exploration. This work proposes to add random perturbation in the latent space in the last layer of the policy network to impose the proper action structure in the perturbation distribution. The presented approach can be potentially integrated into most existing RL algorithms to enhance the exploration capability and is shown to perform great, especially in high-dim musculoskeletal environments.", "questions": "1. The two images in Figure 2(B) is hard to parse. Are those images obtained from the action perturbation training or the latent perturbation approach?\n\n2. In the MyoSuite environments (Figure 4), I noticed Lattice-PPO (T=1) consistently works better than Lattice-PPO (T=4) except for the **Pen** task. Does it mean time-space correlation is not important in general? Do you have any idea on why T=1 works better than T=4 and what is special in **Pen** task that makes T=4 preferable?\n\n3. The proposed Lattice approach benefits the action structure learned by the policy and imposes this learned structure to the action perturbation. However, on the other side is this also a limitation of this approach? Will Lattice suppress the emergence of novel control modes and result in a local-optimal policy? If this is not a problem, which part of the algorithm is the key to addressing this issue?\n\n4. It would be valuable to also ablate the $P_ax$ term in the method. Is this the key to addressing the last point?\n\n5. gSDE (T=4) seems to work consistently worse than other approaches (even worse than vanilla PPO) in all MyoSuite environments. It would be interesting to see the performance of gSDE-PPO (T=1) which differs from PPO on whether having state-dependent perturbation. If gSDE-PPO (T=1) does not work well either, then it might not be reasonable to have $\\alpha$ to toggle between gSDE and Lattice. Instead, using $\\alpha$ to toggle between regular RL perturbation and latent space perturbation might work better?\n\n6. Currently SAC and its variants are used for the Pybullet environments while PPO and its variants are used for MyoSuite environments. It would be also great to provide all of those algorithms on all benchmark problems (maybe in Appendix).", "limitations": "As mentioned in point 3 in the **Questions** section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of the work is clear and intuitive. The provided toy example is a very illustrative example to motivate the idea of using correlated action perturbation.\n\n2. The proposed method is concise and seems to be universal in that it can be easily integrated into most RL frameworks to improve exploration efficiency.\n\n3. The experimental results are convincing to show the effectiveness of the proposed Lattice method.", "weaknesses": "1. The selection of the initial logstd for the $S$ matrices seems to be not straightforward and might need extra experiments to determine.\n\n2. The presented Lattice method relies on a good learned policy to impose the action correlation structure, which might result in local-optimal policy (also see point 3 in the next section).\n\n3. The exposition of the paper can be further improved to be more clear and consistent.\n\n   a) The second motivating example (the one in simulation) is confusing. \n\n   b) In Line 148, $N_l$ is used for the latent space size, while in Figure 1, $N_x$ is used for latent space size.\n\n   c) Line 168: $\\epsilon_a = P_a a \\Rightarrow \\epsilon_a = P_a x$\n\n   d) It was never mentioned in the main paper about the separate action space perturbation until $\\Sigma_a$ appears in Line 152.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and positive evaluation.\n\n*Selection of log_std*\n\nWe have run additional ablation experiments on Hand Pose and Reorient, where we have tested initial values for log_std between -3 and 2. The following results show that values around 0 are optimal:\n\n|std_init |-3|-2|-1|0|1|2|\n|:-|:-|:-|:-|:-|:-|:-|\n|Reorient|0.1815 ± 0.0411|0.3058 ± 0.0624|0.3051 ± 0.0581|0.4824 ± 0.0468|0.2179 ± 0.0808|0.0442 ± 0.0232|\n|Hand Pose|0.0109 ± 0.0094|0.0841 ± 0.0728|0.2006 ± 0.0726|0.4303 ± 0.0528|0.4184 ± 0.0638|0.0001 ± 0.0001|\n\nThanks to the rescaling described in Appendix A3, these values should perform well with different hidden layer sizes.\n\n*Clarification of Figure 2*\n\nWe modified the explanation for improved clarity:\n\"To corroborate this finding about the properties of latent versus action perturbation, we considered a policy network trained to control a realistic musculoskeletal arm model, implemented in MyoSuite [17], featuring three flexor muscles and three extensor muscles. The policy can learn to accurately reach a target angle in every episode, and while doing so learns to alternately activate the extensor or the flexor muscle group (Fig. 2B, left). The training procedure is detailed in Appendix A1. We collected a dataset of 100 episodes where the latent state of the policy network was perturbed with gaussian noise. While applying independent random noise to the muscle activations produces a diagonal covariance matrix, perturbing the latent state of the network leads to a full covariance matrix. As hypothesized, the last layer of the policy network indeed learned to distinguish between agonist and antagonist muscles, namely, the covariance shows positive correlation among muscles of the same group (Fig. 2B, right). We then tested whether this covariance structure leads to higher variance in the task space (joint angle). We collected 100 episodes where, at each step, we computed an action with action space perturbation and one action with latent state perturbation. For consistency of evaluation, the perturbation in the action space was sampled with the same variance magnitude induced by the latent perturbation. We executed each pair of actions in two cloned environments and compared the variance of the joint angles. The latent state perturbation introduces higher variance in the joint angles (Fig. 2C), thus driving more diverse kinematics and thus wider exploration.\"\n\nThe action correlation matrix (heatmap on the left) is relative to a policy network trained with Lattice. We will further clarify the training procedure in Appendix A1.\n\n*Remarks b) - d)*\n\nIn the updated manuscript, we fixed the typos and we specified that we also consider an independent action noise components with diagonal covariance Sigma_a.\n\n*Time correlation in the MyoSuite experiments*\n\nAs previously found in [1], time correlation paired with PPO is less effective than when it’s paired with SAC. Our experiments confirm this general trend for muscle-skeletal models. The Pen environment is a notable exception. We speculate that this might be related to the active stabilisation required to balance the pen, which is usually held between two or three fingers. In the other object manipulation environments (Baoding and Reorient), gravity is countered by keeping the palm below the object. Time-correlated noise might help avoiding the pen to fall, preventing an early termination of the episode.\n\n*Ablation of $P_a x$*\n\nLatent noise alone might not be enough to learn good policies, as it might excessively bias the noise with correlation across action components which might not be ideal, especially in the early stages of the training. We ran an ablation, where we suppress the Pax term of the state-dependent noise. We limited this analysis to episodic updates of the perturbation matrix in the PyBullet environments, because this experiment can be performed in about 2 days (5 random seeds per environment). We report the average reward ± standard error:\n| |Ant|Hopper|Walker|Half Cheetah|Humanoid|\n|:-|:-|:-|:-|:-|:-|\n|$P_a x$|**3846 ± 87**|**2587 ± 24**|**2662 ± 130**|**2948 ± 25**|2902 ± 212|\n|No $P_a x$|938 ± 70|139 ± 91|365 ± 58|-904 ± 337|2960 ± 129|\n\nUnsurprisingly, the performance is very low in Ant, Hopper, Walker and Half Cheetah, showing that independent action noise is necessary for the correct exploration of these environments. Maybe surprisingly, the performance is excellent in the Humanoid, where the perturbation of many degrees of freedom might contribute to provide enough variability to avoid early convergence to a local minimum. \n \n*Performance of gSDE period 1 in MyoSuite: correlated vs. state-dep noise*\n\nWe considered the three pose environments of MyoSuite, as Lattice was on par or a little worse than standard exploration, to verify that the lower performance was due to the introduction of correlated noise, rather than state-dependent exploration. We ran gSDE with period=1 (no time correlation, but state-dependent noise) in Elbow Pose, Finger Pose and Hand Pose (3 seeds per environment). We report the results in terms of solved fraction ± standard error:\n\n| |Elbow Pose|Finger Pose|Hand Pose|\n|:-|:-|:-|:-|\n|gSDE period 1| 0.9014 ± 0.0219|**0.9843 ± 0.0016**|**0.5248 ± 0.0475**|\n|Lattice period 1|**0.9385 ± 0.0043**|0.9137 ± 0.0267|0.4184 ± 0.0638|\n\nThese results support the fact that toggling between gSDE and Lattice can be useful also when the period is 1. In fact, alpha=0 improves Finger pose and Hand pose, bridging the gap with white noise.\n\n*Additional experiments: SAC in MyoSuite and PPO in PyBullet*\n\nWe propose to run at least a subset of the proposed experiments for the camera-ready version, as we do not have enough computational resources to satisfy this request in the rebuttal period. \n\n[1] Antonin Raffin, Jens Kober, and Freek Stulp. \"Smooth exploration for robotic reinforcement learning.\" CORL 2022.\n"}]

Author rebuttal:
This paper proposes a simple and easy-to-implement idea for exploration in reinforcement learning by exploiting the implicit knowledge in the weights of the policy network to drive exploration.  We thank all reviewers for their assessment. We think the quality of our paper improved due to your comments. We provide specific answers for each reviewer and, here, raise two topics here that were mentioned by at least two reviewers. 

Additional baselines: We assessed the capability of Lattice by comparing it to a new baseline exploration noise, pink noise (just published in ICLR by Eberhard et al.), which has been shown to be effective in combination with SAC for continuous control tasks (we report the updated learning curves in Figure 1 of the additional PDF). Pink-SAC’s performance is similar to that of gSDE-SAC. However, the correlation across actuators introduced by Lattice boosts performance for the Humanoid and, to a lesser extent, for the Hopper, where it outperforms standard SAC, SAC with pink noise and SAC with gSDE. We report the table with the test performance of SAC, Pink-SAC, time-independent Lattice-SAC and episodic gSDE-SAC and Lattice-SAC:
|                            | Ant        | Hopper     | Walker     | Half cheetah   | Humanoid   |
|:---------------------------|:-----------|:-----------|:-----------|:---------------|:-----------|
| SAC                        | 3381 ± 30  | 2417 ± 106 | 2741 ± 81  | 2934 ± 27      | 2122 ± 169 |
| gSDE-SAC episode           | 3796 ± 48  | 2472 ± 64  | 2822 ± 32  | **3081 ± 93**      | 2460 ± 160 |
| Pink-SAC                  | 3648 ± 53  | 2438 ± 73  | 2580 ± 164 | 2999 ± 71      | 2116 ± 176 |
| Lattice-SAC (ours)         | 3544 ± 212 | **2610 ± 78**  | 2718 ± 92  | 2900 ± 67      | **2742 ± 77**  |
| Lattice-SAC episode (ours) | 3845 ± 86  | **2586 ± 23**  | 2661 ± 130 | 2948 ± 24      | **2901 ± 211** |

Why does Lattice not improve the results in all the 5 locomotion tasks, but just shows a clear performance increase in the Humanoid?
As suggested by the design of Lattice, we hypothesize that it can improve over baselines when high-dimensional morphologies with redundant degrees of freedom should be controlled (e.g. musculoskeletal systems, or the humanoid). We corroborate this in a novel analysis. We computed the effective (linearized) dimensionality of the learnt policies with respect to the size of the action space (additional pdf: Figure 2). We did this by evaluating the 5 policies trained in each environment for 100 test episodes. We ran the principal component analysis of the actions and computed the cumulative explained variance. We defined the effective dimensionality of a policy as the number of principal components necessary to achieve 90% explained variance.  Lattice tends to discover lower dimensional policies, especially for larger action spaces, which proves to be beneficial for the reward and the energy saving, whereas it performs as good as PPO/SAC when the dimensionality of the actions is low, or when the target space requires to control independently almost all joints. Importantly, Lattice is on par in those cases, so that it could be used as a drop-in replacement. 

Overall, our work pioneers the study of exploratory noise with correlation across actuators in Reinforcement Learning. The experiments with Lattice show for the first time that biasing the action noise with the motor synergies, learnt by the policy itself, leads to specific characteristics of the control policy. The policies trained with Lattice have lower effective dimensionality, meaning that they output actions which can be embedded in a lower-dimensional subspace than the actions output by a policy trained with white or time-only correlated noise. This effect is consistent across many different environments. In those environments with task-irrelevant degrees of freedom (Humanoid) and in the presence of overactuation (musculoskeletal systems), especially in object manipulation environments, this results in higher learning efficiency, higher peak performance and lower energy cost.

Meta-review: This paper proposes to improve the action space exploration of RL by perturbing the network's activations, modeled as a multivariate Gaussian distribution with a full covariance matrix. It showed that this more structural exploration can lead to better learning in simulated locomotion tasks. All the reviewers agree that the paper is clearly written, and the method is simple yet effective. The main concerns raised in the reviews are lack of comparison with other exploration methods, need more RL baselines, and limited improvement in some of the evaluations. The authors' rebuttal has sufficiently addressed these concerns. All the reviewers unanimously voted for acceptance after the rebuttal and discussion. Please incorporate the improvement in the rebuttal and additional promised results into the final version of the paper.
Decision: Accept (poster)