Title: Latent exploration for Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

efficient exploration

curse of dimensionality

motor control

musculoskeletal control
Abstract: In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and off-policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18\% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60\%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks. The code is available at: https://github.com/amathislab/lattice.

Latent Exploration for Reinforcement Learning

Abstract

In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and offpolicy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks. The code is available at: https://github.com/amathislab/lattice.

1 Introduction

Effectively exploring the environment while learning the policy is key to the success of Reinforcement Learning (RL) algorithms. Typically, exploration is attained by using a non-deterministic policy to collect experience, so that the random component of the action selection process allows the agent to visit states that would have not been reached with a deterministic policy. In on-policy algorithms, such as A3C [1] and PPO [2], the policy network parametrizes a probability distribution, usually an independent multivariate Gaussian, from which actions are sampled. In off-policy algorithms, such as DQN [3], DDPG [4] and SAC [5], the policy used to collect experience can be different from the one learned to maximize the cumulative reward, which leaves more freedom in the selection of an exploration strategy. The standard approach in off-policy RL consists in perturbing the policy by introducing randomness (e.g., ϵ -greedy exploration in environments with a discrete action space and Gaussian noise in environments with a continuous action space). In the past years, many notable

Alessandro Marin Vargas

EPFL

Alexander Mathis

EPFL

successes of RL are based on independent noise [1, 2, 4, 6, 7]. However, previous work has challenged the inefficiency of unstructured exploration by focusing on introducing temporal correlations [8, 4, 911]. Indeed, two opposite perturbations of subsequent actions might cancel out the deviation from the maximum probability trajectory defined by the policy, effectively hiding potentially better actions if followed by a more coherent policy. Here, we argue that not only correlation in time [4, 9-11, 8], but also correlation across actuators, can improve noise-driven exploration.

In motor control, precise coordination between actuators is crucial for the execution of complex behaviors [12-15]. Perturbing each actuator independently can disrupt such coordination, limiting the probability of discovering improvements to the current policy. In particular, musculoskeletal systems [16-19] typically feature a larger number of actuators (muscles) than degrees of freedom (joints). Here, we demonstrate that in such systems the exploration achieved by uncorrelated actuator noise is suboptimal. To tackle this problem, we introduce an exploration strategy, named LATent TIme-Correlated Exploration (Lattice). Lattice takes advantage of the synergies between actuators learned by the policy network to perturb the different action components in a structured way. This is achieved by applying independent noise to the latent state of the policy network (Fig. 1 A). With extensive experiments, we show that Lattice can replace standard unstructured exploration [2, 5] and time-only-correlated exploration (gSDE) [8] in off-policy (SAC) and on-policy (PPO) RL algorithms, and improve performance in complex motor control tasks. Importantly, we demonstrate that Lattice-SAC is competitive in standard benchmarks for continuous control, such as the locomotion environments of PyBullet [20]. In the Humanoid locomotion task, Lattice improves training efficiency, final performance and energy consumption. In the complex object manipulation tasks of the MyoSuite library [18], involving the control of a realistic musculoskeletal model of the human hand, Lattice-PPO achieves higher reward, while also finding more energy-efficient policies than standard PPO. To the best of our knowledge, our work is the first to showcase the potential of modeling the action distribution of a policy network with a multivariate Gaussian distribution with full covariance matrix induced by the policy network weights, rather than a diagonal covariance matrix.

A

B

Action

Actionnoise

Latent noise a =W(c +)

Actionnoise

a=Wx+∈

e1

15

e1

dev.

***

0.030

e2

10

e2

4

std.

0.025

e3

5

e3

2

ise

0.020

f

0

nois

0

angle

0.015

-5

0.010

-10

f3

Joint

0.005

e1e2 e3 f1 f2 f3

e1e2 e3 f1 f2 f3

p=3.2·10-17

2 Related work

Exploration based on the learned policy. In on-policy RL, the target policy must be stochastic, so that the experience collected includes enough variability for the agent to discover policy improvements. Algorithms such as A3C [1], PPO [2], and TRPO [21] parametrize each action component with an independent Gaussian distribution (continuous actions) or a categorical distribution (discrete actions). In off-policy RL, most exploration strategies are based on perturbations of the policy. For example, in discrete action spaces an exploratory policy can be derived from the target policy by adding ϵ -greedy exploration, or more sophisticated perturbations such as the upper confidence bound [22, 23], Thompson sampling [24], Boltzmann exploration [25] and Information-Directed Sampling (IDS) [26]. In continuous action space, the perturbation of the actions is provided by independent random noise, with (e.g., SAC [5]) or without (e.g., DDPG [4]) learned noise parameters. Unstructured exploration has several drawbacks [9], which have been tackled by ensuring a temporally coherent noise through colored noise [11], an auto-regressive process [27] or a Markov chain [28]. Another strategy to induce time correlation consists in keeping the perturbation parameters constant for multiple steps. This concept has been applied to perturbations of the network parameters [29] and to state-dependent action perturbations [9, 10, 8, 30]. While these exploration strategies provide smoother noise in time, Lattice aims to pair time-correlated state-dependent exploration with correlation in the action space.

Other exploration strategies. In on-policy RL, count-based exploration [32], curiosity-driven exploration [33] and random network distillation [34] are techniques to encourage visiting rare states. In off-policy RL, bootstrapped DQN [35] employs multiple Q-networks to guide exploration and improve performance. Additionally, unsupervised methods like Diversity Is All You Need [36] can achieve more diverse exploration by maximizing mutual information between a goal-conditioned policy and a set of skill labels. Starting to explore only after returning to a recently visited state allows to solve complex sequential-decision problem [37]. Lattice does not aim to replace curriculum learning [38-40, 15] or exploratory policies, but proposes exploring implicitly via the policy network in a monolithic way and can be combined with other approaches.

Exploration in over-actuated system. Overactuated systems, such as musculoskeletal systems, pose a challenge for exploration. Recent NeurIPS challenges have addressed the problem of learning control policies for locomotion and dexterous hand movements with musculoskeletal models [41, 14, 15]. The winners in these challenges have typically utilized complex training curricula, such as static-to-dynamic stabilization in the Baoding task, where a hand needs to accurately rotate two balls [15], or feature engineering and expert demonstration for locomotion [41]. Exploration can also be encouraged by learning a muscle-coordination network [42] or the state-dependent joint torque limits and energy function [43]. Similarly, Schumacher et al. proposed a self-organizing controller that outperforms previous approaches by inducing state-space covering exploration [19]. Integrating exploratory policies into a RL framework requires ad-hoc solutions, whereas Lattice can be trained end-to-end with both on-policy and off-policy algorithms.

3 Motivating example: The case of a flexor-extensor, single joint arm

To motivate Lattice, consider a simple flexor-extensor system, such as the elbow joint in the human upper arm (Fig. 2A). The activation a f ∈ [0 , 1] of the flexor muscle (biceps) produces a negative angular acceleration at the elbow, while the activation a e ∈ [0 , 1] of the extensor muscle (triceps) produces a positive angular acceleration. Using a first order approximation of the system dynamics, we can write the equation of the angular position of the elbow as ¨ θ = α ( a e -a r ) , where θ is the angle between the upper arm and the lower arm and α [rad s -2 ] is a parameter of the system which converts the muscle activations into angular acceleration. Given this system, we consider a simple control problem, where an agent needs to reach a target angular position of the elbow by activating the flexor and/or the extensor muscle. At every step, the agent observes the difference ∆ θ = θ 0 -θ t between the target angle θ 0 and the angle θ t at time t . For small values of ∆ θ , we assume that the control policy learned by the agent can be approximated with the linear functions a e = 0 . 5 + ∆ θ and a f = 0 . 5 -∆ θ . We now want to compare the effects on the angular acceleration produced by noise applied to the actions ( a e and a f ) or to the latent state ( ∆ θ ) for various cases:

Case 1: action space noise. If we inject independent noise ϵ f and ϵ e following a normal distribution N (0 , σ 2 ) to the muscle activations, we have a f ∼ N ( ⟨ a f ⟩ , σ 2 ) and a e ∼ N ( ⟨ a e ⟩ , σ 2 ) . It can be shown that ¨ θ ∼ N ( α ( ⟨ a e ⟩ - ⟨ a f ⟩ ) , 2 α 2 σ 2 ) (Appendix A.1).

Case 2: latent space noise. If we instead inject Gaussian noise to the latent state ∆ θ , while the marginal distribution of a f and a e is the same as with action space noise, the fact that the two distributions are correlated has an effect on the distribution of ¨ θ . It can be shown that ¨ θ ∼ N ( α ( ⟨ a e ⟩ - ⟨ a r ⟩ ) , 4 α 2 σ 2 ) (Appendix A.1).

This toy example illustrates how perturbing the latent state, instead of the actions, leads to higher variance in the behavior space. In fact, when randomly perturbing two opposing muscles, half of the time the perturbations will be in opposite directions, with a reduced impact on the observed kinematics.

To corroborate this finding about the advantage of latent versus action perturbations, we considered a policy network trained to control a realistic musculoskeletal arm model, implemented in MyoSuite [18], featuring three flexor muscles and three extensor muscles. A neural network trained with PPO and Lattice exploration can learn to accurately reach a target angle in every episode, and while doing so learns to alternately activate the extensor or the flexor muscle group (Fig. 2 B, left). The training procedure is detailed in Appendix A.1. We collected a dataset of 100 episodes where the latent state of the policy network was perturbed with Gaussian noise. While applying independent random noise to the muscle activations produces a diagonal covariance matrix, perturbing the latent state of the network leads to a full covariance matrix (Fig. 2 B, right). Thus, as hypothesized, the last layer of the policy network learned to distinguish between agonist and antagonist muscles, namely, the covariance shows positive correlation among muscles of the same group. We then tested whether sampling action noise with this covariance matrix leads to higher variance in the task space (joint angles) than noise sampled independently for each muscle. We collected 100 episodes where, at each step, we computed an action with action space perturbation and one action with latent state perturbation. For consistency of evaluation, the perturbation in the action space was sampled with the same variance magnitude induced by the latent perturbation. We executed each pair of actions in two cloned environments and compared the variance of the joint angles. The latent state perturbation introduces higher variance in the joint angles (Fig. 2 C), thus driving more diverse kinematics and wider exploration. Overall, we conclude that latent exploration might be advantageous when learning to control over-actuated systems, and thus developed and empirically tested Lattice on different benchmarks.

4 Methods

4.1 LATent TIme-Correlated Exploration (Lattice)

In continuous control, the policy update of on-policy algorithms (e.g., REINFORCE [44], A3C [1], TRPO [21], PPO [2]) and of some off-policy algorithms (e.g., SAC [5]) requires the computation of log π ( a t | s t ) , which is the logarithm of the probability of choosing a certain action a t given the current state s t , according to the current policy π . In such cases, the policy gradient needs to be

Algorithm 1 Standard (e.g., PPO, SAC)

Require:

Algorithm 2 Lattice

Require:

backpropagated through the density function of the action distribution. This is typically accomplished via the reparametrization trick , in which the policy network outputs the parameters of a differentiable probability density function, whose analytical expression is known, so that gradients can flow through the probability estimation. In the standard case, when we apply independent Gaussian noise to the action components, the action probability distribution can be parametrized as a multivariate Gaussian with diagonal covariance matrix (Algorithm 1). We propose that the weights of the policy network can be naturally used to parameterize the amount of exploration noise (Algorithm 2).

While a generic perturbation of the policy network's activations would lead to an action distribution with an unknown probability density function, in Lattice we limit the latent perturbation to the last layer's latent state, which is linearly transformed into the action (Fig. 1 A). Consider the output of the last layer of the policy network x ∈ R N x and the matrix W ∈ R N a × N x , mapping the embedding state to an action according to a = W x . We have indicated the size of the latent space with N x and the size of the action space with N a . If we assume a perturbation of the latent state x with independent Gaussian noise ϵ = N ( 0 , Σ x ) , where Σ x is a positive-definite diagonal matrix, then the perturbed latent state ˜ x is distributed as ˜ x ∼ N ( x , Σ x ) . Thus, the action distribution a ∼ N ( W x , W Σ x W ⊤ +Σ a ) , where with Σ a we have indicated the independent component of the action noise, can be derived as a linear transformation of a multivariate Gaussian distribution (details in Appendix A.2). This formula provides an analytical expression for log π ( a t | s t ) , as a function of the policy network weights and the covariance matrix of the latent state:

The probability of an action depends on the network weights both through its mean value W x and through the perturbation variance matrices Σ x and Σ a . Depending on the RL algorithm, it might be convenient not to propagate the gradients of the policy network through the variance component of the loss. In this way, the expected action trains the policy network, while the loss due to the standard deviation can regulate its magnitude by updating the parameters Σ x and Σ a .

SAC

Lattice-SACT=4

gSDE-SAC T=4

SAC-pink

Lattice-SACT=8

gSDE-SACT=8

Lattice-SAC T=1

Lattice-SAC episode

mean

Ant

Hopper

Walker

Halfcheetah

Humanoid

3K

3K

4K

reward

2K

2K

2K

2K

Episode

2K

1K

1K

0

1K

0

0

0

E

0

5M

10M0

5M

10M

15M0

5M

10M

15M

0

5M

0

10M

20M

Number of steps

4.2 Lattice generalizes time-correlated noise

Lattice can be thought of as an extension of Generalized State-Dependent Exploration (gSDE) [8], as it inherits all its properties, while extending it with the possibility of perturbing the latent state of the policy, besides the actions (Fig. 1 B). In Lattice, the agent learns two sets of parameters, S x and S a , representing the standard deviation of N x × ( N x + N a ) Gaussian distributions ( N x and N a being the size of the latent space and of the action space, respectively). These parameters are used to sample, every T steps, two noise matrices P x and P a (Fig. 1 C). These matrices are used to determine the latent noise ϵ x = αP x x and the action noise ϵ a = P a x . The parameter α ∈ { 0 , 1 } can be used to turn on or off the correlated action noise. For α = 0 , Lattice is equivalent to gSDE. The value of T regulates the amount of time-correlation of the noise. If T is large, P x and P a are sampled infrequently, causing the noise applied to the latent state and to the action to be strongly time-correlated (Algorithm 2).

In short, the action distribution depends on the latent state x , the last linear layer parameters W , the Lattice noise parameters S x and the gSDE noise parameters S a in the following way:

4.3 Implementation details of Lattice

Code to study the implementation details is available at https://github.com/amathislab/lattice.

Noise magnitude. The magnitude of the variance of individual noise components can either be learned or kept fixed, and can or can not be made dependent on the current state. We extend the implementation of gSDE [8, 45], which makes the noise magnitude a learnable parameter independent of the policy network.

Controlling the variance. We introduce two clipping parameters to limit the minimum and the maximum value of the variance of each latent noise component, to avoid excessive perturbations and convergence to a deterministic policy. This clipping does not modify the analytical expression of the action distribution, because it does not affect the sampled values, but rather the variance of the distribution.

Layer-dependent noise rescaling. As the noise vectors are given by ϵ x = αP x x and ϵ a = P a x , assuming the components of x have similar magnitude, the noise scales with the size of the latent state. To remove this effect, we apply a correction to the learned matrices log ˜ S x and log ˜ S a before sampling P x and P a : log S x = log ˜ S x -0 . 5 log( N x ) and log S a = log ˜ S a -0 . 5 log( N x ) . It can be proven that this correction removes the dependence of the noise from the size of the latent state (Appendix A.3), and thus from the network architecture.

PPO

gSDE-PPO T=4

Lattice-PPOT=1

Lattice-PPOT=4

Elbowpose

Finger pose

Hand pose

Fingerreach

1.01

1.0

0.8

0.8

0.6

0.3

0.5

0.5

0.4

0.2

Ifraction

0.2

0.2

0.2

0.1

0.0

0.0

0.0

0.0

0

200K

400K

600K0

2M

4M

6M

0

5M10M15M20M

0

5M

10M

Solved

Hand reach

Reorient

Baoding

Pen

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0

5M

10M

0

5M

10M15M20M0

5M

10M15M20M

0

5M

10M15M20M

Number of steps

Preventing singularity of the covariance matrix. The covariance matrix of the action distribution must be positive definite. It has the expression W Σ x W ⊤ +Σ a (Eq. 2), with Σ x = α 2 Diag ( S 2 x x 2 ) and Σ a = Diag ( S 2 a x 2 ) , where the square operations are to be intended element-wise. By construction, the covariance matrix is positive-semidefinite. However, it could be singular, e.g., when x is the null vector (Appendix A.4 for further details). To prevent the singularity of the covariance matrix, we add a small positive regularization value to its diagonal terms.

5 Experiments

We benchmarked Lattice on standard locomotion tasks [47, 6, 16, 48-50] in PyBullet [20], as well as musculoskeletal control tasks of MyoSuite [18] built in MuJoCo [31]. Both libraries include continuous control tasks of varying complexity. While in PyBullet the actuators apply a torque to each individual joint, in MyoSuite the agent controls its body through muscle activations. We give a complete description of each task in Appendix A.6. We implemented Lattice as an extension of gSDE in the RL library Stable Baselines 3 [45], which we used for all our experiments; the hyperparameters of the algorithms are detailed in Appendix A.7. All the results are averaged across 5 random seeds. The training was run on a GPU cluster, for a total of approximately 10,000 GPU-hours.

5.1 Pybullet locomotion environments

Lattice can be paired with off-policy RL algorithms, such as SAC [5]. We tested this combination in the locomotion environments of PyBullet, where gSDE-SAC achieves state-of-the-art performance [8, 46]. We also included Pink Noise exploration [11] as an additional baseline, which is the stateof-the-art colored noise process for exploration with off-policy RL algorithms. We used the same network architecture and hyperparameters for SAC specified in [46] for all the environments (see Appendix A.7). Preliminary experiments on the parameters of Lattice showed that environments with a smaller action space benefit from a higher initial standard deviation of the exploration matrix, so we set all the elements of log S x and log S a to 0 for Ant and Humanoid and to 1 for the other environments. We found that Lattice achieves similar performance to gSDE-SAC and SAC-pink for lower-dimensional morphologies (Ant, Hopper, Walker and Half Cheetah), while outperforming them substantially for the Humanoid (Fig. 3 and Appendix A.8). This suggests that Lattice can achieve

higher performance when controlling larger-dimensional morphologies. Furthermore, we later show that the policy learnt with Lattice-SAC for the Humanoid is more energy efficient than that learnt with SAC (Section 6).

5.2 Musculoskeletal control: MyoSuite environments

For musculoskeletal control, we tested several tasks from the MyoSuite [18]:

Three pose tasks (Elbow Pose, Finger Pose and Hand Pose). In each session, a target angular position is sampled independently for each joint. This means that the policy has to learn how to control each degree of freedom independently.

Two reach tasks (Finger Reach and Hand Reach). In each session, a target position for each finger tip (one for Finger Reach and five for Hand Reach) is sampled independently.

Three object manipulation tasks (Reorient, Pend and Baoding). In each session, the target is a fixed or moving target for an object (a die in Reorient, a pen in Pen and two Baoding balls in Baoding).

We tested Lattice with a LSTM network and PPO. This choice is motivated by (our) winning solution of the Boading ball task in the 2022 NeurIPS MyoChallenge [15], which used PPO together with a LSTM network [51]. For our experiments we use the same PPO hyperparameters and network architecture, which we keep identical across methods (see Appendix A.7).

Weset the initial value of all the elements of log S x and log S a to 0 in every task, except in Elbow pose and Hand pose, where we set it to 1.

In the reach and challenging object manipulation tasks (Reorient, Baoding and Pen), where the policy network has to control a complex hand model with 39 muscles and receives an observation with more than 100 dimensions, Lattice consistently outperforms the baselines (Fig. 4 and Appendix A.8). In contrast, in the pose tasks PPO performs comparably or better than Lattice-PPO (Fig. 4 and Appendix A.8). Those tasks define dense target states and, perhaps, exploration and coordination is less important (see below).

Interestingly, focusing action noise in the task-relevant space allows Lattice to avoid activating muscles unnecessarily, leading to conspicuous energy saving in the manipulation and reaching tasks (Fig. 5). In reach and object manipulation , Lattice-PPO achieves better reward at lower energy cost. We speculate that injecting correlated noise across muscle activations improves exploration in the

Finger pose Fingerreach Hand pose Hand reach Elbow pose

Pen

Baoding

Reorient

30%

I gain (relative)

20%

Finger reach.

10%

Lattice-PPO better

Pen

Reorient.

Baoding

Hand reach

%0

Elbowpose

Finger pose.

Reward

-10%

PPO better

-20%

Hand pose

-30%

-60%

-40%

-20%

0%

20%

40%

60%

Energy saving (relative)

space of task-relevant body poses and facilitates the discovery of efficient, coordinated movements. We tested this hypothesis next.

6 How does Lattice explore?

In most of the considered muskuloskeletal control environments, Lattice finds a more energy efficient control strategy, without compromising the performance (Fig. 5 and Appendix A.8). Especially in Reorient, where Lattice outperforms standard exploration in cumulative reward, it does so at a

A

Humanoid

C

Reorient

No of episodes

Explained variance

1.00

N° of episodes

0.75

.

0.50

20

0.25

SAC

5

PPO

Hip

RL

LLRALA

0.15

0.20

0.00

0

Lattice-SAC

10

12

W

0.025

0.050

0.00

0

Lattice-PPO

20

40

Action components

Energy

N°of components

Actioncomponents

Energy

N°of components

B

Action

Noise

D

Action

Noise

1.00

1.00

0.6

H

0.75

H

0.4

W

0.75

W

0.4

SAC

LL

RA

RL

-0.00

0.25

0.50

0.50

0.25

RA

RL

LL

口

口

0.2

0.0

0.2

PPO

M

T

R

一

0.25

0.00

0.50

0.25

0.50

M

R

T

0.2

0.0

0.2

LA

0.75

LA

0.4

L

0.75

L

0.4

H

RL

LL RA LA

1.00

H

RL

LL RA LA

W

M

R

1.00

W

T

M

R

L

0.6

1.00

1.00

0.6

Lattice-SAC

0.75

0.50

0.00

0.25

RL

LL

H

口

0.0

0.4

0.2

Lattice-PPO

T

一

W

0.50

0.25

0.00

0.75

W

0.2

0.0

0.4

0.50

0.25

RA

口

0.2

0.25

-0.50 R

M

0.2

LA

H

RL

LL RA LA

1.00

0.75

LA

HP

RL

LL RA LA

0.4

L

WTIMR

L

0.75

1.00

WTIMR

L

0.4

-0.6

H = Hip RL = Right Leg LL = Left Leg RA = Right arm LA = Left Arm

W=Wrist T=Thumb|=Index M=Middle R=Ring L=Little

fraction of the energy cost. We hypothesize that this is a direct consequence of biasing the noise with the same correlation as the actuators, preventing energy-consuming co-activations, which the agent needs to compensate for, despite the reward not being affected by them.

We assessed whether coordination emerged by analyzing policies for the Humanoid locomotion task and the Reorient task. By looking at the relative allocation of action noise across actuators (Fig. 6 A, C), we can see that in the Humanoid, compared to SAC, Lattice-SAC re-directs more exploration noise towards the task-relevant leg motors (SAC: 40% Legs , 45% Arms and 15% Body; Lattice-SAC: 50% Legs , 32% Arms, 18% Body). In Reorient, where all the muscles contribute to controlling the hand, the difference is less evident. Furthermore, both in the Humanoid and Reorient task, the intrinsic dimensionality of the actions output by a policy trained with Lattice is lower. Indeed, consistently across random seeds, fewer principal components explain a higher fraction of the action variance in Lattice than in standard exploration (Fig. 6 A, C). Lattice consistently promotes policies with actions lying on a lower-dimensional manifold, and this effect extends beyond Humanoid and Reorient, to almost all the considered tasks (Appendix A.9). We speculate that this property of Lattice determines when it is to be preferred to uncorrelated noise, i.e., when there exists a low-dimensional solution to the task which allows the agent to take advantage of motor synergies. In environments where the dimensionality of the task is intrinsically low ( reach and object manipulation ), Lattice achieves strong performance. Further experiments in the pose and in the object manipulation tasks, performed with Lattice-SAC, confirm the result that Lattice-SAC is better than SAC for learning to manipulate objects (Appendix A.8).

The intrinsic dimensionality of the actions decreases as an effect of training (Appendix A.9). While training tends to reduce the dimensionality of the policy regardless of the action noise, the decrease is more marked with Lattice. The low dimensionality of the actions is explained by the action correlation matrices, which show increased cross-actuator coordination with Lattice (Fig. 6 B, D). We speculate that this is driven by the correlation structure of the noise. While the off-diagonal elements are close to 0 with uncorrelated noise, those of Lattice present a structure that resembles that of the action correlation matrix (Fig. 6 B, D), also consistent with the motivating example (Section 3). Indeed, if we consider the off-diagonal elements at position ( i, j ) , we have that Cov ( a i , a j ) = W Cov ( x ) W ⊤ , while V [ π ( a | s )] i,j = α 2 ( W Diag ( S 2 x x 2 ) W ⊤ ) i,j (further details in Appendix A.5). In the case where α = 1 , S x is the identity and the components of x are uncorrelated, then the two matrices are identical. While throughout the training the parameters of S x can adapt, we can empirically see that the covariance matrix retains elements of its original structure.

7 Discussion and Limitations

We proposed Lattice, a method that leverages the synergies between action components encoded in the weights of the policy network to drive exploration based on the latent state. Latent exploration has proven effective in discovering high-reward and energy efficient policies in complex locomotion and muscle control tasks. Remarkably, Lattice outperforms random exploration in the reach and object manipulation tasks, with a large reduction in energy consumption. We showed that perturbing the last layer of the policy network introduces enough bias into the action noise. Importantly, this specific perturbation allowed us to find an analytical expression for the action distribution, which makes Lattice a straightforward enhancement for any on-policy or off-policy continuous control algorithm.

This sets Lattice apart from other forms of exploration using an auxiliary policy to collect experience, as they see their application limited to off-policy RL. Furthermore, knowing the action distribution is fundamental for those off-policy algorithms learning a stochastic policy, such as SAC. On the other hand, modeling the action distribution with a multivariate Gaussian with full covariance matrix comes at additional computational cost. In fact, Lattice introduces a training overhead over gSDE and PPO/SAC (approx. 20%-30%, depending on the hardware). This is due to the additional matrix multiplications required to estimate the action probabilities (Eq. 1). This limitation opens an interesting research direction, e.g., introducing sparsity constraints in the distribution matrices. For example, S a and S x could be forced to be diagonal or low-rank, without preventing motor synergies to be included into the noise distribution. We leave this investigation to future work.

Biological motor learning finds efficient and robust solutions [12, 13, 52-55]. Lattice also discovers low-energy solutions and it will be an interesting future question, if the brain is also performing some form of latent-driven exploration. Beyond this specific question, we are enthusiastic that advances in musculoskeletal simulators [18] as well as reinforcement learning are opening up exciting avenues for Neuroscience [56].

Our research aims to facilitate the training of motor control policies for complex tasks. Lattice might be employed as a tool in Artificial Intelligence, Robotics and Neuroscience, contributing to the creation of powerful autonomous agents. The energy efficiency of the learned policies might be of great interest for energy-sensitive robotics applications for reducing the carbon footprint of humans. While the advancement of the research in this field is a great opportunity, the related concerns should be carefully addressed [57].

Acknowledgments

We are grateful to Adriana Rotondo, Stéphane D'Ascoli and other members of the Mathis Group for comments on earlier versions of this manuscript. Our work was funded by Swiss SNF grant (310030_212516) and EPFL. A.M.V.: Swiss Government Excellence Scholarship. A.Z.H.: EPFL School of Life Sciences Summer Research Program. We thank the Neuro-X institute at EPFL for supporting the travel expenses (A.S.C. and A.M.V).

References

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning , pages 1861-1870, 2018.

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning , pages 1329-1338. PMLR, 2016.

Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 , 2017.

Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning , pages 1634-1644. PMLR, 2022.

Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient methods. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2008, Antwerp, Belgium, September 15-19, 2008, Proceedings, Part II 19 , pages 234-249. Springer, 2008.

Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn , 1:14-24, 2010.

Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Stephen H Scott. Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience , 5(7):532-545, 2004.

Emanuel Todorov. Optimality principles in sensorimotor control. Nature neuroscience , 7(9): 907-915, 2004.

Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Seungmoon Song, Yuval Tassa, Massimo Sartori, and Vikash Kumar. Myochallenge: Learning contact-rich manipulation using a musculoskeletal hand. https://sites.google.com/view/myochallenge , 2022.

Vittorio Caggiano, Guillaume Durandau, Huwawei Wang, Alberto Chiappa, Alexander Mathis, Pablo Tano, Nisheet Patel, Alexandre Pouget, Pierre Schumacher, Georg Martius, et al. Myochallenge 2022: Learning contact-rich manipulation using a musculoskeletal hand. In NeurIPS 2022 Competition Track , pages 233-250. PMLR, 2022.

Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4906-4913. IEEE, 2012.

Thomas K Uchida and Scott L Delp. Biomechanics of movement: the science of sports, robotics, and rehabilitation . Mit Press, 2021.

Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600 , 2022.

Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems. arXiv preprint arXiv:2206.00484 , 2022.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011.

Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in neural information processing systems , 30, 2017.

Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Informationdirected exploration for deep reinforcement learning. arXiv preprint arXiv:1812.07544 , 2018.

Dmytro Korenkevych, A Rupam Mahmood, Gautham Vasan, and James Bergstra. Autoregressive policies for continuous control deep reinforcement learning. arXiv preprint arXiv:1903.11524 , 2019.

Herke van Hoof, Daniel Tanneberg, and Jan Peters. Generalized exploration in policy search. Machine Learning , 106(9-10):1705-1724, 2017.

Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905 , 2017.

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems , 30, 2017.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature , 590(7847):580-586, 2021.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv:2003.04664 [cs, stat] , May 2020.

Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 , 2019.

Łukasz Kidzi´ nski, Carmichael Ong, Sharada Prasanna Mohanty, Jennifer Hicks, Sean Carroll, Bo Zhou, Hongsheng Zeng, Fan Wang, Rongzhong Lian, Hao Tian, et al. Artificial intelligence for prosthetics: Challenge solutions. In The NeurIPS'18 Competition: From Machine Learning to Intelligent Conversations , pages 69-128. Springer, 2020.

Shuzhen Luo, Ghaith Androwis, Sergei Adamovich, Erick Nunez, Hao Su, and Xianlian Zhou. Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. Journal of NeuroEngineering and Rehabilitation , 20(1):1-19, 2023.

Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C Karen Liu. Synthesis of biologically realistic human motion using joint torque actuation. ACM Transactions On Graphics (TOG) , 38 (4):1-12, 2019.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/ 20-1364.html .

Antonin Raffin. Rl baselines3 zoo, v1.8.3. https://github.com/DLR-RM/ rl-baselines3-zoo , 2020.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Seshashayee S Murthy and Marc H Raibert. 3d balance in legged locomotion: modeling and simulation for the one-legged case. ACM SIGGRAPH Computer Graphics , 18(1):27-27, 1984.

Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning , pages 1-9. PMLR, 2013.

Alberto Silvio Chiappa, Alessandro Marin Vargas, and Alexander Mathis. Dmap: a distributed morphological attention policy for learning to locomote with a changing body. Advances in Neural Information Processing Systems , 35:37214-37227, 2022.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology , 53(3): 139-154, 2009.

Michale S Fee and Jesse H Goldberg. A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience , 198:152-170, 2011.

Dagmar Sternad. It's not (only) the mean that matters: variability, noise and exploration in skill learning. Current opinion in behavioral sciences , 20:183-195, 2018.

AdamSLowet, Qiao Zheng, Sara Matias, Jan Drugowitsch, and Naoshige Uchida. Distributional reinforcement learning in the brain. Trends in neurosciences , 43(12):980-997, 2020.

Sébastien B Hausmann, Alessandro Marin Vargas, Alexander Mathis, and Mackenzie W Mathis. Measuring and modeling the motor system with machine learning. Current opinion in neurobiology , 70:11-23, 2021.

Maartje MA de Graaf. An ethical evaluation of human-robot relationships. International journal of social robotics , 8:589-598, 2016.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

A Appendix

Contents

A.1 Singe-joint arm: detailed calculations

Case 1: action space noise. By combining the equation of the angular acceleration ¨ θ = α ( a e -a f ) with the distributions of the muscle activations a f ∼ N ( ⟨ a f ⟩ , σ 2 ) and a e ∼ N ( ⟨ a e ⟩ , σ 2 ) , we can compute expected value and variance of the angular acceleration:

In the second derivation we have used the fact that a e and a r are independent random variables to compute their variances separately, and that V [ -a f ] = V [ a f ] .

Case 2: latent space noise. In this case we apply noise to the latent state, so that ∆ θ ∼ N ( ⟨ θ ⟩ , σ 2 ) . While it still holds that a f ∼ N ( ⟨ a f ⟩ , σ 2 ) and a e ∼ N ( ⟨ a e ⟩ , σ 2 ) , the two random variables are no longer independent. This fact does not change the computation of the expectation:

For the variance, we need to make the dependence on θ explicit, using the formulas a e = 0 . 5 + ∆ θ and a f = 0 . 5 -∆ θ :

We can observe that the variance is double in the case of latent space noise.

To test whether these results can be observed also in simulation, we considered the Elbow pose task of MyoSuite, where we trained a policy with PPO and Lattice exploration (with the hyperparameters specified in Table T4). The policy reaches a solved value of 0.95, meaning that the angle between the upper and the lower arm is in the immediate neighborhood of the target angle 95% of the simulation time. We assessed the effect of latent and action perturbations on the angular position by carrying out two simultaneous simulations of the environment per noise type. At every step, one simulation received a noisy action, while the other would receive a deterministic, noise-free action. We then registered the difference in elbow angle between the noisy and the deterministic simulation. To avoid having the results obfuscated by the cumulative perturbation over longer time periods, we synchronized the simulators after every step at the state achieved through the stochastic simulation.

To compute comparable results with latent and action noise, we had to make sure that the variance of each action component would be equivalent in both cases. We accomplished this by first running the simulation with latent noise, with the same variance for each latent state component. We then measured the variance that such perturbation would cause to each action component, and we used

this value to generate the action perturbation noise. This ensures that the difference in the variability of the elbow angle when applying latent or action noise is not due to a difference in scale, but only to the presence of off-diagonal elements in the noise covariance matrix.

A.2 Lattice's Action Distribution Parameterization

In Lattice, the action vector a is a linear transformation of the perturbed latent vector ˜ x = x + ϵ x , further perturbed with independent action noise ϵ a :

Furthermore, the latent noise ϵ x and the action noise ϵ a are defined as follows:

The elements of P x and P a are distributed as independent Gaussians:

where S ( x ) and S ( a ) are learnt parameter matrices whose elements represent the standard deviation of each element of the perturbation matrices. We can therefore compute the distribution of each element of the noise vectors. They are defined as:

meaning that they are the sum of independent Gaussian random variables. We can compute their mean and variance as follows:

̸

The covariance of noise elements at different indices is 0 . Indeed, for i = j :

where we used that different elements of P x and P a come from independent Gaussian distributions. Therefore, the joint distribution of the noise vectors ϵ x and ϵ a is Gaussian, with a diagonal covariance matrix:

where the squares are to be intended element-wise. The distribution of a can be directly computed from the formula of the linear transformations of multivariate Gaussian distributions from Eq. (7):

where we have used the independence of P a and P x to find the covariance matrix of ϵ a + W ϵ x .

A.3 Noise rescaling across networks of different size

Here, we will show that the variance of Lattice's generative noise model (Eq. (18)) depends the dimension of the latent state x . We thus, rescale S x and S a to be invariant to the nework size.

Derivation: We first observe that the standard deviation of each element of the noise vector ϵ x scales with the size of x . Indeed:

Therefore, if we consider the average value of ( S x ) 2 i,j and of x 2 j to be independent of the size of the latent state (at initialization time, it is the case, e.g., with the common Xavier initialization for the network parameters [58]), then we have that V [ ϵ i ] scales with N x . By applying the correction

we have that

In this way the initialization of log ˜ S x can be kept the same across networks with different latent state size. An identical argument is valid for S a , too.

A.4 Conditions on the covariance matrix of the action distribution

First we show that the covariance matrix of the action distribution, defined as W Σ x W ⊤ +Σ a , is positive semidefinite by construction. We start by observing that the matrix Σ x = α 2 Diag ( S 2 x x 2 ) and the matrix Σ a = Diag ( S 2 a x 2 ) are square diagonal matrices, whose elements are larger or equal than 0. We can therefore write Σ x = Σ 1 2 x Σ 1 2 x and Σ a = Σ 1 2 a Σ 1 2 a , where the elevation to 1 2 has to be applied to each element of the matrices. For any vector y ∈ R N x , we have that

The regularization term we apply in Lattice consists in a multiple of the identity matrix by the coefficient γ , so that the minimum eigenvalue of the covariance matrix can never be lower than γ itself.

Without a regularization term, the covariance matrix W Σ x W ⊤ + Σ a might have 0 eigenvalues. In particular, this is common when using an activation function which promotes sparse latent representations, such as ReLU. In the limit case where x is the null vector, the covariance matrix is the null matrix.

A.5 Empirical covariance of the action

The covariance of the action components of a deterministic policy is given by:

A.6 Parameters of the PyBullet and MyoSuite environments

We used all the default parameters for the PyBullet environments [20], including the default reward function and episode length. The MyoSuite environments, instead, come with a defined metric for the success (the solved value), while the reward components are adjustable [18]. We keep the same reward components across all the algorithms, with values which consent achieving good solved values. In Table T1 and T2 we detail the environment and reward parameters we choose for our trainings.

We often add a new reward component, called alive , which is equal to 1 when the episode is not finished. It can be used to promote policies that do not trigger an early termination, e.g., by dropping the object. To be noted that we reduced the range of possible target poses in Hand pose, because no algorithm could solve the environment with the full range (without a curriculum), making the environment unsuitable to test the difference between standard and latent exploration.

A.7 Hyperparameters of SAC, PPO, gSDE and Lattice

Our implementation was based on the library Stable Baselines 3 [45], and the code for Lattice will be shared in an open-source way.

Here, we summarize the parameters of SAC, gSDE-SAC and Lattice-SAC for the PyBullet locomotion tasks (Table T3) and the parameters of PPO, gSDE-PPO and Lattice-PPO in the MyoSuite muscle control tasks (Table T4 and T5). We also assessed the dependency of the final reward depending

on the initial value of log σ in for Hand Pose and Reorient. The default value of 0 seems to perform optimally (Fig. F1).

0.4

HandPose

0.2

Reorient

0.0

-3

-2

-1

0

1

2

Initial value of log(o)

A.8 Detailed reward and energy results of all experiments

We list the performance (energy and reward) of SAC, gSDE-SAC and Lattice-SAC in the PyBullet locomotion tasks (Table T6) and the performance of PPO, gSDE-PPO and Lattice-PPO in the MyoSuite tasks (Tables T7,T8, T9 and T10).

All results are averaged over 5 seeds and we report mean and standard error of mean.

A.9 Evolution of the dimensionality of the policy during training

In the main text we analyzed how Lattice explores the environment (Section 6). As discussed in the main text, here we present the results across learning.

We studied how the dimensionality of the policy changes during training when the policy is trained with Lattice, compared to standard SAC and PPO. While the number of relevant components is similar at the beginning of the training, the policies trained with Lattice converge to actions that lie

on a lower dimensional manifold (Fig. F2). We hypothesize that this is the main reason why Lattice leads to more energy-efficient policies. Indeed, the energy consumption is similar for both Lattice and SAC/PPO at initialization and diverges during training with Lattice achieving more energy-efficient policies; this result is also consistent across seeds(Fig. F3). Overall, the final policies trained with lattice have lower dimensionality, consistently across environments and random seeds (Fig. F4).

Training step

A

Explained variance

1.0

0.8

Lattice SAC

：

0.8M

1.2 M

0.4 M

Explained variance

1.0

0.8

B

N. of components

16

Lattice SAC

Seed 1

Seed2

0.6

3.2M

3.2M

5.2M

0.6

0

Seed 4

Seed3

Seed5

0.4

15M

0.4

0.2

0.2

八

8

0.0

0.0

0

5

10

15

0

5

10

15

0.40.81.223.25.215

N°ofcomponents

N°of components

Training step [M]

1.0

1.0

Explained variance

Training step

Explained variance

N. of components

30

Lattice PPO

Lattice PPO

0.8

Seed2

0.6-

.

5.2 M

0.4 M

2 M

0.6

20

Seed 3

Seed 4

Seed5

0.4

8.8 M

0.4

0.2

.

15.6 M

20 M

0.2

10

0.0

0.0

o

N°of components

20

40

0

N°of components

20

40

0

0.42 5.2 8.812.415.620

Training step [M]

HUMANOID

0.20

0.15

R

Energy

0.15

0.20

SAC

LATTICE

0.20

0.15

Energy

0.20

0.15

Energy

0.15

0.20

0.10

0.10

0.10

0.10

0.10

0.05

0.05

0.05

0.05

0.05

0.00

0

0.4

0.81.223.25.215

Training step [M]

0.00

0

0.4

0.81.223.25.215

Training step [M]

0.00

0

0.4

0.81.223.25.215

Training step [M]

00'0

0

0.4

0.81.223.25.215

Training step [M]

0.00

0

0.4

0.81.223.25.215

Training step [M]

0.5

PPO

0.5

0.6

REORIENT

0.4

0.6

LATTICE

0.4

0.5

0.6

Energy

0.4

0.3

0.2

0.1

0.2

0.2

0.1

0.2

0.1

0.2

0.0

0

0.4

Training step [M]

25.2 8.812.4 15.620

0.0

0

0.4

Training step [M]

2

5.28.812.420

0.0

0

0.4

25.2 8.8 12.4 15.620

Training step [M]

0.0

0

0.4

25.28.812.415.620

Training step [M]

0.0

0

0.4

25.2 8.812.4 15.620

Training step [M]

MyoSuite environments

PyBullet environments

[eff dim/DoF]

Ratio [eff dim/DoF]

0.8

Lattice-SAC

1.0

二

Lattice-PPO

SAC

PPO

0.6

0.8

0.4

0.6

Ratio

0.2

0.4

Ant

Pen

Walker

Reorient

Baoding

Hopper

FingerReach

ElbowPose

HandPose

HandReach

HalfCheetah

Humanoid

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the exploration problem in RL algorithms by considering the action-space correlation. The approach is motivated by the fact that over-actuated systems with high-dof action space require correlated control actions. Such action-space correlation is ignored by action-space random perturbation in a standard RL algorithm which results in inefficient exploration. This work proposes to add random perturbation in the latent space in the last layer of the policy network to impose the proper action structure in the perturbation distribution. The presented approach can be potentially integrated into most existing RL algorithms to enhance the exploration capability and is shown to perform great, especially in high-dim musculoskeletal environments.\n\nSTRENGTHS:\n1. The motivation of the work is clear and intuitive. The provided toy example is a very illustrative example to motivate the idea of using correlated action perturbation.\n\n2. The proposed method is concise and seems to be universal in that it can be easily integrated into most RL frameworks to improve exploration efficiency.\n\n3. The experimental results are convincing to show the effectiveness of the proposed Lattice method.\n\nWEAKNESSES:\n1. The selection of the initial logstd for the $S$ matrices seems to be not straightforward and might need extra experiments to determine.\n\n2. The presented Lattice method relies on a good learned policy to impose the action correlation structure, which might result in local-optimal policy (also see point 3 in the next section).\n\n3. The exposition of the paper can be further improved to be more clear and consistent.\n\n   a) The second motivating example (the one in simulation) is confusing. \n\n   b) In Line 148, $N_l$ is used for the latent space size, while in Figure 1, $N_x$ is used for latent space size.\n\n   c) Line 168: $\\epsilon_a = P_a a \\Rightarrow \\epsilon_a = P_a x$\n\n   d) It was never mentioned in the main paper about the separate action space perturbation until $\\Sigma_a$ appears in Line 152.\n\nQUESTIONS:\n1. The two images in Figure 2(B) is hard to parse. Are those images obtained from the action perturbation training or the latent perturbation approach?\n\n2. In the MyoSuite environments (Figure 4), I noticed Lattice-PPO (T=1) consistently works better than Lattice-PPO (T=4) except for the **Pen** task. Does it mean time-space correlation is not important in general? Do you have any idea on why T=1 works better than T=4 and what is special in **Pen** task that makes T=4 preferable?\n\n3. The proposed Lattice approach benefits the action structure learned by the policy and imposes this learned structure to the action perturbation. However, on the other side is this also a limitation of this approach? Will Lattice suppress the emergence of novel control modes and result in a local-optimal policy? If this is not a problem, which part of the algorithm is the key to addressing this issue?\n\n4. It would be valuable to also ablate the $P_ax$ term in the method. Is this the key to addressing the last point?\n\n5. gSDE (T=4) seems to work consistently worse than other approaches (even worse than vanilla PPO) in all MyoSuite environments. It would be interesting to see the performance of gSDE-PPO (T=1) which differs from PPO on whether having state-dependent perturbation. If gSDE-PPO (T=1) does not work well either, then it might not be reasonable to have $\\alpha$ to toggle between gSDE and Lattice. Instead, using $\\alpha$ to toggle between regular RL perturbation and latent space perturbation might work better?\n\n6. Currently SAC and its variants are used for the Pybullet environments while PPO and its variants are used for MyoSuite environments. It would be also great to provide all of those algorithms on all benchmark problems (maybe in Appendix).\n\nLIMITATIONS:\nAs mentioned in point 3 in the **Questions** section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an exploration method that adds noise to the last linear layer (latent) of the policy network to incorporate time-dependent noise. The method is tested on locomotion tasks and evaluated on the metrics of performance and energy consumption. Evaluation is done on PPO and SAC base algorithm and an existing time-correlated baseline, gSDE.\n\nSTRENGTHS:\n- Tackles an important problem of exploration in tasks with continuous action.\n- The proposed method is rather simple and presented well.\n- Thorough experiments with necessary details and hyperparameters are presented.\n- Performance is evaluated on return and energy conservation which is important for practical implications.\n\nWEAKNESSES:\n- Unclear why the proposed method enabled energy-saving policy compared to another time-correlated baseline. \n- Energy comparison with gSDE is missing, which makes it difficult to evaluate how the noise injected in the latent layer is helping.\n- Few cases, the proposed method outperforms the existing method in return performance.\n\nQUESTIONS:\nIt is unclear why the proposed method enabled energy-saving policies compared to another time-correlated baseline. The comparison is conducted between PPO/SAC and not with gSDE (time-correlated). It is unclear how much the injected noise in the latent space contributes compared to the time-correlated component.\n\nIn Figure 4, the results are mixed. It is important to understand why time correlation sometimes worsens performance. It might be interesting to consider whether the proposed task requires independent or time-correlated noise.\n\nThe description of Figure 6 is currently difficult to understand. Simplifying the description would make it easier to comprehend the implications of the energy comparison.\n\nThe main contribution of the proposed method lies in the addition of noise to the latent space. Therefore, it is crucial to comprehend how this component is implemented. In terms of empirical return, the performance of the proposed method is mostly similar to the baseline. However, it is important to investigate how the introduction of noise in the latent space contributes to the observed energy consumption improvements. This information would provide a better understanding of the implications of the proposed method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method to conduct action exploration based on the latent space representation during the reinforcement learning process. The disturbance noise in this case is based on gSDE, and the further \"interval-hold\" mechanism makes the noise time-correlated. The authors conduct experiments on both classic pybullet benchmarks and musculoskeletal control environments. They show in most cases, the new exploration method can give faster reward learning compared to other baselines (SAC, gSDE-SAC, PPO, gSDE-PPO).\n\nSTRENGTHS:\n1. The paper is clearly written and easy to read.\n2. The idea of injecting time-correlated noise is simple, novel, yet effective.\n3. Extensive results comparison on a wide range of control tasks.\n\nWEAKNESSES:\n1. Lack of exploration algorithms comparison: the authors mention many exploring techniques [20-37] in the related work section, but have not compared to any of those, nor add on top of those techniques to boost the performance.\n2. Lack of RL algorithms implemented: I will be eager to see how the proposed latent exploration idea can work on other RL algorithms (TRPO, TD3, etc).\n3. Lack of ablation studies to provide rationale about: (1) why learn noise magnitude rather than a fixed noise magnitude, (2) the combination of the clipping parameters\n\nQUESTIONS:\n1. In Half cheetah experiment, the standard deviation for the reward curves is much smaller compared to the rest benchmarks. What could be the potential reason for it?\n2. It seems like the proposed algorithm tends to introduce more uncertainty in the reward learning (an observation from Fig. 3-4) in most cases. Does that mean it is hard to get a very stable training from the proposed latent exploration?\n3. A bit curious about the 10,000 hours of training. How many GPUs do you use? Do you think there is a way to reduce the workload to make it more friendly for normal AI labs in universities to conduct this kind of research?\n\nLIMITATIONS:\nThe authors have well discussed the limitations in section 7. For some of the limitations (training overhead), the authors put some thought and suggest new promising work directions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

comments: SUMMARY:
The paper proposes adding time- and actuator-correlated noise instead of simple gaussian noise to improve exploration in RL.

STRENGTHS:
1. The paper is well written and easy to read.
2. The approach can be easily applied to any RL algorithm. And can be thought of as a generalization of earlier work in noise for exploration (gSDE).
3. The improvements on 4 out of 8 tasks in MyoSuite is encouraging.

WEAKNESSES:
1. Is there any intuition as to which tasks are suitable for your method? It appears completely random that PPO outperforms everyone on handpose which is (I expect), an implicit subgoal in tasks like Baoding and Pen. Hence it is unclear why lattice underperforms on hand pose, elbow pose.
2. A nitpick: “ought” is an ambiguous word to use in the abstract and should be replaced.
3. Another nitpick: the arrows in the energy saving vs reward gain plot is not very clear. What do they indicate?

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-28

comments: SUMMARY:
The paper presents an extension to the previously introduced RL exploration method gSDE called Lattice. The main idea is to add noise to the full covariance matrix (instead of only onto the diagonal) of the multivariate Gaussian used for exploration. In particular, the authors multiply state dependent noise with the weights of the last (linear) noise layer of the policy network (so that the resulting action probability distribution is known) and add this term to the weights and action dependent noise (gSDE). In this manner, the authors achieve correlation between action components. Both, the action and state dependent noise terms are sampled periodically, where the period determines the time-correlation in the noise terms.

STRENGTHS:
- thorough experimental validation on classical RL benchmarks and high dimensional muscle RL benchmarks (myosuite)
- detailed implementation details and discussion about how Lattice works
- paper is easy to read and understand

WEAKNESSES:
- no comparison to other exploration methods such as [11], [18] or [30]
- contribution seems rather empirical (no theoretical contribution found)
- Lattice results for classical RL benchmarks (ant, hopper, walker, half cheetah, humanoid) not very convincing even in comparison to previous method (gSDE).

QUESTIONS:
Please address the points in the weakness section.

LIMITATIONS:
The authors appropriately addressed the limitiations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-31