Contextual Subspace Approximation with Neural Householder Transforms
Authors: Kerrick Johnstonbaugh
Michael Przystupa
Jacob Keller
Martin Jagersand
Venue: Submitted to ICLR 2023 (2023)
Keywords: robotics
RL
representation learning

Abstract: Choosing an appropriate action representation is an integral part of solving robotic manipulation problems. Published approaches include latent action models which compress the control space into a low dimensional manifold. These involve training a conditional autoencoder, where the current observation and a low-dimensional action are passed through a neural network decoder to compute high dimensional actuation commands. Such models can have a large number of parameters, and can be difficult to interpret from a user perspective. In this work, we propose that similar performance gains in robotics tasks can be achieved by restructuring the neural network to map observations to a basis for a context-dependent linear actuation subspace. This results in an action interface wherein a user’s actions determine a linear combination of a state-conditioned actuation basis. We introduce the Neural Householder Transform (NHT) as a method for computing this basis. Our results show that reinforcement learning agents trained with NHT in kinematic manipulation and locomotion environments tend to be more robust to hyperparameter choice and achieve higher final success rates compared to agents trained with alternative action representations. NHT agents outperformed agents trained with joint velocity/torque actions, agents trained with an SVD actuation basis, and agents trained with a LASER action interface in the WAMWipe, WAMGrasp, and HalfCheetah environments.

Under review as a conference paper at ICLR 2023
CONTEXTUAL SUBSPACE APPROXIMATION WITH NEURAL HOUSEHOLDER TRANSFORMS
Anonymous authors Paper under double-blind review
ABSTRACT
Choosing appropriate action representation is an integral part of solving robotic manipulation problems, Published approaches include latent action models which compress the control space into low dimensional manifold. These involve training a conditional autoencoder, where the current observation and a low- dimensional action are passed through a neural network decoder to compute high dimensional actuation commands. Such models can have a large number of pa- rameters, and can be difficult to interpret from a user perspective. In this work, we propose that similar performance gains in robotics tasks can be achieved by restructuring the neural network to map observations to a basis for a context- dependent linear actuation subspace. This results in an action interface wherein a user's actions determine a linear combination of a state-condite conee-ooiiiioeed actuation ba- sis. We introduce the Neural Householder Transform (NHT) as method for com- puting this basis. Our results suggest that reinforcement learning agents trained with NHT in kinematic manipulation and locomotion environments are more ro- bust to hyperparameter choice and achieve higher final success rates compared to agents trained with alternative action representations. NHT agents outperformed agents trained with joint velocity/torque actions, agents trained with an SVD actu- ation basis, and agents trained with LASER action interface in the WAMWipe. WAMGrasp, and HalfCheetah environments.
INTRODUCTION
In real-world applications of reinforcement learning, its imperative to choose appropriate represen- tations when defining the Markov decision process. The consequences of poor design decisions can have adverse effects in domains like robotics, where safety (Tosatto et al., 202T) and sample efficiency (Li et al., 2021) are desirable properties. Typically these properties choice of action space. Choices of robot action types distinct from basic joint motor control, such as Cartesian control or impedance control, have been shown to influence the efficiency of robotic learning, depending on the task (Martín-Martin et al.) 2019).
be captured by
Researchers have typically focused learning action representations that can capture variety of robotic motions. This interest has led to developing several different action representation frame- works. One framework includes motor primitives which entire trajectories are encoded as the action PParaschos et al., (2013: Schaal 2006). Motor primitives have seen much success in robotics leading to impressive woal-world experimental results by constraining the action space (Tosatto al., 2021 Kober & Peters 2009).
Another framework is the latent actions framework, which actions-pers pernn------tm--tteee are compressed into a latent subspace. Typically these are conditional auto-encoders trained to predict the high- dimensional actions given the state and latent action. These methods have been used successfully in both learning systems (Allshire et al. 2021 Zhou et al. 2020 van der Pol et al. 2020 as well as human-in-the-loow -n-thh-loop settings (Losey et al. 202 2020 Karamcheti et al. 2021 Jun Jeon et al. 2020).
It remains unclear whether robotics tasks must have deep, complex action models. There is little work comparing latent action models across varying complexity tasks. For example, hand poses complex high dimensional action space can have up to 80% of configurations explained by two principal components (Santello et al.) 1998). This result has been exploited to develop low- dimensional linear control algorithms, but they assume all actions exist in global linear subspace

---

Under review as a conference paper at ICLR 2023
(Matrone et al., 2012 Odest & Jenkins 2007 2022).
Artemiadis & Kyriakopoulos: 2010
Liang et al.
In this work propose approach in which we neural network to produce state-dependent basis for a linear actuation subspace. We refer to this as contextual subspace appreximation. Actu- ation commands (e.g. joint velocities) are locally linear with respect low dimensional inputs, but globally non-linear as the actuation subspace changes as a function of context.
The motivation for contextual subspace approximation and the corresponding solutions be summarized as follows: 1) Contextual subspace approximation requires less data because a k- dimensional subspace is completely determined by just k linearly independent samples. 2) From the agent's ' perspective, action maps change the transition dynamics of the environment, and using simpler functions results in simpler dynamies. 3) Models for contextual subspace approximation can be notably smaller by doing away with the encoder from the latent actions framework.
The model proposed here uses Householder transformations obtain an orthonormal basis for the desired actuation subspace. Householder transformations are often used in QR factorization to ef- ficiently compute least square solutions to over-determined systems of linear equations. This prop- erty has been exploited in several settings to define learnable orthonormal matrices in applications of QR factorization for machine learning (Nirwan & Bertschinger [019 Dass Mahapatra [02T] van den Berg et al. 2018). Additional work has studied applications of Houscholder reflections that include normalizing flows (Tomzaak & Welling, 2016 Mathiasen et al. 2020), network activation functions (Singla al. (021), and decomposition of recurrent and dense layers in neural networks (Mhammedi et al. 201 Zhang et al., (2188) Obukhov et al.) (021). To the best of our knowledge, our work is the first study Householder matrices in the context of latent action models.
We identify our contributions as the following:
We propose contextual subspace approximation as a novel alternative to end-to-end non- linear latent action models for robotic control.
We prove that the Neural Houscholder Transform is smooth with respect changes in con- text, and can output bases for the optimal actuation subspace associated with each context.
Our experiments empirically suggest that in two simulated kinematic manipulation tasks and one locomotion task, reinforcement learning agents trained with Neural Householder Transforms learn more efficiently than agents trained to act in with 7dof, SVD, or LASER action interfaces.
2 BACKGROUND AND PRELIMINARIES
In this section, we formalize our framework for learning action representations. We outline relevant background knowledge to contextualize our work, including deep latent action models, and their combination with Markov decision processes. We compare linear, locally-linear, and nonlinear action mapping approaches by conducting experiments on reinforcement learning problems.
2.1 PROBLEM STATEMENT
We assume that the data we wish to model was observed in some context, and the resulting dataset is a collection of context-datapoint pairs (datapoints and context are both represented by vectors). We formulate the problem of contextual subspace approximation by supposing that, for every context c, there exists an associated subspace that best approximates the data observed in the neighborhood of c.
We use x = (c, u) to denote a tuple consisting of a datapoint u and the context c which was observed. For convenience, we define the following functions to extract the data and context from tuple x, respectively: C(x) = c; l(xx) = u. In addition, we denote the neighborhood point as (C()= (c' ||c c'Il 8) for some d E R.
context
Definition 2.1 (Optimal Contextual Subspace). We define (c),,, the optimal k-dimensional sub- space associated with context c, as the k-dimensional subspace that minimizes the expected projec- tion error of data observed in the neighborhood of c:
2

---

Under review as a conference paper at ICLR 2023
(c) arg min Bx (ecce))ee)))))) - projw (2(x)) the associated optimal contextual subspace.
(1)
W
where W k-dimensional linear subspace of IR", and projw (2((x)) is the orthogonal projection of the data u onto W.
Our goal is to approximate a function (c) that maps context vectors to an orthonormal basis for
Q' c Q col Q) (c)
(2)
where Q Faxk n l matrix of real numbers, and col( Q) is the column space of Q. We assume access to a dataset of datapoint-conies pairs.
2.2 MARKOV DECISION PROCESSES
Markov Decision Process (MDP) is defined by the tuple (S, A, T, p(p)) a, s')) where $ the state space and A is the action space. The transition probability operator T(s, a, a.)): : x A S [0, 1] denotes the probability of transitioning to state s' E when taking an action a E A from state S. p(s)) is the initial state distribution, and r(s. a, s') defines the reward function. In this framework, the optimal policy search problem involves finding some π (s) that maximizes the discounted return: x (s) arg max, Va (s) E a, s,)]
Often in real-world problems, reinforcement learning agents must approximate ='(s) as the state representation is intractably large. As we do not assume access to the underlying state s, we will deal with observations, which serve as our context c. We are interested in representing low-dimensional contextual subspaces that approximate the high-dimensional actuations u of robotic agents.
This paper studies learning action interfaces that map actions a E IR* to raw actuation commands, R" Throughout this work, the action space & will be Rt where k is smaller than the dimen- sionality of the raw actuation space (e.g. number of joints) of the robotic agent. In the MDP frame- work, interpret action interfaces being absorbed in the transition dynamics, T(s, a, s's).
2.3 LATENT POLICY FRAMEWORK
The latent actions framework assumes that the actuation commands produced by the optimal policy rt exist on some lower dimensional manifold. In latent action models, latent actions z E RR are mapped to this manifold. These models have typically been studied in settings where there exists dataset of transition tuples (c,, u, r')r). Here c' is the context observed after the agent performs actuation in context and r is the corresponding reward. We follow this paradigm of learning from offline demonstrations, and leave the study of learning latent action models online as future work, noting that some researchers have previously studied this setting (Alshhre et al. (021I.
Broadly, the class of models previously studied are conditional autoencoders. These models include neural encoder fo u) = which predicts the latent action. If the model is variational CAE, then fe(c, u) = (μ, o). and z is sampled from the Gaussian parameterized by μ, a using the reparameterization trick (Kingma & Welling 2014). These latent actions are then reconstructed with a decoder (z, c) u. where c is assumed to contextualize how the latent action z should map to the higher dimensional space. In some works, there is also latent transition model To(z, c) c'. which is trained to encourage the latent space to be predictive of transitions Allshire et al. 2021 van der Pol et al. 2020).
The most general loss function incorporating the above models is the following:
arg min Treau (c, u, go: fo) + (C, u, So) + oeTaggn (z, c', To).
The first term Lysen is responsible for enforcing that the reconstructed latent actions approximate the demonstration actuations. The second term, Log incorporates all the terms that enforce additional requirements of the latent space. The typical choice are compression terms that pack the latent codes into some desired distribution which can include the Kullback-Leebler divergence, maximum mean
3

---

Under review as a conference paper at ICLR 2023
discrepancy, simply the L2-norm of z. The third term Ldya is used encourage the latent actions to be predictive transitions. The LASER algorithm is representative example of this framework Allshire et al. 2021). LASER trains latent dynamics model in conjunction with variational autoencoder.
3 CONTEXTUAL SUBSPACE APPROXIMATION
In this section, we describe our proposed alternative to the conditional autoencoder paradigm of tent action models. The goal is to compute useful map from low-dimensional task relevant actions a E IR to high-dimensional actuation commands (e.g. motor torques a robotic manipulator) R", where k. Our approach centers optimizing parameterized approximation of Q" (see Equation 2
First, let consider using linear map from the latent space the actuation space. Instead of a linear function 98 a, c > that jointly maps context vectors and actions to the actuation space, we work with a non-linear function 2 с + Q that maps context vectors to a matrix. The matrix Q: + itself is linear map from now-dimensional actions high dimensional actuation commands.
Q serves a similar purpose to go (both map actions to actuation commands), we could con-
Neural Householder Transform
sider Q to be linear decoder in the latent ac- tion framework. Then optimization of the stan- dard reconstruction loss is formulated as fol- lows:
Qo(c)
0" arg min E= |u Qal?
(4)
Actuation Projection
QQ u ú
Where Q = 2x (c) is a function the context 0 represents the optimal parameter vector for Co.
Training Loss
The problem now becomes how to select the ac- tion a E R to use in this optimization. One ap- proach is to follow the conditional autoencoder paradigm and predict a with an encoder neural network. We opt instead compute the opti- mal action a", which define as the action that minimizes equation (4) for fixed u and Q when 0 is held constant:
Figure 1: neural network and Householder transformations to map context vector to an n x matrix Q with orthonormal columns. The data associated with contex is projected onto the column space of Q.
Training procedure for NHT. Qo uses
arg min ||u Qal
(5)
Finding a* is least squares problem, and the solution can be computed with the Moore-Penrose left pseudoinverse Q+ (Q Q) 'Q_. The solution to is given by a" - Now optimization problem becomes:
0r = arg min E - QQ+QQQutuul
(6)
Note that the matrix QQ+ + is orthogonal projector onto span Q). Therefore, when cal- culate QQuu. we are performing an orthogonal projection of onto span(Q). That is, = proj ②) (u).
Now is clear that the solution to Equation (6) is the best approximation attainable by Oo to the optimal actuation subspace W W*(c) (c) defined Equation
3.1 NEURAL HOUSEHOLDER TRANSFORM
It can be desirable for the matrix Q produced by Oo to have orthonormal columns. One reason is that Q t can be trivially computed Q QT. which is computationally cheaper to perform. Our experimental results also indicate that learning an Qo that produces Q with orthonormal columns
4

---

Under review as a conference paper at ICLR 2023
tends to be more robust to hyperparameter choices (see Appendix). For these reasons compute Q by first computing orthogonal matrix Q, and then extracting the first columns:
1k 0
Q =
We can obtaín n n orthogonal matrices by computing Householder transformations. However, in order to span arbitrary k-dimensional subspace, need to chain together & reflections:
H[(v)))((22) ...H(v)))
where H R" + goxa computes the Householder matrix that reflects about the hyperplane or- thogonal to v;:
(v.:) 2v.7, i (1,...A))
where each v; has unit norm. Next, we describe how NHT uses neural network to compute these V, unit vectors.
3.1.1 EXPONENTIAL MAP ON UNIT SPHERE.
We would like leverage neural networks learn map from contexts the v, needed com- pute Q. We can readily obtain unit v; from the output of a typical neural network he by simple normalization: V; ho (c)/ [[]a(())]; Unfortunately, this approach can result in unstable approx- imations. As the norm of (c) shrinks, arbitrarily small perturbations the context can cause disproportionate changes in V;.
a more stable alternative, we make use of the exponential map from Riemannian geometry Absil et al.] 2008), which maps points in the tangent space of a manifold to the manifold itself (in our case, the sphere). We seek unit vectors in R2,, which lie on the (n--))-phere. We can therefore make use of the exponential map on ((n-1) e₁ (the first standard basis vector, C₁ [1, 01, O]
The mapping
ExPe; E
(10)
map tangent vectors &; E (n-11) to unit vectors E R". We require k tangent vectors & that will map to the v; vectors used to compute Q. therefore configure the neural network hg to output vector { = I))=-)), We treat the output of ha as l stacked tangent vectors:
ho : c E E [ E ................................ eX]
(11)
We then use the exponential map each tangent vector, resulting in vector Ran of stacked unit n-vectors:
E E2
Exp
V2
(12)
VVk_
where v V vJ] Each v, is then used to compute a Householder matrix (Eq. which are composed to obtain Q(v) (Eq. 8 Overall, NHT (Q, Q) can be understood the composition each of these computations:
Q(v) (v)
(13)
NHT
where e is a context vector arbitrary dimension, and Q(v) is an n x n orthogonal matrix, and Q(v) is the matrix formed by the first & columns of Q(v).
In the context of learning systems, exponential maps have heen previously studied in the literature on normalizing flows (Rezende et al. 2020).
[0]
For the sphere, the exponential map e; is computed e, er (
sin(lle:
&.
5

---

Under review as a conference paper at ICLR 2023
3.2 EXISTENCE
If we hope to use NHT to approximate arbitrary subspaces, it is important to ensure that for every k-dimensional subspace W of IR", there exists a vector V E R such that W = span (Q(v)..
Remark. Let W _ be an arbitrary k-dimensional subspace. There is sequence of k House- holder reflectors Q = H H2 ... H2 such that the first l columms of Q are an orthonormal basis of W.
x k matrix whose column space is W. By (Trefethen & Bau 1997) Algorithm 10.1 we can construct a QR decomposition, M = QR where Q is the product of exactly k Householder reflections. Now are done because it is basic property of QR decompositions that the first columns of Q are an orthonormal basis for the column space M, which W. L
Proof. Let M be
Thus, given the existence of an optimal contextual subspace W (c), we can be sure that there exists some 4 such that Q(v*) spans W*. It is left to the neural network hg to approximate a set of tangent vectors that map v", given
3.3 SMOOTHNESS OF Qo
We conjecture that low-dimensional action interfaces that change abruptly from state-to-state may degrade learning in RL agents. Thus we are interested in whether or not Q Qo changes smoothly with respect to changes in the context. Concretely, we would like to limit the change in the high-dimensional actuation ú corresponding to an identical low-dimensional action a given that the change in the context is small. Let Q Qs(ci) and Q2 Qs(c2) for two nearby contexts, C1 and C2. Suppose an agent takes the same oww-dimensional action a in both contexts. Denote the corresponding actuation commands as úJ = Q and ú = Q2a, respectively. We would like to limit the magnitude of the change in ú (i.e. jû üz|)) with respect to changes in context. That is, we would like to find a constant T., such that:
úzll S Lallc1 call
(14)
where ||- refers to the vector n-rom. We begin by assuming that the agent is limited to low- dimensional actions with norm less than or equal to M. Then we have:
| ú2ll = ||Q, a Q2ai|
(15)
[Qq Q2)al|
(16)
< [[ Q2)
||a||
(17)
S MÍ Q2)Il
(18)
where the norm in Eq. [8 the matrix norm induced by the vector 2-norm. We now seek to limit this norm by finding scalar constant I. such that
[[O(c.) 2(c2)) s L||c - call
(19)
Given that such an L exists, is called Lipschitz constant, and @ is considered to be L-Lipschitz. turns out that there is well understood procedure for training Lipschitz continuous neural networks Gouk al. 2018). Using this Lipschitz regularization, we can choose constant L,, such that
E E2ll Lallc; -c2l
(20)
where E h((1)) and E. = h(c2). ) The exponential map on the sphere has a Lipschitz constant of 1, so we have the same result for the Lipschitz continuity of v with respect to changes in context. All that remains is to show that Q(v) is Lipschitz continuous.
Theorem 1. Let V1, V2 E Bnk be constructed from stacked unit n-vectors, and Q(v) be the product of the corresponding Householder reflections (as defined in Eq. 8 9.. Then,
[[Q(v;) - Q(v2) < Lo - V2l
(21)
where L.q 2VF.

---

Under review as a conference paper at ICLR 2023
NHT
LASER
SVD
7dof
WAMWipe
WAMGrasp
Success Rate
Success Rate
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
Figure 3: Learning curves corresponding the configurations with the best average final success rate, over all hyperparameter configurations, for each method. Each curve shows the mean success rate over five runs of the best configuration, with the shaded regions indicating the standard error.
Proof. Please see section A.3 in the appendix.
Using the fact that the Lipschitz constant of composition of Lipschitz continuous functions is upper bounded by the product of the constituent Lipschitz constants Gouk et al. (2018), combine the results of equation 20 and theorem to obtain a Lipschitz constant for @: NHT is Lipschitz continuous with L 24. V..
Thus, the low dimensional action is guaranteed result in similar actuations in nearby contexts: ي üر/ن < ML ||c,
-c2l
(22)
where ú = Q and Ú = Q2a, respectively.
4 EXPERIMENTAL SETUP
Our experimental results focus on validating the efficacy of neural householder transforms for learn- ing kinematic tasks within custom MuJoCo simulation of a Barrett WAM robotic manipulator with seven df ofgeres-of-freedom (see Figure 2}. We model each task as an MDP, and report results two environments: WAMWipe and WAMGrasp. Learning involves first training an NHT model
of- fline dataset of demonstrations, and then fixing the parameters of NHT and using Deep Deterministic Policy Gradient (DDPG) (Lillierap al. 2016 learn policy online. We compare DDPG agents trained with an NHT action interface against agents trained with a state-of-the-art latent action model Allshire et al., ).21), agents trained with actuation basis computed by SVD, and agents trained in the raw actuation space of the task (7dof joint velocity control). In our experiments we used a pub- licly available implementation of deep deterministic policy gradient (Andrychowiez et 2017
4.1 WAM ENVIRONMENTS
WAMWipe and WA.GGrap were designed study the effects of using NHT to augment reinforcement learning in kinematic manipulation tasks with binary reward function. These tasks can be classified accord- ing to how their goals are defined, and constraints on the configuration of the manipulator during execution of the task. Table [ in the appendix enumerates the goal type and constraints present in WAMWWpp and WAMGrasp. Section A. includes detailed descrip- tions of these environments.
Position
Target
Position
Target Orientation
(a) WAMWipe
(b) WAMGrasp
Figure 2: Simulated kinematic manipula- tion environments with distinct goal types and constraints.
4.2 ACTION INTERFACE BASELINES
In addition to training NHT from a dataset of demonstrations, we trained LASER Allshire et al., 202 from the same dataset, and computed the singular value decomposition (SVD) of the dataset
Ithough we used the implementation of DDPG introduced in the HER paper, we did not use HER in any of experiments.

---

Under review as a conference paper at ICLR 2023
of joint velocities executed during the demonstrations. In experiments, the clate-conditioned actuation basis computed by NHT. static basis computed by SVD, and nonlinear decoder of LASER all serve different choices of interface between DDPG and the actuation commands that determine the next state of the environment. In our WAMWipe experiments, NHT, LASER, and SVD all exposed two-dimensional action interface DDPG, while in WAMGrasp they all exposed paree-dimensione interfaces. The & E {2, 3 actuation bases provided by SVD were the vectors in IRT corresponding the k largest singular values.
Demonstrations were collected by recording observation-actuation (c, u) pairs from PD controllers that were hand-engineered for each environment. For WAMWipe, the dataset consisted of 20,000 transitions, where single demonstration consisted of roughly 250 transitions on average. In WAMGrasp, the dataset consisted of 100,000 transitions, where single demonstration consisted of roughly 100 transitions on average. In both environments, the observation с upon which the output of NHT and LASER are conditioned was the concatenation of joint angles and Cartesian coordinates of the end-effecctr. LASER is regularized by the KL and dynamics terms in its loss function (see equation 3) while we regularize NHT by enforcing Lipschitz continuity with Lips- chitz constant L at each layer during training (Gouk et al., 2018). The Adam optimizer is used for both NHT and LASER, with learning rate amap. and otherwise default parameters. Likewise, Adam is used the optimizer in our chosen implementation of DDPG, with learning rates Caactor and Ceritic for the policy and value function, respectively.
4.3 HYPERPARAMETE SEARCH
We would like to estimate the performance of the best policy that could be learned by DDPG in finite amount of time for agents trained with (1) an NHT action interface, (2) a
NHT
LASER
SVD
7
Final Performance
1.00
0.75
LASER AAlshhre et al. 2021) action interface, (3) an actuation basis computed by SVD, and joint velocity ac-
0.50
(4) seven ofgrre--o-frreddon tions. In addition, we would like to study the sensitivity of agent Saarningddynamin ferent hyperparameter configurations for each action interface. As such, we performed a ran- dom search over map (i.e. NHT, LASER) and DDPG hyperparameters. We jointly sampled 128 configurations each for NHT + DDPG. LASER + DDPG, DDPG with SVD, and DDPG with joint velocity actions. For the latter two conditions the only hyperparameters of interest are those of DDPG itself. For each configuration, we first trained the mapping function (if applicable), and then trained the DDPG agent, over five runs with different random seeds. We chose randomly sample hyperparameters because of previous work suggesting it to be more computationally efficient to find better hyper parameters (Bergstra & Bengio 2012). The ranges and method of sampling used for each hyperparameter are listed in Table 2 of the appendix.
0.25
0.00
to dif-
WAMWipe
WAMGrasp
Figure 4: Violin plots of final success rates across 128 randomly sampled hyperparameter configura- tions (5 runs each).
4.4 HALFCHEETAH
While main interest for the application of NHT lies in constrained/safe robotic manipulation, there is value in validating the utility of NHT on more standard reinforcement learning environ- ments. In addition, it is important to show that the action interface learned by NHT useful for agents trained with various RL algorithms: not only for DDPG agents. We therefore performed hyperparameter search experiment with a standard implementation DDariwal et al. (0117) of PPO SShulman et al. (2017) on the HalfCheetah-v4 environment from OpenAl Gym Brockman et al.) (2016). This environment has 17-dimensional observation space that includes angular positions and velocities, and a 6-dimensional torque actuation space (compared to the joint velocity actuation space in WAMGrasp and WAMWipe). We compared NHT agents to agents that learned in the stan- dard 6dof actuation space of HalfCheetah, and agents with LASER Allshire et al. (2021) and SVD action interfaces. NHT, SVD, and LASER all learned 2-dimensional action interfaces. This exper- iment precisely mirrored the hyperameter search experiments reported in section 4.3 except that
8

---

Under review as a conference paper at ICLR 2023
NHT
LASFR
SVD
6dof
6000
1000
Finel Performence
5000
4000
3000
Aug
2000
1000
0.2
0.4
0.8
1.0
1000
Timestops
MaltChooial-wd
Figure 5: Results of hyperparameter search for action mapping methods in HaliChectaah-4, Left: Learning curves corresponding the configurations with the best average final success rate. Right: Violin plots of final success rates across 128 randomly sampled hyperparameter configurations (5 runs each).
the demonstrations used to train NHT, SVD and LASER were collected from the best- rest-performing policy learned by the standard 6dof agent. A total of just 1,000 transitions were recorded from this expert policy. The return this demonstration episode and sampling methods for this experiment are summarized in Table (see Appendix).
6,000. The hyperparameter ranges
5 EXPERIMENTAL RESULTS
Figure summarizes the results of the random hyperparameter search in the WAMWipe and WAMGrasp environments. The violin plots represent the distribution of final success rates (success rate after 100 epochs of training) across every randomly sampled hyperparameter configuration. The learning curves in Figure plot the mean success rate during training for the best performing agent in each condition, averaged over five runs.
can be seen that DDPG agents trained with an NHT action interface produced the best performing agents after hyperparameter optimization (higher success rates in fewer epochs) in both WAMWipe addition, the distributions of final success rates across hyperparameter configu- rations suggest that agents trained with NHT are more robust to hyperparameter choices compared to the baselines. Although some runs the 7dof agent managed to reach a success rate 100% in WAMGrasp. the variance of final success rates amongst 7dof agents is much larger than the variance of success rates for NHT agents. In general there was not a strong correlation between any one hyperparameter and the final performance of the agents (coefficient of determination < 0.1).
and WAMGrasp.
The learning curves of the agents with the best average final performance, and the distribution of final agent performances for each method in HalfCheetah- v4 are shown in Figure Interestingly, found that the constant (i.e. nor state dependentdd action interface SVD was sufficient to learn more efficiently than the standard 6dof agent while still achieving the same asymptotic performance. This suggests that all of the instantaneous actuations used by an expert (> 6, 000 return) HalfCheetah agent lie close agent learning in an adaptive actuation subspace with NHT, although the performance gains are small in this environment. The agents learning with NHT tended again be more robust different hyperparameter configurations.
fixed 2-dimensional linear subspace! There appears to be some benefit to the
6 CONCLUSION
We proposed contextual subspace approximation as a novel alternative to deep latent actions models for robotic control. We derived the Neural Householder Transform model as an approach con- textual subspace approximation, and showed that it is smooth with respect to changes in context. In a large hyperparameter search experiment, we found that reinforcement learning agents trained with NHT outperformed agents trained to act in (1) the original actuation space, (2) global linear actuation basis computed by SVD, and (3) a state-of-the- deep latent action model, LASER, in novel WAMWipe and WAMGrasp robotic manipulation environments, well in the standard HalfCheetah environment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pierre-Antoine Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds, volume 78. 2008. ISBN 978-0-691-1329% doi: 10. 10.1515//98814003302222
Arthur Allshire, Roberto Martún-Martún, Charles Lin, Shawn Manuel, Silvio Savarese, and Ani- mesh Garg. LASER: learning a latent action space for efficient reinforcement learning. CoRR, abs/2103.15793, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran As- httpp//pss....ppppp/////////eeee
sociates, Inc., 2017. /ttp///tss///prreeeiis...ssssss 455.aaaesaaaaaaaaaaaffffff533333
Panagiotis Artemiadis and Kostas J. Kyriakopoulos. Emg-based control robot arm using low-dimensional embeddings. IEEE Transactions on Robotics, 26(2):393-398, 2010. doi: 10.
1109/TRO.2009.2099333
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10))281-305, 2012. URL http://t owtpp/pergpppppps///// v 213/bergstral2a.htm
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jyotikrishna Dass and Rabi Mahapatra. Householder sketch for accurate and accelerated least-mean- squares solvers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp-
2467-2477. PMLR, 18-24 Jul 2021. URL https: httt::::/pproceedingg mlr wwl.ppress/v19/
dass2la. html
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. 2017.
https:
7/github. com/ ceemm/ppenaa/baaelineee
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing lipschitz continuity, 2018. URL htt.:pp::/aa.....rr///a/////////
04368
Hong Jun Jeon, Dylan Losey, and Dorsa Sadigh. Shared Autonomy with Learned Latent Ac- tions. In Robotics: Science and Systems XVI. Robotics: Science and Systems Foundation, July 2020. ISBN 978-0-9923747-661. doi: 10. 15.507/RSS.2020.X1T roboticcsprceeednng... orgp://ddiininn.oosssoeeeeeeeeee http://ccceeeaooooo.............
URL hhttp://wwr
Siddharth Karamcheti, Albert J. Zhai, Dylan P. Losey, and Dorsa Sadigh. Learning visually guided latent actions for assistive teleoperation. Ali Jadbabaie, John Lygeros, George J. Pappas, Pablo A. Parrilo, Benjamin Recht, Claire J. Tomlin, and Melanie N. Zeilinger (eds.), Proceedings the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learming Research, pp. 2230-1241. PMLR, 07 08 June 2021. Utt.http://ppoceeeee ngs..
m mlr. w.r.press/vv 4 www.//aaammaaa raacaaaachee.... 21a html
Diederik P. Kingma and M. Welling, Auto-Encoding Variational Bayes. ICLR, 2014.
Jens Kober and Jan Peters. Learning motor primitives for robotics. In 2009 IEEE International Conference Roboties and Automation, pp. 2112-2118, 2009. doi: 10.1109/ROBOT.2009 5152577.
Gen Li, Laixi Shi, Yuxin Chen, Yuantao Gu, and Yuejie Chi. Breaking the sample complexity barrier to repret- model-free reinforcement learning. CoRR, 10.0444, 2021. URL
httpss://aret wttp://w..wga//b///ooooo// 04645
10

---

Under review as a conference paper at ICLR 2023
Hongzhuo Liang, Lin Cong, Norman Hendrich, Shuang Li, Fuchun Sun, and Jianwei Zhang. Mul- tifingered grasping based on multimodal reinforcement learning. IEEE Robotics and Automation Letters, 7(2:1174-1181, 2022. doi: 10. 10.1009/RA.22211113355.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. URL http: 7/axxiv. org/ abaaabas/s5 009. 02971
Dylan P. Losey, K. Srinivasan, Ajay Mandlekar, Animesh Garg, and D. Sadigh. Controlling Assis- tive Robots with Learned Latent Actions. 2020 IEEE International Conference on Robotics and Automation (ICRA). 2020. doi: 10. (W09/CCRAA009452220099999979
Dylan P. Losey, Hong Jun Jeon, Mengxi Li, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Jeannette Bohg, and Dorsa Sadigh. Learning latent actions to control assistive robots. CoRR, htw/2107.029011 2021. URL https: http://axxvv.oroe wttpa//.wwr///////......ooo 02907
Roberto Martin-Martin, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, and Animesh Garg. Variable impedance control in end-effector space: An action space for reinforce- ment learning in contact-rich tasks. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1010-1017, 2019. doi: 10.1 /ttp//000S00897221988866666
Alexander Mathiasen, Frederik Hvilshøj, Jakob Rødsgaard Jørgensen, Anshul Nasery, and Davide Mottin. (Faster Orthogonal Parameterization with Householder Matrices. In (ICMI. Workshop on Invertible Neural Networks and Normalizing Flows, 2020.
Giulia Matrone, Christian Cipriani, Maria Chiara Carrozza, and Giovanni Magenes. Real-time myo- electric control of a multi multi-finggeed hand prosthesis using principal components analysis. Journal of neuroengineering and rehabilitation, 9:40, 06 2012. doi: 10.1 10.1186/1743-0003---46
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2401-2409, PMLR, 06-11 Aug 2017. URL htttss:://prroeeddngee ngg ml www.s/ressvvvv... http:///0000mmmm/mmmme 17a html
Rajbir Nirwan and Nils Bertschinger. Rotation invariant householder parameterizzatio for Bayesian PCA. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- fernational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 4820-4828. PMLR, 09-15 Jun 2019. URL htttp:://proceeea/e .ml....s.m......mmm.mmm........m
press/r97/nirwan1.e
Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang, Stamatios Georgoulis, Dengxin Dai, and Luc Van Gool. Spectral tensor train parameterization of deep learning layers. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Confer- ence on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3547-3555. PMLR, 13-15 Apr 2021. URL. pattps://eat /ttpps://procceediggss ftess/v130/obukhov222
html
Aggeliki Odest and Odest Jenkins. 2d subspaces for user-driven robot grasping. Robotics, Science and Systems Conference: Workshop on Robot Manipulation, 2007.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic move- ment primitives. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL (r0022276628222000ccc222a------ 77
pdff
Danilo Jimenez Rezende, George Papamakarios, Sebastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows tori and spheres. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- ence Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp-
11

---

Under review as a conference paper at ICLR 2023
8083 8092. PMLR, 13 18 Jul 2020. URL htt.:t/ss//ppoeed.nn........eess rezende2Da.. html
Marco Santello, Martha Flanders, and John Soechting. Postural hand synergies for tool use. The Journal of Neuroscience, 1010055,,, 12 1998. doi: 10. 10.1523/NEUROSCI. 18-2330000... 1998.
Stefan Schaal. Dynamic Movement Primitives Framework for Motor Control in Humans and Humanoid Robotics, pp. 261-280. Springer Tokyo, Tokyo, 2006. ISBN 978-4-431-313811 doi:
10.10077444311-31388 23. (tRlhhttp::///ot http://oooor/rg////// -
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL
1777000037
Sahil Singla, Surbhi Singla, and Soheil Feizi. Improved detemminniti 12 robustness cifar- 10 and cifar-100, 2021. httpptttss///axx...rr//////////0
Jakub M. Tomczak and Max Welling. Improving variational auto-encoders using householder flow, 2016.
abs 611. 09630
Samuele Tosatto, Georgia Chalvatzaki, and Jan Peters. Contextual lntent-movements off-policy optimization for robotic manipulation skills. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 10815-10821, 2021. doi: 10.1 10110//CRA445562222..0555
L.N. Trefethen and D. Bau. Numerical Linear Algebra. Other Titles in Applied Mathematics. Society for Industrial and Applied Mathematics, 1997. ISBN 9780898713619, URL https:
//bbooks.gog/// com/ cotm///oookss111--/oo3355ppp.pp
Rianne van den Berg, Leonard Hasenclever, Jakub Tomezak, and Max Welling, Sylvester normaliz- ing flows for variational inference. In proceedings of the Conference on Uncertainty in Artificial
Intelligence (UAI), 2018.
Elise van der Pol, Thomas Kipf, Frans A. Oliehoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgen Systems, AAMAS '20, pp. 1431-1439, Rich- land, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450375184
Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing gradients for deep neural networks via effi- cient SVD parameterization. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 5806-5814, PMLR, 10-15 Jul 2018. URL httph:tss//// http:/ss///prrcceningee mlr.
press/v80/zhang113 html
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. 2020.
A APPENDIX
Here we provide additional details about the WAMWipe and WAMGrasp environments, details about our hyperparameter search, and a proof for the theorem presented in Section (3. of the main paper. We also include in section A.4 supplementary results from an ablation study where we strip NHT of the constraint that must output orthonormal bases. Finally, we compare NHT to a Jacobian pseudoinverse action interface in section A.5)
A.1 RL ENVIRONMENT DETAILS
In both WAMWipe and WAMGrasp, environment step corresponds to 10 MuJoCo simulation time-steps of length 0.002 seconds each. Both environments allow a maximum 200 environment steps per episode.
12

---

Under review as a conference paper at ICLR 2023
Env
Action Dim. Actuation Dim.
Goal Type
Constraints
WAMWipe
Pos
Safety, Contact, Orientation Safety
WAMGrasp
Pos, Orientation
Table I: Properties of reinforcement learning environments in simulation experiments.
A.1.1 WAMWIPE
In WAMWipe the goal is to control the manipulator such that the flat face of the last link remains flush against a table while sliding unless the end-effector is within a small distance of the goal position, in which case the reward is Episode failure occurs if the end-effector:
randomly sampled goal position. The reward is -1 every step
(1) Pushes into the table, (2) Lifts off of the table, or (3) The end-effector tilts such that it is no longer flush with the table. Let denote the unit vector orthogonal to the face of the end-effector, pictured as a purple arrow figure Constraint (3), the orientation constraint, was considered violated when the angle between p and the vector orthogonal to the surface of the table (not pictured) was greater than 1/16 radians. The agent observation in experiments
concatenated vector of joint angles, Cartesian coordinates of the endeeffector, Cartesian coordinates of the goal position, and the unit vector orthogonal to the face of the end- effector. The actions in our WAMWipe experiments were either 7 joint velocity commands, or 2-dimensional actions input to an NHT, SVD, or LASER action interface.
A.1.2 WAMGRASP
In WAMGrasp the goal is to simultaneously reach randomly sampled grasp-point, while achieving goal orientation that is determined by the grasp-point. Let p" denote the unit vee- tor pointing from the grasp-point (small sphere Figure to the object being grasped (large sphere in Figure I.I.. We consider the orienta- tion satisfactory if the angle between p* and the vector orthogonal to the face of the manip- ulator, p. is less than π/ / radians. The reward in WAMGrasp is 1 at every step unless the end- effector is within a small distance of the grasp- point with a satisfactory orientation. Episode failure occurs the end-effector collides with either the object being grasped (large red sphere in Figure A.I or the table. In each episode the grasp-point is randomly sampled from the sur- face of a sphere with the same center but larger radius than the large red sphere in Figure A. The agent observation was a concatenated vector of joint angles, Cartesian coordinates of the end- effector, and Cartesian coordinates of the grasp point. The actions in our WAMGrasp experiments were either 7dof joint velocity commands, or 3-dimensional actions input to an NHT, SVD, or LASER action interface.
0
P
p
Figure A.1: The 3-vector p pictured here was used to determine whether the orienta- tion constraint/goallconttee
was satisfied in WAMWipe/AAMGrap. respectively.
A.2 HYPERPARAMETER SEARCH DETAILS
The hyperparameter search experiment described section 4.3 of the main paper was designed to estimate the performance of the best policy that could be learned by DDPG in finite amount of time for agents trained with (I) an NHT action interface, (2) a LASER Allshire et al. (202I) action interface, (3) actuation basis computed by SVD, and (4) seven degree- of-eee--ofrrreedm joint velocity actions.
The hyperparameter search also enabled us study the sensitivity of agent Cearning-dynamin to different hyperparameter configurations for each action interface. The results serve as empirical evidence with which to answer questions such as: "Could changing the neural architecture of NHT
13

---

Under review as a conference paper at ICLR 2023
cause significant drop in the final success rate of a policy learned by DDPG?" Answering such questions is non-trivial since there may or may not be complex interactions between map (i.e. NHT, LASER) hyperparameters, DDPG hyperparameters, and final agent performance. It is unknown whether NHT hyperparameters tuned for an agent with arbitrary configuration will be the best NHT hyperparameters for agent with different configuration B. For example, is conceivable that DDPG agent with hyperparameter configuration A may perform best with NHT configuration C. while DDPG with configuration B performs best with NHT configuration D. Thus meaningful search should jointly vary the hyperparameters of the mapping models and the DDPG agent.
We jointly sampled 128 configurations each for NHT DDPG, LASER DDPG, DDPG with SVD, and DDPG with joint velocity actions. For each configuration, we first trained the mapping function (if applicable), and then trained the DDPG agent, over five runs with different random seeds. The range of values and sampling method used for each hyperparameter are listed in table 2
For both WAMWipe and WAMGrasp, each agent was trained for one million environment steps, using three workers to generate experience. This resulted in 100 training epochs of 10,000 steps each.
Parameter
Range
Sampling
Omap
(le-6, le-1)
Exponential
8KL (LASER)
(le-6, 1e0)
Exponential
Bäyn (LASER)
(le-6, le0)
Exponential
L (SCL)
(Ie-1, le2)
Exponential
Map Batch Size
(32, 256)
Uniform
Map Activation
[ReLU, sigmoid, tamh]
Uniform
Map Hidden Layers
1, 3, 4] (16, 1024)
Uniform
Map Hidden Units
Geometric
Oaactor
(le-4, 1e-2)
Exponential
Oceittic
(1e-4, le-2)
Exponential
Rand Action
(0, 0.4)
Uniform
Action Noise
(0, 0.4)
Uniform
Penalty on ||a]|
(0, 1)
Uniform
Max ||a||
(1, 10)*
Exponential
DDPG Batch Size
(32, 256)
Uniform
Polyak
(0.9, 0.99)
Uniform
Table Hyperparameter ranges and sampling methods for pre-trained mapping functions (top) and DDPG (bottom). We use lez as shorthand to denote 1 x 10ah Exponential and Geometric sampling of parameters was carried out as described in Bergstra & Bengio (2012). *For the agent with 7dof actions we use the range (0.1, 10).
The parameter ranges and sampling methods for the PPO agent used in the HalfCheetah hyperpa- rameter search are given in table
Parameter
Range
Sampling
(1e-4, 1e-2)
Exponential
Discount Factor
(0.9, 0.99)
Uniform
GAE Parameter
(0.9, 0.99)
Uniform
VF coefficient
(0.25,1)
Uniform
Clipping Parameter
(0.05,0.5)
Uniform
Steps-per-update
(1024.2048,409)
Uniform
Table 3: Hyperparameter ranges for PPO in the HalfCheetah-v4 experiments. We 1ea: short-
hand to denote x 10". Exponential sampling of parameters was carried out as described in Bergstra Bengio (0122)
14

---

Under review as a conference paper at ICLR 2023
A.3 SMOOTHNESS Q(v)
In this section we prove the Lipschitz continuity of Q(V), as stated in theorem |
Theorem 1. Let V1, V2 Ign be constructed from k. stacked unit n-vectors, and Q(v) be the product of the corresponding Householder reflections (as defined in Eq. 8 9.. Then,
[IQ(v,) - Q(v2)) s LQllv - V2l
(21)
where LQ T.
We write H, as shorthand for H (i) I - 2v.v. We write E R denote the concate- nated column vector of V, E : [v] V2 1.... I We denote the map from to the corresponding product of reflections as Q V V Q(v), where
Q(v) = H(v))H(vv)
H(vk)
(23)
We likewise write 8 E TR denote the concatenated vector of perturbations each V;
8
C 61
82
C202
8
(24)
where ||8]| 1, with scalars e E R scaling the unit norm 6, vectors that represent the direction of change for each v.. We consider the directional derivative of Q(V) in the direction of Ó:
Q(v + v(v)
V
lim c00
(25)
e
where ||8| = 1.
The existence of positive constant LQ that bounds Q(V):
V VVQ(v)I implics Lipschitz continuity of
IQ(()) - Q(v2)((11)))))))))())))) & Lo V - v2ll
(26)
for all V1, V2 constructed with A stacked unit n-vectors. We explicitly compute such LQ below. As a first step, we show section A.3.1 that VvH(v)]] = We will then use this result to compute an upper bound on LQ in section A.3.2
A.3.1 LIPSCHITZ CONTINUITY OF H(v)
The directional derivative of H(v) in the direction of ò is defined as:
H(v + ( -H(v)
VoH(v) vv = lim c-0
(27)
sphere, and thus any instantaneous change to must occur a direction tangent to the sphere at v; that is, v. Furthermore, without loss generality let || Thus, a unit vector the direction of the perturbation of v.
where 5 R Recall that is in the
We first simplify the first term in the numerator:
H(v + ed) = 1-2(v
2(v + +v) + =d)T
(28)
I 1-2(vv) cdv + EVōT 288T)
(29)
- 2vv - 2(e8v + +ev + +688
(30)
= H(v) - 2(Eōv T + +evô" T
(31)
Substituting the result into the definition of VoH(v), we have:
2(c5v T + + 268
V&H(v)
lim c-+0
(32)
lim 2(5v voT + c66
(33)
c-1
=- +v8
(34)
15

---

Under review as a conference paper at ICLR 2023
Now compute [v)l())] Note that the symmetry of the sphere guarantees that ||v:H(v)]] is invariant with respect to both and v.
V (v)xll
[vH((v)]
max x=0
(35)
|
The numerator is maximized when x is in the plane spanned by and 6. Given this is the case, we can write x as a linear combination of v and 6. Let
x= ov + +,8
(36)
for some a, 3 E IR. We then have the following:
V [vH((v)x
[vH((v)ll = max
(37)
||
2(0v] T + vδ }x|
max x=0
|||||
(38)
[(vv + vō (av + 1301
(39)
||X|
llavv v + avō v + 15v 3v8
(40)
[x
1 v B38 8||
(41)
|x||
|ad8 + 3v
(42)
||x||
where equation follows from 40 by the fact that 1 v. Equation 42 follows from the fact that both 5 and v have unit norm.
Now, recall that av av 36. The numerator 42 represents a simple change of basis for x. Since 8 and are orthonormal, this change of basis preserves the norm of x. Hence ||ax + 3vll = ||x|| and we have:
V (v)ll) =
(43)
This implies H(v) is Lipschitz continuous with Lipschitz constant 2.
A.3.2 LIPSCHITZ CONTINUITY OF Q
We now consider the directional derivative of Q(v) in the direction of 8:
Q(v oō) Q(v)
V3Q(v)
lim c-+0
(44)
where ||0|| 1. Recall:
8
8,
c2d2
8
(45)
C
Theorem Let V1, V2 Bink be constructed from stacked unit n-vectors, and Q(v) be the product of the corresponding Householder reflections (as defined in Eq. 8 9.. Then,
[[Q(v,) - Q(v2)( < LQ [1 -22l
(21)
where TQ= 2VT.
Proof. We begin by expanding the numerator of V
H(v, o0 H(v2 + c8)) H(v: + +68) Q
VgQ(v) = lim
(46)
c-0
16

---

Under review as a conference paper at ICLR 2023
We now consider the first term in the numerator. In the following write H, shorthand for H(v)), and VH, as shorthand for Vo. Hv,), the derivative of H(v,) as defined in equation (27).
H(v <0 H(v2 + =8)) ... H(v + =0%)
(47)
(H1 + 60I vHI + 602 VH2) (HA + 6ck VH,) O
(48)
= H H2 ... H + ecI H2 ... HA. + 602 H (VH,) 2 H3 ... H -...
(49)
+ c H H2 H (VH)) + O
(50)
k-1
Q(()) e01 (H, H. +
H.)
cc
H (VH + O(e)) 0(E))
t=3
(51)
Q(() +e
Σe Cj
I H))H)) II H:)
+
(52)
J=1
(=++1
and substitute the result into the definition of V (VV):
Q(v) + Ej= oj
H. HH H.) + +02) Q(v)
V8Q(v)= lim c00
(53)
Σe Cj
H. H.)) II H:)
(54)
(=j+1
We now work toward bounding the norm of QVQ(v):
[
Σ C
II H:)
(55)
1=j+1
A II H:)
Σ
9
H))HH,
(56)
1=j+1
(57)
b: VH 12
(58)
|8|
V8, H(v,
(59)
(2)2
(60)
2Vh
(61)
Where equation 57 is thanks to the fact that each of the H; in the preceding equation orthogo- nal, and equation (58) follows by the Cauchy-Sehiee Schwarz incquality.
Hence, the norm of the directional derivative of Q(v) is bounded by 2 Te; that is:
\QQ()| < 2VK
(62)
which implies
[[Q(v;) - Q(v2)) < LQ V1 - -22
(63)
with Lipschitz constant T-q = 2vk.
n
17

---

Under review as a conference paper at ICLR 2023
A.4 ABLATION OF ORTHONORMAL CONSTRAINT
What is the benefit of enforcing orthonormal actuation bases in NHT? Beside the fact that the pseu- doinverse of Q can be computed trivially the transpose during training, wanted to find out if there was any empirical benefit. To answer this question we performed an experiment in which we trained a neural network to produce an arbitrary state-conditioned matrix as an actuation basis for WAMWipe and WAMGrasp. Unlike NHT, this baseline is not constrained to output a matrix with orthonormal columns. We will refer to the baseline as the state-conditioned linear map (SCL) model. The SCL baseline a neural network hg c +> B E Rn* that maps context vectors to an n x k matrix B. In our WAMWipe experiment n = and 2, while for WAMGrasp n = 7 and k 3.
NHT
SCL
1.00
Final Performance
0.75
0.50
0.25
0.00
WAMWipe
WAMGrasp
Figure A.2: Violin plots of final success rates across 128 randomly sampled hyperparameter config- urations (5 runs each) for NHT vs a non-orthonormal state-conditioned linear (SCL) action mapping baseline.
As in section 4.3 of the main text, performed hyperparameter search in which we sampled 128 hyperparameter configurations and trained NHT/SCL and then a DDPG agent with 5 different random seeds. The sampling method and ranges for each parameter were the same for both NHT and SCL, and are listed in Table 2 of section A.2. Figure A.2 summarizes the results of this hy- perparameter search with violin plots of the final performance attained by agents with NHT and SCL action interfaces for the WAMWipe and WAMGrasp environments. The NHT hyperparameter search results reported this figure are the same as those reported in section 4.3 of the main text.
We found that 64 out of 640 (10%) of the runs for SCL. in WAMWipe failed due to numerical in- stability. In these cases the matrices output by the unconstrained neural network had large norms, resulting in very large joint velocity actuations that caused the mujoco simulations to fail. Interest- ingly, we did not observe the same numerical stability issues in the SCL models that were trained for WAMGrasp. Note that, in contrast, for NHT numerical stability is not an empirical issue. The 2-norm of the matrix produced by NHT is guaranteed to be equal to one.
In WAMWipe, the best hyperparameter configurations of SCL resulted in actuation interfaces that were suitable for the DDPG agent to achieve 100% success rate. However, in both WAMWipe and WAMGrasp. the distributions of final agent performance in figure A.2 indicate that NHT more robust than SCL with respect to variation in hyperparameter configurations. This suggests that NHT may be less sensitive different choices of hyperparameters, making it easier to tune in practice.
A.5 COMPARISON TO JACOBIAN PSEUDOINVERSE INTERFACE
Here we compare NHT to an additional choice of action interface that, unlike the baselines discussed in the main text, is not learned from demonstrations, The Jacobian of the robotic manipulator de- scribes the relationship between the joint velocities and the Cartesian and angular velocity of the end-effector. The pseudoinverse of the Jacobian can be used to define six-dimensional action in- terface for an RL agent: dimensions in the agent's 's action space correspond to Cartesian velocity, and the remaining 3 correspond to angular velocity.
18

---

Under review as a conference paper at ICLR 2023
NHT
Jacobian pinv
1.00
Final Performance
0.75
0.50
0.25
0.00
WAMWipe
WAMGrasp
Figure A.3: Violin plots of final success rates across 128 randomly sampled hyperparameter configu- rations (5 runs each) for NHT vs Jacobian pseudoinverse (Jacobian pinv) action mapping baseline.
We compared NHT to the Jacobian pseudoinverse as an action interface for DDPG agent in WAMGrasp and WAMWipe described in section 4.3 of the main text (128 hyperparameter configurations, 5 seeds for each con- figuration). As already noted, the dimensionality of the agent's s action space was when using the Jacobian pseudoinverse interface. NHT used to learn 2-dimensional action interface for WAMWipe, and a d-dimensional action interface for WAMGrasp. The hyperparameter search re- sults are plotted in figure A.3 The NHT hyperparameter search results reported in this figure the same as those reported section 4.3 the main text. The variation in performance for the Jacobian pseudoinverse agents entirely due to different DDPG agent configurations (the Jacobian has hyperparameters)
hyperparameter search experiment with the same methodology
As expected, the agent with the Jacobian pseudoinverse action interface performed poorly in WAMWipe; like the 7dof joint velocity agent, the Jacobian pseudoinverse agent was able to freely jam the end-effector of the robot into the table, lift the end-effector from the table, resulting in an automatic failure for its training episodes. Without a learned action interface, the exploratory behavior inherent in reinforcement learning resulted in destructive behavior that made learning in the highly constrained environment of WAMWipe difficult.
In WAMGrasp, the Jacobian pseudoinverse agents were sometimes able to learn to achieve 100% success rate. However, it is clear from the violin plots in figure A.. that limiting the joint velocity commands to useful subspaces learned by NHT has some benefit over allowing free exploration with the Jacobian pseudoinverse interface. Some of the poorer hyperparameter configurations resulted in close to 0% success rate when interacting with WAMGrasp through Jacobian pseudoinverse interface. NHT tended to concentrate agent performance, over all hyperparameter configurations, toward a success rate of 75% to 100%.
19


Reviews:
[{"review_id": "j79YFuWQ1o", "reviewer": "Reviewer_1kbF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed neural household transforms (NHT) for learning a low-dimensional action representation for RL problem. Prior work in this area usually learns an autoencoder-style action representation using neural networks. In this work, the authors propose to learn a network that maps to an orthogonal matrix instead of the original action space. By doing this, it enables them to avoid the need of encoding the action into the latent space as it can be obtained by an analytical least square solution. To obtain a well-behaved mapping function, exponential map is used to map the neural network output onto a unit sphere, which is then formed into the orthogonal projection matrix. The authors provided theoretical justifications for the approximation error as well as the smoothness of the proposed method. The algorithm is evaluated on two manipulation tasks and a locomotion task in simulation and is demonstrated to outperform baseline methods.", "summary_of_the_review": "As discussed in the weaknesses, my main concerns for the work is the comparison fairness to the baseline as well as using a stronger learning algorithm for the manipulation tasks. I think the algorithm itself is novel and interesting in my knowledge and if the concerns were addressed I believe this would be a good one for the venue.", "strength_and_weaknesses": "Strengths:\n1. The proposed NHT algorithm is intriguing, with theoretical justifications, and without much compromise to make the method practical.\n2. Compared to alternative methods that learn a latent action space, the proposed method appears to perform better.\n\nWeaknesses:\n1. The manipulation tasks were trained with DDPG, while newer methods like SAC or TD3 has shown notably better learning performance. It’s not clear how much impact the algorithm would still make given a stronger algorithm.\n2. The comparison to using the raw action space is not entirely fair. For example in the half-cheetah case an optimal policy is already trained to collect data for learning the action representation. I think a more proper way of comparison is to separate the tasks for training the action space representation and evaluating the representation performance. For example for the two manipulation tasks one can learn the action space with one task and test on the other. For the half-cheetah task one may learn the action space with a running backward policy and test with a running forward policy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O7-5-gIvdN2", "reviewer": "Reviewer_2zLC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new latent action model based on Neural Householder Transform, where context dependent linear actuation subspaces are the outputs of an encoder-like neural network. The encoder outputs can be transformed via the exponential map on the sphere to form Household reflectors, that can be used to compute the context-dependent subspace. These local subspaces when interfaced with standard RL approaches can lead to more efficient and robust (w.r.t hyperparameters) learning in some environments.", "summary_of_the_review": "I would recommend the acceptance of the paper, see above.\n\n=== POST-REBUTTAL EDIT ===\n\nThanks to the authors for providing a careful rebuttal. As mentioned before, the method proposed in the paper is quite nice and novel in my opinion. However upon deeper reading the paper, the comments and the rebuttal, I am not convinced about the results of the paper. In particular relating to the hyperparameter-robustness, I am not convinced of the link between the NHT transform and increased robustness to hyperparameters. It's not clear to me why the context dependent linear subspaces emitted by the NHT transform would lead to such robustness, as opposed to the nonlinear dimensionality reduction achieved by LASER. I would rather expect improved learning curves, and better generalization (as I noted in my reply to the authors). \n\nAs hyperparameter-robustness seems to be the main claim of the paper (as an improvement), I'm lowering my score to 5, but I'm happy to reconsider if the reviewers would like to give some feedback on my concerns.", "strength_and_weaknesses": "The paper is well written and the introduced approach is quite nice in my opinion.\n\nStrengths:\n\n* The Neural Householder Transform and the way it is computed, can yield a flexible way to compute locally well approximating linear subspaces, as indeed shown by the paper. \n\n* By bounding the network with a Lipschitz bound, the authors can also show that the computed transform is smooth. This result can perhaps be used computationally as well, however the authors have not done so (e.g. during training).\n\nWeaknesses:\n\n* The claims of the paper about the contributions are not rigorously demonstrated, the authors themselves use \"tend to\", to describe the effect of the framework (when compared with competing approaches). This can be remedied by either more detailed experiments, and/or by a more stringent analysis of the failures/benefits of the approach. \n\n* It is not clear if some of the NHT is the author's contribution, e.g. the particular output representation utilizing the exponential map.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0yPo1dUGKZ7", "reviewer": "Reviewer_XySg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to learn a linear subspace for actions to control agents. It is well-known in the RL community that choice of action space has a dramatic effect on the learning process. For example, end-effector control in robotic manipulation is much easier than direct torque control for several tasks. Thus, learning better action spaces is an important research problem. This work specifically aims to learn local linear subspaces for actions, and utilizes NHT for efficient learning and inference. The paper also demonstrates that subspaces learned using NHT have desirable local smoothness properties.", "summary_of_the_review": "It is unclear why this problem is important and relevant to the NeurIPS community. The authors can either motivate the problem statement better or submit to a more specialized conference where it might better appeal to the audience. ", "strength_and_weaknesses": "The paper proposes a nice algorithmic approach for the problem studied -- i.e. finding local linear sub-spaces for actions. However, the paper does not do a good job of motivating why this is an important problem. What are the specific advantages of linear subspaces? Why is it preferable to nonlinear per-timestep action-space mappings (nonlinear latent actions) and trajectory-level action-spaces (motor primitives)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject