Bringing robotics taxonomies to continuous domains via GPLVM on hyperbolic manifolds
Authors: Noémie Jaquier
Leonel Rozo
Miguel González-Duque
Viacheslav Borovitskiy
Tamim Asfour
Venue: Submitted to ICLR 2023 (2023)
Keywords: GPLVM
hyperbolic geometry
robotic taxonomies

Abstract: Robotic taxonomies have appeared as high-level hierarchical abstractions that classify how humans move and interact with their environment. They have proven useful to analyse grasps, manipulation skills, and whole-body support poses. Despite the efforts devoted to design their hierarchy and underlying categories, their use in application fields remains scarce. This may be attributed to the lack of computational models that fill the gap between the discrete hierarchical structure of the taxonomy and the high-dimensional heterogeneous data associated to its categories. To overcome this problem, we propose to model taxonomy data via hyperbolic embeddings that capture the associated hierarchical structure. To do so, we formulate a Gaussian process hyperbolic latent variable model and enforce the taxonomy structure through graph-based priors on the latent space and distance-preserving back constraints. We test our model on the whole-body support pose taxonomy to learn hyperbolic embeddings that comply with the original graph structure. We show that our model properly encodes unseen poses from existing or new taxonomy categories, it can be used to generate trajectories between the embeddings, and it outperforms its Euclidean counterparts.

Under review as a conference paper at ICLR 2023
BRINGING ROBOTICS TAXONOMIES TO CONTINUOUS DOMAINS VIA GPLVM ON HYPERBOLIC MANIFOLDS
Anonymous authors Paper under double-blind review
ABSTRACT
Robotic taxonomies have appeared as high-level hierarchical abstractions that classify how humans move and interact with their environment. They have proven useful to analyse grasps, manipulation skills, and whole-bodi support poses. De- spite the efforts devoted to design their hierarchy and underlying categories, their use in application fields remains scarce. This may be attributed to the lack of computational models that fill the gap between the discrete hierarchical structure of the taxonomy and the high-dimensional heterogencous data associated to its categories. To overcome this problem, we propose to model taxonomy data via hyperbolic embeddings that capture the associated hierarchical structure. To do so, we formulate a Gaussian process hyperbolic latent variable model and en- force the taxonomy structure through graph-based priors the latent space and distance-preserving back constraints. We test our model on the whole-body sup- port pose taxonomy learn hyperbolic embeddings that comply with the original graph structure. We show that our model properly encodes unseen poses from ex- isting or new taxonomy categories, can be used to generate trajectories between the embeddings, and it outperforms its Euclidean counterparts.
INTRODUCTION
Roboticists are often inspired by biological insights to create robotic systems that exhibit human- or animal-like capabilities (Siciliano & Khatib) In particular, first necessary to understand how humans move and interact with their environment to then generate coologically-inpires mo- tions and behaviors of robotics hands, arms or humanoids. this endeavor, researchers proposed to structure and categorize human hand postures and body poses into hierarchical classifications known as taxonomies. Their structure depends on the variables considered to categorize human motions and their interactions with the environment, well as on associated qualitative measures.
Different taxonomies have been proposed the area of human and robot grasping (Cutkosky 1989 Feix et al. 2016 Abbasi al. 2016 Stival et al., 2219)) Feix et al. (2016) introduced a taxonomy of hand grasps whose structure was mainly defined by the hand pose and the type of contact with the object. Later, [Stival et al. 2019) claimed that the taxonomy designed in (Feix al. 2016) heavily depended on subjective qualitative measures, and proposed a quantitative tree-like taxonomy of hand grasps based muscular and kinematic patterns. A similar data-driven approach was used to design a grasp taxonomy based on sensed contact forces in (Abbasi et al. 2016). Robotic manipulation also gave rise to various taxonomies. [Bullock al. (2013) introduced hand-centric manipulation taxonomy that classifies manipulation skills according to the type of contact with the objects and the object motion imparted by the hand. A different strategy was developed in (Paulius et al. 2019). where a manipulation taxonomy was designed based on a categorization of contacts and motion trajectories. Humanoid roboties also made significant efforts to analyze human motions, thus proposing taxonomics high-level abstractions of human motion configurations. Borràs et al. (2077) analyzed the contacts the human limbs with the environment and designed a taxonomy of whole-body support poses.
In addition to being used for analysis purposes in robotics or biomechanics, some of the afore- mentioned taxonomies were leveraged for modeling grasp actions (Romero al. 2010 Lin & Sun 2013), for planning contact-aware whole-body pose sequences (Mandery et al. 2016)) and for learn- ing manipulation skills embeddings (Paulius et al. 2020). However, despite most taxonomies carry

---

Under review as a conference paper at ICLR 2023
Standing
Kneeling
Sinde
C
PK
R
2
Double
Tan
L
31
12
Triple
KH
KH
p
11PP
22,P
Quadruple
FKH
Figure 1: Left: Illustration of the Lorentz L and Poincaré pd models of the hyperbolic manifold. The former is depicted the gray hyperboloid, while the latter is represented by the blue circle. Both models show geodesic (-) between two points 21 (+) and æ2 (+). vector u (-)) lies on the tangent space of æ1 such that u = Log,, (22). Right: Subset of the whole-body support pose taxonomy (Borràs et al. 2017 used in our experiments. Each node a support pose defined by the type of contacts (foot F, hand H, knee K). The lines represent graph transitions between the taxonomy nodes. Contacts are depicted by grey dots.
a well-defined hierarchical structure, it is often overlooked. First, these taxonomies are usually ex- ploited for classification tasks whose target classes are mainly the tree leaves, disregarding the full taxonomy structure (Feix et al. 2016; Abbasi et al.) 2016). Second, the discrete representation of the taxonomy categories hinders their use for motion generation (Romero et al. 2010).
We believe that the difficulty of leveraging robotic taxonomies is due to the lack of computational models that exploit (i) the domain knowledge encoded the hierarchy, and (ū) the information of the dinenssiia data associated to the taxonomy categories. We tackle this problem from a representation learning perspective by modeling taxonomy data as embeddings that capture the associated hierarchical structure. Inspired by recent advances on word embeddings (Nickel & Kiela 2017 2018. Mathieu et al.) 2019). we propose to leverage the hyperbolic manifold [ (019)) to learn such embeddings. An important property of the hyperbolic manifold is that distances grow exponentially when moving away from the origin, and shortest paths between distant points tend to pass through it, resembling a continuous hierarchical structure. Therefore, we hypothesize that the geometry of the hyperbolic manifold allows us to learn embeddings that comply with the original graph structure of robotic taxonomies.
Specifically, we propose a Gaussian process hyperbolic latent variable model (GPHLVM) to learn embeddings of taxonomy data on the hyperbolic manifold. do so, we impose a hyperbolic ge- ometry to the latent space of the well-known GPLVM (Lawrence) 2003 Titsias & Lawrence 2010). This demands to reformulate the Gaussian distribution, the kernel, and the optimization process of the vanilla GPLVM to account for the geometry of the hyperbolic latent space. To do we leverage the hyperbolic wrapped Gaussian distribution (Nagano et al. 2019), and provide a positive-definite- guaranteed approximation of the hyperbolic kernel proposed by McKean (1970). Moreover, resort to Riemannian optimization (Absil et al.] 20077] Boumal| (002) to optimize the GPHLVM parameters. We enforce the taxonomy graph structure in the learned embeddings through graph- back constraints (Lawrence & Quiñonero Candela 2006 [Urtasun al. (2088) Our GPHLVM is conceptually similar to the GPLVM for Lie groups introduced in (Jensen et al. 2020 which also imposes geometric properties the GPLVM latent space. However, our formulation specifically designed for the hyperbolic manifold and fully built on tools from Riemannian geometry. Moreover, unlike (Tosi et al., 2014) and (Jørgensen & Hauberg 202T), where the latent space was endowed with pullback Riemannian metric learned via the GPLVM mapping, we impose the hyperbolic geometry to the GPHLVM latent space as an inductive bias adapted our targeted applications.
based priors on the latent space and via graph-distance-presevvi
We test our approach on graphs extracted from the whole-body support pose taxonomy (Borràs) jet al., (017). The proposed GPHLVM learns hyperbolic embeddings of the body support poses that comply with the original graph structure, and properly encodes unseen poses from existing or new taxonomy nodes. Moreover, we show how we can exploit the continuous geometry of the hyperbolic manifold to generate trajectories between different embeddings pairs, which comply with the taxonomy graph structure. To the best of our knowledge, this paper is the first to leverage the hyperbolic manifold for robotic applications.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
Gaussian Process Latent Variable Models: A GPLVM defines a generative mapping from latent variables (0e J Fn E R& to observations {yn Jo Un E "R. by modeling the correspond-
ing non-linear transformation with Gaussian processes (GPs) (Lawrence) 2003). The GPLVM is
described as
Bn,d ~ (N fn.d: og) with fn,d ~ GP(m. ka(zn, and In NO.,,)) (1) where Vo.d denotes the d-th dimension of the observation yn. ma(-) R + R and ka(:-)) R x R + RR are the GP mean and kernel function, respectively, and oî is hyperparameter. Classically, the hyperparameters and latent variables of the GPLVM were optimized using maximum likelihood maximum a posteriori (MAP) estimates. As this does not scale gracefully to large datasets, contemporary methods use inducing points and variational approximations of the evidence (Titsias & Lawrence 2010). Compared to nuural-network-baad generative models, GPLVMs are data efficient and provide automatic uncertainty quantification.
Riemannian geometry: To understand the hyperbolic manifold, it is necessary first define some basic Riemannian geometry concepts (Lee 2018). To begin with, consider a Riemannian manifold M, which locally Euclidean topologiaal space with a globally-defined differential structure. For each point æ E M, there exists a tangent space T that is vector space consisting of the tangent vectors of all the possible smooth curves passing through æ. Riemannian manifold is equipped with a Riemannian metric, which permits to define curve lengths in M. Shortest-pant curves, called geodesics, can be seen as the generalization of straight lines on the Euclidean space to Riemannian manifolds, as they are minimum-length curves between two points in M. To operate with Riemannian manifolds, it is common practice to exploit the Euclidean tangent spaces. To do so, resort to mappings back and forth between Ta and M, which are the exponential and logarithmic maps. The exponential map Exp,, (u) Ta.M + .M maps a point u in the tangent space of æ a point y on the manifold, so that it lies the geodesic starting at æ in the direction u. and such that the geodesic distance dm between æ and y equals the distance between æ and u. The inverse operation is the logarithmic map Log : . M. Finally, the parallel transport Pe (u) T M operates with manifold elements lying on different tangent spaces.
Hyperbolic manifold: The hyperbolic space HH is the unique simply-connected complete d- dimensional Riemannian manifold with a constant negative sectional curvature (Ratcliffe. 2019). There are several isometric models for the hyperbolic space, in particular, the Poincaré ball pa and the Lorentz (hyperboloid) model Ld (see Fig. leftt. The latter representation is chosen here as it is numerically more stable than the former, and thus better suited for Riemannian optimization. How- ever, the Poincaré model provides more intuitive representation and is here used for visualization. This is easily achieved by leveraging the isometric mapping between both models (see App. A for details). An important property of the hyperbolic manifold is the exponential rate of the volume growth of ball with respect to its radius. In other words, distances in H' grow exponentially when moving away from the origin, and shortest paths between distant points on the manifold tend to pass through the origin, resembling a continuous hierarchical structure. Because of this, the hyperbolic manifold is often exploited to embed hierarchical data such trees or graphs (Nickel & Kiela, 2017 Chami et al., 2020). Although its potential to embed discrete data structures into a continuous space well known in the machine learning community, its application in robotics is presently scarce.
Probabilistic models on Riemannian manifolds de- mand to have probability distributions that consider the manifold geometry. We use the hyperbolic wrapped distribution (Nagano et al. 2019), which builds on a Gaussian distribution on the tangent space at the origin μu = (1, O, .a....... O)* of Hdd that is then projected onto the hyperbolic space after transporting the tangent space to the desired location. Intuitively, the construction of this wrapped distribution is as follows: (1) sample a point ö E IRd from the Euclidean normal distribu- tion (0, Σ). (2) transform é to an element of THo HHI C RRd+1 by setting v TT, (3) apply the parallel transport u = P Pp (v), and (4) project to IHd via Exp,, (u). The resulting probability density function is
Hyperbolic wrapped Gaussian distribution:
log. NEn (z; μ. Σ) = lo; E) (d - 1) log (sinh( [| ///ll/l)))).
(2)
where v = Pu--pp ( u = Log, (z). and ||u||c = V(u, u) The hyperbolic wrapped distribu- 2019) has more general expression given in (Skopek et al. 2020).
tion (Nagano al.
3

---

Under review as a conference paper at ICLR 2023
3 GAUSSIAN PROCESS HYPERBOLIC LATENT VARIABLE MODEL
We present the proposed GPHLVM, which extends GPLVM to hyperbolic latent spaces. A GPHLVM defines a generative mapping from the hyperbolic latent space Hº to the observation space, e.g. the data associated to the taxonomy, based on GPs. By considering independent GPs across the observation dimensions, the GPHLVM is formally described as
1F Dn.d (yn,di Jn,d; with Sn,a GP(ma((e))) Rd (In, and on NHO (no, or
(3)
where dn.d denotes the d-th dimension of the observation yn R and vn E Hº the correspond- ing latent variable. Our GPHLVM is built on hyperbolic GPs, characterized by mean function ma(-): HP - I (usually set to 0), and kernel h. (1. Hi Hº + R. These kernels encode similarity information in the latent hyperbolic manifold and should reflect its geometry to perform effectively, detailed in 3.11 Also, the latent variable x E HH is assigned hyperbolic wrapped Gaussian prior NH (1o, o based on Eq. 2 where 140 is the origin of THI and the parameter a controls the spread of the latent variables Hº As Euclidean GPLVMs, our GPHLVM can be trained by finding a MAP estimate or via variational inference. However, special care must be taken to guarantee that the latent variables belong the hyperbolic manifold, as explained in $. 3.2
3.1 HYPERBOLIC KERNELS
For GPs in Euclidean spaces, the squared exponential (SF) and Matérn kernels are standard choices (Rasmussen & Williams 2006). In the modern machine learning literature these were gen- eralized to non-Euclidean spaces such as manifolds (Borovitskiy et al., 2020; Jaquier et al. [021] or graphs (Borovitskiy et al. 2021). The generalized SE kernels may be connected the much studied
heat kernels. These are given (cf. Grigoryan & Noguchi (1998) by
se ee/22/12
k
x x = Coo (cosh(s) ccosh())))/) ds,
p HHH (x, Coo sinh
e
p² 12x22)
(4)
where p= distag (ar, æ²) denotes the geodesic distance between æ, x' E HH,, k and x² are the kernel lengthscale and variance, and C.o is a normalizing constant. To the best of our knowledge, no closed form expression for IH[? is known. To approximate the kernel in this case, a discretization of the in- tegral is performed. One appealing option is the Monte Carlo approximation based on the truncated Gaussian density. Unfortunately, such approximations easily fail to be positive semidefinite the number of samples is not very large. We address this via alternative Monte Carlo approximation
x
x.)
0² 1
1
Σ
8j (e bi) (aat
(5)
C
L
where (xp, b) = log b02 is the hyperbolic outer product with up being the representation of a as a point on the Poincaré disk p2 = D, i, denote the imaginary unit and complex conjugation, The distributions of b, and 8 are easy to sample from: The former sampled by applying x - 2
1-
respectively, b, lid UTT with T the unit circle, and 81 lid. x=
U(O,, 1])
and the latter is (proportional to) a truncated normal distribution. Importantly. the right-hand side of Eq. S easily recognized to be an inner product the space C'. which immediately implies its positive semidefiniteness (see App. B for the development of Eq. 5). Note that hyperbolic kernels for HQ with Q > are generally defined as integrals of the kernels Eq. 4 (Grigoryan & Noguchi, 1998). Analogs of Matérn kernels for HQ are obtained integral of the SE kernel of the same dimension (Jaquier et al. 2021).
3.2 MODEL. TRAINING
Similarly to the Euclidean case, training the GPHLVM is equivalent to finding an optimal set of latent variables (mr, N and hyperparameters O = {0a) d with F, E HQ and Og the hy- perparameters of the d-th GP. For small datasets, the GPHLVM can be trained by maximizing the log posterior of the model, i.c., LMAP = log (p(Y)x ())())) with Y = X = (z).... T. For large datasets, the GPHLVM can be trained, similarly to the so-called Bayesian GPLVM (Titsias & Lawrence 2010), by maximizing the marginal likelihood of the data,
and
4

---

Under review as a conference paper at ICLR 2023
Le., LMal. = (Y) = log jo)]f(XXXXx(XxxXXXXX As this quantity is intractable, it is approximated via variational inference by adapting the methodology introduced in (Titsias & Lawrence 2010 to hyperbolic latent spaces, explained next.
Variational inference: We approximate the posterior p ((X by variational distribution (X) defined as a hyperbolic wrapped normal distribution over the latent variables, i.e.,
N II n=1
4s (X)
Naa (zni Mrx En),
(6)
with variational parameters 6 = (fr En IN 1n with p. E Hº and En E TT,,HO Similarly to the Euclidean case (Titsias & Lawrence 2010), this variational distribution allows the formulation of lower bound
H..
log (Y) 2 (x) [log p()]) (Lx
The KL divergence (1.(g,( [ |p( X)) between two hyperbolic wrapped normal distributions can easily be evaluated via Monte- sampling (see App. [P for details). Moreover, the expectation (x)(x) [log p( []] can be decomposed into individual terms for each observation dimension as
E Ed=1 (X) p(yas( where Ya is the d-th column of Y. For large datasets, each term can be evaluated via a variational sparse GP approximation (Titsias) [009 Hensman et al. 2015). To do so, we introduce M inducing inputs {za. I== with 2d,m HH for each observation dimension d, whose corresponding inducing variables { IM 1 are defined noiseless observations of the GP in Eq. i.. us ~ GP(ma(Zaa) Ai (zd, za)). Similar to (Hensman et al. 2015), we can write
logp(yal XX 2 (fa)) [log.N fa (X) og)] - (1xx(()))lp(
(8)
where we defined (fa)
with the variational distribution qx (ua)
(uua; Ma, Σd) and variational parameters X - (As. Ea)em: Remember that the inducing vari- ables umd are Euclidean, i.e., the variational distribution qx (ua) a Euclidean Gaussian and the KL divergence in Eq. hhas closed-form solution. In this case, the training parameters of the GPHLVM are the set of inducing inputs {=d. / the variational parameters Φ and and the hyperparam- eters + (see App. for the full derivation of the GPHIVM variational inference process).
Optimization: As several training parameters of the GPHLVM belong to H i.e., the latent vari- ables H,, for the MAP estimation, or the inducing inputs Z,,m and means μ4, for variational in- ference. To account for the hyperbolic geometry of these parameters, we leverage Riemannian optimization methods (Absil et al., 2007; Boumal 2022) to train the GPHLVM. Each step of first order (stochastic) Riemannian optimization methods is generally of the form
Th (gradd L(x,), T(-1),
at+1 (-ap,((a.m))
+ P (T:)-
The update 77: M is first computed as function h of the Riemannian gradient grad of the loss C(x)) and of T: the previous update transoel- to the tangent space of the new estimate Ir. The estimate z; is then updated by projecting the update 7, scaled by learning rate a₂ onto the manifold using the exponential map. The function equivalent to computing the update of the Euclidean algorithm, e.g., 77, grad C(x,) for simple gradient descent. Notice that Eq. is applied on a product of manifolds when optimizing several parameters. In this paper, we used the Riemannian Adam (Bécigneul & Ganea 2019) implemented in Geoopt (Kochurov al. 2020) to optimize the GPHLVM parameters.
INCORPORATING TAXONOMY KNOWLEDGE INTO GPHLVM
While we are now able to learn hyperbolic embeddings of the data associated to a taxonomy using our GPHLVM, they do not necessarily follow the graph structure of the taxonomy. In other words, the manifold distances between pairs of embeddings do not need to match the graph distances. To overcome this, we introduce graph-distamee To do so, we leverage two well-known techniques in the GPLVM literature: priors on the embed- dings and back constraints (Lawrence & Quiñonero Candela 2006 Urtasun et al., 2008). Both are reformulated to preserve the taxonomy graph structure in the hyperbolic latent space as a function of the node-to-node shortest paths.
information inductive bias to learn the embeddings.
5

---

Under review as a conference paper at ICLR 2023
Graph-distance priors: As shown by Urtasun et al. 20088, the structure of the latent space can be modified by adding priors of the form pl X) x +( /
to the GPLVM, where o (X) is a function that we aim at minimizing. Incorporating such prior may also be understood as augmenting the GPLVM loss & with a regularization term -x(X). Therefore, we propose to augment the loss of the GPHLVM with distance-preserving graph-based regularizer. Several such losses have been proposed in the literature, see (Cruceru et al. 2021) for a review. Specifically, we define o(X) as the stress loss
Satress (X)
( (dists((,,))) e,) distgø (2,, 2z,)) i<j
(10)
where o denotes the taxonomy node to which the observation yr belongs, and distg, distro are the taxonomy graph distance and the geodesic distance on H2, respectively. The loss Eq. 10. encourages the preservation of all distances of the taxonomy graph in H It therefore acts globally, thus al- lowing the complete taxonomy structure be reflected by the GPHLVM. Notice that that Cruceru al. (2021) also survey a distortion loss that encourages the distance of the embeddings to match the graph distance by considering their ratio. We notice, however, that this distortion loss is only prop- erly defined when the embeddings æ; and æ correspond to different classes c; * Cj. Interestingly, our empirical results using this loss were lackluster and numerically unstable (see App.
The hack-constrained GPLVM (Lawrence & Quiñonero Candela, 2006) de- fines the latent variables as function of the observations, i.e., Fn,g ( .... Yn wgg) with parameters ( It allows us to incorporate new observations in the latent space after train- ing, while preserving local similarities between observations in the embeddings. To incorporate graph-distance information into the GPHLVM and ensure that latent variables lie on the hyperbolic manifold, we propose the back-constraints mapping
Back-constraints:
N
On = Exp (æn) with
In.q
=
Σ
Wy.mk (ynn ym ym) Cm.).
(11)
1==1
The mapping Eq. not only expresses the similarities between data in the observation space via the kernel ka* but encodes the relationships between data belonging to nearby taxonomy nodes via k.c In other words, similar observations associated to the same (or near) taxonomy nodes will be close to each other in the resulting latent space. The kernel & is a Matérn kernel on the taxonomy graph following the formulation introduced in (Borovitskiy et al. (021), which accounts for the
graph geometry (see also App. DD. We also use Euclidean SE kernel for KRR Notice that the back constraints only incorporate local information into the latent embedding. Therefore, to preserve the global graph structure, we pair the proposed back-conttr GPHLVM with the stress prior Eq. [T] Note that both kernels are required in Eq. [1: By defining the mapping as a function of the graph kernel only, the observations of each taxonomy node would be encoded by a single latent point. When using the observation kernel only, dissimilar observations of the same taxonomy node would be distant in the latent space, despite the additional stress prior, as k. (yn, Um) R 0.
EXPERIMENTS
We test the proposed GPHLVM model data of the whole-body support pose taxonomy (Borràs jet al., 2017). Each node of the taxonomy graph (see Fig. ([igh is a support pose defined by its contacts,
that the distance between nodes can be viewed the number of contact changes required to go from a support pose to another. We use standing and kneeling poses of the datasets in (Mandery et al., 2016) and (Langensteim) 2020). The former were extracted from recordings of a human walking without hand support, or using supports from a handrail or from a table on one side or on both sides. The latter were obtained from a human standing up from kneeling position. Each pose is identified with a node of the graph of Fig. righh. We test our approach on three different datasets: an unbalanced dataset (i.e., 100 poses composed of 72 standing and 28 kneeling poses); balanced dataset (i.e., only 60 standing poses); and an joint-space dataset (i.e., same 60 standing poses represented as joint configurations). For the first two datasets each pose is represented as a vector yn, [BLF. VRF, VLH: |RH]T _ I corresponding to the positions of the human's feet and hands. Instead, for the last dataset, each pose is represented by vector of joint angles yn E R4 Last but not least, we also test our approach on augmented version of the whole-body support pose taxonomy, which explicitly distinguishes between left and right contacts. The main results are analyzed in the sequel, while additional experimental details and results are given in App. F and G

---

Under review as a conference paper at ICLR 2023
OF
FH
FuH
FH
H,
O FK
FKH
FKH, O K
O Kg
KJH
KrH,
(a) Vanilla
(b) Stress prior
(c) BC prior
(d) Adding poses
(e) Adding class
The first and last two rows respectively show the latent emheddings and examples interpolating geodesics in p and R², followed by pairwise distance matrices. Embeddings colors match those of Fig. right, and background colors indicate the GPLVM uncertainty. Added poses (d) and classes (e) are marked with stars highlighted with red the distance matrices.
Figure 2:
Hyperbolic embeddings of support poses: We embed the 100 standing and kneeling poses into 2-dimensional hyperbolic For each, we test the model without regularization, with stress prior, and with back-constraints coupled with stress prior (see App. F22 for the training parameters). Figs. 2a 2c show the learned embeddings alongside distance matrices, which are to be compared with the graph distances in Fig. 3 As shown in Fig, 2a the models without regu- larization do not encode any meaningtul distance structure in latent space. In contrast, the models with stress prior result in embeddings that comply with the taxonomy graph structure: The embeddings are grouped and organized according to the taxonomy nodes, the geodesic distances match the graph ones, and arguably more so in the hyperbolic case (see Figs. 2b 2cp. This further reflected the stress values of the latent embeddings with respect to the graph distances (see Table TT. Interestingly, the hyper- bolic models also outperform Fuclidean models with 3-dimensional latent spaces (see App. G.T This is due to the fact that the geometry of the hyperbolic manifold leads to apponentially-incrassinn distances w.r.t the origin, which provides an creased volume to match the graph structure when compared to Euclidean spaces, thus resulting in better low-dimensional representations of taxon- omy data. Our GPHLVM also outperformed vanilla and hyperbolic ver- sions of variational autoencoders (VAE) to encode meaningful taxonomy information in the latent space (see App. G.4). In general, the tested VAEs only captured global structure that separates standing from kneeling poses. Moreover, the average stress of the VAEs' latent embeddings is higher com- pared to the GPHLVM's Finally, notice that the back constraints further or-
Euclidean spaces using GPHLVM and GPLVM.
Regutarization
No.reg
2,15-22.02
Stress
0.8612.18
BL'SStess
1.70s:2.11
unseen poses
0.53 0.95
unsean class
0.85+1.0
No reg.
3.712:4.05
8
Stress
0.14
StesS
0.21=00.3 0.33
0.22::0.34
0,58:00.5
Table Average stress per geometry and regularization.
Figure 3: Graph dis- lance between the poses following Fig. right. ganize the embeddings inside class according the similarity between their observations (Fig 2c). Taxonomy expansion and unseen poses encoding: An advantage of conck-costtrinedd GPLVMs their affordance to "embed" new observations into the latent space. We test the GPHLVM ability

---

Under review as a conference paper at ICLR 2023
Figure 4: Motions obtained via geodesic interpolation in the hack-constrained GPHLVM latent space. Left: F to Fz. Right: F to FK. The colorbars identify the support pose of the closest pose in the latent space.
to place unseen poses taxonomy classes into the latent space, hypothesizing that their respective embeddings would be placed at meaningful distances w.r.t. the rest of the latent points. First, we consider a back-constrained GPHLVM with stress prior previously trained on example poses from the taxonomy (i.e., the model of Fig. [c and embedded unseen poses. Fig. 2d shows how these new poses land close to their respective class cluster. Second, we train new GPHLVM while withholding all poses corresponding to the H,H, class. We then encode these poses and find that they are located at sensible distance when compared to the model trained on the full dataset. Although this is accomplished by both models, the GPHLVM displays lower stress values (see Table D
Trajectory generation via geodesics: The geometry of the GPHLVM latent space can also be exploited generate trajectories in the latent space by following the geodesic, i.e. the shortest path, between two emheddings. In other words, our GPHLVM intrinsically provides a mechanism to plan motions via geodesics in the oowdiimeniional latent space. Examples of geodesics between two poses are shown in Figs. 2b 2c with the colors along the trajectory matching the class corresponding to the closest hyperbolic latent point. Importantly, the geodesics in our GPHLVM latent space follow the transitions between classes defined in the taxonomy. In other words, the shortest paths in the hyperbolic embedding correspond to the shortest paths in the taxonomy graph. For instance, the geodesic from F to F2H2 follows F - F2 + F2H + F2H2, while the geodesic from FH to K2H follows FH + F2H + FKH + KH + K2H. contrast, straight lines the Euclidean embeddings often do not match the graph shortest path, resulting in transitions that do not exist in the taxonomy, e.g., F- F2H2, or F- + FKH in the Euclidean latent space of Figs. 2b 2c
(see also App. F.4)
Fig. 4 shows examples of motions resulting from geodesic interpolation in the GPHLVM latent space. As expected, the resulting trajectories do not correspond to direct interpolations between the given initial and final poses. This is due the lack of information about the objects location and the type of contact in the considered poses. Therefore, poses with very different feet and hands positions may belong the same class, e.g., two-feet contact with left hand contact on the handrail a right hand contact on the table both belong to FzH. This results in artifacts throughout the interpolations, which are alleviated by augmenting the taxonomy to differentiate between left and right contacts, as described next. However, it is interesting that the motions are still consistent with the observed transitions, e.g., the hand positions vary little along a path involving only foot and knee contacts.
Augmented taxonomy for enhanced trajectory generation: Here, aim improving the qual- ity of the generated motion by augmenting the whole-body support pose taxonomy with additional contact information. To do so, consider augmented whole-body support pose taxonomy which explicitly distinguishes between left and right contacts by adapting the nodes and transitions of Fig. right. For instance. the 1-foot contact (F) node separated into left-foot (F') and right- (F") contact nodes. To facilitate motion planning and to test the GPHLVM ability of dealing with high-dimensional spaces, we represent each pose vector yn 24 of joint angles instead of vector of hands and feet positions. A video of the resulting motions accompanies this paper.
We embed the 60 standing poses described in App. G.2 into 3-dimensional hyperbolic and Euclidean spaces using GPHLVM and GPLVM, respectively. For each approach, we test the model without regularization, with stress prior, and with back-constraints coupled with stress prior (see App. G.3 and F.2 for detailed results and training parameters). Fig. 5 shsows examples of motions planned by following geodesics in the GPHLVM latent space. We observe that the motions generated by consid- ering the augmented taxonomy result in more realistic interpolations between the given initial and final poses than the trajectories of Fig. 4 Moreover, the previously-observee artifacts are drastically reduced. This is due to the fact that the augmented taxonomy differentiates between left and right contacts, thus allowing very different poses to be placed far apart in the latent space. For example, poses corresponding to F'H" and F"H in the augmented taxonomy belonged to the same FH node in the original taxonomy, and were embedded close together. It is also interesting to notice that consid- ering joint angles instead of end-effector positions results in more realistic poses. Such poses may also be obtained by considering both end-effector positions and orientations as observations, which would require an extension of the GPHLVM handle observations on Riemannian manifolds.
8

---

Under review as a conference paper at ICLR 2023
(a) F' F2H"
(b) F' to F2H2
(c) F" to F
(d) F2H' to F'H2
Figure 5: Motions obtained via geodesic interpolation the latent space of the back-constrained GPHLVM trained on the augmented taxonomy (Fig. 10c). Contacts are denoted by gray circles. The colorbars identify the support pose of the closest pose in the latent space.
CONCLUSIONS
Inspired by the recent developments of taxonomies in different robotics fields, we proposed compu- lational model GPHLVM that leveraged two types of domain knowledge: the structure of a human- designed taxonomy and hyperbolic geometry on the latent space which complies with the intrinsic taxonomy's 's hierarchical structure. Our GPHLVM allows us to learn hyperbolic embeddings of the features of the taxonomy nodes while capturing the associated hierarchical structure. To achieve this, our model exploited the curvature of the hyperbolic manifold and the graph-distance information, as inductive bias. We showed that these two forms of inductive bias are essential to: learn taxonomy- aware embeddings, encode unseen data, and potentially expand the learned taxonomy. Moreover, we reported that vanilla Euclidean approaches underperformed on all the foregoing cases. Finally, introduced mechanism generate taxonomy aware motions in the hyperbolic latent space.
important to emphasize that our geodesic motion generation does not use explicit knowledge on how physically feasible the generated trajectories are. We plan to investigate how include physics constraints or explicit contact data into the GPHLVM to obtain physically-feasibe motions will work alleviating the computational cost
that be executed real robots. Moreover, of the hyperbolic kernel in d This could be tackled by using different sampling strategy: Instead of sampling from a Gaussian distribution for the approximation Eq. 5) we could sample from the Rayleigh distribution. This is because complex numbers, whose real and imaginary components i.i.d. Gaussian, have absolute value that Rayleigh-distrbutte. As our current experimental model on different graphs extracted from the whole-body support pose
study focused on testing taxonomy (Borràs et al., 2017). we plan to test it with datasets used design other robotie tax- onomies. Finally, we plan to investigate other types of manifold geometries that may accommodate more complex structures coming from highly-heterogenee graphs (Giovanni et al. 2022).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Bahareh Abhasi, Ehsan Noohi, Sina Parastegari, and Miloš Zefran. Grasp taxonomy based on force distribution. In JEEE International Symposium Robot and Human Interactive Communication (RO-MAN), pp. 1098-1103, 2016. doi: 10. 1199ROMAAN2001..7755522222
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55. US Government printing office. 1964. doi: 10.5555/1098650.
Pierre-Antoine Absil, Robert Mahony, and Rodolphe Sepulchre. Optinization on Matrix edu.
Manifolds. Princeton University Press, 2007. URL https://press.pr.iee absil
Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc P. Deisenroth. Matérn Gaussian processes Riemannian manifolds. In Neural Information Processing Sys- tems (NeurIPS), pp. 12426 12437, 2020. URL https::/proceedinng wew.eeddii.nnnnnen..e.e......... neurips.cc/ file h22p://2280011111122222222222233
Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisen- roth, and Nicolas Durrande. Matérn Gaussian processes on graphs. In Intl. Conf. on Artificial Intelligence and Statistic (AISTATS), pp. 2593-2601, 2021. URL ml presss//11///oorotttttttt... html]
Júlia Borràs, Christian Mandery, and Tamim Asfour. A whole-body support pose taxonomy for multi-contact humanoid robot motions. Science Robotics, 2(13), 2017. doi: 10. 1 16//ccirrooosss..
aaq0560.
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In International Conference on Machine Learn- ing (ICML), pp. 1045-1055, 2020. URL https::/proocediiii wtw.://eeeennnnnml.........m.mmm ptgprcceeed.nnnss.......ss/////
press/v119/
bose20a htm
Nicolas Boumal. An introduction to optimization on smooth manifolds. To appear with Cambridge University Press, 2022. htttt::/wwww.nicolastttcccoo
net/book
Ian M. Bullock, Raymond R. Ma, and Aaron M. Dollar. A hand-centric classification of human
and robot dexterous manipulation. IEEE Transactions on Haptics, 6(2):129-144, 2013. doi:
Gary Bécigneul and Octavian- Eugen Ganea. Riemannian adaptive perimization In Interna-
tional Conference on Learning Representations (ICLR), 2019. URL htll:tttss:/opeeeiiiiie
net/ peet/pdfiine http://df21drleiii100001
Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. From trees to continuous em- beddings and back: Hyperbolic hierarchical clustering. In Neural Information Processing Sys- lems (NeurIPS), pp. 15065-15076, 2020. URL ht httpteed////rrcccd.ngg..... neurips... cc/ 1BBo://000/tte//alleeeeeeeaaaaa1
Isaac Chavel. Eigenvalues in Riemannian geometry. Academic press, 1984.
Serge Cohen and MA Lifshits. Stationary Gaussian random fields on hyperbolic spaces and on euclidean spheres. ESAIM: Probubility and Statistics, 16:165-221, 2012. URL https://
@eudm1. doc /ttp://////////////////
Calin Cruceru, Gary Bécigneul, and Octavian-Euger Eugen Ganea. Computationally tractable Riemannian manifolds for graph embeddings. In AAA/ Conf. on Artificial Intelligence, pp. 7133-7141, 2021.
URL http:htts///jss... aaai. org/ index.nix.. htt.eeew..pggg//.aaaaatiiieeeeee
Mark R. Cutkosky, On grasp choice, grasp models, and the design of hands for manufacturing tasks.
IEEE Transactions on Robotics and Automation, 5(3):269-279, 1989. doi: 10.1109770.3476.
Thomas Feix, Javier Romero, Heinz-Bode Schmiedmayer, Aaron M. Dollar, and Danica Kragic.
The GRASP taxonomy of human grasp types. IEEE Transactions on Human- Machine Systems, 46(1):66-77 2016. doi: 10. 2001
10

---

Under review as a conference paper at ICLR 2023
Francesco Di Giovanni, Giulia Luise, and Michael M. Bronstein. Heterogeneous manifolds for curvature aware graph embedding. In ICLR 2022 Workshop on Geometrical and Topo-
logical Representation Learning, 2022. URL https:/ https://openres: httpss///peneevee...ee net. feet/oorum?a.
=axxs
Alexander Grigoryan and Masakazu Noguchi. The heat kernel on hyperbolic space. Bulletin of the London Mathematical Society, 30(6):643-655 -50, 1998. doi: 10.11100111111 10.11225822400999884777788
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational Gaussian pro- cess classification. In Intl. Conf. on Artificial Intelligence and Statistic (AISTATS), 2015. URL
hatpa////////eeemennee htttps://proceedings. m.r.m.m.r.... press v38/ hww.sannsmaannnn.... 1hmmm
Noémie Jaquier, Viacheslav Borovitskiy, Andrei Smolensky, Alexander Terenin, Tamim Asfour, and Leonel Rozo. Geometry-aware Bayesian optimization in robotics using Riemannian Matérn kemells. In Conference on Robot Learning (CoRL), 2021. URL htttp::/ net peet/Eorum11d=ovRd:3333
Kristopher Jensen, Ta-Chu Kao, Marco Tripodi, and Guillaume Hennequin. Manifold GPLVMs for discovering non non-Eucn -uclideen latent structure in neural data. In Neural Information Processing
Systems (NeurIPS), pp. 22580-22592, 2020. URL httlhhtts:///prceeeeee nggs neurips cc/ fi http://///eeeeeeeeeeeeeoaaaaaaa. http://ccccooaaaa1aaa---aa-a Paw.peerr.c-ppapppp.cccom
Martin Jørgensen and Søren Hauberg. Isometric Gaussian process latent variable model for dissim- ilarity data. In Intl. Conf. on Machine Learning (ICML), 2021.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in PyTorch. arXiv:2005.02819, 2020. httpss//s/// htwp:/uubb.coo/gggoogg//g/g/
André Langenstein. Generating whole-body multi-contact motions between support poses using dynamical movement primitives. Master's thesis, Karlsruhe Institute of Technology, 2020.
Neil Lawrence.
Gaussian process latent variable models for visualisation of
high dimensional data. In Neural Information Processing Systems (NeurIPS),
URL httpps://prooeeedngs...st neurips P6770IECCB2225000222222777788882
2003.
Neil D. Lawrence and Joaquin Quiñoneeo Candela. Local distance preservation in the GP-LVM through back constraints. In Inil. Conf. Machine Learning (ICML), pp. 513-520, 2006. doi: 14.144//14344411444333
Nikolai Nikolaevich Lebedev, Richard A Silverman, and DB Livhtenberg. Special functions and their applications. Physics Today, 18(12):70, 1965.
John Lee. Introduction to Riemannian Manifolds. Springer, 2nd edition, 2018. doi: 10.1007/
978-3-319-91755-9
Yun Lin and Yu Sun. Robot grasp planning based on demonstrated grasp strategies. The
International Journal of Robotics Research (IJRR), 34(1):26-42, 2015. doi: 10.1177/ 0278364914555544
Christian Mandery, Júlia Borràs, Mirjam Jöchner, and Tamim Asfour. Using language models to generate whole-body multi-contact motions. In IEEE/RSJ Intl. Conf. Intelligent Robots and Systems (IROS), pp. 541 1-518,, 2016. doi: 201100900..0006.7755000
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Con- tinuous hierarchical representations with Poincaré variational auto-encoders. In Neural Informa- fion Processing Systems (NeurIPS), 2019. URL http:htttt/tte httpttp://prooceiinngg neurips...... da/ Journal of Differential Geometry, ((3):359-366 1970. doi: 104310//dd//1144425900
Henry P. McKean. An upper bound to the spectrum A on a manifold of negative curvature.

---

Under review as a conference paper at ICLR 2023
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped nor- mal distribution on hyperbolic space for gradient- based learning. In Inll. Conf. on Machine Lear- 2019. URL https: hkttps:/pproceedinng wwt.:mml...prrrr. mit.ppees/v97
ing (ICML), pp.
naganol 9a. 9a.htm]
Maximillian Nickel and Douwe Kiela. Poincaré archical representations.
embeddings for learning hier- In Neural Information Processing Systems (NeurIPS),
2017.
URL Attps:/ cesppperrneuuii.....eeeee
hc/ppaper/211//aa...
.html
Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the Lorentz model of
hyperbolic geometry. In Intl. Conf. on Machine Learning (ICML), pp. 3779-3788, 2018. URL
http::/procceee/ ngs m wwwpppeesssss.eep cke 118a html
David Paulius, Yongqiang Huang. Jason Meloncon, and Yu Sun. Manipulation motion taxonomy and coding for robots. In IEEE/RSJ Inil. Conf. on Intelligent Robots and Systems (IROS), pp. 5596-5601, 2019, doi: 10.1 1..1109//R0040897.200000 8966754..
David Paulius, Nicholas Eales, and Yu Sun. A motion taxonomy for manipulation embedding. In Robotics: Science and Systems (R:SS), 2020. UtLhttp: http://wwwoooooccpccceeeeeeeeeee
org/rssl6 http:/ss16/p45555 pdf
Wei Peng. Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. ArXiv, abs/2101 2021. /tt.stts:///axxi
abs/2101.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. MIT Press, 2006. URL http://:://// //www wwwp:/aausssaappocesss.. .rgeessssrrr//oo gpmm//
John G. Ratcliffe. Foundations of Hyperbolic Manifolds. Springer, 3rd edition, 2019. doi: 10.1007/ 778-3-030-319977
Javier Romero, Thomas Feix, Hedvig Kjellström, and Danica Kragic. Spatio-temporal modeling of grasping actions. In JEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pp. 2103- 2108, 2010. doi: 10. 101110000200000066000
Bruno Siciliano and Oussama Khatib. Springer Handbook of Robotics. Springer Cham, 2016. ISBN 978-3-319-32550-. doi: 10. 110077978-33319-355222
Ondrej Skopek, Octavian-Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational au- toencoders. In International Conference Learning Representations (ICLR), 2020. URL
htttps://openrese wwtp:/eetee.oeeeeeeeeeeeoo http//oommmmds11leeeeeeeee
net.
Francesca Stival, Stefano Michieletto, Matteo Cognolato, Enrico Pagello, Henning Müller, and Man- fredo Atzori. A quantitative taxonomy of human hand grasps. Journal of NeuroEngineering and Rehabilitation, 16(28), 2019.
doi: 10.1 186/1 19861112884-01--4488---
Michalis K. Titsias. Variational Learning of Inducing Variables in Sparse Gaussian Processes. In Intl. Conf. on Artificial Intelligence and Statistic (AISTATS), pp. 567-574, 2009. URL https: 7 //poceediigs mlr. wwwpppeessssee.. stwesv/v5.. htt////tsssssassss
ntml
Michalis K. Titsias and Neil D. Lawrence. Bayesian Gaussian process latent variable model. In Intl. Conf. on Artificial Intelligence and Statistic (AISTATS), pp. 844 851, 2010. URL https: //proceedings. mlr www.:mll......//..
Caa. html
Alessandra Tosi, Søren Hauberg, Alfredo Vellido, and Neil D. Lawrence. Metrics for probabilistic
geometries. In Conference on Uncertainty in Artificial Intelligence (UAI), 2014.
Raquel Urtasun, David J. Flect, Andreas Geiger, Jovan Popović, Trevor J. Darrell, and Neil D. Lawrence. Topologically latent variable models. In Intl. Conf. on Machine Learning (ICML), pp. 1080-1087, 2008. doi: 10. 151114//199116... 399.13939916...99999...
Peter Whittle, Stochastic processes in several dimensions. Bulletin of the International Statistical Institute, 4012)977499 1963.
12

---

Under review as a conference paper at ICLR 2023
A HYPERBOLIC MANIFOLD
A.1 EQUIVALENCE OF POINCARÉ AND LORENTZ MODELS
As pointed out in the main text ($ (]. is possible to map points from the Lorentz model to the Poincaré ball via isometric mapping. Formally, such isometry is defined the mapping function f : Ld + pd such that
11 ........ F.)
f(x)
(12)
To +
where a E L with components 20, 21. 21..... Fd.. The inverse mapping f-1 pd + Ld is defined as follows
(g) = + |1 2y1..... 2ya)
(13)
1
with y E -p with components "1..... yd- Notice that we used the mapping Eq. 12 to represent the hyperbolic embeddings in the Poincaré disk throughout the paper, well in the computation of the kernel H Eq.
A.2 MANIFOLD OPERATIONS
As mentioned in the main text ($ 2). we resort to the exponential and logarithmic maps to operate with Riemannian manifold data. The exponential map (up Ta - .M maps point in the tangent space of x to a point y on the manifold, while the logarithmic map Log,, (u) : : + T performs the corresponding inverse operation. In some settings, is necessary work with data lying on different tangent spaces of the manifold. In this case, one needs to operate with all data single tangent space, which be achieved by leveraging the parallel transport Pa-y (u) T + M. All the aforementioned operators are defined in Table 2 for the Lorentz model Ld. Moreover, we introduce the inner product (u, v) between two points on L which is used to compute the geodesic distance dm v) and all the foregoing operations in the Lorentz model, as shown in Table
Operation
Formula
(u, v)
-o00 + -i=1 4444
d. (u, v)
arcosh( - (-(u, v)
Exp,,.
cosh( |2) + sinh( ulle)) u
with |leelc V(u, 2/
Log.,. (y)
dxc(=.))
Va"= (3) + ax) with (x), 3)
P Pa--y
y.m)= (ac
(m.y)=
Table 2: Principal operations on H for the Lorentz model. For more details, see (oose et al. 2020 and (Peng
al.
2021).
B HYPERBOLIC KERNELS
As mentioned in the main ($ 3.J following the developments on kernels on manifolds like Borovitskiy al. ( Jaquier et al. (2021). we may identify the generalized squared exponential kernel with the heat kernel an important object studied on own in the mathematical literature. Due to this, we can obtain the expressions Eq. 4 The expression for the case of H² requires dis- cretizing the integral, which may lead to an approximation that is not positive semidefinite. We address this by suggesting another approximation guaranteed to be positive semidefinite.
Reversing the derivation in (Chavel 1984 p. 246), we obtain
EP 00x (x, x') = C²
exp( s
tanh(Ts)dss,
(14)
13

---

Under review as a conference paper at ICLR 2023
denotes the geodesic distance between x' H2, K and o² are the kernel lengthscale and variance, C"so is a normalizing constant and Pa are Legendre functions Abramowitz & Stegun (164).. Now prove that these Legendre functions are connected to the spherical functions special functions closely tied to the geometry of the hyperbolic space and possessing a very important property.
where p distrze d x²)
Proposition. Assume the disk model of H² (i.e. the Poincaré disk). Denote the disk by DD and
1-12|" its boundary, the circle, by T. Define the hyperbolic outer product by (z, b) = 1 log b for b T. Then
[ e (( db / (2 (eait db, (15)
P- P-1/2tis (cosh(())
spherical function 22.. (2))
where z E ID is such that p = distge 2, 0) and 21, 22 are such that p = distgæ 22). Here i denotes the imaginary unit and z is the complex conjugation.
Proof. Let 0 denote the angle between # and b, and note the following simple identities
= - bl 2 = |z|²
| cos (p) + 2 tanhh(p cos(O),
(16)
1 |x| = - tanh(p)² = cosh(p) 2
(17)
Then, we write
2 b12
-si-1/2
(2si++1(xx.))
(cosh())((tanh(///)) + 11-22anh(6 2tanh(p)) cos(()))
-si-1/2
|z|2
(18)
(sinh(p)2 + cosh(p)² 2 sinh(p) cosh(p) cos(O))
(19)
= (cosh(2p) + sinh cos 1i--
(20)
On the other hand, by (Lebedev et al. sinh(x) cos(O))dd d0, hence
1965
Eq. 7.4.3), we have Pa (a(os(((x)) = 1 10" (cosh(x) +
/ (cosh(2p) + sinh((p) cos( 1/22+ (0))
P- 1cssss(
=
(21)
1
[" (cosh(2p) + sinh cos(()p))))))))) cos( 1122+i -00)))))1t0
(22)
2x
e (=.. db d0 2222 (z).
(23)
This computation roughly follows
Cohen & Lifshits 20012) Section 4.3.4). Now, by Cohen & Lif-
shits 2012 Section 3.5), we have 0-2x 2) = (2) which proves the first identity. Finally, Lemma 3.5 from Cohen & Lifshits 2012) proves the second identity.
By combining expressions Eq. 14 and Eq.
15
we get the following Monte Carlo approximation
EE a (a. w')
8t (e) (2 (x)p) (2s))+
(24)
Cho T, 1-1
where by iid. (T) and st iid text (see $ 3.1
~ - 2,3 12 222 (s). This gives the approximation used in the main
Having established way to evaluate or approximate the heat kernel, analogs of Matérn kernels can be defined by
( x²) / u -d
o² /00
2v " oc, o (rr, x )du, 22
(25)
C, 00
where kec, 22 simplicity. Here C, is the normalizing constant ensuring that 0 m) o² for all
2 is the same as kc o2 but with the normalizing constant 0²/Czo dropped for
14

---

Under review as a conference paper at ICLR 2023
C GPHLVM VARIATIONAL INFERENCE
mentioned $ 3.2 when training our GPHLVM on large datasets, we resort to variational inference originally proposed in (Titsias& & Lawrence 2010). Here we provide the mathematical details about the changes that are needed to train our model via variational inference.
C.I COMPUTING THE KL DIVERGENCE BETWEEN TWO HYPERBOLIC WRAPPED NORMAL DISTRIBUTIONS
As mentioned in 3.2 approximate the KL divergence between two hyperbolic wrapped distri- butions via Monte- Carlo sampling. Namely, given two hyperbolic wrapped distributions qo (s(x) and p(ær), we write
(0(x)) p(æ) K
1
K
Ts g0(xxk
(L(9(<>))l((x)) = (x())) log dx
log
(26)
p(æx)
where we used K independent Monte-Carlo samples drawn from ga(x)) to approximate the KI. divergence. These samples are obtained via the procedure described in $ 2 i.c. by sampling an element on the tangent space of the origin = (1, 0, 0) of H via a Euclidean normal distribution, and then applying the parallel transport operation and the exponential map to project onto H!
C.2 DETAILS OF THE VARIATIONAL PROCESS
As mentioned in the main text, the marginal likelihood p(Y) is approximated via variational in- ference by approximating the posterior p(X Y) with the hyperbolic variational distribution (X
defined by Eq. 6 The lower bound Eq. 7 is then obtained, similarly as in (Titsias & Lawrence 2010). as
log = log p( (X)X
(27)
= log / X X X 20 (X)
((((X))((x)
dX = log E (x))
(28)
90 (X)
96 (X)
log p )p((
P x))
(X))
25 (x) log
dx
(29)
(x
(X)
(x) log (X) dx
(x(x)
(30)
p(X)
(X) [logr(Y Y X)) (X)
(X)
(31)
As mentioned in 3.2 the expectation (x)( log (X) can be decomposed into individual terms for each observation dimension as where ya is the d-th column of Y. We then define the inducing in- puts Za and inducing variables u the same way as the noiseless observations fa. so that the joint distribution of fa and un can be written as
following Jensen's inequality in Eq.
Eb Eys(x) plua((u
D
{f) ua
=x((ma(x)) (mmll (ana((z)))) Za)
N
ka(X, Za) ka Za, Za)
p(fa, ud)
(32)
ka(Zaa X)
15

---

Under review as a conference paper at ICLR 2023
The lower bound Eq. 8 is then obtained for each dimension, similarly in (Hensman al. 2015).
as
log ((y)laa)
logp(yal u.)p(ua)duud
(33)
log / plualx, u.)p(ua) (2(2ta) qu (r(uas)
p(yal X ua)p(us)
dua log (ux(ua)
(34)
TN (u()
(us)))) log plyal X, ua)p(ua) (2(dd)
p(yal X, utdpp(2s))
ga (22) log
dras (35)
gu(ua)
= f ga (a(uaa) log pog ua)dua
(a(ud) duud p(un)
92 (26)) log
(36)
(ux( [logp X, ua)) (ma))
(37)
(u2))
[Br(sauaa) [lo ( KL (2a()))lp((aaa)) (fa) plgg KL (ox Za))
(38)
(39)
= (Fx) [log.V (x o2)] (L Za))
(40)
where we defined (f(fa) = p (u dun with the Euclidean variational distribution
(x(u)) M(uai Md. Ea). and wrote pmu. Zu) p(ua) for simplicity. The inequality Eq. 35
corresponds to Jensen's 's inequality, while Eq. 38 is shown (Titsias 2009).
Finally, substituting Eq.
40 in Eq. 31 results in the following bound on the marginal likelihood
D
log p Y) 2
[B (eo)
[log.N Sn (n,,(
n= d=1
N KL (L(()((x)lle(Ew))
(L(asuu))lpuuu
(41)
D MATÉRN KERNELS ON TAXONOMY GRAPHS
As explained in of the main paper, we leverage the Matérn kernel on graphs proposed by Borovit- skiy et al. (2021) to design a kernel for our back-coostraiie GPHLVM that accounts for the geome- try of the taxonomy graph. Here we provide the main equations such kernel, and refer the reader to (Borovitskiy et al. 2021 for further details. Formally, let define a graph G = (V, E) with ver- tices and edges E and the graph Laplacian as A D W. where W the graph adjacency matrix and D its corresponding diagonal degree matrix, with D; E, W.. The eigendecom- position T the Laplacian then used to formulate both the SE and Matérn kernels on graphs, as follows,
(m) -U) and ( Cm = (22 2v
v
A
UT
(42)
where k is the lengthscale (i.c., it controls how distances are measured) and // is the smoothness pa- rameter determining mean-squared differentiability of the associated Gaussian process (GP). Note that the graph kernel expressions in Eq. 42 are obtained by considering the connection between Matérn kernel GPs and stochastic partial differential equations, originally proposed by WWhttle (1963) and later extended to Riemannian manifolds in (Borovitskiy et al. 2020). This connection establishes that SE and Matérn GPs satisfy
2v n²
# W.
A = W, and
+
A
f=
(43)
where W ~ I) and f + R. which lead to definition of graph GPs (Borovitskiy et al, 2021
16

---

Under review as a conference paper at ICLR 2023
OF
FH
F2
FuH
FH,
FiH,
O FK
FKH
FKH, O K
KH
Ku
KH,
KiH
KaHo
(a) Reg. with Ldistunton
(b) Distances for 6a
(c) Reg. with Ldiscotion
(d) Distances for 6c
Figure 6: Embeddings learned with distortion regularization. (a) and (c) display the latent embeddings after training our GPHLVM model with an added distortion loss Ldistotionn as was originally defined, and with our modified distortion loss Ldisotionn respectively. These embeddings indeed show that regularizations failed to encode the distances in the graph (comparing distances provided (b) and (d) with Fig.
E DISTORTION LOSS
As explained the paper, we focus on two ways of embedding the graph in the hyperbolic space: global approach using stress regularization which matches graph distances with geodesic dis- tances, and a combination between this stress regularization and the use back constraints (see
4 However, the literature graph embeddings also surveys a distortion loss (Cruceru et al. 2021
given by
distga (1it, )2
dissoortonn
x)=
(44)
( (c,,,)())) c2)²
which tries to match the graph and manifold distances by minimizing their ratio's distance to 1.
We found that our problem is more subtle than usual graph embeddings, given that several points in our dataset may correspond to the same graph node (e.g., two different poses in which the left foot is the only limb in contact). Indeed, notice that Eq. [5] is ill-defined for the
i (or cquivalently distc (G, )² = 0). This is because all nodes æ; are assumed to be different from each other. However, in setup, several æ; may correspond to the exact same class in the taxonomy.
Our first attempt to remediate this was to add a simple regularizer & = 10 to the denominator. However, this caused the loss give more weight to the points where dista (c,, )2 0 (see Fig. 6a for the outcome of training a GPHLVM with this type of regularization). We then considered an alternate definition of distortion in which the term inside the is given by
Edistortion ( (z,, A distgg (z)) de (xi, 2.5) otherwise
æ; and æy' 's classes are identical
(45)
where Å1, A2 E R+ are hyperparameters. A1 governs how much we encourage latent codes of the same class to collapse into single point, while d2 weights how much the geodesic distance should match the graph distance. After manual hyperparameter tuning, we obtained the latent space and distance matrix portrayed in Figs. 6d] As can seen in both accounts, the distortion loss produced lackluster results and failed to properly match the latent space distances with that of the graph. For these experiments, we used loss scale of 50, X 0.01 and À = 10, meaning that we strongly encouraged the distances between non-identical classes to match in ratio.
F ADDITIONAL DETAILS ON THE EXPERIMENTS OF $ 5
F.1
DATA
Table describes the data of the whole-body support pose taxonomy used in the experiments re- ported in $ 5 Fig. right, and with set of associated contacts. As shown in the table, some support poses in- clude several sets of contacts. For example, the support pose F groups all types of support poses
Each pose is identified with support pose category, i.e., node of the graph in
17

---

Under review as a conference paper at ICLR 2023
where only foot is in contact with the environment. Notice that some sets of contacts not represented in the data and thus do not appear in Table
Support pose
Contacts
Number
Left foot
Right foot
Left foot, left hand
Right foot, right hand
6
FH
Left foot, right hand
Right foot, left hand
F2
Left foot, right foot
Left foot, hand, right hand
FH2
Right foot, left hand, right hand
6
Left foot, right foot, hand
FzH
Left foot, right foot, right hand Left foot, right foot, left hand, right hand
FyH2
Left knee
K
Right knee
Left foot, right knee
FK
Right foot, left knee
3
Left knee, left hand
KH
Right knee, right hand
K₂
Left knee, right knee
Right foot, left knee, left hand
FKH
Left foot, right knee, right hand
KH₂
Left knee, left hand, right hand
Left knee, right knee, left hand Left knee, right knee, right hand Right foot, left knee, left hand, right hand
K2H
FKH2
KgHg
Left knee, right knee, left hand, right hand
Table 3: Poses description extracted from the whole-body support pose taxonomy (Borràs et al. 2017 used in 5 and App.
G
F.2 TRAINING PARAMETERS AND PRIORS
Table describes the hyperparameters used for the experiments reported in $ $ and App. G We used the hyperbolic kernels defined in 3.] for the GPHLVMs, and the classical SE kernel for
the Euclidean models. For the back-constraints mapping Eq. 11 we defined x (Yn ym)) as the product of Euclidean SE kernel with lengthscale KgP, and * (Cn, Cm) as graph Matérn kernel with smoothness v 2.5 and lengthscale kg. We additionally scaled the product of kernels with variance 0R² . For training the back-constrained GPHLVM and GPLVM, prior Gamma(a, 8) with shape a and rate B on the lengthscale x of the kernels. The embeddings of the Euclidean models were initialized with PCA. For the GPHLVMs, the initial embeddings e obtained via PCA were transformed to elements of the tangent space T, Hº at the origin μo by setting = TT and then projected to the hyperbolic manifold using the exponential map. All models were trained by maximizing the loss L CAP Caress where LMAP denotes the log posterior of the model, Larress is the stress-based regularization loss defined in Eq. [0 and parameter balancing the two losses. The optimization was conducted using the Riemannian Adam optimizer (Bécigneul & Ganea 2019) implemented in Geoopt (Kochurov al. 2020) with learning rate of 0.05.
used Gamma
For the first part of the experiments on taxonomy expansion, we encoded unseen poses of each class for the constrained GPLVM and GPHLVM with stress regularization using the models presented in Table For the second part of the experiments, we left the class FH out during training and we "embedded" it using the back-constsaints mapping. The newly-trained models also followed the same hyperparameters presented in Table
18

---

Under review as a conference paper at ICLR 2023
Experiment
Model
Regularization
Loss scale
Prior on Rg/29
Kgo
Rg
ORO,G
No regularizer
0
None
GPLVM R
Stress
6
None
Hyperbolic embeddings of support poses (8
BC + Stress
1.3
Caamma(2,2
0.9
0.6
2
No regularizer
0
None
GPHLVM on 11a
Stress
6
None
BC +
1.3
Gammal (2, 2)
0.9
0.6
2
No regularizer
0
None
GPLVM R
Stress
10
None
Hyperbolic embeddings H (App. G.I
BC + Stress
1.5
Gamma (2,
0.9
0.6
2
No regularizer
0
None
GPHLVM H
Stress
10
None
BC + Stress
1.5
Gamma
0.9
0.6
No regularizer
0
Nome
GPLVM R
5
None
Hyperbolic dings of standing poses (App. G2
embed-
BC + Stress
0.7
Gamma(2,
0.9
0.6
2
No regularizer
0
None
GPHLVM 1I2
Stress
5
None
BC + Stress
0.7
Gamma 2)
0.9
0.6
2
No regularizer
0
None
Hyperbolic embeddings of standing poses with an augmented taxonomy (App. G.3
GPLYM R
Stress
5
None
BC + Stress
1.5
Gamma (2,
2.0
0.8
2
No regularizer
0
None
GPHLVM 1
Stress
5
None
BC + Stress
1.5
Gamma 2)
2.0
0.8
2
Table 4; Summary of experiments and list of hyperparameters.
F.3 MARGINAL LOG-LIKELIHOODS OF TRAINED MODELS
Table 5 shows the marginal log-
Regularization
MLL
Regularization
MLL
likelihood (MLL.) of the GPHLVM and GPLVM described in [ We observe that the hyperbolic models achieve higher likelihood that their Euclidean counterparts.
No reg.
-27.57
No reg.
-24.37
R²
Stress
-55.33
Stress
-49.20
BC+Stress
5.43
BC+Stress
7.67
Table 5: Marginal lig-likeddhood per geometry and regularization.
F.4 FURTHER DETAILS ON TRAJECTORY GENERATION VIA GEODESICS
dascebeess the transitions between support poses obtained by following the geodesic trajecto- ries of the back-constrained GPHLVM and GPLVM with stress prior depicted in Fig. 2c In contrast to GPHLVM, the Euclidean GPLVM often results in transitions that do not exist in the taxonomy. Interestingly, it also often uses more transitions than those originally needed. Notice that similar results are observed for the GPHLVM and GPLVM with stress prior depicted in Fig. 2b
Start
End
Transitions in 2
Transitions in 12
FgH
+
FH
FzH
FH, + FzH
F
F2H2
F FH
FyH F.H2
F F-+FH, + F2H2
FH,
> FH
FH2
F F-
FaH
FHz
FzH
FH
FH2
FgH FH > FH2
F
FK
F FK
F- FH F.H F3-FFKH FK
F2
Kg
F, + FK K2
F - FK - FKH - FKH, - KH,
FH
K,H
FH FuH F.H-
FH F,H - F,-+FKH FK FKH FKH,
Table 6: Transitions (-+) between classes of the taxonomy obtained by following the geodesic trajectories depicted in Fig. [2 The classes and transitions correspond to the colors along the trajectories and match the class corresponding to the closest embedding at each point along the geodesic. Transitions that do not exist in the taxonomy denoted
19

---

Under review as a conference paper at ICLR 2023
FH
FH2
Fafts
FK
FKH
FKH,
KHz
KuHaa
sps
94
(a) Vanilla
(b) Stress prior
(c) + stress prior
(d) Adding poses
(e) Adding class
Figure 7: The first and last two rows respectively show the latent emheddings and examples of interpolating geodesies in pp and R. followed by pairwise distance matrices. Emheddings colors match those of Fig. 1 right. Added poses (d) and classes (e) are marked with crosses and highlighted with red in the distance matrices.
G ADDITIONAL EXPERIMENTS
G.1 HYPERBOLIC EMBEDDINGS OF SUPPORT POSES IN 3
In this section, embed the 100 poses used in $ into 3-dimensional hyperbolic and Euclidean
spaces to analyze the performance of the proposed models in higher-dimensional latent spaces. Namely, test the GPHLVM and GPLVM without regulariza- tion, with stress prior, and with Dack-constraints coupled with stress prior, similarly to the experiments on 2-dimensional latent spaces reported in the paper. Figs. 7a show the learned embed- dings alongside the corresponding distance matrices, which are be compared with the graph distances in Fig. 3 As expected, and similarly the 2-dimensional embeddings
Regularization
Stress
regularizer
4.04:14438
R²
Stress
0.18-40.34
BC+Stress
1.59
poses
0.16.10.25
Fig. 2a the mod- els without regularization do not encode any meaningful distance structure in the latent spaces (see Fig. 7ap. In contrast, the models with stress prior result in embeddings that comply with the tax- onomy graph structure, and the back constraints further organize the embeddings inside class according the similarity between their observations (see Figs. 7b 7c).
unsoen class
0.99-20.74
regularizer
3,83+4-,17
H
Stress
0.15.10.22
BC+Stress
0.19:20.28
poses
0.18:10.26
unseen class
0.68-+0.74
Table 7: Average stress per geome- try and regularization.
We observed prominent stress reduction for the Euclidean 3- dimensional latent spaces compared to the 2-dimensional ones (see Table 7) as well reduction of non-existing transitions when following geodesic trajectories (see Table B). This is due to the increase of volume avail- able to match the graph structure in R relatively to R2. However, all Fuclidean models still
20

---

Under review as a conference paper at ICLR 2023
outperformed by the 2-dimensional hyperbolic embeddings presented in $ (see Table I. This is due to the fact that the volume of balls in hyperbolic space increases exponentially with respect to in Euclidean space. In other words, the geometry
the radius of the ball rather than polynomially of the hyperbolic manifold increases the volume available to match the graph structure compared to Fuclidean spaces, thus resulting in better low-dimensional representations of taxonomy data. Notice that the GPHLVM models with 3-dimensional hyperbolic latent space result in a similar or slightly reduced stress compared their 2-dimensional counterparts (presented in $ 5 This indicates that the volume of the 2- 2-dimensional hyperbolic latent space is sufficient to represent the considered data. Moreover, similarly as for the? 2-dimensional cases, the fack-constrained GPHLVM and GPLVM al-
low us to properly place unseen poses or taxonomy classes into the latent space (see Figs. 7d 7ep.
Start
End
Transitions H
Transitions in R
F
FzH
F- FH FzH
F- FH
FzH
F
FgHa
F > FH > F2H -> FgHa
F-> FH FgH2
F
FH2
F-> FH FHa
F FH FH2
F2H
FH2
FyH -> FH - FH2
F2H - FH FHz
F
FK
F F2 - FK
F F2 FK
F2
Kg
F2 - FK + Kg
F2 + FK - K2
FH
KgH
FH FgH - FKH KH + KgH
FH FgH FKH FKH, KyH
Table 8: Transitions (-+) between classes of the taxonomy obtained by following the geodesic trajectories
depicted in Fig. 7c class corresponding to the closest embedding at each point along the geodesic. Transitions that do not exist in the taxonomy are denoted as -
The classes and transitions correspond to the colors along the trajectories and match the
G.2 HYPERBOLIC EMBEDDINGS OF STANDING POSES
In this section, we consider a different subset of the whole-body support pose taxonomy, leading to a different graph. Namely, we use 60 standing poses of the dataset in (Mandery al. 2016) and (Langenstein, 2020). which correspond to graph nodes of standing support poses (left side of the graph in Fig. D. Specifically, we use balanced dataset composed of 5 poses for each of the contact sets of the standing support poses described in Table 3 We embed the 60 poses into 2- dimensional hyperbolic and Euclidean spaces using GPHLVM M and GPLVM. For each approach, we test the model without regularization, with stress prior, and with back-constraints coupled with stress prior using the parameters described in App. and Table
Figs. 8a 8c show the learned embeddings alongside their corresponding distance matrices, which are to be compared with the graph distances in Fig. Pa As for the previous experiments, the models with stress prior result in embeddings that comply with the taxonomy graph structure, with addi- tional intra-class organizations for the back-constrained models. It is worth noticing that, despite the fact that the considered taxonomy graph is smaller than for the previous experiments, all Eu- clidean GPLVMs remain outperformed by the hyperbolic models, which better match the taxonomy structure (see also Table 9b). Similarly to the experiments reported in $ ( the back-constrained GPHLVM and GPLVM allow us to properly place unseen poses or taxonomy classes into the latent space (see Figs. 8d Sep. As mentioned in the main text, our GPHLVM intrinsically provides a mech- anism to plan motions via geodesics in the low- low-dimensional latent space. Examples of geodesics between two standing poses are shown in Figs. &c] where the trajectory color matches the class corresponding to the closest latent point. The transitions between standing support poses obtained by following these geodesic trajectories are also described in Table 9 As for our previous experi- ments, the geodesics, i.e., shortest paths, in the GPHLVM latent space correspond to shortest paths in the taxonomy graph. Due to the size of the taxonomy graph, we observe fewer forbidden (i.e. nonexistent) transitions than for the previous experiments in the Euclidean models. However, as their latent space does not match the taxonomy structure, they often require additional transitions and thus do not follow shortest paths in the taxonomy graph.
21

---

Under review as a conference paper at ICLR 2023
OF
FH
O
FSH
FH
FiH:
(a) Vanilla
(b) Stress prior
(c) BC + stress prior
(d) Adding poses
(c) Adding a class
Figure 8: Embeddings of standing poses: The first and last two rows respectively show the latent embeddings of hipedal poses and examples of interpolating geodesics in p and R² followed by pairwise distance matrices. Embeddings colors match those of Fig. righht. and background colors indicate GPLVM uncertainty. Added poses (d) and classes (e) are marked with stars and highlighted with red in the distance matrices.
Regularization
Stres
No regularizer
1.69=1.96
1E3
Stress
0.62=1.41
BC+Stress
0.68=0.96
unseen
0.61 0.84
unseen class
0.47=0.38
No regularizer
1.66=1.95
Stress
0.07-0.09
BC+Stress
0.15-0.18
unseen poses
0.17=0.20
unseen class
0.22-0.31
(a) Graph distance between the standing poses.
(b) Average stress per geometry and regularization.
Figure 9: Embeddings of standing poses: (a) shows the graph distance following the left part of Fig, right. (b) shows the stress resulting from the different embeddings of standing poses.
22

---

Under review as a conference paper at ICLR 2023
Start
End
Transitions in H
Transitions in R
F
FyH
FH FyH
F2 F F2 FH F2H
F
FaH2
F-> FH + FzH + F2H2
F + F2 - F-+FH, FaH1
FHg
- FH + FHg
F + F2 F-
FzH
FH2
FzH
FH
FH2
FaHl - FH
Fa
F2 F->FH2
FaH2 FH2
Table 9: Embeddings of standing poses: Transitions (+) between classes of the taxonomy obtained by fol-
lowing the geodesic trajectories depicted in Fig. [c The classes and transitions correspond to the colors along
the trajectories and match the class corresponding to the closest embedding at each point along the geodesic.
Transitions that do not exist the taxonomy denoted
G.3 HYPERBOLIC EMBEDDINGS OF STANDING POSES WITH AN AUGMENTED TAXONOMY FOR IMPROVED TRAJECTORY GENERATION
As shown in $ geodesics in the hyperbolic latent space of our GPHLVM intrinsically provide a mechanism to plan motions accounting for the underlying taxonomy. However, as discussed in S S the whole-body support pose taxonomy (Borràs et al. 2017) lacks information about the type of contact in the considered poses, thus leading to artifacts in the geodesic-generated motions. In the main paper, we showed that the quality of the generated motion is improved by augmenting the whole-body support pose taxonomy with additional contact information. To do so, we considered an augmented whole-body support pose taxonomy which explicitly distinguishes between left and right contacts. In other words, the nodes and transitions of Fig. right are adapted to consider left and right contacts. For instance, the 1-foot contact (F) node is separated into left-foot (F)) and right-foot (F") contact nodes. To facilitate motion planning and to test the GPHLVM ability of dealing with high-dimensional spaces, we represent each pose vector yn 24 of joint angles instead of vector of hands and feet positions.
We embed the 60 standing poses described in App. G.2 into 3-dimensional hyperbolic and Euclidean spaces using GPHLVM and GPLVM. respectively. For each approach, we test the model without regularization, with stress prior, and with back-constraints coupled with stress prior using the pa- rameters described in App. F2 and Table 4 Figs. [Oa 10c show the learned embeddings alongside their corresponding distance matrices, which are to be compared with the graph distances of the aug- mented taxonomy in Fig. aa Similarly to previous experiments, the models with stress prior result embeddings complying with the taxonomy graph structure (Fig. TOb), with additional intra-class organizations for the cocstronsrr models (Fig. [0c). Notice that the embeddings differentiate between left and right contacts according to the augmented taxonomy: For instance, we observe four clusters of orange embeddings corresponding to F'H', F'H",, F"H', and F"H". As shown in Ta- TTb] the hyperbolic models better represent the taxonomy structure and outperform the Euclidean models. Similarly previous experiments, the back-constraint mapping introduced in 41 allows us to properly place unseen poses or taxonomy classes into the latent space (see Figs. [Od
Examples of motions planned by following geodesics between two standing poses in the hyperbolic latent space are displayed in the main paper (Fig. (5). The corresponding geodesics are shown in Fig. [0c| with the colors along the trajectory matching the class corresponding to the closest hy- perbolic latent point. The resulting transitions are given in Table I0] As mentioned in the main paper, we observe that, in contrast to the trajectories of Fig. 4 the motions generated by considering the augmented taxonomy (Fig. 5) result in more realistic human-like interpolations between the given initial and final poses. Moreover, these motions look more realistic than the motions obtained via linear interpolation in the Euclidean latent space of the vanilla back-constrained GPLVM. As shown in Fig. 2 the motions planned in the Euclidean latent space sometimes result in unrealistic joint configurations and the same posture is associated with different types of contacts (see middle part of the motions). As shown in Table I0, non- oon-exxiting transitions arise more frequently when following trajectories generated by the Fuclidean model.
23

---

Under review as a conference paper at ICLR 2023
F
FHr, PHr, PHr.FHl.. FH
FiRt, FaH
Flyy FH2
(a) Vanilla
(b) Stress prior
(c) BC prior
(d) Adding poses
(e) Adding class
Figure 10: Embeddings of standing poses considering the augmented whole-body support pose taxonomy: The first and last two rows respectively show the latent embeddings and examples of interpolating geodesics in pa and R followed by pairwise distance matrices. Embeddings colors match those of Fig. right. Added poses (d) and classes (e) are marked crosses and highlighted with in distance matrices.
Regularization
rogularizer
3.85=3.63
R
Stress
BC+Stress
4.92=7.60
poses
1.93=2.34
class
2.31
No regularizer
4.05-33.77
B Stress
0.23-0.34
BC+Stress
1.25=2.14
unseen
1.60-
unseen class
3.20=4.59
(a) Graph distance between the standing poses.
(b) Average stress per geometry and regularization.
Figure 11: Embeddings of standing poses considering the augmented whole-bold support pose taxonomy: (a) shows the graph distance (colors follow Fig. right). (b) shows the stress resulting from the different embeddings of standing poses.
24

---

Under review as a conference paper at ICLR 2023
(a) F' to F2H"
(b) F' to F2H2
(c) Fr to FHH2
(d) F2H' F'H2
Figure 12: Motions obtained via linear interpolation in the latent space of the vanilla Euclidean back- constrained GPHLVM trained on the augmented taxonomy (Fig. [0c.. Contacts are denoted by gray circles. The colorbars identify the support pose of the closest pose in the latent space.
Start
End
Transitions in m
Transitions in R F'H F2Ht' FF2H"
FzH"
Fr F*H>
Fr
FzH2
Fl F2H' F2Hta
F1 > PH F2H F2HH" > F2Hta
Fr
F'Ha
F F'H" F'H2
Fr FT FH' FTH2
FzH'
F'Hg
FyH' + F2H2 + F'H2 FyH' + F'H + F'H2
Table 10: Embeddings of standing poses considering the augmented whole-body support pose taxonomy: Tran-
sitions (-+ between classes of the taxonomy obtained by following the geodesic trajectories depicted in Fig [0 The classes and transitions correspond to the colors along the trajectories and match the class corresponding to the closest embedding at each point along the geodesic. Transitions that do not exist in the taxonomy are denoted
G.4 COMPARISON AGAINST VARIATIONAL. AUTOENCODERS
Hyperbolic embeddings of support poses: In this section, we compare the trained GPHLVMs of Fig. 2 with two additional baselines: vanilla variational autoencoder (VAE) and a hyperbolic variant of this VAE which the latent space the Lorentz model of hyperbolic geometry (akin to Mathieu et al. ). Both VAEs are designed with 12 input nodes, 6 hidden nodes, 2- dimensional latent space, and a symmetric decoder. Their encoder specifies the mean and standard deviation of normal distribution (resp. wrapped normal for the hyperbolic VAE), and their decoder specifies the mean and standard deviation of the normal distribution that governs the reconstructions. Both models are trained by maximizing an Evidence Lower Bound (ELBO) under similar regimes as
25

---

Under review as a conference paper at ICLR 2023
OF
FH
F2
FuH
FH
FiHi
O FK
FKH
FKH,
KH
Ku
KH,
KiH
KiHo
(a) p2 Vanilla
(b) R Vanilla
(c) p2 Stress
(d) 2 Stress
Figure 13: Embeddings of the VAE baselines: The first and second rows show the latent spaces of the (hy- perbolic) VAE and the distance matrix between the latent codes, respectively. When comparing these distance matrices and encodings with that of GPHIVMs (see Fig. 2 we notice that proposed model is hetter able to preserve the graph distance structure. We argue because VAEs enforce latent spaces that follow unit Gaussian, which an opposite goal to ours.
the GPHLVMs, i.e., 1000 epochs with a learning rate of 0.05. The KL divergence for the hyperbolic VAE computed using Monte Carlo estimates.
Importantly, the VAE models only seem to capture global structure that separates standing from kneeling poses (except the vanilla hyper- bolic VAE in Fig. T Although adding a stress regularization with GPHLVM (? = 6) helps preserve the graph
Regularization
Stress dor
R2
No
1888+2.57
Stress
0.59.0.89
the same scale distance structure, the embeddings organization is still not competitive with the one achieved by our GPHLVM models (see Fig. 2). Moreover, when compared to our proposed GPHLVM, all VAE models provide subpar uncertainty modeling their latent spaces.
1 No reg. Stress
3.96224.22
0.52:10.71
Table 11: Average stress per geometry regularization for VAE baselines.
Table I1 shows that the average stress of the latent embeddings for the VAE baselines (trained with and without stress regularization) is higher than the average stress of models (see Table T. Overall, proposed GPHLVM consistently outperforms all VAEs to encode meaningful taxonomy information in the latent space. We argue that VAEs are not the right tool for target applications. When training VAEs, the Kullback- to regularize the latent space to match a unit Gaussian. This regularization is in stark contrast with goal of separating the embeddings preserve the taxonomy graph distances.
term in the ELBO tries
Hyperbolic embeddings of standing poses with an augmented taxonomy: We further com- pare our GPHLVM model against the vanilla and hyperbolic VAEs in the experiment described in Sec. G.3 Namely, we consider the augmented whole-body support pose taxonomy which explicitly distinguishes between left and right contacts and we represent each pose as vector of joint angles. This increases the dimensionality of the data to 44.
We tested the vanilla and hyperbolic VAEs without regularization and with a stress regularization with the same scale as for the GPHLVM (r 1.5). Fig. [4 shows the learned embeddings alongside dis- tance matrices, which are to be compared with the GPHLVM model of Figs. 10b 10c and with the ground-truth graph distances Fig. Ia Despite the stress regularization, the VAEs" tendency have unit- normally distributed latent representations hinders the distance match- ing. This is further quantified by the mean stress presented in Table 12 which show a higher mean stress (0.34 and 0.44) than our model the same taxonomy (0.23, see Table 11b..
Regularization
Stress +o
reg
2.33.13.10
Stress
0.3420.37
No reg.
0.4410.63
Table 12: Average stress geometry and regularization for VAE baselines trained augmented taxonomy (see
App. G.3)
26

---

Under review as a conference paper at ICLR 2023
P
FHi, PHr, PH. FH
Fjlt, FUH
PHy. FHH
Fafty
(a) H' Vanilla
(b) R Vanilla
(c) H Stress
(d) I Stress
Figure 14: Embeddings of VAE baselines considering the augmented whole-body support pose taxonomy:
The first and second rows show latent spaces of (hyperholic) VAE the distance matrix between latent codes, respectively.
GPHLVM Q = 2
GPHLVM Q
GPLVM Q =2
GPLVM Q 3
Training
2.5 X 103
8.91
5.9
6.3
Decoding
1.33 10-2
1.57
10
1.16
10
1.22
10-5
Table Average runtime seconds) training decoding phases of our GPHLVM and vanilla GPLVM over 5 experiments, using and 3-dimensional latent spaces for both models. Training time was measured over 500 iterations for both models. The implementations are fully developed on Python, and runtime measurements were taken using standard laptop with 32 GB RAM, Intel Xcon CPU E3-1505M v6 processor and Ubuntu 20.04 LTS.
Fig. 15 shows examples of motions planned by following geodesics between two standing poses the hyperbolic VAE latent space. Similarly as the motions generated in the latent space of proposed GPHIVM (Fig. 5), these motions result in realistic interpolations between the given initial and final poses.
G.5 RUNTIME
In order show the computational cost of approach, average runtime for the training and decoding phases, using and 3-dimensional latent spaces. As reference, we added the runtime measurements of Euclidean counterpart, that is, the vanilla GPLVM. Table shows the runtime measurements. Note that the main computational burden arises in our GPLHVM with a 2-dimensional latent space, which in sharp contrast with the experiments using a 3-dimensional latent space. As discussed in the main paper, this increase in computational cost is mainly attributed the 2-dimensional hyperbolic kernel.
of experiments to measure the
27

---

Under review as a conference paper at ICLR 2023
(a) F' to F2H"
(b) F' F2H2
(c) Fr to F'
(d) F2H' F'H2
Figure 15: Motions obtained via geodesic interpolation in the latent space of the hyperbolic VAE trained on the augmented taxonomy (Fig. [4c). Contacts are denoted by gray circles. The colorbars identify the support pose of the closest pose in the latent space.
28


Reviews:
[{"review_id": "RJczg1j-uaM", "reviewer": "Reviewer_1Gzu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes robot taxonomy, for hierarchical classifications of the human hand postures and body poses, through Gaussian process hyperbolic latent variable models (GPHLVM) built on top of existing GPLVM and using its graph-based distance-preserving priors and back constraints. The learned hyperbolic embeddings comply with the original graph structure and encode the unseen poses better than the euclidean ones. The assumption behind hyperbolic embeddings is that (1) the distance grows exponentially from the origin and (2) the shortest paths between 2 distant points tend to pass through the origin, thereby inducing a hierarchy. The model is evaluated on whole-body support pose taxonomy of 100 human poses of standing or kneeling and compared to euclidean pose embeddings. The ablation study shows that regularization with stress prior + back constraints improves clustering in Poincare ball P^2 and results in better latent space mapping of unseen poses and classes.", "summary_of_the_review": "Overall, the paper provides a perspective on the benefits of using hyperbolic embeddings over euclidean ones for humanoid joint pose taxonomy. The training paradigm for learning embeddings for taxonomy is driven by existing work in terms of using graph-based priors and back constraints, with modifications for the proposed hyperbolic manifold formulation.\nThe two advantages highlighted for the learned taxonomy are (1) generalization to unseen poses and (2) interpolation for generating trajectories. The latter, however, suffers from a lack of representation of contacts due to self-collisions and obstacles - a noted limitation of the current work.\n ", "strength_and_weaknesses": "### Strengths\nThe paper mentions the limitations and future directions for the community, such as the need to (1)  apply physical constraints on the generated motion, and (2) reduce the computational cost of the hyperbolic kernel. The lack of information on contact and object locations results in artifacts during interpolations between initial and final latent pose embeddings. \n\nThe paper is clearly written with sufficient additional details mentioned in the background and appendix sections. \n\nAdditional videos are provided for the generated trajectories on an augmented dataset with left & right side labels for hand and foot contacts.\n\n### Weakness\nIt is unclear to understand the generated trajectories as the contacts are not visualized, only the human joints are shown.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KiXrP3N684", "reviewer": "Reviewer_CG8r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to learn an embedding space, which leverages the human-designed taxonomy, for robotics tasks. It extends Gaussian Process Latent Variable Models (GPLVM) to hyperbolic latent spaces and proposes GPHLVM. The motivation for using hyperbolic space is that distances grow exponentially when moving away from the origin, and the shortest paths between distant points tend to pass through it, resembling a continuous hierarchical structure. They propose graph-distance priors and back constraints to introduce taxonomy knowledge into the GPHLVM.\n\nThe experiments are conducted on whole-body support pose taxonomy. The authors analyze the learned Hyperbolic latent embedding space and compare it with the Euclidean one. They also show that the learned embedding enables motion interpolation in the latent space.", "summary_of_the_review": "I feel that the experimental part is not enough to demonstrate the advantage of the proposed over existing methods. The practical usage is also unclear.", "strength_and_weaknesses": "Strengths:\n1. The idea of utilizing human-designed taxonomy prior in latent space learning is interesting.\n2. The introduction of Hyperbolic space seems to lead to better structure than Euclidean space.\n\n\nWeaknesses:\n1. I feel that the experimental part is not enough to demonstrate the advantage of the proposed GPHLVM:\n(a) There is only one task/dataset (whole-body support pose taxonomy) is used in the experiments. It is thus unclear whether the proposed method can be applied to other tasks as well.\n(b) There are no other alternative approaches for comparison (only Gaussian counterpart for visual analysis).\n(c) There is no quantitative comparison on some concrete tasks (e.g., pose classification) to demonstrate the practical benefit of the latent space.\n\n2. An alternative design would be using a neural network to learn an embedding space. We can still apply the priors (i.e., Hyperbolic and taxonomy-aware priors) to the learned embedding space by designing various loss functions. However, this alternative approach will be more straightforward and efficient to implement. Unfortunately, we fail to find such discussion or comparison in the paper.\n\n3. The proposed method hopes that the latent space complies with the human-designed taxonomy's hierarchical structure. However, as shown in the part of the experiments, when the human-designed taxonomy is coarse-grained, the performance of the proposed method is also limited.\n\n4. The runtime information is missing from the text. It's unclear whether the proposed method is an efficient or computationally intractable one. \n\n5. The current latent space is very low-dimensional. Could the authors provide more discussion on the benefits of learning a low-dimensional latent space? Also, it's interesting to know whether the proposed method can handle high-dimensional latent space as well.\n\n6. How shall we interpret the pairwise distance matrices in Figure 2 and compare them with the graph distances in Figure 3. To be honest, it's not easy to tell which one is closer to Figure 3.\n\n7. The interpolated motions in Figure 4 fail to impress me. The transition is not very natural and smooth. Moreover, it includes many physically-infeasible motions. In contrast, Figure 5 shows better performance. Since only motions generated by the proposed GPHLVM are shown, it is hard to demonstrate the advantage of the proposed GPHLVM, as other methods may generate similar or better performances.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5N9rIdBZj6", "reviewer": "Reviewer_ttyr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to model robotics taxonomy data via a proposed Gaussian process hyperbolic latent variable model (GPHLVM). The GPHLVM maintains the structure of the human-designed taxonomy while embedding the taxonomy node features into a hyperbolic latent space, which naturally encodes a continuous hierarchical structure. The GPHLVM is an extension of GPLVM for hyperbolic spaces, and the authors propose using Riemannian optimization methods for training the GPHLVM and use graph-distance information as inductive bias to learn the embeddings, encouraging the embeddings to adhere to the taxonomy structure. They evaluate the approach on a whole-body support pose taxonomy dataset, and experiments show that the 2-dimensional encodings display desired structure complying with the taxonomy. Experiments also show trajectory generation by following paths in the latent space between two embeddings,", "summary_of_the_review": "The authors introduce a novel idea of embedding robotics taxonomies into a hyperbolic manifold. This is a neat idea as the hyperbolic manifold is a natural representation for hierarchical structures such as trees and graphs. The paper is well written and includes extensive technical details and analysis, and the authors demonstrate nice applications on the whole-body pose dataset, showing various motions obtained via geodesic interpolation in the latent space.", "strength_and_weaknesses": "* The paper is well-written: the authors include extensive technical details in the main paper and appendix.\n* The approach is novel: it is the first to leverage the hyperbolic manifold in encoding hierarchical structures for robotics applications.\n* The experiments are sufficient in demonstrating the approach. The visualizations in Fig. 2 demonstrate that the embeddings comply with the graph structure, and that the hyperbolic approach outperforms the Euclidean counterpart even at higher dimensions.\n* Though physical constraints are not yet considered, the motion generation results look promising and indicate that the learned representations work well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject