Geometrically regularized autoencoders for non-Euclidean data
Authors: Cheongjae Jang
Yonghyeon Lee
Yung-Kyun Noh
Frank C. Park
Venue: ICLR 2023 poster (2023)
Keywords: autoencoders
Riemannian geometry
non-Euclidean data
regularization
score estimation

Abstract: Regularization is almost {\it de rigueur} when designing autoencoders that are sparse and robust to noise. Given the recent surge of interest in machine learning problems involving non-Euclidean data, in this paper we address the regularization of autoencoders on curved spaces. We show that by ignoring the underlying geometry of the data and applying standard vector space regularization techniques, autoencoder performance can be severely degraded, or worse, training can fail to converge. Assuming that both the data space and latent space can be modeled as Riemannian manifolds, we show how to construct regularization terms in a coordinate-invariant way, and develop geometric generalizations of the denoising autoencoder and reconstruction contractive autoencoder such that the essential properties that enable the estimation of the derivative of the log-probability density are preserved. Drawing upon various non-Euclidean data sets, we show that our geometric autoencoder regularization techniques can have important performance advantages over vector-spaced methods while avoiding other breakdowns that can result from failing to account for the underlying geometry.

Published as a conference paper at ICLR 2023
GEOMETRICALLY REGULARIZED AUTOENCODERS FOR MON-EUCLIDEAN DATA
Cheongjae Jang', Yonghyeon Lee Yung-Kyun Noh Frank Chongwoo Park 2.4 'Hanyang University, "Seoul National University, KKorea Institute for Advanced Study, *Saige Research
cj jjjanghanyanng. ac.kr, .r. ylee@kias.rem re.kr, nohyung@hanyang.cco ac. kr, fcp0snu. ac.kr
ABSTRACT
Regularization is almost de rigueur when designing autoencoders that are sparse and robust to noise. Given the recent surge of interest in machine learning prob- lems involving non-Euclidean data, in this paper we address the regularization of autoencoders on curved spaces. We show that by ignoring the underlying ge- ometry of the data and applying standard vector space regularization techniques, autoencoder performance can be severely degraded, to converge. Assuming that both the data space and latent space can be mod- eled Riemannian manifolds, show how to construct regularization terms in a coordinate-invanl way, and develop geometric generalizations of the denois- ing autoencoder and reconstruction contractive autoencoder such that the essential properties that enable the estimation of the derivative of the log-probabilit den- sity are preserved. Drawing upon various Euclidean data sets, we show that our geometric autoencoder regularization techniques can have important perfor- mance advantages over vector-spaced methods while avoiding other breakdowns that can result from failing to account for the underlying geometry.
worse, training can fail
INTRODUCTION
Regularization is almost de rigueur when designing autoencoders that are sparse and robust to noise. With appropriate regularization, autoencoders enable representations useful for downstream appli- cations (Bengio et al. 2013). generate plausible data samples (Kingma & Welling 2013 Rezende al. 2014). or even obtain information on the date-gengg probability density (Vincent al.]
2010
Rifai et al., 10
Existing work on autoencoder regularization has mostly been confined vector spaces, i.e., the data are assumed to be drawn from a vector space. On the other hand, a significant and growing number of problems in machine learning involve data that is non-Euclidean (in some past cases the fact that the data was non non-Euclidean was not recognized or ignored). [Bronstein et al. (2017) reviews several deep neural network architectures and modeling principles to explicitly deal with data defined on Euclidean domains, c.g., data collected from sensor networks, social networks in computational social sciences, or two-dimensional meshes embedded the three-dimensional space. Other works have also addressed manifold-valued data including human mass and shape data (Kendall 1984 Freifeld & Black] 2012), directional data (Mardia 2014), point cloud data (Lee et al.) 2022). and
MRI imaging data (Fletcher & Joshi 2007 Banerjee al. 2015). with several deep neural networks proposed to handle such data in coordinate-invariant way (Huang & Van Gool 2017: Chakraborty
al. 2020).
The fundamental idea behind these works is that the geometrical structure of the curved space from which the non-Euclidean data are drawn needs to be accounted for properly, so that the output of any deep learning network applied to such input data should not depend on the particular choice of coordinates used to parametrize the data. Ignoring the underlying geometry of the data and simply applying standard vector space techniques can severely degrade performance, or worse, cause training to fail. Autoencoder training and its regularization are no exception.

---

Published as a conference paper at ICLR 2023
For example, consider autoencoder training on a set of data points on a sphere as shown in Figure [ When us- ing spherical coordinate representations as inputs to train autoencoder with a contractive regularization, the trained reconstruction function can heavily de- pend on the choice of coordinates. Moreover, often fails learn the cor- rect contractive directions toward data- dense regions, especially near the sin- gularity (or the spherical coordinate ori- gins). On the other hand, an autoencoder that properly reflects the spherical con- straints can recover those directions suc- cessfully and show results almost invari- ant to the choice of coordinates.
(a) Training data
(b) RCAE
(c) GRCAE
Figure 1: Autoencoder training on spherical data sam- pled from the von Mises-Fisher (vMF) distribution. We train the reconstruction contractive autoencoder (RCAE) and the geometric RCAE (GRCAE). For (b)-(c), we plot the reconstruction directions of the autoencoders trained using representations obtained from different coordinate choices. The results from each coordinate choice are color-coded along with corresponding spherical coordi- nate origins. (See Appendix E. for more details.)
In this paper we address the regularization of autoencoders on curved spaces. Any loss function used to train or regularize the autoencoder should be formulated in a coordinate-invarian way, i.e., invariant to the choice of local coordinates used to parametrize the data, and instead depend only on the intrinsic properties of the curved space such as curvature or the choice of metric.
Assuming that both the data space and latent space can be modeled as Riemannian manifolds, we show how to construct regularization terms and objective functions in a coordinate-invaiian way. We also develop geometric generalizations of the denoising autoencoder (DAE) and reconstruction contractive autoencoder (RCAE) such that the essential properties that enable the estimation of the score, i.e., the log-derivative of the data-generating density, are preserved. We provide some ap- plications that use this property, such as sampling, clustering, and filtering for non-Fuclidean data, and also show that the proposed autoencoders can obtain useful representations for non-Euclidean data, especially when noise exists in data. Drawing upon various -un-Euulid data sets, we show that our geometric autoencoder regularization techniques can have important performance advan- tages over vector-spaced methods in some cases by significant margins while avoiding other breakdowns that can result from failing to account for the underlying geometry.
The paper is organized follows. We describe regularized autoencoders for Euclidean data in Section _ and propose their coordinate-invarin generalizations to non-Fuclidean data Section We then provide autoencoder training studies using non-Fuclidean data sets in Section 4
REGULARIZED AUTOENCODERS FOR EUCLIDEAN DATA
Mathematically, an autoencoder can be represented as the composition of two mappings f: R Rd (the encoder) and g R + IR (the decoder), i.e., r g a ff RDD + RDD with the space of hidden variables Rdd Assume there exists a data gentratnnn probability density p : IRD - R from which data points IR' drawn. Autoencoder training in vector space can then be formulated as minimizing the reconstruction error fgo || g(f(x; 01): 12)11 p(z) dx over 0 = (01, 02). where x E I2D denotes an input variable, 0, and 0g are respectively the parameter sets of the maps and g, and 11- is the squared Euclidean norm. In the typical case where d < D, the autoencoder engages in type of dimensionality reduction. By disregarding the assumption that d < D, autoencoders have been modeled in the form of deep artificial neural networks accompanied by certain regularization terms to learn useful representations of the data (Bengio et al. 2007: Vincent et al. 2008 2010 Ranzato et al. 2007: 2008 Kingma Welling, 2013) Rezende et al. 2014 Rifai et 2 For more comprehensive review of autoencoders, we refer the reader to Goodfellow al. (0166).
201 Ibla).
In the meantime, the effects of regularization have been investigated in some detail in Alain & Bengio (2014) for the denoising autoencoder (DAE) (Vincent et al. 2010) and the reconstruction contractive autoencoder (RCAE). They point out that these regularization methods reduce the au- toencoder's sensitivity the input, while the reconstruction error increases the autoencoder's 's sen- sitivity to variations along the region of the highest density in the data space. Reconstruction and regularization together successfully capture variations in such regions while ignoring variations that
2

---

Published as a conference paper at ICLR 2023
orthogonal to those and obtain information the data generatinn probability density. Our focus will be on these two types of regularized autoencoders, i.e,, the DAE and the RCAE, but the methods described in this paper are easily generalizable other autoencoders.
In the standard vector space formulation of the DAE, an imput x E IR is assumed to have been corrupted by some noise density (( (r)) ê E R., iie., 3 ~ (((r.)) We then seek the reconstruction function r = o f: RDD + RD that minimizes
min [ F [Ir(#) - x1 x=|²] p(x) de, 20
where (g(# u) denotes the expectation with respect to the noise density (((())) trivial identity mapping r(x) can be avoided due to the injected noise.
For the vector space formulation of the RCAE, the objective function is
min [ IEP (Ir(x) - + o
(
(r 0r
Ör 0x.
p(x) dx,
where o² is scalar weighting coefficient. The second term in ( acts regularization term and can be interpreted as the Dirichlet energy of r : RD -> IR,, measuring how variable the mapping is (Belkin & Niyogi 2003) [Solomon et al.] 2013). Minimizing this term induces the contraction of the mapping (e.g., in the absence of the reconstruction error term in (2), an extreme contraction of constant would be obtained), preventing r(w: from becoming the identity mapping r(x: x.
01 in reduces the objective function for the contractive Note that replacing Lar with 2rr autoencoder (CAE) (Rifai et al. 2011b).
For both the DAE under Gaussian corruption process ă N(x, o² I) and the RCAE with small, the derivative of the log-probabilit density, which is also referred to as the score, can be estimated from the optimized (1)) as follows (Alain & Bengio 2014):
i log p(x)
1 0p (x) p pdx
r(x) +
(3)
dz
3 REGULARIZED AUTOENCODERS FOR NON-EUCIE CON-EUCLIDEAN DATA
In this section, we address the problem of autoencoder training for the case where the data points are drawn from priori known non-Euclidean space M, possibly with another non-Euclidean latent space N. We formulate the reconstruction error and regularization terms in a perrdinateiivvarian way using notions from Riemannian geometry. (We provide some mathematical backgrounds re- quired for our formulations in Appendix A We then show that, as in the Euclidean case, it is possible to estimate the score for non- -un-Fuclidean data using the trained autoencoders.
3.1
GENERALIZATIONS OF THE AUTOENCODER COMPONENTS
Referring Figure/ let M be dimensional manifold with local coordinates (x+ ....,,""" x²") and Riemannian metric dx'dr' Throughout
d = Em 1 this paper, we use italics to represent local co- ordinates, e.g., a point x E M has local coordi- nates r I and the mapping M + M can be represented in local coordinates as r: mR - R. The metric will also be denoted in matrix form as G(x) = (x)) E xm We now provide coordinate-invariant general- izations of each component in autoencoder train- ing, especially in (1) and (2), while simultane- ously reflecting any intrinsic properties of the manifold M.
G(a)
Glrllb
(0)) oveen(
Figure 2: Local coordinates and Riemannian
metrics for the reconstruction mapping r: M -> M. Local coordinates are denoted in italics.
3

---

Published as a conference paper at ICLR 2023
Reconstruction error: The reconstruction enrrr Riemannian manifolds can be defined as dist(r(x), x)), where yist(x,y) denotes the minimal geodesic distance between points x, y E M..
Probability function: Assume there exists a data-generating probability function Pg M + R from which data points on M are drawn. Denote by Pg : R. + its representation in local coordinates, satisfying (2(x) > O for all z E R and JM ( G(z)dz = 1 Pennec 1999), where det G(r))dr is the natural volume element induced from the metrie. When formu- lating regularized autoencoders later, Pg (or Pg) is used in the form of weighted volume element p(z)dz = Pg (x) det G(z) In practice, the integrations involving p(z) (()dz dz are approximated as an equally weighted finite sum of the integrands (with p(z) excluded) evaluated at given data points.
Contractive regularization: The conventional Dirichlet energy (appearing in (2 has been general- ized to mappings between Riemannian manifolds in the theory of harmonic maps (Eells & Sampson 1964). based on which we formulate the contractive regularization for non- -on-Euclddaan settings.
Let M be the input manifold, and let / be an n-dimensional output manifold with local coordinates y = (v', "u',...,'" y²) and Riemannian metric dr² = >x=1 Σs= has (y) dy The metric will also be denoted in matrix form as (y) = (y)) E R=x" In Eells & Sampson 1964) the Dirichlet energy of a smooth map f: MM- - N is defined as
Ja TTJ HJJT G dol ... dxm".
(4)
where J(xx) = A5
des (x)
R= is the differential dfx Tx.M- - Ty.V denoted in local coordi-
nates. The energy functional in (4) is an intrinsic quantity, i.e., coordinate-invariann, We discuss the poordinate-invariann
of (4) and some physical interpretations of minimizing (4) in Appendix B.1 and refer the reader to the extensive literature on the theory and applications of harmonic maps, e.g.,
Eells & Lemaire (1978) 1988); Park Brockett (1994): Gu al. (2004); Jang al. (202I): Lee et al. (2021b).
To apply this energy functional to autoencoder regularization, we replace the mapping f: M - W and the Riemannian metric H (zz) with the reconstruction mapping M + M and G(r(z)), respectively, and replace the natural volume element /det G dz with the weighted volume element p(z) dr. Also note that for the case of non- non-Euclidean latent space N, the objective in (4) with the volume element p(r) dr can serve as geometric regularizer for the contractive autoencoders. (See Appendix B.2 for more discussions related to non-Fuclidean latent spaces.)
Data corruption process: To corrupt an input X E M, sample a tangent vector E T.M from an isotropic zero-mean multivariate Gaussian, i.e., linear combination of an orthonormal basis for T&M with coefficients sampled from o2 and then apply the exponential map Exp, T.M + M to v. The point x Exp, (v) can then be inter- preted as a corrupted point from x. We denote by ((([x) the noise density that samples x for given x according to this pro- cedure. A data corruption example for data on sphere (S²) is illustrated in Figure 3 and a rationale for adopting this way of corruption is explained in Appendix B.3
TyS²
Expg(s)
Figure 3: A data corruption exam- ple for an input x E S Tangent vectors (the yellow triangles) are sampled from a zero-mean Gaus- sian in T,S² and mapped via the exponential map to black dots, the corrupted points from x. The red dot x = (v) for v E T,S².
3.2 GEOMETRICALLY REGULARIZED AUTOENCODERS.
We now derive coordinate-invarian formulations of the regu- larized autoencoders presented in (T) and (2). For the recon- struction contractive autoencoder (2). the contractive regular- izer modified from (4) (as explained above) is augmented to the reconstruction error term follows:
min / (dis((r(x) x)² Tr
((0r)
(or) (x
Or
G-1
)-1)
G(r)
p(x) dr,
(5)
Ox
4

---

Published as a conference paper at ICLR 2023
where the reconstruction map .M- - M is expressed in local coordinates as r R - R", and
G(r) denotes the metric at point r(æ). We refer to as the geometric RCAE or GRCAE.
Similarly, the geometric version of the DAE referred to GDAE, can be formulated as follows:
min S/a xx(ix) [ist(r(X)), p(x) dr,
(6)
where (x(())) [+] denotes the expectation with respect to the noise density ((]xx) presented above.
From the geometric formulations ())
obtain the following relations between the recon-
struction function r and the log of the probability function Pg Pz(2) det G(x)
p(a))
Theorem 1. Provided o² is small, the derivative of the log of the probability function Pg(x)
p(a) can be approximated for both the GRCAE and GDAE as
v det G(z)
Olog)((o)
1 L.pg (z) = G(x)
02 (((o)==) x - 0
((x)
Ox Pg dr
The proof of Theorem I provided in Appendix Equation 7
can be thought of as generaliza-
tion of for non-Fuclidean data, and we will refer to
Boo ( as the geometric score.
dx
The reconstruction function M .M for proposed autoencoders is modeled neural network in later experiments, with implementation details provided in Appendix [ We also provide there some ideas to deal with the case of manifolds that require multiple coordinate charts and discuss another case where the data space Riemannian metric is not known a priori.
4 EXPERIMENTS
In the experiments, we first demonstrate the geometric score estimation (Theorem D based on our geometrically regularized autoencoders for non-Euclidean data, providing solid basis for future applications autoencoders. We then utilize the proposed autoencoders for various applications, such data sampling based on the Langevin Monte Carlo methods (Girolami & Calderhead, 2011 or clustering and noise filtering based on mode-seeking (Fukunaga & Hostetler] 1975] Cheng 1995
Comaniciu & Meer 2002), involving real-world non-Huclidean data sets. We also examine the usefulness of the proposed autoencoders in the representation learning perspective, using noisy point cloud data.
4.1 GEOMETRIC SCORE ESTIMATION
For geometric score estimation, we consider the data sampled from P(n), the space of n X n sym- metric dositive-definite matrices, endowed with the affine-invarian Riemannian metric (Fletcher & Joshi, 2007). We train the GDAE and GRCAE using synthetic data sampled from m mixtures of isotropic tangent space Gaussians for which the ground truth geometric score values are obtainable. For purposes of comparison, we also use DAE, RCAE, the least-squares log-density gradient method (LSLDG) presented in Sasaki et al. 20014). and also their extension data on Riemannian manifolds (R-LSLDG) in Ashizawa et al. (2017). Full experimental details are provided in Appendix F
For a given data set {II..... NN represented in local coordinates of P(n), the geometric score estimation error (Est. error), which is also defined to be coordinate-invariane is evaluated as follows:
Est. error N
① (logg
(xx)
0log Bi
(x.))
0 llog
(xx)
Ølog Pa 8e
(x:)
(8)
lest
est
where Ooo Pa dur
(I:) and
a dx
(x.) respectively denote the estimated value and the ground truth
value for Ooo po(x) at Fi. The estimation errors measured on 10, 000 test data points are averaged Note that the GDAE, GRCAE, and R-LSLDG methods show superior
five 24 Table 1
performance over DAE, RCAE, and LSLDG in estimating Po
p. (w) Notably, GRCAE shows the dxz
best performance for higher dimensionality and higher number of mixtures in terms of estimation error. We also obtain a similar tendency for another synthetic data set on the hypersphere S" = (p e Rn+1 ||p|| 1), and the results provided in Appendix R.4
5

---

Published as a conference paper at ICLR 2023
Table I: Estimation errors for fo for mixtures of tangent space Gaussian data P(n) with the standard errors in parentheses. Bolds represent the best and comparable methods from the t-test with significance level of 5%.
TR
LSLDG
DAE
RCAE
R-L.LDG
GDAE (ours)
GRCAT (ours)
2
2
10.5 (0.87)
8.72 (3.03)
9.04 (2.49)
(..19)
3.55 (0.31)
2.61 (0.20)
30.0 (0.91)
(1.23)
(1.83)
(0.19)
4.33 (0.34)
3
3
(2.16)
(0.97)
22.4 (1.18)
7.07 (0.31)
6.86 (0.96)
5.85 (1.07)
4
75.5 (0.85)
42.4 (0.69)
43.8 (1.01)
9.67 (0.78)
7.72 (0.60)
6.97 (0.39)
4
90.8 (1.91)
59.5 (4.41)
12.2 (6.3()
11.8 (0.71)
10,5 (1.06)
4
88.4 (0.82)
73.2 (2.98)
(4.76)
13.9 (0.82)
(6.29
12.6 (0.28)
4.2 SAMPLING ON S
We can apply the geometric scores estimated from GDAE and GRCAE to the sampling of non- Euclidean data via Riemannian Langevin Monte Carlo (RLMC) methods. The stochastic process for the RLMC methods in Girolami & Calderhead (2011) can be reformulated using the geometric score in (7) as follows:
dx = (x)
Olog (x) PlogPo()) 1 - (=) 8
03
+ ((x) dt dx
(9)
Or
where dw E I denotes the Brownian motion in an m-dimensional vector space, (xx)
1/ /et G(x), and (xx) I2th is vector whose i-th component is given by Σラ= =1 der with g2) as
8g
the (i, () element of G (x:). discretization of the above process gives the RLMC method (Giro-)
lami & Calderhead 2011). Note that in (9), the terms except for (z) (log (x) to the Brownian motion on manifolds (Brockett 1997).
dt correspond dr
As an illustrative case study, we consider sampling on a sphere (S²) After training DAE, RCAE, GDAF, and GRCAE for data points on S shown in Figure 4 (left), we sample new data points that approximately follow the original data distribution by applying the RLMC methods using the geo- metric scores estimated from GDAE/GRCAL and its Euclidean counterparts using DAE/RCAE, We also report the results obtained from the S-Flow method (a variation of the M-Flow method (Brehmer & Cranmer 2020) for the case of S²). The experimental details provided in Appendix G
For a quantitative comparison of the sampling performances, the maximum mean discrepancy (MMD) (Gretton et al. (0122) between test data set and the obtained samples is provided in Table As shown in the table and Figure GDAE shows the best quantitative and qualitative performance among the considered autoencoders. Even though the samples from GDAE do not yield better nu- merical results than those from the S-Flow method, an algorithm targeted for data sampling, it is observed that plausible samples can obtained. Also note that the samples obtained from RCAE and GRCAE tend to be inferior to those from DAE and GDAE in this task, possibly due to the algo- rithmic properties of RCAE/GRCAE which, unlike DAE/GDAE, the inputs to neural networks are strictly confined to the given training data. Thus, the reconstruction functions may not be accurately trained on other regions visited during the sampling process compared to DAE/GDAE.
Table 2:
The MMD measure (the lower, the better) between the test and the sampled data with standard errors in parentheses.
x x
DAR
RCAF
GDAF (ours)
GRCAF (ours)
S-Flow
four blabs
0.215 (0.015)
0.598 (0.059)
0.188 (0.018)
0.291 (0.052)
0.137 (0.038)
two moons
0.147 (0.022)
0,249 (0.020)
0.144 (0.032)
0.161 (0.020)
0.100 (0.030)
s-curve
(0.042)
0.248
0.242 (0.031)
0.231 (0.072)
(0.063)
circles
0.075 (0.02.)
0.353 (0.012)
0.061 (0.005)
0.323 (0.008)
0.046 (0.093)
4.3 CLUSTERING OF HYPERSPHERICAL DATA
In natural language processing, the similarity between word embeddings, i.c., the vector represen- tations that encode semantic information of the words, is often measured according to the cosine

---

Published as a conference paper at ICLR 2023
Data
DAE
RCAE
GDAE (ours)
GRCAE (ours)
slow
Four bobs
Figure 4: Data sampling on S?
similarity (Mikolov al. 2013). This be equated considering the embeddings points hypersphere and measuring the distance based on the geodesic distance (Straub al. 2015).
Based this idea, group documents in the Newsgroup20 data set (Lang 1995) using the GDAE trained on the document embeddings. The document embeddings are first represented as the average of the word embeddings in the document and then projected hypersphere. For the word embed- dings, we utilize pre-trained 50-dimensional Glo Ve embeddings (Pennington et al. 2014), hence the document embeddings lie on $49 We define four clustering tasks as described in Appendix H.1
We train DAE, GDAE, LSLDG, and R-LSLDG explained in Appendix H.2 Here consider two variations DAE and LSLDG, respectively; the first ones are trained on the spherical coordinate the representations in the ambient space IR++1
representations of the data, and the second After training the models, the document embeddings are iteratively updated along the gradient of the log-probability (for GDAE, this can be performed by repeatedly applying the reconstruction function on the embedding) for fixed number of steps and then grouped as done in the mean shift clustering (Comaniciu Meer 2002). The adjusted Rand index (ARI) is used as the performance metric for the clustering tasks (Hubert & Arabie 1985)
Table The adjusted Rand index for the clustering results on the Newsgroup20 data (S+)) with standard errors in parentheses. The higher, the better.
LSLDG
LSLDG
DAE
R-LSLDG
(ours)
crypt
0.174 (0.016)
0.034 (0.025)
0,205 (0.069)
0.071 (0.040)
0.221 (0.109)
0.465 (0.018)
electronics
0.204 (0.006)
0.036 (0.018)
0.228 (0.082)
0.092 (0.001)
0.192 (0.099)
0.448 (0.044)
med
0.153 (0.012)
0.021 (0.017)
0.227 (0.120)
0.071 (0.040)
0.245 (0.090)
0.451 (0.055)
space
0.176 (0.0199
0.025 (0.026)
0.211 (0.066)
0.089 (0.001)
0.241 (0.081)
0.429 (0.070)
The averaged clustering performance for five runs of each method is presented in Table 3 Note that the vector-spaced methods trained on the ambient representations of the data are hardly successful grouping the data. On average, the R-LSIDD performs slightly better than LSLDG and DAE but with higher variance. The results obtained from GDAE show much higher ARI than others. An additional case study for the clustering covariance matrix data provided in Appendix H.3
4.4 FILTERING OF DIFFUSION TENSOR IMAGING DATA
For the next case study, we consider data ob-
Table 4: The R-squared score for N(2) data filter- ing. The higher, the better.
tained from diffusion tensor imaging (DTT). Mathematically,
DTI datum a three- dimensional image in which the value assigned to each voxel an element of P(3). voxel of DTI data can be treated in the space of three- dimensional normal distributions N(3), by re- garding the voxel location and voxel value
Noise level
DAE
GDAE (ours)
MVKR
0.02
99,80 (0.00)
99.86 (0.00)
96.31 (0.01)
0.05
99.12 (0.05)
99.51 (0.03)
96.21 (0.03)
0.1
97.14
98.51 (0.03)
95.79 (0.04)
0.2
89.44 (0.32)
93,70 (0.47)
94.11 (0.10)

---

Published as a conference paper at ICLR 2023
the mean and covariance of normal distribution, respectively. In this section, following Han & Park (2014) we adopt the Fisher information Riemannian metric (FIRM) for DTI data (see Ap- pendix D33.1 for the definition of the FIRM). We train the GDAE DTI data and apply the
trained GDAE to filter the noise in the data.
data set comprised of N voxels ( Pi)..... (IN, of a raw DTI datum, where It E R and P. P(3) denote the location and value of the i-th voxel, respectively. For the data set, we train the GDAE with the reconstruction function N(3) N(3); the overall training process is explained in Appendix L.I
Consider
Clean data
input
MYVIR
DAE
GDAE
Figure N(2) data filtering (noise 0.2). The redder, the higher error.
By iteratively applying the reconstruction function of the trained GDAE, data points are mapped toward the local modes the probability function (as can be implied from (7). and the voxels with added noise, which usually have a lower probability, Meer 2002). We summarize a DTI filtering algorithm based on the GDAE Appendix I.2
be automatically filtered (Comaniciu &
We first demonstrate the effectiveness of this algorithm in simpler setting by using synthetic data set in wwo-dimensional normal distributions N(2). We train our GDAE on a noisy input artificially corrupted from clean data in Figure and apply the filtering algorithm. The proposed algorithm can effectively filter out the noises qualitatively better than other filtering methods, such as the DAE-based filtering with more remaining noises and a manifold-valuee kernel regression-based filtering approach (MVKR) (Banerjee et al., 2015) which tends to erroneously smooth discontinuous voxel values. guantitative comparison Table also suggests that our GDAE-based filtering can perform well at various noise levels. The experimental details are deferred to Appendix
I3
We now conduct numerical experi- ments of DTI filtering by applying our algorithm. Data used in these experiments were obtained from the Alzheimer's Disease Ncuroimaging Initiative (ADNI) database (adni.loni. usc.edu). See Appendix 1.4 for the data preprocessing details. We also perform filtering based on DAE trained on nine-dimensional vector representations of voxels. Note that, critically, the DAE-based filtering has no guarantee for the filtered voxel val- ues to be positive-definiee while GDAE-based filtering always satisfies the constraint.
(a) Noisy input
(h) DAE
(c) GDAE
Figure 6: DTI filtering results.
The input image and the filtering results from each algorithm are shown in Figure 6 We plot axial slices of the corresponding DTI data in 6 (a)-(c). Voxel values are drawn as ellipsoids with colors representing the direction of the first principal axis.
Since the ground truths of DTI data are not available, only qualitative comparisons between the filtering results can be made. In Figures 6 (a)-(c), it can be seen that the GDAE-based method effectively filters outliers (the abruptly changing colors appearing in Figure 6 (a)) when compared to DAE-based filtering. Using brain anatomical terms, the separation between the cerebrospinal fluid and brain parenchyma became clear after ADDAEbassed filtering, and as a result specifying the sulci and gyri of the brain is much easier than from the raw input DAE-based filtering results. Furthermore, noise in the ventricle area also disappeared after GDAE-based filtering, while DAE- based filtering failed eliminate this noise. We should note here that the DTI filtering results would require further concordance verification procedures with experts such as radiologists to ensure that only spurious artifacts are erased rather than important anatomy.
8

---

Published as a conference paper at ICLR 2023
4.5 ROBUST REPRESENTATION LEARNING OF POINT CLOUD DATA
We now show that can obtain useful representations for non-Fuclidean data from our proposed autoencoders, especially when noise exists in the data. We consider point cloud data in this case study, A point cloud data in RD is a set of n points in RP, represented as X {.i.....33 RDD Recently, a statistical manifold framework has been suggested for point cloud data in Lee al. 2022), and they have observed that reflecting this geometry to train autoencoders can obtain better representations than the vector space counterparts. (See Appendix D.4.1 for more details.)
... 2n
24 E
Based on this choice of geometry, we train GDAE for point cloud data with n 048 and D (hence of dimension nD = 6, 144). We reflect the Fisher information metric proposed in Lee jet al. (2022) in the data corruption process for GDAE and use the modified Chamfer loss as the reconstruction error for point cloud data. For comparison purposes, we also consider DAE trained on the vector representations
point cloud data. We use the FeNet (Yang et al. 2018) as the structure for the reconstruction functions with latent space dimension of 512. Further details are provided in Appendix D.4
To verify the usefulness of the representations obtained from trained autoencoders, utilize them as features to train classifiers in the transfer learning setting based on some benchmark data sets following Yang et al. (2018). More specifically, we train autoencoders using the ShapeNet data set (Chang et a 2015) and obtain representations for the ModelNet data set (Wu et al., 2015). We then train linear SVM using the representations and measure the transfer classification accuracy- We inject varying noises in the data sets to verify if the obtained representations are robust to noise. More experimental details are explained in Appendix
Table Classification accuracy by transfer learning for ModelNet10 and ModelNet40 from ShapeNet under the noise levels of 0.01, 0.05, 0.1, and 0.2.
ModelNet1O GDAB
ModeINer40 GDAE
Noise lovel
AE +R.
GDAE (ours) + R.
AE -R.
GDAE (ours) + R.
DAB
AE
DAE
(ours)
(ours)
0.01
93.1
92.7
93.2
92.8
93.2
87.3
87.0
87.3
88.6
88.8
92.3
93.3
86.8
86.5
0.1
88.9
90.6
91.1
90.2
91.7
75.6
79.9
83.2
80,5
82.4
88.2
84.8
64.6
72.2
76.1
The experimental results are in Table 3] The features obtained from GDAE lead to a better transfer classification accuracy than those from the vanilla autoencoder (AE) or DAE; this tendency gets stronger as the noise level increases. Our approach also shows comparable or better performances compared another regularization method ('AE R.' in Table 5 considered in Lee et al. (2022), which tries to match the pull-back metric of the Fisher information metric (via the decoder mapping) the identity. Combining these two regularization methods (' ('GDAE + R. in Table shows the best overall transfer classification accuracy, demonstrating the usefulness of our geometric regularization methods in obtaining better representations of oon-Euclidean data for downstream tasks.
5 CONCLUSION
In this paper, we have introduced geometrically regularized autoencoders for non-Euclidean data. By constructing regularization terms in a coordinate-invariant way, we have developed two types of geometric autoencoders, the geometric reconstruction contractive autoencoder and the geometric denoising autoencoder. These autoencoders are effective in estimating the derivative of the log of the probability density of non-Fuclidean data and have been successfully applied to several applications such as data sampling, mode-seeking, and representation learning tasks involving real-world non- Euclidean data sets. Although training these models can be computationally demanding and may involve numerical issues when reflecting the geometry of the data, our experiments show that our approach can be a viable option for handling nigh-dimensional and complex non- non-Euclidean data. In the future, it would be worth investigating more efficient and robust methods for reflecting the geometry during training. Additionally, the idea of using geometric regularizations could be applied to other machine learning problems that involve on-Fuclidean data.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
C. Jang and Y.-K. Noh were supported by IITP Artificial Intelligence Graduate School Program for Hanyang University funded by MSIT (Grant No. 2020-0-01373), Y.-K Noh was partly sup- ported by NRE/MSIT (Grant No. (01881A5A70595499 201IM3EDD2001195555) and IITP/MSIT (Grant No. LTTP-2021-0-02066 Y. Lee and F. C. Park were supported in part by SRRC NRF grant 2016R A5A 1938472, IITP-MSIT (Grant 2022-0-00480, Development of Training and Infer- ence Methods for Goal-Oriented Al Agents, 20%), SNU-AIIS, SNU-IAMD, and the SNU Institute for Engineering Research.
REPRODUCIBILITY STATEMENT
We refer the reader the following pointers for reproducibility:
Codes to train the proposed autoencoders: Supplementary Material.
Proof of Theorem [
Appendix C
Implementation details of the proposed autoencoders: Appendix
Experimental settings: Appendix
and
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. Journal of Machine Learning Research, 15(1):3563-359) 2014.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference Learning Representations, 2018.
Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. In International Conference on Artificial Intelligence and Statistics, pp. 631-639, PMLR, 2021.
Mina Ashizawa, Hiroaki Sasaki, Tomoya Sakai, and Masashi Sugiyama. Least-squares log-density gradient clustering for riemannian manifolds. In Artificial Intelligence and Statistics, pp. 537- 546, 2017.
Monami Banerjee, Rudrasis Chakraborty, Edward Ofori, David Vaillancourt, and Baba C Vemuri. Nonlinear regression on riemannian manifolds and its applications to neuro-image analysis. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 719-727. Springer, 2015.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373-11) 1396, 2003.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 1798-18988 2013.
William M Boothby. An introduction to differentiable manifolds and Riemannian geometry, volume 120. Academic press, 1986.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estima- tion. Advances in Neural Information Processing Systems, 33:442-453, 2020.
Roger Brockett. Notes on stochastic processes on manifolds. In Systems and Control in the twenty- first century, pp. 75-100. Springer, 1997.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42 2017.
10

---

Published as a conference paper at ICLR 2023
Rudrasis Chakraborty, Jose Bouza, Jonathan Manton, and Baba C Vemuri. Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint lrxiv: 2015.
Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 17(8):790-799, 1995.
Dorin Comaniciu and Peter Meer. Mcan shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, (5):603-619, 2002.
Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspheri- cal variational auto-encoders, arXiv preprint arXiv:1804.0088 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605 160508803, 2016.
Boris Dubrovin, Anatolij Timofeevič Fomenko, and Sergei Petrovich Novikov. Modern Geom- etry-Methods and Applications Part I. The Geometry of Surfaces, Transformation Groups, and Fields. Springer, 1992.
James Fells and Lue Lemaire. report on harmonic maps. Bulletin of the London Mathematical Society, 10(1):1-68, 1978.
James Bells and Luc Lemaire. Another report harmonic maps. Bulletin of the London Mathe- matical Sociery, 20(5):385-5244 1988.
James Eells and Joseph Sampson. Harmonic mappings of riemannian manifolds. American Journal of Mathematics, 86(1):109-166) 1964.
P Thomas Fletcher and Sarang Joshi. Riemannian geometry for the statistical analysis of diffusion tensor data. Signal Processing, 87(2):250-25) 2007.
Oren Freifeld and Michael Black. Lie bodies: A manifold representation of 3d human shape. In European Conference on Computer Vision, pp. 1-14. Springer, 2012.
Keinosuke Fukunaga and Larry Hostetler. The estimation of the gradient of density function, with applications in pattern recognition. IEEE Transactions on information theory, 21(1):32-40, 1975.
Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2): 123-214, 2011.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/www. deepl ea r ngbook. rgbooookk..ooooooooooooooooooo.. org
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Xianfeng Gu, Yalin Wang, Tony F Chan, Paul M Thompson, and Shing-Tung Yau. Genus zero surface conformal mapping and its application to brain surface mapping. IEEE Transactions on Medical Imaging, 23(8):949 9009999588111 2004.
Minycon Han and Frank C Park. Dti segmentation and fiber tracking using metrics multivariate normal distributions. Journal of mathematical imaging and vision, 49(2):317333, 2014.
Søren Hauberg, Oren Freifeld, and Michael Black. A geometric take on metric learning. Advances in Neural Information Processing Systems, 25, 2012.

---

Published as a conference paper at ICLR 2023
Zhiwu Huang and Lue Van Gool. A riemannian network for spd matrix learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr))695-709, 2005.
Cheongjae Jang, Yung-Kyun Noh, and Frank Chongwoo Park. riemannian geometric framework for manifold learning of non-cuclidean data. Advances in Data Analysis and Classification, 15 (3):673 699, 2021.
Sadeep Jayasumana, Richard Hartley Mathieu Salzmann, Hongdong Li, and Mehrtash Harandi. Kernel methods on riemannian manifolds with gaussian rbf kernels. IEEE Transactions Pattern Analysis and Machine Intelligence, 37(12):2464 220244444477,-0 2015.
Mark Jenkinson, Christian F. Beckmann, Timothy E.J. Behrens, Mark W. Woolrich, and Stephen M.
Smith. Fsl. Neurolmage, 62(2):782 790, 2012. ISSN 1053-8119. doi: https://doi.or org/ nol6fj.merrimmage2001..0.00..
URL http::/wwwl sci wcw.enceirect.com wtt.rrect.tccccc.eeeeeeeeeeeee 20 YEARS OF IMRI.
arricleepp1i/11058111111111
David G Kendall. Shape manifolds, procrustean metrics, and complex projective spaces. Bulletin of the London Mathematical Society, 16(2):81-121, 1984.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. wtt.htttpp/aaxxv.oorr wttp:////rr///////o////o/b. abss//12, 6980
Diederík Kingma and Max Welling. 13Xiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Brian Kulis al. Metrie learning: A survey. Foundations and Trends in Machine Learning, 5(4): 287-364, 2013.
Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pp- 331-339. Elsevier, 1995.
Yonghyeon Lee, Jonghyuk Back, Young Min Kim, and Frank Chongwoo Park. Imat: The iterative medial axis transform. In Computer Graphics Forum, volume 40, pp. 162-181. Wiley Online Library, 2021a.
Yonghyeon Lee, Sangwoong Yoon, MinJun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. International Conference on Learning Representations, 2021b.
Yonghyeon Lee, Seungyeon Kim, Jinwon Choi, and Frank Park. A statistical manifold frame- work for point cloud data. In International Conference on Machine Learning, pp. 12378-12402. PMLR, 2022.
Bastian Leibe and Bernt Schiele. Analyzing appearance and contour based methods for object categorization. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pp. II-409, IEBE, 2003.
Kantilal Varichand Mardia. Statisfics directional data. Academic press, 2014.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Con- tinuous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. arXiv preprint arXiv:1310.454p 2013.
Maher Moakher. A differential geometric approach to the geometric mean symmetric positive- definite matrices. SIAM Journal Matrix Analysis and Applications, 26(3):735-747, 2005.
12

---

Published as a conference paper at ICLR 2023
Sameer A Nene, Shree K Nayar, and Hiroshi Murase. Columbia object image library (coil-20).
Technical Report CUCS-005-96, 1996a.
Sameer A Nene, Shree K Nayar, and Hiroshi Murase. Columbia object image library (coil-100),
Technical Report CUCS-006-96, 1996b.
Frank C Park and Roger W Brockett. Kinematic dexterity of robotic mechanisms. The International Journal of Robotics Research, 13(1):1-15, 1994.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W. W, 2017.
Xavier Pennec. Probabilities and statisties riemannian manifolds: Basic tools for geometric measurements. In NSIP, pp. 194-198. Citeseer, 1999.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543,
2014.
/ www. acl web. org/ http//atttooooooll111aa
Marc'aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann L. Cun. Efficient learning of sparse representations with an energy-baa based model. In Advances in Neural Information Process- ing Systems 19, pp. 1137-1144. MIT Press, 2007.
Marc' aurreio Ranzato, Y lan Boureau, and Yann L. Cun. Sparse feature learning for deep belief networks. In Advances in Neural Information Processing Systems 20, pp. 1185-1192. Curran Associates, Inc., 2008.
Luis A Pérez Rey, Vlado Menkovski, and Jacobus W Portegies. Diffusion variational autoencoders. arXiv preprint arXiv:1901.08991, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Con- ference on Machine Learning, volume 32, pp. 1278-1287 1286.128...6 PMLR, 2014.
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 645-660. Springer, 2011a.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 833-840, 2011b.
Hiroaki Sasaki, Aapo Hyvärinen, and Masashi Sugiyama. Clustering via mode seeking by direct estimation of the gradient of a log-density. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 19-34. Springer, 2014.
Ondrej Skopek, Octavian-Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. arXiv preprint arXiv:1911.0844 2019.
Justin Solomon, Leonidas Guibas, and Adrian Butscher. Dirichlet energy for analysis and synthesis of soft maps. In Computer Graphics Forum, volume 32, pp. 197-206. Wiley Online Library, 2013.
Julian Straub, Jason Chang, Oren Freifeld, and John Fisher IIII A dirichlet process mixture model for spherical data. In Artificial Intelligence and Statistics, 930-938. PMLR, 2015.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference Machine Learning, pp. 1096-1103. ACM, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of Machine Learning Research, 1 ((Dec):3371-344) 2010.
13

---

Published as a conference paper at ICLR 2023
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. arXiv preprint arXiv:1808. 10805, 2018.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 206-215,2011
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A MATHEMATICAL BACKGROUNDS
In this section, we briefly review some notions related to differentiable manifolds and Riemannian geometry. For further mathematical details on differentiable manifolds and Riemannian geometry. we refer the reader to [Boohby (1986) and Dubrovin et al. (1992).
Intuitively, an m-dimennsional differentiable manifold M space which is locally diffeomorphi to mn-dimensional Euclidean space. For every point p E M, there exists coordinate chart (U,*), where U an open subset of M containing p. and x is homeomorphismmfof U to an open subset of R". Applying to gives the coordinates of p. i.e., .r(p) (x (p). x=t (p" (p)) E R of a real-valued function on U, the i-th coordinate function. Here x is called the local coordinates; note that other choices of local coordinates are also possible (c.g., for the sphere, both spherical coordinates and stereographic projection correspond to different local coordinates of the sphere).
A differentiable manifold M endowed with Riemannian metric is called Riemannian manifold. The Riemannian metric is a function defined on the manifold M that assigns to each point e M bilinear mapping 4p T.MM x TpM
R, where TμuM denotes the tangent space M (x) ..... ",, the Riemannian metric can be expressed as Di= Ej=1 Diy (i)(r)dr'ds (dd'dda dx or ds² -E 1 (x)dd') dx'. Here Biz(x) is assumed to be smooth, i.e., infinitely differentiable, and its matrix representation G (gi)) E Rmxx is symmetric positive-definite. The Riemannian metric allows one to calculate lengths, angles, volumes, and even define distance metric on differentiable manifolds in intrinsic way, i.e., in way that is invariant to the choice of local coordinates
Using the local coordinates
x(Ii)
M
m
.x(0)
V
x(1)
x(1)
(a) A curve and a region on M
(b) Corresponding curve and region in local coordinates
Figure A Riemannian manifold M and its local coordinate æ.
The length of curve C (x(1) M | [0, 1}} is caleulated
Length(C) = V V(())G((()))()))) dt,
(10)
where a<() IR" is the local coordinate representation of x(!) (see Figure 7p. Given two fixed boundary points x(0), x(1) E M, the curves that minimize the length (10p are called the minimal geodesics, and the corresponding lengths the minimal geodesic distances.
The equations for geodesics are obtained as
m
pk dx dxl
0, k: = I=1,..., 1,..., m,,
(11)
di
dt dt
where D& denote the Christoffel symbols of the second kind in M, i.e.
Ogia
Ogu
Ögij
(12)
dx
0zd
and ght is the (k entry of G-1 E m
'Two manifolds said to be diffeomorphic if there exists differentiable mapping between the man- ifolds which invertible and its inverse is also differenniable. Such mapping is called diffeomorphism. AA continuous function is called homcomorphism if it is invertible, and its inverse is continuous.
See Appendix B. for discussions the coordinate transformations and the coordinate-ivvariacce.
15

---

Published as a conference paper at ICLR 2023
Consider a geodesic curve [0.1] + M emanating from a point x(0) E M with an initial velocity vector x(0)
Txm- as shown in Figure 8 It is known that such a geodesic is unique if exists (Boothby 1986). and denote by x(1) M the endpoint of the geodesic that propagates for a unit time. The mapping that maps the initial velocity vector
x(O)
x(1)
EXPs(o((V)
the point x(I) called the exponential map (xpx()) Txor-M + M. Note that the distance between x(0) and x(1) is the same as the norm of
M
This corresponds to generalization of propagating line vector space from a starting point along a vector.
Figure 8: A geodesic curve x [0, +
M emanating from x(0) E .M along
On Riemannian manifolds, there exists (natural) volume element induced from the Riemannian metric G(x) which expressed in local coordinates as /det (az) dz .......daa dx The volume of a compact subset V S M is then obtained by the following integral:
VE M.
Volume(V) = J Vdet G(x)) da! ... ...d..
(13)
where V denotes the domain of integration expressed local coordinates" (see Figure 7 The
integration of bounded and continuous function M - R on the integration domain V is also obtained using the volume element as follows:
J (x) G(x) dxl dam
(14)
B FURTHER EXPLANATIONS OF THE GEOMETRIC REGULARIZATION COMPONENTS
B.I CONTRACTIVE REGULARIZATION
To see why (4 is coordinate-invariant, observe that under a pair of local coordinate transformations
+ x' ((x) and y 2/ = (vy), G(x) = (115(1)) E mmxm (y) (b) E 8ax=
and J(x) ((25
or (1) E 2axm transform according to the following rules DDubovvin et al. 1992
() G' Φ- 3-GGT where
Rmxm, (ii) H H' 1 H
where
V
0y By
E R
(iii) J + J = JT where it can be verified that Tr (J HJG-1) remains
the same.
Also note that minimizing induces the contraction (shrinking without distortion) of the map- ping f. As extreme case without any boundary conditions constraints, trivial solutions are obtained as J = 0 or equivalently f= constant, which an extreme contraction. On the other hand, provided the boundary conditions constraints for f are well-specified, a useful physical analogy for minimizing 4 to imagine wrapping a curved object made of marble (N) by an elastic sheet (M); harmonic maps, which are extrema of (4), can be viewed as solutions corresponding to elastic equilibria (Eells & Sampson 1964).
B.2 SOME REMARKS ON NON-EUCLIDEAN LATENT SPACES
Recently, to better capture the structure of data distributions, there have been increasing works on autoencoders that deal with non-Euclidean latent spaces, e.g., hyperspheres (Davidson et al. 2018 Xu & Durrett 2018), hyperbolic spaces or Poincaré balls (Mathieu et al. 2019), their mixtures
WWe may also use V rather than V to denote the domain of integration for notational simplicity.
We can easily verify (i) and (ii) by observing the Riemannian metric should remain the same under the local coordinate transform as ds² [dz]] G[dæ) [dx'] G' (d(d',,, where [dr] [dr' ,.... dn and 8z
[dr'] = Idx". ........" dx""qT are related by [dæ] du [dx']] Φ "dx"]. We can verify (iii) by considering the chain rule 8(y'of)
T 2L
16

---

Published as a conference paper at ICLR 2023
(Skopek et al., 2019), submanifolds embedded in Euclidean spaces (Rey et al. 2019). In the case
of such non-Eonnnieee Euclidean latent spaces, by reflecting the pooddinateiivvarinn contractive regularization we formulate geometric contractive autoencoder (GCAE) follows:
discussed in Section 3.1
xist(r(x),x)² + +o² Tr ((ff)
T
Of dz
G (-)
min gogof
H(f(z))
p(z) dæ,
(15)
where the reconstruction map M + M, the encoder mapping : .M- + N. and the decoder mapping g : M are respectively expressed in local coordinates as r R IR , R TR
Rm
Training the GCAE may induce additional regularization effects that better capture data- concentrated regions or make the model more robust to noise, in addition to the effect of using non-Buclidean latent spaces. Confirming these experimentally left for future work.
As another choice of the Riemannian metric for latent space, we can consider the pullback of the data space metric via decoder mapping (Arvanitidis et al. 2018). It has been observed that applying this metric better characterize the data distances in latent space and provide more meaningful results in analyzing latent representations. Adopting this metric on the latent space reveals an interesting connection between the GCAE in (15) and GRCAE in This pullback metric is obtained as
(y) = (o)) G(g(v)) Bg
R"*", and we can observe that 15) and (5) become identical
when substituting this metric into
and considering the chain rule
Or Uz By
0g af
E <R
B.3 DATA CORRUPTION PROCESS
The corruption process ā of) in vector space is equivalent set 5 = for ((0, g) i.e., the endpoint of a line segment starting from x and extended according to the vector On the Riemannian manifolds, similar discussion is possible using the notion of the geodesic and the exponential map as discussed in Appendix A Therefore, we corrupt an input E M by applying the exponential map Exp, T.M + M tangent vector v T&M sampled from an isotropic mean multivariate Gaussian, i.e., v E1 + + ... + T Em for an orthonormal basis .i...... E,, for T and an m--imennional vector (u' ..... vTn) NO, o²I). By using an isotropic Gaussian, the corruption process x = Exp,(v) (v) does not depend on which orthonormal basis E1,..., F.m used.
C PROOF OF THEOREM
C.1 THE FIRST-ORDER NRCESSARY CONDITIONS FOR GRCAE
Lemma 1. Provided o² is small, the derivative of the log of the probability function Pg =
V det G(x) can be approximated for the geometric reconstruction contractive autoencoder as
p(x)
Olog Pa 1 0pg (x) (x) (x)-)
3p
r(x) - (r(x)=
0² + +0002,
(16)
Ox Pg Or p d.r
where T(x:) E Th is vector whose i-th component is given by 2 Tr (G 1 aG 8z
Proof. The first-order necessary conditions for 5 can be obtained from the following Euler- Lagrange equations
OC Fr
m
i
OC
0, 1,..., ,,,
(17)
Ozi
Drd Bard
where is the integrand (5) with an approximation on the squared geodesic distance to (r(x) x:" (rz)(r(z) - x) provided o² small [ and r² denotes the i-th coordinate representation of the
Thhe difference hetween optimal r(z:) and J. turns out be (o)2). can be verified that this approximation does not affect the final results, since the approximation error of the squared geodesic distance hecomes O(a").
17

---

Published as a conference paper at ICLR 2023
reconstruction function By applying L to the Euler-Lagrange equations,
17
results in
-' =z n/ (2,n, ,r"),
1..... m,
(18)
where r' denotes Or r" denotes the collection of 0²,' dxd for j = ...... m. and 1/' denotes a function of x. r. r', r" Assuming r. G, G-1 are smooth, and their higher-order derivatives are bounded, r¹ ---x' o² (2,,,, r"", 2 holds. By iterating the relation r' x² o 1/ (z, r,, ,r")
Oz
x' + O i... substituting this form of r² into s7' (r,,,,'', r"), the i-th component of r is obtained
for small o² as follows:
(1
0p p dzi
8G Ozi
)) +0((0))
pd x²
o²
Σg0
TTr
G-
(19)
1 8p Pg Oxd
o²
+ O(00),
(20)
where gdd denotes the (i entry of G Pg
G and the identity logg(de G) Tr (G- Oui
used to derive 20.. By gathering (20) for all i, the first-order necessary conditions can be rewritten
1 0pg (x) G(x)
(r(z)= - 0
r(() 02
(21)
Pg dx
C.2 THE FIRST-ORDER NECESSARY CONDITIONS FOR GDAE
To obtain the first-order necessary conditions for GDAE, the integrand of (6) is first represented in local coordinates x = (x',... xz") in Lemma 2
Lemma 2. Provided 0² is small, Eq(R(s) [dit(rr()),x) in (6 can be approximated in local coor-
dinates as
Fy((()) [dist(r(x),x))
(22)
Ör ax
T
Ör dr
= (r(xx) - TG())(((()))))) -z) +0²
Tr
G(r)
G G-1(x)
(
(8Pm') dr?
x²) gij Tr
6-1
Or Ord Or gh - gks 4Σ (r' - x²) (Fisk drl Ox n=
) + +
I
Oz
where T is the i-th component of r, 9ij and 9²و respectively denote the (i,j) entry of and G-1, The Taa and 1l respectively denote the Christoffel symbol of the first and second kind M.
Proof. We represent M and E M in local coordinates as x E R and ã E m.m respectively. Furthermore, at x, consider a nonlinear function á : IR" + R that maps the representations in local coordinates of the points near x to those in the normal coordinates; from these settings, Φ(x) holds and let î = o(²). Denote by To R + R the reconstruction function represented in the normal coordinates. (The functions r and ru are then related by r, rū) ((r())), and (0) = ((((x)) holds.)
Since the distance between M and r(x) M corresponds to the standard vector norm of the representation of r(x) in normal coordinates at x, dist(r(X), = ||r,(")](" holds. Near the origin
The explicit form of n² (xt, r,,,, r"rr") is obtained as th Bgir)as ar
Ejaia 9¹¹ (CE,
Ba(r)aa a dr De dx g + g(r)s
zz 3rd Oz +
kd 1. dp 0z
)) where ge and g(r) respectively denote the (i, entry of G-1 and G(r(z)).
dzt
18

---

Published as a conference paper at ICLR 2023
of the normal coordinates x, r Rm
R (the i-th component of R
R") admits
following Taylor's 's expansion:
dr 8u
(1) r (0)
à
à
(23)
2!
Then (a)ll can be expressed as follows:
|nu(())1 Σ (1)
(24)
dra Ou
Oru
Öru Ou
= =||ra(0)|12 - 2r)( -rr())))))(())))))))
ū + ū
ū
(25)
du
Σ (0) T
+++... +....
where the terms of the order (with respect to ūi higher than three omitted in 25 Consider the ex- pectation with respect to ~ (0,, o² I) I) of (25) (this corresponds to the expectation of dist(r(ī), x)² with respect to q(x(x)). Provided a² is small, obtain
[x((()) (ditt(r()). x = (i)) [[lra(a)](]
(26)
( (ora) Ou
Oru
( ( du
)) + +(O0),
= =Ilra(0)|13 + o² Tr
+
(0) Tr
du
where q(ū) denotes the noise density for ~ N0, I21), and the terms with an order higher than a² are assumed be negligible.
We now represent (26) in local coordinates. For this purpose, the following equations are useful:
Ose
(0) (r(x))
dx (r(z)
=x),
(27)
a (r(z))
Jist 2r Ox! Ör Ox Ous
(28)
Ou
Ou
02r
m
m
m
O5'
Or Dr Oz Ox
(29)
Fu Fuk
Or Oad Oa Oud Du
02d
Ox Ox
Oo Ord 8x
+
Or Oxpdatt dus Our
Orl Oun Oud Ou
1=1
pm
q=1
1=1 n=l
m
Oes Oøs
9ik
Σ
Oxi Oxk
(30)
Dan' Ox Ju Oui"
glk
(31)
Ori
8
Pj:kl
(32)
Owi Ox
1=1
m m
m
m
P
jkg
85k
(33)
Oui Juk
j=1k=1
1=1
8z where Dwt otherwise in 33 Here, the higher-order terms in 29) are obtained from the chain rule. We can derive (30 33) using the properties of the normal coordinates. By applying (73) to (26) and rearranging the terms, we obtain the result in
is the (i.j) entry of
R Rmxm and 85 = for j = k and 85k
Bar
27 can be neglected for small o², and 28
19

---

Published as a conference paper at ICLR 2023
We now provide the first-order necessary conditions for (6) using Lemma Lemma 3. Provided a² is small, the derivative of the log of the probability function Pa(x) be approximated for the geometric denoising autoencoder p(x)
V det G(z)
log Pa (x)
1 0p Ōr Pg dr
xx) = 16p (x) r(x) 0r
(1(0) -
(((x)
=
+ +0(02)
(34)
0²
where ((x) E I is vector whose i-th componen! is given by 1 Tr (G-1
DG Ux
Proof. The first-order necessary conditions for are obtained from the following Euler-Lagrange equations:
OC
V11
a
OC
m
O
OC
0, 1 1 m, (35)
Oxl
Ort Ouet
Oas Ozc
8 Om Uz
j=1
j,k=1
where L denotes of the form T o² (0, (2,,,, r, rr") for i 1, 1,..... m. where go is a function of 2; rrr r
22
approximated according to Lemma Applying to (35) results in equation
Assuming r, G. G- o are smooth and their higher-order derivatives are bounded, we obtain an equation identical to 200) for small o² by iterating r' = x + ppppp (x, (x, = x² (a²))
Hence the first-order necessary conditions are obtained (21
C.3 PROOF THEOREM L
Theorem 2. Provided o² is small, the derivative of the log of the probability function Pg(x))
p(x) can be approximated for both the GRCAE and GDAE as
dee G(z)
OlogP,(z)
1 0pg (x) Pg dr
1 0p (x) - I(x) G(x)
r(z) x
+ +0(0²)
(36)
Or
p dx
where 1'(x) E R is vector whose i-th component is given by 1 Tr (G-1
DG 0z
Proof. Collecting Lemma | from Appendix C.1 and Lemma 3 from Appendix proof.
completes the
D IMPLEMENTATION OF GEOMETRICALLY REGULARIZED AUTOENCODERS
In Appendix D.I D.2 D.3 and D.4 provide the implementation details of the geometrically regularized autoencoders for S P(n). N3), and point cloud data considered in our experiments, respectively. We also provide some ideas to deal with the case of manifolds that require multiple coordinate charts in Appendix D.5) We then discuss the case when the data space Riemannian metric not known priori in Appendix D.6
D.1
S" DATA
D.1.1 DATA CORRUPTION PROCESS
Given + 1-dimensional vector with the unit norm E S" Ra+1 and tangent vector
v E T. C [R*+1 the exponential map Exp, T,S" + S" is defined as follows:
sin ||v|| Exp,(v) cos |M||+x+ x |
(37)
'The explicit form of yp (z, r") is obtained as
Σg dutr)aa d. o.
dri dut det ght + jun ban Bet ght
dr 8r
9ya ass k19 kg
0. 1
+gja oe
(g( grrght M + (Ep(ro x I "h 1 (r )93a
Theg ())
g²
(gsa (r" - >x"9"" p))) where g(r) is the (i, entry of G(r(z)).
2 Bakaal
20

---

Published as a conference paper at ICLR 2023
where ||
is the Euclidean norm.
For the data compptiin process, we first sample an n + 1-dimensional vector N0, 0²1) and project onto T,S" (with the origin identified to that of the ambient Euclidean space) xx )e. As discussed in Section 3.2 the input x is then corrupted to x = (v) according to (37
D.1.2 THE RECONSTRUCTION ERROR
For reconstruction function r S + S", the reconstruction error is defined as follows:
dist(r(8), x)² = arccos(r(x) xx²².
(38)
D.1.3 THE RECONSTRUCTION FUNCTION
We implement regularized autoencoder for S" mapping S - S" The input and output of the mapping are n + I-dimensional vectors with the unit norm. The reconstruction function r is modeled by neural network with two hidden layers, with the hyperbolic tangent (Tanh) activation function as follows:
r(x) = Proj(x + Wah2 - ba), h; Tanh(W,h4-1 + b.),
(39)
=
(40)
where h1, hgg Rdd denote the hidden variables, hg is S" C R=+ is the function to project vector onto a hypersphere by dividing the input vector by its norm, and W,, 6; for i 1, respectively denote the matrix and vector parameters with sizes defined accordingly as above. We set the hidden variable dimensionality d; to 1,000 in all the experiments.
be S IR++1 Proj : Ra+1
D.1.4 CONTRACTIVE REGULARIZATION
For the case S", the contractive term of 5 (without the p(z) term) can be computed using r as Tr
(T-xx)))
(T - xxT)). where 8x E R 1=+1xa+1 and x E S Ra+1
0x
D.1.5 AN EMPIRICAL ANALYSIS OF COMPUTATIONAL TIME
Training geometrically regularized autoencoders requires more computations than training vector space autoencoders since it needs to calculate the geodesic distance, exponential map, and geometric contractive term. We have experimentally measured the computational times for the models applying each geometric component explained above to the autoencoder r M - M. We have used the Pytorch library (Paszke et al. 2017) and have utilized NVIDIA Tesla V100 GPU with Intel Xeon ES-2698 v4 2.2 GHz (20-Core) CPU (also for most of the other experiments).
We report in Table 6 the computational times spent for a gradient update iteration for each model with batch size of T0,000. the table, AE, GAF, GAE + G., GAE+ E., and GAE + C. respectively represent the vanilla autoencoder (of the same input and output dimensions with other models), an autoencoder reflecting the non-Euclidean input and output structure, a model applying the geodesic distance the reconstruction eтror for GAE, a model applying the exponential map in data cor- ruption process for GAE, and a model applying the geometric regularization term for GAE. For comparison, we also consider GDAE and GRCAE. To check the time for various data dimensional- ity, we consider n = 2, 6, 10, 25, 50, 100 for S" data generated as explained in Appendix [F4 From observe that applying our geometric components
the table, S data reasonably well, except for the geometric contractive term, which requires the derivative of the Jacobian & 1=+1xa+1 during the computation of gradients for an update.
be scaled to high-dimensional
D.2 P(n) DATA
D.2.1 MATRIX EXPONENTIAL AND MATRIX LOGARITHM
We first provide the closed-fore expressions the matrix exponential on S(n), the space of n symmetric matrices, and matrix logarithm P(n). Given eigendecomposition of symmetric
21

---

Published as a conference paper at ICLR 2023
Table The computational times (in milliseconds) spent for gradient update iteration for autoen- coders applying different geometric components for S" data.
GAE+E
GDAB
GRCAE
2
5.30
7.53
35.7
10.7
39.8
5.46
5.32
7.59
6.13
68.7
10.9
74.0
10
5.56
5.47
7.44
6.23
103
10.9
110
25
5,46
5.53
7.71
6.31
234
11.0
248
50
5.68
5.58
7.79
6.45
462
10.7
491
11.3
matrix A RDRT E S(n), with R O(n) as the eigenvector matrix and D diag(d)....,, dz)) the matrix of corresponding eigenvalues, the matrix exponential is obtained
Exp(4) R Exp(D) R.,
(41)
where Exp(D) diag(exp(d1)) ..... exp(dx)). Given eigendecomposition of symmetric positive-definite matrix A RDR E P(n) similarly to the above, the matrix logarithm is ob- tained as
Log(A) R Log(D) RT,
(42)
where LDg(D) = (lag((og(d1).... (d)........... log(d,,))
D.2.2 DATA CORRUPTION PROCESS
Given an input P E P(n), we first sample m(x+1) -dimensional vector N(0, o²T). As discussed 2 in Section 3.2 the input P is then corrupted to P P(n) as follows:
p= p Pxp ((+) p!,
(43)
where pł RS$ RT for an eigendecomposition of P RSRT, and the bracket operator is E "R-" as (le))ss = arjes for S j and
defined for (Ei1,..., Eimi 622, 62n, Enn) ([e)); ((t));a for > j. with 0ij = if = and
v otherwise.
D.2.3 THE RECONSTRUCTION ERROR
For reconstruction function P(n) P(n), the reconstruction error is defined from the affine- invariant Riemannian metric as follows (Fletcher Joshi 2007]:
P)) Log (P (p- tr(p)p
(44)
where p-1 = RS-ERT for eigendecomposition of P = RSR Log P(n) + S(n) is the matrix logarithm, and || is the matrix Frobenius norm.
D.2.4 THE RECONSTRUCTION FUNCTION
The reconstruction functions P(n) + P(n) for GDAE and GRCAE should consider the special structure of P(n). By defining function P(n)) + S(n), the reconstruction function at P E P(n) modeled
r(P) Ex
(45)
Hence training reduces to training unconstrained v (when considering only the lower- or upper- triangular part), and denote by P(n) + dimensional vector representation of the lower- (or upper-) triangular part of the output for v.
m(++t) the function that returns the n(++1)
We model by neural networks with two hidden layers, with the hyperbolic tangent (Tanh) activation function as follows:
c(P) Wzhz + b3,
(46)
h; Tanh(W + bi).
i == 1,2,
(47)
22

---

Published as a conference paper at ICLR 2023
where h1 h2 E Rd denote the hidden variables, hg R m(mtt1) to the vector representation of the lower- (or upper-) triangular part of Log(P) with Log P(n) - S(n) the matrix logarithm and W,, b, for accordingly as above. We set the dimensionality of the hidden variables d,, to 1,000 for both GDAE and GRCAE in all the experiments.
2,3 respectively denote the matrix and vector parameters with sizes defindd
D.2.5 CONTRACTIVE REGULARIZATION
For the case P(n), we can consider the upper- (or lower-) triangular part of the matrix represen- tations as its local coordinates, i.e., r(x) r(x) and 2, and calculate the contractive term of (5) straight- forwarll.
D.2.6 AN EMPIRICAL ANALYSIS COMPUTATIONAL TIME
Similarly to Appendix D.I.3 we have experimentally measured the computational times for each autoencoder model for P(n) data. Note that we can calculate the matrix exponential and logarithm faster and numerically more stably using the approximations from Taylor's expansion by assuming small o² (hence inputs becoming near zero and identity for the matrix exponential and logarithm, respectively, in (4 We also precompute and use some quantities frequently appears in 443
45 such as P1 and P
We consider two different models with different degrees of approximation for GDAE, namely (45) the second-order Taylor's expansion on the matrix exponentil and logarithm, i.e., Exp(v) T+ v2 and Log( M) (M - [) (M 7)2
GDAE-vl and GDAE-v2. GDAE-vl applies in 43
GDAE-v2 additionally applies the first-order Taylor's expansion near P for P and Log( P) required to calculate r(P) based on (55) so that can avoid performing additional eigenvalue decompositions during training.
We report in Table the computational times for a gradient update iteration for each model with batch size of 5,000. To check the time for various data dimensionality, we consider = 2, 4, 7, 10, 14 for P(n) data (with dimensionality d = m(++1) 3, 6, 10, 28, 55, 105) gen- erated explained in Appendix F.I. From the table, we can observe that applying our geometric components be scaled to high-dimensional P(n) data at rate slower than the linear rate to d, except for the geometric contractive term of which the computational time increases at a nearly quadratic rate to d when applied. Also note that even if the GDAE-v2 shows faster computational time when n = 2,3, it becomes slower than GDAE-v1 as the data dimensionality increases due to the heavy matrix multiplications required in the approximation. We utilize GDAE-v1 for other experiments using P(n) data.
Table The computational times (in milliseconds) spent for a gradient update iteration for autoen- coders applying different geometric components for P(n) data.
GAE
GAE G.
GAE + E.
GAE C.
GDAE-vI
GDAE-v2
GRCAE
3
3,02
5.10
4.66
14.6
40.1
22.1
11.2
45.0
10
3.29
8.96
8.85
25.1
213
38.1
38.8
220
28
3.32
19.8
19.1
1111
52.5
1126
55
3.34
39.6
39.1
63.2
4573
79.2
164
4594
105
3,73
62.8
62.8
98.6
28949
111
262
28776
D.3 N(3) DATA
In training GDAE for diffusion tensor imaging (DTI) data in Section 4.4 the structure of the manifold N(3) (using the Fisher information Riemannian metric) should be dealt with properly. Here, we provide technical details for the modeling and training of the reconstruction function
r: N(3) - N(3).
It has empirically performed hetter the representation obtained from Log(P) rather than
23

---

Published as a conference paper at ICLR 2023
D.3.1 THE FISHER INFORMATION RIEMANNIAN METRIC
Denoting by (u, 2) normal distribution in R with mean μ and covariance >, the space of n-dimensional normal distributions N(n) is defined as follows:
N(n) (N 2) I R", E P(n)).
(48)
N(n) becomes a differentiable manifold with dimension n + In(n + 1). By using the Fisher infor- mation Riemannian metric with local coordinates (J,, E) (for Σ., only the lower- upper-triangular
is needed part), the inner product of two tangent vectors V = (V Vy),, W = (W,, Wx) for
Vuo W E _2" and Ve, Wy E S(z) at point N (v, E N(n) defined as follows:
(V. W) = V + TV
(49)
It can be easily verified that the Fisher information metric reduces to the affine-invariant metric when the R part of the tangent vectors zero; the geodesic distance between two normal distributions with identical means is then obtained in closed-form from the affine-invarian distance between the two covariance matrices. However, in general, the distance between two normal distributions cannot be obtained in closed-form; numerical algorithm to find the minimal geodesic according to this metric proposed Han & Park (2014), along with some physical insights of adopting the Fisher information Riemannian metric for DTI analysis.
D.3.2 DATA CORRUPTION PROCESS
Given an input voxel data N = (x, Px E N(3), we first sample ep ~ N(0, and six- dimensional vector &c ~ N(00,21) The input voxel P) E N(3) is then corrupted to (i, P) E N(3) as follows:
5 I + Cp.
(50)
p P Exp ([eo)) Pł.
(51)
where pł RRSRR T for an eigendecomposition of P RSR and the bracket operator [ is
defined for = (611, 612, 613, 622, <23, <33) E R as (()))s = aijoi for S and ([{))s = ((c));
for i > j. with aij = /2 if of the data corruption process (x[[x) (for small a²) discussed in Section In the experiments, we have further applied the first-order Taylor's expansion on the matrix exponential in (51).
and otherwise. Note that this corresponds to an approximation
D.3.3 THE RECONSTRUCTION ERROR
Denote the reconstruction function by (rp. Fe), where T, location part and re N(3) + P(3) the value part of the reconstruction function, respectively (we discuss their exact modeling in the later section). Assuming o² is small, the reconstruction error between (x. P) E N(3) and (1,( P), P)) E N(3) is approximated according to the inner product the Fisher information metric defined in (49) as follows:
N(3) R33 corresponds to the
dist ( P), (ū P)), (x,
(52)
z (rp(i, P) - x)) p- (r,
I) + 2 ()P--
where p-t = RS- - RT for an eigendecomposition of P T.ST.
D.3.4 THE RECONSTRUCTION FUNCTION
Next, we model the reconstruction function = (rp, ra) on N(3). By defining function v N(3) S(3), the voxel value part r, of the reconstruction function at (ar, P) EN(3) is modeled as
->
(r, P) PtExp (v(a, P)) P3.
(53)
In this way, training Ie reduces to training unconstrained (when considering only the lower- or upper-triangular part), and we denote by e N(3) + R the function that returns the six-dimensional vector representation of the lower- (or upper-) triangular part of the output for
24

---

Published as a conference paper at ICLR 2023
We model Ty and by neural networks with two hidden layers, with the hyperbolic tangent (Tanh) activation function as follows:
(x, P) I + Wihh b3.
(54)
o0z, P) Wah2 + ba,
(55)
ly Tanh(W b:), i= 1,2,
(56)
where h1, h2 E Rd denote the hidden variables, and ho set to be (x, p) R2 with p E R
as the vector representation of the lower- (or upper-) triangular part of P. Moreover, W, b, for
2, respectively denote the matrix and vector parameters with sizes defined accordingly as above. We set the dimensionality of the hidden variables dn to 1,000.
D.4 POINT CLOUD DATA
D.4.1 THE FISIER INFORMATION RIEMANNIAN METRIC FOR POINT CLOUD DATA
A statistical manifold framework has been suggested to deal with point cloud data in Lee et al. (202).. Briefly speaking, they deem each point a point cloud as a sample from a specific paramet- ric probability density model and identify the data to the density. The Fisher information metric for these density models can then serve as natural Riemannian metric in the point cloud data space.
For point cloud data in Rº represented X {=1,... ................................ ........, Ix E R?), by using X itself as the parameter, following parametric probability density model considered:
p(:; X)
1 A
Σ K (x--- (x))).
(57)
where K RDD - is a kernel function satisfying Sgo ((u)du = 1, and Σ E RDxD is symmetric positive-definite bandwidth matrix. In the experiments, we choose the Gaussian kernel function K (u) 1/ (2T) exp(-u" u/2) with = of following Lee et al. 2022).
With such a choice of density model, the Fisher information metric G E RR is obtained as
Gijka = p(r; X) Olog x) 1logg(1;; X)
dr
(58)
0x
0X
for i,k 1,...,W and j,l L,... D. where Giju is the ((i 1)D + 1)D + entry of G and X is the j-th entry of v.. We approximate the integration in (58) finite sum of the integrands (with p(x; X) excluded) over the data points T1, Tn. We refer the reader to Lee et al. 2022) for more details.
D.4.2 DATA CORRUPTION PROCESS
Since applying the exponential map based the Riemannian metric in (58) is computationally very expensive, we resort to an approximation for the data corruption process. Given a vector representation of the point cloud data x = (1 (=I_....,, ".,,.,,,,,). E RnD we sample a random vector of the same size OT), and corrupt the data as x + VG-L, where G-1 E RaDxxD the inverse of the Riemannian metric G in (58) We further approximate G' to be diagonal matrix by considering only the diagonal elements of 58 that VG-T is computationally tractable.
D.4.3 THE RECONSTRUCTION ERROR
For reconstruction function r R - D consider point cloud X (x In x2 RD) and its reconstructed version r( r(X) Y (v1...., Vn y: R D). We define the reconstruc- tion error by slightly modifying the Chamfer distance in Yang et al. (2018) as follows (Lee et al.] 2022
Reconstruction error
Σ
min | y/ + Y
Σ
min || vll2.
(59)
X
BEY
ZEX
BEX
yEY
where |X] denotes the number of elements in the set X.
25

---

Published as a conference paper at ICLR 2023
Note that using the reconstruction enror in (59) does not perfectly correspond to original defini- tion of GDAE in (6). which uses the geodesic distance as the reconstruction error. However, with a slight abuse of notation, still use the term GDAE for the method minimizing the reconstruction error defined in (59
D.4.4 THE RECONSTRUCTION FUNCTION
We use exactly the same point cloud autoencoder with the reconstruction function r R RTxD (i.e., the composition of the encoder f RaxD Rd and the decoder g RRd - TR
adopted from the FeNet in Yang et al. (2018). We set the latent space dimensionality d as 512.
D.5 A REMARK ON IMPLEMENTATION FOR MANIFOLDS THAT REQUIRE MULTIPLE COORDINATE CHARTS
Since the manifolds mainly considered in the experiments, e.g., S P(n), N(x), and the statistical manifold for point cloud data, could be embedded in Euclidean spaces, we have directly used the manifold elements embedded in Euclidean spaces the form of input and output for r M - M.
If this is not the case, c.g., for the abstract manifolds, we can implement the mappings : R R represented in local coordinates. When multiple charts {(U1, 21), ..... ... (Uc, >c)} required one available approach would be to implement mappings for each chart as separate neural networks and train each mapping by minimizing objective functions weighted by appropriate weight functions. In the case of GDAE, for example, we can solve C optimization problems as follows:
min / Egts s) (X), (((())))))))((()(((((((())) i i=1,...,C,
(60)
where C is the number of charts, r; Imm + R. the mapping for the i-th chart (this ri : U; + M represented in local coordinates), and fi M + I220 is the weight function for the i-th chart satisfying f((x) = for & U, e.g., determined based on partitions of unity (satisfying that, for all x M,, there is neighborhood of where all but finite number of fz()s
zero, and Ei 1). The final reconstruction results for x E MM can be obtained by geometrically averaging (after appropriate coordinate transformations) the outputs (x) from each mapping with corresponding weights (xx). Note that there is no guarantee that (U;)) S (U,))) (or r; (U;) S U,) would always hold. That is, (x(x) may be outside of U, hence (x) and x may not be the same coordinate chart for some U.. For small o², such phenomenon (if it happens) would mostly occur at E U, near the boundary of U, since T; is trained so that ry rx()) is close enough to x, i.e., rı(x)) - x = O(a²), according to Theorem [1] We eliminate any undesirable effect that this phenomenon may have on the optimization of the objective function (60) or on the weighted average of the final results by choosing weight function f; that has the value of zero in sufficiently wide region within U; that includes the boundary of U, As a side effect, this choice would also assign relatively large weights for the inner regions of U, far from the boundary of U, hence inducing larger regularization effect (c.g., denoising or contraction) in those regions. Therefore, this may lead to learning the reconstruction function direct toward the inner regions of U,, which helps prevent the range of (or r,) from going outside of (U,) (or U,).
D.6 A REMARK ON THE CASE WHEN THE DATA SPACE RIEMANNIAN METRIC IS NOT KNOWN a priori
When the Riemannian metric for data space D is not known priori, we can construct the Rie- mannian metric by resorting to metric learning techniques that either utilize some supervision on the desired distance for a given task or reflect some intuition about the data manifold. For example, in Hauberg et al. 2012). given a set of centers {CI...., .........o CK) J. C2 E RD, k = I,..., K and a set of metrics G Gk]. G3 E IR DxD k 1,...,K K corresponding to each center, a smoothly
10Her U. is open subset of M, and x; is a homeomorphism of U, to open subset of Rm for i 1. C. The collection (U1,-.., Uc} covers M, and for all i and j. the transition map x; o x.j is smooth.
26

---

Published as a conference paper at ICLR 2023
varying Riemannian metric G RR D>D
P is constructed by interpolating the metrics as follows:
G(z) Σ wk Ga)
(61)
(x) where wa (2) E (.))) parameter. Here
(x(x)) expl- 2% (a c&) Gx(a -x)), and 0 bandwidth obtain the set of metrics {GI..... Gx} from different metric learning
methods, which learn task-specific distance metrics supervised manner (Kulis et al. (013).
In the unsupervised case, based on the intuition that the shortest path in data space should be near the data manifold, not the region where data are sparse, the Riemannian metric G RR" + RR Dx can be constructed follows:
G(x) (a (a(a)))) + +c)=1.1
(62)
where h 1DD 1200 h(x)) when near the data manifold and h(x) otherwise, and a, > 0 are scalars to control the lower and upper bound of the metric, respectively. (Some methods to construct smoothly varying h(x) are provided in Arvanitidis al. (2021))
Consider formulating geometrically regularized autoencoders that reflect the Riemannian metrics constructed above. The calculations of the geodesics and exponential maps these cases usually require numerical solvers (Hauberg et al. 2012). Therefore, efficient training of these autoencoders would require some methods to approximately apply the geometric components, detailed discus- sion of which is left for future work.
E DETAILS FOR EXPERIMENTS IN THE INTRODUCTION
For the example provided in the introduction, train the RCAE and the GRCAE (modeled as described in Appendix D.I on spherical data sampled from the von Mises-Fisher (vMF) dis- tribution with the concentration parameter of 10. To see the trained results depend on the consider five differ-
30
choice of coordinates, ent spherical coordinate representations of the data for RCAE. We obtain these representations by representing the data with respect to differ- choices of ambient space reference frames shown in Figure 9 (a) and converting them to spherical coordinate representations. We then train the RCAEs using each representation and compare in the ambient space the reconstruc- tion directions obtained from each RCAE[T For the GRCAES using the ambient space rep- resentations, we train the models using data represented in five different reference frames (the same those considered for RCAE) and compare the reconstruction directions from each model after transforming them to an identical ref- erence frame.
Crigiral cata Aug merter duta
33
n/4
(a) Reference frames
(b) example of orig- inal and augmented data in spherical coordinates
considered in the experi- ments
Figure Additional figures for experiments the introduction. In (a), red, green, and blue lines rep- resent each reference frame's 's X-, Y-, and Z-axis, respectively. Note that the reference frames are translated to corresponding spherical coordinate origins for better visualization.
In Figure (b)- we plot the reconstruction directions for a subsample of the data in Figure and provide magnified view near the zenith for better visualization.
"To alleviate any topological issues arising from using spherical coordinate representations (x x²) which parametrizes the three-dimensional points as (cos(æ*) ), sin (2 ') cos()²), sin(x*) sin(²²)), e.g., the fact that, for points with the x² value near -T, nearby points the sphere can be mapped from two distant regions in spherical coordinates, augment data obtained by adding subtracting 2nr from the x? values (as shown in Figure (b)) during training and only the original data when getting the reconstruction directions.
27

---

Published as a conference paper at ICLR 2023
F DETAILS FOR SECTION 4.1
F.1 SYNTHETIC DATA GENERATION
We use 10,000 data sampled from m mixtures of tangent space Gaussian on P(n) in Section 4.1
m[++1] is the For the i-th mixture, we set the mean as μ4 = Exp( <d di)g(e:)) E P(n), where d = 2 dimension of P(z), et R is a standard vector whose i-th element one, and Exp S(n) + P(n)
is the matrix exponential. The covariance of the i-th mixture is represented using an orthonormal basis of T P(n) as Σj diag(O1..... (.......000... .d) Rddd where 0R = 0.1 1+ (i--))n-- (--1)(i-2)
(for i = 1, , v1) and 0% = 0.01 otherwise, and the k-th basis of Tp. P(n) is defined using the bracket operator in (43) as μ4 [ex] 14 E S(n) with eg E IRd as a standard vector whose k-th element is one.
F.2 DEFINITION OF THE ESTIMATION ERROR
For given data set {x1,.... IN} J represented in local coordinates of P(n), the log-density gradient estimation error (Est. error) is evaluated as follows:
Est. error >
Ö log Pe Bx
(2x)
log Pe (z:) G Gx,(x.) 8x
(
log
(2,)
0 log (I.)) (63)
Bu
8x
est
lest
where
0 0log Pv
(I4) and
& log
(I,) respectively denote the estimated value and the ground truth
da
les
value for Blog Px(x) wy. Note that it is usually impossible calculate the estimation performance
63 Pee
based
since the ground truth value of gloggp(()) is generally not available. Instead, (63
82
can be reformulated not to require the plog term by using the integration by parts technique
Be
presented in Hyvärinen (2005). Ignoring the constant terms that do not depend on the estimated value,
63 can be rewritten as
Ei=1
Bloges Dix
(x:)G=²(ss)
elose
(x.) 22())) 2 1 Ex-1
Blog,
(2i)9)((x))) (64)
where I(x) E R" is a vector whose i-th component is given by [T (G 8G Note that this formulation turns out be equivalent the minimization objective of the R-LSLDG method Ashizawa et al. 2017). In the case of GDAE and GRCAE, further simplification is available for
Pigg 64: Σ, Ex Back P2
9thh
Tr Tr θr 1) holds from (7) Equation (64) can be used in
de
hyperparameter tuning to evaluate the performance of the estimation place of
F.3 TRAINING DAE, RCAE, GDAE, GRCAE, LSLDG, AND R-LSLDG
We model the reconstruction functions P(n) + P(t) for GDAE and GRCAE as explained in Appendix D.2 The reconstruction functions r : IRd - Rdd (d =(++1) for P(n)) for DAE and
RCAE are modeled by neural network with two hidden layers, with the hyperbolic tangent (Tanh) activation function as follows:
r(x) = + Wyhs + by.
(65)
h; T by). 1=1,,
(66)
where hi, E de denote the hidden variables, ho is set to be x E Rdd, and W, b, for i = 1,2,3 respectively denote the matrix and vector parameters with sizes defined accordingly as above. We set the dimensionality of the hidden variables dn to 1,000 for both DAE and RCAE.
We optimize the parameters train GDAE and GRCAE (or those in (65) (66) for (T) and (2 to train DAE and RCAE respectively).
(W1, by, Wz, 2, b2, W3. by 46).
for and (5) to respectively
We apply the Adam algorithm (Kingma & Ba 2015) using the Pytorch library (aaskee et al.) 2017) and update the parameters for 500,000 iterations. We use the batch size of 1,000 to train RCAE and
GRCAE and the batch size of 10,000 to train DAE and GDAE The learning rate starts at 2.5e-5 and is divided by ten after 250,000 iterations with weight decay parameter of 1e-12. We have applied the weight initialization scheme of Glorot & Bengio (2010). Dividing the initial W,, by by two helped to improve the estimation error for GDAH and GRCAE.
28

---

Published as a conference paper at ICLR 2023
In discretizing 1 Gb. and (5), the expectations with respect the data- data-generating probability density p(z) are approximated by a finite sum over the N training data points with equal weights N. Given an imput (or z)), the expectation with respect to the noise density ((xxx) in (6) for GDAE (or g(+[))) in (1) for DAF) is also approximated by a finite sum over finite samples of x (or ă) with equal weights.
In training the autoencoders, the noise parameter in the data corruption process or the weighting coefficient for the contractive term (o²) should be chosen carefully. In this experiment, is selected among a E {0.01, 0.025, 0.05 to reduce the modified estimation error in 64). which does not require the value of the estimation results from the autoencoders can vary over iterations due to the stochastic nature of the optimization algorithm, instances with the minimum value of (64) during the training process are taken to be the output for each run of autoencoder training.
8 Blog Pa(#) on randomly selected validation data set of sizes 20,000, Since dx
In LSLDG, the derivative of the log-probability density, i.e.. Blog is modeled weighted Ahr sum of some basis functions, with the weights optimized in a least-squares sense. The R- LSLDG method estimates the derivative of the log-probabilityy function, i.e., Σ Boog (x() 8x7
for i I,..., m in local coordinates, similar to LSLDG but using basis functions de- fined on the Riemannian manifold. For both LSLDG and R-LSLDG, we perform cross- validation for the hyperparameters À E (10- 10 1002, 15,10-., 1.5 100 10 10-0.5, 10 10 100 and 10-1, 10-2,10-1.3 10-1.5,
10-0.5 10") as described in Sasaki et al. (2014) and Ashizawa et al. 2017 The number of the basis functions is set to 500.
a E (10- 10-3 10-2.5 10-2,
The time spent training each model is shown in Table The autoencoders (DAE, RCAE, GDAE,, and GRCAE) take much longer time for training when compared to LSLDG and R-LSLDG, which iinear-i--parmmtter GRCAE (or RCAE) requires more computations than GDAE (or DAE) due the contractive terms containing the derivative of the reconstruction functions. Compared to DAE and RCAE, GDAE and GRCAE involve heavier computations such as the eigenvalue decomposition (EVD) to reflect the non-Euclidean geometry of the data. Efficient parallel computation for the eigenvalue decomposition has been available by using the torch_baccccce hacp__aath______e_e svd repository (MIT License) 13
are linear-in- parameter models and have closed-form solutions for the parameters. Furthermore,
Table 8: The time (in seconds) spent in training each model for P(n) data.
LSLDG
DAE
RCAE
R-LSLDDG
GDAE
GRCAE
3062
17382
126
20671
43254
3 17
4087
19622
306
22499
67580
25120
545
33166
116255
F,4 GEOMETRIC SCORE ESTIMATION OF SYNTHETIC HYPERSPHERICAL DATA
This section provides the result of the geometric score estimation for the data hypersphere S {p E g=+1 ||||| = 1} endowed with the Riemannian metric induced from the Euclidean metric on R=+1, We use 10,000 data generated by mapping samples from m Gaussian mixtures in T,S" via the exponential map Exp, TyS" + S", where set the mean as μ4 J R with e, En as standard vector whose i-th element one, and the covariance as Σ3 (0.1)2T T E Raxx so that the ground truth log-probability derivative values are obtainable.
(1,0,... 0) E R For the i-th mixture, we
We train the DAE, RCAE, GDAE, GRCAE, LSLDG, and R-LSLDG using the data. For the au- toencoder models described in Appendix D. (for GDAE and GRCAE) and Appendix F33 (for DAE and RCAE), we use the Adam optimizer (Kingma & Ba 2015) of the Pytorch library (Paszke al. 2017) to update the parameters for 500,000 iterations. We use the batch size of 1,000 to train RCAE and GRCAE and the batch size of 10,000 train DAE and GDAE. From the initialization scheme of Glorot & Bengio (2010), the initial Wi, by are further divided by 1-5 for the experiments. Other optimization and hyperparameter tuning conditions the same those explained in Appendix F3
The definitions of X and a can be found in Sasaki et al. 2014 and Ashizawa et al.
2017
URRL: http::/githbb..oom/KiiiiitttCttt
29

---

Published as a conference paper at ICLR 2023
For both LSLDG and R-LSLDG, perform cross-validation for the hyperparameters À
(10 10 2, 10 1.5 10 and σ E {1002, 2,10-, 1,101.66
0.5 10 } as described in Sasaki et al. 10 1.00-100 10 -0.5
(014 and Ashizawa et al. (2017). The number of the basis functions is set to 500. The time spent in training each model is shown in T
Table 9: The time (in seconds) spent in training each model for S" data.
LSLDG
DAB
RCAE
L.LDG
GDAE
GRCAE
17165
5236
21804
111
3472
20369
399
6590
25864
7696
The estimation errors for Blog Har obtained from each model are reported in Table 10 Similar to the results in Section 4.1 the GDAE and GRCAE perform much better than other methods (especially for higher dimensionality), while GRCAE gives the least estimation error in most cases.
Table 10: The estimation errors for log Ps. for tangent space Gaussian mixture data on S Bolds dzz
represent the best and comparable methods from the t-test with a significance level of 5%.
n
m
LSLDG
DAE
RCAE
R-LSLDG
GDAE
GRCAE
2
2
1.44 (0.07)
4.94 (0:25)
6,43 (2.05)
1.77 (0.09)
1.91 (0.18)
1.74 (0.15)
6
(10.3)
647 (266)
586 (93.9)
5.33 (0.11)
5.16
4.74 (.74(
6
677 (21.1)
1194 (369)
1175 (259)
(2.11.
14.2
14.1 (1.27)
10
2
782 (6.45)
630 (107)
802 (187)
14.4
8.52 (0.17)
7.58 (0.07)
10
(4.46)
739 (81.1)
42.7
21.3
20.7 (3.57)
10
10
1700 (133)
1380 (54.8)
1655 (82.2)
50.4 (0.27)
39.3 (0.95)
38.2 (1.72)
G DETAILS FOR SECTION 4
G.1 THE RIEMANNIAN LANGEVIN MONTE CARLO (RLMC) METHOO S
To apply the RLMC method for S we should appropriately discretize the Brownian motion drg on an n--iimeniioal manifold written as
9 Recall that, from (9)
G (x) log / det G(z) + ((o) di - (G-
G-((x) drB 2
(67)
8x
where du E R the Brownian motion in an n-dimensional vector space and (ix)) E IR. vector whose i-th component is given by Σ}= 0g daz
with g6d the (i.j) element of G-! (x) R
Consider spherical coordinate representations (x) ......"))) which parametrizes the points on S as x(x x." = (x ........,]], E R=+1 with x = cos(x"), x² = II;=1 sin=)) sin(w") cos(x") for i 2,..., and =+1 11 sin(x"). The Riemannian metric is then obtained as G(x)
diag(1 (1.922 Inn) E IR*x with [T)= sin? for .....
To formulate the RLMC algorithm for S", it is useful to represent the Brownian motion in (67 in the ambient space. Applying the Ito rule, we can obtain such a representation as follows:
Ox G-)(x)
2 log v det G(x) (x)
dxB
dt VG- + +2 E()ddt
(68)
Ox
dr
where
8x de
[(n+1)xx is the Jacobian of the parametrization x(x) with respect and E(x)
Ra+1 is vector whose i-th component given by Tr(vG-1" ap.'
2z VG-1) with
E Ruxn as
the Hessian of x with respect to x.
After a straightforward calculation, the Brownian motion in 68 reduces to
dxg = 2
xdt + Bdw,
(69)
30

---

Published as a conference paper at ICLR 2023
where B
R(++1xxn is matrix whose column vectors form an orthonormal basis Note here that the drift term (-2xdt) is orthogonal to the tangent space T.S".
0x Ba (z-
of S
Each Langevin diffusion step of the RLMC method for S" can then be performed by
Xj+1 Expx, ( s(xg) At -7- (1xxxxx)w)
(70)
where E S" is the point sampled at the j-th step, s(x;) ) E Sr is the estimated geometric score at xj, and W, E R=+1 is random vector sampled from N 7) with (I - x,x, )w; implying the vector obtained by projecting w, onto Tx,S² S" Note that 770) can be approximated up to the order
of Al to the update of Xj+1 Proj xj A (5s(x,) 5ンン + A/ (I xyx])w.) which
straightforwardly considers the Brownian motion in the ambient space in 669
The geometrie score in (70) can be estimated for point S" using the reconstruction function r: S + S" the GDAE trained from a given set training data as
s(x) = o² Log, (r(x)),
(71)
where the logarithm map Log, S" T,S", the inverse of the exponential map in (37. locally defined for an input S near as follows:
(y) = arccos(x (I xx T)y.
(72)
VI-(x (x-(x1) y)2
G.2 EXPERIMENTAL DETAILS
We consider four synthetic data sets S²: four blobs, two moons, s-curve, and circles, which are generated by the following procedure. First, we make two-dimensional data sets by using the python sklearn. dataset package with zero noise level, place them on the plane 2 1, and project them to the spherical surface S². Secondly, we add tangent Gaussian noises with standard deviations of 0.01 for two moons, s-curve, circles, and 0.05 for four blobs, and project those perturbed data to the sphere by Riemannian projections. The number of training data is 800, that of validation data is 200, and that of test data is 1000.
For autoencoder- 4-512-512-512-512-5111 models, we use the fully-connected neural network architecture that has layers with ReLU-ReLU-linen MeeL---inearRReLLURRR For S-Flow that uses the RealNVP model (Dinh et al., 2016), the depth eight and the length of the hidden feature is 512. For all cases, the learning rate is Te-3 and the weight decay parameter is 1e-12.
activation functions.
For nutoeeccoerrbbsed models, we search {0.01, 0.025, 0.05]. We use batch gradient descent, and the maximum training epoch set to 5000, As validation loss used to determine the best model during the training, we use the modified score estimation enror in (64) for autoencoder- based models and negative log-likelihood for S-Flow. In RLMC sampling of the autoencoder- based models, need determine the step size At in (70). The step size is searched over AL E {0.00001, 0.00005, 0.0001, 0.0005, 0.001}. We run the experiment five times with differ- random seeds. In the result table, the averages and standard errors of the best cases are reported, where for each run the best-model-resultinn hyperparameters (i.e., a and step size for autoencoder- based models) are selected by computing the MMD metries (Gretton al., 2012) between sam- pled points and the validation data sets. We compute the MMD metric with the exponential kernel k<x, = d2,, (a. (2 r)²) where dza (x. y) is the squared geodesic distance between data y,y E S and the bandwidth parameter 1 is defined as the mean of the 22,,4455-neare neighbor geodesic distances for all training data.
This he easily verified by the identity of B B
V
VG
since G(x)
T
Ohe
31

---

Published as a conference paper at ICLR 2023
H DETAILS FOR SECTION 4.3
H.1 DEFINITION OF THE CLUSTERING TASKS
We define the clusters by merging relevant topics among 20 groups the Newsgroup20 data set resulting in 'religion.' "computers," 'politics," "automobiles," "sports." and topic in 'science" (one among 'crypt,' "electronies," 'med,' and "space"). We define four clustering tasks by differing the topic in 'science."
H.2 TRAINING DAE, GDAE, LSLDG, AND R-LSLDG
The reconstruction function for GDAE is modeled as explained in Appendix D.I and DAE is mod- eled in the same way explained in Appendix E3
We then optimize the parameters 0 {W1, b1, W2, b2, W3. by for (6) and [1 to train GDAE and DAE, respectively. We the Adam optimizer (Kingma & Ba 2015) of the Pytorch library (Paszke et al. 2 and update the parameters for 500,000 iterations. The learning rate respectively starts at 2.5e-3 and 1c-3 for GDAE and DAE, and is divided by ten after 250,000 iterations. Other opti- mization conditions are kept the same as those in Appendix [33 and the corruption noise standard deviations used selected between E (0.025, 0.05) reduce the modified estimation error in 64) on the training data set.
For both LSLDG and R-LSLDG, we perform cross-validation for the hyperparameters À {10-8, 10-2, 1221,001:4 (00-1.,,,7 10-1] and a E {10-12, 10-1, 11000.5 100) as described in Sasaki et al. (2014) and AAshizawa et al. (2017). The number of the basis functions is set to 300. The time spent in training each model is shown in [I
Table 11: The time (in seconds) spent in training each model for 449 data.
LSLDG
LSLDG
DAE Amb
R-LSLDG
288
166
5006
5015
1435
6108
H.3 CLUSTERING COVARIANCE MATRIX DATA
In this section we provide a case study of the clustering of covariance matrix data. Following Jaya- sumana al. 2015)) we calculate covariance matrices from the features of image data and group the images based on the covariance matrices. For this case study, we consider the ETH-80 data se 16 with 3,240 images from eight categories (Leibe & Schiele, 2003). COIL-20 data se with 1,440 images from twenty categories (Nene et al. 1996a). and CO1L-100 data se with 7,200 images calculating the covariance matrix for each im-
from hundred categories (Nene et al.) 1996b). age, we use the feature (from every pixel of the image) consisting of {x, I, TT IT, where y are the horizontal and vertical pixel locations, / is the intensity of the pixel, and Lg and Ly are the horizontal and vertical gradients of the intensity, respectively! Hence the data in use are in P(5).
We train DAE, GDAE, LSLDG, and R-LSLDG using the covariance matrix data and perform clus- tering using the trained models similarly as explained in Section 4.3 For the GDAE and DAE mod- els respectively described in Appendix D.2 and Appendix F.3
the Adam optimizer (Kingmal & Ba 2015) of the Pytorch library (Paszke et al. (017)) to update the parameters for 400,000 iter- ations. The learning rate starts 1e-4 and is divided by ten after 200,000 iterations with a weight decay parameter of 1e-12.
The corruption noise standard deviations DAE and GDAE. To choose σ. we use a (randomly chosen) half of the data as a validation set to train the DAE and GDAE models and measure the clustering performance. The value that gives the
significantly affect the clustering performance in
URRL http:///qwone.com/ jason/2/Newsgrrupps/.
16RLL: https://gitubb..omKaaii-KOa
UURL:
18 "URL: www.:/cccolumbaaccul.uu ////s/ttaaa///ooiiiiiiiill Fach dimension of the features has been normalized before calculating the covariance matrices.
32

---

Published as a conference paper at ICLR 2023
best clustering performance among {0.025, 0.05, 0.2, is chosen, and the rest half of the data to train the models and report the clustering performance. Other optimization conditions are kept the same as those explained in Appendix F3
For both LSLDG and R-LSLDG, we perform cross-validation for the hyperparameters À E (10-3 10-2, 10-10--..---- 1.6 10 10-1) and {10-2 10-1.0, 10-1001110 10-0.5 100) as described in Sasaki al. (2044) and Ashizawa et al. (017). The number the basis functions is set to 100. The clustering is performed using the identical data used report the results from DAE and GDAE. The time spent in training each model is shown in Table T2
Table 12: The time (in seconds) spent in training each model for P(5) data.
LSLDG
DAE
R-L.SLGG
ETH-80
20
1184
63
14227
COIL-20
15
1221
13282
COIL-100
23
1178
99
14924
The averaged clustering performance for five runs of each method is presented in Table 13 For the ETH-80 data set, R-LSLDG and GDAE show better performance than others on average but without much significance. In the case of the COIL-20 data set, the clustering results from DAE show the best performance. The absence of a significant performance difference between DAE and GDAE in both cases may partially be due the less peculiarity in the covariance matrices, e.g., having extremely small or large eigenvalues, for the relatively small number of categories. (According to the affine-invariant metric, the geometry of the covariance matrices can become more relevant when the relative cigenvalues of the covariance matrices get bigger or smaller.) When the number of categories gets bigger as in the COIL- 100 data set, the results from GDAE show better performance with a large margin compared to DAE. Even though this case study does not show distinct tendency, clustering using GDAE performs well among the considered methods terms of the overall adjusted Rand index.
Table 13: The adjusted Rand index for the covariance matrix data P(5). Bolds represent the best and comparable methods from the 1-test with a significance level of 5%.
Data set
L.SLDG
DAF
R-I.SI.GG
GDAF
(0.039)
0.348 (0.016)
0.362 (0.035)
0.361 (0.022)
COTI
0,170 (0.030)
0.655 (0.012)
0.483 (0.030)
0.616 (0.016)
COIL-100
1.46e-3 (2.9e-4)
0.204 (0.009)
0.314 (0.036)
0.380 (0.015)
I DETAILS FOR SECTION 4.4
1.1 TRAINING OF THE RECONSTRUCTION FUNCTION
We model GDAE for N(3) explained in Appendix D.3.4 We then optimize the parameters 0 ( by W2, b2, W3. by. Wa. b1} for 6 to train GDAE. We model DAE as explained in Ap- pendix F3 with d 9, and the parameters in 65). (66) are optimized for (T). We use the Adam optimizer (Kingma & Ba 2015) of the Pytorch library (Paszke et al. 2017) and update the pa- rameters for 40,000 iterations. The learning rate starts at le-4 and is divided by 100 after 20,000 iterations. In discretizing ( the expectation with respect to the data-generating probability density p(x) and the noise density g (x|x) are dealt with the same way as for the case studies in Section 4.1 (explained in Appendix [33) and we use the approximated reconstruction error in (52). We use the corruption noise standard deviation o = 0.05 in training both DAE and GDAE.
I.2 A GDAE-BASE DTI FILTERING ALGORITHM
Based on the discussions in Section 4.4 we summarize a DTI filtering algorithm based on the GDAE in Algorithm| [ At each iteration, the input points updated toward the reconstructed points of GDAE along the geodesic with a predefined step size e. Assuming small step sizes, we approximate the update along the geodesic on N(3) by treating the updates of the location (in R3) and the value (in
33

---

Published as a conference paper at ICLR 2023
P(3)) separately. The update of the voxel values then reduces to update along the geodesic according
to the invariant Riemannian metric on P(3) (Fletcher & Joshi 2007). as expressed in line
of Algorithm|| | The functions Expo S(3) + P(3) and Logg P(3) S(3) (for Q P(3)) are
defined as follows:
Expo (A) = Q Exp (o 1 AQ- :) Q=
(73)
Log
(P) Q² Log @ PQ +) Q²,
(74)
where Q3 RS' RD and Q RT for eigendecomposition of Q RSR and the functions Exp(-) and (og(:) respectively denote the matrix exponential and logarithm which the closed-forms are provided in Appendix D.2.T When the iteration is terminated, assign the final voxel values to their original locations.
Algorithm DTI Filtering Algorithm Using GDAE
Given: Voxels of DTI datum (z,, P.) E N(3). = I....,N.
Input: Reconstruction function N(3) N(3) the GDAE trained from the given of voxels, step size the number of iterations Naa-
Initialize: (yt.1. Qi,1) = (I,, P.) for i 1, I. Iteration:
N do 2: Compute the reconstruction point (y:- Q:) = r(yi.j, Qiy)-
1: for = L,..... Nu ,.....N
3: Shift the current point according step size
4: Di,t+1 + Vis + <(y; Mr.j),
5: Qi,+1 + -ExpQu ( (LLogq,, (Q:)) (Qis+1 + Qi holds when == 1).
6: end for
7: Assign P., Qi,Nart1 for i L....,N.. N.
Output: Filtered voxels (I,, Pa,f) E EN3)), i= I....,N. N.
I.3 DETAILS FOR N(2) FILTERING EXPERIMENTS
In this experiment, we consider synthetic N(2) data set obtained by gathering the position (in R value (in P(2)) pairs at 1,600 grid points (i.e., N = 1, 600) of P(2) field on R For the P(2) field for clean data, we gather four distinct smoothly-varying P(2) fields as depicted in Figure 5 For noisy data, we corrupt each value in P(2) explained in 43 with noise standard deviations (or noise levels) in (0.02, 0.05, 0.2,
We model the neural networks for both DAE and GDAE as explained in Appendix pendix D.3.4 with appropriate modifications to deal with data in N(2), respectively, and train the reconstruction functions. We use the Adam optimizer (Kingma & Ba 2015) and update the pa- rameters for 20,000 iterations. The learning rate starts at le-4 and is divided by 100 after 10,000 iterations. The corruption noise standard deviation is chosen among a 0.025, 0.05, 0.13 to show the best modified estimation error in (64). For the trained models, we apply Algo- rithm [ (with appropriate modifications to deal with N(2) data) with N 300 and the step size e {0.01, 0.03, 0.1, 0.3, 1.0} chosen to achieve the minimum averaged square distances (according the
F3 and Ap-
metric) to the clean data.
For the MVKR method in Banerjee et al. (2015), the number of considered nearest neighbors k E {10, 20, 30, 40, 100} and the kernel bandwidth parameter o E {1, 10, 20, 30, 50} are chosen to achieve the minimum averaged square distances (according to the affine-invariane metric) to the clean data.
We report the R-squared score to compare the filtering results from each method in Table and the scores are calculated as follows:
Σ, dist(P, Pt.rue) R-squared score
(75)
E, dist( P,, Pi.true)
34

---

Among the Published as a conference paper at ICLR 2023
where dist(P1, P2) is the distance between P1. P3 P(2) obtained according to the affine- invariann metric, Pa, E P(2) is the i-th filtered value, Pi E P(2) is the i-th value of the clean data, and P P(2) is the intrinsic mean of (Pittrue:.... (N,te (Moakeer 2005
1.4 PREPROCESSING DIFFUSION TENSOR IMAGING (DTI) DATA
Data used in the preparation of the experiments performed in Section 4.4 were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database data set, we use the DTI data for a randomly chosen healthy subject.
20
FSL library (Jenkinson et al. 2012) is utilized to fit diffusion tensors from raw diffusion-weighted image data [2 The brain extraction tool (BET), eddy current correction function (eddy_correcty cccrrect),, a linear transformation tool to match the brain template (flirt), and dtifit module are used in the preprocessing of the DTI data. The region of interest (ROI) chosen as an axial slice that intersects with the corpus callosum region of the brain.
J DETAILS FOR SECTION 4.5
Overall experimental settings have been set similarly to those in Lee et al. (2022) We add noise to each point x E R3 in the point cloud of the data sets (ShapeNet, ModelNet1O, and ModelNet40) according + x m m. where is sampled from the Gaussian distribution with zero mean and different levels of standard deviation (0.01, 0.05, 0.1, and 0.2 of the diagonal length of the point cloud bounding box) as done in Lee al. a021a).
R S is uniformly sampled on the unit sphere and m
we use = = with op 0.8 MED, where MED denotes the median of the distances between the points in the point cloud and their nearest points. The mean values of MEDs each data set are 0.0320, 0.0364, 0.0442, and 0.579 for the cases of the noise levels 0.01, 0.05, 0.1, and 0.2, respectively.
For the parameters in the kernel functions 57
To train the autoencoders defined Appendix D.4, we use ADAM (Kingma & Ba 2015) with learning rate of le-4, betas of [0.9, 0.999], a weight decay of 1e-6, and a batch size of 16; the total
number of epochs is 500. The noise parameters a for DAE and GDAE are set to 0.1 and lc-4, respectively. The coefficient for the regularization term (defined in (13) of Lee et al. 2022 is set to À = 8000 for both the 'AE + R.' and 'GDAE + R.' methods Table 5
2hh ADNI was launched in 2003 as unblic-private partnership. led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can combined to measure the progression of mild cognitive impairment (MCT) and early Alzheimer's discase (AD).
The license information is available at URL:: http:///f...mrriooo..aaaa//i//if icence.
35


Reviews:
[{"review_id": "lZhUjjpIRX", "reviewer": "Reviewer_F6Np", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an autoencoder for data residing on a Riemannian manifold alongside a geometric regularize. The reconstruction trivially replaces Euclidean distances with geodesics ones, and the bulk of the paper is concerned with the regularization. I did not fully understand the regulerization scheme (details below), but it seems related to denoising and contractive autoencoders. Empirically, the method is shown to work well on low-dimensional data; it is unclear if the method will scale to high-dimensional manifolds, which arguably is where the dimensionality reduction associated with autoencoders is most valuable.", "summary_of_the_review": "A promising paper, that I did not fully understand. If I understand the work better during the rebuttal phase I expect to have a more positive vote.", "strength_and_weaknesses": "## Strengths\n* The paper approaches an important problem as regularization on manifolds is generally understudied.\n* The idea of extending denoising autoencoders to manifold data is neat as it is really simple to work with. I very much appreciate the practicalities of this idea.\n* The empirical study is quite broad as several data sets from different manifolds are considered (see also Weaknesses below).\n\n## Weaknesses\n* In general, I found the paper somewhat difficult to read. As such, the weaknesses listed here may very well be weaknesses of the reader rather than of the paper.\n* The opening motivation for the paper is that standard vector-based regularizers do not work on manifolds. While I generally believe this to be true, I found the demonstration thereof to be somewhat thin. I appreciate the idea of empirically demonstrating the claim, but I think this demonstration should be more elaborate.\n* I didn't understand the sentence (page 4), \"use the weighted volume element $\\rho(x) dx$ instead of $\\sqrt{\\det G} dx$.\" Does this mean that you ignore the volume element of the metric? That sounds dodgy.\n* I did not understand Eq. 5, in particular the trace-term. Is my reading correct that $\\left(\\frac{\\partial r}{\\partial x}\\right)^T G \\left(\\frac{\\partial r}{\\partial x}\\right)$ is a scalar, which can be moved outside the trace-function? I did not understand why the inverse metric appears in this expression. Perhaps it's obvious, but it wasn't to me.\n* I also had trouble understanding Eq 8. Why does this use the inverse metric? I would have intuitively guessed that it should be the metric (not its inverse). Can some explanation be provided?\n* The experiments generally focus on very low-dimensional manifolds (e.g. 3x3 symmetric positive definite matrices). I would expect that autoencoder-style models have most value on high-dimensional data. I suppose that the method has difficulty scaling to high-dimensional data; what is the main bottleneck?\n\n## Minor issues\n* The paper generally talks of Riemannian latent spaces, but, as far as I could tell, mostly Euclidean latent spaces are considered. Perhaps one should here call on one of the many papers which argue that autoencoder-style latent spaces should be viewed as Riemannian (through the pull-back metric). See e.g. https://arxiv.org/abs/1710.11379", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8D1i13TkEG", "reviewer": "Reviewer_hFuw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a regularization technique for auto-encoders, when the data is non-Euclidean. Conventional regularizers encourage small MSE reconstruction error when input is noised. In contrast, the proposed method replaces MSE reconstruction error with geodesic distances along the data manifold. Empirical study shows that the method is better at estimating geometric score, and could lead to more useful learned features.", "summary_of_the_review": "The paper imports notions from differential geometry and manifold learning. This could have pros and cons.", "strength_and_weaknesses": "The paper has some features that can be considered both as strength and weakness.\n\n1. Insights from differential geometry and manifold learning are introduced. This could educate the machine learning community but also has the risk of not being appreciated.\n\n2. The paper shows extensive empirical results. However, most of them aims at showing a more accurate estimate of geometric score. That is, the gradient of log probability distribution function. While I agree it is necessary to show effectiveness using intrinsic measure like this, I think the community may care more about the practical implications.\n\nAnother weakness is that the method relies on the metric $G(x)$ at each local coordinate. $G(x)$ may not be known a priori. In fact, $G(x)$ is often unknown for many learning problems. The paper should discuss how to handle it in a separate section. Will it require more training data, compared against the conventional regularizer, to figure out the $G(x)$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7dFtOYrSAd", "reviewer": "Reviewer_Ts5i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose geometrically regularized autoencoders for non-Euclidean data. The main contribution of the paper is adapting the regularization terms in the denoising autoencoder (DAE) and the reconstruction contractive autoencoder (RCAE) to the non-Euclidean setting. If the manifold from which the data are sampled are known a priori, the proposed autoencoders achieve improved reconstruction performance, and are able to estimate the score of the non-Euclidean data.", "summary_of_the_review": "I think this is an interesting paper. I have some concern on the practical implementation and its implication on the computational cost. I am willing to adjust my rating based on the authors' response.", "strength_and_weaknesses": "**Strength**\n1. The motivation of the paper is clear. It is also very well-organized. I especially like the different notations for points on the manifold and their coordinates.\n2. Theorem 1 suggests that the geometric score can be estimated from the trained auto-encoder (a generalization of the result by Alain & Bengio, 2014 in the Euclidean setting), and the numerical results are convincing.\n\n**Weakness**\n1. My biggest concern is the increased computational cost after taking into account the geometric information (exponential maps, geodesics, etc.) This seems to be buried under the rug of the paper. I would like to see come comparison of the computational time in training the vanilla autoencoder and the proposed models.\n2. The author seems to be using one single chart for a manifold. However, general manifolds are typically not parametrizable using one chart. What would happen if $r(x)$ and $x$ appear in two different charts?\n3. It might be a good idea to elaborate, in the main text, what \"$\\approx$\" means in equation (7).\n4. Also, is RCAE actively used in the community? There does not seem to be any reference on that in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster