Invertible Manifold Learning for Dimension Reduction
Authors: Siyuan Li
Haitao Lin
Zelin Zang
Lirong Wu
Jun Xia
Stan Z. Li
Venue:  (2021)
Keywords: Manifold Learning
Inverse Model
Representation Learing

Abstract: It is widely believed that a dimension reduction (DR) process drops information inevitably in most practical scenarios. Thus, most methods try to preserve some essential information of data after DR, as well as manifold based DR methods. However, they usually fail to yield satisfying results, especially in high-dimensional cases. In the context of manifold learning, we think that a good low-dimensional representation should preserve the topological and geometric properties of data manifolds, which involve exactly the entire information of the data manifolds. In this paper, we define the problem of information-lossless NLDR with the manifold assumption and propose a novel two-stage NLDR method, called invertible manifold learning ($\textit{inv-ML}$), to tackle this problem. A $\textit{local isometry}$ constraint of preserving local geometry is applied under this assumption in $\textit{inv-ML}$. Firstly, a homeomorphic $\textit{sparse coordinate transformation}$ is learned to find the low-dimensional representation without losing topological information. Secondly, a $\textit{linear compression}$ is performed on the learned sparse coding, with the trade-off between the target dimension and the incurred information loss. Experiments are conducted on seven datasets with a neural network implementation of $\textit{inv-ML}$, called $\textit{i-ML-Enc}$, which demonstrate that the proposed $\textit{inv-ML}$ not only achieves invertible NLDR in comparison with typical existing methods but also reveals the characteristics of the learned manifolds through linear interpolation in latent space. Moreover, we find that the reliability of tangent space approximated by the local neighborhood on real-world datasets is key to the success of manifold based DR algorithms. The code will be made available soon.

Under review as a conference paper at ICLR 2021
INVERTIBLE MANIFOLD LEARNING FOR DIMENSION REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
It is widely believed that a dimension reduction (DR) process drops information inevitably in most practical scenarios. Thus, most methods try to preserve some essential information of data after DR, as well as manifold based DR methods. However, they usually fail to yield satisfying results, especially in high tigh-dimensiooal cases. In the context of manifold learning, we think that a good low-dimensional representation should preserve the topological and geometric properties of data manifolds, which involve exactly the entire information of the data manifolds. In this paper, we define the problem of cermmttion-losless NLDR with the mani- fold assumption and propose novel two-stage NLDR method, called invertible manifold learning (in--LL,, to tackle this problem. A local isometry constraint of preserving local geometry is applied under this assumption in imv-ML. Firstly, a homeomorphic sparse coordinate transformation is learned to find the low- dimensional representation without losing topological information. Secondly, a linear compression is performed on the learned sparse coding, with the trade-off between the target dimension and the incurred information loss. Experiments are conducted on seven datasets with neural network implementation of inv-ML, called i-ML-Enc, which demonstrate that the proposed inv-ML not only achieves invertible NLDR in comparison with typical existing methods but also reveals the characteristics of the learned manifolds through linear interpolation latent space. Moreover, we find that the reliability of tangent space approximated by the local neighborhood on world datasets is key to the success of manifold based DR algorithms. The code will be made available soon.
INTRODUCTION
In real-world scenarios, it is widely believed that the loss of data information is inevitable after dimension reduction (DR), though the goal of DR is to preserve as much information as possible in the low-dimensional space. In the case of linear DR, compressed sensing [Donoho (006) breaks this common sense with practical sparse conditions of the given data. the case of nonlinear dimension reduction (NLDR), however, it has not been clearly discussed, e.g. what is the structure within data and how to maintain these structures after NLDR? From the perspective of manifold learning, the manifold assumption is widely adopted, but classical manifold based DR methods usually fail to yield good results in the many practical case. Therefore, what is the gap between theoretical and world applications of manifold based DR? Here, give the first detailed discussion of these two problems in the context of manifold learning. We think that a good low-dimensional representation should preserve the topology and geometry of input data, which require the NLDR transformation to be homeomorphic. Thus, we propose an invertible NLDR process, called imv-ML, combining sparse coordinate transformation and local isometry constraint which preserve the property of topology and geometry, to explain the cermattion-lssleess NLDR in manifold learning theoretically. We instantiate inv-ML as a neural network called i-ML-Enc via cascade of equidimensional layers and linear transform layer. Sufficient experiments are conduct to validate invertible NLDR abilities of i-ML-EN Enc and analyze learned representations to reveal inherent difficulties of classical manifold learning.
Topology preserving dimension reduction. To start, first make out the theoretical definition of information-losslee DR on a manifold. The topological property is what is invariant under a homeomorphism, and thus what we want to achieve is construct homeomorphism for dimension

---

Under review as a conference paper at ICLR 2021
reduction, removing the redundant dimensions while preserving invariant topology. To be more specific. f MMg - IR is a smooth mapping of a differential manifold into another, and if f is a homeomorphism of MØ into MM f(Mg) C R" we call f is an embedding of Mg into R Assume that the data set X = ( [my|] 1 S S n sampled from the compact manifold Mi C IRm which we call the data manifold and is homeomorphic to Mi- For the sample points we get are represented in the coordinate after inclusion mapping it, we can only regard them as points from Euclidean space I without any prior knowledge, and learn to approximate the data manifold in the latent space Z. According the Whitney Embedding Theorem (Seshadri & Verma 2016). MM is can be embedded smoothly into Rdd by homeomorphiimm Rather than to find the f Mi + Ma our goal is seek a smooth map : Mi + R C22,, where h = o= is homeomorphism of M into MM = h(M)) and d S S 2d < m, and thus the dim((((2))
8, which achieves the DR while preserving the topology. Owing to the homeomorphism we seek as a DR mapping.
the data manifold Mf is reconstructible via M₁ = h- a h(M1), by which we mean h topology preserving DR as well as information-lossles. DR.
R$
M
g
MØ
f
Ma
i
m
g f
(,,,2?,.,,z") ..., =
x = xx,,x2,,.., ...
Figure 1: Illustration of the process of NLDR. The dash line links M4 and æ means x is sampled from MV and it is represented in the Fuclidean space Rm after inclusion mapping i, We aim to approximate Mj from the observed sample æ. For the topology preserving dimension reduction methods, it aims to find homeomorphism g f-1 map æ into z which is embedded in R*.
Geometry preserving dimension reduction. While the topology of the data manifold Mj can be preserved by the homeomorphism h discussed above, it may distort the geometry. To preserve the local geometry of the data manifold, the map should be isometric on the tangent space M for every E M indicating that d d v) = d d (h(u), h(u)), Vu, v E TpM.. By Nash's s Embedding Theorem (Nash 1956), any smooth manifold of class C* with k 2 3 and dimension d can be embedded isometrically in the Euclidean space R with s polynomial in d.
Noise perturbation. In the real-world scenarios, sample points not lied the ideal manifold strictly due to the limitation of sampling, e.g. non-uniform sampling noises. When the DR method is very robust to the noise, is reasonable to ignore the effects of the noise and learn the representation Z from the given data. Therefore, the intrinsic dimension of X' is approximate to d, resulting in the lowest isometric embedding dimension is larger than s.
2 RELATED WORK
Manifold learning. Most classical linear nonlinear DR methods aim preserve the geometric properties of manifolds. The Isomap (Tenenbaum et al.] (2000) based methods aim to preserve the
global metric between every pair of sample points. For example, McQueen al. (2016) can be regarded as such methods based on the push-forward Riemannian metric. For the other aspect, LLE (Roweis & Saul (000) based methods try to preserve local geometry after DR, whose derivatives like LTSA (Zhang& & Zha 2004), MLLE (Zhang & Wang 2007). etc. have been widely used but usually fail in the high-dimensional case. Recently, based on local properties of manifolds, MLDL (Li et al.) 2020) was proposed as robust NLDR method implemented by a neural network, preserving the local geometry but abandoning the retention of topology. In contrast, our method takes the preservation of both geometry and topology into consideration, trying to maintain these properties of manifolds even in cases of excessive dimension reduction when the target dimension s' is smaller than s.
Invertible model. From AutoEncoder (AE) (Hinton & Salakhutdinov 2006). the fundamental neural network based model, having achieved DR and cut information loss by minimizing the reconstruction loss, some AE based generative models like VAE (Kingma & Welling 2014) and manifold based NLDR models like TopoAE (Moor et al, 2020) has emerged. These methods cannot avoid information loss after NLDR, and thus, some invertible models consist of series of
2

---

Under review as a conference paper at ICLR 2021
equidimensional layers have been proposed, some of which aim generate samples by density estimation through layers (Dinh et al. 2015) (Dinh et al. 2017) (Behrmann et al. 2019). and the other of which established for other targets, e.g. validating the mutual information bottleneck (Jacobsen et al. (218)). Different from methods mentioned above, our proposed i-MI.-Ein is a neural network based encoder, with NLDR as well as maintaining structures of raw data points based manifold assumption via a series of cquidimensional layers.
Compressed sensing, The Sinnssonnindesstrausss Theorem (Johnson & Lindenstrauss 1984) pro- vides the lower bound of target dimension for linear DR with the pairwise distance loss. Given a small constant E (0. 1) and samples {x,) in I a lincar projection W 1 - IR" s > O logm can be found, which embeds samples into s-dimensional space with (1 e) distortion of any sample pairs (æ,, æ,)) It adopts a prior assumption that the given samples in high-dimensional space have a relevant dindimmeniioall structure constraint which can be maintained by keeping the pairwise distance. Further, compressed sensing (CS) provides strict sparse conditions of linear DR with great probability to recover the compressed signal, which usually cooperates with sparse dictionary learning Hawe et al. 2013). The core of CS is Restricted Isometry Property (RIP) condition, which reads
c²
(1 1 [-]]]|= x21 S (2; - x²) 12 S + e)(|= æ211?
where c E (0. 1) is rather small constant and W is linear measurement of signal æι and Ø2- Given a signal æ Th with s-sparse representation a Dr on an m-dimensional orthogonal basis Φ, ar can be recovered from the linear measurement sparse optimization
Wa with great probability by the
Wxxx satisfies the RIP condition: arg min; ||ā||o, s.t. y = Wa. The linear measurement is rewritten y V/aa Vz where V/ is low-dimensional orthogonal basis and Φ can be found by the nonlinear dictionary learning. Some reconstructible CS-based NLDR methods (Wei al. 2015) (Wei et al. 2019) are proposed, which are achieved by preserving local geometry on Al-bbased networks, but usually with unsatisfying embedding qualities.
3 PROPOSED METHOD
We will specifically discuss the proposed two-stage invertible NLDR process inv-ML as the first stage in Sec 3.1 in which s-dimensional representation is learned by homeomorphism transformation while keeping all topological and geometric structure of the data manifold; then give applicable conditions in real-world scenarios the second stage in Sec 3.2, in which the dimension is further compressed to s'. We instantiate the proposed inv-ML as a neural network i-ML-Enc in See 3.3
(c)
ges W2
222
L-1
200-13
xWW
or (1)
a
(W,2
Encoder c Rm
(a)
(e)
(a)
(d)
W1-0--D
(b)
wyy
2 c BF
R-tr
W (100y
W o (
WF -
Decader & c g
Figure 2: The network structure for the proposed implementation A-ML-Enc. The first L - 1 layers equidimensional mapping in the green dash box are the first stage which achieves s-sparse, and they have an inverse process in the purple dash box. (a) is a layer of nonlinear homeomorphism transformation (red arrow). (b) linearly transforms (blue arrow) s-sparse representation in IR" into RR as the second stage. (c) are the extra heads by linear transformations. (d) indicates the padding zeros of the I-th layer to force _(a)rse
3

---

Under review as a conference paper at ICLR 2021
3.1 TOPOLOGY AND GEOMETRY PRESERVATION
Canonical embedding for homeomorphism. To seek the smooth homeomorphiim h, we turn to the theorem of local canonical form of immersion (Mei 2013). Let f . - N an immersion, and for any p .,, there exist local coordinate systems (U, の) around p and (V, 1)) around f(p) such that v o ooo-_ ((U) + (VV is a canonical embedding, which reads
oo fofo0 の (x x²,
x²)) (x+ xx² ,,,0,, 0,0,0 .0).
(2)
In our case, let M Mg. and N Mi. any point = (z .2, ,2) E MM R' can be mapped to point in 12th by the canonical embedding
s o
,²* = (2
0,0, 0).
(3)
For the point = is regarded as a point in R Φ = 1 is an identity mapping, and for h = g f-1 is homeomorphism, h is continuous. The Eq. can be written as
(2), 22) h a -----ll( "0-'(2', ,²2 .2,.... ".,,,,,,,000 h(x' 1 x², ,... ,."))
2-,0 0,0, 0)
(4)
Therefore, to reduce dim((X) = m to 8, we can decompose h into tb and h o 2h-1, by firstly finding homeomorphic coordinate transformation v to map x (x x2 ...... xx") into (xx
(z'. .... 2", 0, 0, 0). which is called a sparse coordinate transformation. and h o- can be easily obtained by Eq. 3 We denote h o v- by fup and call it sparse compression. The theorem holds for any manifold, while in our case, we aims to find the mapping of X' C R into R2, so the local coordinate systems can be extended to the whole space of R
Local isometry constraint. The prior local isometry constraint applied under the manifold assumption, which aims to preserve distances (or some other metrics) locally that (u, d.gg (l(g) Vu, v E T.M4.
3.2 LINEAR COMPRESSION
With the former discussed method, manifold-based NLDR can be achieved with topology and geometry preserved, i.e. 8-sparse representation in Th. However, the target dimension s' may be even less than 8, further compression can be performed through the linear compression hú IR - R& instead of sparse compression, where (z)z) = Wmxx' 2, with minor information loss. In general, the sparse compression is particular case of linear compression with ho (z) )) h²(z)) Az, where mxx and 0g,y is the Kronecker delta. We discusses the information loss caused by a linear A = (6x,)mxx compression under different target dimensions s' following cases.
input data after NLDR process without loss of any information by solving the sparse optimization Ideal case. In the case of < s < s', based on compressed sensing, we can reconstruct the raw
problem mentioned in Sec. when the transformation matrix Wmxx' has full rank of the column. In the case of d s' before and after NLDR are not homeomorphic, and it is reduced to local geometry preservation by LIS constraint. However, in the case of s' < d < s, both topological and geometric information is lost to varying degrees. Therefore, we can only try to retain as much geometric structure as possible.
is inevitable to drop the topological properties because the two spaces
Practical case. In real-world scenarios, the target di- mension s' is usually lower than s, even lower than d. Meanwhile, the data sampling rate quite low, and the clustering effect is extremely significant, indicating that it is possible to approximate Mi by low-dimensional hy- perplane in the Euclidean space. In the case of s' < s, we can retain as the prior Euclidean topological structure as additional topological information of raw data points. is reduced to replace the global topology with some relative structures between each cluster.
Figure 3: Sparsity and clustering effect.
4

---

Under review as a conference paper at ICLR 2021
3.3 NETWORK FOR IMPLEMENTATION
Based on Sec 3.1 and Sec 3.2 we propose neural network i-ML-Enc which achieves two-stage NLDR preserving both topology and geometry. as shown in Fig. 2 In this section, we will introduce the function of network structures and loss functions respectively. including the orthogonal loss, padding loss and extra heads for the first stage, and the LIS loss, push-away loss for the second stage.
Cascade of homeomorphisms. Since the sparse coordinate transformation v (and its inverse) can be highly nonlinear and complex, we decompose it into a cascade of L- 1 isometric homeomorphisms v)(4-1) ..... oz(()) v(11, which can be achieved by L-1 equidimensional network layers. For each (0(), is sparse coordinate transformation, where (²(22 2,,(() ... 22%00, o,... 0)
(z11(1+1), 22,((++1,,... 20(+110++1) O..... 0) with 8141 < 81 and - The layer-wise transfor- mation Z(+1) (z()) and its inverse can be written z(++1) (Wix()), za² (o- )),
in which Wi is the I-th weight matrix of the neural network to be learned, and (-)) is nonlinear activation. The bias term is removed here to facilitate its simple inverse structure.
Orthogonal loss. Each layer-wise transformation is thought to be a homeomorphism between z and z(41), and we want it to be a nearly isometric. We force each Wi to be an orthogonal matrix, which allows simple calculation of the inverse of Wi. Based RIP condition, the orthogonal constraint of the weight matrix in the first L - layers can be obtained as
Lorth Σ (1) p(W I),
(6)
where (a are the loss weights. Notice that PWW SUpxERm... , x
Wz is the spectral norm of
W, and the loss term can be written as p(W, Wi I) = supp S E RIP condition in Eq. IT..
[=
which is equivalent to
[=]
Padding loss To force sparsity from the second to (L 1)-th layers, we add a zero padding loss to each of these layers. For the 1-th layer whose target dimension is s/, pad the last m 8y elements of ((+1) with zeros and panish these elements with L1 norm loss:
Lpad Σ
3
(1+1))
(=
where {800] are loss weights. The target dimension 81 can be set heuristically.
Linear transformation head. We use the linear transformation head to achieve the linear com- pression step in our NLDR process, which is transformation between the orthogonal basis of high dimension and lower dimension. Thus, apply the row orthogonal constraint to Wz.
LIS loss Since the linear DR is applied at the end of the NLDR process, we apply locally isometric smoothness (LIS) constraint (Li et al. 2020) to preserve the local geometric properties. Take the LIS loss in the I-th layer as an example:
LLIS
Σ JENT
dx(a;, dz [d 101
(8)
where Aith a of 24's k-nearest neighborhood the input space, and dx and dz are the distance of the input and the latent space, which can be approximated by Euclidean distance in local open sets.
Push-away loss. In the real case discussed in Sec 3.2 the latent space of the (L 1)-th layer can approximately to be a hyperplane in Fuclidean space, so that we introduce push-away loss repel the non-adjacent sample points of each x; in B-radius neighbourhood in the latent space. It deflates the manifold locally when acting together with LLIS in the linear DR. Similarly, Lyash is applied after the linear transformation in the 1-th layer:
Lepush- Σ
Σ
dete!) (a jB log (1+ dz(=((22))))
(9)
1ml JENK
5

---

Under review as a conference paper at ICLR 2021
where 1() (0,1) is the indicator function for the bound of B.
Extra heads. order to force the first layers of the network to achieve NLDR gradually, introduce auxiliary DR branchs, called extra head, layers from the second to the (L - 1).th. The structure of each extra head is same as the linear transformation head and will be discarded after training. Textra is written
L Lestra Σ OL + 4D Lpush):
(10)
where { and {u are loss weights which can be set based on {st}-
Inverse process. The inverse process is the decoder directly obtained by the first layers of the encoder given by Eq. (5). which does not involved in the training process. When the target dimension equal s, the inverse of the layer-I. can be solved by some existing methods such as compressed sensing eigenvalue decomposition.
4 EXPERIMENT
In this section, first evaluate the proposed invertible NLDR achieved by i-ML-Enc in Sec 4.1 then investigate the property data manifolds with i-ML-Enc in Sec 4.2 The properties of i-ML-Enc are
further studied in Sec 4.3 We carry experiments seven datasets: (i) Swiss roll (Pedregosal
et al. 201T). (ii) Spheres (Moor et al. 2020) and Half Spheres, (iii) USPS (Hull) 1994). (iv) MNIST
(eeCun al. 1998). (v) KMNIST (Clanuwat al. (0188), (vi) FMNIST (Xiao et al. 2017), (vii)
COIL--20 (Nene et al. 1996b). The implementation is based on the Py Py 1.3.0 library running on
NVIDIA v100 GPU. The following settings of i-MI.-Enc are used for all datasets: LeakyReI.U with
= 0.1; Adam optimizer (Kingma & Ba 2015 with learning rate ir = 0.001 for 8000 epochs; the
local neighborhood is determined by kNN with k 15; L layers neural network as shown in Fig.
4.1 METHODS COMPARISON
To verify the invertible NLDR ability of i-ML-Enc and analyze different cases of NLDR, compare with several typical methods in NLDR and inverse scenarios on both synthetic (Swiss roll, Spheres and Half Spheres) and real-world datasets (USPS, MNIST, FMNIST and COIL-20). Six methods for manifold learning: MILE (Zhang & Wang 2007). t-SNE (Maaten & Hinton 2008) and ML-Enc
al. 2020) are compared for NLDR; three AE-based methods VAE (Kingma & Welling 2014).
TopoAE Mooor et al. 2020) and MI-AE (Li et al. 2020 are compared for reconstructible manifold learning. Three methods for inverse models: INN (gguyen et al. 2019). i-RevNet (Jacobsen
et al. 20.8). and i-ResNet (Behrmann et al.) 2019h are compared for bijective property. Among them, i-RevNet and i-ResNet are supervised algorithms while the rest are unsupervised. For fair comparison in this experiment, we adopt 8 layers neural network for all the network- methods except i-RevNet and i-ResNet. Hyperparameter values of i-ML-Enn and configurations these datasets such the input and target dimension provided in Appendix A.2
Table 1: Comparison in representation and invertible quality MNIST datasets
Dataset
Algorithm
RMSE
MNE
Trust
Cont
Kmin
Kmax
I-MSE
Acc
MLLE
0,6709
0.6573
1.873
6.7c49
36,80
0.8341
t-SNE
0.9896
0.9886
5.156
324.9
48.07
0,9246
ML-Ene
0.9862
0.9927
1.761
58.91
18.98
0.9326
VAE
0.5263
33.17
0.9712
0,9703
5.837
130.5
22.79
0.8652
TopoAE
0.5178
31.45
0.9915
0.9878
4.943
265.3
24.98
0.8993
MNIST
ML-AE
0.4012
16.84
0.9893
0.9926
1.704
57.48
19.05
0.9340
Anc (L)
0.0457
0.5085
0.9906
0.9912
2.033
60.14
18.16
0.9316
INN
0.0615
0.5384
0.9851
0.9823
1.875
22.38
7.494
0.9176
ReReNet
0.0443
0.4679
0.9118
0.8785
13.41
142.5
6.958
0.9901
i-ResNet
0.0502
0.6422
0,9149
0.8922
1.876
19.28
10.78
0.9925
i-ML-Enc(L-I)
0.0407
0.5085
0.9986
0.9973
1.256
5.201
5.895
0.9580

---

Under review as a conference paper at ICLR 2021
(a) Swiss roll
1-MI-Ene,
L-MI-Ewe,
L-LL-Lmc,
reconstruction
MI--nn
[b] Spere (PCA)
AMM-Eac, 4
[-M Eut. 4
Mi.-Fmc
[c] Half Spere (PCA)
1-16L-En,, bn
J-MI-Fnn,, Ax
Mi.-Enc
(d) MNIST (PCA) 7-MI-Eaa,, fa
Mi-EEn
E-SWE
[e] C011,20 (PCA)
(-MI-EnG, to
ML-ène
C-SNE
Visualization of invertible NLDR results of i-ML-Enc compared to ML-Enc and t-SNE. All the high-dimensional results are visualized by PCA and the target dimension 8' 2. (a) shows the NLDR and its inverse process of i-MI.-Enc
Figure 4:
the test set of Swiss roll in the case of d = s We show the cases s' < d S 8 and s' = d < s by comparing (b)(c): (b) shows the failure case of reducing spheres $100 sampled in IR into 10-D, while (c) shows results of reducing half-spheres 5 sampled in IR 101 into 10-D. The L, layers of i-ML-Enc show the same topology as the input data in both cases, but ML-Enc shows bad topological structures. (d) and (e) show results of two sparse cases on MNIST and COIL-20: The clustering effects of ML-Enc and t-SNE show the local geometric structure but dropping the relationship between sub-manifolds With both of the geometric and topological structures, i-ML-Enc provides more reliable representations of the data manifold.
Evalution metries. We evaluate an invertible NLDR algorithm from three aspects: (1) Invertible property. Reconstruction MSE (RMSE) and maximum norm error (MNE) measure the difference between the input data and reconstruction results by norm-based errors. (2) NLDR quality. Trust- worthiness (Trust) and Continuity (Cont) (Kaski & Venna 2006), latent MSE (I-MSE). Minimum (Kmin) and Maximum (K max) local Lipschitz constant (Li et al.) 2020) used evaluate the quality of the low-dimensional representation. (3) Generalization ability the representation. Mcan accuracy (Acc) of linear classification the representation measures models' generalization ability to downstream tasks. Their exact definitions and purpose are given in Appendix A.1 A.1
Conclusion. Table I compares the i-ML-Enc with the related methods on MNIST, more results and
detailed analysis on other datasets are given in Appendix A.2] The process of invertible NLDR of
i-ML-Enc and comparing results of typical methods are visualized in Fig. 4 We can conclude: (1) i-ML-Enc achieves invertible NLDR in the first stage with great NLDR and generalization qualities. The representation in the 1-th layer of i-ML-Enc mostly outperforms all comparing methods for both invertible and NLDR metrics without losing information of the data manifold, while other methods drop geometric and topological information to some extent. (2) i-ML-Enn tries to keep d S 8. Though the
more geometric and topological structure the second stage in the case representation of the L-th layer of i-ML-Enc achieves the second best in NLDR metrics, it shows high consistency with the L 1-th layer visualization results.
4.2 LATENT SPACE INTERPOLATION
Since the first stage of i-ML-Enc is nearly bomeomorphism, we carry out linear interpolation exper- iments on the discrete data points in both the input space and the (T. - 1)-th layer latent space to analyze the intrinsic continuous manifold, and verify the latent results by its inverse process. A good low-dimensional representation of the manifold should not only preserve the local properties, but also be flatter and denser than the high-dimensional input with lower curvature. Thus, we expect that the local linear interpolation results in the latent space should be more reliable than in the input space. The complexity of data manifolds increases from USPS(256), MNIST(256), MNIST(784), KMNIST(784) to FMNIST(784), which is analyzed in Appendix(A.3.I K-nearest neighbor interpolation. We first verify the reliability of the low-dimensional repre-
7

---

Under review as a conference paper at ICLR 2021
0.
3
8
3
B
3
8
3
0.4
0.0
LESF52251
3
8
02
MNNSSTPRA)
KNN (6)
(a)
Figure 5: (a) shows the proposed geodesics interpolation manifold; (b) reports the MSE loss of 1 to 10 nearest neighbors interpolation results on interpolation datasets.
sentation in small local system by kNN interpolation. Given sample æ;, randomly select æ in a; 's k-nearest neighborhood in the latent space to form sample pair (æz, æt)). Perform linear interpolation of the latent representation of the pair and get reconstruction results for evaluation as: é 1- (tbb(x;)) + (1 f))((ar) t E j0, 1]. The experiment is performed on i-ML-Ene with L= and 15, training with 8000 samples for USPS and MNIST(256), 20000 sapmles for MNIST(784), KMNIST, FMNIST.
Evaluation. (1) Calculate the MSE loss between reconstruction results of the latent interpolation e and the input space result æं, which is the corresponding interpolation results in the local neighborhood of the input space with æ².6 tæ; + (1 f)æz Fig. shows the results of k
1, ..., 10. (2) Visualize the typical results of the input space and the latent space for comparison, as shown in Fig. 6 More results and detailed analysis are given in Appendix A.3.2
S SSSCU55 S
2
5S55555
K
2
2
K 10
(a) USPS (256)
(b) MNIST(784)
K
1
(c) KMNIST
(d) FMNIST
Figure 6: The results of KNN interpolation in latent space. For each dataset, the upper row shows the latent result, while the lower shows the input result. The latent results show more noise but less overlapping and pseudo-contour than the input results.
Geodesic interpolation. Based on 4.2.1, we further employ a more reasonable method to generate the sampling points between two distant samples pairs in the latent space. Given a sample pair (æx, xz)) with & 2 45 from different clusters, we select the three intermediate sample pairs (I,, Fs, ). (Ia, Zig) (Dig xyy) with & < 20 along the geodesic path in latent space for piece-wise linear interpolation in both space. Visualization results are given in Appendix A.3.2
Conclusion. Compared with results the kNN and geodesic interpolation, we can conclude: (1) Because of the sparsity of the high-dimensional latent space, noises are inevitable on the latent results indicating the limitation of linear approximation. Empirically, the reliability of the latent interpolation decreases with the expansion of the local neighborhood the same dataset. (2) We will get worse latent results in the following cases: on the similar manifolds, the sampling rate is lower or the input dimension is higher indicated by USPS(256), MNIST(256) and MNIST(784), with the same sampling rate and input dimension, the manifold is more complex indicated by MN1ST(784),, KMNIST to FMNIST. They indicate that the confidence of the tangent space estimated by local
8

---

Under review as a conference paper at ICLR 2021
neighborhood decreases more complex manifolds with sparse sampling. (3) The interpolation between two samples in latent space is smoother than that in the imput space, validating the flatness and density of the lower-dimensional representation learned by i-ML-Enc. Overall, infer that the unreliable approximation of the local tangent space by the local neighborhood is the basic reason for the manifold learning fails in the real-world case, because the geometry should be preserved in the first place. To come with this common situation, it is necessary to import other prior assumption or knowledge when the sampling rate of the data manifold is quite low, e.g. the Euclidean space assumption, semantic information of down-steam tasks.
4.3 ABLATION STUDY
Analysis on loss terms. We perform an ablation study on MNIST, USPS, KMNIST, FMNIST and COIL-20 to evaluate the effects of the proposed network structure and loss terms in i-ML-Enc for invertible manifold learning. Based on ML,-Enc, three proposed parts are added: the extra head (Ex), the orthogonal loss Lorth (Orth), the zero padding loss Lped (Pad). Besides the previous 8 indicators, we introduce the rank of the output matrix of the layer L - as r(Z"-1), ) to measure the sparsity the -igh-dimensional representation. We conclude that the combination Ex+Orth+Pad the best to achieve invertible NLDR of s-sparse by series of equidimensional layers. The detailed analysis of experiment results are given Appendix A.4.]
Orthogonality and sparsity. We further discuss the orthogonality of weight matrices and learned s-sparse representations in the first stage of i-ML-Enc. We find that the first - 1 layers of i-ML-Enc are nearly strict orthogonal mappings and the output from the L 1-th layer can be converted to s-dimensional representation without information loss. The detailed analysis are provided in Appendix A.4.2 Thus, conclude that invertible NLDR of data manifolds can be learned by i-ML--Enc in the sparse coordinate transformation.
CONCLUSION
A novel invertible NLDR process inv-ML and neural network implementation inv-MLEnnc proposed to tackle two problems of manifold- basedd DR in practical scenarios, i.c., the condition for information-
NLDR and the key issue of manifold learning. Firstly, the sparse coordinate transformation is learned to find a flatter and denser low-dimensional representation with preservation of geometry and topology of data manifolds. Secondly, we discuss the information loss with different target dimensions in linear compression. Experiment results of i-ML-Enc on seven datasets validate its invertibility. Further, the interpolation experiments reveal that finding a reliable tangent space by the local neighborhood on real-work datasets is the inherent defect of manifold based DR methods.
REFERENCES
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, pp. 573-582, 2019.
Tarin Clanuwat, Mikel Bober-Irizan Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv: 2018. URI
http:// rxiv www.ivvvoooooooom orgg abs 18 010118
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In 3rd International Conference Learning Representutions (ICLR), 2015. URL
http:://axivv http:///xxiv.org//ooo abs 8516
Laurent Dinh, Jascha Sohl-Dickster and Samy Bengio. Density estimation using real NVP. In
5th International Conference on Learning Representations (ICLR). OpenReview.net, 2017. URL
https::/ net htt:://orrrrrrreme (oorrrmm1d-kkkpoH999
David L. Donoho. Compressed sensing. IEEE Trans. Inf. Theory, 52:1289-1306, 2006.
Simon Hawe, Matthias Seibert, and Martin Kleinsteuber. Separable dictionary learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. IEEE Computer Society, 2013.

---

Under review as a conference paper at ICLR 2021
Matthias Hein and Jean-Yves Audibert. Intrinsic dimensionality estimation of submanifolds in pp- 289-296, 2005.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-577, 2006.
Jonathan Hull. Database for handwritten text recognition research. JEEE Transactions on Pattern Analysis and Machine Intelligence, 16:550 - 554, 05 1994. doi: 10. 10.1109/34.291440.
Vörn-Henrik Jacobsen, Arnold W. M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In Proceedings of 6th International Conference on Learning Representations (ICLR). OpenReview.net, 2018.
William B. Johnson and JohnsonJoram Lindenstrauss. Extensions of lipschitz maps into hilbert space. Contemporary Mathematics, 206, 01 1984.
Samuel Kaski and Jarkko Venna. Visualizing gene interaction graphs with local multidimensional scaling. In European Symposium on Artificial Neural Networks, pp. 557-562, 2006.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference on Learning Representations (ICLR), 2015. http://://aae
org/ abs/ 1412. 6980
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR), 2014.
Yann LeCun, Léon Bottou, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-238) 2324, 1998.
Stan Z. Li. Zelin Zhang, and Lirong Wu. Markov-lipschitz deep learning. arXiv preprint arXiv:2006.08255 0abs/2000.082566 2020. URL/attps://axxxa wttp/b////abbboeeeeeeeeeege////e 08256
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (ANNv):2579-2605) 2008.
James McQueen, Marina Meila, and Dominique Joneas. Nearly isometric embedding by relaxation. In Proceedings of the 29th Neural Information Processing Systems (NIPS), pp. 2631-2639,, 2016.
Jiaqiang Mei. Introduction to Manifold and Geometry. Beijing Science Press, 2013.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Proceedings of the 37th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, 2020.
John Nash. The imbedding problem for riemannian manifolds. Annals of Mathematics, 63:20 63. 1956.
Sameer Nene, Shree Nayar, and H. Murase. Columbia object image library (coil-100), Technical re- port, 03 1996a. URL https:: wwww cs wwcss.collne columbia httpaaa///ee///eeeaaae////tt//a/
coil-200 20.00..00 php
Sameer A. Nene, Shree K. Nayar, and Hiroshi Murase. Columbia object image library (coil-20). Technical report, Columbia University, 1996b. URL URL/ttps: hhttps://wwwt cs www.oooooocoooooooooooooooooooo. www.umua.....m..... edu/
php
The-Gia Leo Nguyen, Lynton Ardizzone, and Ullrich Köthe. Training invertible neural networks as autoencoders. In Proceedings of 41st German Conference of Pattern Recognition (GCPR), volume 11824, pp. 442-455. Springer, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournaeeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12(85):282- 2830, 2011. / r. http://hh....ooooooo//o/////////
htmll
10

---

Under review as a conference paper at ICLR 2021
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290:2323 2226,,226222 2000.
Harish Seshadri and Kaushal Verma. The embedding theorems of whitney and nash. Resonance, pp. 815-826, 2016.
Joshua Tenenbaum, Vin De Silva, and John C Langford. global geometric framework for nonlinear dimensionality reduction. science, 290(55000)2319-233883 2000.
Xian Wei, Martin Kleinsteuber, and Hao Shen. Invertible nonlinear dimensionality reduction via joint dictionary learning. In 12th Latent Variable Analysis and Signal Separation (LVA/ICA), volume 9237 of Lecture Notes in Computer Science, pp. 279-286. Springer, 2015.
Xian Wei, Hao Shen, Yuanxiang Li, Xuan Tang, Fengxiang Wang, Martin Kleinsteuber, and Yi Lu Murphey, Reconstructible nonlinear dimensionality reduction via joint dictionary learning. IEEE Trans. Neural Networks Learn. Syst., 30(1):175-189 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 170vv7170.00777, 2017. URL htt.::/ttp/atittl
org/abs/1708. 07747
Zhenyue Zhang and Jing Wang. Mile: Modified locally linear embedding using multiple weights. In Advances in Neural Information Processing systems, pp. 1593-1600, 2007.
Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal on scientific computing, 26(1):313-338 2004.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DEFINITIONS OF PERFORMANCE METRICS
As for NLDR tasks, We adopt the performance metries used in MLDL (Li et al. 2020) and TopoAE (Moor et al. 2020) to measure topology-base manifold learning, and add new indicator to evaluate the generalization ability of the latent space. Essentially, the related indicators are defined based on comparisons of the local neighborhood of the input space and the latent representation. As for the invertible property, we adopted the norm-based reconstruction metrics, i.e. L2 and Lio norm errors, which are also based on the inputs. The following notations are used in the definitions d is the pairwise distance in space z(0) A is the set of indices to the k-nearest neighbors (k-NN)
of ( in latent space, and Wi.x is the set of indices to the h-NN of in input space; rijj is the closeness rank of 2)
(1) in the k--N of (1) The evaluation metrics are defined below:
(1) RMSE (invertible quality). This indicator is commonly used measure reconstruction quality. Based on the input x and the reconstruction output c, the mean square error (MSE) of the La norm is defined as:
N Σ t=1
RMSE = N2 x 21) 2))
(2) MNE (invertible quality). This indicator is designed to evaluate the bijective property of a layers neural network model. Specifically, taking each invertible unit in the network, calculate the L norm error of the input and reconstruction output of each corresponding layer, and choose the maximum value among all units. a model is bijective, this indicator can reflect the stability of the model:
MNE max 13KLL-
||| - -2 11,2, I.
(3) Trust (embedding quality). This indicator measures how well neighbors are preserved between the two spaces. The nearest neighbors of point preserved when going from the input space X to space z();
h2
Trust
Σ
Σ
(1
k)
l02
Ai
Mk(2M - 3k
k=
JEA
JEN
where kı and l2 are the bounds of the number of nearest neighbors, averaged for different k-NN numbers.
(4) Cont (embedding quality). This indicator asymmetric Trust. It checks what extent neighbors are preserved from the latent space ( the input space X:
Cont
2
M
Σ
(r
k)
K2
Ki
+
MR(2M 3k
k=kı
JENUN
(5) Kmin and Kmax (embedding quality). Those indicators are the minimum and maximum the local bi-Lipschitz constant for the homeomorphism between input space and the I-th layer, with respect to the given neighborhood system:
Kmin min max K Kmax 15$5
max max Ki,j 1s JEN
JEN
where is that for N--N used defining N and
K max
12

---

Under review as a conference paper at ICLR 2021
(6) 1-MSE (embedding quality). This indicator is to evaluate the distance disturbance between the imput space and latent space with L2 norm-basee error.
N IMSE = N2
N
1
Idx d2(((dz))) ((z))))
(7) ACC (generalization ability). In general, a good representation should have good genera- tion ability to downstream tasks. To measure this ability, logistic regression (Pedregosa et al. 2011] is performed after the learned latent representation. We report the mean accuracy on the test set for 10-fold cross cross-valddttion,
A.2 METHOD COMPARISON
Configurations of datasets. The NLDR performance and its inverse process are verified on both synthetic and real-world datasets. As shown in Table [ we list the type of the dataset, the class number of clusters, the input dimension m. the target dimension s', the intrinsic dimension d which only an approximation for the real-world dataset, the number of train and test samples, and the logistic classification performance on the raw input space. Among them, Swiss roll serves as an ideal example of information-losseer NLDR; Spheres, whose target dimension is lower than the intrinsic dimension s, serves as an excessive case of NLDR compared to Half-spheres; four image datasets with increasing difficulties are used to analyze complex situations in real-world scenarios. Additionally, the lower bound and upper bound of the intrinsic dimension of real-world datasets are approximated by (Hein & Audibert 2005) and AE-based INN (Nguyen al. 2019). Specifically, the upper bound can be found by the grid search of different bottlenecks of the INN, and we report the bottleneck size of each dataset when the reconstruction MSE loss almost unchanged.
Table 2: Brief introduction to the config configuration of datasets for method comparison.
Dataset
Type
Class
Input
Target s'
intrinsic d
Train
Test
Logistic
Swiss roil
synthetic
3
800
8000
Spheres
synthetic
101
10
101
5500
5500
Half-spheres
synthetic
10
5500
5500
USPS
real-world
10
256
10
10 to 80
4649
4649
0.9381
MNIST
real-world
10
784
10
10 to 100
20000
10000
0.8943
FMNIST
world
10
784
10
20
140
20000
10000
0.7984
COIL-20
real-world
4096
20
20 260
1440
1440
0.9974
Hyperparameter values. Basically, i-ML-Enc is trained with Adam optimizer (Kingma & Ba 2015) and learning rate lr = 0.001 for 8000 epochs. We set the layer number L = 8 for most datasets but L = for COIL-20. The bound push-away loss is set B
most datasets but
removed in Spheres and Half-spheres, We set the hyperparameter based on two intuitions: (1) the implementation of sparse coordinate transformation should achieve DR on the premise of maintaining homeomorphism; (2) NLDR should be achieved gradually from the first (L 1)-th layer because NL.DR is impossible to achieve by a single nonlinear layer. Based on (1), we decrease the extra heads weights y linearly for epochs from 2000 to 8000, while linearly increase the orthogonal loss weights a for epochs from 500 to 2000. Based on (2), we approximate the DR trend by exponential series. For the extra heads, the target dimension decrease exponentially from m to s for the 2-th to (L (L-12 1) layer, and the push-away loss weights μ increase linearly. Similarly, the padding weight should increase linearly. Because the intrinsic dimension is different from each real-world dataset, we adjust the prior hyperparameters according to the approximated intrinsic dimension.
Results on toy datasets. The Table compares the i-ML-Enc with other methods in 9 performance metrics on Swiss roll and Half-spheres datasets in the case
s'. Eight methods for manifold learning: Isomap [Tenebbumm et al. 2000), t-SNE MMatten & Hinton 2008), RR (McQueen et al.) 2016), and ML-Ene (Li et al. (2000) are compared for NLDR; four AE- based methods AE (Hinton & Salakhutdinov 2006), VAE (Kingma & Welling 2014). TopoAE (Moor al. 2020), and ML-AE (Li et al.] are compared for reconstructible manifold learning. We report the L-th layer of
i-MI-Enn (the first stage) for the NLDR quality and the (T. 1) layer (the second stage) for the
13

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison in embedding and invertible quality on Swiss roll and Half-spheres datasets.
I-ML-Enc achieves invertible NLDR in the first stage and top three embedding performance in the second stage when d
Dataset
Algorithm
RMSE
MNE
Cont
K'in
Kmax
1-MSE
Isomap
0.9834
0,9812
1.213
43.55
0.0756
t-SNE
0.9987
0,9843
10.96
1097
3,407
RR
0,9286
0,9847
4.375
187.7
0.0453
ML-Enc
0.9999
0.9985
1.000
2.141
0.0039
Swiss Roll
AE
0.3976
10.55
0.8724
0.8333
1.687
4230
0.0308
VAE
0.7944
13.97
0,5064
0.6486
1.51
4809
0.0397
TopoAE
0.5601
11.61
0,9198
0,9881
1.194
220.6
0.1165
ML-AE
0.0208
8.134
0,9998
0,9847
2.462
0.0051
i-ML-Enc (L)
0.0048
0.0649
0.9996
0.9986
1.004
2.471
0.0043
Isomap
0.8701
0.9172
1.845
199.3
0,4046
t-NEE
0.8908
0,9278
25.33
790.9
2.6665
0.8643
0.8516
3.047
201.2
0.4789
ML-Enc
0.8837
0,9305
1.029
46.35
0.0207
Half-spheres
AE
0.7359
11.54
0.6886
0.7069
1.763
4112
0.0937
VAE
0.8474
14.97
0.5398
0,6197
2.361
4682
0.1205
TopoAE
0.9174
13.68
0.8574
0.8226
1,375
154.8
0.4342
MI-AE
0.6339
9.492
0.8819
0.9293
1.025
43.17
0.0218
i-M (L.)
0.1095
0.7985
0.8892
0.9295
1,491
41.25
0.0463
Table 4: Comparison in embedding and invertible quality USPS, FMNIST, and COIL-20 datasets.
ML-Enc shows comparable performance for embedding metrics. Based on ML-Enc, i-ML-Enc achieves invertible NLDR the first stage while maintaining a good generalization ability. It also achieves the top embedding performance for the most NLDR metrics in the second stage when
8.
Dataset
Algorithm
RMSE
MNE
Trust
Cont
Kmin
K Mmax
L-MSE
Ace
I-SNEE
0.9831
0,9889
3.238
194.8
35.53
0.9522
MI.-Ene
0.9874
0.9897
1.562
52.14
14.88
0.9534
AE
0.6201
29.09
0.9845
0.974
4.728
87.41
17.41
0.8952
TopoAE
0.647
30.19
0.9830
0.9852
3.598
126.2
19.98
0.8876
ML-AE
0.4912
11.84
0.9879
0.9905
1.529
$5.32
15.05
0.9576
USPS
i-ML-Ene (L)
0.0253
0.3058
0.9886
0.9861
1.487
60.79
15.16
0.9435
INN
0.0535
0.5239
0.9872
0,9843
1,795
26.38
9.581
0,9305
i-RevNet
0.0337
0.3471
0.9187
0.9096
11.25
183.2
6,209
0,9945
i-ResNet
0.0437
0.5789
0.9205
0.9122
1.635
18.375
9.875
0,9974
i-ML-Ene(L-I)
0.0253
0.3058
0.9934
0.9927
1.165
4.974
5,461
0.9876
t-SNE
0.9896
0,9863
3.247
108.3
48.07
0.7249
ML-Ene
0.9903
0,9896
1.358
89.65
25.18
0.7629
AE
0.2078
27.45
0.9744
0.9689
6.728
102.1
21.98
0.7495
TopoAE
0.2236
31.01
0.9658
0.9813
6,982
115.4
23.53
0.7503
MI.-AE
0.4912
18.84
0.9912
0.9917
1.738
101.7
25.89
0.7665
FMNIST
i-ML-Enc (L)
0.0461
0.3567
0.9923
0.9905
1.295
83.63
20.13
0.7644
INN
0.0627
0.6819
0.9832
0.9744
1.364
21.36
9,258
0.8471
i-RevNet
0.0475
0.3519
0.9157
0.8967
21.58
204.8
6.517
0.9386
i-ResNet
0.0582
0.6719
0.9242
0.9058
1,953
22.75
9.687
0,9477
(LL-Eno(L-1)
0.0461
0,3567
0.9935
0.9959
1.356
6.704
6.017
0.8538
I-SNE
0.9911
0.9954
5.794
101.2
17.22
0.9039
ML-Enc
0.9920
0,9889
1.502
70.79
9.961
0.9564
AE
0.3507
24.09
0,9745
0.9413
4,524
85.09
11.45
0.8958
TopoAE
0,4712
26.66
0,9768
0,9625
5.272
98.33
27.19
0,9043
MI-AE
0.1220
16.86
0.9914
0.9885
1.489
68.63
10.34
0.9548
Coil-20
i-MI--Ene (L)
0.0312
1.026
0.9921
0.9871
1.695
71.86
11.13
0.9386
INN
0.0758
0.8075
0.9791
0.9681
2.033
79.25
8.595
0.9936
i-RevNet
0.0508
0.7544
0.9316
0,9278
11.34
147.2
9,803
1.000
i-ResNet
0.0544
0.7391
0.9258
0,9136
1.821
13.56
10.41
L.000
i-ML- MLL-Enn(L-1)
0.0312
0.9263
0.9940
0.9937
1.297
4.439
7.539
L.000
14

---

Under review as a conference paper at ICLR 2021
Swise
(-MtI-Sw..k
i-M1-8nc
(PCA)
1.M Bhc.
1.M Bnc. b,
(M1 Dec.
rwairstracton
hamap
ML Enc
LML Exc,
SNE
(4 FMNIST (PCA)
1.M2 Enc,
1.ML Enc,
(.ML-Dex,
moond nu
Isomap
ML Enc
L-ML-Ess,
Z-NKR
RNNNIST
1-ML-Bnx.
L-ML-ENs,
Enc.
Bnc,
roconstruction
Isomap
EMC, 4.
Figure 7: Visualization of invertible NLDR results of i-ML-Enc with comparison to Isomap, ML-Ene, and I-SNE on Swiss roll and five real-world datasets. The target dimension s' 2 for all datasets, and the high-dimensional latent space are visualized by PCA. For each row, the left five cells show the NLDR and reconstruction process in the first stage of i-ML-Enc, and the right four cells show 2D results for comparison. ML-Enc and t-SNE show great clustering effects but drop topological information. Compared the classical DR method Isomap (preserving the global geodesic distance) and t-SNE (preserving the local geometry), the representations learned by i-ML-Enc preserves the relationship between clusters and the local geometry within clusters.
01
0
34 4
5
7
889
0
3
4
5
6
7
8
9
0
2.
3
3
5
7
8 9
0
/
2
3
4
5
6
7
8
9
USPS
MNIST
B
1
A
*
KMNIST
FMNIST
C0IL-20
COIL-100
Figure 8: Visualization of reconstruction results of i-ML-Enc six image datasets. For each cell, the upper row shows results of i-ML-Ene while the lower row shows the raw input images. randomly selected 10 images from different classes demonstrate the bijective property of i-ML-Enc.
15

---

Under review as a conference paper at ICLR 2021
invertible NLDR abilitty. ML-Enc performs best in Trust, Kmin, Kmax and I-MSE on Swiss roll which shows its great embedding abilities. Based on ML-Enc, i-ML-Enc achieves great embedding results in the second stage on Half-spheres, which shows its advantages of preserving topological and geometric structures in the high-dimensional case. And i-MI.-Enc outperforms other methods in its invertible NLDR property of the first stage.
Results on real-world datasets. The Table compares the i-ML-Enc with other methods in performance metries on USPS, FMNIST and COIL-20 datasets in the case of s > s'. Six methods for manifold learning: Isomap, t-SNE, and ML-Enc are compared for NLDR; three AE-based methods AE, ML-AE, and TopoAE are compared for reconstructible manifold learning. Three methods for inverse models: INN (guuyen et al. 2019), i-RevNet (Jacobsen et al. (2018), and i-ResNet (Behrmann jt al. 2019) are compared for bijective property. The visualization of NLDR and its inverse process of I-MI--Enc are shown in Fig. 7 together with the NLDR results of Isomap, t-SNE and, ML.-Fnc.
The target dimension for visualization is 2 and the high-dimensional latent space are visualized by PCA. Compared with NLDR algorithms, the representation of the L-th layer of i-ML-Enc nearly achieves the best NLDR metries on FMNIST, and ranks second place on USPS and third place COIL-20. The drop of performance between the (L- 1) and L-th layers of i-ML- Enc are caused by the sub-optimal linear transformation layer, since the representation of its first stage are quite reliable. Compared with other inverse models, i-ML-EEnc outperforms in all the NLDR metrics and inverse metries in the first stage, which indicates that great low-dimensional representation of data manifolds can be learned by a series of equidimensional layers. However, i-ML-Enc shows larger NME on FMNIST and COIL-20 compared with inverse models, which indicates that i-ML- Enc is more unstable dealing with complex datasets in the first stage. Besides, we visualize the reconstruction samples of six image datasets including COIL-100 (Nene et al. 1996a) to show the inverse quality of i-MI.-Enc in Fig. g
A.3 LATENT SPACE INTERPOLATION
A.3.1 DATASETS COMPARISON
Here is a brief introduction four interpolation data sets. We analyze the difficulty of dataset roughly according to dimension, sample size, image entropy, texture, and the performance of classification tasks: (1) Sampling ratio. The input dimension and sample number reflect the sampling ratio. Generally, the sample number has an exponential relationship with the input dimension in the case of sufficient sampling. Thus, the sampling ratio of USPS higher than others. (2) Image entropy. The Shannon entropy of the histogram measures the information content of the image, and it reaches the maximum when the density estimated by the histogram is an uniform distribution. We report the mean entropy of each dataset. We conclude that USPS has richer grayscale than MNIST(256), while the information content of MNIST(784), KMNIST, and FMNIST shows an increasing trend. (3) Texture. The standard deviation (std) of the histogram reflects the texture information in the image, and we report the mean std of each dataset. Combined with the evaluation of human eyes, the texture features become rougher and rougher from USPS, MNIST to KMNIST, while FMNIST contains complex and regular texture. (4) Classification tasks. We report the mean accuracy of 10-fold cross-validation using kNN and logistic classifier (Pedregosa et al. 201I) for each data set. The credibility of the neighborhood system decreases gradually from USPS, MNIST, KMNIST to FMNIST. Combined with the visualization results of each dataset in Fig. 7 it obvious that KMNIST has the worst linear separability. Above all, we can roughly give the order of the difficulty of manifold learning on each data set: USPS 2PPSSMMNNST222222NN777)8)))
<
Table 5: Comparison of manifold learning difficulties of interpolation datasets
Dataset
Class
Train set
Dimension
Entropy
Texture
KNN
Logistic
USPS
10
9298
256
5,479
0.5097
0.9589
0.9381
MNIST(256)
10
9298
256
1.879
10.51
0.9493
0.9099
MNIST(784)
10
20000
784
1,598
39.75
0.9515
0.8943
KMNIST
10
20000
784
2,911
33.01
0,9141
0.6471
FMNIST
10
20000
784
4.115
24.75
0.8133
0.7984
16

---

Under review as a conference paper at ICLR 2021
6
8
8
8
USPS
9
9
9
9
9
7
7
K2 20
K 10
MNIST
Le
6
6
6
K2 25
Le
Le
Le
L
6
6
6
6
6
KMNIST
K2 20
10
FMNIST
K2 25
Figure 9: Visualization of KNN interpolation results of i-ML-Enc on image datasets with S 10 and k 2 20. For each row, the upper part shows results of i-ML-Enc while the lower part shows the raw input images. Both the input and latent results transform smoothly when is small, while the latent results show more noise but less overlapping and pseudo-contour than the input results when is large. The latent interpolation results show more noise and less smoothness when the data manifold becomes more complex.
A.3.2 MORE INTERPOLATION RESULTS
kNN interpolation. We verify the reliability of the low-dimensional representation by kNN interpolation. Comparing the results of different values of k; as shown in Fig. 9 we conclude that: (1) Because the high- -igendimensional latent space is still quite sparse, there is some noise caused by linear approximation on the latent results. The MSE loss and noises of the latent results are increasing with the expansion of the local neighborhood on the same dataset, reflecting the reliability of the local neighborhood system. (2) In terms of the same sampling rate, the MSE loss and noises of the latent results grow from MNIST(784), KMNIST to FMNIST, which indicates that the confidence of the local homeomorphism property of the latent space decreases gradually manifolds. (3) In terms of the similar data manifolds, USPS(256) and MNIST(256) show better latent interpolation results than MNIST(784), which demonstrates that it is harder preserve the geometric properties on higher imput dimension. (4) Though the latent results import some noise, the input results have unnatural transformations such pseudo-contour and overlapping. Thus, the latent
more difficult
17

---

Under review as a conference paper at ICLR 2021
3
3
3
3
3
3
8
8
8
8
8
MNIST
K
3
3
3
3
3
8
8
8
8
8
8
8
8
KMNIST
K 30
FMNIST
8 30
Figure 10: The interpolation results the geodesic interpolation in the latent space, For each dataset, the upper row shows the latent result, while the lower shows the input result. The samples 1, 5, 9, 13 pointed by the arrow are the original samples.
space results by i-ML--Enc is flatter and denser than the input space. In a nutshell, we infer that the difficulty of preserving the geometric properties based on approximation of the local tangent space by the local neighborhood is the key reason for the manifold learning fails in the real-world case.
smooth than the input space, which validates that the latent space learned
We further perform the latent interpolation along the geodesic path between sample pairs when à is large to generate reliable intermediate samples. It might reflect the topological structure of data manifolds when two samples in sample pair are in different clusters. Compared with results of MNIST, KMNIST, and FMNIST, as shown in Fig. T0 we can conclude: (1) The latent results are more reliable than those in the input space which can generate the synthetic samples between two different clusters. (2) Compared with MNIST, KMNNST, and FMNIST, the latent results of more complex datasets are more ambiguous and noisy, which indicates that is more difficult to find a low-dimensional representation of more complex data manifolds with all geometric structure preserved.
Geodesic interpolation.
A.4 ABLATION STUDY
A.4.1 ANALYSIS THE LOSS TERMS
We further conduct ablation study of the exira head (+Ex), the orthogonal loss Lorth (+Orth), and the zero padding loss Lad (+Pad) on MNIST, USPS, KMNIST, FMNIST and COIL-20. The Table reports ablation results in the 8 indicators and the r(ZL-1). We analyze and conclude: (1) The combination of Ex and Orth nearly achieve the best inverse and DR performance on MNIST, USPS, FMNIST, and COIL-20, which indicates that it is the basic factor for invertible NLDR in the first L- layers. (2) When only use Orth, the NLDR the first
layer of the network will degenerate into the identity mapping, and DR is achieved with the linear project layer L. (3) Combined with all three items Ex, Orth and Pad, L.ML-En obtains a sparse coordinate representation, but achieves little worse embedding quality USPS and COIL-200 than using Ex and Orth. (4) Besides the proposed loss items, ML-AE overperforms the other combinations in the Ace metric indicating the reconstruction loss helps improve the generation ability of ML-Ene. Above all, the Ex+Orth+Pad combination, i.c. i-ML-Enc, can achieve the proposed invertible NLDR.
A.4.2 ORTHOGONALITY AND SPARSITY
Orthogonal analysis. We first analyze the orthogonality of weight matrices the first stage by evaluating the orthogonal loss ||, Wi Z||. Using the same experimental settings Sec 4.1 the maximum of diadoiaa elements and the minimum diagonal elements of each layer are calculated in the first stage of i-M different datasets. We find that the margin of the maximum
18

---

Under review as a conference paper at ICLR 2021
value and the minimum value is at least 4 orders of magnitude apart, shown in Fig. I1 conclude that the first L layers in i-ML-Enc are close to strict orthogonal mappings.
We can
Table 6: Ablation study of the proposed loss terms i-MI.-Enc on five image datasets.
Dataset
Algorithm
RMSE
MNE
Trust
Cont
Kmin
Kmax
Ace
r(Zh-1)
MI-AF
0.40T2
16.84
0.9893
0.9926
1.704
57.48
0.9340
15
MI--Ene
0.9862
0.9927
1.761
58.91
0.9326
+Ex
0.9891
0.9812
2.745
78.88
0,9316
MNIST
+Ex+Orth
0.0341
0.4255
0.9874
0.9927
1.817
59.97
0,9298
361
EExOOtt+Paa
0.0457
0.5085
0.9906
0,9912
2.033
60.14
0.9316
125
+Orth
0.0056
0.1275
0.9652
0.9578
1.597
53.21
0.8807
716
ML-AE
0.4912
11,84
0.9879
0.9905
1.529
55.32
0.9576
ML-Enc
0.9874
0.9897
1.562
52.14
0,9534
+Ex
0.9849
0,9836
2.525
78.88
0.9413
11
USPS
+Ex+Orth 0.0395
0.2511
0.9895
0.9875
1.366
58.83
0.9376
192
+Ex+Orth+Pad
0.0253
0.3058
0,9886
0.9861
1.538
60.79
0,9456
116
+Orth
0.0109
0,2043
0.9702
0.9654
1.328
66.25
0.8961
243
ML-AE
0.4912
18.84
0.9781
0.9912
2.478
80.66
0.7639
19
ML-Enc
0.9738
0.9883
2.253
103.4
0.7719
18
+Ex
0.9786
0.9801
5.826
255.1
0.7624
KMNIST
+Ex+Orth
0.0463
0.4661
0.9805
0.9872
2.396
70.89
0.6325
406
Pax+OrthPaad
0.0844
0.4589
0.9875
0.9894
2.697
78. 18.19
0.7513
198
+Orth
0.0223
0.1962
0.9621
0,9593
1.991
60.51
0.5922
732
ML-Fnn 0.4912
ML-AE
18.84
0.9912
0.9917
1.738
101.7
0.7665
0.9903
0.9896
1.358
89.65
0,7629
+Ex
0.9886
0.9726
5.826
279.4
0.7624
16
FMNIST
+ +Ex+Orth 0.0337
0.3194
0.9895
0.9840
1.879
98.66
0.7613
393
PEx+Ort+Paa
0.0461
0.3567
0.9923
0.9905
1.298
83.63
0.7644
182
+Orth
0.0152
0.2975
0.9701
0.9593
2.073
89.03
0.5934
743
ML-AE
0.1220
16.87
0,9914
0,9885
1,489
74.79
0.9564
44
ML-EEnc
0.9920
0.9889
1.502
70.79
0.9564
46
COIL-20
+Ex+Orth
0.0049
0.093
0.9927
0.9852
1.378
66.39
0.9427
1190
+Ex+Orth+Pad
0.0171
1.026
0.9921
0.9871
1.695
71.86
0.9386
746
(a) Swiss roll, W W2
(b) MNIST, W W2
(c) MNISI, ronk eigenvalue in Lg
Figure 11: Visualization analysis of the representation from the L 1-th layer of i-ML-Enc. (a) and (b) show the margin of the non-diagonal elements and the diagonal elements with W? W7 trained on Swiss roll (50x50) and MNIST (784x784). These elements are divided into orders magnitude after min-max normalization, indicating the large margine of the diagonal and non-diagonal elements. (c) shows the rank of eigenvalues of the subspace decomposed (by SVD) from the output of the 8-th layer of i-ML-Enn the test set of MNIST, The number of the main eigenvalues of the output is 125 which is equal to its matrix rank.
Discussion on the s-sparse representation. We first provide a possible way to decompose the leamed dimeiiinnio representation in the 1-th layer of i-ML-Enc. i.c. decomposing the output matrix by PCA to construct a linear subspace. Taking the 8-th layer of i-ML-Ent Enc on the test set of MNIST which is 784-D example, we can construct 171-D orthogonal base vectors in the linear
19

---

Under review as a conference paper at ICLR 2021
subspace from the data matrix after dimension reduction and reconstruct the original space (784-D) without losing information by PCA (Pedregosa et al. 2011 and its inverse transform, as shown in Fig. [ Compared to the matrix rank 125-D in the 784-D space, the extra 46-D in the subspace can be regarded as the machine error in the process of performing PCA because of the large margin between the first 125 eigenvalues and the rest. We notice that the s-sparse achieved by the first stage of i-ML-Enc is higher than the approximate intrinsic dimension on each dataset, e.g. 116-sparse on USPS and 125-sparse on MNIST. We found the following reasons: (I) Because the data manifolds usually quite complex but sampling sparsely, the lowest isometric embedding dimension are between to 2d according to Nash Embedding Theorem and the hyper plane hypothesis. The obtained by i-ML-Enc on each dataset is nearly in the interval of [d. 2d]. which is not the true intrinsic dimension of the manifolds.
(2) The proposed i-MI--Enc not optimized enough which serves as a simple network implementation of inv-MI.. We need to design better implementation model if we want to approach the lower embedding dimension with the preservation both geometry and topology.
20


Reviews:
[{"review_id": "nPoq9mR7Vel", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a nonlinear dimensionality reduction (NLDR) method which is claimed to be invertible. The method inserts an existing idea (LIS) to a two-stage neural network implementation. The proposed method is tested with several known benchmark datasets as well as three synthetic datasets.\n\nThe paper currently is not enough for acceptance.\n\nFirst of all, the problem in attack is unclear. By title, abstract and introduction, it reads that the target is a lossless NLDR method such that the data objects can be perfectly reconstructed from the subspace. However, this is not true in the experiment part. Clearly the loss is not zero after reconstruction. On the other hand, without perfect reconstruction, the proposed method is just another approximation and I don't see major contribution. So I believe the term \"invertibility\" is different from common understanding and needs more careful definition.\n\nIn the definition of \"Topology preserving dimension reduction\", it requires the data set is sampled from a d-dimensional compact manifold with d<<m. This is too ideal and the assumption often does not hold. For example, when the data has m-dimensional noise, then the so-called invertibility does not exist.\n\nFrom Fig.4, I don't see any advantage of using i-ML-Enc for visualization. Clearly the classes are more mixed by using the proposed method.\n\nNo need to refer to the LIS paper. The LIS loss is just about local distance preserving, which is well known in Multidimensional Scaling (MDS) literature.\n\nIt is unknown how to determine the hyperparameters, e.g., gamma^{(l)} and mu^{(l)}.\n\nIt is unknown how to determine network architecture for various data types.\n\nThe writing is fairly good. The main problem is that the paper is not self-contained. Unless the readers dive into the references, it is hard to understand the methodology, especially many symbols and acronyms.\nSome detailed comments:\n* All equations are not numbered.\n* Some presentation is vague and unexplained. For example, what do dash lines mean in Fig.1? At the end of Section 2, what do you mean \"their performance is not impresssive\"?\n* I cannot find the definitions of d_X and d_Z in the LIS-loss\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hH7IVQy7WhH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel manifold learning method, via adding a locally isometric smoothness constraint, which preserves topological and geometric properties of data manifold. Empirical results demonstrate the efficacy of their approach. The authors also show that the reliability of tangent space approximated by its local neighborhood is essential to the success of manifold learning approaches.\n\nOverall I found the ideas in the paper somewhat interesting. Many aspects were unclear to me (refer below).\n\n1) In the abstract, the authors claim that in their first step there is no loss of topological information. How did the authors measure this i.e. the sparse coordinate transformation step does not result in any loss of topological information ? Do the results in the paper demonstrate this ? \n\n2) I was not clear to me as to how the authors in their approach are preserving topology and in general geometry of the space while reducing the dimension of the space simultaneously. I would like the authors to add a section on this and/or improve the clarity of their presentation. \n\n3) What is the intuition for the two step process of their approach ? It definitely fits the encoder-decoder framework but is there some other reasoning behind this approach ?\n\n4) In section 3.3, the authors introduce the orthogonal loss wherein they force the weight matrices to be orthogonal. After training, are the weights orthogonal or how close to orthogonal are the weights ? Can we possibly add some other constraint in place of this ? \n\n5) Given the complexity of the model and the different loss functions associated with the objective, how much overhead is involved in model training and execution. Do the authors plan to share any time complexity results so that we can compare training/execution times with other state-of-the-art models ?\n\n6) In the appendix section A.1, what is the intuition behind the exact mathematical form of the expressions used for the Trust and Cont metrics ? I was curious with regards to the exact mathematical expressions used i.e. the different constants and factors involved.\n\n7) I did not quite understand the notation of Figure 2. What is the meaning of the different colored arrows and boxes indicate and how are they used ? There is not discussion on this as well. Similarly for Figure 4 for which there is no analysis or discussion. I would like the authors to include these without which the notation is cumbersome to follow. It is surprising that there is no additional discussion or analysis of the different figures included in the paper, given the captions are not sufficient by themselves.\n\n8) How are the authors deciding on the values of the different hyper-parameters i.e. gamma and alpha in the Appendix ? Overall it felt very heuristic and I did not quite understand how the authors decided on the values used. \n\n9) Some of the loss based definitions in Section 3.3 are unclear to me. In case the authors used some standard/other loss functions from some paper, they could have added those definitions in the main paper or the Appendix etc. I found that the paper was not self contained as such and I had to navigate to the references quite a bit.\n\nI found quite a few typos in the submitted draft. Kindly proofread and correct these. Overall I felt that the paper does not quite achieve lossless compression as well as there is only marginal improvement in terms of results when compared against other approaches. The approach presented in the paper in not clear in many parts.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hAQawEuBOQR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nFigure 4. (a) is informative, which clearly illustrates the invertible learning process. \n\nThe authors display the failure cases in Fig 4. (b), which is helpful for other researchers working on this filed. However, more analyses and discussion regarding the failure cases are suggested. \n\nMy major concerns are listed as follows.\n\n1, the contribution of this paper seems to be over-claimed. This paper attempts to learn an NLDR without loss of information; however, it is clear that the method in this paper also loses some information in the second step. Moreover, this paper seems to be a combination of previous works (LIS + sparse coordinate transformation), and thus it is important to clearly state the real contributions.\n\n2, the presentation of this paper needs to be improved as many technical details are omitted. For example, in introduction, authors do not well describe how to preserve topology and geometry. This is very important as the preservation of topological and geometric properties of complex structures is a very difficult task [1], while at the same time, low-dimensional structures usually have sophisticated geometric and topological structures [2]. \n\n[1] Cohen, et al. \"A general theory of equivariant CNNs on homogeneous spaces.\" NIPS. 2019.\n[2] Wakin, et al. “The multiscale structure of non-differentiable image manifolds”. \tIn Proceedings o fSPIE, the International Society for Optical Engineering, pages 59141B–1, \t2005.\n\t\n3, The network mentioned in Fig. 2 is not clear. For example, what is the main difference between the blue and red arrows, the difference between the solid lines and the dash lines, and the difference between the gray rectangles and the white rectangles? Moreover, as this network is based on ML-Enc, without introducing the structure of ML-Enc, it is hard to clearly illustrate the network structure of the proposed method. \n\n4, Figure 4 is important as it visualizes the embeddings of different methods; however, there isn’t any discussion related to figure 4. In addition, in figures 4 (c)-(d), I cannot see the advantage of the proposed model over the comparted ones. \n\n5, There is an interesting phenomenon that the L-1 th layer preserves the most NLDR results, which should be elaborated.\n\n\n6. there are some typos in this paper, e.g., a NLDR -> an NLDR, and the image quality like figure 1 can be enhanced. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7aeVd6RaL25", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nPaper presents a novel encoder-decoder framework for invertible dimensionality reduction. It is composed of multiple stages of homeomorphic embedding, sparse representation, linear compression and an inverse (reconstruction) process to learn invertible non-linear representations. Proposed idea is indeed novel and interesting actualization of geometry preserving dimension reduction shown in Figure 1. \n\nStrengths:\nProposed methods combines multiple ideas work on structure-preserving manifold learning, invertible and distance-preserving sparse representation learning.\nEach of the steps above are achieved by NN structure and novel loss functions that impose orthogonality, sparsity and isometry constraints and so on.\nEmpirical results on synthetic and real-world datasets support the approach and shown efficacy of the method. Ablation studies on adding different components show need of each aspect.\n\nWeakness: \nInvertible mapping learned is computed explicitly but can also be learned end-to-end during training. Is there a reason why the prior is preferred. \nPaper doesn't not address convergence aspect of the training and how it affects empirical results. Sparsity count (s) for representation is still heuristic and choice is not obvious. Can the RIP property provide a lower bound for choosing s.\n\nRecommendation:\nPaper is a clear accept as it introduces a novel method achieve Figure 1 using NN. It is well written and technically sound and qualitative results demonstrate effectiveness of the technique in terms of SOTA results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
While reviewers find the ideas in the paper interesting, they also raise several major concerns.
In particular, R1 and R4 find the claims of "invertible" and "lossless" to be potentially misleading.
The bijective property is achieve on the first stage (L-1 layers) due to a sequence of one-to-one mappings, as is done in previous work (e.g. i-RevNet)  so the novelty is limited.  As stated by R3,  since the paper is a combination of previous methods, the writing should be substantially improved to clarify what the real, new contributions are. The interpretation of the results (e.g. Figure 4) should also be better explained.
Decision: Reject