Neighborhood Reconstructing Autoencoders
Authors: Yonghyeon Lee
Hyeokjun Kwon
Frank C. Park
Venue: NeurIPS 2021 Poster (2021)
Keywords: Autoencoders
Manifold Learning
Neighborhood Graph
Geometry

Abstract: Vanilla autoencoders often produce manifolds that overfit to noisy training data, or have the wrong local connectivity and geometry. Autoencoder regularization techniques, e.g., the denoising autoencoder, have had some success in reducing overfitting, whereas recent graph-based methods that exploit local connectivity information provided by neighborhood graphs have had some success in mitigating local connectivity errors. Neither of these two approaches satisfactorily reduce both overfitting and connectivity errors; moreover, graph-based methods typically involve considerable preprocessing and tuning. To simultaneously address the two issues of overfitting and local connectivity, we propose a new graph-based autoencoder, the Neighborhood Reconstructing Autoencoder (NRAE). Unlike existing graph-based methods that attempt to encode the training data to some prescribed latent space distribution -- one consequence being that only the encoder is the object of the regularization -- NRAE merges local connectivity information contained in the neighborhood graphs with local quadratic approximations of the decoder function to formulate a new neighborhood reconstruction loss. Compared to existing graph-based methods, our new loss function is simple and easy to implement, and the resulting algorithm is scalable and computationally efficient; the only required preprocessing step is the construction of the neighborhood graph. Extensive experiments with standard datasets demonstrate that, compared to existing methods, NRAE improves both overfitting and local connectivity in the learned manifold, in some cases by significant margins. Code for NRAE is available at https://github.com/Gabe-YHLee/NRAE-public.

Neighborhood Reconstructing Autoencoders
Yonghyeon Lee'
Hyeokjun Kwon'
Frank C. Park
Scoul National University! Saige Research²
Cyhlee, hj kwon))roboticcs.......o....o.... snu ac. kr fcpesnu kr
Abstract
Vanilla autoencoders often produce manifolds that overfit to noisy training data, have the wrong local connectivity and geometry. Autoencoder regularization techniques, c.g., the denoising autoencoder, have had some success in reducing overfitting, whereas recent graph-based methods that exploit local connectivity information provided by neighborhood graphs have had some success in mitigating local connectivity errors. Neither of these two approaches satisfactorily reduce both overfitting and connectivity errors; moreover, graph- based methods typically involve considerable preprocessing and tuning. To simultaneously address the two issues of overfitting and local connectivity, we propose new graph-based autoencoder, the Neighborhood Reconstructing Autoencoder (NRAE). Unlike existing graph-based methods that attempt to encode the training data some prescribed latent space distribution - one consequence being that only the encoder is the object of the regularization NRAE merges local connectivity information contained the neighborhood graphs with local quadratic approximations of the neighborhood reconstruction loss. Com-
decoder function formulate pared to existing graph-based methods, our new loss function simple and easy to implement, and the resulting algorithm is scalable and computationally efficient; the only required preprocessing step is the construction of the neighborhood graph. Extensive experiments with standard datasets demonstrate that, compared to exist- ing methods, NRAE improves both overfitting and local connectivity in the learned manifold, in some cases by significant margins. Code for NRAE is available https://github..com// YHLee/NRAE-pubii
Introduction
Autoencoders are widely used to identify, and to generate samples from, the underlying low- dimensional manifold structure of given data distribution [14, I. It has been widely observed that vanilla autoencoders quite often produce manifolds that (i) are highly sensitive to noisy training data (see Figure [a), (ii) have the wrong local connectivity and geometry (see Figure [b)), significantly impairing their performance. Regularization techniques have had some success in mitigating the former, e.g.. the Denoising Autoencoder [2I], which uses deliberately corrupted inputs to train the autoencoder, typically learns manifolds that are robust to noise, but not always with the correct local geometry.
Recently, autoencoder regularization methods that use neighborhood graphs have had some success in addressing the incorrect connectivity issue [16] 18 TO 7. Notwithstanding the additional computa- tional overhead of constructing local neighborhood graphs, the local geometric information obtained from these graphs significantly reduce errors in the local geometry of the learned manifold, and make learning more well-behaved and robust.
The underlying premise behind these methods is that since the local geometry and topology of the data captured in the latent space distribution. which is determined entirely by the encoder, regularizing only the encoder should be sufficient; little if any consideration needs to be given to the decoder.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Data manifold and noisy sampled points
Learned manifold overfits the data
(b)
manifold and linite sampled points
Learned manifold with wrong geometry
1d latent space
Figure I:
Learned manifolds that (a) overfit the data (b) have the wrong local geometry.
The flaw with this premise is that although the encoder learns the correct latent space representation of the manifold data, the decoder still susceptible to overfitting of the type shown in Figure TigureTaa. These existing methods moreover rely on computation-intensi
preprocessing steps like manifold learning [OO, linear coefficients computation 7. or computing topological features using persistent homology at each training iteration [66] 2]. each of whose computational requirements can grow significantly with problem dimension and scale.
The main contribution of this paper is a new graph-based autoencoder training method that addresses both the overfitting and connectivity issues illustrated in Figure(Ta))-b)) ab) Like current methods, our method also employs local graphs that capture the local geometry of the data distribution. 'he key idea behind our method, which we call the Neighborhood Reconstructing Autoencoder (NRAE). is to employ a local quadratic (and in some cases linear) approximation of the decoder function to formulate new neighborhood reconstruction loss lieu of the point reconstruction loss typically used for autoencoder training. This idea leads to learning the correct geometry and reducing noise sensitivity, significantly improving the robustness of autoencoder training.
To make things more explicit, let Is R² + R' be the encoder (parametrized by @) and fo R R" be the decoder (parametrized by 0). Whereas vanilla autoencoders are trained to minimize the sum of the point reconstruction errors Σ, |0 fo(g)(z;))1 NRAE minimizes a reconstruction
error of the form
Σ Σ Σ Fo(oo(=))
(1)
(EN()
where (xx)) is the set of neighborhood points of x; (including x;) and fo(;; (1:)))) is a local quadratic (or linear) approximation of fe about (x;). The vanilla autoencoder is obtained by setting (xx) = } } for all i. The key idea here is to locally approximate the decoder only, and to exploit the local geometric information extracted from the decoded manifold represented by the image of fa-
Like other neighborhood graph-based based methods, NRAE also learns the correct local geometry of the decoded manifold. At the same time, the local quadratic (or linear) approximation of fe considerably reduces any overfitting to noisy training data or sensitivity to outlicrs, while maintaining computational efficiency rather than the entire Jacobian or Hessian of fe. only the more easily computed Jacobian- vector and Hessian-vector products are needed for the approximation.
Compared to existing graph- based autoencoder regularization methods, NRAE is easy to implement, computationally efficient, and scalable, requiring only a single prior construction of the graph without additional pre-processing steps. Experiments with both synthetic and image data (MNIST, Fashion- MNIST, KMNIST, Omniglot, SVHN, CIFARIO, CIFARIO0, CelebA) confirm that overall our method better learns the correct geometry of manifolds, showing improved generalization performance vis-á-vis existing graph-based based and other autoencoder regularization methods.
2 Neighborhood Reconstructing Autoencoder
In this section, we first provide high-level mathematical description of the Neighborhood Recon- structing Autoencoder (NRAE), followed by algorithmic details and discussion of the NRAE's properties and behavior. Throughout we consider deterministic autoencoder with encoder
function Js : R= + I. and decoder function fo Rm >> R" (m S 7i), with their composition
denoted by Fo.0 = So o gø. We use the notation = (2r E R m denote the set of observed data points.

---

2.1 Mathematical Description
In what follows we use the notation (a:) to denote the set of neighborhood points of z, with included in (x).. We begin with the following definition:
Definition 1 Let Fo,o(:z) = linear) approximation of fo at 2 = (z).2² "".
where fo(:2) 2) is local quadratic (or in some cases
Ofe dz (2)d2* + Σ
1 fo
+
(z)dz'dz* d
2 02
1,=11
where dæ = =z Fo.( said to be neighborhoud reconstruction of (a)).
If instead of a quadratic approximation we use the linear approximation of fo. the image of Fo the tangent space of the decoded manifold at (z)) and the neighborhood reconstruction of ((x) is a subset of the tangent space; the neighborhood reconstruction in this case contains first-order local geometric information about the decoded manifold.
The key idea behind Definition is that we locally approximate the decoder, and not the encoder, extract and exploit local geometric information the decoded manifold, which is captured in the image of F= (i.e., the local approximation of the decoded manifold), Figure iigustrallsstattes an example where the autoencoder reconstructs the points almost perfectly, but the neighborhood reconstruction of (xx. whose elements lie in the tangent space (here we use the linear approximation of fa) is considerably different from M(x).
Data manifold & training data points
image of (decoded manifold)
Neighborhood reconstruction of (X)
Neighborhood x
Fo (2(());))
Figure 2:
The training data points (blue), the decoded manifold (orange), the neighborhood of x denoted by (xx), and the neighborhood reconstruction (red). The black dotted lines represent the correspondences between x' E (xx) and (x'
Given that the neighborhood reconstruction of (x)) reflects the local geometry of the decoded manifold, minimizing loss function that measures the difference between Nzr) and its image (N))( is means of training an autoencoder to preserve the local geometry of the original data distribution. With that goal in mind, we formulate a neighborhood reconstruction loss L as follows:
Σ ZED [()))
Σ d'EN(x))
K(1'22 .12' =2)
(3)
D
where K(z',a x)) is positive symmetric kernel function that determines the weight for each x' E (xx). Figure 3 Billustrates how the neighborhood reconstruction loss can differentiate among the quality of the learned manifolds whose point reconstruction losses are all the same (close to zero): Case has the smallest neighborhood reconstruction loss compared to Case (wrong local geometry) and Case 2 (overfitting). NRAE converges to the vanilla AE that is, the neighborhood reconstruction loss recovers the point reconstruction loss - if one of the following conditions is met: (1) (i)) {x). (ii) ((',, x)) = 6(2', x)., or (iii) fo is linear.
It is reasonable to ask whether there are any advantages to using an approximation for both the encoder and decoder, i.e., to use a local quadratic (or linear) approximation for the composition map Fe,s rather than just the decoder. As verified below in Section 4.3 using quadratic approximation for both the encoder and decoder results in minimal to no performance improvement (the results for this Extended NRAE (E-NRAE) case are nearly identical to those obtained for NRAE), but the computational requirements increase substantially. intuition suggests, applying a quadratic approximation for the decoder is sufficient in evaluating the neighborhood reconstruction loss.
3

---

Neighborhood (c)
Fag
Case 3
Figure 3: The orange curves represent the learned manifolds, the red points represent the neighborhood reconstruction, and the lengths of the black dotted lines represent the neighborhood reconstruction loss.
2.2 Algorithmic Details
Graph construction. The problem of inferring the geometric structure of a data distribution is typically posed as a graph construction problem [4]. We use one of the simplest graph construction methods, the k-NN graph with the Euclidean distance metric. The robustness of our algorithm with respect to the choice of k is tested in the Supplementary Material.
Kernel design.
We choose the following simple kernel
K(x', x) = + (1 - X) 6(x',z),
(4)
where 0 S A and 5(x', if x' and zero otherwise. This assigns the weight 1 for the center x E (x) and the weight À for the remaining neighborhood points.
Batch sampling. To estimate the gradient of the proposed loss function, both summations over D and (z)). Given a batch C D, we again sample a batch B,, from A (z). We empirically find that forcing each batch B. to include improves convergence. In this paper, set B, = {i, Fn. where xn is uniformly sampled from N N(x)) (x).
batch sampling for
3 Related Work: Regularization of Autoencoders
In this section, we review standard autoencoder regularization techniques and their relation NRAE: i) regularizing latent space distributions, ii) Jacobian regularization, and iii) regularization using neighborhood graphs.
Regularizing latent space distributions. One popular regularization strategy is to enforce the latent space distribution (an approximate posterior) to be close to some user-specified prior distri- bution; some examples include the Variational Autoencoder (VAE) [12]. Adversarial Autoencoder (AAE) [1). and Wasserstein Autoencoder (WAE) [9]. The Gaussian distribution popular choice for the prior, but this choice often leads to wer-regularization Several works improve the VAE by learning more complex priors, or an optimal prior in terms of maximizing the training objective These techniques are developed to make latent space distributions
function of the VAE g 6 close to some easy-to-sampe prior distribution, but are not designed to resolve the two main issues (local geometry and overfitting) addressed in this paper.
20 13
Jacobian regularization. The Contractive Autoencoder (CAB) attempts to enhance robustness of representation by penalizing the Jacobian norm of the encoder function [17]. However, like other encoder-only regularization methods, often learns manifold that overfits the data. More recently [ regularizes the Jacobian of the decoder function to learn flat manifold. The trained decoder function may produce smooth manifold but with the wrong local geometry. The main difference between these methods and NRAE is the use of a neighborhood graph containing local connectivity information.
Regularization using neighborhood graphs. Several recent works have developed graph-based autoencoder regularization methods that, at least implicitly, try to learn manifolds with the right local geometry: Generalized Autoencoder (GAE) [22]. Topological Autoencoder (TopoAE) [16),

---

Witness Autoencoder (W-AE) 18 Geometry Regularized Autoencoder (GRAE) [TO), and Structure- Preserving Variational Autoencoder (Sp-VAE) 77. These works focus almost entirely on the encoder
when regularizing the latent space distribution (with the exception of GAE (22)); using the regular- ization term that does not depend on the decoder lead to noise sensitivity and overfitting issues as described in detail above. These methods are also computationally intensive, especially for large scale sigh-dimensional problems, as a result of expensive pre-prcessiing training [8].
10 or computation in
In contrast, by focusing more on the geometry of the decoded manifold, NRAE is able to learn smooth manifolds with the correct local geometry. simpler to implement, and more scalable since it requires only a single prior construction of the graph without other additional pre-prccesiig steps. Indeed, with the exception of Sp- VAE 77. all other approaches test their algorithms with two-dimensional latent spaces, while we test NRAE for latent spaces with up to 128 dimensions.
For general regularization methods that focus on latent space distributions, we surmise that using our neighborhood reconstruction loss in lieu of the point reconstruction loss may lead to several performance improvements; this is verified in Section 4.3 where we combine our neighborhood reconstruction loss with some existing encoder regularization methods and compare their performance against the original methods.
Experiments
In this section, through extensive experiments with both synthetic and real-world image data, we compare NRAE with a range of existing regularization methods: the Variational Autoencoder (VAE) 12I. Wasserstein Autoencoder (WAE) [9], Denoising Autoencoder (DAE) (T). Contractive Autoencoder (CAE) [77, Geometry Regularized Autoencoder (GRAE) [10), and Structure-Present Variational Autoencoder (Sp-VAE) 77. For comparison with Sp- PAE, we augment the regularization term introduced in 7 to vanilla autoencoder rather than the variational autoencoder we refer to this autoencoder as the Structure-Presserving Preserving Autoencoder (SPAE) - since a direct comparison with the variational autoencoder is already made, and our intent is to examine the effects of the structure-preserving regularization term.
For NRAE, we use both the local quadratic and linear approximations, respectively denoted NRAE-Q and NRAE-L. Our focus is on comparing the manifold smoothing property, geometry preserving property, and generalization ability of NRAE against baseline methods. We refer the reader to the Supplementary Material for a description of the network architectures used in the experiments, together with implementation details including the hyperparameter tuning strategy.
4.1 Manifold Smoothing Property
Smoothness of learned manifolds. We first consider one-dimensional manifold embedded in R², {ar, sin(x)}æ E [[T, x]]. Given 50 randomly sampled data points, we add isotropic Gaussian noise with standard deviation of 0.2. We train the NRAE and baseline AEs with one-dimensional latent spaces. Figure and Table show that NRAE-L and NRAE-Q successfully re-produce smooth manifolds compared to other baseline methods.
Ground-trunt manitold noisy samples
VAC
DAC
NRAC-L
NRAC-Q
Figure 4: The noisy samples (blue) and learned manifolds (orange).
Table The mean- qquane reconstruction errors of 10,000 clean test data. The averages and standard deviations are computed over ten times run with different noises, and multiplied by 100 (the highest and lowest scores are ignored). The best and second-best results are colored red and blue, respectively.
All
VAB
WAB
DAB
CAE
GRAB
SPAE
NRAE&
NKAE
1:00:5 d 0.23
1.45 + 0.39
2.02 0.76
1.09 + 0:21
1.33 + 0.30
1.56 + 0.41
1.28 + 0.30
0.29 de 0.06 0.30 + 0.07
5

---

Second, we rotate 100 MNIST images of the digit eight 100 times by 3.6 degrees. obtaining a set of 10,000 training data. We train the NRAE and baseline AEs with two-dimensional latent spaces, normalize each latent space, and generate rotating images by sampling regular grids of the latent spaces. NRAE-L and NRAF- generate smoothly varying images compared to other baselines (Figure 5 The DAE does not generate smoothly varying samples because the noise statistics (Gaussian noise) used in training is different from the statistical noise in the rotating image data.
5
5
7 saave
Feeer
1188%53 F 111220e4
AU222119
8
3
5
AE
VAE
DAE
SPAE
NRAE-L
NRAE-Q
Figure 5: Generated samples of rotating digit 8 from regular grids of the latent spaces.
Finally, to confirm the denoising effect of NRAE on common image data, we train the NRAE and baseline AEs on MNIST and CIFARIO data corrupted with various levels Gaussian noise (we use sufficiently large epochs for convergence). We then compare denoising performance measured by the Peak Signal-to-Noise Ratio (PSNR). As shown in Table NRAE-L and NRAE-Q Q outperform other baselines including DAE which performed the third best (the noise statistics used in training is the same as the added noise statistics).
Table 2: Comparison of PSNR thebbr--th--ettery The latent space dimensions are 16 for MNIST and 128 for CIFARIO. The best and second-best results are colored red and blue, respectively.
Duraset
Noise
AE
VAL
WAE
DAR
CAE
GRAE
SPAE
NRAB-L
NKAE-U
MNIST
0.2
0,1
20.19
17.15
19.61
16.36
16.96
19.83
20.69
18.37
20.08
17.24
20.06
17.34
20.01
17.34
21.07
18.55
21.27
18.90
0.1
18:22
19.87
19.54
20.42
20.73
19.79
19.43
22.21
22:27
CIFARIO
17.84
18:25
17.38
19.30
18.86
16.98
16.90
21.04
20.9
Curvature of learned manifolds. We train the NRAE and baseline AEs on MNIST images with wwo-dimensional latent spaces, normalize each latent space, then visualize the scalar curvature field (i.e., twice the Gaussian curvature (9] (Figure 6). As shown in Figure 6 NRAE-L and NRAE-Q both learn flatter manifolds compared to our baseline AEs.
AE
VAE
WAE
DAE
CAE
SPAE
NRAE-L
NRAE-Q
Figure 6: The sealar curvature field (brighter-the-largen)
4.2 Geometry Preserving Property.
Swiss roll. Consider one-dimensional Swiss roll r(0) 0.1 0.9 0.9-0 0/((T)) for e [0, 2T). Given 30 randomly sampled data points, we add isotropic Gaussian noise with standard deviation of 0.01. We then train the NRAE and baseline AEs with one-dimensional latent spaces. Figure shows the reconstruction results for 1000 test data points. Only GRAE, SPAE, and NRAE-Q successfully learn manifolds with the right local geometry. However, unlike NRAE-Q, the regularization terms in GRAE and SPAE have the effect increasing reconstruction emoos (observe the red rectangles in Figure 7. NRAE-L fails to reconstruct points around the point ,, circled in red: the nearest neighbor to 3r, the training set, marked by the blue circle, lies close to the tangent space of the decoded manifold at Tr, i.c., the neighborhood reconstruction loss is small.

---

Training set
Test set
AE
VAE
WAE
DAF
CAE
GRAE
SPAE
NRAE-L
NRAE-Q
Swiss roll data trained with one-dimensional latent spaces. The dots below the figures represent the one-dimensional latent space encoding of the training set.
Figure 7:
Rotated/Shifted MNIST. First, as shown in the first row of the two figures in Figure 8 we generate two sets of 20 training data: the rotated MNIST images of the digit 3 and shifted MNIST images of the digit 7. We train NRAE and baseline AEs with one-dimensional latent spaces, encode the training data to the latent spaces, sort the encoded values in ascending order, and decode them to generate the images (Figure Only SPAE, NRAE-L, and NRAE-Q Q successfully learn manifolds of the rotated/shifted images with the correct geometry.
Rotated MNIST digit
Shitted MNIST digit
Iraining set
3 3
3
3
O
h
M
E
E 3 m
7 7
(g
m
m
M
M O
E
E
3
3
VAE
3 3
3
E
E E
M
m
m
7 7 7
7
7
7
SPAC
3
3
3 9
h P
M
E
3 3
7 7 7 7 7 7
7
7
7
7
7
NRAE-L NRAE-Q
3
3
3
> 9 P R
7 7 7 ? ? 7
7
7
7 7
3 3
3
3
P P
8EEE
7 7 7 ?
7
7
space akis
space
Figure 8: Generated rotated/shifted MNIST images. Discontinuities are marked by orange boxes.
Second, we rotate an MNIST image of the digit six 300 times by 1.2 degrees and obtain 300 training data; 40 of these are visualized in the first row of Figure 9 We further generate 1000 test data in similar manner. We then train the AE, DAE, CAE, and NRAE with one-dimensional circular latent spaces by adding which data are encoded to which point in the latent spaces via the cyclic color map. Only NRAE-Q Q is able to learn manifold with the correct local geometry.
layer the end of the encoder corresponding to 2 z/||²||, and visualize
Circular latent space
AE
DAE
CAE
SPAE
NRAE-L
NRAE-Q
Figure 9: Circular latent space encoding of the rotated MNIST images of digit 6.
4.3 Generalization
Test data reconstruction. With various large-scale standard benchmark image data, show that NRAE generalizes better to the test data compared to other baseline methods. We use the test reconstruction Mean Square Error (MSE) for quantitative comparison.
First, we train fully-connected neural networks with the MNIST, FMNIST, and KMNIST data by varying the number of training data from 1000, 2000 ... 10, 000, and compare the test reconstruction MSEs. The numbers for the validation and test data are fixed at 10, 000 and 50, 000, respectively. Figure 10 shows the MSEs number of training data increases, while those of the NRAE are lower than most of the other baselines regardless of the number of training data.
function of the number of training data. The MSEs decrease the
Second, we train convolutional neural networks and compare the test reconstruction MSEs (Table 3). We conduct two experiments with the entire public data denoted by L (large) and ii) with the subset
7

---

MNIST
FMNIST
KMNIST
Reconstruction MSE
0.026
0.020
0.060
0.038
0.055
VAE
0.020
0.022
0.016
0.050
WAE
0.034
0.045
0.040
CAE
GRAE
0.014
0.035
SPAE
0.012
0.030
NRAE-L
2000
4000 6000 8000 10000
2000
4000 6000 8000 10000
2000
4000
8800 10000
NRAE-Q
The number of training data
The number of training data
The number of training data
Figure 10: The test reconstruction MSEs as the number of training data changes.
of data denoted by S (small). The number of selected subsets are around 20 30% of the entire data. NRAE-L and NRAE-Q produce lower reconstruction MSEs than most of the other baselines, meaning that the manifolds learned with our methods better generalize to the test data. The CAE experiment on CELEBA data is excluded because of its high computational cost for high-dimensional data. Other measures such as the Frechet- Incepttinn Distance (FID) scores [1] and the Evidence Lower Bound (ELBO) are reported in the Supplementary Material.
Table 3: The test reconstruction MSEs, the lower the better. The latent space dimensions are 16, 32, 32, 32, 64, 128, 128, 128 for MNIST, FMNIST, KMNIST, Omniglot, SVHN, CIFAR1O, CIFAR 100, and CELEBA, respectively. The best and second-best results are colored red and blue, respectively.
Size
VAB
NRAG-L
NRAE-Q
MNIST
0010022
0.01009
0,00990
0.00998
0.10044
0,00953
0.00968
0.06888
0.00756
0.0690
0.00644
0.09922
0.00666
0.00944
0.00649
0.006R3
FMNIST
S
0.01428
0:01289
0.01277
0.01118
0.01233
0.01066
0.01099
0.01022
0.01060)
0,01065
0.01000
0.01044
KMNIST
$
0.03267
0,03234
0.02283
0.03280
0.03799
0.03206
0.03268
0.03011
0.03021
0,02844
0,02963
0.02776
0.02814
0.02762
0.02753
0,02732
0.02564
0.02602
Onanggic
8
0.00038
0,03627
0.00078
0.03068
0.00714
0.02667
0.02889
0.02668
0.02631
0.02704
0,03192
0.02726
0,02696
0,02567
0.02688
0,02641
0.02578
0.02539
SVIEN
8
0.00174
0.00920
0.00204
0,00420
0.00190
0.00320
0.00309
0L00273
0.00178
0.08317
0.00007
0.00200
0.00192
CIFARIO 10
0.14666
0.01620
0.01431
0.01427
0.01208
0.1452
0.05044
0.00766
0.00691
0.00660
0.01122
0.00863
0,00900
0,00755
0.00832
0.00K29
CJ00587
CIFAR1OO
0.01015
0,01064
0.01463
0.00551
0.01484
0.00862
0.00842
0.00910
0.00912
0.00766
0.00578
0.00717
0.00635
CELERA
T.
$
0.00780
0.06113
0.00646
0.00937
0.00630
0.00830
0.00990
0.00782
0.00595
0.00814
0.00665
0.00861
0.00563
0.00508
0.00965
0.00747
Compatibility with other regularization methods. In this section, we show that the NRAE loss function can be used together with other existing latent space distribution regularization methods to improve their generalization performance. We train convolutional neural networks with the MNIST and CIFAR 10 data (small). As shown in Table 4 the NRAE loss improves WAE, CAE, and SPAE by significant margins. For VAE, there is only little improvement, primarily because the VAE regularization term is too dominant in training. Interestingly, some cases of WAE, CAE, and SPAE combined with the NRAE loss show even better generalization performance compared to NRAE alone. These results imply that proper combination of the neighborhood reconstruction loss and existing regularization methods can lead further improvements in performance.
Extended NRAE. Instead of using a quadratic (or linear) approximation of fo. we can approximate the composition function Fe. = fo o 9o to define the neighborhood reconstruction loss. While product or Hessian- -vectors product
NRAE only requires the computation of the Jacobian-vecti of fo. E-NRAE requires the computation of these quantities for Fe.s. We train convolutional neural networks on the MNIST and CIFAR data (small) and compare their generalization abilities and computational requirements. As shown in Table 5 the extended versions show comparable generalization performance to the original version, yet take a longer per-epoch runtime (100 batch size and 10,000 training data).
8

---

Table 4: Comparison of the test reconstruction MSEs for VAE, WAE, CAE. GRAE, and SPAE before and after being combined with the NRAE loss function, MNIST and CIFARIO data (small) are used.
Dataset
NRAF-I
NRAF-Q
VAR
NRVAE-1
NRVAF-Q
WAF
NRWAF-I
NRWAF-Q
MNIST CIFARIO
0,00953
0.09688
0.01091
0.01089
0.00998
0.01009
0.08522
0
DURITOS
000691
0.01620
0.0613
0.06009
0D1451
0U0020T
0,00844
N
NRAEQQ
CAE
NRCAE
NRCAE-Q
SPAB
NRSPAE-L
NRSPAE Q
MNISTT CIFARIO
0000953
0L00968
0.00998
0L00936
0.00965
0,0989
0.00942
0.00755
0.00768
0.00891
0.01208
0,00118
0.00724
0.01004
0.00723
0.00703
NRAF-I
NRAF-Q
GRAF
NEGRAF-
NRGRAF-Q
MNIST
0.00953
0.00968
0,01001
0.00945
0.00974
CIFARIO
0.00788
0.01452
0.07166
0.00694
Table 5: Comparison of the test reconstruction MSES and per-epoch runtime estimates for NRAE-L, NRAE-Q, E-NRAE-L, and E-NRAE-Q using MNIST and CIFARIO data (small).
Dutaset
Metric
NRAE-L
E-RRAE
NRAE- Q
E-RRAE Q
MNIST
Tuntime
1956
0,00953 21.57 4
0.00000 24.85
0.09968 34.73
0.10017
40.96
CIFARIO
TURLime
mic
39.14
0.00112 61,28
0.00991 89.24
0
100.18
5 Conclusion
This paper has proposed a new graph-based autoencoder, the Neighborhood Reconstructing Au- toencoder (NRAE), that is capable of learning accurate manifolds that are robust to noisy training data and have the correct local connectivity and geometry, while being easy to implement, scalable, and computationally efficient. Neighborhood graphs that capture the local geometry the data distribution are combined with local quadratic (or linear) approximations of the decoder function to formulate a new neighborhood reconstruction loss, which turns out to be a generalization of the origi- nal point reconstruction loss. Through extensive experiments with both synthetic and standard image datasets, we have demonstrated the manifold smoothing property, geometry preserving property, and the generalization performance advantages- in some cases by significant margins of our method,
Further, we have empirically verified that (i) a proper combination of the neighborhood reconstruction loss and existing regularization terms that focus on the latent space distributions can lead to further improvements in performance and (ii) in neighborhood reconstruction loss, approximating the decoder only is sufficient for learning the correct manifold while being computationally more efficient than approximating both the encoder and decoder.
Our algorithm can be further enhanced in a number of different ways. First, the current implementation of NRAE uses the k-NN graph construction with Euclidean distance metric and a simple kernel function that outputs binary values. These choices are made for simplicity, and clearly sub-optimal. Other combinations of the graph construction method (e.g., the persistent homology (24)). D, distance metric (e.g.. using deep metric learning), and kernel function (e.g., the Gaussian kernel) are worth exploring. Second, NRAE can be extended to a stochastic model number of ways, e.g., using the probabilistic autoencoder 3 that uses the latent variable model, or the energy-based approach as done in 23
Acknowledgments and Disclosure of Funding
This work was supported by the NAVER LABS AMBIDEX Project, AIFF20066R/ASA11984722 SNU- IAMD, SNU BK21+ Program in Mechanical Engineering, and the SNU Institute for Engineering Research.
References
[1] G. Arvanitidis, L. K. Hansen, and S. Hauberg. Latent space oddity: the curvature of deep generative models. arXiv preprint arxiv:1710.11379, 2017.

---

[2] M. Balasubramanian, E. L. Schwartz, J. B. Tenenbaum, V. de Silva, and J. C. Langford. The isomap algorithm and topological stability. Science, 295(3552)77-7, 2002.
[3] V. Böhm and U. Seljak. Probabilistic auto-encoder. arXiv preprint arXiv:2000005799, 2020.
[4] C. Carey. Graph construction for manifold discovery. 2017.
[5] N. Chen, A. Klushyn, Ferroni, J. Bayer, and P. Van Der Smagt. learning flat latent manifolds with vaes. arXiv preprint 2002,000,,,4888,,, 2020.
[6] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, Sutskever, and P. Abbecl. Variational lossy autoencoder. arXiv preprint arXiv: 1,0731,,,
[7] X. Chen, C. Wang, X. Lan, N. Zheng, and W. Zeng. Neighborhood geometric structure- pressrvvngg variational autoencoder for smooth and bounded data sourees. IEEE Transactions on Neural Networks and Learning Systemx, 2021.
[8] N. Dilokthanakul, P. A. Mediano, M. Gamelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shana- han. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02645 2016.
[9] M. P. Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
[10] A. F. Duque, S. Morin, G. Wolf, and K. R. Moon. Extendable and invertible manifold learning with geometry regularized autoencoders. arXiv preprint arXiv:2007.07142) 2020.
[11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. arXiv preprint 17X6::1706.0850000 2017.
[12] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
[13] A. Klushyn, N. Chen, R. Kurle, B. Cseke, and P. van der Smagt. learning hierarchical priors in vaes. arXiv preprint arxiv: arXiv: 0408, 2019.
[14] M. A. Kramer. Nonlincar principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-243, 1991.
[15] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv preprint arXiv: 1.15614444, 2015.
[16] M. Moor, M. Horn, B. Rieck, and K. Borgwardt. Topological autoencoders. In International Conference Machine Learning, pages 7045-7054, PMLR, 2020.
[17] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In /cmi, 2011.
[18] S. Schänenherger, A. Varava, V. Polianskii, J. Chung, R. Siegwart, and D. Kragic. Witness autoencoder: Shaping the latent space with witness complexes.
[19] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: arXiv: 1011,01588 2017.
[20] J. Tomezak and M. Welling, Vac with a vampprior. In International Conference on Artificial Intelligence and Staristics, pages 1214-1223, PMLR, 2018
[21] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-. Manzagol, and L. Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal of machine learning research, 11())), 2010.
[22] W. Wang, Y. Huang, Y. Wang, and L. Wang, Generalized autoencoder: A neural network framework for dimensionality reduction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 490-497, 2014.
[23] S. Yoon, Y.-K. Noh, and F. C. Park. Autoencoding under normalization constraints. arXiv preprint arXiv:2105.05735 2021.
[24] A. Zomorodian and G. Carlsson. Computing persistent homology. Discrete Computational Geometry, 33(2):249-274, 2005.
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the Supplementary Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Supplementary Material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We have done five times run and reported standard errors for the main experiments (small) in the Supplementary Material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the Supplementary Material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See the Supplementary Material.
(b) Did you mention the license of the assets? [Yes] See the Supplementary Material.
(c) Did you include any assets either in the supplemental material URL? [Yes]
See the Supplementary Material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] See the Supplementary Material.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]


Reviews:
[{"review_id": "jfYLQD5Xmz", "reviewer": "Reviewer_mRsa", "summary": "This paper proposes a new graph-based autoencoder dubbed ‘Neighborhood Reconstructing Autoencoder’ (NRAE) for learning smoother manifolds. The goal is to (1) learn an AE that is robust to data noise (2) learn a manifold with the correct geometry. The authors claim the focus of regularization to induce a smooth representation should, unlike the previous work, be on the decoder and not the encoder. The proposed model minimizes an objective that encourages neighbouring xs (based on a predefined graph that is constructed using some kernel K(x,x’)) to also be close by in latent space by having a small ‘neighbourhood reconstruction loss’. The authors finally perform an extensive set of experiments on a variety of datasets which demonstrates that the proposed model is able to learn a more smooth manifold compared to other AE baselines, as well as having a better generalization error. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[General Comment]\n\nI enjoyed reading the paper. I think the idea behind the proposed approach certainly makes sense and can be useful in many domains. The extensive number of experiments in various scenarios makes the paper even stronger. While I think there are few things that need clarifying, I think this paper makes a good contribution to NeurIPS and I’m happy to recommend acceptance. \n\n- The authors could correct me if I’m wrong but I think the implicit assumption in this paper is that the ‘correct’ manifold is also in fact a 'smooth' one (therefore proposing to minimize the neighborhood reconstruction loss). In many real-world settings however, the ground-truth manifold is not smooth at all. Could the authors comment on this?\n- Similarly, distinguishing noise from features of interest is also an ill-posed problem as far as I know (in a fully unsupervised manner). That being said, the kernel we use here can act as inductive bias as to what we consider to be ‘neighbour’ in data space which I find very helpful. I think, if there is enough space, the authors should provide more discussion on this.   \n- While I think I understand the claim of focusing on regularizing the decoder instead of encoder, I’m not sure if there is enough explanation in the current draft about ‘why’ this is the case. The authors explain in paragraph 3 in the intro that decoder is also susceptible to overfitting which makes sense, but this doesn't explain why only regularizing the decoder and not encoder works significantly better. The decoder overall has a more difficult task as it has to map low-dimensional to high dimensional data, and in the cases where the training set is large, wouldn't it be very hard for the decoder to overfit?  \n- Could the authors discuss the connection of the proposed method to [1]?\n- I understand kNN is one of the simplest & common methods to construct graphs but euclidean distance in high-dimensional domains such as vision is typically not a good way to quantify similarity, especially if the data is sparse. To put it another way, there is no strict reason for why two points that are close in x-space (w.r.t L2 distance) should also be close in latent space. I understand that addressing this issue might be beyond the scope of this paper given that this is a fundamental problem, but I would still appreciate it if the authors could comment on it.  \n- The experiment Section in particular was very good I think. The experiments done here are very extensive and I think are designed very well, perfectly making the case for learning smoother manifolds and their benefit in terms of reconstruction error. \n\n[Clarity]\n\nThis paper is very well written. Most of the figures were very well designed and informative. \n\n[Novelty & Prior work]\n\nI am not very familiar with the field, but the discussion Related work section seems sufficient to me. The few papers that I think should be cited but not cited here are [1-3] \n\n\n[Minor Comments/Questions]\n\n- In Figure 4, what happens if you simply use weight regularization to smooth the manifold of a vanilla autoencoder; AE_LossL + λ||w||?\n\n- Figures 4, 6, & 7, and Table 2 are all very good!\n\n- I’m curious about the Experiment in Figure 9. Sorry if I missed it but could the authors elaborate on ‘training with a circular latent space’? How was the latent space designed here exactly?  \n\n\n\n[References]\n\n[1] Yeh, Chin-Chia Michael, et al. \"Representation Learning by Reconstructing Neighborhoods.\" arXiv preprint arXiv:1811.01557 (2018).\n\n[2] Wang, Wei, et al. \"Generalized autoencoder: A neural network framework for dimensionality reduction.\" Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2014.\n\n[3] Liang, Kongming, et al. \"Representation learning with smooth autoencoder.\" Asian conference on computer vision. Springer, Cham, 2014.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did a find job discussing the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VudLIJZJj4x", "reviewer": "Reviewer_AQVH", "summary": "Rather than reconstructing a given input $x$, the first- or second- order* Taylor approximations (centered at $x$) of the decoder are optimized to accurately reconstruct some neighborhood $\\mathcal{N}(x)$ of $x$. More formally, the loss for a single $x, x'$ pair, where $x' \\in \\mathcal{N}(x)$ is written as\n$K(x, x') \\cdot \\left \\Vert x - \\tilde{f}(g(x')) \\right \\Vert^2$\nwhere $g$ is the encoder and $\\tilde{f}(x')$ is a Taylor approximation of the decoder $f$, centered at $x$. The authors evaluate the method across a wide array of empirical experiments and show that it produces superior autoencoding performance by many measures.\n\n*There are two variants of the algorithm -- one that uses a first-order Taylor approximation, and one that uses a second-order Taylor approximation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear and easy to read --- this owes as much to the elegance of the idea as it does to the writing of the authors. Additionally, the method is both well-motivated, and well-validated in synthetic and real-world experiments. Neighborhood reconstruction via local Taylor approximations is charming in its simplicity, yet clearly quite far-reaching in the way it shapes the latent embedding.\n\nTo my knowledge, such an approach has not been previously made. I find the idea both original and worthy of discussion at this conference.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some concerns over the robustness of the hyperparameters of the model added by the neighborhood reconstruction terms. I note that the supplemental material explores the robustness of the work to the number of nearest neighbors. Have the authors also explored the robustness of the method to the kernel hyperparameter $\\lambda$? Or the neighborhood batch size, which I understand has been set to 2?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "8S2iQma02d9", "reviewer": "Reviewer_jN5d", "summary": "An auto-encoding (AE) loss that reconstructs data according to a local approximation of the autoencoding function. In particular, for each $x_i$ in the dataset, the model reconstructs data around a neighborhood of $x_i$ using a low-order Taylor expansion of the encoder-decoder composition around $x_i$. The loss to each point in the neighborhood is assigned a weight according to a simple kernel function. The author demonstrates empirical advantage compared to many other AE losses in extensive evaluations. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nI am familiar with AE but not with tree-based AE methods. I find the idea of this work novel interesting. In particular, the local approximation of the full encoder and decoder composition seems novel. \n\n# Quality\nThis work appears to be very much intuition-based and easy to understand. The authors also provided key visualizations of their method at work. My main concerns are\n1. the discussion on the source of overfitting is confusing, and I do not very much follow the argument. Although I see it is connected with the later proposal of approximating the encoder-decoder composition. \n2. a lack of theoretical analysis\n3. The toy experiments may be quite unfair to traditional methods, and a think the authors should either emphasize or remind readers of the assumptions made or run more experiments in a fairer comparison. In particular, for datasets with very few data points, I think the comparison with many other AEs is a bit unfair due to finite data. The problem of failing to denoise should go away if using more data.\n4. Please provide intuition why NRAE-Q can be better or worse compared with NRAE-L, especially in the context of Figure 7.\n\n# Clarity\nVery clear and easy to understand, although some visualizations can be improved to avoid confusion at first sight. Details to follow. \n\n# Significance\nDespite the lack of theoretical components, I find this work to be interesting and the empirical validations thorough, although the toy data raises some questions that I hope to see answers to from the authors. \n\n# Detailed comments\n\nDetailed comments:\n\na. Line 37-38: I don't see whether Figure 1(a) provides the intuition for the argument in this line. It could be that the encoder does not prune the noise (if z is 2-d), giving overfitting. How does it show the distinction of overfittings caused by encoder or decoder?\n\nb. Line 53-63: I found these detail to be repetitive of what follows, and I spent quite a while understanding these, and then realize that the details in Section 2 are much more clear.\n\nc. Line 87: $\\hat{F}$ of a set $\\mathcal{N}$ is not defined, but I guess it maps to a set through element-wise operation?\n\nd. Figure 2 is very confusing but can be very useful if the readers are reminded that the start blue point is on the short spiky part of the data manifold in the second panel, and thus the red points seem to be crossing the apparent data manifold. I see the pale orange model manifold in the fourth panel as well, but the important spike was pretty much covered by the red dots. I was confused for quite a while.\n\ne. Line 123: Please provide more details for the batch sampling. For each minibatch $B = [x_1, x_2,...,x_k]$, do you sample a neighbour for each of the $k$ samples? Does B_x only contain two samples? B_x doesn't really appear afterward. \n\nf. Line 132.  I find (5) a bit confusing. What is the domain of optimization? Is $x\\in\\mathcal{D}$ fixed? but somehow it is optimized over? I'm not familiar with the original literature so excuse me if this should be obvious.\n\ng. Line 139. What is the notion of convergence to a set with countable elements? Better replace with =?\n\nh. Line 144. It is identical to the vanilla AE *with linear decoder*?\n\ni. Section 4.1, second experiment, how do you normalize each latent space? I'm not sure I understood the last sentence about DAE.\n\nj. Line 212, Table 4.1 -> Table 1\n\nk. Figure 7. The problem of the failing cases seems to be that there is a spurious manifold jumping from the bottom point circled red to the inner tip of the spiral, spiraling around to the other point at the bottom and then tracing back to the inner tip. Is it just a consequence of the bigger gap between the points in circles? Again, this should not happen if the dataset is larger.  I really want to see why NRAE-Q avoided this problem since it seems that with a quadratic approximation this problem can happen more easily.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The very last section of limitations did not really talk about limitations but mostly future works and extensions. Please consider my comments above and refine. An ideal work should address theory, explain why a version of the algorithm can avoid problems of another version (linear vs quadratic). ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1foqaDAcqyQ", "reviewer": "Reviewer_JTcy", "summary": "The paper goes into detail about two major obstacles that frequent Autoencoders, namely overfitting to training data, and learning the correct local geometry of the data. They then claim that all existing methods struggle with one or the other, or they are so computationally taxing that they aren't scalable for most problems. The paper's main contribution is that they provide a new training method/loss function that uses a local 2nd order approximation of the decoder function. The paper claims that the decoder is more important than the encoder in correctly learning the local geometry of the data. They then go into the algorithm's details and then explore the related work of regularization of autoencoders. Finally, they run experiments to demonstrate the algorithm's properties as well as show that NRAE is compatible with other methods (VAE, WAE, CAE, and SPAE) and often gets better results combining methods.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach appears to be original and unique. However I have concerns about some of the claims in the paper.\n\nFirst, the authors are overly pessimistic on the scalability of methods such as GRAE. By my estimation, both GRAE and NRAE should have the same asymptotic performance computationally, although the constants may differ. Thus the authors should compare to GRAE in their experiments as well.\n\nThe authors also do not compare to SPAE in Figure 9. This should be corrected as in Figure 8, SPAE was the only other method that correctly learned the manifold. \n\nThe authors also claim that \"it is in fact the decoder, and not the encoder, that has a greater impact on learning the correct manifold\". The authors do not sufficiently justify this. They show that their approach seems to work well, but that is not sufficient justification for their claim. For one thing, regularizing the bottleneck has a direct effect on the decoder as well as it changes the inputs to it. Thus it is incorrect to claim that any method that only regularizes the bottleneck is ignoring the decoder. The authors should do more experiments demonstrating very clearly that focusing on the output of the decoder is superior to other methods that perform regularization on other parts of the autoencoder.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces NRAE -- Neighborhood Reconstructing Autoencoder. The high level goal is regularization to avoid overfitting to noise and learning a smooth manifold with the 'correct' geometry. The authors introduce an objective that encourages a small neighbourhood reconstruction loss’ based on a kernel. The paper is easy to follow, clearly written and includes an extensive set of experiments on a variety of datasets. 


During the rebuttal, the authors were able to successfully address the concerns raised by the reviewers, in particular, comparisons with GRAE, SPAE and nuanced discussion of the claims about decoder regularization. There is a consensus among the reviewers that the paper deserves acceptance. 
Decision: Accept (Poster)