Autoencoder Image Interpolation by Shaping the Latent Space
Authors: Alon Oring
Zohar Yakhini
Yacov Hel-Or
Venue:  (2021)
Keywords: deep learning
autoencoders
deep generative models
representation learning

Abstract: One of the fascinating properties of deep learning is the ability of the network to reveal the underlying factors characterizing elements in datasets of different types. Autoencoders represent an effective approach for computing these factors. Autoencoders have been studied in the context of enabling interpolation between data points by decoding convex combinations of latent vectors. However, this interpolation often leads to artifacts or produces unrealistic results during reconstruction. We argue that these incongruities are due to the structure of the latent space and to the fact that such naively interpolated latent vectors deviate from the data manifold. In this paper, we propose a regularization technique that shapes the latent representation to follow a manifold that is consistent with the training images and that forces the manifold to be smooth and locally convex. This regularization not only enables faithful interpolation between data points, as we show herein, but can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation

Under review as a conference paper at ICLR 2021
AUTOENCODER IMAGE INTERPOLATION BY SHAPING THE LATENT SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
Autoencoders represent an effective approach for computing the underlying factors characterizing datasets of different types. The latent representation of autoencoders have been studied in the context of enabling interpolation between data points by decoding convex combinations of latent vectors. This interpolation, however, often leads to artifacts or produces unrealistic results during reconstruction. We argue that these incongruíties are due to the structure of the latent space and because such naively interpolated latent vectors deviate from the data manifold. In this paper, we propose a regularization technique that shapes the latent representation to follow manifold that is consistent with the training images and that drives the manifold to be smooth and locally convex. This regularization not only enables faithful interpolation between data points, as a general regularization technique to avoid overfitting or to produce new samples for data augmentation.
show herein, but also be used
INTRODUCTION
Given set of data points, data interpolation extrapolation aims at predicting novel data points between given samples (interpolation) or predicting novel data outside the sample range (extrapola- tion). Faithful data interpolation between sampled data be seen capacity of a learning system (Berthelot et al. (018).. In the context of computer vision and computer graphics, data interpolation may refer generating novel views of an object between two given views or predicting in-between animated frames from key frames.
measure of the generalization
Interpolation that produces novel views of scene requires input such as the geometric and photo- metric parameters of existing objects, camera parameters and additional scene components, such as lighting and the reflective characteristics of nearby objects. Unfortunately, these characteristics are not always available or are difficult to extract in real-world scenarios. Thus, in such cases, we can apply data-driver interpolation that is deduced based sampled dataset drawn from the scene taken under various acquisition parameters.
Light source: (8.$)
Vertical pale
/
Plane
Cast Shadow
Figure 1: Left: A vertical pole casting a shadow. Yellow blocks-top row: Cross-dissolve phenomena result of linear interpolation in the input space. Yellow blocks-bottom row: Image reconstruction obtained by a linear latent space interpolation of an autoencoder. Unrealistic artifacts are introduced.

---

Under review as a conference paper at ICLR 2021
-30
Figure 2: The latent manifold of the data embedded in 2D latent space (leftmost plot) and 3D latent space (second plot from the left) learned by vanilla autoencoders. Gridlines represent the (0, o) parameterization. The second image from the right was generated from the latent point denoted 'A'. The rightmost image generated from the latent point denoted 'B'.
The task of data interpolation is to extract new samples (possibly continuous) between known data samples. Clearly, linear interpolation between two images in the input (image) domain does not work as it produces a cross wross-dissolve effect between the intensities of the two images. Adopting the manifold view of data (Goodfellow et al., 2016; Verma al., 2018 Bengio al.) 2013). this task can be seen as sampling new data points along the geodesic path between the given points. The problem that this manifold is unknown in advance and one has to approximate from the given data. Alternatively, adopting the probabilistic perspective, interpolation can be viewed drawing samples from highly probable areas in the data space.
One fascinating property of unsupervised learning is the network's 's ability to reveal the underlying fac- tors controlling a given dataset. Autoencoders (Doersch) 2016 Kingma & Welling 2013: Goodfellow et al. 2016; Kramer 1991 Vincent et al. 2010) represent effective approach for exposing these factors. Researchers have demonstrated the ability to interpolate between data points by decoding a convex sum of latent vectors (Shu et al. 2018 Mathieu et al. 2016); however, this interpolation often incorporates visible artifacts during reconstruction.
To illustrate the problem, consider the following example: A scene is composed of vertical pole the center of a flat plane (Figure [efft). A single light source illuminates the scene and accordingly, the pole projects shadow onto the plane. The position of the light source can vary along the upper hemisphere. Hence, the underlying parameters controlling the generated scene are (6, Φ), the elevation and azimuth, respectively. The interaction between the light and the pole produces a cast shadow whose direction and length are determined by the light direction. A set of images of this scene acquired from fixed viewing position (from above) with various lighting directions. Our goal in this example is to train a model that is capable of interpolating between two given images. Figure top row, depicts a set of interpolated images, between the source image (left image) and the target image (right image), where the interpolation is performed in the input domain. As illustrated, the interpolation is not natural as it produces cross-dee dioss-dissolve effects in image intensities. Training a standard autoencoder and applying linear interpolation in its latent space generates images that are much more realistic (Figure T bottom row). Nevertheless, this interpolation is not perfect as visible artifacts occur in the interpolated images. The source of these artifacts can be investigated by closely inspecting the 2D manifold embedded in the latent space.
Figure shows two manifolds embedded in latent spaces, one with data embedded in 2D latent space (left plot) and one with data embedded in 3D latent space (2nd plot from the left). In both cases, the manifolds are 2D and are generated using vanilla autoencoders. The grid lines represent the (0, o) parameterization. It can be seen that the encoders produce non-smooth and non-convex surfaces in 2D as well in 3D. Thus, linear interpolation between two data points inevitably produces be-bewween points outside of the manifold. In practice, the decoded images of such points are unpredictable and may produce non-realistic artifacts. This issue demonstrated in the two right images in Figure 2 When the interpolated point is on the manifold (an empty circle denoted 'A'), a faithful image is generated by the decoder (2nd image from the right). When the interpolated point departs from the manifold (the circle denoted 'B') the resulting image is unpredictable (right image).
In this paper, we argue that the common statistical view of autoencoders is not appropriate when dealing with data that have been generated from continuous factors. Alternatively, the manifold structure of continuous data must be considered, taking into account the geometry and shape of the manifold. Accordingly, propose new interpolation regularization mechanism consisting
2

---

Under review as a conference paper at ICLR 2021
(4)
(9)
(C)
(D)
((3)
f(x)
Recons, advers. eyde
cydle
Figure 3: Data interpolation using autoencoders. Two points 3; æ; located the input data manifold (solid black line). The encoder fax maps input points into the latent space 2i, 2j (red arrows). Linear interpolation in the latent space is represented by the blue dashed line. The interpolated latent codes are mapped back into the input space by the decoder g(z) (blue arrows). See Section 2.2 for the contribution of each loss component for an admissible interpolation.
of an adversarial loss, a cycle-consistency loss, and a smoothness loss. The adversarial loss drives optimized against discriminator that learns to tell
the interpolated points to look realistic apart real from interpolated data points. The cycle-consistency and the smoothness losses encourage smooth interpolations between data points. We show empirically that these combined losses prompt the autoencoder to produce realistic and smooth interpolations while providing a convex latent manifold with bijective mapping between the input and the latent manifolds. This regularization mechanism not only enables faithful interpolation between data points, but can also be used general regularization technique to avoid overfitting or to produce new samples for data augmentation, suggested, among others, by Zhang al. (2018).
To conclude, the contributions the papers are: We define what constitutes an admissible interpola- tion between two data points continuous manifold. In particular added the conssscons and the smoothness terms and show their importance in generating admissible interpolations. II. We empirically demonstrate how the combination of the four losses; the reconstruction, adversarial, cycle-consistency and the smoothness losses, contribute to admissible interpolations and produce state of the art results.
2 MANIFOLD DATA INTERPOLATION
Before presenting the proposed approach we would like to define what constitutes a proper interpola- tion between two data points. There are many possible paths between two points on the manifold. Even if we require the interpolations to be on a geodesic path, there might be infinitely many such paths between two points. Therefore, we relax the geodesic requirement and define less restrictive conditions. Formally, assume we are given dataset sampled from target domain X. We are interested in interpolating between two data points x; and æ; from X. Let the interpolated points be -;-s(a) (a) for [o, 1] and let P(x) be the probability that data point belongs to X. We define an interpolation to be an admissible interpolation if Ei+z (a) satisfies the following conditions:
1. Boundary conditions: (1-)(0) = x, and a; &; (1) xy
2. Monotonicity: We require that under some defined distance on the manifold d(æ,, x'), the interpolated points will depart from x; and approach æy, as the parameterization goes from 0 to 1. Namely, Ya' 2 a.
(())))(a), m;)) < d d( ((')) a;;)
and similarly:
d( m.)) < d( (a),
3. Smoothness: The interpolation function (a) ) Lipschitz continuous with a constant
K:
| 2 (a t)ll < K|t
4. Credibility: Va E [0. We require that it is highly probable that interpolated images, (a) belong to X. Namely,
P(t,,,(a)) 2 11-8. 1-8, for some constant 3 2 0
3

---

Under review as a conference paper at ICLR 2021
2.1 PROPOSED APPROACH
Following the above definitions for an admissible interpolation, we propose a new approach, called Autoencoder Adversarial Interpolation (AEAI), which shapes the latent space according the above requirements. The general architecture comprises standard autoencoder with an encoder, == f(x), and a decoder क g().. We also train discriminator ((x) to differentiate between real and interpolated data points. For pairs of input data points 2;, 2., we linearly interpolate between them in the latent space: _ (a) (1 a: 2; + a22 where a [0. 1]. The first requirement is that we would like (a) = g - (a)) to look real and fool the discriminator D. Additionally, we add cycle-consistency loss that encourages the latent representation of (i) to be mapped back f'o(z=
should be similar to za))((a), Finally, we add a smoothness loss that drives the linear parameterization to form smooth interpolation. Putting everything together we define the loss Ci-ss between pairs 2, and xj follows:
into Zi-z (a) again; namely, (a))(a))
Ci = L 1i + A + Aoco + As
(1)
where CR, LA. Lc, Ls are the reconstruction, adversarial, cycle, and smoothness losses, respectively. The first term LR is standard reconstruction loss and is calculated for the two endpoints æ; and æz;
R = L(x,, .;) + La,, k.))
where -(.,-) some loss function between the two images (we used the L2 distance or the perceptual loss (Johnson et al. 2016)) and d = {(f(z=)). LA is the adversarial loss that encourages the network to fool the discriminator so that interpolated images are indistinguishable from the data in the target domain X::
M
C
Σ
log D ((d=-)(n/MM)
where ((x)) [0, is discriminator trying to distinguish between images in the training set and the interpolated images. The cycle-consistency loss Lo encourages the encoder and the decoder to produce bijective mapping:
c
Σ (
- bn
where (a) = f(g(2) (a))). The last term Ls is the smoothness loss encouraging +(a) to produce smoothly varying interpolated points between 2;; and æy:
di ( Oa
c
==0
n/M
where
means that the derivative it taken at a = Og- The three losses CA, Le and Ls are accumulated over M sampled points, from a 0/M up to a = M/M. Finally, we sum the (i loss over many sampled pairs.
In the next section, we explain the motivation for each of the four losses comprising (i in Equation I and describe how these losses promote the four conditions defined in Section
2.2 JUSTIFICATION FOR THE PROPOSED APPROACH
Figure 3 iilustrates the justification for introducing the four losses. As seen in Plot A in Figure B the images Ø;, Z.j. which lie on the data manifold in the image space (solid black curve), are mapped back the original images thanks to the reconstruction loss L Li This loss promotes the boundary conditions defined above. The reconstruction loss, however, is not enough as it neither directly affects in-between points in the image space nor the interpolated points in the latent space. Introducing the adversarial loss C prompts the decoder (()+=)(a)) to map interpolated latent vectors back into the image manifold (Plot B). Considering the output of the discriminator D(x) as the probability of image x to be in the target domain X' (namely, to be on the image manifold), the adversarial loss promotes the credibility condition defined above. As indicated in Plot B, the encoder / f(x) (red
4

---

Under review as a conference paper at ICLR 2021
f
2,
g
{(a)
((01))
z(a)
8
(()
(0.13
x)
9
Figure 4: Our proposed architecture. Dotted lines represent the loss functions. h is non-learned layer that performs latent linear interpolation.
arrows) might, nevertheless, still map in-between images to latent vectors that are distant from the linear line the latent space. Adding the cycle-consistency loss f forces the reconstruction of interpolated latent vectors to be mapped back into the original vectors in the latent space (Plot C). The adversarial and sycle-consisteng, losses encourage bijective mapping (one-to-one and onto) between the imput and the latent manifolds, while providing realistic reconstruction of interpolated latent vectors. Lastly, the parameterization of the interpolated points, namely, a E [0, 1]. does not necessarily provide smooth interpolation in the image space (Plot C); constant velocity interpolation in the parameter a may not generate smooth transitions in the image space. The smoothness loss Cs resolves this issue as it requires the distance between and 3j to be evenly distributed along a E [0, (due to the L2 norm). This loss fulfillss the smoothness condition defined above (Plot D). If we consider the latent representation as normed space representing the manifold distance d(r;;, vT) = ||= z, the linear interpolation in the latent space also satisfies the monotonicity condition defined above.
2.3
IMPLEMENTATION
The proposed architecture is visualized in Figure 4 At each iteration, we sample two images from our dataset. The two images (z,, 2zy) are encoded by the encoder f into (z,, zy). respectively. We sample uniformly between [0, 1] and pass (a. 2;, 2,) to h, a non-learned layer, which calculates the linear interpolation in the latent space, namely, Zi-z (a) = (1 a)z; + azj. We then decode 2,, 2j and calculate the reconstruction loss L 3_ Subsequently, we decode (a=)(a:)) and alternately provide the discriminator 1 with samples either from the training
from (.)+((a) = (a)) The discriminator is optimized using the standard GAN loss and is updated after every iteration. We then calculate the smoothness loss - by taking the derivative of a );))((a) with respect to Finally, we pass (a))(a)) through the encoder f obtain (,))(a) = ( for the cycle-consistency loss and add the loss f (==(()))))
The chosen encoder architecture VGG-inspired (Simonyan & Zisserman 2014). We extract the features using convolutional blocks starting from 16 feature maps, gradually increasing the number of feature maps reach 128 by the last convolutional block. We then flatten the extracted features and pass them through fully connected layers until we reach our desired latent dimensionality. The decoder architecture symmetrical that of the encoder. We use max-pooling after each convolutional block and batch normalization with ReLU activations after each learned layer. A random 80%-20% wraning-testtin split was chosen for all experiments, using the same batch size and total number of examples in the dataset. During log grid- search hyperparameter optimization, we found that Xa Ào 1 and Äs 10-1 produce the best results. All experiments were performed using single NVIDIA V100 GPU.
RELATED WORK
In its simplest version, the autoencoder (Goodfellow et al. 2016 Kramer 1991) trained obtain a reduced representation of the input, removing data redundancies while revealing the underlying factors of the data set. The reduced space, namely, the latent space, be viewed 'useful"
5

---

Under review as a conference paper at ICLR 2021
representation space in which data interpolation can be attempted. Many autoencoder improvements have been proposed in recent years, including new techniques designed for improved convergence and accuracy. Among these are the introduction of new regularization terms, new loss objectives (such as adversarial loss) and new network designs (Doersch] 2016 Kingma & Welling, 2013 Larsen let al.] 2015 Makhzani al. 2015 Vincent et al. [ Gooofellow et al. [016)]
| Vincent al. 2010).
Other new autoencoder techniques provide frameworks that attempt to shape the latent space to be efficient with respect factor disentanglement or make it conducive to latent space interpolation
(Kingma & Welling 2013 Bouchacourt et al., 2017 Vincent et al., 2008 Yeh et al. 2016 Higgins
ct al. 2016).
Within this second category, the variational autoencoder (VAE) and its derivatives were shown to be very successful in applying interpolation the latent space, in particular for multimodal distributions, such as MNIST. The KI. term in the VAE loss tends to cluster the modes in the latent space close to each other (Dieng et al.] 2019). Consequently, linearly interpolating between different modes in the latent space may provide pleasing results that smoothly transition between the modes. Unfortunately, this cannot be applied to data points whose generating factors are continuous (in contrast to multimodal distributions) given that the KL loss term tends to fold the manifold tightly into a compact space making it highly non-convex.
Berthelot et al. (0018) propose using critic network to predict the interpolation parameter a E [0, 1] while an autoencoder is trained to fool the critic. The motivation behind this approach is that the interpolation parameter can be estimated for badly-interpolated images, while it is unpredictable for faithful interpolation. While this approach might work for multimodal data, it does not seem to work for data sampled from continuous manifold. In such cases, the artifacts and the unreallstti-gennrrtte data do not provide any hint about the interpolating factor.
The GAIA method of Sainnurg et al. 2018) uses BEGAN architecture composed of generator and a discriminator, both based on autoencoders. The discriminator is trained to minimize the pixel-wise loss of real data and to maximize the pixel-wise loss of generated data (including interpolations). On the other hand, the generator is trained to minimize the loss of the discriminator for the interpolated data. The GALA method is devoted to synthesizing loookigic-ookking images while ignoring the objective of image diversity and the need for smooth transitions between data points.
Perhaps the method most similar to our approach is the adversarial mixup resynthesis (AMR) of Beckham et al. (2019). With the AMR method, decoded mixup of latent codes Mix(zi, zÿ) are encouraged to be indistinguishable from real samples by fooling a trained discriminator. This is similar to the adversarial loss introduced in our framework. Nevertheless, as claborated in Section [2.2 and illustrated in Figure 3 (Plot B), the adversarial loss alone only amounts to generating realistic- looking interpolations, where the latent space is prone mode collapse and sharp transitions along the interpolation paths.
In contrast to these methods, our additional smoothness and cycle-consistency requirements not only generate smooth transitions between data points but also ensure a diverse generation of realistic- looking images while avoiding mode collapse and sharp transitions along the interpolating paths. This characteristic will be demonstrated in Section 4 and in the ablation study provided in the appendix.
4 RESULTS
Evaluating the realisticality of interpolation is often elusive. In the unsupervised scenario, where the ground-truth parameterization is unavailable, defining a path between two points pa. Py in the parameter space depends on the parameterization of the underlying factors governing the data, which unknown. For example, our synthetic pole dataset, the parameter space (0, Φ) and there are infinitely many possible paths between any two points in that space, each of which can yield an admissible interpolation. Nevertheless, we evaluate the interpolation faithfulness both qualitatively and quantitatively on various datasets based on the conditions we defined in Section 2
4.1 DATASET
We tested our method against two different datasets: the synthetic pole dataset, which was rendered using the Unity game engine, where all images were taken from a fixed viewing position (from above)

---

Under review as a conference paper at ICLR 2021
AEAI
AAE
ACAI
B-VAE
AMR
GAIA
Figure 5: Each of the six rows presents linear interpolation of images from COIL-100 and our synthetic dataset for each of the methods tested.
and the COIL-100 dataset. For the first dataset, single illumination source was rotated intervals of 5 degrees along the azimuth different altitudes, ranging from 45 to 80 degrees with respect to plane in 5-degree intervals. This dataset contains total of 576 images. In the second dataset, to test our method against real images with complex geometric and photometric parameterization, used the COIL-100 dataset (Nene et al. 1996) containing color images 100 objects. The objects placed motorized turntable against black background. The images were taken intervals of 5 degrees resulting in a total of 72 images for each class. All architectures were trained using a dataset that contained single class from COIL-100 the synthetic pole dataset.
4.2 QUALITATIVE ASSESSMENTS
Each one of the six rows in Figure presents a linear interpolation of a single object from the COIL-100 dataset (left) and our pole dataset (right) sampled from the validation dataset. We compared the results of the 3- V-Variational Autoencoder (3-VAE) (Higgins et al. (0166). the Adversarial Autoencoder (AAE) Makhzani et 2015), the Adversarially Constrained Autoencoder Interpolation (ACAI) (Berthelot et al. [2018). the Generative Adversarial Interpolative Autoencoding method (GAIA) (Sainburg et al. 2018). the Adversarial Mixup Resythhesis (AMR) (Beckham et al. 2019) and our approach-Autoentee Autoencoder Adversarial Interpolation (AEAI). In the experiments with both datasets, we used a latent dimensionality of 256. From Figure 5 it can be seen that our proposed method provides realistic-looking reconstructions and an admissible interpolation between modes. The AAE and 3-VAE interpolations change abruptly between modes and introduce small artifacts during reconstruction. The ACAI produces unrealistic transitions and artifacts during reconstruction, especially in the mid-range of the a-values. The GAIA method produces realistic transitions with small artifacts during reconstruction while the AMR produces significant artifacts and gradual transitions between modes. Additional qualitative results and an ablation study are presented in the Appendix.
Alpha
Alphe
Interval
Our Approach
0333
Acversarial Autoencoder
ACAI
Bata
MSE (M02))
AMR
0.000
GAIA
0.82
6.50
0.00
0.50
0.2
20
03
0.5
Aphha
Alpha
Isterval (Dagreos)
Figure 6: use the parameterization of the dataset to evaluate the reconstruction accuracy of the AAE, ACAI, 3-VAE, AMR, GAIA and our proposed method. Left graph: Averaged MSE vs. values. Middle graph: STD of MSE vs. a values. Right: Averaged MSE of the interpolated images vs. the interval length.
7

---

Under review as a conference paper at ICLR 2021
4.3 QUANTITATIVE ASSESSMENTS
For quantitative comparison we used the COIL-100 dataset. We fixed an interval length, which is a multiplicative of degrees, and calculated the reconstruction error (MSE) against the available ground-truth images. We used interval length of 80 degrees that resulted in 14 in ne-between images of a single object. The reconstruction error the interpolated images is presented in Figure 6 Clearly, method reduces the mean MSE and the standard deviation of the MSE for different alpha values. We then inspected the average reconstruction error on multiple intervals ranging from to 80 degrees as presented in the right part of F Note that our proposed method is able to reduce the reconstruction emrrr of interpolated images consistently even when the interval length increases.
Our Approach
Adversarial Autoencoder
Predicted Alpha
Predicted Alpha
04
64
0.0
02
0.6
08
10
0.0
02
0.4
0.6
088
2.0
True Alpha
True Alpha
ACAI
Beta VAE
0.
0.3
Predicted Alphe
Predicted Alpht
0.
00
64
True Alpha
True Alpha
AMR
GAIA
Perfect Prediction
Predicted Median Alpha
6.
6.3
Predicted Alpha
Predicted Alpht
6G
60
0.0
0.6
Alpha
Alpha
Figure 7: Predicting the interpolated alpha value based on the L2 distance the interpolated image to the closest image in the dataset. The dots represent the median and the colored area corresponds to the interquartile range.
To assess the transition smoothness from one sample to the other, we compared each interpolated image (;));(a) to the closest image in the dataset in terms of the L, distance and assigned the alpha value for the interpolated image according to the retrieved image. We repeated this process for all the intervals of length 70. Figure presents the scatter diagrams for each method. It is demonstrated that our framework consistently retrieves the best value alpha with smaller interquartile range (IQR). The next experiment was applied to the synthesized pole dataset. As above, we retrieved the closest image terms of MSE in the image space, and measured the L2 distance the parameter space between the interpolated image and the source image (a and between the interpolated image and the target image (a = 1). We repeated this process on multiple intervals of different lengths on both 0 and and present the average distance from the source and target images function of
8

---

Under review as a conference paper at ICLR 2021
Our Approach
AAE
ACAI
L2 Parameier Distance
Beta VAE
AMR
GAIA
L2 Parameter Distance
Perfect interpolation Smoothness Source Target
Alpha
Alpha
Alpha
Figure 8: We sampled two images I₁ Fyy and linearly interpolated between them in latent space. For each interpolated image, we retrieved the closest image terms of MSE from the dataset. The blue and orange lines present the averaged L2 distance, in the parameter space (0, Φ), between the retrieved image and Tú Iy, respectively. The red lines represent perfect interpolation smoothness.
the interpolation variable, a. Figure g shows the results for each tested method. It demonstrated that all methods exhibit monotonicity characteristics; however our approach outperforms the other methods with respect the smoothness of the parameterization. Our approach tightly follows the linear parameterization due to the explicit smoothness term we incorporated into the training scheme.
5 CONCLUSION & DISCUSSION
The problem of realistic and faithful interpolation in the latent spaces of generative models has been tackled successfully that deal with manifold data not common multimodal data, and this misinterpretation of manifold data harms the competence of generative models to deal with them successfully. In this work, argue that the manifold structures of data generated from continuous factors should be taken into account. Our main contribution is applying convexity regularization using adversarial and cycle-consistency losses. Applying this technique on small datasets of images, taken from various viewing conditions, we managed to greatly improve the fidelity of interpolated images. We also implemented smoothness loss and improved the non-uniform parameterization of the latent manifold. In future work, we intend to further investigate properties of latent manifolds, in particular, capable of generating admissible interpolation between both categorized and continues data, and use the proposed approach as a general regularizer method for generative models with few training examples.
last few years. Nevertheless, our opinion that generative approaches
REFERENCES
Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In Advances in neural information processing systems, pp. 4346-4357, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8), 2013.

---

Under review as a conference paper at ICLR 2021
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. arXiv preprint arXiv:1807 007.07543, 2018.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoen- coder: Learning disentangled representations from grouped observations. arXiv preprini arXiv: 1705.0810, 2017.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397-2405. PMLR, 2019.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.0988 2016.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Arxiv, 2016.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In The European Conference on Computer Vision (ECCV), 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114,222 2013.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-243, 1991.
Anders Boesen Lindbo Larsen, Søren Kaac Sønderby, Hugo Larochelle, and Ole Winther. Au- toencoding beyond pixels using learned similarity metric. arXiv preprint 11X1v:151..09300, 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 1 2015.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in neural information processing systems, pp. 5040-5048, 2016.
Sameer A. Nene, Shree K. Nayar, and Hiroshi Murase. object image library (coil-100). Technical report, 1996.
Tim Sainburg, Marvin Thielk, Brad Theilman, Benjamin Migliori, and Timothy Gentner. Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions. arXiv preprint 1007,0680706666 2018
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling shape and appearance. The European Conference on Computer Vision (ECCV), September 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.156. 2014.
Vikas Verma, Alex Lamb, Christopher Beckham, Aaron Courville, Ioannis Mitliagkis, and Yoshua Bengio. Manifold mixup: Encouraging meaningful on-manifold interpolation regularizer. arXiv preprint arXiv: 1eo...028600.. 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1100 103. ACM, 2008.
10

---

Under review as a conference paper at ICLR 2021
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations a deep network with local denoising criterion. Journal of machine learning research, ((Deec)333713340S 2010.
Raymond Yeh, Ziwei Liu, Dan B Goldman, and Aseem Agarwala. Semantic facial expression editing using autoencoded flow. arXiv preprint 1rXiv:161 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.

---

Under review as a conference paper at ICLR 2021
APPENDIX
A ADDITIONAL RESULTS
In this section, we provide additional results showing the interpolation behavior of different architec- tures we studied. Each of the following figures is composed of six blocks, each of which presents a bilinear interpolation of the latent representation of four ground truth images that reside in each corner of the block. Figures 2
and T0 demonstrate bilinear interpolation on a differnet COIL-100 objects and Figure TT shows the interpolation results on our synthetic pole dataset.
We demonstrate that our technique produces admissible interpolation while other techniques fail to reconstruct in-between images realistically or to transition smoothly from mode to mode. For example, in Figure I we show that the AMR, AAE and the ACAI create a cross-dissolve effect during interpolation while the interpolation of the 3-VAE and GAIA method changes abruptly while showing little progression between interpolation frames. Additionally, in Figures that the AAE and AMR exhibit artifacts in interpolated images while the ACAI shows blurry images during reconstruction. The 3-VAE and GAIA shows mostly realistic reconstructions; however, the transition is not smooth nor consistent.
9 and I0 we show
B ABLATION STUDY
We present an ablation study of our unsupervised interpolation framework presented in Figure above. As seen in the bottom left part of Figures 12 and [3] without significant contribution from the discriminator, the reconstructed images resulting from interpolating latent vectors are unrealistic and exhibit severe artifacts and non- ooootho transition between modes. Without the yycle-consistency loss, interpolated images are relatively realistic; however, they change modes abruptly, exhibiting artifacts when transitioning from one mode to another. Adding both cycle consistency and the discriminator results in realistic transitions from mode to mode as can be seen in the bottom right part of Figure [2] Nevertheless, there are instances of consecutive interpolated images that show little to no change as can be seen in the two lefumost columns of the bottom right part of Figure [2] When introducing the we get both smooth transitions and
smoothness loss, as can be seen in the top left part of Figure 12 realistic reconstructions of interpolated latent vectors.
We present a quantitative analysis of our ablation study in Figures 14 and [5] For each case, we demonstrate the average MSE error between the interpolated image and ground truth images retrieved from our dataset. In the top part of Figure 14 we fix an interval length of 80 degrees and iterate over all such intervals in dataset. We split the interval into 16 images separated by 5 degrees and obtain 14 in-between images. For every interpolated image we retrieve the corresponding ground truth image in our dataset and present the average MSE and standard deviation on all such intervals. In the bottom part of Figure 14 we repeat this process on multiple interval sizes ranging from 15 degrees to 80 degrees. We show that each element in our loss function I contributes reducing the mean and variance of the reconstruction error. We further support our justification presented in Figure by showing that the largest contribution to the reduction the MSE stems from the introduction of the discriminator, which keeps the reconstruction of interpolated images consistent with the dataset. The addition of the socle-consistency and smoothness losses further improves the results by encouraging smooth bijective mapping.
In Figure 15 we present the results of predicting the interpolated alpha value by querying the dataset for the closest image in terms of the Le distance to the interpolated image. We present the median for each alpha on all intervals with the corresponding interquartile range. The red line demonstrates the perfect retrieval of the predicted alpha value. It is shown that the absence of the discriminator greatly affects the interpolation faithfulness while the addition of the cycle-consistency and smoothness losses contributes to the consistency of retrieving the correct alpha value.
12

---

Under review as a conference paper at ICLR 2021
Our Approach
Adversarial Autoencoder
ACAI
Beta VAE
AMR
GAIA
Figure 9: Each of the six blocks shows a bilinear interpolation of four ground truth images that reside in each corner of the block.
13

---

Under review as a conference paper at ICLR 2021
Our Approach
Adversarial Autoencoder
ACAI
Beta VAE
AMR
GAIA
Figure 10: Each of the six blocks shows bilinear interpolation of four ground truth images that reside in each corner of the block.
14

---

Under review as a conference paper at ICLR 2021
Our Approach
Adversarial Autoencoder
ACAI
Beta VAE
AMR
GAIA
Figure 11: Each of the six blocks shows bilinear interpolation of four ground truth images that reside in each corner of the block.
15

---

Under review as a conference paper at ICLR 2021
Our Approach
No Cycle Consistency
No Discriminator
No Smoothing
Figure 12: Each of the four blocks presents bilinear interpolation of four ground truth images that reside in each corner of the block. Top left: Bilinear interpolation results of our approach with all loss components. Top right: Removing the cycle-consistency contribution from the loss function. Bottom left: Removing the discriminator contribution. Bottom right: Removing the smoothing contribution.
16

---

Under review as a conference paper at ICLR 2021
Our Approach
No Cycle Consistency
No Discriminator
No Smoothing
Figure 13: Each of the four blocks presents bilinear interpolation of four ground truth images that reside in each corner of the block. Top left: Bilinear interpolation results of our approach with all loss components. Top right: Removing the cycle-consistency contribution from the loss function. Bottom left: Removing the discriminator contribution. Bottom right: Removing the smoothing contribution.
17

---

Under review as a conference paper at ICLR 2021
MSE-MEAN vs Alpha
MSE-STD vs Alpha
Our Approach
No Cycle Consistency
Discriminator
No Smoothness
0.004
MSE (Mean)
0.003
MSE (STD)
0.003
0.001
0.000
0.000
Alpha
Alpha
3.007
Our Approach
Cycle Consistency
8.006
Discriminator
No Smoothness
0.005
Average MSE
0.004
0.002
Interval (Degrees)
Figure 14: Top graph: Average reconstruction error and standard deviation vs. a values. Bottom: Average MSE of the interpolated images vs. the interval length.
Appreach
Cycle Consistency
1.0
Pertect Alpha Prediction
15
Predicted Median Alphe
Predicted Mlsha
0.6
0.8
Predicted
6.4
66-
True Alpha
Tree Alpha
No Discriminator
No Smoothness
1.-
15-
Precicted Alpha
0.8
044
68
00
True Alpha
True Alpha
Figure 15: Predicting the interpolated alpha value based on the L2 distance of the interpolated image the closest image the dataset. The dots represent the median and the colored area corresponds to the interquartile range.
18


Reviews:
[{"review_id": "mQvGWbZMOQU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper proposes new regularization terms (smoothness and cycle consistency) for more realistic latent interpolations in auto-encoders. It is experimentally tested by measuring interpolation error on two datasets: a new custom pole dataset introduced by the paper and the COIL-100 dataset.\n\n# Pros\n1. The method feels intuitively motivated and makes sense.\n2. The results appear convincing compared to the (limited choice of) baselines.\n3. Ablation study helps comprehend the contribution of each loss term.\n\n# Cons\n1. No direct comparison to state of the art baselines, namely GAIA and AMR.\n2. Results only on artificial tasks. For example, there’s no measure of the effects of the better conditioned latent space on downstream tasks such as a classification on SVHN or CIFAR10 like some other works in the domain checked.\n3. Novelty is not clearly identified: a good place would be the end of section 1, e.g. “Our contributions are …”. I deduced it was the cycle consistency loss (which seems inspired from cycle GANs) and smoothness loss since the other loss terms look like what previous techniques already do.\n4. Ablation should really be in the main section of the paper since it’s important. On the other hand Figures (3) and (4) which didn’t really add to my comprehension could be removed to save space.\n5. Without the appendix the paper is lacking essential information.\n6. The bibliography seems to be lacking (see questions and nits)\n\n# Questions and nits\n1. “This regularization ... can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation.” It would be good to see it demonstrated in results.\n2. In several places, you cite auto-encoders but really refer to variational auto-encoders, not crediting earlier works.\n3. “Researchers have demonstrated the ability to interpolate between data points by decoding a convex sum of latent vectors (Shu et al., 2018) ...”  I believe there’s prior art before that, for example, just to name one: https://arxiv.org/abs/1611.03383 \n4. “... interpolated points to look reliable as it is optimized ...” In several places the word “reliable” is used, I suppose you meant “realistic”. If not can you clarify what it means?\n5. “... while providing a convex latent manifold with a bijective mapping between the input and latent spaces.” It seems counter intuitive to me, the input space is much larger than the latent space, how can it be bijective? In addition you later introduce $L_c$ the cycle consistency loss which tries to map $f(g(z_{i,j,\\alpha}))$ to $z_{i,j,\\alpha}$, in other words it looks surjective only. I understand that perceptually you want $g(f(x))$ to be close to $x$, but they are not the same, or are they? If you are referring to the mapping from the latent space onto itself by $f(g(.))$, then the term `identity mapping` would seem clearer to me.\n6. “Credibility” I don’t understand why it is written in such a complicated manner. Why not simply write $P(\\hat{x}_{i,j,\\alpha}) \\geq 1 - \\beta$ for a constant $\\beta \\geq 0$. \n7. Concerning the smoothness loss $L_s$ it seems to be derived from the k-Lipschitz constraint (3). But in its implementation it’s really 0-Lipschitz, it would be great to clarify this aspect. \n8. In addition, concerning $L_s$, the formulation seems odd. I’d expect it to be written as $\\sum ||\\frac{d\\hat{x}_{i,j}(\\alpha)}{d\\alpha}(n/M)||^2$ since in its current form $\\alpha$ does not appear in the numerator.\n9. One last thing concerning $L_s$, it would be good to give the reader a simple explanation: if I’m correct you’re simply minimizing the pixel distance between successive images on the interpolation line. That would be really helpful to state it clearly - assuming I understood correctly.\n10. “2.2 JUSTIFICATION FOR THE PROPOSED APPROACH”. I found it was already justified by the intro to section 2 for the most part. One exception is the justification for the cycle consistency loss L_c which was not covered in the introduction of section 2. So you could simplify/remove this section by moving it to section 2.\n11. On the topic of L_c, it is justified as “the cycle-consistency loss L_c forces the encoder-decoder architecture to map linearly interpolated latent vectors onto the image manifold …”. I don’t see why it should. As I understand it, the loss term itself only seems to say that it should map to an image that projects to the same latent representation. This could use clarification.\n12. In the comparison to other methods, did you use the same number as training examples? I’m asking this question because the loss term L_s has an implicit batch size of M which could lead your method to see M times more samples than the methods under comparison (in particular AAE and ACAI).\n13. “The two images (xi, xj) are encoded by the shared-weight encoder…”. I didn’t follow what the weights are shared with.\n14. The loss terms are called $L_{a,c,s}$ but the weighting hyper-parameters are called $\\lambda_{1,2,3}$. Using the same letter as their corresponding loss term instead of 1,2,3 would make reading more friendly.\n15. “The GAIA method of Sainburg et al. (2018) is similar in spirit to the AMR framework.” The GAIA method came first, so this is the other way around (B is similar in spirit to A, if B came after A). Actually it should probably be presented before AMR. \n16. “but also ensure a diverse generation ... while avoiding mode collapse“. How does it ensure it? I didn’t really get what loss terms directly prevented modal collapse nor diversity? Or if the effect is indirect, could you give more insights on what causes it?\n17. The results section comes just after Related Work and it’s striking that there’s no comparison to the two methods yours is most similar to and that were covered sentences ago. \n   1. I understand you claim that “Comparisons with AMR and GAIA methods (Beckham et al., 2019; Sainburg et al., 2018) are analogous to the ablation study presented in the Appendix, where the smoothness and cycle-consistency losses are missing.” but it’s not clear whether you ran the experiment and observed it was indeed analogous or whether it should be analogous.\n   2. In addition, in the appendix, I didn’t see a curve in the ablation study with *both* smoothness and cycle-consistency loss removed.\n   3. Not presenting this result with the other methods and putting it in the appendix gives an incomplete and possibly misleading impression to someone quickly eyeballing results like many readers tend to do. I think it is highly detrimental to the results presentation.\n18. On the COIL-100 dataset, it is not clear whether during training you used interpolations only between objects of the same class or between classes as well.\n19. “Results on other datasets can be seen in the Appendix.” What other datasets? I seem to only see results for COIL-100 and the pole dataset.\n20. In addition it would be interesting to see what happens for interpolations between classes.\n\n=====POST-REBUTTAL COMMENTS======== I thank the authors for the response and the efforts in the updated draft. Most of my queries were clarified and I raised my rating accordingly. I understand the authors view that some of my queries fall outside their desired scope for the paper, however I still think the paper could benefit from such contents.", "extra_scores": {}, "extra_text": {}}, {"review_id": "i_SPGZ1kER5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces several autoencoder (AE) regularization terms that aim at reproducing continuous realistic deformation by interpolating latent codes of images. The authors assume there is a continuous process generating the data and introduce three novel loss terms (in addition to the standard AE reconstruction loss). The first term is a GAN loss for decoded interpolated latents $\\hat{x}(\\alpha)$ where this terms makes sure the interpolated latents are decoded to images similar to the train images. The second term is called cycle-consistency and enforce injectivity of the decoder. The last term is enforcing smoothness of the decoded image as a function of the interpolated latent. Combining these three losses with the original loss leads to natural interpolations of latents that enjoy both smoothness and realism.  The method is tested on a synthetic \"pole shadow\" example, and COIL 100. The method seem to improve upon several baselines on these datasets. \n\nMore details:\n\n- The method is simple and able to improve latent interpolations. I think the choice of losses and properties they enforce is well explained.  The results are convincing and seem to provide a good arrangement of the latent space. \n- In terms of contributions: the incorporation of a discriminator loss in training of AE to provide more realistic interpolations was done before, as the authors acknowledge (e.g., Beckham et al. 2019). This diminish some of this paper's contribution. However, as shown in Figure 12, discriminator loss alone does not solve the continuous latent problem.\n- In Figure 12, I cannot see the benefit in the smoothing term, that is, top-left and bottom-right images look almost identical to me. Can you state if there is some difference?  If not, did you encounter a benefit in incorporating the smoothness term somewhere else? \n- Are the corners in the square interpolation grids (e.g., Figures  5, 9, 10, 11, 12, 13) train or test examples? How does grid interpolation of test examples look? The last question is interesting both for in-distribution test examples, as well as out-of-distribution test examples. \n- How does the discriminator $D$ is trained? Do you use standard GAN loss? Missing info here. \n- How do you set the different parameters $\\lambda_i$ of the model?\n- How is the loss $\\mathcal{L}_S$ approximated? Do you use automatic differentiation, numerical differentiation? Do you use stochastic approximation?  \n- Figure 8 was not clear to me.\n- Why L1 distance of images is used in 4.2 and L2 in 4.3?\n- Minor: typo \"qualitatively and qualitatively\" ; distorted text in Figure 6\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eE4IIJeaYCa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focused on developing a new regularization technique for autoencoders, which shapes the latent representation to follow a manifold that is consistent with the training images and that drives the manifold to be smooth and locally convex. The authors suggest that the manifold structure of continuous data must be considered to include the geometry and shape of the manifold. The new interpolation regularization mechanism consists of an adversarial loss, a cycle-consistency loss, and a smoothness loss. So the architecture of the proposed model includes a standard autoencoder, a discriminator and the loss mentioned above. \n\nStrengths: The motivation of using four losses were clearly described and the architecture of the model was straightforward. The authors tested the proposed AEAI method on one synthetic plot dataset and COIL-100 dataset and compared the results with three other models. The effectiveness of the proposed technique is tested by both visual inspection and comparing the reconstruction error against the available ground-truth images. The authors also examined the transition smoothness from one sample to the other on both datasets. The comparison of the alpha values showed that the proposed technique did generate more smooth results than other methods. By visual inspection, the results looked pretty good for the proposed technique while other interpolations methods change abruptly between modes and introduce smallartifacts during reconstruction. The evaluation of MSE also showed the proposed technique achieved lower MSE. In general, I think the analysis of the proposed technique was convincing, and the authors also provided ablation study with smoothness and cycle-consistency losses missing, showing the necessity of adding these losses.\n\nWeakness: The proposed model is a mixture of known techniques, without a unifying theme. The authors can try to compare the latent space directly with manifold learning techniques or autoencoders that directly penalize using manifold learning. One example is the recent paper at ICML https://arxiv.org/abs/2002.04881 (Chen et al. \"Learning Flat Latent Manifolds with VAEs.\" ICML 2020) or https://www.sciencedirect.com/science/article/abs/pii/S1063520317300957 (Mishne et al, Diffusion Nets, Applied and Computational Harmonic Analysis.\") The authors also don't quantify the manifold learning capability of the latent space. A potential metric for this is the Denoised Manifold Affinity-Preservation Metric (DeMAP) proposed in Moon et al., Nature Biotechnology 2019. I would also suggest visualizations of the latent space using the PHATE technique from the same paper.  \n\nI also feel that the authors should try a more irregular/noisy dataset to further show the effectiveness of the model. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QZbLhNAmTX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper presents a method of regularising the latent space of an Autoencoder in a way that pressures the data manifold to be convex. This allows interpolation within the latent space which does not leave the data manifold and results in realistic reconstructions as one moves from one point to another.\n\nThis is done through the introduction of adversarial and cycle-consistency losses, over and above the usual reconstruction loss and a smoothness loss. The adversarial loss ensures that the interpolated reconstructions are realistic, while cycle-consistency encourages a bijective mapping.\n\n## Quality & Clarity\n\nThe paper is clearly written and without typographical or grammatical errors. It is structured logically and the authors' arguments are easily followed.\n\nThe results are compelling and the proposed AEAI technique clearly outperforms other methods in the qualitative experiments, with quantitative results to substantiate it.\n\n## Originally & Significance\n\nThe contribution of the paper is clear in that it imposes convexity regularisation to the latex space. The approach is compared with modern competing techniques and the work is well positioned among recent literature in the field.\n\n## Outcome\n\nThis is a clear, high-quality paper with compelling results.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a technique called Autoencoder Adversarial Interpolation (AEAI). The key idea is to train autoencoder architectures that explicitly "shapes" trajectories in the encoder (latent) space to correspond to smooth geodesics between data points. This is achieved by a combination of several loss terms that are fairly intuitive. The authors empirically justify each term via ablation studies on simple datasets.

Initial review scores had wide variance. The reviewers liked the overall approach as well as the clarity with which the theory and experiments were presented, but raised several concerns. The authors provided succinct responses that seem to have satisfied the reviewers on average.

Unfortunately, after having carefully read this paper (and the authors' responses), I have to go against the wishes of the majority of the reviewers, and recommend a reject. My two main concerns are as follows:
a) The synthetic pole dataset, as well as the COIL-100 dataset, are far too simplistic to evaluate performance. It is now standard to report results on considerably more challenging datasets.
b) Echoing R2 -- the authors should articulate why a shaped latent space should actually matter in applications, beyond giving intuitive(I guess?) visualizations and reconstruction error curves. Results on downstream tasks may be one avenue to achieve this.
Decision: Reject