Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck
Authors: Ilja Manakov
Markus Rohm
Volker Tresp
Venue: Submitted to ICLR 2023 (2023)
Keywords: autoencoders
unsupervised learning
representation learning
investigation

Abstract: In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck.
Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox.
Researchers and practitioners employ CAEs for various tasks, ranging from outlier detection and compression to transfer and representation learning.
Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the CAE's emergent properties.
We demonstrate that increased bottleneck area (i.e., height $\times$ width) drastically improves generalization in terms of reconstruction error while also speeding up training.
The number of channels in the bottleneck, on the other hand, is of secondary importance.
Furthermore, we show empirically that CAEs do not learn to copy their input, even when all layers have the same number of neurons as there are pixels in the input (i.e. there is no bottleneck).
Besides raising important questions for further research, our findings are directly applicable to two of the most common use-cases for CAEs:
In image compression, it is advantageous to increase the feature map size in the bottleneck as this greatly improves reconstruction quality.
For reconstruction-based outlier detection, we recommend decreasing the feature map size so that out-of-distribution samples will yield a higher reconstruction error.

Under review as a conference paper at ICLR 2023
WALKING THE TIGHTROPE: AN INVESTIGATION OF THE CONVOLUTIONAL AUTOENCODER BOTTLENECK
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck. Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox. Researchers and practitioners employ CAEs for various tasks, ranging from outlier detection and compression to transfer and representation learning. Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the CAE's emergent properties. We demonstrate that increased bottleneck area (i.e., height x width) drastically improves generalization in terms of reconstruction error while also speeding up training. The number of channels in the bottleneck, the other hand, is of secondary importance. Furthermore, we show empirically that CAES do not learn an identity mapping, even when all layers have the same number of neurons as there are pixels in the input. Besides raising important questions for further research, our findings are directly applicable to of the most common use-cases for CAEs: In image compression. it is advantageous to increase the feature map size in the bottleneck as this improves reconstruction quality greatly. For reconstruction beaadd outlier detection, we recommend decreasing the feature map size so that out-of-distribution samples will yield higher reconstruction error.
INTRODUCTION
Autoencoders (AE) are an integral part of the neural network toolkit. They are class of neural networks that consist of an encoder and decoder part and are trained by reconstructing datapoints after encoding them. Due to their conceptual simplicity, autoencoders often appear teaching materials as introductory models to the field of unsupervised deep learning. Nevertheless, autoencoders have enabled major contributions in the application and research of the field. The main areas of application include outlier detection Xia et al. (2015); Chen et al. (2017), Zhou & Palfenroth (2017): Baur et al. (2019), data compression Yildirim et al. (2188: : Cheng et al. 2008): Dumaa et al. 2018), and image enhancement Mao al. 2016); Lore et al. (2017). Additionally ly, autoencoders can used catalysts in the training of deep neural networks. The layers of the target network can be greedily pre-trained by treating them autoencoders with hidden layer [Bengio al. (2007). Subsequently, |Erhan et al. (2009) demonstrated that autoencoder pre-training also benefits generalization. Currently, researchers in the field of representation learning frequently rely on autoencoders for learning nuanced and high-level representations of data Kingma & Welling (2013) Tretschk et al. (2019); Shu et al. (2018):
Makhzani al. (015): Berthelot al. (2018).
However, despite its widespread use, we propose that the (deep) autoencoder model is not well understood. Many papers have aimed to deepen our understanding of the autoencoder through theoretical analysis Nguyen et al. 2018); Arora et al. (2013); Baldi (2012); Alain & Bengio 2).. While such analyses provide valuable theoretical insight, there is significant discrepancy between the theoretical frameworks and actual behavior of autoencoders in practice, mainly due the assumptions made (e.g., weight tying, infinite depth) or the simplicity of the models under study. Others have approached this issue from (2013): Vincent et al. (2008); Berthelot et al. (2019); Radhakrishhan et al. (2018). Such investigations part of an ongoing effort to understand the behavior of autoencoders in variety of settings.
experimental angle Arpit et al. (2015): [Bngio al. (2013a) Le
The focus of most such investigations so far has been the traditional autoencoder setting with fully connected layers. When working with image data, however, the default choice is to use convolutions,

---

Under review as a conference paper at ICLR 2023
as they provide prior that is well suited to this type of data Ulyanov al. (2018). For this reason, Masci et al. (2011) introduced the convolutional autoencoder (CAE) by replacing the fully connected layers in the classical AE with convolutions. In autoencoder, the layer with the least amount of neurons is referred to as bottleneck. In the regular AE, this bottleneck is simply vector (rank-1 tensor). In CAES, however, the bottleneck assumes the shape of multichannel image (rank-3 tensor, height x width X channels) instead. This bottleneck shape prompts the question: What is the relative importance of bottleneck depth (i.e., the number of channels) versus the bottleneck area (i.e., feature map size) in determining the tightness of the CAE bottleneck? Intuitively, we might expect that only the total number of neurons should matter since convolutions with one-hot filters can distribute values across channels.
In this paper, we share new insights into the properties of convolutional autoencoders, which we gained through extensive experimentation. We address the following questions:
How do bottleneck area and depth impact
reconstruction quality?
generalization ability?
knowledge transfer to downstream tasks?
How and when do CAEs overfit?
Are CAEs capable of learning "copy function" if the CAE is complete (i. c., when the number of pixels in input equals the number of neurons bottleneck)? By copy function we are referring to type of identity function, in which the input pixel values are transported through the bottleneck and copied to the output. The hypothesis that AEs learn an identity mapping is common for fully connected AES and can sometimes be encountered for CAEs (see Sections and 5 in Masci et al. 201T).
We begin the following section by formally introducing convolutional autoencoders and explaining the convolutional autoencoder model we used in our experiments. Additionally, we introduce our three datasets and the motivation for choosing them. Section 3] we outline the experiments and their respective aims. Afterward, we present and discuss our findings in Section E All of our code, results, and trained models and datasets, is published on github We invite interested readers to take a look and experiment with models.
MATERIALS AND METHODS
2.1 AUTOENCODERS AND CONVOLUTIONAL AUTOENCODERS
The regular autoencoder, as introduced by Rumelhart et al. 1985), is a neural network that learns mapping from data points the input space IR"" code vector in latent space h E I and back. Typically, unless we introduce some other constraint, m is set to be smaller than d to force the autoencoder to learn higher-level abstractions by having to compress the data. In this context, the encoder is the mapping f(x) Rd - I and the decoder is the mapping g(h) Rm + Rd The layers in both the encoder and decoder are fully connected:
g++1 oW''t' + bb).
Here, I is the activation vector in the i-th layer, W and b' are the trainable weights and is element-wise non-linear activation function. If necessary, we can tie weights in the encoder to the ones in the decoder such that W (Wn-ij7' where is the total number of layers. Literature refers to autoencoders with this type of encoder-decoder relation as weight-tied.
The convolutional autoencoder keeps the overall structure of the traditional autoeneoder but replaces the fully connected layers with convolutions:
L [+1 oW L' b')
where * denotes the convolution operation and the bias is broadcast to match the shape of L' such that the j-th entry in b' is added to the j-th channel in L'. Whereas before the hidden code was an m-dimensional vector, it is tensor with rank equal to the input tensor's rank. In the case
2

---

Under review as a conference paper at ICLR 2023
of images, that rank is three (height, width, and the number of channels). CAEs generally include pooling layers or convolutions with strides > I or dilation in the encoder to reduce the size the input. In the decoder, unpooling transposed convolution layers Dumoulin & Visin (206) inflate the latent code to the size of the input.
2.2 OUR MODEL
Our model consists of five strided convolution layers in the encoder and five up-sampling convolution layers (bilinear up-sampling followed by padded convolution) Odena et al. (2016) in the decoder. We chose to use five such layers so that the bottleneck area would be 3x3 (input dimensions are 96 x 96). To have a deeper model, we added two residual blocks He et al. (2016) with two convolutions each after each strided / up-sampling convolution layer. We applied instance normalization Ulyanov et al.) (2016) and ReLU activation Nair & Hinton 2010) following every convolution in the architecture.
Our goal was to understand the effect latent code shape has on different aspects of the network. Therefore, wanted to change the shape of the bottleneck between experiments while keeping the rest of the network constant. To this end, we quadrupled the number of channels with every strided convolution s and reduced it by factor of four with every up-sampling convolution u'. In effect, this means that the volume (i. e., height x width x channels) of the activations is identical to that of the input in all layers up to the bottleneck:
s'L') E ER //xx"" //axx///xmnn
/ax4m for L' RR"xx"' xa''x"" x
(3)
(L(L') E 22 x2w 4b/4 for E R='x''x
(4)
In this sense, the bottleneck the only moving part in our experiments. The resulting models have number of parameters ranging from ~ 50M to 90M, depending on the bottleneck shape. The variants with smaller bottleneck area have more parameters due to the large number of channels needed to achieve the same relative volume to the input as variants with a bigger area.
2.3 DATASETS
To increase the robustness of our study, we conducted experiments on three different datasets:
Chess The first dataset is a collection of synthetic images showing chess positions The images generated by randomly sampling the board and piece icon style and the number of pieces present on the board. Each image comes with string of characters describing the position in FEN notation. We convert this description into binary labels that indicate each possible piece's presence, excluding kings, as they are always on the board. In total, the dataset consists of 100,000 400 pixel images. To keep the training time within acceptable bounds, we resized all images to be 96x96 pixels.
CelebA Our second dataset is the CelebA faces dataset Liu al. (015)). This dataset is collection of 202,600 images showing celebrity faces, each with a 40-dimensional attribute vector (attributes such as smiling/not smiling, male/female). For our purposes, we resized the images to be 96x96 pixels. The original size was 178x218 pixels.
STL-10 For our last dataset, we picked STL- Coates et al. (011 This dataset consists 96 x6 pixel natural images and is divided into three splits: 5,000 training images (10 classes). 8,000 test images (10 classes), 100,000 unlabeled images. The unlabeled images also include objects that are not covered by the ten classes in the training and test splits.
3 EXPERIMENTS
3.1 AUTOENCODER TRAINING
The first experiment we conducted, and which forms the basis for all subsequent experiments, consists of training of autoencoders with varying bottleneck shapes and observing the dynamics of their training and test losses. This experiment probes the relative importance area versus depth in the bottleneck. Additionally, it provides insight into how and when models overfit. We also
3

---

Under review as a conference paper at ICLR 2023
tested the hypothesis that autoencoders learn to "copy" the input if there is no bottleneck (i.e., the volume of activations relative to imput is always 1). To be able to observe overfitting behavior, we used only the first 10,000 images for training and the last 2,000 images for testing in each dataset. For S0,,-10, we used images from the unlabeled split.
Our CAE model, introduced in Section 2.2
yields bottleneck area An- We obtain two additional bottleneck areas A,, i E {1,2] by changing the stride in the last i strided convolution layers in the encoder from 2 to
Ain
A,
E f0, 1,2), =5
(5)
4-
In this equation, nt = is the number of strided convolution layers in the vanilla network, and Ain is the area (height x width) of the images in the dataset. The figures in Section 4 refer to Ao, A1, A2 as S, M, L (small, medium, large), which we believe to more intuitive.
For each area setting we then fix four levels of compression oj E (1/44, 1/18. 1/4, 1 and calculate the necessary number channels Aay according to
CyAin New
i i {0, 10, j E (1,2,3,4)
(6)
A;
Here, ne.. is the number of channels in the input image. When presenting our results, we use the levels of compression, rather than the number of channels, as the latter changes based on the chosen area for the same level of compression.
All resulting autoencoders have the same number of parameters in all layers except the convolutional layer and residual block directly preceding and following the bottleneck. We use mean squared error (MSE) between reconstruction and input as our loss function. After initializing all models with the same seed, we train each for 1,000 epochs, computing the test error after every epoch. We repeat this process for three different seeds each dataset. Training a single model for 1000 epochs took approximately two days on a Nvidia RTX 2080Ti using half-precision.
3.2 SCALING WITH DATASET SIZE
As described above, we limited the training data to 10,000 samples for the training of our CAEs. To estimate the effect of the training data amount has on CAE training, we train CAE models with six different amounts of training data. For this experiment, we use the CelebA dataset and four of the twelve models from the first experiment (S 1/64, S 1/64, M 1/16, M 1/16). We train the models on the first 1% (2k samples), 5% (10k samples), 10% (20, samples), 25% (50k samples), 50% (100k samples), and 95% (190k samples) of the samples in the dataset while reserving the last 5% for testing. Each model trains for 35,000 iterations, which translates to roughly half of the compute each model received in the first experiment. For each split and model, use three random seeds, which are different from the ones used in Section 3.1
3.3 KNOWLEDGE TRANSFER
Another goal of our investigation is to estimate the effect of the latent code shape on transferability. Here, we train a logistic regression model on latent codes to predict the corresponding labels for each dataset. Since logistic regression can only learn linear decision boundaries, this approach allows us catch a glimpse of the sort of knowledge present in the latent code and its linear separability. Furthermore, this serves as another test for the "copying" hypothesis. If the encoder has indeed learned to copy the input, the results of the logistic regression will be the same for the latent codes and the input images. We perform this experiment separately for data that the CARs saw during training and unseen data. the first step, we export latent codes for all 108 trained models. From the Chess and CelebA datasets, we use the 10,000 samples in the training data and another 10,000 unseen images from the end of the dataset. Since we trained CAEs on samples from the unlabeled split of STL-10, there are labels available for the training data. Consequently, we only use the 8,000 images in the STL-10 test split for classification.
We train the classifiers with a one-cycle policy (mmihh (2018) and early stopping for maximum of 100 epochs. For each classifier, we use 60% of data for training, 20% as a validation set for early stopping, and 20% for final testing. Additionally, train each classifier three times different
4

---

Under review as a conference paper at ICLR 2023
seeds and different samples in the splits. Besides, we also train models directly on the image data for every dataset to serve as a bascline for comparison (also with three seeds).
3.4 OUTLIER DETECTION
Chess Training Loss
Chess Test Loss
Since CAES are widely used in reconstruction- based outlier detection, we investigate the impact of the bottleneck shape in this setting as well. To this end, used the models trained on the CelebA (all configurations and all three seeds) data calcu- ated the reconstruction loss of 2000 samples from the CelebA and STL-10 datasets each. We then clip the losses to and calculate the area under the ROC curve as an indicator for the effectiveness of the AE as an outlier detector.
CelebA Training Loss
CelebA Test
0:33
6:31
STL-10 Train ng Lass
STL.10 Test Lass
6:33
3.5 AUTOENCODER INTERPOLATION AND ITERATION
6.22
6:00
230
420
835
830
3303
230
400
050
830
3309
Epochs
Epochs
To get a more holistic (but qualitative) impression of the differences between the different bottleneck configurations we employ methods for visualizing different aspects of the latent space learned by the models on the CelebA data. The first such method latent-space interpolation. Here, we pick pairs of test samples from the CelebA dataset and en- code them. Once we have the latent codes, we perform liner interpolation between them, recon- structing the intermediates along the way. Visually inspecting the resulting intermediate reconstructions is good method for discovering qualitative differences that hard to capture using metrics Oring et al. (2020).
Figure I: Loss plots for the three datasets. Ev- ery bottleneck configuration is shown as dis- tinct line. Each line is the average over three runs. Configurations that have common bottle- neck area share the same color. Color intensity represents the bottleneck depth (darker more channels)
The other method we use is iteration of the CAE. Starting from test sample repeatedly feed the reconstructed image back into the autoencoder. Beeeeoddeer.BBiiooooo et al. (2013b) showed that this can be thought of walking along the latent space manifold learned by the model. In our experiment iterate over each model 100 times. Compared to the interpolation, this method samples different portions of the latent space, as the process not directed.
RESULTS AND DISCUSSION
4.1 AUTOENCODER TRAINING
Generalization Improves with Bottleneck Area We observe that bottleneck shape critically affects generalization (Figure[T) While bottleneck depth has almost no effect on generalization, increasing the bottleneck area vastly improves generalization and enables the models to be trained longer before overfitting occurs. To measure the strength of this effect, we looked at the minimal test error achieved by each model, and the epoch at which this minimum occurred (Figure 2). Models with a larger bottleneck area have much lower test eтror than ones with small area. Additionally, their test error curves reach their minima at later epochs, meaning that such models start overfitting later in the training cycle. The differences in the minimum test error were statistically significant for the different bottleneck areas with p-values below 0.001, according to a Wilcoxon rank-sum test (Table in the Appendix). This finding is surprising, given the hypothesis that only the total amount of neurons matters. The better generalization of models with larger bottleneck area is also clearly visible when inspecting reconstructions of test samples. We have added figures showing the reconstructions of random samples in the Appendix.
This effect is relevant for many applications of CAEs. For instance, if compression restoration is the goal, a larger bottleneck area is helpful as we demonstrate with Figures B.I B.2 B.33 in the supplementary material. On the other hand, if the CAE is intended for outlier detection, smaller
5

---

Under review as a conference paper at ICLR 2023
bottleneck area is preferable, this increases the difference in reconstruction enror between valid samples and outliers. To show this, we have plotted the difference in reconstruction error for CAEs trained on CelebA data when reconstructing unseen samples from CelebA and STL-10 in Figure
Training Speed Increases with Bottleneck Area Although all bottleneck configurations re- sult in similar final training errors, a bigger bot- tleneck area accelerates the model training. This effect is apparent in the loss curves in Figure I Loss curves from models with bigger bot- leencck area reach lower training and test error sooner than their counterparts. We have quan- tified the effect by looking at the first epoch to reach training error below 0.01 (Figure (2). We find that only the bottleneck area is strongly correlated with convergence speed, while bot-
ColobA
Chess
STL-10
Syoch g
134
151
141
135
Mosnnal fext tree
tleneck depth does not seem to affect it. A Wilcoxon rank-sum test shows that the epoch at which the training loss falls beneath 0.01 is sig-
Epach Tast
274
nificantly different for the bottleneck areas, with
all p-values being below 0,001, except in two cases, where they were 0.02 and 0.08. [Table in the Appendix). The transition from small to medium area yielded the largest speed up in our experiments.
Figure Plots showing the first epoch to fall beaneath 0.01 training error, the minimal test er- ror and the epoch at which the model achieved (see Section 4.1. Columns represent the results for each dataset. Each number is the average over three runs. Marginals show the mean and standard deviation of the row / column. More desirable values have brighter colors.
CAEs without Bottleneck Do Not Learn an Identity Mapping CAES, where the total num- ber of neurons in the bottleneck is the same as the number of pixels in the input, do not show signs of simply copying images. If the CAEs would indeed copy images, the test error would go to zero, yet we do not observe this case in any of the datasets.
3.4
5%
10%
25%
59%
95%
1/64 Training
1/64 Training
0.05
0.04
0.03
What is more, these complete CAEs follow the same pattern as the under-complete ones and often converge to similar values. In essence, it suggests that even complete CAES learn ab- stractions from data, and raises the question: What prevents the CAE from simply copying its
0.02
0.02
0.00
1/64 Test
1/64 Test
0.05
0.04
0.03
0.02
input? We believe that the answer to this ques- tion could potentially lead to new autoencoder designs that exploit this limitation to learn bet- ter representations. Hence, we argue that it is an exciting direction for future research. Addi- tionally, the trends we derive from results suggest that this finding likely extends to over- complete CAEs as well. However, experiments with over-complete CAES are required to test this intuition.
0.03
0.00
S 1/16 Training
M 1/16 Training
0.05
0.04
0.02
0.03
1/16 Test
M 1/16 Test
0.05
0.04
a.2
0.02
0.00
0.5
3.0
2.0
2.5
3.0
0.5
2.0
2.5
3.0
Steps
Steps
4.2 SCALING WITH DATASET SIZF.
Figure 3: Loss plots for the scaling experiment (see Section 3.2h averaged over three runs. Each quadrant shows training and test losses from one model. Colors correspond to the fraction of data used for training, as shown in the legend the top. Brighter means more training data.
We present the results of the dataset scaling ex-
periment from Section 3.2 in Fig. 3 Train- ing with more data unsurprisingly results in bet- ter generalization but with diminishing returns. When using more than 25% of the full datasets (Rs 50,000 samples), the test error does not improve significantly, it converges to the training error.

---

Under review as a conference paper at ICLR 2023
These models did not plateau after 35,000 training iterations, meaning that differences could arise if we trained the models longer.
We can confirm observation from Section 4.1 that a bigger bottleneck area results faster train- ing. As can be seen from the scaling experiment, this effect is independent of the training set size.
Unseen
CelebA
0.81
0.02
0.33
0.81
5.81
0.02
0.83
0.04
The effect of the bottleneck area on generalization, which we discovered in the first experiment, is present for all fractions of the training data. Ad- ditionally, we find the decrease in test eπror to be more pronounced if we train the model on fewer samples.
0.55
0.45
0.45
0.85
2.55
0.84
0.24
0.45
1Ak
04
0.55
0:53
0.62
0.63
5.57
0.42
0.63
0.54
Chess
4.3 KNOWLEDGE TRANSFER
0:53
0.52
0.43
0.65
5.55
0.63
0.61
0.04
If we look the results of our knowledge transfer experiments (Fig. 4. we find further evidence that contradicts the identity mapping hypothesis. Al- though the loss curves and reconstructions already indicate that the CAE does not copy its input, the possibility remains that the encoder distributes the input pixels along the channels, but the decoder is unable to reassemble the image. Here, we see that the results from the linear model trained on latent codes of complete CAEs perform better than those trained directly on the inputs (marked "baseline" in the figure). As such, it is implausible to assume that the encoder copied the input to the bottleneck.
STL-10
0.37
0.4
0.35
0.33
0.34
0.34
0.38
Figure Results linear probes. AUROC for Chess and CelebA, F1 for STL-10. Marginals show the mean and standard devia- tion. Red signifies improvement over baseline.
Overall, we find that bottleneck shape only slightly affects knowledge transfer with linear classifiers. Interestingly, the number of channels in the code seems to improve the performance of the classifi- cation slightly. It is not clear, however, whether this effect is due to the structure of the codes or their high dimensionality. Perhaps projecting the representations to have the same dimensionality (using PCA or UMAP) before classification could answer this question. Additionally, we find that the classifier'' performance almost the same on seen and unseen data. Given the differences in overfitting between the different models, we would expect otherwise. This may suggest that encoder and decoder exhibit different degrees overfitting. One possible explanation would be that overfitting happens mostly in the decoder, while the encoder retains most of its generality. We believe that this question warrants further in- vestigation, especially light of the widespread use of transfer learning methods.
0.00
0.023
MSE
0.13
$1.509
$1000
H 1,s0
250
Loow
1..2%
Lite
Bottleneck Shape
Figure 5: Difference in the average MSE on both datasets in the outlier detection experiment.
0.0
0.5680.557
0.557
0.551
0.506
0.685
0.466
0.512
0.6697
0.714
0.705
0.655
1/64
1/1
0's
4.4 OUTLIER DETECTION
Figure 6: AUROC organized into a grid of area number of channels
Calculating the difference between the average MSE in both the CelebA and STL-10 samples shows that models with larger bottleneck area perform significantly better due to their improved generalization capabilities. This poses an issue for reconstruction-base based outlier detection however, can be seen in the results for the calculated
7

---

Under review as a conference paper at ICLR 2023
AUROC values (see Fig. 66. In conclusion observe, that for outlier detection tasks smaller bottleneck area is preferable. It appears that the outlier detection performance also decreases with increasing number of channels, although this effect is much weaker and not clear.
4.5 AUTOENCODER INTERPOLATION AND ITERATION
Looking at the results of the interpolation and it- eration experiments (see Figures and 7 we find that smaller the bottleneck areas encourage the learning of "concepts". We can see this e.g. in the iteration results, where the models with the smallest bottleneck area still show faces after 100 iterations, while the images of the larger ones dis- solve into indistinguishable patterns. Similarly, the iteration from the larger-area models seem more closely resemble aplha-blending, whereas the models with smaller show gradual mor- phological change of their contents.
6X6 758 36
5 CONCLUSION
In this paper, we presented the findings of our in- depth investigation of the CAE bottleneck. We could not confirm the intuitive assumption that the total number of neurons sufficiently charac- terizes the CAE bottleneck. We demonstrate that height and width the feature maps the bottleneck define its tightness, while the number of channels plays secondary role. Larger bot- tleneck area (i.e., height x width) is also critical in achieving lower test errors, while simultane- ously speeding up training. These insights directly transferable to the two main areas of ap- plication for CABs, outlier detection and compres- sion/denoising: In the case of outlier detection, the model should yield a high reconstruction error on out-of-die oi-diitribution samples. Using smaller bot- tleneck sizes to limit generalization proves useful in this scenario. On the other hand, compression and denoising tasks seek to preserve image details while reducing file size and discarding unneces- sary information, respectively. In this case, big- it increases
Figure 7: Results of the autoencoder interpola- tion. Images grouped by area and increasing with number channels.
4B
102
3x3
1072768
3072
12
768 102 48
6x6
192
ger bottleneck size is preferable, reconstruction quality at the same level of com- pression. Furthermore, we could not confirm the hypothesis that complete CAE (i. e., CAFS with the same number of neurons the bottleneck as pixels in the input) will learn an identity mapping. On the contrary, even complete CAEs appear to follow the same bottleneck size dynamics, as stated above.
Figure 8: Results of the iteration experiment. Each column shows the reconstruction after iterations.
In knowledge transfer experiments, we have also shown that CAEs that overfit retain good predictive power the latent codes, even on unseen samples. interesting question that follows from these experiments is how overfitting manifests itself in CAEs. Does it occur mainly in the encoder or decoder or equally both?
8

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guillaume Alain and Yoshua Bengio. What Regularized Auto-Encoders Learn from the Data Generating Distribution. arXiv e-prints, art. 1xXiv11211.42444 Nov 2012.
Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma. Provable bounds for learning some deep representations. CoRR, abs/1310.6343, 2013.
6343
Devansh Arpit, Yingbo Zhou, Hung Ngo, and Venu Govindaraju. Why Regularized Auto-Encoders learn Sparse Representation? arXiv e-prints, art. 1rXiv:1505.0555 May 2015.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham Taylor, and Daniel Silver (eds.), Proceedings of ICML Workshop on Unsupervised and Transfer Learning. volume 27 of Proceedings of Machine Learning Research, pp. 37-49, Bellevue, Washington, USA, 02 Jul 2012. PMLR. URL http: http://proceecingg m r. wt.press/v2//// wwwp//////a/lili html
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding models for unsupervised anomaly segmentation in brain mr images. In Alessandro Crimi, Spyridon Bakas, Hugo Kuijf, Farahani Keyvan, Mauricio Reyes, and Theo van Walsum (eds.), Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 161-169, Cham, 2019, Springer International Publishing. ISBN 978-3-030-11123-8.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf. J. C. Platt, and T. Hoffman (eds.), Advances in Neural Information
Processing Systems 19, pp. 153-160. MIT Press, 2007. URL http:: nips cc/
ayer wwwe-wweeeeeeeeeeeeeeeeeeeeeeeee -are
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising nodoesss as generative models. CoRR, abs/ 305.6663,, 2013a. URL http::/aaxwv..og/////ooa/
6663
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized Denoising Auto-Encoders as Generative Models. pp. 1-9, 2013b. ISSN 10495258. URL httphtttt///xvv..oo wwwpvvoorrgraaao /bbs/
1305. 6663
arXiv: 1305.6663 ISBN: 10495258.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via adversarial regularizer. arXiv preprint arXiv: 1rXiv:1807.077 2018.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Improving interpolation in autoen- coders. 2019. URL htt.hhttss//pppeeeeieeee net.
wwwp:r/wwwwweeeeeddeeeeeeeeeeeee Sid=S1 FQSiCoYi
Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga. Oulier Delection with Autoencoder Ensembles, pp. 90-98. 2017. doi: 19113/7999711119999999977
URL https: /epubs
siam. 2ttpa///r...aaaaa...............
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Deep convolutional autoencoder- based lossy image compression. In 2018 Picture Coding Symposium (PCS), pp. 253-257, June 2018. doi: 10.1 PCS.2018.8456303
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 225,222,222 2011.
T. Dumas, A. Roumy, and C. Guillemot. Autoencoder based image compression: Can the learning be quantization independent? 2018 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1188-1192, April 2018. doi: D0.1109//CSSSPP2188.88888 8462263.
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. arXiv preprint :Xi3:/603.00288 2016.

---

Under review as a conference paper at ICLR 2023
Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect unsupervised pre- wre-training. In David van Artificial
Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 153-160, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR. URL
http: /ttp://d/oceeennggs..mc www. htw.e///wnn...p..eeeeeee.ee.. erwan html
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114 Dec 2013.
Quoc V. Le. Building high-level features using large scale unsupervised learning. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8595-8598, 2013. ISSN 1520-6149. doi: 10. 10111//ASSS.2211.666935525 URL
document/ 66393333666333333/
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Llnet: A deep autoencoder approach to natu-
ral low-light image enhancement. Pattern Recognition, 61:650 662, 2017. ISSN 0031-3203. doi: com/
URL http::/ww..cceeneeeiiccccc.
CoRR, 6sfaanec//atccee/////60012211222
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and lan Goodfellow, Adversarial autoencoders. 2015.
arwiv..iii.vv..........a.aaa..v. wwt/abs.//aaeeeoeee..../e..e.aae aataab/s/11l....... 05644
Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Image Restoration Using Convolutional Auto- encoders with Symmetric Skip Connections. arXiv e-prints, art. arXiv: (66660899111 Jun 2016.
Jonathan Masci, Ueli Meier, Dan Ciresan, and Jürgen Schmidhuber. Stacked convolutional auto- encoders for hierarchical feature extraction. In International Conference on Artificial Neural Networks, pp. 52-59. Springer, 2011.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Thanh V. Nguyen, Raymond K. W. Wong, and Chinmay Hegde. Autoencoders Learn Generative Linear Models. arXiv e-prints, art. mrXiv:1806.00572 Jun 2018.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard arti- facts. Distill, 2016. doi: 10.2391 //distill.l., URI. http::///istt111.uub///
Alon Oring, Zohar Yakhini, and Yacov Hel-Or. Autoencoder image interpolation by shaping the latent space. arXiv preprint arXiv:2008.01458 01487, 2020.
Adityanarayanan Radhakrishnan, Karren Yang. Mikhail Belkin, and Caroline Uhler. Memorization in Overparameterized Autoencoders. 2018. URL
wttp:////a////..11111111 10333
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In The European Conference on Computer Vision (ECCV), September 2018.
Leslie Smith. A disciplined approach to neural network batch size, momentum, and weight decay. arXiv preprint arXiv: 0803.0880, 2018.
Part 1 l-learning rate,
10

---

Under review as a conference paper at ICLR 2023
Edgar Tretschk, Ayush Tewari, Michael Zollhöfer. Vladislav Golyanik, and Christian Theobalt. DEMEA: deep mesh autoencoders for non-rigidly deforming objects. CoRR, 1909. 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprinl arXiv: urXvv:660.080022, 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008.
Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Ozal Yildirim, Ru San Tan, and U. Rajendra Acharya. An efficient compression of ecg sig- nals using deep convolutional autoencoders. Cognitive Systems Research, 52:198 211, 2018. ISSN 1389-0417. doi: http://bbb.or///...000...ssss.s.
URL http:
/ htt.://wwc........ooooooo...eeaa
Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 17, pp. 665-674, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-4887
4. doi: 00.1145/30979833009885 URL http://dott acm 015
3098052


Reviews:
[{"review_id": "rLtvPuGrV3X", "reviewer": "Reviewer_fPDr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically investigates the impact of the bottleneck shape of convolutional autoencoders (CAE) on generalization, training speed,  outlier detection and transferability of features. Based on numerical experiments on a CAE, authors conclude that larger spatial resolution in the bottleneck corresponds to better generalization.  Furthermore, they claim that convergence speed increases with larger bottleneck area. They also find that CAEs do not learn to copy the input to the output, even when the bottleneck volume matches the input image volume. Based on their findings, authors recommend large bottleneck area for compression/denoising and small bottleneck area for outlier detection.", "summary_of_the_review": "Overall, in its current form I would lean towards borderline rejecting this paper.  I have concerns with respect to the experimental methodology and the significance of the findings as it is unclear whether the phenomena hold in higher dimensions and on models deployed in practice.  Furthermore, the provided experimental details are insufficient to determine whether the different models were fairly compared. The paper is definitely investigating interesting questions however, and the direction on better understanding the nature of overfitting of AEs would be an interesting addition to the paper.", "strength_and_weaknesses": "Strenghts: \n\n- The paper is fairly well-written and easy to follow. It is organized in a logical manner.\n\n- The paper's motivation is clear, it is important to gain better understanding of the trade-offs and design considerations of common deep learning models.\n\n- The paper identifies an interesting question with respect to AE overfitting: is overfitting mostly happening in the decoder only?\n\nWeaknesses:\n\n- I have some concerns with respect to the experiments. The datasets used in the core experiments are small (10k images) with small resolution images (96x96). The paper's findings may not necessarily hold in higher dimensions, however in most applications where the findings may be relevant (medical image denoising, outlier detection) the images are typically of much higher resolution. \n\n- The methodology of the experiments can also be somewhat questionable. Changing the bottleneck layers results in models of different size (50M-90M according to the authors) and potentially very different training dynamics depending on bottleneck size. Therefore, using the same training hyperparameters for all models may not guarantee a fair comparison: larger models may need more time to converge properly, the optimal learning rate for different models may vary. It would be important to see how these parameters were set and whether the comparison is really fair.  \n\n- I am not convinced that bottleneck area is necessarily the cause. For instance, the observed phenomenon might be explained as follows: all models are heavily overparameterized due to the small dataset/low-res images, the smallest model (in terms of # of parameters) has the strongest implicit regularization, therefore will provide the best generalization. In case of this paper, the smallest model is the one with the largest bottleneck area.\n\n- I don't see how \"bottleneck depth has almost no effect on generalization\".  According to Figure 1,  there is a very significant spread between different compression ratios, however typically the higher ratio the better. \n\n- I am unsure what was the purpose of the interpolation experiments, they are not directly referenced and no conclusion has been drawn.\n\n- If the authors' conclusion is correct and in fact larger bottleneck area is better, why did we stop at the largest resolution investigated in the paper? What happens if we do not use any spatial downsampling?\n\n- The paper has only 8 content pages out of 9. Many interesting questions raised by the authors could fill the extra page with additional experimentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mWCNesANHlE", "reviewer": "Reviewer_7HZy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts an analysis of the bottleneck size/channels of the convolutional autoencoder (CAE). The authors find that an increased bottleneck area (i.e., height × width) improves generalization in terms of reconstruction error while also speeding up training. In contrast, decreasing the feature map size will make out-of-distribution samples yield a higher reconstruction error, which may be beneficial for outlier detection.", "summary_of_the_review": "Personally, I think this is a good paper. Although there are some notable weaknesses, I vote for borderline acceptance.", "strength_and_weaknesses": "Strength:\n- The analysis is comprehensive and insightful. The conclusions seem rational.\n- Several realistic applications are considered, i.e., image compression, outlier detection, and classification.\n\nWeaknesses:\n- I'm not fully convinced by the claim that \"a smaller bottleneck area is preferable for outlier detection\". From Figure 1 one can observe that a smaller bottleneck area leads to overfitting. The test reconstruction loss of both normal samples and outliers will be large. Why is this beneficial for outlier detection?\n- Are there some comparisons against the recently proposed methods of image compression? The improvements on top of state-of-the-art baselines may be strong evidence to demonstrate the practical importance of the conclusions of this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cVAotQPg5Y", "reviewer": "Reviewer_VXUw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors conduct a study of convolutional auto-encoders to assess the importance of different dimensions of the autoencoder bottleneck on model quality for a range of tasks. They additionally assess whether or not these models learn identity functions when the number of pixels in the input matches the number of features in the bottleneck. Based on their analysis, the authors make practical recommendations for the designing autoencoder architectures. ", "summary_of_the_review": "Overall, the paper is well written and the authors' results are potentially practically useful. My primary concern is the single architecture considered and whether the authors' results will generalize beyond this setup.", "strength_and_weaknesses": "The paper is clear and focused and I found it to be easy to follow. The study is practically useful and the results are interesting, although I am not an expert on autoencoders and do not know of related work to this study. \n\nMy main concern with the paper is the utility of the authors conclusions given the single model configuration that was studied. It is not clear to me that these observations will generalize to a different convolutional autoencoders architectures.\n\nI also found the training time of 1000 epochs to be surprisingly long - roughly an order of magnitude more than is typical for vision datasets like ImageNet, which is much larger than the datasets studied here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CxsU0SRoOvM", "reviewer": "Reviewer_AyQZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the convolutional autoencoder (CAE) bottleneck, more specifically on the impact of the bottleneck shape. Empirically, under the experimental settings chosen, it observes that increased bottleneck spatial resolution (i.e., height × width) improves generalization in terms of reconstruction error while speeding up training, while the chosen number of channels in the bottleneck is of secondary importance.\nThe ablated model consists of a convolutional auto encoder with five conv. layers in the encoder and another 5 in the decoder, such that given an input with dimension 96x96, produces a bottleneck with spatial resolution of 3x3, and kept the 'volume' (height × width × channels) constant across experiments. \nThe models with different spatial/depth composition is trained on 3 datasets: Chess (chess positions), CelebA (40-attribute vestor) and STL-10 (10 classes + unlabeled), resized to 96x96 pixels. \nThe results presented are comparison of training and test loss curves for the 3 datasets, and bottlenecks scaled by 1, 1/4 and 1/64. \nIt conjectures that the observations can be used to recommendations on outlier detection and compression/denoising, suggesting to increase the feature map spatial resolution for image compression but reduce it for reconstruction based outlier detection.\n", "summary_of_the_review": "The paper is clear, but does not present a robust experimental setting to sustain the conclusions/recommendations made. In that sense, it is hard to evaluate its impact and validity outside the experiments performed into real  use-cases of CAE.", "strength_and_weaknesses": "The paper findings are highly experimental, which is not a defect in itself, but creates an expectation for robust experimental settings that reflect the hypotheses investigated. In this sense, the major weakness of the paper is the lack of extensive experiments to support the recommendations and conclusions made and/or the lack or real case study. In the same direction, the paper mentions that the insights are 'directly transferable' to outlier detection and compression/denoising, but no evidence of such is given. \n \nAll references pointed are dated a few years ago with a single reference from 2020. Again here, using older references is not a defect in itself, but should not be the case in the lines where authors indicate 'currently' state of the art/on going research directions (Intro- first paragraph). This indicates that the paper contextualization is outdated as it contrasts with the velocity of new findings in the main application areas tackled in the paper (compression and outlier detection).  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject