Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck
Authors: Ilja Manakov
Markus Rohm
Volker Tresp
Venue:  (2020)
Keywords: convolutional autoencoder
bottleneck
representation learning

Abstract: In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck.
Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox.
Researchers and practitioners employ CAEs for a variety of tasks, ranging from outlier detection and compression to transfer and representation learning.
Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the emergent properties of the CAE.
We demonstrate that increased height and width of the bottleneck drastically improves generalization, which in turn leads to better performance of the latent codes in downstream transfer learning tasks.
The number of channels in the bottleneck, on the other hand, is secondary in importance.
Furthermore, we show empirically, that, contrary to popular belief, CAEs do not learn to copy their input, even when the bottleneck has the same number of neurons as there are pixels in the input.
Copying does not occur, despite training the CAE for 1,000 epochs on a tiny (~ 600 images) dataset.
We believe that the findings in this paper are directly applicable and will lead to improvements in models that rely on CAEs.

Under review as a conference paper at ICLR 2020
WALKING THE TIGHTROPE: AN INVESTIGATION OF THE CONVOLUTIONAL AUTOENCODER BOTTLENECK
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, present in-depth investigation of the convolutional autoen- coder (CAE) bottleneck. Autoencoders (AE), and especially their convolutional variants, play vital role in the current deep learning toolbox. Researchers and practitioners employ CAEs for a variety of tasks, ranging from outlier detection and compression transfer and representation learning. Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the emer- gent properties of the CAE. We demonstrate that increased height and width of the bottleneck drastically improves generalization, which in turn leads to better per- formance of the latent codes in downstream transfer learning tasks. The number of channels in the bottleneck, thermore, we show empirically that, contrary to popular belief, CABs do not learn to copy their input, even when the bottleneck has the same number of neurons there are pixels the input. Copying does not occur, despite training the CAE for 1,000 epochs tiny (=s 600 images) dataset. We believe that the findings in this paper are directly applicable and will lead to improvements in models that rely on CAEs.
the other hand, is secondary in importance. Fur-
INTRODUCTION
Autoencoders (AE) are an integral part of the neural network toolkit. They are class neural net- works that consist of an encoder and decoder part and are trained by reconstructing datapoints after encoding them. Due to their conceptual simplicity, autoencoders often appear in teaching materi- als as introductory models to the field of deep unsupervised learning. Nevertheless, autoencoders have enabled major contributions application include outlier detection (Xia et al., 2015; Chen et al. 2017 Zhou & Paffenroth 2017]
the application and research of the field. The main areas of
Baur et al., 2019), data compression Yildirim et al. 2018 Cheng et al. 2018 Dumas et al. 2018), and image enhancement (Mao et al.) 2016; Lore et al. 2017). In the early days of deep learning, autoencoders were a crucial tool for the training of deep models. Training large (by the standards of the time) models challenging, due to the lack of big datasets and computational resources. One way around this problem was to pre-train some or all layers of the network greedily by treating them as autoencoders with one hidden layer (Bengio al. (007).. Subsequently, EErhan et al. (2009) demonstrated that autoencoder pre-training also benefits generalization. Currently, researchers in the field of representation learning frequently rely on autoencoders for learning nuanced and high-level
representations of data (Kingma & Welling 2013: Tretschk et al. 2019 Shu et al. 2018 Makhzani
al. 2015 Berthelot al. 2018).
However, despite its widespread use, we propose that the (deep) autoencoder model is not well understood. Many papers have aimed deepen our understanding of the autoencoder through theo- retical analysis (Nguyen et al. 2018 Arora et al. 2013) Baldi 2012) AAlin & Bengio 2012). While such analyses provide valuable theoretical insight, there is a significant discrepancy between the the- oretical frameworks and actual behavior of autoencoders in practice, mainly due to the assumptions made (e.g.. weight tying, infinite depth) or the simplicity of the models under study. Others have approached this issue from more experimental angle (Arpit et al. 2015; Bengio et al., 20133 Le 2013) Vincent et al. 2008 Berthelot et al. 2019). Such investigations are part of an ongoing effort understand the behavior of autoencoders in variety of settings.

---

Under review as a conference paper at ICLR 2020
The focus of most such investigations far has been the traditional autoencoder setting with fully connected layers. When working with image data, however, the default choice is to use convolutions, as they provide prior that is well suited to this type of data (Ulyanov al. (018). For this reason, Masci et al. (011) introduced the convolutional autoencoder (CAF) by replacing the fully connected layers in the classical AE with convolutions. In autoencoder, the layer with the least amount of neurons is referred to as a bottleneck. In the regular AE, this bottleneck is simply vector ( rrank-I tensor). In CAES, however, the bottleneck assumes the shape of multichannel image (rank-3 tensor, height x width X channels) instead of a vector. This bottleneck shape prompts the question: What the relative importance of the number of channels versus the height and width (hereafter referred to as size) in determining the tightness of the CAE bottleneck? Intuitively, we might expect that only the total number of neurons should matter since convolutions with one-hot filters can distribute values across channels. Generally, the study of CAE properties appears to be underrepresented in literature, despite their widespread adoption.
In this paper, we share new insights into the properties of convolutional autoencoders, which gained through extensive experimentation. We address the following questions:
How does the number of channels and the feature map size in the bottleneck layer impact
reconstruction quality?
generalization ability?
the structure of the latent code?
knowledge transfer to downstream tasks?
How and when do CAEs overfit?
How does the complexity of the data distribution affect all of the above?
Are CAEs capable of learning a "copy function" if the CAE is complete (i. e., when the number of pixels in input equals the number of neurons in bottleneck)? This "copying CAE" hypothesis is a commonly held belief that was carried over from regular AEs (see Sections and in Masci et al. 201T).
We begin the following section by formally introducing convolutional autoencoders and explain- ing the convolutional autoencoder model used in our experiments. Additionally, we intro- duce our three datasets and the motivation for choosing them. In Section 3 we outline the ex- periments and their respective aims. Afterward, we present and discuss our findings in Sec- tion [ All of our code, as well as the trained models and datasets, will be published at https://github.cc com/ mously Anno/ullyno///alkiiiin Walking The Tightrope This repository will also include an in- teractive Jupyter Notebook for investigating the trained models. We invite interested readers to take a look and experiment with our models.
MATERIALS AND METHODS
2.1 AUTOENCODERS AND CONVOLUTIONAL AUTOENCODERS
The regular autoencoder, as introduced by Rumelhart et al. (1985). is a neural network that learns a mapping from data points in the input space x E Rd code vector in latent space h E I and back. Typically, unless we introduce some other constraint, m is set to be smaller than d to force the autoencoder learn higher-level abstractions by having to compress the data. In this context, the encoder is the mapping fz) Ed + I and the decoder is the mapping g(h) R" + Rd. The layers in both the encoder and decoder are fully connected:
g++1 = oWW' + b').
Here, I' is the activation vector in the i-th layer, W and b' are the trainable weights and is element-wise non-linear activation function. If necessary, we can tie weights in the encoder to the ones in the decoder such that W (W"-ijr, where the total number of layers. Literature refers to autoencoders with this type of encoder-decoder relation as weight-tied.
The convolutional autoencoder keeps the overall structure of the traditional autoencoder but replaces the fully connected layers with convolutions:
L ++1
a ('' L' b²),
(2)
2

---

Under review as a conference paper at ICLR 2020
where denotes the convolution operation and the bias b' is broadcast to match the shape of L' such that the j-th entry in b' is added to the j-th channel in L'. Whereas before the hidden code was an m-dimensional vector, it is now a tensor with a rank equal to the rank of the input tensor. In the case of images, that rank is three (height, width, and the number of channels). CAES generally include pooling layers or convolutions with strides > 1 or dilation > 1 in the encoder to reduce the size of the input. In the decoder, unpooling or transposed convolution layers (Dumoulin & Visin 2016) inflate the latent code to the size of the input.
2.2
OUR MODEL
Our model consists of five strided convolution layer in the encoder and five up-sampling convolution layers (bilinear up-sampling followed by padded convolution) (Odena et al. (016) in the decoder. We chose to use five layers so that the size of the latent code, after the strided convolutions, would be 4x4 or 3x3 depending on the dataset. To increase the level of abstraction in the latent code, we increased the depth of the network by placing two residual blocks (He et al.) (016) with two convolutions each after each every strided / up-sampling convolution layer. We applied instance normalization (Ulyanov et al.) (2166) and ReLU activation (Nair & Hinton 2010) following every convolution in the architecture.
One our goals was to understand the effect latent code shape has on different aspects of the network. Therefore, we wanted be able change the shape of the bottleneck from one experiment to another, while keeping the rest of the network constant. To this end, we quadrupled the number of channels with every strided convolution s' and reduced it by a factor of four with every up-sampling convolution u'. In effect, this means that the volume (i. e., height x width x channels) of the feature maps is identical to the input in all layers up the bottleneck:
s'(L') E IR
/2xdn', for L' E RB xx"'n
(3)
v'((') E 2222222x2 x2 16/4 for L' E "''''='xxn
(4)
In this regard, our model, differs from CAES commonly found literature, where it is customary to double/halve the number of channels with every down-/up-sane saapllingg layer. However, our scheme allows us to test architectures with different bottleneck shapes while ensuring that the volume of the feature maps stays the same as the input until the bottleneck. In this sense, the bottleneck is the only moving part in our experiments. The resulting models range from having ~ 50M to 90M parameters.
2.3 DATASETS
To increase the robustness of our study, we conducted experiments on three different datasets. Ad- ditionally, the three datasets allowed us to address the question, how the difficulty of the data (i. e., the complexity of the data distribution) affects learning in the CAE. To study this effect, we de- cided run our experiments on three datasets of varying difficulty. We determined the difficulty of each dataset based on intuitive heuristics. In the following, we present the datasets in the order of increasing difficulty and our reasoning for the difficulty grading.
2.3.1 POKEMON
The first dataset is blend of the images from "Pokemon Images Dataset from "The Complete Pokemon Dataset [ both of which are available on Kaggle. Our combined dataset consists of 793 256 256 pixel images of Pokemon and their primary and secondary types as labels. To keep the training time within acceptable bounds, we resized all images to be 128x 128 pixels. We chose this dataset primarily for its clear structure and simplicity. The images depict only the Pokemon without background, and each image centers on the Pokemon it is showing. Additionally, the variation in poses and color palettes limited in the images, and each image contains large regions of uniform color. Due to the above reasons and its small size, we deemed this dataset to be the "easy" dataset our experiments. We trained our models on the first 80% of images and reserved the rest for testing.
Mand and the type information
http://wwww.kaagge..coommoaaaaa
3

---

Under review as a conference paper at ICLR 2020
2.3.2 CELEBA
A step up from the Pokemon dataset in terms of difficulty is the CelebA faces dataset (Liu et al, 2013). This dataset is a collection of celebrity faces, each with a 40-dimensional attribute vector (attributes such as smiling/not smiling, male/female) and five landmarks (left and right eye. nose and left and right corner of the mouth). To be able to observe overfitting behavior, we used only the first 10,000 images in the dataset for training and the last 2,000 images for testing. Since the images also contain backgrounds of varying complexity, we argue that this leads to more complex data distribution. Furthermore, the lighting conditions, quality, and facial orientation can vary sig- nificantly in the images. However, some clear structure is still present in this dataset, as the most substantial portion of each image shows human face. For those reasons, have "medium" " difficulty. For our purposes, we resized the images to be 96 x 96 pixels. The original size was x78x218 pixels.
defined this dataset to
2.3.3 STL-10
For our last dataset, we picked STL-10 (Coates et al. 2011). This dataset consists of 96x96 pixel natural images and is divided into three splits: 5,000 training images (10 classes), 8,000 test images (10 classes), 100,000 unlabeled images. The unlabeled images also include objects that are not covered by the ten classes the training and test splits. Analogously to CelebA, we used the first 10,000 images from the unlabeled split for training and the last 2,000 for testing of the CAE. In the experiments regarding knowledge transfer (see Section ).2), we used all 8,000 labeled images from the test split of the dataset. As the images in this dataset show many different scenes, from varying viewpoints and under a multitude of lighting conditions, we find samples from this dataset to be the most complex and, therefore, the most difficult of the three.
3 EXPERIMENTS
3.1 AUTOENCODER TRAINING
The first experiment we conducted, and which forms the basis for all subsequent experiments, con- sists of training of autoencoders with varying bottleneck sizes and observing the dynamics of their training and test losses. This experiment probes the relative importance of latent code size versus its number of channels. Additionally, it meant to provide insight into how and when our models overfit and if the data complexity (see Section 2.33 plays a discernible role in this. We also tested the widespread hypothesis that autoencoders learn "copy" the input if there is no bottleneck. For each dataset (as introduced in Section 2.3) we selected three latent code sizes (=height=width) s;
i {1. 2.3
8;
Sinput
with i (1,2,3), =5
(5)
2
In this equation, ny = 5 is the number of strided convolutions in the network, and Simput is the height (= width) of the images in the dataset. Throughout the rest of the paper, we mean width and height when we refer to the size of the bottleneck. To obtain latent codes with size (sa). we changed the strides in the last (two) strided convolution layer(s) from two to one. For each size then fixed four levels of compression Cj E (//1,/ //16, //. 1} and calculated the necessary number of channels ne, according
eypptt Becoput
ney
with i E (1,2,3), j (1,2,3,4)
Here, Booppst is the number of channels in the input image. This way, the autoencoders had the same number of parameters in all layers except the ones directly preceding and following the bottleneck. We used mean squared error (MSE) between reconstruction and input as our loss function. After initializing all models with the same seed, we trained each for 1,000 epochs and computed the test error after every epoch. We repeated this process for two different seeds and used the models from the first seed in further experiments.
4

---

Under review as a conference paper at ICLR 2020
3.2 KNOWLEDGE TRANSFER
Another goal of our investigation was to estimate the effect of the latent code shape on transferabil- ity. Here, our idea was to train logistic regression on latent codes predict the corresponding labels for each dataset. Since logistic regression can only learn linear decision boundaries, this ap- proach allows us to catch glimpse of the sort of knowledge present in the latent code and its linear separability. Furthermore, this serves as another test for the "copying" hypothesis. If the encoder has indeed learned to copy the input, the results of the logistic regression will be the same for the latent codes and the input images. In the first step, we exported all latent codes for the training and testing data from the Pokemon and CelebA datasets. For STL- 10, we extracted the latent codes for the test split since we trained on the unlabeled split, where no labels are available. In the case of CelebA, we additionally trained linear regression models to predict the facial landmarks provided in the dataset. For every autoencoder setting, we used fivefold cross-validation to strengthen the reliability of the results. We trained the linear models for 200 epochs (50 epoch in the case of CelebA landmarks) with weight decay of 0.01 and learning rate of //64 (referring to Section 2.2 Besides, also trained models directly on the image data for every dataset to serve as a baseline for comparison.
3.3 PATR-WISS REPRESENTATION SIMILARITY
In our final experiment, we used the recently published singular vector canonical correlation analy- sis (SVCCA) (Raghu et al. 2017 technique gauge the pair-wise similarity of the learned latent codes. SVCCA takes two sets of neuron activations of the shape number of neurons x data points and estimates aligned directions in both spaces that have maximum correlation. First, SVCCA cal- culates the top singular vectors that explain 99% of the variance using singular value decomposition (SVD). Subsequently, SVCCA finds affine transformations for each set of singular vectors that max- imize their aligmment in the form of correlation. Lastly. it averages the correlation for each direction in the discovered subspace produce a scalar similarity score. In convolutional neural networks, this computation can become prohibitively expensive, due to the large size of the feature maps. For such cases, the |Raghu et al. (2017) recommend transforming the feature maps using discrete Fourier transformation (DFT). In the publication, the authors show that DFT leaves SVCCA invari- ant (if the dataset is translation invariant) but results in block diagonal matrix, which enables exact SVCCA computation by computing SVCCA for each neuron mend down-sampling bigger feature maps in Fourier space when comparing them to smaller ones. In this experiment, we investigated the effect of latent code shape on its structure and content.
time. Additionally, they recom-
RESULTS AND DISCUSSION
Looking at the error curves for the CAES (Fig. D. we make several observations:
1. The total amount of neurons in the bottleneck does not affect training much as expected. All CAES converge to a similar training error. We find this unexpected, as the smallest bottlenecks have only 1.56% of total neurons compared to the largest ones. Although the final differences in training error are small, we discover that the size of the bottleneck feature maps has a more substantial effect on training error than the number of channels. The larger the bottleneck width and height, the lower the training error. An interesting outlier presents itself in the plots for the Pokemon dataset. Here, we see that late in the training of the CAE with the 8x8x48 bottleneck training error suddenly spikes. At the same time, the test error drops significantly approximately to the same level as the training error. We verified that this was not due to an unintended interruption in training, by retraining the model with the same seed and obtained an identical result. Currently, it is unclear to us how such drastic change in model parameters came about at such a late stage in training. Usually, we expect the loss landscape to become smoother the longer we train a model (Goodfellow et al. 2014). Whether this outlier is fluke or has implications for the loss landscape of CAES remains to be seen as our understanding of the training dynamics of neural networks deepens.
2.
We observe that bottleneck shape critically affects generalization. Increasing the number of channels in the bottleneck layer seems to improve test error only slightly and not in all cases. The relationship between bottleneck size and test error, on the other hand, is clear
5

---

Under review as a conference paper at ICLR 2020
Pakemon training loss
CelebA training loss
STL-10 training loss
0.27
0.07
0.56
0.06
2000
0.05
2000/0
0.06
0.04
0.04
0.23
0.02
0.02
9.01
0.31
0.01
Pokemon test loss
CelebA test loss
STL-10 test loss
0.07
9.66
0.06
0.06
0.25
0.05
0.
0.04
0.03
0.23
0.02
0.01
0.01
0.91
0.0
202
400
600
800
1030
0.0
200
400
600
800
1000
200
400
600
800
1000
opochs
opachs
op ochs
(a) Seed 0
Pokemon training loss
CelebA training loss
STL-10 training loss
2.57
0.37
0.07
16000000
30
0.05
0.05
0.05
MS6
2
0.24
02
0.04
9.02
0.32
0.02
0.01
0.01
0.0
0.0
Pokemon test loss
CelebA test loss
10 test loss
0.67
0.07
0.07
0.26
0.05
0.05
0.04
W 0.24
0.04
0.52
0.32
0.92
0.01
0.33
0.03
0.0
spechs
epachs
opachs
(b) Sced
Figure 1: Loss plots for the three datasets for both seeds. Each columns corresponds to a dataset. From left to right: a) Pokemon, b) CelebA, e) STL-10. The top polot shows the training error, while the bottom one depicts test error. Every bottleneck configuration is shown as distinct line. Contigurations that have a common feature map size share the same color. Color intensity represents the amount of channels in the bottleneck (darker more channels)
cut. Larger bottleneck size correlates with a significant decrease in test error. This finding is surprising, given the hypothesis that only the total amount of neurons matters. The CAE reconstructions further confirm this hypothesis. We visually inspected the reconstructions of our models (samples are shown in Fig. 2 and in the Appendix) and found that recon- struction quality improves drastically with the size of the bottleneck, yet no so much with the number of channels. As expected from the loss plots, the effect is more pronounced for samples from the test data.
3. Bottleneck shape also affects overfitting dynamics. We would expect the test score to in- crease after reaching a minimum, the CAE overfits the data. Indeed, we observe this behavior in some cases, especially in CAFS with smaller bottleneck sizes or the minimum amount of channels. In other cases, predominantly in CAEs with a larger bottleneck size, the test error appears to platcau instead. In the plot for the CelebA dataset, the curves for 12x 1x48 and 12x12x119 even appear to decrease slightly over the full training duration. This overfitting behavior implies that CAEs with larger bottleneck size can be trained longer before overfitting occurs.
4. CAES, where the total number of neurons in the bottleneck is the same as the number of pixels in the input, do not show signs of simply copying images. If the CAES would indeed copy images, the test error would go to zero, yet we do not observe this case in any of the datasets. What is more, these complete CAEs follow the same pattern the under-

---

Under review as a conference paper at ICLR 2020
(a) Pokemon training sample
(b) Pokemon test sample
(c) CelebA training sample
(d) CelebA test sample
(e) STL-10 training sample
(1) STL-10 test sample
Figure 2: Reconstructions of randomly picked samples. The left column contains samples from the training data, while on the right, we show samples from the test data. In each subfigure, the rows correspond to CAEs with the same bottleneck size (height, width), increasing from top to bottom. The columns group CAEs by the number of channels in the bottleneck, expressed as percentage relative to input given bottleneck size. The image to the left of each grid is the input image.
complete ones and often converge to similar values. This finding directly contradicts the popular hypothesis about copying CAEs. essence, it suggests that even complete CAES learn abstractions from data, and raises the question: What prevents the CAE from simply copying its input? We believe that the answer to this question could potentially lead to new autoencoder designs that exploit this limitation to learn better representations. Hence, argue that it is an exciting direction for future research. Additionally, the trends we derive from our results suggest that this finding likely extends over-complete CAEs well. However, experiments with over-complete CAÉS are required to test this intuition.
Furthermore, the loss curves and reconstruction samples appear to only marginally reflect the notion of dataset difficulty, defined in Section 2.3 One thing that stands out is the large generalization gap on the Pokemon dataset, which is most likely due to the comparatively tiny dataset size of R 600 training images. Comparing the results for CelebA and STL-10, we find that overall general- ization appears to be slightly better for CelebA, which is the less difficult dataset of the two. The test errors on STL-10 exhibit greater variance than CelebA, although the number of samples and training epochs are equal between the two. This effect also shows itself in the reconstruction quality. On CelebA, even the CAEs with the smallest bottlenecks manage produce decent reconstructions
7

---

Under review as a conference paper at ICLR 2020
Pakeman train
Celebbattriutbsstm train
feaure mar 206
Parit
1011
14:3
1923
2011
valuma 18 MEve baseira
valum relativ bassine
2303
voune Netvee toteiir
5224
Pokemon test
attrbutts test
STL-10 test
Raltrr moceennpss sue
map Saa
velume 10 bastline
valump inlatvv baseliee
TMME 1036
volume re baseline
Figure 3: Results from training linear models on latent codes to predict the labels associated with each dataset. For Pokemon, CelebA attributes and STL-10 (macro) f'lsccoe is shown. The plots for CelebA regression show MSE. The top row corresponds to models trained on latent codes from the CAE training data, while the bottom row is from CAE test data. Color is based on difference to baseline, where red signifies an improvement.
Poicemnn
Pakemon test
Celebá train
Colebd test
STL-10 test
093
90-3
563
16.0
12.4
13.1
13.
311
610
6-18
Figure 4: Results of pair-wise SVCCA. Labels on the x and y axis correspond to (height=width)- (number of channels) in the bottleneck.
on test data, whereas the test sample reconstructions on STL-10 are often unrecognizable for those models. Overall, this effect is weak and warrants separate investigation of the relationship be- tween data complexity and CAE characteristics, especially in the light of compelling results from curriculum learning research (Bengio al. 2009).
If we look at the results of our knowledge transfer experiments (Fig. 3). we find further evidence that contradicts the copying autoencoder hypothesis. Although the loss curves and reconstructions already indicate that the CAE does not copy its input, the possibility remains that the encoder dis- tributes the input pixels along the channels but the decoder unable to reassemble the image. Here, we see that the results from the linear model trained on latent codes perform drastically better, than the ones trained on the inputs (marked "baseline" in the figure). The only deviation from this pattern seems to be the prediction of attributes on the CelebA dataset, where the performance is more or less the same for all settings. However, the prediction of landmarks on the same dataset strongly favors latent codes over raw data. As such, it seems implausible to assume that the encoder copied the input to the bottleneck. Overall, we find that knowledge transfer also seems to work better on latent codes with greater size, although the effect is not distinct as in the loss curves.
Another point of interest to us is the discrepancy between models trained on the CAE training and test data from the Pokemon dataset. Oddly, the linear models perform better on the test data, despite the evident overfitting of the CAEs as seen in the reconstructions and loss curves. This discrepancy raises the question if overfitting happens mostly in the decoder, while the encoder retains most of its generality. We believe that this question warrants further investigation, especially in light of the recent growth in the popularity of transfer learning methods.
8

---

Under review as a conference paper at ICLR 2020
We notice that the latent codes from bottlenecks with the same size have higher SVCCA similarity values, as can be seen in Fig. 4 in the blocks on the diagonal. This observation further supports our hypothesis that latent code size, and not the number of channels, dictates the tightness of the CAE bottleneck. Finally, we wish to point out some observations in the SVCCA similarities as possible inspiration for future research:
Overall, similarity appears to be higher in latent codes from test data than in codes from training data
Latent codes from complete CAEs show high similarity all latent codes from all other CAEs
SVCCA similarity with the raw inputs tends to increase with the number of channels
5 CONCLUSION
In this paper, we presented the findings of our in-depth investigation of the CAE bottleneck. The intuitive assumption that its total amount of neurons characterizes the CAE bottleneck could be confirmed. We demonstrate that the height and width of the feature maps in the bottleneck are what defines its tightness, while the number of channels plays secondary role. Larger bottleneck size (i. e., height and width) is also critical in achieving better generalization as well as a lower training error. Furthermore, we could not confirm the commonly held belief that complete CAE (i. e., CAEs with the same number of neurons in the bottleneck pixels in the input) will learn to copy its input. On the contrary, even complete CAEs appear to follow the same dynamics of bottleneck size, as stated above. In knowledge transfer experiments, we have also shown that CAEs that overfit retain good predictive power in the latent codes, even on unseen samples. These insights are directly transferable to the two main areas of application for CAEs, outlier detection and compression/denoising:: In the case of outlier detection, the model should yield a high reconstruction eтror on out-of out-of-distripe distribution samples. Using smaller bottleneck sizes limit generalization could prove useful in this scenario. Compression and denoising tasks, on the other hand, seek to preserve image details while reducing file size and discarding unnecessary information, respectively. In this case, bigger bottleneck size is preferable, as it increases reconstruction quality at the same level of compression.
Our investigation yielded additional results that spark new research questions. Data complexity, as estimated by human intuition, did not lead to significant differences in the training dynamics of our models. On the flipside, curriculum learning, which rests on a similar notion of difficulty, has been shown to lead to improvements in the training of classifiers and segmentation networks. The link between those two empirical results is still unclear. Another interesting question that arose from our experiments is how overfitting manifests itself in CAFs. Does it occurs mainly in the encoder or decoder or equally in both?

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. What Regularized Auto-Encoders Learn from the Data Gen- crating Distribution. arXiv e-prints, art. 12Xiv:1211.42440 Nov 2012.
Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma. Provable bounds for learning some deep representations. CoRR, abs/1310.6343,, 2013. URL
Devansh Arpit, Yingbo Zhou, Hung Ngo, and Venu Govindaraju. Why Regularized Auto-Encoders learn Sparse Representation? arXiv e-prints, art. 1rXiv:1505.05560 May 2015.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham Taylor, and Daniel Silver (eds.), Proceedings of ICML Workshop on Unsupervised and Transfer Learning. volume 27 of Proceedings of Machine Learn- ing Research, pp. 37-49, Bellevue, Washington, USA, 02 Jul 2012. PMLR. URL http: \/proceed ngs. ml prwss v2 / l dii 12a html
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding mod- els for unsupervised anomaly segmentation in brain mr images. In Alessandro Crimi, Spyridon Bakas, Hugo Kuijf, Farahani Keyvan, Mauricio Reyes, and Theo van Walsum (eds.). Brainle- sion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 161-169, Cham, 2019. Springer International Publishing. ISBN 978-3-030-11723-4
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf, J. C. Platt, and T. Hoffman (eds.), Advances in Neural Infor- mation Processing Systems 19, pp. 153-160. MIT Press, 2007. http:htttp//pppeeee nips. pc/ 304///34885-0 100aaaee//00---eeed--ee layer wise of oww.aanngo..-ddde---eeetooo..... .ddf
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. CoRR, abs/1305.6663,, 2013. UtL.
6663
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via adversarial regularizer. arXiv preprint urXiv:1807.005 2018.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Improving interpolation in autoen- coders. 2019. URL http:htttt:://peeerriee net /tt.tett/.ff.e ?id= CQSiCcYm
Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga Oulier Detection with Au- loencoder Ensembles, pp. 90-98. 2017. doi: 10.1 10.1137/1.9781111990
URL https:
//epubs. siam. org/ wogg/do////oooooooooooooo//ooooo abs/ /00 1tt.///1.111111111. 9781 /11
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Deep convolutional autoencoder-based lossy image compression. In 2018 Picture Coding Symposium (PCS), pp. 253-257, June 2018. doi: 10.1 PCS.2018.8456305
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, 2011.
T. Dumas, A. Roumy, and C. Guillemot. Autoencoder based image compression: Can the learning be quantization independent? 2018 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1188-1192, April 2018. doi: 10. (ttp:/CAAASS22188842222222
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. arXiv preprint /603.00203 2016.
10

---

Under review as a conference paper at ICLR 2020
Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty training deep architectures and the effect unsupervised pre-training. In David van Artificial
Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference Intelligence and Statistics, volume 5 Proceedings of Machine Learning Research, pp. 153- 160, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR. URL
m. wwt.e/ppess.eeeeeeeeeeee
html
lan J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv: 1412.654,, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114 Dec 2013.
Quoe V. Le. Building high-level features using large scale unsupervised learning. 2013 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 8595-8598, 2013. ISSN 1520-6149. doi: 10. 11110//CASSP2223..699944 URUhtttp:/eeexxoooooo leee.
org doo.ummettt6333333
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Linet: A deep autoencoder approach to
natural low-light image enhancement. Puttern Recognition, 61:650 662, 2017. ISSN 0031-3203. doi: /ccpaann///rrrreeee/////01111310 com/
URL http::/www.w http///wwwsciiencdrrrre.e..
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian J. Goodfellow. Adversarial autoen- coders. CoRR, abs/151 2015. URL URL /ttt::///axxiii wwwpii..rr/abs//////.. 05644
Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Image Restoration Using Convolutional Auto- encoders with Symmetric Skip Connections. arXiv e-prints, art. arXiv:1606.0888 Jun 2016.
Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. Stacked convolutional auto- encoders for hierarchical feature extraction. In International Conference Artificial Neural Networks, pp. 52-59. Springer, 2011.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Thanh V. Nguyen, Raymond K. W. Wong, and Chinmay Hegde. Autoencoders Learn Generative Linear Models. arXiv e-prints, art. mmXiv:1866.00572 Jun 2018.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard arti- facts. Distill, 2016. doi: 10.23955/iittill00000.. URL http:::////ssill.uubbbb
Geconv-checkerbonn
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by emrrr propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and lasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In The European Conference Computer Vision (ECCV), September 2018.
11

---

Under review as a conference paper at ICLR 2020
Edgar Tretschk, Ayush Tewari, Michael Zollhöfer, Vladislav Golyanik, and Christian Theobalt. DE- MEA: deep mesh autoencoders for non rigidly deforming objects. CoRR, abs/1905.102900 2019.
http:tt:///ax...oo////poo// 1905 10290
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini urXiv:1607.0060 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008.
Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Ozal Yildirim, Ru San Tan, and U. Rajendra Acharya. An efficient compression of ecg sig- nals using deep convolutional autoencoders. Cognitive Systems Research, 52:198 211, 2018. ISSN 1389-0417. doi: http:://bb..og///00000///cssossp www.::/.cccsss........0000
URL http: www.
Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining. KDD 17, pp. 665-674, New York, NY, USA, 2017. ACM. ISBN
4. doi: 10.1145/3099983 309805...
URL sttp://dee acm. 1r9/1001145/3397993
3098052
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
130
227
20
110%
10.0%
1.0
Figure A.I: Reconstructions of randomly picked samples from the Pokemon dataset. The left col- umn contains samples from the training data, while on the right, we show samples from the test data. In each subfigure, the rows correspond to CAEs with the same bottleneck size (height, width), in- creasing from top to bottom. The columns group CAES by the number of channels in the bottleneck, expressed percentage relative to input given bottleneck size. The image to the left of each grid is the input image.
13

---

Under review as a conference paper at ICLR 2020
AIT
110%
4.7%
Figure A.2: Reconstructions of randomly picked samples from the CelebA dataset. The left column contains samples from the training data, while on the right, we show samples from the test data. In each subfigure, the rows correspond to CAEs with the same bottleneck size (height, width), increas- ing from top bottom. The columns group CAES by the number of channels in the bottleneck, expressed as percentage relative to input given bottleneck size. The image to the left of each grid is the input image.
14

---

Under review as a conference paper at ICLR 2020
Figure A.3: Reconstructions of randomly picked samples from the STL-10 dataset. The left column contains samples from the training data, while on the right, we show samples from the test data. In each subfigure, the rows correspond to CAEs with the same bottleneck size (height, width), increas- ing from top bottom. The columns group CAES by the number of channels in the bottleneck, expressed as percentage relative to input given bottleneck size. The image to the left of each grid is the input image.
15


Reviews:
[{"review_id": "S1li_V7ecS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates convolutional autoencoder (CAE) bottleneck. The research problem is important, as CAE is widespread adopted. It cam be interesting to shed insight into how the bottleneck works. \n\nIn particular, two observations are made: (1) By measuring the performance of the latent codes in downstream transfer learning tasks, the authors show that  increased height/width of the bottleneck drastically improves generalization; The number of channels in the bottleneck is secondary in importance.  (2) CAEs do not learn to copy their input, even when the bottleneck has the same number of neurons as there are pixels in the input.\n\nIt would make this submission more convincing if the authors show the some important applications of their findings. For example, \n\nA. How does (1) above help architecture search when designing CAEs in a new application? \nB. How does (2) above help style transfer if ``\"CAEs do not learn to copy their input\" \n\nConcerns: If (2) is true, Could the authors explain that almost perfect reconstruction results are shown in [*] ?\n\n[*] Generating Diverse High-Fidelity Images with VQ-VAE-2", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xTS8cptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies some of the properties of fully convolutional autoencoders (CAE) as a function of the shape and total size of the bottleneck. They train and test CAEs with bottlenecks consisting of different ratios of spatial resolution versus number of channels, as well as different total number of neurons. The authors investigate which type of change in the bottleneck is most influential on training behavior, generalization to test set, and linear separability for classification/regression. Their first main finding is that the spatial resolution of the bottleneck is a stronger influencer of generalization to the test set than the number of channels and the total number of neurons in the bottleneck. The second main finding is that even when the total number of neurons in the bottleneck is equal to the data input size, the neural network does not appear to simply learn to copy the input image into the bottleneck. \n\n\nDecision: \nWeak reject: It is always refreshing to see papers that address/challenge/investigate common assumptions in deep learning. However, I find the experimental findings and discussion of borderline quality for a full conference paper. It might be more suitable as a good workshop paper.\n\nSupporting arguments for decision:\nIt is unclear to me why the authors have chosen to only take a subset of the CelebA and STL-10 datasets for training and testing. It seems like dataset size is also an important factor that increases the complexity for training a model, and it certainly affects how a model can generalize. When auto-encoders are studied in other literature it is uncommon practice to restrict the dataset sizes like this, so this makes me question the applicability of this paper’s results to the literature. \n\nIt seems that the experimental validation is based on one run per CAE model with one single seed. This is on the low side of things, especially when quite extensive claims are made. An example of such a claim is on page 6 when discussing a sudden jump in training and test scores for 8x8x48 model trained on the Pokemon dataset. Because the same behavior appeared when the authors repeated the experiment with the same seed, the authors conclude “This outlier suggests, that the loss landscape might not always be as smooth towards the end of training, as some publications (Goodfellow et al., 2014) claim and that ‘cliffs” (i. e., sudden changes in loss) can occur even late in training.” Making this claim based on something that occurs with one single model for a single seed is not convincing and overstating this finding. \nAnother example is on page 7 under bullet point 4, where the authors discuss the obtained evidence against copying behaviour when the bottleneck is of the same size as the input. The authors state “We believe this finding to have far-reaching consequences as it directly contradicts the popular hypothesis about copying CAEs.” The paper definitely shows some empirical evidence that supports the claim that copying does not occur, but these findings are all done with a single seed and by considering small subsets of datasets (celebA and stl-10). In my opinion, it is therefore too much to state that the current findings have far reaching consequences. It has potential, but I wouldn’t go much further than that. \n\nOn page 7 in the second to last paragraph the influence of dataset complexity is discussed. The authors state “the loss curves and reconstruction samples do not appear to reflect the notion of dataset difficulty we defined in Section 2.3” and “This lack of correspondence implies that the intuitive and neural network definitions of difficulty do not align. Nevertheless, a more detailed study is required to answer this question definitively as curriculum learning research that suggests the opposite (Bengio et al., 2009) also exists.” It is unclear to me what the authors expected to find here. Moreover, the absence of major differences across the chosen datasets does not immediately make me doubt or question results from curriculum learning. My skepticism is again enhanced by the fact that the authors have taken a subset of the data for the more complex celebA and STL-10 datasets. Dataset size seems like a crucial part of dataset complexity.\n\nAn interesting smaller finding is that linear separability of the latent codes for classification is better on the test set for the pokemon dataset, even though the training and test reconstruction losses showed signs of overfitting. The authors hypothesize that overfitting might occur more in the decoder than in the encoder.\n\nAdditional feedback to improve the paper (not part of decision assessment)\n- Section 2.3.1: what was the original resolution of the pokemon dataset?\n- Section 3.1, c_j is used as the compression level, but in eq 3 and 4 c^i is also used to indicate the number of channels. - For clarity please use n_ci in eq 2 and 3 or choose a different all together for the compression level.\n- Please increase the font size of the plots in figures 1, 3 and 4.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byg17qZy9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors evaluate convolutional autoencoders (CAE) by varying the size (width & height) and depth of the bottleneck layer on three datasets and compare test and training performance. They furthermore evaluate the quality of the bottleneck activations for linear classification. The authors also investigate the belief that a bottleneck layer of size equal to the input image will copy the image. \n\nI am not an expert in the field of (C)AEs. As such, I cannot approriately judge the relevance of the questions which are answered here. In the following, I will therefore make the assumption that those questions are relevant.\nIf so, I (weakly) recommend accepting the paper. \n\nWhile it does not propose any novel algorithms, it does ask a clear question and provides a compelling experimental answer, which (assuming the question is relevant), should be interesting for the community. \nOn the other hand, further experiments to provide initial insights into the further questions raised by the authors would improve the 'novelty' aspect of the paper. \n\nMinor comment: \nI believe the paper \"Reconciling modern machine learning practice and the bias-variance trade-off\" by Belkin et. al is relevant. \n\nEdit:\nThank you for your response.\n\nI wasn't referring to any specific experiments, but the questions you are raising in the paper, for example in the conclusion section. \n\nRegarding Belkin et al: Sorry for the misunderstanding, I didn't mean to imply that it's highly relevant, but I thought it might be a good addition to the related work section as it also relates to the generalization of neural networks.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the effect of convolutional information bottlenecks to generalization. The paper concludes that the width and height of the bottleneck can greatly influence generalization, whereas the number of channels has smaller effect. The paper also shows evidence against a common belief that CAEs with sufficiently large bottleneck will learn an identity map. 

During the rebuttal period, there was a long discussion mainly about the sufficiency of the experimental setup and the trustworthiness of the claims made in the paper. A paper that empirically investigates an exiting method or belief should include extensive experiments of high quality in to enable general conclusions. I’m thus recommending rejection, but encourage the authors to improve the experiments and resubmitting.
Decision: Reject