Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

In this paper , we present an in-depth investigation of the convolutional autoencoder ( CAE ) bottleneck . Autoencoders ( AE ) , and especially their convolutional variants , play a vital role in the current deep learning toolbox . Researchers and practitioners employ CAEs for a variety of tasks , ranging from outlier detection and compression to transfer and representation learning . Despite their widespread adoption , we have limited insight into how the bottleneck shape impacts the emergent properties of the CAE . We demonstrate that increased height and width of the bottleneck drastically improves generalization , which in turn leads to better performance of the latent codes in downstream transfer learning tasks . The number of channels in the bottleneck , on the other hand , is secondary in importance . Furthermore , we show empirically that , contrary to popular belief , CAEs do not learn to copy their input , even when the bottleneck has the same number of neurons as there are pixels in the input . Copying does not occur , despite training the CAE for 1,000 epochs on a tiny ( ≈ 600 images ) dataset . We believe that the findings in this paper are directly applicable and will lead to improvements in models that rely on CAEs . 1 INTRODUCTION . Autoencoders ( AE ) are an integral part of the neural network toolkit . They are a class of neural networks that consist of an encoder and decoder part and are trained by reconstructing datapoints after encoding them . Due to their conceptual simplicity , autoencoders often appear in teaching materials as introductory models to the field of deep unsupervised learning . Nevertheless , autoencoders have enabled major contributions in the application and research of the field . The main areas of application include outlier detection ( Xia et al. , 2015 ; Chen et al. , 2017 ; Zhou & Paffenroth , 2017 ; Baur et al. , 2019 ) , data compression ( Yildirim et al. , 2018 ; Cheng et al. , 2018 ; Dumas et al. , 2018 ) , and image enhancement ( Mao et al. , 2016 ; Lore et al. , 2017 ) . In the early days of deep learning , autoencoders were a crucial tool for the training of deep models . Training large ( by the standards of the time ) models was challenging , due to the lack of big datasets and computational resources . One way around this problem was to pre-train some or all layers of the network greedily by treating them as autoencoders with one hidden layer ( Bengio et al. , 2007 ) . Subsequently , Erhan et al . ( 2009 ) demonstrated that autoencoder pre-training also benefits generalization . Currently , researchers in the field of representation learning frequently rely on autoencoders for learning nuanced and high-level representations of data ( Kingma & Welling , 2013 ; Tretschk et al. , 2019 ; Shu et al. , 2018 ; Makhzani et al. , 2015 ; Berthelot et al. , 2018 ) . However , despite its widespread use , we propose that the ( deep ) autoencoder model is not well understood . Many papers have aimed to deepen our understanding of the autoencoder through theoretical analysis ( Nguyen et al. , 2018 ; Arora et al. , 2013 ; Baldi , 2012 ; Alain & Bengio , 2012 ) . While such analyses provide valuable theoretical insight , there is a significant discrepancy between the theoretical frameworks and actual behavior of autoencoders in practice , mainly due to the assumptions made ( e.g. , weight tying , infinite depth ) or the simplicity of the models under study . Others have approached this issue from a more experimental angle ( Arpit et al. , 2015 ; Bengio et al. , 2013 ; Le , 2013 ; Vincent et al. , 2008 ; Berthelot et al. , 2019 ) . Such investigations are part of an ongoing effort to understand the behavior of autoencoders in a variety of settings . The focus of most such investigations so far has been the traditional autoencoder setting with fully connected layers . When working with image data , however , the default choice is to use convolutions , as they provide a prior that is well suited to this type of data ( Ulyanov et al. , 2018 ) . For this reason , Masci et al . ( 2011 ) introduced the convolutional autoencoder ( CAE ) by replacing the fully connected layers in the classical AE with convolutions . In an autoencoder , the layer with the least amount of neurons is referred to as a bottleneck . In the regular AE , this bottleneck is simply a vector ( rank-1 tensor ) . In CAEs , however , the bottleneck assumes the shape of a multichannel image ( rank-3 tensor , height × width × channels ) instead of a vector . This bottleneck shape prompts the question : What is the relative importance of the number of channels versus the height and width ( hereafter referred to as size ) in determining the tightness of the CAE bottleneck ? Intuitively , we might expect that only the total number of neurons should matter since convolutions with one-hot filters can distribute values across channels . Generally , the study of CAE properties appears to be underrepresented in literature , despite their widespread adoption . In this paper , we share new insights into the properties of convolutional autoencoders , which we gained through extensive experimentation . We address the following questions : • How does the number of channels and the feature map size in the bottleneck layer impact – reconstruction quality ? – generalization ability ? – the structure of the latent code ? – knowledge transfer to downstream tasks ? • How and when do CAEs overfit ? • How does the complexity of the data distribution affect all of the above ? • Are CAEs capable of learning a “ copy function ” if the CAE is complete ( i. e. , when the number of pixels in input equals the number of neurons in bottleneck ) ? This “ copying CAE ” hypothesis is a commonly held belief that was carried over from regular AEs ( see Sections 4 and 5 in Masci et al . ( 2011 ) . We begin the following section by formally introducing convolutional autoencoders and explaining the convolutional autoencoder model we used in our experiments . Additionally , we introduce our three datasets and the motivation for choosing them . In Section 3 , we outline the experiments and their respective aims . Afterward , we present and discuss our findings in Section 4 . All of our code , as well as the trained models and datasets , will be published at https : //github.com/YmouslyAnon/WalkingTheTightrope . This repository will also include an interactive Jupyter Notebook for investigating the trained models . We invite interested readers to take a look and experiment with our models . 2 MATERIALS AND METHODS . 2.1 AUTOENCODERS AND CONVOLUTIONAL AUTOENCODERS . The regular autoencoder , as introduced by Rumelhart et al . ( 1985 ) , is a neural network that learns a mapping from data points in the input space x ∈ Rd to a code vector in latent space h ∈ Rm and back . Typically , unless we introduce some other constraint , m is set to be smaller than d to force the autoencoder to learn higher-level abstractions by having to compress the data . In this context , the encoder is the mapping f ( x ) : Rd → Rm and the decoder is the mapping g ( h ) : Rm → Rd . The layers in both the encoder and decoder are fully connected : li+1 = σ ( W ili + bi ) . ( 1 ) Here , li is the activation vector in the i-th layer , W i and bi are the trainable weights and σ is a element-wise non-linear activation function . If necessary , we can tie weights in the encoder to the ones in the decoder such that W i = ( W n−i ) T , where n is the total number of layers . Literature refers to autoencoders with this type of encoder-decoder relation as weight-tied . The convolutional autoencoder keeps the overall structure of the traditional autoencoder but replaces the fully connected layers with convolutions : Li+1 = σ ( Wi ∗ Li + bi ) , ( 2 ) where ∗ denotes the convolution operation and the bias bi is broadcast to match the shape of Li such that the j-th entry in bi is added to the j-th channel in Li . Whereas before the hidden code was an m-dimensional vector , it is now a tensor with a rank equal to the rank of the input tensor . In the case of images , that rank is three ( height , width , and the number of channels ) . CAEs generally include pooling layers or convolutions with strides > 1 or dilation > 1 in the encoder to reduce the size of the input . In the decoder , unpooling or transposed convolution layers ( Dumoulin & Visin , 2016 ) inflate the latent code to the size of the input . 2.2 OUR MODEL . Our model consists of five strided convolution layer in the encoder and five up-sampling convolution layers ( bilinear up-sampling followed by padded convolution ) ( Odena et al. , 2016 ) in the decoder . We chose to use five layers so that the size of the latent code , after the strided convolutions , would be 4x4 or 3x3 depending on the dataset . To increase the level of abstraction in the latent code , we increased the depth of the network by placing two residual blocks ( He et al. , 2016 ) with two convolutions each after each every strided / up-sampling convolution layer . We applied instance normalization ( Ulyanov et al. , 2016 ) and ReLU activation ( Nair & Hinton , 2010 ) following every convolution in the architecture . One of our goals was to understand the effect latent code shape has on different aspects of the network . Therefore , we wanted to be able to change the shape of the bottleneck from one experiment to another , while keeping the rest of the network constant . To this end , we quadrupled the number of channels with every strided convolution si and reduced it by a factor of four with every up-sampling convolution ui . In effect , this means that the volume ( i. e. , height×width× channels ) of the feature maps is identical to the input in all layers up to the bottleneck : si ( Li ) ∈ R hi/2×w i /2×4nic , for Li ∈ Rh i×wi×nic ( 3 ) ui ( Li ) ∈ R2h i×2wi×n i c/4 , for Li ∈ Rh i×wi×nic ( 4 ) In this regard , our model , differs from CAEs commonly found in literature , where it is customary to double/halve the number of channels with every down-/up-sampling layer . However , our scheme allows us to test architectures with different bottleneck shapes while ensuring that the volume of the feature maps stays the same as the input until the bottleneck . In this sense , the bottleneck is the only moving part in our experiments . The resulting models range from having ∼ 50M to 90M parameters . 2.3 DATASETS . To increase the robustness of our study , we conducted experiments on three different datasets . Additionally , the three datasets allowed us to address the question , how the difficulty of the data ( i. e. , the complexity of the data distribution ) affects learning in the CAE . To study this effect , we decided to run our experiments on three datasets of varying difficulty . We determined the difficulty of each dataset based on intuitive heuristics . In the following , we present the datasets in the order of increasing difficulty and our reasoning for the difficulty grading .

Summary: This paper studies some of the properties of fully convolutional autoencoders (CAE) as a function of the shape and total size of the bottleneck. They train and test CAEs with bottlenecks consisting of different ratios of spatial resolution versus number of channels, as well as different total number of neurons. The authors investigate which type of change in the bottleneck is most influential on training behavior, generalization to test set, and linear separability for classification/regression. Their first main finding is that the spatial resolution of the bottleneck is a stronger influencer of generalization to the test set than the number of channels and the total number of neurons in the bottleneck. The second main finding is that even when the total number of neurons in the bottleneck is equal to the data input size, the neural network does not appear to simply learn to copy the input image into the bottleneck.