Downsampling leads to Image Memorization in Convolutional Autoencoders

1 INTRODUCTION AND RELATED WORK . As deep convolutional neural networks ( CNNs ) become ubiquitous in computer vision due to their applicability and strong performance on a range of tasks ( Goodfellow et al. , 2016 ) , recent work has begun analyzing the memorization properties of such networks in classification . For example , Zhang et al . ( 2017 ) show that popular CNNs can achieve almost zero training error on randomly labeled datasets , indicating that CNNs have the capacity to “ memorize ” large training data sets . Arpit et al . ( 2017 ) and Sablayrolles et al . ( 2018 ) build on the experiments from Zhang et al . ( 2017 ) to better understand and evaluate the extent to which CNNs memorize training data . Arpit et al . ( 2017 ) show that CNNs , when trained on large datasets , are able to learn patterns from realistic data before memorizing training images . Sablayrolles et al . ( 2018 ) present experiments on “ membership inference ” ( i.e . determining whether an image was used during training ) and conclude that modern architectures are capable of “ remember [ ing ] a large number of images and distinguish [ ing ] them from unseen images ” . Although the above methods analyze memorization in the classification setting , they do not provide a mechanism through which memorization of training data occurs . We here present downsampling as one mechanism by which deep CNNs memorize specific training images . We will focus our study on the memorization properties of linear and nonlinear fully convolutional autoencoders . The architectures we use ( such as U-Net , ( Ronneberger et al. , 2015 ) ) are commonly employed in imageto-image tasks , see e.g . Ulyanov et al . ( 2017 ) . However , we will use these architectures only in the autoencoding framework . We primarily focus on autoencoders ( Baldi , 2012 ) for the following reasons : ( 1 ) components of convolutional autoencoders are building blocks of many CNNs ; and ( 2 ) layerwise pre-training using autoencoders is a technique to initialize individual layers of CNNs to improve training ( Bengio et al . ( 2007 ) , Erhan et al . ( 2010 ) ) . It is important to note that there are many potential solutions to the autoencoding problem when using over-parameterized autoencoders . In particular , in the linear case , these models may range from learning the ( full rank ) identity function ( which has 0 error in the autoencoding task ) to low rank solutions where each training example corresponds to an eigenvector with eigenvalue 1 . Thus , understanding how autoencoders learn is of interest in order to gain insights into how deep CNNs memorize training data . Figures 1a and 1b provide two examples of memorization : A typical U-Net architecture ( the same as e.g . used in Ulyanov et al . ( 2017 ) for large hole impainting ) when trained on a single image “ memorizes ” the training image in the sense that for any input , the output always contains the training image ( even if the input is random noise or an arbitrary white square ) . This paper provides a mechanism for this phenomenon . The outline is as follows : After introducing some notation in Section 2 , we will show in Section 3 that memorization is tightly coupled with downsampling and also occurs in the simpler setting of linear autoencoding CNNs . In the linear setting , the neural network corresponds to matrix multiplication . In Section 4 , we show how to extract this matrix representation and we provide our main conjecture , namely that linear combinations of the training images are stored as eigenvectors of this matrix , whose rank is given by the dimension of the span of the training set . We also provide strong evidence for this conjecture on 2 × 2 images . In Section 5 , we analyze the eigenvalue decay and show in various examples that using downsampling linear CNNs , linear combinations of the training examples are stored as eigenvectors with eigenvalues close to 1 . Finally , we return to the nonlinear setting in Section 6 , providing evidence that memorization is an even stronger phenomenon in nonlinear networks , since the actual training images ( in contrast to linear combinations of training images ) are memorized . We end with a short discussion in Section 7 . 2 NOTATION . In this section , we introduce the mathematical framework for our work and highlight two different functions learned by autoencoding CNNs , namely the identity function and the point map . We denote a training set of n square images by X = { x1 , x2 , ... , xn } , where each xi ∈ Rc×s×s with c being the number of color channels ( or filter channels ) and s denoting the width and height of an image . We will focus on convolutional autoencoders , i.e. , CNNs trained to map between the same image . In particular , we will consider linear CNNs , by which we mean convolutional autoencoders with layers being either nearest neighbor upsampling or convolutional with kernel size 3 , zero padding , no activation functions , and no biases . To simplify the computations in Section 4 , we assume throughout that the input image as well as the stride size are a power of 2 . We denote the function learned by a CNN on the training set X by CX : Rc×s×s → Rc×s×s . The training procedure minimizes a given loss function between the input image and its reconstruction by CX . We use the mean squared error loss throughout ; thus the loss function is given by∑i=n i=1 1 ncs2 ( CX ( xi ) − xi ) 2 , where subtraction and exponentiation are taken elementwise . For linear CNNs , we denote the matrix corresponding to CX by AX . Denoting the vectorized ( or “ flattened ” ) version of an image y ∈ Rc×s×s by yf ∈ Rcs 2 , then AX satisfies AXyf = ( CX ( y ) ) f . In this work , we identify and analyze an architectural mechanism that is able to fundamentally alter the function CX learned by an autoencoding CNN on a given training set . The following two functions will play an important role in the subsequent analysis : ( 1 ) the identity function given by CX ( y ) = y for any y ∈ Rc×s×s and ( 2 ) the point map given by CX ( y ) = CX ( x0 ) for any y ∈ Rc×s×s , where x0 is a particular element in span CX ( X ) where span CX ( X ) = span { CX ( xi ) , 1 ≤ i ≤ n } . An extreme form of memorization occurs when a CNN learns the point map on a training set of size one , i.e. , it maps any image to the same fixed image . 3 DOWNSAMPLING : A MECHANISM FOR MEMORIZATION . In this section , we will illustrate how downsampling acts as a mechanism for learning the point map in deep CNNs . In particular , we will show that even if a downsampling network has the capacity to learn the identity function , it prefers to learn the point map . We consider the linear CNNs defined by Network ND ( non-downsampling ) and Network D ( downsampling ) in Figure 2a . Both networks employ 128 filters in each convolutional layer in all but the last layer , which contains 3 filters . The two networks only differ by the property that Network D uses filters of stride 8 in the first layer to downsample the image and then later uses nearest neighbor upsampling with scale factor 8 to rescale the image , while Network ND does not perform any downsampling . In concordance with the severe downsampling from 224×224 to 7×7 images performed by ResNet and VGG ( He et al . ( 2016 ) ; Simonyan & Zisserman ( 2015 ) ) , Network D downsamples from 32 × 32 to 4 × 4 images . However , we will show in Section 5 that the same memorization phenomenon is also observed in networks that only downsample using multiple convolutional layers of stride 2 as is done in U-Nets ( Ronneberger et al . ( 2015 ) ) . In the examples , the networks were initialized using the default in the PyTorch deep learning library ( Paszke et al. , 2017 ) , namely each layer ’ s weights were drawn i.i.d . from a uniform distribution U ( −a , a ) with a = 1/ √ P where P is the number of parameters in that layer . Both networks were trained for 2000 iteration on a single 3 × 32 × 32 image from CIFAR10 ( Krizhevsky , 2009 ) of a frog . Network ND reconstructed the digit 3 with a training loss of 10−4 and Network ND with loss 10−2 . We then applied the trained networks to test images consisting of both realistic images and random images with each pixel in the image being drawn from the standard normal distribution . As shown in Figure 2c and 2b , although trained on the same input for the same number of iterations Network D ( with downsamling ) learned the point map while Network ND ( no downsampling ) learned the identity function up to sign . The possible sign flip is a consequence of using no biases . In fact , this contrasting behavior is very robust throughout the training process . In Figure 3 , we see that regardless of when we stop training , even after only 10 iterations , Network D learns a point map , while Network ND learns a function visually similar to the identity map . One may conjecture that the memorization phenomenon that occurs using Network D could be due to the small number of filters in the downsampling layer , so that not all the features from the input could be preserved and the network would not be capable of learning a full rank identity map . This is not the case . To demonstrate that this is in general not the reason why downsampling networks learn a point map , we next present a downsampling network which , as we show , has the capacity to learn the identity function , but still prefers to learn the point map instead as the result of optimization . Example . Consider the linear CNN defined by the small downsampling Network DS presented in Figure 4a , which is trained on a single 1×2×2 image for 2000 iterations . Figure 4b shows that there exists an initialization such that the network learns the identity function . A full description of how we computed this manual initialization is provided in Appendix A . In contrast , as shown in Figure 4c initializing Network DS with the default PyTorch uniform distribution results in the point map . This provides the first example showing that even though the network has the capacity to learn the identity function , it prefers to learn a point map using the default PyTorch initialization . We observed the same effects using the Xavier uniform initialization , which is based on the assumption that activations are linear ( Glorot & Bengio ( 2010 ) ) . However , the results are not observed for linear networks when using Kaiming initialization , which is expected since this initialization is meant for nonlinear networks with ReLU/PReLU activations ( He et al . ( 2015 ) ) . In Appendix E , we show that nonlinear networks memorize training images under any of these initializations .

Summary: The paper tries to provide an explanation for a memorization phenomenon observed in convolutional autoencoders. In the case of memorization, the autoencoder always outputs the same fixed image for any input image, even when the input image is random noise. The authors provide an empirical analysis that connects such a phenomenon to strides in convolutional layers of the autoencoder. Then, a possible theoretical explanation is given in the form of conjecture with some empirical evidence.