Reviving Autoencoder Pretraining

1 INTRODUCTION . While approaches such as greedy layer-wise autoencoder pretraining ( Bengio et al. , 2007 ; Vincent et al. , 2010 ; Erhan et al. , 2010 ) arguably paved the way for many fundamental concepts of today ’ s methodologies in deep learning , the pressing need for pretraining neural networks has been diminished in recent years . This was primarily caused by numerous advances in terms of regularization ( Srivastava et al. , 2014 ; Hanson & Pratt , 1989 ; Weigend et al. , 1991 ) , network architectures ( Ronneberger et al. , 2015 ; He et al. , 2016 ; Vaswani et al. , 2017 ) , and improved optimization algorithms ( Kingma & Ba , 2014 ; Loshchilov & Hutter , 2017 ; Reddi et al. , 2019 ) . Despite these advances , training deep neural networks that generalize well to a wide range of previously unseen tasks remains a fundamental challenge ( Neyshabur et al. , 2017 ; Kawaguchi et al. , 2017 ; Frankle & Carbin , 2018 ) . Inspired by techniques for orthogonalization ( Ozay & Okatani , 2016 ; Jia et al. , 2017 ; Bansal et al. , 2018 ) , we re-visit the classic idea of unsupervised autoencoder pretraining in the context of reversible network architectures . Hence , we propose a modified variant that relies on a full reverse pass trained in conjunction with a given training task . A key insight is that there is no need for ” greediness ” , i.e. , layer-wise decompositions of the network structure , and it is additionally beneficial to take into account a specific problem domain at the time of pretraining . We establish links between singular value decomposition ( SVD ) and pretraining , and show how our approach yields an embedding of problem-aware dominant features in the weight matrices . An SVD can then be leveraged to conveniently gain insights about learned structures . Most importantly , we demonstrate that the proposed pretraining yields an improved performance for a variety of learning and transfer tasks . Our formulation incurs only a very moderate computational cost , is very easy to integrate , and widely applicable . The structure of our networks is influenced by invertible network architectures that have received significant attention in recent years ( Gomez et al. , 2017 ; Jacobsen et al. , 2018 ; Zhang et al. , 2018a ) . However , instead of aiming for a bijective mapping that reproduces inputs , we strive for learning a general representation by constraining the network to represent an as-reversible-as-possible process for all intermediate layer activations . Thus , even for cases where a classifier can , e.g. , rely on color for inference of an object type , the model is encouraged to learn a representation that can recover the input . Hence , not only the color of the input should be retrieved , but also , e.g. , its shape . In contrast to most structures for invertible networks , our approach does not impose architectural restrictions . We demonstrate the benefits of our pretraining for a variety of architectures , from fully connected layers to convolutional neural networks ( CNNs ) , over networks with and without batch normalization , to GAN architectures . We discuss other existing approaches and relate them to the proposed method in the appendix . Below , we will first give an overview of our formulation and its connection to singular values , before evaluating our model in the context of transfer learning . For a regular , i.e. , a non-transfer task , the goal usually is to train a network that gives optimal performance for one specific goal . During a regular training run , the network naturally exploits any observed correlations between input and output distribution . An inherent difficulty in this setting is that typically no knowledge about the specifics of the new data and task domains is available when training the source model . Hence , it is common practice to target broad and difficult tasks hoping that this will result in features that are applicable in new domains ( Zamir et al. , 2018 ; Gopalakrishnan et al. , 2017 ; Ding et al. , 2017 ) . Motivated by autoencoder pretraining , we instead leverage a pretraining approach that takes into account the data distribution of the inputs . We demonstrate the gains in accuracy for original and new tasks below for a wide range of applications , from image classification to data-driven weather forecasting . 2 METHOD . With state-of-the-art methods , there is no need for breaking down the training process into single layers . Hence , we consider approaches that target whole networks , and especially orthogonalization regularizers as a starting point ( Huang et al. , 2018 ) . Orthogonality constraints were shown to yield improved training performance in various settings ( Bansal et al. , 2018 ) , and can be formulated as : Lort = n∑ m=1 ∥∥MTmMm − I∥∥2F , ( 1 ) i.e. , enforcing the transpose of the weight matrix Mm ∈ Rs out m×s in m for all layers m to yield its inverse when being multiplied with the original matrix . I denotes the identity matrix with I = ( e1m , ... e sinm m ) , ejm denoting the jth column unit vector . Minimizing equation 1 , i.e . M T mMm− I = 0 is mathematically equivalent to : MTmMme j m − ejm = 0 , j = 1 , 2 , ... , sinm , ( 2 ) with rank ( MTmMm ) = s in m , and e j m as eigenvectors of M T mMm with eigenvalues of 1 . This formulation highlights that equation 2 does not depend on the training data , and instead only targets the content of Mm . Inspired by the classical unsupervised pretraining , we re-formulate the orthogonality constraint in a data-driven manner to take into account the set of inputs Dm for the current layer ( either activation from a previous layer or the training data D1 ) , and instead minimize LRR = n∑ m=1 ( MTmMmd i m − dim ) 2 = n∑ m=1 ( ( MTmMm − I ) dim ) 2 , ( 3 ) where dim ∈ Dm ⊂ Rs in m . Due to its reversible nature , we will denote our approach with an RR subscript in the following . In contrast to classical autoencoder pretraining , we are minimizing this loss jointly for all layers of a network , and while orthogonality only focuses onMm , our formulation allows for minimizing the loss by extracting the dominant features of the input data . Let q denote the number of linearly independent entries in Dm , i.e . its dimension , and t the size of the training data , i.e . |Dm| = t , usually with q < t. For every single datum dim , i = 1 , 2 , ... , t , equation 3 results in MTmMmd i m − dim = 0 , ( 4 ) and hence dim are eigenvectors of M T mMm with corresponding eigenvalues being 1 . Thus , instead of the generic constraint MTmMm = I that is completely agnostic to the data at hand , the proposed formulation of equation 4 is aware of the training data , which improves the generality of the learned representation , as we will demonstrate in detail below . As by construction , rank ( Mm ) = r 6 min ( sinm , s out m ) , the SVD of Mm yields : Mm = UmΣmV T m , with { Um = ( u 1 m , u 2 m , ... , u r m , u r+1 m , ... , u soutm m ) ∈ Rs out m×s out m , Vm = ( v 1 m , v 2 m , ... , v r m , v r+1 m , ... , v sinm m ) ∈ Rs in m×s in m , ( 5 ) with left and right singular vectors in Um and Vm , respectively , and Σm having square roots of the r eigenvalues of MTmMm on its diagonal . u k m and v k m ( k = 1 , ... , r ) are the eigenvectors of MmM T m and MTmMm , respectively ( Wall et al. , 2003 ) . Here , especially the right singular vectors in V T m are important , as they determine which structures of the input are processed by the transformation Mm . The original orthogonality constraint with equation 2 yields r unit vectors ejm as the eigenvectors of MTmMm . Hence , the influence of equation 2 on Vm is completely independent of training data and learning objectives . Next , we show that LRR facilitates learning dominant features from a given data set . For this , we consider an arbitrary basis for spanning the space of inputsDm for layerm . Let Bm : 〈 w1m , ... , w q m 〉 denote a set of q orthonormal basis vectors obtained via a Gram-Schmidt process , with t > q > r , and Dm denoting the matrix of the vectors in Bm . As we show in more detail in the appendix , our constraint from equation 4 requires eigenvectors of MTmMm to be w i m , with Vm containing r orthogonal vectors ( v1m , v 2 m , ... , v r m ) from Dm and ( sinm − r ) vectors from the null space of M . We are especially interested in how Mm changes w.r.t . input in terms of Dm , i.e. , we express LRR in terms of Dm . By construction , each input dim can be represented as a linear combination via a vector of coefficients cim that multiplies Dm so that d i m =Dmc i m. Since Mmdm = UmΣmV T mdm , the loss LRR of layer m can be rewritten as LRRm = ( MTmMmdm − dm ) 2 = ( VmΣTmΣmV Tmdm − dm ) 2 = ( VmΣ T mΣmV T mDmcm −Dmcm ) 2 , ( 6 ) where we can assume that the coefficient vector cm is accumulated over the training data set size t via cm = ∑t i=1 c i m , since eventually every single datum inDm will contribute to LRRm . The central component of equation 6 is V TmDm . For a successful minimization , Vm needs to retain those w i m with the largest cm coefficients . As Vm is typically severely limited in terms of its representational capabilities by the number of adjustable weights in a network , it needs to focus on the most important eigenvectors in terms of cm in order to establish a small distance to Dmcm . Thus , features that appear multiple times in the input data with a corresponding factor in cm will more strongly contribute to minimizing LRRm . To summarize , Vm is driven towards containing r orthogonal vectors wim that represent the most frequent features of the input data , i.e. , the dominant features . Additionally , due to the column vectors of Vm being mutually orthogonal , Mm is encouraged to extract different features from the input . By the sake of being distinct and representative for the data set , these features have the potential to be useful for new inference tasks . The feature vectors embedded inMm can be extracted from the network weights in practical settings , as we will demonstrate below . Realization in Neural Networks Calculating MTmMm is usually very expensive due to the dimensionality of Mm . Instead of building it explicitly , we constrain intermediate results to realize equation 3 when training . A regular training typically starts with a chosen network structure and Under review as a conference paper at ICLR 2021 trains the model weights for a given task via a suitable loss function . Our approach fully retains this setup and adds a second pass that reverses the initial structure while reusing all weights and biases . E.g. , for a typical fully connected layer in the forward pass with dm+1 = Mmdm +bm , the reverse pass operation is given by d ′ m = M T m ( dm+1 − bm ) , where d ′ m denotes the reconstructed input . a m o un t o f s h a re d in fo rm a tio n b et w e e n la ye r a n d amount of shared information between layer ℒ and 𝑋 𝐿𝑎𝑦𝑒𝑟 1 2 3 𝑚 𝑛 𝐿𝑎𝑦𝑒𝑟 𝑚 𝒅𝒎 𝒅𝒎 𝟏 𝒅𝒎 𝟏𝒅𝒎 𝒅𝟏 𝒅𝟏 | 𝑓 𝐵𝑁 𝑐𝑜𝑛𝑣 𝑓 ( 𝑀 , 𝒃𝒎 ) 𝑑𝑒𝑐𝑜𝑛𝑣 ( 𝑀 , −𝒃𝒎 ) 𝐵𝑁 𝑤𝑒𝑖𝑔ℎ𝑡 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝒗 𝒗 𝒗 0.427 0.400 0.489 0.420 0.394 0.445 0.213 0.216 0.153 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 𝒗 𝒗 𝒅 … … ( 50 images ) ( 50 images ) ( 1,0 ) ( 0,1 ) 𝒅 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 𝑎 𝑀𝑁𝐼𝑆𝑇 𝑇𝑒𝑠𝑡 𝑤𝑖𝑡ℎ 𝑙𝑖𝑛𝑒𝑎𝑟 𝑚𝑜𝑑𝑒𝑙 𝑏 𝑃𝑒𝑎𝑘 𝑇𝑒𝑠𝑡 𝑤𝑖𝑡ℎ 𝐵𝑁 𝑎𝑛𝑑 𝑅𝑒𝐿𝑈 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 ( 0,0,0,0,0,0,0,1,0,0 ) ( 0,0,1,0,0,0,0,0,0,0 ) 0.499 0.502 0.499 0.491 0.233 0.202 𝒅 𝒅 𝑆𝑡𝑑 𝑂𝑟𝑡 𝑅𝑅 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 0 2 4 6 8 10 12 more information about both 𝑋 & 𝑌 more information about 𝑌 only 𝐼 ( 𝑋 ; 𝒟 ) 𝐼 ( 𝒟 ; 𝑌 ) less shared information between layer ℒ and 𝑋 le s s sh ar ed in fo rm at io n b et w ee n la ye r ℒ a n d 𝑌 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 0 2 4 6 8 10 12 𝑂𝑟𝑡 𝑅𝑅 𝑅𝑅 𝑆𝑡𝑑 𝐼 ( 𝑋 ; 𝒟 ) 𝐼 ( 𝒟 ; 𝑌 ) 0.9 0.92 0.94 0.96 0.98 1 1.02 0 2 4 6 8 10 12 𝑂𝑟𝑡 𝑅𝑅 𝑅𝑅 𝑆𝑡𝑑 𝐼 ( 𝑋 ; 𝒟 ) 𝐼 ( 𝒟 ; 𝑌 ) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 0 2 4 6 8 10 12 𝑂𝑟𝑡 𝑅𝑅 𝑅𝑅 𝑆𝑡𝑑 𝐼 ( 𝑋 ; 𝒟 ) 𝒟 𝒟 𝒟 𝒟 𝒟 𝒟 𝑆𝑡𝑑 𝐼 ( 𝒟 ; 𝑌 ) more information about 𝑋 only ( a ) Mutual Information Plane , How to Read ( b ) Mutual Information for Task A ( c ) After fine-tuning for A ( d ) After fine-tuning for B no information … … … … 𝒅𝟐 𝒅𝟐 | 𝒅𝟑 𝒅𝟑 | 𝒅𝒏 𝒅𝒏 | 𝒐𝒖𝒕𝒑𝒖𝒕 Layers of 𝑅𝑅 models exhibit strong MI with in- & output 𝒗 𝒗 0.43 0.49 0.42 0.45 0.21 0.22 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 𝒗 𝒗 𝒅 ( 1,0 ) ( 0,1 ) 𝒅 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 𝑎 𝐿𝑖𝑛𝑒𝑎𝑟 𝑚𝑜𝑑𝑒𝑙 𝑏 𝑊𝑖𝑡ℎ 𝐵𝑁 𝑎𝑛𝑑 𝑅𝑒𝐿𝑈 7 2 0.50 0.50 0.50 0.49 0.23 0.20 𝒅 𝒅 𝑆𝑡𝑑 𝑂𝑟𝑡 𝑅𝑅 …… ( 50 imgs ) …… ( 50 imgs ) Our goal with the reverse pass is to transpose all operations of the forward pass to obtain identical intermediate activations between the layers with matching dimensionality . We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass , as illustrated in figure 2 . Unlike greedy layer-wise autoencoder pretraining , which trains each layer separately and only constrains d1 and d ′ 1 , we jointly train all layers and constrain all intermediate results . Due to the symmetric structure of the two passes , we can use a simple L2 difference to drive the network towards aligning the results : LRR = n∑ m=1 λm ∥∥∥dm − d′m∥∥∥2 F . ( 7 ) Here dm denotes the input of layer m in the forward pass and d ′ m the output of layer m for the reverse pass . λm denotes a scaling factor for the loss of layer m , which , however , is typically constant in our tests across all layers . Note that with our notation , d1 and d ′ 1 refer to the input data , and the reconstructed input , respectively . Next , we show how this setup realizes the regularization from equation 3 . For clarity , we use a fully connected layer with bias . In a neural network with n hidden layers , the forward process for a layer m is given by dm+1 = Mmdm +bm , , with d1 and dn+1 denoting in- and output , respectively . For our pretraining , we build a reverse pass network with transposed operations starting with the final output where dn+1 = d ′ n+1 , and the intermediate results d ′ m+1 : d ′ m = M T m ( d ′ m+1 − bm ) , ( 8 ) which yields ∥∥∥dm − d′m∥∥∥2 F = ∥∥MTmMmdm − dm∥∥2F . When this difference is minimized via equation 7 , we obtain activated intermediate content during the reverse pass that reconstructs the values computed in the forward pass , i.e . d ′ m+1 = dm+1 holds . As in equation 10 the reverse pass activation d ′ m depends on dm+1 ′ , this formulation yields a full reverse pass from output to input , which we use for most training runs below . In this case d ′ m = M T m ( d ′ m+1 − bm ) = MTm ( dm+1 − bm ) = MTmMmdm , ( 9 ) which is consistent with equation 3 , and satisfies the original constraint MTmMmdm − dm = 0 . This version is preferable if a unique path from output to input exists . For architectures where the path is not unique , e.g. , in the presence of additive residual connections , we use a local formulation d ′ m = M T m ( dm+1 − bm ) , ( 10 ) which employs dm+1 for jointly constraining all intermediate activations in the reverse pass . Up to now , the discussion focused on simplified neural networks without activation functions or extensions such as batch normalization ( BN ) . While we leave incorporating such extensions for future work , our experiments consistently show that the inherent properties of our pretraining remain valid : even with activations and BN , our approach successfully extracts dominant structures and yields improved generalization . In the appendix , we give details on how to ensure that the latent space content for forward and reverse pass is aligned such that differences can be minimized . To summarize , we realize the loss formulation of equation 7 to minimize ∑n m=1 ( ( M T mMm−I ) dm ) 2 without explicitly having to construct MTmMm . Following the notation above , we will refer to networks trained with the added reverse structure and the additional loss terms as RR variants . We consider two variants for the reverse pass : a local pretraining equation 10 using the datum dm+1 of a given layer , and a full version via equation 8 which uses d ′ m+1 incoming from the next layer during the reverse pass . Embedding Singular Values Below , Std denotes a regular training run ( in orange color in graphs below ) , while RR denotes our models ( in green ) . Pre and Ort will denote regular autoencoder pretraining and orthogonality , respectively , while a subscript will denote the task variant the model was trained for , e.g. , StdT for task T. While we typically use all layers of a network in the constraints , a reduced variant that we compare to below only applies the constraint for the input data , i.e. , m=1 . A network trained with this variant , denoted by RR1A , is effectively trained to only reconstruct the input . It contains no constraints for the inner activations and layers of the network . For the Ort models , we use the Spectral Restricted Isometry Property algorithm ( Bansal et al. , 2018 ) . We verify that the column vectors of Vm of models from RR training contain the dominant features of the input with the help of a classification test , employing a single fully connected layer , i.e . d2 = M1d1 , with batch normalization and activation . To quantify this similarity , we compute an LPIPS distance ( Zhang et al. , 2018b ) between vim and the training data ( lower values being better ) . 𝒗 𝒗 0.43 0.49 0.42 0.45 0.21 0.22 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 𝑎 𝐿𝑖𝑛𝑒𝑎𝑟 𝑚𝑜𝑑𝑒𝑙 7 2 𝒅 𝒅 𝐋𝐏𝐈𝐏𝐒𝐏𝐫𝐞 0.14 0.31 𝒗 𝒗 1,0 𝑎𝑛𝑑 ( 0,1 ) 𝐋𝐏𝐈𝐏𝐒𝐒𝐭𝐝 𝐋𝐏𝐈𝐏𝐒𝐎𝐫𝐭 𝐋𝐏𝐈𝐏𝐒𝑹𝑹 0.50 0.50 0.50 0.49 0.23 0.20 𝑆𝑡𝑑 𝑂𝑟𝑡 𝑅𝑅 …… …… 𝑃𝑟𝑒 𝐋𝐏𝐈𝐏𝐒𝐏𝐫𝐞 0.24 0.40 + We employ a training data set constructed from two dominant classes ( a peak in the top left , and bottom right quadrant , respectively ) , augmented with noise in the form of random scribbles . Based on the analysis above , we expect the RR training to extract the two dominant peaks during training . The LPIPS measurements confirm our SVD argumentation above , with average scores of 0.217±0.022 for RR , 0.319±0.114 for Pre , 0.495± 0.006 for Ort , and 0.500± 0.002 for Std . I.e. , the RR model fares significantly better than the others . At the same time , the peaks are clearly visible for RR models , an example is shown in figure 3 ( b ) , while the other models fail to extract structures that resemble the input . Thus , by training with the full network and the original training objective , our pretraining yields structures that are interpretable and be inspected by humans . The results above experimentally confirm our formulation of the RR loss and its ability to extract dominant and generalizing structures from the training data . Next , we will focus on quantified metrics and turn to measurements in terms of mutual information to illustrate the behavior of our pretraining for deeper networks .

Summary: This paper proposes to use orthogonal weight constraints for autoencoders. The authors demonstrate that under orthogonal weights (hence invertible), more features could be extracted. The theory is conducted under linear cases while the authors claim it can be applied to more complicated scenarios such as higher dimension and with nonlinearity. The experiments demonstrate the performance of proposed model on classification tasks and generative tasks. Several baselines are compared.