Learning General and Reusable Features via Racecar-Training

1 INTRODUCTION . Humans spend surprising amounts of time assembling and disassembling objects . These deconstruction tasks serve a wide range of purposes from a hobby for motorists , to an important source of learning and exploration for children ( Gopnik et al. , 1999 ) . Motivated by this behavioral trait of humans , we propose a surprisingly simple , yet powerful modification of neural network training : in addition to a regular forward pass , we add a reverse pass that is constrained to reconstruct all in-between results of the forward pass as well as the input . As we will demonstrate below , this palindromic structure yields substantial improvements for generalization of the learned features in a wide range of architectures , and in many cases even improves the baseline achieved with regular training . Our results indicate that the reversible nature of the proposed training setup , which we will subsequently refer to via the palindrome ” racecar ” , encourages the formation of general and reusable features that benefit a wide range of learning tasks . With our approach we specifically target transfer learning applications . For a regular , i.e. , a nontransfer task , the goal usually is to train a network that gives the optimal performance for one specific goal , as has been demonstrated in many success stories over the years ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ; Goodfellow et al. , 2014 ; He et al. , 2016 ) . In such a case , the network naturally exploits any observed correlations between input and output distribution . E.g. , if the color of an object in any way correlates with its type , the training of a classifier should find and use this information . In recent years , even networks trained only for a very specific task were shown to be powerful starting points for training models with different tasks ( Zamir et al. , 2018 ; Gopalakrishnan et al. , 2017 ; Ding et al. , 2017 ) . In many cases , the original network contained features that were applicable to different data domains and beneficial for new inference tasks . An inherent difficulty in this setting is that typically no knowledge about the specifics of the new data and task domains is available at training time of the source model . While it is common practice to target broad and difficult tasks with the hope that this will yield learned features that are applicable in new domains , we instead specifically target improving the generalizing capabilities of learned features while training the source model . The core idea of our approach is to add a reverse path during training that is constrained to be as reversible as possible every step of the way . This makes the training task more difficult at first , but at the same time encourages the network to learn reversible and general features . Motivated by the disassembly and assembly processes of humans , and in contrast to previous work on invertible networks ( Gomez et al. , 2017 ; Jacobsen et al. , 2018 ; Zhang et al. , 2018 ) , we constrain the network to represent a as-reversible-as-possible process for all intermediate layer activations , instead of only perfectly reproducing the input . Thus , even for cases where a classifier can , e.g. , rely on color for inference of an object type , the model is encouraged to learn a representation that can recover the input in order to not only reconstruct the color of an object but also its shape . Hence the internal representation the network builds naturally has to encode as many aspects of the input data distribution as its representational capabilities permit , in order to recover the input . We demonstrate the benefits of our approach for a variety of architectures , from pure convolutional neural networks ( CNNs ) with and without batch normalization , to networks that include fully connected layers , as well as GAN architectures . 2 RELATED WORK . Transfer learning with deep neural networks has been very successful for a variety of tasks , such as image classification ( Duan et al. , 2012 ; Kulis et al. , 2011 ; Zhu et al. , 2011 ) , multi-language text classification ( Zhou et al. , 2014b ; Prettenhofer & Stein , 2010 ; Zhou et al. , 2014a ) , and medical imaging problems ( Ravishankar et al. , 2016 ) . A central question in transfer learning is whether base and target tasks are related or not . Zamir et al . ( 2018 ) proposed an approach to obtain task relationship graphs for different tasks . Another critical point is that we need the neural networks to learn general features from the data set , which are useful for both base and related tasks , rather than some specific features . But how to improve generalization of trained neural networks and reuse them for different related tasks is still a challenge . In this paper we propose a modified training approach for improving generalization via explicitly building a reverse pass network in addition to a regular forward pass . While Zhang et al . ( 2018 ) proposed reverse connected modules , they are primarily used to transfer information from deep layers to shallow layers for a given task . Recovering all input information from hidden representations of a network is generally very difficult ( Dinh et al. , 2016 ; Mahendran & Vedaldi , 2016 ) , due to the loss of information over the course of the layer transformations . In this context , Tishby & Zaslavsky ( 2015 ) proposed the information bottleneck principle , which states that for an optimal representation , information unrelated to the current task is reduced . This highlights the common specialization of regular training approaches . Ardizzone et al . ( 2018 ) , Jacobsen et al . ( 2018 ) and Gomez et al . ( 2017 ) also build reversed networks , but mainly focus on how to make a network fully invertible via introducing special structures . As a consequence , the path from input to output is actually different from the reverse path that translates output to input . In addition , the proposed structures of previous work can not be easily attached to other network architectures . In contrast , we show that it is not necessary to strive for perfect reversibility to obtain an improved performance . Our racecar training also fully preserves an architecture for the backward path , and does not require operations that are not part of the source network . As such it can easily be applied in new settings , such as for adversarial training ( Goodfellow et al. , 2014 ) . Aiming for related goals , Bansal et al . ( 2018 ) introduced orthogonality regularizations to the loss function . However , the proposed constraints are relatively weak , and make it difficult to arrive at invertible networks . Besides , the orthogonality regularization still focuses on improving performance of a known , given task . This means the training process only extracts features which the network considers useful for improving the performance of the current task . Hence , unlike our method , orthogonality does not necessarily improve generalization or improved transfer performance ( Torrey & Shavlik , 2010 ) . 3 METHOD . The goal of transfer learning is to reuse a basic model trained for task A for a related new task B . The performance for B naturally depends crucially on whether the content of the basic model yields benefits for the new task . In line with previous work ( Yosinski et al. , 2014 ) , we consider the generality of features learned on task A as the extent to which the features can be used for task B . This can be measured in terms of the difference between the transfer performance pAB and training a model for task B from scratch yielding performance pB . Our central goal is improving the transferability of the basic model w.r.t . the performance metric pAB − pB , which at the same time can be seen as a measure of the generalizing capabilities of the basic model . Regular training approaches typically construct a model with a certain network structure , and train the model weights for a given task via a suitable loss function . Our approach does not modify this initial structure , but adds a second pass that reverses the initial structure while reusing all weights and biases . E.g. , for a typical fully connected layer , the forward pass , the operation L2 = M×L1+b is changed to L ′ 1 = M T × ( L2 − b ) for the reverse pass , where L1 and L2 denote input and output , respectively . Here , M and b denote weight matrix and bias , while L ′ 1 denotes the input regenerated via the reverse pass . We will show and discuss L ′ for several examples below . Our goal with the reverse pass is to invert all operations of the forward pass to obtain identical intermediate activations between the layers with matching dimensionality . We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass . Due to the symmetric structure of the two passes , we can use a simple L2 difference to drive the network towards aligning the results : Lracecar = ∑n m=1 λm ∥∥∥Lm − L′m∥∥∥ 2 ( 1 ) Here Lm denotes the input of layer m in the forward pass and L ′ m the output of layer m for the reverse pass . λm denotes a scaling factor for the loss of layer m , which , however , is typically constant in our tests across all layers . An illustration of this process for a CNN structure is shown in Fig . 1 . While the construction of the reverse is straight-forward for all standard operations , i.e. , fully connected layers , convolutions , pooling etc. , batch normalization ( BN ) and activation function require slight adjustments to map Lm and L ′ m to the same range of values such that they can be compared in the loss . Hence , we use the BN parameters and the activation function of layer m − 1 from the forward pass for layer m in the reverse pass , Note that with our notation , L1 and L ′ 1 refer to the input I , and the regenerated input I ′ , respectively . In the following , we will refer to networks trained with the added reverse structure and the loss terms of equation 1 as racecar training . The constraints of equation 1 intentionally only minimize differences in an averaged manner with an L2 norm , as we don ’ t strive for a perfectly bijective mapping between input and output domains . Rather , our goal with the racecar training is to encourage the network to extract features that preserve as much information from the input data set as possible with the given representative capabilities of the chosen architecture . Hence , while a regular forward training allows a model to specialize its extracted features to maximize performance for a given task , our approach encourages the network to consider the full input data distribution , such that it ideally can be recovered from the latent-space representation of all layers . To differentiate variants , we will use the following naming scheme : StdA/B , RRsA/B , OrtA/B for trained base models . Here , Std denotes a regular training run ( always shown in yellow color in graphs below ) , while RRs denotes models trained with our racecar training ( in green below ) . Here , the s superscript denotes how many intermediate results of layers are used , e.g. , RR3 means n = 3 in equation 1 , which constrains the input data as well as the next two layers of the original structure . Correspondingly , a network trained with RR1 means the network is trained to regenerate the input without any constraints for the activations inside of the network . This represents a special , and somewhat sub-optimal case of our method , as we will demonstrate below . Ort additionally denotes models trained with orthogonal constraints ( Bansal et al. , 2018 ) ( in blue ) . The subscripts A/B denote the task A/B the model was trained for . We will call direct training for either task phase I in the following . In phase II , we reuse a model trained in phase I for new tasks ( typically B ) , with a regular training approach . I.e. , in phase II , all models are only trained with the forward structure and original loss , and do not use the racecar loss or other modifications . We will use the naming scheme StdAA/AB , RRsAA/AB , OrtAA/AB for trained models in phase II . Here AA/AB mean the model was trained for task A during phase I , and is then trained for task A/B in phase II . It is worth pointing out that the additional constraints of our racecar training lead to increased requirements for memory and additional computations during phase I , e.g. , it is 61.13 % slower per epoch for the MNIST tests . And while the constraints can also lead to a slight deterioration of performance , e.g. , a 0.17 % lower accuracy for MNIST tests during phase I , we will demonstrate that our models outperform baselines during phase II even for the original task , and thus in practice justify introducing the racecar training .

Summary: This paper proposes a scheme for training layered feedforward neural networks with backwards accumulation of gradients. In the proposed scheme, the intermediate activations during a forward pass are constrained, using an L2 norm penalty, to be close to the activations of a model that inverts the operations of each layer and transforms the data in reverese order (from output to input). The second, inverse, network shares all parameters with the feedforward network.