Title: LEARNING GENERAL AND REUSABLE FEATURES VIA RACECAR-TRAINING

Abstract: We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.e. racecar training, leads to significantly more general features to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outperforms baseline models trained in a regular fashion. This improved performance is visible for a wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved performance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.

1. INTRODUCTION:

Humans spend surprising amounts of time assembling and disassembling objects. These deconstruction tasks serve a wide range of purposes from a hobby for motorists, to an important source of learning and exploration for children (Gopnik et al., 1999) . Motivated by this behavioral trait of humans, we propose a surprisingly simple, yet powerful modification of neural network training: in addition to a regular forward pass, we add a reverse pass that is constrained to reconstruct all in-between results of the forward pass as well as the input. As we will demonstrate below, this palindromic structure yields substantial improvements for generalization of the learned features in a wide range of architectures, and in many cases even improves the baseline achieved with regular training. Our results indicate that the reversible nature of the proposed training setup, which we will subsequently refer to via the palindrome "racecar", encourages the formation of general and reusable features that benefit a wide range of learning tasks.

With our approach we specifically target transfer learning applications. For a regular, i.e., a nontransfer task, the goal usually is to train a network that gives the optimal performance for one specific goal, as has been demonstrated in many success stories over the years (LeCun et al., 1998; Krizhevsky et al., 2012; Goodfellow et al., 2014; He et al., 2016) . In such a case, the network naturally exploits any observed correlations between input and output distribution. E.g., if the color of an object in any way correlates with its type, the training of a classifier should find and use this information. In recent years, even networks trained only for a very specific task were shown to be powerful starting points for training models with different tasks (Zamir et al., 2018; Gopalakrishnan et al., 2017; Ding et al., 2017) . In many cases, the original network contained features that were applicable to different data domains and beneficial for new inference tasks. An inherent difficulty in this setting is that typically no knowledge about the specifics of the new data and task domains is available at training time of the source model. While it is common practice to target broad and difficult tasks with the hope that this will yield learned features that are applicable in new domains, we instead specifically target improving the generalizing capabilities of learned features while training the source model.

The core idea of our approach is to add a reverse path during training that is constrained to be as reversible as possible every step of the way. This makes the training task more difficult at first, but at the same time encourages the network to learn reversible and general features. Motivated by the disassembly and assembly processes of humans, and in contrast to previous work on invertible networks (Gomez et al., 2017; Jacobsen et al., 2018; Zhang et al., 2018) , we constrain the network to represent a as-reversible-as-possible process for all intermediate layer activations, instead of only perfectly reproducing the input. Thus, even for cases where a classifier can, e.g., rely on color for inference of an object type, the model is encouraged to learn a representation that can recover the input in order to not only reconstruct the color of an object but also its shape. Hence the internal representation the network builds naturally has to encode as many aspects of the input data distribution as its representational capabilities permit, in order to recover the input. We demonstrate the benefits of our approach for a variety of architectures, from pure convolutional neural networks (CNNs) with and without batch normalization, to networks that include fully connected layers, as well as GAN architectures.

2. RELATED WORK:

Transfer learning with deep neural networks has been very successful for a variety of tasks, such as image classification (Duan et al., 2012; Kulis et al., 2011; Zhu et al., 2011) , multi-language text classification (Zhou et al., 2014b; Prettenhofer & Stein, 2010; Zhou et al., 2014a) , and medical imaging problems (Ravishankar et al., 2016) . A central question in transfer learning is whether base and target tasks are related or not. Zamir et al. (2018) proposed an approach to obtain task relationship graphs for different tasks. Another critical point is that we need the neural networks to learn general features from the data set, which are useful for both base and related tasks, rather than some specific features. But how to improve generalization of trained neural networks and reuse them for different related tasks is still a challenge.

In this paper we propose a modified training approach for improving generalization via explicitly building a reverse pass network in addition to a regular forward pass. While Zhang et al. (2018) proposed reverse connected modules, they are primarily used to transfer information from deep layers to shallow layers for a given task. Recovering all input information from hidden representations of a network is generally very difficult (Dinh et al., 2016; Mahendran & Vedaldi, 2016) , due to the loss of information over the course of the layer transformations. In this context, Tishby & Zaslavsky (2015) proposed the information bottleneck principle, which states that for an optimal representation, information unrelated to the current task is reduced. This highlights the common specialization of regular training approaches. Ardizzone et al. (2018) , Jacobsen et al. (2018) and Gomez et al. (2017) also build reversed networks, but mainly focus on how to make a network fully invertible via introducing special structures. As a consequence, the path from input to output is actually different from the reverse path that translates output to input. In addition, the proposed structures of previous work can not be easily attached to other network architectures. In contrast, we show that it is not necessary to strive for perfect reversibility to obtain an improved performance. Our racecar training also fully preserves an architecture for the backward path, and does not require operations that are not part of the source network. As such it can easily be applied in new settings, such as for adversarial training (Goodfellow et al., 2014) .

Aiming for related goals, Bansal et al. (2018) introduced orthogonality regularizations to the loss function. However, the proposed constraints are relatively weak, and make it difficult to arrive at invertible networks. Besides, the orthogonality regularization still focuses on improving performance of a known, given task. This means the training process only extracts features which the network considers useful for improving the performance of the current task. Hence, unlike our method, orthogonality does not necessarily improve generalization or improved transfer performance (Torrey & Shavlik, 2010) .

3. METHOD:

The goal of transfer learning is to reuse a basic model trained for task A for a related new task B. The performance for B naturally depends crucially on whether the content of the basic model yields benefits for the new task. In line with previous work (Yosinski et al., 2014) , we consider the generality of features learned on task A as the extent to which the features can be used for task B. This can be measured in terms of the difference between the transfer performance p AB and training a model for task B from scratch yielding performance p B . Our central goal is improving the transferability of the basic model w.r.t. the performance metric p AB -p B , which at the same time can be seen as a measure of the generalizing capabilities of the basic model.

Regular training approaches typically construct a model with a certain network structure, and train the model weights for a given task via a suitable loss function. Our approach does not modify this initial structure, but adds a second pass that reverses the initial structure while reusing all weights and biases. E.g., for a typical fully connected layer, the forward pass, the operation L 2 = M ×L 1 +b is changed to L 1 = M T × (L 2 -b) for the reverse pass, where L 1 and L 2 denote input and output, respectively. Here, M and b denote weight matrix and bias, while L 1 denotes the input regenerated via the reverse pass. We will show and discuss L for several examples below.

Our goal with the reverse pass is to invert all operations of the forward pass to obtain identical intermediate activations between the layers with matching dimensionality. We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass. Due to the symmetric structure of the two passes, we can use a simple L 2 difference to drive the network towards aligning the results:

L racecar = n m=1 λ m L m -L m 2 (1)

Here L m denotes the input of layer m in the forward pass and L m the output of layer m for the reverse pass. λ m denotes a scaling factor for the loss of layer m, which, however, is typically constant in our tests across all layers. An illustration of this process for a CNN structure is shown in Fig. 1 . While the construction of the reverse is straight-forward for all standard operations, i.e., fully connected layers, convolutions, pooling etc., batch normalization (BN) and activation function require slight adjustments to map L m and L m to the same range of values such that they can be compared in the loss. Hence, we use the BN parameters and the activation function of layer m -1 from the forward pass for layer m in the reverse pass, Note that with our notation, L 1 and L 1 refer to the input I, and the regenerated input I , respectively. In the following, we will refer to networks trained with the added reverse structure and the loss terms of equation 1 as racecar training.

The constraints of equation 1 intentionally only minimize differences in an averaged manner with an L 2 norm, as we don't strive for a perfectly bijective mapping between input and output domains. Rather, our goal with the racecar training is to encourage the network to extract features that preserve as much information from the input data set as possible with the given representative capabilities of the chosen architecture. Hence, while a regular forward training allows a model to specialize its extracted features to maximize performance for a given task, our approach encourages the network to consider the full input data distribution, such that it ideally can be recovered from the latent-space representation of all layers.

To differentiate variants, we will use the following naming scheme: Std A/B , RR s A/B , Ort A/B for trained base models. Here, Std denotes a regular training run (always shown in yellow color in graphs below), while RR s denotes models trained with our racecar training (in green below). Here, the s superscript denotes how many intermediate results of layers are used, e.g., RR 3 means n = 3 in equation 1, which constrains the input data as well as the next two layers of the original structure. Correspondingly, a network trained with RR 1 means the network is trained to regenerate the input without any constraints for the activations inside of the network. This represents a special, and somewhat sub-optimal case of our method, as we will demonstrate below. Ort additionally denotes models trained with orthogonal constraints (Bansal et al., 2018) (in blue). The subscripts A/B denote the task A/B the model was trained for. We will call direct training for either task phase I in the following.

In phase II, we reuse a model trained in phase I for new tasks (typically B), with a regular training approach. I.e., in phase II, all models are only trained with the forward structure and original loss, and do not use the racecar loss or other modifications. We will use the naming scheme Std AA/AB , RR s AA/AB , Ort AA/AB for trained models in phase II. Here AA/AB mean the model was trained for task A during phase I, and is then trained for task A/B in phase II.

It is worth pointing out that the additional constraints of our racecar training lead to increased requirements for memory and additional computations during phase I, e.g., it is 61.13% slower per epoch for the MNIST tests. And while the constraints can also lead to a slight deterioration of performance, e.g., a 0.17% lower accuracy for MNIST tests during phase I, we will demonstrate that our models outperform baselines during phase II even for the original task, and thus in practice justify introducing the racecar training.

4. EVALUATION IN TERMS OF MUTUAL INFORMATION:

We now evaluate our approach in terms of the mutual information (MI) between input and output data distributions.

As our approach hinges on the introduction of the reverse pass, we will show that our approach succeeds in terms of establishing mutual information between the input and the constrained intermediates inside a network.

More formally, the mutual information I(X; Y ) of random variables X and Y measures how different the joint distribution of X and Y is w.r.t.

the product of their marginal distributions, i.e., the Kullback-Leibler divergence: A with a full racecar loss. For comparison, we also show the MI planes for Std A (a regularly trained model), and Ort A (trained with orthogonal constraints (Bansal et al., 2018) ). We additionally include a version RR 1 A , i.e. trained with only one racecar loss term λ 1 |L 1 -L 1 | 2 ), which means that only the input is constrained to be recovered. Thus, RR 1

I(X; Y ) = D KL [P (X,Y ) ||P X P Y ].

A represents a sim-plified version of our approach which receives no constraints that the intermediate results of forward and backward pass should match. The resulting information planes are shown in Fig. 2 . We can see that the end points of RR 6 A , i.e., the yellow clusters of points for the final state of the model, are located in the middle part of the graph. This means that all layers successfully encode information about the inputs as well as the outputs. In contrast, for Std A and Ort A , I(X; L) values of later layers are very low (points near the top left), while I(L; Y ) is high throughout. This indicates that the outputs were successfully encoded, and that increasing amounts of information about the inputs are discarded by Std A and Ort A . Hence, both models are highly specialized for the given task. Comparing RR 1

A with RR 6 A , the yellow end points of the latter are located in a centralized region, which indicates that every layer contains similar information about X and Y . It also indicates that the path from input to output is similar to the path from output to input. The end points of RR 1 A , on the other hand, are located in a scattered region. I.e., this network has different amounts of mutual information across its layers, and potentially a very different path in each direction. RR 1

A is only constrained to be able to regenerate its input, while the full racecar loss for RR 6

A ensures that the network learns features which are beneficial for both directions. This test highlights the importance of the constraints throughout the depth of a network in our racecar loss. AA slightly outperforms the other variants for the original task. We will more clearly demonstrate this positive trait of our racecar training with more complex tasks below. We now analyze a new task, to check whether the model learned specific or general features, and reverse output labels as transfer learning task B. E.g., if the output is [0,1] in the original data set, we invert it to [1, 0] . Based on RR 1 A , RR 6 A , Ort A and Std A , we train RR 1 AB , RR 6 AB , Ort AB and Std AB for the modified data set. For comparison, we also train a model trained Std B from scratch. MI planes and accuracy comparisons are shown in Fig. 4 . While the range in performance is relatively small across most models, Std AB stands out with a very low accuracy. This model from a regular training run has large difficulties to adapt to the new task. Model Ort AB also performs worse than Std B . RR 6

AB shows the best performance in this setting, confirming the previous MI plane visualizations, and demonstrating that our loss formulation helped to learn more general features from the input data, improving the performance for related tasks such as the inverted outputs.

5. EXPERIMENTAL RESULTS:

We now turn to more complex network structures (CNNs, GANs, Auto-Encoders), with different data sets (MNIST, Cifar, smoke, ImageNet) and different tasks (such as classification and synthesis) to show that models trained with our approach succeed in learning very general features that transfer to new tasks. The MNIST data set is a commonly used data set for hand written digit classification (LeCun et al., 1998) . At first, we train three models for regular MNIST data set classification as task A, one with racecar loss RR 3

5.1. DIGIT CLASSIFICATION:

A , a regular model Std A , and one with orthogonal Ort A constraints (Bansal et al., 2018) , as the latter also aims for improving CNN performance. Usually, convolutional layers take up most of the model parameters, so we correspondingly compute the racecar loss for the convolutional layers, omitting the fully connected layers that would be required for class label inference. To highlight the properties of our algorithm, we show comparisons between I and the regenerated I in Fig. 5 . We can see that for racecar training, most of the features from the input are recovered. Trying to invert the network in the same way for a regular training run or training with orthogonal constraints largely fails, as the extracted features are extracted according to the digit classification and discard information unrelated to this task. Based on Ort A , Std A , RR 3 A , we continue training to obtain models Ort AA , Std AA , and RR 3 AA . Results are shown in Fig. 6 . We repeated these runs 5 times, in order to draw reliable conclusions. We can see that RR 3 AA outperforms Ort AA and Std AA at the end of training, which indicates that racecar training yields general features that can also improve performance for original task. Council (2000) illustrated that balanced learning of both general and specific features is more effective for human learning, and our results here are consistent with this intuition.

As general features are more robust than specific features (Novak et al., 2018) , we investigate a perturbed data set for the transfer task B. We apply Ort A , Std A and RR 3 A to n-MNIST, a data set for classification with motion blur (Basu et al., 2017) . Performance results are likewise given in Fig. 6 . Based on the same CNN structure and parameters, RR 3 A achieves the best performance. This indicates that RR 3

A learned more general features via racecar training than Ort A and Std A . Natural images arise in many important application scenarios. Hence, we evaluate our approach with the Cifar data set (Krizhevsky et al., 2009) . At first, we train two models as Cifar-10 data set classification task A, RR 13 A and Std A . We again continue training to obtain RR 13 AA and Std AA , results for which are shown in Fig. 7 . The racecar training also improves performance for this natural image classification task. For transfer learning task B, we reuse RR 13 A and Std A for Cifar-100 data set classification (RR 13 AB and Std AB ). Results are also shown in Fig. 7 . Training with the same CNN structure and parameters, Std AB has difficulties adjusting to the new task, while our model from the initial racecar training slightly outperforms the model trained for scratch for B.

5.2. NATURAL IMAGE CLASSIFICATION:



5.3. GENERATIVE ADVERSARIAL MODELS:

In the previous sections, we employed commonly used data sets and transfers to related tasks. Next, we test our approach for a transfer situation with a more challenging transfer from a synthetic single channel data set of synthetic smoke simulations, to RBG videos of real smoke clouds. We use GAN and auto-encoder structures for the following models. A can also recover low resolution versions from high resolution data. Std A fails in regenerating the low resolution images. This shows that RR 6

A not only focused on generating good features for the super-resolution task, but also preserves information about the input.

We reuse the generator models RR 6:

A and Std A to train two sets of auto-encoder networks: once for the synthetic smoke data as transfer task B 1 , and as a second task B 2 for RBG videos of real-world smoke clouds (example frames are given in the appendix). This yields models RR 6 AB1,2 and Std AB1,2 , respectively. The resulting accuracies, summarized in Fig. 9 , show that racecar training performs best for both auto-encoding tasks. This is especially encouraging for task B 2 , as it represents a transfer from fully synthetic training to real-world images. Synthetic data and real-world images have significant differences in terms of their features. In the worst case, most of the features learned by the regular learning process Std A can not be transferred to the new task. Besides, an inappropriate starting point of the transfer learning task can guide the whole training process towards a wrong direction. Instead, a model with more general features such as RR 6

A can more easily be reused in new tasks, and can provide a better starting point for learning. This indicates the potential of racecar training to obtain generalizable features from synthetic data sets that can be used for tasks working with real-world data. In order to provide a qualitative evaluation in a complex, visual scenario, we turn to the popular task of image stylization. We use V GG19 networks (Simonyan & Zisserman, 2014) in the following, and we train two V GG19 networks RR 16 A and Std A with ImageNet data set (Deng et al., 2009) . Based on these, we train RR 16 AA and Std AA , top5 accuracy of which are 77.59% and 75.28%, respectively. Consistent with previous test, RR 16 AA outperforms the Std AA baseline even in this complex case for a network with ca. 143 million weights. Gatys et al. (2016) achieve excellent stylization results, but the results strongly depend on the pre-trained VGG network, i.e., it stylizes and changes structures that are represented by the features in the VGG network. Hence, we employ stylization to visualize which features, general or specialized, the two VGG versions focus on. Gatys et al. (2016) generate stylized results via optimizing two losses, L style and L content , which measure style difference and content difference, respectively (details are given in the appendix.

To compare the feature extracting capabilities between the RR 16 As a first test, we use two simple shapes as source and style images, as shown in Fig. 11 . They have the same background and object color. The only difference is the large-scale shape of the object. Thus, as we aim for applying an style that is identical to the source, we can test whether the features can cleanly separate and preserve the large scale shape features from the ones for the localized style. Comparing the results of RR 16

A and Std A , the output of RR 16

A is almost identical to the input, while stylization with Std A changes the shape of the object introducing undesirable streaks around the outline. This result indicates that RR 16 A is able to cleanly encode the shape of the object and preserve it during the stylization.

We show more stylization tests in Fig. 10 , which optimize both L content and L style . For the left example of Fig. 10 with a Van Gogh style image, both models can generate good stylized results. However, from the results obtained with Std A (orange square area), we can see a star within the building, which indicates that this model mixed style and context information in its features. Across our tests we found that models like this one, trained for more specific features, more often performed simple patch-based matches rather than generating meaningful transfers of styles. However, if the model focuses on more general and abstract features, it can perform a better semantic matching and generate improved stylization results. For the right part of Fig. 10 , we show a test transferring a horse image to a zebra style. RR 16 A focuses more on general features, such as grass, zebra, and their positional relationships. As a result the model successfully distinguishes foreground from background, and changes the style for both foreground and background objects with only smaller mistakes. Std A 's performance is significantly worse, leading to large grass-like areas within the body of the horse. We confirm the capabilities of the VGG model from racecar training with a low-to high-resolution transfer where the goal is purely to add detail. I.e., we can compute in place errors w.r.t. high-resolution reference. The result is shown in Fig. 12 . The VGG model obtained with a regular training run yields significantly higher errors, as visible on the right side of the figure . RR 16 A can extract more general information from the input, which is helpful with generating semantically meaningful results. However, we found that this property can also confuse the model in some specific situations. For instance, as shown in Fig. 13 , at first, RR 16 A can distinguish foreground and background and generate better stylization results than Std A . However, RR 16 A also stylizes the shadow of the horse. This also indicates that RR 16

A focuses more on general features. As a consequence, all horse shape objects are extracted and then stylized with zebra textures, such as the shadow in Fig. 13 . However, we generally think that this can provide potentially helpful information in other situations, e.g., for classification tasks. Overall, the quality of the racecar transfer also still surpasses the regular version.

A APPENDIX Here we will present more details, such as data set information, network structures and training parameters, for all of our mentioned tests above: Mutual Information Sec. A.1, MNIST classification Sec. A.2, Cifar Sec. A.3, smoke Sec. A.4 and VGG Sec. A.5. We will use C(k, l, s), D(k, l, s) to represent convolutional and deconvolutional operations, respectively, and fully connected layers are noted with F (l), where k, l, s denote kernel size, output channels and stride size, respectively. Bias of CNN layer is denoted with b. I/O(z) denote input/output and their dimensionality is given by z. I r denotes the input of reverse pass network. tanh, relu, lrelu denote corresponding activation functions, where we typically use a leaky tangent of 0.2 for the negative half space. U P , M P and BN denote 2× nearest-neighbor up sampling, max pooling with 2 × 2 filters and stride 2, and batch normalization, respectively. All performance numbers were measured on a Nvidia GeForce GTX 1080 Ti GPUs and Intel Core i7-6850K CPUs.

A.1 MUTUAL INFORMATION TEST:

Below we will introduce more details about tests in Sec. 4.For the numerical task (Shwartz-Ziv & Tishby, 2017) , input variable X are 12 binary digits that represent 12 uniformly distributed points on a 2D sphere, and this task is about binary decision rules which are invariant under O(3) rotations of the sphere. X has 4096 different patterns, and they are divided into 64 disjoint orbits of the rotation group, which form a minimal sufficient partition/statistics for spherically symmetric rules (Kazhdan et al., 2003) . To generate input-output distribution P (X, Y ), Shwartz-Ziv & Tishby (2017) applied a stochastic rule p(y = 1|x) = Ψ(f (x) -θ), (x ∈ X, y ∈ Y ), where Ψ is a standard sigmoidal functionΨ(u) = 1/(1 + exp(-γu)). Shwartz-Ziv & Tishby (2017) use a spherically symmetric real valued function of the pattern f (x) (evaluated through its spherical harmonics power spectrum (Kazhdan et al., 2003) and compared it to a threshold θ, which was selected to make p(y = 1) =

x p(y = 1|x)p(x) ≈ 0.5, with uniform p(x). γ is high enough to keep the mutual information I(X; Y ) ≈ 0.99 bits. 80% of the data (3277 data pairs) are used for training and rests (819 data pairs) are used for testing. The forward and reverse pass structures of the fully connected neural networks are in Table 7 . Hyper parameters used for training are listed in Table 8 .

All layer are used in racecar loss. All layers in the RR 6

A and Std A are reused for training RR 6

AA/AB and Std AA/AB . For Ort A , we used the Spectral Restricted Isometry Property (SRIP) regularization (Bansal et al., 2018) ,

EQUATION

)

where W is the kernel; I denotes an identity matrix; β represents the regularization coefficient;

σ(W ) = sup z∈R n ,z =0 Wz z

denotes the spectral norm of W . Details about all models' accuracy are shown in Table 1 and Table 2 Mutual information (MI) plane is a powerful tool for neural networks analysis but admittedly not very intuitive. Here we will use MI plane of Std A in Fig. 2 as an example to illustrate MI plane in details. From Table 7 , we can see that neural network in this numerical task has 5 middle layers (L 1∼5 ) and one output layer (L 6 ). The X axis of the MI plane represents the quantity I(X; L), i..e the mutual information between input variable X and output of each layer L. The Y axis of the MI plane represents I(L; Y ), the mutual information between output of each layer L and output variable Y . Besides, every point in the graphs is colored w.r.t. the training epochs, i.e., initially black, and yellow once the training is finished. Hence, we can see six lines changing from black to yellow in figure 14 of the updated version. According to the information bottleneck principle [5], the outputs of the early layers contain more information from the input, which means a high value for I(X; L) and I(L; Y ). We can see that early layers L 1 3 are located in the top right part of the graph. For later layers, such as L 6 , parameters of L 6 are randomly initialized before training, so there are almost no direct relationships between the output of L 6 and X or Y , which means low values of I(X; L) and I(L; Y ). Once the model is well trained, L 6 of Std A is able to generate data which has the same distribution with Y , so I(L; Y ) has increased and moved to the top left corner of the graph. The line of L 6 starts from the bottom left part of the graph, and moves to the top left part during training. For RR 6

A in figure 2 , the L2 constraint is explicitly applied to decrease difference between L and L , and the output is pushed to recover lost information of the input, so I(X; L) of later layers is increased, and I(X; L) of early layers is decreased to make the task easier.

A.2 MNIST CLASSIFICATION:

This section gives details for Sec. 5.1. The regular MNIST data set (LeCun et al., 1998) contains 55k images for training and 10k images for testing. For the n-MNIST motion blur data set (Basu et al., 2017) , there are 60k images for training and 10k images for testing. Image size of them are all 28 × 28. Example data from MNIST and n-MNIST with motion blur are shown in Fig. 15 . Details about the forward and reverse pass network structures are shown in Table 9 . Hyper parameters are listed in Table 10 . 3 and Table 4 . A outperform Std A in both task A and B. We show details about all models in Table 5 .

A.4 SMOKE DATA SETS TEST:

In this section, we will introduce details for Sec. 5.3. The smoke simulation data was generated with a standard fluid solver (Stam, 1999) with MacCormack advection and MiC-preconditioned CG solver via the mantaf low library. We generated 20 simulations with 120 frames for every simulation. 10% of the data was used for training. Smoke inflow region, inflow velocity and buoyancy force were randomized to produce varied data. The low resolution data was down-sampled from the high-resolution data by a factor of 4. Data augmentation, such as flipping and rotation was used in addition. The smoke capture data set contains 2500 smoke images (Eckert et al., 2018) , and we again used 10% as training data set. The forward and reverse pass structures of the network are in Table 13 . Hyper parameters are listed in Table 14 . Note that inputs of discriminator contain high resolution data (64, 64, 1) and low resolution (16, 16, 1), which is up-sampled to (64, 64, 1) and concatenated with high resolution data. 

where ω t is are the weighting factors for layer t; G is the Gram matrix; N t denotes filter numbers of layer t, and M t is the dimension of layer t's filter; T denotes the number of layers included in the style loss. 

:

CONCLUSIONWe have proposed a novel training approach for improving neural network generalization by adding a constrained reverse pass. We have shown for a range of tasks that this yields networks with more general features that more easily transfer to new tasks, and even outperform baselines for the original task. Our training approach is very general, and imposes no requirements regarding network structure or training method. As future work, we believe it will be very interesting to evaluate our approach for other architectures from the vast zoo of existing work, e.g., we are particularly interested in analyzing recurrent structures such as LSTMs and GRUs(Hochreiter & Schmidhuber, 1997;Cho et al., 2014).

Figures and Tables:

Figure 1: Left: An overview of the regular forward pass (blue) and corresponding reverse pass (yellow). The right side illustrates how parameters are reused for a convolutional layer. conv and deconv denote convolution and deconvolutional operations, and fm and BNm denote activation function and batch normalization of layer m, respectively. Shared kernel and bias are represented by km and bm, and we assume that the deconvolution internally transposes the kernel tensor.

Figure 2: MI planes for the initial models RR 1 A , RR 6 A , StdA and OrtA. End points of RR 6 A are located in the middle of the graphs. But for StdA and OrtA, I(L; Y ) values of every layer are very high, which means more specific features about the given output data set are learned by them. Tishby & Zaslavsky (2015) proposed a mutual information plane to analyze the training process, which shows I(X; L) and I(L; Y ) for each layer L over the course of the training epochs. Points in the resulting graphs are colored w.r.t. the epoch, i.e., initially black, and yellow once the training is finished. Each graph shows the 5 middle and output layers. The MI planes visualize how much information about input and output distribution is retained at each layer, and how these relationships change within the network. For regular training, the information bottleneck principle (Tishby & Zaslavsky, 2015) states that early layers contain more information about the input (high I(X; L) and I(Y ; L)). Hence they are often visible at the top-right. Later layers have no relationship with the output I(Y ; L) initially, which increases over the course of a successful training. Thus, they typically move to the top-left. The intuition behind MI planes is explained in more detail in Appendix A.1. We use the same numerical studies as in Shwartz-Ziv & Tishby (2017) as task A, i.e. a regular feedforward neural network with 6 fully-connected layers. The inputs consist of 12 binary digits, and outputs are 2 binary digits. Models are trained using cross entropy as base loss function. Details of this architecture (and following ones) are given in the Sec. A.1. At first, we train a base model RR 6A with a full racecar loss. For comparison, we also show the MI planes for Std A (a regularly trained model), and Ort A (trained with orthogonal constraints(Bansal et al., 2018)). We additionally include a version RR 1 A , i.e. trained with only one racecar loss term λ 1 |L 1 -L 1 | 2 ), which means that only the input is constrained to be recovered. Thus, RR 1A represents a sim-

Figure 3: MI plane and accuracy comparisons for continued training (RR 1 AA , RR 6 AA , StdAA and OrtAA). The RR 6 AA model slightly outperforms the other three models, indicating the positive effect of the full racecar training.As the discussion above focused on phase I, we now turn to analyzing phase II, i.e., follow-up training runs with models trained during phase I. Based on RR 1 A , RR 6A , Ort A and Std A , we continue training the models only for the original task, i.e., RR 1 AA , RR 6 AA , Ort AA and Std AA . The resulting MI planes are shown in Fig.3. While all models now focus on the output (yellow points at the top, maximizing I(L; Y ), there are differences in the distributions of the yellow points along the x axis, i.e., how much MI with the input is retained. We can see that for model RR 6 AA , the final I(X; L 6 ) value at the end of the training is higher than for Std AA , Ort AA and RR 1 AA (final I(L 6 ; Y ) values are shown above each graph). While the final accuracy is high throughout due to the relatively simple setup, RR 6 AA slightly outperforms the other variants for the original task. We will more clearly demonstrate this positive trait of our racecar training with more complex tasks below.

Figure 4: MI plane and accuracy comparisons for a task transfer for models RR 1 AB , RR 6 AB , OrtAB and StdAB. RR 6 AB successfully reuses features from task A for task B, and outperforms StdB. The regular model StdAB yields a very low performance.

Figure 5: Comparisons between regenerated inputs. F.l.t.r.: OrtA, StdA, RR 3 A , and the reference. Only RR 3 A recovers most of the input information successfully.

Figure 8: Regenerated low resolution results and high resolution outputs comparison between StdA, RR 6 A and reference. Only RR 6 A

Figure 10: Left: stylization test from a natural image to the starry night painting style. Right: stylization test from horse to zebra.

Figure 12: Stylization comparison from low resolution to high resolution. RR 16 A 's results are closer to the high resolution.

Figure 13: RR 16A generates a zebra pattern in the horse's shadow, but yields generally improved results compared to the regular transfer in StdA.

.

Figure 14: MI plane of StdA as an example.

Figure 15: Left: data from MNIST; Right: data from n-MNIST with motion blur.

Figure 16: Left: training processes of RR 3 A (blue, accuracy: 0.9810, cost: 5.675 seconds/epoch), StdA (orange, accuracy: 0.9827, cost: 3.522 seconds/epoch) and OrtA (red, accuracy: 0.9792, cost: 4.969 seconds/epoch). Middle: training processes of RR 3 AA (blue), StdAA (orange) and OrtAA (red). Right: training processes os RR 3 AB

Figure 18: Training processes of RR 6 AB 1 (orange), StdAB 1 (blue) and StdB 1 (red). The green inset shows the final loss values.

Figure 22: Top5 accuracy of RR 16 A (red, accuracy: 0.6673, cost: 0.636 second/batch) and StdA (blue, accuracy: 0.7324, cost: 0.308 second/batch).

Figure 23: Top5 accuracy of RR 16 AA (grey) and StdAA (blue).

Model accuracy of MI base task tests

Table content: <table><tr><td>training runs</td><td>OrtA</td><td>RR 1 A</td><td>RR 6 A</td><td>StdA</td><td>OrtAA RR 1 AA</td><td>RR 6 AA</td><td>StdAA</td></tr><tr><td>1</td><td colspan="7">0.976 0.682 0.767 0.973 0.973 0.855 0.997 0.956</td></tr><tr><td>2</td><td>0.944</td><td>0.75</td><td colspan="3">0.779 0.986 0.954 0.909</td><td>0.99</td><td>0.973</td></tr><tr><td>3</td><td colspan="5">0.975 0.731 0.855 0.964 0.979 0.875</td><td>0.99</td><td>0.986</td></tr><tr><td>4</td><td colspan="7">0.993 0.923 0.855 0.981 0.994 0.984 0.995 0.965</td></tr><tr><td>5</td><td colspan="3">0.965 0.405 0.852</td><td>0.94</td><td colspan="3">0.967 0.965 0.999 0.986</td></tr><tr><td>Avg.</td><td colspan="7">0.971 0.707 0.822 0.967 0.973 0.938 0.994 0.973</td></tr><tr><td colspan="8">Std. Dev. 0.018 0.187 0.045 0.018 0.015 0.056 0.004 0.013</td></tr></table>

Model accuracy of MI transfer task tests

Table content: <table><tr><td>training runs</td><td colspan="2">OrtAB RR 1 AB</td><td>RR 6 AB</td><td>StdAB</td><td>StdB</td></tr><tr><td>1</td><td colspan="5">0.978 0.914 0.996 0.136 0.966</td></tr><tr><td>2</td><td colspan="2">0.948 0.956</td><td>1</td><td colspan="2">0.136 0.958</td></tr><tr><td>3</td><td colspan="5">0.976 0.961 0.993 0.136 0.984</td></tr><tr><td>4</td><td>0.994</td><td>0.97</td><td colspan="3">0.996 0.136 0.974</td></tr><tr><td>5</td><td colspan="5">0.968 0.959 0.998 0.136 0.989</td></tr><tr><td>Avg.</td><td colspan="5">0.973 0.956 0.997 0.136 0.974</td></tr><tr><td colspan="4">Std. Dev. 0.017 0.022 0.003</td><td>0</td><td>0.013</td></tr></table>

Model accuracy of MNIST base task tests

Table content: <table><tr><td>training runs</td><td>OrtA</td><td>StdA</td><td>RR 3 A</td><td>OrtAA</td><td>StdAA</td><td>RR 3 AA</td></tr><tr><td>1</td><td colspan="2">0.9792 0.9827</td><td>0.981</td><td colspan="3">0.9819 0.9854 0.9856</td></tr><tr><td>2</td><td colspan="3">0.8322 0.9815 0.9824</td><td>0.982</td><td colspan="2">0.9837 0.9851</td></tr><tr><td>3</td><td colspan="4">0.9511 0.9815 0.9828 0.9792</td><td>0.983</td><td>0.9862</td></tr><tr><td>4</td><td colspan="6">0.8841 0.9817 0.9816 0.9814 0.9842 0.9867</td></tr><tr><td>5</td><td colspan="6">0.8266 0.9829 0.9816 0.9816 0.9846 0.9861</td></tr><tr><td>Avg.</td><td>0.895</td><td>0.982</td><td>0.982</td><td>0.981</td><td>0.984</td><td>0.986</td></tr><tr><td colspan="7">Std. Dev. 0.0689 0.0007 0.0007 0.0011 0.0009 0.0006</td></tr></table>

Model accuracy of MNIST transfer task tests

Table content: <table><tr><td>training times</td><td>OrtAB</td><td>StdAB</td><td>RR 3 AB</td><td>StdB</td></tr><tr><td>1</td><td>0.949</td><td>0.956</td><td colspan="2">0.9612 0.8714</td></tr><tr><td>2</td><td colspan="3">0.8629 0.9458 0.954</td><td>0.9657</td></tr><tr><td>3</td><td colspan="4">0.9458 0.9543 0.9648 0.9659</td></tr><tr><td>4</td><td colspan="4">0.9495 0.9559 0.9604 0.8789</td></tr><tr><td>5</td><td colspan="4">0.8564 0.9466 0.9649 0.9638</td></tr><tr><td>Avg.</td><td>0.913</td><td>0.952</td><td>0.961</td><td>0.929</td></tr><tr><td colspan="5">Std. Dev. 0.0485 0.0051 0.0044 0.0494</td></tr></table>

Model accuracy of Cifar tests

Table content: <table><tr><td>training runs</td><td>StdA</td><td>RR 13 A</td><td>StdAA</td><td>RR 13 AA</td><td>StdAB</td><td>RR 13 AB</td><td>StdB</td></tr><tr><td>1</td><td colspan="3">0.8263 0.5784 0.8351</td><td>0.869</td><td colspan="3">0.2063 0.2602 0.2581</td></tr><tr><td>2</td><td colspan="7">0.7868 0.5553 0.7936 0.8538 0.1838 0.2687 0.2659</td></tr><tr><td>3</td><td colspan="7">0.7729 0.7686 0.7718 0.8449 0.1795 0.2803 0.2386</td></tr><tr><td>4</td><td>0.865</td><td colspan="6">0.7382 0.8472 0.8692 0.1738 0.2969 0.2418</td></tr><tr><td>5</td><td>0.781</td><td colspan="6">0.6775 0.7787 0.8418 0.1704 0.2733 0.2621</td></tr><tr><td>Avg.</td><td>0.806</td><td>0.664</td><td>0.805</td><td>0.856</td><td>0.183</td><td>0.276</td><td>0.253</td></tr><tr><td colspan="8">Std. Dev. 0.0387 0.0945 0.0340 0.0130 0.0141 0.0138 0.0123</td></tr></table>

Model L2 loss of smoke tests. Results for B2: ×10 7 .

Table content: <table><tr><td>training runs</td><td>StdAB 1</td><td>RR 6 AB 1</td><td>StdB 1</td><td colspan="2">StdAB 2 RR 6 AB 2</td><td>StdB 2</td></tr><tr><td>1</td><td>22883</td><td>211.9</td><td>31911</td><td>213</td><td>2.95</td><td>6.13</td></tr><tr><td>2</td><td>493.3</td><td>139</td><td>22291</td><td>2.37</td><td>1.79</td><td>2.66</td></tr><tr><td>3</td><td>1828</td><td>210.2</td><td>31911</td><td>11.4</td><td>2.32</td><td>3.43</td></tr><tr><td>Avg.</td><td>8401.43</td><td colspan="2">187.03 28704.33</td><td>75.6</td><td>2.36</td><td>4.07</td></tr><tr><td colspan="2">Std. Dev. 12559.15</td><td>41.61</td><td>5554.11</td><td>119</td><td>0.578</td><td>1.82</td></tr></table>

Peer review (iclr2020):
This paper proposes a scheme for training layered feedforward neural networks with backwards accumulation of gradients. In the proposed scheme, the intermediate activations during a forward pass are constrained, using an L2 norm penalty, to be close to the activations of a model that inverts the operations of each layer and transforms the data in reverese order (from output to input). The second, inverse, network shares all parameters with the feedforward network. The paper hypothesizes that such training, which they call racecar training, results in features that are transferable between tasks; i.e. using racecar training to learn the features, then applying regular training on a novel task. To support this hypothesis, the paper provides an empirical analysis of the mutual information between inputs and intermediate features, and between intermediate features and outputs (as proposed in the information bottleneck literature). Under this analysis, the paper shows that using racecar training the intermediate layers contain less information about the outputs than standard feedforward training. The paper states that this makes the features learned by racecar training better for transfer: features that enable the network to achieve high predictive accuracy on a particular tasks but that carry little information about the output distribution, thus less specialized. The paper continues this analysis to the transfer setting, first to the same initial task, and then the a new task. In both cases one of the variants of the proposed method appears to produce higher accuracy than standard pre-training. I*m inclining to reject this paper given that the results on the main hypothesis (i.e. transferability of features) seem to provide only marginal improvement, and we have no idea about the repeatability of the results ( how many times did the authors run the experiments for figures 3,4,6,7,9? What*s the spread of the results? Are these results significant?). The results on style transfer and super resolution are promising, but these are only illustrative examples: these results do not provide insights on how well the method works in general, or when does it fail.