Year: 2019

Abstract:
We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.e. racecar training, leads to significantly more general features to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outperforms baseline models trained in a regular fashion. This improved performance is visible for a wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved performance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.

heading: 1 INTRODUCTION
text: Humans spend surprising amounts of time assembling and disassembling objects. These deconstruction tasks serve a wide range of purposes from a hobby for motorists, to an important source of learning and exploration for children (Gopnik et al., 1999). Motivated by this behavioral trait of humans, we propose a surprisingly simple, yet powerful modification of neural network training: in addition to a regular forward pass, we add a reverse pass that is constrained to reconstruct all in-between results of the forward pass as well as the input. As we will demonstrate below, this palindromic structure yields substantial improvements for generalization of the learned features in a wide range of architectures, and in many cases even improves the baseline achieved with regular training. Our results indicate that the reversible nature of the proposed training setup, which we will subsequently refer to via the palindrome ”racecar”, encourages the formation of general and reusable features that benefit a wide range of learning tasks.
With our approach we specifically target transfer learning applications. For a regular, i.e., a nontransfer task, the goal usually is to train a network that gives the optimal performance for one specific goal, as has been demonstrated in many success stories over the years (LeCun et al., 1998; Krizhevsky et al., 2012; Goodfellow et al., 2014; He et al., 2016). In such a case, the network naturally exploits any observed correlations between input and output distribution. E.g., if the color of an object in any way correlates with its type, the training of a classifier should find and use this information. In recent years, even networks trained only for a very specific task were shown to be powerful starting points for training models with different tasks (Zamir et al., 2018; Gopalakrishnan et al., 2017; Ding et al., 2017). In many cases, the original network contained features that were applicable to different data domains and beneficial for new inference tasks. An inherent difficulty in this setting is that typically no knowledge about the specifics of the new data and task domains is available at training time of the source model. While it is common practice to target broad and difficult tasks with the hope that this will yield learned features that are applicable in new domains, we instead specifically target improving the generalizing capabilities of learned features while training the source model.
The core idea of our approach is to add a reverse path during training that is constrained to be as reversible as possible every step of the way. This makes the training task more difficult at first, but at the same time encourages the network to learn reversible and general features. Motivated by the disassembly and assembly processes of humans, and in contrast to previous work on invertible
networks (Gomez et al., 2017; Jacobsen et al., 2018; Zhang et al., 2018), we constrain the network to represent a as-reversible-as-possible process for all intermediate layer activations, instead of only perfectly reproducing the input. Thus, even for cases where a classifier can, e.g., rely on color for inference of an object type, the model is encouraged to learn a representation that can recover the input in order to not only reconstruct the color of an object but also its shape. Hence the internal representation the network builds naturally has to encode as many aspects of the input data distribution as its representational capabilities permit, in order to recover the input. We demonstrate the benefits of our approach for a variety of architectures, from pure convolutional neural networks (CNNs) with and without batch normalization, to networks that include fully connected layers, as well as GAN architectures.

heading: 2 RELATED WORK
text: Transfer learning with deep neural networks has been very successful for a variety of tasks, such as image classification (Duan et al., 2012; Kulis et al., 2011; Zhu et al., 2011), multi-language text classification (Zhou et al., 2014b; Prettenhofer & Stein, 2010; Zhou et al., 2014a), and medical imaging problems (Ravishankar et al., 2016). A central question in transfer learning is whether base and target tasks are related or not. Zamir et al. (2018) proposed an approach to obtain task relationship graphs for different tasks. Another critical point is that we need the neural networks to learn general features from the data set, which are useful for both base and related tasks, rather than some specific features. But how to improve generalization of trained neural networks and reuse them for different related tasks is still a challenge.
In this paper we propose a modified training approach for improving generalization via explicitly building a reverse pass network in addition to a regular forward pass. While Zhang et al. (2018) proposed reverse connected modules, they are primarily used to transfer information from deep layers to shallow layers for a given task. Recovering all input information from hidden representations of a network is generally very difficult (Dinh et al., 2016; Mahendran & Vedaldi, 2016), due to the loss of information over the course of the layer transformations. In this context, Tishby & Zaslavsky (2015) proposed the information bottleneck principle, which states that for an optimal representation, information unrelated to the current task is reduced. This highlights the common specialization of regular training approaches. Ardizzone et al. (2018), Jacobsen et al. (2018) and Gomez et al. (2017) also build reversed networks, but mainly focus on how to make a network fully invertible via introducing special structures. As a consequence, the path from input to output is actually different from the reverse path that translates output to input. In addition, the proposed structures of previous work can not be easily attached to other network architectures. In contrast, we show that it is not necessary to strive for perfect reversibility to obtain an improved performance. Our racecar training also fully preserves an architecture for the backward path, and does not require operations that are not part of the source network. As such it can easily be applied in new settings, such as for adversarial training (Goodfellow et al., 2014).
Aiming for related goals, Bansal et al. (2018) introduced orthogonality regularizations to the loss function. However, the proposed constraints are relatively weak, and make it difficult to arrive at invertible networks. Besides, the orthogonality regularization still focuses on improving performance of a known, given task. This means the training process only extracts features which the network considers useful for improving the performance of the current task. Hence, unlike our method, orthogonality does not necessarily improve generalization or improved transfer performance (Torrey & Shavlik, 2010).

heading: 3 METHOD
text: The goal of transfer learning is to reuse a basic model trained for task A for a related new task B. The performance for B naturally depends crucially on whether the content of the basic model yields benefits for the new task. In line with previous work (Yosinski et al., 2014), we consider the generality of features learned on task A as the extent to which the features can be used for task B. This can be measured in terms of the difference between the transfer performance pAB and training a model for task B from scratch yielding performance pB . Our central goal is improving the
transferability of the basic model w.r.t. the performance metric pAB − pB , which at the same time can be seen as a measure of the generalizing capabilities of the basic model.
Regular training approaches typically construct a model with a certain network structure, and train the model weights for a given task via a suitable loss function. Our approach does not modify this initial structure, but adds a second pass that reverses the initial structure while reusing all weights and biases. E.g., for a typical fully connected layer, the forward pass, the operation L2 = M×L1+b is changed to L ′
1 = M T × (L2 − b) for the reverse pass, where L1 and L2 denote input and output,
respectively. Here, M and b denote weight matrix and bias, while L ′
1 denotes the input regenerated via the reverse pass. We will show and discuss L ′ for several examples below.
Our goal with the reverse pass is to invert all operations of the forward pass to obtain identical intermediate activations between the layers with matching dimensionality. We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass. Due to the symmetric structure of the two passes, we can use a simple L2 difference to drive the network towards aligning the results:
Lracecar = ∑n m=1 λm ∥∥∥Lm − L′m∥∥∥ 2
(1)
Here Lm denotes the input of layer m in the forward pass and L ′
m the output of layer m for the reverse pass. λm denotes a scaling factor for the loss of layer m, which, however, is typically constant in our tests across all layers. An illustration of this process for a CNN structure is shown in Fig. 1. While the construction of the reverse is straight-forward for all standard operations, i.e., fully connected layers, convolutions, pooling etc., batch normalization (BN) and activation function require slight adjustments to map Lm and L ′
m to the same range of values such that they can be compared in the loss. Hence, we use the BN parameters and the activation function of layer m − 1 from the forward pass for layer m in the reverse pass, Note that with our notation, L1 and L ′
1 refer to the input I , and the regenerated input I ′ , respectively. In the following, we will refer to networks trained with the added reverse structure and the loss terms of equation 1 as racecar training.
The constraints of equation 1 intentionally only minimize differences in an averaged manner with an L2 norm, as we don’t strive for a perfectly bijective mapping between input and output domains. Rather, our goal with the racecar training is to encourage the network to extract features that preserve as much information from the input data set as possible with the given representative capabilities of the chosen architecture. Hence, while a regular forward training allows a model to specialize its extracted features to maximize performance for a given task, our approach encourages the network to consider the full input data distribution, such that it ideally can be recovered from the latent-space representation of all layers.
To differentiate variants, we will use the following naming scheme: StdA/B, RRsA/B, OrtA/B for trained base models. Here, Std denotes a regular training run (always shown in yellow color in graphs below), while RRs denotes models trained with our racecar training (in green below). Here, the s superscript denotes how many intermediate results of layers are used, e.g., RR3 means n = 3 in equation 1, which constrains the input data as well as the next two layers of the original structure.
Correspondingly, a network trained with RR1 means the network is trained to regenerate the input without any constraints for the activations inside of the network. This represents a special, and somewhat sub-optimal case of our method, as we will demonstrate below. Ort additionally denotes models trained with orthogonal constraints (Bansal et al., 2018) (in blue). The subscripts A/B denote the task A/B the model was trained for. We will call direct training for either task phase I in the following.
In phase II, we reuse a model trained in phase I for new tasks (typically B), with a regular training approach. I.e., in phase II, all models are only trained with the forward structure and original loss, and do not use the racecar loss or other modifications. We will use the naming scheme StdAA/AB, RRsAA/AB, OrtAA/AB for trained models in phase II. Here AA/AB mean the model was trained for task A during phase I, and is then trained for task A/B in phase II.
It is worth pointing out that the additional constraints of our racecar training lead to increased requirements for memory and additional computations during phase I, e.g., it is 61.13% slower per epoch for the MNIST tests. And while the constraints can also lead to a slight deterioration of performance, e.g., a 0.17% lower accuracy for MNIST tests during phase I, we will demonstrate that our models outperform baselines during phase II even for the original task, and thus in practice justify introducing the racecar training.

heading: 4 EVALUATION IN TERMS OF MUTUAL INFORMATION
text: We now evaluate our approach in terms of the mutual information (MI) between input and output data distributions. As our approach hinges on the introduction of the reverse pass, we will show that our approach succeeds in terms of establishing mutual information between the input and the constrained intermediates inside a network. More formally, the mutual information I(X;Y ) of random variables X and Y measures how different the joint distribution of X and Y is w.r.t. the product of their marginal distributions, i.e., the Kullback-Leibler divergence: I(X;Y ) = DKL[P(X,Y )||PXPY ].
Tishby & Zaslavsky (2015) proposed a mutual information plane to analyze the training process, which shows I(X;L) and I(L;Y ) for each layer L over the course of the training epochs. Points in the resulting graphs are colored w.r.t. the epoch, i.e., initially black, and yellow once the training is finished. Each graph shows the 5 middle and output layers. The MI planes visualize how much information about input and output distribution is retained at each layer, and how these relationships change within the network. For regular training, the information bottleneck principle (Tishby & Zaslavsky, 2015) states that early layers contain more information about the input (high I(X;L) and I(Y ;L)). Hence they are often visible at the top-right. Later layers have no relationship with the output I(Y ;L) initially, which increases over the course of a successful training. Thus, they typically move
to the top-left. The intuition behind MI planes is explained in more detail in Appendix A.1. We use the same numerical studies as in Shwartz-Ziv & Tishby (2017) as task A, i.e. a regular feedforward neural network with 6 fully-connected layers. The inputs consist of 12 binary digits, and outputs are 2 binary digits. Models are trained using cross entropy as base loss function. Details of this architecture (and following ones) are given in the Sec. A.1. At first, we train a base model RR6A with a full racecar loss. For comparison, we also show the MI planes for StdA (a regularly trained model), and OrtA (trained with orthogonal constraints (Bansal et al., 2018)). We additionally include a version RR1A, i.e. trained with only one racecar loss term λ1|L1 − L ′
1|2), which means that only the input is constrained to be recovered. Thus, RR1A represents a sim-
plified version of our approach which receives no constraints that the intermediate results of forward and backward pass should match. The resulting information planes are shown in Fig. 2. We can see that the end points of RR6A, i.e., the yellow clusters of points for the final state of the model, are located in the middle part of the graph. This means that all layers successfully encode information about the inputs as well as the outputs. In contrast, for StdA and OrtA, I(X;L) values of later layers are very low (points near the top left), while I(L;Y ) is high throughout. This indicates that the outputs were successfully encoded, and that increasing amounts of information about the inputs are discarded by StdA and OrtA. Hence, both models are highly specialized for the given task. Comparing RR1A with RR 6 A, the yellow end points of the latter are located in a centralized region, which indicates that every layer contains similar information about X and Y . It also indicates that the path from input to output is similar to the path from output to input. The end points of RR1A, on the other hand, are located in a scattered region. I.e., this network has different amounts of mutual information across its layers, and potentially a very different path in each direction. RR1A is only constrained to be able to regenerate its input, while the full racecar loss for RR6A ensures that the network learns features which are beneficial for both directions. This test highlights the importance of the constraints throughout the depth of a network in our racecar loss.
As the discussion above focused on phase I, we now turn to analyzing phase II, i.e., follow-up training runs with models trained during phase I. Based on RR1A, RR6A, OrtA and StdA, we continue training the models only for the original task, i.e., RR1AA, RR 6 AA, OrtAA and StdAA. The resulting MI planes are shown in Fig. 3. While all models now focus on the output (yellow points at the top, maximizing I(L;Y ), there are differences in the distributions of the yellow points along the
x axis, i.e., how much MI with the input is retained. We can see that for model RR6AA, the final I(X;L6) value at the end of the training is higher than for StdAA, OrtAA and RR1AA (final I(L6;Y ) values are shown above each graph). While the final accuracy is high throughout due to the relatively simple setup, RR6AA slightly outperforms the other variants for the original task. We will more clearly demonstrate this positive trait of our racecar training with more complex tasks below.
We now analyze a new task, to check whether the model learned specific or general features, and reverse output labels as transfer learning task B. E.g., if the output is [0,1] in the original data set, we invert it to [1,0]. Based on RR1A, RR 6 A, OrtA and StdA, we train RR 1 AB, RR 6 AB, OrtAB and StdAB for the modified data set. For comparison, we also train a model trained StdB from scratch. MI planes and accuracy comparisons are shown in Fig. 4. While the range in performance is relatively small across most models, StdAB stands out with a very low accuracy. This model from a regular training run has large difficulties to adapt to the new task. Model OrtAB also performs worse than StdB. RR6AB shows the best performance in this setting, confirming the previous MI plane visualizations, and demonstrating that our loss formulation helped to learn more general features from the input data, improving the performance for related tasks such as the inverted outputs.

heading: 5 EXPERIMENTAL RESULTS
text: We now turn to more complex network structures (CNNs, GANs, Auto-Encoders), with different data sets (MNIST, Cifar, smoke, ImageNet) and different tasks (such as classification and synthesis) to show that models trained with our approach succeed in learning very general features that transfer to new tasks.
5.1 DIGIT CLASSIFICATION
The MNIST data set is a commonly used data set for hand written digit classification (LeCun et al., 1998). At first, we train three models for regular MNIST data set classification as task A, one with racecar loss RR3A, a regular model StdA, and one with orthogonal OrtA constraints (Bansal et al., 2018), as the latter also aims for improving CNN performance. Usually, convolutional layers take up most of the model parameters, so we correspondingly compute the racecar loss for the convolutional layers, omitting the fully connected layers that would be required for class label inference. To
highlight the properties of our algorithm, we show comparisons between I and the regenerated I ′ in Fig. 5. We can see that for racecar training, most of the features from the input are recovered. Trying to invert the network in the same way for a regular training run or training with orthogonal constraints largely fails, as the extracted features are extracted according to the digit classification and discard information unrelated to this task.
Based on OrtA, StdA, RR3A, we continue training to obtain models OrtAA, StdAA, and RR3AA. Results are shown in Fig. 6. We repeated these runs 5 times, in order to draw reliable conclusions. We can see that RR3AA outperforms OrtAA and StdAA at the end of training, which indicates that racecar training yields general features that can also improve performance for original task. Council (2000) illustrated that balanced learning of both general and specific features is more effective for human learning, and our results here are consistent with this intuition.
As general features are more robust than specific features (Novak et al., 2018), we investigate a perturbed data set for the transfer task B. We apply OrtA, StdA and RR3A to n-MNIST, a data set for classification with motion blur (Basu et al., 2017). Performance results are likewise given in Fig. 6. Based on the same CNN structure and parameters, RR3A achieves the best
performance. This indicates that RR3A learned more general features via racecar training than OrtA and StdA.
5.2 NATURAL IMAGE CLASSIFICATION
Natural images arise in many important application scenarios. Hence, we evaluate our approach with the Cifar data set (Krizhevsky et al., 2009). At first, we train two models as Cifar10 data set classification taskA, RR13A and StdA. We again continue training to obtain RR13AA and StdAA, results for which are shown in Fig. 7. The racecar training also improves performance for this natural image classification task. For transfer learning task B, we reuse RR13A and StdA for Cifar-100 data set classification (RR 13 AB and StdAB). Results are also shown in Fig. 7. Training with the same CNN structure and parameters, StdAB has difficulties adjusting to the new task, while our model from the initial racecar training slightly outperforms the model trained for scratch for B.

heading: 5.3 GENERATIVE ADVERSARIAL MODELS
text: In the previous sections, we employed commonly used data sets and transfers to related tasks. Next, we test our approach for a transfer situation with a more challenging transfer from a synthetic single channel data set of synthetic smoke simulations, to RBG videos of real smoke clouds. We use GAN and auto-encoder structures for the following models.
At first, we use a GAN structure (one generator and one discriminator network) for super resolution of the simulation data as task A. StdA is trained via regular training, and RR6A uses racecar training for the generator network of the GAN. Regenerated low resolution results and high resolution outputs comparisons are shown in Fig. 8. Both RR6A and StdA can up scale low resolution data 4 times larger to high resolution data very well. But RR6A can also recover low resolution versions from high resolution data. StdA fails in regenerating the low resolution images. This shows that RR6A not only focused on
generating good features for the super-resolution task, but also preserves information about the input.
We reuse the generator models RR6A and StdA to train two sets of auto-encoder networks: once for the synthetic smoke data as transfer task B1, and as a second task B2 for RBG videos of real-world smoke clouds (example frames are given in the appendix). This yields models RR6AB1,2 and StdAB1,2 , respectively. The resulting accuracies, summarized in Fig. 9, show that racecar training performs best for both auto-encoding tasks. This is especially encouraging for task B2, as it represents a transfer from fully synthetic training to real-world images.
Synthetic data and real-world images have significant differences in terms of their features. In the worst case, most of the features learned by the regular learning process StdA can not be transferred to the new task. Besides, an inappropriate starting point of the transfer learning task can guide the whole training process towards a wrong direction. Instead, a model with more general features such as RR6A can more easily be reused in new tasks, and can provide a better starting point for learning. This indicates the potential of racecar training to obtain generalizable features from synthetic data sets that can be used for tasks working with real-world data.
5.4 VGG19 STYLIZATION
In order to provide a qualitative evaluation in a complex, visual scenario, we turn to the popular task of image stylization. We use V GG19 networks (Simonyan & Zisserman, 2014) in the following, and we train two V GG19 networks RR16A and StdA with ImageNet data set (Deng et al., 2009). Based on these, we train RR16AA and StdAA, top5 accuracy of which are 77.59% and 75.28%, respectively. Consistent with previous test, RR16AA outperforms the StdAA baseline even in this complex case for a net-
work with ca. 143 million weights. Gatys et al. (2016) achieve excellent stylization results, but the results strongly depend on the pre-trained VGG network, i.e., it stylizes and changes structures that are represented by the features in the VGG network. Hence, we employ stylization to visualize which features, general or specialized, the two VGG versions focus on.
Gatys et al. (2016) generate stylized results via optimizing two losses, Lstyle and Lcontent, which measure style difference and content difference, respectively (details are given in the appendix. To compare the feature extracting capabilities between the RR16A and StdA runs, we only optimize Lstyle.
𝑅𝑅𝑠𝑜𝑢𝑟𝑐𝑒 (𝑙𝑜𝑤 𝑟𝑒𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛)
𝑠𝑡𝑦𝑙𝑒 (ℎ𝑖𝑔ℎ 𝑟𝑒𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛) 𝑆t𝑑
𝐸
𝐸
Figure 12: Stylization comparison from low resolution to high resolution. RR16A ’s results are closer to the high resolution.
As a first test, we use two simple shapes as source and style images, as shown in Fig. 11. They have the same background and object color. The only difference is the large-scale shape of the object. Thus, as we aim for applying an style that is identical to the source, we can test whether the features can cleanly separate and preserve the large scale shape features from the ones for the localized style. Comparing the results of RR16A and StdA, the output of RR16A is almost identical to the input, while stylization with StdA changes the shape of the object introducing undesirable streaks around the outline. This result indicates that RR16A is able to cleanly encode the shape of the object and preserve it during the stylization.
We show more stylization tests in Fig. 10, which optimize both Lcontent and Lstyle. For the left example of Fig. 10 with a Van Gogh style image, both models can generate good stylized results. However, from the results obtained with StdA (orange square area), we can see a star within the building, which indicates that this model mixed style and context information in its features. Across our tests we found that models like this one, trained for more specific features, more often performed simple patch-based matches rather than generating meaningful transfers of styles. However, if the model focuses on more general and abstract features, it can perform a better semantic matching and generate improved stylization results. For the right part of Fig. 10, we show a test transferring a horse image to a zebra style. RR16A focuses more on general features, such as grass, zebra, and their positional relationships. As a result the model successfully distinguishes foreground from background, and changes the style for both foreground and background objects with only smaller
mistakes. StdA’s performance is significantly worse, leading to large grass-like areas within the body of the horse.
We confirm the capabilities of the VGG model from racecar training with a low- to high-resolution transfer where the goal is purely to add detail. I.e., we can compute in place errors w.r.t. high-resolution reference. The result is shown in Fig. 12. The VGG model obtained with a regular training run yields significantly higher errors, as visible on the right side of the figure.
RR16A can extract more general information from the input, which is helpful with generating semantically meaningful results. However, we found that this property can also confuse the model in some specific situations. For instance, as shown in Fig. 13, at first, RR16A can distinguish foreground and background and generate better stylization results than StdA. However, RR16A also stylizes the shadow of the horse. This also indicates that RR16A fo-
cuses more on general features. As a consequence, all horse shape objects are extracted and then stylized with zebra textures, such as the shadow in Fig. 13. However, we generally think that this can provide potentially helpful information in other situations, e.g., for classification tasks. Overall, the quality of the racecar transfer also still surpasses the regular version.

heading: 6 CONCLUSION
text: We have proposed a novel training approach for improving neural network generalization by adding a constrained reverse pass. We have shown for a range of tasks that this yields networks with more general features that more easily transfer to new tasks, and even outperform baselines for the original task. Our training approach is very general, and imposes no requirements regarding network structure or training method. As future work, we believe it will be very interesting to evaluate our approach for other architectures from the vast zoo of existing work, e.g., we are particularly interested in analyzing recurrent structures such as LSTMs and GRUs (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).

heading: A APPENDIX
text: Here we will present more details, such as data set information, network structures and training parameters, for all of our mentioned tests above: Mutual Information Sec. A.1, MNIST classification Sec. A.2, Cifar Sec. A.3, smoke Sec. A.4 and VGG Sec. A.5. We will use C(k, l, s), D(k, l, s) to represent convolutional and deconvolutional operations, respectively, and fully connected layers are noted with F (l), where k, l, s denote kernel size, output channels and stride size, respectively. Bias of CNN layer is denoted with b. I/O(z) denote input/output and their dimensionality is given by z. Ir denotes the input of reverse pass network. tanh, relu, lrelu denote corresponding activation functions, where we typically use a leaky tangent of 0.2 for the negative half space. UP , MP and BN denote 2× nearest-neighbor up sampling, max pooling with 2× 2 filters and stride 2, and batch normalization, respectively. All performance numbers were measured on a Nvidia GeForce GTX 1080 Ti GPUs and Intel Core i7-6850K CPUs.

heading: A.1 MUTUAL INFORMATION TEST
text: Below we will introduce more details about tests in Sec. 4.For the numerical task (Shwartz-Ziv & Tishby, 2017), input variableX are 12 binary digits that represent 12 uniformly distributed points on a 2D sphere, and this task is about binary decision rules which are invariant under O(3) rotations of the sphere. X has 4096 different patterns, and they are divided into 64 disjoint orbits of the rotation group, which form a minimal sufficient partition/statistics for spherically symmetric rules (Kazhdan et al., 2003). To generate input-output distribution P (X,Y ), Shwartz-Ziv & Tishby (2017) applied a stochastic rule p(y = 1|x) = Ψ(f(x) − θ), (x ∈ X, y ∈ Y ), where Ψ is a standard sigmoidal functionΨ(u) = 1/(1 + exp(−γu)). Shwartz-Ziv & Tishby (2017) use a spherically symmetric real valued function of the pattern f(x) (evaluated through its spherical harmonics power spectrum (Kazhdan et al., 2003) and compared it to a threshold θ, which was selected to make p(y = 1) =∑
x p(y = 1|x)p(x) ≈ 0.5, with uniform p(x). γ is high enough to keep the mutual information I(X;Y ) ≈ 0.99 bits. 80% of the data (3277 data pairs) are used for training and rests (819 data pairs) are used for testing. The forward and reverse pass structures of the fully connected neural networks are in Table 7. Hyper parameters used for training are listed in Table 8.
All layer are used in racecar loss. All layers in the RR6A and StdA are reused for training RR 6 AA/AB and StdAA/AB. For OrtA, we used the Spectral Restricted Isometry Property (SRIP) regularization (Bansal et al., 2018), LSRIP = βσ(WTW − I) , (2) where W is the kernel; I denotes an identity matrix; β represents the regularization coefficient; σ(W ) = supz∈Rn,z 6=0 ‖Wz‖ ‖z‖ denotes the spectral norm of W . Details about all models’ accuracy are shown in Table 1 and Table 2.
Mutual information (MI) plane is a powerful tool for neural networks analysis but admittedly not very intuitive. Here we will use MI plane of StdA in Fig. 2 as an example to illustrate MI plane in details. From Table 7, we can see that neural network in this numerical task has 5 middle layers (L1∼5) and one output layer (L6). TheX axis of the MI plane represents the quantity I(X;L), i..e the mutual information between input variable X and output of each layer L. The Y axis of the MI plane represents I(L;Y ), the mutual information between output of each layer L and output variable Y . Besides, every point in the graphs is colored w.r.t.
the training epochs, i.e., initially black, and yellow once the training is finished. Hence, we can see six lines changing from black to yellow in figure 14 of the updated version. According to the information bottleneck principle [5], the outputs of the early layers contain more information from the input, which means a high value for I(X;L) and I(L;Y ). We can see that early layers L1 3 are located in the top right part of the graph. For later layers, such as L6, parameters of L6 are randomly initialized before training, so there are almost no direct relationships between the output of L6 and X or Y , which means low values of I(X;L) and I(L;Y ). Once the model is well trained, L6 of StdA is able to generate data which has the same distribution with Y , so I(L;Y ) has increased and moved to the top left corner of the graph. The line of L6 starts from the bottom left part of the graph, and moves to the top left part during training. For RR6A in figure 2, the L2 constraint is explicitly applied to decrease difference between L and L ′ , and the output is pushed to recover lost information of the input, so I(X;L) of later layers is increased, and I(X;L) of early layers is decreased to make the task easier.

heading: A.2 MNIST CLASSIFICATION
text: This section gives details for Sec. 5.1. The regular MNIST data set(LeCun et al., 1998) contains 55k images for training and 10k images for testing. For the n-MNIST motion blur data set(Basu et al., 2017), there are 60k images for training and 10k images for testing. Image size of them are all 28× 28. Example data from MNIST and n-MNIST with motion blur are shown in Fig. 15. Details
about the forward and reverse pass network structures are shown in Table 9. Hyper parameters are listed in Table 10.
All 3 convolutional layers are used for racecar loss and all layers in RR3A, StdA and OrtA are reused for training RR3AA/AB, StdAA/AB and OrtAA/AB. Example training processes of the MNIST tests are shown in Fig. 16. We can see that racecar loss increases the task difficulty, so RR3A yields a lower performance and longer training time than StdA in the first phase. In the second phase RR3A outperforms StdA and OrtA in both task A an B, which indicates that racecar training is helpful with general feature extraction. For OrtA, we also use equation 2 as orthogonal regularization. All models’ accuracy results are listed in Table 3 and Table 4.

heading: A.3 CIFAR TEST
text: All 13 convolutional layers are used for racecar loss and all layers except the last fully connected layer (because of different output size) in RR13A and StdA are reused for training RR 13 AA/AB and StdAA/AB. Example training processes of the Cifar tests are shown in Fig. 17. In phase II, the model RR13A outperform StdA in both task A and B. We show details about all models in Table 5.

heading: A.4 SMOKE DATA SETS TEST
text: In this section, we will introduce details for Sec. 5.3. The smoke simulation data was generated with a standard fluid solver (Stam, 1999) with MacCormack advection and MiC-preconditioned CG
solver via the mantaflow library. We generated 20 simulations with 120 frames for every simulation. 10% of the data was used for training. Smoke inflow region, inflow velocity and buoyancy force were randomized to produce varied data. The low resolution data was down-sampled from the high-resolution data by a factor of 4. Data augmentation, such as flipping and rotation was used in addition. The smoke capture data set contains 2500 smoke images (Eckert et al., 2018), and we again used 10% as training data set. The forward and reverse pass structures of the network are in Table 13. Hyper parameters are listed in Table 14. Note that inputs of discriminator contain high resolution data (64, 64, 1) and low resolution (16, 16, 1), which is up-sampled to (64, 64, 1) and concatenated with high resolution data.
All generator layers are involved in racecar loss. All 6 layers of RR6A and StdA are reused for training RR6AB1 and StdAB1 , but only 5 layers are reused for training RR 6 AB2 and StdAB2 because of different data size. The following results give further details for the auto-encoder transfer learning task AB1 which uses synthetic, i.e., simulated fluid data. Example training processes of RR6AB1 (orange), StdAB1 (blue) and model trained from scratch StdB1 (red) are shown in Fig. 18. RR 6 AB1 achieved the lowest L2 loss after training. Example outputs of RR6AB1 , StdAB1 and StdB1 are shown in Fig. 19. It becomes clear that model RR6AB1 gives the best performance across these models.

heading: L2_loss
text: We similarly illustrate the behavior of the transfer learning task AB2 for images of real-world fluids. This example likewise uses an auto-encoder structure. Example training processes of RR6AB2 , StdAB2 and model trained from scratch StdB2 are shown in Fig. 20. RR 6 AB2 yields the best performance at the end of training. Example output comparisons of RR6AB2 , StdAB2 and StdB2 are shown in Fig. 21. We can see that error of RR6AB2 is lower than StdAB2 and StdB2 . Detail comparisons between different models are shown in Table 6.

heading: L2_loss
text: A.5 VGG TEST
We now give details of the tests in Sec. 5.4. For the ImageNet data set (Deng et al., 2009), 1281167 images of 1000 classes are used for training, and 50k images are used for testing. Image size is 224× 224. The forward and reverse pass of VGG19 are in Table 16. Hyper parameters are listed in Table 15.
All 16 convolutional layers are used for racecar loss. To speed up training process of RR16A , we firstly train a model without racecar loss for 6 epochs with batch size 64, as regular training. And then we reuse this model for training RR16A and StdA with batch size 24. Example training processes of RR16A and StdA are shown in Fig. 22, while those of RR 16 AA and StdAA are shown in Fig. 23.
Next, we will give additional details for the stylization tests. Gatys et al. (2016) use Ltotal = ηLcontent + δLstyle for stylization optimizations, where η and δ are coefficient factors. Lcontent is used to calculate content difference between source image p and generated image g, as shown in equation 3.
Lcontent(p, g, t) = 1 2 ∑ m,n(F m,n,t p − Fm,n,tg )2, (3)
where Fm,n,tp is p ′s feature representation of the mth filter at position n in layer t. Lstyle is used to calculate style difference between style image a and generated image g, as shown in equation 4. The style loss can be written as:
Gm,n,ta = ∑ f F m,f,t a F n,f,t a , Et = 1
4N2t M 2 t
∑ m,n(G m,n,t g −Gm,n,ta )2,
Lstyle(a, g) = ∑T t=0 ωtEt,
(4)
where ωt is are the weighting factors for layer t; G is the Gram matrix; Nt denotes filter numbers of layer t, and Mt is the dimension of layer t’s filter; T denotes the number of layers included in the style loss.

Review summaries:
This paper attempts to learn general and reusable features for transfer learning tasks. The authors propose a training paradigm called Racecar Training. The core idea of it is to operate a reverse pass for the network. The authors use mutual information to analyze the network for its improved generalizing capabilities. They also conduct experiments on classification, regression and stylization to validate their method’s effectiveness.

This paper proposes a scheme for training layered feedforward neural networks with backwards accumulation of gradients. In the proposed scheme, the intermediate activations during a forward pass are constrained, using an L2 norm penalty, to be close to the activations of a model that inverts the operations of each layer and transforms the data in reverese order (from output to input). The second, inverse, network shares all parameters with the feedforward network.