Learning General and Reusable Features via Racecar-Training
Authors: You Xie
Nils Thuerey
Venue:  (2020)
Keywords: transfer learning
neural networks
generalization
reusable features

Abstract: We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.e. racecar training, leads to significantly more general features to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outperforms baseline models trained in a regular fashion. This improved performance is visible for a wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved performance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.

Under review as a conference paper at ICLR 2020
LEARNING GENERAL AND REUSABLE FEATURES VIA RACECAR-TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.c. racecar training, leads to significantly more general fea- tures to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outper- forms baseline models trained in a regular fashion. This improved performance is visible for wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved per- formance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.
INTRODUCTION
Humans spend surprising amounts of time assembling and disassembling objects. These decon- struction tasks serve wide range of purposes from hobby for motorists, to an important source of learning and exploration for children (Gopnik et al. 1999). Motivated by this behavioral trait of humans, we propose a surprisingly simple, yet powerful modification of neural network training: in addition to a regular forward pass, we add a reverse pass that is constrained to reconstruct all in-between results of the forward pass as well as the input. As we will demonstrate below, this palindromic structure yields substantial improvements for generalization of the learned features in a wide range of architectures, and in many cases even improves the baseline achieved with regular training. Our results indicate that the reversible nature of the proposed training setup, which we will subsequently refer to via the palindrome "racecar", encourages the formation of general and reusable features that benefit a wide range of learning tasks.
With our approach we specifically target transfer learning applications. For a regular, i.e., a non- transfer task, the goal usually is to train network that gives the optimal performance for one spe- cific goal, as has been demonstrated in many success stories over the years (LeCun et al., [998] Krizhevsky al. 2012; Goodfellow et al. 2014 He et al. 2016). In such case, the network nat- urally exploits any observed correlations between input and output distribution. E.g., if the color of object in any way correlates with its type, the training of classifier should find and use this information. In recent years, even networks trained only for a very specific task were shown to be powerful starting points for training models with different tasks (Zamir al. 2018 Gopalakrishnan et al. 2017; Ding et al. 2017). In many cases, the original network contained features that were applicable to different data domains and beneficial for new inference tasks. An inherent difficulty in this setting is that typically no knowledge about the specifics of the new data and task domains is available at training time of the source model. While it common practice to target broad and dif- ficult tasks with the hope that this will yield learned features that are applicable in new domains, instead specifically target improving the generalizing capabilities of learned features while training the source model.
The core idea of our approach is to add a reverse path during training that is constrained to be as reversible possible every step of the way. This makes the training task more difficult at first, but at the same time encourages the network to learn reversible and general features. Motivated by the disassembly and assembly processes of humans, and in contrast previous work invertible

---

Under review as a conference paper at ICLR 2020
constrain the network to represent a as-reversible-Ba alsssssee possible process for all intermediate layer activations, instead of only perfectly reproducing the input. Thus, even for cases where classifier can, e.g., rely color for inference of an object type, the model is encouraged to learn a representation that can recover the input in order to not only reconstruct the color of an object but also its shape. Hence the internal rep- resentation the network builds naturally has to encode as many aspects of the input data distribution as its representational capabilities permit, in order to recover the input. We demonstrate the benefits of our approach for variety of architectures, from pure convolutional neural networks (CNNs) with and without batch normalization, to networks that include fully connected layers, as well as GAN architectures.
networks (Gomez al. 2017: Jacobsen et al. 2018 Zhang al. 2018),
2 RELATED WORK
Transfer learning with deep neural networks has been very successful for a variety of tasks, such as image classification (Duan et al. 2012 Kulis et al., 2001 Zhu et al.] 2011), multtilanguage text classification (Zhou et al. 2014b) Prettenhofer & Stein 2010 Zhou et al. 2014a), and medical imaging problems (Ravishankar et al.) (016). A central question in transfer learning is whether base and target tasks are related or not. Zamir et al. (2018) proposed an approach to obtain task relationship graphs for different tasks. Another critical point that we need the neural networks to learn general features from the data set, which are useful for both base and related tasks, rather than some specific features. But how to improve generalization of trained neural networks and reuse them for different related tasks is still a challenge.
In this paper we propose a modified training approach for improving generalization via explicitly building reverse pass network in addition to a regular forward pass. While While/Zhang et al. (2018) pro- posed reverse connected modules, they are primarily used to transfer information from deep layers to shallow layers for a given task. Recovering all input information from hidden representations of network is generally very difficult (Dinh et al. 2016 Mahendran & Vedaldi 2016),, due to the loss of information over the course of the layer transtormations. In this context, [Tishby & Zaslavsky (2015) proposed the information bottleneck principle, which states that for an optimal representation, infor- mation unrelated to the current task is reduced. This highlights the common specialization of regular training approaches. Ardizzone et al. (2018). Jacobsen et al. (2018) and Gomez et al. (2017) also build reversed networks, but mainly focus on how to make network fully invertible via introducing special structures. As a consequence, the path from input to output is actually different from the reverse path that translates output to input. In addition, the proposed structures of previous work can not be easily attached to other network architectures. In contrast, we show that it is not necessary to strive for perfect reversibility to obtain an improved performance. Our racecar training also fully preserves an architecture for the backward path, and does not require operations that are not part of the source network. As such it can easily be applied in new settings, such as for adversarial training (Goodfellow et al. 2014).
Aiming for related goals, Bansal et al. 2018 introduced orthogonality regularizations to the loss function. However, the proposed constraints are relatively weak, and make it difficult to arrive at in- vertible networks. Besides, the orthogonality regularization still focuses on improving performance of known, given task. This means the training process only extracts features which the network considers useful for improving the performance of the current task. Hence, unlike our method, or- thogonality does not necessarily improve generalization or improved transfer performance (Torrey & Shavlik 2010).
METHOD
The goal of transfer learning is to reuse a basic model trained for task A for a related new task B. The performance for B naturally depends crucially on whether the content of the basic model yields benefits for the new task. In line with previous work (Yosinski et al. 2014), we consider the generality of features learned on task A the extent to which the features can be used for task B. This can be measured in terms of the difference between the transfer performance Рad and training a model for task B from scratch yielding performance pa. Our central goal is improving the
2

---

Under review as a conference paper at ICLR 2020
transferabilityy of the basic model w.r.t. the performance metric PAB PB. which at the same time can be seen as a measure of the generalizing capabilities of the basic model.
Regular training approaches typically construct model with certain network structure, and train the model weights for a given task via a suitable loss function. Our approach does not modify this initial structure, but adds second pass that reverses the initial structure while reusing all weights and biases. E.g., for a typical fully connected layer, the forward pass, the operation L2 = M x L +b is changed to L = M x (L2 - b) for the reverse pass, where L and L2 denote input and output, respectively. Here, M and b denote weight matrix and bias, while L, denotes the input regenerated via the reverse pass. We will show and discuss L' for several examples below.
Our goal with the reverse pass is to invert all operations of the forward pass to obtain identical intermediate activations between the layers with matching dimensionaliiy. We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass. Due to the symmetric structure of the passes, we can simple L difference drive the network towards aligning the results:
Erccecur Σn=
Am
Lm
I'm
Here Lm denotes the input of layer m in the forward pass and L'm the output of layer m for the reverse pass. Am denotes a scaling factor for the loss of layer m, which, however, is typically constant in our tests across all layers. An illustration of this process for CNN structure is shown in Fig. | While the construction of the reverse is straight-forward for all standard operations, i.e., fully connected layers, convolutions, pooling etc., batch normalization (BN) and activation function require slight adjustments to map Lm and L," to the same range of values such that they can be compared in the loss. Hence, we use the BN parameters and the activation function layer m - from the forward pass for layer m in the reverse pass, Note that with our notation, L1 and Lj refer to the input I, and the regenerated input I', respectively. In the following, we will refer to networks trained with the added reverse structure and the loss terms of equation | as racecar training.
The constraints of equation I intentionally only minimize differences in an averaged manner with an L norm, as we don't 't strive for a perfectly bijective mapping between input and output domains. Rather, our goal with the racecar training is to encourage the network to extract features that preserve as much information from the input data set as possible with the given representative capabilities of the chosen architecture. Hence, while regular forward training allows a model to specialize its extracted features to maximize performance for given task, our approach encourages the network to consider the full input data distribution, such that it idcally can be recovered from the latent-space representation of all layers.
To differentiate variants, we will use the following naming scheme: Std RR Ortди for trained base models. Here, Std denotes a regular training run (always shown in yellow color in graphs below), while RR denotes models trained with our racecar training (in green below). Here, the superscript denotes how many intermediate results of layers are used, e.g., RRS means n = 3 in equation which constrains the input data well as the next two layers of the original structure.
layar
layar
Li
(conoolhh R.,)) b())
output
fre (BN, NN (decon - B R.
Figure 1: Left: An overview the regular forward pass (blue) and corresponding reverse pass (yellow). The right side illustrates how parameters are reused for convolutional layer. conv and deconv denote convolution and deconvolutional operations, and fm and B.... denote activation function and batch normalization of layer m, respectively. Shared kernel and bias represented by km, and bm. and assume that the deconvolution internally transposes the kernel tensor.
3

---

Under review as a conference paper at ICLR 2020
Correspondingly, network trained with RR' means the network is trained regenerate the input without any constraints for the activations inside of the network. This represents a special, and somewhat sub-optimal case of our method, as we will demonstrate below. Ort additionally denotes models trained with orthogonal constraints (Bansal et al. 2018) (in blue). The subscripts A/ B denote the task A/ B the model was trained for. We will call direct training for either task phase in the following.
In phase II, we reuse model trained in phase I for new tasks (lypically B), with a regular training approach. Ie., in phase II, all models are only trained with the forward structure and original loss, and do not use the racecar loss or other modifications. We will use the naming scheme SAdAA/AB- RRAAA OπlдлAaB for trained models in phase II. Here AA/AB mean the model was trained for task during phase I, and is then trained for task A/ B phase II.
It is worth pointing out that the additional constraints of our racecar training lead to increased re- quirements for memory and additional computations during phase I, e-g., 61. 133% slower per epoch for the MNIST tests. And while the constraints can also lead to a slight deterioration of performance, e.g., a 0.17% lower accuracy for MNIST tests during phase we will demonstrate that our models outperform baselines during phase II even for the original task, and thus in practice justify introducing the racecar training.
4 EVALUATION IN TERMS OF MUTUAL INFORMATION
We evaluate approach in terms of the mutual information (MI) between input and output data distributions. As our approach hinges on the introduction of the re- verse pass, we will show that our approach succeeds in terms of establishing mutual in- formation between the input and the constrained intermediates inside network. More formally, the mutual information I(X;Y) of random variables X and Y measures how different the joint distribution of X and Y is w.r.t. the product of their marginal distributions, i.e. the Kullback-Leebler divergence: T(X;Y) Tishby & Zaslavsky (2015) proposed a mu- tual information plane to analyze the train- ing process, which shows I (X; L) and [L; Y) for each layer L over the course of the train- ing epochs. Points in the resulting graphs are colored w.r.t. the epoch, i.e., initially black, and yellow once the training is finished. Each graph shows the 5 middle and output layers. The MI planes visualize how much informa- tion about input and output distribution is re- tained at each layer, and how these relation- ships change within the network. For regular training, the information bottleneck principle (Tishby & Zaslavsky| 2015) states that early layers contain more information about the in- put (high I(X:L) L) and I(Y: L)). Hence they are often visible at the top-right. Later layers have relationship with the output T(Y; T.) initially, which increases over the course of successful training. Thus, they typically move to the top-left. The intuition behind MI planes is explained in more detail in Appendix A.I We use the same numerical studies as in Shwartz-Ziv & Tishby (2017) as task A, i.e. a regular feed- forward neural network with 6 fully-comnected layers. The inputs consist of 12 binary digits, and outputs are 2 binary digits. Models are trained using cross entropy as base loss function. De- tails of this architecture (and following ones) are given in the Sec. A. At first, we train a base model RR with a full racecar loss. For comparison, we also show the MI planes for Std (a regularly trained model), and Orta (trained with orthogonal constraints (Bansal et al. 2018 We additionally include version RR!, i.e. trained with only racecar loss term X L1 L/1g), which means that only the input is constrained be recovered. Thus, RR represents sim-
(Pxx((xx))) Reg
RR]
HA-T)
10:0)
12t.
Orty
Std,
RED
106
Figure 2: MI planes for the initial models RR RR Std, and Ort,, End points of RR are located the middle of the graphs. But for Std, and Ort T(E; Y) values of every layer are very high, which means more specific features about the given output data set learned by them.
4

---

Under review as a conference paper at ICLR 2020
plified version of our approach which receives constraints that the intermediate results of for- ward and backward pass should match. The resulting information planes are shown in Fig. 2 We can see that the end points of RRA i.., the yellow clusters of points for the final state of the model, are located in the middle part of the graph. This means that all layers successfully encode information about the inputs well the outputs. In contrast, for Stdı and OIta. T(X; L) values of later layers are very low (points near the top left), while [(LY) Y) is high throughout. This indicates that the outputs were successfully encoded, and that increasing amounts of information about the inputs are discarded by Stda and Orta. Hence, both models are highly specialized for the given task. Comparing RRA with RR the yellow end points of the latter are located which indicates that every layer contains similar information about X and Y. It also indicates that path from input to output similar to the path from output to input. The end points of RR on the other hand, are located in a scattered region. I.e., this network has different amounts of mutual information across its layers, and potentially a very different path in each direction. RRA is only constrained to able to regenerate its input, while the full racecar loss for RRA ensures that the network learns features which are beneficial for both directions. This test highlights the importance of the constraints throughout the depth a network in our racecar loss.
centralized region,
As the discussion above fo- cused on phase I, we now turn to analyzing phase II. ie., follow-up training runs with models trained during phase I. Based on RRA RRS Ort, and Stda, we continue training the mod- els only for the original task,
PRix Final (LLe)) 3.8299
Jides Phai 10::33 0.9229
Orhe: fixal Hla 2.511
105.))
Teot Acturayy
0.873
i... RR RR Ortлa and SIdaA. The resulting MI planes are shown in Fig. While all models now fo- cus on the output (yellow points at the top, maximiz- ing [(;; Y), there are differ- ences in the distributions of the yellow points along the axis, i.e. how much MI with the input is retained. We I(X; La) value the end of the training is higher than for StdAA. Ortлa and RRIA (final [(L;; Y) values are shown above each graph). While the final accuracy is high throughout due to the rela- tively simple setup, RRA slightly outperforms the other variants for the original task. We will more clearly demonstrate this positive trait of our racecar training with more complex tasks below.
reur)
(000
Ortaa
MRL
Figure 3: MI plane and accuracy comparisons for continued training (RR)A- RRAA. Stdaa and Ortлa). The RRA model slightly outperforms the other three models, indicating the positive effect of the full racecar training.
that for model RR AA the final
881e [G,:) F2 0.9699
85%, final (l.: 0L9922
Sclae (C,,
1981)
(0:0)
0rSaffnst (2)7))
Stds, final/Ge))
Accurnyy siez
10043
122
31ds
Figure 4: MI plane and accuracy comparisons for task transfer for models RRA,. RR Ortas and Stdaa- RR successfully reuses features from task A for task B. and outperforms Stdg. The regular model Stday yiclds very low performance.
5

---

Under review as a conference paper at ICLR 2020
We now analyze new task, to check whether the model learned specific or general features, and reverse output labels as transfer learning task B. E.g. if the output is [0.1] in the original data set, invert it to [1,0]. Based RR), RR; Ort, and Std train RR RR On and Std for the modified data set. For comparison, we also train a model trained Stdą from scratch. MI planes and accuracy comparisons across most models, StdAB stands out with a very low accuracy. This model from a regular training run has large difficulties to adapt to the new task. Model OrTAB also performs worse than Stdg. RRAB shows the best performance in this setting, confirming the previous MI plane visualizations, and demonstrating that our loss formulation helped to learn more general features from the input data, improving the performance for related tasks such as the inverted outputs.
shown in Fig. 4 While the range in performance is relatively small
5 EXPERIMENTAL RESULTS
We now turn to more complex network structures (CNNs, GANs, Auto-Encoders with different data sets (MNIST, Cifar, smoke, ImageNet) and different tasks (such as classification and synthesis) to show that models trained with our approach succeed in learning very general features that transfer to new tasks.
5.1 DIGIT CLASSIFICATION
The MNIST data set is a commonly used data set for hand writ- ten digit classification (LeCun al., 1998). At first, we train three models for regular MNIST data set classification with racecar loss RR A. a regular model Stdx, and one with orthogo- nal Oit, constraints improving CNN performance. Usually, convolutional layers take up most of the model parameters, we correspondingly compute the racecar loss for the convolutional layers, omitting the fully con- nected layers that would be required for class label inference. To highlight the properties of our algorithm, we show comparisons between and the regenerated l' in Fig. 5 We can see that for racecar training, most of the features from the input are recovered. Trying to invert the network in the same way for regular training run or training with orthogonal constraints largely fails, as the extracted features are extracted according the digit classification and discard information unrelated to this task.
7
task A, one
Figure 5: Comparisons between regenerated inputs. Fltr:: Orta, Std,, RRA and the reference. Only RRA recovers most of the input information successfully.
(Bansal et al. 2018), the latter also aims for
Based on Orta, Std,, RR we continue training to obtain models ORtAA Std and RR3 AA Results are shown in Fig. 6 We repeated these runs 5 times, in order to draw reliable con- clusions. We can see that RRAA outperforms Ortąa and STdAa at the end of training, which indicates that racecar training yields general features that can also improve performance for original task. [Council (2000) illustrated that balanced learn- ing of both general and specific features is more effective for human learning, and our results here are consistent with this intuition.
TestAcouraoy
0061
Orta
RRC
Striee
RR
Shdp
Figure 6: Accuracy comparisons for
As general features more robust than specific features (No- vak et al.) 2018), we investigate a perturbed data set for the transfer task B. We apply Ort,, Std, and RR3 n-MNIST, data set for classification with motion blur (Basu et al. 2017). Performance results are likewise given in Fig. 6 Based on the same CNN structure and parameters, RRA achieves the best performance. This indicates that RRA learned more general features via racecar training than Orta and Std
base task and transfer learning task B. RRA and RRA, achieve the best performance for the base task and transfer learning task, respectively.
5.2 NATURAL IMAGE CLASSIFICATION

---

Under review as a conference paper at ICLR 2020
Tost Aocuraty
Natural images arise in many important application scenar- ios. Hence, we evaluate our approach with the Cifar data set (Krizhevsky al. 2009). At first, we train two models Cifar- 10 data set classification task A, RRA and Stdy. We again continue training to obtain RRA and StdAA, results for which are shown in Fig. [ The racecar training also improves performance for this natural image classification task. For transfer learning task B, we reuse RRA and StdA for Cifar-100 data set classification (RR AB and StdAB). Results are also shown in Fig. 7 Training with the same CNN structure and parameters, StdAB has difficulties adjust- ing to the new task, while our model from the imitial racecar training slightly outperforms the model trained for scratch for B.
0.506
0323
Figure 7: Accuracy compar- isons of task A and task B. RR and R got bestperformance for tast A and B.
5.3 GENERATIVE ADVERSARIAL MODELS
In the previous sections, we employed commonly used data sets and transfers related tasks. Next, we test our approach for a transfer situation with a more challenging transfer from a synthetic single channel data set of synthetic smoke simulations, RBG videos of real smoke clouds. We use GAN and auto-encoder structures for the following models.
At first, we use GAN structure (one generator and one discriminator net- work) for super resolution of the sim- ulation data task A. Std, is trained via regular training, and RRA uses racecar training for the generator net- work of the GAN. Regenerated low resolution results and high resolu- tion outputs comparisons are shown in Fig. g Both RR and Stda can up scale low resolution data times larger to high resolution data very well. But RR can also recover low resolution versions from high reso- lution data. Std, fails in regenerat- ing the low resolution images. This shows that RRA not only focused on generating good features for the super-resolution task, but also preserves information about the in- put.
Imput'
Stela
RR2
ref
Output
Stulg
Figure 8: Regenerated low resolution results and high resolution outputs comparison between Stda, RR and reference. Only RR successfully recovers the input.
We reuse the generator models RR and Stda to train two sets of auto- encoee networks: once for the synthetic smoke data transfer task B1, and as second task B2 for RBG videos of real-world smoke clouds (example frames are given in the appendix). This yields mod- els RRS R and SidAB,,2 respectively. The resulting accuracies, summarized in Fig. show that racecar training performs best for both auto-encoding tasks. This is especially encourag- ing for task B2, as it represents a transfer from fully synthetic training to real-world images. Synthetic data and real-world images have significant dif- ferences in terms of their features. In the worst case, most of the features learned by the regular learning process Std, can not be transferred to the new task. Besides, an inappropriate starting point of the transfer learning task can guide the whole training process towards a wrong di- rection. Instead, a model with more general features such as RR can more easily be reused in new tasks, and can provide a better starting point for learning. This indicates the potential of racecar training to obtain generalizable features from synthetic data sets that can be used for tasks working with real-world data.
MO-D
400
Figure 9: Accuracy comparisons trans- for learning tasks. RR and RR got best performance for task B; and Bz.

---

Under review as a conference paper at ICLR 2020
5.4
VGG19 STYLIZATION
style
Figure 10: Left: stylization test from a natural image to the starry night painting style. Right: stylization test from horse to zebra.
In order to provide qualitative evaluation in complex, visual scenario, we turn to the popular task image styl- ization. We use VGG19 networks (Simonyan & Zisser- man 2014) in the following, and we train two VGG19 networks RR, and StdA with ImageNet data set (Deng al. 2009). Based on these, train RR AA and StdAA, top5 accuracy of which are 77.59% and 75.28%, respec- tively. Consistent with previous test, RR outperforms the StdAa baseline even this complex case for a net-
16
Figure 11: First primary stylization test, RRA did not change specific features in results.
work with 143 million weights. Gaays al. (2016) achieve excellent stylization results, but the results strongly depend on the pre-trained VGG network, i.e., it stylizes and changes structures that are represented by the features the VGG network. Hence, we employ stylization to visualize which features, general or specialized, the two VGG versions focus on.
Gatys
(2016) generate stylized results via optimizing two losses, Latyle and Lecntents which measure style difference and content difference, respec- tively (details are given in the appendix.
To compare the feature extract- runs, we only optimize Lstyle-
ing capabilities between the RR and Std, a first test, we use two simple shapes as source and style images, as shown in Fig. 11 They have the same background and object color. The only difference is the large-scale shape of the object. Thus, plying an style that is identical to the source, we can test whether the features can cleanly separate and preserve the large scale shape features from the ones for the localized style. Comparing the results of RR16 and Stda, the output of RR is almost identical the input, while stylization with Std, changes the shape of the object introducing un- desirable streaks around the outline. This result indicates that RR 16 is able to cleanly encode the shape of the object and preserve during the stylization.
aim for ap-
Stylization comparison from low resolution high resolution. RR 16 re- sults are closer to the high resolution.
Figure 12:
We show more stylization tests in Fig. 10, which optimize both Leontent and Letyle- For the left example of Fig. [0] with Van Gogh style image, both models can generate good stylized results. However, from the results obtained with Std, (orange square area), we can see star within the building, which indicates that this model mixed style and context information in its features. Across our tests we found that models like this one, trained for more specific features, more often performed simple patch-based matches rather than generating meaningful transfers styles. However, the model focuses more general and abstract features, it can perform a better semantic matching and generate improved stylization results. For the right part of Fig. I0 we show test transferring a horse image to a zebra style. RR focuses more general features, such as grass, zebra, and their positional relationships. As a result the model successfully distinguishes foreground from background, and changes the style for both foreground and background objects with only smaller
8

---

Under review as a conference paper at ICLR 2020
mistakes. Stda 's performance is significantly worse, leading to large grass- of the horse.
graks-like like areas within the body
We confirm the capabilities of the VGG model from race- car training with a low- to high-resolution transfer where the goal is purely add detail. Lee., we can compute in place errors w.r.t. high-resolution reference. The result shown in Fig. [2 The VGG model obtained with regular training run yields significantly higher errors, as visible on the right side of the figure.
Surre
RRA can extract more general information from the in- put, which is helpful with generating semantically mean- ingful results. However, we found that this property can also confuse the model some specific situations. For in- stance, shown in Fig. 13 at first, R 16 can distinguish foreground and background and generate better styliza- tion results than Std,. However, RRA also stylizes the shadow of the horse. This also indicates that R 16 fo-
Figure 13: RR generates zebra pat- tern in the horse's shadow, but yields gen- erally improved results compared to the reg- ular transfer in Stda.
cuses more on general features. As a consequence, all horse shape objects are extracted and then stylized with zebra textures, such as the shadow in Fig. [3 However, generally think that this can provide potentially helpful information in other situations, e.g., for classification tasks. Overall, the quality of the racecar transfer also still surpasses the regular version.
CONCLUSION
We have proposed novel training approach for improving neural network generalization by adding a constrained reverse pass. We have shown for range of tasks that this yields networks with more general features that more casily transfer to new tasks, and even outperform basclines for the original task. Our training approach is very general, and imposes no requirements regarding network structure or training method. As future work, we believe it will be very interesting to evaluate our approach for other architectures from the vast zoO of existing work, e.g., we are particularly interested in analyzing recurrent structures such as LSTMs and GRUs (Hochreiter Schmidhuber) 1997 Cho et al. 2014).
REFERENCES
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint 1808 04730, 2018.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular- izations in training deep cnns? In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 4266-422777 Curran Associates Inc., 2018.
Saikat Basu, Manohar Karki, Sangram Ganguly, Robert DiBiano, Supratik Mukhopadhyay, Shree- kant Gayaka, Rajgopal Kannan, and Ramakrishna Nemani. Learning sparse feature representa- tions using probabilistic quadtrees and deep belief nets. Neural Processing Letters, 45(3):855- 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
National Research Council. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. The National Academies Press, 2000.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.

---

Under review as a conference paper at ICLR 2020
Hui Ding, Shaohua Kevin Zhou, and Rama Chellappa. Facenet2expnet: Regularizing deep face recognition net for expression recognition. In 2017 12th IEEE International Conference on Auto- matic Face & Gesture Recognition (FG 2017), pp. 118-126, IEEE, 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv 08803, 2016.
Lixin Duan, Dong Xu, and Ivor Tsang. Learning with augmented features for heterogeneous domain adaptation. arXiv preprint 1206v 4660, 2012.
M-L Eckert, Wolfgang Heidrich, and Nils Thuerey. Coupled fluid density and motion from single views. In Computer Graphics Forum, volume 37, pp. 47-58. Wiley Online Library, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and patter recog- nition, pp. 2414 2423, 2016.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger Grosse. The reversible residual net- work: Backpropagation without storing activations. In Advances in neural information processing systems, pp. 2214 2224, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kasthurirangan Gopalakrishnan, Siddhartha K Khaitan, Alok Choudhary, and Ankit Agrawal. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pave- ment distress detection. Construction and Building Materials, 157:322-330, 2017.
Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl. The scientist in the crib: Minds, brains, and how children learn. William Morrow & Co, 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Hönn-Henrik Jacobsen, Amool Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprini arXiv: 1802.00088 2018
Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewiez. Rotation invariant spherical harmonic representation of 3 shape descriptors. In Symposium on geometry processing, vol- ume 6, pp. 156-164, 2003.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Brian Kulis, Kate Saenko, and Trevor Darrell. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In CVPR 201 /, pp. 1785-1792. IEEE, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2212-222222 1998.
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255 255,, 2016.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv: 1802. 2018.
10

---

Under review as a conference paper at ICLR 2020
Peter Prettenhofer and Benno Stein. Cross-language text classification using structural correspon- dence learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp.
1118 (118-1127 1127, 2010.
Hariharan Ravishankar, Prasad Sudhakar, Rahul Venkataramani, Sheshadri Thiruvenkadam, Pavan Annangi, Narayanan Babu, and Vivek Vaidya. Understanding the mechanisms of deep transfer learning for medical images. In Deep Learning and Data Labeling for Medical Applications, pp- 188-196. Springer, 2016.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: I 770.00080000 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409.1556, 2014.
Jos Stam. Stable fluids. In Siggraph, volume 99, pp. 121-128, 1999.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Lisa Torre and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp.
IGI Global, 2010.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
Lijing Zhang, Yao Lu, Ge Song, and Hanfeng Zheng. Re-enn: Reverse connected convolutional neural network for accurate player detection. In Pacific Rim International Conference on Artificial Intelligence. pp. 438 088-446. Springer, 2018.
Joey Tianyi Zhou, Sinno Jialin Pan, Ivor W Tsang, and Yan Yan. Hybrid heterogeneous trans- fer learning through deep learning. In Twenty-eighth AAAI conference on artificial intelligence, 2014a.
Joey Tianyi Zhou, Ivor W Tsang, Sinno Jialin Pan, and Mingkui Tan. Heterogeneous domain adap- tation for multiple classes. In Artificial Intelligence and Statistics, pp. 1095-1103, 2014b.
Yin Zhu, Yuqiang Chen, Zhongqi Lu, Sinno Jialin Pan, Gui-Rong Xue, Yong Yu, and Qiang Yang- Heterogeneous transfer learning for image classification. Twenty-Fifih AAAI Conference on Artificial Intelligence, 2011.
A
APPENDIX
Here we will present more details, such as data set information, network structures and training parameters, for all of our mentioned tests above: Mutual Information S.c. MNIST classification Sec. A.2 Cifar Sec. A.3] smoke Sec. A.4 and VGG Sec. A.5 We will use (,), D(k, L, to represent convolutional and deconvolutional operations, respectively, and fully connected layers are noted with Fl). where k, s denote kernel size, output channels and stride size, respectively. Bias of CNN layer is denoted with 1/O(z) denote /nput/outpul and their dimensionality given by Ir denotes the input of reverse pass network. tanh, relu, Irelu denote corresponding activation functions, where we typically use leaky tangent of 0.2 for the negative half space. UP, M and pooling with 2 x 2 filters and stride 2. and batch
BN denote 2x neireot-n up sampling, normalization, respectively. All performance numbers were measured on a Nvidia GeForce GTX 1080 Ti GPUs and Intel Core 17-6850K CPUs.

---

Under review as a conference paper at ICLR 2020
A.1 MUTUAL INFORMATION TEST
Below we will introduce more details about tests in Sec, For the numerical task (Shwartz-Ziv & Tishby 2017). input variable X are 12 binary digits that represent 12 uniformly distributed points on a 2D sphere, and this task is about binary decision rules which are invariant under ((3) rotations of the sphere. X has 4096 different patterns, and they are divided into 64 disjoint orbits of the rotation group. which form a minimal sufficient partition/ partition/statisties for spherically symmetric rules (Kazhdan al. ).03). To generate input-output distribution P(X, Y). Shwartz-Ziv & Tishby (217)) applied
stochastic rule ply 1|x) (f(() 0). (I E X.y E Y). where V is a standard sigmoidal function (u) = 1/(1 + exp(--r)). Shwartz-Ziv & Tishby (017)) use a spherically symmetric real valued function of the pattern f(x) (evaluated through its spherical harmonics power spectrum Kazhdan et al., 2003) and compared it to a threshold 6, which was selected to make pyy 1) = E.Ply Ply = [() z) ~ 0.5, with uniform p(z). Y is high enough to keep the mutual information T(X;Y) R 0.99 bits. 80% of the data (3277 data pairs) are used for training and rests (819 data pairs) are used for testing. The forward and reverse pass structures of the fully connected neural networks are in Table 7 Hyper parameters used for training are listed in Table
All layer are used in racecar loss. All layers in the RRA and Stda are reused for training RR AAAAB and StdAwAB-- For ORta, we used the Spectral Restricted Isometry Property (SRIP) regularization Bansal et al. 2018
SSUP = (W - I)
(2)
where W is the kernel; I denotes an identity matrix; 3 represents the regularization coefficient; [[W: o(W) sup=ez" ,=#0 ||=|| denotes the spectral norm of W. Details about all models' accuracy are shown in and Table
Mutual information (MI) plane is powerful tool for neural networks analysis but admittedly not very intuitive. Here will use MI plane of Std, in Fig. 2 as an example to illustrate MI plane in details. From Table 7 we can see that neural network in this numerical task has 5 middle layers (I1-s) and
19908
Std_A
0.8
06
Epechs
output layer (Lo). The X axis of the MI plane represents the quantity / ((;L), L). i.e the mutual information between input variable X and output of each layer L. The Y axis of the MI plane represents [[L; Y). the mutual information between out- put of each layer L and output variable Y. Be- sides, every point in the graphs is colored w.r.t. the training epochs, i.c., initially black, and yellow once the training is finished. Hence, we can see six lines changing from black to yellow in figure 14 of the updated version. According to the information bottleneck principle [5), the outputs of the early layers contain more information from the input, which means high value for T(X; L) and YL; We can see that early layers L2 3 located in the top right part of the graph. For later layers, such as Lp. parameters of Le are randomly initialized before training, there are almost no direct relationships between the output of Le and X or Y, which means low values of I (X; L) and (LL Once the model is well trained, Ls of Std, able to generate data which has the same distribution with Y. so [L: Y) has increased and moved to the top left corner of the graph. The line of LG starts from the bottom left part of the graph, and moves to the top left part during training. For RRC in figure 2, the 1.2 constraint is explicitly applied to decrease difference between L and L', and the output is pushed to recover lost informa- tion of the input, so I(X; L) of later layers is increased, and I(X; L) of early layers is decreased to make the task easier.
02
108:0
Figure 14: MI plane of Sida an example.
A.2 MNIST CLASSIFICATION
This section gives details for Sec. 5.1 The regular MNIST data LeCnn et al. 1998) contains 55A images for training and 10k images for testing. For the n-MNIST motion blur data set ct al. (017). there are 60k images for training and 10k images for testing. Image size of them are all 28 x 28. Example data from MNIST and n-MNIST with motion blur are shown in Fig. 15 Details
12

---

Under review as a conference paper at ICLR 2020
about the forward and reverse pass network structures shown in Table Hyper parameters
listed Table
7
Figure 15: Left: data from MNIST; Right: data from n-MNIST with motion blur.
All 3 convolutional layers are used for racecar loss and all layers in RRA, StdA and Orta are reused for training RR3 RRAAAB SIdAnAAB and OrtAA/AB- Example training processes of the MNIST tests are shown in Fig. 16 We can see that racecar loss increases the task difficulty, so RRA yields lower performance and longer training time than StdA the first phase. In the second phase RRA out- performs Stda and Ort, in both task A an B, which indicates that racecar training is helpful with general feature extraction. For Orta, we also use equation as orthogonal regularization. All models' accuracy results are listed in Table3ann Table
800UENy
BCOUP9CY
BCOJraE
0.967
1.00
0.978
0.985
0.974
0.070
0.880
0.966
0.060
0.581
Why
0.840
Figure 16: Left: training processes of RRA (blue, accuracy: 0.9810, cost: 5.675 scconds/epnch), Stda (orange,
accuracy: 0.9827, cost: 3.522 seconds/epach) and Ort, (red, accuracy: 0.9792, cost: 4.969 seconds/epoch). Middle: training processes RRIA (blue). Stdaa (orange) and Ortдa (red). Right: training processes RR (green), StdAu (pink), OrlAz (blue), and Stdy (grey).
A.3 CIFAR TEST
test_aocuracy
test_acouracy
test_acouracy
0.800
0 1.000
0.400
0.200
200
0.00
Figure 17: Left: training pracesses of RR (blue, accuracy: 0.5784, cost: 64 seconds/epoch) and Std, (orange, accuracy: 0.8272, cost: 63 seconds/epoch). Middle: training processes of RR (green) and Stdaa (blue). Right: training processes of RR (blue), Stdau (pink) and Stdg (green).
All 13 convolutional layers are used for racecar loss and all layers except the last fully connected layer (because of different output size) in RR and Std, reused for training RR13 StdANAB- Example training processes of the Cifar tests are shown in Fig. 17 In phase II, the model RR33 outperform StdA both task and We show details about all models in Table
AAAAB and
A.4 SMOKE DATA SETS TEST
In this section, we will introduce details for Sec. 5.3 The smoke simulation data was generated with standard fluid solver (Stam 1999] with MacCorack advection and MiC-pprconditioned CG
13

---

Under review as a conference paper at ICLR 2020
solver via the mantaflou library. We generated 20 simulations with 120 frames for every simu- lation. 10% of the data was used for training. Smoke inflow region, inflow velocity and buoyancy force were randomized to produce varied data. The low resolution data was down-sampled from the high-resolution data by a factor of 4. Data augmentation, such as flipping and rotation was used in addition. The smoke capture data set contains 2500 smoke images (Eckert et al.) 2018), and again used 10% as training data set. The forward and reverse pass structures of the network are in Table 13 Hyper parameters are listed in Table [4] Note that inputs of discriminator contain high resolution data (64, 64, 1) and low resolution (16, 16, 1), which is up up-sampled to (64, 64, 1) and concatenated with high resolution data.
All generator layers are involved in racecar loss. All 6 layers of RR and Stda are reused for training RRAB, AB, and StdAB, but only 5 layers are reused for training R AB, and StdAB, because of different data size. The following results give further details for the auto-encoder transfer learning task AB1 which uses synthetic, i.e., simulated fluid data. Example training processes of RR AB, (orange), StdAB, (blue) and model trained from scratch Stdg, (red) shown in Fig. 18 RRAB, achieved the lowest L2 loss after training. Example outputs of RR An. StdAB, and Std are shown in Fig. 19 becomes clear that model RR gives the best performance across these models.
L2 loss
33530m
25522-4
1353c
1 13222
Figure 18: Training processes of RR, R, (orange), StdAr, (blue) and Stdg, (red). The green inset shows the final loss values.
Stda,
Stdar,
RR
re refereen ference
Figure 19:
Example outputs comparisons hetween RR AB, StdAB, Stdy, and reference. We can see that RRS, works better than StdAa,, while Stdp, failed for this task producing a mostly black image.
We similarly illustrate the behavior of the transfer learning task AB, for images of real-world flu- ids. This example likewise uses an auto-encoder structure. Example training processes of RR AB Atd, and model trained from scratch Std,, are shown in Fig. 20 RR A yields the best perfor- mance at the end of training. Example output comparisons of RR StdAy, and Stdp, are shown in Fig. 21 We can see that error RR is lower than Std, and Std Detail comparisons between different models are shown in Table
14

---

Under review as a conference paper at ICLR 2020
joss
5.00201
1100201
2000018
2.00048
19.08
Figure 20: Training processes RRA (orange), St (blue) and Std (red). The green insct shows the final loss values.
RRAB. PRRZS :65.26
StdaR.
Std 02
ref eeeeece
EsTdAD, 57.008
Estd pz :75.81
Figure 21: Example outputs mean absolute error comparisons between RRAB, Stdae,, Stda, and reference. We see that error of RRAS, is lower than Std and Std
A.5 VGG TEST
We now give details of the tests in Sec. 5.4 For the ImageNet data set (Deng al. 2009). 1281167 images of 1000 classes are used for training, and 50k images are used for testing. Image size is 224 x 224. The forward and reverse pass of VGG19 are Table 16 Hyper parameters are listed in Table 15
All convolutional layers are used racecar loss. To speed up training process of RR16 we firstly train model without racecar loss for 6 epochs with batch size 64, as regular training. And then we reuse this model for training RR and StdA with batch size 24. Example training processes of RR16 and Std, are shown in Fig. 22 while those of 16 and Stdaa are shown Fig. 23
Next, we will give additional details for the stylization tests. Gatys et al. (2016) use Liotal Locntent + &istyle for stylization optimizations, where and are coefficient factors. Loontent used to calculate content difference between source image and generated image g, shown in equation g
Ioontend (p. g. Em. (Fm.. - Fm.....2
(3)
15

---

Under review as a conference paper at ICLR 2020
Figure Top5 accuracy of RRA (red, accuracy: 0.6673, cost: 0.636 second/batch) and Std, (blue, accuracy: 0.7324, cost: 0.308 second/batch).
ax
000
Figure 23: Top5 accuracy of RR (grey) and Stdaa (blue).
where F.... is y's feature representation of the th filter position in layer t. Estyle is used to calculate style difference between style image a and generated image g. as shown in equation The style loss can be written as:
Gm..n. -E, EEP pm..tt Fn....
E2 IN;M?
Em,a (Gim.nd Gm).
(4)
Lstylel (a.9) We=
where wa is are the weighting factors for layer i; G is the Gram matrix; N, denotes filter numbers of layer t, and M, is the dimension of layer t's filter; T denotes the number of layers included in the style loss.
16

---

Under review as a conference paper at ICLR 2020
Table
Model accuracy of MI base tests
training
Ofta
RRA
RRA
Std,
RRAA
R
StdAa
0.976
0.682
0.767
0.973
0.973
0.855
0.997
0.956
2
0.944
0.75
0.779
0.986
0.954
0.909
0.99
0.973
0.975
0.731
0.855
0.964
0.979
0.875
0.99
0.986
0.993
0.923
0.855
0.981
0.994
0.984
0.995
0.965
5
0.965
0.405
0.852
0.94
0.967
0.965
0,999
0.986
Avg.
0.971
0.707
0.822
0.967
0.973
0.938
0.994
0.973
Std. Dev.
0.018
0.187
0.045
0.018
0.015
0.056
0.004
0.013
Table
Model accuracy of MI transfer task tests
training
Ortab
RRAB
R
StdAB
Stdp
0.978
0.914
0.996
0.136
0.966
2
0.948
0.956
0.136
0.958
3
0.976
0.961
0.993
0.136
0.984
4
0.994
0.97
0.996
0.136
0.974
5
0.968
0.959
0.998
0.136
0.989
Avg.
0.973
0.956
0.997
0.136
0.974
Std. Dev.
0.017
0.022
0.003
0.013
Table
Model accuracy of MNIST base task tests
training runs
StdA
RRA
Ortдa
Stdaa
RRA
1
0.9792
0.9827
0.981
0.9819
0.9854
0.9856
2
0.8322
0.9815
0,9824
0.982
0.9837
0.9851
0.9511
0.9815
0,9828
0.9792
0.983
0,9862
4
0.8841
0.9817
0,9816
0,9814
0,9842
0.9867
5
0.8266
0.9829
0,9816
0,9816
0.9846
0,9861
Avg.
0.895
0.982
0.982
0.981
0.984
0.986
Std. Dev.
0.0689
0.0007
0.0007
0.0011
0.0009
0.0006
Table 4: Model accuracy of MNIST transfer task tests
training times
Ortar
SidAr
R
Sida
0.949
0,956
0,9612
0.8714
0.8629
0.9458
0.954
0.9657
0.9458
0.9543
0.9648
0,9659
4
0.9495
0,9559
0,9604
0.8789
0.8564
0.9466
0.9649
0.9638
Avg.
0.913
0.952
0.961
0.929
Std. Dev.
0.0485
0.0051
0.0044
0.0494
Table
5: Model accuracy of Cifar tests
training
Stda
RR
Stdaa
RR
SIdAD
RR
Stdp
runs
1
0.8263
0.5784
0.8351
0.869
0.2063
0.2602
0.2581
2
0.7868
0.5553
0.7936
0.8538
0.1838
0.2687
0.2659
3
0.7729
0.7686
0.7718
0.8449
0. 0.1795
0.2803
0.2386
4
0.865
0.7382
0.8472
0.8692
0.1738
0.2969
0.2418
5
0.781
0.6775
0.7787
0.8418
0.1704
0.2733
0.2621
Avg.
0.806
0.664
0.805
0.856
0.183
0.276
0.253
Std. Dev.
0.0387
0.0945
0.0340
0.0130
0.0141
0.0138
0.0123
Table 6: Model 1.2 loss of smoke tests. Results for B2: x x10"
training runs
StdAp,
R AB
Std
Std
RR TRAB
Stdæ,
22883
211.9
31911
213
2.95
6.13
2
493.3
139
22291
2.37
1.79
2,66
3
1828
210.2
31911
11.4
2.32
3.43
Avg.
8401.43
187.03
28704.33
75.6
2.36
4.07
Std. Dev.
12559.15
41.61
5554.11
119
0.578
17

---

Under review as a conference paper at ICLR 2020
Table 7: Forward reverse pass the neural network
Forward pass (294 weights):
I(12) tanh((C((L0) + b,) waaaaaaa(aiia bz) tooh((F((())))))) by) tanh(FC(4)+
(a()(FF((3 + ba) (onh( + -
Reverse pass: O(2) by - Lenb(FC(()) by - Lanh(FFC(4)) b4 lanh(FC((5) - by Lank(FFC(T)) - bz + tenh(FC(10)) + tanh(FC(12)) + (12).
Table 8: Hyper parameters of MI tests
Batch size
512
Learning rate 0,0004
A
1E 2
Training Epochs
20000
RRAAaA SidaawAB: 40000 Stda
Table 9: Forward and reverse pass network of MNIST Classification tests
Forward pass (38645 weights): 7(28, 28, 1) (C3 64, b1) MP relu b2) - MP re..1,1111 by) T, FC((1) O(10)
Reverse pass: by
relu D(3, 64, 1)) UP b2 10
Table 10: Hyper parameters of MNIST Classification tests
Batch size
Ai-a
1E 5
0.001 for RRA, Std, and Orta:
Learning rate
0.0001 for AAAB Std and OrtлNAв
100 for RR StdA and Orta;
Training Epochs
400 for RR Stdaa and Ortas:
700 for RRA StdAr and OrtaB
Table 11: Forward and reverse pass of neural network Cifar tests
Forward pass:
1[32, 32, +
relu(BN(C(3,,, 12) +
(M 256, bs))
relu(BNN 256, 1) +b) redu(BN(Ci3.52221)) C 512, bl))
512, bx))
re3u
(C3 +b))
b2))
- +cl((DN((((A.2222))+m)m)) ( bu))
+ (NNC(BN((CC(4000 + b15))
Reverse pass:
b13 relu(BN(D(3,51)) 512, 1))) b12 relu(BNN(((35512.1)) 11 rel 1)))
+ UP 0o + www.eeuBBB (DD( 512, 1))) bo reN
- h8 relu (BN ( 256. 1)))
+ UP - by relu 256, 1))) by + relu 256, 1))) by + relu(BN 128, 1)))
+ UP - bu reNu 128, 1))) by relu 64, 1))) + UP b2
relu(BNN(((3641 1))) (BN 1))) (32,32.3).
Table 12: Hyper parameters of Cifar
Batch size
200
A1-18
1E -
0.1 (0 epochs);
Learning rate
0.01 (81 to epochs);
0.001 (after 120 epochs)
180 for RR13 MAA and Stdans:
Training Epochs
50 for RR13 and Stdawa
18

---

Under review as a conference paper at ICLR 2020
Table 13: Forward reverse pass network smoke tests
Generator forward pass: T(16, 16, 1) relu (C,5, 64, 1) + bi) -P relu 128, 1) bb) +P (C5, 128, 1) + ba) + relu(C((, 64, 1) + b4) + +
relu( b②) 0611,1,111 I.-
Generator reverse pass:
- he (eeu(D(a 32, 1)) by + -1)
rel, 128,
relu 128, 1))
.MP
relu ((5, 64, 1)) MP
16.
Discriminator:
[(61, 61, + Irelu (5 32, 1) 1) bi)) Irelu (CN 1) bb))
Irelu (C 128, 1) ba)) Irelu(BBN(CC)) (N, 256, 1) b.)) - FC(() + bs 0(1).
Table 14: Hyper parameters smoke tests
Batch size
64
Learning rate 0.0002 Å
0.1
Training
40000 for RRA and Stdı;
Epochs
1000 for RRAZ, RRA StdAe, and Std.r
Table 15: Hyper parameters of VGG19 training
A1-16
IE I0
0.01 (0 to 7 epochs); 0.001 (7 to 10 epochs);
Learning rate
0.0001(10 to 15 epochs); 0,00001 (after 15 epochs)
36 epochs for RR and Stda;
Training Epochs
22 epochs for A
and Stdaa
Table 16: Forward and reverse pass of VGG19 network
Forward pass:
b2) MP relu(C(
reduCCCC 128, 1) + bs)
relu(C(3, 12 - MP r
relu(C33, 256, 1) + b6)
relu(C(3,256) 25), - -b rel 512, by)
relu(C(3,512) (12, +1))bbm) - =n)
b1) MP
relu(C(3, b1s) + relu bın)
(112, +16)) + r relu
(18)
relu(FCC(100) ( 0(1000).
Reverse pass:
- bis (=
(122(())))111) 1 - 1)) b13
bıo rreu([(335512 (12,1) - -by + =rcb nel 256, 1)) b7
+ relu(D(3, 256, 1)) - by + - UP bu
re3,(D(3, 128, 1)) bg ()) ( -bi - relu(D(3,
1' (224, (24, 3).
19


Reviews:
[{"review_id": "rkxHedg05S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a scheme for training layered feedforward neural networks with backwards accumulation of gradients. In the proposed scheme, the intermediate activations during a forward pass are constrained, using an L2 norm penalty, to be close to the activations of a model that inverts the operations of each layer and transforms the data in reverese order (from output to input). The second, inverse, network shares all parameters with the feedforward network.\n\nThe paper hypothesizes that such training, which they call racecar training, results in features that are transferable between tasks; i.e. using racecar training to learn the features, then applying regular training on a novel task. To support this hypothesis, the paper provides an empirical analysis of the mutual information between inputs and intermediate features, and between intermediate features and outputs (as proposed in the information bottleneck literature). Under this analysis, the paper shows that using racecar training the intermediate layers contain less information about the outputs than standard feedforward training. The paper states that this makes the features learned by racecar training better for transfer: features that enable the network to achieve high predictive accuracy on a particular tasks but that carry little information about the output distribution, thus less specialized. The paper continues this analysis to the transfer setting, first to the same initial task, and then the a new task. In both cases one of the variants of the proposed method appears to produce higher accuracy than standard pre-training.\n\nI'm inclining to reject this paper given that the results on the main hypothesis (i.e. transferability of features) seem to provide only marginal improvement, and we have no idea about the repeatability of the results ( how many times did the authors run the experiments for figures 3,4,6,7,9? What's the spread of the results? Are these results significant?). The results on style transfer and super resolution are promising, but these are only illustrative examples: these results do not provide insights on how well the method works in general, or when does it fail.\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1lfbmsKtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to learn general and reusable features for transfer learning tasks. The authors propose a training paradigm called Racecar Training. The core idea of it is to operate a reverse pass for the network. The authors use mutual information to analyze the network for its improved generalizing capabilities. They also conduct experiments on classification, regression and stylization to validate their method’s effectiveness.\n\nPros:\nThe reverse pass idea is similar to auto-encoder paradigm that discards redundant information and only save the essential low dimensional one by comparing the original data and the decoding data. The general feature the authors mentioned is like the essential low dimensional information, which is reasonable.\n\nCons:\n1.\tI think the main drawbacks of this paper is that the authors make a poor presentation. The authors talk about learning general features with which the model can use on new task in the title, introduction and even the whole paper. However, there are no explicit general features learned during the learning procedure. They only perform the reverse pass when learning in the original task. Even the structure of networks does not change at all. The general and reusable feature is only an explanation of the improved performance. I think the authors should change this explanation to a more convincing one. For example, the network may learn general and reusable weights or parameters since it is the model learned that will be applied to new tasks instead of the features. \n2.\tThe analysis by mutual information makes the paper hard to follow. The figures such as figure 2 are so confusing. There are so many points and lines in each picture. What do they mean? What are the x-axis and y-axis? The authors also do not explain what the meaning of different mutual information are. \n3.\tThe symbols are chaotic. The authors explain “RR^3” means n=1 in equation 1. Then what does n equal to in RR^1? The authors explain “AB” means the model was trained for task A during phase I, and is then trained for task B as transfer in phase II. Then what does AA/AB in “Std_{AA/AB}” mean? \n4.\tIn the last paragraph of page 3, the authors mention orange color. However, there are only green, blue and yellow color in Figure 1.\n5.\tIn experiments, I do not see obvious advantage of the proposed method. For example, in Figure 6, the test accuracy of Std_{AA} is 0.9842 while that of RR^3_{AA} is only 0.9859. In Figure 7, the test accuracy of Std_{AA} is 0.8377 while that of RR^13_{AA} is 0.8711. The transfer tasks (referred as AB) also show minor advantages. With minor advantages but increased requirements for memory and additional computations (e.g., 61.13% slower per epoch for the MNIST tests mentioned by the authors), the proposed method shows very limited values.\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Both reviewers (we apologize for the lack of a 3rd review) did not feel the paper should be accepted. The rebuttal offered did not change the reviewer scores. So the paper cannot be accepted unfortunately. But the authors should use the feedback to improve their paper and resubmit.
Decision: Reject