Reviving Autoencoder Pretraining
Authors: You Xie
Nils Thuerey
Venue:  (2021)
Keywords: unsupervised pretraining
greedy layer-wise pretraining
transfer learning
orthogonality

Abstract: The pressing need for pretraining algorithms has been diminished by numerous advances in terms of regularization, architectures, and optimizers. Despite this trend, we re-visit the classic idea of unsupervised autoencoder pretraining and propose a modified variant that relies on a full reverse pass trained in conjunction with a given training task. We establish links between SVD and pretraining and show how it can be leveraged for gaining insights about the learned structures. Most importantly, we demonstrate that our approach yields an improved performance for a wide variety of relevant learning and transfer tasks ranging from fully connected networks over ResNets to GANs. Our results demonstrate that unsupervised pretraining has not lost its practical relevance in today's deep learning environment.

Under review as a conference paper at ICLR 2021
REVIVING AUTOENCODER PRETRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
The pressing need for pretraining algorithms has been diminished by numerous advances in terms of regularization, architectures, and optimizers. Despite this trend, re-visit the classic idea of unsupervised autoencoder pretraining and propose a modified variant that relies on full reverse pass trained in conjunction with given training task. We establish links between SVD and pretraining and show how it can be leveraged for gaining insights about the learned structures. Most importantly, we demonstrate that our approach yields an improved perfor- mance for wide variety of relevant learning and transfer tasks ranging from fully connected networks over ResNets to GANs. Our results demonstrate that unsu- pervised pretraining has not lost its practical relevance in today's ' deep learning environment.
INTRODUCTION
While approaches such as greedy layer-wise autoencoder pretraining (Bengio et al., [007] Vincent al.) 2010 Erhan al.) 2010) arguably paved the way for many fundamental concepts of today's methodologies in deep learning, the pressing need for pretraining neural networks has been dimin- ished recent years. This was primarily caused by numerous advances terms of regularization Hanson & Pratt 1989, Weigend et al. 1991), network architectures (Ron- He al. 2016 Vaswani et al. 2017), and improved optimization algorithms Loshchilov & Hutter 2017 Reddi al. 2019.. Despite these advances, train- ing deep neural networks that generalize well to a wide range of previously unseen tasks remains a
(Srivastava et al. 2014;
neberger et al. 2015
(Kingma & Ba
2014
fundamental challenge (Neyshabur al. 2017 Kawaguchi al. 2017 Frankle Carbin 2018).
Inspired by techniques for orthogonalization (Ozay & Okatani 2016 Jia et al. 2017 Bansal et al., 2018), we re-visit the classic idea of unsupervised autoencoder pretraining the context of versible network architectures. Hence, we propose a modified variant that relies on a full reverse
a) Texture-Shan Shape Classification
c) CIFAR 10 (T)
Prediction (1)
sirplano
Classification image
224
Ax
0.90
100
0:332
Stvf
Selus
1.902
200
0.134
3.1
b) Mutual Information
Digt
0
2
3
4
6
6
8
Rotation
L
(
/
/
Thicknasa
0
Ne fermos
Our pretraining (denoted as RR) yields improvements for numerous applications: a): For difficult shape classification tasks. it outperforms existing approaches (Stdrs, Ortys, Prers): the RRTs model classi- fies the airplane shape with significantly higher confidence. b): Our approach establishes mutual information between input and output distributions. c): For CIFAR classification with Resnetl RRcw yields sub- stantial practical improvements over the ofate-of-the-at art. d): Learned weather forecasting has strictly limited real-wor weal-world data: our pretraining yields improvements for pressure (Z500, zoomed in regions shown above), atmospheric temperature (T850) as well as ground temperature (T2M).
Figure 1:

---

Under review as a conference paper at ICLR 2021
pass trained in conjunction with given training task. A key insight is that there is need for "greediness", ie.. layer-wise decompositions of the network structure, and it is additionally bene- ficial to take into account specific problem domain at the time of pretraining. We establish links between singular value decomposition (SVD) and pretraining, and show how our approach yields an embedding of problem-aware dominant features in the weight matrices. An SVD can then be leveraged to conveniently gain insights about learned structures. Most importantly, we demonstrate that the proposed pretraining yields an improved performance for variety of learning and transfer tasks. Our formulation incurs only a very moderate computational cost, is very easy to integrate, and widely applicable.
The structure of our networks is influenced by invertible network architectures that have received significant attention in recent years (Gomez et al. 2017 Jacobsen et al., 2018 ZZhng et al. 2018a). However, instead of aiming for bijective mapping that reproduces inputs, we strive for learning a general representation by constraining the network to represent an as-reversible-as-pen possssill process for all intermediate layer activations. Thus, even for cases where a classifier can, e.g., rely on color for inference of an object type, the model is encouraged to learn a representation that can recover the input. Hence, not only the color of the input should be retrieved, but also, e.g., its shape. In contrast to most structures for invertible networks, our approach does not impose architectural restrictions. We demonstrate the benefits of our pretraining for variety of architectures, from fully connected layers to convolutional neural networks (CNNs), over networks with and without batch normalization, GAN architectures. We discuss other existing approaches and relate them to the proposed method in the appendix.
Below, we will first give an overview of our formulation and its connection singular values, before evaluating our model in the context of transfer learning. For regular, i.e., a non-transfer task, the goal usually to train a network that gives optimal performance for one specific goal. During a regular training run, the network naturally exploits any observed correlations between input and output distribution. An inherent difficulty in this setting that typically no knowledge about the specifics of the new data and task domains is available when training the source model. Hence, it is common practice to target broad and difficult tasks hoping that this will result in features that are applicable in new domains (Zamir et al., 2018 Gopalakrishnan al. 2017: Ding et al. (017). Motivated by autoencoder pretraining. we instead leverage a pretraining approach that takes into account the data distribution of the inputs. We demonstrate the gains in accuracy for original and new tasks below for a wide range of applications, from image classification to data-driven weather forecasting.
METHOD
With ttate-of-the-ar methods, there is no need for breaking down the training process into single layers. Hence, we consider approaches that target whole networks, and especially orthogonalizattion regularizers as starting point (Huang et al. 2018). Orthogonality constraints were shown to yield improved training performance in various settings (Bansal et al. 2018). and can be formulated as:
Lon Σ MM - 1||²"
i.e., enforcing the transpose of the weight matrix Mn, E T for all layers m to yield its inverse when being multiplied with the original matrix. T denotes the identity matrix with =
= ), ei,, denoting the juh column unit vector. Minimizing equation | i.c. M - I = (e), ...e."
is mathematically equivalent to:
MT.Meeee - - - 0, - 1,2, ...y Sm
(2)
Mn.) sin and e/2 as eigenvectors of MT Mm, with eigenvalues of 1. This formu- lation highlights that equation 2 does not depend on the training data, and instead only targets the content of Mm. Inspired by the classical unsupervised pretraining, we re-formulate the orthogonal- ity constraint in a data-driven manner to take into account the set of inputs Dm for the current layer (either activation from previous layer the training data D1), and instead minimize
with
LRR
Σ m=l
(M Mm di ==m Σ (MM,M, - ))
(3)
m=l
2

---

Under review as a conference paper at ICLR 2021
where d'n, E Dm C R Due to its reversible nature, will denote approach with RR subscript in the following. In contrast to classical autoencoder pretraining, we are minimizing this loss jointly for all layers of a network, and while orthogonality only focuses on Mn our formulation allows for minimizing the loss by extracting the dominant features of the input data.
Let q denote the number of linearly independent entries in Dm, i.e. its dimension, and t the size of the training data, i.c. [Dmll t, usually with < t. For every single datum dm i = 2,,,.... t,
equation 3 results in
MM - d 0.
and hence di are eigenvectors of M.... with corresponding eigenvalues being 1. Thus, instead of the generie constraint Mm = / that completely agnostic the data at hand, the proposed formulation of equation Ais aware of the training data, which improves the generality of the learned representation, we will demonstrate in detail below.
As by construction, rank(Mm) =r r & min ", the SVD of Mm yields:
Un, = (u u U u
E X R
Mm = Um VT. mi with
(5)
Vn = (v v2 V..... Vn v
with left and right singular vectors in Um and Vm. respectively, and Em having square roots of the r eigenvalues of M Mm on its diagonal. ut, and v (!....., are the eigenvectors of Mn M and M M,,,,,,, Mm. respectively (Wall et al. (203)). Here, especially the right singular vectors in V are
?)
important, as they determine which structures of the input are processed by the transformation Mn- The original orthogonality constraint with equation yields unit vectors ci, as the eigenvectors of Mm. Hence, the influence of equation 2 on Vm. is completely independent of training data and learning objectives.
Next, we show that LRR facilitates learning dominant features from given data set. For this, we consider arbitrary basis for spanning the space of inputs Dm, for layer Let B, (w",, ... w4, denote a set of orthonormal basis vectors obtained via a Gram- Schmidt process, with t g rr, and D. denoting the matrix of the vectors in Bm- As our constraint from equation requires eigenvectors of MT, to be w with Vm containing r orthogonal vectors (v 2 ... v,m)) from Dm and (s - r) vectors from the null space of M.
show in more detail in the appendix,
We are especially interested in how Mm changes w.r.t. input in terms of Dms i.e., we express LRR in terms of Dm By construction, each input d' can be represented as linear combination via vector of coefficients cn that multiplies Dm so that d', = -'m Since Mm = Um VT dms the loss CrR of layer m can be rewritten as
CRR... Mm (V,, ET E., E.TV V.T.D.DD.. D Cm))
2m (V DT Em Em dm,²
(6)
where we can assume that the coefficient vector cm is accumulated over the training data set size f. via C. = Ei= c',, since eventually every single datum D,, will contribute to LRR... The central component of equation 6 is V Dm For a successful minimization, Vm needs to retain those w m with the largest Cya coefficients. As VT, is typically severely limited in terms of its representational capabilities by the number of adjustable weights in a network, it needs to focus on the most im- portant eigenvectors terms of Cm order establish small distance to Dm Cm. Thus, features that appear multiple times in the input data with a corresponding factor in Cm will more strongly contribute minimizing CRR...
To summarize, Vm is driven towards containing r orthogonal vectors wi that represent the most frequent features of the input data, i.e., the dominant features. Additionally, due to the column vectors of Vn being mutually orthogonal, Mm. is encouraged to extract different features from the input. By the sake of being distinct and representative for the data set, these features have the potential to be useful for new inference tasks. The feature vectors embedded in Mn, can be extracted from the network weights in practical settings, as we will demonstrate below.
Realization in Neural Networks Calculating M, is usually very expensive due to the di-
mensionality of M,,- Instead of building it explicitly, we constrain intermediate results to realize equation 3 when training. A regular training typically starts with chosen network structure and
3

---

Under review as a conference paper at ICLR 2021
trains the model weights for given task via suitable loss function. Our approach fully retains this setup and adds a second pass that reverses the initial structure while reusing all weights and biases. E.g., for typical fully connected layer in the forward pass with d Mm, d + b the reverse pass operation is given by d MM, (dm+1 bm). where dm denotes the reconstructed input.
Our goal with the reverse pass is to transpose all oper- ations of the forward pass obtain identical intermedi- ate activations between the layers with matching dimen- sionality. We can then constrain the intermediate results of each layer of the forward pass to match the results of the backward pass, as illustrated in figure 2 Unlike greedy layer-wise autoencoder pretraining, which trains each layer separately and only constrains d, and dj. we jointly train all layers and constrain all intermediate re- sults. Due to the symmetric structure of the two passes, we can use a simple 1.2 difference to drive the network towards aligning the results:
Layer
2
m
id
outpatt
Figure A visual overview of the regular forward pass (bluc) and the corresponding re- verse pass for pretraining (ycllow).
LRR Σ
Am
dm
d'm
(7)
m==1
Here dm denotes the input of layer in the forward pass and d'm the output of layer for the reverse pass. Am denotes a scaling factor for the loss of layer m, which, however, is typically constant in our tests across all layers. Note that with our notation, d₁ and di refer to the input data, and the reconstructed input, respectively.
Next, we show how this setup realizes the regularization from equation 3 For clarity, we use a fully connected layer with bias. In neural network with hidden layers, the forward process for layer m is given by dm+1 = Mm + bm., with d1 and d denoting in- and output, respectively. For our pretraining, build reverse pass network with transposed operations starting with the final output where dn+1 d and the intermediate results dm+1i
d M (d (m bm),
which yields dm. - d/m |I Mm d l| When this difference is minimized via equa-
tion 7 we obtain activated intermediate content during the reverse pass that reconstructs the values computed in the forward pass, i.c. d' dm+1 holds. As in equation the reverse pass activa- tion d',, depends on dm+1's this formulation yields a full reverse pass from output to input, which we use for most training runs below. In this case
d = (d m+1 b,) (d - bm)) = MMT Mm
which is consistent with equation and satisfies the original constraint M MM - dm = 0. This version is preferable if unique path from output input exists. For architectures where the path is not unique, e.g., in the presence of additive residual connections, we use a local formulation
d = M (dm+1 - bm),
(10)
which employs dm+1 for jointly constraining all intermediate activations in the reverse pass.
Up to now, the discussion focused on simplified neural networks without activation functions or extensions such as batch normalization (BN). While we leave incorporating such extensions for future work, our experiments consistently show that the inherent properties of our pretraining remain valid: even with activations and BN, our approach successfully extracts dominant structures and yields improved generalization. In the appendix, we give details on how ensure that the latent space content for forward and reverse pass is aligned such that differences can be minimized.
To summarize, we realize the loss formulation of equation to minimize ( M,, -/)dm. without explicitly having to construct M Mr. Following the notation above, we will refer to networks trained with the added reverse structure and the additional loss terms as RR variants. We consider two variants for the reverse pass: a local pretraining equation 10 using the datum dm+1 of a given layer, and a full version via equation which uses d incoming from the next layer during the reverse pass.
4

---

Under review as a conference paper at ICLR 2021
Embedding Singular Values Below, Std denotes regular training run (in orange color in graphs below), while RR denotes our models (in green). Pre and Ort will denote regular autoencoder pretraining and orthogonality, respectively, while subscript will denote the task variant the model was trained for, e.g., Stdy for task T. While we typically use all layers of network in the constraints, reduced variant that we compare to below only applies the constraint for the input data, i.e., m=1. A network trained with this variant, denoted by RRA, is effectively trained to only reconstruct the input. It contains no constraints for the inner activations and layers of the network. For the Ort models, we use the Spectral Restricted Isometry Property algorithm (Bansal et al., 2018).
We verify that the column vectors of Vm of models from RR training contain the dominant features of the imput with the help of a classification test, employing single fully connected layer, i.c. d2 = M1, with batch normalization and activation. To quantify this similarity, we compute an LPIPS distance (Zhang et al. 2018b) between e/ and the training data (lower values being better). We employ a training data set constructed from two dominant classes (a peak in the top left, and bottom right quadrant, respectively). augmented with noise in the form of random scribbles. Based on the analysis above, we expect the RR training to extract the two dominant peaks during training. The LPIPS measure- ments confirm our SVD argumentation above, with av- erage scores of 0.217=0.022 for RR, 0.319 114 for Pre, 0.495 + 0.006 for Ort, and 0.500 = 0.002 for Std. Le.. the RR model fares significantly better than the others. At the same time, the peaks are clearly visible for RR models, an example is shown in figure (bb),, while the other models fail to extract structures that resemble the input. Thus, by training with the full network and the original training objective, pre- training yields structures that are interpretable and be inspected by humans.
Exquit: d, x 50 ungs)
Std
Pre
0.24
0.50 0.49
0.4G
0.20
Figure 3: Column vectors of Vn for different trained models Std, Ort, Pre and RR for peaks, Input features elearly are successfully embed- ded in the weights of RR, as confirmed by the LPIPS scores.
The results above experimentally confirm our formulation of the RR loss and its ability to extract dominant and generalizing structures from the training data. Next, will focus quantified metries and turn to measurements in terms of mutual information to illustrate the behavior of our pretraining for deeper networks.
3 EVALUATION IN TERMS OF MUTUAL INFORMATION
As our approach hinges on the introduction of the reverse pass, we will show that succeeds in terms of establishing mutual information (MI) between the input and the constrained intermediates inside network. More formally, MI I(X; Y' of random variables X and Y measures how different the joint distribution of X and Y is w.r.t. the product of their marginal distributions, i.e the Kullback-Leibler J. (Tishyy& & Zaslavsky (2015) proposed MI plane to
divergence T(X; Y) = L (P(x. (x,)))))lll))). || analyze trained models, which show the MI between the input and activations of layer Dm i.c., [(X; (m) and [(Dm, Y). i.c., MI of layer Dm, with output Y. These two quantities indicate how much information about the in- and output distributions are retained at each layer, and we use them to show to which extent our pretraining succeeds at incorporating information about the inputs throughout training.
The following tests employ networks with six fully connected layers with the objective to learn the mapping from 12 binary inputs to 2 binary output digits (Shwartz-Ziv & Tishby 2017). with results accumulated over five runs. We compare the versions Stda, Prea. ORta, RRA, and variant of the latter: RR), i.e. a version where only the input d, is constrained to be reconstructed. While figure 4p) visually summarizes the content of the MI planes, the graph in (b) highlights that training with the RR loss correlates input and output distributions across all layers: the cluster of green points in the center of the graph shows that all layers contain balanced MI between in- as well as output and the activations of each layer. RR) fares slightly worse, while Std, and Orta almost exclusively focus on the output with I (Dmm Y) being close to one. Pre instead only focuses on reconstructing inputs. Thus, the early layers cluster in the right-top corner, while the last layer [(r;; Y) fails to
5

---

Under review as a conference paper at ICLR 2021
Madual intomaaliitttmttoo Flane, low to Read
19 Michal Information for A
After Pww-tuning for
(4) After Mit-erttn--annggtrr
mnin Mi with in Sapps
AYa.
Jula
100
102
Figure 4: MI planes for different models: a) Visual overview of the contents. b) Plane for task A. Points on each line correspond to layers of one type of model. All points of RR are located in the center of the graph. while Std, and Orta, exhibit large [(Dm: Y). ie., specialize on the oulput. Pres strongly focuses on reconstructing the input with high ;(;; for early layers. c.d): After fine-tuning for A/B, The last layer Dr of RRA and A successfully builds the strongest relationship with Y. yielding the highest accuracy.
align with the outputs. Once we continue fine-tuning these models without regularization, the MI naturally shifts towards the output, as shown in figure 4 (c). Here, RRAA outperforms the other models in terms of final performance. Likewise, RR AB performs best for transfer task B with switched output digits, as shown in graph (d). The final performance for both tasks across all runs summarized in figure 5 These graphs visualize that the proposed pretraining succeeds in robustly establishing mutual information between inputs and targets across a full network, in addition to extracting reusable features.
MI has received attention recently as a learning ob- jective, e.g., in the form of the InfoGAN approach
Accurscy Compartoons Source Transter Tacks
(Chen et al. (016) for learning disentangled and in- terpretable latent representations. While MI typ- ically challenging to assess and estimate (Walters- Williams & Li. 2009), the results above show that
Zcom-in
approach provides straightforward and robust way for including it as a learning objective. In this way, can, e.g.. reproduce the disentangling sults from (Chen et al.) 2016). which are shown in figure A generative model with our pretraining extracts intuitive latent dimensions for the different digits, line thickness, and orientation without any additional modifications of the loss function. The joint training of the full network with the proposed reverse structure, including non- non-linearinn ooneainessssie and nor- malization, yields a natural and intuitive decomposi- tion.
Pro
00r
RK
"Auxai"
Figure 5: Performance for MI source and trans- fer tasks for the models of figure 4 Due the large standard deviation of Ort, we show min/max value ranges. The dashed gray line and region show baseline accuracy for Stds. The top-left in- highlights the stability of the high accuracy sults from RR training.
4 EXPERIMENTAL RESULTS
We now turn to a broad range of network structures, i.e.. CNNs, Autoencoders, and GANs, with a variety of data sets and tasks to show our approach succeeds in improving inference accuracy and generality for modern day applications and architectures.
Transfer-learning Benchmarks We first evaluate our approach with two state-of-the-ar bench- marks for transfer learning. The first one uses the texture-shape data set from (Geirhos al.) 2018)
which contains challenging images of various shapes combined with patterns and textures to be classified. The results below are given for 10 runs each. For the stylized data shown in figure 6 (a), the accuracy of Prets is low with 20.8%. This result is in line with observations in previous work and confirms the detrimental effect of classical pretraining. Stdıs yields a performance of 44.2%, and Ortys improves the performance to 47.0%, while RRTs yields performance of 54.7% (see figure 6p). Thus, the accuracy of RRys is 162.98% higher than Prets. 23.76% higher than StdTs, and 16.38% higher than Ortrs- To assess generality, we also apply the models to new data without

---

Under review as a conference paper at ICLR 2021
Accunaccy Comparsorrs Urtra Jadry Mlls
Acourasy Comparisons for CIFAR with Resnet 110
313)
BA2S)
12211
A16ST
balaaa
Comp ton CFAN 10
Sidex
Maa
Sikoa
09
Figure 6: set. (b, c, d) Texture-shape test accuracy compar- isons of Prets, Ortrs, Stdrs and RR for different data sets.
Examples from texture-shape data
Figure 7: Left: Examples from CIFAR 10.1 data set. Right: Accuracy comparisons when applying models trained CIFAR 10 CIFAR 10. data.
Outpput
outpus
I Loss Comparisons
Task B;
Task B; (x 10")
4.41
Reconstruuced Inpust (di)
Pre, (d+-)
586.18
(649.8
1.80
1.56
202.24
Press
Stdan
RRan
Prean,
Stean,
RRu,,
(0)
Figure 8: (a) Example output and reconstructed inputs, with the reference shown right. Only RR, successfully recovers the input, Stda produces a black image, while Pre fares poorly. (b) f.2 loss comparisons for two different generative transfer learning tasks (averaged across 5 runs each). The RR models show the best performance for both tasks.
re-training, i.e. an edge and filled data set, also shown in figure 6 (a). For the edge data set, RR outperforms Preys, Stdys and Ortys by 178.82%, 50% and 16.75 66,75% respectively,
worth pointing out that the additional constraints of our training approach lead moderately increased requirements for memory and computations, c.g., 41.86% more time per epoch than reg- ular training for the texture-shape test. the other hand, it allows us to train smaller models: we can reduce the weight count by 32% for the texture-shape case while still being on-par with Ortys in terms of classification performance. By comparison, regular layer-wise pretraining requires a sig- nificant overhead and fundamental changes the training process. Our pretraining fully integrates with existing training methodologies and can easily be deactivated via Am = 0.
As second test case, use CIFAR-based task transfer (Recht et al. 2019 that measures how well models trained on the original CIFAR 10, generalize to a new data set (CIFAR 10.1) collected according to the same principles as the original one. Here use Resnetl 10 with 110 layers and 1.7 million parameters, Due to the consistently low performance of the Pre models (Alberti) et al. 2017).
In terms of accuracy across runs, Orlcio outperforms Stde1o by 0.39%, while RRc10 outperforms Ortcıo by another 0.28% in terms of absolute test accuracy (figure 7.. This increase for RR training matches the gains reported for orthogonality in previous work (Bansal et al.] 2018), thus showing that our approach yields substantial practical improvements over the latter. especially interesting how well performance for CIFAR 10 translates into transfer performance for CIFAR 10.1. Here, RRc1o still outperforms OrtCIo and Stdc1o by 0.22% and 0.95%, respectively. Hence, the models from our pretraining very successfully translate gains in performance from the original task to the new one, indicating that the models have successfully learned a set more general features. To summarize, both benchmark cases confirm that the proposed pretraining benefits generalization.
focus Std, Ort and RR for this test
In this section, we employ our pretraining in the context of gen- erative models for transferring from synthetic to real-world data from the ScalarFlow data set (Eckert al. 2019). As super-resolution task A, we first use a fully-convolutionall generator network, adver- sarially trained with discriminator network the synthetic flow data. While regular pretraining is
Generative Adversarial Models

---

Under review as a conference paper at ICLR 2021
Std
RR
Reference
(o) Three prediction exomples 2500
(8) Three pradiction exomples of T830
(c) Three prodiction exomptes of T2M
Figure 9: Details of the three physical quantitics the weather forecasting test (full frames are shown in the appendix), As confirmed by the quantified results, RR predicts results closer to the reference.
more amenable generative tasks than orthogonal regularization, can not be directly combined with adversarial training. Hence, we pretrain a model Pre for reconstruction task at high-resolution without discriminator instead. Figure 8 a) demonstrates that our method works well in conjunction with the GAN training: As shown in the bottom row, the trained generator succeeds in recovering the input via reverse pass without modifications. A regular model Stda, only yields black im- age in this case. For Prex, the layer-wise nature of the pretraining severely limits its capabilities to learn the correct data distribution (Zhou et al., 2014), leading to a low performance.
We now mirror the generator model from the previous task to evaluate autoencoder structure that we apply to two different data sets: the synthetic smoke data used for the GAN training (task B1), and real-world RGB data set of smoke clouds (task Bz). Thus both variants represent transfer tasks, the second one being more difficult due to the changed data distribution. The resulting losses, summarized in figure b), show that RR training performs best for both autoencoder tasks: the L loss of RR AB, is 68.88%4 lower than Std while is 13.3% lower for task B2. The proposed pretraining also clearly outperforms the Pre variants. Within this series of tests, the RR performance for task B2 is especially encouraging, as this task represents a synthetic to real transfer.
Weather Forecasting Pretraining particularly attractive in situations where the amount of data for training is severely limited. Weather forecasting is such a case, systematic and accurate data for many relevant quantities are only available for approximately 50 years. We target three-day forecasts of pressure, ground temperature, and mid-level atmospheric temperature based on public benchmark dataset (Rasp et al. 2020). This dataset contains worldwide observations from ERA5 (Hersbach et al.] 2020) six-hour intervals with 5.625 resolution. For the joint inference of atmospheric pressure (500 hPa geopotential, Z500), ground temperature (T2M), and atmospheric temperature (at 850 hPa, T850), As regular pretraining is not compatible with residual connections, we omit it here.
convolutional ResNet architecture with 19 residual blocks.
We train model regular model (about 6.36M trainable parameters) with data from 1979 to 2015, and compare its inference accuracy across all datapoints from years 2017 and 2018 to similar model that employs our pretraining. While the regular model was trained for 25 epochs, the RR model was pretrained for 10 epochs and fine-tuned for another 15 epochs. Across all three physical quantities, the RR model clearly outperforms the regular model, as summarized in figure (d) and figure p (details are given in the appendix). Especially for the latitude-weighted RMSE of Z500, it yields improvements of 5.5%. These improvements point to an improved generalization of the RR model via the pretraining and highlight its importance for domains where data is scarce.
5 CONCLUSIONS
We have proposed novel pretraining approach inspired by classic methods for unsupervised au- toencoder pretraining and orthogonality constraints. In contrast the classical methods, we employ constrained reverse pass for the full non-linear network structure and include the original learning objective. We have shown for a wide range of scenarios, from mutual information, over transfer learning benchmarks to weather forecasting, that the proposed pretraining yields networks with bet- ter generalizing capabilities. Our training approach general, easy to integrate, and imposes no requirements regarding network structure training methods. Most importantly, results show that unsupervised pretraining has not lost its relevance in today's 's deep learning environment.
As future work, believe it will be exciting to evaluate approach in additional contexts, e.g., for temporal predictions (Hochreiter & Schmidhuber 1997 Cho et al.] 2014), and for training ex-
plainable and interpretable models (Zeiler & Fergus 2014 Chen et al. 2016 Du al. 2018).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Michele Alberti, Mathias Scuret, Rolf Ingold, and Marcus Liwicki. A pitfall of unsupervised pre- training. arXiv preprint 17X3.v:703.04533 2017.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf Klessen, Lena Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with 2018.
invertible neural networks. arXiv preprint
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regu- larizations in training deep cnns? In Advances in Neural Information Processing Systems, pp- 4266 4276. Curran Associates Inc., 2018.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Joan Bruna, Arthur Szlam, and Yann LeCun. Signal recovery from pooling representations. arXiv preprint arXiv: 131 1.4025, 2013.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arxiv:14000 1078, 2014.
Hui Ding, Shaohua Kevin Zhou, and Rama Chellappa. Facenet2expnet: Regularizing deep face recogmition net for expression recognition. In 2017 12th IEEE International Conference on Auto- matic Face & Gesture Recognition (FG 2017), pp. 118-126. IEEE, 2017.
Laurent Dinh, Jascha Sohl-Dic,stein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
Mengnan Du Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. arXiv preprint arXiv:1808.0000 000333, 2018.
Marie-Lena Bckert, Kiwon Um, and Nils Thuerey. Scalarflow: large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM Transac- tions on Graphics (TOG), 38(6):239, 2019.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 201-208, 2010.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv: 1.////1.1120 12231, 2018.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger Grosse. The reversible residual net- work: Backpropagation without storing activations. In Advances in Neural Information Process- ing Systems, pp. 2214-2224, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Kasthurirangan Gopalakrishnan, Siddhartha K Khaitan, Alok Choudhary, and Ankit Agrawal. Deep convolutional neural networks with transfer learning for computer vision- based data-driven pave- ment distress detection. Construction and Building Materials, 157:322-330, 2017.

---

Under review as a conference paper at ICLR 2021
Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with prok-propaggtton. In Advances in Neural Information Processing Systems, pp. 177-185, 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín SanozeSabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730); (9977300:19992044 1999-2049, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Lei Huang. Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv:1802.07088, 2018
Kui Jia, Dacheng Tao, Shenghua Gao, and Xiangmin Xu. Improving training of deep neural net- works via singular value bounding. In JEEE Conference on Computer Vision and Puttern Recog- nition, pp. 4344 4352, 2017.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv: 171005564 2017.
Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewiez. Rotation invariant spherical harmonic representation of 3 shape descriptors. In Symposium on geometry processing, vol- ume 6. pp. 156-164, 2003.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:14/2.6980, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255, 2016.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947-5956, 2017.
Mete Ozay and Takayuki Okatani. Optimization on submanifolds of convolution kernels in cnns. 2016.
arXiv preprint
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in neural information processing systems, 2015.
pp.
Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: A benchmark dataset for data-driven weather forecasting, arXiv preprint arXiv:2002.00466, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv: 19040097, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer, 2015.
10

---

Under review as a conference paper at ICLR 2021
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint 0tt:::/703.00000 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):12291198, 2014.
Yunfei Teng and Anna Choromanska. Invertible autoencoder for domain adaptation. Computation, 7(2):20, 2019.
Nils Thuerey and Tobias Pfaff. MantaFlow, 2018. http://manttflow.com..
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Lisa Torey and Jude Shavlik. Transfer learning. Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp. 242 264. IGI Global, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11((2), 2010.
Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and prin- cipal component analysis. In A practical approach to microarray data analysis, pp. 91-109. Springer, 2003.
Janett Williass and Yan Li. Estimation of mutual information: A survey. In International Conference Rough Sets and Knowledge Technology, pp. 389-396. Springer, 2009.
Andreas S Weigend, David E Rumelhart, and Bernardo A Huberman. Generalization by weight- elimination with application to forecasting. In Advances in Neural Information Processing Sys- tems, pp. 875-882, 1991.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3300 (0000322,200 2014.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722 2018.
Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Lijing Zhang, Yao Lu, Ge Song, and Hanfeng Zheng. Re-enn: Reverse connected convolutional
neural network for accurate player detection. In Pacific Rim International Conference Artificial Intelligence, pp. 438 446. Springer, 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018b.
Yingbo Zhou, Devansh Arpit, Ifeoma Nwogu, and Venu Govindaraju. Is joint training better for deep auto-encoders? arXiv preprint arXiv: 1405. / 1380, 2014.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
To ensure reproducibilityy, source code and data for all tests will be published. Runtimes were mea- sured on a machine with Nvidia GeForce GTX 1080 Ti GPUs and an Intel Core i7-6850K CPU.
A.1 DISCUSSION OF RELATED WORK
Greedy layer-wise pretraining was first proposed by (Bengio et al. 2007), and influenced a large number of follow up works, providing a crucial method for enabling stable training runs of deeper networks. A detailed evaluation was performed by (Erhan et al.) 20TO), also highlighting cases were can be detrimental. These problems were later on also detailed in other works, c.g., by (Alberti ct al. 2017). The transfcrability of learned features was likewise a topic of interest for transfer learn- ing applications (Yosinski et al., 2014). Sharing similarities with our approach, (Rasmus et al. 2015) combined supervised and unsupervised learning objectives, but focused denoising autoencoders and a layer-wise approach without weight sharing. We demonstrate the importance of leveraging state-of-the-art methods for training deep networks, i.e. without decomposing or modifying the network structure. This not only improves performance, but also very significantly simplifies the adoption of the pretraining pass in new application settings.
Extending the classic viewpoint of unsupervised autoencoder pretraining, several prior methods employed "hard orthogonal constraints" to improve weight orthogonality via singular value decom- position (SVD) at training time (Huang et al. 2018 Dia et al. 2017 Ozay & Okatani (016). Bansal al. (Bansal al. 2018) additionally investigated efficient formulations the orthogonality con- straints. In practice, these constraints are difficult to satisfy, and correspondingly only weakly im- posed. addition, these methods focus on improving performance for a known, given task. This means the training process only extracts features that the network considers useful for improving the performance of the current task, not necessarily improving generalization or transfer performance Torrey & Shavlik 2010). While our approach shares similarities with SVD-based constraints, it can be realized with a very efficient I.2- formulation, and takes the full input distribution into account.
Recovering all input information from hidden representations of a network is generally very difficult (Dinh et al. 2016 Mahendran & Vedaldi 2016), due the loss of information throughout the layer transformations. In this context, (Tishyy & Zaslavsky 2015) proposed the information bottleneck principle, which states that for an optimal representation. information unrelated to the current task is omitted. This highlights the common specialization of conventional training approaches.
Reversed network architectures were proposed in previous work (Ardizzone al. 2018 Jacobsen et al. 2018 Gomez et al., 2017), but mainly focus on how to make a network fully invertible via augmenting the network with special structures. As a consequence, the path from input to output is different from the reverse path that translates output to input. Besides, the augmented structures of these approaches can be challenging to apply to general network architectures. In contrast, our approach fully preserves an existing architecture for the backward path, and does not require any operations that were not part of the source network. As such, can easily be applied in new settings, e.g., adversarial training (Goodfellow et al. 2014). While methods using reverse connections were previously proposed (Zhang et al.) 2018a] Teng & Choromanska 2019). these modules primarily focus transferring information between layers for given task, and auto-ent auto-encoder structures for domain adaptation, respectively.
A.2 PRETRAINING AND SINGULAR VALUE DECOMPOSITION
In this section we give more detailed derivation of our loss formulation, extending Section 3 of the main paper. As explained there, our loss formulation aims for minimizing
LRR = Σ (M M, - )
(1I)
m==
where Mm E Th axxxx x denotes the weight matrix of layer m, and data from the input data set Dmn. is denoted by d C RR=, i -,,,.... ...,, l. Here denotes the number of samples in the input data set.
12

---

Under review as a conference paper at ICLR 2021
Minimizing equation 11 is mathematically equivalent to
Ma. Mm d' - d
(12)
for all d Hence, perfectly fulfilling equation 11 would require d' eigenvectors of M Mm with corresponding eigenvalues being 1. As in Sec. 3 of the main paper, we make use of an auxiliary orthonormal basis Bm <W"" m "...., W for which q (with & S t) denotes the number of linearly independent entries in D. While B,, never has to be explicitly constructed for our method, can, e.g., be obtained via Gram-Schmidt. The matrix consisting of the vectors in Bm is denoted by Dm
Since the w/ (h = ...)...q) necessarily can be expressed as linear combinations di,, equation 11 similarly requires w to be eigenvectors of M Mm. with corresponding eigenvalues being 1, i.e.:
M Mm w m
w m
(13)
We denote the vector of coefficients to express d via Dm with cim i.e d'n = Dm equation 12 can be rewritten as:
Then
M Mm Dm em
Dm
(14)
Via SVD of the matrix Mm in equation 14 we obtain
M T Mm mm - Dm Cy
m
Ma. Mm Mm.. wht Cma
w Cma..
(15)
V En E VT wh Cmy,
w
m Cmn
where the coefficient vector Cm is accumulated over the training data set size t via Cm = Ei= cim. Here we assume that over the course of a typical training run eventually every single datum in Dm will contribute to LRR... This form of the loss highlights that minimizing LRR requires alignment of Vm DT .m Fm VT m Cm,, and wh m Cma.
By construction, Em contains the square roots of the eigenvalues of Mm its diagonal en- tries. The matrix has rank r = rank(M, Mm). and since all eigenvalues are required to be by equation 13 the multiplication with Σ,, in equation equutiio vectors from Vn. Hence, we can focus on the interaction between the basis vectors Wm and the active column vectors of Vn
performs selection of r column
V EI Im Vi' wh m Cma w Cma V2' w m
Cm. (Vm E'l'
Em
wh) wh v. w ).
(16)
Cma
f=1
As Vin is obtained via an SVD it contains r orthogonal eigenvectors of M Mm. equation requires w m..... w to be eigenvectors of MT. Mr, but since typically the dimension of the input data set is much larger than the dimension of the weight matrix, i.e. r S q. in practice only r vectors from B,, can fulfill equation 13 This means the vectors v1 mm ... v in Vm are a subset of the orthonormal basis vectors Bn (Win..., w9.) with )2 = 1. Then for any wh we have
13
m
if if w (v) m wh = O, otherwise.
(v T 1,
(17)
Thus if Vn, contains w we have
(vd )r Wwt v.f wh m
(18)
13

---

Under review as a conference paper at ICLR 2021
and we trivially fulfill the constraint
Cma
) wht v.f w
0.
(19)
However, due being smaller than q in practice, Vn typically can not include all vectors from Bm Thus, if Vm does not contain w m we have (v ) = 0 for every vector v in Vn,, which means
w
Σ (v m
v 0.
(20)
As consequence, the constraint equation 12 is only partially fulfilled:
wht v/ wh)) -Cm w
CYTA
(21)
As the wht have unit length, the factors Cm determine the contribution of a datum to the overall loss. A feature wh, m that appears multiple times in the input data will have correspondingy larger factor in Cm and hence will more strongly contribute to LRR. The L formulation of equation [1 leads to the largest contributors being minimized most strongly, and hence the repeating features of the data, i.e., dominant features, need to be represented in Vn to minimize the loss. Interestingly, this argumentation holds when additional loss terms are present, e.g., a loss term for classification. In such a case, the factors Cm will be skewed towards those components that fulfill the additional loss terms, i.e. favor basis vectors w that contain information for about the loss terms. This, e.g., leads to clear digit structures being embedded in the weight matrices for the MNIST example below.
In summary, to minimize CRR, Vm is driven towards containing r orthogonal vectors w which represent the most frequent features of the input data, i.e. the dominant features, It worth empha- sizing that above B,, is only an auxiliary basis, i.e., the derivation does not depend on any particular choice of Bm.
A.3 EXAMPLES OF NETWORK ARCHITECTURES WITH PRETRAINING
While the proposed pretraining is significantly more easy to integrate into training pipelines than classic autoencoder pretraining, there are subtleties w.r.t. the order the operations in the reverse pass that we clarify with examples in the following sections. To specify NN architectures, we use the following notation: C(k, I, 9). and D(k, I, q) denote convolutional and deconvolutional operations, respectively, while fully connected layers are denoted with F()), where k, q denote kernel size, output channels and stride size, respectively. The bias of a CNN layer is denoted with b. T/O(z) denote input/ output. their dimensionality given by 2. I,- denotes the input of the reverse pass network. tanh, relu, Irelu denote hyperbolic tangent, ReLU, and leaky ReLU activation functions (AF), where we typically use leaky tangent of 0.2 for the negative half-space. UP, MP and BN nearest-neighbor
denote 2x nearest- neighbor up-sampling, max pooling with 2 x 2 filters and stride 2, and batch normalization, respectively.
Below we provide additional examples how to realize the pretraining loss Cn in a neural network architecture. As explained in the main document, the constraint equation 11 formulated via
n
d'm
Cn =
Am dm
(22)
m=1
with dm, and Am denoting the vector of activated intermediate data in layer from the forward pass, and scaling factor, respectively. d',, denotes the activations of layer m from the reverse pass. E.g., let Lm() denote the operations of a layer m in the foward pass, and L', () the corresponding operations for the reverse pass. Then d = Lm (d..), and d',, = L',, (d'm+))-
When equation 22 minimized, we obtain activated intermediate content during the reverse pass that reconstructs the values computed in the forward pass, i.e. d'#t1 dm+1 holds. Then d',, can be reconstructed from the incoming activations from the reverse pass, i.c., d or from the output of layer m, i.e., dmt1. Using d' d results in global coupling of input and output throughout
14

---

Under review as a conference paper at ICLR 2021
Layer
Layer
Amii
BN
fre
(M Aud
outprrt
d'
Fm--1
(N -b.)
Figure 10: Left: An overview of the regular forward pass (blue) and the corresponding reverse pass (yellow). The right side illustrates how parameters are reused for convolutional layer. conv/ deconv denote convolu- tion/deconvoluttional operations. Jm and BN.. denote the activation function and batch normalization of layer m, respectively. Shared kernel and bias represented by Mn and bm
all layers, i.e., the full loss variant. On the other hand, dm+1 yields variant that ensures local reversibility of each layer, and yields very similar performance, as we will demonstrate below. We employ this local loss for networks without a unique, i.c., bijective, connection between two layers. Intuitively, when inputs cannot be reliably reconstructed from outputs.
Full Network Pretraining: An illustration of a CNN structure with AF and BN and full loss is shown in figure TO While the construction of the reverse pass is forward for all standard operations, i.e., fully connected layers, convolutions, pooling, etc., slight adjustments are necessary for AF and BN. It is crucial for our formulation that dm and d', contain the same latent space content in terms of range and dimensionality, such that they can be compared in the loss. Hence, we use the BN parameters and the AF of layer m from the forward pass for layer m in the reverse pass, An example is shown figure 14
To illustrate this setup, we consider an example network employing convolutions with mixed AFs, BN, and MP. Let the network receives field of 322 scalar values as input. From this input, 20, 40, and 60 feature maps are extracted in the first three layers. Besides, the kernel sizes are decreased from x to 3 x 3. To clarify the structure, we use ReLU activation for the first convolution, while the second one uses a hyperbolic tangent, and the third one sigmoid function. With the notation outlined above, the first three layers of the network are
(32,33, 1) = d. + C1 20, 1) + b1 + BN1 + relu
+ d2 + MP - C2(4, 40, 1) + b + BN2 - tanh
(23)
- d3 + MP (3, 1) + b3 > BN, -> sigm
d, +
The reverse pass for evaluating the loss re-uses all weights of the forward pass and ensures that all intermediate vectors of activations, dm and d'm. have the same size and content in terms of nor- malization and non-linearity. We always consider states after activation for L... Thus, dm denotes activations before pooling in the forward pass and d'm contains data after up-sampling the reverse pass, in order to ensure matching dimensionality. Thus, the last three layers of the reverse network for computing C,, take the form:
d + bb Dz(3, 40, 1) - BN2 + tanh + UUP d + bb + D2(4, 20, + BN; + relu + UP
(24)
-b (5,
d O 1).
Here, the de-convolutions D, in the reverse network share weights with Cz in the forward network. Le., the x 20 x 40 weight matrix of C2 is reused its transposed form as matrix in D2. Additionally, it becomes apparent that AF and BN of layer 3 from the forward pass do not appear in the listing the three last layers of the reverse pass. This is caused by the fact that both are required to establish the latent space of the fourth layer. Instead, d3 in our example represents the activations after the second layer (with BN, and tanh), and hence the reverse pass for d' reuses both functions. This ensures that d,,, and d',, contain the same latent space content in terms of range and dimensionality, and can be compared in equation 22
x 40 x 20
15

---

Under review as a conference paper at ICLR 2021
For the reverse pass, we additionally found it beneficial to employ an AF for the very last layer if the output space has suitable content. E.g., for inputs in the form of RGB data we employ an additional activation with a ReLU function for the output to ensure the network generates only positive values.
Localized Pretraining: In the example above,
a full pretraining with d to reconstruct the activations d'.. The full structure establishes a slightly stronger relationship among the loss terms of different layers, and allows earlier layers to decrease the accumulated loss of later layers. However, if the architecture of the original network makes use of operations between layers that not bijective, we instead use the local loss. E.g., this happens for residual connections with an addition or non-invertible pooling operations such as max-pooling. In the former, we cannot b given only And unless special care is taken (Bruna et al. 2013), the source neuron of an output is not known for regular max-pooling operations. Note that our loss formulation has no problems with irreversible operations within a layer, c.g,, most convolutional or fully-connected layers typically are not fully invertible. In all these cases the loss will drive the network towards state that is as-invertible-ass asspossilleee for the given input data set. However, this requires a reliable vector of target activations in order to apply the constraints. If the connection betweeen layers is not bijective, we cannot reconstruct this target for the constraints, as in the examples given above.
uniquely determine the b,c in
In such cases, we regard every layer as an individual unit to which we apply the constraints by building localized reverse pass. For example, given a simple convolutional architecture with d - (5, 20, 1) + b = d2
(25)
in the forward pass, we calculate di with
(d2 b1) D1 (5 di
(26)
We, c.g., use this local loss in the Resnetl 10 network below. It is important to note that despite being closer to regular autoencoder pretraining, this formulation still incorporates all non- oonelincrssies of the original network structure, and jointly trains full networks while taking into account the original learning objective.
A.4 MNIST AND PEAK TESTS
Below we give details for the peak tests from Sec. 3 of the main paper and show additional tests with the MNIST data set.
(5) One hiddon loyer, without EN, Without
d. E ERTTHHI MM 2257M dg
Std
Ort
Pre
RR
unns...
LE175
Peak Test: For the Peak test we generated data set of 110 images shown in figure 12 55 images contain a peak located in the upper Teft corner of the image. The other 55 contain peak located in the bottom right corner. We added random scrib- bles in the images to complicate the task. All 1 110 images were labeled with a one-hot encoding of the two possible positions of the peak. We use 100 im- ages as training data set, and the remaining 10 for testing. All peak models are trained for 5000 epochs with a learning rate of 0.0001, with À = 1e 6 for RR,, To draw reliable conclusions, we show re- sults for five repeated runs here. The neural network in this case contains one fully connected layer, with BN and ReLU activation. The results are shown in figure 13 with both peak modes being consistently embedded into the weight matrix of RR,, while reg- ular and orthogonal training show primarily random singular vectors.
0.312
0.902
0.394
(h) One Nitden layer, with BNN with Rei UP
d, 2.3244 ga dd gest
Stel
Ort
Pre
RR*
0,490
C.499
3,239
0.233
V3
0.502
C.481
3,400
0.202
hicklen lagers, with
d. 7,7 My
18336784 M,, 22-122 1a2x
Std
Ort
Pre
RR
0.958
0.437
0.433
0.207
Figure 14; Right singular vectors of M1 for peak tests with different network architectures. Across the three architectures, RR, successfully extracts dominant and salient features.
We also use different network architectures in fig- 14 to verify that the dominant features are successfully extracted when using more complex
16

---

Under review as a conference paper at ICLR 2021
+5
140
LTPS
LPPS
3
128
GV1
1.0
ASX
DAY
STEEY
0.00
Figure 11: SVD of the Mi matrix for five tests with random two digit images as training data. I.PIPS distances (Zhag al. 2018b) of RR are consistently lower than Std and Ort.
training data
test data
f
p
o
e
7
D
2
b
Figure 12: Data set used for the peak tests.
Repeated Test and
Fi
Fi
Pre
Pre
1411
2.12
C10
260
LPIPS
Figure 13: Five repeated tests with the peak data shown in Sec. 3 of the main paper. RRa robustly extracts dominant features from the data set. The two singular vectors strongly resemble the peak modes of the training data. This confirmed by the LPIPS measurements.
17

---

Under review as a conference paper at ICLR 2021
network structures. Even for two layers with BN and ReLU activations, our pretraining clearly ex- tracts the two modes of the training data. The visual resemblance is slightly reduced in this case, as the network has the freedom to embed the features in both layers. Across all three cases (for which we performed 5 runs each), our pretraining clearly outperforms regular training and the orthogonal- ity constraint in terms of extracting and embedding the dominant structures of the training data set in the weight matrix.
MNIST Test: We additionally verify that the column vectors of Vn of models from RR training contain the dominant features of the input with MNIST tests, which employ a single fully connected layer, i.e d2 = d In the first MNIST test, the training data consists only of 2 different images. All MNIST models are trained for 1000 epochs with a learning rate of 0.0001, and À = le 5 for 2 After training, we compute the SVD for M SVDs of the weight matrices of trained models can be seen in figure IT The LPIPS scores show that features embedded in the weights of RR are consistently closer to the training data set than all other methods, i.c., regular training Std, classic autoencoder pretraining Pre, and regularization via orthogonalization Ort. While the vectors of Std and Ort contain no recognizable structures.
Overall, our experiments confirm the motivation of our pretraining formulation. They additionally show that employing an SVD of the network weights after our pretraining yields a simple and con- venient method to give humans intuition about the features learned by a network.
B MUTUAL INFORMATION
This section gives details of the mutual information and disentangled representation tests from Sec. 4 of the main paper.
B.1 MUTUAL INFORMATION TEST
Mutual information (MI) measures the dependence of random variables, i.e., higher MI means that there is more shared information between two parameters. More formally, the mutual infor- mation [[X; Y) of random variables X and Y measures how different the joint distribution of X and Y is w.r.t. the product of their marginal distributions, i.c., the Kullback-Leibler diver- gence Y(X:Y)= = Kl.[P(x.y)]I Py), where KI. denotes the Kullback-Leibler divergence. Let (() denote the mutual information between the activations of layer Dm, and input X. Sim- ilarly [(D,, Y) denotes the MI between layer m and the output Y. We use MI planes in the main paper, which show I(X; Dm) and I(Dm: Y) 2D graph for the activations of each layer D,, of a network after training. This visualizes how much information about input and output distribution retained at each layer, and how these relationships change within the network. For regular train- ing, the information bottleneck principle (Tishby & Zaslavsky 2015) states that early layers contain more information about the input, i.c., show high values for [(:; Dm) and 1 (Dmi mi YY. Hence in the MI plane visualizations, these layers are often visible at the top-right corner. Later layers typically share a large amount of information with the output after training, i.c. show large (Dmi Y) values, and correlate less with the input (low ((::,,)) Thus, they typically show up in the top-left corner of the MI plane graphs.
Training Details: We use the same numerical studies as in (Shwartz-Ziv & Tishby 2017) as task A, i.e. a regular feed-forward neural network with 6 fully-connected layers. The input variable X contains 12 binary digits that represent 12 uniformly distributed points on a 2D sphere. The learning objective is to discover binary decision rules which are invariant under ((3) rotations of the sphere. X has 4096 different patterns, which are divided into 64 disjoint orbits of the rotation group. forming minimal sufficient partition for spherically symmetric rules (Kazhdan et al. 2003). To generate the
input-output distribution P(X, Y), We apply the stochastic rule p(y = 1|x)
(f 0), (x e X.y E Y), where // is a standard sigmoidal function (n) 1/(11 exp( (ur)) following SSwwatz--Zii & Tishby 2017). We then use a spherically symmetric real valued function of the
pattern f(z), evaluated through its spherical harmonics power spectrum (Kazhdan et al., (2003), and
compare with a threshold 8, which was selected to make p(y = 1) = >e p(y = [0]p(z] R 0.5, with uniform p(r). 7 is high enough keep the mutual information T T(xY) R 0.99 bits.
18

---

Under review as a conference paper at ICLR 2021
For the transfer learning task B, reverse output labels to check whether the model learned specific or generalizing features. E.g. if the output is [0.1] in the original data set, we swap the entries to [1,0]. 80% of the data (3277 data pairs) are used for training and rests (819 data pairs) are used for testing.
For the MI comparison in Fig. 4 of the main paper, we discuss models before and after fine-tuning separately, in order to illustrate the effects of regularization. We include model with greedy layer- wise pretraining Pre, a regular model Stda, one with orthogonality constraints Orta, and our regular model ARA, all before fine-tuning. For the model A all layers are constrained to be recovered in the backward pass. We additionally include the version RRA, i.e. a model trained with only one loss term \1|d1 dd, l2. which means that only the input is constrained to be recovered. Thus, RR3 represents a simplified version of our approach which receives no constraints that intermediate results of the forward and backward pass should match. For Orta, we used the Spectral Restricted Isometry Property (SRIP) regularization (Bansal et al. 2018).
LSRP = Bo W II),
(27)
where W is the kernel, I denotes identity matrix, and 3 represents the regularization coefficient. [Wal denotes the spectral norm of W.
(W)= = suppssu .=0 ||=||
As explained in the main text, all layers of the first stage, i.e. from RR RRA, OIta Prea and Std are reused for training the fine-tuned models without regularization, i.e. RRAA RRAA OrtAA Pre^a and Stdaa- Likewise, all layers of the transfer task models RR RRA, OrTAB PreAn and StdAB are initialized from the models of the first training stage.
Analysis of Results: We first compare the version only constraining input and output reconstruc- tion (RR.) and the full loss version RR₃, Fig. 4(b) of the main paper shows that all points of RRA are located in a central region of the MI place, which means that all layers successfully encode information about the inputs as well as the outputs. This also indicates that every layer contains a similar amount of information about X and Y. and that the path from input to output is similar to the path from output to input. The points of RRA, on the other hand, form a diagonal line. I... this network has different amounts of mutual information across its layers, and potentially a very differ- ent path for each direction. This difference in behavior is caused by the difference of the constraints in these two versions: RRA is only constrained to be able to regenerate its input, while the full loss for RR, A ensures that the network learns features which are beneficial for both directions. This test highlights the importance of the constraints throughout the depth of network in our formulation. In contrast, the I (X; values of later layers for Std, and Orta exhibit small values (points near the left side), while (D;; Y) is high throughout. This indicates that the outputs were successfully encoded and that increasing amounts of information about the inputs are discarded. Hence, more specific features about the given output data-set are learned by Std, and Ort,, This shows that both models are highly specialized for the given task, and potentially perform worse when applied to new tasks. Prea only focuses on decreasing the reconstruction loss, which results in high [(X; DD values for early layers, and low I(D;Y) values for later layers.
During the fine-tuning phase for task A (i.e. regularizers being disabled), all models focus on the output and maximize [DD; Y). There are differences in the distributions of the points along the y-axis, i.c., how much MI with the output is retained, as shown in Fig. 4(c) of the main paper. For model RRAA, the I [D; Y) value is higher than for StdAa, ORtAs, PReĄs and RR which means outputs of RRAA are more closely related to the outputs, i.e., the ground truth labels for task A. Thus, RRAA outperforms the other variants for the original task.
In the fine-tuning phase for task B, StdAn stands out with very low accuracy in Fig. 5 of the main paper. This model from regular training run has large difficulties to adapt to the new task. Pre aims at extracting features from inputs and reconstructed them. PIсAB outperforms StdAB, which means features helpful for task B are extracted by PreA, however, it's hard to guide the feature extracting process. Model OrtAb also performs worse than Stdg- RRAS shows the best performance in this setting, demonstrating that our loss formulation yielded more generic features, improving the performance for related tasks such as the inverted outputs for B.
We also analyze the two variants of our pretraining: the local variant IRR, and the full version RR, in terms of mutual information. figure 15 shows the MI planes for these two models, also
19

---

Under review as a conference paper at ICLR 2021
(4) Mutuul infermatio T
34 Afti Wnwraniinggg
12) Acconacy Compaturre Course Trandee Tau
152
2ooeve
M5a
2o0g 38 H3 By & aupu
101)
155
100
0.5
03)
Figure 15: (a-c) MI plane comparisons for local (IRR.) versus full models (RR,). Points on each line correspond layers of one type of model. MI Plane for task A. All points of RR, and the majority of points for IRRs (five out seven) are located in the center of the graph, i.c., successfully connect in- and ouput distributions. b,c): After fine-tuning for A/B. The last layer D7 of RRAa builds the strongest relationship with Y. [D;;; Y) of IRR, is only slightly lower than RRAA- d): Accuracy comparisons among different models: RRAA yields the highest performance, while IRRA performs similarly with RRAA-
showing RR! for comparison. Despite the local nature of IRRA manages to establish MI for the majority of the layers, as indicated by the cluster of layers in the center of the MI plane. Only the first layer moves towards the top right corner, and the second layer is affected slightly. Lee. these layers exhibit a stronger relationship with the distribution of the outputs. Despite this, the overall performance when fine-tume toning or for the task transfer remains largely unaffected, e.g., the IRRA still clearly outperforms RR² This confirms our choice use the full pretraining when network connectivity permits, and employ the local version in all other cases.
B.2 DISENTANGLED REPRESENTATIONS
The InfoGAN approach (Chen et al., 2016) demonstrated the possibility to control the output of generative models via maximizing mutual information between outputs and structured latent vari- ables. However, mutual information is very hard to estimate in practice (Walters- Williams & Li, 2009). The previous section and Fig. 4(b) of the main paper demonstrated that models from our pretraining (both RR) and RRA) can increase the mutual information between network inputs and outputs. Intuitively, the pretraining explicitly constrains the model to recover an input given an output, which directly translates into an increase of mutual information between input and output distributions compared to regular training runs. For highlighting how our pretraining can yield dis- entangled representations (as discussed in the later paragraphs of Sec. 4 of the main text), we follow the experimental setup of InfoGAN (Chen et al. 2016): the input dimension of our network is 74, containing ten-dimensional category code cı, 2 continuous latent codes 02 C3 U(-1,)) 1) and 62 noise variables. Here, 2 denotes a uniform distribution.
Training Details: As InfoGAN focuses on structuring latent variables and thus only increases the mutual information between latent variables and the output, we also focus the pretraining on the corresponding latent variables. Le.,, the goal is to maximize their mutual information with the output of the generative model. Hence, we train model RR' for which only latent dimensions C₁, C2 C3 of the input layer are involved in the loss. We still employ a full reverse pass structure in the neural network architecture. C1 is ten-dimensional category code, which is used for controlling the output digit category, while C2 and C3 are continuous latent codes, to represent (previously unknown) key thickness. Building relationship between ei and
properties of the digits, such orientation outputs is more difficult than for (2 or cg. since the 10 different digit outputs need to be encoded in a sinlge continuous variable C1. Thus, for the corresponding loss term for ei we use slightly larger factor (by 33%) than for 02 and c3. Details of our results are shown in figure [6] Models are trained using a GAN loss (Goodfellow al. 2014) as the loss function for the outputs.
Analysis of Results: In figure 16 we show additional results for the disentangling test case. It is visible that our pretraining of the RR model yields distinct and meaningful latent space dimensions for C12233 While e₁ controls the digit, 62,3 control the style and orientation of the digits. For com- parison, a regular training run with model Std does result in meaningful or visible changes when adjusting the latent space dimensions. This illustrates how strongly the pretraining can shape the
20

---

Under review as a conference paper at ICLR 2021
9
9
9
0
2
3
4
b
2
2
C
0
2
3
4
6
6
7
8
4
R
8
4
b
7
&
9 4
5
4
S
6
&
?
2
4
6
7
G
(4) Vantin f 30d tha dear mearing)
[B] Waringe 38
[Pg 1yp4)
3c) Vinving s for At1 Doyl tweakrrg)
[6) Varying deferitt (Hotatoo))
Figure 16:
Additional results for the disentangled representations with the MNIST data: For every row in the figures, we vary the corresponding latent code (left to right), while keeping all other inputs constant. Different rows indicate a different random noise input. For example, in (b): every column contains five results which are generated with different noise samples, but the same latent codes C1w3. In every row, 10 results generated with 10 different values of cı, which correspond no clear correspondence between e₁ and the outputs are apparent (similarly for 02,3). (c): Different O2 values result in a tweaked style, while ey controls the orientation of the digit, as shown in (d). Thus, in contrast to Std, the pretrained model learns meaningful, disentangled representation.
digit each for (b). (a): For a regular training (Std),
latent space, and in addition to an intuitive embedding of dominant features, yield disentangled representation.
C DETAILS OF EXPERIMENTAL RESULTS
C.I TEXTURE-SHAPE BENCHMARK
Training Details: All training data of the texture-shape tests were obtained from (Geirhos et al. 2018 The stylized data set contains 1280 images, 1120 images are used training data, and 160 as test data. Both edge and filled data sets contain 160 images each, all of which ing only. All three sets (stylized, edge, and filled) contain data for 16 different classes.
Texdure-Srapp Teols Par-clasa Accures acy
used for test-
10
Figure 17: Separate per-class test accuracies for the four model variants. The RRis model exhibits a consistently high accuracy across all 16 classes.
Analysis of Results: For a detailed com- parison, we list per-class accuracy of styl- ized data training runs for Ortrs, Stdrs, Prets and RRTs in figure 17 RRTs out-
performs the other three models for most of the classes. RRTs requires an additional 41.86% for training compared to Stdrs, but yields a 76% higher performance. (Training times for these mod- els are given in the supplementary document.) All models saturated, i.c. training Stdys or Ortys longer does not increase classification accuracy any further. We also investigated how much we can reduce model size when using our pretraining in comparison to the baselines. A reduced model only uses 67.9 .44%% of the parameters, while still outperforming Ortns-
C.2 GENERATIVE ADVERSARIAL MODELS
Training Details: The data of smoke simulation was generated with Navier-Stokes solver from an open-source library (Thuerey& & Pfaff 2018). We generated 20 randomized simulations with 120 frames each, with 10% of the data being used for training. The low-resolution data were down-sampled from the high-resolution data by a factor of 4. Data augmentation, such as flipping and rotation was used in addition. As outlined in the main text, we consider building an autoencoder model for the synthetic data as task B1, and generating samples from real-world smoke data set as task B3. The smoke capture data set for B3 contains 2500 smoke images from the ScalarFlow data set (Eckert al. 2019), and again used 10% of these images training data set.
Task A: We use a fully convolutional works. Note that the inputs of the discriminator contain high resolution data (64, 64, 1), well as low resolution (16, 16, 1), which is up-sampled to (64, 64, 1) and concatenated with the high resolution data. In line with previous work (Xie al. 2018), RR, A and Std, are trained with
based architecture for generator and discriminator net-
21

---

Under review as a conference paper at ICLR 2021
Prean
STEAR
RBAS
Figure 18: Example outputs for PreAn, StdAA,, , RR AB. The reference is shown for comparison. RR produces higher quality results than StdAn; and Pre Ary
Outputs
E 21.51
Estaaey: 4.16
:3.78
reference
MAE
Figure 19: Mean Absolute Error (MAE) comparisons for smoke task B2 models. AR shows the smallest and additionally achieves the best visual quality amongst the different models.
non-saturating GAN loss, feature space loss and L2 loss as base loss function. All generator layers involved in the pretraining loss. As greedy layer-wise autoencoder pretraining is compatible with adversarial training, we pretrain Prea for reconstructing the high resolution data instead.
Task Br: All encoder layers are initialized from RR, and Std, when training RR AB, and Std It is worth noting that the reverse pass of the generator is also constrained when training PReĄ and RR So both encoder and decoder are initialized with parameters from Pre and RR, when train- ing Pre, and RRAB, respectively. This is not possible for regular network like Std as the weights obtained with a normal training run are not suitable convolutions of At are initialized randomly.
transposed. Hence, the de-
Task By: As the data set for the task B, substantially different and contains RBG images (instead of single channel gray-scale images), we choose the following setups for the RR Pre and Std, models: parameters from all six layers of Std, and RR, are reused for initializing decoder part of At, and RR AB parameters from all six layers of PreĄ are reused for initializing the encoder part of Ar. Specially, when initializing the last layer of Pre Std and ARA,, we copy and stack the parameters from the last layer of Pre Std, and RR respectively, into three channels to match the dimenions of the outputs for task Bz. Here, the encoder part of RR, AB, and the decoder of Ar, not initialized with RR, and Prex, due the significant gap between training data sets of task B, and task B2. Our experiments show that only initializing the decoder part of RR AB, (avg- loss:1.56ed std. dev.:3.81e5) outperforms initializing both encoder and decoder (avg. loss:1.82e77 + 2.07e6), and only initializing the encoder part of Pre AB, (avg. 1oss:4.4le7 + 6.36e6) outperforms initializing both encoder and decoder (avg. Poss:9.42e7 7 + 6.11e7). We believe the reason that initializing both encoder and decoder part makes it more difficult to adjust the parameters for new data that very different from the data set of the source task.
Analysis of Results: Example outputs of PreAB, StdAB, and RR AB, shown in figure 18 is clearly visible that RRAB, gives the best performance among these models. We similarly illustrate the behavior of the transfer learning task B2 for images of real-world fluids. This example
22

---

Under review as a conference paper at ICLR 2021
(1)
(2)
(2)
8R
(6) prodiction saample of 2500
0 prsdiction eramples of 7880
(c) 7hrve prodicton exampless 72N
Figure 20: A comparison of additional Z500, T850, T2M predictions (zoomed in regions). The predictions inferred by the RR model are closer to the observed references.
(3)
(3)
(B)
Std
Yex 1.646
Rox (110
Fox 6.11
Reu 3:
Foe 0.105
Fax A.11
Sur 8074
For 666
Find
Kv 6.99
Kry 0.139
Fix 8.6
Fax 6:65
Fax
Reference
(17 feee prediction encrepies of X503
(h prstiction examplas TT55
Theee peodiction exumples T2M
Figure 21: MSE value comparisons between RR and Std(ERR for RR and Esta for Std), RR consistently yields lower errors than Std.
likewise uses an autoencoder structure. Visual comparisons are provided in figure 19 where RRAB generates results that are closer the reference. Overall, these results demonstrate the benefits of our pretraining for GANs, and indicate its potential to obtain more generic features from synthetic data sets that can be used for tasks involving real-world data.
C.3 WEATHER FORECASTING
The weather forecasting scenario discussed in the main text follows the method- ology of the WeatherBench benchmark (Rasp et al.) (020)). This benchmark contains 40 years of data from the ERA5 reanalysis project Hersbach et al. (020)) which was re-sampled
Training Details:
5.625 resolu- tion, yielding 32 x 64 grid points in ca. two-hour intervals. Data from the year of 1979 to 2015 (i.e., 1621 14 samples) and 2018) are used as test data. All RMSE measurements are seitudde-eiggteed to account for area distortions from the spherical projection.
used for training. the year of 2016 for validation. The last two years (2017
The neural networks for the forecasting tasks employ a ResNet architecture with 19 layers, all of which contain 128 features with 3 x 3 kernels (apart from batch normalization, leaky ReLU activation (tangent 0.3), and dropout with strength 0.1. As inputs, the model receives feature- wise concatenated data from the WeatherBench data for consecutive time steps, i.e., f, - 6h, and - 12h, yielding 117 channels in total. The last convolution jointly generates all three output fields, i.e., pressure 500 hPa (Z500), temperature 850 hPa (T850), and the 2-meter temperature (T2M).
in the first layer). All layers
Analysis of Results: In addition to the quantitative results for both years test data given in main text, figure 20 and 2T contain additional example visualizations from the test data set. A vi- sualization of the spatial error distribution w.r.t. ground truth result in also shown in figure 21 becomes apparent that our pretraining achieves reduced errors across the whole range of samples. Both temperature targets contain larger number of smaller scale features than the pressure fields. While the gains from pretraining approach are not huge (on the order of 3% in both cases), they represent important steps forward. The learning objective is highly non-trivial, and the improve- were achieved with the same limited set of training data. Being very easy to integrate into existing training pipelines, these results indicate that the proposed pretraining methodology has the potential to yield improved learning results for a wide range of problem settings.
23


Reviews:
[{"review_id": "mHSjWDP76Xn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use orthogonal weight constraints for autoencoders. The authors demonstrate that under orthogonal weights (hence invertible), more features could be extracted. The theory is conducted under linear cases while the authors claim it can be applied to more complicated scenarios such as higher dimension and with nonlinearity. The experiments demonstrate the performance of proposed model on classification tasks and generative tasks. Several baselines are compared.\n\nThe paper is poorly written. It is full of inconsistent and irrelevant claims. The method is not clarified. All experiments are in low quality. \n\n+ves: \n\n+ This paper discusses its connection to several of topics such as mutual information, greedy learning, SVD etc.\n\n \nConcerns: \n\n- No novelty. Using orthogonal weight regularization has been widely studied.\n\n- The theory does not apply to higher dimensional cases or nonlinear cases. The discussion seems trivial. There are no connection of corresponding theory and the model in the experiments.\n\n- Throughout the paper, the \"pretraining\" process is not clarified. \n\n- The authors claims applying their method to GAN but I don't see how they combine their model with it. \n\n- The classification and generation experiment results are not convincing. In Figure 6, 7, the difference are in range of error bar. In Figure 8, 9, there is no advantage from proposed method.\n\n=====POST-REBUTTAL COMMENTS========\n\nI would like to thank the authors for their response. The authors have clarified the method in their response. I appreciate all the experimental details in the appendix. I tend to agree this is a promising idea and worth explored.\n\nHowever, this paper clearly cannot be accepted in its current form. The paper is poorly organized and poorly written. The method in the paper needs to be clarified. Their theory goes nowhere and proves nothing. In terms of the experimental results, the authors choose unclear baselines (which they claim state-of-the-art) and report improvement in terms of percentage increase (percentage over percentage). None of this is convincing to me.\n\nI slightly increased my score.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "innsq3xI-kC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present an approach to pre-training of an ANN which utilizes a purportedly novel approach.  This approach aims to be \"data aware\" by incorporating the individual data points in the orthogonality constraint in the loss function.\n\nThe paper is well-written and his mathematically rigorous.  The mathematics appears to be correct.  It is a concept paper, which I appreciate.\n\nThe paper puts the proposed approach in juxtaposition with the greedy, layer-by-layer approach.\n\nThe authors show improved performance on a handful of tasks against some baseline tasks.  The baselines were, mostly, well selected with a major exception (see below).\n\nThe authors provide some experimental data analysis showing that the features learned by the hidden layers in the proposed approach are more easily interpretable by humans.  Furthermore, the features learned by the baseline features appear to be noisy whereas the features learned by the proposed approach capture the structure seen in the data.  Finally, the authors have some interesting results indicating that the features learned by this approach lead to greater generalization or transferability between tasks/data sets.\n\nMy largest critique of this work is the lack of discussion/comparison with the long-standing auto-encoder approaches which can already compute PCA/SVD/orthogonal basis vectors utilizing a single hidden layer.  Are these methods unrelated in some way?  If so, why?  This work should be tied back to these approaches.\n\nAnother critique of this work is the lack of future work directions.  The authors offered no questions about the proposed approach which were generated during their line of inquiry.  No analysis of where the proposed approach might be inferior to existing approaches was provided or discussed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SNUhhiKXQXf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality\n\nThe paper is well written and mathematically precise. Statements are supported through helpful illustrations and the appendix provides the reader with sufficient information about the experimental setup and evaluations.\n\nClarity\n\nThe approach is clearly motivated and is described in full mathematical detail. The authors also show a highly efficient way to integrate the actually computational very expensive loss into neural networks. The derivation is moderately easy to follow for readers with a background in linear algebra.\n\nOriginality\n\nThe proposed is, while inspired by previous orthogonalization approaches, a novel idea and its relation to previous work is discussed appropriately.\n\nSignificance\n\nUnsupervised pretraining in itself has a large significance for deep learning, even though it lost in popularity due to other approaches that achieved a similar result without the extra preparation phase of the neural network training. It is important and useful to keep the research in this area alive and the authors contributed very valuable knowledge with this paper.\n\nPros:\n\n-- very promising approach for pretraining of neural networks\n\n-- well written paper with good illustrations\n\n-- large experimental evaluation\n\nCons\n\n-- derivation only for networks without activation function\n\n-- evaluation very much focused on image tasks\n\n-- Figure 5: legend hardly readable (kind of true for many figures)\n\n-- source code not referenced (might be due to remain anonymous during review)", "extra_scores": {}, "extra_text": {}}, {"review_id": "M2J_DwsBc92", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an auto-encoder pre-training approach for regularising the neural network parameters, which can be used in many different existing neural models. The proposed approach is build based on the unsupervised auto-encoder pre-training and the orthogonality constraints. A number of classical applications are shown to be improved using the proposed models.\n\n\n\nStrengths of the paper:\n\n1. The proposed approach is a general approach, easy to integrate into the existing models.\n\n2. The experimental results demonstrate the gains in accuracy for original and new tasks below for a wide range of applications\n\n\n\nWeaknesses of the paper:\n\n1. The writing of the paper needs to be improved. Many typos and informal/improper places can be easily found, e.g\n\n   - Left double quotation marks in latex should be $``$, rather than $”$.\n\n   - Paragraph before Eq (9), \"equation 10 \" should be \"equation 8 \", since equation 10 has no $d_{m+1}{′}$.\n\n   - $L^2$ and $L2$, $L_{rr}$ and $L_{RR}$, are confused in this paper.  \n   - CNNs, Autoencoders, and GANs, although are well-known, should cite with the corresponding references.\n\n2. The proposed approach is very simple, which is incremented from (Bansal et al., 2018), hence the contribution is limited.\n\n3. The motivation of the proposed models but not utilize other models for the problem is not clear.\n\n4. My main concern is the organisation of this paper, where the current version is not suitable for general readers in different domains. The preliminary knowledge of the proposed model is unclear. I see at least two key points need to be discussed at the beginning of the paper, i.e. regularisers, and unsupervised auto-encoder pertaining, however, are not well illustrated. \n\n5. The model section should be organised with subsections, where the differences of the preliminary models and the proposed approaches should be explicitly indicated.\n\n6. What is missing when doing the orthogonality constraint, comparing with other regularisation approaches, should be discussed.\n\n7. Comparing the proposed $L_{RR}$ (Eq. (3)) with Eq. (1) (Bansal et al., 2018), the only difference is the weighting/regularizing of parameters by the input data $d_{m}^{I}$. However, the input space could be arbitrarily in value scales, Eq. (3) could punish the large/sharp values of the input space, which may not be desirable for some tasks where the input spaces are very sparse and discrete.  \n\nOverall, this is good work that uses a simple method to improve the generalising capabilities of many models, but there is a number of weaknesses as indicated above.\n\n*************\nUpdates: Thanks for the authors' response. However, not of all my concerns are well-addressed. Based on the current methodology and the novelty of this paper, I remain my overall rating of this paper unchanged.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a practical unsupervised pretraining strategy that does not require layer-wise training stages. Clearly this is an area that has lot of potential and the work seems to head in the right direction. 
However, despite a very positive review, I share the same concerns raised by the remaining 3 reviewers. Better motivation and clarity is needed and, considering the proposed approach, a much more thorough comparison and analysis of the theoretical advantages and guarantees of such an approach.
A main argument is that the method can handle arbitrary networks due to the way it is implemented, it is however not clear how practical that would be and how that would work in presence of non-linearities. 
Experiments require also additional work to be presented with clear and standard baselines, not by presenting SOTA on arbitrary tasks. This seemed to be a main concern of the reviewers.
Decision: Reject