Title: SEMI-SUPERVISED LEARNING VIA NEW DEEP NETWORK INVERSION

Abstract: We exploit a recently derived inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework that applies to a wide range of systems and problems. The approach reaches current state-of-the-art methods on MNIST and provides reasonable performances on SVHN and CIFAR10. Through the introduced method, residual networks are for the first time applied to semi-supervised tasks. Experiments with one-dimensional signals highlight the generality of the method. Importantly, our approach is simple, efficient, and requires no change in the deep network architecture.

1. INTRODUCTION:

Deep neural networks (DNNs) have made great strides recently in a wide range of difficult machine perception tasks. They consist of parametric functionals f Θ with internal parameters Θ. However, those systems are still trained in a fully supervised fashion using a large set of labeled data, which is tedious and costly to acquire.

Semi-supervised learning relaxes this requirement by leaning Θ based on two datasets: a labeled set D of N training data pairs and an unlabeled set D u of N u training inputs. Unlabeled training data is useful for learning as unlabeled inputs provide information on the statistical distribution of the data that can both guide the learning required to classify the supervised dataset and characterize the unlabeled samples in D u hence improve generalization. Limited progress has been made on semi-supervised learning algorithms for DNNs Rasmus et al. (2015) ; Salimans et al. (2016) ; Patel et al. (2015) , but today's methods suffer from a range of drawbacks, including training instability, lack of topology generalization, and computational complexity.

In this paper, we take two steps forward in semi-supervised learning for DNNs. First, we introduce an universal methodology to equip any deep neural net with an inverse that enables input reconstruction. Second, we introduce a new semi-supervised learning approach whose loss function features an additional term based on this aforementioned inverse guiding weight updates such that information contained in unlabeled data are incorporated into the learning process. Our key insight is that the defined and general inverse function can be easily derived and computed; thus for unlabeled data points we can both compute and minimize the error between the input signal and the estimate provided by applying the inverse function to the network output without extra cost or change in the used model. The simplicity of this approach, coupled with its universal applicability promise to significantly advance the purview of semi-supervised and unsupervised learning.

1.1. RELATED WORK:

The standard approach to DNN inversion was proposed in Dua & Gupta (2000) , and the only DNN model with reconstruction capabilities is based on autoencoders Ng (2011) . While more complex topologies have been used, such as stacked convolutional autoencoder Masci et al. (2011) , there exists two main drawbacks. Firstly, the difficulty to train complex models in a stable manner even when using per layer optimization. Secondly, the difficulty to leverage supervised information into the learning of the representation.

The problem of semi-supervised learning with DNN has been attempted by several groups. Firstly, GAN based methods are a natural way to tackle this problem. The improved generative adversarial network (GAN) technique Salimans et al. (2016) couples two deep networks: a generative model that can create new signal samples, and a discriminative network. Both neural nets are trained jointly as in typical GAN framework. Triple Generative Adversarial Nets Li et al. (2017) propose a extension of the GAN framework for the particular task of semi-supervised by introduction of a third player. The task thus becomes simpler as there exist one discriminator labeling images (fake or real) and another predicting if the couples (image,label) are fake or not. Through this, better stability is reached. Finally, Good Semi-supervised Learning That Requires a Bad GAN Dai et al. (2017) currently hold SOTA method. This work lessen the same problem of GAN for semi-supervised of Li et al. (2017) by deriving analytical conditions and better formulation for the GAN objective hence providing a finer loss function as opposed to a third network.

The semi-supervised with ladder network approach Rasmus et al. (2015) employs a per-layer denoising reconstruction loss, which enables the system to be viewed as a stacked denoising autoencoder which is a standard and until now only way to tackle unsupervised tasks. By forcing the last denoising autoencoder to output an encoding describing the class distribution of the input, this deep unsupervised model is turned into a semi-supervised model. The main drawback of this method is the lack of a clear path to generalize it to other network topologies, such as recurrent or residual networks. Also, the per-layer "greedy" reconstruction loss might be too restrictive unless correctly weighted pushing the need for a precise and large cross-validation of hyper-parameters. The probabilistic formulation of deep convolutional nets presented in Patel et al. (2015) natively supports semi-supervised learning. The main drawbacks of this approach lies in the requirement that the activation functions be ReLU and that the overall network topology follows a deep convolutional network. Temporal Ensembling for Semi-Supervised Learning Laine & Aila (2016) propose to constrain the representations of a same input stimuli to be identical in the latent space despite the presence of dropout noise. This search of stability in the representation is analogous to the one of a siamese network Hoffer & Ailon (2015) but instead of presenting two different inputs, the same is used through two different models (induced by dropout). This technique provides an explicit loss for the unsupervised examples leading to the Π model just described and a more efficient method denoted as temporal ensembling. Distributional Smoothing with Virtual Adversarial Training Miyato et al. (2015) proposes also a regularization term contraining the regularity of the DNN mapping for a given sample. Based on this a semi-supervised setting is derived by imposing for the unlabeled samples to maintain a stable DNN. Those two last described methods are the closest one of the proposed approach in this paper for which, the DNN stability will be replaced by a reconstruction ability, closely related to the DNN stability.

1.2. CONTRIBUTIONS:

This paper makes two main contributions: First, we propose a simple way to invert any piecewise differentiable mapping, including DNNs. We provide a formula for the inverse mapping of any DNN that requires no change to its structure. The mapping is computationally optimal, since the input reconstruction is computed via a backward pass through the network (as is used today for weight updates via backpropagation). Second, we develop a new optimization framework for semisupervised learning that features penalty terms that leverage the input reconstruction formula. A range of experiments validate that our method improves significantly on the state-of-the-art for a number of different DNN topologies.

2. DEEP NEURAL NETWORK INVERSION:

In this section we review the work of Balestriero & Baraniuk (2017) aiming at interpreting DNNs as linear splines. This interpretation provides a rigorous mathematical justification for the reconstruction in the context of deep learning.

2.1. DEEP NEURAL NETS AS LINEAR SPLINE OPERATORS:

Recent work in Balestriero & Baraniuk (2017) demonstrated that DNNs of many topologies are or can be approximated arbitrary closely by multivariate linear splines. The upshot of this theory for this paper is that it enables one to easily derive an explicit input-output mapping formula. As a result, DNNs can be rewritten as a linear spline of the form

EQUATION

where we denote by f Θ a general DNN, x a generic input, A[x], b[x] the spline parameters conditioned on the input induced activations. Based on this interpretation, DNNs can be considered as template matching machines, where A[x] plays the role of an input-adaptive template.

To illustrate this point we provide for two common topologies the exact input-output mappings. For a standard deep convolutional neural network (DCN) with succession of convolutions, nonlinearities, and pooling, one has

EQUATION

where z ( ) (x) represents the latent representation at layer for input x. The total number of layers in a DNN is denoted as L and the output of the last layer z (L) (x) is the one before application of the softmax application. After application of the latter, the output is denoted by ŷ(x). The product terms are from last to first layer as the composition of linear mappings is such that layer 1 is applied on the input, layer 2 on the output of the previous one and so on. The bias term results simply from the accumulation of all of the per-layer biases after distribution of the following layers' templates. For a Resnet DNN, one has

EQUATION

We briefly observe the differences between the templates of the two topologies. The presence of an extra term in

1 =L-1 A ( ) σ,in C ( ) in + C ( ) out as opposed to 1 =L-1 A ( ) ρ A ( )

σ C ( ) provides stability and a direct linear connection between the input x and all of the inner representations z ( ) (x), hence providing much less information loss sensitivity to the nonlinear activations.

Based on those findings, and by imposing a simple 2 norm upper bound on the templates, it has been shown that the optimal templates DNNs to perform prediction has templates proportional to the input, positively for the belonging class and negatively for the others Balestriero & Baraniuk (2017) . This way, the loss cross-entropy function is minimized when using softmax final nonlinearity. Note that this result is specific to this setting. For example in the case of spherical softmax the optimal templates become null for the incorrect classes of the input.

Theorem 1. In the case where all inputs have identity norm ||X n || = 1, ∀n and assuming all templates denoted by A

[X n ] c , c = 1, . . . , C have a norm constraint C c=1 ||A[X n ] c || 2 ≤ K, ∀X n , then the unique globally optimal templates are A * [X n ] c =    C-1 C KX n , if c = Y n - K C(C-1) X n , else. (4)

We now leverage the analytical optimal DNN solution to demonstrate that reconstruction is indeed implied by such an optimum.

2.2. OPTIMAL DNN LEADS TO INPUT RECONSTRUCTION:

Based on the previous analysis, it is possible to draw implications based on the theoretical optimal templates of DNNs. This is formulated through the corollary below. First, we propose the following inverse of a DNN as

f -1 Θ (X n ) =A[X n ] T (A[X n ]X n + b[X n ]) = C c=1 ( A[X n ] c , X n + b[X n ] c )A[X n ] c .

(5)

Following the analysis from a spline point of view, this reconstruction is leveraging the closest input hyperplane, found through the forward step, to represent the input. As a result this method provides a reconstruction based on the DNN representation of its input and should be part away from the task of exact input reconstruction which is an ill-posed problem in general.

The bias correction present has insightful meaning when compared to known frameworks and their inverse. In particular, when using ReLU based nonlinearities we will see that this scheme can be assimilated to a composition of soft-thresholding denoising technique. We present further details in the next section where we also provide ways to efficiently invert a network as well as describing the semi-supervised application.

2.3. IMPLEMENTATION AND NEW LOSS FUNCTION FOR SEMI-SUPERVISED LEARNING:

We now apply the above inverse strategy to a given task with an arbitrary DNN. As exposed earlier, all the needed changes to support semi-supervised learning happen in the objective training function by adding extra terms. In our application, we used automatic differentiation (as in TheanoBergstra et al. ( 2010) and TensorFlowAbadi et al. ( 2016)). Then it is sufficient to change the objective loss function, and all the updates are adapted via the change in the gradients for each of the parameters. The efficiency of our inversion scheme is due to the fact that any deep network can be rewritten as a linear mapping Balestriero & Baraniuk (2017) . This leads to a simple derivation of a network inverse defined as f -1 that will be used to derive our unsupervised and semi-supervised loss function via

EQUATION

The main efficiency argument thus comes from

EQUATION

which enables one to efficiently compute this matrix on any deep network via differentiation (as it would be done to back-propagate a gradient, for example). Interestingly for neural networks and many common frameworks such as wavelet thresholding, PCA, etc., the reconstruction error as

( df (x) dx ) T f (x)

is the definition of the inverse transform. For illustration purposes, Tab. 1 gives some common frameworks for which the reconstruction error represents exactly the reconstruction loss. 

<x,W i > ||W i || 2 ||x -i αi df (x) i dx || 2 + λ||α||1 NMF Learned < x, Wi > ||x -i αi df (x) i dx || 2 s.t. J f (x) ≥ 0 PCA f (x)i < x, Wi > ||x -i αi df (x) i dx || 2 s.t. J f (x) orth. Soft Wavelet Thres. f (x)i max | < x, Wi > | -bi, 0 sig(< x, Wi >) ||x -i αi df (x) i dx || 2 Hard Wavelet Thres. f (x)i 1 |<x,W i >|-b i >0 < x, Wi > ||x -i αi df (x) i dx || 2 Best Basis (WTA) f (x)i 1 i=argmax i <x,W i > ||W i || 2 < x, Wi > ||x -i αi df (x) i dx || 2 k-NN 1 1 i=argmax<x,W i >-||W i || 2 /2 < x, Wi > ||x -i αi df (x) i dx || 2

We now describe how to incorporate this loss for semi-supervised and unsupervised learning. We first define the reconstruction loss R as

EQUATION

)

While we use the mean squared error, any other differentiable reconstruction loss can be used, such as cosine similarity. We also introduce an additional "specialization" loss defined as the Shannon entropy of the class belonging probability prediction

EQUATION

)

This loss is intuitive and complementary to the reconstruction for the semi-supervised task. In fact, it will force a clustering of the unlabeled examples toward one of the clusters learned from the supervised loss and examples. We provide below experiments showing the benefits of this extraterm. As a result, we define our complete loss function as the combination of the standard cross entropy loss for labeled data denoted by L CE (Y n , ŷ(X n )), the reconstruction loss, and the entropy loss as

EQUATION

with α, β ∈ [0, 1] 2 . The parameters α, β are introduced to form a convex combination of the losses,with α controlling the ratio between supervised and unsupervised loss and β the ratio between the two unsupervised losses. This weighting is important, because the correct combination of the supervised and unsupervised losses will guide learning toward a better optimum (as we now demonstrated via experiments).

3. EXPERIMENTAL VALIDATION:



3.1. SEMI-SUPERVISED EXPERIMENTS:

We now present results of our approach on a semi-supervised task on the MNIST dataset, where we are able to obtain reasonable performances with different topologies. MNIST is made of 70000 grayscale images of shape 28 × 28 which is split into a training set of 60000 images and a test set of 10000 images. We present results for the case with N L = 50 which represents the number of samples from the training set that are labeled and fixed for learning. All the others samples form the training set are unlabeled and thus used only with the reconstruction and entropy loss minimization. We perform a search over (α, β) ∈ {0.3, 0.4, 0.5, 0.6, 0.7} × {0.2, 0.3, 0.5}. In addition, 4 different topologies are tested and, for each, mean and max pooling are tested as well as inhibitor DNN (IDNN) as proposed in Balestriero & Baraniuk (2017) . The latter proposes to stabilize training and remove biases units via introduction of winner-share-all connections. As would be expected based on the templates differences seen in the previous section, the Resnet topologies are able to reach the best performance. In particular, wide Resnet is able to outperform previous state-of-the-art results.

Running the proposed semi-supervised scheme on MNIST leads to the results presented in Tab. 2. We used the Theano and Lasagne libraries; and learning procedures and topologies are detailed in the appendix.The column 1000 corresponds to the accuracy after training of DNNs using only the supervised loss (α = 1, β = 0) on 1000 labeled data. Thus, one can see the gap reached with the same network but with a change of loss and 20 times less labeled data.

We further present performances on CIFAR10 with 4000 labeled data in Tab. 3 and SVHN with 500 labeled data in Tab. 4. For both tasks we constrain ourselves to a deep CNN models, similar as the LargeCNN of Patel et al. ( 2015). Also, one of the cases correspond to the absence of entropy loss when β = 1. Furthermore to further present the generalization of the inverse technique we provide results with the leaky-ReLU nonlinearity as well as the sigmoid activation function. 

3.2. SUPERVISED REGULARIZATION FOR IMPROVED GENERALIZATION:

We now present and example of our approach on a supervised task on audio database (1D). It is the Bird10 dataset distributed online and described in Glotin et al. (2017) . The task is to classify 10 bird species from their songs recorded in tropical forest. It is a subtask of the BirdLifeClef challenge. CIFAR4000, CIFAR8000, Improved GAN Salimans et al. (2016) 81.37 ± 2.32 82.28 ± 1.82 LadderNetwork Rasmus et al. (2015) 79.6 ± 0.47 -catGAN Springenberg (2015) 80.42 ± 0.46 -DRMM +KL penalty Patel et al. (2015) 76.76 -Triple GAN Li et al. (2017) 83.01 ± 0.36 -Semi-Sup Requires a Bad GAN Dai et al. (2017) We train here networks based on raw audio using CNNs as detailed in the appendix. We vary (α, β) over 10 runs to demonstrate that the non-regularized supervised model is not optimal. The maximum validation accuracies on the last 100 epochs (Fig. 2 ) show that the regularized networks tend to learn more slowly, but always generalize better than the not regularized baseline (α = 1, β = 0).

4. CONCLUSIONS AND FUTURE WORK:

We have presented a well-justified inversion scheme for deep neural networks with an application to semi-supervised learning. By demonstrating the ability of the method to best current state-of-theart results on MNIST with different possible topologies support the portability of the technique as well as its potential. These results open up many questions in this yet undeveloped area of DNN inversion, input reconstruction, and their impact on learning and stability.

Among the possible extensions, one can develop the reconstruction loss into a per-layer reconstruction loss. Doing so, there is the possibility to weight each layer penalty bringing flexibility as well as meaningful reconstruction. Define the per layer loss as

EQUATION

with

EQUATION

)

Doing so, one can adopt a strategy in favor of high reconstruction objective for inner layers, close to the final latent representation z (L) in order to lessen the reconstruction cost for layers closer to the input X n . In fact, inputs of standard dataset are usually noisy, with background, and the object of interest only contains a small energy with respect to the total energy of X n . Another extension would be to update the weighting while performing learning. in time such as the current epoch or batch, we now have the previous loss becoming

L(X n , Y n ; Θ) = α(t)L CE (Y n , ŷ(X n ))1 {Yn =∅} + β(t)E(X n )1 {Yn=∅} + L-1 =0 γ ( ) (t)R ( ) (X n ).

(13) One approach would be to impose some deterministic policy based on heuristic such as favoring reconstruction at the beginning to then switch to classification and entropy minimization. Finer approaches could rely on explicit optimization schemes for those coefficients. One way to perform this, would be to optimize the loss weighting coefficients α, β, γ ( ) after each batch or epoch by backpropagation on the updates weights. Define

EQUATION

as a generic iterative update based on a given policy such as gradient descent. One can thus adopt the following update strategy for the hyper-parameters as γ ( ) (t + 1) = γ ( ) (t) -dL(X n , Y n ; Θ(t + 1)) dγ ( ) (t) ,

and so for all hyper-parameters. Another approach would be to use adversarial training to update those hyper-parameters where both update cooperate trying to accelerate learning.

EBGAN (Zhao et al. (2016) ) are GANs where the discriminant network D measures the energy of a given input X. D is formulated such as generated data produce high energy and real data produce lower energy. Same authors propose the use of an auto-encoder to compute such energy function.

We plan to replace this autoencoder using our proposed method to reconstruct X and compute the energy; hence D(X) = R(X) and only one-half the parameters will be needed for D.

Finally, our approach opens the possibility of performing unsupervised tasks such as clustering.

In fact, by setting α = 0, we are in a fully unsupervised framework. Moreover, β can push the mapping f Θ to produce a low-entropy, clustered, representation or rather simply to produce optimal reconstruction. Even in a fully unsupervised and reconstruction case (α = 0, β = 1), the proposed framework is not similar to a deep-autoencoder for two main reasons. First, there is no greedy (per layer) reconstruction loss, only the final output is considered in the reconstruction loss. Second, while in both case there is parameter sharing, in our case there is also "activation" sharing that corresponds to the states (spline) that were used in the forward pass that will also be used for the backward one. In a deep autoencoder, the backward activation states are induced by the backward projection and will most likely not be equal to the forward ones.

:

ACKNOWLEDGMENTWe thank PACA region and NortekMed, and GDR MADICS CNRS EADM action for their support.

Figures and Tables:

Figure 1: Reconstruction of the studied DNN models for Resnet2-32 test set samples. The columns from left to right correspond to: the original image, mean-pooling reconstruction, max-pooling reconstruction, inhibitor connections.

Figure 2: Top: Accuracy for Bird10 baseline (α = 1, β = 0), versus regularized networks. Bottom Left: Learning curves of the baseline model. Bottom Right: Learning curve of the best test (α = 0.7, β = 0.3).

Examples of frameworks with similar inverse transform definition.

Table content: <table><tr><td/><td>αi</td><td>f (x)i</td><td>loss</td></tr><tr><td>Sparse Coding</td><td>Learned</td><td/></tr></table>

Accuracy on MNIST test set for 50 labeled examples in the training. The column N L = X gives the accuracy of the same networks trained with a X labeled samples and the remaining of the training set as unlabeled examples.

Table content: <table><tr><td>N L</td><td>50,</td><td>100,</td><td>Sup1000,</td></tr><tr><td/><td>(α, β)</td><td/><td>(α = 1, β = 0)</td></tr><tr><td>SmallCNNmean</td><td>99.07,(0.7, 0.2)</td><td>-</td><td>94.9</td></tr><tr><td>SmallCNNmax</td><td>98.63,(0.7, 0.2)</td><td>-</td><td>95.0</td></tr><tr><td>SmallUCNN</td><td>98.85,(0.5, 0.2)</td><td>-</td><td>96.0</td></tr><tr><td>LargeCNNmean</td><td>98.63,(0.6, 0.5)</td><td>-</td><td>94.7</td></tr><tr><td>LargeCNNmax</td><td>98.79,(0.7, 0.5)</td><td>-</td><td>94.8</td></tr><tr><td>LargeUCNN</td><td>98.23,(0.5, 0.5)</td><td>-</td><td>96.1</td></tr><tr><td>Resnet2-32mean</td><td>99.11,(0.7, 0.2)</td><td>-</td><td>95.5</td></tr><tr><td>Resnet2-32max</td><td>99.14,(0.7, 0.2)</td><td>-</td><td>94.9</td></tr><tr><td>UResnet2-32</td><td>98.84,(0.7, 0.2)</td><td>-</td><td>95.6</td></tr><tr><td>Resnet3-16mean</td><td>98.67,(0.7, 0.2)</td><td>-</td><td>95.4</td></tr><tr><td>Resnet3-16max</td><td>98.56,(0.5, 0.5)</td><td>-</td><td>94.8</td></tr><tr><td>UResnet3-16</td><td>98.7,(0.7, 0.2)</td><td>-</td><td>95.5</td></tr><tr><td>(Mean,Std)</td><td>(98.768, 0.249)</td><td>-</td><td>(95.266, 0.462)</td></tr><tr><td>Improved GAN</td><td>97.79 ± 1.36</td><td>99.07 ± 0.065</td><td>-</td></tr><tr><td>Salimans et al. (2016)</td><td/><td/><td/></tr><tr><td>Auxiliary Deep Generative</td><td>-</td><td>99.04</td><td>-</td></tr><tr><td>Model Maaløe et al. (2016)</td><td/><td/><td/></tr><tr><td>LadderNetwork</td><td>-</td><td>98.94 ± 0.37</td><td>-</td></tr><tr><td>Rasmus et al. (2015)</td><td/><td/><td/></tr><tr><td>Skip Deep Generative Model</td><td>-</td><td>98.68</td><td>-</td></tr><tr><td>Maaløe et al. (2016)</td><td/><td/><td/></tr><tr><td>Virtual Adversarial</td><td>-</td><td>97.88</td><td>-</td></tr><tr><td>Miyato et al. (2015)</td><td/><td/><td/></tr><tr><td>catGAN Springenberg (2015)</td><td>-</td><td>98.61 ± 0.28</td><td>-</td></tr><tr><td>DGN Kingma et al. (2014)</td><td>-</td><td>96.67 ± 0.14</td><td>-</td></tr><tr><td>DRMM Patel et al. (2015)</td><td>-</td><td>87.97</td><td>-</td></tr><tr><td>Triple GAN Li et al. (2017)</td><td>-</td><td>99.09 ± 0.58</td><td>-</td></tr><tr><td>Semi-Sup Requires a Bad</td><td>-</td><td>99.2±0.09</td><td>-</td></tr><tr><td>GAN Dai et al. (2017)</td><td/><td/><td/></tr></table>

Accuracy on CIFAR10 test set for 4000 labeled examples in the training. Presented results include the average and standard deviation over 5 runs with random sampling of the labeled examples among the training set.

Table content: <table><tr><td>N L</td><td colspan="2">ReLU-CIFAR4000, Sigmoid-CIFAR4000,</td></tr><tr><td>(0.7,0.1)</td><td>76.97 ± 0.39</td><td>74.81 ± 0.29</td></tr><tr><td>(0.7,0.2)</td><td>77.75±0.25</td><td>75.25±0.31</td></tr><tr><td>(0.7,0.3)</td><td>76.42 ± 0.43</td><td>74.39 ± 0.42</td></tr><tr><td>(0.8,0.1)</td><td>76.77 ± 0.32</td><td>75.02 ± 0.49</td></tr><tr><td>(0.8,0.2)</td><td>76.62 ± 0.33</td><td>75.08 ± 0.3</td></tr><tr><td>(0.8,0.3)</td><td>76.98 ± 0.07</td><td>74.44 ± 0.09</td></tr><tr><td>(0.85,1)</td><td>75.28 ± 0.2</td><td>73.05 ± 0.4</td></tr><tr><td/><td>N L</td><td/></tr></table>

Accuracy on SVHN test set for 500 labeled examples in the training. Presented results include the average and standard deviation over 5 runs with random sampling of the labeled examples among the training set.

Table content: <table><tr><td/><td>85.59 ± 0.30</td><td>-</td></tr><tr><td>ΠModelLaine &amp; Aila (2016)</td><td>83.45 ± 0.29</td><td>-</td></tr></table>

Hence, if we denote by t the position

Table content: <table><tr><td/><td>α</td><td>β</td><td/><td/><td colspan="2">train mean (std)</td><td/><td/><td/><td colspan="2">valid mean (std)</td><td colspan="3">test mean (std)</td></tr><tr><td/><td>1</td><td>0</td><td/><td/><td colspan="2">90.41 (4.106)</td><td/><td/><td/><td colspan="2">53.01 (2.534)</td><td colspan="3">51.40 (1.198)</td></tr><tr><td/><td colspan="15">0.7 0.5 still being computed computing still being computed still being computed</td></tr><tr><td/><td colspan="2">0.7 0.4</td><td/><td/><td colspan="2">91.88 (1.041)</td><td/><td/><td/><td colspan="2">51.76 (1.401)</td><td colspan="3">52.23 (1.700)</td></tr><tr><td/><td colspan="2">0.7 0.3</td><td/><td/><td colspan="2">92.24 (1.166)</td><td/><td/><td/><td colspan="2">52.45 (2.853)</td><td colspan="3">51.53 (1.441)</td></tr><tr><td/><td colspan="2">0.7 0.2</td><td/><td/><td colspan="2">91.56 (1.287)</td><td/><td/><td/><td colspan="2">52.24 (1.935)</td><td colspan="3">51.77 (1.826)</td></tr><tr><td/><td colspan="2">0.7 0.1</td><td/><td/><td colspan="2">91.15 (1.101)</td><td/><td/><td/><td colspan="2">52.87 (3.538)</td><td colspan="3">52.04 (1.738)</td></tr><tr><td/><td colspan="2">0.6 0.5</td><td/><td colspan="4">still being computed</td><td colspan="8">still being computed still being computed</td></tr><tr><td/><td colspan="2">0.6 0.4</td><td/><td/><td colspan="2">91.63 (1.391)</td><td/><td/><td/><td colspan="2">51.97 (3.316)</td><td colspan="3">52.00 (1.756)</td></tr><tr><td/><td colspan="2">0.6 0.3</td><td/><td/><td colspan="2">91.51 (1.259)</td><td/><td/><td/><td colspan="2">51.43 (1.933)</td><td colspan="3">52.95 (1.239)</td></tr><tr><td/><td colspan="2">0.6 0.2</td><td/><td/><td colspan="2">92.41 (0.9555)</td><td/><td/><td/><td colspan="2">50.32 (1.207)</td><td colspan="3">52.07 (1.216)</td></tr><tr><td/><td colspan="2">0.6 0.1</td><td/><td/><td colspan="2">90.27 (3.050)</td><td/><td/><td/><td colspan="2">53.11 (3.604)</td><td colspan="3">51.74 (1.631)</td></tr><tr><td/><td colspan="2">0.5 0.5</td><td/><td colspan="4">still being computed</td><td colspan="8">still being computed still being computed</td></tr><tr><td/><td colspan="2">0.5 0.4</td><td/><td/><td colspan="2">91.52 (0.9859)</td><td/><td/><td/><td colspan="2">52.21 (2.968)</td><td colspan="3">51.92 (0.990)</td></tr><tr><td/><td colspan="2">0.5 0.3</td><td/><td/><td colspan="2">90.50 (1.341)</td><td/><td/><td/><td colspan="2">52.04 (3.335)</td><td colspan="3">51.18 (0.937)</td></tr><tr><td/><td colspan="2">0.5 0.2</td><td/><td/><td colspan="2">90.13 (1.901)</td><td/><td/><td/><td colspan="2">52.71 (3.431)</td><td colspan="3">51.46 (1.390)</td></tr><tr><td/><td colspan="2">0.5 0.1</td><td/><td/><td colspan="2">88.52 (3.152)</td><td/><td/><td/><td colspan="2">53.86 (3.579)</td><td colspan="3">51.19 (1.479)</td></tr><tr><td/><td>1.0</td><td/><td/><td/><td>baseline</td><td/><td/><td/><td/><td/><td colspan="3">loss + reconstruction</td><td/></tr><tr><td/><td>0.9</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Accuracy on segments</td><td>0.5 0.6 0.7</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.4</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Train set Validation set</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Test set</td></tr><tr><td/><td>0.2</td><td>0</td><td>50</td><td>100</td><td>150</td><td>200</td><td>250</td><td>300</td><td>0</td><td>50</td><td>100</td><td>150</td><td>200</td><td>250</td><td>300</td></tr><tr><td/><td/><td/><td/><td/><td>Epoch number</td><td/><td/><td/><td/><td/><td colspan="2">Epoch number</td><td/><td/></tr></table>

Peer review (iclr2018):
 After reading the revision: The authors addressed my detailed questions on experiments. It appears sometimes the entropy loss (which is not the main contribution of the paper) is essential to improve performance; this obscures the main contribution. On the other hand, the theoretical part of the paper is not really improved in my opinion, I still can not see how previous work by Balestriero and Baraniuk 2017 motivates and backups the proposed method. My rating of this paper would remain the same. ============================================================================================ This paper propose to use the reconstruction loss, defined in a somewhat unusual way, as a regularizar for semi-supervised learning. Pros: The intuition is that the ReLU network output is locally linear for each input, and one can use the conjugate mapping (which is also linear) for reconstructing the inputs, as in PCA. Realizing that the linear mapping is the derivative of network output w.r.t. the input (the Jacobian), the authors proposed to use the reconstruction loss defined in (8). Different from typical auto-encoders, this work does not require another reconstruction network, but instead uses the "derivative". This observation is neat in my opinion, and does suggest a different use of the Jacobian in deep learning. The related work include auto-encoders where the weights of symmetric layers are tied. Cons: The motivation (Section 2) needs to be improved. In particular, the introduction/review of the work of Balestriero and Baraniuk 2017 not very useful to the readers. Notations in eqns (2) and (3) are not fully explained (e.g., boldface c). Intuition and implications of Theorem 1 is not sufficiently discussed: what do you mean by optimal DNN, what is the criteria for optimality? is there a generative assumption of the data underlying the theorem? and the assumption of all samples being norm 1 seems too strong and perhaps limits its application? As far as I see, section 2 is somewhat detached from the rest of the paper. The main contribution of this paper is supposed to be the reconstruction mapping (6) and its effect in semi-supervised learning. The introduction of entropy regularization in sec 2.3 seems somewhat odd and obscures the contribution. It also bears the questions that how important is the entropy regularization vs. the reconstruction loss. In experiments, results with beta=1.0 need to be presented to assess the importance of network inversion and the reconstruction loss. Also, a comparison against typical auto-encoders (which uses another decoder networks, with weights possibly tied with the encoder networks) is missing.