Title: SEMANTIC NOISE MODELING FOR BETTER REPRESENTATION LEARNING

Abstract: Latent representation learned from multi-layered neural networks via hierarchical feature abstraction enables recent success of deep learning. Under the deep learning framework, generalization performance highly depends on the learned latent representation. In this work, we propose a novel latent space modeling method to learn better latent representation. We designed a neural network model based on the assumption that good base representation for supervised tasks can be attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables. From this base model, we introduce a semantic noise modeling method which enables semantic perturbation on the latent space to enhance the representational power of learned latent feature. During training, latent vector representation can be stochastically perturbed by a modeled additive noise while preserving its original semantics. It implicitly brings the effect of semantic augmentation on the latent space. The proposed model can be easily learned by back-propagation with common gradient-based optimization algorithms. Experimental results show that the proposed method helps to achieve performance benefits against various previous approaches. We also provide the empirical analyses for the proposed latent space modeling method including t-SNE visualization.

1. INTRODUCTION:

Enhancing the generalization performance against unseen data given some sample data is the main objective in machine learning. Under that point of view, deep learning has been achieved many breakthroughs in several domains such as computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016) , natural language processing (Collobert & Weston, 2008; Bahdanau et al., 2015) , and speech recognition (Hinton et al., 2012; Graves et al., 2013) . Deep learning is basically realized on deep layered neural network architecture, and it learns appropriate taskspecific latent representation based on given training data. Better latent representation learned from training data results in better generalization over the future unseen data. Representation learning or latent space modeling becomes one of the key research topics in deep learning. During the past decade, researchers focused on unsupervised representation learning and achieved several remarkable landmarks on deep learning history (Vincent et al., 2010; Hinton et al., 2006; Salakhutdinov & Hinton, 2009) . In terms of utilizing good base features for supervised learning, the base representation learned from unsupervised learning can be a good solution for supervised tasks (Bengio et al., 2007; Masci et al., 2011) .

The definition of 'good' representation is, however, different according to target tasks. In unsupervised learning, a model is learned from unlabelled examples. Its main objective is to build a model to estimate true data distribution given examples available for training, so the learned latent representation normally includes broadly-informative components of the raw input data (e.g., mutual information between the input and the latent variable can be maximized for this objective). In supervised learning, however, a model is learned from labelled examples. In the case of classification, a supervised model learns to discriminate input data in terms of the target task using corresponding labels. Latent representation is therefore obtained to maximize the performance on the target supervised tasks.

Since the meaning of good representations vary according to target tasks (unsupervised or supervised), pre-trained features from the unsupervised model are not be guaranteed to be useful for subsequent supervised tasks. Instead of the two stage learning strategy (unsupervised pre-training followed by supervised fine-tuning), several works focused on a joint learning model which optimizes unsupervised and supervised objectives concurrently, resulting in better generalization performance (Goodfellow et al., 2013; Larochelle & Bengio, 2008a; Rasmus et al., 2015; Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014) .

In this work, we propose a novel latent space modeling method for supervised learning as an extension of the joint learning approach. We define a good latent representation of standard feed-forward neural networks under the basis of information theory. Then, we introduce a semantic noise modeling method in order to enhance the generalization performance. The proposed method stochastically perturbs the latent representation of a training example by injecting a modeled semantic additive noise. Since the additive noise is randomly sampled from a pre-defined probability distribution every training iteration, different latent vectors from a single training example can be fully utilized during training. The multiple different latent vectors produced from a single training example are semantically similar under the proposed latent space modeling method, so we can expect semantic augmentation effect on the latent space.

Experiments are performed on two datasets; MNIST and CIFAR-10. The proposed model results in better classification performance compared to previous approaches through notable generalization effect (stochastically perturbed training examples well cover the distribution of unseen data).

2. METHODOLOGY:

The proposed method starts from the existing joint learning viewpoint. This section first explains the process of obtaining a good base representation for supervised learning which is the basis of the proposed latent space modeling method. And then, we will describe how the proposed semantic noise modeling method perturbs the latent space while maintaining the original semantics.

2.1. BASE JOINT LEARNING MODEL:

In a traditional feed-forward neural network model (Figure 1 (a)), output Y of input data X is compared with its true label, and the error is propagated backward from top to bottom, which implicitly learns a task-specific latent representation Z of the input X. As an extension of a joint learning approach, an objective to be optimized can be described in general as below (Larochelle & Bengio, 2008b) :

min θ λL unsup + L sup (1)

where L unsup and L sup are respectively an unsupervised loss and a supervised loss, and θ and λ are model parameters to be optimized during training and a loss weighting coefficient, respectively. In terms of modeling L unsup in Eq. ( 1), we assume that good latent representation Z is attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables; i.e. the sum of the mutual information between the input X and the Z and the mutual information between the Z and the output Y . Each mutual information is decomposed into an entropy and a conditional entropy terms, so the sum of hierarchical mutual informations is expressed as follows: The total correlation is equal to the sum of all pairwise mutual informations (Watanabe, 1960) .

I(X; Z) + I(Z; Y ) = H(X) -H(X|Z) + H(Z) -H(Z|Y ) (2) X Z Y X Z Y X R Z R X Z Y X R Z R Z P Y P (a) (b) (c)

Our objective is to find the model parameters which maximize I(X; Z) + I(Z; Y ). Since H(X) and H(Z) are non-negative, and H(X) is constant in this case, the lower bound on I(X; Z) + I(Z; Y ) can be reduced to 1 :

EQUATION

It is known that maximizing -H(X|Z) can be formulated as minimizing the reconstruction error between the input x (i) (i-th example sampled from X) and its reconstruction x

(i)

R under the general audo-encoder framework (Vincent et al., 2010) . Since H(X|Z) + H(Z|Y ) is proportional to the sum of reconstruction errors of x (i) (with its reconstruction x (i) R ) and z (i) (with its reconstruction z (i) R ), the target objective can be expressed as follows (refer to Appendix (A1) for the details of mathematical derivations):

EQUATION

where L rec is a reconstruction loss.

Figure 1 (b) shows the target model obtained from the assumption that good latent representation Z can be obtained by maximizing the sum of hierarchical mutual informations. Given an input sample x, feed-forward vectors and their reconstructions are attained deterministically by:

z = f θ1 (x) y = f θ2 (f θ1 (x)) x R = g θ 1 (z) = g θ 1 (f θ1 (x)) z R = g θ 2 (y) = g θ 2 (f θ2 (f θ1 (x)) .

(5)

1 Although H(Z) is an upper bound of H(Z|Y ), H(Z) is anyway affected by the process of H(Z|Y ) being minimized in Eq. (3). In Section 4, we experimentally show that we can obtain good base model even from the relatively loose lower bound defined in Eq. (3).

Given a set of training pairs (x (i) , t (i) ) where x (i) and t (i) are the i-th input example and its label, target objective in Eq. (1) under the model described in Figure 1 (b) can be organized as below (with real-valued input samples, L2 loss L L2 is a proper choice for the reconstruction loss L rec ):

EQUATION

where L N LL is a negative log-likelihood loss for the target supervised task. Note that Eq. ( 6) represents the 'proposed-base' in our experiment (see Section 4.3).

2.2. SEMANTIC NOISE MODELING:

Based on the architecture shown in Figure 1 (b) with the target objective in Eq. ( 6), we conjecture that stochastic perturbation on the latent space during training helps to achieve better generalization performance for supervised tasks. Figure 1 (c) shows this strategy which integrates the stochastic perturbation process during training. Suppose that Z P is a perturbed version of Z, and Y P is an output which is feed-forwarded from Z P . Given a latent vector z = f θ1 (x) from an input sample x, z = z + z e and ŷ = f θ2 (z )

where z and ŷ are a perturbed latent vector and its output respectively, and z e is an additive noise used in the perturbation process of z. Based on the architecture shown in Figure 1 (c), target objective can be modified as:

EQUATION

Using random additive noise directly on z e is the most intuitive approach ('proposed-perturb (random)' in Section 4.3). However, preserving the semantics of the original latent representation z cannot be guaranteed under the direct random perturbation on the latent space. While the latent space is not directly interpretable in general, the output logit y of the latent representation z is interpretable, because the output logit is tightly coupled to the prediction of the target label. In order to preserve the semantics of the original latent representation after perturbation, we indirectly model a semantic noise on the latent space by adding small random noise directly on the output space.

Based on the output (pre-softmax) logit y, the semantic-preserving variation of y (i.e. y ) can be modeled by y = y + y e , where y e is a random noise vector stochastically sampled from a zeromean Gaussian with small standard deviation σ; N (0, σ 2 I). Now, the semantic perturbation z can be reconstructed from the random perturbation y through the decoding path g θ 2 in Figure 1(c) .

From the original output logit y and the randomly perturbed output logit y , semantic additive noise z e on the latent space can be approximately modeled as below:

EQUATION

By using the modeled semantic additive noise z e and the original latent representation z, we can obtain the semantic perturbation z as well as its output ŷ via Eq. ( 7) for our target objective Eq. ( 8).

From the described semantic noise modeling process ('proposed-perturb (semantic)' in Section 4.3), we expect to achieve better representation on the latent space. The effect of the proposed model in terms of learned latent representation will be explained in more detail in Section 4.4. 

3. RELATED WORKS:

Previous works on deep neural networks for supervised learning can be categorized into two types as shown in Figure 2 ; (a) a general feed-forward neural network model (LeCun et al., 1998; Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016) , and (b) a joint learning model which optimizes unsupervised and supervised objectives at the same time (Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014) . Here are the corresponding objective functions:

EQUATION

where λ is a loss weighting coefficient between unsupervised and supervised losses.

Since the feed-forward neural network model is normally implemented with multiple layers in a deep learning framework, the joint learning model can be sub-classified into two types according to the type of reconstruction; reconstruction only with the input data x (Eq. ( 11)) and reconstruction with all the intermediate features including the input data x as follows:

EQUATION

)

where h

(i) j and h (i) j R are the j-th hidden representation of the i-th training example and its reconstruction.

Another type of the joint learning model, a ladder network (Figure 3 ), was introduced for semisupervised learning (Rasmus et al., 2015) . The key concept of the ladder network is to obtain robust features by learning de-noising functions (g θ ) of the representations at every layer of the model via reconstruction losses, and the supervised loss is combined with the reconstruction losses in order to build the semi-supervised model. The ladder network achieved the best performance in semi-supervised tasks, but it is not appropriate for supervised tasks with small-scale training set (experimental analysis for supervised learning on permutation-invariant MNIST is briefly summarized

+ noise + noise

Figure 3 : Ladder network; a representative model for semi-supervised learning (Rasmus et al., 2015) .

in Appendix (A2)). The proposed model in this work can be extended to semi-supervised learning, but our main focus is to enhance the representational power on latent space given labelled data for supervised learning. We leave the study for semi-supervised learning scenario based on the proposed methodology as our future research.

4. EXPERIMENTS:

For quantitative analysis, we compare the proposed methodology with previous approaches described in Section 3; a traditional feed-forward supervised learning model and a joint learning model with two different types of reconstruction losses (reconstruction only with the first layer or with all the intermediate layers including the first layer). The proposed methodology includes a baseline model in Figure 1 In Figure 4 , z is perturbed directly from z by adding Gaussian random noise for random perturbation. For semantic perturbation, z is indirectly generated from y which is perturbed by adding Gaussian random noise on y based on Eq. ( 9). For perturbation, base activation vector (z is the base vector for the random perturbation and y is the base vector for the semantic perturbation) is scaled to [0.0, 1.0], and the zero-mean Gaussian noise with 0.2 of standard deviation is added (via elementwise addition) on the normalized base activation. This perturbed scaled activation is de-scaled with the original min and max activations of the base vector.

Initial learning rates are 0.005 and 0.001 for MNIST and permutation-invariant MNIST, and 0.002 for CIFAR-10, respectively. The learning rates are decayed by a factor of 5 every 40 epochs until the 120-th epoch. For both datasets, the minibatch size is set to 100, and the target objective is optimized using Adam optimizer (Kingma & Ba, 2015) with a momentum 0.9. All the λ's for reconstruction losses in Eq. ( 11) and Eq. ( 12) are 0.03 and 0.01 for MNIST and CIFAR-10, respectively. The same weighting factors for reconstruction losses (0.03 for MNIST and 0.01 for CIFAR-10) are used for λ 1 in Eq (8), and 1.0 is used for λ 2 .

Input data is first scaled to [0.0, 1.0] and then whitened by the average across all the training examples. In CIFAR-10, random cropping (24×24 image is randomly cropped from the original 32×32 image) and random horizontal flipping (mirroring) are used for data augmentation. We selected the network that performed best on the validation dataset for evaluation on the test dataset. All the experiments are performed with TensorFlow (Abadi et al., 2015) . 

5. DISCUSSION:

We introduced a novel latent space modeling method for supervised tasks based on the standard feed-forward neural network architecture. The presented model simultaneously optimizes both supervised and unsupervised losses based on the assumption that the better latent representation can be obtained by maximizing the sum of hierarchical mutual informations. Especially the stochastic perturbation process which is achieved by modeling the semantic additive noise during training enhances the representational power of the latent space. From the proposed semantic noise modeling process, we can expect improvement of generalization performance in supervised learning with implicit semantic augmentation effect on the latent space.

The presented model architecture can be intuitively extended to semi-supervised learning because it is implemented as the joint optimization of supervised and unsupervised objectives. For semisupervised learning, however, logical link between features learned from labelled and unlabelled data needs to be considered additionally. We leave the extension of the presented approach to semisupervised learning for the future.

APPENDIX (A1) DERIVATION OF RECONSTRUCTION ERRORS FROM CONDITIONAL ENTROPY TERMS:

Extended from Section 2. From the lower bound in Eq. ( 3), we consider the following optimization problem (refer to 'Section 2. From mutual information to autoencoders' in (Vincent et al., 2010) ):

EQUATION

Here, we denote q(X, Z, Y ) an unknown joint distribution. Note that Z and Y are respectively the variables transformed from parametric mappings Z = f θ1 (X) and Y = f θ2 (Z) (see Fig. 1 ). q(X, Z, Y ) then can be reduced to q(X) from q(Z|X; θ

1 ) = δ(Z -f θ1 (X)) and q(Y |Z; θ 2 ) = δ(Y -f θ2 (Z))

where δ denotes Dirac-delta function.

From the Kullback-Leibler divergence that D KL (q||p) ≥ 0 for any two distributions p and q, the optimization in Eq. ( 13) corresponds to the following optimization problem where p(•) denotes a parametric distribution:

EQUATION

By replacing q(X) with a sample distribution q 0 (X) and putting all parametric dependencies between X, Z and Y , we will have

EQUATION

For a given input sample x of X, it is general to interpret x R and z R as the parameters of distributions p(X|X R = x R ) and p(Z|Z R = z R ) which reconstruct x and z with high probability (i.e. x R and z R are not exact reconstructions of x and z). Since x R and z R are real-valued, we assume Gaussian distribution for these conditional distributions, that is,

EQUATION

The assumptions yield

-log p(•|•) ∝ L L2 (•, •).

With the following relations for log terms in Eq. ( 15),

EQUATION

the optimization problem in Eq. ( 15) corresponds to the minimization problem of reconstruction errors for input examples x (i) as below:

EQUATION

(A2) LADDER NETWORK, A REPRESENTATIVE SEMI-SUPERVISED LEARNING MODEL Extended from Section 3. We performed experiments with a ladder network model (Rasmus et al., 2015) in order to estimate the performance on pure supervised tasks according to different sizes of training set. We used the code (https://github.com/rinuboney/ladder.git) for this experiment. The network architecture implemented on the source code is used as is; (784-1000-500-250-250-250-10) . Based on the same network architecture, we implemented the proposed stochastic perturbation model described in Figure 1 (c) and compared the classification performance with the ladder network as described in Table 2 (we did not focus on searching the optimal hyperparameters for the proposed model in this experiment). As summarized in the bottom of the table (mean over 3 random trials), the proposed semantic noise modeling method shows a fairly large performance gain compared to the ladder network model with small-scale datasets (e.g., in a case of 10 per-class training examples, the proposed method achieves 22.11% of error rate, while the ladder network shows 29.66%). 10) . Further experiments with denoising constraints are also included. Zero-mean Gaussian random noise with 0.1 standard deviation is used for noise injection. Denoising function helps to achieve slightly better performance on MNIST, but it results in performance degradation on CIFAR-10 (we did not focus on searching the optimal parameters for noise injection in this experiments). 

:



Figures and Tables:

Figure 1: (a) Standard feed-forward neural network model, (b) feed-forward neural network model with reconstruction paths, and (c) feed-forward neural network model with reconstruction and stochastic perturbation paths.

Figure 2: Previous works for supervised learning; (a) traditional feed-forward model, and (b) joint learning model with both supervised and unsupervised losses.

Figure 4 shows the architecture of the neural network model used in this experiment. W 's are convolution or fully-connected weights (biases are excluded for visual brevity). Three convolution (3×3 (2) 32, 3×3 (2) 64, 3×3 (2) 96, where each item means the filter kernel size and (stride) with the number of filters) and two fully-connected (the numbers of output nodes are 128 and 10, respectively) layers are used for MNIST. For the permutation-invariant MNIST setting, 784-512-256-256-128-10 nodes of fully-connected layers are used. Four convolution (5×5 (1) 64, 3×3 (2) 64, 3×3 (2) 64, and 3×3 (2) 96) and three fully-connected (128, 128, and 10 nodes) layers are used for CIFAR-10. Weights on the decoding (reconstruction) path are tied with corresponding weights on the encoding path as shown in Figure 4 (transposed convolution for the tied convolution layer and transposed matrix multiplication for the tied fully-connected layer).

Figure 4: Target network architecture; 3 convolution and 2 fully-connected layers were used for MNIST, 5 fully-connected layers were used for permutation-invariant MNIST, and 4 convolution and 3 fully-connected layers were used for CIFAR-10.

Figure 2(a), joint learning model with recon-one Figure 2(b), joint learning model with recon-all Figure 2(b), respectively) and the proposed methods (proposed-1, 2, 3 are baseline Figure 1(b), random perturbation Figure 1(c), semantic perturbation Figure 1(c), respectively) is summarized.

QUANTITATIVE ANALYSIS Three previous approaches (a traditional feed-forward model, a joint learning model with the input reconstruction loss, and a joint learning model with reconstruction losses of all the intermediate layers including the input layer) are compared with the proposed methods (the baseline model in Figure 1(b), and the stochastic perturbation model in Figure 1(c) with two different perturbation methods; random and semantic). We measure the classification performance according to varying sizes of training set (examples randomly chosen from the original training dataset). Performance is averaged over three different random trials.

Figure 5: Examples reconstructed from the perturbed latent vectors via (a) random perturbation, and (b) semantic perturbation (top row shows the original training examples). More examples are summarized in Appendix (A4.1).

QUALITATIVE ANALYSISAs mentioned before, random perturbation by adding unstructured noise directly to the latent representation cannot guarantee preserving the semantics of the original representation. We compared two different perturbation methods (random and semantic) by visualizing the examples reconstructed from the perturbed latent vectors (Figure5). Top row is the original examples selected from training set (among 2k per-class training examples), and the rest are the reconstructions of their perturbed latent representations. Based on the architecture described in Figure1(b), we generated five different perturbed latent representations according to the type of perturbation, and reconstructed the perturbed latent vectors through decoding path for reconstruction.

Figure5(a) and(b)  show the examples reconstructed from the random and semantic perturbations, respectively. For both cases, zero-mean Gaussian random noise (0.2 standard deviation) is used for perturbation. As shown in Figure5(a), random perturbation partially destroys the original semantics; for example, semantics of '1' is mostly destroyed under random perturbation, and some examples of '3' are reconstructed as being similar to '8' rather than its original content '3'. Figure5(b)shows the examples reconstructed from the semantic perturbation. The reconstructed examples show subtle semantic variations while preserving the original semantic contents; for example, thickness difference in '3' (example on the third row) or writing style difference in '8' (openness of the top left corner).

Figure 6 shows the overall effect of the perturbation. In this analysis, 100 per-class MNIST examples are used for training. From the trained model based on the architecture described in Figure 1(b), latent representations z of all the 50k examples (among 50k examples, only 1k examples were used for training) are visualized by using t-SNE (Maaten & Hinton, 2008). Only the training examples of three classes (0, 1, and 9) among ten classes are depicted as black circles for visual discrimination in

Figure 6: Training examples (circles or crosses with colors described below) over the examples not used for training (depicted as background with different colors); (a) training examples (black circles), (b) training examples (yellow circles) with 3× random-perturbed samples (blue crosses), and (c) training examples (yellow circles) with 3× semantic-perturbed samples (blue crosses). Best viewed in color.

Figure 6(a). The rest of the examples which were not used for training (approximately 4.9k examples per class) are depicted as a background with different colors. We treat the colored background examples (not used for training) as a true distribution of unseen data in order to estimate the generalization level of learned representation according to the type of perturbation. Figure 6(b) and (c) show the training examples (100 examples per class with yellow circles) and their perturbed ones (3× sampled from each example with blue crosses) through random and semantic perturbations, respectively.In Figure6(b), perturbed samples are distributed near the original training examples, but some samples outside the true distribution cannot be identified easily with appropriate classes. This can be explained with Figure5(a), since some perturbed samples are ambiguous semantically. In, however, most of the perturbed samples evenly cover the true distribution. As mentioned before, stochastic perturbation with the semantic additive noise during training implicitly incurs the effect of augmentation on the latent space while resulting in better generalization. Per-class t-SNE results are summarized in Appendix (A4.2).

Figure 7: For each example, top row is the original examples selected from the training set, and the rest are reconstructed from the perturbed representations via random (left) and semantic (right) perturbations.

Error rate (%) on the test set using the model with the best performance on the validation set. Numbers on the first row of each sub-table are the number of randomly chosen per-class training examples. The average performance and the standard deviation of three different random-split datasets (except for the case using the entire training set in the last column) are described in this table (error rate on each random set is summarized in Appendix (A3)). Performance of three previous approaches (with gray background; previous-1, 2, 3 are feed-forward model

Table content: <table/>



Table content: <table><tr><td>the</td></tr></table>

Classification performance (error rate in %) of the ladder network and the proposed model on three different sets of randomly chosen training examples (MNIST). Among the total 50k and 40k training examples in MNIST and CIFAR-10, we randomly select the examples for training. Classification performance according to three different randomly chosen training sets are summarized inTable 3 (MNIST) and Table 4 (CIFAR-

Table content: <table><tr><td>set No.1 (# training examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td>(all) 5k</td></tr><tr><td>ladder network model; Figure 3</td><td>25.85</td><td>16.48</td><td>9.26</td><td>6.00</td><td>4.66</td><td>3.07</td><td>2.15</td><td>1.26</td><td>0.91</td></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>19.76</td><td>12.33</td><td>8.77</td><td>6.06</td><td>4.59</td><td>2.93</td><td>1.87</td><td>1.31</td><td>0.93</td></tr><tr><td>set No.2 (# training examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>ladder network model; Figure 3</td><td>33.14</td><td>17.46</td><td>10.44</td><td>6.67</td><td>4.43</td><td>2.82</td><td>1.94</td><td>1.37</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>23.36</td><td>15.35</td><td>9.43</td><td>5.75</td><td>4.43</td><td>2.99</td><td>1.87</td><td>1.39</td><td/></tr><tr><td>set No.3 (# training examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>ladder network model; Figure 3</td><td>29.99</td><td>16.99</td><td>9.73</td><td>7.34</td><td>4.39</td><td>3.00</td><td>2.12</td><td>1.47</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>23.21</td><td>13.98</td><td>8.83</td><td>6.51</td><td>4.32</td><td>2.94</td><td>2.22</td><td>1.49</td><td/></tr><tr><td>mean over 3 random trials</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td>(all) 5k</td></tr><tr><td>ladder network model; Figure 3</td><td>29.66</td><td>16.98</td><td>9.81</td><td>6.67</td><td>4.49</td><td>2.96</td><td>2.07</td><td>1.37</td><td>0.91</td></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>22.11</td><td>13.89</td><td>9.01</td><td>6.11</td><td>4.45</td><td>2.95</td><td>1.99</td><td>1.40</td><td>0.93</td></tr></table>

Classification performance (error rate in %) on three different sets of randomly chosen training examples (MNIST).

Table content: <table><tr><td>Set No.1 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td>(all) 5k</td></tr><tr><td>feed-forward model; Figure 2(a)</td><td>22.61</td><td>14.20</td><td>11.25</td><td>6.37</td><td>4.34</td><td>2.63</td><td>1.83</td><td>1.56</td><td>1.04</td></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>18.69</td><td>12.21</td><td>7.84</td><td>5.17</td><td>4.02</td><td>2.58</td><td>1.79</td><td>1.47</td><td>1.12</td></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>20.39</td><td>11.91</td><td>7.41</td><td>4.64</td><td>3.65</td><td>2.57</td><td>1.97</td><td>1.53</td><td>0.97</td></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>18.82</td><td>12.82</td><td>9.34</td><td>6.43</td><td>5.23</td><td>4.12</td><td>2.68</td><td>2.42</td><td>1.87</td></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>17.93</td><td>11.76</td><td>7.32</td><td>4.78</td><td>3.91</td><td>3.04</td><td>2.52</td><td>1.99</td><td>1.36</td></tr><tr><td>proposed-base; Figure 1(b)</td><td>20.23</td><td>10.18</td><td>6.47</td><td>3.89</td><td>3.04</td><td>1.89</td><td>1.33</td><td>0.91</td><td>0.80</td></tr><tr><td>proposed-base with denoising constraints</td><td>19.88</td><td>10.89</td><td>6.62</td><td>4.26</td><td>3.40</td><td>2.44</td><td>2.11</td><td>1.54</td><td>1.13</td></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>18.38</td><td>10.58</td><td>6.64</td><td>3.78</td><td>3.14</td><td>1.90</td><td>1.21</td><td>0.89</td><td>0.65</td></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>19.33</td><td>9.72</td><td>5.98</td><td>3.47</td><td>2.84</td><td>1.84</td><td>1.16</td><td>0.84</td><td>0.62</td></tr><tr><td>Set No.2 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>feed-forward model; Figure 2(a)</td><td>28.84</td><td>17.36</td><td>10.14</td><td>6.20</td><td>4.78</td><td>3.02</td><td>1.61</td><td>1.41</td><td/></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>26.09</td><td>14.40</td><td>7.98</td><td>5.18</td><td>4.17</td><td>2.29</td><td>1.94</td><td>1.52</td><td/></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>27.69</td><td>13.11</td><td>6.95</td><td>5.07</td><td>3.54</td><td>2.37</td><td>1.83</td><td>1.28</td><td/></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>24.01</td><td>14.13</td><td>8.98</td><td>6.84</td><td>5.44</td><td>3.51</td><td>2.98</td><td>2.18</td><td/></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>23.05</td><td>13.29</td><td>7.79</td><td>5.12</td><td>3.92</td><td>3.01</td><td>2.27</td><td>1.84</td><td/></tr><tr><td>proposed-base; Figure 1(b)</td><td>22.95</td><td>12.98</td><td>6.27</td><td>4.43</td><td>3.22</td><td>2.14</td><td>1.37</td><td>0.96</td><td/></tr><tr><td>proposed-base with denoising constraints</td><td>26.96</td><td>12.21</td><td>6.45</td><td>4.62</td><td>3.13</td><td>2.53</td><td>1.88</td><td>1.49</td><td/></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>22.10</td><td>12.52</td><td>5.97</td><td>4.26</td><td>2.86</td><td>1.94</td><td>1.23</td><td>0.92</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>21.22</td><td>11.52</td><td>5.75</td><td>3.91</td><td>2.61</td><td>1.73</td><td>1.14</td><td>0.89</td><td/></tr><tr><td>Set No.3 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>feed-forward model; Figure 2(a)</td><td>22.20</td><td>16.43</td><td>9.67</td><td>7.16</td><td>5.02</td><td>3.17</td><td>2.25</td><td>1.39</td><td/></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>20.23</td><td>14.19</td><td>7.73</td><td>5.96</td><td>4.22</td><td>2.62</td><td>1.79</td><td>1.35</td><td/></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>19.32</td><td>12.25</td><td>7.44</td><td>5.39</td><td>3.58</td><td>2.37</td><td>1.49</td><td>1.56</td><td/></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>17.51</td><td>14.12</td><td>9.12</td><td>7.04</td><td>5.49</td><td>4.05</td><td>3.08</td><td>2.25</td><td/></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>17.07</td><td>12.50</td><td>7.86</td><td>5.48</td><td>4.05</td><td>2.97</td><td>2.02</td><td>1.98</td><td/></tr><tr><td>proposed-base; Figure 1(b)</td><td>20.86</td><td>11.79</td><td>6.25</td><td>4.63</td><td>2.96</td><td>1.91</td><td>1.16</td><td>0.96</td><td/></tr><tr><td>proposed-base with denoising constraints</td><td>19.89</td><td>11.30</td><td>6.26</td><td>4.57</td><td>3.50</td><td>2.63</td><td>1.61</td><td>1.47</td><td/></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>20.02</td><td>11.94</td><td>6.12</td><td>4.32</td><td>3.13</td><td>1.81</td><td>1.28</td><td>1.08</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>19.78</td><td>10.53</td><td>6.03</td><td>4.00</td><td>2.70</td><td>1.76</td><td>1.14</td><td>0.92</td><td/></tr></table>

Classification performance (error rate in %) on three different sets of randomly chosen training examples (CIFAR-10).

Table content: <table><tr><td>Set No.1 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td>(all) 4k</td></tr><tr><td>feed-forward model; Figure 2(a)</td><td>73.30</td><td>69.25</td><td>62.42</td><td>55.65</td><td>47.71</td><td>34.30</td><td>27.04</td><td>21.06</td><td>17.80</td></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>75.19</td><td>70.38</td><td>62.25</td><td>55.30</td><td>46.89</td><td>34.12</td><td>26.63</td><td>21.05</td><td>17.68</td></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>73.72</td><td>68.20</td><td>61.99</td><td>55.23</td><td>46.64</td><td>36.37</td><td>29.78</td><td>25.53</td><td>21.73</td></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>74.79</td><td>68.33</td><td>62.92</td><td>56.24</td><td>51.37</td><td>40.30</td><td>30.91</td><td>26.49</td><td>22.71</td></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>76.56</td><td>69.67</td><td>64.53</td><td>57.88</td><td>52.74</td><td>42.24</td><td>36.90</td><td>30.93</td><td>27.41</td></tr><tr><td>proposed-base; Figure 1(b)</td><td>70.79</td><td>66.57</td><td>59.91</td><td>52.98</td><td>43.29</td><td>32.25</td><td>26.19</td><td>20.92</td><td>17.45</td></tr><tr><td>proposed-base with denoising constraints</td><td>71.03</td><td>67.49</td><td>60.37</td><td>53.52</td><td>44.28</td><td>33.40</td><td>28.00</td><td>25.06</td><td>21.34</td></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>71.89</td><td>67.12</td><td>59.22</td><td>52.79</td><td>43.87</td><td>31.82</td><td>25.04</td><td>20.97</td><td>17.43</td></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>71.59</td><td>66.90</td><td>58.64</td><td>52.34</td><td>42.74</td><td>30.94</td><td>24.45</td><td>20.10</td><td>16.16</td></tr><tr><td>Set No.2 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>feed-forward model; Figure 2(a)</td><td>72.39</td><td>69.49</td><td>60.45</td><td>54.85</td><td>46.91</td><td>33.39</td><td>26.73</td><td>21.00</td><td/></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>74.06</td><td>69.14</td><td>60.71</td><td>54.54</td><td>45.70</td><td>33.54</td><td>27.43</td><td>20.90</td><td/></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>76.40</td><td>69.33</td><td>60.28</td><td>55.38</td><td>47.40</td><td>36.29</td><td>29.31</td><td>24.60</td><td/></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>72.28</td><td>67.60</td><td>61.53</td><td>56.65</td><td>49.99</td><td>42.08</td><td>32.99</td><td>26.33</td><td/></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>73.90</td><td>69.23</td><td>61.90</td><td>57.99</td><td>52.35</td><td>45.12</td><td>37.23</td><td>30.14</td><td/></tr><tr><td>proposed-base; Figure 1(b)</td><td>72.49</td><td>65.62</td><td>57.82</td><td>52.66</td><td>43.20</td><td>32.24</td><td>25.60</td><td>21.32</td><td/></tr><tr><td>proposed-base with denoising constraints</td><td>72.99</td><td>66.75</td><td>57.78</td><td>53.81</td><td>44.33</td><td>33.56</td><td>28.40</td><td>25.03</td><td/></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>71.84</td><td>65.98</td><td>58.08</td><td>53.37</td><td>43.44</td><td>31.56</td><td>25.69</td><td>21.03</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>72.85</td><td>66.65</td><td>57.44</td><td>52.21</td><td>42.74</td><td>31.17</td><td>24.99</td><td>20.54</td><td/></tr><tr><td>Set No.3 (# train examples per class)</td><td>10</td><td>20</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1k</td><td>2k</td><td/></tr><tr><td>feed-forward model; Figure 2(a)</td><td>75.78</td><td>68.24</td><td>61.02</td><td>54.29</td><td>46.28</td><td>33.38</td><td>26.11</td><td>20.85</td><td/></tr><tr><td>joint learning model with recon-one; Figure 2(b)</td><td>77.79</td><td>67.62</td><td>61.37</td><td>55.22</td><td>45.96</td><td>33.21</td><td>26.29</td><td>21.81</td><td/></tr><tr><td>joint learning model with recon-one with denoising constraints</td><td>76.60</td><td>69.27</td><td>61.13</td><td>55.10</td><td>47.50</td><td>37.12</td><td>29.63</td><td>24.88</td><td/></tr><tr><td>joint learning model with recon-all; Figure 2(b)</td><td>72.92</td><td>66.97</td><td>63.31</td><td>56.23</td><td>50.16</td><td>41.41</td><td>33.75</td><td>26.31</td><td/></tr><tr><td>joint learning model with recon-all with denoising constraints</td><td>76.83</td><td>68.53</td><td>65.58</td><td>58.29</td><td>52.43</td><td>45.42</td><td>39.01</td><td>32.32</td><td/></tr><tr><td>proposed-base; Figure 1(b)</td><td>71.60</td><td>66.31</td><td>58.99</td><td>52.30</td><td>43.88</td><td>31.10</td><td>25.48</td><td>20.95</td><td/></tr><tr><td>proposed-base with denoising constraints</td><td>72.39</td><td>67.20</td><td>60.60</td><td>52.64</td><td>44.62</td><td>33.52</td><td>28.01</td><td>25.25</td><td/></tr><tr><td>proposed-perturb (random); Figure 1(c)</td><td>71.34</td><td>67.15</td><td>59.55</td><td>52.86</td><td>43.81</td><td>32.01</td><td>25.78</td><td>20.42</td><td/></tr><tr><td>proposed-perturb (semantic); Figure 1(c)</td><td>70.06</td><td>67.07</td><td>58.83</td><td>52.41</td><td>43.47</td><td>30.61</td><td>25.00</td><td>19.94</td><td/></tr></table>

Peer review (iclr2017):
The paper presents a new regularization technique for neural networks, which seeks to maximize correlation between input variables, latent variables and outputs. This is achieved by defining a measure of total correlation between these variables and decomposing it in terms of entropies and conditional entropies. Authors explain that they do not actually maximize the total correlation, but a lower-bound of it that ignores simple entropy terms, and only considers conditional entropies. It is not clearly explained what is the rationale for discarding these entropy terms. Entropies measures are applying to probability distributions (i.e. this implies that the variables in the model should be random). The link between the conditional entropy formulation and the reconstruction error is not made explicit. In order to link these two views, I would have expected, for example, a noise model for the units of the network. Later in the paper, it is claimed that the original ladder network is not suitable for supervised learning with small samples, and some empirical results seek to demonstrate this. But a more theoretical explanation why it is the case would have been welcome. The MNIST results are shown for a particular convolutional neural network architecture, however, most ladder network results for this dataset have been produced on standard fully-connected architectures. Results for such neural network architecture would have been desirable for more comparability with original ladder neural network results.