Title: Semantic Noise Modeling for Better Representation Learning
Authors: Hyo-Eun Kim
Sangheum Hwang
Kyunghyun Cho
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: Latent representation learned from multi-layered neural networks via hierarchical feature abstraction enables recent success of deep learning. Under the deep learning framework, generalization performance highly depends on the learned latent representation. In this work, we propose a novel latent space modeling method to learn better latent representation. We designed a neural network model based on the assumption that good base representation for supervised tasks can be attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables. From this base model, we introduce a semantic noise modeling method which enables semantic perturbation on the latent space to enhance the representational power of learned latent feature. During training, latent vector representation can be stochastically perturbed by a modeled additive noise while preserving its original semantics. It implicitly brings the effect of semantic augmentation on the latent space. The proposed model can be easily learned by back-propagation with common gradient-based optimization algorithms. Experimental results show that the proposed method helps to achieve performance benefits against various previous approaches. We also provide the empirical analyses for the proposed latent space modeling method including t-SNE visualization.

TL;DR: A novel latent space modeling method to learn better representation

Review discussion:
The paper proposes a new regularization method for neural network based on maximization of the total correlation between input, hidden variables and output. How does this regularization technique compares to other recently proposed techniques such as adversarial training?
The method is tested for two specific convolutional architectures and particular data extension schemes. How well is the method performing in a more standard setting (e.g. permutation invariant MNIST)?
Thanks for the comments. We summarized the answers below and updated the manuscript.
(1) Comparison to the adversarial training
Training with an adversarial objective function (i.e. adversarial training) also has regularization effect (Goodfellow et al., 2015). The regularization effect comes from perturbed input which lies within the max-norm box around the original input. This perturbed input is generated by “fast gradient sign method” which computes the perturbation greatly causing the change in activation.
Therefore, adversarial training can be seen as “training with hard example mining framework” as noted in the paper.
One significant difference between our proposed method and the adversarial training is that our method achieves regularization effect by directly perturbing the final hidden layer. We do not expect such regularization effect from the adversarial training, since the model should have enough capacity to learn to resist the added adversarial noise (low capacity incurs underfitting). We believe that the regularization effect is maximized if we add class-conditionally perturbed noise on the final hidden layer, since the final hidden layer contains the most significant semantic features. This difference also can be observed via visualization of perturbed examples; our method generates different examples with subtle semantic variations (Fig. 5(b)), but the adversarial training cannot show such examples (see Fig. 1 in Goodfellow et al., 2015).
We think that the two methods have a complementary relationship, so there might exist a synergy in terms of regularization effect if we were to apply both approaches simultaneously.
(2) Verification on a permutation-invariant MNIST setting
We further verify the proposed method on a permutation-invariant MNIST task with a standard neural network; 784-512-256-256-128-10 nodes of fully-connected layers are used. Classification performance is measured against three different sizes of training sets (1k, 2k, and 5k per-class training examples). The final performance is averaged over three different random trials.
“Proposed-perturb (class-conditional)” achieves the best performance among all the configurations; 2.57%, 1.82%, and 1.28% error rates for 1k, 2k, and 5k per-class training examples, respectively. We append all the experimental results below. For each size of training set, six results (three for previous works followed by three for the proposed) are summarized in order; [previous] a feed-forward model, a joint learning model with input reconstruction, and a joint learning model with reconstruction of all the intermediate layers including the input, [proposed] a baseline model, a random stochastic perturbation model, and a class-conditional stochastic perturbation model (the same order described in Table 1).
[1k] 3.35%  2.72%  3.30%  2.82%  2.61%  2.57%
[2k] 2.21%  1.97%  2.32%  1.90%  1.87%  1.82%
[5k] 1.52%  1.38%  1.44%  1.30%  1.28%  1.28%
As X and Z are fixed, the criterion you use is very similar to that optimized via deep CCA (which is maximizing the information Z has that contains the joint info between X and Z, as measured via correlation). Some comparison with this wold be good, as they are both effectively trying to solve the same problem.
CCA is a method for exploring the relationships between two sets of vectors (X1, X2), all measured on the same individual. In a Deep CCA (DCCA) framework, two mappings are constructed with multiple stacked layers of nonlinear transformations, and DCCA learns two deep nonlinear mappings of two views of given data (e.g., left and right halves of an image) that are maximally correlated [ref.1].
In our framework, Z is a latent representation obtained from a learned deep nonlinear mapping of the observation X; Z=f(X).
So, CCA is not directly applicable on X and Z.
However, for argument’s sake, if we use the latent vector Z sampled from the learned model, CCA then can be employed on a pair of the observation X and the sampled Z. Since the Z is already transformed from the corresponding observation X, we cannot guarantee additional benefit from the transformation g1 and g2 via CCA (by maximizing the correlation between g1(X) and g2(f(X))), which might be redundant.
[ref.1] Andrew, Galen, Raman Arora, Jeff A. Bilmes, and Karen Livescu. Deep Canonical Correlation Analysis. In International Conference on Machine Learning (ICML), 2013.
These two approaches are certainly different from each other, but we agree that it'll be interesting to follow up on their similarities and a way to combine them in a single model.
This paper introduces a maximum total correlation procedure, adds a target and then adds noise perturbations.
Technical issues:
The move from (1) to (2) is problematic. Yes it is a lower bound, but by igoring H(Z), equation (2) ignores the fact that H(Z) will potentially vary more significantly that H(Z|Y).
As a result of removing H(Z), the objective (2) encourages Z that are low entropy as the H(Z) term is ignored, doubly so as low entropy Z results in low entropy Z|Y. Yes the -H(X|Z) mitigates against a complete entropy collapse for H(Z), but it still neglects critical terms.
In fact one might wonder if this is the reason that semantic noise addition needs to be done anyway, just to push up the entropy of Z to stop it reducing too much.
In (3) arbitrary balancing paramters lamda_1 and lambda_2 are introduced ex-nihilo - they were not there in (2). This is not ever justified.
Then in (5), a further choice is made by simply adding L_{NLL} to the objective.
But in the supervised case, the targets are known and so turn up in H(Z|Y).
Hence now H(Z|Y) should be conditioned on the targets.
However instead another objective is added again without justification, and the conditional entropy of Z is left disconnected from the data it is to be conditioned on. One might argue the C(X,Y,Z) simply acts as a prior on the networks (and hence implicitly on the weights) that we consider, which is then combined with a likelihood term, but this case is not made.
In fact there is no explicit probabilistic or information theoretic motivation for the chosen objective.
Given these issues, it is then not too surprising that some further things need to be done, such as semantic noise addition to actually get things working properly. It may be the form of noise addition is a good idea, but given the troublesome objective being used in the first place, it is very hard to draw conclusions.
In summary, substantially better theoretical justification of the chosen model is needed, before any reasonable conclusion on the semantic noise modelling can be made.
(1) Ignoring H(Z) from the lower bound
I wonder why H(Z) will be lowered as a result of ignoring H(Z) (based on the term “the objective (2) encourages Z that are low entropy as the H(Z) term is ignored”). We agree with the following term “low entropy Z results in low entropy Z|Y”, since H(Z) >= H(Z|Y).
However, minimizing H(X|Z)+H(Z|Y) (our objective) does not necessarily encourage a decrease in H(Z).
We have empirically confirmed that the assumption (maximizing the lower bound defined without taking H(Z) into consideration) would yield a good base representation for supervised learning. From the experimental results that "proposed-base (without noise addition)" shows better performance than previous approaches, it can be confirmed that the formulation is not the reason that semantic noise addition needs to be done anyway.
I agree, however, that considering H(Z) might be a better model than ignoring H(Z).
But this requires an appropriate prior to P(Z), which may require fairly uncertain assumptions.
(2) L_{NLL} and loss weighting coefficients lambda_1 & lambda_2
Remark that X, Y are an observation and its output, and Z is a latent representation of this framework as already described in the manuscript. From the base feedforward model described in Fig.1(a), we assume that good base representation Z for supervised learning (that’s why we consider the classification loss L_NLL) can be obtained by maximizing total correlation between the variables.
So, eq(2) starts from “maximization of total correlation + L_{NLL}” (we will clarify these contents through manuscript revision).
From [ref.1], it is known that maximization of -H(X|Z) can be formulated as minimization of L_rec(x, x_R) averaged over the observations x.
Similarly, it is easy to show that -H(Z|Y) corresponds to L_rec(z, z_R) averaged over z = f_{theta_1}(x). Since H(X|Z) and H(Z|Y) in eq(2) are respectively proportional to L_rec(x, x_R) and L_rec(z, z_R) in eq(3), lambda_1 and lambda_2 are required.
[ref.1] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research (JMLR), 11:3371–3408, 2010.
(3) Adding the perturbation NLL loss
The addition of supervised loss is to make the model as discriminative as possible when finite capacity exists [ref.2]. We can use the proposed model as is (“proposed-base” in the manuscript), but we empirically show that the addition of supervised loss via class-conditional perturbation (“proposed (class-conditional)” in the manuscript which is the main idea of this paper) generalizes better.
[ref.2] Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In International Conference on Machine Learning (ICML), 2008.
In conclusion, please note that this mathematical model is not a complete analytical model but a motivating model for proposed semantic noise modeling method.
P.S. We are glad to clarify any misunderstanding in our manuscript through active discussion with the reviewers as well as other readers. If there's any unclear point, we kindly ask you to point it out here in the discussion forum.
> However, minimizing H(X|Z)+H(Z|Y) (our objective) does not necessarily encourage a decrease in H(Z).
It does, relative to what the paper claims we want to be minimizing: H(X|Z)+H(Z|Y)-H(Z). This latter encourages higher entropy Z as it explicitly is lowered by higher entropy Z. Ignoring that term necessarily encourages a decrease in H(Z) relative to the original objective.
>We have empirically confirmed that the assumption (maximizing the lower bound defined without taking >H(Z) into consideration) would yield a good base representation for supervised learning. From the >experimental results that "proposed-base (without noise addition)" shows better performance than >previous approaches, it can be confirmed that the formulation is not the reason that semantic noise >addition needs to be done anyway.
It may be that an approach of minimizing H(X|Z)+H(Z|Y); it is just that the current justification appears spurious.
Also the experimental results do not test this per-se, they test a supervised perturbation of this.
> Remark that X, Y are an observation and its output, and Z is a latent representation of this
> framework as already described in the manuscript. From the base feedforward model described in
> Fig.1(a), we assume that good base representation Z for supervised learning (that’s why we consider
> the classification loss L_NLL) can be obtained by maximizing total correlation between the variables.
> So, eq(2) starts from “maximization of total correlation + L_{NLL}” (we will clarify these contents
> through manuscript revision).
So X is an observation and Y is the "output". Here I read that Y is simply the output that is the result of mapping X through the network:.
So effectively, under the data processing inequality the stated objective is to maximize the information available at Y about X (via Z), which is simply biased towards choosing parameters that maximizing the information transmission, under a restricted architecture that prevents identity mapping. I suggest this paper is effectively using this measure as a prior over networks for using in combination with supervised input.
Hence the combination with the NLL.
However at the beginning the paper states we are trying to maximize C(X,Z,Y), where X corresponds to the inputs and Y corresponds to the "outputs" - usually understood to be the predictor of the target.
Hence where the target is known for a given X, the reader might reasonably assume you would want optimize Z for providing the information Z gives about the target T. For a fully observed dataset, this would be similar to a nonlinear CCA, which has been shown to be a latent variable model, which one can interpret as maximizing the information Z contains about X and Y. This is not what is done here, but it is worth being clear in distinguishing that the total correlation formalisms provides a x-data driven prior for good (potentially informative) models, that is then used to combine with the target data during learning to minimize potential confusion.
>From [ref.1], it is known that maximization of -H(X|Z) can be formulated as minimization of L_rec(x,
>x_R) averaged over the observations x.
Similarly, it is easy to show that -H(Z|Y) corresponds to
>L_rec(z, z_R) averaged over z = f_{theta_1}(x). Since H(X|Z) and H(Z|Y) in eq(2) are respectively
>proportional to L_rec(x, x_R) and L_rec(z, z_R) in eq(3), lambda_1 and lambda_2 are required.
Under the justification, the "constants" of proportionality are required rather than an arbitrary lambda_1 and lambda_2.
Furthermore these "constants" are not constant in this setting as the latter is dependent on the density of Z, which is dependent on the upstream network parameters.
Hence the move from (2) to (3) is perhaps questionable on many grounds.
In the stacked denoising autoencoder work, they greedily stack layers.
So in that paper, H(Z) is always constant, and they use that as pretraining rather than including it with the objective. It is reasonable to consider how to use stacked autoencoders in what are effectively semi-supervised settings, as is done in this paper. It is just that the approach used for doing that has many theoretical holes. I do agree that others have used combinations of discriminative and generative models before (inc with hybrid discriminative Boltzmann machines);
however the approach still should be justified.
Indeed in this setting this is interpretable as a combination of a prior energy surface corresponding to "informative" models and a log likelihood term, and a justification in those terms could work.
On to the issue of the semantic noise, I have some difficulty motivating (7). Why should that be the objective?
When proposing an objective to the reader it is most helpful if there is an explanation as to why that should be the objective; here it appears that proposal comes out of nowhere.
The perturbation assumption that the output of the network is a Gaussian perturbed y is problematic, as in most neural networks for classification, the output vector y is a probability vector. Gaussian perturbation does not satisfy probabilistic constraints. I assume this is done on presoftmax values?
But I really don't know what the Gaussian assumption is. What is the random variable this assumption is made about? Clearly not Y|X as that is deterministic. I assume it is \Yhat|X not Y|X contrary to what is said?
But I am not sure.
In the introduction there was talk of a class conditional noise distribution, but there is no class conditioning here (the target class t doesn't even turn up in the noise model). There seems a mismatch between what happens and the description.
Finally the noise process is dependent on the model parameters. This interaction is not discussed, despite the big implications of this for the training. Altogether the noise process is not motivated at all in the paper - it is called "semantic" but I never see a statement regarding what is "semantic" about it.
In summary, this may be a really neat idea.
But I find it hard to see the motivation for each decision, and it is hard for me as a reader to work out if it is a good idea, and if so why is a good idea, beyond "look it works".
So though this may be really good, at the moment there is, at least for me, a lack of motivation, some obfuscation and big jumps that seem to occur in the theoretical basis that lack further discussion.
(issue 1)
>> From reviewer: It does, relative to what the paper claims we want to be minimizing: H(X|Z)+H(Z|Y)-H(Z). This latter encourages higher entropy Z as it explicitly is lowered by higher entropy Z. Ignoring that term necessarily encourages a decrease in H(Z) relative to the original objective.
>> To reviewer: As you say, we should consider carefully about the relative relationship between H(Z) and H(Z|Y) in the problem of H(X|Z) + H(Z|Y) - H(Z) being minimized, since higher H(Z) can help in decreasing H(X|Z) + H(Z|Y) - H(Z).
However, in our formulation to minimize H(X|Z) + H(Z|Y), minimizing H(Z|Y) does not guarantee a lower H(Z). We agree that H(Z) is affected by the process of H(X|Z) + H(Z|Y) being minimized.
However, since H(Z) is the upper bound of H(Z|Y), we cannot conclude that our objective of minimizing H(Z|Y) necessarily reduces H(Z). As we mentioned in the previous answer, we agree that considering H(Z) might be a better model than ignoring H(Z).
But, maximizing the lower bound defined in this paper under the supervised learning model also shows better results compared to other previous approaches.
>> From reviewer: It may be that an approach of minimizing H(X|Z)+H(Z|Y); it is just that the current justification appears spurious.
Also the experimental results do not test this per-se, they test a supervised perturbation of this.
>> To reviewer: As mentioned, it cannot be known how minimizing H(X|Z) + H(Z|Y) will affect H(Z). We have shown from experiments that the lower bound we set can be of benefit to the supervised task and the overall results are better than the previous approaches. From this result, the defined lower bound is acceptable under the supervised task.
(issue 2)
>> From reviewer: So X is an observation and Y is the "output". Here I read that Y is simply the output that is the result of mapping X through the network:.
So effectively, under the data processing inequality the stated objective is to maximize the information available at Y about X (via Z), which is simply biased towards choosing parameters that maximizing the information transmission, under a restricted architecture that prevents identity mapping. I suggest this paper is effectively using this measure as a prior over networks for using in combination with supervised input.
Hence the combination with the NLL.
>> To reviewer: Right. Since our target is supervised learning, we can count of it as a suggestion of a regularization method for better supervised learning.
Therefore, we tried to obtain better regularization effect from maximizing total correlation of X, Z, Y defined in general supervised model which is basically based on a likelihood objective. We defined relatively loose lower bounds, and we have confirmed through experiments that the defined lower bound can help to obtain good regularization effects. We wanted to show this fact via this paper.
In summary, we have experimentally verified that maximization of the defined lower bound functions as a prior that helps us to obtain a better regularization effect for supervised learning.
>> From reviewer: However at the beginning the paper states we are trying to maximize C(X,Z,Y), where X corresponds to the inputs and Y corresponds to the "outputs" - usually understood to be the predictor of the target.
Hence where the target is known for a given X, the reader might reasonably assume you would want optimize Z for providing the information Z gives about the target T. For a fully observed dataset, this would be similar to a nonlinear CCA, which has been shown to be a latent variable model, which one can interpret as maximizing the information Z contains about X and Y. This is not what is done here, but it is worth being clear in distinguishing that the total correlation formalisms provides a x-data driven prior for good (potentially informative) models, that is then used to combine with the target data during learning to minimize potential confusion.
>> To reviewer: Your description is correct. Y is a prediction of a target T, and we aim to find a nonlinear mapping that can get appropriate Z from X so that Y maximally corresponds to T. From this point of view, we can consider it as a nonlinear CCA for X and T, as you say.
However, as you know, our target is supervised learning, where T is the target (e.g., class label) for observation X. Usually, CCA is to find two nonlinear mappings from two different views of some data, where the correlation of the two views of data can be maximized. Two views of data can be the left half (28x14 pixels) and the right half (28x14 pixels) of a 28x28 MNIST image as used in the DCCA paper, or in more general, it can be the X-ray data and MRI data of a patient. We did not consider deeply about the nonlinear CCA approach between observation X and its target label T, but it would be nice to take a close look. (however, I think this is not an issue to be covered in this paper.)
(issue 3)
>> From reviewer: Under the justification, the "constants" of proportionality are required rather than an arbitrary lambda_1 and lambda_2.
Furthermore these "constants" are not constant in this setting as the latter is dependent on the density of Z, which is dependent on the upstream network parameters.
Hence the move from (2) to (3) is perhaps questionable on many grounds.
>> To reviewer: The lower bound defined in this work can be reduced to a sum of two expectations; one is for the negative log-likelihood of X|Z over the observations x, and the other one is for the negative log-likelihood of Z|Y over the z which is a feed-forwarded from corresponding observation x via the mapping f_{theta_1}. As you mentioned, exact ‘constant’ is required rather than the rough lambda’s we defined. We do not set the constants via extensive grid search, but we defined the lambda’s differently for each dataset for dataset-specific calibration of the defined deterministic lower bound.
>> From reviewer: In the stacked denoising autoencoder work, they greedily stack layers.
So in that paper, H(Z) is always constant, and they use that as pretraining rather than including it with the objective. It is reasonable to consider how to use stacked autoencoders in what are effectively semi-supervised settings, as is done in this paper. It is just that the approach used for doing that has many theoretical holes. I do agree that others have used combinations of discriminative and generative models before (inc with hybrid discriminative Boltzmann machines);
however the approach still should be justified.
Indeed in this setting this is interpretable as a combination of a prior energy surface corresponding to "informative" models and a log likelihood term, and a justification in those terms could work.
>> To reviewer: The co-optimization of supervised and unsupervised losses covered in this paper and in various existing papers can be seen as a combination of a prior (unsupervised loss) and a likelihood (supevised loss) as you pointed out, so we will clarify the contents through revision.
(issue 4)
>> From reviewer: On to the issue of the semantic noise, I have some difficulty motivating (7). Why should that be the objective?
When proposing an objective to the reader it is most helpful if there is an explanation as to why that should be the objective; here it appears that proposal comes out of nowhere.
>> To reviewer: Since our purpose is to take better generalization for supervised model, we add an additional likelihood term to make the model as discriminative as possible. The added NLL loss in eq(7) is for the semantic perturbation of latent representation z, the main idea of ​​this paper. As we mentioned in the paper, we conjectured that if the perturbation in latent space guarantees the original semantics well, we could get a better generalization effect on the given examples. We confirmed the expected effect of the proposed idea on the difference between the most commonly considered “random perturbation” and the final proposed “semantic perturbation”.
>> From reviewer: The perturbation assumption that the output of the network is a Gaussian perturbed y is problematic, as in most neural networks for classification, the output vector y is a probability vector. Gaussian perturbation does not satisfy probabilistic constraints. I assume this is done on presoftmax values?
But I really don't know what the Gaussian assumption is. What is the random variable this assumption is made about? Clearly not Y|X as that is deterministic. I assume it is \Yhat|X not Y|X contrary to what is said?
But I am not sure.
>> To reviewer: The gaussian assumption is a very simple assumption to generate a small perturbation around each element of output y. Small random noise can be sampled stochastically from the gaussian distribution, and this sampled noise is added to each element of y to produce perturbed y (yes, y is a pre-softmax logit). We can perform the same process using target t, but because t does not include a semantics difference between intra-class observations, performing perturbation from each output y corresponding to each observation x separately is right.
>> From reviewer: In the introduction there was talk of a class conditional noise distribution, but there is no class conditioning here (the target class t doesn't even turn up in the noise model). There seems a mismatch between what happens and the description.
Finally the noise process is dependent on the model parameters. This interaction is not discussed, despite the big implications of this for the training. Altogether the noise process is not motivated at all in the paper - it is called "semantic" but I never see a statement regarding what is "semantic" about it.
>> To reviewer: Since the latent vector z is the representation that best describes the observation x under the defined architecture, we denote this z as a semantic vector. The term "class-conditional" means that y obtained through supervised learning using label t implicitly becomes a vector representing class label t, which does not mean that we use t as a direct condition in the modeling process. Sorry for the unclear expression. Since output y of an observation x is obtained from z which best represents the semantics of the observation x, we need to generate a perturbation from y, not t, in order to perform a semantic perturbation on z.
(summary)
>> From reviewer: In summary, this may be a really neat idea.
But I find it hard to see the motivation for each decision, and it is hard for me as a reader to work out if it is a good idea, and if so why is a good idea, beyond "look it works".
So though this may be really good, at the moment there is, at least for me, a lack of motivation, some obfuscation and big jumps that seem to occur in the theoretical basis that lack further discussion.
>> To reviewer: We are sorry that we did not clearly explain what we would like to express in our paper. Through this open review process, and through careful discussion with you, it seems to be a good opportunity to explain more clearly what we want to say. We will post a revision that includes a clearer explanation that reflects this discussion. Thank you very much.
An updated version of the paper has been uploaded. This version includes more clear explanation of the base model, reflecting the issues raised by AnnoReviewer3.
The paper presents a new regularization technique for neural networks, which seeks to maximize correlation between input variables, latent variables and outputs. This is achieved by defining a measure of total correlation between these variables and decomposing it in terms of entropies and conditional entropies.
Authors explain that they do not actually maximize the total correlation, but a lower-bound of it that ignores simple entropy terms, and only considers conditional entropies. It is not clearly explained what is the rationale for discarding these entropy terms.
Entropies measures are applying to probability distributions (i.e. this implies that the variables in the model should be random). The link between the conditional entropy formulation and the reconstruction error is not made explicit. In order to link these two views, I would have expected, for example, a noise model for the units of the network.
Later in the paper, it is claimed that the original ladder network is not suitable for supervised learning with small samples, and some empirical results seek to demonstrate this.
But a more theoretical explanation why it is the case would have been welcome.
The MNIST results are shown for a particular convolutional neural network architecture, however, most ladder network results for this dataset have been produced on standard fully-connected architectures. Results for such neural network architecture would have been desirable for more comparability with original ladder neural network results.
First of all, we have made overall revisions to the manuscript including detailed mathematical derivations for the base model (Section 2.1 with Appendix_A1) and clear explanations of the proposed noise modeling method (Section 2.2).
>> From reviewer: The paper presents a new regularization technique for neural networks, which seeks to maximize correlation between input variables, latent variables and outputs. This is achieved by defining a measure of total correlation between these variables and decomposing it in terms of entropies and conditional entropies. Authors explain that they do not actually maximize the total correlation, but a lower-bound of it that ignores simple entropy terms, and only considers conditional entropies. It is not clearly explained what is the rationale for discarding these entropy terms.
>> To reviewer: It is a relatively loose lower bound which is defined without taking H(Z) into consideration. We agree with that H(Z) is anyway affected by the process of H(Z|Y) being minimized in our target objective.
However, H(Z) is an upper bound of H(Z|Y), so minimizing H(Z|Y) does not necessarily encourage a decrease in H(Z). We experimentally show that we can obtain good base model even from this relatively loose lower bound (Section 4.3).
>> From reviewer: Entropies measures are applying to probability distributions (i.e. this implies that the variables in the model should be random). The link between the conditional entropy formulation and the reconstruction error is not made explicit. In order to link these two views, I would have expected, for example, a noise model for the units of the network.
>> To reviewer: The derivation of reconstruction errors from the conditional entropies corresponds to the general auto-encoder framework described in “Stacked denoising autoencoders, JMLR2010”. We added the details of mathematical derivation (including noise models for probabilistic analysis of the neural network model) in the revised manuscript (Appendix_A1).
>> From reviewer: Later in the paper, it is claimed that the original ladder network is not suitable for supervised learning with small samples, and some empirical results seek to demonstrate this.
But a more theoretical explanation why it is the case would have been welcome. The MNIST results are shown for a particular convolutional neural network architecture, however, most ladder network results for this dataset have been produced on standard fully-connected architectures. Results for such neural network architecture would have been desirable for more comparability with original ladder neural network results.
>> To reviewer: We experimented again with the same architecture as the ladder network, and updated the results to the revised manuscript (Appendix_A2). From the results, the ladder network and our model show similar performance in over 200 per-class examples.
However, we can see that the proposed method shows notable performance benefit as the dataset scale goes down to below 200 per-class examples (e.g., in a case of 10 per-class training examples, the proposed method achieves 22.11% of error rate, while the ladder network shows 29.66%). Thanks for the advice.
The paper introduces supervised deep learning with layer-wise reconstruction loss (in addition to the supervised loss) and class-conditional semantic additive noise for better representation learning. Total correlation measure and additional insights from auto-encoder are used to derive layer-wise reconstruction loss and is further combined with supervised loss.
When combining with supervised loss the class-conditional additive noise model is proposed, which showed consistent improvement over the baseline model. Experiments on MNIST and CIFAR-10 datasets while changing the number of training examples per class are done extensively.
The derivation of Equation (3) from total correlation is hacky.
Moreover, assuming graphical model between X, Y and Z, it should be more carefully derived to estimate H(X|Z) and H(Z|Y). The current proposal, encoding Z and Y from X and decoding from encoded representation is not really well justified.
Is \sigma in Equation 8 trainable parameter or hyperparameter? If it is trainable how it is trained? If it is not, how are they set? Does j correspond to one of the class? The proposed feature augmentation sounds like simply adding gaussian noise to the pre-softmax neurons. That being said, the proposed method is not different from gaussian dropout (Wang and Manning, ICML 2013) but applied on different layers.
In addition, there is a missing reference (DisturbLabel: Regularizing CNN on the Loss Layer, CVPR 2016) that applied synthetic noise process on the loss layer.
Experiments should be done for multiple times with different random subsets and authors should provide mean and standard error.
Overall, I believe the proposed method is not very well justified and has limited novelty.
First of all, we have made overall revisions to the manuscript including detailed mathematical derivations for the base model (Section 2.1 with Appendix_A1) and clear explanations of the proposed noise modeling method (Section 2.2).
>> From reviewer: The paper introduces supervised deep learning with layer-wise reconstruction loss (in addition to the supervised loss) and class-conditional semantic additive noise for better representation learning. Total correlation measure and additional insights from auto-encoder are used to derive layer-wise reconstruction loss and is further combined with supervised loss.
When combining with supervised loss the class-conditional additive noise model is proposed, which showed consistent improvement over the baseline model. Experiments on MNIST and CIFAR-10 datasets while changing the number of training examples per class are done extensively.
>> To reviewer: Yes, we applied hierarchical reconstruction losses derived from hierarchical conditional entropies which is obtained from the sum of hierarchical mutual informations. Strictly speaking, the target unsupervised objective for the base supervised learning model consists of the reconstruction loss of the observation x (with x_R reconstructed from the top most latent representation z) and the reconstruction loss of the latent representation z (with z_R reconstructed from the output y); not all of the layer-wise reconstruction losses.
>> From reviewer: The derivation of Equation (3) from total correlation is hacky.
Moreover, assuming graphical model between X, Y and Z, it should be more carefully derived to estimate H(X|Z) and H(Z|Y). The current proposal, encoding Z and Y from X and decoding from encoded representation is not really well justified.
>> To reviewer: The derivation of reconstruction errors from the conditional entropy terms corresponds to the general auto-encoder framework described in “Stacked denoising autoencoders, JMLR2010”. We added the details of mathematical derivation (including noise models for probabilistic analysis of the neural network model) in the revised manuscript (Appendix_A1).
>> From reviewer: Is \sigma in Equation 8 trainable parameter or hyperparameter? If it is trainable how it is trained? If it is not, how are they set? Does j correspond to one of the class? The proposed feature augmentation sounds like simply adding gaussian noise to the pre-softmax neurons. That being said, the proposed method is not different from gaussian dropout (Wang and Manning, ICML 2013) but applied on different layers.
In addition, there is a missing reference (DisturbLabel: Regularizing CNN on the Loss Layer, CVPR 2016) that applied synthetic noise process on the loss layer.
>> To reviewer: As we described in the paper, we assumed that if the perturbation in latent space guarantees to preserve the original semantics well, we could get a better generalization effect on the given training examples.
So, we propose a method for augmenting in the latent space Z (while preserving the original semantics well), not the output space Y. Yes, we simply add gaussian noise to the pre-softmax output neurons in order to obtain perturbed Y.
But, the perturbed Y is used for making the “semantic noise” on the latent space Z (that’s exactly what we want to introduce in this paper) via the reconstruction path of the based model (perturbation on the output space is not directly used for the target loss as in “DisturbLabel, CVPR2016”). From experiments (in Section 4.4), we showed the difference between the most commonly considered “random perturbation directly on Z” and the proposed “semantic perturbation indirectly modeled from the direct perturbation of Y”. “Gaussian dropout, ICML2013” is a kind of a computationally efficient version of the dropout, which drops a subset of hidden units randomly sampled during training. The proposed noise modeling method focuses on semantic-preserving perturbation on the latent space, so it is different from the dropout (or gaussian dropout) approaches.
>> From reviewer: Experiments should be done for multiple times with different random subsets and authors should provide mean and standard error.
Overall, I believe the proposed method is not very well justified and has limited novelty.
>> To reviewer: As described in the original manuscript, Table.1 shows the average of 3 trials with different random splits, and the results according to each trial are summarized in Table.2 (in Appendix_A3). We added std_dev in Table.1. Thanks for the comments.
We have made overall revisions to the manuscript in order to clarify mathematical justification, explanations, and several empirical analyses. Among the revised contents, we especially focused on the methodology part (Section 2). Section 2 is divided into two subsections. Section 2.1 explains the base model as an extension of joint learning approaches. This subsection includes details of derivations for mathematical justification of the base model (See Appendix_A1). Section 2.2 shows the proposed latent space modeling method. This subsection focuses on explaining how to model the semantic-preserving perturbation on the latent space. We are sorry for the incompleteness of the early version of manuscript. We ask the reviewers to revisit the updated manuscript once again.

Meta-review:
role: PC
data: comment: The reviewers all expressed concerns with the technical quality of this work. In particular, the reviewers are concerned that ignoring certain entropy terms in the objective is problematic and would require significantly more justification theoretically and empirically. The reviewers believe that the authors had to resort to unjustified tricks such as adding noise in order to compensate for the missing terms in the objective. Some of the reviewers also had concerns with the choice of experiments, expressing that the authors did not choose the right baseline comparisons to compare to (e.g. convolutional networks vs. fully connected networks on MNIST). Hopefully the thorough feedback and lengthly discussion, along with the authors' responses (both in the text and additions to the paper and appendix), will lead to a stronger submission to a future conference.

Decision: Reject