Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces supervised deep learning with layer-wise reconstruction loss (in addition to the supervised loss) and class-conditional semantic additive noise for better representation learning. Total correlation measure and additional insights from auto-encoder are used to derive layer-wise reconstruction loss and is further combined with supervised loss. When combining with supervised loss the class-conditional additive noise model is proposed, which showed consistent improvement over the baseline model. Experiments on MNIST and CIFAR-10 datasets while changing the number of training examples per class are done extensively.

The derivation of Equation (3) from total correlation is hacky. Moreover, assuming graphical model between X, Y and Z, it should be more carefully derived to estimate H(X|Z) and H(Z|Y). The current proposal, encoding Z and Y from X and decoding from encoded representation is not really well justified.

Is \sigma in Equation 8 trainable parameter or hyperparameter? If it is trainable how it is trained? If it is not, how are they set? Does j correspond to one of the class? The proposed feature augmentation sounds like simply adding gaussian noise to the pre-softmax neurons. That being said, the proposed method is not different from gaussian dropout (Wang and Manning, ICML 2013) but applied on different layers. In addition, there is a missing reference (DisturbLabel: Regularizing CNN on the Loss Layer, CVPR 2016) that applied synthetic noise process on the loss layer.

Experiments should be done for multiple times with different random subsets and authors should provide mean and standard error. Overall, I believe the proposed method is not very well justified and has limited novelty. 
review_title: null

final_score: rating: null
review_content: The paper proposes a new regularization method for neural network based on maximization of the total correlation between input, hidden variables and output. How does this regularization technique compares to other recently proposed techniques such as adversarial training?

The method is tested for two specific convolutional architectures and particular data extension schemes. How well is the method performing in a more standard setting (e.g. permutation invariant MNIST)?
review_title: benchmark results

final_score: rating: 2: Strong rejection
review_content: This paper introduces a maximum total correlation procedure, adds a target and then adds noise perturbations.

Technical issues:

The move from (1) to (2) is problematic. Yes it is a lower bound, but by igoring H(Z), equation (2) ignores the fact that H(Z) will potentially vary more significantly that H(Z|Y). As a result of removing H(Z), the objective (2) encourages Z that are low entropy as the H(Z) term is ignored, doubly so as low entropy Z results in low entropy Z|Y. Yes the -H(X|Z) mitigates against a complete entropy collapse for H(Z), but it still neglects critical terms. In fact one might wonder if this is the reason that semantic noise addition needs to be done anyway, just to push up the entropy of Z to stop it reducing too much.

In (3) arbitrary balancing paramters lamda_1 and lambda_2 are introduced ex-nihilo - they were not there in (2). This is not ever justified.

Then in (5), a further choice is made by simply adding L_{NLL} to the objective. But in the supervised case, the targets are known and so turn up in H(Z|Y). Hence now H(Z|Y) should be conditioned on the targets. However instead another objective is added again without justification, and the conditional entropy of Z is left disconnected from the data it is to be conditioned on. One might argue the C(X,Y,Z) simply acts as a prior on the networks (and hence implicitly on the weights) that we consider, which is then combined with a likelihood term, but this case is not made. In fact there is no explicit probabilistic or information theoretic motivation for the chosen objective.

Given these issues, it is then not too surprising that some further things need to be done, such as semantic noise addition to actually get things working properly. It may be the form of noise addition is a good idea, but given the troublesome objective being used in the first place, it is very hard to draw conclusions.

In summary, substantially better theoretical justification of the chosen model is needed, before any reasonable conclusion on the semantic noise modelling can be made.
review_title: Semantic noise modelling

final_score: rating: null
review_content: As X and Z are fixed, the criterion you use is very similar to that optimized via deep CCA (which is maximizing the information Z has that contains the joint info between X and Z, as measured via correlation). Some comparison with this wold be good, as they are both effectively trying to solve the same problem.
review_title: comparison with nonlinear CCA

final_score: rating: 3: Clear rejection
review_content: The paper presents a new regularization technique for neural networks, which seeks to maximize correlation between input variables, latent variables and outputs. This is achieved by defining a measure of total correlation between these variables and decomposing it in terms of entropies and conditional entropies.

Authors explain that they do not actually maximize the total correlation, but a lower-bound of it that ignores simple entropy terms, and only considers conditional entropies. It is not clearly explained what is the rationale for discarding these entropy terms.

Entropies measures are applying to probability distributions (i.e. this implies that the variables in the model should be random). The link between the conditional entropy formulation and the reconstruction error is not made explicit. In order to link these two views, I would have expected, for example, a noise model for the units of the network.

Later in the paper, it is claimed that the original ladder network is not suitable for supervised learning with small samples, and some empirical results seek to demonstrate this. But a more theoretical explanation why it is the case would have been welcome.

The MNIST results are shown for a particular convolutional neural network architecture, however, most ladder network results for this dataset have been produced on standard fully-connected architectures. Results for such neural network architecture would have been desirable for more comparability with original ladder neural network results.
review_title: unclear relation between the total correlation maximization idea, and the actual training scheme based on local reconstructions

Meta-review: The reviewers all expressed concerns with the technical quality of this work. In particular, the reviewers are concerned that ignoring certain entropy terms in the objective is problematic and would require significantly more justification theoretically and empirically. The reviewers believe that the authors had to resort to unjustified tricks such as adding noise in order to compensate for the missing terms in the objective. Some of the reviewers also had concerns with the choice of experiments, expressing that the authors did not choose the right baseline comparisons to compare to (e.g. convolutional networks vs. fully connected networks on MNIST). Hopefully the thorough feedback and lengthly discussion, along with the authors' responses (both in the text and additions to the paper and appendix), will lead to a stronger submission to a future conference.
Decision: Reject