Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
Based on ideas within the context of kernel theory, the authors consider post-training of NNs as an extra training step, which only optimizes the last layer of the network.
This additional step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task (which is also reflected in the experiments).

According to the authors, the contributions are the following:
1. Post-training step: keeping the rest of the NN frozen (after training), the method trains the last layer in order to "make sure" that the representation learned is used in the most efficient way.
2. Highlighting connections with kernel techniques and RKHS optimization (like kernel ridge regression).
3. Experimental results.

Clarity:
The paper is well-written, the main ideas well-clarified. 

Importance:
While the majority of papers nowadays focuses on the representation part (i.e., how we get to \Phi_{L-1}(x)), this paper assumes this is given and proposes how to optimize the weights in the final step of the algorithm. This by itself is not enough to boost the performance universally (e.g., if \Phi_{L-1} is not well-trained, the problem is deeper than training the last layer); however, it proposes an additional step that can be used in most NN architectures. From that front (i.e., proposing to do something different than simply training a NN), I find the paper interesting, that might attract some attention at the conference.

On the other hand, to my humble opinion, the experimental results do not show a significant gain in the performances of all networks (esp. Figure 3 and Table 1 are within the range of statistical error). In order to state something like this universally, either one needs to perform experiments with more than just MNIST/CIFAR datasets, or even more preferably, prove that the algorithm performs better.

Originality:
It would be great to have some more theory (if any) for the post-training step, or investigate more cases, rather than optimizing only the last layer.

Comments:
1. I assume the authors focused in the last layer of the NN for simplicity, but is there a reason why one might want to focus only on the last layer? One reason is convexity in W of the problem (2). Any other? 

2. Have the authors considered (even in practice only) to include training of the last 2 layers of the NN? The authors state this question in the future direction, but it would make the paper more complete to consider it here.

review_title: Overall, this is a nice idea, with experimental results showing its merits. Some of the empirical outcomes do not show that much improvement. However it is a step that could be used either way, even for a slight gain. It would have been more preferable to consider the future directions stated in the paper in this work.

final_score: rating: 3: Clear rejection
review_content: This paper proposes to fine-tune the last layer while keeping the others fixed, after initial end-to-end training, viewing the last layer learning under the light of kernel theory (well actually it's just a linear model).

Summary of evaluation

There is not much novelty in this idea (of optimizing carefully only the last layer as a post-training stage or treating the last layer as kernel machine in a post-processing step), which dates back at least a decade, so the only real contribution would be in the experiments. However the experimental setup is questionable as it does not look like the same care has been given to control overfitting with the 'regular training' method.

More details

Previous work on the same idea: at least a decade old, e.g., Huang and LeCun 2006. See a review of such work in 'Deep Learning using Linear Support Vector Machines' more recently.

Experiments

You should also have a weight norm penalty in the end-to-end ('regular training') case and make sure it is appropriately and separately tuned (not necessarily the same value as for the post-training). Otherwise, the 'improvements' may simply be due to better regularization in one case vs the other, and the experimental curves suggest that interpretation is correct.

review_title: Review of 'Post-Training in Deep Learning'

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper demonstrate that by freezing all the penultimate layers at the end of regular training improves generalization. However, the results do not convince this reviewer to switch to using 'post-training'.

Learning features and then use a classifier such as a softmax or SVM is not new and were actually widely used 10 years ago. However, freezing the layers and continue to train the last layer is of a minor novelty. The results of the paper show a generalization gain in terms of better test time performance, however, it seems like the gain could be due to the \lambda term which is added for post-training but not added for the baseline. c.f. Eq 3 and Eq 4.
Therefore, it's unclear whether the gain in generalization is due to an additional \lambda term or from the post-training training itself.

A way to improve the paper and be more convincing would be to obtain the state-of-the-art results with post-training that's not possible otherwise.

Other notes, 

Remark 1: While it is true that dropout would change the feature function, to say that dropout 'should not be' applied, it would be good to support that statement with some experiments.

For table 1, please use decimal points instead of commas.

review_title: This paper demonstrate that by freezing all the penultimate layers at the end of regular training improves generalization.

Meta-review: * the proposed fine-tuning of only the last layer is not novel enough
* experiments are not sufficient to isolate the differences to support the benefit of post-training

Decision: Reject