Task: Language Identification
Input (Spanish, English): ResNet and other architectures that use shortcuts have shown empirical success in several domains and therefore, studying the optimization for such architectures is very valuable. This paper is an attempt to address some of the properties of networks that use shortcuts. Some of the experiments in the paper are interesting. However, there are two main issues with the current paper:  1- linear vs non-linear: I think studying linear networks is valuable but we should be careful not to extend the results to networks with non-linear activations without enough evidence. This is especially true for Hessian as the Hessian of non-linear networks have very large condition number (see the ICLR submission "Singularity of Hessian in Deep Learning") even in cases where the optimization is not challenging. Therefore, I don't agree with the claims in the paper on non-linear networks. Moreover, one plot on MNIST is not enough to claim that non-linear networks behave similar to linear networks.  2- Hessian at zero initial point: The explanation of why we should be interested in Hessain at zero initial point is not acceptable. The zero initial point is not interesting because it is a very particular point that cannot tell us about the Hessian during optimization.
Output (English): en