Task: Language Identification
Input (Spanish, English): This paper proposes new initialization for particular architectures and a correction trick to batch normalization to correct variance introduced by dropout. While authors state interesting observations, the claims are not supported with convincing results.  I guess Figure 1 is only for mnist and for only two values of p with one particular network architecture, the dataset and empirical setup is not clear.  The convergence is demonstrated only for three dropout values in Figure 2 which may cause an unfair comparison. For instance how does the convergence compare for the best dropout rate after cross-validation (three figures each figure has three results for one method with different dropouts [bests cv result for each one])? Also how is the corresponding validation error and test iterations?  Also only mnist does not have to generalize to other benchmarks.  Figure 3 gives closer results for Adam optimizer, learning rate is not selected with random search or bayesian optimization, learning decay iterations fixed and regularization coefficient is set to a small value without tuning. A slightly better tuning of parameters may close the current gap. Also Nesterov based competitor gives unreasonably worse accuracy compared to recent results which may indicate that this experiment should not be taken into account.   In Table 2, there is no significant improvement on CIFAR10. The CIFAR100 difference is not significant without including batch normalization variance re-estimation. However there is no result for 'original with BN update' therefore it is not clear whether the BN update helps in general or not. SVHN also does not have result for original with BN update.  There should be baselines with batch normalizations for Figure 1,2 3 to support the claims convincingly.  The main criticism about batch normalization is additional computational cost by giving (Mishkin et al, 2016 ) as reference however this should not be a reason to not to compare the initialization to batch-normalization.  In fact, (Mishkin et al, 2016) performs comparison to batch normalization and also with and without data augmentation with recent state of art architectures.  None of the empirical results have data augmentation. It is not clear if the initialization or  batch normalization update will help or make it worse for that case.  Recent state of art methods methods like Res Net variant and Dense Net scale to many depths and report result for ImageNet. Although the authors claim that this can be extended to residual network variants, it is not clear if there is going to be any empirical gain for that architectures.    This work requires a comprehensive and fair comparison. Otherwise the contribution is not significant.
Output (English): en