Task: Language Identification
Input (Spanish, English): Encouraging orthogonality in weight features has been reported useful for deep networks in many previous works. The authors present a explicit regularization cost to achieve de-correlation among weight features in a layer and encourage orthogonality. Further, they also show why and how negative correlations can and should be avoided for better de-correlation.   Orthogonal weight features achieve better generalization in case of large number of trainable parameters and less training data, which usually results in over-fitting. As also mentioned by the authors biases help in de-correlation of feature responses even in the presence of correlated features (weights). Regularization techniques like OrthoReg can be more helpful in training deeper and leaner networks, where the representational capacity of each layer is low, and also generalize better.  Although the improvement in performances is not significant the direction of research and the observations made are promising.
Output (English): en