Task: Language Identification
Input (Spanish, English): The starting point of this work is the understanding that by having decorrelated neurons (e.g. neurons that only fire on background, or only on foreground regions) one provides independent pieces of information to the subsequent decisions. As such one gives "complementary viewpoints" of the input to the subsequent layers, which can be thought of as performing ensembling/expert combination within the model, rather than using an ensemble of networks.   For this, the authors propose a sensible method to decorrelate the activations of intermediate neurons, with the aim of delivering complementary inputs to the final classification layers: they split intermediate neurons to a "foreground" and a "background" subset, and append side-losses that force them to be zero on background and foreground pixels respectively.   They demonstrate that this can improve classification on a mid-scale classification example (a fraction of imagenet, and a ResNet with 18, rather than 150 layers), when compared to a "vanilla" baseline that does not use these losses.  I enjoyed reading the paper because the idea is simple, smart, and seems to be effective.  But there are a few concerns; -firstly, the way of doing this seems very particular to vision. In vision one knows that masking the features (during both training and testing) helps, e.g.
Output (English): en