Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
The paper highlights two counterintuitive properties of neural networks: (1) a tendency to encode factors of variation in multiple units rather than single units, and (2) an absence of local generalization. The first hypothesis is tested by finding a set of samples correlating most with certain directions in the feature space and visually inspecting their similarity. The second hypothesis is tested by generating 'adversarial samples' that are close neighbors to the data points but predicted of different class.

I think that the first property makes a lot of sense in the context of sigmoidal networks, as the squashing effect of the sigmoid clearly encourages the representation to be redundantly encoded in a large number of neurons in order to preserve the principal components of the data. However, I am wondering whether these findings apply broadly or whether some modeling decision (e.g. sparsity, type of nonlinearity, competition between units, etc) do favor the alignment of independent components to the canonical coordinates.

The second property is very interesting and shows that imperceptible deviations from a data point in the input space can cause the neural network to change its prediction. This finding is likely to generate further research on the nature of this effect. In particular, I would find it interesting to study whether these adversarial samples are a degenerate effect caused by the limited capacity of the network or a more fundamental issue with the learning algorithm that would occur for networks of any size.

Authors provide several examples of adversarial samples for both the MNIST and the image datasets. However, I could not find whether these samples are typical (with distortions corresponding to the average minimum distortion) or whether they have been preselected to have low distortion.