Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
This paper studies the interesting case of showing where deep nets fail. It proposes a simple mechanism to demonstrate that even small (visually almost imperceptible distortion) to the training samples can cause drastic changes in recognition (ie, decision boundaries). 

This is a fresh perspective and I applaud the authors for providing interesting diagnostic means to help to understand deep nets.

A few comments:

(1) the paper actually does not state how to address the issue of deep nets learning very rugged decision boundaries:  a few ideas come into mind, though. Using drop-out in training process might eliminate this problem? If # of classes is small, would this problem be less severe? (For example, the required distortion is large so one can detect ).   

(2) I could not follow the analysis in section 4.3 -- not clear where it leads to

(3) Looking at Table 1/2: I notice that the required distortion is larger for more complex models. So perhaps this model needs to be better regularized?  Have you tried contractive autoencoders?