Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
Hi Folks,
Interesting manuscript that I enjoyed reading, but I have some concerns about how you establish the conclusion that 'The explanation is that the set of adversarial negatives is of extremely low probability, and thus is never (or rarely) observed in the test set, yet it is dense (much like the rational numbers, and so it is found near every
virtually every test case.'
FWIW I think this is very likely the right conclusion, but I really don't think you show that.

Filling in the gaps, I assume you did some sort of monte carlo using the images with Gaussian white noise, found few that were unclassifiable, and then concluded that the adversarial images must have low probability?
I think you should say that, if that is what you did, and give the number of trials and enough other detail to replicate the experiments. Otherwise I think you should give enough detail so that we can understand how you reached this conclusion - your assertion is so blunt I can't imagine it is just a conjecture, and I don't think I'm the only reader who would like to know how you reached this conclusion.

But that is not my main concern.

A more serious problem, I think, is that you were probably looking in the wrong place. I can give you both a heuristic argument and a formal one for why I think this:
 A heuristic argument for why you should believe this is that the adversarial images in the paper all look like the originals, while the noisy ones don't.
A formal argument would be that with n=784, adding Gaussian white noise with std dev sigma is essentially the same as shifting the image in a random direction by sqrt{784}*sigma - this follows from well-known results in measure concentration. Of course, that is not close to the original images or to the adversarial examples that you found.

Fortunately this is easy to correct however - take the images that you generated adversarial versions of (or a random subsample of them) and simply do monte carlo with Gaussian white noise added to the original image where sigma = Av. distortion of the adversarial version of that image. Then you are searching the sphere where the adversarial image lies and if there aren't many adversaries there, then you won't find them even with a whole bunch of MC trials.

(Of course, if you do find them then that's even more intriguing - why should small distortions of the images become unclassifiable but large ones not...?)