Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
Having read the paper, I have some more comments.  I can only speak to my own experience, of course, and may be plain wrong about a few things.  Many of my comments are minor edits.  

First, it seems to me that x_eps, r, and n_i are all used variously to refer to the 'adversarial shift' (i.e. middle columns of 5a,b).  I don't see any reason not to standardize.

In the second paragraph of section 1, I would remove 'Moreover'.  In the same paragraph, I did not know what 'up to a rotation of the space' meant.

2nd to last paragraph of section 1: hyperparemeter -> hyperparameter

In section 3, e_i is undefined.  Also, I would use different names for the different definitions of x'.  I think the sentence before the second definition is not a proper sentence, as well.  And there is an extraneous in ('random basis in for') in the sentence after.  Some quotes point the wrong way as well.

Section 4, paragraph 2: 'In other words...' I would add 'according to [2]' or something else that clarifies that this is the hypothesis represented in that paper, and not an assertion you are making.

paragraph 4:  I think the statement 'this can never occur with smooth classifiers by their definition' is too strong.  While the perturbations are imperceptible, they are not arbitrarily small.  I think this statement would only be justified if you could always change the classification output by changing the intensity of one pixel by the minimum amount possible given your discretization.

paragraph 5:  'finding adversarial' -> 'find adversarial'

Figure 5: I think at least on larger example like this would be nice.  The perturbations might be more perceptible with larger images.

paragraph before 4.1:  I'm not sure what it means 'the training set is then changed...' in general (eg paragraph before 4.3), I was not clear on what dataset (training/test/etc.) was used for what part of the procedure.  

In 4.1, I don't think D(x,l) was ever defined.  

Based on the last line of 4.1, it appears that your optimization problem uses a soft constraint on the misclassification.  The write-up suggests that you are forcing a misclassification (hard constraint).  

bottom of page 6: 'The columns of the table show the error (ratio of disclassified' -> '...(proportion of misclassified'.   I'm not sure what it means: 'The last two rows are special'.  

Table 3 caption: what does it mean 'trained to study'?  

Section 4.3 I found confusing in general.
I don't know what 'max(0,x) is contractive' means.  I would also specify the norm that you use for W_k in the next line (I assume operator norm).  

I found the equation for Wx hard to parse and I don't think w_{c,f} is defined.  There are missing commas before and after the ...'s.  

'x_c denoted the c input feature image' -> 'x_c denoted the c-th input feature image' ?

Finally, In the discussion:
'the set of adversarial negatives is of extremely low probabilities' - did you show this?  
'...yet it is dense' - this I don't think makes sense mathematically in a discrete space.  I would not use this terminology unless you can prove it using the formal definition.