Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
The paper presents some empirical analysis of various types of neural network
with regard to finding examples close to the training examples which are
misclassified.  It also proposes a technique to train on these perturbed
examples, and they mention some improved results on MNIST (it would be nice
to see more focus on this method, rather than so much on the analysis).

Novelty: it's very novel, I'd say.
Quality: so-so.  The experimental justification of the training method is quite
   limited, and there are some flaws in the analysis in my opinion.

Pro: The paper has an original idea.
Con: There are various weaknesses in the analysis and the presentation.
  The paper gives the impression of being a little half-baked.  But I think it's
  interesting enough to publish; the issues could probably be fixed without too
  much trouble.



Detailed comments are below.

------
typo in abstract:
contains of->contains
extend->extent

When you say 'Our experiments show that ... properties.'...  I don't think what
you are talking about is really a scientific experiment.  It's something
anecdotal and informal.  A proper experiment would have human test subjects
judging the extent of semantic similarities of sets of images, where the tests
had been chosen using different methods and the test subjects didn't know which
method the set of images had been chosen from.  Until you do this and show
statistical analysis of some kind, you can't really say anything for sure.
Of course, you can still report the anecdotal observations, just don't say they
are experiments.

When you say 'this can never occur with smooth classifiers by their
definition'...  I think your definition must be problematic.  There will always
be a classification boundary, and it will always be possible to find images
that are close to that boundary.  The only question is how easy is it to find
such images.  Unless you quantify that somehow, and compare across different
modeling strategies (e.g. the average distance you have to go to find
the negative example), I don't think you can say very much.

Fig. 5: I think you can safely assume the reader knows what an ostrich is.

You might want to check that you don't have your description of hard negatives
backwards (I'm not a computer vision expert, but something seems wrong).

disclassified->misclassified

The sentence 'The instability is expressed mathematically as...' is to me quite
problematic.  I think you are trying to formalize something that is really a little
vague, and not doing it quite right.
The rest of the spectral analysis is interesting though, but with the stuff
regarding A(omega), I think it would be more useful if you tried to explain in
words what is happening rather than force the reader to wade through notation.

I don't think the comparison to the set of rational numbers is useful, given
that we know the last hidden layer is a continuous function of the input
(and thus the sets we're dealing with are much more well-behaved than the
set of rational numbers).

Overall, some of the analysis and explanation in the paper is a little problematic,
and sometimes over-stated, but the basic idea is I think quite interesting, and I think
you should emphasize more your training method, which in my mind is probably more
interesting than the analysis itself.