Intriguing properties of neural networks
Authors: Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, Dumitru Erhan
ICLR 2014

Abstract: Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.


Peer review:
[Review]:
It is a nicely written and interesting find, but I think some of the unexplained anomalies and deviations from intuition about the behavior of a deep neural network discussed in the paper have some intuitive justification. For example - 
'Moreover, it puts into question the notion that neural networks disentangle variation factors across coordinates.' I think this statement is not fully justified by the observation that even a random direction in the final layers output space can lead to some sort of semantic clustering of the inputs. For instance, lets plot a thousand people on weight vs height 2D graph. Not all, but a lot of straight lines emanating from the origin will have a clearly perceivable semantic meaning associated with them to a human, like a line with very little slope will have super skinny people, line with big slope will have fat people, something in between we will have fit people and so on. I am not saying that height and weight are totally independent but are fairly untangled features. This simply analogy sort of hints towards an intuitive explaination of observance of semantic features when using random directions in higher layer output space because this higher layers have a very rich bag of weakly untangled variations and their random combination can still provide us with strongly perceivable semantic features, because humans are really good at finding patterns in a collection of object.
Regarding the blind-spot observation, I think the paper sort of played around the well-known insensitivity of human visual system towards high-frequency structured noise in images. Whereas, for machine vision, high-frequency signals play a vital role, therefore, it is easy to make a drastic change for the machine while keeping it almost imperceptible for the humans. The other way round of this phenomenon ie drastic change to the human eye but no change to the machine is already in use as the basis of a very popular edge extractor, 'Edge detection with embedded confidence' by Peter Meer. And such behaviour is sort of expected from a deep network because the final layer neurons when projected back to the visual input nodes using Deconvolution networks are mostly edge-profiles of the objects and thus making small adversarial changes to them is expected to alter the final classification output. 
The use of this phenomenon in training is interesting, but an evaluation of the incorrect classifications on some datasets like Image-net to see if a small imperceptible change to the wrongly classified images can indeed alter the final label to the correct label of the image, will possibly be a way to improve the results of the network even further.
The comments are general observations and my own thoughts after reading the paper and not intended towards any sort of review or feedback, scholarly or otherwise.