Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors aim to better understand the classification of neural networks. The authors explore the latent space of a variational auto encoder and consider the perturbations of the latent space in order to obtain the correct classification. They evaluate their method on CelebA and MNIST datasets.

Pros:

 1) The paper explores an alternate methodology that uses perturbation in latent spaces to better understand neural networks 
2) It takes inspiration from adversarial examples and uses the explicit classifier loss to better perturb the $z$ in the latent space
3) The method is quite simple and captures the essence of the problem well

Cons:
The main drawback of the paper is it claims to understand working of neural networks, however, actually what the authors end up doing are perturbations of the encoded latent space. This would evidently not explain why a deep network generates misclassifications for instance understanding the failure modes of ResNet or DenseNet cannot be obtained through this method. Other drawbacks include:

1)     They do not show how their method would perform against standard adversarial attack techniques, since by explaining a neural network they should be able to guard against attacks, or at-least explain why they work well. 
2) The paper reports results on 2 datasets, out of which on 1 of them it does not perform well and gets stuck in a local minima therefore implying that it is not able to capture the diversity in the data well. 

3) The authors provide limited evaluation on a few attributes of CelebA. Extensive evaluation that would show on a larger scale with more attributes is not performed.

4) The authors also have claimed that the added parts should be interpretable and visible. However, the perturbations of the latent space would yield small $\epsilon$ variation in the image and it need not actually explain why the modification is yielding a correct classification, the same way an imperceptible adversarial attack yields a misclassification. Therefore there is no guarantee that the added parts would be interpretable. What would be more reasonable to claim would be that the latent transformations that yield correct classifications are projected into the original image space. Some of these yield interpretations that are semantically meaningful and some of these do not yield semantically meaningful interpretations.

5)  Solving mis-classification does not seem to equate with explaining the neural network, but rather only suggest where it makes mistakes. That is not equal to an explanation about how it is making a classification decision. That would rather be done by using the same input and perturbing the weights of the classifier network. 

In conclusion, the paper in its current form provides a direction in terms of using latent space exploration to understand classification errors and corrections to them in terms of perturbations of the latent space. However, these are not conclusive yet and actually verifying this would need a more thorough evaluation.
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method for explaining the classification mistakes of neural networks. For a misclassified image, gradient descent is used to find the minimal change to the input image so that it will be correctly classified. 

My understanding is that the proposed method does not explain why a classifier makes mistakes. Instead, it is about: what can be added/removed from the input image so that it can be correctly classified. Strictly speaking, "Explaining the Decisions of Neural Networks" is not the most relevant title for the proposed method. 

Based on my understanding about what the paper proposes, I am not sure how useful this method is, from the application point of view. It is unclear to me how this method can shed light to the mistakes of a classifier. 

The technical aspects of the paper are straight forward optimization problem, with a sensible formulation and gradient descent optimization problem. There is nothing extraordinary about the proposed technique. 

The method assumes the availability of a generative model, VAE. The implicit assumption is that this VAE performs well, and it raises a concern about the application domains where VAE does not work well. In this case, would the visualization reflect the shortcoming of VAE or the mistake of the classifier? 

review_title: Limited technical novelty. Unclear applicational benefit. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors propose a method for explaining why neural networks make mistakes by learning how to modify an image on a mistaken classification to make it a correct classification. They do this by perturbing the image in an encoded latent space and then reconstructing the perturbed image.  The explanation is the difference between the reconstructed perturbed encoded image and the reconstructed original encoded image.

The title is too general as this paper only offers an explanation in the area of image classification, which by itself, is still interesting.  

A method for explaining the results of neural networks is still open ended and visually to the human eye, this paper does offer an explanation of why the 8 is misclassified. However, if this works very well for MNIST, more examples should be given. This single example is interesting but not sufficient to illustrate the success of this method.  

The examples from CelebA are interesting but inconclusive. For example, why should adding blue to the glasses fix the misclassification. If the explanation is simply visual for a human, then this explanation does not suffice. And the two examples with one receding and the other not receding hairlines look like their correct classifications could be flipped.

Regarding epsilon, it is unclear what a small euclidean distance for epsilon is without more examples. It would also help to see how the euclidean distance changes along the path.  But also it is not clear why we care about the size of epsilon, but rather the size of the perturbation that must be made to the original image, which is what is defined in the paper as the explanation. 

Since it is the encoded image that is perturbed, and this is what causes the perturbations to be selective to particular features of the image, an analysis of what features in the encoded space that are modified would greatly help in the interpretability of this explanation. The fact that perturbations are made in the latent space, and that this perturbation gets reflected in particular areas in the reconstructed image, is the most interesting part of this work.  More discussion around this would greatly enhance the paper, especially since the technical tools of this method are not very strong.

Pros: Interesting explanation, visually selects certain parts of the image relevant to classification rather than obscure pixels

Cons: No discussion or analysis about the latent space where perturbations occur. Only one easy example from MNIST shown and examples on CelebA are not great. No way (suggested) to use this tool outside of image recognition. 
review_title: Interesting explanation for images, but lacks discussion about perturbations in the latent space which is most interesting part.

Meta-review: The paper proposes a way to find why a classifier misclassified a certain instance. It tries to find pertubations in the input space to identify the appropriate reasons for the misclassification. The reviewers feel that the idea is interesting, however, it is insufficiently evaluated.  Even for the datasets they do evaluate not enough examples of success are provided. In fact, for CelebA the results are far from flattering.
Decision: Reject