Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper details a way of investigating the space of pre-images that lead to a particular output from a ReLU layer, with the goal of using the inverted representations as a way to understand the deep neural networks. Three experiments are proposed where the authors claim that the computed pre-images help interpret the network decision making. 

Overall the paper is interesting, however I am not certain of the novelty as some related work is not discussed. Additionally, although the practical application of the method is interesting, the clarity could be improved for the last experiment. 

Positives:
* Understanding the invariances of neural networks can potentially lead to more interpretable models, and one way to investigate this is by looking at the preimages for a network. 
* The paper is a nice mix of theoretical results which lead to practical applications

Questions and Concerns:
* The authors state that maxpool can be rewritten in terms of a linear component and a ReLU, but this is non obvious. If this is true, a mathematical formulation should be explicitly included in the paper. 
* The paper is missing some potentially related references. Previous work has investigated how multiple stimuli can get mapped onto (approximately) the same point in recognition networks by inverting the representations via iterative gradient descent (Mahendran & Vedaldi 2015, and recent work including invarianced based adversarial examples in Jacobsen et al. 2019 or model metamers in Feather et al. 2019). How does the proposed preimage computation help improve model interpretability beyond this previous work, especially given the authors statement that the method is intractable for large networks? 
* The paper does not discuss invertible networks which have a bijective mapping between the input and output (ie Invertible Residual Networks in Behrmann et al. 2019). Discussing this work seems relevant if the goal is to make models such that one can start with hypothetical outputs and understand the inputs that lead to them. 
* The final example of using this method in practice for ACAS systems is interesting, but it is difficult to follow what “success” would mean for this experiment

Minor points: 
* The following sentence on page 3 seems to be missing something “Preimages are most insightful and useful when the inputs and outputs have definite interpretation – application areas where the need for massive networks is less.”.
* There it a typo in the last sentence of page 3 (“bu”->”but”)

review_title: interesting discussion of using network pre-images for interpretation 

final_score: rating: 4: Ok but not good enough - rejection
review_content: There are many issues in the paper that can be improved.

The title is not appropriate, this work does not address safety applications. It is worth noting that the word safety is not defined and not used in the main body of the paper.

It is difficult to follow the presentation of the paper since mainly the applications are presented and then some contributions given, in the same presentation as the abstract.

A major issue it that the paper is missing some important theoretical analysis. Of particular interest is the existence of the preimages, because not all outputs have inputs. Moreover, the uniqueness of the solution needs to be studied. These properties should depend on the used nonlinearities and architecture of the neural network.

There are many spelling and grammatical errors, such as “suprising”, “have been been”, “Coincidentlly”, “configuations”
review_title: The quality of the paper needs to be improved

final_score: rating: 3: Clear rejection
review_content: The paper presents a method to verify if a NN is performing as expected in sensitive applications. Although the general area is very important in machine learning, the paper is not very well presented: The problem is not well stated, the approach is not very clear, and the results are not well justified.

- The presentation and the writing of the paper should be improved. Unfortunately, with the current format it is hard to glean the idea of the paper. There are some typos (e.g., 'bu' in page 3, 'plot' in page 4, etc.).
- There are some concepts that are not defined early on and maybe never in the paper. For example, what is the problem that the paper tries to solve mathematically? It is not very clear. What is the mathematical definition of a preimage?
- The authors say: "Preimages are most insightful and useful when the inputs and outputs have definite interpretation –
application areas where the need for massive networks is less". Its hard to fully understand but it seems that the method suffers scalability issues. Can this be formally analyzed? What is the complexity of the algorithm in time and space? Why is there a scalability issue? Is it a fundamental problem? How does this limit the scope and applicability of the method? Also, what does "definite interpretation" mean?
- The NN used in the experiments are very tiny. I would consider experiments that reflect more realistic situations in the real-world. Current setup significantly limits the scope of the method.
- It is not clear how to verify the performance of the method. Results in Figure 1 and 2 does not show us the quality of the method, is it doing good or bad? I found the results in Figure 1 surprising as the moon data is fairly symmetric while the preimage is biased towards one class. Is there a reason for that?
review_title: Paper needs substantial improvements.

final_score: rating: 2: Strong rejection
review_content: Deep neural networks are known to be brittle, and can lead to dangerous consequences if left unverified. Forward reach set computation can be used as a basic primitive to verify properties of deep neural networks used in a robotic setting. There has been a rising interest in verifying larger neural networks used in safety critical setting. 
 In this paper,  the authors propose a way to compute reachable sets for a neural network in a backward sense. Starting from the outputs of the neural network, and  then work it's way to the inputs. This is an interesting way to look at the problem itself,  but as the authors point out it is an intractable problem.

My concern about this paper is I don't see the use of a pre-image computation algorithm as being very useful. A forward reachability tool works pretty well for the size of neural networks considered in the paper. Pre-image computation does not provide any advantage in terms of scalability, as is apparent from the experiments. Moreover, almost any safety constraint that needs to be verified with system dynamics in the loop always should ideally work forward in time. Thus for the neural network controller from the inputs to the outputs. 

Cartpole example : The authors come up with rules about, which output behaviors are correct for a few of the input regions. Then use this as a specification for the verification algorithm. But the very specifications, comes from reasoning about the forward behavior of the system dynamics itself. The idea of forward reach sets computation would generalize much better to a wide range of examples therefore.  Without the need to come up with such handcrafted rules. 

The authors do make a convincing case for the ACASXu example. But this example is less interesting given the amount of attention it has received recently. 
review_title: Not convinced that pre-image computation offers any realistic advantage

Meta-review: Thank you for your submission to ICLR.  The reviewers and I unanimously felt, even after some of the clarifications provided, that while there was some interesting element to this work, ultimately there were substantial issues with both the presentation and content of the paper.  Specifically, the reviewers largely felt that the precise problem being solved was somewhat poorly defined, and the benefit of the proposed preimage technique wasn't always clear.  And while the ACAS system was a nice application, it seems to be difficult to quantify the real benefit of the proposed method in this setting (especially given that other techniques can similarly be used to verify NNs for this size problem).  The answer that this paper provides seems to be something along the lines of "ease of visual interpretation" of the pre-image conditions, but this needs to be quantified substantially more to be a compelling case.
Decision: Reject