Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors showed that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability to query the network. The studied problem is very interesting. I have the following questions about this paper:

1. Can the authors provide detailed explanation of Figure 1? For instance start from input (x_1, x_2), and the weight in layer 1 and layer 2, what is the exact form of the function plotted in the middle panel? Also, how the input space is partitioned? I appreciate the authors provide this simple example, but detailed math will help readers to understand this easily.

2. How about the efficiency of the proposed method? Is it NP-hard? I would like to see some analysis of the computational complexity and also some related experimental results.

3. If the ReLU network can be reconstructed, can the input also be reconstructed based on the output? It would be very interesting to show a few example on reconstructing the input. Also, is that possible to even reconstruct the training data based on the released model?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper introduces a procedure for reconstructing the architecture and weights of deep ReLU network, given only the ability to query the network (observe network outputs for a sequence of inputs).  The algorithm takes advantage of the piecewise linearity of ReLU networks and an analysis by [Hanin and Rolnick, 2019b] of the boundaries between linear regions as bent hyperplanes.  The observation that a boundary bends only for other boundaries corresponding to neurons in earlier network layers leads to a recursive layer-by-layer procedure for recovering network parameters.  Experiments show ability to recover both random networks and networks trained for a memorization task.  The method is currently limited to ReLU networks and does not account for any parameter-sharing structure, such as that found in convolutional networks.

The networks used in experiments appear to be substantially smaller (e.g. input/output dimensions on the order of 10 neurons) than those used in real applications.  Is the proposed approach practical to apply to networks used in actual applications?  How does the number of queries per parameter scale? (page 5 mentions sample complexity for recovering the first layer, but it would be helpful to clarify the situation for subsequent layers).

Page 7 states that the proposed algorithm also holds for ResNets, with slight modifications, but defers details to future work.  If the modifications are indeed slight, it would better to include them here as this is an important special case and would increase the potential impact of the paper.

Overall, while the paper does appear to rely heavily on developments made by [Hanin and Rolnick, 2019b], there is a potentially interesting contribution here.  I would appreciate clarification on concerns over practicality and the extension to ResNets.

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 
Main contribution of the paper
- The paper proposes a new method to recover the unknown structure of  the network by utilizing the piecewise linearity of ReLU network.
- Some theoretical explanation of the method is provided.

Note & Questions
- As far as the author understands, the algorithm does not suppose a fully black-box condition. By seeing the section 4.1 and 4.2, it seems possible to access neurons in the intermediate layers.
- Also, the proposed method seems to target only a MLP.


Strong-points
- This field is not that thoroughly investigated, and the author proposes a creative method to infer the hidden statistics of the neuron.

Concerns
- Most of all, the information the experiments conveys is too small to convince the argument of the author. The reviewer could not find the dataset they train (in the Experiment section), and the graph only shows the case of two-layered networks. Moreover, the reviewer couldn't find the explanation of the graph, including their legend (for example, Memorization).
The author suggests that this method can be applied to various networks. Still, the reviewer couldn't find any clue that the method actually worked for various settings: different activations, convolutional networks, and so on. More experimental results supporting the argument of the authors are required.
- Assuming that the network was trained by MNIST and we infer the weight of the networks by the proposed method. Can the recovered network classify the number as well? Then, how the accuracy change?
More quantitative results regarding the asking are required.
- Experimental results for more-than-two layered networks should be provided.
- Oh.et.al (https://arxiv.org/abs/1711.01768) proposed a blackbox reverse-engineering method and provided experimental settings as well. The author should clarify the novelty and the strong-points of the works compared to the mentioned work.

Conclusion
- The author proposes a new method to recover the weight and bias of the network.
- The reviewer could not find much clue supporting the author's argument from the experiment section.

inquiries
- See the Concerns parts.
review_title: Official Blind Review #2331

final_score: rating: 3: Weak Reject
review_content: This paper introduces an approach to recover weights of ReLU neural networks by querying the network with specifically constructed inputs. The authors notice that the decision regions of such networks are piece-wise linear corresponding to activations of individual neurons. This allows to identify hyperplanes that constitute the decision boundary and find intersection points of the decision boundaries corresponding to neurons at different layers of the network. However, weights can be recovered only up to permutations of neurons in each layer and up to a constant scaling factor for each layer.

The algorithm consists of two parts: Identifying parameters of the first layer and subsequent layers. First they sample a lot of line segments and find their intersections with the decision boundary, i.e. where pre-activations equal 0. Then they sample some points around the intersections and estimate the hyperplanes up to a scaling factor and a sign. For the first layer they check whether some of the hyperplanes belong to it. For the consecutive layers they proceed by moving from the intersection points along the hyperplanes until the decision boundary bends. Again, by identifying which of the bends correspond to the intersection of the current layer's hyperplanes with the previous layers' ones, they are able to recover parameters of the current layer by computing the angles of these intersections.

This paper tackles a very interesting and important problem that might have huge implications for security and many other aspects. However, I'm leaning towards Reject for the following reasons:

1. The algorithm's description is either incomplete or unclear. There are such core functions as PointsOnLine and TestHyperplane, whose pseudo-code would be very helpful for understanding. For example, the authors say that PointsOnLine performs a "binary search," but then this function can find only one (arbitrary) intersection of a line segment with a decision boundary, while each sampled line can intersect multiple ones. If it is not binary search, then the asymptotic analysis given in the end of Sec. 4.2 is incorrect. Even more mysterious is TestHyperplane, from the provided intuition I do not understand how it is possible to distinguish hyperplanes corresponding to the first layer vs. the other layers. In Sec. 4.3, second paragraph, the choice of R is unclear. How to chose it to make sure that the closest boundary intersects it?

The authors consider a very limited setting of only fully-connected (linear) layers with ReLU activations. In this case it is easy to see that the resulting decision boundary is indeed piece-wise linear with a lot of bending. Authors themselves notice, that "the algorithm does not account for weight sharing." For CNN this will lead to each virtual neuron in each channel to have its own kernel weights, although there must be one kernel per channel. Also the authors admit, that pooling layers affect partitioning of the activation regions, making the proposed approach inapplicable. The authors did not discuss whether the proposed approach can handle batchnorm layers. Such non-linear transformations could pose serious problems. All this rules out applications to, for example, all CNN-based architectures, that prevail in computer vision. The authors mention, that their "algorithm holds with slight modification" for ResNets, but as mentioned earlier convolutional, pooling and batchnorm layers make it not so trivial (if at all possible).

2. Experimental evaluation is extremely limited: It is all contained in just one paragraph. Although it is mentioned that "it is often possible to recover parameters of deep ReLU networks," they evaluated their approach on very shallow and narrow networks (only 2 layers, 10 to 50 neurons in each). The immediate question here is why this algorithm is not applied to sufficiently deep NN? At least a network that could classify MNIST reasonably well. Actually, this would be a better proof-of-concept: Given a pre-trained MNIST classifier, apply the proposed method, recover the weights and check if you get the same output as from the original network. Whereas here the evaluation is given as a normalized relative error of the estimated vs. the true weights. Which raises the question of how the scaling factor was chosen? Recall, that the proposed method estimates network's parameters only up to an arbitrary scaling factor. My guess, is that for the Figures 3 and 4 (both right) the estimated weights were re-scaled optimally to minimize the relative error. But in the end, one is interested in recovering the original weights of the network, not relative ones.

I am very confused by Fig. 3 left: Why is the number of queries going down as the number of neurons increases? Should it not be that with more neurons the ambiguity also increases, requiring more queries? Again, this analysis is very limited, it would be very interesting to see, how many more queries one needs for deeper layers of the network. But for this experiments with deeper than 2 layers networks are necessary.

3. The choice of parameters is unclear and not discussed. How long should the line segments be, how many of them. How many points are sampled and within which radius to identify hyperplanes, how to choose 'R'. And how all these choices affect accuracy and performance.

Overall, the paper looks rather incomplete to me and requires a major revision. It will definitely benefit if the "slight modification" for the case of ResNets is included. Also, experimental evaluation should be completely re-done and extended.
review_title: Official Blind Review #1

Meta-review: This article studies the identifiability of architecture and weights of a ReLU network from the values of the computed functions, and presents an algorithm to do this. This is a very interesting problem with diverse implications. The reviewers raised concerns about the completeness of various parts of the proposed algorithm and the complexity analysis, some of which were addressed in the author's response. Another concern raised was that the experiments were limited to small networks, with a proof of concept on more realistic networks missing. The revision added experiments with MNIST. Other concerns (which in my opinion could be studied separately) include possible limitations of the approach to networks with no shared weights nor pooling. The reviewers agree that the article concerns an interesting topic that has not been studied in much detail yet. Still, the article would benefit from a more transparent presentation of the algorithm and theoretical analysis, as well as more extensive experiments. 
Decision: Reject