Identifying Weights and Architectures of Unknown ReLU Networks
Authors: David Rolnick
Konrad P. Kording
Venue:  (2020)
Keywords: deep neural network
ReLU
piecewise linear function
linear region
activation region
weights
parameters
architecture

Abstract: The output of a neural network depends on its parameters in a highly nonlinear way, and it is widely assumed that a network's parameters cannot be identified from its outputs. Here, we show that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability to query the network. ReLU networks are piecewise linear and the boundaries between pieces correspond to inputs for which one of the ReLUs switches between inactive and active states. Thus, first-layer ReLUs can be identified (up to sign and scaling) based on the orientation of their associated hyperplanes. Later-layer ReLU boundaries bend when they cross earlier-layer boundaries and the extent of bending reveals the weights between them. Our algorithm uses this to identify the units in the network and weights connecting them (up to isomorphism). The fact that considerable parts of deep networks can be identified from their outputs has implications for security, neuroscience, and our understanding of neural networks.

Under review as a conference paper at ICLR 2020
IDENTIFYING WEIGHTS AND ARCHITECTURES OF UNKNOWN RELU NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The output of a neural network depends on its parameters in highly nonlinear way, and it is widely assumed that a network's parameters cannot be identified from its outputs. Here, we show that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability query the network. ReLU networks between pieces correspond to inputs for which one of the ReLUs switches between inactive and active states. Thus, first-layer ReLUs can be identified (up to sign and scaling) based on the orientation of their associated hyperplanes. Later-layer ReLU boundaries bend when they cross earlier-layer boundaries and the extent of bending reveals the weights between them. Our algorithm uses this to identify the units in the network and weights connecting them (up to isomorphism). The fact that considerable parts of deep networks can be identified from their outputs has implications for security, neuroscience, and our understanding of neural networks.
piecewise linear and the boundaries
INTRODUCTION
The behavior of deep neural networks is complex it is powerful. The relation of individual pa- rameters to the network's ' output is highly nonlinear and is generally unclear to an external observer. Consequently, it has been widely supposed in the field that it impossible recover the parameters of a network merely by observing its output on different inputs.
Beyond informing our understanding of deep learning, going from function to parameters could have serious implications for security and privacy. In many deployed deep learning systems, the output is freely available, but the network used generate that output not disclosed. The ability to uncover a confidential network not only would make it available for public use but could even expose data used train the network if such data could be reconstructed from the network's weights.
This topic also has implications for the study of biological neural networks. Experimental neurosci- entists can record some variables within the brain (e.g. the output a complex cell in primary visual cortex) but not others (e.g. the pre-synaptic simple cells), and many biological neurons appear to be well modeled as the ReLUU of a linear combination of their inputs (Chance et al. 2002). would be highly useful if we could reverse engineer the internal components of a neural circuit based on recordings of the output and our choice of input stimuli.
In this work, we show that it is, in fact, possible in many cases recover the structure and weights of an unknown ReLU network by querying it. Our method leverages the fact that a ReLU network is piecewise linear and transitions between linear pieces exactly when one of the ReLUs of the network transitions from its inactive to its active state. We attempt to identify the piecewise linear surfaces in input space where individual neurons transition from inactive active. For neurons in the first layer, such boundaries are hyperplanes, for which the equations determine the weights and biases of the first layer (up to sign and scaling). For neurons subsequent layers, the boundaries are "bent hyperplanes" that bend where they intersect boundaries associated with earlier layers. Measuring these intersections allows us to recover the weights between the corresponding neurons.
Our major contributions are:
We identify how the architecture, weights, and biases a network can be recovered from the arrangement of boundaries between linear regions in the network.

---

Under review as a conference paper at ICLR 2020
We implement this procedure and demonstrate its success in recovering trained and un- trained ReLU networks.
We show that this algorithm "degrades gracefully," providing partial weights even when full weights are not recovered, and show that these situations can indicate intrinsic ambiguities in the network.
Layer
Layer 2
Function outpur
Input
Output
Input alm 2
Input Input dim
Input dim
Input dim
Figure 1: Left: Architecture of ReLU network .(xx) R + R with two hidden layers, each of width 5. Center: Graph of the piecewise linear function (xx) as a function of the two input variables. Right: Boundaries between linear regions of the network M, essentially "flattened" version of the center image. Boundaries corresponding to neurons from the first layer are shown in blue, from the second layer red. Green shading indicates the gradient ON / along input dimension I; note that the gradient is constant within each region since N(x) is piecewise linear.
RELATED WORK
Various works within the deep learning literature have considered the problem learning network given its output on inputs drawn adaativvy) from given distribution. It is known that this problem is general hard (Goel al.) 2017), though positive results have been found for certain specific choices of distribution in the case that the network has only one or two layers (Ge et al., 2019 Goel & Klivans 2017). By contrast, we consider the problem of learning about network of arbitrary depth, given the ability to issue queries specified input points. In this work, we leverage the theory of linear regions within a ReLU network, an area that has been studied e.g. by Tel Telgarsky (2015); Raghu et al. (2017); Hanin & Rolnick (2019a). Most recently Hanin & Rolnick (2019b) considered the boundaries between linear regions as arrangements of "bent hyperplanes" Milli al. 2019); Jagielski al. (219)) show the effectiveness of this strategy for networks with one hidden layer. For inference of other properties of unknown networks, see e.g. Oh et al. (2019).
Neuroscientists have long considered similar problems with biological neural networks, albeit armed with prior knowledge about network structure. For example, it is believed that complex cells in the primary visual cortex, which are often seen as wanslation-invariant edge detectors, obtain their invariance through what is effectively a two-layer neural network (Kording et al. 2004). first layer believed extract edges, while second layer essentially implements maxpooling. Heggelund (198I) perform physical experiments akin to our approach of identifying one ReLU at time, by applying inputs that move individual neurons above their eritical threshold one by one. Being able to solve such problems more generically would be useful for a range of neuroscience applications.
PRELIMINARIES
3.1 DEFINITIONS
In general, we will consider fully connected, feed-forward neural networks (multilayer perceptrons) with ReLU activations. Each such network N defines a function ((x) from input space "h to output space R We denote the layer widths of the network by 7in (input layer), R1, R2.... ndd nou (output layer). We use W to denote the weight matrix from layer (k - 1) to layer k where layer is the input; and b² denotes the bias vector for layer k. Given neuron 2 in the network, use =xx to denote its preactivation for input x E Ri Thus, for the jth neuron in layer k. we have
Rk-1
xx)x =
Wh ReLU k- + bbt).
2

---

Under review as a conference paper at ICLR 2020
For each neuron 2, will use B2 denote the set of x for which z(x) 0. In general" B, will be an (man- - -1)ddmmenso piecewise linear surface in R" (see Figure in which input dimension is 2 and the B= simply lines). We call B₂ the boundary associated with neuron and we say that B = U 33 is the boundary of the overall network. We refer to the connected components of R \ as regions. Throughout this paper, we will make the Linear Regions Assumption: The set of regions is the set of linear pieces of the piecewise linear function (xx.. While this assumption has tacitly been made in the prior literature, where it does not hold for example, if an entire layer of the network is zeroed out for some inputs.
noted Hanin & Rolnick (2019b) that there are cases
3.2 ISOMORPHISMS OF NETWORKS
Before showing how to infer the parameters of a neural network, we must consider to what extent these parameters be inferred unambiguously. Given network N, there are number of other networks VV' that define exactly the same function from input space to output space. We say that such networks are isomorphic to A. For multilayer perceptrons with ReLU activation, we consider the following network isomorphisms:
Permutation. The order of neurons in each layer network N does not affect the underlying function. Formally, let Pk.o (A) be the network obtained from / by permuting layer &o according to (along with the corresponding weight vectors and biases). Then, Piko(A7 is isomorphic A for every layer and permutation d.
Scaling, Due to the ReLU's equivariance under multiplication, it is possible to scale the incoming weights and biases of any neuron, while inversely scaling the outgoing weights, leaving the overall function unchanged. Formally, for z the ith neuron in layer k and any positive constant, (2,( be the network obtained from N by replacing W .:. b and W by cW is. cb and (1/c)W; respectively. It is simple to prove that (2,) ) is isomorphic to A (see Appendix (A).
Thus, we can hope to recover a network only up to layer-wied wise permutation and neuron-wits wiieen-wise scaling. Formally, Pi,- (NV) and (N) are generators for group of isomorphisms of A. (As shall in (5) some networks also possess additional isomorphisms.)
THE ALGORITHM
Identifying the first layer
Additional layers
Explore boundary intersections
Sample boundary points
P
P.
Infer hyperplanes
P
Unused boundary points
x
Test hyperplanes
Figure 2: Schematic of our algorithms for identifying the first layer of N and the additional layers.
4.1 INTUITION
Consider network A and neuron 2 E N,, so that B, is the boundary associated with neuron Recall that B2 is piecewise linear. We say that B2 bends at point if B, is nonlinear at that point (that is, if the point lies the boundary of several regions). As observed in Hanin & Rolnick 2019b). Bg can bend only at points where it intersects boundaries B for in an earlier layer of
'oore precisely, this holds for all but measure zero set of networks, and any network for which this is not may simply he perturbed slightly.
3

---

Under review as a conference paper at ICLR 2020
the network. In general, the converse also holds; B3 bends wherever it intersects such boundary By (see Appendix A]. Then, for any two boundaries B, and By, one of the following must hold: B_ bends at their intersection (in which case 2 occurs in a deeper layer of the network), B bends (in which case z' occurs in a deeper layer), or neither bends (in which case z and z' occur in the same layer). It is not possible for both B_ and B_r to bend their intersection unless that intersection is also contained in another boundary. which is vanishingly unlikely in general. Thus, the architecture of the network can be determined by evaluating the boundaries B3 and where they bend in relation to one another.
Moving beyond architecture, the weights and biases of the network can also be determined from the boundaries, one layer at time. Boundaries for neurons in the first layer do not bend and are simply hyperplanes; the equations of these hyperplanes expose the weights from the input to the first layer (up to permutation, scaling, and sign). For each subsequent layer, the weight between neurons and 2' can determined by calculating how Byr bends when it crosses Bg. The details of our algorithm below are intended to make these intuitions concrete and perform efficiently even when the input space high-dimensional.
Algorithm The first layer
Algorithm 2 Additional layers
Initialize P1 P3 = S1 #
Input Ps and ,,....,
for t 1,..., L do
Initialize S& = ()
Sample line segment &
for p1 E Pk-1 on boundary B, do
Pi Pi PointsonLine(()
Initialize A = {PI), L3 H₂ = 0
end for
while La 2 Layer k do
for p E Pi do
Pick p;
and v
H Infertlypeerpaaee())
p'. B = Closest
if Ifeesttyperrplane(H then
p on boundary then
S Si U GetParams
A= + A₂ U (p'+
else P2 + P2U (p)
L -- I, U (2'}
end if
H. f H. U/Tneertyyerppaee(ep)
end for return Parameters S unused sample points P3
else Px + Ps U (p:): break
end if
end while
if L, 2 Layer k k- then S₁ + GetParams (T.) end if
end for return Parameters Sp, unused sample points Pk+1
4.2 THE FIRST LAYER
We begin by identifying the first layer of the network N, for which we must infer the number of neurons, the weight matrix W and the bias vector b². As noted above, for each = 27 the first layer, the boundary B, is hyperplane with equation W bj = 0. For each neuron in a later layer of the network, the boundary B, will, in general, bend and not be a (complete) hyperplane (see Appendix [A). We may therefore find the number of neurons in layer 1 by counting the hyperplanes contained in the network's boundary B, and we can infer weights and biases by determining the cquations of these hyperplanes.
Boundary points along line. Our algorithm is based upon the identification of points on the boundary B. One of our core algorithmic primitives is a subroutine PointsOnLine that takes as input line segment C Rn and approximates the set & B of boundary points along f. The algorithm proceeds by leveraging the fact that boundary points subdivide into regions within which N N(x) is linear. We maintain a list of points in order along & (initialized to the endpoints and midpoint of () and iteratively perform the following operation: For each three consecutive points on our list, X3, we determine if the vectors (N (x2) -x(()) -N(x))) /)/l|x>>
X1 l|2 and (N((x) -N(x))) [xxx))/[[>> X2 la are equal (to within computation error) - so, we remove the point X2 from our list, otherwise we add the points (x1 + 2x2)/3 and (x3 - 2xg) /3 to our list The points in the list converge by binary search to the set of discontinuities of the gradient (>(x)/dx (xx, which are our desired boundary points. Note that Point sOnLine is where make use of our ability to query the network.
X1,
X2, X2,
These weighted averages speed up the search algorithm by hiasing it towards points closer towards the center of the segment, which where we expect the most intersections given our choice of segments.
4

---

Under review as a conference paper at ICLR 2020
Sampling boundary points. In order to identify the boundaries B, for in layer 1, we begin by identifying a set of boundary points with at least one on each B. A randomly chosen line segment through input space will intersect some of the B= indeed, if it is long enough, it will intersect any fixed hyperplane with probability 1. We sample line segments e in RR and run PointsOnl ne choose to sample
on each. Many sampling distributions are possible, but in our implementation segments of fixed (long) length, tangent at their midpoints to a sphere of fixed (large) radius. This ensures that each of our sample lines remains far from the origin, where boundaries are in closer proximity and therefore more easily confused with one another (this will become useful in the next step). Let P1 be the overall set of boundary points identified on our sample line segments.
Inferring hyperplanes. We now proceed to fit a hyperplane to each of the boundary points we have just identified. For each p E P1, there a neuron 2 such that p lies on Bz. The boundary B₂ is piecewise linear, with nonlinearities only along other boundaries, and with probability 1, P does not lie on a boundary besides Bg. Therefore, within a small enough neighborhood of p. Bg is given by a hyperplane, which we call the local hyperplane at p. If is in layer 1, then B3 equals the local hyperplane. The subroutine InferHyperr rp ane takes as input a point p on a boundary B2 and approximates the local hyperplane within which p lies. This algorithm proceeds by sampling many small line segments around p. running Point sOnLine to find their points of intersection with B,, and performing a linear regression to find the equation of the hyperplane containing these points.
Testing hyperplanes. Not all of the hyperplanes we have identified are actually boundaries for neurons in layer 1, we need to test which hyperplanes are contained in B in their entirety, and which are the local hyperplanes of boundaries that bend. The subroutine Test Hepeyplael takes as input a point p and hyperplane H containing that point, and determines whether the entire hyperplane H is contained in the boundary B of the network. This algorithm proceeds by sampling points within H that lie far from p and applying Point Pointson sOnLine to a short line segment around each such point to check whether these points all lie on B. Applying Test Hyperplaa to those hyperplanes inferred in the preceding step allows us to determine those B2 for which = is in layer 1.
From hyperplanes to parameters. Finally, we identify the first layer of N from the equations of hyperplanes contained in B. The number of neurons in layer 1 is given simply by the number of distinct B3 which are hyperplanes. As we have observed, for
z in layer 1, the hyperplane B3 is given by W x + b O. We can thus determine W:, and b; up to multiplication by a constant. However, have already observed that scaling W. and by by a positive constant (while inversely scaling W?.) is a network isomorphism (33.2) Therefore, we need only determine the true sign of the multiplicative constant, corresponding determining which side of the hyperplane is zeroed out by the ReI.U. This determination of sign will be performed below in $4.3
Sample complexity. We expect the number of queries necessary obtain weights and biases (up to sign) for the first layer should grow as Om n2) log n1)), which for constant width networks is only slightly above the number of parameters being inferred. Assuming that biases in the network are bounded above, each sufficiently long line has at least a constant probability of hitting a given hyperplane, suggesting that log n1 lines are required according to a coupon collector-style argument. Hanin & Rolnick (2019a) show that under natural assumptions, the number of boundary points intersecting a given line through input space grows linearly in the total number of neurons in the network. Finally, each boundary point on a line requires ((m.) queries in order to fit a hyperplane.
4.3 ADDITIONAL LAYERS
We now assume that the weights W ..... W and biases b². b²....,,u - have already been deter- mined within the network N, with the exception of the sign choice for weights and biases at each neuron in layer along with the correct signs for W and b
We now show how is possible determine the weights W and biases b*
Closest boundary along line. In this part of our algorithm, we will need the ability to move along a boundary to its intersection with another boundary. For this purpose, the subroutine will be useful. It takes as input a point p. a vector v and the network pa- rameters as determined up to layer k - 1, and outputs the smallest >> 0 such that q p + ev lies on B, for some 2 in layer at most h - 1. In order to compute c, we consider the region R within which p lies, which is associated with certain pattern of active and inactive ReLUs. For each boundary B., we can calculate the hyperplane equation which would define B, were it to intersect R, due to the fixed pattern of active and inactive neurons within R, and can calculate the distance
5

---

Under review as a conference paper at ICLR 2020
from P to this hyperplane. While not every boundary B, intersects R, the closest boundary does, allowing us to find the desired c.
Unused boundary points. In order to identify the boundaries B, for in layer k, we wish to identify a set of boundary points with at least one on each such boundary. However, in previous steps of our algorithm, set Pk-1 of boundary points was created, of which some were used in ascertaining the parameters of earlier layers. We now consider the subset Pk C Pk-1 of points that were not found to belong to Bz, for in layers 1 through k 1. These points have already had their local hyperplanes determined.
Exploring boundary intersections. Consider point P1 Ps such that P1 E B2. Note that B_ will, in general. have nonlincarities where it intersects each By for which 2' lies in an earlier layer than z. We explore these intersections, and in particular attempt to find a point of B3g n B.y for every z' in layer k- - 1. Given the local hyperplane H at P1. we pick a direction v along H and apply closestBoundat
to calculate the closest point of intersection p' with B. for all a' already identified in the network. (Below we discuss how best to pick v.) Note that if 2 is in layer k, then p' must be on B2 as well as By, while if is in a later layer of the network, then there may exist unidentified neurons in layers below and therefore B₃ may bend before meeting By We check if p' lies on Bg by applying Point s onniine, and if so apply In In.erHHperrp ane to calculate the local hyperplane of B₂ on the other side of B,'' from P1. We select a representative point P2 this local hyperplane. We repeat the process of exploration from the points PI, P2.... until one of the following occurs: (i) point of B₂ B_r has been identified for every x' in layer h I (this may be impossible; see 5 (ii) is determined to be in layer deeper than k (as a result of p' not lying on B2), or (iii) a maximum number of iterations has been reached.
How to explore. An important step in our algorithm is exploring points of B3 that lie on other boundaries. Given a set of points Ag eral methods for picking a point p; and direction v along the local hyperplane at p; to apply Closest "losestBcundald Bennddrd... One approach and a random direction v: this has the advantage of simplicity. However, it is somewhat faster to consider for which 2/ the intersection B= B. has not yet been identified and attempt specifically to find points on these intersections. One approach for this is to pick a missing 2' and identify for which p; the boundary By lies on the boundary of the region containing P; and solve a linear program to find v. Another approach is to pick missing 2' and point Pi. calculate the hyperplane H which would describe B.' under the activation pattern of pt. and choose v along the local hyperplane to p: such that the distance to H is minimized. This is the approach which we take in implementation, though more sophisticated approaches may exist and present an interesting avenue for further work.
{p1. P2...., pm, on Bz. we briefly consider sev-
to pick a random point P, from those already identified
From boundaries to parameters. We now identify layer l of N, along with the sign of the param- eters of layer A: --11 1, by measuring the extent to which hyperplanes bend at their intersection. We are, in addition, able identify the correct signs at layer k: 1 by solving an overconstrained system of constraints capturing the influence of neurons in layer k - 1 on different regions of input space. The following theorem formalizes the inductive step that allows us to go from what we know at layer k 1 (weights and biases, up to scaling and sign) to the equivalent set of information for layer k plus filling in the signs for layer ls
The proof is given in Appendix B
Theorem. The following holds true for deep multi-layer perceptrons N satisfying the Linear Region Assumption (3..)) excluding a set of networks with measure zero:
Suppose that the weights and biases of N are known up through layer k 1,,, with the exception that for each neuron in layer l - 1, the sign of the incoming weights and the bias is unknown. Suppose also that for each in layer k, there exists an ordered set of points A, = {PI,, P2...., pmm such that: (i) Each point lies on the boundary of Bz, and in (the interior of) a distinct region with respect to the earlier-layer boundaries already known; (ii) each point (except for P1) has a precursor in an adjacent region; (ii)) for each such pair of points, the local hyperplanes of B, are known, as is the boundary B. dividing them (z' in an earlier layer): (iv) the set of such a! includes all of layer 1 1.
Then, it is possible to recover the weights and biases for layer k, with the exception that for each neuron, the sign of the incoming weights and the bias is unknown. It is also possible to recover the sign for every neuron in layer k - 1.
Note that even when assumption (iv) of the Theorem is violated, the algorithm recovers the weights corresponding to whichever boundaries are successfully crossed (as we verify empirically in T6).

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION
We here explore some reasons why our algorithm may fail, motivate our recursive approach, and discuss the potential for generalizations to different architectures.
Non-intersecting boundaries. It is possible that for some neurons and z' in consecutive layers, there is no point of intersection between the boundaries BN and B (or that this intersection is very small), making it impossible to infer the weight between 2 and 2' by our algorithm. Some such cases represent an ambiguity in the underlying network- - an additional isomorphism to those described in 43.2 Namely, B2 n B is empty if one of the following cases holds: (1) whenever 2 is active, is inactive; (2) whenever 2 is active, z' is active; (3) whenever is inactive, z' is inactive; or (4) whenever is inactive, z' is active. In case I, observe that a slight perturbation to the weight w. between and z' has no effect upon the network's output; thus w is not uniquely determined. Cases 2-4 present a more complicated picture; depending on the network, there may or may not be additional isomorphisms.
Boundary topology. For simplicity in our algorithm, we have not considered the relatively rare cases where boundaries B₂ are disconnected bounded. If B- is disconnected, then it may not be possible to find a connected path along it that intersects all boundaries arising from the preceding layer. In this case, it is simple to infer that two independently identified pieces of the boundary belong to the same neuron to infer the full weight vector. Next, if B3 is bounded for some z, then it is a closed (d 1)-dimensional surface within d-dimensional input space 3l While our algorithm requires no modification in this case. bounded B, may be more difficult to find by intersection with randomly chosen lines, and more principled sampling method may be helpful.
Our recursive approach. Our approach proceeds layer by layer, leveraging the fact that each boundary bends only those for those boundaries corresponding to earlier neurons in the network. Our approach in the first layer is, however, distinct from (and simpler than) the algorithm for sub- sequent layers. One might wonder why, once the first - layers have been identified, it is not possibly simply to apply our first-layer algorithm to the n4 -dimensional "input space" arising from activations of layer k- 1. Unfortunately, this is not possible in general, as this would require the ability to evaluate layer & for arbitrary settings of layer k 1. ReLU networks are hard to invert, and therefore it is unclear how one could manufacture an input for a specified layer kt - activation, even while knowing the parameters for the first h 11 layers.
Other architectures. While we have expressed our algorithm in terms of multilayer perceptrons with ReLU activation, it also extends to various other architectures of neural network. Other piece- wise linear activation functions admit similar algorithms. For a network with convolutional layers, it is possible to use the same approach to infer the weights between neurons, with two caveats: (i) As we have stated it, the algorithm does not account for weight-sharing the number of "neurons" in each layer is thus dependent on the input size, and is very large for reasonably sized images. (ii) Pooling layers do affect the partition into activation regions, and indeed introduce new discontinu- ities into the gradient; our algorithm therefore does not apply. For skip connections as in ResNets (He et al. 2016), our algorithm holds with slight modification, which we defer until future work.
6 EXPERIMENTS
We verified the success of our algorithm on both untrained and trained networks. In keeping with literature on ReLU network initialization (He et al. 2013] Hanin & Rolnick] 2018), networks were initialized using i.i.d. normal weights with variance 2/ 2/fan-in and i.i.d. normal biases with unit vari- ance. We trained networks on either the MNIST dataset or a memorization task of 1000 "datapoints" of dimension 10 with coordinates drawn i.i.d. from a unit Gaussian and given arbitrary binary labels. Training was performed using the Adam optimizer and a cross-entropy loss applied to the softmax of the final layer, over 20 epochs for MNIST and 1000 epochs for the memorization task. The trained networks (when sufficiently large) were able to attain near- near-perfect accuracy. We observed that both the first-layer algorithm and additional-layer algorithm identified weights and biases to within ex- tremely high accuracy (see Figures 3 and A.. Even in cases where, for the additional- algorithm, a small fraction of neurons were not identified (see (5), the algorithm was able to correctly predict the remaining parameters.
For 2D input, such B2 must be topological circles, but for higher dimensions, it is conceivable for them to he more complicated surfaces, such as toroiaal polyhedra.
7

---

Under review as a conference paper at ICLR 2020
log ferror norm) for learned weights
layers untraited
layers, untraneed
500
layer, meroottttonn tadk
layers N01S7
queries learned
400
-14
Layer 1 nnuuons
Layer neurons
Figure 3: Results of our first-layer algorithm, applied to networks with two or more hidden layers as the width of the first layer varies. All other layers have fixed width 10. Untrained networks have input and output dimension 10, those trained on the memorization task have input dimension 10 and output dimension 2, and those trained on MNIST have input dimension 784 and output dimension 10. Left: The number of queries issued by algorithm per parameter identified in the network AV; the algorithm is terminated once the desired number neurons have been identified. Right: Log normalized error log(I W |/ (1 for W the approximated weights. Weight vectors were scaled to unit norm to account for isomorphism (see (3.2). Curves are averaged over 5 runs in the case of MNIST and 40 runs otherwise, with standard deviations shown.
log (error norm) for learned params
laysts untrentc
leyess remmruatte tedk
layrrs untreiec- Stace
Layer 2 neurons found
layers memortation task veihtt
50
20
10
Layer
Figure 4: Results of algorithm for additional layers, applied to networks with layers, the first layer width 10, as the width of the second layer varies. Left: Number of estimated layer 2 neurons. Right: Log normalized error between estimated and corresponding Figure 3 above) for approximated weights W and biases b Curves are averaged over 4 runs, with standard deviations between runs shown as shaded regions.
neurons (as in
CONCLUSION
In this work, have shown that it is often possible recover the architecture, weights, and biases of deep ReLU networks by repeated queries. We proceed by identifying the boundaries between linear regions of the network and the intersections of these boundaries. Our approach is theoretically justified and empirically validated on networks before and after training. Where the algorithm does succeed in giving a complete set of weights, is nonetheless able give partial set of weights, and incompleteness in some cases reflects unresolvable ambiguities about the network.
Our approach works for a wide variety of networks, though not all. It limited to ReLU or otherwise piecewise linear activation functions, though we believe it possible that continuous version of this method could potentially be developed in future work for use with sigmoidal activation. If used with convolutional layers, our method does not account for the symmetries of the network and therefore scales with the size of the input as well as the number features, resulting high computation. Finally, the method is not robust to defenses such as adding noise the outputs of the network, and therefore can be thwarted by a network designer that seeks to hide their weights/arcciitectue..
We believe that the methods we have introduced here will lead to considerable advances in identi- fying neural networks from their outputs, both in the context of deep learning and, more specula- tively. in neuroscience. While the implementation instances, we anticipate future work that optimizes these methods for efficient use with different architectures and scale.
have demonstrated here is effective in small
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Frances S Chance, Larry F Abbott, and Alex D Reyes. Gain modulation from background synaptic input. Neuron, 35(4):773-782, 2002.
Rong Ge, Rohith Kuditipudi, Zhize Li, and Xiang Wang. Learning two-layer neural networks with
symmetric inputs. In International Conference Learning Representations (ICLR), 2019.
Surbhi Goel and Adam Klivans. Learning neural networks with two nonlinear layers in polynomial time. Preprint arXiv1170.06600 2017.
Surbhi Goel, Varun Kanade, Adam Klivans, and Justin Thaler. Reliably learning the ReLU in polynomial time. In Conference on Learning Theory (COLT), 2017.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In Neural Information Processing Systems (NeurIPS), 2018.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning (ICML), 2019a.
Boris Hanin and David Rolnick. Deep ReLU networks have surprisingly few activation patterns. In Neural Information Processing Systems (NeurIPS), 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on ImageNet classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
P Heggelund. Receptive field organization of simple cells in cat striate cortex. Experimental Brain Research, 42(1):899998 98, 1981.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High- 2019.
fidelity extraction of neural network models. arXiv preprint
Konrad P Kording, Christoph Kayser, Wolfgang Einhauser, and Peter Konig, How are complex cell properties adapted to the statistics of natural stimuli? Journal of neurophysiology. 91(1):206-212, 2004.
Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountabillty, and Trans- parency, pp. 1-9. ACM, 2019.
Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 121- 144. Springer, 2019.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In International Conference Machine Learning (ICML), 2017.
Matus Telgarsky, Representation benefits of deep feedforward networks. 15%.vv1500.08/0/ 2015.
Preprint
A USEFUL LEMMATA
Lemma 1 (Isomorphism under scaling), Given an MLP N with ReLU activation, the network (N) is isomorphic to N for every neuron and constant 0.

---

Under review as a conference paper at ICLR 2020
Proof. Suppose that 2 is the ith neuron in layer k. Then, for each neuron 2ft1 in layer & +
of the network N, we have:
k+ (x)
w (x) + bb²)
W ReLU Σ
wh. ReLU(zh (xx + b/: b)
+ by
(I)
By comparison, in network 82 (N), we have:
2/ (x)
w ReLU( + (x)
W ReLU Σ cw ReLLUL (x) + b b + +ob²
ng. w ReLU
nk-1 wh
ReLU(>> (x) + b): bb²
where we used the property that ReLU(cz) cRc(U(x) for any 0.
As expressions T and (2 are equal, we conclude that (2,c(N) is isomorphic to N.
Lemma (Bending hyperplanes). The set of networks N with the following property has measure
zero in the space of networks: There exist neurons 2;-1 and 2 in consecutive layers such that the boundary Byg intersects B Ba-- but does not bend at the intersection.
Proof. Observe that Bug defined by the equation:
0 マ (x) Σ w ReLU(= (x) + b
As it does not bend when it intersects B the gradient of the RHS must remain unchanged when ReLUU= (x) bb-²) flips between active and inactive. Unless another neuron transitions simuitaneously with 2;¹ (an event that occurs with measure zero), this can happen only if W 0, which itself a measure zero event.
B PROOF OF THEOREM
In this proof, we will show how the information we are given by the assumptions of the theorem is enough to recover the weights and biases for each neuron in layer k. We will proceed for each & individually, progressively learning weights between z and each of the neurons the preceding layer (though for skip connections this procedure could also easily be generalized to learn weights from z to earlier layers).
For each of the points p; E A2, suppose that H, is the local hyperplane associated with p; boundary B. The gradient dz 0x (p:) at p; is orthogonal to H,, and we thus already know the direction unknown to us. We will proceed in order through the points de
of the gradient, but its magnitude Pi, P2...., Pm. with the goal of identifying Øx (p.) for each p., to single scaling factor, as this computation will end up giving us the incoming weights for
We begin with Pı by assigning 0s (Pr) arbitrarily to either one of the two unit vectors orthogonal to H4. Due to scaling invariance (Lemma []. the weights of N can be rescaled without changing the function so that 8z
8x (p:) is multiplied by any positive constant. Therefore, our arbitrary choice can be wrong at most its sign, and we need not determine the sign at this stage. Now, suppose towards induction that we have identified dz dx (p:) (up to sign) for 1,...,8 8 1. We wish to identify
8z (pz).
10

---

Under review as a conference paper at ICLR 2020
By assumption (ii), there exists precursor Pr to Pa such that H, and H₃ intersect boundary By Let v, = tz 8x (pr) be our estimate of 2x (pr). for unknown sign ig E [+1,-1] Let Va be 0=
unit normal vector to H,, so that of v. so that it has the same orientation as v, with respect to the surface B,, and thus > Finally, let v = tz' 8x (pr) = 'z' tx (p+) be our estimate of the gradient of z'; where fyr E {+1,- -11 is also an unknown sign (recall that since z' is in layer k 1 we know its gradient up to sign). We will use v and v, to identify v..
ct= ix (P.) for some unknown constant We pick the sign
Suppose that = = is the jth neuron in layer is and that =' = % k-1 is the lih neuron layer Recall that
z(x) xx) E w (eL + b,-
As By is the boundary between inputs for which = is active and inactive, (xL((xx/1(xx)))) by b must equal zero either (Case 1) on H, or (Case 2) on Hg.
In Case 1, we have
0= (p.)
Oz (pr)
W Oe' (pr);
0x
0x
hi Ox
or equivalently:
d2v, 12,, Whsls'V
which gives us the equation:
CV, V, Whitety
Since we know the vectors Va, Vr, v, we are able to deduce the constant c.
A similar equation arises in Case 2:
V, -V,- Wistzte.
giving rise to the same value of c. We thus may complete our induction. In the process, observe that we have calculated a constant W W where the sign t' is +1 in Case 1 and -1 in Case Note that lz-ll can calculated based on whether points towards pr or Ps- Therefore, we have obtained Wh, kytz. which is exactly the weight (up to z-dependent sign) that wished find. Once we have all weights incoming to (up to sign), it is simple to identify the bias for this neuron (up to sign) by calculating the equation of any known local hyperplane for B, and using the known weights and biases from earlier layers.
To complete the proof, we must now also calculate the correct signs l, of the neurons in layer k- 1. Pick some 2 in layer &t and observe that for all points A, there corresponds an equation, obtained by taking gradients in equation
Oz (p.)
Ozt dzk-1 1,
Σ
w
-p.),
Ox
dx
d=1
where i., equals 1 if is on the active side of B Ba-- We can substitute in our (sign-unknown) values for these various quantities:
n2 w Iiat
Now, we may estimate i by function 14 that is 1 if Ps and v, on the same side of B B- This estimate will be wrong exactly when f 1 = -1. Thus, 1 = (1 + tug 11/,)/2, giving us the equation:
1 ++1+- 1 is
tV
W
2
i=1
1
W (t (11 + 11
2
2,
11

---

Under review as a conference paper at ICLR 2020
All the terms of this equation are known, with the exception of t, and the n4-1 variables t giving us linear system in n + variables. For given zÿ, there are n different ps representing the intersections with B. for each z' in layer l 1; choosing these p. should in general give linearly independent constraints. Moreover, the equation in fact a vector equality with dimension Rini hence, it is highly overconstrained system, enabling us to identify the signs L-- for each k-1 This completes the proof of the theorem.
12


Reviews:
[{"review_id": "SyxqKXM6KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach to recover weights of ReLU neural networks by querying the network with specifically constructed inputs. The authors notice that the decision regions of such networks are piece-wise linear corresponding to activations of individual neurons. This allows to identify hyperplanes that constitute the decision boundary and find intersection points of the decision boundaries corresponding to neurons at different layers of the network. However, weights can be recovered only up to permutations of neurons in each layer and up to a constant scaling factor for each layer.\n\nThe algorithm consists of two parts: Identifying parameters of the first layer and subsequent layers. First they sample a lot of line segments and find their intersections with the decision boundary, i.e. where pre-activations equal 0. Then they sample some points around the intersections and estimate the hyperplanes up to a scaling factor and a sign. For the first layer they check whether some of the hyperplanes belong to it. For the consecutive layers they proceed by moving from the intersection points along the hyperplanes until the decision boundary bends. Again, by identifying which of the bends correspond to the intersection of the current layer's hyperplanes with the previous layers' ones, they are able to recover parameters of the current layer by computing the angles of these intersections.\n\nThis paper tackles a very interesting and important problem that might have huge implications for security and many other aspects. However, I'm leaning towards Reject for the following reasons:\n\n1. The algorithm's description is either incomplete or unclear. There are such core functions as PointsOnLine and TestHyperplane, whose pseudo-code would be very helpful for understanding. For example, the authors say that PointsOnLine performs a \"binary search,\" but then this function can find only one (arbitrary) intersection of a line segment with a decision boundary, while each sampled line can intersect multiple ones. If it is not binary search, then the asymptotic analysis given in the end of Sec. 4.2 is incorrect. Even more mysterious is TestHyperplane, from the provided intuition I do not understand how it is possible to distinguish hyperplanes corresponding to the first layer vs. the other layers. In Sec. 4.3, second paragraph, the choice of R is unclear. How to chose it to make sure that the closest boundary intersects it?\n\nThe authors consider a very limited setting of only fully-connected (linear) layers with ReLU activations. In this case it is easy to see that the resulting decision boundary is indeed piece-wise linear with a lot of bending. Authors themselves notice, that \"the algorithm does not account for weight sharing.\" For CNN this will lead to each virtual neuron in each channel to have its own kernel weights, although there must be one kernel per channel. Also the authors admit, that pooling layers affect partitioning of the activation regions, making the proposed approach inapplicable. The authors did not discuss whether the proposed approach can handle batchnorm layers. Such non-linear transformations could pose serious problems. All this rules out applications to, for example, all CNN-based architectures, that prevail in computer vision. The authors mention, that their \"algorithm holds with slight modification\" for ResNets, but as mentioned earlier convolutional, pooling and batchnorm layers make it not so trivial (if at all possible).\n\n2. Experimental evaluation is extremely limited: It is all contained in just one paragraph. Although it is mentioned that \"it is often possible to recover parameters of deep ReLU networks,\" they evaluated their approach on very shallow and narrow networks (only 2 layers, 10 to 50 neurons in each). The immediate question here is why this algorithm is not applied to sufficiently deep NN? At least a network that could classify MNIST reasonably well. Actually, this would be a better proof-of-concept: Given a pre-trained MNIST classifier, apply the proposed method, recover the weights and check if you get the same output as from the original network. Whereas here the evaluation is given as a normalized relative error of the estimated vs. the true weights. Which raises the question of how the scaling factor was chosen? Recall, that the proposed method estimates network's parameters only up to an arbitrary scaling factor. My guess, is that for the Figures 3 and 4 (both right) the estimated weights were re-scaled optimally to minimize the relative error. But in the end, one is interested in recovering the original weights of the network, not relative ones.\n\nI am very confused by Fig. 3 left: Why is the number of queries going down as the number of neurons increases? Should it not be that with more neurons the ambiguity also increases, requiring more queries? Again, this analysis is very limited, it would be very interesting to see, how many more queries one needs for deeper layers of the network. But for this experiments with deeper than 2 layers networks are necessary.\n\n3. The choice of parameters is unclear and not discussed. How long should the line segments be, how many of them. How many points are sampled and within which radius to identify hyperplanes, how to choose 'R'. And how all these choices affect accuracy and performance.\n\nOverall, the paper looks rather incomplete to me and requires a major revision. It will definitely benefit if the \"slight modification\" for the case of ResNets is included. Also, experimental evaluation should be completely re-done and extended.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeEalATKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nMain contribution of the paper\n- The paper proposes a new method to recover the unknown structure of  the network by utilizing the piecewise linearity of ReLU network.\n- Some theoretical explanation of the method is provided.\n\nNote & Questions\n- As far as the author understands, the algorithm does not suppose a fully black-box condition. By seeing the section 4.1 and 4.2, it seems possible to access neurons in the intermediate layers.\n- Also, the proposed method seems to target only a MLP.\n\n\nStrong-points\n- This field is not that thoroughly investigated, and the author proposes a creative method to infer the hidden statistics of the neuron.\n\nConcerns\n- Most of all, the information the experiments conveys is too small to convince the argument of the author. The reviewer could not find the dataset they train (in the Experiment section), and the graph only shows the case of two-layered networks. Moreover, the reviewer couldn't find the explanation of the graph, including their legend (for example, Memorization).\nThe author suggests that this method can be applied to various networks. Still, the reviewer couldn't find any clue that the method actually worked for various settings: different activations, convolutional networks, and so on. More experimental results supporting the argument of the authors are required.\n- Assuming that the network was trained by MNIST and we infer the weight of the networks by the proposed method. Can the recovered network classify the number as well? Then, how the accuracy change?\nMore quantitative results regarding the asking are required.\n- Experimental results for more-than-two layered networks should be provided.\n- Oh.et.al (https://arxiv.org/abs/1711.01768) proposed a blackbox reverse-engineering method and provided experimental settings as well. The author should clarify the novelty and the strong-points of the works compared to the mentioned work.\n\nConclusion\n- The author proposes a new method to recover the weight and bias of the network.\n- The reviewer could not find much clue supporting the author's argument from the experiment section.\n\ninquiries\n- See the Concerns parts.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx7Jukc9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors showed that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability to query the network. The studied problem is very interesting. I have the following questions about this paper:\n\n1. Can the authors provide detailed explanation of Figure 1? For instance start from input (x_1, x_2), and the weight in layer 1 and layer 2, what is the exact form of the function plotted in the middle panel? Also, how the input space is partitioned? I appreciate the authors provide this simple example, but detailed math will help readers to understand this easily.\n\n2. How about the efficiency of the proposed method? Is it NP-hard? I would like to see some analysis of the computational complexity and also some related experimental results.\n\n3. If the ReLU network can be reconstructed, can the input also be reconstructed based on the output? It would be very interesting to show a few example on reconstructing the input. Also, is that possible to even reconstruct the training data based on the released model?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJxjoEJUcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a procedure for reconstructing the architecture and weights of deep ReLU network, given only the ability to query the network (observe network outputs for a sequence of inputs).  The algorithm takes advantage of the piecewise linearity of ReLU networks and an analysis by [Hanin and Rolnick, 2019b] of the boundaries between linear regions as bent hyperplanes.  The observation that a boundary bends only for other boundaries corresponding to neurons in earlier network layers leads to a recursive layer-by-layer procedure for recovering network parameters.  Experiments show ability to recover both random networks and networks trained for a memorization task.  The method is currently limited to ReLU networks and does not account for any parameter-sharing structure, such as that found in convolutional networks.\n\nThe networks used in experiments appear to be substantially smaller (e.g. input/output dimensions on the order of 10 neurons) than those used in real applications.  Is the proposed approach practical to apply to networks used in actual applications?  How does the number of queries per parameter scale? (page 5 mentions sample complexity for recovering the first layer, but it would be helpful to clarify the situation for subsequent layers).\n\nPage 7 states that the proposed algorithm also holds for ResNets, with slight modifications, but defers details to future work.  If the modifications are indeed slight, it would better to include them here as this is an important special case and would increase the potential impact of the paper.\n\nOverall, while the paper does appear to rely heavily on developments made by [Hanin and Rolnick, 2019b], there is a potentially interesting contribution here.  I would appreciate clarification on concerns over practicality and the extension to ResNets.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article studies the identifiability of architecture and weights of a ReLU network from the values of the computed functions, and presents an algorithm to do this. This is a very interesting problem with diverse implications. The reviewers raised concerns about the completeness of various parts of the proposed algorithm and the complexity analysis, some of which were addressed in the author's response. Another concern raised was that the experiments were limited to small networks, with a proof of concept on more realistic networks missing. The revision added experiments with MNIST. Other concerns (which in my opinion could be studied separately) include possible limitations of the approach to networks with no shared weights nor pooling. The reviewers agree that the article concerns an interesting topic that has not been studied in much detail yet. Still, the article would benefit from a more transparent presentation of the algorithm and theoretical analysis, as well as more extensive experiments. 
Decision: Reject