Identifying Weights and Architectures of Unknown ReLU Networks

1 INTRODUCTION . The behavior of deep neural networks is as complex as it is powerful . The relation of individual parameters to the network ’ s output is highly nonlinear and is generally unclear to an external observer . Consequently , it has been widely supposed in the field that it is impossible to recover the parameters of a network merely by observing its output on different inputs . Beyond informing our understanding of deep learning , going from function to parameters could have serious implications for security and privacy . In many deployed deep learning systems , the output is freely available , but the network used to generate that output is not disclosed . The ability to uncover a confidential network not only would make it available for public use but could even expose data used to train the network if such data could be reconstructed from the network ’ s weights . This topic also has implications for the study of biological neural networks . Experimental neuroscientists can record some variables within the brain ( e.g . the output of a complex cell in primary visual cortex ) but not others ( e.g . the pre-synaptic simple cells ) , and many biological neurons appear to be well modeled as the ReLU of a linear combination of their inputs ( Chance et al. , 2002 ) . It would be highly useful if we could reverse engineer the internal components of a neural circuit based on recordings of the output and our choice of input stimuli . In this work , we show that it is , in fact , possible in many cases to recover the structure and weights of an unknown ReLU network by querying it . Our method leverages the fact that a ReLU network is piecewise linear and transitions between linear pieces exactly when one of the ReLUs of the network transitions from its inactive to its active state . We attempt to identify the piecewise linear surfaces in input space where individual neurons transition from inactive to active . For neurons in the first layer , such boundaries are hyperplanes , for which the equations determine the weights and biases of the first layer ( up to sign and scaling ) . For neurons in subsequent layers , the boundaries are “ bent hyperplanes ” that bend where they intersect boundaries associated with earlier layers . Measuring these intersections allows us to recover the weights between the corresponding neurons . Our major contributions are : • We identify how the architecture , weights , and biases of a network can be recovered from the arrangement of boundaries between linear regions in the network . • We implement this procedure and demonstrate its success in recovering trained and untrained ReLU networks . • We show that this algorithm “ degrades gracefully , ” providing partial weights even when full 2 RELATED WORK . Various works within the deep learning literature have considered the problem of learning a network given its output on inputs drawn ( non-adaptively ) from a given distribution . It is known that this problem is in general hard ( Goel et al. , 2017 ) , though positive results have been found for certain specific choices of distribution in the case that the network has only one or two layers ( Ge et al. , 2019 ; Goel & Klivans , 2017 ) . By contrast , we consider the problem of learning about a network of arbitrary depth , given the ability to issue queries at specified input points . In this work , we leverage the theory of linear regions within a ReLU network , an area that has been studied e.g . by Telgarsky ( 2015 ) ; Raghu et al . ( 2017 ) ; Hanin & Rolnick ( 2019a ) . Most recently Hanin & Rolnick ( 2019b ) considered the boundaries between linear regions as arrangements of “ bent hyperplanes ” . Milli et al . ( 2019 ) ; Jagielski et al . ( 2019 ) show the effectiveness of this strategy for networks with one hidden layer . For inference of other properties of unknown networks , see e.g . Oh et al . ( 2019 ) . Neuroscientists have long considered similar problems with biological neural networks , albeit armed with prior knowledge about network structure . For example , it is believed that complex cells in the primary visual cortex , which are often seen as translation-invariant edge detectors , obtain their invariance through what is effectively a two-layer neural network ( Kording et al. , 2004 ) . A first layer is believed to extract edges , while a second layer essentially implements maxpooling . Heggelund ( 1981 ) perform physical experiments akin to our approach of identifying one ReLU at a time , by applying inputs that move individual neurons above their critical threshold one by one . Being able to solve such problems more generically would be useful for a range of neuroscience applications . 3 PRELIMINARIES . 3.1 DEFINITIONS . In general , we will consider fully connected , feed-forward neural networks ( multilayer perceptrons ) with ReLU activations . Each such network N defines a function N ( x ) from input space Rnin to output space Rnout . We denote the layer widths of the network by nin ( input layer ) , n1 , n2 , . . . , nd , nout ( output layer ) . We use Wk to denote the weight matrix from layer ( k − 1 ) to layer k , where layer 0 is the input ; and bk denotes the bias vector for layer k. Given a neuron z in the network , we use z ( x ) to denote its preactivation for input x ∈ Rnin . Thus , for the jth neuron in layer k , we have zkj ( x ) = nk−1∑ i=1 Wkij ReLU ( z k−1 i ( x ) + b k i ) . For each neuron z , we will use Bz to denote the set of x for which z ( x ) = 0 . In general1 , Bz will be an ( nin−1 ) -dimensional piecewise linear surface in Rnin ( see Figure 1 , in which input dimension is 2 and the Bz are simply lines ) . We call Bz the boundary associated with neuron z , and we say that B = ⋃ Bz is the boundary of the overall network . We refer to the connected components of Rnin \B as regions . Throughout this paper , we will make the Linear Regions Assumption : The set of regions is the set of linear pieces of the piecewise linear function N ( x ) . While this assumption has tacitly been made in the prior literature , it is noted in Hanin & Rolnick ( 2019b ) that there are cases where it does not hold – for example , if an entire layer of the network is zeroed out for some inputs . 3.2 ISOMORPHISMS OF NETWORKS . Before showing how to infer the parameters of a neural network , we must consider to what extent these parameters can be inferred unambiguously . Given a network N , there are a number of other networks N ′ that define exactly the same function from input space to output space . We say that such networks are isomorphic to N . For multilayer perceptrons with ReLU activation , we consider the following network isomorphisms : Permutation . The order of neurons in each layer of a network N does not affect the underlying function . Formally , let pk , σ ( N ) be the network obtained fromN by permuting layer k according to σ ( along with the corresponding weight vectors and biases ) . Then , pk , σ ( N ) is isomorphic to N for every layer k and permutation σ . Scaling . Due to the ReLU ’ s equivariance under multiplication , it is possible to scale the incoming weights and biases of any neuron , while inversely scaling the outgoing weights , leaving the overall function unchanged . Formally , for z the ith neuron in layer k and c any positive constant , let sz , c ( N ) be the network obtained from N by replacing Wk·i , bki , and Wk+1 by cWk·i , cbki , and ( 1/c ) W k+1 i· , respectively . It is simple to prove that sz , c ( N ) is isomorphic to N ( see Appendix A ) . Thus , we can hope to recover a network only up to layer-wise permutation and neuron-wise scaling . Formally , pi , σ ( N ) and sz , c ( N ) are generators for a group of isomorphisms of N . ( As we shall see in §5 , some networks also possess additional isomorphisms . ) 4 THE ALGORITHM . 4.1 INTUITION . Consider a network N and neuron z ∈ N , so that Bz is the boundary associated with neuron z . Recall that Bz is piecewise linear . We say that Bz bends at a point if Bz is nonlinear at that point ( that is , if the point lies on the boundary of several regions ) . As observed in Hanin & Rolnick ( 2019b ) , Bz can bend only at points where it intersects boundaries Bz′ for z′ in an earlier layer of 1More precisely , this holds for all but a measure zero set of networks , and any network for which this is not true may simply be perturbed slightly . the network . In general , the converse also holds ; Bz bends wherever it intersects such a boundary Bz′ ( see Appendix A ) . Then , for any two boundaries Bz and Bz′ , one of the following must hold : Bz bends at their intersection ( in which case z occurs in a deeper layer of the network ) , Bz′ bends ( in which case z′ occurs in a deeper layer ) , or neither bends ( in which case z and z′ occur in the same layer ) . It is not possible for both Bz and Bz′ to bend at their intersection – unless that intersection is also contained in another boundary , which is vanishingly unlikely in general . Thus , the architecture of the network can be determined by evaluating the boundaries Bz and where they bend in relation to one another . Moving beyond architecture , the weights and biases of the network can also be determined from the boundaries , one layer at a time . Boundaries for neurons in the first layer do not bend and are simply hyperplanes ; the equations of these hyperplanes expose the weights from the input to the first layer ( up to permutation , scaling , and sign ) . For each subsequent layer , the weight between neurons z and z′ can be determined by calculating how Bz′ bends when it crosses Bz . The details of our algorithm below are intended to make these intuitions concrete and perform efficiently even when the input space is high-dimensional . Algorithm 1 The first layer Initialize P1 = P2 = S1 = { } for t = 1 , . . . , L do Sample line segment ` P1 ← P1 ∪ PointsOnLine ( ` ) end for for p ∈ P1 do H = InferHyperplane ( p ) if TestHyperplane ( H ) then S1 ← S1 ∪ GetParams ( H ) else P2 ← P2 ∪ { p } end if end for return Parameters S1 , unused sample points P2 Algorithm 2 Additional layers Input Pk and S1 , . . . , Sk−1 Initialize Sk = { } for p1 ∈ Pk−1 on boundary Bz do Initialize Az = { p1 } , Lz = Hz = { } while Lz 6⊇ Layer k − 1 do Pick pi ∈ A and v p′ , Bz′ = ClosestBoundary ( pi , v ) if p′ on boundary then Az ← Az ∪ { p′ + } Lz ← Lz ∪ { z′ } Hz ← Hz ∪ { InferHyperplane ( pi ) } else Pk ← Pk ∪ { p1 } ; break end if end while if Lz ⊇ Layer k − 1 then Sk ← GetParams ( Tz ) end if end for return Parameters Sk , unused sample points Pk+1

Summary: This paper introduces a procedure for reconstructing the architecture and weights of deep ReLU network, given only the ability to query the network (observe network outputs for a sequence of inputs).  The algorithm takes advantage of the piecewise linearity of ReLU networks and an analysis by [Hanin and Rolnick, 2019b] of the boundaries between linear regions as bent hyperplanes.  The observation that a boundary bends only for other boundaries corresponding to neurons in earlier network layers leads to a recursive layer-by-layer procedure for recovering network parameters.  Experiments show ability to recover both random networks and networks trained for a memorization task.  The method is currently limited to ReLU networks and does not account for any parameter-sharing structure, such as that found in convolutional networks.