Identifying Weights and Architectures of Unknown ReLU Networks

1 INTRODUCTION . The behavior of deep neural networks is as complex as it is powerful . The relation of individual parameters to the network ’ s output is highly nonlinear and is generally unclear to an external observer . Consequently , it has been widely supposed in the field that it is impossible to recover the parameters of a network merely by observing its output on different inputs . Beyond informing our understanding of deep learning , going from function to parameters could have serious implications for security and privacy . In many deployed deep learning systems , the output is freely available , but the network used to generate that output is not disclosed . The ability to uncover a confidential network not only would make it available for public use but could even expose data used to train the network if such data could be reconstructed from the network ’ s weights . This topic also has implications for the study of biological neural networks . Experimental neuroscientists can record some variables within the brain ( e.g . the output of a complex cell in primary visual cortex ) but not others ( e.g . the pre-synaptic simple cells ) , and many biological neurons appear to be well modeled as the ReLU of a linear combination of their inputs ( Chance et al. , 2002 ) . It would be highly useful if we could reverse engineer the internal components of a neural circuit based on recordings of the output and our choice of input stimuli . In this work , we show that it is , in fact , possible in many cases to recover the structure and weights of an unknown ReLU network by querying it . Our method leverages the fact that a ReLU network is piecewise linear and transitions between linear pieces exactly when one of the ReLUs of the network transitions from its inactive to its active state . We attempt to identify the piecewise linear surfaces in input space where individual neurons transition from inactive to active . For neurons in the first layer , such boundaries are hyperplanes , for which the equations determine the weights and biases of the first layer ( up to sign and scaling ) . For neurons in subsequent layers , the boundaries are “ bent hyperplanes ” that bend where they intersect boundaries associated with earlier layers . Measuring these intersections allows us to recover the weights between the corresponding neurons . Our major contributions are : • We identify how the architecture , weights , and biases of a network can be recovered from the arrangement of boundaries between linear regions in the network . • We implement this procedure and demonstrate its success in recovering trained and untrained ReLU networks . • We show that this algorithm “ degrades gracefully , ” providing partial weights even when full 2 RELATED WORK . Various works within the deep learning literature have considered the problem of learning a network given its output on inputs drawn ( non-adaptively ) from a given distribution . It is known that this problem is in general hard ( Goel et al. , 2017 ) , though positive results have been found for certain specific choices of distribution in the case that the network has only one or two layers ( Ge et al. , 2019 ; Goel & Klivans , 2017 ) . By contrast , we consider the problem of learning about a network of arbitrary depth , given the ability to issue queries at specified input points . In this work , we leverage the theory of linear regions within a ReLU network , an area that has been studied e.g . by Telgarsky ( 2015 ) ; Raghu et al . ( 2017 ) ; Hanin & Rolnick ( 2019a ) . Most recently Hanin & Rolnick ( 2019b ) considered the boundaries between linear regions as arrangements of “ bent hyperplanes ” . Milli et al . ( 2019 ) ; Jagielski et al . ( 2019 ) show the effectiveness of this strategy for networks with one hidden layer . For inference of other properties of unknown networks , see e.g . Oh et al . ( 2019 ) . Neuroscientists have long considered similar problems with biological neural networks , albeit armed with prior knowledge about network structure . For example , it is believed that complex cells in the primary visual cortex , which are often seen as translation-invariant edge detectors , obtain their invariance through what is effectively a two-layer neural network ( Kording et al. , 2004 ) . A first layer is believed to extract edges , while a second layer essentially implements maxpooling . Heggelund ( 1981 ) perform physical experiments akin to our approach of identifying one ReLU at a time , by applying inputs that move individual neurons above their critical threshold one by one . Being able to solve such problems more generically would be useful for a range of neuroscience applications . 3 PRELIMINARIES . 3.1 DEFINITIONS . In general , we will consider fully connected , feed-forward neural networks ( multilayer perceptrons ) with ReLU activations . Each such network N defines a function N ( x ) from input space Rnin to output space Rnout . We denote the layer widths of the network by nin ( input layer ) , n1 , n2 , . . . , nd , nout ( output layer ) . We use Wk to denote the weight matrix from layer ( k − 1 ) to layer k , where layer 0 is the input ; and bk denotes the bias vector for layer k. Given a neuron z in the network , we use z ( x ) to denote its preactivation for input x ∈ Rnin . Thus , for the jth neuron in layer k , we have zkj ( x ) = nk−1∑ i=1 Wkij ReLU ( z k−1 i ( x ) + b k i ) . For each neuron z , we will use Bz to denote the set of x for which z ( x ) = 0 . In general1 , Bz will be an ( nin−1 ) -dimensional piecewise linear surface in Rnin ( see Figure 1 , in which input dimension is 2 and the Bz are simply lines ) . We call Bz the boundary associated with neuron z , and we say that B = ⋃ Bz is the boundary of the overall network . We refer to the connected components of Rnin \B as regions . Throughout this paper , we will make the Linear Regions Assumption : The set of regions is the set of linear pieces of the piecewise linear function N ( x ) . While this assumption has tacitly been made in the prior literature , it is noted in Hanin & Rolnick ( 2019b ) that there are cases where it does not hold – for example , if an entire layer of the network is zeroed out for some inputs . 3.2 ISOMORPHISMS OF NETWORKS . Before showing how to infer the parameters of a neural network , we must consider to what extent these parameters can be inferred unambiguously . Given a network N , there are a number of other networks N ′ that define exactly the same function from input space to output space . We say that such networks are isomorphic to N . For multilayer perceptrons with ReLU activation , we consider the following network isomorphisms : Permutation . The order of neurons in each layer of a network N does not affect the underlying function . Formally , let pk , σ ( N ) be the network obtained fromN by permuting layer k according to σ ( along with the corresponding weight vectors and biases ) . Then , pk , σ ( N ) is isomorphic to N for every layer k and permutation σ . Scaling . Due to the ReLU ’ s equivariance under multiplication , it is possible to scale the incoming weights and biases of any neuron , while inversely scaling the outgoing weights , leaving the overall function unchanged . Formally , for z the ith neuron in layer k and c any positive constant , let sz , c ( N ) be the network obtained from N by replacing Wk·i , bki , and Wk+1 by cWk·i , cbki , and ( 1/c ) W k+1 i· , respectively . It is simple to prove that sz , c ( N ) is isomorphic to N ( see Appendix A ) . Thus , we can hope to recover a network only up to layer-wise permutation and neuron-wise scaling . Formally , pi , σ ( N ) and sz , c ( N ) are generators for a group of isomorphisms of N . ( As we shall see in §5 , some networks also possess additional isomorphisms . ) 4 THE ALGORITHM . 4.1 INTUITION . Consider a network N and neuron z ∈ N , so that Bz is the boundary associated with neuron z . Recall that Bz is piecewise linear . We say that Bz bends at a point if Bz is nonlinear at that point ( that is , if the point lies on the boundary of several regions ) . As observed in Hanin & Rolnick ( 2019b ) , Bz can bend only at points where it intersects boundaries Bz′ for z′ in an earlier layer of 1More precisely , this holds for all but a measure zero set of networks , and any network for which this is not true may simply be perturbed slightly . the network . In general , the converse also holds ; Bz bends wherever it intersects such a boundary Bz′ ( see Appendix A ) . Then , for any two boundaries Bz and Bz′ , one of the following must hold : Bz bends at their intersection ( in which case z occurs in a deeper layer of the network ) , Bz′ bends ( in which case z′ occurs in a deeper layer ) , or neither bends ( in which case z and z′ occur in the same layer ) . It is not possible for both Bz and Bz′ to bend at their intersection – unless that intersection is also contained in another boundary , which is vanishingly unlikely in general . Thus , the architecture of the network can be determined by evaluating the boundaries Bz and where they bend in relation to one another . Moving beyond architecture , the weights and biases of the network can also be determined from the boundaries , one layer at a time . Boundaries for neurons in the first layer do not bend and are simply hyperplanes ; the equations of these hyperplanes expose the weights from the input to the first layer ( up to permutation , scaling , and sign ) . For each subsequent layer , the weight between neurons z and z′ can be determined by calculating how Bz′ bends when it crosses Bz . The details of our algorithm below are intended to make these intuitions concrete and perform efficiently even when the input space is high-dimensional . Algorithm 1 The first layer Initialize P1 = P2 = S1 = { } for t = 1 , . . . , L do Sample line segment ` P1 ← P1 ∪ PointsOnLine ( ` ) end for for p ∈ P1 do H = InferHyperplane ( p ) if TestHyperplane ( H ) then S1 ← S1 ∪ GetParams ( H ) else P2 ← P2 ∪ { p } end if end for return Parameters S1 , unused sample points P2 Algorithm 2 Additional layers Input Pk and S1 , . . . , Sk−1 Initialize Sk = { } for p1 ∈ Pk−1 on boundary Bz do Initialize Az = { p1 } , Lz = Hz = { } while Lz 6⊇ Layer k − 1 do Pick pi ∈ A and v p′ , Bz′ = ClosestBoundary ( pi , v ) if p′ on boundary then Az ← Az ∪ { p′ + } Lz ← Lz ∪ { z′ } Hz ← Hz ∪ { InferHyperplane ( pi ) } else Pk ← Pk ∪ { p1 } ; break end if end while if Lz ⊇ Layer k − 1 then Sk ← GetParams ( Tz ) end if end for return Parameters Sk , unused sample points Pk+1

Summary: This paper introduces an approach to recover weights of ReLU neural networks by querying the network with specifically constructed inputs. The authors notice that the decision regions of such networks are piece-wise linear corresponding to activations of individual neurons. This allows to identify hyperplanes that constitute the decision boundary and find intersection points of the decision boundaries corresponding to neurons at different layers of the network. However, weights can be recovered only up to permutations of neurons in each layer and up to a constant scaling factor for each layer.