Span Recovery for Deep Neural Networks with Applications to Input Obfuscation

1 INTRODUCTION . Consider the general framework in which we are given an unknown function f : Rn → R , and we want to learn properties about this function given only access to the value f ( x ) for different inputs x . There are many contexts where this framework is applicable , such as blackbox optimization in which we are learning to optimize f ( x ) ( Djolonga et al. , 2013 ) , PAC learning in which we are learning to approximate f ( x ) ( Denis , 1998 ) , adversarial attacks in which we are trying to find adversarial inputs to f ( x ) ( Szegedy et al. , 2013 ) , or structure recovery in which we are learning the structure of f ( x ) . For example in the case when f ( x ) is a neural network , one might want to recover the underlying weights or architecture ( Arora et al. , 2014 ; Zhang et al. , 2017 ) . In this work , we consider the setting when f ( x ) = M ( x ) is a neural network that admits a latent low-dimensional structure , namely M ( x ) = σ ( Ax ) where A ∈ Rk×n is a rank k matrix for some k < n , and σ : Rk → R is some neural network . In this setting , we focus primarily on the goal of recovering the row-span of the weight matrix A . We remark that all our results generalize in a straightforward manner to the case when A is rank r < k. Span recovery of general functions f ( x ) = g ( Ax ) , where g is arbitrary , has been studied in some contexts , and is used to gain important information about the underlying function f . By learning Span ( A ) , we in essence are capturing the relevant subspace of the input to f ; namely , f behaves identically on x as it does on the projection of x onto the row-span of A . In statistics , this is known as effective dimension reduction or the multi-index model Li ( 1991 ) ; Xia et al . ( 2002 ) . Another important motivation for span recovery is for designing adversarial attacks . Given the span of A , we compute the kernel of A , which can be used to fool the function into behaving incorrectly on inputs which are perturbed by vectors in the kernel . Specifically , if x is a legitimate input correctly classified by f and y is a large random vector in the kernel of A , then x+ y will be indistinguishable from noise but we will have f ( x ) = f ( x+ y ) . Several works have considered the problem from an approximation-theoretic standpoint , where the goal is to output a hypothesis function f̃ which approximates f well on a bounded domain . For instance , in the case that A ∈ Rn is a rank 1 matrix and g ( Ax ) is a smooth function with bounded derivatives , Cohen et al . ( 2012 ) gives an adaptive algorithm to approximate f . Their results also give an approximation Ã to A , under the assumption that A is a stochastic vector ( Ai ≥ 0 for each i and ∑ iAi = 1 ) . Extending this result to more general rank k matrices A ∈ Rk×n , Tyagi & Cevher ( 2014 ) and Fornasier et al . ( 2012 ) give algorithms with polynomial sample complexity to find approximations f̃ to twice differentiable functions f . However , their results do not provide any guarantee that the original matrix A itself or a good approximation to its span will be recovered . Specifically , the matrix Ã used in the hypothesis function f̃ ( x ) = g̃ ( Ãx ) of Tyagi & Cevher ( 2014 ) only has moderate correlation with the true row span of A , and always admits some constant factor error ( which can translate into very large error in any subspace approximation ) . Furthermore , all aforementioned works require the strong assumption that the matrix of gradients is well-conditioned ( and full rank ) in order to obtain good approximations f̃ . In contrast , when f ( x ) is a non-differentiable ReLU deep network with only mild assumptions on the weight matrices , we prove that the gradient matrix has rank at least k/2 , which significantly strengthens span recovery guarantees since we do not make any assumptions on the gradient matrix . Finally , Hardt & Woodruff ( 2013 ) gives an adaptive approximate span recovery algorithm with poly ( n ) samples under the assumption that the function g satisfies a norm-preserving condition , which is restrictive and need not ( and does not ) hold for the deep neural networks we consider here . On the empirical side , the experimental results of Tyagi & Cevher ( 2014 ) for function approximation were only carried out for simple one-layer functions , such as the logistic function in one dimension ( where A has rank k = 1 ) , and on linear functions gT ( Ax+ b ) , where g ∈ Rk has i.i.d . Gaussian entries . Moreover , their experiments only attempted to recover approximations Ã to A when A was orthonormal . In addition , Fornasier et al . ( 2012 ) experimentally considers the approximation problem for f when f ( Ax ) is a third degree polynomial of the input . This leaves an experimental gap in understanding the performance of span recovery algorithms on non-smooth , multi-layer deep neural networks . When f ( x ) is a neural network , there have been many results that allow for weight or architecture recovery under additional assumptions ; however nearly all such results are for shallow networks . Arora et al . ( 2014 ) shows that layerwise learning can recover the architecture of random sparse neural networks . Janzamin et al . ( 2015 ) applies tensor methods to recover the weights of a two-layer neural network with certain types of smooth activations and vectorvalued output , whereas Ge et al . ( 2019 ) ; Bakshi et al . ( 2019 ) obtain weight recovery for ReLU activations . Zhang et al . ( 2017 ) shows that SGD can learn the weights of two-layer neural networks with some specific activations . There is also a line of work for improperly two-layer networks , where the algorithm outputs an arbitrary hypothesis function which behaves similarly to the network under a fixed distribution ( Goel et al. , 2017 ; Goel & Klivans , 2019 ) . Learning properties of the network can also lead to so-called model extraction attacks or enhance classical adversarial attacks on neural networks ( Jagielski et al. , 2019 ) . Adversarial attacks are often differentiated into two settings , the whitebox setting where the trained network weights are known or the blackbox setting where the network weights are unknown but attacks can still be achieved on external information , such as knowledge of the dataset , training algorithm , network architecture , or network predictions . Whitebox attacks are well-studied and usually use explicit gradients or optimization procedures to compute adversarial inputs for various tasks such as classification and reinforcement learning ( Szegedy et al. , 2013 ; Huang et al. , 2017 ; Goodfellow et al. , 2014 ) . However , blackbox attacks are more realistic and it is clear that model recovery can enhance these attacks . The work of Papernot et al . ( 2017 ) attacks practical neural networks upon observations of predictions of adaptively chosen inputs , trains a substitute neural network on the observed data , and applies a whitebox attack on the substitute . This setting , nicknamed the practical blackbox setting ( Chen et al. , 2017 ) , is what we work in , as we only observe adaptively chosen predictions without knowledge of the network architecture , dataset , or algorithms . We note that perhaps surprisingly , some of our algorithms are in fact entirely non-adaptive . 1.1 OUR CONTRIBUTIONS . In this paper , we provably show that span recovery for deep neural networks with high precision can be efficiently accomplished with poly ( n ) function evaluations , even when the networks have poly ( n ) layers and the output of the network is a scalar in some finite set . Specifically , for deep networks M ( x ) : Rn → R with ReLU activation functions , we prove that we can recover a subspace V ⊂ Span ( A ) of dimension at least k/2 with polynomially many non-adaptive queries.1 First , we use a volume bounding technique to show that a ReLU network has sufficiently large piece-wise linear sections and that gradient information can be derived from function evaluations . Next , by using a 1We switch to the notation M ( x ) instead of f ( x ) to illustrate that M ( x ) is a neural network , whereas f ( x ) was used to represent a general function with low-rank structure . novel combinatorial analysis of the sign patterns of the ReLU network along with facts in polynomial algebra , we show that the gradient matrix has sufficient rank to allow for partial span recovery . Theorem 3.4 ( informal ) Suppose we have the network M ( x ) = wTφ ( W1φ ( W2φ ( . . .Wdφ ( Ax ) ) . . . ) , where A ∈ Rk×n is rank k , φ is the ReLU and Wi ∈ Rki×ki+1 are weight matrices , with ki possibly much smaller than k. Then , under mild assumptions , there is a non-adaptive algorithm that makesO ( kn log k ) queries toM ( x ) and returns in poly ( n , k ) -time a subspace V ⊆ span ( A ) of dimension at least k2 with probability 1− δ . We remark that span recovery of the first weight layer is provably feasible even in the surprising case when the neural network has many “ bottleneck '' layers with small O ( log ( n ) ) width . Because this does not hold in the linear case , this implies that the non-linearities introduced by activations in deep learning allow for much more information to be captured by the model . Moreover , our algorithm is non-adaptive , which means that the points xi at whichM ( xi ) needs to be evaluated can be chosen in advance and span recovery will succeed with high probability . This has the benefit of being parallelizable , and possibly more difficult to detect when being used for an adversarial attack . In addition , we note that this result generalize to the case when A is rank r < k , in which setting our guarantee will instead be that we recover a subspace of dimension at least r2 contained within the span of A . In contrast with previous papers , we do not assume that the gradient matrix has large rank ; rather our main focus and novelty is to prove this statement under minimal assumptions . We require only two mild assumptions on the weight matrices . The first assumption is on the orthant probabilities of the matrix A , namely that the distribution of sign patterns of a vector Ag , where g ∼ N ( 0 , In ) , is not too far from uniform . Two examples of matrices which satisfy this property are random matrices and matrices with nearly orthogonal rows . The second assumption is a non-degeneracy condition on the matrices Wi , which enforces that products of rows of the matrices Wi result in vectors with non-zero coordinates . Our next result is to show that full span recovery is possible for thresholded networks M ( x ) with twice differentiable activation functions in the inner layers , when the network has a 0/1 threshold function in the last layer and becomes therefore non-differentiable , i.e. , M ( x ) ∈ { 0 , 1 } . Since the activation functions can be arbitrarily non-linear , our algorithm only provides an approximation of the true subspace Span ( A ) , although the distance between the subspace we output and Span ( A ) can be made exponentially small . We need only assume bounds on the first and second derivatives of the activation functions , as well as the fact that we can find inputs x ∈ Rn such that M ( x ) 6= 0 with good probability , and that the gradients of the network near certain points where the threshold evaluates to one are not arbitrarily small . We refer the reader to Section 4 for further details on these assumptions . Under these assumptions , we can apply a novel gradient-estimation scheme to approximately recover the gradient of M ( x ) and the span of A. Theorem 4.3 ( informal ) Suppose we have the network M ( x ) = τ ( σ ( Ax ) ) , where τ : R → { 0 , 1 } is a threshold function and σ : Rk → R is a neural network with twice differentiable activation functions , and such that M satisfies the conditions sketched above ( formally defined in Section 4 ) . Then there is an algorithm that runs in poly ( N ) time , making at most poly ( N ) queries to M ( x ) , where N = poly ( n , k , log ( 1 ) , log ( 1 δ ) ) , and returns with probability 1 − δ a subspace V ⊂ Rn of dimension k such that for any x ∈ V , we have ‖PSpan ( A ) x‖2 ≥ ( 1− ) ‖x‖2 where PSpan ( A ) is the orthogonal projection onto the span of A. Empirically , we verify our theoretical findings by running our span recovery algorithms on randomly generated networks and trained networks . First , we confirm that full recovery is not possible for all architectures when the network layer sizes are small . This implies that the standard assumption that the gradient matrix is full rank does not always hold . However , we see that realistic network architectures lend themselves easily to full span recovery on both random and trained instances . We emphasize that this holds even when the network has many small layers , for example a ReLU network that has 6 hidden layers with [ 784 , 80 , 40 , 30 , 20 , 10 ] nodes , in that order , can still admit full span recovery of the rank 80 weight matrix . Furthermore , we observe that we can effortlessly apply input obfuscation attacks after a successful span recovery and cause misclassifications by tricking the network into classifying noise as normal inputs with high confidence . Specifically , we can inject large amounts of noise in the null space of A to arbitrarily obfuscate the input without changing the output of the network . We demonstrate the utility of this attack on MNIST data , where we use span recovery to generate noisy images that are classified by the network as normal digits with high confidence . We note that this veers away from traditional adversarial attacks , which aim to drastically change the network output with humanly-undetectable changes in the input . In our case , we attempt the arguably more challenging problem of drastically changing the input without affecting the output of the network .

Summary: The paper studies how to recover the span of a NN from a limited number of queries. The problem belongs to the general question of how to reconstruct functions from black-box interaction and it may find application in obfuscation attacks where very large perturbations of the input do not affect the output. The main contribution of the paper is on the theoretical analysis of a simple non-adaptive and a more sophisticated adaptive algorithm. The main finding is that under mild conditions on the structure of the NN partial recovery is possible. The empirical validation show that in practice, it is often the case the full span recover is actually possible, as the structure and weights of common NN are "friendly" enough.