Span Recovery for Deep Neural Networks with Applications to Input Obfuscation

1 INTRODUCTION . Consider the general framework in which we are given an unknown function f : Rn → R , and we want to learn properties about this function given only access to the value f ( x ) for different inputs x . There are many contexts where this framework is applicable , such as blackbox optimization in which we are learning to optimize f ( x ) ( Djolonga et al. , 2013 ) , PAC learning in which we are learning to approximate f ( x ) ( Denis , 1998 ) , adversarial attacks in which we are trying to find adversarial inputs to f ( x ) ( Szegedy et al. , 2013 ) , or structure recovery in which we are learning the structure of f ( x ) . For example in the case when f ( x ) is a neural network , one might want to recover the underlying weights or architecture ( Arora et al. , 2014 ; Zhang et al. , 2017 ) . In this work , we consider the setting when f ( x ) = M ( x ) is a neural network that admits a latent low-dimensional structure , namely M ( x ) = σ ( Ax ) where A ∈ Rk×n is a rank k matrix for some k < n , and σ : Rk → R is some neural network . In this setting , we focus primarily on the goal of recovering the row-span of the weight matrix A . We remark that all our results generalize in a straightforward manner to the case when A is rank r < k. Span recovery of general functions f ( x ) = g ( Ax ) , where g is arbitrary , has been studied in some contexts , and is used to gain important information about the underlying function f . By learning Span ( A ) , we in essence are capturing the relevant subspace of the input to f ; namely , f behaves identically on x as it does on the projection of x onto the row-span of A . In statistics , this is known as effective dimension reduction or the multi-index model Li ( 1991 ) ; Xia et al . ( 2002 ) . Another important motivation for span recovery is for designing adversarial attacks . Given the span of A , we compute the kernel of A , which can be used to fool the function into behaving incorrectly on inputs which are perturbed by vectors in the kernel . Specifically , if x is a legitimate input correctly classified by f and y is a large random vector in the kernel of A , then x+ y will be indistinguishable from noise but we will have f ( x ) = f ( x+ y ) . Several works have considered the problem from an approximation-theoretic standpoint , where the goal is to output a hypothesis function f̃ which approximates f well on a bounded domain . For instance , in the case that A ∈ Rn is a rank 1 matrix and g ( Ax ) is a smooth function with bounded derivatives , Cohen et al . ( 2012 ) gives an adaptive algorithm to approximate f . Their results also give an approximation Ã to A , under the assumption that A is a stochastic vector ( Ai ≥ 0 for each i and ∑ iAi = 1 ) . Extending this result to more general rank k matrices A ∈ Rk×n , Tyagi & Cevher ( 2014 ) and Fornasier et al . ( 2012 ) give algorithms with polynomial sample complexity to find approximations f̃ to twice differentiable functions f . However , their results do not provide any guarantee that the original matrix A itself or a good approximation to its span will be recovered . Specifically , the matrix Ã used in the hypothesis function f̃ ( x ) = g̃ ( Ãx ) of Tyagi & Cevher ( 2014 ) only has moderate correlation with the true row span of A , and always admits some constant factor error ( which can translate into very large error in any subspace approximation ) . Furthermore , all aforementioned works require the strong assumption that the matrix of gradients is well-conditioned ( and full rank ) in order to obtain good approximations f̃ . In contrast , when f ( x ) is a non-differentiable ReLU deep network with only mild assumptions on the weight matrices , we prove that the gradient matrix has rank at least k/2 , which significantly strengthens span recovery guarantees since we do not make any assumptions on the gradient matrix . Finally , Hardt & Woodruff ( 2013 ) gives an adaptive approximate span recovery algorithm with poly ( n ) samples under the assumption that the function g satisfies a norm-preserving condition , which is restrictive and need not ( and does not ) hold for the deep neural networks we consider here . On the empirical side , the experimental results of Tyagi & Cevher ( 2014 ) for function approximation were only carried out for simple one-layer functions , such as the logistic function in one dimension ( where A has rank k = 1 ) , and on linear functions gT ( Ax+ b ) , where g ∈ Rk has i.i.d . Gaussian entries . Moreover , their experiments only attempted to recover approximations Ã to A when A was orthonormal . In addition , Fornasier et al . ( 2012 ) experimentally considers the approximation problem for f when f ( Ax ) is a third degree polynomial of the input . This leaves an experimental gap in understanding the performance of span recovery algorithms on non-smooth , multi-layer deep neural networks . When f ( x ) is a neural network , there have been many results that allow for weight or architecture recovery under additional assumptions ; however nearly all such results are for shallow networks . Arora et al . ( 2014 ) shows that layerwise learning can recover the architecture of random sparse neural networks . Janzamin et al . ( 2015 ) applies tensor methods to recover the weights of a two-layer neural network with certain types of smooth activations and vectorvalued output , whereas Ge et al . ( 2019 ) ; Bakshi et al . ( 2019 ) obtain weight recovery for ReLU activations . Zhang et al . ( 2017 ) shows that SGD can learn the weights of two-layer neural networks with some specific activations . There is also a line of work for improperly two-layer networks , where the algorithm outputs an arbitrary hypothesis function which behaves similarly to the network under a fixed distribution ( Goel et al. , 2017 ; Goel & Klivans , 2019 ) . Learning properties of the network can also lead to so-called model extraction attacks or enhance classical adversarial attacks on neural networks ( Jagielski et al. , 2019 ) . Adversarial attacks are often differentiated into two settings , the whitebox setting where the trained network weights are known or the blackbox setting where the network weights are unknown but attacks can still be achieved on external information , such as knowledge of the dataset , training algorithm , network architecture , or network predictions . Whitebox attacks are well-studied and usually use explicit gradients or optimization procedures to compute adversarial inputs for various tasks such as classification and reinforcement learning ( Szegedy et al. , 2013 ; Huang et al. , 2017 ; Goodfellow et al. , 2014 ) . However , blackbox attacks are more realistic and it is clear that model recovery can enhance these attacks . The work of Papernot et al . ( 2017 ) attacks practical neural networks upon observations of predictions of adaptively chosen inputs , trains a substitute neural network on the observed data , and applies a whitebox attack on the substitute . This setting , nicknamed the practical blackbox setting ( Chen et al. , 2017 ) , is what we work in , as we only observe adaptively chosen predictions without knowledge of the network architecture , dataset , or algorithms . We note that perhaps surprisingly , some of our algorithms are in fact entirely non-adaptive . 1.1 OUR CONTRIBUTIONS . In this paper , we provably show that span recovery for deep neural networks with high precision can be efficiently accomplished with poly ( n ) function evaluations , even when the networks have poly ( n ) layers and the output of the network is a scalar in some finite set . Specifically , for deep networks M ( x ) : Rn → R with ReLU activation functions , we prove that we can recover a subspace V ⊂ Span ( A ) of dimension at least k/2 with polynomially many non-adaptive queries.1 First , we use a volume bounding technique to show that a ReLU network has sufficiently large piece-wise linear sections and that gradient information can be derived from function evaluations . Next , by using a 1We switch to the notation M ( x ) instead of f ( x ) to illustrate that M ( x ) is a neural network , whereas f ( x ) was used to represent a general function with low-rank structure . novel combinatorial analysis of the sign patterns of the ReLU network along with facts in polynomial algebra , we show that the gradient matrix has sufficient rank to allow for partial span recovery . Theorem 3.4 ( informal ) Suppose we have the network M ( x ) = wTφ ( W1φ ( W2φ ( . . .Wdφ ( Ax ) ) . . . ) , where A ∈ Rk×n is rank k , φ is the ReLU and Wi ∈ Rki×ki+1 are weight matrices , with ki possibly much smaller than k. Then , under mild assumptions , there is a non-adaptive algorithm that makesO ( kn log k ) queries toM ( x ) and returns in poly ( n , k ) -time a subspace V ⊆ span ( A ) of dimension at least k2 with probability 1− δ . We remark that span recovery of the first weight layer is provably feasible even in the surprising case when the neural network has many “ bottleneck '' layers with small O ( log ( n ) ) width . Because this does not hold in the linear case , this implies that the non-linearities introduced by activations in deep learning allow for much more information to be captured by the model . Moreover , our algorithm is non-adaptive , which means that the points xi at whichM ( xi ) needs to be evaluated can be chosen in advance and span recovery will succeed with high probability . This has the benefit of being parallelizable , and possibly more difficult to detect when being used for an adversarial attack . In addition , we note that this result generalize to the case when A is rank r < k , in which setting our guarantee will instead be that we recover a subspace of dimension at least r2 contained within the span of A . In contrast with previous papers , we do not assume that the gradient matrix has large rank ; rather our main focus and novelty is to prove this statement under minimal assumptions . We require only two mild assumptions on the weight matrices . The first assumption is on the orthant probabilities of the matrix A , namely that the distribution of sign patterns of a vector Ag , where g ∼ N ( 0 , In ) , is not too far from uniform . Two examples of matrices which satisfy this property are random matrices and matrices with nearly orthogonal rows . The second assumption is a non-degeneracy condition on the matrices Wi , which enforces that products of rows of the matrices Wi result in vectors with non-zero coordinates . Our next result is to show that full span recovery is possible for thresholded networks M ( x ) with twice differentiable activation functions in the inner layers , when the network has a 0/1 threshold function in the last layer and becomes therefore non-differentiable , i.e. , M ( x ) ∈ { 0 , 1 } . Since the activation functions can be arbitrarily non-linear , our algorithm only provides an approximation of the true subspace Span ( A ) , although the distance between the subspace we output and Span ( A ) can be made exponentially small . We need only assume bounds on the first and second derivatives of the activation functions , as well as the fact that we can find inputs x ∈ Rn such that M ( x ) 6= 0 with good probability , and that the gradients of the network near certain points where the threshold evaluates to one are not arbitrarily small . We refer the reader to Section 4 for further details on these assumptions . Under these assumptions , we can apply a novel gradient-estimation scheme to approximately recover the gradient of M ( x ) and the span of A. Theorem 4.3 ( informal ) Suppose we have the network M ( x ) = τ ( σ ( Ax ) ) , where τ : R → { 0 , 1 } is a threshold function and σ : Rk → R is a neural network with twice differentiable activation functions , and such that M satisfies the conditions sketched above ( formally defined in Section 4 ) . Then there is an algorithm that runs in poly ( N ) time , making at most poly ( N ) queries to M ( x ) , where N = poly ( n , k , log ( 1 ) , log ( 1 δ ) ) , and returns with probability 1 − δ a subspace V ⊂ Rn of dimension k such that for any x ∈ V , we have ‖PSpan ( A ) x‖2 ≥ ( 1− ) ‖x‖2 where PSpan ( A ) is the orthogonal projection onto the span of A. Empirically , we verify our theoretical findings by running our span recovery algorithms on randomly generated networks and trained networks . First , we confirm that full recovery is not possible for all architectures when the network layer sizes are small . This implies that the standard assumption that the gradient matrix is full rank does not always hold . However , we see that realistic network architectures lend themselves easily to full span recovery on both random and trained instances . We emphasize that this holds even when the network has many small layers , for example a ReLU network that has 6 hidden layers with [ 784 , 80 , 40 , 30 , 20 , 10 ] nodes , in that order , can still admit full span recovery of the rank 80 weight matrix . Furthermore , we observe that we can effortlessly apply input obfuscation attacks after a successful span recovery and cause misclassifications by tricking the network into classifying noise as normal inputs with high confidence . Specifically , we can inject large amounts of noise in the null space of A to arbitrarily obfuscate the input without changing the output of the network . We demonstrate the utility of this attack on MNIST data , where we use span recovery to generate noisy images that are classified by the network as normal digits with high confidence . We note that this veers away from traditional adversarial attacks , which aim to drastically change the network output with humanly-undetectable changes in the input . In our case , we attempt the arguably more challenging problem of drastically changing the input without affecting the output of the network .

Summary: The paper considers the problem of recovering the span of the latent variables of a neural network with various activation functions. More precisely, if we write a neural network as M(x) = f(Ax), for some neural network f, and a matrix A: R^{k x d} -> R^{k}, k << d, we wish to recover the row span of A. The authors consider ReLU activations -- in which case they can recover at least "half" of the row span with ~kd queries to M(x), and smooth activations -- in which case they can approximately recover the row span in poly(k,d) queries. The authors also consider (empirically) applications of these algorithms to "input obfuscation": namely generating samples which are effectively noise, but the network classifies them as "structure" (e.g. digits on MNIST).