LEARNING DISTRIBUTIONS GENERATED  BY SINGLE-LAYER RELU  NETWORKS  IN  THE PRESENCE  OF ARBITRARY OUTLIERS

d δ ) sam- ples are sufficient for our algorithm to estimate the NN parameters within an error of with probability 1− δ when the probability of a sample being uncorrupted is p and the problem dimension is d. Our theoretical and simulation results provide insights on how the estimation of the NN parameters depends on the probability of a sample being uncorrupted , the number of samples , and the problem dimension . 1 INTRODUCTION . A fundamental challenge in machine learning and statistics is to estimate a high dimensional distribution given a set of observed data samples from the distribution . One solution technique is the deep generative method that models the unknown distribution as the output distribution of a neural network ( NN ) when the input of the NN is drawn from a known distribution like standard Gaussian which is motivated from the case of image generation by generative adversarial networks ( GANs ) ( Goodfellow et al . ( 2014 ) ; Arjovsky et al . ( 2017 ) ; Radford et al . ( 2015 ) ; Kingma & Welling ( 2013 ) ; Van Oord et al . ( 2016 ) ) . Then , the unknown distribution is estimated by learning the NN parameters from the data samples . Several approaches like GANs ( Goodfellow et al . ( 2014 ) ; Arjovsky et al . ( 2017 ) ; Radford et al . ( 2015 ) ) , variational autoencoders ( Kingma & Welling ( 2013 ) ) , and autoregressive models ( Van Oord et al . ( 2016 ) ) have been proposed to train the NN . However , they do not provide theoretical guarantees for the parameter learning . So , we address the open problem of computing the sample complexity for the parameter estimation of a single-layer NN with rectified linear unit ( ReLU ) activation using a corrupted dataset in the unsupervised learning framework . Here , the corrupted dataset refers to the model wherein the output samples consists of a fraction of arbitrary outliers introduced by an adversary ( Byzantine ) . Motivated by the deep generative models , we consider the unsupervised learning framework , i.e. , we assume the knowledge of the output samples but , the corresponding inputs are unknown and drawn from the standard Gaussian distribution . We start with a brief review of the related literature . The NN parameter estimation has been considered under both supervised and unsupervised learning frameworks . The estimation rely on the stochastic gradient descent ( SGD ) -based algorithms ( Goel et al . ( 2018 ) ; Allen-Zhu et al . ( 2019 ) ; Chen et al . ( 2020 ) ; Oymak ( 2019 ) ; Mazumdar & Rawat ( 2018 ) ; Wu et al . ( 2019 ) ; Lei et al . ( 2020 ) ) ; or the gradient descent ( GD ) -based approach ( Cao & Gu ( 2019 ) ; Du et al . ( 2019 ) ) . A major shortcoming of these works is that they assume that the available samples are not corrupted by noise or outliers . Several works in the literature have addressed the problem of corrupted samples in the context of supervised learning . The existing works have considered the learning of the parameters of a NN with ReLU activation using both SGD-based ( Bakshi et al . ( 2019 ) ; Goel et al . ( 2019 ) ; Mukherjee & Muthukumar ( 2020 ) ) and GD-based algorithms ( Zhang et al . ( 2019 ) ; Frei et al . ( 2020 ) ; Vempala & Wilmes ( 2019 ) ) . However , to the best of our knowledge , the estimation of a NN in the presence of noise or outliers in the unsupervised learning framework has not been studied in the literature . Our setting of unsupervised parameter estimation is also related to the area of robust statistics as described below . We recall that the goal of our paper is to estimate the parameters of a NN using corrupted output samples assuming that the input distribution is Gaussian . Mathematically , this problem is equivalent to estimating the parameters of a truncated Gaussian distribution ( see Section 2 for details ) . The area of robust statistics also deals with the estimation of high dimensional distributions like Gaussian , Gaussian product , and Gaussian mixture distributions where a constant fraction of the samples were corrupted by noise ( Huber ( 2004 ) ; Hampel et al . ( 2011 ) ; Lai et al . ( 2016 ) ; Diakonikolas et al . ( 2019 ) ; Diakonikolas & Kane ( 2019 ) ; Kane ( 2021 ) ) . However , in this paper , we consider parameter estimation from truncated Gaussian samples in the adversarial setting which has not been studied in the literature . We present our major contributions as the following . • Parameter Estimation ( Sections 2 and 3 ) : We estimate the parameters ( weight matrix and bias vector ) of a single-layer NN with ReLU in the unsupervised learning framework where each output sample can potentially be an arbitrary outlier with a fixed probability . We propose an estimation algorithm that has two steps : ( i ) we estimate the row norms of the weight matrix and the bias vector using GD along with either the median or trimmed mean-based filters to mitigate the effect of arbitrary outliers , and ( ii ) we estimate the angles between any two row vectors of the weight matrix using a simple geometric result ( Williamson & Shmoys , 2011 , Lemma 6.7 ) . • Theoretical guarantees ( Section 4 ) : We show that the proposed algorithm requires Ω ( 1 p4 log d δ ) samples to estimate the network parameters within an error of with probability 1− δ when the probability of a sample being uncorrupted is p. Here , > 0 represents the estimation error . • Empirical validation ( Section 5 ) : We evaluate the performance of our algorithm empirically in terms of its relation to the probability of a sample being uncorrupted , number of samples , and dimension ( see Figs . 1 and 2 ) . We see that our proposed filtering schemes ensure robustness to the arbitrary outliers . Also , the performance of our algorithm improves with the increase in the probability of a sample being uncorrupted and the number of samples , as expected . Further , we note that the estimation error increases slowly as the dimension grows for a fixed number of samples in the presence of arbitrary outliers . These observations from the empirical results are consistent with our theoretical results . To summarize , in this paper , we estimate the parameters of a single-layer NN with ReLU activation in the presence of arbitrary outliers . The results obtained in this paper provide insights to generalize to a multi-layer NN with various activation functions and obtain generalization bounds . 2 LEARNING A NEURAL NETWORK MODEL USING CORRUPTED OUTPUT SAMPLES . We consider a single-layer ReLU NN . Let the weight matrix of the NN be denoted by W ∈ Rd×k and the bias vector by b ∈ Rd . The input to the NN is denoted by the latent variable z ∈ Rk . We assume that the variable z is drawn from the standard Gaussian distribution . The Gaussianity assumption is for mathematical tractability , and it is motivated by a popular generative approach to estimate a high-dimensional distribution from observed samples in the case of image generation by GANs Goodfellow et al . ( 2014 ) ; Radford et al . ( 2015 ) ; Arjovsky et al . ( 2017 ) . Thus , the output of the network is the random vector x ∈ Rd given by x = ReLU ( Wz + b ) , where z ∼ N ( 0 , Ik ) , ( 1 ) where N ( 0 , Ik ) denotes the Gaussian distribution with mean 0 and covariance matrix Ik which is the identity matrix of size k×k . Let the distribution of the random vector x be denoted byD ( W , b ) . Then , our goal is to estimate the unknown parameters W and b of the distribution D ( W , b ) using the knowledge of n output samples x1 , x2 , . . . , xn . Here , we assume that a fraction of the samples are arbitrary outliers . The distribution of the observed samples are modeled using the Huber ’ s pcontamination model 1 defined as the following . Definition 2.1 ( Huber ’ s p-Contamination Model Huber ( 1964 ) ) . The observed samples are said to be following Huber ’ s contamination model if any given sample is drawn from the true distribution D and an arbitrary distribution G with probability p , and 1 − p , respectively . In other words , the observed samples are drawn from the mixture distribution , Dp ( W , b ) given by Dp ( W , b ) = pD ( W , b ) + ( 1− p ) G. ( 2 ) Note that if b has large negative values , then most of the output samples are zeros due to the ReLU operation . Further , if W is a zero matrix , then any negative coordinate of b can not be identified as it is reset to zero after the ReLU operation . Further , in Wu et al . ( 2019 ) , the authors have shown that if b ∈ Rd then exponentially large number of samples are required to estimate the bias . This holds true in our case as in the presence of arbitrary outliers if b has large negative values , then we would require larger number of samples to estimate the bias vector b compared to that in Wu et al . ( 2019 ) . Hence , we also assume b to be non-negative . Our estimation problem is challenging due to two reasons : 1 ) ReLU operation is not invertible , and therefore , estimation of W and b utilizing maximum likelihood of the probability density function of the random vector x is intractable ; 2 ) the distribution of the corrupted samples is unknown . We tackle these issues using a new formulation combining the GD algorithm and a filtering technique . Note that the GD algorithm is similar to that in Wu et al . ( 2019 ) wherein the authors use the SGD algorithm . The proposed algorithm is presented in the next section . 3 ESTIMATION OF PARAMETERS OF SINGLE-LAYER NN . To design our estimation algorithm , we first note that the weight matrix W may not be identifiable from the distribution D ( W , b ) . This is because of the fact that for any matrices W1 and W2 if W1W T 1 = W2W T 2 then there exists a unitary matrix Q such that W2 = W1Q . As z ∼ N ( 0 , Ik ) , we have Qz ∼ N ( 0 , Ik ) . Hence , for any vector b , we have D ( W1 , b ) = D ( W2 , b ) ( Wu et al . ( 2019 ) ) . Since our goal is to learn the distribution , learning either W1 or W2 is sufficient . In short , we focus on the learnability of the underlying distribution and not the learnability of the NN parameters . Therefore , our proposed algorithm estimates WWT ∈ Rd×k and b ∈ Rd from the observed samples from Dp ( W , b ) . Next , we note that ( i , j ) -th entry ( WWT ) ( i , j ) of the symmetric matrix WWT is ‖W ( i , : ) ‖2‖W ( j , : ) ‖2 cos ( θij ) , where θij = θji is the angle between vectors W ( i , : ) and W ( j , : ) which are the i-th and j-th rows of matrix W , respectively . Also , note that the ` p norm of a vector is given by ‖x‖p = ( ∑ i |x ( i ) |p ) 1/p . Thus , we can construct the matrix WWT using the row norms { ‖W ( i , : ) ‖2 } di=1 and angles { θij } d i , j=1 . Our estimation algorithm consists of two steps : 1 ) estimation of row norms of W and bias vector b ; and 2 ) estimation of angles between the row vectors of W. To estimate ‖W ( i , : ) ‖2 ∈ R and b ( i ) ∈ R , for i ∈ [ d ] = { 1 , 2 , . . . , d } , suppose x ∼ D ( W , b ) . Its i-th coordinate can be written as x ( i ) = ReLU ( W ( i , : ) Tz + b ( i ) ) , ( 3 ) where W ( i , : ) Tz + b ( i ) ∼ N ( b ( i ) , ‖W ( i , : ) ‖22 ) . Thus , to compute b ( i ) and ‖W ( i , : ) ‖22 , it is enough to consider the i-th entries of the observed samples . Also , in ( 3 ) , the ReLU operator sets x ( i ) to zero if W ( i , : ) Tz + b ( i ) is negative . Consequently , we estimate b ( i ) and ‖W ( i , : ) ‖22 using the positive entries of the observed samples . Hence , estimation of b ( i ) and ‖W ( i , : ) ‖2 is equivalent to estimating the parameters of a one-dimensional normal distribution using positive samples , i.e. , the samples that belong to the set R > 0 : = { x ( i ) ∈ R : x ( i ) > 0 } . We use µ∗ = b ( i ) and σ∗2 = ‖W ( i , : ) ‖2 to denote the parameters of this one-dimensional normal distribution . The estimation of µ∗ and σ∗2 using corrupted samples is discussed next . 1Our algorithm and its analysis follow even if the outliers are not sampled from the same arbitrary distributions G. However , we need to assume that all the true samples come from the same distribution D. Algorithm 1 : Learning one-layer ReLU NN with outliers Input : Samples x1 , . . . , xn ∈ Rd 1 for i ∈ [ d ] do 2 X > 0 ← { xm , m ∈ [ n ] : xm ( i ) > 0 } 3 v̂ = ProjGD ( X > 0 ) 4 Σ̂i , i ← 1/v̂ ( 1 ) 5 b̂ ( i ) ← max { 0 , v̂ ( 2 ) /v̂ ( 1 ) } 6 for i < j ∈ [ d ] do 7 Compute θ̂ij using ( 11 ) 8 Σ̂ ( i , j ) ← √ Σ̂ ( i , i ) Σ̂ ( j , j ) cos ( θ̂ij ) 9 Σ̂ ( j , i ) ← Σ̂ ( i , j ) Output : Σ̂ ∈ Rd×d , b̂ ∈ Rd Algorithm 2 : ProjGD Input : Samples X > 0 , Parameters T , γt , and nb 1 Compute { xj ( i ) } nbj=1 from X > 0 using ( 7 ) 2 g̃x ← filter ( { xj ( i ) } nbj=1 ) 3 v0 ← [ 0 0 ] 4 for t = 1 , 2 , . . . , T do 5 µ← vt−1 ( 2 ) /vt−1 ( 1 ) 6 σ2 ← 1/vt−1 ( 1 ) 7 g̃z← [ − ( σ̆2 + µ̆2 ) /2 µ̆ ] T using ( 5 ) and ( 6 ) 8 g̃t ← g̃x + g̃z 9 ṽt ← vt−1 − γt−1g̃t−1 10 vt ← P ( ṽt ) using ( 10 ) Output : vT ∈ R2 We determine the parameters of the univariate normal distribution using maximum likelihood estimation ( Daskalakis et al . ( 2018 ) ) given by arg minv ` ( v ) where ` ( v ) is the expected negative log-likelihood with respect to x ( i ) and v = [ 1/σ2 ; µ/σ2 ] ∈ R2 . This optimization problem can be solved using a learning algorithm like GD or SGD . SGD utilizes only one sample for the gradient computation which introduces large variance due to the stochasticity and low reliability in the presence of arbitrary outliers . Thus , we use GD that utilizes all the samples for the gradient . To derive our algorithm based on GD , we compute the gradient ( Daskalakis et al . ( 2018 ) ) , ∇ ` ( v ) = Ex , z [ g ( i ) ] where g ( i ) = gx ( i ) + gz ( i ) = [ x2 ( i ) /2 −x ( i ) ] T + [ −z2 ( i ) /2 z ( i ) ] T . ( 4 ) Here , x ( i ) ∼ N ( µ∗ , σ∗2 ; R > 0 ) and z ( i ) ∼ N ( µ , σ2 ; R > 0 ) . Also , µ∗ and σ∗2 are the parameters of the true distribution , and µ and σ2 are functions of v. In ( 4 ) , Ez [ gz ( i ) ] admits a closed form expression ( Johnson et al . ( 1995 ) ) , g̃z = Ez [ gz ( i ) ] = [ − ( σ̆2 + µ̆2 ) /2 µ̆ ] T . Here , we define µ̆ = E [ z ( i ) |0 < z ( i ) < ∞ ] = µ+ φ ( −µ/σ ) 1− Φ ( −µ/σ ) σ , ( 5 ) σ̆2 = Var ( z ( i ) |0 < z ( i ) < ∞ ) = σ2 ( 1− µ σ φ ( −µ/σ ) 1− Φ ( −µ/σ ) − ( φ ( −µ/σ ) 1− Φ ( −µ/σ ) ) 2 ) , ( 6 ) where φ ( · ) and Φ ( · ) denote the probability density function and the cumulative distribution function of the standard normal distribution , respectively . We also need Ex [ gx ( i ) ] to compute Ex , z [ g ( i ) ] . Since the value of Ex [ gx ( i ) ] is not known , we use the sample mean to estimate Ex [ gx ( i ) ] . For this , all the positive samples are partitioned into nb batches . Let Sj be the j-th batch . We then compute the vector xj ( i ) ∈ R2 as follows : xj ( i ) = 1 |Sj | [ ∑ x ( i ) ∈Sj x ( i ) 2/2 − ∑ x ( i ) ∈Sj x ( i ) ] T . ( 7 ) We then combine the nb vectors { xj ( i ) } nbj=1 to estimate Ex , z [ g ( i ) ] . Recall that the observed samples are from the mixture distribution Dp ( W , b ) . Therefore , to compute the expectation with respect to the true distribution D ( W , b ) , a filter is applied on this set of nb vectors to mitigate the effect of the arbitrary outliers which is a common tool in the robust statistics literature and obtain Ex [ gx ( i ) ] . We consider two filters : median and trimmed mean . Median : The median based filter possesses the following robustness property . Typically , a median is the value separating the higher half from the lower half of a given set of points . If more than half of a given set of points are in [ −M , M ] for some M > 0 , then their median must be in [ −M , M ] . Thus , the median of { xj ( i ) } nbj=1 is the vector computed either from all the true samples or outliers whose magnitudes are comparable to the true samples . Trimmed Mean : The trimmed mean removes the vectors among the set of nb vectors with relatively large and small values and computes the estimate of Ex [ gx ( i ) ] as the mean of the remaining vectors . Here , we use the parameter β to indicate the number of vectors to be discarded . This technique prunes the vectors xj ( i ) , j ∈ [ nb ] that are computed from the batches with outliers having relatively high or low magnitudes . Therefore , we obtain the gradient estimate as g̃ ( i ) = g̃x ( i ) + g̃z ( i ) = filter ( { xj ( i ) } nb j=1 ) + [ ( σ̆2 + µ̆2 ) /2 µ̆ ] T , ( 8 ) where filter is either median or trimmed mean , and µ̆ and σ̆2 are given in ( 5 ) and ( 6 ) , respectively . Having computed the gradient estimate , we next present the proposed GD algorithm . In the t-th iteration , the algorithm performs the following three steps : The first step is Gradient Computation Step where the gradient g̃t ∈ R2 is computed from the observed positive samples using ( 8 ) . The second step is Update Step where the gradient computed in the previous step is used to perform the GD update : ṽt = vt−1 − γt−1g̃t−1 , ( 9 ) where γt−1 > 0 is the diminishing step size . The last step is Projection Step where the objective function l ( v ) is a strongly convex function of v , if v belongs to a bounded region . To control the strong-convexity of the objective function , we project the update vector v into the domain Dr = { v ∈ R2 : 1/r ≤ v ( 1 ) ≤ r , |v ( 2 ) | ≤ r } . Thus , the projection is P ( v ) = [ min { max { v ( 1 ) , 1/r } , r } min { max { v ( 2 ) , −r } , r } ] . ( 10 ) The GD algorithm is run for T iterations where T is the algorithm parameter . The overall algorithm is summarized in Algorithm 2 . This completes the first step of our algorithm that uses the GD algorithm to obtain the estimate of the bias vector and the row norms of the weight matrix . Finally , using the estimates b̂ and Σ̂ obtained using the GD algorithm , we estimate θ̂ij similar to Wu et al . ( 2019 ) using ( Williamson & Shmoys , 2011 , Lemma 6.7 ) . Specifically , we have θ̂ij = π − 2π ( 1 n n∑ m=1 1 ( xm ( i ) > b̂ ( i ) ) 1 ( xm ( j ) > b̂ ( j ) ) ) , ( 11 ) where 1 ( · ) denotes the indicator function and b̂ is the estimate of the bias vector computed in the previous step using the projected GD algorithm . The overall algorithm is given in Algorithm 1 where Σ̂ is the estimate of WWT . Here , Steps 1-5 estimate the row norms of W and b , and Steps 6-9 estimate the angles between any two rows of W. Our algorithm is similar to that in Wu et al . ( 2019 ) except for the filter which is also a standard technique in robust statistics . However , our algorithm is different from the algorithm in Wu et al . ( 2019 ) in the following aspects . We do not rescale the unknown parameters ( see Step 1 of Algorithm 2 in Wu et al . ( 2019 ) ) . We employ median or trimmed mean based filter to compute the full gradient and utilize the GD algorithm instead of SGD used in Wu et al . ( 2019 ) . We note that our main contribution is the analysis rather than the algorithm which is presented in the next section .

Summary: Given observations $x_1, \cdots, x_n$ assume that a fraction $p$ of the observations are drawn from  $$ x = ReLU(Wz + b), \text{ where } z \sim N(0, I_k) $$ and the rest are drawn from an arbitrary distribution $\mathcal{G}.$  The authors study the problem of estimating $W, b$ in this model.  They assume that $b$ has non-negative entries.  The authors separate the problem in two parts:  * estimate $\| W(i,:) \|, b_i$ from the $i$th coordinates of the observations * estimate the angles between the rows of $W$  They use a gradient descent algorithm for the first part and use a clever random projections based idea for the second part.  They derive error bounds of their method and demonstrate the performance of their method through simulations.