LEARNING DISTRIBUTIONS GENERATED  BY SINGLE-LAYER RELU  NETWORKS  IN  THE PRESENCE  OF ARBITRARY OUTLIERS

d δ ) sam- ples are sufficient for our algorithm to estimate the NN parameters within an error of with probability 1− δ when the probability of a sample being uncorrupted is p and the problem dimension is d. Our theoretical and simulation results provide insights on how the estimation of the NN parameters depends on the probability of a sample being uncorrupted , the number of samples , and the problem dimension . 1 INTRODUCTION . A fundamental challenge in machine learning and statistics is to estimate a high dimensional distribution given a set of observed data samples from the distribution . One solution technique is the deep generative method that models the unknown distribution as the output distribution of a neural network ( NN ) when the input of the NN is drawn from a known distribution like standard Gaussian which is motivated from the case of image generation by generative adversarial networks ( GANs ) ( Goodfellow et al . ( 2014 ) ; Arjovsky et al . ( 2017 ) ; Radford et al . ( 2015 ) ; Kingma & Welling ( 2013 ) ; Van Oord et al . ( 2016 ) ) . Then , the unknown distribution is estimated by learning the NN parameters from the data samples . Several approaches like GANs ( Goodfellow et al . ( 2014 ) ; Arjovsky et al . ( 2017 ) ; Radford et al . ( 2015 ) ) , variational autoencoders ( Kingma & Welling ( 2013 ) ) , and autoregressive models ( Van Oord et al . ( 2016 ) ) have been proposed to train the NN . However , they do not provide theoretical guarantees for the parameter learning . So , we address the open problem of computing the sample complexity for the parameter estimation of a single-layer NN with rectified linear unit ( ReLU ) activation using a corrupted dataset in the unsupervised learning framework . Here , the corrupted dataset refers to the model wherein the output samples consists of a fraction of arbitrary outliers introduced by an adversary ( Byzantine ) . Motivated by the deep generative models , we consider the unsupervised learning framework , i.e. , we assume the knowledge of the output samples but , the corresponding inputs are unknown and drawn from the standard Gaussian distribution . We start with a brief review of the related literature . The NN parameter estimation has been considered under both supervised and unsupervised learning frameworks . The estimation rely on the stochastic gradient descent ( SGD ) -based algorithms ( Goel et al . ( 2018 ) ; Allen-Zhu et al . ( 2019 ) ; Chen et al . ( 2020 ) ; Oymak ( 2019 ) ; Mazumdar & Rawat ( 2018 ) ; Wu et al . ( 2019 ) ; Lei et al . ( 2020 ) ) ; or the gradient descent ( GD ) -based approach ( Cao & Gu ( 2019 ) ; Du et al . ( 2019 ) ) . A major shortcoming of these works is that they assume that the available samples are not corrupted by noise or outliers . Several works in the literature have addressed the problem of corrupted samples in the context of supervised learning . The existing works have considered the learning of the parameters of a NN with ReLU activation using both SGD-based ( Bakshi et al . ( 2019 ) ; Goel et al . ( 2019 ) ; Mukherjee & Muthukumar ( 2020 ) ) and GD-based algorithms ( Zhang et al . ( 2019 ) ; Frei et al . ( 2020 ) ; Vempala & Wilmes ( 2019 ) ) . However , to the best of our knowledge , the estimation of a NN in the presence of noise or outliers in the unsupervised learning framework has not been studied in the literature . Our setting of unsupervised parameter estimation is also related to the area of robust statistics as described below . We recall that the goal of our paper is to estimate the parameters of a NN using corrupted output samples assuming that the input distribution is Gaussian . Mathematically , this problem is equivalent to estimating the parameters of a truncated Gaussian distribution ( see Section 2 for details ) . The area of robust statistics also deals with the estimation of high dimensional distributions like Gaussian , Gaussian product , and Gaussian mixture distributions where a constant fraction of the samples were corrupted by noise ( Huber ( 2004 ) ; Hampel et al . ( 2011 ) ; Lai et al . ( 2016 ) ; Diakonikolas et al . ( 2019 ) ; Diakonikolas & Kane ( 2019 ) ; Kane ( 2021 ) ) . However , in this paper , we consider parameter estimation from truncated Gaussian samples in the adversarial setting which has not been studied in the literature . We present our major contributions as the following . • Parameter Estimation ( Sections 2 and 3 ) : We estimate the parameters ( weight matrix and bias vector ) of a single-layer NN with ReLU in the unsupervised learning framework where each output sample can potentially be an arbitrary outlier with a fixed probability . We propose an estimation algorithm that has two steps : ( i ) we estimate the row norms of the weight matrix and the bias vector using GD along with either the median or trimmed mean-based filters to mitigate the effect of arbitrary outliers , and ( ii ) we estimate the angles between any two row vectors of the weight matrix using a simple geometric result ( Williamson & Shmoys , 2011 , Lemma 6.7 ) . • Theoretical guarantees ( Section 4 ) : We show that the proposed algorithm requires Ω ( 1 p4 log d δ ) samples to estimate the network parameters within an error of with probability 1− δ when the probability of a sample being uncorrupted is p. Here , > 0 represents the estimation error . • Empirical validation ( Section 5 ) : We evaluate the performance of our algorithm empirically in terms of its relation to the probability of a sample being uncorrupted , number of samples , and dimension ( see Figs . 1 and 2 ) . We see that our proposed filtering schemes ensure robustness to the arbitrary outliers . Also , the performance of our algorithm improves with the increase in the probability of a sample being uncorrupted and the number of samples , as expected . Further , we note that the estimation error increases slowly as the dimension grows for a fixed number of samples in the presence of arbitrary outliers . These observations from the empirical results are consistent with our theoretical results . To summarize , in this paper , we estimate the parameters of a single-layer NN with ReLU activation in the presence of arbitrary outliers . The results obtained in this paper provide insights to generalize to a multi-layer NN with various activation functions and obtain generalization bounds . 2 LEARNING A NEURAL NETWORK MODEL USING CORRUPTED OUTPUT SAMPLES . We consider a single-layer ReLU NN . Let the weight matrix of the NN be denoted by W ∈ Rd×k and the bias vector by b ∈ Rd . The input to the NN is denoted by the latent variable z ∈ Rk . We assume that the variable z is drawn from the standard Gaussian distribution . The Gaussianity assumption is for mathematical tractability , and it is motivated by a popular generative approach to estimate a high-dimensional distribution from observed samples in the case of image generation by GANs Goodfellow et al . ( 2014 ) ; Radford et al . ( 2015 ) ; Arjovsky et al . ( 2017 ) . Thus , the output of the network is the random vector x ∈ Rd given by x = ReLU ( Wz + b ) , where z ∼ N ( 0 , Ik ) , ( 1 ) where N ( 0 , Ik ) denotes the Gaussian distribution with mean 0 and covariance matrix Ik which is the identity matrix of size k×k . Let the distribution of the random vector x be denoted byD ( W , b ) . Then , our goal is to estimate the unknown parameters W and b of the distribution D ( W , b ) using the knowledge of n output samples x1 , x2 , . . . , xn . Here , we assume that a fraction of the samples are arbitrary outliers . The distribution of the observed samples are modeled using the Huber ’ s pcontamination model 1 defined as the following . Definition 2.1 ( Huber ’ s p-Contamination Model Huber ( 1964 ) ) . The observed samples are said to be following Huber ’ s contamination model if any given sample is drawn from the true distribution D and an arbitrary distribution G with probability p , and 1 − p , respectively . In other words , the observed samples are drawn from the mixture distribution , Dp ( W , b ) given by Dp ( W , b ) = pD ( W , b ) + ( 1− p ) G. ( 2 ) Note that if b has large negative values , then most of the output samples are zeros due to the ReLU operation . Further , if W is a zero matrix , then any negative coordinate of b can not be identified as it is reset to zero after the ReLU operation . Further , in Wu et al . ( 2019 ) , the authors have shown that if b ∈ Rd then exponentially large number of samples are required to estimate the bias . This holds true in our case as in the presence of arbitrary outliers if b has large negative values , then we would require larger number of samples to estimate the bias vector b compared to that in Wu et al . ( 2019 ) . Hence , we also assume b to be non-negative . Our estimation problem is challenging due to two reasons : 1 ) ReLU operation is not invertible , and therefore , estimation of W and b utilizing maximum likelihood of the probability density function of the random vector x is intractable ; 2 ) the distribution of the corrupted samples is unknown . We tackle these issues using a new formulation combining the GD algorithm and a filtering technique . Note that the GD algorithm is similar to that in Wu et al . ( 2019 ) wherein the authors use the SGD algorithm . The proposed algorithm is presented in the next section . 3 ESTIMATION OF PARAMETERS OF SINGLE-LAYER NN . To design our estimation algorithm , we first note that the weight matrix W may not be identifiable from the distribution D ( W , b ) . This is because of the fact that for any matrices W1 and W2 if W1W T 1 = W2W T 2 then there exists a unitary matrix Q such that W2 = W1Q . As z ∼ N ( 0 , Ik ) , we have Qz ∼ N ( 0 , Ik ) . Hence , for any vector b , we have D ( W1 , b ) = D ( W2 , b ) ( Wu et al . ( 2019 ) ) . Since our goal is to learn the distribution , learning either W1 or W2 is sufficient . In short , we focus on the learnability of the underlying distribution and not the learnability of the NN parameters . Therefore , our proposed algorithm estimates WWT ∈ Rd×k and b ∈ Rd from the observed samples from Dp ( W , b ) . Next , we note that ( i , j ) -th entry ( WWT ) ( i , j ) of the symmetric matrix WWT is ‖W ( i , : ) ‖2‖W ( j , : ) ‖2 cos ( θij ) , where θij = θji is the angle between vectors W ( i , : ) and W ( j , : ) which are the i-th and j-th rows of matrix W , respectively . Also , note that the ` p norm of a vector is given by ‖x‖p = ( ∑ i |x ( i ) |p ) 1/p . Thus , we can construct the matrix WWT using the row norms { ‖W ( i , : ) ‖2 } di=1 and angles { θij } d i , j=1 . Our estimation algorithm consists of two steps : 1 ) estimation of row norms of W and bias vector b ; and 2 ) estimation of angles between the row vectors of W. To estimate ‖W ( i , : ) ‖2 ∈ R and b ( i ) ∈ R , for i ∈ [ d ] = { 1 , 2 , . . . , d } , suppose x ∼ D ( W , b ) . Its i-th coordinate can be written as x ( i ) = ReLU ( W ( i , : ) Tz + b ( i ) ) , ( 3 ) where W ( i , : ) Tz + b ( i ) ∼ N ( b ( i ) , ‖W ( i , : ) ‖22 ) . Thus , to compute b ( i ) and ‖W ( i , : ) ‖22 , it is enough to consider the i-th entries of the observed samples . Also , in ( 3 ) , the ReLU operator sets x ( i ) to zero if W ( i , : ) Tz + b ( i ) is negative . Consequently , we estimate b ( i ) and ‖W ( i , : ) ‖22 using the positive entries of the observed samples . Hence , estimation of b ( i ) and ‖W ( i , : ) ‖2 is equivalent to estimating the parameters of a one-dimensional normal distribution using positive samples , i.e. , the samples that belong to the set R > 0 : = { x ( i ) ∈ R : x ( i ) > 0 } . We use µ∗ = b ( i ) and σ∗2 = ‖W ( i , : ) ‖2 to denote the parameters of this one-dimensional normal distribution . The estimation of µ∗ and σ∗2 using corrupted samples is discussed next . 1Our algorithm and its analysis follow even if the outliers are not sampled from the same arbitrary distributions G. However , we need to assume that all the true samples come from the same distribution D. Algorithm 1 : Learning one-layer ReLU NN with outliers Input : Samples x1 , . . . , xn ∈ Rd 1 for i ∈ [ d ] do 2 X > 0 ← { xm , m ∈ [ n ] : xm ( i ) > 0 } 3 v̂ = ProjGD ( X > 0 ) 4 Σ̂i , i ← 1/v̂ ( 1 ) 5 b̂ ( i ) ← max { 0 , v̂ ( 2 ) /v̂ ( 1 ) } 6 for i < j ∈ [ d ] do 7 Compute θ̂ij using ( 11 ) 8 Σ̂ ( i , j ) ← √ Σ̂ ( i , i ) Σ̂ ( j , j ) cos ( θ̂ij ) 9 Σ̂ ( j , i ) ← Σ̂ ( i , j ) Output : Σ̂ ∈ Rd×d , b̂ ∈ Rd Algorithm 2 : ProjGD Input : Samples X > 0 , Parameters T , γt , and nb 1 Compute { xj ( i ) } nbj=1 from X > 0 using ( 7 ) 2 g̃x ← filter ( { xj ( i ) } nbj=1 ) 3 v0 ← [ 0 0 ] 4 for t = 1 , 2 , . . . , T do 5 µ← vt−1 ( 2 ) /vt−1 ( 1 ) 6 σ2 ← 1/vt−1 ( 1 ) 7 g̃z← [ − ( σ̆2 + µ̆2 ) /2 µ̆ ] T using ( 5 ) and ( 6 ) 8 g̃t ← g̃x + g̃z 9 ṽt ← vt−1 − γt−1g̃t−1 10 vt ← P ( ṽt ) using ( 10 ) Output : vT ∈ R2 We determine the parameters of the univariate normal distribution using maximum likelihood estimation ( Daskalakis et al . ( 2018 ) ) given by arg minv ` ( v ) where ` ( v ) is the expected negative log-likelihood with respect to x ( i ) and v = [ 1/σ2 ; µ/σ2 ] ∈ R2 . This optimization problem can be solved using a learning algorithm like GD or SGD . SGD utilizes only one sample for the gradient computation which introduces large variance due to the stochasticity and low reliability in the presence of arbitrary outliers . Thus , we use GD that utilizes all the samples for the gradient . To derive our algorithm based on GD , we compute the gradient ( Daskalakis et al . ( 2018 ) ) , ∇ ` ( v ) = Ex , z [ g ( i ) ] where g ( i ) = gx ( i ) + gz ( i ) = [ x2 ( i ) /2 −x ( i ) ] T + [ −z2 ( i ) /2 z ( i ) ] T . ( 4 ) Here , x ( i ) ∼ N ( µ∗ , σ∗2 ; R > 0 ) and z ( i ) ∼ N ( µ , σ2 ; R > 0 ) . Also , µ∗ and σ∗2 are the parameters of the true distribution , and µ and σ2 are functions of v. In ( 4 ) , Ez [ gz ( i ) ] admits a closed form expression ( Johnson et al . ( 1995 ) ) , g̃z = Ez [ gz ( i ) ] = [ − ( σ̆2 + µ̆2 ) /2 µ̆ ] T . Here , we define µ̆ = E [ z ( i ) |0 < z ( i ) < ∞ ] = µ+ φ ( −µ/σ ) 1− Φ ( −µ/σ ) σ , ( 5 ) σ̆2 = Var ( z ( i ) |0 < z ( i ) < ∞ ) = σ2 ( 1− µ σ φ ( −µ/σ ) 1− Φ ( −µ/σ ) − ( φ ( −µ/σ ) 1− Φ ( −µ/σ ) ) 2 ) , ( 6 ) where φ ( · ) and Φ ( · ) denote the probability density function and the cumulative distribution function of the standard normal distribution , respectively . We also need Ex [ gx ( i ) ] to compute Ex , z [ g ( i ) ] . Since the value of Ex [ gx ( i ) ] is not known , we use the sample mean to estimate Ex [ gx ( i ) ] . For this , all the positive samples are partitioned into nb batches . Let Sj be the j-th batch . We then compute the vector xj ( i ) ∈ R2 as follows : xj ( i ) = 1 |Sj | [ ∑ x ( i ) ∈Sj x ( i ) 2/2 − ∑ x ( i ) ∈Sj x ( i ) ] T . ( 7 ) We then combine the nb vectors { xj ( i ) } nbj=1 to estimate Ex , z [ g ( i ) ] . Recall that the observed samples are from the mixture distribution Dp ( W , b ) . Therefore , to compute the expectation with respect to the true distribution D ( W , b ) , a filter is applied on this set of nb vectors to mitigate the effect of the arbitrary outliers which is a common tool in the robust statistics literature and obtain Ex [ gx ( i ) ] . We consider two filters : median and trimmed mean . Median : The median based filter possesses the following robustness property . Typically , a median is the value separating the higher half from the lower half of a given set of points . If more than half of a given set of points are in [ −M , M ] for some M > 0 , then their median must be in [ −M , M ] . Thus , the median of { xj ( i ) } nbj=1 is the vector computed either from all the true samples or outliers whose magnitudes are comparable to the true samples . Trimmed Mean : The trimmed mean removes the vectors among the set of nb vectors with relatively large and small values and computes the estimate of Ex [ gx ( i ) ] as the mean of the remaining vectors . Here , we use the parameter β to indicate the number of vectors to be discarded . This technique prunes the vectors xj ( i ) , j ∈ [ nb ] that are computed from the batches with outliers having relatively high or low magnitudes . Therefore , we obtain the gradient estimate as g̃ ( i ) = g̃x ( i ) + g̃z ( i ) = filter ( { xj ( i ) } nb j=1 ) + [ ( σ̆2 + µ̆2 ) /2 µ̆ ] T , ( 8 ) where filter is either median or trimmed mean , and µ̆ and σ̆2 are given in ( 5 ) and ( 6 ) , respectively . Having computed the gradient estimate , we next present the proposed GD algorithm . In the t-th iteration , the algorithm performs the following three steps : The first step is Gradient Computation Step where the gradient g̃t ∈ R2 is computed from the observed positive samples using ( 8 ) . The second step is Update Step where the gradient computed in the previous step is used to perform the GD update : ṽt = vt−1 − γt−1g̃t−1 , ( 9 ) where γt−1 > 0 is the diminishing step size . The last step is Projection Step where the objective function l ( v ) is a strongly convex function of v , if v belongs to a bounded region . To control the strong-convexity of the objective function , we project the update vector v into the domain Dr = { v ∈ R2 : 1/r ≤ v ( 1 ) ≤ r , |v ( 2 ) | ≤ r } . Thus , the projection is P ( v ) = [ min { max { v ( 1 ) , 1/r } , r } min { max { v ( 2 ) , −r } , r } ] . ( 10 ) The GD algorithm is run for T iterations where T is the algorithm parameter . The overall algorithm is summarized in Algorithm 2 . This completes the first step of our algorithm that uses the GD algorithm to obtain the estimate of the bias vector and the row norms of the weight matrix . Finally , using the estimates b̂ and Σ̂ obtained using the GD algorithm , we estimate θ̂ij similar to Wu et al . ( 2019 ) using ( Williamson & Shmoys , 2011 , Lemma 6.7 ) . Specifically , we have θ̂ij = π − 2π ( 1 n n∑ m=1 1 ( xm ( i ) > b̂ ( i ) ) 1 ( xm ( j ) > b̂ ( j ) ) ) , ( 11 ) where 1 ( · ) denotes the indicator function and b̂ is the estimate of the bias vector computed in the previous step using the projected GD algorithm . The overall algorithm is given in Algorithm 1 where Σ̂ is the estimate of WWT . Here , Steps 1-5 estimate the row norms of W and b , and Steps 6-9 estimate the angles between any two rows of W. Our algorithm is similar to that in Wu et al . ( 2019 ) except for the filter which is also a standard technique in robust statistics . However , our algorithm is different from the algorithm in Wu et al . ( 2019 ) in the following aspects . We do not rescale the unknown parameters ( see Step 1 of Algorithm 2 in Wu et al . ( 2019 ) ) . We employ median or trimmed mean based filter to compute the full gradient and utilize the GD algorithm instead of SGD used in Wu et al . ( 2019 ) . We note that our main contribution is the analysis rather than the algorithm which is presented in the next section .

Summary: The authors study the problem of learning the parameters of a generative model defined by a single-layer relu network in the presence of outliers from the Huber contamination model.  They derive a gradient-based algorithm for estimating the norms and angles between rows of the generating matrix, which suffices for estimating the matrix WW^T (which is the only estimable parameters in this setting).  Their analysis is largely the same as that of Wu et al. '19, with additional care needed to take care of outliers.  This is done via trimmed mean or median estimates.