Robust Regression Revisited: Acceleration and Improved Estimation Rates

1 Introduction . Parameter estimation in generalized linear models ( GLMs ) , such as linear and logistic regression problems , is among the most fundamental and well-studied statistical optimization problems . It serves as the primary workhorse in statistical studies arising from a variety of disciplines , ranging from economics [ Smi12 ] , biology [ VGSM05 ] , and the social sciences [ Gor10 ] . Formally , given a link function γ : R2 → R and a dataset of covariates and labels { ( Xi , yi ) } i∈ [ n ] ⊂ Rd × R drawn from an underlying distribution DXy , the problem of statistical ( generalized linear ) regression asks to estimate θ ? : = argminθ∈Rd { E ( X , y ) ∼DXy [ γ ( 〈θ , X〉 , y ) ] } . ( 1 ) For example , when γ ( v , y ) = 12 ( v−y ) 2 , ( 1 ) corresponds to ( statistical ) linear regression . The problem ( 1 ) also has an interpretation as computing a maximum likelihood estimate for a parameterized distributional model for data generation , and indeed is only tractable under certain distributional 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . assumptions , since we only have access to samples from DXy rather than the underlying distribution itself ( see e.g . [ BP21 ] for tractability results in the linear regression setting ) . However , in many modern settings these strong distributional assumptions fail to hold . In practically relevant settings , regression is often performed on massive datasets , where the data come from a poorly-understood distribution and have not been thoroughly vetted or cleaned of outliers . This has prompted the study of robust regression , i.e . regression under weak distributional or corruption assumptions . In this work , we study ( 1 ) in the strong contamination model . In this model , we assume the data points we receive are independently drawn from DXy , but that an arbitrary -fraction of the samples are then adversarially contaminated or replaced . The strong contamination model has garnered significant recent interest in the algorithmic statistics and learning communities for several reasons . Firstly , it is a flexible model of corruption and can be used to study both truly adversarial data poisoning attacks ( where e.g . part of the dataset is sourced from malicious respondents ) , as well as model misspecification , where the generative DXy does not exactly satisfy our distributional assumptions , but is close in total variation to a distribution that does . Furthermore , a line of work building upon [ DKK+16 , LRV16 ] ( discussed in our survey of prior work in Section 1.2 ) has achieved remarkable positive results for mean estimation and related problems under strong contamination , with statistical guarantees scaling independently of the dimension d. This dimension-free error promise is important in modern high-dimensional settings . 1.1 Our results . We give multiple nearly-linear time algorithms1 for problem ( 1 ) under the strong contamination model , with improved statistical or runtime guarantees compared to the state-of-the-art . Prior algorithms for ( 1 ) under the strong contamination model in the literature typically followed one of two frameworks . The first , which we refer to as robust gradient descent , was pioneered by [ PSBR20 ] , and is based on reframing ( 1 ) as a problem where we have noisy gradient access to an unknown function we wish to optimize , coupled with the design of a noisy gradient oracle based on a robust mean estimation primitive . The second , which we refer to as Sever , originated in work of [ DKK+19 ] , and uses the guarantees of stationary point finders such as stochastic gradient descent to repeatedly perform outlier removal . Interestingly , we show that both approaches can be dramatically sped up , and give two complementary types of algorithms within these frameworks . Robust acceleration . We demonstrate that under the noisy gradient estimation framework for solving well-conditioned regression problems ( 1 ) , an accelerated rate of optimization can be achieved , answering an open question of [ PSBR20 ] . We give the following result for smooth statistical regression , where we assume the uncorrupted data is drawn from DXy with marginals DX and Dy ; throughout , Õ hides polylogarithmic factors in problem parameters and failure probabilities . Finally , we remove κ dependences in sample complexity statements , as they are subsumed by dependences due to assumed bounds on κ or κ2 . Theorem 1 ( informal , cf . Theorem 7 , supplement ) . Suppose γ : R2 → R is such that γy ( v ) : = γ ( v , y ) is convex and has ( absolute ) first and second derivatives at most 1 for all y in the support of Dy , and DX has second moment matrix Σ ? LI . For some µ ≥ 0 , let κ : = max ( 1 , Lµ ) and let θ ? reg : = argminθ∈Rd { E ( X , y ) ∼DXy { γ ( 〈θ , X〉 , y ) } + µ 2 ‖θ‖22 } be the solution to the true regularized statistical regression problem.2 There is an algorithm that given n : = Õ ( d ) -corrupted samples from DXy , for κ 2 at most an absolute constant , obtains θ with ∥∥θ − θ ? reg∥∥2 = O ( √κ µ ) with high probability in time Õ ( nd√κ ) . A canonical example of a link function γ satisfying assumptions of Theorem 1 is the logit function γ ( v , y ) = log ( 1 + exp ( −vy ) ) , when the labels y are ±1 . To contextualize Theorem 1 , [ PSBR20 ] 1Throughout , we reserve the description “ nearly-linear ” for runtimes scaling linearly in the dataset size nd , and polynomially in −1 and the condition number , up to a polylogarithmic overhead in problem parameters . 2To simplify bounds and avoid estimation error for non-strongly convex statistical regression problems scaling with the initial search radius ( which may be dimension-dependent ) , we focus on regularized problems . There is a substantial line of work on reductions between rates for strongly convex and convex smooth optimization in the non-robust setting , see e.g . [ ZH16 ] , and we defer an analogous exploration in the robust setting to future work . obtains a similar statistical guarantee in its setting , using Õ ( κ ) calls to a noisy gradient oracle , implemented via a subroutine based on robust mean estimation . Since then , [ CAT+20 ] showed that for the case of linear regression ( see Theorem 3 for the formal setup , as the linear regression link function is not Lipschitz ) , the framework was amenable to nearly-linear time mean estimation techniques of [ CDG19 ] , and gave an algorithm with runtime Õ ( ndκ −6 ) . Theorem 1 represents an improvement to these results on two fronts : we apply tools from [ DHL19 ] to remove the poly ( −1 ) runtime dependence for a general class of regression problems , and we achieve an iteration count of Õ ( √ κ ) , matching the accelerated runtime of [ Nes83 ] for ( non-robust ) smooth optimization . We remark that the application of tools inspired by [ DHL19 ] is fairly straightforward , and not a primary contribution of our work compared to the accelerated dependence on κ . We demonstrate the generality of our acceleration framework by applying it to optimizing the Moreau envelope for Lipschitz , but possibly non-smooth , link functions γ ; a canonical example is the hinge loss γ ( v , y ) = max ( 0 , 1 − vy ) with ±1 labels , used in training support vector machines . The Moreau envelope is an extremely well-studied smooth approximation , and is an everywhere additive approximation if the original function is Lipschitz ( see e.g . [ Sho97 ] ) . In the non-robust setting many state-of-the-art rates for Lipschitz optimization are attainable by accelerated optimization of an appropriate Moreau envelope [ TJNO20 ] . We show that even without explicit access to the Moreau envelope , we can approximately minimize it with our robust acceleration framework . Theorem 2 ( informal , cf . Theorem 8 , supplement ) . Suppose γ : R2 → R is such that γy ( v ) : = γ ( v , y ) is convex and has ( absolute ) first derivative at most 1 for all y in the support of Dy , and DX has bounded second moment matrix . For some µ , λ ≥ 0 , let κ = max ( 1 , 1λµ ) and let θ ? env : = argminθ∈Rd { F ? λ ( θ ) + µ 2 ‖θ‖22 } , where F ? λ ( θ ) : = inf θ′ { F ? ( θ′ ) + 1 2λ ‖θ − θ′‖22 } is the Moreau envelope of F ? ( θ ) : = E ( X , y ) ∼DXy { γ ( 〈θ , X〉 , y ) } . There is an algorithm that given n : = Õ ( d ) -corrupted samples from DXy , for κ 2 at most an absolute constant , obtains θ with ∥∥θ − θ ? reg∥∥2 = O ( √κ µ ) with high probability in time Õ ( nd√κ ) . To obtain this result , we give a nearly-linear time construction of a noisy gradient oracle for the Moreau envelope , which may be of independent interest ; we note similar gradient oracle constructions in different settings have been developed in the optimization literature ( see e.g . [ CJJS21 ] ) . Robust linear regression . Perhaps the most ubiquitous example of statistical regression , the specific problem of robust linear regression has received substantial attention in the literature ( cf . Section 1.2 ) . Amongst the algorithms developed for the variant of this problem that we study , the only nearlylinear time algorithm is the recent work of [ CAT+20 ] .3 For a robust linear regression problem with noise variance bounded by σ2 and covariate second moment matrix Σ ? : = EX∼DX [ XX > ] , [ DKK+19 , PSBR20 , CAT+20 ] attain distance to the true regression minimizer θ ? scaling as σκ √ in the Σ ? norm ( the “ Mahalanobis distance ” ) 4 under a bounded 4th moment assumption . We give one result ( Theorem 3 ) which improves the runtime of [ DKK+19 , PSBR20 , CAT+20 ] under the noisy gradient descent framework , and one result ( Theorem 4 ) which improves its estimation rate , under the Sever framework . We first demonstrate that applying our robust acceleration framework leads to a similar estimation guarantee as [ DKK+19 , PSBR20 , CAT+20 ] under the same assumptions , but with improved runtime . Theorem 3 ( informal , cf . Theorem 6 , supplement ) . Suppose DX is 2-to-4 hypercontractive with second moment matrix Σ ? = EX∼DX [ XX > ] satisfying µI Σ ? LI , and y ∼ Dy is generated as 〈θ ? , X〉 + δ , for δ ∼ Dδ with variance at most σ2 independent of X . Let κ : = Lµ . There is an algorithm that given n : = Õ ( d ) -corrupted samples fromDXy , for κ 2 at most an absolute constant , obtains θ with ‖θ − θ ? ‖Σ ? = O ( σκ √ ) with high probability in time Õ ( nd √ κ ) . 3Another algorithm was recently given by [ ] under different assumptions , most notably a known second moment matrix . 4We measure error in the Σ ? norm as it is scale invariant and the natural norm to measure the underlying ( quadratic ) statistical regression problem under ; some prior works gave ` 2 norm guarantees , which we convert . We give a formal definition of 2-to-4 hypercontractivity in Section 2 . We remark that attaining estimation rates for robust linear regression scaling polynomially in is known to be impossible under only bounded second moments ( [ BP21 ] ) ; the 4th moment bound we require is the minimal assumption known in the literature under which such robust estimation is possible.5 Theorem 3 matches the distribution assumptions and error of [ CAT+20 ] , while obtaining an accelerated runtime . Interestingly , under the 4th moment bound used in Theorem 3 , [ BP21 ] showed that the informationtheoretically optimal rate of estimation in the Σ ? norm is independent of κ , and presented a matching upper bound under an analogous , but more stringent , distributional assumption.6 However , thus far robust linear regression algorithms have broadly fallen under two categories . The first family ( e.g . [ KKM18 , ZJS20 , BP21 ] ) , based on the sum-of-squares paradigm for algorithm design , sacrifices practicality to obtain improved error rates by paying a large runtime and sample complexity overhead ( as well as requiring stronger distributional assumptions ) . The second ( e.g . [ DKK+19 , PSBR20 , CAT+20 ] ) , which opts for more practical approaches to algorithm design , has been bottlenecked at Mahalanobis distance O ( σκ √ ) and the requirement that κ2 = O ( 1 ) . We present a nearly-linear time method for robust linear regression overcoming this bottleneck for the first time amongst non-sum-of-squares algorithms , attaining improved statistical rates compared to Theorem 3 while only requiring κ = O ( 1 ) . Theorem 4 ( informal , cf . Theorem 5 , supplement ) . Suppose DX is 2-to-4 hypercontractive with second moment matrix Σ ? = EX∼DX [ XX > ] satisfying µI Σ ? LI , and y ∼ Dy is generated as 〈θ ? , X〉+ δ , for δ ∼ Dδ , a 2-to-4 hypercontractive distribution with variance at most σ2 independent of X . Let κ : = Lµ . There is an algorithm that given n : = Õ ( ( d2 + d 4 ) ) samples from DXy , for κ at most an absolute constant , uses Õ ( 1 ) calls to an empirical risk minimization algorithm 7 and Õ ( nd ) additional runtime , and obtains θ with ‖θ − θ ? ‖Σ ? = O ( σ √ κ ) with probability at least 910 . This second algorithm does require more resources than that of Theorem 3 : the sample complexity scales quadratically in d , and the runtime is never faster . Further , we make the slightly stronger assumption of hypercontractive noise for the uncorrupted samples . On the other hand , the improved dependence on the condition number in the error can be significant for distributions in practice , which may be far from isotropic . All told , Theorem 4 presents an intermediate tradeoff inheriting some statistical gains of the sum-of-squares approach ( albeit still depending on κ ) without sacrificing a nearly-linear runtime . Interestingly , we obtain Theorem 4 by reinterpreting an identifiability proof used in the algorithm of [ BP21 ] , and combining it with tools inspired by the Sever robust estimation framework . We note that our sample complexity dramatically improves that of [ DKK+17 ] ’ s original linear regression algorithm in the Sever framework in terms of the dependence on d ( but not ) , which requires Õ ( d 5 2 ) samples ( in addition to beating their weaker error guarantee ) . We elaborate on these points further in Section 4 ; we believe it is an interesting open problem to understand if the worse sample complexity of Theorem 4 is truly necessary for the improved statistical guarantees .

Summary: The paper develops fast algorithms for robust stochastic optimization in the presence of strong contamination model, where the objective function belongs to one of the following: linear regression, (regularized) generalized linear models, and (regularized) Moureau Envelope. The main contributions of the paper are as follows: 1. Building on the gradient descent framework of [PSBR20], the paper develops accelerated gradient descent method using a noisy gradient oracle. This allows them to get better dependence on the condition number in the all of the mentioned models (with similar statistical rate as existing works). 2. Inspired by the identifiability proof of [BP21], the paper gives a practical algorithm for linear regression that has better statistical rates than existing works (at the cost of additional sample complexity). This is the first work that achieves this rate without certifiable hypercontractivity.