Dynamic Sasvi: Strong Safe Screening for Norm-Regularized Least Squares

1 Introduction . Sparse models such as Lasso [ 23 ] and group Lasso [ 26 ] have been widely studied in the areas of statistics and machine learning and used for various applications such as compressed sensing [ 6 ] and biomarker discovery [ 4 ] . Although sparse models can be formulated as a simple convex optimization problem , the computational cost can be large if the numbers of samples and dimensions are extremely large . To address this problem , a technique called safe screening has been introduced [ 10 ] for Lasso problems . Specifically , it eliminates variables that are guaranteed to be zero in the Lasso solution before solving the original Lasso optimization problem . Many safe screening methods have been proposed for various problems [ 10 , 19 , 24 , 14 , 25 ] . These are called sequential screening rules because they require the solution to a more strongly regularized problem . A recent technique , called dynamic screening , has been proposed to eliminate variables through an estimated solution in an iterative solver [ 3 ] . In particular , Gap Safe [ 7 , 16 ] , a dynamic screening framework , is widely used owing to its generality and efficiency [ 17 , 21 , 1 , 20 , 18 ] . In this paper , we propose a dynamic safe screening algorithm that is stronger than Gap Safe for the Lasso-Like problem , which includes norm-regularized least squares . To this end , we first propose a general screening framework based on the Fenchel–Rockafellar duality and then derive Dynamic Sasvi , a strong safe screening rule for Lasso-like problems . Our framework can be regarded as a generalization of the Gap Safe framework , and thus , we can derive Gap Safe simply using our results . Moreover , owing to this generalization , we can use the strong problem adaptive inequality . The derived screening rule for Lasso-like problems can be seen as a dynamic variant of safe screening with variational inequalities ( Sasvi ) [ 14 ] , which is a sequential screening rule for Lasso . Therefore , we refer to this as Dynamic Sasvi . Unlike the original Sasvi , dynamic Sasvi does not require an exact solution to the problem with another hyper-parameter and , hence , operates safely in practice . Moreover , we propose the use of dynamic enhanced dual polytope projection ( EDPP ) [ 24 ] , which is a relaxation of dynamic Sasvi due to the introduction of a minimum radius sphere . We show , both 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . theoretically and experimentally , that the screening power and computational costs of Dynamic Sasvi and Dynamic EDPP compare favorably with those of other state-of-the-art Gap Safe methods . Contribution : The contributions of our study are summarized as follows . • We propose a flexible screening framework based on the Fenchel–Rockafellar duality , which is a generalization of the Gap Safe framework [ 17 ] . • We propose two novel dynamic screening rules for norm-regularized least squares : a dynamic variant of Sasvi [ 14 ] and a dynamic variant of EDPP . • We show that Dynamic Sasvi always eliminates more features and increases the speed of the solver compared with Gap Safe [ 7 , 17 ] . 2 Preliminary . In this section , we first formulate the problem and introduce the key techniques used in this study . 2.1 Notation . Given h : Rm → [ −∞ , ∞ ] , the domain of h is defined by dom ( h ) : = { z ∈ Rm | |h ( z ) | < ∞ } and h ? : Rm → [ −∞ , ∞ ] , which is the Fenchel conjugate of h , is defined as h ? ( v ) : = supz∈Rm v > z − h ( z ) . If h is proper , the Fenchel–Young inequality , h ( z ) + h ? ( v ) ≥ v > z , ( 1 ) can be proven directly based on definition of the Fenchel conjugate . The subdifferential of a proper function , h : Rm → ( −∞ , ∞ ] , at z is given as ∂h ( z ) : = { v ∈ Rm | ∀w ∈ Rm v > ( w − z ) + h ( z ) ≤ h ( w ) } . The next proposition is important for deriving safe-screening algorithms . Proposition 1 Assume that h : Rm → ( −∞ , ∞ ] is a proper lower semi-continuous convex function and z , v ∈ Rm . We then have v ∈ ∂h ( z ) ⇐⇒ h ( z ) + h ? ( v ) = v > z ⇐⇒ z ∈ ∂h ? ( v ) . See Section 16 of [ 2 ] for the proof . For convex set C ⊂ Rm , the relative interior of C is defined as relint ( C ) : = { v ∈ C | ∀w ∈ C ∃ > 0 s.t . v + ( v − w ) ∈ C } . 2.2 Problem Formulation . In this study , we consider an optimization problem , formulated as minimize β∈Rd f ( Xβ ) + g ( β ) , ( 2 ) where β ∈ Rd is the optimization variable , X ∈ Rn×d is a constant matrix , and f : Rn → ( −∞ , ∞ ] and g : Rd → ( −∞ , ∞ ] are proper lower semi-continuous convex functions . We assume ∃β ∈ relint ( dom ( g ) ) s.t.Xβ ∈ relint ( dom ( f ) ) and the existence of the optimal point , i.e. , ∃β̂ ∈ dom ( P ) s.t . P ( β̂ ) = inf β∈Rd P ( β ) , where P : Rd → R is defined as P ( β ) = f ( Xβ ) + g ( β ) . Moreover , we focus on the cases in which g induces sparsity . Although all theorems in this paper hold , we can not eliminate any variables without sparsity . This is a popular class of optimization problem , with the most popular example being Lasso [ 23 ] : minimize β∈Rd 1 2 ‖y −Xβ‖22 + λ‖β‖1 . Many extensions of Lasso , including Group-Lasso [ 26 ] , elastic net [ 27 ] , and sparse logistic regression [ 15 ] , are in this class . The dual problems of a support vector machine [ 5 ] and a support vector regression [ 22 ] are also in this class . 2.3 Dual Problem . In the derivation of a safe screening rule for the optimization problem , Eq . ( 2 ) , the Fenchel– Rockafellar dual formulation , plays an important role . Theorem 2 ( Fenchel–Rockafellar Duality ) If all assumptions for the optimization problem ( 2 ) are satisfied , we have the following : min β∈Rd f ( Xβ ) + g ( β ) = max θ∈Rn −f ? ( −θ ) − g ? ( X > θ ) . ( 3 ) The proof of Theorem 2 is provided in the appendix . We denote −f ? ( −θ ) − g ? ( X > θ ) by D ( θ ) . For primal/dual solutions , there are many conditions that are equivalent to optimality . Herein , we provide a list of such conditions for convenience . Proposition 3 ( Optimal Condition ) If all assumptions for the optimization problem ( 2 ) are satisfied , the following are equivalent . ( a ) β̂ ∈ argmin β∈Rd P ( β ) ∧ θ̂ ∈ argmax θ∈Rn D ( θ ) ( b ) P ( β̂ ) = D ( θ̂ ) ( c ) f ( Xβ̂ ) + f ? ( −θ̂ ) = −θ̂ > Xβ̂ = −g ( β̂ ) − g ? ( X > θ̂ ) ( d ) −θ̂ ∈ ∂f ( Xβ̂ ) ∧X > θ̂ ∈ ∂g ( β̂ ) ( e ) Xβ̂ ∈ ∂f ? ( −θ̂ ) ∧ β̂ ∈ ∂g ? ( X > θ̂ ) ( Proof ) ( a ) ⇐⇒ ( b ) is directly derived from the strong duality . ( b ) ⇐⇒ ( c ) is derived from the Fenchel–Young inequality ( 1 ) . ( c ) ⇐⇒ ( d ) ⇐⇒ ( e ) are derived from Proposition 1 . 2.4 Relationship of Dual Safe Region and Screening . For the optimization problem ( 2 ) , we can eliminate some features by constructing a simple region that contains θ̂ . Assume that the dual optimal point , θ̂ , is within regionR . According to Proposition 3 , we have β̂ ∈ ∂g ? ( X > θ̂ ) ⊂ ⋃ θ∈R ∂g ? ( X > θ ) . Hence , if ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β | βi = 0 } holds , we obtain β̂i = 0 . A simple example is the following corollary . Example 4 Consider an optimization problem , i.e. , Eq . ( 2 ) with g ( β ) = ‖β‖1 . Assume that θ̂ ∈ R. Then , we have max θ∈R |x > i θ| < 1 =⇒ β̂i = 0 . ( Proof ) Based on the definition of g , we have ∂g ? ( X > θ ) ⊂ { β | βi = 0 } ⇐⇒ |x > i θ| < 1 . When maxθ∈R |x > i θ| < 1 , we have β̂ ∈ ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β | βi = 0 } . To eliminate features safely , we should first construct a simple region R ⊂ Rn which contains θ̂ and then determine whether ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β ∈ Rd | βi = 0 } is guaranteed . In this study , we provide a novel general framework for constructing a simple safe region , R ⊂ Rn . By combining it with existing methods to calculate an upper bound of ⋃ θ∈R ∂g ? ( X > θ ) ( cf . [ 17 ] for the un-overlapping group L1 norm and sparse group L1 norm , [ 1 ] for the ordered weighted L1 norm ) , we can formulate strong safe screening rules . 3 General Framework for Constructing Safe Region . Herein , we propose a general framework for constructing a dual region that contains the solution to the optimization problem expressed by Eq . ( 3 ) . Our framework consists of a general lower bound and a problem-adaptive upper bound of the optimal value . Hence , we can derive a narrower region than the framework with a general upper bound under certain situations . The general lower-bound is derived from the optimal condition and the L-strong convexity . Assume that f ? is L-strongly convex ( L ≥ 0 ) . Then , as D is L-strongly concave and θ̂ is the optimal point , we have D ( θ̃ ) ≤ D ( θ̂ ) − L2 ‖θ̂ − θ̃‖ 2 2 for ∀θ̃ ∈ Rn . Thus , we have θ̂ ∈ { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ D ( θ ) } . Because this region is too complicated for screening , we use a simple upper bound of D ( θ ) to construct a simple safe region . Theorem 5 Consider the optimization problem expressed by Eq . ( 3 ) and assume that f ? is L-strongly convex ( L ≥ 0 ) . Let θ̂ be the solution to Eq . ( 3 ) . Assume that D ( θ ) is upper-bounded by u ( θ ) , i.e. , ∀θ ∈ Rn D ( θ ) ≤ u ( θ ) . Then , for ∀θ̃ ∈ Rn , we have θ̂ ∈ R ( θ̃ , u ) = { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ u ( θ ) } . The complexity ofR ( θ̃ , u ) depends on the complexity of u . For example , if u is linear , thenR ( θ̃ , u ) is a sphere . We can construct a narrow , simple , and safe region with a tight simple upper-bound u . In fact , the Gap Safe Sphere region [ 7 , 17 ] can be derived easily from this theorem and weak duality . Corollary 6 ( Gap Safe Sphere ) Consider the optimization problem described by Eq . ( 3 ) and assume that f ? is L-strongly convex ( L ≥ 0 ) . Let θ̂ be the solution to Eq . ( 3 ) . For ∀β̃ ∈ Rd and ∀θ̃ ∈ Rn , we have θ̂ ∈ { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ P ( β̃ ) } . ( Proof ) Based on a weak duality , we have ∀θ D ( θ ) ≤ P ( β̃ ) . Using this constant function as an upper bound in Theorem 5 , we can derive the corollary directly . Hence , our framework can be seen as a generalization of Gap Safe . Owing to this generalization , we can use a stronger problem-adaptive upper bound rather than a weak duality . In the next section , we derive specific regions for the Lasso-Like problem . Some regions for other problems are presented in the appendix .

Summary: The paper proposes a screening test for sparse models, i.e. a way to remove features from the problem once it is sure that they are not active at optimum. The rule is a generalization of the popular Gap Safe screening. As in GSR, a dual safe region is constructed using strong convexity of the dual objective (obtained from Lipschitz smoothness of the datafit in the primal)