Dynamic Sasvi: Strong Safe Screening for Norm-Regularized Least Squares

1 Introduction . Sparse models such as Lasso [ 23 ] and group Lasso [ 26 ] have been widely studied in the areas of statistics and machine learning and used for various applications such as compressed sensing [ 6 ] and biomarker discovery [ 4 ] . Although sparse models can be formulated as a simple convex optimization problem , the computational cost can be large if the numbers of samples and dimensions are extremely large . To address this problem , a technique called safe screening has been introduced [ 10 ] for Lasso problems . Specifically , it eliminates variables that are guaranteed to be zero in the Lasso solution before solving the original Lasso optimization problem . Many safe screening methods have been proposed for various problems [ 10 , 19 , 24 , 14 , 25 ] . These are called sequential screening rules because they require the solution to a more strongly regularized problem . A recent technique , called dynamic screening , has been proposed to eliminate variables through an estimated solution in an iterative solver [ 3 ] . In particular , Gap Safe [ 7 , 16 ] , a dynamic screening framework , is widely used owing to its generality and efficiency [ 17 , 21 , 1 , 20 , 18 ] . In this paper , we propose a dynamic safe screening algorithm that is stronger than Gap Safe for the Lasso-Like problem , which includes norm-regularized least squares . To this end , we first propose a general screening framework based on the Fenchel–Rockafellar duality and then derive Dynamic Sasvi , a strong safe screening rule for Lasso-like problems . Our framework can be regarded as a generalization of the Gap Safe framework , and thus , we can derive Gap Safe simply using our results . Moreover , owing to this generalization , we can use the strong problem adaptive inequality . The derived screening rule for Lasso-like problems can be seen as a dynamic variant of safe screening with variational inequalities ( Sasvi ) [ 14 ] , which is a sequential screening rule for Lasso . Therefore , we refer to this as Dynamic Sasvi . Unlike the original Sasvi , dynamic Sasvi does not require an exact solution to the problem with another hyper-parameter and , hence , operates safely in practice . Moreover , we propose the use of dynamic enhanced dual polytope projection ( EDPP ) [ 24 ] , which is a relaxation of dynamic Sasvi due to the introduction of a minimum radius sphere . We show , both 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . theoretically and experimentally , that the screening power and computational costs of Dynamic Sasvi and Dynamic EDPP compare favorably with those of other state-of-the-art Gap Safe methods . Contribution : The contributions of our study are summarized as follows . • We propose a flexible screening framework based on the Fenchel–Rockafellar duality , which is a generalization of the Gap Safe framework [ 17 ] . • We propose two novel dynamic screening rules for norm-regularized least squares : a dynamic variant of Sasvi [ 14 ] and a dynamic variant of EDPP . • We show that Dynamic Sasvi always eliminates more features and increases the speed of the solver compared with Gap Safe [ 7 , 17 ] . 2 Preliminary . In this section , we first formulate the problem and introduce the key techniques used in this study . 2.1 Notation . Given h : Rm → [ −∞ , ∞ ] , the domain of h is defined by dom ( h ) : = { z ∈ Rm | |h ( z ) | < ∞ } and h ? : Rm → [ −∞ , ∞ ] , which is the Fenchel conjugate of h , is defined as h ? ( v ) : = supz∈Rm v > z − h ( z ) . If h is proper , the Fenchel–Young inequality , h ( z ) + h ? ( v ) ≥ v > z , ( 1 ) can be proven directly based on definition of the Fenchel conjugate . The subdifferential of a proper function , h : Rm → ( −∞ , ∞ ] , at z is given as ∂h ( z ) : = { v ∈ Rm | ∀w ∈ Rm v > ( w − z ) + h ( z ) ≤ h ( w ) } . The next proposition is important for deriving safe-screening algorithms . Proposition 1 Assume that h : Rm → ( −∞ , ∞ ] is a proper lower semi-continuous convex function and z , v ∈ Rm . We then have v ∈ ∂h ( z ) ⇐⇒ h ( z ) + h ? ( v ) = v > z ⇐⇒ z ∈ ∂h ? ( v ) . See Section 16 of [ 2 ] for the proof . For convex set C ⊂ Rm , the relative interior of C is defined as relint ( C ) : = { v ∈ C | ∀w ∈ C ∃ > 0 s.t . v + ( v − w ) ∈ C } . 2.2 Problem Formulation . In this study , we consider an optimization problem , formulated as minimize β∈Rd f ( Xβ ) + g ( β ) , ( 2 ) where β ∈ Rd is the optimization variable , X ∈ Rn×d is a constant matrix , and f : Rn → ( −∞ , ∞ ] and g : Rd → ( −∞ , ∞ ] are proper lower semi-continuous convex functions . We assume ∃β ∈ relint ( dom ( g ) ) s.t.Xβ ∈ relint ( dom ( f ) ) and the existence of the optimal point , i.e. , ∃β̂ ∈ dom ( P ) s.t . P ( β̂ ) = inf β∈Rd P ( β ) , where P : Rd → R is defined as P ( β ) = f ( Xβ ) + g ( β ) . Moreover , we focus on the cases in which g induces sparsity . Although all theorems in this paper hold , we can not eliminate any variables without sparsity . This is a popular class of optimization problem , with the most popular example being Lasso [ 23 ] : minimize β∈Rd 1 2 ‖y −Xβ‖22 + λ‖β‖1 . Many extensions of Lasso , including Group-Lasso [ 26 ] , elastic net [ 27 ] , and sparse logistic regression [ 15 ] , are in this class . The dual problems of a support vector machine [ 5 ] and a support vector regression [ 22 ] are also in this class . 2.3 Dual Problem . In the derivation of a safe screening rule for the optimization problem , Eq . ( 2 ) , the Fenchel– Rockafellar dual formulation , plays an important role . Theorem 2 ( Fenchel–Rockafellar Duality ) If all assumptions for the optimization problem ( 2 ) are satisfied , we have the following : min β∈Rd f ( Xβ ) + g ( β ) = max θ∈Rn −f ? ( −θ ) − g ? ( X > θ ) . ( 3 ) The proof of Theorem 2 is provided in the appendix . We denote −f ? ( −θ ) − g ? ( X > θ ) by D ( θ ) . For primal/dual solutions , there are many conditions that are equivalent to optimality . Herein , we provide a list of such conditions for convenience . Proposition 3 ( Optimal Condition ) If all assumptions for the optimization problem ( 2 ) are satisfied , the following are equivalent . ( a ) β̂ ∈ argmin β∈Rd P ( β ) ∧ θ̂ ∈ argmax θ∈Rn D ( θ ) ( b ) P ( β̂ ) = D ( θ̂ ) ( c ) f ( Xβ̂ ) + f ? ( −θ̂ ) = −θ̂ > Xβ̂ = −g ( β̂ ) − g ? ( X > θ̂ ) ( d ) −θ̂ ∈ ∂f ( Xβ̂ ) ∧X > θ̂ ∈ ∂g ( β̂ ) ( e ) Xβ̂ ∈ ∂f ? ( −θ̂ ) ∧ β̂ ∈ ∂g ? ( X > θ̂ ) ( Proof ) ( a ) ⇐⇒ ( b ) is directly derived from the strong duality . ( b ) ⇐⇒ ( c ) is derived from the Fenchel–Young inequality ( 1 ) . ( c ) ⇐⇒ ( d ) ⇐⇒ ( e ) are derived from Proposition 1 . 2.4 Relationship of Dual Safe Region and Screening . For the optimization problem ( 2 ) , we can eliminate some features by constructing a simple region that contains θ̂ . Assume that the dual optimal point , θ̂ , is within regionR . According to Proposition 3 , we have β̂ ∈ ∂g ? ( X > θ̂ ) ⊂ ⋃ θ∈R ∂g ? ( X > θ ) . Hence , if ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β | βi = 0 } holds , we obtain β̂i = 0 . A simple example is the following corollary . Example 4 Consider an optimization problem , i.e. , Eq . ( 2 ) with g ( β ) = ‖β‖1 . Assume that θ̂ ∈ R. Then , we have max θ∈R |x > i θ| < 1 =⇒ β̂i = 0 . ( Proof ) Based on the definition of g , we have ∂g ? ( X > θ ) ⊂ { β | βi = 0 } ⇐⇒ |x > i θ| < 1 . When maxθ∈R |x > i θ| < 1 , we have β̂ ∈ ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β | βi = 0 } . To eliminate features safely , we should first construct a simple region R ⊂ Rn which contains θ̂ and then determine whether ⋃ θ∈R ∂g ? ( X > θ ) ⊂ { β ∈ Rd | βi = 0 } is guaranteed . In this study , we provide a novel general framework for constructing a simple safe region , R ⊂ Rn . By combining it with existing methods to calculate an upper bound of ⋃ θ∈R ∂g ? ( X > θ ) ( cf . [ 17 ] for the un-overlapping group L1 norm and sparse group L1 norm , [ 1 ] for the ordered weighted L1 norm ) , we can formulate strong safe screening rules . 3 General Framework for Constructing Safe Region . Herein , we propose a general framework for constructing a dual region that contains the solution to the optimization problem expressed by Eq . ( 3 ) . Our framework consists of a general lower bound and a problem-adaptive upper bound of the optimal value . Hence , we can derive a narrower region than the framework with a general upper bound under certain situations . The general lower-bound is derived from the optimal condition and the L-strong convexity . Assume that f ? is L-strongly convex ( L ≥ 0 ) . Then , as D is L-strongly concave and θ̂ is the optimal point , we have D ( θ̃ ) ≤ D ( θ̂ ) − L2 ‖θ̂ − θ̃‖ 2 2 for ∀θ̃ ∈ Rn . Thus , we have θ̂ ∈ { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ D ( θ ) } . Because this region is too complicated for screening , we use a simple upper bound of D ( θ ) to construct a simple safe region . Theorem 5 Consider the optimization problem expressed by Eq . ( 3 ) and assume that f ? is L-strongly convex ( L ≥ 0 ) . Let θ̂ be the solution to Eq . ( 3 ) . Assume that D ( θ ) is upper-bounded by u ( θ ) , i.e. , ∀θ ∈ Rn D ( θ ) ≤ u ( θ ) . Then , for ∀θ̃ ∈ Rn , we have θ̂ ∈ R ( θ̃ , u ) = { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ u ( θ ) } . The complexity ofR ( θ̃ , u ) depends on the complexity of u . For example , if u is linear , thenR ( θ̃ , u ) is a sphere . We can construct a narrow , simple , and safe region with a tight simple upper-bound u . In fact , the Gap Safe Sphere region [ 7 , 17 ] can be derived easily from this theorem and weak duality . Corollary 6 ( Gap Safe Sphere ) Consider the optimization problem described by Eq . ( 3 ) and assume that f ? is L-strongly convex ( L ≥ 0 ) . Let θ̂ be the solution to Eq . ( 3 ) . For ∀β̃ ∈ Rd and ∀θ̃ ∈ Rn , we have θ̂ ∈ { θ | L 2 ‖θ − θ̃‖ 2 2 +D ( θ̃ ) ≤ P ( β̃ ) } . ( Proof ) Based on a weak duality , we have ∀θ D ( θ ) ≤ P ( β̃ ) . Using this constant function as an upper bound in Theorem 5 , we can derive the corollary directly . Hence , our framework can be seen as a generalization of Gap Safe . Owing to this generalization , we can use a stronger problem-adaptive upper bound rather than a weak duality . In the next section , we derive specific regions for the Lasso-Like problem . Some regions for other problems are presented in the appendix .

Summary:  The concern of  the paper is the design of screening rule for sparsity-based regularized convex  optimization problem  such as the Lasso-type problem.  The aim is to screen out  irrelevant parameters during  the optimization  procedure. The  designed screening rule hinges on Fenchel-Rockafellar duality and  encompasses existing  methods (Gap Safe Sphere, Gap Safe Dome, Sasvi). The  paper establishes that the  constructed  safe region for the dual solution is of reduced size than the  ones the aforementioned methods. Hence, the new rule induces a more efficient screening. Empirically evaluation on Lasso problem show that the proposed screening rule brings computation gain when compared to existing methods.