Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a heuristic (non-safe) screening rule to deal with l1-regularized estimation problems such as linear regression and logistic regression. The proposed method can be viewed as a generalization of the strong rule used in the glmnet, which only used first-order information.  The paper reports experiments on both synthetic data and real-world data to illustrate the performance of the proposed method.
strengths_and_weaknesses: Strengths:
* the paper addresses an interesting problem in l1-regularized estimation for linear regression and logistic regression. Screening rules are an effective strategy to speed up such estimation.
* Although the proposed rule is heuristic in nature, the simplicity in its formulation and the effectiveness shown in the experiments offers some advantages of the proposed method. This is also a not-so-common approach that makes use of second-order information for screening.
* Experiments are exhaustive. six well-known alternative methods are compared on a wide variety of synthetic and real-world data.

Weakness:
* The authors may consider carrying out experiments on other settings covered by the proposed method such as poison regression and elastic net. It may also be interesting (somewhat orthgonal) to understand how the proposed method performs or scale compared to SGD based approach on very large datasets.
questions: I don't have additional questions for the authors.
limitations: Limitations are discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: A Hessian screening rule for lasso and its generalized linear model extension for logistic regression was presented to take advantage of the high-order information for more efficient screening, specifically in cases with highly correlated predictors/covariates. The proposed Hessian screening rule together with several speedup tricks has been shown to be effective in both simulated and real datasets. 
strengths_and_weaknesses: The proposed Hessian screening rule extends the Strong Rule and Working Set by taking advantage of the high-order information for more efficient screening, specifically in cases with highly correlated predictors/covariates. The Hessian screening rule together with several speedup tricks has been shown to be effective in both simulated and real datasets. 

It is not clear based on the current presentation why the Hessian rule can be less conservation from Section 3. In addition to Theorem 3.1, some theoretical analysis for that may help further improve the quality of the submission. The actual final screening in fact was based on the modifications as described from line 133 to 151. It may be interesting also to have ablation comparison to see clearly what led to improved efficiency. 

Since, the screening rules are not "safe", in particular for logistic regression. In addition to investigating the efficiency, the authors may also need to provide the performance comparison with respect to both predictor selection and model prediction accuracy. 

In the real data experiments, the authors may want to provide some explanations why the Hessian rule performs significantly worse on arcene and rcv2 datasets, for which p is much larger than n, especially for arcene. It is clearly not the case when p is similar as n as discussed in Section 5. 

Finally, there are language problems in the submission. For example, in line 188-189 (page 5): "... this is not a surprising find."  In line 257-258 (page 7), "we also stop the path whenever the number of coefficients ever to be active predictors exceeds p." The number of coefficients can be equal to p but will never exceed p. The authors may need to improve the presentation of the submission. 
questions: 1. Are there any theoretical guarantee that the Hessian rule will be always less conservative than the Strong rule or Working Set method? 

2. How the corresponding prediction accuracy of the resulting models by different screening rules? Are the final "screened" predictors all the same as the ones from the optimal solution from the model fitting without screening? 

3. There are multiple screening rules published more recently, including: https://proceedings.neurips.cc/paper/2020/hash/11348e03e23b137d55d94464250a67a2-Abstract.html, how does the Hessian screening rule compare with these new rules? 
limitations: N/A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the predictor screening rules over the lasso optimization problem. It proposes a Hessian screening rule which utilizes the second-order information. This rule is effective not only in screening but also in accurate warm starts. Updating the second-order information has high computational complexity, and to deal with it, this work replies on the sweep operator. In the experiments, the proposed rule is compared to many baselines and outperforms them significantly on both simulated and real-world data.


strengths_and_weaknesses: Originality

Previous works on screening rules overlooked the study on the second-order information. Thus I think the direction of this work novel. The proposed rule takes the warm start and the Hessian matrix computation problems into account, and resolves them soundly. Besides, the insightful discussions on the proposed screening rule are provided. They are helpful to understand the rule and to differentiate this approach with previous methods. Therefore, I think the contributions of the work novel.

Quality

I am satisfactory with most contents of this paper. The paper presents a clear overview of the question and previous rules both in words and math. The proposed rule is based on the Hessian matrix and is actually in a form of the second-order Taylor approximation. Speeding up the Hessian matrix computation is based on the sweep operator and the warm starts also benefit from the Hessian matrix. These arguments are demonstrated in the experiments. As far as I checked, the theoretical analyses have no problem.

However, I have a concern with the study method. Normally, people select the predictors for better fitting accuracy, but this paper relies heavily on the time cost and the minimum number of active predictors to measure the performance. I am curious about why not including fitting accuracy.
 
Clarity

The paper is clearly written for me. I can easily follow the contents on the approach and the experiment.

Significance

The idea of the approach is sound and practical performance is better than baselines. However, because just the lasso problem is involved, I am afraid that the audience will be very limited. Therefore, I think the significance of this paper is limited.
questions: See the concerns in the above section.
limitations: Yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a screening strategy for L1 sparse modelings. The basic idea is to use a prediction of an optimal solution as a function of the regularization parameter, derived from optimal conditions (which is aka solution path). The authors further propose combining working set selection screening by predicted solution with so-called strong rule to make screening more efficient. 
strengths_and_weaknesses: Overall, the paper is easy to follow and technical quality is fine. The purpose is clear and the procedure is written in detail. However, a critical issue is that the idea of main proposal (Hessian screening) is not novel. Approaches based on a similar idea have been studies in the context of the path following though it is not fully mentioned. Detailed comments are as follows.

Closely related papers in the path following literature are missed. For example, since the following two papers contains conceptually quite similar approaches, the difference should have been discussed in detail, though currently nothing is mentioned:
[Rosset2004] S. Rosset, Following Curved Regularized Optimization Solution Paths, NeurIPS 2004. 
[Park2006] M. Y. Park and T. Hastie, L1 Regularization Path Algorithm for Generalized Linear Models, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2006.

The proposed algorithm can be seen as a so-called 'predictor-corrector' method in general (e.g., discussed in [Park2006]). Further, [Park2006] also discusses the working set selection based on the update equation (6).

Therefore, most importantly, the idea using Theorem 3.1 to predict the variable c (working set criterion) of the next \lambda has been already known. Therefore, I do not think the concept of 'Hessian screening' is novel.

[Rosset2004] also discusses a similar idea of approach (Hessian based update) based on an essentially quite similar theoretical property to Theorem 3.1. Further, this paper also provides the error analysis of the predictor.

Combination with strong rule and additional adjustment would be novel, but its technical significance is a bit marginal because these are quite simple heuristics.

Techniques in the 'Updating the Hessian' paragraph have been also known (the same techniques repeatedly discussed in the path following literature). 

The 'Warm Starts' has also been widely known (e.g., [Park2006]).

Minor comments:
- Since Theorem 3.1 has been widely known, it should clarify more explicitly rather than noting it only in the footnote.  
questions: Convergence to the optimal solution is not mentioned. Could you give any information about the optimality of solution in a sense of the KKT conditions of (1)?
limitations: In Section 5, limitations were discussed.
review_title: null

Meta-review: metareview: This papers proposed a Hessian screening rule for lasso and its generalized linear model extension for logistic regression. The proposed screening rules have been demonstrated to be effective in both simulated and real datasets. The idea is novel and the evaluation is convincing. The authors mention that extensions to MCP and SCAD may also be possible, even though the objective may not be convex. A brief discussion will be helpful. 
confidence: Less certain
Decision: Accept