Title: FastSurvival: Hidden Computational Blessings in Training Cox Proportional Hazards Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: survival analysis

cox proportional hazards model

optimization

first-order method

sparse learning
Abstract: Survival analysis is an important research topic with applications in healthcare, business, and manufacturing. One essential tool in this area is the Cox proportional hazards (CPH) model, which is widely used for its interpretability, flexibility, and predictive performance. However, for modern data science challenges such as high dimensionality (both $n$ and $p$) and high feature correlations, current algorithms to train the CPH model have drawbacks, preventing us from using the CPH model at its full potential. The root cause is that the current algorithms, based on the Newton method, have trouble converging due to vanishing second order derivatives when outside the local region of the minimizer. To circumvent this problem, we propose new optimization methods by constructing and minimizing surrogate functions that exploit hidden mathematical structures of the CPH model. Our new methods are easy to implement and ensure monotonic loss decrease and global convergence. Empirically, we verify the computational efficiency of our methods. As a direct application, we show how our optimization methods can be used to solve the cardinality-constrained CPH problem, producing very sparse high-quality models that were not previously practical to construct. We list several extensions that our breakthrough enables, including optimization opportunities, theoretical questions on CPH's mathematical structure, as well as other CPH-related applications.

FastSurvival: Hidden Computational Blessings in Training Cox Proportional Hazards Models

Abstract

Survival analysis is an important research topic with applications in healthcare, business, and manufacturing. One essential tool in this area is the Cox proportional hazards (CPH) model, which is widely used for its interpretability, flexibility, and predictive performance. However, for modern data science challenges such as high dimensionality (both n and p ) and high feature correlations, current algorithms to train the CPH model have drawbacks, preventing us from using the CPH model at its full potential. The root cause is that the current algorithms, based on the Newton method, have trouble converging due to vanishing second order derivatives when outside the local region of the minimizer. To circumvent this problem, we propose new optimization methods by constructing and minimizing surrogate functions that exploit hidden mathematical structures of the CPH model. Our new methods are easy to implement and ensure monotonic loss decrease and global convergence. Empirically, we verify the computational efficiency of our methods. As a direct application, we show how our optimization methods can be used to solve the cardinality-constrained CPH problem, producing very sparse high-quality models that were not previously practical to construct. We list several extensions that our breakthrough enables, including optimization opportunities, theoretical questions on CPH's mathematical structure, as well as other CPH-related applications.

1 Introduction

Survival analysis, which studies time-to-event data, is an important research topic with a wide range of real-world applications. In medicine, survival analysis has been employed to model when a patient will die [44, 38, 10]. In business, it is useful for attrition prediction [30] (when an employees resigns) and churn prediction [31] (when a customer unsubscribes), and in manufacturing, it is used to predict when a physical system breaks down [42, 50]. A fundamental tool in analyzing such data is the Cox proportional hazards (CPH) model [8], a linear model under the assumption that features have a multiplicative effect on the risk of failure/event. Simple yet powerful, the CPH model has enjoyed great popularity due to its modeling flexibility (when coupled with additive models [27, 28, 67, 11, 1]). Moreover, in contrast to black box models, it is both interpretable and accurate.

However, with the advent of larger sample and feature spaces and more complex data, new challenges arise in using the CPH model to its full potential. Ideally, practitioners want to produce CPH models repeatedly, with feature engineering and preprocessing between iterations. Additionally, they want the CPH model to identify important variables [64, 14], even in presence of highly correlated features. However, current optimization methods for training the CPH model do not meet these needs. Current algorithms [62, 22, 23, 54], based on the generic Newton's method, are computationally intensive. More importantly, due to both vanishing second-order derivatives [53] and the use of approximation strategies that trade precision for efficiency, existing optimization methods have trouble converging, either with the loss blowing up or the algorithm converging very slowly when we require the precision

necessary to handle correlated variables. The latter issue is the core reason for incorrect variable selection when features are highly correlated.

In this work, we propose new optimization methods to train the CPH model and show that there is not necessarily a precision-efficiency tradeoff. Despite the CPH model being seemingly amenable to classical optimization approaches such as coordinate descent, this has not been attempted for the original CPH loss function due to its daunting complexity. However, through careful examination, we show instead that the complexity of the CPH loss function is really a blessing, rather than a curse. We discover hidden mathematical structures that allow us to design very efficient algorithms. We show both the first and second-order derivatives at each coordinate can be computed exactly in linear time complexity ( O ( n ) ). Moreover, we show both derivatives are Lipschitz-continuous by making novel connections with the second and third central moment calculation in probability theory and statistics. All these discoveries lead us to design algorithms that essentially minimize a quadratic surrogate function and a cubic surrogate function, respectively. They are extremely easy to implement.

Empirically, we demonstrate the superior speed of our algorithms on large-scale datasets. In general, ours are significantly faster than all existing methods and rapidly converge to optimal high-precision solutions. Because our methods produce high-quality solutions, we apply them for variable selection in challenging regimes where features are highly correlated. We solve difficult cardinality-constrained CPH problems and produce models that are much sparser than the state-of-the-art methods.

In summary, our contributions are: (1) We find a critical flaw in the current optimization algorithms for the CPH method by pinpointing that they converge slowly with low precision. Sometimes, the loss does not converge and explodes. (2) To circumvent this issue, we propose novel algorithms that minimize a quadratic and a cubic surrogate function, respectively, with guaranteed convergence and loss descent at each iteration. The core novelty lies in discovering hidden mathematical structure, which allows for an efficiency way ( O ( n ) ) of calculating the second-order partial derivatives exactly. In addition, we show the first and second order partial derivatives are Lipschitz continuous. To calculate these Lipschitz constants, we leverage second and third central moments from theoretical statistics and probability theory. (3) Empirically, our method enjoys fast speed in training the loss function and results in superior performance when solving cardinality-constrained problems.

Our work constitutes a methodological breakthrough in training CPH models. At the end of the paper, we also discuss several exciting extensions and follow-up questions, showing how our new perspectives and discoveries open doors to many new research opportunities.

2 Preliminaries

Given a time-to-event dataset of n samples with { x i , t i , δ i } n i =1 , where x ∈ R p is the feature vector with length p , t i ∈ R is the observation time, and δ i ∈ { 0 , 1 } is an indicator with 1 indicating that a failure event has happened, the CPH model can be used to learn and predict the risk of failure, commonly known as the hazard function in survival analysis. The CPH model predicts the hazard h i ( t ) for sample i in a semiparametric way [62]. For review of related work, please see Appendix B.

where h 0 ( t ) is a baseline hazard function shared by all samples, and β ∈ R p is the parameter of interest. The nice thing about the CPH model is that h 0 ( t ) cancels out if we look only at the ratio of hazards of sample i vs. all remaining samples at time t i , i.e. ,

where R i := { j | t j ≥ t i } is the set of indices whose observation time is greater than or equal to that of sample i . Such ratios are also called partial likelihoods. To estimate the parameter of interest, β , we maximize the joint partial likelihood of all samples with failure events, which can be written as

This is equivalent to minimizing the negative log partial likelihood [62], which is defined as

The loss function ℓ ( β ) , while convex, is very mathematically involved. In addition to the double sum, the inner summation over j is with respect to a different index set R i , for each outer summation index i . Such daunting complexity makes it difficult to employ first-order methods such as gradient descent because we cannot easily pick the right step size for each iteration, which plays a crucial role in practical running time. Therefore, past efforts have been focused on developing Newton-type (second-order) methods, where the loss function is approximated by a second-order Taylor expansion:

The function f (∆ β ) can be minimized by solving a linear system: ∆ ˆ β = -( ∇ 2 β ℓ ( β )) -1 ∇ β ℓ ( β ) . To reveal the computational nuances more explicitly, we use an intermediate variable η with η = Xβ . Then we can rewrite the approximation function f (∆ β ) as:

At each iteration, calculating the Hessian matrix ∇ 2 η ℓ ( η ) requires O ( n 2 ) complexity. Past methods on the CPH model have resorted to various approximation strategies by replacing ∇ 2 η ℓ ( η ) with H ( η ) to reduce the computational complexity:

where diag ( · ) constructs a matrix with its diagonal equal to the input vector and other entries equal to 0. There are two major problems with the above approaches. One common problem is that the these Newton-type methods inherently have trouble converging beyond the local region of minimizers without backtrack line search [53]. We provide a concrete example to demonstrate this issue in the experiment section. Ideally, we want to avoid backtracking because this increases the running time. In contrast, our methods do not have this flaw and guarantee global convergence.

The other problem is that when the above approaches do converge to the optimal solutions, none of them can converge with high precision fast enough (in a practical sense). The exact Newton's method [22] has a local quadratic convergence rate, but each iteration can take a long time. Quasi Newton [62] and proximal Newton [51] 1 methods are computationally much cheaper to evaluate per iteration, but they make less progress toward the optimal solution. In the next section, we show that, by exploiting hidden mathematical structure, we can obtain the best of both worlds : cheap evaluation per iteration and fast convergence with respect to the number of iterations.

3 Methodology

3.1 New Formulas for First, Second, and Third Order Partial Derivatives

As we have mentioned, the reason for the diagonal approximations of ∇ 2 η ℓ ( η ) is to reduce the complexity of the mathematics and associated high computational cost. Here, we take a completely different approach from past methods. First, we avoid making any approximations and embrace the full Hessian matrix. Second, we bypass the intermediate step of calculating the Hessian in the sample space η and focus on the Hessian in the feature space β . The involved mathematics may already sound complicated, but we do not stop here. We apply these two ideas not only to the second order partial derivatives but the third order partial derivatives as well. Although this seems like a burdensome task, we show that the end result is very elegant and has an intuitive interpretation. Out of complexity comes simplicity. We summarize the relevant results in the first following theorem:

Theorem 3.1. For the CPH loss function defined in Equation (4) , the first, second, and third order partial derivatives with respect to coordinate l are:

1st order partial derivative :

2nd order partial derivative :

3rd order partial derivative :

The proof can be found in Appendix A. The first, second, and third order partial derivatives all have a probabilistic interpretation. Notice that for any i , the coefficients in front of X kl , X 2 kl , and X 3 kl are nonnegative and sum up to 1, i.e. , e η k / ( ∑ j ∈ R i e η j ) ≥ 0 and ∑ k ∈ R i [ e η k / ( ∑ j ∈ R i e η j )] = 1 . Then, we can regard these coefficients as a discrete probability distribution. Thus, for Equation (8), the term inside [ · ] resembles the variance or second order central moment formula: E [ X 2 ] -( E [ X ]) 2 = E [( X -E [ X ]) 2 ] . For Equation (9), the term inside [ · ] resembles the skewness or third order central moment formula: E [ X 3 ] + 2( E [ X ]) 3 -3 E [ X 2 ] E [ X ] = E [( X -E [ X ]) 3 ] .

One may wonder whether for higher orders (order r ≥ 4 ), the relationship between the r -th order partial derivative and r -th central moment still preserve. The answer is no and this can be easily deduced from the following lemma. The proof can be found in Appendix A.

Lemma 3.2. Let us define C r to be the r -th central moment with

Then we can calculate the partial derivative of C r with respect to β l as:

From Lemma 3.2, we can see why the connection to central moment does not work for higher order partial derivatives. If r = 2 , the second term in Equation (11) disappears, i.e. , C r -1 = C 1 = 0 . Therefore, we get ∂C 2 /∂β l = C 3 . However, for r ≥ 3 , C r -1 in general is not zero, so we cannot extrapolate this pattern to higher order partial derivatives.

Theorem 3.1 forms the basis upon which we build everything else. These results are not only mathematically interesting but also have significant implications for computation, which we elaborate in the next two sections.

3.2 Time Complexity of First and Second Order Partial Derivative Calculation

From the connections to the second and third central moment, we have the following corollary regarding the time complexity of calculating the first, second, and third order derivatives:

This is a surprising result, especially for the second order partial derivatives. The intermediate Hessian, ∇ 2 η ℓ ( η ) , takes O ( n 2 ) to compute, so we would expect the second order partial derivative, ∂ 2 ℓ ( β ) ∂β 2 j = e T j X T ∇ 2 η ℓ ( η ) Xe j , would take O ( n 2 ) to compute as well. Yet, the time complexity is just O ( n ) . We use the first order partial derivative formula, Equation (7), as an example to explain why this happens. We ignore the second term ∑ n i =1 δ i X il because it is just a constant. The first term in Equation (7) can be rewritten as:

Let us focus on the numerator inside the parenthesis for now. For the entire sequence ( i = 1 , 2 , ..., n ) of numerator terms, we can obtain all of them together at the cost of O ( n ) by performing reverse cumulative summation. The same is true when we obtain the entire sequence of denominators. Once we have all numerators and denominators, calculating the entire sequence of ratios inside the parenthesis also costs O ( n ) . Finally, multiplying each ratio with δ i and summing up all these products costs O ( n ) as well. Therefore, the computational cost to calculate the first order partial derivative is O ( n ) . We can apply the same idea to the second order partial derivative formula, Equation 8, to show that the computational complexity is also O ( n ) . Note that the reverse cumulative summation trick has already been explored in [62] for calculating the diagonal of ∇ 2 η ℓ ( η ) in the sample space η , but this trick has not been used to calculate the partial derivatives in the feature space β .

We will later see how this O ( n ) time complexity allows us to design a second order optimization method whose evaluation cost per iteration is just as cheap as a first order optimization method. Before we discuss that, let us continue and discuss another computational implication of Theorem 3.1.

3.3 Lipschitz-Continuity Property of First and Second Order Partial Derivatives

The connection to the central moment calculation allows us to conclude that the first and second order partial derivatives are Lipschitz-continuous. Moreover, we can calculate these Lipschitz constants explicitly. Recall that for a univariate function f ( x ) , we say that the function is Lipschitzcontinuous [5] if there exists L ≥ 0 such that for any two points in the domain of f , i.e. , x, y ∈ D ( f ) , we have | f ( x ) -f ( y ) | ≤ L | x -y | . The value L is called the Lipschitz constant for function f ( · ) . If the function is continuously differentiable, the previous definition is equivalent to the condition where the first order derivative is bounded, | f ′ ( x ) | ≤ L for any x ∈ R [5].

Not only can we say that the first and second order partial derivatives are Lipschitz-continuous, but we can also calculate the Lipschitz constants explicitly. We summarize the results in the theorem below. The proof can be found in Appendix A

Theorem 3.4. For the second order partial derivatives in Equation (8) , its absolute values are bounded by the following formula:

For the third order partial derivatives in Equation (9) , its absolute values are bounded by the following formula:

The availability of these Lipschitz constants suggests that we might construct surrogate functions.

3.4 Quadratic and Cubic Surrogate Functions

We now have all the tools at hand to attack the original optimization problem. For a univariate convex f ( x ) , if we have access to L 2 , the Lipschitz constant for its first order derivative, then we can construct the following quadratic surrogate function g x ( · ) [53]:

If we have access to L 3 , the Lipschitz constant for its second order derivative, then we can construct the following cubic surrogate function h x ( · ) [53]:

A nice thing about these surrogate functions is that their minimizers can be computed analytically:

where the function sgn ( · ) extracts the sign ( + or -) of the input. The analytical solution to the quadratic surrogate function is well known, but the analytical solution to this cubic surrogate function has not been well studied. We provide a derivation for Equation (18) in Appendix A.

Since these surrogate functions are convex and are upper bounds of the original functions, minimizing them will lead to a decrease of the original function f ( x ) as well. This explains why our methods ensure monotonic decrease in loss and guarantee global convergence. The final algorithms are very easy to understand and can be thought of as coordinate descent-type methods. We anticipate these core ideas can be applied to solve a wide range of problems related to the CPH model. In the next subsection, we showcase two problems our algorithms can tackle.

3.5 Applications to Regularized and Constrained Problems

Regularized Problem The first problem is the regularized CPH problem whose penalty terms are separable. The penalties that qualify for this category include LASSO [64], ElasticNet [72], SCAD [15], MCP [68], etc. For the ℓ 1 -regularized problems, we can in fact find analytical solutions 2 .

For the quadratic surrogate function, solving the ℓ 1 -regularized problem in Equation (15) is equivalent to solving the following optimization problem (with a = f ′ ( x ) , b = L 2 , and c = x ),

The solution for the above problem is

For the cubic surrogate function, solving the ℓ 1 -regularized problem of Equation (16) is equivalent to solving the following optimization problem (with a = f ′ ( x ) , b = f ′′ ( x ) , c = L 3 , and d = x ):

whose solution is:

Equation (20) is well known in a slightly different format. Equation (22) has not been well studied in the past. We provide derivations for both in Appendix A.

Constrained Problem The second problem is the cardinality-constrained CPH problem. Recently, the beam search framework (a combination of the beam search method [66] from natural language processing and generalized orthogonal matching pursuit [13]) has shown promise in finding near optimal solutions for a class of l 0 -constrained nonconvex problems, including sparse ridge regression [47] and sparse logistic regression [48].

Similar to the generalized orthogonal matching pursuit algorithm, we expand our support (starting from an empty set) by adding one feature at a time until the cardinality is satisfied. However, instead of selecting features based on partial derivatives, we select features based on which coefficient, if optimized, can result in the largest decrease of the loss function. After the feature is added into the support, we fine-tune all nonzero coefficients in the support. Additionally, during each support expansion step, we select multiple feature candidate instead of the best one, similar to the core idea in beam search. We use our coordinate descent methods to solve the feature selection step and the coefficient fine-tuning step.

Although the beam search framework has already been proposed for other cardinality-constrained problems, it cannot be applied directly to the CPH model without our coordinate descent methods to select features, especially in the highly correlated settings.

4 Experiments

We test the effectiveness of our optimization methods on both synthetic and real-world datasets. We run experiments for both regularized and constrained problems mentioned in Section 3.5. Our main objectives are: 1) When minimizing the same objective functions, how fast can our methods converge to the optimal solutions when compared with all existing optimization methods for the CPH model? 2) When coupled with the beam search framework, how well can our methods help with variable selection when compared with the state-of-the-arts methods, especially for challenging scenarios where features are highly correlated?

4.1 Accessing How Fast Our Methods Converge to Optimal Solutions

We compare our methods (one based on the quadratic surrogate function and the other based on the cubic surrogate function) with the existing optimization methods outlined in Section 2: exact Newton method, the quasi Newton method, and the proximal Newton method. We run on both ℓ 2 -regularized CPH problems and ℓ 1 + ℓ 2 -regularized CPH problems. The choices of these regularizations are: λ 2 = { 0 , 1 } and λ 2 = { 1 , 5 } . The coefficients are all initialized to be 0 . In the main paper, we show results on the Flchain dataset in Figure 1. More results on other datasets can be found in Appendix D. During each iteration, the baseline methods [62, 51] optimize all coefficients at once, whereas our methods optimize coefficients sequentially with respect to the original loss function. To assess the per-iteration convergence rate, we plot the CPH loss against the number of iterations. To assess the practical running speed, we plot the CPH loss against the overall time elapsed (wall clock). From the left two plots of loss vs . number of iterations, we see that the Newton-type baselines sometimes have losses that blow up or increase during the initial phase of optimization. This is a common problem of Newton's method. Our methods are the only ones with monotonically decreasing loss curves. This is the main reason why only our method can be used for the beam search framework in the variable selection experiments. From the right two plots of loss vs . overall time elapsed, we can see that our methods are significantly faster than the baselines . This is due to the fact that both our first and second order partial derivatives are very cheap to compute (with time complexity O ( n ) ), as we have explained in Section 3.2.

4.2 Accessing How Well Our Methods Perform Variable Selection

We compare our method with both Cox-based methods and other model classes. For Cox-based models, we run on both synthetic datasets and real-world datasets. For other model classes, we only run on the real-world datasets. To assess how well different methods select important variables, features are highly correlated in all datasets. Synthetic datasets are generated with high correlation level, ρ = 0 . 9 . For each continuous feature on the real-world datasets, we perform binary thresholding for preprocessing [49] to obtain many one-hot encoded binary features. This preprocessing step result in highly correlated features on which it is challenging to perform variable selection. We use the

Loss vsIters

Loss vs Time

Loss vsIters

Loss vs Time

lambda1=0,lambda2=1

lambda1=0,lambda2=1

lambda1=1,lambda2=5

lambda1=1,lambda2=5

15000

15000

14500

14500

losses

105

105

14000

14000

13500

13500

13000

13000

12500

12500

12000

12000

104

104

0

5

10

15

20

0

2

4

6

0

5

10

0.0

2.5

5.07.510.012.5

iters

elapsed_times(sec)

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

SupportSizevsf1_score

Support Size vsf1_score

SupportSizevsf1_score

Dataset:SyntheticHighCorrHighDim1

Dataset:SyntheticHighCorrHighDim2

Dataset:SyntheticHighCorrHighDim3

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

S

0.4

f

0.4

0.4

0.2

0.2

0.2

100

101

102

100

101

102

100

101

Support Size

SupportSize

SupportSize

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

following metrics to evaluate our solution qualities: CPH loss, CIndex, and IBS. On the synthetic datasets where we know the true coefficients, we also calculate the F1 score. We perform 5-fold cross validation and report the mean and standard deviation of different metrics on both the training and test sets. For details about the experimental setup, please see Appendix C. For Cox-based methods, we compare our method with Coxnet, Abess, and Adaptive Lasso.

Results on the synthetic datasets are shown in Figure 2. We plot support size vs F1 score. The F1 score is closely related to the support recovery rate. Our method performs significantly better than the baselines. In particular, on the leftmost plot with 1200 samples, our method is the only one to achieve 100% recovery rate ; the true support size is 15 and we recover all 15 features with a model of size 15. Results for the Employee Attrition dataset are shown in Figure 3. We plot support size vs . CIndex and support size vs . IBS. Similar to the trend on the synthetic datasets, our method performs significantly better than the baselines in terms of both metrics . Lastly, we compare our method with other model classes on the Dialysis dataset. The results are shown in Figure 4. We plot support size vs . CIndex and support size vs . IBS. The results indicate that other model classes are prone to overfitting on the training sets. Our method achieves the best accuracy-sparsity tradeoff. We are able to obtain solutions with the smallest number of coefficients without losing predictive performance.

SupportSizevstrain_cindex

SupportSizevstest_cindex

SupportSize vs train_ibs

SupportSizevstest_ibs

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

0.90

0.88

0.90

0.88

T

0.18

0.20

0.18

0.16

_cindex

0.84

0.86

cindex

0.86

0.84

ibs

0.14

ibs

0.16

0.14

=

0.82

test_

10.82

tra

0.12

0.12

trai

0.80

0.80

0.10

0.10

0.78

0.78

0.08

0.08

0.76

0.76

0.06

0

10

20

30

0

10

20

30

0

10

20

30

0

10

20

30

SupportSize

SupportSize

SupportSize

SupportSize

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

SupportSize vs train_cindex

SupportSizevstest_cindex

SupportSizevs train_ibs

Support Size vs test_ibs

Dataset:Dialysis

Dataset:Dialysis

Dataset:Dialysis

Dataset:Dialysis

1.00

0.95

0.74

0.175

0.20

0.72

train_cindex

0.90

cindex

0.70

0.150

0.19

0.85

0.68

b

S

0.125

S

0.18

0.75

0.80

0.70

0.66

0.64

0.62

u

=

0.075

0.050

0.100

0.16

0.17

0.65

0.60

0.025

0.15

0.60

0.58

0.14

0

200000400000600000

101

103

105

101

103

105

101

103

105

SupportSize

SupportSize

SupportSize

SupportSize

sksurvTree

sksurvRSF

sksurvGBST

sksurvNaiveSVM

★

sksurvFastSVM

BeamSearch(Ours)

All these results demonstrate the superior sparse learning capability of our method. For more results, with all baselines on all datasets, please see Appendix D.

Limitations of FastSurvival Our work focuses on efficient training and effective variable selection of the CPH model. Other model classes, such as trees, random forests, and neural networks, have their own unique merits in capturing complex patterns when the linear (or in our case, additive) model assumption is not satisfied. Another limitation is using the CPH model itself, since its assumptions do not always hold. Handling this question is out of scope for this work.

5 Conclusion and Future Outlook

We presented new optimization methods to train the Cox proportional hazards (CPH) model by constructing and minimizing either a quadratic or a cubic surrogate function. We achieve computational efficiency by exploiting the hidden mathematical structures discovered for the CPH model. Our algorithms are able to train the model significantly faster than previous approaches while avoiding the issue of loss explosion. Furthermore, when applied to the variable selection problem, our method can produce solutions with much fewer parameters while maintaining predictive performance. There are many possible extensions to build upon this work. On the optimization side, it will be interesting to see whether we can derive analytical solutions for other types of regularizers mentioned in Section 3.5. On the theoretical side, questions remain whether higher order partial derivatives are Lipschitz-continuous and how to compute these Lipschitz constants. On the application side, we can apply our method to solve the CPH models with time-varying features [16], stratifications [40], and feature interactions [45].

Code Availability

Implementations of FastSurvival discussed in this paper are available at https://github.com/ jiachangliu/FastSurvival .

References

A. Bender, A. Groll, and F. Scheipl. A generalized additive model approach to time-to-event analysis. Statistical Modelling , 18(3-4):299-321, 2018.

Q. Bertrand, Q. Klopfenstein, P.-A. Bannier, G. Gidel, and M. Massias. Beyond l1: Faster and better sparse models with skglm. Advances in Neural Information Processing Systems , 35:38950-38965, 2022.

D. Bertsimas, J. Dunn, E. Gibson, and A. Orfanoudaki. Optimal survival trees. Machine Learning , 111(8):2951-3023, 2022.

D. Bertsimas, A. King, and R. Mazumder. Best subset selection via a modern optimization lens. The Annals of Statistics , pages 813-852, 2016.

S. Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports , 8(1):6085, 2018.

T. Ching, X. Zhu, and L. X. Garmire. Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data. PLoS Computational Biology , 14(4):e1006076, 2018.

D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological) , 34(2):187-202, 1972.

I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 57(11):1413-1457, 2004.

A. Dispenzieri, J. A. Katzmann, R. A. Kyle, D. R. Larson, T. M. Therneau, C. L. Colby, R. J. Clark, G. P. Mead, S. Kumar, L. J. Melton III, et al. Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population. In Mayo Clinic Proceedings , volume 87, pages 517-523. Elsevier, 2012.

V. B. Djeundje and J. Crook. Identifying hidden patterns in credit risk survival data using generalised additive models. European Journal of Operational Research , 277(1):366-376, 2019.

E. Drysdale. Survset: An open-source time-to-event dataset repository. arXiv preprint arXiv:2203.03094 , 2022.

E. R. Elenberg, R. Khanna, A. G. Dimakis, and S. Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics , 46(6B):3539-3568, 2018.

J. Fan, Y. Feng, and Y. Wu. High-dimensional variable selection for Cox's proportional hazards model. In Borrowing Strength: Theory Powering Applications-A Festschrift for Lawrence D. Brown , volume 6, pages 70-87. Institute of Mathematical Statistics, 2010.

J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association , 96(456):1348-1360, 2001.

L. D. Fisher and D. Y. Lin. Time-dependent covariates in the cox proportional-hazards regression model. Annual review of public health , 20(1):145-157, 1999.

S. Fotso et al. PySurvival: Open source package for survival analysis modeling, 2019.

J. Friedman, T. Hastie, H. Höfling, and R. Tibshirani. Pathwise coordinate optimization. The Annals of Applied Statistics , 1(2):302 - 332, 2007.

J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics , 9(3):432-441, 2008.

J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software , 33(1):1, 2010.

E. Giunchiglia, A. Nemchenko, and M. van der Schaar. Rnn-surv: A deep recurrent model for survival analysis. In Artificial Neural Networks and Machine Learning-ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27 , pages 23-32. Springer, 2018.

J. Goeman. penalized: L1 (lasso) and L2 (ridge) penalized estimation in glms and in the Cox model. R package version 09-21 2008 , 2008.

J. J. Goeman. L1 penalized estimation in the Cox proportional hazards model. Biometrical journal , 52(1):70-84, 2010.

E. Graf, C. Schmoor, W. Sauerbrei, and M. Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine , 18(17-18):25292545, 1999.

J. Gui and H. Li. Penalized Cox regression analysis in the high-dimensional and low-sample size settings, with applications to microarray gene expression data. Bioinformatics , 21(13):30013008, 2005.

F. E. Harrell Jr, K. L. Lee, and D. B. Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine , 15(4):361-387, 1996.

T. Hastie and R. Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association , 82(398):371-386, 1987.

T. Hastie and R. Tibshirani. Generalized additive models for medical research. Statistical methods in medical research , 4(3):187-196, 1995.

H. Hazimeh and R. Mazumder. Fast best subset selection: Coordinate descent and local combinatorial optimization algorithms. Operations Research , 68(5):1517-1537, 2020.

S. Hill. Employee churn prediction, 2017.

H. Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC77.

T. Hothorn, K. Hornik, and A. Zeileis. Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics , 15(3):651-674, 2006.

T. Hothorn, K. Hornik, and A. Zeileis. ctree: Conditional inference trees. The Comprehensive R Archive Network , 8, 2015.

C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan. A dual coordinate descent method for large-scale linear svm. In Proceedings of the 25th international conference on Machine learning , pages 408-415, 2008.

T. Huisman, J. G. van der Linden, and E. Demirovi´ c. Optimal survival trees: A dynamic programming approach. arXiv preprint arXiv:2401.04489 , 2024.

H. Ishwaran and U. B. Kogalur. Random survival forests for r. R News , 7(2):25-31, 2007.

H. Ishwaran, U. B. Kogalur, E. H. Blackstone, and M. S. Lauer. Random survival forests. Annals of Applied Statistics , 2(3):841-860, 2008.

J. D. Kalbfleisch and R. L. Prentice. The statistical analysis of failure time data . John Wiley & Sons, 2011.

J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Medical Research Methodology , 18(1):1-12, 2018.

D. G. Kleinbaum and M. Klein. Survival analysis a self-learning text . Springer, 1996.

J. Ko. Solving the Cox proportional hazards model and its applications . PhD thesis, Master's thesis. EECS Department, University of California, Berkeley, 2017.

B. Kunzer, M. Berges, and A. Dubrawski. The digital twin landscape at the crossroads of predictive maintenance, machine learning and physics based modeling. arXiv preprint arXiv:2206.10462 , 2022.

M. LeBlanc and J. Crowley. Survival trees by goodness of split. Journal of the American Statistical Association , 88(422):457-467, 1993.

S. Lemeshow, S. May, and D. W. Hosmer Jr. Applied survival analysis: regression modeling of time-to-event data . John Wiley & Sons, 2011.

R. Li and L. Chambless. Test for additive interaction in proportional hazards models. Annals of epidemiology , 17(3):227-236, 2007.

Y. li, V . Rakesh, and C. Reddy. Project success prediction in crowdfunding environments. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining , WSDM'16, pages 247-256, 02 2016.

J. Liu, S. Rosen, C. Zhong, and C. Rudin. Okridge: Scalable optimal k-sparse ridge regression. Advances in Neural Information Processing Systems , 36, 2024.

J. Liu, C. Zhong, B. Li, M. Seltzer, and C. Rudin. Fasterrisk: Fast and accurate interpretable risk scores. Advances in Neural Information Processing Systems , 35:17760-17773, 2022.

J. Liu, C. Zhong, M. Seltzer, and C. Rudin. Fast sparse classification for generalized linear and additive models. Proceedings of machine learning research , 151:9304, 2022.

G. Moat and S. Coleman. Survival analysis and predictive maintenance models for non-sensored assets in facilities management. In 2021 IEEE international conference on big data (Big Data) , pages 4026-4034. IEEE, 2021.

B. Moufad, P.-A. Bannier, Q. Bertrand, Q. Klopfenstein, and M. Massias. skglm: improving scikit-learn for regularized generalized linear models. Journal of Machine Learning Research , 23:1-5, 2023.

Y. Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018.

Y. Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Mathematical programming , 108(1):177-205, 2006.

M. Y. Park and T. Hastie. L1-regularization path algorithm for generalized linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology , 69(4):659-677, 2007.

J. Platt. Sequential minimal optimization: A fast algorithm for training support vector machines. Technical report, Microsoft, April 1998.

S. Pölsterl. scikit-survival: A library for time-to-event analysis built on top of scikit-learn. Journal of Machine Learning Research , 21(212):1-6, 2020.

S. Pölsterl, N. Navab, and A. Katouzian. Fast training of support vector machines for survival analysis. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part II 15 , pages 243-259. Springer, 2015.

T. Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica , 9(129-145):20, 1935.

B. D. Ripley and R. M. Ripley. Neural networks as statistical methods in survival analysis. Clinical Applications of Artificial Neural Networks , 237:255, 2001.

M. SÁ Carvalho, R. Henderson, S. Shimakura, and I. P. S. C. Sousa. Survival of hemodialysis patients: modeling differences in risk of dialysis centers. International Journal for Quality in Health Care , 15(3):189-196, 2003.

R. Sharma, M. Gupta, and G. Kapoor. Some better bounds on the variance with applications. Journal of Mathematical Inequalities , 4(3):355-363, 2010.

N. Simon, J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for c ox's proportional hazards model via coordinate descent. Journal of Statistical Software , 39(5):1, 2011.

A. Tarkhan and N. Simon. Bigsurvsgd: Big survival data analysis via stochastic gradient descent. arXiv preprint arXiv:2003.00116 , 2020.

R. Tibshirani. The lasso method for variable selection in the Cox model. Statistics in medicine , 16(4):385-395, 1997.

V. Van Belle, K. Pelckmans, J. A. Suykens, and S. Van Huffel. Support vector machines for survival analysis. In Proceedings of the third international conference on computational intelligence in medicine and healthcare (cimed2007) , pages 1-8, 2007.

S. Wiseman and A. M. Rush. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1296-1306, 2016.

S. N. Wood. Generalized additive models: an introduction with R . chapman and hall/CRC, 2017.

B. C.-H. ZHANG. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , 38(2):894-942, 2010.

H. H. Zhang and W. Lu. Adaptive lasso for Cox's proportional hazards model. Biometrika , 94(3):691-703, 2007.

R. Zhang, R. Xin, M. Seltzer, and C. Rudin. Optimal sparse survival trees. arXiv preprint arXiv:2401.15330 , 2024.

J. Zhu, X. Wang, L. Hu, J. Huang, K. Jiang, Y. Zhang, S. Lin, and J. Zhu. abess: a fast bestsubset selection library in python and r. Journal of Machine Learning Research , 23(202):1-7, 2022.

H. Zou and T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301-320, 2005.

Appendix to FastSurvival: Hidden Computational Blessings in Training Cox Proportional Hazards Models

Table of Contents

A Derivations and Proofs

A.1 First, Second, and Third Order Partial Derivatives

The derivation for the first order partial derivative is shown in Section A.1.1. The derivation for the second order partial derivative is shown in Section A.1.2. The derivation for the third order partial derivative is shown in Section A.1.3.

A.1.1 First Order Partial Derivative

We want to show that

because R i is a set of indices whose time is greater than or equal to y i

A.1.2 Second Order Partial Derivative

We want to show that

= n ∑ i =1 δ i [ n ∑ k 2 =1 ∑ k ∈ R i ( e η k ✶ k = k 2 ∑ j ∈ R i e η j X kl X k 2 l -e η k e η k 2 ✶ y k 2 ≥ y i (∑ j ∈ R i e η j ) 2 X kl X k 2 l )] # distribute X kl X k 2 l inside ( · ) = n ∑ i =1 δ i [ ∑ k ∈ R i n ∑ k 2 =1 ( e η k ✶ k = k 2 ∑ j ∈ R i e η j X kl X k 2 l -e η k e η k 2 ✶ y k 2 ≥ y i (∑ j ∈ R i e η j ) 2 X kl X k 2 l )] # exchange summation orders; sum over k 2 and then k = n ∑ i =1 δ i [ ∑ k ∈ R i ( n ∑ k 2 =1 e η k ✶ k = k 2 ∑ j ∈ R i e η j X kl X k 2 l -n ∑ k 2 =1 e η k e η k 2 ✶ y k 2 ≥ y i (∑ j ∈ R i e η j ) 2 X kl X k 2 l )] # distribute ∑ n k 2 =1 inside ( · ) = n ∑ i =1 δ i [ ∑ k ∈ R i ( e η k ∑ j ∈ R i e η j X 2 kl -∑ k 2 ∈ R i e η k e η k 2 (∑ j ∈ R i e η j ) 2 X kl X k 2 l )] # for the first term, only k 2 = k is left because of the indicator ✶ k = k 2 ; for the second term, the expression can be simplified because R i is a set of indices whose time is greater than or equal to y i = n ∑ i =1 δ i ( ∑ k ∈ R i e η k ∑ j ∈ R i e η j X 2 kl -∑ k ∈ R i ∑ k 2 ∈ R i e η k e η k 2 (∑ j ∈ R i e η j ) 2 X kl X k 2 l ) # distribute ∑ k ∈ R i inside ( · ) = n ∑ i =1 δ i [ ∑ k ∈ R i e η k ∑ j ∈ R i e η j X 2 kl -( ∑ k ∈ R i e η k ∑ j ∈ R i e η j X kl )( ∑ k 2 ∈ R i e η k 2 ∑ j ∈ R i e η j X k 2 l )] # the double sum of the second term can be rewritten as product of two sums = n ∑ i =1 δ i [ ∑ k ∈ R i e η k ∑ j ∈ R i e η j X 2 kl -( ∑ k ∈ R i e η k ∑ j ∈ R i e η j X kl ) 2 ]

# the two terms of the product are exactly the same and can be simplified because k and k 2 are independent

A.1.3 Third Order Partial Derivative

We want to show that

A.2 Partial Derivative of r -th Central Moment

Proof. Recall that the r -th central moment C r is defined as:

Let us focus on the solution for the subproblem ∂ ∂β l [ e η k / ( ∑ j ∈ R i e η j )] .

Let us now plug this result back into the original problem:

A.3 First and Second Order Partial Derivatives Are Lipschitz-Continuous

Proof. To show that the first and second order partial derivatives are Lipschitz-continuous, we need to show that the second and third order partial derivatives are bounded, respectively.

First order partial derivative is Lipschitz-continuous To show that the first order partial derivative with respect to each coordinate is Lipschitz, we need to show that the second order partial derivative with respect to each coordinate is bounded. Recall that the second order partial derivative with respect to the l -th coordinate can be expressed as:

It suffices to show that each term inside the bracket is bounded.

If we interpret the expression probabilistically, then the coefficients e η k ∑ j ∈ Ri e η j in front of X 2 kl and X kl can be thought of as the probability of a particular distribution because all terms are greater than or equal to 0 and sum up to 1 .

For notational convenience, let us use a to denote the probability of this specific distribution with a k = e η k ∑ j ∈ Ri e η j . Then we can rewrite each term inside [ · ] as:

The right-hand side is nothing but the variance of { X kl } k ∈ R i with respect to the distribution { a k } k ∈ R i . Since the variance is always non-negative, we have

Let us now denote a := min k ∈ R i X kl as the minimum of this given set, b := max k ∈ R i X kl as the maximum of this given set, Z as a random variable with values restricted to [ a, b ] . We are going to show that

We achieve this through two steps.

First, suppose the random variable can only take finite | R i | number of values { Z 1 , Z 2 , ..., Z | R i | } with probability { p 1 , p 2 , ..., p | R i | } , where | R i | is the cardinality of the set R i . Then, we have

where the expectation is taken with respect to the distribution p . The above inequality holds because the left-hand side is a specific instance of the expression inside [ · ] , so taking max( · ) produces the inequality above.

Next, if we drop the assumption that the random variable Z can only take | R i | number of values, then the maximum variance we can achieve is no smaller than before. Mathematically, this means that

Combining Inequality (24) and Inequality (25), we arrive at Inequality (23). Lastly, note that the Popoviciu Inequality [58] tells us that for a bounded random variable restricted to [ a, b ] , the maximum variance it can achieve is ( a -b ) 2 4 , i.e. ,

This allows us to conclude that

Plugging in this inequality into the expression of the second order partial derivative, we can show that the second order partial derivative is bounded:

Second order partial derivative is Lipschitz-continuous This proof is similar to the first part above. We need to show that the third order partial derivative with respect to each coordinate is bounded. Recall that the third order partial derivative can be expressed as:

As we have done in the first part, if we use a to denote the probability of this specific distribution with a k = e η k ∑ j ∈ Ri e η j , we can rewrite each term inside [ · ] as

Similarly, as in the first part, let us now denote a := min k ∈ R i X kl as the minimum of this given set, b := max k ∈ R i X kl as the maximum of this given set, Z as a random variable with values restricted to [ a, b ] . Using the exact same logic, we have

The expression inside |·| on the right-hand side is known as the third central moment (skewedness) in statistics. Fortunately, we can derive an explicit formula for the maximum of the absolute third central moment of a bounded variable.

According to [61], we have the following inequality involving the second and third central moment:

From this, we can derive an upper bound on the third central moment:

For notational convenience, let us denote V := E [( Z -E [ Z ]) 2 ] , then the right-hand side above can be expressed as a function of V:

Because V is the variance, V ∈ [0 , 1 4 ( b -a ) 2 ] . Additionally, because √ · is monotonically increasing, the maximum is achieved either at the points where the first order derivative of ( b -a ) 2 V 2 -4 V 3

with respect to V is 0 or at the boundary, 0 and 1 4 ( b -a ) 2 . Let us calculate the points where the first order derivative is 0 :

To obtain the maximum value achievable, we calculate the values at points V = 0 , V = ( b -a ) 2 6 , and V = 1 4 ( b -a ) 2 and pick the maximum value afterwards:

Therefore, the maximum value achievable is 1 6 √ 3 | b -a | 3 .

We now show that upper bound on the absolute value of the third central moment is actually tight by providing with a concrete example. For a random variable Z, let P [ Z = a ] = 1 4 , P [ Z = b ] = 1 4 , and P [ Z = a + b 2 ] = 1 2 . We can verify that E [( Z -E [ Z ]) 3 ] = 1 6 √ 3 | b -a | 3 , thus proving that this upper bound is indeed tight.

This helps us to arrive at the following inequality:

Therefore, for the upper bound of the third order partial derivative, we have the following explicit formula:

A.4 Analytical Solution to the Cubic Surrogate Problem

Let f ( x ) be a convex function whose first, second, and third derivatives all exist. Let h x (∆ x ) := f ( x ) + f ′ ( x )∆ x + 1 2 f ′′ ( x )∆ x 2 + 1 6 L 3 | ∆ x | 3 be the cubic surrogate function [53] of f ( x ) , where L 3 > 0 is the Lipschitz-constant of the second derivative f ′′ ( x ) . Then, the minimum of this surrogate function is achieved at the following point:

Proof. We discuss three cases: f ′ ( x ) > 0 , f ′ ( x ) < 0 , and f ′ ( x ) = 0 .

Case 1 f ′ ( x ) > 0 .

If f ′ ( x ) > 0 , then ∆ˆ x < 0 . For the sake of contradiction, suppose ∆ˆ x > 0 , which means h x (∆ x ) achieves its minimum at some point with ∆ x > 0 . However, we arrive at a contradiction because

Therefore, the minimum is achieved either at ∆ x = 0 or ∆ x < 0 . However, since ∆ x 2 and | ∆ x | 3 grow slower than | ∆ x | when ∆ x is close to 0, there exists some ∆ x < 0 such that f ′ ( x )∆ x + 1 2 f ′′ ( x )∆ x 2 + 1 6 L 3 | ∆ x | 3 < 0 . Thus, h x (0) cannot be the minimum value, and we are left with the

minimum value achieved at some ∆ x < 0 . If ∆ x < 0 , h x (∆ x ) = f ( x ) + f ′ ( x )∆ x + 1 2 f ′′ ( x )∆ x 2 -1 6 L 3 (∆ x ) 3 . Note that the second order derivative of h x (∆ x ) is greater than or equal to 0 since

Therefore, h x (∆ x ) is a convex function with respect to ∆ x when ∆ x < 0 , and its minimum value is achieved when the first order derivative is 0 . When the derivative with respect to ∆ x is 0 , we have

Since f ′ ( x ) > 0 , only one root f ′′ ( x ) -√ ( f ′′ ( x )) 2 +2 L 3 f ′ ( x ) L 3 satisfing the condition ∆ x < 0 . Thus, when f ′ ( x ) < 0 , we have

Case 2 f ′ ( x ) < 0 .

If f ′ ( x ) < 0 , we have ∆ˆ x > 0 using the same logic as above. If ∆ x > 0 , h x (∆ x ) = f ( x ) + f ′ ( x )∆ x + 1 2 f ′′ ( x )∆ x 2 + 1 6 L 3 (∆ x ) 3 . Note that the second order derivative of h x (∆ x ) is also greater than or equal to 0 since

Therefore, h x (∆ x ) is a convex function with respect to ∆ x when ∆ x > 0 , and its minimum value is achieved when the first derivative is 0 . When the derivative with respect to ∆ x is 0 , we have

Since f ′ ( x ) < 0 , only one root -f ′′ ( x )+ √ ( f ′′ ( x )) 2 -2 L 3 f ′ ( x ) L 3 satisfing the condition ∆ x > 0 . Thus, when f ′ ( x ) < 0 , we have

Case 3 f ′ ( x ) = 0 .

When f ′ ( x ) = 0 , the minimum value of h x (∆ x ) is achieved at ∆ x = 0 .

The explicit formulas for the three cases above can be unified into one succinct formula below:

A.5 Analytical Solution to the ℓ 1 -regularized Quadratic and Cubic Surrogate Problems

ℓ 1 -regularized quadratic surrogate problem We have the following ℓ 1 -regularized quadratic surrogate problem:

The solution for the above problem is

Proof. Since the function a ∆ x + 1 2 b ∆ x 2 + λ 1 | c +∆ x | is convex, the condition for this function to achieve the minimum value is for its differential to include 0 . The differential of this function is:

For the first condition, if the differential contains 0 , we have

However, because we require ∆ x > -c , we have

For the second condition, if the differential contains 0 , we have

However, because we require ∆ x < -c , we have

For the third condition, if the differential contains 0 , we have

However, because we require ∆ x = -c , we have

ℓ 1 -regularized cubic surrogate problem We have the following ℓ 1 -regularized cubic surrogate problem:

The solution to the above problem is

Proof. Like the first part, since the function a ∆ x + 1 2 b ∆ x 2 + 1 6 c | x | 3 + λ 1 | d +∆ x | is convex, the condition for this function to achieve the minimum value is for its differential to include 0 . We discuss the differential of this function in two cases: d ≥ 0 and d < 0 .

When d ≥ 0 , the differential of this function is:

We discuss these 5 cases one by one.

For the first condition, if the differential contains 0, we have

However, because we require ∆ x > 0 , we have

This means that we can only have one root because the other root violates ∆ x > 0 :

For the second condition, if the differential contains 0, we have

However, because we require ∆ x = 0 , we have

For the third condition, if the differential contains 0, we have

However, because we require -d < ∆ x < 0 , we have

This means that we can only have one root because the other root violates the condition -d < ∆ x < 0 :

Moreover, since the root is between -d and 0 , and the coefficients in front of ∆ x , -1 2 c , is negative, we have a -bd -1 2 cd 2 + λ 1 < 0 .

For the fourth condition, if the differential contains 0, we have

However, because we require ∆ x = -d , we have

For the fifth condition, if the differential contains 0, we have

However, because we require ∆ x < -d , we have

This means that we can only have one root because the other root violates the condition ∆ x < -d ≤ 0 :

Moreover, because the left root is less than -d and the coefficient in front ∆ x 2 , -1 2 c , is negative, we have a -bd -1 2 cd 2 -λ 1 > 0 .

When d < 0 , the differential of this function is

Similar to the previous part when d ≥ 0 , we discuss the 5 cases one by one but omit the details because the logic and the reasoning process are exactly the same:

For the first condition, if the differential contains 0 , we have

For the second condition, if the differential contains 0 , we have

For the third condition, if the differential contains 0 , we have

For the fourth condition, if the differential contains 0 , we have

For the fifth condition, if the differential contains 0 , we have

We can combine the two situations where d ≥ 0 and d < 0 and obtain a unified formula:

The first and second equations above can be further unified into just one equation sgn ( d ) ( -b

+ √ b 2 -2 c ( sgn ( d ) a + λ 1 ) ) /c if sgn ( d ) a + λ 1 ≤ 0 , so we finally have

B Related Work

Optimization for CPH One way to train the CPH model is through gradient descent [63]. However, because the CPH loss is complex, it is difficult to pick the right step sizes, so gradient descent tends to be slow when we want to obtain solutions with high precision. To speed up the training process, people have used the Newton method [25, 22, 23]. The drawback of this approach is that it is computationally intensive to calculate the full Hessian matrix. Moreover, the vanilla Newton method cannot be applied to solve the ℓ 1 -regularized problem. To alleviate this problem, quasi Newton [62] and proximal Newton [51] methods have been proposed. However, as we have shown in our experiments, these Newton methods have the flaw of training loss blow up due to vanishing second order derivatives. A generic binary search method [41] has also been proposed, but the algorithm has been shown to be slower than the quasi Newton method. In contrast to all these approaches, our method is computationally efficient, can easily handle different regularizers, and guarantees that the loss decreases monotonically.

Modern First and Second-order Optimization Methods Our work is greatly inspired by the modern developments for convex optimization [52], but the general principle cannot be rigidly applied. For first order methods, as we mentioned about gradient descent in the last paragraph, it is difficult to choose the right stepsize for fast convergence. Instead of performing gradient descent, our method performs coordinate descent, which has been shown to be effective in training other statistical models [9, 18, 19, 20, 34, 55]. We give an explicit formula, by leveraging the Popoviciu's inequality on variances [58], to calculate the Lipschitz constant at each coordinate. We also design a second order method (still under the coordinate descent framework) based on the cubic-regularization of the Newton method [53]. To calculate the Lipschizt constant, we make the connections to the third central moment and Bhatia-Davis's inequality [61]. Moreover, we are able to exploit the mathematical structures of the CPH model to compute the second order partial derivatives at the computational complexity of O ( n ) , making the evaluation per iteration of our second order method as fast as that of our first order method.

Variable Selection and Interpretability for the CPH Model If we can find sparse solutions whose predictive performances are as good as dense solutions, we can better interpret which features play important roles. A popular way to select important variables for the CPH model is adding an ℓ 1 regularization term [64], commonly known as the LASSO method. We can also use the ElasticNet method, which adds an ℓ 1 + ℓ 2 regularization term [62]. Another way is to apply the Adaptive LASSO [69], which repeatedly use the absolute values of coefficients obtained from the previous iteration as weights of ℓ 1 regularizations for parameters in the current iteration. These above approaches all use convex regularizers and have difficulty obtaining high-quality solutions when the support size is small. The reason is that these convex regularizers penalize the magnitude of the coefficients while promoting sparsity. To avoid this issue, recently, solving the ℓ 0 -constrained problem [49, 29, 4] has attracted lots of attention and shown to produce much sparser models without losing accuracy. For the ℓ 0 -constrained CPH problem, ABESS [71] has proposed to use a hybrid method of greedy selection and feature swapping to solve the problem heuristically. However, as we have shown in our experiments, this method cannot handle highly correlated features. Our method also solves the ℓ 0 -constrained CPH problem but uses the beam search framework [66, 48, 47]. Although the beam search framework has already existed, this frameowork cannot be applied to the CPH model without our coordinate descent algorithm. We need to use coordinate descent for support expansion as well as coefficient finetuning, in which other Newton-type methods all have issues with losses potentially blowing up.

Other Model Classes for Survival Analysis In addition to the CPH model, there are some other model classes that can be applied to analyze time-to-event data. One model class is the survival tree models [70, 3, 35]. Survival trees have the advantage of capturing non-linear interactions between features. However, when sparse models are desired, the accuracy of sparse trees is compromised by the fact that all samples in the same leaf node share the same predictions. One way to overcome this issue is to construct ensembles of trees using random forest or boosting techniques [36, 32, 33]. Another model class for survival analysis is based on neural networks [39, 7, 6, 59, 21]. However, for all these other model classes mentioned, they are not very interpretable due to large parameter space. The CPH model, which is the focus of our work, provides both interpretability and good accuracy. For applications involving high stakes decisions, it is desirable to produce models that are as sparse

as possible without losing accuracy. In this work and especially the variable selection experiments, we push the limit of sparsity-accuracy tradeoff curve for this model class.

C Experimental Setup Details

C.1 Computing Platforms

All experiments were run on the Intel(R) Xeon(R) CPU E5-2680 v3 Processor, 2.50GHz. We set the memory limit to be 100GB.

C.2 Datasets, Baselines, and Licenses

We have a summary of datasets for experiments in Table 1.

Synthetic Data Generation Process The synthetic data used in the paper is generated according to the following process, similar to [71]:

Firstly, from a Gaussian distribution N ( 0 , Σ) where the first entry is the mean and the second entry is the covariance matrix with size p × p , we sample features:

The covariance matrix is defined as Σ jl = ρ | j -l | , where ρ ∈ (0 , 1] is the correlation parameter. When ρ is large, the features in x i are more correlated. We create a k-sparse coefficient vector β ∗ ∈ R p . The entries of β ∗ are either 1 or 0 . If j mod ( p/k ) = 0 , then β ∗ j = 1 ; otherwise, β ∗ j = 0 .

Secondly, we generate the death time t i according to the following equation:

where V i ∼ U (0 , 1) (samples are drawn from a uniform distribution on the interval [0 , 1] ) and s is a hyperparameter. In our experiments, we set s = 0 . 1 .

Lastly, we generate the censoring time, the censoring indicator, and change the death time to observation time. We sample the censoring time from a uniform distribution: C i ∼ U (0 , 1) . If the death time is bigger than the censoring time, we have the indicator equal to 1; otherwise, we have the indicator equal to 0. Specifically, we have:

δ i = ✶ t i >C i (30) Afterwards, we change the death time to observation time, taking into consideration of censoring:

We form a triplet ( x i , t i , δ i ) and return this triplet as one sample.

Real-world survival data:

Flchain : Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population [10]. The event is death.

Kickstarter1 : Data from a popular crowdfunding platform, used to predict project success [46]. We used the version from https://dmkd.cs.vt.edu/projects/survival/ data/ .

Dialysis : Data from a survival study of dialysis patients, which aims to assess quality of renal replacement therapy at dialysis centers in Rio de Janeiro, Brazil [60].

EmployeeAttrition : The task of predicting when an IBM employee will quit. The event is an IBM employee's leaving [30].

Licenses We list the licenses of the software packages used in this paper:

Abess : The license of this package is GPL-3.

skglm : The license of this package is BSD-3.

Scikit-survival (SkSurv) : The license of this package is GPL-3.

Flchain : We use the dataset from the Scikit-Survival [56] package. The GitHub link to this dataset is https://github.com/sebp/scikit-survival/tree/master/ sksurv/datasets/data . The license of this package is GPL-3.

Kickstarter1 : We use the dataset from the Virginia Tech. The link to this dataset is https://dmkd.cs.vt.edu/projects/survival/data/ . There is no license associated with this dataset. This means we cannot modify any part of the dataset, which we have obeyed while doing experiments on this dataset.

Dialysis : We use the dataset from the SurvSet [12] package. The GitHub link to this dataset is https://raw.githubusercontent.com/ErikinBC/SurvSet/main/ SurvSet/_datagen/output/Dialysis.csv . The license of this package is GPL-3.

EmployeeAttrition : We use the dataset from the PySurvival [17] package. The GitHub link to this dataset is https://github.com/square/pysurvival/blob/master/ pysurvival/datasets/employee_attrition.csv . The license of this package is Apache-2.

Baselines We compared our method against various survival models:

Abess: Adaptive Best-Subset Selection (ABESS) algorithm [71] for Cox proportional hazards model. We used the Cox model in abess python package Version 0.4.6.

SkglmALassoCox : Cox model with the adaptive Lasso regularization [69]. We used the implementation from skglm [2, 51].

SksurvCoxnet : Cox's proportional hazard's model with elastic net penalty [62]. We used the implementation from Scikit-survival (SkSurv): scikit-survival version-0.20.0 ( https: //scikit-survival.readthedocs.io/en/stable/index.html ).

SksurvTree : A greedy decision tree model using log-rank splitting rule [43]. We used the implementation from sksurv.

SksurvRSF : Random survival forest [37] algorithm. We used the implementation from Sksurv.

SksurvGBST : Gradient-boosted Cox proportional hazards loss with regression trees as base learner. In each stage, a regression tree is fit on the negative gradient of the loss function. We used the implementation from Sksurv.

SksurvNaiveSVM: Naive version of linear Survival Support Vector Machine [65]. We used the implementation from Sksurv.

SksurvFastSVM : Efficient Training of linear Survival Support Vector Machine [57]. We used the implementation from Sksurv.

Evaluation Metrics

CIndex : The full name of this metric score is Harrell's Concordance Indices [26]. It is used to evaluate the discrimination ability of a survival model. It assesses how well the model ranks observations based on their predicted risk of experiencing an event (e.g., death, disease recurrence) over time. The higher the CIndex score, the better the model.

IBS : The Integrated Brier score was proposed by [24] to evaluate survival models across all possible time threshold. The IBS score takes the Brier score a step further by integrating it across all possible time points within the follow-up period of interest. This provides a single score summarizing the model's performance over the entire time range. The lower the IBS score, the better the model.

F1-score, Precision, Recall : Suppose the true coefficients are β ∗ and the estimated coefficients are ˆ β . Then the precision score can be calculated as P = | supp ( β ∗ ) ∪ supp ( ˆ β ) | / | supp ( ˆ β ) | , where supp ( · ) extracts the support (indices whose coefficients are nonzero) of the input vector. The recall score can be calculated as R = | supp ( β ∗ ) ∪ supp ( ˆ β ) | / | supp ( β ∗ ) | . We calculate the F1 score as F1 = 2 PR/ ( P + R ) .

C.3 Details about Variable Selection Experiments

Collection and Setup: We ran 5-fold cross-validation (random seed 0) on the following datasets: Dialysis, Flchain, Kickstarter1, EmployeeAttrition, SyntheticHighCorrHighDim1, SyntheticHighCorrHighDim2, SyntheticHighCorrHighDim3. In order to create highly correlated features, we encoded continuous features into binarized features, by considering 1000 quantiles for each continuous column. For each dataset, we ran algorithms with different configurations and evaluated fitted models with metrics described in Appendix C.2:

Abess: We ran this algorithm with 30 different configurations: support size, k , ranging from 1 to 30, forcing the number of non-zero coefficients in the Cox model to be exact k . We set primary _ model _ fit _ max _ iter to be 20, approximate _ Newton to be False. All other parameters were set to the default.

SksurvCoxnet: We ran this algorithm with 30 different configurations: support size, k , ranging from 1 to 30, forcing the number of non-zero coefficients in the Cox model to be exact k . We set l 1 _ ratio to be 1.0, alpha _ min _ ratio to be 0.01. All other parameters were set to the default.

SkglmALassoCox : We ran this algorithm with 9 different L1 regularization penalty parameters (alpha): 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100. All other parameters were set to the default.

SksurvTree : We ran this algorithm with 8 different configurations: max depth limit, d , ranging from 2 to 9, and a corresponding maximum leaf limit 2 d . The random state was set to 2024 and all other parameters were set to the default.

SksurvRSF, SksurvGBST : We ran this algorithm with 8 × 5 configurations: max depth limit, d , ranging from 2 to 9, and 5 different total numbers of estimators (10, 50, 100, 500, 100). The random state was set to 2024 and all other parameters were set to the default.

SksurvNaiveSVM, SksurvFastSVM : We ran this algorithm with 9 different ℓ 2 regularization penalty parameters (alpha): 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100. All other parameters were set to the default.

SksurvCoxPHBeamSearch (our method) : We ran this algorithm with 30 different configurations: support size, k , ranging from 1 to 30, forcing the number of non-zero coefficients in the Cox model to be exact k .

Recording experimental results: For each method with specific configuration, we have a set of up to 5 fitted models on each dataset. Some metrics may be unavailable:

Precision, recall, and f1-score are not available on real-world data as we do not know the true coeffcients.

The losses on the training and testing folds of cox models are not applied to non-Cox models.

SksurvNaiveSVM and SksurvFastSVM were not able to provide IBS and AUC.

Some methods' training time exceeded our 3-hour time limit (We noticed that sksurvNaiveSVM often timed out).

For Cox and SVM models, we recorded the number of non-zero coefficients as the support size. For tree based models we recorded the number of nodes as the support size. We plotted the standard deviation of support size and various metric scores as corresponding error bars.

D Additional Results

D.1 Optimization on ℓ 1 and ℓ 1 + ℓ 2 -regularized Problems

D.1.1 Results on Flchain

Loss vsIters

Loss vs Time

Dataset:Flchain

Dataset: Flchain

105

105

losses

104

104

0

5

10

15

20

0

1

2

3

4

5

6

iters

elapsed_times (sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vsIters

Loss vs Time

Dataset: Flchain

Dataset: Flchain

600000

600000

500000

500000

400000

400000

sses

S

300000

300000

S

200000

200000

100000

100000

0

0

0

2

4

6

8

10

12

14

0

5

10

15

20

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vsIters

Loss vs Time

Dataset:Flchain

Dataset:Flchain

105

105

losses

104

104.

0

5

10

15

20

0

1

2

3

4

5

6

7

8

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

LossvsIters

Loss vs Time

Dataset:Flchain

Dataset:Flchain

15000

15000

14500

14500

14000

14000

sses

S

S

13500

13500

13000

13000

12500

12500

12000

12000

0

2

4

6

8

10

12

14

0

2

4

6

8

10

12

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

D.1.2 Results on Employee Attrition

Loss vsIters

Loss vs Time

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

24500

24500

24000

24000

23500

23500

23000

23000

22500

22500

22000

22000

21500

21500

0

2

4

6

8

10

12

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vs Iters

Loss vs Time

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

24500

24500

24000

24000

23500

23500

3

S

23000

23000

22500

22500

22000

22000

21500

21500

0

2

4

6

8

10

12

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vs Iters

Loss vs Time

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

24500

24500

24000

24000

23500

23500

S

23000

23000

22500

22500

22000

22000

21500

21500

0

2

4

6

8

10

12

0

1

2

3

4

5

6

7

8

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

LossvsIters

Loss vs Time

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

24500

24500

24000

24000

23500

23500

S

23000

23000

22500

22500

22000

22000

21500

21500

0

2

4

6

8

10

12

0

1

2

3

4

5

6

7

8

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

D.1.3 Results on Kickstarter1

Loss vs Iters

Loss vs Time

Dataset:kickstarterl

Dataset:kickstarterl

105

105

losses

104

104

0

5

10

15

20

0

20

40

60

80

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vsIters

Loss vs Time

Dataset:kickstarterl

Dataset:kickstarterl

105

105

losses

104

104

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

0

20

40

60

80

100

120

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

LossvsIters

Loss vs Time

Dataset:kickstarterl

Dataset:kickstarterl

6 × 104

6 ×104

4 × 104

4 × 104

losses

S

3 ×104

3 × 104

2 × 104

2 ×104

104

104

0.0

2.5

5.0

7.5

10.0

12.515.0

17.5

0

20

40

60

80

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vs Iters

Loss vs Time

Dataset:kickstarterl

Dataset:kickstarterl

300000

300000

250000

250000

200000

200000

loss

S

150000

150000

100000

100000

50000

50000

0

0

0

2

4

6

8

10

12

14

16

0

20

40

60

80

100

120

iters

elapsed_times (sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

D.1.4 Results on Dialysis

Loss vsIters

Loss vs Time

Dataset: Dialysis

Dataset:Dialysis

10300

10300

10200

10200

losses

S

10100

10100

10000

10000

9900

9900

9800

9800

0

2

4

6

8

10

0

1

2

3

4

5

6

iters

elapsed_times (sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vs Iters

Loss vs Time

Dataset:Dialysis

Dataset:Dialysis

10300

10300

10200

10200

losses

10100

10100

10000

10000

9900

9900

0

2

4

6

8

0.0

0.5

1.0

1.5

2.0

2.5

3.0

iters

elapsed_times(sec)

NewtonExact

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vsIters

Loss vs Time

Dataset: Dialysis

Dataset: Dialysis

10300

10300

10200

10200

losses

10100

10100

10000

10000

9900

9900

0

2

4

6

8

10

0.0

0.5

1.0

1.5

2.0

2.5

iters

elapsed_times(sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

Loss vsIters

Loss vs Time

Dataset: Dialysis

Dataset: Dialysis

10300

10300

10250

10250

10200

10200

S

10150

10150

10100

10100

10050

10050

10000

10000

9950

9950

0

1

2

3

4

5

6

7

8

0.0

0.5

1.0

1.5

2.0

2.5

iters

elapsed_times(sec)

NewtonProx

NewtonQuasi

QuadraticSurrogate(Ours)

CubicSurrogate(Ours)

D.2 Variable Selection for the CPH Model

D.2.1 Results on Dialysis

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:Dialysis

Dataset:Dialysis

0.75

0.75

cindex

0.70

ndex

0.70

cin

0.65

0.65

trai

0.60

0.60

0.55

0

10

20

30

0

10

20

30

Support Size

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:Dialysis

Dataset:Dialysis

1.0

0.75

0.9

cindex

_cindex

0.70

0.8

train_

test_

0.65

0.7

0.60

0.6

101

103

105

101

103

105

Support Size

Support Size

sksurvTree

★

sksurvRSF

★

sksurvGBST

★

sksurvNaiveSVM

★

sksurvFastSVM

BeamSearch(Ours)

Support Size vs train_ibs

Support Size vs test_ibs

Dataset: Dialysis

Dataset: Dialysis

0.19

0.19

0.18

0.18

ibs

0.17

S

0.17

t

pi

0.16

0.16

0.15

0.15

0.14

0

10

20

30

0

10

20

30

Support Size

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

Support Size vs train_ibs

Support Size vs test_ibs

Dataset:Dialysis

Dataset: Dialysis

★

★

0.20

0.15

ibs

test_ibs

0.18

train

0.10

0.16

0.05

0.14

101

103

105

101

103

105

Support Size

Support Size

sksurvTree

sksurvRSF

sksurvGBST

BeamSearch(Ours)

Support Size vs train_loss

Support Size vs test_loss

Dataset:Dialysis

Dataset:Dialysis

1.65

1.90

1.88

1.60

loss

loss

1.86

1.55

1.84

test

pi

1.50

1.82

1.80

1.45

0

10

20

30

0

10

20

30

SupportSize

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

D.2.2 Results on EmployeeAttrition

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

0.90

0.90

_cindex

test_cindex

0.85

0.85

train_

0.80

0.80

0.75

0

10

20

30

0

10

20

30

SupportSize

SupportSize

skglmALassoCox

abess

sksurvCoxnet

BeamSearch (Ours)

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

0.95

0.95

cindex

0.90

cindex

0.90

train_

0.85

0.85

0.80

0.80

0.75

101

103

105

101

103

105

Support Size

Support Size

sksurvTree

★

sksurvRSF

★

sksurvGBST

★

sksurvNaiveSVM

★

sksurvFastSVM

BeamSearch(Ours)

Support Size vs train_ibs

Support Size vs test_ibs

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

0.200

0.175

0.175

0.150

S

0.150

=

0.125

0.125

trai

0.100

0.100

0.075

0.075

0

10

20

30

0

10

20

30

Support Size

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

SupportSizevstrain_ibs

Support Size vs test_ibs

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

0.20

0.15

0.15

sq!

=

0.10

test_

trai

0.10

0.05

0.05

101

103

105

101

103

105

Support Size

SupportSize

sksurvTree

★

sksurvRSF

sksurvGBST

BeamSearch(Ours)

SupportSizevstest_loss

Dataset:EmployeeAttrition

Dataset:EmployeeAttrition

1.95

1.90

1.6

SSOI

test_loss

train_

1.85

1.5

1.80

1.75

1.4

0

10

20

30

0

10

20

30

SupportSize

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

D.2.3 Results on Kickstarter1

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:kickstarterl

Dataset:kickstarterl

1.0

1.0

cindex

0.9

_cindex

0.9

0.8

0.8

pi

0.7

0.7

0.6

0.6

0

20

40

60

80

0

20

40

60

80

Support Size

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

Support Size vs train_cindex

Support Size vs test_cindex

Dataset:kickstarterl

Dataset:kickstarterl

1.0

0.95

**★

0.90

train_cindex

_cindex

0.9

0.85

0.80

0.8

0.75

101

103

105

101

103

105

Support Size

Support Size

sksurvTree

★

sksurvRSF

★

sksurvGBST

★

sksurvNaiveSVM

★

sksurvFastSVM

BeamSearch(Ours)

Support Size vs train_ibs

Support Size vs test_ibs

Dataset:kickstarterl

Dataset:kickstarterl

0.4

0.4

0.3

0.3

_ibs

_ibs

train_

0.2

test_

0.2

0.1

0.1

0

20

40

60

80

0

20

40

60

80

Support Size

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

Support Size vs test_ibs

Dataset: kickstarterl

Dataset:kickstarterl

0.15

0.150

0.125

0.10

S

S

0.100

0.075

0.05

0.050

★★★

0.025

101

103

105

101

103

105

Support Size

Support Size

sksurvTree

★

sksurvRSF

sksurvGBST

BeamSearch(Ours)

Support Size vs train_loss

Support Size vs test_loss

Dataset:kickstarterl

Dataset:kickstarterl

4.0

2.8

_loss

3.5

test_loss

2.6

train_

2.4

3.0

2.2

2.5

2.0

0

20

40

60

80

0

20

40

60

80

SupportSize

Support Size

skglmALassoCox

abess

sksurvCoxnet

BeamSearch(Ours)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: The abstract and intro clearly state the contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed above the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] ,

Justification: All proofs are provided in the appendix. All assumptions are provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental setups are clearly stated in the Appendix. Code for the method and paper's experiments is included in a public GitHub repository.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All baselines and real-world datasets are publicly accessible. The synthetic data generation process is clearly stated in the Appendix. Code for the method and paper's experiments is included in a public GitHub repository.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This is described in the experimental section and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: Error bars are present.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This was reported in the manuscript. See Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ? [Yes]

Justification: The paper follows the full code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper involves survival analysis. We have never heard of survival analysis having negative societal impacts. It has positive impacts, particularly in medicine and reliability analysis for equipment/machinery.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper develops its own assets. Other works (datasets and baseline software packages) are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer:[Yes]

Justification: Documentation is provided with our code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

[NA] .

Justification: This is not relevant to us.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an alternative optimization method for the Cox proportional hazards model. They derive quadratic and cubic upper bounds on the loss and minimize these upper bounds with respect to a single model parameter at a time (similar to coordinate descent) via explicit formulas. They test their method on standard survival analysis benchmarks and also apply it to a feature selection problem with highly correlated features.\n\nSTRENGTHS:\nThe topic addressed by the paper (survival analysis with large datasets/high dimensionality) is relevant to the community. The prose of the paper was generally easy to follow. The background information provided on survival analysis should also make it easier for a non-expert to understand the paper.\n\nWEAKNESSES:\nI do not believe the paper makes a significant contribution to the literature.\n\nThe runtime reduction results for computing derivatives of the partial likelihood are trivial. The example given on lines 147-148, showing a \"surprising\" improvement from $O(n^2)$ to $O(n)$ computation time, seems to use a deliberately inefficient method for computing derivatives is a point of comparison: there is no need to compute the full Hessian (which is where the authors derive the $O(n^2)$ runtime) when we only want some diagonal terms, and indeed one would expect computing derivatives of a function of the form $\\sum_{i=1}^n f_i(x)$ to take $O(n)$ time provided the derivatives of each function in the summation can be computed in constant time. To that end, the only optimization over a naive implementation is the use of partial sums to avoid repeated calculation of sums with $O(n)$ terms, which is a very standard technique.\n\nThe most important problem is that the experimental results on runtime improvements are misleading. The Flchain dataset is a standard benchmark in survival analysis. It contains 7874 datapoints with 9 features, 4 of which are numeric. After standard 1-hot encoding of the categorical features, the features become 38-dimensional. Requiring 10-20 seconds for convergence (Fig. 1, rightmost plot) on such a modestly-sized dataset does not seem reasonable.\n\nI tested this myself using the scikit-survival (sksurv) library. Sksurv implements Newton's method for fitting the Cox model with an L2 penalty, as can be seen from their source code: https://github.com/sebp/scikit-survival/blob/v0.22.2/sksurv/linear_model/coxph.py#L189\n\nI ran the following code based on the sksurv documentation:\n\n```\nfrom sksurv.datasets import load_flchain\nfrom sksurv.linear_model import CoxPHSurvivalAnalysis\nfrom sksurv.preprocessing import OneHotEncoder\nfrom sklearn.impute import SimpleImputer\n\nX, y = load_flchain() # Load the data\nXt = OneHotEncoder().fit_transform(X) # One-hot encode categorical variables\nimputer = SimpleImputer().fit(Xt) # Dataset contains missing values, impute with mean\nXt = imputer.transform(Xt)\n\ncph = CoxPHSurvivalAnalysis(alpha=1.0, verbose=1) # Fit the model with regularization lambda_2 = 1 as in Fig. 1\ncph.fit(Xt, y)\n```\n\nThe fitting procedure converged for me on my laptop (Apple M2 Pro chip) in a Jupyter notebook in 9 iterations and 0.6s wall-clock time. I also had no problems getting the model fit to converge even with no regularization, contrary to the claim that \"the losses blow up when regularization is weak\" for Newton's method (caption of Fig. 1a). Thus, the apparent gains in performance are most likely due to poor implementations of Newton's method/other baselines.\n\nIn addition to these major issues, there are several other problems:\n1. In equation (6), $\\ell$ had been defined as a function of a $p$-dimensional vector $\\beta$, so it is not clear what is meant by making it a function of the $n$-dimensional vector $\\eta$. The reader can figure out what the authors probably meant, but this should be fixed and defined precisely.\n\n2. The probabilistic interpretations given by the authors for some of the derivative expressions have appeared in very early works on the Cox model, see e.g. Section 2 of [1].\n\n3. Equation (17) is exactly a Newton step, just in a single coordinate and with an upper bound on the second derivative instead of the second derivative itself. It's then not at all clear why we should expect this to be an improvement over Newton's method, especially for the specific cases the authors mention (e.g. high correlations among the features).\n\n4. I'm not sure what is meant by \"the analytical solution to this cubic surrogate function has not been well studied\" (lines 189-190). The cubic formula, which can be used to solve a general cubic equation, has existed for several hundred years (https://en.wikipedia.org/wiki/Cubic_equation).\n\n5. Lines 191-192: The claim that minimizing a convex upper bound of the original function will lead to a decrease in the original function as well is incorrect, barring some important additional assumptions. In particular, I don't believe the authors have justified their claim that the proposed method will \"ensure monotonic loss decrease and global convergence\".\n\n**References:**\n\n[1] Schoenfeld, David. Partial Residuals for the Proportional Hazards Regression Model. Biometrika (1982).\n\nQUESTIONS:\nIs there a mathematical justification for why we should expect the method to be robust to highly correlated features?\n\nPlease also address the concerns listed in the Weaknesses section.\n\nLIMITATIONS:\nLimitations are briefly discussed (lines 278-282), mostly regarding limitations of the Cox model itself.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-17"}

comments: SUMMARY:
This paper explores the optimisation of the Cox model. Through careful mathematical analysis, the authors identified efficient ways to calculate the exact derivatives and surrogate loss functions necessary for efficient optimisation, addressing existing strategies' imprecision and time limitations.

STRENGTHS:
**Clarity**   
Despite not being familiar with the literature on optimising the Cox model, the problem is well-described, and the mathematical expressions are rigorous and well-detailed.

**Relevance**
The Cox model remains a pillar in multiple fields; ensuring its stability and quick convergence is a significant contribution.

WEAKNESSES:
Excellent paper, only a few minor typos and questions, see below.

QUESTIONS:
- It would be beneficial to add a pointer on the proof of convexity (line 82) of the nll.
- Doesn't the cumulative reverse sum require times to be sorted, i.e. an initial O(n log(n)) computation?
- Typo Line 195 - 'wise' should be 'wide'.

- In Appendix A1.1.1, the 3rd equation, should have for second element $\eta_i$ not $\eta_j$; then no difference between 4th and 5th

LIMITATIONS:
As an optimization paper, the authors mention that the work's limitations are the same as those of the standard Cox model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new optimization algorithm for the Cox model, which is a classical algorithm for survival analysis presented in 1972.\n\nSTRENGTHS:\nThis paper is well-written.  I am not an expert on optimization algorithms for convex functions, but I think I could understand the proposed algorithm and I enjoyed reading this paper.\n\nThe proposed algorithm converges faster than other methods due to the new idea of exploiting high order derivatives (up to the third order derivatives) in the Cox model, and it utilizes the surrogate functions for the second and third derivatives.  The proposed algorithm is reasonable and the experimental results are convincing.\n\nWEAKNESSES:\nAs a researcher in the field of survival analysis, I would like to note that the impact of this paper is moderate within the field of survival analysis. As mentioned on page 9 of this paper, the Cox model is based on a strong assumption, the proportional hazard assumption, which is unlikely to hold in practice. Therefore, many state-of-the-art models have been developed using much weaker assumptions, and the Cox model is becoming obsolete. However, I would also like to note that the Cox model still shows a certain level of popularity thanks to its simplicity and interpretability.\n\nMisc:\n- Line 52: we are -> ours are\n- Lines 93-94: the formula for quasi Newton method can be simplified by using the diag() function as in the formula for the proximal Newton method.\n- Line 116: result result\n- Line 124: a comma is missing\n- Line 195: wise -> wide\n- Line 257: it is better to clarify which correlation coefficient is used: Pearson, Spearman, or any other one?\n- Page 15: nothing is changed in the statement \"move the partial derivative operator\"\n- Lines 539-540: Redundant vertical lines in the inequality.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nDiscussed in page 9.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an optimization of the Cox proportional hazards model based on minimizing surrogate functions obtained from exploiting the Lipschitz continuity property of the first and second order partial derivatives of the loss wrt coefficients. The authors show that the optimization works for sparse penalties or constrained problems like using a cardinality constraint. The authors conduct experiments to show the efficiency and stability of the proposed optimization procedure and empirically show high performance against CoxPH with Newton-Raphson as well as non-linear predictors.\n\nSTRENGTHS:\n- The size of modern data sets wrt n and p can be a limiting factor for applying the CoxPH model, so innovation in optimization strategy is needed\n- Strongly correlated variables can be an issue for solving the CoxPH model due to colinearity\n- The method works well compared to standard CoxPH empirircally\n- Introducing beamsearch for l0 regularization and covariate selection due to a well behaved loss function leads to interesting performance improvements for large data sets and prevents overfitting\n\nWEAKNESSES:\n- Comparison to an O(n) gradient descent framework like fastCPH (Yang et al) and BigSurvSGD would be helpful in the experiment section (including the lassonet implementation) to substantiate the claim that it is slow (in runtime) compared the proposed procedure and compare stability.\n- When comparing performances wrt survival metrics (c-index, ibs), including the lasso version of the optimization model would be helpful to understand the exact difference between standard lasso cox and the proposed optimization\n- A discussion of the tightness of the bound seems necessary when optimizing the bound instead of the loss function\n\nQUESTIONS:\n- With the proposed method, is it possible to obtain the variance-covariance matrix to construct confidence intervals for the coefficients? If so, the calibration of these would be interesting to look at.\n- Do I understand it correctly that the survival times have to be sorted for the first and second derivative to be O(n) in the current implementation? Would that imply that the complexity of the algorithm is O(n log n)?\n- In future outlook time-varying features are named as an application, does this amount to just changing the risk set in the derivatives as is the case for the loss function?\n\nLIMITATIONS:\nThe authors have addressed some of the limitations of the model.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}