Title: Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretable Machine Learning

Generalized Additive Models

Concurvity

Multicollinearity

Regularization

Time-Series Forecasting

Interpretability
Abstract: Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability, which arises from expressing the target value as a sum of non-linear transformations of the features. Despite the current enthusiasm for GAMs, their susceptibility to concurvity — i.e., (possibly non-linear) dependencies between the features — has hitherto been largely overlooked. Here, we demonstrate how concurvity can severly impair the interpretability of GAMs and propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables. This procedure is applicable to any differentiable additive model, such as Neural Additive Models or NeuralProphet, and enhances interpretability by eliminating ambiguities due to self-canceling feature contributions. 
We validate the effectiveness of our regularizer in experiments on synthetic as well as real-world datasets for time-series and tabular data. Our experiments show that concurvity in GAMs can be reduced without significantly compromising prediction quality, improving interpretability and reducing variance in the feature importances.

Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models

Abstract

Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability, which arises from expressing the target value as a sum of non-linear transformations of the features. Despite the current enthusiasm for GAMs, their susceptibility to concurvity - i.e., (possibly nonlinear) dependencies between the features - has hitherto been largely overlooked. Here, we demonstrate how concurvity can severely impair the interpretability of GAMs and propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables. This procedure is applicable to any differentiable additive model, such as Neural Additive Models or NeuralProphet, and enhances interpretability by eliminating ambiguities due to self-canceling feature contributions. We validate the effectiveness of our regularizer in experiments on synthetic as well as real-world datasets for time series and tabular data. Our experiments show that concurvity in GAMs can be reduced without significantly compromising their prediction quality, improving interpretability and reducing variance in the feature importances. 1

1 Introduction

Interpretability has emerged as a critical requirement of machine learning models in safety-critical decision-making processes and applications subject to regulatory scrutiny. Its importance is particularly underscored by the need to ensure fairness, unbiasedness, accountability and transparency in

4

Task:

Fit periodic timeseries with

daily and weekly seasonality

RMSE

= 3.85 · 10

-

1

Corr weekly

,

daily

=

-

6.0 · 10

(

)

-

10

Many

Fourier terms

w/ concurvity regularization

RMSE

= 4.68 · 10

-

3

Corr weekly

,

daily

=

-

0.64

(

)

RMSE

= 4.37 · 10

-

3

Corr weekly

,

daily

= 5.2 · 10

(

)

-

Weekly

Component

Prediction

=

+

=

+

Daily

Component

=

+

Many

Fourier terms

Few

Fourier terms

(NeuralProphet defaults)

✗

✓

✗

✓

✓

✓

Fri

many applications and domains [6, 42], such as medical diagnoses [10], loan approvals [4], and hiring practices [16]. In such cases, model interpretability can even be favored over prediction accuracy [31].

Apopular model class for interpretable machine learning is Generalized Additive Models ( GAMs ) [24], in which the target variable is expressed as a sum of non-linearly transformed features. GAMs combine the interpretability of (generalized) linear models with the flexibility of capturing non-linear dependencies between the features and the target. GAMs have recently seen a resurgence in interest with prominent examples being Neural Additive Models ( NAMs ) [2] and its variants [11, 19, 20, 39, 50] for tabular data, as well as Prophet [45] and NeuralProphet [46] for time-series forecasting. Both domains will be further explored in our experiments.

A significant obstacle to the interpretability of additive models is the phenomenon of concurvity [9]. As a non-linear analog to multicollinearity, concurvity refers to the presence of strong correlations among the non-linearly transformed feature variables. Similarly to multicollinearity, concurvity can impair interpretability because parameter estimates become unstable when features are correlated [40], resulting in highly disparate interpretations of the data depending on the model initialization. Although this issue is known and addressed by various techniques such as variable selection [28] in traditional GAMs, it has been overlooked in more recent works. Unlike the prevalent GAM package mgcv [47], we are not aware of any differentiable GAM implementations that include concurvity metrics.

In this work, we propose a novel regularizer for reducing concurvity in GAMs by penalizing pairwise correlations of the non-linearly transformed features. Reducing concurvity improves interpretability because it promotes the isolation of feature contributions to the target by eliminating potentially correlated or self-canceling transformed feature combinations. As a result, the model becomes easier to inspect by clearly separating individual feature contributions. In addition, our regularizer encourages the model to learn more consistent feature importances across model initializations, which increases interpretability. The trade-off between increased interpretability and prediction quality will be further explored in Section 4.

Figure 1 provides a first intuition of the practical effects of concurvity and how they can be addressed by our regularizer. We use the additive time-series model NeuralProphet [46] in this example, restricted to daily and weekly seasonality components. Here, each component is modeled via periodic functions with adjustable complexity. For more details on the experiment, see Section 4.2. We find that while the default NeuralProphet parameters effectively mitigate concurvity by producing a very

simple model, they provide a worse fit to the data than the more complex models. However, if left unregularized, a more complex model is subject to strong correlation between the seasonalities, an effect visually apparent in self-canceling periodic components in the middle column of Figure 1. In contrast, when using our regularization, the seasonalities are less correlated, resulting in a clearer separation between the components. While the predictive performance of the two complex models is comparable, the regularized model is more interpretable because daily and weekly effects are clearly separated.

This idea forms the basis of our argument: Comprehending a standard GAM typically requires answers to two questions: How do the features relate to my output? How do the features interact, possibly negating their contributions? By applying concurvity regularization, we simplify the interpretation of a GAM by eliminating the need for the second question.

Our main contributions can be summarized as follows:

We showcase the susceptibility of modern GAMs to concurvity and present a revised formal definition of the concept.

We propose a concurvity regularizer applicable to any differentiable GAM.

We validate our approach experimentally on both synthetic and real-world data, investigating the trade-off between concurvity and prediction quality, as well as the impact of regularization on interpretability.

2 Background

2.1 Generalized Additive Models

Generalized Additive Models ( GAMs ) [24] form a class of statistical models that extends Generalized Linear Models [36] by incorporating non-linear transformations of each feature. Following [24], GAMs can be expressed as:

where Y = ( y 1 , . . . , y N ) ∈ R N is a vector of N observed values of a target (random) variable, X = [ X 1 , . . . , X p ] ∈ R N × p assembles the observed feature variables X i = ( x i, 1 , . . . , x i,N ) ∈ R N , and f i : R → R are univariate, continuous shape functions modeling the individual feature transformations. 4 Furthermore, β ∈ R is a learnable global offset and g : R → R is the link function that relates the (expected) target value to the feature variables, e.g., the logit function in binary classification or the identity function in linear regression. The shape functions f i precisely describe the contribution of each individual feature variable in GAMs, and can be visualized and interpreted similarly to coefficients in a linear model. This allows practitioners to fully understand the learned prediction rule and gain further insights into the underlying data.

While early GAMs primarily used splines [24] or boosted decision trees [10, 34, 35] to model f i , more recent GAMs such as Neural Additive Models ( NAMs ) [2] use multilayer perceptrons (MLPs) to fit the functions f i , benefitting from the universal approximation capacity of neural networks [15] as well as the support of automatic differentiation frameworks [1, 8, 38] and hardware acceleration. As a result, one can now solve the GAM fitting problem

by common deep learning optimization techniques such as mini-batch stochastic gradient descent (SGD). Here, L : R × R → R is a loss function and H ⊂ { ( f 1 , . . . , f p ) | f i : R → R } any function class with differentiable parameters, e.g., MLPs or periodic functions like in NeuralProphet [46].

2.2 Multicollinearity and Concurvity

Multicollinearity refers to a situation in which two or more feature variables within a linear statistical model are strongly correlated. Formally, this reads as follows: 5

Definition 2.1 (Multicollinearity) . Let X 1 , . . . , X p ∈ R N be a set of feature variables where X i = ( x i, 1 , . . . , x i,N ) ∈ R N represents N observed values. We say that X 1 , . . . , X p are (perfectly) multicollinear if there exist c 0 , c 1 , . . . , c p ∈ R , not all zero, such that c 0 + ∑ p i =1 c i X i = 0 .

According to the above definition, every suitable linear combination of features that fits a target variable, say Y ≈ d 0 + ∑ p i =1 d i X i , can be modified by adding a trivial linear combination c 0 + ∑ p i =1 c i X i = 0 , i.e., Y ≈ ( c 0 + d 0 ) + ∑ p i =1 ( c i + d i ) X i . So there exist other (possibly infinitely many) equivalent solutions with different coefficients. This can make individual effects of the features on a target variable difficult to disambiguate, impairing the interpretability of the fitted model. However, even in the absence of perfect multicollinearity, difficulties may arise. 6 For example, if two features are strongly correlated, estimating their individual contributions becomes challenging and highly sensitive to external noise. This typically results in inflated variance estimates for the linear regression coefficients [40], among other problems [17].

The notion of concurvity was originally introduced in the context of GAMs to extend multicollinearity to non-linear feature transformations [9]. In analogy with our definition of multicollinearity, we propose the following definition of concurvity:

Definition 2.2 (Concurvity) . Let X 1 , . . . , X p ∈ R N be a set of feature variables and let H ⊂ { ( f 1 , . . . , f p ) | f i : R → R } be a class of functions. We have (perfect) concurvity w.r.t. X 1 , . . . , X p and H if there exist ( g 1 , . . . , g p ) ∈ H and c 0 ∈ R such that c 0 + ∑ p i =1 g i ( X i ) = 0 with c 0 , g 1 ( X 1 ) , . . . , g N ( X N ) not all zero.

Technically, concurvity simply amounts to the collinearity of the transformed feature variables, and Definition 2.1 is recovered when H is restricted to affine linear functions. Concurvity poses analogous challenges to multicollinearity: Any non-trivial zero-combination of features can be added to a solution of GAM-Fit in Equation (1), rendering the fitted model less interpretable as each feature's contribution to the target is not immediately apparent. Finally we note that, although concurvity can be considered as a generalization of multicollinearity, neither implies the other in general, see Appendix A.2(2) for more details.

3 Concurvity Regularizer

Concurvity easily arises in highly expressive GAMs, such as NAMs, since the mutual relationships between the functions f i are not constrained while solving GAM-Fit in Equation (1). This results in a large, degenerate search space with possibly infinitely many equivalent solutions. To remedy this problem, it appears natural to constrain the function space H of (1) such that the shape functions f i do not exhibit spurious mutual dependencies. Here, our key insight is that pairwise uncorrelatedness is sufficient to rule out concurvity . Indeed, using H from Definition 2.2, let us consider the subclass

̸

where Corr( · , · ) is the Pearson correlation coefficient. It is not hard to see that concurvity w.r.t. X 1 , . . . , X p and H ⊥ is impossible, regardless of the choice of H (see Appendix A.1 for a proof). From a geometric perspective, H ⊥ imposes an orthogonality constraint on the feature vectors. The absence of concurvity follows from the fact that an orthogonal system of vectors is also linearly independent. However, it is not immediately apparent how to efficiently constrain the optimization domain of Equation (1) to H ⊥ . Therefore, we rephrase the above idea as an unconstrained optimization problem:

where R ⊥ : H× R N × p → [0 , 1] denotes our proposed concurvity regularizer :

Using the proposed regularizer, GAM-Fit ⊥ simultaneously minimizes the loss function and the distance to the decorrelation space H ⊥ . In situations where high accuracy and elimination of

concurvity cannot be achieved simultaneously, a trade-off between the two objectives occurs, with the regularization parameter λ ≥ 0 determining the relative importance of each objective. An empirical evaluation of this objective trade-off is presented in the subsequent experimental section.

Since R ⊥ is differentiable almost everywhere, GAM-Fit ⊥ in Equation (2) can be optimized with gradient descent and automatic differentiation. Additional computational costs arise from the quadratic scaling of R ⊥ in the number of additive components, although this can be efficiently addressed by parallelization (see Appendix E.3.2 for a runtime analysis). A notable difference to traditional regularizers like ℓ 1 - or ℓ 2 -penalties is the dependency of R ⊥ on the data { X i } i . As a consequence, the regularizer is also affected by the batch size and hence becomes more accurate with larger batches. It is also worth noting that our regularizer does not necessarily diminish the quality of the data fit, for instance, when the input feature variables are stochastically independent, see Appendix A.2(3).

Common spline-based concurvity metrics proposed in the literature [28, 47] are not directly applicable to other differentiable models such as NAMs or NeuralProphet. Thus, similarly to [40], we decided to report the average R ⊥ ( { f i } i , { X i } i ) ∈ [0 , 1] as a model-agnostic measure of concurvity.

4 Experimental Evaluation

In order to investigate the effectiveness of our proposed regularizer, we will conduct evaluations using both synthetic and real-world datasets, with a particular focus on the ubiquitous applications of GAMs: tabular and time-series data. For the experiments involving tabular data, we have chosen to use Neural Additive Models (NAMs), as they are differentiable and hence amenable to our regularizer. For time-series data, we investigate NeuralProphet models which contain an additive component modeling seasonality. Further elaboration on our experimental setup, including detailed specifications and parameters, can be found in Appendix C.

4.1 Toy Examples

In the following, we design and investigate two instructive toy examples to facilitate a deeper comprehension of the proposed regularizer as well as the relationship between the regularization strength λ and the corresponding model accuracy. Results for an additional toy example from [28] are presented in Appendix E.2.

Toy Example 1: Concurvity regularization with and without multicollinearity To compare the effect of concurvity regularization on model training in the presence of multicollinearity, we generate synthetic data according to the linear model Y = 1 · X 1 +0 · X 2 , where X 1 and X 2 are each drawn from a uniform distribution. We consider three different settings of input feature correlation: (1) the stochastically independent case where X 1 and X 2 are independently sampled, (2) the strongly correlated case where Corr ( X 1 , X 2 ) = 0 . 9 , and (3) the perfectly correlated case with X 1 = X 2 .

We first investigate the effect of concurvity regularization on the contribution of each feature to the target by measuring the correlation of the transformed features f 1 ( X 1 ) , f 2 ( X 2 ) with the target variable Y . The results are shown in Figure 2a. In the stochastically independent case, the NAM accurately captures the relationship between the input features and target variable regardless of the regularization setting, as observed by the high correlation for f 1 ( X 1 ) and zero correlation for f 2 ( X 2 ) with the target. This result emphasizes the minimal impact of the regularizer when features are independent (see Appendix A.2(2) for details). In the perfectly correlated case, the NAM trained without concurvity regularization displays a high correlation for both f 1 ( X 1 ) and f 2 ( X 2 ) with the target, thus using both features for its predictions. In contrast, when concurvity regularization is applied, the NAM is pushed towards approximately orthogonal f i , which encourages feature selection, as indicated by high correlation of either f 1 ( X 1 ) or f 2 ( X 2 ) with the target as there is no natural preference. Interestingly, the situation is slightly different in the strongly correlated case, as X 1 is more likely to be selected here, which means that the model has correctly identified the true predictive feature. This nicely illustrates the impact of the proposed regularization on correlated feature contributions. A more detailed visualization of the impact of regularization under perfectly correlated features can be found in Figure 8 in Appendix E.1.

Secondly, we examine the trade-off between the validation RMSE and concurvity R ⊥ in the perfectly correlated case in Figure 2b. Our findings suggest that with moderate regularization strengths λ , we can effectively eradicate concurvity without compromising the accuracy of the model. Only when

W/o regularization

W/ regularization

Corr

(

X

1

, X

2

)=1

Corr

(

X

1

, X

2

)=

.

9

Corr

(

X

1

, X

2

)=0

0

1

Corr

(

f

2(

X

2)

, Y

)

0

1

Corr

(

f

2(

X

2)

, Y

)

0

1

Corr

(

f

1(

X

1)

, Y

)

0

1

Corr

(

f

2(

X

2)

, Y

)

0

1

Corr

(

f

1(

X

1)

, Y

)

(a)

0.0

0.2

0.4

0.6

0.8

Val. RMSE

0.0

0.2

0.4

0.6

0.8

1.0

Val.

R

Individual seeds

Seed average

10

6

10

4

10

2

10

0

(b)

0.0000 0.0005 0.0010 0.0015

f

1(

X

1)

1

0

f

2(

X

2)

W/o regularization

(Corr

=0

.

92

)

0.01

0.00

0.01

f

1(

X

1)

W/ regularization

(Corr

=

-

0

.

03

)

(a)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Val. RMSE

0.0

0.2

0.4

0.6

0.8

Val.

R

Individual seeds

Seed average

10

6

10

4

10

2

10

0

(b)

the regularization strength is considerably high, the RMSE is adversely affected, without any further reduction in the measured concurvity.

Toy Example 2: Concurvity regularization in the case of non-linearly dependent features Next, we evaluate our regularizer in the case of non-linear relationships between features, a setting to which it is equally applicable. To this end, we design an experiment with feature variables that are uncorrelated, but not stochastically independent due to a non-linear relationship. We choose X 1 = Z and X 2 = | Z | where Z is a standard Gaussian, and let Y = X 2 be the target variable. In this case, there is no multicollinearity by design, but the model may still learn perfectly correlated feature transformations. For example, a NAM could learn f 1 = | · | and f 2 = id , then f 1 ( X 1 ) = f 2 ( X 2 ) , which are fully correlated, yet providing a perfect fit. For our experiment, we use the same NAM model configuration as in the previous toy example.

The transformed features for the NAM fitted with and without regularization are visualized in Figure 3a. We find that the regularizer has effectively led the NAM to learn decorrelated feature transformations f 1 and f 2 , reflected in the striking difference in feature correlation ( Corr = -0 . 03 for the regularized NAM compared to Corr = 0 . 92 for the unregularized model). Moreover, these results suggest that the regularized model seems to have learned the relationship between the features, where f 2 ( X 2 ) seems to approximate | f 1 ( X 1 ) | .

Finally, a trade-off curve of the validation RMSE and R ⊥ is shown in Figure 3b, illustrating that even in the case of non-linearly dependent features, our proposed regularizer effectively mitigates the measured concurvity R ⊥ with minimal impact on the model's accuracy as measured by the RMSE.

4.2 Time-Series Data

In this section, we provide more context and additional results for the motivational example in Figure 1 on time-series forecasting using NeuralProphet [46], which decomposes a time-series into various additive components such as seasonality or trend. In NeuralProphet, each seasonality S p is modeled using periodic functions as

where k denotes the number of Fourier terms, p the periodicity, and a j , b j are the trainable parameters of the model. In the synthetic example of Figure 1, we restrict the NeuralProphet model to two components, namely a weekly and daily seasonality. Our overall reduced model is therefore given by ˆ y t = S 24h ( t ) + S 7d ( t ) .

If k is sufficiently large, it can cause the frequency ranges of S 24h and S 7d to overlap, leading to concurvity in the model. The default values of k provided by NeuralProphet for S 24h ( k = 6 ) and S 7d ( k = 3 ) are intentionally kept low to avert such frequency overlap, as demonstrated in Figure 1 (left). However, this 'safety measure' comes at the cost of prediction accuracy due to reduced model complexity.

Analogously to the above toy examples, we present a trade-off curve between RMSE and concurvity, averaging over 10 random initialization seeds per regularization strength λ . In this experiment, we choose

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Val. RMSE

0.05

0.10

0.15

0.20

0.25

Val.

R

Individual seeds

Seed average

10

6

10

4

10

2

10

0

k = 400 components for both daily and weekly seasonality, to allow concurvity to occur and fit the data almost exactly. Our findings are identical to the toy examples, demonstrating a steep decline in concurvity when increasing λ with only a small increase in RMSE.

Finally, we note that concurvity can often be identified by visual inspection for additive univariate time-series models as each component is a function of the same variable, see Figure 1. In contrast, on multivariate tabular data, concurvity may go unnoticed if left unmeasured and hence lead to false conclusions, as we investigate next.

4.3 Tabular Data

In our final series of experiments, we investigate the benefits of the proposed regularizer when applied to NAMs trained on real-world tabular datasets - a domain often tackled with conventional machine learning methods such as random forests or gradient boosting. We concentrate our analysis on six well-studied datasets: Boston Housing [23], California Housing [37], Adult [18], MIMICII [29], MIMIC-III [27] and Support2 [14]. These datasets were selected with the aim of covering different dataset sizes (ranging between N = 506 for Boston Housing and N = 48 , 842 for Adult) as well as target variables (binary classification for Adult, MIMIC-II & -III, and Support2, and regression for California and Boston Housing). NAMs are used throughout the evaluation, subject to a distinct hyperparameter optimization for each dataset; more details are presented in Appendix B.1. Additionally, we provide a traditional, spline-based GAM using pyGAM [43] for comparison; details on the HPO for pyGAM can be found in Appendix B.2.

First, we explore the trade-off between the concurvity measure R ⊥ and validation fit quality when employing the proposed concurvity regularization. Figure 5 displays the results for the tabular datasets, including the pyGAM baseline. It is clear that the concurvity regularizer effectively reduces the concurvity measure R ⊥ without significantly compromising the model fit quality across all considered datasets, in particular in the case of small to moderate regularization strengths. For example, on the California Housing dataset, we are able to reduce R ⊥ by almost an order of magnitude from around 0.2 to 0.05, while the validation RMSE increases by about 10% from 0.59 to 0.66. Additionally, we

4

6

8

Val. RMSE [1k $]

0.1

0.2

0.3

Val.

R

Boston Housing

0.6

0.8

1.0

Val. RMSE [100k $]

0.1

0.2

California Housing

0.30

0.35

0.40

0.45

Val. BCE

0.025

0.050

0.075

0.100

Adult

Individual seeds

Seed average

PyGAM

10

6

10

4

10

2

10

0

0.30

0.32

0.34

Val. BCE

0.01

0.02

0.03

0.04

0.05

Val.

R

MIMIC2

0.26

0.28

0.30

Val. BCE

0.02

0.03

0.04

0.05

MIMIC3

0.42

0.44

0.46

0.48

Val. BCE

0.03

0.04

0.05

0.06

Support2

10

6

10

4

10

2

10

0

observe the variation in the scale of R ⊥ across the datasets, exemplified by the Adult dataset where the transformed features are nearly decorrelated even in the unregularized case, potentially implying a reduced necessity for regularization. In practice, trade-off curves between concurvity and model accuracy can serve as a valuable tool for identifying the optimal level of regularization strength. As expected, pyGAM obtains similar levels of concurvity as the unregularized NAM at equally similar levels of model performance. We refer to Appendix E.3.1 for more verbose trade-off curves.

Case study: California Housing Our preceding experiments demonstrated that concurvity reduction can be achieved when training NAMs on tabular data. However, the practical significance of this observation in relation to interpretability remains unclear so far. To address this, we perform a more detailed analysis of NAMs trained on the California Housing dataset (see Appendix E.3.3 for similarly detailed results on the other datasets). In the following analysis, we compare NAMs trained with and without concurvity regularization. More specifically, we evaluate λ = 0 . 1 (determined based on Figure 5) and λ = 0 . 0 both for 60 random weight initializations.

First, we assess the effect of the regularizer on the model fit, finding that regularization increases the mean test RMSE by about 10% from about 0.58 to 0.64 and slightly decreases the spread between the seeds, as shown in Figure 6d. Note that the result in the non-regularized case is on par with the original NAM evaluation [2] serving as a sanity check of our experimental setup.

Second, we juxtapose the feature correlations of non-linearly transformed features for models trained with and without regularization. The results, as displayed in Figure 6a (upper right triangular matrices), are contrasted with the raw input feature correlations (lower left triangular matrices). It is evident that without regularization, high input correlations tend to result in correlated transformed features, as seen in the left correlation matrix. Conversely, the right correlation matrix reveals that concurvity regularization effectively reduces the correlation of transformed features. This effect is especially pronounced for previously highly correlated features such as Longitude and Latitude , or Total Bedrooms and Households .

Third, we investigate how concurvity impacts the estimation of the individual feature importances, which is of key interest for interpretable models such as NAMs. Following [2], we measure the importance of feature i as 1 N ∑ N j =1 | f i ( x i,j ) -f i | where f i denotes the average of the shape function f i over the training datapoints. We visualize the distribution of feature importances over our regularized and unregularized ensembles of NAMs in Figure 6b. 7 It is apparent that feature importances tend to have a larger variance in the unregularized case compared to the regularized case, a pattern which is

Longitude

Latitude

Housing median age

Total rooms

Total bedrooms

Population

Households

Median income

Longitude

Latitude

Housing median age

Total rooms

Total bedrooms

Population

Households

Median income

W/o regularization

Longitude

Latitude

Housing median age

Total rooms

Total bedrooms

Population

Households

Median income

W/ regularization

1.0

0.5

0.0

0.5

1.0

Corr

0.0

0.1

0.2

0.3

Importance

Households

Housing median age

Latitude

Longitude

Median income

Ocean proximity

Population

Total bedrooms

Total rooms

0.0

0.1

400

800

1200

1600

Households

0.0

0.5

Contribution [100k $]

2

4

6

8

10

Median income [10k $]

0.0

0.5

1.0

0

0.1

PyGAM

1500

3000

4500

Population

0.5

0.0

0.5

0.0

0.1

Reg. strength

0.60

0.65

Test RMSE [100k $]

(c) Average and individual shape functions of 3 out of the 9 selected features. A kernel density estimate of the training data distribution is depicted on top. The results for pyGAM are shown for comparison, including 95% confidence intervals.

(d) Test RMSEs.

particularly clear for the strongly correlated features which we identified in Figure 6a. Such variance in feature importance can detrimentally impair the interpretability of the models, due to potential inconsistencies arising in absolute importance orders. However, our proposed concurvity regularizer effectively counteracts this issue, resulting in more consistent and compact feature importances across different random seeds. With regards to the varying effect of regularization on the respective features, two observations are particularly interesting: (1) Features that are mostly uncorrelated remain unaffected by the regularization - an effect we have previously seen in Toy Example 1 which can, for example, be observed in the case of the Median income feature. (2) Input correlations lead to a bi-modal distribution in the corresponding feature importance as, for example, observable in the case of the Longitude and Latitude or Total bedrooms and Households features. Similarly to Toy Example 1, we see that the regularizer encourages feature selection for correlated feature pairs.

Finally, to visualize the impact of the regularization on model interpretability in more detail, the shape functions of three features are shown in Figure 6c. Here, the features Households and Population are strongly negatively correlated (see Figure 6a) which leads to their feature contributions largely canceling each other out. This problem is effectively mitigated by the proposed regularization, revealing naturally low contributions for both features. For comparison, the contribution of the mostly non-correlated Median income feature remains virtually unchanged by the regularization. Due to the concurvity in the data, the shape functions of the pyGAM model exhibit large variance which impedes their interpretability [40]. A similar behavior can be observed for the remaining feature contributions, which are depicted in Figure 11 in Appendix E.3.3.

In a supplementary experiment in Appendix E.3.4, we compare concurvity regularization to classical L1 regularization on the California Housing dataset. We find that L1 more strongly prunes features and removes finer details of each feature's contribution, which are more gently preserved for concurvity regularization.

In summary, our case study on the California Housing dataset establishes that concurvity regularization significantly enhances interpretability and consistency of a GAM in terms of shape functions and feature importance, whilst maintaining high model accuracy.

5 Related Work

Classical works on concurvity in GAMs The term concurvity was first introduced in [9]; for a well-written introduction to concurvity, we refer the reader to [40]. Numerous subsequent works have developed techniques to address concurvity, such as improving numerical stability in spline-based GAM fitting [25, 48] and adapting Lasso regularization for GAMs [5]. Partial GAMs [21] were proposed to address concurvity through sequential maximization of Mutual Information between response variables and covariates. More recently, [28] compared several feature selection algorithms for GAMs and found algorithms selecting a larger feature set to be more susceptible to concurvity, a property first noticed in [22]. In addition, [28] proposes a novel feature selection algorithm that chooses a minimal subset to deal with concurvity. We refer to [28] for a thorough comparison of different concurvity measures. In contrast, our proposed regularizer adds no additional constraints on the feature set size and does not explicitly enforce feature sparsity.

Modern neural approaches to GAMs Recent advancements in neural approaches to GAMs, such as Neural Additive Models (NAMs) [2] and NeuralProphet [46], have provided more flexible and powerful alternatives to classical methods [24]. These have spurred interest in the subject leading to several extensions of NAMs [11, 19, 39]. Our approach is compatible with the existing methodologies and can be readily integrated if they are implemented in an automatic differentiation framework.

Regularization via decorrelation Similar types of decorrelation regularizers have previously been proposed in the machine learning literature but in different contexts. [13] found that regularizing the cross-covariance of hidden activations significantly increases generalization performance and proposed DeCov. OrthoReg [41] was proposed to regularize negatively correlated features to increase generalization performance by reducing redundancy in the network. Similarly, [49] proposes to add a regularizer enforcing orthonormal columns in weight matrices. More recent approaches, such as Barlow Twins [51], leverage decorrelation as a self-supervised learning technique to learn representations that are invariant to different transformations of the input data.

6 Conclusion

In this paper, we have introduced a differentiable concurvity regularizer, designed to mitigate the often overlooked issue of concurvity in differentiable Generalized Additive Models (GAMs). Through comprehensive empirical evaluations, we demonstrated that our regularizer effectively reduces concurvity in differentiable GAMs such as Neural Additive Models and NeuralProphet. This in turn significantly enhances the interpretability and reliability of the learned feature functions, a vital attribute in various safety-critical and strongly regulated applications. Importantly, our regularizer achieves these improvements while maintaining high prediction quality, provided it is carefully applied. We underscore that while the interpretability-accuracy trade-off is an inherent aspect of concurvity regularization, the benefits of increased interpretability and consistent feature importances across model initializations are substantial, particularly in real-world decision-making scenarios.

Nonetheless, our study is not without its limitations. The validation of our approach, while diverse, was limited to three real-world and three synthetic datasets. As such, we acknowledge that our findings may not fully generalize across all possible datasets and use cases.

An intriguing avenue for future work could be to examine the impact of our regularizer on fairness in GAMs. While prior work [12] suggests that GAMs with high feature sparsity can miss patterns in the data and be unfair to minorities, our concurvity regularizer does not directly enforce feature sparsity. Thus, a comparison between sparsity regularizers and our concurvity regularizer in unbalanced datasets would be of high interest. In addition, future work could explore how the joint optimization of concurvity and model fit could be improved by framing it as a multi-objective problem. Finally, our concurvity regularizer could be adapted to differentiable GAMs that incorporate pairwise interactions. Specifically, it would be interesting to contrast such an extension with the ANOV A decomposition proposed in [30] in terms of single and pairwise interactions.

We conclude by encouraging researchers and practitioners to 'curve your enthusiasm' - that is, to seriously consider concurvity in GAM modeling workflows. We believe this will lead to more interpretable models and hence more reliable and robust analyses, potentially avoiding false conclusions.

References

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. TensorFlow: a system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) , pages 265-283, 2016.

R. Agarwal, L. Melnick, N. Frosst, X. Zhang, B. Lengerich, R. Caruana, and G. E. Hinton. Neural Additive Models: Interpretable Machine Learning With Neural Nets. In Advances in Neural Information Processing Systems 34 , 2021.

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 2623-2631, 2019.

K. Arun, G. Ishan, and K. Sanmeet. Loan approval prediction based on machine learning approach. IOSR Journal of Computer Engineering , 18(3):18-21, 2016.

M. Avalos, Y. Grandvalet, and C. Ambroise. Regularization methods for additive models. In International Symposium on Intelligent Data Analysis , pages 509-520, 2003.

S. Barocas and A. D. Selbst. Big Data's Disparate Impact. California Law Review , pages 671-732, 2016.

J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems 24 , 2011.

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: Composable transformations of Python+NumPy programs. http://github.com/google/jax , 2018.

A. Buja, T. Hastie, and R. Tibshirani. Linear smoothers and additive models. The Annals of Statistics , pages 453-510, 1989.

R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1721-1730, 2015.

C.-H. Chang, R. Caruana, and A. Goldenberg. NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning. In International Conference on Learning Representations , 2022.

C.-H. Chang, S. Tan, B. Lengerich, A. Goldenberg, and R. Caruana. How Interpretable and Trustworthy are GAMs? In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 95-105, 2021.

M. Cogswell, F. Ahmed, R. B. Girshick, C. L. Zitnick, and D. Batra. Reducing Overfitting in Deep Networks by Decorrelating Representations. In International Conference on Learning Representations , 2016.

A. F. Connors Jr, N. V. Dawson, C. Thomas, F. E. Harrell Jr, N. Desbiens, W. J. Fulkerson, P. Kussin, P. Bellamy, L. Goldman, and W. A. Knaus. Outcomes following acute exacerbation of severe chronic obstructive lung disease. The SUPPORT investigators (Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments). American Journal of Respiratory and Critical Care Medicine , 154(4):959-967, 1996.

G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems , 2(4):303-314, 1989.

B. Dattner, T. Chamorro-Premuzic, R. Buchband, and L. Schettler. The legal and ethical implications of using AI in hiring. Harvard Business Review , 25, 2019.

C. F. Dormann, J. Elith, S. Bacher, C. Buchmann, G. Carl, G. Carré, J. R. G. Marquéz, B. Gruber, B. Lafourcade, P. J. Leitão, et al. Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography , 36(1):27-46, 2013.

D. Dua and C. Graff. UCI Machine Learning Repository, 2017.

A. Dubey, F. Radenovic, and D. Mahajan. Scalable Interpretability via Polynomials. In Advances in Neural Information Processing Systems 35 , 2022.

J. Enouen and Y. Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In Advances in Neural Information Processing Systems 35 , 2022.

H. Gu, T. Kenney, and M. Zhu. Partial Generalized Additive Models: An Information-Theoretic Approach for Dealing With Concurvity and Selecting Variables. Journal of Computational and Graphical Statistics , 19(3):531-551, 2010.

M. A. Hall. Correlation-based feature selection for machine learning . PhD thesis, The University of Waikato, 1999.

D. Harrison Jr and D. L. Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management , 5(1):81-102, 1978.

T. Hastie and R. Tibshirani. Generalized Additive Models: Some applications. Journal of the American Statistical Association , 82(398):371-386, 1987.

S. He. Generalized Additive Models for data with concurvity: Statistical issues and a novel model fitting approach . PhD thesis, University of Pittsburgh, 2004.

D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs). Preprint arXiv:1606.08415, 2016.

A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data , 3(1), 2016.

L. Kovács. Feature selection algorithms in generalized additive models under concurvity. Computational Statistics , pages 1-33, 2022.

J. Lee, D. J. Scott, M. Villarroel, G. D. Clifford, M. Saeed, and R. G. Mark. Open-Access MIMIC-II Database for Intensive Care Research. 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society , 2011:8315-8318, 2011.

B. Lengerich, S. Tan, C.-H. Chang, G. Hooker, and R. Caruana. Purifying interaction effects with the functional anova: An efficient algorithm for recovering identifiable additive models. In International Conference on Artificial Intelligence and Statistics , pages 2402-2412, 2020.

B. Letham, C. Rudin, T. H. McCormick, and D. Madigan. Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics , 9(3):1350-1371, 2015.

I. Loshchilov and F. Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations , 2017.

I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations , 2019.

Y. Lou, R. Caruana, and J. Gehrke. Intelligible Models for Classification and Regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 150-158, 2012.

Y. Lou, R. Caruana, J. Gehrke, and G. Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 623-631, 2013.

J. A. Nelder and R. W. Wedderburn. Generalized Linear Models. Journal of the Royal Statistical Society: Series A (General) , 135(3):370-384, 1972.

R. K. Pace and R. Barry. Sparse Spatial Autoregressions. Statistics & Probability Letters , 33(3):291-297, 1997.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 , 2019.

F. Radenovic, A. Dubey, and D. Mahajan. Neural Basis Models for Interpretability. In Advances in Neural Information Processing Systems 35 , 2022.

T. O. Ramsay, R. T. Burnett, and D. Krewski. The Effect of Concurvity in Generalized Additive Models Linking Mortality to Ambient Particulate Matter. Epidemiology , 14(1):18-23, 2003.

P. Rodríguez, J. Gonzàlez, G. Cucurull, J. M. Gonfaus, and X. Roca. Regularizing CNNs with Locally Constrained Decorrelations. In International Conference on Learning Representations , 2017.

C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong. Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges. Statistic Surveys , 16:1-85, 2022.

D. Servén and C. Brummitt. pyGAM: Generalized Additive Models in Python. URL: https://zenodo. org/records/1476122 , 2018.

M. Signoretto, K. Pelckmans, and J. A. Suykens. Functional ANOVA models: Convex-concave approach and concurvity analysis. Technical Report, URL: https://ftp.esat.kuleuven.be/pub/ sista/kpelckma/kp08-203.pdf , 2008.

S. J. Taylor and B. Letham. Forecasting at Scale. The American Statistician , 72(1):37-45, 2018.

O. Triebe, H. Hewamalage, P. Pilyugina, N. Laptev, C. Bergmeir, and R. Rajagopal. NeuralProphet: Explainable forecasting at scale. Preprint arXiv:2111.15397, 2021.

S. N. Wood. MGCV: GAMs and generalized ridge regression for R. R news , 1(2):20-25, 2001.

S. N. Wood. Fast stable direct fitting and smoothness selection for Generalized Additive Models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(3):495-518, 2008.

D. Xie, J. Xiong, and S. Pu. All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6176-6185, 2017.

S. Xu, Z. Bu, P. Chaudhari, and I. J. Barnett. Sparse neural additive model: Interpretable deep learning with feature selection via group sparsity. In ICLR 2022 Workshop on PAIR 2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data , 2022.

J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In International Conference on Machine Learning , pages 12310-12320, 2021.

A Additional Remarks and Theoretical Results for the Proposed Regularizer

A.1 The Decorrelation Space H ⊥ Rules Out Concurvity

In this section, we formalize our claim from Section 3 that the space H ⊥ provably rules out concurvity. In the following, ⟨· , ·⟩ and ∥ · ∥ 2 denote the Euclidean scalar product and norm, respectively. For v = ( v 1 , . . . , v N ) ∈ R N , we set ¯ v := 1 N ∑ N l =1 v l and denote the all-one vector by ✶ := (1 , . . . , 1) ∈ R N . The standard (empirical) Pearson's correlation coefficient is then given by 8

Lemma A.1. Let X 1 , . . . , X p ∈ R N be a set of feature variables with p > 1 and let H ⊂ { ( f 1 , . . . , f p ) | f i : R → R } be a class of functions. Consider the following subclass of H :

̸

Then we do not have concurvity w.r.t. X 1 , . . . , X p and H ⊥ .

Proof. Towards a contradiction, assume that there are ( g 1 , . . . , g p ) ∈ H ⊥ and c 0 ∈ R such that c 0 + ∑ p i =1 g i ( X i ) = 0 .

✶ We set v i := g i ( X i ) and trivially add the mean vectors to the linear combination:

̸

Using that Corr( v i , v j ) = 0 for i = j , we then obtain

̸

A.2 Additional Remarks on Concurvity and our Regularizer

We conclude that v j = ¯ v j ✶ , i.e., v j is a constant vector. But this contradicts the definition of H ⊥ because we would have Corr( v i , v j ) = ∞ with any i = j .

(1) The definitions of multicollinearity (Definition 2.1) and concurvity (Definition 2.2) are based on a fixed (deterministic) feature design, but one could also formulate probabilistic versions, cf. [44]. The latter typically facilitates a theoretical analysis, which, however, is not the focus of our work. Moreover, a probabilistic definition would not cover an important practical source of multicollinearity, namely underdetermined systems where N < p .

(2) Although closely related, multicollinearity does not necessarily imply concurvity and vice versa. Indeed, one can easily come up with setups where perfectly correlated features become decorrelated after a non-linear transform. Similarly, uncorrelated input features can be made perfectly correlated with a non-linearity. Two simple (toy) examples are presented in Section 4.1.

(3) Our concurvity regularizer R ⊥ does not automatically affect the predictive performance of a GAM. For example, assuming that the input features are drawn from stochastically independent random variables, we can conclude that Corr( f i ( X i ) , f j ( X j )) ≈ 0 for a large enough sample size N , since non-linear transforms of independent random variables remain independent. Consequently, we have that R ⊥ ( { f i } i , { X i } i ) ≈ 0 , so that no (in this case undesirable) regularization takes effect.

B Hyperparameter Optimization

B.1 Tabular Datasets

For hyperparameter optimization we use Tree-structured Parzen Estimator (TPE) [7] as implemented in Optuna [3]. We run the optimization for a budget of 500 function evaluations and optimize w.r.t. validation RMSE for Boston Housing and California Housing or validation binary cross entropy loss for Adult, MIMIC-2, MIMIC-3, and Support 2.

The hyperparameter space and default parameters are shown in Table 1 and the hyperparameters per dataset are shown in Figure 7.

Hyperparameter

Value

Learning Rate

1e-3

Weight Decay

0.0

Activation

GELU

# of neurons per layer

128

# of hidden layers

3

Num. Epochs

50

128

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(a) Toy Example 1&2

Hyperparameter

Value

Learning Rate

7.93e-4

Weight Decay

1.79e-2

Activation

ELU

# of neurons per layer

75

# of hidden layers

6

Num. Epochs

91

128

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(b) Boston Housing

Hyperparameter

Value

Learning Rate

9.46e-3

Weight Decay

3.73e-3

Activation

ReLU

# of neurons per layer

72

# of hidden layers

5

Num. Epochs

39

512

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(c) California Housing

Hyperparameter

Value

Learning Rate

2.64e-3

Weight Decay

1.64e-3

Activation

GELU

# of neurons per layer

204

# of hidden layers

4

Num. Epochs

200

512

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(d) Adult

Hyperparameter

Value

Learning Rate

3.31e-3

Weight Decay

1.08e-3

Activation

GELU

# of neurons per layer

190

# of hidden layers

3

Num. Epochs

20

512

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(e) MIMIC-II

Hyperparameter

Value

Learning Rate

3.88e-3

Weight Decay

1.10e-3

Activation

GELU

# of neurons per layer

168

# of hidden layers

3

Num. Epochs

23

512

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(f) MIMIC-III

Hyperparameter

Value

Learning Rate

3.42e-3

Weight Decay

1.05e-2

Activation

GELU

# of neurons per layer

127

# of hidden layers

3

Num. Epochs

30

512

Correlation Denominator

ϵ

1e-12

Start Conc. Reg. after x% of steps

5

(g) Support2

B.2 pyGAM

We optimized the second-order derivative penalties on the spline functions in pyGAM [43] using random search. We used a budget of 300 function evaluations within a search range of [1e-3, 1e3].

C Experimental Details

C.1 NAMExperiments

In all of our NAM experiments, we use the AdamW optimizer [33] and adjust our learning rate using Cosine Annealing [32] and decay to 0. For all regression problems we use the Mean Squared Error (MSE) and for all binary classfication problems the Binary Cross Entropy (BCE) as loss function L .

In all of our experiments, we apply the concurvity regularization after a warm-up phase consisting of 5% of the total optimization steps for stability reasons. After this warm-up phase, regularization is performed at each step.

C.2 Toy Examples

We sample 10,000 datapoints from the model and use 7,000, 2,000, 1,000 for training, validation and testing respectively. We use the validation split to find adequate hyperparameters via a small manual search. Our NAM has 3 layers per feature NN with 128 hidden units and uses the GeLU [26] activation function. We use no weight decay, train for 50 epochs at an initial learning rate of 1e-3 and a batch size of 128. Otherwise, the experimental setup is the same as described earlier in Section C.1.

D Dataset Details

Boston Housing The Boston Housing Dataset [23], compiled by Harrison and Rubinfeld in the 1970s, is a benchmark dataset employed in machine learning and statistical modeling for housing price prediction. Consisting of 506 neighborhoods in the Boston metropolitan area, it features 13 attributes, including crime rate, zoning information, industrial acreage, Charles River proximity, air quality, housing characteristics, accessibility to employment centers, highways, and education, as well as demographic factors. The primary objective is to predict the median value of owner-occupied homes using these features.

California Housing The California Housing Dataset [37] is a widely-used benchmark dataset for machine learning and statistical modeling, particularly in the domain of housing price prediction. Originally derived from the 1990 California Census, it consists of 20,640 samples, each representing a census block group. The dataset contains information on various housing-related attributes, such as median income, housing median age, average number of rooms, average number of bedrooms, population and average household size. It also includes the geographical location (latitude and longitude) of each block group. The objective is to predict the median house value. We obtained the dataset from [18].

Adult The Adult Dataset [18] is also known as the 'Census Income' dataset. Extracted from the 1994 United States Census Bureau data, it comprises 48,842 records, each representing an individual. The dataset contains 14 features, including age, work class, education, marital status, occupation, relationship, race, sex, capital gain, capital loss, hours worked per week, and native country. The objective is to predict whether an individual's annual income exceeds $50,000.

MIMIC-II The MIMIC-II (Multiparameter Intelligent Monitoring in Intensive Care) dataset [29] is a public database, offering clinical data from a multitude of Intensive Care Unit (ICU) patients. It is managed by the MIT Lab for Computational Physiology and encompasses a wide variety of data points, such as patient demographics, vital signs, lab test results, medications, procedures, caregiver notes, and imaging reports, as well as mortality rates both in and out of the hospital.

MIMIC-III MIMIC-III (Medical Information Mart for Intensive Care III) [27] is the successor to the MIMIC-II database. It contains additional, more recent patient records and provides more detailed data, including free-text interpretation of imaging reports, allowing for more granular research and improved application in areas like machine learning, health informatics, and predictive modeling.

SUPPORT2 The Support2 [14] (Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments) dataset is a clinical database that contains detailed medical information from a large cohort of seriously ill hospitalized adults. The dataset was created as part of a multi-center study designed to understand the outcomes of decisions made in the course of medical treatment. It provides extensive variables, including demographic data, physiological measurements, diagnostic information, treatment plans, and outcomes such as survival and quality of life. The data collected span a diverse set of medical conditions, making the SUPPORT2 dataset a valuable resource for researchers seeking to study clinical decision-making, prognosis evaluation, and healthcare outcomes.

E Additional results

E.1 Toy Example 1

Figure 8 shows an additional scatter plot comparing the contributions of the transformed features.

(a) w/o concurvity regularization

(b) w/ concurvity regularization (

λ

= 0.1)

1.0

0.5

0.0

0.5

1.0

0.01

0.00

f_0(input_0)

0.01

0.02

0.03

0.04

0.05

0.5

0.0

f1(input_1)

0.5

1.0

1.5

1.0

0.50.0

0.5

1.0

0.04

0.02

1.5

1.00.5

0.0

0.5

105.1

f_0(input_0)

f_1(input_1)

1.0

0.5

10f10

0.0

0.5

1.0

0.15

0.10

(0ndu)o]

0.05

0.00

S0°0-

0.10

0.15

0.5

f1(input_1)

0.0

0.5

1.0

-1.0

0.5

0.0

0.5

1.0

0.1

0.0

0.1

1.0

0.5

0.0

0.5

target

f_0(input_0)

f_1(input_1)

E.2 Additional Toy Example from [28]

In response to a suggestion to include a more complex toy example during the review process, we have replicated the toy example from [28] using our NAM setup. To recap, this example contains 7 input features:

where the standard deviations are sufficiently small to create severe concurvity among the features ( σ 1 = 0 . 05 , σ 2 = 0 . 5 ). We simulated 10 , 000 data points from this model and created a 7:3 train/test split. We fitted 20 random initializations of a NAM in the unregularized, concurvity-regularized, and L1 regularized setting. The regularization parameter was determined separately for each regularization type based on trade-off curves. For concurvity regularization, we used λ = 0 . 1 , and for L1, we used λ = 0 . 05 .

The results, reported as the R 2 on the test set, are reported in Table 2, with the top three rows from [28] for comparison. Confidence intervals of the mean are estimated on 10 , 000 bootstrap samples. Features are presented in descending order of their importance (as defined in the main paper) for the best-fitting model in each setting, while the actual importances are reported below.

Wenote that both concurvity regularization and L1 regularization correctly identify the three predictive features X 1 , X 5 and X 6 , on which Y directly depends. This is not the case without regularization. Furthermore, we find that concurvity regularization effectively reduces R ⊥ , unlike L1 regularization.

We also note that the R 2 values of all NAM implementations are lower than those reported in [28], which we believe is due to the inductive biases in spline-based models being particularly well-suited to the mostly polynomial-based problem. This notwithstanding, our results further underline the effectiveness of concurvity regularization.

E.3 Tabular Data

E.3.1 Verbose Trade-off Curves

In addition to the trade-off curves presented in the main text (see Figure 5), Figure 9 shows a more verbose version, separating the visualization for the measured validation concurvity and validation accuracy as a function of the regularization strength λ . This explicit visualization allows us for an easier choice of λ , for example, by determining the elbow in the validation accuracies, after which stronger regularization would lead to a significant loss of performance. For example, this can be nicely observed in the case of the Boston Housing dataset at λ ≈ 1 .

0.1

0.2

0.3

Val.

R

Boston Housing

0.1

0.2

Val.

R

California Housing

0.025

0.050

0.075

0.100

Val.

R

Adult

Individual seeds

Seed average

10

5

10

3

10

1

10

1

Reg. strength

4

6

8

Val. RMSE [1k $]

10

5

10

3

10

1

10

1

Reg. strength

0.6

0.7

0.8

0.9

1.0

Val. RMSE [100k $]

10

5

10

3

10

1

10

1

Reg. strength

0.30

0.35

0.40

0.45

Val. BCE

0.01

0.02

0.03

0.04

0.05

Val.

R

MIMIC2

0.02

0.03

0.04

0.05

Val.

R

MIMIC3

0.03

0.04

0.05

0.06

Val.

R

Support2

10

5

10

3

10

1

10

1

Reg. strength

0.30

0.32

0.34

Val. BCE

10

5

10

3

10

1

10

1

Reg. strength

0.26

0.28

0.30

Val. BCE

10

5

10

3

10

1

10

1

Reg. strength

0.44

0.46

0.48

Val. BCE

E.3.2 Runtime Analysis

Due to calculating pair-wise correlations, our proposed regularizer has a quadratic computational complexity. However, using an efficient implementation leveraging vectorization, the scaling constant can be well controlled. To investigate the computational overhead of our regularizer, we measure the mean absolute runtime across 100 repetitions for different numbers of input features and batch sizes. Furthermore, we evaluate the relative overhead of our regularizer compared to a NAM model. The relative overhead clearly depends on the size and implementation details of the NAM. Here, we compare with a NAM for which each feature-dependent network consists of a 3-layer MLP with 128 neurons each. The calculation over the features is vectorized using batched matmul s. The results are shown in Figure 10, all obtained with an M1 MacBook Pro. As expected, we observe a power-law

scaling of the runtime of the regularizer, which is in particular observable for an increasing numbers of input features. In particular, for moderate numbers of input features between 64 and 256, the absolute runtime is about 1 ms. Compared with the forward pass of a 3-layer NAM, the proposed regularizer typically implies a relative overhead of well below 10 %. Note that the results compare only with the forward pass of the NAM model and do not include data loading, batch collation, loss calculation, and backward passes. We note that especially for larger batch sizes, the relative overhead introduced by our regularizer is well below 5 %. On the other hand, for smaller batch sizes, the relative overhead can become large, due to the reduced effectiveness of vectorization. In absolute terms, however, the overhead is very small for smaller batch sizes as well. We therefore conclude that the overhead introduced by regularization is negligible in commonly considered model setups.

2

4

8

16

32

64

128

256

512

1024 2048 4096

Number of input features

10

1

10

0

10

1

10

2

Runtime in ms

Batch size

64

128

256

512

2

4

8

16

32

64

128

256

512

1024 2048 4096

Number of input features

0

10

20

30

40

50

60

Relative overhead in %

E.3.3 In-detail Results for the Remaining Tabular Datasets

In addition to the results of our case study with the California Housing dataset in Section 4.3, we here present the shape functions of the complete set of features, see Figure 11.

Furthermore, we show the analogous results for the remaining datasets Adult (Figure 12), Boston Housing (Figure 13), MIMIC-II (Figure 14), MIMIC-III (Figure 15 & 16), and Support2 (Figure 17 & 18).

E.3.4 Comparing Concurvity Regularization to L1 Regularization

Although we chose pyGAM as our baseline in the main paper, we have additionally compared our concurvity regularizer to L1 regularization, as a popular alternative to eliminating redundant features. Results on the California Housing dataset are included in Figure 19. We performed L1 regularization on the terms w i f i ( x i ) in Equation (GAM), with regularization strength λ = 0 . 1 , as determined by trade-off curves on feature L1 norms versus RMSE.

The figure suggests that L1 regularization prunes features more strongly than concurvity regularization, which may be desirable in some situations. However, this comes at a cost of a larger test RMSE, as well as some seemingly predictive features being pruned. For example, L1 regularization removes the two local peaks in the Latitude dimension corresponding to San Francisco and Los Angeles. In addition, unlike L1 regularization, concurvity regularization does not prune uncorrelated features.

Finally, we note that there may be different motivations behind choosing sparsity regularization (as few features as possible) or concurvity regularization (features become as decorrelated as possible). If the main goal is to address concurvity, then sparsity regularization is a blunt tool, and vice versa.

400

800

1200

1600

Households

0.4

0.2

0.0

0.2

0.4

0.6

Contribution [100k $]

10

20

30

40

50

Housing median age

0.0

0.1

0.2

0

0.1

PyGAM

34

36

38

40

Latitude [°]

0.6

0.4

0.2

0.0

0.2

0.4

123.0

121.5

120.0

118.5

117.0

Longitude [°]

0.50

0.25

0.00

0.25

0.50

Contribution [100k $]

2

4

6

8

10

Median income [10k $]

0.25

0.00

0.25

0.50

0.75

1.00

<1h Ocean

Inland

Island

Near Bay

Near Ocean

Ocean proximity

0.2

0.1

0.0

0.1

0.2

1500

3000

4500

Population

0.50

0.25

0.00

0.25

0.50

Contribution [100k $]

500

1000

1500

2000

Total bedrooms

0.4

0.2

0.0

0.2

0.4

0.6

2500

5000

7500

10000

Total rooms

0.15

0.10

0.05

0.00

0.05

age

fnlwgt

educational-num

capital-gain

capital-loss

hours-per-week

age

fnlwgt

educational-num

capital-gain

capital-loss

hours-per-week

W/o regularization

age

fnlwgt

educational-num

capital-gain

capital-loss

hours-per-week

W/ regularization

0.4

0.2

0.0

0.2

0.4

Corr

0

50

100

150

200

250

Importance

age

capital-gain

capital-loss

education

educational-num

fnlwgt

gender

hours-per-week

marital-status

native-country

occupation

race

relationship

workclass

0.0

1.0

30

45

60

age

1000

500

0

Log-Odds

0

4000

8000

12000

capital-gain

100

50

0

0

1.0

0

500

1000

1500

2000

capital-loss

400

200

0

0

5

10

15

education

0.5

0.0

0.5

3

6

9

12

15

educational-num

2

0

2

Log-Odds

100000

200000

300000

400000

500000

fnlwgt

0.25

0.00

0.25

0.0

0.5

1.0

gender

0.50

0.25

0.00

0.25

15

30

45

60

75

hours-per-week

1

0

0

2

4

6

marital-status

0

2

Log-Odds

0

20

native-country

5.0

2.5

0.0

0

5

10

occupation

1

0

1

1

2

3

4

race

0.4

0.2

0.0

Corr

crim

zn

indus

nox

rm

age

dis

rad

tax

ptratio

b

lstat

crim

zn

indus

nox

rm

age

dis

rad

tax

ptratio

b

lstat

W/o regularization

crim

zn

indus

nox

rm

age

dis

rad

tax

ptratio

b

lstat

W/ regularization

0.4

0.2

0.0

0.2

0.4

0.00

0.05

0.10

0.15

0.20

0.25

Importance

age

b

crim

dis

indus

lstat

nox

ptratio

rad

rm

tax

zn

0.0

1.0

20

40

60

80

100

age

0.0

0.1

Log-Odds

80

160

240

320

b

0.2

0.0

0

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.1

0.2

10

20

30

40

crim

0.3

0.2

0.1

0.0

2

4

6

8

10

dis

0.2

0.0

Log-Odds

5

10

15

20

25

indus

0.2

0.1

0.0

8

16

24

32

lstat

0.5

0.0

0.5

0.4

0.5

0.6

0.7

0.8

nox

0.2

0.0

14

16

18

20

ptratio

0.0

0.2

Log-Odds

5

10

15

20

rad

0.2

0.1

0.0

0.1

5

6

7

8

rm

0.0

0.5

1.0

200

300

400

500

600

tax

0.0

0.2

Age

GCS

SBP

HR

Temperature

PFratio

Renal

Urea

WBC

CO2

Na

K

AdmissionType

AIDS

Lymphoma

Age

GCS

SBP

HR

Temperature

PFratio

Renal

Urea

WBC

CO2

Na

K

AdmissionType

AIDS

Lymphoma

W/o regularization

Age

GCS

SBP

HR

Temperature

PFratio

Renal

Urea

WBC

CO2

Na

K

AdmissionType

AIDS

Lymphoma

W/ regularization

0.4

0.2

0.0

0.2

0.4

Corr

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Importance

AIDS

AdmissionType

Age

CO2

GCS

HR

K

Lymphoma

Na

PFratio

Renal

SBP

Temperature

Urea

WBC

0.0

1.0

0.0

0.4

0.8

1.2

1.6

2.0

AdmissionType

0.0

0.1

0.2

Log-Odds

4

8

12

16

0

1

2

0

1.0

18

21

24

27

30

CO2

0.0

0.5

1.6

2.4

3.2

4.0

4.8

GCS

0.5

0.0

0.5

Log-Odds

75

100

125

150

HR

0.0

0.5

1.0

1.5

0.0

0.2

0.4

0.6

0.8

1.0

Lymphoma

0.0

0.5

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

Log-Odds

120

126

132

138

144

Na

0.00

0.25

0.50

0.75

250

500

750

1000

PFratio

1

0

1

0

40

80

120

160

SBP

0.0

0.5

1.0

Log-Odds

3.2

4.0

4.8

5.6

Temperature

0.0

0.5

10

20

30

40

50

WBC

0.00

0.25

0.50

age

eth_asian

eth_white

admType_URGENT

sysbp_min

diasbp_max

meanbp_mean

tempc_min

spo2_max

glucose_mean

bilirubin

hematocrit

phosphate

inr

wbc

age

adult_icu

eth_hispanic

admType_ELECTIVE

admType_URGENT

heartrate_mean

sysbp_mean

diasbp_mean

meanbp_mean

resprate_mean

tempc_mean

spo2_mean

glucose_mean

bicarbonate

chloride

hemoglobin

phosphate

ptt

sodium

W/o regularization

age

eth_asian

eth_white

admType_URGENT

sysbp_min

diasbp_max

meanbp_mean

tempc_min

spo2_max

glucose_mean

bilirubin

hematocrit

phosphate

inr

wbc

W/ regularization

0.4

0.2

0.0

0.2

0.4

Corr

0.0

0.1

0.2

0.3

0.4

Importance

admType_ELECTIVE

admType_EMERGENCY

admType_NEWBORN

admType_URGENT

adult_icu

age

albumin

aniongap

bicarbonate

bilirubin

bun

chloride

creatinine

diasbp_max

diasbp_mean

diasbp_min

eth_asian

eth_black

eth_hispanic

eth_other

eth_white

first_hosp_stay

first_icu_stay

glucose

glucose_max

glucose_mean

glucose_min

heartrate_max

heartrate_mean

heartrate_min

hematocrit

hemoglobin

inr

lactate

magnesium

meanbp_max

meanbp_mean

meanbp_min

phosphate

platelet

potassium

pt

ptt

resprate_max

resprate_mean

resprate_min

sodium

spo2_max

spo2_mean

spo2_min

sysbp_max

sysbp_mean

sysbp_min

tempc_max

tempc_mean

tempc_min

wbc

0.0

1.0

0.0

0.2

0.4

0.6

0.8

1.0

admType_ELECTIVE

0.5

0.0

Log-Odds

0.0

0.2

0.4

0.6

0.8

1.0

admType_EMERGENCY

0.5

0.0

0

1.0

30

45

60

75

age

0.5

0.0

0.5

1.8

2.4

3.0

3.6

4.2

albumin

0.0

0.5

1.0

12

18

24

30

aniongap

0.5

0.0

0.5

Log-Odds

4

8

12

16

20

bilirubin

0

1

25

50

75

100

bun

0.5

0.0

0.5

0.0

0.2

0.4

0.6

0.8

1.0

eth_black

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

eth_other

0.0

0.5

Log-Odds

80

120

160

200

240

glucose_mean

0.5

0.0

0.5

90

120

150

180

heartrate_max

0.5

0.0

0.5

60

75

90

105

120

heartrate_mean

0.0

0.5

40

60

80

100

heartrate_min

0.0

0.5

Log-Odds

2.5

5.0

7.5

10.0

lactate

0

1

200

400

600

800

1000

platelet

0.5

0.0

0.5

24

32

40

48

resprate_max

0.25

0.00

0.25

15

20

25

30

resprate_mean

0.0

0.5

Log-Odds

87

90

93

96

99

spo2_mean

0.0

0.5

1.0

45

60

75

90

spo2_min

0.0

0.5

75

100

125

150

sysbp_mean

0.0

0.5

60

80

100

120

sysbp_min

0

1

Log-Odds

36.8

37.6

38.4

39.2

40.0

tempc_max

0.5

0.0

0.5

36.0

36.6

37.2

37.8

38.4

tempc_mean

0.0

0.5

1.0

35

36

37

38

tempc_min

0.0

0.5

1.0

age

edu

hday

dementia

wblc

resp

pafi

bili

sod

glucose

urine

adls

age

numco

edu

scoma

hday

diabetes

dementia

meanbp

wblc

hrt

resp

temp

pafi

alb

bili

crea

sod

ph

glucose

bun

urine

adlp

adls

adlsc

W/o regularization

age

edu

hday

dementia

wblc

resp

pafi

bili

sod

glucose

urine

adls

W/ regularization

0.4

0.2

0.0

0.2

0.4

Corr

0.0

0.2

0.4

0.6

0.8

Importance

adlp

adls

adlsc

age

alb

bili

bun

ca

crea

dementia

diabetes

dzclass

edu

glucose

hday

hrt

income

meanbp

numco

pafi

ph

race

resp

scoma

sex

sod

temp

urine

wblc

0.0

1.0

0.0

1.5

3.0

4.5

6.0

adlp

1

0

Log-Odds

0.0

1.5

3.0

4.5

6.0

adls

0.2

0.0

0.2

0.4

0

1.0

0.0

1.5

3.0

4.5

6.0

adlsc

0.0

0.5

30

45

60

75

90

age

1.0

0.5

0.0

0

1

2

3

4

alb

1.0

0.5

0.0

Log-Odds

0

8

16

24

32

bili

0

1

0

30

60

90

120

bun

0.5

0.0

0

1

2

ca

0.0

0.2

0.4

2.5

5.0

7.5

10.0

crea

0.0

0.5

Log-Odds

0.0

0.2

0.4

0.6

0.8

1.0

dementia

1.0

0.5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

diabetes

0.2

0.0

0

1

2

3

dzclass

0.5

0.0

0.5

0

5

10

15

20

edu

0.00

0.25

0.50

Log-Odds

0

150

300

450

glucose

0.4

0.2

0.0

10

20

30

40

hday

0.0

0.5

1.0

80

120

160

200

hrt

0.0

0.5

1.0

0

1

2

3

income

0.2

0.0

0.2

Log-Odds

50

75

100

125

150

meanbp

0.0

0.5

1.0

0.0

1.5

3.0

4.5

6.0

numco

0.50

0.25

0.00

0.25

0

150

300

450

pafi

0.5

0.0

0

2

4

6

ph

0.2

0.0

Log-Odds

1

2

3

4

race

0.2

0.0

10

20

30

40

resp

0.2

0.0

0.2

0.4

0

20

40

60

80

100

scoma

0

1

2

0.0

0.5

1.0

sex

0.1

0.0

0.1

Log-Odds

126

132

138

144

150

sod

0.0

0.5

36.0

37.5

39.0

temp

0.00

0.25

0.50

0

1500

3000

4500

urine

0.5

0.0

0.0

0.1

0.2

0.3

Importance

Households

Housing median age

Latitude

Longitude

Median income

Ocean proximity

Population

Total bedrooms

Total rooms

0.0

0.1

L1

(a) Feature importances.

0

0.1

L1

Reg. strength

0.60

0.65

0.70

Test RMSE [100k $]

(b) Test RMSEs.

400

800

1200

1600

Households

0.4

0.2

0.0

0.2

0.4

0.6

Contribution [100k $]

10

20

30

40

50

Housing median age

0.0

0.1

0.2

0

0.1

L1

34

36

38

40

Latitude [°]

0.6

0.4

0.2

0.0

0.2

0.4

123.0

121.5

120.0

118.5

117.0

Longitude [°]

0.50

0.25

0.00

0.25

0.50

Contribution [100k $]

2

4

6

8

10

Median income [10k $]

0.25

0.00

0.25

0.50

0.75

1.00

<1h Ocean

Inland

Island

Near Bay

Near Ocean

Ocean proximity

0.2

0.1

0.0

0.1

0.2

1500

3000

4500

Population

0.50

0.25

0.00

0.25

0.50

Contribution [100k $]

500

1000

1500

2000

Total bedrooms

0.4

0.2

0.0

0.2

0.4

0.6

2500

5000

7500

10000

Total rooms

0.15

0.10

0.05

0.00

0.05

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper concerns the issue of concurvity in the context of Generalized Additive Models (GAMs), which can be considered as an extension of multicolinearity to GAMs. The authors propose a regularization scheme to reduce the concurvity between learned functions, hoping to improve interpretability of the model.\n\nSTRENGTHS:\n- Multicolinearity/concurvity are indeed  serious problems in statistics, and any method that can address such issues is of interest to the community.\n- The proposed concurvity penalty is new and has not been discussed before.\n- The numerical experiments show the promise of the proposed methodology.\n\nWEAKNESSES:\n- I think the numerical results although interesting, are rather limited. As far as I could tell, authors only discuss 4 tabular datasets, which is not enough. More datasets and baselines should be added---see [1] as an example.\n\n[1] Chang, Chun-Hao, Rich Caruana, and Anna Goldenberg. \"NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning.\" International Conference on Learning Representations.\n\nQUESTIONS:\n- Connections to sparse regularization: As pointed out by the authors, the issue of multicolinearity can also arise in the context of linear models. It is well-known that regularization schemes such as sparse regularization can help to deal with correlation. For example, see [1,2]. The notion of variable selection in GAMs is also well-explored, see [3] and references therein. A natural question would be how such penalization schemes deal with concurvity. Intuitively, if one can reduce the total number of features used, a more compact model can be obtained that is less likely to suffer from concurvity.\n\n- How should one choose the parameter $\\lambda$? The authors mention the elbow method but I think this will need more exploration. It seems that the concurvity regularization can reduce validation accuracy. So, if one uses validation performance to choose a model, they'll end up with a model with no regularization. It is not clear to me how a practitioner can use the concurvity regularization if they have to sacrifice accuracy. Going back to the previous point, in the context of linear models, sparsity regularization usually comes with improved out-of-sample performance.\n\n- Extensions: Another interesting direction is that GAMs might be too simple for more complex datasets, and methods that use higher-order interaction terms have been developed [3,4]. Would it be possible to extend the concurvity regularization to such interaction models? This definitely falls out of the focus of this paper, but a short discussion from the authors would be appreciated.\n\n\n\n\n[1] Figueiredo, M., & Nowak, R.. (2016). Ordered Weighted L1 Regularized Regression with Strongly Correlated Covariates: Theoretical Aspects.\n\n[2] Hazimeh, H., & Mazumder, R. (2020). Fast best subset selection: Coordinate descent and local combinatorial optimization algorithms. Operations Research, 68(5), 1517-1537.\n\n[3] Chang, Chun-Hao, Rich Caruana, and Anna Goldenberg. \"NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning.\" International Conference on Learning Representations.\n\n[4] Enouen, J., & Liu, Y. Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection. In Advances in Neural Information Processing Systems.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors highlight concurvity as a relevant concern when developing additive models. They introduce a differentiable regularizer that aims to reduce concurvity (which will in general raise RMSE, though there may be a regularization strength that gives a good trade-off).\n\nThe proposed regularizer is applied to a neural additive model, which is demonstrated on three toy examples (two 2-variable examples where $Y$ equals one of the variables, one time series example with different weekly and daily step functions) and three UCI data sets (2x regression, 1x classification).\n\nSTRENGTHS:\nThe paper provides a nice overview of concurvity and related works and why it is an important issue to be aware of. It introduces a novel regularizer that can easily be combined with any differentiable models.\n\nWEAKNESSES:\nThe authors include a large set of references to related work on concurvity. Yet there is no comparison to other models (with the same proposed regularizer) or other methods of dealing with concurvity (despite mentioning multiple related approaches).\nThe paper claims that their concurvity regularization encourages feature selection (line 169) but there is no demonstration of this nor comparison to actual feature selection approaches.\n\nVery limited evaluation on real-world data sets; see for example [b] in the same space which has an extensive comparison both on many data sets and across multiple methods. Cited work such as [25] (Kovács 2022) uses toy examples to much better effect; a comparison of the proposed NAM/regularizer approach on the same toy example would make for a much stronger submission.\n\nOf course minimizing a given metric (such as the proposed concurvity regularizer) reduces that same metric, but I am missing a stronger demonstration of the practical benefits/relevance of this. The paper claims the resulting model is more interpretable, but does not really demonstrate this. See [a] for a discussion of interpretability of GAM features.\n\nThough the discussion is all about concurvity and non-linearity, only scalar correlation values are shown in Fig. 6 (a). Pair plots of both $X_i$ vs $X_j$ and $f_i(X_i)$ vs $f_j(X_j)$ might demonstrate the effect of the regularization more easily. Claims such as “their feature contributions largely [cancel] each other out” (lines 282-284) could easily be shown by showing their sum across the data set.\n\n### References:\n- [a] *\"How Interpretable and Trustworthy are GAMs?\"* Chang, Tan, Lengerich, Goldenberg, Caruana (2021)\n- [b] *\"Additive Gaussian Processes Revisited\"* Lu, Boukouvalas, Hensman (2022)\n\nQUESTIONS:\n### Substantial questions:\n\nQ1. Time series with multiple seasonalities are in some sense a “degenerate” example as all functions depend on the same feature $t$. How, if at all, would time series have to be treated differently from “regular” data sets with multiple input features $X_1, \\dots, X_p$?\n\nQ2. Definition 2.2: you claim this “revised formal definition” as a main contribution of your work. How is it different from (and what are the similarities to) previous definitions?\n\nQ3. Section 3: your description sounds like concurvity is purely a *model* issue, but is it not a property of the *data*?\n\nQ4. Proof in Appendix A.1: this seems to hinge on your definition of correlation which assigns $\\infty$ to the correlation with a constant vector. However, this correlation is simply ill-defined - and one could equally define it to be zero. Does this not invalidate your proof?\n\nQ5. Your proposed regularizer scales quadratically with the number of additive components: how would you efficiently address this by parallelization? (Does the scaling not remain quadratically?)\n\nQ6. “Our concurvity regularizer is agnostic to the function class …” (lines 140-142): “metrics proposed in the literature [42, 25] are not directly applicable.” [42] (Wood 2001) does not mention concurvity at all and while [25] (Kovács 2022) discusses concurvity, I could not find any mention of score/metric proposed in there. Moreover, what is the similarity of your proposed metric to that of [35] (Ramsay et al)?\n\nQ7.a) Toy Example 2: Fig. 3 (a) suggests that without regularization, the model learns that $f_1(X_1) = 0$ *more* accurately (range of $f_1$ ~ 0.0015) than with concurvity regularization (range of $f_1$ ~ 0.02), how does this support the claimed benefit of regularization?\nQ7.b) How do you see that $f_2$ approximates $|f_1|$ (lines 192-193)?\n\nQ8. lines 238-241: could it be that this is due to it being classification rather than regression?\n\nQ9. You discuss the distinction between multicollinearity and concurvity. Is it possible to have multicollinearity, but *not* have concurvity?\n\n### Notation and clarity:\n\nC10. Regarding the origin of concurvity (lines 100-101, 293), you might want to cite the work by Buja, Donnell, Stuetzle (1986) cited within your reference [9].\n\nC11. The sum $\\sum_{l=1}^N$ in the (GAM-Fit) and (GAM-Fit$_\\perp$ equations seems to be inconsistent with the vector notation (the subscript $l$ is not used anywhere).\n\nQ12. line 85: the notation used in the equation $\\mathcal{H} \\subset \\dots$ is not immediately clear to me. Is $\\mathcal{H}$ supposed to be a space of $p$-tuples?\n\nQ13. “every suitable linear combination of features can be modified by adding a trivial linear combination” (lines 93-94) can you clarify what you mean here? Having gone through some of your references I understand, but your description on its own is rather confusing.\nLikewise, “any non-trivial zero-combination of features can be added to a solution of (GAM-Fit)” (line 109-110): what does this mean?\n\nC14. Simply writing “(GAM-Fit)” e.g. in line 110 made me take a while to realize that this is referring to an equation and to find where it was. How about having the equation label include a number, e.g. “(2; GAM-Fit)”, and then you can explicitly refer to “a solution of Eq. (2; GAM-Fit)” for example?\n\nC15. Additional remarks in Appendix A.2: these are very helpful, would be great to at least summarize the points in the main text.\n\nQ16. Figure 2 (b) is great to show that there is a value of $\\lambda$ that reduces the concurvity measure without affecting RMSE perceptibly, but how sensitive is this to the value?\nThis might be easier to see in two separate plots of $\\lambda$ vs $R_\\perp$ and $\\lambda$ vs RMSE overlaid on top of each other (to identify the range of $\\lambda$ in which both are low simultaneously).\n\nQ17. line 172: what is Fig. 8 supposed to demonstrate? This is not clear to me.\n\nQ18. lines 175, 177: what would be the “scale” of $\\lambda$? What does “moderate” or “considerably high” mean?\n\nQ19. line 243: Are you aware of any other references that discuss the “elbow technique”? Thorndike’s “Who belongs in the family?” does not even mention “elbow”, and I would hope a better reference in this context is available (though it was a thoroughly enjoyable read which I thank the authors for bringing to my attention!)\n\nC20. The references are inconsistent.\n\nLIMITATIONS:\nThe authors acknowledge the limited validation of their approach (though I would not call it \"diverse\"). They also acknowledge that there is an interpretability-accuracy trade-off in concurvity regularization (though I find their demonstration of \"increased interpretability\" lacking).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGeneralized additive models (GAMs) offer greater interpretability than other machine learning methods while having greater flexibility than generalized linear models. This paper addresses how to account for concurvity (the non-linear analog to multicollinearity in linear regression). The authors suggest directly penalizing correlations between the features during training. Concurivity poses problems with interpretability (which (transformed) feature to attribute the contributions to) and can increase the variance in the model. The proposed regularizer was found to reduce concurvity while maintaining strong predictive performance. The viability and success of this method are shown in time series forecasting as well as tabular regression tasks.\n\nSTRENGTHS:\n- Clear motivation for the problem and general organization of the paper\n- Method is light on assumptions regarding the feature transformations, $f_i$ so this can be applied in many settings\n- Improves model interpretability while reducing variance in fitted functions\n\nWEAKNESSES:\n- The figures were very information dense (particularly fig 6). More discussion of these results and how to interpret them would aid the reader\n- Not much space was spent on the time-series experiments relative to how much was dedicated to it in the introduction\n\n\nMinor\n- line 75: (linear) regression\n- line 83: minimize over $\\beta$ as well?\n- line 126: minimize over $\\beta$?\n\nQUESTIONS:\n- How much do the $f_i$ differ depending on $\\lambda$? How fair is it to compare feature importances across different levels of regularization if the resulting $f_i$ are not the same? (Possibly addressed in figure 6.c?)\n\n- line 126.5: in the $gam-fit_\\perp$ what is the $l$ index for? It does not show up in any of the other terms\n\n- What is the additional computational burden of computing the regularizer? How well does this scale?\n\n- How does gam-fit perform in the presence of covariates that are neither perfectly correlated nor entirely uncorrelated (as they were in the toy examples)?\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a regularization using pairwise correlations of shape functions for differentiable GAM, aimed at reducing concurvity between shape functions\n\nSTRENGTHS:\n1.\tThe idea of the paper is well-explained and straightforward.  The correlated terms can be self-cancelled to avoid to unnecessary complexity. So reducing concurvity may potentially enhance generalizability. \n2.\tThe paper’s story, theory and the numerical experiments all demonstrate that regularization effectively controls the correlations of the non-linearly transformed features.\n3.\tThe proposed component is easy to implement and optimize and as far as I understand, it can be incorporated into any differentiable GAM.\n\nWEAKNESSES:\n1.\tThe foundation of this paper – the reason why correlated terms should be cancelled out is still beyond me. For the time series scenario in the introduction, I understand that high frequency terms of feature pairs hinder interpretation. But what if the correlation does lie in the ground truth? Especially for the tabular data, the visualized results in the public dataset show that most of the features have almost no prediction power for the target variable (e.g. Population, households, housing age before 30 years, total bedrooms and total rooms, etc.), which seem to be counterintuitive. Little concurvity is allowed, but it happens in the real world. Is it possible that restricting concurvity may in general push the shape functions away from their natural relationship (trained to be not effective but actually affects the target)? The author may provide further evidence and explanation regarding the negative effects of concurvity, to make the article more convincing. \n2.\tWhile previous literature found that regularizing the cross-covariance increases generalization performance, the numerical results presented in this paper indicate that controlling the concurvity will sacrifice the generalization ability anyhow. The accuracy is slightly worse than the one without regularization, in exchange of the stated ‘interpretability’, which unfortunately is not straightforward in the figures. Therefore, it appears that the trade-off between accuracy and correlation may not be worth it in this case.\n3.\tThe paper showcases the motivation using a time-series example, but no visual results for time series are presented to address the issue. It is doubtful whether the regularization does control the frequency or some uninterpretable parts, rather than merely force terms to be uncorrelated (in an unnecessary manner).\n4.\tIt is not shown how to do the optimization. Is the calculation of regularization time-consuming and unstable? What would happen if the regularization is implemented in every iteration rather than only 5% of the total optimization steps as introduced in the article? It seems a mismatch between the design and the realization.\n\nQUESTIONS:\n1.\tIn Figure 2(a), we see the model with regularization chooses a random point on a curve. This is because both features are identical in the setting, making the model unidentifiable. What if the two features are correlated but not perfectly correlated? Is the regularizer able to identify the true predictive variable X1, and not so affected by random seeds?\n2.\tHow to determine a proper level of the strength of regularization (lambda)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}