SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We optimized the second-order derivative penalties on the spline functions in pyGAM [43] using random search. We used a budget of 300 function evaluations within a search range of [1e-3, 1e3].
C.1 NAM Experiments
In all of our NAM experiments, we use the AdamW optimizer [33] and adjust our learning rate using Cosine Annealing [32] and decay to 0. For all regression problems we use the Mean Squared Error (MSE) and for all binary classification problems the Binary Cross Entropy (BCE) as loss function  $L$ .
In all of our experiments, we apply the concavity regularization after a warm-up phase consisting of  $5\%$  of the total optimization steps for stability reasons. After this warm-up phase, regularization is performed at each step.
C.2 Toy Examples
We sample 10,000 datapoints from the model and use 7,000, 2,000, 1,000 for training, validation and testing respectively. We use the validation split to find adequate hyperparameters via a small manual search. Our NAM has 3 layers per feature NN with 128 hidden units and uses the GeLU [26] activation function. We use no weight decay, train for 50 epochs at an initial learning rate of 1e-3 and a batch size of 128. Otherwise, the experimental setup is the same as described earlier in Section C.1.
Boston Housing The Boston Housing Dataset [23], compiled by Harrison and Rubinfeld in the 1970s, is a benchmark dataset employed in machine learning and statistical modeling for housing price prediction. Consisting of 506 neighborhoods in the Boston metropolitan area, it features 13 attributes, including crime rate, zoning information, industrial acreage, Charles River proximity, air quality, housing characteristics, accessibility to employment centers, highways, and education, as well as demographic factors. The primary objective is to predict the median value of owner-occupied homes using these features.
California Housing The California Housing Dataset [37] is a widely-used benchmark dataset for machine learning and statistical modeling, particularly in the domain of housing price prediction. Originally derived from the 1990 California Census, it consists of 20,640 samples, each representing a census block group. The dataset contains information on various housing-related attributes, such as median income, housing median age, average number of rooms, average number of bedrooms, population and average household size. It also includes the geographical location (latitude and longitude) of each block group. The objective is to predict the median house value. We obtained the dataset from [18].
Adult The Adult Dataset [18] is also known as the "Census Income" dataset. Extracted from the 1994 United States Census Bureau data, it comprises 48,842 records, each representing an individual. The dataset contains 14 features, including age, work class, education, marital status, occupation, relationship, race, sex, capital gain, capital loss, hours worked per week, and native country. The objective is to predict whether an individual's annual income exceeds $50,000.
MIMIC-II The MIMIC-II (Multiparameter Intelligent Monitoring in Intensive Care) dataset [29] is a public database, offering clinical data from a multitude of Intensive Care Unit (ICU) patients. It is managed by the MIT Lab for Computational Physiology and encompasses a wide variety of data points, such as patient demographics, vital signs, lab test results, medications, procedures, caregiver notes, and imaging reports, as well as mortality rates both in and out of the hospital.
MIMIC-III MIMIC-III (Medical Information Mart for Intensive Care III) [27] is the successor to the MIMIC-II database. It contains additional, more recent patient records and provides more detailed data, including free-text interpretation of imaging reports, allowing for more granular research and improved application in areas like machine learning, health informatics, and predictive modeling.
SUPPORT2 The Support2 [14] (Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments) dataset is a clinical database that contains detailed medical information from a large cohort of seriously ill hospitalized adults. The dataset was created as part of a multi-center study designed to understand the outcomes of decisions made in the course of medical treatment. It provides extensive variables, including demographic data, physiological measurements, diagnostic information, treatment plans, and outcomes such as survival and quality of life. The data collected span a diverse set of medical conditions, making the SUPPORT2 dataset a valuable resource for researchers seeking to study clinical decision-making, prognosis evaluation, and healthcare outcomes.
E.1 Toy Example 1
Figure 8 shows an additional scatter plot comparing the contributions of the transformed features.
(a) w/o concavity regularization
(b) w/ concurvity regularization  $(\lambda = 0.1)$
Figure 8: (Toy Example 1) Pair plot demonstrating the difference in shape functions learned without and with concavity regularization. The features  $X_{1}$  and  $X_{2}$  are fully correlated in this example. Each subplot is generated by a scatter plot over a large set of random samples from the input feature pair  $(X_{1}, X_{2})$ . As expected, the model trained without regularization shows strong (almost perfect) linear correlation between the transformed features  $f_{1}(X_{1}), f_{2}(X_{2})$  and the target variable  $Y = X_{1}$ , while our regularizer effectively decorrelates  $f_{1}(X_{1})$  and  $f_{2}(X_{2})$ . But note that  $f_{1}(X_{1})$  and  $f_{2}(X_{2})$  are not stochastically independent, as the plots in (b) clearly show a non-linear functional relationship between the two features.
E.2 Additional Toy Example from [28]
In response to a suggestion to include a more complex toy example during the review process, we have replicated the toy example from [28] using our NAM setup. To recap, this example contains 7 input features:
$$
X _ {1} \sim X _ {2} \sim X _ {3} \sim U (0, 1),
$$
$$
X _ {4} = X _ {2} ^ {3} + X _ {3} ^ {2} + N (0, \sigma_ {1}),
$$
$$
X _ {5} = X _ {3} ^ {2} + N (0, \sigma_ {1}),
$$
$$
X _ {6} = X _ {2} ^ {2} + X _ {4} ^ {2} + N (0, \sigma_ {1}),
$$
$$
X _ {7} = X _ {1} \cdot X _ {2} + N (0, \sigma_ {1}),
$$
$$
Y = 2 X _ {1} ^ {2} + X _ {5} ^ {3} + 2 \sin (X _ {6}) + N (0, \sigma_ {2}),
$$
where the standard deviations are sufficiently small to create severe concavity among the features  $(\sigma_{1} = 0.05, \sigma_{2} = 0.5)$ . We simulated 10,000 data points from this model and created a 7:3 train/test split. We fitted 20 random initializations of a NAM in the unregularized, concavity-regularized, and L1 regularized setting. The regularization parameter was determined separately for each regularization type based on trade-off curves. For concavity regularization, we used  $\lambda = 0.1$ , and for L1, we used  $\lambda = 0.05$ .
The results, reported as the  $R^2$  on the test set, are reported in Table 2, with the top three rows from [28] for comparison. Confidence intervals of the mean are estimated on 10,000 bootstrap samples. Features are presented in descending order of their importance (as defined in the main paper) for the best-fitting model in each setting, while the actual importances are reported below.
Model	Selected features	R2(%) (5%, 95%)	R⊥ (5%, 95%)
Full model [28]	Full model	84.99
Stepwise [28]	X1, X4, X5, X6	85.11
Hybrid [28]	X1, X5, X6	85.31
Unregularized (ours)	X1, X6, X4, X2, X5, X3, X7	80.77 (80.31 / 80.95)	0.22 (0.20 / 0.23)
→ Feature importance	0.129, 0.097, 0.066, 0.053, 0.043, 0.013, 0.004
Concurvity Reg. (ours)	X1, X6, X5, X2, X7, X4, X3	79.28 (78.52 / 79.88)	0.03 (0.02 / 0.03)
→ Feature importance	0.132, 0.125, 0.088, 0.070, 0.006, 0.002, 0.002
L1 Reg. (ours)	X6, X1, X5, X7, X4, X3, X2	79.12, (78.50 / 79.47)	0.21, (0.20 / 0.21)
→ Feature importance	0.147, 0.106, 0.037, 0.009, 0.008, 0.005, 0.0
Table 2: Results of replication experiment of the toy example proposed in [28]. All results are calculated over the test split, with our results reported as mean values and confidence intervals. In addition to our three NAM settings (unregularized, concavity regularization, and L1 regularization), we report the values from the original paper [28] for comparison.
We note that both concavity regularization and L1 regularization correctly identify the three predictive features  $X_{1}, X_{5}$  and  $X_{6}$ , on which  $Y$  directly depends. This is not the case without regularization. Furthermore, we find that concavity regularization effectively reduces  $R_{\perp}$ , unlike L1 regularization.
We also note that the  $R^2$  values of all NAM implementations are lower than those reported in [28], which we believe is due to the inductive biases in spline-based models being particularly well-suited to the mostly polynomial-based problem. This notwithstanding, our results further underline the effectiveness of concavity regularization.
E.3 Tabular Data
E.3.1 Verbose Trade-off Curves
In addition to the trade-off curves presented in the main text (see Figure 5), Figure 9 shows a more verbose version, separating the visualization for the measured validation concavity and validation accuracy as a function of the regularization strength  $\lambda$ . This explicit visualization allows us for an easier choice of  $\lambda$ , for example, by determining the elbow in the validation accuracies, after which stronger regularization would lead to a significant loss of performance. For example, this can be nicely observed in the case of the Boston Housing dataset at  $\lambda \approx 1$ .
Figure 9: Verbose trade-off curves between the model fit quality and measured concavity  $R_{\perp}$  for 50 levels of the concavity regularization strength  $\lambda$ . Each regularization strength is evaluated over 10 initialization seeds to account for training variability, which is particularly noticeable in smaller datasets.
E.3.2 Runtime Analysis
Due to calculating pair-wise correlations, our proposed regularizer has a quadratic computational complexity. However, using an efficient implementation leveraging vectorization, the scaling constant can be well controlled. To investigate the computational overhead of our regularizer, we measure the mean absolute runtime across 100 repetitions for different numbers of input features and batch sizes. Furthermore, we evaluate the relative overhead of our regularizer compared to a NAM model. The relative overhead clearly depends on the size and implementation details of the NAM. Here, we compare with a NAM for which each feature-dependent network consists of a 3-layer MLP with 128 neurons each. The calculation over the features is vectorized using batched matmuls. The results are shown in Figure 10, all obtained with an M1 MacBook Pro. As expected, we observe a power-law
scaling of the runtime of the regularizer, which is in particular observable for an increasing numbers of input features. In particular, for moderate numbers of input features between 64 and 256, the absolute runtime is about  $1\mathrm{ms}$ . Compared with the forward pass of a 3-layer NAM, the proposed regularizer typically implies a relative overhead of well below  $10\%$ . Note that the results compare only with the forward pass of the NAM model and do not include data loading, batch collation, loss calculation, and backward passes. We note that especially for larger batch sizes, the relative overhead introduced by our regularizer is well below  $5\%$ . On the other hand, for smaller batch sizes, the relative overhead can become large, due to the reduced effectiveness of vectorization. In absolute terms, however, the overhead is very small for smaller batch sizes as well. We therefore conclude that the overhead introduced by regularization is negligible in commonly considered model setups.
Figure 10: Left plot: Absolute runtime of the proposed regularizer for different numbers of input features and batch sizes. Right plot: Computational overhead relative to the forward pass of a 3-layer NAM, with 128 neurons each, for different numbers of input features and batch sizes.
E.3.3 In-detail Results for the Remaining Tabular Datasets
In addition to the results of our case study with the California Housing dataset in Section 4.3, we here present the shape functions of the complete set of features, see Figure 11.
Furthermore, we show the analogous results for the remaining datasets Adult (Figure 12), Boston Housing (Figure 13), MIMIC-II (Figure 14), MIMIC-III (Figure 15 & 16), and Support2 (Figure 17 & 18).
E.3.4 Comparing Concavity Regularization to L1 Regularization
Although we chose pyGAM as our baseline in the main paper, we have additionally compared our concavity regularizer to L1 regularization, as a popular alternative to eliminating redundant features. Results on the California Housing dataset are included in Figure 19. We performed L1 regularization on the terms  $w_{i}f_{i}(x_{i})$  in Equation (GAM), with regularization strength  $\lambda = 0.1$ , as determined by trade-off curves on feature L1 norms versus RMSE.
The figure suggests that L1 regularization prunes features more strongly than concavity regularization, which may be desirable in some situations. However, this comes at a cost of a larger test RMSE, as well as some seemingly predictive features being pruned. For example, L1 regularization removes the two local peaks in the Latitude dimension corresponding to San Francisco and Los Angeles. In addition, unlike L1 regularization, concavity regularization does not prune uncorrelated features.
Finally, we note that there may be different motivations behind choosing sparsity regularization (as few features as possible) or concavity regularization (features become as decorrelated as possible). If the main goal is to address concavity, then sparsity regularization is a blunt tool, and vice versa.
Figure 11: Shape functions of all input features in the case of the California Housing dataset. Depicted are the average as well as individual contributions with and without concavity regularization using 60 initialization seeds each. The pyGAM results plotted for comparison include the estimated  $95\%$  confidence intervals.
(a) Average feature correlations of the features (lower left) and non-linearly transformed features (upper right). Showing non-categorical features only.
(b) Combined box and strip plot of the models' feature.   
importances.
(c) Average and individual shape functions. A kernel density estimate of the training data distribution is depicted on top.
Figure 12: Results for the Adult dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
Figure 13: Results for the Boston Housing dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
Figure 14: Results for the MIMIC-II dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
(a) Average feature correlations of the features (lower left) and non-linearly transformed features (upper right).
Figure 15: Results (part 1) for the MIMIC-III dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
Figure 16: Results (part 2) for the MIMIC-III dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
(a) Average feature correlations of the features (lower left) and non-linearly transformed features (upper right). Showing non-categorical features only.
Figure 17: Results (part 1) for the Support2 dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
Figure 18: Results (part 2) for the Support2 dataset. The considered NAMs were trained with and without concavity regularization using 60 model initialization seeds each.
(a) Feature importances.
(b) Test RMSEs.
(c) Average and individual shape functions of the 9 selected features. A kernel density estimate of the training data distribution is depicted on top.
Figure 19: Comparison of concavity and L1 regularization on the California Housing dataset. The considered NAMs were trained using 60 model initialization seeds each. Note that for L1, the regularization strength is set to  $\lambda = 0.1$ .