SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The contextual explanation networks in Al-Shedivat et al. (2020) are cousins of the contextual lasso. These neural networks input contextual features and output interpretable models for explanatory features. They include the (nonsparse) contextual linear model, a special case of the contextual lasso where  $\lambda = \infty$ . In their terminology, the contextual linear model is a "linear explanation" model with a "deterministic encoding" function. They also explore a "constrained deterministic encoding" function that involves a weighted combination of individual fixed linear models with weights determined by the contextual features. To avoid overfitting, they apply  $\ell_1$ -regularization to the individual models. However, they have no mechanism that encourages the network to combine these sparse models such that the result is sparse. In contrast, the contextual lasso directly regularizes the sparsity of its outputs.
The contextual lasso is also related to several estimators that allow varying sparsity patterns. Yamada et al. (2017) devise the first of these—the localized lasso—which fits a linear model with a different coefficient vector for each observation. The coefficients are sparsified using a lasso regularizer that relies on the availability of graph information to link the observations. Yang et al. (2022) and Yoshikawa and Iwata (2022) follow with LLSPIN and NGSLL, neural networks that produce linear models with varying sparsity patterns via gating mechanisms. These approaches are distinct from our own, however. First, they do not dichotomize into  $\mathbf{x}$  and  $\mathbf{z}$ , making the resulting model  $\mathbf{x}^{\top} \boldsymbol{\beta}(\mathbf{x})$  difficult to interpret. Second, the sparsity level (NGSSL) or nonzero coefficients (LLSPIN) are fixed across observations, making them unsuitable for the contextual setting where both may vary.
Hallac et al. (2015) introduce the network lasso which has a different coefficient vector per observation, clustered using a lasso-style regularizer. They consider problems similar to ours, for which contextual information is available, but do not impose sparsity on the coefficients. Deleu and Bengio (2021) induce structured sparsity over neural network weights to obtain smaller, pruned networks that admit efficient computation. In our work, we leave the weights as dense and instead induce sparsity over the network's output for interpretability. Wang et al. (2020) propose a network quantization scheme with activation functions that output zeros and ones. Though our approach involves an activation that outputs zeros, we also allow a continuous output. Moreover, their end goal differs from ours; whereas they pursue sparsity to reduce computational complexity, we pursue sparsity for interpretability.
Our work also advances the broader literature at the intersection of feature sparsity and neural networks, an area that has gained momentum over the last few years. See, e.g., the lassonet of Lemhadri et al. (2021a,b) which selects features in a residual neural network using an  $\ell_1$ -regularizer on the skip connection. This regularizer is combined with constraints that force a feature's weights on the first hidden layer to zero whenever its skip connection is zero. See also Scardapane et al. (2017) and Feng and Simon (2019) for earlier ideas based on the group lasso, and Chen et al. (2021) for another approach. Though related, these methods differ from the contextual lasso in that they involve uninterpretable neural networks with fixed sparsity patterns. The underlying optimization problems also differ—whereas these methods regularize the network's weights, ours regularizes its outputs.
The contextual lasso is evaluated here via experimentation on synthetic and real data. As benchmark methods, we consider the (nonsparse) contextual linear model, which uses no projection layer, and a deep neural network, which receives all explanatory and contextual features as inputs. We further include the lasso, lassonet, and LLSPIN, which also receive all features. The localized lasso does not scale to the experiments that follow, so we instead compare it with the contextual lasso on smaller experiments in Appendix G. Appendix H provides the implementation details of all methods.
We consider three different settings of increasing complexity: (1)  $p = 10$  and  $m = 2$ , (2)  $p = 50$  and  $m = 2$ , and (3)  $p = 50$  and  $m = 5$ . Within each setting, the sample size ranges from  $n = 10^2$  to  $n = 10^5$ . The full simulation design is detailed in Appendix I. As a prediction metric, we report the square or logistic loss relative to the intercept-only model. As an interpretability metric, we report the proportion of nonzero features. As a selection metric, we report the F1-score of the selected features; a value of one indicates all true positives recovered and no false positives. All three metrics are evaluated on a testing set with tuning on a validation set, both constructed by drawing  $n$  observations independently of the training set. Figure 3 reports the results for regression (i.e., continuous response).
The contextual lasso performs comparably with most of its competitors when the sample size is small. On the other hand, the contextual linear model (the contextual lasso's unregularized counterpart) can perform poorly here (its relative loss had to be omitted from some plots to maintain the aspect ratio). As  $n$  increases, the contextual lasso begins to outperform other methods in prediction, interpretability, and selection. Eventually, it learns the correct map from contextual features to relevant explanatory features, recovering only the true nonzeros. Though its unregularized counterpart performs nearly as well in terms of prediction for large  $n$ , it remains much less interpretable, using all explanatory features. In contrast, the contextual lasso uses just  $10\%$  of the explanatory features on average.
The deep neural network's performance is underwhelming for most  $n$ . Only for large sample sizes does it begin to approach the prediction performance of the contextual lasso. The lassonet often performs somewhere between the two. These three methods should predict equally well for large enough  $n$ , though the function learned by the deep neural network and lassonet will remain opaque. The lasso makes some gains with increasing sample size, but lacks the expressive power of the contextual lasso needed to adapt to the complex sparsity pattern of the true model. LLSPIN—the only other method to allow for varying sparsity patterns—is the second best feature selector for  $p = 10$ , though its mediocre performance more generally is likely due to it not exploiting the explanatory-contextual feature dichotomy and not allowing its nonzero coefficients to change.
4.2 Energy consumption data
We consider a real dataset containing energy readings for a home in Mons, Belgium (Candanedo et al., 2017). Besides this continuous response feature, the dataset also contains  $p = 25$  explanatory features in the form of temperature and humidity readings in different rooms of the house and local weather
5The contextual linear model corresponds to the contextual explanation network with a linear explanation and deterministic encoding in Al-Shedivat et al. (2020).
The F1-score := 2 TP / (2 TP + FP + FN), where TP, FP, and FN are the number of true positive, false positive, and false negative selections.
- Deep neural network - Lasso - LLSPIN
- Contextual linear model - Lassonet - Contextual lasso
Figure 3: Comparisons on synthetic regression data. Metrics are aggregated over 10 synthetic datasets. Solid points are averages and error bars are standard errors. Dashed horizontal lines in the middle row indicate the true sparsity level.
data. We define several contextual features from the time stamp to capture seasonality: month of year, day of week, hour of day, and a weekend indicator. To reflect their cyclical nature, the first three contextual features are transformed using a sine and cosine, leading to  $m = 7$  contextual features.
The dataset, containing  $n = 19$ , 375 observations, is randomly split into training, validation, and testing sets in 0.6-0.2-0.2 proportions. We repeat this random split 10 times, each time recording performance on the testing set, and report the aggregate results in Table 1. As performance metrics, we consider the relative loss and average sparsity level (i.e., average number of selected explanatory features). Among all methods, the contextual lasso leads to the lowest test loss, outperforming even
Table 1: Comparisons on the energy consumption data. Metrics are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative loss	Avg. sparsity
Deep neural network	0.433 ± 0.004	25.0 ± 0.0
Contextual linear model	0.387 ± 0.003	25.0 ± 0.0
Lasso	0.690 ± 0.002	11.6 ± 0.4
Lassonet	0.423 ± 0.003	25.0 ± 0.0
LLSPIN	0.639 ± 0.005	24.5 ± 0.1
Contextual lasso	0.356 ± 0.003	2.8 ± 0.4
the deep neural network and lassonet. $^{7}$  Importantly, this excellent prediction performance is achieved while maintaining a high level of interpretability. In contrast to most other methods, which use all (or nearly all) available explanatory features, the predictions from the contextual lasso arise from linear models containing just 2.8 explanatory features on average! These linear models are also much simpler than those from the lasso, which typically involve more than four times as many features.
The lassonet with tuned  $\lambda$  uses nearly all features here. However, manually choosing  $\lambda$  to attain a sparsity level similar to the contextual lasso substantially degrades its performance.
Figure 4: Explanatory feature sparsity as a function of hour of day for the estimated energy consumption model. The sparsity level varies within each hour because the other contextual features vary.
The good predictive performance of the contextual lasso suggests a seasonal pattern of sparsity. To investigate this phenomenon, we apply the fitted model to a randomly sampled testing set and plot the resulting sparsity levels as a function of the hour of day in Figure 4. The model is typically highly sparse in the late evening and early morning. Between  $10\mathrm{pm}$  and  $6\mathrm{am}$ , the median proportion of nonzero coefficients is  $0\%$ . There is likely little or no activity inside the house at these times, so sensor readings from within the house—which constitute the majority of the explanatory features—are irrelevant. The number of active explanatory features rises later in the day, peaking around lunchtime and dinnertime. Overall, a major benefit of the contextual lasso, besides its good predictions, is the ability to identify a parsimonious set of factors driving energy use at any given time of day.
4.3 Parkinson's telemonitoring data
We illustrate the contextual lasso on grouped explanatory features using data from a study on the progression of Parkinson's disease in 42 patients (Tsanas et al., 2009). The task is to predict disease progression (a continuous variable) using 16 vocal characteristics of the patients as measured at different times throughout the study. As Tsanas et al. (2009) point out, these vocal characteristics can relate nonlinearly to disease progression. To account for these effects, we compute a five-term cubic regression spline per explanatory feature  $(p = 16 \times 5 = 80)$ . Each spline forms a single group of explanatory features  $(g = 16)$ . The contextual features are the age and sex of the patients  $(m = 2)$ .
The dataset of  $n = 5,875$  observations is again partitioned into training, validation, and testing sets in the same proportions as before. As a new benchmark, we evaluate the group lasso, which is applied to splines of all explanatory and contextual features. The deep neural network and lassonet are applied to the original (nonspline) features. $^{8}$  The lasso is also applied to the original features to serve as a linear benchmark. The results are reported in Table 2. The purely linear estimator—the lasso—performs
Table 2: Comparisons on the Parkinson's telemonitoring data. Metrics are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative loss	Avg. sparsity
Deep neural network	0.367 ± 0.015	16.0 ± 0.0
Lasso	0.885 ± 0.005	3.1 ± 0.1
Group lasso	0.710 ± 0.006	4.2 ± 0.4
Lassonet	0.263 ± 0.007	15.5 ± 0.2
Contextual group lasso	0.113 ± 0.006	1.6 ± 0.3
worst overall. The group lasso improves over the lasso, supporting claims of nonlinearity in the data. The contextual group lasso is, however, the star of the show. Its models predict nearly three-times better than the next best competitor (lassonet) and are sparser than those from any other method.
Setting aside predictive accuracy, a major benefit of the contextual group lasso (compared with the deep neural network and lassonet) is that it remains highly interpretable. To illustrate, we consider the fitted spline function (i.e., the spline multiplied by its coefficients from the contextual group lasso) on the detrended fluctuation analysis (DFA) feature, which characterizes turbulent noise in speech. Figure 5 plots the function at three different ages of patient. For 70-year-olds, the function is zero, indicating DFA is not yet a good predictor of Parkinson's. At 75, the function becomes nonzero,
<sup>8</sup> Inputting the splines to these methods does not improve their performance.
Figure 5: Fitted spline function from the contextual lasso for the detrended fluctuation analysis (DFA) explanatory feature. The age explanatory feature is varied while the sex feature is set to female.
taking on a concave shape. It becomes even more concave and negative 80. The models reported in Tsanas et al. (2009) also had this coefficient negative, but fixed across all ages. In contrast, the contextual lasso identifies DFA and other features as relevant only for patients of certain ages and sex.
4.4 Additional experiments
Experiments for classification on synthetic and real data are available in Appendix J. In Appendix K, we report high-dimensional experiments with  $p = 1,000$  and fixed coefficient experiments (i.e., the lasso's home court). Appendix L investigates the stability of the contextual lasso with respect to the random initialization. Appendix M provides hyperlinks to the datasets used throughout the paper.
Contextual sparsity is an important extension of the classical notion of feature sparsity. Rather than fix the relevant features once and for all, contextual sparsity allows feature relevance to depend on the prediction context. To tackle this intricate statistical learning problem, we devise the contextual lasso. This new estimator utilizes the expressive power of deep neural networks to learn interpretable sparse linear models with sparsity patterns that vary with the contextual features. The optimization problem that defines the contextual lasso is solvable at scale using modern deep learning frameworks. Grouped explanatory features and side constraints are readily accommodated by the contextual lasso's neural network architecture. An extensive experimental analysis of the new estimator illustrates its good prediction, interpretation, and selection properties in various settings. To the best of our knowledge, the contextual lasso is the only tool currently available for handling the contextually sparse setting.
The problem of deciding the explanatory-contextual feature split is the same as that faced with varying-coefficient models. Though the literature on varying-coefficient models is extensive, there are no definitive rules for partitioning the features in general. In the housing and energy examples, the contextual features are spatial or temporal effects, which are distinct from the remaining features. In the telemonitoring example, the patient attributes (age and sex) differ fundamentally from the vocal characteristics. Ultimately, the partition for any given application should be guided by domain expertise with consideration to the end goal. If one needs to interpret the exact effect of a feature, that feature should be an explanatory feature. If a feature's effect is of secondary interest, or it is suspected that the feature influences the structural form of the model, that feature should be a contextual feature. If the user determines there are no contextual features, the ordinary lasso is a more appropriate tool.
It remains an important avenue of future research to establish a solid theoretical foundation for the contextual lasso. The statistical properties of the lasso in terms of estimation, prediction, and selection are now well-established in theory (Bunea et al., 2007; Raskutti et al., 2011; Shen et al., 2013; Zhang et al., 2014). The synthetic experiments in our paper suggest that the contextual lasso satisfies similar properties, though theoretically establishing these results is challenging. Statistical convergence results for vanilla feedforward neural networks (e.g., Schmidt-Hieber, 2020) do not apply in our setting due to the projection layer. Moreover, to our knowledge, no statistical guarantees exist for neural networks configured with convex optimization layers that otherwise might apply here. It is also important to understand when the contextual lasso's performance is matched by a deep neural network, since both should predict well for large samples in the contextually sparse linear regime.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-85, 2022.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 241:81-89, 2017.
Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. Annals of Statistics, 48:1875-1897, 2020.
Xiaotong Shen, Wei Pan, Yunzhang Zhu, and Hui Zhou. On constrained and regularized high-dimensional regression. Annals of the Institute of Statistical Mathematics, 65(5):807-832, 2013.
Ryan Thompson, Edwin V Bonilla, and Robert Kohn. Contextual directed ayclic graphs, 2023. arXiv: 2310.15627.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58:267-288, 1996.
Athanasios Tsanas, Max A Little, Patrick E McSharry, and Lorraine O Ramig. Accurate telemonitoring of Parkinson's disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering, 57:884-893, 2009.
Ewout van den Berg, Mark Schmidt, Michael P Friedlander, and Kevin Murphy. Group sparsity via linear-time projection. Technical Report TR-2008-09, Department of Computer Science, University of British Columbia, 2008.
Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12192-12199, 2020.
Makoto Yamada, Koh Takeuchi, Tomoharu Iwata, John Shawe-Taylor, and Samuel Kaski. Localized lasso for high-dimensional regression. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 54:325-333, 2017.
Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally sparse neural networks for tabular biomedical data. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 25123-25153, 2022.
Yuya Yoshikawa and Tomoharu Iwata. Neural generators of sparse local linear models for achieving both accuracy and interpretability. Information Fusion, 81:116-128, 2022.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68:49-67, 2006.
Yuchen Zhang, Martin J Wainwright, and Michael I Jordan. Lower bounds on the performance of polynomial-time algorithms for sparse linear regression. In Proceedings of the 27th Conference on Learning Theory, volume 35, pages 921-948, 2014.
Yichen Zhou and Giles Hooker. Decision tree boosted varying coefficient models. Data Mining and Knowledge Discovery, 36:2237-2271, 2022.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67:301-320, 2005.
A House pricing data
We expand the analysis in Section 1 and compare the statistical performance of the contextual lasso with its competitors using the house pricing data. Following Zhou and Hooker (2022), the explanatory features are the elevator indicator, renovation condition, floor of the apartment, and number of living rooms and bathrooms  $(p = 5)$ . The contextual features are longitude and latitude  $(m = 2)$ . We randomly sample training, validation, and testing sets of size  $n = 15,000$  and report the results across 10 random data splits in Table 3. The contextual lasso delivers sparse models and with competitive
Table 3: Comparisons of methods on the house pricing data. Metrics are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative loss	Avg. sparsity
Deep neural network	0.515 ± 0.003	5.0 ± 0.0
Contextual linear model	0.505 ± 0.002	5.0 ± 0.0
Lasso	0.892 ± 0.001	5.0 ± 0.0
Lassonet	0.521 ± 0.003	5.0 ± 0.0
LLSPIN	0.729 ± 0.031	4.5 ± 0.2
Contextual lasso	0.498 ± 0.002	2.9 ± 0.3
prediction performance. The contextual linear model trails closely in prediction, though it does not offer a similar level of parsimony. While also producing good predictions, the deep neural network and lassonet do not offer the same interpretability. The lasso lags far behind the contextual lasso and other neural network-based methods, suggesting that the contextual features have nonlinear effects.
B Grouped explanatory features
Algorithm 2 presents the routine for projecting onto the group  $\ell_1$ -ball. To summarize the algorithm,
Algorithm 2 Projection onto group  $\ell_1$ -ball
input Dense group coefficients  $\pmb{\eta}_{1}^{(k)},\dots,\pmb{\eta}_{n}^{(k)}$ $(k = 1,\dots ,g)$  and radius  $\lambda$    
Compute group-wise norms  $\xi_{i}^{(k)} = \| \pmb{\eta}_{i}^{(k)}\|_{2}$  for  $i = 1,\ldots ,n$  and  $k = 1,\ldots ,g$    
Run Algorithm 1 with  $\xi_1^{(k)},\ldots ,\xi_n^{(k)}$ $(k = 1,\dots ,g)$  and  $\lambda$  to get  $\bar{\xi}_1^{(k)},\dots,\bar{\xi}_n^{(k)}$ $(k = 1,\dots ,g)$    
Compute  $\beta_{1}^{(k)},\dots,\beta_{n}^{(k)}$  as  $\beta_{i}^{(k)} = \pmb{\eta}_{i}^{(k)}\bar{\xi}_{i}^{(k)} / \xi_{i}^{(k)}$  for  $i = 1,\dots ,n$  and  $k = 1,\dots ,g$    
output Group-sparse coefficients  $\beta_{1}^{(k)},\dots,\beta_{n}^{(k)}$ $(k = 1,\dots ,g)$
the norm of each group is projected onto the  $\ell_1$ -ball using Algorithm 1, and then each set of group coefficients is rescaled by the resulting projected norms. These projected norms can be zero after thresholding, yielding sparsity across the groups. For the correctness of Algorithm 2, refer to Theorem 4.1 in van den Berg et al. (2008), which establishes the validity of this type of thresholding.
C Side constraints
To simplify notation here, we refer to  $\pmb{\eta}(\mathbf{z}_i)$  by the shorthand  $\pmb{\eta}_i$ . The  $\ell_1$ -projection with sign constraints is
$$
\min  _ {\boldsymbol {\beta} _ {1}, \dots , \boldsymbol {\beta} _ {n}} \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \| \boldsymbol {\eta} _ {i} - \boldsymbol {\beta} _ {i} \| _ {2} ^ {2}
$$
$$
\text {s . t .} \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \| \beta_ {i} \| _ {1} \leq \lambda \tag {6}
$$
$$
\beta_ {i j} \geq 0, i = 1, \ldots , n, j \in \mathcal {P}
$$
$$
\beta_ {i j} \leq 0, i = 1, \dots , n, j \in \mathcal {N}.
$$
Here,  $\mathcal{P} \subseteq \{1, \dots, p\}$  and  $\mathcal{N} \subseteq \{1, \dots, p\}$  index the explanatory features whose coefficients are restricted nonnegative and nonpositive, respectively. Proposition C.1 states that (6) reduces to a simpler problem directly solvable by Algorithm 1.
Proposition C.1. Let  $\eta_1, \ldots, \eta_n \in \mathbb{R}^p$ . Define  $\tilde{\eta}_1, \ldots, \tilde{\eta}_n$  elementwise as