The Contextual Lasso: Sparse Linear Models via Deep Neural Networks
Authors: Ryan Thompson
Amir Dezfouli
Robert Kohn
Venue: NeurIPS 2023 poster (2023)
Keywords: feature selection
sparsity
sparse regression
varying coefficients
deep learning

Abstract: Sparse linear models are one of several core tools for interpretable machine learning, a field of emerging importance as predictive models permeate decision-making in many domains. Unfortunately, sparse linear models are far less flexible as functions of their input features than black-box models like deep neural networks. With this capability gap in mind, we study a not-uncommon situation where the input features dichotomize into two groups: explanatory features, which are candidates for inclusion as variables in an interpretable model, and contextual features, which select from the candidate variables and determine their effects. This dichotomy leads us to the contextual lasso, a new statistical estimator that fits a sparse linear model to the explanatory features such that the sparsity pattern and coefficients vary as a function of the contextual features. The fitting process learns this function nonparametrically via a deep neural network. To attain sparse coefficients, we train the network with a novel lasso regularizer in the form of a projection layer that maps the network's output onto the space of $\ell_1$-constrained linear models. An extensive suite of experiments on real and synthetic data suggests that the learned models, which remain highly transparent, can be sparser than the regular lasso without sacrificing the predictive power of a standard deep neural network.

The Contextual Lasso:
Sparse Linear Models via Deep Neural Networks
Ryan Thompson' University of New South Wales CSIRO's Data6l
Amir Dezfouli BIMLOGIQ
Robert Kohn University of New South Wales
Abstract
Sparse linear models of several core tools for interpretable machine learning, a field of emerging importance as predictive models permeate decision decision-making in many domains. Unfortunately, sparse linear models are far less flexible as functions their input features than black-box models like deep neural networks. With this capability gap in mind, study not-uncommon situation where the input features dichotomize into two groups: explanatory features, which are candidates for inclusion variables in interpretable model, and contextual features, which select from the candidate variables and determine their effects. This dichotomy leads us the contextual lasso,
statistical estimator that fits sparse linear model to the explanatory features such that the sparsity pattern and coefficients vary function of the contextual features. The fitting process learns this function nonparametrically via deep neural network. To attain sparse coefficients, we train the network with novel lasso regularizer in the form of projection layer that maps the network's 's output onto the space of É --constrrieeed linear models. An extensive suite of experiments on real and synthetic data suggests that the learned models, which remain highly transparent, can be sparser than the regular lasso without sacrificing the predictive power of a standard deep neural network.
Introduction
Sparse linear models-linear predictive functions in a small subset of features- have long history in statistics, dating back at least to the 1960s (Garside, 1965). Nowadays, against the backdrop of elaborate, black-box models such as deep neural networks, the appeal of sparse linear models is largely their transparency and intelligibility. These qualities are sought in decision making settings (e.g.. consumer finance and criminal justice) and constitute the foundation of interpretable machine learning, topic that has recently received significant attention (Murdoch et al., 2019; Molnar et al., 2020; Rudin al., 2022; Mareinkeviěs and Vogt, 2023). Interpretability, however, comes price when the underlying phenomenon cannot be predicted accurately without a more expressive model capable of apro-ippro complex functions, such as neural network. Unfortunately, one must forgo direct interpretation of expressive models and instead resort to post hoc explanations (Ribeiro et al., 2016; Lundberg and Lee, 2017), which have their own flaws (Laugel et al., 2019; Rudin, 2019).
Motivated by a desire for interpretability and expressivity, this paper focuses on a setting where sparse linear models and neural networks can collaborate together. The setting is characterized by a not-uncommon situation where the input features dichotomize into two groups, which we call explanatory features and contextual features. Explanatory features are features whose effects are of primary interest. They should be modeled via a low-complexity function such sparse linear model for interpretability. Meanwhile, contextual features describe the broader predictive context,
Cooresppnningg author. Email: ryan. thompson 1 @unss. edu. I Part of this work was carried out while the author was at CSIRO's Data61.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Intercept
Elevator
Floor
Renovation condition
Latitude
Labtule
Lattude
1.5
1.0
0.04
0.00
0.5
0.02
-0.05
0.0
-0.10
Longitude
Longitude
Longitude
Longitude
Figure 1: Fitted coefficient functions from the contextual lasso for the house pricing dataset. Colored points indicate coefficient values at different locations. Grey points indicate coefficients equal to zero.
e.g. the location of the prediction in time or space (see the house pricing example below). These inform which explanatory features are relevant and, for those that are, their exact low-complexity effects. Given this role, contextual features are best modeled via an expressive function class,
The explanatory-contexue feature dichotomy described above leads to the seemingly previously unstudied contextually sparse linear model:
(Ely x, 2)) = Σ 128,(Z).
(ES(2)
To parse the notation, y E R is response variable, x x= (x (xp
TR are explanatory features, (Z1,..., Zm) E IR contextual features, and is link function (e.g., identity for regression or logit for classification). 3 Via the contextual features, the set-valued function S(z) encodes the indices of the relevant explanatory features (typically, small set of js), while the coefficient functions (z) encode the effects of those relevant features. The model (1) draws inspiration from the varying- coefficient model (Hastic and Tibshirami, 1993; Fan and Zhang, 2008; Park et al., 2015), special case that assumes all explanatory features are always relevant, i.c., S(z) = {l..... p}} for all z e R". We show that this new model is more powerful for various problems, including energy forecasting and disease prediction. For these tasks, sparsity patterns can be strongly context-dependent.
The main contribution of our paper is new statistical estimator for (1) called the contextual lasso. The new estimator is inspired by the lasso (Tibshirani, 1996), a classic sparse learning tool with excellent properties (Hastie et al., 2015). We focus tabular datasets these are the most common use case for the lasso and its cousins. Whereas the lasso fits sparse linear model that fixes the relevant features and their coefficients once and for all (i.e.. S(z) and BJ (z) constant), the contextual lasso fits a contextually sparse linear model that allows the relevant explanatory features and coefficients to change according to the prediction context. To Iearn the map from contextual feature vector to sparse coefficient vector, we use the expressive power of neural networks. Specifically, we train a feedforward neural network output a vector of linear model coefficients sparsified via novel lasso regularizer. In contrast to the lasso, which constraints the coefficients' fn-nrmm, our regularizer constraints the expectation of the coefficients' & f--ormm with respect to To implement this new regularizer, we include a novel projection layer at the bottom of the network that maps the network's output onto the space of e--constraened linear models by solving a convex optimization problem.
To briefly illustrate our proposal, we consider data on property sales in Beijing, China, studied in Zhou and Hooker (2022). We use the contextual lasso to learn a pricing model with longitude and latitude as contextual features. The response is price per square meter. Figure plots the fitted coefficient functions of three property attributes (explanatory features) and intercept. The relevance and effect of these attributes can vary greatly with location. The elevator indicator, e.g., is irrelevant throughout inner Beijing, where buildings tend to be older and typically do not have elevators. The absence of elevators also makes it difficult to access higher floors, hence the negative effect of floor on price. Beyond the inner city, the floor is mostly irrelevant. Naturally, renovations are valuable everywhere, but more so for older buildings in the inner city than elsewhere. The flexibility of the contextual lasso to add remove attributes by location, and simultaneously determine their coefficients, equips sellers with personalized interpretable models containing only the attributes most relevant to them. At the same time, these models outpredict both the lasso and a deep neural network; see Appendix A.
The intercept omitted throughout this paper ease notation.

---

The rest of paper is organized as follows. Section 2 introduces the contextual lasso and describes techniques for its computation. Section discusses connections with earlier related work. Section reports experiments on synthetic and real data. Section 5 closes the paper with discussion.
2 Contextual lasso
This section describes our estimator. To facilitate exposition, we first rewrite the contextually sparse linear model (1) more concisely:
(Ely x, z)) ((z).
The notation (zz) := (z)(z).... (2))) represents a vector coefficient function which is sparse over its codomain. That is, for different values z, the output of ((z) contains zeros at different po- sitions. The function S(z). which encodes the of active explanatory features in (1), is recoverable as S( = (j : 3, (z) 0}.
2.1 Problem formulation
At the population level, the contextual lasso comprises minimization of the expectation of loss function subject to an inequality on the expectation a constraint function:
min BEF
E (x B(2).u)
S. t.
[[]](())]n s
where the set F is a class of functions that constitute feasible solutions and : R2 - the loss function, e.g., square loss 1(g. y) (y 5)² for regression logistic loss I(0. -y log(ū) (1 1--1) 14) log 9) for classification. Here, the expectations are taken with respect to the random variables y, and The parameter À > 0 controls the level of regularization. Smaller values of encourage (z)) towards zero over more of its codomain. Larger values have the opposite effect. The contextual lasso thus generalizes the lasso, which learns (zz) as constant:
min B
E [
v] S.
||3|| S X.
To reiterate the difference: the lasso coaxes the fixed coefficients B towards zero, while the contextual lasso coaxes the expectation of the function B(z) to zero. The result for the latter is coefficients that can change in value and sparsity with provided the function class F is suitably chosen.
Given sample (y,, x,, 2:) the data version of the population problem (2) replaces the unknown expectations with their sample counterparts:
n
min BEF
(x]]((zz))
S. t.
| B3zz) S A. 72 Σ
n
The set of feasible solutions optimization problem (3) coefficient functions that lie in the -ball of radius À when averaged over the observed data." To operationalize this estimator, we take F= (.w((z) w). where Bw (z) is certain architecture of feedforward neural network (described shortly) parameterized by weights w. This choice leads to our core proposal:
1 [ (x] (z:). n
min
t.
Σ || (2,)1l S
Configuring feedforward neural network such that its outputs are sparse and satisfy the ---osstrrintt not trivial. We introduce novel network architecture address this challenge.
2.2 Network architecture
The neural network architecture depicted in Figure 2 2-involves two key components. The first and most straightforward component is vanilla feedforward network (7) Rm - RP. The purpose of the network is to capture the nonlinear effects of the contextual features on the explanatory features.
'The ε -all is the convex compact (x RRP ||x||| & A}.

---

Hidden layers
Projection layer
3:
72
Figure 2: Network architecture. The contextual features pass through series of hidden layers. The resulting dense coefficients ...-... !hp then enter a projection layer to produce sparse coefficients ,I,..., By. Here, the last coefficient is gray to illustrate that is zeroed-out by the projection layer.
Since the network involves only hidden layers with standard affine transformations and nonlinear maps (e.g.. rectified linear activation functions), the coefficients they produce generally do not satisfy the contextual lasso constraint and are not sparse. To enforce the constraint and attain sparsity, employ novel projection layer as the second main component of our network architecture.
The projection layer takes the dense coefficients mz)) from the network and maps them to sparse coefficients (z)) by performing a projection onto the falbll. Because the contextual lasso does not constrain each coefficient vector to the fj-ball, but rather constrains the average coefficient vector, we project all n coefficient vectors (7(2)...... m7.... together. That is, we take the final sparse coefficients ((2)). ((2x) as the minimizing arguments of convex optimization problem:
n
(zz)). ...)....................... ((z.) =
arg min B B En BB
Σ [m(,.) - 3.112.
(5)
72
The minimizers of this optimization problem typically sparse thanks the geometry of the f-abll.. The idea of including optimization as a layer in a neural network is explored in previous works (see, e.g.. Amos and Kolter, 2017; Agrawal et al., 2019). Yet, to knowledge, previous work has studied optimization layers (also known as implicit layers) for inducing sparsity in neural network.
The optimization problem (5) does not admit analytical solution, though it is solvable by general purpose convex solvers (see, e.g., Boyd and Vandenberghe, 2004). However, because (5) is a highly structured problem, it is also amenable more specialized algorithms. Such algorithms facilitate the type of scalable computation necessary for deep learning. Duchi et al. (2008) provide a low- complexity algorithm for solving (5) when
1. Algorithm below is an extension to > 1. The algorithm consists of two main steps: (1) computing a thresholding parameter 0 and (2) soft-
Algorithm 1 Projection onto bi-ball
input Dense coefficients 7h .p..... 7h and radius À Set μ (Im l...., |nm]]
Sort e in decreasing order: μı M4 for all <
Set kimxx max {K pe > (E -mn-- /k)
Set 0
(Eken
nd) /kmix
Compute BI, ... ,,n as 3rg = sign(n)) max(Ingl 0,00) for -1..., and .,....P
output Sparse coefficients BI,...,
thresholding the inputs using the computed 6. Critically, the operations comprising Algorithm are suitable for computation GPU, meaning the model can be trained end-to-end at scale.
Although the projection algorithm involves some nondifferentiable operations, most deep learning libraries provide gradients for these operations, e.g., sort is differentiated by permuting the gradients and abs differentiated by taking the subgradient zero zero. The gradient obtained by automatically differentiating through Algorithm 1 is the same that from implicitly differentiating through convex solver (c.g., using cvxpylayers of Agrawal et al., 2019), though the latter is slower. In subsequent work, Thompson et al. (2023) derive analytical gradients for projection layer that maps matrices onto an ball. Their result readily adapts to the vector case dealt with here.
Computation of the thresholding parameter is performed only during training. For inference, the estimate 9 from the training set is used for soft-thresholding. That is, rather than using Algorithm I

---

as an activation function when performing inference, we use T(x) := sign(æ) max([2]] 8, 0).
The purpose of using the estimate 0 rather than recomputing 0 via the algorithm is because the f1-constraint applies to the expected coefficient vector. It need not be the case that every coefficient vector produced at inference time lies in the f1-ball, which would occur if the algorithm is rerun.
Algorithm 1 computes the thresholding parameter using the full batch of n observations. The algorithm can also be applied to mini-batcher during training. Once training is complete, the estimate i for inference can be obtained via single forward pass of the full batch through the network.
2.3 Grouped explanatory features
In certain settings, the explanatory features may be organized into groups such that all the features in group should be selected together. These groups may emerge naturally (c.g., genes in the same biological path) or be constructed for a statistical task (e.g.. hasis expansions regression). The prevalence of such problems has led to the development of sparse estimators capable of handling groups, one of the most well-known being the group lasso (Yuan and Lin, 2006; Meier et al., 2008). Perhaps unsurprisingly, the contextual lasso extends gracefully to grouped selection.
nonparametric
{L....:P] }} be of g nonoverlapping groups, and let (z) and X be the coefficient function and explanatory features restricted to group Gg- In the noncontextual setting, the group lasso replaces the & f1-norm || of the lasso with sum of group- wise 2--norms Ei=1 | 2. To define the contextual group lasso, we make the analogous modification to (4):
Let G1,.. ......, G,
n
min W
Σ
x Gr (w.)) Vi
BL (2:)l2 S
12
Similar to how the absolute values of the & fn-ormm nondifferentiable zero, which causes individual explanatory features to be selected, the f--norm is nondifferentiable at the zero vector, causing grouped explanatory features be selected together. To realize the grouped estimator, adopt the same architecture as before but replace the previous (ungrouped) projection layer with its grouped counterpart. This change demands different projection algorithm, presented in Appendix B.
2.4 Side constraints
Besides the contextual (group) lasso constraint, our architecture readily accommodates side constraints on B(z) via modifications to the projection. For instance, follow Zhou and Hooker (2022) in the house pricing example (Figure 1) and constrain the coefficients on the clevator and renovation features to be nonnegative. Such sign constraints reflect domain knowledge that these features should not impact price negatively. Appendix presents the details and proofs this extension.
2.5 Pathwise optimization
The lasso regularization parameter A, controlling the size of the f and thus the sparsity level, typically treated as a tuning parameter. For this reason, algorithms for the lasso usually provide multiple models over grid of varying X, which can then be compared (Friedman et al., 2010). Towards this end, it can be computationally efficient to compute the models pathwise by sequentially warm-starting the optimizer. As Friedman al. (2007) point out, pathwise computation for many À can be as fast as for a single X. For the contextual lasso, warm starts also reduce run time compared with initializing at random weights. More importantly, however, pathwise optimization improves the training quality. This last advantage is a consequence of the network's nonconvex optimization surface. Building up sophisticated network from simple one helps the optimizer navigate this surface. Appendix D presents our pathwise algorithm and an approach for setting the grid.
2.6 Relaxed fit
A possible drawback to the contextual lasso, and indeed all lasso estimators, is bias of the linear model coefficients towards zero. This bias, which a consequence of shrinkage from the -nrmm, can help or hinder depending on the data. Typically, bias is beneficial when the number of observations is low or the level of noise is high, while the opposite is true in the converse situation (see, e.g., Hastie et al., 2020). This consideration motivates relaxation of the contextual lasso that unwinds some, or all, of
5

---

the bias imparted by the & norm. We describe an approach in Appendix E that extends the proposal of Hastie al. (2020) for relaxing the lasso. Their relaxation, which simplifies an earlier proposal by Meinshausen (2007), involves a convex combination of the lasso's 's coefficients and "polished" coefficients from an unregularized least squares fit on the lasso's s selected features. We extend this idea from the lasso's ' fixed coefficients to the contextual lasso's varying coefficients. The benefits of the relaxation are demonstrated empirically in Appendix E, where we present an ablation study.
2.7 Computational complexity
A forward or backward pass through the vanilla feedforward component of the network takes O(md + hd2 pd) time, where h is the number of hidden layers, d is the number of nodes per layer, m is the number of contextual features, and p is the number of explanatory features. A forward or backward pass through the projection algorithm takes (pp) time (Duchi al., 2008). The time complexity for pass through the full network over n observations is thus O(n((m + hdd + p)). This result suggests that the training time linear in the sample size n and number of features m and p. Actual training times are reported in Appendix F that demonstrate linear complexity empirically.
2.8 Package
We implement the contextual lasso described in this section in the Julia (Bezanson et al., 2017) package Contextuallasso. For training the neural network, we use the deep learning library Flux (Innes et al., 2018). Though the experiments throughout this paper involve square or logistic loss functions, our package supports any differentiable loss function, e.g., those in the family of generalized linear models (Nelder and Wedderburn, 1972). Contextualaasoo is available at
https://gtttubb.cm//raaaaattt http:///hh...c.oooo/////////o/tt
3
Related work
The contextual explanation networks in Al-Shedivat et al. (2020) are cousins of the contextual lasso. These neural networks input contextual features and output interpretable models for explanatory features. They include the (nonsparse) contextual linear model, special case of the contextual lasso where À = co. In their terminology, the contextual linear model is a "linear explanation" model with a "deterministic encoding" function. They also explore a "constrained deterministic encoding" function that involves a weighted combination of individual fixed linear models with weights determined by the contextual features. To avoid overfitting, they apply & firregularization to the individual models. However, they have no mechanism that encourages the network to combine these sparse models such that the result is sparse. In contrast, the contextual lasso directly regularizes the sparsity of its outputs.
The contextual lasso is also related several estimators that allow varying sparsity patterns. Yamada et al. (2017) devise the first of these-the localized lasso-which fits a linear model with a different coefficient vector for each observation. The coefficients are sparsified using a lasso regularizer that relies on the availability of graph information to link the observations. Yang et al. (2022) and Yoshikawa and Iwata (2022) follow with LLSPIN and NGSLL, neural networks that produce linear models with varying sparsity patterns via gating mechanisms. These approaches are distinct from our own, however. First, they do not dichotomize into x and z, making the resulting model x T 3xx) difficult to interpret. Second, the sparsity level (NGSSL) or nonzero coefficients (LLSPIN) are fixed across observations, making them unsuitable for the contextual setting where both may vary.
Hallac et al. (2015) introduce the network lasso which has different coefficient vector per observation, clustered using a lasso-style regularizer. They consider problems similar ours, for which contextual information is available, but do not impose sparsity on the coefficients. Deleu and Bengio (2021) induce structured sparsity over neural network weights to obtain smaller, pruned networks that admit efficient computation. In our work, we leave the weights as dense and instead induce sparsity over the network's 's output for interpretability. Wang et al. (2020) propose a network quantization scheme with activation functions that output zeros and ones. Though our approach involves an activation that outputs zeros, we also allow a continuous output. Moreover, their end goal differs from ours; whereas they pursue sparsity to reduce computational complexity, pursue sparsity for interpretability.

---

Our work also advances the broader literature at the intersection of feature sparsity and neural networks, an area that has gained momentum over the last few years. See, e.g.. the lassonet of Lemhadri et al. (2021a,b) which selects features in a residual neural network using an f--reullarizer on the skip connection. This regularizer combined with constraints that force feature's weights on the first hidden layer to zero whenever its skip connection is zero. See also Scardapane et al. (2017) and Feng and Simon (2019) for earlier ideas based on the group lasso, and Chen et al. (2021) for another approach. Though related, these methods differ from the contextual lasso in that they involve uninterpretable neural networks with fixed sparsity patterns. The underlying optimization problems also lefferr-wheree these methods regularize the network's ' weights, ours regularizes its outputs.
4 Experiments
The contextual lasso is evaluated here via experimentation on synthetic and real data. As benchmark methods, we consider the (nonsparse) contextual linear model, which uses no projection layer, and deep neural network, which receives all explanatory and contextual features as inputs. We further include the lasso, lassonet, and LLSPIN, which also receive all features. The localized lasso does not scale to the experiments that follow, so we instead compare it with the contextual Jasso on smaller experiments in Appendix G. Appendix H provides the implementation details of all methods.
4.1 Synthetic data
We consider three different settings of increasing complexity: (1) 10 and m = 2, (2) = 50 and m 2. and (3) p 50 and m 5. Within each setting, the sample size ranges from n 1002 to n = 10 The full simulation design is detailed in Appendix I. As a prediction metric, we report the square or logistic loss relative to the intercept-only model. As an interpretability metric, we report the proportion of nonzero features. As a selection metric, we report the Fi-score of the selected features; a value of one indicates all true positives recovered and no false positives. 6 All three metrics are evaluated on a testing set with tuning on a validation set, both constructed by drawing n observations independently of the training set. Figure reports the results for regression (i.e., continuous response).
The contextual lasso performs comparably with most of its competitors when the sample size small. On the other hand, the contextual linear model (the contextual lasso's 's unregularized counterpart) can perform poorly here (its relative loss had be omitted from some plots to maintain the aspect ratio). As increases, the contextual lasso begins to outperform other methods in prediction, interpretability. and selection. Eventually, learns the correct map from contextual features to relevant explanatory features, recovering only the true nonzeros. Though its unregularized counterpart performs nearly as well in terms of prediction for large n, it remains much less interpretable, using all explanatory features. In contrast, the contextual lasso uses just 10% of the explanatory features on average.
The deep neural network's 's performance is underwhelming for most n. Only for large sample sizes does it begin to approach the prediction performance of the contextual lasso. The lassonet often performs somewhere between the two. These three methods should predict equally well for large enough n, though the function learned by the deep neural network and lassonet will remain opaque. The lasso makes some gains with increasing sample size, but lacks the expressive power of the contextual lasso needed to adapt to the complex sparsity pattern of the true model. LLSPIN the only other method to allow for varying sparsity patterns- the second best feature selector for p = 10, though its mediocre performance more generally is likely due to it not exploiting the conttett feature dichotomy and not allowing its nonzero coefficients to change.
4.2 Energy consumption data
We consider a real dataset containing energy readings for home in Mons, Belgium (Candanedo et al., 2017). Besides this continuous response feature, the dataset also contains p = 25 explanatory features in the form of temperature and humidity readings in different rooms of the house and local weather
'The contextual linear model corresponds the contextual explanation network with linear explanation and deterministic encoding in Al-Shedivat et al. (2020). 'The f'l-score =- 2 2TP /2 TP + FP + FN), where TP, FP, and FN are the number of true positive, false positive, and false negative selections.
7

---

Deep neural network
Lasso LLSPN
Relative loss
0.0
Prop. nonzeros
0.5
1.0
F1 of nonzeros
0.5
0.0
10'
10"
10"
10' 10/
10"
10a
10'
10/
10"
10a
10
p= 10, m =
p 50,
=
p= 50,
Sample size
Figure Comparisons on synthetic regression data. Metrics are aggregated over 10 synthetic datasets. Solid points are averages and error bars are standard errors. Dashed horizontal lines in the middle row indicate the true sparsity level.
data. We define several contextual features from the time stamp to capture seasonality: month of year, day of week, hour of day, and weekend indicator. To reflect their cyclical nature, the first three contextual features are transformed using sine and cosine, leading to m contextual features.
The dataset, containing n. 19, 19, observations, is randomly split into training, validation, and testing sets in 0.6-0.2-0.2 proportions. We repeat this random split 10 times, each time recording performance the testing set, and report the aggregate results in Table 1. As performance metrics, we consider the relative loss and average sparsity level (i.e., average number of selected explanatory features). Among all methods, the contextual lasso leads the lowest test loss, outperforming even
Table Comparisons on the energy consumption data. Metries are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative
Avg. sparsity
Deep neural network
0.433 0.004
25.0 0.0
Contextual linear model
0.387 + 0.008
25.0 0.0
Lasso
0.690 + 0.002
11.6
Lassonet LISPPN
0.423 = 0.003
2.0
0.639 0.005
2..5
Contextual lasso
0.356 + 0.003
2.8 0.4
the deep neural network and lassonet." Importantly, this excellent prediction performance is achieved while maintaining high level of interpretability. In contrast most other methods, which use all (or nearly all) available explanatory features, the predictions from the contextual lasso arise from linear models containing just 2.8 explanatory features on average! These linear models are also much simpler than those from the lasso, which typically involve more than four times as many features.
'The lassonet with tuned À uses nearly all features here. However, manually choosing X to attain a sparsity level similar the contextual lasso substantially degrades its performance.

---

Prop. nonzeros
0.4
0.3
0.2
0.2
0.0
15
20
Hour of day
Figure 4: Explanatory feature sparsity as function of hour of day for the estimated energy consump- tion model. The sparsity level varies within each hour because the other contextual features vary.
The good predictive performance of the contextual lasso suggests a seasonal pattern of sparsity. To investigate this phenomenon, we apply the fitted model randomly sampled testing set and plot the resulting sparsity levels as function of the hour of day in Figure 4. The model is typically highly sparse in the late evening and early morning. Between 10 pm and 6 am, the median proportion of nonzero coefficients is 0%. There is likely little or no activity inside the house at these times, so sensor readings from within the house which constitute the majority of the explanatory features are irrelevant. The number of active explanatory features rises later in the day. peaking around lunchtime and dinnertime. Overall, a major benefit of the contextual lasso, besides its good predictions, the ability to identify parsimonious set of factors driving energy use at any given time of day.
4.3 Parkinson's telemonitoring data
We illustrate the contextual lasso on grouped explanatory features using data from study on the progression of Parkinson's 's disease in 42 patients (Tsanas et al., 2009). The task is to predict disease progression (a continuous variable) using 16 vocal characteristics of the patients as measured different times throughout the study. As Tsanas et al. (2009) point out, these vocal characteristics can relate nonlinearly to disease progression. To account for these effects, we compute five-term cubic regression spline per explanatory feature (p = 16 x 5 = 80). Each spline forms a single group of explanatory features (g- 16). The contextual features are the age and sex of the patients (m = 2).
The dataset of n = 5, 875 observations is again partitioned into training, validation, and testing sets in the same proportions as before. As new benchmark, evaluate the group lasso, which is applied to splines of all explanatory and contextual features. The deep neural network and lassonet are applied to the original (nonspline) features." The lasso is also applied to the original features to serve as linear benchmark. The results are reported in Table 2. The purely linear estimator-the lasso-performs
Table 2: Comparisons on the Parkinson's 's telemonitoring data. Metrics are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative loss
Avg. sparsity
Deep neural network
0.367 1 0.015
16.0
Lasso
0.885 1 0.005
Group lasso
0.710 + 0.006
4.2
Lassonct
0.263 = 0.007
1.2
Contextual group lasso
0.113 = 0.006
1.6 0.3
worst overall. The group lasso improves over the lasso, supporting claims of nonlinearity in the data. The contextual group lasso is, however, the star of the show. Its models predict nearly three-times better than the next best competitor (lassonet) and sparser than those from any other method.
Setting aside predictive accuracy, major benefit of the contextual group lasso (compared with the deep neural network and lassonet) is that it remains highly interpretable. To illustrate, we consider the fitted spline function (i.e., the spline multiplied by its coefficients from the contextual group lasso) on the detrended fluctuation analysis (DFA) feature, which characterizes turbulent noise in speech. Figure plots the function at three different ages patient. For 70-year-olds, the function zero, indicating DFA is not yet good predictor of Parkinson's ss At 75, the function becomes nonzero,
Innuttinn the splines to these methods does not improve their performance.

---

-50
ftx)
110
-100
0.9
70-year-old
75-year-old
80-year-old
Detrended flucation analysis
Figure 5: Fitted spline function from the contextual lasso for the detrended fluctuation analysis (DFA) explanatory feature. The age explanatory feature is varied while the sex feature is set female.
taking on concave shape. It becomes even more concave and negative 80. The models reported in Tsanas et al. (2009) also had this coefficient negative, but fixed across all ages. In contrast, the contextual lasso identifies DFA and other features as relevant only for patients of certain ages and sex.
4.4 Additional experiments
Experiments for classification on synthetic and real data are available in Appendix J. In Appendix K, we report high-dimensional experiments with p 1, 000 and fixed coefficient experiments (i.e., the lasso's 's home court). Appendix L. investigates the stability of the contextual lasso with respect to the random initialization. Appendix M provides hyperlinks the datasets used throughout the paper.
Discussion
Contextual sparsity is important extension of the classical notion of feature sparsity. Rather than fix the relevant features once and for all, contextual sparsity allows feature relevance to depend on the prediction context. To tackle this intricate statistical learning problem, we devise the contextual lasso. This new estimator utilizes the expressive power of deep neural networks to learn interpretable sparse linear models with sparsity patterns that vary with the contextual features. The optimization problem that defines the contextual lasso is solvable at scale using modern deep learning frameworks. Grouped explanatory features and side constraints are readily accommodated by the contextual lasso's neural network architecture. An extensive experimental analysis of the new estimator illustrates its good prediction, interpretation, and selection properties various settings. To the best of our knowledge, the contextual lasso is the only tool currently available for handling the contextually sparse setting.
The problem of deciding the explanatory-contese varying-coeffictt models. Though the literature on varying-coefficieet models is extensive, there are no definitive rules for partitioning the features in general. In the housing and energy examples, the contextual features are spatial or temporal effects, which are distinct from the remaining features. In the telemonitoring example, the patient attributes (age and sex) differ fundamentally from the vocal characteristics. Ultimately, the partition for any given application should be guided by domain expertise with consideration to the end goal. If one needs interpret the exact effect of a feature, that feature should be an explanatory feature. If a feature's 's effect is of secondary interest, or it is suspected that the feature influences the structural form of the model, that feature should be a contextual feature. If the user determines there are no contextual features, the ordinary lasso is more appropriate tool.
feature split is the same as that faced with
remains an important avenue of future research establish solid theoretical foundation for the contextual lasso. The statistical properties of the lasso in terms of estimation, prediction, and selection are now well-established in theory (Bunea et al., 2007; Raskutti et al., 2011; Shen al., 2013; Zhang et al., 2014). The synthetic experiments in our paper suggest that the contextual lasso satisfies similar properties, though theoretically establishing these results is challenging. Statistical convergence results for vanilla feedforward neural networks (e.g., Schmidt-Hieber 2020) do not apply in our setting due to the projection layer. Moreover, to our knowledge, no statistical guarantees exist for neural networks configured with convex optimization layers that otherwise might apply here. It is also important to understand when the contextual lasso's 's performance matched by a deep neural network, since both should predict well for large samples in the contextually sparse linear regime.
10

---

References
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, volume 32, 2019.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623-2631, 2019.
Maruan Al-Shedivat, Avinava Dubey, and Eric Xing. Contextual explanation networks. Journal of Machine Learning Research, 21:1-44, 2020.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as layer neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 136-145, 2017.
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM Review, 59:65-98, 2017.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Patrick Breheny and Jian Huang. Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors. Statistics and Computing, 25:173-187, 2015.
Florentina Bunea, Alexandre B Tsybakov, and Marten H Wegkamp. Aggregation for Gaussian
regression. Annals of Statistics, 35(4):1674-1697, 2007.
Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in low-energy house. Energy and Buildings, 140:81-97, 2017.
Yao Chen, Qingyi Gao, Faming Liang, and Xiao Wang. Nonlinear variable selection via deep neural networks. Journal of Computational and Graphical Statistics, 30:484-492, 2021.
Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learning, 2021. arXiv: 2102.03869.
John Duchi, Shai Shalev- Shwwarzz Yoram Singer. and Tushar Chandra. Efficient projections onto the ba-ball for learning in high dimensions. In Proceedings of the 25th International Conference Machine Learning, pages 272-279, 2008.
Jianqin Fan and Wenyang Zhang. Statistical methods with varying coefficient models. Statistics and Its Interface, 1:179- 17799-19,,91 195, 2008
Jean Feng and Noah Simon. Sparse-input neural networks for high-dimensional nonparametric regression and classification, 2019. arXiv: 1711.07592.
Kelwin Fernandes, Pedro Vinagre, and Paulo Cortez. A proactive intelligent decision support system for predicting the popularity of online news. In Progress in Artificial Intelligence, volume 9273, pages 535 546, 2015.
Jerome Friedman, Trevor Hastie, Holger Höfling, and Robert Tibshirani. Pathwise coordinate optimization. Annals of Applied Statisties, 1:302-332, 2007.
Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33:1-22, 2010.
M J Garside. The best sub-set in multiple regression analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 14:196-200, 1965.
David Hallac, Jure Leskovec, and Stephen Boyd. Network lasso: Clustering and optimization in large graphs. In Proceedings of the 2/ ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 387-396, 2015.

---

Trevor Hastie and Robert Tibshirani. Varying- models. Journal of the Royal Statistical Society: Series (Methodological), 55:757-796, 1993.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations, CRC Press, 2015.
Trevor Hastie, Robert Tibshirani, and Ryan Tibshirani. Best subset, forward stepwise or lasso?
Analysis and recommendations based on extensive comparisons, Statistical Science, 35:579-592, 2020.
Michael Innes, Elliot Saba, Keno Fischer, Dhairya Gandhi, Marco Concetto Rudilosso, Neethu Mariya Joy, Tejan Karmali, Avik Pal, and Viral B Shah. Fashionable modelling with Flux. In Workshop on Systems for ML and Open Source Software at NeurIPS 2018, 2018.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marein Detyniecki. The dangers of post-hoe interpretability: Unjustified counterfactual explanations. Proceedings of 2019.
the 28th International Joint Conference on Artificial Intelligence, pages
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature spansity. Journal of Machine Learning Research, 22:1-29, 2021a.
Ismael Lemhadri, Feng Ruan, and Robert Tibshirani. Lassonet: Neural networks with feature sparsity. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, volume 130. pages 10-18, 2021b.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30, 2017.
Ričards Marcinkevičs and Julia E Vogt. Interpretable and explainable machine learning: A methods- centric overview with concrete examples. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13, 2023.
Lukas Meier, Sara van de Geer, and Peter Bühlmann. The group lasso for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70:53-71, 2008.
Nicolai Meinshausen. Relaxed lasso. Computational Statistics and Data Analysis, 52:374-393, 2007.
Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable machine learning A brief history, state-of-the-ar and challenges. In ECML PKDD 2020 Workshops, volume 1323, pages 417-431, 2020.
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences of the United States of America, 11:222071-22088 2019.
J A Nelder and W M Wedderburn. Generalized linear models. Journal of the Royal Statistical Society: Series A (General). 135:370-384, 1972.
Byeong Park, Enno Mammen, Young K Lee, and Eun Ryung Lee. Varying coefficient regression models: A review and new developments. International Statistical Review, 83:36-66 64, 2015.
Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Minimax rates of estimation for high-dimensionall linear regression over balls... IEEE Transactions on Information Theory, 57710)4697--694, 2011.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1:206-215, 2019.
12

---

Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1 16:, 2022.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regular- ization for deep neural networks. Neurocomputting. 241:81-88 2017.
Johannes Schmidt- Hieber. Nonparametric regression using deep neural networks with ReLU activa- tion function. Annals of Statistics, 48:1875-1889 2020.
Xiaotong Shen, Wei Pan, Yunzhang Zhu, and Hui Zhou. On constrained and regularized high- dimensional regression. Annals of the Institute of Statistical Mathematics, 65(5):807-832, 2013.
Ryan Thompson, Edwin V Bonilla, and Robert Kohn. Contextual directed ayclic graphs, 2023. arXiv:
15627.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58:267-2881 1996.
Athanasios Tsanas, Max A Little, Patrick E McSharry, and Lorraine O Ramig. Accurate telemon- itoring of Parkinson's 's disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering, 57:884-893, 2009.
Ewout van den Berg, Mark Schmidt, Michael Friedlander, and Kevin Murphy. Group sparsity via linear-time projection. Technical Report TR-2008-09, Department of Computer Science, University of British Columbia, 2008
Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12192-12199, 2020.
Makoto Yamada, Koh Takeuchi, Tomoharu Iwata, John Shawe Tayyor,, and Samuel Kaski. Localized lasso for high-dimensional regression. Proceedings of the 20th International Conference Artificial Intelligence and Statistics, 54:325-333. 2017.
Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally sparse neural networks for tabular biomedical data. In Proceedings of the 39th International Conference Machine Learning, volume 162, pages 25123-25153, 2022.
Yuya Yoshikawa and Tomoharu Iwata. Neural generators of sparse local linear models for achieving both accuracy and interpretability. Information Fusion, 81:116-128, 2022.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68:49-67, 2006.
Yuchen Zhang, Martin Wainwright, and Michael Jordan. Lower bounds on the performance of polynomial-time algorithms for sparse linear regression. In Proceedings of the 27th Conference on Learning Theory, volume 35, pages 921-99119911191 221-948, 2014.
Yichen Zhou and Giles Hooker. Decision boosted varying coefficient models. Data Mining and Knowledge Discovery, 36:2237-2271, 2022.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series (Statistical Methodology). 67:301-320, 2005.
13

---

A House pricing data
We expand the analysis in Section 1 and compare the statistical performance of the contextual lasso with its competitors using the house pricing data. Following Zhou and Hooker (2022), the explanatory features the elevator indicator, renovation condition, floor of the apartment, and number of living rooms and bathrooms (p 5). The contextual features are longitude and latitude (m = 2). We randomly sample training, validation, and testing sets of size n 15, 000 and report the results across 10 random data splits in Table 3. The contextual lasso delivers sparse models and with competitive
Table Comparisons of methods on the house pricing data. Metrics are aggregated over random splits of the data. Averages and standard errors are reported.
Relative loss
Avg. sparsity
Deep neural network
0.515 0.003
5.0
Contextual linear model
0.505 1 0.002
5.0 0.0
Lasso
0.892 1 0.001
5.0 0.0
Lassonet
0.521 + 0.003
5.0
LLSPIN
0.729 0.031
4.2
Contextual lasso
0.498 + 0.002 2.9 0.3
prediction performance. The contextual linear model trails closely in prediction, though it does not offer similar level of parsimony. While also producing good predictions, the deep neural network and lassonet do not offer the same interpretability. The lasso lags far behind the contextual lasso and other neural network-based methods, suggesting that the contextual features have nonlinear effects.
B Grouped explanatory features
Algorithm 2 presents the routine for projecting onto the group e b-bll.. To summarize the algorithm,
Algorithm 2 Projection onto group -ball
input Dense group coefficients 17/ ..... nh (k 1,..... 9) and radius
Compute group-wise norms c(*) [17, 1l2 for i 1, ....., and & 1 .......
(k)
Run Algorithm with (k)
c (Is 1,...,g) () and A to get 214)
El (k=1,....
Compute g(a) ..... h as g(() n (1) g( (e(h) for i 1, .,..., and lo =1,.. 1,...,9
output Group Group-sparse coefficients ((a) ..... 2 (k 1, .9....9)
the norm of each group is projected onto the -all using Algorithm 1, and then each set of group coefficients is rescaled by the resulting projected norms. These projected norms can be zero after thresholding, yielding sparsity across the groups. For the correctness of Algorithm 2, refer to Theorem 4.1 in van den Berg et al. (2008), which establishes the validity of this type of thresholding.
C
Side constraints
To simplify notation here, we refer (7z)) by the shorthand n, The 4 l--rojectioon with sign constraints is
min B B.......
|| Ball
s.
|| A
1=1
Bu > 0, i I,..., E P
Bij i0 j, E.
Here, P C (1,....,p) and N C (1,...,,p index the explanatory features whose coefficients are restricted nonnegative and nonpositive, respectively. Proposition C.I states that (6) reduces simpler problem directly solvable by Algorithm
14

---

Proposition C.1. Let mh
E R². Define ....... nn elementwise as
if THi OAjEP if MH >OjEW, i= ......................... ......p otherwise
nl
Then optimization problem (6) admits the same optimal solution as
min B
Mi
3:112
S. L.
| S A.
==1
The proof of this proposition requires the following lemma. Lemma C.2. A solution B Br. to (6) must satisfy the inequality Th 2 for all and j.
Proof. We proceed using proof by contradiction along the lines of Lemma
Duchi al. (2008).
Suppose there exists solution BI..... B,, such that 7syBi < 0 for some and j. Take .r...., 3 equal .......BB. G, except at index (i.j), where we set Biy = O. Note that f ...... Sr continues to satisfy the sign constraints and L fy-oonstraint, and hence remains feasible for (6). We also have that
n
Σ 11 - 3.112
Σ 17 3
B
2ny > Big 0.
Thus, Bi...., attains lower objective value than solution Bi,.... Br. This contradiction yields the statement of the lemma.
The proof of Proposition C.I now follows.
Proof. If Miy < 0 and E P, then by Lemma C.2, solution to (6) must satisfy B.g = 0. Likewise, My and N,, then it must hold that Bij 0. Hence, by setting any The that violate the sign constraints to zero (i.c., îkj), and noting that a solution to (7) must satisfy B.g = 0 when Mh we arrive at the result of the proposition.
D Pathwise optimization
In a spirit similar to Friedman et al. (2007), we take the sequence of regularization parameters (A(9), as a grid of values that yields a path between the unregularized model (no sparsity) and the fully regularized model (all coefficients zero). Specifically, we set ((1) such that the contextual lasso regularizer does not impart any regularization, i.c., (1) = n E.= [[swer (z.)ll.. where the weights (1) are a solution to (4) from setting À = co. We then construct the sequence as a grid of linearly spaced values between (1) and (7)) O, the latter forcing all coefficients to zero. Linear spacing of the sequence of ( generally yields linearly spaced sparsity levels. The sequence should be decreasing the optimizer can build networks that increase in sparsity.
Algorithm 3 summarizes the complete pathwise optimization process, with gradient descent employed as the optimizer. To parse the notation used in the algorithm, L(w; X) = n En= ((x, Bw (z,), vi) the loss as function of the network's 's weights w given X, and V Vw X) its gradient.
E Relaxed fit
Denote by (x(z) contextual lasso network fit with regularization parameter X. To unwind bias in (x(z,, we train polished network (z( that selects the same explanatory features but does not impose any shrinkage. For this task, we introduce the function S (z) 1Tmh + {0, 1)P that outputs
vector with elements equal to wherever 3g is nonzero and elsewhere is zero. We then fit the
15

---

Algorithm 3 Pathwise optimization
input Initial weights ( step size a, and numher of regularization parameters T Initialize (1) oC
for f= ,,...,, do
Initialize W(o = w (t-1)
Initialize m = 0
while Not converged do
Update ((m+1) W(m) a V [w)(W
Update m m +
end while
(()) ((m)
= then
Set ( n Σ 1 m (2))l1 1 and A =
Equispace (2)) ,..., (3--1) between ( and (
end if
end for
output Fitted weights (1)), (
polished network (z) 17())= (x),,, where means clement- multiplication and 17 is the same architecture as used for the original contextual lasso network before the projection layer. The effect of including (z), which is fixed when training B (z), is twofold. First, guarantees the coefficients from the polished network are nonzero in the same positions as the original network, i.e., the same features are selected. Second, it ensures explanatory features only contribute gradients for observations in which they are active, i.e., Tij does not contribute if the jth component of $ (z.)) zero. Because the polished network does not project onto -balll, its coefficients are not shrunk.
To arrive the relaxed contextual lasso fit, we combine (z) and the fitted polished network / (z):
(x) == (1 7) + (2),, S S 1.
(8)
When y = O, we recover the original biased coefficients, and when 1, we attain the unbiased polished coefficients. Between these extremes lies a continuum of relaxed coefficients with varying degrees of bias. Since the original and polished networks need only be computed once, we may consider any relaxation on this continuum at virtually no computational expense over and above that of the two networks. In practice, we choose among the possibilities by tuning y on validation set.
To illustrate the benefits of relaxation, we present Figure 6, which compares the relaxed and nonrelaxed variants of the contextual lasso under the synthetic experimental design of Section 4. As far as prediction accuracy is concerned, the benefits of relaxation are marginal. However, for selection and interpretation, the story is quite different. The relaxation yields models that are both sparser and contain more true positives and/or fewer false positives. These gains are most pronounced in larger samples. In smaller samples, relaxation is less beneficial because the bias from shrinkage helps stabilize the models. Yet, the relaxed variant of the contextual lasso typically does no worse than its nonrelaxed counterpart because it adapts to the best level of bias (shrinkage) by tuning ?
F
Run times
The complexity analysis in Section 2.7 suggests that the training time for the contextual lasso should be linear in the sample size n and number explanatory features p. To verify this result, we record the time taken to fit the contextual lasso over sequence of 50 values of X. The number of hidden layers is three and the number of neurons per layer 100. Figure reports the results. The run times are indeed linear as a function of and and, overall, quite reasonable for real world applications.
G Localized lasso
We compare the contextual lasso with the localized lasso of Yamada et al. (2017). The graph information required by the localized lasso estimated from the contextual features using the nearest neighbors approach of Yamada et al. (2017). We focus on smaller sample sizes than in the main
16

---

Contextual lasso (relaxed) Contextual lasso (non-relaxed)
1.0
Relative loss
0.5
Prop. nonzeros
0.5
of nonzeros
10'
10"
10"
10' 10/
10"
10a
10'
10/
10"
10a
10'
p= 10, m
p= 50,
=
p= 50,
=
Sample size
Figure 6: Relaxation ablation on synthetic regression data. Metrics are aggregated over synthetic datasets. Solid points are averages and error bars are standard errors. Dashed horizontal lines in the middle row indicate the true sparsity level.
Run time
30
10
20
10
8000
1001
Sample size
Number of explanatory features
Figure 7: Run time in seconds to fit the contextual lasso over a sequence 50 values of X, measured over 10 synthetic datasets. The number of explanatory features 50 in the left plot and the sample size n 000 in the right plot. The number of contextual features m 5. Solid points are averages and error bars are one standard errors.
experiments since each observation requires new coefficient vector that each constitute additional optimization variables. Figure 8 reports the results for regression (the localized lasso does not support classification). The localized lasso's prediction loss improves with growing n, but nearly as well as the contextual lasso. The graph information may be insufficient to encode the underlying nonlinearity. Furthermore, the localized lasso's 's regularizer never induces fully sparse coefficient vectors (all zeros), which may be limiting if there are no relevant explanatory features for certain z.
H Implementation details
The contextual lasso fit using our Julia package Contextuallasso. The network is configured with three hidden layers. The number of neurons, which are spread equally across these hidden layers, set so that the dimensionality of the weights is approximately the network size scales roughly linearly with the number of features. The contextual linear model uses the same architecture, excluding the projection layer. The deep neural network is set up similarly. These methods all use rectified linear activation functions in the hidden layers and are optimized
x p m. This setting ensures
17

---

Locelized lasso . Contextual lasso
1.0
0.0
1014
10:3
10
101.
1079 1050
1014
1070
10F1
1070 1050
1015
1070
1077
1079
Relative loss
Prop, of nonzeros
F1 of nonzeros
Sample size
Figure 8: Comparisons with the localized lasso synthetic regression data. Metries aggregated over 10 synthetic datasets. Solid points are averages and error bars are standard errors. The dashed horizontal line in the middle indicates the true sparsity level.
using Adam (Kingma and Ba, 2015) with learning rate of 0.001. Convergence is monitored validation set with the optimizer terminated after 30 iterations without improvement.
The lasso is fit using the Julia package GLMNet (Friedman et al., 2010). The group lasso is fit using the R package grpreg (Breheny and Huang, 2015). Since contextual features are always relevant, the regularizer for the lasso is applied only to explanatory features and interactions, not the contextual features." 'Te (group) lasso and contextual lasso all allow for relaxed fits, as discussed in Section 2.6. The regularization parameter À is swept over grid of 50 values computed automatically from the data using Contextuallasso, GLMNet, or grpreg. For each value of the relaxation parameter ? is swept over the grid {0, 0.1, 1}.
The lassonet is fit using the Python package lassonet (Lemhadri et al., 2021b), which also performs its own relaxation. LLSPIN is fit using the authors' Python implementation (see Yang et al., 2022). Their implementation relies on the hyperparameter optimization framework Optuna (Akiba al., 2019) to determine the regularization parameter and learning rate. We use the default grid for tuning the learning rate, but increase the maximum regularization parameter to 10, which is roughly the smallest value required to achieve fully sparse solution in our experiments. LLSPIN does not shrink and so does not admit a relaxation. Lassonet and LLSPIN use the same convergence criterion, number of regularization parameters, and network configuration as the other deep learning methods.
For all methods, the input features standardized prior training. Standardization of the planatory features is necessary for the lasso estimators as it places all coefficients on the same scale, ensuring equitable regularization. ContextualLasso automates standardization and expresses all final coefficients on their original scale.
All experiments are on Linux platform with NVIDIA RTX 4090 GPUs.
I Synthetic data generation
The explanatory features X1. Xn, are generated iid as p-dimensional No, Σ) random variables, where the covariance matrix Σ has elements Σg = 0.5 The contextual features Zı...., .,, Z, are generated iid m-dimensional random variables uniform [-1, 1jmh, independent of the x,. With the features drawn, we simulate a regression response as
3 N(4,1)
K x, B(z,),
or a classification response as
1
y4 Bernoullipp)).
p. 1 exp ( (-R: x ((z)))
for = 1,..., Here, & > 0 controls the signal strength vis-à-vis the variance of x, B(zz). We first estimate the variance of x B(z:) on the training set and then set x so the variance of the signal
ggrpreg does provide support for this functionality.
18

---

1.0
B
1.0
Ba
0.9
0.8
-0.5
0.7
0.0
-1.0
0.5
Figure 9: Illustration of coefficient function (9) for 3 explanatory features and m contextual features. The centers Cj correspond the dark red in the middle of each sphere.
five. The coefficient function B(z) == to nonzero value whenever lies within hypersphere of radius rj centered at cy:
is constructed such that 3z(z.) maps
2r, 2: -cyll
if |2, 0yl1a srj
(z))
0
otherwise
This function attains the maximal value one when z; Cj and the minimal value zero when || Cyll2 > Tj. The centers CI....,yy are generated with uniform probability on [-1, 1| and the radii r Tp are chosen to achieve sparsity levels that vary between 0.05 and 0.15 (average 0.10). Figure 9 provides a visual illustration. This function is inspired by the house pricing example in Figure 1, where the coefficients are nonzero some central region of the contextual feature space.
Classification results
J.1
Synthetic data
Figure 10 reports the experimental results for classification on synthetic data, analogous to those for regression in Section 4. The findings are broadly in line with the regression ones. The contextual lasso performs best overall and is the only method ever able to recover the true nonzeros accurately.
J.2 News popularity data
We turn real dataset of articles posted to the news platform Mashable (Fernandes al., 2015). The task is to predict if an article will be popular, defined in Fernandes et al. (2015) as more than 1 1400 shares. In addition to the zero-one response feature for popularity, the dataset has predictive features that quantify the articles (e.g., number of total words, positive words, and images). The data channel feature, which identifies the category of the article (lifestyle, entertainment, business, social media, technology, world, viral), is taken as the contextual feature. is expressed as a sequence of indicator variables yielding m contextual features. There remain p = explanatory features.
Table reports the results over 10 random splits of the dataset's 39, 643 observations into training, validation, and testing sets in 0.6-0.2-0.2 proportions. contrast the previous datasets, all
Table 4: Comparisons on the news popularity data. Metrics are aggregated over 10 random splits of the data. Averages and standard errors are reported.
Relative loss
Avg. sparsity
Deep neural network
0.903 + 0.003
51.0 + 0.0
Contextual linear model
0.906 + 0.003
51.0
Lasso
0.914 0.002
2.7
Lassonet
0.894 0.002
50.7 + 0.3
LLSPIN
0.923 1 0.009
51.0 1: 0.0
Contextual lasso
0.906 1 0.002
12.6
12.60.99 0.9
methods predict similarly well here. The lassonet performs marginally best overall, while L.ISPIN performs marginally worst. Though predicting neither best worst, the contextual lasso retains
19

---

Deep neural network
Lasso
LLSPN
Relative loss
Prop. nonzeros
0.5
1.0
F1 of nonzeros
0.5
10'
10"
10
10' 10/
10a
10a
10'
10/
10a
10a
10'
p= 10, m =2
p 50,
=
p= 50,
=
Sample size
Figure 10: Comparisons on synthetic classification data. Metrics are aggregated over 10 synthetic datasets. Solid points are averages and error bars are standard errors. Dashed horizontal lines in the middle row indicate the true sparsity level.
significant lead in terms of sparsity, being twice sparse as the next sparsest method (lasso). Sparsity crucial for this task allows author editor focus on a small number changes necessary to improve the article's s likelihood of success. Other methods such as the uninterpretable deep neural network, or fully dense contextual linear model, are not nearly as useful for the same purpose.
K High-dimensional and fixed coefficient results
Besides interpretability, major appeal of the lasso is its good performance in high-dimensional regimes, where the number of features is comparable to the sample size. It intriguing to consider whether the contextual lasso remains useful in this setting. To this end, we extend the experiments of Section p 1, 000 explanatory features. Typically, when there are so many features, only small number are relevant for predicting the response,
adjust the synthetic data generation process so that only 10 explanatory features are relevant some values of the contextual features z. The remaining 990 explanatory features remain irrelevant for all z. Figure I1 reports the results. The contextual lasso performs highly competitively, even when n < 1, 000 and there are more explanatory features than observations. As with the lower-dimensional experiments, the contextual lasso still filter out the irrelevant explanatory features and achieves near-perfect support recovery for large n.
A second regime that might arise in practice is where contextual features are present but have no effect on the explanatory features. That is, the explanatory features have fixed sparsity pattern and fixed coefficients for all z. Figure 12 reports the results in this setting. Here, the contextual lasso is outperformed by the lasso (which in home territory) and lassonet, both of which assume the sparsity pattern is fixed. Nonetheless, the contextual lasso is able to recover the true nonzeros by learning a constant function for (z) via the bias term in each neuron. Provided the sample size sufficiently large, is reasonable to expect the contextual lasso to remain competitive with the lasso.
20

---

Deep neural network
Lasso
LLSPN
1000
0.0
10
10a
1014
10 1020
1021
16
10
10t0
102
10.00
1012
10t0
Relative loss
Prop. of nonzeros
nonzeros
Sample size
Figure 11: Comparisons high-dimensional synthetic regression data. Metrics are aggregated over 10 synthetic datasets, Solid points are averages and error bars are standard errors. The dashed horizontal line in the middle indicates the true sparsity level.
Deep neural network Lasso
Contextual linear model Lassonet Contextual lasso
1.0
100.
10a
107
10
10a 10
107
10a
10 10
10'
10a
10
Relative loss
Prop. of nonzeros
F1 of nonzeros
Sample size
Figure 12: Comparisons on fixed coefficient synthetic regression data. Metrics are aggregated over 10 synthetic datasets. Solid points are averages and error bars are standard errors. The dashed horizontal line in the middle indicates the true sparsity level.
L Stability analysis
Since the contextual lasso involves a highly-parameterized neural network, it insightful to consider its selection stability when trained using different random initializations of the network weights. To this end, we report Figure 13, which shows selection instability as measured by the Hamming distance (scaled by p) between two independently initialized networks. Unsurprisingly, the statistically difficult
0.3
Selection instabilit
0.0
10'
10a
10'
10
10
10a
10'
10'
10'
10a
10'
p= 10, m =
p 50, m
p= 50, m =
Sample size
Figure 13: Selection instability of the contextual lasso over 10 synthetic datasets. Solid points represent averages and emrrr bars denote standard errors.
regimes, where the sample size small or the number of explanatory/contextuaa features large, correspond to higher instability. In any case, the contextual lasso grows increasingly stable with the sample size. Even when p = and = the contextual lasso almost entirely stable by the time 10, 000, which is typical of our real data examples. The concave shape of the stability curve
21

---

when p = 50 is consistent with the proportion of nonzeros reported in Figure 3, where the contextual lasso initially produces highly sparse models that more restricted and hence more stable.
M
Dataset availability
The datasets used throughout this paper are publicly available at the following URLs.
House pricing data in Section 1: attps://www.aagg) com htmm///attaeeetsrrruuur/iaaaaaaa
Energy consumption data in Section 4:
https://archive. ics. uci.
Parkinson''s telennnirinng data in Section 4:
https://archive. ics www.uuci.e..... http://www//////////////////aaaa
News popplarity data Appendix J:
https://archivve www.:iiisiiiiiii..... /ttpt//////tttttt/////////ooooor
N
Limitations
The contextual lasso has strong prediction, interpretation, and selection properties. Naturally, it also has several weaknesses. As with the ordinary lasso, the contextual lasso may select single feature from a group of highly correlated features. This shortcoming could be remedied by introducing an L3 l2-regularizer in spirit similar to the elastic (Zou and Hastie, 2005). Another potential drawback is that the contextual lasso does not guarantee the complete exclusion of an explanatory feature. That is, it cannot ensure the coefficient for a feature nonzero for every possible z, which might limit interpretability in certain settings. Full exclusion could be achieved by adding an é re-reuulari on the weights of the first layer, similar to the lassonet. Finally, as is typical neural networks, the objective function of the contextual lasso is nonconvex, which can complicate analysis of its properties.
22


Reviews:
[{"review_id": "F2fYG9wnZq", "reviewer": "Reviewer_Eybr", "summary": "This paper proposes a contextual lasso, which makes the coefficients of features sparse depending on the context.\nUnlike the standard lasso, the contextual lasso regularizes prediction loss with the expectation of L1 regularization.\nThe sparse coefficients are produced by neural networks whose last layer is a projection layer that maps dense coefficients onto l1-ball.\nThe experimental results on synthetic and real datasets show that the contextual lasso outperforms the linear lasso and 'dense' contextual lasso while achieving the high sparseness of the coefficients.\n\n", "questions": "- When contextual features are useful but explanatory features are uninformative, I think that the predictions using the contextual lasso would be worse than the standard neural networks, which use both contextual and explanatory features as input. Is there any solution to this?\n- It would be easier to read this paper if figures were displayed at the top of the page.", "limitations": "I could not find the description of the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, this paper is well-organized and well-written. Section 2, which describes the proposed method, is easy-to-follow.\nThe proposed contextual lasso is technically sound, and the high expandability similar to the lasso is described.\nThe experiments are organized well, and the experimental results are convincing.\n", "weaknesses": "In the experiments, the contextual lasso is assessed only on tabular datasets with relatively low dimensional features.\nTherefore, it is unclear whether the contextual lasso consistently works well for high-dimensional features and data other than tabular data, such as images and texts.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Comment:** In the experiments, the contextual lasso is assessed only on tabular datasets with relatively low dimensional features. Therefore, it is unclear whether the contextual lasso consistently works well for high-dimensional features and data other than tabular data, such as images and texts.  \n**Response:** We focus on tabular datasets as these are the most common use case for the lasso and its cousins. We will clarify the introduction to state our focus. The question of higher-dimensional datasets is interesting, and we now include experiments where the number of explanatory features is 1000 ($20\\times$ larger than before), and 990 of these are zero for all values of $\\mathbf{z}$. The results are plotted in Figure 1 of the PDF. Importantly, the contextual lasso can still filter out the irrelevant explanatory features and achieves near-perfect support recovery for large $n$.\n2. **Comment:** When contextual features are useful but explanatory features are uninformative, I think that the predictions using the contextual lasso would be worse than the standard neural networks, which use both contextual and explanatory features as input. Is there any solution to this?  \n**Response:** We expect the contextual lasso to outperform a standard neural network in this setting due to its ability to remove the effect of the explanatory features by zeroing out their coefficients. In contrast, a standard neural network has no such capability, i.e., the effect of the explanatory features will flow through the network unencumbered.\n\n3. **Comment:** It would be easier to read this paper if figures were displayed at the top of the page.  \n**Response:** We will make this change.\n\n4. **Comment:** I could not find the description of the limitations.  \n**Response:** We have added a dedicated discussion on limitations---please see our response to Comment 7 of Reviewer BSpt."}, {"review_id": "s3sQ5wDNMh", "reviewer": "Reviewer_BSpt", "summary": "In this paper, the authors present the contextual lasso, a new estimator that fits a sparse linear model on the explanatory features, allowing for the coefficients to vary as a function of the contextual features. This estimator generalizes the original lasso and the varying-coefficient model. The authors introduce a projection layer for efficient model training and furthermore, they develop a group version of the contextual lasso, which is analogous to the extension of lasso to the group lasso.", "questions": "Can the authors provide additional details regarding the experimental results on property sales shown in Figure 1? Merely displaying the fitted coefficients does not clearly indicate the reliability of the results. Are the model fits sufficiently accurate to validate that the interpretation of the coefficient functions is meaningful? Without further context of data, it's difficult to understand if the learned model accurately represents the true structure of the data.\n\nIntroducing the projection layer presents an issue with the non-differentiability of the objective function. How does this impact the actual training of the model? For example, is the learning process consistently stable across multiple random initializations?\n\nIn Section 4, the contextual lasso (or contextual group lasso) seems to outperform deep neural networks, but it's unclear whether the comparison is fair. The performance of deep neural networks can exceed the contextual lasso, depending on the chosen architecture. If the sample size is sufficient, deep neural networks should potentially offer superior performance compared to the contextual lasso, and depending on the applications, some practioners may favor the use of deep neural networks at the cost of sacrificing interpretability. It's important to make this clear in the section; currently, it reads that the contextual lasso surpasses deep neural networks in both predictive accuracy and interpretability.\n\nIn the first line of introduction, it might be more appropriate to revise the statement \"Sparse linear models are a gold standard tool for interpretable machine learning,\" as interpretability really depends on the context of specific problems.", "limitations": "It would be beneficial to discuss the limitations of the contextual lasso, especially in relation to those of the ordinary lasso. For instance, the lasso often selects only one feature from a group of highly correlated features. Does the contextual lasso suffer a similar issue? Furthermore, how sensitive are the selected features to minor adjustment in the hyperparameter $\\lambda$? Such stability considerations are important to enable better interpretation of the model.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and structured. The contextual lasso has the potential to provide substantial advantages for interpretation over neural networks, while leveraging the expressive power of these neural networks through the coefficient functions. The experiments conducted on property sales data (depicted in Figure 1), as well as real data tested in Section 4, are convincing to see the interpretability of the contextual lasso without compromising its predictive accuracy.", "weaknesses": "Although the method has not been explored in previous studies, it appears to be a relatively direct extension of the lasso and the varying-coefficient model. Besides the architecture of the projection layer for model training, the remaining parts of the paper, such as the extension to the group version, side constraints, pathwise optimization, and relaxed fit, are all straightforward extension from existing lasso literature.\n\nThe synthetic data experiments seem somewhat artificial. It's unclear whether these experiments provide any numerical evidence supporting the benefits of the contextual lasso. Given that the data is generated from the same model as the contextual lasso, it's expected that the method which leverages the true structure of the data would outperform or perform comparable to other methods. Furthermore, using the proportion of nonzero features as an interpretability metric is not appropriate, as interpretability is much broader than just sparsity. It would be more convincing to readers if the authors re-evaluate this experiment and supplement it with additional experiments using real data, for instance, in classification scenarios.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Comment:** Although the method has not been explored in previous studies...  \n**Response:** Indeed, many of the techniques for the contextual lasso extend gracefully from the extensive, existing lasso literature. A crucial difference is that the existing techniques are designed for fixed coefficients. In contrast, we adapt all these techniques to the challenging setting of coefficients that vary in sparsity, magnitude, and sign.\n2. **Comment:** The synthetic data experiments seem somewhat artificial...  \n**Response:** The motivation for the existing synthetic design is to validate that the contextual lasso works correctly in the setting for which it is designed. However, we appreciate it is helpful to understand its properties in other settings. In response to your concern, we added a new set of experiments that evaluate the contextual lasso in a setting where the contextual features are irrelevant. Please see Figure 2 of the PDF. Notably, the contextual lasso is still able to recover the true nonzeros by learning a constant function for $\\boldsymbol{\\beta}(\\mathbf{z})$, something it can achieve thanks to the bias term in each neuron. To address your question about interpretability metrics, we added a new experiment analyzing the stability of the selected features from the contextual lasso. Please see our response to your Comment 4. On your last point, we draw your attention to Appendix G of the supplement, which contains results for classification under real and synthetic data.\n3. **Comment:** Can the authors provide additional details regarding the experimental results on property sales shown in Figure 1?...  \n**Response:** We agree the paper would benefit from additional details regarding the housing data and will add a new section to the appendix with a complete explanation of the dataset. Regarding accuracy, we report below the test loss (over 10 random draws) from the contextual lasso and its competitors.\n\n   |                        | Relative loss    | Avg. sparsity    |\n   |------------------------|------------------|------------------|\n   | Deep neural network    | $0.515\\pm0.003$  | $5.0\\pm0.0$      |\n   | Contextual linear model| $0.505\\pm0.002$  | $5.0\\pm0.0$      |\n   | Lasso                  | $0.892\\pm0.001$  | $5.0\\pm0.0$      |\n   | Lassonet               | $0.521\\pm0.003$  | $5.0\\pm0.0$      |\n   | LLSPINN                | $0.729\\pm0.031$  | $4.5\\pm0.2$      |\n   | Contextual lasso       | $0.498\\pm0.002$  | $2.9\\pm0.3$      |\n\n   We will include these results in the appendix.\n\n4. **Comment:** Introducing the projection layer presents an issue with the non-differentiability of the objective function...  \n**Response:** The projection layer does involve some nondifferentiable operations (e.g., abs, sort). However, `Flux` provides gradients for these operations, e.g., sort is differentiated by permuting the gradients, and abs is differentiated by taking the subgradient zero at the point zero. We remark that the gradient obtained by differentiating through Algorithm 1 is the same as that obtained by using implicit differentiation with a convex solver (e.g., `cvxpylayers` of Agrawal et al. 2019), though the latter is slower. The training process is indeed fairly stable across random initializations. To demonstrate, we fit the contextual lasso under two random initializations and measured the hamming distance between the resulting sparsity patterns. We repeated this process under different values of $n$ and plotted the results in Figure 4 of the PDF. For $n>1000$, variation across the 50 explanatory features is relatively small.\n5. **Comment:** In Section 4, the contextual lasso (or contextual group lasso) seems to outperform deep neural networks...  \n**Response:** You are right that the deep neural network and contextual lasso should perform equally well (in terms of prediction) for a sufficiently large sample, and the former can outperform if the ground truth is nonlinear in the explanatory features. While we touch on this point in the experiments (see lines 237-238), we appreciate that it should be better drawn to the reader's attention. We will clearly state it in the paper's conclusion.\n6. **Comment:** In the first line of introduction, it might be more appropriate to revise the statement \"Sparse linear models are a gold standard tool for interpretable machine learning,\"...  \n**Response:** We agree that interpretability is context-specific, and sparsity/linearity alone may not be sufficient for interpretability in some cases. We will rewrite the sentence as: \"Sparse linear models are one of several core tools in interpretable machine learning, ...\".\n7. **Comment:** It would be beneficial to discuss the limitations of the contextual lasso...  \n**Response:** We will add the following paragraph to a new limitations section in the appendix:\n> The contextual lasso has strong prediction, selection, and interpretation properties but also has several weaknesses. As with the ordinary lasso, the contextual lasso tends to select a single feature from a group of highly correlated features. This shortcoming could be remedied by introducing an $\\ell_2^2$-penalty in a spirit similar to the elastic net (Zou and Hastie 2005). Another potential drawback is that the contextual lasso does not guarantee the complete exclusion of an explanatory feature. That is, it cannot ensure the coefficient for a feature is nonzero for every possible $\\mathbf{z}$, which might limit interpretability in certain settings. Full exclusion could be achieved by adding an $\\ell_1$-penalty on the weights of the first layer, similar to the lassonet. Finally, as is typical of neural networks, the objective function of the contextual lasso is nonconvex. Nonconvexity complicates the analysis of its computational and statistical properties.\n\n   Regarding stability, it is our experience that the solution is not sensitive to small changes in $\\lambda$ due to continuity of the regularizer."}, {"review_id": "1el4QxVEgZ", "reviewer": "Reviewer_mL1j", "summary": "This paper considers the predictive problem setting in which the input features are dichotomized into two groups: explanatory features and contextual features. For such a setting, the authors propose the contextual lasso, a statistical estimator that leverages the expressiveness of deep neural networks to learn interpretable sparse linear models with sparsity patterns that can vary based on the contextual features. More precisely, the authors proposed using a neural network to model the nonlinear effects of the contextual features on the explanatory features. In addition, to enforce sparsity among the coefficients for the explanatory features, the network is trained with a regularizer in the form of a projection layer that maps the output (dense) to sparse coefficients by performing a projection onto the $\\ell_1$-ball. Experiments on real and synthetic data suggest that the contextual lasso leads to models which remain interpretable and can be sparser than the regular lasso, without sacrificing the predictive accuracy of standard deep neural networks.", "questions": "My questions/suggestions are provided together with the weaknesses of the paper (where applicable and necessary).", "limitations": "A separate “Limitations” section has not been included in this paper. No potential negative societal impact has been declared in the main manuscript. For my improvement suggestions, please refer to the “Weaknesses” section of this review.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work appears to be the first to propose a sparsity-inducing projection layer that maps a dense output of a neural network onto a space of $\\ell_1$-constrained linear models by solving a constrained quadratic program.\n\nThe benefit from the proposed estimator is that it can utilize the expressive power of deep neural networks and yet learn interpretable sparse linear models by encouraging contextual sparsity through an additional projection layer; all while allowing feature relevance to depend on the prediction context as opposed to determining a fixed set of relevant features. In addition to the classical contextual sparse learning setting, a variant of the sparsity-encouraging projection layer is also devised for the group lasso setting.\nThe proposed estimator and the formulation of the quadratic program for determining the sparse coefficients are technically sound and well-supported. Moreover, the particular sparse learning setting that is being considered is well-supported by examples where the input features are inherently dichotomized into explanatory features and contextual features, and the need for contextual sparsity in such a setting is well-justified.\n\nThe effectiveness of the contextual lasso has been evaluated on both synthetic and real data (in the standard setting and a setting with grouped explanatory features). The findings seem to suggest that, in contrast to other sparse linear models and a conventional DNN, which use either all or almost all available explanatory features, contextual lasso leads to linear models that are much simpler and involve less explanatory features on average.\n\nThe paper is rather well written and organized. The notation is very clear, easy to follow, unambiguous and consistent throughout the paper.", "weaknesses": "The problem considered in this work is a rather general one, as many applications involve a certain context that can be leveraged to select candidate features and determine their effects. Nevertheless, this generality of the problem is not matched by the extensiveness of the conducted experiments. It seems that the contribution’s potential significance is limited by (1) the narrow settings of the experimental design (particularly in the synthetic data generation process) and (2) the consideration of only two real-world applications given the pervasiveness of sparse learning problems surrounded by a certain prediction context. I would encourage the authors to consider additional datasets characterized by both explanatory and contextual features.\n\nAlthough this work is aimed at capturing the prediction context by leveraging the expressive power of neural networks to map contextual features to a sparse coefficient vector, there are other works that take different approaches to the same end. For instance, [1] proposed adaptive optimizers for structured sparsity for deep learning, but also discusses unstructured sparsity ($\\ell_1$) which is directly relevant to this work. There are also other works, such as [2], that take a non-neural based approach to capturing prediction context but I believe are still relevant. Namely, [2] addresses the house pricing problem (provided as an example in Section 1 of this work) using a generalization of the group lasso to a graph setting, where the graph represents the context defined by the distances between the house locations. Finally, there are also approaches that are not directly applicable to the setting considered in this paper, however, in a more general sense, they are relevant to inducing sparsity in neural networks and should be considered as a part of the related work (e.g., [3]).\n\n[1] Deleu, T., & Bengio, Y. (2021). Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint arXiv:2102.03869.\n[2] Hallac, D., Leskovec, J., & Boyd, S. (2015, August). Network lasso: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 387-396).\n[3] Wang, P., He, X., Li, G., Zhao, T., & Cheng, J. (2020, April). Sparsity-inducing binarized neural networks. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 12192-12199).\n\nAs previously suggested, other sparse linear estimator variants have been proposed to utilize contextual information for the same or similar setting as the one considered in this work. Such models include Network Lasso (mentioned in the previous comment), Graphical Lasso and its associated extensions, among others. In that regard, I am wondering why the authors did not consider including such estimators as additional baselines. Moreover, since this work proposes a separate projection layer to induce sparsity in the output of a neural network, I also believe that a comparison to a conventional neural network with a dropout layer applied on its outputs ($\\eta$ values) would be appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Comment:** The problem considered in this work is a rather general one, as many applications involve a certain context that can be leveraged to select candidate features and determine their effects. Nevertheless, this generality of the problem is not matched by the extensiveness of the conducted experiments. It seems that the contribution’s potential significance is limited by (1) the narrow settings of the experimental design (particularly in the synthetic data generation process) and (2) the consideration of only two real-world applications given the pervasiveness of sparse learning problems surrounded by a certain prediction context. I would encourage the authors to consider additional datasets characterized by both explanatory and contextual features.  \n**Response:** We expanded the synthetic data generation process to incorporate two new scenarios: (1) a high-dimensional scenario with $p=1,000$ (Figure 1 of the PDF) and (2) a scenario where the contextual features are irrelevant (i.e., the lasso's home court, Figure 2 of the PDF). The first scenario is statistically challenging, though the contextual lasso offers highly competitive results. The second scenario is designed to evaluate the contextual lasso outside its home court. The contextual lasso remains competitive and recovers the ground truth for sufficiently large $n$. Regarding the real-data experiments, we highlight the third real-data example in Appendix G. We have also expanded the analysis of the house pricing data (see our response to Reviewer BSpt Comment 3), making for four real examples in total.\n2. **Comment:** Although this work is aimed at capturing the prediction context by leveraging the expressive power of neural networks to map contextual features to a sparse coefficient vector, there are other works that take different approaches to the same end. For instance, Deleu and Bengio (2021) proposed adaptive optimizers for structured sparsity for deep learning, but also discusses unstructured sparsity () which is directly relevant to this work. There are also other works, such as Hallac, Leskovec, and Boyd (2015), that take a non-neural based approach to capturing prediction context but I believe are still relevant. Namely, Hallac, Leskovec, and Boyd (2015) addresses the house pricing problem (provided as an example in Section 1 of this work) using a generalization of the group lasso to a graph setting, where the graph represents the context defined by the distances between the house locations. Finally, there are also approaches that are not directly applicable to the setting considered in this paper, however, in a more general sense, they are relevant to inducing sparsity in neural networks and should be considered as a part of the related work  (e.g., Wang et al. 2020).  \n**Response:** We have read all three papers and agree on their relevance. We will add each of them to the related works section. We briefly recap the differences in our work. Deleu and Bengio (2021) induce structured sparsity over network weights to obtain smaller, pruned networks that admit efficient computation. In our work, we leave the weights as dense and instead induce sparsity over the network's output for interpretability. Hallac, Leskovec, and Boyd (2015) introduce the Network Lasso with a different coefficient vector per observation and cluster these coefficients using a lasso-style regularizer. They consider problems similar to ours, for which contextual information is available, but do not induce sparsity over the coefficients (their regularizer clusters but does not sparsify).  Wang et al. (2020) study network quantization and devise a scheme for activation functions that output zeros and ones. Though our approach involves an activation that outputs zeros, we also allow a continuous output instead of a binary one. Moreover, their end goal differs from ours; whereas they pursue sparsity to reduce computational complexity, we pursue sparsity for interpretability.\n3. **Comment:** As previously suggested, other sparse linear estimator variants have been proposed to utilize contextual information for the same or similar setting as the one considered in this work. Such models include Network Lasso (mentioned in the previous comment), Graphical Lasso and its associated extensions, among others. In that regard, I am wondering why the authors did not consider including such estimators as additional baselines. Moreover, since this work proposes a separate projection layer to induce sparsity in the output of a neural network, I also believe that a comparison to a conventional neural network with a dropout layer applied on its outputs ( values) would be appropriate.  \n**Response:** As a new benchmark, we added the localized lasso  (Yamada et al. 2017). The localized lasso extends the network lasso by adding sparsity. We estimated the graph information from the contextual features using the nearest neighbour approach of  Yamada et al. (2017). A challenge with the localized lasso (and the network lasso) is that each new observation requires a new coefficient vector, i.e., additional optimization variables. Consequently, we found it difficult to scale the method to our main experiments (synthetic and real), so we ran smaller-scale experiments and plotted the results in Figure 3 of the PDF. The localized lasso's prediction loss improves with growing $n$, but at a slower rate than the contextual lasso. Regarding your last suggestion, it is an intriguing idea. However, initial experimentation indicates it does not work well in our setting. At $n=10,000$, $p=10$, and $m=2$, the dropout network yielded $\\sim0.95$ relative test loss with the dropout rate set to reflect the true sparsity level (vs $\\sim0.2$ for the contextual lasso). Cross-validating the dropout rate led to a dropout of zero. This result seems to be due to the contextually sparse setting involving a deterministic sparsity pattern, whereas dropout is random."}, {"review_id": "CE7SQUbBoV", "reviewer": "Reviewer_DGcD", "summary": "This paper proposed the contextual lasso model to combine the interpretability of linear model and the express power of deep neural network. The input feature are split into explanatory features and contextual features. The explanatory features are used as input features for a generalized linear model, and the contextual features are feed into a neural network with a projection layer on top to compute the parameters of the linear model. The projection layer are formed as a constrained optimization to force the linear model parameters lie in a $L_1$ ball. Grouped features and other extensions are also discussed. Experiment results show the proposed model outperform traditional DNN model and contextual linear model without LASSO sparsity.", "questions": "1. The projection layer works on the average coefficient vector, how does it work with mini-batch training for DNN, does it just look at the coefficient in that mini-batch?\n\n2. In the experiments, for DNN type model, does contextual features included in the input? In line 280, the DNN and Bassinet are applied to the original features, it might be beneficial to include results of DNN or lassonet with all spline features and contextual features as input\n\n3. Looking at the results of lassonet, it almost select all variables for Parkinson and Energy data sets and select all variables when n gets large for simulation data. How is the regularization parameter determined for lassonet. If we chose a large regularization parameter manually to force the model to have similar sparsity level as the contextual lasso, would the prediction performance be much worse?\n", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. Using contextual features to compute parameters for an interpretable sparse linear model seems to be an interesting and novel idea. Many extensions are discussed. The empirical results also support that the proposed model can select sparse model while maintain better prediction performance.", "weaknesses": "A straightforward idea for a sparse DNN model is to include all explanatory features and contextual features as input to a DNN, train the model with some regularizations, after some thresholding, check the weights in the first layer(or multiplication of indicator matrix for all layers) to decide whether a variable is select or not. With appropriate regularization, I think this can also select relevant variables, and the variable importance can be measured by the gradient to the variable. Conceptually, why does the contextual model better? For real data, how should we decide which features are explanatory features or contextual features?\n\nAnd empirically, It was a bit surprise for me that the baseline Lassonet always keep a relatively dense model, is it due to some poor hyper-parameter choice? please see the Questions section for some clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Comment:** A straightforward idea for a sparse DNN model is to include all explanatory features and contextual features as input to a DNN, train the model with some regularizations, after some thresholding, check the weights in the first layer(or multiplication of indicator matrix for all layers) to decide whether a variable is select or not. With appropriate regularization, I think this can also select relevant variables, and the variable importance can be measured by the gradient to the variable. Conceptually, why does the contextual model better?  \n**Response:** Your idea is related to the lassonet, which inputs all features (explanatory and contextual) and thresholds weights on the first layer (using a lasso penalty) to select variables. This approach underperforms relative to the contexutal lasso because the sparsity pattern in our real/synthetic experiments varies, whereas the weights (which are the basis for thresholding) are fixed. Another drawback is limited interpretability because all features pass through the network.\n2. **Comment:** For real data, how should we decide which features are explanatory features or contextual features?  \n**Response:** Please see our response to Comment 1 of Reviewer kkJW---the same question is raised there.\n3. **Comment:** The projection layer works on the average coefficient vector, how does it work with mini-batch training for DNN, does it just look at the coefficient in that mini-batch?  \n**Response:** We use full batch training in our experiments because all datasets readily fit in GPU memory, even those with $n=10^5$ observations. In any case, the projection layer indeed applies with mini batch training by running the projection algorithm on each mini batch. The drawback is that the average coefficient vector is noisier to estimate than with the full batch. We will add the details to the paper.\n4. **Comment:** In the experiments, for DNN type model, does contextual features included in the input? In line 280, the DNN and Bassinet are applied to the original features, it might be beneficial to include results of DNN or lassonet with all spline features and contextual features as input  \n**Response:** Yes, the DNN receives all features (contextual and explanatory) as inputs. We will clarify this point in the manuscript. We reran the results for the Parkinson's data with the DNN and lassonet receiving all explanatory splines and contextual features.\n\n   |                            | Relative loss      | Avg. sparsity      |\n   |----------------------------|--------------------|--------------------|\n   | Deep neural network (w/ spline) | $0.638\\pm0.077$ | $16.0\\pm0.0$   |\n   | Lassonet (w/ spline)       | $0.335\\pm0.007$   | $15.5\\pm0.2$   |\n\n   The spline versions witness a decline in performance relative to the existing non-spline versions (compare with Table 2 in the paper). The neural networks can approximate splines themselves, so the additional dimensionality from the splines likely introduces further noise. We will state in the text that using splines with these methods does not improve their performance.\n\n5. **Comment:** Looking at the results of lassonet, it almost select all variables for Parkinson and Energy data sets and select all variables when n gets large for simulation data. How is the regularization parameter determined for lassonet. If we chose a large regularization parameter manually to force the model to have similar sparsity level as the contextual lasso, would the prediction performance be much worse?  \n**Response:** The fact that lassonet often uses all the available features was initially a surprise to us too. However, this behavior seems consistent with the results in Lemhadri et al. (2021), where the best MSE or classification accuracy is typically attained at a dense model (see Figures 1-5 and 7 in that paper). As for the regularization parameter, it is tuned on a validation set in the same manner as the lasso and contextual lasso (see lines 484-491 in the appendix). Manually setting $\\lambda$ to achieve a sparsity level similar to the contextual lasso substantially degrades performance on the energy data: the prediction loss is 0.804, roughly twice that from the tuned $\\lambda$."}, {"review_id": "0ItpBC7Hdp", "reviewer": "Reviewer_kkJW", "summary": "This paper proposes the contextual lasso, considering the contextual sparsity which allows feature relevance to depend on the prediction context.To solve the optimization problem of contextual lasso, deep neural networks is utilized to learn interpretable sparse linear models. Besides, an extensive experimental analysis of the new estimator illustrates its good prediction, interpretation, and selection properties in various settings.  ", "questions": "1. The authors should provide detailed explanation or definition about the contextual feature. Moreover, how to achieve the additional contextual feature from the real-world data should also be included.\n2. The authors should add more comparison methods including the contextual explanation networks.\n3. The authors should offer the time complexity analysis of contextual lasso.\n4. At least, some discussions are necessary for the theoretical properties of contextual lasso, e.g., generalization ability compared with the previous Lasso-type methods. ", "limitations": "One of the limitations of contextual lasso referred by authors is that it can result in bias of the linear model coefficients towards zero. This drawback is also due to the L1-norm and indeed belongs to all lasso estimators. The authors propose a possible relaxing approach.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The main novelty of this paper is that it simultaneously considers the explanatory feature and contextual feature in the classical lasso framework. In this case, the classical lasso can be viewed as a special case of contextual lasso.\n2. The performance compared to lasso shows that combining the contextual feature and explanatory feature as proposed in the paper leads to performance benefits.\n3. The paper is well-written and technically sound.", "weaknesses": "1. An essential problem of this paper is that how to accurately define or distinct the explanatory feature and contextual feature still remains unclear. The authors should offer more definition or evidence to explain the contextual feature.\n2. The comparison methods are insufficient. Especially, since contextual explanation networks is a cousin of the contextual lasso, it is necessary to show the superiority of contextual lasso against contextual explanation networks (Al-Shedivat et al.,2020).\n3. Though some experiments can validate the effectiveness of contextual lasso over lasso, it also lacks a time complexity analysis.\n4. Theoretical analysis (e.g., generalization, optimization) seems weak  for the proposed approach.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Comment:** The authors should provide detailed explanation or definition about the contextual feature. Moreover, how to achieve the additional contextual feature from the real-world data should also be included.  \n**Response:** We will add the following paragraph to the paper's conclusion:\n> In practice, one must decide on the split into explanatory and contextual features. The problem of deciding on this split is the same as that faced with varying-coefficient models. Though the literature on varying-coefficient models is extensive, there are no definitive rules for partitioning the features in general. In the housing and energy examples, the contextual features are temporal or spatial effects, which are distinct from the remaining features. In the telemonitoring example, the patient attributes (age and sex) differ fundamentally from the vocal characteristics, so the delineation is clear there. Ultimately, the partition for any given application should be guided by domain expertise with consideration to the end goal. If the user needs to understand the exact effect of a feature, that feature should be an explanatory feature. If a feature's effect is of secondary interest, or it is suspected that the feature influences the structural form of the model, that feature should be a contextual feature. If the user determines there are no contextual features, the ordinary lasso is a more appropriate tool.\n2. **Comment:** The authors should add more comparison methods including the contextual explanation networks.  \n**Response:** The contextual linear model evaluated in our paper is the contextual explanation network with deterministic encoding and linear explanation in Al-Shedivat, Dubey, and Xing (2020). Please see \"varying-coefficient models\" (Section 5.1) and \"deterministic encoding\" (Section 3.1.1) of that paper. They also consider a \"constrained deterministic encoding\" (Section 3.1.2), which involves soft-attention over a finite-set of linear models (described in the literature review of our paper). We found this variant did not perform as well during the development of our approach. We will revise the related works section of our paper to clarfiy the connection. As an additional benchmark, we now also evaluate the localized lasso (Yamada et al. 2017). The localized lasso requires graph information, which we extract from the contextual features. It's prediction loss improves with growing $n$, but at a slower rate than the contextual lasso. Please see Figure 3 of the PDF.\n2. **Comment:** The authors should offer the time complexity analysis of contextual lasso.  \n**Response:** A forward/backward pass through the vanilla feedforward component of the network takes $O(md+hd^2+pd)$ time, where $h$ is the number of hidden layers, $d$ is the number of nodes per layer, $m$ is the number of contextual features, and $p$ is the number of explanatory features. A forward/backward pass through the projection algorithm takes $O(p)$ time (Duchi et al. 2008). Over a sample of size $n$, these become $O(nd(m+hd+p))$ and $O(np)$, respectively. The time complexity for the full network over $n$ observations is thus $O(nd(m+hd+p))$. We will add these details to the paper. We also added runtime experiments that empirically demonstrate linear complexity as a function of $n$ and $p$. Please see Figure 5 of the PDF. As a point of comparison, in the left plot at $n=10,000$, Lassonet takes $\\sim50$ seconds and LLSPINN takes $\\sim470$ seconds. In the right plot, at $p=1,000$, Lassonet takes $\\sim35$ seconds and LLSPINN takes $\\sim95$ seconds. The run time for other methods is negligible ($\\sim 1$ second).\n3. **Comment:** At least, some discussions are necessary for the theoretical properties of contextual lasso, e.g., generalization ability compared with the previous Lasso-type methods.  \n**Response:** We will add the following paragraph to the paper:\n> The statistical properties of the lasso in terms of estimation, prediction, and selection are now well-established in theory  (Bunea, Tsybakov, and Wegkamp 2007; Raskutti, Wainwright, and Yu 2011; Shen et al. 2013; Zhang, Wainwright, and Jordan 2014). The synthetic experiments in our paper suggest that the contextual lasso satisfies similar properties, though theoretically establishing these results is challenging. Convergence results for vanilla feedforward neural networks (e.g., Schmidt-Hieber 2020) do not apply in our setting due to the projection layer. Moreover, to our knowledge, no statistical guarantees exist for networks configured with convex optimization layers in general. It remains an important avenue of future research to establish a solid theoretical foundation for the contextual lasso."}]

Author rebuttal:
We thank all the reviewers for their time and careful reviews of our paper. We have now responded to all the questions and concerns raised. Besides making clarifications and textual modifications to the manuscript, we made the following major improvements:
- High-dimensional experiments with $p=1000$ explanatory features. For Reviewers Eybr and mL1j.
- Experiments where the contextual features are irrelevant. For Reviewers BSpt and mL1j.
- New benchmark comparator: the localized lasso. For Reviewers mL1j and kkJW.
- Stability analysis of the contextual lasso. For Reviewer BSpt.
- Time complexity analysis of the contextual lasso. For Reviewer kkJW.
- Extended analysis of the housing price dataset of the introduction. For Reviewer BSpt.
- Dedicated discussion of the contextual lasso's limitations. For Reviewers Eybr and BSpt.
- Discussion on the partition of the contextual/explanatory features. For Reviewers DGcD and kkJW.

If the reviewers have further questions, we are happy to address them.

**New figures are available in the attached PDF.**

*References*

1. Agrawal, A., Amos, B., Barratt, S., Boyd, S., Diamond, S., and Kolter, J. Z. (2019). "Differentiable convex optimization layers". Advances in Neural Information Processing Systems. Vol. 32.

2. Bunea, F., Tsybakov, A. B., and Wegkamp, M. H. (2007). "Aggregation for Gaussian regression". Annals of Statistics 35.4, pp. 1674–1697.

3. Deleu, T. and Bengio, Y. (2021). "Structured sparsity inducing adaptive optimizers for deep learning". arXiv: 1711.07592.

4. Duchi, J., Shalev-Shwartz, S., Singer, Y., and Chandra, T. (2008). "Efficient projections onto the ℓ1-ball for learning in high dimensions". Proceedings of the 25th International Conference on Machine Learning, pp. 272–279.

5. Hallac, D., Leskovec, J., and Boyd, S. (2015). Network lasso: Clustering and optimization in large graphs. Association for Computing Machinery, pp. 387–396.

6. Lemhadri, I., Ruan, F., Abraham, L., and Tibshirani, R. (2021). "LassoNet: A neural network with feature sparsity". Journal of Machine Learning Research 22, pp. 1–29.

7. Raskutti, G., Wainwright, M. J., and Yu, B. (2011). "Minimax rates of estimation for high-dimensional linear regression over ℓq -balls". IEEE Transactions on Information Theory 57.10, pp. 6976–6994.

8. Schmidt-Hieber, J. (2020). "Nonparametric regression using deep neural networks with ReLU activation function". Annals of Statistics 48 (4), pp. 1875–1897.

9. Al-Shedivat, M., Dubey, A., and Xing, E. (2020). "Contextual explanation networks". Journal of Machine Learning Research 21, pp. 1–44.

10. Shen, X., Pan, W., Zhu, Y., and Zhou, H. (2013). "On constrained and regularized high-dimensional regression". Annals of the Institute of Statistical Mathematics 65.5, pp. 807–832.

11. Wang, P., He, X., Li, G., Zhao, T., and Cheng, J. (2020). "Sparsity-inducing binarized neural networks". Vol. 34, pp. 12192–12199.

12. Yamada, M., Takeuchi, K., Iwata, T., Shawe-Taylor, J., and Kaski, S. (2017). "Localized lasso for high-dimensional regression". Proceedings of the 20th International Conference on Artificial Intelligence and Statistics 54, pp. 325–333.

13. Zhang, Y., Wainwright, M. J., and Jordan, M. I. (2014). "Lower bounds on the performance of polynomial-time algorithms for sparse linear regression". Proceedings of the 27th Conference on Learning Theory. Vol. 35, pp. 921–948.

14. Zou, H. and Hastie, T. (2005). "Regularization and variable selection via the elastic net". Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67 (2), pp. 301–320.

Meta-review: There are two "borderline" reviews which mention several strong points, such as the novelty of the proposed integration of explanatory and contextual information, the idea  using a sparsity-inducing projection layer, and the experimental evaluation.
On the negative side, these reviewers mention problems related to the lacking formal definition of "context", the limited theoretical depth and the conceptual similarity to other approaches.

The more "positive" reviewers also mention the novelty of the idea and quality of the experimental evaluation, which, for them, outweigh some weaknesses, such as possible limitations due to the assumption of an existing  "explanatory\contextual structure" in the underlying problem domain.

After going again over the paper, all reviews, the rebuttal and the discussion, I come to the conclusion that this is indeed a borderline case, but for me the strengths regarding the over-all novelty of the idea and the conclusive experimental evaluation (slightly) outweigh several conceptual weaknesses. However, since this is not a clear-cut case, I would like to encourage the authors to improve the final version of their paper by including further clarifications and experimental results from the discussion phase.
Decision: Accept (poster)