NIMO: a Nonlinear Interpretable MOdel
Authors: Shijian Xu
Marcello Massimo Negri
Volker Roth
Venue: ICLR 2026 Poster (2026)
Keywords: linear model
lasso
interpretability
feature effect
deep learning
Primary area: interpretability and explainable AI

Abstract: Deep learning has achieved remarkable success across many domains, but it has also created a growing demand for interpretability in model predictions. Although many explainable machine learning methods have been proposed, post-hoc explanations lack guaranteed fidelity and are sensitive to hyperparameter choices, highlighting the appeal of inherently interpretable models. For example, linear regression provides clear feature effects through its coefficients. However, such models are often outperformed by more complex neural networks (NNs) that usually lack inherent interpretability. To address this dilemma, we introduce NIMO, a framework that combines inherent interpretability with the expressive power of neural networks. Building on the simple linear regression, NIMO is able to provide flexible and intelligible feature effects. Relevantly, we develop an optimization method based on parameter elimination, that allows for optimizing the NN parameters and linear coefficients effectively and efficiently. By relying on adaptive ridge regression we can easily incorporate sparsity as well. We show empirically that our model can provide faithful and intelligible feature effects while maintaining good predictive performance.

Published as a conference paper at ICLR 2026
NIMO: A NONLINEAR INTERPRETABLE MODEL
Shijian Xu
Marcello Massimo Negri University of Basel
University of Basel shijian.
hwaijian.uuuunibeee.. xu@unibas.ch
Volker Roth University of Basel volker. roteerrottuunbbbbe
ABSTRACT
Deep learning has achieved remarkable success across many domains, but it has also created growing demand for interpretability in model predictions. Although many explainable machine learning methods have been proposed, post-hoc explanations lack guaranteed fidelity and are sensitive to hyperparameter choices, highlighting the appeal of inherently interpretable models. For example, lincar regression provides clear feature effects through its coefficients. However, such models often outperformed by more complex neural networks (NNs) that usually lack inherent interpretability. To address this dilemma, we introduce NIMÓ, framework that combines inherent interpretability with the expressive power of neural networks. Building on the simple linear regression, NIMO is able provide flexible and intelligible feature effects. Relevantly, we develop an optimization method based on parameter elimination, that allows for optimizing the NN parameters and linear coefficients effectively and efficiently. By relying adaptive ridge regression we can easily incorporate sparsity well. We show empirically that our model can provide faithful and intelligible feature effects while maintaining good predictive performance.
INTRODUCTION
Over the past decade, neural networks have achieved remarkable success across domains, from computer vision (Krizhevsky et al.) 2012) to natural language processing (Vaswani et al. 2017). At the same time, their deployment in high-stakes domains such healtheare created a pressing demand for interpretability (Esteva et al. 2017). However, neural networks in general lack interpretability of the predictions in terms of the input features, hence the term "black-box" models. In contrast, classical models such linear regression and decision trees often considered as highly interpretable because of their transparent structure and simple decision rules, but they can lack predictive power on complex, nigh-dimensional, or highly nonlinear tasks (Hastie et al. 2009 Breiman et al. 2017). This tension between accuracy and interpretability has motivated much recent work on models and methods that combine the expressive power of neural networks with the transparency of simpler models (Lemhadri et al. 2021 Thompson et al.) 2023 Wu et al., 2017 (021). Meanwhile, many model-agnostic, post-hoc explainers have been proposed, such as SHAP (Lundberg Lee 2017) and LIME (Ribeiro et al. 2016), providing instance-leve feature attributions for arbitrary predictors. But these host-hoe explanations are approximations and may depend on the choices of parameters, such as the background distribution and kernel width. Among various ways to achieve interpretability. feature effects (Scholbeck et al., 2019) provide a direct, quantitative description of how each feature influences the prediction, something post-hoc methods can approximate but do not guarantee.
We use the term feature effects in broad sense, referring to how changes in input features influence model predictions. A common formalization of this idea is through marginal effects (Nguyen) 2020), which define feature effects as the direction and magnitude of the change in prediction due to an infinitesimal change in feature value. Linear models offer immediately intelligible marginal effects through their coefficients (Molnar 2025), but are outperformed by more complex nonlinear models, particularly neural networks. Unfortunately, these expressive models typically lose the property

---

Published as a conference paper at ICLR 2026
Linear model
Neural network
NIMO (ours)
actwork
h4
Pa
individual
global coefficient
nomiiinsee
Interpretabiäty Expressivity
Interpretability Emprrssivily
Interpretabliity
Expressivity
Comparison of linear models, neural networks, and our method. Linear models offer straightforward interpretabilityy but limited expressivity while neural networks provide high expres- sivity but lack interpretability. Our method combines the strengths of both by building on the linear model and adjusting the global linear coefficients with individual nonlinear corrections.
Figure 1:
of providing such inherently intelligible effects. In such models, marginal effects must instead be obtained by computing derivatives of the fitted function with respect the inputs, usually through computationally expensive automatic differentiation (autograd) frameworks (Paszke et al. (017). To bridge this gap, we propose intrinsically interpretable hybrid model that preserves atteliggbble marginal effects while retaining the expressivity of neural networks. Specifically, we start from a linear model y E, 3z and adjust each By by multiplicative factor hj. which is the output of neural network NNu, (x) evaluated at the input instance x:
Σ (13))1(()) with hy(x) = NNa, (x)
In Figure M we illustrate how the proposed approach retains the high-level structure of a linear model while integrating per-instance nonlinear corrections through neural networks.
Within this framework, we distinguish between local explanations and global interpretations. Local explanations describe how changes a feature influence the prediction for a specific instance, enabled here by the per-instance corrections (x) from the neural network. Global interpretations, in contrast, summarize the overall influence of a feature across the dataset, which we later formalize through the marginal effects at the mean (MEM). In our model, these coincide exactly with the global linear coefficients. To illustrate, consider a medical application where we aim to estimate a patient's 's risk of developing disease. A local explanation would ask: given this patient's 's age, weight, and other characteristics, how does a small change in age affect their individual risk? A global interpretation instead asks: if all other variables are held fixed, how does age in general influence disease risk across the population? ? For linear models, these two perspectives coincide, but in our framework they differ due to potential nonlinear feature interactions. Our network is designed so that the model preserves the classical interpretation of the linear coefficients B, while simultancously providing per-instance corrections through hy (x). This enables us unify global summaries via MEM with local explanations at the instance level within single coherent framework.
Since the network parameters and the linear coefficients are tightly coupled, jointly optimizing them is non-trivial. To address this, we develop an optimization method based on parameter elimina- tion. Specifically, we derive closed-form expression for the linear coefficients / function of the NN parameters u. Substituting this expression back into the objective reduces the problem to optimizing only over the NN parameters. To further enhance interpretability, we impose sparsity by formulating the problem as adaptive ridge regression (Grandvalet, 1998). In the network, we further apply group & regularization (Yuan Lin 2006) to the weight matrix of the first fully connected layer, encouraging feature- level sparsity and clarifying how inputs contribute predictions.
Overall, the contributions the present work can be summarized as follows:
We introduce nonlinear interpretable model that combines the expressivity of neural net- works with the interpretability linear models, providing intelligible global interpretability via marginal effects at the mean (MEM) and per-instance corrections through the network.
2

---

Published as a conference paper at ICLR 2026
We introduce optimization algorithm via parameter elimination that effectively and efficiently optimizes the linear coefficients and the neural network parameters
We systematically evaluate NIMO on synthetic and real datasets, verifying that it recovers correct feature effects, achieves accurate per-instance explanations, and offers favorable trade-off between interpretability and predictive performance compared to existing methods.
2 RELATED WORK
Interpretability in machine learning. Interpretability refers to the capacity to express what a model has learned and the factors influencing its outputs in a manner that is clear and understandable to humans. In the literature, the two terms interpretability and explainability interchangeably. However, they have subtle yet important differences. Explainability focuses on providing reasons for the model's output. Most of the methods, especially in deep learning, provide post-hoc explanations. The most popular ones include LIME (Ribeiro et al.) 2016), SHAP (Lundberg & Lee 2017) and Grad-CAM (Selvaraju al. 2020). On the other hand, interpretability focuses on understanding the inner mechanisms and decision processes of the model, and most of the methods that claimed to be interpretable involve inherently understandable models, such as decision trees (Wu 2017 )021). Lasso [Tibshirani] 1996) and Explainable Boosting Machines (Lou et al. 2013) The literature on interpretable machine learning is vast and we refer readers to Rudin et al. (2022) for a more in-depth review.
sometimes used
et al.
and Molnar 2025
Linear Models and Input Feature Effects. Coefficients from linear regression models provide inherently interpretable feature effects, directly quantifying how changes in input features influ- ence predictions (Nguyen 2020 Molnar 2025). This transparency makes linear models especially attractive in domains where interpretability is critical. Regularization methods such the Lasso (Tib. shirani, 1996) further enhance interpretability by selecting the most relevant input features. However, linear models cannot capture nonlinearities or interactions without complicated feature engineering, e.g., basis expansion, which limits their ability to reflect complex data-generating processes. This motivates our approach, which preserves the interpretability of linear coefficients while extending their expressiveness to capture more complex data patterns.
Hybrid models. Several works have proposed combining linear models with neural networks to improve interpretability. Neural Additive Models (NAMs) (Agarwal et al. 202T) extend generalized additive models (GAMs) by learning one neural network per feature, whose outputs are summed to form the prediction. This design preserves interpretability at the feature level but does not have the ability to capture feature interactions. NODE-GAM (Chang et al. 202T) is also neural generalized additive model that uses differentiable tree ensembles to learn interpretable univariate and pairwise shape functions. However, similar to NAM, both of their interpretability comes from the shape function, which is input dependent, and cannot provide a global population-levs level summary. LassoNet (Lemhadri et al. 2021) integrates a neural network with Lasso regression by enforcing that feature can be used in the nonlinear part only if its corresponding linear coefficient is nonzero. Interpretable Mesomorphic Networks (IMN) (adra et al.) (022)) use a hypernetwork to predict linear coefficients on per-instance basis. While this increases flexibility, it sacrifices global interpretability of the coefficients, making it difficult to recover clear baseline effects. Interpretable Mixture of Experts (IME) (Ismail al. 2022) routes each sample to an interpretable expert (e.g.. linear model), ensuring that for that sample the entire decision process is an exact and transparent explanation of which expert was chosen and why. While IME can maintain high accuracy and provide faithful local explanations, its global interpretability remains limited. In contrast, our proposed model (NIMO) preserves the global interpretability of linear coefficients while augmenting them with nonlinear corrections. This unifies global summaries via MEM with instance-level explanations within single coherent framework.
3 PROPOSED APPROACH
3.1 A NONLINEAR INTERPRETABLE MODEL.
Model definition Let X E gaxd be n observations with d dimensions and let y E R" be the targets. In the following, we assume the data X to be standardized, i.e. that each feature has zero mean
3

---

Published as a conference paper at ICLR 2026
individual nonlinear correction
global linear coefficient
81
9v, (x--)
By (u,(z_)))
X
Bi
ad
Figure 2: NIMO architecture. The model consists of global linear coefficients 83 modulated by the individual data point x through hj. In partieular, ht depends on X-j, i.e. all features except the j-th.
and unit standard deviation. A linear regression model is simply defined as y 30 + 2ー=1 Ty In linear models the coefficient 33 can be interpreted as the additive effect of the j-th feature on the output when all other feature values remain fixed. The idea of our model is maintain the intelligible global interpretation at the population level while allowing for local explanations at the per-instance level. To do so, we define our approach starting from a linear model and multiplying the linear coefficients 83 by nonlinear correction term hy that depends on the data point x:
d
f(x) = B0 +
33 + Iu, ()) s.t.
Juy (0) = 0
where x-j is the vector without the j-th component and Ju, (.) are d different scalar-valued func- tions defined by neural networks, each parametrized by uj. We provide illustration of the proposed approach Figure 2 Due to the specific design of our model, it exhibits the following properties. First, by explicitly excluding the j-th feature as input of the neural network, i.e. gu, (x-j). 2y con- tributes to the prediction only through the linear term 3y. preserving the behavior and interpretation of the coefficient B,. Second, is zero. Therefore, since by construction 9u, (0) = when all other features are fixed at the mean value (i.e. at zero), the model reduces to linear model. In other words, the marginal effect of feature j is exactly BJ. as in a purely linear model. This ensures that the nonlinear correction term does not alter the global interpretability of the linear coefficients.
assume the data is standardized, the mean value of the features
Implementation details A naive implementation of Eq. 2 requires d separate neural networks, which is impractical in high- nigh-dimensional settings. To address this, we use a single shared network Ju and provide it with positional information to distinguish between features. Concretely, we append positional encoding of the index j to the masked input vector x-j. where the j-th component of is set to zero. This way, Ju can learn feature-specife corrections while reusing parameters across all features. Finally, to enforce the interpretability constraint gu(0) (0) = O, we simply subtract the mean prediction in each forward pass: Gu (x.x)) := 9u (x-)) gn (0).
3.2 INTERPRETABILITY VIA MARGINAL EFFECTS
Following Nguyen 2020). the marginal effect (ME) of feature j at input x is defined as
ME, 0x)
for any model fx). By definition, ME, measures the instantaneous change in prediction when varying Zj. However, marginal effects in nonlinear models are not constant, they depend on the other input features. One way to summarize them is by computing marginal effects at the mean (MEM), which estimates marginal effects the average values of the input features:
0f(x)
MEM,
dx,
xx=
4

---

Published as a conference paper at ICLR 2026
Based on the definition, compute the marginal effects for NIMO as below:
Dgu, (x-1)
ME, 3,(1 Iu, (x-1))
Σ
dzj
1
MEM, 33
Note that the mean value of input features x 0 after standardization. Due the design of our model, MEM, coincides with the linear coefficient. In principle, such marginal effects can also be computed via automatic differentiation for other differentiable models, but these do not necessarily yield interpretable results. In contrast, for NIMO, the MEMs directly correspond to the linear coefficients B3, capturing the additive population contribution of each feature.
While MEM is widely used as global effect measure in statistics, it is important to acknowledge its general limitations, as discussed in Scholbeck al. (2024). MEM meaningful and interpretable in models that include an explicit global linear component, but for highly nonlinear functions the MEM may fail to reflect the true relevance of individual features. However, NIMO is explicitly designed to avoid such cases: its architecture explicitly separates global linear component (captured by () from local, per-instance nonlinear adjustments. This structural constraint ensures that the global linear contribution remains identifiable and stable. Consequently, MEM remains appropriate and interpretable within NIMO's s model class, even though it may be unsuitable as a universal feature-importance metric for fully unconstrained nonlinear functions.
3.3 TRAINING VIA PARAMETER ELIMINATION
For illustrative purposes, in this section provide high-level description of the optimization algorithm used to train NIMO. More detailed training procedure can be found in Appendix B
Optimization for sparse regression Consider the data X _nd and targets y R" As argued before, we can model the d networks with one network Iu and represent its outputs in a matrix G. = Ou (X) E Rxxd The model in Eq. 2 can then be re-written in matrix form as fXX) B,,3, where B, = X + X Gu and "o" denotes element-wise multiplication. Let us first consider the standard ridge regression setting: L(,3, u) = |ly B,, + || To separate the optimization over 3 and Murphy & Van der Vaart (2000). The key idea is to eliminate ,3 by first solving for its closed-form expression in terms of u, and then substituting it back into the objective. This reduces the problem to an optimization over u only:
take inspiration from the profile likelihood approach (Venzon & Moolgavkar 1988
min L(B, u)
min C(B(U) u), where B(u) = (B B.TTB.. Bu + -
(5)
We can then efficiently optimize the objective with gradient descent over u only. Furthermore, within this framework we can incorporate sparsity by replacing the lg penalty with the & penalty:
min | B + X|1B111 B.u
(6)
Unlike ridge regression, Lasso does not admit a closed-form solution for B. To eliminate 3 in this setting, we use adaptive ridge regression, which admits at each step a closed-form expression for 3 in terms of and is equivalent to Lasso at the optimum (Grandvalet) 1998); see Appendix A for a detailed proof. This enables us to efficiently optimize the problem using gradient descent while still enforcing sparsity. The full procedure is summarized in Algorithm I in the Appendix. A well-known limitation of Lasso the over-shrinkage effect, where large coefficients are excessively penalized (Fan & Li 2001). In some applications, this lead significant bias and degrade model performance. To mitigate this issue, we can extend the & norm regularization to sub-Li pseudo-norm, which penalizes large coefficients less aggressively while still promoting sparsity. A detailed introduction to this approach is provided in Appendix B.3
Extension to generalized linear models The same idea can be extended to any generalized linear model of the form: f(EYY|x) Bu + f1œ1 +... + Spxp xT 3. where f is link function. For simplicity, consider logistic regression with é, penalty. As before, we first get a closed-fone expression for B in terms of and then substitute it back optimize the neural network
5

---

Published as a conference paper at ICLR 2026
Linear coefficients
Nonlinear terms
NN sparsity in first layer
tanh)1Drg)
1.0
1.0
Learned
0.5
0utbun Neurens
10
0.0
20
-05-
-0.5
80
30
-2 Kg
-7 9G
Toy example: y= 3- 21 + tanh(102z)) - 3 -3222 (1 sin( sin(-2c:)) + 0 +
Figure 3: Illustrative toy example. Left: NIMO learns the GT coefficients and also finds the correct sparsity. Middle: the learned nonlinearities coincide with the GT. Right: the first fc layer of the NN learns the correct sparsity: only features # and 22 have non-zero weights, while æg ignored.
parameters u. What is different is that in each optimization step we use the iteratively reweighted least squares (IRLS) surrogate (Green 1984 McCullagh & Nelder 1989), which is weighted least-squares approximation of the original problem:
1/ BuB) 1| 1|
min 8.u
where Wa (m)) (m)) and n, BuB. The so-called working response z is computed as 7 (y- (n)) Eq. is also Lasso problem and we can employ adaptive ridge regression to find closed form expression for 3(u). Then, we substitute ((u) back and optimize over the neural networks parameters u via gradient descent. Appendix B.2 includes detailed derivation.
3.4 PROOF-OF-CONCEPT: TOY EXAMPLE
We illustrate NIMO on simple ftree-imeenioos. toy example. The data is generated as:
= 3 21 (1 + ttanh(1022) (10x:)) + (-3) 22 (1 sin(-2n:) + 0 23 +
(8)
where (0,00.1) Gaussian noise. This example involves three features {x1, 22, 233. but D3 is inactive since is multiplied by zero coefficient. We generate 400 samples and use 200-100- 100-100 train- seinn--aiiaaiion-tee split. As baseline, we compare against Lasso regression. We further apply group l2 regularization to the weight matrix of the first fully connected layer in the neural network to enhance interpretability. Further implementation details are given Appendix C.2 The results shown in Figure On the left, see that the estimated coefficients 33, which also represent the marginal effects at the mean, learned by NIMO align with the ground truth; in particular, NIMO correctly identifies B3 0, i.e. that 23 uninformative. In the middle, the nonlinear corrections gx(x recover the true underlying interactions for both 21 and T2. This highlights an important interpretability aspect: while 81 and 33 -3 represent the global population-level effects, the neural network outputs vary across individual inputs, providing per-instance corrections. Therefore, NIMO not only preserves the global interpretability of the coefficients but also captures the correct local deviations at the per-instance level. Finally, the right panel shows that neurons connected to Z3 in the first fully connected layer are sparse, permitting an additional layer of interpretability: a' does not contribute to nonlinear interactions.
4 EXPERIMENTS
In this section, we experimentally evaluate the proposed NIMO, focusing on interpretability through marginal effects. We first introduce the baseline methods and the experimental protocol used for comparison. Next, we formulate three hypotheses and present empirical results that examine both the interpretability and predictive performance of NIMO. Additional results provided in Appendix D
4.1 METHODS AND PROTOCOL.
Methods: In this paper, we compare our method against Lasso Tibshirani] 1996). vanilla neural network (NN) (Goodfellow et al. 2016), and several state of the interpretable ap-

---

Published as a conference paper at ICLR 2026
proaches, namely LassoNet (Lemhadri et al.) wal et al. 2021) and Interpretable Mesomorphic Networks (IMNs) Kadra et al. 2024
2021), Neural Additive Models (NAMs) (A gar
LassoNet encourages the network to only a subset of the available input features by en- forcing sparsity with novel objective. NAMs learn a linear combination of neural networks that each attend single input feature. IMNs optimize deep hypernetworks to generate ex- plainable linear models on a per-instance basis.
Table I: MSE loss for synthetic regression settings.
Method
Setting 1
Setting
Setting 3
(#features)
(5)
(10)
(50)
Lasso
3.164
3,340
13.122
NN
1.109
1.482
13.718
NAM
3,427
5.126
16.543
Protocol: We conduct experiments both synthetic and real datasets to verify the inter- pretability and performance of our model. For synthetic datasets, we create them by explicitly controlling the linear coefficients and the nonlin- earities, similarly to Eq. [ We explore multiple nonlinearities, different dimensionalities and also use several uninformative features (i.e. with zero coefficient) to test if we recover sparse signals. The details about the exact data generating process are provided in the Appendix D.2 For real datasets, we choose several popular and well-studied datasets from UCI Machine Learning Repository !
IMN
0.137
1.188
6.308
LassoNet
0.078
2.612
1.738
NIMO
0.024
0.197
0.380
4.2 HYPOTHESES AND EXPERIMENT RESULTS
Hypothesis 1: NIMO outperforms other methods synthetic datasets and remains robust in low-data regime.
We compare NIMO with the above methods the synthetic datasets and evaluate predictive perfor- mance using mean squared error (MSE). The results are presented in Table In each setting, we use 200 samples for training. In this low-data regime, the naive neural network tends to overfit even with a relatively shallow MLP, while the linear model is too simple to capture the complex underlying nonlinearities. NAM, which assigns a separate MLP to each feature component, even more prone to overfitting than the naive neural network. Overall, the results show that NIMO not only outperforms the other methods by effectively capturing complex nonlinear feature interactions, but also remains robust in low-data settings due to the regularization and optimization strategies employed.
Hypothesis 2: NIMO accurately estimates the marginal effects at the mean (MEM), preserving the additive contribution of each feature at the population-level.
Vanilla regression
Setting
Setting
Setting 3
NIMD
P.
P.
B
-4
Figure 4: Learned B coefficients synthetic regression datasets and comparison with Lasso regres- sion and Ground truth. Vanilla regression (left) is purely linear while Settings 1, 3 (right) have different nonlinearities. Both NIMO and Lasso recover the correct nonzero coefficients (informative features) but only NIMO recovers the correct sparsity of the zero coefficients (uninformative features).
This experiment evaluates whether NIMO can recover population-leva interpretability terms of marginal effects at the mean. For the other methods that rely on automatic differentiation. MEM can certainly be computed, but they do not yield meaningful interpretations. They reduce to arbitrary

---

Published as a conference paper at ICLR 2026
Diabctes
Boston housing
Superconductivity
0.6
0.25
Test MSE
0.20
0.2
0.1
Lasso
NN
NAM
IMN
NN
NAM
IMN LasseNet
Lasso
NN
NAM
IMN
Figure 5: Predictive performance (MSE) the diabetes, Boston housing and superconductiivty dataset.
HEM Poalurcc und Nathoz
hodlurs Methed
62
-14
L200
(a) Diabetes
(b) Boston
Figure MEMs for Lasso and NIMO on the diabetes (leff) and Boston housing dataset (right).
numerical values that neither explain the prediction for an instance nor capture population- effects. In contrast, the specific structure of NIMO ensures that the marginal effects the mean coincide with the linear coefficients, as in the linear models, providing clear and intelligible interpretation. Therefore, we restrict the comparison
Lasso only, which directly yields globally interpretable coefficients. As a sanity check, we first test NIMO on a Vanilla regression dataset with purely linear coefficients in {-5, -4.... 3, 4} and no nonlinearities. This verifies whether the neural network component of NIMO interferes with the linear part when nonlinear effects are present. As shown Figure (left), NIMO perfectly recovers the ground truth coefficients, confirming that the neural network does not distort the linear coefficients, maintaining the intelligibility of MEM. We then extend the analysis to three additional regression settings (Setting 3) with varying nonlinearities and sparsity levels (see Appendix D.2 for details). Unlike Lasso. NIMO not only recovers the true coefficients but also correctly identifies the sparsity pattern, i.c., which features are uninformative (Figure 4 bottom-right). Based these results, we conclude that NIMO estimates the accurate marginal effects at the mean and preserves the additive population-leve contribution each feature.
Hypothesis 3: NIMO achieves a favorable trade-off between interpretability and predictive perfor- mance in practice.
This experiment aims to evaluate whether NIMO can achieve strong predictive performance on real datasets and how the results can be interpreted. We select several widely used datasets from the UCI Machine Learning Repository: the diabetes dataset (Efron et al.] 2004). the Boston housing dataset (Belsley et al. (205), and the superconductivity dataset ((amidich][2018) Each dataset randomly shuffled and split multiple times with different seeds (9 seeds for the diabetes dataset and the Boston housing dataset, 5 seeds for the superconductivity dataset), and the experiments are performed once each split using identical model settings. We first examine the predictive performance of NIMO. The box plot of the mean squared error for each method is shown in Figure 5 NIMO generally achieves performance compatible with the other methods and attains the best performance on the superconductivity dataset. To assess interpretability, we first visualize the marginal effects the mean (MEM) obtained from Lasso and NIMO for the diabetes and Boston housing datasets
8

---

Published as a conference paper at ICLR 2026
in Figure 6 Recall that MEM estimates marginal effects the average values of the imput features, reflecting the additive population-level contribution each feature. We observe that the MEM profiles of Lasso and NIMO differ across several features. These differences do not necessarily imply the presence of nonlinear feature interactions; rather, both datasets contain groups of strongly correlated predictors, making feature selection inherently unstable. Consequently, Lasso and NIMÓ may highlight different features within the same correlated group, even if their predictive roles are similar. Appendix D.8 we experimentally showcase the degree of nonlinearities in each dataset. For the superconductivity dataset, which involves much larger set of features, the corresponding MEM visualization is provided the Appendix in Figure
We then use MEM to rank the features on the diabetes and Boston housing datasets. For comparison, we also apply the post-hoc explainer SHAP (Lundberg Lee 2017) our model and rank features On the diabetes dataset, all methods behave similarly identifying the most important features, which is expected since the data lacks complex nonlinear interactions. This is further supported by the comparable predictive performance across methods, as shown Figure (left). Notably, SHAP produces identical feature rankings MEM for NIMO. In contrast, for the Boston housing dataset, the methods yield inconsis- tent feature rankings, suggesting that nonlinear interactions play a crucial role in explaining the data. Although MEM also be computed for other nonlinear methods, they do not provide the same level of intelligible interpretability as our model. For NIMO, SHAP rankings are largely consistent with MEM, though not perfectly aligned. The most pronounced difference appears with LassoNet, which assigns high importance to features considered least relevant by other methods, such as INDUS. We attribute this how LassoNet handles nonlinearities.
according to the global SHAP values. The results are shown in Table 2
Table 2: Feature rankings according MEMs the diabetes and Boston housing datasets. A lower ranking is associated with a higher feature importance. Features whose rankings are consistently similar across multiple methods highlighted in yellow.
Boston Housing Dataset
Diabetes Dataset
SHAP
NIMO
LassoNer
NAM
IMN
SHAP
NIMO
LassoNet
NAM
IMN
CRIM
10
9
8
age 7
9
9
10
ZN
13
13
10
11
12
8
5
INDUS
hmi
2
CHAS
12
10
11
3
3
NOX
RM
11
10
9
10
10
7
7
8
AGE
DIS
3
6
7
3
5
12
2
RAD
5
12
s4
8
4
TAX
4
I1
6
6
PTRATIO
7
4
5
s6
10
10
LSTAT
2
1
4.3 ABLATION STUDY
To verify the effectiveness of masking out the j-th component of an instance when passing it through the neural network, we conducted an ablation study. Specifically, we examined the effect of masking out the j-th component as well as the effect removing position encoding. From the ablation results across the three settings (Figure Figure 15 and Figure 16). observe several consistent patterns:
Without masking the j-th component, the first fc layer of the neural network can still exhibit sparsity patterns, but they are less distinct than when masking is applied. We attribute this to the position encoding, which enables the network to identify the feature index.
Without masking the j-th component, the MEMS are confounded by nonlinear interactions, their interpretation as additive regression coefficients and the meaning of their magnitudes are lost.
Further removing the position encoding distorts the sparsity patterns in the first fc layer and alters the magnitude of the coefficients, undermining interpretability.

---

Published as a conference paper at ICLR 2026
Orrsated PosElc irrradedd PSP.C.
Figure 7: The effect of feature masking and position encoding the linear coefficients and the sparsity of the first fc layer on setting I.
4.4 LIMITATIONS
The proposed approach is designed preserve the global, population-level interpretability inherited from linear models while simultaneously learning nonlinear corrections to linear predictions through neural networks, thereby enhancing the expressive capacity of línear models. Thus, the underlying assumption is that the data can be well described by adding nonlinear refinements on top of a linear model. In settings dominated by strong self inferinteratsoons highly nonlinear regimes, NIMO cannot fully capture the underlying relationships in an interpretable manner. Assume the data generating process contains self-interacting term of the form zj (x,), where f is linear or nonlinear function (e.g. x sin(x)) 2²)). To maintain interpretability of the linear coefficients, we imposed the constraint that Ju, (x-j) does not depend on which prevents NIMO from learning such self-interacting nonlinearities.
However, if such a feature is known to play an important role in the dataset, simple feature engineering can resolve the issue. To experimentally and explicitly illustrate both the limitation of NIMO and straightforward remedy, we conduct an ablation study on a synthetic dataset. The data are generated as
= == 2 +
Empirically, we demonstrate that naively applying NIMO to this dataset results in poor performance. However, after applying polynomial basis expansion of degree 2, NIMO can perfectly solve the problem while still providing global, population-leve interpretability through the linear coefficients B. More detailed experimental results can be found in Appendix D.6
CONCLUSIONS
In this paper, we propose nonlinear interpretable model (NIMO). Unlike other hybrid approaches. our model preserves the interpretability of linear coefficients, capturing the additive contribution of each feature to the prediction, while also accommodating nonlinear interactions. Interpretability is formalized through marginal effects, with the marginal effects at the mean (MEM) providing global, population-level summaries that align directly with the linear coefficients. To enable effective training, we introduce an optimization algorithm based parameter elimination, and enforce sparsity through adaptive ridge regularization. We demonstrate the effectiveness of the proposed approach on both synthetic and real datasets, showing competitive predictive performance while delivering meaningful and globally consistent interpretations.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoftrey E. Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699-4711, 2021.
David A Belsley, Edwin Kuh, and Roy E Welsch. Regression diagnostics: Identifying inftuential data and sources of collinearity. John Wiley & Sons, 2005.
Leo Breiman, Jerome Friedman, Richard Olshen, and Charles J Stone. Classification and regression trees. Chapman and Hall/CRC, 2017.
Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. Node-gam: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.00 111000000611,... 2021.
Bradley Efron, Trevor Hastie, lain Johnstone, and Robert Tibshirani. Least angle regression. The doi: 1..12400095336040000066
Annals of Statisticss, 32(2, April 2004. ISSN 0090-5364.
URL http: dei wtt.a//...................0....0
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, (4277639):15--188 2017.
William Falcon and The PyTorch Lightnnng team. PyTorch Lightning, March 2019. URL https
//github. com
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456): 134--1360, 2001.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
w/wwwddeeplearaanngiiee
Yves Grandvalet. Least absolute shrinkage is equivalent to quadratic penalization. In ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Networks, Skövde, Sweden, 2-4 September 1998 8. pp. 201-206. Springer, 1998.
Yves Grandvalet and Stéphane Canu. Outcomes of the equivalence of adaptive ridge with least absolute shrinkage. Advances in neural information processing systems, 11, 1998.
Peter Green. Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. Journal of the Royal Statistical Society: Series B (Methodological), 1692):149-170, 1984.
Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a supercon- ductor. Computational Materials Science, 154:346-355 2018
Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009.
Aya Abdelsalam Ismail, Sercan Ö Arik, Jinsung Yoon, Ankur Taly, Soheil Feizi, and Tomas Pfister. Interpretable mixture of experts. arXiv preprint arXiv:2206.02107, 2022.
Arlind Kadra, Sebastian Pineda Arango, and Josif Grabocka. Interpretable mesomorphic networks for tabular data. Advances in Neural Information Processing Systems,
2024.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learming Research, 22(127):1-22) 2021.
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 623-631, 2013.
11

---

Published as a conference paper at ICLR 2026
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
McCullagh and J.A. Nelder. Generalized Linear Models, Second Edition. Chapman Hal//CRC Monographs Statistics & Applied Probability. Taylor & Francis, 1989. ISBN 9780412317606.
Christoph Molnar. Interpretable Machine Learning. Christoph Molnar, 3 edition, 2025. ISBN 978-3- book
911578-03-5. URLhttts:://dit st.ohm. weeeipph.....iiii. gittu...... htt.tt..o..ttetrrrreeeeeeeeeeeee hww.n//erreeeaaeleeeeeeeeeee
Susan A Murphy and Aad W Van der Vaart. On profile likelihood. Journal of the American Statistical Association, 95(450):4494465 2000.
Marcello Massimo Negri, Fabricio Arend Torres, and Volker Roth. Conditional matrix flows for gaussian graphical models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Mike Nguyen. Guide on Data Analysis. Bookdown, 2020. URL BtLttttss////oooownt org/ mi tilkeeddata_aaallssi..
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veto,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. Advances in Neural Information Processing Systems, 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion. O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2011.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-1144, 2016.
Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1.0: an open source dataset collection for benchmarking machine learning methods. arXiv preprint (000::20/2.0005882 2021.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
Christian A Scholbeck, Christoph Molnar, Christian Heumann, Bernd Bischl, and Giuseppe Casalic- chio. Sampling, intervention, prediction, aggregation: generalized framework for model-agnostic interpretations. In Joint European conference on machine learning and knowledge discovery in databases, pp. 205-216. Springer, 2019.
Christian A Scholbeck, Giuseppe Casalicchio, Christoph Molnar, Bernd Bischl, and Christian Heumann. Marginal effects for non-linear prediction functions. Data Mining and Knowledge Discovery, 3855)299773342, 2024.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-basee based localiza- tion. International journal of computer vision, 128:336 359, 2020.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Ryan Thompson, Amir Dezfouli, and Robert Kohn. deep neural networks. Advances in Neural Information Processing Systems, 19:19940-19961 2023.
The contextual lasso: Sparse linear models via
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1 (16672277-288 1996.
12

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
D. Venzon and S. H. Moolgavkar, A method for computing profile-iikelihodd-beeee confidence intervals. Journal of the Royal Statistical Society. Series C (Applied Statistics), 37(1):87-94, 1988. ISSN 00359254, 14679876.
Mike Wu, M Hughes, Sonali Parbhoo, and Doshi-Velez. Beyond sparsity: Tree-based regulariza- tion of deep models for interpretability. In In; Neural Information Processing Systems (NIPS) Conference. Transparent and Interpretable Machine Learning in Safety Critical Environments (TIML) Workshop, 2017.
Mike Wu, Sonali Parbhoo, Michael C Hughes, Volker Roth, and Finale Doshi Velez. Optimizing for interpretability in deep neural networks with tree regularization. Journal of Artificial Intelligence Research, 72:1-37, 2021.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49-6) 67, 2006.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Equivalence of Adaptive Ridge regression and Lasso
15
Training via parameter elimination
16
B.1
Regression setting: adaptive ridge regression
16
B.2
Logistic regression: Iteratively reweighted least squares
17
B.3
Extension sub-( pscudo-norms
18
NIMO: Implementation details
19
Model Definition
19
Practical Implementation
19
Experiments
20
D.I
Methods implementation
20
D.2
Synthetic experiments: Setup
21
D.3
Synthetic experiments: Additional results
22
D.4
Real-dataset experiments
................................
23
D.5
Ablation study of feature masking
25
D.6
Limitations of NIMO
................................
26
D.7
Local Explanations
27
D.8
Comparison of the Degree of Nonlinearity
28
................................
D.9
Ablation study of the sparse penalty on the first FC
layer
29
D. 10 Extra Results PMLB Benchmark
29
LLM statement
29
14

---

Published as a conference paper at ICLR 2026
A EQUIVALENCE OF ADAPTIVE RIDGE REGRESSION AND LASSO
The equivalence in the solution of Adaptive ridge regression and Lasso regression is well-known fact (Tibshirani 1996; Grandvalet, 1998, Grandvalet & Canu 1998). Since both our model relies on this fact both in the regression and classification settings, we report its proof below.
Proof. Let X E Raxd be n d-dimensional observations and IR" be the targets. In linear model the relationship between target and data assumed to be linear:
y x + 6,,
(10)
where [ RR is the coefficient vector and (N is standard Gaussian noise. In Ridge Regression a penalized version the objective is minimized:
min X3 - y1
33.
(11)
where À weighs the penalty term. In Adaptive Ridge regression, instead of single X, a set of feature-specific penalties {1r]/= are introduced. The objective then reads as
t. Σ d
min X33 y/1² + 3v
Σ
(12)
i=1
where V/ 2 0 Výj and a predefined value. One can solve the above constrained optimization through the method of Lagrange multipliers:
1
((v, n) || y/12
(13)
1=1 1/
where j is the Lagrangian multiplier and v = (.,,..., va)². The stationary points can found by taking the gradients with respect to follows:
0L(U, Ovk
$3 1wv/ = 0
V = n/vk or vx = 1/2
(14)
we now substitute val = 12 2 in the Adaptive Ridge objective in Eq. 12 we get the Lasso objective:
f(8,v) = |XB yy12 +
= |X3 - y/12 +
1: 18.1
(15)
d
|XB - y11 +
4 |3.1
1=1
X y112 + 1
[.
which gives the relationship between adaptive ridge regression and Lasso regression, with the Lasso penalty parameter optimum to be 2=/2 2 Σ [3c]:
Σ Pk = Σ / equation pd
3. = p/V2
Ub
X
Σ
Σ vn /3ll 1.8e
equction
4
1BB 11 1/2
(16)
X
3
1/2
13
15

---

Published as a conference paper at ICLR 2026
B TRAINING VIA PARAMETER ELIMINATION
The proposed model in Eq. 2 has two sets of parameters: the linear coefficients B and the neural network parameters Since these two sets of parameters tightly entangled, jointly optimizing them via gradient descent not straightforward ity of the model, also aim achieve sparsity in the learned coefficients. In this section, show how to achieve these optimization goals via parameter climination. Specifically, we eliminate by expressing it in terms of the neural network parameters u and substituting it back into the optimization objective. The resulting problem can then be optimized effectively and efficiently using gradient descent. Sparsity achieved by reformulating the problem as an adaptive ridge regression. We further show that this approach can be extended to generalized linear models, and we illustrate it detail for logistic regression.
practice. Furthermore, to increase the interpretabil-
B. REGRESSION SETTING: ADAPTIVE RIDGE REGRESSION
We already showed that the proposed model in Eq. 2 can be re-written in matrix form as:
f(X) BuB with Bu X X
(17)
where E Rd are the linear coefficients, G. E Raxd is the neural network output with inputs X E Rxd and "o" denotes element-wis multiplication. The linear model part is then expressed as:
B,, +
(18)
where y E R denotes the targets. Adaptive Ridge regression minimizes the following objective:
d BuB - y112 + Σ v.8 s.t. 8,u
d Σ 1=1
d X
min
(19)
where > 0 are the penalty parameters for each coefficient 3,, and À >> 0 is a predefined param- eter. To avoid divergent solutions we use the re- re-parameterization employed Grandvalet (1988;
Grandvalet & Canu (1998):
%t = (2//A)1/28, c; (A/v;)2/2.
(20)
we substitute 8 and v; into the Adaptive Ridge regression objective, we get the following opti- mization problem:
d
min y BaD +|bb| t.t. Σ 2 = 4 2 0, 7.c.u
(21)
where De is diagonal matrix whose diagonal elements given by the vector [c, ........ cdl T. Applying the parameter elimination trick, if we fix e and u, 7 can be expressed in a closed-form in terms of and u:
(c, u) (D. A1) 1 D.B..
(22)
We substitute (c, u) back into the original problem and take the constraint into consid- eration. This leads us to a Lagrangian minimization problem in the scaling variable e and neural network parameters u:
min y BuDef(c, u + Al|f(c, w)ll m/cl C.U
(23)
where μ is the Lagrangian multiplier introduced for c, and we can optimize together through gradient descent. The training algorithm for this re- re-parameterized adaptive ridge regression is summarized in Alg.1
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 Training of Re- Re-parameterized Adaptive Ridge Regression
Require: Training data X E Raxd, training target y E Re, neural network gu(+), regularization parameter À and jr, learning rate maximum iterations T. 1: Initialize network parameters u and scaling coefficients c (with oj 2 0).
2: for t = 1...., do
3: Bu = =x= +
// Compute modified design matrix // Form diagonal scaling matrix
4: De diag(c)
5:
4 = (DeB u Bu De + NT) B.
// Solve for f // Compute regression coefficients // Compute loss function // Gradient descent updates
6:
ß coÿ 9
7:
L Jly B + ^lllll + p||c|22
8:
C c c V
u + v
9:
end for
10: Return 3,c,u
// Output final parameters
B.2 LOGISTIC REGRESSION: ITERATIVELY REWEIGHITE LEAST SQUARES
The procedure followed in the previous section can be extended to generalized linear models. Here we showcase how to do it for logistic regression. Logistic regression is a classification model that describes the probability that binary outcome {0, } occurs given feature vector E Rd:
P(y 1 x)) (x ()
(24)
where (=)
I+e- is the sigmoid function. The model can then be trained by minimizing the negative log- log-likelihood. Similarly to the regression case discussed in the previous section, we add an adaptive ridge regression penalty to the objective:
((3) = > In log (x{ (1 y.) log(1- o xxf
ox
B))]+
Σ
1388 j=1
s. t.
Σ j=1
1
(25)
Fj
Now we can use the same re- paraameteriiztion used in the previous section for regression. We get the following standard ridge-penalized logistic regression:
min Y.C
y log ((x, ²c) + (1 y:) log (1 ((x, (c) A//7l1
(26)
s.t.
Σ
o
Cj 20,
where c:= [ci, ca) is element-wise multiplication. The same as what we done for regression, we can eliminate y by expressing it in closed-form with respect to c. To get this closed-form expression of y, use the iteratively reweighted least squares (IRLS) algorithm. The associated update reads as:
((+1 (c) = (XTWOX X (0)
(27)
where X is the modified design matrix with entries ž.j CyTig W(( is diagonal weight matrix with entries w!at p; (t)
(1 P. (t)
),), ),pd Ite u0y and z is the the vector of working response
with entries 2.0h (0) 2- 2ー We then substitute > ((+1) back into the original problem,
resulting the optimization nonlinear term as well, so we would need to replace X with B. = X + o Gu- The rest goes the same as the case for regression.
e only. Note that in model, design matrix X contains the
17

---

Published as a conference paper at ICLR 2026
B.3 EXTENSION TO SUB-L1 PSEUDO-NORMS
In some cases sparsity is a crucial aspect in the learning process. One of the limitations of lasso is the well-known over-shrinkage effect, which means that shrinkage of uninformative features could be achieved at the expense of shrinking the retained features as well. One way to avoid this is to use sub-&1 pseudo-norms, which the limits of the pseudo-norm achieve sparsity without any shrinkage. The main challenge with sub-&1 pseudo-norm is that the resulting objective function becomes non-convex and harder to optimize. However, recent work has successfully used sub-L norms as prior on regression problems (Negri et al.) 2023), which is similar to our setting.
We now show how to use a sub-L1 prior in our Adaptive Ridge regression objective, which we report below for completeness:
d s.t.
d
Σ
min XB yl1 +
(28)
X
In order to generalize to sub-l pseudo-norms, we need the following simple modification on 24:
Σ Σ 06851.
(29)
We now apply the same re-parameterization:
= (v: = 11/2))1/2
(30)
The objective function then reads
s.t. min Iyy XD27 1|yy1 Y.C
Ee X.C, 4 a>0 0.
(31)
i=1
Fixing c, we can eliminate y by expressing it in closed-form in terms of
+(c) (D.X* + A1) DeX
(32)
Substituting it back into the original optimization problem and taking the constraint into consideration lead us Lagrangian minimization problem in the scaling variable only:
min |yy - XD. Al(((c) +
26
(33)
and we can optimize it through gradient descent.
To verify that this constraint leads sub-Cı pseudo-norm to the original B3, first we reformulate the original objective with the method of Lagrangian multipliers:
((v, |X3 y11 +
(34)
Then, we find stationary points for vs by taking the gradients and setting them to zero:
0C(v,p) .ōo;
- (6+1)
0
6+1
pd
0v:
3
(35)
μu
The penalty in the original adaptive ridge regression be rewritten
(36)
Note that when 5 = 1, we recover the Lasso e norm. When 8 + 0, it approaches the lo pseudo-norm, which basically counts the number of nonzero entries.
18

---

Published as a conference paper at ICLR 2026
C NIMO: IMPLEMENTATION DETAILS
In this section, we provide detailed introduction to the structure of NIMO, along with the strategies used to improve training.
C. MODEL DEFINITION
We define NIMO as follow:
f(x) Bo + Σ (1 + Iu, (x-;))
s.t. Ju, (0) 0.
(37)
The architecture of our model consists of two components, as illustrated in Fig. The left component is nonlinear neural network that operates on the imput features X-j, where the j-th element has been masked out. The right component is linear model, in which coefficients 3y are multiplied by the corresponding neural network outputs and subsequently summed.
In practice, for each feature component zy of the input vector x, the value temporarily masked to form x-j. which is then passed through neural network gu,- The output of this network is added to constant and multiplied by the original feature value Ty, resulting in a rescaled version of the feature:
(38) This rescaling can be interpreted as an element-wise modulation of the input features based on context. The rescaled features are then linearly combined using fixed coefficients 33 to produce the final prediction:
f = x. (1 Duy (x-)))
y 3o
33
C.2 PRACTICAL IMPLEMENTATION
We implement our model with PyTorch Lightning (Falcon & The PyTorch Lightning team 2019). While the model definition is simple, in practice, several strategies are needed to make the training more efficient and effective.
Positional Encoding In principle each feature x, could have its dedicated network Juy, but this can be computationally expensive for high-dimensional inputs. To reduce overhead, we instead shared neural network Ju across all components. To ensure the network remains aware of which feature is currently masked out, we append a positional encoding to x-j before passing it through the shared network. We assign each masked input feature x-j unique binary vector corresponding to its index j. represented in fixed-length binary form using [log, d + 1 bits. This compact encoding allows the shared neural network to remain index-awar with minimal overhead.
Group Penalty and Noise Injection In the main paper we showed that linear coefficients & in NIMO are interpretable as linear model. Even though the neural network contribution is not interpretable per se, can still identify relevant nonlinear interactions by means of sparsity. The idea is to force the first layer the neural network to be sparse, hence to use only relevant features. To do so, apply group penalty the weight matrix W E Roxd of the first fully-connected (fc) layer, where d the number of input features and p is the hidden dimension of the first fc layer. In order obtain sparsity at the input feature level, we need consider neurons connected to the same input feature and penalize them as a group:
L groupp
A Agroup
1wwll2 122
(39)
where w, W|:jj] Rº is the weight vector acting the j-th feature. With this penalty the first Ec layer, we encourage to select only certain input features. However, since an MLP computes composition of functions, the subsequent fc layers still compensate for the sparsity induced
19

---

Published as a conference paper at ICLR 2026
by the regularization in the first layer. Specifically, even if the first layer produces sparse outputs, subsequent layers can reweigh and transform these outputs to restore their influence on the final prediction. To mitigate this compensation effect, we inject noise into the output of the first layer. By doing so, we disrupt the deterministic signal path, making it harder for the subsequent layers to simply exploit the resulting weak, regularized, and noisy signal. As result, the network is encouraged to focus on strong and robust signals associated with informative input features. This approach works very well in practice and allows easily achieve sparsity in the neural network.
Other Details After the second fc layer, sin activation function is applied to capture potentially repeating or high-frequency cyclic patterns in the data (Sitzmann et al., 2020). The learning rate 1e-3 for synthetic experiments, and 5e-3 3 for real-world datasets. We choose Adam (Kingma) 2014) the default optimizer. In all experiments, the lasso and group penalties are selected by performing grid search over a predefined set of candidate values.
D EXPERIMENTS
We conduct experiments on both synthetic and real-world datasets. All experiments conducted on a workstation equipped with a single user-grade NVIDIA GeForce RTX 5080 GPU. The methods we compare with are linear model (Lasso for regression, Logistic Regression for classification), vanilla neural network (NN) (Goodfellow et al., 2016), and several state of the art interpretable approaches. namely LassoNet (Lemhadri et al. 2021). Neural Additive Models (NAMs) (Agarwal al. 2021)
and Interpretable Mesomorphic Networks (IMNs) (Kadra et al. 2024
D.1
METHODS IMPLEMENTATION
Linear Model We use scikit-learn (Pedregosa al. 2011 regression model. More specifically, we use LassoCV and default 5-fold cross validation select the best model. For Logist welssiicReeee wetsssiiieeeesiinnyy..... the L penalty is applied.
implement the Lasso and Logistic
with
Neural Network We implement the naive neural network (NN) with PyTorch Lightnnng (Falcon & The PyTorch Lightning team 2019). For a fair comparison, the NN has the same structure as the nonlinear part in our NIMO model, except no positional encoding. We use Adam (Kingma 2014) as the default optimizer. The learning rate is 1e-3 for synthetic experiments, and 5e-3 for real-world datasets. Strong dropout regularization (p=0. 6) is used to avoid overfitting for small scale datasets (including synthetic datasets, diabetes and Bostong housing datasets). For the relatively large superconductivity dataset, small dropout rate (p=0. 1) is used.
LassoNet We use the original implementation of LassoNet from their GitHub repository Using the same API as shown in the LassoNet document, we configure the model to have a structure and number of parameters comparable to our model. All other hyperparameters are kept at their default values. The best model is automatically selected along the entire regularization path, as demonstrated in their paper. The one with the smallest validation loss chosen.
NAM The official NAM implementation was in Tensorflow. We chose a well cell-maintained Py Torch implementation from the GitHub repository This implementation provides scikit-learn style interface, which can be easily used.
IMN
We use the original Torch implementation of IMN from their GitHub repository f All the arguments kept at their default values except modifying the data loading logic to fit our datasets.
For all the methods above, we tried our best to make sure that they achieve reasonably good and consistent performance on all the datasets.
hhttss//iihuub.com/asooo//////ss httpp://ggihub..co///mmmeh// httpps///gihuu.oo//A/iiiintiiii
20

---

Published as a conference paper at ICLR 2026
D.2 SYNTHETIC EXPERIMENTS: SETUP
We create the synthetic datasets by first sampling the features, and then compute the targets according different nonlinear pattems.. Random Gaussian noises are added the targets. The feature values are sampled uniformly between -2 and 2. For logistic regression, we pass the targets through logistic (sigmoid) function to get probabilities between 0 and 1, and then sample the final binary labels using Bernoulli distribution with those probabilities. In all the settings, we use 200 training samples and 100 samples each for validation and test.
Settings for regression All features are indexed starting from 1. N/ 0.12) is Gaussian noise.
Setting 0 (toy example), n = 200, 3
y = +3 x1 [1+ (10x2)
-3.22
[1 sin( (22x))
(40)
+
Setting 1, n 200, p
y 3 -1
[1 (20 1)]
2 32
(41)
2 2.3
+ e
Setting 2, n = 200, p = 10
y = + 1 Ø1
[] + tanh(2223 + sin(x44))
+ 22.22 2.22
[ + sin(2x))
2
(42)
1 1.23
1+ (rctan(x221))
+
Setting 3, n = 200, p = 50
y = 21 [1 + tanh(52244)
+ 22.22
1+ arctan(24 - xz)
(43)
3.114
[1 tanh(12 sin())))]
1 1.15
[1 [1+ (20(21)11 - 1)]
+
Setting 4 (vanilla regression). = 200, p 10
y - 4 x2 3- +0
+ 11 + xg + 44.
(44)
+
Settings for classification In order to create data for synthetic classification we follow a similar procedure as in the regression cases, where we directly control the linear coefficients and the nonlinear interactions. The only difference is that we further apply link function and binarize the output and get the labels. Note that this way we loose information about the original linear coefficients, which cannot be recovered exactly anymore. The sparsity in the coefficients still remains crucial. For all the settings, we generate y first, and then generate labels through the following procedure:
label Bernoulli())
1+e-v
21

---

Published as a conference paper at ICLR 2026
Table 3: Classification accuracy for synthetic classification settings. regression, NN and LassoNet. In each setting we use 200 samples.
We compare with logistic
Features
Log. Regr.
NN
NIMO
LassoNet
Setting I
0.59
0.90
0.92
0.89
Setting 2
10 50
0.68
0.83
0.85
0.82
Setting 3
0.74
0.69
0.84
0.90
Setting 1, n 200, p
y = + 21
[1 tanh(2z)]
2 22
[1+ 32n()))) tanh(2x1))
(45)
+ +
Setting 2, 200, 10
y = + 10. [1 2 tanh(2x2) + sin(x1))
+ 20 -2
[1 cos (2x))
20 13
[1+ + arctan(
(46)
+ 10 F4
10 +
Setting 3, n 200, 50
= 20.20 01 [1 + tanh tanh(12244)
2
+ 20-22
1+ arctan( x35)
(47)
30 24 [1 + tanh(x2 + sin(xs)))
10
[1 (20(Z111) - 1)]
+
D.3 SYNTHETIC EXPERIMENTS: ADDITIONAL RESULTS
Below, we present additional experimental results for the synthetic settings described above.
Regression As we mentioned in the main paper, we further apply group & regularization to the weight matrix of the first fully connected (fc) layer, encouraging feature-level sparsity and clarifying how inputs contribute to predictions. We provide the details in Appendix C.2 Here, we show the sparse features selected by the neural network Figure We observe that even though all features input to the neural network, only small subset of features
selected. Moreover, these selected features align with those involved in the nonlinearity computation in our settings, which demonstrates the model's ability to effectively identify and focus the most relevant features for the task.
Classification Our method can be extended to generalized linear models and here we showcase it for logistic regression. Also in this case we show various settings for different dimensionality, nonlinear terms and different number of uninformative features. In Table we we report the classification accuracy on the test for all methods. Overall, our model significantly outperforms logistic regression on all settings. When analyzing the learned coefficient, we can see that only NIMO recovers the correct sparsity patterns. As argued before, this allows to learn the correct nonlinear interactions. Other nonlincar models and hybrid approaches perform similarly to NIMO but fail to provide the interpretability of the coefficients as in a linear model.
In classification, the presence of the link function can obscure the underlying signal, so while linear decision boundary may suffice, recovering the true generative coefficients is more difficult than in regression, where the model directly learns to approximate the target surface. Across all settings, the goal not to recover the precise magnitudes of the ground truth coefficients, but to accurately identify their sparsity pattern and directional effects (i.e., signs). We compare the recovered coefficients from
22

---

Published as a conference paper at ICLR 2026
First FC Layer Weights
635
20
20
Hagrituue
30-
20
-33
00
30
11
16
21
26
31
36
Irpuc Festures
Input Footures
Input Features
Figure 8: Sparsity and feature selection in the first fc layer of NIMO for regression settings. Left: Setting 1. Middle: Setting 2. Right: Setting 3.
logistic regression and our model across three classification settings, as shown in Figure 9 Since recovering the exact magnitudes of the coefficients not feasible, we normalize all coefficients, including the ground truth, by dividing them by the maximum absolute value among the coefficients, to enable a fair comparison.
Sotting
Setting
Setting 3
1.0
0.0
-0.5
Log. Rog NIMO
Ground Truth
Figure 9: Learned coefficients for synthetic classification settings. We compare NIMO and Logistic regression with the ground truth. Since we binarized labels is impossible retrieve the exact coefficients. Instead, we compare normalized coefficients, such that the maximum is always 1.
We observe that logistic regression struggles to identify the relevant features and their correct signs. In contrast, our model generally succeeds in selecting the correct features, and the neural network component effectively captures sparse but important features that contribute to nonlinearity. It is worth noting that NIMO also selects some noisy features, especially in Setting 3. However, our experiments primarily intended as proof of concept demonstrate that model can be easily extended to generalized linear models. With careful hyperparameter tuning, NIMO capable of selecting more accurate features. The sparse features selected by the neural network are shown in Figure 10
D.4 REAL-DATASET EXPERIMENTS
We conduct experiments three real-worle datasets: the diabetes dataset (Efron al. 2004), the Boston housing dataset (Belsley et al. 2005) and the superconductivity dataset (Hamidich] 2018). Each dataset is randomly split into training (60%), validation (20%), and test (20%) sets, repeated
23

---

Published as a conference paper at ICLR 2026
First FC Layer Weights
1.00
0.75
10-
10
0.50
15
0.25
Weigght Magnitude
Nourors
30
30
20
0.00
-0.25
50
35
-1.00
11
16
21
26
31
36
41
46
Features
Inpuc
Figure 10: Sparsity and feature selection in the first fc layer of NIMO for classification settings. Left: Setting Middle: Setting Right: Setting 3.
over 5 independent runs. For each split, perform grid search to select the optimal penalty parameter based on the validation loss (or validation accuracy for classification tasks). Error bars and confidence intervals reported across these different splits. The main results shown in Figure
Diabetes dataset The diabetes dataset is classic regression dataset consisting of 442 patients, each with 10 numeric features, and the target measures disease progression one year after baseline. As shown in Figure 11 (left), we observe that all methods achieve similar MSE losses and the neural network (also the hybird methods) shows no clear advantage on this dataset. Since the diabetes dataset is commonly used for linear regression tasks, it is not surprising that the data does not exhibit complex nonlinear interactions among features. We can verify this claim by looking at the weight sparsity of the first fc layer in NIMO, shown in Figure IT (right). Relevantly, NIMO achieves significantly sparser solution than Lasso regression, see Figure (middle). This is particularly evident by looking the coefficients associated with the feature "sl" "s2" and "s4".
0.4
Metrod
Coefficient Value
02-
2.50
2-2
1080 381 1001
Figure 11: Diabetes datasets results. Left: MSE loss on test set. All methods achieve comparable results. Middle: Learned linear coefficients by Lasso and NIMO. NIMO selects sparser coefficients ("s1, "s2", ; "s4"). Right: the first fc layer of NIMO very sparse; few nonlinearities are relevant.
Boston dataset The Boston housing dataset contains 506 instances, each with 13 features, and the target is the median value of owner-occupied homes. As shown in Figure 12 (left), our method performs significantly better than Lasso regression and shows compatible MSE test loss with the other nonlinear and hybrid approaches. This suggests that nonlinear interactions crucial to explain the data. While the other methods are able to capture those nonlinearities, they lack the interpretability provided by model. It is important note that the Boston housing dataset has been criticized
24

---

Published as a conference paper at ICLR 2026
for containing controversial features The feature of interest is termed "B". From the coefficient plot Figure 12 (middle). we observe that the significance of the feature "B" is greatly reduced in model compared to Lasso regression. This surprising result suggests that the role of the feature "B" towards the prediction might be irrelevant. Note that the feature "B" is also not relevant for the nonlinear terms, shown in Figure (right).
Pittcc
Coefficient Value
3.2
Whightt Magnitude
0.1
033
-010 750
crop
Figure 12: Boston datasets results. Left: Comparison of MSE losses. NIMO performs on par with other methods and significantly outperforms Lasso, suggesting nonlinear interactions are crucial. Middle: Learned linear coefficients by Lasso and NIMO. The two models learn coefficient with comparable sparsity. Relevantly, the feature "B" almost irrelevant for NIMO. Right: sparsity in the first fc layer of NIMO.
The superconductivity dataset contains 21,263 instances, each with features. The target is to predict the critical temperature based on the features extracted. In Figure 13 we
Superconductivity
Firs PC Layer Wagt
1.00
0.25
0.50
0.50
0.00
0.355
59
0.125
605
0300
Figure 13: Superconductivity dataset results. Left: Comparison of MSE losses. Right: sparsity in the first fc layer NIMO.
observe that NIMO achieves the best performance comparable general neural network. With respect to sparsity in the first fc layer, the presence of strong correlations among several of the 81 features makes it challenging obtain clear sparsity patterns. Nevertheless, certain patterns can still be discemed (Figure 13 right). Figure 14 shows the coefficients for Lasso and NIMO on this dataset.
D.5 ABLATION STUDY OF FEATURE MASKING
To verify the effectiveness of masking out the j-th component of an instance when passing it through the neural network, we conducted an ablation study. Specifically, we examined the effect masking out the j-th component as well as the effect of removing position encoding. From the ablation results across the three settings (Figure Figure and Figure 16). we observe several consistent patterns:
https//farttemm.ogg/aannssee quide/dlataseet/hostonn housing htta.html
25

---

Published as a conference paper at ICLR 2026
Figure 14: Coefficients (also MEMs) for Lasso and NIMO on superconductivity dataset.
Without masking the j-th component, the first fo layer of the neural network can still exhibit sparsity patterns, but they are less distinct than when masking is applied. We attribute this to the position encoding, which enables the network to identify the feature index.
Without masking the j-th component, the magnitudes of the linear coefficients are distorted, reducing the interpretability of the marginal effects at the mean (MEM).
Further removing the position encoding distorts the sparsity patterns in the first fc layer and substantially alters the magnitude of the linear coefficients, thereby undermining inter- pretability.
Unar Coafidess
Figure 15: The effect of feature masking and position encoding on the linear coefficients and the sparsity of the first fc layer on setting 2.
The effect of feature masking and position encoding on the linear coefficients and the sparsity of the first layer on setting 3. Coefficients are displayed for the first 10 features.
Figure 16:
D.6 LIMITATIONS OF NIMO
To clearly and explicitly illustrate the limitations of our model, we conduct an additional set of ablation studies in this subsection. As discussed above, due to its architectural constraints, NIMO is not able to capture datasets with strong nonlinearity or purely self-interacting features. To demonstrate this, create the following synthetic dataset:
.
2 + 23 + +
(48)
where the feature values sampled uniformly from [- and (00 1) is Gaussian noise. We compare the predictive performance of Lasso, NIMO, and LassoNet in Table As expected, NIMO performs similarly poor to Lasso, while LassoNet is still able model this dataset effectively. From
26

---

Published as a conference paper at ICLR 2026
Figure (left), observe that the coefficients learned by NIMO closely resemble those of Lasso, and NIMO effectively learns only a constant intercept 8g. In addition, we plot the weight matrix of the first fc layer of the neural network and observe that all weights confirms that, in this setting, NIMO reduces to purely linear model without contributing additional nonlinear modeling capacity. This outcome is fully consistent with expectations and illustrates known limitation of NIMO: it cannot recover sell-nonlinearities such as zj due to its interpretability- preserving architectural design. Although LassoNet is able model this type of nonlinearity, it does not offer the same ability as NIMO or linear models to provide global, population-leve interpretability through stable and directly interpretable learned coefficients.
close to zero. This further
Table Predictive performance on the synthetic self-interaction dataset. NIMO performs similarly to Lasso in the original feature space but succeeds after basis expansion, where quadratic terms are explicitly included.
Method
Original
Basis Expansion
(#fcatures)
(3)
(9)
Lasso
5.307
1.009
LassoNet
1.027
1.012
NIMO
5.340
1.009
Lonk
Figure 17: Learned coefficients and first-layer weights on the self-interactin dataset. (Left) In the original feature space. NIMO collapses to a constant model, matching Lasso. (Right) With basis expansion, NIMO recovers the quadratic effects.
However, if we know or suspect that certain feature interactions play role in the dataset, we can model them explicitly through feature engineering. this experiment, we apply second-order polynomial basis expansion, transforming the original feature space [æ1, #2, 23) into [21, Ø2, æ3, 21: 22 22 22, T243, xz]- From Table we can see that after basis expansion, both NIMO and Lasso achieve performance comparable to LassoNet. In Figure 17 (right), NIMO also correctly recovers the linear coefficients associated with the quadratic terms af. 23. and az. demonstrating that NIMO can model self noniineninsssss once they are explicitly included in the feature representation. In summary, while NIMO cannot capture self-interactions in the raw feature space due its interpretabilityy prreeeivig architecture, simple basis expansions allow it to model such nonlinearities effectively while retaining global interpretability.
D.7 LOCAL EXPLANATIONS
NIMO provides intelligible global interpretability through the marginal effects at the mean (MEM) and per-instance adjustments via the correction networks. In the sections above, have demonstrated the effectiveness of NIMO's global interpretation. Here, we briefly showcase its ability to provide meaningful local, per-instance explanations.
Local Explanation on Synthetic Dataset. for each feature,
For local explanations, NIMO defines local coefficient
s, sx(x) (1 Iu, (x-))))
(49)
which represents how the global effect B, is modulated by the specific input context. Based on this, we compute the local contribution
Cs (x(x) 2. (x))
(50)
27

---

Published as a conference paper at ICLR 2026
for each feature at each instance. The scatter plots in Figure compare the ground-truth contribution term for feature j with NIMO's estimated local contribution cy for synthetic setting 1. In this setting, the dataset contains five features, but only the first three have nonzero linear coefficients in the data-generating process. As shown in the figure, the first three features exhibit scatter points that align closely with the identity line, while the remaining two features produce scatter distributions centered near the origin. These results indicate that NIMO's local explanations faithfully capture the underlying structure of the data.
03-
-184
To. Dhei0
Toe
Figure 18: Local explanations for each feature on the synthetic setting 1.
Local Explanation on Real Dataset. For real datasets, where ground-truth local contributions are unavailable, we compare NIMO's 's local contributions (x(x) with SHAP values. As shown in Figure I9 the scatter plots for most features align closely with the identity line, indicating that NIMO's local explanations are consistent with widely used post-hoc explanation tools while remaining intrinsic to the model. Due to the sparsity constraint, some coefficients learned by NIMO are exactly zero, which forces the corresponding local contributions to be zero as well. In such cases, the scatter plot reflects this behavior clearly. For example, the local contributions for the feature NOX collapse to zero for all instances.
0.10
GING
628
6105
8004
800
800
-808
25 -73
2.5
-50
-301
630
+400
540
A22
MO
-2
J228
660
179070
42
18
Figure 19: Local explanations for each feature on the Boston housing dataset.
D.8 COMPARISON OF THE DEGREE OF NONLINEARITY
In the main paper (Sec. Figure 5). we observed substantial difference in predictive performance between the Diabetes and Boston Housing datasets, and attributed this phenomena the relatively low level of nonlinearity present in the Diabetes dataset. To experimentally and quantitatively verify this claim, we compute the absolute magnitude of the nonlinear correction for each feature.
ny (x) 1 1 9u, (x-;))-
(51)
and then average these values over all test samples for each feature. Figure 20 shows the resulting mean nonlinearity levels for both datasets. The Boston Housing dataset exhibits noticeably larger deviations from across many features, indicating stronger nonlinear interactions. In contrast, the
28

---

Published as a conference paper at ICLR 2026
Diabetes dataset shows values much closer to for all features, confirming that its underlying structure primarily linear. These results support our interpretation of the performance differences observed in the main experiments.
Diabetes: Mean 2,4UJ1
Boston: Mean
yod Now Nontharrty
20-
Maar hommarrty
20
Casd
LSTer
Figure Comparison the degree of nonlinearity the Diabetes and Boston Housing datasets.
D.9 ABLATION STUDY OF THE SPARSE PENALTY ON THE FIRST FC LAYER
To verify whether applying the group penalty to the first fc layer of the neural network is beneficial, conduct ablation study on the synthetic dataset (setting 1). From Figure 21
that
Compaa Ion
Loatted Conffcieess
Wiiggtes (Miruur Perals)
Waightt With Preally
0.90
Stoet Wet TC seliity
0.08
0.05
0.02
6.00
0.00
WA Fourlty
Figure 21: Ablation study of the group penalty the first fc layer.
removing the group penalty on the first layer leads to worse final performance, Furthermore. comparing the learned coefficients shows that they deviate more from the ground truth. Inspecting the weight matrix of the first layer, we also observe that the neural network begins to use the two redundant features, feature3 and feature4, even though they were not used in the data generation process for setting 1 (see D.2).
D.10 EXTRA RESULTS PMLB BENCHMARK
We conduct a more extensive evaluation of prediction performance using the PMLB benchmark Ro- mano et al. (202I) We filter out datasets that were either too small (less than 1,000) or too large (more than 16,000), resulting in total of 10 regression datasets. For each dataset, compare the performance of Lasso, LassoNet, a standard Neural Network, and NIMO. Hyperparameters for all models selected via grid search. The results presented in Figure 22
As discussed the limitations of NIMO (Sec. D.6). some datasets require additional feature engi- neering for NIMO to perform well. Within the PMLB benchmark, indced identified such a case. The 225 puma 8NH dataset cannot be well fitted by NIMO directly (yielding an MSE around 0.4). However, after applying a polynomial basis expansion, NIMO achieves performance comparable LassoNet and Neural Networks.
E
LLM STATEMENT
ChatGPT is used only for polishing the sentences.
29

---

Published as a conference paper at ICLR 2026
Test MSE by Model and Datases
Simgro KN
2.5
20.
229 pomanter
Geopremi attigaattenn tottiaataiicc
1G2 stna
Sa9 poien
204 catetie
AR
Figure 22: Comparison of prediction performance on PMLB.
30


Reviews:
[{"review_id": "T3OLVn51wA", "reviewer": "Reviewer_cUUF", "summary": "Neural Networks are powerful but not inherently interpretable. Linear probes are interpretable but lack expressiveness. The authors aim to bridge the gap by introducing a new model - NIMO. NIMO is motivated by the marginal effect $ME_j$, which measures how much a prediction $f(x)$ changes when changing the value of $x_j$. Specifically the metric marginal effect at the mean $MEM_j$ factorizes out nicely, so that under the mean the marginal effect $MEM_j = \\beta$, the fixed and learned linear coefficient. For the optimization of the model they utilize a profile likelihood approach which reduces the problem, so that only the parameters $u$ need to be trained. Furthermore they extend the idea to generalized linear models. In S3.4 the authors demonstrate that Nimo is able to retrieve accurate linear coeffcients $\\beta$ and that $g_{u_j}(x_{-j})$ is accurately predicting the nonlinear scalars of feature $f_j$. In section 4.2 H1 the authors demonstrate that in 3 synthetic low data settings their method outperforms competitors in MSE. In 4.2 H2 they showcase how Nimo accurately captures the linear coeffcients in a vanilla regression setting and in 3 non linear settings which are further explained in the appendix. Sec 4.2 H3 compares a) the performance of Nimo on real world datsets (diabetis, housing and superconductivity) in which they perform competitive and b) how the shap feature contributions align with the NIMO coefficients.\n\nin 4.3 the authors run ablation studies regarding the design of $g_u$\n\nIn general Nimo contributes a novel model architecture and training recipe to enable interpretable by design and yet powerful AI models.", "questions": "- Did the authors evaluate the statistics of $\\beta_i + g_{u_j}(x_{-j})$? It would be interesting to find out whether the degree of non-linearity can be assessed through such statistics. For example, if the impact of $f_i$ were bi-modal, $\\beta_i$ might be close to zero. I would be particularly interested in the differences between the Diabetes and Boston Housing datasets, as the authors state that the Diabetes dataset exhibits more linear interactions. It would be great if this were directly reflected in the statistics of the effective contributions.\n\n- I would like to see analysis of scenarios in which features do not contribute linearly e.g. on the xor problem. On of my concerns is that Nimo would still be able to solve the problem, but would not be faithful in its $\\beta_i + g_{u_j}(x_{-j})$ score. Is it possible to give guarantees to empirical bounds to the faithfulness of the feature contribution?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors take the definition of MEM seriously and introduce a novel and clever method that enhances MEM interpretability.\n- The Introduction, as well as Sections 3.1 and 3.2, are particularly clear and easy to follow.\n- NIMO can model non-linear relationships while providing linear interpretations at any sample x.\n- The toy model in Section 3.4 is excellent for illustrative purposes.\n- NIMO is competitive with other interpretable models in terms of performance, while offering intriguing properties for feature ranking.", "weaknesses": "* The limitations of this model are not analyzed in great detail. I appreciate the analysis of what this Nimo can do equally as much as the discussion of the limitations. However the limitations are only discussed in a single sentence. Addressing these problems convincingly e.g. by experiments that clearly display key limitations, is crucial imo.\n* The authors themselves acknowledge the importance of both local and global explanations, yet they do not provide any local explanations, but only global explanations.\n* No code is available, severely diminishing means of reproduction.\n* Minor things: In line 414 it would be great to link to the Figure in the Appendix", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l91nsynun9", "reviewer": "Reviewer_F6aQ", "summary": "A new interpretable hybrid neural network architecture is presented. The architecture differs from existing ones by perturbing the weights of a linear model via a neural network. A novel optimization algorithm is presented to fit said architecture, and it is compared to state-of-the-art hybrid neural networks on synthetic and real tabular datasets.", "questions": "In Figure 5, why are plain Neural Networks worse than LassoNet and NIMO on superconductivity? This result is surprising since NNs are in theory more expressive than either of these two other models. Is it because there is some overfitting going on? Were the hyperparameters for NN chosen to maximize validation set performance? Appendix D.1 describes the hyperparameters used, but not why these values were chosen.\n\nAt line 413, it is states that \"[Figure 6 reflects] NIMO’s ability to capture nonlinear feature interactions that Lasso cannot represent.\" I am not sure how to interpret this statement along-side the Figure 6. This is because some features are considered important according to one method and not the other. In diabetes, Lasso considers s1 important while NIMO does not. For Boston, NIMO considers age important, while Lasso does not. Hence it is not immediately apparent that NIMO always captures something that Lasso does not.\nAlso, can the MEM really be used to make statements about \"feature interactions\"? A feature could have a large $\\beta_i$ but still not interact with other features if its perturbation network has a negligible output $f_{-i}(x_{-i})=0$? To provide evidence for feature interactions, it could be useful to report the strength of the coefficient perturbation $|(1+f_{-i}(x_{-i}))|$ for all features.\n\nEnforcing coefficient sparsity is relevant in lasso because it allows for feature selection. However, in NIMO, a coefficient $\\beta_i=0$ does not imply that the model is not relying on $x_i$ to predict. Indeed, $x_i$ is still used in the networks that modify the weights of other features. Given this observation, why is inducing sparsity of $\\beta$ in NIMO (via adaptive ridge regression) desirable in the first place? Are there ways sparsity in NIMO could potentially be used for feature selection?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The optimization algorithm consists of expressing the linear coefficients as the closed-form solution of the loss minimization, and then back-propagate through this closed-form solution to get the gradient of the neural network parameters. This is a novel theoretical and practical contribution.\n\nThe sections that describe the theory and algorithm behind the approach (sections 3.3 and Appendix A, B, C) are very clear and well written.\n\nThe method is compared on a variety of tabular datasets : small (~500 instances ~10 features) and large (20K instances 80 features). This demonstrates that the proposed algorithm scales.", "weaknesses": "## The MEM\n\nThe main motivation behind the proposed architecture is that the Marginal Effect at the Mean (MEM)\\\n$$ \\frac{\\partial f(x)}{\\partial x\\_i}\\bigg\\vert\\_{x=\\bar{x}}$$\nis a useful measure for interpreting the model. Indeed, the main motivation behind architecture is that the learned coefficients $\\beta$ will coincide with the MEM. Hence, the merits of the MEM should be discussed in more depth. For instance, why should we use this measure of feature importance instead of the mean derivative  $\\mathbb{E}\\big[\\frac{\\partial f(X)}{\\partial x\\_i}\\vert\\_{x=X}\\big]$ or the average derivative amplitude \n$\\mathbb{E}\\big[\\big|\\frac{\\partial f(X)}{\\partial x\\_i}\\vert\\_{x=X}\\big|^2\\big]$ as proposed in [1].\n\n## Choice of Synthetic Data\n\nThe ground-truth functions used in the synthetic experiments all take the form $y(x)=\\beta_0 + \\sum_{i=1}^d\\beta_i x_i(1 + f_{-i}(x_{-i}))$, which is the exact form of the NIMO model. This explains why NIMO performs better than competing methods Table 1. It would be fairer to compare NIMO with others architectures on toy models that take a variety of forms e.g. $y(x) = x_1^2 + x_2^2 + x_3^2$ . I would expect NIMO to struggle in this example because the derivative is null at the origin. NIMO cannot perfectly model this function because enforcing a null derivative at the origin requires $\\beta_i=0$ for all indices $i$. So, NIMO must output a constant prediction $f(x)=\\beta_0$. If NIMO can find an adequate estimate of $y(x) = x_1^2 + x_2^2 + x_3^2$, it would be interesting to plot its behavior near the origin.\n\n## Other Hybrid Architectures\n\nAlternative hydrid architectures have been proposed to enable partial interpretability. Mixture of experts [2] employ a neural network to infer weights $w_k(x)$ that balance the decisions of various experts $e_k(x)$  (e.g. linear models). The model takes the form $f(x) = \\sum_{k=1}^K w_k(x) e_k(x)$ and offers local interpretability : when $w_k(x)$ is close to 1 the expert $e_k(x)$ is responsible for the classification of x.\nNODE-GAM [3] restrict the network to take the form $f(x) = \\sum_{i=1}^d f_i(x_i) + \\sum_{i\\lt j} f_{ij}(x_{ij})$, which enables interpretability by plotting the main effects $f_i$ and pair-wise interactions $f_{ij}$. \n\nDiscussing these articles in the related work would strengthen the need for the novel NIMO architecture. Notably, how is reporting the MEM (which is the build-in feature importance of NIMO) better or worse than reporting the coefficients of the local experts in a Mixture of Experts, or reporting the main effects and pair-wise interactions in a NODE-GAM?\n\n\n[1] I.M. Sobol’, S. Kucherenko. (2009) \"Derivative based global sensitivity measures and their link with global sensitivity indices\".\nMathematics and Computers in Simulation, 79(10), 3009-3017.\n\n[2] Ismail, Aya Abdelsalam, et al. \"Interpretable Mixture of Experts.\" Transactions on Machine Learning Research.\n\n[3] Chang, C. H., Caruana, R., & Goldenberg, A. NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning. In International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LgrNs0CBbH", "reviewer": "Reviewer_DuVs", "summary": "The authors introduce NIMO (Nonlinear Interpretable MOdel), a model that fits a neural network to learn non-linear corrections to a linear model's predictions. The model optimizes the linear coefficients analytically while optimize over the neural network parameters. At inference time, the model can be interpreted using the linear parameters. In a toy example, synthetic experiments, and two tabular datasets, NIMO outperforms some standard baselines (Lasso, LassoNet, and a neural net); in real-world datasets it performs on par with these baselines.", "questions": "The rationale for excluding a feature's values when computing its coefficient seems unclear. A nonlinear network should be able to effectively deduce the feature's value anyhow, especially if there are any correlations between features.\n\nSimilarly, the sparsity/noise added to the first layer of the FC network seems strange, as the network can still recombine different features in any nonlinear manner. Can the authors perform some ablations showing the effect of this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- S1: The authors tackle an important problem\n- S2: The authors approach is novel\n- S3: The authors' optimization approach seems generally reasonable", "weaknesses": "- W1: The main weakness seems to be that the model does not provide significant global interpretability, as the model coefficients are a nonlinear function of the input. Thus, the model's provides interpretability similar to instance-level posthoc methods such as LIME or SHAP. The authors should provide a deeper discussion of this and a quantitative comparison showing a scenario where NIMO can provide more interpretability than post-hoc baselines (rather than just a qualitative comparison).\n- W2: The real-dataset experiments are fairly limited, it would be nice to see a more substantial evaluation of prediction performance on popular tabular datasets, e.g. the Tab-Arena benchmark or PMLB.\n- W3: The authors discuss how NIMO reaches a sparser solution than a Lasso baseline (e.g. Fig 4), but it is unclear whether this is useful as NIMO can simply use the non-linear NN to use these features", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)