Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new GAM-like model, denoted as Neural Basis Model (NBM) to analyze the learned feature in machine learning models.  Comparing with the traditional GAM, NBM learns a set of shape functions for each feature using a single neural network. Comparing with NAM, NBM requires less parameters when the dimensionality is greater than ten. The experiment shows NBM can achieve similar or slightly higher results to NAM on multiple datasets using less parameters. The visualized shape functions show that NBM is more stable than NAM, a lower standard deviation is achieved on 100 runs.
strengths_and_weaknesses: The number of parameters of NBM is less than NAM when the dimensionality is high, the trick behind is the convolution-like shared shape function (with a fixed small number of B parameters, Eq.(6)), and this NBM can be extend to the bi-variate situation based on Eq(5). This simple yet effective design outperforms the previous state-of-the-art NAM.
questions: Generally, this draft is well- written and organized, the only question I have is about the "sparse architecture"(line 120). It would be even better to add one more column about the average sparseness of those tabular datasets to Table 1, which could be another factor behind the acceleration shown in Table 2. 
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: # After Rebuttal

I appreciate the authors including multiple datasets in rebuttal, but the performance improvement is still not as big. 
The idea is still somewhat incremental in my idea, but the evaluation seems complete and the speed-up looks good compared to NAM. I believe this paper will have more impact if it releases a good codebase, and also shows more GAM graphs in the image datasets. 

# Original Review

This paper proposes to improve the NAM model by learning a shared basis function for each feature function i.e. for each MLP of each feature, the first few layers are the same across features. The proposed method, called NBM, allows a strong reduction of the # of parameters without sacrificing the accuracy. It also has higher speed and scales to higher number of features. It also models the interaction term, called NB2M, by modeling each pairwise interaction term through another MLP. The performance of NBM is slightly better than NAM in several tabular and image datasets although consistently. 
strengths_and_weaknesses: # Strengths
- Clear writing.

# Weaknesses
- The datasets, IMO, are poorly chosen in this paper. Several image datasets are used, but no one would use GAM on pixels to claim any interpretability. Also, several original datasets in NAM are not compared. In contrast, some rare datasets like CoverType are used. It makes me wonder if there are dataset picking and the reported performance is not representative.

- Low originality: I believe the idea is not new as sharing basis functions is common, e.g. CNN v.s. MLP and multi-task learning. The basis sharing might work better but the hard sharing in NBM can incur negative transfer that lowers the performance, especially in tabular datasets when the features are very heterogenous. Therefore, I think it's more important the author shows more comprehensive evidences.

- Even with the selected datasets, the performance benefit does not seems to be big. 
questions: 1. **Dataset picking**: Why are there several datasets in NAM not compared? Also, image datasets are a very poor use of GAM - no one would want to visualize the GAM plot of a single pixel. I get that the shared basis functions of NBM should work better with images since each feature is homogenous, but again no one would use GAM on images. If the idea is to test NB2M in high-dimensional features, there are multiple high-dimensional tabular datasets used in the NodeGAM paper that can be compared to. Also, why do you mostly include multi-class datasets while NAM uses binary ones? 

2. Doesn't the NB2M still have the quadratic growth of the parameters? In Eq. (5), the parameter $b_{i,j,k}$ still grows quadratic.

3. IMHO, the feature selection method in NodeGAM by using attention and back-propogation is not complex, just a type of regular deep learning architecture.

4. It would be nice to show if NBM is better than NodeGAM since it's also a neural GAM and should have fewer #params and faster throughput than NAM already. It also supports multi-class classification unlike EB2M. But I understand it can be perpendicular to the paper's main point to improve from NAM.


[1] Chang, Chun-Hao, Rich Caruana, and Anna Goldenberg. "Node-gam: Neural generalized additive model for interpretable deep learning." arXiv preprint arXiv:2106.01613 (2021).
limitations: One limitation I can think of is that the hard sharing could lead to negative transfer among feature networks and deteriorates the performance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a Neural Basis Model (NBM) that utilizes basis decomposition of shape function for regression, binary classification and multi-class classification tasks in the context of achieving state-of-the-art accuracy, model size, and throughput. The central idea of the proposed approach is to dissociate the input feature and individually feed it to a set of shared bases that allows for efficient in inference and compact model size.

strengths_and_weaknesses: What's good:
1) The paper organization and presentation are clear mostly.
2) The idea of decomposing each feature’s shape function into a small set of basis functions seems novel and efficient.

To be improved:
1) Seems the main motivation of the paper is interpretability (as addressed in the title). If so, I would like to see a detailed analysis of the shape of the learned basis and a visualization of how these basis represent the inputs. 
2) The comparison approaches are related old I think, how does NBM perform compared with recent implicit functions?
3) The idea of basis learning is also related to the radial basis functions, I would recommend also including some related discussion.
questions: 
1) Have you try the proposed method on the ill-posed tasks? such as image pinpointing and radiance field reconstruction? How does it perform?
2) Is there any similarity between the learned basis and the basis extracted from PCA-like approaches?
3) Any further advantages of interpretable modeling apart from the visualization?
limitations: The paper lacks a limitations analysis.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new transparent model called NBM. NBM improves upon NAM, which learns a NN (neural network) for each feature and the final output is a learned weighted-sum of the outputs of the neural networks. Instead of training independent NNs for each feature like NAM, NBM trains a set of NNs as basis and used them across all features. 
NBM improves the scalability issue of NAM by reducing the number of independent NNs. NBM is more stable than NAM because of the use of shared NNs. 
Experimental results show NBM and its variant NB^2M can get better results than NAM and NA^2M.
strengths_and_weaknesses: Strengths:
- The proposed model achieved very good experimental results.
- The proposed model is easy to follow.

Weeknesses
- I had to switch back and forth multiple times while reading the paper. For example, the NAM model, an extension of GAM and the predecessor of the proposed NBM, is not mentioned in Section 3.1 Background but in Section 3.3 Dicussion after NBM is introduced. I feel it's more natural to introduce GAM, NAM, then NBM, which helps readers understand the contribution of this paper.
- The discussion regarding the limitation of NBM is quite limited and can be improved. See my questions below.
questions: - Do you do any data normalization before training NBM? I didn't remember any related discussions in the paper or supplimentary PDF. If no, when the features' magnitudes vary, does NBM essentially work the same way as NAM since each NN is supposed to work for a particular range of inputs? I believe it's worth to discuss this issue a little bit more.
- Why does NBM achieve better experimental results than NAM? NAM seems to be more powerful in its representative power. Is it the case where NAM achieves better training accurracy/loss but NBM gets better validation/test results? If that's case, does the shared NN work as some implicit regularizations? Will there be some regularizations that make NAM achieve similar results as NBM?
limitations: - It would be great if the authors can discuss more about what kind of high dimension data can/can't be handled by NBM.
review_title: null

Meta-review: metareview: The paper proposes an approach, Neural Basis Model (NBM), that can be seen as a new subfamily of Generalized Additive Models for interpretability. The proposed model is compared to alternatives, showing competitive performance while being computationally more efficient. The authors successfully addressed questions raised by reviewers. As also noted by the authors, a major limitation of the paper remains to be the requirement of the input features being interpretable. This would limit the applicability and utility of the proposed model, limiting the significance of the contribution.
confidence: Less certain
Decision: Accept