Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Sparse Interaction Additive Networks (SIANs). On an array of tabular datasets, SIANs perform comparably to neural networks and offer more interpretability than neural networks. 
strengths_and_weaknesses: **Strength**. The paper is very well written. The goal of the paper -- scaling up interpretable additive models while making their training to be tractable is very well motivated.   

**Weakness**. In my opinion, this paper can be further improved by incorporating a discussion / empirical comparison between the interpretability of DNNs and SIANs. To my understanding, the main advantage of SIANs is not their data-fitting ability (DNNs are better in these perspectives) but their interpretability. However, there has been a thrust of research in the post-hoc interpretation of neural networks such as saliency maps. These DNN-based saliency maps (or other attributed features) give rise to meaningful interpretation while being very flexible: They can be applied to DNNs in a model-agnostic way. The applicability of SIANs, however, is currently confined to tabular datasets and MLPs. 
questions: 1. In Tables 2, 3, and 4, are DNN baselines here also the reference DNNs used in Feature Interaction Selection (FIS)?

2. Would it be possible to have a discussion / empirical comparison between the interpretability of SIANs and DNNs undergone posthoc feature-attribution? As a concrete example, the authors can train a standard DNN on the California Housing dataset and visualize key features crucial to its performance. Are these features meaningful as well? And do they give rise to the same interpretaion as by SIANs in Figure 3?  
limitations: In my opinion, the interpretability of SIANs comes at the cost of being only able to deal with tabular datasets. In fact, even for tabular datasets, it is unclear to me whether the interpretability of SIANs is indeed greater than equipping conventional DNNs with feature attribution methods -- I encourage the authors to provide such a comparison.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed Sparse Interaction Additive Networks (SIAN), which can effectively tackle the exponential number of high-order feature interactions and scale up to larger scale datasets. By leveraging heredity and interaction detection, SIAN achieves competitive performance across multiple datasets. The work also provides further insights into the generalization and capacity trade-off and a block sparse implementation of neural additive models, which achieve better training speed and memory efficiency of neural-based additive models.
strengths_and_weaknesses: #### Strength:

1. The paper is well organized and easy to follow.
2. Sufficient implementation details are provided for reproduction.
3. The experiment section consists of both quantitative results and several visualization analyses.

#### Weakness:

1. As shown in Table 2, SIAN-1 performs the best in Appliances Energy dataset, while SIAN-2/3 is better in Bike Sharing and SIAN-5 in the rest. So how to effectively choose the feature intersection order used in SIAN for better performance.
2. When we increase the intersection order of additive models, will it reduce the model's interpretability to some extent, which may degrade the value of performance improvement.
3. Letter case should remain consistent, e.g., sian-1/2/3/5 in Table2, but SIAN in the main text.
questions: How do the training speed and storage change along with the increasing of feature intersection orders in SIAN ?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper discusses how to generalize neural additive models to higher-order interactions and how to do them efficiently. This is done in three steps: 1) a general DNN is trained; 2) a feature interaction algorithm based on higher-order derivatives with a large norm is used to choose which features to focus on for learning neural additive models. These are then applied to an additive network to learn the final features. Experimental results demonstrate reasonable performance on regression and classification datasets
strengths_and_weaknesses: Strength:
- A sound solution for addressing the efficiency issues in selecting higher-order interactions for generalized additive models. 
- Extensive experiments on various classification and regression datasets. 
- The explanation of MIMIC III is particularly interesting. 
- Block sparse implementations massive improves training, and compression reduces memory cost.

Weaknesses:
- Relatively weak direct technical novelty as it mostly combines Archipelago and generalized additive models. 
questions: Q1: Table 2 is a bit confusing. Both regression and classification metrics and datasets are listed, and for one higher is better, for the other lower is better. It would be clearer to clearly indicate which is which, as the reader might have a hard time switching context between Section 4 and it. Also, it is not all datasets since MIMIC III is in Table 3.

Q2: It might be nice to show the visualization in the discussion via some "automated" process, as it would illustrate that these type of data analysis/visualization can be done automatically (what are the shared components in these post-processing steps that can be written into programs easily?). The results in the discussion section are nice, but it is also possible to produce these with some "manual" exploratory data analysis methods. 
limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes a scheme to augment a trained neural network (considering in particular the case of unstructured, tabular data) by extending generalized additive models to the multi-layer neural setting in an unusual manner by using higher-order derivatives from an initial deep neural network to select a sparse set of higher order feature interactions on which to fit their augmented network.

Reviewers considered the paper well written, easy to follow, considered the method sound and well-motivated, and praised experiments as thorough and detail as adequate for reproducibility. Q2gD wondered specifically how the interpretability of these models measures against post-hoc DNN interpretation methods; the authors responded with a new section in the appendix, causing Q2gD to raise their score. FZvZ had questions about the selection of the model order and how that might affect interpretability which were adequately addressed in rebuttal. wcBj points to "relatively weak direct technical novelty", to which the authors reasonably respond that their forward selection method for interaction terms stands apart from typical approaches that involve backward selection or pruning; several confusing aspects and an suggestion for an explanatory visualization comprises a new section in the Appendix.

The experimental results seem well chosen and are convincing, even if the proposed method SIAN is not uniformly the best method on all tasks considered, it is a strong contender overall and in several cases handily outperforms the DNN baseline. The selection procedure seems well motivated and clever, while the architecture of the resultant additive model seems like one particular choice in a sea of possibilities. I doubt this paper will be the last word on the matter. Nonetheless, this seems like a valuable contribution to the literature on applying DNNs to tabular data and the intersection of GAM techniques with deep learning. I thus recommend acceptance.
confidence: Certain
Decision: Accept