Title: Revisiting Feature Interaction Selection in Neural Additive Models
Primary area: learning theory
Keywords: additive models, feature interactions, deep learning theory, staircase phenomenon

Abstract: In this work, we revisit the paradigm of feature interaction selection for additive models.
This paradigm generalizes the selection of a model's input features to the selection of a model's feature interactions by equipping any model with the additive structure of a generalized additive model.
When applied to neural networks,
this restricts the network's learned representations to interactions between the specified sets of features.
In the study of the training dynamics of these neural additive models,
we discover a new phenomenon which we call `medium dimensionality',
corresponding to a balance between data complexity and model complexity.
We find that this phenomenon helps to explain the good performance of additive models on tabular datasets.
We moreover find that the tool of additive models is able to 
unify insights for many of the recently explored phenomenon of deep learning theory:
double descent, grokking, leap dynamics, and the staircase property.
Finally, we present developments on selections algorithms and neural additive models, benchmarking performance across a suite of tabular datasets.