Emb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models
Authors: Chandan Singh
Jianfeng Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Interpretability
Explainability
Additivity
Generalized additive model
Linearity

Abstract: Deep learning models have achieved impressive prediction performance but often sacrifice interpretability, a critical consideration in high-stakes domains such as healthcare or policymaking. In contrast, generalized additive models (GAMs) can maintain interpretability, but often suffer from poor prediction performance due to their inability to effectively capture feature interactions. In this work, we aim to bridge this gap by using pre-trained large-language models to extract embeddings for each input before learning a linear model in the embedding space. The final model (which we call Emb-GAM) is a transparent, linear function of its input features and feature interactions.
Leveraging the language model allows Emb-GAM to learn far fewer linear coefficients, model larger interactions, and generalize well to novel inputs (e.g. unseen ngrams in text). Across a variety of natural-language-processing datasets, Emb-GAM achieves strong prediction performance without sacrificing interpretability. All code for using Emb-GAM and reproducing our results is made available on github.

Under review as a conference paper at ICLR 2023
EMB-GAM: AN INTERPRETABLE AND EFFICIENT PRE- DICTOR USING PRE-TRAINED LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning models have achieved impressive prediction performance but often sacrifice interpretability and speed, critical considerations in high-stakes domains and compute-limited settings. In contrast, generalized additive models (GAMs) can maintain interpretability and speed but often suffer from poor prediction per- formance due their inability to effectively capture feature interactions. This work aims to bridge this gap by using pre- pre-trained neural language models to ex- tract embeddings from each input before aggregating them and learning a linear model in the embedding space. The final model (which we call Emb-GAM) is transparent, linear function of its input features and feature interactions. Leverag- ing the language model allows Emb-GAM to learn far fewer linear coefficients, model larger interactions, dramatically speed up inference, and generalize well to novel inputs (e.g. unseen ngrams in text). Across a variety of natural-languaae processing datasets, Emb-GAM achieves strong prediction performance without sacrificing interpretability speed. All code is made available on Github.
INTRODUCTION
Large neural language models (LLMs) have demonstrated impressive predictive performance due to their ability to learn complex, non-linear, relationships between variables. However, the inabil- ity of humans to understand these relationships has led LLMs to be characterized as black boxes, often limiting their use in high-stakes applications such as science (Angermueller et al.] 2016), medicine (Kornblith et al. 2022), and policy-making (Brennan & Oliver (013)). Moreover, the use of black-box models such as LLMs has come under increasing scrutiny settings where users re- quire explanations where models struggle with issues such as fairness (Dwork et al.) 2012) and regulatory pressure (Goodman & Flaxman 2016). Simultaneously. recent black-box models have grown to massive sizes, making them costly and difficult to deploy, particularly for edge devices such as mobile phones.
As alternative large black-box models, transparent models, such generalized additive mod- els (astie & Tibshirani 1986) and rule-based models (Breiman et al. 1984) can maintain inter- computationally efficient than black-box models. While transparent models can sometimes perform as well as black-bor models (e.g. Rudin al. (202T): Ha al. (22T)) Mignan & Broccardo (2019): Tan et al. 2022) ) in many settings such as natural Luuuralllnguuge processing (NLP), there is olten large gap in the perfor- mance between transparent models and black-box models.
pretability. Additionally, transparent models tend to be faster and
This work aims to minimize this gap by leveraging pre-trained LLM to learn more effective transparent model. Specifically, we extract LLM embeddings for different feature interactions (e.g. ngrams in text) and then learn a generalized additive model on top of these embeddings. The final learned model (which we call Emb-GAM) a transparent, linear function of its input features and feature interactions, but the use of the LLM allows Emb-GAM to intelligently reduce its number of learned parameters (see Fig M. Rather than learning a linear model over all possible feature interactions (which scales exponentially with the order of the interaction and the feature dimen- sion), Emb-GAM requires learning only a fixed set of linear coefficients (the size of the embedding extracted by the LLM).
As a result, Emb-GAM GAM can efficiently model high-order interactions, generalize well to novel inter- actions, and even vary the number of features used test-time for prediction. Moreover, inference

---

Under review as a conference paper at ICLR 2023
with Emb-GAM is extremely fast, requiring only looking up coefficients from a dictionary and then summing them. Experiments on a variety of NLP classification datasets show that Emb-GAM can achieve better generalization accuracy than transparent baseline methods. Moreover, learned Emb- GAM models are easily interpretable, both for individual predictions and at the level of an entire dataset, enabling cases in high-stakes settings. In what follows, Sec 2 covers the background and related work, See B explains the Emb-GAM pipeline, Sec f shows the results, and Sec S con- cludes with discussion.
(i) Extract ngrams not
(ii) Fixed-size embeddings
(iv) Linear model
(iii) Sum
2,
"not good movie"
good movie
2
negative
prediction
W
LLM
not good
Each ngram gets
good movie
2g
linear coefficient 2 w
Figure 1: Overview of Emb-GAM. A pretrained neural network is used to extract fixed-size em- beddings for ngrams in a given sequence. Then, these embeddings are summed and used to train supervised linear model. At test time, the model can be interpreted exactly as a generalized additive model, with a linear coefficient for each ngram in the input.
2
BACKGROUND AND RELATED WORK
GAMs There is a large literature on additive models being used for interpretable modeling. This includes generalized additive models (GAMs) (Hastie & Tibsshirani 1986) that have evolved to achieve strong performance by modeling individual component funetions/interactions using regular- ized boosted decision trees @Caruana et al. 2015) and more recently using neural networks (Agarwal et al. T02Tb. While GAMs have been successful a variety of domains, existing GAM methodol- ogy is limited in its ability model the high-order feature interactions that arise NLP.
Meanwhile, NLP has seen great success in models which build strong word-level representations. e.g. word2vec (Mikolov et al. 20T3a b GloVe (Pennington al. 2014), and ELMo (Peters al., 2018). By replacing such models with LLM embeddings, LAmb-GAM enables easily modeling ngrams of different lengths without training new model. Moreover, LLMs can incorporate formation about labels into the embeddings (e.g. by first finetuning an LLM on downstream prediction task).
Other transparent models Rule-based methods, such trees (Breiman et al. 1984 Lin al. 2020). regularized trees (Agarwal et al. 2022), rule sets (Friedman & Popescu 2008). lists (An- gelino al. 2017 Singh al. 2021). and tree sums (Tan et al.) (022) perform well for variety of tasks, but are often less effective in NLP tasks, where the number of required rules tends to grow with the size of the vocabulary.
An alternative line of work aims to make neural networks more interpretable. For example, models make predictions by comparing inputs prototypes (Li al., 2018 Chen et al. 2019) by predicting intermediate interpretable concepts (Koh et al. 2020). Alternatively, the entire model can be distilled into transparent model (c.g. adaptive wavelets (Ha et al. 202T). However, these works have been largely been restricted to computer vision.
Feature and feature-interaatioo importances Loosely related this work are post hoc methods that aim to help understand a black-box model, i.e. by providing feature importances using methods such as LIME Ribeiro et al. 2016). SHAP (Lundberg et al., 2019). and others (Friedman 2001 Devlin et al. 2019). However, these post-hoc methods lose some information by summarizing the model and suffer from issues with summarizing interactions (Rudin) 2018 Murdoch et al.) 2019). Slightly more related are works which aim explain feature interactions transformations in
2

---

Under review as a conference paper at ICLR 2023
neural networks @anizek et al., 2021; Singh et al. 2019 2020). but these works fail to explain the model as a whole and are again less reliable than having a fully transparent model.
METHODS: EMB-GAM PIPELINE DESCRIPTION
3.1 PRELIMINARIES: GAMs
Generalized additive models, GAMs (Hastie & Tibssirani] 1986) take the form:
g (E( 3 3 + fi (a1) f2 (122 ... fK (IK),
where x = (I1, T2, x is the input with K features. y is the target variable, g(.) is the link function (e-g., logistic function) and each f; univariate shape function with EE [fa] 0. Due to the function's additivity, each component function Si can be interpreted independently. Generalized linear models, such as logistic regression, are a special form of GAMs where each fa is restricted to be linear.
GAMs fail to effectively capture interactions in NLP In natural-language processing, GAMs usually take the form of a bag-of-words model, in which each feature æ; is a binary indicator (or count) of the presence of single token (e.g. the word good). However, this model fails to capture interactions between features (e.g. not good is different than the of not and good), which is crucial in NLP tasks.
The most common way deal with interactions including ngrams features (a bag-of-ngrams model); in this case, each feature is formed by concatenating tokens, c.g. not good would be 2-gram, also known as bigram. However, the number of ngrams in a dataset grows exponentially with n and the vocab-size. Even for a modest vocab-size of 10,000 tokens, the number of possible trigrams is already 10 This makes exceedingly difficult to learn accurate coefficients for linear models on ngrams in the training set and impossible to learn coefficients for ngrams not seen in the training set.
3.2 THE EMB-GAM PIPELINE
To remedy the issues with the bag-of-ngrams model, propose Emb-GAM, an intuitive model which leverages pre-trained language model ø to extract a better feature representation = Φ(x,) for each input ngram 24 (see Fig This allows learning only a single linear weight vector that has the same dimension 2i, regardless of the number of ngrams. The learned model is still GAM, ensuring that the model can cleanly interpreted as a linear function of its inputs:
(F())) B + w. Σ ((z))
Emb-GAM consists of four steps, each of which can be modified slightly:
(i) Extracting ngrams. To begin, user specifies procedure extract a of ngram features from an input text sequence. It is important that the extracted ngrams be semantically meaningful. For NLP, word-level tokenizer can extract meaningful unigrams from the text; all experiments here use the spaCy tokenizer (Honnibal & Montani 2017). The order of interaction to be used (i.e. the length of an ngram) can be pre-specified or selected via vooss-vaiiattin. Note that the longer the included interactions become, the less interpretable the resulting model will be. Domain knowledge can be used to improve the ngram extraction process: for example, common stopwords can be removed as features
iogguage-speciff parse trees be used extract out key ngrams.
(ii) Extracting embeddings. In the embedding step, each input (i.e. ngram) is fed through the model retrieve fixed-size embedding. If transformer returns a variable-length embedding (c.g. the embedding is the size of the sequence length), we average over its variable-length dimensior
A common alternative for hi-directional (masked) language models is to use the embedding for special token (i.e. [CI.S]), but aim keep the approach here more general.
3

---

Under review as a conference paper at ICLR 2023
For Cmb--AAM to work well, it is important that the pre-training task (e.g. next-word prediction) contains useful information about the interactions which are used in a downstream task (e.g. senti- ment classification).
(iii) Summing embeddings. In the summation step, the embeddings of each ngram in the input are summed to yield single fixed-size vector, ensuring additivity of the final model. While conduct a naive sum here, the sum could be adjusted with weights (e.g. weighting ngrams of different orders differently).
(iv) Fitting the final linear model to make predictions. Finally, we obtain a prediction by train- ing linear model on the summed embedding vector. Importantly, the fitted model Eq. 2 can still be exactly decomposed into an additive combination of terms: for each ngram TG, its linear contribution the final prediction is simply w ((x;). the classification experiments performed here, we choose the link function to be the logit function (or the softmax for multi-class) and also add l regularization over parameters w Eq.
Computational considerations Emb-GAM is inexpensive to fit as (i) the pre-trained language model is used only for inference and (ii) it only requires fitting a linear model to relatively few features. After training, the model can be converted to dictionary of linear coefficients for each ngram, making inference extremely fast. Making a prediction requires simply looking up the scalar coefficient for each ngram in sample, where the coefficient is the dot product between the ngram's embedding and the learned linear weight w, which cached during training. Nevertheless, it may still be useful to retain the embedding model test-time infer coefficients for previously unseen ngrams.
4 RESULTS: EMB-GAM PREDICTS WELL WHILE MAINTAINING INTERPRETABILITY
4.1 DATA OVERVIEW
In this work, we study four widely used NLP classification datasets spanning different domains including classifying the emotion of tweets (Saravia et al., 2018). the sentiment of financial news sentences (Malo et al. 2014), or the sentiment of movie reviews (Pang & Lee 2005 Socher et al., 2013) (see Table for an overview).
Table 1: Overview of datasets used here. The number of ngrams explodes with the size of the ngram.
Financial phrasehank
Rotten tomatoes
SST2
Emotion
Samples (train)
2,313
8,530
67.349
16,000
Samples (val)
1,140
1,066
872
2,000
Classes
3
2
Majority class fraction
0.62
0.5
0.56
0.34
Unigrams
7.169
16,631
13,887
15,165
Bigrams
28.481
93,921
72,501
106.201
Trigrams
39,597
147,426
108,800
201,404
Fraction of trigrams
0.91
0.93
0.13
0.89
appearing only once
Across datasets, the number of bigrams and trigams quickly explodes, making it difficult to fit an accurate bag-of-ngrams model. Morcover, many ngrams appear very rarely; for example, in the rotten tomatoes datasets, 93% of trigrams appear only once in the training dataset.
4

---

Under review as a conference paper at ICLR 2023
4.2 PREDICTION RESULTS
For each dataset, fit Emb-GAM along with two baselines: bag-of-ngrams model and the popular TF-IDF model (Jones 1972 In each case, model is fit via cross-validatio on the training set (to tune the amount of &g regulaniattion added) and its accuracy is evaluated on the validation set.
Generalization as function of ngram size Fig 2 shows the generalization accuracy of the dif- ferent models across different datasets as a function of the included ngram size. In this plot, the language model used to extract embeddings ø is BERT model (Devlin et al. 2018), finetuned each individual dataset Emb-GAM performs well compared to the baselimes, achieving a consid- erable increase in accuracy across three of the four datasets. Notably, Emb-GAM performance tends to increase as higher-order ngrams are added, whereas the baseline methods do not.
Financial phrasebank
Rotten tomatoes
0.821
0.80
Accuracy
Accuracy
0.78
0.76
0.74
ngram size
ngram size
Emb-GAM (BERT finetuned)
SST2
Emotion
Bag ngrams
TF-IDF
0.86
0.88
Accuracy
Accuracy
0.86
0.84
0.82
0.82
6
ngram size
ngram size
Figure 2: Generalization accuracy as function of ngram size. As the size of the included ngram increases, the gap between the BERT finetuned model and the original models grows. Averaged over three random cross cross-validation splits; error bars are standard errors of the mean (many are within the points).
Best generalization results Table shows the best results when choosing the order of ngrams via cross-validation (again using finetuned BERT model to extract embeddings for Emb-GAM), When comparing the interpretable baselines (GloVE (Pennington al. 2014 in addition to Bag of ngrams and TF-IDF), Emb-GAM shows considerable gains on three of the datasets (Financial phrasebank, Rotten tomatoes, and SS72) and only minor gain the tweet dataset (Emotion), likely because this dataset requires learning less high-order interactions. When restricting Emb-GAM to only use unigrams, performance longer improves interpretable baseline methods.
Thh TF-IDF ("Term frequency-invent document frequency") model an additive model ngrams which represents each ngram by its count rescaled by its frequency across the training dataset. Pre-trrineed language models retrieved from HuggingFace (Wolf al. 2019) (see Table A1 for details on models and downloadable checkpoints).
WWe use the pre-trained Glove emheddings trained on Common Crawl (840 billion tokens, 2.2 mi lilnoo vocab-size, cased, 300-dimensional vectors).
5

---

Under review as a conference paper at ICLR 2023
Table 2: Emb-GAM yields strong gains in validation accuracy for three datasets and is competitive for the two others. When restricted to only unigrams, the performance of Emb-GAM drops consid- erably.
Financial phrascbank
Rotten tomatoes
SST2
Emotion
Emb-GAM
92.8% % + 0.37%
81.6% + 0.05%
86.9% 1 0.1%
89.5% % 0.03%
Emb-GAM
Emb-GAM
86.4% + 0.13%
76.8% 0.19%
81.7% 0.07%
87.2% = 0.06%
(unigrams only)
Bag of ngrams
85.0% + 0.11%
75.0% +: 0.09%
82.8% 1: 0.0%
89.0% + 0.09%
Interpretable baselines
TF-IDF
84.9% + 0.16%
75.9% 1: 0.06%
83.4%
0.11% 1%
89.2% + 0.04%
GloVe
80.5% 0.06%
78.7% 0.03%
80.1%
0.1%
73.1% = 0.09%
Black-box baseline
BERT finetuned
98.0%
87.5%
92.4 %
93.6%
Comparing Emb-GAM performance with a black-box baseline In the studied data sets, the black-box baseline (a BERT finetuned model) outperforms Emb-GAM by 4%-6% accuracy. This is potentially reasonable tradeoff in settings where interpretability. memory, or speed are critical. In cases involving inference memory/speed, Emb-GAM can be converted a dictionary coefficients roughly the size of the number of ngrams that appeared in training (see Table I for trigram model, this yields roughly a 1,000-fold reduction in model size (compared to the ~ ~110 million trainable parameters in BERT), with much room for further size reduction (e.g. simply removing coefficients for trigrams that appear only once yields another 10-fold size reduction). Inference is nearly instantaneous, as it simply requires looking up coefficients in dictionary and then a single sum (and does not require GPU).
Using Emb-GAM together with black-box baseline In some situations, it may be useful to use Emb-GAM on some fraction of samples (for anerpreabbily/mennoryppeedd but relegate the remaining samples to black-box model. Here, we first predict each sample with Emb-GAM, then assess its confidence (how close its predicted probability for the top class to 1). is above pre-specified probability threshold, we use the Emb-GAM prediction. Otherwise, sample's prediction using a finetuned BERT model. Fig S shows the validation accuracy entire dataset
compute the
vary the percentage of samples predicted with Emb-GAM. Since Emb-G. GAM yields probabilities which are reasonably calibrated (see Fig A2),. rather than the accuracy linearly interpolating between Emb-GAM and BERT, large percentage of samples be predicted with Emb-GAM while incurring little to no drop in accuracy. For example, when using Emb-GAM on 50% of samples, the average drop in validation accuracy is only 0.0053.
Validation accuracy
(over entire dataset)
0.95
Financial phrasebank Emotion
0.90
SST2
0.85
Rotten tomatoes
10
20
30
40
50
60
70
80
90
100
Percentage of samples es predicted with Emb-GAM (%)
(Remaining samples predicted with Finetuned BERT)
Figure 3: Accuracy when using Emb-GAM in combination with BERT. A large percentage of sam- ples can be accurately predicted with Emb-GAM.
Varying Emb-GAM model settings Table 3 shows how generalization accuracy changes when the LLM used to extract embeddings varied between BERT, DistilBERT (Sanh et al. 2019) and RoBERTa (Liu et al.] 2019) Across the variations, embeddings from finetuned models yield siderably better results than embeddings from non-finetuned models. We also investigate using different layers and ngram selection techniques in Table A2

---

Under review as a conference paper at ICLR 2023
Table 3: Generalization accuracy varies depending the model used extract embeddings. Fine- tuning the embedding model improves Emb-GAM performance, and using DistilBERT lowers per- formance. Top two methods in each column are bolded.
Financial phrasebank
Rotten tomatoes
SST2
Emotion
BERT finetuned
92.8% % + 0.37%
81.6% + 0.05%
86.9% + 0.1%
89.5% + 0.03%
BERT
84.1% + 0.08%
78.1% + 0.16%
82.8% - 0.27%
67.1% 1 0.06%
DistilBETT finetuned
85.8% 1 0,34%
78.5% +:: 0.34%
81.7% 1: 0.07%
68.8% 1 0.11%
DistilBERT
81.7% + 0.34%
79.8% = 0.08%
86.8% + 0.1%
87.5% + 0.11%
RoBERTa finetuned
77.8% + 0.31%
83.6% + 0.03%
89.1% 1%1 +: 0.24%
88.5% + 0.19%
4.3 INTERPRETING A LEARNED MODEL
In this section, investigate how to interpret fitted Emb-GAM model, focusing the SS72 dataset, where the task is to classify whether a movie review's sentiment is positive or negative. We inspect Emb-GAM model fitted using unigram and bigram features extracted from the BERT finetuned model; this model achieves 84% validation accuracy.
Learned coefficients match human sentiment scores A trained Emb-GAM model can be inter- preted for a single prediction (i.e. getting score for each ngram in a single input, as in Fig | or for entire dataset (i.e. by inspecting its learned coefficients). Fig AA visualizes the learned Emb- GAM coefficients (i.e. the contribution to the prediction w" d(x:)) with the largest absolute values across the SST2 dataset. just the top ngrams. The learned coefficients interactions (c.g. not very is assigned to be very negative whereas without resorting is assigned to be very positive). Note that this form of model visualization easily allows a to audit and edit the model with prior knowledge, e.g. by altering a coefficient.
show a diversity of ngrams, we show every fifth ngram rather than
semantically reasonable and many contain strong
Fig 4 shows how well the learned Emb-GAM coefficients match human- sentiment phrase scores for umigrans/bigrams in SST (note: these continuous scores are separate from the binary sentence labels used for training in the SST2 dataset). Both are centered, that 0 is neutral senti- ment and positive/negative values correspond to positive/negative sentiment, respectively. There is a strong positive correlation between the coefficients and the labeled scores.
undi per Forera
mur taaly inscivv
atheratre drowns
wruullyy pratel plotees
unimatsty empty
1.0
renahs
ui ctr foauue
competing
los iideenyy
fai to
nether
nos very
mene
on
nut well
0.3
chumming soniin
optuun
Mepp tressaa
Bugunea smripw
0.0
-55
neturel by aertiiae the
-10
0.4
-15
Ropeconiibbuding nggaam warmed during training
Emb-GAM coefricient
(A)
(B)
Figure 4: Top and bottom contributing ngrams to an Emb-GAM model trained on SST2 are (A)
qualitatively semantically accurate and (B) match human-labeled phrase sentiment scores.
Inferred coefficients for unseen ngrams match human sentiment scores One strength of Emb- GAM is its ability infer linear coefficients for ngrams that were not seen during training. Whereas
7

---

Under review as a conference paper at ICLR 2023
baseline models generally assign each unknown ngram the same coefficient (e.g. 0), Emb-GAM can effectively assign these new ngrams accurate coefficients (as long as the new ngram consists of tokens in the vocabulary of the LLM being used).
As one example, Fig A shows that the Emb-GAM model trained only on bigrams in Fig Rl can automatically infer coefficients for trigrams (none of which were explicitly learned during training). The learned coefficients are semantically meaningful, even capturing three-way interactions, such as not very amasing. To show diversity of ngrams, we show every 20th ngram rather than just the top ngrams. Fig 5 shows the learned trigram coefficients compared to the human-labeled SST phrase sentiment for all trigrams in SST. Again, there is strong correlation, suggesting that the coefficients are semantically accurate.
mach
Intornet Cereettreoo ooe toe www.ooofft..o
nt good
SST phraze sentmmett tbbel
cont COOCETTL
sentiis Keszzuu en mum
trons 1/0 yeus
0.2
0.0
petter's
-0.4
Emb confiicttt
(A)
(B)
Figure 5: Inferred trigrams coefficients for Emb-GAM model trained only bigrams (A) quali- tatively semantically accurate and (B) match human-labelee phrase sentiment scores.
Test-time accuracy-interpretabllly tradeoff The ability to effectively generalize to unseen to- kens in Fig raises the question of whether one can vary the order of ngrams used at test-time, to get tradeoff between accuracy and interpretability (i.e. how many features are used to make a prediction). Fig suggests this may be feasible; it shows the accuracy of an Emb-GAM model (using finetuned BERT) fitted using 4-grams as the order of ngrams used only for testing is varied. As the number of features used for testing increases, the performance tends to increase but interpre- tations become more difficult. Depending on the relative importance of accuracy and interpretability context, one may select to use a different number of features for testing.
Fig 6 characterizes the full tradeoff between the number of ngrams used for fitting and for testing for SST2. Generally, the best performance is achieved when the same number of ngrams is used for training and testing (the diagonal). Performance tends to degrade significantly when fewer ngrams are used for testing than training (lower-left). Results for all datasets show similar patterns (see Fig
Comparison with post-hoc feature importance The coefficients learned by Emb-GAM often differ from importances assigned by post-hoc feature-importance methods. Emb-GAM learns a single coefficient for each ngram across the dataset, allowing for auditing/editing the model with visualizations such as Fig In contrast, popular methods for post-hoc feature importance, such as LIME (Ribeiro et al. 2016 and SHAP (Lundberg & Lee 2016) yield importance scores that vary based on the context in each input. This can be useful for debugging complex nonlinear models, but these scores (i) are approximations, (ii) must summarize nonlinear feature interactions, and (iii) vary across predictions, making transparent models preferable whenever possible.
Fig shows an example of the Emb-GAM coefficients for the SST2 model from Fig 4 for different ngrams when making prediction for the phrase not very good. While Emb-GAM yields scores for each subphrase that match human judgement (as seen in Fig and Fig BB) posthoc feature importance methods summarize the interactions between different ngrams into individual words, potentially making interpretation difficult. Scores are rescaled to be between - and to make them comparable. See Emb-GAM scores for many top-interacting phrases in Fig A3
8

---

Under review as a conference paper at ICLR 2023
SST2
Tgrem Cgram 5gram dgram 1yram 7pram 1pam
Retten tomeloes phrasebank
$1.0
84.2
83
65.5
83.6
83.33
83.6
Emotion
0.85
SST2
Fostures traiting
64.2
81.-
115
85.
86.2
25.5
05.4
0.75
Text
0.70
59.9
73.4
82:
85
86
86.8
86.9
0.65
0.60
0.55
62.5
72.8
8144
83.8
85.7
86.7
86.7
1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram Features used for testing
1gram
3grom 4gram Scram sgram 7gram Features
testing
(A)
(B)
Figure 6: Emb-GAM performance when varying the number of ngrams used for testing. (A) Per- formance for a model fitted using 4- g-grams. (B) Full breakdown when varying the ngrams used for training and testing.
Posiive
not very good
not
very
good
Emb-GAM
very
very good
not
very
good
SHAP
not
very
good
Neguöve
Comparing Emb-GAM ngram coefficients (left) word-level feature importances from posthoc methods (right): LIME and SHAP.
Figure 7:
DISCUSSION
Emb-GAM helps bridge the gap between black-box models and transparent models in NLP in terms of accuracy and speed. This potentially opens the door for introducing NLP in high-stakes domains, such as medical docision-making and on compute-limited hardware. While the work here focuses on NLP, the same pipeline could be applied to other domains wherever effective language models are available to extract meaningful embeddings (e.g. computer vision).
One limitation of this work that performance does not improve unless using an LLM finetuned on downstream dataset. Future work could train the entire Emb-GAM pipeline end-to-end, po- tentially improving the embedding representation. Such a representation could allow Emb-GAM to outperform even a black-box finetuned neural network, as the GAM inductive bias may help prevent overfitting in data-limited settings.
There are many potentially useful extensions of Emb-GAM. One notable extension would build on the nonlinearity present in GAMs such as the explainable boosting machine (Caruana et al. 2015). to nonlinearly transform the embedding for each ngram with model before summing obtain the final prediction.
Finally, there are many ways Emb-GAM can be improved and used in the real-world. When domain knowledge is available, more meaningful inputs can be used to extract ngrams or to make a sparser model. Additionally, compression techniques verting to a dictionary of coefficients to reduce memory requirements. Finally, Emb-GAM can also be used for tasks beyond classification, such sequence prediction outlier detection. We hope that the introduction of Emb-GAM can help push improved performance prediction into high-stakes applications in the real world and reduce unnecessary energy/compute usage.
be applied to the Emb-GAM model after

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhineet Agarwal. Yan Shuo Tan, Omer Ronen, Chandan Singh, and Yu. Hierarchical shrinkage: im- proving the accuracy and interpretabilityy of tree-based methods. arXiv:2202.006 008.0000008888.0 fcs, statl, 2022. URL arXiv: 2202.00858.
htto://arx.v.orr//aa/////2 wtt.:orr//abbaaoaoa/
30858
Rishabh Agarwal, Levi Melnick. Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geof- frey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699-4711, 2021.
Hanna Abi Akl, Dominique Mariko, and Hugues De Mazancourt. Yseop finsim-3 shared task 2021: Spe- cializing financial domain learning with phrase representations. arXiv preprint arXiv:2108. 09485, 2021.
Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. arXiv preprint arXiv:1704.0170) 2017.
Christof Angermueller, Tanel Pärnamaa, Leopold Parts, and Oliver Stegle. Deep learning for computational biology. Molecular systems biology, 12(7):878, 2016.
L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Wadsworth and Brooks, Monterey. CA. 1984. URL htttp:://ww. http://wwwww.o.tt.eee.... Classiticat ien- book/97804 0/01///77701224411 12484188
Classification and Regression Trees.
Tim Brennan and William L Oliver. The emergence machine learning techniques in criminology. Criminol- ogy & Public Policy, 12(3):51--62. 2013.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2 ACM SIGKDD international conference knowledge discovery and data mining, pp. 1721-1730. 2015.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 2018.
Summer Devlin, Chandan Singh, W James Murdoch, and Bin Yu. Disentangled attribution curves for interpret- ing random forests and boosted trees. arXiv preprint 1905 2016,
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through aware- Proceedings of the 3rd innovations theoretical computer science conference, pp. 214-226. ACM, 2012.
Friedman and B. E. Popescu. Predictive learning via rule ensembles. The Annals of Applied Statistics, (3):916-954, 2008. doi: 10. 10.1224/07-aoas 148.
Jerome H Friedman. Greedy function approximation: a gradient hoosting machine. Annals of statistics, 1189-1232, 2001.
Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a" right explanation". arXiv preprint arXiv: 0ww.116660.8811 2016.
Wooseok Ha. Chandan Singh, Francois Lanusse, Srigokul Upadhyayula, and Bin Yu. Adaptive wavelet distil- lation from neural networks through interpretations. Advances Neural Information Processing Systems, 34, 2021.
Trevor Hastic and Robert Tibshirani. Generalized additive models. Statistical Science, 1(3):297-3) 1986.
Ahmed Hazourli. Financialbert pretrained language model for financial text mining. 02 2022. doi: 10. 23140/RG.2.2.340033 12803.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Joseph D Janizek, Pascal Sturnfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. J. Mach. Learn. Res., 22:104-1 2021.
10

---

Under review as a conference paper at ICLR 2023
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. PMLR, 2020.
Aaron E. Kornblith, Chandan Singh, Gabriel Devlin, Newton Addo, Christian J. Streck, James Holmes, Nathan Kuppermann, Jacqueline Grupp-Phclan, Jeffrey Fineman, Atul J. Butte, and Bin Yu. Predictabil- ity and stability testing to assess clinical decision instrument performance for children after blunt torso trauma. PLOS Digital Health, 2022. doi: http:://oii http://doo..or/111113//////mmmm 13jo///urlll/ pdig 0000076. URL bttps:/ journals plws. erg/ httg/diiitaalinn wawaalhhhlaatt
http:e/tww.eeetttaaaaeeeeeeeeeee wttptt///rrrrceeeeeeeeee 1tt.::...1111oorroe pdig. CC00076
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through proto- types: A neural network that explains its predictions. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jimmy Lin. Chudi Zhong, Diane Hu, Cyathia Rudin, and Margo Seltzer. Generalized and scalable optimal sparse decision trees. International Conference on Machine Learning, pp. 6150-6160. PMLR, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1r0iv: 1692, 2019.
Scott Lundherg and Sn-In Lee. An unexpected unity among methods for interpreting model predictions. arXiv preprint 161 1.07477. 2016.
Scott M Lundherg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. Explainable ai for trees: From local explanations global understanding, arXiv preprint 1xXx:1905.006100 2019.
Malo, A. Sinha, P. Korhonen, Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Arnaud Mignan and Marco Broccardo. One neuron versus deep learning aftershock prediction. Nature, 574 (7776):E1-E3. 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3788 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013b.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05009, 2020.
W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of National Academy of Sciences of the United States of America, 116/44) 2019. doi: 10.1073/pmas. 19077///aa.90005411666 1900654116.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary VeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Fabian Pedregosa, Ga è Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-leamn Machine leamning python. the Journal of machine Learning research, 12(Oct):2825 2830, 2011. URL http: html
Jeffrey Pennington. Richard Socher, and Christopher D Manning. Glove: Global vectors for word representa- tion. In Procedings of the 2014 conference on empirical methods in natural language processing (EMNIP), pp. 1532-1543, 2014.
11

---

Under review as a conference paper at ICLR 2023
Mathhew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). pp. 2227 -2277, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.1866.530 1/1/5//1//1/// I/N11- 88 101188-1200 1202. URLpttps: Setetps://aclattiinnn /ww.aattctooooy.o.oooo....o org/N18 1202
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: tions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144. ACM, 2016.
Explaining the predic-
Cynthia Rudin. Please stop explaining black box models for high stakes decisions. arXiv preprint arXiv:811.11014, 2018.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong, Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint arXiv:103.112551, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilhert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01 108, 2019.
Elvis Saravia, Hsien- Chi Toby Liu, Ven-Hao Huang. Junlin Wu, and Yi-Shin Chen. Carer: Contextualized affect representations for emotion recognition. Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3687-3697, 2018.
Chandan Singh, James Murdoch, and Bin Yu. Hicrarchical interpretations for neural network predictions. International Conference on Learning Representations, pp. 26, 2019. URL htttps://openreeieek
fett/forum? htte/oorumiidddSe Sii.rummiddSSSerootttt..
Chandan Singh, Wooseok Ha, Francois Lanusse, Vanessa Bochm, Jia Liu, and Bin Yu. Transformation impor- tance with applications to cosmology, 2020.
Chandan Singh, Keyan Nasseri, Yan Shuo Tan, Tiffany Tang, and Bin Yu, imodels: a python package for fitting interpretable models. Journal of Open Source Software, 6(61):3192, 2021. doi: 10.21 105/joss.03192 URL
hittps:/ddo www/eo..//.... 20105/ 03192
Richard Socher, Alex Perelygin, Jcan Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Yan Shuo Tan, Chandan Singh, Keyan Nasseri, Abhineet Agarwal, and Bin Yu. Fast interpretable greedy- sums (figs). 19X1v:2201.1193// [cs, stat), 1 2022. URL http:://aar....oogaaa//////0///0 11931
arXiv: 2201.11931.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cis- tac, Tim Rault, Rémi Louf, Morgan Funtowicz, et Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377,, 2019.
12


Reviews:
[{"review_id": "u0nZoQkWK4", "reviewer": "Reviewer_2xXf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors identify that deep learning models (DLM)  give state of the art performance on NLP tasks, but are black box models which are not interpretable in themselves.  They propose using a generalized additive model (GAM) setup which uses the sum of ngram embeddings for text classification tasks to bridge the gap between DLM performance and GAM interpretability.  Their Emb-GAM model is interpretable and decomposable because a prediction is a linear combination of n-gram embeddings fed into a softmax or logit function.  They show the model’s accuracy performance compared with two interpretable baselines ( bag of words and TF-IDF ) on 5 datasets, along with ablations on what variant of BERT and BERT output to use and the effect of the number of n-gram features used by the model ( generally the higher the n, the higher task accuracy and the potentially less interpretable ).  They also show how to interpret model results by inspecting n-gram coefficients at a global level and how the model can infer the importance of unseen n-grams during training at test time.", "summary_of_the_review": "The idea is novel and there is need for interpretability in NLP, but the performance of their model compared with deep learning models which the authors claim to bridge is missing and needs to be a part of this work and discussion.\nSome ablations, experiments and analysis of the model could have been improved/added to strengthen the argument for what is an interesting and potentially compelling line of research.\n", "strength_and_weaknesses": "**Strengths:**  \nThe paper is well written and situates itself well against the existing literature,\nThe idea is novel and there is need for interpretability in NLP.\n\n**Weaknesses:**  \nHowever the performance of their model compared with deep learning models ( for which they aim to bridge the gap ) is not given ( ie, Emb-GAM is compared against simple, interpretable baselines, which is fine if one of the main claims of the paper is to bridge the gap in performance between GAMs and DNNs ).  How big is the gap?  A quick look shows for SST2,  it looks to be 8 to 10 points.  These results need to be included and discussed in the paper.\n\nAlso the ablation with respect to using DistilBERT or fine-tuning and the pooling strategy could have been relegated to an appendix in lieu of other experiments (or one of the Figures from Appendix Figure 1a).  For instance, DistilBERT is an optimized/pruned version of BERT whose advantage is that its smaller, parameter wise while getting near BERT performance, so using it will definitely not improve results and the speed gain is negligible in your model ( which uses BERT solely as an embedder ).  It would have been better to use DeBERTa or RoBERTa for comparisons.\n\nThe utility of the interpretability of the coefficients in the experiments could have been made stronger by a human study of the learned coefficient as opposed to showing global coefficient values for bi-grams / trim-grams and saying they seem qualitatively reasonable.  \n\nAlso the argument for the learning of interactions via ngram embeddings was a little unconvincing/unclear in that I’m not sure the use of 10 bigrams and their constituent unigrams was the best argument here.  Yes the Emb-GAM model’s bigram coefficient is not the simple sum of unigram coefficients its learned, but its not clear why/how that suggests the model has successfully learned interactions?  \nThe sum of the embedding for word1  and the embedding for word2 is not equal to the embedding of the combined word1 and word2  so it stands to reason a GAM will have different coefficients for all 3.   Why not compare against LIME or SHAP at the individual level since this model purports to handle interactions better and show it to humans for quality comparisons?  A better case needs to be made for how/when to use this method if the performance gap with deep learning models is high and post hoc feature attribution methods are available ( although I agree with all the possible limitations listed of using post-hoc methods )\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JmU6q22kzYM", "reviewer": "Reviewer_HyrP", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "# Summary\nEmb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models\n\n## What is the problem?\nDeep neural networks for NLP are high-performing, but uninterpretable. They are also computationally more intensive.\n\n## Why is it impactful?\nSome people argue that intepretability is essential to produce trust in models in high-stakes domains. However, it is important to note that this view has some detractors, https://hai.stanford.edu/news/should-ai-models-be-explainable-depends. A possibly more universal argument in favor of interpretability is that interpretability is valuable as a model validation, debugging, and de-biasing tool. \n\nI highlight these two different views as your method would, I believe, only help support one of them (the former), not the latter.\n\n## Why have existing approaches failed?\nThe authors only comment briefly on prior approaches for producing interpretable DNN-like models, but argue that they lose some information by summarizing the model, or fail to explain the model as a whole.\n\n## What is this paper's contribution?\nThe authors propose to use a DNN to produce feature embeddings of n-grams in text, then to use a GAM to provide a low-capacity, efficient, interpretable output prediction of the final task given these LLM contextualized embeddings. \n\n## How do they validate their contributions?\nThe authors demonstrate the superiority of their approach against a Bag-of-ngrams and TF-IDF approach on 4 NLP datasets, and examine the qualitative semantic alignment between identified high-importance n-grams and the underlying task.\n", "summary_of_the_review": " Summary of the Review\n  1. {Correctness}\n     (incorrected or not at all supported - well-supported and correct)\n     I'm scoring this a 2 here, as the results seem correct but the conclusions (that this model is an acceptable trade-off of interpretability, efficiency, and performance) don't seem well supported as interpretability is not quantitatively evaluated against any baselines and performance is not contrasted against underlying LLMs).\n  2. {Technical Novelty and Significance}\n     (neither significant nor novel - significant and do not exist in prior works)\n     I'm scoring this a 1, as I don't feel this is novel nor does it show sufficient technical significance.\n  3. {Empirical Novelty and Significance}\n     (neither significant nor novel - significant and do not exist in prior works)\n     Not Applicable\n  4. {Flag for Ethics Review}\n     No flag\n  5. Recommendation\n     (strong reject - strong accept)\n     I'm recommending strong reject here as I don't think the approach is well justified, either conceptually or via their provided experiments, and it also doesn't meet the technical sophistication/novelty bar for ICLR.\n\n## What would make me raise my score? (Things that you can do that would, pending their results and the manner in which you accomplish them, make me raise my score)\nI would need to see (a) additional experiments establishing that this method does significantly better than comparable, simpler approaches, such as those leveraging word2vec or glove embeddings as the source of contextual embeddings in raw performance, (b) that it performs comparably or better than (on some scale, preferablly some kind of human evaluation) at interpretability than existing interpretability methods, including post-hoc methods like LIME and attentional analysis from BERT, and (c) that it doesn't perform too much worse than just using BERT alone that it is still a viable solution.\n\nI suspect these experiments are too large in scope to fit within a revision, and would necessitate a full re-write. Additionally, even if those experiments were performed to my satisfication, I'm still not sure ICLR would be the right fit for this paper, and would probably recommend the authors look at a more focused NLP venue instead given the limited technical sophistication of the underlying methods here.\n", "strength_and_weaknesses": "# Strengths and Weaknesses\n\n## Key Strengths (reasons I would advocate this paper be accepted)\nNone\n\n## Key Weaknesses (reasons I would advocate this paper be rejected; unordered)\n  1. This approach is insufficiently technically sophisticated or novel. Ultimately, all that you are doing is producing n-gram embeddings using a LLM, then training a traditional, low-capacity model on those embeddings. This is not a new idea (especially as you produce non-contextualized word embeddings, this is equivalent in terms of technical sophistication to prior, static embedding approaches such as word2vec, elmo, etc.), and your experiments likewise do not produce dramatically novel findings to fit the ICLR venue.\n  \n  2. This approach does not seem well founded. You propose feeding _isolated_ n-grams through a large language model (like BERT) to produce n-gram embeddings. But, large language models like BERT are worthwhile over static n-gram embedding approaches precisely because they integrate longer-range context into their embeddings (thereby producing contextualized word embeddings). Under your system, you're explicitly producing _non-contextual_ embeddings using a _contextualized_ approach, which is slightly problematic. It certainly isn't likely to not work, but the significant expense of a system like BERT seems wasted in your setting and warrants more justification than you give it, both qualitatively and quantitatively. Furthermore, this also actually subverts your claim of interpretability -- While I understand that BERT embeddings are frozen prior to your final use via the GAM model, it still isn't the case that you could easily answer the question of why one n-gram is preferred over another given your interpretability analysis. With a bag-of-ngrams representation and linear coefficient based interpretability study, you can reason that one n-gram is preferred over another because it is more correlated with the output in question in your dataset; however, by using BERT to extract features, if I see from the GAM approach that \"very good\" and \"super great\" have wildly different final coefficients, I can't reason about why this is. Thus, it isn't clear how your approach is likely to solve the problem you identify; namely, producing interpretable, efficient predictions while leveraging the advantages of a DNN.\n  \n  3. You do not sufficiently justify your model empirically to motivate its use. In particular,\n   - You lack sufficient comparisons against baselines, as you only compare against bag of n-grams and TF-IDF. You don't look at nearest neighbor approaches, methods leveraging simpler static embeddings like  word2vec, or existing, post-hoc interpretability methods.\n   - You don't quantify the performance gap your interpretable method induces relative to high-capacity system. if BERT (as traditionally used) performs dramatically better on these tasks compared to your method, that is relevant to the overall impact of this work. \n   - You don't attempt to quantify the gain in interpretability your approach offers over existing interpretability methods. As interpretability is the primary goal of your work, this omission is a major lack.\n\n## Minor Strengths (things I like, but wouldn't sway me on their own)\n  1. I think interpretability for validation/debugging/debiasing purposes is very useful, and so work in that space is valuable. This is only a minor strength as I'm not sure your approach can meaningfully contribute towards the validation/debugging/debiasing goals given it proposes a fully new model.\n  2. The paper is, in general, well written.\n\n## Minor Weaknesses (things I dislike, but wouldn't sway me on their own)\nNone\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "B2Z8lrjTN0", "reviewer": "Reviewer_Ks5M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple and interpretable additive model called Emb-GAM by leveraging pretrained language model.", "summary_of_the_review": "Although this paper handles a very important task, but the idea and evaluation are both not enough. Besides, there are many limitations as mentioned in the paper. Therefore, I tend to reject this paper.", "strength_and_weaknesses": "Strengths\n- The paper is easy to read.\n\nWeaknesses\n- Only compare with two basic baselines.\n- Only evaluate on classification datasets.\n- The idea is not novel. The procedure of Emb-GAM is already common especially in cross-domain retrieval.\n- The length of ngram still need to be specified. And this matters. As higher-order ngrams are added, the computation also explodes.\n- It depends on language model. For each new dataset, finetuning is needed.\n- I have doubt about using a linear model to learn high-dimensional feature interaction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject