Understanding Text Classification Data and Models
Using Aggregated Input Salience
Sebastian Ebert, Alice Shoshana Jakobovits, Katja Filippova {eberts,jakobovits,katjaf}@google.com

Introduction
Goal● understand and ﬁnd general patterns that amodel learned from text data
● e.g., (mis-)predictions, shortcuts, etc.
● aimed at model developers

Approach● take a method that, given a model, explains a single
data example● aggregate the explanations over an entire dataset● aggregated explanations help gaining insights into themodel and data (not just a single example)

Model Developer's Questions● What patterns did the model learn fromthe data? (this poster)● What is the model sensitive to? (paper)● How can puzzling predictions be
explained? (paper)

MethodSalience Maps (Grad L2)charming

aand

movie

this‘

donotsamovie

stink

##s

silly

comedy

…

it

funny

watch

it,

3/

10

Analysis Methods: clustering, nearest neighbors, visualizations

Single Example Input Salience● gives an importance weight to every input token● gradient-based method, here: Grad L2 for BERT

Analysis Methods

Problems of Single Example Explanations● may lead the developer to discover false generalizations ("not"as indicator of contradiction in NLI)
● time-consuming to go through a large number of inputs to ﬁndpatterns● may be diﬃcult to spot unintuitive patterns from individual
examples even when the salience maps hints at its presence(a) Clustering(c) Visualizations(b) Nearest Neighbors

Solution: aggregation of ﬁxed-length representations

Fixed-Length Representations

…
…
…
…

…

…

…

Baseline Representations● vocabulary with token PMI weighting● average BERT word piece embedding● CLS encoding of BERT model
Salience Representations (ours)● vocabulary with salience weighting● sum of word piece embedding weighted by salience

Results
Question: What patterns did the model learn
from the data?
● IMDB: movie reviews● binary polarity classiﬁcation (negative,positive)

IMDB numeric clusters● t-SNE on ﬁxed-length representation
reveals interesting artifact● e.g., bottom center examples (red)contain "1"● --> many examples contain "1 / 10stars","3 out of 10","4 / 10"● this is a potential shortcut● ﬁnd numeric patterns with regexes (245/2500 examples)● cluster representation matrix● evaluate representations based on highest precision cluster(out of 5) (see word cloud of such a cluster)● prec: ratio of numeric examples within the cluster● recall: ratio of numeric examples in the cluster out of all
numeric examples

Conclusion

Toxicity REDIRECT Talk: cluster● Toxicity classiﬁcation● revealed pattern: "REDIRECT
Talk: {Article Title}"● contentless comments, containno information about toxicity● 1-1.5% of Toxicity data follows
the same pattern and is useless● We propose aggregated salience
representations for discovering
patterns that a model learned fromthe data.
● We can answer distinct, yetcommon model developerquestions.
○ What patterns did the model
learn from the data?
○ What is the model sensitive to?
○ How can puzzling predictions
be explained?

● The answers are a ﬁrst step for
improving the model.
● Use clustering and visualization on
your own models with LIT and theembedding projector.