Can you stop the dog from…
barking

Interpreting
Language Models
with ContrastiveExplanations

Whynot?

Kayo Yin and Graham Neubig
1

EMNLP 2022

Interpreting Language Models withContrastive Explanations
Kayo Yin

Graham Neubig

2Why did the LM predict “barking”?
Input: Can you stop the dog from

Output: barking

3Why did the LM predict “barking”?
Input: Can you stop the dog from

Output: barking

4Language modeling is complex
Input: Can you stop the dog from

Output: barking

POS?

Verbinﬂection?

Negativeaction?

Actionverb?

Canineaction?
5

Language modeling is complex
Input: Can you stop the dog from

Output: barking…
barkerbarkhan
Barkhausenbarking
BarklaBarkley
barklouse…

6

Contrastive explanations are more intuitivetarget

Can you stop the dog from…
barking

Whynotfoil?

7

Why did the LM predict “barking” instead of “crying”?
Input: Can you stop the dog from

Output: barking

8Why did the LM predict “barking” instead of “crying”?
Input: Can you stop the dog from

Output: barking

9Why did the LM predict “barking” instead of “walking”?
Input: Can you stop the dog from

Output: barking10

Why did the LM predict “barking” instead of “walking”?
Input: Can you stop the dog from

Output: barking

11

Contrastive explanations for language models●

Gradient●

Input x gradient12

Contrastive explanations for language models●

Gradient●

Input x gradient13

Contrastive explanations for language models●

Gradient●

Input x gradient●Contrastive gradient14

Contrastive explanations for language models●

Gradient●

Input x gradient●Contrastive gradient●Contrastive input x gradient15

Contrastive explanations for language models●

Gradient●

Input x gradient●Contrastive gradient●Contrastive input x gradient16

Contrastive explanations for language models●

Contrastive input x gradient●Contrastive gradient norm●

Contrastive erasure

17

Contrastive explanations for language models●

Contrastive input x gradient●Contrastive gradient norm●

Contrastive erasure

18

Contrastive explanations for language models●

Contrastive input x gradient●Contrastive gradient norm●

Contrastive erasure

19

Contrastive explanations for language models●

Contrastive input x gradient●Contrastive gradient norm●

Contrastive erasure

20

Contrastive explanations for language models●

Contrastive input x gradient●Contrastive gradient norm●

Contrastive erasure

21

Result 1:
Contrastive explanations can better identify
linguistically appropriate evidence
for LM decisions

22

Deﬁning evidence for grammatical acceptability
BLiMP dataset (Warstadt et al., 2020): minimal pairs of grammatical acceptability

Many teenagers were helping themselves.
Many teenagers were helping themself.
Anaphor number agreement23

Deﬁning evidence for grammatical acceptability●

Anaphor agreement: extract input tokens coreferent with the target

Many teenagers were helping themselves.
Many teenagers were helping themself.
24

Deﬁning evidence for grammatical acceptability●●

Anaphor agreement: extract input tokens coreferent with the target
Argument structure: extract the main verb

Amanda was respected by some waitresses.
Amanda was respected by some picture.
25

Deﬁning evidence for grammatical acceptability●●●

Anaphor agreement: extract input tokens coreferent with the target
Argument structure: extract the main verb
Determiner-noun agreement: extract the determiner of the target noun

Phillip was lifting this mouse.
Phillip was lifting this mice.
26

Deﬁning evidence for grammatical acceptability●●●●

Anaphor agreement: extract input tokens coreferent with the target
Argument structure: extract the main verb
Determiner-noun agreement: extract the determiner of the target noun
NPI licensing: extract the NPI

Even these trucks have often slowed.
Even these trucks have ever slowed.
27

Deﬁning evidence for grammatical acceptability
●
●
●
●
●

Anaphor agreement: extract input tokens coreferent with the targetArgument structure: extract the main verb
Determiner-noun agreement: extract the determiner of the target noun
NPI licensing: extract the NPI
Subject-verb agreement: extract subject of the target verb

A sketch of lights doesn’t appear.
A sketch of lights don’t appear.
28

Quantifying alignment between linguistic rules andexplanations
Dot product00001

10-.5.1.3.1

29

Quantifying alignment between linguistic rules andexplanations
Dot product00001

10-.5.1.3.1

More alignment metrics in paper:
Probes needed
Mean reciprocal rank
30

Contrastive explanations can better identify linguisticallyappropriate evidence for LM decisions
Non-contrastive
Contrastive

Contrastive explanations have better alignment thannon-contrastive explanations
31

Contrastive explanations can better identify linguisticallyappropriate evidence for LM decisions

Contrastive explanations have better alignment thannon-contrastive explanations, especially when-> the model predicts correctly
32

Contrastive explanations can better identify linguisticallyappropriate evidence for LM decisions

Contrastive explanations have better alignment thannon-contrastive explanations, especially when-> the model predicts correctly-> the target and appropriate evidence are distant33

Result 2:
Contrastive explanations improvemodel simulatability

34

Model simulatability test

35Model simulatability test●

10 participants○

ML graduate students

36Model simulatability test●

10 participants○●

ML graduate students

20 pairs of highly confusable
words○

Corpus-driven confusion metric

37

Model simulatability test●

10 participants○●

20 pairs of highly confusablewords○●ML graduate students

Corpus-driven confusion metric

Balanced data38

Contrastive explanations improve model simulatability

39

Contrastive explanations improve model simulatability

40

Result 3:
Contrastive explanations help us characterize
how LMs make decisions

41

What context do LMs use for certain decisions?

Hypothesis: linguistically similar decisions
have similar contrastive explanations

42

Clustering foils by explanation
Input: General relativity predicts the existence of
Output: black
Englishredyellow
Asiandog
43

Clustering foils by explanation
Input: General relativity predicts the existence of
Output: black

EnglishAsianredyellow
dog

44

Clustering foils by explanation
Input: General relativity predicts the existence of
Output: black

redyellow
color

45

Clustering foils by explanation
Input: General relativity predicts the existence of
Output: black

redmeanyellow
color

46

Scaling up the cluster analysis●

Target: 10 most frequent words for each POS

47

Scaling up the cluster analysis●●

Target: 10 most frequent words for each POS
Foil: 10,000 most frequent vocabulary items

48

Scaling up the cluster analysis●●●

Target: 10 most frequent words for each POS
Foil: 10,000 most frequent vocabulary items
Input sentence: 500 randomly selected49

Foils related to a linguistic phenomenon are clustered
together50

Foils related to a linguistic phenomenon are clustered
together51

Foils related to a linguistic phenomenon are clustered
together52

Foils related to a linguistic phenomenon are clustered
together53

Contrastive explanations help us characterize how LMs
make decisions

See paper for aggregate analysis oflinguistic distinctions and results

54

Contrastive explanations help us characterize how LMs
make decisions●55

Bonus: interpreting NMT models with contrastive
explanations
Why did the model predict “carnet”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau

56Bonus: interpreting NMT models with contrastive
explanations
Why did the model predict “carnet”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau

57Bonus: interpreting NMT models with contrastiveexplanations
Why did the model predict “carnet”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau

Why did the model predict “carnet” instead of “ordinateur”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau
58Bonus: interpreting NMT models with contrastive
explanations
Why did the model predict “carnet”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau

Why did the model predict “carnet” instead of “ordinateur”?
En: I like my old notebook better than my new notebook.
Fr: J’aime mieux mon ancien carnet que mon nouveau
59Summary
Contrastive explanations…
1. can better identify linguisticallyappropriate evidence
2. improve model simulatability
3. help us characterize how LMs makedecisions

60