DEIM Forum 2020 P1-19(day1 p84)

Do Neural Models for Response Generation
Fully Exploit the Input Natural Language Text?
Lingfeng ZHANG† and Tetsuya SAKAI†† Department of Computer Science and Engineering, Waseda University
3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan
E-mail: †zhang-lingfeng@moegi.waseda.jp, ††tetsuyasakai@acm.org
Abstract The advent of deep neural networks has enabled us to build natural language response generation systems. However, due to the black-box nature of these networks, it is unclear how they utilize diﬀerent parts of the input text to generate
responses. To answer this question, previous work has tried perturbing input by reversing, shuﬄing and dropping words to
explore how models react to the perturbation. In the present study, we extend this approach by replacing original words in the
input text with randomly chosen ones. We test four RNN-based seq2seq models and one Transformer-based seq2seq modelon two open-domain dialogue datasets, and ﬁnd that context information can be exploited by models much better than word
order. Hence it is useful to diﬀerentiate context information and word order when designing perturbations.
Key words Seq2seq Model, Chat Bot, Natural Language Processing

1 Introduction
From RNN-based models to Transformer-based models, deep

The former work only tested three models with attention mechanism. Our work extends the experiment to ﬁve models with attention mechanism and bidirectional structureneural network has become more and more advanced and usable(2) Diﬀerentiate context information and word orderin natural language response generation tasks. To better understandWe notice that all perturbation types tested in the former work

models and evaluate their performance, a lot of researches have been

actually inﬂuence both context information and word order. It is notdone mainly in two ways. One way is comparing the performanceclear whether the performance diﬀerence is caused by context infor-of diﬀerent models. This can be seen in almost all model design

mation or by word order. Our work has designed a new perturbation

works [4, 5]. The other way is analyzing the interior structure oftype to separate the inﬂuence of context information and word ordermodels to ﬁnd out whether speciﬁc structures or mechanisms help

clearly.

improve model performance [1, 6]. However, little attention has(3) Generalizability

been paid to the information exploited by a model. Few work has

Sankar et al. [14] only tested models’ performance on its own

investigated what kind and how much of the input information has

dataset. Hence, it is not clear whether their models perform well

the model exploited.

on a diﬀerent dataset. In our study, we have tested models’ per-To answer the question above, our work（注 1） attempts to explore

formance not only on the datasets where they are trained but also

the performance changes on multiple response generation models

another irrelevant dataset so that we can investigate whether there

when input information is modiﬁed. The idea follows the work of

is a trade-oﬀ between exploits of input text and generalizability for

Sankar et al. [14]. Perturbations including word drop, word shuf-

models.

ﬂe and word reverse are manually introduced to input, and then the

The paper is structured as follows: Section 2 introduces some re-performance diﬀerence before and after perturbations will be com-lated work. Section 3 describes our experiment details from three

puted as models’ sensitivity. The performance diﬀerence also showsparts: dataset, perturbation type and model. Section 4 shares thethe models’ ability to exploit the information which is inﬂuenced byexperiment results and our ﬁndings. Section 5 provides conclusions

perturbations.

and discusses future work.

Compared with their work, we have extended the approach of
Sankar et al. [14] in three ways to better understand what type andamount of input text information have been exploited by models:(1) More models

2 Related WorkA recent experiment done by Khandelwal et al. [9] trained an
LSTM-based language model on Penn Treebank [11] and Wikitext2 [12] datasets to make prediction on the next word after one se-（注 1）：Code can be found at https://github.com/Lingfeng-Z/ParlAIquence. They tested how model performance changes with a de-creasing context size under various conditions including word func-

higher conditional probability P(y|X) of each word y in the response

tion, word order, word drop, etc. Diﬀerences in perplexity un-utterance is expected. Thus the conditional probability of the total

der speciﬁc conditions were recorded with decreasing context size.

response can be computed as:

This evaluates how much information that match speciﬁc conditions
P(y|X) =be learnt by the model. They observed that the LSTM-based lan-n∏P(yi |y < i, x1 , x2 , ..., xn )i=1

guage model has an eﬀective context size of about 200 tokens on
average and local word order only matters for the most recent 20

Higher conditional probability of the whole response means

tokens. However, their work is mainly about language modelinglower perplexity of models. Therefore, perplexity is used as theinstead of response generation. Since seq2seq models are in themetric to evaluate the performance of diﬀerent models in our exper-

encoder-decoder structure and both parts of a seq2seq model caniments.

inﬂuence the ﬁnal performance, other researchers have conducted

Perplexity = P(y|X)−1/i

experiments on each of them. The decoder part has been investi-

Manual perturbations are introduced to input and perplexity dif-

gated by Ford et al. [7]. They divided the response generation pro-ference before and after perturbations will be recorded. For exam-

cess into two steps: First, generating part of the words in responses

ple, after shuﬄing the input, if perplexity increase 1.0, we can assert

under manual restrictions. Second, generating the other words tothat the loss of information caused by shuﬄing lead to the perplex-

ﬁll the blanks and complete the sentences. The restrictions include

ity diﬀerence 1.0. In other words, our model can learn that kind of

POS (part-of-speech), word frequency, etc. A Transformer model

information and we can quantify its sensitivity by using perplexity

has been trained and they found that generating common words anddiﬀerence. Note that all models are trained normally and all pertur-function words in the ﬁrst step leads to a better model performance.

bations are added only in the test processes.

This suggests that Transformer can hardly learn word frequency and3.1 Dataset

syntactic information.

Experiments are conducted on two diﬀerent chit-chat datasets:

Compared with modifying the encoder, operations on decoder are

Dailydialog is an open-domain dataset [10] containing chit-chatmuch more indirect and complicated. Therefore, modifying the en-

dialogues with annotations for topic, emotion and utterance acts.

coder part may be more eﬀective and straightforward. The encoder

Table 1 shows the basic statistics of Dailydialog dataset.

part has been discussed by Sankar et al. [14]. Their work is inspiredby the above language model. They trained diﬀerent models on

Total Dialogues

several dialogue datasets and tested their performance under a lot of

Average Speaker Turns Per Dialogue

7.9

diﬀerent perturbations such as drop, shuﬄe and reverse. The perfor-

Average Tokens Per Dialogue

114.7

Average Tokens Per Utterance

14.6

mance was evaluated by the diﬀerence in perplexity before and after

13,118

Table 1 Basic Statistics of Dailydialog

perturbation. They found that seq2seq models can exploit quite little
information from input text. In addition, they found that attention

Persona Chat is a crowdsourced dataset [19] collected via Ama-

mechanism can increase the models’ sensitivity to long-term con-zon Mechanical Turk. In each dialogue, paired turkers are given

text information. Although they conducted the experiments at boththeir personas randomly before conversation and are required to bet-utterance level and word level, they have not done analysis in de-ter know each other through chatting. The statistics of Persona Chattail about why there is a diﬀerence in performance before and after

dataset can be found in Table 2.

introducing perturbations and what kind of information the models
exploited from the input text.
To address the above questions, we extend the approach of Sankar
et al. [14] by testing more models with popular structures and more

Total Dialogues

10,907

Average Speaker Turns Per Dialogue

14.9

Table 2 Basic Statistics of Persona Chattypes of well-designed perturbations, and attempt to investigate
what types of information in the input text and what amount of them

Selecting these two chit-chat datasets is not only because they areare exploited by the response generation models. In addition, howeasy to get but we also want to keep a coherence with the former

attention mechanism and birdirectional structure help models ex-

work. With the same datasets, the ﬁndings of diﬀerent researches

ploit input information can be clearly demonstrated.

can be compared and discussed together easily and generalized to

3 Experiment
Our work follows the experiment setup of Sankar et al. [14] andbased on the ParlAI [13] framework. Given an input X contain-

many other situations.
3.2 Perturbation Type
Here we list all perturbation types we use in our experiment andbrieﬂy introduce the details of perturbations.

ing utterances x1 , x2 , ..., xn , a response generation model is expected(1) Drop

to generate a reasonable utterance of response y. This means that a

Drop perturbation consists of three parts: Word Drop, Verb Dropand Noun Drop. Drop means that we will omit some part of the(2) Transformer-based Models

input. Specially, we drop 30% of all the words in the input in Word

We train a transformer model [18] with an embedding size andDrop, all verbs in Verb Drop and all nouns in Noun Drop. Drophidden state of 300, a batch size of 32, a two-head attention and no

perturbation makes the input lose part of its words, and it can test:

positional embedding. Limited by the GPU memory size, the batch(1) Whether some words hold higher importance than the others.

size of our transformer model on Dailydialog dataset has been set(2) Whether words with some speciﬁc POS can be understood by

to 3 and we raise the value of epoch to 40 instead of 20 to reach amodels better than the others.

closing training eﬀect.

(2) Shuﬄe & Reverse

All of our models have not been carefully ﬁne-tuned, as recent

This part of perturbation types includes Word Shuﬄe and Word

research [9] shows that hyperparameters will not inﬂuence models’Reverse. Word Shuﬄe randomly changes the positions of all words

sensitivity to input information. Since we are interested only in thein the input while Word Reverse feeds the input to models in de-perplexity diﬀerence before and after perturbation, we believe that

scent order. These two perturbation types can test if models canhyperparameter changes will not substantially aﬀect our experimen-

learn context information and order information from the input.

tal results.

(3) Replace
Replace perturbation consists of three variants, which are
10%Replace, 20%Replace and 30%Replace. It means that we randomly replace 10%, 20% and 30% of all the words in the input

4 Results and Discussion
The main experiment results are shown in Table 3. They suggest
that:with other words randomly picked from the dictionary. Being re-(1) Attention mechanism and bidirectional structure can help im-

placed by other words may cause a diﬀerent inﬂuence on context

prove the model performance. Comparing the Test PPL column

information and word order information. After replacement, theof each dataset in a chain of seq2seq lstm → seq2seq lstm att →context information of every word in the input has been changed

seq2seq bidir → seq2seq bidir att, the models’ perplexities with-

but the word order of most of the words in the input has not beenout perturbation decrease, meaning better performance on test sets.

changed. Meanwhile, the percentage of words been replaced will(2) Response generation models are still struggling in general-

increase 10% every two variants. As a result, we can evaluate how

izability. Results in the Generalization column show that all mod-much context information are exploited by the models.

els have a large increase in perplexity after switching to another(4) Generalization

irrelevant dialogue dataset. This indicates that none of them have

Generalization means testing the model performance on othergood generalizability. Among all models, transformer has the best

datasets. Here we choose Dialog bAbI dataset [3] and pick one

generalizability, which suggests that it learns more general fea-

batch of samples to replace the present test cases. In generalization,tures owned by all dialogue datasets and less unique features of theboth input and label are replaced with new pair of ones from Dialog

dataset where it is trained than the other models.

bAbI dataset. With this modiﬁcation, models’ generalizability can(3) All models learn a rather small amount of input text informa-

be tested. Higher generalizability means that the model can learntion, which is the same as the ﬁndings of Sankar el al. [14]. From

more commonly owned information from the tested dataset.

Word Drop column to 30%Replace column, all models have a small

3.3 Model

diﬀerence in perplexity after introducing perturbations. The biggest

Experiments have been conducted on diﬀerent models with someincrease in perplexity occurs on Dailydialog dataset when runningof the popular structures widely used nowadays.

seq2seq bidir model and perturbing with Word Reverse. Even in(1) RNN-based Models

that situation, the increase of perplexity is only 5.12, which is not a

We have trained a representative RNN-based model, which is alarge number for perplexity increment.

seq2seq encoder-decoder structure with LSTM units [17]. In ad-(4) Attention mechanism and bidirectional structure help im-

dition, the widely-used attention mechanism [2] and bi-directional

prove model’s sensitivity to context information. In the columns

LSTM structure [15] have also been added into the model. There-of all perturbations except Generalization, seq2seq bidir att has thefore, we have four models to test and compare the performance:

highest performance in nearly half of our tests. seq2seq lstm att andseq2seq lstm (standard seq2seq model), seq2seq lstm att (seq2seq

seq2seq bidir also performs comparably well in most cases. It may

model with attention mechanism), seq2seq bidir (seq2seq model

because attention mechanism can give weights to all words in thewith a bidirectional LSTM encoder), seq2seq bidir att (seq2seq

encoder and bidirectional structure can let a model get in touch withmodel with a bidirectional LSTM encoder and attention mecha-bidirectional context information so models with attention mecha-nism). The encoder and the decoder have two layers of LSTM. The

nism and bidirectional structure can exploit input information better

hidden state has been set to 256 and the dropout rate has been set tothan other models.

0.3 to avoid overﬁtting.

(5) Although traditional attention helps the models learn input
information, the same thing does not happen on self-attention. As

Models

Test PPL Word Drop Verb Drop Noun Drop Word Shuf Word Rev 10%Replace 20%Replace 30%Replace Generalization
Dailydialog(∆PPLavg )seq2seq lstm

33.00

+1.53

+0.73

+0.96

+2.79

+2.68

+0.60

+1.14

+1.71

+514.20

seq2seq lstm att

29.48

+1.61

+0.94

+2.07

+2.76

+3.69

+0.94

+1.98

+3.22

+535.82

seq2seq bidir

28.55

+2.39

+2.99

+2.45

+3.80

+5.12

+0.96

+1.97

+2.42

+656.15

seq2seq bidir att

28.36

+2.36

+3.69

+2.45

+2.98

+2.76

+1.02

+1.94

+3.11

+605.84

transformer

35.03

+0.56

+0.14

+1.26

+0.00

-0.03

+3.37

+3.83

+4.20

+297.07

Persona Chat(∆PPLavg )seq2seq lstm

43.34

+1.09

+0.69

+0.54

+0.82

+1.25

+0.18

+0.85

+1.31

+392.56

seq2seq lstm att

41.37

+2.30

+1.02

+1.24

+1.78

+2.19

+0.37

+1.17

+1.99

+434.53

seq2seq bidir

39.68

+1.94

+0.60

+0.17

+1.77

+2.56

+0.40

+0.94

+1.73

+435.42

seq2seq bidir att

39.35

+2.70

+1.28

+1.26

+2.02

+2.81

+0.31

+1.06

+1.92

+2902.65

transformer

40.04

+0.06

+-0.06

+0.02

-0.09

-0.03

+0.28

+0.50

+0.60

+171.06

Table 3 Diﬀerent models’ performance in multiple experiments. Column 2 shows the models’ average perplexity without perturbation over 3 times’ tests.
Other columns behind it show the average diﬀerence of perplexity over 3 times’ tests after doing the perturbations. The increases in perplexity aremarked as positive numbers while decrease ones are marked as negative. The best-performance values are marked as bold. It should be clear that
only to Test PPL and Generalization column, lower value is better while to all the other columns higher is better.

the two rows of transformer show, transformer performs poorly on

Perturbation

Model

10%Replace 20%Replace 30%Replace
Dailydialog

most of the tests, especially on Word Reverse and Word Shuﬄe,showing that transformer cannot learn word order information atall. It is caused by the structure of self-attention mechanism which

Word Shufis not a sequential structure like RNN. Therefore, positional information can hardly be learnt.
(6) Context information is exploited much more by models thanword order. From the results of n%Replace columns, we can noWord Rev

tice that the perplexity diﬀerence caused by introducing 30%Replace is near the ones caused by Word Shuﬄe and Word Reverse.

seq2seq lstm

21.5%

40.9%

61.3%

seq2seq lstm att

34.1%

71.7%

116.7%

seq2seq bidir

25.3%

51.8%

63.7%

seq2seq bidir att

34.2%

65.1%

104.3%

tranformer

N/A

N/A

N/A

seq2seq lstm

22.4%

42.6%

63.8%

seq2seq lstm att

25.5%

53.7%

87.3%

seq2seq bidir

18.8%

38.5%

47.3%

seq2seq bidir att

37.0%

70.3%

112.7%

N/A

N/A

N/A

tranformer

Speciﬁcally, transformer has the best score among all models in the

Persona Chat

n%Replace test of Dailydialog dataset, showing that:– The Transformer model can learn context information though
Word Shuf

hard to learn word order information.
– Context and word order are diﬀerent kinds of information andshould be discussed separately.
To analyze the diﬀerence clearly, we compute the ratio ofWord Rev

n%Replace’s performance to Word Shuﬄe/Reverse’s as Table 4shows.
The results in 30%Replace column of Table 4 show that to mostseq2seq lstm

22.0%

103.7%

160.0%

seq2seq lstm att

20.8%

65.7%

118.0%

seq2seq bidir

22.6%

53.1%

97.7%

seq2seq bidir att

15.3%

52.5%

95.0%

tranformer

N/A

N/A

N/A

seq2seq lstm

14.4%

68.0%

104.8%

seq2seq lstm att

16.9%

53.4%

90.9%

seq2seq bidir

15.6%

36.7%

67.6%

seq2seq bidir att

11.0%

37.7%

68.3%

tranformer

N/A

N/A

N/A

of the models, 30%Replace can have more than 65% performance

Table 4 The ratio of n%Replace’s perplexity diﬀerence to Word Shuf-of Word Shuﬄe and Word Reverse. The rest are also very close toﬂe/Reverse’s. The ﬁrst column lists the perturbation type com-the threshold we set. Empirically, perturbations like Word Shuﬄepared with n%Replace. The second column lists the model name.

and Word Reverse have changed both context information and word

Columns 3-5 list the ratios. The ﬁrst row shows n%Replace’s type.
All ratios are written in percentage and ”N/A” represents an inﬁn-order, so it is reasonable that the perplexity diﬀerence is caused by

ity number or a negative number which has no value in this exper-

both of them. However, the results in Table 4 provides evidence

iment. The ratios larger than 65% have been marked as bold tothat context information has played a much more important role inshow that context information is much better understood by mod-the ﬁnal perplexity diﬀerence than we expected, which suggests thatels than word order here.

context information can be better understood and exploited, and ismore important than word order.

much of them have been exploited by generative deep-neural net-

5 Conclusions and Future Workworks. We ﬁnd that:

This research studies what kinds of input information and how•

All RNN-based models and the Transformer model cannot ex-ploit the input information well.
•

Attention mechanism and bidirectional structure do help improve the model’s performance as well as make better use of
input information.

•

The Transformer model can learn more general informationfrom dialogues, leading to a better generalizability. In addition, there is a trade-oﬀ between exploits of information andgeneralizability to the models.

•Context information and word order information are diﬀerent.
In most cases, context information is better exploited by themodels.
Although our work has conducted experiments on several models

and perturbation types, further steps in testing more advanced models (BERT [5], hierarchical seq2seq model [16], skip-connection[8], etc.) and designing perturbation types which can more accurately evaluate context, word order or syntactic information stillshould be explored in future work.

References[1] Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav
Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207, 2016.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neuralmachine translation by jointly learning to align and translate. arXiv
preprint arXiv:1409.0473, 2014.
[3] Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning endto-end goal-oriented dialog. arXiv preprint arXiv:1605.07683, 2016.
[4] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime
Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformerxl: Attentive language models beyond a ﬁxed-length context. arXiv
preprint arXiv:1901.02860, 2019.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. Bert: Pre-training of deep bidirectional transformers forlanguage understanding. arXiv preprint arXiv:1810.04805, 2018.
[6] Tobias Domhan. How much attention do you need? a granular analysis of neural machine translation architectures. In Proceedings of the56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1799–1808, 2018.
[7] Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, andGeorge E Dahl. The importance of generation order in language
modeling. arXiv preprint arXiv:1808.07910, 2018.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep
residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pages 770–
778, 2016.
[9] Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp
nearby, fuzzy far away: How neural language models use context.
arXiv preprint arXiv:1805.04623, 2018.
[10] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi
Niu. Dailydialog: A manually labelled multi-turn dialogue dataset.
arXiv preprint arXiv:1710.03957, 2017.
[11] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.
Building a large annotated corpus of english: The penn treebank.
1993.
[12] Stephen Merity, Caiming Xiong, James Bradbury, and Richard
Socher.
Pointer sentinel mixture models.
arXiv preprint
arXiv:1609.07843, 2016.
[13] Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. Parlai: A dialog

research software platform. arXiv preprint arXiv:1705.06476, 2017.
[14] Chinnadhurai Sankar, Sandeep Subramanian, Christopher Pal, Sarath
Chandar, and Yoshua Bengio. Do neural dialog systems use theconversation history eﬀectively? an empirical study. arXiv preprint
arXiv:1906.01603, 2019.
[15] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neuralnetworks. IEEE Transactions on Signal Processing, 45(11):2673–
2681, 1997.
[16] Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conferenceon Information and Knowledge Management, pages 553–562. ACM,
2015.
[17] I Sutskever, O Vinyals, and QV Le. Sequence to sequence learningwith neural networks. Advances in NIPS, 2014.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion
Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing
systems, pages 5998–6008, 2017.
[19] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe
Kiela, and Jason Weston. Personalizing dialogue agents: I have adog, do you have pets too? arXiv preprint arXiv:1801.07243, 2018.