Dialogue Act Classification in Reference Interview
Using Convolutional Neural Network
with Byte Pair Encoding
Seiya Kawano, Koichiro Yoshino, Yu Suzuki, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST), Japan
http://isw3.naist.jp/~seihttp://isw3.naist.jp/~seiya-ka/index.html

2018©Seiya KAWANO AHC-Lab, IS, NAIST

Background
Demand for conversation based seacrh

 Many users cannot clarify their “information-needs”
 Clarification of the requirement through interactions is important
 e.g. confirmation, asking for users “motivation” or “background”

How to model the dialogue strategy for clarification?
 Focus on the behaviour of human expert such as librarian
2018©Seiya KAWANO AHC-Lab, IS, NAIST

1

Reference Interview
Reference Service Information navigation service in library
 Library users can ask the s at the librarian
for helping to find information

Reference Interview
 Structured interview for clarify the information-needs
 librarian works with the user to clarify their ambiguous needs

Improves the accuracy of information navigation
[Ross et al. 2002]
2018©Seiya KAWANO AHC-Lab, IS, NAIST

2

Example of Reference Interview

question(ambiguous)Confirm: type of information

I'm having trouble finding the volumes
of postal mail throughout the 20th century

Hi, so you are trying to find some
measure of the volume of mail sent
through the US postal service

feedback: yes
Yes

confirm: search history
Ok, Let me see if I can find something
Can you tell me what you have done already?

answer
OK, let me look a little.... please hold
Do you think this page would help?
https://www.usps.com/cpim/pub100.htm

feedback: search history
I found one site, about a week ago,
but I just realized it's more recent data
and the paper is for '20th century' history,
so I want to try to focus on statistics

Feedback: positive
Wow, this is perfect!!
Thanks a lot

follow-up
So, will that answer your question?

Yes thanks.
Bye.

Utterances: Librarian
2018©Seiya KAWANO AHC-Lab, IS, NAIST

closing

Library User
3

Toward Modeling the Reference Interview
How do we model the dialogue strategy?
① Abstract of the utterance: e.g. dialogue act, dialogue state Tracking and predicting the speaker's intentions
② Modeling the dialogue strategy with reinforcement learning
 Understanding the dynamics of reference interview
 Imitate the librarian behaviors
③ Construct the response generation module
 Corresponds to each response action of librarian

 Toward the dialogue management
Focus the dialogue act classification task
2018©Seiya KAWANO AHC-Lab, IS, NAIST

4

Available Corpus of Reference Interview
QuestionPoint Transcripts [Radford et al. 2011]
 QuestionPoint: chat based reference service
 600 dialogue sessions, 12634 utterances (preprocessed)
 Personal information are anonymized

Dialogue act tag in reference interview [Inoue 2013]
 Defined the two intent levels dialogue act
• Dialogue Act Function; DAF (5 class)
• Dialogue Act Domain; DAD (19 class)

2018©Seiya KAWANO AHC-Lab, IS, NAIST

5

Dialogue Act in Reference Interview
Table. 5 Class DA Categories (DAF) [Inoue 2013]

2018©Seiya KAWANO AHC-Lab, IS, NAIST

6

Dialogue Act in Reference Interview
Table. 19 Class DA Categories (DAD) [Inoue 2013]

2018©Seiya KAWANO AHC-Lab, IS, NAIST

7

Problem of DA Classification
Supervised dialogue act (DA) classification

A problem existing DA classification approach
 Requires enough training data with labels
• Sparseness: Lack of training data for rare and unusual words
• It is critical in open-domain task such as reference interview

We need handle OOVs
2018©Seiya KAWANO AHC-Lab, IS, NAIST

8

Subword Approach
Words can be divided into Subwords
 Can reduce OOVs
• Character-ngram
• Byte Pair Encoding, etc.
[Gage et al. 1994, Sennrich et al. 2016]
Table. Variation of units for text tokenization

Units

Reduce OOVs

Consideration of word structure

Word××Characters

◎

×Character-ngram

〇

△Byte Pair Encoding◎〇

2018©Seiya KAWANO AHC-Lab, IS, NAIST

9

Byte Pair Encoding Compression
Bottom-up character merging

 Recursively merges most frequent consecutive symbols
into one symbol
 Starting point: character-level representation
 Hyper parameter: when to stop the merge operation
vocabulary size = number of merges + unique characters

 E.g. training data = {_low, _lowest, _newer, _wider}
1.
2.
3.
4.

Start = {_ l o w, _ l o w e s t, _ n e w e r, _ w i d e r }
_ l → _l
 Can segment the any text using_l o → _lo
merge operation rule_lo w → _low
• e.g. _lowly
e r → er
 Tokenized: _low｜l｜y2018©Seiya KAWANO AHC-Lab, IS, NAIST

10

Solution
Handle OOVs & Build an better vocabulary units
 Apply the Byte Pair Encoding for subword tokenization
• BPE is regarded as a domain-dependent feature extractor
Domain adaption of Subword tokenizer
Merge rule for tokenization
Target domain corpus
[BPE compression]
Parameter:
number of merge operation

Chunk a with b
Chunk b with c…

Adaptation to neural dialogue act classification model• Applied for a simple CNN-based classifier
2018©Seiya KAWANO AHC-Lab, IS, NAIST

11

Diagram of Proposed Method

Byte Pair EncodingCNN

2018©Seiya KAWANO AHC-Lab, IS, NAIST

12

CNN based DA Classifier
BPE-Unit-Level Convolutional Neural Network
 Embedding Layer
• Convert one-hot BPE vector to dense vector
 1D Convolution layer
 Global Max-Pooling Layer
• pool size = input length

Character vs. word vs. Subword (BPE-Unit)
• Comparison with a simple model
• Adapt the simple CNN

Figure. The Input Generation to CNN

2018©Seiya KAWANO AHC-Lab, IS, NAIST

13

Experimental Setup
Dataset: QuestionPoint transcripts
• Labeled 5,327 utterances, unlabeled 7, 307 utterances
labeled
Train Corpus

Tokenize

BPE
Tokenizer

Tokenized
Train Corpus
Training

Unlabeled
Corpus

Evaluation

Training

DA
Classifier

Domain adaptation

• Predict the 5 class & 19 class DA categories [Inoue 2013]
• 10-fold cross validation with paired t-test (5,327 utterances)
2018©Seiya KAWANO AHC-Lab, IS, NAIST

14

Comparison
DA classifiers
Method

Unit

BPE-Unit-Level CNN

BPE-Unit

Character-Level CNN

Character

Word-Level CNN

Word

Word-Level LSTM

WordMLP

Word

MLP w/o addition

WordRFWord

RF w/o additional

Word

Baseline features of RF & MLP
• Basic: bag-of-words (BoW), bag-of-bigrams (BoW)
• Additional : speaker, length of tokenized utterance, order of utterance
2018©Seiya KAWANO AHC-Lab, IS, NAIST

15

Statistics of BPE Tokenization
Number of characters per token Length of tokenized utterances

Vocabulary size of BPE

Average number of OOVs
 Word-Unit
• 334 OOVs

Vocabulary size of BPE

 Character & BPE-Unit
• Less OOVs (< 2)

2018©Seiya KAWANO AHC-Lab, IS, NAIST

16

Experimental Results
Method
BPE-Unit-Level CNN (v=200)
Character-Level CNN
Word-Level CNN
Word-Level LSTM
MLP
MLP w/o additional
RFRF w/o additional

DAF (5 class)
DAD (19 class)
0.8684 ***
0.7256 *
0.8538
0.7124
0.8438
0.6937
0.8286
0.6745
0.8515
0.7145
best of baselines
0.8498
0.7119
0.8367
0.7008
0.8292
0.6790

＊ p < 0.05＊＊ p < 0.01＊＊＊ p < 0.001：Comparison with MLP

BEP-Unit Level CNN ＞Character-Level CNN ＞＞Word-Level CNN
2018©Seiya KAWANO AHC-Lab, IS, NAIST

17

Results of Several BPE Settings
DAF (5 class) results
***********

**
Char-CNN (0.8538)
MLP (0.8515)
MLP (0.8498)
w/o additional
Word-CNN (0.8438)

＊ p < 0.05＊＊ p < 0.01＊＊＊ p < 0.001：Comparison with MLP
2018©Seiya KAWANO AHC-Lab, IS, NAIST

18

Results of Several BPE Settings
DAF (19 class) results
**
MLP (0.7145)
Char-CNN (0.7124)
MLP (0.7119)
w/o additional
Word-CNN (0.6937)

＊ p < 0.05＊＊ p < 0.01＊＊＊ p < 0.001：Comparison with MLP
2018©Seiya KAWANO AHC-Lab, IS, NAIST

19

Conclusion
We proposed dialogue act classification model in reference interview using CNN with Byte Pair Encoding
 Achieved the best performance without complicated feature
engineering and additional features

CNN with Character vs. word vs. Subword (BPE-Unit)
 BPE-Unit > Character >> Word
 BPE-Unit-Level CNN improved accuracy than Character-Level CNN
 Possibility of eliminating sparseness & acquiring the better unit

2018©Seiya KAWANO AHC-Lab, IS, NAIST

20

Future work
More Improve the dialogue act classification model
 Automatic parameter decision in BPE
 Combining some additional information (e.g. dialogue history)

Improve the current annotation scheme
 e.g. apply the ISO-24617-2 [Bunt et al. 2013, Yoshino et al. 2018]
 Define the dialogue state for the reference interview

Apply the reinforcement learning
 Understanding unknown reward structure in a reference interview
• E.g. Inverse reinforcement learning (IRL)
2018©Seiya KAWANO AHC-Lab, IS, NAIST

21

End Slide.

Example of BPE Tokenizatione.g.
we need simple explanations for the nervous and lymphatic system .

 Vocabulary size= 100
_we _n e ed ▁s i m p le ▁ e x p l an tion s ▁for ▁the ▁n er v o u s ▁and ▁l y m p h a t ic ▁s y st e m .

 Vocabulary size = 500
▁we ▁need ▁s i m p le ▁ex pl an tion s ▁for ▁the ▁n er v o us ▁and ▁ly m p h at ic
▁system .

 Vocabulary size = 1000
▁we ▁need ▁simple ▁ex pl an tion s ▁for ▁the ▁ne r v o us ▁and ▁ly m pha tic
▁system .
2018©Seiya KAWANO AHC-Lab, IS, NAIST

23

Input Generation to CNN
Text Tokenization: BPE or Character or Word

Set the optimal parameter
1. Max length of segmentation
2. Number of filters
3. Kernel size of convolution
4. Stride length of convolution

Figure. The Input Generation to CNN

2018©Seiya KAWANO AHC-Lab, IS, NAIST24