[Review] Sequential Dialogue Context Modeling for Spoken Language Understanding

## Details of the paper
Title : Sequential Dialogue Context Modeling for Spoken Language Understanding
Authors : Ankur Bapna, Gokhan Tur¨, Dilek Hakkani-Tur¨, Larry Heck
Published Year : 2017
Link : https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46224.pdf

## Introduction
They have explored novel approaches for modeling dialogue context in a recurrent neural network (RNN) based language understanding system, which is the Sequential Dialogue Encoder Network. That architecture is able to consider and embed the context to the nets from the dialogue history in chronological order.
Then they have evaluated the two models

- Memory Network
- Previous Utterance Encoder
- Sequential Dialogue Encoder Network

And they did experiments with a multi-domain dialogue dataset demonstrate that the proposed architecture results in reduced semantic frame error rates.

## Previous Work
Technically they have extended the previous model which is Hierarchical Recurrent Encoder Decoders (HRED) (Sordoni et al.,2015).
So in this section, let me brief their architecture as well.


### 1. Previous Utterance Encoder

#### Details of the paper
Title : Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Authors : Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
Published Year : 2014
Link : https://arxiv.org/pdf/1412.3555.pdf

#### Issues:
No, they have just compared the different types of RNN families in the tasks of polyphonic music modeling and speech signal modeling.

#### Model Architectures of LSTM and GRU
<img width="724" alt="Screen Shot 2018-06-12 at 16.41.41.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/f969efb1-2a42-b468-687a-9ed5109186a9.png">

**there is nothing new to mention here....**

### 2. Memory Network

#### Details of the paper
Title : End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding
Authors : Yun-Nung Chen, Dilek Hakkani-Tur, Gokhan Tur, Jianfeng Gao, and Li Deng
Published Year : 2016
Link : https://pdfs.semanticscholar.org/df07/45ce821007cb3122f00509cc18f2885fa8bd.pdf

#### Issues:
1. errors from previous turns may be propagated and then degrade the performance of the current turn.
2. knowledge mentioned in the long history may not be carried into the current turn.

#### Approach: end-to-end memory network model for multi-turn SLU
This paper addresses the above issues by proposing an architecture using end-to-end memory networks to model knowledge carryover in multi-turn conversations, where utterances encoded with intents and slots can be stored as embeddings in the memory and the decoding phase applies an attention model to leverage previously stored semantics for intent prediction and slot tagging simultaneously.

**Problem Setting**
<img width="483" alt="Screen Shot 2018-06-12 at 16.25.48.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/38d3bfb8-e3f0-50ab-7195-115ccba3849b.png">

**Model Architecture**
<img width="962" alt="Screen Shot 2018-06-12 at 16.26.14.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/9db1b3c7-4345-7f59-51d4-3501af34f5d9.png">
<img width="488" alt="Screen Shot 2018-06-12 at 16.26.46.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/c0e2fdb6-618c-fc92-9849-a0a726012cbb.png">
<img width="482" alt="Screen Shot 2018-06-12 at 16.26.56.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/82c2e1bd-60ef-566c-0b0c-61ff3191f0de.png">

#### Evaluation
They have used Cortana.

#### Result
<img width="1027" alt="Screen Shot 2018-06-12 at 16.27.39.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/a04a9f30-c42b-29c1-8ff6-abfe48904291.png">



### 3. Sequential Dialogue Encoder Network(HRED)
#### Details of the paper
Link: https://arxiv.org/pdf/1507.02221.pdf
Authors: Alessandro Sordonif, Yoshua Bengiof, Hossein Vahabig, Christina Liomah, Jakob G. Simonsenh, Jian-Yun Nief
Published Year: 2015

#### Issue: Achieving context awareness is challenging due to data sparsity
Search engines assist the users by presenting query suggestions. To preserve the original search intent, suggestions should be context-aware and account for the previous queries issued by the user

#### Approach: hierarchical recurrent encoder-decoder architecture
This model is able to account for the sequences of previous queries of arbitrary lengths.
Given a query in the session, the model encodes the information seen up to that position and tries to predict the following query.

**Model Architecture**
<img width="1030" alt="Screen Shot 2018-06-12 at 16.11.28.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/afee85a8-5fb6-8aa0-37bf-f8e7d78f8d76.png">

**Generated Suggestions from the net**
<img width="483" alt="Screen Shot 2018-06-12 at 16.16.19.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/a81a2991-41e2-93a5-695e-0a35ee278a62.png">


**So, let' go back to our original paper here!**

## Problem Setting
**Example Semantic Framework in reservation of the restaurant**
<img width="486" alt="Screen Shot 2018-06-12 at 15.56.21.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/31e1e514-b15f-a085-753f-35067f8851c0.png">

|variable|description|
|---|-------|
|u|utterance|
|S|slot     |
|D|domain   |
|I|intent   |
* Following IOB (in-out-begin) representation for slot values.

## Data Source of the dialogues
1. buying movie tickets
2. searching for a restaurant
3. reserving tables at a restaurant

**Sample DataSet**
<img width="853" alt="Screen Shot 2018-06-12 at 16.51.12.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/3367cd27-b6af-6621-0a5b-d80bb7d26d18.png">

**Slots in each domain**
<img width="722" alt="Screen Shot 2018-06-12 at 16.55.12.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/d589958f-d708-0ffc-57a8-25ea8d55feaf.png">

## Data Recombination
They have merged the dialogues collected from above format following procedures.
- Sample dialogues x and y from dataset1 and dataset2 respectively.
- Find the first user utterance labeled with a task intent in y. Let this be turn l.
- Randomly sample an insertion point in dialogue x. Let this be turn k.
- • The new recombined dialogue is
- 
```math
{(t_{x1, fx1), ...(t_{x_k}, f_{x_k}),(t_{y_l}, f_{y_l}), ...(t_{y_{ny}} , f_{y_{ny}})}.
```

## Tasks for experiments
We compare the domain classification, intent classification and slot-filling performances, and the overall frame error rates of the encoder-decoder, memory network and sequential dialogue encoder network on the dataset described above.

## Model Architectures
Please check the previous works above.
- Memory Network
- Previous Utterance Encoder
- Sequential Dialogue Encoder Network

On top of these models, they have used the RNN tagger, which was introduced by ([Hakkani-Tur et al. , 2016](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46224.pdf)).

**Stacked RNN Tagger**
<img width="857" alt="Screen Shot 2018-06-12 at 16.45.56.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/714e82a9-e9e6-ab60-0807-72c31ae95be9.png">
*layers details*
First layer: GRU
Second layer: LSTM


### [Notations]

$D_t = \\{u_1, ..., u_t\\}$
$u_k = \\{x_{1}^k, ..., x_{n^k}^k\\}$
$h_t = H(D_t)$ 

where:
$D_t =$ dialog (sequence of utterances $u_k$)
$u_k =$ utterance (sequence of tokens $x_{i}^{k}$)
$t =$ current time step
$n^k =$ total tokens in utterance $k$
$H =$ Encoder
$h_t =$ vector representation of the dialog context


### A. Context Encoder
Three architectures were used for experiments:

#### 1. Previous Utterance Encoder
Baseline architecture of the context encoder.

- Use the embeddings of the tokens from the previous system utterance $u_{t-1} = \\{x_{1}^{t-1}, ..., x_{n^{t-1}}^{t-1}\\}$:
$h_t = BiGRU_c(u_{t-1})$, $c =$ context

- Single BiRNN layer with GRU cells and 128 dimensions (64 for each direction).

#### 2. Memory Network

Same as [[Chen et al., 2016]](https://pdfs.semanticscholar.org/df07/45ce821007cb3122f00509cc18f2885fa8bd.pdf)

- All previous context utterances $\\{u_1, ..., u_{t-1}\\}$ are encoded into memory vectors $\\{m_1, ..., m_{t-1}\\}$:
$m_k = BiGRU_m(u_k)$, for $k = 0, ..., t-1$; $m =$ memory

    - BiGRU encoder with 128 dimensions (64 in each direction)

- The current utterance is encoded into a context vector $c$ using another BiGRU encoder:
$c = BiGRU_c(u_t)$, $c =$ context

- Use a matrix $M$ where rows are memory vectors $m_i$.

- Obtain the cosine similarity between each memory vector $m_i$ and the context vector $c$.

- Get the attention distribution over $M$, and use an attention weighted sum of $M$ to produce $h_t$:
$a = softmax(Mc)$
$h_t = a^TM$

#### 3. Sequential Dialogue Encoder Network

Improvement of the Memory Network architecture by adding a session encoder that temporally combines $c$ and $\\{m_1, ..., m_{t-1}\\}$

- Concatenate $m_k$ and $c$, and produce 128-dimensions context encodings $g_k$:
$g_k = sigmoid(FF(m_k, c))$, for $k = 0, ..., t-1$; $FF =$ Feed-forward layer

- Feed $g_k$ as token level inputs:
$h_t = BiGRU_s(\\{g_1, ..., g_{t-1}\\})$, $s =$ Session


### B. Tagger

Stacked BiRNN to jointly model:

- domain classification
- intent classification
- slot filling

#### 1st layer: GRU, 256 dimensions (128 for each direction)

$\mathbf{o^1} = BiGRU_1(u_t)$
$\mathbf{o^1}: \\{o_{1}^{1}, ..., o_{n^t}^{1}\\} \rightarrow$ token level outputs

#### 2nd layer: LSTM, 256 dimensions (128 for each direction)

$\mathbf{o^2}, s^2 = BiLSTM_2(\mathbf{o^1}, h_t)$
$\mathbf{o^2}: \\{o_{1}^{2}, ..., o_{n^t}^{2}\\} \rightarrow$ token level outputs

- $s^2$ is used as input to classification layers for domain and intent classification:
$p^{domain} = softmax(Us^2)$
$p^{intent} = sigmoid(Vs^2)$

- $\mathbf{o^2}$ outputs are used as input to a softmax layer that outputs the IOB slot labels:
$p_{i}^{slot} = softmax(So_{i}^{2})$, for $i = 0, ..., n^t$



### google duplex model
<img width="662" alt="Screen Shot 2018-06-12 at 15.51.12.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/98d241e1-b50a-1e29-0d79-5510183e922b.png">