## 3 Model
### Multi-Agent Dialogue Model (MADM)
#### 3.2.1 Dialogue Manager  
Dialogue manager consists of two parts: an observation encoder and a manager policy as shown in Figure 1. The observation encoder is employed to map historical dialogue sequence to observation representation. As some slot dependent actions (e.g., _confirm_()) need to combine with slot values from user utterances to make up an integral action, observation encoder also produces the slot values from user utterance through slot filling and intent recognition. The manager policy is applied to map the observation representation to a selected action for responding to user simulator. Observation encoder and manager policy are described in detail, respectively, as follows. **Observation encoder**: the historical dialogue sequence \(h_{t}=[a_{0}^{m},u_{1},...,a_{t-1}^{m},u_{t}]\) is encoded to an observation representation \(o_{t}^{m}\), meanwhile, the slot values information \(y^{t}\) and the intent recognition information \(z^{t}\) are output, where \(a_{t-1}^{m}\) denotes the selected action from manager in time step \(t-1\), \(u_{t}=[w_{t}^{1},w_{t}^{2},...,w_{t}^{1}]\) denotes the user utterance in time step \(t\), \(w_{t}^{i}\) denotes the \(i\)-th word (or \(i\)-th character in Chinese) in the user utterance \(u_{t}\), and \(\hat{y}_{t}=[\hat{y}_{t}^{1},\hat{y}_{t}^{2},...,\hat{y}_{t}^{1}]\) denotes the slot label information on user utterance \(u_{t}\). To this end, a hierarchical recurrent neural network (HRNN) is applied to model observation encoder. In the bottom layer of HRNN, a bidirectional LSTM [34] with attention pooling is employed to obtain the sentence representation \(e_{t}^{m}\) for user utterance \(u_{t}\), which is computed as follows:  
\[\overrightarrow{c}_{t}^{i} =LSTM_{fud}(\overrightarrow{c}_{t}^{i-1},e(w_{t}^{i})) \tag{1}\] \[\overleftarrow{c}_{t}^{i} =LSTM_{bnd}(\overleftarrow{c}_{t}^{i+1},e(w_{t}^{i}))\] (2) \[e_{t}^{i} =\sum_{i=1}^{I}\alpha_{i}[\overrightarrow{c}_{t}^{i}\oplus \overleftarrow{c}_{t}^{i}]\] (3) \[\alpha_{i}= \frac{\exp q_{t}^{i}}{\sum_{i=1}^{I}\exp q_{t}^{i}}\] (4) \[q_{t}^{i} =g([\overrightarrow{c}_{t}^{i}\oplus\overleftarrow{c}_{t}^{i}]), \tag{5}\]  
where \(\overrightarrow{c}_{t}^{i}\) and \(\overleftarrow{c}_{t}^{i}\) are the outputs of forward and backward LSTM in bottom layer of HRNN, respectively, \(e_{t}^{i}\) denotes the embedding of word \(w_{t}^{i}\), \(\oplus\) is the concatenation operator, \(\alpha_{i}\) is the attention  
Figure 1: The cooperative multi-agent dialogue model between dialogue manager and user simulator. weights, and \(g\) is a feed-forward neural network. The bidirectional LSTM also outputs the slot values information \(\hat{y}_{t}\) and the intent recognition information \(\hat{z}^{t}\), which is computed as follows:  
\[\hat{y}_{t}^{i} =\underset{l}{\arg\max}(softmax(\overrightarrow{c}_{t}^{i} \oplus\overleftarrow{c}_{t}^{i})) \tag{6}\] \[\hat{z}_{t} =\underset{k}{\arg\max}(softmax(\overrightarrow{c}_{t}^{i} \oplus\overleftarrow{c}_{t}^{0})), \tag{7}\]  
where \(l\) denotes the set of slot labels and \(k\) denotes the set of intent labels. In top layer of HRNN, a forward LSTM is applied to integrate the last observation representation \(o_{t-1}^{m}\), last manager action \(a_{t-1}^{m}\) and current sentence representation \(e_{t}^{m}\) into current observation representation \(o_{t}^{m}\), which is computed as follows:  
\[d_{t}^{m}= e_{t}^{m}\oplus o(a_{t-1}^{m}) \tag{8}\] \[\overrightarrow{c}_{t}^{i} =LSTM_{fud}(\overrightarrow{c}_{t}^{i-1},e(w_{t}^{i})) \tag{1}\]0  
where \(d_{t}^{m}\) is the concatenation of sentence representation \(e_{t}^{m}\) and last action representation \(o(a_{t-1}^{m})\), and \(o(a_{t-1}^{m})\) is a one-hot vector with the corresponding action position set to 1. **Manager policy:** the observation representation \(o_{t}^{m}\) is projected to the selected action \(a_{t}^{m}\).