## 3 Model
### Multi-Agent Dialogue Model (MADM)
#### 3.2.2 User Simulator  
User simulator is composed of four parts: a simulator observation maintainer, a goal generator, a simulator policy, and an NLG as shown in Figure 1. The observation maintainer is applied to obtain the observation representation for user simulator. The goal generator is used to produce the user goal (e.g., slot value) and simulate the goal change during a dialogue. The simulator policy is applied to map the observation representation to a selected action for generating a user utterance. The NLG is applied to generate the next user utterance to dialogue manager. The four parts of user simulator are described in detail, respectively, as follows.  
**Observation maintainer:** the observation representation \(o_{t}^{s}\) is a concatenated vector composed of three parts: an embedding \(o(a_{t}^{m})\) for manager action \(a_{t}^{m}\), a binary variable \(b_{t}\) that indicates whether the slot value in manager action \(a_{t}^{m}\) is null, and an indicative vector \(v_{t}\) that denotes which type of slot value in confirm-action received from manager is different from user goal \(g_{t}\) in time step \(t\).  
**Goal generator:** the user goal is generated at the start of the dialogue by sampling the candidate slot values uniformly. As the initial goal may change in a real user dialogue, the variation of user goals are also simulated during the interaction. For each session, the user goals are sampled from the candidate slot values randomly at the beginning of the dialogue, meanwhile, an indicative vector \(c_{c}\), which counts the number of variations for each slot, is set to be a zeroes vector. This indicative vector \(c_{c}\) is used to limit the number of variations for each slot to avoid overly complex conversations. In each turn, a variation probability \(p_{v}\) is sampled from \([0,1]\) randomly, if this variation probability \(p_{v}\) is bigger than threshold probability \(p_{th}\), then a random slot is selected to change the corresponding value to another one from candidate slot values. Once a slot value is changed, the corresponding value of variation slot in indicative vector \(c_{c}\) is added 1. If the number of variations for some slots exceed the limitation number, those slots will not be changed, even though the variation probability \(p_{v}\) is bigger than threshold probability \(p_{th}\).  
**Simulator policy:** the observation representation \(\sigma_{t}^{s}\) is mapped to the selected action \(a_{t}^{s}\). To this end, a multi-layer perceptron (MLP) is applied to model simulator policy, which is computed as follows:  
\[\pi^{s}(a_{t}^{s}|\sigma_{t}^{s})=softmax(MLP(\sigma_{t}^{s})), \tag{11}\]  
where policy function \(\pi^{s}(a_{t}^{s}|\sigma_{t}^{s})\) is a probability distribution on the action space. The selected action \(a_{t}^{s}\) is drawn from the distribution \(\pi^{s}(a_{t}^{s}|\sigma_{t}^{s})\). For convenience, \(\pi^{s}(a_{t}^{s}|\sigma_{t}^{s};\theta^{s})\) is denoted as policy function of user simulator, where \(\theta^{s}\) are the parameters of the simulator policy.  
**NLG:** the selected action \(a_{t}^{s}\) is projected to next user utterance \(u_{t+1}\) for replying to dialogue manager. A template-based NLG is used to produce such user utterances. The responding template is drawn from a set of pre-defined templates according to the selected action \(a_{t}^{s}\). To assure the generalization and expressiveness, the templates are delexicalized by replacing concrete slot values with their slot names. For some slot dependent actions (e.g., _inform_()), the drawn template is lexicalized with the goal slot values to generate the final user utterance. An example of user utterance generation is shown in Figure 2, where B-loc, I-loc and O denote the slot labels of the beginning character of a location, inter character of a location and other characters, respectively.