## 3 Model
### Cooperative Training  
**Policy gradient:** the policy gradient is applied to compute an estimate of the gradient of policy parameters in order to maximize the long-term discounted reward. In a cooperative dialogue, the gradient of manager policy and simulator policy are denoted as follows:  
\[\
abla_{\theta^{m}}J(\theta^{m})=\mathbb{E}_{\pi^{m},\pi^{s}}[A^{ m}(a^{m},o^{m})\
abla_{\theta^{m}}\log\pi^{m}(a^{m}|o^{m})] \tag{12}\] \[\
abla_{\theta^{s}}J(\theta^{s})=\mathbb{E}_{\pi^{m},\pi^{s}}[A^ {s}(a^{s},o^{s})\
abla_{\theta^{s}}\log\pi^{s}(a^{s}|o^{s})], \tag{13}\]  
where \(A^{m}(a^{m},o^{m})\) is the advantage function of manager, and \(A^{s}(a^{s},o^{s})\) is the advantage function of simulator. REINFORCE with a baseline algorithm [35] is applied to estimate the advantage functions. Thus, the advantage function \(A^{m}(a^{m},o^{m})\) and the advantage function \(A^{m}(a^{m},o^{m})\) are computed as follows:  
\[A^{m}(a_{t}^{m},o_{t}^{m})=\sum_{j=0}^{J}\gamma^{j}r_{t+j}-V^{ \pi^{m}}(o_{t}^{m};\phi^{m}) \tag{14}\] \[A^{s}(a_{t}^{s},o_{t}^{s})=\sum_{j=0}^{J}\gamma^{j}r_{t+j}-V^{ \pi^{s}}(o_{t}^{s};\phi^{s}), \tag{15}\]  
Figure 2: An example of user utterance generation. where \(V^{\pi^{m}}(o_{t}^{m};\phi^{m})\) is the value function of manager with parameters \(\phi^{m}\) to estimate the return on observation \(o_{t}^{m}\), and \(V^{\pi^{s}}(o_{t}^{s};\phi^{s})\) is the value function of simulator with parameters \(\phi^{s}\) to estimate the return on observation \(o_{t}^{s}\). The loss functions of \(V^{\pi^{m}}(o_{t}^{m};\phi^{m})\) and \(V^{\pi^{s}}(o_{t}^{s};\phi^{s})\) are computed as follows:  
\[J(\phi^{m})=\frac{1}{2}[A^{m}(a_{t}^{m},o_{t}^{m})]^{2} \tag{16}\] \[J(\phi^{s})=\frac{1}{2}[A^{s}(a_{t}^{s},o_{t}^{s})]^{2}. \tag{17}\]  
The value function \(V^{\pi^{m}}(o_{t}^{m};\phi^{m})\) and policy function \(\pi^{m}(a_{t}^{m}|o_{t}^{m};\phi^{m})\) share the same parameters, meanwhile, the slot filling and intent recognition are optimized in a supervised manner jointly. To this end, the total loss function of dialogue manager is computed as follows:  
\[J_{r}(\theta^{m})=-A^{m}(a^{m},o^{m})\
abla_{\theta^{m}}\log \pi^{m}(a^{m}|o^{m})+\frac{1}{2}[A^{m}(a^{m},o^{m})]^{2} \tag{18}\] \[J_{s}(\theta^{m})=\sum_{t=1}^{T}\sum_{i=1}^{I}g_{t}^{i}\log y_{t }^{i}+\sum_{t=1}^{T}\hat{z}_{t}\log z_{t}\] (19) \[J_{w}(\theta^{m})=(1-\lambda)J_{r}(\theta^{m})+\lambda J_{s}( \theta^{m}), \tag{20}\]  
where \(\lambda\in(0,1]\) is a balance coefficient. Similar to dialogue manager, the value function \(V^{\pi^{s}}(o_{t}^{s};\phi^{s})\) and policy function \(\pi^{s}(a_{t}^{s}|o_{t}^{s};\theta^{s})\) share the same parameters in user simulator. The total loss function of user simulator is computed as follows:  
\[\
abla_{\theta^{s}}J(\theta^{s})=\mathbb{E}_{\pi^{m},\pi^{s}}[A^ {s}(a^{s},o^{s})\
abla_{\theta^{s}}\log\pi^{s}(a^{s}|o^{s})], \tag{13}\]0  
The two total-loss functions are optimized cooperatively after a complete dialogue. In this way, the dialogue manager and the user simulator are optimized cooperatively and simultaneously. The alternate training method was tried to optimize dialogue manager and user simulator, and empirical results show that alternate training method (every 10 training steps alternately) has slower convergence than joint training method and achieves the same performance with training jointly. Above all, the dialogue manager and the user simulator are optimized cooperatively in a one-to-one manner. To improve the dialogue manager generalization performance, this one-to-one cooperation is generalized to one-to-many cooperation where a dialogue manager cooperates with various user simulators. These various user simulators are obtained through changing the settings of adjacency pairs as described in the next paragraph. For one training step, dialogue manager interacts with one user simulator to fulfill a complete dialogue, then the dialogue manager and the current simulator are optimized via one-to-one training.