# AidIR: An Interactive Dialog System to Aid Disease Information Retrieval
## 3 Architecture of AidIR
### Dialog Policy Learning
#### 3.5.2 Online Learning Stage  
In the online learning stage, we use the A2C algorithm to update the weights that we train in the warm-up stage. The A2C algorithm can be divided into two parts. The first part is the actor network, which predicts the action that the agent will perform. The second part is the critic network, which will predict the \(Q\) value related to the action that the agent performs. A2C is a TD learning method, which means that A2C will bootstrap the reward forward. In AidIR, we use the TD(0) learning method, which will only bootstrap the reward one step forward. The actor network in AidIR is similar to the network in the warm-up stage. However, we did not use the cross-entropy loss to update the actor network. Instead, we employed the policy gradients to update the model. Then, we used the critic network to generate the \(Q\) value of the state \(s\) and the action \(a\) at turn \(t\) and the \(Q\) value of action \(a\) and state \(s\) at turn \(t+1\). The training objective of the critic network is to minimize to mean square error of the advantage function. We mathematically defined policy gradients for dialog policy prediction and token prediction in the actor network in Equations (9) and (10):  
\[\bigtriangledown\theta I_{dp}(\theta)=\mathbb{E}_{a\sim\tau_{\theta}}\Big{[} \bigtriangledown_{\theta}\ln\Big{(}\pi_{\theta}^{dp}(s_{t},a_{t})\Big{)}\hat{ A}_{dp}(s_{t},a_{t})\Big{]} \tag{9}\]  
\[\bigtriangledown_{\theta}I_{token}(\theta)=\mathbb{E}_{a\sim\tau_{\theta}} \Big{[}\bigtriangledown_{\theta}\ln\Big{(}\pi_{\theta}^{token}(s_{t},a_{t}) \Big{)}\hat{A}_{token}(s_{t},a_{t})\Big{]} \tag{10}\]  
where \(\bigtriangledown_{\theta}\!\!J_{dp}(\theta)\) is the policy gradient for dialog policy prediction. \(\bigtriangledown_{\theta}\!\!J_{token}(\theta)\) is the policy gradient for token prediction. We will only calculate the policy gradient for token prediction if the dialog policy is either request or check. \(\pi_{\theta}(s_{t},a_{t})\) is the probability of performing the action \(a\) under state \(s\) at turn \(t\). This is calculated using the softmax function. \(\hat{A}_{dp}(s_{t},a_{t})\) and \(\hat{A}_{token}(s_{t},a_{t})\) are the advantage functions for dialog policy prediction and token prediction at turn \(t\) which are defined in Equations (11) and (12), respectively. The total loss of the actor network is defined in Equation (13):  
\[\hat{A}_{dp}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma Q_{dp}(s_{t+1},a_{t+1})-Q_{dp }(s_{t},a_{t}) \tag{11}\]  
\[\hat{A}_{token}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma Q_{token}(s_{t+1},a_{t+1})- Q_{token}(s_{t},a_{t}) \tag{12}\]  
\[\mathcal{L}_{actor}\Big{(}\bigtriangledown_{\theta}\!\!J_{dp}(\theta), \bigtriangledown_{\theta}\!\!J_{token}(\theta)\Big{)}=\bigtriangledown_{ \theta}\!\!J_{dp}(\theta)+\bigtriangledown_{\theta}\!\!J_{token}(\theta) \tag{13}\]  
where \(\mathcal{L}_{actor}\Big{(}\bigtriangledown_{\theta}\!\!J_{dp}(\theta), \bigtriangledown_{\theta}\!\!J_{token}(\theta)\Big{)}\) is the total loss of the actor network. \(r(s_{t},a_{t})\) is the immediate reward while performing \(a\) under state \(s\) at turn \(t\). \(\gamma\) is a discount factor. \(Q_{dp}(s_{t},a_{t})\) is the \(Q\) value of the dialog policy prediction generated by the critic network for state \(s\) and action \(a\) at turn \(t\). \(Q_{token}(s_{t},a_{t})\) is the \(Q\) value of token prediction generated by the critic network for state \(s\) and action \(a\) at turn \(t\). This is the TD(0) learning method which bootstraps the reward one step forward. In addition, we define the mean square errors of dialog policy prediction and token prediction in the critic network in Equations (14) and (15). \[MSE_{dp}(s_{t},a_{t})=\frac{1}{N}\sum_{t=1}^{N}\Big{[}\hat{A}_{dp}(s_{t},a_{t} )\Big{]}^{2} \tag{14}\]  
\[MSE_{token}(s_{t},a_{t})=\frac{1}{N}\sum_{t=1}^{N_{dp-request,check}}\big{[} \hat{A}_{token}(s_{t},a_{t})\big{]}^{2} \tag{15}\]  
where \(N\) is the number of turns in one episode.