## 4 Proposed ms-SDRGN Solution For SAG-MCS
### Learn Stochastic Policies with Adjustable Action Entropy  
Based on the Q-value produced by DRGN, we can learn a deterministic policy, where each Q-value represents a fixed probability of the corresponding action. However, deterministic policies can easily jump into local optimum and lack for exploration in complex, real-world scenarios. Inspired by the maximum entropy RL framework [50; 51], we utilize soft Q-loss to learn a stochastic policy in SAG-MCS, with the objective of maximizing expected reward and optimizing the action entropy towards a certain target. A flow chart of the training process is presented in Figure 4.  
Firstly, we sample previous interaction experiences from the replay buffer as training inputs. The ms-SDRGN learned model infers a set of Q-value from the experiences. Then, we apply temperatured softmax operation to Q-value for getting the action probability:  
\[\pi(\mathcal{O}_{t},a_{t})=\text{softmax}_{a_{t}}\bigg{(}\frac{Q(\mathcal{O}_{ t},a_{t})}{\alpha}\bigg{)}=\exp\bigg{(}\frac{Q(\mathcal{O}_{t},a_{t})}{\alpha}- \log\Sigma_{a_{t}}\exp\bigg{(}\frac{Q(\mathcal{O}_{t},a_{t})}{\alpha}\bigg{)} \bigg{)}, \tag{13}\]  
where \(\alpha\) is an adjustable temperature parameter, and Q-value \(Q(\mathcal{O}_{t},a_{t})\) is produced by the learned model when receiving \(\mathcal{O}_{t}\) and \(a_{t}\) as inputs. Specific action during simulation is sampled from the action probability. Then, we use Equation (13) to estimate the action entropy by calculating the information entropy expectation from sampled experiences:  
\[\mathbb{E}[\mathcal{H}_{\pi}(\mathcal{O},a)]=\mathbb{E}[-\Sigma_{a_{t}\sim\pi} \pi(\mathcal{O}_{t},a_{t})\cdot\log\pi(\mathcal{O}_{t},a_{t})], \tag{14}\]  
The action entropy represents the action uncertainty of policy \(\pi\), which can be adjusted by the temperature parameter \(\alpha\). Therefore, we preset a target action entropy as \(\mathcal{H}_{\pi}^{target}=p_{\alpha}\cdot\max\mathcal{H}_{\pi}\), where the maximum action entropy is determined by action space as \(\max\mathcal{H}_{\pi}=\log(\dim\mathcal{A})\), and \(p_{\alpha}\) is a hyper-parameter named target entropy factor. Note that different RL tasks require different levels of exploration, so \(p_{\alpha}\) shall be modified according to specific scenarios. More concretely, our goal is to let the action entropy \(\mathbb{E}[\mathcal{H}_{\pi}(\mathcal{O},a)]\) approach the pre-defined target action entropy \(\mathcal{H}_{\pi}^{target}\), by updating the temperature parameter \(\alpha\) through gradient descent:  
\[\
abla_{\alpha}=f\Big{(}\mathcal{H}_{\pi}^{target}-\mathbb{E}[\mathcal{H}_{\pi }(\mathcal{O},a)]\Big{)}. \tag{15}\]  
Figure 4: The training process of ms-SDRGN. In the flow chart, solid lines indicate feed forward propagation, and dashed lines denote updating parameters by backpropagation.  
where \(f\) is a customized activation function and \(\mathcal{H}_{\pi}^{target}\) denotes the target action entropy. The configurable action entropy mentioned above guarantees the balance between interaction stability and exploration capability of the policy.  
Following Soft Q-learning [50], we also include the temperature parameter \(\alpha\) to help define a V-value function for the target model. Finally, we use the mean squared error calculated by Q-value function and V-value function as \(Q_{loss}\):  
\[V(\mathcal{O}_{t})=\alpha\cdot\log\Sigma_{at}\exp\Big{(}\frac{Q(\mathcal{O} _{t},a_{t})}{\alpha}\Big{)}, \tag{16}\]  
\[Q_{loss}=\frac{1}{S}\Sigma(r_{t}+V(\mathcal{O}_{t+1})-Q(\mathcal{O}_{t},a_{t} ))^{2}. \tag{17}\]  
where \(r_{t}\) is the reward earned in timestep \(t\), \(V(\mathcal{O}_{t})\) denotes the V-value function and \(S\) is the batch size. The Q-value function \(Q(o,a)\) of the learned model is updated through minimizing the \(Q_{loss}\) in Equation (17). In the learning process, ms-SDRGN target model will be updated periodically by duplicating the parameters of the learned model directly.