## 3 Materials and Methods
### Maximum Entropy Reinforcement Learning-Soft Actor-Critic  
An RL problem can be seen as a sequential decision problem in a Markov Decision Process (MDP). It is defined as a tuple \((S,A,P,R,\gamma)\), where \(S\) and \(A\) are, respectively, the set of states and actions, \(P\) denotes state transition probability matrix, specifically, the probability of transiting to a successor state \(s_{t+1}\) from the state \(s_{t}\) by taking action \(a_{t}\). The reward function \(R:S\times A\rightarrow\mathbb{R}\), which returns a number indicating the utility of performing an action in the current state. The last element is the discount factor \(\gamma\in[0,1]\), which leverages the importance on short-term rewards against long-term rewards. In RL, the agent interacts with the environment to collect experience and tries to learn an optimal policy \(\pi^{*}\) such that the return, namely cumulative reward, is maximized. \[\pi^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^{t}r_ {t+1}\right],\]  
where \(r_{t}\) refers to the _immediate reward_ at time point \(t\), \(\tau\) is the trajectory, characterized as a sequence of \(\{s_{0},a_{0},r_{1},\ldots,s_{T},a_{T},r_{T+1}\}\) following the policy \(\pi\) and \(T\) is the time horizon to reach the terminal state. The maximum entropy RL algorithm Soft Actor Critic (SAC) [33; 34] differs from the standard RL in that it changes the goal by incorporating an additional weighted policy entropy term \(\mathcal{H}(\pi(\cdot\mid s_{t}))\). The policy entropy describes the stochasticity of the policy, indicating the degree of exploration. In this manner, the greedy policy returned by SAC includes an internal exploration bonus, which is advantageous for maximum entropy RL, as no explicit exploration strategy needs to be formulated. The objective function of SAC is defined as:  
\[\pi^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^{t}(r _{t+1}+\alpha\mathcal{H}(\pi(\cdot\mid s_{t})))\right],\]  
where \(\alpha\) is the temperature coefficient determining the weight for policy entropy. In [33], the weights are pre-determined by the users and need manual running for different tasks. As an improvement, the work [34] proposed automatically adjusting \(\alpha\) for different tasks. To enable a learnable \(\alpha\), the authors impose a constraint that \(\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{\pi}}[\mathcal{H}(\pi( \cdot\mid s_{t}))]\geq\overline{\mathcal{H}}\), where \(\overline{\mathcal{H}}\) is a pre-defined entropy lower bound to ensure a minimal level of exploration. The constraint can be cast into a dual optimization problem shown below so that the temperature \(\alpha\) turns a learnable parameter. For an exact theoretical derivation, please refer to the original work [34]. \[\alpha_{t}^{*}=\arg\min_{a_{t}}\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{t}^{*}}\left[ -\alpha_{t}\log\pi_{t}^{*}(\mathbf{a}_{t}\mid\mathbf{s}_{t};\alpha_{t})- \alpha_{t}\overline{\mathcal{H}}\right].\]  
The critic part learns the Q-values with the additional policy entropy term based on a re-defined Bellman update:  
\[\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t})=r_{t+1}+\gamma\Big{(}Q_{\hat{\varphi}} (s_{t+1},\tilde{a}_{t+1})-\alpha\log\pi_{\theta}(\tilde{a}_{t+1}\mid s_{t+1}) \Big{)},\]  
where \(\tilde{a}_{t+1}\sim\pi_{\theta}(\cdot\mid s_{t+1})\), \(\phi\) and \(\tilde{\phi}\) refer, respectively, to the running and target critic network for training stability similar to DQN. The critic loss is then computed by sampling a minibatch from the replay buffer \(\mathcal{D}\). \[J_{Q}(\theta)=\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\mathcal{D}} \left[\frac{1}{2}\big{(}Q_{\hat{\varphi}}(\mathbf{s}_{t},\mathbf{a}_{t})-\hat {Q}(\mathbf{s}_{t},\mathbf{a}_{t})\big{)}^{2}\right].\]In the policy improvement step, the actor is updated towards an exponential of the Q-values to still allow for a distribution of the policy via Kullback-Leibler divergence.