## Appendix B  
In order to evaluate the balance effect of exploration and exploitation, regret \(l_{t}\) is defined to represent the average possible loss at each step:  
\[l_{t}=\mathbb{E}[V^{*}-Q(a_{t})]\] (A6)where, \(V^{*}=Q^{*}(a^{*})=\max\limits_{a\in\mathcal{A}}Q^{*}(a)\) represents the expected return value corresponding to the optimal action. Total regret \(L_{t}\) is defined to represent the total loss:  
\[L_{t}=\mathbb{E}[\sum\limits_{\tau=1}^{t}V^{*}-Q(a_{\tau})]=\sum\limits_{a\in \mathcal{A}}\mathbb{E}[N_{t}(a)](V^{*}-Q(a))=\sum\limits_{a\in\mathcal{A}} \mathbb{E}[N_{t}(a)]\Delta_{a} \tag{10}\]  
where, \(\Delta_{a}=(V^{*}-Q(a))\) represents the gap between the value of the action and the optimal action, and \(N_{t}(a)\) represents the number of times that the action \(a\) has been selected. The goal of maximizing cumulative returns is actually equivalent to minimizing total regret, and an excellent algorithm is hoped to reduce the number of choices for actions with large gaps. For such an evaluation system, the most crucial issue is that the gap in practical problems is difficult to obtain, as \(V^{*}\) is unknown and \(Q^{*}(a)\) also needs to be estimated. However, this evaluation system can still have guiding significance for the evaluation of strategies. It is usually assumed that \(\Delta_{a}\) is a known constant value when discussing. If the total regret \(L_{t}\) increases linearly with the increase of iteration, it indicates that the probability of selecting each action in the algorithm does not change at each time interval. That is, the regret \(l_{t}\) at each step does not change, and the information obtained from exploration is not better utilized. A strategy that can better balance exploration and exploitation should have a sublinear total regret. As the iteration progresses, the regret \(l_{t}\) for each time interval gradually decreases, and the selection of algorithms will gradually abandon those that are likely to achieve lower returns. [PERSON] and [PERSON] has proven that for all total regrets that may become an optimal strategy, an asymptotic lower bound in the form of logarithmic growth is required [16; 21; 22]:  
\[\lim\limits_{t\to\infty}L_{t}\leq 8\ln t\sum\limits_{a|\Delta_{a}}\frac{ \Delta_{a}}{KL(R^{a}||R^{a^{*}})} \tag{11}\]  
The strategy of \(\varepsilon\)-greedy strategy can ensure that all actions are sampled infinitely as the number of iterations increases. Thereby the convergence of \(Q(a)\) is ensured. This means that the probability of selecting the optimal action can converge to greater than \(1-\varepsilon\), which is close to certainty. However, the probability of selecting the optimal action is only approaching a value, and its effectiveness cannot be guaranteed in practice. Obviously, the probability of each choice of action by \(\varepsilon\)-greedy is more than \(\frac{\varepsilon}{\mathcal{A}}\). So the regret \(l_{t}\) is more than \(\frac{\varepsilon}{\mathcal{A}}\sum\limits_{a\in\mathcal{A}}\Delta_{a}\), which is a linear total regret. However, although \(\varepsilon\)-greedy is not the optimal method in theory, [PERSON] and [PERSON] [23] have demonstrated through extensive experimental data that \(\varepsilon\)-greedy can often achieve better results than some complex methods in practice. Set \(episode=t\), \(max\_episodes=T\). We will write the value of \(\varepsilon\) in the improved \(\varepsilon\)-greedy as follows:  
\[\varepsilon_{t}=\left\{ \begin{array}{ll}(0.5+\sqrt{0.25-(t/T)^{2}}),\ (t/T)\in[0,0.5]\\ (0.5-\sqrt{0.25-(1-t/T)^{2}}),\ (t/T)\in(0.5,1]\end{array} \right. \tag{12}\]  
To demonstrate the convergence of the improved \(\varepsilon\)-greedy strategy, we calculate the regret of the strategy:  
\[l_{t}=\pi(a|s)\sum\limits_{a\in\mathcal{A}}\Delta_{a}=\left\{