## 4 OWS Q-Learning Algorithm
### OWS Q-Learning Algorithm
#### 4.3.3 Improve \(\epsilon\)--Greedy Strategy  
The entire learning process of reinforcement learning can be divided into two stages: (1) exploration stage: the agent randomly explores the environment to learn the current environment; (2) utilization stage: the agent uses the environmental information that has been explored to achieve the goal. The balance of exploration and utilization is also a key challenge in reinforcement learning, commonly used in a heuristic exploration algorithm \(\epsilon-greedy\). The algorithm selects the action with the largest Q value from the action set A with a certain probability, randomly selects the action with the probability of \(\epsilon\), and selects the action with the largest Q value with a probability of \(1-\epsilon\). The mathematical expression for the \(\epsilon-greedy\) strategy is:  
\[\pi^{\epsilon}(a)=\left\{\begin{array}{c}\underset{a\in A}{\arg\max}Q(a), \quad\text{take probability }1-\epsilon\\ \text{randomly select action A,}\quad\text{take probability }\epsilon\end{array}\right\} \tag{19}\]  
where \(a\) is a randomly generated real number between \((0,1)\).  
Strategy \(\epsilon-greedy\) can update and adjust the strategy in time according to the selected action and the instant reward obtained, so as to avoid the algorithm falling into a suboptimal state. Parameter \(\epsilon\) is difficult to set accurately. When \(\epsilon\) is large, the algorithm has strong flexibility, a fast exploration rate, and can quickly explore potential high rewards, so the convergence speed of the algorithm is faster and the adaptability to environmental changes is strong, but the overall cumulative reward may be very low. When \(\epsilon\) is small, the algorithm has strong stability, the exploration rate is slower, and there is more probability of making good use of the better reward that have been explored, but it will make the convergence speed of the algorithm slower, and will make the algorithm less adaptable to environmental changes, but it is possible to obtain a higher cumulative reward in the end.  
This paper optimizes and improves the value of \(\epsilon\), and sets the value of \(\epsilon\) to the form of segments. In order to balance the relationship between exploration and utilization in reinforcement learning, to ensure that the agent can fully explore in the early stage of training, and as the training progresses, the agent can make full use of the information explored for more effective learning, and the initial value of \(\epsilon\) is set to 0.85 under multiple experiments, as shown in Equation (20):  
\[\epsilon=\left\{\begin{array}{c}0.85+episode*0.0001,\ episode\leq m\\ 1,\ \ \ \ \ \ \ \ \ else\end{array}\right. \tag{20}\]where \(m\) is a positive integer less than the maximum number of rounds, set according to the experimental environment.  
The pseudocode for the OWS Q-learning Algorithm 3 is as follows:  
```
Input: state set \(S\), action set \(A\), reward function \(R\) Output: optimal state-action value function \(Q_{s}(s,a)\), from the optimal state-action value function to obtain the optimal strategy \(\pi_{s}(S)\) Process:
1. Initialize \(Q(s,a)=M(s,a),s\), and have a Q value of 0 for all state-action pairs for \(i=1\) to episodes
2. Repeat:
3. Based on the value function Q, the improved \(\varepsilon-greedy\) strategy is used to select action \(a\) under state \(s\);
4. Execute action \(a\), obtain instant reward \(r\), obtain the next state \(s^{\prime}\);
5. \(a^{*}\gets argmaxQ(s^{\prime},a^{\prime})\) \(a^{\prime}\gets M(s,a)\)
6. \(\sigma\gets M(s,a)\)
7. \(M(s,a)\leftarrow\) Q\((s^{\prime},a^{*})\)
8. \(Q(s,a)\leftarrow\alpha\{r+\gamma[\beta M(s,a)+(1-\beta)\sigma]-Q(s,a)\}+(1- \alpha)\gamma[M(s,a)-\sigma]\)
9. \(s\gets s^{\prime}\)  Until \(s\) is terminated End for
```  
**Algorithm 3**OWS Q-learning algorithm