## 4 OWS Q-Learning Algorithm  
Reinforcement learning, as a learning method of machine learning, is an artificial intelligence algorithm that does not require prior knowledge and acts based on feedback from the environment. Through continuous interaction with the environment, trial, and error, the specific purpose is finally achieved, or the overall action benefit is maximized. It does not need the label of the training data, but it needs to know whether the feedback given by the environment for each step is a reward or a punishment. The feedback can be quantified, and the behavior of the training object is constantly adjusted based on thefeedback. Reinforcement learning is mainly composed of five parts: agent, environment, state, action, and reward. Agent: The subject of reinforcement learning training. Environment: The environment in which the agent is located. State (S): The state in which the current environment and agent are located, because the position of the agent is constantly changing, the entire state is changing, and the state contains the state of the agent and environment. Action (A): An action set that the agent can take based on the current state. Reward (R): The agent takes a specific action under the current state and obtains certain feedback from the environment as a reward. State transition is represented by the \(p\) function:  
\[p(s^{\prime}|s,a)=p\big{(}S^{\prime}=s^{\prime}|S=s,A=a\big{)} \tag{7}\]  
This is a conditional probability density function that represents the probability of the \(p\) function output if the current state \(s\) and the action \(a\) are observed. The framework of reinforcement learning is shown in Figure 5:  
\(S_{t}\) is the state of the environment at \(t\) moment, \(A_{t}\) is the action that the agent performs at \(t\) moment in the environment, \(A_{t}\) makes the state of the environment change to \(S_{t+1}\). In the new state, the environment generates a new feedback \(R_{t+1}\), and the agent performs a new action \(A_{t+1}\) according to \(S_{t+1}\) and \(R_{t+1}\), and so on until the end of the iteration. In reinforcement learning, agents learn to solve sequential decision problems, which are usually described by the Markov decision process (MDP). It is used to describe and solve the problem hypothesis that agents learn strategies to maximize returns or achieve specific goals in the process of interacting with the environment. The quaternion of the [PERSON] decision process (MDP) is \(<S,A,R,P>\), where \(S\) is the set of states, \(A\) is the set of actions, \(R\) is the reward function, \(R_{t}=R(S_{t},A_{t})\) is the reward obtained by the agent after executing \(A_{t}\) under state \(S_{t}\), \(P\) is the state transition probability, denoted as \(P(S_{t+1},R_{t}|S_{t},A_{t})\). It indicates the probability distribution of \(t\) moment state \(S_{t}\), after executing action \(A_{t}\), obtaining a reward \(R_{t}\) and the next state is \(S_{t+1}\).The complete Markov decision model is shown in Figure 6.  
In MDP, the status \(S_{t+1}\) and reward \(R_{t+1}\) depend only on the previous state \(S_{t}\) and action \(A_{t}\), as described by the following function:  
\[p\big{(}s^{\prime},r|s,a\big{)}=Pr\big{\{}R_{t+1}=r,S_{t+1}=s^{\prime}|S_{t},A _{t}\big{\}} \tag{8}\]  
Figure 5: Reinforcement learning framework. Figure 6: Markov decision chain. In the process of learning, the agent tries to maximize the cumulative return, and the estimated discounted return \(\gamma\) expressed by the discount rate \(G_{t}\) is:  
\[G_{t}=\sum\limits_{k=0}^{\infty}\gamma^{k}R_{t+k+1},\ \ 0\leq\gamma\leq 1 \tag{9}\]  
The agent follows the policy \(\pi(a|s)\), which is a function of the probability of selecting each possible action from that state. It represents the probability that the agent chooses to perform action \(A_{t}=a\) in the case of \(S_{t}=s\).