# Obtaining Human Experience for Intelligent Dredger Control: A Reinforcement Learning Approach
## 4 Reinforcement Learning to Find Optimal Control Policy
### Model-Free Learning Algorithm  
In a RL task, if the learning agent has prior knowledge about the state transitions and the coming rewards, it only needs to search the optimal control policy based on the model of the environment. To this end, we can apply model-based algorithms, such as dynamic programming [21], to calculate the expected return by traversing all possible actions and states. However, in this work the state transitions and the reward functions are not initially available for the agent to perform model-based calculation. Thus, it must interact with the virtual environment constructed in the above section to establish the model of the environment. As a typical model-free learning paradigm, Temporal difference (TD) learning can update estimates based on previous learned estimates, without calculating the true expected return. According to Equation (6), the optimal control policy can be obtained if the optimal action-value function is found. In this work, we employ the on-policy SARSA-Learning algorithm [22] that updates the policy \(\pi\) as it follows. Specifically, the learning agent will not choose the best future \(Q(s,a)\), but select the action \(a^{\prime}\) to execute and at time same time to update \(Q(s,a)\). The SARSA learning rule is defined as:  
\[Q(s,a)\gets Q(z,a)+\alpha\bigg{[}p(s)+\gamma Q(z^{\prime},a^{\prime})-Q(z, a)\bigg{]}, \tag{8}\]where \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and \(s^{\prime}\) and \(a^{\prime}\) represents the next state and action, respectively. The above SARSA learning rule is also named as one-step SARSA learning, because the reward will only be propagated one step backwards. A disadvantages of one-step learning is that the learning agent often needs to take many non-exploitive actions until the optimal control policy is obtained. Comparatively, the Monte Carlo learning can speed up the learning process because it can update the entire trajectory from the beginning state to the final state. Thus, the combination of one-step learning and Monte Carlo learning can be a choice. The eligibility trace \(Tr(s,a)\in\mathbb{R}^{+}\) can be defined to indicate to what extent the current reward will influence \(Q(s,a)\) at time-step \(t\). Naturally, the current reward should exert more influence on the recently visited states. To do so, the state-action pair indicating the eligibility trace should be decayed by a parameter in each time-step. Here we use \(\gamma\lambda\) to decay the eligibility trace, and then Equation (8) can be formalized as  
\[ \begin{split}& Q(s,a)\gets Q(s,a)+\alpha\delta Tr(s,a)\\ &\delta=p(s)+\gamma Q(s^{\prime},a^{\prime})-Q(s,a)\end{split}  \tag{9}\]  
where  
\[Tr(s,a)= \begin{cases}\gamma\lambda Tr(s,a)+1\text{ if }&(s,a)=(s_{t},a_{t})\\ \gamma\lambda Tr(s,a)\text{ if }&(s,a)\
eq(s_{t},a_{t}).\end{cases}  \tag{10}\]  
We use \(\delta\) to represent the TD. According to the above equation, the eligibility trace of state-action pairs are required to be updated in each episode. In episode tasks, the state-action pairs that will not be visited during an episode will be zero. However, for non-episode tasks, it will be a challenge to store all the previously visited state-action pairs since the entire list can be very large. Thus, in the cutting process problem, we need to consider how to configure the episodes for updating the state-action pairs. In this work, we will evaluate three intelligent control models, i.e., Q-learning, SARSA, and SARSA(\(\lambda\)). The state space is discretized as: \(C_{v}=\{0,5,10,\ldots,70\}\), \(V_{s}=\{0,1,2,\ldots,16\}\), \(I_{c}=\{900,905,915,\ldots,970\}\), \(I_{p}=\{80,85,90,\ldots,200\}\), \(D_{v}=\{0,-5,-10,\ldots,-80\}\), and \(V_{f}=\{4.0,4.2,4.4,\ldots,8.0\}\). It should be noted that although we have developed tabular-based learning methods, we do not create a fixed list to store the Q-values and the eligibility trace for all the states when implementing them.