# Data-Driven Reinforcement-Learning-Based Automatic Bucket-Filling for Wheel Loaders
## 5 Automatic Bucket-Filling Algorithm  
### Automatic Bucket-Filling Algorithm Based on Q-Learning  
Reinforcement learning is learning what to do--how to map situations to actions--so as to maximize the expected discounted long-term reward. The two most important distinguishing features of reinforcement learning are trial-and-error search and delayed reward. The learner and decision-maker are called the agent. At each time-step, the agent takes action \(a\) according to the current environmental state \(S_{t}\) and the policy \(\pi\) which is a mapping from perceived states to actions. Therefore, as a consequence of action, the environmental state transits from \(S_{t}\) to \(S_{t+1}\) and the agent gets a reward \(r\). The agent and  
Figure 7: Transition probability for medium coarse gravel. (**a**) Velocity of wheel loader. (**b**) Lift cylinder pressure. (**c**) Tilt cylinder pressure. environment generate the trajectories (\(S_{1}\); \(A_{1}\); \(R_{1}\)), (\(S_{2}\); \(A_{2}\); \(R_{2}\)),..., (\(S_{T}\); \(A_{T}\); \(R_{T}\)) [23], until an episode is over. The basic architecture of RL is shown in Figure 8.  
Q-learning is a widely used RL algorithm. Similar to other classical RL methods, the goal of Q-learning is to obtain an optimal policy that maximizes the long-term reward. In the Q-learning algorithm, the agent receives the reward and updates the Q-function corresponding to the action-state. The Q-function represents the expected estimated accumulated reward for the action-state pair under a policy. For example, \(Q(S_{t},A_{t})\) is denoted as the expected long-term reward starting from state \(S_{t}\), taking action \(A_{t}\). By continuous exploitation and exploration, the agent will eventually obtain the optimal Q-function \((Q_{*})\) which determines the action selection policy. The optimal policy \(\pi^{*}(S_{t})\) can be calculated by the following equation:  
\[\pi^{*}(S_{t})=\arg\max_{a\in A}Q_{*}(S_{t},A), \tag{3}\]  
where \(Q_{*}(S_{t},A)\) is the maximum Q-function over all policies. The optimal policy \(\pi^{*}(S_{t})\) is to select the action that maximizes the \(Q_{*}(S_{t},A)\). The Bellman equation of the optimal Q-function \((Q_{*})\) is:  
\[Q_{*}(S_{t},A_{t})=\sum_{S_{t+1},r}P(S_{t+1},r|S_{t},A_{t})\bigg{[}r+\gamma \max_{A_{t+1}}Q_{*}(S_{t+1},A_{t+1})\bigg{]}, \tag{4}\]  
where \(\gamma\in[0,1]\) is the discount factor that determines the present value of future rewards. The Q-learning algorithm is designed by the Bellman equation and contraction mapping theorem. Q-learning is defined by  
\[Q(S_{t},A_{t})\gets Q(S_{t},A_{t})+\alpha\Big{[}R_{t+1}+\gamma\max_{a}Q(S_ {t+1},a)-Q(S_{t+1},A_{t})\Big{]}, \tag{5}\]  
where \(\alpha\in[0,1]\) is the learning rate which reflects the influence of the new experience on the current estimation \(Q(S_{t},A)\). Q-learning starts with an initial \(Q(S_{1},A_{1})\) for each state-action pair. At each time-step, the agent performs an action based on a commonly used exploration method \(\varepsilon\) greedy strategy that selects the greedy action with probability \(1-\varepsilon\), but every once in a while, it selects randomly from all the actions with equal probability \(\varepsilon\) independently of the action-value estimates. Each time an action \(a\) is taken in state \(s\), then the reward \(r\) is fed back from the environment and the next state \(s\) is observed, thus the Q-value is updated with a combination of its current value and the Temporal-Difference Error (TDE)(date-drive). The pseudo-code of the Q-learning algorithm is shown in Figure 9. The code can be found in supplementary materials. In this study, Q-learning based on a prediction model was used to optimize the choice of actions. The Q-learning architecture in automatic bucket-filling is illustrated in Figure 10. By using the real data, the environmental characteristics can be abstracted into the prediction model. Q-learning is trained on the prediction model until the algorithm converges.