# Data-Driven Reinforcement-Learning-Based Automatic Bucket-Filling for Wheel Loaders
## 6 Results and Discussions
Figure 15 shows the convergence curve of rewards for Q-learning and transfer Q-learning in the different bucket-pile interaction models. In Figure 15a, Q-learning is only trained on the MCG-pile model, and transfer Q-learning is first trained on the SCG-pile model and then trained on the MCG-pile model. In Figure 15b, Q-learning is only trained on the SCG-pile model, and transfer Q-learning is first trained on the MCG-pile model and then trained on the SCG-pile model. The convergence rate (learning rate efficient) of Q-learning and transfer Q-learning in two bucket-pile interaction models are compared, as presented in Table 3. Using Q-learning as the benchmark, the convergence speed of transfer Q-learning in the medium coarse gravel and small coarse gravel model is improved by 30.3% and 34.1%, respectively. This means that the proposed algorithm has a good transfer learning capability. This improvement can be ascribed to the fact that Q-learning stores the learned knowledge in the Q-function and the transfer Q-learning transfers the Q-function learned from the source task to the Q-function of the target task. Therefore, the agent no longer needs to learn the basic action characteristics in the bucket-filling phase. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Pile** & \
\begin{tabular}{c} **Average Fuel Consumption of** \\ **Recorded Bucket-Filling Phase (mL)** \\ \end{tabular} & **Variance** &
\begin{tabular}{c} **Convergence** \\ **Value (mL)** \\ \end{tabular} & **Improvement (\%)** \\ \hline medium coarse gravel & 36.2 & 61.6 & 33.3 & 8.0 \\ small coarse grave & 51.0 & 42.8 & 45.6 & 10.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The comparison of fuel consumption. Figure 13: Digging force reward of agent on different prediction models. (a) Medium coarse gravel. (b) Small coarse gravel. When the two piles have similar characteristics, such as in category and shape, transfer Q-learning might have better performance in the new bucket-filling task due to the similarity of the optimal Q-function in two tasks [24]. Finally, the amount of data used to build the interaction model also has an impact on the performance of transfer Q-learning on the prediction model. The more data there is, the more states and actions the developed prediction model contains. Therefore, different prediction models have more identical states and actions, and the Q-function of the target task can learn more knowledge from the Q-function of the source task. However, the transfer learning method potentially does not work or even harm the new tasks [25] when the piles or environment are greatly different. Figure 14: Fuel consumption of agent on different interaction prediction models. (**a**) Medium coarse gravel. (**b**) Small coarse grave. Figure 15: Comparison between Q-learning and transfer Q-learning. (**a**) Q-learning is only trained on the MCG-pile interaction prediction model, and transfer Q-learning is first trained on the SCG-pile interaction prediction model and then trained on the MCG-pile interaction prediction model. (**b**) Q-learning is only trained on the SCG-pile model, and transfer Q-learning is first trained on the MCG-pile model and then trained on the SCG-pile interaction prediction model. (**c**) Q-learning is only trained on the SCG-pile model, and transfer Q-learning is first trained on the MCG-pile model and then trained on the SCG-pile interaction prediction model. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Pile**} & **Convergence Episode of** & **Convergence Episode of** & **Improvement (\%)** \\  & **Q-Learning** & **Transfer Q-Learning** & **Improvement (\%)** \\ \hline medium coarse gravel & 825 & 575 & 30.3 \\ small coarse grave & 1025 & 675 & 34.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The comparison of Q-learning and transfer Q-learning on convergence speed.