# Automating the Short-Loading Cycle: Survey and Integration Framework
## 3 Automation of Wheel Loaders
### Survey
#### 3.2.1 Bucket-Filling
However, the solution could still not beat the operator in terms of bucket-filling time. This time-delayed neural network was later adapted to learn how to scoop another type of material through the use of reinforcement learning [28]. Within 40 bucket-filling trials, the bucket-filling solution improved the productivity by 5-10% over directly deploying the old network. A simple random forest controller has successfully performed bucket-filling in summer conditions, outperforming a neural network controller in the same environment [29], trained on data collected from non-expert operators and using both low-level signals and vision-based sensors as input. However, when the complexity of the environment increased such as when the temperature decreased, a random forest controller started failing, hence requiring a more complex controller such as that that adapting the attention mechanism [30]. Furthermore, adding pressure sensors as input to the controller also increased the performance of the controller. The results presented in the above two works indicate that data-driven approaches are capable of effectively operating in more complex environments under challenging conditions. Both of these solutions were tested on real machines in realistic environments, showing good performance. Furthermore, the former could match the expert operator in terms of fill factor, which shows that this type of approach has the potential to result in a higher-performing solution. Lastly, reinforcement learning (RL) has been used to attempt to perform bucket-filling. RL has the benefit of allowing us training of an agent to perform the bucket-filling without having to collect data from a real operator, as this can be expensive. Furthermore, in the future, there might not exist human data to train from, hence RL might be the only way to train an agent. A deep RL controller has been trained in simulation to perform the approach towards the pile and scooping of material [31]. The controller was trained using the actor-critic DPPG algorithm [32], purely by exploring the environment. Both the actor and the critic had three hidden layers, mapping a 12 d vector containing sensor data to the wheel loader's tilt, lift, and velocity. The controller was transferred to a miniature wheel loader and tested with no additional training. The solution showed good performance compared to the simulation in terms of the weight of the material within the bucket and the bucket-filling time. Two sequential deep learning agents learned how to perform the approach and scooping of material in a high-fidelity 3D simulation [33]. The two sequential agents were a mucking position agent which determined the scooping point from camera input and a mucking agent which approached and scooped from the determined scooping point. The mucking agent used a variety of sensors such as position, velocity, force, and LIDAR to navigate towards the scooping point and scoop. This type of solution achieved an average of 75% of the max loading capacity. Q-learning has also been applied to the bucket-filling problem. This was achieved by removing the physical model of the wheel loader and instead using a statistical model predicting the state of the wheel loader at some time step [34]. From this model, an agent was trained to perform the bucket-filling process. The trained agent showed good performance in terms of fuel consumption, adaptability, and convergence even with the absence of a physical model. Furthermore, the trained agent could, with transfer learning, be adapted to other types of material. Many of the solutions presented above, rule-based and data-driven alike, relied on the reaction forces between bucket and pile, or pressure in the hydraulics, as input to the system. This seems like a reasonable input choice for an autonomous bucket-filling system, even if it does not translate to what human operators use generally, as with this information autonomous systems can outperform human operators at least in terms of fuel efficiency [35].