# Goal-Conditioned Reinforcement Learning within a Human-Robot Disassembly Environment
## 4 Experiments and Results
### Simulation Results  
The policies learned were then evaluated with static and dynamic obstacles. Instead of generating dynamic obstacles with a random or linear motion in a single plane, which may underrepresent hazardous situations, actual human movements related to the task to be performed were used. Specifically, three movements were generated through a virtual reality system. The movements envisaged the robot removing the magnetic gasket so that the worker would have to open the refrigerator door, extract the items inside and close the door. For this purpose, two types of refrigerators were considered:  
* A refrigerator with two doors, one being the refrigerator and the other the freezer, where the co-worker would open one door, remove drawers and shelves from inside, put them aside, close the door and do the same with the second door. * A second refrigerator with a single door on which the co-worker could work either to the robot's left or right (Figure 5). 10 evaluations were performed with the best neural weights from each agent training curve. Regarding the evaluations with static obstacles, the three agents executed the task successfully with average success rates of 87.82%, 77.10%, and 84.43% for PPO, DDPG, and SAC, respectively. The violation of the rotation threshold conditioned all those episodes that failed to perform a successful disassembly. Figure 6 shows the position of the obstacle with respect to the rotation of the part in all episodes sampled in a single evaluation for each agent, as well as the box and whiskers chart of the 10 evaluations performed with each agent. These statistics are represented according to the scores obtained by each agent in 30\({}^{\circ}\) intervals. The violations occurred mostly when the obstacle was close to the \(y\)-axis (60-90\({}^{\circ}\), 90-120\({}^{\circ}\), 240-270\({}^{\circ}\) and 270-300\({}^{\circ}\)), in a neutral standoff position, where the extraction of the part did not pose a risk. This may be due to the uncertainty of the neural networks and the exploration during learning. Nevertheless, the proportion of completed tasks for all the agents could reach a higher value with static, but more realistic positions, placing the obstacle partially to one side of the workpiece. In addition, PPO was the agent with the least variability in scores across the 10 evaluations, with a mean standard deviation of \(\pm\)3.76. In contrast, SAC and DDPG obtained more distributed values around the means, with mean standard deviations of \(\pm\)7.74 and \(\pm\)20.41, respectively. Figure 4: Mean reward and standard deviation of the reward perceived by each of the agents during simulation training. On the other hand, when it comes to the evaluations with dynamic obstacles, the performance of the three agents dropped significantly, with success rates of 76.31%, 72.64%, and 75.14% for PPO, DDPG, and SAC, respectively. This is primarily due to the design of the reward function and how the continuous movement of the worker's hand causes multiple switches between the disassembly and risk-penalty functions. Nevertheless, the co-worker's safety was almost completely assured during disassembly. In about 97% of the cases in which the episode was considered failed, regardless of the agent, it was the removal of the peg that failed to execute, while the agent was able to direct the removal to the opposite side of the co-worker. In less than 3% of the failed episodes, the extraction was performed correctly, albeit to the co-worker's side. Figure 5: Hand movements acquired from virtual reality system to evaluate the learned policies. In the two-door refrigerator (**bottom left**), the black and yellow lines represent the opening and closing of the doors, respectively, the red line represents the transition of the worker from the refrigerator to the freezer behind the robot, and the pink, blue and green lines show the removal of shelves and drawers inside the refrigerator and freezer.