# Goal-Conditioned Reinforcement Learning within a Human-Robot Disassembly Environment
###### Abstract  
The introduction of collaborative robots in industrial environments reinforces the need to provide these robots with better cognition to accomplish their tasks while fostering worker safety without entering into safety shutdowns that reduce workflow and production times. This paper presents a novel strategy that combines the execution of contact-rich tasks, namely disassembly, with real-time collision avoidance through machine learning for safe human-robot interaction. Specifically, a goal-conditioned reinforcement learning approach is proposed, in which the removal direction of a peg, of varying friction, tolerance, and orientation, is subject to the location of a human collaborator with respect to a 7-degree-of-freedom manipulator at each time step. For this purpose, the suitability of three state-of-the-art actor-critic algorithms is evaluated, and results from simulation and real-world experiments are presented. In reality, the policy's deployment is achieved through a new scalable multi-control framework that allows a direct transfer of the control policy to the robot and reduces response times. The results show the effectiveness, generalization, and transferability of the proposed approach with two collaborative robots against static and dynamic obstacles, leveraging the set of available solutions in non-monotonic tasks to avoid a potential collision with the human worker. collaborative robots; machine learning; reinforcement learning; contact-rich tasks; disassembly; collision avoidance +
Footnote †: journal:  
1222211610122221161012222116101222211610122221161012222116101222211610122221161012222211610122222  
an unstructured environment, this drawback makes it important to improve these systems' perception and decision-making [4]. In this regard, artificial intelligence (AI) and, specifically, machine learning (ML) emerge as promising solutions. Indeed, there has been a growing interest in applying these techniques in multiple domains, such as healthcare [5], autonomous driving [6] or automation, and robotics [7], even in complex scenarios. In particular, reinforcement learning (RL) enables a robot to discover optimal behavior through trial-and-error interactions with its environment autonomously. However, conventional RL only addresses the case of a single goal, specified by a single reward function [8]. This approach leads to limitations such as long learning times, poor generalization, and performance drops when transferring a policy learned in simulation to a real robot when the agent must learn multiple goals. In addition, most current approaches use third-party libraries or communication tools, such as Robot Operating System (ROS), to send the RL policy action commands to the robot. This communication slows the robot's response time, compromising the co-worker's safety in highly dynamic environments. Consequently, these difficulties continue to limit RL's potential and its deployment in realistic applications. Goal-conditioned RL formalizes the problem of learning different goals in one environment [9]. Similar to conventional RL, the main difference in goal-conditioned RL is that, in this case, the reward function depends on the agent's goals, which are represented by the states of the system. This approach may lead to multiple sources of reward during learning. Most goal-oriented RL approaches proposed in the literature include the goal as part of the observation, understood as a final pose [10; 11]. However, in this work, the goal is defined as task completion, giving the robot the freedom to select available trajectories. Thus, this paper presents an approach that simultaneously tackles disassembly and collision avoidance with deep RL techniques in an HRI scenario in which the robot must extract a peg while considering the co-worker's position (Figure 1). This approach uses a model-free algorithm, avoiding modeling collision forces in disassembly or co-worker behavior.