## 4 Deep Reinforcement Learning in the Production Industry
### Robotics  
Robot manipulation encompasses a wide range of tasks, from assembly operations, such as screwing and peg-in-hole, to robot grasping and pick-and-place operations [151, 152]. The characteristics of DRL make it very suitable for robotic tasks, which has produced a close relationship between both fields for many years, leading to promising results in the future [50, 120]. This section includes a mini-review of the most recent DRL applications in this vast field. Firstly, this review starts with the peg-in-hole assembly, the robotic manipulation task with the most DRL applications according to the literature search, and its high precision characterises it. For this task, PPO is the most commonly applied algorithm with applications such as [103, 153, 154]. Among them, the PPO controller developed by [PERSON] et al. (2021) [103] should be noted, where the algorithm is trained through curriculum learning. This technique consists of dividing the learning problem into several subtasks and learning them in ascending order of complexity, which allows the learning of the simpler tasks to be used to learn the more complex ones and improves generalisation skills [155]. Although PPO applications abound, other DRL algorithms can be found. For example, [PERSON] et al. (2021) propose an actor-critic-based algorithm that improves the stability and sample efficiency of other state-of-the-art algorithms such as DDPG and TD3 [103, 104]. In addition, training this algorithm with hierarchical reinforcement learning (HRL) notably increases the generalisation capability to other assembly tasks. HRL consists of decomposing tasks into simpler and simpler sub-tasks, establishing levels of hierarchy in which more complex parent tasks are formed by simpler child tasks. With this technique, the most basic tasks are learned, which allow for the development of more complex tasks [156]. Furthermore, among the applications of hybrid algorithms, the work of [PERSON] et al. (2020) [104], which uses SAC to learn contact-rich manipulation tasks and tests the algorithm with a real robot arm, and the proposed uses of DDPG to control the force in contact-rich manipulation in [157] and to enhance the flexibility of assembly lines in [158] are noteworthy. The latter is particular in that it uses a digital twin model of the assembly line to train the DDPG algorithm, and once trained, this model is used to monitor the assembly lines and predict failures during the production stage. Digital twins are a technology that is increasingly important in I.40 and I.5.0, which seems to be crucial to the development of smart manufacturing. Indeed, some DRL control applications, such as [158], leverages this technology to increase their data efficiency and robustness. [PERSON] et al. (2022) train a DQN algorithm with the digital twin model of a robot arm that has to perform a grasping task [159]. In this line, [PERSON] et al. (2021) do the same with DQN and DDQN + PER for a pick-and-place task [69]. Both cases highlighted the smoother transfer of knowledge from the simulation to the real environment thanks to digital twin models. Finally, another robot manipulation task to which DRL is currently applied is pick-and-place, which in turn includes other tasks such as motion planning, grasping and reaching a point in space [50]. As in other robotic tasks, the use of DDPG is predominant [160]. Some recent examples are [161], whose objective is reaching a point and measuring the influence of different reward functions, and [162], where the application of DDPG results in robust grasping in pick-and-place operations. In addition, the joint use of DDPG and HER is common, highlighting the work of [PERSON] et al. (2021), that DDPG + HER is used together with HRL to learn complex pick-and-place tasks [163]. Nonetheless, other state-of-the-art algorithms are used in this field, such as TD3 + HER for the motion planning of robot manipulators [164] and PPO and SAC for a grasping task with an outstanding success rate [165].