# Smart Industrial Robot Control Trends, Challenges and Opportunities within Manufacturing
## 4 Deep Reinforcement Learning-Based Control  
After the tremendous victory of AlphaGo [63], the attention on reinforcement learning (RL) has significantly increased whilst many potential application scenarios have emerged. Industrial robot control and the field of smart manufacturing is not an exception. However, robotics as an RL domain is substantially different from most common RL benchmark problems [64]. RL methods are based on trial and error [65], where the agent learns optimal behavior through interactions with the environment. However, the best problem representations in the field of robotics often consist of high-dimensional, continuous states and actions, which are hardly manageable with classical RL approaches [64]. RL in combination with deep learning defines the field of Deep-Reinforcement Learning (DRL). As DRL combines the technique of giving rewards that are based on agents' actions and the usage of neural networks for learning feature representations, it enables agents to make decisions in complex environment scenarios when input data are unstructured and high-dimensional [66]. The DRL model typically consists of an agent that takes actions in an environment, whereas the environment or \"interpeter\" returns an observation of how the environment changed after the particular action was taken. Figure 4 illustrates a simplified agent and environment interaction. In the context of industrial robots, the input to the agent is a current state, whereas information in this state depends on the type of action space or learning task. The state can consist of such parameters as robot joint angles, tool position, velocity, acceleration, target pose and other sensor information, which typically is environmental information acquired by a 3D camera. The action space is categorized by two main approaches: pixel-wise actions and actions that directly control robot motions. These different categories also require distinct information. For example, within pixel-wise action space, the agent selects pixels to determine where the action primitive, typically a predefined grasping trajectory, should be executed. In the context of robotic grasping and manipulation, the goal of DRL is to provide an optimal sequence of commands to succeed at the given task [66]. As illustrated in Figure 5, the most commonly applied DRL algorithms can be grouped into model-free and model-based methods. Model-based methods have access to the environment transition dynamics, which is either given or learned from experience. Even though the model-based approaches are more sample efficient, they are limited to the exact or approximate models of the environment. Thus, accurate models are rarely available and often are difficult to learn. Model-free methods, however, learn control policies without building or having access to a model. The ability to provide an optimal control sequence without the use of a model can be advantageous in dynamically complex environments, which is mostly the case in the context of industrial robot control. However, model-free approaches can require a large amount of learning data [67]. To overcome this issue, benefits of combining model-based and model-free methods have been explored in [68]. Both types can be divided into two categories based on the learning approach: policy-based learning and value-based learning [69]. These methods can also be combined, which results in an actor-critic approach [70]. Even though many deep-learning-based computer vision solutions effectively deal with the grasping problem, there are limitations on how much can be achieved without interacting with the environment. When the robotic grasping is approached by DRL methods, the system architecture changes as the optimal policies are learned by interactions with the environment. Unlike computer-vision based techniques, where robot control is mostly performed separately (e.g., by generating trajectories to successfully grasp the object of interest), in DRL-based techniques, the robots' performance is mostly evaluated as a whole.