# Smart Industrial Robot Control Trends, Challenges and Opportunities within Manufacturing
## 3 Computer Vision-Based Control  
The recent achievements in the computer vision and deep learning fields [27; 28] have raised awareness of the benefits that such systems can provide to industrial robot systems, manufacturing and the business itself. Computer vision-based methods in combination with the control loop of the robot are one of the most common approaches [29] to deal with the uncertainty of the environment and to interact with it. After data acquisition, as illustrated in Figure 2, the first step in the workflow of such systems is to detect objects or the presence of the objects or humans. For some scenarios, such as monitoring of safety zones in human-robot collaboration [11] it is enough to detect just a presence of an object or human to continue, pause or stop task execution. In the majority of scenarios, it is not enough, and all the required information about detected objects needs to be extracted  
Figure 1: Smart industrial robot concept. and taken into account in the planning step. In these scenarios, the typical actions after object detection are object pose-estimation, grasp pose determination, motion planning and control in accordance. Although, as illustrated in Figure 2, this is only one structure of a workflow, and some solutions are bypassing the pose-estimation step by directly looking at possible grasping points [30]. The novel object detection methods are usually deep-learning-based [31; 32]. Even though the main focus of object detection algorithm development historically has been addressed in different fields [33], the need for such systems is also present in the smart manufacturing context to deal with uncertainties of the environment. Thus, such deep learning-based approaches have proven to be quite flexible if conditions changes [34]. Object detection in industrial robotic tasks differs from other tasks as extracted information needs to be more detailed and precise in order to successfully manipulate objects. The precision and accuracy of the computer vision system can be directly connected to the quality of the product. Several different scenarios or combinations of conditions can be distinguished, such as working with overlapping, similar, different, multiple, complex objects with undefined positions, etc., where some situations are illustrated in Figure 3. Additionally, the objects can be known, whether information is available beforehand, or otherwise, the objects are unknown. Some combinations of conditions make the task more complex and vice versa, and accordingly, available methods are similar for some scenarios. The required industrial robot control methods briefly can be distinguished in two different cases, respectively:  
* Tasks that require the robots' end of arm tooling to be precisely positioned for the entire trajectory,
* Tasks that require explicit grasp planning. Figure 2: Typical pipelines of computer-vision based industrial robot control. The first task can be considered to have low complexity in regards to computer-vision as the usual region of interest includes a single object that is not obstructed by other objects. The application scenarios are multiple, but most applicable cases in the smart manufacturing context are connected to pose estimation of the part and more precise feature extraction for further processing such as path generation in autonomous spray painting [35], welding [39], inspection [40], polishing, etc. Historically, in these kinds of scenarios, CAD model comparison-based methods have been proposed [41] or other solutions that are based on geometrical characteristics [42]. In some cases, CAD models of the part are not available or accurate enough; therefore, such methods are not capable of handling surfaces without mathematical representations. More recent methods can automatically generate tool paths by utilizing point cloud [43,44]. The application scenarios for the second task has the most occurrence in modern industrial robot use-cases.