# Automatic Drones for Factory Inspection: The Role of Virtual Simulation

Omid Maghazei, Torbjørn H. Netland, Dirk Frauenberger, Tobias Thalmann

► To cite this version:

Omid Maghazei, Torbjørn H. Netland, Dirk Frauenberger, Tobias Thalmann. Automatic Drones for Factory Inspection: The Role of Virtual Simulation. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.457-464, 10.1007/978-3-030-85910-7_48. hal-03806551

HAL Id: hal-03806551

https://inria.hal.science/hal-03806551

Submitted on 7 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Automatic Drones for Factory Inspection:
The Role of Virtual Simulation

Omid Maghazei[0000-0002-2257-3550], Torbjørn H. Netland[0000-0001-7382-1051],
Dirk Frauenberger, and Tobias Thalmann

ETH Zürich, Switzerland

omaghazei @ethz.ch

**Abstract.** Manufacturers experiment with the use of drones for various processes such as surveillance, inspection, cycle counting, and intralogistics, but implementation into routine operations remains rare. One reason for low adoption rates relates to the manual control requirements of most drone systems. This paper studies the use of automatic drones in manufacturing. A virtual simulation of drone flights in a manufacturing facility was developed to identify and evaluate the potential of automatic drones for thermal inspection of injection molding machines. This paper reports the implementation of the virtual simulation and discusses how such simulations can inform the use of automatic drones for factory inspections.

**Keywords:** Unmanned Aerial Vehicles, Automatic Drones, Virtual Simulation.

# 1 Introduction

Unmanned vehicles, such as automated guided vehicles (AGV), can take over routine tasks previously done by humans. Automated vehicles require advanced algorithms to detect and avoid static and moving obstacles [1], as well as systems for path planning and traffic management [2]. Mobility becomes an order of magnitude more difficult when the vehicle moves through the air. Yet unmanned aerial vehicles (UAVs)—particularly known as drones—have high potential in both outdoor and indoor manufacturing environments [3].

Indoor industrial applications of drones are much less common than outdoor applications [3]. For example, drones are already finding applications in emergency operations [4], infrastructure maintenance [5], logistics [6], and warehouse management [7]. Indoor drone flights pose additional challenges compared to outdoor flights, such as navigation, obstacle detection, explosion risk, and collision avoidance in the three-dimensional (3D) factory space [8]. There is also a considerable risk to both humans and assets involved when drone systems fail in indoor environments. Other limiting factors are noise and privacy concerns. Most importantly, established alternatives to drones such as floor-based AGVs, conveyor belts, and wall- or roof-mounted cameras and sensors can do many of the tasks usually attributed as potential applications of drones (e.g., surveillance, inspection, or intralogistics).
From an industrial management perspective, the challenge is to identify niche applications where drones can outcompete existing technologies in terms of accessibility, speed, safety, cost, or accuracy—or do tasks not previously carried out by any other technology. To address this challenge, the authors engaged in field research with a leading manufacturer of products for household construction, Household Inc. The authors identified several potential use cases and employed piloted drones to test the use cases’ feasibility. High potential was found in using drones in routine inspections of thermal losses in injection molding machines, which would normally require manual inspection. However, with piloted drones, the return on investment would be marginal because a human operator would still be needed. It was concluded that the use case is viable, but only with automated drones.

It is not without reason that most reported drone implementations in manufacturing use piloted drones [9]. Automated drone technology is still subject to basic research [10] and—at the time of writing—is only used for “simple” tasks such as cycle counting in warehouses. This paper seeks to address this limitation using the simulation of automated drone inspection flights in the factory floor environment of Household Inc. The purpose is to demonstrate the role of simulation for automated drones and identify challenges and opportunities for real implementation in manufacturing.

## 2 Simulation of automatic drone flights

Automation is defined as the partial or full replacement of a human operator’s intended task (e.g., decision selection) with a machine agent, usually a computer [11]. There are various approaches to classify the ‘level of automation’ of drones. This study uses a framework that classifies drone automation in five levels based on the level of involvement of drone pilots (see Table 1) [12, 13]. The literature recommends step-wise automation. This paper’s primary focus is to examine the operational and technological challenges of moving from level 0, ‘no automation,’ to automatic drones indoors with level 2, ‘partial automation.’

<table><caption>**Table 1.** Five levels of automation for drones (adapted from [12, 13])</caption><thead><tr><th>Level 0</th><th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th></tr></thead><tbody><tr><td>None</td><td>Low</td><td>Partial</td><td>Conditional</td><td>High</td><td>Full</td></tr><tr><td>Entirely manual flight</td><td>Piloted flight with autopilot assistance</td><td>Automated flights with predefined waypoints, pilot supervision when necessary</td><td>Automated flight with waypoints and adaptive safety systems but with pilot involvement</td><td>Automated flight with failsafe systems, with no pilot involvement</td><td>Autonomous flight</td></tr></tbody></table>

Simulation is an established method in industrial engineering and management. It is a practical approach when field implementation is costly, risky, or impossible [14, 15]. Simulation can ‘extend our understanding into the realm of possibility, imagination, and projection of future possibilities for pretesting yet-to-be-implemented possibilities’
[16, p. 1196]. There is a wide range of simulation approaches, such as virtual simulation, system dynamics, discrete event simulation, Monte Carlo simulation, agent-based simulation, and intelligent simulation [see, 17]. This paper focuses on virtual simulation.

Virtual simulations can be used to visualize and test digital copies of smart factories (e.g., digital twins). It can help optimize current factory processes or avoid future failures. For instance, virtual simulations can allow for optimizing planning and maintenance decisions concerning the operations of a manufacturing system or assist in layout and factory planning. Virtual simulation also has potential in the study of intelligent robotics and industrial automation [18] by simulating and visualizing operations at the micro level using 3D graphics models [19].

## 3 Implementation of drone flight simulation

In this study, a simulation was used to explore the potential of automated drones for the inspection of injection molding machines in Household Inc. The simulation model was based on an area of Household Inc.'s shop floor with nine injection molding machines. A simulation model mirroring the real shop floor was developed. The simulation model was implemented in five steps.

The first step was to simulate drone operations. The research team used Gazebo, an open-source '3D dynamic simulator with the ability to accurately and efficiently simulate populations of robots in complex indoor and outdoor environments' [20]. Gazebo was installed on a Linux operating system (OS) with a Ubuntu OS that supports the robot operating system (ROS)—a collection of tools, libraries, and conventions that helps software developers to build robot applications [21, 22]. Then RotorS, a modular drone simulation framework, was used to assemble a drone in Gazebo [22]. Figure 1 shows the real AscTec Firefly hex-rotor drone from Ascending Technologies (owned by Intel) and the simulated Firefly drone in Gazebo.

<figure><img src="image_3.png" /><figcaption>Fig. 1. (a) Real AscTec Firefly drone; (b) Simulated Firefly hex-rotor drone in Gazebo.</figcaption></figure>

The modular framework included drone components, namely, a body, a fixed number of rotors, and sensors attached to the body (e.g., inertial measurement unit, a generic odometry sensor, a visual-inertial sensor, and sensors developed by the user), all of which were 'designed to be analogous to its real-world counterparts' [22, p. 595]. The framework also included a position controller to access various levels of commands and a state estimator to obtain information about the drone state [22]. The RotorS simulator provided a fully functional trajectory tracking controller, which allowed for implementing higher-level tasks such as collision avoidance, path planning, and vision-based problems [22].
The second step was to simulate the environment for drone operations by building a virtual factory. Dimensions, appearance, and positions were measured and noted in the same production area, where the human-assisted drone experiment had been conducted [23]. The digital model of the shop floor was created in Fusion 360 software. Having knowledge about the operating environment helped better determine the drone path for precise navigation in a global positioning system–denied environment [see 24].

The third step was to define feasible airspace areas or corridors for the safe operation of automatic drones in factories, in part due to the air turbulence generated around drones. In this study, a safe area was defined by following the suggested flying space boundary [see, 8], allocating a 50 centimeter safety distance around the drone. An area was designated for safe take-off and landing for the automatic drone. Figure 2 shows the 3D model of the safe flight zone in the factory.

<figure><img src="image_4.png" /><figcaption>Fig. 2. 3D model of the safe flight zone in the shop floor by Fusion 360.</figcaption></figure>

The fourth step was to develop flight scenarios. In this study, two scenarios were defined based on a trade-off between drone flight duration and coverage. Each scenario consisted of a flight trajectory with predefined waypoints for thermal inspection of the nine injection molding machines. The scenarios considered obstacles in the indoor physical environments that limited the maneuverability of automatic drone flights. The scale and duration of drone operations were also limited by current battery technologies. The first scenario was described as ‘quick and limited’ and designed to collect data from restricted angles (see Figure 3a) through 49 waypoints. The second scenario was described as ‘slow and detailed’ and designed to collect more data from each machine from multiple available angles in the safe zone, most notably by going on top of the machines with movable robotic arms (see Figure 3b) through 163 waypoints.

<figure><img src="image_5.png" /><figcaption>Fig. 3. A schematic representation of the (a) quick and limited scenario and (b) slow and detailed simulated drone flight scenarios.</figcaption></figure>
The final step was to launch the simulation of drone flights in the virtual factory in Gazebo. Figure 4 shows the file structure of the simulation model that combines the files from RotorS, including both unchanged and changed libraries, Gazebo files, waypoints from Microsoft Excel software, and a 3D model of the shop floor from Fusion 360 software. Having combined the various files in Gazebo allowed for the development of AscTec drone flights in the 3D model of the company's shop floor.

<figure><img src="image_6.png" /><figcaption>Fig 4. The main file structure of the implemented Gazebo simulation.</figcaption></figure>

## 4 Results and discussion

The virtual model simulated drone flights close to the real behavior of automated drones. It allowed for no-risk virtual tests and exploration of technological and operational opportunities and challenges of using automatic drones indoors. In this study, the simulation runs were evaluated on the basis of speed and safety.

The quick and limited model, with 49 waypoints, took 3 minutes and 52 seconds. The slow and detailed scenario, with 163 waypoints, required 5 minutes and 47 seconds. In the case of Household Inc., both these durations were acceptable and considerably faster than manual inspections, which were estimated to on average 90 minutes for 9 machines (10 minutes per machine including preparations and breaks). Household
Inc.'s production plant in Switzerland has 80 injection molding machines that are currently manually inspected four times a year. In a scenario with automated drones, inspections could be run much more frequently.

Concerning the safety assessment, a 50 centimeter flying-space boundary was considered the minimum safety requirement by Household Inc. For some automatic drone flights, particularly for the inspection of machines with movable robotic arms, this safety distance was classified as risky. To assess potential risks for the two scenarios, an analysis of the minimum distance of the drone from three particularly critical machines with robotic arms was conducted (i.e., the three first machines in Figure 3(a) and 3(b)). The drone's minimum distance to any object in the area 90° straight down and 45° to the front was measured. This view resembled the effective coverage area of the thermal camera. A Python script was written that read the depth sensor data published by the drone over the ROS channels. This sensor provided a point cloud, each point representing an observation of a surface. The script iterated for each published set (i.e., frame) through each point and determined the one closest to the drone. This minimal distance was then logged in a text file. Figure 5 shows the result of this test. It compares the two scenarios according to potential risks, defined as any flight position closer than 1 meter from an object. The horizontal axis represents the flight completion percentage, and the vertical axis shows the minimum distance measures of the down-view to objects. The quick and limited and slow and detailed scenarios had 0 and 13 potential risks, respectively.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Minimum distance measures for the two scenarios.</figcaption></figure>

The results demonstrated that the virtual simulation helped Household Inc. evaluate the potential of automatic drones for thermography inspection and could quickly be adapted for different use cases and scenarios. The potential implementation of automated, routine thermal inspection of injection molding machines could help increase productivity at Household Inc. The company could discover heat losses and irregularities earlier than it can now, which could (1) reduce defective parts produced or improve the material quality of products, (2) increase machine uptime, (3) reduce maintenance
time (e.g., by reducing time to search for undetected heat losses, sources of oil leakages, and defective band heaters, isolations, and cooling systems), and (4) reduce energy costs. The potential savings to maintenance costs alone were estimated to be USD 24,000 annually by Household Inc. managers. Given that the technology would be available, safe, and cost-efficient to implement and operate, there should be a positive business case for automated drone flights for inspection. With the current development of drone technology, such implementation could be possible within the next few years.

# 5 Conclusions

Automatic drones can improve indoor manufacturing processes, particularly repetitive tasks such as inspection operations, intralogistics for small parts, cycle counting in warehouses, security monitoring, ergonomic monitoring, risk monitoring, and building-information modeling. Implementing automatic drones involves considerable technological and operational challenges. This paper showed that virtual simulations can animate automatic drone flights in 3D and help better understand the technological and operational challenges of using new technologies in real manufacturing environments and support decision-making and problem-solving processes. For instance, factory planners can use simulation models to evaluate, allocate, and design airspace for drone flights. Virtual simulation of automatic drones also helps set and define safe flight zones for indoor airspace, which helps quickly evaluate various scenarios, such as trade-offs between coverage, speed, and risks. Furthermore, factory managers can use 3D models to communicate and collaborate with drone technology developers, as well as sensor manufacturers and robotics companies, by virtually presenting the technological requirements of using automatic drones with embedded sensors/technologies in their settings. For instance, primary metal companies would require drones and thermal cameras that can cope with high temperatures, petrochemical companies require explosion-proof drone systems, and e-commerce warehouses may benefit from collaborative drone technologies (i.e., swarming) with advanced path planning and fleet management systems.

# References

1. Franke, J., Lütteke, F., Versatile Autonomous Transportation Vehicle for Highly Flexible Use in Industrial Applications. CIRP Annals, 61 (1), 407-410 (2012).

2. Gyulai, D., Bergmann, J., Váncza, J., Adaptive Network Analytics for Managing Complex Shop-Floor Logistics Systems. CIRP Annals, 69 (1), 393-396 (2020).

3. Maghazei, O., Netland, T., Drones in Manufacturing: Exploring Opportunities for Research and Practice. Journal of Manufacturing Technology Management, 31 (6), 1237-1259 (2019).

4. Park, Y., Moon, I., UAV Set Covering Problem for Emergency Network. In: Ameri F., Stecke K., von Cieminski G., Kiritsis D. (eds), Advances in Production Management Systems (APMS), Towards Smart Production Management Systems, IFIP Advances in Information and Communication Technology, p. 84-90, Springer, Switzerland (2019).
5. Maghazei, O., Steinmann, M., Drones in Railways: Exploring Current Applications and Future Scenarios Based on Action Research. European Journal of Transport and Infrastructure Research, 20 (3), 87-102 (2020).

6. Herrera, S., Cervantes, A., The Drones Are Coming! How Amazon, Alphabet and Uber Are Taking to the Skies, https://www.wsj.com/articles/the-drones-are-coming-11571995806, last accessed 2019/09/30.

7. Wawrla, L., Maghazei, O., Netland, T., Applications of Drones in Warehouse Operations. Whitepaper, D-MTEC, ETH Zurich, (2019).

8. Khosiawan, Y., Nielsen, I., A System of UAV Application in Indoor Environment. Production & Manufacturing Research, 4 (1), 2-22 (2016).

9. Mazur, M., Wiśniewski, A., Clarity from Above, https://www.pwc.pl/en/publikacje/2016/clarity-from-above.html, last accessed 2019/07/22.

10. Floreano, D., Wood, R. J., Science, Technology and the Future of Small Autonomous Drones. Nature, 521 (7553), 460-466 (2015).

11. Parasuraman, R., Riley, V., Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39 (2), 230-253 (1997).

12. Nanduri, A., Building the Autonomous Future, https://nari.arc.nasa.gov/sites/default/files/attachments/6%20Nanduri-Building%20the%20Autonomous%20Future.pdf, last accessed 2020/07/18.

13. Radovic, M., Tech Talk: Untangling the 5 Levels of Drone Autonomy, https://www.droneii.com/project/drone-autonomy-levels, last accessed 2019/03/20.

14. Shafer, S. M., Smunt, T. L., Empirical Simulation Studies in Operations Management: Context, Trends, and Research Opportunities. Journal of Operations Management, 22 (4), 345-354 (2004).

15. Negahban, A., Smith, J. S., Simulation for Manufacturing System Design and Operation: Literature Review and Analysis. Journal of Manufacturing Systems, 33 (2), 241-261 (2014).

16. Burton, R. M., Obel, B., Computational Modeling for What-Is, What-Might-Be, and What-Should-Be Studies—and Triangulation. Organization Science, 22 (5), 1195-1202 (2011).

17. Jahangirian, M., Eldabi, T., Naseer, A., Stergioulas, L. K., Young, T., Simulation in Manufacturing and Business: A Review. European Journal of Operational Research, 203 (1), 1-13 (2010).

18. Posada, J., Toro, C., Barandiaran, I., Oyarzun, D., Stricker, D., de Amicis, R., Pinto, E.B., Eisert, P., Döllner, J. and Vallarino, I., Visual Computing as a Key Enabling Technology for Industrie 4.0 and Industrial Internet. IEEE Computer Graphics and Applications, 35 (2), 26-40 (2015).

19. Orady, E. A., Osman, T., Bailo, C. P., Virtual Reality Software for Robotics and Manufacturing Cell Simulation. Computers & Industrial Engineering, 33 (1-2), 87-90 (1997).

20. Gazebo, Why Gazebo?, http://gazebosim.org/, last accessed 2020/03/08.

21. ROS, Documentation, http://wiki.ros.org/, last accessed 2020/02/28.

22. Furrer, F., Burri, M., Achtelik, M., Siegwart, R., Rotors—a Modular Gazebo Mav Simulator Framework. In: Koubaa A. (eds) Robot Operating System (ROS), p. 595-625, Springer, Switzerland, (2016).

23. Zhong, Y., Shirinzadeh, B., Virtual Factory for Manufacturing Process Visualization. Complexity International, 12 (1), 1-22 (2008).

24. De Croon, G., De Wagter, C., Challenges of Autonomous Flight in Indoor Environments. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1003-1009, Madrid, Spain (2018).