# Optimization of Production-Oriented Logistics Processes Through Camera-Based Identification and Localization for Cyber-Physical Systems

Marcus Lewin, Helmut Weber, Alexander Fay

► To cite this version:

Marcus Lewin, Helmut Weber, Alexander Fay. Optimization of Production-Oriented Logistics Processes Through Camera-Based Identification and Localization for Cyber-Physical Systems. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2017, Hamburg, Germany. pp.168-176, 10.1007/978-3-319-66923-6_20. hal-01666182

HAL Id: hal-01666182

https://inria.hal.science/hal-01666182

Submitted on 18 Dec 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Optimization of production-oriented logistics processes through camera-based identification and localization for Cyber-physical Systems

Marcus Lewin, Helmut Weber, Alexander Fay

Helmut Schmidt University / University of the Federal Armed Forces Hamburg, Institute of Automation Technology, Holstenhofweg 85, 22043 Hamburg (Germany)
lewinm@hsu-hh.de, weberh@hsu-hh.de, alexander.fay@hsu-hh.de

**Abstract.** The use of sensor data as well as the combination of different data from diverse systems in production and logistics lead to new opportunities for monitoring, controlling and optimization of processes within the scope of Industry 4.0. New developments of camera-based systems support this trend, which is particularly relevant for the control of cyber physical systems (CPS). This paper discusses a new approach for camera-based dynamic identification and localization, including speed and orientation determination, in combination with joined data from different sources and data analysis for CPS. In order to assess the potential of the approach, various possibilities and methods for camera-based optimizing CPS are discussed. A production-oriented logistics application shows the technical feasibility of the approach.

**Keywords:** camera, camera-based sensor, combining data, logistics, Cyber-physical Systems, optimization, visualization, Industry 4.0

## 1 Introduction

Increasing customer requirements and individualized products lead to a growth of complexity in production processes and the amount logistics processes due to distributed production sites and higher complex delivery networks [1]. To meet these higher requirements, a new level of organization and control of the entire value chain over the lifecycle of products is necessary [2]. This is summarized under the term Industry 4.0 based on Cyber-Physical Systems (CPS). CPS are embedded open-ended socio-technical systems and enable a range of new functions, services and features for production, logistics, engineering, coordination and management processes as well as internet services [3]. An essential basis is the availability of all relevant information in real time through directly captured physical data by sensors and networking of entities involved in the value creation as well as the ability to derive optimal value-added flows from data obtained [2].

Industry 4.0 contains vertical integration and networking of production systems [4]. These aspects include the areas of sensor networks as well as intelligence, flexibility
and changeability of systems through comprehensive solutions. In particular, sensor
networks provide a new way of process understanding by permanently collecting and
analyzing sensor data from production and logistics. [5] This includes further develop-
ment of camera-based sensor technology to intelligent, configurable and networked
sensor systems with case-specific and adaptive measurement strategies [6]. After an
overview of related work, we show an approach using camera-based sensors in combi-
nation with joined data from different sources, data analysis and visualizations to ana-
lyze and control CPS to improve production-oriented logistics. An implementation in a
modular production system and opportunities of the approach are discussed subse-
quently.

## 2 Developments in sensor technology and related work

Various methods for obtaining production and material flow data, such as manual data
collection by employees, applications of auto-ID techniques (bar code or RFID tech-
niques), subsequent analysis of film recordings, analysis of production documents and
statistics or data collection from ERP systems are available [7]. Through the continuous
development of sensor technologies, new and more effective methods for obtaining
these data could be developed [8].

Camera-based data collection systems can provide a useful complement to existing
data collection methods. This increases the availability of distributed information from
a wide range of sources, which also have to be analyzed in real time and can be used
for system control. Providing information for self-control becomes increasingly im-
portant. New camera models for camera-based systems have the capability to build the
function of different sensors and to generate virtual sensor data in the desired quality.
This allows an analysis of the effects for sensor-specific and process-specific parame-
ters on situation analysis [9].

Concerning the use of camera-based sensors in logistics, the number of applications
and research projects is limited [10, 11]. Computer vision is used as a method for real-
izing and replacing different sensors on forklifts trucks with just one technology at the
project "Staplerauge" [12]. The replacement or supplementation for various sensors
such as localization of hoist height determination and fork-occupied detection leads to
decreasing integration costs on forklifts trucks and background IT-systems. In practice
the technical feasibility and a high potential of the vision-based state monitoring for
forklifts has been proved [13].

A different approach pursue the marker-based location of resources at the project
MarLO [8]. Fixed infrastructure is equipped with appropriate video sensors to detect
optical markers on different mobile logistics resources like forklift trucks. Marked ob-
jects are tracked from the outside and the acquisition and calculation systems are firmly
connected with the sensor processing units. This system is advantageous for spatially
restricted application environments as well as for free surfaces, in which numerous mo-
bile operating means are used.

At the project tracing intelligent logistics objects (TiLo) the feasibility of data acqui-
sition and analysis by the industrial image processing and the use of simulation studies
are demonstrated [14]. The used camera covers only individual areas and the obtained data is only used for simulations. There is also no connection of the camera data with production and sensor data.

The presented projects often show only individual elements of Industry 4.0. A comprehensive connection of camera-based data with further production data analysis and evaluation in combination with visualization in real time and for simulation is not achieved in the production environment.

# 3 Technical concept

In addition to the necessary basic conditions, the technical concept includes four stages. First, the type of identifiers as well as the type of identification and localization methods are defined. An interface to existing production databases is created subsequently. In a third step the camera-based data and the data from the existing systems have to be prepared and evaluated in order to make decisions and to support workers through appropriate visualizations in a final step. Such virtual sensors are a step ahead to reach Industry 4.0 goals. Figure 1 shows the complete process.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Complete process of the approach</figcaption></figure>

## 3.1 Requirements and necessary conditions

The use of camera-based sensors needs various minimum requirements and conditions, which must be taken in order to achieve a secure identification and localization of objects in a required quality. For locating different equipment by camera-based systems, two essential technical approaches are possible. Either the mobile object itself captures its surroundings. For this purpose, cameras are installed directly on the objects and markers are placed firmly in the environment. Or the mobile object is captured, realized by cameras attached in the infrastructure and markers mounted on the objects.

The size of the covered area is to be taken into consideration. Depending on the design, several camera-based sensors have to be used for larger areas. This leads to an increase in system complexity. Furthermore, the type of the relevant objects have to be taken into account. In addition to natural identifiers, artificial identifiers can be used too. The size of the objects observed as well as the distance leads to minimum requirements for the resolution of camera-based sensors. Influence on the minimum sampling rate of the sensors has among other things the speed of the objects. The light conditions on the spot have an influence on the quality of the object detection. These factors influence the exact design of the camera-based sensors.
## 3.2 Image Processing and Localization Algorithm

Because artificial identifiers often lead to a more stable detection of objects, further matrix codes are discussed. ArUco markers are synthetic square marker composed by a black border and an inner binary matrix containing a number as an ID. The ID and the direction and size of these markers is easy to detect with software libraries to analyze camera images, like the open source computer vision OpenCV. For the recognition of the markers, they need at least the space of about 30x30 pixels of the camera image.

The resulting data of the recognition – ID, Coordinates of the edges – are used to compute more derived information: direction and speed of the objects, distances to other objects etc. These data may be seen as received from virtual sensors and are saved for further investigation. The object itself does not need any sensors or wireless communication facilities, but if it is able to communicate, it could be provided with all these data concerning itself or all other marked objects. The data seems to come from a subset of sensor built in – virtual sensors. To reach the goals of Industry 4.0, these data may be distributed to the different elements of the production in real time.

## 3.3 Connection and processing of data from different sources

A software interface to relevant production database has to be implemented to read the state of all involved elements to combine these data with the data of marker recognition and derived data together with the timestamp to build a database of the production process containing all relevant information. This data could be used to monitor the production process just in time or later in simulations. Chapter 5 discusses these possibilities.

## 3.4 Visualization and worker support

These compacted data from different sources are used to visualize the production process in a different view like augmented vision or a virtual picture of a system. The augmented vision combines the real camera image with additional superimpositions. Thus provides a detailed insight into systems along with necessary added information. The camera images are not necessary needed any longer for the virtual view – may be stored for some time for controlling purposes – and the compact sensor may be transferred to any place to see the production in real time or later. The commented and with all information added data can be used to monitor and control the production. Based on this added data new decisions could be made to optimize production processes.

# 4 Demonstrator and implementation on an CPS

The presented concept is implemented in a modular production system shown in figure 2. This CPS produces individualized valves in small batches up to lot size 1. A work piece carrier equipped with a RFID tag contains orders, work plans and work pieces. The production line consists of six connection points for various production modules and a conveyor belt circuit. To realize an improved material flow control, the system
has two switch points to skip stations. Each module is decentral controlled via its own PLC. Orders and certain production steps are also documented centrally in a MES with appropriate databases. In the first module, three different basic bodies are provided from stock. The second module performs several machining operations, such as drilling. Quality control is performed in the third module. A robot assembles individual components of a valve in the fourth module, which can subsequently be stored in the fifth module. Module six commissions and deliveries finished orders to customers.

Based on this system, various hardware changes were performed based on the process shown in Figure 1. This includes in a first step the placement of ArUco markers on the work piece carriers for identification and localization. A camera was installed centrally above the modular production system and provides images of the whole system. The analysis software, located on an external PC, recognizes the parameters of all markers inside the image and continuously calculates the position of all carriers almost in real time, shown in figure 2. The parameters of the markers (ID, position of the center and the position of the corners of the markers) are stored in a database table together with time stamps. The alignment, the speed and the direction of movement of the markers can be calculated from these image sequences. This design of the camera system is easy to implement with minimal interferences of the production and logistics processes.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Augmented camera picture</figcaption></figure>

Secondly, the production databases of the MES were connected to the software through a two-way interface. During the data connection and analysis, the tables of the production database are constantly read out. These data are combined with the data from the position calculation of the markers. According to this, in a third step the system derives different evaluations and results. In the example shown in figure 3, eight carriers are used. Two carriers (3 and 5) being filled with a basic cylinder. Carrier 3 moves to the station "Camera" for a quality check. Carrier 5 transported its basic cylinder to the station "Driller" and waits for the completion of the drilling process.

The detailed view of production and logistics processes leads to abilities for self-optimization, self-configuration and self-diagnosis, which are essential for Industry 4.0. Standstills of the conveyor belt can be detected immediately. The blockage of the belt by a jammed carrier also between the stations triggers an alarm message. In addition, the evaluation determines the service life of a carrier at a station, whether the processes run stably or require more time than usual. The position of the individual carriers is determined continuously and independently from sensors of the individual modules.
This allows a more precise localization of the carriers and a higher transparency of logistics processes. The position and the type of parts on the carriers can also be determined precisely. The software identifies the markers of the loaded parts and evaluates the images in combination with the data from the MES. Thereby a plausibility check of the work pieces (presence, color and diameter) is possible. Different lengths of processing steps on individual modules can cause queues, which cannot be adequately regulated by the decentralized control, without an image of the entire system.

In addition to a real time evaluation, the data can be used for later simulations too. Without intervention in the plant, the influence of different parameters and strategies on the production can be tested. This includes changes in the belt speed, changes in the duration of individual production steps or influence of the distance of the stations. Layout decisions and optimizations can also be determined.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Virtual view of the system with camera-based identification and analytics</figcaption></figure>

Several views are implemented for the visualization of the system within the fourth process step. One view shows the camera image of the plant with augmented information of identified carriers and important parameters from the production database. Figure 3 shows the virtualized representation of the plant with the current status of the orders, work pieces and work piece carriers. This view gives a good overview of the production and logistics processes including messages and warnings.

# 5 Optimization methods and opportunities of the approach

The presented concept leads to a shift of hardware functions to software functions via software modules. These software modules are the basis for higher flexibility and changeability for CPS and in production-oriented logistics.
The presented camera-based systems have the advantage that they do not need a direct connection to the existing controls of the systems. Therefore, they collect information independently from existing automation and control data of production plants. Changes of the camera and evaluation software can be made during production. This leads to minimal disturbances in production and logistics. In this way, camera-based systems can collect data on details that cannot be captured or can only be captured at great expense by installed sensors in existing production facilities.

The continuous connection of identification and localization with different production data can be used to monitor a production. Already in a simple structure, an augmented representation of the current production is possible over the network or internet. Standstills of transport systems, such as conveyor belts, can be detected immediately and an alarm message can be triggered. Processing and handling times of production stations can be determined and evaluated, too. Without direct encroachment on plants, data for simulations can be collected. The data can be used to test the influence of different parameters and strategies in production and logistics. Thus leads to an evaluation and optimization of individual production stations, identification of bottlenecks and to improved layout decision for production networks. In addition to the presented possibilities, there are numerous other possibilities for optimization for CPS in Industry 4.0:

* Providing high-resolution state data and relevant information to entities and systems in real time through directly captured physical data and data analysis for an adaptive production control

* Plausibility comparison with the signals from other sensors, e.g. combination of artificial with natural identification features or production data

* Replacement and extension of various existing sensors

* Continuous monitoring and Real time control of processes

* Increasing the transparency and system understanding of the production and logistics facilities by recording and analyzing services and loads

* Support for workers through clear visualization and decision-making

* Later made simulations based on the data could help to discover and eliminate weaknesses and errors and to optimize production

# 6 Conclusion and Outlook

In this paper, various possibilities for the use of camera-based sensor data in production and logistics were presented. Camera-based sensor technologies become more effective, more powerful and more cost-effective, and will be used largely in logistics and production systems. By combining the resolved sensor data with other data from the production and logistics, CPS control can be improved and optimized. The presented methods of visualization provide a good basis for system monitoring and decision support of workers. Therefore, combining of different data resources leads to an improvement and optimization of processes. The technical feasibility was demonstrated by an implementation in a modular system with production-related logistics processes.

In the further work, additional implementations for evaluation will be developed. This will includes extended use and combination of different data sources as well as
new methods for visualization of data and providing relevant information for CPS in Industry 4.0.

# 7 References

1. Bauernhansl, T.: Die Vierte Industrielle Revolution - Der Weg in ein wertschaffendes Produktionsparadigma. In: Vogel-Heuser, B., Bauernhansl, T., ten Hompel, M.: Handbuch Industrie 4.0 Bd. 4. Springer Berlin Heidelberg. pp. 1-31. (2017).

2. Dorst, W., Glohr, C., Hahn, T. (2016). Implementation Strategy Industrie 4.0. Report on the results of the Industrie 4.0 Platform. Berlin, Frankfurt am Main: BITKOM, VDMA, ZVEI.

3. Kagermann, H., Helbig, J., Hellinger, A., Wahlster, W. (2013). Recommendations for Implementing the strategic initiative INDUSTRIE 4.0: securing the future of German manufacturing industry; final report of the Industrie 4.0 working group. Forschungsunion.

4. Adolphs, P., Bedenbender, H., Dirzus, D., Ehlich, M., Epple, U., Hankel, M., ..., Koziolek, H. (2015). Status Report-Reference Architecture Model Industrie 4.0 (RAMI4. 0). VDI ZVEI.

5. Bauernhansl, T., Diegner, B., Diemer, J., Dümmler, M., Eckert, C., Herfs, W., ..., Kubach, U. (2015). Industrie 4.0-Whitepaper FuE-Themen. Plattform Industrie 4.0.

6. Bischoff, J., Hegmanns, T., Prasse, C., Hompel, M., Henke, M., Wolter, D., ..., Guth, M. (2015). Erschließen der Potenziale der Anwendung von Industrie 4.0 im Mittelstand. agiplan GmbH, Mühlheim an der Ruhr.

7. Bohács, G., Semrau, K. F. (2012) Automatische visuelle Datensammlung aus Materialflusssystemen und ihre Anwendung in Simulationsmodellen. In: Logistics Journal: Not reviewed. Vol. 2012, Iss. 01. pp. 1-7 (2012).

8. Richter, K., Poenicke, O., Kirch, M., Nykolaychuk, M.: Logistiksysteme. In: Schenk, M.(ed.): Produktion und Logistik mit Zukunft. Springer Berlin Heidelberg, pp. 245–281. (2015).

9. Nationaler IT-Gipfel Berlin 2015 Plattform „Digitale Netze und Mobilität (Eds.): Echtzeitdaten für die Logistik. (2015).

10. Özgür, Ç., Alias, C., Noche, B.: Comparing sensor-based and camera-based approaches to recognizing the occupancy status of the load handling device of forklift trucks. In: Logistics Journal: Proceedings (05), 2016, pp. 1–9. (2016).

11. Borstell, H., Plate, C.: Bildbasierte Situationsanalyse unterstützt Logistikprozesse. In: Schenk, M.(ed.): Jahresbericht 2014 des Fraunhofer-Instituts für Fabrikbetrieb und -automatisierung IFF, Magdeburg. Magdeburg, pp. 28–29, (2014).

12. Jung, M., Hohenstein, F., Günthner, W.: "Staplerauge": A Framework for Camera-Based Sensor Functions on Forklift Trucks. In: Clausen, U., Hompel, M., Meier, F. (eds.): Efficiency and innovation in logistics: Proceedings of the International Logistics Science Conference (ILSC) 2013. Springer International Publishing. pp. 77–88. (2014).

13. Hohenstein, F.; Jung, M.; Günthner, W. A.: „Das Staplerauge“ - Vision und Wirklichkeit. In: Hebezeuge Fördermittel, vol. 54 (4), 2014, pp. 188–191. (2014).

14. Koç, E., Alias, C., Kalkan, Y., Noche, B., Stange, C.: Nutzung von Realdaten in Simulationsmodellen durch industrielle Bildverarbeitung. In: Dangelmaier, W., Laroque, C., Klaas, A. (eds.): Simulation in Produktion und Logistik 2013: 15. ASIM-Fachtagung Simulation in Produktion und Logistik. pp. 349-360 (2013).