# Applying Machine Learning for Adaptive Scheduling and Execution of Material Handling in Smart Production Logistics

Erik Flores-García, Yongkuk Jeong, Magnus Wiktorsson

► To cite this version:

Erik Flores-García, Yongkuk Jeong, Magnus Wiktorsson. Applying Machine Learning for Adaptive Scheduling and Execution of Material Handling in Smart Production Logistics. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.28-36, 10.1007/978-3-030-85914-5_4. hal-03897871

HAL Id: hal-03897871

https://inria.hal.science/hal-03897871

Submitted on 14 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Applying Machine Learning for Adaptive Scheduling and Execution of Material Handling in Smart Production Logistics

Erik Flores-García¹[0000-0003-0798-0753], Yongkuk Jeong¹[0000-0003-1878-773X], and Magnus Wiktorsson¹[0000-0001-7935-8811]

¹ Department of Sustainable Production Development, KTH Royal Institute of Technology, Södertälje 15136, Sweden
efs01@kth.se

**Abstract.** Combining Smart Production Logistics (SPL) and Machine Learning (ML) for adaptive scheduling and execution of material handling may be critical for enhancing manufacturing competitiveness. SPL and ML may help identify, adapt, and respond to scheduling changes originating from disturbances in and enhance the operational performance of material handling. However, the literature combining SPL and ML for material handling is scarce. Accordingly, the purpose of this study is to propose a framework applying ML for the dynamic scheduling and execution of material handling tasks in SPL. The study proposes an architecture including Cyber Physical System (CPS) and Internet of Things (IoT) applying ML for the dynamic scheduling and execution of material handling. Then, we describe the ML inputs, interactions, and work flow for realizing the proposed architecture. Finally, the study presents digital services in a simulation environment exemplifying the dynamic scheduling and execution of material handling in SPL. The study concludes with essential implications to the manufacturing industry.

**Keywords:** Smart Production Logistics, Adaptive Scheduling, Machine Learning.

## 1 Introduction

Combining Smart Production Logistics (SPL) and Machine Learning (ML) offers advantages essential for addressing unexpected events that arise during material handling and enhancing the competitiveness of manufacturing companies [1]. On the one hand, SPL involves applying digital technologies including Cyber Physical Systems (CPS) and Internet of Things (IoT) for making machines and material handling equipment smart [2]. Accordingly, SPL facilitates the perception, active response, and autonomous decision-making for the movement of materials and information within the boundaries of a factory [3]. Particularly, SPL is essential for identifying, adapting, and reducing costly mistakes originating from disturbances or machine breakdowns affecting material handling. On the other hand, ML is a subset of artificial intelligence focusing on
autonomous computer knowledge gain [4]. ML includes algorithms identifying and ex-
tracting valuable patterns in data, and allows computers to solve problems without ex-
plicit programming [5]. One area of increasing interest in SPL is applying ML for adapt-
ing the scheduling and execution of material handling as a response to unexpected
events including malfunctioning machines, changing priorities, illness of staff, delays,
defects, or shortage of materials [6].

ML provides distinct benefits essential for addressing unexpected disturbances and adapting scheduling and execution of material handling tasks. For example, ML provides fast acceptable solutions that conform to the operation of a dynamic environment [7]. The timeliness of response in ML is advantageous over current alternatives such as simulation approaches requiring intensive computational effort and lacking adaption to changes in the factory floor. Additionally, ML reduces the effort to acquire knowledge, identify a strategy, and adapt to changes of environments automatically [8]. Finally, ML can classify and predict responses based on knowledge learned from training examples [9].

Recent research efforts advance the use of ML in SPL, yet address scheduling and material handling separately [10]–[13]. These studies are critical for addressing unexpected events arising in material handling, yet the manufacturing industry may obtain only partial benefits when applying scheduling and execution of material handling independently. Instead, manufacturing companies require adaptive scheduling capable of identifying disturbances, adjusting changes instantly, and executing material handling.

Addressing this need, the purpose of this study is to propose a framework applying
ML for adaptive scheduling and execution of material handling in SPL. The main con-
tribution of this work includes a CPS architecture containing a physical, virtual, and an
adaptive control layer. We apply ML in the adaptive control layer for knowledge learn-
ing, and adaptive adjustment including scheduling and execution of material handling.
Additionally, the study includes the description of the phases in ML for achieving a
closed loop adaptive scheduling based and execution of material handling. Finally, the
study presents the visualization of a SPL system including the proposed framework in
a laboratory environment. The remainder of this paper includes the following sections.
Section 2 reviews related works about SPL in material handling and ML. Section 3
describes the proposed framework applying ML for adaptive scheduling and execution
of material handling. Section 4 presents the visualization of a SPL system. Section 5
draws the conclusions of this study and suggests future research.

## 2 Related Works

This section presents current understanding about SPL essential for addressing unex-
pected events in material handling. The section describes the importance of IoT devices
and CPS for adaptive scheduling and execution of material handling. Then, the section
outlines key elements of existing architectures of CPS in SPL. The section finalizes
identifying essential aspects for applying ML to existing IoT and CPS architectures.

CPS are critical for developing the vision of SPL including the acquisition, handling,
and use of real-time data facilitating autonomous decisions in the shop floor. CPS and
IoT involve multi-layered architectures that facilitate acquiring, handling, and using
real-time and historic information essential for adapting scheduling and executing ma-
terial handling as events occur. In relation to scheduling, Rossit et al. [14] propose a
decision-making schema for smart scheduling intended to yield flexible and efficient
production schedules on the fly. Feng et al. [15] suggest a dynamic scheduling system
based on CPS improving the energy efficiency of the job-shop. Jha et al. [16] present a
CPS for manufacturing processing in IIoT environments, in which physical actor re-
sources take part in the scheduling process. In relation to material handling, Zhang et
[2] al. investigate the mechanism of intelligent modeling and active response of manu-
facturing resources in the infrastructure layer; and present a self-organizing configura-
tion for SPL. Zhang et al. [17] design a CPS control model for material handling.

The literature on CPS and IoT underscores key elements of existing architectures that may be essential for adaptive scheduling and execution of material handling. For example, Guo et al. [6] propose a three layered approach of a SPL system including a physical, virtual, and control layers addressing self-adaptive collaborative control. The virtual layer involves IoT technologies, sensors and automatic identification sensors for acquiring and transmitting real-time information about physical devices in the literature terms a physical layer. The virtual layer comprehends synchronizing real-time information and dynamic behavior of physical devices, the encapsulation of information, and digital services for the use of information by staff. Finally, a control layer detects external and internal exceptions using real-time status information and a knowledge base. The control layer identifies and proposes strategies for addressing known exceptions. Additionally, the control layer analyzes new exceptions, extracts knowledge, and updates the database.

Recent advances suggest applying ML for adapting scheduling to unexpected events in combination with IoT and CPS [10]. Applying ML requires developing a knowledge base for simultaneously acquiring different information from the manufacturing system in real-time and selecting an appropriate strategy for responding to unexpected events [18]. There are four essential aspects for developing and using a knowledge base for the adaptive scheduling and execution of material handling. Firstly, acquiring information including system structure and capacity (static data) and manufacturing process, material handling and behavior (dynamic data). Secondly, processing information including the identification and classification of status of the manufacturing system and real-time and historical disturbances. Thirdly, knowledge learning involving the selection of a best response for each disturbance and evaluation of each response in relation to scheduling and execution of material handling. Fourthly, applying an adjusted scheduling strategy and the execution of material handling tasks.

3 Applying ML for Adaptive Scheduling and Execution of Material Handling in SPL

This section proposes a framework applying ML for the adaptive scheduling and exe-
cution of material handling in SPL. The framework presents a direct response to the
need of the manufacturing industry for addressing unexpected events arising in material
handling. Accordingly, the framework includes a CPS architecture together with an
adaptive control layer involving ML. Figure 1 presents the proposed SPL framework
applying ML for adaptive scheduling and execution of material handling.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Proposed SPL framework applying ML for adaptive scheduling and execution of material handling.</figcaption></figure>

The CPS architecture for adaptive scheduling and execution of material handling
includes a physical, virtual, and adaptive control layers. The physical layer comprès-
hends the real-world manufacturing system. This layer contains machines, operators,
products, forklifts, AGVs, pallets, and warehouses in the shop floor. Additionally, the
physical layer comprises IoT devices such as RFIDs and sensors, cameras, and net-
working devices connected to forklifts and AGVs. The activities in the physical layer
are two-fold. Firstly, the physical layer executes scheduled tasks in material handling.
Secondly, IoT devices collect, transmit, and receive real-time information from the
shop floor indispensable for applying ML for adaptive scheduling and execution of
material handling.

The virtual layer contains elements and activities for synchronizing material han-
dling in the shop floor with its virtual counterpart. The virtual layer contains three sets
of activities focusing on information encapsulation, handling, and use. Information en-
capsulation refers to the segmentation of information from each device of the manufac-
turing system including its attributes, structure, and resources. Information encapsula-
tion includes real-time and historical information involving resources, unexpected
events, schedule of tasks, and performance in material handling. For example, resource
data refers to type, time, position, distance travelled, and speed of an AGV. Information
about unexpected events includes time, obstacle, and AGV downtime, and scheduling information comprehends order number, location and time for picking and delivering orders. Information about performance of material handling includes throughput, on time delivery, makespan, and distance travelled by forklifts or AGVs. Information handling comprehends the processing and analysis of historical and real-time information. Importantly, information handling targets the calculation and interrelation of information about resources, unexpected events, schedules and performance in material handling. Consider the case of the parameter for on time delivery requiring the integration of information about AGV location, schedule, and time. Finally, information use concerns the configuration, monitoring, and execution of scheduled tasks in material handling. Digital services provide an interface between staff and the virtual manufacturing system and facilitate automatic decisions. Accordingly, digital services facilitate the execution of the schedule for material handling, and determine how and when material handling resources carry out tasks in the shop floor.

ML is the core element of the adaptive control layer. Accordingly, real-time acquisition of information and a knowledge base detect and determine the impact of unexpected events. Figure 2 exemplifies addressing unexpected events with the proposed SPL framework applying ML. The knowledge base contains responses to unexpected events that occurred in the past. Additionally, the processing of real-time information detects new unexpected events and renews the knowledge base. The knowledge base presents an online decision adjusting the scheduling strategy based on the goals of the adaptive scheduling that maintain the performance of SPL. Then, the adaptive control layer transfers an adaptive solution including optimal schedule to the virtual layer. Finally, material handling resources execute the optimal schedule in the physical layer. The proposed framework applies ML in four phases for achieving a closed loop adaptive scheduling based and execution of material handling based on prior findings [10].

The four phases for applying ML in the adaptive control layer include acquiring information, identifying disturbances, adapting a scheduling strategy, and generating a schedule. Information acquisition requires two steps. Firstly, transferring information from the physical layer of a CPS to the adaptive control layer. Secondly, examining and selecting information critical for adaptive scheduling and execution of material handling is essential.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Addressing unexpected events with the proposed SPL framework applying ML.</figcaption></figure>

The identification of disturbances in the adaptive control layer involves three aspects. Firstly, determining the type of disturbance; secondly, recognizing the effects of a disturbance; and thirdly, establishing a response. Establishing a response involves
analyzing dynamically and defining the limits of operational performance. Accordingly, adjustments to a schedule respond to maintaining or improving a level of operational performance.

The adapting scheduling strategy is an output of the knowledge base. Therefore, the scheduling strategy requires gathering knowledge learned from the manufacturing process including historical and real-time information. Additionally, the knowledge base provides real-time information consisting of updates to the scheduling strategy with the purpose of achieving a desired level of operational performance. Finally, the adaptive control layer presents an adaptive schedule of its execution in material handling. The adaptive schedule depends on the effects of disturbances on the operational performance of the manufacturing system. Additionally, the adaptive schedule presents information including material handling tasks, location and time for pickup and delivery, and the resource assigned to the execution of a task to the virtual layer.

# 4 Prototype Applications in SPL Test-Bed Environment

This section describes a prototype application for testing the proposed framework of this study. This prototype application includes a SPL test-bed environment, and a variety of robots and IoT devices. An internal network connects all robots and devices, and facilitate the transmission of material handling data including resource type, location, distance, speed, and status in real time.

A database stores real-time data used for the learning of the ML models and visualizing the execution of material handling by the AGV in the virtual world. Figure 3 shows the execution of material handling by the AGV in the SPL test-bed environment. Figure 3 (a) presents how the AGV finds the route in the as-is situation. In this case, the AGV receives the mission to deliver material from the planning system and performs the given mission according to the order. The AGV utilizes a radar sensor and pathfinding algorithm for identifying nearby obstacles, and determining the path during the execution of material handling. However, as shown in Figure 3 (a) the AGV cannot react in real-time to events that occur during the execution of material handling. Upon encountering and unexpected event, the AGV will stop the execution of material handling, and recalculate the optimal path to its destination.

However, adaptive and predictive responses are possible by applying the framework proposed in this study. First, Figure 3 (b) shows an adaptive response method when an unexpected event occurs while the AGV is moving. In this instance, the AGV can execute ML-based routing by combining real-time and historical data in the ML model. The ML model provides two advantages originating from its knowledge base and training including faster response than optimal path finding algorithms and alternatives to unexpected situations. Therefore, the AGV reacts to occurrences in real-time and does not require stopping and recalculating during the execution of material handling.

Next, Figure 3 (c) shows the predictive response to a wide planning horizon in material handling. Traditionally, the AGV executes multiple missions sequentially. However, ML-based scheduling calculates in advance the total material handling time of all
missions including different scheduling scenarios. Therefore, the proposed framework
may contribute to real-time routing and a wide planning horizon in material handling.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Example of AGV with logistics tasks in SPL test-bed environment</figcaption></figure>

ML-based real-time routing can be implemented through the framework we propose in this paper. Currently, AGV scans its surroundings and identifies obstacles before staring the delivery task, and the shortest is calculated based on this. If there is any obstacle in the path that was not recognized from the beginning, the operation is temporarily stopped, and the optimal path is recalculated as shown in Fig. 4 (a). This problem can be improved through online adaptive route planning by the framework prosed in this paper. The reinforcement learning (RL) algorithm can be used to respond to the dynamic situation of the SFL environment. Unlike any other ML models such as supervised and unsupervised learning models, the RL algorithm does not requires labeled or unlabeled data. But it trains the model by rewards when the agent performs an action appropriate for its purpose. The trained model can make the right decision by itself, not only in a given environment but also in a completely new environment. Fig. 4 (b) represents this situation, and the continuous revised path can be calculated by periodically calling the trained RL model. Since the RL model has al-ready been trained, it can immediately calculate the revised path even for dynamic obstacles.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Comparison between offline route planning and online adaptive route planning</figcaption></figure>

5 Conclusions

The purpose of this study was to propose a framework applying ML for the adaptive scheduling and execution of material handling in SPL. The main contribution of this work included a CPS architecture containing a physical, virtual, and an adaptive control layer. We applied ML in the adaptive control layer for knowledge learning, and adaptive adjustment including scheduling and execution of material handling. The study presented the visualization of a SPL system including the proposed framework in a laboratory environment. Future research will focus on verifying the proposed framework in an industrial case. Additionally, future studies will involve additional tasks in including packaging, transportation, or warehousing.

Acknowledgments

The authors would like to acknowledge the support of the Swedish Innovation Agency (VINNOVA). This study is part of the Cyber Physical Assembly and Logistics Systems in Global Supply Chains (C-PALS) project. The SMART EUREKA CLUSTER on Advanced Manufacturing program funded this project.
References

1. Woschank, M., Rauch, E. & Zsifkovits, H. A Review of Further Directions for Artificial Intelligence, Machine Learning, and Deep Learning in Smart Logistics. Sustainability 12, 3760 (2020).

2. Zhang, Y., Guo, Z., Lv, J. & Liu, Y. A Framework for Smart Production-Logistics Systems Based on CPS and Industrial IoT. IEEE Transactions on Industrial Informatics 14, 4019–4032 (2018).

3. Wang, W., Zhang, Y. & Zhong, R. Y. A proactive material handling method for CPS enabled shop-floor. Robotics and Computer-Integrated Manufacturing 61, 101849 (2020).

4. Sharp, M., Ak, R. & Hedberg, T. A survey of the advancing use and development of ma-
chine learning in smart manufacturing. Journal of Manufacturing Systems 48, 170–179
(2018).

5. Wuest, T., Weimer, D., Irgens, C. & Thoben, K.-D. Machine learning in manufacturing: advantages, challenges, and applications. Production & Manufacturing Research 4, 23–45 (2016).

6. Guo, Z., Zhang, Y., Zhao, X. & Song, X. CPS-Based Self-Adaptive Collaborative Control for Smart Production-Logistics Systems. IEEE Transactions on Cybernetics 51, 188–198 (2021).

7. Shiue, Y.-R., Guh, R. & Lee, K. Development of machine learning-based real time schedul-
ing systems: using ensemble based on wrapper feature selection approach. International
Journal of Production Research 50, 5887–5905 (2012).

8. Wang, H., Sarker, B. R., Li, J. & Li, J. Adaptive scheduling for assembly job shop with uncertain assembly times based on dual Q-learning. International Journal of Production Research 1–17 (2020).

9. Shiue, Y.-R., Lee, K.-C. & Su, C.-T. Real-time scheduling for a smart factory using a rein-
forcement learning approach. Computers & Industrial Engineering 125, 604–614 (2018).

10. Qiao, F., Liu, J. & Ma, Y. Industrial big-data-driven and CPS-based adaptive production scheduling for smart manufacturing. International Journal of Production Research 1–21 (2020).

11. Mourtzis, D. & Vlachou, E. A cloud-based cyber-physical system for adaptive shop-floor scheduling and condition-based maintenance. Journal of Manufacturing Systems 47, 179–198 (2018).

12. Umar, U. A., Ariffin, M. K. A., Ismail, N. & Tang, S. H. Hybrid multi objective genetic algorithms for integrated dynamic scheduling and routing of jobs and automated-guided vehicle (AGV) in flexible manufacturing systems (FMS) environment. The International Journal of Advanced Manufacturing Technology 81, 2123–2141 (2015).

13. Hu, H., Jia, X., He, Q., Fu, S. & Liu, K. Deep reinforcement learning based AGVs real-time scheduling with mixed rule for flexible shop floor in industry 4.0. Computers & Indus-trial Engineering 149, 106749 (2020).

14. Rossit, D. A., Tohmé, F. & Frutos, M. Industry 4.0: Smart Scheduling. International Journal of Production Research 57, 3802–3813 (2019).

15. Feng, Y., Wang, Q., Gao, Y., Cheng, J. & Tan, J. Energy-Efficient Job-Shop Dynamic Scheduling System Based on the Cyber-Physical Energy-Monitoring System. IEEE Access 6, 52238–52247 (2018).

16. Jha, S. B., Babiceanu, R. F. & Seker, R. Formal modeling of cyber-physical resource sched-
uling in IIoT cloud environments. Journal of Intelligent Manufacturing 31, 1149–1164
(2020).
17. Zhang, Y., Zhu, Z. & Lv, J. CPS-Based Smart Control Model for Shopfloor Material Handling. IEEE Transactions on Industrial Informatics 14, 1764–1775 (2018).

18. Qu, T. et al. IoT-based real-time production logistics synchronization system under smart cloud manufacturing. The International Journal of Advanced Manufacturing Technology 84, 147–164 (2016).

19. Priore, P., Gómez, A., Pino, R. & Rosillo, R. Dynamic scheduling of manufacturing systems using machine learning: An updated review. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 28, 83–97 (2014).