42 43 Multi -agent Reinforcement Learning (RL) frameworks for job scheduling and navigation control of 44 autonomous mobile robots are becoming increasingly common as a means of increa sing productivity in 45 manufacturing. Centralized and decentralized frameworks have emerged as the two dominant archetypes 46 for these systems. However, the tradeoffs of these competing archetypes in terms of efficiency , stability, 47 robustness , accuracy, generalizability, and scalability are not well -understood . This work investigates the 48 time efficiency , learning stability , and robustness to operational disruptions of an exemplar decentralized 49 RL framework in comparison to a centralized RL framework. Specifically, several policies with increasing 50 computational budgets are trained using both frameworks and then evaluated on the throughput and safety 51 of the shop floor in static and dynamic tests . We observe that the decentralized framework yields a high 52 performing policy at a significantly lower training budget than the centralized one . However, the centralized 53 framework exhibits superior learning stability as well as robustness to the initialization of robot locations in 54 static testing. Furthermore, we compare the robustness of the frameworks in dynamic tests to find that the 55 decentralized framework provides better compensation for processing delays and failures in real time. 56 57 Keywords: Artificial intelligence, Machine learning, Multi -agent reinforcement learning , Manufacturing 58 systems, Intelligent manufacturing, Autonomous mobile robots, Job scheduling, Robotics 59 60 3 1. INTRODUCTION 61 62 Industry 4.0 is an information -intensive transformation in manufacturing, powered by 63 advanced technologies such as the internet of things, robotics, and artificial intelligence 64 [1–5], with the goal of enhanc ing productivity, customization, and safety [6]. Aligned with 65 the Industry 4.0 design principles of decentralization and real -time capability [7], there is 66 an increasing interest in adopting Autonomous Mobile Robots (AMRs) controlled by 67 multi -agent deep Reinforcement Learning (RL) algorithms [8–12] in shop floors . However, 68 the tradeoffs of these decentralized frameworks , in terms of efficiency , stability , 69 robustness , accuracy, generalizability, and scalability, as compared to competing 70 centralized frameworks, are not well -understood . To address this gap, t his work 71 investigates the time efficiency , learning stability , and robustness to operational 72 disruptions of an exemplar decentralized RL framework in comparison to a centralized RL 73 framework. 74 Intelligent manufacturing system s are composed of multiple interacting autonomous 75 agents involved in tasks such as enterprise -level collaboration, process planning, job 76 scheduling, and shop floor control [13–15]. When properly designed, such m ulti-agent 77 systems can provide benefit s like increased flexibility, a higher degree of modularity, 78 better reconfigurability, and more robust adaptability . However , they face distinct 79 challenges as well, including agent organization, agent coordination, and negotiation . 80 Deep RL can be used to overcome several of these challenges , primarily due to the 81 potential to learn coordinated behavior between agents , without an engineer having to 82 encode specific desirable system behaviors [11,16 –21]. 83 4 The agents that make up a multi -agent intelligent manufacturing system can represent 84 a variety of entities, including decision -making software, human operators, and 85 autonomous machines. In a job scheduling task, which is the focus in this work, these 86 entities are tasked with collaborating to complet e a variety of parallel and sequential jobs 87 through effective shop floor control. In doing so they must contend with limited resources 88 and timing constraints. Competing approaches for accomplishing job scheduling can 89 generally be categorized as centralized or decentralized frameworks [11,22,23] . A 90 centralized framework holds the advantage of possessing comprehensive system 91 information, enabling a holistic approach to decision -making . However, it can lead to high 92 computational complexity and is prone to deadlocks triggered by a single point of failure. 93 In contrast, a well -designed decentralized framework can be more scalable and enable 94 better local decision -making capabilities in response to d ynamic changes in the task 95 environment. Such changes are common in manufacturing environments and may include 96 rush jobs, processing time delays, and even failures like machine breakdowns and robot 97 malfunctions . 98 This work investigates the efficiency, learning stability, and robustness of a 99 decentralized RL framework for an autonomous mobile robot driven shop floor in 100 comparison to a centralized framework . The rest of the paper is organized as follows. In 101 Section 2, we discuss relevant background and prior literature . In Section 3, we review 102 the specific multi -agent system from a prior work t hat serves as a testbed for comparing 103 decentralized and centralized RL frameworks for mobile robot driven shop floors . This is 104 followed by t he methodology for conducting the comparison. Section 4 details the results 105 5 of the decentralized versus centralized frameworks' performance on time efficiency , 106 learning stability , and robustness to robot placement and dynamic challenges like delays 107 and failures. Section 5 summarizes the contributions of the work and proposes future 108 directions. 109 110 2. BACKGROUND 111 112 2.1 Intelligent Manufacturing with Autonomous Mobile Robots 113 114 Intelligent manufacturing is actively evolving through the integration of c yber - 115 physical agents [24–27] like autonomous mobile robots [8–12] on the shop floor. T he 116 cyber portion of these robots consists of software entities that handle perception , action , 117 and communication to undertake intelligent real-time decision -making [28]. The mobile 118 robots are often tasked with transporting material across machines that process it to 119 finished or semi -finished part s. Human workers perform monitoring on the shop floor by 120 report ing anomalies and doing quality checks on the machines . This multi -agent system 121 work s towards meeting various shop floor objectives involving throughput, equipment 122 utilization, energy consumption, maintenance cost, and safety. The various characteristics 123 of such an autonomous mobile robot driven shop floor and the associated Industry 4.0 124 principles are illustrated in Figure 1 and discussed hereinafter. 125 6 126 FIGURE 1: CHARACTERISTICS OF AUTONOMOUS MOBILE ROBOT DRIVEN SHOP 127 FLOORS 128 Autonomous mobile robots , embodying the Industry 4.0 principle s of 129 decentralization and real time capability [7], can dynamically plan their path based on 130 sensor y information from the shop floor as well as communicated information from other 131 agents . This enables them to undertake decentralized material handling and job 132 scheduling at various machines that process this material. With their physical ability to 133 navigate and handle material freely and cyber ability to make decisions independently, 134 dynamic changes like machine processing delays and rush jobs , and failures like machine 135 breakdowns and robot malfunctions are handled more robustly than traditional 136 7 manufacturing systems. This reduces congestion and material queues ultimately lead ing 137 to an improvement in the throughput of the shop floor . Further , all cyber -physical assets 138 communicate with each other through standard protocols. This facilitates interoperability 139 [7] across systems from different manufacturers, a fundamental principle of Industry 4.0 140 design. 141 The successful integration of human workers with cyber -physical systems also 142 plays a vital role in the advancement of intelligent manufacturing [29–31]. Humans can 143 work alongside robots, monitoring shop floor activities and reporting any irregularities, 144 thus enhancing the system's real -time capability [7]. Ensuring the safety of human 145 workers in close proximity to robots is a critical objective for such shop floors . Mobile 146 robots should adapt their behavior to compensate for the stochastic nature of human 147 workers' activity in its proximity . The integrated job scheduling and navigation control of 148 mobile robots to meet the objectives of throughput and safety in this context is a 149 challenging problem in intelligent manufacturing systems. 150 2.2 Reinforcement Learning for Intelligent Manufacturing 151 152 Reinforcement learning (RL) [32–36] is a machine learning approach where an 153 agent or multi -agent system learns to make decisions in an environment by trial -and- 154 error, aiming to maximize long -term rewards. It is applicable to problems that can be 155 modeled as Markov decision processes (MDP) [34]. Further, RL can be classified into 156 single -agent RL or multi -agent RL. A single -agent RL framework can involve either a single 157 entity, or multiple entities sending their perceptions and receiving actions from a 158 centralized agent. In the latter case, the centralized agent essentially integrates all the 159 8 entities as a single combined agent [37]. A single -agent RL framework for a system with 160 multiple entities can provide a strong global optimization capability as the decisions of 161 the agent are based on system -wide information. However, such frameworks can pose 162 challenges involving computational expense and can be more sensitive to unanticipated 163 behavior in stochastic systems [38]. Multi -agent RL frameworks [18,20] , consisting of 164 multiple homogeneous interacting agents coexisting in an environment, can learn 165 different regions of a high -dimensional policy space and share their experience to reduce 166 memory and computational cost. Moreover, they are more tolerant to failures as the 167 agents can compensate for each other’s failures. 168 The use of RL is transforming several operational paradigms of manufacturing, 169 including order selection, process planning, job scheduling, robotic manipulation, and 170 motion planning [16,21] . This work focusses on the application of RL for job scheduling 171 and navigation control of autonomous mobile robot s. Within this context, the efficiency, 172 stability, robustness, accuracy, generalizability, and scalability of the RL framework is vital 173 to advanc e its applicability to real -world scenarios [39–42]. Efficiency is characterized by 174 aspects like time efficiency ( speed of learning ), sample efficiency (training samples 175 needed ) and resource utilization (computing systems needed) [40,42,43] . Stability of the 176 learning algorithm entails the consistency of the algorithm's performance as it learns over 177 time, ensuring that overall improvements in decision -making do es not introduce variance 178 in the process [41,42] . Robustness [39–41] in the algorithms ensures resilience against 179 diverse shop floor states as well as dynamic changes or failures , allowing robots to re - 180 route and re-schedule tasks without significant delays [8]. Accuracy, particularly in 181 9 navigation, is essential to avoid collisions and guarantee materials are delivered safely . 182 The generalizability [39] of the algorithms is its ability to cater across various shop floor s 183 layouts, machine configurations, and job requirements without extensive retraining [44]. 184 Lastly, scalability [40] ensures that the algorithm can handle an increasing number of 185 robots and machines as manufacturing demands evolve . This work focusses on 186 investigating the time efficiency, learning stability , and robustness to robot locations and 187 operational disruptions of a decentralized R L framework in comparison to a centralized 188 one. The subsequent sections will delve into the background of RL's applicability for job 189 scheduling and navigation in mobile robots, further motivating the investigation of 190 efficiency, stability , and robustness 191 192 2.3 Reinforcement Learning for Job Scheduling and Robot Navigation 193 194 Job scheduling in manufacturing , aimed at the efficient production of finished or 195 semi -finished parts from raw materials , can be solved using various techniques including 196 heuristics, meta -heuristics, and knowledge -based systems [45–47]. In contrast to these 197 approaches, m achine learning -based scheduling approaches [48–52] leverage insights 198 from historical scheduling data and utilize them to outperform heuristic and meta - 199 heuristic approaches . Furthermore, for shop floors involving automated guided vehicle s 200 (AGV s), RL algorithm s have showcased enhanced performance, efficiency, and robustness 201 than heuristic and metaheuristic approaches [53–56]. In another work, Mayer et al. [23] 202 highlight the generalizability of RL in an AGV driven shop floor to diverse manufacturing 203 scenarios . 204 10 Many of the scheduling techniques based on the above algorithms are centralized 205 or hierarchical in nature, leading to problems like high computational effort , high 206 communication cost, lack of adaptability, and lack of recovery when the equipment of the 207 centralized agent malfunctions. Decentralized multi -agent RL frameworks present 208 advantages including flexibility, modularity, reconfigurability, and adaptability. For 209 instance, Dittrich and Fohlmeister [57] have shown that a co -operative multi -agent 210 scheduling system based on RL can achieve better and comparable performances than 211 sequential (round -robin) scheduling and centralized capacity -based scheduling 212 respectively. In AGV driven shop floor s, multi -agent RL frameworks showcase enhanced 213 efficiency and robustness as compared to centralized genetic algorithm s and RL 214 frameworks [58,59] . More recently , Fang et al. [60] successfully integrated an adaptive 215 mechanism within a digital twin -driven scheduling system by utilizing decentralized RL to 216 counteract manufacturing disruptions like processing delays and machine breakdowns . 217 Robot navigation can also be achieved using several techniques including classical 218 algorithms, heuristics, and learning -based approaches [61]. Like job scheduling, l earning - 219 based approaches offer greater adaptability to dynamic environments by map ping raw 220 sensory observations to navigation controls. More recently, RL is being extensively used 221 for the navigation of ground robots for obstacle avoidance in dynamic environments [62– 222 67]. This approach is suitable for the navigation control of mobile robots to various 223 machines on a manufacturing shop floor. However, similar to centralized job scheduling, 224 centralized navigation control of robots also suffers from computational limitations. For 225 instance, Li et al. [68] investigated a centralized navigation approach that cannot meet 226 11 real-time demands (0.1 seconds to return planning results) for a large -scale system of 227 AGVs . In response to such challenges, multi -agent approaches to robot navigation have 228 gained traction. For instance , a multi -agent RL algorithm has shown promise for the 229 integrated target assignment and path planning of aerial vehicles [69]. Multi -agent RL has 230 also been used to train aerial vehicles to cooperatively perform field coverage making it 231 suitable for surveillance in intelligent manufacturing shop floors [70]. Further, deep RL 232 has enabled ground and aerial vehicles to form a coalition that is complementary and 233 cooperative for completing tasks that they are incapable of achieving alone [71,72] . 234 When integrated, job scheduling and navigation of autonomous mobile robots is 235 often formulated as a decentralized task allocation problem [73–75]. Like the individual 236 tasks of job scheduling and robot navigation, the combined task also benefits from 237 decentralized approaches. For instance, in multi -robot additive manufacturing involv ing 238 both scheduling and navigation , a swarm principle -based decentralized algorithm 239 outperforms a centralized method (based on genetic and A* algorithms ), in terms of 240 efficiency, robustness, and scalability [76]. Malus et al. [9] focused on an autonomous 241 mobile robot shop floor wherein decentralized RL agents bid on individual transport jobs 242 based on the start location of the current job, the location of the mobile robot, the 243 number of jobs assigned and the end location of the last assigned job. 244 This work specifically builds on an earlier investigation by Agrawal et al . [8] in 245 which mobile robot actions were map ped to their own local observations with only a 246 limited set of communicated observations from other agents . While such a f ine 247 granularity enhance s robustness and manufacturing productivity , it introduces a 248 12 compromise in the learning efficiency of the framework . In the current work, we 249 specifically explore the efficiency, stability, and robustness distinctions between 250 decentralized and centralized RL frameworks in mobile robot driven shop floor s. 251 252 3. METHODOLOGY 253 254 The aim of this work is to compare the efficiency, learning stability , and robustness of 255 a decentralized RL framework to that of a centralized RL framework in an autonomous 256 mobile robot driven shop floor. The decentralized framework that is utilized in this work 257 is adapted from prior work by Agrawal et al. [8] and is illustrated in Figure 2. In this 258 framework, robots and machines are represented by agents that partially perceive and 259 autonomously act upon the shop floor environment through embedded sensors and 260 controllers. Human workers are represented as operator agents that also perceive the 261 shop floor and act based on their own intelligence. All agents communicate with each 262 other through the communication server agent . This multi -agent system aims to 263 maximize throughput and safety. 264 13 265 FIGURE 2: MULTI -AGENT SYSTEMS FRAMEWORK 266 (adapted from Agrawal et al. [8]) 267 268 The learning problem for the decentralized framework is modelled as a n MDP with 269 communication amongst homogenous robot agents sharing a common policy network, 𝜋 270 during training . This is illustrated in Figure 3 and is often referred to as parameter sharing 271 [77,78] . The observation, action, and rewards for the framework are adapted from prior 272 work by Agrawal et al. [8] 273 14 274 275 FIGURE 3: DECENTRALIZED TRAINING SCENARIO 276 277 For the centralized framework, the MDP assumes a joint model for the actions and 278 states [78]. It involves a single agent with a policy network, 𝜋0 that maps the joint 279 observations of the robots to a joint action as shown in Figure 4. The observation of this 280 agent is the union of the observations of each agent from the decentralized training 281 scenario. The action space of the agent is the cartesian product of the action spaces from 282 the decentralized training scenario. To make this case comparable with the previous one, 283 the reward function is one -third the sum of the rewards each agent would have received 284 in the decentralized scenario. 285 15 286 FIGURE 4: CENTRALIZED TRAINING SCENARIO 287 288 A proximal policy optimization algorithm with curiosity driven exploration [79,80] 289 is used for training both frameworks . We make use of Unity's built in ML Agents Toolkit 290 for our reinforcement learning algorithm [81], but this work can be translated to other 291 simulators and algorithmic implementations . Specifically, the training is performed on an 292 NVIDIA GeForce GTX 1650 with 8 instances of the shop floor environment running in 293 parallel. Each instance involves random initialization for the location of the mobile robots. 294 To compare the efficiency , stability , and robustness of both the frameworks, 295 several policies with increasing time budgets are trained using both the frameworks . The 296 following tests are conducted to assess the performance of all the trained policies: 297 1. Static testing: The trained policies are evaluated for a fixed period (referred to 298 as an evaluation run) with repeated machine cycles, with each cycle having the 299 same processing time. To test the robustness of the policies to initialization, 300 several such evaluation runs are performed by randomly varying the initial 301 position of robots on the shop floor in each run. 302 16 2. Dynamic testing: T o test the robustness of the policies in dynamic scenarios 303 and failures, the trained policies are evaluate d with machine processing time 304 variations of up to 30%. Moreover, two failures are introduced in the form of a 305 machine breakdown and a robot malfunction after a few cycles . Specifically, 306 Machine 5 gets stuck in the processing state and Robot 2 gets stuck in the halt 307 state . Similar to static testing, several evaluation runs are performed by 308 randomly varying the initial position of robots on the shop floor in each run. 309 310 For each evaluation run , we compute the resulting throughput (t otal number of 311 jobs completed across all machines ) and s afety (total number of collisions amongst all 312 entities on the shop floor ). 313 314 4. RESULTS AND DISCUSSION 315 316 The cumulative reward received by the agents trained using the centralized and 317 decentralized frameworks with various compute budgets is shown in Figure 5. We 318 observe that the reward increases over time in both scenarios with the decentralized 319 case s gaining rewards equivalent to the centralized cases at a lower computational 320 expense . This show s that the decentralized framework is more efficient than the 321 centralized framework. However, we observe higher fluctuations in the cumulative 322 reward in the training runs of decentralized framework, indicating a lower stability in the 323 learning process. 324 17 325 FIGURE 5: CUMULATIVE REWARD COMPARISON ACROSS COMPUTE BUDGETS 326 327 To further investigate the efficiency and robustness of the decentralized and 328 centralized frameworks, several static and dynamic tests are conducted using all the 329 policies that were trained across the set of budgets and evaluated for throughput and 330 safety. In each of these runs, the initial location of the mobile robots is randomly assigned 331 on the shop floor. Figure 6 shows the results of all the testing scenarios . 332 We observe that both frameworks yield high performance (i.e., throughput and 333 safety) in static testing when trained using a high compute budget. Specifically, this 334 corresponds to a high number of jobs scheduled and low number of collisions at a training 335 time of approximately 10 hours in Figure 6(a) and 6(b), respectively. However, the 336 decentralized framework yields a high er performing policy than the centralized one at a 337 reduced budget in the static testing. Specifically, this corresponds to the policies that are 338 trained for approximately 5 hours. This emphasizes that the decentralized framework is 339 more efficient than the centralized one. Further, the variance in the number of jobs 340 18 scheduled at the highest budget (around 10 hours) is lower for the centralized case as 341 compared to the decentralized case. This indicates that the centralized case is more 342 robust to initialization than the decentralized one, albeit at a lower computational 343 efficiency. 344 345 FIGURE 10: RESULTS OF STATIC AND DYNAMIC TESTING 346 When comparing the effect of dynamic and static testing for the decentralized 347 framework, we observe that the number of jobs scheduled are reduced upon introducing 348 dynamic changes across all budgets. However, the number of collisions does not change 349 significantly upon introducing processing time variations and failures. This is because the 350 robots still navigat e around obstacles in a similar manner despite their altered 351 trajectories. 352 When comparing the effect of dynamic and static testing for the centralized 353 framework, we observe that the number of jobs scheduled are reduced upon introducing 354 dynamic changes like the previous comparison . However, the reduction is more 355 prominent in the high -performance case trained at a budget of approximately 10 hours. 356 The number of collisions does not change significantly for the dynamic case at this budget, 357 like the previous comparison. However, for the low -performance policies, the number of 358 19 collisions is reduced upon introduction of dynamic changes. This is attributed to the 359 failure of robot 2 which contributes less to the overall number of collisions on the shop 360 floor. This is mainly observed in the low -performance cases as the robots have not learnt 361 to effectively navigate around obstacles to reach the machines. 362 When comparing the centralized and decentralized frameworks in dynamic testing , 363 the r esults of static testing at a high budget serve as a suitable baseline performance . 364 Although the number of jobs completed in both frameworks is comparable under these 365 static conditions, the introduction of processing time variations and failures leads to a 366 more significant performance reduction in the centralized framework. This shows that 367 the decentralized framework is more robust to dynamic changes than the centralized one. 368 Overall, t he results of this study reveal that the decentralized framework 369 outperforms the centralized one in terms of efficiency and robustness to dynamic 370 changes and failures . The decentralized framework's advantage in efficiency can be 371 attributed to its distributed computing capabilities . Instead of mapping joint observations 372 to joint actions, each robot maps its own local observations and a limited set of 373 communicated observations from other robots to its own local actions, leading to 374 reduced compu tational complexity and a more efficient solution. Furthermore, its local 375 decision -making capabilities allow for enhanced robustness to dynamic changes such as 376 processing time delays, machine breakdowns, and robot malfunctions. Specifically, each 377 robot can make decisions independently to maximize its contribution to the overall 378 throughput even with such dynamic events. Further , upon extensive training, the 379 centralized framework is also able to yield a high -performance shop floor . Moreover, a 380 20 reduced variance across several evaluations runs is indicative of its robustness to 381 initialization of robot locations . This enhanced robustness is credited to the centralized 382 agent's comprehensive access to shop floor information and its control over all the robots. 383 Further more , the decentralized framework's training exhibit s higher fluctuations, 384 signalling a lower stability in the learning process as compared to the centralized one. This 385 is attributed to multiple robots sharing parameters while training the same policy despite 386 making independent decisions. While t his setup improves the learning speed by pooling 387 experiences , the variability in individual experiences coupled with collective policy 388 training results in fluctuations in learning. 389 Important ly, the extensive training required for the centralized framework to 390 achieve high performance is often impractica l. The time and computational resources 391 needed for such training can be prohibitive in high -fidelity simulators , making the 392 decentralized framework more valuable, particularly given its inherent robustness to 393 dynamic changes and failures. 394 5 Conclusion 395 396 This work compares a decentralized RL framework for the integrated job 397 scheduling and navigation control of a system involving autonomous mobile robots with 398 a comparable centralized framework when learning a control policy. On one hand, we 399 observe that the centralized framework exhibits higher learning stability and is more 400 robust to the initial location of the robots on the shop floor when comparing the policies 401 trained with the highest budget. On the other hand, the decentralized framework 402 showcases superior time efficiency and robustness to dynamic changes like processing 403 21 time delays, machine breakdowns and robot malfunctions . This study underscores the 404 decentralized framework's superiority over centralized approaches in managing 405 autonomous mobile robot -driven shop floors, highlighting its efficiency, and robustness 406 to dynamic changes and failures as key factors that make it a more practical and effective 407 solution for real -world applications. 408 Future work should aim to extend the investigation beyond the metrics of 409 efficiency, stability, and robustness to include a comprehensive analysis of accuracy, 410 generalizability, and scalability of decentralized frameworks in comparison to centralized 411 ones . This could involve an evaluation of the accuracy of robot actions in a high -fidelity 412 simulator using both the frameworks . An extension to this work could also be an 413 investigation of the generaliz ability across different shop floors with diverse layouts . 414 Additionally, assessing the scalability of these systems with a larger number of robots and 415 machines will be key to determining their viability for large -scale industrial applications. 416 Addressing these dimensions will provide a more holistic view of the potential and 417 limitations of decentralized frameworks, thereby guiding the development of more 418 effective and adaptable autonomous mobile robot -driven shop floors. 419 420 22