# The Applicability of Reinforcement Learning Methods in the Development of Industry 4.0 Applications

Authors: Tamás Kegyes; Zoltán Süle; János Abonyi
Journal: Complexity
 (2021)

---

## Abstract

Reinforcement learning (RL) methods can successfully solve complex optimization problems. Our article gives a systematic overview of major types of RL methods, their applications at the field of Industry 4.0 solutions, and it provides methodological guidelines to determine the right approach that can be fitted better to the different problems, and moreover, it can be a point of reference for R&D projects and further researches.


---

## Body

## 1. Introduction

Reinforcement learning (RL) has a significant chance to revolutionize the artificial intelligence (AI) applications by serving a novel approach of machine learning (ML) developments that lets the user to handle large-scale problems efficiently. These techniques together with widespread Internet of things tools have opened up new possibilities for optimizing complex systems, including domains of logistics, project planning, scheduling, and further industry-related domains. Extracting this potential can result in a fundamental progress of Industry 4.0 transformation [1]. During this digital transformation, the vertical and horizontal integration will be strengthened, the flexibility should be raised, and the human control and supervision need to be focused [2, 3]. Furthermore, the data produced by the integrated tools are increasing exponentially that requires a higher level of autonomous processes and decisions. Reinforcement learning can serve as a valuable tool in the development of self-optimising and organising Industry 4.0 solutions. The main challenge of developing these applications is that there are several methods and techniques and a wide range of parameters that need to be defined. As the definition of these parameters requires detailed knowledge of the nature of the RL algorithms, the main goal of this paper is to provide a comprehensive overview of RL methods from the viewpoint of Industry 4.0 and smart manufacturing.On the basis of our best knowledge, there exists no similar overview article of reinforcement learning methods in Industry 4.0 applications. Next to the fundamental book [4], there are several overviews of reinforcement learning methods from theoretical point of view. A detailed semantic overview of Industry 4.0 frameworks [5] and a categorization of Industry 4.0 research fields are also described. An overview of key elements of Industry 4.0 researches and several application scenarios [6] highlighted the wide scope of smart manufacturing. Although many authors found that there is a lack of extensive review of Industry 4.0 revolution from different aspects, according to their persistent work nowadays, several articles are available in this topic [7]. A survey on the applications of optimal control to scheduling in production, supply chain, and Industry 4.0 systems [8] focused on maximum principle-based studies. Most of the surveys and review articles of Industry 4.0 declare the importance of optimization, but mostly only general approaches are discussed, and there are no detailed guidelines extracted. A comprehensive survey at field of Industry 4.0 and optimization [9] discussed the recent developments in data fusion and machine learning for industrial prognosis, placing an emphasis on the identification of research trends, niches of opportunity, and unexplored challenges. Even if it considered several ML methods and algorithms, RL was mentioned only shortly without extracting its key fundamentals.The above collected facts strengthened our motivation of preparing a detailed overview of RL applications and methods used in the field of Industry 4.0. Our main goals with this are:Presenting a hands-on reference for researchers who are interested in RL applicationsGiving compact descriptions of applicable RL methodsServing a guideline to support them easily identify the best fitting subset of RL methods to their problems and hence letting them focus on the relevant part of the literatureOur systematic review is based on an examination of the literature available from Scopus by following the PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols). The PRISMA-P workflow contains a 17-item checklist that supports to facilitate the preparation and reporting of a robust protocol in a standardized way for systematic reviews. The literature source list was queried in February 2021 with the following keywords: TITLE-ABS-KEY (“reinforcement learning” AND (“smart factory” OR “IOT” OR “smart manufacturing” OR “industry 4.0” OR “CPS”)).Both author keywords and index keywords were involved into the analysis. The keyword processing started with an extensive data cleansing process by:Building up a standardized keyword unit (SKU) list and splitting complex keywords into SKUsAssigning SKUs to one of the following keyword classification types:(i)
Principle captured(ii)
Industrial field of application(iii)
Application field of solution(iv)
Mathematical approach of application methodologyIdentifying major classification groups by classification types781 articles were involved into the analysis. Out of 14,035 original author and index keywords, 2,579 duplications were filtered out. The remaining 11,456 keywords were sliced into 45,824 SKUs. Finally, 12,017 keywords were assigned to classification types that provide the major tendencies and relations of industrial applications of reinforcement learning methods. Figure1 shows the change of the assessed literature size over the PRISMA steps.Figure 1
PRISMA processing flow.Our article stands for the following major parts:First, in Section2, we will give a short general introduction of reinforcement learning framework and summarize some major mathematical properties behind RL techniques. Furthermore, we will present a classification of RL methods that lets the reader to have a map for the further discussions.As a next step in Sections3.1–3.3, we will present the key findings of systematic review and a hands-on reference for further researches.Then, in Section3.4 and in Section 3.5, we will discuss the conclusions and give a detailed guideline to help the reader to choose to most adequate RL method for the different problems.Finally, in Appendices A–H, we will provide a compact overview of 18 different RL methods.


## 2. Theoretical Background of Reinforcement Learning

In this section, we will summarize the fundamental concept of reinforcement learning, then we will present a general classification of RL methods.There are three main paradigms in machine learning: supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, a functional relationship of a regression model of a classifier is learnt based on data that represent the input and output of the model. In unsupervised learning, the hidden structure of the data is explored, usually by clustering [9].Reinforcement learning (RL) also refers to learning problems. As Figure2 represents the process, an agent takes observations of the environment; then on the basis of that, it executes an action At. As a result of the action in the environment, the agent will get a reward Rt and it can take a new observation Ot from the environment and the cycle is repeated. The problem is to let agent learning so as to maximize the total reward. Reinforcement learning concept was introduced in ([4], Section 3.1). While in supervised and unsupervised learning, the model fitting requires a complete set of observations; in reinforcement learning, the learning process is sequential.Reinforcement learning is based on the reward hypothesis which states that all goals can be described by the maximisation of expected cumulative rewards. Formally, the history is the sequence of observations, actions, and rewards: Ht=O1,R1,A1,…,At−1,Ot,Rt.Figure 2
Reinforcement learning framework.A state contains all the information to determine what happens next. Formally, state is a function of the history:St=fHt. Let Gt denote the total discounted reward from time-step t: Gt=Rt+1+γRt+2+…=∑k=0∞γkRt+k+1.The state-value functionvs gives the expected total discounted return if starting from state s:vs=EGt∣St=s. Policy covers the agent’s behaviour in all possible cases, so it is essentially a map from states to actions. There are two major categories in it: (1) deterministic policy: a=πs, (2) stochastic policy: πa∣s=ℙAt=a∣St=s. The action-value function qπs,a is the expected return starting from state s, taking action a, and then following policy π: qπs;a=EπGt∣St=s,At=a.Practically, state-value function is a prediction of expected present values (PV) of future rewards that allows evaluating the goodness of states, so it is a map from states to scalars:vπs=EπRt+1+γRt+2+γ2Rt+3+…∣St=s. The optimal state-value function v∗s is the maximum state-value function overall policies: v∗s=maxπvπs. It is easy to find that in case if an optimal state-value function is known that an optimal action-value function and an optimal policy can be derived.Reinforcement learning concept is based on stochastic processes and on Markov chains. Markov property is fundamental of mathematical basis of reinforcement learning methods. A state is Markov if and only if theℙSt+1∣St=ℙSt+1∣S1,…,St condition holds. By definition, a Markov decision process (MDP) is a tuple of S;A;P;ℛ;γ, where S is a finite set of states, A is a finite set of actions, P is a state transition probability matrix, Pss′a=ℙSt+1=s′∣St=s,At=a, ℛ is a reward function, ℛsa=ERt+1∣St=s,At=a, γ is a discount factor, γ∈0;1, and t time-steps are discrete. The Bellman equation practically states that state-value function of an MDP can be decomposed into two parts: immediate reward and discounted value of successors states: vs=St=Rt+1+γvSt+1.Environments can be distinguished by its observability. Let us denoteSta as the agent’s state at time-step t and Ste as the environment’s state. Environment can be (1) fully observable if the agent directly observes all states of environment Ot=Sta=Ste, or partially observable if the agent has indirect observations Sta=ℙSte=s1,…,ℙSte=sn.Figure3 summarizes a classification of reinforcement learning methods in tree structure. Further details of the different RL methods are described in Appendix.Figure 3
Classification tree of reinforcement learning methods.


## 3. Overview of the Industry 4.0 Relevant Applications

In this section, we will present the hands-on references in tabular format based on the results of our data cleansing process and some major results of systematic literature analysis that will highlight some general trends which is able to lead the reader to a successfully applicable RL methods by preventing the usage of inappropriate trials and hence shortening development periods. In the final part of the section, we will present a hands-on guideline to summarize the key conclusions.


### 3.1. Classification of Applications by Principle Captured

The main goal of this section is to give an overview what are the principal captured problem types that reinforcement learning was applied for and describe the major tools that gave an impressive performance for each and every problem category and finally to highlight some typical issues that needed to be taken care of during the implementation.By performing SKU analysis, we identified the most relevant keywords that are assigned to a principle captured. In Table1, the associated publications are listed by principle captured categories.Table 1
Publication reference by principle captured.
Principle capturedReferred publicationsPrediction, forecasting, estimation, planning[10–39]Detection, recognition, prevention, avoidance, protection[10, 29, 39–71]Evaluation, assessment[18, 32, 54, 66, 72–81]Classification, clustering[35, 42, 66, 69, 69, 81–95]Decision making[11, 13, 17, 20, 21, 24, 38, 43, 61, 62, 66, 69, 82, 89, 93], [96–131]Allocation, assignment, resource management[20, 22, 31, 32, 39, 45, 60, 65, 67, 70, 75, 78, 83, 87, 91, 96, 97, 99, 100], [103, 104, 113, 119, 121, 121, 125, 127, 130], [130, 131, 131–154], [154–156, 156–178], [179–196], [196–202]Scheduling, queuing, planning[12, 19, 21, 24, 32, 72, 87, 88, 91, 93, 96, 99, 110, 113, 122, 125, 131, 150, 151, 160, 184, 188, 203–222]Control[12, 14, 15, 18, 23, 27, 31, 36, 37, 40, 56, 69, 70, 91, 93, 99, 101], [105–107, 111, 112, 122, 123, 129, 130], [149, 150, 167–169, 171, 180], [188, 189, 199, 200, 205, 217], [223–245], [245–265]Furthermore, Figure4 shows the principle captured classes by reinforcement learning methods. Although the related frequency table does not meet all the required criteria, in Table 2, a χ2-test, calculation is presented, yet by principle captured classes, it makes the identification of some significant deviations from the overall distribution of RL methods possible.In the class of prediction, forecasting, and estimation, planning value function approximation methods and Markov decision processes are over-represented. This lets us to conclude that the complex methods in the focus are less, which is fully in line with the goal to understand better the behaviour of the environment without strong optimization aims.In the class of detection, recognition, prevention, avoidance, and protection, the policy gradient methods are over-represented, while MDPs are under-represented. This shows us that researchers are interested more in complex models with a higher predictive performance than in basic solutions.In the classes of evaluation, assessment and allocation, assignment, and resource management, the multiagent methods are more in focus, which tells us that this field is on the way to distribute the tasks to lower level tools instead of centralized data processes. But while in the first class, the distribution of further RL methods follows the overall distribution, and in the second class, the policy gradient methods are over-represented which comes from the fact that allocation-related problems prefer to create an optimal policy.In the classes of classification, clustering and decision making and scheduling, queuing, and planning, the situation is opposite: multiagent methods are under-represented, which means that researches of these kinds of operations are still focusing to a centralized solution.In the class of control, the temporal-difference methods and Markov decision process contractions and multiagent methods are over-represented, while complex approaches, like policy gradient methods, are under-represented.Discussions of specific parts of RL solution design problems occur in smaller number of cases, but these kinds of publication demonstrate the fact that constructing an appropriate RL application is not always trivial. We can highlight state space design [12, 25, 33, 107, 144, 179, 193, 208, 217, 220, 222, 224, 227, 266, 267] and action space design [109, 220, 246, 268], reward construction [14, 76, 110, 199, 220, 226, 246, 269–273], and exploration strategy planning [86, 274] which can be determinants from the whole application point of view.Figure 4
Distribution of RL methods by principal captured classes.Table 2
χ2-test table of principle captured classes by RL method types.
Principle capturedMarkov decision processMultiarmed banditDynamicTemporal differenceValue function approximationPolicy gradientMultiagentEdge computingPrediction, forecasting, estimation, planning1.96−0.580.12−0.323.080.31−0.39−4.17Detection, recognition, prevention, avoidance, protection−3.091.51−1.11−0.170.9988, 0.9782, 0.97850.390.9935, 0.8769, 0.87852.2−1.381.65Evaluation, assessment−0.820.63−0.08−0.62−1.21−0.62.96−0.27Classification, clustering−3.610.561.281.411.540.65−2.881.05Decision making3.12.47−0.841.730.950.02−10.823.39Allocation, assignment, resource management−2.69−1.950.21−11.18−4.742.0810.547.74Scheduling, queuing, planning2.17−1.33−2.171.241.61−0.04−2.631.16Control2.67−1.342.357.52−1.72−4.934.27−10.82


### 3.2. Classification of Publications by Industrial Field of Application

Similarly, as we have shown in Section3.1, by performing SKU analysis, we also identified the most relevant keywords that are assigned to industrial fields. In Table 3, the associated publications are listed by industrial field categories.Table 3
Publication reference by industrial field of application.
Industrial fieldReferred publicationsEnergy, solar, power, electric[14, 17, 18, 23, 31, 32, 37, 45, 51, 53, 70, 72, 75, 75, 80, 81, 83, 85, 88, 100], [101, 107, 108, 118, 123, 133, 134, 136, 144, 145, 154, 159, 161, 165, 167, 168, 171], [174–177, 179, 183, 188, 192, 198, 200], [204, 208, 210, 213, 216, 221, 222, 225–227], [230–232, 234, 235, 240, 243, 246, 248, 252, 255, 257, 259], [264–266], [275–310]Telecommunication, communication, networking, internet, 5G, Wi-Fi, mobile[12, 17, 22, 28, 33, 35, 38, 39, 41, 45, 50, 50, 51, 65, 70, 72, 75, 75, 82, 83, 87, 88, 90], [91, 95, 100], [102, 104, 106, 109, 112, 120, 121, 133, 134, 136, 137, 141–143, 145], [146, 151, 157, 159, 160, 165–171, 174, 175], [177–179, 181, 187, 188, 190, 196, 197, 200, 201, 203, 206, 207, 211, 214, 218, 223], [234, 236, 239, 243, 245, 248, 252, 254, 258, 275, 277–279], [284, 286–292, 294, 297, 300], [305, 306, 309–343]Wireless, radio, antenna, signal[14, 18, 28, 32, 33, 38, 41, 45, 46, 51, 64, 68, 70, 72, 82, 83, 85, 86, 91, 94, 96, 100], [102, 104, 106–109, 128, 131, 135, 136, 140, 141, 145, 151, 158, 160], [164, 166, 168, 170, 173, 178, 179, 188, 191, 198, 200, 201, 203, 206, 207, 209, 215–218, 223, 230], [234–236, 239, 243–245, 248, 248], [251, 252, 254, 255, 257, 272, 275, 277, 279, 284, 286, 288, 293, 299, 300, 302, 304, 305, 307–309], [311, 312, 315–318, 321, 323, 328–336, 340, 341, 343–351]Vehicle, unmanned aerial vehicle, drone, aircraft[10, 68, 79, 90, 91, 151, 161, 170, 181, 183, 197, 217, 220, 223, 228, 229, 232, 244, 257, 259, 266], [272, 277, 282, 299, 305, 309–311, 313, 320, 325, 336, 341, 352–354]Cyber-physical system, robot[15, 21, 43, 50, 56, 66, 69, 73, 74, 107, 109, 112, 122, 131, 149, 188, 189, 209, 224, 225, 229, 233], [238, 241, 242, 247–249, 258, 264, 266, 268, 282, 288, 305, 350, 354–359]Manufacturing, factory[19, 36, 69, 79, 93, 99, 123, 128, 149, 205, 209, 228, 247, 279, 347, 359–362]City, building[18, 31, 45, 66, 140, 174, 201, 204, 213, 235, 244, 282, 292, 332, 340, 355, 363–368]Similarly, as we presented categories of principal captured, we also prepared Figure5 that shows the industrial field classes by reinforcement learning methods. Although the related frequency table does not meet all the required criteria, in Table 4, a χ2-test, calculation is presented, yet by industrial field classes, it makes the identification of some significant deviations from the overall distribution of RL methods possible.In the class of energy, solar, power, electric, the applications of Q-learning methods are over-represented, while more basic methods and policy gradient methods are under-representedIn the class of telecommunication, communication, networking, internet, 5G, Wi-Fi, and mobile, the policy gradient methods are over-represented and there is a strong focus on the applications of edge computingIn the class of wireless, radio, antenna, and signal, the applications of Markov decision processes are highlightedSimilarly, in the class of vehicle, unmanned aerial vehicle, drone, and aircraft, the applications of Markov decision process are over-represented together with policy gradient methods, while the multiagent solutions are less discussedIn the classes of cyber-physical system, robot and manufacturing, and factory, the basic dynamic methods and Q-learning approaches are more popularFinally, in the class of city and building, the multiagent methods are over-representedFigure 5
Distribution of RL methods by industrial field classes.Table 4
χ2-test table of industrial field classes by RL method types.
Principle capturedMarkov decision processMultiarmed banditDynamicTemporal differenceValue function approximationPolicy gradientMultiagentEdge computingEnergy, solar, power, electric0.521.7−12.7721.870.94−11.542.05−2.77Communication, networking, internet, 5G, Wi-Fi, mobile−8.61−3.516.29−18.792.2313.24−3.1412.29Wireless, radio, antenna, signal5.773.07−1.73−6.621.33−1.762.68−2.73Vehicle, unmanned aerial vehicle, drone, aircraft8.44−1.39−2.32−2.89−2.495.9−6.941.68Cyber-physical system, robot−1.7508.143.86−2.23−2.141.97−7.86Manufacturing, factory−3.33−0.554.161.361.23−2.541.52−1.84City, building−1.050.69−1.771.21−1.01−1.171.871.23


### 3.3. Classification of Publications by Mathematical Approach of Application Methodology

Similarly, as we have shown in the previous sections, we also performed the SKU analysis for the third major dimension of keywords which is the methodological approach of the solution. The most relevant keywords were identified, and then in Table5. the associated publications are listed by methodological approach categories.Table 5
Publication reference by methodological approaches.
ApproachReferred publicationsMarkov decision process[12, 23, 24, 37, 64, 70, 75, 84, 96, 100, 101, 104, 127, 130, 133, 138, 144, 153, 165, 167, 170, 177, 188, 191, 199], [203, 207, 211, 212, 214, 217, 220, 231, 252, 256–259, 263, 264, 272, 274, 281, 291, 309, 313, 320, 340, 343, 346], [369–376]Multiarmed bandit[61, 66, 102, 198, 351, 377, 378]Dynamic programming[16, 19, 27, 52, 68, 70, 84, 90, 93, 107, 119, 120, 132, 135, 141, 145, 155, 156, 161, 162, 189, 191, 198, 201, 207], [209, 212, 222, 236, 242, 247, 254, 258, 259, 278, 280, 288, 289, 304, 306, 313, 321, 331, 340, 347, 357, 371, 372, 379, 380]Q-learning[10, 17, 24, 44, 47, 50, 64, 68, 70, 80, 81, 83, 91, 92, 94, 101, 110, 116, 124, 125, 127, 129, 133, 145, 152, 172, 179], [180, 183, 187, 201, 203, 205, 206, 208, 210, 212, 215, 219, 222–225, 227, 231, 242, 244, 246, 248, 250, 254, 262], [264, 280, 282, 283, 291, 294–296, 321, 326, 327, 343, 347, 353, 356, 366, 367, 372, 374, 381–386]SARSA[14, 127, 240, 246, 280, 384]Deep Q-network[17, 47, 83, 99, 125, 133, 190, 210, 254, 291, 294, 347, 387]Deep deterministic policy gradient[229, 260, 338]Gradient descent[26, 28, 216, 388]Deep reinforcement learning[19, 20, 32, 41, 47, 50, 60, 75, 77, 84, 88, 90, 95, 98, 100, 103, 117, 131, 134, 147, 154, 159, 165, 168, 176, 179, 182], [193, 199, 201, 207, 210, 220, 221, 223, 236, 241, 260, 261, 273, 275, 281, 294, 299, 301, 302, 305, 309, 311, 317, 323, 333, 338, 341, 346, 352, 355, 361, 363, 375, 380, 389–391]Actor-critic[15, 17, 33, 97, 120, 176, 196, 221, 237, 337, 338, 349, 392]Double deep Q-network[47, 83, 125, 210, 254, 294, 387, 393]Imitation[226, 265, 355]Multiagent[32, 60, 70, 77, 103, 145, 163, 168, 173, 175, 176, 188, 195, 200, 209, 218, 219, 225, 231], [245, 251, 263, 267, 280, 289, 323, 330, 338, 344, 361, 367, 377, 394, 395]Distributed[45, 56, 60, 73, 91, 119, 133, 145, 187, 261, 282, 348, 394]Centralized[60, 147, 187, 243, 296]Cooperative[16, 81, 170, 200, 338, 344]Collaborative[45, 137, 174, 196, 237, 248, 325, 381, 396]Although it is not feasible to summarize all the different methodological approaches in details, we would like to highlight some specialities of selected cases to demonstrate how widely RL approaches are used and motivate researchers to find a solution for their problems from a new perspective.As we described in Section2, reinforcement learning methods are based on Markov property and hence it is fundamental to model the problems as Markov decision processes (MDPs), which is far not trivial in several cases. By formulating an MDP, we need to take care about state space design, especially guaranteeing that a state representation contains all the relevant information to evaluate a situation, or with other words anytime, when the system is in the same particular action, the environment will take its response by the same characteristic for a particular action [96, 104, 191, 203, 313, 346].Actor-critic methods are model-free learning methods that learn both the optimal policy for taking an action and the value function for most accurate evaluating of the current state. Most of the publications discuss mainly distributed autonomous IoT device networks. In these cases, the focus is shifted towards the learning and knowledge transfer solutions:Stochastic model of cloud-based IoT for fog computing computation offload and radio resource allocation [97].Centralized joint resource allocation solution for handling shortage of frequency resources of cellular systems by using a neural network embedded reinforcement learning algorithm [176].Determining optimal sampling time for IoT devices for energy harvesting by saving batteries. Hence state space contains continuous quantities, a linear function approximation was used and a set of novel features were introduced to represent the large state space [349].A bio-inspired RL modular architecture is able to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. Its architecture is based on a RL actor-critic model where both the actor and critic have a hierarchical structure, inspired by the mixture-of-experts model [392].Deep reinforcement learning-based cooperative edge caching approach [338].Multiple IoT devices are sending data parallel, but in general, they do not provide additional information to the existing knowledge. So, it is not necessary to permanently send data. By using actor-critic method, it can be determined which data packages need to be sent to prevent redundant or irrelevant communication [221].Mobile edge computing and energy harvesting framework of centralized training with decentralized execution by adopting MD-hybrid-AC method [120].Asynchronous advantage actor-critic method for mobile edge computing because computation offloading cannot have good performance in many situations, but the optimal algorithm can be chosen to use on IoT side [196].Optimization of the robustness of IoT network topology with a scale-free network model which has good performance in random attacks. A deep deterministic learning policy (DDLP) is proposed to improve the stability for large-scale IoT applications [337].IoT devices have lack of storage capacity, therefore a jointly cache content placement and delivery policy for the cache-enabled D2D networks was constructed. [17].A federated reinforcement learning architecture was presented where each agent working on its independent IoT device shares its learning experience (i.e., the gradient of loss function) with each other [237].By applying multiagent methods, there are multiple ways to organize learning:Local learning and no centralized knowledge (see Figure6(a))Local knowledge deployment, local learning, and central knowledge collectionLocal knowledge deployment and local learning with knowledge transfer to close neighborhoods (see Figure6(b))Local knowledge deployment and centralized learning (see Figure6(c))Figure 6
Cooperation concepts of multiagent learning systems. (a) Full local learning. (b) Knowledge sharing. (c) Centralized learning.
(a)(b)(c)


#### 3.3.1. Centralized and Federated Methods

As Internet of things (IoT) services and applications are growing rapidly, most of the current optimization-based methods lack a self-adaptive ability in dynamic environments. To handle these challenges, learning-based approaches are implemented generally in a centralized way. However, network resources may be over-consumed during the training and data transmission process. To solve the complex and dynamic control issues, a federated deep reinforcement learning-based cooperative edge caching (FADE) framework is presented. FADE enables base stations (BSs) to cooperatively learn a shared predictive model by considering the first-round training parameters of the BSs as the initial input of the local training and then uploads near-optimal local parameters to the BSs to participate in the next round of global training [16].Although the first researches have focused on designing learning algorithms with provable convergence time, but other issues, such as incentive mechanism, were explored later: a deep reinforcement learning-based incentive mechanism has been designed to determine the optimal pricing strategy for the parameter server and the optimal training strategies for edge nodes [147].


#### 3.3.2. Hierarchical Methods

Hierarchical approaches are applied primarily to solve communication channel or information processing capacity issues. The model structure usually follows the structure of the information path. In a two-layer approach, a local IoT device needs to transfer information to a local hub and then the local hub transmits the collected information to the central decision maker. In this case, separated models can be set up for both layers to find optimal scheduling order for communication.A new crowd sensing framework is introduced based on hierarchical structure to organize different resources and it is solved by using deep reinforcement learning-based strategy to ensure quality of service [88]. A hierarchical correlated Q-learning (HCEQ) approach is presented to solve the dynamic optimization of generation command dispatch (GCD) for automatic generation control (AGC) [231]. An enhanced version of a bio-inspired reinforcement learning modular architecture is presented to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. TERL architecture is based on a RL actor-critic model where both the actor and critic have a hierarchical structure, inspired by the mixture-of-experts model, formed by a gating network that selects experts specializing in learning the policies or value functions of different tasks [392]. A new cloud computing model is proposed that is hierarchically composed of two layers: a cloud control layer (CCL) and a user control layer (UCL). The CCL manages cloud resource allocation, service scheduling, service profile, and service adaptation policy from a system performance point of view. Meanwhile, the UCL manages end-to-end service connection and service context from a user performance point of view. The proposed model can support nonuniform service binding and its real-time adaptation using metaobjects by intelligent service-context management using a supervised and reinforcement learning-based machine learning framework [150]. A new cooperative resource allocation algorithm is presented which couples reinforcement learning networks and prediction neural networks for accurate mobile targets tracking. Specifically, a hierarchical structure that performs collaborative computing is designed for alleviating computing pressure of front-end devices which are supported by edge servers [397]. A slightly different approach is applied at a resilient control problem studied for cyber-physical systems (CPSs) under the denial-of-service (DoS) attack. The term resilience is interpreted as the ability to be robust to the physical layer external disturbance and defending against cyber layer DoS attacks. The overall resilient control system is described by a hierarchical game, where the cyber security issue is modeled as a zero-sum matrix game, and physical minimax control problem is described by a zero-sum dynamic game. In virtue of the reinforcement learning method, the defense/attack policy in the cyber layer can be obtained, and additionally, the physical layer control strategy can be obtained by using the dynamical programming method [398]. Further publications in hierarchical RL topics are related to balancing timeliness and criticality when gathering data from multiple sources [116], ubiquitous user connectivity, and collaborative computation offloading for smart cities [248].


#### 3.3.3. Distributed and Parallel Methods

It can be stated with certainty that the biggest potential of industrial applications is in intelligent devices. In this context, intelligence means some kind ability for taking autonomously decisions and furthermore being able to perform learning steps locally. There were made significant efforts to develop functional solution to reach this goal.Computation offloading can provide a solution for the issue of the high computation requirement of resource-constrained mobile devices. The mobile cloud is the well-known existing offloading platform, which is usually far-end network solution, but this can cause other issues, such as higher latency or network delay, which negatively affects the real-time mobile Internet of things (IoT) applications. Therefore, a deep Q-learning-based autonomic management framework is proposed as a near-end network solution of computation offloading in mobile edge [133].Another way to extend single reinforcement learning applications is to handle multiple objectives. There are two major solution practices to handle such kind of problems. The most obvious idea is to construct a mixed reward function that returns a combined result according to the different objectives [161, 259, 370]. Another possible way is to combine multiobjective ant colony optimization methods with RL techniques like deep reinforcement learning or double Q-learning algorithms [83, 142].


### 3.4. General Trends of RL Applications

Before the beginning of Industry 4.0 revolution, the general methodology was based on centralized data collection, data processing, and predictive model development solutions. By spreading Internet of things (IoT) devices, it turns possible to delegate more computational task to them. This kind of potential gets being exploited by reacting to another major issue which is the lack of communication capability. On the one hand, the communication between IoT devices and central servers or nodes are relative energy intensive processes; on the other hand, there are significant limitations on communication channels or frequencies.By distributing computational tasks to IoT devices, a fundamental change gets required: it is not possible to assign as much human effort to data processing and predictive model development supervision as before during the centralized era. This was the major reason of appreciating RL methods because it provides a general self-learning framework that basically requires no manual or human interactions to maintain.The early researches focused on the applicability of reinforcement learning techniques with single agents. Then, more and more complex problems were solved, and the multiagent solutions started to analyze. In the last years, the focus of the researchers is shifting to multiagent structures. The set-up of the agents and their goals or reward functions are showing very creative solutions. At a new wave of researches, the agents are defined with different roles often with attacker-defender objectives and let each of the agent to be trained an optimal strategy according to it. Then, the stability and robustness of the system can be analyzed and the weakest items can be purposefully improved.As Figure7 demonstrates, the number of Industry 4.0-related reinforcement learning-based researches dynamically increases, and there is no sign for expecting a slowing in it.Figure 7
Distribution of publications processed in our analysis by publication years.


### 3.5. Discussion and Guideline Process to Determine Appropriate RL Method to Use

On the basis of the previous section, it can be highlighted that there are several ways and methods how reinforcement learning can be applied for Industry 4.0-related problems, and it is far not trivial which one can provide a successful solution.We prepared a questionnaire and we presented it in a decision flow diagram in Figure8. Our primary goal was to set up a method to help the readers in formulating their RL tasks. The first questions of the questionary-based process verify whether state and action spaces are appropriately defined and how the reward can be obtained. The further questions systematically narrow down the set of applicable RL methods. The possibility of using simulation or learning from own experience can determine the general learning mechanism. In contrast, the nature of reward propagation can determine a smaller subset of the RL methods that can be applicable. Even if the conclusions are soft-defined, a user with some basic knowledge of RL methods can easily interpret them, or it can be a basis of some RL methods selector wizard. We believe that researchers will have fewer failed attempts by using our guideline, and the time-to-solution can be reduced significantly.Figure 8
Guideline process to determine appropriate RL method to use.We should keep in mind that the whole reinforcement learning concept is based on Markov decision processes. A direct conclusion is that the state space should be constructed in a way that all the potential states should contain all the relevant information that can have any influence on the outcomes. Moreover, the action space should be constructed similarly: the effects of an action in a particular state should be based on the same deterministic or stochastic behaviour. This will let the RL agent to learn the effect mechanism behind.Once the state and action spaces are defined, it needs to be investigated whether performing simulations is an option or not. If we are able to determine the environment’s behaviour when an action is made in a particular state, so deriving the reward value and the state transition, then an extensive learning process can be executed by using model-based RL methods in a cost-efficient way without significant risk of applying untrained agents. The general rule is also true in this case: the RL solution will be as adequate as the simulation is. If there is an option to validate the simulation outcomes to the real environment, then this can help to ensure the validity of the solution.


## 3.1. Classification of Applications by Principle Captured

The main goal of this section is to give an overview what are the principal captured problem types that reinforcement learning was applied for and describe the major tools that gave an impressive performance for each and every problem category and finally to highlight some typical issues that needed to be taken care of during the implementation.By performing SKU analysis, we identified the most relevant keywords that are assigned to a principle captured. In Table1, the associated publications are listed by principle captured categories.Table 1
Publication reference by principle captured.
Principle capturedReferred publicationsPrediction, forecasting, estimation, planning[10–39]Detection, recognition, prevention, avoidance, protection[10, 29, 39–71]Evaluation, assessment[18, 32, 54, 66, 72–81]Classification, clustering[35, 42, 66, 69, 69, 81–95]Decision making[11, 13, 17, 20, 21, 24, 38, 43, 61, 62, 66, 69, 82, 89, 93], [96–131]Allocation, assignment, resource management[20, 22, 31, 32, 39, 45, 60, 65, 67, 70, 75, 78, 83, 87, 91, 96, 97, 99, 100], [103, 104, 113, 119, 121, 121, 125, 127, 130], [130, 131, 131–154], [154–156, 156–178], [179–196], [196–202]Scheduling, queuing, planning[12, 19, 21, 24, 32, 72, 87, 88, 91, 93, 96, 99, 110, 113, 122, 125, 131, 150, 151, 160, 184, 188, 203–222]Control[12, 14, 15, 18, 23, 27, 31, 36, 37, 40, 56, 69, 70, 91, 93, 99, 101], [105–107, 111, 112, 122, 123, 129, 130], [149, 150, 167–169, 171, 180], [188, 189, 199, 200, 205, 217], [223–245], [245–265]Furthermore, Figure4 shows the principle captured classes by reinforcement learning methods. Although the related frequency table does not meet all the required criteria, in Table 2, a χ2-test, calculation is presented, yet by principle captured classes, it makes the identification of some significant deviations from the overall distribution of RL methods possible.In the class of prediction, forecasting, and estimation, planning value function approximation methods and Markov decision processes are over-represented. This lets us to conclude that the complex methods in the focus are less, which is fully in line with the goal to understand better the behaviour of the environment without strong optimization aims.In the class of detection, recognition, prevention, avoidance, and protection, the policy gradient methods are over-represented, while MDPs are under-represented. This shows us that researchers are interested more in complex models with a higher predictive performance than in basic solutions.In the classes of evaluation, assessment and allocation, assignment, and resource management, the multiagent methods are more in focus, which tells us that this field is on the way to distribute the tasks to lower level tools instead of centralized data processes. But while in the first class, the distribution of further RL methods follows the overall distribution, and in the second class, the policy gradient methods are over-represented which comes from the fact that allocation-related problems prefer to create an optimal policy.In the classes of classification, clustering and decision making and scheduling, queuing, and planning, the situation is opposite: multiagent methods are under-represented, which means that researches of these kinds of operations are still focusing to a centralized solution.In the class of control, the temporal-difference methods and Markov decision process contractions and multiagent methods are over-represented, while complex approaches, like policy gradient methods, are under-represented.Discussions of specific parts of RL solution design problems occur in smaller number of cases, but these kinds of publication demonstrate the fact that constructing an appropriate RL application is not always trivial. We can highlight state space design [12, 25, 33, 107, 144, 179, 193, 208, 217, 220, 222, 224, 227, 266, 267] and action space design [109, 220, 246, 268], reward construction [14, 76, 110, 199, 220, 226, 246, 269–273], and exploration strategy planning [86, 274] which can be determinants from the whole application point of view.Figure 4
Distribution of RL methods by principal captured classes.Table 2
χ2-test table of principle captured classes by RL method types.
Principle capturedMarkov decision processMultiarmed banditDynamicTemporal differenceValue function approximationPolicy gradientMultiagentEdge computingPrediction, forecasting, estimation, planning1.96−0.580.12−0.323.080.31−0.39−4.17Detection, recognition, prevention, avoidance, protection−3.091.51−1.11−0.170.9988, 0.9782, 0.97850.390.9935, 0.8769, 0.87852.2−1.381.65Evaluation, assessment−0.820.63−0.08−0.62−1.21−0.62.96−0.27Classification, clustering−3.610.561.281.411.540.65−2.881.05Decision making3.12.47−0.841.730.950.02−10.823.39Allocation, assignment, resource management−2.69−1.950.21−11.18−4.742.0810.547.74Scheduling, queuing, planning2.17−1.33−2.171.241.61−0.04−2.631.16Control2.67−1.342.357.52−1.72−4.934.27−10.82


## 3.2. Classification of Publications by Industrial Field of Application

Similarly, as we have shown in Section3.1, by performing SKU analysis, we also identified the most relevant keywords that are assigned to industrial fields. In Table 3, the associated publications are listed by industrial field categories.Table 3
Publication reference by industrial field of application.
Industrial fieldReferred publicationsEnergy, solar, power, electric[14, 17, 18, 23, 31, 32, 37, 45, 51, 53, 70, 72, 75, 75, 80, 81, 83, 85, 88, 100], [101, 107, 108, 118, 123, 133, 134, 136, 144, 145, 154, 159, 161, 165, 167, 168, 171], [174–177, 179, 183, 188, 192, 198, 200], [204, 208, 210, 213, 216, 221, 222, 225–227], [230–232, 234, 235, 240, 243, 246, 248, 252, 255, 257, 259], [264–266], [275–310]Telecommunication, communication, networking, internet, 5G, Wi-Fi, mobile[12, 17, 22, 28, 33, 35, 38, 39, 41, 45, 50, 50, 51, 65, 70, 72, 75, 75, 82, 83, 87, 88, 90], [91, 95, 100], [102, 104, 106, 109, 112, 120, 121, 133, 134, 136, 137, 141–143, 145], [146, 151, 157, 159, 160, 165–171, 174, 175], [177–179, 181, 187, 188, 190, 196, 197, 200, 201, 203, 206, 207, 211, 214, 218, 223], [234, 236, 239, 243, 245, 248, 252, 254, 258, 275, 277–279], [284, 286–292, 294, 297, 300], [305, 306, 309–343]Wireless, radio, antenna, signal[14, 18, 28, 32, 33, 38, 41, 45, 46, 51, 64, 68, 70, 72, 82, 83, 85, 86, 91, 94, 96, 100], [102, 104, 106–109, 128, 131, 135, 136, 140, 141, 145, 151, 158, 160], [164, 166, 168, 170, 173, 178, 179, 188, 191, 198, 200, 201, 203, 206, 207, 209, 215–218, 223, 230], [234–236, 239, 243–245, 248, 248], [251, 252, 254, 255, 257, 272, 275, 277, 279, 284, 286, 288, 293, 299, 300, 302, 304, 305, 307–309], [311, 312, 315–318, 321, 323, 328–336, 340, 341, 343–351]Vehicle, unmanned aerial vehicle, drone, aircraft[10, 68, 79, 90, 91, 151, 161, 170, 181, 183, 197, 217, 220, 223, 228, 229, 232, 244, 257, 259, 266], [272, 277, 282, 299, 305, 309–311, 313, 320, 325, 336, 341, 352–354]Cyber-physical system, robot[15, 21, 43, 50, 56, 66, 69, 73, 74, 107, 109, 112, 122, 131, 149, 188, 189, 209, 224, 225, 229, 233], [238, 241, 242, 247–249, 258, 264, 266, 268, 282, 288, 305, 350, 354–359]Manufacturing, factory[19, 36, 69, 79, 93, 99, 123, 128, 149, 205, 209, 228, 247, 279, 347, 359–362]City, building[18, 31, 45, 66, 140, 174, 201, 204, 213, 235, 244, 282, 292, 332, 340, 355, 363–368]Similarly, as we presented categories of principal captured, we also prepared Figure5 that shows the industrial field classes by reinforcement learning methods. Although the related frequency table does not meet all the required criteria, in Table 4, a χ2-test, calculation is presented, yet by industrial field classes, it makes the identification of some significant deviations from the overall distribution of RL methods possible.In the class of energy, solar, power, electric, the applications of Q-learning methods are over-represented, while more basic methods and policy gradient methods are under-representedIn the class of telecommunication, communication, networking, internet, 5G, Wi-Fi, and mobile, the policy gradient methods are over-represented and there is a strong focus on the applications of edge computingIn the class of wireless, radio, antenna, and signal, the applications of Markov decision processes are highlightedSimilarly, in the class of vehicle, unmanned aerial vehicle, drone, and aircraft, the applications of Markov decision process are over-represented together with policy gradient methods, while the multiagent solutions are less discussedIn the classes of cyber-physical system, robot and manufacturing, and factory, the basic dynamic methods and Q-learning approaches are more popularFinally, in the class of city and building, the multiagent methods are over-representedFigure 5
Distribution of RL methods by industrial field classes.Table 4
χ2-test table of industrial field classes by RL method types.
Principle capturedMarkov decision processMultiarmed banditDynamicTemporal differenceValue function approximationPolicy gradientMultiagentEdge computingEnergy, solar, power, electric0.521.7−12.7721.870.94−11.542.05−2.77Communication, networking, internet, 5G, Wi-Fi, mobile−8.61−3.516.29−18.792.2313.24−3.1412.29Wireless, radio, antenna, signal5.773.07−1.73−6.621.33−1.762.68−2.73Vehicle, unmanned aerial vehicle, drone, aircraft8.44−1.39−2.32−2.89−2.495.9−6.941.68Cyber-physical system, robot−1.7508.143.86−2.23−2.141.97−7.86Manufacturing, factory−3.33−0.554.161.361.23−2.541.52−1.84City, building−1.050.69−1.771.21−1.01−1.171.871.23


## 3.3. Classification of Publications by Mathematical Approach of Application Methodology

Similarly, as we have shown in the previous sections, we also performed the SKU analysis for the third major dimension of keywords which is the methodological approach of the solution. The most relevant keywords were identified, and then in Table5. the associated publications are listed by methodological approach categories.Table 5
Publication reference by methodological approaches.
ApproachReferred publicationsMarkov decision process[12, 23, 24, 37, 64, 70, 75, 84, 96, 100, 101, 104, 127, 130, 133, 138, 144, 153, 165, 167, 170, 177, 188, 191, 199], [203, 207, 211, 212, 214, 217, 220, 231, 252, 256–259, 263, 264, 272, 274, 281, 291, 309, 313, 320, 340, 343, 346], [369–376]Multiarmed bandit[61, 66, 102, 198, 351, 377, 378]Dynamic programming[16, 19, 27, 52, 68, 70, 84, 90, 93, 107, 119, 120, 132, 135, 141, 145, 155, 156, 161, 162, 189, 191, 198, 201, 207], [209, 212, 222, 236, 242, 247, 254, 258, 259, 278, 280, 288, 289, 304, 306, 313, 321, 331, 340, 347, 357, 371, 372, 379, 380]Q-learning[10, 17, 24, 44, 47, 50, 64, 68, 70, 80, 81, 83, 91, 92, 94, 101, 110, 116, 124, 125, 127, 129, 133, 145, 152, 172, 179], [180, 183, 187, 201, 203, 205, 206, 208, 210, 212, 215, 219, 222–225, 227, 231, 242, 244, 246, 248, 250, 254, 262], [264, 280, 282, 283, 291, 294–296, 321, 326, 327, 343, 347, 353, 356, 366, 367, 372, 374, 381–386]SARSA[14, 127, 240, 246, 280, 384]Deep Q-network[17, 47, 83, 99, 125, 133, 190, 210, 254, 291, 294, 347, 387]Deep deterministic policy gradient[229, 260, 338]Gradient descent[26, 28, 216, 388]Deep reinforcement learning[19, 20, 32, 41, 47, 50, 60, 75, 77, 84, 88, 90, 95, 98, 100, 103, 117, 131, 134, 147, 154, 159, 165, 168, 176, 179, 182], [193, 199, 201, 207, 210, 220, 221, 223, 236, 241, 260, 261, 273, 275, 281, 294, 299, 301, 302, 305, 309, 311, 317, 323, 333, 338, 341, 346, 352, 355, 361, 363, 375, 380, 389–391]Actor-critic[15, 17, 33, 97, 120, 176, 196, 221, 237, 337, 338, 349, 392]Double deep Q-network[47, 83, 125, 210, 254, 294, 387, 393]Imitation[226, 265, 355]Multiagent[32, 60, 70, 77, 103, 145, 163, 168, 173, 175, 176, 188, 195, 200, 209, 218, 219, 225, 231], [245, 251, 263, 267, 280, 289, 323, 330, 338, 344, 361, 367, 377, 394, 395]Distributed[45, 56, 60, 73, 91, 119, 133, 145, 187, 261, 282, 348, 394]Centralized[60, 147, 187, 243, 296]Cooperative[16, 81, 170, 200, 338, 344]Collaborative[45, 137, 174, 196, 237, 248, 325, 381, 396]Although it is not feasible to summarize all the different methodological approaches in details, we would like to highlight some specialities of selected cases to demonstrate how widely RL approaches are used and motivate researchers to find a solution for their problems from a new perspective.As we described in Section2, reinforcement learning methods are based on Markov property and hence it is fundamental to model the problems as Markov decision processes (MDPs), which is far not trivial in several cases. By formulating an MDP, we need to take care about state space design, especially guaranteeing that a state representation contains all the relevant information to evaluate a situation, or with other words anytime, when the system is in the same particular action, the environment will take its response by the same characteristic for a particular action [96, 104, 191, 203, 313, 346].Actor-critic methods are model-free learning methods that learn both the optimal policy for taking an action and the value function for most accurate evaluating of the current state. Most of the publications discuss mainly distributed autonomous IoT device networks. In these cases, the focus is shifted towards the learning and knowledge transfer solutions:Stochastic model of cloud-based IoT for fog computing computation offload and radio resource allocation [97].Centralized joint resource allocation solution for handling shortage of frequency resources of cellular systems by using a neural network embedded reinforcement learning algorithm [176].Determining optimal sampling time for IoT devices for energy harvesting by saving batteries. Hence state space contains continuous quantities, a linear function approximation was used and a set of novel features were introduced to represent the large state space [349].A bio-inspired RL modular architecture is able to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. Its architecture is based on a RL actor-critic model where both the actor and critic have a hierarchical structure, inspired by the mixture-of-experts model [392].Deep reinforcement learning-based cooperative edge caching approach [338].Multiple IoT devices are sending data parallel, but in general, they do not provide additional information to the existing knowledge. So, it is not necessary to permanently send data. By using actor-critic method, it can be determined which data packages need to be sent to prevent redundant or irrelevant communication [221].Mobile edge computing and energy harvesting framework of centralized training with decentralized execution by adopting MD-hybrid-AC method [120].Asynchronous advantage actor-critic method for mobile edge computing because computation offloading cannot have good performance in many situations, but the optimal algorithm can be chosen to use on IoT side [196].Optimization of the robustness of IoT network topology with a scale-free network model which has good performance in random attacks. A deep deterministic learning policy (DDLP) is proposed to improve the stability for large-scale IoT applications [337].IoT devices have lack of storage capacity, therefore a jointly cache content placement and delivery policy for the cache-enabled D2D networks was constructed. [17].A federated reinforcement learning architecture was presented where each agent working on its independent IoT device shares its learning experience (i.e., the gradient of loss function) with each other [237].By applying multiagent methods, there are multiple ways to organize learning:Local learning and no centralized knowledge (see Figure6(a))Local knowledge deployment, local learning, and central knowledge collectionLocal knowledge deployment and local learning with knowledge transfer to close neighborhoods (see Figure6(b))Local knowledge deployment and centralized learning (see Figure6(c))Figure 6
Cooperation concepts of multiagent learning systems. (a) Full local learning. (b) Knowledge sharing. (c) Centralized learning.
(a)(b)(c)


### 3.3.1. Centralized and Federated Methods

As Internet of things (IoT) services and applications are growing rapidly, most of the current optimization-based methods lack a self-adaptive ability in dynamic environments. To handle these challenges, learning-based approaches are implemented generally in a centralized way. However, network resources may be over-consumed during the training and data transmission process. To solve the complex and dynamic control issues, a federated deep reinforcement learning-based cooperative edge caching (FADE) framework is presented. FADE enables base stations (BSs) to cooperatively learn a shared predictive model by considering the first-round training parameters of the BSs as the initial input of the local training and then uploads near-optimal local parameters to the BSs to participate in the next round of global training [16].Although the first researches have focused on designing learning algorithms with provable convergence time, but other issues, such as incentive mechanism, were explored later: a deep reinforcement learning-based incentive mechanism has been designed to determine the optimal pricing strategy for the parameter server and the optimal training strategies for edge nodes [147].


### 3.3.2. Hierarchical Methods

Hierarchical approaches are applied primarily to solve communication channel or information processing capacity issues. The model structure usually follows the structure of the information path. In a two-layer approach, a local IoT device needs to transfer information to a local hub and then the local hub transmits the collected information to the central decision maker. In this case, separated models can be set up for both layers to find optimal scheduling order for communication.A new crowd sensing framework is introduced based on hierarchical structure to organize different resources and it is solved by using deep reinforcement learning-based strategy to ensure quality of service [88]. A hierarchical correlated Q-learning (HCEQ) approach is presented to solve the dynamic optimization of generation command dispatch (GCD) for automatic generation control (AGC) [231]. An enhanced version of a bio-inspired reinforcement learning modular architecture is presented to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. TERL architecture is based on a RL actor-critic model where both the actor and critic have a hierarchical structure, inspired by the mixture-of-experts model, formed by a gating network that selects experts specializing in learning the policies or value functions of different tasks [392]. A new cloud computing model is proposed that is hierarchically composed of two layers: a cloud control layer (CCL) and a user control layer (UCL). The CCL manages cloud resource allocation, service scheduling, service profile, and service adaptation policy from a system performance point of view. Meanwhile, the UCL manages end-to-end service connection and service context from a user performance point of view. The proposed model can support nonuniform service binding and its real-time adaptation using metaobjects by intelligent service-context management using a supervised and reinforcement learning-based machine learning framework [150]. A new cooperative resource allocation algorithm is presented which couples reinforcement learning networks and prediction neural networks for accurate mobile targets tracking. Specifically, a hierarchical structure that performs collaborative computing is designed for alleviating computing pressure of front-end devices which are supported by edge servers [397]. A slightly different approach is applied at a resilient control problem studied for cyber-physical systems (CPSs) under the denial-of-service (DoS) attack. The term resilience is interpreted as the ability to be robust to the physical layer external disturbance and defending against cyber layer DoS attacks. The overall resilient control system is described by a hierarchical game, where the cyber security issue is modeled as a zero-sum matrix game, and physical minimax control problem is described by a zero-sum dynamic game. In virtue of the reinforcement learning method, the defense/attack policy in the cyber layer can be obtained, and additionally, the physical layer control strategy can be obtained by using the dynamical programming method [398]. Further publications in hierarchical RL topics are related to balancing timeliness and criticality when gathering data from multiple sources [116], ubiquitous user connectivity, and collaborative computation offloading for smart cities [248].


### 3.3.3. Distributed and Parallel Methods

It can be stated with certainty that the biggest potential of industrial applications is in intelligent devices. In this context, intelligence means some kind ability for taking autonomously decisions and furthermore being able to perform learning steps locally. There were made significant efforts to develop functional solution to reach this goal.Computation offloading can provide a solution for the issue of the high computation requirement of resource-constrained mobile devices. The mobile cloud is the well-known existing offloading platform, which is usually far-end network solution, but this can cause other issues, such as higher latency or network delay, which negatively affects the real-time mobile Internet of things (IoT) applications. Therefore, a deep Q-learning-based autonomic management framework is proposed as a near-end network solution of computation offloading in mobile edge [133].Another way to extend single reinforcement learning applications is to handle multiple objectives. There are two major solution practices to handle such kind of problems. The most obvious idea is to construct a mixed reward function that returns a combined result according to the different objectives [161, 259, 370]. Another possible way is to combine multiobjective ant colony optimization methods with RL techniques like deep reinforcement learning or double Q-learning algorithms [83, 142].


## 3.3.1. Centralized and Federated Methods

As Internet of things (IoT) services and applications are growing rapidly, most of the current optimization-based methods lack a self-adaptive ability in dynamic environments. To handle these challenges, learning-based approaches are implemented generally in a centralized way. However, network resources may be over-consumed during the training and data transmission process. To solve the complex and dynamic control issues, a federated deep reinforcement learning-based cooperative edge caching (FADE) framework is presented. FADE enables base stations (BSs) to cooperatively learn a shared predictive model by considering the first-round training parameters of the BSs as the initial input of the local training and then uploads near-optimal local parameters to the BSs to participate in the next round of global training [16].Although the first researches have focused on designing learning algorithms with provable convergence time, but other issues, such as incentive mechanism, were explored later: a deep reinforcement learning-based incentive mechanism has been designed to determine the optimal pricing strategy for the parameter server and the optimal training strategies for edge nodes [147].


## 3.3.2. Hierarchical Methods

Hierarchical approaches are applied primarily to solve communication channel or information processing capacity issues. The model structure usually follows the structure of the information path. In a two-layer approach, a local IoT device needs to transfer information to a local hub and then the local hub transmits the collected information to the central decision maker. In this case, separated models can be set up for both layers to find optimal scheduling order for communication.A new crowd sensing framework is introduced based on hierarchical structure to organize different resources and it is solved by using deep reinforcement learning-based strategy to ensure quality of service [88]. A hierarchical correlated Q-learning (HCEQ) approach is presented to solve the dynamic optimization of generation command dispatch (GCD) for automatic generation control (AGC) [231]. An enhanced version of a bio-inspired reinforcement learning modular architecture is presented to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. TERL architecture is based on a RL actor-critic model where both the actor and critic have a hierarchical structure, inspired by the mixture-of-experts model, formed by a gating network that selects experts specializing in learning the policies or value functions of different tasks [392]. A new cloud computing model is proposed that is hierarchically composed of two layers: a cloud control layer (CCL) and a user control layer (UCL). The CCL manages cloud resource allocation, service scheduling, service profile, and service adaptation policy from a system performance point of view. Meanwhile, the UCL manages end-to-end service connection and service context from a user performance point of view. The proposed model can support nonuniform service binding and its real-time adaptation using metaobjects by intelligent service-context management using a supervised and reinforcement learning-based machine learning framework [150]. A new cooperative resource allocation algorithm is presented which couples reinforcement learning networks and prediction neural networks for accurate mobile targets tracking. Specifically, a hierarchical structure that performs collaborative computing is designed for alleviating computing pressure of front-end devices which are supported by edge servers [397]. A slightly different approach is applied at a resilient control problem studied for cyber-physical systems (CPSs) under the denial-of-service (DoS) attack. The term resilience is interpreted as the ability to be robust to the physical layer external disturbance and defending against cyber layer DoS attacks. The overall resilient control system is described by a hierarchical game, where the cyber security issue is modeled as a zero-sum matrix game, and physical minimax control problem is described by a zero-sum dynamic game. In virtue of the reinforcement learning method, the defense/attack policy in the cyber layer can be obtained, and additionally, the physical layer control strategy can be obtained by using the dynamical programming method [398]. Further publications in hierarchical RL topics are related to balancing timeliness and criticality when gathering data from multiple sources [116], ubiquitous user connectivity, and collaborative computation offloading for smart cities [248].


## 3.3.3. Distributed and Parallel Methods

It can be stated with certainty that the biggest potential of industrial applications is in intelligent devices. In this context, intelligence means some kind ability for taking autonomously decisions and furthermore being able to perform learning steps locally. There were made significant efforts to develop functional solution to reach this goal.Computation offloading can provide a solution for the issue of the high computation requirement of resource-constrained mobile devices. The mobile cloud is the well-known existing offloading platform, which is usually far-end network solution, but this can cause other issues, such as higher latency or network delay, which negatively affects the real-time mobile Internet of things (IoT) applications. Therefore, a deep Q-learning-based autonomic management framework is proposed as a near-end network solution of computation offloading in mobile edge [133].Another way to extend single reinforcement learning applications is to handle multiple objectives. There are two major solution practices to handle such kind of problems. The most obvious idea is to construct a mixed reward function that returns a combined result according to the different objectives [161, 259, 370]. Another possible way is to combine multiobjective ant colony optimization methods with RL techniques like deep reinforcement learning or double Q-learning algorithms [83, 142].


## 3.4. General Trends of RL Applications

Before the beginning of Industry 4.0 revolution, the general methodology was based on centralized data collection, data processing, and predictive model development solutions. By spreading Internet of things (IoT) devices, it turns possible to delegate more computational task to them. This kind of potential gets being exploited by reacting to another major issue which is the lack of communication capability. On the one hand, the communication between IoT devices and central servers or nodes are relative energy intensive processes; on the other hand, there are significant limitations on communication channels or frequencies.By distributing computational tasks to IoT devices, a fundamental change gets required: it is not possible to assign as much human effort to data processing and predictive model development supervision as before during the centralized era. This was the major reason of appreciating RL methods because it provides a general self-learning framework that basically requires no manual or human interactions to maintain.The early researches focused on the applicability of reinforcement learning techniques with single agents. Then, more and more complex problems were solved, and the multiagent solutions started to analyze. In the last years, the focus of the researchers is shifting to multiagent structures. The set-up of the agents and their goals or reward functions are showing very creative solutions. At a new wave of researches, the agents are defined with different roles often with attacker-defender objectives and let each of the agent to be trained an optimal strategy according to it. Then, the stability and robustness of the system can be analyzed and the weakest items can be purposefully improved.As Figure7 demonstrates, the number of Industry 4.0-related reinforcement learning-based researches dynamically increases, and there is no sign for expecting a slowing in it.Figure 7
Distribution of publications processed in our analysis by publication years.


## 3.5. Discussion and Guideline Process to Determine Appropriate RL Method to Use

On the basis of the previous section, it can be highlighted that there are several ways and methods how reinforcement learning can be applied for Industry 4.0-related problems, and it is far not trivial which one can provide a successful solution.We prepared a questionnaire and we presented it in a decision flow diagram in Figure8. Our primary goal was to set up a method to help the readers in formulating their RL tasks. The first questions of the questionary-based process verify whether state and action spaces are appropriately defined and how the reward can be obtained. The further questions systematically narrow down the set of applicable RL methods. The possibility of using simulation or learning from own experience can determine the general learning mechanism. In contrast, the nature of reward propagation can determine a smaller subset of the RL methods that can be applicable. Even if the conclusions are soft-defined, a user with some basic knowledge of RL methods can easily interpret them, or it can be a basis of some RL methods selector wizard. We believe that researchers will have fewer failed attempts by using our guideline, and the time-to-solution can be reduced significantly.Figure 8
Guideline process to determine appropriate RL method to use.We should keep in mind that the whole reinforcement learning concept is based on Markov decision processes. A direct conclusion is that the state space should be constructed in a way that all the potential states should contain all the relevant information that can have any influence on the outcomes. Moreover, the action space should be constructed similarly: the effects of an action in a particular state should be based on the same deterministic or stochastic behaviour. This will let the RL agent to learn the effect mechanism behind.Once the state and action spaces are defined, it needs to be investigated whether performing simulations is an option or not. If we are able to determine the environment’s behaviour when an action is made in a particular state, so deriving the reward value and the state transition, then an extensive learning process can be executed by using model-based RL methods in a cost-efficient way without significant risk of applying untrained agents. The general rule is also true in this case: the RL solution will be as adequate as the simulation is. If there is an option to validate the simulation outcomes to the real environment, then this can help to ensure the validity of the solution.


## 4. Conclusions

As we pointed out that reinforcement learning methods have a high potential also in Industry 4.0 applications which is a common agreement of researchers, one of the biggest reasons behind is that smart tools require a high level of optimizations which cannot be satisfied with human interventions. This continuously raises the demand of self-learning solutions, and RL techniques have been proven their efficiency at multiple fields. A major goal of our article was to give an overview of RL applications at the field of Industry 4.0. As a first step, we served a high-level overview of the general RL framework and a classification of RL methods to easily see through the possibilities, while we also presented a more detailed summary of the most widely used RL methods of Industry 4.0 applications in Appendix. Therefore, our publication can serve a starting point of further researches for RL applications.Then, we highlighted the results of our systematic literature overview of reinforcement learning applications at the field of Industry 4.0. An extensive keyword analysis drove us to identify some typical patterns by choosing an adequate RL method for some particular combinations of principal captures and industrial fields. Although there are no unique optimal RL methods, there are RL methods that provide efficient solution for some problems. Our summary can be used as a hands-on-reference for further researches and it can help researchers to shorten the preparation time for their researches.Furthermore, we prepared a questionnaire that provides a methodology to set up the reinforcement learning system in a proper way and to choose an appropriate method for the learning problem that the researcher is facing to. We believe that an extension of our questionnaire can be a basis of a wizard tool that enables the user to find the most fitting RL method for the learning task and guiding through the set-up processes. On the other hand, by knowing the key properties of the different RL methods, it becomes faster to adopt an existing one or to modify it to fit the specific needs and hence develop an own RL method.We hope that our article lets the researchers strengthen to decide using RL methods for further applications as numerous successful applications show the high efficiency of them.



---

*Source: 7179374-2021-11-30.xml*