## 5 Service Management in the Cloud Continuum
### Markov Decision Process for Reinforcement Learning Algorithms  
Concerning the application of PPO and the DQN to the service management problem, we need to formulate a decision-making problem using the Markov Decision Process (MDP) framework [71]. Specifically, we define two slightly different MDP problems, one specific to the DQN and the other one to PPO. The difference between the two MDPs is related to the definition of possible actions, which can be more elaborated for PPO. Both MDPs define a set of states \(S\), each one defined as the above deployment array (10) to represent the allocation of service components on devices \(X_{c_{i},d_{j}}\in C\), and a reward function \(R\), which is the immediate reward \(R_{a}(S,S^{\prime})\) received after performing an action \(a\in A\) in a state \(s\).  
The main difference between the MDPs is related to how the deployment array in (10) is analyzed and, therefore, the actions that the agent can perform. For the DQN MDP, the agent has to analyze each element of the deployment array sequentially. Specifically, at each timestamp \(t_{i}\), the DQN agent analyzes an element of the deployment array \(SC[t_{i}]\), starting from the beginning to the end. When analyzing \(SC[t_{i}]\), the agent performs an action \(a\in A\) on the element \(SC[t_{i}]\) to modify the active instances for application components \(c_{i}\) on \(d_{j}\), the value of \(X_{c_{i},d_{j}}\in C\). To do so, the agent can either (i) do nothing, (ii) activate up to two new instances, or (iii) deactivate up to two instances. Let us note that we encoded these actions as integer values in \([0,5]\).  
Instead, for PPO, \(A\) has the shape of a multi-discrete action space, i.e., a vector that extends the discrete action space over a space of independent discrete actions [72]. Our proposal consists of two vectors of choices: to perform the first choice, the agent picks an element of \(C\), corresponding to the number of active instances for service component \(c_{i}\) on device \(d_{j}\). Then, for the second choice, the agent modifies the number of active instances to improve the current allocation according to the three actions described for the DQN. Therefore, according to this formulation, the PPO agent does not scan the deployment array in a sequential fashion; instead, it can learn a smarter way to improve the overall value of (9).  
Concerning the reward definition, we modeled the reward by performing an action \(a\) in state \(S\) bringing to a new state \(S^{\prime}\), i.e., \(R_{a}(S,S^{\prime})\), as the difference in (9) calculated between two consecutive time steps. Specifically, with this reward function, we want to verify if a specific action can improve or not the value of (9).  
\[R_{a}(S,S^{\prime})=R*_{a}(S,S^{\prime})-\Phi \tag{11}\]  
Specifically, \(R*_{a}(S,S^{\prime})\) and \(\Phi\) are defined as:  
\[R*_{a}(S,S^{\prime})=f(S^{\prime})-f(S) \tag{12}\]  
\[\Phi=\#infeasible*\gamma \tag{13}\]  
where \(f(S^{\prime})\) and \(f(S)\) are the objective functions calculated, respectively, in states \(S^{\prime}\) and \(S\) and \(\Phi\) is the penalty quantified in the number of accumulated infeasible allocations during the simulation multiplied by a facto \(\gamma\), which we set to 0.1. If there is an improvement from one time step to the next one (\(R*_{a}(S,S^{\prime})>0\)), the agent receives an additional bonus of 3.0 as compensation for its profitable move. Otherwise, the action taken is registered as a wrong pass since it is not remunerative in improving the objective function calculated previously. If the agent reaches an amount of 150 wrong passes, the training episode terminates immediately and resets the environment to the initial state.