## 2 Research Methodology  
RL is a data-driven approach to solve decision-making problems in the form of Markov Decision Processes (MDP). Figure 1 shows the workflow where an agent is fed with an observation \(\mathbf{s}\) and a reward \(r\) so that it can decide an action \(\mathbf{a}\) to take on an environment, with the scope of optimizing the cumulative reward \(\sum r\) over time [60,61]. In this work, the algorithm used for training the agent is the Deep Deterministic Policy Gradient (DDPG) [62], an off-policy, model-free, actor-critic structured approach. The agent is composed of four neural networks, namely the \(Q\)-function \(\mathbf{Q}_{\theta}\), the policy \(\boldsymbol{\pi}_{\varphi}\), and their respective target lagged copies \(\mathbf{Q}^{\prime}_{\theta}\), and \(\boldsymbol{\pi}^{\prime}_{\phi^{\prime}}\). The \(\mathbf{Q}_{\theta}\) network is used to approximate the \(Q\)-function of the environment, that is the function predicting the cumulative reward for a given \(\mathbf{s}\) and \(\mathbf{a}\). The policy \(\boldsymbol{\pi}_{\varphi}\) maps the state with an action. Each of the target copies contains lagged parameters of its corresponding neural  
Figure 1: Workflow scheme of the RL approach. network. The DDPG algorithm works through the joint learning of the optimal \(Q\)-function and the policy. The reinforcement learning loop begins with the random initialization of the four neural networks. Then, the agent begins to act in the environment to collect experiences in an experience buffer of dimension \(D\). Each entry of the experience buffer is a tuple with the structure \((\mathbf{s},\ \mathbf{a},\ r,\ \mathbf{s}^{\prime})\), where \(\mathbf{s}^{\prime}\) represents the new state of the system after taking action \(\mathbf{a}\). Then, in order to approximate the optimal \(Q\)-function, the Mean Squared Bellman Error (MSBE) is minimized by employing the loss function given by:  
\[L_{MSBE}=\frac{1}{d}\sum_{i=1}^{d}\left(\mathbf{Q}_{\theta}(\mathbf{s}_{i},\ \mathbf{a}_{i})-y_{i}(\mathbf{s}_{i},\ \mathbf{a}_{i},\ r_{i},\ \mathbf{s}_{i}^{\prime})\right)^{2} \tag{1}\]  
where \(\mathbf{Q}_{\theta}(\mathbf{s},\ \mathbf{a})\) is the neural network with the current approximation of the optimal \(Q\)-function, \(d\) is the size of the replay buffer of previous experiences randomly sampled from the experience buffer of length \(D\) containing a subset of all the transitions in \((\mathbf{s},\ \mathbf{a},\ r,\ \mathbf{s}^{\prime})\) tuples, \(\mathbf{s}^{\prime}\) is the observation of the subsequent state, and \(y_{i}\) is the target value function defined as:  
\[y_{i}=\left\{\begin{array}{l}r_{i}+\gamma\mathbf{Q}_{\theta^{\prime}}^{ \prime}\left(\mathbf{s}_{i}^{\prime},\ \boldsymbol{\pi}_{\theta^{\prime}}^{\prime}(\mathbf{s}_{i}^{\prime})\right), \quad t<t_{end}\\ r_{i},\quad t=t_{end}\right. \tag{2}\]  
where \(t\) is the time of the observation \(\mathbf{s}^{\prime}\), \(t_{end}\) is the total duration of the episode of training, and \(\gamma\) is the discount factor. In Equation (2), the target lagged copies \(\boldsymbol{\pi}_{\varphi^{\prime}}^{\prime}(\mathbf{s}_{i}^{\prime})\) and \(\mathbf{Q}_{\theta^{\prime}}^{\prime}\left(\mathbf{s}_{i}^{\prime},\boldsymbol {\pi}_{\varphi^{\prime}}^{\prime}(\mathbf{s}_{i}^{\prime})\right)\) are used to avoid the instability in the training process derived from the recursive approach to approximate the \(Q\)-function. Then, the neural network \(\mathbf{Q}_{\theta}\) is updated through the use of a one-step gradient by employing \(L_{MSBE}\) defined in Equation (1). Subsequently, from the current \(\mathbf{Q}_{\theta}\), the gradient ascent \(\boldsymbol{\
abla}\) is computed to update the policy \(\boldsymbol{\pi}_{\varphi}\).