## 4 Deep Reinforcement Learning-Based Algorithms
### Deep Deterministic Policy Gradient Algorithm  
In [21], the DDPG was proposed by integrating DQN and the actor-critic method to solve problems with a continuous, high-dimensional action space. DDPG is one of the agents that perform policy learning using the gradient of the objective function (22) that represents the expected long-term reward. The DDPG agent learns a policy that is continuous and deterministic. To guarantee the exploration of the environment, the trajectories generated by the agent come from another policy, which is stochastic and makes DDPG an off-policy method. The space of observations can be discrete or continuous, but the space of actions has to be continuous. The DDPG algorithm is model-free, i.e., it does not need the model of the environment, and to perform learning, it learns both the deterministic policy \(\mu(s\,|\,\theta^{\mu})\), parameterized by \(\theta^{\mu}\), and the value function of the action-state pair for this policy \(Q(s,a\,|\,\theta^{Q})\). Therefore, it uses actor-critic architecture. \[\
abla_{\theta^{\mu}}J\approx E[\
abla_{\theta^{\mu}}Q(s(t),a(t)\,|\,\theta^{ Q})\,|\,_{s=(t),a=\mu(s(t)\,|\,\theta^{\mu})}]. \tag{22}\]  
The authors in [22] show that the gradient of this objective function concerning the parameters \(\theta^{\mu}\) is given by:  
\[\
abla_{\theta^{\mu}}J\approx E[\
abla_{a}Q(s(t),a(t)\,|\,\theta^{Q})\,|\,_{s =(t),a=\mu(s(t))}\cdot\
abla_{\theta^{\mu}}\mu(s(t)\,|\,\theta^{\mu})\,|\,_{s= (t)}]. \tag{23}\]  
Note that the equation above is in the format of the actor-critic architecture, where the actor is related with \(\
abla_{\theta^{\mu}}\mu(s(t)\,|\,\theta^{\mu})\) and the critic is related with \(\
abla_{a}Q(s(t),a(t)\,|\,\theta^{Q})\). The actor-critic method consists of a neural network, called the actor, and another, called the critic. The actor network takes the state as its input and produces a value for the action, while the critic network is a Q-value network that takes both state and action as the input and produces the Q-value as a single output. As in the DQN, the DDPG also uses target networks that are time-delayed copies of their original networks, which slowly follow the learned networks. Another DQN feature used in the DDPG is the replay buffer (\(R_{b}\)) for the experiment sample to update the neural network parameters, as shown in Figure 3. Figure 3 shows the learning process of the DDPG-KRP algorithm in steps. In Algorithm 1, the parameters of the copies are updated periodically at every \(\tau\) time in a smoothed manner concerning the originals through a first-order filter with a smoothing factor (\(\rho\)) of \(10^{-5}\). These copies are called target networks. Using these target network values greatly improves the learning stability [20]. Algorithm 1 maintains a parameterized actor function, \(\mu(s\,|\,\theta^{\mu})\), that specifies the current policy, deterministically mapping states to a specific action. The critic network \(Q(s,a\,|\,\theta^{Q})\) learns using the Equation (24). The actor is updated following the chain rule applied to the expected return from the start of the \(J\) distribution as Equation (23) [5,21]. \[L=\frac{1}{N}\sum_{i=1}^{N}(y(i)-Q(s(i),a(i)\,|\,\theta^{Q}))^{2}, \tag{24}\]  
where \(y(i)\) for \(i=1,...,N\) are the desired output values for the training input samples given by the Bellman loss function, i.e., \(y(i)=r(i)+\gamma\times Q^{\prime}(s(i+1),\mu^{\prime}(s(i+1)\,|\,\theta^{\mu^{ \prime}})\,|\,\theta^{Q^{\prime}})\) where \(\mu^{\prime}(s(i+1)\,|\,\theta^{\mu^{\prime}})\) and \(Q^{\prime}(s(i+1),\mu^{\prime}(s(i+1)\,|\,\theta^{\mu^{\prime}})\,|\,\theta^ {Q^{\prime}})\) are the target networks of the actor and the critic, respectively. Since DDPG generates a deterministic policy, to ensure the exploration of the environment, it was proposed to add a noise (\(\eta\)) in the actor output.