## 6 Actor-Critic Methods
### Deep Deterministic Policy Gradient
#### 6.1.1 Assisted DDPG (AsDDPG)  
In the work by [PERSON] et al. [27], Assisted DDPG (AsDDPG) extends the standard DDPG algorithm by incorporating an external controller to accelerate training and stabilize learning. The external controller acts as a supplementary policy during the early stages of training and is progressively replaced by the learned policy as training converges. The integration of the external controller introduces an adaptive switching mechanism, enabling the algorithm to dynamically choose between the controller and the actor network. ```
Input: Learning rate \(\alpha\), discount factor \(\gamma\), replay buffer size \(N\), batch size \(B\), target update rate \(\tau\) Output: Trained actor \(\pi(s;\theta^{\pi})\) and critic \(Q(s,a;\theta^{Q})\)  Initialize actor network \(\pi(s;\theta^{\pi})\) and critic network \(Q(s,a;\theta^{Q})\) with random weights;  Initialize target networks \(\pi^{\prime}(s;\theta^{\pi^{\prime}})\) and \(Q^{\prime}(s,a;\theta^{Q^{\prime}})\) with \(\theta^{\pi^{\prime}}\leftarrow\theta^{\pi}\), \(\theta^{Q^{\prime}}\leftarrow\theta^{Q}\);  Initialize replay buffer \(\mathcal{D}\) with capacity \(N\); for each episodedo  Initialize environment and receive initial state \(s_{0}\); for each time step \(t\)do  Select action \(a_{t}\) using the exploration policy: \[a_{t}=\pi(s_{t};\theta^{\pi})+\mathcal{N}_{t},\quad\mathcal{N}_{t}\sim\text{OU }(\mu,\sigma)\] (48)  Execute action \(a_{t}\), observe reward \(r(s_{t},a_{t})\) and next state \(s_{t+1}\);  Store transition \((s_{t},a_{t},r(s_{t},a_{t}),s_{t+1})\) in replay buffer \(\mathcal{D}\); if replay buffer \(\mathcal{D}\) has at least \(B\) samplesthen  Sample a mini-batch of \(B\) transitions \((s,a,r,s^{\prime})\) from \(\mathcal{D}\);  Compute the target for each transition: \[y=r+\gamma Q^{\prime}(s^{\prime},\pi^{\prime}(s^{\prime};\theta^{\pi^{\prime}}) ;\theta^{Q^{\prime}})\] (49)  Update the critic network by minimizing the loss: \[L(\theta^{Q})=\frac{1}{B}\sum\Bigl{(}Q(s,a;\theta^{Q})-y\Bigr{)}^{2}\] (50)  Update the actor network using the sampled policy gradient: \[\
abla_{\theta^{\pi}}J(\pi)=\frac{1}{B}\sum\
abla_{a}Q(s,a;\theta^{Q})\big{|} _{a=\pi(s;\theta^{\pi})}\
abla_{\theta^{\pi}}\pi(s;\theta^{\pi})\] (51)  Soft update the target networks: \[\theta^{\pi^{\prime}} \leftarrow\tau\theta^{\pi}+(1-\tau)\theta^{\pi^{\prime}}\] (52) \[\theta^{Q^{\prime}} \leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\] (53)
```  
**Algorithm 3**Deep Deterministic Policy Gradient (DDPG) Algorithm  
The core innovation in AsDDPG is the inclusion of a Critic-DQN network that evaluates both the learned policy and the external controller. At each time step \(t\), the Critic-DQN decides which policy to use based on the Q-value estimates:  
\[\sigma_{t}=\arg\max_{\sigma\in\{\text{policy,controller}\}}Q(x_{t},\sigma; \theta^{Q}), \tag{54}\]  
where \(\sigma_{t}\) is the switching action and \(Q(x_{t},\sigma;\theta^{Q})\) represents the Q-value of the state-action pair under the selected policy. The critic network in AsDDPG consists of two branches: the _Critic branch_ and the _DQN branch_. The Critic branch evaluates the Q-value of the actor's actions and computes the policy gradient, while the DQN branch estimates the Q-values of both the external controller and the actor network. The advantage function \(A(x,a)\) is computed to improve stability:  
\[A(x,a)=Q(x,a;\theta^{Q})-V(x;\theta^{V}), \tag{55}\]  
where \(V(x;\theta^{V})\) is the state value function learned using a dueling architecture.