## 2 Materials and Methods
### Proposed Solution: Deep Reinforcement Learning
#### 2.2.3 DRL Algorithm

Table 2: Deep ANN Assigner topology Parameters. Figure 2: Dueling-architectured DDQN topology for our SFC Deployment agent: A two-stream deep neural network. One stream approximates the state-value function, and the other approximates the action advantage function. These values are combined to get the state-action value estimation in the output layer. The inputs are instead the action is taken and the current state. Notice that \(\epsilon(0)=\epsilon_{0}\) and  
\[\lim_{\tau\rightarrow+\infty}\epsilon(\tau)=\epsilon_{final}\]  
We call our algorithm Enhanced-Exploration Dense-Reward Duelling DDQN (\(E2\)-\(D4 QN\)) SFC Deployment. Algorithm 1 describes the training procedure of our E2-D4 QN DRL agent. We call _learning network_ the ANN approximator used to choose actions. In lines 1 to 3, we initialize the replay memory, the parameters of the first layers (\(\theta_{1}\)), the action-advantage head (\(\theta_{2}\)), and the state-value head (\(\theta_{3}\)) of the ANN approximator. We then initialize the target network with the same parameter values of the learning network. We train our agent for \(M\) epochs, each of which will contain \(N_{\epsilon}\) MDP transitions. In lines 6-10 we set an _ending episode signal_\(\tau_{end}\). We need such a signal because, when the final state of an episode has been reached, the loss should be computed with respect to the pure reward of the last action taken, by definition of \(Q(s,a)\). At each training iteration, our agent observes the environment conditions, takes an action using the \(\epsilon\)-greedy mechanism, obtains a correspondent reward, and transits to another state (lines 11-14). Our agent stores the transition in the replay buffer and then randomly samples a batch of stored transitions to run the stochastic gradient descent on the loss function in (24) (lines 14-25). Notice that the target network will only be updated with the parameter values of the learning value each \(U\) iterations to increase training stability, where \(U\) is a fixed hyper-parameter. The complete list of the training hyper-parameters used for training is enlisted in Appendix A.4. ```
1:Initialize \(\mathcal{D}\)
2:Initialize \(\theta_{1}\), \(\theta_{2}\), and \(\theta_{3}\) randomly
3:Initialize \(\theta_{1}^{-}\), \(\theta_{2}^{-}\), and \(\theta_{3}^{-}\) with the values of \(\theta_{1}\), \(\theta_{2}\), and \(\theta_{3}\), respectively
4:forepisode \(e\in\{1,2,...,M\}\)do
5:while\(\tau\leq N^{e}\)do
6:if\(\tau=N^{e}\)then
7:\(\tau_{end}\leftarrow True\)
8:else
9:\(\tau_{end}\leftarrow False\)
10:endif
11: Observe state \(s_{\tau}\) from simulator. 12: Update \(\epsilon\) using (28). 13: Sample a random assignation \(a_{t}\) action with probability \(\epsilon\) or \(a_{\tau}\leftarrow\underset{a}{\operatorname*{argmax}}Q(s_{\tau},a;\Theta)\) with probability \(1-\epsilon\). 14: Obtain the reward \(r_{\tau}\) using (18), and the next state \(s_{\tau+1}\) from the environment. 15: Store transition tuple \((s_{\tau},a_{\tau},r_{\tau},s_{\tau+1},\tau_{end})\) in \(\mathcal{D}\). 16: Sample a batch of transition tuples \(\mathcal{T}\) from \(\mathcal{D}\). 17:for all\((s_{j},a_{j},r_{j},s_{j+1},\tau_{end})\in\mathcal{T}\)do
18:if\(\tau_{end}=True\)then
19:\(y_{j}\gets r_{j}\)
20:else
21:\(y_{j}\gets r+\gamma Q(s_{j+1},\underset{a}{\operatorname*{argmax}}Q(s_{j +1},a;\theta),\theta^{-})\)
22:endif
23: Compute the temporal difference error \(\mathcal{L}(\theta)\) using (24). 24: Compute the loss gradient \(\
abla\mathcal{L}(\theta)\). 25:\(\Theta\leftarrow\Theta-l_{r}\cdot\
abla\mathcal{L}(\theta)\)
26: Update \(\Theta^{-}\leftarrow\Theta\) only every \(U\) steps. 27:endfor
28:endwhile
29:endfor
```  
**Algorithm 1** E2-D4 QN.