## 2 Materials and Methods
### Proposed Solution: Deep Reinforcement Learning
#### 2.2.3 DRL Algorithm
[41] proposed a change in the architecture of the ANN approximator of the Q-function: they used a decomposition of the action value function in the sum of two other functions: the _action-advantage function_ and the _state-value function_:  
\[Q_{\pi}(s,a)=V_{\pi}(s)+A_{\pi}(a) \tag{25}\]  
Authors in [41] proposed a two-stream architecture for an ANN approximator, where one stream approximated \(A_{\pi}\) and the other approximated \(V_{\pi}\). They integrate such contributions at the final layer of the ANN \(Q_{\pi}\) using:  
\[Q(s,a;\theta_{1},\theta_{2},\theta_{3})=V(s;\theta_{1},\theta_{3})+(A(s,a; \theta_{1},\theta_{2})-\frac{1}{|\mathcal{A}|}\sum_{a^{\prime}}A(s,a^{\prime} ;\theta_{1},\theta_{2})) \tag{26}\]  
where \(\theta_{1}\) are the parameters of the first layers of the ANN approximator, while \(\theta_{2}\) and \(\theta_{3}\) are the parameters encoding the action-advantage and the state-value heads, respectively. This architectural innovation works as an attention mechanism for states where actions have more relevance with respect to other states and is known as _Dueling DQN_. Dueling architectures have the ability to generalize learning in the presence of many similar-valued actions. For our SFC Deployment problem, we propose the usage of the DDQN algorithm [55] where the ANN approximator of the Q-value function uses the dueling mechanism as in [41]. Each layer of our Q-value function approximator is a fully connected layer. Consequently, it can be classified as a multilayer Perceptron (MLP) even if it has a two-stream architecture. Even if we approximate \(A\pi(a)\) and \(V\pi(s)\) whit two streams, the final output layer of our ANN approximates the Q-value for each action using (26). The input neurons receive the state-space vectors \(s_{\tau}\) specified in Section 2.2.1. Figure 2 schematizes the proposed topology for our ANN. The parameters of our model are detailed instead in Table 2.  
We index the training episodes with \(e\in[0,1,...,M]\), where \(M\) is a fixed training hyper-parameter. We assume that an episode ends when all the requests of a fixed number of simulation time-steps \(N_{ep}\) have been processed. Notice that each simulation time-step \(t\) may have a different number of incoming requests, \(|R_{t}|\), and that every incoming request \(r\) will be mapped to an SFC of length \(|K|\), which coincides with the number of MDP transitions on each SFC deployment process. Consequently, the number of transitions in an episode \(e\) will be then given by  
\[N^{e}=\sum_{t\in[t_{0}^{e},t_{r}^{e}]}|K|\cdot|R_{t}| \tag{27}\]  
where \(t_{0}^{e}=t\cdot N_{ep}\) and \(t_{f}^{e}=t\cdot(N_{ep}+1)\) are the initial and final simulation timesteps of episode \(e\), respectively (Recall that \(t\in\mathbb{N}\)). To improve training performance and avoid convergence to local optima, we use the \(\epsilon\)-greedy mechanism. We introduce a high number of randomly chosen actions at the beginning of our training phase and progressively diminish the probability of taking such random actions. Such randomness should help to reduce the bias in the initialization of the ANN approximator parameters. In order to gradually lower the number of random moves as our agent learns the optimal policy, our \(\epsilon\)-greedy policy is characterized by an exponentially decaying \(\epsilon\) as:  
\[\epsilon(\tau)=\epsilon_{final}+(\epsilon_{0}-\epsilon_{final})\cdot e^{ \frac{\epsilon-\tau}{\epsilon_{decay}}},\forall\tau\in\mathbb{N}^{+} \tag{28}\]  
where we define \(\epsilon_{0}\), \(\epsilon_{final}\), and \(\epsilon_{decay}\) as fixed hyper-parameters such that  
\[Q_{\pi}(s,a)=E_{\pi}[G_{\tau}\|s_{\tau}=s,a_{\tau}=a] \tag{21}\]0 \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Action-advantage hidden layers & 2 \\ State-value hidden layers & 2 \\ hidden layers dimension & 128 \\ Input layer dimension & \(2\cdot|N_{H}|+(|N_{UC}|+|N_{CP}|+|K|+1)\) \\ Output layer dimension & \(|N_{H}|\) \\ Activation function between hidden layers & ReLU \\ \hline \hline \end{tabular}
\end{table}