# Virtual Network Function Embedding under Nodal Outage Using Deep Q-Learning
## 5 Deep Neural Network  
The architecture of the NN is controlled by two hyper-parameters: (1) the number of hidden layers \(W\); i.e., the depth of the NN, and (2) the number of neurons in each hidden layer; i.e., the width of the NN, since \(W\)'s best value cannot be discovered using any analytical calculation but rather a systematic experiment. Thus, this segment discusses the impact of \(W\) in our model where these hidden layers are fully connected with a drop probability of 0. We are considering a wide range of \(W\) ranging from 2 to 10 layers.  
Recalling the Primary and Target NNs, these NNs are composed of the same architecture; thus, \(W\)'s change will influence both NNs. To examine this performance, we have considered the Netrail Topology, where the deployment of 100 VNF-FG per episode will occur. Furthermore, we assumed that each VNF-FG is comprised of three VNFs with diverse resource requirements. This experiment is repeated for various Nodal capacity,i.e., 2, 4, 8, and 12 Core CPU, as well as Core capacity (2, 4, 8, and 16 VNFs per Core), as shown in the Figure 10. The obtained results are evaluated in terms of the Runtime and Service Acceptance Ratio (SAR), as shown in Figures 10 and 11, respectively.  
```
1 Initialize Learning Rate \(\eta\), Discounting Factor \(\gamma\), Exploration Rate \(\epsilon\), and Latency
2 Initialize Replay Memory
3 Initialize action-value Q(s,a) with random weights \(\theta\)
4 Initialize Target action-value with random weights \(\theta^{\prime}\)
5 foreachepisode \(i\)=1..., \(N\)do
6 Initialize Computing Resources
7 Initialize Observed State
8 foreach\(timestep\)\(t\) = 1... \(T\)do
9 Select \(s_{0}\) from state space \(S\) as initial state
10 Using \(e\)-Greedy method select an action \(a\)
11 if\(random(0,1)>\epsilon\)then
12\(a_{t}=\arg\max_{a}Q(s_{t},a;\theta)\)
13 else
14 Execute Algorithm 1
15\(a_{t}=random(A(s))\)
16
17 end if
18 ifAction successfully embeds VNFthen
19 Reward is given
20 else
21 Penalty
22
23 end if
24 Store transition \((s_{t},a_{t},r_{t},s_{t+1})\) in Replay Memory with local reward
25 Execute Algorithm 3
26 Uniformly Sample the transitions from Replay Memory
27 Estimate the Target Value
28 ifSession at Terminal Step \(x\)then
29 Target Value = \(r_{x}\)
30 else
31 Target value = \(r_{t}+\gamma\arg\max_{a}Q(s_{t+1},a;\theta^{\prime})\)
32 end if
33 Estimate the Loss function with respect to \(\theta\)
34 Update the Target NN after every \(\tau\) steps
35 end if
36
37 end while
```  
**Algorithm 2**VNF placement based on DQL  
With the increase in \(W\), the complexity of the NNs intensifies, resulting in elevated Runtime to solve the problem. Thus, when \(W=10\), it takes \(\sim\)27 s longer to deploy 4 more VNF-FGs (per Run where, 1 Run = 100 Episodes) than \(W=2\). Further escalation in the Runtime happens with the increase in Nodal capacity and Core capacity. We have considered a model with two hidden layers because \(W=2\) provides roughly the same average SAR per Scenario as compared with a higher degree of W, at lower Runtime, as presented in Figure 11.  
```
1 ifall VNFs are placedthen
2 x \(\leftarrow\) source(m),y \(\leftarrow\) Destination(m)
3 Latency (x,y) \(\leq\) Latency
4 Reward is given
5 else
6 Penalty is given
7 Store transition (\(s_{t},a_{t},r_{t},s_{t+1}\)) in Replay Memory with global reward
```  
**Algorithm 3**Virtual Link Placement  
Figure 11: Hidden Layer performance in terms of SAR.  
Figure 10: Hidden Layer performance in terms of Runtime.  
In this section, we examined how the NN parameters affect the VNF-FG embedding. The remaining results are presented in Section 7.