## 4 Dynamic Placement of SFC Based on Meta Reinforcement Learning and Fuzzy Logic
### Inner Model  
Figure 5 shows an illustration of the proposed DDQN-based inner model. After we obtain the optimal initial parameters of the inner model, we can use them to train the inner model, which is directed at the new running IoT-MEC scenario. The inner model is used to make the placement and routing decisions. In this paper, a neural network is utilized to approximate the value function. The distribution of the Q function can be approximated more precisely and achieve an effective SFC placement strategy through continuous training of the DDQN network. The experience replay pool is employed by the DDQN network to store empirical samples from each iteration, and network parameters are updated by randomly extracting data from the experience replay pool. This approach helps to break the correlation between data. As we use the DDQN model to solve the two subproblems, i.e., the VNF placing network and SPFN, we use a similar method to train the neural network for both. First, the parameters of the neural network of the inner model are initialized as the parameters of the trained general network. The agent interacts with the environment to generate trajectories stored in experience memory. Then, we extract a batch of trajectories to train the neural network of the inner model, which aims at achieving fast adaptation in the current IoT-MEC scenario within a few steps. Finally, we can use the trained inner model to make the placement/routing decisions. We present the algorithmic process of meta adapting in Algorithm 2. Figure 4: The general training process of outer model. Figure 5: Dynamic SFC placement framework based on DDQN. After training the neural network, we can use the trained model to realize dynamic SFC placement in IoT-MEC networks, which can be broken down into three stages: (1) Acquire the current network state \(S_{s}\) using the SDN controller and send it to the agent; then, use \(DDQN_{1}\) to select the optimal combination of VNF instances, i.e., determine the cloudlets for IoT-SRs to traverse. Then, dynamically place VNF instances according to the action with the highest reward. (2) Conduct a preliminary evaluation of each link by considering available bandwidth, delay, and packet loss rate using fuzzy logic. (3) Choose the top \(k\) links in terms of performance and use \(DDQN_{2}\) to further choose the routing action with the highest reward based on the output of \(DDQN_{1}\) and the current state (\(S_{p}\)). The pseudocode of the online running algorithm is depicted in Algorithm 3. ```
0: VNF placing network: \(\theta_{s}\); The network for searching SFC path: \(\theta_{p}\); Remaining resource ratios: \(\left\{\omega_{1}^{cpu},\omega_{2}^{cpu},\ldots,\omega_{|C|}^{cpu}\right\}\) and \(\left\{\omega_{1}^{bw},\omega_{2}^{bw},\ldots,\omega_{|L|}^{bw}\right\}\); IoT-SRs set \(R\). 0: The routing path of IoT-SR which is denoted as \(\Omega\). 1:while\(R\
eq\emptyset\)do
2: Choose an IoT-SR\({}_{i}\) from \(R\) which is represented as \(\left(S_{i},D_{i},F_{i},R_{i,p}^{cpu},R_{i}^{bw},R_{i}^{delay}\right)\). 3: Initialize routing path \(\Omega_{i}=\emptyset\). 4: The SDN controller is interacted with by the agent to obtain the network status and information, serving for the input \(s_{s}\) for the VNF placing network. 5: Get the trained \(DDQN_{1}\) model based on Algorithm 2. 6: The output is obtained by applying \(s_{s}\) to \(DDQN_{1}\)\(\theta_{s}\), and it is assigned to \(y_{s}\). 7: The action \(a_{s}\) which has the highest Q value among \(y_{s}\) is selected. 8: By executing action \(a_{s}\), the VNF instances are dynamically placed according to its specifications. This action yields a reward \(r_{s}\) and leads to the acquisition of the subsequent status \(s^{\prime}_{s}\). The obtained sample \((s_{s},a_{s},r_{s},s^{\prime}_{s})\) is then stored within the experience replay buffer \(E_{s}\). 9: Get the trained \(DDQN_{2}\) model based on Algorithm 2.