## 4 Dynamic Placement of SFC Based on Meta Reinforcement Learning and Fuzzy Logic
### MDP Model  
In a real network scenario, it is difficult to find an optimal routing path in a dynamic network as a result of the changing of the network and the stochastic arrival of IoT-SRs. The next SFC placement is only relevant to the state of the network at the moment. Thus, we can use an MDP model to model the dynamic network. First, we depict the input of the VNF placing network, i.e., the network state (\(s_{\text{s}}\)), which includes the following several elements:  
* The ratio of the remaining CPU resource in the cloudlets \(\left\{\omega_{1}^{cpu},\omega_{2}^{cpu},\ldots,\omega_{|C|}^{cpu}\right\}\), where \(|C|\) is the cardinality of set \(C\);
* The CPU resource consumption (\(R_{i,p}^{cpu}\)) of VNF type \(p\in F_{i}\) of IoT-SR\({}_{i}\);
* The requirement for end-to-end delay (\(R_{i}^{delay}\)) of IoT-SR\({}_{i}\);  
Subsequently, a vector can be utilized to characterize the network state \(s_{\text{s}}\):  
\[s_{\text{s}}=\left\{\left\{\omega_{1}^{cpu},\omega_{2}^{cpu},\ldots,\omega_{| C|}^{cpu}\right\},R_{i,p}^{cpu},R_{i}^{delay}\right\} \tag{16}\]  
The input of SPFN \(s_{p}\) is described as follows:  
* The ratio of the remaining bandwidth in the links is \(\left\{\omega_{1}^{hp},\omega_{2}^{hw},\ldots,\omega_{|L|}^{hp}\right\}\), where \(|L|\) is the number of links in IoT-MEC networks;
* Binary codes are used to represent the path \((e,e+1)\) in IoT-SR\({}_{i}\): \(\left\{B_{e},B_{e+1}\right\}\), where \(e=1,2,\ldots,|F_{i}|+1\). These codes encode the node location information using binary coding. For instance, when considering a network comprising 60 nodes and assuming the second node as the source node and the fifteenth node as the destination node, their binary codes would be \((0,0,0,0,1,0)\) and \((0,0,1,1,1,1)\), respectively. The length of the binary code \(a\) is \(\left\lceil\log_{2}|V|\right\rceil\);
* The bandwidth requirement of IoT-SR\({}_{i}\): \(R_{i}^{hw}\);
* The requirement for end-to-end delay in IoT-SR\({}_{i}\): \(R_{i}^{delay}\). * The maximum tolerated packet loss rate of IoT-SR\({}_{i}\): \(R_{i}^{loss}\). Consequently, a vector formulation can be used to represent the network state \(s_{p}\):  
\[s_{p}=\left\{B_{e},B_{e+1},\left\{\omega_{1}^{hw},\omega_{2}^{hw},\ldots, \omega_{|L|}^{hw}\right\},R_{i}^{hw},R_{i}^{delay},R_{i}^{loss}\right\} \tag{17}\]  
Second, the definition of the action space is presented. The action set (\(a_{s}\)) is defined as the collection of all possible combinations of cloudlets. Specifically, in IoT-MEC networks with \(N\) cloudlets and an IoT-SR requiring traversal of \(K\) types of VNF instances, the task involves selecting three VNF instances among the \(N\) cloudlets. Once an action is chosen, we must check whether there is a corresponding VNF instance on the cloudlet; if not, the VNF instance is dynamically placed immediately. Figure 2: A framework for placing dynamic SFCs based on meta reinforcement learning and fuzzy logic. Subsequently, the action \(a_{p}\) is denoted as the pathway employed to establish connectivity between the initial and terminal nodes. Since \(|F_{i}|+1\) paths needed to be generated to connect the source and destination nodes and the nodes chosen by the VNF placing network in predefined order, the agent, i.e., _inner model_ (DDQN2), must determine the path iteratively. Third, the reward function of the VNF placing network is defined as follows:  
\[r_{s}=-\psi_{1}(\sum_{c\in\mathcal{C}}d_{c})-\psi_{2}(\sum_{p\in F_{i}}\sum_{c \in\mathcal{C}}x_{pc}(\eta_{a})^{1-\omega_{c}^{cpu}}-Cost_{i}(d)) \tag{18}\]  
The selected set of cloudlets, denoted as \(\hat{\mathcal{C}}\), holds significance in the formulation of the reward function. In order to maintain algorithm stability, scaling factors \(\psi_{1}\) and \(\psi_{2}\) are employed to equalize the magnitudes of the two rewards. Prior to computing the reward based on the given state and selected action, it is necessary to assess whether the chosen cloudlets possess adequate resources to accommodate VNF placement.