## 4 Mapping Algorithm
### Training Approach  
Our training process for virtual network mapping tasks typically involves the following steps:  
1. Initialization: first, the policy network's parameters are initialized at random. 2. Interaction: the policy network interacts with the environment. It takes the virtual nodes' feature matrix out of the substrate network and feeds it into the policy network. 3. Policy network output: in each interaction, a list of potential substrate nodes and the likelihood that each will be chosen are produced by the policy network. 4. Feedback from the environment: the environment provides feedback based on the policy network's output. This feedback is known as the reward signal. The agent uses the latest reward signal to determine if its behavior is appropriate. If the environment provides positive feedback, it motivates the agent to continue this behavior in the future. If the feedback is unclear or negative, the current behavior should be avoided. The reward function plays a crucial role here. The algorithm's performance and rate of convergence can be directly impacted by the reward function which has been correctly designed and optimized. The reward function used in this paper is as follows: \[reward=\alpha\cdot(revToCost(req))+\beta\cdot bw\_sa+\lambda\cdot dl\_sa\] (21) where \(revToCost(req)\) denotes the ratio of revenue to cost of request deployment and \(\alpha,\beta\), and \(\eta\) are weights coefficient of different factors. In addition, \(bw\_sa\) represents the satisfaction of bandwidth and \(dl\_sa\) represents the satisfaction of delay. \(bw\_sa\) and \(dl\_sa\) are defined as follows: \[bw\_sa=\sum_{l_{i}^{pr}\in L^{VR}}\frac{1-abs(act_{bw}(l_{i}^{pr})-req_{bw}(l _{i}^{pr}))}{|L^{VR}|}\] (22) \[dl\_sa=\sum_{l_{i}^{pr}\in L^{VR}}\frac{1-abs(act_{dl}(l_{i}^{pr})-req_{dl}( l_{i}^{pr}))}{|L^{VR}|}\] (23) where \(act_{bw}(l_{i}^{pr})\) represents the actual bandwidth, \(req_{bw}(l_{i}^{pr})\) represents the required bandwidth, \(act_{dl}(l_{i}^{pr})\) represents the actual delay, and \(req_{dl}(l_{i}^{pr})\) represents the required delay. 5. Calculation of loss: a loss value is calculated based on the action and feedback. This loss reflects the discrepancy between the policy network's prediction and the actual environmental feedback. In this paper, we calculate cross-entropy loss: \[L(p)=-\sum_{i}^{|N^{q}|}y_{i}\times log(p_{i})\] (24)  
Figure 2: Policy network. This figure shows the hierarchical structure of the policy network. The input layer receives the feature matrix, the convolutional layer extracts local features and performs linear transformation, and the Softmax layer outputs the probability of each node being selected. where \(y_{i}\) is the manually labeled selection of the \(i\)th node and \(p_{i}\) represents the possibility of the \(i\)th physical node being mapped by current virtual node. 6. Update policy network: an optimization algorithm is then employed to make the parameter of the policy network update to minimize the calculated loss. In gradient descent, parameters are updated by traveling across the gradient in the other direction to gradually decrease the loss function. The learning rate \(\eta\) controls the step size of each parameter update. If \(\eta\) is too large, the model will start to sway. Large steps may skip the optimal solution, causing the training process to converge or only converge to a local optimal solution. If \(\eta\) is too small, it can result in very slow convergence. \[g=\eta\cdot r\cdot g_{f}\] (25) where \(r\) represents the reward when a virtual request is successfully mapped and \(g_{f}\) represents the gradient used to adjust the optimization direction of the loss function. 7. Iteration: repeat the above steps until a stopping condition is met. Throughout the training process, the policy network continuously interacts with the environment and summarizes the best behavioral decisions through trial and error, gradually learning how to take effective actions under given network states to maximize returns (i.e., revenue-to-cost ratio).