## 5 SD-CFN Routing
### Reinforcement Learning Agent  
(1) State Space (\(S\)): Each state in the state space corresponds to a forwarding device in the data plane, specifically referring to SDN switches in this context. State transitions correspond to links connecting two switches. Therefore, the size of the state space is equal to the number of switches in the network topology. (2) Action Space (\(A\)): It corresponds to the set of all actions taken for all states within the state space \(A\). For a given state \(s_{i}\in S\), the reinforcement learning agent can take any action, and each action leads to a transition from state \(s_{i}\) to one of its neighboring states. The neighboring states of state \(s_{i}\) correspond to the adjacent switches related to the state \(s_{i}\). Therefore, the number of actions for state \(s_{i}\) is equal to its degree \(dr(s_{i})\), which is the number of switches connected to that state. The size of the action space \(A\) can be defined as \(|A|\equiv\sum_{s_{i}\in S}dr(s_{i})\). (3) Reward Function: The reward is inversely proportional to the available bandwidth \(bwa_{link}\), and directly proportional to the delay \(d_{link}\) and packet loss rate \(l_{link}\). \(\beta_{1},\beta_{2},\beta_{3}\in[0,1]\) are tunable parameters, allowing for different weights to be set for different features. \[R=\beta_{1}\frac{1}{bwa_{link}}+\beta_{2}d_{link}+\beta_{3}l_{link} \tag{17}\]  
Due to the different dimensions of bandwidth \(bwa_{link}\), latency \(d_{link}\), and packet loss rate \(l_{link}\), it is necessary to normalize using Equation (2). Equation (18) provides the normalized reward function:  
\[R^{\prime}=\beta_{1}\frac{1}{bwa_{link}^{\prime}}+\beta_{2}d_{link}^{\prime} +\beta_{3}l_{link}^{\prime} \tag{18}\]  
(4) Optimal Policy: The goal of the optimal policy is to minimize the reward values in the Q-Learning routing process. In this way, the agent can choose paths with high available bandwidth, low latency, and a small packet loss rate during the routing process. The intelligent agent approximates the optimal Q-function by accessing all action-state pairs. It updates and stores Q-values in the Q-table, which is used to find the optimal path from the computing request source to the target computing node. The Q-value is a measure of the overall expected reward when the reinforcement learning agent is instate \(S_{t}\) and takes action \(A_{t}\). The reinforcement learning agent updates the Q-value using Equation (19). In Equation (19), the updated Q-value (\(Q_{t+1}\)) depends on the previous value \(Q_{t}\), and it is influenced by the result \((S_{t},A_{t},R_{t},S_{t+1})\) and \(\alpha\). \(R_{t}\) is the reward at time \(t\), and \(\alpha\) determines the weight relationship between the newly acquired information and previous information. \(\alpha=0\) would make the reinforcement learning agent unable to learn from the latest \((S_{t},A_{t})\) pair, while \(\alpha=1\) allows the agent to retain learned information by considering the immediate reward \(\mathrm{Rt}\) for the \((S_{t},A_{t})\) pair. \[Q_{t+1}(S_{t+1},A_{t+1})=Q_{t}(S_{t},A_{t})+\alpha[R_{t}+min_{A}Q_{t}(S_{t+1}, A)-Q_{t}(S_{t},A_{t})] \tag{19}\]  
(5) Exploration and Exploitation Strategy: In Q-Learning, there is a trade-off between choosing the expected optimal action (exploitation) and choosing different actions in the hope of obtaining greater rewards in the future (exploration) [47]. This paper adopts the \(\epsilon\)-greedy exploration and exploitation method. \(\epsilon\)-greedy uses \(\epsilon\in[0,1]\) as a tunable parameter, allowing the intelligent agent to exploit with probability \(pr=\epsilon\) and explore with probability \(pr=1-\epsilon\). Therefore, this parameter determines the degree to which the reinforcement learning agent explores and exploits during the learning process. The agent uses Equation (20) to choose the next action for a specific state. At each step, it generates a random value \(x\in[0,1]\). If \(x<\epsilon\), the agent exploits; otherwise, the agent explores. \[\mathcal{A}=