## 5 Algorithm
### E2E Operation and DGA Architecture
```
1:DDQN params \(\tau\), \(\gamma\), and \(\epsilon\), number of candidate paths \(k\), and domain and overlay IDs \(\lambda,\psi\)
2:request queue \(q_{f}\) for flows \(f\), for each DQR and M-QR: replay buffer \(\mathcal{H}\), primary and target Q-networks \(Q_{\theta}\), \(Q_{\theta^{-}}\) with random weights \(\theta\) and \(\theta^{-}\), virtual path allocation buffer \(B\) for tuples (Q-value, post-allocation state, action, path)
3:for episode \(\{1,2,\ldots,\epsilon\}\)do
4:for step \(t\) in episode do
5:\((v_{src},v_{dst},f_{q_{i}})\leftarrow\)\(q_{f}.dequeue()\)\(\triangleright\) Figure 3, Step 1
6:\(a_{list}\leftarrow[],a_{i}^{\dagger,exp}\leftarrow[],f_{d}\gets\)
7:\(a_{t}^{\dagger}\leftarrow\) GetPath\((\psi,v_{src},v_{dst},q_{i},k)\)\(\triangleright\) Figure 3, Steps 2-4
8:if\(a_{t}^{\dagger}\) not empty then
9:ensure: conditions (17) for \(a_{t}^{\dagger}\); else return
10:\(a_{t}^{\texttt{seg}}\leftarrow\) splitPath\((a_{t}^{\dagger})\)\(\triangleright\) Figure 3, Step 5
11:for\((v_{src},v_{dst})\in a_{i}^{\texttt{seg}}\)do\(\triangleright\)\(v_{src},v_{dst}\in V^{B}\)
12:\(\lambda\leftarrow\) getDomain\((v_{src},v_{dst})\)
13:\(a_{t}^{\dagger}\leftarrow\) getPath\((\emptyset,v_{src},v_{dst},q_{i},k)\)\(\triangleright\) Figure 3, Steps 6-9
14: ensure: conditions (18) for \(a_{t}^{\dagger}\); else return
15:\(a_{t}^{\dagger,exp}\leftarrow\) expandPath\((a_{t}^{\dagger},a_{list})\)
16: ensure: conditions (19) for \(a_{t}^{\dagger,exp}\); else return\(\triangleright\) Figure 3, Step 10
17:\(f_{d}\gets 1\)
18:for\(\
u\in\) getDomains\((a_{t}^{\texttt{seg}})\)do
19:\(s_{t}^{\
u},a_{t}^{\
u},s_{t+1}^{\
u},r_{t}^{\
u}\leftarrow\)GetTransition\((\
u,t)\)
20:\(\mathcal{H}^{\
u}.add(s_{t}^{\
u},a_{t}^{\
u},s_{t+1}^{\
u},r_{t}^{\
u})\)
21:every \(|t|\) steps: trainAgent\((\
u)\)
22:return\(f_{d},a_{t}^{\emptyset,exp}\)
23:proceduregetPath\((\
u,src,dst,q_{i},k)\):\(\triangleright\) graph IDs, \(\
u\in\)\(\emptyset\cup\{\psi\}\)
24:\(s_{t}\leftarrow\) getEnvironmentState\((\
u)\)
25:\(A_{t}\) = getShortestPath\((G^{\
u},src,dst,k)\)\(\triangleright\)\(A_{t}\)--candidate paths list
26:for\(a_{t}^{k}\) in \(A_{t}\)do
27:\(s_{t}^{k}\) = virtualPathAlloc\((s_{t},\)\(p_{k},q_{i})\)
28:\(q\omega l_{t}^{\
u}=Q_{\theta}^{\
u}(s_{t}^{k},a_{t}^{k})\)
29:\(B^{\
u}.store(s_{t}^{k},a_{t}^{k},q\omega l_{t}^{k})\)
30:\(index\leftarrow\) argmax\({}_{qual}B^{\
u}\)
31:\(s_{t}^{\
u},a_{t}^{\
u},q\omega l_{t}=B^{\
u}[index]\)
32:return\(a_{t}^{\
u}\)
33:proceduresplitPath\((a_{t}^{\psi})\):
34:subpaths \(\leftarrow[]\)
35:for\(i\in 1,2\ldots,len(a_{t}^{\psi})-1\)do
36:subpaths.add\(((a_{t}^{\psi}[i],a_{t}^{\psi}[i+1]))\)
37:return:subpaths \(\triangleright\)\([(v_{1},v_{2})\ldots(v_{i},v_{j})];v_{i},v_{j}\in V^{B}\)
38:proceduretrainAgent\((\
u,t)\):
39: get sample: \(b_{t}^{\
u}=(s_{t}^{\
u},a_{t}^{\
u},r_{t}^{\
u},s_{t+1}^{\
u})\sim\mathcal{H}^{\
u}\)
40:\(Q^{\
u,s}(s_{t}^{\
u},a_{t}^{\
u})\approx r_{t}^{\
u}+\gamma Q_{\theta}^{\
u} (s_{t+1}^{\
u},argmax_{a^{\
u}}-Q_{\theta^{-}}^{\
u}(s_{t+1}^{\
u},a^{\
u-})\)
41:grad. descent step: \(((Q^{\
u,s}(s_{t}^{\
u},a_{t}^{\
u})-Q_{\theta}^{\
u}(s_{t}^{\
u},a_{t}^{\
u}

such as learning rate \(\tau\), decay coefficient \(\gamma\), exploration rate \(\epsilon\), the number of candidate paths \(k\), etc.). Initialisation ends with creating the queue \(q_{\text{f}}\) inside M-SDNC, which buffers arriving flow allocation requests represented by \(f\). The episode's duration is specified by the number of steps, where each step is a single flow allocation request processed by 3 DQR. Each step begins with retrieving an allocation request (getFlowRequest) from the queue \(q_{\text{f}}\). First, M-QR calculates the rough E2 EP containing only the BNs--\(a_{t}^{\psi}\), using the state visible at the overlay level \(s_{t}^{\psi}=(G^{\psi},X^{\psi},F^{\psi})\) (GetPath--described later in this section).