# Intelligent Caching with Graph Neural Network-Based Deep Reinforcement Learning on SDN-Based ICN
## 4 Proposed Methodology
### The GNN-DDQN Agent  
The GNN-DDQN agent operates based on the procedure described in Algorithm 1. In the beginning, we initialize a replay buffer \(P\), a Q-network (\(Q\)) implemented as a GNN with randomly generated parameters \(\theta\), and a target Q-network (\(\hat{Q}\)) with the same network architecture and parameters as \(Q\). Each episode corresponds to a complete round of experimentation, and the time is divided into \(T\) slots. The GNN-DDQN agent takes actions at each time step, denoted as \(t_{l}\) (starting from \(t_{1}\), as \(t_{0}\) represents the initial point of the experimentation). To balance exploration and exploitation, we utilize an \(\epsilon\)-greedy exploration strategy [40]. This strategy involves randomly selecting actions with a probability of \(\epsilon\) and selecting the action with the highest expected Q-value with a probability of \(1-\epsilon\). The purpose is to encourage initial exploration and gradually decrease exploration over time. We employ an exponential decay strategy for \(\epsilon\), starting with an initial value of \(\epsilon_{s}=0.9\) and decaying to a minimum value of \(\epsilon_{e}=0.01\) with a decay rate of 0.01, denoted as \(\epsilon_{d}=100\). Since each router with caching capability has a cache size of \(z\), the GNN-DDQN agent selects \(z\) actions for each node at each time step. It chooses the top \(z\) actions with the highest Q-values for each node during greedy action selection. For random actions, it randomly selects \(z\) actions for each node. It is crucial to emphasize that the agent precisely chooses \(z\) actions for each node at every time step. However, it only executes these actions for router nodes with caching capabilities, excluding others. ```
1:number of episodes \(E\), batch size \(B\), target network update step \(K\), replay buffer capacity \(R\), epsilon start \(\epsilon_{s}\), epsilon end \(\epsilon_{e}\), epsilon decay \(\epsilon_{d}\), number of steps \(k\), discount factor \(\gamma\)
2:Initialize replay buffer \(P\) with capacity \(R\)
3:Initialize \(Q\)-network with random weights \(\theta\)
4:Initialize target \(\hat{Q}\)-network with weights \(\hat{\theta}=\theta\)
5:for episode \(\in\{1,\dots,E\}\)do
6:for\(t_{l}\in\{t_{1},\dots,t_{T}\}\)do
7: Randomly pick \(e^{\prime}\in[0,1]\)
8:\(e_{t}=\epsilon_{e}\cdot(\epsilon_{s}-\epsilon_{e})\cdot\exp\big{(}\frac{-k}{ \epsilon_{d}}\big{)}\)
9:\(k=k+1\)
10:for\(n_{k}\in\{n_{1},\dots,n_{N}\}\)do
11:if\(e^{\prime}<\epsilon_{t}\)then
12: Randomly select \(z\) actions
13:else
14: Select \(z\) actions with the highest \(Q(\mathcal{S}_{t_{l}},\mathcal{A}_{t_{l}}|\theta)\)
15:endif
16:endfor
17: Take action \(\mathcal{A}_{t_{l}}\), get reward \(\mathcal{R}_{t_{l}}\) and next state \(\mathcal{S}_{t_{l+1}}\)
18: Store transition \((\mathcal{S}_{t_{l}},\mathcal{A}_{t_{l}},\mathcal{R}_{t_{l}},\mathcal{S}_{t_{l +1}})\) into \(P\)
19: Randomly sample B transitions \((\mathcal{S}_{b_{j}},\mathcal{A}_{b_{j}},\mathcal{R}_{b_{j}},\mathcal{S}_{b_{j +1}})\) from \(P\)
20: Use Equation (10) to compute \(\mathcal{Y}_{b_{j}}\)
21: Perform a gradient descent step on \(L(\theta)\) with respect to the network parameters \(\theta\), where \(L(\theta)\) is computed in Equation (9)
22: Update \(\hat{\theta}=\theta\) every \(K\) steps
23:endfor
24:endfor
```  
**Algorithm 1** GNN-DDQN Agent Operation  
At time step \(t_{l}\), the GNN-DDQN agent interacts with the environment by taking action \(A_{t_{l}}\) and receiving a reward \(R_{t_{l}}\) and the subsequent state \(S_{t_{l+1}}\) at time step \(t_{l+1}\). The rewards, denoted by \(\mathcal{R}_{t_{l}}\), are node-level rewards, where each node has C rewards corresponding to different actions. The newly generated transition \((\mathcal{S}_{t_{l}},\mathcal{A}_{t_{l}},\mathcal{R}_{t_{l}},\mathcal{S}_{t_{l +1}})\) is then stored in the replay buffer \(P\). We train the Q-network by randomly sampling a batch of transitions \((\mathcal{S}_{b_{j}},\mathcal{A}_{b_{j}},\mathcal{R}_{b_{j}},\mathcal{S}_{b_{j+1}})\) from the replay buffer \(P\).