# Multi-Agent Deep-Q Network-Based Cache Replacement Policy for Content Delivery Networks
## 5 Multi-Agent Deep-Q Network Learning Model
 \tag{7}\]  
Additionally, we set \(\epsilon_{1}\) and \(\epsilon_{2}\) to decrease over time. Thus, the model will gradually opt for the optimal decision. If the agent performs poorly, exploration is encouraged. By keeping track of a set of recent rewards (\(\mathcal{R}_{H}\)), we update \(\epsilon_{x}\) (where \(x\in\{1,2\}\)) as follows:  
\[\epsilon_{x} = \begin{cases}\epsilon_{x}+\delta^{+},&\mathbb{E}[\mathcal{R}_{H} ]<r_{th}\\ \epsilon_{x}-\delta^{-},&\mathbb{E}[\mathcal{R}_{H}]\geq r_{th}\end{cases}  \tag{8}\]  
where \(r_{th}\) is a reward threshold, and \(\delta^{+}\) and \(\delta^{-}\) are the step sizes for updating the probability \(\epsilon_{x}\). For the chosen action (\(a_{t}\)), the agent receives an immediate reward (\(r_{t+1}\)) based on (3), transitions to the next state (\(s_{t+1}\)), and stores the transition in the replay buffer with a size of \(N_{RB}\). Throughout the training phase, agent \(i\) chooses a mini-batch of size \(B\) from the replay buffer and employs mini-batch gradient descent to train the main network. To ensure stability in learning, the target network duplicates the main network every \(J\) steps. The proposed MADQN algorithm is presented in Algorithm 1. MADQN enables the chosen edge server \(i\) to act as an agent and initializes the main network, target network, and replay buffer in Steps 2 to 5. From Steps 7 to 9, it starts selecting neighboring agents and exchanging information. The action selection, reward calculation, storing experience, mini-batch training of the main network (\(\mathbf{\theta}_{m}\)), computation of the target \(y_{k}\) over steps \(k=\{1\cdots B\}\), loss function calculation, and use of gradient descent to update the weights are presented in Steps 11 to 25. The target network parameters are updated every \(J\) steps in Step 27. With the exchanged knowledge, the agent calculates the Q-value of the MADQN algorithm as follows:  
\[Q_{t+1}(s_{t},a_{t})\leftarrow(1-\alpha)Q_{t}(s_{t},a_{t};\mathbf{ \theta}_{m})+\alpha\big{[}r(s_{t},a_{t},s_{t+1})\\ +\gamma\max_{a}Q_{t}(s_{t+1},a;\mathbf{\theta}_{tg})-Q_{t}(s_{t},a_{t}; \mathbf{\theta}_{m})\big{]}\\ +\sum_{j\in N_{e}}\omega_{ij}Q_{t-1}(s_{t-1},a_{t-1};\mathbf{\theta}_ {m}), \tag{9}\]  
where \(\omega_{ij}\) is used to emphasize the impact of neighbor \(j\) on agent \(i\) and is modeled inversely proportional to the traffic cost (\(q_{E,xy}\)) between \(j\) and \(i\)[30]. ```
1:Procedure:
2:Initialize agent \(i\)
3:Initialize main network \(\theta_{m}\)
4:Initialize target network \(\theta_{lg}\)
5:Initialize replay buffer with size (\(N_{RB}\))
6:for episode = 1:\(N_{ep}\)do
7: Select \(N_{e}\) neighboring agents
8: Send Q-value to neighboring agents \(j\in N_{e}\)
9: Receive \(\max_{a_{j}\in\mathcal{A}}Q_{t}(s,a)\) from agent \(j\in N_{e}\)
10: Observe current state (\(\mathbf{s}_{t}\))
11:for\(t=1:T\)do
12: Select action using (7) (exploration/exploitation)
13: Receive immediate reward \(r_{(}s_{t},a_{t},s_{t+1})\) and next state \(s_{t+1}\)
14: Store experience in the replay buffer. The record \((s_{t},a,r_{(}s_{t},a,s_{t+1}),s_{t+1})\), is saved. 15: Choose a mini-batch of experience (\(B\))
16:for steps \(k=1:B\)do
17:If episode ends then
18:\(y_{k}=r_{(}s_{k},a,s_{k+1})\)
19:else
20:\(y_{k}=r(s_{k},a_{k},s_{k+1})+\gamma\max_{a\in\mathcal{A}}Q_{t}(s_{k+1},a; \theta_{lg})\)
21:endif
22: Calculate the loss function:
23:\(L(\theta_{m})=\frac{1}{B}\sum_{i=1}^{B}(Q(s_{k},a_{k};\theta_{m})-y_{k})^{2}\)
24: Update Q value using gradient descent on main network parameters (\(\theta_{m}\))
25:endfor
26: Every \(j\) steps update \(\theta_{lg}=\theta_{m}\)
27: Update Q-value using Equation (9)
28:endfor
29:endfor
30:End Procedure
```  
**Algorithm 1** MADQN algorithm