# Multi-Agent Deep-Q Network-Based Cache Replacement Policy for Content Delivery Networks
## 6 Experimental Results  
The implementation details of our proposed cache replacement algorithm are presented in this section. We consider a mesh network with four edge servers, where each edge server is connected to other edge servers and the source server. The network configuration is assumed to be optimized. The source server slot capacity is set to \(M=60\), and the edge servers' capacities are specified in each simulation. A chosen \(i\)th edge server has \(N_{e}=4\in\mathcal{E}_{nb}\) neighboring edge servers. The link costs between the edge servers (\(q_{E,xy}\)) and the source CDN server (\(q_{5,z}\)) are initialized randomly. This setup is used to investigate the capability of our algorithm in choosing the correct neighboring edge servers with the minimum traffic costs. The Zipf probabilistic distribution, which is used to generate content requests, has a distribution parameter set to 1.2 [27; 28]. Typically, \(N_{req}=10^{4}\) requests are generated for each agent in the simulations unless stated otherwise. We set the parameters of the reward function design in (3) and (5) as follows: \(\lambda=0.9\), and \(\beta=10\), \(\rho=30\), and \(\kappa=1\). Each agent \(i\) has a main network and a target network. Both neural networks are implemented with one hidden layer consisting of 256 nodes. A deeper network is not adopted in order to accelerate the convergence of the network. Hyperparameter tuning via cross-validation is used to obtain the optimal DQN network parameters which are specified as follows: The learning rate and batch size are set to \(\alpha=0.01\) and \(B=32\), respectively. The size of the replay buffer is set to \(N_{RB}=1000\). The target network is updated every \(J=100\) steps. The discount factor in (8) is set to \(\gamma=0.8\). The probabilities for the exploration policies \(\epsilon_{1}\) and \(\epsilon_{2}\) are initialized at 0.1 and 0.3, respectively. Both parameters \(\delta^{+}\) and \(\delta^{-}\) are set to 0.005 for \(\epsilon_{1}\). For \(\epsilon_{2}\), \(\delta^{+}\) and \(\delta^{-}\) are set to 0.1 and 0.001, respectively. We present our experimental results below. We evaluated our proposed cooperative MADQN with respect to several network parameters. Comparisons are presented to depict the performance enhancements achieved by our proposed algorithm. We compared our algorithm with existing non-cooperative heuristic cache replacement algorithms and a non-cooperative reinforcement learning-based algorithm. Moreover, we present how the number of cooperative nodes affects performance. Figure 3 depicts the cache hit ratio with respect to the capacity of the \(i\)th edge server for different cache replacement policies. We compared the performance of our MADQN approach with that of LRU, LFU, and random cache replacement policies. Furthermore, comparisons were made with a single-agent DQN (SADQN) and an actor-critic algorithm based on [17] to show the performance differences. For instance, with an edge cache capacity of \(C=20\), our MADQN approach with only one neighboring edge server achieved percentage increases of 2.74%, 3.33%, 12.51%, 28.569%, and 50.01% compared to SADQN, the algorithm based on [17], LRU, LFU, and random cache replacement policies, respectively. Hence, compared to the reinforcement learning algorithm based on [17] and SADQN, our cooperative MADQN algorithm achieved better performance in terms of the cache hit ratio. Moreover, it can be observed that with a higher number of neighboring edge servers (\(N_{c}\)), the cache hit ratio increased. For instance, MADQN with two neighboring edge servers (\(N_{e}=2\)) achieved a cache hit ratio increase of 0.08984 compared to MADQN with only one neighboring edge server (\(N_{e}=1\)) when the edge server capacity was \(C=40\). Figure 4 depicts the performance of our MADQN policy when varying the number of neighboring edge servers for three different cases: \(N_{req}=\{10,000,5000,1000\}\). The \(i\)th edge server capacity \(C\) was set to 50 for all considered scenarios.