# Multi-Agent Deep-Q Network-Based Cache Replacement Policy for Content Delivery Networks
## 6 Experimental Results
The final cache hit ratio value after convergence, when all corresponding requests were satisfied, is depicted in Figure 4. In Figure 4, for all three cases, it can be seen that the cache hit ratio increased with the number of neighboring edge servers. Furthermore, when the \(i\)th edge server worked cooperatively with three neighboring edge servers, compared to Case C, the cache hit ratios of Case B and Case A decreased by 0.009 and 0.0191, respectively. Another observation is that there was only a slight decrease in the cache hit ratio with an increased number of content requests (\(N_{req}\)) when employing our proposed MADQN algorithm. Figure 3: Cache hit rate versus edge server capacity.The actor-critic-based algorithm is based on [17]. Figure 5 compares various caching algorithms in terms of the average delay and average cache hit ratio. We set \(N_{rq}=5000\) and simulated over ten episodes (\(N_{ep}=10\)). The average delay was measured as the average time each policy took to cater to 5000 content requests over the ten episodes. It can be observed that traditional cache replacement policies had a higher average delay than our proposed MADQN policy. MDQN-I corresponds to the case where the neural network implemented in the \(i\)th agent contained only one hidden layer, while MADQN-II contained two hidden layers. The average delay of the MADQN-II exhibited a slight increase compared to MADQN-I, as increasing the number of hidden layers in the neural network led to a longer convergence time in a dynamic environment. Moreover, it can be observed that the SADQN approach had a slightly lower delay (a difference of only 0.21 s) compared to MADQN-I, but it had a lower cache hit ratio compared to our MADQN approach. The algorithm based on [17] achieved a cache hit ratio and average delay that fell between those of SQDQN and MQDQN-I. Finally, it can be highlighted that our proposed MADQN policy with cooperative learning outperformed traditional cache replacement policies and the algorithm based on [17]. Figure 4: Cache hit rate versus no. of neighboring servers. Figure 5: Comparison of caching policies (average delay and average cache hit ratio). The actor-critic-based algorithm is based on [17]. Figure 6 compares the performance of various caching algorithms with different sizes of \(M\) content items on the source server. The capacity C of the \(i\)th edge server was set to 50. The sizes of the \(M\) content items were chosen randomly, and the cache hit ratio was plotted against the size of the largest object. The largest possible size of a content item was set to half of the edge server capacity. It can be observed that the cache hit ratio decreased as the content size increased. Here, the cache size was fixed, so as the content item size increased, a smaller fraction of the total content could be stored in the cache at any given time. This increased the likelihood of a cache miss occurring. Moreover, with a larger content set, the cache needed to evict items more frequently to make space for new content, which led to a degradation in the cache hit ratio. However, compared with existing cache replacement algorithms, our proposed cooperative MADQN algorithm achieved better cache hit ratios as content item size increased. Figure 7 depicts the convergence of our algorithm when the cache capacity (C) was varied. We utilized our algorithm with two neighboring nodes and simulated the network for 1000 requests. The cache hit rate was monitored from the 100 th content request to the 1000 th request. It can be observed that the convergence of our algorithm slightly depended on the cache capacity of the edge server. For instance, for the three edge cache capacities (\(C\in\{30,40,50\}\)) investigated, the algorithm converged around 350 th, 250 th, and 150 th requests, respectively. The convergence of our algorithm improved with the cache capacity of the edge server.