# Multi-Agent Deep-Q Network-Based Cache Replacement Policy for Content Delivery Networks
## 3 Network Model  
We examine a CDN with edge servers interconnected as a mesh, where each edge server handles some load of end-user requests, as shown in Figure 1. We designate the set of edge servers as \(\mathcal{E}=\{1,2,\cdots,i,\cdots,E\}\). The links connecting these edge servers are represented as \(L_{E,xy}\), where \(x\) and \(y\) denote the IDs of the respective edge servers forming the connection. For example, \(L_{E,12}\) is the link between edge servers 1 and 2. Each edge server is connected to a CDN source server via a backbone link \(L_{S,z}\), where \(z\) denotes the corresponding edge server that forms the link to the source server. Moreover, the traffic costs of links \(L_{E,xy}\) and \(L_{S,z}\) are denoted as \(q_{E,xy}\) and \(q_{S,z}\), respectively. We assume that the source server has \(M\) number of slots to store all the contents distributed and populated in the network. We assign an ID to each content in the source server, ranging from 1 to \(M\). A uniform content volume is assumed for simplicity. The capacity of each edge server is limited to a size of \(C\) (\(C<M\)), and the cache of each edge server is initially filled with some content. We model the request pattern as an independent reference model with a Zipf probability distribution, which reflects the common pattern of content popularity observed in traffic measurements [26]. Hence, the content popularity is modeled by a Zipf distribution [27; 28]. For simplicity, the analysis does not explicitly account for individual user devices or the connections between these devices and their respective edge servers.  
The requests from users to the \(i\)th edge server can be represented as a temporal sequence of content IDs \(\mathcal{R}_{i}=\{R_{i,1},\cdots,R_{i,t},\cdots,R_{i,T}\}\), where \(R_{i,t}\in\{1,\cdots,M\}\) is the content request at time \(t\) for the \(i\)th edge server. Similarly, the cached content of the \(i\)th edge server at time \(t\) can be written as a sequence of content IDs such that \(C_{i,t}=\{C_{1},\cdots,C_{k},\cdots,C_{C}\}\), where \(C_{k}\in\{1,\cdots,M\}\). It can be seen that \(C_{i,t}\) varies with content requests at each time instance \(t\). A cache miss occurs when \(R_{i,t}\
otin C_{i,t}\) and \(C_{i,t}\
eq C_{i,t+1}\) after the replacement occurs.  
We use the temporal sequence of content IDs \(\mathcal{R}_{i}=\{R_{i,1},\cdots,R_{i,t},\cdots,R_{i,T}\}\) to represent the requests to the \(i\)th edge server, where \(R_{i,t}\in\{1,\cdots,M\}\) denotes the content request received at time \(t\) for the corresponding \(i\)th edge server. Similarly, the cache content of the \(i\)th edge server at time \(t\) can be written as a sequence of content IDs such that \(C_{i,t}=\{C_{1},\cdots,C_{k},\cdots,C_{C}\}\), where \(C_{k}\in\{1,\cdots,M\}\). The composition of \(C_{i,t}\) can be updated in response to the content requests received at each time step \(t\). If the requested content \(R_{i,t}\) is not found in \(C_{i,t}\), a cache miss occurs, and \(C_{i,t}\) will differ from \(C_{i,t+1}\) after the replacement process has taken place.  
In a network architecture, intercommunication among edge servers is crucial for optimizing content retrieval. When a user requests specific content from their designated edge server, the request is processed through the following steps: First, if the content is available in the local cache of the edge server, it is retrieved immediately. If the local cache  
Figure 1: Chosen network model.  
does not contain the requested content, the edge server searches its neighboring peer edge servers. If a neighboring server has the content, it is retrieved and sent to the user. If neither the local cache nor the neighboring servers have the content, the edge server fetches it directly from the source server and stores it in the local cache. This final scenario, known as a cache miss, indicates that all local and adjacent caches were exhausted before retrieving the content from the source server.