# Multi-Agent Deep-Q Network-Based Cache Replacement Policy for Content Delivery Networks
## 5 Multi-Agent Deep-Q Network Learning Model  
For the formulated MDP, a reinforcement learning approach is necessary to acquire knowledge of the rewards and the probabilistic distribution of transitions. We adopt a model-free reinforcement learning approach that does not require specifically learning the dynamics of the environment. Traditional Q-learning, a value-based model-free technique, involves agent \(i\) observing state \(s_{t}\) and selecting the best possible action \(a\) at time \(t\) to transition to state \(s_{t+1}\). Subsequently, the agent receives an immediate reward \(r(s_{t},a,s_{t+1})\) at time \(t+1\), and updates the Q-value that represents this scenario as  
\[Q_{t+1}(s_{t},a_{t})\quad\quad\leftarrow\quad(1\ -\ \alpha)Q_{t}(s_{t},a_{t})\ +\ \alpha\big{[}r(s_{t},a_{t},s_{t+1})\ +\ \gamma\max_{a}Q_{t}(s_{t+1},a)\big{]}, \tag{6}\]  
where \(\alpha\in[0,1]\) denotes the learning rate, \(\gamma\) is a discount factor in reinforcement learning, and \(\gamma\max_{a}^{\prime}Q_{t}(s^{\prime},a^{\prime})\) represents the discounted reward. For agent \(i\), a two-dimensional look-up table with the dimensions \(\mathcal{S}\times\mathcal{A}\) is used to store the Q-values for each possible state and action. Thus, in a complex environment, the size of the Q-table increases exponentiallywith the number of states and actions. This introduces the curse of dimensionality, leading to longer convergence times to obtain the optimal action [29]. To address the curse of dimensionality problem, we propose a deep Q-learning-based approach where a neural network is used as a function approximator to generalize and approximate the value function. The proposed MADQN model is presented in Figure 2. The MADQN model consists of multiple agents interacting with the caching environment and their neighboring nodes. In our case, each edge server is considered an agent. In the multi-agent environment, each agent has a main network parameterized by (\(\theta_{m}\)) to estimate Q-values, a target network (\(\theta_{tg}\)) that copies the main network to provide a stable target for learning, and a replay buffer to store the agent's experiences. A deep neural network (DNN) can be implemented as a fully connected network consisting of an input layer, some hidden layers, and an output layer. The output layer produces the Q-values for all possible actions, and the action with the maximum Q-value is considered the optimal decision. Initially, at time \(t\), agent \(i\) observes the content request and its states, exchanges information (states (\(\mathbf{s}_{t}\)) and policies) with neighboring agents, and chooses an action (\(a_{t}\)). Here, it should be noted that agent \(i\) selects the neighboring node with the minimum traffic cost. For instance, consider a case where the content request cannot be satisfied by agent \(i\). Then, the agent will then look at the neighboring agents. Assume that the corresponding content is in two neighboring agents (\(\{j,q\}\in\mathcal{E}_{nb}\)) with traffic costs \(q_{E,ij}\) and \(q_{E,ij}\), where \(q_{E,ij}<q_{E,ij}\). In this case, the neighboring agent with traffic cost \(q_{E,ij}\) is selected. Figure 2: The proposed MADQN model. The agent takes an action based on two methods: exploitation and exploration. Exploitation occurs when the agent selects the action with the maximum Q-value as the optimal decision. Active exploration is also integrated into our DQN agent. With a probability (\(p\)) of \(\epsilon_{1}\), the agent is encouraged to take a stochastic step, while with a probability of \(\epsilon_{2}\), the agent follows a different cache replacement policy, such as LRU or LFU. Otherwise, the agent strictly follows the optimal action evaluated by the Q-values. The exploration policy can be written as  
\[a_{t} = \begin{cases}\arg\max_{a\in\mathcal{A}}Q(s_{t},a),&p=1-\epsilon_ {1}-\epsilon_{2}\\ \text{random }a\in\mathcal{A},&p=\epsilon_{1}\\ \text{Other replacement policy }a\in\mathcal{A},&p=\epsilon_{2}\end{cases}