## 3 Proposed Algorithm  
In this section, we propose an RL algorithm to optimize the objective function considering the evolving network dynamics and the complexities of multidimensional action. First, the data placement problem is modeled through an MDP, then Q-learning, a model-free RL approach, is exploited to determine the action selection policy for optimizing caching decisions. The Q-learning approach optimizes caching without any prior knowledge about the underlying dynamics of the system. This makes Q-learning suitable to optimize caching through learning and adjusting strategies based on the obtained rewards in the dynamic environment. Therefore, we define a tuple \(\{\  
Our objective is to increase the rewards associated with optimizing network performance through optimal data placement. We propose a mechanism aimed to improve the rewards in terms of minimizing delay in the following manner:  
* When a cache \(c\) receives requests for content \(m\), the cache state \(s^{m}_{it}\in\mathcal{S}\) is checked to determine whether the content \(m\) is cached or not during any given time \(t\). * After observing the cache state, an action \(a^{m}_{ct}\) is executed, leading to a transition to the next state \(s^{m}_{c(t+1)}\in\mathcal{S}\). A reward \(R^{m}_{ct}\) is also achieved. * This complete procedure is repeated. To maximize the reward, it is imperative to pick the optimal action from \(\mathcal{A}\), which is a function \(\omega^{*}:\mathcal{S}\rightarrow\mathcal{A}\) defined to map a state \(s\in\mathcal{S}\) to an action \(a\in\mathcal{A}\). Hence, \(\omega^{*}(\cdot)\) determines which action \(a^{m}_{ct}\in\mathcal{A}\) is beneficial to be executed during the state \(s^{m}_{ct}\) in order to increase the reward \(R^{m}_{ct}\). A state-value function \(V\) is defined to evaluate the data placement efficiency as follows:  
\[V=\sum_{t=0}^{\infty}\alpha^{t}R^{m}_{ct}, \tag{16}\]  
where \(\alpha\in(0,1)\) denotes the discount parameter, which controls the long-term rewards resulting from the current actions to capture the system dynamics. We exploit Q-learning to determine an optimal \(\omega^{*}(\cdot)\) to improve the accuracy of \(V_{c}\), which is determined as follows:  
\[V_{c}=\sum_{t=0}^{\infty}\alpha^{t}R^{m}_{ct}. \tag{17}\]  
To optimize the \(V_{c}\) estimation accuracy, the Q-value for each state-action pair of a SBS \(c\) is determined by  
\[ \begin{split} Q_{c}(s^{m}_{ct},a^{m}_{ct})&=\{R(s^ {m}_{ct},a^{m}_{ct})\}+\alpha\sum_{s^{m}_{c(t+1)}\in\mathcal{S}}P\big{(}s^{m}_{ c(t+1)}|s^{m}_{ct},a^{m}_{ct}\big{)}\\ &\max_{a^{m}_{c(t+1)}\in\mathcal{A}}\mathcal{Q}\big{(}s^{m}_{c(t+ 1)},a^{m}_{b(t+1)}\big{)},\end{split}  \tag{18}\]  
where \(s^{m}_{ct}\) and \(s^{m}_{c(t+1)}\) denote the current and next cache state, respectively, in response to the execution of action \(a^{m}_{ct}\) during state \(s^{m}_{ct}\). Moreover, the probability of state transition is determined by \(P\big{(}s^{m}_{c(t+1)}|s^{m}_{ct},a^{m}_{ct}\big{)}=P\big{(}v^{m}_{c(t+1)},x^{ m}_{c(t+1)}|v^{m}_{ct},x^{m}_{ct},a^{m}_{ct}\big{)}\). We exploit the potential of the greedy policy to optimize action selection in order to maximize the rewards. The \(c\) greedy policy performs a trade-off between the exploration and exploitation of actions. During the exploitation phase, those actions are selected that can improve the long-term rewards. Since this approach may halt the system's performance, it is necessary to also check the impact of new actions that can further improve long-term rewards. Consequently, exploration is performed to find new actions that can significantly enhance the achievable rewards in terms of network delay minimization. Hence, the \(\epsilon\) greedy policy provides a balance between exploitation and exploration by a factor of \(\epsilon\in(0,1)\)[40].