## 5 Q-Leaning Based Data Caching  
An illustration of RL-based data caching is provided in Figure 2. The efficiency of the data caching mechanism is measured by determining the delay experienced by the users accessing the data from the serving SBS. Therefore, we devise Q-learning based data caching at the SBS level. Q-learning constituted by state, action, and reward. * State: State (\(\mathcal{M}\)) is the set of cache states of serving SBS \(s\), \(\mathcal{M}=\{m_{s}:\forall s\in\mathcal{S}\}\), where \(m_{s}\) represents the contents placement of SBS \(s\) that is represented as \(m_{s}=(m_{s}(1),m_{s}(2),\ldots,m_{s}(f),\ldots,m_{s}(F))\). * Action: Action (\(\mathcal{A}\)) is the set of actions to optimize caching at SBS \(s\). We define action set as \(a_{s}=(a_{s}(1),a_{s}(2),\ldots,a_{s}(f),\ldots,a_{s}(F))\) to perform an adjustment of cached contents in the SBS \(s\). * Reward: \(\psi(m,a)\) is the reward provided to the SBS in return for performing an action \(a\) during state \(m\), which is an expected reduction in data access delay from the cache resource. The process of caching in SBS \(s\) at time \(t\) is described as
* The current cache set of SBS \(s\) is sensed at time \(t\), i.e., \(m_{st}\in\mathcal{M}\). * Based on the current cache state \(m_{st}\in\mathcal{M}\), an action \(a_{st}\in\mathcal{A}\) is selected by SBS \(s\). * The system transfers to a new state \(m_{s(t+1)}\in\mathcal{M}\) in result of action \(a_{st}\in\mathcal{A}\). Moreover, in result of this transition a reward \(\psi_{st}=\psi(m_{st},a_{st})\) is obtained. * The reward is provided to the SBS \(s\) and the whole process is repeated. Our objective is to enhance rewards to reduce data access delay via cached resources. To increase rewards, it is necessary to perform optimal actions. A state-value function \(V\) is defined to determine data caching efficiency in the following manner:  
\[V=\sum_{t=0}^{\infty}\alpha^{t}\psi^{t}, \tag{12}\]  
where \(\alpha\in(0,1)\) represents the discount factor to determine the effect of future rewards on the current action decisions. We utilize Q-learning to determine the optimal caching policy \(\zeta(s)\) for SBS \(s\), which corresponds to \(V\) without prior system information. Since the cache decision is taken in an independent manner, optimal \(V_{s}\) for SBS \(s\) is determined as follows:  
\[V_{s}=\sum_{t=0}^{\infty}\alpha^{t}\psi_{st}. \tag{13}\]  
Moreover, \(V_{s}\) is estimated by defining Q-value for every state-action pair. An optimal Q-value for SBS \(s\) is defined in the following manner:  
Figure 2: An illustration of RL based data caching in wireless network. \[Q_{s}(m_{st},a_{st})=\{\psi(m_{st},a_{st})\}+\alpha\sum_{m_{s(t+1)} \in\mathcal{M}}P_{m_{st},m_{s(t+1)}}(a_{mt}) \tag{14}\] \[\max_{a_{s(t+1)}\in\mathcal{A}}Q\Big{(}m_{s(t+1)},a_{s(t+1)}\Big{)},\]  
where \(m_{s}\) represents the current state of SBS \(s\), \(m_{s(t+1)}\) represents the state transfer in result of performing an action \(a_{s}\) at state \(m_{s}\), and \(P_{m_{s},m_{s(t+1)}}\) represents the probability of system transition from state \(m_{s}\) to \(m_{s(t+1)}\). \(\psi(m_{st},a_{st})\) is evaluated by calculating the difference between gain of caching fresh content and loss of removing old content as follows:  
\[v_{st}^{f^{+}}=\sum_{f=1}^{F}r_{st}^{f^{+}}, \tag{15}\]  
where \(v_{st}^{f^{+}}\) represents increase in data access via cache resources in result of caching new content. However, loss of replacing old content is evaluated as  
\[v_{st}^{f^{-}}=\sum_{f=1}^{F}r_{st}^{f^{-}}, \tag{16}\]  
where \(v_{st}^{f^{-}}\) represents decrease in data access via cached resources as a result of removing previous content.