# Delay-Tolerance-Based Mobile Data Offloading Using Deep Reinforcement Learning
## 2 Delay-Tolerance-Based Mobile Data Offloading Using Deep Reinforcement Learning
### Transmission Control by Deep Reinforcement Learning  
We determine the transmission control of the UE by using deep reinforcement learning, and the RL server learns using a double deep Q-network (DDQN) [20]. DDQN is a reinforcement learning method that uses deep learning with Q-learning [21]. Q-learning is focused on maximizing a function \(Q(s,a)\). \(Q(s,a)\) represents the value of action \(a\) taken in state \(s\). Q-learning updates \(Q(s_{t},a_{t})\) at time \(t\) as follows:  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha\Big{(}r+\gamma\underset{a^{ \prime}}{\max}Q(s_{t+1},a^{\prime})-Q(s_{t},a_{t})\Big{)} \tag{1}\]  
where \(\alpha\) is the learning rate, and \(\gamma\) is the discount factor. Each parameter is defined as \(0\leq\alpha,\ \gamma\leq 1\). Furthermore, Q-learning selects the action with the highest Q-value. If we use Q-learning to perform learning, it is necessary to prepare a table function \(Q(s,a)\) which is a combination of all states \(s\) and actions \(a\) in advance. However, it is difficult to prepare this table function because there are innumerable situations in mobile networks. In the case of such a problem, deep Q-network (DQN) is used, which approximates \(Q(s,a)\) with a neural network [15]. Although a method for approximating \(Q(s,a)\) with a neural network has been previously proposed, it was known that the learning diverges as the number of parameters increases. Because the correlation between the data is high, the policy of the selecting action is changed significantly on updating \(Q(s,a)\). In order to prevent the divergence of the learning, DQN uses experience replay and neural fitted Q. Experience replay stabilizes the learning by using randomly sampled states and actions of the past. Neural-fitted Q fixes parameters to be approximated with a neural network for stabilizing the learning. DQN updates the Q value using the loss function \(L(\theta)\) as follows:  
\[L(\theta)\ =\ \frac{1}{2}(r+\gamma\underset{a^{\prime}}{\max}Q(s_{t+1},a^{ \prime})-Q(s_{t},a_{t}))^{2} \tag{2}\]  
However, DQN overestimates the action because \(Q(s,a)\) selection and \(Q(s,a)\) estimation models are the same. In contrast, the DDQN uses different models for the selection and estimation. Thus, the overestimation of the action in the case of DDQN is reduced compared with that in the case of DQN. In addition, we apply dueling-network architecture to DDQN [22]. Dueling-network represents \(Q(s,a)\) as follows:  
\[Q(s,a)\ =\ V(s)+A(s,a) \tag{3}\]  
\(V(s)\) is a state value function and \(A(s,a)\) is a state-dependent action advantage function. \(V(s)\) shows the worth of a particular state \(s\), and \(A(s,a)\) represents a relative measure of the importance of each action \(a\). On using dueling-network architecture, it is possible to directly express a value of the state without using the value of the action. Furthermore, it is confirmed that the convergence of the state value occurs faster as a result. Hence, we use DDQN and dueling-network architecture to construct a transmission rate control model. Defining the states, actions, and reward is essential for learning using deep reinforcement learning. Table 1 lists the learning parameters of the proposed method at \(t\). In our method, we construct a transmission rate control model for each eNB. Additionally, when multiple UE exists in the same eNB, the MDOP server aggregates UE information from each UE. Besides, the RL server determines the QoSlevel for each UE and performs bandwidth allocation. We define the environment information as state \(s\). The RL server periodically receives environment information from the MDOP server. The environment information includes UE information and connected eNB information. The UE information includes the remaining amount of content (\(content_{n}\)), the content delay tolerance (\(content_{dt}\)), and the destination eNB ID. Moreover, the eNB information is that gathered by the MDOP server, such as the available bandwidth and current load of the eNB.