# Deep Learning Models Applied to Prediction of 5G Technology Adoption
## 3 Methodology
### Deep Learning Models
#### 3.1.3 Deep Reinforcement Learning  
Reinforcement learning is one of the best strategies for making decisions in real time. As it acts and recognises things in the world, it learns [41]. At each level of interaction, the agent chooses an action that modifies the environment based on the current state of the environment. Whether an activity is useful or not, the agent receives feedback in the form of a reward or a penalty.  
To describe RL, the research uses the notation of a Markov decision process (MDP) tuple, written as (S, A, R, P), where S is the current state of the environment, A is the current action being taken, R is the current reward, and P is the probability of a successful state transition. The goal of RL is to maximise the total discounted reward across all states by learning the optimal strategy [42]. That idea can be presented in Equation (12):  
\[J_{\pi^{*}}=\underset{\pi}{\text{max}}J_{\pi}=\underset{\pi}{\text{max}}E_{\pi} \left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right] \tag{12}\]  
where \(\pi^{*}\) and \(\pi\) denote the optimal policy and policy respectively, \(J_{\pi}\) denotes total expected reward, \(E_{\pi}[.]\) is the expectation based on the policy \(\pi\) and the transition probabilities, and \(\gamma\) is the discount factor in the range \([0,\,1)\). The agent becomes opportunistic about the present reward when \(\gamma=0\), and strives for long-term great reward when \(\gamma=1\). \(r_{t}\) denotes the reward at time \(t\).  
The achievable return for execution of an action \(a\) in a state \(s\) is represented by the value function \(Q(s,a)\). This can be updated according to each state-action pair till a given threshold turns out to be greater than the highest change in the value, as presented in Equation (13):  
\[Q(s,a)\leftarrow\sum_{s^{\prime}}p\big{(}s^{\prime}|s,a\big{)}\left[r\big{(}s, a,s^{\prime}\big{)}+\gamma\underset{a}{\text{max}}Q\big{(}s^{\prime},a^{ \prime}\big{)}\right] \tag{13}\]  
where \(p(s^{\prime}|s,a)\) denotes the transition probability from state \(s\) to state \(s^{\prime}\) when action \(a\) has been executed, and the reward is denoted by \(r(s,a,s^{\prime})\). Following the convergence of the algorithm, the optimal policy is achieved by performing a greedy action on each state \(s\). This is expressed in Equation (14):  
\[a^{*}=arg\underset{a}{\text{max}}Q^{*}(s,a),\ (\forall s\epsilon S) \tag{14}\]  
In situations where the system does not have prior knowledge about the environment, optimal policies can be achieved by a type of RL algorithm known as Q-learning. Given \(\alpha_{t}\) as the learning rate, such that when \(\alpha_{t}=0\), the agent becomes incapable of learning, and when \(\alpha_{t}=1\), the agent only considers the most recent information. The updating rule of Q-learning is provided in Equation (15):  
\[Q(s_{t},a_{t})=(1-\alpha_{t})Q(s_{t},a_{t})+\alpha_{t}\Big{(}r_{t+1}+\gamma \underset{a}{\text{max}}Q\big{(}s_{t+1},\ a^{\prime}\big{)}\Big{)} \tag{15}\]  
This implies at time step \(t\), state \(s_{t}\) is observed by the agent and an action \(a_{t}\) is chosen, reward \(r_{t+1}\) is received by the agent for execution of the action \(a_{t}\). Q-learning always tries to choose the optimal action by considering the state-action pair with the best Q value. RL algorithms are very good for solving various problems, especially problems relating to messaging and mobile networks [43].