# A Survey of Deep Learning for Data Caching in Edge Network
## 3 Deep Learning Outline
### Deep Reinforcement Learning (DRL)
#### 3.5.1 DNN as Critic (Value-Based)  
In value-based method, DNN does not get involved with policy decision but estimates the policy performance. Two functions are introduced for the measurement: \(V^{\pi}(s)\) represents the reward expectation of policy \(\pi\) starting from state \(s\); \(Q^{\pi}(s,a)\) illustrates the reward expectation of policy \(\pi\) starting from state \(s\) and taking action \(a\). In addition, \(V^{\pi}(s)\) is the expected value of \(Q^{\pi}(s,a)\). If we can estimate \(Q^{\pi}(s,a)\), the policy \(\pi\) can also be improved by choosing the action \(a^{*}\) hold \(Q^{\pi}(s,a^{*})\geq V^{\pi}(s)\). As aforementioned, the conventional Q Learning scheme is impractical when dealing with tasks containing enormous discrete states and actions. So, the DNN employed in agent is approximating function \(Q^{\pi}(s,a)\), where the inputs are state \(s\) and action \(a\) and output is the estimated value \(Q^{\pi}(s,a)\). There are some representative critic methods, like Deep Q Networks (DQN) [50,51] and its variants Double DQN [52], Dueling DQN [53], etc. As shown in the left side of Figure 5, the DNN \(Q\) is applied to estimate the state-action reward \(Q^{\pi}(s,a)\) in the DQN scheme. There are two main problems in training step if we only employ a single DNN: (1) the target is unstable, where the objective function/loss function for training DNN parameters depends on these parameters themselves; (2) these training samples are strongly correlated instead of independent (for example, samples of playing video games are generated from continuous frames), which makes the gradient descent towards a deterministic direction then the training process has a high risk to be not convergent. To deal with the aforementioned problems, DQN takes two measures accordingly: (1) Freezing target DNN \(Q^{\prime}\), in Figure 5, we put a lock symbol, indicating the parameters are fixed during several training steps, to keep the learning objective steady;(2) Experience replay memory \(D\), which consists of interactions between the client and the environment. Initially, the target DNN \(Q^{\prime}\) is a replica of DNN \(Q\). In each training episode, the client observes a current state from the environment, executes an action depending on the estimation of \(Q\), and receives a reward. Those interactions (i.e., the current state \(s\), action \(a\), reward \(r\), and state after transition \(s\)s) are stored in \(D\). Then, target DNN \(Q^{\prime}\) samples random batches of interactions from \(D\) and calculates the reward value \(q^{\prime}\) as follows:  
\[q^{\prime}= \begin{cases}r,&\text{if $s$ is a terminated state}\\ r+\gamma\cdot\max_{a^{\prime}}Q^{\prime}(ss,a^{\prime}),&\text{otherwise}\end{cases} . \tag{2}\]  
The loss function is defined as the mean square error: \((q^{\prime}-Q(s,a))^{2}\). After several steps, the parameters of \(Q^{\prime}\) would be updated by the setting from \(Q\). In Formula (2), the target DNN \(Q^{\prime}\) is responsible for both actions selection and evaluation. If any \(Q^{\prime}(ss,a^{\prime})\) is overestimated, the entire evaluation is overestimated accordingly since we always tend to select the action \(argmax_{a^{\prime}}Q^{\prime}(ss,a^{\prime})\). To avoid this upward bias, Double DQN decouples the calculation operation by applying the training DNN \(Q\) to selection actions, while employing the target DNN \(Q^{\prime}\) to evaluate actions. If \(Q\) is overestimated and select an improper action, \(Q^{\prime}\) will rectify the bias. As shown in the right side of Figure 5, the Dueling DQN modifies the DNN structure by splitting the \(Q\) evaluation into two channels: a scalar \(V\) representing the contribution of state only and a vector \(A\) describing the reward from both state and action. It is worth noting that the expectation of \(A\) is limited to be 0.