## 4 UbiPriSEQ: The Proposed Framework
### Device Module (DevM)
#### 4.3.5 DevM: Q-Value  
DevM comprises a DQN that consists of an input layer, two hidden convolutional layers, a rectifier linear unit layer, and two fully connected output layer. The weights in the CNN are updated using gradient descent optimization [58] based on the previous attack experiences.  
The state transits from the state \(\mathbf{s_{t}}\) to \(\mathbf{s_{t+1}}\) when the agents offload data to the selected fog node at state \(\mathbf{s_{t}}\). Based on different offloading experiences denoted by \((\mathbf{s_{t}},\mathbf{a_{t}},ud_{d}^{(t)},\mathbf{s_{t+1}})\), the Q-function update of the state-action pair, using general RL based technique can be computed as  
\[Q(\mathbf{s_{t}},\mathbf{a_{t}})\leftarrow(1-\alpha)Q(\mathbf{s_{t}},\mathbf{a_{t}})+\alpha \big{(}ud_{d}^{(t)}+\gamma\max_{\mathbf{\overline{u_{t}}}\in A}Q(\mathbf{s_{t+1}},\mathbf{ \overline{a_{t}}})\big{)}, \tag{3}\]  
where \(\alpha\in(0,1]\) is the learning rate denoting the weight of the current experience and \(\gamma\in[0,1]\) denotes the view of the agents regrading the future rewards. However, the DQN computes the Q-value for each time \(t\) by learning from the replay memory (Experience Sequence). The experience sequence (replay memory) \(\Phi_{t}\) consists of the current system state \(\mathbf{s_{t}}\) and the previous \(K\) state-action pairs, given by:  
\[\Phi_{t}=\left((\mathbf{s_{t-K}},\mathbf{a_{t-K}}),\cdots,(\mathbf{s_{t-1}},\mathbf{a_{t-1}}), \mathbf{s_{t}}\right). \tag{4}\]  
The experience sequence \(\Phi_{t}\) is fed as input to the first convolutional layer in DQN. The output of the first convolutional layer is then passed through a rectified linear unit (ReLU) as an activation function. The output of ReLU is passed through the second convolutional layer. The outputs from second convolutional layer is then passed to fully connected layers to obtain the Q-value for all the possible actions \(|A|\) (offloading strategies) at current sequence \(\Phi_{t}\). Subsequently, in time \(t\), DevM takes an action for offloading based on the Q-value with a probability \(1-\epsilon\), or takes a random action with a minute probability \(\epsilon\) (to explore various rewards and reach global optimum reward).  
Once the results are obtained from the offloaded computation the agents update their reward using Equation (2) and the offloading experience \((\Phi_{t},\mathbf{a_{t}},ud_{d}^{(t)},\Phi_{t+1})\) is stored in the memory pool denoted by \(\mathcal{P}\). As per the experience replay technique, the agents randomly select an experience from the pool to update the CNN weight \(\theta_{t}\). The mean squared error is used to calculate the loss between the networks output and the optimal Q-value. The Q-function for each state-action pair in a DQN is given below:  
\[Q(\Phi_{t},\mathbf{a_{t}},\theta_{t})=\mathbb{E}\left[ud_{d}^{(t)}+\gamma\max_{ \mathbf{\overline{u_{t}}}\in A}Q(\overline{\Phi_{t}},\mathbf{\overline{a_{t}}})|\Phi_ {t},\mathbf{a_{t}}\right], \tag{5}\]  
where \(\overline{\Phi_{t}}\) is the next state sequence based on the selection of the offloading policy \(\mathbf{a_{t}}\) at state \(\Phi_{t}\) and \(\theta_{t}\) is the weight of DNN at time \(t\). Furthermore, every \(T\) time steps, the Q-network is duplicated to obtain a target network to obtain the target Q-values so that updated DNN weights can be obtained. The target optimal Q-function \(Q_{opt}(\Phi_{t},\mathbf{a_{t}},\theta_{t})\) is given as follows:  
\[Q_{opt}(\Phi_{t},\mathbf{a_{t}},\theta_{t})=\left[ud_{d}^{(t)}+\gamma\max_{\mathbf{ \overline{a_{t}}}\in A}Q(\overline{\Phi_{t}},\mathbf{\overline{a_{t}}},\theta_{ t-1})|\Phi_{t},\mathbf{a_{t}}\right] \tag{6}\]  
and hence the loss is computed as follows [59]:  
\[\mathcal{L}(\theta_{t})=Q(\Phi_{t},\mathbf{a_{t}},\theta_{t})-Q_{opt}(\Phi_{t},\bm {a_{t}},\theta_{t}). \tag{7}\]This explains the loss calculation and weight updating in DQN as well as \(Q\)-value compuations using DQN.