## 4 Problem Definition and Optimization Approaches
### Deep Q-Learning Approach
#### 4.2.3 Deep Q-Learning Algorithm  
The domain of states in our model for mobile and edge devices are \(\mathbb{N}^{6}\) and \(\mathbb{N}^{7}\), respectively, and some of the values such as \(l_{w}\) can become relatively large. As a result, using a Q-table to approximate the total reward for each state \(s_{t}\) and action \(a_{t}\) is inefficient, taking a huge amount of time for the table to converge. This, in turn, renders this method unusable, as our model should quickly adapt to changes in a changing environment. To overcome this issue, we can use an artificial neural network (ANN) to replace our Q-table. That is, we feed a state \(s_{t}\) and an action \(a_{t}\) as the input of our ANN and receive one real value as the output, which is our Q-value \(Q_{\theta}(s_{t},a_{t})\). The parameters in our ANN collectively are denoted by \(\theta\). To train our ANN, we define the loss function as [6]  
\[loss(\theta,\theta^{\prime})=\mathbb{E}_{(s_{t},a_{t}s_{t+1},r)\sim U(T_{b})} [(r+\gamma.\max_{a\in A}Q_{\theta^{\prime}}(s_{t+1},a)-Q_{\theta}(s_{t},a_{t} ))^{2}] \tag{32}\]  
where \(\theta\) is the current value of our ANN parameters and \(\theta^{\prime}\) is a snapshot of our ANN parameters updated after every \(n\) number of trainings. ANNs are known to be difficult to converge when used as the Q-function; hence, the deep Q-learning model employs two techniques to alleviate this issue [6]. The first technique is to break the correlation between consecutive transitions by randomly sampling from a pool of transitions. This method is known as experience replay. The second technique is to use a snapshot of network parameters \(\theta^{\prime}\) when calculating the loss function, which should reduce the correlation with the current state of network parameters \(\theta\). With regard to the MDP, to calculate the state of a mobile or edge device, we need information about the workload and task counts at the edges to which the node has connections. To satisfy this requirement, each edge device periodically sends its workload and task counts to all the devices to which it has a connection. Algorithm 3 illustrates this process. The algorithm repeats for the lifetime of the network. At each iteration, first, it retrieves \(e\) and \(q\), which are the unique identifier and task count in the execution queue of the corresponding edge device. Then, it sends this pair \((e,q)\) to every node it has a connection to. Lastly, it waits for \(c\) seconds before repeating the process. Here, \(c\) is an arbitrary constant; if the network has a high data rate, \(c\) can be very small, and if the data rate is low, \(c\) can have a larger value. When \(c\) has a smaller value, mobile and edge devices will have a more recent view of the state of the network and will be able to make better decisions. The procedure for when a task arrives at a mobile device is depicted in Algorithm 4. The procedure is repeated until all tasks have arrived. When a task arrives, it is stored in variable \(t\). ```
1:repeat
2: wait until a task arrives
3:\(t\leftarrow\) newly arrived task
4:\(s\leftarrow\) retrieve current state updated with task \(t\)
5:\(x\sim U(0,1)\)
6:if\(x<e\)then
7:\(a\leftarrow\) randomly choose from \(\{0,1\}\)
8:else
9:\(a\gets argmax_{a}(Q_{\theta}(s,a))\) where \(a\in\{0,1\}\)
10:endif
11:if\(a=0\)then
12: insert task \(t\) into local execution queue
13:else
14: insert task \(t\) into transmission queue
15:endif
16:\(s^{\prime}\leftarrow\) retrieve current state
17: store \((t,s,a,s^{\prime})\) into partial transition buffer
18:until all tasks have arrived
```  
**Algorithm 4** Task arrival algorithm for mobile devices  
Next, an updated state of the mobile device, illustrated in Equation (28), is recorded using the arrived task into variable \(s\). Information received from the connected edge device using Algorithm 3 is used to calculate the value for \(r_{q}\). Additionally, the values of \(l_{q}\) and \(t_{q}\) are increased by one.