## 4 DRTO: Deep Reinforcement Learning for Task Offloading
### Generate the Offloading Location  
As shown in the upper part of Figure 3, in each time frame, the fully connected DNN takes the current channel gain \(\mathbf{h}\) as input, and generates a relaxed offloading location \(\hat{\mathbf{x}}=[\hat{x}_{1},\hat{x}_{2},\ldots,\hat{x}_{N}]\) (each entry is relaxed into \([0,1]\) interval). Then, the relaxed location \(\hat{\mathbf{x}}\) is quantized into \(K\) binary locations. Given a candidate location \(\mathbf{x}_{k}\), DRTO solves a bandwidth allocation convex problem and obtains the offloading cost. Subsequently, the optimal offloading location \(\mathbf{x}^{*}\) is selected according to the minimal offloading cost.  
Although the mapping from the channel state to the offloading location is unknown and complex, thanks to the universal approximation theorem [36], we adopt a fully connected DNN to approximate this mapping. The DNN is characterized by the weights that connect the hidden neurons, and composed of four layers, namely the input layer, two hidden layers and output layer. Here, we respectively use ReLU and the sigmoid activation function in the hidden layers and output layer; thus, each entry of the output relaxed offloading location satisfies \(\hat{x}_{n}\in(0,1)\).  
Then, the \(\hat{\mathbf{x}}\) is quantized into \(K\) candidate binary offloading locations, where \(K\in[1,2^{N}]\). Intuitively, a larger \(K\) creates higher diversity in the candidate offloading location set, thus increasing the chance of finding the global optimal offloading location but resulting in higher computational complexity. We adopt an order-preserving quantization method proposed in [37] for the trade-off of performance and complexity. In order-preserving quantization, \(K\) is relatively small, but the diversity of the candidate offloading locations is guaranteed. Its main idea is preserving the order when quantization, i.e., for each quantized location \(\mathbf{x}_{k}=[x_{k,1},x_{k,2},\ldots,x_{k,N}]\), \(x_{k,n}\leq x_{k,m}\), should be held if \(\hat{x}_{n}\leq\hat{x}_{m}\) for all \(n,m\in\{1,2,\ldots,N\}\). Specifically, a series of \(K\) quantized locations \(\{\mathbf{x}_{k}\}\) are generated as the following:  
(1) Each entry of the 1 st binary offloading location \(\mathbf{x}_{1}\) is given by  
\[x_{1,n}=\begin{cases}1&\hat{x}_{n}>0.5,\\ 0&\hat{x}_{n}\leq 0.5.\end{cases}n=1,2,\ldots,N \tag{11}\]  
(2) As for the remaining \(K-1\) offloading locations, we first sort each entry of \(\hat{\mathbf{x}}\) according to its distance to 0.5, i.e., \(|\hat{x}_{(1)}-0.5|\leq|\hat{x}_{(2)}-0.5|\leq\cdots\leq|\hat{x}_{(N)}-0.5|\), where  
Figure 3: The diagram of DRTO.  
\(\hat{x}_{(n)}\) denotes the sorted \(n\)-th entry. Hence, each entry of the \(k\)-th offloading location \(x_{k}\), \(k=2,3,\ldots,K\) is given by  
\[x_{k,n}=\begin{cases}1&\hat{x}_{n}>\hat{x}_{(k-1)},\\ 1&\hat{x}_{n}=\hat{x}_{(k-1)}\text{ and }\hat{x}_{(k-1)}\leq 0.5,\\ 0&\hat{x}_{n}=\hat{x}_{(k-1)}\text{ and }\hat{x}_{(k-1)}>0.5,\\ 0&\hat{x}_{n}<\hat{x}_{(k-1)}.\end{cases} \tag{12}\]  
Here, we obtain \(K\) candidate offloading locations; given a candidate offloading location \(x_{k}\), the original offloading cost minimization problem \(\mathcal{P}\) is transformed into a convex problem on \(\alpha\)  
\[\mathcal{P}^{\prime}:\min_{\alpha}F(x_{k},\alpha) \tag{13a}\] \[s.t.\quad 0\leq\sum_{n=1}^{2N}\alpha_{n}\leq 1 \tag{13b}\]  
which can be solved by a convex optimization tool like CVXPY [38]. Then, we obtain the optimal bandwidth allocation \(\alpha_{x_{k}}^{*}\) and minimum offloading cost \(F\Big{(}\mathbf{x}_{k},\mathbf{\alpha}_{x_{k}}^{*}\Big{)}\) with the given candidate offloading location \(x_{k}\). By repeatedly solving the problem \(\mathcal{P}^{\prime}\) for each candidate offloading location, the best offloading location is selected by  
\[\mathbf{x}^{*}=\arg\min_{\{x_{k}\},k=1,2,\ldots K}F\Big{(}\mathbf{x}_{k},\mathbf{\alpha}_{ x_{k}}^{*}\Big{)} \tag{14}\]  
along with its corresponding optimal bandwidth allocation \(\mathbf{\alpha}^{*}\).