## 5 Proposed Algorithm
### DQN-Based Ellipsoidal Trajectory Optimization
The agent eventually reduces the use of random actions as it becomes more familiar with the environment. This reduction acts as a switch towards exploitation, forcing the agent to choose the actions that work best. This adjustment aids the agent in finding the ideal balance between utilizing what it has already determined as best and attempting new things, enabling it to get the best results over time by combining both previously successful strategies and new information. The agent takes action to change the center position, length of a radius, or rotation angle of the ellipsoidal path, to determine the position, size, and shape of the ellipse. By performing the above actions, it observes the next state (line 10). This state transition leads to a change in the UAV-MEC server's position. Consequently, the associations between MDs and UAV-MEC servers also change. The association between each MD and UAV-MEC server is determined based on throughput. In particular, for each MD, the UAV-MEC server that yields the highest throughput is selected. Subsequently, after computing  
Figure 2: The DETO framework. the association between MDs and the UAV-MEC server, the agent obtains the reward (lines 11, 12). ```
0: Initial center position, radius, rotation angle of the ellipsoidal trajectory of each UAV-MEC server
0: Optimal center position, radius, rotation angle of the ellipsoidal trajectory to minimize the system objective
1:for simulation \(=1\) to \(N\)do\(\to N:\) Total simulation number
2: Initialize memory dataset \(E\) with capability \(B\)
3: Initialize the \(Q\)-network's parameter values \(\theta\)
4: Initialize the target \(Q\)-network's parameter values \(\theta^{-}\leftarrow\theta\)
5:for episode \(=1\) to \(L\)do\(\to L:\) Total episode number
6: Initial state
7:for\(t=1\) to \(T\)do\(\to T:\) Max length of episode
8: Current state \(s_{t}\)
9: Select an action according to \(\epsilon-\)greedy algorithm
10: Perform action \(a_{t}\) and observe the next state \(s_{t+1}\)
11: Compute the association of MDs with the UAV-MEC server
12: Obtain reward \(r_{t}\), using (33)
13: Store current experience \((s_{t},a_{t},r_{t},s_{t+1})\) in \(E\)
14: Randomly sampling a batch of experiences \((s_{j},a_{j},r_{j},s_{j+1})\) from \(E\)
15: Compute target \(Q\)-value for the next state, using target \(Q\)-network
16: Apply a gradient descent step with regard to the network parameter values \(\theta\), to reduce the squared difference between target \(Q\)-value and current \(Q\)-value, using (36)
17: Periodically, reset the target \(Q\)-network's parameter values from the \(Q\)-network: \(\theta^{-}\leftarrow\theta\)
18: Set \(s_{t}=s_{t+1}\)
19:endfor
20:endfor
21: Test the model and save the results
22:endfor
23: choose the best result
```  
**Algorithm 1** DQN-based ellipsoidal trajectory optimization (DETO)  
To make the learning process stable and to enhance sample effectiveness, the agent stores each transition that includes state, action, reward, and the next state in memory dataset \(E\) (line 13). When training Q-network parameters, a batch of transitions from the memory dataset is randomly chosen by the agent (line 14). This sampling process is used to break the correlation between consecutive transitions, ensuring the training data are sufficiently diverse, which aids the network's ability to acquire useful weights that can effectively handle a variety of data values. During training, the aim is to reduce the gap between the target Q-network's output, \(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i-1})\), and the Q-network's output, \(Q(s,a;\theta_{i})\), which is achieved by utilizing the mean squared error (MSE) loss function (lines 15, 16):  
\[L(\theta_{i})=\mathbb{E}[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime}; \theta_{i-1})-Q(s,a;\theta_{i})^{2}]. \tag{36}\]  
The parameter values of the Q-network are updated using the Adam optimizer with an initial learning rate of 0.001. Then, the parameter values from the Q-network are copied periodically to the target Q-network (line 17).