## 5 Proposed Algorithm
### DQN-Based Ellipsoidal Trajectory Optimization  
RL has gained significant attention in recent times for solving complex optimization problems. Instead of using static data, an agent interacts with the environment, explores its surroundings, and learns through trial and error to develop a policy that maximizes the total reward:  
\[\pi^{*}=\arg\max_{\pi}\sum_{t=0}^{\infty}\gamma^{t}r_{t}, \tag{34}\]  
where \(\gamma\) denotes the discount factor, reducing the influence of future rewards on the agent's decision making. In RL, particularly in traditional Q-learning [48], the Q-value function serves as a tool to evaluate the effectiveness of selecting a specific action in a particular state. It is updated iteratively, using the Bellman equation:  
\[Q(s,a)=\mathbb{E}\Big{[}r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\mid s,a\Big{]}. \tag{35}\]  
However, Q-learning records each combination of state \(s\) and action \(a\) in a Q-table, which is impractical for many real-world scenarios. The dimensions of the Q-table rise dramatically as the sizes of the state and action spaces grow, and Q-learning needs a huge memory space to save them. Deep-Mind [49] introduced the DQN to overcome this issue by utilizing function approximation through an NN, to estimate the Q-values for individual state-action combinations. Considering the large state space and high-dimensional action space in our UAV-assisted MEC system, we employ the DQN framework to address the MDP problem discussed earlier, called DQN-based ellipsoidal trajectory optimization (DETO). The framework is depicted in Figure 2 and consists of two NNs (the Q-network and the target Q-network) and a memory dataset to store the experiences. Each network has multiple layers that are connected to one other, to correlate the state and action. The input of the NN is used to represent the current state of the environment and the probability assigned to each possible action is represented by the output of the networks. The memory dataset is depicted as a queue-like data structure. The agent creates new transitions by interacting with the environment and stores them in memory. When the memory reaches its limit, it begins to remove the earliest transitions, to make way for the newest ones. Therefore, the most recent transitions are kept in memory while removing the oldest ones, which primarily helps in tracking the agent's most recent interactions with the environment. The DETO training steps are presented in Algorithm 1. At the beginning of each simulation during the learning process, a number of elements, including the capacity of the memory dataset and the parameters of both the Q-network and the target network, are initialized. For the NN learning process, these parameter values serve as starting points. This initialization is performed through lines 2, 3, and 4 of the algorithm. Note that the DQN uses a dual-network strategy to estimate the current Q-value and target Q-value, where both the Q-network and the target Q-network have the same neural architecture. However, these networks have a different set of parameter values to make these estimations. Each training episode begins with the agent starting from an initial state, and each episode runs for a given number of steps (lines 5, 6, and 7). The initial deployment positions for each UAV-MEC server are determined using the K-means clustering technique and are integrated into the initial state information. In each step of the training episode, the NN receives the current state (line 8) as an input value, and it returns the Q-values of every action. The agent then selects an action using an epsilon greedy algorithm (line 9). Due to the lack of background knowledge about the environment, the agent emphasizes exploration at first. To do this, it occasionally selects random actions under the guidance of a probability parameter, \(\epsilon\). With the aid of this exploratory strategy, the agent discovers the dynamics of the environment, gathers important data, and stays away from locally optimal solutions.