## 3 Methodology
### Q-Learning Integration in GA Mutation  
Traditional GA mutation works with random changes, frequently resulting in inferior local solutions because direct feedback mechanisms are lacking. In our method, we incorporate Q-learning to optimize such mutations according to environmental feedback in real time, thus enhancing the efficiency and adaptability of the optimization process. Every drone is considered an independent agent that learns positional changes incrementally to maximize its fitness through several iterations. Q-learning decision making is encoded by the Q-table, which, at all times, is updated based on the following:  
State Representation: The state \(\mathrm{S_{t}}\) at time t encapsulates the drone's positional coordinates, its coverage contribution, connectivity status, and energy level. This multi-dimensional state representation ensures that decisions consider both immediate and long-term fleet objectives. Action Space: The action set \(\mathrm{A_{t}}\) includes small positional adjustments \(\mathrm{\delta_{x}}\), \(\mathrm{\delta_{y}}\), and \(\mathrm{\delta_{z}}\) within predefined bounds, ensuring that drones refine their location with minimal displacement while maintaining coverage efficiency. In this context, \(\mathrm{\delta_{x}}\), \(\mathrm{\delta_{y}}\), and \(\mathrm{\delta_{z}}\) refer to the discretization step sizes in the x, y, and z axes, respectively. These values determine the granularity of the positional adjustments made by drones during Q-learning-based mutation. For all experiments, we set \(\mathrm{\delta_{x}}=\mathrm{\delta_{y}}=\mathrm{\delta_{z}}=10\) m. This value was selected empirically to strike a balance between movement flexibility and computational efficiency. Smaller step sizes would result in more precise but slower convergence, while larger values could overlook fine-grained optimizations. Reward Function: A tailored reward function \(\mathrm{R_{t}}\) incentivizes actions that lead to improved fleet performance. The function formulation can be seen in Equation (5). In Equation (4), the parameters \(\mathrm{\alpha}\), \(\mathrm{\beta}\), \(\mathrm{\gamma}\), and \(\mathrm{\lambda}\) are scalar weights that control the contribution of different components in the reward function: \(A_{t}\)--penalizes overlapping coverage between UAVs, \(L_{t}\)--rewards the preservation of inter-UAV links, \(D_{t}\)--promotes a shorter average communication distance, and \(H_{t}\)--encourages UAVs to maintain a minimum height above ground level to avoid obstacles or interference. For all experiments, we empirically set \(\alpha=0.3\), \(\beta=0.4\), \(\gamma=0.2\), and \(\lambda=0.1\) to achieve a balanced trade-off between spatial distribution, connectivity, and efficiency. These values were chosen based on preliminary trials and sensitivity to reward shaping:  
\[\mathrm{R_{t}}=\alpha\mathrm{C_{t}}+\beta\mathrm{N_{t}}+\gamma\mathrm{E_{t}}- \lambda\mathrm{P_{t}}, \tag{5}\]  
where \(\mathrm{C_{t}}\) represents the increase in coverage, \(\mathrm{N_{t}}\) quantifies connectivity improvements, \(\mathrm{E_{t}}\) denotes energy conservation, and \(\mathrm{P_{t}}\) penalizes excessive movement that may destabilize the fleet. The Q-value update follows the standard Bellman equation which can be seen in Equation (6). In this formulation, the value of the current state-action pair \(\mathrm{Q(S_{t},\,A_{t})}\) is updated based on the immediate reward \(\mathrm{R_{t}}\) and the discounted maximum estimated Q-value from the next state \(\mathrm{S_{t+1}}\):  
\[\mathrm{Q(S_{t},\,A_{t})}\leftarrow\mathrm{Q(S_{t},\,A_{t})}+\eta\,[\mathrm{ R_{t}}+\delta\max_{a}\mathrm{Q(S_{t+1},\,a)}-\mathrm{Q(S_{t},\,A_{t})}], \tag{6}\]  
where \(\eta\) is the learning rate and \(\delta\) is the discount factor applied only to the estimated future reward. The term \(\max_{a}\mathrm{Q(S_{t+1},\,a)}\) represents the off-policy nature of Q-learning, in which the agent updates its estimates assuming the optimal action will be taken in the future state.