## 4 Problem Definition and Analysis
### Markov Decision Process Modeling  
This section discusses the construction of a MDP model for electromagnetic spectrum operations in complex urban environments and critical areas. The discussion is based on an analysis of various factors and constraints that red team UAVs must consider when perceiving and localizing anomalous radiation sources. The MDP framework is particularly suitable for capturing the sequential decision-making nature and environmental uncertainties inherent in this problem. The interaction is illustrated in Figure 2.  
As shown in Figure 2, the framework follows a MDP where state \(S_{t}\) represents the agent's observation of the environment at time step \(t\), which transitions to the next state \(S_{t+1}\) after the agent takes an action. Action \(A_{t}\) defines the agent's possible operations. The reward signals include both immediate reward \(R_{t}\) and the previous time step's reward \(R_{t-1}\), providing feedback on the agent's performance in both current and past states. Through this temporal interaction loop, the agent learns to optimize its behavior based on both current observations and historical experience. This MDP framework is implemented in our urban electromagnetic environment where UAVs act as agents to perform sensing and detection tasks. The key components of our MDP framework are detailed as follows:  
* State: The state primarily focuses on the red team UAV's current position \((x,y)\), distance \(d\), detection direction \(dir\), current time step \(t\), the number of successfully detected UAVs \(n\), the anomalous radiation source position of the most recent effective scan \((x_{v},y_{v})\), and its distance \(d_{v}\). The state space is constructed as a set \(\langle x,y,d,dir,t,n,x_{v},y_{v},d_{v}\rangle\). Different UAVs have varying attributes and represent different systems. Based on their specific characteristics, the state transitions for movement and detection are configured accordingly;
* Action: The red team UAV's actions are primarily divided into movement direction \(m_{d}\) and detection direction \(d_{d}\), where \(m_{d}\) is determined by velocity \(\mathbf{v}_{i}(t)\) and heading angle \(\mathbf{\phi}_{i}(t)\), while \(d_{d}\) is determined by sensor orientation \(\mathbf{\psi}_{i}(t)\). For movement, the red team UAV has nine options: moving in cardinal and diagonal directions or staying. For detection, the red team UAV also has nine options: scanning or not scanning in 45-degree increments on the current plane. The action space is constructed as a set \(\langle m_{d},d_{d}\rangle\). The actual radiation detection depends on both sensor orientation and RSSI measurements. When three red team UAVs simultaneously detect the anomalous radiation source (i.e., \(|VM(t)|\geq 3\)), it triggers the three-point localization function;
* Reward: The environment generates a scalar signal as reward feedback based on the state and actions taken by the red team UAV. In this paper, we design rewards from multiple aspects, providing rewards for individual UAV successful detection, first successful detection, and successful three-point localization. This multi-faceted reward design balances individual performance and team collaboration, while addressing both task speed and accuracy. The individual detection reward keeps all red team UAVs motivated, the first successful detection reward speeds up task completion through competition, and the three-point localization reward encourages coopera  
Figure 2: The MDP framework and its implementation: (**a**) standard MDP interaction framework; (**b**) implementation in urban electromagnetic environment with red team UAVs as agents. tion, enhancing localization accuracy.