## 4 Agent Training with Markov Decision Process (MDP)
### DRL-Based Solution  
To address the sequential decision-making challenge in epoch weighting, we have established the MDP model as detailed in Section 3. Given the complexity of the multi-level state space \(S(t)\) and the discrete nature of action space \(A(t)\) while pursuing long-term cumulative reward maximization, we employ DRL for optimal policy derivation.  
While conventional value-based deep reinforcement learning approaches like the Deep Q-Network (DQN) [29] offer theoretically sound solutions for Markov decision processes, they exhibit significant limitations when applied to the complex state-action spaces characteristic of our epoch weighting problem. The algorithm's well-documented propensity for Q-value overestimation [30] originates from its fundamental architecture--specifically, the use of a single network to perform dual functions: both selecting the optimal action and evaluating its value. To mitigate the Q-value overestimation issue inherent in the DQN algorithm and enhance both the stability of the learning process and the performance of the final policy, we adopt the DDQN algorithm.  
The key improvement of DDQN lies in its decoupling of action selection and value estimation in the target Q-value computation, achieved through the use of an online network and a target network. Specifically, the online network \(Q(s,a;\theta)\) (with parameters \(\theta\)), which undergoes continuous updates and is responsible for action exploration and selection based on current knowledge, determines the optimal action: \(a^{*}=\operatorname*{argmax}_{\theta}Q(s^{\prime},a^{\prime};\theta)\), where \(s\) denotes the current state, \(a\) denotes the currently determined selected action, \(s^{\prime}\) denotes the next state, and \(a^{\prime}\) denotes the set of admissible actions available at state \(s^{\prime}\). The value of the selected action is then evaluated by the target network \(Q(s,a;\theta^{\prime})\) (with parameters \(\theta^{\prime}\)), which periodically synchronizes its parameters from the online network to provide a relatively stable benchmark for value estimation: \(Q(s^{\prime},a^{*};\theta^{\prime})\). This decoupling mechanism effectively mitigates the Q-value overestimation problem.  
While DDQN is conventionally applied in video game domains--where raw pixel frames serve as sequential input states processed through convolutional neural networks (CNNs) for feature extraction and direct-action generation (e.g., movement controls), with rewards derived from game scores to enable end-to-end learning--their direct adaptation to satellite navigation presents fundamental challenges. In LEO SOP applications, raw signal waveform images are unintelligible to agents without engineered state representations. Direct positioning output would cause exponential action space expansion, while undefined reward mechanisms for navigation problems require careful design.  
To address these limitations, we innovatively customize DDQN:  
* Structured State Representation: Extracting positioning-relevant features from satellite, epoch, signal, and positioning layers using lightweight neural networks, enabling efficient interpretation of mathematical relationships.
* Collaborative Decision Framework: Integrating the agent with RLS within the AWR-PF, where the agent evaluates epoch credibility and outputs weight for RLS-based positioning, dramatically reducing decision complexity.
* Differential Reward Mechanism: Incorporating both absolute and proportional positioning error changes--rather than instantaneous error alone--to account for convergence dynamics during iterative positioning.  
These adaptations transform DDQN from a gaming solution into a viable satellite positioning optimizer: the DDQN-based Epoch Weighting (DDQN-EW) algorithm. Subsequently, we will detail DDQN-EW's implementation.