## 4 Agent Training with Markov Decision Process (MDP)
### Design of the MDP Model
#### 4.1.1 Definitions of Each Element in the MDP Model  
Consider a continuous sequence of \(M\) epochs \(\{E_{n+1},E_{n+2},\ldots,E_{n+M}\}\) from the actual collected epoch set \(\mathcal{E}=\{E_{n}\}_{n=1,2,\ldots,N}\) as the weighting targets for a complete iterative positioning process, where \(M=M_{1}+M_{2}\). The first \(M_{1}\) epochs \(\{E_{n+1},E_{n+2},\ldots,E_{n+M_{1}}\}\) are used for initial positioning, while the weights of the subsequent \(M_{2}\) epochs \(\{E_{n+M_{1}+1},E_{n+M_{1}+2},\ldots,E_{n+M_{1}+M_{2}}\}\) are determined by the agent. Next, we define that at each timestep \(t\), the agent observes the environmental state and makes weighting decisions, completing one interaction with the environment (i.e., AWR-PF). After \(T\) timesteps, the final positioning result is obtained, where \(T=M_{2}\). We define the agent's interaction with the environment over these \(T\) timesteps as an episode. At any timestep \(t\), the agent's decision-making process for the weight of epoch \(E_{n}\) can be represented by the MDP model as a tuple \((S(t),A(t),R(t),S^{\prime}(t))\). The following will introduce each element of the tuple in detail. At timestep \(t\), the state \(S(t)\) observed by the agent from the environment is represented as \(S(t)=\left\{S_{\text{ Satellite}}(t),S_{\text{epoch}}(t),S_{\text{signal}}(t),S_{\text{GDOP}}(t)\right\}\), with a state space dimension of 9 (comprising 9 components). \(S^{\prime}(t)\) denotes the next state transitioned to after the agent observes \(S(t)\) and takes action \(A(t)\), expressed as \(S^{\prime}(t)=S(t+1)\). The satellite state \(S_{\text{ Satellite}}(t)\) observed by the agent at timestep \(t\) is defined as:  
\[S_{\text{ Satellite}}(t)=\left\{S_{\text{age}}(t),S_{\text{azimuth}}(t),S_{ \text{elevation}}(t),S_{\text{distance}}(t)\right\} \tag{20}\]  
where \(S_{\text{age}}(t)\) denotes the ephemeris age (corresponding to \(EA\) in Equation (15)) derived from the observed ephemeris data, \(S_{\text{azimuth}}(t)\) denotes the azimuth angle (corresponding to \(A_{a}\) in Equation (15)) of the satellite associated with epoch \(E_{n}\), \(S_{\text{elevation}}(t)\) denotes the elevation angle (corresponding to \(A_{e}\) in Equation (15)) of the same satellite, and \(S_{\text{distance}}(t)\) denotes the satellite-to-receiver distance (corresponding to \(SRD\) in Equation (15)). These components collectively characterize the satellite state for the agent's positioning decisions. The epoch state \(S_{\text{epoch}}(t)\) observed by the agent at timestep \(t\) is defined as:  
\[S_{\text{epoch}}(t)=\{S_{\text{id}}(t),S_{\text{rank}}(t)\} \tag{21}\]  
where \(S_{\text{id}}(t)\) denotes the satellite identifier (corresponding to \(ID_{\text{Sat}}\) in Equation (15)) associated with epoch \(E_{n}\), and \(S_{\text{rank}}(t)\) denotes the cumulative observation count (corresponding to \(R_{\text{SatID}}\) in Equation (15)) of the corresponding satellite from the initial timestep up to the current timestep \(t\). This provides essential epoch identification and historical tracking information for the agent. The signal state \(S_{\text{signal}}(t)\) observed by the agent at timestep \(t\) is defined as:  
\[S_{\text{signal}}(t)=\left\{S_{\text{snr}}(t),S_{\text{Doppler}}(t)\right\} \tag{22}\]  
where \(S_{\text{snr}}(t)\) denotes the SNR (corresponding to \(SNR\) in Equation (15)) of the observed epoch \(E_{n}\), and \(S_{\text{Doppler}}(t)\) denotes the Doppler frequency (corresponding to \(f_{\text{d}}\) in Equation (15)) of the corresponding signal. These measurements provide critical signal quality metrics for the agent's weighting decisions. Additionally, \(S_{\text{GDOP}}(t)\) denotes the latest GDOP value (corresponding to \(GDOP_{n-1}\) in Equation (15)) obtained by the agent at timestep \(t\) through the weighted RLS algorithm, which serves as the agent's observation of the current positioning system state.