## 4 Algorithm
### Markov Modeling  
The state represents the characteristics or information of the environment at a certain moment. The state determines the observations of the agent and influences future decisions and rewards. In the process of in situ data management in the InS system, the state at each time step \(t\) consists of three current system evaluation metrics: \(AWD\), \(ARR\), and \(ARL\). These metrics comprehensively characterize the service performance of the InS system at each time step.  
However, since the original values of these three indicators are real numbers with unfixed upper and lower bounds, directly using them in reinforcement learning would result in an infinite state space, which prevents the convergence of the Q-learning algorithm. To control the value range of the evaluation indicators without affecting their relative differences, the three system-level evaluation indicators are normalized as follows:  
\[x^{\text{norm}}=\frac{x}{1+x},\quad x\in\{AWD,ARR,ARL\} \tag{24}\]  
where \(x\) denotes the original indicator value and \(x^{\text{norm}}\) represents the normalized value. After normalization, in order to discretize the state space for Q-learning, each normalized indicator value is further divided into \(n\) (default is 5) fixed intervals and mapped to discrete integers. The discretization process is defined as follows:  
\[x^{\text{disc}}=\lfloor n\cdot x^{\text{norm}}\rfloor,\quad x^{\text{norm}} \in(0,1) \tag{25}\]  
After normalization and discretization, the system state is ultimately represented as a three-dimensional discrete vector. Each dimension takes values from a finite set, effectively transforming the originally continuous state space into a finite discrete state space, which provides theoretical support for the stability and convergence of subsequent algorithm training. \(s_{t}\) denotes the state at time step \(t\), and \(\mathcal{S}\) denotes the finite discrete state space, defined as follows:  
\[s_{t}=\left[AWD^{\text{disc}},ARR^{\text{disc}},ARL^{\text{disc}}\right]\in \mathcal{S} \tag{26}\]The action represents the decision made by the agent in a specific state, determined by the policy function \(\pi\), and directly affects the next state of the environment and the reward received. In the process of in situ data management, \(a_{t}\) denotes the action selected at time step \(t\), _i.e._, the target access demand queue responded to in the current round. \(\mathcal{A}\) denotes the prioritized discrete action space, which can be represented as follows:  
\[a_{t}\in\mathcal{A}=\{Que_{low},Que_{medium},Que_{high}\} \tag{27}\]  
The reward represents the feedback signal received by the agent after executing a certain action, serving as a measure of the decision's effectiveness. The agent's objective is to maximize the cumulative reward. To ensure the stability of the reward function during the training process and to satisfy the boundedness assumption required by Q-learning, the reward function \(R(s_{t},a_{t})\) at time step \(t\) is composed of three normalized system metrics:  
\[R(s_{t},a_{t})=-(AWD^{\text{norm}}+ARR^{\text{norm}}+ARL^{\text{norm}}) \tag{28}\]  
The policy \(\pi\) defines the rules or strategy by which the agent selects an action in each state. The policy can be defined as a function \(\pi(s)\) mapping state \(s_{t}\) to action \(a\), which can be specifically expressed as follows:  
\[\pi(s_{t})=a_{t} \tag{29}\]  
In summary, although the system state at each time step is derived from real-time system metrics rather than based on an explicit state transition probability function, the management process of access demand for in situ data in the InS system can still be formalized as a Markov Decision Process (MDP) that satisfies the Markov property, defined as follows:  
\[\mathcal{M}=(\mathcal{S},\mathcal{A},R,\gamma) \tag{30}\]  
where the discount factor \(\gamma\) is used to balance the trade-off between immediate and future rewards. In MDP modeling, the policy is optimized through reinforcement learning algorithms to maximize the cumulative reward.