## 3 Preliminaries
### Deep Reinforcement Learning  
Reinforcement learning is a technique that is used to enhance the model's coping ability through interaction with the environment. It iteratively trains the model by selecting actions in a constantly changing environment so as to achieve the model's ability to choose high-scoring actions [30; 31; 32; 33; 34; 35]. In general, the Markov decision process (MDP) has been utilized to describe the reinforcement learning process. Specifically, in a certain environmental state, the agent of the reinforcement learning makes action choices in a certain way or probability, thus promoting the transformation of the environmental state and obtaining corresponding rewards. Mathematically, we denote the environmental state at time \(t\) as  
Figure 1: Schematic diagram of CSI data collection process in this paper. \(s_{t}\), and the selected action as \(a_{t}\); therefore, a numerical reward \(r_{t}\) can be acquired, and the subsequent state of the environment would be transformed into \(s_{t+1}\) at the next moment. The quadruple \(\langle\mathbf{S},\mathbf{A},\mathbf{P},\mathbf{R}\rangle\) with \(\mathbf{S},\mathbf{A},\mathbf{P},\mathbf{R}\) respectively represent the state space, action space, reward set, and state transition probability matrix, and they are usually adopted to describe the learning process of the reinforcement learning. The decision-making strategy of the agent \(\pi_{t}(s,a)\) that indicates the action choice recommendation for state \(s\) at time \(t\) can be updated once the experience sequence \(\{(s_{t},a_{t},r_{t},s_{t+1}),\ldots\}\) has been obtained during the agent's ongoing exploratory learning. The target of reinforcement learning is to ensure that the agent has the ability to obtain the maximum expected cumulative reward in the decision-making process. Then, we can phrase it as follows:  
\[R_{t}=\sum_{l=0}^{\infty}\vartheta^{l}r_{t+l}, \tag{6}\]  
in which \(\vartheta\in[0,1]\) denotes the discount ratio. Q-learning is a popular approach among reinforcement learning methods [34; 35], whereby the Q-values of the model are updated via interactions with the environment. The Q-value mirrors the effectiveness evaluation of the appropriating action \(a\) under policies \(\pi\) in circumstantial state \(s\). The environment can be expressed as follows:  
\[Q^{\pi}(s,a)=E[R_{t}\mid s_{t}=s,a_{t}=a]. \tag{7}\]  
We denote \(Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a)\) as the optimal action value function and, according to the Bellman optimality equation, we can acquire the following formula:  
\[Q^{*}(s,a)=E\big{[}r_{t+1}+\theta\max_{a^{\prime}}Q^{*}\big{(}s^{\prime},a^{ \prime}\big{)}\mid s_{t}=s,a_{t}=a\big{]}, \tag{8}\]  
in which \(s^{\prime}\) denotes the next state after the action \(a\) was selected and utilized by the agent. By utilizing the experience sequence acquired through the interaction with the environment, the agent of Q-learning can continuously update the optimal action value function. By denoting \(q(s_{t},a_{t})\) as the estimated Q-value of the iterative process, we can express the update process of Q-learning using the following formula:  
\[q(s_{t},a_{t}){\leftarrow}q(s_{t},a_{t})+\xi(r_{t}+\theta\max_{a^{\prime}}q(s_ {t+1},a_{t+1}){-}q(s_{t},a_{t})), \tag{9}\]  
in which \(\xi\in[0,1]\) represents the learning rate. Exploration and exploitation are two important processes in reinforcement learning, which should be properly handled to achieve a reasonable balance. The exploitation process signifies the agent adopting the learned decision-making strategies, i.e., selecting actions solely based on the model's output. Meanwhile, the exploration process means that the agent selects actions in a random manner in order to explore the decision space. It should be noted that an agent should not solely favor one process to preclude scenarios in which model training is absent or model decisions are overly random; these are detrimental to the convergence or performance improvement of the model.