# A Survey on Optimal Channel Estimation Methods for RIS-Aided Communication Systems
## 3 Channel Estimation in the RIS-Assisted Communication Systems
### MISO Systems
#### 3.2.3 Pseudocode of Asynchronous One-Step Q-Learning  
In [92], the authors propose an RL algorithm in an RIS-assisted MU-MISO system, where instead of repeating experience, they asynchronously execute multiple agents in parallel over multiple instances of the environment. This parallelism likewise decouples the agents' data into a more stationary procedure, because at each time step, the parallel agents experience a variety of different states. This allows a wider range of fundamental RL algorithms within policy, such as critical action methods, and non-policy RL algorithms, such as Q-learning, to be robust and efficiently implemented using deep neural networks.  
They believe the standard RL setting, where an agent relates with environment E over a few discrete time steps. At each time step t, the agent obtains a state \(s_{t}\) and takes an action from some set of possible actions A, according to its policy \(\pi\). In response, the agent obtains the next state \(s_{t+1}\) and obtains a scalar reward \(r_{t}\). The method remains until the agent achieves a terminal state, after which the procedure restarts. The return \(\mathrm{R_{t}}=\sum_{k=0}^{\infty}\omega^{k}r_{t+k}\) is the total accumulated return from time step t with \(\omega\in(0,1]\). The purpose of the agent is to improve the expected return from each state \(s_{t}\)[92].  
In value-based model-free RL methods, the action value function is characterized using a function approximator, such as a neural network. Let \(\mathcal{Q}(s,\alpha;\theta)\) be an approximate action-value function with parameters \(\theta\). The updates to \(\theta\) are obtained from a variety of RL algorithms. One example of such an algorithm is Q-learning, which directly estimates the optimal action value function: \(\mathcal{Q}^{*}(s,\alpha)\approx\mathcal{Q}(s,\alpha;\theta)\). In one-step Q-learning, the parameters \(\theta\) of the action value function \(\mathcal{Q}(s,\alpha;\theta)\) are studied by iteratively minimizing a sequence of loss functions, where the \(i\)-th loss function is described as  
\[L_{i}(\theta_{i})=\mathbb{E}\big{(}r+\omega max_{\alpha^{\prime}}\mathcal{Q} \big{(}s^{\prime},\alpha^{\prime};\theta_{i-1}\big{)}-\mathcal{Q}(s,\alpha; \theta_{i})\big{)}^{2} \tag{24}\]  
where \(s^{\prime}\) is the state encountered after state s [92].  
The above method is mentioned as one-step Q-learning because it renews the energy value \(\mathcal{Q}(s,\alpha)\) to the one-step return \(r+\omega max_{\alpha^{\prime}}\mathcal{Q}(s^{\prime},\alpha^{\prime};\theta_{ i})\). A disadvantage of utilizing one-step methods is that acquiring a reward r only immediately impacts the value of the pair of state actions s that led to the reward. The values of the other pairs of state actions are only indirectly affected through the updated value \(\mathcal{Q}(s,\alpha)\). This can make the learning process slow, as many updates are expected to generate a reward to the relevant previous states and actions [92]. For more details, see Algorithm 1 in [92].