# A Survey on Optimal Channel Estimation Methods for RIS-Aided Communication Systems
## 3 Channel Estimation in the RIS-Assisted Communication Systems
### MISO Systems
#### 3.2.2 PPO Algorithm  
As we mentioned in Section 2, the authors of [74] use the PPO algorithm on the RIS-assisted MU-MISO system to solve optimization problem P1. PPO is presented in [27] and has been shown to outperform other benchmark algorithms, as it is easy to set up, useful and easy to execute, and is a model-free, policy-based, actor-critic strategy ranking method.  
Think about an infinite-horizon discounted Markov decision process (MDP), specified by the tuple \((S,A,P,r,\omega)\), where P:\(S\times A\times S\to R\) is the transition probability distribution and finally, \(\omega\in[0,\,1]\). The purpose of PPO is to maintain the consistency of the trust region policy optimization (TRPO) algorithms because they guarantee monotonic improvements considering the Kullback-Leibler (KL) variance of policy updates using first-order optimization methods. The TRPO increases the objective purpose and limits the size of the policy update [74]. Specifically,  
\[max_{\theta}\hat{\mathbb{E}}_{t}\bigg{[}\frac{\pi_{\theta}(\alpha_{t}|s_{t})} {\pi_{\theta_{old}}(\alpha_{t}|s_{t})}\hat{A}_{t}\bigg{]}, \tag{17}\]  
\[subject\,\,to\,\hat{\mathbb{E}}\big{[}KL\big{[}\pi_{\theta_{old}}(.|s_{t}),\pi _{\theta}(.|s_{t})\big{]}\big{]}\leq\delta,\]  
with the expectation \(\hat{\mathbb{E}}_{t}[.]\) showing the observed average over a fixed batch of samples, in an algorithm that varies between sampling and optimization. Furthermore, \(\hat{A}_{t}\) is an estimator of the gain function at timestep t and is given by  
\[A_{t}=Q(s_{t},\alpha_{t})-V(s_{t}), \tag{18}\]  
where \(Q(.,.)\) and \(V(.)\) are the action-value of the value functions, accordingly, and are specified as follows  
\[Q(s_{t},\alpha_{t})=\mathbb{E}_{s_{t+1}\alpha_{t+1}\cdots}\bigg{[}\sum_{l=0}^ {\infty}\omega^{l}r(s_{t+1})\bigg{]}, \tag{19}\]  
\[V(s_{t})\!\!=\mathbb{E}_{a_{t}s_{t+1}\cdots}\bigg{[}\sum_{l=0}^{\infty}\omega ^{l}r(s_{t+1})\bigg{]}, \tag{20}\]  
where \(s_{t+1}\sim P(s_{t+1}|s_{t},a_{t})\). The valuation of the benefit task in the interval \(t\in[0,\,\mathrm{T}]\) is given by [27]  
\[\hat{A}_{t}=\delta_{t}+(\omega\kappa)\delta_{t+1}+\ldots+(\omega\kappa)^{T-t+1 }\delta_{T-1}, \tag{21}\]  
with \(\delta_{t}=r_{t}+\omega V(s_{t+1})-V(s_{t})\) and \(\kappa\) is a hyperparameter and represents the factor for the trade-off of bias and variance for the generalized advantage estimator (GAE) [74].  
Then, let \(\rho_{t}(\theta)\) signify the probability ratio \(\rho_{t}(\theta)=\pi_{\theta}(\alpha_{t}|s_{t})/\pi_{\theta old}(\alpha_{t}|s_{t})\), then obviously \(\rho_{t}(\theta_{old})\)= 1, hence, according to (17), the TRPO maximizes  
\[\mathcal{L}^{CPI}(\theta)=\hat{\mathbb{R}}_{t}[\rho_{t}(\theta)]\hat{A}_{t}, \tag{22}\]  
where CPI is the conservative policy iteration. The major problem of optimization in (22) is the probability of a large policy update. If \(\pi_{\theta old}\ll\pi_{\theta}\) then \(\rho_{t}(\theta)\) becomes very large and policies change dramatically. To unravel this issue, PPO alters the objective function in (22), as follows  
\[\mathcal{L}^{CPI}(\theta)=\hat{\mathbb{R}}_{t}\big{[}\min\big{(}\rho_{t}( \theta)\hat{A}_{t},clip(\rho_{t}(\theta),1-\varepsilon_{c},1+\varepsilon_{c} )\hat{A}_{t}\big{)}\big{]}, \tag{23}\]  
where \(\varepsilon_{c}\) is a hyperparameter. The simplification of the objective function (22) was carried out because using a small set of experiment sizes; the algorithm is not too acquisitive in favoring actions with positive benefit functions, nor too fast in preventing actions with negative benefit functions.  
The PPO is an MDP with observation and action spaces. When resolving the joint active and passive beamforming problem on the BS, the RIS and UE of the MISO system are declared in the E environment, and the agent is the BS that controls the RIS. For more details, see Algorithm 1 in [74].