## 4 ML Applications for RIS-Assisted Communications
### Beamforming Applications  
ML-based beamforming for RIS-assisted wireless systems applications considers pilot signals or channel realizations as input data for training, similarly to the CSI approaches listed in Section 4.2. However, in most of the reported state-of-the-art research frameworks, the optimization problem relies on maximizing the sum rate using a DL scheme. Figure 14 shows the process of beamforming solutions based on DL for RIS-assisted systems, where the benchmark is similar to the one depicted in Figure 13, except for the learning stage. The \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Reference** & **ML Algorithm** & \multicolumn{2}{c}{**Architecture and ML Methods**} & **Database Used** & **Available Source Code?** \\ \hline
[71] & DL &
\begin{tabular}{c} 2 CNNs of 9 layers, dropout, SGD optimizer, \\ minibatch \\ \end{tabular} & Synthetic & Yes, MATLAB R2018b \\ \hline
[72] & DL &
\begin{tabular}{c} Adapted NN with variable number of layers, \\ ReLU activation function \\ \end{tabular} & DeepMIMO [56] & Yes, MATLAB R2018b \\ \hline
[73] & DL &
\begin{tabular}{c} Adapted NN with a variable number of layers, \\ tanh activation function, Adam optimizer \\ \end{tabular} & Synthetic & No \\ \hline
[74] & DL &
\begin{tabular}{c} CNN of 15 convolutional layers, 64 filters of size \\ \(3\times 3\times 64\), ReLU activation function, Adam optimizer \\ \end{tabular} & Synthetic & Yes, MATLAB R2018b \\ \hline
[75] & DL &
\begin{tabular}{c} CNN-based EDSR, MDSR architectures, ReLU \\ activation \\ \end{tabular} & Synthetic & Yes, Python \\ \hline
[76] & DL &
\begin{tabular}{c} CNN-based FFDNet, filters of \(3\times 3\) and conv2D \\ layers, ReLU activation, batch normalization \\ \end{tabular} & Synthetic & Yes, Python \\ \hline
[77] & DL &
\begin{tabular}{c} CNN-based network, 32 filters, ReLU activation \\ \end{tabular} & DeepMIMO [56] & Yes \\ \hline
[78] & DL &
\begin{tabular}{c} NN-based network, ReLU activation, NMSE loss \\ function, ReLU activation \\ \end{tabular} & Synthetic & Yes, Python \\ \hline
[79] & DL &
\begin{tabular}{c} ANN with linear layers, sigmoid activation, \\ Adam optimizer \\ \end{tabular} & Synthetic & Yes, Python \\ \hline \hline \end{tabular}
\end{table} 
Table 3: ML architectures, methods, and data sources for the estimation of CSI in RIS-aided communication systems. Figure 13: Diagram of ML-based CSI estimation for RIS communication systems. sum-rate maximization is added to the optimizing process in the former case. The outputs consist of beamforming and phase-shift matrices at the RIS [82]. Several solutions have been initiated using DRL schemes for beamforming rather than DL, avoiding the training of datasets. Here, channel matrices can be considered an RL environment to build the state space; the actions are the optimized phase shift and the beamforming matrices at each channel realization, while the sum rate is used as an immediate reward. Figure 15 shows a general approach for applying the DRL loop to RIS-aided wireless communications. The action \(A^{(t)}\) consists of a space given by the phase shifts and beamforming matrices at a given time \(t\). The state \(S^{(t)}\)consists of a space given by all channel matrices defined in Equation (1) and the last action performed by the agent, whose objective is to maximize the accumulated sum of rewards. For the task of beamforming, a common choice for the rewards \(R^{(t)}\) is the _sum-rate_ metric \(\rho_{k}\)[82; 83]. A model is built at each time interval by updating the actions using a DRL algorithm such as DDPG, TD3, or SAC (since the output is continuous). As soon as the sum-rate rewards are maximized, the closed loop depicted in Figure 15 updates and optimizes the beamforming and phase-shift matrices. In Table 4, we show the revised applications of ML in RIS-aided communication systems. The contributions and remarks for each approach are listed. In the first revised approach, [PERSON] et al. [84] propose a novel RIS architecture where all elements are passive except for a few active elements.