## 4 ML Applications for RIS-Assisted Communications
### Beamforming Applications
This framework aimed to maximize UAVs' worst-case secrecy energy efficiency (SEE) via a joint optimization of the flight trajectory, UAV active beamforming, and RIS passive beamforming. The TD3-based proposed scheme had two agents composed of three networks each. All networks were based on MLP architectures. The state space consisted of the local information at the UAVs, where the action space captured the flying direction and completed the UAV trajectory using SEE as a reward. As in Section 4.2, we compared the ML algorithms and parameters for beamforming applications in Table 5. We list the source database used for each approach as the source code availability. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Reference** & **ML Algorithm** & **Architecture and ML Methods** & **Database Used** & **Available Source Code?** \\ \hline [84] & DL & \begin{tabular}{c} Perceptron with variable fully connected layers, \\ RELU activation, MSE loss \\ \end{tabular} & Deep MIMO [56] & No \\ \hline [85] & DL & \begin{tabular}{c} 2 NNs with fully connected layers, ReLU \\ activation, Adam optimizer \\ \end{tabular} & Synthetic & No \\ \hline [86] & DL & \begin{tabular}{c} 5 fully connected layers ANN with variable \\ number of neurons, ReLU activation, \\ Adam optimizer \\ \end{tabular} & Synthetic & No \\ \hline [87] & DL & \begin{tabular}{c} DNN to parametrize pilots GNN to capture \\ interactions among users, Adam optimizer, ReLU \\ activation function \\ \end{tabular} & Synthetic & \begin{tabular}{c} Yes, MATLAB R2018b \\ and Python \\ \end{tabular} \\ \hline [88] & DL & \begin{tabular}{c} 2 DNN (BS and UE) fully connected networks, \\ ReLU and sigmoid activation functions, \\ cross-entropy loss function, Adam optimizer \\ \end{tabular} & Synthetic & \begin{tabular}{c} Yes, MATLAB R2018b \\ and Python \\ \end{tabular} \\ \hline [89] & DL & \begin{tabular}{c} NN-based architecture, ReLU activation, \\ Adam optimizer \\ \end{tabular} & Synthetic & Yes, Python \\ \hline [83] & RL & \begin{tabular}{c} DDPG algorithm with both actor and critic \\ networks, tanh activation function \\ \end{tabular} & n/a & \begin{tabular}{c} Yes, Python \\ (reproduction work) \\ \end{tabular} \\ \hline [90] & RL & \begin{tabular}{c} SAC algorithm with 3 MLPs, Adam and Xavier \\ optimization, tanh activation function \\ \end{tabular} & n/a & Yes, Python \\ \hline [91] & RL & \begin{tabular}{c} Q-learning-based scheme with DQN agent with 4 \\ layers, 128 units, MSE loss function, \\ Adam optimizer \\ \end{tabular} & n/a & Yes, Python \\ \hline [92] & RL & \begin{tabular}{c} DQN and DDPG algorithms, 2-layer networks \\ with 30 units, ReLU activation, Adam optimizer \\ \end{tabular} & n/a & Yes, Python \\ \hline [93] & RL & \
\begin{tabular}{c} SD3 algorithm for dual-domain energy \\ harvesting and joint optimization of phase shifts \\ and transmit power \\ \end{tabular} & n/a & Yes, Python \\ \hline [94] & RL &
\begin{tabular}{c} 2 agents based on TD3 algorithm with 3 networks \\ each, MLP architecture \\ \end{tabular} & n/a & Yes, Python \\ \hline \hline \end{tabular}
\end{table} 
Table 5: ML algorithms, parameters, databases, and source code availability for the revised beamforming applications on RIS-aided wireless systems.