## 4 RISs Controlled by AI
### Reinforcement Learning  
RL is one of the most popular methods for training a model for RISs. The most important advantage of RL is adaptability as it does not require training on existing data because RL models learn and improve after deployment. Thus, the algorithm may be tailored to specific requirements of each scenario. Moreover, implementation of RL tends to be easier to deploy and integrate in existing networks. A plethora of algorithms have been used for the application of RL in RIS-related scenarios. On the one hand, there exist value-based algorithms, such as Deep Q-Network (DQN) and Double Deep Q-Network (DDQN); these algorithms are rather simple in architecture, but as they work with values and actions they are constrained in specific values and actions spaces. On the other hand, policy-based algorithms are more complex in architecture by using optimal policies to make decisions; thus, they perform better in dynamic environments. Examples of such algorithms are the Deep Deterministic Policy Gradient (DDPG), the Proximal Policy Optimization (PPO) and the Twin Delayed DDPG (TD3). The latter may also be considered as hybrid. The studies presented in Table 4 showcase diverse applications of RL, demonstrating the potential of the latter in addressing complex optimization problems in RIS-enabled systems and addressing challenges ranging from security enhancement [38] to beamforming optimization [39; 40] and energy efficiency [41]. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Ref.** & **Problem Description** & **Algorithm Used** & **Achievement** \\ \hline [PERSON] and [PERSON], 2023 [42] & Review paper & Deep RL & Potential of Deep RL for RIS technologies \\ \hline [PERSON] and [PERSON], 2022 [43] & Real-time phase control of IRS for SRE & Deep RL with DDQN & Model-free control of IRS and improvement of its adaptivity to different channel dynamics \\ \hline [PERSON] et al., 2023 [38] & Network Security for IoT communications under trusted-untrusted device diversity & Deep RL with a DDPG QoS of all devices \\ \hline [PERSON] et al., 2020 [39] & Joint design of BS beamforming matrix and IRS phase shift matrix in Massive-MIMO wireless Communications systems & Deep RL with DDPG environments with high efficiency \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Selection of research works demonstrating RL (various approaches and different algorithm types). [PERSON] et al. [42] provide a broad perspective on the potential of Deep RL for RIS technologies, offering foundational insights. In [43], the conventional approaches have turned out to be insufficient in a RIS-aided SRE. Thus, a model-free design for controlling the Intelligent Reflecting Surface (IRS), independent of the sub-CSI, has been proposed in order to create SRE. Deep RL has been applied to perform real-time coarse phase control of IRS, followed by extremum seeking control (ESC) for the fine phase control. Optimization of beamforming has drawn strong interest over the last years and a lot of studies have turned to the application of RL in RIS-aided systems. Optimal performance has been achieved for a BS antenna beamforming matrix in conjunction with RIS phase shift matrix in massive-MIMO systems by applying DDPG [39], while phase-shift optimization for URLLC systems has been effectively handled through TD3 [40]. The latter reduced overestimations commonly found in DDPG and provided robust performance under short-packet communications. The potential of RIS to enhance IoT network performance has been leveraged by several researchers [38; 41; 44]. The IoT network security has been addressed in [38], by diversifying the devices as trusted and untrusted while maintaining the quality of service (QoS) of all devices.