## 4 ML Applications for RIS-Assisted Communications
### Beamforming Applications
The architecture consisted of one input layer, one hidden layer, and one output layer for the BS network. The layers were fully connected with a ReLU activation. The user DNN had the same architecture as the first layer but used the Xavier initialization and Adam optimization. The framework suggested a type of transfer-learning algorithm. In [89], a DL approach is proposed to maximize the weighted sum-rate (WSR) [8; 95; 96] for beamforming. The system is called RISnet, a dedicated architecture where all antennas share the same parameters. It applies an alternating optimization procedure for the phase shift and precoding matrices. The structure of the NN consists of a variable number of layers according to the channel inputs, followed by ReLU activations and the Adam optimizer. The output is the estimated phase shift, where the NN takes a separate channel feature for each user. When training is performed online, input data are not required for DRL-based beamforming schemes. In [83], the DDPG algorithm was used for the joint design of transmit beamforming and phase-shift matrices at the RIS. The sum rate was used to train the DRL agent as the immediate reward. The method required an actor and a critic network, whose architecture comprised one input layer, one hidden layer, and one output layer with a _tanh_ activation function at the output. The number of neurons corresponded to the number of users and antennas at the BS and RIS elements. The scheme performed a whitening process to avoid correlations between entries. [PERSON] et al. [90] addressed the beamforming optimization research problem under imperfect CSI and hardware impairments. The algorithm employed a closed form to model the phase-dependent amplitude entries developed in [97]. Using the SAC DRL method, the agent learned the optimal beamforming and phase-shift matrices (actions) that maximized the sum rate (reward). The SAC framework consisted of two Q-networks and one stochastic policy network, with the Adam optimization, a _tanh_ activation for the output layer, and a ReLU activation for the hidden layers. A model-free design to configure reflections on RIS-aided communications was proposed in [91]. This DRL approach did not require the CSI of the sub-channels and operated under the TDD-MIMO scheme. The DRL-based method used a double DQN (DDQN) to perform coarse phase control in real time, enhancing the phase-shift optimization. The Q-learning-based architecture consisted of four layers with 128 units, each using the Adam optimizer and MSE loss function. DRL approaches have a 3D trajectory and phase-shift-enhanced design for RIS-assisted UAV systems. The authors in [92] perform this optimization using DDPG and DDQN algorithms. The framework explores the mobility of the UAV in a 3D space and reaches the optimal phase shift of the RIS with high accuracy. The architecture consists of two-layer networks with 30 units, applying a ReLU activation and the Adam optimization. The state space consists of channel realizations and the locations of UAVs. At the same time, the reward proposed is a ratio of the total quantity of data, state, location of UAVs, flight time, and scheduling. Another framework that applies DRL for joint optimization in a UAV environment is proposed in [93]. In that paper, the authors design a scheme for energy harvesting for resource allocation and convex optimization of phase shifts in a UAV-RIS system. The proposed method is based on an SD3 scheme that uses a pair of critic networks. The reward is evaluated in terms of the harvested energy. The state space consists of the equivalent channels from the BS to the UAV-RIS, the distance between the UAV elements and metasurface, the meta-surface location, and each UAV antenna's position. The action space consists of the phase shift at the RIS, the transmit power at each UAV, and the scheduling. The physical layer security in RIS-assisted mmWave UAV communications under multiple eavesdroppers and imperfect CSI was investigated in [94].