## 3 Machine Learning for IRS-Assisted Communication Systems
### Reinforcement Learning for IRS-Enhanced Communication Systems  
RL is constructed to transform the AI field. In addition, it represents a next step with regard to the creation of autonomous systems with a higher-level knowledge of the visual world. Currently, RL is also applied to the field of robotics, allowing control policies for robots to be learned directly from camera inputs in the real world [59]. The learning capability of the RL model exploits learning from the environment, learning from the feedback of UE, and learning from its mistakes, which may mitigate the challenges encountered in conventional IRS-enhanced wireless networks, and thus the performance can be improved. In recent years, several works related to RL-based IRS wireless communication have been carried out to enhance the communication performance, focusing on 6G network application [60; 61; 62]. Figure 7 shows a typical application scenario for RL-based communication. Each user sends the feedback through a relay buffer, which is used to update the beamforming policy by the base station. In [63], the authors proposed deep deterministic policy gradient (DDPG)-based models in the IRS wireless network with the multiple input single output (MISO) communication system. Based on the simulation result, the IRS-NOMA downlink outperforms IRS orthogonal multiple access (OMA) downlink transmission significantly. However, the complexity of the proposed methods increases exponentially as the number of reflecting element increases.  
The phase shifts of the IRS structure are mainly of discrete form, so the deep Q-network (DQN) learning model is appropriate to mitigate the difficulty of the phase shift design. For acquiring the whole advantages of an IRS wireless communication system with the aid of an RL algorithm, the architectures of the joint transmit and passive beamforming of an IRS-assisted communication system were considered in MISO systems [64; 65], OFDM-based systems [66], wireless security systems [67], and millimeter wave systems [68]. Moreover, the RL-based algorithm is susceptible to simultaneously design the alternating optimization (AO) method for the transmit beamforming at the BS and the passive beamforming at the IRS device. For more details in [64], the authors proposed a DDPG-based algorithm by utilizing the sum rate as instant rewards for training the model to gain a maximum throughput. In the applied model, the successive transmit beamforming and IRS phase shift were cooperatively optimized with less complexity. A deep RL approach in [65] was proposed for IRS phase shift design. Under the Rician fading channel, the proposed approach almost achieves an upper bound in terms of the received SNR value. By using the direct optimizing interaction matrices from the sampled channel knowledge in [66], the authors applied a deep RL-based algorithm for maximizing the achievable communication rate in the IRS wireless network. The only single beam was used for training episodes in the applied deep RL algorithm. Thus, the proposed algorithm can avoid the data collection phase during the training period, which reduces the computational time. RL-based approaches can ensure security in IRS-based communication, which was reported in [67]. The optimal beamforming for both the base station and IRS reflection maximizes the worst-case secrecy rate. The proposed scheme can enable a secrecy rate with a satisfactory quality of service. In [68], authors proposed a deep RL-based algorithm that acquired a maximum throughput by estimating the perfect and imperfect CSI and modeling a return distribution for each state-action pair, and a univariate regression algorithm was also proposed, which modeled the inherent randomness interconnection between the IRS and communicating environment. In addition, a post-decision state and prioritized experience replay schemes were used to enhance the learning efficiency and secrecy performance.