# Machine-Learning-Based Optimization for Multiple-IRS-Aided Communication System
## 4 Proposed Algorithm for Dynamic UE Scenario
### MDP Formulation  
As can be noted, solving the expressed optimization problem \((P1)\) by the traditional alternating approach is challenged by high complexity. Meanwhile, it needs to alternate over the relaxed sub-problems with the driven convex formulations to reach converged solutions of the coupled decision variables of the beamforming. Moreover, solving the optimization problem with the proposed algorithm presented in Section 3 eliminates the AO complexity at run time. However, it is still restricted to the static UE scenario. This is because the algorithm developed in Section 3 is a supervised-learning-based approach that needs to collect a large data set to reach an optimal regression surface to predict solutions based on static locations for UEs. Thus, when considering mobile UEs in a dynamic scenario, this algorithm cannot efficiently work. As motivated by DRL, the optimal beamforming strategy can be formed from observing the network states and the returned reward in the case of dynamic UEs. Thus, we propose a DRL-based approach to optimally solve the optimization problem and efficiently handle the dynamic UE scenario. Note that although the algorithm proposed in this section can work for the static UE case, it is still meaningful to adopt the algorithm in Section 3. This is because the GRNN model can be deployedmore quickly with a ready data set of feasible samples compared to the DRL-based model, which needs sufficient training to form its optimal decision policy. To utilize DRL, the first step is to convert the optimization problem into an MDP form. The MDP generally consists of five fundamental elements, i.e., agent, state \(s_{t}\), action \(a_{t}\), reward function \(r_{t}\), and policy \(\pi_{t}\)[35]. As shown in Figure 3a, at each time step \(t\), the agent receives instant observation or state \(s_{t}\in S\) from the environment, where \(S\) is the possible set of states, and responds to that state by selecting an action \(a_{t}\in A\), where \(A\) is the set of possible actions. Then, the environment updates its state to the next state \(s_{t+1}\) and provides performance metric reward \(r_{t}\in R\) to this state-action pair \((s_{t},a_{t})\). The mapping from states to actions based on the agent's policy, \(\pi_{t}(s_{t},a_{t})\), denotes the probability of choosing action \(a_{t}\) for the current state, \(s_{t}\). Over time, the agent aims to learn an optimal policy by maximizing the rewards received. Under our considered optimization problem, since the source BS decides on the optimal beamforming based on the surrounding environmental state, it functions as an agent. The state, action, rewards, and policy are described below. 1. State: The state \(s_{t}\) consists of all the current observed channel matrices of direct and indirect paths at time step \(t\), i.e., \((\{G_{R|}\}_{t=1}^{L})_{t}\) and \((\mathbf{H}_{d})_{t}\); the previous transmission power from the source to each \(k\)-th active UE at time step \(t-1\), i.e., \((\binom{p_{k}^{trans}}{k}_{t-1}\); and the previously received powers at each \(k\)-th active UE at time step \(t-1\), i.e., \((\mathbf{P}_{k}^{rec})_{t-1}\). Here, the transmission power for each \(k\)-th UE is defined as \((\mathbf{P}_{k}^{trans})_{t-1}=(\|\mathbf{w}_{k}\|^{2})_{t-1}\), which leads to \(K\) inputs stacked to the state. As each \(k\)-th UE in the system can receive its desired signal and interference signals due to other \(k-1\) UEs, the received power for each \(k\)-th UE is defined as \((\mathbf{P}_{k}^{rec})_{t-1}=|\mathbf{h}_{k}^{H}\mathbf{W}|^{2}\). This induces \(K\) inputs for each UE; consequently, we have \(K^{2}\) stacked in total to the state. Mathematically, the state at the time step \(t\) is given by \[s_{t}=\ \{(\{\mathbf{G}_{R|}\}_{t=1}^{L})_{t},(\mathbf{H}_{d})_{t},(\{P_{k}^{trans}\}_{k=1} ^{K})_{t-1},(\{\mathbf{P}_{k}^{rec}\}_{k=1}^{K})_{t-1}\}\ \in S.\] (18) Note that since the complex-valued features of channel inputs are treated as two separate features, the instant state dimension is \(2(LMK+MK)+K+K^{2}\). 2.