## 5 Outer and Inner Loop Optimization Algorithm with Online Learning
### Inner Loop Optimization: A TD3-Based Algorithm for RIS Phase Shifting and Power Allocation
The action \(a^{(t)}\) is given by \[a^{(t)}=\{\mathbf{\Theta},\{\mathbf{W}_{i}\}_{i\in N}\}\] (22)
* _Reward function:_ The agent receives an immediate reward \(r_{i}^{t}\), which is the energy efficiency defined in Equation (16). This reward is affected by factors such as the channel conditions, RIS phase shifts, and device-to-device (D2D) power allocations, i.e., \[r_{i}^{t}=\eta_{EE,i}^{t}\] (23)  
_(2) Phase Shifting and Power Allocation Algorithm Based on TD3_  
In the architecture of our TD3 deep reinforcement learning (DRL) model, the actor network plays a pivotal role in action selection, while the critic network is responsible for evaluating actions. Parameterized by \(\theta_{\pi},\theta_{q1}\), and \(\theta_{q2}\), the actor and two critic networks are illustrated in Figure 3. The actor network selects actions denoted as \(a=\pi(s|\theta_{\pi})\) based on the current state \(s\). On the other hand, the critic networks take the current state \(s\) and action \(a\) as input, producing the Q value of the action under the policy \(\pi\) as: \(Q_{\pi}(s,a)=r(s,a)+\gamma*E[Q_{\pi}(s^{\prime},a^{\prime})]\), where \(s^{\prime}\) and \(a^{\prime}\) represent the next state and action, with \(a^{\prime}\) sampled from policy \(\pi_{s^{\prime}}\). The immediate reward is denoted as \(r(s,a)\), and \(\gamma\) signifies the discount factor. Figure 3: TD3 network structure. The critic networks approximate the Q value as \(Q(s,a|\theta_{qi})\), with \(i=(1,2)\). The target networks mirror the structure of the main networks. This innovative model architecture significantly enhances the learning capabilities and overall performance. In our approach, a replay memory is employed to archive experience pairs s, a, r, s\({}^{\prime}\), and we adopt the strategy of random batch sampling from this memory to compute the loss value and subsequently update the critic networks. The initial step involves using the target actor network to determine actions for the state \(s^{\prime}\), expressed as \(a^{\prime}=\pi^{\prime}(s^{\prime}|\theta^{\pi^{\prime}})\). Then, we introduce noise to the target action \(a^{\prime}\), with target policy smoothing regularization:  
\[a^{\prime}=a^{\prime}+\epsilon=\pi^{\prime}(s^{\prime}|\theta^{\pi^{\prime}})+\epsilon \tag{24}\]  
where \(\epsilon\sim\text{clip}(N(0,\sigma),-c,c)\) represents clipped noise with bounds of \(-c\) and \(c\). This strategy enhances the robustness of the learning process, promoting stability and convergence in the model. Persisting with the concept of utilizing dual networks, the computation of the target value follows a meticulous procedure. Specifically, the target value is determined as  
\[y=r+\gamma min_{i=1,2}Q_{i}^{\prime}(s^{\prime},a^{\prime}|\theta^{qi}) \tag{25}\]  
Ultimately, we employ the gradient descent algorithm to minimize the loss function associated with the critic networks. This loss function is defined as  
\[L_{ci}=(Q_{i}(s,a|\theta^{qi})-y)^{2}\quad(i=1,2) \tag{26}\]  
After completing \(d\) steps of updating the critic1 and critic2 networks, we start the update of the actor network. We utilize the actor network to compute the action for the state \(s\) as \(a_{new}=\pi(s|\theta_{\pi})\). Then, we perform the evaluation of the state-action pair \((s,a_{new})\) using either the critic1 or critic2 network, assuming, in this context, the use of the critic1 network. \[q_{new}=Q_{1}(s,a_{new}|\theta^{q1}) \tag{27}\]  
Finally, we apply a gradient ascent algorithm to maximize \(q_{\text{new}}\), thereby finalizing the update process for the actor network. The process of updating the target networks incorporates a soft update technique. This method introduces a learning rate or momentum parameter \(\tau\), which calculates a weighted average between the previous target network parameters and the new network parameters. The result is then assigned to the target network.