## 5 Outer and Inner Loop Optimization Algorithm with Online Learning
### Inner Loop Optimization: A TD3-Based Algorithm for RIS Phase Shifting and Power Allocation  
The Twin Delayed Deep Deterministic Policy Gradient (TD3) is fundamentally an off-policy model that is well suited for continuous high-dimensional action spaces. Similar to DDPG, TD3 adopts an actor-critic structure. The actor network is responsible for approximating the policy function \(\pi(s,\theta_{\pi})\), where the weights \(\theta_{\pi}\) are trained to return the best action for a given state. Concurrently, the critic network assesses the value of the chosen action through the value function approximation \(q(s,a,\theta_{\sharp 1})\) based on the neural network, with its weights \(\theta_{\sharp 1}\) trained to represent the long-term reward function \(q(\cdot)\). TD3 is primarily an off-policy model suitable for continuous high-dimensional action spaces. Similar to DDPG, TD3 follows an actor-critic structure. The actor network approximates the policy function \(\pi(s,\theta_{\pi})\), with weights \(\theta_{\pi}\) trained to return the best action for a given state. Simultaneously, the critic network assesses the value of the chosen action through the value function approximation \(q(s,a,\theta_{\sharp 1})\) based on the neural network, with its network weights \(\theta_{\sharp 1}\) trained to represent the long-term reward function \(q(\cdot)\). In comparison to DDPG, TD3 offers the following advantages [27]. **Reduced Overestimation Bias:** TD3 introduces a second critic network (\(\theta_{\sharp 2}\)) and a second target critic network (\(\tilde{\theta}_{\sharp 2}\)) to address the overestimation bias that can occur in DDPG. By considering the minimum of the value estimates from the two critics, TD3 aims to provide more accurate value estimates and is less susceptible to overestimating Q-values. **Clipped Double Q-Learning:** TD3 incorporates a second critic network to enhance the stability of the learning process. This clipped double Q-learning approach contributes to improving the learning stability. **Delayed Policy Updates:** TD3 updates the actor network and all target networks at a lower frequency than the critic network. This decoupling of actor and critic updates reduces the interdependence between the networks, resulting in more stable learning and fewer oscillations in the policy. In this section, an intelligent resource allocation algorithm is proposed based on the TD3 framework. For this inner loop problem, each device-to-device (D2D) pair has selected RIS \(m\) and obtained the partition if more than one player has chosen the same RIS and RB in time \(t\). Then, the transmitter (Tx) controller of the D2D pair is treated as the agent, and the RIS-assisted ad-hoc communication is considered as the environment. Building upon the settings of the TD3 network, we first define the state, action, and reward settings for our problem and then provide an illustration of the proposed solution. These are outlined as follows. _(1) Problem Reformulation Based on MDP_  
The MDP problem includes the agent, state, action, reward, and environment. The elements of the MDP are illustrated as follows. * _State space:_ Let \(S\) be the state space, which contains the following components: (i) information about the current channel conditions, denoted as \(h_{i}^{t}\), \(f_{i}^{t}\) and \(g_{i}^{t}\); (ii) the positions and statuses of device-to-device (D2D) pairs \(p_{i}\); (iii) the actions, including the phase-shifting settings of the RIS elements and power allocation of \(Tx_{i}\) taken at time \(t-1\); (iv) the energy efficiency at time \(t-1\). Thus, \(S\) comprises \[s^{(t)}=\{\{h_{i}^{t},f_{i}^{t},g_{i}^{t}\}_{i\in N},p_{i},a^{(t-1)},\{\eta_{ EE,i}^{t}\}_{i\in N}\}\] (21)
* _Action space:_ Denote \(A\) as the action space, which consists of the actions that the agent can take. In this case, it includes the phase shifting of each RIS element and the transmission power of the transmitter (Tx) of the device-to-device (D2D) pair.