## 4 Causal MDP-Based RIS-Assisted Resource Allocation Optimization with Online Reinforcement Learning
### An Algorithm Utilizing Causal Factor-A3C for RIS Phase Shifting and Power Allocation  
The overall concept is shown in Figure 6. The causal factor extraction network learns from the SCM of the environment and trains on the agent networks. A3C facilitates parallel training by allowing multiple worker threads to interact with their copies of the environment and update the global network asynchronously. The A3C algorithm utilizes an asynchronous architecture, enabling parallel training across multiple worker threads. Each worker thread interacts with its instance of the environment and updates the global actor and critic networks asynchronously. At the core of the A3C method lies the central actor network, responsible for determining which actions to take based on the current situation represented by \(z\). Guided by its parameters \(\theta_{\text{actor}}\), this network is fine-tuned to select the best possible action given the state, denoted as \(a=\pi(z|\theta_{\text{actor}})\). Meanwhile, the critic network evaluates the value of chosen actions by estimating the Q-value function \(Q(z,a|\theta_{\text{critic}})\), where \(\theta_{\text{critic}}\) represents the critic network's parameters. These parameters are adjusted during training to capture the overall long-term rewards represented by \(q(\cdot)\). In our A3C approach (Algorithm 2), we directly update the global actor and critic networks based on experiences collected by multiple worker threads. Here is how the A3C network would be regenerated without the replay buffer. ```
1:Input: Global network parameters \(\theta^{\text{global}}\), number of threads \(N_{\text{threads}}\)
2:Initialize global network parameters \(\theta^{\text{global}}\)
3:for thread = 1 to \(N_{\text{threads}}\) in parallel do
4: Initialize thread-specific environment and network parameters \(\theta^{\text{local}}\leftarrow\theta^{\text{global}}\)
5: Initialize episode counter \(episode=0\)
6:while not done do
7: Receive initial observation state \(s_{1}\)
8: Put state \(s_{t}\) into the causal factor extraction network to obtain causal factor \(z_{t}\)
9:for t = 1 to \(T_{\text{max}}\)do
10: Select action \(a_{t}=\pi(z_{t}|\theta^{\text{local}})\) using policy network
11: Execute action \(a_{t}\) and observe reward \(r_{t}\) and new state \(s_{t+1}\)
12: Put state \(s_{t+1}\) into the causal factor extraction network to obtain causal factor \(z_{t+1}\)
13: Perform gradient ascent on global network parameters using the observed transition
14: Synchronize local network parameters with global network parameters
15:endfor
16: Increment episode counter episodes
17:endwhile
18:endfor
```  
**Algorithm 2** CF-A3C-based resource allocation optimization algorithm  
Figure 6: Overall concept. Global actor and critic networks are initialized with parameters \(\theta^{\text{actor}}\)global and \(\theta^{\text{critic}}\)global. Multiple worker threads are created, each with its instance of the environment and local actor and critic networks. Each worker thread interacts asynchronously with its environment. At each time step \(t\), the worker receives the current state \(s_{t}\) from the environment and puts it into the causal factor extraction network to get the causal output \(z_{t}\). They then select an action \(a_{t}\) according to the policy \(\pi(z_{t}|\theta_{\text{actor}})\), execute the action \(a_{t}\) and observe the reward \(r_{t}\) and the next state \(s_{t+1}\). Also, they obtain \(z_{t+1}\) through the same process. They store the experience tuple \((z_{t},a_{t},r_{t},z_{t+1})\), calculate the advantage function using the sampled batch, update the actor parameters \(\theta^{\text{actor}}_{\text{global}}\), and calculate the loss for the critic network using the sampled batch to update the critic parameters \(\theta^{\text{critic}}_{\text{global}}\). The gradients computed by each worker are applied asynchronously to the global actor and critic networks.