## 4 Causal MDP-Based RIS-Assisted Resource Allocation Optimization with Online Reinforcement Learning
### Causal Factor Encoder and Decoder with a Deep Neural Network  
There are \(L\) electric vehicle (EV) users in the charging network and \(L\) corresponding subcarriers in the network, where each subcarrier's state information is represented as the concatenation of \(\mathbf{s}_{i}\) described in Equation (9) and \(a\) described in Equation (10), \([s_{t}^{i},a_{t}]\in\mathbb{R}^{d}\), where \(i=1,2,\dots,L\), and \(d\) is the dimensionality of the state features. The input sequence is represented as \(\mathbf{S}=\big{[}[\mathbf{s}_{1}^{i},a_{t}],[\mathbf{s}_{2}^{i},a_{t}],\dots,[\mathbf{s}_{L}^{L},a_{t}]\big{]}\in\mathbb{R}^{d\times L}\). The corresponding feature  
Figure 4: A3C network structure.  
representation \(Z_{t}^{i}\in\mathbb{R}^{d\times 1}\) can be regarded as the global environmental information for subcarrier \(i\), \(s_{j}^{t}\) is subcarrier \(j\)'s state and \(\alpha_{ij}\) is a weight for \(s_{t}^{j}\):  
\[z_{t}^{i}=\text{Softmax}\Big{(}h_{t}^{j}\Big{)}=\frac{\exp\Big{(}\text{relu} \big{[}\alpha^{T}\big{[}h_{t}^{i}\big{|}h_{t}^{j}\big{]}\big{]}\Big{)}}{\sum_{v \in N_{i}}(\text{relu}\big{[}\alpha^{T}\big{[}h_{t}^{i}\big{|}h_{t}^{v}\big{]} \big{]})}, \tag{18}\]  
where \(\alpha\in\mathbb{R}^{d\times 1}\) is the attention vector.  
For the state set of the subcarrier i's neighborhood \(L_{i}\), i.e., \(s_{t}^{L_{i}}=\{s_{j}^{t}\}_{j\in Ni}\), we employ fully connected (FC) layers (i.e., a shared weight matrix \(W_{i}\in\mathbb{R}^{d\times 1}\), where \(d\) is set to 256 in the simulation section) to transform the input state \(s_{j}^{t}\) of each subcarrier j and then obtain the embedding \(z_{j}^{t}\in\mathbb{R}^{d\times 1}\). Thus, there will be a total of \(|L_{i}|\) embeddings, i.e., \(\{z_{j}^{t}\}_{j\in L_{i}}\). Finally, an attention mechanism layer aggregates all these embeddings to obtain \(Z_{t}^{L_{i}}\in\mathbb{R}^{d\times 1}\), which can be considered as the current global embedding of subcarrier i. The network architecture is shown in Figure 5. The complementation is provided in Algorithm 1 below.  
```
1:Input: State set of each subcarrier \(s_{j}^{t}\), \(\forall j\); subcarrier neighborhood sets \(L_{i}\), \(\forall i\); weight matrix \(W_{i}\); attention vector \(l\)
2:Position Encoding:
3: Initialize the position encoding for each state vector to consider the sequence order.
4:Encoder:
5:for each time step do
6: Concatenate the state vectors of all subcarriers into a matrix \(X\).
7:for encoder layer do
8: Pass the output through a feed-forward neural network layer to capture non-linear relationships.
9: Compute the self-attention scores for each subcarrier's state vector.
10: Apply the self-attention mechanism to obtain the weighted sum of each user's state representation.
11:endfor
12: Obtain the causal factor for each subcarrier from the final encoder layer output.
13:endfor
14:return Causal factor for each subcarrier.
```  
**Algorithm 1** Causal factor extraction based on a self-attention mechanism  
Figure 5: Causal factor extraction network architecture.  
After the last hidden layer, the extracted causal factor representation for the system can be utilized for subsequent tasks, which is the resource allocation optimization of the communication network. The algorithm is provided in the following.