## 2 Materials and Methods
### Proposed Method
#### 2.2.4 Details of Proposed Method  
We can find the process of proposed method as shown in Algorithm 1. First, we initialize ERM, online NN, and target NN. Then, the algorithm enters LC-MARL training stage. The training phase contains M episodes. In each episode, agents learn by continuously exploring environment until it satisfies termination condition. In D-TDD configuration task, the termination condition is that all UE buffers are cleared or exploration step reaches the maximum value. In each step, each BS uses online NN to estimate Q-value of all available duplex patterns under local observation \(S_{b}(t)\), and then selects the duplex patterns \(\alpha_{b}(t)\) according to \(\epsilon\)-_greedy_ strategy. After \(N_{t}\) time slots, the wireless network environment  
Figure 5: Binary LSTM. feeds back the reward of each BS. Leniency controller of each agent calculates the leniency value of state-action pair \((S_{b}(t),\alpha_{b}(t))\) in this step, and stores the data of this step into ERM. Finally, each agent extracts a mini-batch from the ERM to train the online network, and synchronizes the online NN and target NN every G step. ```
1:Initialize the wireless network environment
2:Initialize the ERM of each BS
3:Initialize the online NN of each BS with weight \(\theta_{b}\)
4:Initialize the target NN of each BS with weight \(\theta_{b}^{*}=\theta_{b}\)
5:/* Traning */
6:forepisode \(=1,2,\ldots,M\)do
7: Initialize uplink and downlink buffer length of each UE
8:\(t=0\)
9:while the termination condition is not satisfied do
10: /*Action Seclect*/
11:for BS \(b\) in \(B\)do
12: Observe local state \(S_{b}(t)\)
13: Use online NN to estimate Q-value of all \((S_{b}(t),\alpha_{b}),\forall\alpha_{b}\in W\)
14: Choose duplex pattern \(\alpha_{b}(t)=\operatorname*{argmax}_{\text{ff}_{b}\in\text{FW}}\text{Q}(S_{b} (t),\text{ff}_{b})\) with probability \(1-e\)
15:endfor
16: Global duplex pattern \(\alpha(t)=[\alpha_{1}(t),\ldots,\alpha_{N_{bs}}(t)]\)
17: Feed global duplex pattern to wireless communication environment
18: Calculate the SINR of each UE \(u\) according to Equations (3) and (4)
19: Calculate reward \(R_{b}(t)\) and evolved state \(S_{b}(t+N_{t})\) of each BS \(b\)
20: /*Online NN Training*/
21:for BS \(b\) in \(B\)do
22: Calculate leniency value \(l(S_{b}(t),\alpha_{b}(t))\) according to Equation (21)
23: Store experience tuple \((S_{b}(t),\alpha_{b}(t),R_{b}(t),S_{b}(t+N_{t}),l(S_{b}(t),\alpha_{b}(t)))\) to ERM
24: Sample a mini-batch of ERM
25: Calculate benchmark value according to Equation (20)
26: Perform gradient descent to train online NN of BS \(b\)
27:endfor
28: /* parameter synchronization */
29: Every \(G\) steps, set \(\theta_{b}^{*}=\theta_{b}\)
30:\(t=t+N_{t}\)
31:endwhile
32:endfor
33: Output trained online NN of each BS
34: Obtain D-TDD configuration of each BS by Formula (29)
35: Obtain global D-TDD configuration by Formula (30)
```  
**Algorithm 1** LC-MARL based decentralized D-TDD configuration  
After training process, we obtain each BS's D-TDD configuration policy \(\pi_{b}(\alpha_{b}|S_{b}(t))\) by trained online NN of each BS, as shown in Equation (29). \[\pi_{b}(\alpha_{b}|S_{b}(t))\leftarrow\left\{ \begin{array}{ll}1&if\alpha=A^{ *},A^{*}=argmax_{\alpha\in W}Q(\phi(S_{b}(t)),\alpha_{b}(t),\theta)\\ 0&else\end{array} \right. \tag{29}\]  
Because each BS obtains its own D-TDD configuration through independent training, we obtain the global D-TDD configuration policy \(\pi(\alpha|S(t))\) by Formula (30). \[\pi(\alpha|S(t)) = \prod_{b\in\mathbb{B}}\pi_{b}(\alpha_{b}|S_{b}(t))\] \[\alpha = [\alpha_{1},\alpha_{2},\ldots a_{N_{\text{bs}}}]\] \[S(t) = [S_{1}(t),S_{2}(t),\ldots S_{N_{\text{bs}}(t)}] \tag{30}\]  
It is worth mentioning that because LC-MARL independently learns D-TDD configuration strategy of each BS, it does not require UE service information of other BSs, which reduces signaling overhead and avoids data transmission between BSs. At the same time, our proposed method can be deployed on the MEC.