## 3 Results  
In this section, we carry out a simulation to validate our proposed lenient-MARL based D-TDD duplex control framework. First, we investigate the convergence of our proposed MARL algorithms. Second, we evaluate the performance of the framework in terms of overall system rate and each BS data transmission rate. For the simulation environment, we set up a server powered by a dual Intel(R) Xeon CPU 2.40 GHz 4-cores and 16 GB RAM. We use python3.8 for the wireless communication simulation. Further, we build BLSTM, auto-encoder, and deep learning static graph with tensorflow1.0. A summary of the wireless environment simulation parameters is provide in Table 3. We assumed that each BS uses the same frequency. All UEs are distributed within BS coverage area, and the number of UEs in the coverage area follow Poisson distribution with expectation \(\lambda_{UE}\). Each UE selects BS which provides the max reference signal received power RSRP as a serving BS. \(\lambda_{h}\) and \(\lambda_{l}\) represent the high-load and low-load packet generation rates, respectively. The UE uplink and downlink high-load probability is represented by \(p_{UL}^{high}\) and \(p_{DL}^{high}\). Moreover, four types of traffic are considered in our simulation, namely high uplink load, high downlink load, high load, and low load. The specific parameters of the four flow types are shown in Table 4.  
Table 5 summarizes the hyper-parameters of LC-MARL. In order to estimate Q-value of time series state, we extract feature of the time series state through the LSTM layer. Then, we construct a three-layer fully connected neural network, with the number of cells in each layer decreasing. For each learning step, _Adam_ is adopted to optimize the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Traffic Type** & \(p_{UL}^{high}\) & \(p_{DL}^{high}\) \\ \hline High uplink load & 0.8 & 0.2 \\ High downlink load & 0.2 & 0.8 \\ High load & 0.8 & 0.8 \\ Low load & 0.2 & 0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Traffic type. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline BS number & 2, 3, 4, 5 \\ Sub-channel number & 10 \\ Sub-channel bandwidth & 1 Mhz \\ BS power & 46 dbm \\ BS distance & 500 m \\ \(\lambda_{UE}\) & 8 \\ \(\lambda_{h}\) & 325 \\ \(\lambda_{l}\) & 125 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation parameters of wireless environment. hyper-parameter of online NN. This method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients [34], and the initial learning rate is shown in Table 5. The discount rate is 0.9. A higher discount rate value enhances the effect of future reward on Q-value and encourages agents to learn a policy to reach the termination faster. In this case, that means the data in the buffer are transmitted in time. Figure 6a,b plot the convergence of our proposed LC-MARL algorithm with different packet size. In the Figure, the \(x\)-axis and \(y\)-axis represents episode steps and total system reward. It is clear that total system reward increases with continuous training. Unlike Figure 6a,b has a larger total reward jitter. For network performance analysis, we compare the proposed LC-MARL solution with decentralized deep reinforcement learning (DDRL) in terms of overall system rate and rate of distribute agents (each BS). For the detail of DDRL, we expand deep reinforcement learning proposed by paper [26] to multi-agent domain, which is a common learning cooperation MARL solution [35]. In DDRL based method, each BS obtains the D-TDD configuration through parallel distributed learning. At each scheduling moment, BS performs a D-TDD configuration based on state information of wireless environment, and obtains reward and next state feedback from environment at next scheduling moment, then stores data in memory. At each scheduling moment, BS trains a NN to estimate the Q-value through memory replay method.