## 2 Materials and Methods
### Proposed Method
#### 2.2.2 Leniency Control Based Distributed MARL  
The idea of leniency control is derived from the application of lenient learning in [30] by [PERSON] and [PERSON], they applied lenient learning to extend traditional GA which improves the performance of solving complex problems. Ref. [31; 32] applied lenient learning to MARL. In [31], the author verified that leniency control can effectively prevent relative over-generalization. The motivation of [32] is to help co-evolutionary algorithms converge towards a sub-optimal policy, which is a difficult task because the noise generated by other agent's exploration strategy makes learning environment non-stationary. When we use distributed MARL to solve the D-TDD configuration problem, the noise is caused by other BS's D-TDD configuration strategy. So, we apply a leniency control to solve the problem of non-stationary learning environment in D-TDD configuration tasks. As described in Equation (19), leniency controller maintains a leniency value for each tuple \((S_{b}(t),\alpha_{b}(t))\).  
\[l(S_{b}(t),\alpha_{b}(t))=1-e^{-K\kappa T(S_{b}(t),\alpha_{b}(t))} \tag{19}\]  
where \(K\) is a leniency moderation factor, and \(T(S_{b}(t),\alpha_{b}(t))\) is temperature function. Each state-action pair initially assigned a defined maximum temperature value, and temperature value \(T(S_{b}(t),\alpha_{b}(t))\) decayed each time that the state-action pair is visited. Following the update, \(T_{t}(S_{b}(t),\alpha_{b}(t))\) is decayed using a discount factor \(\beta\in[0,1]\), such that \(T_{t+1}(S_{b}(t),\alpha_{b}(t))=\beta T_{t}(S_{b}(t),\alpha_{b}(t))\). Thus, we can find that leniency value of a state-action pair decreases when the times it is visited increases. Given a TD-Error \(\delta\), where \(\delta=Y(S_{b}(t),\alpha_{b}(t))-Q(S_{b}(t),\alpha_{b}(t),\theta)\), leniency control is applied to the calculation of benchmark value as follows:  
\[Y_{l}(S_{b}(t),\alpha_{b}(t))=\left\{\begin{array}{ll}Q(S_{b}(t),\alpha_{b} (t),\theta)+\delta&\delta>0\ or\ x>l(S_{b}(t),\alpha_{b}(t))\\ Q(S_{b}(t),\alpha_{b}(t),\theta)&\delta\leq 0\ and\ x\leq l(S_{b}(t),\alpha_{b}(t)) \end{array}\right. \tag{20}\]  
The random variable \(x\sim U(0,1)\) is adopted to ensure that an update on a negative \(\delta\) is executed with a probability \(1-l(S_{b}(t),\alpha_{b}(t))\). As shown in Equation (20), when the TD-error of the Q-value is less than 0, the lenient controller tends to ignore the sampling of this action. This reduces the impact of noise on learning in a non-stationary learning environment. On the other hand, the lenient controller does not ignore all samples when TD-error is less than 0, but effectively ignores it with probability \(1-l(S_{b}(t),\alpha_{b}(t))\). According to Equations (19) and (20), leniency control tends to forgive (ignore) sub-optimal Q-value iteration in the initial exploration phase. As the agent explores, the probability of leniency control ignoring the sub-optimal Q-value decreases. That is to say, an agent is less lenient to a frequently visited state-action tuple Q-value iteration, while quite lenient in unexplored  
Figure 2: State transition diagram.  
areas. This also avoids agents iterating towards a local optimum instead of a global optimum, that is, a relative over-generalization. Thus, leniency control increases the ability of convergence towards the globally optimal solution in distributed MARL. The specific process of LC-MARL is shown in Figure 3.  
LC-MARL fulfill decentralized D-TDD configuration task, but it has a shortcoming: weak learning ability in a high-dimensional space environment. In D-TDD configuration learning task, the state is defined as a time series of the UE buffer, whose space dimension is high due to that the number of slots in a frame is large. So, maintaining a leniency value of each visited state-action pair is no longer feasible. Motivated by this shortcoming, we apply a binary auto-encoder based self-supervised temporal hashing method to improves the learning ability of LC-MARL in a high-dimensional environment.