## 2 Materials and Methods
### Proposed Method
#### 2.2.1 Reinforcement Learning  
In reinforcement learning, agents continuously collect states from environment and choose action based on feedback rewards. In our considered scenario, agents are BSs, and actions are candidate D-TDD configurations (UL/DL slots ratio in a frame). After BSs choose the configuration, the network run for the next time frame and feedback of network performance are collected to calculate the rewards of the corresponding actions. We define state, action, and Q-value as follow. 1. State: we format the state \(s_{b}^{t}\) of BS \(b\) at time slot \(t\) to a tuple, where the element is uplink and downlink buffer length of BS \(b\). This is described as: \[s_{b}^{t}=(\Omega_{b}^{ul}(t),\Omega_{b}^{dl}(t))\] (13) where \(\Omega_{b}^{ul}(t)\) and \(\Omega_{b}^{ul}(t)\) represents the uplink and downlink buffer length of BS \(b\) at time slot \(t\). We normalize all of the state parameters values. In our proposed method, the scheduling time interval is one frame, that is, \(N_{t}\) time slots. Instead of conventional reinforcement learning that only considers the state of one time slot, we consider states of historical time sequence with \(N_{t}\) time slots in a frame for representing the state \(S_{b}(t)\) of BS \(b\) at scheduling moment \(t\). Thus, \(S_{b}(t)\) can be formulated as: \[S_{b}(t)=(s_{b}^{t-N_{t}},s_{b}^{t-N_{t}+1},\dots,s_{b}^{t-1})\] (14) where the number of \(N_{t}\) is obtained by Table 1 according to numerology \(\mu\). BS \(b\) performs D-TDD configuration at scheduling moment \(t\). After the \(N_{t}\) time slots, BS obtains the next state \(S_{b}(t+N_{t})\) feedback by the wireless environment. Then, BS \(b\) performs the configuration at the scheduling moment \(t_{N_{t}}\). The process of state transition is described in Figure 2, where the termination condition is that all UE buffers are cleared. 2. Action: in D-TDD configuration task, the action space \(W\) of each BS is the available configurations of UL/DL ratio. According to [26], we consider that there are six different types of duplex pattern. Thus, the action of BS \(b\) at time \(t\) is defined as \[\alpha_{b}(t) =(N_{up},N_{down})\in W\] \[W =\{(1,9),(2,8),(3,7),(4,6),(5,5),(6,4)\}\] (15)where \(N_{up}\) and \(N_{down}\) are uplink and downlink ratio. BS selects action following \(\epsilon-greedy\) policy according to Q-value of tuple \((S_{b}(t),\alpha_{b}(t))\), and the calculation method of Q-value is described in the next part. 3. Q-value: after agent selects action, the reward \(R_{b}(t)\) is calculated to evaluate the feedback of tuple \((S_{b}(t),\alpha_{b}(t))\) and agent transits to the next state \(S_{b}(t+N_{t})\). As described in Equation (11), our purpose is to transmit uplink an downlink services data in time. Therefore, the reward function of tuple \((S_{b}(t),\alpha_{b}(t))\) can be defined as: \[R_{b}(t)=-(r_{b}^{up}(t)+r_{b}^{down}(t))=-(\sum_{u\in b}\omega_{u}^{ul}(t+N_ {t})+\sum_{u\in b}\omega_{u}^{dl}(t+N_{t}))\] (16)  
In the Formula (16), \(r_{b}^{ul}(t)\) and \(r_{b}^{dl}(t)\) denote the uplink and downlink reward at time \(t\), which are negatively correlated with the sum of intra-BS UE uplink buffer length \(\omega_{u}^{ul}(t+N_{t})\) and downlink buffer length \(\omega_{u}^{dl}(t+N_{t})\) at the beginning of the next scheduling moment \(t+N_{t}\). In terms of Bellman Equations for deterministic policies [27], the Q-value of tuple \((S_{b}(t),\alpha_{b}(t))\) can be written as:  
\[Q(S_{b}(t),\alpha_{b}(t))=R_{b}(t)+\gamma\text{max}_{\alpha\in W}Q(S_{b}(t+N_ {t}),\alpha_{b}(t+N_{t})) \tag{17}\]  
For high dimensions of states, it is difficult to maintain a Q-table for agents. So we use a neural network (NN) to estimate the Q-value. Agents save \((S_{b}(t),\alpha_{b}(t),S_{b}(t+N_{t}),R(S_{b}(t),\alpha_{b}(t))\) of each exploration into experience reply memory (ERM). During training, agents select a training mini-batch from ERM to train the online NN.