## 2 Materials and Methods
### System Model
#### 2.1.3 Problem Formulation  
We consider a static power transmission scheme, where uplink transmission power of all UEs is the same, and downlink transmission power of all BSs is also the same. Uplink transmission power and downlink transmission power between UE \(u\) and BS \(b\) on each sub-channel \(c\) are denoted as \(P_{u,b}^{ul}\) and \(P_{u,b}^{dl}\), respectively. In order to ensure the fairness of UEs, we use a round-robin (RR) algorithm for sub-channel resource allocation. If sub-channel \begin{table}
\begin{tabular}{c c c c} \hline \(\mu\) & **SCS** & **Number of Slots Per Frame** & **Time Length of Slot** \\ \hline
0 & 15 khz & 10 & 1 ms \\
1 & 30 khz & 20 & 0.5 ms \\
2 & 60 khz & 40 & 0.25 ms \\
3 & 120 khz & 80 & 0.125 ms \\
4 & 240 khz & 60 & 0.0625 ms \\ \hline \end{tabular}
\end{table} 
Table 1: Supported transmission numerologies and additional info. \begin{table}
\begin{tabular}{c c c} \hline \(\delta\) **(ms)** & \(\mu\) & **SCS (kHz)** \\ \hline
0.625 & 3 & 120 \\
1.25 & 2, 3 & 60, 120 \\
2.5 & 1, 2, 3 & 30, 60, 120 \\ \hline \end{tabular}
\end{table} 
Table 2: Supported transmission numerologies and period of UL/DL pattern. is allocated to UE \(u\), we set the indicator \(\kappa^{c}_{u,b}\) to 1, otherwise 0. Therefore, the uplink and downlink transmission SINR of UE \(u\) at time \(t\) can be written as Equations (3) and (4). \[\gamma^{ul}_{u}(t)=\frac{\sum_{b\in B}\sum_{c\in C}\eta_{u,b}\cdot\kappa^{c}_{u,b }\cdot P^{ul}_{u,b}\cdot\delta^{c}_{u,b}}{\sigma^{2}+\sum_{u\in U}\sum_{b\in B} \sum_{c\in C}l^{ul}_{u,b}(t)+\sum_{u\in U}\sum_{b\in B}\sum_{c\in C}l^{dl}_{u,b }(t)} \tag{3}\]  
\[\gamma^{dl}_{u}(t)=\frac{\sum_{b\in B}\sum_{c\in C}\eta_{u,b}\cdot\kappa^{c}_{u,b}\cdot P^{dl}_{u,b}\cdot\delta^{c}_{u,b}}{\sigma^{2}+\sum_{u\in U}\sum_{b\in B }\sum_{c\in C}l^{ul}_{u,b}(t)+\sum_{u\in U}\sum_{b\in B}\sum_{c\in C}l^{dl}_{u, b}(t)} \tag{4}\]  
\[l^{ul}_{u,b}(t)=(1-\tau_{b}(t))\eta_{u,b}\cdot\kappa^{c}_{u,b}\cdot P^{ul}_{u, b}\cdot\delta^{c}_{u,b} \tag{5}\]  
\[l^{dl}_{u,b}(t)=\tau_{b}(t)\eta_{u,b}\cdot\kappa^{c}_{u,b}\cdot P^{dl}_{u,b} \cdot\delta^{c}_{u,b} \tag{6}\]  
where \(\sigma^{2}\) is additive white Gaussian noise power spectral density, \(l^{ul}_{u,b}(t)\) and \(l^{dl}_{u,b}(t)\) are UL and DL interference, respectively. Furthermore, UL and DL transmission rate \(r^{ul}_{u}(t)\) and \(r^{dl}_{u}(t)\) of UE \(u\) are computed as follows:  
\[r^{ul}_{ul}(t)=BW\cdot\log_{2}(1+\gamma^{ul}_{u}) \tag{7}\]  
\[r^{ul}_{dl}(t)=BW\cdot\log_{2}(1+\gamma^{dl}_{u}) \tag{8}\]  
where \(BW\) is the bandwidth of sub-channel. We assume \(\varrho^{ul}_{u}(t)\) and \(\varrho^{dl}_{u}(t)\) are the UL/DL packet size generated by UE \(u\) at time \(t\), then we can obtain total uplink buffer length \(\Omega^{ul}(t)\) and total downlink buffer length \(\Omega^{dl}(t)\) by  
\[\Omega^{ul}(t)=\sum_{b\in B}\Omega^{ul}_{b}(t)=\sum_{b\in B}\sum_{u\in b} \omega^{ul}_{u}(t)=\sum_{b\in B}\sum_{u\in b}(\omega^{ul}_{u}(t-1)+\varrho^{ ul}_{u}(t)-r^{u}_{ul}(t)) \tag{9}\]  
\[\Omega^{dl}(t)=\sum_{b\in B}\Omega^{dl}_{b}(t)=\sum_{b\in B}\sum_{u\in b} \omega^{dl}_{u}(t)=\sum_{b\in B}\sum_{u\in b}(\omega^{dl}_{u}(t-1)+\varrho^{ dl}_{u}(t)-r^{u}_{dl}(t)) \tag{10}\]  
where \(\Omega^{ul}_{b}(t)\), \(\Omega^{dl}_{b}(t)\), \(\omega^{ul}_{u}(t)\) and \(\omega^{dl}_{u}(t)\) are uplink buffer length of BS \(b\), downlink buffer length of BS \(b\), uplink buffer length of UE \(u\) and downlink buffer length of UE \(u\). In this research, our goal is to find a D-TDD configuration policy \(\pi(\alpha|(\Omega^{ul}(t),\Omega^{dl}(t))\)\(=(x,y))\) that maximizes the expected discount return of all UE's sum rate. That is to say, the expected discount return of all UE's sum rate increases means the UE uplink and downlink data are transmitted in time. In order to achieve this goal, we obtained the utility function \(f(t,(\Omega^{ul}(t),\Omega^{dl}(t))=(x,y))\) described in Equation (11).