## 3 Proposed MTU-Net-Based Speech Enhancement
### Inference  
After finishing the training procedure described in Section 3.2, the clean and noise magnitude spectra for a given noisy input spectrum are estimated and are denoted as \(\left|\hat{X}_{i}(k)\right|\) and \(\left|\hat{D}_{i}(k)\right|\), respectively. As the first approach to estimate clean speech, \(\left|\hat{X}_{i}(k)\right|\) can be directly used as an estimate of the clean speech magnitude spectrum. Thus, the estimated clean speech, \(\hat{x}_{i}(n)\), is reconstructed by applying the phase of the noisy input speech, \(\angle Y_{i}(k)\) as  
\[\hat{x}_{i}(n)=\text{IFFT}\left\{\left|\hat{X}_{i}(k)\right|\exp(j\angle Y_{i }(k))\right\}. \tag{4}\]  
As a second approach, the mask is estimated and then applied to the spectrum of the noisy input speech. In other words, the IRM can be inferred from \(\left|\hat{X}_{i}(k)\right|\) and \(\left|\hat{D}_{i}(k)\right|\) as  
\[H_{i}^{IRM}(k)=\frac{\left|\hat{X}_{i}(k)\right|}{\left|\hat{X}_{i}(k)\right| +\left|\hat{D}_{i}(k)\right|} \tag{5}\]  
Figure 2: Network architecture of the proposed multi-task learning U-Net (MTU-Net)-based single-channel speech enhancement.  
where \(H_{i}^{IRM}(k)\) is the IRM of the \(k\)-th frequency bin at the \(i\)-th frame. Simultaneously, the IBM of the \(k\)-th frequency bin at the \(i\)-th frame, \(H_{i}^{IBM}(k)\), can be estimated as  
\[H_{i}^{IBM}(k)=\left\{\begin{array}{ll}1&\text{if }H_{i}^{IRM}(k)>\theta_{IBM} \\ 0&\text{otherwise}\end{array}\right. \tag{6}\]  
where \(\theta_{IBM}\) is the threshold to obtain the IBM from the IRM and is set to 0.5 according to [11]. This is because the SNR between the estimated clean and noise magnitude spectra, \(\left|\hat{X}_{i}(k)\right|\) and \(\left|\hat{D}_{i}(k)\right|\), becomes zero if \(\left|\hat{X}_{i}(k)\right|=\left|\hat{D}_{i}(k)\right|\), i.e., \(\text{SNR}_{\text{i}}(\text{k})=20\log\left(\frac{\left|\hat{X}_{i}(k)\right| }{\left|\hat{D}_{i}(k)\right|}\right)=0(\text{dB})\). Next, the estimated clean speech, \(\overline{x}_{i}(n)\), is reconstructed by multiplying the IRM or IBM with the noisy input speech spectrum, \(Y_{i}(k)\), followed by an IFFT, as follows:  
\[\overline{x}_{i}(n)=\text{IFFT}\!\left\{H_{i}(k)\!\left|Y_{i}(k)\right|\exp \!\left(j\angle Y_{i}(k)\right)\right\} \tag{7}\]  
where \(H_{i}(k)\) corresponds to either \(H_{i}^{IRM}(k)\) or \(H_{i}^{IBM}(k)\).  
Figure 3 summarizes the three different methods for estimating clean speech from MTU-Net: (i) the estimation of clean speech from the estimated clean speech spectrum and the phase of noisy input speech by using (4), referred to as MTU-Net(spec); (ii) the estimation of clean speech by applying IRM using (5) and (7), referred to as MTU-Net(IRM); and (iii) estimation by applying IBM using (6) and (7), referred to as MTU-Net(IBM). Since the loss function of MTU-Net takes into account the squared error between \(\left|\hat{X}_{i}(k)\right|\) and \(\left|X_{i}(k)\right|\), sometimes \(\left|\hat{X}_{i}(k)\right|\) is greater than \(\left|X_{i}(k)\right|\) or greater than \(\left|Y_{i}(k)\right|\). This degrades the quality of the estimated clean speech reconstructed by MTU-Net(spec) due to the fluctuation of the trajectory of \(\left|\hat{X}_{i}(k)\right|\) over all the frame against that of \(\left|X_{i}(k)\right|\). On the other hand, \(H_{i}(k)\!\left|Y_{i}(k)\right|\) produced by either MTU-Net(IBM) or MTU-Net(IRM) is always smaller than \(\left|Y_{i}(k)\right|\) since \(0\leq H_{i}(k)\leq 1\). Therefore, it is expected that MTU-Net(IBM) or MTU-Net(IRM) will provide better performance than MTU-Net(spec).