## 1 Introduction  
Speech enhancement (SE) has been widely used as a preprocessing step in speech-related tasks, such as automatic speech recognition, speaker recognition, hearing aids, and enhanced mobile communication. It attempts to remove background noise from a noisy signal using a single microphone or a microphone array. There have been many studies on statistical SE techniques, including Wiener filtering, the minimum mean square error (MMSE)-based spectral amplitude estimator [1], and non-negative matrix factorization (NMF) [2, 3]. Among them, sparse NMF (SNMF) achieves the best performance in noise reduction with matched noise that is modeled by the noise basis. In the last decade, single-channel SE methods based on deep learning have significantly improved the performance of such statistical approaches, even though these techniques require a large amount of training data due to their more complex neural network (NN) architectures for better SE performance. Deep learning-based SE methods can largely be classified into two categories depending on what they estimate. The SE methods in the first category estimate the magnitude spectrum of clean speech,such as the deep denoising autoencoder [4], deep recurrent NN (DRNN) [5], and convolutional NN (CNN) [6]. These methods provide a high signal-to-noise ratio (SNR) due to their good estimated magnitude spectrum matching to clean speech, but the intelligibility of the estimated clean speech is somewhat degraded when using the noisy input speech phase for clean speech estimation. To overcome this problem, the SE methods belonging to the second category estimate time-domain clean speech using WaveNet [7] or the SE generative adversarial network (SEGAN) [8]. While these approaches in the time domain improve speech intelligibility, they can lead to the issue of missing high-frequency components, thus resulting in degraded speech intelligibility [9; 10]. On the other hand, an NN can be applied in the time-frequency domain [11; 12; 13; 14; 15] and is motivated by the computational auditory scene analysis (CASA). As used in CASA, which has shown considerable promise in preserving speech intelligibility [16], various deep learning-based SE methods have been proposed to estimate the ideal binary mask (IBM) [12]. Although speech intelligibility could be improved through IBM-based SE, this improvement depends largely on the IBM estimation performance. To reduce this dependency, the ideal ratio mask (IRM) is defined by the ratio of the clean signal energy to the mixture energy at each time-frequency bin, and the IRM-based SE method provides better speech quality and intelligibility than the IBM-based one [17]. Among the many IRM estimation methods, the U-shaped NN (U-Net) has shown excellent speech enhancement performance as the skip connections in the U-Net helped restore the speech spectrum [15]. Since the U-Net was trained only using the clean speech spectrum as the target of the network, as with most deep learning-based methods, the accuracy of the estimated IRM was highly dependent only on the accuracy of the clean speech estimate [5]. However, if an NN could simultaneously estimate both clean speech and noise that has been contaminated in the noisy input speech, the estimation performance of the IRM could be improved by computing the IRM as the ratio between the estimated clean spectrum and the sum of the estimated clean and noise spectra at each time-frequency bin. In addition to SE, voice activity detection (VAD) plays an important role in speech-related applications [18]. Typically, VAD is carried out using the clean speech estimates from SE modules in noisy environments [19; 20; 21]. In this case, statistical or NN-based VAD requires hand-labeled annotations to train the VAD models. In practice, it is hard to annotate precise voice activity for noisy speech signals when their corresponding clean speech signals are not available; this problem is more severe for NN-based VAD [22; 23; 24] because it requires a large amount of training data.