Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport

1 Introduction . The goal of speech enhancement ( SE ) is to convert low-quality speech signals to ones with improved quality and intelligibility . SE serves as an important regression task in the speech-processing field and has been widely used for a pre-processor in speech-related applications , such as speech coding [ 1 ] , automatic speech recognition ( ASR ) [ 2 ] , speaker recognition [ 3 ] , and assistive hearing devices [ 4 , 5 ] . Recent advances in machine learning have made significant progress to the SE technology . Generally , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning-based SE approaches estimate a transformation to characterize the mapping function from noisy to clean speech signals in the training phase [ 6 ] . The estimated transformation converts noisy speech signals to generate clean-like signals in the testing phase . Various neural network models have been used to characterize noisy-to-clean transformations . Well-known examples of such models include fully connected neural network [ 7 ] , deep denoising autoencoder [ 8 ] , convolutional neural network [ 9 ] , long-short-term memory [ 10 ] , and Transformer [ 11 ] . To effectively handle diverse noisy conditions , we usually prepare a considerable amount of training data that cover various noise types to train SE models . However in real-application scenarios , the noise types in the testing data may not always be involved in the training set . Consequently , the noisy-to-clean transformation learned from the training data can not be suitably applied to handle the testing noise , resulting in limited enhancement performance . This training-testing mismatch is generally called a domain mismatch issue for SE . An effective solution is required to perform domain adaptation to adjust the SE models with formulating a precise noisy-to-clean transformation that matches the testing conditions . Most existing domain adaptation methods rely on at least one of the following adaptation mechanisms : aligning domain-invariant features [ 12 , 13 , 14 ] and adversarial training , where a discriminator is introduced during training as a domain classifier [ 15 , 16 , 17 ] . This study aims to solve the unsupervised domain adaptation problem for SE by introducing optimal transport ( OT ) . In particular , we consider the circumstance where SE is tested on a target domain with completely unlabeled data , and only labeled data from the source domain is available for reference . Generally speaking , OT theory compares two ( probability ) distributions and considers all possible transportation plans in between to find one with a minimal displacement cost . The concept of OT can be applied to minimize domain mismatch and consequently achieve unsupervised domain adaptation . Even with the mathematical characteristics offered by OT , obstacles to excellent SE performance persist due to the complex structure possessed by human speech . To further overcome the obstacles , another concept from Generative Adversarial Network ( GAN ) is integrated to assist attaining sophisticated SE domain adaptation . Although an existing domain transition technique “ domain adversarial training ” and our proposal share similarity in names , the fundamental constructions are substantially different . A key element in our method lies in a discriminator utilized to examine speech output characteristics , instead of a domain classifier . More precisely , the discriminator in our method was employed to govern the output speech quality by learning the probability distribution of the source labels . This novel approach was designed especially for the unsupervised SE domain adaptation to exhibit excellent performance , which was verified on the VoiceBank and TIMIT datasets . Contributions We proposed a novel method designed specifically for unsupervised domain adaptation in a regression setting . This area of study still has very limited results ; moreover , the existing methods often require additional classification of source domains or may not yet be supported by strong regression applications . Conversely , our approach does not require any additional input information other than the source samples , source labels , and target samples . Our approach was applied to two standardized SE tasks , namely VoiceBank-DEMAND and TIMIT , and achieved superior adaptation performance in terms of both Perceptual Evaluation of Speech Quality ( PESQ ) and Short-Time Objective Intelligibility ( STOI ) scores . Furthermore , owing to the simple input requirements , we can easily investigate the effect of target sample complexity on our method by increasing the number of noise types allowed in the target domain , which has not been reported by previous literature to the best of our knowledge . 2 Related work . Adversarial domain adaptation : The main objective of Domain Adversarial Training ( DAT ) is to train a deep model ( from the source domain ) capable of adapting to other similar domains by leveraging a considerable amount of unlabeled data from the target domain [ 15 , 18 ] . A conventional DAT system consists of three parts , deep feature extractor , label predictor , and domain classifier . By using a gradient reversal layer , the extracted deep features are discriminative for the main learning task and invariant with shifts between the source and target domains . The DAT approach has been applied and confirmed to effectively compensate for the mismatch of source ( training time ) and target ( testing time ) conditions in numerous tasks , such as speech signal processing [ 19 , 20 ] , image processing [ 15 , 21 ] , and wearable sensor signal processing [ 22 ] . A later development in Multisource Domain Adversarial Networks ( MDAN ) [ 23 ] extended the original DAT to lift the constraint of single domain transition , utilizing multiple domain classifiers to extract discriminative deep features for the main learning task while being invariant to multiple domain shifts [ 3 , 24 ] . Optimal transport for domain adaptation : Hitherto , OT [ 25 , 26 ] has been utilized to domain adaptation [ 27 , 28 ] with related analytical results [ 29 ] . Furthermore , the concepts of OT have proved itself even more useful under a joint distribution framework [ 30 , 31 ] . More recently , to improve the sensitivity of OT to outliers , Robust OT was proposed [ 32 ] . Moreover , a method for combining the notion of adversarial domain adaptation with OT and margin separation has been proposed [ 33 ] . However , almost all experiments performed in these studies were classification problems , unlike the SE task that is the focus of our study . Domain adaptation in speech enhancement : Existing domain adaptation in SE approaches can be divided into two categories : supervised and unsupervised . For supervised domain adaptation , paired noisy and clean speech signals for the testing conditions are available to adjust the parameters in the SE models . In [ 34 , 35 ] , transfer-learning-based approaches have been proposed to adapt SE models to alleviate corpus mismatches . To combat the catastrophic forgetting issue , Lee et al. , proposed a SERIL algorithm that combines curvature-based regularization and path optimization augmenting strategies when preforming domain adaptation on SE models [ 36 ] . Conversely , for unsupervised domain adaptation , only noisy speech signals are provided , and the corresponding clean counterparts are not accessible . Generally , unsupervised domain adaptation has good applicability to real-world scenarios . In [ 37 ] , unsupervised domain adaptation for SE was performed by minimizing the Kullback-Leibler divergence between posterior probabilities produced by teacher and student senone classifiers without paired noisy-clean adaptation data . In [ 19 , 38 ] , the DAT approach was used to adapt SE models to new noisy conditions . Despite yielding promising performance , the existing unsupervised domain adaptation approaches require additional information , such as word labels , language models , and noise-type labels . In this paper , we propose a new approach : discriminator-constrained OT network ( DOTN ) to perform unsupervised domain adaptation on SE . In contrast to related works , DOTN does not require additional label information when adapting the original SE models to match new noisy conditions . Our experiments show that DOTN can effectively adapt the SE models to new testing conditions , and that it achieves better adaptation performance than previous adversarial domain adaptation methods , which require additional noise type information . 3 Method . 3.1 Problem setting and notation . Consider a source domain with paired data ( Xs , Ys ) = { ( xsi , ysi ) } Ns i=1 , where x s i ∈ Rn , ysi ∈ Rm stand for the input and the corresponding label of sample i . The unsupervised domain adaptation assumes that another target domain exists containing only data unlabeled , Xt = { xti ∈ Rn } Ns i=1 . The goal is to seek a ground truth estimator ( or statistical hypothesis ) f : Rn → Rm for the target labels Yt = { yti } Nt i=1 ( exist but not known ) , based on the knowledge provided by the source domain . The probability distribution of a dataset D is denoted by PD , where D is either Xs , Ys , Xt or Yt in our discussion . Our problem is to find a function f such that f induces a probability distribution Pf ( Xt ) in Yt with Pf ( Xt ) → PYt under certain measure . We propose using the concept of OT to solve this problem . 3.2 Proposed model : Discriminator-Constrained Optimal Transport Network ( DOTN ) . Given a pair of distributions PD1 and PD2 and a displacement cost matrix C ≥ 0 , OT solves for the transportation plan γ ∈ ∏ ( PD1 , PD2 ) that minimizes the total cost ( in a discrete setting ) min γ∈ ∏ ( PD1 , PD2 ) 〈C , γ〉F , ( 1 ) where ∏ ( PD1 , PD2 ) denotes the space of joint distributions with marginal PD1 and PD1 . 〈· , ·〉F is the Frobenius product , and the entry Cij of C represents the displacement cost of the ith and jth samples . It can be proven that the minimum of this problem is a distance and called the Wasserstein distance when the corresponding cost is a norm [ 25 , 26 ] . Our proposed method consists of two parts : OT alignment and Wasserstein Generative Adversarial Network ( WGAN ) training [ 39 , 40 ] . Both steps are based on OT ; however , they are considered with two different pairs of distributions and employ different algorithms . Adaptation by Joint Distribution Optimal Transport Our adaptation mechanism relies on the alignment between the joint distributions of source and target domains ( for the target domain , the label is estimated label ) . In particular , we approximate f by minimizing the OT loss between the joint distributions PXs × PYs and PXt × Pf ( Xt ) , with a chosen cost matrix , Cij = α ‖xsi − xtj‖2 + β ‖ysi − f ( xtj ) ‖2 , ( α , β > 0 ) ( 2 ) By aligning the joint distributions of the source and target domains , noise adaptation is naturally achieved as OT seeks the source sample that is the most “ similar ” for each target sample . Although the OT provides accurate estimates for each sample , the effect of each estimation error could accumulate in the training process and mislead f to a convenient local minimum without preserving the speech data structure . To avoid this situation , we employ Wasserstein Generative Adversarial Network ( WGAN ) training to complement and enhance our adaptation system . Discriminative training on outputs and source labels Distinct from the adaptation where we consider the joint distributions of the inputs and labels , we focus on WGAN training for the source label distribution PYs and output distribution Pf ( Xt ) . In the terminology of generative adversarial training , we consider f a generator and introduce a convolutional neural network-based discriminator , h , as the ’ critic ’ . In general , we use the discriminator to decide whether the outputs of f are ’ similar ’ to the source labels . Formally , the WGAN algorithm solves min f max h∈L { Ey∼PYs ( h ( y ) ) − Ex∼PXt ( h ( f ( x ) ) ) } ( 3 ) by Kantorovich-Rubinstein duality [ 25 ] , where L is the set of 1-Lipschitz functions . In this case , under an optimal discriminator minimizing the value function with respect to the generator parameters minimizes the Wasserstein distance between the distributions PYs and Pf ( Xt ) . This discriminative training complements our alignment and provides additional constraints from the explicit relation between the source labels and estimations of target labels . These constraints support the joint distribution alignment and provides further guidance in the gradient descent training process . The performance of the experiments is considerably improved when our discriminative training supports the joint distribution alignment .

Summary: This paper proposes using a discriminator-constrained optimal transport method for unsupervised speech enhancement (SE). The method is specifically designed for a regression task. It involves 2 parts. The first parts estimates a generator function as an OT problem to align the source and target labels. The source-target alignments ($\gamma$) and the generator function parameters (${\theta}_f$) are estimated in an interleaving fashion. The second part incorporates a discriminative training component on the label distributions using WGAN. Two discriminative loss functions were used, one for the generator (${\cal L}_f$) and another for the discriminator (${\cal L}_h$). The discriminator losses are applied periodically at a pre-defined interval ($n_f$ and $n_h$).   The proposed method is referred to as the discriminative-constrained optimal transport network (DOTN). It does not require any additional knowledge about the noise types during training. This paper compares DOTN with two other adversarial domain adaptation methods, namely DAT and MDAN, using two datasets (VoiceBank and TIMIT). Different types of noise are artificially added to these datasets using the DEMAND dataset at different SNR levels to create different noise domains. The quality of the speech enhancement methods are evaluated using the PESQ and STOI metrics.  The overall results show that the proposed DOTN outperforms DAT and MDAN on both datasets. However, the model sizes and computational complexities of the models are not reported, making it difficult to compare different methods. Besides, there is lack of detailed discussions/analyses about the design of the training algorithm (see main review).