Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport

1 Introduction . The goal of speech enhancement ( SE ) is to convert low-quality speech signals to ones with improved quality and intelligibility . SE serves as an important regression task in the speech-processing field and has been widely used for a pre-processor in speech-related applications , such as speech coding [ 1 ] , automatic speech recognition ( ASR ) [ 2 ] , speaker recognition [ 3 ] , and assistive hearing devices [ 4 , 5 ] . Recent advances in machine learning have made significant progress to the SE technology . Generally , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning-based SE approaches estimate a transformation to characterize the mapping function from noisy to clean speech signals in the training phase [ 6 ] . The estimated transformation converts noisy speech signals to generate clean-like signals in the testing phase . Various neural network models have been used to characterize noisy-to-clean transformations . Well-known examples of such models include fully connected neural network [ 7 ] , deep denoising autoencoder [ 8 ] , convolutional neural network [ 9 ] , long-short-term memory [ 10 ] , and Transformer [ 11 ] . To effectively handle diverse noisy conditions , we usually prepare a considerable amount of training data that cover various noise types to train SE models . However in real-application scenarios , the noise types in the testing data may not always be involved in the training set . Consequently , the noisy-to-clean transformation learned from the training data can not be suitably applied to handle the testing noise , resulting in limited enhancement performance . This training-testing mismatch is generally called a domain mismatch issue for SE . An effective solution is required to perform domain adaptation to adjust the SE models with formulating a precise noisy-to-clean transformation that matches the testing conditions . Most existing domain adaptation methods rely on at least one of the following adaptation mechanisms : aligning domain-invariant features [ 12 , 13 , 14 ] and adversarial training , where a discriminator is introduced during training as a domain classifier [ 15 , 16 , 17 ] . This study aims to solve the unsupervised domain adaptation problem for SE by introducing optimal transport ( OT ) . In particular , we consider the circumstance where SE is tested on a target domain with completely unlabeled data , and only labeled data from the source domain is available for reference . Generally speaking , OT theory compares two ( probability ) distributions and considers all possible transportation plans in between to find one with a minimal displacement cost . The concept of OT can be applied to minimize domain mismatch and consequently achieve unsupervised domain adaptation . Even with the mathematical characteristics offered by OT , obstacles to excellent SE performance persist due to the complex structure possessed by human speech . To further overcome the obstacles , another concept from Generative Adversarial Network ( GAN ) is integrated to assist attaining sophisticated SE domain adaptation . Although an existing domain transition technique “ domain adversarial training ” and our proposal share similarity in names , the fundamental constructions are substantially different . A key element in our method lies in a discriminator utilized to examine speech output characteristics , instead of a domain classifier . More precisely , the discriminator in our method was employed to govern the output speech quality by learning the probability distribution of the source labels . This novel approach was designed especially for the unsupervised SE domain adaptation to exhibit excellent performance , which was verified on the VoiceBank and TIMIT datasets . Contributions We proposed a novel method designed specifically for unsupervised domain adaptation in a regression setting . This area of study still has very limited results ; moreover , the existing methods often require additional classification of source domains or may not yet be supported by strong regression applications . Conversely , our approach does not require any additional input information other than the source samples , source labels , and target samples . Our approach was applied to two standardized SE tasks , namely VoiceBank-DEMAND and TIMIT , and achieved superior adaptation performance in terms of both Perceptual Evaluation of Speech Quality ( PESQ ) and Short-Time Objective Intelligibility ( STOI ) scores . Furthermore , owing to the simple input requirements , we can easily investigate the effect of target sample complexity on our method by increasing the number of noise types allowed in the target domain , which has not been reported by previous literature to the best of our knowledge . 2 Related work . Adversarial domain adaptation : The main objective of Domain Adversarial Training ( DAT ) is to train a deep model ( from the source domain ) capable of adapting to other similar domains by leveraging a considerable amount of unlabeled data from the target domain [ 15 , 18 ] . A conventional DAT system consists of three parts , deep feature extractor , label predictor , and domain classifier . By using a gradient reversal layer , the extracted deep features are discriminative for the main learning task and invariant with shifts between the source and target domains . The DAT approach has been applied and confirmed to effectively compensate for the mismatch of source ( training time ) and target ( testing time ) conditions in numerous tasks , such as speech signal processing [ 19 , 20 ] , image processing [ 15 , 21 ] , and wearable sensor signal processing [ 22 ] . A later development in Multisource Domain Adversarial Networks ( MDAN ) [ 23 ] extended the original DAT to lift the constraint of single domain transition , utilizing multiple domain classifiers to extract discriminative deep features for the main learning task while being invariant to multiple domain shifts [ 3 , 24 ] . Optimal transport for domain adaptation : Hitherto , OT [ 25 , 26 ] has been utilized to domain adaptation [ 27 , 28 ] with related analytical results [ 29 ] . Furthermore , the concepts of OT have proved itself even more useful under a joint distribution framework [ 30 , 31 ] . More recently , to improve the sensitivity of OT to outliers , Robust OT was proposed [ 32 ] . Moreover , a method for combining the notion of adversarial domain adaptation with OT and margin separation has been proposed [ 33 ] . However , almost all experiments performed in these studies were classification problems , unlike the SE task that is the focus of our study . Domain adaptation in speech enhancement : Existing domain adaptation in SE approaches can be divided into two categories : supervised and unsupervised . For supervised domain adaptation , paired noisy and clean speech signals for the testing conditions are available to adjust the parameters in the SE models . In [ 34 , 35 ] , transfer-learning-based approaches have been proposed to adapt SE models to alleviate corpus mismatches . To combat the catastrophic forgetting issue , Lee et al. , proposed a SERIL algorithm that combines curvature-based regularization and path optimization augmenting strategies when preforming domain adaptation on SE models [ 36 ] . Conversely , for unsupervised domain adaptation , only noisy speech signals are provided , and the corresponding clean counterparts are not accessible . Generally , unsupervised domain adaptation has good applicability to real-world scenarios . In [ 37 ] , unsupervised domain adaptation for SE was performed by minimizing the Kullback-Leibler divergence between posterior probabilities produced by teacher and student senone classifiers without paired noisy-clean adaptation data . In [ 19 , 38 ] , the DAT approach was used to adapt SE models to new noisy conditions . Despite yielding promising performance , the existing unsupervised domain adaptation approaches require additional information , such as word labels , language models , and noise-type labels . In this paper , we propose a new approach : discriminator-constrained OT network ( DOTN ) to perform unsupervised domain adaptation on SE . In contrast to related works , DOTN does not require additional label information when adapting the original SE models to match new noisy conditions . Our experiments show that DOTN can effectively adapt the SE models to new testing conditions , and that it achieves better adaptation performance than previous adversarial domain adaptation methods , which require additional noise type information . 3 Method . 3.1 Problem setting and notation . Consider a source domain with paired data ( Xs , Ys ) = { ( xsi , ysi ) } Ns i=1 , where x s i ∈ Rn , ysi ∈ Rm stand for the input and the corresponding label of sample i . The unsupervised domain adaptation assumes that another target domain exists containing only data unlabeled , Xt = { xti ∈ Rn } Ns i=1 . The goal is to seek a ground truth estimator ( or statistical hypothesis ) f : Rn → Rm for the target labels Yt = { yti } Nt i=1 ( exist but not known ) , based on the knowledge provided by the source domain . The probability distribution of a dataset D is denoted by PD , where D is either Xs , Ys , Xt or Yt in our discussion . Our problem is to find a function f such that f induces a probability distribution Pf ( Xt ) in Yt with Pf ( Xt ) → PYt under certain measure . We propose using the concept of OT to solve this problem . 3.2 Proposed model : Discriminator-Constrained Optimal Transport Network ( DOTN ) . Given a pair of distributions PD1 and PD2 and a displacement cost matrix C ≥ 0 , OT solves for the transportation plan γ ∈ ∏ ( PD1 , PD2 ) that minimizes the total cost ( in a discrete setting ) min γ∈ ∏ ( PD1 , PD2 ) 〈C , γ〉F , ( 1 ) where ∏ ( PD1 , PD2 ) denotes the space of joint distributions with marginal PD1 and PD1 . 〈· , ·〉F is the Frobenius product , and the entry Cij of C represents the displacement cost of the ith and jth samples . It can be proven that the minimum of this problem is a distance and called the Wasserstein distance when the corresponding cost is a norm [ 25 , 26 ] . Our proposed method consists of two parts : OT alignment and Wasserstein Generative Adversarial Network ( WGAN ) training [ 39 , 40 ] . Both steps are based on OT ; however , they are considered with two different pairs of distributions and employ different algorithms . Adaptation by Joint Distribution Optimal Transport Our adaptation mechanism relies on the alignment between the joint distributions of source and target domains ( for the target domain , the label is estimated label ) . In particular , we approximate f by minimizing the OT loss between the joint distributions PXs × PYs and PXt × Pf ( Xt ) , with a chosen cost matrix , Cij = α ‖xsi − xtj‖2 + β ‖ysi − f ( xtj ) ‖2 , ( α , β > 0 ) ( 2 ) By aligning the joint distributions of the source and target domains , noise adaptation is naturally achieved as OT seeks the source sample that is the most “ similar ” for each target sample . Although the OT provides accurate estimates for each sample , the effect of each estimation error could accumulate in the training process and mislead f to a convenient local minimum without preserving the speech data structure . To avoid this situation , we employ Wasserstein Generative Adversarial Network ( WGAN ) training to complement and enhance our adaptation system . Discriminative training on outputs and source labels Distinct from the adaptation where we consider the joint distributions of the inputs and labels , we focus on WGAN training for the source label distribution PYs and output distribution Pf ( Xt ) . In the terminology of generative adversarial training , we consider f a generator and introduce a convolutional neural network-based discriminator , h , as the ’ critic ’ . In general , we use the discriminator to decide whether the outputs of f are ’ similar ’ to the source labels . Formally , the WGAN algorithm solves min f max h∈L { Ey∼PYs ( h ( y ) ) − Ex∼PXt ( h ( f ( x ) ) ) } ( 3 ) by Kantorovich-Rubinstein duality [ 25 ] , where L is the set of 1-Lipschitz functions . In this case , under an optimal discriminator minimizing the value function with respect to the generator parameters minimizes the Wasserstein distance between the distributions PYs and Pf ( Xt ) . This discriminative training complements our alignment and provides additional constraints from the explicit relation between the source labels and estimations of target labels . These constraints support the joint distribution alignment and provides further guidance in the gradient descent training process . The performance of the experiments is considerably improved when our discriminative training supports the joint distribution alignment .

Summary: The authors propose an approach for speech enhancement based on optimizing the empirical OT loss between the joint distribution of noisy/clean training data and noisy/cleaned target data (c.f. (1), line 139). The cleaned speech outputs are additionally optimized under a standard WGAN formulation (c.f. line 129). Their approach consistently outperforms DAT[19] and MDAN[44] on VoiceBank-DEMAND and Helicopter-TIMIT mixtures, and performs roughly on-par or slightly worse on TIMIT-(Crowd-party, Cafeteria, Baby-cry) mixtures which contain speech, according to the standard PESQ and STOI metrics.