Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport
Authors: Hsin-Yi Lin
Huan-Hsin Tseng
Xugang Lu
Yu Tsao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain adaptation
Optimal Transport
adversarial training
speech enhancement

Abstract: This paper presents a novel discriminator-constrained optimal transport network (DOTN) that performs unsupervised domain adaptation for speech enhancement (SE), which is an essential regression task in speech processing. The DOTN aims to estimate clean references of noisy speech in a target domain, by exploiting the knowledge available from the source domain. The domain shift between training and testing data has been reported to be an obstacle to learning problems in diverse fields. Although rich literature exists on unsupervised domain adaptation for classification, the methods proposed, especially in regressions, remain scarce and often depend on additional information regarding the input data. The proposed DOTN approach tactically fuses the optimal transport (OT) theory from mathematical analysis with generative adversarial frameworks, to help evaluate continuous labels in the target domain. The experimental results on two SE tasks demonstrate that by extending the classical OT formulation, our proposed DOTN outperforms previous adversarial domain adaptation frameworks in a purely unsupervised manner.

Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Contrraiiee Optimal Transport
Hsin-Yi Lin
The Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder, CO, 80309, USA NOAA Physical Sciences Laboratory, Boulder, CO, 80305, USA hylin@colorado edu
Huan-Hsin Tseng
Research Center for Information Technology Innovation
Academia Sinica, Taiwan
atsseg@citi.siniie edu. tw
Xugang Lu
National Institute of Information and Communications Technology (NICT), Japan xugang lu
Yu Tsao
Research Center for Information Technology Innovation Academia Sinica, Taiwan
yu. tsaoocitiiii. sinica. edu.
Abstract
This paper presents a novel discriminator-constiin optimal transport network (DOTN) that performs unsupervised domain adaptation for speech enhancement (SE), which is essential regression task in speech processing. The DOTN aims to estimate clean references of noisy speech in a target domain, by exploiting the knowledge available from the source domain. The domain shift between training and testing data has been reported to an obstacle to learning problems in diverse fields. Although rich literature exists unsupervised domain adaptation for classi- fication, the methods proposed, especially in regressions, remain searce and often depend on additional information regarding the input data. The proposed DOTN approach tactically fuses the optimal transport (OT) theory from mathematical analysis with generative adversarial frameworks, help evaluate continuous labels in the target domain. The experimental results on two SE tasks demonstrate that by extending the classical OT formulation, our proposed DOTN outperforms previous ad adversarial domain adaptation frameworks in purely unsupervised manner.
Introduction
The goal of speech enhancement (SE) is convert low-quality speech signals with improved quality and intelligibility. SE serves as an important regression task in the has been widely used for pre-processor in speech-related applications, such speech coding (1. automatic speech recognition (ASR) (2). speaker recognition [3) and assistive hearing devices 54
field and
Recent advances machine learning have made significant progress to the SE technology. Generally,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

soaming-based SE approaches estimate transformation to characterize the mapping function from noisy clean speech signals in the training phase [6. The estimated transformation converts noisy speech signals to generate dlean-like signals in the testing phase. Various neural network models have been used to characterize noisy-to-clean transformations. Well-known examples of such models include fully connected neural network 77, deep denoising autoencoder [8], convolutional neural network [9]. long-short-temm memory [10], and Transformer [[]. To effectively handle diverse noisy conditions, we usually prepare a considerable amount of training data that cover various noise types to train SE models. However in real-application scenarios, the noise types in the testing data may not always be involved in the training set. Consequently, the noisy-to-clean transformation learned from the training data cannot be suitably applied to handle the testing noise, resulting in limited enhancement performance. This saining-testing mismatch is generally called domain mismatch issue for SF. An effective solution is required to perform domain adaptation to adjust the SE models with formulating a precise noisy-to-clean transformation that matches the testing conditions. Most existing domain adaptation methods rely on at least one of the following adaptation mechanisms: aligning Nomain-invariaal features [ 13 [4] and adversarial training, where a discriminator is
introduced during training as a domain classifier [15 16 17).
This study aims to solve the unsupervised domain adaptation problem for SE by introducing optimal transport (OT). In particular, we consider the circumstance where SE is tested on a target domain with completely unlabeled data, and only labeled data from the source domain is available for reference. Generally speaking. OT theory compares two (probabilityy) distributions and considers all possible transportation plans in between to find one with a minimal displacement cost. The concept of OT can be applied to minimize domain mismatch and consequently achieve unsupervised domain adaptation. Even with the mathematical characteristics offered by OT, obstacles to excellent SE performance persist due to the complex structure possessed by human speech. To further overcome the obstacles, another concept from Generative Adversarial Network (GAN) is integrated to assist attaining sophisticated SE domain adaptation. Although an existing domain transition technique "domain adversarial training" and our proposal share similarity in names, the fundamental constructions are substantially different. A key element in our method lies in a discriminator utilized examine speech output characteristics, instead of domain classifier. More precisely, the discriminator in our method was employed to govern the output speech quality by learning the probability distribution of the source labels. This novel approach was designed especially for the unsupervised SE domain adaptation to exhibit excellent performance, which was verified on the VoiceBank and TIMIT datasets.
Contributions We proposed a novel method designed specifically for unsupervised domain adapta- tion in regression setting. This area of study still has very limited results; moreover, the existing methods often require additional classification of source domains may not yet be supported by strong regression applications. Conversely, our approach does not require any additional input infor- mation other than the source samples, source labels, and target samples. Our approach was applied to two standardized SE tasks, namely VoiceBank- DEEANDDD and TIMIT, and achieved superior adapta- tion performance in terms of both Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) scores. Furthermore, owing to the simple input requirements, we can easily investigate the effect of target sample complexity on our method by increasing the number of noise types allowed in the target domain, which has not been reported by previous literature to the best of our knowledge.
2 Related work
Adversarial domain adaptation: The main objective of Domain Adversarial Training (DAT) is to train a deep model (from the source domain) capable of adapting to other similar domains by leveraging considerable amount of unlabeled data from the target domain [15] T8]. A conventional DAT system consists of three parts, deep feature extractor, label predictor, and domain classifier. By using a gradient reversal layer, the extracted deep features are discriminative for the main learning task and invariant with shifts between the source and target domains. The DAT approach has been applied and confirmed to effectively compensate for the mismatch of source (training time) and target (testing time) conditions in numerous tasks, such as speech signal processing [19 20 image processing [5 21), and wearable sensor signal processing [2]. A later development in Multisource Domain Adversarial Networks (MDAN) [23 extended the original DAT to lift the constraint of single

---

domain transition, utilizing multiple domain classifiers to extract discriminative deep features for the main learning task while being invariant to multiple domain shifts 24
Optimal transport for domain adaptation: Hitherto, OT 22 26 has been utilized domain adaptation [27 28 with related analytical results [29]. Furthermore, the concepts of OT have proved itself even more useful under a joint distribution framework [3 311. More recently, to improve the sensitivity of OT to outliers, Robust OT was proposed [32]. Moreover, a method for combining the notion of adversarial domain adaptation with OT and margin separation has been proposed [33]. However, almost all experiments performed in these studies were classification problems, unlike the SE task that is the focus of our study.
Domain adaptation in speech enhancement: Existing domain adaptation in SE approaches can be divided into two categories: supervised and unsupervised. For supervised domain adaptation, paired noisy and clean speech signals for the testing conditions are available adjust the parameters in the SE models. In
34 351, transfer-lcarning-ue basedd approaches have been proposed to adapt SE models to alleviate corpus mismatches. To combat the catastrophic forgetting issue, Lee et al., proposed a SERIL algorithm that combines pervtture-based regularization and path optimization augmenting strategies when preforming domain adaptation on SE models [6]. Conversely, for unsupervised domain adaptation, only noisy speech signals are provided, and the corresponding clean counterparts are not accessible. Generally, unsupervised domain adaptation has good applicability to real-world scenarios. In [77, unsupervised domain adaptation for SE was performed by minimizing the Kullback-L.enh
divergence between posterior probabilities produced by teacher and student senone classifiers without paired noisy-clean adaptation data. In [19 38), the DAT approach was used to adapt SE models to new noisy conditions.
Despite yielding promising performance, the existing unsupervised domain adaptation approaches require additional information, such as word labels, language models, and noise-type labels. In this paper, we propose a new approach: discriminator-consttaine OT network (DOTN) to perform unsupervised domain adaptation on SE. In contrast related works, DOTN does not require additional label information when adapting the original SE models to match new noisy conditions. Our experiments show that DOTN can effectively adapt the SE models to new testing conditions, and that it achieves better adaptation performance than previous adversarial domain adaptation methods, which require additional noise type information.
3 Method
3.1 Problem setting and notation
Consider a source domain with paired data (X Y = [(x], y where x R", y E Rm stand for the input and the corresponding label of sample The unsupervised domain adaptation assumes that another target domain exists containing only data unlabeled, X = fxf E ) The goal is to seek ground truth estimator (or statistical hypothesis) f R m for the target labels Y ( (exist but not known), based on the knowledge provided by the source domain.
[==]
The probability distribution of a dataset in our discussion. Our problem is to find function f such that f induces a probability distribution F Y with P f(x)) + Py: under certain measure. We propose using the concept of OT to solve this problem.
denoted by Pp, where D is either X Y X' or Y
3.2 Proposed model: Discriminator-Consttaii Optimal Transport Network (DOTN)
Given pair of distributions PD, and Pp, and displacement cost matrix C 2 0, OT solves for the transportation plan 7 (IP,, PD,) that minimizes the total cost (in a discrete setting)
min (T
F)
Pr,))
where [PP,, PD.) denotes the space of joint distributions with marginal PD, and Pp,, (:. )) is the Frobenius product, and the entry Cg of C represents the displacement cost of the 7th and jth
3

---

samples. It can be proven that the minimum of this problem is a distance and called the Wasserstein distance when the corresponding cost norm [25 26
Our proposed method consists of two parts: OT alignment and Wasserstein Generative Adversarial Network (WGAN) training [39 40]. Both steps based OT; however, they considered with two different pairs of distributions and employ different algorithms.
Adaptation by Joint Distribution Optimal Transport Our adaptation mechanism relies on the alignment between the joint distributions of source and target domains (for the target domain, the label is estimated label). In particular, we approximate by minimizing the OT loss between the joint distributions Px Py and Ip x P((x)), with chosen cost matrix,
Cyy a || - x5I1 + B 1ly? -
(a, 0)
(2)
By aligning the joint distributions of the source and target domains, noise adaptation is naturally achieved as OT seeks the source sample that is the most "similar" for each target sample.
Although the OT provides accurate estimates for each sample, the effect of each estimation error could accumulate in the training process and mislead f to a convenient local minimum without preserving the speech data structure. To avoid this situation, we employ Wasserstein Generative Adversarial Network (WGAN) training to complement and enhance our adaptation system.
Discriminative training outputs and source labels Distinct from the adaptation where consider the joint distributions of the inputs and labels, we focus on WGAN training for the source label distribution Py and output distribution Py(x))- In the terminology of generative adversarial training, we consider f' generator and introduce a convolutional neural network-based discriminator, h, as the 'critic', In general, we the discriminator to decide whether the outputs of f are 'similar' to the source labels. Formally, the WGAN algorithm solves
min max REC
Eyy-y- (I(2)) E. Ez- (h((())))
(3)
by Sesooovvcch-Rbinsteeee duality (25], where the set of I-Lipschitz functions. In this case, under an optimal discriminator minimizing the value function with respect the generator parameters minimizes the Wasserstein distance between the distributions Py, and PP (()))
This discriminative training complements our alignment and provides additional constraints from the explicit relation between the source labels and estimations of target labels. These constraints support the joint distribution alignment and provides further guidance in the gradient descent training process. The performance of the experiments is considerably improved when our discriminative training supports the joint distribution alignment.
3.3 Loss functions and the proposed algorithm
Our domain alignment can be achieved by solving the following optimization problem:
Σ Iyy f(x+)12 + Tie (alxx ||] - -x,11 3 Ily f(x))1)), (4)
min Li + L2 min 7.
7.8 No
where a, 3
are the parameters chosen for balance. Notably, the first term emphasizes the knowledge from the source domain is not to be forgotten during training, which has been revealed in several works [31 41 42). Without this emphasis, the source domain knowledge cannot be well- maintained, and thus the overall performance may degrade. Such phenomenon the SE experiments. The second term intended for domain alignment. To show some intuitions, consider the ideal case where Eq.
also observed in
(4) is completely minimized zero, which leads to
| 2 0 and ||y f(x))11 =0
x = x² and y f(x))
for all i.j. This indicates that for each given target domain sample x/. an identical sample x from the source domain is found, and the unknown target label is then constructed by the corresponding source label. Even though practically the ideal case of zero loss unlikely to happen, the OT loss aims to search for the most "similar" correspondence, which entails the intuition of domain alignment

---

outpat f(rt)
input (x", y"),
discriminator
target estimator f
Figure I: DOTN network structure.
From this point of view, it is noted that although the term ||* -xl1 in Cit (in Eq. 2) not directly related to the network backpropagations of f and h, may not be ignored as the discard of this term will result in wrong transportation plan Yij and eventually lead to undesired alignment.
for Eq. 4
For discriminative training. the discriminator h is trained by the discriminator loss function LA. m h where is the batch size. As there are multiple sets of parameters 7, h and f in our framework, one set of parameters is updated each time, while the other sets of parameters are fixed.
1 ΣE h(y?) (xx)), and f follows the generator loss function Ly -
Algorithm DOTN, proposed algorithm
Require: x", source domain inputs. y", source domain labels. x', target domain inputs. c, the clipping parameter. m, the batch size. nf, nh, n4. number of iterations of OT per generator training, discriminator training, and source domain training, respectively. number of iterations.
Require: 0,, initial parameters of estimator f. 0%, initial parameters of discriminator h.
1: for each batch of source samples (x".y") and target samples (y") do
2: fix fy, solve for y in Eq. 44 by OT.
3: fix y, By A V,, L2. 0%
4: if mod ni, = then
5:
O, < Adam Vo, C1, ff 0n) end if
6:
7:
if mod ny = 0 then
8:
0, Adamm C,, O,, 0n).
9:
end if
10:
if mod ni = 0 then
11:
An Adamm Vo, Ln, f 0n) 9% c (6.. -c,
12:
13:
end if
14: end for
4 Experiments
We evaluated method in SE on two datasets: Voice Bank corpus 43 and TIMIT 44 Although there is abundant literature on unsupervised domain adaptation, very limited methods have been successfully applied to SE or regression problems. Closely relevant methods often require additional input structures or domain label. Unlike previous methods, our approach does not rely on additional compared results with most relevant adversarial domain
data information. Nevertheless, adaptation methods: DAT [19 and MDAN (3). The original MDAN was designed for classification problems and cannot be directly applied SE. Preserving the fundamental ideas, modified the MDAN structure for regressions, so that SE experiments can be performed for comparison. The implementations summarized in the supplementary material and codes available Githut
Transport-Netwoon

---

4.1 Comparisons to DAT and MDAN
As both DAT and MDAN utilize a domain classifier to derive domain invariant features, domain labels become essential in their adaptation mechanism. Thus, both these methods are considered weakly supervised. Specifically, they request inputs of the form {x]+ yf, Ca and {x). o with
C; Cj E K., where KC is an index set specifying the origin of data domains. For DAT, K. {0, 13 in- dicate whether sample belongs to the source or target domain, while for MDAN K. = {0, 1,..,,,, K} to indicate that there are K distinct source domains and one target domain. The requirement on domain labels poses restrictions in certain circumstances, such as a new target sample may not always fall into any existing categories, or the data origin is simply unknown. In contrast, the proposed method DOTN is not bounded by specifications of data origin and simply receives inputs of the form (x yi. x a. With less input requirement, DOTN is more flexible to be applied in various scenarios, more approachable to real-world applications.
4.2 Voice Bank with DEMAND noise database
Dataset In the first set of experiments, the pre-selected subset of Voice Bank provided by 45 was used to test the proposed DOTN. For source domain data, 14 male speakers and 14 female speakers were randomly selected out of totally 84 speakers (42 male and 42 female). and each speaker pronounced around 400 sentences. As result, the clean data in source domain contained 5,724 male utterances and 5,848 female utterances, amounting to 1 1,,72 utterances in total. We then mixed the 11 11,572 clean utterances with noise from DEMAND [46] in transportation category: "Bus", "Car", "Metro" at 7 Signal-to-noise ratio (SNR) levels (-9, -6, -3, 0, 3, 6, and 9 dB). Accordingly, for this set of source domain data, both noisy speech signals and the corresponding clean references are prepared.
A target domain data contained 5, 768 noisy utterances mixed by 824 clean ones from two random speakers (1 male, female) (followed the design in [47]) with one of the three noise types from DEMAND, in STREET category: "Traffic", "Cafe", or "Public square" and 7 SNR ratios (-9, -6, -3, 0, 3, 6, and 9 dB). No clean labels were given under target domain. That is, for the target domain data, only noisy speech signals are provided and the corresponding clean references are not accessible.
All source samples (both noisy utterances and the corresponding clean references, prepared from the three transportation noise types) and the target samples (only noisy utterances without the corresponding clean references, prepared from the one street noise type) were included in our training set. We conducted the experiments under single-target noise-type circumstances and compared the results with DAT and MDAN. More specifically. we ran this setting for all three cases, where the target domain contained either cafe, public square, or traffic noise.
Results Table I and Table 2 list the PESQ and STOI scores, respectively, of DAT, MDAN and DOTN under three noise types at 7 SNR levels. "Avg" denotes the averaged scores over SNR levels. From the PESQ scores reported in Table we note that the proposed DOTN outperforms both DAT and MDAN consistently over different noise types and SNR levels, except for the Cafe noise type at dB SNR. This might be owning to a potential limitation of DOTN, which will be detailed in the next section. Next, from Table 2 we note that STOI scores show very similar trends to that of PESQ scores, as listed Table
"DAT consists of three components, a deep feature extractor E : - 2, (task) label predictor Fy
2 + )), and a domain classifier Fp 2 + K., to form two functional pairs: FD o E and Fy o E. Here
X, y the input space and (lask) label space respectively. Z denotes the invariant (latent) feature space; [O source, 1 target) as the domain label classes. The pair FD o : X' + K. formed by Fp. works against each other, by a Gradient Reversal Layer, to derive domain invariant features in Z. Another pair =X++3 Fy o E : + y' demands that the invariant features in Z encode sufficient information for (main task) label classifications at the same time. Via adversarial training, the two pairs eventually reach balance completing the main learning task as well as eliminating the domain mismatch.
= K=
"MDAN inherits the DAT architecture with an extension to K source domains and K domain classifiers FD, 2 (0, 1). L...K with similar adversarial training applied.

---

Table 1: PESQ scores for DEEMAND
noise type
Traffic
Cafe
Public square
SNRodB)/model
DAT 19
MDAN 23
DOTN
DAT
MDAN
DOTN
DAT
MDAN
DOTN
-9
1.307
1.670
1,863
1.058
1.539
1.497
1.436
1.929
2.037
-6
1.446
1.920
2.182
1.184
1.735
1.853
1.655
2.119
2.258
-3
1.718
2.153
2.395
1.362
1.949
2.089
1.939
2.318
2.439
0
2.081
2.366
2.591
1.599
2.153
2.333
2.268
2.512
2.601
3
2.381
2.535
2.740
1.865
2.345
2,490)
2.575
2.670
2.761
6
2.712
2.708
2,888
2.189
2.534
2.661
2.867
2.824
2.889
9
3.016
2.854
3.015
2.493
2.695
2.783
3.120
2.966
3,057
Avg
2.094
2.315
2.525
1.679
2.136
2.244
2.266
2.477
2.577
Table 2: STOI scores for VoiceBank-DEMAND
noise type
Traffic
Cafe
Public square
SNRodB)/model
DAT 19
MDAN
DOTN
MDAN
DOTN
DAT
MDAN
DOTN
-9
0,584
0.708
0.721
0,557
0.643
0.633
0.667
0.747
0.765
-6
0.659
0.761
0.790
0.616
0.702
0.723
0.725
0.792
0.815
-3
0.728
0.810
0.833
0.687
0.760
0.779
0.771
0.830
0.850
0
0.785
0.849
0.871
0.741
0.805
0.831
0.815
0.859
0.877
3
0.824
0.874
0.894
0.785
0.842
0.861
0.845
0.881
0.900
6
0.860
0.896
0.914
0.825
0.871
0.887
0.872
0.898
0.915
9
0.885
0.910
0.927
0.854
0.893
0.905
0.893
0.914
0.931
Avg
0.761
0.830
0.850
0.724
0.788
0.803
0.798
0.846
0.865
4.3 TIMIT
Dataset For the second part of experiments, TIMIT was used to prepare source and target sam- N. ples. The clean speech of the source domain { for training consists of utterances, contributed by 48 male speakers and 24 female speakers from 8 dialect regions. Each speaker had 8 sentences, including 5 SX (phonetically compact sentences) and 3 SI (phonetically diverse sentences), according to the official suggestion of TIMIT. The number of the speakers selected was to maintain the balance of the original data.
The above clean utterances were used to mix with stationary noise types (Car, Engine, Pink, Wind, and Cabin) utterances, be the noisy speech the source domain {xf NN with N, 25, 920.
SNR levels (-12, -9, -6, -3, 0. 6, 9, and 12 dB), amounting to 25,920 noisy
For the target domain, a total of 24 speakers suggested by TIMIT core test set was all used to have 192 clean utterances for the target domain y/. which subsequently mingled with one of the four non-stationary noise types: "Helicopter", "Cafeteria", "Baby-cry", or Crowd-party: under SNRs (-9, -6, -3, 0, 3, 6, and 9 dB) as target inputs {x]] with N₂ = 1, 144. The choice of noise types for the source and target domain was let the learning algorithms adapt from distinguished environments in the real-world.
Results Table lists the PESQ and STOI scores of the DAT, MDAN, and DOTN under four noise types at seven SNR levels. From the table, we first note that DOTN consistently outperforms DAT and MDAN in terms of both "Avg" PESQ and STOI scores among the four noise types. With more careful investigation, the DOTN achieves higher PESQ and STOI scores over DAT and MDAN for all SNR conditions in the Helicopter noise. For the Crowd-party and Cafeteria noises, DOTN outperforms DAT and MDAN in most higher SNR conditions for both PESQ and STOI scores. However, for the Baby-cry noise, DOTN outperforms DAT and MDAN STOI but underperforms MDAN in PESQ. Note that Cafeteria, Crowd- and Baby-cry noise types involved clear human speech components, which may cause confusions when DOTN tries to retrieve the target speech (clean reference). Thus, DOTN yields sub-optimal performance when dealing with these noise types, especially under very low SNR conditions, where background speech components might overwheim the target speech. Nevertheless, the overall average PESQ and STOI scores of DOTN (P=1.838; Q=0.7238) are still higher than that DAT (P=1.5728; Q=0.6298) and MDAN (P=1.8001: Q=0.7038), where P and Q denote the PESQ and STOI scores, respectively, over the noise types and

---

Table 3: TIMIT results
noise type
Helicopter
Crowd-party
model
DAT
19
MDAN
23
DOTN
DAT
MDAN
DOTN
SNR(dB)
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STOI
-9
1.031
0.392
1.252
0.517
1,455
0,577
1.483
0.544
1.150
0,440
1.056
0.451
-6
1.015
0.431
1,443
0.594
1.669
0.649
1.484
0.560
1.356
0.516
1.302
0.538
-3
1.094
0.497
1.664
0.670
1.890
0.716
1.528
0.592
1,560
0.600
1.559
0.621
1.268
0.566
1,902
0.742
2.104
0.775
1.596
0.636
1.776
0.684
1.816
0.709
1,518
0.637
2.134
0.801
2.289
0.822
1.736
0.690
1.986
0.762
2.042
0.782
1.779
0.701
2.363
0.849
2.497
0.865
1.953
0.750
2.179
0.823
2.236
0.838
2.094
0.759
2.563
0.884
2.677
0.895
2.200
0.801
2.355
0.865
2.447
0.885
Avg
1.400
0.569
1,903
0.722
2.083
0.757
1.711
0.653
1.766
0.670
1.780
0.690
noise type
Cafeteria
Baby-cry
model
DAT
MDAN
DOTN
DAT
MDAN
DOTN
SNR(dB)
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STO1
PESQ
STOI
PFSQ
STOI
-9
1.196
0.440
1.248
0.471
1.185
0.436
1.109
0.580
1.116
0.561
0.984
0.597
1.206
0.471
1.395
0.535
1.419
0.528
1.313
0.630
1.313
0.636
1.180
0.668
1.244
0.519
1.608
0.614
1.631
0.620
1.495
0.665
1.500
0.699
1.431
0.732
1.408
0.579
1.827
0.692
1.859
0.699
1.621
0.707
1.734
0.768
1.665
0.789
1.631
0.651
2.031
0.764
2.075
0.769
1.801
0.746
1.909
0.809
1.889
0.835
1.915
0.719
2.244
0.822
2.271
0.826
1.973
0.782
2.112
0.849
2.105
0.871
2.216
0.782
2.422
0.864
2.458
0,873
2.133
0.814
2.274
0,880
2.277
0.890
1.545
0.594
1.825
0.680
1.833
0.679
1.635
0.703
1.708
0.743
1.647
0.769
SNR levels. Please also note that DAT and MDAN require additional domain label information while DOTN performs domain adaptation in a purely unsupervised manner.
To show the advantages and flexibility of DOTN, we further explored and applied DOTN to circumstances when multiple noise types are included in the target domain. Fig 2 and Fig 3 respectively, present the PESQ and STOI results over four SNR levels on multiple target noise types; here H, Ca, Cr. and B denote Helicopter, Cafeteria, Crowd-party. and Baby-cry noise types, respectively. In Fig 2 when the testing noise is "Helicopter", the PESQ scores of using two noise comparable.
types (H+Ca, H+Cr, H+B) and three noise types (H+Cr+B, H+Cr+Ca, H+B+Ca) Further, the PESQ scores of these six systems are also comparable to that of using one noise type (H). The results confirmed that the DOTN has minor effects catastrophic forgetting, which is common issue in domain adaptation approaches [48] 49] In the case of 4 noise types (the system sequentially learned Helicopter, Baby-ery, Cafeteria, and Crowd-party noise types and tested on the Helicopter noise), the achieved performance drops moderately. We observe very similar trends for the other three noise types (baby-ery, cafeteria, and crowd-party) in Fig
From Fig 3 similar trends as those from Fig are observed: (1) The seven results using 1, 2. and 3 noise types are comparable, thereby confirming that the DOTN has minor effects catastrophic forgetting. (2) When there are noise types for sequential learning, the achievable STOI scores start to decrease.
5 Conclusion
In this study, we proposed novel DOTN method, which was designed specifically for unsupervised domain adaptation in regression setting. Our approach skillfully fuses OT and generative adversarial frameworks to achieve unsupervised learning in target domain based on the information provided from the source domain, requiring no additional structure or inputs, such as multi-source domain and noise type labels. Our experiments show that the proposed method is capable of superior adaptation performance in SE by outperforming other adversarial domain adaptation methods on both PESQ and STOI scores for the VoiceBank-DEMAA moderately increasing the complexity of the target samples (by increasing the number of noise types in the target domain), only a small degree of degradation was observed. This suggests that our method robust to sample complexity in the target domain.
and TIMIT datasets. Further, we show that when

---

PESQ scores: helicopter
PESQ crowd-party
24
74.
24
2-2
PE5O
10
H+Ca
HeCr
H+g
HeCred
HecraCa
HeB
HeB-CCaCr
CryCa
GMH
Cre
Ca
CreBww
CrsCaeH
crecasti
nois by aes target domain
noise types in target domain
PESQ scores: cafeteria
PESQ scores: baby-cry
9:55
2.6
41
dh
3:00
3
P550
14-
1.5
14-
14
12-
10
CarCr
Caeff
Ce1g
carcreH
Casc +0
ca+ 8
CastteBCCr
1.0
B+Ca
B+Cr
B4H
BeffsCr
B-Ca+ BeCo+H
B+Co+HH
noise types target domain
noise types in target obmain
Figure 2: Comparisons in PESQ average for cases with multiple noise types in target domain, where H. Ca, Cr, and B denote Helicopter, Cafeteria, Crowd-party, and Baby-cry noise types.
STOI scores: helicopter
STOI scores: crowd party
28
1:35
2:28
-145
3.49
2.-
5T0
STCI
0.7
0.
H*CK
H+Cr
H+B
HeCre
HcCreCa
W+B+Ca
H+B-CaaeCr
Gr+Ca
Gr+Hl
Crog
G+BxCa
Cre +f
C Creaae
GreCaetted
naise types target domain
noise types target domain
STOI scores: cafeteria
STOI scores: baby-cry
CD
2.
2:5
05-
2.00
STC everage
5.
0.7
86-
CarCr
Ca+11
Crvg
CasCre
CarCh+
C
Ca-tteascr
B+ca
Blcr
BeH
BeHeer
BeCakCr
BeCa+H
BsCasHtG
neise types domsin
target domain
Figure 3: Comparisons in STOI average for cases with multiple noise types target domain.

---

References
[1] Junfeng Li. Lin Yang, Jianping Zhang, Yonghong Yan, Yi Hu, Masato Akagi, and Phili- pos C Loizou. Comparative intelligibilitt investigation of single-channel noise-reduction algorithms for chinese, japanese, and english. The Journal of the Acoustical Society of America, 129(5):3291-332) -3001, 2011.
[2] Jinyu Li, Li Deng, Reinhold tion: a bridge to practical applications. 2015.
and Yifan Gong. Robust automatic speech recogni-
[3] Daniel Michelsanti and Zheng- Hua Tan. Conditional generative adversarial networks for speech enhancement and noise-robust speaker verification. arXiv preprint arXiv:1709.01100: 2017.
[4] DeLiang Wang. Deep learning reinvents the hearing aid. IEEE spectrum, 54(3):32 2017.
[5] Ying-Hui Lai, Fei Chen, Syu-Siang Wang. Xugang Lu, Yu Tsao, and Chin-Hui Lee. A deep denoising autoencoder approach to improving the intelligibility of vocoded speech in cochlear implant simulation. IEEE Transactions Biomedical Engineering.
(777))1568-5577 2016.
[6] DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10): 1702- 1726, 2018.
[7] Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee. A regression approach to speech enhance- ment based on deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(1);1119 2014.
[8] Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based on deep denoising autoencoder. In Interspeech, volume 2013, pages 436-440, 2013.
(9] Szu-Wei Fu, Yu Tsao, and Xugang Lu. Sor-aware convolutional neural network modeling for speech enhancement. In Interspeech, pages 3768-3772, 2016.
[10] Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John R Hershey. Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
[11] Yuma Koizumi, Kohei Yaiabe, Marc Delcroix, Yoshiki Maxuxama, and Daiki Takeuchi. Speech enhancement using self-adaptation and multi-head self-attention. In ICASSP 2020-2020 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 181-185. IEEE, 2020.
[12] Eric Tzeng, Judy Hoffman, Ning Zhang. Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474. 2014.
[13] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
[14] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv: 2017.
[15] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1 1180-1189, PMLR, 2015.
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. networks. The journal of machine learning research, 17(1):2096-200 2030, 2016.
training of neural
[17) Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-Adversarr Training of Neural Networks. 09 2017.
10

---

[19] Chien-Feng Liao, Yu Tsao, Hung- Yi Lee, and Hsin-Min Wang. Noise adaptive speech enhance- ment using domain adversarial training. arXiv preprint vrXiv:187.0050011 2018.
[20] Dmitriy Serdyuk, Kartik Audhkhasi, Philémon Brakel, Bhuvana Ramabhadran, Samuel Thomas, and Yoshua Bengio. Invariant representations for noisy speech recognition. arXiv preprint 16Xiv: 128, 2016.
[21] Zhenjie Tang, Bin Pan, Enhai Liu, Xia Xu, Tianyang Shi, and Zhenwei Shi. Srda-net: Super- resolution domain adaptation networks for semantic segmentation. arXiv e-prints, pages arXiv-2005, 2020.
[22] Kai-Chun Liu, Michael Chan, Yeh Hsieh, Hsiang-Yun Huang, Chia-Tai Chan, and Yu Tsao. Domain-adaptive fall detection using deep adversarial training. arXiv preprint arXiv:2012.10911, 2020.
[23] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31:8559-85700 2018.
[24] Guillaume Richard, Antoine de Mathelin, Georges Hébrail, Mathilde Mougeot, and Nicolas
Vayatis. Unsupervised multi-source domain adaptation for regression. 2020.
[25] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science Business Media, 2008.
[26] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 1((-6))335-6677 2019.
[27] Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 274-289. Springer, 2014.
[28] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamoniy. Optimal transport for domain adaptation. IEEE transactions pattern analysis and machine intelligence, 39(9): 1853- 1865, 2016.
[29] levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737-753. Springer, 2017.
[30] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. arXiv preprini arXiv: 1exivv/705: 0et.//0055.888. 2017.
[31] Bharath Bhushan Damodaran, Benjamin Kellenherger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 447-463, 2018.
[32] Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. arXiv preprint arXiv:2010.05822 2020.
[33] Sofien Dhouib, levgen Redko, and Carole Lartizien. Margin- awwrre adversarial domain adapta- tion with optimal transport. In International Conference on Machine Learning, pages 2514 2524. PMLR, 2020.
[34] Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee. Cross-language transfer learning for deep neural network based speech enhancement. In The 9th International Symposium on Chinese Spoken Language Processing, pages 336-340 340. IEEE, 2014.
[35] Dong Wang and Thomas Fang Zheng. Transfer learning for speech and language processing. In 2015 Asia-Pacific Signal and Information Processing Association Anmual Summit and Conference (APSIPA), pages 1225-1237. IEEE, 2015.
[36] Chi-Chang Lee, Yu-Chen Lin, Hssun-TTie Lin, Hsin-Min Wang, and Yu Tsao. Seril: Noise adaptive speech enhancement using regularization-ban incremental learning. arXiv preprint arXiv:2005.11760 2020.

---

[37] Sicheng Wang. Wei Li, Sabato Marco Siniscalchi, and Chin-Hui Lee. A cross-task transfer learning approach to adapting deep speech enhancement models unseen background noise using paired senone classifiers. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6219-6223. IEEE, 2020.
[38] Nana Hou, Chenglin Xu, Eng Siong Chng, and Haizhou Li. Domain adversarial training for speech enhancement. In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 667-672. IEEE, 2019.
[39] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In International conference machine learning, pages 214-223. PMLR, 2017.
[40] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. arXiv preprint arXiv:1704.0002. 2017.
[41] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(11):2935- 2947, 2017.
[42] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE International Conference on Computer Vision, pages 3400-3409, 2017.
[43] Christophe Veaux, Junichi Yamagishi, and Simon King. The Voice Bank Corpus: Design, collection and data analysis of large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (--COCOSDA/CASLEE) pages 1-4. IEEE, 2013.
[44] John Garofolo, Lori Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Getting started with the DARPA TIMIT CD-ROM: An acoustic phonetic continuous speech database. National Institute of Standards and Technology (NIST), Gaithersburgh, MD, 107:16, 1988.
[45] Cassia Valentini-Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating RNN-based speech enhancement methods for noise-robust text-to-speech. In SSW, pages 146-155 2016.
[46] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. DEMAND: a collection of multi- channel recordings of acoustic noise in diverse environments, June 2013. Supported by Inria under the Associate Team Program VERSAMUS.
[47] Santiago Pascual, Antonio Bonafonte, and Joan Serra. Segan: Speech enhancement generative adversarial network. arXiv preprint 1703009703 2017.
[48] Min Chee Choy, Dipti Srinivasan, and Ruey Long Cheu. Neural networks for continuous online learning and control. IEEE Transactions on Neural Networks, 1116):151 2006.
[49] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-basee bassed neural networks. arXiv preprint 131.v.1 2013.
12


Reviews:
[{"review_id": "oKMeAEzMAJs", "reviewer": "Reviewer_uPz2", "summary": "The authors propose an approach for speech enhancement based on optimizing the empirical OT loss between the joint distribution of noisy/clean training data and noisy/cleaned target data (c.f. (1), line 139). The cleaned speech outputs are additionally optimized under a standard WGAN formulation (c.f. line 129). Their approach consistently outperforms DAT[19] and MDAN[44] on VoiceBank-DEMAND and Helicopter-TIMIT mixtures, and performs roughly on-par or slightly worse on TIMIT-(Crowd-party, Cafeteria, Baby-cry) mixtures which contain speech, according to the standard PESQ and STOI metrics.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- OT is a very natural formulation to domain adaptation, and has been adopted by others.\n- Gains on non-speech noise over baselines.\n- Code made available, so should be reproducible.\n\nLimitations:\n\n- Lower novelty, standard training metrics for domain adaptation.\n- The baselines are not SOTA speech enhancement techniques, and the results are on non-standard datasets, reducing the significance of the results.\n- Standard metrics are reported but no audio demonstrations or human evaluations (MOS) are provided, these are important ways to assess true performance.\n- The approach does not improve mixtures containing other speech, which is another signficant limitation.\n\nCurrent Assessment:\n\n- Falls short wrt ML novelty for Neurips and significance to the ML and speech enhancement communities.\n\n---- \n\nPost-rebuttal (updated)\n\nThank you to the reviewers for their detailed responses. \n- The audio demonstrations reveal that all of the methods compared are performing quite poorly, far below SOTA. \n- (update 1) The follow-up MetricGAN+ (SOTA) results provided show that the approach achieves slightly better STOI scores, but significantly worse PESQ scores. However, the fact that the MetricGAN+ is optimized on PESQ makes this comparison more difficult to interpret. \n- (update 2) Follow-up MetricGAN+ and Transformer audio demos are subjectively significantly better than the proposed method. MOS or A/B preference listening tests vs. MetricGAN+ and/or the SE Transformer should be included in the results. The current audio demos suggest that the method would be outperformed by MetricGAN+ and the SE Transformer in terms of MOS or A/B assessment, but only 2 audio examples were provided.\n- The manuscript needs substantial revision. The paper in general, and particularly the experimental section is not detailed enough to establish the contributions the paper makes, and does not describe the approach, particularly in mapping abstract losses and inputs/outputs to a concrete system and associated details.\n- As a type of kernel-based method for adaptation/reconstruction, the training sets are likely too small, and the details regarding reconstruction (chunking etc.) likely not fully optimized, perhaps explaining the results.\n- (update 3) Based on the additional results provided, I have increased my score, but remain concerned that the manuscript and results are not complete enough to justify an acceptance recommendation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "cUAzUuYoMO0", "reviewer": "Reviewer_Km9T", "summary": "This paper presented a novel method to achieve unsupervised domain adaptation for speech enhancement. There is little work in unsupervised domain adaptation for regression tasks and the paper combined techniques from optimal transport theory and generative adversarial networks, accomplishing better results on two datasets, VoiceBank-DEMAND and TIMIT, while requiring less supervision than prior work.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although there exists many works on unsupervised domain adaptation, the authors proposed a novel method to tackle the speech enhancement task, which resembles regression rather than classification, an area that had seen much less success previously. Compared to prior work on unsupervised domain adaptation for speech enhancement, the proposed discriminator-constrained optimal transport network (DOTN) does not require any additional labels. The paper uses a novel combination of known techniques to solve an important problem. The related work and contributions were stated very clearly.\n\nQuality: The authors demonstrated the strength of their novel approach well using two standard datasets (VoiceBank-DEMAND and TIMIT) and two metrics (PESQ and STOI), while comparing against two other prior work on unsupervised domain adaptation for speech enhancement. How the datasets were cleaned and prepared was described in detail and results and comparisons were analyzed in depth. The authors also tested and stated the limitations well by performing the multiple noise source experiment, testing the effects of catastrophic forgetting. \n\nClarity: The submission was very well written and clearly organized. One minor suggestion is to flip the row orders of noise type and methods in the tables to make comparisons clearer. \n\nSignificance: The results are quite significant as speech enhancement could benefit many downstream tasks and the paper was able to achieve better performance than prior works on unsupervised domain adaptation while not needing additional labels. Having a truly unsupervised setting is quite important for real world use since knowing the type of noise is an unrealistic assumption in many cases. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed limitations by discussing performance drops when too many sources of noises are present simultaneously. The paper do not have any immediate negative impacts. The only concern is that by improving the performance of various speech tasks, there can be more privacy violation or surveillance. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "JG2KAkJ2bW2", "reviewer": "Reviewer_8JUP", "summary": "This paper proposes using a discriminator-constrained optimal transport method for unsupervised speech enhancement (SE). The method is specifically designed for a regression task. It involves 2 parts. The first parts estimates a generator function as an OT problem to align the source and target labels. The source-target alignments ($\\gamma$) and the generator function parameters (${\\theta}_f$) are estimated in an interleaving fashion. The second part incorporates a discriminative training component on the label distributions using WGAN. Two discriminative loss functions were used, one for the generator (${\\cal L}_f$) and another for the discriminator (${\\cal L}_h$). The discriminator losses are applied periodically at a pre-defined interval ($n_f$ and $n_h$). \n\nThe proposed method is referred to as the discriminative-constrained optimal transport network (DOTN). It does not require any additional knowledge about the noise types during training. This paper compares DOTN with two other adversarial domain adaptation methods, namely DAT and MDAN, using two datasets (VoiceBank and TIMIT). Different types of noise are artificially added to these datasets using the DEMAND dataset at different SNR levels to create different noise domains. The quality of the speech enhancement methods are evaluated using the PESQ and STOI metrics.\n\nThe overall results show that the proposed DOTN outperforms DAT and MDAN on both datasets. However, the model sizes and computational complexities of the models are not reported, making it difficult to compare different methods. Besides, there is lack of detailed discussions/analyses about the design of the training algorithm (see main review).\n\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe use of optimal transport (OT) for adversarial domain adaptation has been investigated in prior work for classification tasks. This work focuses on applying it to speech enhancement, which is a regression problem. However, the paper did not discuss the specific challenges in applying OT domain adaptation to a regression problem.\n\nQuality:\n\nThe experiments are limited to comparing the overall SE performance of DOTN, DAT and MDAN on two datasets. Some important details are missing and more ablation studies could have been done. For example:\n\n- How important is it to include the discriminator training stage? \n- How were the hyper-parameters ($\\alpha$, $\\beta$, $n_f$ and $n_h$) tuned? How sensitive were they on speech enhancement performance?\n\nClarity:\n\nThe paper is generally clear enough to understand and follow through. Given that the paper focuses on comparing DOTN with DAT and MDAN, it can be useful to include a section that compare and contrast these methods. Some minor comments:\n- In Algorithm 1, $t$ is used in lines 4 & 7. So, it should be included in the **Require:** section.\n- In Algorithm 1, in lines 5 & 8, both $\\theta_f$ and $\\theta_h$ should be passed to $Adam(...)$ since both $f(.)$ and $h(.)$ are used to compute ${\\cal L}_f$ and ${\\cal L}_h$.\n- In Eq. 1, is it necessary to include the $\\alpha || x_i^s - x_j^t||^2$ term (since it is independent of $\\gamma$ and $f$)?\n\nSignificance:\n\nThe problems being investigated in this paper is specific to speech enhancement (or regression problem in general). The experimental setup is limited to adapting to different noise domains and the noisy data are obtain by artificially adding noise to clean datasets. The study would have been more impactful if realistic noisy data were used.\n\n---\n\nAfter the rebuttal: I have read the rebuttal by the authors. Given the new results provided on CHiME3 dataset, I have increase my score by 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The results reported in the paper are based on artificially created noisy speech data. It is unclear how well the same method will perform on realistic noisy data. Furthermore, the lack of details about the models being evaluated makes it difficult to make a fair comparison.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GAV2escP_5V", "reviewer": "Reviewer_Ryjo", "summary": "The paper presents an unsupervised noise adaptive speech enhancement method by combining optimal transport network and Wasserstein generative adversarial network. Experimental results show that the method is effective. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty of this work is to propose a discriminator-constrained optimal transport network for unsupervised domain adaptation for speech enhancement. It is unsupervised and outperforms two weakly-supervised methods. The written of the paper is generally okay but the clarity could be improved as exemplified below. \n\nOptimal transport (OT) is the theory fundamental to this work. However, it is neither well explained (thinking OT is less known for the speech enhancement community) nor well motivated (namely why it is a good solution for this problem). \n\nThe abbreviation OT is not introduced when it is used the first time, but later only. \n\nNo details are provided about how the two reference methods DAT and MDAN were implemented for the experiments done in this work. When comparing the performance numbers (e.g. PESQ scores) of DAT in this paper and the numbers of DAT in its original paper [19], they are different for similar testing conditions; of course, various settings can impact on the results, which makes it important to explain the implementation details and ensure the reference works are reproduced correctly.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a discriminator-constrained optimal transport network (DOTN) for unsupervised speech enhancement.  The authors apply joint distribution optimal transport with a discriminator under Wasserstein GAN to generate enhanced speech signals.  The novelty of the work is the combination of joint distribution OT and WGAN for both domain adaptation and unsupervised learning simultaneously in the application of speech enhancement, which is a regression setting.  The authors conduct experiments on Voice Bank and TIMIT datasets under various conditions and show good performance under the PESQ and STOI metrics.  The paper is well written.  The work is theoretically solid and the performance under PESQ and STOI is decent.  However, there are a few standing concerns. First of all, the authors should make it clear that both joint distribution optimal transport and WGAN are existing techniques by directly citing the previous work when mathematically formulating the problem in Section 3.2 even though they are mentioned in the related work.  Second, there has been a significant concern regarding the quality of the generated speech by DOTN in the demo where distortions are quite noticeable perceptually.  It is not clear whether this is due to the unsupervised setting.  As suggested by one of the reviewers, the authors upload the comparative performance between MetricGAN+ and the proposed DOTN in terms of PESQ and STOI.  It shows that DOTN has slightly better STOI and is quite a bit lower in PESQ since SOTA of MetricGAN+ is optimized under PESQ.  This appears to clear up the concern to some degree.  All reviewers consider the work interesting and the authors are responsive in the discussion period.  I would recommend acceptance.  But the authors should include the additional results in the rebuttal and discussion in the revised version.  Furthermore, it would be greatly helpful to include subjective evaluation results in the revision to make the work more convincing.   
Decision: Accept (Poster)