Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an unsupervised noise adaptive speech enhancement method by combining optimal transport network and Wasserstein generative adversarial network. Experimental results show that the method is effective. 
main_review: The main novelty of this work is to propose a discriminator-constrained optimal transport network for unsupervised domain adaptation for speech enhancement. It is unsupervised and outperforms two weakly-supervised methods. The written of the paper is generally okay but the clarity could be improved as exemplified below. 

Optimal transport (OT) is the theory fundamental to this work. However, it is neither well explained (thinking OT is less known for the speech enhancement community) nor well motivated (namely why it is a good solution for this problem). 

The abbreviation OT is not introduced when it is used the first time, but later only. 

No details are provided about how the two reference methods DAT and MDAN were implemented for the experiments done in this work. When comparing the performance numbers (e.g. PESQ scores) of DAT in this paper and the numbers of DAT in its original paper [19], they are different for similar testing conditions; of course, various settings can impact on the results, which makes it important to explain the implementation details and ensure the reference works are reproduced correctly.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose an approach for speech enhancement based on optimizing the empirical OT loss between the joint distribution of noisy/clean training data and noisy/cleaned target data (c.f. (1), line 139). The cleaned speech outputs are additionally optimized under a standard WGAN formulation (c.f. line 129). Their approach consistently outperforms DAT[19] and MDAN[44] on VoiceBank-DEMAND and Helicopter-TIMIT mixtures, and performs roughly on-par or slightly worse on TIMIT-(Crowd-party, Cafeteria, Baby-cry) mixtures which contain speech, according to the standard PESQ and STOI metrics.
main_review: Strengths:

- OT is a very natural formulation to domain adaptation, and has been adopted by others.
- Gains on non-speech noise over baselines.
- Code made available, so should be reproducible.

Limitations:

- Lower novelty, standard training metrics for domain adaptation.
- The baselines are not SOTA speech enhancement techniques, and the results are on non-standard datasets, reducing the significance of the results.
- Standard metrics are reported but no audio demonstrations or human evaluations (MOS) are provided, these are important ways to assess true performance.
- The approach does not improve mixtures containing other speech, which is another signficant limitation.

Current Assessment:

- Falls short wrt ML novelty for Neurips and significance to the ML and speech enhancement communities.

---- 

Post-rebuttal (updated)

Thank you to the reviewers for their detailed responses. 
- The audio demonstrations reveal that all of the methods compared are performing quite poorly, far below SOTA. 
- (update 1) The follow-up MetricGAN+ (SOTA) results provided show that the approach achieves slightly better STOI scores, but significantly worse PESQ scores. However, the fact that the MetricGAN+ is optimized on PESQ makes this comparison more difficult to interpret. 
- (update 2) Follow-up MetricGAN+ and Transformer audio demos are subjectively significantly better than the proposed method. MOS or A/B preference listening tests vs. MetricGAN+ and/or the SE Transformer should be included in the results. The current audio demos suggest that the method would be outperformed by MetricGAN+ and the SE Transformer in terms of MOS or A/B assessment, but only 2 audio examples were provided.
- The manuscript needs substantial revision. The paper in general, and particularly the experimental section is not detailed enough to establish the contributions the paper makes, and does not describe the approach, particularly in mapping abstract losses and inputs/outputs to a concrete system and associated details.
- As a type of kernel-based method for adaptation/reconstruction, the training sets are likely too small, and the details regarding reconstruction (chunking etc.) likely not fully optimized, perhaps explaining the results.
- (update 3) Based on the additional results provided, I have increased my score, but remain concerned that the manuscript and results are not complete enough to justify an acceptance recommendation.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presented a novel method to achieve unsupervised domain adaptation for speech enhancement. There is little work in unsupervised domain adaptation for regression tasks and the paper combined techniques from optimal transport theory and generative adversarial networks, accomplishing better results on two datasets, VoiceBank-DEMAND and TIMIT, while requiring less supervision than prior work.
main_review: Originality: Although there exists many works on unsupervised domain adaptation, the authors proposed a novel method to tackle the speech enhancement task, which resembles regression rather than classification, an area that had seen much less success previously. Compared to prior work on unsupervised domain adaptation for speech enhancement, the proposed discriminator-constrained optimal transport network (DOTN) does not require any additional labels. The paper uses a novel combination of known techniques to solve an important problem. The related work and contributions were stated very clearly.

Quality: The authors demonstrated the strength of their novel approach well using two standard datasets (VoiceBank-DEMAND and TIMIT) and two metrics (PESQ and STOI), while comparing against two other prior work on unsupervised domain adaptation for speech enhancement. How the datasets were cleaned and prepared was described in detail and results and comparisons were analyzed in depth. The authors also tested and stated the limitations well by performing the multiple noise source experiment, testing the effects of catastrophic forgetting. 

Clarity: The submission was very well written and clearly organized. One minor suggestion is to flip the row orders of noise type and methods in the tables to make comparisons clearer. 

Significance: The results are quite significant as speech enhancement could benefit many downstream tasks and the paper was able to achieve better performance than prior works on unsupervised domain adaptation while not needing additional labels. Having a truly unsupervised setting is quite important for real world use since knowing the type of noise is an unrealistic assumption in many cases. 
limitations_and_societal_impact: The authors adequately addressed limitations by discussing performance drops when too many sources of noises are present simultaneously. The paper do not have any immediate negative impacts. The only concern is that by improving the performance of various speech tasks, there can be more privacy violation or surveillance.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes using a discriminator-constrained optimal transport method for unsupervised speech enhancement (SE). The method is specifically designed for a regression task. It involves 2 parts. The first parts estimates a generator function as an OT problem to align the source and target labels. The source-target alignments ($\gamma$) and the generator function parameters (${\theta}_f$) are estimated in an interleaving fashion. The second part incorporates a discriminative training component on the label distributions using WGAN. Two discriminative loss functions were used, one for the generator (${\cal L}_f$) and another for the discriminator (${\cal L}_h$). The discriminator losses are applied periodically at a pre-defined interval ($n_f$ and $n_h$). 

The proposed method is referred to as the discriminative-constrained optimal transport network (DOTN). It does not require any additional knowledge about the noise types during training. This paper compares DOTN with two other adversarial domain adaptation methods, namely DAT and MDAN, using two datasets (VoiceBank and TIMIT). Different types of noise are artificially added to these datasets using the DEMAND dataset at different SNR levels to create different noise domains. The quality of the speech enhancement methods are evaluated using the PESQ and STOI metrics.

The overall results show that the proposed DOTN outperforms DAT and MDAN on both datasets. However, the model sizes and computational complexities of the models are not reported, making it difficult to compare different methods. Besides, there is lack of detailed discussions/analyses about the design of the training algorithm (see main review).




main_review: Originality:

The use of optimal transport (OT) for adversarial domain adaptation has been investigated in prior work for classification tasks. This work focuses on applying it to speech enhancement, which is a regression problem. However, the paper did not discuss the specific challenges in applying OT domain adaptation to a regression problem.

Quality:

The experiments are limited to comparing the overall SE performance of DOTN, DAT and MDAN on two datasets. Some important details are missing and more ablation studies could have been done. For example:

- How important is it to include the discriminator training stage? 
- How were the hyper-parameters ($\alpha$, $\beta$, $n_f$ and $n_h$) tuned? How sensitive were they on speech enhancement performance?

Clarity:

The paper is generally clear enough to understand and follow through. Given that the paper focuses on comparing DOTN with DAT and MDAN, it can be useful to include a section that compare and contrast these methods. Some minor comments:
- In Algorithm 1, $t$ is used in lines 4 & 7. So, it should be included in the **Require:** section.
- In Algorithm 1, in lines 5 & 8, both $\theta_f$ and $\theta_h$ should be passed to $Adam(...)$ since both $f(.)$ and $h(.)$ are used to compute ${\cal L}_f$ and ${\cal L}_h$.
- In Eq. 1, is it necessary to include the $\alpha || x_i^s - x_j^t||^2$ term (since it is independent of $\gamma$ and $f$)?

Significance:

The problems being investigated in this paper is specific to speech enhancement (or regression problem in general). The experimental setup is limited to adapting to different noise domains and the noisy data are obtain by artificially adding noise to clean datasets. The study would have been more impactful if realistic noisy data were used.

---

After the rebuttal: I have read the rebuttal by the authors. Given the new results provided on CHiME3 dataset, I have increase my score by 1.
limitations_and_societal_impact: The results reported in the paper are based on artificially created noisy speech data. It is unclear how well the same method will perform on realistic noisy data. Furthermore, the lack of details about the models being evaluated makes it difficult to make a fair comparison.
review_title: null

Meta-review: This paper proposes a discriminator-constrained optimal transport network (DOTN) for unsupervised speech enhancement.  The authors apply joint distribution optimal transport with a discriminator under Wasserstein GAN to generate enhanced speech signals.  The novelty of the work is the combination of joint distribution OT and WGAN for both domain adaptation and unsupervised learning simultaneously in the application of speech enhancement, which is a regression setting.  The authors conduct experiments on Voice Bank and TIMIT datasets under various conditions and show good performance under the PESQ and STOI metrics.  The paper is well written.  The work is theoretically solid and the performance under PESQ and STOI is decent.  However, there are a few standing concerns. First of all, the authors should make it clear that both joint distribution optimal transport and WGAN are existing techniques by directly citing the previous work when mathematically formulating the problem in Section 3.2 even though they are mentioned in the related work.  Second, there has been a significant concern regarding the quality of the generated speech by DOTN in the demo where distortions are quite noticeable perceptually.  It is not clear whether this is due to the unsupervised setting.  As suggested by one of the reviewers, the authors upload the comparative performance between MetricGAN+ and the proposed DOTN in terms of PESQ and STOI.  It shows that DOTN has slightly better STOI and is quite a bit lower in PESQ since SOTA of MetricGAN+ is optimized under PESQ.  This appears to clear up the concern to some degree.  All reviewers consider the work interesting and the authors are responsive in the discussion period.  I would recommend acceptance.  But the authors should include the additional results in the rebuttal and discussion in the revised version.  Furthermore, it would be greatly helpful to include subjective evaluation results in the revision to make the work more convincing.   
Decision: Accept (Poster)