Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes diffusion models based speech enhancement. In particular, it extends enhancement to a large number of degradations beyond additive noise. The degradation includes common speech degradations such as codec artifacts, bandwidth reduction, reverb etc. The proposed approach puts together different pieces of diffusion-based learning in the context of speech enhancement and then different variations on the top of the base approach are applied to achieve improved performances. 
strength_and_weaknesses: Strengths


1. Diffusion based approaches have been very successful in generative tasks in other domains. While their applications to the audio domain are starting to show up, the success is perhaps not yet at the level in other domains like images. To this end, the authors' attempt on using diffusion based learning for speech enhancement is a good idea. Extending it to degradations beyond additive noise also makes sense. 


2. I liked that authors did subjective tests as well. 


Weaknesses


1. The experimental section leaves quite a few things unanswered. More on it in the review summary below - for which I would like to see the author's response. 


2. I think the novelty of the diffusion-learning process is limited. It largely follows prior works. 


3. Some questions below are part of the weaknesses. 

clarity,_quality,_novelty_and_reproducibility: The paper is largely clear and the writing quality is good. The novelty is limited to some extent. I did not see any comments on reproducibility like code, model release etc. 

summary_of_the_review: This work proposes speech enhancement through diffusion based approaches. It trains a single model to learn from a large number of speech degradations. 


1. There is a waveform direct waveform based loss too - the significance of which seems to be very high. Doesn’t that default the whole process to so many other speech enhancement models where MSE  on waveform or spectrograms is common. Moreover, there are other latent targets which are also added like pitch, VAD, loudness. One of the versions uses noise level information too. I wonder how much some of the other prior methods can be improved just by adding these additional latent targets and  these auxiliary information. 



2. In Section 4.2.1 authors discuss a regression based approach with Universe architecture. Along with waveform and STFT losses are other losses  (as in UNIVERSE like latent targets etc, ) used here ? 


3. I am not sure how fair the comparison is with prior works here. 
First of all the proposed model is likely orders of magnitude bigger than several of the prior works. 
In fact, some of them might be causal as well (which proposed model is not) and causality can have significant impact on performance.
Moreover, the training data is different for different algorithms whereas here a large corpus of private data is used. But importantly, several of these prior works were never trained on such degradations.  I wonder what performance would look like if we just take them and train them with all these other degradations. 
I think decoupling the point above is necessary because it will tell us whether variety in speech degradations is important or diffusion-based learning is bringing something to the table. 



3. It would be good to show how this model performs in really difficult conditions. For example, a large number of state of the art methods do a really good job on enhancement but struggle in low-SNRs.  A model like the current one (very large, non-causal) are clearly not the most practical speech enhancement systems which often have real-time uses. Nevertheless, this work puts together a whole bunch of pieces to build the enhancement system. Considering this, stronger results would have been expected, and especially in situations where current state of the art struggles. How does this model compare against prior works in really difficult quality conditions?



4. While I like subjective tests in the paper, I think adding MOS subjective scores would add a lot of value. It will clarify a lot of different aspects. Comparative tests do not give an idea of absolute performance of UNIVERSE or others. 

-- updates after rebuttal --
updated the score and comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on various types of speech distortions (a total of 55) and aims to realize universal speech enhancement based on score-diffusion-based models. The paper carefully describes their complicated architecture based on the various attempts motivated by related studies and incrementally improves the architecture described in Section 3. Overall, their proposed neural network architecture and training methods are novel. The paper also shows the effectiveness of the proposed method by comparing various other state-of-the-art methods with their speech distortion setups based on the subjective preference test.
strength_and_weaknesses: Strength
- the first part (abstract and introduction) of this paper is very well written, and I can easily understand the motivation of this work.
- the related studies for diffusion-based models are adequate.
- although the proposed architecture and training method are complicated, the authors provide detailed explanations of how they evolved their architecture and method with the experimental evidence.
- task setup and their modeling are novel

Weaknesses
- The paper has weak reproducibility (see below).
- The explanation of the model (especially Section 3.2) and some experimental discussions (e.g., Table 4) are not self-consistent (see my comments below).
- The modeling efforts and task setup should be separately evaluated. For example, it is better to show the effectiveness of the proposed model with the same training data for the VCTK+demand experiments since the training data is different. We could also train the conventional models with the same training data.
- another significant distortion comes from interference speakers. I expect the paper to have bubble noise, but speech separation should also be considered if we call this method "universal" speech enhancement.


clarity,_quality,_novelty_and_reproducibility: Clarity
- The paper's first parts (abstract and introduction) are well written. However, Section 3 (especially Section 3.2) is not self-consistent within the main document. (it requires reading the appendix to understand the model). Of course, I know we always have page limitations and cannot put all our content together. However, the authors could make more effort by briefly providing the theory and practical parts in the appendix to Section 3.2. Section 3.3 is informative, but we can move some development history parts to the appendix instead. 

Quality
- I understand the importance of the subjective evaluation since this is a new setup. However, this paper also has a significant improvement in the modeling part. To validate their model, it is better to have more fair comparisons with the other methods based on the existing database and metrics. Table 4 corresponds to this validation, but due to the difference in the training setup, we cannot compare them. Also, Table 4 is in the appendix and is not considered as the main result.

Novelty
- The prior study does not have such significant variations, and its problem setup is novel.
- The modeling part evolved from the conventional method (described in Section 3.3).

Reproducibility
- The paper has an issue with reproducibility. The database and scoring (based on SESQA, correct me if I'm wrong) are internal. The source code is not available. The source code part is not critical in general; however, this method is very complicated, as described in Section 3.3, and it would not be easy for other people to re-implement this technique.
summary_of_the_review: The paper has sufficient novelties in the problem setup and modeling parts. However, I'm concerned about the lack of reproducibility of this method and the fair comparisons with the existing methods in the conventional metrics.

Other suggestions:
- It is better to discuss speech separation since it is regarded as another critical speech distortion in the broad sense.
- Section 2, the last paragraph: I'm not very sure about the effectiveness of the generative model method for this setup. Since we could simulate most of these distortions, we can make a discriminate model with the artificially created pair data. Please clarify this point.
- Why does the evaluation metric not include PESQ?
- Figure 1(a): I'm very curious how we predict "g" since $\log (\sigma)$ seems to be a scale, and it may not have enough information to predict it. Probably, I misunderstand $\log (\sigma)$. It would be great if the authors clarify it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a universal diffusion model for speech enhancement for a wide variety of distortions (Table 3 in the appendix), in contrast to prior studies which focused on a limited set of distortion types (cf Table 2). The authors first presented a series of exploration of model architectures and auxiliary losses based on which the authors arrived at their final model. The authors then demonstrated that the proposed model, UNIVERSE, performs better than prior models developed for specific types of distortion as well as ablated models that are trained with regression loss or trained for a specific type of distortion. 
strength_and_weaknesses: Strengths
- This paper appears to be the first general model for speech enhancement based on diffusion. Empirical results are very strong. The samples provided in the supplementary materials are impressive especially when tested on the real world samples.
- The presentation of how the authors arrived at their final model architecture and training losses is excellent. The authors motivated the design well. Results of intermediate models are included, serving as a nice reference for future development of audio enhancement models
- It is also informative to include a baseline that is trained with regression loss.

Weaknesses
- The comparison with prior work may not be entirely fair considering that the proposed model is trained on a different dataset. Hence, it cannot be concluded from the paper that UNIVERSE is better than prior work because diffusion is a better objective of enhancement and/or the proposed model architecture is better. In addition, the target UNIVERSE is trained to predict is also enhanced as described in Appendix C, which might be another advantage of the proposed model over baselines. This question can be answered if a) the authors retrain the strongest baseline (e.g.,HiFi-GAN-v2) using the same dataset, or b) train the proposed model using the same public data as the prior work.
- While the authors share the type of distortions used for corrupting clean speech as input, the details of the parameter used in each distortion are not provided (e.g., SNR for colored noise). How clean speech was selected /what noise dataset was used / how many hours of noise are used for training are not specified. It would be hard to reproduce the data setup given the paper.
- The authors mentioned in the appendix that the target was enhanced (Sec C) but results comparing using original speech versus enhanced speech are not provided.

clarity,_quality,_novelty_and_reproducibility: This paper is well written with high quality results. While diffusion models are not new for speech, this work presents very strong empirical results and universality across distortion types. The main concern would be reproducibility and comparability with past/future work, since the model is only trained on internal data.
summary_of_the_review: It is exciting to see a universal model for speech enhancement based on diffusion with very strong performance. The quality of the work is good. However, it is hard to assess exactly why the proposed model is superior to prior work and how future work can be compared with the proposed model due to the use of non-public datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a "universal" speech enhancement model that handles 55 different distortions including background noise, reverberation, codec artifacts, etc. The model uses a UNet-like generator that has been trained with a diffusion loss. The paper further conducts a side-by-side evaluation with 22 expert listeners, who preferred the model's output compared to competing models.
strength_and_weaknesses: Strengths:
- To the best of my knowledge, this paper is the first in the speech enhancement literature to handle this many speech distortions. 55 distortions from 10 different families are considered: bandwidth reduction, clipping, codec artifacts, silent gaps, excessive dynamics
compression/expansion, sub-optimal equalization, noise gating, and others.
- While it is not the first paper that proposes to use diffusion for speech enhancement, the authors show that it performs better than existing approaches on subjective side-by-side evaluations, as well as automated metrics on the VoiceBank-Demand test set.

Weaknesses:
- While an impressive engineering feat, the paper contributes little to the community's theoretical or empirical understanding of speech enhancement or diffusion.
- The findings in the paper are also generally not reproducible. The authors use internal training data and programmatically generate distorted speech, but neither the training data nor code for the programmatic distortions are made available. The paper proposes a very "hacky" U-Net architecture with numerous non-trivial and non-standard modifications, but while the ablation study is useful, it is hard to see how others can easily replicate this without the authors releasing the code for it. Finally, the side-by-side evaluations are also subjective and not generally reproducible. I think it is a huge missed opportunity, because the authors could have taken the chance to establish a new benchmark task and associated metrics for "universal" speech enhancement.


clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. While an impressive feat of engineering, the paper's contributions are neither novel nor reproducible.
summary_of_the_review: As written, I do not think this paper is suitable for ICLR. While the authors claim impressive results on subjective side-by-side evaluations on an impressive number of speech distortions, it is hard to see how others in the ICLR community can build upon this work since there is no new ML technique introduced and the empirical findings are generally not reproducible. At ICLR, there is similar precedent for a well-engineered visual speech recognition system that also did not introduce new ML techniques and whose empirical findings are generally not reproducible (https://openreview.net/forum?id=HJxpDiC5tX). The reviewers recommended that the paper would be more suitable at an applications-oriented venue. I would make the same recommendation in this case.

If the authors would like to revise their work to make it more suitable for ICLR, I would suggest the following:
1) Create a benchmark task for evaluating "universal" speech enhancement. This would involve open-sourcing code for programmatically generating their 55 speech distortions.
2) Propose new automated metrics that holistically evaluates "universal" speech enhancement, and show that they correlate well with subjective side-by-side listener evaluations. This builds upon a long precedent in the speech enhancement literature (PESQ, COVL, STOI, etc) and would be an immensely valuable contribution for the community.
3) Use a simpler and less "hacky" deep learning architecture that is easy for others to replicate by themselves, or release code for it to aid others in their reproducibility efforts.

Finally, I would like to point the authors' attention to several related work in the literature for Table 4. [1], [2], [3], [4] outperforms Universe-Denoise on VoiceBank-Demand. [1], [3] outperforms Universe-Denoise-E on VoiceBank-Demand. [1] outperforms both Universe-Denoise and Universe-Denoise-E on IS20 DNS Challenge. Given that Universe-Denoise-E is an expectation rather than the output of a deterministic model, I think it also helps to provide distribution statistics like median and std to help readers understand its performance. Unfortunately, the related work does cast some doubt on whether a "universal" speech enhancement model can perform as competitively as specialized speech enhancement models, but in my opinion, the presence of better-performing specialized models would not detract from the contribution of a "universal" model.

[1] DPT-FSNet: Dual-Path Transformer Based Full-Band and Sub-Band Fusion Network for Speech Enhancement. Dang et al.

[2] Single-Channel Speech Enhancement using Learnable Loss Mixup. Chang et al.

[3] CMGAN: Conformer-based Metric GAN for Speech Enhancement. Cao et al.

[4] Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement. Yu et al.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a model for ‘universal’ speech enhancement, to address a wide-range of distortions at the same time. The model itself is a diffusion-based denoiser that uses score-based diffusion. 

The reviewers agree that this is one of the first papers that addresses all the varied distortions in a single model, with good empirical results. While the term ‘universal’ may need reconsideration since the work does not address multitalker separation, which is a very popular use case for speech frontends. 

There is also some novelty in the application of score-based diffusion models to enhancement, and the overall model architecture, which is presented in a nice way despite being complex. Although score based diffusion is from prior work, reviewers agreed that its application to enhancement is somewhat novel. 

One of the shortcomings raised by multiple reviewers is that it lacks enough comparisons. For instance, there are various components to the model formulation, architecture, losses, and the data. It is not always clear what parts are contributing to the final performance. The reviewers noted that the mismatch in training data and the differences in model capacity likely makes the presented comparisons with prior art unfair. While the authors argue in their rebuttal that their model is evaluated on held-out data, that does not fully address the fact that the models compared against are at a disadvantage. This makes it difficult to assess what parts of the model are contributing to the gains (discounting the universal nature of the new model, which is indeed unique to the presented work). 

Along similar lines, the authors do not train on public datasets nor make the code available. The presented technique is complex, all of which makes it a bit challenging to replicate. What would have helped is if the authors presented results also using public datasets for training, to make it easier to compare, and would have served as a useful target for researchers trying to replicate the technique. This could have also made comparisons with prior art more fair.
justification_for_why_not_higher_score: Reproducibility and fair comparison were identified as significant enough to recommend rejection.
Decision: Reject