D4AM: A General Denoising Framework for Downstream Acoustic Models
Authors: Chi-Chang Lee
Yu Tsao
Hsin-Min Wang
Chu-Song Chen
Venue: ICLR 2023 poster (2023)
Keywords: audio processing
speech enhancement
robust automatic speech recognition
auxiliary task learning

Abstract: The performance of acoustic models degrades notably in noisy environments. Speech enhancement (SE) can be used as a front-end strategy to aid automatic speech recognition (ASR) systems. However, existing training objectives of SE methods are not fully effective at integrating speech-text and noise-clean paired data for training toward unseen ASR systems. In this study, we propose a general denoising framework, D4AM, for various downstream acoustic models. Our framework fine-tunes the SE model with the backward gradient according to a specific acoustic model and the corresponding classification objective. In addition, our method aims to consider the regression objective as an auxiliary loss to make the SE model generalize to other unseen acoustic models. To jointly train an SE unit with regression and classification objectives, D4AM uses an adjustment scheme to directly estimate suitable weighting coefficients rather than undergoing a grid search process with additional training costs. The adjustment scheme consists of two parts: gradient calibration and regression objective weighting. The experimental results show that D4AM can consistently and effectively provide improvements to various unseen acoustic models and outperforms other combination setups. Specifically, when evaluated on the Google ASR API with real noisy data completely unseen during SE training, D4AM achieves a relative WER reduction of 24.65% compared with the direct feeding of noisy input. To our knowledge, this is the first work that deploys an effective combination scheme of regression (denoising) and classification (ASR) objectives to derive a general pre-processor applicable to various unseen ASR systems. Our code is available at https://github.com/ChangLee0903/D4AM.

Published as a conference paper at ICLR 2023
D4AM: A GENERAL DENOISING FRAMEWORK FOR DOWNSTREAM ACOUSTIC MODELS
Chi-Chang L Yu Tsao², Hsin-Min Wang", Chu-Song Chen 1.2
1 National Taiwan University, Taipei, Taiwan
2 PAcademia Sinica, Taipei, Taiwan
ABSTRACT
The performance of acoustic models degrades notably in noisy environments. Speech enhancement (SE) can be used as a front-end strategy aid automatic speech recognition (ASR) systems. However, existing training objectives of SE methods are not fully effective at integrating speech-text and noisy-clean paired data for training toward unseen ASR systems. In this study, propose general denoising framework, D4AM, for various downstream acoustic models. Our frame- work fine-tunes the SE model with the backward gradient according specific acoustic model and the corresponding classification objective. In addition, our method aims to consider the regression objective as an auxiliary loss to make the SE model generalize to other unseen acoustic models. To jointly train an SE unit with regression and classification objectives, D4AM uses an adjustment scheme di- rectly estimate suitable weighting coefficients rather than undergoing grid search process with additional training costs. The adjustment scheme consists of two parts: gradient calibration and regression objective weighting. The experimental results show that D4AM can consistently and effectively provide improvements to various unseen acoustic models and outperforms other combination setups. Specifically, when evaluated on the Google ASR API with real noisy data completely unseen during SE training, D4AM achieves relative WER reduction of 24.65% % compared with the direct feeding of noisy input. To our knowledge, this is the first work that deploys effective combination scheme of regression (denoising) and classifica- tion (ASR) objectives to derive a general pre-processor applicable to various unseen ASR systems. Our code is available at httts///zttbbbocoomm/mmmccc/uDDD
INTRODUCTION
Speech enhancement (SE) aims extract speech components from distorted speech signals to obtain enhanced signals with better properties (Loizou 2013). Recently, various deep learning models (Wang et al. 2020 Lu et al., 2013: Xu et al. 2015 Zheng et al. 2021 Nikzad et al. (020) have been used to formulate mapping functions for SE. which treat SE as a regression task trained with noisy-clean paired speech data. Typically, the objective function is formulated using signal-level distance measure (e.g., LI norm (Pandey & Wang. 2018, Yue et al., 2022), L.2 norm (Ephraim & Malah
1984 Yin al.) 22200 Xu et al. 2020), SI-SDR (Le Roux al. 2019 Wisdom et al. 2020 Lee et al. 2020), or multiplerreoolution loss (Défossez et al. (0200). ). In speech-related applications, SE units are generally used as key pre-processors improve the performance of the main task in noisy environments. To facilitate better performance on the main task, certain studies focus on deriving suitable objective functions for SE training.
For human-human oral communication tasks. SE aims to improve speech quality and intelligibility. and enhancement performance is usually assessed by subjective listening tests. Because large- scale listening tests are generally prohibitive, objective evaluation metrics have been developed to objectively assess human perception of a given speech signal (Rix et al., 2001 Taal et al. 2010
Jensen & Taal 2016 Reddy al. 2021 Perceptual evaluation of speech quality (PESQ) (iix et al. 2001) and short-time objective intelligibility (STOI) (Taal et al., 2010 Jensen & Taal 2016) popular objective metries designed to measure speech quality and intell intelligbbility respectively.
Recently, DNSMOS (Reddy et al. (22TT) has been developed as a non-instructive assessment tool that predicts human ratings (MOS scores) of speech signals. In order to obtain speech signals with

---

Published as a conference paper at ICLR 2023
improved speech quality and intelligibilliy, many SE approaches attempt to formulate objective functions for SE training directly according to speech assessment metrics (Fu et al. 2018 Fu et al. 2019; 202T Another group of approaches, such deep feature loss (Germain et al. 2 and
HiFi-GAN (Su et al. 2020). propose to perform SE by mapping learned noisy latent features to clean ones. The experimental results show that the deep feature loss can enable the enhanced speech signal to attain higher human perception scores compared with the conventional LI and L2 distances.
Another prominent application of SE is to improve automatic speech recognition (ASR) in noise (Selizer et al. 2013; Weninger et al. 2013b; Li et al., 2014; Cui et al. 202T). ASR sys- tems perform sequential classification, mapping speech utterances to sequences of tokens. Therefore, the predictions of ASR systems highly depend on the overall structure of the input utterance. When regard to noisy signals, ASR performance will degrade significantly because noise interference corrupts the content information of the structure. Without modifying the ASR model, SE models can be trained separately and "universally" used as pre-processor for ASR to improve recognition accuracy. Several studies have investigated the effectiveness of SE's model architecture and objective function in improving the performance of ASR in noise (Geiger et al. 2014a] Wang et al. 2020
Zhang et al., 2020 Chao al.) 2021 Meng et al., 2017 Weninger et al. 2015a Du et al. 2019 Kinoshita et al. 2020; Meng et al. 2018). The results show that certain specific designs, including model architecture and input format, are favorable for improving ASR performance. However, has also been reported that improved recognition accuracy in noise is not always guaranteed when the ASR objective not considered in SE training (Geiger et al. 2014b), A feasible approach to tune the SE model parameters toward the main ASR task is to prepare the data of (noisy) speech-text pairs and backpropagate gradients on the SE model according to the classification objective provided by the ASR model. That is. SE models can be trained on a regression objective (using noisy-clean paired speech data) or/and a classification objective (using speech-text paired data). Ochiai et al. (201 b) proposed multichannel end-to-end (E2E) ASR framework, where mask-based MVDR (minimum variance distortionless response) neural beamformer is estimated based on the classification objective. Experimental results on CHiME-4 (Jon et al. (017)) confirm that the estimated neural beanformer can achieve significant ASR improvements under noisy conditions. Mcanwhile, Chen et al. (2015) and Ma et al. (2021) proposed to train SE units by considering both regression and classification objectives, and certain works (Chen et al.) [2013] Ochiai et al.) 2017a proposed to train SE models with E2E-ASR classification objectives. A common way to combine regression and classification objectives is to use weighting coefficients to combine them into a joint objective for SE model training.
Notwithstanding promising results, the use of combined objectives in SE training has two limitations. First, how effectively combine regression and classification objectives remains an issue. large- scale grid search is often employed to determine optimal weights for regression and classification objectives, which requires exhaustive computational costs. Second, ASR models are often provided by third parties and may not be accessible when training SE models. Moreover, due to various training settings in the acoustic model, such as label encoding schemes (c.g., word-piece (Schuster) & Nakajimal 2012), byte-pair-encoding (BPE) (Gage 1994, Sennrich et al.] 2016)) and character), model architectures (e.g., RNN (Chiu et al. 2018 Rao et al. [207]]He et | Sainath et al.) 2020), transformer (Vaswani et al.) 2017 Zhang al. (020), and conformer (Gulati al. (020)), and objectives (c.g.. Connectionist Temporal Classification (CTC) (Graves et al. 2006), Attention (NLL) (Chan al. 2016), and their hybrid version (Watanabe et al. 2017)) SE units trained according to a specific acoustic model may not generalize well to other ASR systems. Based on the above limitations, we raise the question: Can we effectively integrate speech-text and noisy-clean paired data to develop a denoising pre-processor that generalizes well to unseen ASR systems?
In this work, we derive novel denoising framework, called D4AM, be used as "universal" pre- rre-processor to improve the performance of various downstream acoustic models in noise. To achieve this goal, the proposed framework focuses on preserving the integrity of clean speech signals and trains SE models with regression and classification objectives jointly. By using the regression objective as an auxiliary loss, we circumvent the need to require additional training costs to grid search the appropriate weighting coefficients for the regression and classification objectives. Instead, D4AM applies an adjustment scheme to determine the appropriate weighting coefficients automatically and efficiently. The adjustment scheme is inspired by the following concepts: (1) we attempt to adjust the gradient yielded by a proxy ASR model so that the SE unit can be trained to improve the general recognition capability; (2) we consider the weighted regression objective as a regularizer and, thereby,
2

---

Published as a conference paper at ICLR 2023
prevent over-fitting while training the SE unit. For (I), we derive coefficient agelb (abbreviation for gradient calibration) according to whether the classification gradient conflicts with the regression gradient. When the inner product of the two gradient sets is negative, agelb is the projection of the classification gradient on the regression gradient; otherwise, it is set to O. For (2), we derive coefficient O'srpr (abbreviation for surrogate prior) based ARML (auxiliary task reweighting for minimum-data learning) (Shi et al., (020) and formulate the parameter distribution induced by the weighted regression objective as the surrogate prior when training the SE model. From the experimental results on two standard speech datasets, we first notice that by properly combining regression and classification objectives, D4AM can effectively improve the recognition accuracy of various unseen ASR systems and outperform the SE models trained only with the classification objective. Next, by considering the regression objective as an auxiliary loss, D4AM can be trained efficiently to prevent over-fitting even with limited speech-text paired data. Finally, D4AM mostly outperforms grid search. The main contribution of this study is two-fold: (1) to the best of our knowledge. this is the first work that derives general denoising pre- pre-processor applicable to various unseen ASR systems: (2) we deploy a rational coefficient adjustment scheme for the combination strategy and link it to the motivation for better generalization ability.
auxiliary learning method called
2 MOTIVATION AND MAIN IDEA
Noise Robustness Strategies for ASR. Acoustic mismatch, often caused by noise interference, long-standing problem that limits the applicability of ASR systems. To improve recognition performance, multi-condition training (MCT) (Parihar et al.) 2004: Rajnoha 2009 Zhang et al. 2020) is a popular approach when building practical ASR systems. MCT takes noise-corrupted utterances augmented data and jointly trains an acoustic model with clean and noisy utterances. In addition to coise-corrupted utterances, enhanced utterances from SE models can also be included in the training data for MCT training. Recently, PPasad et al. 2021) 1) studied advanced MCT training strategies and conducted a detailed investigation of different ASR fine-tuning techniques, including
freezing partial layers (Zhang a 2020). multi-task learning (Saon al.. 2017 Prasad al.
2021). adversarial learning (Serdyuk et al., 2016: Denisov et 201888 Zhang et al. 2020). and
training involving enhanced utterances from various SE models (Braithwaite & Kleijn 2019 Zhang
ctt al. 2020; Défossez et al. 2020). The results reveal that recognition performance in noise can be remarkably improved by tuning the parameters in the ASR system. However, an ASR system is generally formed by complex structure and estimated with large amount of training data. Also, for most users, ASR systems provided by third parties are black-box models whose acoustic model parameters cannot be fine-tuned to improve performance in specific noisy environments. Therefore, tuning the parameters in an ASR system for each specific test condition is usually not a feasible solution for real-world scenarios.
In contrast, employing an SE unit to process noisy speech signals to generate enhanced signals, which are then sent to the ASR system, is viable alternative for externally improving the recognition accuracy in noise as a two-stage strategy of robustness. The main advantage of this alternative is that SE units can be pre-prepared for direct application to ASR systems without the need to modify these ASR systems or to know the system setups. Furthermore, ASR is usually trained with a large amount of speech-text pairs, requiring expensive annotation costs; on the contrary, for SE, the required large number of noisy-clean training pairs can be easily synthesized. In this study, we propose to utilize the regression objective as a regularization term, combined with the classification objective, for SE unit training to reduce the need for speech-text data and improve the generalization ability.
Regression Objective as an Auxiliary Loss. Since our proposed framework focuses on improving ASR performance in noise, the classification objective is regarded as the main task, and the regression objective is used as an auxiliary loss. Generally, an auxiliary loss aims to improve the performance of the main task without considering the performance of the auxiliary task itself. The auxiliary task learning framework searches for adequate coefficients to incorporate auxiliary information and thus improve the performance of the main task. Several studies (Hu et al. 2019 Chen et al.) 2018 Du et al. 2018 Lin et al. 2019 Dery et al. 2021 Shi et al., 2020) have investigated and confirmed the benefits of auxiliary task learning. In D4AM, given the SE model parameter 8, we consider the classification objective C.,(()) as the main task and the regression objective (re as the auxiliary task. Then, aim jointly fine-tune the SE model with (0) and Lreg (0), where
3

---

Published as a conference paper at ICLR 2023
their corresponding training data are Tels (speech-text paired data) and Treg (noisy-clean paired speech data), respectively. Here, D4AM uses - log P(Tcia P (f) and - log p Tregl (). respectively, as the classification objective La, (0) and the regression objective Lreg (0). Fig. shows the overall training flow of D4AM. Notably, D4AM partially differs from conventional auxiliary task learning algorithms, where the main and auxiliary tasks commonly share the same model parameters. In D4AM, the main task objective Lets (0) cannot be obtained with only the SE model. Instead, proxy acoustic model is employed. That is, the true classification objective (0) ()) is not attainable, and alternatively, with the proxy acoustic model Φ, a surrogate term - p( 10, Φ). termed 2%, (0), is used to update the SE parameters. Since there is no guarantee that reducing C (0) will reduce (eis()), constraint is needed to guide the gradient of C (0) toward improving general recognition performance.
Meanwhile, we believe that, for any particular downstream acoustic recognizer, the best enhanced output produced by the SE unit from any noisy speech input is the corresponding clean speech. Therefore, any critical point of (0) should be "covered" by the critical point space of Leg (0) and we just need guidance from speech-text paired data to determine the critical point in the Legg
space. That is, when an update of SE model parameters decreases both Ca. and Creg that update should be beneficial to general ASR systems. On the other hand, compared to Lets (0).
the calculation of Creg()) (0) does not require the proxy model when we use it to update SE model parameters. Accordingly, we derive the update of 0 at the I-th step as:
0 arg min ola (0) subject to (0) S (0e)
As indicated in Lopez-Paz & Ranzato (2017) and Chaudhry et al. (2019), the model function is supposed to be locally linear around small optimization steps. For ease computation, we replace Greg(0) (0) I (0e with (V (6'), Vregg((') 2 0. That is, we take the alignment between V. (09) and V (0*) as the constraint to guarantee the generalization ability to unseen ASR systems at each training step. When the constraint is violated, V V
ota (6') needs to be calibrated. Following the calibration method proposed in [Chaudhry et al. (2019) to search for the non- non-violating candidate closest to the original V Ca (0*). we formulate the calibrated gradient g" as:
g" = arg min ž و| - v (. subject to (g. Virre 2 O.
(2)
In addition to guiding SE unit training toward improving ASR performance, D4AM also aims to alleviate over-fitting in the classification task. In ARML (Shi et al.] (020), auxiliary task learning was formulated as problem of minimizing the divergence between the surrogate prior and true prior. shown that when a suitable surrogate prior is identified, the over-fitting issue caused by the data limitation of the main task can be effectively alleviated. Following the concept of ARMI,, D4AM aims to estimate an appropriate coefficient Creg to approximate the log prior of the main task objective Cote to alleviate over-fitting, Finally, D4AM combines the calibrated gradient g and auxiliary task learning (ARML) to identify a critical point that ensures better generalization to unseen ASR systems and alleviates over-fitting in the classification task. In Sec. we illustrate the details of our proposed framework.
METHODOLOGY
D4AM aims estimate an SE unit that can effectively improve the recognition performance of general downstream ASR systems. As shown in Fig. Ta the overall objective consists of two parts, the classification objective (f) given the proxy model é and the regression objective (f) As shown in Fig. Tb, we intend guide the update of the model from 0 6, with the regression objective Lreg (0) to improve the performance of ASR in noise. In this section, we present an adjustment scheme to determine a suitable weighting coefficient to combine (0) and Crey (0) instead of grid search process. The adjustment scheme consists of two parts. First, we calibrate the direction of V V. (0) to promote the recognition accuracy, as described in Eq. 2 Second, we follow the ARML algorithm to derive a suitable prior by weighting the regression objective. Finally, the gradient descent of D4AM at the t-th step for SE training is defined as:
g--1 0² < L (0') + (Aigetb + (ssrpr) V V.reeg (0')) )) + mp>
4

---

Published as a conference paper at ICLR 2023
Text Label
Lreg
cls
AM (frozen)
Eats
V.C (f)
SE model /
Ed
Noisy
Synthesizer
Clean
Speech
Speech
(a) Training flowchart
(b) Schematic diagram training trajectory
Figure 1: (a) Training flow of D4AM. Given a proxy acoustic model (AM) parameterized by c 0 is updated by training with both Lits and Freg weighted by a. (b) Illustration of the training trajectories. The dashed line (from 0g to 0%) represents the training process without considering Creg. The solid line (from 0g to 0,) represents the training process considering Frgg.
where é, is the learning rate, and 7/ ~N (0, 26x) is a Gaussian noise used to formulate the sampling of 9. The coefficient O'gclt is responsible for calibrating the gradient direction when C (f*) does not satisfy (V (0'), Vrey (0*) 2 Mathematically. agell the result projecting V C ((') (0*) into the space formed by using V Vrey (0*) (0)), Leeg (0*)) 2 is introduced Section 3.1 The coefficient Osrpe
basis. The connection between the projection process
and the constraint responsible for weighting Lreg (0*) to approximate the log prior of parameters and is estimated to minimize the divergence between the parameter distributions the surrogate and true priors. The determination of Clrppr, the sampling mechanism for 0, and the overall framework proposed in ARML are described in Section 3.2 Finally, we introduce the implementation of the D4AM framework in Section 3
3.1 GRADIENT CALIBRATION FOR IMPROVING GENERAL RECOGNITION ABILITY
This section details the gradient calibration mechanism. First, we assume that any critical point of (0) is located in the critical point space of (O). When the derived g" satisfies (g"", (orr 2 0, the direction of g" can potentially yield recognition improvements to gen- eral acoustic models. The constrained optimization problem described in Eq, / can be solved in a closed form while considering only a mini-batch of samples. We then design criterion C -VLL. (0'), (frr to examine the gradient direction and formulate the solution
g" = v Ca (0') agelh V Lreeg (0'). where Oggolb -CC <0]
(4)
V (0'
where [.] is an indicator function whose output is if the argument condition is satisfied; otherwise, the output O. agehh is derived based on the Lagrangian of the constrained optimization problem. The details of the derivation are presented in Appendix A.7
3.2 REGRESSION OBJECTIVE WEIGHTING AS THE SURROGATE PRIOR
In this section, we discuss the relationship between the classification and regression objectives with the main ASR task. A weighted regression objective derived to approximate the log prior for SE training by modifying the ARML (Shi et al., 2020) algorithm, which considers the weighted sum of auxiliary losses as a surrogate prior and proposes framework for reweighting auxiliary losses based on minimizing the divergence between the parameter distributions of the surrogate and true priors. We aim to identify coefficient (ssppr such that O Ossppr = arg min, Eg [logp" (0) a log(p((reg) (0)) where pz is the true prior distribution of f, and Strppr (Tr is the surrogate prior corresponding to the weighted regression objective -Carpr log(p(TTe 0)). However, p" (0) is not practically available. To overcome the unavailability of (0). ARML uses an alternative solution for p" (0) and a sampling scheme of in p" (#). . For the alternative solution of p" (#), ARML assumes that the main task distribution pl (Tets (0) will cover p" (0) since the samples of 9 from p" (9) are likely to induce high values in p( Taa 9). Thus, ARML takes log TTat ()) as the alternative of log p" (0). For the sampling scheme of 0 in p" (0), for convenience, ARMI acquires the derived parameters from the joint loss
5

---

Published as a conference paper at ICLR 2023
each training step and combines them with Langevin dynamics (Neal al. 2011 Welling & Teh 2011) to approximate the sampling of 9 with an injected noise 17; ~ (O, 2e, Thus, the optimization Log(p(Trrg) 10a) )), where p represents
problem of Csrpr becomes min, Ep- [log(p(Ta...)) the distribution for sampling 0 from the above described joint training process with respect to the original sampling. More details about the ARML algorithm are available in [Shi et. (2020). Finally, at the t-th step, the overall parameter change (g)g) of the auxiliary learning framework given an arbitrary gradient g of the main task (ASR) can be written as:
(g = G + Orrpe Vere (0')) + 1/2-
(5)
we can obtain only VL (0') instead of the ideal ASR task gradient (O'). Therefore, the gradient should be calibrated, as described in Section 3.1 To this end, we update 0 by accordingly taking A0(g"), ), as described in Eq. 3
Nonetheless, the aforementioned problem remains:
Algorithm 1 D4AM (A General Denoising Framework for Downstream Acoustic Models)
Learnable-Parameter:: model parameter 0 (initialized by 89); task weight Oarpr (initialized by 1)
Hyper-Parameter: learning rate of the 1-th iteration et. learning rate of task weight /3 1: for iteration f = 0 to T do
2: O'gcib + project (V C (0*), Virre
3: gi+1 t 8i (V L (d'((')) ) + (Oigetb + Assppr) V Lreg )6') + 1/2
4: 9a Jasepr IV od (8') + (Agelt- Csrpr) Vrreg )12
5: O'srpr f Clerpr
B clamp (ga, maxx = +1, min = 1)
6: end for
3.3 ALGORITIIM
In this section, we detail the implementation of the proposed D4AM framework. The complete algorithm is shown in Algorithm [1] The initial value of Osrpr is set 1. and the learning rate 33 is set to 0.05. We first update 5F by the mechanism presented in Eq. and collect @ samples at each iteration. As described in Section 3.2] Carrpr is optimized to minimize the divergence between p" (0) and patarpr (Tr Referring to ARML (Shi et al. 2020). we use the Fisher di- vergence to measure the difference between the surrogate and true priors. That is. we optimize a by min, Egr [I] (o') a (frre(('))13) Because we can obtain only C (8"), we use the calibrated version g" as the alternative of V Then, the optimization of a becomes min, Egr-p=' [|] Ca proximate the above minimization problem, since it is not feasible collect all 0 samples. Practically, we only have access to one sample of per step. which may cause instability while updating Carpp- Therefore, increase the batch size of 0. we modify the update scheme by accumulating the gradients of Carpr- The update period of Okerpr is set to 16 steps. Unlike ARML (Shi et al. 2020). we do not restrict the value of asrpr to the range 0 1, because C and Lrey have different scales. In addition, we clamp the gradient of Gerpr with the interval [-1, 1] while updating Carpr to increase training stability. Further implementation details are provided in Section 4 and Supplementary Material.
+ (a ax) (11))) In addition, we use gradient descent to ap-
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
The training datasets used in this study include; noise signals from DNS- DNS-Challenge (Reddy et al. 2020) and speech utterances from LibriSpeech (Panayotov et al. 2013). DNS-Challenge, hereinafter DNS. is a public dataset that provides 65,303 background and foreground noise signal samples. LibriSpeech is a well-known speech corpus that includes three training subsets: Libri-100 (28,539 utterances), Libri-360 (104,014 utterances). and Libri-500 (148,688 utterances). The mixing process (of noise signals and clean utterances) to generate noisy-clean paired speech data was conducted during training, and the -ignal-to-noise ratio (SNR) level was uniformly sampled from -4dB to 6dB.
The SE unit was built on DEMUCS (Défossez et al. 2019 2020), which employs a skip-connected encoder-decoder architecture with multi-resolution STFT objective function. DEMUCS has been

---

Published as a conference paper at ICLR 2023
shown to yield state-of-the-an results music separation and SE tasks, We divided the whole training process into two stages: pre- vee-traiing and fine-tuning. For pre-training. we only used Creg to train the SE unit. At this stage. we selected Libri-360 and Libri-500 as the clean speech corpus. The SE unit was trained for 500,000 steps using the Adam optimizer with 31 = 0.9 and /82 = 0.999, learning rate 0.0002, gradient clipping value 1, and batch size 8. For fine-tuning, Frey and Cat. to re-train the SE model initialized by the checkpoint selected from the pre-training stage. We used Libri-100 100 to prepare the speech-text paired data for Lals and all the training subsets (including Libri-100, Libri-360, and Libri-500) to prepare the noisy-clean paired speech data for Creg- The SE unit was trained for 100,000 steps with the Adam optimizer with 3 = 0.9 and 32 = 0.999, learning rate 0.0001, gradient clipping value 1, and batch size 16. For the proxy acoustic model Φ. we adopted the conformer architecture and used the training recipe provided in the SpeechBrain toolkit (Ravanelli et al. 202Tp. The model was trained with all training utterances in LibriSpeech, and the training objective was hybrid version (Watanabe et al. (017) of CTC (Graves et al.. 2006 and NLL (Chan et al. 2016).
used both
Since the main motivation of our study was to appropriately adjust the weights of the training objectives to better collaborate with the moisy-clean/speech-ee
paired data, we focused on comparing different objective combination schemes on the same SE unit. To investigate the effectiveness of each component in D4AM, our comparison included: (1) without any SE processing (termed NOIS), (2) solely trained with Creg in the pre- pre-training stage (termed INIT), (3) solely trained with Ca in the fine-tuning stage (termed CISSO), (4) jointly trained with L and Lreg by solely taking dggeb in the fine-tuning stage (termed GCLB), and (5) jointly trained with Lats and Greg by solely taking G'rrpr in the fine-tuning stage (termed SRPR). Note that some of the above schemes can be viewed as partial comparisons with previous attempts. For example, CLSO is trained only with the classification objective, the same as (Ochiai al. 201 b): SRPR, GCLB, and D4AM use joint training, the
same as (Chen et al. 2013] Ma et al. 2021). but (Chen et al. 2015 Ma et al. 2021) used predefined weighting coefficients. In our ablation study. we demonstrated that our adjustment achieves optimal results without additional training costs compared with grid search. Since our main goal was to prepare an SE unit as a pre-processor to improve the performance of ASR in noise, the word error rate (WER) was used as the main evaluation metric. Lower WER values indicate better performance.
To fairly evaluate the proposed D4AM framework, we used CHiME-4 (Jon et al.) 2017) and Aurora- 4 (Parihar et al. (204)) as the test datasets, which have been widely used for robust ASR evaluation. For CHiME-4, we evaluated the performance on the development and test sets of the 1-st track. For Aurora-4, we evaluated the performance on the noisy parts of the development and test sets. There are two types of recording process, namely wvl and wv2. The results of these two categories are reported separately for comparison. For the downstream ASR systems, tokens by removing all punctuation, mapping numbers to words, and converting all characters to their upper versions. To further analyze our results, we categorized the ASR evaluation into -f-main and in-domain scenarios. In the out-of-domain scenario, we aim to verify the generalization ability of ASR when the training and testing data originate from different corpora. That is, in this scenario, the testing data is unseen to all downstream ASR systems. In the in-domain scenario, we aim to investigate the improvements achievable in standard benchmark settings. Therefore, we set up training and testing data for ASR from the same corpus. Note that ASR systems typically achieve better performance in the in-domain scenarios than in the out-domain scenario (Likhomanenko et al. 2021).
normalized all predicted
4.2 EXPERIMENTAL RESULTS
D4AM Evaluated with Various Unseen Recognizers in the Out-of-Domain Scenario. We first
investigated the D4AM performance with various unseen recognizers. We prepared these recognizers
by the SpeechBrain toolkit with all the training data in LibriSpeech. The recognizers include: (1) conformer architecture trained with the hybrid loss and encoded by BPE-5000 (termed CONF), (2) a transformer architecture trained with the hybrid loss and encoded by BPE-5000 (termed TRAN). (3) an RNN architecture trained with the hybrid loss and encoded by BPE-5000 (termed RNN), and (4) a self-supervised wav2vec2 (Baevski al. 2020) framework trained with the CTC loss and encoded by characters (termed W2V2). To prevent potential biases caused by language models, CONF, TRAN and RNN decoded results only from their attention decoders, whereas W2V2 greedily
7

---

Published as a conference paper at ICLR 2023
Table I: WER (%) results on CHiME-4 and Aurora-4.
CHIME-4
didS
real
dioS-simu
et05 real
e105 simu
CONF
TRAN
RNN
W2V2
CONT
TRAN
RNN
W2V2
CONF
TRAN
RNN
W2v2
TRAN
RNN
NOIS
40.53
34.56
56.76
29.84
45,40
38.86
58.75
35.37
57.23
46.10 72.16 45.61
52.06
44.96
64.93
44.10
INIT
30.98
29.61
38.61
26.93
35.02
32.49
41.76
31.84
40.32
38.77
49.67
39.37
42.78
40.51
49.83
41.04
CLSO
26.66
27.83
35.25
24.63
29.09
29.89
38.00
29.16
34.00
35.19
46,02
36.08
35,02
36.62
45.66
37,23
SRPR
26.88
26.73
34.83
24.51
30.12
29.22
38.73
29,86
86
34,53
33,93
46.08
35,21
36.68
36.55
46.27
38,80
GCLB
26.76
26.51
23.55
28.73
37.55
34.18
33.76
45.75
35.15
35.69
36.09
45.37
37.18
D4AM
26.70
26.07
34.43
23.30
29.45
28.61
37.49
28.40
33.71
33.20
45.07
34.43
35.97
35.75
45.36
37.30
Aurora-4
del-wel
dev-wv2
test-wel
test-wv2
CONF
TRAN
RNN
W2v2
CONF
TRAN
RNN
W2v2
CONF
TRAN
RNN
W2V2
CONF
TRAN
RNN
W2V2
NOIS
21.69
19.66
27.36
14.44
31.56
27.96
42.12
25.47
19.27
17.64
23.98
13.60
29.77
27,54
38.91
26.37
INIT
18.97
18.24
21.71
13.59
27.85
26.16
34.58
24.38
16.76
16,25
18.45
12.26
26.55
25.23
31,98
24.65
CL.SO
17.75
18.23
20.98
13.31
24.76
25.05
33.08
23,48
15.43
17.03
17.53
12.16
24.24
25.85
32.33
25.79
SRPR
17.77
17.20
20.40
12.97
24.67
23.20
31.66
21.57
15.52
15,50
16.82
11.89
23.83
23.00
29.74
22.95
GCLB
17,70
17.16
20.35
12.91
24.65
22.99
31.73
21.92
15.51
15.66
16.82
12
24.51
23.19
30.39
23.36
D4AM
17.73
17.08
20.30
12.84
24.31
22.78
31.06
20.90
15.55
15.44
17.18
11.66
23.35
22.68
29.65
22.02
Table 2: WER (%) results of CHiME-4's baseline ASR system.
method
NOIS
INIT
CLSO
SRPR
GCLB
D4AM
dt05-real
11.57
10.28
7.95
8.28
8.15
7.94
d105-simu
12.98
11.80
10.07
10.28
9.56
9.48
et05-real
23.71
19.91
16.28
16.33
16.32
15.66
et05-simu
20.85
20.39
18.37
18.18
16.86
16.31
decoded results following the CTC rule. Table I shows the WER results under different settings. Note that we cascaded CONF to train our SE model, so CONF is not a completely unseen recognizer in our evaluation. Therefore, we marked the results of CONF in gray and paid more attention to the results of other acoustic models. Obviously, NOIS yielded the worst results under all recognition and test conditions. INIT performed worse than others because it did not consider the ASR objective in the pre-training stage. CLSO was worse than GCLB, SRPR, and D4AM, verifying that training with c alone does not generalize well to other unseen recognizers. Finally, D4AM achieved the best performance on average, confirming the effectiveness of gradient calibration and using regression objective weighting as a surrogate prior in improving general recognition ability.
D4AM Evaluated with a Standard ASR System in the In-Domain Scenario. In this experiment, we selected the official baseline ASR system from CHiME-4, a DNN-HMM ASR system trained in an MCT manner using the CHiME-4 specifce training data, as the evaluation recognizer. This baseline ASR system is the most widely used to evaluate the performance of SE units. As observed in Ta the result of INIT is comparable to the state-of-the-art results of CHiME-4. Meanwhile,
D4AM achieved notable reductions in WER compared with NOIS. This trend is similar to that observed in Table and again validates the effectiveness of D4AM.
D4AM Evaluated with a Practical ASR API. We further evaluated D4AM using the Google ASR API, well-known black- system for users. To prepare a "real" noisy testing dataset, merged the real parts of the CHiME-4 development and test sets. Table shows the average WER values obtained by the Google ASR API. The table reveals trends similar to those observed in Table T but two aspects are particularly noteworthy. First, NOIS still yielded the worst result, confirming that all SE units contribute to the recognition accuracy of the Google ASR API on our testing data. Second, CLSO was worse than INIT, confirming that an SE unit trained only by the classification
Table 3: WER (%) results of the Google ASR API on the CHiME-4 real sets.
method
NOIS
INIT
CLSSO
SRPR
GCLB
D4AM
WER (%)
35.09
28.02
29.26
26.79
26.54
26.44
8

---

Published as a conference paper at ICLR 2023
objective according to ASR model is not guaranteed to contribute other unseen recognizers. In contrast, improved recognition results (confirmed by the improvements produced by SRPR, GCLB, and D4AM) can be obtained by well considering the combination of objectives.
4.3 ABLATION STUDIES
Regression Objective Weighting for Handling Over-fitting, As mentioned earlier, ARML ad- dresses the over-fitting issue caused by the limited labeled data of the main task by introducing an appropriate auxiliary loss. For D4AM, we considered the regression objective as an auxiliary loss to alleviate the over-fitting issue. To verify this feature, we reduced the speech-texi paired data to only 10% (2853 utterances) of the Libri-100 dataset. The WER learning curves of CLSO and D4AM on privately mixed validation set within 100,000 training steps are shown in Fig. 2a The figure illustrates that the learning curve of CLSO ascends after 50,000 training steps, which is a clear indication of over-fitting. In contrast, D4AM displays a different trend. Fig. confirms the effectiveness using the regression objective to improve generalization and mitigate over-fitting.
gnd count D43
265
20.03
SRI'S
20
We- ome
Word
285
Trwniin Iepg
11eae of acrA
(a) Learning curves.
(b) WER (%) results of grid search, SRPR, and D4AM.
Figure 2: (a) Learning curves of CLSO and D4AM trained with 10% transcribed data of Libri-100. (b) Comparison of WER (%) results of grid search, SRPR, and D4AM. The dashed line represents the average performance of the best seven results of grid search.
D4AM versus Grid Search. One important feature of D4AM is that automatically and effectively determines the weighting coefficients to combine regression and classification objectives. In this section, we intend to compare this feature with the common grid search. Fig. 2b compares the performance of D4AM, SRPR, and grid search with different weights the CHiME-4 dt05-real set. For the grid search method, the figure shows the results for different weights, including O, 0.1, 1.0, 10, 20, 30, 40, 50, and 60. The average performance of the best seven results is represented by the dashed green line. From Fig. 2bb we first note that D4AM evidently outperformed SRPR. Furthermore, the result of D4AM is better than most average results of grid search. The results confirm that D4AM can efficiently and effectively estimate coefficients without an expensive grid search process.
To further investigate its effectiveness, we tested D4AM on high SNR testing conditions, MCT- trained ASR models, and another acoustic model setup. Additionally, we tested human perception assessments using the D4AM-enhanced speech signals. All the experimental results confirm the effectiveness of D4AM. Owing to space limitations, these results are reported in APPENDIX.
5 CONCLUSION
In this paper, we proposed a denoising framework called D4AM, which aims to function as a pre- processor for various unseen ASR systems boost their performance in noisy environments. D4AM employs gradient calibration and regression objective weighting to guide SE training to improve ASR performance while preventing over-fitting The experimental results first verify that D4AM can effectively reduce WERs for various unseen recognizers, even practical ASR application (Google API). Meanwhile, D4AM can alleviate the over-fitting issue by effectively and efficiently combining classification and regression objectives without the exhaustive coefficient search. The promising results confirm that separately training an SE unit to serve existing ASR applications may be a feasible solution to improve the performance robustness of ASR in noise.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The source code for additional details the model architecture, training process, and pre pre-processing steps, as well as demo page, are provided in the supplemental material. Additional experiments are provided in Appendix.
ACKNOWLEDGEMENT
This work was partly supported by the National Science and Technology Council, Taiwan (NSTC I11-2634-F-002-0223)
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-super learning of speech representations. In Proc. NeurIPS, 2020.
D.T. Braithwaite and W. Bastiaan Kleijn. Speech enhancement with variance constrained autoen- coders. In Proc. Interspeech, 2019.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In Proc. ICASSP, 2016.
Fu-An Chao, Shao- Wei Fan-Jiang, Bi-Cheng Yan, Jeih-Wei Hung. and Berlin Chen. Tenet: time-reversal enhancement network for noise-robust asr. arXiv preprint arXiv:2107.0153/ 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In Pruc. ICLR, 2019.
Zhao Chen, Vijay Badrinarayanan, Chen Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proc. ICML, 2018.
Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John Hershey. Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In Proc. Interspeech, 2015.
Chung-Cheng Chiu, Tara Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski. and Michiel Bacchiani. State-of-the-ar speech recognition with sequence-to -o-suuencce models. In Proc. ICASSP, 2018
Xiaodong Cui, Songtao Lu, and Brian Kingsbury. Federated acoustic modeling for automatic speech recognition. Proc. ICASSP, 2021.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.1325. 2019.
Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Proc. Interspeech, 2020.
Pavel Denisov, Ngoc Thang Vu, and Mare Ferras. Unsupervised domain adaptation by adversarial learning for robust speech recognition. In ITG Symposium on Speech Communication, 2018.
Lucio M. Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral. In Proc. ICLR, 2021.
Yunshu Du, Wojeiech M Czamecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: 022224 2018.
Zhihao Du, Xueliang Zhang, and Jiqing Han. Investigation of monaural front-end processing for robust speech recognition without retraining or joint-training. In Proc. APSIPA ASC, 2019.
10

---

Published as a conference paper at ICLR 2023
Y. Ephraim and D. Malah. Speech enhancement using minimum square enror short-time spectral amplitude estimator. JEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1109-1121, 1984.
S. Fu, T. Wang. Y. Tsao, X. Lu, and H. Kawai. End-to waveform utterance enhancement for direct evaluation metries optimization by fully convolutional neural networks. /EEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1570-1584, 2018.
Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In Proc. ICML, 2019.
Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang lu, and Yu Tsao. MetricGAN+: An improved version MetricGAN for speech enhancement. Proc. Interspeech, 2021.
Philip Gage, A new algorithm for data compression. Users Journal, 12(2):23-38, 1994.
Jürgen T Geiger, Jort Gemmeke, Björn Schuller, and Gerhard Rigoll. Investigating NMF speech enhancement for neural network based acoustic models. In Proc. Interspeech, 2014a.
Jürgen T. Geiger, Felix Weninger, Jort F. Gemmeke, Martin Wöllmer, Björn Schuller, and Gerhard Rigoll. Memory-cnhanced neural networks and NMF for robust asr. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1037-1046, 2014b.
François Germain, Qifeng Chen, and Vladlen Koltun. Speech denoising with deep feature losses. In Proc. Interspeech, 2019.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. ICMI., 2006.
Eric Guizzo, Christian Marinoni, Marco Pennese, Xinlei Ren, Xiguang Zheng, Chen Zhang, Bruno Masiero, Aurelio Uncini, and Danilo Comminiello. 1.3das22 challenge: Learning 3d audio sources in a real office environment. In Proc. ICASSP, 2022.
Anmol Gulati, Chung-Chern Chiu, James Qin, Jiahui Yu, Niki Parmar, Ruoming Pang, Shibo Wang, Wei Han, Yonghui Wu, Yu Zhang, and Zhengdong Zhang. Conformer: Convolution-augmentte transformer for speech recognition. In Proc. Interspeech, 2020.
Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Álvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo yiin Chang, Kanishka Rao, and Alexander Gruenstein. Streaming end-to-end speech recognition for mobile devices. Proc. ICASSP, 2019.
Guoning Hu and Del.iang Wang- A tandem algorithm for pitch estimation and voiced speech segregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 2067-2079, 2010.
Hanzhang Hu, Dehadeepta Dey, Martial Hebert, and Andrew Bagnell. Learning anytime predictions in neural networks via adaptive loss balancing. In Proc. AAAI, 2019.
Jensen and C. Taal. An algorithm for predicting the intelligibillit of speech masked by modulated noise maskers. IEEE/ACM Transactions Audio, Speech, and Language Processing, pp. 20-2000002222 2016.
Barker Jon, Marxer Ricard, Vincent Emmanuel, and Watanabe Shinji. The third 'CHiME' speech separation and recognition challenge: Analysis and outcomes. Computer Speech Language, pp. 605-626, 2017.
Keisuke Kinoshita, Tsubasa Ochiai, Mare Delcroix, and Tomohiro Nakatani. Improving noise robust automatic speech recognition with single-channel time-domain enhancement network. In Proc. ICASSP, 2020.
11

---

Published as a conference paper at ICLR 2023
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. SDR-half-bake well done? In Proc. ICASSP, 2019.
Chi-Chang Lee, Yu-Chen Lin, Hsuan-Tien Lin, Hsin-Min Wang, and Yu Tsao. SERIL: noise adaptive speech enhancement using regularization-bent bbased incremental learning. Proc. Interspeech, 2020.
Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umee Umbach. An overview of noise-robust automatic speech recognition. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, pp- 745-777, 2014.
Tatiana Likhomanenko. Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking Evaluation in ASR: Are Our Models Robust Enough? In Proc. Interspeech, 2021.
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In Proc. NeurIPS, 2019.
Philipos C. Loizou. Speech Enhancement: Theory and Practice, CRC Press, Inc., USA, 2nd edition, 2013. ISBN 1466504218.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proc. NeurIPS, 2017.
Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based on deep
denoising autoencoder. In Proc. Interspeech, 2013.
Duo Ma, Nana Hou, Van Tung Pham, Haihua Xu, and Eng Siong Chng. Multitask-based joint learning approach to robust asr for radio communication speech. In Proc. APSIPA ASC, 2021.
Zhong Meng, Shinji Watanabe, John R. Hershey, and Hakan Erdogan. Deep long short-term memory adaptive beamforming networks for multichannel robust speech recognition. In Proc. ICASSP, 2017.
Zhong Meng, Jinyu Li, Yifan Gong, and Swang- (Fred) Juang. enhancement. In Proc. Interspeech, 2018
Cycle-consistent speech
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Mohammad Nikzad, Aaron Nicolson, Yongsheng Gao, Jun Zhou, Kuldip K. Paliwal, and Fanhua Shang. Deep residual-dense lattice network for speech enhancement. In Proc. AAAI, 2020.
Tsubasa Ochiai, Shinji Watanabe, Takaaki Hori, and John R. Hershey. Multichannel end-to-end speech recognition. In Proc. ICML, 2017a.
Tsubasa Ochiai, Shinji Watanabe, and Shigeru Katagiri. Does speech enhancement work with end-to-end asr objectives?: Experimental analysis of multichannel end-to-end asr. In Proc. IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), 2017b.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. ICASSP, 2015.
Ashutosh Pandey and Deliang Wang. A new framework for supervised speech enhancement in the time domain. In Proc. Interspeech, 2018.
N. Parihar, J. Picone. D. Pearce, and H. G. Hirsch. Performance analysis of the Aurora large vocabulary baseline system. In Proc. The 12th European Signal Processing Conference, 2004.
Archiki Prasad, Preethi Jyothi, and Rajbabu Velmurugan. An investigation of end-to-end models for robust speech recognition. In Proc. ICASSP, 2021.
Josef Rajnoha. Multi-condition training for unknown environment adaptation in robust ASR under real conditions. 2009.
12

---

Published as a conference paper at ICLR 2023
Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with mnntrrnnddueer. In Proc. ASRU, 2017.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung- Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. Speechbrain: A general-purpose speech toolkit. arXiv preprint ww..iv:2106.046660 2021.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In Proc. Interspeech, 2020.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. Proc. ICASSP, 2021.
A. W. Rix, J. G. Beerends, M. P. Hollier, and P. Hekstra. Perceptual evaluation of speech quality (P new method for speech quality assessment of telephone networks and codecs. In Proc. ICASSP. 2001.
Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, et al. A streaming on-device end-to-end model surpassing server-side conventional model quality and latency. Proc. ICASSP, 2020.
George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi, and Phil Hall. English conversational telephone speech recognition by humans and machines. Proc. Interspeech, 2017.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In Proc. ICASSP, 2012.
Michael L. Seltzer, Dong Yu, and Yongqiang Wang. An investigation of deep neural networks for noise robust speech recognition. In Proc. ICASSP, 2013.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. ACL, 2016.
Dmitriy Serdyuk, Kartik Audhkhasi, Philémon Brakel, Bhuvana Ramabhadran, Samuel Thomas, and Yoshua Bengio. Invariant representations for noisy speech recognition. arXiv preprint arXiv: 1612.01928, 2016.
Baifeng Shi, Judy Hoffman, Kate Saenko, Trevor Darrell, and Huijuan Xu. Auxiliary task reweighting for minimum-data learning. In Proc. NeurlPS, 2020.
Jiaqi Su, Zeyu Jin, and Adam Finkelstein. HiFi-GAN:
High-fidelity denoising and dereverberation based on speech deep features adversarial networks. Proc. Interspeech, 2020.
C. H. Taal, R. C. Hendriks, R. Heusdens, and Jensen. A short-time objective intelligibility measure for time-freyuency weighted noisy speech. In Proc. ICASSP. 2010.
Andrew Varga and Herman J. M. Steeneken. Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment study the effect of additive noise on speech recognition systems. Speech Communication, pp. 247-251, 1993.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proc. NeurIPS, 2017.
Peidong Wang, Ke Tan, and De Liang Wang. Bridging the gap between monaural speech enhancement and recognition with distortion-indeeeedent acoustic modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 39 48, 2020.
S. Wang, W. Li, S. M. Siniscalchi, and C. Lee. A cross-task transfer learning approach to adapting deep speech enhancement models to unseen background noise using paired senone classifiers, In Proc. ICASSP, 2020.
13

---

Published as a conference paper at ICLR 2023
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, pp. 1240-1253, 2017.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proc. ICML, 2011.
Felix Weninger, Hakan Erdogan, Shinji Watanabe, Emmanuel Vincent, Jonathan Le Roux, John R. Hershey, and Björ Schuller. Speech enbancement with Istm recurrent neural networks and its application noise-robust asr. In International Conference on Lalent Variable Analysis and Signal Separation, pp. 91-99, 2015a.
Felix Weninger, Hakan Erdogan, Shinji Watanabe, Emmanuel Vincent, Jonathan Le Roux, John R Hershey, and Björn Schuller. Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ast. In International Conference on Latent Variable Analysis and Signal Separation, 2015b.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Proc. NeurIPS, 2020.
Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng. Listening to sounds of silence for speech denoising. In Proc. NeurIPS, 2020.
Y. Xu, J. Du, L. Dai, and C. Lee. A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2015.
Dacheng Yin, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. PHASEN: A pharonind-harmonics- aware speech enhancement network. In Proc. AAAI, 2020.
Huanjing Yue, Wenxin Duo, and Jingyu Yang. Reference-basee speech enhancement via feature alignment and fusion network. Proc. AAAI, 2022.
Q. Zhang, A. Nicolson, M. Wang, K. K. Paliwal, and C. Wang. DeepMMSE: A deep learning approach MMSE-basedd noise power spectral density estimation. JEEF/ACM Transactions on Audio, Speech, and Language Processing, pp. 1404-1415, 2020.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss. In Proc. ICASSP, 2020.
Zhang, C. T. Do, Doddipatla, and Renals. Learning noise invariant features through transfer learning for robust end-to-end speech recognition. In Proc. ICASSP, 2020.
Shucong Zhang, Cong-Thani Do, Rama Doddipatla, and Steve Renals. Learning noise invariant features through transfer learning for robust end-to-end speech recognition. In Proc. ICASSP, 2020.
Chengyu Zheng, Xiulian Peng, Yuan Zhang, Sriram Srinivasan, and Yan Lu. Interactive speech and noise modeling for speech enhancement. In Proc. AAAI, 2021.
A APPENDIX
A.1 I D4AM EVALUATED ON HIGH SNR DATA
Most SE research focuses on "average" denoising on the target ASR task, which implies that we consider the distortion caused by the SE model as tolerable if is sufficiently low, Accordingly, we conducted additional experiments using cleaner speech utterances testing data. The WERs of the Google ASR API are shown in Table where test-clean represents the clean LibriSpeech test set, and the others are test-clean mixed with noisex92 (Varga & Steeneken 1993) and Nonspeech (Hu & Wang 2010)) noise sources, uniformly sampled at high SNR levels between 10dB to 20dB (average 15 dB). From Table illustrates that the WER of the clean speech processed by D4AM is marginally
14

---

Published as a conference paper at ICLR 2023
higher than that of the original clean speech (ORIG). The result is 16.61% versus 16.27%. However, for both types of high SNR noisy speech, using D4AM reduced the WER (19.24% versus 19.67% for noisex92 (Varga & Steeneken 1993) and 20.19% versus 20.55% for Nonspeech (Hu & Wang 2010)). The results show that D4AM can yield performance comparable (better) to that achieved using the original clean (high SNR noisy) speech.
Table 4: WER (%) results on high SNR test sets.
method
test-clean
test-noisex
test-nonspeech
ORIG
16.27
19.67
20.55
D4AM
16.61
19.24
20.19
A.2 D4AM EVALUATED WITH MCT-TRAINED ASR MODELS
The results of the Google ASR API in Table 3 are supposed to be representative results of the MCT- trained ASR systems. We conducted additional ASR experiments CHiME-4 with MCT-CONE, MCT-W2V2, and MCT-TRAN corresponding to the ASR model architectures in Table and trained on the LibriSpeech and DNS noise datasets with the SNR levels -4 to 6. Note that W2V2 consists of a wav2vec2 module and a linear layer. During training, the wav2vec2 module was fixed and used as feature extractor, and only the linear layer was updated. It is evident from Table 5 that D4AM consistently outperformed NOIS for all MCT-trained ASR models.
Table 5: WER (%) results of MCT-trained ASR models on the CHiME-4 dt-05 real set.
method
MCT-CONF
MCT-W2V2
MCT-TRAN
NOIS
25.69
28,86
22.82
D4AM
24.22
22.89
22.59
A.3 D4AM TRAINED WITH ANOTHER ACOUSTIC MODEL
We selected the conformer-based ASR model the proxy acoustic model because it has lower memory requirement than other model architectures in SpeechBrain. In terms of implementation. model complexity a key concem, particularly for researchers similar us with limited computing resources. We believe our claim that any critical point of (0) should be "covered" by the critical most cases. By considering this appropriately, we can reduce the bias of different settings of the proxy acoustic model used for training. In addition, previous work (Ochiai et al. 2017a b) has shown empirically that training an SE unit with one acoustic model can improve another ASR system. To show that the proposed D4AM is insensitive to the selection of proxy acoustic model, we conducted additional experiments using the W2V2-CTI CTC model as the proxy acoustic model. The results evaluated with LibriSpeech-train trained ASR system on the CHiME-4 dt-05 real set are shown in Table where Proxy-CONF and Proxy- www--2222---T denote conformer- and W2V2-CTC-based proxy acoustic models, respectively.
point space of Lrey (0) applies
As shown in T D4AM can effectively reduce the bias between proxy acoustic models and no- tably improve the performance. Since Conformer and W2V2-CTC have different model architectures and use different label types, the results confirm that D4AM can consistently produce promising results using different types of proxy acoustic models.
A.4 D4AM EVALUATED WITH THE L3DAS22 MICRO-A SETS
The input of L3DAS22 Guizzo et al. 2022 is multi-channel signal. In our experiments, we focused on the single-channel SE task and considered different channels as separate signals for training and testing. Also, please note that the WER definition of L3DAS22 differs from the regular version. In L3DAS22, the reference target for computing WER is NOT true transcription. The WER is the word-level distance between the ASR model's predicted transcripts for the clean and enhanced
15

---

Published as a conference paper at ICLR 2023
Table 6: WER (%) results of different setups on the CHiME-4 dt-05 real
method
CONF
TRAN
RNN
Proxy-CONF-CLSO
26.66
27.83
35.25
Proxy-CONF-SRRR
26.88
26.73
34.83
Proxy-CONF-GCLB
26.76
27.51
34.62
Prox--CONF-D4AM
26.7
26.07
34.43
Proxy-W2Y-CCTC-CLSS
30.61
28.98
40.30
Proxy-W2Y2-CTC-SRPR
29.09
27.57
36.67
Proxy-W2V2-CTC-GCLBB
29.25
27.91
36.92
Proxy-22V-CCT--DAAM
28.46
26.89
35.33
INIT
30.98
29.61
38.61
signals. Furthermore, the WER value is calculated sample basis. In contrast, the conventional WER value calculated using ground truth transcripts as targets and derived by summing the emror across all testing samples.
By training our SE model with their pre-mixed training data, we executed our algorithm on the set with the following results:
Table 7: WER results on the L3DAS22 micro-A sets.
L3DAS22-dev-microu
L3DAS22-tcxt-micrro
method
TRAN
RNN
W2V2
TRAN
RNN
W2V2
NOISY
31.14
60.80
38.85
27.59
57.98
35.70
INIT
22.93
35.65
28.00
21.94
33.72
26.67
CLSO
19.31
32.88
24.34
18.35
30.99
22.78
SRPR
19.03
31.82
23.68
18.08
29.95
22.54
GCLB
19.40
32.07
23.75
18.50
30.17
22.55
D4AM
18.68
31.61
23.59
17.88
29.71
22.19
A.5 HUMAN PERCEPTION EVALUATION
Finally, we evaluated the enhanced speech signal using objective perceptual metric, DNS- MOS (Reddy et al. 2021). The DNSMOS value represents the perception score of human hearing, with higher scores indicating better quality. Tislossshows the DNSMOS scores of enhanced speech produced by various SE methods. The real sets of CHiME-4 described in Section 4.2 were used as the testing data. rebeals that D4AM produced the highest DNSMOS score among all models trained with the ASR objective, and its DNSMOS score is marginally lower than that of INIT (only considering the regression objective for training). The result again substantiates our conjecture that any critical point of (0) should be "covered" by the critical point space of (0)
Table 8: DNSMOS results the CHiME-4 real sets.
method
NOIS
INIT
CLSO
SRPR
GCLB
D4AM
DNSMOS
2.5574
3.2401
2.7665
3.0598
3.0398
3.1450
16

---

Published as a conference paper at ICLR 2023
A.6 VARIATION IN Carpr DURING TRAINING
Fig. 3 shows the variation in aarp with and without gradient calibration applied over training steps. It is evident that the Osrypr values of both SRPR and D4AM converge rapidly, indicating that their learning process is stable.
Value alpha
D4AM
D4AM avg
SRPR
SRPR
20000
40000
60000
80000
Training
Figure 3: The change in the Carper value during training. The dashed lines represent the average Carpr values.
A.7 DERIVATION OF Oggelb
To derive d'gcib, the Lagrangian of the constrained optimization problem defined Eq. can be written as:
1 711o VC (09) a(g, VLreg
E(g. a)
(6)
To determine the optimal value g" arg min, C(g. a). we set the derivatives L(g, a:) with respect to O. By solving for V vgC(a,g) g) O, obtain g v c (0') + a Vireg (0*). Then, we can substitute g = v V (0') + a Lreg (0*) into L(a, and get the minimum C* (a)):
1 L (0') 1| const
(a) - (0'), v Creg (0')
2
where const is the summation of those regardless of a. Here, the solution of O'getb is the critical point of L (a).. Therefore, by deriving a such that (.L*(a) = 0, we can obtain the final result of Ogelb:
L V Lreg (0))
Qtyclb = -IVV v (09) < 0O
(8)
V Lreg |1
17


Reviews:
[{"review_id": "tG5FkN-3dKK", "reviewer": "Reviewer_jXQS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a denoising approach for speech enhancement for use in downstream tasks such as automatic speech recognition (ASR). They augment the typical end-to-end training with an auxiliary loss function that also seeks to minimize distortion with the notion of ASR-independent generalization. The results have been evaluated on 2 noisy ASR evaluation datasets and show a small improvement over a couple of previous systems.", "summary_of_the_review": "The paper builds on a host of related work and shows small improvements on medium vocabulary ASR tasks and is computationally more efficient than a grid-search-based approach. Without LVCSR or large-scale MCT comparisons though, the power of the approach might need to come from its applicability to other tasks, e.g., intelligibility improvements of enhanced speech.", "strength_and_weaknesses": "Strengths\n1.Good literature review.\n2. Insightful connection to related efforts in Section 3. \n3. Evaluation across a variety of recognizers and datasets.\n4. Helpful wav and spectrogram files in the supplementary section.\n5. The attached source code promotes reproducibility.\n6. The writing is very clear.\n\nGrowth opportunities\n1. The authors claim that \"tuning parameters in an ASR system is not a feasible solution for practical scenarios.\" With large-scale multi-condition training, it is not clear how much tuning is necessary.\n2. The annotation cost comment on MCT is misleading. Typically, only the clean speech is annotated, incurring no additional cost for MCT.\n3. What is the justification for the learning rate and update period values used? How sensitive is the solution to these parameters?\n4. Why is the SNR range being investigated in the range from -4dB to 6dB?\n5. CHIME-4 and Aurora-4 are medium vocabulary ASR tasks. It would be beneficial to evaluate the proposed approach on LVCSR tasks.\n6. Only small improvement over GCLB is observed. On the Aurora-4 and Google ASR API on CHIME-4 tasks, even SRPR performs competitively.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gmBk7fiYIFa", "reviewer": "Reviewer_DCcQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a denoising frontend for automatic speech recognition. The method combines speech enhancement (regression) and recognition (classification) loss. The paper presents a scheme for automatically balancing the regression and classification losses. The paper shows the effectiveness of the proposed method in the efficacy of noisy speech recognition experiments.", "summary_of_the_review": "The paper does not have sufficient novelty in terms of the technical/algorithm part. Various researchers have already studied the joint training of both enhancement and recognition losses (as shown in Section 2). The automatic balancing scheme seems novel, but the effectiveness compared with the grid search is marginal.\n\nOther suggestions\n- Please discuss the reverberation cases. This is very critical for actual ASR deployment.\n- Can we use this method without the regression loss? In the practical scenario (real data situations), we cannot prepare the clean data, and we cannot construct the regression loss. I want the authors to discuss such cases and how the proposed method works.\n- Abstract: It is better to provide more concrete descriptions (e.g., with some numbers) to claim the effectiveness of the proposed method experimentally. \n- Can you discuss whether the proposed automatic balancing scheme more generally applies to the other machine learning problems? I think such a discussion makes the potential effectiveness of this scheme strong.\n- The introduction (section 1) and motivation (section 2) have some similar logical flows, and this part can be refined.\n- Why did you use DEMUCS? Can you add some references about the state-of-the-art performance of DEMUCS in some SE tasks?\n- Table 1. Please use the same decimal place for all numbers (2nd decimal place).\n- Section 4.2 \"As seen in Table 2, the result of INIT is comparable to the state-of-the-art results of CHiME-4\" Can you add a reference? Which system are you comparing?\n\n\n\n\n", "strength_and_weaknesses": "Strength\n- consistent improvements of the ASR performance even with a black box ASR API. The problem setup of developing universal speech enhancement techniques for various ASR systems is crucial. \n- automatic balancing scheme of regression and classification losses seem to be novel\n- showing the robustness of the balancing scheme experimentally.\n\nWeaknesses\n- optimization of speech enhancement with ASR loss is not novel. Of course, a part of dealing with unseen ASR systems is novel, but it is not a sufficient novelty.\n- the method is specific to speech processing problems and would not attract general machine learning and AI researchers. The automatic balancing scheme seems to be a general scheme, and if this has more critical or different applications, this part will be improved more.\n- Although the balancing scheme is sophisticated, the result is not very different from a conventional grid search. It is not difficult to tune a single hyper-parameter with the dev set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ewS8mij35uk", "reviewer": "Reviewer_mD2F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for Speech Enhancement (SE) applied to Automatic Speech Recognition (ASR). In the proposed approach, the SE model is trained to enhance noisy speech and to keep improving recognition performance of the acoustic model. This is accomplished by training the SE model with multi-task learning, where one of the losses is an estimation of the ASR performance, together with an adjustment scheme to learn the weights directly. The approach is evaluated on several noisy ASR benchmarks, including an unseen ASR system and is shown to improve peformance.", "summary_of_the_review": "Overall, the approach seems novel and the findings are significant. The domain is very specific (using SE for ASR) and might not be of huge interest to the whole ICLR community, but the algorithm with the adjustment could be useful and is worth publishing. I thus recommend to accepting the paper. ", "strength_and_weaknesses": "Strengths:\n - The approach presented in the paper is significant as it is designed for unseen ASR systems and yields state-ot-the-art performance.\n - The adjustment scheme is very interesting and makes the approach very practical and appealing to use.\n - The experiments are thorough and clearly show the viability of the proposed approach.\n\nWeaknesses:\n - The paper doesn't situate well the proposed approach with the literature: in the third paragraph of Section 2, several related works are cited which seems to be very similar to the proposed approach as they all use the auxiliary task approach, but it's unclear which tasks and how it is relating to the proposed approach. Please describe the relevant related works and clarify.\n- The clarity of the paper could be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster