Generative approach using the noise generation
models for DNN-based speech synthesis trained
from noisy speech
Masakazu Une∗† , Yuki Saito† , Shinnosuke Takamichi† , Daichi Kitamura† ,Ryoichi Miyazaki∗ and Hiroshi Saruwatari†∗ National Institute of Technology, Tokuyama College, Gakuendai Shunan, Yamaguchi 745–8585, Japan

E-mail: {i12une, miyazaki}@tokuyama.ac.jp† Graduate School of Information Science and Technology, The University of Tokyo,

7-3-1 Hongo Bunkyo-ku, Tokyo 133–8656, Japan
E-mail: {yuuki saito, shinnosuke takamichi, daichi kitamura, hiroshi saruwatari}@ipc.i.u-tokyo.ac.jp

Abstract—This paper proposes a generative approach to construct high-quality speech synthesis from noisy speech. Studioquality recorded speech is required to construct high-quality
speech synthesis, but most of existing speech has been recorded ina noisy environment. A common method to use noisy speech fortraining speech synthesis models is reducing the noise before thevocoder-based parameterization. However, such multi-step processes cause an accumulation of spectral distortion. Meanwhile,
statistical parametric speech synthesis (SPSS) without vocoders,which directly generates spectral parameters or waveforms, hasbeen proposed recently. The vocoder-free SPSS will enable us totrain speech synthesis models considering the noise addition process generally used in signal processing research. In the proposed
approach, newly introduced noise generation models trained bya generative adversarial training algorithm randomly generates
spectra of the noise. The speech synthesis models are trained tomake the sum of theirs output and the randomly generated noise
close to spectra of noisy speech. Because the noise generation
model parameters fit the spectrum of the observed noise, theproposed method can alleviate the spectral distortion found in
the conventional method. Experimental results demonstrate thatthe proposed method outperforms the conventional method interms of synthetic speech quality.

I. I NTRODUCTION
Statistical parametric speech synthesis [1] is a techniquethat synthesizes natural-sounding speech from the text using
statistical models. A variety of techniques has been proposed
to improve speech naturalness [2], [3], [4]. Deep neuralnetwork (DNN)-based speech synthesis [5] has significantly
improved speech quality. In order to construct high-quality
speech synthesis, studio-quality speech data is required. However, most of the existing speech data has been recorded ina noisy environment [6]. Besides training using such poorquality speech is a challenging task in DNN-based speechsynthesis. There are several kinds of noisy environments, suchas narrow frequency bands [7] and poor-quality of communications network [8]. In our research, we focused on speechrecorded in a more stationary noise environment such as in ahouse [9].
The common approach is to perform noise reduction prior
to speech synthesis training (as shown in the upper graph Fig.

1. Noise reduction for speech synthesis is different from thatfor speech recognition because it needs the vocoder parameters (e.g., mel-cepstral coefficients of STRAIGHT [10] andWORLD [11]) to train the acoustic models (speech synthesis
models)1 . Noise reduction for speech synthesis can be classified into two types: 1) direct estimation of vocoder parametersfrom noisy speech [12] and 2) vocoder-based parameterization after noise reduction. The former trains the statisticalmodels in advance to identify the vocoder parameters fromthe noisy speech parameters. This method can perform anonlinear conversion using the DNNs but its resistance to theunseen 2 noise is not guaranteed. The latter type can adopt a
signal-processing-based unsupervised noise reduction such as
spectral subtraction [13]. It works even when the unseen noise
is observed, but the quality of vocoder-based parameterization
after the noise-suppressed speech is not guaranteed. Also, thecommon problem in both types is that the spectral distortion
caused by the DNN-based or signal-processing-based advanceestimation is accumulated in the speech synthesis training. In
our research, we focused on constructing vocoder-free DNNbased speech synthesis from noisy speech [5], [14]. It is aframework that directly estimates the spectral parameters or
waveforms, not the vocoder parameters. Avoiding the vocoders
makes it possible to apply spectrum-domain or waveformdomain computation used in signal processing research. The
noise reduction for speech synthesis described above canbe applied in the assessment of the spectral parameters or
waveforms. Regarding robustness to unseen noise, the lattermethod (noise reduction and speech synthesis training) ispreferred. However, the accumulation of the spectral distortion
explained above remains a critical problem.
In this paper, we propose a generative approach to training
high-quality statistical parametric speech synthesis from noisy
speech using speech synthesis and noise generation models.
1 Usually, a statistical model that predicts speech from a text is called acoustic model, but in this paper we call it speech synthesis model to differentiate
it from the noise generation model introduced below.
2 Here, unseen means noise signals not contained in the training data.

Conventional approach2

1

Speechsynthesis

Noise
reduction

Text

Speech in noise(observed)

clean speech(estimated)

Human’s speech production
Text

Speechproduction

Noiseadditionclean speech(unobserved)

Speech in noise(observed)

Fig. 1. Speech synthesis training procedures using noisy speech. Conventional
method (upper graph) first performs noise reduction, then performs speechsynthesis training to predict the noise-suppressed speech parameters. Our
method (lower graph) directly predicts parameters of noisy speech using thenoise addition process of human speech production.

The noise generation models are trained by generative adversarial training [15] so that the generated spectral parameters
have the natural statistics of the observed noise. In synthesis,
the models randomly generate the spectral parameters of thenoise. The speech synthesis models are trained so that the sumof the output and the randomly generated noise is close to thespectral parameters of the noisy speech. Namely, the speechsynthesis models are trained using the noise addition process
as shown in the lower graph in Fig. 1. The proposed method
can more efficiently model the statistics of the observed
noise compared to the conventional spectral subtraction-basednoise reduction. Also, the proposed generative approach canalleviate the spectral distortion observed in the conventional
method by using noise reduction. We conducted the evaluation
with several settings of the signal-to-noise ratio and noise suppression ratio. The experimental evaluation demonstrated thatthe proposed method outperformed the conventional method
in terms of synthetic speech quality.
II. C ONVENTIONAL METHOD : SPECTRAL SUBTRACTION
AND SPEECH SYNTHESIS MODEL TRAINING

Using the observed noisy speech, we performed spectral
subtraction to suppress the observed noise first. Then we
performed speech synthesis model training based on the meansquared error criterion using the suppressed speech.
A. Spectral subtraction
Spectral subtraction [13] approximates the distribution of
the power spectrum of the noise signals and subtracts thenoise components from the power spectrum of noisy speech.
Let the log amplitude sequence of the noise signal be y n =
>
>
>
[y >
, and that of the noisy speechn,1 , · · · , y n,t , · · · , y n,Tn ]>>>be y ns = [y ns,1 , · · · , y ns,t , · · · , y >ns,T ] , where Tn and T
are the total frame lengths of the noise and noisy speech,>
respectively. y n,t = [yn,t (1) , · · · , yn,t (f ) , · · · , yn,t (F )]>and y ns,t = [yns,t (1) , · · · , yns,t (f ) , · · · , yns,t (F )] denotethe log amplitude of the noise and noisy speech at frame t,
respectively. f is the frequency bin and F is the total numberof the frequency bins. Here, y n indicates the non-speechFig. 2. Architectures of the proposed method. Noise generation model Gn (·)
is trained using noise discrimination model D n (·) and it samples the noise
randomly. Speech synthesis model Gs (·) is trained so that the sum of its
output and the sampled noise is close to spectra of the noisy speech.

period of y ns . The log amplitude suppressed by spectral(SS)
subtraction, y ns , is given asp2
 exp{yns,t (f )} − β ȳn,t (f )(SS)
exp{yns,t (f )} =if exp{yns,t (f )}2 > β ȳn,t (f ) ,
0 otherwise(1)Tȳn,t (f ) =n
1 X
exp{yn,t (f )}2 ,Tn t=1(2)where, β is the noise suppression ratio that adjusts the amountof noise suppression.
B. Speech synthesis model training
Let the speech synthesis model predicting the log amplitudeof speech from the input contexts be Gs (·) described as theneural networks [5], [14]. Here, let the input context sequence> >>be x = [x>1 , · · · , xt , · · · , xT ] , the model parameter ofGs (·) is estimated by minimizing mean squared error between(SS)
the generated log amplitude ŷ s = Gs (x) and y ns , which is
given as


> 

1
LMSE ŷ s , y (SS)=ŷ s − y (SS)ŷ s − y (SS)
. (3)
ns
ns
ns
T
C. Issues
Spectral subtraction causes distortion of the suppressed
log amplitude because the statistic of the noise is approximated with the expectation value. Also, it causes the socalled musical noise [16] which is known as a harsh artificial
noise. Moreover, this kind of spectral distortion is accumulated
during speech synthesis model training, so the quality of the
final synthesized speech is significantly lower.
III. P ROPOSED METHOD : GENERATIVE APPROACH TO
TRAIN SPEECH SYNTHESIS AND NOISE GENERATIONMODELSThe DNN architecture of the proposed method is shown in
Fig. 2. In addition to the speech synthesis model Gs (·) of
the conventional method, a noise generation model Gn (·) is

A. Generative adversarial training of the noise generation
model
Generative adversarial training algorithm [15] is used for
efficiently modeling the statistics of the observed noise. An
input of Gn (·) is a Tn -frame prior noise sequence n =
>
> >
[n>
randomly sampled frame by frame
1 , · · · , nt , · · · , nTn ]from a known probability distribution (e.g., uniform distribution). The nt indicates the prior noise vector at frame t. The
Gn (·) is updated with the noise discrimination models D n (·)
which distinguishes the generated noise ŷ n = Gn (n) from the(G)observed noise y n . The loss functions LGAN (·) for Gn (·) and(D)
LGAN (·) for D n (·) are
T(G)

LGAN (ŷ n ) = −n
1 Xlog Dn (ŷ n,t ),Tn t=1

(4)

T

(D)

LGAN (y n , ŷ n ) = −n
1 Xlog Dn (y n,t )Tn t=1

T

−

n

1 X
log 1 − Dn (ŷ n,t ) ,Tn t=1(5)respectively. The adversarial training minimizes the approximated Jensen-Shannon divergence between y n and ŷ n distributions. After the training, Gn (·) can randomly sample thenoise spectrum that has the statistics of the observed noise.
B. Speech synthesis model training using noise generation
model
We assumed that the additivity of speech and noise holds inthe amplitude domain and we ignored the phase information.
While fixing the estimated parameters of Gn (·), the speechsynthesis model Gs (·) was trained to minimize the following
loss function:
1>
LMSE (ŷ ns , y ns ) = (ŷ ns − y ns ) (ŷ ns − y ns ) , (6)Tŷ ns = log (exp ŷ s + exp ŷ n ) ,(7)Note that the sequence length of ŷ n here is T . In synthesis, the log amplitude of the synthesized speech is given
as ŷ s = Gs (x). The final synthesized speech is obtained byapplying Griffin-Lim’s phase reconstruction algorithm [17].
In this paper, the speech synthesis and the noise generation
model predict the log amplitude, and the predicted outputs

Freq. [ kHz] Freq. [ kHz]

introduced here. Gn (·) transforms the prior known distribution
to the observed noise distribution and randomly generates thenoise spectrum. The speech synthesis model Gs (·) is trained
so that the sum of its output and noise spectrum generated by
Gn (·) is close to the observed spectrum of the noisy speech.
In the preliminary experiment, we tried to simultaneously
train the Gs (·) and Gn (·), but the separation performanceof speech and noise is limited. Therefore, we first trained thenoise generation model Gn (·) to have the natural statisticsof the noise spectrum using the observed noise spectrum y n .
Then, while fixing the model parameter of Gn (·), the Gs (·)
was trained using the spectrum of the noisy speech.

8
6
4
2
0
8
6
4
2
0
0

1

2

3

Tim e [ s]Fig. 3. Spectrograms of observed noise (upper graph) and generated noise(lower graph). The generated noise is randomly sampled frame by frame
independently. We can see some temporal stripes in the generated noise, but
its overall tendency is similar to the observed noise.

are summed up in the linear amplitude domain. A simpler
implementation is one in which the models output the linear
amplitude. However, our method did not work well in this
implementation.
C. Discussion
The proposed method does not define the explicit probability distribution and expresses only the empirical distribution
using generative adversarial network (GAN). Therefore, theproposed method reduces the spectral distortion that causes
musical noise. The proposed method currently captures only
stationary noise in the amplitude domain but it can be extended
to recurrent architectures to capture non-stationary noise,
conditional GAN [18] to capture context-dependent noise(e.g., pop noise), and WaveNet [19] for waveform-domain
calculation [20]. Also, our GAN-based approach is expected
to extend to GAN-based model adaptation [21]. Pre-recorded
clean speech data will be used to build prior models.
IV. E XPERIMENTAL EVALUATION
A. Experimental conditions
We used approximately 3000 sentences which wererecorded in the anechoic room by a Japanese female speaker.
We artificially generated Gaussian noise as an observed noise
and added it to the recorded speech. The evaluation data was
53 sentences of subset J form ATR Japanese database [22].
The training data was sampled at 16 kHz. The window
length was 400 samples (25 ms), the frame shift length
was 80 samples (5 ms), and the size of the fast Fouriertransform (FFT) length was 512. The window function was
the Hamming window. The speech and noise generation
models predicted 257-dimensional log amplitude which did
not include the dynamic feature. The synthesized speech
waveform was synthesized by applying Griffin-Lim’s phasereconstruction [17] to the predicted log amplitude. Because
we confirmed that the residual noise was included in the synthesized speech in the conventional and proposed methods, weapplied weak spectral subtraction to the generated spectra sothat their speech components were not perceptually distorted.
No post-emphasis method, such as cepstrum [23], global
variance [24], and modulation spectrum [3]-based methods,

SS+MSEβ =0.5

0.368

0.632

β =1.0

0.312

0.688

0.312

0.688

β =2.0

SS+MSEβ =0.5

0.268

0.732

β =1.0

0.292

0.707

β =2.0

0.256

0.744

Proposed

β =5.0 0.253
0.747
0.00
0.50
0.75
0.25
1.00
Preference score
Fig. 4. Preference scores on synthetic speech quality (SNR = 0 dB).

SS+MSEβ =0.5

0.292

0.708

β =1.0

0.320

0.680

β =2.0

0.323

0.677

Proposed

β =5.0 0.288
0.712
0.00
0.50
0.75
0.25
1.00
Preference score
Fig. 6. Preference scores on synthetic speech quality (SNR = 10 dB).

Proposed: the proposed method where the speech synthesis model is trained so that the sum of its output andgenerated noise is close to spectra of noisy speech.
Note that vocoder-based approach was not used in this evaluation because the purpose of this research was to compare
the vocoder-free DNN-based speech syntheses. The signalto-noise ratio (SNR) was set as 0, 5, and 10 dB. Because
the tradeoff between the amount of speech distortion (or the
residual noise) and speech recognition accuracy was empirically known in DNN-based speech recognition, we believe thatthe same applies in DNN-based speech synthesis. Therefore,
we used several kinds of noise suppression ratio of spectral
subtraction, i.e., β = 0.5, 1.0, 2.0, and 5.0. The smaller β
makes less distortion of speech (but larger amount of residual
noise) and the larger β makes stronger distortion. Preference
AB tests were conducted to evaluate the naturalness of thesynthetic speech in every pair of SNR and noise suppression
ratio. The tests were done using our cloud-sourcing evaluation
system. Twenty five listeners participated in each test; the total
number of listeners was 300.
Figs. 4, 5, and 6 show the results of our experiments. Our
method outperformed the conventional method in all cases.
The p-values between the methods were smaller than 10−6 ,meaning that the results were statistically significant. In Fig.
4, we can see that the score of the conventional method
became worse as β increased in 0 dB SNR. We found thatthe speech distortion caused by strong spectral subtraction was
significantly higher in speech synthesis training with the worse
SNR. We think that this observation is an important factor in
vocoder-free DNN-based speech synthesis trained from noisy
speech.
•Proposed

β =5.0 0.216
0.784
0.00
0.50
0.75
0.25
1.00
Preference score
Fig. 5. Preference scores on synthetic speech quality (SNR = 5 dB).

was used. Contextual features consisted of 439-dimensional
linguistic features, 3-dimensional duration features, continuous
log F0 and voiced/unvoiced label [14]. In practice, the duration
feature, continuous log F0 and voiced/unvoiced labels must
be extracted from the noisy speech. However, we used theseones extracted from clean speech to avoid speech quality
degradation caused by extraction of these features from thenoisy speech [25]. In the training phase, the context x andnoisy speech log spectrum y ns were normalized to have zeromean unit-variance. In the synthesizing phase, ŷ s = Gs (x)was un-normalized using the statistics of y ns . Note that the
un-normalization process is an ill-posed problem, i.e., only
the scale of y ns scale is known, but we need to un-normalize
the component y s . The implementation of un-normalizationis one aspect of the future work. The input of the noise
generation model nt was 100-dimensional vector randomly
generated by the uniform distribution. 90% of the silenceframes were removed in the speech synthesis model training
phase. The DNN architecture for the speech synthesis model,
noise generation model, and noise discrimination model werefeed-forward neural networks, and the conventional and proposed methods used the same architectures of the speechsynthesis models. The architecture for the speech and noise
generation models included 3 × 512-unit leaky rectified linear
unit (leaky ReLU) [26] hidden layers and a 257-unit linear
output layer. Also, the architecture for the noise discrimination
model included 3 × 512-unit leaky ReLU hidden layers anda one-unit sigmoid output layer. We initialized each model
parameter using a random number and adoptive gradientalgorithm AdaGrad [27] as the optimization algorithm.
B. Results of subjective evaluation
We compared two synthetic speech samples of the following
methods:• SS+MSE: the conventional method where the speechsynthesis model is trained based on minimum squared
error criterion after the spectral subtraction.

V. C ONCLUSION
In this paper, we introduced a generative approach to DNNbased speech synthesis trained from noisy speech using the
proposed noise generation model. The noise generation model
was trained to have natural statistics of the observed noise andto randomly generate the noise spectra. The speech synthesis
model (acoustic model) was trained so that the sum of its
output and generated noise was close to noisy speech spectra.
We compared the proposed method and conventional method
using noise reduction and standard speech synthesis training
in synthetic speech quality. The results demonstrated that our
method outperformed the conventional method under several
signal-to-noise ratios and noise suppression ratios. As future
work, we will introduce an activation matrix of non-negativematrix factorization [28] and compare it with the vocoderbased methods.
ACKNOWLEDGMENT
Part of this work was supported by SECOM Science andTechnology Foundation, and JSPS KAKENHI Grant Number
18K18100.
R EFERENCES[1] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speechsynthesis,” Speech Communication, vol. 51, no. 11, pp. 1039–1064,
2009.
[2] S. Takamichi, K. Tomoki, and H. Saruwatari, “Sampling-based speechparameter generation using moment-matching network,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017.
[3] S. Takamichi, T. Toda, A. W. Black, G. Neubig, S. Sakti, and S. Nakamura, “Postfilters to modify the modulation spectrum for statisticalparametric speech synthesis,” IEEE Transactions on Audio, Speech, andLanguage Processing, vol. 24, no. 4, pp. 755–767, 2016.
[4] Y. Saito, S. Takamichi, and H. Saruwatari, “Training algorithm todeceive anti-spoofing verification for DNN-based speech synthesis,” inProc. ICASSP, Orleans, U.S.A., Mar. 2017.
[5] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speechsynthesis using deep neural networks,” in Proc. ICASSP. Vancouver,
Canada, May 2013.
[6] S. A.-E.-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan,
and S. Vijayanarasimhan, “YouTube-8M: A large-scale video classification benchmark,” vol. abs/1609.08675, 2016.
[7] Y. Ohtani, M. Tamura, M. Morita, and M. Akamine, “Statistical
bandwidth extension for speech synthesis based on Gaussian mixture
model with sub-band basis spectrum model,” IEICE Transactions on
Information and Systems, vol. E99-D, no. 10, pp. 2481–2489, 2016.
[8] A. Saeb, R. Menon, H. Cameron, W. Kibira, J. Quinn, and T. Niesler,“Very low resource radio browsing for agile developmental and humanitarian monitoring,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug.
2017, pp. 2118–2122.
[9] S. Takamichi and H. Saruwatari, “Croudsourcing large-scale parallel
speech corpora of japanese dialect,” in 2017 autumn meeting of Acoustic
Society of Japan, 1-8-4, Sep. 2017, (in japanese).
[10] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based F0 extraction: Possible role of arepetitive structure in sounds,” Speech Communication, vol. 27, no. 3–4,
pp. 187–207, 1999.
[11] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoder-based
high-quality speech synthesis system for real-time applications,” IEICE
transactions on information and systems, vol. E99-D, no. 7, pp. 1877–
1884, 2016.
[12] C. V.-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Speechenhancement for a noise-robust text-to-speech synthesis system using
deep recurrent neural networks,” in Proc. INTERSPEECH, Sep. 2016,
pp. 352–356.
[13] S. F. Boll, “Suppression of acoustic noise in speech using spectral
subtraction,” IEEE Transactions on Acoustic, Speech, and Signal
Processing, vol. ASSP-27, no. 2, pp. 113–120, 1979.
[14] S. Takaki, H. Kameoka, and J. Yamagishi, “Direct modeling of
frequency spectra and waveform generation based on phase recovery
for DNN-based speech synthesis,” in Proc. INTERSPEECH, Stockholm,
Sweden, Aug. 2017.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
Proc. NIPS, pp. 2672–2680, 2014.
[16] R. Miyazaki, H. Saruwatari, T. Inoue, Y. Takahashi, K. Shikano, andK. Kondo, “Musical-noise-free speech enhancement based on optimized
iterative spectral subtraction,” IEEE Transactions on Audio, Speech, andLanguage Processing, vol. 20, no. 7, pp. 2080–2094, Sep. 2012.
[17] D. W. Griffin and J. S. Lim, “Signal estimation from modified shorttime fourier transform,” IEEE Transactions on Acoustics, Speech andSignal Processing, vol. 32, no. 2, pp. 236–243, Apr. 1984.
[18] M. Mirza and S. Osindero, “Conditional generative adversarial nets,”
arXiv:1411.1784, 2015.

[19] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves,N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “WaveNet: A
generative model for raw audio,” vol. abs/1609.03499, 2016.
[20] K. Qian, Y. Zhang, S. Chang, X. Yang, D. Florencio, and M. H.Johnson, “Speech enhancement using Bayesian Wavenet,” in Proc.
INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 2013–2017.
[21] M. Sato, H. Manabe, H. Noji, and Y. Matsumoto, “Adversarial training
for cross-domain universal dependency parsing,” in Proc. the CoNLL
2017 Shared Task, Vancouver, Canada, Aug. 2017, pp. 71–79.
[22] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara, “A large-scale Japanese speech database,” in ICSLP90,
Kobe, Japan, Nov. 1990, pp. 1089–1092.
[23] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,“Simultaneous modeling of spectrum, pitch and duration in HMM-based
speech synthesis,” in Proc. EUROSPEECH, Budapest, Hungary, Apr.
1999, pp. 2347–2350.
[24] T. Toda and K. Tokuda, “A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,” IEICE
Transactions on Information and Systems, vol. E90-D, no. 5, pp. 816–
824, 2007.
[25] P. Baljekar and A. W. Black, “Utterance selection techniques for TTS
systems using found speech,” in Proc. SSW9, Sunnyvale, CA, USA,
Sep. 2016, pp. 199–204.
[26] L. A. Maas, Y. A. Hannun, and Y. A. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” in Proc. ICML, 2013, vol. 30.
[27] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods
for online learning and stochastic optimization,” EURASIP Journal on
Applied Signal Processing, vol. 12, pp. 2121–2159, 2011.
[28] F. Cedric and J. Idier, “Algorithms for nonnegative matrix factorization
with the β-divergence,” Neural computation, vol. 23, no. 9, pp. 2421–
2456, Aug. 2011.