ASRU 201719th Dec. 2017.

Subband WaveNet with overlapped single-sideband filterbanks
Takuma OKAMOTO1, Kentaro Tachibana1, Tomoki Toda2,1, Yoshinori Shiga1, and Hisashi Kawai11National Institute of Information and Communications Technology, Japan, 2Nagoya University, Japan

1. Introduction

3. Experiments
Japanese speech corpora with a sampling frequency of 32 kHz
Female and male corpora: 4.7 and 3.7 hours for training sets, 100 utts for test sets
Filterbank setting (Decimation factor: M=4)
LPF-MD: N=4 with simple lowpass filter
LPF-OL: N=9 with simple lowpass filter
SQRT-Hann-OL: N=9LPF-MDSQRT-Hann-OL
with overlapped square-root Hann window-based analysis/synthesis filter
WaveNet models
Unconditional WaveNet training and synthesis without additional inputx̂ = [x̂(1), · · · , x̂(T )]Predicting x̂(t) from correct input [x(1), · · · , x(t 1)]

Target: High-quality statistical parametric speech synthesis
Text
Linguistic features
Acoustic features
Speech waveforms

Conventional:
State-of-the-art:

HMM to DNN
STRAIGHT, WORLD, GlottDNN etc..
1. WaveNet
2. End-to-end: Char2wav, Deep voice, Tacotron
WaveNet: Dilated causal convolutional NN-based raw audio generative model
Autoregressive model to directly predict raw audio samples
TYp(x|h) =p(x(t)|x(1), · · · , x(t 1), h)

···

#M

x1

x2···

xN

WaveNetfor 2nd band···fs20(b)#M"M

fs2(c)

Aliasing components0(d)fs2M······0(e)Hz
fs2Hz0(f)fs2x̂2WaveNetfor 1st band

WaveNetfor 2nd band"Mx̂2"M···0.56-30

123-600123[dB]···"M-208-30678 -40

00123Time [s]3π/4π/2π

Softmax loss

0.5694-6000[dB]

21.5 ± 0.35
13.6 ± 0.28
13.1 ± 0.22

8.72 ± 0.07
7.68 ± 0.07
7.16 ± 0.05

2.45 ± 0.03
2.80 ± 0.04
1.95 ± 0.04

MethodMale
f s = 32 kHz

FullbandLPF-MD
LPF-OLLPF-MD
LPF-OLSQRT-Hann-OL

13.0 ± 0.27

6.45 ± 0.09

1.72 ± 0.04

23.2 ± 0.26

9.51 ± 0.08

2.75 ± 0.03

14.0 ± 0.25
13.4 ± 0.26
13.6 ± 0.31

プ条件であり，分析，合成フィルタは共に，-20⎧ $-30⎨
cos( N 2−1 ω) for − Nπ−1 ≤ ω ≤ Nπ−1H(ω) =-40
⎩ 0otherwise-50の周波数特性 (ω：各周波数) を実現する。分析 → 合-60

成後の周波数特性 h(t) ∗ g(t) がFrequency
Fig. 3(b)に相当する。
[kHz]
0481212Time [s]FullbandMCD [dB](WaveNet)

16

離散逆フーリエ変換により 1024 サンプルの FIR フィ
Single-sideband modulation-based synthesis filtering

8.17 ± 0.08
7.48 ± 0.06
7.26 ± 0.10

2.5

3-30

1-15-15M

311413 279 3

Frequency [kHz]43955≪ 104 −10≪ 10−1010-10-208-306-402 4-502-600[dB]3-40456-50
Frequency band-60012312-1010-208-30

674-50-60013[dB]

SQRT-Hann-OL

2(A) Original
1LPF-OL
SQRT-Hann-OL

123(B) Estimated by WaveNet

456Frequency band

Z-score-45−18.82Time [s]78(C) Residual between
original and estimated

9(D) Re-analyzed-48-48-50-50-52-54-56

1-400[dB]

LPF-OL-40−18.0092Time [s]-35-508Fig. 6. Results of softmax loss averaged in last 1 k out of 100 k
training times for female training set with a sampling frequency of32 kHz.

-20
Table
2 Results of paired comparison -20listening
-25
-25
test.
C, P and N are answer number of conventional-30-30
fullband WaveNet, proposed subband WaveNet with33

LPF-OLSQRT-Hann-OL

12Softmax loss comparison-10-50
F
00-10

1.5

0.5-10SQRT-Hann-OL
and neutral.
-40
CPNp-value-45

1014LPF-MD

2.75 ± 0.05
2.10 ± 0.04
2.07 ± 0.06

-3516082SD [dB]
(WaveNet)

1014-20-50

SNR [dB](WaveNet)

16010Time [s]Original

112Fig. 6. Results of softmax loss averaged in last 1 k out of 100 ktraining times for female training set with a sampling frequency ofTable 1 Results of objective evaluations with 100 test set utterances.
32 kHz.

0
WaveNet-10 N = 9，M = 4 のオーバーラッ
for N-th bandSQRT-Hann-OL:

x̂N-10464 5-50
2Frequency
band-400

SQRT-Hann-OL···

π/4

Angular frequency [rad]Frequency [kHz]8Femalef s = 32 kHzx̂N0π10Frequency [kHz]10Fullbandx̂1x̂1t(n 1/2)

Shift with WN
Hz12LPF-OL -10
10SQRT-Hann-OL-20(a) Training stage

Hzfs201

WaveNetfor N-th band

Additional input h3π/4π/2

Power spectrum [dB]#M

-60-80π/4Power spectrum [dB]

(a)

14

Power spectrum [dB]0160Softmax loss

#M

WaveNetfor 1st band

Real value
Hz10142

Complex conjugate

Hz1604t(n 1/2)

Complex value

1014Frequency [kHz]Single-sideband modulation-based analysis filtering

Subband WaveNetShift with WN

16

Power spectrum [dB]Single-sideband (SSB) filterbank
1. Dividing fullband signal into N subband signals
2. Decimating them with a factor M
Signal length and sampling frequency: 1/M
3. Upsampling and inverse processing
Reconstructing fullband signal-40

Receptive field: 0.192 s, mini-batch size: 2.5 s with Adam optimizer
Dilation channel: 32, Residual channel: 32, Skip channel: 512
Number of parameter update: 20 k (Fullband), 10 k (Subband)
Results of synthesis speed with CPUs: Female (3.85 s) and male (3.87 s)
Fullband: 11.40 and 11.21 mins, Subband: 2.68 and 2.65 mins
about 4 times
2.5Results of objective evaluations
2.5
Proposed SQRT-Hann-OL colors each subband
waveform
22Improving prediction accuracy: Realizing
higher-quality synthesis than fullband
1.5
1.5

12Fullband waveform x-20

Angular frequency [rad]Frequency [kHz]Additional input h-600Softmax loss

2. Subband WaveNet-40-80t=1Categorical problem (8bit μ-law encoding) rather than regression
Synthesis speed problem since past samples are requiredParallel WaveNet: Fast generation but complicated training architecture
Proposal: Rapid and high quality synthesis for WaveNet with simple approach
Parallel training and synthesis based on Multirate signal processing
Realizing high-quality synthesis with square-root Hann window-based filterbanks

0

Relative level [dB]

VocoderFrequency [kHz]Acoustic modelPower spectrum [dB]

Text analysis

Relative level [dB]0-202Frequency [kHz]34-58-52-54-56

6

78
Frequency [kHz]910-58678
Frequency [kHz]910Fullband
LPF-OL (2nd)SQRT-Hann (2nd) LPF-OL (5th)SQRT-Hann (5th)Table 3. Results of paired comparison listening test with a sampling
Results
evaluations:with21 Japaneselistening
subjects
frequencyof 32 kHz ofwithsubjective
21 adultマックス損失
Japanese(最後nativespeakers. C, P, Paired comparison
frequency of 32 kHz with
21 adult
native speakers.
C, P,1000 回平均) を示す。WaveNet はx̂(t) を推定し，生成サンプル x̂ = [x̂(1), · · · , x̂(T )]and N are answers of conventionalSNR
fullbandWaveNet,proposedsubandNareanswersofconventional
fullbandWaveNet,proposed subを最大化するように学習されるため，誤差は各周
Significant
improvement
byproposed
SQRT-Hann-OLを出力とする。WaveNet における受容野は0.192 s quality
band WaveNet with SQRT-Hann-OL, and neutral.
band WaveNet with SQRT-Hann-OL, and neutral.
ルタで実現した。

Fullband waveform x̂

Subband WaveNet(b) Synthesis stage
WaveNet training and synthesis for each subband waveform
Enabling parallel synthesis: M times synthesis speed(女性テストセット) を示す。また，Fig. 6 に，LPF条件なし WaveNet
Table による学習，生成を行う。補助
3. Results of paired comparison listening test with a sampling
入力 h は用いず，正解入力 [x(1), · · · , x(t−1)] から， OL および SQRT-Hann-OL の学習セットのソフト波数帯に均一に分散され，(B) の周波数特性は全てフFullband SQRT-Hann-OL
Neutralp-valueWaveNetZ-score
ラットになる。そのため，フルバンドは高1024} × 3 = 33 層，8 万サンプルを，サブバンド
SD 18.0やFemale79⌧ 10 10(f s = 8 kHz) では，{1, 2, 4, · · · ,33256} × 3 = 27 層，413域のスペクトル構造がモデル化できておらず，MCD は劣化する。
(%)
(6.3)
(78.7)(15.0)LPF-OL と SQRT-Hann-OL
2 万サンプルをそれぞれ拡張型畳み込み層，バッファでは，分割周波数帯域は共通で，異なるのは分析，合サイズ (2.5 s) とした。拡張チャネル，初期フィルタとし，フル帯域 (f s = 32 kHz) では，{1, 2, 4, · · · ,Male(%)

Fullband31(5.9)SQRT-Hann-OL
439(83.6)

Neutral55(10.5)

p-value⌧ 10 10Z-score18.8