2019 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP2019)
Honolulu, Hawaii, USA, March 4-7, 2019

Experimental Evaluation of WaveRNN Predictor for Audio Lossless Coding
Shota Amada1 , Ryosuke Sugiura2 , Yutaka Kamamoto2 , Noboru Harada2 , Takehiro Moriya2 ,Takeshi Yamada1 and Shoji Makino1
2

1

University of Tsukuba
1–1–1 Tennoudai, Tsukuba-shi, Ibaraki, 305–0006, Japan
E-mail: s1720651@s.tsukuba.ac.jp

Abstract
This paper describes a new scheme of speech and audio lossless coding. MPEG-4 Audio Lossless Coding(ALS) is the international standard lossless compression
method of audio signals. It uses linear predictor andcompresses the signal by converting the signal into information such as prediction residuals and prediction coefficients. In the compressed signal with MPEG-4 ALS,
the prediction residuals occupies a large amount of information. Improving the performance of the predictor is directly related to improving compression performance. Using non-linear predictors in lossless coding
has the possibility to perform flexible prediction and improve prediction performance. WaveRNN is the deepneural networks performing a non-linear prediction. The
outputs of WaveRNN are the predicted probability distribution of the target sample. Arithmetic coding can
generate an optimal code for a set of arbitrary symbols and probabilities. WaveRNN is trained to minimize
the bitrate after encoding when using arithmetic coding
for the output of WaveRNN. This paper proposes thescheme of speech and audio lossless coding that combines WaveRNN with arithmetic coding. Experimental
evaluation confirmed that the developed method reduced
around 0.7 bits per sample compared to MPEG-4 ALS
in speech coding. DNN-based speech and audio coding
techniques are expected to transcend the international
standard technologies.
1. Introduction
Broadband networks and digital devices have developed and there is an increasing demand for high quality
audio signals. The amount of information required forarchive and broadcast is dramatically increasing. The
compression method with less deterioration of quality isrequired because the demand of using large amount of
information with high quality, such as high-resolusion
audio is increasing. (Generally, high-resolusion audio isthe audio files that have a higher sampling frequency
and/or bit depth than that of Compact Disc Digital Audio, which is specified at 16 bit/44.1 kHz.) Lossless compression meets the demand as it can reduce the rate while

NTT Communication Science Laboratories,
Nippon Telegraph and Telephone Corporation,
3–1 Morinosatowakamiya, Atsugi-shi, Kanagawa,
243–0198, Japankeeping the perfect reconstruction of the signal. Therefore, it plays an important role for the high quality soundtransmission such as high-resolusion audio. Considering
to improve the performance of lossless compression, thecompression ratio and the calculation amount attracts
attention as the performance index because the original
signal is losslessly reconstructed. In this paper, we aim
to improve the encoding performance about the compression ratio in lossless compression method.
MPEG-4 Audio Lossless Coding (ALS) is the international standard lossless compression method of audio
signals[1, 2, 3]. It can handle many types of input signals
including high-resolution audio such as one with sampling frequency up to about 4 GHz, quantization bitdepth up to 32 bit and up to 65536 channels. It uses
linear prediction analysis and the performance of linear
predictor aﬀects its compression ratio. The studies toimprove the linear predictor in MPEG-4 ALS has been
conducted so far[4]. According to these studies, it seems
to exist signals that are diﬃcult for the conventional linear predictor to predict. Using non-linear predictors inlossless coding has the possibility to perform flexible prediction that is not limited to linear prediction and improves prediction performance.
WaveNet and WaveRNN are the deep neural networks
proposed to directly generate audio waveform in the fieldof speech synthesis[5, 6]. Recently, lossy speech coding
methods using WaveNet as a non-linear predictor havebeen reported[7]. WaveNet has the problem that it works
with up to 8 bit (µ-law) signal and takes large amountof calculation. WaveRNN solves this problem. It works
with up to 16 bit signal and takes smaller amount ofcalculation than WaveNet.
WaveRNN outputs the predicted probability distribution of the target sample using the previous sample. In
signal compression, if the probability distributions of signals are known, compression can be eﬃciently performedby optimal entropy coding for the distributions. The optimal entopy coding can be realized by arithmetic codeing. Generally, arithmetic coding can generate an optimal code for a set of arbitrary symbols and probabilities,
and the codelength becomes the valuse close to the entropy. Therefore, by training WaveRNN to optimize en-3152019 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP2019)
Honolulu, Hawaii, USA, March 4-7, 2019

tropy, it is possible to obtain the predictor that directly
minimizes the codelength in arithmetic coding.
We have already done the initial examination of the
speech lossless coding using WaveRNN as predictor[8].
This paper describes a new scheme of speech and audio
lossless coding using WaveRNN as a nonlinear predictor.

Residual

Input audio

𝑒(𝑛)

𝑥(𝑛)

−

Coded
bitstream

Residual

𝑒(𝑛)

𝑥(𝑛)

+
𝑥6(𝑛)

RLS-LMSpredictor

Estimate

Figure 1: MPEG-4 ALS encoder (top) and decoder (bottom) with the cascaded RLS–LMS predictor.
𝐑

c"#$
f"#$

𝐎𝟏

𝐎𝟐

𝑃(c" )

𝐈
𝑃(f" )c"

𝐎𝟑

𝐎𝟒

Figure 2: The network structure of WaveRNN
bits at the target time (P (ct ), P (ft )) from coarse bits
and fine bits at the previous time (ct−1 , ft−1 ).
xt =[ct−1 , ft−1 , ct ]ut =σ(Ru ht−1 + Iu⋆ xt )rt =σ(Rr ht−1 + Ir⋆ xt )et =
τ (rt ◦ (Re ht−1 ) + Ie⋆ xt )ht =ut ◦ ht−1 + (1 − ut ) ◦ etyc , yf =split(ht )P (ct ) = sof tmax(O2 relu(O1 yc ))P (ft ) = sof tmax(O4 relu(O3 yf ))3. WaveRNN
The network structure of WaveRNN is shown in Figure
2. WaveRNN consists of single recurrent neural network(RNN) layer and two full-connected neural network (NN)layer. WaveRNN uses the previous sample as input topredict the probability distribution of the target sample.
The sample bits converted into coarse bits and fine bits.
The coarse bits is higher 8-bits of the 16-bits sample, andfine bits is lower 8-bits of the 16-bits sample. In Figure
2, c and f at the inputs represents coarse bits and fine
bits respectively. P (c) and P (f ) at the outputs are theoccurrence probability of each coarse bits and fine bitsin the next sample. Eq.(1) shows the overall flow for
obtaining occurrence probabilities of coarse bits and fine

Entropy
decoder

Estimate

Decoded audio

In the MPEG-4 ALS, the ALS with LPC (linear
predictive coding) is the general audio lossless coding
method. In the ALS with LPC, the PARCOR coeﬃcients and the linear prediction residual signal are transmitted and the original signal is perfectly reconstructedby using them. Furthermore, the ALS with cascaded recursive least square–least mean square (RLS–LMS) prediction is defined[9]. In ALS with RLS-LMS prediction, it is not necessary to transmit PARCOR coeﬃcients. Compared with the above-mentioned method,
this method has not been widely used since it has a large
calculation amount. However, MPEG-4 ALS with RLSLMS prediction provides higher compression ratio than
MPEG-4 ALS with LPC because it does not send PARCOR coeﬃcients. The proposed scheme has high similarity to the MPEG-4 ALS with RLS–LMS prediction,
we use it as the comparison method.
The block diagram of MPEG-4 ALS encoder and decoder with the cascaded RLS–LMS predictor id shownin Figure 1. In the encoder, the prediction signal isacquired by the cascaded RLS-LMS predictor for eachframe of imput signal, and the prediction residual signal is calculated from the input signal and the prediction signal. The prediction residual signal is entropy
coded and transmitted as a compressed signal. In the
decoder, the transmitted compressed signal is entropy
decoded to obtain the residual signal. The decoder canalso calculate the prediction signal in the same way as
the encoder, because the cascaded RLS-LMS predictor
outputs the target prediction sample by using previous
samples. Therefore, the original signal is perfectly reconstructed from the residual signal and the prediction
signal obtained by the cascaded RLS-LMS predictor.

Coded
bitstream𝑥6(𝑛)RLS-LMSpredictor

2. MPEG-4 ALS

Entropycoder(1)Here, Ru , Rr , and Re are the values obtained from multiplying the hidden value of the RNN layer by the recursive
weight, and these are used for obtaining ut , rt , and et respectively. σ and τ are sigmoid function and hyperbolic
tangent function of standard activation functions. Also,⋆ represents a masked matrix whereby ct is used only forcomputing ft .

316

2019 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP2019)
Honolulu, Hawaii, USA, March 4-7, 2019

4. Proposed RNN-based prediction method

Table 1: Specifications of Multilingual Speech Database
The block diagram of the encoder and decoder of our 2002.
proposed method is shown in Figure 3. WaveRNN uses
16 kHzSampling rate
the previous sample as input to predict the probability
Amplitude resolutin
16 bitdistribution of the target sample. Arithmetic coding is
Number of channels
1 chperformed according to combinations of the sample valRecoding time per item4s
ues and the probability distributions and it is possible
Number of items
17,000to generate optimal codelengths for these combinations.
WaveRNN learns to minimize the cross entropy of thetrue probability distribution and the predicted probabil- Table 2: Specifications of classic music items in RWC
ity distribution. Here, the true probability distribution Music Database.
is that of the input, so it become one-hot vector. Whenperfoming arithmetic coding with the probability distriSampling rate
44.1 kHzbution outputs by WaveRNN, the codelength after enAmplitude resolutin
16 bitcoding depends on the cross entropy at learning. Thus,
Number of channels
2 chlearning in WaveRNN performs aiming at minimizing
Number of items
50
the codelength after arithmetic coding.
Total recoding time 5.48 hours
Input audio

Entropycoder𝑥(𝑛)WaveRNNpredictor
Decoded audio

WaveRNNpredictor

5.2 Criteria for performance evaluation
In this experiment, the coding performance of WaveRNN predictor is evaluated by the average codelengthof coded signal. The average codelength indicate theaverage cross entropy per one sample of signal. This indicate the theoretical bit rate [bits/sample] in compression. The average codelength H is shown in Equation(2). Here, N is the number of samples, and pi (xi ) is thepredicted probability of the true sample value xi .

𝑃6 (𝑥(𝑛))Probability distribution

Entropy
decoder

𝑥(𝑛)

Coded
bitstream

Coded
bitstream

𝑃6 (𝑥(𝑛))Probability distribution

1 ∑− log2 pi (xi )
N i=1
N

H=

Figure 3: The encoder (top) and decoder (bottom) with
WaveRNN predictor(2)

5.3 Evaluation for speech item
5. Evaluation
5.1 Experimental conditions

In this experiment, 300 items were used for training WaveRNN and 100 items were used for evaluation.
These were American English speech items by 5 men and5 women. Figure 4 and Table 3 show the evaluation results of the performance of MPEG-4 ALS and WaveRNN
predictor for speech items. In Figure 4, the error bars
represent 95 % confidence intervals. WaveRNN predictor
showed better compression ratio than MPEG-4 ALS in
items of 95 % or more. In fact, it showed better compression ratio than MPEG-4 ALS at all items for evaluation.
In the Table 3, the number in the parentheses represents
the diﬀerence of average codelength from MPEG-4 ALS.
The performance of WaveRNN predictor for speech item
was 0.77 bits/sample better than MPEG-4 ALS in average.

In speech coding experiment, the speech items of Multilingual Speech Database 2002 are used[10]. Table 1
shows specifications of the speech items used in this experiment. In music coding experiments, classic music
items of RWC Music Database are used[11]. Table 2
shows specifications of the music items used in this experiment. In each experiment, only some of the items
were used from each database.
In speech coding experiment, the non-utterance sections are cut out from the speech items. In music coding
experiment, the music items are resampled to 16 kHz
and each items are split every 4 seconds. WaveRNN is Table 3: The average codelengths of speech coding withproposed for speech synthesis. Therefore, it is expected MPEG-4 ALS and WaveRNN predictor.
that the performance will be degraded in music encoding.
WaveRNN predictor MPEG-4 ALS
7.00 (-0.77)
7.77
bits/sample

317

2019 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP2019)
Honolulu, Hawaii, USA, March 4-7, 2019

Bits par sample

8
7.8

Table 4: The average codelengths of music coding with
MPEG-4 ALS and WaveRNN predictor.

7.6
7.4

bits/sample

7.2
7
6.8
6.6
6.4
MPEG-4 ALS

WaveRNN

Figure 4: The evaluation results of the performance of
MPEG-4 ALS and WaveRNN predictor for speech items
5.4 Evaluation for music item
In this experiment, 100 items were used for training
WaveRNN and 50 items were used for evaluation. Figure 5 and Table 4 show the evaluation results of theperformance of MPEG-4 ALS and WaveRNN predictor
for music items. In Figure 5, the error bars represent
95 % confidence intervals. WaveRNN predictor showedworse compression ratio than MPEG-4 ALS in the average compression ratio. However, WaveRNN showed
better compression ratio than MPEG-4 ALS for somesound sources (21 items in 50 items for evaluation). In
the Table 4, the number in the parentheses represents
the diﬀerence of average codelength from MPEG-4 ALS
in average. The performance of WaveRNN predictor formusic items was 0.64 bits/sample worse than MPEG-4
ALS in average.

WaveRNN
9.38 (+0.64)

MPEG-4 ALS
8.74

speech coding, experimental evaluation confirmed thatthe proposed method reduced around 0.8 bits per sample compared to MPEG-4 ALS. In audio coding, theproposed method showed worse compression ratio than
MPEG-4 ALS in the average compression ratio. This isbecause WaveRNN used in this experiment was adjusted
for speech synthesis. However, the proposed method
showed better compression ratio than MPEG-4 ALS forsome sound sources (21 items in 50 items for evaluation). The proposed method has the potential to obtainbetter compression performance than MPEG-4 ALS inmusic coding. It is considered that the compression performance will be improved by tuning parameters of WaveRNN for music audio. DNN-based speech and audio
coding techniques will transcend the international standard technologies.
References

Bits par sample[1] ISO/IEC 14496-3:2009/Amd 3:2015, 2009 Information technology – Coding of audio-visual objects –Part 3: Audio.
[2] T. Liebechen, et al. ”The MPEG-4 audio lossless coding (ALS) standard - Technology and applications,”in Proc. 119th AES Convention, pp. 1-14, Oct, 2005.
[3] T. Liebchen and Y. Reznik, ”MPEG-4 ALS：anemerging standard for lossless audio coding,” in Proc.
IEEE Data Compression Conference, pp. 439-448,
2004.
12[4] S. Amada, et al. ”Experimental evaluation of encoding parameters of MPEG-4 ALS for high-resolution
10
audio,” in Proc. IEEE Global Conference on Consumer Electronics, Oct, 2017.
8[5] A. van den Oord, et al. ”WAVENET: A generativemodel for raw audio,” arXiv:1609.03499, 2016.
6[6] N. Kalchbrenner, et al. ”Eﬃcient neural audio syn4
thesis,” arXiv:1802.08435, 2018.
[7]W.
B. Kleijn, et al. ”WAVENET based low rate2speech coding,” arXiv:1712.01120, 2017.
0[8] S. Amada, et al. ”Experimental evaluation of the audio lossless coding with WaveRNN,” in Proc. AcousMPEG-4 ALS
WaveRNNtical Society of Japan 2018 Autumn Meeting, 2-4-9,
Sept, 2018 (in Japanese).
[9]
H.
Huang, et al. ”Cascaded RLS-LMS prediction in
Figure 5: The evaluation results of the performance of
MPEG-4 lossless audio coding,” in Proc. ICASSP,
MPEG-4 ALS and WaveRNN predictor for music items
pp. 181-184, May, 2006.
[10] Multilingual Speech Database 2002,
http://www.ntt-at.com/product/speech2002/
6. Conclusions[11] RWC Music Database,This paper presented a new scheme of speech and auhttps://staﬀ.aist.go.jp/m.goto/RWC-MDB/
dio lossless coding using WaveRNN as a predictor. In

318