REPRESENTATION OF SPECTRAL ENVELOPE
WITH WARPED FREQUENCY RESOLUTION
FOR AUDIO CODER
R. Sugiura1 , Y. Kamamoto2 , N. Harada2 , H. Kameoka2 , T. Moriya2
1

Graduate School of Information Science and Technology, The University of Tokyo
2
NTT Communication Science Labs., Nippon Telegraph and Telephone Corp.
ABSTRACT

We have devised a method for representing frequency spectral envelopes with warped frequency resolution based onsparse non-negative matrices aiming at its use for frequency
domain audio coding. With optimally prepared matrices, we
can selectively control the resolution of spectral envelopesand enhance the coding efficiency. We show that the devised
method can enhance the subjective quality of the state-of-theart wide-band coder at 16 kbit/s at a cost of minor additional
complexity. The method is therefore, expected to be useful for low-bit-rate and low-delay audio coder for mobile
communications.
Index Terms— audio coding, signal processing, frequency warping, non-negative matrix, TCX
1. INTRODUCTION
For years, speech coders have been developed for use in voicecommunication tools such as mobile phones. However, there
is a demand for higher quality not only in speech, but also in
the other audio signals such as music.
3GPP Extended Adaptive Multi-Rate WideBand (AMRWB+) and MPEG-D Unified Speech and Audio Coding(USAC) [1,2] are the state-of-the-art speech and audio coders.
Both have at least two different modes and switch from one
to the other depending on the input signals. Voice signals
are coded in the time domain, and the other audio signals
are coded in the frequency domain by Transform Coded eXitation (TCX). The goal of the present work is to design a
low-bit-rate audio coder with higher quality, with lower algorithmic delay than AMR-WB+ or USAC so that the coder canbe used in mobile communications. Here, to achieve higher
quality in the audio coder, we discuss modifying the TCX
coder.
The efficiency of TCX is highly dependent on the parameters representing frequency spectral envelopes of the inputs.
In this paper, we introduce a model using frequency warpingfor a more efficient representation of spectral envelopes witha similar motivation such as in [3–6]. To achieve low computational complexity for the warping, we construct sparse
matrices with non-negative elements that approximate warping and inverse warping. In addition, perceptual weighting
for the envelopes of this model is also considered.
In section 2, we outline the TCX coder. Then, in section 3,
the model of spectral envelopes is introduced. Finally, in sec-tion 4, the model is evaluated by quantitative and subjective
assessments.
2. FREQUENCY DOMAIN AUDIO CODING
2.1. TCX coder
The state-of-the-art TCX coder uses Modified Discrete Cosine Transform (MDCT) [7] and is used in USAC. Fig. 1
describes the TCX process. The coder quantizes and codes
two kinds of information: linear prediction (LP) coefficients,
which represent the spectral envelopes, and residual spectra,
the frequency spectra divided by their envelopes. LP coefficients are first transformed into line spectrum pairs (LSP),
which are robust for interpolation and quantization, and thenvector quantized. On the other hand, residual spectra are compressed by entropy coding after they are scalar quantized foreach frequency bin.
Envelopes are perceptually weighted when divided into
the spectra. Normally, envelopes are calculated from LP coefficients by using an all-pole filter:∑
2πk
Hk = 1/|1 +an e−j N n |, (k = 0, · · · , N − 1) (1)nwhere N and an are the length of the envelopes and the coefficient of the nth order, respectively. To make the quantizationnoise in the spectra less annoying, the envelopes are smoothedby the weighting defined as∑
2πkH̃k = 1/|1 +an γ n e−j N n |, (0 < γ < 1).
(2)nThis weighting shapes the quantization noise into approximately H̃k /Hk , eventually resulting in smaller distortion in
the spectral peaks, which is more important than the spectral
valleys for human perception. It is experimentally known thatthe perceptual weighting works well when 0.92 is chosen for
the value of the parameter γ.
Moreover, the envelopes can improve the performance ofthe entropy coding. The values of the residual spectra canbe roughly estimated from the values of their envelopes: thehigher the values of the envelopes are, the more likely thevalues of the residual spectra are to be high.
2.2. Limitation of linear prediction
As stated above, envelopes can shape the quantization noise in
the spectra to realize a perceptually efficient quantization andcan also estimate the values of the residual spectra. However,this noise shaping approximation and value estimation standFig. 1. Outline of TCX encoder.
Fig. 2. Flowchart for extracting resolution warped envelopes
only if the spectral envelopes properly represent the shapesof their spectra. Generally, the envelopes extracted by linear
prediction have about (length of the signal) /(order of LP) frequency resolution and this resolution is uniform over the frequency axis. This limitation on the resolution makes, in somecases, the envelopes fail to represent their spectra, which leads
to an unexpected distribution of the quantization noise andestimation in the entropy coding. Indeed, this failure can be
avoided by increasing the order of the prediction, but this also
increases the parameters that must to be transmitted. In fact,
most natural sounds have a lot of power in the lower frequencies so that the information content of quantized signals tends
to be biased to the lower band. Therefore, instead of transmitting more parameters, we modify the model of envelopes
to warp the frequency resolution into, for example, a Melfrequency scale.
3. DESIGN OF THE ENVELOPE MODEL3.1. Extracting envelopes using frequency warping
A previous work on modeling spectral envelopes with warped
resolution used an all-path filter to modify the all-pole modeleq. (1) [4]. This modified method is called Mel Linear Predictive Coding (Mel-LPC) and is a special case of Mel generalized cepstral analysis [5], of which use was considered ina time-domain speech coder [6]. However, estimating the parameters of the model from inputs and calculating envelopes
both requires much more computational complexity than linear prediction. In addition, this filter has only one parameterfor tuning the warping and is lacking in flexibility.
To reduce computational costs, the model introduced
here uses sparse non-negative matrices for approximating
frequency warping. Fig. 2 shows the processes for extracting resolution warped envelopes. First, power spectra arefrequency warped by a warping matrix and Fourier transformed. Then, the transformed spectra are used as a pseudoautocorrelation function to estimate parameters by linear
predictive analysis. Finally, envelopes are calculated bysubstituting the parameters into the all-pole model eq. (1),followed by inverse warping with another matrix operation.
Since we use the linear predictive analysis, the parameters are
explicitly found and the stability can be easily guaranteed.
3.2. Approximation of warping by a matrix operation
With discrete signals, frequency warping is an irreversible operation, which involves a risk of unexpected transformations
when the signals are warped and inversely warped by simple
sinc interpolation. To reduce the risk, it is reasonable to findthe warping and inverse warping matrices in advance by optimizing for a training data set. For the measurement of thisoptimization, Itakura-Saito distance is selected, as in linear
prediction [8]. The objective function to minimize for optimizing the warping matrix W is defined as
)
∑(
Y
Yij
∑ ij
− log ∑
−1
(3)
k Wik Xkj
k Wik Xkj
i,jwhere X and Y respectively stands for the power spectra of
the training data set and the spectra warped by sinc interpolation. Subscript i and k indicate the number of frequency
bin, and j indicates the number of frame. The function abovemeans an Itakura-Saito distance of W X from Y . Since thespectra warped by W must still be power spectra, the elements of W have to be found in a non-negative condition.
This minimization problem cannot be solved explicitly, whichleads to a need for an implicit approach such as the auxiliary
function method [9] as follows.
First of all, we consider only the terms in the objectivefunction eq. (3) related to W and minimizing it:
)
(
∑
∑
∑
Yij
∑
+
Wik Xkj ≡ L(W ). (4)log
k Wik Xkj
i,j
i,j
k

Since the function f (x) =inequality holds as

1
x is convex in x > 0, Jensen’s

1
1
=∑
(5)
W
X
λ
(W
ik
kj
ijk
ik Xkj /λijk )
k
k
(
)
∑
∑
λijk
≤
λijk ≥ 0,
λijk = 1 .
Wik Xkj /λijk∑kk

In addition, the concavity of the logarithmic function leads to()∑∑
Wik Xkjlog, (ϕij > 0) .
Wik Xkj ≤ log ϕij + k
ϕij
k
(6)
Using both inequalities (5) and (6), the upper bound can be
set to the function L(W ) by an auxiliary function as
L(W ) ≤

∑

∑

λ2ijk
(7)
Wik Xkj
i,j
k
∑
)
∑(
k Wik Xkj+
log ϕij +≡ G(W )
ϕij
i,j
Yij

provided that the equality is attained if and only if the following equation holds∑
Wik Xkj
λijk = ∑, ϕij =
Wik Xkj .
(8)
k Wik Xkj
kBecause of the convexity, the auxiliary function G(W ) can be
minimized, with λ and ϕ both fixed, at the stationary point of
W which is found as∂G(W )|W =W̃
∂Wmn
(
) ∑
∑
1
Xnj
=
Ymj λ2mjn /Xnj −
=0
+
2
ϕmj
W̃
mn
j
j
√∑

⇐⇒ W̃mn =2
j Ymj λmjn /Xnj∑

j Xnj /ϕmj.

(9)

At last, by eqs. (8) and (9), λ, ϕ and W̃ are iteratively updated, which makes the objective function decrease monotonically until it ends up in a local optimum. This iteration canbe summarized as follows.
Respectively λ, ϕ are updated by W (l) , which indicatesW̃ in the lth iteration, as
∑ (l)
W Xkj
, ϕij =
Wik Xkj .
λijk = ∑ ik (l)kk Wik Xkj

3.3. Perceptual weighting for warped envelopes
As mentioned in section 2.1, envelopes extracted from the inputs must be smoothed by the perceptual weighting according
to equation (2). However, the weighting needs a modification to apply it to the resolution-warped envelopes statedabove. Applying the weighting eq. (2) means smoothing theenvelopes with the warped-frequency domain regarded asa linear frequency domain. This mismatch of the domains
causes a difference from the expectation in the smoothing
result. To distribute the quantization noise according to theshape of H̃k /Hk , which is known to be less annoying, γ has
to be modified for each frequency bin.
Here, we approximate the perceptual weighting as follows. Suppose there are N points in both the linear andwarped-frequency domains at equal intervals up to the Nyquist
frequency. Then, define k in eq. (2) as an index of frequency
in the warped domain and f (k) as an index corresponding to
k in the linear domain, provided f (0) = 0. Approximation of
the weighting can be performed as∑
H̃0 = 1/|1 +an γ n |(13)H̃k = 1/|1 +

∑

n

an γ

f (k)
k n

e

−j 2πkN n|, (k = 1, · · · , N − 1).

n

(l)

(10)

Substituting them into eq. (9) leads to W̃ in the (l + 1)th
iteration:
v
)2
(∑
u∑
(l)
(l)2
u
W
X
X
/
Y
W
mn
u
kj
nj
mj
k
j
mk
(l+1)
. (11)
Wmn
=t
∑
∑
(l)
k Wmk Xkj
j Xnj /
This results in the following updating rule:v
u∑2
u j Ymk Xnj /Ŷmj
(l+1)
(l) t
Wmn = Wmn
, Ŷ = W (l) X. (12)
∑
X
/
Ŷ
nj
mj
j
The updating rule to optimize the inverse warping matrix U
can also be derived by using a corresponding method to minimize Itakura-Saito distance of U W X from X.
The updating rules stated above are products of the respective elements and non-negative numbers. Accordingly,
the elements of W and U stay positive when the initial values
are positive. Moreover, the elements stay zero when the initial values are zero, which enables us to optimize W and U
while restricting the number of non-zero elements in the eachrow. This means the computational complexity of the warpingand the inverse warping operations are controllable. In termsof flexibility, this method is capable of designing warping at
will by changing the warped spectra Y of the training data.
In the following discussions, we use W and U as respectively the Mel-frequency scale warping matrix and the inverse
one, both optimized under the condition of having at mostseven elements in each row. Thus, the warping and the inverse warping needs only 7× (frame length) operations ofmultiplication each, meaning these two operations result in
little additional complexity.

Additionally, f (k) stated above can be approximated by(f (0), . . . , f (N − 1))T ≃ U (0, . . . , N − 1)T(14)

and the smoothing also works in this case.
4. EVALUATIONS
4.1. Fidelity of the envelopes
To evaluate the method described above, we compared envelopes extracted in different ways. For each frame in the
input signals, envelopes with resolution warped into the Melfrequency scale were extracted respectively by Mel-LPC
method and by the method proposed in an earlier section,using either the optimized matrix or sinc interpolation forfrequency warping. The warp matrix was optimized using training data set containing six minutes of audio andspeech signals other than the following test items. Frequency
warping by sinc interpolation simply warped the frequency
according to the Mel-frequency function and inverse warped
according to the inverse function.
Fig. 3 is the clear example for the comparison of the envelopes extracted from one frame by linear prediction andthe method using the optimized matrix. The spectrum is presented in linear-amplitude domain since it is quantized in thisdomain when coded. It is obvious that the resolution of theenvelope was warped: the fidelity of the envelope improvedin the lower band, which is more important for the coding, atthe cost of the higher band.
We also performed a quantitative evaluation. In eachframe of the test data, the improvements from linear prediction in Itakura-Saito distance were compared (Fig. 4). The
three methods had similar influence on the fidelity of theenvelopes. The comparison between the two warping methods reveals the contribution of the optimization to preventing
unexpected transformations in the higher band. Therefore itFig. 3. Comparison of the envelopes. The green dotted line
shows the spectrum of the input. The blue solid line and thered dashed line respectively indicate its envelopes extractedusing linear prediction and the proposed method. The LP order of the envelopes was 16.
can be said that the optimization succeeded in making an appropriate interpolation for the inverse warping. On the other
hand, compared with Mel-LPC method, the proposed method
showed almost the same performance.
4.2. Perceptual weighting
Secondly, we compared the perceptual weighting for envelopes with warped resolution. Generally, smoothing by
the weighting eq. (2) has more influence on the steep peaks
than on the gentle slopes of the envelopes. However, as
Fig. 5(a), for example, shows, the weighting failed to smooththe resolution-warped envelopes, with peaks remaining in thelower band. This problem arose because the weighting eq. (2)smoothed the envelope in the warped frequency domain instead of the linear domain. By modifying the weighting
according to eq. (13), as shown in Fig. 5(b), we were able
to smooth the envelope appropriately in accordance with thewarping. The local roughness in the smoothed envelopes
were caused by the approximation errors in the warping andinverse warping.
4.3. Subjective evaluation of sound quality
Finally, for subjective evaluation, we made a coder based on
TCX using the warped envelopes. Fig. 6 shows the outline ofthe process. The residual spectra of inputs were Golomb-Ricecoded [11] after scalar quantization, with the optimal Riceparameter estimated from their envelopes. Thus, the fidelityof the envelopes increases the efficiency of the entropy coding
as well as the efficiency of the distribution of the quantizednoise. Additionally, the harmonics of the inputs are detected
and transmitted, which roughly indicates in which frequency
the non-zeros are likely to be. This harmonics information
is used for modifying the Rice parameters and also for zero
run-length coding. This coder codes inputs for 320 samples
per frame and foresees 320 samples per frame at a 16-kHz
sampling rate. Thus 40 ms of algorithmic delay occurs.
We compared the performance of the coder explained
above using and without using the warped envelopes. Ten
seconds of audio signals each from six categories in the RWC

Fig. 4. The average and standard deviation of the improvements from linear prediction in each band. In the band where
the improvements are positive, the envelopes were closer totheir spectra in Itakura-Saito distance compared with the envelopes extracted by linear prediction. For the test data,
30 second segments from 15 music files in the RWC Music
Database were randomly selected. 16-kHz sampling rate, using 256 samples for each frame, and the LP order of the envelopes was 16. We used the Mel-LPC method in Speech
Signal Processing Toolkit [10].
music database, a harpsichord piece for classic, a synthesizer
piece for house, a piano piece for jazz, a guitar piece forpops, and male/female vocal pieces for vocal, were compressed into 16 kbps by the coder under the two conditions.
We also compressed the inputs by AMR-WB+ for a benchmark. AMR-WB+ compresses the inputs for 1024 samples
per frame at a 16-kHz sampling rate, and 72 ms of algorithmic
delay occurs.
ITU-R BS.1534-1 MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) was carried out for the evaluation. Seven participants were presented the original signals
for references, the signal compressed under each of the three
conditions, and 3.5-kHz band-limited ones for anchors. The
closeness of the signals to the references were graded fromzero to a hundred points.
Fig. 7(a) displays the item-wise difference of scores in the
MUSHRA test with the confidential intervals of 95 %. The
coder described above showed almost the same performanceas AMR-WB+ in scores. Fig. 7(b) shows the item-wise difference of the improvement in the scores by using the warping.
Indeed, the influence of the warping varied by the items, but
all the scores improved on average. Furthermore, in three outof the six items, qualities were enhanced at the significancelevel of 5 %. This results proves that the warping resulted in
an enhancement of the qualities.
5. SUMMARY
We devised a representation of envelopes with warped resolution, which approximates frequency warping and its inverseby sparse and non-negative matrices. The resolution-warped
envelopes showed almost the same fidelity as those of MelLPC method despite less computational complexity for extraction. In addition, we pointed out the necessity of modifying the perceptual weighting when applying it to the warped

(a) Smoothed by perceptual weighting eq. (2).

(a) MUSHRA scores. The three types of bars represents, respectively
from the left, TCX coder with warping, TCX coder without warping,and AMR-WB+.

(b) Smoothed by the modified weighting eq. (13).

Fig. 5. The resolution-warped envelope smoothed by eachmethod. The blue dashed line and the red solid line respectively indicate the envelope before and after smoothing. The
LP order of the envelopes was 16.

Fig. 6. Outline of the TCX based encoder.
envelopes. Moreover, the enhancement in subjective quality
by the warping was proven statistically.
A future issue is to verify under which conditions thewarping has greater influence on enhancing the quality of thecoding. We used the Mel-frequency scale for warping in thispaper, but the scale for the warping is also still left for further
modifications.
6. REFERENCES[1] 3GPP TS 26.290 version 11.0.0 Release 11, 3GPP, 2012.
[2] M. Neuendorf, et al., ”MPEG Unified Speech and Audio
Coding - The ISO/MPEG standard for high-efficiency audio coding of all content types”, AES, 2012.

(b) Improvements in the scores by using warping.

Fig. 7. Results of the subjective evaluation with confidence
interval of 95 %. Items with an asterisk in the label had adifference at the significance level of 5 %.
[3] S. Wabnik, et.al., ”Frequency warping in low delay audio
coding,” ICASSP, IEEE, vol.3, pp. /181- /184, 2005.
[4] H.W.Strube, ”Linear prediction on a warped frequency
scale,” J.Acoust.Soc.Am., vol.68, no.4, pp.1071-1076,
1980.
[5] K. Tokuda, ”Spectral estimation of speech by Melgeneralized cepstral analysis,” Electron. Commun. Japan,
vol.3, no.2, pp.30-43, 1993.
[6] K. Koishida, et al., ”A wideband CELP speech coder
at 16 kbit/s based on Mel-generalized cepstral analysis,”
ICASSP, IEEE, vol. 1, pp.161-164, 1998.
[7] G. Fuchs, et al., ”MDCT-Based coder for highly adaptive
speech and audio coding,” EUSIPCO, IEEE, pp.12641268, 2009.
[8] F. Itakura and S. Saito, ”A statistical method for estimation of speech spectral density and formant frequencies,”
Electron. Commun. Japan, vol. 53-A, pp. 36-43, 1970.
[9] H. Kameoka, et al., ”Selective amplifier of periodic
and non-periodic components in concurrent audio signals
with spectral control envelopes,” IPSJ SIG Technical Reports, vol.2006-MUS-66, pp.77-84, Aug. 2006.
[10] http://sp-tk.sourceforge.net/ (as of Jan. ’14).
[11] R. F. Rice, ”Some practical universal noiseless coding
techniques - part I-III,” JPL Technical Report, vol. JPL79-22, JPL-83-17, JPL-91-3, 1979, 1983, 1991.