Amplitude Spectrogram Prediction from Mel-Frequency Cepstrum Coefﬁcientsand Loudness Using Deep Neural Networks
Shoya Kawaguchi1 and Daichi Kitamura1
1National Institute of Technology, Kagawa College
355 Chokushi, Takamatsu, Kagawa 761–8058, Japan

Abstract
Timbre conversion of musical instrument sounds utilizing
deep neural networks (DNNs) has been the subject of extensive research and continues to elicit signiﬁcant interest in thedevelopment of more advanced techniques. We aim to propose a novel algorithm for timbre conversion utilizing a variational autoencoder. However, this system must possess thecapability of predicting the amplitude spectrogram from themel-frequency cepstrum coefﬁcient (MFCC) and loudness.
The present research aims to build a DNN-based decoder thatutilizes the MFCC and loudness as inputs to predict the amplitude spectrogram. Experiments using a musical instrument
sound dataset indicate that a decoder incorporating bidirectional long short-term memory yields accurate predictions ofamplitude spectrograms.
1. Introduction
The generation of musical instrument sounds and the conversion of timbre have been the focus of extensive researchin recent years. Various techniques based on deep neural networks (DNNs) have been proposed to tackle this problem.
For example, differentiable digital signal processing [1] is asynthesizer of musical instrument sounds, which utilizes multiple sinusoidal waves and ﬁltered noise, whose parameters
are generated by pre-trained DNNs. Another approach involves the use of a variational autoencoder (VAE) [2] to generate musical instrument sounds [3, 4, 5]. In this method,
VAE is used to extract disentangled latent features of pitchand timbre.
Similar to the aforementioned approach, this paper also
focuses on a VAE-based system for the timbre conversionof musical instruments. Currently, we are in the process ofdeveloping a new timbre conversion system, as depicted inFig. 1. In this system, we propose to incorporate the three fundamental elements of musical instrument sounds: pitch, timbre, and volume. To represent these features, we employ traditional, well-established features: note number as a parameter of pitch, mel-frequency cepstrum coefﬁcient (MFCC) [6]
as a parameter of timbre, and loudness as a parameter of vol-Figure 1: Process ﬂow of proposed timbre conversion systemume. The timbre conversion of an input sound is achieved bymodifying only the MFCC based on the pre-trained VAE. We
anticipate that the proposed system will enable the interpolation of timbre across multiple types of musical instruments,
thus contributing to the advancement of new art and music.
The proposed system (Fig. 1) requires a decoding processthat involves the calculation of an amplitude spectrogramfrom the pitch, manipulated timbre, and volume. This decoder cannot be realized in an analytical manner. To solve
this problem, in this paper, we propose the utilization of three
DNN architectures and evaluate the suitable DNN for predicting amplitude spectrograms.
2. Proposed system and its DNN decoder
2.1 Overview of proposed system
As depicted in Fig. 1, the proposed system ﬁrst computes
an amplitude spectrogram of the input sound via a short-time
Fourier transform (STFT). Subsequently, the pitch, MFCC,and loudness are extracted. While the pitch can be extracted using various techniques, including DNN-based methods such as [7], the MFCC and loudness are obtained throughdeterministic calculations. In the proposed system, it is assumed that the MFCC is a reliable feature that represents
the timbre of the sound and is disentangled from the pitchand volume. To train the timbre-embedded latent space, theMFCCs of various from multiple musical instruments are
input into the VAE. This training enables us to manipulate
only the timbre of the sounds after the training of the entire proposed system. Finally, the pitch, manipulated MFCC,

Figure 2: Training process ﬂow of proposed DNN-based timbre decoderand loudness are decoded to the amplitude spectrogram, andthe waveform of the generated sound is obtained via inverse
STFT and phase recovery techniques.
2.2 Motivations and contributions of this paper
In the proposed system, an output of the VAE (generatedMFCC) must be decoded into an amplitude spectrogram using pitch and loudness. Since MFCC is a dimensionalityreduced feature, this decoding cannot be achieved throughlinear operations or analytical manners. To circumvent thisissue, we employ a DNN as the decoder of the proposed system, namely, the DNN decoder predicts an amplitude spectrogram of synthesized sound from the inputted pitch, MFCC,and loudness. In this paper, we investigate a suitable architecture for this DNN decoder and evaluate its accuracy. The
evaluation of the entire system of Fig. 1 is our future work.
The procedure for training the DNN decoder is depicted
in Fig. 2. As the decoder, multiple DNNs are independently
prepared and trained for each sound note number (from C3 toB5). Thus, the estimated pitch is utilized to select the pitchspeciﬁc DNN, and MFCC and loudness are inputted into theselected one. Thanks to this pitchwise model training, a generalization capability for pitch is not required for each DNN,
resulting in a high prediction performance.
In this paper, we conduct experimental investigations to determine the suitable DNN architecture for the DNN decoder.
The performance of multilayer perception (MLP), bidirectional gated recurrent unit (BiGRU) [8], and bidirectional
long short-term memory (BiLSTM) [9] are compared. BiGRU and BiLSTM are referred to as bidirectional recurrent
neural networks (BiRNNs) [10], which can effectively capture latent structures of time-series data.
2.3 Architecture of DNN decoders
I×J
Let Y ∈ R≥0and yij be the amplitude spectrogram of
an input sound and its element at frequency i and time j, respectively, as obtained∑I by STFT. The loudness of this signal iscomputed by vj = i=1 (yij + ε), where ε represents a smallvalue to avoid zero-division. The amplitude spectrogram isFigure 3: Architecture of BiRNN used as DNN decoder
normalized using the loudness as y ij = yij /vj . Then, theMFCC is computed with the normalized power spectrogram
I×JP ∈ R≥0, whose element is y 2ij . This MFCC is denoted as
C ∈ RK×J , where K is the order of MFCC, which is equal
to the number of ﬁlters in the bandpass ﬁlter called the melﬁlter bank used in the MFCC computation. The input data ofthe DNN decoder is deﬁned as[ ]C
X = T ∈ R(K+1)×J ,(1)vwhere v = [v1 , v2 , · · · , vJ ]T ∈ RJ .
For the training of MLP, we vectorize X and input thevector into the ﬁrst layer. The label (reference of the DNNprediction) is deﬁned as y ∈ RIJ≥0 , a vectorized version of
the original amplitude spectrogram Y . This MLP consists ofthree fully connected hidden layers with 1024–512–512 dimensions, and each hidden layer has a rectiﬁed linear unit as
the activation function.
The network architecture of BiGRU or BiLSTM is illustrated in Fig. 3. The column vectors of X, denoted as
xj ∈ RK+1 , are input into the ﬁrst BiRNN layer. Thisbidirectional computation is applied four times, acquiring the(f)(f)(f)
forward output vectors h1 , h2 , · · · , hJ ∈ RI≥0 and the(b)

(b)

(b)

backward output vectors hJ , hJ−1 , · · · , h1 ∈ RI≥0 . The
dimension of each vector is increased from K + 1 to I at theﬁrst BiRNN layer and is maintained throughout the remaining layers. Finally, the predicted amplitude spectrogram iscomposed as
I×JŶ = [h1 h2 · · · hJ ] ∈ R≥ε,()(f)(b)
hj = max hj ⊙ hj , ε ∀j,(2)(3)

where ⊙ denotes the Hadamard product and max(·, ·) returnsthe maximum value of inputs in each element.
All the DNNs are trained by minimizing the mean squarederror ∥y − ŷ∥22 , where ŷ ∈ RIJ≥ε is a vectorized version of theTable 1: Experimental conditions
Window and shift lengths in STFT
64/32 ms
Window function in STFT
Hann window
Maximum frequency of mel ﬁlter bank
8 kHz
Minimum frequency of mel ﬁlter bank
0 kHz
Number of mel-ﬁlters (K)64
Flooring value ε
2.0 × 10−7

Figure 5: Example of spectrograms for test data (synth. keyboard sound): (a) original, (b) predicted by MLP, (c) predicted by BiGRU, and (d) predicted by BiLSTM

Figure 4: Example of spectrograms for test data (acousticﬂute sound): (a) original, (b) predicted by MLP, (c) predicted
by BiGRU, and (d) predicted by BiLSTM
predicted amplitude spectrogram Ŷ and ∥ · ∥2 is the L2 norm.

respectively, where ŷij and ckj are the elements of Ŷ and C,
respectively, and ĉkj is the MFCC calculated from the predicted amplitude spectrogram Ŷ . Small values of ARSE andMRSE indicate accurate prediction performance. It should be
noted that for calculating MRSE, we only used the MFCCs
from k = 2 to 14, as these dimensions speciﬁcally encompass the timbre characteristics.
3.2 Results

Figs. 4, 5, and 6 show examples of original and predicted
amplitude spectrograms. These results conﬁrm that the MLP
consistently fails to predict the amplitude spectrogram: the3.1 Dataset and conditions
predicted results contain numerous spectral holes, resulting
To evaluate the performance of the DNN decoder, we in artiﬁcial distortions. Conversely, BiGRU and BiLSTM acconducted an experiment on amplitude spectrogram predic- curately predict the harmonic structures and temporal transition using the neural audio synthesis (Nsynth) dataset [11]. tions within the original amplitude spectrogram. This is due
Nsynth is an audio dataset comprising four-second-long sig- to the recurrent architecture in BiGRU and BiLSTM, whichnals of various musical instrument sounds and consists of effectively captures the time-series structures of the input
305,979 signals. These signals were split into 289,205 train- data. Also, Fig. 7 shows ARSE and MRSE results averaged
ing, 12,678 validation, and 4,096 test data. The other exper- over all test data for each musical instrument. In both scores,imental conditions are shown in Table 1. As evaluation cri- BiLSTM consistently provides the best result for all the interia, we used amplitude relative squared error (ARSE) and struments. The results for mallets are inferior as compared toother musical instruments. This is likely due to the fact thatMFCC relative squared error (MRSE), deﬁned as
mallets are typically classiﬁed as percussion instruments, and∑J ∑I
2their sounds are characterized by complex harmonic strucj=1
i=1 (yij − ŷij )[dB],(4) tures and time transitions, as shown in Fig. 6. From theseARSE = 10 log10
∑J ∑I
2
j=1
i=1 yij
results, we found that BiLSTM provides satisfactory perfor∑J ∑142
mance in predicting the amplitude spectrogram and, thus, is(c−ĉ)kj
j=1
k=2 kj
MRSE = 10 log10[dB], (5) a preferred choice for the DNN decoder in the proposed sys∑J ∑14 2
j=1
k=2 ckjtem.
3. Experimental evaluation of DNN decoders

Figure 6: Example of spectrograms for test data (acousticmallet sound): (a) original, (b) predicted by MLP, (c) predicted by BiGRU, and (d) predicted by BiLSTM

MLP

BiGRU

BiLSTM

MLP

BiGRU

BiLSTM

Figure 7: (a) ARSE and (b) MRSE results averaged over test
data for each musical instrument
and timbre for isolated musical instrument sounds,” inProc. ISMIR, pp 700–707, 2020.

4. Conclusions
We examined the efﬁcacy of predicting amplitude spectrograms from MFCC and loudness using DNNs. Experiments using the Nsynth dataset showed that BiLSTM consistently outperforms MLP and BiGRU models. In future work,
we intend to construct the proposed sound generation system(Fig. 1) utilizing the BiLSTM-based DNN decoder.
Acknowledgment
This work was partially supported by JSPS KAKENHI Grant
number 22H03652 and Ono Charitable Trust for Acoustics.
References[1] J. Engel, L. Hantrakul, C. Gu, and A. Roberts, “Differentiable Digital Signal Processing,” in Proc. ICLR,
2020.
[2] D. P. Kingma, and M. Welling, “Auto-Encoder Variational Bayes”, in Proc. ICLR, 2014.

[5] K. Tanaka, R. Nishikimi, Y. Bando, K. Yoshii, andS. Morishima, “Pitch-timbre disentanglement of musical instrument sounds based on VAE-based metriclearning,” in Proc. ICASSP, pp. 111–115, 2021.
[6] F. Zheng, G. Zhang, and Z. Song, “Comparison ofdifferent implementations of MFCC,” J. Comput. Sci.
Technol., vol. 16, pp. 582–589, 2001.
[7] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “CREPE:
A convolutional representation for pitch estimation,” inProc. ICASSP, pp. 161–165, 2018.
[8] K. Cho, B. V. Merriënboer, C. Gulcehre, D. Bahdanau,F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder
for statistical machine translation,” arXiv: 1406.1078,2014.
[9] K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and J. Schmidhuber, “LSTM: A search spaceodyssey,” IEEE Trans. Neural Netw. Learn. Syst.,
vol. 28, no. 10, pp. 2222–2232, 2016.

[3] Y. J. Luo, K. Agres, and D. Herremans, “Learning dis- [10] M. Schuster and K. K. Paliwal, “Bidirectional recurrent
entangled representations of timbre and pitch for muneural networks,” IEEE Trans. Signal Process., vol. 45,
sical instrument sounds using Gaussian mixture varino. 11, pp. 2673–2681, 1997.
ational autoencoders,” in Proc. ISMIR, pp 746–753,[11] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck,
2019.
K. Simonyan, and M. Norouzi, “Neural audio synthesis[4] Y. J. Luo, K. W. Cheuk, T. Nakano, M. Goto, andof musical notes with WaveNet autoencoders,” in Proc.
D. Herremans, “Unsupervised disentanglement of pitchICML, pp. 1068–1077, 2017.