Sparse modeling of musical instruments sounds in time-frequency domain
Hiromu Ogi(1) , Kohei Yatabe(2) , Yasuhiro Oikawa(3) , Yusuke Miyagi(4) , Koji Oishi(5)(1)

Department of Intermedia Art and Science, Waseda University, Japan, hiromu.ogi.6626@ruri.waseda.jp(2)

Department of Intermedia Art and Science, Waseda University, Japan, k.yatabe@asagi.waseda.jp(3)

Department of Intermedia Art and Science, Waseda University, Japan, yoikawa@waseda.jp(4)

KORG INC., Japan, miyagi@korg.co.jp(5)

KORG INC., Japan, koji@korg.co.jp

Abstract
Sample-based synthesis is one of the most common synthesis methods of the digital synthesizer. It can synthesize musical instrument sounds well because it plays recorded sounds instead of generating sounds such asadditive and frequency modulation syntheses. The recorded sounds are usually stored in the time domain. If
the recorded sounds are stored in the time-frequency domain, sound processing can be simplified. Users canprocess audio signals intuitively by operating the time-frequency bins of the spectrograms. However, the representation of sounds in the time-frequency domain usually requires more memory than that in the time domain.
In synthesizers, if much data is needed, hardware costs are increased correspondingly. Therefore, a techniqueto reduce the amount of data is required. In this paper, to reduce the amount of data, we introduce a sparsemodeling technique for musical instrument sounds in the time-frequency domain. We propose an optimization
algorithm of sparse modeling with four shrinkage operators. Numerical experiments show that the data quantityof the signals reduced by over 95 percent by using the technique.
Keywords: Sample-based synthesis, Spectrogram, Sparse modeling, Alternating direction methods of multipliers (ADMM)

1

INTRODUCTION

There are many synthesis methods of digital synthesizers, such as additive synthesis, subtractive synthesis, wavetable synthesis, frequency modulation synthesis, and sample-based synthesis [1, 2]. Especially,
sample-based synthesis is a powerful method when synthesizers imitate traditional musical instrument
sounds like piano, violin or trumpet. The synthesizers of this type have read-only memory (ROM) in
which the recorded sounds are stored, and synthesize sounds by playing them back. Generally, therecorded sounds are stored in the time domain.
If the recorded sounds are stored in the time-frequency domain, sound processing can be simplified.
Users can process audio signals intuitively by operating the time-frequency bins of the spectrograms
because time-frequency bins have information for a certain frequency band in a certain time. However,
the representation of sounds in the time-frequency domain usually requires more memory than that in
the time domain. In synthesizers, if much data is needed, hardware costs are increased correspondingly.
Therefore, a technique to reduce the amount of data without sacrificing quality is required.
It is expected that the time-frequency bins necessary to represent the musical instrument sounds exist
sparsely since sounds have harmonic structure. If it is possible to express the musical instrument soundsparsely, the reduction of the amount of data is realized.

6467

One of the effective tools for providing sparse representation is sparse modeling. This approach is
widely used in signal processing, image processing, and machine learning [3]. For effective modeling,
it is necessary to consider the characteristics of the signals and its transformation. Thus, we focus
on that the window function extends spectral peaks and introduce "social sparsity [4]" to the sparsemodeling for musical instrument sounds.
In this paper, we propose some algorithms of sparse modeling for musical instrument sounds in thetime-frequency domain. We also found that these algorithms are effective in the viewpoint of theamount of data compared with the usual time domain representation.

2

PRELIMINARIES

2.1 Time-frequency Representation
The short-time Fourier transform (STFT) is one of the most popular methods of the time-frequency
representation [5]. The discrete STFT is given by
X[n, m] =Xtwa [t − ∆n]x[t]e−i2π(t−∆n)m/L ,(1)

where x[t] is the time domain signal, wa is an analysis window, L is window length, ∆ is a window
shifting step parameter, and i is the imaginary unit. Inverse transformation of STFT (iSTFT) is also
given by
!
X
X
1 L−1
X[n, m]ei2π(t−∆n)m/L ,(2)
x̃[t] =
ws [t − ∆n]m
L m=0where ws is a synthesis window [6]. In STFT, one sample point in the time domain is analyzed by
Q = L/∆ windows. As a result, the time-frequency representation has Q times as many time-frequency
bins as the sample points of the time domain representation. In other words, STFT usually provides
redundant expression. If the signal is expressed redundantly, it is possible to represent it in various
ways. Thus, we propose algorithms to find the representation where the number of zeros is maximal(i.e., sparsest) in the time-frequency representation.
2.2 Sparse ModelingSparse modeling is an effective method to solve underdetermined problems, and it is possible to reduce
the amount of data significantly when signals admit sparse representation [3]. Studies based on thesparsity is widely conducted in acoustic engineering, including sound source separation [7–10], speechenhancement [11, 12] and restoration of the sound field [13–15].
Assuming that the musical instruments sound is x[t] and the observed musical instrument sound including noise is x̂[t], the time-frequency representation of x[t] is given by
Find X s.t. kF ∗ X − x̂k∞ ≤ ε ,(3)where kyk∞ = maxi |yi |, F ∗ is a operator which indicates iSTFT, and ε is a small constant whichindicates the tolerance of the error between x[t] and x̂[t]. This equation is an underdetermined system,
hence there are innumerable solutions to it.

6468

To get the sparsest solution, we should solve the optimization problem:min ιC (X) + λΛ(X) ,

(4)

X

(

ιC (X) =0(X ∈ C),∞ (otherwise)

C = {X ∈ CM ×N | kF ∗ X − bk∞ ≤ ε} .

(5)

The ideal function as Λ(X) giving a sparse solution is `0 norm defined as a function representing thenumber of nonzero elements of X [16]. However, it is difficult to find the optimal solution of thisminimization problem because it is combinational optimization of high-dimensional data. Therefore,the `0 norm is often relaxed with the convex function, `1 norm, to find the optimal solution [17–20].
2.3 Alternating Direction Methods of Multipliers (ADMM)
There are many algorithms for optimization problems. One of them is the alternating direction methods of multipliers (ADMM) [21]. Many optimization problems including the sparse modeling, can beinterpreted as the following minimization problem:
min f (x) + g(z) s.t. Kx + Lz = c ,(6)where f and g are real-valued cost functions, x ∈ RN , z ∈ RN , K ∈ RS×N , L ∈ RS×N , and c ∈ RS .
ADMM can solve the above minimization problem, and it is written as the following procedure:ρx= argmin f (x) + kKx + Lz [n] − c + u[n] k22
2
x


ρ
[n+1]
[n+1]
[n] 2
g
= argmin g(z) + kKx+ Lz − c + u k22zu[n+1] = u[n] + Kx[n+1] + Lz [n+1] − c ,
[n+1]





(7)
(8)
(9)

where ρ is a positive constant. The strengths of the ADMM is that it can be applied even if thecost function is not differentiable. By this fact, we can solve the optimization problem which has
non-differentiable functions such as `1 norm and `∞ norm. The effectiveness of ADMM has been
substantiate by many studies [22, 23].

3

PROPOSED METHODS

3.1 Proposed Algorithm
Applying ADMM to Eq. (4), it is formulated as
f (x) = ιC (x),g(z) = λΛ(z),K = I,

L = −I,

c=0,

(10)

where I is the identity matrix. Using Eq. (7) to Eq. (9), an algorithm for the sparse modeling isderived, and it is summarized in Algorithm 1. Here, sign(y) is the signum function,(sign(y) =y/|y|0(y 6= 0),(y = 0)

(11)

6469

Algorithm 1 Proposed Algorithm
Initialization: u[0] ∈ CM ×N ,repeatx0 ← F ∗ (z [k] − u[k] ) ;e ← x0 − b ;e ← sign(e) min{|e|, };x[k+1] ← (z [k] − u[k] ) + F (b + e − x0 ) ;z [k+1] ← P· (x[k] + u[k] ) ;u[k+1] ← u[k] + x[k+1] − z [k+1] ;k ←k+1;
until convergence;
F is a operator which indicates STFT, and P· (v) indicates the proximity operator of λΛ(X). The
proximity operator [24] is defined by
1
P· (v) = proxλΛ (v) = argmin λΛ(z) + kz − vk22 .
2
z




(12)

3.2 Thresholding and Shrinkage
3.2.1 Soft-Thresholding
Let Λ(X) be a `1 norm, P`1 is given by[P`1 (z)][n] = max {1 − λ/|z[n]| , 0} z[n] ,(13)and it is known as a soft-thresholding operator. `1 norm is used in various optimization problem since
it is a convex function [17–20].
3.2.2 Log-Thresholding
When Λ(X) is `1 norm, if there are large time-frequency bins in X, it is expected that these are
reduced by the optimization problem Eq. (4). For this reason, it has been proposed relaxing the effectof `1 norm using a logarithmic function [25, 26].
Let Λ(X) beΛ(X) = log(δ + |X|) ,(14)
where δ is a small positive constant, Plog is given by

 
p
1
2 − 2λ

(z[n]
−
δ)
+
(z[n]
+
δ)

2
 

p
[Plog (z)][n] = 12 (z[n] + δ) − (z[n] + δ)2 − 2λ




0

6470

(z[n] >
(z[n] <√√

2λ − δ)2λ − δ) .

(otherwise)(15)

Figure 1. Schematic diagram of social shrinkage

3.2.3 Social Shrinkage
In the time-frequency representation of instrumental sound, it is considered that energy is concentrated
in the grouped part of the spectrograms because the window function extends spectral peaks. Hence,
we propose to combine social sparsity [4, 27, 28]. This is sparse modeling in which the informationof surrounding time-frequency bins is added as a weight to take a decision: keeping or discarding thebin. The effects of this modeling are expected that the time-frequency bins of the spectrogram becomegrouped and noise which is not related to the signal is reduced. A schematic diagram of social sparsity
is shown in Fig. 1. Introducing the social sparsity, Psocial is given by[Psocial ](z)[n] = max{1 − λ/V w [n], 0} z[n] ,V w [n] =q(16)w ∗ |z[n]|2 ,(17)

where w is a non-negative kernel, and ∗ indicates the convolution operation. Compared to the Eq. (13),Eq. (16) can be regarded as a kind of soft-thresholding.
3.2.4 Social-Log Shrinkage
Finally, we propose combining social sparsity and log-thresholding. In Eq. (15), Psocial-log can be determined by replacing z[n] with V w [n], and Psocial-log is given by

 
p
1
w [n] − δ) +w [n] + δ)2 − 2λ

(V
(V
2 


p
[Psocial-log (z)][n] = 21 (V w [n] + δ) − (V w [n] + δ)2 − 2λ




0

(V w [n] >(V w [n] <√√2λ − δ)2λ − δ) .

(18)(otherwise)

By combining social sparsity and log-thresholding, it is expected that non-zero elements of spectrograms can be grouped while suppressing the effects of bias.

6471

0.075

880

0.075

0.07

0.07

-20

Frequency (normalized)

Frequency (Hz)Frequency (normalized)

0.06

660
0.055
0.05

0.06

-50

0.055

-60

0.05

550
0.045

0.045

0.04

0.04

-70

-80

-60

0.05

-70

0.045

-80

-90

440

0.04
-100

0.035

Non-zeroElements

22

44
66
Time (samples)

0.03

88 10 4
104

-90

22

0.06

-60

0.05

-70

0.045

-80

0.03

88 10 4
104

22

-30
-30

-40

-40
-40

0.06

-50

-50
-50

0.055

-60

-60
-60

0.05

-70

-70
-70

-80

-80
-80

-90

-90
-90

-100

-100
-100

0.045

-90

0.04

-100

0.035
-110

44
66
Time (samples)

-50

0.055

0.04

0.035
-110

0.03

-30

0.065
-40

-100

0.035

-20
-20

-30

-50

0.055

-20

0.07

0.065
-40

0.06

-10
-10

-20

-30

-40

-10

0.075

0.07

0.065

Frequency (normalized)

770

0.065

0.075

0.07
-30

0.08
-10

0.075
-20

0.065

0.08
-10

Frequency (normalized)

-10

0.035

4

22

-110
-110

-110

-110

0.03

44
66
88 10 4
Time (samples)
10
Time (samples)

Level (dB)

0.08

0.08

Frequency (normalized)

0.08

44
66
Time (samples)

0.03

88 10104
4

22

44
66
Time (samples)

88 10 4

1270385811 (4.57%)

4286 (3.37%)

5263 (4.14%)

4099 (3.23%)(Original)

(a)

(b)

(c)

(d)

104

Figure 2. Amplitude spectrograms and numbers of non-zero elements

4

EXPERIMENTS

Using the proposed method, we modeled the recorded piano sound to be a sparse representation in thetime-frequency domain. In the simulation, the target sound was piano sound A4 (442 Hz), the windowfunction was Hanning window, L was 212 sample, ∆ was 211 sample, λ was 0.1, ε was 3.92 × 10−7 ,w was a Gaussian kernel with a size of 7 rows and 3 columns, and iteration number was 3000. For
comparison, we performed sparse modeling in four cases: (a) soft-thresholding, (b) log-thresholding,(c) social shrinkage, and (d) social-log shrinkage. An expanded view of the amplitude spectrogramsand the number of non-zero elements of the spectrograms are shown in Fig. 2.
From the results, it was confirmed that a large amount of data was reduced by sparse modeling. In (a)and (b), non-zero elements were scattered, while in (c) and (d), non-zero elements were collectively
present due to the effect of social sparsity. In addition, we found that log-thresholding is able to reduce
more non-zero elements.

5

CONCLUSION

In this paper, we propose a method to represent instrument sounds sparsely in the time-frequency domain. The simulation shows that it is possible to reduce most of the data by sparse modeling, and the
combination of log-thresholding and social sparsity is effective. In future works, we plan to consider
the implementation of high-speed iSTFT using sparsity so that the synthesizer can sound in real timeeven if sounds are contained as time-frequency representation.

6472

REFERENCES[1] M. V. Mathews, F. R. Moore, and J. C. Risset, “Computers and future music,” Science 183, 263–
268 (1974).
[2] H. F. Olson and H. Belar, “Electronic music synthesizer,” The Journal of the Acoustical Societyof America 27, 595–612 (1955).
[3] A. Bruckstein, D. Donoho, and M. Elad, “From sparse solutions of systems of equations to sparsemodeling of signals and images,” SIAM Review 51, 34–81 (2009).
[4] M. Kowalski, K. Siedenburg, and M. Dörfler, “Social sparsity! neighborhood systems enrich structured shrinkage operators,” IEEE Trans. Signal Processing 61, 2498–2511 (2013).
[5] K. Yatabe, Y. Masuyama, T. Kusano, and Y. Oikawa, “Representation of complex spectrogram via
phase conversion,” Acoustical Science and Technology 40, 170–177 (2019).
[6] T. Kusano, Y. Masuyama, K. Yatabe, and Y. Oikawa, “Designing nearly tight window for improving time-frequency masking,” arXiv e-prints arXiv:1811.08783 (2018).
[7] T. Virtanen, “Monaural sound source separation by nonnegative matrix factorization with temporalcontinuity and sparseness criteria,” IEEE Transactions on Audio, Speech, and Language Processing
15, 1066–1074 (2007).
[8] H. Kameoka, M. Nakano, K. Ochiai, Y. Imoto, K. Kashino, and S. Sagayama, “Constrained andregularized variants of non-negative matrix factorization incorporating music-specific constraints,”
in “2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),”(2012), pp. 5365–5368.
[9] Y. Masuyama, K. Yatabe, and Y. Oikawa, “Phase-aware harmonic/percussive source separation
via convex optimization,” in “ICASSP 2019 - 2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP),” (2019), pp. 985–989.
[10] A. Hiruma, K. Yatabe, and Y. Oikawa, “Separating stereo audio mixture having no phase difference by convex clustering and disjointness map,” in “International Workshop on Acoustic Signal
Enhancement (IWAENC),” (2018), pp. 266–270.
[11] C. D. Sigg, T. Dikk, and J. M. Buhmann, “Speech enhancement with sparse coding in learned
dictionaries,” in “2010 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP),” (2010), pp. 4758–4761.
[12] J. F. Gemmeke, T. Virtanen, and A. Hurmalainen, “Exemplar-based sparse representations fornoise robust automatic speech recognition,” IEEE Transactions on Audio, Speech, and Language
Processing 19, 2067–2080 (2011).
[13] K. Yatabe and Y. Oikawa, “Optically visualized sound field reconstruction based on sparse selection of point sound sources,” in “2015 IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP),” (2015), pp. 504–508.

6473

[14] T. Tachikawa, K. Yatabe, and Y. Oikawa, “3d sound source localization based on coherenceadjusted monopole dictionary and modified convex clusterin,” Applied Acoustics 139, 267–281(2018).
[15] K. Kobayashi, D. Takeuchi, M. Iwamoto, K. Yatabe, and Y. Oikawa, “Parametric approximationof piano sound based on kautz model with sparse linear prediction,” in “2018 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP),” (2018), pp. 626–630.
[16] G. Su, J. Jin, Y. Gu, and J. Wang, “Performance analysis of l0 norm constraint least mean squarealgorithm,” IEEE Transactions on Signal Processing 60, 2223–2235 (2012).
[17] S. Chen and D. Donoho, “Basis pursuit,” in “Proceedings of 1994 28th Asilomar Conference on
Signals, Systems and Computers,” , vol. 1 (1994), vol. 1, pp. 41–44 vol.1.
[18] S. Chen, D. Donoho, and M. Saunders, “Atomic decomposition by basis pursuit,” SIAM Review
43, 129–159 (2001).
[19] D. L. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory 52, 1289–1306(2006).
[20] E. J. Candes and T. Tao, “Near-optimal signal recovery from random projections: Universal encoding strategies?” IEEE Transactions on Information Theory 52, 5406–5425 (2006).
[21] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statisticallearning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3, 1–122(2011).
[22] S. Diamond, R. Takapoui, and S. Boyd, “A general system for heuristic minimization of convex
functions over non-convex sets,” Optimization Methods and Software 33, 165–193 (2018).
[23] Y. Masuyama, K. Yatabe, and Y. Oikawa, “Griffin–Lim like phase recovery via alternating direction method of multipliers,” IEEE Signal Processing Letters 26, 184–188 (2019).
[24] N. Parikh and S. Boyd, “Proximal algorithms,” Found. Trends Optim. 1, 127–239 (2014).
[25] M. Fazel, H. Hindi, and S. P. Boyd, “A rank minimization heuristic with application to minimum
order system approximation,” in “Proceedings of the 2001 American Control Conference. (Cat.
No.01CH37148),” , vol. 6 (2001), vol. 6, pp. 4734–4739 vol.6.
[26] D. Malioutov and A. Aravkin, “Iterative log thresholding,” in “2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),” (2014), pp. 7198–7202.
[27] M. Kowalski and B. Torrésani, “Structured sparsity: from mixed norms to structured shrinkage,”
Proceeding of Signal Processing with Adaptive Sparse Structured Representations (SPARS) (2009).
[28] K. Siedenburg, M. Kowalski, and M. Dörfler, “Audio declipping with social sparsity,” in “2014IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),” (2014),
pp. 1577–1581.

6474