Feasibility Study of Deep Frequency Modulation
Synthesis
Keiji Hirata1⋆ , Masatoshi Hamanaka2 , and Satoshi Tojo3
1

Future University Hakodate hirata@fun.ac.jp
2
Riken AIP masatoshi.hamanaka@riken.jp
3
JAIST tojo@jaist.ac.jp

Abstract. Deep Frequency Modulation (FM) synthesis is the methodof generating approximate or new waveforms by the network inspiredby the conventional FM synthesis. The features of the method include
that the activation functions of the network are all vibrating ones with
distinct parameters and every activation function (oscillator unit) shares
an identical time t. The network learns a training waveform given in thetemporal interval designated by time t and generates an approximating
waveform in the interval. As the first step of the feasibility study, we
examine the basic performances and potential of the deep FM synthesis
in small-sized experiments. We have confirmed that the optimization
techniques developed for the conventional neural networks is applicableto the deep FM synthesis in small-sized experiments.
Keywords: Frequency modulation synthesis · neural networks · activation function · backpropagation.

1

Introduction

Frequency Modulation (FM) synthesis is a well-known technique for generating musical sound [1] and has been employed for many commercial products
of digital synthesizers such as DX7 of Yamaha, which is one of the bestsellingsynthesizers [7]. Also many variations of the FM synthesis have been developed[6, pp.224-250]). FM synthesis can generate rich sounds despite a simple configuration, i.e., the small number of parameters; on the other hand, it is known
that FM synthesis requires some skills for manipulating parameters when generating new desired sounds. It is mainly because the relationships among output
sounds, parameter values, and configurations of connecting oscillators are not
suﬃciently intuitive. Hence, to create new sounds easily, many of digital synthesizers employing the FM synthesis provides the presets which are built-in
connection patterns of oscillators with predefined parameters of amplitudes andcarrier and modulating frequencies.
After considering these presets provided, we would come up with an idea ofa general form of the FM synthesis which looks like a neural network, the activation functions of which are oscillators. Suppose, in the network, an oscillator⋆

This work has been supported by JSPS Kakenhi 16H01744.

2

K. Hirata et al.

X at a layer can receive modulating waveforms from the ones at one-level lower
layer Y1 , Y2 , · · ·. The weight of the connection between X with Yk corresponds
to amplitude parameters. The carrier and modulating frequencies are definedas parameters within each oscillator. Then, all the oscillators should refer to an
identical current time and simultaneously generate waveforms along with thetime as in the conventional FM synthesis. If we would apply the learning techniques developed for conventional neural networks to the network inspired by theconventional FM synthesis, we might generate target sounds without taking careof the relationships among output sounds, parameter values, and configurationsof connecting oscillators.
Gashler and Ashmore [2] have surveyed various networks to model and predict time-series data and oﬀered a useful idea for categorization of approaches.
Gashler and Ashmore claim that at a high level, the neural networks to predicttime-series data are broadly categorized into three major approaches: here we refer to them as WaveNet approach, RNN approach, and extrapolation approach.
Among them, there have been proposed several neural networks that belong to
the extrapolation approach and employ vibrating activation functions such as asinusoidal function and wavelet [2, 5, 3]. In these architectures, the current time
for generating a waveform is treated as the explicit input given at an input layer.
That is, neither all the activation functions (i.e., oscillators) share the current
time, nor the activation functions at the hidden layers refer to it.
In the paper, we propose deep Frequency Modulation (FM) synthesis, whichis the method of generating an approximating waveform based on the network
inspired by the conventional FM synthesis. According to the Gashler and Ashmore’s categorization, the deep FM synthesis basically belongs to the extrapolation approach. Thus, we hope the deep FM sysntesis could generate unknown
yet good sounds by extrapolating already existing sounds in a diﬀerent way fromprevalent sound generation methods such as sampling and WaveNet. To study
the feasibility and utility of the deep FM synthesis, we investigate the basic
characteristics and performances of it; for instance, how accurate the deep FM
synthesis can approximate a target waveform, what size of the network we needfor reconstructing a target waveform, how and what conventional techniques for
optimizing networks can be applied to the deep FM synthesis, and so on.

2

Deep Frequency Modulation Synthesis

2.1

Architecture

For theoretical consideration, we think of a simple, typical architecture shownin Figure 1, which presents how oscillator units are interconnected with weights
w; the depth is M + 1, the width is N + 1, and Σ w stands for weighted sum. The
input to the network is vector {A0 , A1 , · · · AN }, and the output is waveform S.
For an oscillator unit by a typical vibrating function, we here adopt a sinusoidal
function y = sin 2π((x + c) t + p) with input x (the bottom of the oscillator unit
in the figure) and output y (the top). Each oscillator unit has two parametersc and p to be tuned corresponding to frequency and phase, respectively. All

Feasibility Study of Deep Frequency Modulation Synthesis

3

ܵ
‫ݐ‬

‫ݐ‬

ȭ‫ݓ‬

osc
ȭ‫ݓ‬
osc
ȭ‫ݓ‬

ܿM,0
‫݌‬M,0

‫ݐ‬

ȭ‫ݓ‬

‫ݐ‬
ܿM−1,0 osc‫݌‬M−1,0
ȭ‫ݓ‬

‫Ͳܣ‬

ܿ0,0
‫݌‬0,0

‫ݐ‬

osc

‫ͳܣ‬



ܿM−1,1 
‫݌‬M−1,1‫ݐ‬‫ݐ‬

ȭ‫ݓ‬
osc

ȭ‫ݓ‬



ܿ0,1‫݌‬0,1osc

ܿM,N‫݌‬M,NܿM−1,N‫݌‬M−1,N
Training waveform T

osc

ܿM,1‫݌‬M,1



‫ݐ‬

osc

‫ݐ‬

osc

‫ܰܣ‬

ܿ0,N
‫݌‬0,N

𝑇𝑇s

𝑇𝑇𝑒𝑒

𝑡𝑡

Output waveform S

Fig. 1. Network Configuration of Deep FM Synthesis

oscillator units share the identical timing signal t, which moves between startingtime Ts and ending time Te , to compute the output waveform (the red solid
curve in Figure 1). The network attempts to fit the output waveform to thetraining waveform only between Ts and Te (dark blue). Thus, in the ranges outof the interval between Ts and Te , the network does not take care of the output
waveform (red dashed curves).
The forward propagation in the deep FM synthesis works as conventional
neural networks; at layer n, input waveform x is given to each oscillator unit tocompute output y with parameters cm,n and pm,n at time t. Then, the output
waveforms calculated at layer n, that is y, are summed up with weights, and thesum is provided to the input waveform at layer n + 1 as x. An output waveform
is made of the series of values St , which represent the samples at time instantt between Ts and Te . In other words, given time t at which we want to obtainvalue St , the network calculates St in a bottom-up manner.
We may assume that all the elements of the input vector {A0 , A1 , · · · AN }are constant values. The assumption is justified by the design decision we madethat the deep FM synthesis works as a multiple-wave generator depending on the
input vector. In reality, the input vector can be made of either constant values
or any waveforms synchronized by the timing signal t. Theoretically, the input
vector can be either constants or any waveforms synchronized by the timing
signal t, since giving waveforms to the input is equivalent to the extension of the
network in the direction of depth with constants given to the input. Therefore,

4

K. Hirata et al.

the multi-layered architecture absorbed such subtle diﬀerences, and we can put
the assumption without loss of generality.
2.2

Backpropagation

We would apply the standard backpropagation technique to optimize the deep
FM synthesis as follows [4]. For notational simplification, we assume the network
size is depth M + 1 by width N + 1, and the width is unchanged from the top
layer to the bottom. The final output of the deep FM synthesis at time t isdenoted as St . The input and output of the n-th oscillator unit at layer m aredenoted as xm,n and ym,n , respectively. The weight between adjacent layers isdenoted as wm,n′ ,n , which stands for the weight from n′ -th unit at layer m ton-th unit at layer m + 1. Only at the topmost layer, we write wM,n , omitting
the second n. Then, the final output is straightforwardly defined in a topdown
manner:N∑
St =wM,n · yM,n(1)
n=0

For m = M .. 1 and n = 0 .. N, we define ym,n = sin 2π((xm,n + cm,n )t + pm,n )∑N
and xm,n = n′ =0 wm−1,n′ ,n · ym−1,n′ . For m = 0 and n = 0 .. N (the bottom
layer), we define y0,n = sin(x0,n t + p0,n ), and x0,n = An . We always put c0,n = 0
because the elements of the input {A0 , A1 , · · · AN } are all assigned to constantvalues.
A single network of deep FM synthesis is trained, considering the set of time
instants within the designated period between Ts and Te . Let us denote thetraining waveform (target waveform) at time t as Tt . Then, the loss function isdefined as follows:∑∑1(St − Tt )2 ,
(2)
E=
Et =
2
t
t
∑
where t means the summation over the set of the time instants (Ts ≤ t ≤ Te ).
We present the gradient descent method for optimizing the network [4]; let
us compute the partial diﬀerential of the loss in Equation (2) with respect toeach parameter contained in the network in the standard manner. First of all,for the topmost weights wM,n , from Equation (1) we have
∂Et
∂Et
∂St
=
·
= (St − Tt ) · yM,n∂wM,n∂St ∂wM,nNote that although the same timing signal t is provided to all oscillator units, thegradient of the loss can be derived as in conventional neural networks. Also fortt
the parameters within each oscillator, ∂c∂Eand ∂p∂E
can be derived similarly.
M,nM,nThen, to simply express the derivatives of ym,n , we introduce term cos 2π((xm,n +cm,n )t + pm,n ) and denote it as zm,n . Due to space limitation, we omit thetechnical details and show only the result of parameter optimization. We derive
the following entire inductive definition of the gradient chain Γ and the feedback
values for gradient descent:Feasibility Study of Deep Frequency Modulation Synthesis

5A
Base step: ΓM,n
= (St − Tt ) · wM,nInduction step:

A
Γm,n
=

N
∑

B
Γm+1,i· wm,n,ii=0BAΓm,n = Γm,n
· 2πt zm,n(m = 0 .. M-1)

(m = 0 .. M)(3)

Using the series of the gradient chain above, we obtain the partial diﬀerentialsof the parameters as follows:∂Et= (St − Tt ) · yM,n
∂wM,n
∂Et
B
= Γm+1,n· ym,n′ (m = 0 .. M-1)
∂wm,n′ ,n∂EtA= Γm,n· 2πt zm,n (m = 0 .. M)
∂cm,n
∂Et
A
= Γm,n· 2π zm,n (m = 0 .. M)
∂pm,n

3

Experiments and Results

3.1

Implementation

Following the standard backpropagation techniques [4], for optimization, we employ stochastic gradient descent, Adam, and L2 regularization with soft threshold. Within an epoch, as many time instants at which the loss is calculated as thesize of the mini-batch are generated by the uniform random number generator
over the designated temporal interval. In the following experiments, the minibatch size is always set to 5. For simplicity, all parameters c’s, p’s, and w’s are
initialized by the normal distribution with the average being 0.0 and the variance being 0.1. Each layer, consisting of the oscillator units, is fully-connectedto adjacent layers.
Here, we give a notice in setting the period of time t to preserve stability
for the deep FM synthesis. As some may already noticed in Equation (3), thegradient chain inevitably includes term tn , where n is the depth from the top
layer. It follows that the amount of the loss feedback is proportional to tn .
Thus, if 0.0 < t < 1.0, tn may always become almost 0.0; on the other hand, if
t > 1.0, tn may become a larger value. Therefore, in the following experiments,
the period of time t is put from 1.0 to 2.0. These values have been determinedthrough several trials we made.
3.2

Loss Convergence

At the very first step, we would check if the backpropagation technique introduced in the previous section can work for the deep FM synthesis. Figure 2 shows

6

K. Hirata et al.

the loss convergences calculated by mean squared error as the epoch increases(Equation (2)), when the network size is depth 5 by width 5 and the training
waveforms are rectangle and sawtooth with two cycles in the period of 1.0 to
2.0.
Loss
Loss

Rectangle

Epoch

Sawtooth

EpochFig. 2. Loss Convergences along with Epoch

Figure 3 shows the intermediate waveforms generated by the network at
epochs 0, 500, and 4400 for rectangle and at epochs 0, 1000, and 8400 for sawtooth, respectively. Note that the training and output waveforms are shown only
between Ts and Te (i.e., 1.0 and 2.0). To obtain the above results, it took several
minutes or less for each training phase, using Surface Pro 3 featuring Intel Core
i7 CPU @ 1.70 GHz.
3.3

Network Size

We examine the relationship between the network size (depth × width) andthe loss. The left-hand graph in Figure 4 shows the results for rectangle as thetraining waveform, and the right-hand graph sawtooth. In the both graphs, thedepth is changed from 2 to 5 (colored broken lines), and the width from 1 to 8
(horizontal axes).
The loss in sawtooth converges faster along with the width increased thanrectangle. At present, we presume the result could be understood by the complexity of a waveform as a figure. For example, while a cycle of rectangle contains
two steep changes (-1.0 to 1.0 and 1.0 to -1.0), that of sawtooth contains one(1.0 to -1.0).
3.4

Training by Multiple Waves

We are interested in the learnability of the network, that is, how many wave
forms the network can learn at the same time. Then, for training, the networkis given the more than one pair of input vector and output wave form. In theexperiment, during an epoch, three trainings are performed. During a single

Feasibility Study of Deep Frequency Modulation Synthesis




















WLPH

At Epoch 0 (Initialized)

1.5

0.15

3

1

4

5

2

6

3

7

4

5

8

0

depth=2
depth=3
depth=4
depth=5

depth=5

1

2

0

3

0.1



0.06




WLPH

2

3

3
6

4
7

5
8

6

7

8

6

7

4

8

5

6

Width

0

7

depth=58

depth=2

depth=2

depth=3

depth=3

depth=4

depth=4

depth=5

depth=5

0.06
0.04
0.02

1

2

Width

Trained by Rectangle with
Changing Network Size

5

Width

3

0.08

0.08

0.02
2
5

2

0.1

0.04

1
4

4

1

0.12

0.1

depth=4

depth=4

Loss
0.12

depth=3

depth=30.02

1depth=2

depth=2
At Epoch 8400

0.04

0.06

0.1

0.05

0.05



0.08

0.04

8

depth=4

0.15

LOSS

LOSS

7

depth=30.2

0.1

depth=4

depth=2

0.25

0.15

6

Width

Loss

0.2

depth=3

0.12

0.02
0
Fig. 3. Waveforms
Generateddepth=5
During Training
for Rectangle
and Sawtooth depth=5
depth=5

Width

0.25

0.08

At Epoch 1000
0.06

depth=402depth=3

Width vs Loss

Width vs Loss


WLPH

0.1

LOSS

0.05


WLPH



LOSS

0.05

depth=2

depth=2

At Epoch
0 (Initialized)
0.1

0.1

 0.12



Loss

0.15



2.0
time

sawtoothsawtooth

Loss

0.2

1.0

0.2

Loss

Width vs Loss

Width vs Loss







0.25

-1.0

0



At Epoch 4400

Loss

rectangle

0.0

1


WLPH



rectangle0At Epoch 500

1.0

0.25







7

0

3

1
4

2
5

3
6

4
7

5
8

Width
Trained by Sawtooth with
Changing Network SizeFig. 4. Network Size and Losstraining, the network is given a pair of input and output by a mini-batch. The
first pair is made of input vector of {1, 0, 0, · · ·} and output signal of two-cycle

6

7

8

8

K. Hirata et al.

rectangle; the second {0, 1, 0, · · ·} two-cycle sawtooth, and the third {0, 0, 1, · · ·}two-cycle sinusoid.
Figure 5 shows the results of training the three pairs; the waves generated by
the deep FM synthesis are drawn in the colored curves, and the training waves
in black. We use the network of depth 5 by width 5.
























WLPH






WLPH






WLPH

Fig. 5. Three Waves Generated by Network When Given Three Distinct Input Vectors

The leftmost graph in the figure is obtained at epoch 23000 with loss of 6.57
×10−2 (near the optimal point for the rectangle), when given input {1, 0, 0, · · ·}.
Similarly in the middle for sawtooth, the graph is obtained at epoch 21370 withloss of 2.12 ×10−2 , and for sinusoid, at epoch 11680 with loss of 6.62 ×10−3 .
For instance, let us compare the leftmost colored graph in Figure 5 with theupper rightmost in Figure 3 (“At Epoch 4400”). It seems that the generated
wave form in the former is hardly deteriorated, even if that network learns three
diﬀerent wave forms at the same time, however, the epochs required for training
is about 5 times larger.

4

Concluding Remarks

We propose the deep FM synthesis which is inspired by the conventional FM
synthesis; it has the network architecture like neural networks and can be optimized by the backpropagation technique as neural networks. We have demonstrated that the deep FM synthesis works well to some extent for small-sized
artificial training waveforms. Figure 6 shows an example when an unsteady, alittle complicated training waveform (black curve in the figure) is given; the wave
length and amplitude of it is varied along time. The horizontal axis in the figure
stands for time in the unit of time instant. The conditional behavior presented
in Section 3.4 is also promising for reconstructing multiple training waves.
The network attempts to approximate the output waveform (blue) to thetraining waveform; the network size is here depth 4 by width 20, and the network
achieves the loss of 0.0583 at epoch 7570. However, the realistic waveforms ofnatural tones of acoustic instruments such as pianos or flutes are far longer

Feasibility Study of Deep Frequency Modulation Synthesis

9and contain many cycles. At present, the network of the deep FM synthesis
cannot properly learn and reconstruct such real waveforms, unfortunately. The
output waveform of the network does not suﬃciently converge on a given training
waveform under the current optimization method.

Fig. 6. Approximating Unsteady Waveform

Future work will include improving the approximation to the simple waveforms as given in Sections 3.2 and 3.3, and developing a tractable optimization
method that can work eﬀectivtly when learning realistic, long, complicated waveforms such as natural tones of acoustic instruments and voices. For the purpose,
we would investigate other vibrating functions to be used for an activation function which must be diﬀerentiable and not necessarily periodic such as wavelet
and phase modulator.

Acknowledgments
This work has been supported by JSPS Kakenhi 16H01744. The authors would
like to thank to Prof. Ichiro Fujinaga of McGill University, Mr. Adrien Ycart
of Queen Mary Univeristy, and Mr. Masafuji Takahashi of Future University
Hakodate for fruitful discussions and valuable suggestions.

References
1. Chowning, J.M.: The Synthesis of Complex Audio Spectra by Means of Frequency
Modulation. J. the Audio Engineering Society 7(21), 526–534 (1973)
2. Gashler, M.S., Ashmore, S.C.: Training Deep Fourier Neural Networks To Fit TimeSeries Data (2014), arXiv preprint arXiv:1405.2262v1 (2014)
3. Godfrey, L.B.: Parameterizing and Aggregating Activation Functions in Deep Neural
Networks. Ph.D. thesis, University of Arkansas (May 2018)
4. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press (2016)

10

K. Hirata et al.

5. Mingo, L., Aslanyan, L., Castellanos, J., Daz, M., Riazanov, V.: Fourier Neural Networks: An Approach With Sinusoidal Activation Functions. International Journal“Information Theories & Applications” 11, 52–55 (2004)
6. Roads, C.: The Computer Music Tutorial. The MIT Press (1996)
7. Wikipedia: Yamaha DX7. https://en.wikipedia.org/wiki/Yamaha DX7, accessed:
2019/01/24