JAIST Repository
https://dspace.jaist.ac.jp/

Title

Procedural Content Generation of Rhythm Games
Using Deep Learning Methods

Author(s)Liang, Yubin; Li, Wanxiang; Ikeda, Kokolo

CitationInternational Conference on Entertainment
Computing (ICEC) & Joint Conference on
Serious Games (JCSG), 11863: 134‑145

Issue Date

2019‑11‑04

Type

Conference Paper

Text version

author

URL

http://hdl.handle.net/10119/18205

RightsCopyright IFIP International Federation for
Information Processing 2019. Yubin Liang,
Wanxiang Li, Kokolo Ikeda, Lecture Notes in
Computer Science, 11863, 2019, 134‑145. The
final publication is available at Springer
via https://doi.org/10.1007/978‑3‑030‑34644‑
7̲11

Description

International Conference on Entertainment
Computing (ICEC) & Joint Conference on
Serious Games (JCSG), Arequipa, Perú, 12th
November 2019Japan Advanced Institute of Science and Technology

Procedural Content Generation of Rhythm
Games Using Deep Learning Methods
Yubin Liang1 , Wanxiang Li1 ? , and Kokolo Ikeda1
School of Information Science, Japan Advanced Institute of Science and Technology,
Nomi, Ishikawa, Japan
liang yubin@yahoo.com
wanxiang.li,kokolo@jaist.ac.jp

Abstract. The rhythm game is a type of video game which is popular
to many people. But the game contents(required action and its timing)of rhythm game are usually hand-crafted by human designers. In thisresearch, we proposed an automatic generation method to generate gamecontents from the music file of the famous rhythm game “OSU!” 4k mode.
Generally, the supervised learning method is used to generate such gamecontents. In this research some new methods are purposed, one is called“fuzzy label” method, which shows better performance on our training
data. Another is to use the new model C-BLSTM. On our test data,
we improved the F-Score of timestamp prediction from 0.8159 to 0.8430.
Also, it was confirmed through experiments that human players could
feel the generated Beatmap is more natural than previous research.
Keywords: Procedural Content Generation · Rhythm Game · C-BLSTM.

1Introduction

Rhythm game is a genre of music-themed (action video) game in which players
play by taking actions in accordance with rhythm and music [8]. Since music
and songs are familiar to ordinary people, it is easy to understand how to playsuch games. In addition, both of easy stage and hard stage can be created fromone music, therefore rhythm game becomes a popular game genre in the wholeworld.
In many cases, the contents (required action and its timing) of rhythm gameare hand-crafted by human designers from music material. Also, there are countless pieces of music, but only a part of them have already been used as gamecontents. Therefore, to generate contents automatically for rhythm game is required. In this research, two models are used to generate contents from music
materials. One inputs the audio data, outputs timestamps (timing of action),
another inputs the timestamp and outputs action type.
As a Machine Learning task, this approach has some difficulties such as:(1) Because same music may be processed by different authors or has different?

Corresponding author

2

Y. Liang et al.

level settings, one music may have many different Beatmaps. (2) Proportion of
positive/negative samples is ill. To deal with those problems, in this research
we adopt that: (1) Handle the difficulty settings as an input feature. (2) Using
fuzzy labels to increase positive samples.
Proposed training methods and new model C-BLSTM are evaluated throughF-Score, shows a better performance than previous research on Timestamp generation. And from human experiments we can find that Beatmaps which generated by purposed method are more natural than Beatmaps generated by previous
research.

2

BackgroundThe system structure of purposed method is shown as Fig.1. The main workflow
is from audio files generating the timestamp, and using those timestamps togenerate action type by second model. Combining the generated timestamp andaction type, we can obtain Beatmap for input music.

Fig. 1: System structure of purposed method.

2.1

Rhythm game: OSU!

To generate the contents of rhythm game, we use open-sourced game “OSU!”
which has over 10 million registered users [1] as our test bench.
“OSU!” consists of many types (or mode) of game, but in this research we
just use the mode called mania 4k. Players are required to push the button whenmarkers are dropping from the top of screen. When players push the correct
button, they would get a high score. On the other hand if the timing or required
action was incorrect, system would judge this action as missing. By pushing the
buttons, players would feel like they are playing this rhythm by themselves.
These required actions (4 types: click, long push start, long push end andnone) and their timing consist the game contents. In “OSU!”, Beatmap records

PCG of Rhythm Games Using Deep Learning Methods

3

these information. Generally, Beatmap are usually hand-crafted by human designers. So even if there are countless pieces of music, the number of contents
that can be played in the game is limited. An automatic generation method isneeded in this case.
2.2

Previous research

In the field of rhythm game and game contents generation research. There aresome previous researches which is essential to our approaches.
Capturing the characteristics of music is essential in rhythm game content
generating. Generally, selection of action timings is based on the sounds or
rhythm which can be clearly heard by human. Since the beginning or changing of music is most distinguishable to human, we usually set actions aroundthese timing in rhythm games. To extract these timing, Schlüter et.al proposed
an research using deep learning methods to detect the beginning of musical notes(musical onsets) [10]. Their method is used to extract features in our research.
Long Short-Term Memory (LSTM) is a kind of Recurrent Neural Network(RNN) which is effective to suppress gradient vanishing over time [5], and it has
been proved to be effective for time-based task such as voice detection [7] andtranslation [6], etc.
Music or rhythm is generally regarded as a type of time-based data, and in
the field of rhythm game content generation, LSTM was used to generate gamecontent of a rhythm game called “Dance Dance Revolution” in Donahue et al.’s
research [3]. However, to generate a content which correspond to specify difficult
level, or some special patterns (e.g. patterns that player good at or not) is still
not so gratifying. Hence, this research is aimed at generating game contents that
correspond to specific player’s level in rhythm game.
In many rhythm games, long press buttons are basically corresponds to along sound of music. So it is useful for analyzing the melody to generate thelong press properly. In Donahue et al.’s research, although the effectiveness of
melody information in action generation is examined, but its application has notbeen taken yet [3]. In this research, we use the method which propose by Salamon
et al. [9], for extracting the main melody of music. By using their method, we
can detect long sounds in music and attempt to employ them in long press action
generation.

3

Preparation of training data

3.1

Data and difficulty definition

In this research, all training data is collected from the homepage of “OSU!” [1]on June, 2017. Considering to use the most popular Beatmaps as our traning
data, only Beatmaps which has been played over 100 thousands times will bechosen.
In general, one music file corresponds to various Beatmaps depending on
the different authors and different difficulties. And players can play a suitable

4

Y. Liang et al.

difficulty according to their ability. The statistical information of used training
data is shown as Table 1.

Table 1: Statistical information of traning data
Number of
authors∗

Number ofmusic

Number of
Beatmap

Number of
all action∗

Action persecond∗

300

473

1655

1690000

7.85∗

Approximately.

Since there are various difficulties in the training data, and we aims to generate Beatmaps with different difficulties from one music. So a proper definitionof difficulty is highly required.
Generally, the difficulty is depending on: (1) Complexity of action combination. (2) Number of actions per second (density of actions). (3) Speed of action
marker. (4) The strictness of judging miss (max allowable time difference). (5)Number of miss that could be permitted.
Because (3), (4) and (5) is the more relative to setting of game, we don’t
consider these factors in this research. Also, individual players have different
action combinations that they are not good at, so it is hard to use factor (1)in automatic generation method. More actions in same time means Beatmap ismore difficult, so in this research we just use density of actions to define thedifficulties for each Beatmap.
We divide the training data to 10 levels by density, that is (a) number ofaction under 3 per second is level-0, (b) number of action from 3 to 17 per
second is used for level-1 to level-8 equally (e.g. 3 to 4.75 is level-1 and 4.75 to
6.5 is level-2), (c) number of action more than 17 per second is level-9.

3.2

Feature extraction

In order to use the audio data in neural networks, some feature extraction process
is needed. Firstly we compute a multiple-timescale short-time Fourier transform(STFT) of every audio file by three window sizes 23ms, 46ms and 93ms andstride of 10ms as 1 frame (that means there are 100 frames per second). Shorter
window sizes preserve low-level features such as pitch and timbre while large
window sizes provide more context for high-level features such as melody andrhythm[4].
Then we compute the Mel-scale from 27.5Hz to 16kHz with 80 bands of
STFT magnitude spectra to better represent human perception of loudness [12].
Finally we prepend and append n frames of past and future context for eachframe, so the final audio representation is a 80 * 3 * (2n + 1) tensor.

PCG of Rhythm Games Using Deep Learning Methods

4

Timestamp generation

4.1

Network structure

5

Donahue et al. uses C-LSTM to generate timestamp [3], which has the advantage
to learn not only the current frame but also the past and future frames. Structureof C-LSTM is shown as Fig.2. We use 15 * 80 * 3 tensor (n = 7) as input. First
layer is a convolution layer which has the size of 7 * 3 * 3 and 1 * 3 max-pooling,
output 9 * 26 * 10 matrix to next layer. Second layer is a convolution layer
which has the size of 3 * 3 * 10 and 1 * 3 max-pooling, output 7 * 8 * 20 matrix.
Adding the 10 units one-hot density (which present the level of difficulty), we
input the matrix to 2 LSTM layer which have 200 units. After LSTM there
are 2 full-connected layer with 258 units and 128 units separately. Finally weoutput the result by a single Sigmoid unit, which presents the possibility of beenchoosing as a timestamp. LSTM layers use tanh function as activation function,
while other layers use ReLU.

Fig. 2: Structure of C-LSTM.

C-LSTM can only use the information “until now”. In this research, “OSU!”
has a lot of “long push” action, since most “long push start” actions are at
the onset of long melody, we need to consider not only the information in past,
but also information in the future. One way is to use more frames (i.e. maken bigger), but the high calculation cost and performance reduction is unbearable sometimes. Another way to get more future information without increasing
frames is using the C-BLSTM [11].

Fig. 3: Structure of C-BLSTM.

6

Y. Liang et al.

The structure of C-BLSTM is shown as Fig.3. C-BLSTM can predict timestamp using more future information, but has the similar calculation cost as
C-LSTM.
4.2

Fuzzy label

There are about 7.85 actions per second in our training data. Since we have 100
frames in 1 second, there are just few positive data (less than 8%). One way tosolve this problem is to calculate feature with wider stride. For example using
50ms stride will get 20 frames per second, which will make the rate of positive
data to 40%. But longer stride will lost some rhythm or melody information.
In this research a method called fuzzy label that can solve the problem is
proposed. Shown as Fig.4, the original data use 0 (no action in this frame) and1 (there is action in this frame) label which have a drastic changing. And fuzzylabel modify the data using Gaussian distribution, transform the data from “haveor not have action” to “the possibility that have action”, makes the changing
smoother.

Fig. 4: Example of fuzzy label.

4.3

Threshold selection

Output of network is the “possibility whether one frame should have actions”.
In practice, we need to transform the possibility into a two-value number (0 or1), to decide whether there should have actions in this frame. Using some rules
such as “when possibility bigger than a threshold t, there should have action”
can be helpful for judging.
When threshold was too large, output action would decrease, and output
action would increase when it was too small. Both situations are not good fortimestamp selection, so a proper threshold is needed. Some data which called“threshold deciding data” is randomly selected to decide the threshold t. The
threshold deciding data is independent to training data and will not take partin the training process.

PCG of Rhythm Games Using Deep Learning Methods

4.4

7

Experiment

Difficulty distribution in original data is unbalanced, there are just few low
difficulty data and high difficulty data. Such a ill data distribution is harmfuland causing unstable learning [2]. To solve this problem, we copy the training
data in low and high difficulty to make original data distribution balanced.
Validation experiment In this experiment, all model uses 80*3*15 (frame
length n = 7) tensor as input and same learning hyper-parameters. All networks
are trained for 25 epochs (1000 batches per epoch), the mean F-score of eachepoch is shown as Fig.5.

Fig. 5: Result of validation experiment.

From the result, we can find that comparing to the C-LSTM, C-BLSTM
shows a better performance in the beginning, but suddenly facing the problem
of degradation in performance. We also find that training error of C-BLSTM
deceasing follow the process, which means C-BLSTM faced a serious overfitting
problem in the mid of training.
When comparing the result of C-LSTM to C-LSTM with fuzzy label (label
length is 3), we can find that fuzzy label improved the performance dramatically.
Moreover, from the result of C-BLSTM with fuzzy label, we can find that fuzzylabel not only improved the performance, but also solved the overfitting problemof C-BLSTM.
Experiment of different fuzzy label We also have experiments on different
fuzzy label lengths to compare their performance. Tn this experiment, all models
use C-BLSTM and 80*3*15 (frame length n = 7) tensor as input, also, they use
same learning hyper-parameters.
Mean F-score of each epoch is shown as Fig.6. We can find that our method
improved the naive C-BLSTM comparing to the result with no fuzzy label. Also,

8

Y. Liang et al.

we find best fuzzy label length is 3 on our data. One possible reason is whenlength was 3, the rate of positive data would be closed to 50%, which is a wellbalanced data distribution.

Fig. 6: Result of fuzzy label experiment.

Experiment of different frame length In this experiment, all model use
C-BLSTM, same fuzzy label length of 3 and same learning hyper-parameters.
The only difference is the frame length, which will influence the input tensor
size. For example when frame length n was 7, the input size would be 80*3*15,
and when frame length n was 9, the input size would be 80*3*19.
The result of n =4, 5, 6 and 7 is shown as Fig.7a. We can find that whenn is 4, we would get the best performance, and when n is 7 we would get thesimilar performance to n =4. From Fig.7b we can find that n =7 showed thebest performance. Overall, from n =4 to 10, the performance firstly goes down
and then goes up. The result shows that we cannot get better performance bysimply adding the by frame length.

(a) Result of n =4, 5, 6 and 7(b) Result of n =7, 8, 9 and 10

Fig. 7: Result of different frames

PCG of Rhythm Games Using Deep Learning Methods

5

Action type generation

5.1

Input feature and output

9

In our Action type generation model, we use (1) Action before one frame, (2)Difficulty and (3) Time interval of past/future frame as input. And 4 types ofaction will be generated and used as input, that is (a) No action, (b) Click, (c)Long push start and (d) Long push end. 4 bit one-hot vector is used to present
action on single channel. And since we have 4 channels, the action will be four
4 bit one-hot vectors.
The time interval is the distance between two actions. We use a 8 bit onehot vector to present it. This 8 bit one-hot vector means the interval is ∼50ms,∼100ms, ∼200ms, ∼400ms, ∼800ms, ∼1600ms, ∼3200ms and 3200ms∼. Sinceboth the past time interval and future interval is used, the time interval will
be presented as two 8 bit one-hot vector. And a 10 units one-hot density label(which present the level of difficulty) is also needed in this model. The input
feature is combined with the density, time interval and past action, so input
feature has 7 vectors and the data size is 42 bit.
The output is probabilities of all action combinations. Since we have 4 channels and each channel has 4 types of action, the number of all valid actions is 256(44 ). Finally we will choose one action combination that have biggest probability.
The example of input feature and output probabilities is shown as Fig.8.

Fig. 8: Example of input features and output.

5.2

Network structure

In Action type generation part, we uses the network frame called LSTM64 [3].
Shown as Fig.9, after input layer the first layer uses 128 LSTM units, and secend

10

Y. Liang et al.

layer uses 128 LSTM units. Finally this network output 256 probabilities of all
valid actions.

Fig. 9: Structure of Beatmap generation model.

5.3

Experiment of predict actions

Same training data as timestamp generation model are used for training thismodel. Also, since the difficulty distribution in original data is unbalanced, training data in low and high difficulty is copied to make original data distribution
balanced.
The finial accuracy of predicting action is 0.4366, which is not a high accuracy. But we find that even in one music, there are different Beatmaps. And
according to the different designers, the Beatmap is various. For example inFig.10, we can find that about half actions are not same between two humandesigner, even they have used the same music.

Fig. 10: Compare of two Hand-crafted Beatmaps.

Fig.11 shows a result of Beatmap generated by network and human designer
for same music. We can find that even some actions are different, these two
Beatmaps are similar, which player may not notice the difference. According tothese evidence, we think the result that predicting accuracy is 0.4366 is not bad.
Prediction accuracy is not enough for estimating how human players can enjoy
the generated Beatmaps, so it will be an interesting future work to propose sucha measurement.

PCG of Rhythm Games Using Deep Learning Methods

11

Fig. 11: Compare of Hand-crafted and purposed method.

6

Evaluation of naturalness

We conduct a experiment though 10 human subjects. 15 Beatmaps (5 of them arehand-crafted by human designer, 5 of them are generated by previous researchand 5 of them are generated by proposed method) are used in this experiment.
“Purposed method” data and “Existing method” data use same action typegeneration model, but “Purposed method” data use C-BLSTM model with fuzzylabel (length=3), frame size n is 7 to generate timestamp. timestamp of “Existing
method” data are generated by C-LSTM without fuzzy label, and frame size n
is 7.
All Beapmaps are randomly watched by every subjects, and subjects are
asked to judge the naturalness for each Beatmap. The result is hand-crafted
Beatmaps get 4.52 points, existing method get 3.30 points and purposed method
get 3.72 points. From the result we can find that even Beatmaps which generated by purposed method are not as natural as hand-crafted Beatmaps, thenaturalness is better than Beatmaps generated by existing method.

7

Conclusion

Overall, a new method “fuzzy label” is introduced in game contents generationof rhythm game, and new model C-BLSTM is used in our timestamp generation
method. We have proofed that purposed method improved the performance ofTimestamp generation, and the F-Score is increased from 0.8159 to 0.8460.
Moreover, a generation model from timestamp to Beatmap is introduced.
And by experiment on human subjects, we have proofed that Beatmaps generated by purposed method is more natural than previous research.
Acknowledgments. This research is financially supported by Japan Society
for the Promotion of Science (JSPS) under contract number 17K00506.

References
1. Osu! homepage, https://osu.ppy.sh/home

12

Y. Liang et al.

2. Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance
problem in convolutional neural networks. Neural Networks 106, 249–259 (2018)
3. Donahue, C., Lipton, Z.C., McAuley, J.: Dance dance convolution. In: Proceedingsof the 34th International Conference on Machine Learning-Volume 70. pp. 1039–
1048. JMLR. org (2017)
4. Hamel, P., Bengio, Y., Eck, D.: Building musically-relevant audio features throughmultiple timescale representations (2012)
5. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation
9(8), 1735–1780 (1997)
6. Kalchbrenner, N., Danihelka, I., Graves, A.: Grid long short-term memory. arXiv
preprint arXiv:1507.01526 (2015)
7. Parascandolo, G., Huttunen, H., Virtanen, T.: Recurrent neural networks for polyphonic sound event detection in real life recordings. In: 2016 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6440–6444.
IEEE (2016)
8. Pasinski, A.: Possible benefits of playing music video games (2014)
9. Salamon, J., Gómez, E.: Melody extraction from polyphonic music signals using
pitch contour characteristics. IEEE Transactions on Audio, Speech, and Language
Processing 20(6), 1759–1770 (2012)
10. Schlüter, J., Böck, S.: Improved musical onset detection with convolutional neuralnetworks. In: 2014 ieee international conference on acoustics, speech and signal
processing (icassp). pp. 6979–6983. IEEE (2014)
11. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11), 2673–2681 (1997)
12. Stevens, S., Volkmann, J., Newman, E.: The mel scale equates the magnitude ofperceived differences in pitch at different frequencies. Journal of the AcousticalSociety of America 8(3), 185–190 (1937)