<figure><img src="image_1.png" /></figure>

HAL
open science

# RoughON: A Portable Music Sketch Production System by Real Time Input of Melody

Kazuhisa Akimoto, Junichi Hoshino

► To cite this version:

Kazuhisa Akimoto, Junichi Hoshino. RoughON: A Portable Music Sketch Production System by Real Time Input of Melody. 16th International Conference on Entertainment Computing (ICEC), Sep 2017, Tsukuba City, Japan. pp.32-37, 10.1007/978-3-319-66715-7_4 . hal-01771240

HAL Id: hal-01771240

https://inria.hal.science/hal-01771240

Submitted on 19 Apr 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# RoughON: A Portable Music Sketch Production System by Real Time Input of Melody

Kazuhisa Akimoto and Junichi Hoshino¹

Graduate School of Information and Engineering, University of Tsukuba,
Tsukuba-shi, 305-8577 Japan,
akimoto.kazuhisa@entcomp.esys.tsukuba.ac.jp

**Abstract.** In order to produce a music sketch (a combination of melody and accompaniment) in the beginning of musical composition, it is effective to use a composition means with high portability capable of inputting a melody in real time. In this paper, we propose a system that enables a beginner of musical composition or instrument easily and readily to create a music sketch by constructing a system capable of inputting a natural melody on a smartphone in real time in concert with accompaniment. In addition, we implemented the proposed system as an application to evaluate the proposed system through extensive use experiments by releasing it to the general public, showing the effectiveness of the proposed system in musical composition.

**Keywords:** making music, real-time, production system, portable

## 1 Introduction

In recent years, smartphone composing applications and music posting sites are well-supported[1-3]. As the environment in which composing activities can be easily done has become improved, novice musical composers and instrument users begin to occupy a part of users who use composition tools. In this paper, we propose a system to support novice musical composers.

For a novice musical composer or instrument user who is not easy to play, it is difficult "to produce a natural combination of melody and accompaniment" such as accompaniment production and the harmony of melody and accompaniment. Musical compositing support research thus far has realized a system to produce a combination of melody and accompaniment (hereinafter referred to as "music sketch") by automatic musical composition such as automatic accompaniment generation [4] and automatic lead sheet generation [5]. However, for users such as who want to publish a produced song, because they have a main purpose of self-expression, it is desirable to be able to directly and easily input the composing elements of a melody such as a rhythm, tones and a chord. Furthermore, portability is also necessary to form their inspiration. Portable systems [2, 3] are also available, but their support is not enough compared to the existing composition support researches.
Therefore, in this paper, we propose a portable system that allows users to input a melody in real time during automatic playing of accompaniment, thereby enabling to directly and easily enter the composition elements of a melody. In this system, we use a smartphone to generate a corrected melody so as to be able to input a natural melody even on the small screen so that it is consistent with the accompaniment and the melody up to the time of input. It also outputs musical score and MIDI, enabling to coordinate with instrument playing and DTM. In this paper, in order to verify its usefulness, we released it to the general public as iOS and Android applications[6] to conduct extensive use experiments.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Schematic chart of the proposed system.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Illustration of input screen.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3. Input screen of application.</figcaption></figure>

## 2 System Overview

We show the system outline in Fig. 1 to illustrate the flow of the proposed system following the numbers in the Figure.
The user produces a music sketch by 1) Selecting one out of 12 kinds of accompaniments prepared in advance, and 2) Inputting an melody extemporarily in real time during the automatic playing. The input screen at this time is shown in Figs. 2 and 3 and the select screen in shown in Figs. 4. As it is difficult to input it in real time in concert with the accompaniment at sight reading, display of a line showing the tempo(Tempo line and Time line), addition of percussion, and quantization of sound to reproduce were implemented as simplifying functions.

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Screen of selecting an accompaniment.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Screen of a music score.</figcaption></figure>

During the time of 2), since the system generates 3) pitches constituting a natural melody, input by the user becomes a natural melody. The pitch candidates to generate is multiplied by a key and constraints imposed by the last user's input. Instead of arranging the pitch candidates randomly on the screen, in order to follow the relative pitch, the pitch candidates are arranged so that the higher the sound becomes as it advances in the positive direction of the horizontal axis of the screen, and the lower the sound becomes as it advances in the negative direction. Also, as long as the user continues to enter at the same position, the pitches are arranged from the position the user input last so that the same pitch is played.

<figure><img src="image_8.png" /><figcaption>**Fig. 6.** The algorithm of arranging pitches.</figcaption></figure>
This pitches arrangement can be defined by a recurrence formula starting from i as follows. The algorithm is shown in Figs. 6. Let $S^{(k)} = \{s_1^{(k)}, s_2^{(k)}, \dots, s_{|N|}^{(k)}\}$ be the pitch sequence on the input screen at the time of input number $k$ and $P^{(k)} = \{p_1^{(k)}, p_2^{(k)}, \dots, p_{|M|}^{(k)}\}$ be the usable pitch candidate array. $P^{(k+1)}$ is generated depending on the input pitch $s_i^{(k)}$ and chord. At that time, the pitch sequence $S^{(k+1)}$ at the time of input number $k+1$ becomes as follows.

$$s_j^{k+1} = \begin{cases} s_i^k & (i = j) \\ \max_{m,s_{j+1}^{(k+1)} > p_m^{(k+1)}} p_m^{(k+1)} & (i > j) \\ \min_{m,s_{j-1}^{(k+1)} < p_m^{(k+1)}} p_m^{(k+1)} & (i < j). \end{cases} \quad (1)$$

After inputting the melody, the system edits the input melody, and 4) outputs the music sketch in the form of music score or MIDI. The score screen is shown in Figs. 5

## 3 Experiment

<table><caption>**Table 1.** Questions for evaluation.</caption><thead><tr><th>Question</th><th>Method</th><th>Number of subjects</th></tr></thead><tbody><tr><td colspan="3">Subjects' Experience</td></tr><tr><td>Q1. Do you have any experience in musical composition?</td><td>choices</td><td>84</td></tr><tr><td>Q2. Can you compose with a musical instrument?</td><td>choices</td><td>84</td></tr><tr><td colspan="3">System Evaluation</td></tr><tr><td>Q3. Is RoughON useful in supporting the early stage of composition?</td><td>points</td><td>46</td></tr><tr><td>Q4. Which would you like to use: RoughON or other applications?</td><td>choices</td><td>84</td></tr><tr><td>Q5. Could a music sketch become a clue to music production?</td><td>points</td><td>38</td></tr><tr><td>Q6. Do you want to use a music sketch in the future?</td><td>choices</td><td>37</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>**Fig. 7.** Graph of experimental results. (Q3, Q5 show the average and SE.)</figcaption></figure>
We implemented the system as a smartphone application "RoughON"[6] to release it to the general public. As of January 27, 2017, the number of unique users of the application reached 4300. In addition, the cumulative number of music sketch productions exceeds 10,000 (music: 9800, chord: 3500), and the cumulative output number in MIDI format reaches 3300. Furthermore, even now it is used about 100 times every day.

A questionnaire was used for the evaluation experiment. We show the questionnaire, the each method and the number of subjects in Table. 1. The user who installed the application and made music sketch more than 5 times was the subject, and the questionnaire was divided into 2 times according to the period. The number of subjects at the first time was 46 people, and the number of subjects at the second time was 38 people. However, because of questionnaire by public disclosure, the number of answers is different for each question item.

Question contents are classified into two types: question asking the attributes of subjects and question asking system evaluations obtained by subjects using "RoughON". The question method selectively uses the choices (Q1, 2, 4, 6) and the points (Q3, 5), and furthermore a free answer column was provided. In the scoring system, we made a 5-point evaluation with 1 step (5: very good evaluation, 1: very bad evaluation). The results of the evaluation experiment are shown in Fig. 7.

In Q1 and 2, we asked the subjects' experience in musical composition and instrument. In question Q1 asking the experience in musical composition, those who have no experience in musical composition and beginners accounted for about 80%. For Q4 asking the difficulty of music composition by musical instruments, about 40% answered that musical composition using musical instruments was possible, but nearly all of the respondents felt difficulty in musical composition using musical instruments.

Q3 and Q4 are questions concerning the evaluation as a system for supporting the music composition. Q3 asked the support of initial music composing, which is the purpose of the proposed system, having gotten 3.57 with a 5-point score evaluation. Compared to the score evaluation, in the free description column, many favorable opinions were obtained, saying that "the operation was intuitive and easy." Besides, there are also answers: "we can compose a music with a game feeling," "it is difficult to compose a music from the beginning, but we can easily take the first step if we use RoughON." When Q4 asked which one would you like to use: RoughON or other composition applications?, and 47 people out of 57 people who had use experience of other music composition applications answered that they want to use RoughON. As a reason for that, there were many answers: "simple and easy to use compared to other applications," similarly in Q3. In addition, there was also an answer that "Although other applications inhibit the pleasure of making a music because of a full automatic system, RoughON reasonably supports the composer." Besides, favorable answers for the accompaniment function, such as "it is easier to do if accompaniment is available at first" and "it can superimpose a sound easily" were obtained. In Q3 and 4, it
was shown that the proposed system is useful for supporting the early stage of
music composition.

Q5 and 6 are questions about the sufficiency of music sketch to develop it into
a music. In the score evaluation conducted in Q5 asking whether music sketch
would become a clue to complete a music, the average value was 4.0. Also, in
Q6 asking if you would like to actually use music sketch, there were 23 subjects
who answered that they would use it. They answered various uses, such as "they
make use of it to their own song" as a place to use and "they use for the BGM
of radio, game and movie." Q5 and 6 showed that the music sketch produced by
the proposed system plays a role of helping to complete a music

4 Conclusion

In this paper, we proposed a portable system for music composition and instru-
ment novices to produce a music sketch with real time input. In addition, the
proposed system was implemented, released to the general public, and evaluated
as a smartphone application: "RoughON." As a result, it was shown that the
proposed system played a role of supporting the first step of music composition,
revealing that its output, music sketch, is suitable as an intermediate product
for completing a music.

A wide range of experiments were carried out by public questionnaire, so
many subjects were able to be obtained. However, problems such as not being
able to confirm the detailed operation of the examinee, not being able to confirm
whether the subject used the music sketch produced by the proposed system
or not occurred. In the future, we will acquire operation history and plan to
investigate more detailedly for users who use the proposed system frequently.

References

1. C. Christopher. "The YouTube effect: How YouTube has provided new ways to consume, create, and share music," International Journal of Education & the Arts, vol.12, no.6, pp.1-30, 2011.

2. "Garage Band," Apple Computer,
http://www.apple.com/jp/ilife/garageband/, 2011.

3. "Music Memo," Apple Computer,
http://www.apple.com/jp/music-memos/, 2016.

4. I. Simon, D. Morris, and S. Basu, "MySong: automatic accompaniment generation for vocal melodies," Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.725-734, 2008.

5. A. Papadopoulos, P. Roy, and F. Pachet. "Assisted Lead Sheet Composition Using FlowComposer," International Conference on Principles and Practice of Constraint Programming, pp. 769-785, 2016.

6. "RoughON," K. Akimoto,
https://peraichi.com/landing_pages/view/x2sow, 2016.