# Auditory-Centered Vocal Feedback System Using Solmization for Training Absolute Pitch Without GUI

Nozomu Yoshida, Kosaku Namikawa, Yusuke Koroyasu, Yoshiki Nagatani,
Yoichi Ochiai

► To cite this version:

Nozomu Yoshida, Kosaku Namikawa, Yusuke Koroyasu, Yoshiki Nagatani, Yoichi Ochiai. Auditory-Centered Vocal Feedback System Using Solmization for Training Absolute Pitch Without GUI. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.3-19, 10.1007/978-3-030-85610-6_1 . hal-04215532

HAL Id: hal-04215532

https://inria.hal.science/hal-04215532v1

Submitted on 22 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Nozomu Yoshida<fnref n="1" />, Kosaku Namikawa<fnref n="1" />, Yusuke Koroyasu<fnref n="1" />, Yoshiki Nagatani<fnref n="2" />, and Yoichi Ochiai<fnref n="1" />,<fnref n="2" />

**Abstract.** This study proposes an auditory-centered training system using solmization to artificially acquire and train absolute pitch by providing vocal feedback of musical notes through a musician's voice. Most current training systems and applications for absolute pitch acquisition have focused on providing visual feedback. However, many people having perfect pitch describe that they hear music as words rather than envisioning visual notes. Therefore, we propose a training system that does not require a graphical user interface. In an experiment with 10 participants, our system's training with vocal feedback improved six non-potential absolute pitch users' absolute pitch by approximately 25%, although extant system with visual feedback didn't make an improvement.

**Keywords:** Auditory feedback · learning method · pitch training.

## 1 Introduction

This study proposes an auditory-centered training system using solmization to artificially acquire and train absolute pitch by providing vocal feedback of musical notes by musician's voice. Absolute pitch is the ability to name or produce a note of a given musical tone without an external reference tone [24]. People with this ability are rare, and its prevalence in the general population in North America and Europe is estimated to be less than 10,000 people [2,20,24]. Absolute pitch is not an essential skill; however, having absolute pitch can sometimes be beneficial, especially for professional musicians. For example, it is difficult to detect minor pitch errors by determining pitch relations in atonal music [5]. Additionally, a composer with absolute pitch can sometimes work efficiently with one note at a time, compared to a composer without absolute pitch who has to work in two-note increments [15].

Most current absolute pitch training systems provide musical note names and pitches via the graphical display as visual feedback. Here, visual feedback

<footnote n="1">¹ University of Tsukuba, Tsukuba 305-8577, Japan
{nozo,namikawa,koroyu,wizard}@digitalnature.slis.tsukuba.ac.jp
https://digitalnature.slis.tsukuba.ac.jp</footnote>

<footnote n="2">² Pixie Dust Technologies, inc., Tokyo 101-0061, Japan
yoshiki.nagatani@pixiedusttech.com
https://pixiedusttech.com</footnote>
refers to the feedback of information via the user's vision. However, most individuals having absolute pitch state that they hear music as words rather than envisioning it as notes [12,17]. Therefore, the proposed method adopts a musical note-singing method called solmization; a method of singing while calling out each sound by its note name (e.g., *do*, *re*, *mi*). Solmization provides singers with vocal feedback of notes in the form of words, facilitating pitch training. Here, vocal feedback refers to the feedback of information via the user's hearing by speech. Furthermore, our proposed system does not require a Graphical User Interface (GUI), in contrast to traditional training systems, such as tuners and the visualized guide melody of Karaoke.

Our contributions are as follows. First, we constructed a system that allows users to recognize note names aurally via solmization. Second, we compared our vocal feedback system with existing visual feedback systems and identified which type is more effective.

# 2 Related Work

## 2.1 Pitch Training

Various training methods have been developed to improve the musical sense of pitch [23,9,10,16,21]. Solfege, which applies solmization at the scale of C, uses the *sol-fa* syllables to name or represent the tones of a melody or scale. It refers to training that involves writing down the music heard by ear onto a score and singing the melody while looking at it. Additionally, musical-instrument and singing instruction can be regarded as sound training that requires a teacher. It is known that paying attention to the pitch of both voice and instrument improves the vocal and aural senses.

Karaoke provides a visualized training tool for similar purposes. For example, the Live Dam Stadium DX-G<fnref n="3" /> is an analytical scoring system for Karaoke, and it feeds back the singer's pitch in real-time as a visualized melody. There are several such pitch-training applications. In the work of [13], a self-training mobile application was built that improved the sense of pitch as the user sang into an interactive user interface. Additionally, [6] provided a singing and pitch-training application for children with cochlear implants. With the Ear Training Myu-Tre<fnref n="4" /> and OtoAte<fnref n="5" /> applications, users train their absolute pitch by listening to a specific pitch and guessing the note. The C-major scale (i.e., CDEFGAB) is displayed on the screen to assist the user visually. The Pitch<fnref n="6" /> application provides information on note names and pitches of 3-7 scales, and by guessing the presented notes, users train their relative pitch. This tool requires a GUI. The Singing Assessment and Development [11,26] tool was a pioneer project that incorporated real-time visual feedback into educational software for singing.

<footnote n="3">³ https://www.clubdam.com/app/dam/seimitsusaiten/, accessed on Jan 18, 2021</footnote>

<footnote n="4">⁴ https://apps.apple.com/jp/app/id1015269208, accessed on Jan 18, 2021</footnote>

<footnote n="5">⁵ https://apps.apple.com/jp/app/id763325064, accessed on Jan 18, 2021</footnote>

<footnote n="6">⁶ https://apps.apple.com/jp/app/id942373213, accessed on Jan 18, 2021</footnote>
It provided a pedagogical tool for assessing and developing the singing voices of UK primary school children using real-time visual feedback technology. The ALBERT tool [22] provides acoustic output while monitoring laryngeal action. Thus, the system provides a greater diversity of feedback displays and parameters, such as F0, larynx closed quotient, spectral ratio, sound-pressure level (amplitude), shimmer, and jitter. The SING and SEE [3] tools distinguished three parameters (i.e., pitch, vowel identification, and timbre). Although SING and SEE focused on maximizing visual feedback, the VOXed project [25] involved psychologists, vocalists, singing instructors, and students to investigate the usefulness of direct feedback using commercially available visual feedback software. MiruSinger [18] proposed a method of automatically assessing a user's singing ability by providing visual feedback of musical information without using musical notation.

## 2.2 HCI Study Using Auditory Feedback

Kuber et al. demonstrated the potential for non-visual speech-based audio and haptic feedback systems in the design of accessible memory games to aid visually impaired users who would otherwise remain excluded from most mainstream gaming applications [14]. They recommended that speech-based feedback be presented in conjunction with haptic cues when developing a non-visual game. Stina et al. proposed a running-technique training system that combined real-time visual and audio feedback [19]. In their proposed system, the visual feedback displayed acceleration, and the audio feedback conveyed the runner's rhythm using beeps. This study suggested that, if the users were in a situation where the screen could not be seen, the audio feedback would provide suitable feedback. Therefore, when inducing user movement, voice feedback may be suitable if the amount of data transmitted is succinct, such as that of rhythm or note names. Guardati et al. proposed a method to assist with writing rehabilitation [8]. The system was designed for human-in-the-loop operations, and it could analyze handwriting in real-time while providing vocal feedback to guide the patient during exercise. The advantage of the proposed method is that it could be easily used at home without a trainer. Christiansen et al. investigated in-vehicle input and output techniques to compare their effects on driving behavior and attention, finding that using audio resulted in significantly fewer eye glances but longer task completion times with inferior primary driving task performance compared with visual cues [4]. Gaver et al. proposed Auditory Icon, an alternative approach to the use of sound in computer interfaces that emphasized the role of sound in conveying information about the world to the listener [7]. By combining auditory and visual feedback, all information could be accessed more efficiently. For example, when an error occurred on a computer, a beeping sound in addition to a warning screen display would instantly inform the user of the situation.
## 2.3 Vocal Feedback as Cybernetics

Cybernetics is a research field that attempts to study the structure of control and communication/information transmission in a wide range of mechanical, biological, and social organizations from basically the same methodological perspective [27]. The core concept of the discipline is circular causality or feedback, i.e., where the outcomes of actions are taken as inputs for further action [1]. The vocal feedback system also includes the concept of Cybernetics in terms of performing a feedback loop. The user communicates with the vocal feedback system by actions (such as one's voice or UI operation), receives feedback by listening to the system output, and performs self-regulation after referring to it.

## 2.4 Position of This Study

The research question for this study is “can absolute pitch be trained by a vocal feedback system?” Extant pitch-training methods provide visual feedback containing the note information. However, individuals with absolute pitch describe their experience as audible instead of visual. Furthermore, previous studies indicate that auditory feedback is useful for providing simple information. Therefore, we construct a system that allows users to aurally recognize note names. Then, we compare the auditory feedback system with the extant method to identify which one is more effective.

# 3 Implementation

## 3.1 Design Principles

Figure 1 shows the design of our proposed system. It consists of five modules.

**Random Test Generator:** A module that determines the correct pitch. The fundamental tone is 60 (C4, 261.6Hz), and it randomly selects the correct pitch from the C-major scale range at one octave from the fundamental tone.

**User Input:** A module in which the user inputs a note number via a slider. The slider's pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.

**Oscillator:** A module that sounds input and correct pitches. The triangular-wave oscillator reproduces the pitch corresponding to the input. The user interface has a fundamental tone button, an input tone button, and correct tone button, but they are not played simultaneously to prevent delivering hints for guessing.

**Vocal Feedback:** A module that outputs vocal feedback corresponding to the input pitch. From the input sounds, the microtonal music is rounded off to the nearest semitone unit. The sound source material used for vocal feedback is sampled from a vocalist who had mastered vocal music.

**Display:** A module that returns the user's input pitch in text form. The displayed pitch entered with the slide bar is rounded off to the nearest semitone unit.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Design of absolute pitch trainer consisting of five modules: random test generator, user input, oscillator, vocal feedback, and display. In our system, the note name is given auditory feedback in real-time, unlike the existing software learning method that visually feeds back the note name.</figcaption></figure>

## 3.2 User Interface

Figure 2 provides a screenshot of the interface, and Table 1 describes each button of the display interface. The interface provides a random pitch test for training, and it is used as the proposed system for absolute pitch training in an experiment. In addition to the vocal feedback feature, our application has a visual feedback feature for comparative experiments, as shown in Nos. 12 and 13 in Table 1. The visual feedback feature displays staff notation with note names. In the proposed vocal feedback system, GUI-free pitch training is possible. However, because the experiment requires the user to provide the pitch input using a slide bar, a GUI was adopted. The only difference between vocal and visual feedback is the type of note-name fed back, and the other specifications are the same.

<footnote n="7">⁷ These features are for relative pitch training. In this study experiment, we focused only on absolute pitch, so we did not use them.</footnote>
<table><caption>**Table 1.** User interface description. Each number in table 1 corresponds to the number in Fig. 2.</caption><thead><tr><th>No</th><th>Name</th><th>UI Type</th><th>Description</th></tr></thead><tbody><tr><td>1</td><td>Random Test</td><td>Display</td><td>The designated note name as random test is displayed.</td></tr><tr><td>2</td><td>Slide Bar</td><td>Drag</td><td>Manipulate the pitch by dragging the slide bar.</td></tr><tr><td>3</td><td>Keynote<sup>7</sup></td><td>Button</td><td>Listen to the fundamental triangle wave by pressing a button.</td></tr><tr><td>4</td><td>Start</td><td>Button</td><td>The bar's pitch's triangular wave at the current position will be fed back by pressing the button. Also, it resumes oscillator feedback from Pause.</td></tr><tr><td>5</td><td>Enter</td><td>Button</td><td>Confirm the pitch of the bar at the current position as an answer.</td></tr><tr><td>6</td><td>Pause</td><td>Button</td><td>Stop playing the oscillator by pressing a button.</td></tr><tr><td>7</td><td>Input Note</td><td>Display</td><td>The input note name is displayed</td></tr><tr><td>8</td><td>Correct Pitch</td><td>Button</td><td>The note name corresponding to the entered pitch is displayed.</td></tr><tr><td>9</td><td>Next Question</td><td>Button</td><td>Move on to the next question by pressing a button.</td></tr><tr><td>10</td><td>Mode Selection<sup>7</sup></td><td>Switch</td><td>Switch between absolute pitch mode and relative pitch mode.</td></tr><tr><td>11</td><td>Vocal Feedback</td><td>Toggle</td><td>Turn on/off the vocal feedback of the note name.</td></tr><tr><td>12</td><td>Visual Feedback</td><td>Toggle</td><td>Turn on/off the visual feedback of the note name.</td></tr><tr><td>13</td><td>Visual Feedback</td><td>Display</td><td>The staff notation and the note name are fed back when the visual feedback is on.</td></tr></tbody></table>

# 4 Experiment

We anticipated that the vocal feedback of the note name presented by the proposed system would be more effective in improving the absolute pitch than the visual feedback of the existing system. Thus, we conducted an experiment to determine the same. The experiment consisted of a preliminary test and a main test. Additionally, a pre-questionnaire was conducted to determine attributes, such as the participants' musical experience prior to the experiment. A post-survey was also conducted to evaluate the experiment process and the usability of the system. The experiment was conducted with the approval of our institution's ethics review committee.

## 4.1 Participants

We recruited 10 men with no reported physical or mental disability ranging in age from 20 to 24 years old as participants (M: 22; SD: 1.3). We accepted participants regardless of their musical experience. All participants have never received singing instruction or music theory instruction, though four people (P2, P8, P9, P10) had experience playing the piano or electric piano, and three people (P2, P8, P10) had ever taken the piano lesson.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** User interfaces for training and learning absolute pitch. Each number in Fig. 2 corresponds to the number in table 1. There are two types of training modes: the auditory feedback mode (proposed system) and the visual feedback mode (existing system). There is also a test mode for absolute pitch test, which excludes the training mode's feedback function.</figcaption></figure>

## 4.2 Preliminary Test

Using an existing pitch-training application (Ear Training Myu-Tre), the participants attempted the absolute pitch quiz 10 times and recorded their scores. The contents of the test was as follows. First, for each trial, one of the seven C-major scale notes was given as a piano note sound: *Do/C*, *Re/D*, *Mi/E*, *Fa/F*, *Sol/G*, *La/A*, and *Si/B*. Next, the participants attempted to identify the note name at the same pitch as the given piano note sound. The score was recorded as “correct” or “incorrect,” but the pitch distance between the correct pitch and the input pitch was not recorded.

## 4.3 Main Test

We conducted the main test to verify the hypothesis: vocal feedback of the note name by the proposed system is more effective in improving the absolute pitch than the visual feedback of the note name by the existing system-based counterpart. For the convenience of explanation, we defined the training provided by the proposed system as A. The training provided by the existing system is B. The absolute pitch test, C, is shown in Table 2. In this experiment, the participants were trained in absolute pitch using both systems (A and B). Additionally, we quantified the improvement of the absolute pitch tests conducted before and after the training. They took test C before and after training A and B. To reduce
possible bias caused by the training order, five participants trained on the proposed system first, and the other five trained on the existing system first. The order of the experiments is shown in Table 3.

<table><caption>**Table 2.** Definition of the Experimental Process</caption><thead><tr><th>Process</th><th>Definition</th><th>Time</th></tr></thead><tbody><tr><td>A</td><td>Absolute pitch training with the vocal feedback of the proposed system</td><td>5 min</td></tr><tr><td>B</td><td>Absolute pitch training with the visual feedback of the existing system</td><td>5 min</td></tr><tr><td>C</td><td>An Absolute pitch test with 10 questions</td><td>until finish</td></tr></tbody></table>

**Training Task: A & B** The procedure of the training tasks A and B is shown in table 4. The only difference between A and B is the note name's type of feedback; The note name is fed back auditory in the A and visually in the B.

**Absolute Pitch Test: C** the procedure of the absolute pitch test is shown in Table 5. Two values were recorded in the system: an integer value as the midi-note number of the correct note name and a real value as the midi-note number of the input note name. Therefore, it could measure the distance between the correct and input pitches. For example, when the correct pitch was C4 (midi-note #60) and the input pitch was C#4 (midi-note #61). A distance of 1.0 was recorded as an error. Both the correct and input pitches could be octaves, and the distance was calculated by the difference between the input pitch and the nearest correct pitch.

# 5 Result

## 5.1 Preliminary Test

For grouping purposes, in the preliminary test, the four people who recorded perfect scores were defined as “potential absolute pitch,” and the other six were defined as “non-potential absolute pitch.” The scores from the preliminary test for potential and non-potential absolute pitch are shown in the second columns of Tables 6 and 7, respectively. All potential absolute pitches had experience playing the piano or electric piano, and three (P2, P8, P10) had ever taken the piano lesson for several years between 4 and 13 years old. All non-potential

<table><caption>**Table 3.** Experiment Pattern</caption><thead><tr><th>Pattern</th><th># of Participants</th><th>Order</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>C → A → C → Interval → C → B → C</td></tr><tr><td>2</td><td>5</td><td>C → B → C → Interval → C → A → C</td></tr></tbody></table>
<table><caption>**Table 4.** Procedure of Training Task: A, B</caption><thead><tr><td>Step</td><td>Description</td></tr></thead><tbody><tr><td>1</td><td>A randomly specified note name is displayed; participants imagine the pitch and note name.</td></tr><tr><td>2</td><td>Operate the slide bar to approach the pitch you imagined while listening to the oscillator sound.</td></tr><tr><td>3</td><td>Stop the bar at the position where the specified note name is auditory (A) or visually (B) fed back, and press the enter button. At that time, be sure that the pitch name and pitch are associated and memorized.</td></tr><tr><td>4</td><td>Listen to the correct pitch. At that time, be sure that the floor name and pitch are associated and memorized.</td></tr><tr><td>5</td><td>Click the "next question button" to move to the next question.</td></tr><tr><td>6</td><td>Repeat a series of steps 1-5 for 5 min.</td></tr></tbody></table>

<table><caption>**Table 5.** Procedure of Absolute Pitch Test: C</caption><thead><tr><td>Step</td><td>Description</td></tr></thead><tbody><tr><td>1</td><td>A randomly specified note name is displayed; participants imagine the pitch of that note name.</td></tr><tr><td>2</td><td>Operate the slide bar to approach the pitch you imagined while listening to the oscillator sound.</td></tr><tr><td>3</td><td>Stop the bar at the position that is thought to be correct and press the enter button.</td></tr><tr><td>4</td><td>The note name corresponding to the input pitch is displayed, and the real value of the midi note number is recorded in the system.</td></tr><tr><td>5</td><td>Click "the next question button" to move to the next question.</td></tr><tr><td>6</td><td>Repeat a series of steps 1-5 10 times.</td></tr></tbody></table>

absolute pitches had neither experience playing the instruments nor lessons of any musical instruments (except for the class in the school).

## 5.2 Main Test

The results were divided into two categories: potential and non-potential absolute pitch. The average error is the average difference between the midi number of the correct pitch and that of the input pitch from the 10 questions of test C (c.f. 4.3). The improvement value refers to the difference between the average error before and after training. The improvement rate refers to the value obtained by dividing the average error before training by that after training.

**Potential Absolute Pitch** Table 6 shows the results of the experiment of potential absolute pitch. Generally, neither the proposed nor the existing system showed significant improvement. The average improvement value was slightly negative (−0.09), and the average improvement rate was positive (+11%) in the proposed system. The average improvement value was slightly negative (−0.07), and the average improvement rate was almost 0 in the existing system.
<table><caption>**Table 6.** Result of potential absolute pitch (PS: Pretest Score, BT: Before Training, AT: After Training, IV: Improvement Value, IR: Improvement Rate).</caption><thead><tr><th rowspan="2">ID</th><th rowspan="2">PS</th><th colspan="4">Proposed System</th><th colspan="4">Existing System</th></tr><tr><th colspan="2">Average Error</th><th colspan="2">Improvement</th><th colspan="2">Average Error</th><th colspan="2">Improvement</th></tr><tr><th></th><th></th><th>BT</th><th>AT</th><th>IV</th><th>IR</th><th>BT</th><th>AT</th><th>IV</th><th>IR</th></tr></thead><tbody><tr><td>2</td><td>10</td><td>0.29</td><td>0.16</td><td>0.13</td><td>+82%</td><td>0.23</td><td>0.21</td><td>0.02</td><td>+12%</td></tr><tr><td>8</td><td>10</td><td>0.34</td><td>0.49</td><td>-0.16</td><td>-32%</td><td>0.50</td><td>0.52</td><td>-0.02</td><td>-4%</td></tr><tr><td>9</td><td>10</td><td>0.41</td><td>0.39</td><td>0.02</td><td>+4%</td><td>0.33</td><td>0.32</td><td>0.01</td><td>+2%</td></tr><tr><td>10</td><td>10</td><td>2.97</td><td>3.30</td><td>-0.33</td><td>-10%</td><td>2.87</td><td>3.16</td><td>-0.30</td><td>-9%</td></tr><tr><td>Ave</td><td>10</td><td>1.00</td><td>1.09</td><td>-0.09</td><td>+11%</td><td>0.98</td><td>1.05</td><td>-0.07</td><td>+0%</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Average error transition of potential absolute pitch.</figcaption></figure>

**Non-Potential Absolute Pitch** Table 7 shows the result of non-potential absolute pitch. Generally, the proposed system showed some improvement, whereas the existing system did not. In the proposed system, both the average improvement value and the average improvement rate were positive (0.65 and +25%, respectively). In particular, P4 had an improvement value of 1.40, implying that the absolute pitch accuracy improved by more than a semitone. In contrast, in the existing system, both the average improvement value and the average improvement rate were negative (-0.49 and -12%, respectively). The existing system did not show any improvement, except for one person on P7.

**Statistical Hypothesis Testing** We conducted a Wilcoxon signed-rank test at a significance level of 5% to compare the average error before and after training in the proposed/existing system. The null hypothesis stated that there would be no difference in the average error before and after training with the proposed/existing system, and the alternative hypothesis stated that there would be a difference in the average error before and after training with the proposed/existing system.
<table><caption>**Table 7.** Result of non-potential absolute pitch (PS: Pretest Score, BT: Before Training, AT: After Training, IV: Improvement Value, IR: Improvement Rate).</caption><thead><tr><th rowspan="2">ID</th><th rowspan="2">PS</th><th colspan="4">Proposed System</th><th colspan="4">Existing System</th></tr><tr><th>BT</th><th>AT</th><th>IV</th><th>IR</th><th>BT</th><th>AT</th><th>IV</th><th>IR</th></tr></thead><tbody><tr><td>1</td><td>6</td><td>1.91</td><td>1.88</td><td>0.02</td><td>+1%</td><td>1.66</td><td>2.20</td><td>-0.54</td><td>-25%</td></tr><tr><td>3</td><td>4</td><td>3.47</td><td>3.04</td><td>0.43</td><td>+14%</td><td>1.91</td><td>3.53</td><td>-1.63</td><td>-46%</td></tr><tr><td>4</td><td>4</td><td>4.15</td><td>2.76</td><td>1.40</td><td>+51%</td><td>2.74</td><td>3.76</td><td>-1.02</td><td>-27%</td></tr><tr><td>5</td><td>2</td><td>4.17</td><td>3.30</td><td>0.87</td><td>+26%</td><td>2.35</td><td>3.41</td><td>-1.07</td><td>-31%</td></tr><tr><td>6</td><td>5</td><td>3.11</td><td>2.47</td><td>0.64</td><td>+26%</td><td>2.47</td><td>2.45</td><td>0.02</td><td>+1%</td></tr><tr><td>7</td><td>5</td><td>2.28</td><td>1.73</td><td>0.56</td><td>+32%</td><td>3.58</td><td>2.27</td><td>1.31</td><td>+58%</td></tr><tr><td>Ave</td><td>4.33</td><td>3.18</td><td>2.53</td><td>0.65</td><td>+25%</td><td>2.45</td><td>2.94</td><td>-0.49</td><td>-12%</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Average error transition of non-potential absolute pitch.</figcaption></figure>

- Potential absolute pitch with the proposed system: We confirmed that there was no significant difference (N=4, p-value=0.625, Z=-0.730).

- Potential absolute pitch with the existing system: We confirmed that there was no significant difference (N=4, p-value=0.875, Z=-0.365).

- Non-Potential absolute pitch with the proposed system: We confirmed a significant difference (N=6, p-value=0.0313, Z=2.20).

- Non-Potential absolute pitch with the existing system: We confirmed that there was no significant difference (N=6, p-value=0.438, Z=-0.943).

In summary, it showed significance only when people with non-potential absolute pitch trained with the proposed system. No significance was shown when the non-potential absolute pitch trained with the existing system or when the potential absolute pitch trained with either the proposed system or the existing system.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Self-Evaluation of the Pitch of the Participants</figcaption></figure>

## 5.3 User Study

**Self-Evaluation of the Pitch** Figure 5 shows the results of the self-evaluation of the participants' sense of pitch before and after the experiment. After the training task, some participants responded that their sense of pitch improved slightly, but there was no significant difference. The participants were asked the reason for the evaluation after the experiment. Positive opinions included “I learned how to get to the scale (P3)” and “I felt that there was less deviation (P6)”. However, some participants reviewed their self-evaluation of the sense of pitch by saying, “I noticed that there was no sense of pitch (P4, P10).”

**Usability** After completing all the experimental tasks, we asked the questions about the system's usability as shown in Fig. 6 and 7. The following is a summary of the subjective feedback. More than 50% of the participants answered that they were satisfied with the usability. Although the difficulty level of the sound test and training task was not appropriate for P8, who had with musical experience, 70% of the participants in the experiment answered that it was appropriate. 90% said they were less tired overall, and the degree of fatigue was only 50% for the eyes and 30% for the ears. Additionally, we asked whether the existing system or the proposed system could be used for long-term sound training (e.g., 1 hour/day for a month). 80% said that they would like to use the proposed system. When asked about improvements, they mentioned some related to the user interface and feedback system. One of the user interface improvements was to change the button layout. Regarding vocal feedback, it was recommended that the voice of the vocal feedback should be selectable. There was also a desire for multiple choices of feedback voice and for shortening the long word, “sharp”.

**User Interview** Figure 8 displays the impressions of the users of the proposed system. 90% of the respondents displayed a lack of confidence regarding listening to music and singing with others. Most said that the proposed system did not affect their motivation for music activities: “music is fun even if you do not know the scale (P4),” and “music is naturally fun (P9).” However, more than 50% said
<figure><img src="image_8.png" /><figcaption>Fig. 6. Questionnaire about Usability</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Questionnaire about Usability2</figcaption></figure>

that they became more aware of pitch (Third question in Fig. 8) and more than 80% said they thought the proposed system is a great way to improve their pitch (Fourth question in Fig. 8).

# 6 Discussion

For the “potential absolute pitch” participants, as discussed in section 5.2, neither the proposed system nor the existing one showed significant training effect. One possible reason is that the error recorded in test C before training for potential absolute pitch participants was initially small. Thus, it was difficult to show improvement. However, as a result of the discussion in section 5.2, in the case of non-potential absolute pitch participants, the average improvement rate was 25% higher than before training and 37% higher with the existing system. In addition, Wilcoxon signed-rank test comparing the average errors before and after the proposed system’s training confirmed a significant difference (section 5.2). Thus, the proposed system may show beneficial results regarding the absolute pitch training of non-potential absolute pitch participants.
<figure><img src="image_10.png" /><figcaption>Fig. 8. User Interview</figcaption></figure>

Figure 9 shows the distribution of pitch improvement values in the proposed system between non-potential and potential absolute pitch. From this figure, it is clear that the non-potential absolute pitch had a higher improvement value than did the potential absolute pitch. This is because the potential absolute pitch score before training was already high. Thus, the score after training was unlikely to change. Therefore, as shown in Section 5.2, we investigated the improvement value by limiting it to the non-potential absolute pitch. As a result, there was a significant improvement in the absolute pitch. Thus, it is considered that the proposed system has certain advantages over the existing system.

# 7 Future Work

## 7.1 Relative Pitch Training

In this study, the experiment was designed to train absolute pitch. However, initially, it could also train relative pitch. When training the relative pitch, the idea of movable *Do* is applied. For example, with the voice that feeds back the pitch name of *Si/B* when *La/A* is used as the fundamental tone, the pitch remains *Si/B*, but the pitch name becomes *Re/D*. Because it is possible to test the relative pitch using this principle, future verification is needed.

## 7.2 Non-GUI Application

In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions. However, by limiting the system input to an auditory system only, ubiquitous sound training becomes possible, as opposed to the GUI-based system.

**Voice Input** In the training task, by inputting the user's voice into the system input, it is possible to perform sound training by vocalization. Therefore, the
<figure><img src="image_11.png" /><figcaption>Fig. 9. Distribution of Pitch Improvement values in the Proposed System</figcaption></figure>

users can train their pitch sense by simply wearing earphones with a microphone. Additionally, users can improve their ability to utter their voice at the height of the pitch they imagined. As a case study, users could walk around outside while training their pitch sense and vocalization.

**Gesture Input** By using a pressure sensor-type slide bar attached to the hand as a system input, the users can perform a training task by indicating pitch by gesturing the sliding of the finger. As a case study, users can employ gesturing when in crowds or when it would otherwise be difficult to sing.

8 Conclusion

This study proposes a novel auditory-centered training system using solmization to artificially acquire and train absolute pitch by providing vocal feedback to a musician's musical notes. Our contributions are as follows. First, we constructed a system that allows users to recognize note names aurally via solmization. Second, we compared our vocal feedback system with existing visual feedback systems and identified which type is more effective. In an experiment with 10 participants, our system's training improved six non-potential absolute pitch users' absolute pitch by approximately 25%. We also proposed GUI-free pitch training by vocalization or reflecting pitch with finger motion, which is beneficial in situations where it is difficult to hear or speak. Future verification is needed on relative pitch training and GUI-free application with vocal feedback.
References

1. Ashby, W.R.: An introduction to cybernetics. Chapman & Hall Ltd (1961)

2. Bachem, A.: Absolute Pitch 161 (1948), 1180–1185 (1948)

3. Callaghan, J., Thorpe, W., van Doorn, J.: The science of singing and seeing pp. 15–18 (05 2004)

4. Christiansen, L.H., Frederiksen, N.Y., Jensen, B.S., Ranch, A., Skov, M.B., Thiru-ravichandran, N.: Don't look at me, i'm talking to you: Investigating input and output modalities for in-vehicle systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 675–691. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)

5. Crutchfield, W.: There may be more to music than meets a typical ear. The New York Times (1990, December 23)

6. Duan, Z., Gupta, C., Percival, G., Grunberg, D., Wang, Y.: Seccima: Singing and ear training for children with cochlear implants via a mobile application (2017)

7. Gaver, W.W.: Auditory icons: Using sound in computer interfaces. Hum.-Comput. Interact. 2(2), 167–177 (Jun 1986). https://doi.org/10.1207/s15327051hc102023

8. Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453

9. Hoppe, D., Sadakata, M., Desain, P.: Development of real-time visual feedback assistance in singing training: a review. Journal of Computer Assisted Learning 22(4), 308–316 (2006). https://doi.org/10.1111/j.1365-2729.2006.00178.x

10. Howard, D.M.: Technology for real-time visual feedback in singing lessons. Research Studies in Music Education 24(1), 40–57 (2005). https://doi.org/10.1177/1321103X050240010401

11. Howard, D.M., Welch, G.F.: Microcomputer-based singing ability assessment and development. Applied Acoustics 27(2), 89 – 102 (1989). https://doi.org/10.1016/0003-682X(89)90002-9

12. Itoh, K., Suwazono, S., Arao, H., Miyazaki, K., Nakada, T.: Electrophysiological correlates of absolute pitch and relative pitch. Cerebral cortex (New York, N.Y. : 1991) 15, 760–9 (07 2005). https://doi.org/10.1093/cercor/bhh177

13. Kin Wah Edward, L., Anderson, H., Hamzeen, H., Lui, S.: Implementation and evaluation of real-time interactive user interface design in self-learning singing pitch training apps (09 2014). https://doi.org/10.13140/RG.2.1.1757.1682

14. Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)

15. LANGFELD, H.S.: Psychological monographs. Psychological Review Publication (1932)

16. Leong, S., Cheng, L.: Effects of real-time visual feedback on pre-service teachers' singing. Journal of Computer Assisted Learning 30(3), 285–296 (2014). https://doi.org/10.1111/jcal.12046

17. Matsuda, M., Igarashi, H., Itoh, K.: Auditory t-complex reveals reduced neural activities in the right auditory cortex in musicians with absolute pitch. Frontiers in Neuroscience 13, 809 (2019). https://doi.org/10.3389/fnins.2019.00809, https://www.frontiersin.org/article/10.3389/fnins.2019.00809
18. Nakano, T., Goto, M., Hiraga, Y.: Mirusinger: A singing skill visualization interface using real-time feedback and music cd recordings as referential data. In: Ninth IEEE International Symposium on Multimedia Workshops (ISMW 2007). pp. 75-76 (2007). https://doi.org/10.1109/ISM.Workshops.2007.19

19. Nylander, S., Jacobsson, M., Tholander, J.: Runright - real-time visual and audio feedback on running (04 2014). https://doi.org/10.1145/2559206.2574806

20. Profita, J., Bidder, T.G.: Perfect pitch. American journal of medical genetics **29**(4), 763-771 (1988). https://doi.org/10.1002/ajmg.1320290405

21. Richardson, P., Kim, Y.: Beyond fun and games: A framework for quantifying music skill developments from video game play. Journal of New Music Research **40**(4), 277-291 (2011). https://doi.org/10.1080/09298215.2011.565350

22. Rossiter, D., Howard, D.: Albert: a real-time visual feedback computer tool for professional vocal development. Journal of voice : official journal of the Voice Foundation **10**(4), 321-336 (December 1996), http://europepmc.org/abstract/MED/8943135

23. Shiraishi, M., Ogasawara, K., Kitahara, T.: Hamokara: A system that enables amateur singers to practice backing vocals for karaoke. Journal of Information Processing **27**, 683-692 (2019). https://doi.org/10.2197/ipsjjip.27.683

24. Takeuchi, A.H., Hulse, S.H.: Absolute pitch. Psychological Bulletin **113**(2), 345-361 (1993). https://doi.org/10.1037/0033-2909.113.2.345

25. Welch, G., Himonides, E., Howard, D., Brereton, J.: VOXed: Technology as a meaningful teaching aid in the singing studio, pp. 166-167 (01 2004)

26. Welch, G., Howard, D., C, R.: The singad (singing assessment and development) system: A classroom-based pilot study with poor pitch singers. Canadian journal of Music Education, Research Edition (01 1988)

27. Wiener, N.: Cybernetics: or Control and Communication in the Animal and the Machine. MIT Press, Cambridge, MA, 2 edn. (1948)