Proceedings of the IEEE International Conference on Multimedia and Expo (ICME) 201710-14 July 2017

STRUMMER: AN INTERACTIVE GUITAR CHORD PRACTICE SYSTEM
Shunya Ariga1 , Masataka Goto2 , Koji Yatani1
1 Interactive Intelligent Systems Laboratory, The University of Tokyo{ariga, koji}@iis-lab.org
2 National Institute of Advanced Industrial Science and Technology (AIST)

m.goto@aist.go.jp
ABSTRACTMusical instrument playing is a skill many people desire toacquire, and learners now have a wide variety of learningmaterials. However, their volume is enormous, and novice
learners may easily get lost in which songs they should practicefirst. We develop Strummer: an interactive multimedialsystem for guitar practice. Strummer provide data-driven andpersonalized practice for learners in order to identify importantand easy-to-learn chords and songs. This practice design isintended to encourage smooth skill transfers to songs thatlearners even have not seen. Our user study confirms thebenefits and possible improvements of the Strummer system.
In particular, participants expressed their positive impressions
on lessons provided by the system.
Index Terms— Musical instrument training; musicalchords; data-driven approaches.
1. INTRODUCTION
Playing musical instruments requires motor skills of fingers
and hands. Training for these instruments can be demanding
and time-consuming though many people have strong desire
to acquire such skills. This problem is more prominent inpolyphonic instruments, such as guitars and pianos. These
instruments involve playing chords (i.e., harmonic sets of morethan two notes). Learners thus have to develop sufficientdexterity to play chords, which can be discouraging.
There are many possible approaches for self-training.
Learning materials for self-training include chord books(book illustrating finger placements for chords); music theory
books discussing relationships between chords; and textbooks
providing comprehensive tutorials and exercises. Many online
resources for music training are also available. Examples arechordify1 , GuitarTricks2 and Yousician3 . However, the volume
of available materials is enormous, and learners without
experience and knowledge cannot easily identify songs tochoose for their training. For instance, novice learners may not

be aware of which songs are easy or difficult to practice. Somesongs may have important characteristics for effective training(e.g., songs including chords frequently used in other songs).
This motivates us to explore another computer-supported
music training approach mainly targeted to novice learners.
We present Strummer, an interactive guitar chord practicesystem. Strummer considers three characteristics of chords
in a song for prioritization within the given set of music
annotations. These characteristics include: chord difficulty,
chord frequency, and chord transition difficulty. It thenprioritizes songs including chords and their transitions thatare frequently-used and easy-to-play. Learners can go throughstep-by-step, interactive tutorials that are designed to practicefrom a chord to the entire song. Tutorial contents also change
depending on song sets that learners choose in order to provide
personalized tutorials for accommodating their preferences.
Our contributions are two-folded:• A chord and song prioritization algorithm: Ouralgorithm identifies which songs users should practice firstbased on three elements of chords.
• An interactive system that provides guitar chord
training: Strummer provides step-by-step tutorials topractice chords and songs in a guitar. It can also analyze
audio recordings of guitar performances by learners tojudge whether each chord is correctly performed.

1 https://chordify.net/
2 https://www.guitartricks.com/
3 https://get.yousician.com/

978-1-5090-6067-2/17/$31.00 ©2017 IEEE

2. RELATED WORK
It is pedagogically desirable that learners are directly
taught by an instructor from the beginning, and start withbasic principles [1]. However, human instructors are
expensive and not always available. Furthermore, supporting
self-motivation and independence is important for successful
skill acquisition [2].
Playing musical instruments requires a considerable
amount of training and effort. Prior work has investigated
how computers can support learners’ practice of musicalinstruments. Dannenberg et al.’s Piano Tutor [3] is one of
the early work in this space. This system analyzes learners’
errors by taking matches with the given score and using
training heuristics. Takegawa et al. [4] developed a system

ICME 20171057

that recognizes learners’ finger and hand positions on the
piano, and then offers visual feedback onto the piano keys
by a projector on the ceiling. Wearable devices are often
used for supporting musical instrument practice. Johnson etal. [5] created a system to support a practice of playing theviolin with wearable sensors. Huang et al.’s Piano touch [6]informs learners of finger movements in piano playing throughvibrotactile feedback with glove-shaped devices.
There are several projects targeted to guitar practice.
Motokawa et al. [7] created a system to teach finger placementon the guitar strings by displaying a hand position image of thegiven chord. Barthet et al.’s Hotttabs [8] offers online practicematerials, containing multimedia contents, including video
tutorials and guitar TABs.
There are commercially available games and systems
for encouraging learners’ practice, and they greatly inspired
some of our interface design. Guitar Hero4 uses a simplified
guitar device, and users press buttons as the display instructs.
Rocksmith5 provides a similar game environment to Guitar
Hero, but users can connect a real guitar to the system.
Yousician is an online training system for pianos, guitars,
and ukuleles. It offers step-by-step tutorials and immediate
feedback on the user’s performance. Yousician is one of theclosest systems to Strummer, but we incorporate a data-driven
approach for prioritizing chords and songs for guitar practices.
3. MUSIC AND CHORD DATASET
3.1. Preprocessings
We used Billboard dataset [9] for the source of chord practicematerials. The dataset includes chord notations of popular 890
songs (151 songs are duplicated thus it contains 739 uniquesongs) which ranked in the Billboard “Hot 100” chart in U.S.
at least once. The notation is described in a context-free chordtransition representation developed by Harte et al. [10].
We decided to remove twelve songs that only containfewer than three chords because they are considered to havelittle training effect. We built a parser and analyzer for thischord dataset. In pre-processing, the parser first extracts thebasic information of a song, including the title and artist.
It then parses the chord information that contains timing
and chord symbols. Most songs in the music dataset haverepeated section patterns (e.g., verses and choruses). The
parser performs simple matching of descriptions available inthe dataset to mark such patterns. During the parsing, we
broke down each song to smaller components for our tutorials.
We fixed the time signature as 4/4 (i.e., each bar consists four
quarter notes). A bar containing more than three different
chords can be difficult to practice, thus we divide it into
two half notes. We define a segment as a part of a section

containing four bars (with an exception that the last segment
can contain fewer bars).
We performed two kinds of simplification before
developing Strummer. First, we performed representation
simplification (e.g., C : ma j to C). We then executed chord
simplification on tension chords except add 9th chords. For
example, the chord of minor 9th is converted to minor 7th.
We also converted inversions and fraction chords into simple
forms. The types of guitar chords we use in Strummer
are: ma jor, minor, 7th, major 7th, minor 7th, minor major
7th, augumented, diminished, half-diminished 7th, major 6th,
minor 6th, add 9th, suspended 4th, and 5th(power chord).
After finishing all processing, our parser then converts
all information into a JSON file in the format compatible
with Songle Widget [11]. We use this converted data in ourStrummer system.
3.2. Chord and Transition Difficulty Rating
As our application is guitar practice, we also need to consider
how difficult each chord transition is to play. We first
determined chord difficulty through manual rating. Two raters,
including one of the authors, independently evaluated chorddifficulty with a 5-Likert scale (1: easiest – 5: most difficult).
The raters were instructed to consider only the complexity of
the finger placement for each chord. They rated 180 chords
including 163 appeared in the dataset. As a result, 52.2% of
the chords had the exact same ratings. The square-weighted
Cohen’s Kappa was κ = .81 (95% CI: [.75, .85], indicating asubstantial agreement. We use the average rating as a score ofchord difficulty in our system.
We also developed difficulty ratings for chord transitions.
However, the number of possible chord combinations in thedataset is theoretically 26,569, and it is infeasible to manuallyrate all of them. We thus decided to use a statistical approachto determining chord transition difficulty. First, we obtained
5-Likert scale ratings for 100 most frequently-used transitions
in a similar manner to the chord difficulty through manual
labeling by two raters. In addition, we randomly chose 30 ofchord transitions, and rated in a similar manner to broaden the
coverage of our model. Through our manual ratings, 60.0% of
the 130 transitions had the exact same difficulty scores. The
square-weighted Cohen’s Kappa was κ = .76 (95% CI: [.66,.84], indicating a substantial agreement.
Using these sampled transitions with ratings, we ran alinear regression to create a model to estimate the difficulty of
all chord transitions. We initially started with two potential
independent variables, and used AIC for determining the
optimal model. As a result, we had the following independent
variables and estimated coefficient values (p < .001 for all)besides the intercept of 0.790:

4 https://www.guitarhero.com/
5 http://rocksmith.ubi.com/rocksmith/1058• diff_t (0.310): The difficulty of the target chord.
• f ig_num (0.211): The increase or decrease of the numberof middle, ring, and little fingers (ranged from -2 to 2). We

excluded the index finger for this count as it is involved in
more than 80% of the chords.
• log (move_base + 1) (0.336): The logarithm of the absolute
movement of the hand, defined as the difference in the fretpositions.
• move_ f igR (0.233): The absolute movement of the ring
finger. It is defined as the Euclidean distance of the ring
finger’s position the between the origin and target chord.
For example, if the ring finger is moved from the second
fret in the fourth√ string to the third fret in the fifth string,
this value is 2. It is zero when the origin or target chord
does not involve the ring finger.
• move_ f igL (0.242): The absolute movement of the little
finger. The definition is similar to move_ f igR.
• barre_t (0.481): The target chord is a barre (1 if true,
otherwise 0).
• slide (-1.401): Both origin and target chords have the samebarre form (1 if true, otherwise 0). Their positions can bedifferent.
This model achieved high R-squared value, R2 = .786. We
use this model to estimate the difficulty of the given chordtransition, expressed in the value ranging [1.0, 5.0].
4. CHORD AND SONG PRIORITIZATION
Strummer needs to identify which chords and songs learners
should practice first. This information determines which songs
they should prioritize for practice.
We first map all chords in the coordinate of difficulty
and inverse frequency of occurrences. The frequency ofoccurrences of the chord ci is calculated as fci = Nci / ∑C Nci .
The denominator represents the total number of occurrences of
all chords in the given song set. We define chord primariness,
CPci , for the chord ci as an Euclidean
distance from the originq

in this coordinate: CPci = − w f (1/ fci )2 + wd dci 2 , where drepresents the difficulty of a chord, and w f and wd are theassociated coefficients (w f = 0.1 and wd = 10 in our current
prototype system). We add the negative coefficient of −1 sothat higher CP represents a more important and easy-to-learn
chord.
With CP, we can calculate the primariness of each song,indicating how likely it includes chords that are important andeasy to learn. In addition, this calculation should consider
the difficulty of chord transitions. Therefore, we formulate
the song primariness, or SP, as SP = ∑C CPci + wTh ∑Th T D +wT f ∑T f T D, where C, Th , and T f represent all chords, its
transitions in the half note, and transitions in the whole note.
We set wTh and wT f to −1. In this manner, we can only penalize
highly difficult transitions in the whole note. The maximum
computational complexity of SP is O(n(n + m)) where n is theaverage number of chords (10.2) in the given songs and m is
its total number (727). It is possible to run this calculation for
all songs in real time.

1059

Fig. 1: Plots of k%-playable songs (genre: pops) across the number ofsongs completed (k = 80 and 100) in two ranking algorithms (SP and
BL (the baseline method based on the total chord difficulty scores)).

To validate benefits of SP, we define an unseen song to
be “k%-playable” if learners would be able to perform k% of
the chords in that song. 100%-playable means that learners
have already practiced all chords in the given song. We regard
song prioritization as effective if it achieves a higher numberof k%-playable with practice in a small number of song.
We compared SP against the total chord difficulty (TCD)
as a baseline approach (BL). TCD is defined as the sum ofdifficulty scores of all chords in a song. Sorting with the BL
method means that the system prioritizes songs that have fewer
and easy-to-play chords. Fig. 1 shows the number of 80%
and 100%-playable songs under an assumption that learners
have completed practicing the first up to 20 songs in the two
rankings. In both 80% and 100%-playable, SP constantly
exhibits a higher number than BL. This means that the chords
learners would practice can be more useful to play unseen
songs in SP. Strummer thus uses SP to rank songs in the given
set, and sorts them in a descending order of SP.
5. STRUMMER SYSTEMThe Strummer interface provides step-by-step tutorials to offer
efficient chord practices. When learners log into the system,
they first can choose genres, artists, and songs they desire
to practice. Strummer then displays the songs in the chosen
set in the order of SP by default (Fig. 2a). This list also
provides information about the number of chords and sections
in each song, and learners can sort them if they like. Learners
can select a song to start tutorials. Our accompanying video
provides a demonstration of the entire system.
5.1. Strummer Tutorials
Tutorials in the Strummer system are designed to provide
step-by-step practice for the given song. As we describedFig. 2: The walkthrough of the Strummer system. Refer to the accompanying video for the full demonstration. (a) Learners choose songs topractice. They are sorted by SP. (b) Each song is divided into sections to make practices manageable. (c) Strummer has six practice stages.
(d) Learners can listen to chords along with finger placements in the Listen stage. (e) In the Chord Practice stage, The chords are sorted ina way that learners start with frequently-used and easy-to-play chords. (f) Learners practice chords one-by-one in the Chord Practice stage.
(g) Strummer offers additional, lower-level tutorials for unsuccessful chords (Bm in this example). (h) The system records a partial finger
combition of a chord learners can play, and only enables performance assessment in later tutorials. Note that the finger placement of Bm isdifferent from the original (shown in (f)). In this manner, Strummer encourages learners to practice the entire song even if they cannot playsome chords well. (i) Learners practice chord transitions in a similar way to individual chords. (j) A screen for practicing a segment of thegiven song. (k) Learners practice a portion of the song in the Rehearsal stage. (l) Learners can also practice the entire song with a similarinterface to music games.

above, a song generally has some repetitions (i.e., sections).
The system first removes such repetitions and generates lessons
each of which contains a unique set of chords. When learners
choose a song, the system presents a set of tutorials each ofwhich focuses on practicing one of the sections (Fig. 2b).
Learners can start a tutorial by simply clicking it. During atutorial, Strummer provides six practice stages explained laterin this section (Fig. 2c).
• Listen Stage: At the beginning of a practice, learners areencouraged to listen to the entire section before starting
practice. They can listen to the song along with chord
information (Fig 2d).
• Chord Practice Stage: In this stage, learners practice
all chords used in the chosen section one-by-one. The
system sorts the chords by CP in the descending order(Fig 2e). When learners click a chord, the Strummer system
displays the finger placement in a similar fashion to guitar
TAB (Fig 2f). The system plays the chord in a strummingstyle. Learners can then confirm what the chord sounds
like. When learners feel ready, they press the start button
to check their performance. The system then activates1060

an audio-based chord recognition module (we explain its
details later). The Strummer interface then informs themwhether they have successfully played the given chord.
Upon unsuccessful performance, it offers feedback about
which strings the learners failed to play (Fig. 2f). If they areable to play the chord three times continuously, the system
records the completion.
• Supplemental Chord Practice: When they miss the chordmore than five times, or missed three times in sequence,the system enables a button for additional tutorials. The
learners can continue to practice the chord separately inthose tutorials (Fig. 2g). This separate tutorials offers
lower-level training. It starts with practice for each finger,
and progressively increases the number of fingers to be
used. However, learners still may struggle playing a chordeven after this additional training. The system records withwhich finger combinations the learner have been successful,and enables performance assessment only for these fingers
in later tutorials (Fig. 2f, g). For example, suppose that alearner can play the chord C with the index and middle

finger, but not with all three fingers. The system thendisables performance assessment for the ring finger.
• Chord Transition Practice Stage: Learners can practicechord transitions in a similar manner (Fig. 2h and i). The
chord transition practice stage offers the same tutorial
except that learners practice transitions.
• Segment Practice Stage: After completing the chordtransition practice stage, learners move to practice thechosen segment. Similarly to the previous stage, the system
displays the chord name and finger placement (Fig. 2j).
When the learners press the start button, an animation
starts over the chord sequence, indicating the timing toplay. The system also activates the chord recognition
module, and provides feedback about whether the learners
have successfully played each chord at the right timing.
The learners can complete this stage by correctly playing
the segment at least once. For the chords learners havenot completed even after additional training, the systemdisables the recognition module, and synthesizes the soundon behalf of the learners as scaffolding. In this manner,
the learners can continue their practice without being
constrained to chords they cannot play well.
• Rehearsal Stage: This stage (Fig. 2k) is intended toprovide additional repetitions in practice, helping skill
fixation [12]. The system marks the stage as completed
when learners are able to play all chords successfully. But
they can move to the next stage even if they have notcompleted.
• Live Concert Stage: At the end of the tutorial, learners areinvited to play the song they have been practicing as if theywould do in a concert. We take a similar user interface topopular music games and training systems, such as Guitar
Hero and Yousician (Fig. 2l). In this stage, learners are
asked to play all sections they have completed as well as
what they are currently practicing. After the performance,
the screen displays the score based on the number ofsuccessfully-played chords. Learners can complete thelesson by being able to perform 60% of the chords.

5.2. Chord Recognition
Fig. 3 shows the overview of our chord recogition workflow.
We implement the Strummer system in Ruby on Rails and
JavaScript. Fast Fourier Transform (FFT) in Web API does
not take sufficiently long samples for chord recognition. We
thus run the chord recognition module locally on the learners’
machine. It is written in Python with the numpy and pyaudio
libraries. The sampling rate is 44.1 kHz, and the window
size of FFT is 8192 samples. We used the Hanning window
in sampling. The module then matches the fundamental
frequencies of a chord with the observed frequency peaks
to determine whether it is correctly played. The module sends
the recognition results to the main system via HTTP.

6. USER EVALUATION
We conducted a preliminary user study to examine the effect ofStrummer. We mainly investigated the user experience of thefeatures and tutorials to validate the design of the Strummer
interface. We recruited 5 people (P1 – P5, all male, the averageage of 21.6). All of them had nearly no experience of playing
the guitar.
We designed our experiment to be five days long. We
asked our participants to come to our lab every day during theexperiment. They were instructed to practice with the given
condition for at least one hour. Two of our participants were
not able to accommodate this schedule, and they practiced fortwo hours at the fourth day. We lent the participants a classicalguitar during the experiment.
At the beginning of the first day, we explained the interface
and experimental procedure. We also manually provided
basic tutorials about how to play the guitar. After thispre-experimental briefing, the participants were asked topractice starting from the first song in the given list.
When participants completed practicing one song, we
conducted a short interview to understand their experience.
In addition, we conducted another short interview to elicit any
opinion about the system at the end of each day.
On the last day, we asked participants to perform a newsong in the dataset that they had not practiced. The intentionof this sight reading task was to emulate experience of playing
at sight, requiring sufficient skill fixation on chords. We
also wanted to ensure that the participants learned how toplay chords. We chose “(Sittin’ On) The Dock Of The Bay”
by Otis Redding. This song requires quick transitions infrequently-used chords (C, D, G), and also includes bar chords(F, B) and unseen chords (A, B). After 15-minute practice, they
played the song in front of the experimenter. We audio-taped
their performance for later analysis.
Our interviews were not in English because none of ourparticipants was native or proficient in the language. We
transcribed and translated the interviews to English for thereport in this paper. We offered the participants approximately
50 USD in local currency as compensation at the end of thestudy.
7. RESULTS
All participants successfully completed the study. One of theauthors evaluated the performance of the sight reading task.
Table 1 presents the number of successful chord playing in thesight reading tasks, showing that all participants performed atleast half of all chords correctly.
Our interviews also uncovered perceived benefits of thefeatures in the Strummer interface. Participants agreed thatstep-by-step tutorials were beneficial. P3 expressed his
appreciation of the tutorials as follows:1061

Fig. 3: An overview of our chord recognition workflow.
Seen
Chord

Em

Occurrences
Difficulty

UnseenB[2] G. Percival et al., “Effective use of multimedia forcomputer-assisted musical instrument tutoring,” in Proc.
EMME, 2007, pp. 67–76.

D

C

E

G

F

A

Total

2

4

5

1

14

1

3

2

32

1

1.5

2

2

2

3

2

3.5

2.13(ave.)

P1
P2
P3
P4
P5

2
1
1
2
2

4
4
0
4
4

3
3
3
5
5

1
0
0
1
1

14
12
12
14
13

0
1
0
0
1

3
0
0
3
3

1
0
0
0
2

28
21
16
29
31

[3] R. Dannenberg et al., “A computer-based multi-media
tutor for beginning piano students,” Journal of New
Music Research, vol. 19, no. 2–3, pp. 155–173, 1990.

Success Rate (%)

85

82

76

50

93

40

57

25

78

[4] Y. Takegawa et al., “Design and implementation of a
piano practice support system using a real-time fingering
recognition technique,” in Proc. ICMC, 2011, pp.
387–394.

Table 1: The results of the sight reading task.

It’s like I complete one-by-one, step-by-step. That gave
me a good motivation. [P3]
In the step-by-step tutorials, the system included the chord
recognition module to assess how well the participants played.
This feature was also useful to the participants.
I really like that the system checks my performance...
When the system showed unsucessful placement, I foundthat I did not put my fingers strongly enough. [P2][5] R. Johnson et al., “Real-time feedback for learning theviolin,” in Proc, BCS-HCI, 2011, pp. 535–538.
[6] K. Huang et al., “Pianotouch: A wearable haptic piano
instruction system for passive learning of piano skills,”
in Proc. IEEE ISWC, 2008, pp. 41–44.
[7] Y. Motokawa and H. Saito, “Support system for guitar
playing using augmented reality display,” in Proc. IEEE
ACM ISMAR, 2006, pp. 243–244.

I think the sound feedback was helpful when I was busy
with the guitar and was not able to pay attention to thedisplay. [P4][8] M. Barthet et al., “Music recommendation for music
learning: Hotttabs, a multimedia guitar tutor,” in Proc.
WOMRAD, 2011, pp. 7–13.

8. CONCLUSION
Many people have a desire to acquire musical instrument
playing skills. Although recent online services offer avariety of training materials, the volume of such resources isoverwhelming, and learners may get lost in which to practice.
We develop models and algorithms to identify importantand easy-to-learn chords given song genres and artists. We
instantiate them in Strummer, an interactive system for guitar
chord practice tailored toward novice learners. Strummer
provides step-by-step tutorials to make practice manageable.
Our user study confirmed that participants were able tocontinue their practice over five days and showed positive
attitudes toward our Strummer system.
9. REFERENCES[1] A. Glise, Classical Guitar Pedagogy: A Handbook for
Teachers, Mel Bay Publications, Inc., 1997.

1062[9] J. A. Burgoyne et al., “An expert ground truth set foraudio chord recognition and music analysis,” in Proc.
ISMIR, 2011, pp. 633–638.
[10] C. Harte et al., “Symbolic representation of musicalchords: A proposed syntax for text annotations,” in Proc.
ISMIR, 2005, pp. 66–71.
[11] M. Goto et al., “Songle widget: Making animation andphysical devices synchronized with music videos on theweb,” in IEEE ISM, 2015, pp. 85–88.
[12] P. M. Fitts and M. I. Posner, Human performance.,
Brooks/Cole, 1967.