This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
1Computational Intelligence in Music Composition:
A Survey
Chien-Hung Liu and Chuan-Kang Ting

Abstract—Composing music is an inspired yet challenging task,
in that the process involves many considerations such as assigning
pitches, determining rhythm, and arranging accompaniment.
Algorithmic composition aims to develop algorithms for music
composition. Recently, algorithmic composition using artificial
intelligence technologies received considerable attention. In particular, computational intelligence is widely used and achieves
promising results in the creation of music. This paper attempts
to provide a survey on the computational intelligence techniques
used in music composition. First, the existing approaches arereviewed in light of the major musical elements considered incomposition, to wit, musical form, melody, and accompaniment.
Second, the review highlights the components of evolutionary
algorithms and neural networks designed for music composition.
Index Terms—computational intelligence, music composition,
evolutionary computation, neural networks.

I. I NTRODUCTION
Music plays an essential role in our daily life. It serves asa significant medium to entertain people, deliver messages,
improve productivity, and express moods and emotions. Composing melodious music is a challenging task since manymusical elements need to be considered, such as pitch, rhythm,
chord, timbre, musical form, and accompaniment [99], [125].
In the past, music composition was usually accomplished bya few talented people. The algorithmic composition, which
formulates the creation of music as a formal problem, facilitates the development of algorithms for music composition.
The concept of algorithmic composition can be traced backto the Musikalisches Würfelspiel (musical dice game), which
is often attributed to Mozart. In the game, a music pieceis generated by assembling some music fragments that are
randomly selected. Mozart’s manuscript K.516f, written in
1787, is commonly viewed as an example piece of the musicaldice game because it contains many two-bar fragments, eventhough the random selection by dice is not evidenced.
The algorithmic composition enables automatic composition
by using computers and mathematics. In 1957, Hiller andIsaacson [60] first programmed the Illinois automatic computer(ILLIAC) to generate music algorithmically. The music piece
was composed by the computer and then transformed into
a score for a string quartet to perform. Xenakis [150] in1963 created a program as an assistant to produce data forhis stochastic composition. The research on algorithmic composition has significantly grown since then. Some overviews
on automatic composition can be found in [9], [10], [45],The authors are with the Department of Computer Science and Information
Engineering, National Chung Cheng University, Chiayi 621, Taiwan (e-mail:lch101p@cs.ccu.edu.tw; ckting@cs.ccu.edu.tw).

[68], [107]. Moreover, there have been an abundant amountof software and applications for computer composition andcomputer-aided composition, such as Iamus [1], GarageBand[2], Chordbot [3], and TonePad [4]. Notably, the Iamus system[1] is capable of creating professional music pieces, some ofwhich were even played by human musicians (e.g., the London Symphony Orchestra). GarageBand [2] is a well-known
computer-aided composition software provided by Apple. It
supports numerous music fragments and synthetic instrument
samples for the user to easily compose music by combiningthem.
The methodology of algorithmic composition includes
mathematics, grammar, and artificial intelligence. First, froma mathematical perspective, composing music can be viewed
as a stochastic process and, therefore, the mathematical models such as Markov chains are useful for composition [34].
The music composition using mathematical models have theadvantages of low complexity and fast response, which areadequate for real-time application. Sertan and Chordia [126]utilized the variable-length Markov model that considers thepitch, rhythm, instrument, and key, to predict the subsequent
sequence of Turkish folk music. Prechtl et al. [116] generated
music for games by using the Markov chains with musicalfeatures such as tempo, velocity, volume, and chords. Voss
and Clark [146] observed and proposed the 1/f noise forcomposition. Hsü and Hsü [64], [65] adopted fractal geometry
in the expression of music and presented the self-similarity forthe musical form.
Second, music can also be regarded as a language with
distinctive grammar [120]. Composing music turns out to be aprocess of constructing sentences using the musical grammar,
which is usually comprised of the rules about rhythm andharmony. Howe [63] codified the musical elements in themulti-dimensional arrays and used the computer to compose
music by determining the locations of pitches and rhythms
through five operators. Steedman [128] and Salas et al. [122],[123] adopted linguistics and grammar to generate music. The
proposed approaches describe music as a language, and thenlearn its patterns from music pieces, to build the model forcomposing a new melody.
Third, the advances of artificial intelligence (AI) promote
its application to algorithmic composition. The composition
systems based on AI technologies such as cellular automata[22], [100], knowledge-based systems [11], [42], machinelearning [41], and evolutionary computation [101], [148], havereceived several encouraging results [47], [112]. In addition
to composition, the AI technologies are applied to expressive
performance of music [33], [78]. Recently, the use of computa-

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence2

tional intelligence (CI) in music composition has emerged. The
CI techniques, including neural networks, fuzzy systems, andevolutionary computation, have achieved remarkable results
and render powerful tools for modeling, learning, uncertainty
handling, search, and optimization. These techniques havebeen applied to music composition. In particular, the evolutionary composition system progresses by making widespreaduse of genetic algorithm (GA) [59], [61], genetic programming(GP) [79], particle swarm optimization (PSO) [74], [75], antcolony optimization (ACO) [38], [39], and other evolutionary algorithms [121]. The population-based, quality-driven,and stochastic nature of evolutionary algorithms makes themespecially suitable for music composition and computational
creativity. Neural networks are often utilized in the learningand modeling processes for music composition. Bharucha andTodd [13] proposed a model using neural networks for theprediction of notes. This work is followed by plenty of studies
on the use of neural networks to compose music. In addition,neural networks are applied to assist the evaluation of compositions [18], [58]. Fuzzy systems cater to the classification
and analysis of music; nonetheless, they are seldom used formusic composition.
This paper provides a survey on the CI techniques usedin music composition to reflect the recent advances in thisarea. The survey is organized from two perspectives: musicalelements and CI technology. First, in the light of musicalelements, the task of music composition entails deliberating
the musical form, creating the melody, and arranging theaccompaniment. The musical form is associated with thefundamental structure, phrases, motives, and music genre. In
composing a melody, the musical elements such as timbre,
pitch, rhythm, harmony, and tune need to be properly assigned.
The emotion is an advanced consideration in generating compositions. As for the accompaniment, the musicians arrange
the main accompaniment, chords, and bass, to enhance theharmony and euphony of the composition. Second, the different aspects of the CI techniques need to be pondered upon formusic composition. In the evolutionary composition systems,
various designs for the representation, crossover, and mutation
are proposed. The music systems using neural networks aredeveloped to generate compositions and assist evolutionary
composition. In addition, the evaluation of compositions is
a paramount issue that needs to be addressed in the CI-based
composition systems.
The paper is organized as follows. Section II reviews thestudies on music composition using musical form, including
motive, phrases, and genre. Section III reviews the approaches
based on CI for composing the melody in terms of timbre,
pitch, rhythm, and emotion. Section IV recapitulates the CI
methods for the accompaniment arrangements. Section V surveys the designs of the CI techniques for music composition.
Section VI provides some suggestions for future research
topics. Finally, a summary of this paper is presented in
Section VII.
II. C OMPOSITION U SING M USICAL F ORMMusic composition involves many musical elements, suchas timbre, pitch, rhythm, motive, phrase, and chord. Figure 1

summarizes the musical elements in composition, which canbe divided into three major categories: musical form, melody,
and accompaniment. Musicians usually deliberate over themusical form as the basic structure and then fill in pitches
to construct the melody. Afterward, The accompaniment is
assigned to strengthen the harmony or complement the composition.
This section reviews the research on the utilization ofmusical form in CI-based composition systems. Moreover,
we discuss the music genre as an advanced consideration forcomposing music.
A. Musical Form
Musical form serves as the fundamental structure of acomposition. While composing music in a specific genre, composers consider the musical form to establish its frameworkand then develop motives and phrases. For example, the form
AABA is ordinarily applied in sonatas, and ABAB and ABCA
are commonly used in modern popular music. The musicalform constitutes the main frame of a composition, where thenotes will be filled in to construct the melody afterward.
In the composition system, Xu et al. [151] adopted four conventional forms of popular music, i.e., ABAB, ABAA, AAAA,
and ABBB (or ABCA). The first two motives are repeated inthese forms for intensification. Liu and Ting [86] considered
the musical forms used by a famous Asian pop-music singer.
Although musical form is a fundamental component in music
composition, there exist few studies addressing this issue in the
CI-based composition approaches. Design and development ofmusical form is hence a direction for future studies.
B. Motive
Motive represents the minimum component repeating insome phrases. The form of repeats can be classified into two
types: sequence imitation and repetition imitation. The former
indicates that the repeated motives are similar but different,
whereas the latter requires they are identical. Both types ofimitation are beneficial to cultivate the hook for the music
and make phrases or compositions memorable.
1) Sequence Imitation: Ricanek et al. [119] considered thethematic bridging issue at music composition. They designed
a fitness function favoring the similarity between phrases
for sequence imitation. In the composition system of Papadopoulos and Wiggins [111], the users can specify theirpreferred motives. The fitness function for the GA will countthe interval patterns matched. Towsey et al. [136] also took thefeatures of patterns into account. The sequence imitation andrepetition imitation are achieved by copying three or four notesfrom repeated rhythm patterns and repeated pitch patterns,
respectively. Calv and Seitzer [24] used GA to generate themelodic motives and adopted the genetic algorithm traversal
tree (GATT) to construct the musical structure. The fitness
function especially rewards the intervals matching the Fibonacci sequence. Long et al. [89] claimed that the melody andrhythm in different phrases should be similar for the coherent
effect. When generating a new phrase, the system checks if theprevious melody can be partially adopted. Loughran et al. [90]2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
3

Harmony

Main Accompaniment
Tune
Pitch

Chord

RhythmComposition
Bass
TimbrePhrase & Motive
Imitation:
Sequence

Figure 1.

Imitation:
Repetition

Musical elements in composition

used grammatical evolution (GE) to generate the sequences fora long melody. The composition was created by merging the
top four individuals obtained from the final generation. The
best individual will share its motifs with others, which will beslightly varied in regards to rhythm and pitch.
2) Repetition Imitation: The system proposed by Chiu andShan [28] repeats the motives resting upon the analysis on
the structure of input music. It divides the phrases into several
segments to form the motives. Liu and Ting [84], [85] and Wu
et al. [149] observed and applied three types of repetitions
commonly used in the creation of music. According to thetype of repetition, the measures involving the repetition arecompared and the best one will replace others to form therepetition. Liu and Ting [86] adopted both the repetition
imitation and sequence imitation to mimic the compositionof a famous Asia pop-music singer. The better motif in thegenerated music will replace other motifs originating from themusical form in the original song.
C. Genre
Music genre is an important consideration for composition.
A genre is formed by a composition style in a specific region,
culture, instrument, or group. Composing music for a certain
genre involves three elements: rhythm, scale, and structure.
Most genres have unique rhythm patterns or scales. For
example, Chinese music uses the pentatonic scale composedof only five pitches. In addition, music genres usually havespecial structures.

Jazz is a music genre popularly studied in the CI and nonCI based composition systems because of its salient featuresin scale and rhythm. For composing jazz music, Papadopoulos
and Wiggins [111] proposed a GA with a representation anda fitness function based on the jazz scales. Biles [16], [17]developed the GenJam system supporting the trade four in jazz
impromptu. The GenJam is capable of improvisation throughinteraction with the jazz player. Tzimeas and Mangina [138]presented a GA-based system to transform the compositionsof Bach into jazz music. The user, who must be familiar withjazz music, listens to an input song of Bach and identifies thenotes with a jazz flavor during the evolutionary process of GA.
Aside from jazz, other music genres are also consideredin the composition systems using CI. The GA composer of
McIntyre [98] focuses on the four-part Baroque harmony,consisting of soprano, alto, tenor, and bass. In light of the
Baroque genre, the fitness function includes the rules specially designed for chord spelling, doubling, voice leading,smoothness, and resolution. Dimitrios and Eleni [36] proposed
the SENEgaL to create Western Africa rhythm music. In thesystem, they introduced different styles of Western Africa
rhythm: Gahu, Linjen, Nokobe, Kaki Lambe, and Fanga. Each
type has different rhythmic patterns that are played on two
or more instruments. The user can specify the proportion ofWestern Africa patterns considered while composing music.
Wang et al. [147] extracted the features, including the repeat
rhythm patterns and special intervallic motives, from Chinese
Jiangnan ditties. They found that the Jiangnan ditties have2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence4

many three-note and four-note intervallic motives; precisely,
more than 80% are three-note intervallic motives. Hence thefitness function rewards the intervallic motives to form thespecialty of Chinese Jiangnan ditties.
Some studies have focused on the composition style ofa specific musician or a music group. Liu and Ting [85]determined the weights for the rule-based fitness function
by using the music and charts information of the famousrock band Guns N’ Roses. They [86] further utilized thesequential pattern mining technique for the patterns of Jay
Chou’s composition style. The patterns obtained are used as
the genes for the GA to create new compositions.
III. C OMPOSITION OF M ELODY
This section introduces the musical elements and the studies
of using CI-based techniques in the composition of melody.
A. Timbre
Timbre represents the tone color of sound. When composing
music, timbre should be considered and properly arranged
because each instrument or vocal type (soprano, contralto,
tenor, baritone, and bass) has its sound characteristics. The
instruments and vocals, for example, have their charactersand need to be organized for the ensemble. Timbre can beclassified into two types: instrument and vocal. For instrumental music, many instruments have their own unique tones,
which should be considered when composing music for them.
In addition, playing techniques, e.g., the bowing techniquesfor violin, influence the timbre of instruments and hence may
be taken into account in composition. As for vocal music,
the compositions are subject to the timbre of each part, i.e.,
soprano, alto, tenor, and bass, of the singers.
Timbre can be applied to the evaluation criterion for theassignment of instruments. Wang et al. [147] extracted thefeatures from Chinese Jiangnan ditties for the fitness function,in which the timbre serves as an evaluation criterion using
the spectral centroid to fit the instruments. Dimitrios andEleni [36] proposed the SENEgaL system generating theWestern Africa rhythm music. In the interactive interface,
the user can choose different timbres of drum to play themusic. Considering the elements and playing techniques ofdrums, Dostál [40] designed a chromosome representation for
GA to generate drum rhythms in the human-like rhythmic
accompaniment system.
B. Pitch
Assigning the pitch for each note is a paramount task inmusic composition. The assignment pertains to range, tune,
and harmony of pitches. Tune is associated with the sequenceof pitches, whereas harmony concerns the concurrence ofmultiple pitches.
1) Range: In the assignment of pitches, the composers must
include the range of instruments or vocals for consideration.
The violin, for instance, has a limit of four and a half
octaves in playable notes, although the range is determinedby the player’s skills. A composition system therefore needs

to consider whether the generated tunes or chords are playable
or not; otherwise, the composition will turn out to be ameaningless adornment with musical notations.
Regarding the range of playable pitches, Tuohy and Potter[137] presented a music system using GA, where the pitch
range is specifically determined for guitars in the composition.
Some studies concern the polyphonic music composition. In[84], [85], the tracks for different instruments are represented
and composed separately according to their individual pitch
ranges and playing techniques. The rhythmic composition
focuses on the features of rhythm, e.g., the stroke, flam, drag,and ruff of drumming. Oliwa [108] designed a GA-based
system treating the tracks of lead guitar, rhythm guitar, rockorgan/piano, and drums. The representation of each track is
specialized according to the purpose, pitch range, and theplaying techniques of the instruments.
The limitation of pitch is crucial to vocal music. A common
genre is the four-part choral music, comprising the soprano,alto, tenor, and bass. McIntyre [98] proposed using GA to generate the four-part Baroque chorus. The fitness function takes
account of the vocal range; moreover, the system suggests
leaving proper spacing between the voice and the melody in all
four parts. Phon-Amnuaisuk et al. [114] also used the fitness
function to keep the pitches within the proper range. Donnelly
and Sheppard [37] and Maddox [91] considered the vocal
range in the fitness evaluation of the four-part chorus music,
where each part should exclude the out-of-range pitches. Geis
and Middendorf [57] presented an ant colony optimizer togenerate the four-part accompaniment, for which the vocal
range is also considered in the fitness evaluation. Long et al.
[89] developed a composition system for human singing. The
system restricts the use of pitches to the range of two octaves
that humans can sing. Muñoz et al. [104] formulated theconstruction of the figured bass as an optimization problem.
With the input of a bass line, the local composer agentscooperate with the population-based agents to generate thecomplete four-part music pieces.
2) Tune: Tune is a pivotal musical component that affects
the listenability of a composition. It is associated with theintervals between pitches and is usually used as the basis forthe assessment of a melody. To assign the sequences of pitches
for a good tune is difficult due to the enormous combinations
and constraints on the notes. A considerable number of studies
have proposed the CI-based methods to deal with this problem.
The methods can be divided into four types: generating thewhole-tune, combining music segments, constructing note bynote, and imitation.
First, the most common way is to generate the whole tune
at one time. This way is usually used in the evolutionary composition systems, where the series of notes can be generated
randomly or using some specially designed operators.
• Several studies have created the tunes at random andthen used the fitness evaluation to sort out the goodtunes [8], [92], [96], [97], [108], [110], [136], [155].
Papadopoulos and Wiggins [111] constructed the tunes
given the chord progression. The pitches are assigned
randomly from the notes of the corresponding chords.
In the GA proposed by Towsey et al. [136], the tunes

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence5

are evaluated according to 21 musical features regarding
pitch, tonality, and contour. The pitch features are used
to fix the pitch range and measure the pitch diversity;
the tonality features keep the proportion of non-scales
and dissonant intervals; the contour features deal withthe tendency of melody, decrease large leaps, and handle
the return of large leaps. Marques et al. [96] manipulated
the tunes according to the intervals between consecutive
notes. Blackwell and Bentley [19], [20], [21] designed a
PSO approach to generate the tune, of which the pitch,volume, and the duration of notes are represented as aparticle. Geis and Middendorf [57] manipulated the ants
to crawl around the notes for composing the tune.
• Instead of random creation, the music theory about consonance, chord information, and music patterns are also
employed to generate the tunes. Biles [16], [17], [18] usedthe chord information and jazz theory to assign the notes
appropriate for impromptu. In the composition systems ofUnehara and Onisawa [109], [140], [141], [142], a tune is
generated from the database of music theory. Maeda and
Kajihara [92] utilized the twelve-tone technique to create
simple music pieces. The process begins with a tonerow created by twelve tones and then evolves depending
upon the harmonious degree. Liu and Ting [85] proposedcreating tunes for consonance hinged on the chords ineach measure. The T-Music composition system [89]handles the tunes considering the chord progression andcadence. The system follows the cadence principle toassign the ending note of phrase.
• Some studies have utilized the mathematical equations
or human assistance in the construction of tunes. Yoon[154] used fractals with musical seed data to generatemeaningful music segments. Phon-Amnuaisuk et al. [114]presented the four-voice harmony system, in which theuser inputs the soprano information as the tune, and theGA then harmonizes it by generating the alto, tenor, andbass parts. Reddin et al. [118] proposed a composition
system based on GE, which adopts six measurements, i.e.,pitch variety, dissonance, pitch contour, melodic direction
stability, note destiny, and pitch range, for the fitness
function to handle the tune. Loughran et al. [90] used
GE to create tonal music. The tunes are generated withreference to the grammar, such that most notes are in the
middle range of piano.
• The pitch range and scale are concerned in the generationof tunes. For example, Loughran et al. [90] restricted thepitch range to four octaves by grammar, while Donnelly
and Sheppard [37] limited the range to an interval of 13th
and the interval of consecutive notes to 9th. KaliakatsosPapakostas et al. [73] developed a hybrid system using
PSO to create the musical features of tune and rhythm,
which will be used in the evaluation criterion for GA.
Osana et al. [53], [76], [131] generated the tunes using thefeatures on pitch, rhythm, and chord progression obtained
from the sample melody. Wang et al. [147] proposed acomposing system for the Chinese Jiangnan ditty. They
first extracted the features, including pitch range, melodic
interval, mode, note duration, timbre, rhythm pattern, andintervallic motive, from the existing Jiangnan ditties, andsubsequently used the features for the fitness evaluation.
The tunes are handled through the Chinese pentatonic
mode.
Second, the tune can be constructed by combining musicalpatterns or pieces. Chiu and Shan [28] used pattern mining
to extract the features, including motif, chord, and structure,from the input music. The extracted motifs are selected tocompose the tune for a melody, where the selection depends
on the chord sequences to ensure the harmony of the melodic
progression. Liu and Ting [86] proposed using frequent patternmining to extract the musical patterns repeatedly used by acomposer. The tune is created by assembling these patterns.
The third way is to generate the series of notes sequentially.
The rationale behind this approach is that the current andthe previous notes provide the information for determining
an appropriate pitch or rhythm for the subsequent notes. The
neural networks and grammatical models are often used in thisapproach. Bharucha and Todd [13], [133] designed a neuralnetwork to predict the next note, given the current note as
input. The learning process of the neural network is performed
with culture-specific modes and chords. Mozer [102], [103]used the recurrent neural network (RNN) to predict notes forcreating the composition. The RNN was trained with a numberof Bach’s music pieces and traditional European folk songs.
Chen and Miikkulainen [26] also adopted neural networks toaddress the tune issue. They used a whole measure as the input
data to generate its subsequent measure. The generated tunes
are then examined for transposition. Chen et al. [27], [52] useda table defining the fitness of notes based on the music theory
to create the initial sequences of notes. Tomari et al. [135]employed the N-gram model to predict the next note in theevaluation of compositions. The N-gram model was trainedwith 30 Japanese children’s songs.
Fourth, the tune can be formed by imitating the existingmusic. Wu et al. [149] and Ting et al. [132] presented anovel composing system, which initializes the population withexisting songs. In particular, the system addresses the issue oftunes by imitation; that is, it attempts to change the notesin phrases without altering the melodic progression of theoriginal song. Tuohy and Potter [137] developed a system
to convert music pieces into the compositions appropriate forguitar playing. The conversion includes four considerations forthe tune: 1) The arranged note should be playable for a guitar
instead of a causal assignment; 2) the fitness function rewards
consecutive notes in a moving line; 3) the notes occurring inboth of the original and new compositions are preferred; and4) the chord notes are important but should not be abused.
Alfonseca et al. [8] proposed creating the tunes by the minimal
normalized compression distance. Vargas et al. [145] tried to
mimic the composition process of musicians to create musicpieces. They used a music database to initialize four measures
as a lick and constructed the tunes in accordance with thegiven chords and scales.
3) Harmony: Harmony is associated with two formationsof pitches: 1) The vertical one concerns multiple pitches,from the same or different tracks, sounding at the same time
(beat); and 2) the horizontal one is related to multiple pitches

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence6

sounding in tandem to form the chord progression. Harmony
is crucial to music composition. A proper arrangement ofpitches can achieve the consonance, thereby making musicdelightful and peaceful. In the composition systems using
evolutionary algorithms, harmony is ordinarily considered in
the fitness function. For the four-part chorus, the four parts canbe viewed and handled as four different instruments sounding
human voices. Phon-Amnuaisuk et al. [114] dealt with theharmony in the four-part chorus through several criteria to
avoid parallel unison, parallel perfect fifths, parallel octaves,crossing voices, and hidden fifths. Maddox [91] developed a
GA for the four-part chorus. For harmony, the fitness function
penalizes the parallel fifths, parallel unison and octaves, leap
greater than one octave, improper resolution of a large leap,and overlapping voices. Donnelly and Sheppard [37] examined
the harmony for generating the four-part chorus by GA. The
harmonic rules are defined in the fitness function to avoid theparallel octaves, parallel fifths, and dissonant intervals such as
seconds, fourths, and sevenths.
Chords serve as an important basis for forming harmony.
The chord progression constitutes the orientation of pitches
in a melody. Some studies have utilized this notion to tackle
the harmony in music composition. McIntyre [98] used GA togenerate a four-part Baroque chorus. The GA handles harmonyin the fitness function that considers several conditions in thechords, including chord correspondence, start and stop notesin chords, and resolutions for chords. Marques et al. [96]presented a fitness function favoring the concurrent notes thatcan conform to a chord, especially the major or minor chord,for forming the harmony. Chang and Jiau [25] addressed theharmony issue by adopting the suitable chords for a melody.
The harmony of notes across multiple tracks is anotherissue in music composition. Oliwa [108] presented a GA-based
composition system, in which the fitness function regulates
that the lead guitar should use the chords corresponding tothose used by the rhythm guitar and the piano for harmony.
Xu et al. [151] adopted the music theory, harmonics, as thebasis for fitness evaluation. The fitness function treats harmony
via mapping the mode to the chord progression. In addition,
each period (A to C) of generated music has its own rules tomaintain harmony. Liu and Ting [84] proposed using the music
theory rules derived from consonance to tackle the harmony
issue across the tracks of different instruments. The fitness
function rewards the consonance of the main, bass, and chordaccompaniments that are evolved through GA.
C. Rhythm
Rhythm endows the connection between notes as well as
an essential factor in another dimension, by contrast to thedimension of pitch, to be considered in composition. It assigns
the duration for each note (including the rest note) to makethe music vivid or form a specific genre. Establishing rhythms
is not easy because of the huge variety in allocating notes tothe beats.
Like handling the pitch, several evolutionary composition
approaches utilize the fitness evaluation to find the appropriate
rhythms. Maeda and Kajihara [93] included rhythm in theevaluation criteria of their GA for automatic composition[92]. The fitness values are determined by the duration, ratio,
and time interval. Moreover, they proposed another evaluation
criterion regarding the rhythm and the statistics on the timing
variation of sound. Wang et al. [147] presented a fitness
function that evaluates the rhythms using statistical results.
The frequent durations and rhythm patterns in the extracted
compositions are rewarded with a high fitness value.
Moreover, Towsey et al. [136] devised a GA that evaluates
the rhythms according to the rhythmic features, including thenote density, rest density, rhythmic variety, rhythmic range,and syncopation. Tomari et al. [135] used the N-gram modelfor the evaluation criterion. The N-gram model was learned
from 30 Japanese children’s songs. To deal with rhythm, themodel considers the transition of rhythm every two bars. Longet al. [89] developed the T-music system, which takes into
account the rhythm components, such as the length of the lastnote in a phrase, and the similarity of rhythm between two
phrases. Loughran et al. [90] used GE to establish the rhythms.
The system prefers short notes in the grammar because theauthors argued that the existence of many long notes will makethe music dull. In Oliwa’s rock music composition system[108], the fitness function also rewards short notes for a fast
tapping and playing of the lead guitar.
Rhythm is a major issue at percussion accompaniment.
Dostál [40] developed a human-like rhythmic system thatgenerates the drum’s rhythm for accompaniment. The fitness
function favors the coincidence of beats in the melody anddrum accompaniment. Yamamoto et al. [152] proposed aninteractive GA to generate the drum fill-in patterns, which
are represented by no-sound, snare drum, high-tam, low-tam,floor-tam, open-high-hat, and closed-high-hat.
Some studies use neural networks to model rhythms andassist the fitness evaluation. Gibson and Byrne [58] adopted aneural network as the fitness function for GA in composition.
The system first learns the rhythm and then responds withthe goodness of the input rhythm. Chen and Miikkulainen[26] proposed a neural network to generate melodies. The
system adjusts the duration of notes to form the rhythm.
Yuksel et al. [155] combined RNN and evolutionary algorithm
to manage the tune and rhythm simultaneously. The neuralnetwork is trained with a set of pitches and duration of notes.
Tokui and Iba [134] applied the interactive GA and GP to
create the rhythmic patterns. The back-propagation neuralnetwork is applied to model user’s evaluation, in order toreduce the number of interactive evaluations and lessen thefatigue caused.
Few studies propose directly using the rhythm of the original composition and focus the efforts on the assignment oftunes. For instance, Liu and Ting [86] mined the frequent
musical patterns consisting of both tune and rhythm. Music
composition turns out to be a task of recombining the patterns,
thereby arranging the rhythm as well. Chiu and Shan [28]also used music pattern mining for composition. The rhythm
assignment was omitted since the structure of measures in theirsystem was predetermined.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence7

D. Emotion
The emotion conveyed through music is also a key consideration in music composition. The emotion of a compositionis associated with timbre, rhythm, chord progression, scale,etc. For example, the bright timbre and fast tempo are usuallyused to express spirited, happy, and excited emotions.
Zhu et al. [157] designed an interactive GA to generatecompositions with a specified emotion. They adopted the KTH
rule system [51], which models the performance principles
used by musicians, for the evaluation rules. The weights of
the rules are determined by the response (happy or sad) ofusers. Onisaw et al. [109] also used an interactive evolutionary
system to compose a joyful or sorrowful melody to reflect theuser’s feeling. They concluded that a joyful melody is harderthan a sorrowful melody to evaluate because the former needs
to consider more elements, such as rhythm and tempo. Xu
et al. [151] introduced the harmony emotion rules to theircomposing system, which employs a major mode with fast
tempo for happiness and a minor mode with slow tempo forsadness.
IV. C OMPOSITION WITH ACCOMPANIMENT
Accompaniment is vital to reinforce the melody. Good
accompaniment can promote harmony, enhance the music
structure, and intensify the expressiveness of the compositions.
The accompaniment can be divided into three parts: main
accompaniment, chord accompaniment, and bass.
A. Main Accompaniment
The main accompaniment usually collocates with the
melody note by note to make up or intensify it. Acevedo[7] used the counterpoint method for a GA to build the main
accompaniment of the input melody. The fitness function considers different aspects of the accompaniment: The proportionof notes in the same key and the length of measures in thegenerated accompaniment should be similar with those in the
input melody. The beginning and ending notes need to beconsonant with the melody. The intervals and repetitions ofnotes are also limited. Liu and Ting [84] developed a GA
for making the polyphonic accompaniment, where the threeaccompaniments (main, bass, and chord) have their own fitness
functions. The main accompaniment was designed to enhance
the harmony and complement the insufficient rhythm in the
melody.
B. Chord Accompaniment
Chord accompaniment is one of the key factors in music composition. The music would become boring andmonotonous without the chords supporting harmony. The CI
techniques for handling the chord accompaniment can beclassified into two types: Some assume the chords are given
and compose the music accordingly; some produce the chord
progression while creating the composition.
For the first type, supposing the chord progression ispredetermined, the composition system will focus on the
search for the appropriate pitches, scales, rhythms, and soon. The GA developed by Liu and Ting [84] evaluates thefitness of compositions according to the chords given by theuser. The chords follow a predefined progression but will
be evolved into different forms by the GA. Furthermore,they [86] proposed generating compositions by following
the original chord progression. The chords are employed toevaluate the consonance with the generated melody. Xu etal. [151] presented a varying chord progression for harmony.
The selection of chords depends on the mode, intervals, endingparts, and three different periods. Chiu and Shan [28] used
the pattern mining technique to extract chord progressions
from the input music. These progressions are adopted as thebasis for evaluating the sequence of randomly selected chords,
where a high similarity in chord progression leads to a highfitness value.
For the second type, chords and melody are handled concurrently. Chang and Jiau [25] developed a system to automatically generate chords for the melody in two phases: The local
recognition phase determines the chord candidates according
to the common chord templates and rhythm. The global
decision phase further selects the most suitable chords fromthe candidates based on the chord progression rules. Nakamura
and Onisawa [106] designed an interactive GA system toconstruct the chord sequence. The chords are evolved withthe compositions by the GA with the listener’s feedback.

C. Bass Part
The bass part is an important element helping to stabilize
the progression of music. For composition, Liu and Ting [84]proposed evolving the bass line using GA. In the composition
system, the candidate bass lines are generated with reference
to the chords used. The fitness function evaluates the basslines according to their harmony with the melody and otheraccompaniments.
In addition, for the four-part music, the bass part is ordinarily composed of chord notes, especially the root of chord.
The relevant studies have been described in Section III.B.

V. D ESIGNS OF C OMPUTATIONAL I NTELLIGENCE
This section introduces the various designs of computational intelligence techniques for music composition. First,
we recapitulate the studies on the operators used in theevolutionary composition systems. Next, we review the neuralnetworks proposed for music composition. The research on
the application of fuzzy systems to music is also briefed.
Finally, the subsection reviews the methods for the evaluationof compositions.

A. Evolutionary Computation
Evolutionary algorithms have been widely used for automatic composition and achieved several favorable results. The
design of an evolutionary algorithm pertains to representation,
selection, crossover, mutation, and fitness function.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
81) Representation: The integer string is commonly usedas the representation for the elements of music composition,such as timbre, pitch, and rhythm. Unemi and Nakada [143],[144] presented a two dimensional array as the representation
for the guitar, bass and drum. The representation indicates theinformation of both the rhythm and the notes for the three
tracks. The proportion of rest and tenuto are determined withrespect to the representation form. Papadopoulos and Wiggins[111] proposed a representation resting on the degrees of scale
relative to the current chord for composing a jazz melody.
The representation is capable of handling the information
about the scales and responding chords rather than only theabsolute pitches. Vargas et al. [145] designed an improved
chromosome representation using a numerical sequence basedon the chords assigned to the measure. For example, the chord
Cmaj7 corresponds to a major scale (avoid 4th) according to
the musical theory. Therefore, the available notes will be C,
D, E, G, A, B, octave C, octave D, octave E, and so on, which
are indicated by the numbers 1 to 15 for representation.
Instead of a fixed length, Donnelly and Sheppard [37]proposed the variable length chromosome for representationof the four-part music composition. Each chromosome contains four parts for the musical line, namely, soprano, alto,
tenor, and bass. The variable length representation allows
the composition to grow and shrink caused by the insertionand deletion of the mutation operator, respectively. Oliwa[108] presented different representations for the tracks of
instruments, including the lead guitar, rhythm guitar, keyboard,and drums, considering the differences in their components
and playing techniques. For instance, the representation for thelead guitar preserves the space for the tapping technique, whilethat for the drum is based on the rhythm patterns. Biles [16]took account of the pitches and rhythms in the representation.
In addition, the notes of hold and rest are considered as events.
The chromosome uses the information about the chords andscales to ensure the represented notes are playable by humanplayers, which is critical for the impromptu interactive process.
2) Crossover: The well-known crossover operators, suchas k-point crossover and uniform crossover, are usually applicable to the evolutionary algorithms for music composition.
In addition, some studies have developed special crossover
operators that consider the musical characters or enhanced
performance.
Marques et al. [96] devised the note crossover and octave
crossover. The former regulates that the crossover should takeplace in the note with sound; namely, the rest and tenuto notes
are excluded. The latter keeps the tone of the chosen note but
exchanges in one octave. Vargas et al. [145] suggested that thecrossover should avoid disrupting compositions. The proposedcrossover operator produces valid compositions according to
the distance of intervals. Likewise, Liu and Ting [84], [85]proposed placing the cutting points between measures toprevent destroying the structure of music sections. Wu et al.
[149] designed three crossover operators considering different
restrictions: 1) The chord-based switch locates the cutting
points between measures with absolutely identical chords; 2)
the root-based switch picks the cutting points between themeasures having up to one different note and the same rootof chords; and 3) the analogous switch is similar to the rootbased switch but permits cutting points locating between themeasures with different roots of chords.
3) Mutation: Some studies have devised the mutation operator to improve the efficiency in composing music. Beyond
simply flipping genes, Biles [16] presented two mutation
operators with musical meaning. The first mutation operator
alters chromosomes in light of transposition, retrograde, rotation, inversion, sorting, and retrograde-inversion. The second
operator acts as a high level mutation processing musicalphrases. Marques et al. [96] proposed two mutation operatorson the notes and octave. The note mutation is performed on thenotes excluding the rest and tenuto notes. The octave mutation
shifts the pitch of the selected note by one octave. Matić
[97] designed three mutation operators to increase the fitnessof compositions. The first mutation operator shifts a tone
into its lower octave to reduce the number of large intervals.
The second operator changes the pitch if the current note isdissonant with its subsequent note. The third mutation simply
swaps two consecutive notes. Donnelly and Sheppard [37] presented a series of mutation rules: repeat, split, arpeggiate, leap,
upper neighbor, lower neighbor, anticipation, delay, passing
tone, deleting note, and merging note. When performed, themutation probabilistically selects one of the rules to change
the composition. Vargas et al. [145] developed three mutation
operators to ensure the consistency of phrases. These mutation
operators retain the pleasant sound without damaging theresults of crossover by transposing two down, octave, andhemiola operators. The transpose two down operator shifts anentire phrase down by two tones to introduce diversity to thephrase without destroying the consonance. The octave operator
decreases a note with a large interval by one octave to reduce
the horizontal interval. The hemiola operator repeats a groupof notes throughout a phrase.
B. Neural Networks
Neural networks are often used to evaluate compositions
or predict notes in music composition. In these applications,neural networks are trained to model the evaluation or arrangement of musical elements. Regarding the evaluation, neuralnetworks are commonly adopted to assist the fitness functionin the evolutionary composition systems. Biles et al. [18]adopted the neural networks as the fitness function for the
impromptu system GenJam. Tokui and Iba [134] employedneural networks to model the user’s preference and thereby
mitigate the fatigue issue at interactive composition. Burton
and Vladimirova [23] applied the adaptive resonance theory(ART) network for the fitness evaluation in GA. The ART
network is trained to model the clusters of drum patterns invarious genres of music, including rock, funk, disco, Latin andfusion.
As for the prediction of notes, Bharucha and Todd [13],[133] utilized neural networks to predict the next note. Afterthe learning process, the neural network is seeded with an
initial value and then generates a new composition note bynote. Similarly, Chen and Miikkulainen [26] adopted a neuralnetwork to generate the next measure, given the current

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence9

ones. The neural network takes both the tune and rhythm
into account. Mozer [102], [103] trained an RNN with themusic pieces of Bach and traditional European folk songs
to generate compositions. In addition, he proposed learning
the AABA phrase patterns in order to maintain the musicalstructure. Eck and Schmidhuber [43], [44] suggested using thelong short-term memory (LSTM) to improve the composition
process of RNN. Considering the issue that the compositions
made by RNN usually lack the global structure, the LSTM
is adopted to learn the musical chord structure. The authors
believed the well-trained chord structure can form a cycle
for a global musical structure. Liu et al. [87] also proposedusing RNN to generate music. Specifically, they adopted theresilient propagation (RProp) in place of the common backpropagation. The LSTM was adopted to solve the issue atthe long phrases. Franklin [48] compared several types of
RNN for music composition and presented a composition
system using the LSTM and music representation. Coca et
al. [29], [30] applied the chaotic algorithm [31] to generatechaotic melodies for inspiration. The proposed RNN system
has two input units: melodic and chaotic units. The former isdevised for learning, whereas the latter is used for inspiring
the composition process.
C. Fuzzy Systems
The notions of fuzzy sets and fuzzy logic are widely usedin the classification of music genre [12], [46], [115], recognition of music emotions [50], [72], [153], recommendation
systems [81], [113], music retrieval [88], [129], [156], copyright protection [77] and authentication [82]. Although theseapproaches focus on processing audio signals and metadata
rather than create music compositions, they can be used todeal with the emotion and music genre in the compositions. In
addition, fuzzy sets and logic can further assist the evaluationof compositions owing to the intrinsic fuzzy nature of themusic genre, emotion, and evaluation.
D. Evaluation of Compositions
The CI techniques ordinarily require feedback to adapt themodel or guide the evolution. Evaluation of the existing or
generated compositions caters for the major feedback and has
a vital influence on the learning and evolutionary process. As
the above sections described, various designs for the fitness
function are presented to deal with the pitch, harmony, rhythm,
and accompaniment in music composition.
The evaluation methods used in CI for music composition
can be classified into three categories: interaction-based, rulebased, and learning-based. The following recapitulates thestudies on these three types of music evaluation.
1) Interaction-based Evaluation: The interaction-based
evaluation relies on the listener’s feedback on the generated
music. The methods collect the listener’s responses, e.g.,
evaluation scores, preferences, or physiological signals such asheartbeat, pulse, and skin conductance, to a sample phrase orcomposition; and then use the collected information to evaluate
the sample. The interaction-based evaluation methods provide
a direct and useful way to evaluate the generated compositions;

therefore, they are commonly used in CI/AI-based composition
systems.
Based on the interactive evaluation, several studies proposed
the interactive evolutionary algorithms (IEAs) for music composition [130]. In the IEAs, the fitness of a composition isdetermined as per the listener’s evaluation result. Johanson andPoli [71] presented an interactive GP to arrange the pitches ofnotes, while Tokui and Iba [134] devised another interactive
GP to evolve the rhythm of compositions. Yoon [154] proposed
a composition system based on fractals and GA, which relieson the evaluation results from the listeners to evolve the generated compositions. Nakamura and Onisawa [105] developed
a lyrics/music composition system that considers the user’s
impression and music genre. Given the user’s choice about
genre, the system creates the lyrics using the Markov chain
and a lyrics database, while the music is composed of the
melody and one of the three accompaniments for the genre.
The combination of the music and lyrics is then evaluated by
the user.
Instead of an entire composition, some evaluation approaches require the listeners to take note of and evaluate
only certain parts of the composition to lighten their loading.
The GenJam system [14] uses the real-time human evaluation on the segments of generated jazz solos. Jacob [67]developed an IEA to evolve the weights, phrase lengths, andtransposition table for a new melody, given the motif andthe chord progression. Like GenJam, the IEA requires theaudience to evaluate a partial composition to reduce the loading in evaluation. Onisaw et al. [109] designed an interactive
composition system to generate the emotional melody. The
listeners evaluate only four bars as to whether or not the
melody transfers the correct emotion. Unehara and Onisawa[140] adopted the interaction with humans for choosing theirfavorite 16-bar music. Fukumoto and Ogawa [54] proposed aninteractive differential evolution (IDE) for creating the eightnote sign sounds. The evaluation of the sounds generated
by the DE is based on comparison: Each time the listeners
choose their preferred one from two sounds, and from foursounds at different evolutionary phases in the first and second
experiments, respectively. The authors further compared theperformances of the IDE and IGA in music composition [55],[56]. Chen et al. [27], [52] utilized the user’s feedback, inthe form of preference scores, to evaluate the music phrases
generated by the proposed evolutionary algorithm. The fitnessof a music block is defined by the average score of all themusic phrases containing this block.
Another form of interaction-based evaluation is through thecollaboration with musicians for professional feedback. The
GenJam [16], [17] can do trade fours at a jazz impromptu.
It listens to the human player performing four bars and thenconverts the music into the chromosome structure to evolve.
The resultant chromosome soon responds to accomplish thetrade fours. Diaz-Jerez [35] developed the composition system
Melomics, which makes use of the professional composers’scored results. Manaris [94] designed an interactive music
generator using GA, Markov model, and power laws. The
music generator creates music in response to the humanplayer’s performance.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
10

The results of interaction-based evaluation provide a genuine feedback on the compositions. However, a crucial issue
lies in the fatigue caused by the repeated listening, which
gradually runs out of human evaluator’s patience and decreases
their music sensitivity. Hence, the number of evaluations basedon interaction is usually very limited in order to maintain thequality and consistence of the evaluation results.
2) Rule-based Evaluation: The rule-based evaluation
adopts explicit rules for evaluation of generated compositions.
The evaluation rules are developed according to personal
experience or music theory in composition, considering themusical elements such as rhythm, phrase, scale, and chord.
The rule-based evaluation renders the objective measures, bycontrast to the personal and subjective measures of interactionbased evaluation. Moreover, the rule-based evaluation saves
the high cost at interaction and benefits the efficiency of thefitness evaluation in evolutionary music composition.
Horner and Goldberg [62] presented a GA that creates
compositions by iteratively producing the segments bridging
music parts, where the fitness evaluation depends upon thepredetermined static patterns. Phon-Amnuaisuk et al. [114]adopted some evaluation rules from music theory considering
leaps and intervals. Özcan and Erçal [110] used the evaluation
rules based on notes, intervals, and pitch contour, in their GA
for generating improvisations. Oliwa [108] proposed a GA
using a set of fitness functions tied to different instruments
in rock music. For example, the fitness function for the notesof a lead guitar is associated with the ascending/descending
tone scale, tapping, and flatten. Matić [97] presented a fitness
function that compares the mean and variance of intervals,
and the proportion of scale notes in a measure between
the generated melodies and the input melodies. Acampora
et al. [6] adopted a set of evaluation rules concerning the
progression of notes in creating the four-part harmony. Morespecifically, the rules are involved with the modulation andtonality of the subsequent chords. The composition system ofGeis and Middendorf [57] evaluates the melody and the fourpart accompaniments separately. The evaluation rules for the
melody consider the smoothness, contour, resolution, and endof-tonic, whereas that for the accompaniments considers thechord, voice distance and leading, progression, smoothness,and resolution. Towsey et al. [136] defined the 21 pitch,
tonality, and contour features hinged on the music theory
and used them in the evaluation rules. Freitas and Guimaraes[49] used a bi-objective evaluation method to address thedissonance and simplicity of harmony. The fitness functions
are based on six rules considering the triads, dissonant chords,
invalid pitches and chords, unison, and tomic position. Liu
and Ting [84] proposed evaluating the compositions according
to the music theory. They adopted a set of rules associatedwith chord notes, leap, harmony, and rhythm in the fitness
evaluation. This method was further applied to an automaticcomposition system based on phrase imitation [132].
The rule-based evaluation methods are useful for creating
the compositions of a specific music genre. Some studies
leverage the music theory or the well-formed music structure
for a genre, e.g., Baroque and jazz, to construct the evaluation
rules. McIntyre [98] took account of the four-part Baroqueharmony in the fitness evaluation. Given the chords and thefour-part Baroque style, the GA arranges the notes to the fourmusic tracks whilst considering the harmony and stability inthe progression. Tzimeas and Mangina [138] constructed a
GA to transform the Baroque music to jazz music. The fitness
is determined by a critically-damped-oscillator function forvarying the evolutionary direction in accordance with the genre
and rhythm [139]. In addition, they used multiple objectives
to deal with the rhythm, where the weights of the objectives
are defined by the similarity to the target rhythm. Wang etal. [147] focused on the Chinese Jiangnan ditty. The fitness
function in the proposed composition system uses the features extracted from several Jiangnan ditties, including range,
melodic interval, mode, note duration, timbre, rhythm pattern,
and intervallic motive. Oliwa [108] designed polyphony composition depending on the features of rock music, such as
the tendency of scale, duration of notes, and coordination ofmusical instruments.
3) Learning-based Evaluation: The learning-based evaluation constructs the models from music data for the evaluationof compositions. Machine learning techniques, such as neuralnetworks, are widely used for the evaluation model. Furthermore, they are applied to analyze, cluster, or classify the music
data, and the results are employed as the basis for evaluation.
Gibson and Byrne [58] utilized neural networks to construct
the model for the fitness evaluation on the combinations ofrhythms. Spector and Alpern [127] used neural networks with
Fahlman’s quickprop algorithm to characterize and evaluate
music in the GP-based composition system. Dannenberg et al.
[32] applied the linear and Bayesian classifiers on music data
and also utilized neural networks to extract the characteristics
of music for creating compositions. Burton and Vladimirova[23] designed a GA for the arrangement of percussion, in
which the fitness function evaluates compositions according to
the results of a clustering algorithm on the music data. Similarly, Manaris et al. [95] developed a GP method using a fitness
function based on the classification results (classical, popular,
and unpopular music) of neural networks. Osana et al. [53],[76], [131], [135] trained N-gram models with 30 Japanesechildren’s songs for evaluating the compositions. They further
improved the N-gram model by additional evaluation rules.
Yuksel et al. [155] established an RNN as the model forevaluating compositions. The RNN was trained using a seriesof human compositions. During training, the neural network
takes the input notes sequentially and the output is the pitchand duration of the next note.
The learning-based evaluation methods can be incorporated
with the interaction-based or rule-based evaluation methods.
Ramirez et al. [117] used a classification approach and a
GA to learn the rule-based expressive performance model.
Compared to interaction-based evaluation, the learning-based
evaluation methods leverage the extant compositions and theirevaluations, instead of using real-time feedback from humans.
Hence, the learning-based evaluation methods can be used
to reduce the loading and fatigue issue at interaction-based
evaluation. Biles et al. [15], [16], [18] used neural networks in
GenJam to learn the feedback to the generated impromptu Jazz
music. The neural networks help to moderate the distraction

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence11

and unstable evaluation caused by the fatigue from long-time
listening.
VI. F UTURE D IRECTIONS AND T OPICS
In this section, we present a number of directions and topics
for future research.
A. Representation
Most of the existing studies have adopted simple representations for the pitches and duration. These representations,nonetheless, cannot support the diverse use of musical symbols, such as triplet, vibrato, and ornament. The design of new
representation for supporting comprehensive usage of musicalsymbols will benefit the composing of music. In addition, a
representation that can exclude inappropriate settings for theelements (e.g., pitches, chords, and beats) is desirable.
B. Evaluation
As aforementioned, evaluation of the generated compositions is imperative for the CI-based composition approaches.
However, there exist some weaknesses in the extant designsof evaluation methods. The interaction-based evaluation canreflect the user’s genuine preference among the generated
compositions, but its practicability seriously detracts fromthe fatigue and decreased sensitivity caused by the long-time
listening. The rule-based evaluation provides a fast way toevaluate the compositions; however, the determination of therules and their weights raises another problem to be addressed.
The learning-based evaluation can build the evaluation modelfrom the music data but encounters the issues existing in
the machine learning methods, such as training time, model
construction, and fidelity.
To design an evaluation approach that can address theabove issues is a crucial research direction for the CI-based
composition systems. A promising way is to hybrid thethree evaluation methods to remedy their own weakness. For
example, learning-based approaches can be used to build thesurrogate for the interaction-based evaluation to reduce thetime and human loading at interaction [69], [70], [83]. The
evaluation rules in the rule-based approaches can serve as
the basis of an evaluation model, while the learning-based
methods are capable of learning the weights from personal
experiences or music data.
C. Deep Learning
Deep learning has thrived and gained many exciting
achievements over recent years [80], [124]. For example, thefamous music streaming service company, Spotify, adopts thedeep learning technique to analyze the user’s preferences forbetter service and experience. In music composition, Huang
and Wu [66] presented a work on the generation of music
through deep learning. They used a multi-layer LSTM andcharacter-level language model to learn the musical structure. In addition, Google has recently released their Magenta
project [5], which uses the RNN on the TensorFlow platformto generate piano melody. Although the current results are
preliminary, to explore the great potential of deep learning inmusic composition is a significant topic for future research.

VII. C ONCLUSIONS
Computational intelligence plays a pivotal role in algorithmic composition. This paper presents a survey on the researchof CI in music composition. In this survey, we review anddiscuss the CI techniques for creating music in the lightof musical elements and algorithmic design. The first part
reviews the existing approaches for dealing with the threeprincipal elements in music composition, i.e., musical form,
melody, and accompaniment. The second part recapitulates
the various designs of evolutionary algorithms and neuralnetworks for creating compositions. In addition, we categorize
the evaluation methods into interaction-based, rule-based, andlearning-based evaluation, and discuss the existing approaches
for these three types of evaluation manners.
The above reviews and discussions show that the CI techniques are highly capable and promising for algorithmic
composition. This survey also reflects the importance of evaluation, representation, and learning of compositions in musicalcreativity. Enhancement and advanced designs on these aspects
are suggested as the significant directions for future researchon CI in music composition.
ACKNOWLEDGMENT
The authors would like to thank the editor and reviewers
for their valuable comments and suggestions, and Chung-Ying
Huang, Chia-Lin Wu, and Chun-Hao Hsu for their commentson the manuscript.
R EFERENCES[1] [Online]. Available: http://melomics.com/[2] [Online]. Available: http://www.apple.com/mac/garageband/[3] [Online]. Available: http://chordbot.com/[4] [Online]. Available: http://tonepadapp.com/[5] [Online]. Available: http://magenta.tensorflow.org/[6] G. Acampora, J. M. Cadenas, R. De Prisco, V. Loia, E. Munoz, andR. Zaccagnino, “A hybrid computational intelligence approach for automatic music composition,” in Proceedings of the IEEE International
Conference on Fuzzy Systems, 2011, pp. 202–209.
[7] A. G. Acevedo, “Fugue composition with counterpoint melody generation using genetic algorithms,” in Proceedings of the International
Symposium on Computer Music Modeling and Retrieval, 2004, pp. 96–
106.
[8] M. Alfonseca, M. Cebrián, and A. Ortega, “A simple genetic algorithm
for music generation by means of algorithmic information theory,” inProceedings of the IEEE Congress on Evolutionary Computation, 2007,
pp. 3035–3042.
[9] A. Alpern, “Techniques for algorithmic composition of music,” Hampshire College, Tech. Rep., 1995.
[10] C. Ames, “The Markov process as a compositional model: A survey
and tutorial,” Leonardo Music Journal, vol. 22, no. 2, pp. 175–187,
1989.
[11] T. Anders and E. R. Miranda, “Constraint programming systems formodeling music theories and composition,” ACM Computing Surveys,
vol. 43, no. 4, p. 30, 2011.
[12] C. Baccigalupo, E. Plaza, and J. Donaldson, “Uncovering affinity ofartists to multiple genres from social behaviour data,” in Proceedings ofthe International Society for Music Information Retrieval Conference,
2008, pp. 275–280.
[13] J. J. Bharucha and P. M. Todd, “Modeling the perception of tonal
structure with neural nets,” Computer Music Journal, vol. 13, no. 4,
pp. 44–53, 1989.
[14] J. A. Biles, “GenJam: A genetic algorithm for generating jazz solos,”in Proceedings of the International Computer Music Conference, 1994.
[15] ——, “Life with GenJam: Interacting with a musical IGA,” in Proceedings of the IEEE International Conference on Systems, Man, andCybernetic, 1999, pp. 652–656.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence12[16] ——, “GenJam: Evolution of a jazz improviser,” Creative evolutionary
systems, vol. 168, p. 2, 2002.
[17] ——, “Straight-ahead jazz with GenJam: A quick demonstration,” inProceedings of the MUME Workshop, 2013.
[18] J. A. Biles, P. Anderson, and L. Loggi, “Neural network fitness
functions for a musical IGA,” in Proceedings of the Symposium ofIntelligent Industrial Automation and Soft Computing, 1996, pp. B39–
B44.
[19] T. Blackwell, “Swarm music: Improvised music with multi-swarms,”
in Proceedings of the AISB Symposium on Artificial Intelligence andCreativity in Arts and Science, 2003, pp. 41–49.
[20] ——, Swarming and music. Springer, 2007, vol. 1.
[21] T. Blackwell and P. Bentley, “Improvised music with swarms,” inProceedings of IEEE Congress on Evolutionary Computation, 2002,
pp. 1462–1467.
[22] D. Burraston and E. Edmonds, “Cellular automata in generative electronic music and sonic art: A historical and technical review,” Digital
Creativity, vol. 16, no. 3, pp. 165–185, 2005.
[23] A. R. Burton and T. Vladimirova, “Genetic algorithm utilising neuralnetwork fitness evaluation for musical composition,” in Proceedingsof the International Conference on Artificial Neural Nets and Genetic
Algorithms, 1998, pp. 219–223.
[24] A. Calvo and J. Seitzer, “MELEC: Meta-level evolutionary composer,”
Journal of Systemics, Cybernetics and Informatics, vol. 9, no. 1, pp.
45–50, 2011.
[25] C.-W. Chang and H. C. Jiau, “An improved music representation
method by using harmonic-based chord decision algorithm,” in Proceedings of the IEEE International Conference on Multimedia andExpo, 2004, pp. 615–618.
[26] C.-C. Chen and R. Miikkulainen, “Creating melodies with evolving
recurrent neural networks,” in Proceedings of the International Joint
Conference on Neural Networks, 2001, pp. 2241–2246.
[27] Y.-p. Chen, “Interactive music composition with the CFE framework,”
ACM SIGEVOlution, vol. 2, no. 1, pp. 9–16, 2007.
[28] S.-C. Chiu and M.-K. Shan, “Computer music composition based on
discovered music patterns,” in Proceedings of the IEEE International
Conference on Systems, Man and Cybernetics, 2006, pp. 4401–4406.
[29] A. E. Coca, D. C. Corrêa, and L. Zhao, “Computer-aided music
composition with LSTM neural network and chaotic inspiration,” inProceedings of the International Joint Conference on Neural Networks,
2013, pp. 1–7.
[30] A. E. Coca, R. A. F. Romero, and L. Zhao, “Generation of composedmusical structures through recurrent neural networks based on chaotic
inspiration,” in Proceedings of the International Joint Conference on
Neural Networks, 2011, pp. 3220–3226.
[31] A. E. Coca, G. O. Tost, and L. Zhao, “Characterizing chaotic melodies
in automatic music composition,” Chaos, vol. 20, no. 3, p. 033125,
2010.
[32] R. Dannenberg, B. Thom, and D. Watson, “A machine learning
approach to musical style recognition,” in Proceedings of the International Computer Music Conference, 1997, pp. 344–347.
[33] R. L. De Mantaras and J. L. Arcos, “AI and music: From composition
to expressive performance,” AI magazine, vol. 23, no. 3, pp. 43–57,
2002.
[34] G. Diaz-Jerez, “Algorithmic music: Using mathematical models inmusic composition,” Ph.D. dissertation, The Manhattan School ofMusic, 2000.
[35] ——, “Composing with melomics: Delving into the computational
world for musical inspiration,” Leonardo Music Journal, vol. 21, pp.
13–14, 2011.
[36] T. Dimitrios and M. Eleni, “A dynamic GA-based rhythm generator,”
in Trends in Intelligent Systems and Computer Engineering. Springer,
2008, pp. 57–73.
[37] P. Donnelly and J. Sheppard, “Evolving four-part harmony using
genetic algorithms,” in Proceedings of the European Conference on
the Applications of Evolutionary Computation, 2011, pp. 273–282.
[38] M. Dorigo, V. Maniezzo, and A. Colorni, “Ant system: Optimization
by a colony of cooperating agents,” IEEE Transactions on Systems,
Man, and Cybernetics, Part B: Cybernetics, vol. 26, no. 1, pp. 29–41,
1996.
[39] M. Dorigo and L. M. Gambardella, “Ant colony system: A cooperative
learning approach to the traveling salesman problem,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 53–66, 1997.
[40] M. Dostál, “Genetic algorithms as a model of musical creativity—on
generating of a human-like rhythmic accompaniment,” Computing andInformatics, vol. 24, no. 3, pp. 321–340, 2012.

[41] S. Dubnov, G. Assayag, O. Lartillot, and G. Bejerano, “Using machinelearning methods for musical style modeling,” Computer, vol. 36,
no. 10, pp. 73–80, 2003.
[42] K. Ebcioğlu, “An expert system for harmonizing four-part chorales,”Computer Music Journal, vol. 12, no. 3, pp. 43–51, 1988.
[43] D. Eck and J. Schmidhuber, “Finding temporal structure in music:
Blues improvisation with LSTM recurrent networks,” in Proceedings ofthe IEEE Workshop on Neural Networks for Signal Processing, 2002,
pp. 747–756.
[44] ——, “A first look at music composition using LSTM recurrent neuralnetworks,” Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale,
Tech. Rep. 103, 2002.
[45] M. Edwards, “Algorithmic composition: Computational thinking inmusic,” Communications of the ACM, vol. 54, no. 7, pp. 58–67, 2011.
[46] F. Fernández and F. Chávez, “Fuzzy rule based system ensemble
for music genre classification,” in Proceedings of the International
Conference on Evolutionary and Biologically Inspired Music and Art,
2012, pp. 84–95.
[47] J. D. Fernández and F. Vico, “AI methods in algorithmic composition:
A comprehensive survey,” Journal of Artificial Intelligence Research,
vol. 48, pp. 513–582, 2013.
[48] J. A. Franklin, “Recurrent neural networks for music computation,”
INFORMS Journal on Computing, vol. 18, no. 3, pp. 321–338, 2006.
[49] A. R. R. Freitas and F. G. Guimaraes, “Melody harmonization in evolutionary music using multiobjective genetic algorithms,” in Proceedingsof the Sound and Music Computing Conference, 2011.
[50] A. Friberg, “A fuzzy analyzer of emotional expression in music
performance and body motion,” in Proceedings of Music and Music
Science, 2004.
[51] A. Friberg, R. Bresin, and J. Sundberg, “Overview of the KTH rulesystem for musical performance,” Advances in Cognitive Psychology,
vol. 2, no. 3, pp. 145–161, 2006.
[52] T.-Y. Fu, T.-Y. Wu, C.-T. Chen, K.-C. Wu, and Y.-p. Chen, “Evolutionary interactive music composition,” in Proceedings of the Annual
Conference on Genetic and Evolutionary Computation, 2006, pp.
1863–1864.
[53] K. Fujii, M. Takeshita, and Y. Osana, “Automatic composition systemusing genetic algorithm and n-gram model-influence of n in n-grammodel,” in Proceedings of the IEEE International Conference on
Systems, Man, and Cybernetics, 2011, pp. 1388–1393.
[54] M. Fukumoto and S. Ogawa, “A proposal for optimization of sign
sound using interactive differential evolution,” in Proceedings of theIEEE International Conference on Systems, Man, and Cybernetics,
2011, pp. 1370–1375.
[55] M. Fukumoto, R. Yamamoto, and S. Ogawa, “The efficiency ofinteractive differential evolution in creation of sound contents: In
comparison with interactive genetic algorithm,” in Proceedings of the
ACIS International Conference on Software Engineering, Artificial
Intelligence, Networking and Parallel & Distributed Computing, 2012,
pp. 25–30.
[56] ——, “The efficiency of interactive differential evolution in creationof sound contents: In comparison with interactive genetic algorithm,”
International Journal of Software Innovation, vol. 1, no. 2, pp. 16–27,
2013.
[57] M. Geis and M. Middendorf, “An ant colony optimizer for melody
creation with baroque harmony,” in Proceedings of the IEEE Congress
on Evolutionary Computation, 2007, pp. 461–468.
[58] P. M. Gibson and J. Byrne, “Neurogen, musical composition using
genetic algorithms and cooperating neural networks,” in Proceedingsof the International Conference on Artificial Neural Networks, 1991,
pp. 309–313.
[59] D. E. Goldberg, Genetic Algorithms in Search, Optimization andMachine Learning. Addison Wesley, 1989.
[60] L. Hiller and L. Isaacson, Experimental Music: Composition With an
Electronic Computer. McGraw-Hill, 1959.
[61] J. Holland, Adaptation in Natural and Artificial Systems. Universityof Michigan Press, 1975.
[62] A. Horner and D. E. Goldberg, “Genetic algorithms and computerassisted music composition,” in Proceedings of the International Conference on Genetic Algorithms, 1991, pp. 337–441.
[63] H. S. Howe, “Composing by computer,” Computers and the Humanities, vol. 9, no. 6, pp. 281–290, 1975.
[64] K. J. Hsü and A. J. Hsü, “Fractal geometry of music,” Proceedings of
the National Academy of Sciences, vol. 87, no. 3, pp. 938–941, 1990.
[65] K. J. Hsü and A. Hsü, “Self-similarity of the 1/f noise called music,”
Proceedings of the National Academy of Sciences, vol. 88, no. 8, pp.
3507–3509, 1991.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
13[66] A. Huang and R. Wu, “Deep learning for music,” arXiv:1606.04930,
Tech. Rep., 2016.
[67] B. Jacob, “Composing with genetic algorithms,” in Proceedings of theInternational Computer Music Conference, 1995.
[68] H. Järveläinen, “Algorithmic musical composition,” in Proceedings of
the Seminar on Content Creation Art, 2000.
[69] Y. Jin, “A comprehensive survey of fitness approximation in evolutionary computation,” Soft Computing, vol. 9, no. 1, pp. 3–12, 2005.
[70] ——, “Surrogate-assisted evolutionary computation: Recent advancesand future challenges,” Swarm and Evolutionary Computation, vol. 1,
pp. 61–70, 2011.
[71] B. Johanson and R. Poli, “GP-music: An interactive genetic programming system for music generation with automated fitness raters,” inProceedings of the Annual Conference on Genetic Programming, 1998,
pp. 181–186.
[72] S. Jun, S. Rho, B.-j. Han, and E. Hwang, “A fuzzy inference-based
music emotion recognition system,” in Proceedings of the International
Conference on Visual Information Engineering, 2008, pp. 673–677.
[73] M. A. Kaliakatsos-Papakostas, A. Floros, and M. N. Vrahatis, “Interactive music composition driven by feature evolution,” SpringerPlus,vol. 5, no. 1, pp. 1–38, 2016.
[74] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of IEEE International Conference on Neural Networks, 1995,
pp. 1942–1948.
[75] J. Kennedy, “Particle swarm optimization,” in Encyclopedia of Machine
Learning. Springer, 2010, pp. 760–766.
[76] M. Kikuchi and Y. Osana, “Automatic melody generation considering
chord progression by genetic algorithm,” in Proceedings of the World
Congress on Nature and Biologically Inspired Computing, 2014, pp.
190–195.
[77] K.-H. Kim, M. Lim, and J.-H. Kim, “Music copyright protection systemusing fuzzy similarity measure for music phoneme segmentation,” inProceedings of the IEEE International Conference on Fuzzy Systems,
2009, pp. 159–164.
[78] A. Kirke and E. R. Miranda, “A survey of computer systems forexpressive music performance,” ACM Computing Surveys, vol. 42,
no. 3, pp. 1–41, 2009.
[79] J. R. Koza, Genetic programming: On the programming of computers
by means of natural selection. MIT press, 1992.
[80] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol.
521, no. 7553, pp. 436–444, 2015.
[81] M. Lesaffre, M. Leman, and J.-P. Martens, “A user-oriented approach tomusic information retrieval,” in Dagstuhl Seminar Proceedings, 2006.
[82] W. Li, X. Zhang, and Z. Wang, “Music content authentication basedon beat segmentation and fuzzy classification,” EURASIP Journal on
Audio, Speech, and Music Processing, vol. 1, no. 11, pp. 1–13, 2013.
[83] D. Lim, Y. Jin, Y. S. Ong, and B. Sendhoff, “Generalizing surrogateassisted evolutionary computation,” IEEE Transactions on Evolutionary
Computation, vol. 14, no. 3, pp. 329–355, 2010.
[84] C.-H. Liu and C.-K. Ting, “Polyphonic accompaniment using genetic
algorithm with music theory,” in Proceedings of the IEEE Congress on
Evolutionary Computation, 2012.
[85] ——, “Evolutionary composition using music theory and charts,” inProceedings of the IEEE Symposium on Computational Intelligence
for Creativity and Affective Computing, 2013, pp. 63–70.
[86] ——, “Music pattern mining for chromosome representation in evolutionary composition,” in Proceedings of the IEEE Congress on
Evolutionary Computation, 2015, pp. 2145–2152.
[87] I. T. Liu and B. Ramakrishnan, “Bach in 2014: Music composition
with recurrent neural network,” arXiv:1412.3191, Tech. Rep., 2014.
[88] M. Liu, C. Wan, and L. Wang, “Content-based audio classification
and retrieval using a fuzzy logic system: Towards multimedia searchengines,” Soft Computing, vol. 6, no. 5, pp. 357–364, 2002.
[89] C. Long, R. C.-W. Wong, and R. K. W. Sze, “T-music: A melody
composer based on frequent pattern mining,” in Proceedings of theIEEE International Conference on Data Engineering, 2013, pp. 1332–
1335.
[90] R. Loughran, J. McDermott, and M. O’Neill, “Tonality driven piano
compositions with grammatical evolution,” in Proceedings of the IEEE
Congress on Evolutionary Computation, 2015, pp. 2168–2175.
[91] T. Maddox and J. Otten, “Using an evolutionary algorithm to generate
four-part 18th century harmony,” in Proceedings of the International
Conference on Mathematics and Computers in Business and Economics, 2000.
[92] Y. Maeda and Y. Kajihara, “Automatic generation method of twelve
tone row for musical composition used genetic algorithm,” in Proceed-

ings of the IEEE International Conference on Fuzzy Systems, 2009, pp.
963–968.
[93] ——, “Rhythm generation method for automatic musical composition
using genetic algorithm,” in Proceedings of the IEEE International
Conference on Fuzzy Systems, 2010, pp. 1–7.
[94] B. Manaris, D. Hughes, and Y. Vassilandonakis, “Monterey mirror:
Combining Markov models, genetic algorithms, and power laws,” inProceedings of the IEEE Congress on Evolutionary Computation, 2011.
[95] B. Manaris, P. Roos, P. Machado, D. Krehbiel, L. Pellicoro, andJ. Romero, “A corpus-based hybrid approach to music analysis andcomposition,” in Proceedings of the National Conference on Artificial
Intelligence, 2007, p. 839.
[96] M. Marques, V. Oliveira, S. Vieira, and A. C. Rosa, “Music composition using genetic evolutionary algorithms,” in Proceedings of the IEEE
Congress on Evolutionary Computation, 2000, pp. 714–719.
[97] D. Matić, “A genetic algorithm for composing music,” Yugoslav Journal of Operations Research, vol. 20, no. 1, pp. 157–177, 2010.
[98] R. A. McIntyre, “Bach in a box: The evolution of four part baroqueharmony using the genetic algorithm,” in Proceedings of the IEEE
Conference on Evolutionary Computation, 1994, pp. 852–857.
[99] M. Miller, The Complete Idiot’s Guide to Music Composition. Alpha,2005.
[100] E. R. Miranda, “Cellular automata music: An interdisciplinary project,”Journal of New Music Research, vol. 22, no. 1, pp. 3–21, 1993.
[101] E. R. Miranda and J. A. Biles, Evolutionary Computer Music.
Springer, 2007.
[102] M. C. Mozer, “Neural network music composition by prediction:
Exploring the benefits of psychoacoustic constraints and multi-scale
processing,” Connection Science, vol. 6, no. 3, pp. 247–280, 1994.
[103] ——, “Neural network music composition by prediction: Exploring
the benefits of psychoacoustic constraints and multiscale processing,”
Musical Networks: Parallel Distributed Perception and Performance,
vol. 227, pp. 227–260, 1999.
[104] E. Muñoz, J. M. Cadenas, Y. S. Ong, and G. Acampora, “Memetic
music composition,” IEEE Transactions on Evolutionary Computation,
vol. 20, no. 1, pp. 1–15, 2016.
[105] C. Nakamura and T. Onisawa, “Music/lyrics composition system considering user’s image and music genre,” in Proceedings of the IEEE
International Conference on Systems, Man and Cybernetics, 2009, pp.
1764–1769.
[106] S. Nakashima, Y. Imamura, S. Ogawa, and M. Fukumoto, “Generationof appropriate user chord development based on interactive genetic
algorithm,” in Proceedings of the International Conference on P2P,
Parallel, Grid, Cloud and Internet Computing, 2010, pp. 450–453.
[107] G. Nierhaus, Algorithmic Composition: Paradigms of Automated Music
Generation. Springer, 2009.
[108] T. Oliwa, “Genetic algorithms and the abc music notation language forrock music composition,” in Proceedings of the Annual Conference on
Genetic and Evolutionary Computation, 2008, pp. 1603–1610.
[109] T. Onisawa, W. Takizawa, and W. Unehara, “Composition of melody
reflecting user’s feeling,” in Proceedings of the Annual Conference ofIEEE Industrial Electronics Society, 2000, pp. 1620–1625.
[110] E. Özcan and T. Erçal, “A genetic algorithm for generating improvised
music,” in Artificial Evolution, 2008, pp. 266–277.
[111] G. Papadopoulos and G. Wiggins, “A genetic algorithm for the generation of jazz melodies,” in Proceedings of the Finnish Conference on
Artificial Intelligence, vol. 98, 1998.
[112] ——, “AI methods for algorithmic composition: A survey, a criticalview and future prospects,” in Proceedings of the AISB Symposium onMusical Creativity, 1999, pp. 110–117.
[113] H.-S. Park, J.-O. Yoo, and S.-B. Cho, “A context-aware music recommendation system using fuzzy bayesian networks with utility theory,”in Proceedings of the International Conference on Fuzzy Systems andKnowledge Discovery, 2006, pp. 970–979.
[114] S. Phon-Amnuaisuk, A. Tuson, and G. Wiggins, “Evolving musicalharmonisation,” in Proceedings of the International Conference on
Artificial Neural Nets and Genetic Algorithms, 1999, pp. 229–234.
[115] S. Poria, A. Gelbukh, A. Hussain, S. Bandyopadhyay, and N. Howard,“Music genre classification: A semi-supervised approach,” in Proceedings of the Mexican Conference on Pattern Recognition, 2013, pp. 254–
263.
[116] A. Prechtl, R. Laney, A. Willis, and R. Samuels, “Algorithmic music
as intelligent game music,” in Proceedings of the AISB Anniversary
Convention, 2014.
[117] R. Ramirez, A. Hazan, E. Maestre, and X. Serra, “A genetic rule-basedmodel of expressive performance for jazz saxophone,” Computer Music
Journal, vol. 32, no. 1, pp. 38–50, 2008.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TETCI.2016.2642200, IEEE
Transactions on Emerging Topics in Computational Intelligence
14[118] J. Reddin, J. McDermott, and M. O’Neill, “Elevated pitch: Automatedgrammatical evolution of short compositions,” in Proceedings of theWorkshops on Applications of Evolutionary Computation, 2009, pp.
579–584.
[119] K. Ricanek, A. Homaifar, and G. Lebby, “Genetic algorithm composesmusic,” in Proceedings of the Southeastern Symposium on System
Theory, 1993, pp. 223–227.
[120] C. Roads and P. Wieneke, “Grammars as representations for music,”
Computer Music Journal, vol. 3, no. 1, pp. 48–55, 1979.
[121] J. J. Romero and P. Machado, The Art of Artificial Evolution: A
Handbook on Evolutionary Art and Music. Springer, 2007.
[122] H. A. G. Salas, A. Gelbukh, and H. Calvo, “Music composition basedon linguistic approach,” in Proceedings of the Mexican International
Conference on Artificial Intelligence, 2010, pp. 117–128.
[123] H. A. G. Salas, C. H. Gelbukh, Alexander, and F. Galindo Soria,“Automatic music composition with simple probabilistic generativegrammars,” Polibits, vol. 44, no. 44, pp. 59–65, 2011.
[124] J. Schmidhuber, “Deep learning in neural networks: An overview,”
Neural Networks, vol. 61, pp. 85–117, 2015.
[125] A. Schoenberg, G. Strang, and L. Stein, Fundamentals of MusicalComposition. Faber & Faber, 1967.
[126] S. Sertan and P. Chordia, “Modeling melodic improvisation in Turkish
folk music using variable-length Markov models,” in Proceedings ofthe International Society for Music Information Retrieval Conference,
2011, pp. 269–274.
[127] L. Spector and A. Alpern, “Induction and recapitulation of deep
musical structure,” in Proceedings of International Joint Conferenceon Artificial Intelligence, 1995, pp. 20–25.
[128] M. J. Steedman, “A generative grammar for jazz chord sequences,”
Music Perception, vol. 2, no. 1, pp. 52–77, 1984.
[129] J.-H. Su, C.-Y. Wang, T.-W. Chiu, J. J.-C. Ying, and V. S. Tseng,“Semantic content-based music retrieval using audio and fuzzy-musicsense features,” in Proceedings of the IEEE International Conferenceon Granular Computing, 2014, pp. 259–264.
[130] H. Takagi, “Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation,” Proceedings of theIEEE, vol. 89, no. 9, pp. 1275–1296, 2001.
[131] M. Takano and Y. Osana, “Automatic composition system using genetic
algorithm and n-gram model considering melody blocks,” in Proceedings of the IEEE Congress on Evolutionary Computation, 2012, pp.
1–8.
[132] C.-K. Ting, C.-L. Wu, and C.-H. Liu, “A novel automatic composition
system using evolutionary algorithm and phrase imitation,” IEEE
Systems Journal, 2015.
[133] P. M. Todd, “A connectionist approach to algorithmic composition,”
Computer Music Journal, vol. 13, no. 4, pp. 27–43, 1989.
[134] N. Tokui and H. Iba, “Music composition with interactive evolutionary computation,” in Proceedings of the International Conference on
Generative Art, 2000, pp. 215–226.
[135] M. Tomari, M. Sato, and Y. Osana, “Automatic composition basedon genetic algorithm and n-gram model,” in Proceedings of the IEEE
Conference on System, Man and Cybernetics, 2008.
[136] M. Towsey, A. Brown, S. Wright, and J. Diederich, “Towards melodic
extension using genetic algorithms,” Educational Technology & Society, vol. 4, pp. 54–65, 2001.
[137] D. R. Tuohy and W. D. Potter, “GA-based music arranging for guitar,”in Proceedings of the IEEE Congress on Evolutionary Computation,
2006, pp. 1065–1070.
[138] D. Tzimeas and E. Mangina, “Jazz Sebastian Bach: A GA system
for music style modification,” in Proceedings of the International
Conference on Systems and Networks Communications, 2006, pp. 36–
36.
[139] ——, “Dynamic techniques for genetic algorithm based music systems,” Computer Music Journal, vol. 33, no. 3, pp. 45–60, 2009.
[140] M. Unehara and T. Onisawa, “Composition of music using humanevaluation,” in Proceedings of the IEEE International Conference on
Fuzzy Systems, 2001, pp. 1203–1206.
[141] ——, “Music composition system with human evaluation as human
centered system,” Soft Computing, vol. 7, no. 3, pp. 167–178, 2003.
[142] ——, “Music composition by interaction between human and computer,” New Generation Computing, vol. 23, no. 2, pp. 181–191, 2005.
[143] T. Unemi and E. Nakada, “A tool for composing short music pieces
by means of breeding,” in Proceedings of the IEEE International
Conference on Systems, Man, and Cybernetics, 2001, pp. 3458–3463.
[144] T. Unemi and M. Senda, “A new musical tool for composition andplay based on simulated breeding,” in Proceedings of Second Iteration,
2001, pp. 100–109.

[145] F. V. Vargas, J. A. Fuster, and C. B. Castanón, “Artificial musicalpattern generation with genetic algorithms,” in Proceedings of the Latin
America Congress on Computational Intelligence, 2015, pp. 1–5.
[146] R. F. Voss and J. Clarke, “1/f noise in music and speech,” Nature, vol.
258, pp. 317–318, 1975.
[147] X. Wang, Y. Zhan, and Y. Wang, “Study on the composition rules for
Chinese Jiangnan ditty,” in Proceedings of the International Conferenceon Information Science and Technology, 2015, pp. 492–497.
[148] G. Wiggins, G. Papadopoulos, S. Phon-Amnuaisuk, and A. Tuson,“Evolutionary methods for musical composition,” International Journalof Computing Anticipatory Systems, 1999.
[149] C.-L. Wu, C.-H. Liu, and C.-K. Ting, “A novel genetic algorithm considering measures and phrases for generating melody,” in Proceedingsof the IEEE Congress on Evolutionary Computation, 2014, pp. 2101–
2107.
[150] I. Xenakis, Formalized Music: Thought and Mathematics in Composition. Pendragon Press, 1992.
[151] B. Xu, S. F. Wang, and X. Li, “An emotional harmony generation
system,” in Proceedings of the IEEE Congress on Evolutionary Computation, 2010, pp. 1–7.
[152] R. Yamamoto, S. Nakashima, S. Ogawa, and M. Fukumoto, “Proposal
for automated creation of drum’s fill-in pattern using interactive genetic
algorithm,” in Proceedings of the International Conference onBiometrics and Kansei Engineering, 2011, pp. 238–241.
[153] Y.-H. Yang, C.-C. Liu, and H. H. Chen, “Music emotion classification:
A fuzzy approach,” in Proceedings of the ACM International Conference on Multimedia, 2006, pp. 81–84.
[154] J. Yoon, “An interactive evolutionary system IFGAM,” in Proceedingsof the Annual Conference of IEEE Industrial Electronics Society, 2004,
pp. 3166–3171.
[155] A. C. Yuksel, M. M. Karcı, and A. S. Uyar, “Automatic music
generation using evolutionary algorithms and neural networks,” in Proceedings of the International Symposium on Innovations in Intelligent
Systems and Applications, 2011, pp. 354–358.
[156] B. Zhang, J. Shen, Q. Xiang, and Y. Wang, “Compositemap: A novel
framework for music similarity measure,” in Proceedings of the ACM
SIGIR International Conference on Research and Development inInformation Retrieval, 2009, pp. 403–410.
[157] H. Zhu, S. Wang, and Z. Wang, “Emotional music generation using
interactive genetic algorithm,” in Proceedings of the International
Conference on Computer Science and Software Engineering, 2008, pp.
345–348.

Chien-Hung Liu (S’12) received the B.S. degree incomputer science and information engineering from
National Chung Cheng University, Chiayi, Taiwan,
in 2011, where he is currently working toward the
Ph.D. degree at the Department of Computer Scienceand Information Engineering.
His research interests include evolutionary computation, memetic algorithm, computer composition,and creative intelligence.

Chuan-Kang Ting (S’01–M’06–SM’13) received
the B.S. degree from National Chiao Tung University, Hsinchu, Taiwan, in 1994, the M.S. degree from
National Tsing Hua University, Hsinchu, Taiwan, in
1996, and the Dr. rer. nat. degree from the Universityof Paderborn, Paderborn, Germany, in 2005.
He is currently an Associate Professor at Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi,
Taiwan. His research interests include evolutionary
computation, computational intelligence, machinelearning, and their applications in music, art, networks, and bioinformatics.
Dr. Ting is an Associate Editor of the IEEE C OMPUTATIONAL I NTELLI GENCE M AGAZINE and IEEE T RANSACTIONS ON E MERGING T OPICS INC OMPUTATIONAL I NTELLIGENCE, and an editorial board member of Soft
Computing and Memetic Computing journals.

2471-285X (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.