CONNECTIVE FUSION:LEARNING TRANSFORMATIONAL JOINING OF SEQUENCES
WITH APPLICATION TO MELODY CREATION

pitch

Taketo Akama
Sony Computer Science Laboratories, Tokyo, Japan
taketo.akama@sony.com
8075

pitch pitch

1c T. Akama. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: T. Akama,“Connective Fusion: Learning Transformational Joining of Sequenceswith Application to Melody Creation”, in Proc. of the 21st Int. Society
for Music Information Retrieval Conf., Montréal, Canada, 2020.

46

2

time (bars)

3

4

3

0

4

1

0

1

2

time (bars)

2

time (bars)

3

4

3

4

pitch pitch

2

time (bars)

3

4

0

1

2

time (bars)

3

4

4

pitch
pitch

pitch
pitch

pitch
pitch

1

time (bars)(b) Transformed outputs of iteration 1,2, and 3

Figure 1: Iterated application of transformation.
81

81

2

76

3

66
61
56

1

0

1

4

81
time
(bars)
76
2 71
3

4

0

time
(bars)
3
4
66

2

1

5

6

71

81

3 4
71

0

7

66

8

61

2

3

56

4

time5 (bars) 6

time61(bars)

61

pitch

1

71

pitch

pitch

76

0

1

2

3

4

7 time (bars)
8

0

1

2

1

2

3

4

3

4

time (bars)(a)56 Given inputs: random
combinationof two sequences
76
56
1

2

3

4

pitch

time (bars)

0

1

2

pitch

0

time (bars)
61

76

56

71

2

3

4

0

1

2

3

4

time (bars)

81

66

76

pitch

56
81

71
66
61

76

pitch

1

time (bars)

66

81

61

56

71

81

66

76

pitch

61
56
81

71
66
61

76

pitch

Spurred by the progress of deep neural networks, symbolic
music generation systems, especially the ones with user
controllability have gathered renewed interests these days.
Most of the systems with controllability can be categorized
into generating continuation [1], regenerating arbitrary positions [2, 3], unsupervised conditional generation [4, 7],
or transforming musical sequences such that musical attributes, concepts, or styles are altered [4–8].
In this paper, we seek another category, generation byfusing input musical sequences as ideas. Specifically, we
propose Connective Fusion, a generative transformation
scheme which allows two input musical sequences to betransformed such that the concatenated musical sequenceis musically plausible (See Fig.1 for the illustration). Afterthe transformation, each of the two sequences has meaningful similarity with the one before the transformation.
Two input musical sequences can also be transformed it-

2time (bars)

time (bars)

pitch

1. INTRODUCTION

1

1
2random
3 combination
4
0
1 two 2sequences
3(a) Given inputs:
of

pitch

We present Connective Fusion, a music generation schemeby transformational joining of two musical sequences forcreative purposes. Given two shorter sequences as inputs,
our model transforms each of them such that their concatenation is more coherent to form a longer sequence, while
each of the transformed shorter sequences retains meaningful similarity with the corresponding input sequence. In
short, our model connects and fuses two contextually unrelated sequences in a coherent way. This transformation canbe applied iteratively to gradually fuse the input sequences.
The style latent space is simultaneously learned, allowing
users to control how the two sequences are merged. Our
approach comprises two steps of unsupervised learning: a
deep generative model with a latent space is learned, followed by adversarial learning of the transformation function in the latent space. We demonstrate the usefulness of
our method through the task of melody creation using asymbolic music dataset.

0
80
75
70
65
80
60
75
55
70
65 0
60
55
80 0
75
70
65
80
60
75
55
70 0
65
60
55
80
80
75
75
70
70
65
65
60
60
55
55
80
80
75
75
70
70
65
65
60
55
60
80
55
75
80
70
75
65
70
60
65
55
60
80
55
75
80
70
75
65
70
60
65
55
60
80
55
75
0
70
81
65
76
60
55
71
0
66

pitch
pitch

ABSTRACT

70
65
60
55

56

71

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

time (bars)

66

time (bars)

8

0

56

0

1

2

3

4

5

6

time (bars)

7

8time (bars)

time (bars)

5

6

time (bars)

618(b) Transformed outputs

Figure 2: Transformation with style space exploration.

eratively to gradually increase the coherency of joining(Fig.1). Our generative transformation differs from puretransformation in that it permits users to explore or sample
from the style space of how the two sequences are combined (Fig.2). The application of Connective Fusion includes i) providing users with musical ideas based on not

78

Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020(a) Pre-training(b) Training(c) Transformation

Figure 3: Connective Fusion schematic diagrams.
(i)necessarily polished ideas on hand, and ii) creating novel
musical flows by combining and fusing musical fragmentsof different characteristics in a coherent way.
Technically, our approach is training a transformation
function in an unsupervised learning manner, consisting of
two steps: the pre-training step and the training step. In the
pre-training step, a Variational Auto-Encoder (VAE) [9] istrained to obtain mappings between the data space and thelatent space. A given pair of musical sequences are fed into
the encoder of the VAE to be mapped to the corresponding
latent vectors in the latent space. In the following training step, models are trained upon the representation of the
VAE latent space. The transformation function, the generator is trained adversarially [10] such that transformed
results are indistinguishable from the human-made musical sequences. Together with the adversarial loss, the similarity loss, consisting of the latent space distance before
and after the transformation, is simultaneously taken into
account in order to train models with desired amount oftransformation.
In experiments, we empirically demonstrate that our
method outperforms a baseline method with various parameter settings in terms of reality and five musical statistics at each transformation amount. We also quantitatively
show that iterated application of our transformation allows
gradual transformation while having comparable to or better performance than single (non-iterative) transformation.
Our contributions of this paper are: i) proposing a new
problem setting/task and its solution, ii) presenting a modeland procedure for learning style space, iii) introducing iterated application of transformation for gradual transformation, and iv) demonstrating the performance and the application for melody creation.

and xR are musical sequences of 4 bars.
Using the dataset D, the scheme of Connective Fusion
basically solves the following task: given two sequences
xL , xR ∈ X , transforming xL to x0L and xR to x0R such thatthe concatenated sequence x0L ⊕ x0R becomes more probable to be a sample in Y than xL ⊕xR , while the transformed
sequences x0L , x0R retain meaningful similarity to the given
sequences xL , xR , respectively. For example, this serves asa solution for generating longer musical sequences given
shorter musical sequence pairs of any combinations as inspiration, which is useful for composing new music basedon unpolished musical ideas on hand.
2.2 Approach
Schematic diagrams of pre-training, training, and transformation of Connective Fusion are depicted in Fig.3.
2.2.1 Pre-training
Our approach is first training a Variational Auto-Encoder(VAE) model for obtaining bidirectional mappings between each sequence x ∈ X and its latent vector z ∈ Z ⊂Rdz , which has the compressed information of x.
2.2.2 Training
Then we train a generator G that transforms any pair0
) ∈ Z × Z that are in(zL , zR ) ∈ Z × Z to (zL0 , zR
distinguishable from the pairs in the dataset. The generator also takes a style vector s ∈ S from the stylespace S = [0, 1]ds as an additional input. Formally, the0generator can be written as (zL0 , zR
) = G(zL , zR , s) andG : Z × Z × S → Z × Z. The generator G is trained together with the discriminator D : Z × Z → [0, 1] with theadversarial learning framework [10]. In addition to the adversarial loss, we add the similarity loss in order to adjust
the degree of similarity before and after the transformation.

2. METHODOLOGY
2.1 Problem Scenario
Suppose we have a dataset of sequences Dy = {y (i) =
(i)
(i)
(i)
xL ⊕ xR }N∈ Y is the concatenation of two
i=1 , where y(i)(i)
subsequences xL ∈ X and xR ∈ X . We can consider(i)(i)
Dy as a sequence pair dataset D = {(xL , xR )}N
i=1 . For(i)instance, y (i) is a musical sequence of 8 bars, whereas xL

2.2.3 Transformation
Given two sequences xL and xR , these sequences are first
fed to the encoder of the VAE to obtain zL and zR , respectively. The latent vectors zL and zR together with a stylevector s ∈ S sampled from S = [0, 1]ds are then fed to the47Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020
0generator G to transform zL , zR to zL0 , zR
, respectively.
00
Finally, the transformed latent vectors zL , zR
are fed to thedecoder to generate x0L , x0R , respectively. The concatenation x0L ⊕ x0R is the generated sequence. Instead, zL and zR
can also be sampled from the prior distribution p(z), providing users with scratch generation functionality followedby the transformation functionality.
The style space S allows us to control how the two sequences are connectively fused. For example, as illustrated
in Fig.2 and explained in the experiments Sec.3.3, users
can choose preferred styles of fusion among interpolated
styles on a 2D plane.
We also propose iterated transformation with G, which
consists in applying G repeatedly, drawing two interrelated trajectories in the latent space. The higher the number of iterations becomes, the farther the latent vectors tend
to move from the original position. This is useful for users
to control the degree of similarity in transformation.

2.3 Advantage of Approach Using Latent Space
Using the representation in Z but not the one in X i) is useful for defining similarity before and after the transformation, ii) is computationally inexpensive, and iii) bypasses
the difficult problem of back-propagating through discrete
sampling of sequences.
2.4 Pre-training Detail: Encoder and Decoder
Training with Auto-Encoding VB algorithm
To train the VAE, another dataset D0 = {x(j) }2N
j=1 is(i)(i)

derived from the dataset D = {(xL , xR )}N
i=1 , where(i)(i)
x(2i−1) = xL and x(2i) = xR . The encoder modelqθ (x|z) and the decoder model pθ (z|x) in the VAE aretrained with the following optimization problem:
max E 0E[log pθ (x|z)] − KL (qθ (z|x)||p(z)) ,θx∼Dz∼qθ (z|x)(1)
which is the maximization of the variational lower bound[9], the lower bound of the marginal log likelihood. Here,KL denotes the Kullback-Leibler (KL) divergence. Notethat the sampling operation z ∼ qθ (z|x) is differentiable
using the reparametrization trick. For the purpose of visualization in Fig.3, we name the two terms in the optimiza-tion problem Lrec = −Ex∼D0 Ez∼qθ (z|x) [log pθ (x|z)]
and Lpri = Ex∼D0 [KL (qθ (z|x)||p(z))]. For the rest of
this paper, θ̂ denotes the estimated model parameter after
the optimization of Eq.1. We use normal distribution for
the encoder model qθ (z|x).

(i)argmaxz qθ̂ (z|xR ), and z (j) = argmaxz qθ̂ (z|x(j) ). The
generator G and the discriminator D are parametrized byψ and φ, respectively. In the following, we use short
hand LpDφ (zL , zR ) = − log Dφ (zL , zR ), LnDφ (zL , zR ) =−(1 − log Dφ (zL , zR )), and U = U(0, 1)ds .
2.5.1 Discriminator Loss
In the adversarial learning framework, the discriminatorclassifies two sets of samples: the real class and the fakeclass. In Connective Fusion, samples in the real class arelatent vector pairs (zL , zR ) sampled from the dataset Dz .
Formally, the loss function for the real class of the discriminator ishi
LpDφ (zL , zR ) .
(2)
Lpdis =
E
(zL ,zR )∼Dz

The fake class for the discriminator are latent vector pairs(zL , zR ) which are obtained by i) sampling independently
with the uniform distribution over the dataset Dz0 (1st termof Eq.3), ii) sampling independently from the prior p(z)(2nd term of Eq.3), iii) the generator G transforming samples (zL , zR , s), where zL , zR are sampled in the same way
as i, while s are sampled from U, the uniform distribution
over S = [0, 1]ds (3rd term of Eq.3), and iv) the generator G transforming samples (zL , zR , s), where zL , zR aresampled in the same way as ii, while s are sampled from U(4th term of Eq.3). Formally, the loss function ishi
Lndis = E 0 E 0 LnDφ (zL , zR )zL ∼Dz zR ∼Dz
h
i
+ E
E
LnDφ (zL , zR )zL ∼p(z)zR ∼p(z)hi+ E 0 E 0 E LnDφ (Gψ (zL , zR , s))zL ∼Dz zR ∼Dz s∼U
h
i
+ E
E
E LnDφ (Gψ (zL , zR , s)) . (3)zL ∼p(z)zR ∼p(z)s∼U

Finally, the overall loss function for the discriminator becomes
Ldis = Lpdis + Lndis .
(4)
2.5.2 Generator Loss
Based on the discriminator, a generator is trained such thatthe generated samples become more like the real samples
rather than the fake samples. We consider two kinds ofgenerated samples which are sampled in the same way asiii and iv in the previous section 2.5.1. Formally, the loss
function ishi
Lpgen = E 0 E 0 E LpDφ (Gψ (zL , zR , s))zL ∼Dz zR ∼Dz s∼U
h
i
+ E
E
E LpDφ (Gψ (zL , zR , s)) . (5)zL ∼p(z)zR ∼p(z)s∼U

2.5 Training Detail: Generator and Discriminator
Training with Adversarial Learning
For notational simplicity, we introduce datasets of latent(i) (i)0(j) 2N
vectors Dz = {(zL , zR )}N}j=1 correi=1 and Dz = {z
(i)
(i)
0
(j) 2N
sponding to D = {(xL , xR )}N
i=1 and D = {x }j=1
(i)
(i)
(i)
respectively, where zL = argmaxz qθ̂ (z|xL ), zR =48

Additionally, we introduce the similarity loss which is thelatent space distance between samples before and after thetransformation:
Lsim =
+

E

E

E

E

E [Ldist (zL , zR , s)]zL ∼Dz0 zR ∼Dz0 s∼U

E [Ldist (zL , zR , s)] ,zL ∼p(z)zR ∼p(z)s∼U(6)

Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020

Figure 4: Evaluation of our model on six metrics vs z-distance. The upper/lower rows are transformation results where
inputs are randomly created pairs of test/generated (sampled from prior) data. The color gradient corresponds to the numberof iterations for LP. For all metrics, higher values are better. See Sec.3.5.
eraged P
over all the training dataset. Precisely, σ̄z =1/|D0 | x∈D0 σ(x). Finally, the overall loss function for
the generator becomes
Lgen = Lpgen + λLsim .

(8)

3. EXPERIMENTS
3.1 Dataset
Figure 5: Musical statistics are corrected after our transformation. Brighter color indicates higher probability. See
the second paragraph of 3.4.

We create datasets from LMD-matched of Lakh MIDI
dataset [11], comprising 45,129 files matched to the songidentity entries in the Million Song Dataset [12]. Eachsong has one or several different versions of MIDI files.
We first extract files with 4/4 time signature, use accompanying tempo information to determine beat locations,and quantize each beat into 4. We then split the songidentities into the proportion of 11:1:6:1:1 to create train1, validation-1, train-2, validation-2, and test dataset, respectively. Train-1 and validation-1 datasets are for training proposed and baseline models, whereas train-2 andvalidation-2 datasets are for training evaluation models.
We filter out non-monophonic tracks, bass or drum tracks,
and the tracks outside the pitch range of [55, 84]. We conduct data augmentation by transposing tracks to all possible keys if the transposed tracks stay within the pitch
range of [55, 84]. We retrieve 8-bar sliding windows(with a stride of 1 bar) from each track followed by filtering out windows that have more than one bar consecutive rests. Finally, for each split of train-1, validation-1,
train-2, validation-2, and test dataset, we create a dataset(i)(i)
D = {(xL , xR )}N
i=1 by assigning the first 4-bars and the(i)(i)
last 4-bars of each 8-bar window to xL and xR , respectively. For encoding musical sequences, we use Melodicorhythmic encoding proposed in [3].

Figure 6: Analysis of z-distance in the latent space Z. See
the last paragraph of 3.4.
where


1 1
2
log 1 + (zL0 − zL )2dz σ̄z
1


1 1
2
0
+
log 1 + (zR − zR )dz σ̄z2
1

Ldist (zL , zR , s) =0with (zL0 , zR
) = Gψ (zL , zR , s).

(7)

Following the latent constraint paper [6], σ̄z ∈ Rdz ischosen to be the standard deviation σ(x) ∈ Rdz of theencoder model qθ̂ (z|x) = N µ(x), diag σ 2 (x) av49

Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020

Figure 7: Evaluation of iterated application of our model. Iterated application of models with larger λ tends to compare favorably with others. The upper/lower rows are transformation results where inputs are randomly created pairs oftest/generated (sampled from prior) data. The color gradient corresponds to the number of iterations for our method. For
all metrics, higher values are better. See Sec.3.6.
3.2 Implementation Details
2-layered long short term memory (LSTM) [13] is used forthe encoder and the decoder of the VAE, with the numberof latent dimensionality dz = 64. 5-layered multilayer perceptron (MLP) is used for the generator and the discriminator, where input vectors are simply concatenated and ReLU
activation is employed. Following the latent constraint paper [6], we use the gating mechanism for each of the out0
. For the decoder to generate sequences,puts zL0 and zR
argmax operation is utilized at each time step greedily forsampling each token. In the adversarial learning, the parameter updates of the model are alternating between updating the discriminator 10 times and updating the generator once. Adam optimizer [14] is used for training with theparameters (α, β1 , β2 ) = (0.000005, 0, 0.9).
3.3 Transformation Examples
Fig.1 and Fig.2 are the transformation examples for λ =2.0 and λ = 0.3, respectively. In Fig.1b at iteration 1, theused pitch set of xL becomes aligned to that of xR , and thenote at time around 4 transforms to the one with longer duration to join two input sequences. As the number of iterations increases, the note durations become more uniform,
while retaining similarity to the input sequences in termsof pitch contours or rhythmic properties. In Fig.2b, four
corners are generated by feeding s randomly sampled from{0, 1} for each dimension and the others are interpolations.
Interestingly, the time 4 to 8 of the bottom left sequence isthe exact transposition of the right input of Fig.2a, whereas
other sequences has variations and transpositions.
3.4 Evaluation Metrics
We use reality, correlations of five musical statistics evaluated against the test data, and z-distance as evaluation50

metrics. Reality is defined as the probability that a generated sequence is classified as human-made. Two-layeredtransformer binary classification model is trained on train2 and validation-2 datasets, where the pair sequences fromthe dataset D are labeled as positive, while the randomly
shuffled pair sequences from the same dataset are labeled
as negative. The mean of output values of this classification model and its accuracy are 0.959 and 0.953 on the test
dataset. Reality is a holistic metric of quality which quantifies how likely a sequence is human-made.
As musical statistics, we choose to use key, mean-pitch(MP), mean-duration (MD) for evaluating whether the first
and the last 4-bars are more musically consistent after
transformation. We also employ pitch-transition (PT) andduration-transition (DT) for evaluating if transitions between the first and the last 4-bars are smooth and natural.
Fig.5 illustrates these five musical statistics. For key, MP,
and MD, these values are estimated for the first and thelast 4-bars, which are interpreted as Markov transitions ofmusical states, creating matrices as in Fig.5. For PT andDT, we extract two or three consecutive notes around theboundary of the first and the last 4-bars, and again compute Markov transition matrices of each statistics as states.
Finally, as a quantitative metric, we compute Pearson correlation coefficient between the matrix made from the test
dataset and the one from the samples outputted by models.
z-distance is a scaled squared Mahalanobis distancedist(z 0 , z) = 1/dz (z 0 − z)Σ−1 (z 0 − z)| , where Σ =diag(σ̄z2 ). We analyze z-distance in the latent space Z using song id annotations of the dataset. We randomly sampled 1,000,000 pairs of 4-bar sequences from the entire
dataset, where pairs are sampled either from the same ordifferent song id, corresponding to intra- and inter-song in
Fig.6. Pairs with identical sequences are filtered out to analyze the similarity rather than the identicality. Fig.6 showsProceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020

quality is as satisfactory as single (non-iterative) transformation with models of smaller λ.

that sequences from different song ids are several orders ofmagnitude less likely to have smaller (say, less than 1.0 or3.0) z-distances, compared with those from the same ids.
The difference of z-distances probably comes from the factthat a lot of variations can be seen within each song id andsimilar themes are rarely seen among different song ids
especially for melody, suggesting that smaller z-distances
includes variations and that z-distance encodes musicallymeaningful similarity.

4. RELATED WORK
Concatenative Synthesis (CS) methods [15, 16] typicallyuse a large database of source audio segmented into units,
and search units that match each segment of the target audio by unit selection algorithms. Mashup methods [17, 18]
combine two or more songs to create entertaining musical results. Typically tracks from different songs are superimposed and combined. Our Connective Fusion differsfrom CS and Mashup in that i) ours is generative rather
than searching and using existing segments of music, ii)
ours tries to combine any segments, whereas CS/Mashup
typically combine some searched segments, iii) ours is unsupervised learning which does not need any human annotation in the dataset or musical expertise for training models, and iv) ours is symbolic which is often not the case in
CS/Mashup.
Learning transformation in the latent space has beenstudied for image, texts, and music [6, 19–23]. Notably,
Mueller et al. proposed to transform texts to have specified
attributes based on optimization in the latent space [21].
Engel et al. proposed to transform image or music to become more realistic or to have specified attributes in thelatent space by using adversarial learning [6]. Lore et al.
and more recently Jahanian et al. instead proposed to transform image by learning latent space directions [22, 23].
Learning transformation with adversarial learning without paired data has been extensively studied. The input
and output domain of transformation in these studies canbe categorized into the same data domain with different
classes [24] or attributes [6,25], the semantic domain to thedata domain [24], and semantically similar domain suchas simulation to real [26, 27] and unsupervised languagetranslation [28, 29]. On the other hand, the input and output domains of transformation in this paper are essentiallythe same except that they have different lengths/counts.

Algorithm 1 Latent Perturb (baseline method).
Input: zL , zR , and σLP
1: Smax ← argmaxx pθ̂ (x|zL ) ⊕ argmaxx pθ̂ (x|zR )2: Rmax ← estimate reality of Smax
3: for i = 1 to N umIter do
24:
L , R ∼ N (0, diag(σLP))
5:SL ← argmaxx pθ̂ (x|zL + L )6:
SR ← argmaxx pθ̂ (x|zR + R )7:
Rtmp ← estimate reality of Stmp = SL ⊕ SR
8:
if Rtmp > Rmax then9:
Smax ← Stmp ; Rmax ← Rtmp
10:
zL ← zL + L ; zR ← zR + R
11:
end if
12: end for
13: return Smax

3.5 Comparisons of Methods
We introduce a naive but strong baseline method called
Latent Perturb (LP), which is summarized in Algorithm
1. Note that the reality estimation model is trained withthe train-1 dataset, and the argmax operation is approximated with the greedy sampling scheme. Compared todata space, perturbation in the latent space can efficiently
search similar samples with high reality.
Fig.4 shows the comparisons of baseline methods andours evaluated on z-distance vs the other evaluation metrics, where z-distance is the average of the mean of zdistances for zL and zR . The reason for choosing LP with
σLP in [0.01, 0.2] is to make sure their resulting evaluation
metrics span the range that the proposed technique “ours”
spans. For samples from the prior as well as from the test
dataset, our methods, especially with larger λ tend to outperform in all the metrics, even though LP methods use
abundant computational budgets—100 forward computations of the decoder model and the classification model.

5. CONCLUSION
We introduced Connective Fusion, a new scheme of interactively generating sequences for creative purposes. Given
two sequences of random combination, our model transforms each of them to similar one such that their concatenation is more indistinguishable from human-madesequences. The transformation model is adversariallylearned in the latent space. Our model equips with user
control—choosing the amount of transformation as well as
exploring in the style space. In experiments of melody creation on a symbolic music dataset, we empirically demonstrated that our method outperforms a baseline method ofvarious parameter settings, and that iterative gradual transformation not just introduced new functionality but also
works as satisfactory as or sometimes better than noniterative transformation.

3.6 Evaluation of Iterated Transformation
Fig.7 illustrates the performance of iterated application of
our transformation. Interestingly, performance of models
with larger λ after several iterations are often comparable to or better than that of models with smaller λ. Thismeans that a model with larger λ can be used for transformation with different distances, without sacrificing the performance. For example, as illustrated in Fig.1, the modelwith λ = 2.0, if iteratively applied, can be used for gradual transformations with several different distances. Their
51

Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020

6. REFERENCES[1] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon,
C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman,
M. Dinculescu, and D. Eck, “Music transformer: Generating music with long-term structure,” in International Conference on Learning Representations, 2019.
[2] G. Hadjeres, F. Pachet, and F. Nielsen, “DeepBach:a steerable model for Bach chorales generation,” inProc. of the 34th International Conference on Machine
Learning, 2017.
[3] G. Hadjeres and F. Nielsen, “Anticipation-rnn: enforcing unary constraints in sequence generation,with application to interactive music generation,”
Neural Computing and Applications, vol. 32,
no. 4, pp. 995–1005, 2020. [Online]. Available:
https://doi.org/10.1007/s00521-018-3868-4[4] J. Gillick, A. Roberts, J. H. Engel, D. Eck, and D. Bamman, “Learning to groove with inverse sequence transformations,” in Proceedings of the 36th International
Conference on Machine Learning, ICML, 2019.
[5] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and
D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in Proc. of the 35th
International Conference on Machine Learning, 2018.
[6] J. Engel, M. Hoffman, and A. Roberts, “Latent constraints: Learning to generate conditionally from unconditional generative models,” in International Conference on Learning Representations, 2018.
[7] T. Akama, “Controlling symbolic music generation
based on concept learning from domain knowledge,” inProceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR, 2019.
[8] G. Brunner, A. Konrad, Y. Wang, and R. Wattenhofer,“MIDI-VAE: modeling dynamics and instrumentationof music with applications to style transfer,” in Proceedings of the 19th International Society for Music
Information Retrieval Conference, ISMIR, E. Gómez,X. Hu, E. Humphrey, and E. Benetos, Eds., 2018.
[9] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on
Learning Representations, ICLR, 2014.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27, 2014.
[11] C. Raffel, “Learning-based methods for comparing sequences, with applications to audio-to-midi alignment
and matching,” Ph.D. dissertation, 2016.
[12] T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and
P. Lamere, “The million song dataset,” in Proceedingsof the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011.
52[13] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, pp. 1735–80, 12
1997.
[14] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on
Learning Representations, ICLR, 2015.
[15] A. J. Hunt and A. W. Black, “Unit selection in a concatenative speech synthesis system using a large speech
database,” in 1996 IEEE International Conference on
Acoustics, Speech, and Signal Processing Conference
Proceedings, vol. 1, 1996, pp. 373–376 vol. 1.
[16] A. Zils and F. Pachet, “Musical Mosaicing,” in Proceedings of the International Conference on Digital
Audio Effects, 2001.
[17] N. Tokui, “Massh! - a web-based collective music
mashup system,” in Proceedings - 3rd International
Conference on Digital Interactive Media in Entertainment and Arts, DIMEA 2008, 2008.
[18] M. E. P. Davies, P. Hamel, K. Yoshii, and M. Goto,“Automashupper: Automatic creation of multi-songmusic mashups,” IEEE/ACM Trans. Audio, Speech andLang. Proc., 2014.
[19] A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and
J. Clune, “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks,”
in Advances in Neural Information Processing Systems
29, 2016.
[20] A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy, and
J. Clune, “Plug & play generative networks: Conditional iterative generation of images in latent space,”in Computer Vision and Pattern Recognition (CVPR),
2017.
[21] J. Mueller, D. Gifford, and T. Jaakkola, “Sequence tobetter sequence: Continuous revision of combinatorial
structures,” in Proceedings of the 34th International
Conference on Machine Learning, 2017.
[22] L. Goetschalckx, A. Andonian, A. Oliva, and P. Isola,“Ganalyze: Toward visual definitions of cognitive image properties,” in The IEEE International Conference
on Computer Vision (ICCV), 2019.
[23] A. Jahanian*, L. Chai*, and P. Isola, “On the "steerability" of generative adversarial networks,” in International Conference on Learning Representations, 2020.
[24] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired
image-to-image translation using cycle-consistent adversarial networks,” in IEEE International Conference
on Computer Vision, ICCV, 2017.
[25] G. Lample, S. Subramanian, E. Smith, L. Denoyer, M. Ranzato, and Y.-L. Boureau, “Multipleattribute text rewriting,” in International Conference
on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=H1g2NhC5KQ

Proceedings of the 21st ISMIR Conference, Montréal, Canada, October 11-16, 2020[26] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan,
and D. Krishnan, “Unsupervised pixel-level domain
adaptation with generative adversarial networks,” in2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
[27] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind,
W. Wang, and R. Webb, “Learning from simulated andunsupervised images through adversarial training,” in2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
[28] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato,“Unsupervised machine translation using monolingual
corpora only,” in International Conference on Learning Representations, 2018.
[29] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in International
Conference on Learning Representations, 2018.

53