Connective Fusion:
Learning Transformational Joining of Sequenceswith Application to Melody Creation

Taketo Akama
Sony Computer Science Laboratories (CSL), Tokyo
taketo.akama@sony.com

Introduction

Model Deployment

We present• Connective Fusion, a sequence generation scheme by transformational joining of two sequences for creative
purposes.
• Our model connects and fuses possibly unrelated sequences in a coherent way.
• This transformation can be applied iteratively to gradually fuse the input sequences. (Fig1, left)• The style latent space is simultaneously learned, allowing users to control how the two sequences are merged.
(Fig1, right)
Connective Fusion in music is useful for• providing users with musical ideas based on not necessarily polished ideas on hand.
• creating novel musical flows by combining and fusing musical fragments of different characteristics in a coherent way.

• Using qθ̂ , pθ̂ , and Gψ̂ , conduct generative transformation. (Fig.2, Deploy)

Fig.4 compares ours with baseline methods. In most cases, ours are over the baseline methods (LP), even though
LP uses abundant computational budgets. LP consists in iterated random search in the latent space to have a higher
reality classifier output.

• Iterated transformation is done by feeding the transformed latent vectors back into the input.
• Style space exploration is done by altering the input s.

Loss Function Detailsp

We use short hand LD (zL, zR) = − log Dφ(zL, zR), LnDφ (zL, zR) = −(1 − log Dφ(zL, zR)), and U = U(0, 1)ds . Dz
φ0and Dz denote paired and unpaired dataset of latent vectors, respectively. The loss function for the discriminator ispLdis = Ldis + Lndis,(1)wherepLdis =E(z ,z )∼DLRzhp
LD (zL, zR)
φhi(2)
hii

LnDφ (zL, zR)
EE 0 LnDφ (zL, zR) + EzL∼Dz zR∼DzzL∼p(z)zR∼p(z)hihi
E
E LnDφ (Gψ (zL, zR, s)) .
+ E 0 E 0 E LnDφ (Gψ (zL, zR, s)) + E

Lndis =E 0zL∼Dz zR∼Dz s∼UzL∼p(z)zR∼p(z)s∼U

The loss function for the generator isp

Lgen = Lgen + λLsim,(3)(4)

wherehp

Lgen =Lsim =p

LD (Gψ (zL, zR, s)) +E 0

E 0 E

E 0

E 0 E [Ldist(zL, zR, s)] +zL∼Dz zR∼Dz s∼UzL∼Dz zR∼Dz s∼Uhi

φEEpLDGψ (zL, zR, s)

EE

E

E [Ldist(zL, zR, s)]zL∼p(z)zR∼p(z)s∼U

φzL∼p(z)zR∼p(z)s∼U

i(5)
(6)

withFigure 1: Illustration of Application Examples


1 1
112
20 −z
0 −z
Ldist(zL, zR, s) =+log
1+zlog1+zLRLR2dz σ̄z2dσ̄zz110 ) = G (z , z , s).
(zL0 , zRψ L R

Figure 4: Evaluation of ours vs baseline methods (LP). The upper/lower rows are transformation results where inputs are randomly created pairs of test/generated (sampled from prior) data. The color gradient corresponds to the number of iterations for LP. For all metrics,higher values are better. Higher λ enforces the model to search the neighbour solution in the latent space (see Eq.4).

Fig.5 compares different number of iterations of our method for evaluating iterated transformation. Ours withlarger λ (e.g., ?) perform well in different z-distances. Iterated transformation with larger λ allows users to explore
the good balance between the coherency and the similarity.

(7)

Model TrainingExperiments• Train variational auto-encoder (VAE) to obtain qθ̂ and pθ̂ . (Fig.2, Pre-train)• Using qθ̂ , train mapping Gψ with adversarial learning. (Fig.2, Train)

Higher coherency at a given similarity is considered to be better. To evaluate similarity, we use z-distance. To
evaluate coherency, we use a reality classifier and 5 musical statistics. As depicted on the left of Fig.3, transition
matrices are estimated for musical statistics. Higher correlation with test data can be seen after transformation.
The right of Fig.3 suggests that sequences from different song ids are much less likely to have smaller z-distances.

Figure 5: Evaluation of Iterated Transformation. The color gradient corresponds to the number of iterations for our method.

Related Work• Concatenative Synthesis. Mashup.
• Learning transformation in the latent space.
Figure 2: Model Schematic

Figure 3: Visualization of musical statistics and analysis of latent space.

• Learning transformation with adversarial learning without paired data.