Title: MODULATED VARIATIONAL AUTO-ENCODERS FOR MANY-TO-MANY MUSICAL TIMBRE TRANSFER

Abstract: Generative models have been successfully applied to image style transfer and domain translation. However, there is still a wide gap in the quality of results when learning such tasks on musical audio. Furthermore, most translation models only enable one-to-one or one-to-many transfer by relying on separate encoders or decoders and complex, computationally-heavy models. In this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to perform musical timbre transfer. We define timbre transfer as applying parts of the auditory properties of a musical instrument onto another. First, we show that we can achieve this task by conditioning existing domain translation techniques with Feature-wise Linear Modulation (FiLM). Then, by replacing the usual adversarial translation criterion by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an adversarial objective. This allows a faster and more stable training along with a controllable latent space encoder. By further conditioning our system on several different instruments, we can generalize to many-to-many transfer within a single variational architecture able to perform multi-domain transfers. Our models map inputs to 3-dimensional representations, successfully translating timbre from one instrument to another and supporting sound synthesis from a reduced set of control parameters. We evaluate our method in reconstruction and generation tasks while analyzing the auditory descriptor distributions across transferred domains. We show that this architecture allows for generative controls in multi-domain transfer, yet remaining light, fast to train and effective on small datasets 1 .

1. INTRODUCTION:

Music information can be analyzed in many forms, each of which conveys different specificities over musical qualities. Among these, timbre is the set of properties that distinguishes one instrument from another playing at the same pitch and loudness. Timbre has become a core concept in music composition since the 19 th century (McAdams (2013) ). It has been studied using human dissimilarity ratings to construct timbre spaces, which exhibit the perceptual relationships between instruments (Grey (1977) ). However, these spaces are not invertible to the signal domain and do not generalize to new examples (McAdams et al. (2006) ). The heavy reliance on hand-crafted audio descriptors to analyze timbre perception altogether leads to a lack of established models to understand and generate timbres (McAdams (2013) ). Moreover, the specific nature of music tasks requires tailored evaluation principles that are yet to be ascertained (Jaffe (1995) ).

Recent advances in generative models open alternative avenues to analyze highly dimensional data and tackle complex subsequent tasks. Amongst these, the idea of style transfer (Gatys et al. (2015) ) has recently gained a flourishing interest. This approach allows to modify the stylistic features of an image while preserving its overall content and led to the more generic question of domain translation. In the recent UNsupervised Image-to-image Translation (UNIT) model, Liu et al. (2017) proposed to learn a shared latent space with a Variational Auto-Encoder (VAE) and translate between different data domains with an adversarial criterion. However, specific properties of the generation cannot be controlled and that discriminative objective might lead to an unstable and longer training. Here, we first extend this approach to musical transfer while improving it by introducing Modulated Variational auto-Encoders (MoVE) that offer control over the generation through conditioning. Furthermore, by replacing the discriminative networks by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an additional adversarial training specific to each domain.

Although UNIT provides a powerful framework, it only applies to one-to-one transfer. This implies that a different model has to be trained for each pair of domains. To mitigate this issue, Choi et al. (2017) proposed StarGAN which performs many-to-many transfer between several domains. However, it relies solely on Generative Adversarial Networks (GANs) and do not learns an implicit task representation to interact with. In the music realm, Mor et al. (2018) proposed Universal Music Translation (UMT), which does not use GANs. Although it enables translation across multiple complex audio domains, this method requires to learn a separate decoder for each domain, which leads to a prohibitive training time. In contrast to these methods, we show that MoVE can be further conditioned on domain information and generalizes to many-to-many transfer with a single encoder and decoder architecture able to perform multi-domain transfer. The resulting models are rather lightweight and fast to train while effective on a moderate amount of examples.

Here, we define timbre transfer as applying a variable part of the auditory properties of a musical instrument onto another. We circumvent the lack of definition for timbre by considering each instrument as a separate domain that maps onto a common latent representation. We further address the crucial need for interactivity and control in creative applications such as audio synthesis. Accordingly, our method yields 3-dimensional latent spaces that can be explored and controlled through high-level explicit variables such as pitch and octave values. This supports sound generation with smoothly evolving timbre qualities and complex domain transfers from a reduced set of parameters. Finally, we analyze traditional audio descriptor distributions when transferring between multiple domains or decoding across latent dimensions to demonstrate the generative capacities of our model.

2. RELATED WORKS:

Style transfer and image translation. In computer vision, style transfer (Gatys et al. (2015) ) has been proposed to generate images that preserve the content from a source image but feature stylistic qualities belonging to another target image. Although this technique provides compelling results, it operates on local textural information and fails to capture higher-level semantic properties of the style. Further research has been carried to address this question of domain translation, first proposed by Isola et al. (2017) . In the fully supervised setting, this translation would require paired samples. However, such datasets are scarce and the concept of existing samples exactly matching the translation task is restrictive from a generative perspective. In the UNIT approach (Liu et al. (2017) ), the underlying assumption is that two hypothetically matching samples should map onto the same point in a shared latent space. Hence, translation is achieved by partially weight-shared VAEs in order to map the two separate domains to a common latent representation. Learning is performed with an auxiliary pair of adversarial discriminators which push translated samples to match the distributions of their respective domains. An additional cycle-consistency objective (Zhu et al. (2017) ) reinforces the shared learning by ensuring that translated samples can be retrieved back to their original domains. However, this architecture can only operate on single domain pairs. In order to provide many-to-many translations, Choi et al. (2017) proposed to replace weight-sharing by conditioning a single GAN. This allows to train on multiple domains simultaneously, while enabling control over the generative process. However, the authors evaluate only on highly similar domains (eg. face attributes). Furthermore, this approach relies solely on GANs, which are notoriously difficult to train, prone to lack full support over the data (Grover et al. (2017) ) and do not provide a latent space encoder.

Recently, Feature-wise Linear Modulation (FiLM) has been proposed to improve conditioning by learning conditional bias and scaling throughout a network (Perez et al. (2017) ). This method was successfully used in image stylization (Ghiasi et al. (2017) ) where adaptive modulation conditioned on a style image is applied after each intermediate instance normalization. Here, we show that by relying on FiLM layers for domain conditioning, we can perform many-to-many domain translation with a single VAE architecture, as depicted in Fig. 1(c ). Moreover, by using a MMD criterion, we alleviate the need for GANs or specific adversarial discriminators. Hence, we obtain an unsupervised, lightweight and easy to train model with a general and controllable latent space. 2016)) or SampleRNN (Mehri et al. (2017) ). Most of these proposals target voice signals and there is still a large gap when addressing musical data. Some approaches have tackled musical style transfer (Verma & Smith (2018) ). However, as pointed by Dai et al. (2018) , musical style is a multimodal and multi-scale notion, which implies a variety of underlying factors. Specifically for domain translation, Mor et al. (2018) proposed a Universal Music Translation (UMT) network that globally translates musical recordings between different genres and instrument domains. Using a single Wavenet encoder and separate decoders for each domain, this approach is able to transform a given melody so that it is played by different instruments. By design, this method requires to train a specific Wavenet decoder for each of the target domain. It does not provide control over audio synthesis and the learned representation does not allow direct visualization nor transfer of only specific parts of timbre attributes. Hence, it does not enable informed generative processes, musical interaction and creativity. Our proposal targets 3-dimensional latent spaces supporting timbre transfer and continuous synthesis paths with explicit control over musical attributes.

MUSICAL TIMBRE TRANSFER:

Musical timbre can be defined as the set of auditory qualities that distinguishes two instruments playing the same note at the same loudness. Seminal studies relying on human dissimilarity ratings provided an interesting step towards understanding music perception (McAdams et al. (1995) ). However, the ordination techniques used yield non-invertible and fixed timbre spaces. Hence, they do not support audio synthesis nor do they provide a way to manipulate timbre structures. Signal processing techniques have also been developed to process and alter timbre. However, these rely on complex analysis schemes that decompose sounds into large sets of parameters (Serra (1997) ), precluding intuitive control over the audio synthesis process.

Here, we propose to use generative models in order to perform musical timbre transfer. In order to circumvent the complexity of defining timbre, our underlying hypothesis is that each instrument defines a timbral domain, which contains all style qualities that shape its identity. Musical timbre transfer can be achieved by transforming a certain amount of the auditory features of a musical instrument according to another (eg. like playing a saxophone with a bow). Transferring all timbre properties of an instrument leads to domain translation, while partial modification of these amounts to style transfer. Furthermore, our goal is to obtain a controllable model that can be used for creative purposes. Hence, we aim to obtain 3-dimensional latent spaces along with high-level musical parameters that enable human interaction and control over the generation. This type of transfer can be performed in several ways, as depicted in Fig. 1 . First, one-to-one transfer models such as UNIT map samples from a given pair of domains to a shared latent space. By learning separate layers and weight-shared layers in both the decoder and encoder, domain translation can be assessed through adversarial discriminators. We first adapt this model to timbre transfer and show that we can alleviate the need for GAN training by using an alternative MMD objective.

It leads to a faster and more stable learning that we further enhance by modulating shared layers with FiLM layers (Perez et al. (2017) ) on pitch and octave. This provides an explicit control over generation, altering or not the pitch regardless of timbre. The one-to-many transfer models (UMT) allow to work with multiple domains but require to learn a different decoder for each. This leads to a more complex and longer training and reduces the generalization ability of the model gained through multi-task learning. Here, we show that our proposed Modulated Variational auto-Encoder (MoVE) allows to perform many-to-many transfers with a single VAE simultaneously processing all domains. The success of our solution relies on an efficient domain conditioning, together with external control variables, performed through FiLM layers acting on the whole network. This solution offers a greater generalization power by jointly learning all transfer tasks within a single architecture. The resulting latent space successfully models joint and conditional distributions over several instrument domains. This also enables control with semantic labels, while providing interactive 3-dimensional spaces to synthesize novel tones from a reduced set of control parameters.

3.1. ONE-TO-ONE TRANSFER:

Our one-to-one transfer model is based on an architecture similar to UNIT (Liu et al. (2017) ) where the core idea is to learn a latent space that is shared between two domains X 1 and X 2 . Based on samples x 1 ∈ X 1 and x 2 ∈ X 2 , we aim to model the joint distribution p X1,X2 (x 1 , x 2 ) over the two domains. By learning domain-specific encoders E 1 and E 2 , matching samples drawn from each marginal distribution p X1 (x 1 ) and p X2 (x 2 ) should map onto the same latent code z = E 1 (x 1 ) = E 2 (x 2 ). Equally, any latent code can be decoded back to any of the two domains d Each VAE is trained with a reconstruction loss on its own domain, by approximating the intractable latent conditional p(z|x) with a parametric encoding network q φ (z|x) with φ ∈ Φ. In comparison to UNIT, we both use a Gaussian encoder q φ and decoder p θ so that z ∼ q φ (z|x) = N (µ φ (x), σ φ (x)) and x ∼ p θ (x|z) = N (µ θ (z), σ θ (z)). Training the model amounts to optimize {θ ; φ} on the Evidence Lower Bound Objective (ELBO), defined as a Negative Log-Likelihood (NLL) term on the output prediction error and a β-weighted Kullback-Leibler Divergence (KLD) term that assesses the error from the approximate latent density against the intractable true posterior distribution.

EQUATION

This inference objective allows to learn structured low-dimensional and invertible representations of the data, while disentangling generative factors in the encoded variables (Higgins et al. (2016) ).

Translation is performed by switching domains between the encoding and decoding stages eg.

x 1→2 = D 2 • E 1 (x 1 ). However, there is usually no matching sample x * 2 that could allow to perform the optimization of the reconstruction error L 1→2 θ2,φ1 = err(x 1→2 x * 2 ). To circumvent this challenge, UNIT relies on GANs to discriminate the generated translations against the target data distributions they model. However, this GAN criterion leads to a more complex and possibly unstable training process. Here, we show that we can efficiently replace the adversarial criterion by a differentiable distance measure on the probability distribution spaces. We minimize the Maximum Mean Discrepancy (MMD), a non-parametric kernel method (Gretton et al. (2012) ), between the set of transferred samples x 1→2 ∼ p θ2 (x|z, φ 1 ) and a randomly sampled set from the target domain

x2 ∼ p X2 L 1→2 θ2,φ1 = MMD[x 1→2 x2 ] = E x,x [k(x, x )] -2 * E x,x [k(x, x)] + E x,x [k(x, x )] ∀{x, x } ∈ x 1→2 and ∀{x, x } ∈ x2 (2)

where k is a Radial Basis Functions (RBF) kernel k

(x, x ) = n i=1 exp -αi x-x 2 .

Reconstruction and translation objectives are jointly optimized with an extra circle-consistency (CC) criterion. It consists in encoding a translated sample back to the latent space and decoding it to its source domain so that

x cc1 = D 1 • E 2 (x 1→2 ).

Hence, this double translation should retrieve the initial sample and the reconstruction error can be optimized with a NLL loss.

EQUATION

Finally, the complete optimization objective is defined as

EQUATION

where λ MMD and λ CC allow to weigh the relative influence of different objectives. For the purpose of controllable musical timbre transfer, we further apply conditioning at the input of the weightshared networks by concatenating one-hot encoded pitch classes and octaves. This pushes the shared encoder to structure note-agnostic features, while providing control over the generation.

3.2. MANY-TO-MANY TRANSFER:

In order to alleviate the one-to-one limitation that requires a different training for each domain pair, we propose the single MoVE architecture as depicted in Figure 2 . All layers are shared over the multiple domains processed, by learning a single modulated encoder E s and decoder D s . Transfer is performed by switching the categorical condition between different instruments. Hence, the practical success of this method highly depends on the conditioning strategy, which must also retain the pitch and octave control. To do so, we use an input embedding that jointly maps these categorical conditions to dense vectors fed into FiLM generators. We replace each intermediate batch normalization with instance normalization and activation is followed by a FiLM modulation layer (conditional instance normalization). Biasing and scaling are either applied feature-wise for 1-dimensional activations or channel-wise after 2-dimensional feature maps. A different generator output is used for modulating each instance normalization layer depending on its shape. The MoVE model trains in reconstruction with the ELBO and in transfer with the MMD, which is separately computed for each instrument against each of the others.

2x 3x:

Generation controls 

0 0 0 1 0 p ✓ (x | z) < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k

E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l S

r l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l S

r l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q (z | x)

M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d

< l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > 

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

4. EXPERIMENTS:

Dataset. In order to learn our timbre transfer models, we rely on the Studio-On-Line (SOL) database of orchestral instrument note recordings (Ballet et al. (1999) ). We selected 12 instruments across the 4 families of wind (Alto-Saxophone, Bassoon, Clarinet, Flute, Oboe), brass (English-Horn, French-Horn, Tenor-Trombone, Trumpet), string (Cello, Violin) and keyboard (Piano). We consider each instrumental subset as a timbral domain X i , which contains the full tessitura of each instrument at different velocities (amounting to around 100 to 200 samples per domain). We split these subsets into 90% training notes and 10% test set. The audio waveforms are down-sampled to 22050Hz before computing the Non-Stationary Gabor Transform (NSGT) (Balazs et al. (2011) ). This spectral transform allows to map to a perceptual pitch scale, while remaining iteratively invertible to the signal domain (Perraudin et al. (2013) ). NSGTs are computed on a scale of 500 Mel bins ranging from 10Hz to 11000Hz. The resulting matrix data is sliced into chunks of 16 temporal frames, amounting for a context of about 120ms. This yields a final input size of 16x500 dimensions.

We keep only the magnitude information and lowest values are floored to 6e -5 before applying a logarithmic transform. Finally, we normalize the entire dataset by computing a zero-mean unit-range normalization on all training samples.

Implementation details. The one-to-one transfer network is implemented as follows. The first encoding stacks e d are domain-specific, each composed of two 2-dimensional strided convolutions, an intermediate flattening step and a fully-connected (FC) layer. The intermediate representation is either concatenated (denoted as CAT-po models) with the conditioning vector of size 21 (12 pitch and 9 octave classes) or modulated by FiLM (denoted as FiLM-po models). Follows a weight-shared set e ws of two FC layers and two Gaussian encoder output layers mapping the input to a shared 3dimensional latent space. All intermediate layers are followed by batch normalization and a Leaky-ReLU non-linearity. This structure is mirrored in the decoders, where the latent code is conditioned and fed into a weight-shared block d ws of 3 FC layers. Follow separate decoding stacks d d each with a FC layer, two transpose convolutions that up-sample the representation and two Gaussian decoder outputs. The final output activation is a Tanh applied to the decoded means, according to the initial data scaling. Full details of the architectures are given in appendix A.

The many-to-many transfer model relies on the same architecture, but without domain-specific encoders and decoders. Hence, all layers are similar but a single network jointly processes all domains thanks to FiLM layers (denoted as FiLM-poi models). For these, an embedding layer maps our categorical vocabulary of pitch, octave and instrument classes to dense vectors which are processed by two FiLM generators, one for the encoder and one for the decoder. Each has 3 FC layers followed by scaling and biasing output pairs that each map to the size of the modulated layer. We replace all batch normalization by instance normalization and apply FiLM generator outputs as linear transform modulating normalized hidden activations. Conditional instance normalization is performed feature-wise for 1-dimensional vectors and channel-wise for 2-dimensional feature maps.

Regarding optimization, all training objectives are simultaneously back-propagated throughout all networks. We use a Xavier weight initialization and the ADAM optimizer with an initial learning rate of 1e -4 . Following the β-warmup procedure (Sønderby et al. ( 2016)), only the NLL reconstruction objective is optimized in the first epochs and the KLD strength is gradually increased from 0 to 1 during half the total number of training epochs. Similarly, we introduce the translation objective after 40 epochs and the optional circle-consistency objective after 60 epochs. We train on mini-batches of size 128 and the MMD is evaluated against batches of size 2048 sampled from the target distributions and computed with three Gaussian kernel parameter values {0.05, 0.1, 1}. We found the magnitude of MMD gradients to be much smaller than that of the ELBO. Hence, we set λ MMD to 1e 5 . Given that our models are light, the training over instrument pairs or triplets can be done in less than 24 hours on a single mid-range GPU (eg. NVIDIA TITAN Xp 12Gb).

First, we compare our MoVE proposal to UNIT on the one-to-one transfer task. To do so, we learn a different model for each pair of instruments. We perform incremental comparisons by ablating certain aspects of our proposal to assess their importance. First, we add concatenative conditioning of pitch and octave to UNIT (noted UNIT(GAN;C-po)). Then we add our proposed alternative MMD criterion replacing the GAN objective (UNIT(MMD;C-po)). Then, we introduce the FiLM layers leading to our MoVE proposal. The first version still features separate domain-specific encoders and decoders, so it is noted MoVE* (MMD; F-po). By further introducing domain conditioning and relying on a single VAE (as in Figure 2 ), we obtain our proposal MoVE (MMD; F-pod).

Evaluation scores. To evaluate reconstruction performances, we compute several criteria between input samples x and reconstructions x. The Root-Mean-Square Error RM SE= √ (x-x) 2 and Log-Spectral Distortion LSD= (10 * log 10 ( x 2 x2 )) 2 provide different assessments of how various models are able to reconstruct samples from the test set. Therefore, they only assess reconstruction abilities without domain transfer. To evaluate the quality of domain transfers, we compute the Maximum Mean Discrepancy (MMD) and the non-differentiable k-Nearest Neighbour (k-NN) test (Friedman & Rafsky (1983) ). Both are dissimilarity measures computed between the target data distribution and transferred samples. Hence, we evaluate test set transfers between different target domains.

Reconstruction and domain transfer. The averaged reconstruction and transfer results are presented in Table 3 , while separate evaluations for different pairs are in Annex B. As we can see, the UNIT-MMD model obtains the highest within-domain reconstruction score, while the MoVE model achieves better domain translation. Hence, it appears that the MMD increases reconstruction performance, and that the FiLM conditioning ameliorates the transfer. It also seems that relying on a single encoder and decoder for domain transfer might provide better generalization, as can be verified by looking at the relative MMD and kNN scores on the transfer task. Indeed, it seems that the modulated but separate layers approach perform worse, while the single architecture performs better on most evaluations. Audio descriptors topology. Audio descriptors are features used to compare the qualities of different sounds (Peeters et al. (2011) ). Hence, we rely on these to assess the effect of transfer, while providing a deeper understanding of its behavior. We compute the spectral flatness, centroid, roll-off and loudness on test samples reconstructed on their own domain or transferred to the other domain. Distribution and sample-specific plots for the spectral centroid are presented in Figure 3 . As we can see, the transfer produces an almost exact match of the descriptor distribution to the target domain. This shows the success in transferring multimodal distributions of auditory properties, as all the modes of the descriptors' distributions are preserved. The scatter plot also suggests that the centroid transfer is highly influenced by the loudness of the sample. This correlates to perception studies, as playing an instrument louder usually leads to a higher centroid McAdams et al. (1995) .

In order to further understand how the latent space is organized with respect to audio descriptors, we provide their spatial topology in Figure 4 . To compute this, we define a sampling grid over the latent space and decode the audio at each point to compute their descriptors. As we can see, the audio descriptors are locally very smooth. Furthermore, one key observation is that the latent space of both conditioned target domains follow the same overall topology. Animations showing the complete latent descriptor topology are available on the supporting webpage.

Latent space synthesis and performance. As the latent space provides continuous audio synthesis and that our method introduce high-level conditioned controls, we can use our proposal as a full musical synthesizer. Furthermore, as we map to 3-dimensional spaces, the user can directly interact with the space while performing timbre transfer. Furthermore, although these models are trained to transfer single instrumental notes, they still can be used to transfer a full melody recording between timbre domains. To do so, the recording is split and iteratively reconstructed by transfering each signal window to the target domain. Audio examples of applying this strategy to transfer a complete instrumental solo are also available in the supporting webpage.

Here, we evaluate the application of MoVE to perform many-to-many transfer. Given our new architecture, this simply consists in training on multiple domains at once by modulating with the appropriate domain information. This architecture allows us to train a single model for different domains and thus to perform multi-domain translation. The conditioning vector is then composed of the pitch, the octave and here the instrument of the corresponding example. This conditioning vector is then processed by an embedding to ease the FILM conditioning. Results are presented in Table 2 : we can see that the MoVE architecture is able to reconstruct and transfer multiple domains at the same time at the cost of a slight decrease in performance, even in the case of diverse domains (here Alto-Sax, Flute, Violin and French-Horn). 

5. CONCLUSION:

We introduced the Modulated Variational auto-Encoders (MoVE), which perform many-to-many domain transfer within a single architecture and without adversarial training while providing highlevel control over the generation. We effectively adapted this technique to musical timbre transfer and showed the successes of our method for audio synthesis. As our technique is generic, it could be applied to other types of data such as image or video. The architecture itself opens up a range of potential sonic applications such as playing style conditioning, transfers between acoustical and electronic instruments, and even with non-musical sound domains. Another avenue of research to be investigated is controlling the amount of transfer performed by the model. 

:

Audio examples, source code and animations available at https://github.com/anonymous124/iclr2019MoVE

Figures and Tables:

Figure 1: Different approaches to domain translation. (a) One-to-one transfer models such as UNIT are restricted to domain pairs. (b) One-to-many transfer models such as UMT require to train a different decoder for each domain. (c) Our proposed many-to-many transfer model (MoVE) allows to perform multi-domain transfer with a single encoder and decoder, while providing control over the generation with external high-level conditioning variables.

∈ {1, 2} by learning appropriate decoders x * d = D d (z). A paired VAE implements this assumption through separate domain-specific layers {e d ; d d } alternated with weight-shared ones {e ws ; d ws }. The full encoders and decoders are defined by the composition of both parts E d = e ws •e d and D d = d d •d ws .

o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A

s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q gZ O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l Sr l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 ds T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 yL o H F c M Y 2 K e X l Sr l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t >Output FC layerTranspose conv.

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

< l a t e x i t s h a 1 _ b a s e 6 4 = " r Qf g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X Xr D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S k e H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e oo U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s BM a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S ke H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0 x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e o o U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S k e H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e oo U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R Li I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S ke H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0 x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e o o U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > ⌃ x < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

l W x m 9 W m + F C n p A G P p 2 j + D k 7 K J Z u m f l g p N m r Z q P P Y w C a 2 a Z 5 7 a O A A T b T I + x Z P e M a L w Y w 7 4 9 5 4 + C o 1 c p l m D T + W 8 f g J l u m a f Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

l W x m 9 W m + F C n p A G P p 2 j + D k 7 K J Z u m f l g p N m r Z q P P Y w C a 2 a Z 5 7 a O A A T b T I + x Z P e M a L w Y w 7 4 9 5 4 + C o 1 c p l m D T + W 8 f g J l u m a f Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

Figure 2: The Modulated Variational auto-Encoder (MoVE) provides a single architecture able to perform many-to-many transfer while controlling the generation with external parameters. Both the domain and control information are processed to modulate different layers of the architecture.

Figure 3: Understanding the effect of musical timbre transfer through audio descriptor distributions.

Figure 4: Topology of the latent space with respect to audio descriptors.

Evaluations of various models on the test sets

Table content: <table><tr><td/><td colspan="2">reconstructions</td><td/><td colspan="2">transfers</td></tr><tr><td>RMSE</td><td>LSD</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td></tr><tr><td colspan="2">UNIT (GAN) 0.3412 718.47</td><td>2.117e-2</td><td>57269</td><td>2.038 e-2</td><td>43180</td></tr><tr><td colspan="2">UNIT (GAN; C-po) 0.3011 693.22</td><td>1.989 e-2</td><td>57806</td><td>9.112 e-2</td><td>43414</td></tr><tr><td colspan="2">UNIT (MMD; C-po) 0.3036 692.41</td><td>2.125 e-2</td><td>57102</td><td>2.304 e-2</td><td>43878</td></tr><tr><td colspan="2">MoVE* (MMD; F-po) 0.3134 762.51</td><td>9.632 e-3</td><td>57273</td><td>3.153 e-2</td><td>43443</td></tr><tr><td colspan="2">MoVE (MMD; F-pod) 0.3339 781.11</td><td>2.587 e-3</td><td>57509</td><td>1.747 e-2</td><td>43173</td></tr></table>

Many-to-Many MoVE reconstruction & transfer scores

Table content: <table><tr><td/><td colspan="3">averaged reconstructions</td><td colspan="2">averaged transfers</td></tr><tr><td>RMSE</td><td>LSD</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td></tr><tr><td colspan="2">Alto-Saxophone 0.5327 835.67</td><td>2.117 e-2</td><td>42299</td><td>2.386 e-2</td><td>59157</td></tr><tr><td colspan="2">Flute 0.4593 761.46</td><td>2.119 e-2</td><td>49719</td><td>1.975 e-2</td><td>57277</td></tr><tr><td colspan="2">Violin 0.3271 773.65</td><td>5.659 e-3</td><td>58013</td><td>1.452 e-2</td><td>55379</td></tr><tr><td colspan="2">French-Horn 0.6239 869.69</td><td>3.404 e-3</td><td>70946</td><td>2.086 e-2</td><td>51317</td></tr></table>

Pair-wise evaluations on Clarinet and Piano test sets

Table content: <table><tr><td/><td/><td colspan="2">reconstructions Clarinet</td><td/><td colspan="2">transfers to Piano</td></tr><tr><td/><td>RMSE.</td><td>LSD.</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td></tr><tr><td>UNIT (GAN; C-po)</td><td>0.6915</td><td>778.33</td><td>3.918 e-2</td><td>76035</td><td>2.827 e-2</td></tr><tr><td>MoVE* (MMD; F-po)</td><td>0.7351</td><td>916.88</td><td>1.144 e-2</td><td>75440</td><td>1.287 e-2</td></tr><tr><td>MoVE (MMD; F-pod)</td><td>0.6878</td><td>884.63</td><td>1.614 e-2</td><td>75604</td><td>1.462e-2</td></tr><tr><td/><td/><td colspan="2">reconstructions Piano</td><td/><td colspan="2">transfers to Clarinet</td></tr><tr><td colspan="3">UNIT (GAN; C-po) 6.682 e-2 543.09</td><td>2.610 e-2</td><td>70925</td><td>4.071 e-2</td></tr><tr><td colspan="3">MoVE* (MMD; F-po) 6.522 e-2 582.98</td><td>1.276 e-2</td><td>70055</td><td>6.790 e-3</td></tr><tr><td colspan="3">MoVE (MMD; F-pod) 5.727 e-2 578.66</td><td>1.547 e-2</td><td>70044</td><td>1.884 e-2</td></tr></table>

many-to-many MoVE evaluation on Clarinet Cello, Tenor-Trombone (T-Trombone) and Piano test sets (350 epochs only)

Table content: <table><tr><td>input</td><td/><td colspan="2">reconstructions</td><td/><td>targets</td><td colspan="2">transfers</td></tr><tr><td/><td>RMSE.</td><td>LSD.</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td><td/><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td></tr><tr><td/><td/><td/><td/><td/><td>Cello</td><td>1.205 e-2</td></tr><tr><td>Clarinet</td><td>0.6832</td><td>873.17</td><td>9.776 e-3</td><td>76093</td><td>T-Trombone</td><td>7.450 e-3</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1233</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>1.345 e-3</td></tr><tr><td>Cello</td><td>0.3043</td><td>755.58</td><td>2.115 e-3</td><td>61756</td><td>T-Trombone</td><td>1.326 e-3</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1595</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>2.089 e-2</td></tr><tr><td>T-Trombone</td><td>0.7362</td><td>1047.1</td><td>1.281 e-2</td><td>25629</td><td>Cello</td><td>1.652 e-2</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1649</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>0.1482</td></tr><tr><td>Piano</td><td colspan="2">4.965 e-2 542.72</td><td>9.773 e-3</td><td>69573</td><td>Cello</td><td>0.1155</td></tr><tr><td/><td/><td/><td/><td/><td>T-Trombone</td><td>0.2038</td></tr></table>

Peer review (iclr2019):
 This work proposes a hybrid VAE-based model (combined with an adversarial or maximum mean discrepancy (MMD) based loss) to perform timbre transfer on recordings of musical instruments. Contrary to previous work, a single (conditioned) decoder is used for all instrument domains, which means a single model can be used to convert any source domain to any target domain. Unfortunately, the results are quite disappointing in terms of sound quality, and feature many artifacts. The instruments are often unrecognisable, although with knowledge of the target domain, some of its characteristics can be identified. The many-to-many results are clearly better than the pairwise results in this regard, but in the context of musical timbre transfer, I don*t feel that this model successfully achieves its goal -- the results of Mor et al. (2018), although not perfect either, were better in this regard. I have several further concerns about this work: * The fact that the model makes use of pitch class and octave labels also raises questions about applicability -- if I understood correctly, transfer can only be done when this information is present. I think the main point of transfer over a regular generative model that goes from labels to audio is precisely that it can be done without label information. * The use of fully connected layers also implies that it requires fixed length input, so windowing and stitching are necessary for it to be applied to recordings of arbitrary length. Why not train a convolutional model instead? * I think the choice of a 3-dimensional latent space is poorly justified. Why not use more dimensions and project them down to 3 for visualisation and interpetation purposes with e.g. PCA or t-SNE? This seems like an unnecessary bottleneck in the model, and could partly explain the relatively poor quality of the results. I appreciated that the one-to-one transfer experiments are incremental comparisons, which provides valuable information about how much each idea contributes to the final performance. Overall, I feel that this paper falls short of what it promises, so I cannot recommend acceptance at this time. Other comments: * In the introduction, an adversarial criterion is referred to as a "discriminative objective", but "adversarial" (i.e. featuring a discriminator) and "discriminative" mean different things. I don*t think it is correct to refer to an adversarial criterion as discriminative. * Also in the introduction, it is implied that style transfer constitutes an advance in generative models, but style transfer does not make use of / does not equate to any generative model. * Some turns of phrase like "recently gained a flourishing interest", "there is still a wide gap in quality of results", "which implies a variety of underlying factors", ... are vague / do not make much sense and should probably be reformulated to enhance readability. * Introduction, top of page 2: should read "does not learn" instead of "do not learns". * Mor et al. (2018) do actually make use of an adversarial training criterion (referred to as a "domain confusion loss"), contrary to what is claimed in the introduction. * The claim that training a separate decoder for each domain necessarily leads to prohibitive training times is dubious -- a single conditional decoder would arguably need more capacity than each individual separate decoder model. I think all claims about running time should be corroborated by controlled experiments. * I think Figure 1 is great and helps a lot to distinguish the different domain translation paradigms. * I found the description in Section 3.1 a bit confusing as it initially seems that the approach requires paired data (e.g. "matching samples"). * Section 3.1, "amounts to optimizing" instead of "amounts to optimize" * Higgins et al. (2016) specifically discuss the case where beta in formula (1) is larger than one. As far as I can tell, beta is annealed from 0 to 1 here, which is an idea that goes back to "Generating Sentences from a Continuous Space" by Bowman et al. (2016). This should probably be cited instead. * "circle-consistency" should read "cycle-consistency" everywhere. * MMD losses in the context of GANs have also been studied in the following papers: - "Training generative neural networks via Maximum Mean Discrepancy optimization", Dziugaite et al. (2015) - "Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy", Sutherland et al. (2016) - "MMD GAN: Towards Deeper Understanding of Moment Matching Network", Li et al. (2017) * The model name "FILM-poi" is only used in the "implementation details" section, it doesn*t seem to be referred to anywhere else. Is this a typo? * The differences between UNIT (GAN; C-po) and UNIT (MMD; C-po) in Table 1 seem very small and I*m not convinced that they are significant. Why does the MMD version constitute an improvement? Or is it simply more stable to train? * The descriptor distributions in Figure 3 don*t look like an "almost exact match" to me (as claimed in the text). There are some clearly visible differences. I think the wording is a bit too strong here.