Title: MODULATED VARIATIONAL AUTO-ENCODERS FOR MANY-TO-MANY MUSICAL TIMBRE TRANSFER

Abstract: Generative models have been successfully applied to image style transfer and domain translation. However, there is still a wide gap in the quality of results when learning such tasks on musical audio. Furthermore, most translation models only enable one-to-one or one-to-many transfer by relying on separate encoders or decoders and complex, computationally-heavy models. In this paper, we introduce the Modulated Variational auto-Encoders (MoVE) to perform musical timbre transfer. We define timbre transfer as applying parts of the auditory properties of a musical instrument onto another. First, we show that we can achieve this task by conditioning existing domain translation techniques with Feature-wise Linear Modulation (FiLM). Then, by replacing the usual adversarial translation criterion by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an adversarial objective. This allows a faster and more stable training along with a controllable latent space encoder. By further conditioning our system on several different instruments, we can generalize to many-to-many transfer within a single variational architecture able to perform multi-domain transfers. Our models map inputs to 3-dimensional representations, successfully translating timbre from one instrument to another and supporting sound synthesis from a reduced set of control parameters. We evaluate our method in reconstruction and generation tasks while analyzing the auditory descriptor distributions across transferred domains. We show that this architecture allows for generative controls in multi-domain transfer, yet remaining light, fast to train and effective on small datasets 1 .

1. INTRODUCTION:

Music information can be analyzed in many forms, each of which conveys different specificities over musical qualities. Among these, timbre is the set of properties that distinguishes one instrument from another playing at the same pitch and loudness. Timbre has become a core concept in music composition since the 19 th century (McAdams (2013) ). It has been studied using human dissimilarity ratings to construct timbre spaces, which exhibit the perceptual relationships between instruments (Grey (1977) ). However, these spaces are not invertible to the signal domain and do not generalize to new examples (McAdams et al. (2006) ). The heavy reliance on hand-crafted audio descriptors to analyze timbre perception altogether leads to a lack of established models to understand and generate timbres (McAdams (2013) ). Moreover, the specific nature of music tasks requires tailored evaluation principles that are yet to be ascertained (Jaffe (1995) ).

Recent advances in generative models open alternative avenues to analyze highly dimensional data and tackle complex subsequent tasks. Amongst these, the idea of style transfer (Gatys et al. (2015) ) has recently gained a flourishing interest. This approach allows to modify the stylistic features of an image while preserving its overall content and led to the more generic question of domain translation. In the recent UNsupervised Image-to-image Translation (UNIT) model, Liu et al. (2017) proposed to learn a shared latent space with a Variational Auto-Encoder (VAE) and translate between different data domains with an adversarial criterion. However, specific properties of the generation cannot be controlled and that discriminative objective might lead to an unstable and longer training. Here, we first extend this approach to musical transfer while improving it by introducing Modulated Variational auto-Encoders (MoVE) that offer control over the generation through conditioning. Furthermore, by replacing the discriminative networks by a Maximum Mean Discrepancy (MMD) objective, we alleviate the need for an additional adversarial training specific to each domain.

Although UNIT provides a powerful framework, it only applies to one-to-one transfer. This implies that a different model has to be trained for each pair of domains. To mitigate this issue, Choi et al. (2017) proposed StarGAN which performs many-to-many transfer between several domains. However, it relies solely on Generative Adversarial Networks (GANs) and do not learns an implicit task representation to interact with. In the music realm, Mor et al. (2018) proposed Universal Music Translation (UMT), which does not use GANs. Although it enables translation across multiple complex audio domains, this method requires to learn a separate decoder for each domain, which leads to a prohibitive training time. In contrast to these methods, we show that MoVE can be further conditioned on domain information and generalizes to many-to-many transfer with a single encoder and decoder architecture able to perform multi-domain transfer. The resulting models are rather lightweight and fast to train while effective on a moderate amount of examples.

Here, we define timbre transfer as applying a variable part of the auditory properties of a musical instrument onto another. We circumvent the lack of definition for timbre by considering each instrument as a separate domain that maps onto a common latent representation. We further address the crucial need for interactivity and control in creative applications such as audio synthesis. Accordingly, our method yields 3-dimensional latent spaces that can be explored and controlled through high-level explicit variables such as pitch and octave values. This supports sound generation with smoothly evolving timbre qualities and complex domain transfers from a reduced set of parameters. Finally, we analyze traditional audio descriptor distributions when transferring between multiple domains or decoding across latent dimensions to demonstrate the generative capacities of our model.

2. RELATED WORKS:

Style transfer and image translation. In computer vision, style transfer (Gatys et al. (2015) ) has been proposed to generate images that preserve the content from a source image but feature stylistic qualities belonging to another target image. Although this technique provides compelling results, it operates on local textural information and fails to capture higher-level semantic properties of the style. Further research has been carried to address this question of domain translation, first proposed by Isola et al. (2017) . In the fully supervised setting, this translation would require paired samples. However, such datasets are scarce and the concept of existing samples exactly matching the translation task is restrictive from a generative perspective. In the UNIT approach (Liu et al. (2017) ), the underlying assumption is that two hypothetically matching samples should map onto the same point in a shared latent space. Hence, translation is achieved by partially weight-shared VAEs in order to map the two separate domains to a common latent representation. Learning is performed with an auxiliary pair of adversarial discriminators which push translated samples to match the distributions of their respective domains. An additional cycle-consistency objective (Zhu et al. (2017) ) reinforces the shared learning by ensuring that translated samples can be retrieved back to their original domains. However, this architecture can only operate on single domain pairs. In order to provide many-to-many translations, Choi et al. (2017) proposed to replace weight-sharing by conditioning a single GAN. This allows to train on multiple domains simultaneously, while enabling control over the generative process. However, the authors evaluate only on highly similar domains (eg. face attributes). Furthermore, this approach relies solely on GANs, which are notoriously difficult to train, prone to lack full support over the data (Grover et al. (2017) ) and do not provide a latent space encoder.

Recently, Feature-wise Linear Modulation (FiLM) has been proposed to improve conditioning by learning conditional bias and scaling throughout a network (Perez et al. (2017) ). This method was successfully used in image stylization (Ghiasi et al. (2017) ) where adaptive modulation conditioned on a style image is applied after each intermediate instance normalization. Here, we show that by relying on FiLM layers for domain conditioning, we can perform many-to-many domain translation with a single VAE architecture, as depicted in Fig. 1(c ). Moreover, by using a MMD criterion, we alleviate the need for GANs or specific adversarial discriminators. Hence, we obtain an unsupervised, lightweight and easy to train model with a general and controllable latent space. 2016)) or SampleRNN (Mehri et al. (2017) ). Most of these proposals target voice signals and there is still a large gap when addressing musical data. Some approaches have tackled musical style transfer (Verma & Smith (2018) ). However, as pointed by Dai et al. (2018) , musical style is a multimodal and multi-scale notion, which implies a variety of underlying factors. Specifically for domain translation, Mor et al. (2018) proposed a Universal Music Translation (UMT) network that globally translates musical recordings between different genres and instrument domains. Using a single Wavenet encoder and separate decoders for each domain, this approach is able to transform a given melody so that it is played by different instruments. By design, this method requires to train a specific Wavenet decoder for each of the target domain. It does not provide control over audio synthesis and the learned representation does not allow direct visualization nor transfer of only specific parts of timbre attributes. Hence, it does not enable informed generative processes, musical interaction and creativity. Our proposal targets 3-dimensional latent spaces supporting timbre transfer and continuous synthesis paths with explicit control over musical attributes.

MUSICAL TIMBRE TRANSFER:

Musical timbre can be defined as the set of auditory qualities that distinguishes two instruments playing the same note at the same loudness. Seminal studies relying on human dissimilarity ratings provided an interesting step towards understanding music perception (McAdams et al. (1995) ). However, the ordination techniques used yield non-invertible and fixed timbre spaces. Hence, they do not support audio synthesis nor do they provide a way to manipulate timbre structures. Signal processing techniques have also been developed to process and alter timbre. However, these rely on complex analysis schemes that decompose sounds into large sets of parameters (Serra (1997) ), precluding intuitive control over the audio synthesis process.

Here, we propose to use generative models in order to perform musical timbre transfer. In order to circumvent the complexity of defining timbre, our underlying hypothesis is that each instrument defines a timbral domain, which contains all style qualities that shape its identity. Musical timbre transfer can be achieved by transforming a certain amount of the auditory features of a musical instrument according to another (eg. like playing a saxophone with a bow). Transferring all timbre properties of an instrument leads to domain translation, while partial modification of these amounts to style transfer. Furthermore, our goal is to obtain a controllable model that can be used for creative purposes. Hence, we aim to obtain 3-dimensional latent spaces along with high-level musical parameters that enable human interaction and control over the generation. This type of transfer can be performed in several ways, as depicted in Fig. 1 . First, one-to-one transfer models such as UNIT map samples from a given pair of domains to a shared latent space. By learning separate layers and weight-shared layers in both the decoder and encoder, domain translation can be assessed through adversarial discriminators. We first adapt this model to timbre transfer and show that we can alleviate the need for GAN training by using an alternative MMD objective.

It leads to a faster and more stable learning that we further enhance by modulating shared layers with FiLM layers (Perez et al. (2017) ) on pitch and octave. This provides an explicit control over generation, altering or not the pitch regardless of timbre. The one-to-many transfer models (UMT) allow to work with multiple domains but require to learn a different decoder for each. This leads to a more complex and longer training and reduces the generalization ability of the model gained through multi-task learning. Here, we show that our proposed Modulated Variational auto-Encoder (MoVE) allows to perform many-to-many transfers with a single VAE simultaneously processing all domains. The success of our solution relies on an efficient domain conditioning, together with external control variables, performed through FiLM layers acting on the whole network. This solution offers a greater generalization power by jointly learning all transfer tasks within a single architecture. The resulting latent space successfully models joint and conditional distributions over several instrument domains. This also enables control with semantic labels, while providing interactive 3-dimensional spaces to synthesize novel tones from a reduced set of control parameters.

3.1. ONE-TO-ONE TRANSFER:

Our one-to-one transfer model is based on an architecture similar to UNIT (Liu et al. (2017) ) where the core idea is to learn a latent space that is shared between two domains X 1 and X 2 . Based on samples x 1 ∈ X 1 and x 2 ∈ X 2 , we aim to model the joint distribution p X1,X2 (x 1 , x 2 ) over the two domains. By learning domain-specific encoders E 1 and E 2 , matching samples drawn from each marginal distribution p X1 (x 1 ) and p X2 (x 2 ) should map onto the same latent code z = E 1 (x 1 ) = E 2 (x 2 ). Equally, any latent code can be decoded back to any of the two domains d Each VAE is trained with a reconstruction loss on its own domain, by approximating the intractable latent conditional p(z|x) with a parametric encoding network q φ (z|x) with φ ∈ Φ. In comparison to UNIT, we both use a Gaussian encoder q φ and decoder p θ so that z ∼ q φ (z|x) = N (µ φ (x), σ φ (x)) and x ∼ p θ (x|z) = N (µ θ (z), σ θ (z)). Training the model amounts to optimize {θ ; φ} on the Evidence Lower Bound Objective (ELBO), defined as a Negative Log-Likelihood (NLL) term on the output prediction error and a β-weighted Kullback-Leibler Divergence (KLD) term that assesses the error from the approximate latent density against the intractable true posterior distribution.

EQUATION

This inference objective allows to learn structured low-dimensional and invertible representations of the data, while disentangling generative factors in the encoded variables (Higgins et al. (2016) ).

Translation is performed by switching domains between the encoding and decoding stages eg.

x 1→2 = D 2 • E 1 (x 1 ). However, there is usually no matching sample x * 2 that could allow to perform the optimization of the reconstruction error L 1→2 θ2,φ1 = err(x 1→2 x * 2 ). To circumvent this challenge, UNIT relies on GANs to discriminate the generated translations against the target data distributions they model. However, this GAN criterion leads to a more complex and possibly unstable training process. Here, we show that we can efficiently replace the adversarial criterion by a differentiable distance measure on the probability distribution spaces. We minimize the Maximum Mean Discrepancy (MMD), a non-parametric kernel method (Gretton et al. (2012) ), between the set of transferred samples x 1→2 ∼ p θ2 (x|z, φ 1 ) and a randomly sampled set from the target domain

x2 ∼ p X2 L 1→2 θ2,φ1 = MMD[x 1→2 x2 ] = E x,x [k(x, x )] -2 * E x,x [k(x, x)] + E x,x [k(x, x )] ∀{x, x } ∈ x 1→2 and ∀{x, x } ∈ x2 (2)

where k is a Radial Basis Functions (RBF) kernel k

(x, x ) = n i=1 exp -αi x-x 2 .

Reconstruction and translation objectives are jointly optimized with an extra circle-consistency (CC) criterion. It consists in encoding a translated sample back to the latent space and decoding it to its source domain so that

x cc1 = D 1 • E 2 (x 1→2 ).

Hence, this double translation should retrieve the initial sample and the reconstruction error can be optimized with a NLL loss.

EQUATION

Finally, the complete optimization objective is defined as

EQUATION

where λ MMD and λ CC allow to weigh the relative influence of different objectives. For the purpose of controllable musical timbre transfer, we further apply conditioning at the input of the weightshared networks by concatenating one-hot encoded pitch classes and octaves. This pushes the shared encoder to structure note-agnostic features, while providing control over the generation.

3.2. MANY-TO-MANY TRANSFER:

In order to alleviate the one-to-one limitation that requires a different training for each domain pair, we propose the single MoVE architecture as depicted in Figure 2 . All layers are shared over the multiple domains processed, by learning a single modulated encoder E s and decoder D s . Transfer is performed by switching the categorical condition between different instruments. Hence, the practical success of this method highly depends on the conditioning strategy, which must also retain the pitch and octave control. To do so, we use an input embedding that jointly maps these categorical conditions to dense vectors fed into FiLM generators. We replace each intermediate batch normalization with instance normalization and activation is followed by a FiLM modulation layer (conditional instance normalization). Biasing and scaling are either applied feature-wise for 1-dimensional activations or channel-wise after 2-dimensional feature maps. A different generator output is used for modulating each instance normalization layer depending on its shape. The MoVE model trains in reconstruction with the ELBO and in transfer with the MMD, which is separately computed for each instrument against each of the others.

2x 3x:

Generation controls 

0 0 0 1 0 p ✓ (x | z) < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Z B g 6 o c X Q z P H k F d 2 f s F R u h C G U T Y = " > A A A C 6 X i c j V H L S s N A F D 2 N r / q u u n Q T r E L d l E Q E X R b c u K x g H 9 B K m a R T O z Q v k

E d m q h o U X t N F F C B c p f H A E k I Q 9 M C T 0 t G D D Q k T c G Y b E x Y S E z n O M s E D a l K o 4 V T B i B / Q 9 p 6 i V s Q H F y j P R a p d 2 8 e i N S W l i h z Q h 1 c W E 1 W 6 m z q f a W b G / e Q + 1 p z r b F f 2 d z M s n V q J P 7 F + 6 c e V / d a o X i R 4 O d Q + C e o o 0 o 7 p z M 5 d U 3 4 o 6 u f m l K 0 k O E X E K d y k f E 3 a 1 c n z P p t Y k u n d 1 t 0 z n 3 3 S l Y l X s Z r U p 3 t U p a c D 2 z 3 F O g v p e 2 b b K 9 s l + s X K Y j T q P T W y h R P M 8 Q A X H q K J G 3 j d 4 x B O e j Y F x a 9 w Z 9 5 + l R i 7 T b O D b M h 4 + A A a i n n A = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l S

r l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l S

r l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q (z | x)

M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 d

< l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z u 

x k v 1 b x i V j f k 3 o V q F Q p l d X w Y 6 o = " > A A A C 6 X i c j V H L S s N A F D 2 N 7 / q K u n Q T r E L d l E Q E u

V O 3 B E z + 3 k V 0 P D D n j b G M W X w 8 2 W X j F r p l r G T 2 A V o I J i 7 U f 6 M 2 y 0 E c F D h g A M I Q R h H w 5 S e k 5 h w U R M 3 B l y 4 h J C X O U Z h i i T N q M q R h U O s X 3 6 d i k 6 L d i Q Y u m Z K r V H u / j 0 J q Q 0 s E G a i O o S w n I 3 Q + U z 5 S z Z 3 7 x z 5 S n P N q C / W 3 g F x A r 0 i P 1 L N 6 r 8 r 0 7 2 I t B B X f X A q a d Y M b I 7 r 3 D J 1 K 3 I k x u f u h L k E B M n c Z v y C W F P K U f 3 b C h N q n q X d + u o / K u q l K

y M v a I 2 w 5 s 8 J Q 3 Y + j 7 O n + B o q 2 a Z N e t g u 9 K o F 6 O e x i r W U K V 5 7 q C B P e y j S d 7 X e M A j n r S + d q P d a n c f p V q p 0 K z g y 9 L u 3 w F g h 5 6 U < / l a t e x i t > 

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

O V L x 1 w 0 T G k / h M D l N 2 G X m 9 m H d 5 4 E m i z i 8 i T / b 9 7 u h 2 f F W u O F V H L 3 s a u A Z U Y F Y 9 K b / g A t d I E C B H B I Y Y k n A I D x k 9 H b h w k B J 3 i R F x g h D X c Y Y x S q T N K Y t R h k f s g L 4 9 2 n U M G 9 N e e W Z a H d A p I b 2 C l D Z 2 S J N Q n i C s T r N 1 P N f O i v 3 N e 6 Q 9 1 d 2 G 9 P e N V 0 S s R J / Y v 3 S T z P / q V C 0 S X R z q G j j V l G p G V R c Y l 1 x 3 R d 3 c / l K V J I e U O I W v K S 4 I B 1 o 5 6 b O t N Z m u X f X W 0 / E 3 n a l Y t Q 9 M

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

0 F y v V d u 2 b a J U u v I r L V D A u v a O M K I V y k 8 M E R Q B L 2 w J D Q c w 4 b F i L i L j A k L i Y k d J 5 j h D n S p l T F q Y I R 2 6 d v l 6 L z j A 0 o V p 6 J V r u 0 i 0 d v T E o T W 6 Q J q S 4 m r H Y z d T 7 V z o r 9 z X u o P d X Z B v R 3 M i + f W I k e s X / p x p X / 1 a l e J D q o 6 R 4 E 9 R R p R n X n Z i 6 p v h V 1 c v N b V 5 I c I u I U v q J 8 T N j V y v E 9 m 1 q T 6 N 7 V 3 T K d f 9 e V i

4. EXPERIMENTS:

Dataset. In order to learn our timbre transfer models, we rely on the Studio-On-Line (SOL) database of orchestral instrument note recordings (Ballet et al. (1999) ). We selected 12 instruments across the 4 families of wind (Alto-Saxophone, Bassoon, Clarinet, Flute, Oboe), brass (English-Horn, French-Horn, Tenor-Trombone, Trumpet), string (Cello, Violin) and keyboard (Piano). We consider each instrumental subset as a timbral domain X i , which contains the full tessitura of each instrument at different velocities (amounting to around 100 to 200 samples per domain). We split these subsets into 90% training notes and 10% test set. The audio waveforms are down-sampled to 22050Hz before computing the Non-Stationary Gabor Transform (NSGT) (Balazs et al. (2011) ). This spectral transform allows to map to a perceptual pitch scale, while remaining iteratively invertible to the signal domain (Perraudin et al. (2013) ). NSGTs are computed on a scale of 500 Mel bins ranging from 10Hz to 11000Hz. The resulting matrix data is sliced into chunks of 16 temporal frames, amounting for a context of about 120ms. This yields a final input size of 16x500 dimensions.

We keep only the magnitude information and lowest values are floored to 6e -5 before applying a logarithmic transform. Finally, we normalize the entire dataset by computing a zero-mean unit-range normalization on all training samples.

Implementation details. The one-to-one transfer network is implemented as follows. The first encoding stacks e d are domain-specific, each composed of two 2-dimensional strided convolutions, an intermediate flattening step and a fully-connected (FC) layer. The intermediate representation is either concatenated (denoted as CAT-po models) with the conditioning vector of size 21 (12 pitch and 9 octave classes) or modulated by FiLM (denoted as FiLM-po models). Follows a weight-shared set e ws of two FC layers and two Gaussian encoder output layers mapping the input to a shared 3dimensional latent space. All intermediate layers are followed by batch normalization and a Leaky-ReLU non-linearity. This structure is mirrored in the decoders, where the latent code is conditioned and fed into a weight-shared block d ws of 3 FC layers. Follow separate decoding stacks d d each with a FC layer, two transpose convolutions that up-sample the representation and two Gaussian decoder outputs. The final output activation is a Tanh applied to the decoded means, according to the initial data scaling. Full details of the architectures are given in appendix A.

The many-to-many transfer model relies on the same architecture, but without domain-specific encoders and decoders. Hence, all layers are similar but a single network jointly processes all domains thanks to FiLM layers (denoted as FiLM-poi models). For these, an embedding layer maps our categorical vocabulary of pitch, octave and instrument classes to dense vectors which are processed by two FiLM generators, one for the encoder and one for the decoder. Each has 3 FC layers followed by scaling and biasing output pairs that each map to the size of the modulated layer. We replace all batch normalization by instance normalization and apply FiLM generator outputs as linear transform modulating normalized hidden activations. Conditional instance normalization is performed feature-wise for 1-dimensional vectors and channel-wise for 2-dimensional feature maps.

Regarding optimization, all training objectives are simultaneously back-propagated throughout all networks. We use a Xavier weight initialization and the ADAM optimizer with an initial learning rate of 1e -4 . Following the β-warmup procedure (Sønderby et al. ( 2016)), only the NLL reconstruction objective is optimized in the first epochs and the KLD strength is gradually increased from 0 to 1 during half the total number of training epochs. Similarly, we introduce the translation objective after 40 epochs and the optional circle-consistency objective after 60 epochs. We train on mini-batches of size 128 and the MMD is evaluated against batches of size 2048 sampled from the target distributions and computed with three Gaussian kernel parameter values {0.05, 0.1, 1}. We found the magnitude of MMD gradients to be much smaller than that of the ELBO. Hence, we set λ MMD to 1e 5 . Given that our models are light, the training over instrument pairs or triplets can be done in less than 24 hours on a single mid-range GPU (eg. NVIDIA TITAN Xp 12Gb).

First, we compare our MoVE proposal to UNIT on the one-to-one transfer task. To do so, we learn a different model for each pair of instruments. We perform incremental comparisons by ablating certain aspects of our proposal to assess their importance. First, we add concatenative conditioning of pitch and octave to UNIT (noted UNIT(GAN;C-po)). Then we add our proposed alternative MMD criterion replacing the GAN objective (UNIT(MMD;C-po)). Then, we introduce the FiLM layers leading to our MoVE proposal. The first version still features separate domain-specific encoders and decoders, so it is noted MoVE* (MMD; F-po). By further introducing domain conditioning and relying on a single VAE (as in Figure 2 ), we obtain our proposal MoVE (MMD; F-pod).

Evaluation scores. To evaluate reconstruction performances, we compute several criteria between input samples x and reconstructions x. The Root-Mean-Square Error RM SE= √ (x-x) 2 and Log-Spectral Distortion LSD= (10 * log 10 ( x 2 x2 )) 2 provide different assessments of how various models are able to reconstruct samples from the test set. Therefore, they only assess reconstruction abilities without domain transfer. To evaluate the quality of domain transfers, we compute the Maximum Mean Discrepancy (MMD) and the non-differentiable k-Nearest Neighbour (k-NN) test (Friedman & Rafsky (1983) ). Both are dissimilarity measures computed between the target data distribution and transferred samples. Hence, we evaluate test set transfers between different target domains.

Reconstruction and domain transfer. The averaged reconstruction and transfer results are presented in Table 3 , while separate evaluations for different pairs are in Annex B. As we can see, the UNIT-MMD model obtains the highest within-domain reconstruction score, while the MoVE model achieves better domain translation. Hence, it appears that the MMD increases reconstruction performance, and that the FiLM conditioning ameliorates the transfer. It also seems that relying on a single encoder and decoder for domain transfer might provide better generalization, as can be verified by looking at the relative MMD and kNN scores on the transfer task. Indeed, it seems that the modulated but separate layers approach perform worse, while the single architecture performs better on most evaluations. Audio descriptors topology. Audio descriptors are features used to compare the qualities of different sounds (Peeters et al. (2011) ). Hence, we rely on these to assess the effect of transfer, while providing a deeper understanding of its behavior. We compute the spectral flatness, centroid, roll-off and loudness on test samples reconstructed on their own domain or transferred to the other domain. Distribution and sample-specific plots for the spectral centroid are presented in Figure 3 . As we can see, the transfer produces an almost exact match of the descriptor distribution to the target domain. This shows the success in transferring multimodal distributions of auditory properties, as all the modes of the descriptors' distributions are preserved. The scatter plot also suggests that the centroid transfer is highly influenced by the loudness of the sample. This correlates to perception studies, as playing an instrument louder usually leads to a higher centroid McAdams et al. (1995) .

In order to further understand how the latent space is organized with respect to audio descriptors, we provide their spatial topology in Figure 4 . To compute this, we define a sampling grid over the latent space and decode the audio at each point to compute their descriptors. As we can see, the audio descriptors are locally very smooth. Furthermore, one key observation is that the latent space of both conditioned target domains follow the same overall topology. Animations showing the complete latent descriptor topology are available on the supporting webpage.

Latent space synthesis and performance. As the latent space provides continuous audio synthesis and that our method introduce high-level conditioned controls, we can use our proposal as a full musical synthesizer. Furthermore, as we map to 3-dimensional spaces, the user can directly interact with the space while performing timbre transfer. Furthermore, although these models are trained to transfer single instrumental notes, they still can be used to transfer a full melody recording between timbre domains. To do so, the recording is split and iteratively reconstructed by transfering each signal window to the target domain. Audio examples of applying this strategy to transfer a complete instrumental solo are also available in the supporting webpage.

Here, we evaluate the application of MoVE to perform many-to-many transfer. Given our new architecture, this simply consists in training on multiple domains at once by modulating with the appropriate domain information. This architecture allows us to train a single model for different domains and thus to perform multi-domain translation. The conditioning vector is then composed of the pitch, the octave and here the instrument of the corresponding example. This conditioning vector is then processed by an embedding to ease the FILM conditioning. Results are presented in Table 2 : we can see that the MoVE architecture is able to reconstruct and transfer multiple domains at the same time at the cost of a slight decrease in performance, even in the case of diverse domains (here Alto-Sax, Flute, Violin and French-Horn). 

5. CONCLUSION:

We introduced the Modulated Variational auto-Encoders (MoVE), which perform many-to-many domain transfer within a single architecture and without adversarial training while providing highlevel control over the generation. We effectively adapted this technique to musical timbre transfer and showed the successes of our method for audio synthesis. As our technique is generic, it could be applied to other types of data such as image or video. The architecture itself opens up a range of potential sonic applications such as playing style conditioning, transfers between acoustical and electronic instruments, and even with non-musical sound domains. Another avenue of research to be investigated is controlling the amount of transfer performed by the model. 

:

Audio examples, source code and animations available at https://github.com/anonymous124/iclr2019MoVE

Figures and Tables:

Figure 1: Different approaches to domain translation. (a) One-to-one transfer models such as UNIT are restricted to domain pairs. (b) One-to-many transfer models such as UMT require to train a different decoder for each domain. (c) Our proposed many-to-many transfer model (MoVE) allows to perform multi-domain transfer with a single encoder and decoder, while providing control over the generation with external high-level conditioning variables.

∈ {1, 2} by learning appropriate decoders x * d = D d (z). A paired VAE implements this assumption through separate domain-specific layers {e d ; d d } alternated with weight-shared ones {e ws ; d ws }. The full encoders and decoders are defined by the composition of both parts E d = e ws •e d and D d = d d •d ws .

o m o p T / g z p 2 4 9 Q f c 6 o + I f 6 B / 4 Z 0 x B b W I T k h y 7 r n 3 n J k 7 1 4 k 8 k U j L e s 0 Z U 9 M z s 3 P 5 + Y X F p e W V 1 c L a e j 0 J 0 9 j l N T f 0 w r j p s I R 7 I u A 1 K a T H m 1 H M m e 9 4 v O E M j l S + c c H j R I T B q b y K + J n P z g P R E y 6 T R H U K 2 1 G n L f t c s l L b Z 7 L v 9 I a X I 7 P t i 6 4 5 j q 9 H u 5 1 C 0 S p b e p m T w M 5 A

s T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q gZ O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 y L o H F c M Y 2 K e X l Sr l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "M z n L e N B W v H L d K 0 c w 0 H l 8 O t h u w J g = " > A A A C 1 3 i c j V H L S s N A F D 2 N r 1 p f t S 7 d B I v g q i Q i 6 L L g x m U F + 5 C 2 l C S d t k P z I p l I S y j u x K 0 / 4 F b / S P w D / Q v v j C m o R X R C k j P n 3 n N m 7 r 1 2 6 P J Y G M Z r T l t a X l l d y 6 8 X N j a 3 t n e K u 6 V G H C S R w + p O 4 A Z R y 7 Z i 5 n K f 1 Q U X L m u F E b M 8 2 2 V N e 3 w u 4 8 0 b F s U 8 8 K / E N G R d z x r 6 f M A d S x D V K 5 Y 6 g r t 9 l n Y 8 S 4 z s Q T q Z z X r F s l E x 1 N I X g Z m B M r J V C 4 o v 6 K C P A A 4 S e G D w I Q i 7 s B D T 0 4 Y J A y F x X a T E R Y S 4 i j P M U C B t Q l m M M i x i x / Q d 0 q 6 ds T 7 t p W e s 1 A 6 d 4 t I b k V L H I W k C y o s I y 9 N 0 F U + U s 2 R / 8 0 6 V p 7 z b l P 5 2 5 u U R K z A i 9 i / d P P O / O l m L w A B n q g Z O N Y W K k d U 5 m U u i u i J v r n + p S p B D S J z E f Y p H h B 2 l n P d Z V 5 p Y 1 S 5 7 a 6 n 4 m 8 q U r N w 7 W W 6 C d 3 l L G r D 5 c 5 yL o H F c M Y 2 K e X l Sr l a z U e e x j w M c 0 T x P U c U F a q i T 9 w S P e M K z d q 3 d a n f a / W e q l s s 0 e / i 2 t I c P y K m X g g = = < / l a t e x i t >Output FC layerTranspose conv.

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

y y 4 c a l g V T B S k n T a D s 3 L Z C L W 0 B 9 w 5 0 7 c + g N u 9 U f E P 9 C / 8 M 6 Y g g 9 E J y Q 5 9 9 x 7 z s y d 6 8 Y + T 4 V p v p S 0 s f G J y a n p m f L s 3 P z C o r 6 0 f J R G W e K x p h f 5 U X L i O i n z e c i a g g u f n c Q J c w L X Z 8 d u f 1 f m j y 9 Y k v I o P B S D m J 0 F T j f k H e 4 5 g q i W v n 7 e y u 2 4 x 4 d

< l a t e x i t s h a 1 _ b a s e 6 4 = " r Qf g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X Xr D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r Q f g 3 l T m F 5 J r k D I p a 8 L G U I B z Y + M = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d F l w 4 8 4 K 9 o F V J I n T d m h e T C Z i q X X r D 7 j V 3 x L / Q P / C O + M U 1 C I 6 I c m Z c + 8 5 M / d e P w 1 5 J h 3 n t W D N z M 7 N L x Q X S 0 v L K 6 t r 5 f W N Z p b k I m C N I A k T 0 f a 9 j I U 8 Z g 3 J Z c j a q W B e 5 I e s 5 Q +

b o 5 3 d U s a s P t z n N O g u V d 1 n a p 7 u l + p 1 c y o i 9 j C N n Z p n g e o 4 R h 1 N M g 7 x i O e 8 G y d W L l 1 Z 9 1 / p l o F o 9 n E t 2 U 9 f A D H k 5 O b < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S k e H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e oo U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s BM a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S ke H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0 x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e o o U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R L i I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S k e H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e oo U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " j 5 t W u n e s I s B M a R Li I H T w o n n x L H 8 = " > A A A C 3 H i c j V G 5 T s N A E H 0 x V w i X g Y K C x i J C C k 1 k J 4 R A h 0 R D C R I 5 J B I i 2 2 z C K r 5 k r x E o S ke H a P k B W v g e x B / A X z C 7 O B I U C N a y P f P m v d k 5 n M j j i T D N t 5 w 2 N T 0 z O 5 e f L y w s L i 2 v 6 K t r z S R M Y 5 c 1 3 N A L 4 7 Z j J 8 z j A W s I L j z W j m J m + 4 7 H W s 7 w S M Z b 1 y x O e B i c i d u I d X 1 7 E P A + d 2 1 B U E / f 6 P h p x + G D U s e 3 x Z X T H 9 2 M p b v T 0 4 t m 2 V T H M M u 1 6 p 5 V r 5 F R P 6 h W a j X D y k J F Z O c k 1 F / R w S V C u E j h g y G A I N u D j Y S e c 1 g w E R H W x Y i w m C y u 4 g x j F E i b E o s R w y Z 0 S N 8 B e e c Z G p A v c y Z K 7 d I t H r 0 x K Q 1 s k y Y k X k y 2 v M 1 Q 8 V R l l u h v u U c q p 6 z t l v 5 O l s s n V O C K 0 L 9 0 E + Z / d b I X g T 7 2 V Q + c e o o U I r t z s y y p m o q s 3 P j W l a A M E W H S v q R 4 T L a r l J M 5 G 0 q T q N 7 l b G 0 V f 1 d M i U r f z b g p P m S V t O D J F o 3 f j W a l b J l l 6 3 S 3 e L i f r T q P T W y h R P u s 4 x D H O E F D 1 f + E Z 7 x o F 9 q d d q 8 9 f F G 1 X K Z Z x 4 + j P X 4 C M P i Z K w = = < / l a t e x i t > ⌃ x < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

l W x m 9 W m + F C n p A G P p 2 j + D k 7 K J Z u m f l g p N m r Z q P P Y w C a 2 a Z 5 7 a O A A T b T I + x Z P e M a L w Y w 7 4 9 5 4 + C o 1 c p l m D T + W 8 f g J l u m a f Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

l W x m 9 W m + F C n p A G P p 2 j + D k 7 K J Z u m f l g p N m r Z q P P Y w C a 2 a Z 5 7 a O A A T b T I + x Z P e M a L w Y w 7 4 9 5 4 + C o 1 c p l m D T + W 8 f g J l u m a f Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T D F z 8 z B + n a a X m U I e M N E L 8 S N / T u A = " > A A A C 3 3 i c j V H L S s N A F D 2 N r / q u u t N N s A i 6 K U l p t d 0 V 3 L i s a F W w I p M 4 b Y f m R T I R S y m 4 c y d u / Q G 3 + j f i H + h f e G d M Q R e i E 5 K c e + 4 9 Z + b O d S J P J N K y 3 n L G x O T U 9 E x + d m 5 + Y X F p u b C y e p K E a e z y l h t 6 Y X z m s I R 7 I u A t K a T H z 6 K Y M 9 / x + K n T 3 1 f 5 0 2 s e J y I M j u U g 4 h c + 6 w a i I 1 w m i b o s r L e P R N d n b U d 0 t 9 s + k z 2 n M 7 w Z q X D n s l C 0 S r W K X a 3 W T a t k V 3 b r 9 T

Figure 2: The Modulated Variational auto-Encoder (MoVE) provides a single architecture able to perform many-to-many transfer while controlling the generation with external parameters. Both the domain and control information are processed to modulate different layers of the architecture.

Figure 3: Understanding the effect of musical timbre transfer through audio descriptor distributions.

Figure 4: Topology of the latent space with respect to audio descriptors.

Evaluations of various models on the test sets

Table content: <table><tr><td/><td colspan="2">reconstructions</td><td/><td colspan="2">transfers</td></tr><tr><td>RMSE</td><td>LSD</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td></tr><tr><td colspan="2">UNIT (GAN) 0.3412 718.47</td><td>2.117e-2</td><td>57269</td><td>2.038 e-2</td><td>43180</td></tr><tr><td colspan="2">UNIT (GAN; C-po) 0.3011 693.22</td><td>1.989 e-2</td><td>57806</td><td>9.112 e-2</td><td>43414</td></tr><tr><td colspan="2">UNIT (MMD; C-po) 0.3036 692.41</td><td>2.125 e-2</td><td>57102</td><td>2.304 e-2</td><td>43878</td></tr><tr><td colspan="2">MoVE* (MMD; F-po) 0.3134 762.51</td><td>9.632 e-3</td><td>57273</td><td>3.153 e-2</td><td>43443</td></tr><tr><td colspan="2">MoVE (MMD; F-pod) 0.3339 781.11</td><td>2.587 e-3</td><td>57509</td><td>1.747 e-2</td><td>43173</td></tr></table>

Many-to-Many MoVE reconstruction & transfer scores

Table content: <table><tr><td/><td colspan="3">averaged reconstructions</td><td colspan="2">averaged transfers</td></tr><tr><td>RMSE</td><td>LSD</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td><td>MMD (α = 0.05)</td><td>k-NN (k = 10)</td></tr><tr><td colspan="2">Alto-Saxophone 0.5327 835.67</td><td>2.117 e-2</td><td>42299</td><td>2.386 e-2</td><td>59157</td></tr><tr><td colspan="2">Flute 0.4593 761.46</td><td>2.119 e-2</td><td>49719</td><td>1.975 e-2</td><td>57277</td></tr><tr><td colspan="2">Violin 0.3271 773.65</td><td>5.659 e-3</td><td>58013</td><td>1.452 e-2</td><td>55379</td></tr><tr><td colspan="2">French-Horn 0.6239 869.69</td><td>3.404 e-3</td><td>70946</td><td>2.086 e-2</td><td>51317</td></tr></table>

Pair-wise evaluations on Clarinet and Piano test sets

Table content: <table><tr><td/><td/><td colspan="2">reconstructions Clarinet</td><td/><td colspan="2">transfers to Piano</td></tr><tr><td/><td>RMSE.</td><td>LSD.</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td></tr><tr><td>UNIT (GAN; C-po)</td><td>0.6915</td><td>778.33</td><td>3.918 e-2</td><td>76035</td><td>2.827 e-2</td></tr><tr><td>MoVE* (MMD; F-po)</td><td>0.7351</td><td>916.88</td><td>1.144 e-2</td><td>75440</td><td>1.287 e-2</td></tr><tr><td>MoVE (MMD; F-pod)</td><td>0.6878</td><td>884.63</td><td>1.614 e-2</td><td>75604</td><td>1.462e-2</td></tr><tr><td/><td/><td colspan="2">reconstructions Piano</td><td/><td colspan="2">transfers to Clarinet</td></tr><tr><td colspan="3">UNIT (GAN; C-po) 6.682 e-2 543.09</td><td>2.610 e-2</td><td>70925</td><td>4.071 e-2</td></tr><tr><td colspan="3">MoVE* (MMD; F-po) 6.522 e-2 582.98</td><td>1.276 e-2</td><td>70055</td><td>6.790 e-3</td></tr><tr><td colspan="3">MoVE (MMD; F-pod) 5.727 e-2 578.66</td><td>1.547 e-2</td><td>70044</td><td>1.884 e-2</td></tr></table>

many-to-many MoVE evaluation on Clarinet Cello, Tenor-Trombone (T-Trombone) and Piano test sets (350 epochs only)

Table content: <table><tr><td>input</td><td/><td colspan="2">reconstructions</td><td/><td>targets</td><td colspan="2">transfers</td></tr><tr><td/><td>RMSE.</td><td>LSD.</td><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td><td/><td>MMD. (α = 0.05)</td><td>k-NN. (k = 10)</td></tr><tr><td/><td/><td/><td/><td/><td>Cello</td><td>1.205 e-2</td></tr><tr><td>Clarinet</td><td>0.6832</td><td>873.17</td><td>9.776 e-3</td><td>76093</td><td>T-Trombone</td><td>7.450 e-3</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1233</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>1.345 e-3</td></tr><tr><td>Cello</td><td>0.3043</td><td>755.58</td><td>2.115 e-3</td><td>61756</td><td>T-Trombone</td><td>1.326 e-3</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1595</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>2.089 e-2</td></tr><tr><td>T-Trombone</td><td>0.7362</td><td>1047.1</td><td>1.281 e-2</td><td>25629</td><td>Cello</td><td>1.652 e-2</td></tr><tr><td/><td/><td/><td/><td/><td>Piano</td><td>0.1649</td></tr><tr><td/><td/><td/><td/><td/><td>Clarinet</td><td>0.1482</td></tr><tr><td>Piano</td><td colspan="2">4.965 e-2 542.72</td><td>9.773 e-3</td><td>69573</td><td>Cello</td><td>0.1155</td></tr><tr><td/><td/><td/><td/><td/><td>T-Trombone</td><td>0.2038</td></tr></table>

Peer review (iclr2019):
 Summary ------- This paper describes a model for musical timbre transfer which builds on recent developments in domain- and style transfer. The proposed method is designed to be many-to-many, and uses a single pair of encoders and decoders with additional conditioning inputs to select the source and target domains (timbres). The method is evaluated on a collection of individual note-level recordings from 12 instruments, grouped into four families which are used as domains. The method is compared against the UNIT model under a variety of training conditions, and evaluated for within-domain reconstruction and transfer accuracy as measured by maximum mean discrepancy. The proposed model seems to improve on the transfer accuracy, with a slight hit to reconstruction accuracy. Qualitative investigation demonstrates that the learned representation can approximate several coarse spectral descriptors of the target domains. High-level comments ------------------- Overall, this paper is well written, and the various design choices seem well-motivated. The empirical comparisons to UNIT are reasonably thorough, though I would have preferred more in-depth evaluation of the MoVE model as well. Specifically, the authors introduced an extra input (control) to encode the pitch class and octave information during encoding. I infer that this was necessary to achieve good performance, but it would be instructive to see the results without this additional input, since it does in a sense constitute a form of supervision, and therefore limits the types of training data which can be used. While I understand that quantifying performance in this application is difficult, I do find the results difficult to interpret. Some of this comes down to incomplete definition of the metrics (see detailed comments below). However, the more pressing issue is that evaluation is done either sample-wise within-domain (reconstruction), or distribution-wise across domains (transfer). The transfer metrics (MMD and kNN) are opaque to the reader: for instance, in table 1, is a knn score of 43173 qualitatively different than 43180? What is the criteria for bolding here? It would be helpful if these scores could be calibrated in some way, e.g., with reference to MMD/KNN scores of random partitions of the target domain samples. Since the authors do additional information here for each sample (notes), it would be possible to pair generated and real examples by instrument and note, rather than (in addition to) unsupervised, feature-space pairing by MMD. This could provide a slightly stronger version of the comparison in Figure 3, which shows that the overall distribution of spectral centroids is approximated by transfer, but does not demonstrate per-sample correspondence. Detailed comments ----------------- At several points in the manuscript, the authors refer to "invertible" representations (e.g., page 4, just after eq. 1), but it seems like what they mean is approximately invertible or decodable. It would be better if the authors were a little more careful in their use of terminology here. In the definition of the RBF kernel (page 4), why is there a summation? What does this index? How are the kernel bandwidths defined? How exactly are reconstruction errors calculated: using the NSGT magnitude representation, or after resynthesis in the time domain?