TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer

1 INTRODUCTION . Timbre is a perceptual characteristic that distinguishes one musical instrument from another playing the same note with the same intensity and duration . Modeling timbre is very hard , and it has been referred to as “ the psychoacoustician ’ s multidimensional waste-basket category for everything that can not be labeled pitch or loudness ” 2 . The timbre of a single note at a single pitch has a nonlinear dependence on the volume , time and even the particular way the instrument is played by the performer . While there is a substantial body of research in timbre modelling and synthesis ( Chowning ( 1973 ) ; Risset and Wessel ( 1999 ) ; Smith ( 2010 ; 2011 ) ) , state-of-the-art musical sound libraries used by orchestral composers for analog instruments ( e.g . the Vienna Symphonic Library ( GmbH , 2018 ) ) are still obtained by extremely careful audio sampling of real instrument recordings . Being able to model and manipulate timbre electronically carries importance for musicians who wish to experiment with different sounds , or compose for multiple instruments . ( Appendix A discusses the components of music in more detail . ) In this paper , we consider the problem of high quality timbre transfer between audio clips obtained with different instruments . More specifically , the goal is to transform the timbre of a musical recording to match a set of reference recordings while preserving other musical content , such as pitch and loudness . We take inspiration from recent successes in style transfer for images using neural networks ( Gatys et al. , 2015 ; Johnson et al. , 2016 ; Ulyanov et al. , 2016 ; Chu et al. , 2017 ) . An appealing strategy would be to directly apply image-based style transfer techniques to time-frequency representations of images , such as short-time Fourier transform ( STFT ) spectrograms . However , needing to convert the generated spectrogram into a waveform presents a fundamental obstacle , since accurate reconstruction requires phase information , which is difficult to predict ( Engel et al. , 2017 ) , and existing techniques for inferring phase ( e.g. , Griffin and Lim ( 1984 ) ) can produce characteristic artifacts which are undesirable for high quality audio generation ( Shen et al. , 2017 ) . 1Link to the demo video : www.cs.toronto.edu/~huang/TimbreTron/index.html Code available at : https : //github.com/huangsicong/TimbreTron 2McAdams and Bregman ( 1979 ) , pg 34 Recent years have seen rapid progress on audio generation methods that directly generate high-quality waveforms , such as WaveNet ( van den Oord et al. , 2016 ) , SampleRNN ( Mehri et al. , 2016 ) , and Tacotron2 ( Shen et al. , 2017 ) . WaveNet ’ s ability to condition on abstract audio representations is particularly relevant , since it enables one to perform manipulations in high-level auditory representations from which reconstruction would have previously been impractical . Tacotron2 performs high-level processing on time-frequency representations of speech , and then uses WaveNet to output high-quality audio conditioned on the generated mel spectrogram . We adapt this general strategy to the music domain . We propose TimbreTron , a pipeline that performs CQT-based timbre transfer with high-quality waveform output . It is trained only on unrelated samples of two instruments . For our time-frequency representation , we choose the constant Q transform ( CQT ) , a perceptually motivated representation of music ( Brown , 1991 ) . We show that this representation is particularly well-suited to musical timbre transfer and other manipulations due to its pitch equivariance and the way it simultaneously achieves high frequency resolution at low frequencies and high temporal resolution at high frequencies , a property that STFT lacks . TimbreTron performs timbre transfer by three steps , shown in Figure 1 . First , it computes the CQT spectrogram and treats its log-magnitude values as an image ( discarding phase information ) . Second , it performs timbre transfer in the log-CQT domain using a CycleGAN ( Zhu et al. , 2017 ) . Finally , it converts the generated log-CQT to a waveform using a conditional WaveNet synthesizer ( which implicitly must infer the missing phase information ) . Empirically , our TimbreTron can successfully perform musical timbre transfer on some instrument pairs . The generated audio samples have realistic timbre that matches the target timbre while otherwise expressing the same musical content ( e.g. , rhythm , loudness , pitch ) . We empirically verified that the use of a CQT representation is a crucial component in TimbreTron as it consistently yields qualitatively better timbre transfer than its STFT counterpart . 2 BACKGROUND . 2.1 TIME-FREQUENCY ANALYSIS . Time-frequency analysis refers to techniques that aim to measure how the signal ’ s frequency domain representation changes over time . Short Time Fourier Transform ( STFT ) The STFT is one of the most commonly applied techniques for this purpose . The discrete STFT operation can be compactly expressed as follows : STFT { x [ n ] } ( m , ωk ) = ∞∑ n=−∞ x [ n ] w [ n−m ] e−jωkn The above formula computes the STFT of an input time-domain signal x [ n ] at time step m and frequency ωk . w refers to a zero-centered window function ( such as Hann Window ) , which acts as a means of masking out the values that are away from m. Hence , the equation above can be interpreted as the discrete Fourier transform of the masked signal x [ n ] w [ n−m ] . An example spectrogram is shown in Figure 2 . Constant Q Transform ( CQT ) . The CQT ( Brown , 1991 ) is another time-frequency analysis technique in which the frequency values are geometrically spaced , with the following particular pat- tern ( Blankertz ) : ωk = 2 k b ω0 . Here , k ∈ { 1 , 2 , 3 , ... kmax } and b is a constant that determines the geometric separation between the different frequency bands . To make the filter for different frequencies adjacent to each other , the bandwidth of the kth filter is chosen as : ∆k = ωk+1 − ωk = ωk ( 2 1 b − 1 ) . This results in a constant frequency to resolution ratio ( as known as the “ quality ( Q ) factor ” ) : Q = ωk ∆k = ( 2 1 b − 1 ) −1 Huzaifah ( 2017 ) showed that CQT consistently outperformed traditional representations such as Mel-frequency cepstral coefficients ( MFCCs ) in environmental sound classification tasks using CNNs . Rainbowgram . Engel et al . ( 2017 ) introduced the rainbowgram , a visualization of the CQT which uses color to encode time derivatives of phase ; this highlights subtle timbral features which are invisible in a magnitude CQT . Examples of CQTs and rainbowgrams are shown in Figure 2 . 2.2 WAVEFORM RECONSTRUCTION FROM SPECTROGRAMS . Synthesis ( waveform reconstruction ) from the aforementioned time-frequency analysis techniques can be performed in the presence of both magnitude and phase information ( Allen and Rabiner , 1977 ) ( Holighaus et al. , 2013 ) . In the absence of phase information , one of the common methods of synthetically generating phase from STFT magnitude is the Griffin-Lim algorithm ( Griffin and Lim , 1984 ) . This algorithm works by randomly guessing the phase values , and iteratively refining them by performing STFT and inverse STFT operations until convergence , while keeping the magnitude values constant throughout the process . Developed to minimize the mean squared error between the target spectrogram and predicted spectrogram , this algorithm is shown to reduce the objective function at each iteration , while having no optimality guarantees due to the non-convexity of the optimization problem ( Griffin and Lim , 1984 ; Sturmel and Daudet ) Although recent developments in the field have enabled performing the inverse operation of CQT ( Velasco et al. , 2011 ; Fitzgerald et al. , 2006 ) , these techniques still require both phase and magnitude information . 2.3 WAVENET . WaveNet , proposed by van den Oord et al . ( 2016 ) , is an auto-regressive generative model for generating raw audio waveform with high quality . The model consists of stacks of dilated causal convolution layers with residual and skip connections . WaveNet can be easily modified to perform conditional waveform generation ; for example , it can be trained as a vocoder for synthesizing natural , high-quality human speech in TTS systems from low-level acoustic features ( e.g. , phoneme , fundamental frequency , and spectrogram ) ( Arik et al. , 2017 ; Shen et al. , 2017 ) . One limitation of WaveNet is that the generation of waveforms can be expensive , which is undesirable for training procedures that require auto-regressive generation ( e.g. , GAN training , scheduled sampling ) . 2.4 GAN AND CYCLEGAN . Generative Adversarial Networks ( GANs ) are a class of implicit generative models introduced by Goodfellow et al . ( 2014 ) . A GAN consists of a discriminator and a generator , which are trained adversarially via a two-player min-max game , where the discriminator attempts to distinguish real data from samples , and the generator attempts to fool the discriminator . The objective is : G∗ , D∗ = arg min G max D Ex∼X [ logD ( x ) ] + Ez∼Z [ log ( 1−D ( G ( z ) ) ) ] , ( 1 ) where D is the discriminator , G is the generator , z is the latent code vector sampled from Gaussian distribution Z , and x is sampled from data distribution X . GANs constituted a significant advance over previous generative models in terms of the quality of the generated samples . CycleGAN ( Zhu et al. , 2017 ) is an architecture for unsupervised domain transfer : learning a mapping between two domains without any paired data . ( Similar architectures were proposed independently by Yi et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Kim et al . ( 2017 ) . ) The CycleGAN learns two generator mappings : F : X → Y and G : Y → X ; and two discriminators : DX : X → [ 0 , 1 ] and DY : Y → [ 0 , 1 ] . The loss function of CycleGAN consists of both adversarial losses ( Eqn . 1 ) , combined with a cycle consistency constraint which forces it to preserve the structure of the input : Lcyc ( F , G , X , Y ) = Ex∼X [ ‖G ( F ( x ) ) − x‖1 ] + Ey∼Y [ ‖F ( G ( y ) ) − y‖1 ] ( 2 ) 3 MUSIC PROCESSING WITH CONSTANT-Q-TRANSFORM REPRESENTATION . This section focuses on the first and last steps of the TimbreTron pipeline : the steps related to the transforming raw waveforms to and from time frequency representations . We explain our reasoning for choosing the CQT representation and introduce our conditional WaveNet synthesizer which converts a ( possibly generated ) CQT to a high-quality audio waveform . 3.1 CQT FOR MUSIC REPRESENTATION . The CQT representation ( Brown , 1991 ) has desirable characteristics that make it especially suitable for processing musical audio signals . It uses a logarithmic representation of frequency , where the frequencies are generally chosen to exactly cover all the pitches present in the twelve tone , welltempered scale . Unlike the STFT , the CQT has higher frequency resolution towards lower frequencies , which leads to better pitch resolution for lower register instruments ( such as cello or trombone ) , and higher time resolution towards higher frequencies , which is advantageous for recovering the fine timing of rhythms . Since individual notes contain information across many frequencies ( due to their pattern of overtones ) , this combination of resolutions ought to allow simultaneous recovery of pitch and timing information for any particular note . ( While this information is preserved in the signal , waveform recovery is a difficult problem in practice ; this is discussed in Section 3.2 ) . Another key feature of the CQT representation in the context of TimbreTron is ( approximate ) pitch equivariance . Thanks to the geometric spacing of frequencies , a pitch shift corresponds ( approximately ) to a vertical translation of the “ spectral signature ” ( unique pattern of harmonics ) of musical instruments . This means that the convolution operation is approximately equivariant under pitch translation , which allows convolutional architectures to share structure between different pitches . A demonstration of this can be seen in Figure 3 . Since the harmonics of a musical instrument are approximately integer multiples of the fundamental frequency , scaling the fundamental frequency ( hence the pitch ) corresponds to a constant shift in all of the harmonics in log scale . We also want to emphasize on some of the reasons why the equivariance is only approximate : • Imperfect multiples : In real audio samples from instruments , the harmonics are only approximately integer multiples of the fundamental frequency , due to the material properties of the instruments producing the sound . • Dependence of spectral signature on pitch and beyond : For each pitch , each instrument has a slightly different spectral signature , meaning that a simple translation in the frequency axis can not completely account for the changes in the frequency spectrum . Furthermore , even at a given pitch it can still change depending on how it ’ s played . We used 16ms frame hop ( 256 time steps under 16kHz ) . More details can be found in Appendix B .

Summary: Main Idea: The authors use multiple techniques/tools to enable neural timbre transfer (converting music from one instrument to another, ex: violin to flute) without paired training examples. The authors are inspired by the success of CycleGANs for image style transfer, and by the success of Wavenet for generating realistic audio waveforms. Even without the CycleGAN, the use of CQT->WaveNet for time stretching and pitch shifting of a single piece is an interesting and valuable contribution.