Unsupervised Disentanglement of Pitch and Timbre for Isolated Musical Instrument Sounds
Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, Dorien Herremans
SummaryTackle unsupervised disentanglement of pitch and timbre
Leverage pitch-shifting to further improve disentanglement
Design a quantitative metric that accounts for disentanglement

ModelIdea: Introduce inductive biases through architectural constraints
Generation
Model a note of musical instruments x
as being generated by- a pitch (discrete c) and- a timbre (continuous z)
latent variablepθ(x, z, c) = pθ(x | z, c)p(z)p(c)- p(c) = U(0, 1)- p(z) = 𝒩(0, 1)- pθ(x | z, c) = 𝒩(μθ(z, c), 1), decoder (D)Inference
Follow the framework of variational inference, introducing a factorized
approximated posterior to approximate the true posterior
Approximated posterior qϕ(z, c | x) = qϕ(z | x)qϕ(c | x)
- qϕ(c | x) = Cat(c | πϕ(x)), pitch encoder- qϕ(z | x) = 𝒩(μϕ(x), diag(σϕ2(x))), timbre encoder

Learning
Reparameterization tricks allow for stochastic gradient descent- Gaussian for z [Kingma et al., ICLR 2014]- Hard Gumbel-softmax for c (one-hot vectors) [Jang et al., ICLR 2017]
Maximize Evidence Lower BOund (ELBO)ℒELBO = 𝔼qϕ(z,c|x)[log pθ(x | z, c)]−DKL(qϕ(z, c | x)∥pθ(z, c))Parameters
Number of Mel-frequency bins F = 256
Dimension of timbre latent variable L = 8
Number of categories for pitch latent variable K = 82

Auxiliary Losses

DatasetAssumption: Moderate pitch-shiftings ps( ⋅ ) do not change timbreStudio-On-Line [Ballet et al., JIM 1999]
1,885 samples of 12 musical instruments and 82 pitches
waveform (22,050Hz) → STFT (w = 92ms, h = 11ms) → Mel-Spec(F = 256) → log-scaled → normalized to [−1,1] → x (200-th frame)xps( ⋅ )

zx̂

zswap

cxswapcswap

Quantitative Resultsx′(x*, x′*)(z*, z′*)z′x′̂

z′swapc′x′swap

c′swapCreate pseudo data pairs (x, x′) where x′ denotes x pitch-shifted by δℒregression = ∥z − z′∥22exp(sim(zi, z′i)/τ)[Chen et al., ICML 2020]ℒcontrast = − log∑z≠z exp(sim(zi, z)/τ)iℒcycle = ∥zswap − z∥22 + ∥z′swap − z′∥22 + CE(cswap, k′) + CE(c′swap, k),where k = arg max(c) [Zhu et al., ICCV 2017]ℒsurrogate = CE(c′, y′), where pseudo pitch label y′ = arg max(c) + δ
The final objective function to be maximized becomesℒ = ℒELBO − (λ1ℒregression + λ2ℒcontrast + λ3ℒcycle + λ4ℒsurrogate)
Evaluation
Pitch Variable
Pitch classification accuracy (ACC and pitch mapping, need labels)̂̂ )];
Consistency-Diversity Score (CDS) = 𝔼k[DKL(pk(y | x))∥𝔼k[pk(y | x)]pk(y | x)̂ = p(y | D(z, ck)) is posterior of a pre-trained pitch classifier,where the one-hot vector ck denotes c ∋ k = arg max (c)- pk(y | x)̂ should be consistent and have low entropy given a fixed ck
̂ should be as uniformly distributed as possible- 𝔼k[pk(y | x)]

Timbre Variable
Pitch and Instrument classification accuracy (need labels)Fréchet Inception Distance (FID) [Heusel et al., NeurIPS 2017]- FIDrecon: FID between true and reconstructed data (upper-bound)- FIDrand: FID between true and randomly sampled data♭: Supervised model trained with pitch labels♯: Unsupervised model trained without pitch-shifting

M0 - M6: Proposed unsupervised models with different losses activatedSupervised model does not yield good generation quality (FID)
Pitch-shifting alone improves disentanglement
No auxiliary loss alone yields consistent improvement for all metrics
Activating ℒsurrogate on top of the rest reaches the best-performing
model (M5→M6)

Qualitative Results
Perform pitch-conditioning spectrum generation- Last row: seeds (three seeds per model)- First to third rows: three different k′s
Spectral distribution stays consistent per column
Spectrums generated given a k are expected tohave a consistent pitch (consistency)
Different k′s render different pitches (diversity)

Future Works
Perform pitch mapping without referring to pitch labels
Trade off between capacity and constraint for pitch representation c
Model larger time scale (temporal variable)