<figure><img src="image_1.png" /></figure>

HAL
open science

# Translation-Equivalent Self-Supervised Learning for Pitch Estimation with Optimal Transport

Bernardo Torres, Alain Riou, Gaël Richard, Geoffroy Peeters

<figure><img src="image_2.png" /></figure>

## To cite this version:

Bernardo Torres, Alain Riou, Gaël Richard, Geoffroy Peeters. Translation-Equivalent Self-Supervised Learning for Pitch Estimation with Optimal Transport. Extended Abstracts for the Late-Breaking Demo Session of the 26th International Society for Music Information Retrieval Conference (ISMIR), Sep 2025, Daejon, South Korea. hal-05321597

HAL Id: hal-05321597

https://hal.science/hal-05321597v1

Submitted on 19 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# TRANSLATION-EQUIVARIANT SELF-SUPERVISED LEARNING FOR PITCH ESTIMATION WITH OPTIMAL TRANSPORT

Bernardo Torres*

Alain Riou*

Gaël Richard

Geoffroy Peeters

LTCI, Télécom-Paris, Institut Polytechnique de Paris, France

*Equal contribution, correspondence to bernardo.torres@telecom-paris.fr

## ABSTRACT

In this paper, we propose an Optimal Transport objective for learning one-dimensional translation-equivariant systems and demonstrate its applicability to single pitch estimation. Our method provides a theoretically grounded, more numerically stable, and simpler alternative for training state-of-the-art self-supervised pitch estimators.

## 1. INTRODUCTION

Pitch estimation is a core task in audio analysis, long studied in the speech and Music Information Retrieval (MIR) communities [1]. It involves estimating the fundamental frequency of harmonic or quasi-harmonic signals, with traditional methods relying on signal processing techniques to extract harmonicity cues [2–4], or by matching the input spectrum to that of a synthetic waveform [5].

Recently, supervised deep learning approaches leveraging large annotated datasets (such as CREPE [6]) have achieved impressive accuracy, but come with notable challenges. In particular, labeling audio with the temporal precision needed for training (typically within a few milliseconds) is labor-intensive and prone to errors. While synthetic data can help alleviate this [7], supervised models remain sensitive to out-of-distribution scenarios—e.g., when trained on music and evaluated on speech [8].

To address these limitations, self-supervised learning has emerged as an efficient and scalable alternative [9–11]. To create pitch information without any labels, a pair of audio frames are pitch-shifted artificially by a known shift k and a Siamese network [12] is trained to be equivariant to such pitch-shift by optimizing a criterion parametrized by k. In practice, a Constant-Q Transform (CQT) [13] is used as a frontend to the network since it maps frequencies to a log scale, where the frequency difference between harmonic components is independent of the fundamental frequency. As a result, pitch shifting operation for harmonic sounds is roughly equivalent to a simple translation.

<figure><img src="image_4.png" /></figure>

© B. Torres, A. Riou, G. Richard, and G. Peeters. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). **Attribution:** B. Torres, A. Riou, G. Richard, and G. Peeters., "Translation-Equivalent Self-Supervised Learning for Pitch Estimation with Optimal Transport", in *Extended Abstracts for the Late-Breaking Demo Session of the 26th Int. Society for Music Information Retrieval Conf.*, Daejeon, South Korea, 2025.

This technique was originally proposed in SPICE [9], in which the pitch estimator is trained by minimizing an equivariance objective between scalar pitch predictions. Later, PESTO [10, 14] proposed an alternative equivariance criterion that operates on estimated pitch distributions, while also taking advantage of a lightweight transposition-equivariant architecture.

Moreover, equivariance to translations is not restricted to single pitch estimation, and further works leveraged similar principles for tasks such as multi-pitch [11], tempo [15, 16], tonality [17] and key [18] estimation.

In this paper, we simplify the training of PESTO by introducing an Optimal Transport (OT) [19] inspired objective between pitch distributions. OT provides a framework to compare distributions taking into account their horizontal displacement, and it has been used audio for frequency-domain comparison of audio signals [20–22]. We evaluate our proposed objective in the same cross-evaluation framework as [14] and show it has encouraging results.

## 2. OPTIMAL TRANSPORT

Let $\mu = \sum_{i=1}^{n} a_i \delta_{p_i}$ and $\nu = \sum_{j=1}^{m} b_j \delta_{q_j}$ be discrete distributions with weights $a_i$ at positions $p_i$ and weights $b_j$ at positions $q_j$, where $\delta_p$ is the Dirac measure at location $p$. Let $(a, b)$ belong to the space of probability vectors, i.e. $a \in \Sigma_n$ and $b \in \Sigma_m$, for $\Sigma_n = \{a \in \mathbb{R}_+^n; \sum_{i=1}^{n} a_i = 1\}$.

Given a cost function $c(p_i, q_j)$ representing the cost of transporting a unit of mass from locations $p_i$ to $q_j$, the OT problem seeks to find the optimal path of moving $\mu$ to $\nu$ such that the total transportation cost $\mathcal{L}_c(\mu, \nu)$, for ground cost function $c$, is minimal. When $c(p_i, q_j) = |p_i - q_j|^p$, $\mathcal{L}_c(\mu, \nu)^{1/p}$ is called the p-Wasserstein distance ($W_p$).

**Closed form for 1D Wasserstein distances:** Let $F_\mu: \mathbb{R} \to [0, 1]$ denote the cumulative distribution function (CDF) of $\mu: F_\mu(t) = \sum_{i=1}^{n} u(t - p_i)a_i$, where $u(t)$ is the step function. Let additionally $F_\mu^{-1}: [0, 1] \to \mathbb{R}$ be its pseudoinverse (the quantile function). For one dimensional distributions, $W_p$ can be conveniently expressed in closed form by integrating $|F_\mu^{-1}(r) - F_\nu^{-1}(r)|^p$ on the real line [23], and can be approximated by a discrete sum [24]:

$$W_p(\mu, \nu) = \sum_{i=1}^{n} |F_{\mu}^{-1}(r_i) - F_{\nu}^{-1}(r_i)|^p (r_i - r_{i-1}). \quad (1)$$
**Wasserstein distance under translations:** A key property of the 2-Wasserstein distance is its simple behavior under translations. For a distribution $\mu$ and its version $\mu_k$ translated by $k$, it follows that $W_2(\mu, \mu_k) = |k|$ [23].

This property is highly relevant to audio signal processing. As shown in [20], if we consider the Power Spectral Densities ($S_1, S_2$) of two signals where one is a frequency-modulated version of the other ($s_2(t) = s_1(t)e^{2i\pi\xi_0 t}$), $W_2(S_1, S_2) = |\xi_0|$ (it measures precisely the frequency shift $\xi_0$). This property inspired Spectral Optimal Transport (SOT) [22] to employ its discrete sum approximation (Eq. 1) as a loss function on magnitude spectrograms.

However, this does not directly apply to musical pitch shifting (i.e., transposition) when using a standard Fourier linear frequency representation. By working in the CQT domain instead, the frequency coordinates are now logarithmically spaced, and $W_2$ becomes a convex measure of the pitch shift (in semitones).

### 3. SELF-SUPERVISED EQUIVARIANCE LOSS BASED ON OPTIMAL TRANSPORT

We base this work on the improved version of PESTO [14], in which the Variable-Q Transform (VQT) is used as the frontend for improved time localization. The encoder $\mathcal{F}$ is trained by optimizing a combination of three losses that favor equivariance to pitch-shifting and invariance to pitch-preserving transforms (noise, gain...), respectively [10]:

$$ \mathcal{L}_{\text{PESTO}} = \underbrace{\lambda_{\text{equiv}} \mathcal{L}_{\text{equiv}} + \lambda_{\text{SCE}} \mathcal{L}_{\text{SCE}}}_{\text{equivariance}} + \underbrace{\lambda_{\text{inv}} \mathcal{L}_{\text{inv}}}_{\text{invariance}}, \quad (2) $$

where the $\lambda_*$ are dynamically updated based on losses' respective gradients, as detailed in [14].

From pitch-shifted VQT frames $\tilde{x}$ and $\tilde{x}^{(k)}$, the equivariance loss terms are computed between $\tilde{y} = \mathcal{F}(\tilde{x})$ and $\tilde{y}^{(k)} = \mathcal{F}(\tilde{x}^{(k)})$, $k$ being the translation/pitch-shift between $\tilde{x}$ and $\tilde{x}^{(k)}$. $\mathcal{L}_{\text{equiv}}$ projects both distributions to scalar values using a vector of power series $(\alpha, \alpha^2, \dots, \alpha^n)^{\top}$ and compare the ratio of these scalars with $\alpha^k$. In addition, $\mathcal{L}_{\text{SCE}}$ forces $\tilde{y}$ and $\tilde{y}^{(k)}$ to match up to a translation/pitch-shift of $k$ bins by minimizing their cross-entropy.

Here, we propose replacing the equivariance terms of Eq. 2 by the 1D Wasserstein distance (Eq. 1, as implemented by [22]) between $\tilde{y}$ and $\tilde{y}^{(k)}$ (corrected by $k$). Let $\tau_k: \Sigma^n \to \Sigma^n$ denote the translation operator which shifts distribution $y$ by $k$ bins to the right$^1$. We train the model to minimize:

$$ \mathcal{L}_{\text{OT}}(\tilde{y}, \tilde{y}^{(k)}, k) = W_2(\tilde{y}, \tau_{-k}(\tilde{y}^{(k)})). \quad (3) $$

Contrary to $\mathcal{L}_{\text{equiv}}$ and $\mathcal{L}_{\text{SCE}}$, this loss consists of a single term and has several desirable theoretical properties:

* **Symmetry:** $\mathcal{L}_{\text{OT}}(y_1, y_2, k) = \mathcal{L}_{\text{OT}}(y_2, y_1, -k)$.

* **Invariance:** $W_p(y_1, y_2) = W_p(\tau_k(y_1), \tau_k(y_2))$.

<table><caption>**Table 1.** Comparison of our model with the PESTO model [14] on different datasets. Rows and columns correspond to training and evaluation sets, respectively.</caption><thead><tr><th rowspan="2"></th><th colspan="2">MIR-1K</th><th colspan="2">MDB</th><th colspan="2">PTDB</th></tr><tr><th>RPA</th><th>RCA</th><th>RPA</th><th>RCA</th><th>RPA</th><th>RCA</th></tr></thead><tbody><tr><td colspan="7"><b>PESTO [14]</b></td></tr><tr><td>MIR-1K</td><td>97.7</td><td>98.0</td><td>94.8</td><td>95.9</td><td>87.7</td><td>90.3</td></tr><tr><td>MDB</td><td>94.6</td><td>96.1</td><td>97.0</td><td>97.1</td><td>88.3</td><td>89.9</td></tr><tr><td>PTDB</td><td>95.6</td><td>96.9</td><td>96.3</td><td>96.6</td><td>89.7</td><td>91.2</td></tr><tr><td colspan="7"><b>PESTO-OT (Ours)</b></td></tr><tr><td>MIR-1K</td><td>97.8</td><td>98.1</td><td>86.6</td><td>95.1</td><td>88.0</td><td>90.1</td></tr><tr><td>MDB</td><td>91.6</td><td>94.0</td><td>95.3</td><td>95.6</td><td>88.3</td><td>89.8</td></tr><tr><td>PTDB</td><td>86.4</td><td>92.9</td><td>93.7</td><td>94.5</td><td>89.0</td><td>90.8</td></tr></tbody></table>

* **Linear scaling under $\tau_k$:** $W_2(y, \tau_k(y)) \propto |k|$.

* **Stability:** $\mathcal{L}_{\text{OT}}$ avoids the large floating-point powers $\alpha^i$ that can cause numerical instability in $\mathcal{L}_{\text{equiv}}$.

These properties are moderated in practice by the fidelity of the pitch-shifting approximation (i.e., VQT cropping) and any boundary artifacts introduced by the shifting operation. The full objective becomes:

$$ \mathcal{L}_{\text{PESTO-OT}} = \lambda_{\text{OT}} \mathcal{L}_{\text{OT}} + \lambda_{\text{inv}} \mathcal{L}_{\text{inv}}, \quad (4) $$

with $\lambda_{\text{OT}}$ and $\lambda_{\text{inv}}$ updated as in [14], based on gradients.

### 4. EXPERIMENTS

Our evaluation protocol strictly follows the one from [14]. We use the same architecture, hyperparameters and training schedule as the configuration with VQT parameter $\gamma = 7$. We train and evaluate our model on three datasets, covering singing voice (MIR-1K [25]), music (MDB-stem-synth [7]) and speech (PTDB [26]). We refer to [14] for further details about the protocol and datasets.

We report in Table 1 cross evaluation Raw Pitch Accuracy (RPA) and Raw Chroma Accuracy (RCA) values for the considered datasets. Despite no hyperparameter tuning, our method achieves competitive performances with the state-of-the-art PESTO baseline, especially when training on MIR-1K. The biggest gap is observed when evaluating on MDB, which spans a wider pitch range.

These encouraging results, combined with the properties outlined in Section 3, suggest the relevance of OT-based loss functions for equivariant SSL.

### 5. CONCLUSION

In this work, we introduce a novel loss based on Optimal Transport for enforcing translation equivariance. We show promising results on the task of self-supervised pitch estimation, with performances close to state-of-the-art.

Furthermore, equivariance to translations is at the core of various MIR tasks such as self-supervised tempo [16], key [17, 18], chord or multi-pitch [11] estimation. We therefore believe that our contribution may have useful applications beyond the scope of this paper. In particular, variants such as Circular OT [27] may be of particular interest for tasks such as key and chord estimation.

<footnote n="1">¹ In practice, $\tilde{y}$ and $\tilde{y}^{(k)}$ are padded with $k_{\text{max}}$ zeros on both sides before applying the loss, then $\tau_k$ is implemented as a circular shift.</footnote>
# 6. ACKNOWLEDGMENTS

This work was funded by the European Union (ERC, HI-Audio, 101052978). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

# 7. REFERENCES

[1] A. M. Noll, "Cepstrum pitch determination." *The Journal of the Acoustical Society of America*, vol. 41 2, pp. 293-309, 1967.

[2] A. de Cheveigné and H. Kawahara, "YIN, a fundamental frequency estimator for speech and music," *The Journal of the Acoustical Society of America*, vol. 111, no. 4, pp. 1917-1930, 2002.

[3] M. Mauch and S. Dixon, "PYIN: A fundamental frequency estimator using probabilistic threshold distributions," in *ICASSP*, vol. 1, 2014, pp. 659–663.

[4] B. S. Lee and D. P. W. Ellis, "Noise robust pitch tracking by subband autocorrelation classification," in *INTERSPEECH*, 2012, pp. 707–710.

[5] A. Camacho and J. G. Harris, "A sawtooth waveform inspired pitch estimator for speech and music," *The Journal of the Acoustical Society of America*, vol. 124, no. 3, pp. 1638–1652, 2008.

[6] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, "Crepe: A Convolutional Representation for Pitch Estimation," in *ICASSP*, 2018, pp. 161–165.

[7] J. Salamon, R. M. Bittner, J. Bonada, J. J. Bosch, E. Gómez, and J. P. Bello, "An analysis/synthesis framework for automatic f0 annotation of multitrack datasets," *ISMIR*, pp. 71–78, 2017.

[8] M. Morrison, C. Hsieh, N. Pruyne, and B. Pardo, "Cross-domain Neural Pitch and Periodicity Estimation," *arXiv preprint arXiv:2301.12258*, 2023.

[9] B. Gfeller, C. Frank, D. Roblek, M. Sharifi, M. Tagliasacchi, and M. Velimirovic, "SPICE: Self-Supervised Pitch Estimation," *TASLP*, vol. 28, pp. 1118–1128, oct 2020.

[10] A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, "PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objective," in *ISMIR*, 2023.

[11] F. Cwitkowitz and Z. Duan, "Toward Fully Self-Supervised Multi-Pitch Estimation," *arXiv preprint arXiv:2402.15569*, 2024.

[12] R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping," *CVPR*, vol. 2, pp. 1735–1742, 2006.

[13] J. C. Brown, "Calculation of a constant Q spectral transform," *Journal of the Acoustical Society of America*, vol. 89, no. 1, pp. 425-434, 1991.

[14] A. Riou, B. Torres, B. Hayes, S. Lattner, G. Hadjeres, G. Richard, and G. Peeters, "PESTO: Real-time pitch estimation with self-supervised transposition-equivariant objective," 2025.

[15] E. Quinton, "Equivariant Self-Supervision for Musical Tempo Estimation," in *ISMIR*, 2022.

[16] A. Gagneré, S. Essid, and G. Peeters, "Adapting Pitch-Based Self Supervised Learning Models for Tempo Estimation," in *ICASSP*, 2024, pp. 956–960.

[17] Y. Kong, V. Lostanlen, G. Meseguer-Brocal, S. Wong, M. Lagrange, and R. Hennequin, "STONE: Self-supervised tonality estimator," in *ISMIR*, 2024, pp. 954–961.

[18] Y. Kong, G. Meseguer-Brocal, V. Lostanlen, M. Lagrange, and R. Hennequin, "S-KEY: Self-supervised learning of major and minor keys from audio," *ICASSP*, pp. 1–5, 2025.

[19] C. Villani et al., *Optimal Transport: Old and New.* Springer, 2009, vol. 338.

[20] E. Cazelles, A. Robert, and F. Tobar, "The Wasserstein-Fourier distance for stationary time series," *TASLP*, vol. 69, pp. 709–721, 2020.

[21] R. Flamary, C. Févotte, N. Courty, and V. Emiya, "Optimal spectral transportation with application to music transcription," *NeurIPS*, vol. 29, 2016.

[22] B. Torres, G. Peeters, and G. Richard, "Unsupervised harmonic parameter estimation using differentiable DSP and spectral optimal transport," in *ICASSP*, 2024, pp. 1176–1180.

[23] G. Peyré and M. Cuturi, "Computational optimal transport," *Foundations and Trends in Machine Learning*, vol. 11, no. 5-6, pp. 355–607, 2019.

[24] R. Flamary et al., "POT: Python optimal transport," *Journal of Machine Learning Research*, vol. 22, no. 78, pp. 1–8, 2021.

[25] C.-L. Hsu and J.-S. R. Jang, "On the Improvement of Singing Voice Separation for Monaural Recordings Using the MIR-1K Dataset," *TASLP*, vol. 18, no. 2, pp. 310–319, 2010.

[26] G. Pirker, M. Wohlmayr, S. Petrik, and F. Pernkopf, "A pitch tracking corpus with evaluation on multipitch tracking scenario," in *INTERSPEECH*, 2011, pp. 1509–1512.

[27] J. Rabin, J. Delon, and Y. Gousseau, "Transportation distances on the circle," *Journal of Mathematical Imaging and Vision*, vol. 41, no. 1-2, pp. 147–167, 2011.