Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Authors: Joan Serrà
R. Oguz Araz
Dmitry Bogdanov
Yuki Mitsufuji
Venue: ICML 2025 poster (2025)
Keywords: audio
music
versions
segments
weakly-labeled
supervised
contrastive
retrieval
Primary area: applications->everything_else
TL;DR: We propose a method to deal with weakly labeled segments and a contrastive loss, both for musical version matching

Abstract: Detecting musical versions (different renditions of the same piece) is a challenging task with important applications. Because of the ground truth nature, existing approaches match musical versions at the track level (e.g., whole song). However, most applications require to match them at the segment level (e.g., 20s chunks). In addition, existing approaches resort to classification and triplet losses, disregarding more recent losses that could bring meaningful improvements. In this paper, we propose a method to learn from weakly annotated segments, together with a contrastive loss variant that outperforms well-studied alternatives. The former is based on pairwise segment distance reductions, while the latter modifies an existing loss following decoupling, hyper-parameter, and geometric considerations. With these two elements, we do not only achieve state-of-the-art results in the standard track-level evaluation, but we also obtain a breakthrough performance in a segment-level evaluation. We believe that, due to the generality of the challenges addressed here, the proposed methods may find utility in domains beyond audio or musical version matching.

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Joan Serrà R. Oguz Araz?
Dmitry Bogdanov Yuki Mitsufuji 13
Abstract
can be present between versions, which go beyond typical augmentations used by the machine learning community. Two musical versions may feature different instrumentation or timbre, together with tonality and chord modifications, altered melodies, substantial changes to rhythm and tempo, an alternate temporal development or structure, and many more (Yesiler et al., 2021). Yet, musical versions retain their essence, to the point that can generally agree whether two of them correspond to the same piece or not'. Therefore, learnt version representations need to encapsulate multiple characteristics shared between versions that, at the same time, can discriminate them from other pieces.
Detecting musical versions (different renditions of the same piece) is a challenging task with im- portant applications. Because of the ground truth nature, existing approaches match musical ver- sions at the track level (e.g., whole song). How- ever, most applications require to match them the segment level (e.g.. s0 chunks). In addition, existing approaches resort to classification and triplet losses, disregarding more recent losses that could bring meaningful improvements. this pa- per, we propose a method to learn from weakly an- notated segments, together with contrastive loss variant that outperforms well-studied alternatives. The former is based on pairwise segment distance reductions, while the latter modifies an existing loss following decoupling, hyper-parameter and geometric considerations. With these two ele- ments, we do not only achieve state-of-the-art results in the standard track-level evaluation, but we also obtain breakthrough performance in a segment- levell evaluation. We believe that, due to the generality of the challenges addressed here, the proposed methods may find utility in domains beyond audio or musical version matching.
Musical version matching has several relevant applica- tions (Serrà, 2011; Yesiler et al., 2021), including specific applications to plagiarism and near-duplicate detection Beyond business impact (Page, 2023) and cultural/artistic appreciation, some applications have become even more relevant today, given the sustained rise and improvement of music generative models (e.g., Copet et al., 2023; Evans et al., 2024: Liu al., 2024). Indeed, the recent efforts on assessing music data replication, memorization, and attri- bution in such models exploit some form of music similar- ity (Barnett et al., 2024; Bralios et al., 2024) or, for improved results, musical version matching (Batle-Roca et al., 2024).
A fundamental limitation of version matching approaches is that they operate at the full-track level, learning and extract- ing individual representations from relatively long record- ings (for instance, few-minute song). This is due ground truth version annotations being only available per track. However, the segments of interest, for both classical and modern applications, are much shorter than the track length (for instance, around 10-20 This mismatch between the learning and inference stages, as we will see, causes a dra- matic performance degradation (Sec. 5). Another challenge is that, in contrast to standard supervised learning tasks, musical version data sets contain only few items per class.
1. Introduction
When two audio tracks contain different renditions of the same musical piece, they are considered musical versions'. Musical versions are inherent in human culture and predate recorded music and notation, as ancient music was transmit- ted solely through playing and listening (Ball, 2010), which naturally led to variations in tunes, rhythms, structures, etc. Learning representations of musical versions is a challeng- ing task due to the degree and amount of variations that
'Sony AI MMusic Technology Group, Universitat Pompeu Fabra 'Sony Group Corporation. Correspondence to: Joan Serrà <joan.serra@e wonn..serr@sooy..omm. com>.com>.
'A related hut restrictive and biased term in the literature is "cover songs". better understand this restriction and bias, see the discussion found in, for example, Yesiler et al. (2021).
AA well-known source collecting this information is https: // asecordaandsongs com.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Note that musical version matching may expand and subsume traditional music fingerprinting (Cano et al., 2005).

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Table Comparison of characteristics for numher of existing approaches and the proposed method CLEWW.. We exclude multi-featur and/or multi- modal approaches (for example fusing CQT and melody estimations or leveraging audio and lyrics information). For further details and approaches we refer to survey by Yesiler et al. (2021).
NAME(S)
MAIN
INPUT
ARCIL
SEGMENT LEARNING
PARTIAL MATCH
LOss / TRAIN
RETRIEVAL
REFERENCE
CONCEPT
DISTANCE
CQTNET
ET AL. (2020)
CQT
CONVNET
X
CLASSIE.
COSINE
DORAS&PPETERR,
DORAS PEETERS (2020)
HCQT
CONVNET
X
TRIPLET
MOVE/RE-MOVE
YESTLER ET AL. (2020A)
CREMA
CONVNET
X
TRIPLET
EUCLIDEAN
PICKINET
O' H'NNLNN ET AL. (2021)
CQT
CONVNET
X
X
CLASSIF:CCNTEER
COSINE
LYRACNET
HU ET AL. (2022)
CQT
WIDERESNET
X
X
CLASSIF.
COSINE
BYTECOVER1/2
DU ET AL. (2022)
CQT
RESNET
X
X
CLASSIF TRIPLET
COSINE
COVERHUNTER
LIU ET AL. (2023)
CQT
CONFORMER
X
J
CLASSIF.+FOCAL CENTER
COSINE
BYTECOVER3/3.5
Du ET AL. (2023)
CQT
RESNET
X
CLASSIF MENSSIIF+TKAE +TRIPLET
COSINE
DVINEE/DVINET+
ARAZ ET AL. (2024A)
CQT
CONVNET
X
TRIPLET
COSINE
CLEWS (PROPOSED)
THIS PAPER
CQT
RESNET
CONTRASTIVE
EUCLIDEAN
For instance, up to 56% of a recent realistic large-scale data set of around 500 tracks is formed by only 2-item classes, with an average of 5 items per class (Araz al., 2024a). This characteristic suggests that, besides the tradi- tional focus on classification and triplet losses, a supervised contrastive learning approach (Sec. 2) could also work well.
2. Background
After a history of rule-, feature-, and model-based ap- proaches (Serrà, 2011; Yesiler et al., 2021), musical ver- sion matching is currently tackled as a supervised learning problem, focusing on full-track pairwise matching (Table 1). However, two versions do not necessarily need to match for their entire duration, and actually several applications rely on few-second partial matches. Only couple of approaches base their learning or retrieval stages on segments or par- tial matches, respectively. ByteCover3 (Du et al., 2023) pioneered learning from segments with their "maxmean" operator. However, such operator still does not allow for partial matches, as it forces all segments of track to match some segment from another track. CoverHunter (Liu et al., 2023) is able to detect partial matches of around 45 s. How- ever, the learning strategy to do so is based on a two-stage brute-force approach. First, it trains a coarse detector model on 15-second segments using classification, focal, and cen- ter losses. Then, it resorts to this first-stage model and a nule-based approach to (weakly) label 45-second segments, which are finally used to train the second stage model with the same losses. In both stages, CoverHunter treats seg- ments as full tracks. To our knowledge, we are the first to consider an entirely segment-based approach for both learning and retrieval stages.
In this paper, we consider a full music track as a succession of weakly-labeled audio segments, and learn contrastive representation using such weak supervision. To do so, we introduce two main methods. First, we develop number of pairwise distance selection strategies, which reduce segment-level distance matrix into track-level distance matrix. This enables the direct utilization of track-level annotations without statically assigning them to some or all of the segments. Second, we reformulate the alignment and uniformity (A&U) loss of Wang & Isola (2020), origi- nally introduced for self- self-supervised learning, to operate (weakly) supervised learning task. Motivated by decoupling, hyper-parameter, and geometric considerations, we intro- duce several changes that convert A&U into a new loss func- tion: the strict decoupling of positives and negatives (Yeh et al., 2022), the simplification of hyper-parameters, the native operation in Euclidean geometry (cf. Koishekenov et al., 2023), and smoothing constant for negative pairs. With both distance reduction and contrastive learning strate- gies, we do not only outperform existing approaches in the segment-level evaluation by a large margin, but we also achieve state-of-the-art results in the standard track-level evaluation. We also perform an extensive ablation study to empirically compare the proposed methods with several alternatives, including additional reduction strategies and common contrastive losses. We believe that, due to the generality of the challenges addressed here, the proposed methods may find utility in further domains beyond mu- sical version matching. To facilitate understanding and reproduction, we share our code and model checkpoints in https: pttps://githin //ttbbb...tttt/ com/ sony/ clews.
The literature on musical version matching has tradition- ally considered number of classification (Sun al., 2014) and triplet (Schroff et al., 2015) loss variants, and their combination (Table 1). However, given the same ground truth, another approach to learning version representations would be to consider a supervised contrastive loss like N- pairs (Sohn, 2016) or SupCon (Khosla et al., 2021). In addi- tion, a number of wetllestablished losses for self- superuupee learning like InfoNCH/NTT-X Xent (Van den Oord et al., 2018; 88; Chen et al., 2020), alignment and uniformity (Wang & Isola, 2020), SigLIP (Zhai et al., 2023) could also be adapted. An analysis of the relations between many of such losses is
2

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
carried out by Koromilas al. (2024). Apart from the loss function, other considerations such as positive/negative de- coupling (Yeh et al., 2022) and the correspondence between distance and space geometry (Koishekenov et al., 2023) are potentially relevant in a practical case. To our mnwwledge, we are the first to consider a supervised contrastive loss for musical version matching.
D
ij
2)
23
3. Contrastive Learning from Weakly-Labeled Audio Segments
4/41
We now detail our approach to perform contrastive learning from weakly-labeled audio segments (CLEWS). The first part deals with track-level labels and their allocation to segment distances (we base our development on distances, but it can be easily reformulated using similarities). The second part details the contrastive loss function we use. The third part explains our architecture and training procedure.
K(41
2/41
Figure 1. Illustration of four reduction functions R over pair- wise segment distances di... They are depicted on different sub- rectangles D,y, where tracks i, j. and j + are versions (green squares) and track i + 1 is not (orange squares). The four func- tions correspond to: Rmeumin (top left). Ropwes (top right), Riess10 (hottom left), and R.in (bottom right). The Repur strategy de- picts its minimum/masking recursion in increasingly dark levels (green/purple cells). The sub-rectangles for Riyer and Rmin also exemplify dcaling with different lengths by masking (gray cells).
3.1. Segment Distance Reduction
Framework - Given the k-th waveform segment of the i-th music track, x we compute latent representations 2t =x{{), where FF represents a neural network that pools the time-varying information of the segment into single vector (architecture details can be found in Sec. 3.3 and Appendix A). Then, for every possible pair of segments k and 1 of every possible pair of tracks i and j. we com- pute their distance d:f and obtain the distance matrix D (Fig. 1). At this point, if there are n query tracks and m candidate tracks with u and v segments*, respectively, we have D Rauxme However, since labels are only pro- 20 vided at the track level, our binary ground truth assignments (1 for version/positive and for non- version/negaaive) are A E Znxm. Therefore, we need some strategy to (weakly) allocate n x m labels to nu X me segment distances.
and reduce nu x mv segment distances to n m track dis- tances, matching the dimensions of positive/negative assign- ments A. More specifically, consider several reduction functions R, producing D RD),, where D E R=* In addition, we employ different reduction functions for positive and negative pairs, R+ and R- respectively.
Reduction Functions - The naïve strategy outlined above would correspond a mean reduction over the entire sub- rectangle determined by D.y:
A naive strategy to do such allocation would be to propa- gate all positiveeneeative track assignments to all segment comparisons in the sub-rectangle D., defined by a pair of tracks and j. This is the approach implicitly followed by CoverHunter (Liu et al., 2023). However, besides its poor performance (Sec. 5), this strategy incurs a fundamental error, in the sense that it is teaching the model that all pos- itive segments are 'similar' to all other positive segments, which in the case of musical versions is false (two segments could reproduce two different motives of the same song: even though it is the same song, the segments are usually not the same, unless it is extremely repetitive song). In- stead of trying to allocate n x positive/negaaiie distances to nu X me segment distances, we take the opposite view
dj R.mea (Di))
Tkl
WU
IKKCH
131Se
Instead, if we just consider the r best matches across seg- ments, we have
di Rhestr (D:))
Σ topr (D:s) 1815
for S uv, where topr(D) is function that returns the lowest distances in D (Fig. 1, bottom left). Another possi- bility is to consider just the single best correspondence in the entire sub-rectangle (Fig. 1, bottom right), yielding
'Notiee that, similar cross-ates aossstattttionn in Transformers, can deal with different track lengths by taking a maximum length u, v and masking. See Fig. (right) for a couple of examples.
de Rmin (D.)) = min 7kl
dij
1 1
1813"
3

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
A further alternative is use an operator like the one used in ByteCover3 (Du al., 2023), which first searches for the candidate best match per query and then averages across all query segments. Reformulating it for distances (Fig. 1, top left), we obtain
Positives and Negatives Reduction We note that the previous distance reduction strategies have some concep- tual parallelism with the negative/positive mining strategies used with triplet losses (Schroff et al., 2015) or in some contrastive approaches (cf. Kalantidis et al., 2020). In our case, for instance, Rmin could correspond to a hard min- ing strategy, while Rhest.r or Repwe.r could be regarded as semi-hard mining of segment pairs. Thus, inspired by those strategies, we decide to study if applying different reduc- tion strategies for positives and negatives has some effect in our setup. To obtain a track-based pairwise distance matrix that combines different reductions for positive and negative pairs, we calculate
dij- = memammnnn (Dis)
1 u
Σ ISkSS
min dal ISISe 17
Notice that the Remean Reest r. and Remeammin strategies above do not explicitly prevent multiple consecutive segments of track& being assigned to the same segment of track j (ver- tical or horizontal traces in Fig. 1). As mentioned, this is unrealistic for the majority of music tracks (and in general for any signal or sequence presenting a minimal variability with time). Notice also that Rmean and meaanmin force full-track match, that is, they are teaching the model that all segments in track i should find a match in track j. Again, this is an unrealistic assumption for musical versions where the structure changes (and in general for any signal or se- quence featuring only partial matches).
D = A - (1- -1-
(2)
where @ denotes element-wist multiplication and 1 is the all- ones matrix (recall that the elements in A are 1 for positives and 0 otherwise). The reductions R+ and R can be chosen among the ones presented above.
3.2. Contrastive Loss
Best-pair Reduction Motivated by the issues of con- secutive and global segment matching above, we decide to design an additional reduction strategy that explicitly deals with both. We term it bpwr-r, for 'best pair without replace- ment' with a threshold r. In a nutshell, Repwer operates by sorting all distances in the D.y sub-rectangle in increasing order, then taking the first one (say the one involving seg- ments & and I), removing all distances computed by using either one or the other segment (either & or i), and iterating r times. It then takes the average among those r best pairwise distances. More formally, we can express as
Motivation - After computing pairwise track-level dis- tances D, we need a contrastive loss that can exploit them and that, ideally, can outperform the existing losses in the considered task. For that, one can consider any supervised contrastive loss function that operates on distances, or adapt existing self-supervised loss to the supervised frame- work (Sec. 2). In our case, we opt for the latter and choose the A&U loss of Wang & Isola (2020) due to its appeal- ing properties and intuitive derivation. One of the practical properties we value is that, by using expectations, we have a similar behavior for different batch sizes (Wang & Isola 2020; see also Koromilas et al. 2024). In our analysis, we will use the concept of "potential" as introduced by Wang & Isola (2020), but nonetheless will depart from the concept of uniformity in the hypersphere.
dij = Rapwer (Dis)) = 1 Σ
(1) Rmin Djj
(I)
15450
forr S min(u,,) v), with the recursion
5 Di
for q 1, for > 1,
Changes to Alignment and Uniformity - The A&U loss, designed for self-supervised contrastive learning, expects one positive for each item in the batch (obtained through some augmentation) while negatives correspond to all other elements in the batch. To adapt A&U to supervised con- trastive learning with multiple positives per anchor, we need to carefully define both positive and negative sets. In partic- ular, we want to preserve the decoupling of the alignment and uniformity terms as, apart from respecting the original idea of A&U, typically yields improved performance (Ych et al., 2022). Therefore, from all pairwise assignments A in the batch, we need to gather positive A and negative A assignment sets such that A+ n A- = 0. This also implies discarding comparisons of one track against itself (that is, the diagonals of D and A, and potentially other spurious cells corresponding to sampling the same track more than once in the same batch).
( 1)
maskmin
where maskmin (D is a function that masks the row and the column corresponding the minimum element in D, such that those elements are not eligible by the Rmin of Eq. 1 in iterations q > 1. A schema of R is illustrated in Fig. (top right; recursion is depicted by progressively darker colors). Notice that masking rows and columns avoids the issue of consecutive segment matching". and that a thresh- old r < min(u, v) avoids the issue of full- till-track matching, which only happens when min(u, v).
*There are some specific cases where a pattern could occupy two consecutive segments (c.g., when it is split between them or when it is reproduced at half the speed). We claim that, in such
cases, learning from just one of the two consecutive segments is enough (our results in Sec. 5 support this claim).
4

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Given the sets A+ and A we can write decoupled, su- pervised version of A&U over a batch
Depending if the pair is in A or A have
ac
2dg IA+
Odgy
(1.))A-
d3 + Alg
Σ ((J)E
A+
A-
Or
(
(3)
A 0C Odgy (1J)EA
-2ydije 2
where| denotes set cardinality and a, X, and y hyper- parameters. Wang & Isola (2020) do not report strong per- formance differences by changing a, X, and y within a certain range, and generally set a = 2, À = 1, and 7 = 3 for their experiments. In preliminary experiments, and for our task, we find similar conclusions for a and X, but not for y. In addition, we are motivated to use a 2 and À as that makes alignment and uniformity terms more compara- ble (same distance and same weight; see also the gradient analysis below).
|A- - (4)
where -odf corresponds to the negative potential for the pair i.j (Wang & Isola, 2020) and the constant is the sum of all potentials that do not feature (i,j). To facilitate our analysis. we view as reference potential and redefine é = |A- We can then consider three cases with regard to the relation of é and the potential for the negative pair i, lf < e -ndy (case 1), we have V ~ -2ydi and, if e (case 2), we have V -nd;j- In both cases, and thanks having set a 2 after Eq. 3, the terms in V are similar to the ones in V (thus positive and negative pairs have comparable influence). Moreover, we see that ry is also acting as weight for the negative pairs' gradient (thus taking a similar role as the original À in Eq. 3). Finally, if E >> e ydl (case 3), we have
The original A&U loss employs Euclidean distances on the hypersphere, using Le-normalized z vectors. This, in our view, presents potential issue, in the sense that the employed distance function does not match with the geo- metric structure of the space (Euclidean vs. hypersphere surface, respectively). In our approach, instead of consid- ering the negative are length (which corresponds to the geodesic distance in the hypersphere) to improve perfor- mance like Koishekenov et al. (2023), we opt for the plain Fuclidean space (of which the Euclidean distance its have
2yd 74g
geodesic distance). Thus, we do not constrain unit norm. Notice that, with this change, the uniformity concept does not apply, we are not in an hypersphere anymore. Nonetheless, we can still reason and base our in- tuitions on the kernel and potential concepts used to derive the uniformity term in Wang & Isola (2020).
which implies a progressively vanishing gradient with in- creasing di (e (e=n;;; decreases much faster than 7dg in- creases). Thus, we obtain a smooth transition to zero gra- dient after the potential e eddy crosses a threshold that is a function of ε. A visualization of the effect of y and E on V (Eq. 4) is given in Appendix A.
With the decoupling, hyper-parameter, and geometric con- siderations above, we formulate the CLEWS loss as
3.3. Architecture and Training
We now overview CLEWS' network architecture F and its training procedure (further details are available in Ap- pendix A and in our code). To obtain segment embedding vectors z on which to compute distances, we start from the full-track audio waveform and uniformly randomly cut 2.5 min block x from it. We further cut x into 8 non- overlapping 20-second segments x* (we repeat-pad the last segment). We then compute its constant-Q spectrogram, downsample it in time by a factor of 5, and normalize it between and 1, all following similar procedures as com- mon version matching approaches (Yesiler et al., 2021). After that, we pass it to a learnable frontend, formed by two 2D strided convolutions, batch normalization, and a ReLU activation. Next, we employ pre-activation ResNet50 backbone (He et al., 2016) with ReZero (Bachlechner et al., 2021) and instance- normalization (Pan et al., 2018). We pool the remaining generalized mean pooling (Radenović et al., 2019), and
L= IA+
dy + +og (iJ)AA
Σ
+ |A-|
1
Σ (1.))A
e eddyy
where dij distances after reduction (Eq. 2) and 8 0 are hyper-parameters. We use clidean distances (root mean squared differences), as this does not affect our geometric considerations (it just adds a constant) and facilitates maintaining the same hyper- parameters when changing the dimensionality of z. The e hyper-parameter is initially introduced for numerical sta- bility. However, we note that it also has soft thresholding or smoothing effect for the potential between negative pairs.
Eu-
Role of the Hyper-pauee parammtassseters - We now briefly and intuitively study the role of 7 and & in L (a full analysis is beyond the scope of the present paper). To do so, we consider the gradient of L for specific distance pair dij.
information with
5

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
project a 1024-dimensional representation z using batch normalization and linear layer. We train CLEWS with R+ = Ropwr-s, R = Rmin- 7 5, and = 10 as de- faults, and study the effect of such choices in Sec. 5. Since test sets also contain tracks longer than the 2.5 min used for training, in CLEWS we use our proposed 10 for track matching, together with a segment hop size of s.
ries of models are known be non-reproducible (O (O'Hanlon et al., 2021; Hu et al., 2022), with all attempts to date sub- stantially under-performing the reported results. We also implement our version of the ByteCover models and, for the first time, are able to obtain results that come close to the ones reported in the original papers (Sec. 5). In the follow- ing, we denote our approximations to ByteCover with i symbol. Having retrained/repllictted baselines provides us estimation of their performance in scenarios that have not yet been considered in the literature, such as with the DVI data set or the segment-level evaluation proposed below.
We train all models with Adam using a learning rate of 2-10-4, following reduce-on- onllata schedule with 10- epoch patience and an annealing factor of 0.2. The only exception is in ablation experiments, where we train for 20 epochs featuring a final 5-epoch polynomial learning rate annealing. In every epoch, we group all tracks into batches of 25 anchors and, for each of them, we uniformly sample with replacement positives from the corresponding version group (excluding the anchor). Thus, we get initial (track-based) batch size of 100. For every track in the batch, we uniformly sample 2.5 min from the full-length music track and create the aforementioned 8 segments per track. Thus, we get a final (segment- based) batch size of 800. We only use time stretch, pitch roll, and Specnngment augmentations (Liu al., 2023).
Evaluation - During testing, to compute candidate em- beddings, we treat all models as if they were segment-based and extract overlapping blocks or segments using the same length as in training and hop size of 5 (this yielded a marginal improvement for full-track baselines trained on 2.5 min blocks). With these candidate embeddings, we per- form both track- and segment-level evaluations. The former is equivalent to the usual evaluation setup in musical ver- sion matching, while the latter focuses on the retrieval of best-matching segments. For the track-level evaluation, we use the same segment length (the training segment length) and hop size for both queries and candidates. Then, mea- sure the performance of the system working at the full-track level, we use Reeammia to compute the final query -unry-can distance. For the segment-level evaluation, we keep the same segment configuration as in the track-level case, but we vary the query segment length т. This way we assess a model's performance on different query lengths found in real-world scenarios. Then, measure the performance of the system working at the segment level, we use Rmii to compute best-match query-candidate distance. With this best-match approach, we simulate the performance of an equivalent segment-based retrieval system using all raw segments as candidates (see Appendix B). As evaluation measures, we compute the usual mean average precision (MAP) plus an enhanced version of the normalized average rank (NAR; see Appendix B). MAP focuses on the precision in the top candidates while NAR focuses more on the overall recall, which we think is better option for musical version matching, especially for segment- based applications.
4. Evaluation Methodology
Data - We train and evaluate all models on the publicly- available data sets Discogs VI-YT (DVI; Araz et al., 2024a) and SHS100K-v2 (SHS; Yu et al., 2020), using the prede- fined partitions. SHS is a well-established reference data set. However, since it is based on YouTube links, it is almost impossible to gather it entirely nowadays (we managed to gather 82% of it). In addition, one could consider it slightly biased, as the version group sizes are unrealistically large (cf. Doras & Peeters, 2020; Araz et al., 2024a). Instead, the re- cently proposed DVI data set is 5 times larger and better represents the real-world distribution of version group sizes. For both data sets, we use 16 kHz mono audio and cap the maximum length to the first 10 min.
Baselines- - To compare the performance of the proposed approach with the state of the art, we consider several base- lines: CQTNet (Yu et al., 2020), MOVE (Yesiler et al., 2020a), LyraC-Net (Hu et al., 2022). CoverHunter (Liu et al., 2023), DVINet+ (Araz et al., 2024b), Bytecover2 (Du et al., 2022), ByteCover3 (Du et al., 2023), and ByteCover3.5 (Du et al., 2024). For CoverHunter, we just consider the first "coarse" stage, as that is the part dealing with segments. CoverHunter, CQTNet, and DVINet have convenient source code available, and thus we can produce results by using it in our own pipeline (this way we can compare those basclines with our model rigorously under the same setting). Due to GPU memory restrictions, we train with randomly-sampled audio blocks of 2.5 min. For the other baselines, we can only use already reported results as reference. The ByteCover se-
5. Results
Comparison with the State of the Art First of all, we focus the track-level evaluation and compare with the state of the art. We observe that CLEWS outperforms all considered approaches, many of them by a large margin (Table 2). CLEWS obtains a NAR of 2.70 on DVI-Test and a MAP of 0.876 on SHS- setting a new state-of-the-art result on both data sets. Besides CLEWS, an interesting
"See for instance httpp://githiii com/ /mmm/rfiuml bytecover/isso
6

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Table Track-level evaluation and comparison with state the art. symbol denotes that our implementation and
symbol denotes 95% confidence intervals.
APPROACH
DVI-TEST
SHS-TEST
NAR +
MAP
NAR
MAP
COVERHUNTER-CONRE (LIU ET AL., 2023)
10.36 + 0.07
0.157 + 0.001
4.09 1 0.17
0.491 + 0.007
MOVE (YESILER ET AL., 2020A)
N/A
N/A 0.493 0,002
N/A
0.519
CQTNET (YU ET AL., 2020)
6.68 0.07
2.67 = 0.16
0.677 + 0.007
DVINET+ (ARAZ ET AL., 2024B)
3.69 + 0.06 N/A
0.643 + 0.002 N/A 0.513 + 0.002
2.39 = 0.16
0.720 I 0.007
LYRAC-NET (HU ET AL.,, 2022)
N/A
0.765
BYTECOVER3f (BASED ON DU ET AI.,, 2023)
5.64 + 0.05
1.91 + 0.14
0.783 + 0.006
BYTECOVER 1/2f (BASED ON DU AI.., 2022)
4.98 0.06
0.595 + 0.002
1.95 + 0.14
0.813 + 0.006
BYTECOVER3 (DU ET AL., 2023)
N/A
N/A
N/A
0.824
BYTECOVER3.5 (DU ET AL., 2024)
N/A
N/A
N/A
0.857
BYTECOVER2 (DU ET AL.. 2022) CLEWS (PROPOSED)
N/A
N/A 0.774 + 0.002
N/A
0.863
2.70 + 0.05
1.27 = 0.12
0.876 + 0.005
50
1.0
Coverilunter Course
CQTNet
0.8
20
Byseoover
ByleCover3 +
10
MAP 7
0.6
CLEWS (proposed)
NAR
0.2
0.0
10
40
[s]
[s]
Figure 2. Segment-level evaluation with Des-Test. NAR (left) and MAP (right) for different query segment lengths (notice the logarithmic axis NAR). The shaded regions correspond to confidence intervals (barely visible due to the size DVi-Test). Comparatively similar results for SHS-Test and also for an alternative evaluation protocol are available in Appendix C.
observation make is that the rank of existing approach considerably varies between SHS-Test and DVI-Test. That is the case of DVINet+, which obtains a modest performance on SHS-Test Test, but achieves the best performance among the existing approaches on DVI-Test. We hypothesize that this thanks being only considered approach (apart from CLEWS) that does not learn from a classification loss: be- cause SHS-Train has more items per class than the more real- istic DVI-Train (an average of 12 vs. 2, respectively), clas- sification loss is able to learn a useful representation from SHS-Train, but not from DVI-Train. Both DVINet+ and CLEWS, utilizing triplet and contrastive losses, respectively, do not suffer from this issue and maintain a comparatively similar performance across the two data sets. Another obser- vation make with regard the consideration of segments is that treating them independently (CoverHunter- Coarse) yields worse results than developing some specific strategies (ByteCover3, ByteCover3t, and CLEWS), and that learning from a global match (ByteCover3 and ByteCover3)) is not as optimal as learning from a partial match (CLEWS). The latter is further supported by our ablations below.
Segment-based Version Matching We focus the segment-level evaluation and study the performance as function of the query segment length We observe that CLEWS again outperforms all considered models both in DVI-Test (Fig. 2) and SHS-Test (Appendix C), and for both NAR and MAP measures. Importantly, CLEWS maintains high performance for all considered lengths (Fig. 2). The only exception with ing experience, it is sometimes difficult even for human to establish if two audio segments are versions or not. Ac- cording to our segment-level evaluation, ByteCover3) fea- tures some noticeable performance degradation with large т, perhaps due to the global match approach. CQTNet and DVINett, both based on the same plain convolutional architecture, show early performance decline for < 60.
where, according to our listen-
A Ablations and hyper-parrmmters: Finally, focus attention possible variations to the default CLEWS. We start by studying the effect of positive R+ and negative R segment distance reductions (Table 3). For R+ if we keep R Rmin- we observe that, depending on the evaluation

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Table 3. Results on DVI- Valid for different positive Rt and nega-
0.825
tive R distance reductions. The default CLEWS reductions are
3.25
R + Repwess and R
Rmin. As above, + denotes 95% c.i.
0.800
3,00
MAP
R * R
NAR
MAP1 f
NAR
2.75
0.775
CLEWS (PROPOSED)
2.57 + 0.09
0.804 + 0.003
2.50
0.750
3
Rmin
2.60 + 0.09
0.809 + 0.003
Ropusss
Rmin
2,51 + 0.09
0.789 + 0.003
2.25
0.725
Raesnmin
Rmin
2.58 + 0.09
0.798 + 0.003
5
Rees-10
Rmin
2.63 + 0.09
0.795 + 0.003
Y (using
10 9)
Rain
Rmin
2.79 + 0.09
0.799 + 0.003
Repwe-S
Reest 10
2.82 0.10
0.779 0.003
0.825
3.25
Repuess
R Rapwes
2.88 = 0.10
0.778 + 0.003
0.800
Repues5
R Remanmin
4.95 0.12
0,488 + 0.004
3.00
NAR
MAP1
2.75
0.775
2.50
0.750
Table Results on DVI-V for different loss functions using the default CLEWS reductions of R Repars and R Rmin.
2.25
0.725
10
10"
5
10
Loss FUNCTION
NAR 4
MAP
(using 5)
CLEWS (PROPOSED)
2.57 + 0.09
0.804 + 0.003
Figure 3. Effect of hyper-parameters Y (top) and (bottom) on DVI-Valid Shaded regions correspond to 95% confidence inter- vals, and the default value is highlighted with a square marker.
SUPCON
2.69 + 0.09
0.676 + 0.004
SIGLIP
2.79 0.09
+ 0.684 0.004
TRIPLET
3.08 0.11
0.717 0.004
DUPCON-DECOUPLED
0.11
0.739
0.004
A&U-DECOUPLED
3.25 125510.11 0.11
0.620 1 0.004
CLASSIFICATION XENT
8.91 0.14
0.205 1 0.003
observe an opposite trend for the two evaluation measures (Fig. 3): with progressively decreasing NAR (better perfor- mance), obtain a progressively decreasing MAP (worse performance). This indicates that hyper-parameters 7 and can be deliberately tuned to benefit one or the other measure (we did not extensively tune them for the results reported previously). Morcover, we actually see that setting = 2 could have provided a better NAR and a slighlty increased MAP. Overall, however, if we zoom ourselves out from the resolution shown in Fig. 3, essentially observe a plateau of performance between y E [2, 8) and E E [10-$, [10-8,10-5]
measure, we have two options that are better than the default: Repwr-8 for NAR and R bpww-3 for MAP. Nonetheless, we decide to keep the default one as a compromise between the two. With such compromise in mind, we observe that, for positive reductions, global matches (Rapwr-s and Reeammii) under-perform partial matches (Rypwx-)). and that learning from consecutive segments 10 and R is not as competitive avoiding them (Ropwr-). For R if we keep RR = =====Ropws Rapwr-s, we see that all considered negative reductions under-perform the default Rmin- We hypothesize that, as with triplet losses, a hard negative mining or worst-case strategy is beneficial (cf. Schroff et al., 2015; Kalantidis al., 2020). Reduction strategies based on Rmean did not learn well, both for R+ and R (not shown).
6. Conclusion
In this paper, we tackle the task of segment-based musi- cal version matching, and propose both strategy to deal with weakly-labeled segments and contrastive loss that outperforms well-studied alternatives. Through a series of extensive experiments, we show that our approach not only achieves state-of-thed oh-thc-arttttttt results in two different datasets and two different metrics, but also that it significantly outper- forms existing approaches evaluation. We also study the effect of different reduction strategies, compare against existing losses, and analyze the effect of the parpmmtersssters in our ablation studies. As weakly labeled segment information is ubiquitous in many research areas, and since the concepts exploited here are gen- eral to a wide range of contrastive learning tasks, we believe our methods could serve as inspiration or find usefulness in domains beyond audio and musical version matching.
The next aspect we study is the effect of the loss function given the default reduction strategies for R* and R (Ta- ble 4). We observe that the proposed CLEWS loss performs better in both NAR and MAP, with a significant difference the latter measure. Standard losses like SupCon, SigLIP, and Triplet (Sec. 2) come next, not being able take as much profit from R+ and R- as CLEWS in the task we study. As already mentioned, a standard classification loss based on cross-entropy does not perform well, especially when training with very few instances per class.
best-match, segment-level
The last aspect we study is the effect of papeer-rrameters y and &. we zoom in the resolution of NAR and MAP, we
8

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Acknowledgements
terns. In Proc. of the Ini. Conf. on Multimedia and Ubiq- uitous Engineering (MUE), pp. 361-365, 2007.
We thank Toshimitsu Uesaka for his comments on an earlier version of the paper. R. Oguz Araz is supported by the pre-doctoral program AGAUR-FI ajuts (2024 FI-3 00065) Joan Oró, and the Cátedras ENIA program "IA y Música: Cátedra en Inteligencia Artificial y Música" (TSI-100929- 2023-1).
Bralios, D., Wichern, G., Germain, G., Pan, Z., Khurana, S., Hori, C., and Le Roux, Generation or replication: auscultating audio latent diffusion models. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 1156-1160, 2024.
Cano, P., Batlle, E.,, Kalker, T., and Haitsma, J. review of audio fingerprinting. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology. 41:271- 284, 2005.
Impact Statement
This paper presents work whose goal is advance the fields of machine learning and music information retrieval. Musi- cal version matching can be used to enhance music discov- ery, preserve cultural heritage, and support fair copyright management. By connecting versions across styles and per- formances, musical version matching also fosters creativity, promotes artistic appreciation, and paves the way for more equitable solutions the music industry. benefiting society at large. As with any machine learning tool, however, there always exists the possibility of some potential misuses of itself or of some of its components, none of which we feel must be specifically highlighted here.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, simple framework for contrastive learning of visual repre- sentations, In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 1597-1607, 2020.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. In Advances in Neural Information Processing Systems (NeurlPS), volume 36, pp. 47704- 47720. 2023.
Doras, G. and Peeters, G. prototypical triplet loss for cover detection. Proc. of the IEEE Int. Conf. on Acous- tics, Speech and Signal Processing (ICASSP), pp. 3797- 3801, 2020.
References
Araz, R. O., Serra, X., and Bogdanov, D. Discogs- VI: musical version identification dataset based on public editorial metadata. In Proc. of the Int. Soc. for Music Information Retrieval Conf. (ISMIR), pp. in press, 2024a.
Du, X., Chen, K., Wang, Z., Zhu, B., and Ma, ByteCover2: towards dimensionality reduction of latent embedding for efficient cover song identification. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 616-620, 2022.
Araz, R. O., Serrà, J., Serra, X., Mitsufuji, Y., and Bog- danov, D. Discoos-VINNe-MIRRE In Music Information Retrieval Evaluation eXchange (MIREX), 2024b.
Du, X., Wang, Z., Liang, X., Liang, H.,, Zhu, B.,, and Ma, ByteCover3: accurate cover song identification short queries. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2023.
Bachlechner, T., Majumder, B., Mao, H., Cottrell, G., and McAuley, J. ReZero is all you need: fast convergence at large depth. In Proc. of the Conf. on Uncertainty in Artificial Intelligence (UAI), pp. 1352-1361, 2021.
Du, X., Liu, M., and Zou, P. X-Cover: better music ver- sion identification system by integrating pretrained ASR model. In Proc. of the Int. Soc. for Music Information Retrieval Conf. (ISMIR), pp. in press, 2024.
Ball, P. The music instinct. Random House, London, UK, 2010.
Barnett, J., Flores Garcia, H., and Pardo, B. Exploring musical roots: applying audio embeddings to empower influence attribution for a generative music model. ArXiv, 2401. 2024.
Evans, Z.. Carr, C. J., Taylor, J., Hawley, S. H.,, and Pons, J. Fast connig-conditioned latent audio diffusion. In Proc. of the Int. Conf. on Machine Learning (ICML), volume 235, pp. 12652-12663 2024.
Batlle-Roca, R., Liao, W.H.., Serra, X., Mitsufuji, Y., and Gómez, E. Towards assessing data replication in music generation with music similarity metrics on raw audio. In Proc. of the Int. Soc. for Music Information Retrieval Conf. (ISMIR), pp. in press, 2024.
He, K., Zhang, X., Ren, S., and Sun, Identity mappings in deep residual networks. Proc. of the European Conf. on Computer Vision (ECCV), pp. 630-645, 2016.
Hu, S., Zhang, B., Lu, J., Jiang, Y., Wang, W., Kong, L., Zhao, W., and Jiang, T. WideResNet with joint repre- sentation learning and data augmentation for cover song
Bosteels, K. and Kerre, E. E. Fuzzy audio similarity mea- sures based on spectrum histograms and fluctuation pat-

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
identification. In Proc. of the Conf. of the Int. Speech Association (INTERSPEECH), pp. 4187-4191, 2022.
Radenović, F,, Tolias, G., and Chum, O. Fine-tuning CNN image retrieval with no human annotation. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 41 (7):1655- 1668, 2019.
Kalantidis, Y., Sariyildiz, M. B., Pion, N., Weinzaepfel, P., and Larlus, D. Hard negative mixing for contrastive learning, In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 21798-21809, 2020.
Schroff, F., Kalenichenko, D., and Philbin, J. FaceNet: unified embedding for face recognition and clustering. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 815-823, 2015.
Khosla, P., Teterwak, P., Wang. C.,, Sarna, A., Tian, Y., Isola, P., Masacchint,, A., Liu, C., and Krishnan, D. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 18661- 2021.
Serrà, J. Identification of versions of the same musical com- position by processing audio descriptions. PhD Thesis, Universitat Pompeu Fabra, 2011.
Koishekenov, Y., Vadgama, S., Valperga, R., and Bekkers, E. J. Geometric contrastive learning. In Proc. of the IEEE/CVF Inl. Conf. on Computer Vision Workshops (ICCVW), pp. 206-215, 2023.
Sohn, K. Improved deep metric learning with multi-class N- N-pair loss objective. In Advances in Neural Information Processing Systems (NIPS), volume 29, pp- 1857-1865. 2016.
Sun, Y., Wang, X., and Tang. X. Deep learning face repre- sentation from predicting 10,000 classes. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1891-1898, 2014.
Koromilas, P., Bouritsas, G., Giannakopoulos, T., Nicolaou, M., and Panagakis, Y. Bridging mini-batch and asymp- totic analysis in contrastive learning: from InfoNCE to kernel-based losses. In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 25276-25301, 2024.
Van den Oord, A., Li, Y., and Vinyals. O. Representa- tion learning with contrastive predictive coding. ArXiv, 1807.03748, 2018
Liu, F., Tuo, D., Xu, Y., and Han, X. CoverHunter: cover song identification with refined attention and alignments. In Proc. of the IEEE Int. Conf. on Multimedia and Expo (ICME), pp. 1080-1085, 2023.
Wang. T. and Isola, P. Understanding contrastive repre- sentation learning through alignment and uniformity on the hypersphere. In Proc. of the Int. Conf. on Machine Learning (ICML), volume 119, pp. 9929-9939, 2020.
Liu, H., Tian, Q. Yuan, Y., Liu, X., Mei, X., Kong, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. AudioLDM 2: learning holistic audio generation with self-supervised pretraining. IEEF/ACM Trans. on Audio, Speech, and Language Processing, 32:2871-2883, 2024.
Ych, C.-H., Hong, C.-Y,, Hsu, Y.-.., Liu, T.-L., Chen, Y., and LeCun, Y. Decoupled contrastive learning. In Proc. of the European Conf. Computer Vision (ECCV), pp. 668-684, 2022.
Müller, H., Müller, W., Squire, D. M., Marchand-Maille S., and Pun, T. Performance evaluation in content- based image retrieval: overview and proposals. Pattern Recog- nition Letters, 22(5):593 601, 2001.
Yesiler, F., Serrà, J., and Gómez, E. Accurate and scal- able version identification using musically-motivatee em- beddings. In Proc. of the IEEE Int. Conf. on Acous- tics, Speech and Signal Processing (ICASSP), pp. 21-25, 2020a.
O' Hanllo, K., Benetos, E., and Dixon, S. Detecting cover songs with pitch class key-invariant networks. In Proc. of the IEEE Int. Workshop on Machine Learning for Signul Processing (MLSP), 2021.
Yesiler, F., Serrà, J., and Gómez, E. Less is more: faster and better music version identification with embedding distillation. In Proc. of the Int. Soc. for Music Information Retrieval Conf. (ISMIR), pp. 884 802, 2020b.
Page, W. Music smashes box office records: global value of music copyright soars to $45.5 bn, now worth more than cinema, 2023. URL
Yesiler, F., Doras, G., Bittner, R. M., Tralie, C. J., and Serrà, J. Audio-basse musical version identification: elements and challenges. JEEE Signal Processing Magazine, 38 136, 2021.
com/
Yu, Z., Xu, X., Chen, X., and Yang, D. Learning a rep- resentation for cover song identification using convolu- tional neural network. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 541-545, 2020.
Pan, X., Luo, P., Shi, J., and Tang, X. Two at once: enhanc- ing learning and generalization capacities via IBN-Net. In Proc. of the European Conf. on Computer Vision (ECCV), pp. 484-500, 2018
10

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proc. of the IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. pp.
11975-11986, 2023.

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
APPENDIX
In this supplementary part of the paper provide further information on the proposed method (Appendix A). We also explain with detail evaluation methodology (Appendix B). Finally, show additional results that could not fit in the main manuscript (Appendix C).
A. Method Details
A.1. Loss Gradient Visualization
In Sec. 3.2 the main mamuscript, we study the effect of hyper-purametters y and E on the gradient negative pairs V- Here, to further facilitate understanding, we plot the result of V (Eq. 4) for a range of potentials e e->df under different values of y and in Fig. 4. We do using | 128 and (|4-|- 1)e >el, 1)e-, With the latter, approximate the case where the average negative potential is not far from the potential of the i, pair.
10
0.00
0.00
0.00
-0.05
-0.05
-0.05
0.10
0.10
0.10
0.15
0.15
-0.15
10
10-9
10
10
10"
10-12
10-9
10
10
10
10
10
10
10
10
Figure Plot V as a function the negative pair potential e- for different values of and s. From left to right, we show y 12,5, 10}. From darker to lighter, colors correspond (10-5 each e value (notice that, in C. e is compared to an average negative pair potential, hence placing e as a reference the potential axis makes sense).
10-7 10 10-5, 10 10-3). Dash-dotted lines indicate
A.2. A More Numerically-Friendt Version of C
For conducting all our experiments, we found no issue in the use of L with regard to numerical stability with 32-bit precision. However, we should note that L. as written in the main manuscript, may have some numerical instability, especially when employing abnormally small/large values of e/7, or potentially when using a numerical precision below 32 bits. In such cases, we recommend switching to the formulation below.
First of all, multiply the terms inside the logarithm by 1/e:
Σ & + log (LJ)AA
c
nde = IA+
Σ
d, + log
1+
Σ e ede + +log(E). (i,))
LA+
A
E|A-
(K.))AA
(K.))A+
With this, obtain the term log(E), which is just constant that does not affect the gradient and can thus be dropped. Next, we perform the change of variable 1/(s) A-I) I) Beh, where b 2 0 is a constant we will set for the upper numerical limit we allow to the exponential. With this change and a few simple operations, we arrive to
L
1
dy log 1+ 3 Σ c (++
Σ
A+
(KJ)EA-
where 3 = \|e)) We can now choose as compromise between the overflow of eb when d, 0 and the underflow of b-,d,, when dey large. For normalized Euclidean distances d and the ranges of and y we consider, we choose b = 10. Note that, in addition, log(1 æ) can be implemented with loglp ( (x) in most scientific programming languages.
12

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
A.3. Model
We here provide fürther specification of our network architecture (for full detail we refer the interested reader to the published code). As mentioned, we use kHz mono audio with maximum length of 10 min for both training and evaluation. For training, we cut 2.5 min blocks uniformly at random. For CLEWS, we divide such blocks into 8 non-overlapping 20-second segments. As the last segment is only s we take the opportunity to repeat-pad such segment and also consider it in our training, with the hope that this will facilitate retrieval with query lengths shorter than 20 (which we also repeat-pad as they would not be long enough to accommodate the total striding factor of our architecture).
After obtaining segments, we apply constant-Q transform (CQT) with 20 ms hop size, spanning octaves (from a minimum frequency of 32.7 Hz), and with 12 bins per octave (we use the nnAudio library" in non-trainable mode, with the rest of the parameters set as default). We then take the CQT magnitude and average in time every 5 consecutive frames without overlap. This CQT representation is sent to three data augmentation functions (explained in the next subsection).
The neural network architecture starts by taking the square root of the CQT magnitude, normalizing every segment's ' representation between O and 1, and applying a learnable affine transformation. Next, we apply a 128-channel 2D convolution with a frequency-time kernel size of 12x3 and a frequency-time stride of (1,2). This is followed by batch normalization (BN), a ReLU activation, and 256-channel 2D convolution with kernel size of 12x3 and stride of (2.2). This constitutes our frontend. Unless stated otherwise, we use the default PyTorch² parameters from version 2.3.1.
As mentioned in the main manuscript, our backbone is formed by pre-activation ResNet modules with ReZero and instance- batch normalization (IBN), We use 3, 4, 6, and 3 residual blocks with 256, 512, 1024, and 2048 channels, respectively. The strides are (1,1). (2,2), (2,2), and (1,1) for each block. The residual blocks have an MON-ReLU-ccon-BN-RLLUU structure, with a kernel of 3x3 in the convolution layers. To reduce GPU memory consumption, we employ half the channel dimension inside the residual block. If there is some channel or stride change, the skip connection features a BN-ReLU-coom block also with 3x3 kernel.
The output of the backbone is time- and frequency-pone poonled by generalized mean pooling operation with single learnable exponent. Finally, the result is processed with BN and projected to 1024 dimensions by a linear layer. None of our linear or convolutional layers feature bias terms. As mentioned in the main manuscript, we use normalized squared Euclidean distances (mean squared differences) between embedding vectors.
A.4. Training
We train all models with Adam using the default PyTorch parameters, a learning rate of 2-10-4, and batch size of 800 segments chosen from 100 tracks featuring 3 positives per anchor. In the main experiments, we follow a reduce-on- plateau strategy for the learning rate, monitoring an average between MAP and NAR measures on the validation set. We define an epoch as using all training tracks as anchor once, and set 10-epoch patience period and an annealing factor of 0.2. Using this strategy, training CLEWS on SHS and DVI takes approximately 2 and 9 days, respectively, using two NVIDIA H100-80GB GPUs In the ablation experiments, to reduce the computational burden, we only train for 20 epochs and, during the last 5 epochs, we apply polynomial learning rate annealing with an exponent of 2.
During training, we employ three CQT data augmentation functions: SpecAugment, we mask a maximum of 15% of the time/frequency tiles. For time stretch, we resample by a uniformly sampled factor between 0.6 and 1.8. For pitch roll, we choose a uniform value between - -22 and +22. In the DVI data set, we use a probability of 0.1 independently for each augmentation. However, since the SHS data set is considerably smaller than DVI and potentially features less variability, we find some benefit in increasing such probability for SHS. In that case, set the augmentation probabilities to 0.4, 0.3, and 0.5 for SpecAugment, time stretch, and pitch roll, respectively.
SpecAugment, time stretch, and pitch roll. For
B. Evaluation Methodology Details
B.1. Track- and Segment-level Evaluations
For the track-level evaluation, we cut the entire raw waveform (up to the first min) into overlapping blocks or segments using a hop size of 5.. For both the queries and the candidates, the length of such blocks/segments corresponds to the same
/tttps:://gihhbb.com hww.:///w...aaaaeeeeeeeaaaaeaaae hhttpp::/ppytorcc..rrr////ss///
13

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
length used to train each model (that is, 2.5 min for CQTNet, DVINet+, and ByteCover1/2f and 20 for CoverHunter, ByteCover3f, and CLEWS). Next, we compute pairwise distances for each query-candidate block/segment and apply distance reduction function. We use Recanmin for all models except CLEWS, which exploits the newly proposed Ropwr-10- After reduction we obtain a track-based distance matrix that we can use to sort candidates per query and compute common evaluation measures.
For the segment-level evaluation, we also cut the entire raw waveform into overlapping blocks/segments with a hop size of s. For candidates, we also use the same length that we used to train each model (same as in the track-level evaluation). However, for the queries, we extract multiple-length segments with hop size of 5 (we consider segment lengths = {5, 10, 20, 30, 40, 60, 90) s). Then, given segment length, compute pairwise distances for each query- candidate block/segment, and apply the Rmmnn distance reduction to obtain a track-based distance matrix. After that, the evaluation proceeds as with the track-level evaluation (and any common evaluation protocol in musical version matching), The usage min puts the focus on the best-matching segment per track, and is equivalent to performing version matching on an index formed by all possible segments, treating them independently, and removing duplicate track names after sorting.
B.2. Normalized Average Rank
To evaluate retrieval performance and complement mean average precision (MAP), we employ an enhanced version of the normalized average rank (NAR), originally proposed by Müller et al. (2001). Given a list retrieved items R. sorted in descending order of predicted relevance to a query q. and containing set of target matches M = {mi:... .mp.. M R. Bosteels Kerre (2007) redefined NAR
JMI NAR, R Σ
(m:, R) -t);
where the function rank(m, Rt) [1, R|] returns the rank of m in R. This definition, as well the one of Müller et al. (2001), yields 0 for perfect retrieval, 0.5 for random retrieval, and approaches I as performance worsens. However, a value equal to one is never obtained. Not only that, but the maximum bound inversely depends on M and, therefore, can be different for each query q. To avoid that, one should replace the number of retrieved items [R] in the denominator by the number of non-relevant retrieved items [ M) Hence, we correct the definition of Bosteels Kerre (2007) and employ
JMI
NAR,
100
Σ
rankk(m,, R) - -i).
M ( M))
which additionally yields a convenient value, now between and 100 for all sizes of M. Our final number is the average over all queries Q:
NAR
NAR,
GEQ
Note that, in research evaluation scenarios, one must compute both MAP and NAR measures excluding the query from the candidate list.
C. Additional Results
C.1. Segment-level Evaluation with the Best Match Protocol
In the main manuscript, we present the results for the segment-level evaluation on DVI-Test (Fig. 2). The exact numbers for such plots can be found here in Table 5. For SHS-Test, we obtain comparable results, which can be found below in Fig. 5 and Table 6.
C.2. Segment-level Evaluation with the Random Segment Protocol
In our segment-level evaluation, we adopt a best match protocol as specified in Sec. 4. However, Du et al. (2023) introduced what could be termed as the 'random segment' protocol: "For each query, we constructed a query set consisting of the original -all-track recording, and music clips randomly cut from it, with the duration being 6, 10, 15, 20, 25, 30, 40, 50 and 60 seconds respectively" (Du et al., 2023). Apart from lacking further specification, we claim that using random segments
14

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
biases the evaluation, have match in version song). Furthermore, the objective is match tracks by their segments, helieve using random segments for evaluation may implicitly favor approaches exploiting more generic or global track characteristics than the specific matching-segmann segment information. These are the reasons why we introduce our segment-based protocol. However, in the spirit of comparing with existing reported values, and to avoid any doubt on the performance of the proposed approach, we replicate such protocol (to our best) and compute again results for all methods considered here. They are shown in Fig. 6 and Table below, together with the MAP values of ByteCover2, ByteCover3, and Re-MOVE (Yesiler et al., 2020b) reported by Du et al. (2023).
never reach perfect accuracy (a random segment from song does necessarily need
C.3. Runtime
To conclude, we also provide an informal runtime analysis for the considered models (Table 8), For a fair comparison, inference times are measured with the segment-bas based evaluation protocol, thus all models perform the same task of segment- based retrieval with 5.s hop size, using Rmin- We should also note that the time complexity of the na've implementation of the reductions studied above is (uv)) for Rmean and Rinn Ouuv u) for Rmeanmin O(r + we log(uv)) for Roest-r and Orr(uv + u- v)) for Rapwer: where I min( u, v) and are the number of considered segments in a sub-rectangle (see main text). Note that the values for r, and are small for today's 's computation standards. For instance, 5 min song with 20 s segments and no overlap yields u = 15.
Table Segment-lewe evaluation with DVI-Test. NAR (top) and MAP (bottom) results for different lengths of query segments т. The symbol marks 95% confidence intervals.
APPROACH
T [SS]
5
90
COVERHUNTER COARSE
14.97 0.07
12.01 0.07
11.01 + 0.07
10.89 + 0.07
10.88 + 0.07
10.95
0.07
11.07 0.07
CQTNET
0.110
48.48=0.11
16.35
0.07
0.007
0.07
6.60
DVINET+
0.13
4..11:0.12
10.42
0.06
4.20
0.04
3.76
BYTECOVER
0.11
6.77
0.06
0.06
0.19
BYTECOVER3
0.11
18.45 = 0.08
8.66 0.06
0.06
0.06
0.00
7.06 0.06
CLEWS (OURS)
5.10 + 0.06
3.02 + 0.05
2.86 0.05
0.05
2.85 0.05
0.07
2.89 0.05
COVERHUNTER COEEREE
0.001
0.132 0.001
0.001
0.009
0.127 0.001
CQTNET
0,001 0.000
0.001 = 0.000
0.011 0,000
0.008
0.282 0.002
0.026
0.475 0.002
DVINET+
0.001 0.000
0.002 + 0.000
0.026 0.000
0.001
0.421 0.002
0.561
0.616 0.002
BYTECOVERI/2
0.008 + 0.000
0.083 0.001
0.363 = 0.002
0.483
0.529 0.002
0.564
0.582 0.002
BYTECOVER3
0,001 + 0.000
0.062 : 0.001
0.358 = 0.002
0.002
0.508 0.002
0.503 4.03000000
0.473 0.002
CLEWS (OURS)
0.271 + 0.002
0.670 + 0.002
0.754 0.002
0.756 + 0.002
0.755 0.002
0.747 0.002
0.738 0.002
CoveHenttnCaaeee
CQTNet
0.8
DVINete
20
ByleCover/22
10
MAP1
0.6
CLEWS (propoeed)
NAR
0.2
0.0
10
20
30
40
60
10
20
30
40
60
[s]
[s]
Figure 5. Segment-level evaluation with SHS-Test. NAR (left) and MAP (right) for different lengths of query segments (notice the logarithmic axis for NAR). The shaded regions correspond to 95% confidence intervals.
15

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Table Segment-level evaluation with SHS-Test. NAR (top) and MAP (bottom) results for different lengths of query segments т. The + symbol marks 95% confidence intervals.
APPROACH
[8]
5
10
20
30
60
90
COVERHUNTER-COAAE
10.78 0.23
6.61 + 0.20
4,90 # 0.18
4.62 + 0.18
4.54 0.18
4.52 0.18 0.18
4.56 +: 0.18
CQTNET
0..48
44.98=0.4
9.45
4.32
0.16
2.14
2.67 2.67:0.16
DVINET+
49.44
43.12 =
16.05 0.33
5.45
3.29
2.62
2.52
BYTECOVER
22.68 0.442
6.53
2.71
0.15
0.15
2.07
2.03 2.03 0.14
BYTECOVER31
0.32
=
0.16
2.97
0.15
2.68
4.53 333300.0 4.53
CLEWS (OURS)
3.39
1.49 +
1.33
1.35
1.37
1.47
COVERHUNTER-COARSE
0,099 + 0.004
0.007
0.414
0.440 0.000
0.439
0.435 0.007
CQTNET
0.603
DVINET+
0,003 + 0.000
0.001
0.093
0.365 0.007
0.000
0.691 0.007
BYTECOVERI/2
0.033 + 0.002
0.006
0.007
0.008
0.006
0.006
0.800 0.006
BYTECOVER3 t
0.098 + 0.004
0.237 + 0.007
0.628 0.008
0.687 0.008
0.690
0.007
0.674 0.007
0.576 0.008
CLEWS (OURS)
0.394 + 0.007
0.806 0.006
0.859 = 0.005
0.861 0.005
0.859 0.005
0.852 0.006
0.847 0.006
50
1.0
Covasuuner
CQTNa
DVINet
20
BylsCoerr
RyleCwert ?
10
MAP1
0.6
Re-MOVE
NAR
ByleCover2
0.4
CLEWS (poposol)
0.0
5
10
20
30
40
60
90
5 10
20
30
40
60
90
[8]
[s]
Figure 6. Segment-level evaluation with SHS-TTss using the random segment protocol of Du et al. (2023). NAR (left) and MAP (right) for different lengths of random query segments (notice the logarithmic axis for NAR). The shaded regions correspond to 95% confidence intervals. The dotted lines correspond to values reported by Du et al. (2023).
Table Segment-level evaluation with SHS-Test using the random segment protocol of Du et al. (2023). NAR (top) and MAP (bottom) results for different lengths of random query segments т. The 1 symbol marks 95% confidence intervals. Du et al. (2023) did not report any confidence interval.
APPROACII
[8]
5
11
20
30
60
90)
COVERHUNTER-COME
15.15 0.30
9.97 + 0.25
7.42 + 0.22
6.56
+ 0.21
6.03 - 0.21
5.62 0.20
5.27 + 0.19
CQTNET
49.73
46.67 = 0.47
18.55
0.32
10.21 = 0.25
7.18 0.23
4.88
3.78 - 0.18
DVINET+
0.55
0.51
22.65
11.41
7.09 0.24
4.57
3.50 0.19
BYTECOVER
31,96
19.16
0.26
3.77
3.02
BYTECOVER3
0.337
10.95
0.22
0.23
4.68 0.20
4.03
5.52
CLEWS (OURS)
0.30
8.09 0.25
0.119
0.10
0.11
2.32
2.07 0.15
CoVERHUNTER-COAEE
0.0007
0.385
0.399
COTNET
0.003 1 0.000
0.003 0.000
0.019
0.276 0.003
0.074
0.586 0.007
DVINET+
0.003 0.000
0,004 0.000
0.016 0.000
0.009
0.0006
0.49810.007
0.623 0.007
BYTECOVERI/2
0.022 + 0.001
0.110 + 0.004
0.357
0.006
0.516+0.000
0.607 0.007
0.691 + 0.007
0.746 0.007
BYILCOVER3
0.044 + 0.002
0.133 + 0.004
0.432
0.007
0.511 0.007
0.528
0.007
0.539 1:39:00 0.007
0.469 0.008
RE-MOVE
0.023
0.069
0.196
0.308
0.407
0.505
N/A
BYTECOVER3
0.084
0.257
0.496
0.600
0.666
0.732
N/A
BYTECOVER2
0.016
0.074
0,282
0.442
0.564
0.684
N/A
CLEWS (OURS)
0.140 +: 0.004
0.455 + 0.006
0.652 + 0.007
0.714 + 0.006
0.746 0.006
0.780 0.006
0.802 + 0.006
16

---

Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching
Table Training and inference runtimes (informal) using a single NVIDIA H100 GPU. Training runtime is measured using a batch construction specified the main (2.5 min audio blocks, 25 anchors, positives per anchor, total 100 audio blocks). Inference runtime is measured following the segment-base based evaluation protocol (5s hop size, T =00 R Rmin). Retrieval time corresponds to database with 2000 candidates, and grows linearly with them approaches.
APPROACH
PARAMETERS
TRAINING [S/BATCH]
INFERENCE
EMBEDDING [MS/SONG] RETRIEVAL [MS/QUERY]
COVERHUNTER-COARE
28 M
0.68
22.5
2.9
CQTNET
35 M
0.48
57.6
2.0
DVINET+
11 M
0.38
60.9
2.2
BYTECOVER3 f
0.71
27.4
BYTECOVERI/2 1/2
202
0.50
62.2
5.1
CLEWS
199 M
1.19
40.3
3.5
17


Reviews:
[{"review_id": "y94N3NgNoI", "reviewer": "Reviewer_oJzQ", "summary": "The proposed paper is about cover song detection or musical version matching task. This task is to identify different versions of a same song, for example, if there exist an original song and this song has been sung by other singers with different vocal styles, this can be regarded as a cover song. Also, if there exist some electronic or sped-up remix of the original song, that's also considered as a different version. So, the task is to detect these different versions of a song in a single category. To do that, most of the previous researches conducted song or track-level matching methods. So, they compared the full two audio and measure the similarity. However, in this work, the authors tried to compare the segments of the two audio and proposed aggregation method over segment-to-segment similarity matrix. They compared few possible methods and among that the proposed aggregation method, best-pair without replacement, reached the SOTA level performance on this task. To training segment-level similarity measure network, they used contrastive learning with weak labels. So, if the two audio are in the same version, then they regarded all the segments in the songs are similar even though there exist some possibility that the two segments have different characteristics (e.g. different section with different styles) even through they are in the same version. Overall, the paper is well-written and easy-to-read. However, I personally think more contribution might be needed to have solid novelty.\n\nI think the core novelty of the proposed method lies in segment-level similarity learning with the proposed aggregation method. However, as the authors noted that, this can be regarded as one of a triplet sampling method, or it can also be viewed as one of the multiple instance learning method. Therefore, if the main contribution lies in the method, then more experimental results might be needed on datasets with different modalities, etc.\n\n## update after rebuttal\nAs the authors replied that their method is novel on soft sampling method within a song, I changed the decision to weak accept. Thanks for the hard work.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for his/her comments and insight. We want to clarify two misunderstandings by the Reviewer, and comment on the further analyses part. We think that the Reviewer may increase his/her score once these two misunderstandings are clarified, as they are the only two strong criticisms we could find in the review.\n\n1) The Reviewer mentions that the method regards all the segments in the songs as similar (\"then they regarded all the segments in the songs are similar\") ---- We want to stress that it is actually the opposite. As mentioned in the manuscript, considering all compared segments between two songs as similar corresponds to the $R_{\\text{mean}}$ strategy. We clearly motivate why this is unrealistic for music (Sec. 3.1) and further corroborate that models do not learn well with this reduction (Sec. 5). In addition, we have to note that all reduction strategies developed after $R_{\\text{mean}}$ go precisely in the direction of avoiding the situation where \"all segments in the songs are similar\". In particular, the proposed $R_{\\text{bpwr-r}}$ reduction explicitly avoids that by construction.\n\n2) The Reviewer mentions that the proposed approach lacks novelty, \"as the authors noted that, this can be regarded as one of a triplet sampling method, or it can also be viewed as one of the multiple instance learning method.\" ---- We believe that this is not true, and we want to state that we do not explicitly mention that. On the one hand, none of the proposed techniques corresponds to triplet sampling, nor we use any triplet loss (except for showing better results with the proposed loss). On the other hand, the fact that we try to conceptually relate some of the reduction techniques we propose (e.g., $R_{\\text{min}}$) to concepts employed in other multiple-instance learning methods (e.g., hard mining of examples) does not mean that we use a similar technique or implementation. Actually, we propose $R_{\\text{bpwr-r}}$, which we do not find a clear parallelism with hard, semi-hard, or random instance mining. Besides, the development of the proposed loss is novel and does not employ triplet sampling or existing multiple instance learning methods.\n\nRegarding further analyses with different modalities, we want to recall that our main focus is on the audio retrieval field, and that the manuscript is submitted to the \"Applications\" track, using the \"Application-Driven Machine Learning\" tag. We believe a number of the techniques we employ could be transferred to other domains as they do not include audio-specific processing. However, demonstrating it in a rigorous way would require much more experimentation and results which are beyond the scope of the current manuscript. Regarding the analysis of some specific examples and on how two audio recordings are compared, we are studying to possibility to add an additional figure to the manuscript, as this was also mentioned by another reviewer. We thank the Reviewer for this suggestion.", "claims_and_evidence": "Yes, the experimental results supports the superiority of the proposed method.", "methods_and_evaluation_criteria": "The authors used the conventional evaluation methods for the cover song identification task, so it's good.", "theoretical_claims": "Multiple-instance learning using segment-level similarity learning with instance-level aggregation method seems solid.", "experimental_designs_or_analyses": "More experiments with datasets with different modalities might enhance the versatility of the proposed method. Also, if more analysis on some examples, like how the actual two audio are compared or highlighted in the proposed aggregation method might be helpful for the readers.", "relation_to_broader_scientific_literature": "Multiple-instance learning using segment-level similarity learning with instance-level aggregation method might be the one for broader readers.", "essential_references_not_discussed": "It's good.", "other_strengths_and_weaknesses": "Dive into the segment-level similarity comparison for cover song detection itself is one that need to be conquered in the future, so motivation is good.", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Avd1xhvRNm", "reviewer": "Reviewer_s9r9", "summary": "This paper presents a novel approach for musical version matching at the segment level using weakly labeled audio segments and a supervised contrastive learning framework. The authors highlight two key contributions: (1) a method to effectively learn from weakly annotated musical segments, which enables more granular version matching beyond track-level approaches, and (2) a modified contrastive loss function that improves retrieval performance by incorporating decoupling, hyper-parameter tuning, and geometric considerations. The proposed method achieves state-of-the-art performance at the track level and delivers a significant breakthrough in segment-level evaluations. The generalizability of this approach suggests potential applications beyond music, such as other weakly labeled retrieval tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for his/her assessment of our work and the constructive feedback. We first answer to the questions posed by the Reviewer and later discuss on a few further aspects that appear in the main review.\n\n1) Label inconsistencies ---- Like the big majority of machine learning models, our model assumes a low (or nonexistent) number of noisy labels. Hence, it does not explicitly handle song label inconsistencies. In the weak label assignment, most of our reduction strategies try to assign the song label to the best (lowest distance) segments in a pairwise comparison, while discarding the rest of the comparisons. According to the obtained results, we believe that the model ends up assigning the weak labels to the correct segments pairs (otherwise we would not be obtaining good performance). Nonetheless, it is true that perhaps one could implement further mechanisms to explicitly handle possible inconsistencies. We plan to actively consider this situation in the next version of our approach.\n\n2) Contrastive loss modifications ---- We think our modifications to the contrastive loss could be beneficial for other retrieval tasks, and we think the examples provided by the Reviewer are perhaps the ones that have more potential, as they also correspond to domains where sequence information is important. As mentioned to Reviewer oJzQ, the extension of the proposed approach to other domains is out of the scope of the current manuscript, but nonetheless a very interesting direction for future work.\n\n3) Self-supervised learning ---- We did consider the use of self-supervised learning techniques in addition to weak supervision. However, we did not have a clear idea on which self-supervised/proxy task to include. Predicting future segments as an aid to weak labeling was one task we played with during our experimentation, but we were not able to obtain any meaningful performance increase. We also studied additional data augmentations to create different views of the segments (we share them in the code), but again found no benefit when evaluating with the largest data set we consider (DVI). We hypothesize that such data set is large enough to not benefit from trivial data augmentations, and that the variability it naturally includes is more informative than the artificial augmentations we introduced.\n\n4) Analysis of failure cases ---- This is a good suggestion, and we are studying which is the best way to include such analysis in the manuscript or the Appendix. Unfortunately, we have not developed a demo of the system yet. Informally, we can comment that it is hard to find a clear denominator for failure cases, as most of the time we do not see a clear or systematic failure that we could address in a further development. For instance, some true positives show a spectacular robustness towards genre variations, while still we find similar genre variations of the original song among the false negatives.\n\n5) Ablation of loss modifications ---- We agree with the Reviewer. It was our original intention to include ablations for every loss modification. However, we finally did not do it due to two main reasons. On the one hand, the decoupling modification introduces just a marginal improvement while, in the other hand, we cannot isolate the other modifications as each one is conceptually and practically related to the other. For instance, the introduction of the smooth threshold $\\varepsilon$ is needed because the sum of exponential terms inside the logarithm can become zero, and the latter is due to switching from cosine to Euclidean distance, which can grow indefinitely and cause a numerical error. Therefore, we cannot study the effect of switching from cosine to Euclidean without introducing the soft smoothing threshold $\\varepsilon$. We can however study the effect of different values of the latter, which we do in Sec. 5.\n\n6) Segment-level boundary effects ---- We think we do not fully understand the question, but we will nonetheless try to address it the better we can. In the way we perform the segment-based evaluation, segments are extracted before concatenating all candidate songs, and segment embeddings (computed independently) are then stacked together in the candidate database. At the retrieval stage, we know which is the song identifier for each segment embedding, and can thus avoid mixing different songs. Since segment embeddings are computed independently, we also avoid boundary effects with other segments and songs.\n\nRegarding confidence intervals, we want to mention that we do report 95% confidence intervals in all our results (both tables and figures). We will add further clarification in the Table captions to avoid such confusion. Regarding additional citations to self-supervised literature in music information retrieval, we are totally open to include them. If the Reviewer has specific examples in mind, we will consider them in the next revision of the manuscript.", "claims_and_evidence": "The claims made in the paper are well-supported by empirical evidence, particularly in demonstrating improvements in both track-level and segment-level musical version matching. The proposed contrastive loss function is compared against existing triplet and classification-based losses, and the reported results indicate superior performance. However, further clarity on the sensitivity of hyper-parameters in different datasets would strengthen the claim of generalizability. Additionally, it would be beneficial if the authors provided an ablation study to isolate the contributions of each component separately.", "methods_and_evaluation_criteria": "The methods and evaluation criteria appear sound and well-aligned with the problem at hand. The choice of contrastive learning is appropriate given the retrieval-based nature of the task. The benchmark datasets used for evaluation are suitable, although further discussion on the robustness of weakly labeled annotations would be helpful. It would also be valuable to see a qualitative analysis of retrieval cases where the proposed method succeeds or fails, to provide deeper insights into its behavior.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The experimental design is solid, with clear comparisons to existing methods. The results convincingly demonstrate the superiority of the proposed approach, but additional experiments on unseen datasets or different genres of music would further support the generalizability claim. It would also be beneficial to include confidence intervals or statistical significance tests to reinforce the reliability of the reported improvements.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in contrastive learning, weakly supervised learning, and music retrieval. It appropriately references fundamental works in these areas but could benefit from additional citations related to self-supervised learning in music information retrieval.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a well-motivated problem with clear practical applications.\n2. The proposed approach outperforms prior state-of-the-art methods in both track-level and segment-level evaluation.\n3. The use of weakly labeled data is an important contribution that has broader applicability beyond musical version matching.\n4. The contrastive loss modifications are well-motivated and lead to clear performance gains.\n\nWeaknesses:\n\n1. The impact of weakly labeled data quality on model performance is not deeply analyzed.\n2. The paper could benefit from more qualitative examples to illustrate cases where the method succeeds or fails.\n3. The generalizability claim would be stronger with additional experiments on varied datasets beyond music retrieval.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Potential Issues with Weakly Labeled Data Assumptions. The method assumes that weakly labeled data is sufficiently informative for training, but in many cases, weak labels can introduce ambiguities (e.g., segments may contain overlapping themes from different songs).\nIf a weak label incorrectly associates a segment with a version it does not belong to, how does the model handle such inconsistencies?\n2. Would the proposed contrastive loss modifications be beneficial for other retrieval tasks, such as speech or video?\n3. Have you considered integrating self-supervised learning techniques in addition to weak supervision?\n4. Can you provide further analysis on failure cases and potential biases in retrieval?\n5. Ablation Studies on Loss Modifications Are Limited. The proposed contrastive loss function is modified in multiple ways, but the individual contributions of each modification are not clearly separated. An ablation study isolating each modification (decoupling, geometric tuning, etc.) would clarify which aspects contribute most to performance.\n6. Segment-Level Evaluation Requires More Justification. While the paper claims a “breakthrough” in segment-level version matching, there is little discussion on the limitations of segment-based evaluation. Segment retrieval can be highly sensitive to boundary effects—how does the model perform when segments are slightly misaligned?\n7.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLGmR5Ifi8", "reviewer": "Reviewer_ysSi", "summary": "In this paper, authors propose a method (CLEWS) to learn from weakly annotated segments, together with a contrastive loss variant that outperforms well studied alternatives. State-of-the-art performance on both track-level and segment-level evaluations, significantly outperforming existing methods. The approach addresses the mismatch between track-level annotations and segment-level inference needs in real-world applications.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank a lot the Reviewer for his/her assessment and constructive feedback. We answer to and discuss on the questions posed by the Reviewer below. As suggested by the Reviewer, we are considering adding a figure to describe the CLEWS model structure and pipeline, especially along with the shared code to facilitate understanding.\n\n1) Computational cost ---- This is a good suggestion. Given track lengths of $N$ and $M$, the computational cost of $R_{\\text{bpwr-r}}$ is $O(NM+N+M)$. Essentially, like the cost of the other reductions considered in the paper, it is quadratic (considering $N\\approx M$). We will add the cost of all reductions to the Appendix. It should be noted that, with non-overlapping 20-second segments (the configuration we use for training), $N$ and $M$ are relatively small. For instance, for a 5-minute song we get $N=15$ segments, which is orders of magnitude smaller than the typical length we usually see in other quadratic and sequential operations like the ones performed, for instance, in Transformer's cross-/self-attention. Regarding computational resources used during training, we used two NVIDIA H100 80GB GPUs. We will add this information to the manuscript. Regarding retrieval runtime, we want to clarify that all approaches, when operating at a segment level, incur a natural increase as compared to track-based retrieval. Thanks to the Reviewers' suggestion, we collected these numbers and will report them in the Appendix. We paste them below for convenience. In summary, we do not see any big issue that would severely limit the utilization of the proposed approach in front of the other evaluated segment-based alternatives.\n\n2) Different kinds of musical material --- In contrast to the SHS data set, which essentially features pop and rock songs, the DVI data set contains a wide variety of musical genres and material (from jazz standards to electronic versions, including musicals, rock, reggae, etc.). The test set of DVI is very large compared to existing evaluation standards, including over 100k different tracks. We believe that DVI-Test is representative enough to assess the generalizability of the approach with regard to western contemporary music.\n\n3) Fine-grained variations in music versions --- We believe the DVI data set is representative enough of the real world, including cases such as the ones outlined by the Reviewer (for instance the one in which lyrics remain consistent while the musical form varies significantly). Our approach, like the vast majority of version matching approaches, focuses on the audio content, and it is true that it may underperform in cases where only the lyrics, but not the melody, are preserved. For those cases, which are not the most common ones for musical versions (according to what we see in the available data sets and sources), including a lyrics transcription system and combining with CLEWS could be a fruitful approach.\n\n4) Song plagiarism --- This is an interesting aspect. Song plagiarism is a controversial and multifaceted topic in music copyright infringement. Often times, court decisions on music plagiarism cases are not only rooted on musical concepts and similarity, but go beyond that by considering external factors such as the possibility of access to the source by the offender, the amount of money involved, the resources employed or the collaborating artists, etc. Focusing on similarity alone, we believe the current data sets, and especially DVI, have the potential to start supporting comprehensive similarity searches, at least for western contemporary music (with the exception perhaps of some niche genres like experimental music).\n\n---\nTable: Training and inference runtimes (informal) using a single NVIDIA H100 GPU. Training runtime is measured using a batch construction as specified in the main text (2.5 min audio blocks, 25 anchors, 3 positives per anchor, total of 100 audio blocks). Inference runtime is measured following the segment-based evaluation protocol (5 s hop size, $\\tau=20$ s, $R=R_\\text{min}$). Retrieval time corresponds to a database with 2000 candidates, and grows linearly with candidates.\n```\nApproach             Parameters   Training [s/batch]                  Inference\n                                                       Embedding [ms/song]   Retrieval [ms/query]\n- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\nCoverHunter-Coarse   28M          0.68                 22.5                  2.9\nCQTNet               35M          0.48                 57.6                  2.0\nDVINet+              11M          0.38                 60.9                  2.2\nByteCover3           969M         0.71                 27.4                  5.1\nByteCover1/2         202M         0.50                 62.2                  5.1\nCLEWS                199M         1.19                 40.3                  3.5\n```", "claims_and_evidence": "1) The effect of hyperparameters on the loss function is presented in fig3 and appendix a, which alleviates my concerns regarding the potential excessive sensitivity of the loss function to hyperparameter variations.\n\n2) The superiority of CLEWS over existing methods (table 2, appendixc, fig. 2) is supported by extensive experiments on two datasets (DVI and SHS). \n\n3) The effectiveness of bpwr-r reduction and modified A&U loss is validated through ablation studies in table3 and table4", "methods_and_evaluation_criteria": "Strgenths:\n1) Segment reduction strategies (e.g., bpwr-r) effectively address partial matching and avoid consecutive segment biases.\n2) The modified loss decouples positives/negates and simplifies hyperparameters, aligning with contrastive learning principles.\n\nEvaluation:\n1) MAP and NAR are appropriate metrics for retrieval tasks. The corrected NAR formula (Appendix B) improves interpretability.\n2) Segment-level evaluation protocols (best-match vs. random-segment) provide comprehensive insights.", "theoretical_claims": "I have check the theoretical proofs presented in the paper and found them to be logically sound and free from inconsistencies.", "experimental_designs_or_analyses": "I have carefully reviewed the experimental setup proposed in the paper and found it to be well-designed and comprehensive. The experiments include detailed analyses of hyperparameters, and the descriptions of data augmentation techniques and evaluation protocols are both sufficient and specific. Additionally, it would be beneficial to include a brief statement regarding the computational resources consumed during training (e.g., GPU memory usage, such as 24GB) to provide further clarity on the practical requirements for reproducing the results.", "relation_to_broader_scientific_literature": "The work builds on contrastive learning (Wang & Isola, 2020), triplet losses (Schroff et al., 2015), and version matching (Serra`, 2011; Yesiler et al., 2021). Key innovations include adapting contrastive learning (A&U loss ) to supervised settings and introducing segment reduction for weak supervision, advancing partial matching in music retrieval.", "essential_references_not_discussed": "I have thoroughly reviewed the theoretical foundations presented in the paper, along with their associated references, and found no instances of related work that were overlooked or inadequately discussed. The ideas and contributions claimed in the paper are original and do not appear to be derived from prior works without proper attribution. Furthermore, the experiments conducted in the study are compared against several state-of-the-art (SOTA) methods from recent literature, ensuring a fair and up-to-date evaluation of the proposed approach.", "other_strengths_and_weaknesses": "Strengths:\nPractical impact: Addresses real-world needs for segment-level matching in copyright detection and generative model attribution.\n\nWeaknesses:\nLimited discussion on computational costs (e.g., bpwr-r’s runtime for large-scale datasets).", "other_comments_or_suggestions": "Add a fig to describe the CLEWS model structure and the pipeline", "questions_for_authors": "Q1: How does the computational cost of bpwr-r scale with track length (e.g., 10-min vs. 2.5-min tracks)? Would this limit real-world deployment?\nImpact: A high runtime for bpwr-r would weaken practicality claims.\n\nQ2: Have you tested CLEWS on different kinds musical to assess generalizability not only focus on the testset.\nImpact: Positive results would strengthen the claim of broader applicability.\n\nQ3：In the context of music version matching, certain musical pieces are expressed in diverse forms, such as the same lyrics being rendered in distinct musical styles or rappers performing familiar songs with different beats. Can the proposed approach, which leverages distance metrics and contrastive learning, effectively distinguish such fine-grained variations in music versions? This raises questions about the method's ability to capture subtle differences in musical expression, particularly when the underlying content (e.g., lyrics) remains consistent while the musical form (e.g., rhythm, instrumentation) varies significantly.\n\nQ4：Do the DVI and SHS datasets used in the current experiments sufficiently support training models for verifying song plagiarism and evaluating AI-generated music? While I acknowledge the challenges in collecting a more objective and larger dataset, this limitation should still be recognized as a constraint of the current work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dvLZ377eM9", "reviewer": "Reviewer_phRw", "summary": "The paper addresses the challenge of matching different renditions of the same musical piece at segment level. The authors propose a method that uses pairwise segment distance reductions for weakly annotated data, combined with a modified alignment-and-uniformity contrastive loss. This approach achieves state-of-the-art track-level results and a breakthrough in segment-level evaluation", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the Reviewer for his/her assessment and comments. We now answer and discuss about the main questions raised in the Review.\n\n1) Additional data sets ---- This is a relevant suggestion that is worth discussing. We originally computed the results for covers80 and we paste them below. However, we finally decided to not include them in the manuscript due to the multiple issues and the low representativity of covers80. Covers80 is a very small data set, consisting of only 80 tracks (as a reference, DVI-Test contains around 100k tracks). The small size of covers80, together with the fact that different existing approaches train with different existing datasets in order to report results on covers80, makes the data set problematic and non-reliable as a reference for performance measurement (this can also be seen with the high confidence intervals reported for covers80). The Da-TACOS data set comes with pre-extracted features, which difficults a fair comparison with audio-input methods due to different feature extraction pipelines and data augmentation procedures. Like covers80, Da-TACOS does not provide a train set, which additionally creates issues for fair comparison as models are typically not trained with the same data. Finally, and related to the latter, our internal analysis shows that around 90% of Da-TACOS is contained in the DVI data set. Thus, we consider that using DVI is preferable (and enough) for comparing approaches.\n\n2) Analysis comparing runtimes ---- Thanks to the Reviewers' suggestion we performed both a complexity and a runtime analysis, and we will include them in the Appendix of the manuscript. For convenience, we reproduce the runtime results in the answer to Q1 from Reviewer ySSi, who asked a similar question, and we refer to it for further details due to space limitation. In summary, we do not see any big issue that would severely limit the utilization of the proposed approach in front of the other segment-based evaluated approaches.\n\n---\nTable: Track-level evaluation on the Covers80 test set. We distinguish between models trained on DVI and models trained on SHS. The +- symbol denotes 95% confidence intervals.\n```\nApproach                        Trained on DVI               Trained on SHS\n                                NAR         MAP              NAR         MAP\n- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \nCoverHunter-Coarse (our impl)   7.85+-2.37  0.558+-0.068     5.80+-2.52  0.735+-0.061\nCQTNet (reported)               n/a         n/a              n/a         0.840\nCQTNet (our impl)               3.33+-2.05  0.869+-0.048     2.10+-1.14  0.861+-0.050\nDVINet+ (our impl)              0.88+-0.51  0.897+-0.043     1.95+-1.35  0.889+-0.045\nByteCover3 (our impl)           1.13+-0.88  0.900+-0.043     0.96+-0.75  0.916+-0.038\nByteCover3 (reported)           n/a         n/a              n/a         0.927\nByteCover2 (reported)           n/a         n/a              n/a         0.928\nByteCover1/2 (our impl)         1.46+-0.93  0.905+-0.042     1.67+-1.18  0.929+-0.038\nCLEWS (proposed)                0.72+-0.63  0.955+-0.030     1.31+-1.11  0.944+-0.034\n```", "claims_and_evidence": "Yes. The claims are well supported by the experiment results.", "methods_and_evaluation_criteria": "Yes. The authors propose methods (best-pair distance reductions + a modified contrastive loss CLEWS) that directly address the need to match musical versions at the segment level and track level. Their evaluation metrics MAP and NAR are make sense for the task.", "theoretical_claims": "Yes, I reviewed the CLEWS loss, especially the decoupling contrastive loss into alignment and uniformity terms and the gradient analysis involving γ and ε. I did not encounter any obvious mistakes.", "experimental_designs_or_analyses": "Yes. The experiments include both track-level and segment-level evaluations and compare with baselines. The paper also includes ablation studies on the impact of various reduction functions, loss functions, and hyperparameters.", "relation_to_broader_scientific_literature": "The paper’s contributions are mainly in the following two parts:\n1. Segment-level music match approaches: although some recent studies have considered fine-grained analyses beyond track level, like ByteCover3 (Du et al., 2023) and CoverHunter (Liu et al., 2023), fully segment-based approaches to musical version detection are lacked. This paper is the first to consider an entirely segment-based approach for both learning and retrieval stages. \n2. Novel supervised contrastive loss: earlier methods (CQTNET, DORAS&PEETERS, MOVE/RE-MOVE, PICKINET, LYRACNET, BYTECOVER, COVERHUNTER, DVINET/DVINET+) often used classification-style losses or triplet losses like shown in Table 1; instead, the authors introduce a supervised variant of alignment and uniformity specifically for this task.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n1. Handling short segments and showing improved results at both track and segment levels.\n2. Proposes a novel supervised contrastive loss for detecting musical versions\n\nWeaknesses:\n1. It would be beneficial to include a detailed analysis comparing running times across different methods.\n2. The experimental evaluation could be further strengthened by adding results on two additional widely used datasets: Covers80 and Da-tacos.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The experimental evaluation could be further strengthened by adding results on two additional widely used datasets: Covers80 and Da-tacos.\n2. (Optinal) It would be beneficial to include a detailed analysis comparing running times across different methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel method to train a model for musical version matching using weakly-labeled audio segments.

- Strengths
	- Exploiting weakly-annotated segments for the training
	- Proposes novel supervised contrastive loss for music version matching
	- The proposed method shows superior performance over previous methods.

Besides the strengths, the reviewers suggested further qualitative case studies, more rigorous ablation studies, or applying it to other tasks than music version matching, although these suggestions would be difficult to be included in a single paper. 

Overall, this is a strong paper with clear application and technical contribution.
Decision: Accept (poster)