AMG-Embedding: a Self-Supervised Embedding Approach for Audio Identification

Authors: yuhang  su
fan  zhang
qiming  xu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 06:31:20

Abstract: Audio Identification aims to precisely retrieve exact matches from a vast music repository through a query audio snippet. The need for specificity and granularity has traditionally led to representing music audio using numerous short fixed-duration overlapped segment/shingle features in fingerprinting approaches. However, fingerprinting imposes constraints on scalability and efficiency, as hundreds or even thousands of embeddings are generated to represent a typical music audio. In this paper, we present an innovative self-supervised approach called Angular Margin Guided Embedding (AMG-Embedding). AMG-Embedding is built on a traditional fingerprinting encoder and aims to represent variable-duration non-overlapped segments as embeddings through a two-stage embedding and class-level learning process. AMG-Embedding significantly reduces the number of generated embeddings while achieving high-specific fragment-level audio identification simultaneously. Experimental results demonstrate that AMG-Embedding achieves retrieval accuracy comparable to the based fingerprinting approach while consuming less than $1/10th$ of its storage and retrieval time. The efficiency gains of our approach position it as a promising solution for scalable and efficient audio identification systems.

Main paper: 
 introduction: 
 Audio identification, often referred to as audio fingerprinting [3] , is an audio-based music information retrieval (MIR) [17] task. It plays a crucial role in various applications, including the identification of unknown music from microphone input, the detection of duplicated music tracks. Precisely recognizing music from a vast database when presented with an input audio snippet, especially when the snippet is just a small fragment exposed to signal distortions like noise, reverberation, or compression, poses a significant challenge.Audio identification is known for its exacting demand in terms of specificity and granularity, involving high-specific fragment-level retrieval tasks [17] . This implies that an audio identification system is expected to precisely return copies of the query, which might consist of only a short fragment. To meet the high-specificity fragmentlevel requirement, cutting-edge audio identification approaches [4, 16, 35, 60] opt to divide music audios into numerous consecutive short fixed-duration overlapped segments or shingles. Each segment is represented as an 𝐿 2 -normalized feature/embedding, where the inner product of features can measure similarities between segments. Therefore, audio identification can be viewed as a feature sequence matching problem. Moreover, owing to the fragmentlevel demands of audio identification, the duration and overlapping length of these segments are typically quite short. For instance, in neural fingerprinting [4] , the duration is 1𝑠 with a 0.5𝑠 overlap. As a result, a typical music track, even one lasting several minutes, may necessitate hundreds of such features for representation. Consequently, these audio identification approaches face scalability and efficiency challenges due to the expanding feature repositories in large-scale music databases.Recently, representing a long variable-length music with an embedding, known as Music Embedding, has demonstrated success in MIR, such as music classification [62] , recommendation [23] , and version identification [14] . However, existing Music Embedding methods primarily target low-specificity document-level MIR tasks and may not simultaneously meet the specificity and granularity requirements of audio identification.In this paper, we introduce a novel self-supervised Angular Margin Guided Embedding approach (AMG-Embedding) for music audio identification 1 . AMG-Embedding aims to represent variableduration (but with an upper limit, for example, less than 30 seconds) audio segments as a single embedding, while ensuring that the inner product can still measure the similarity between segments with different durations. Moreover, there is no longer a necessity for overlap between segments. Hence, AMG-Embedding allows us to represent music with relatively few segments and embeddings. Figure 1 : Audio identification using traditional fingerprinting and our AMG-Embedding. To measure similarity between a 36𝑠 audio and a 12𝑠 query audio which is actually a segment of the former, the fingerprinting method requires generating 94 features in total, whereas AMG-Embedding only requires 6 features.As illustrated in Figure 1 , consider a 36𝑠 audio, 71 features are required to represent the entire audio by using the traditional fingerprinting methods with fixed-duration segments of 1𝑠 and 0.5𝑠 overlap. In contrast, using AMG-Embedding, we can segment the audio into 4 non-overlapping segments (three 10𝑠 and one 6𝑠) with an upper duration limit of 10𝑠(𝐷𝑢𝑟 𝑚𝑎𝑥 as referred to in the paper), requiring only 4 embeddings to represent the entire audio. Therefore, in the case of the 12𝑠 querying audio, the fingerprinting generates 23 features, while AMG-Embedding only produces 2 embeddings. Furthermore, inner product of features can still measure the similarity between corresponding segments, even with different durations. In other words, the more identical content shared between two audios, the larger the inner product of their features. Consequently, both long music pieces and short snippets can be represented with much shorter feature sequences compared to existing fingerprinting systems, allowing feature sequence matching to remain effective for measuring audio similarity.AMG-Embedding is a two-stage embedding approach. The first stage employs existing fingerprinting approaches to extract feature sequences from input audios, and then these feature sequences are further compressed in the second stage by using a standard Transformer Encoder trained with our proposed class-based Proxyanchor Aligned Margin loss function PAM-Loss. AMG-Embedding offers a significant advantage in terms of retrieval efficiency and scalability compared to previous fingerprinting methods, while maintaining a comparable level of accuracy. The main highlights of the work are summarized as follows:(1) We proposed a two-stage approach AMG-Embedding to represent a variable-duration audio as an embedding, which can significantly reduce the size of feature sequences while concurrently meeting the granularity and specificity requirement of audio identification. Moreover, any SOTA fingerprinting approach can seamlessly integrated in AMG-Embedding. (2) We examined the constraints associated with commonly used proxy-based losses in the context of variable duration audio embedding. In response, we introduced a new proxy-based loss, PAM-Loss, specifically crafted to enhance the efficacy of our embeddings. PAM-Loss also stands as a commendable endeavor to harness class-level learning, aiming to elevate the performance of Music Information Retrieval (MIR) tasks. (3) We performed experimental analyses, presenting both quantitative and qualitative results to convincingly showcase the robust effectiveness of AMG-Embedding. 
related work 2.1 audio identification: 
 Audio identification is typically tackled in two steps: the extraction of feature/fingerprint sequences from audio, and the construction of indexes for retrieval. Handcrafted features have been extensively employed. Shazam [41] extracts sets of spectral peaks in the spectrogram as fingerprintings. Similar quad combination was adopted in [37] . Binary fingerprints based on energy changes across spectral-temporal space are employed in [19] . Waveprint [2] computes fingerprints using wavelets, and [34] generates fingerprints by using pseudo-sinusoidal components. Nevertheless, handcrafted features often fall short in real-world applications. Recently, Deep learning, particularly unsupervised learning, has been applied and achieves impressive performance in audio identification. Autoencoders consisting of LSTM layers are proposed in [1, 50] , and CNN models [4, 16, 35, 57, 60] are widely employed as encoders recently. These encoders are trained to generate embeddings of each audio segment using the metric learning loss functions, such as Contrastive loss [26] , Triplet loss [16, 50, 60] , N-Pairs/InfoNCE/NT-Xent loss [4, 35, 57] . In these approaches, each source music and query snippet is represented as a sequence/set of features for retrieval, which results in the creation of massive feature databases. For example, 56M segment features are generated for a subset of the fma-full [7] consisting only 278s audio clips of 100k songs. To ensure efficient retrieval, audio identification systems must employ fast and effective indexing algorithms, such as hashing [2, 35, 41, 60] and inverted indexing [4, 16] .A logical approach is to use a compact embedding to represent either a short audio snippet or a complete music as variable duration embeddings. Embedding-based approaches can obtain compact representations that expedite the retrieval process, reduce storage requirements, and enable efficient similarity estimation. As mentioned above, music embedding have recently found applications in certain low-specificity document-level MIR tasks, such as music classification [9, 10, 52, 62] , recommendation [5, 23, 42] , version identification(also referred as cover song detection, CSI ) [11, 13, 15, 33, 53, 54, 58, 59] , etc. In many of them, MIR tasks are formulated as classification or metric learning challenges employing CNN/Transformerbased encoders to map music to their embedding vectors. For instance, Swin Transformer [25] and InfoNCE loss are used in S3T [62] for music classification, ResNet-IBN and classification-triplet joint losses are employed in ByteCover series [13] [14] [15] for cover song identification. It should be noticed that ByteCover3 implements short-query CSI by borrowing finger printing techniques in audio identification, and its segment length is fixed 20𝑠 with 10𝑠 overlapping. In addition, several self-supervised learning approaches [31, 38, 55, 63] have emerged with a focus on learning music representation (music embedding). These embeddings can be applied in various downstream music-related tasks, also primarily in music classification and version identification. Embedding solutions have not yet been effectively applied to audio identification due to their limitations in terms of specificity and granularity.Metric learning seeks to map data into an embedding space where similar data points cluster together, while dissimilar ones are separated by a significant margin. There are two main paradigms in metric learning: class-level learning and pair-wise learning. Class-level learning usually incorporates a weight matrix, where each column (serve as a proxy) corresponds to a particular class in order to transform the embedding space into class probability vectors. These methods are also known as proxy-based learning. The most basic approach is the normalized softmax loss [44, 61] , where the columns of the weight matrix are L2 normalized. A variation of this approach is ProxyNCA [27] , which employs cross entropy on the Euclidean distances between embeddings and the weight matrix. Several additive angular margin loss functions, including SphereFace [24] , SphereFace2 [49] , CosFace [43, 45] , and ArcFace [8] , further modify the cross entropy loss with angular margins, and achieve great improvement. Additionally, the SoftTriple loss [30] expands the weight matrix to have multiple columns per class to provide more flexibility.Pair-wise learning directly learns similarity between training samples in the embedding space and does not require proxies. Contrastive loss [18] and triplet loss [47] are two fundamental approaches. Numerous losses have been developed based on them, including the angular loss [46] , the margin loss [51] , the lifted structure loss [28] . Recently, mining more relationships (positive and negative pairs) of samples becomes popular, such as the N-Pair loss(also known as InfoNCE and NT-Xent) [6, 29, 36] , the tuplet margin loss [56] , and the PSL [48] .Metric learning is widely adopted for addressing MIR tasks. However, it's important to note that despite its notable success in some retrieval approaches, there has been relatively limited emphasis on class-level learning in the context of MIR. In many scenarios, pair-wise learning is preferred because it is more convenient to specify similarities between samples using pair or triplet relationships. Furthermore, it's essential to highlight that current classlevel/proxy-based learning approaches are not particularly suitable for variable duration audio embeddings. A more detailed discussion on this topic will be provided in Section 3.3.1. 
proposed approach: 
 A self-supervised embedding-based audio identification approach, named AMG-Embedding, is proposed in this paper. Embeddingbased methods are well-suited for low-specificity document-level MIR tasks. However, their application in audio identification has proven to be elusive. To tackle this issue, AMG-Embedding employs a two-stage embedding strategy tailored for audio identification.AMG-Embedding is a two-stage embedding approach, where a piece of variable-duration music audio is converted into a sequence of segment features by using a self-supervised fingerprinting system (neural fingerprinting [4] in our implementation) in the first stage (named Fingerprinting Stage), then this sequence is further compressed to a much shorter embedding sequence by employing a transformer-based [35] encoder trained with PAM-Loss, a proxybased metric loss, in the second stage (named Embedding Stage).Fingerprinting Stage. In this stage, we adopt a self-supervised neural fingerprinting approach [4] . As illustrated in Figure 2 into the corresponding time-frequency representation, denoted as {𝒔 𝒕 } 𝑛 𝑡 =1 . The number of 𝒔 𝒕 mirrors the duration of the input audio 𝑿 . Subsequently, we utilize a fingerprinting encoder referred to as 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 , to project {𝒔 𝒕 } 𝑛 𝑡 =1 into a sequence of 𝐿 2 -normalized 𝑑dimensional embedding vectors {𝒛 𝒕 ∈ R 𝑑 } 𝑛 𝑡 =1 used for retrieval in common fingerprinting systems. Due to the need for granularity at the fragment-level, each segment must have a small time duration, and adjacent segments also need to have an overlap, for instance, the segment length is 1s and the overlap is 0.5s indicated in [4] .FullyEmbedding Stage. In this stage, the sequence {𝒛 𝒕 } 𝑛 𝑡 =1 should be further compressed into a new much shorter feature sequence using 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 . In our implementation, 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 is a standard Transformer Encoder [40] in which an Average Pooling layer and a Fully Connected layer are applied to output the final embedding. It's worth noting that 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 comes with an input size limitation of 𝐿 (corresponding to the upper duration 𝐷𝑢𝑟 𝑚𝑎𝑥 ), which results in the sequence {𝒛 𝒕 } 𝑛 𝑡 =1 being divided into ⌈𝑛/𝐿⌉ non-overlapping sub-sequences. Each sub-sequence is then independently fed into 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 to generate its final fixed-length 𝐿 2normalized 𝑓 -dimensional embedding 𝒀 𝒊 ∈ R 𝑓 . Consequently, a variable-duration audio 𝑿 is mapped to ⌈𝑛/𝐿⌉ embeddings, denoted as {𝒀 𝒊 } ⌈𝑛/𝐿⌉ 𝑖=1 . In our implementation, 𝐿 is typically set to 32 or 64 (corresponding to about 15𝑠 and 30𝑠 in duration), significantly reducing the embedding size. Note that 𝑓 can differ from the dimension of the fingerprinting embedding 𝑑, but in our implementation, we set 𝑓 = 𝑑 because the objective of AMG-Embedding is reducing feature. Moreover, the same dimension of 𝒛 𝑖 and 𝒀 𝑖 also can simplify Proxy Initialization process introduced in Section 3.3.2.As the result, a music is represented as a sequence of short embedding vectors after Embedding Stage. Consequently, these embedding vectors can be used to build indexing for searching. 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 and 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 are employed in AMG-Embedding for two-stage embedding as 𝑿 -→ {𝒛 𝒕 } 𝑛 𝑡 =1 - → {𝒀 𝒊 } ⌈𝑛/𝐿⌉𝑖=1 . The choice to structure AMG-Embedding as a two-stage method can be attributed to two factors:(1) By replacing 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 , any state-of-the-art fingerprinting approach can be seamlessly integrated into AMG-Embedding, thereby enhancing the capabilities of AMG-Embedding. This allows for the incorporation of advanced techniques to improve the overall performance of the embedding. (2) Combining the two stages into a unified process, specifically, training one encoder to replicate the current performance, introduces notable challenges. The requirement for an enlarged model input size brings about a tenfold escalation in model complexity and parameter count. Consequently, the training of the consolidated encoder becomes difficult to achieve performance parity with our two-stage solution.𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 plays a pivotal role in AMG-Embedding, and it actually compresses embeddings by projecting a variable-length sequence of 𝑘 features (1 ≤ 𝑘 ≤ 𝐿) derived from an audio segment into a final embedding 𝒀 , and the inner product of 𝒀 serves as a metric for measuring the similarity between corresponding segments.The model architecture, the chosen metric loss, and the training paradigm of 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 should be carefully designed. Firstly, independent audios are collected as training data, with specific segments (anchor zone in the figure), whose durations are 𝐷𝑢𝑟 𝑚𝑎𝑥 , chosen as the anchor samples for each audio. Each specific anchor is associated with a distinct class in class-level learning. Moreover, we subject these audios to fragment&quality augmentation to simulate signal distortions: a segment of the original audio, with a duration also not exceeding 𝐷𝑢𝑟 𝑚𝑎𝑥 , is extracted firstly. Importantly, these segments must exhibit content overlap with their corresponding anchor. Then the quality augmentation is applied to these segments to generate the augmented samples through the same steps as neural fingerprinting [4] : time offset modulation, background mixing, impulse response filtering, cutout, and spec-augment.After the Fingerprinting Stage, both anchor and augmented samples are converted into feature sequences, and then these sequences are transformed into final embeddings by using 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 , and the embeddings of an anchor sample and its augmented counterparts share the same unique class label. The assignment of identical class labels to both the anchor samples and their augmented replicas suggests that the embedding of an audio segment exposed to signal distortions should exhibit greater similarity to the embedding of its exact musical match than to embeddings of other music tracks. This constitutes a critical aspect of audio identification, ensuring accurate matching of distorted or fragment-level audios to their original sources.In theory, any class-level metric loss functions, such as ArcFace [8] , can be utilized to train 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 by encouraging each final embedding to approach its class proxy while simultaneously moving away from proxies of other classes. However, we found that common class-level metric losses are not suitable for this task (discussed in Section 3.3.1). Therefore, we proposed Proxy-anchor Aligned Margin Loss (PAM-Loss) to significantly solve the problem.PAM-Loss is a proxy-based loss function, derived from ArcFace-Loss [8] , primarily designed to improve variable-length embeddings. Let's first revisit ArcFace-Loss and analyze why it is not an appropriate choice for training 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 . , where 𝑚 > 0 represents an additive angular margin, and 𝑠 is the scaling parameter.ArcFace-Loss hasn't found effective application in MIR, especially in audio identification. Triplet and N-pairs losses play more important roles in these tasks. It's evident that loss functions without proxies offer a more straightforward approach for retrieval tasks. Furthermore, ArcFace exhibits some limitations when applied to variable-length audio embeddings as follows:(1) Firstly, ArcFace's primary objective is to drive the embeddings of various augmented fragment samples from the same music track to converge towards their proxies. However, in practical audio identification, the central goal is to accurately match a distorted audio snippet to a complete high-quality audio in databases. The two objectives are not entirely equivalent, as ArcFace lacks a mechanism to align the proxy with the embeddings of such high-quality audio. (2) Secondly, ArcFace incorporates a fixed angular margin into its loss function. However, it becomes unreasonable to persist with this fixed angular margin when dealing with our task, where the degree of content overlap between the augmented sample and the anchor sample can vary. (3) Finally, but most importantly, a significant challenge arises during the training process-ensuring the convergence of embeddings from different augmented samples towards their respective proxies. The diverse content of augmented samples within the same class may lead to weak relationships between them, because each augmented sample may only contain a fraction of the information present in the corresponding anchor, resulting in a lack of complete class knowledge. Consequently, it is unreasonable to require that these embeddings closely resemble the proxy when the proxy itself lacks complete class knowledge. This requirement for full class knowledge in the proxy is crucial for effective training, distinguishing it significantly from other scenario, such as face recognition, where each training sample inherently encapsulates the entire class information. Experiments in Section 4.4.2 demonstrate this limitation apparently. Notably, most MIR tasks heavily rely on pair-wise instead of class-lever learning. However, this limitation does not stem from the inherent capabilities of class-level learning itself but rather from the fact that commonly used class-level losses are not well-suited for MIR tasks.ArcFace-Loss by incorporating proxy-anchor consistency, overlapaware margin, and proxy initialization to effectively address the three limitations outlined above.Assuming there are 𝑁 samples in a training batch, and 𝑀 samples of them are original anchor samples. These 𝑁 samples are denoted as {𝒀 𝑖 , 𝑙 𝑖 , 𝛾 } 𝑁 𝑖=1 , where 𝒀 𝑖 and 𝑙 𝑖 share the same definition as in Section 3.3.1, and 𝛾 indicates that the training sample is an anchor sample (𝛾 = 1) or an augmented sample (𝛾 = 0). The objective of PAM-Loss is expressed asEQUATION, whereEQUATIONEQUATION. 𝐿 𝑎𝑙𝑖𝑔𝑛 𝑖 is the proxy-anchor alignment loss and 𝐿 𝑜𝑎𝑚 𝑖 is the overlapaware margin loss of the 𝑖-th sample. The hyper-parameters 𝜆 𝑎𝑙𝑖𝑔𝑛 is used to trade-off between two losses. 𝛿 𝑖 represents the overlapping duration ratio to 𝐷𝑢𝑟 𝑚𝑎𝑥 . Following measures are integrated in PAM-Loss to overcome the limitations of ArcFace-Loss:Proxy-anchor Alignment (PA). 𝐿 𝑎𝑙𝑖𝑔𝑛 𝑖 represents a straightforward angular cosine embedding loss function. The primary objective of this loss is to maintain consistency between the embedding of the anchor sample and its associated proxy. Given that the training dataset predominantly consists of high-quality samples, this loss function establishes a mechanism for aligning the proxy with the embedding of the complete high-quality audio. This alignment ensures that the training objective of the PAM-Loss aligns perfectly with the application goal of audio identification.Overlap-aware Margin (OM). 𝐿 𝑑𝑎𝑚 𝑖 is derived from ArcFace-Loss with a dynamic overlap-aware margin 𝑚(𝛿 𝑖 ). We define function 𝑚(𝛿 𝑖 ) as a linear function asEQUATION, where 𝑚 𝑢 and 𝑚 𝑙 are the upper and lower of the angular margin range. This design follows a natural intuition: the embeddings of long samples should be closer to class proxy 𝑃 𝑙 𝑖 via 𝑚(𝛿 𝑖 ). 𝐿 𝑑𝑎𝑚 simultaneously further strengthens the consistency between class proxies and their corresponding original samples (their duration ratio 𝛿 𝑖 = 1.0). Proxy Initialization (PI). As mentioned above, applying a proxy-based loss in audio identification requires that the proxy contains complete class knowledge in the training process. Otherwise, achieving convergence becomes challenging. Assuming there is an original music 𝑿 in the training dataset, its class label is 𝑙. We can initially convert 𝑿 into a feature sequence {𝒛 𝑡 } 𝑡 𝑛 𝑡 =𝑡 1 in Fingerprinting Stage. We have empirically found that the mean of these features, 𝒛 𝒎𝒆𝒂𝒏 = 1 𝑛 𝑡 𝑛 𝑡 =𝑡 0 𝒛 𝑡 , is an excellent initial value for the corresponding class proxy 𝑷 , because 𝒛 𝒎𝒆𝒂𝒏 effectively encapsulates knowledge of the class, as demonstrated in the experimental section. This initialization significantly improve the performance of 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 , and also serves as a compelling demonstration of the necessity for a two-stage embedding process, as 𝒛 𝒎𝒆𝒂𝒏 is a result of Fingerprinting Stage.A proxy should possess full knowledge of its corresponding anchor sample. Proxy Initialization greatly alleviates this challenge. Furthermore, we dynamically adjust the distribution of 𝛿 to further enhance the consistency between proxies and their anchor samples. 𝛿 serves as a measure of the proportion of full knowledge contained in the augmented samples. As we initiate the training process of 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 , our objective is to construct proxies that contain full knowledge, so we predominantly utilize samples with larger 𝛿. As the training processes, we gradually increase more samples with smaller 𝛿 that are considered challenging samples with lower knowledge content as hard-sample mining. Hence, we employ a function 𝛿 𝑖 = 𝛼 𝛽 , where 𝛼 ∼ Uniform[0, 1] and 𝛽 ∈ [0.5, 3.0] is used to adjust the distribution, to randomly generate the overlap duration ratio 𝛿 𝑖 for the augmented samples. As the SOTA open source fingerprinting solution, the encoder of neural fingerprinting [4] is adopted as 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 . Initially, we trained 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 using the source codes, settings and Training-FP-DB provided by the authors of neural fingerprinting. As standard Transformer Encoders, 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 models share a hidden feature dimension of 256, 8 attention heads, and consist of 10 encoder layers. These models were trained using the Adam optimizer on the dataset named Traning-EB-DB. The initial learning rate of 1e-3 was reduced to 1e-4 using cosine decay over 50 epochs. A training batch has 640 samples, in which augmented samples and original anchor samples are mixed in a 9:1 ratio to balance 𝐿 𝑜𝑎𝑚 and 𝐿 𝑎𝑙𝑖𝑔𝑛 . We set 𝑚 𝑢 = 0.4, 𝑚 𝑙 = 0.2, scaling 𝑠 = 32, and 𝜆 𝑎𝑙𝑖𝑔𝑛 = 10.0 in all experiments. All encoders were trained on a server with 8-TitanX GPUs, and the retrieval results were obtained using a standard PC with an Intel Core i7-10700K CPU, 128GB RAM, NVIDIA GeForce RTX 3060 GPU with 12GB VRAM, and SSD hard disks. Querying method. The retrieval database consists of audio songs from Test-Dummy-DB and Test-Query-DB, each assigned a unique ID and converted into embeddings. To retrieve, a random clip from Test-Query-DB is converted into embeddings. For each query embedding, top K similar embeddings are retrieved from the database. Cumulative similarity scores are calculated for embeddings sharing the same song ID, and the song IDs are ranked based on these scores.Evaluation metric. We use retrieval accuracy, retrieval duration, and indexing feature size together to evaluate the performance. Top-1 hit rate is employed to measure the accuracy. The metric of Top-1 hit rate(%) is defined as [4] . 𝑆𝑖𝑧𝑒 𝑏𝑑 represents the embedding count for building the retrieval database. 𝐷𝑈 𝑅1 𝑏𝑑 and 𝐷𝑈 𝑅2 𝑏𝑑 are the duration of building retrieval database using Flat and IVF-PQ index in Faiss [22] , and the duration associated with AMG-Embedding include inference time of 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 . 𝐷𝑈 𝑅1 𝑞𝑟 and 𝐷𝑈 𝑅2 𝑞𝑟 represent the average duration of querying a 30𝑠 audio using Flat and IVF-PQ index, and the duration include the inference time of 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 for all methods and additionally 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 for AMG-Embedding. EQUATION 
comparisons: 
 We firstly compare our work with classic systems, including two handcrafted feature systems (our imeplemented Shazam [41] and a popular Shazam-style open-source Dejavu [12] ), and two deeplearning systems (Now-playing(NP) [16] , and Neural fingerprinting system(NFS) [4] ). The fingerprints of NP and NFS were built with fixedduration segments of 1𝑠 and 0.5𝑠 overlap. Moreover, the encoder of NFS is also employed as 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑓 𝑝 in AMG-Embedding. We created 21 query segments for each audio file in the Test-Query-DB, which includes the original 30𝑠 audio and 5 randomly cropped segments for each duration: 2𝑠, 3𝑠, 5𝑠, and 10𝑠. Therefore, this results in a total of 10,500 test query segments. Each query is synthesized through the quality augmentation pipeline used in neural fingerprinting [4] .All dimensions of the embedding are set to 128 for Now-playing, NFS and AMG-Embedding. In AMG-Embedding, we trained the 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 with 𝐷𝑢𝑟 𝑚𝑎𝑥 = 5𝑠. In NP, NFS, and AMG-Embedding, GPUs were employed during encoder inference, whereas the retrieval was performed using the CPU without parallel processing (querying one embedding at a time). The Top1 accuracy results for AMG-Embedding are obtained using the Flat index in Faiss [22] , while for NFS, due to its computationally intensive nature (see Section 4.3), the results are derived from the IVF-PQ index with a product quantizer (PQ) with the configuration: 256 centroids, each with a code size of 64 and 8 bits per index.The results are listed in Table 1 . It is evident that hand-crafted feature methods exhibit significantly lower performance in retrieval accuracy, retrieval duration, and feature size. The results also indicate that NFS achieves the highest retrieval accuracy. However, it does not exhibit advantages in retrieval duration and indexing feature size. Given that our proposed AMG-Embedding builds upon the foundation of NFS, the primary objective of experiments is to demonstrate that the AMG-Embedding can effectively preserve the accuracy of the underlying NFS while achieving a substantial reduction in the embedding size and the retrieval times. Therefore, we conducted the following experiments to further demonstrate the effectiveness of AMG-Embedding.To comprehensively demonstrate the effectiveness of our method, we trained several 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 with different 𝐷𝑢𝑟 𝑚𝑎𝑥 = {5𝑠, 10𝑠, 15𝑠, 30𝑠}, and different dimension 𝑑 = 𝑓 = {128, 256} for comparisons. As shown in Table 2 , NFS achieves the higher accuracy, and the accuracy of AMG-Embedding is influenced by the 𝐷𝑢𝑟 𝑚𝑎𝑥 apparently, with shorter durations yielding higher accuracy. Additionally, as the retrieval segment becomes shorter, the accuracy gap with NFS becomes more noticeable. However, once the queried segment exceeds 5 seconds, the accuracy gap between AMG-Embedding and NFS becomes minimal. Furthermore, a larger dimension of embeddings is preferred for higher retrieval accuracies. The final embedding quantity generated by AMG-Embedding is significantly reduced. Beyond the storage advantage, this reduction brings efficiency benefits to retrieval process.As for querying duration and indexing feature size, we initially employed the Flat index in Faiss for comparisons. Table2 illustrates that the time required to build retrieval databases (referred to as 𝐷𝑈 𝑅1 𝑏𝑑 ) is roughly comparable between AMG-Embedding and NFS. Of particular note is the negligible duration difference between AMG-Embedding and NFS in building retrieval databases, especially when compared to the nearly 5-hour consumption on fingerprinting computation shared by NFS and AMG-Embedding. However, NFS necessitates more query times due to the larger number of features it generates. Additionally, the Faiss feature index created by NFS is notably large, leading to a significant increase in single-query duration. Consequently, the time required for NFS to complete retrievals for a long audio sample is significantly greater than that of AMG-Embedding. When querying a 30𝑠 audio, NFS generates 59 embeddings, while AMG-Embedding (𝐷𝑢𝑟 𝑚𝑎𝑥 = 10𝑠) generates only 3 embeddings. Thus, their querying durations were 112𝑠 and 0.32𝑠, respectively.Secondly, we adopted the IVF index to reduce the querying duration. While this technique enhances efficiency (𝐷𝑈 𝑅2 𝑞𝑟 ) significantly, it is essential to acknowledge the costs: a potential slight decrease in retrieval accuracy and an extra time-consuming task of computing centroids (see 𝐷𝑈 𝑅2 𝑏𝑑 ). Considering the 30𝑠 audio, the querying durations of NFS and AMG-Embedding were 8.71𝑠 and In a summary, AMG-Embedding introduces additional computations and incurs a slight decrease in retrieval accuracy, but the reduction in embedding size results in a dramatic advantage in retrieval efficiency and storage consumption.In this section, we present a series of ablation experiments designed to highlight the effectiveness of the employed techniques. For the sake of universality, we set 𝐷𝑢𝑟 𝑚𝑎𝑥 = 10𝑠 and 𝑓 = 𝑑 = 128, ensuring a standardized comparison. We also generated 2500 query segments for each query length in these experiments.Encoder. AMG-Embedding employs a standard transformer encoder as 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 . We also trained CNNbased ResNet-18 and ResNet-50 [20] , DenseNet121 [21] , EfficientNet-B3 [39] and a standard sequence-to-sequence Bi-directional 8-layer (256 dimension) LSTM [32] encoders for comparisons. Based on the results in Table 3 , it is evident that, even with fewer parameters, Transformer-based encoders exhibit notable performance advantages compared to both CNN and LSTM encoders.Furthermore, our 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 can be regarded as a feature compression or dimensionality reduction method. Therefore, we also attempted to use machine learning dimensionality reduction techniques, including PCA, T-SNE, etc., to compress features. Because the input is variable-length features, padding is needed before applying the above methods. Nevertheless, our experimental results indicate that these methods cannot effectively represent the features, with accuracy scores consistently at 0 for variable duration retrieval.4.4.2 Effect of PAM-Loss. Our proposed metric loss function, PAM-Loss, plays a crucial role in AMG-Embedding. To assess the impact of PAM-Loss, we trained 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑒𝑏 with various loss functions, including Triplet-Loss, InfoNCE (N-Pairs/NT-Xent)-Loss, ArcFace-Loss, and PAM-Loss. The accuracy results are listed in Table 4 . Among these losses, Triplet-Loss and InfoNCE-Loss are widely employed in MIR-related tasks for pair-wise learning. According to the results, we can see that these two losses can be utilized in our task, and InfoNCE-Loss outperforms Triplet-Loss significantly. As discussed above, though ArcFace-Loss has achieved considerable success in numerous tasks, it is not suitable in variable length audio embedding. It is proved by results in Table 4 . In fact, the encoder trained with ArcFace-Loss failed to converge to a level suitable for practical applications. PAM-Loss addresses the limitations of ArcFace-Loss in handling the variable-length embedding problem by incorporating three measures: Proxy-anchor Alignment (PA), Overlap-aware Margin (OM), and Proxy Initialization (PI). We gradually integrated them into ArcFace-Loss to demonstrate their effectiveness. According to the results in Table 4 , it is obviously that each measure can boost the performance apparently, and PI plays the most important role in PAM-Loss. The results also demonstrate the effectiveness of PA and OM.The primary goal of AMG-Embedding is to address challenges related to resource consumption and retrieval efficiency in large feature database generated by fingerprinting methods. However, AMG-Embedding, by design, does not excel in providing precise matching positions as fingerprinting. Retrieval accuracy of AMG-Embedding is also not expected to surpass the based fingerprinting method, especially for very short retrieval segments. Additionally, the Embedding Stage introduces additional computational overhead, but this overhead can be effectively mitigated by retrieval accelerations. As a result, AMG-Embedding is more suitable for scenarios involving large retrieval database where storage limitations and retrieval efficiency are critical considerations. 
conclusion: 
 In this paper, we introduce AMG-Embedding, a novel approach to audio identification. Our main aim is to represent audio with much shorter embedding sequences compared to fingerprinting, while still supporting high-specific fragment-level retrieval. AMG-Embedding achieves this through a two-stage variable-length embedding using a Transformer encoder trained with PAM-Loss. PAM-Loss is also a valuable exploration for applying class-level learning in audio identification, showing potential for broader use in other MIR tasks. AMG-Embedding achieves accuracy levels comparable to fingerprinting solutions, yet it provides a significant advantage in retrieval efficiency and storage consumption. 
acm reference format:: 
 Yuhang Su, Wei Hu, Fan Zhang, and Qiming Xu. 2024. AMG-Embedding: a Self-Supervised Embedding Approach for Audio Identification. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24), October 28-November 1, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3664647.3681647 
 
 Appendix: Code is available at https://github.com/yuhangsu82/AMG-Embedding. 


Decision: **Conclusion: Accept as a Poster Paper**

After thoroughly evaluating the three independent reviews, I recommend accepting this submission as a Poster Paper for ACM MM 2024. The submission is a significant contribution to the field of audio retrieval systems, addressing critical issues of efficiency and scalability. Although some limitations are noted, the strengths and innovative aspects of the approach justify its acceptance.

**Strengths:**

1. **Innovative Two-Stage Embedding Process**: The proposed method integrates traditional fingerprinting with a Transformer-based encoder, significantly reducing the number of embeddings needed for audio identification. This innovation effectively addresses the inefficiency of previous methods.
2. **Efficiency Improvements**: The method substantially decreases computational and storage requirements, making it suitable for large-scale audio databases. Experimental results show that AMG-Embedding achieves comparable retrieval accuracy while reducing storage and retrieval time to less than one-tenth of the traditional methods.
3. **Comprehensive and Clear Presentation**: The paper is well-written, with a clear and logical structure. The figures and tables are informative and easy to understand, providing a thorough explanation of the PAM-loss and its components.

**Weaknesses:**

1. **Limited Comparison with Latest Approaches**: The paper primarily compares AMG-Embedding with an older method (NFS) and lacks comparisons with the latest fingerprinting techniques, such as MERT, which represents the state-of-the-art in music audio embedding.
2. **Accuracy Trade-offs**: While AMG-Embedding significantly reduces database size and query duration, it slightly underperforms in terms of accuracy compared to the NFS method. This trade-off needs further investigation and justification.
3. **Database Construction Duration**: The duration for building a retrieval database is longer for AMG-Embedding than for NFS, likely due to the two-stage embedding process. This aspect requires further optimization to enhance the method's practical applicability.

In summary, the reviewers collectively acknowledge the innovative contributions and practical significance of AMG-Embedding in improving audio retrieval systems. The strengths in methodology, efficiency, and comprehensive presentation outweigh the noted limitations. Given the overall positive assessment and the potential impact on the multimedia community, this paper is recommended for acceptance as a Poster Paper at ACM MM 2024.

Reviews: review: paper_summary: paper_summary: The paper presents AMG-Embedding, a self-supervised embedding approach designed for audio identification. The method aims to improve the efficiency and scalability of audio retrieval systems by reducing the number of embeddings needed to represent audio tracks. Traditional fingerprinting methods require numerous short, fixed-duration, and overlapping segments, leading to large feature databases and high computational costs. AMG-Embedding addresses these issues by representing variable-duration non-overlapping segments through a two-stage embedding process. The approach integrates a fingerprinting encoder and a Transformer-based encoder trained with a novel loss function, Proxy-anchor Aligned Margin Loss (PAM-Loss). Experimental results demonstrate that AMG-Embedding achieves comparable retrieval accuracy to traditional methods while significantly reducing storage and retrieval time.
strength_weakness: strength_weakness: - The two-stage embedding process combining traditional fingerprinting with a Transformer-based encoder is well-conceived.
- The method significantly reduces the number of embeddings for audio identification.
- The reduced computational and storage requirements make AMG-Embedding suitable for large-scale audio databases.
- The experiments demonstrate that the proposed method is competitive concerning retrieval accuracy while offering efficiency improvements.
- The approach can integrate any state-of-the-art fingerprinting method.
limitations: limitations: - The wording uses 'audio' a lot while focusing on 'music'.
- Given that the first stage already exists in the literature, the overall contribution is rather shallow.
- The comparison is very limited. One could use traditional fingerprinting methods with less overlapped (or non-overlapped) sections to equal with AMG-Embedding in terms of effiency.
- MERT is not considered, which currently represents the state of the art for music audio embedding.
- In general, the evaluation does not showcase the impact of the proposed method in scientific or practical terms.
paper_summary: paper_summary: This paper present AMG-Embedding, which is built on a traditional fingerprinting encoder and aims to represent variable-duration nonoverlapped segments as embeddings through a two-stage embedding. AMG-Embedding reduces the number of generated embeddings while achieving high-specific fragment-level audio identification.
Experimental results demonstrate that AMG-Embedding achieves retrieval accuracy comparable to the based fingerprinting approach while consuming less than 1/10𝑡ℎ of its storage and retrieval time.
strength_weakness: strength_weakness: 1.	In light of the inefficiency of previous audio identification approaches, AMG-Embedding effectively reduces the size of feature sequences. The current fingerprinting approach can be seamlessly integrated into AMG-Embedding.
2.	Based on ArcFace loss, this paper introduces the design of PAM-loss. It thoroughly explains why ArcFace is ineffective for MIR tasks and how PAM-loss was designed. The ablation study validates the effectiveness of the PAM-loss, specifically highlighting the contributions of the PI, OM, and PA modules in PAM.
3.	The experiments concerning AMG-embedding are very comprehensive, particularly in terms of size and duration, which allow readers to clearly understand the efficiency of AMG-Embedding.
limitations: limitations: 1.	Table 1 compares the baseline methods, which do not include the latest approaches. The focus is on comparing with NFS, and although the size is reduced to 1/10, the performance does not surpass NFS. NFS is a neural fingerprinting work published in 2021, and it would be valuable to include comparisons with the latest fingerprinting methods.
2.	In Table 4, I believe that PAM1, specifically PI, achieved the most significant performance improvement within the PAM loss. My question is why a margin of 0.3 was used when only PI was applied, given that the default margin in the original ArcFace paper is 0.5. This margin controls the clustering degree of class angles, and I suppose that an appropriate margin could potentially allow PI to outperform PI+OM. On the other hand, it would be insightful to explore whether PAM loss can form an effective feature space. Using T-SNE visualization might better illustrate the progression from the original ArcFace to PI, PI+OM, and finally to PI+OM+PA.
paper_summary: paper_summary: This paper proposed a new self-supervised embedding method for audio identification named AMG-embedding. The AMG-embedding can represent the audio in a variable-duration manner, and improve the granularity and specificity.
strength_weakness: strength_weakness: 1. This paper is very well written and easy to follow. The focuses are highlighted. The structure is clear and reasonable. The figures and tables are informative and easy to understand.

2. This paper proposed two-stage training where the second embedding stage could compress the fixed-duration audio features to smaller embeddings. 

3. This paper proposed PAM-loss to replace ArcFace loss to improve variable-length embeddings.

4. The experiment results show that the proposed AMG-embedding significantly reduces the size of the retrieval database and the query duration.
limitations: limitations: 1. The embedding stage is based on the fingerprinting stage (i.e., the NFS in Table 1). So why is the AMG-embedding slightly less accurate than NFS in query?

2. The duration for building a retrieval database of AMG-embedding is quite longer than NFS (in Table 2). Is it because the AMG-embedding the combination of NFS and embedding stage?