SONICS: Synthetic Or Not - Identifying Counterfeit Songs

## Abstract

Abstract The recent surge in AI-generated songs presents exciting possibilities and challenges. While these inventions democratize music creation, they also necessitate the ability to distinguish between human-composed and synthetic songs to safeguard artistic integrity and protect human musical artistry. Existing research and datasets in fake song detection only focus on singing voice deepfake detection (SVDD), where the vocals are AI-generated but the instrumental music is sourced from real songs. However, these approaches are inadequate for detecting contemporary end-to-end artificial songs where all components (vocals, music, lyrics, and style) could be AI-generated. Additionally, existing datasets lack music-lyrics diversity, long-duration songs, and open-access fake songs. To address these gaps, we introduce SONICS 1 1 1 Code, Data and Models will be available at https://github.com/awsaf49/sonics , a novel dataset for end-to-end Synthetic Song Detection (SSD), comprising over 97k songs (4,751 hours) with over 49k synthetic songs from popular platforms like Suno and Udio. Furthermore, we highlight the importance of modeling long-range temporal dependencies in songs for effective authenticity detection, an aspect entirely overlooked in existing methods. To utilize long-range patterns, we introduce SpecTTTra, a novel architecture that significantly improves time and memory efficiency over conventional CNN and Transformer-based models. In particular, for long audio samples, our top-performing variant outperforms ViT by 8% F1 score while being 38% faster and using 26% less memory. Additionally, in comparison with ConvNeXt, our model achieves 1% gain in F1 score with 20% boost in speed and 67% reduction in memory usage. Other variants of our model family provide even better speed and memory efficiency with competitive performance.

## 1 Introduction

The rapid advancements in AI-generated music present a substantial threat to the music industry, potentially reducing the demand for professional musicians and stifling new talent development [, ]. To preserve the unique value of human creativity, it is crucial to develop robust methods for detecting AI-generated music, ensuring a fair and vibrant creative ecosystem.

Singing Voice Synthesis (SVS) [] and Singing Voice Conversion (SVC) [] have recently achieved significant progress, enabling the creation of synthetic singing voices that closely mimic real singers’ styles. When combined with instrumental music from real songs, these synthetic voices can produce convincing counterfeit songs. Although related to synthetic speech detection, detecting fake songs is particularly challenging due to the unique rhythmic patterns and artistic vocal traits of singing []. To address this, researchers have turned their attention to Singing Voice Deepfake Detection (SVDD) [, , ]. However, current methods relying on datasets composed of SVS and SVC-generated songs face several limitations. These datasets are bound to use only instrumental music from real songs, leading to artifacts like the ”Karaoke effect” (volume discrepancies between music and vocals) and limited music-lyrics diversity. Moreover, existing methods overlook the long-context temporal relationships inherent in songs, such as repeated verses, music, rhythm, and emotional dynamics, which are critical for effective detection. Availability of only short duration songs in current datasets further hampers the use of these patterns. Additionally, copyright restrictions on some existing datasets limit the public availability of generated fake songs, hindering broader usage. Furthermore, the SVDD task requires separate tools for voice identification and separation during data processing [, ], increasing computational overhead.

Recently, platforms like Suno [] and Udio [] have gained significant traction on social media. They can synthesize not only vocals but also entire songs, including synthetic music, styles, and lyrics, further complicating the situation. Due to their end-to-end nature, these fake songs differ significantly from those generated by SVS and SVC methods, rendering existing SVDD methods and datasets inadequate for detecting them. This necessitates an urgent need for a detection system specifically designed for end-to-end synthetic song detection (SSD).

To address these shortcomings, we introduce SONICS, a large-scale dataset comprising 97,164 songs (4,751 hours), including 49,074 end-to-end synthetic songs (1,971 hours) generated by Suno and Udio, alongside 48,090 real songs (2,780 hours) curated from YouTube. With an average duration of 176 second (sec.), the SONICS dataset supports the use of long-context relationships in songs for accurate fake song detection. Furthermore, SONICS addresses the issue of music-lyrics diversity by including a wide range of music styles and both real and synthetic lyrics. Additionally, the fake songs in our dataset are free from copyright issues, as they are generated through paid subscriptions that provide a license to use and share the content legally. A unique feature of SONICS is that it includes the text lyrics of songs, which can aid future research.

Despite the availability of long songs in our dataset, utilizing long-context relationships presents additional challenges. For instance, CNN-based models struggle to capture long-range dependencies due to their inherently local receptive fields. While Transformer-based models can capture these dependencies with global attention, they are computationally expensive with longer audio inputs. To mitigate this trade-off, we introduce Spectro-Temporal Tokens Transformer (SpecTTTra), which uses a Spectro-Temporal Tokenizer to significantly reduce computational costs while employing global attention.
Our contributions are summarized as follows:

- •
We introduce SONICS, a large-scale dataset for synthetic song detection that addresses the limitations of existing datasets, including the absence of end-to-end fake songs, limited music-lyrics diversity, short-duration songs, and open fake songs.
- •
We highlight the importance of modeling long-context temporal relationships in songs, an aspect entirely overlooked in existing approaches.
- •
We propose a faster and memory efficient model, SpecTTTra, which effectively captures long-context temporal relationships in songs while outperforming popular methods.
- •
We offer a human benchmark for fake song detection, a feature missing in previous works.
- •
We establish a traditional AI benchmark for popular CNN and Transformer-based models.

**Table 1: Comparison of Proposed and Existing Fake Song Datasets**
| \vcellDataset | \vcell<br>End-To-End<br>Fake Songs | \vcell<br>Text Lyrics | \vcell<br>Song Style | \vcell<br>Music-Lyrics<br>Diversity | \vcell<br>Open Fake Songs | \vcell<br>Open Real Songs | \vcell<br>Open-Source<br>Models | \vcell<br>Language | \vcell<br>Average Length<br>(sec) | \vcell<br># Algorithms | \vcell<br># Speakers | \vcell<br># Real Songs | \vcell<br># Fake Songs | \vcell<br># Total Songs |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom |
| FSD [] |  |  |  |  | ✓ | ✓ | ✓ | Chinese | 216.00 | 5 | 60 | 200 | 450 | 650 |
| \vcellSingFake [] | \vcell | \vcell | \vcell | \vcell | \vcell | \vcell | \vcell | \vcellMulti | \vcell13.75 | \vcell- | \vcell40 | \vcell634 | \vcell671 | \vcell1,305 |
| \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellbottom | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle |
| CtrSVDD [] |  |  |  |  | ✓ | ✓ | ✓ | Multi (no English) | 4.87 | 14 | 164 | 32,312 | 188,486 | 220,798 |
| \vcellSONICS (ours) | \vcell✓ | \vcell✓ | \vcell✓ | \vcell✓ | \vcell✓ | \vcell | \vcell | \vcellEnglish | \vcell176.03 | \vcell5 | \vcell9,096+ | \vcell48,090 | \vcell49,074 | \vcell97,164 |
| \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellbottom | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle |

## 2 Related Works

Synthetic Speech Detection: The domain of synthetic speech detection, closely tied to synthetic song detection through their shared audio modality, has been extensively explored due to advancements in voice conversion [] and synthesis techniques []. These advancements have spurred the development of audio spoofing attacks on speaker verification systems and deepfake audio targeting human listeners []. Synthetic speech detection methods include Light CNN (LCNN) with Max-Feature-Map activations [], Transformer encoders with ResNet architectures [], RawNet2 with sinc layers and GRU blocks [], and heterogeneous graph attention networks []. However, the unique complexities of songs—such as rhythm, melody, and emotional nuance—present challenges that traditional speech detection methods are not equipped to handle, as shown by [] and []. Thus, following CtrSVDD [], we opted not to conduct similar experiments in our study.

Synthetic Song Detection: Synthetic song detection, a relatively newer and more complex challenge, has gained attention recently. In early 2024, SingFake [] introduced a dataset of counterfeit songs using Singing Voice Conversion (SVC), along with the task of Singing Voice Deepfake Detection (SVDD) and associated model benchmarks. Subsequent work [, ] combined Singing Voice Synthesis (SVS) with SVC to create phoneme-based songs, leading to specialized detection datasets. Methods in this area include convolutional networks for feature extraction followed by classification using graph neural network  [], wav2vec2-based extraction coupled with graph neural networks [], Linear-Frequency Cepstral Coefficients (LFCC) used with ResNet18 models [] and combination of music-specific models (MERT) & linguistic models (wav2vec2.0) with targeted augmentations [].
However, SVC and SVS-based datasets retain original background music, leading to a detectable ”Karaoke effect” artifact. Recent end-to-end fake songs by Suno [] and Udio [], can produce divergent fake songs where all musical components (e.g., background music, styles, and lyrics) can be synthetic, presenting a severe detection challenge. As shown in Table 2, models trained on SingFake dataset, perform poorly when tested on the end-to-end fake songs (SONICS dataset), with a significant drop in detection performance (F1 score) ranging from 20% to 64%.

Long Audio Classification: Songs exhibit long-range temporal patterns, such as repeated verses, rhythms, etc. setting them apart from speech []. Despite their potential to enhance detection performance, these patterns have been largely overlooked in existing methods [, , ]. Meanwhile, long audio classification remains an uncharted area in audio research. Although automatic speech recognition handles long audio data [], it struggles with end-to-end processing of extended audio due to its high computational cost, thus often uses sliding window techniques [, ] to manage costs. This further complicates the task of leveraging long-context features for synthetic song detection.

## 3 Methodology

### 3.1 SONICS Dataset

The development of a modern synthetic song detection system necessitates a dataset that meets several stringent criteria, which are conspicuously absent in existing music datasets. These criteria include: 1) songs where all components—not just vocals—can be AI-generated; 2) song lengths sufficient to capture long-term contextual relationships; 3) a diverse spectrum of music-lyrics combinations; and 4) a quantity of data substantial enough to serve as a generative model benchmark. Addressing these needs, we introduce the SONICS dataset, a comprehensive collection of end-to-end AI-generated songs produced using the latest audio generative models, spanning lengths from 32 to 240 sec and encompassing an extensive array of music-lyrics styles. A detailed comparison of SONICS with existing datasets is presented in Table 1. It clearly illustrates that datasets such as FSD [], SingFake [], and CtrSVDD [] fall short of fulfilling all the outlined criteria when juxtaposed with SONICS. Additionally, a comprehensive distribution summary of the SONICS dataset is provided in Table 3.

**Table 2: Performance of SingFake-trained models on SingFake vs SONICS dataset**
| Model | SingFake | SONICS |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EER $\downarrow$ | F1 $\uparrow$ | Sens. $\uparrow$ | Spec. $\uparrow$ | EER $\downarrow$ | F1 $\uparrow$ | Sens. $\uparrow$ | Spec. $\uparrow$ |  |
| ConvNeXt | 0.20 | 0.86 | 0.90 | 0.65 | 0.38 | 0.33 | 0.22 | 0.88 |
| ViT | 0.29 | 0.84 | 0.97 | 0.35 | 0.49 | 0.64 | 0.87 | 0.16 |
| EfficientViT | 0.19 | 0.88 | 0.93 | 0.64 | 0.50 | 0.35 | 0.28 | 0.71 |

**Table 3: Dataset Summary**
| Split | \vcellFull Fake | \vcellHalf Fake | \vcellMostly Fake | \vcellReal | Total |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \printcellmiddle | \printcellmiddle | \printcellmiddle | \printcellmiddle |  |  |  |  |  |  |  |  |  |  |  |  |
|  | \vcell<br>Suno v3.5 | \vcell<br>Udio 130 | \vcell<br>Suno v3 | \vcell<br>Suno v2 | \vcell<br>Udio 32 | \vcell<br>Suno v3.5 | \vcell<br>Suno v3 | \vcell<br>Suno v2 | \vcell<br>Suno v3.5 | \vcell<br>Udio 130 | \vcell<br>Suno v3 | \vcell<br>Suno v2 | \vcell<br>Udio 32 | \vcell<br>YouTube |  |
|  | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom | \printcellbottom |  |
| # Train | 774 | 656 | 0 | 0 | 0 | 4558 | 0 | 0 | 11863 | 16172 | 0 | 0 | 0 | 32686 | 66709 |
| # Test | 77 | 53 | 206 | 100 | 203 | 396 | 634 | 309 | 1117 | 1566 | 2797 | 1350 | 3970 | 13237 | 26015 |
| # Valid | 9 | 12 | 37 | 16 | 30 | 67 | 117 | 51 | 196 | 286 | 494 | 258 | 700 | 2167 | 4440 |

#### 3.1.1 Real Songs

The dataset’s Real songs segment comprises original compositions by human artists. This portion of the dataset was initially compiled by random sampling from the Genius Lyrics Dataset [], which provides metadata including lyrics, titles, and artist information. Subsequently, leveraging this metadata, a search was conducted dynamically on YouTube to retrieve the corresponding audio files. This process yielded 48,090 songs performed by 9,096 artists.

#### 3.1.2 Fake Songs

For generating end-to-end fake songs, we utilized the only two currently available audio generative model families: Suno [] and Udio []. Specifically, we used three variants from Suno—Suno v2, v3, and v3.5. Suno v2, the earliest iteration, generates songs up to 80 seconds (sec.) in length. Suno v3 extends this capability to 120 sec. The latest iteration, Suno v3.5, produces songs up to 240 sec. From the Udio family, we used Udio 32 and Udio 130, capable of generating songs of 32 and 130 sec, respectively. For generating and downloading songs from Suno, we utilized the community API []. In contrast, for Udio, we manually generated the songs from the website and then downloaded them using community API []. Subsequently, we used PyAnnote [] to filter out audio files lacking vocal components, which were predominantly found among the Udio-generated song.

Figure: Figure 1: End-to-end pipeline of Full Fake song generation process. Here a random combination of topic, genre, and mood is utilized to create fake lyrics and fake song styles by prompt engineering LLM GPT-4o. The generated lyrics and styles are utilized to create synthetic songs through audio-generative models from Suno and Udio.
Refer to caption: extracted/5900426/images/sonics-full-fake-pipeline.jpg

Figure: Figure 2: End-to-end pipeline of Mostly Fake song generation process. Firstly, lyrics features are extracted using an LLM from real song lyrics, which in turn is utilized to create fake lyrics through a second call to the LLM GPT-4o. Secondly, style information is extracted from real songs using multimodal Generative model Gemini 1.5 Flash. Finally, the generated lyrics and styles are utilized to create synthetic songs through audio-generative models from Suno and Udio.
Refer to caption: extracted/5900426/images/sonics-mostly-fake-pipeline.jpg

The Fake songs are generated with Suno and Udio models by two primary text inputs: 1) lyrics and 2) song-style (e.g., instruments, genre, vocal type) where for each combination of (lyrics, style) input, two songs are generated. Based on the nature of these inputs, the Fake songs are categorized into three distinct groups: i) Full Fake (FF)—featuring both AI-generated lyrics and song-style; ii) Mostly Fake (MF)—where lyrics are AI-generated based on real lyrics features and song-style is derived from real songs; and iii) Half Fake (HF)—where the lyrics are directly sourced from real songs, with song-style also extracted from real songs. To generate FF songs, we curated a rich set of metadata, including 57 broad topics (e.g., friendship, betrayal), 292 specific topics (e.g., star trek, pokemon), 49 music genres (e.g., rock, metal), and 72 moods (e.g., calm, angry). The full list can be found in the Appendix. Random combinations of these elements were used to generate final lyrics and styles via a Large Language Model (LLM). The complete pipeline for FF song generation is illustrated in Fig. 1. Here the prompt generator generates a prompt for later stages by filling in the variable values into the placeholders of a predefined prompt template (details in Appendix D). In contrast, the creation of MF and HF songs involved extracting song-styles from real songs using the multimodal LLM Gemini 1.5 Flash, which uniquely supports audio input. Additionally, for MF songs, lyrics were generated using the GPT-4o LLM, which extracts features from existing real song lyrics to produce new lyrics with a similar distribution, avoiding direct replication of existing content. The pipeline for generating MF songs is detailed in Fig. 2. The HF songs, differing only in the use of real lyrics directly sourced from existing songs, follow a similar process as MF songs. It is noteworthy that, Udio models cannot generate songs for lyrics from real songs; thus, HF songs contain only Suno models. In total, we generated 6,132 HF songs, 40,769 MF songs, and 2,173 FF songs using Suno and Udio models. The prompt templates used in the LLM to extract lyrics features and song styles, as well as to generate lyrics and styles, can be found in the Appendix.

### 3.2 Model

The architecture of an audio classification model depends on how the input audio is processed for feature extraction. In our benchmark, we use Mel-spectrogram due to its versatile usage and effective performance across various audio processing tasks [, , , , ]. Spectrograms, resembling 2D shape of an image, allow the usage of image classification models for audio classification. Since image classification models are generally categorized into CNN-based and Transformer-based architectures, we use popular models from both categories in our study, such as ConvNeXt [] and ViT [], for benchmarking. Additionally, we employ EfficientViT [], a hybrid model that integrates both Transformer and CNN components. However, these models encounter challenges when dealing with the inherent long-range temporal dependencies in songs. Specifically, ConvNeXt struggles to capture long-range dependencies due to its local receptive field, a characteristic inherited from CNNs. On the other hand, while ViT is capable of capturing long-range dependencies, it incurs significant computational costs as the number of patches/tokens rapidly increases with longer audio inputs. Similarly, EfficientViT, despite its linear global attention, becomes computationally expensive for long audio due to the large number of tokens combined with its multi-scale operations. To address this issue, we introduce SpecTTTra, which utilizes global attention similar to ViT but employs a Spectro-Temporal Tokenizer to reduce the number of tokens considerably, thereby lowering computational costs and enhancing efficiency significantly.

Figure: Figure 3: Proposed Spectro-Temporal Tokens Transformer (SpecTTTra) model. First, the input mel-spectrogram undergoes separate temporal and spectral slicing to generate corresponding clips, which are tokenized into temporal and spectral tokens using separate tokenizers. Next, separate positional embeddings are added to these tokens, which are then passed through a Transformer encoder. The resulting globally contextualized features are then average pooled and finally passed to the classifier.
Refer to caption: extracted/5900426/images/sonics-specttra-v2.jpg

#### 3.2.1 Spectro-Temporal Tokens Transformer (SpecTTTra)

As illustrated in Fig. 3, the proposed SpecTTTra model begins by applying temporal and spectral slicing to copies of the input spectrogram, generating distinct temporal and spectral clips/patches indicated by $|t_{i}|$ and $|f_{i}|$ respectively where $i$ denotes the position of each clip. These clips are then processed by the Spectro-Temporal Tokenizer, where temporal clips are embedded as temporal tokens $\langle t_{i}\rangle$ and spectral clips as spectral tokens $\langle f_{i}\rangle$, using separate tokenizers, where each token is represented as a vector of shape $\left(1,embed_dim\right)$. Subsequently, separate positional embeddings ($p_{i}^{t}$ for temporal and $p_{i}^{f}$ for spectral) are added to the respective tokens. Separate embeddings are used because there is no positional relationship between temporal and spectral tokens.

The positionally aware tokens are then fed into a ViT-like Transformer encoder, where they are contextualized with one another through global attention. This process enables the temporal tokens to become aware of both other temporal tokens and spectral tokens, and vice versa for spectral tokens. These globally aware features, now of shape $\left(n_tokens,embed_dim\right)$, are average-pooled across the $n_tokens$ dimension to aggregate the temporal and spectral information. Finally, the accumulated features are passed to a classifier, where they are classified as either real or fake songs.

It is important to note that while previous work has attempted to utilize both spectral and temporal information, these approaches come with significant limitations. For instance, [, ] focuses solely on temporal tokens, neglecting the rich spectral information. On the other hand, methods like [, ] employ separate attentions for spectral and temporal information but with ViT-like tokens, which, as discussed in the following section, become highly inefficient and computationally expensive as they rapidly increase with longer audio inputs. In contrast, our approach disentangles spectral and temporal information at the tokenization level and later contextualizes them with attention, leading to greater efficiency.

#### 3.2.2 Spectro-Temporal Tokenization

The ViT model creates grids of small square patches by simultaneously dividing a 2D spectrogram along both temporal and spectral dimensions, resulting in each patch having access to only limited spectral and temporal information. More critically, as indicated in Eq. 1, the number of patches/tokens ($N_{\nu}$) increases rapidly with the temporal dimension $T$. For example, for a spectrogram of size $F=128$ (spectral dimension) with a short audio duration of 5 sec ($T=128$) , ViT (patch size, $p=16$) generates tokens $N_{\nu}=64$. However, for longer audio with 120 sec ($T=3744$), the number of tokens surges to $N_{\nu}=1872$, nearly 30x more than for the shorter audio. Since the computational cost for global attention in ViT scales quadratically with the number of tokens, this makes it less practical for long audio classification. Similarly, EfficientViT, despite having linear global attention, multi-scale operations coupled with numerous tokens still result in high complexity.

| Equation for ViT Token Count |
| --- |
| If $T$ and $F$ denote the temporal and spectral dimensions of the spectrogram, respectively, and $p$ is the patch size, then the number of patches (or tokens) generated by a ViT ($N_{\nu}$) can be expressed as:<br>$\displaystyle N_{\nu}$<br>$\displaystyle=\left(\frac{F}{p}\right)\times\left(\frac{T}{p}\right)$<br>(1) |
| Equation for SpecTTTra Token Count |
| If $t$ and $f$ represent the sizes of the temporal and spectral clips, respectively, then the number of patches (or tokens) generated by SpecTTTra, denoted as $N_{\psi}$, is given by:<br>$\displaystyle N_{\psi}$<br>$\displaystyle=N_{spectral}+N_{temporal}$<br>(2)<br>$\displaystyle=\left(\frac{F}{f}\right)+\left(\frac{T}{t}\right)$<br>(2) |

Figure: Figure 4: Comparison of the number of tokens generated by the ViT model and the three SpecTTTra variants ($\alpha$, $\beta$, and $\gamma$) as a function of the number of time frames.
Refer to caption: extracted/5900426/images/time-vs-tokens-with-ratios.png

In contrast, SpecTTTra performs slicing independently for temporal and spectral dimensions, as shown in Fig. 3. This approach ensures that each temporal patch has access to the full spectral information, and each spectral patch contains the complete temporal information. This design leverages the observation that meaningful correlations can exist between distant temporal clips (e.g., between $|t_{0}|$ and $|t_{4}|$, capturing repeated song verses) or between distinct spectral clips (e.g., between $|f_{0}|$ and $|f_{2}|$, capturing harmonics). Importantly, as demonstrated in Eq. 2, due to the additive nature of token generation in SpecTTTra, the number of tokens grows more slowly compared to ViTs, significantly reducing computational costs. For instance, using the same parameters as before and setting the temporal and spectral clip sizes to $t=7$ and $f=5$, the number of tokens in SpecTTTra for short audio is $N_{\psi}=43$, and for long audio, it increases to $N_{\psi}=560$, which is approximately 3.4x fewer than in the ViT model. This substantial reduction in token count makes SpecTTTra significantly more computationally efficient. We further explore three variants of SpecTTTra, differentiated by the sizes of their temporal ($t$) and spectral ($f$) patches: SpecTTTra-$\alpha$ ($f=1$, $t=3$), SpecTTTra-$\beta$ ($f=3$, $t=5$), and SpecTTTra-$\gamma$ ($f=5$, $t=7$). Fig. 4 illustrates the rate at which the number of tokens increases for different SpecTTTra variants and ViT as the audio length grows.

The resulting spectral and temporal clips are processed by the Spectro-Temporal Tokenizer (STT) to create spectral and temporal tokens, respectively. The proposed STT block consists of separate Spectral and Temporal Tokenizers, which share identical architecture but differ in their objectives based on their input. These tokenizers use a Linear layer to map the spectral/temporal clips to spectral/temporal tokens, followed by GELU activation, addition of learnable positional embeddings similar to ViT, and finally, layer normalization. For efficiency, during implementation, the slicing and tokenization operations are merged using 1D CNN layer. Given an input spectrogram $x\in\mathbb{R}^{F\times T}$, the mathematical expression for tokenization is as follows:

$$ $\displaystyle x_{t}$ $\displaystyle=\text{Conv1D}\left(x,c_{i}=F,c_{o}=D,k=t,s=t\right)$ (3) $\displaystyle\langle t\rangle$ $\displaystyle=\text{LayerNorm}\left(\text{GELU}\left(x^{\top}_{t}\right)+\hat{ p}^{t}\right)$ $\displaystyle x_{f}$ $\displaystyle=\text{Conv1D}\left(x^{\top},c_{i}=T,c_{o}=D,k=f,s=f\right)$ $\displaystyle\langle f\rangle$ $\displaystyle=\text{LayerNorm}\left(\text{GELU}\left(x^{\top}_{f}\right)+\hat{ p}^{f}\right)$ $$

Here, $\langle t\rangle\in\mathbb{R}^{\frac{T}{t}\times D}$ and $\langle f\rangle\in\mathbb{R}^{\frac{F}{f}\times D}$ denote the temporal and spectral tokens, where $D$ is the token embedding dimension. The parameters $c_{i}$, $c_{o}$, $k$, and $s$ represent the input channels, output channels, kernel size, and stride size of the CNN layers. Additionally, $\hat{p}$ and $\top$ denote the learnable positional embeddings and Transpose operation. Further implementation details can be found in the Appendix.

## 4 Experiments and Results

### 4.1 Dataset

We conduct all experiments using the proposed SONICS dataset, which is divided into train, valid, and test sets. To ensure comprehensive evaluation, the valid and test sets include cases with unseen algorithms (e.g., Suno v2, Suno v3, Udio 32) and unseen singers. We also prevent data leakage by ensuring that song pairs from the same (lyrics, style) inputs are exclusively in either the training or valid-test sets, not in both. The distribution of the train, test, and valid sets is shown in Table 3.

### 4.2 Implementation Details

Generating the SONICS dataset costs $1,055, allocated across GPT-4o ($405), Gemini 1.5 Flash ($80), Suno ($390), and Udio ($180). To train models, we generate spectrograms with a sample_rate = 16kHz, n_fft = win_length = 2048, hop_length = 512, and n_mels = 128, yielding a $128\times 128$ spectrogram for 5 sec and $128\times 3744$ for 120 sec audio. Any song shorter than input length is zero-padded randomly, while for longer songs, a random crop is used. We also apply MixUp [] and SpecAugment [] augmentations during training to improve generalization. However, during the test, to maintain determinism, padding is done on the right side and cropped segments are taken from the middle. We conduct our training on an NVIDIA A6000 GPU with 48GB RAM, using WandB [] for tracking. We use ViT-small (patch size = 16) and ConvNeXt-tiny along with EfficientViT-B2 from the timm [] library. In SpecTTTra, we use the same model configuration as ViT-small. We train all models for 50 epochs from scratch using Binary Cross-Entropy loss with 0.02 label smoothing []. Optimization is performed with AdamW [] and a cosine learning rate scheduler from timm, including a 5-epoch warm-up. While existing methods [, , ] use Equal Error Rate (EER) as a metric, we prioritize the F1 score (binary average, threshold = 0.5) due to EER’s susceptibility to class imbalance. We also evaluate Sensitivity (Sens.) and Specificity (Spec.) to assess performance across fake and real classes.

### 4.3 Benchmarks

#### 4.3.1 AI Benchmark

The comparative analysis of the proposed SpecTTTra models against other existing models is presented in Table 4. The results reveal a significant performance gain (6% for ConvNeXt, 8% for EfficientViT, 10% for ViT, and 17% for SpecTTTra-$\alpha$) in the overall F1 score when using long songs. This finding substantiates our claim that leveraging long-context information is crucial for enhancing fake song detection. Additionally, the advantage of longer audio duration is more prevalent in transformer-based models such as ViT and SpecTTTra, as well as in the hybrid EfficientViT model, compared to the CNN-based ConvNeXt. Notably, the proposed SpecTTTra-$\alpha$, while trailing ConvNeXt by 10% in the F1 score for short audio, outperforms it in long audio. This can be attributed to the global attention mechanism in transformer models, which effectively captures long-range dependencies within the input data. In contrast, models with CNN components tend to perform better on shorter audio. Specifically, ConvNeXt and EfficientViT achieve overall F1 scores of 90% and 87%, respectively, outperforming all transformer-based models on short audio. However, despite the absence of global attention, ConvNeXt demonstrates competitive performance compared to SpecTTTra-$\alpha$ on long audio and outperforms ViT, EfficientViT, and other SpecTTTra variants in both short and long audio scenarios. We hypothesize that this is due to the inherent inductive biases present in CNNs, which are lacking in transformers, leading the latter to require larger datasets to reach their true potential [, ]. Another intriguing observation is the performance of ViT, which, despite its large number of tokens (or patches), is outperformed by the $\alpha$ and $\beta$ variants of SpecTTTra and is only on par with the $\gamma$ variant in terms of overall F1 score for long audio, reinforcing SpecTTTra’s effectiveness. We hypothesize that this is due to an overload of redundant information from ViT’s numerous patches, which may not contribute effectively to the detection task. Moreover, it can also be observed across all models that real songs are more easily identified than fake ones, as indicated by higher specificity and lower sensitivity scores.

Diving deeper into different partitions of test data, we observe that all detection models achieve better performance on seen algorithms (Suno v3.5 and Udio 130) compared to unseen ones (Suno v2, Suno v3, and Udio 32). Particularly, they struggle more with the Udio algorithms, with the most pronounced difficulty observed for Udio 32. However, ConvNeXt and SpecTTTra-$\alpha$ perform relatively well in detecting the Udio 32 algorithm, achieving a sensitivity of 96% and 95% respectively. Interestingly, despite being an unseen algorithm, the detectors perform comparably well on Suno v3 as they do on the seen Suno v3.5 algorithm, suggesting a possible algorithmic similarity between the two. Conversely, for short audio samples, the detectors perform slightly better on songs with seen speakers than those with unseen speakers, a gap that diminishes when longer audio is used. Finally, in Fake Type partitions, all detectors excel in detecting HF songs, which can be linked to the exclusive presence of Suno algorithms, where the detectors generally perform better compared to Udio algorithms. Among MF and FF songs, the models exhibit a slightly lower performance pattern on FF songs.

**Table 4: AI Benchmark**
| Len.<br>(sec) | Model | Algorithm<br>(Sens.) | Singer<br>(Spec.) | Fake Type<br>(Sens.) | Overall |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Suno<br>v2 | Suno<br>v3 | Suno<br>v3.5 | Udio<br>32 | Udio<br>130 | Seen | Unseen | Half<br>Fake | Mostly<br>Fake | Full<br>Fake | F1 | Sens. | Spec. |  |  |
| 5 | ConvNeXt | 0.62 | 0.99 | 0.99 | 0.62 | 0.99 | 0.99 | 0.99 | 0.90 | 0.82 | 0.80 | 0.90 | 0.82 | 0.98 |
| ViT | 0.79 | 0.95 | 0.98 | 0.57 | 0.86 | 0.79 | 0.79 | 0.92 | 0.78 | 0.76 | 0.79 | 0.80 | 0.79 |  |
| EfficientViT | 0.66 | 0.98 | 0.99 | 0.49 | 0.97 | 0.99 | 0.98 | 0.90 | 0.76 | 0.74 | 0.87 | 0.78 | 0.98 |  |
| SpecTTTra-$\gamma$ | 0.51 | 0.98 | 0.99 | 0.10 | 0.99 | 0.98 | 0.97 | 0.87 | 0.61 | 0.62 | 0.76 | 0.63 | 0.98 |  |
| SpecTTTra-$\beta$ | 0.61 | 0.98 | 0.99 | 0.18 | 0.99 | 0.95 | 0.94 | 0.89 | 0.66 | 0.66 | 0.78 | 0.69 | 0.94 |  |
| SpecTTTra-$\alpha$ | 0.68 | 0.99 | 0.99 | 0.26 | 0.99 | 0.93 | 0.92 | 0.91 | 0.69 | 0.70 | 0.80 | 0.71 | 0.92 |  |
| 120 | ConvNeXt | 0.77 | 0.99 | 0.99 | 0.95 | 1.00 | 0.98 | 0.98 | 0.94 | 0.95 | 0.93 | 0.96 | 0.95 | 0.98 |
| ViT | 0.82 | 0.99 | 1.00 | 0.53 | 0.99 | 0.99 | 0.98 | 0.95 | 0.80 | 0.80 | 0.89 | 0.82 | 0.98 |  |
| EfficientViT | 0.73 | 0.98 | 1.00 | 0.95 | 1.00 | 0.97 | 0.97 | 0.92 | 0.92 | 0.94 | 0.95 | 0.94 | 0.97 |  |
| SpecTTTra-$\gamma$ | 0.98 | 0.99 | 1.00 | 0.37 | 1.00 | 0.99 | 0.99 | 0.99 | 0.77 | 0.76 | 0.88 | 0.79 | 0.99 |  |
| SpecTTTra-$\beta$ | 0.87 | 0.99 | 1.00 | 0.62 | 0.99 | 0.99 | 0.99 | 0.96 | 0.84 | 0.82 | 0.92 | 0.86 | 0.99 |  |
| SpecTTTra-$\alpha$ | 0.78 | 0.99 | 1.00 | 0.96 | 1.00 | 0.99 | 0.99 | 0.98 | 0.89 | 0.87 | 0.97 | 0.96 | 0.99 |  |

#### 4.3.2 Human-AI Benchmark

To evaluate Human performance in comparison to AI-based models, we selected a subset of 520 samples from our large test data. This evaluation employed a dynamic scoring system, similar to LMSYS [], allowing public participation and live leaderboard updates, which will be made publicly available after decision of this paper. Three human participants were involved in this benchmark, with their performance summarized in Table 6. In contrast to the AI benchmark using short (5 sec) or long (120 sec) audio samples, this human benchmark employed 25 sec clips. This choice stems from the observation that short clips hinder human identification due to subtle inaudible artifacts easily detected by AI, while longer clips do not necessarily improve human performance due to how difficult it is to notice long-range temporal dependencies.

As shown in Table 6, AI-based methods consistently outperform human participants across all test partitions. However, both humans and AI models struggle most with Udio algorithms, particularly Udio 32, where human sensitivity dropped to 23%. Conversely, Suno algorithms, especially Suno v3.5, are easier to detect, with a human sensitivity of 82%. This mirrors the findings in the AI benchmark, where models demonstrated higher specificity than sensitivity, indicating greater ease in identifying real songs compared to fake ones. Further analysis revealed distinct patterns within real and fake songs. For instance, Suno algorithms often produced synthetic or mechanical-sounding vocals, while Udio 32 algorithm occasionally created the ”Karaoke effect.” Furthermore, Udio algorithms demonstrated the ability to create songs with multiple voices and higher notes, a feature absent in Suno algorithms. On the other hand, real songs exhibit unique features such as a wide note range, diverse timbre, complex rhythms, clear vocals, and unique sounds like flutes and finger snaps.

**Table 5: Human-AI Benchmark**
| Partition | ConvNeXt | ViT | EfficientViT | SpecTTTra-$\gamma$ | SpecTTTra-$\beta$ | SpecTTTra-$\alpha$ | Human |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Algorithm<br>(Sens.) | Suno v2 | 0.65 | 0.80 | 0.65 | 0.54 | 0.64 | 0.72 | 0.69 |
| Suno v3 | 0.99 | 0.96 | 0.96 | 0.98 | 0.98 | 0.99 | 0.75 |  |
| Suno v3.5 | 0.99 | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 | 0.82 |  |
| Udio 32 | 0.67 | 0.56 | 0.58 | 0.18 | 0.23 | 0.33 | 0.23 |  |
| Udio 130 | 0.99 | 0.87 | 0.98 | 0.99 | 0.99 | 0.99 | 0.55 |  |
| Fake Type<br>(Sens.) | Half Fake | 0.91 | 0.93 | 0.90 | 0.88 | 0.90 | 0.92 | 0.71 |
| Mostly Fake | 0.84 | 0.79 | 0.81 | 0.64 | 0.69 | 0.72 | 0.66 |  |
| Full Fake | 0.83 | 0.77 | 0.78 | 0.64 | 0.68 | 0.72 | 0.63 |  |
| Real (Spec.) | 0.98 | 0.80 | 0.98 | 0.97 | 0.95 | 0.94 | 0.78 |  |
| Fake (Sens.) | 0.85 | 0.82 | 0.82 | 0.66 | 0.72 | 0.75 | 0.66 |  |
| Overall (F1) | 0.92 | 0.82 | 0.87 | 0.78 | 0.80 | 0.83 | 0.71 |  |

#### 4.3.3 Efficiency Benchmark

To comprehensively evaluate the efficiency of the proposed SpecTTTra model alongside other methods, we measure various metrics across different song lengths using a P100 16GB GPU. The metrics considered include Speed (A/S $\rightarrow$ Audio per Second), Floating Point Operations (FLOPs), GPU Memory Consumption (Mem.) during the forward pass with a batch size of 12, activation count (# Act.), and parameter count (# Param.). The results are summarized in Table 6. Our analysis reveals that while ViT is the fastest model for 5 sec songs, it becomes the slowest for 120 sec songs (SpecTTTra-$\alpha$ is 38% faster) and exhibits significant memory consumption (SpecTTTra-$\alpha$ uses 26% less memory), rendering it less practical for longer sequences. However, ViT remains the most efficient in terms of parameter count across both short and long songs. On the other hand, ConvNeXt, despite its strong detection performance, becomes very resource-intensive for longer sequences. It consumes a large amount of memory and has the highest FLOPs and parameter count in that category. EfficientViT shows decent performance but with a surprisingly slow speed for short songs, which is over 2x slower than other models. However, in long songs, it shows better speed and lesser FLOPs than ViT and ConvNeXt but has the largest memory requirement and activation count. In contrast, the SpecTTTra model variants excel in their efficiency without compromising competitive performance in longer sequences. For example, in 120 sec songs, the SpecTTTra-$\gamma$ variant emerges as the fastest and most memory-efficient model, being nearly 3x faster and computationally more economical than ViT while showing competitive performance to it. Similarly, the SpecTTTra-$\beta$ variant is more than 2x faster than ViT and uses 2x less memory, all while achieving 3% higher performance. Performance increase culminates in the SpecTTTra-$\alpha$ variant, which outperforms ConvNeXt and EfficinetViT by 1% and 2% respectively, and achieves the highest F1 score of 97%. It achieves this by being 20% and 9% faster while using nearly 67% and 74% less memory, respectively. Therefore, SpecTTTra has the overall best performance while also being the most efficient model in the detection benchmark.

## 5 Limitations and Future Work

The proposed SONICS dataset contains real songs dynamically queried from YouTube using their titles and artist names, which can sometimes result in incorrect audio retrieval. A manual analysis of 600 random samples suggests that this issue affects approximately 0.5% of the dataset. To address this minor noise, we utilized label smoothing [] during training. Another limitation is that the fake songs generated by the Udio platform cannot include lyrics from real songs, limiting the comprehensive evaluation in the Half Fake songs category to only those generated by the Suno platform. Our current benchmarks are based solely on Mel Spectrogram inputs; hence, we aim to incorporate raw audio and explore other feature extraction methods, such as LFCC and MFCC, to enhance the robustness of our evaluations. Due to resource constraints, we trained and compared only smaller versions of all models. In the future, we plan to compare larger versions of all models. Furthermore, we trained all models from scratch to ensure a fair comparison, because the proposed SpecTTTra model is designed specifically for audio, while other models like ConvNeXt and ViT only have pretrained weights available for images (ImageNet). In the future, we plan to pretrain all models on an large-scale audio dataset from scratch before training them on our proposed dataset for maximizing the performance.

## 6 Conclusion

In this paper, we introduced SONICS, a comprehensive dataset for end-to-end synthetic song detection, addressing limitations in existing datasets, such as lack of music diversity, short duration, and most importantly, the absence of end-to-end AI-generated songs. Moreover, we proposed the SpecTTTra model, which efficiently captures long-range temporal relationships in songs, achieving comparable performance to existing popular models while reducing computational costs significantly. Through extensive experiments, we established both AI-based and human benchmarks, demonstrating the dataset’s effectiveness in advancing synthetic song detection research. This work paves the way for future research to more effectively distinguish AI-generated music, thereby aiding in the preservation of human musical artistry.

## 7 Acknowledgment

We express our gratitude to the Suno and Udio platforms, which we used for generating end-to-end synthetic songs. We are equally thankful to the singers whose songs we used as real song references. We also appreciate the GPU credits provided by Jarvislabs.ai, which were partially used for training our model.

## Appendix A Dataset

Long Form Correlations: An example of the long context correlation can be observed on Fig. 5, where a real song would stay consistent throughout the repetition of the phrases, refrains, rythms, etc. In comparison, synthetic generation methods can have difficulty generating consistent refrains, due to the long form context that might be required to model this information. Therefore, a synthetic song detection model that can capture the nuances of long context correlations present in songs would be able to better differentiate real and synthetic songs utilizing this property.

Figure: Figure 5: Long Context Correlation. The red highlighted regions in the spectrogram and raw audio indicate repetition of the same verse ("Near, far, wherever you are, I believe that the heart does go on"), rhythms, and music. Such consistency is a characteristic of real songs and can be challenging for synthetic generation methods to replicate.
Refer to caption: extracted/5900426/images/sonics-long-context-v4.jpg

Figure: Figure 6: Duration distribution between Training and Testing sets.
Refer to caption: extracted/5900426/images/sonics-duration-dist.png

Song Duration: The distribution of the song duration across training and test sets is shown on Fig. 6. From the figure, it can be observed that aside from a few minor outliers, the distribution of fake and real song duration across both training and test sets are similar.

Embedding Space: The t-SNE plots presented in Fig. 7 illustrate the data distributions within the embedding space, generated using an EfficientNetB0 [] encoder with Mel Spectrogram audio inputs.

- •
Subfigure (a) shows a significant overlap between the training (Green) and testing (Orange) datasets. However, distinct clusters along the edges, predominantly composed of training data, suggest that the training set is more diverse than the test set. This diversity is observed despite the test set containing previously unseen algorithms and speaker scenarios.
- •
Subfigure (b) indicates a substantial overlap between real (Red) and fake (Blue) songs, with a few regions exclusively occupied by fake data or outliers. This highlights the inherent challenge of synthetic song detection within this dataset.
- •
Subfigure (c) shows that the embedding space is primarily occupied by Suno v3.5 (Green) and Udio 133 (Orange), which are seen algorithms. Despite Suno v2 (Blue), Suno v3 (Red), and Udio 32 (Violet) being unseen algorithms, their embeddings fall within regions covered by seen algorithms, demonstrating underlying similarities. A noteworthy observation is the considerable overlap between Suno v3 (Red) and Suno v3.5 (Green), which likely contributes to the similar detection performance in these algorithms, as indicated by Table 4.
- •
Subfigure (d) depicts the distribution of fake types. Although Half Fake (Orange) songs form discernible clusters, Full Fake (Red) songs exhibit a more scattered distribution, indicating a lack of common features. Conversely, Mostly Fake (Green) songs are spread broadly across the embedding space, reflecting their diverse characteristics.

Figure: Figure 7: Data distributions in the embedding space. This plot illustrates the clustering of different data categories such as target, algorithms, fake types, and data split.
Refer to caption: extracted/5900426/images/sonics-tsne.png

Song Style: In order to imitate the song style observed in real songs, we had LLMs perform a stylistic analysis of the real song lyrics (prompt shown in Table 9). The wide variety of styles extracted from real songs are utilized to generate the synthetic songs. It is important to have the distribution of synthetic song style as close to the real song styles in the dataset, since it would be a better indicator of real life cases. A wide diversity of synthetic song style in training data is also required so that the models trained on this dataset do not rely on the distinct characteristics of any particular song style as a feature to detect synthetic songs. Additionally, it is also important to evaluate performance of the trained model on a testing set across a wide variety of song style as well, so that the generalization capability of the detection model across multiple styles is evaluated. A word cloud representation of the song styles that have been extracted from real songs and utilized to generate songs is show on Fig. 8. The figure indicates that there are diverse song styles across both training and testing sets.

Figure: Figure 8: Data distributions of song-style between train and test split in the word cloud.
Refer to caption: extracted/5900426/images/sonics-word-cloud.png

## Appendix B Implementation Details

SingFake Training: To demonstrate the limitations of SVDD (Singing Voice Deepfake Detection) models in detecting end-to-end fake songs, we first trained the models on the SingFake dataset, which consists of 597 training songs and 480 testing/validation songs. We then evaluated the performance of these models on our SONICS-test dataset, with the results summarized in Table 2. It is important to note that due to some inactive links in the SingFake dataset, we were able to download only 1077 songs, as opposed to the 1305 songs originally reported in the paper. For training, we maintained the same model configuration as in other experiments, with one exception: we resized the $128\times 128$ (5 sec) Mel spectrograms to $224\times 224$ to leverage the pretrained ImageNet weights, compensating for the smaller size of the SingFake dataset.

Augmentation: To enhance the robustness of our models, we apply MixUp [] augmentation with $\alpha=2.5$ and a 50% probability. Additionally, we utilize SpecAugment [], applying two time masks of size 8 and one frequency mask of size 8, each with a 50% probability.

Benchmark: For the efficiency benchmark, we utilized a single P100 GPU for all experiments. To measure the inference time of each model, we performed 5 warm-up runs followed by 100 test runs with a batch size of 1 to record the processing time. The results were averaged and then inverted to determine the inference speed of each model. To compute GPU memory consumption, we used a batch size of 14 across all models, measuring the peak memory usage during a single forward pass. For calculating FLOPs, we employed the fvcore [] library.

Model: For the proposed SpecTTTra model, we use the vit_small_patch16 configuration from timm [] library with an embedding dimension of 384, 6 attention heads, 12 Transformer layers, and an MLP ratio of 2.67.

To clarify the core components of the SpecTTTra model, we provide PyTorch-like code for the Spectro-Temporal Tokenizer ($st_{t}okenizer$) below. Note that while the code is presented in a functional format for clarity, the actual implementation follows an object-oriented approach.

[bgcolor=bgcolor, fontsize=]python
import torch
import torch.nn as nn

def st_tokenizer(x, t_clip, f_clip, embed_dim):
B, F, T = x.size()

# Temporal tokens
t_tokens = tokenizer(
x, F, embed_dim, t_clip, T // t_clip
)

# Spectral tokens
f_tokens = tokenizer(
x.transpose(1, 2),
T,
embed_dim,
f_clip,
F // f_clip,
)

# Spectro-Temporal tokens
st_tokens = torch.cat(
(t_tokens, f_tokens), dim=1
)
return st_tokens

def tokenizer(
x, input_dim, token_dim, clip_size, n_clips
):
# Slicing and Tokenization
conv1d = nn.Conv1d(
in_channels=input_dim,
out_channels=token_dim,
kernel_size=clip_size,
stride=clip_size,
bias=False,
)
x = conv1d(x).gelu().transpose(1, 2)

# Positional Embedding
pos_embeds = nn.Parameter(
torch.randn(1, n_clips, token_dim) * 0.02
)
x = x + pos_embeds

# Layer Normalization
x = nn.LayerNorm(token_dim, eps=1e-6)(x)
return x

In this code, the $tokenizer$ function represents the spectral or temporal tokenizer used to embed spectral or temporal clips (patches) into tokens. Here, $f_clip$ and $t_clip$ denote the sizes of the spectral and temporal clips, respectively, while $embed_dim$ signifies the feature dimension of each token. The dimensions $T$ and $F$ correspond to the temporal and spectral dimensions of the input spectrogram.

## Appendix C Human-AI Benchmark

To assess human performance in synthetic song detection, we developed a Huggingface space called ”Song Arena” as illustrated in Fig. 10. In this space, users can evaluate whether a randomly selected song from a subset of the proposed dataset (comprising 520 samples) is synthetic or not. The space also features a leaderboard (shown in Fig. 10) that records human detection performance for songs generated by different algorithms and generation methods. The evaluation metrics used to assess the detectability of synthetic songs include the F1 score, Sensitivity (True Positive Rate), and Specificity (True Negative Rate). These metrics provide a comprehensive measure of the difficulty humans face in detecting synthetic songs generated by various algorithms.

Figure: Figure 9: Song Arena on Huggingface.
Refer to caption: extracted/5900426/images/hf-arena-light.png

## Appendix D Prompt Engineering

Selection of LLMs: For lyrics and song-style generation, as well as lyrics feature extraction, we evaluated several proprietary LLMs, including GPT-4o, Claude 3, and Gemini 1.5, along with open-source models like LLama 3, Gemma 2, and Mistral Large. Among these models, GPT-4o demonstrated superior performance, particularly in maintaining rhythm and coherence and accurately following the content of prompts. Based on these qualities, we selected GPT-4o for both lyrics and song-style generation tasks.

For song-style analysis, only the proprietary models Gemini 1.5 Pro and Gemini 1.5 Flash, as well as the open-source model Qwen-Audio [], are equipped to process audio inputs. Our evaluations indicated that both Gemini 1.5 Pro and Gemini 1.5 Flash models deliver similarly accurate performance. In contrast, Qwen-Audio frequently struggled to follow instructions correctly. Considering that the Gemini 1.5 Flash model is ten times more cost-effective than the Gemini 1.5 Pro, we selected the Gemini 1.5 Flash model for song-style analysis.

Half Fake: These songs are generated using lyrics and song style extracted from real songs. To extract the song style, the prompt template mentioned in Table 9 is used. This template extracts song style information such as vocal type, musical instruments, mood, etc.

Mostly Fake: These songs are generated similarly to Half Fake songs, except that the lyrics are AI-generated. The lyrics are created using an LLM (Large Language Model) with the prompt template shown in Table 8, where lyrics features are used as input. These lyrics features are also extracted from real song lyrics using the prompt template shown in Table 7. The use of lyrics features instead of direct lyrics prevents the LLM from copying the original content, encouraging it to generate lyrics with a similar distribution rather than duplicating them.

Full Fake: These songs are generated using AI-generated lyrics and song style, created using the prompt template provided in Table 10. In this process, the genre, topic, and mood were selected randomly from the lists provided below:

- •
List of Genres: alternative, baroque, blues, bollywood, c-pop, celtic, christian rock, classical, country, crunk, dance, dancehall, disco, doom metal, electronic, folk, funk, fusion, gospel, gothic, grime, grunge, hard rock, heavy metal, hip hop, indie rock, j-pop, jazz, k-pop, lo-fi, lounge, metal, metalcore, new age, opera, orchestral, pop, pop rock, progressive metal, progressive rock, punk, r&b, rap, reggae, salsa, smooth jazz, soul, sufi, world music.
- •
List of Moods:
adventurous, ambivalent, amused, angry, anxious, apathetic, bittersweet, blissful, calm, carefree, cautious, chaotic, confident, confused, curious, desperate, determined, disenchanted, distracted, drained, dreamy, empathetic, enchanted, energetic, exhilarated, focused, forgiving, frustrated, gloomy, grateful, hateful, humble, inspired, introspective, jealous, joyful, liberated, lonely, loving, melancholic, mischievous, motivated, mournful, mysterious, nostalgic, optimistic, passionate, pensive, pessimistic, playful, powerless, proud, rebellious, regretful, reluctant, restless, romantic, sarcastic, satisfied, shocked, skeptical, submissive, sympathetic, tense, timid, trapped, uninspired, vengeful, vulnerable, whimsical, yearning, zealous.
- •
List of Broad Topics:
alien invasion, ancient civilizations, augmented reality, betrayal, childhood memories, climate change, cyber crime, dimensional portals, dreams and aspirations, dystopian future, empowerment, endangered species, extraterrestrial contact, family, fashion, financial struggles, first kiss, forgiveness, friendship, futuristic cities, generation gap, grief and loss, heartbreak, social media anxiety, interstellar travel, loneliness in a crowd, long-distance relationships, love, love at first sight, lunar colonization, nanotechnology, nature’s beauty, nostalgia, ocean exploration, overcoming adversity, pandemic experiences, parallel universes, political revolution, politics, quantum physics, reincarnation, never give up, road trip adventures, robotic emotions, save the planet, sibling rivalry, social influencers, social justice, space exploration, space tourism, survival in the wild, technology addiction, time capsules, time paradoxes, time travel, unconditional love, work-life balance.
- •
List of Specific Topics: elon musk vs yann lecun (AI), yann lecun vs geoffrey hinton (AI), convolution vs transformer (AI), gan vs diffusion models (AI), tensorflow vs pytorch (AI), pytorch vs jax (AI), twilight zone (TV), star trek (TV), game of thrones (TV), breaking bad (TV), stranger things (TV), big bang theory (TV), friends (TV), simpsons (TV), house of cards (TV), how I met your mother (TV), the office - US (TV), sherlock (TV), avatar - the last airbender (TV), pokemon (anime), dragon ball z (anime), naruto (anime), one piece (anime), attack on titan (anime), my hero academia (anime), death note (anime), jujustu kaisen (anime), fullmetal alchemist (anime), demon slayer (anime), neanderthals (anthropology), pyramids of giza (archaeology), machu picchu (archaeology), stonehenge (archaeology), egyptian mummies (archaeology), giza sphinx (archaeology), notre dame cathedral (architecture), london bridge (architecture), big ben (architecture), eiffel tower (architecture), versailles (architecture), arc de triomphe (architecture), mona lisa (art), van gogh’s starry night (art), sistine chapel (art), solar eclipses (astronomy), black holes (astronomy), big bang theory (astronomy), supernovas (astronomy), dark matter (astronomy), andromeda galaxy (astronomy), elon musk vs bezos (business), taylor swift (celebrity), tom cruise (celebrity), brad pitt (celebrity), angelina jolie (celebrity), jennifer aniston (celebrity), leonardo dicaprio (celebrity), meryl streep (celebrity), robert de niro (celebrity), michael jackson (celebrity), matt damon vs jimmy kimmel (celebrity), jimmy kimmel vs jimmy fallon (celebrity), marvel vs dc (comics), batman vs superman (comics), justice league vs avengers (comics), thor vs hulk (comics), iron man vs captain america (comics), batman vs joker (comics), john constantine (comics), marvel universe (comics), dc comics (comics), big bang theory (cosmology), russian ballet (culture), bollywood (culture), hollywood (culture), alphago - the movie (documentary), the great hack (documentary), walt disney (entertainment), rose bowl parade (event), coachella (event), wimbledon (event), kentucky derby (event), rio olympics (event), tokyo olympics (event), beijing olympics (event), columbus (exploration), marco polo (exploration), nike vs adidas (fashion), world war III (fiction), pepsi vs coke (food), messi vs ronaldo (football), pele vs maradona (football), brazil vs argentina (football), monopoly (game), chess (game), go (game), dungeons and dragons (game), minecraft (game), fortnite (game), call of duty (game), mario (game), pac-man (game), sonic the hedgehog (game), fifa (game), cyberpunk 2077 (game), grand theft auto (game), the last of us (game), assassin’s creed (game), resident evil (game), halo (game), the witcher 3 (game), god of war (game), alphago vs lee sedol (games), deep blue vs kasparov (games), playstation vs xbox (gaming), mount everest (geography), grand canyon (geography), dead sea (geography), ring of fire (geography), venetian canals (geography), rocky mountains (geography), volcanic eruptions (geology), ice age (geology), mariana trench (geology), great depression (history), moon landing (history), titanic (history), viking explorers (history), stone age (history), aztec empire (history), mayan calendar (history), vikings (history), cold war (history), space race (history), moon landing (history), fall of the berlin wall (history), industrial revolution (history), edison’s light bulb (invention), the wright brothers (invention), tesla vs edison (inventors), taj mahal (landmark), great wall of china (landmark), central park (landmark), mount fuji (landmark), fifa world cup (sports), empire state building (landmark), statue of liberty (landmark), hollywood sign (landmark), golden gate bridge (landmark), niagara falls (landmark), times square (landmark), king arthur (legend), robin hood (legend), loch ness monster (legend), yeti (legend), camelot (legend), shakespearean sonnets (literature), harry potter (literature), the hobbit (literature), lord of the rings (literature), edgar allan poe (literature), charles dickens (literature), dracula (literature), frankenstein (literature), the great gatsby (literature), pride and prejudice (literature), alice in wonderland (literature), romeo and juliet (literature), moby dick (literature), war and peace (literature), little women (literature), treasure island (literature), oliver twist (literature), peter pan (literature), narnia (literature), aesop’s fables (literature), sherlock holmes (literature), the da vinci code (literature), CNN vs Fox News (media), godzilla (movie), inception (movie), matrix (movie), interstellar (movie), john wick (movie), jason bourne (movie), james bond (movie), spiderman (movie), dark knight (movie), avengers (movie), star wars (movie), indiana jones (movie), back to the future (movie), jurassic park (movie), avatar (movie), wizard of oz (movie), star wars vs star trek (movies), star wars (movies), indiana jones (movies), titanic (movies), fight club (movies), the dark knight (movies), the green mile (movies), gladiator (movies), the departed (movies), the lion king (movies), aladdin (movies), beauty and the beast (movies), little mermaid (movies), frozen (movies), tangled (movies), mulan (movies), sleeping beauty (movies), cinderella (movies), snow white (movies), the social network (movies), the beatles (music), bob dylan (music), bermuda triangle (mystery), crop circles (mystery), atlantis (myth), hercules (myth), perseus (myth), pandora’s box (myth), trojan war (myth), achilles (myth), hades (myth), olympus (myth), zeus (myth), hera (myth), apollo (myth), artemis (myth), athena (myth), poseidon (myth), phoenix (mythical creature), medusa (mythical creature), greek mythology (mythology), yellowstone (national park), yosemite (national park), grand tetons (national park), amazon rainforest (nature), sahara desert (nature), great barrier reef (nature), victoria falls (nature), niagara falls (nature), northern lights (nature), southern lights (nature), bioluminescent bays (nature), blue holes (nature), dinosaurs (paleontology), democrats vs republicans (politics), scientist vs engineer (profession), python vs c++ (programming), java vs python (programming), javascript vs java (programming), paper book vs e-book (reading), physics (science), chemistry (science), biology (science), astronomy (science), mathematics (science), albert einstein (scientist), isaac newton (scientist), charles darwin (scientist), marie curie (scientist), summer vs winter (season), twitter vs facebook (social media), olympics (sports), world cup (sports), super bowl (sports), tour de france (sports), wimbledon (sports), nba finals (sports), nfl playoffs (sports), elon musk vs mark zuckerberg (tech), ai vs human intelligence (tech), nvidia vs amd (tech), intel vs amd (tech), nvidia vs intel (tech), google vs openai (tech), iphone vs android (tech), mac vs pc (tech), ai revolution (tech), rubik’s cube (toy), barbie (toy), lego (toys), europe vs asia (travel), usa vs canada (travel), australia vs new zealand (travel), usa vs europe (travel), switzerland vs sweden (travel), beach vs mountains (vacation), city vs countryside (vacation).

**Table 7: Prompt template for extracting lyrics features (e.g subject matter, theme, mood) from real songs. Here, {lyrics} indicates placeholder for input lyrics.**
| Instructions: |
| --- |
| Analyze the provided song lyrics and extract the following elements: |
| 1. Subject Matter: Write what the song is about by providing a summary of the story, narrative, central topic, or events discussed in the song lyrics. |
| 2. Theme: Identify the main theme or message (emotion, life experience, social commentary, or philosophical concept). |
| 3. Target Audience: Define the intended audience (age group, cultural background, or specific interests). |
| 4. Narrative: The story and point of view (e.g., first person or second person). |
| 5. Character Analysis: Main characters (e.g., Protagonist, Antagonist) and their traits. |
| 6. Song Structure: Outline the structure (number of verses, choruses, bridges, intros, outros) and note any unique elements or deviations. |
| 7. Mood: Describe the overall mood (upbeat, melancholic, introspective, etc.). |
| 8. Reference: Identify any cultural, social, time, place, or contextual references. |
| How to respond: |
| You should provide your answer below after the “Answer” section. You are not allowed to use any text formatting (bold, italic, etc.) and narrative (‘Here is the answer’, ‘Below is the response’, etc.) in your answer. Only answer using the following format: |
| subject_matter: “…..” |
| theme: “….” |
| target_audience: “….” |
| narrative: “….” |
| character_analysis: “….” |
| song_structure: “….” |
| mood: “….” |
| reference: “….” |
| Lyrics: |
| {lyrics} |
| Answer: |

**Table 8: Prompt template for generating song lyrics from lyrics features. Here, {lyrics_feature} indicates placeholder for input lyrics features.**
| Task: |
| --- |
| You are a talented songwriter tasked with creating a song based on the following lyrics features. The song must include all the features described below in the “Lyrics Features” section. The song should not be long; rather, it should be medium-length. |
| Lyrics Features: |
| {lyrics_feature} |
| Instructions: |
| You should write the song with metatags following the song structure from ”Lyrics Features”. In some very rare cases, you can also scarcely include ad libs, or non-lexical vocables. |
| You can add metatags to your lyrics on top of a section in [square brackets] that will create certain styles. Some examples of metatags are [Verse], [Chorus], [Bridge], [Solo], [Outro], [Pre-Chorus], [Bridge], [Hook], [Opening], [Intro], [Instrument], [Build], [Drop], [Breakdown], [Refrain], [Spoken], [Interlude], [Prelude], [Sample], etc. Adding a blank newline between sections yields the best results. |
| In some very rare cases, you can also scarcely use Ad libs (vocal embellishments) to your prompts in (parentheses), only when necessary. Examples include (yeah), (alright), (come on), (whoa), etc. Ad libs tend to work best at the end of a line but can also work mid-line. Unlike metatags, ad libs are sung/verbalized. |
| In some very rare cases, you can also scarcely use non-lexical vocables, only when necessary. Examples include la la la, na na na, sha na na. |
| Example: |
| [Verse] |
| I’ve been tryna call |
| I’ve been on my own for long enough |
| Maybe you can show me how to love, maybe |
| [Chorus] |
| I said, ooh, I’m blinded by the lights |
| No, I can’t sleep until I feel your touch |
| I said, ooh, I’m drowning in the night |
| Oh, when I’m like this, you’re the one I trust |
| Hey, hey, hey |
| Write the lyrics below after the “Lyrics:” section. You are not allowed to add any narrative or text before or after your response such as “Here’s your answer” or “Below is the response”. |
| Lyrics: |

**Table 9: Prompt template for extracting song style (e.g. vocal type, instruments) from audio song. Here, {song} indicates placeholder for input audio song.**
| Task: |
| --- |
| Given a song, you need to conduct a comprehensive stylistic analysis and extract all relevant information about the song’s style. This includes identifying characteristics such as instruments used (e.g., guitar, drums, piano, violin, synthesizers, bass, orchestral, solo, acoustic, trumpet, saxophone, flute, cello, harmonica, banjo, accordion, etc.), vocals types (male or female), genres (e.g., rock, pop, pop rock, indie rock, hard rock, metal, heavy metal, r&b, electronic, soul, jazz, country, reggae, classical, hip hop, blues, folk, punk, funk, disco, alternative, grunge, etc.), tempo (slow, moderate, fast), mood (e.g., melancholic, upbeat, aggressive, melodic, sad, happy, excited, nostalgic, mellow, serene, joyful, dark, gothic, etc.), and any other stylistic elements (e.g., dance, party, cinematic, dreamy, energetic, relaxing, anthemic, atmospheric, groovy, etc.) that contribute to the overall vibe, environment, or atmosphere of the song. |
| Your response should be structured as follows: |
| Answer: |
| <start> |
| style1, style2, style3, ..., styleN |
| <end> |
| For example: |
| Answer: |
| <start> |
| male vocals, electronic, guitar, piano, energetic, pop rock, violin, upbeat, dance, synth, sad, soul, trumpet, reggae |
| <end> |
| Please note that the list of style elements should be comprehensive and cover all relevant aspects of the song’s style. Ensure that your response follows strictly to the specified formatting, including the use of angle brackets, commas, and space separating each element written in lowercase. There must not be any narrative or text in the answer (e.g., ’Here’s your answer’ or ’Below is the response’), only the listed style elements. |
| {song} |

Table: Table 10: Prompt template for generating song lyrics and style from genre, topic and mood. It also generates lyrics feature and song title as by-product. Here, {genre}, {topic}, {mood} indicates placeholders for input genre, topic and mood of the song.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I think the title of the paper and the term real and fake are a bit too generalized terms. 
For example, the proposed dataset is about detection on Suno and Udio generated contents. (not counterfeit songs, maybe synthetic is okay) 
I think the authors can carefully suggest a better title.
soundness: 3
strengths: Overall, the paper is well-written and easy-to-read, and the details of the proposed method, experimental design, the dataset, and the experimental results are well addressed. I think the following researchers will be easily follow and use the proposed dataset, and the models as the authors described them well.
summary: This paper presented a dataset for detecting AI generated music from well-known AI generated music services, Suno and Udio. The authors basically configured two group of music which are fake and real songs. For real songs, they retrieved 48,090 real songs from YouTube and for fake songs, they generated music from Suno and Udio. Finally, they trained a model for detecting whether the song is real or fake. To do this, SpecTTTra model is proposed. In this model, the authors proposed to use spectral and temporal patches (full-size in one dimension) to construct spectral and temporal tokens for their transformer-based classification model. The results showed that the proposed model is highly effective in detection. Also, the authors compared the model performance with Human performance.
weaknesses: It is good to see some research paper in the direction of detecting AI generated content in the domain of music. However, I think the paper lacks of analysis part. The scores that's shown in result tables seem to prove the effectiveness of the proposed method, however, without such in-depth analysis, the proposed dataset and the proposed method might not be useful in the future if the algorithms are updated. Therefore, I think having some analysis section would be beneficial for further future researches. (I think leaving some comments that is related to more high-level explanation on why the proposed model is detecting the AI generated content well, e.g. vocal characteristics, temporal/timbral characteristics, etc) At least, some additional case study on Appendix would be nice. For example, If some real song is detected as a fake, then try to give some analysis on why it's been detected as fake or vice versa.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Major questions/comments:
- Line 81f: "the fake songs in our dataset are free from copyright issues, as they are generated through paid subscriptions that provide a license to use and share the content legally" --- Given that the generation models used are very likely to have been trained on copyrighted data, how can you ensure that the fake songs in your dataset are free from copyright issues? What if generations are identical or similar to existing real music recordings - in particular, when real lyrics and styles are used for generation (cf. "half fake" subset).
- Several steps in the dataset preparation pipeline will probably produce very noisy or incorrect results, but their performance is never evaluated. This is concerning, given that the dataset is the main contribution of this paper. In particular:
  - How well does PyAnnotate work for vocal detection on this data (line 189)?
  - How accurate are the extracted song-styles? This, in particular, I would expect to work rather poorly. Vocal type classification from music audio, for example, is a hard problem.
  - How accurate are the transcribed lyrics?
  - In the cases were generated instead of transcribed lyrics were used: How often do these regurgitate the lyrics from real songs?
- In the proposed SpecTTTra model, I am unconvinced that the proposed spectral clips can provide very helpful information for long songs. The resulting spectral tokens would contain something similar to an average spectrum, which would not be very informative for real vs. fake classification. Indeed, increasing the number $f$ of spectral clips does not improve results in Table 4. However, this was not independently evaluated from the choice of $t$.
- Following up on the previous comment, it seems like results for the SpecTTTra-$\alpha$ model are best, even though this utilizes the proposed architecture the least (low $f$ and $t$). 

Minor questions/comments:
- In Table 1, consider adding "Total hours".
- Line 142f: "Meanwhile, long audio classification remains an uncharted area in audio research." --- This statement seems too strong and should be removed. A wide range of music audio classification tasks involve long audio classification, for example: genre classification, global key detection, global tempo estimation, etc.
- Line 211f: How was the list of topics for the full fake generation obtained? The list given in the appendix seems very biased. Is "matt damon vs jimmy kimmel" really a representative example of a topic for a real song?
- Line 288: Why were significantly more MF songs than HF or FF songs included?
- Line 471: What was the musical experience level of the human annotators? Did they have access to spectral analysis tools (which could be helpful for the fake vs. real classification task)?

Additional comments:
- Line 12: "these inventions democratize music creation" --- This is a highly questionable, opinionated slogan that has its place in marketing but not in science. It should be removed.
- Line 68 (and others): Please do not use the bibliography to refer to companies such as Suno or Udio. A company is not a scientific publication. Same, e.g., for wandb (line 392).
- Table 4: Consider clarifying in the caption what the positive and negative classes are (i.e. what is real and what is fake). Furthermore, indicate which algorithms are seen and unseen during training.
soundness: 3
strengths: The introduction of a dataset for fake vs. real music classification is a welcome and very timely contribution. Different systems by two major commercial providers are considered and the dataset is separated in terms of different levels of fake (fully, mostly, half), which provides interesting insights - notably the half-fake songs with real lyrics seem to be easier to recognize. In the provided transformer architecture, I believe it makes a lot of sense to have one token per time step rather than several tokens for different frequency patches.
summary: This paper introduces a dataset of real and "fake" music recordings, where the fake recordings have been generated using commercial music generation systems. An attempt has been made to match some characteristics between the real and fake recordings. In addition, a new transformer architecture is proposed that is aimed at capturing longer-term temporal context in music audio classification. Finally, results on the fake vs. real music classification task are presented.
weaknesses: Unfortunately, the steps of the dataset preparation pipeline do not seem to be evaluated rigorously, which limits the usefulness of the resulting dataset. I am also not very convinced by the usefulness of the proposed spectral tokens.
Overall, because of these and some other issues (see "Questions"), I recommend to reject this paper, although this is a borderline recommendation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could the authors provide a representative set of audio samples from different categories (Full Fake, Half Fake, Mostly Fake) and platforms (Suno, Udio)?
- What quality control measures were used to ensure the generated songs are suitable for the dataset?
- Could the authors provide ablation studies demonstrating how the separate temporal and spectral tokenization contributes to capturing long-range patterns?
- What was the rationale behind the specific clip sizes chosen for the three variants (α, β, γ)? And how sensitive is the model performance to these hyperparameters?
soundness: 3
strengths: - The paper presents a clear and reproducible workflow for synthetic song generation, with detailed documentation of the prompt engineering process, generation parameters, and data processing steps. 
- The generated song dataset can be an important addition to SVS and SVDD community.
- The Spectro-Temporal tokenizer uses a simple but seemingly effective way to capture the long-range information.
summary: The paper introduces SONICS (Synthetic Or Not - Identifying Counterfeit Songs), featuring the largest synthetic song dataset to date with over 97,000 songs (4,751 hours), including 49,074 AI-generated songs from Suno and Udio platforms. SONICS includes end-to-end generated content with both music and lyrics, and features longer duration songs (averaging 176 seconds). To effectively detect these synthetic songs, the authors propose SpecTTTra (Spectro-Temporal Tokens Transformer), a novel architecture specifically designed for capturing long-range patterns in music. This new model demonstrates superior efficiency compared to traditional approaches, running 38% faster than ViT while using 26% less memory and achieving better detection performance, particularly with longer audio samples. This work provides essential tools for detecting AI-generated music as these technologies become increasingly sophisticated and widespread.
weaknesses: - The paper lacks sample demonstrations of the generated synthetic songs for evaluation. Without access to audio examples, it is difficult for reviewers and readers to assess the quality and diversity of the generated content, as well as validate the claims about different types of synthetic artifacts (e.g., "Karaoke effect"). The authors should provide a representative set of samples across different generation categories (Full Fake, Half Fake, Mostly Fake) and generation platforms (Suno, Udio).
- The naming "SpecTTTra" (Spectro-Temporal Tokens Transformer) is potentially misleading as the architecture only employs a transformer encoder component rather than a full transformer architecture. Furthermore, while the model claims to better capture long-range dependencies, the paper lacks ablation studies to demonstrate this capability. The authors should provide specific experiments showing how the separate temporal and spectral tokenization contributes to capturing long-range patterns in music.
- The performance improvements over baseline models, particularly ConvNeXt, are modest given similar computational costs. The F1 score gain of 1% with 20% speed improvement doesn't strongly justify the introduction of a new architecture. The authors should either provide additional experiments demonstrating clear advantages of SpecTTTra (e.g., ablation studies, analysis of captured patterns), or consider separating the dataset contribution from the model architecture into two papers.
- The quotation mark is used incorrectly.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. The dataset's limited scope to Suno and Udio-generated songs may introduce bias, overlooking other AI music tools like SeedMusic and SkyMusic.
2. The model's high accuracy might reflect genre classification rather than distinguishing real from fake songs due to the focused style of Suno and Udio.
3. The study lacks evidence to confirm the model's ability to identify fake songs independently of audio quality differences.
4. The paper's experimental tables fail to highlight the best-performing metrics, hindering clear comparison.
5. An Ablation Study is missing to assess the model's reliance on Spectral or Temporal information alone.
6. The paper does not compare the model's performance with other methods from the Synthetic Song Detection literature.
soundness: 2
strengths: The paper presents three data acquisition pipelines for fake songs, significantly expanding the dataset for synthetic song detection and providing valuable reference for peers looking to augment this dataset. The paper also introduces a new model for synthetic song detection that achieves better detection results while reducing training costs, offering a new reference method for the field.
summary: This paper introduces SONICS, a large-scale dataset designed for end-to-end synthetic song detection, addressing the limitations of existing datasets such as lack of musical diversity, short song durations, and absence of fully AI-generated music. They also propose a novel model, SpecTTTra, which excels in capturing long-range temporal dependencies within songs, outperforming current models in terms of speed and memory efficiency. The paper establishes both human and AI benchmarks to advance research in distinguishing AI-generated music, thereby protecting artistic integrity.
weaknesses: 1. The data collection for fake songs is not comprehensive, leading to potential biases. Only songs generated by Suno and Udio were collected, while other AI music generation tools such as SeedMusic and SkyMusic exist in the market.
2. Music generated by AI tools from the same company tends to have a consistent style. The authors need to demonstrate how they can distinguish between real and fake songs, rather than just identifying the music styles of Suno and Udio. Suno and Udio's song styles are concentrated in certain genres that, while present in real-world music, are less common. For instance, if the model achieves an 80% accuracy rate, it might mean that 20% of the ground truth song collection consists of pop and rock styles generated by Suno and Udio, while the remaining 80% are styles rarely generated by these tools. Thus, the model might be performing genre classification rather than accurately distinguishing between real and fake songs.
   I suggest that to reduce this interference and truly identify AI and non-AI differences, rather than stylistic differences, the authors could introduce songs generated by other AI tools, different from the training data, as a test set. For example, if the model is trained to distinguish between real and fake songs using data from Suno and Udio, it could be tested with songs generated by SkyMusic, SeedMusic, and other models. If the model can still identify songs from SkyMusic and SeedMusic as fake, it would demonstrate the model's effectiveness.

3. The authors need to prove that they are identifying real songs versus fake songs, not just differences in audio quality. The dataset includes real songs and AI-generated songs. Real songs are sourced from YouTube, where the sampling rate is generally higher and the audio quality is better. In contrast, AI-generated songs mentioned in the paper have a sampling rate of 16kHz/32kHz, which is of poorer quality, and even if upscaled to match YouTube's quality, there would be significant loss in the high-frequency spectrum.
   The paper does not explore or explain this aspect, nor does it mention whether the audio quality was normalized before training and testing. It is unclear whether the model has truly learned to distinguish AI-generated songs or if it has been judging audio quality.

   Whether it's distinguishing between 16kHz resolution songs and high-resolution songs or differentiating between up-sampled 16kHz songs and original high-resolution songs, it's clear that these are audio quality tasks, not real song vs fake song recognition tasks. Only by reducing all to the same low-quality audio can the bias of audio quality on the model be avoided.

4. The experimental tables in the paper do not indicate which metrics are the best, making it difficult to intuitively compare the effectiveness from the table.
5. There is a lack of Ablation Study to show the model's performance using only Spectral or Temporal information.
6. There is no experimental comparison with other models and methods used in Synthetic Song Detection papers.

Meta-review: This paper presented a dataset and a system for detecting AI generated music, using songs from YouTube and fake songs generated from Suno and Udio. The detection system SpecTTTra uses spectral and temporal patches with a transformer-based classification model. 
In the evaluation, the test sets include unseen algorithms and unseen singers, showing a reduced performance on OOD data. 
Considering that the primary contribution of the paper is providing the first dataset for detecting AI generated songs, this seems as valid accomplishment on its own. The objective of creating a robust detection system is only partly achieved as generalization remains a problem, especially considering later experiments with SkyMusic and SeedMusic.

Decision: Accept (Poster)