SONICS: Synthetic Or Not - Identifying Counterfeit Songs
Authors: Md Awsafur Rahman
Zaber Ibn Abdul Hakim
Najibul Haque Sarker
Bishmoy Paul
Shaikh Anowarul Fattah
Venue: ICLR 2025 Poster (2025)
Keywords: deepfake detection
fake song detection
synthetic song detection
efficient model
dataset
audio processing
Primary area: datasets and benchmarks
TL;DR: We introduce SONICS, a large-scale dataset of end-to-end synthetic songs, propose SpecTTTra, an efficient model that captures long-range temporal patterns for effective fake song detection, and provide Human-AI benchmark for comprehensive analysis.

Abstract: The recent surge in AI-generated songs presents exciting possibilities and challenges. These innovations necessitate the ability to distinguish between human-composed and synthetic songs to safeguard artistic integrity and protect human musical artistry. Existing research and datasets in fake song detection only focus on singing voice deepfake detection (SVDD), where the vocals are AI-generated but the instrumental music is sourced from real songs. However, these approaches are inadequate for detecting contemporary end-to-end artificial songs where all components (vocals, music, lyrics, and style) could be AI-generated. Additionally, existing datasets lack music-lyrics diversity, long-duration songs, and open-access fake songs. To address these gaps, we introduce SONICS, a novel dataset for end-to-end Synthetic Song Detection (SSD), comprising over 97k songs (4,751 hours) with over 49k synthetic songs from popular platforms like Suno and Udio. Furthermore, we highlight the importance of modeling long-range temporal dependencies in songs for effective authenticity detection, an aspect entirely overlooked in existing methods. To utilize long-range patterns, we introduce SpecTTTra, a novel architecture that significantly improves time and memory efficiency over conventional CNN and Transformer-based models. For long songs, our top-performing variant outperforms ViT by 8% in F1 score, is 38% faster, and uses 26% less memory, while also surpassing ConvNeXt with a 1% F1 score gain, 20% speed boost, and 67% memory reduction.

Published as a conference paper at ICLR 2025
SONICS: SYNTHETIC OR NOT IDENTIFYING COUN- TERFEIT SONGS
Md Awsafur Rahman" UC Santa Barbara, USA awsa f@ucsb.
Zaber lbn Abdul Hakim; Najibul Haque Sarker' Virginia Tech, USA (zaberhakim666, hajibulhaquel edu
Bishmoy Paul
Shaikh Anowarul Fattah
Santa Clara University, USA bpaul@scu. edu
BUET, Bangladesh
fattah@eee buet acc ac.bc
ABSTRACT
The recent surge in Al-generated songs presents exciting possibilities and chal- lenges. These innovations necessitate the ability distinguish between human- composed and synthetic songs to safeguard artistic integrity and protect human musical artistry. Existing research and datasets in fake song detection only focus on singing voice deepfake detection (SVDD). where the vocals are Al-generated but the instrumental music is sourced from real songs. However, these approaches are inadequate for detecting contemporary end-to-end artificial songs where all components (vocals, music, lyrics, and style) could be Al-generated. Addition- ally, existing datasets lack music-lyrics diversity, long-duration songs, and open- access fake songs. To address these gaps, we introduce SONICS" novel dataset for end-to-end Synthetic Song Detection (SSD), comprising over 97k songs (4,751 hours) with over 49k synthetic songs from popular platforms like Suno and Udio. Furthermore, we highlight the importance of modeling long-range temporal de- pendencies in songs for effective authenticity detection, an aspect entirely over- looked in existing methods. To utilize long-range patterns, introduce SpecTT-- Tra, a novel architecture that significantly improves time and memory efficiency over conventional CNN and Transformer-based models. For long songs, our top- performing variant outperforms VIT by 8% in F1 score, is 38% faster, and uses 26% less memory, while also surpassing ConvNeXt with a 1 1% F1 score gain, 20% speed boost, and 67% memory reduction.
INTRODUCTION
The rapid advancements in Al-generated music present substantial threat the music indus- try, potentially reducing the demand for professional musicians and stifling new talent develop- ment [McMahon 2024 Derbyshire al. (023). To preserve the unique value of human creativity, it is crucial to develop robust methods for detecting Al-generated music, ensuring a fair and vibrant creative ecosystem.
Singing Voice Synthesis (SVS) Liu et al., 2022a) and Singing Voice Conversion (SVC) (Jayashankar al., 2023) have recently achieved significant progress, enabling the creation of synthetic singing voices that closely mimic real singers' styles. When combined with in- strumental music from real songs, these synthetic voices can produce convincing counterfeit songs. Although related to synthetic speech detection, detecting fake songs is particularly challenging due the unique rhythmic patterns and artistic vocal traits of singing (Zang al.) (024b). To address this, researchers have turned their attention to Singing Voice Deepfake Detection (SVDD) (Xie) jet al. 20224] Zang et al. 2024b a). However, current methods relying on datasets composed of SVS and SVC- SVC-generated songs face several limitations. These datasets bound to use only instru- mental music from real songs, leading to artifacts like the "Karaoke effect" (volume discrepancies
"Equal contribution.
Code & Data available wwthttts////gtt httphhh://gthhhb.coom.

---

Published as a conference paper at ICLR 2025
Table Comparison Proposed and Existing Fake Song Datasets
Open Fake Songs
Open Real Sungs
Average Length
Fint- Tinl-Te-Faa
Lyries
Musiela a Diverityy
Open Sourcz
Models
Language
(s00)
Beel Songs
Fake Sonts
Total Hears
Fake
Sone
Dataset
Xie 2014
Chinese
Singl hike Cam K04b
Mulli
13.75
634
671
1,308
58
CDSVDD Zaig e aAzeia
Molti (oo English)
4.57
14
164
32,312
188. 498.486
220,796
307
SONICS (10ns)
Engfish
176.03
5
9,006+
48,090
49.074
97,164
4,751
between music and vocals) and limited music-lyrics diversity. Moreover, existing methods overlook the long-context temporal relationships inherent in songs, such as repeated verses, music, rhythm, and emotional dynamics, which are critical for effective detection. Availability of only short duration songs in current datasets further hampers the use of these patterns. Additionally, copyright restrictions on some existing datasets limit the public availability of generated fake songs, hindering broader usage. Furthermore, the SVDD task requires separate tools for voice identification and separation during data processing (Xie et al. 2024 Zang al.) 2024b), increasing computational overhead.
Recently, platforms like Sunc and Udiq have gained significant traction on social media. They can synthesize not only vocals but also entire songs, including synthetic music, styles, and lyrics, further complicating the situation. Due to their end-to-end nature, these fake songs differ significantly from those generated by SVS and SVC methods, rendering existing SVDD methods and datasets inadequate for detecting them. This necessitates an urgent need for a detection system specifically designed for end-to-end synthetic song detection (SSD).
To address these shortcomings, we introduce SONICS, a large-scale dataset comprising 97,164 songs (4,751 hours), including 49,074 end-to-end synthetic songs (1,971 hours) generated by Suno and Udio, alongside 48,090 real songs (2,780 hours) curated from YouTube. With an average du- ration of 176 second (sec.), the SONICS dataset supports the use of long-context relationships in songs for accurate fake song detection. Furthermore, SONICS addresses the issue of music-lyrics diversity by including a wide range of music styles and both real and synthetic lyrics. A unique feature of SONICS is that it includes the text lyrics of songs, which can aid future research.
Despite the availability of long songs in our dataset, utilizing long-contexs relationships presents additional challenges. For instance, CNN-based models struggle to capture long-range dependencies due to their inherently local receptive fields. While Transformer-baser models can capture these dependencies with global attention, they are computationally expensive with longer audio inputs. To mitigate this trade-off, we introduce soeecrr-Temmpral Tokens Transformer (SpecTTTra), which Tokenizer significantly reduce computational costs while employing global attention. Our contributions are summarized as follows:
We introduce SONICS, the first large-scale dataset for end-to-end synthetic song detection that addresses the limitations of existing datasets, including limited music-lyrics diversity, short-duration songs, and open fake songs.
We provide a human benchmark for fake song detection, filling a gap in previous work, and establish standard AI benchmark for CNN and Transformer-base models.
We highlight the importance of modeling long-context temporal relationships in songs, an aspect entirely overlooked in existing approaches.
We propose faster and memory efficient model, SpecTTTra, which effectively captures long-col oog-context temporal relationships in songs while outperforming popular methods.
2 RELATED WORKS
Synthetic Speech Detection: The domain of synthetic speech detection, closely tied to synthetic song detection through their shared audio modality, has been extensively explored due to advance-
ments in voice conversion (Zhao et al. 2020) and synthesis techniques (Wang et al.) 2021 These
hhtpss//suno.com 2022. Accessed: 2024-06-27
https//uuio.com, 2023, Accessed: 2024-07-09
2

---

Published as a conference paper at ICLR 2025
Table 2: Performance of SingFake-trained models on SingFake SONICS dataset
SingFake
SONICS
Model
EER
F1
Sens.
Spec.
EER
F1 f
Sens.
Spec.
ConvNeXi
0.20
0,86
0,90
0.65
0.38
0.33
0.22
0.88
ViT
0.29
0.84
0.97
0.35
0.49
0.64
0.87
0.16
EfficientViT'
0.19
0.88
0.93
0.64
0.50
0.35
0.28
0.71
advancements have spurred the development of audio spoofing attacks speaker verification sys- tems and deepfake audio targeting human listeners (Kawa et al.] (023). Synthetic speech detection methods include Light CNN (LCNN) with Max-Feature-Map activations (Lavrentyeva al. 2019). Transformer encoders with ResNet architectures (Zhang et al., 2021b). RawNer2 with sinc lay- ers and GRU blocks (Tak al.) 202I), and heterogeneous graph attention networks (Jung et al.) (022). However, the unique complexities of songs songs-such as rhythm, melody, and emotional nu- ance -present challenges that traditional speech detection methods are not equipped to handle, as shown by Xie et al. (2024) and Zang et al. (2024b). Thus, following CtrSVDD (Zang et al. 2024a), we opted not to conduct similar experiments our study.
Synthetic song detection, a relatively newer and more complex chal- lenge, has gained attention recently. In early 2024, SingFake (Zang et al.] 2024b) introduced a dataset of counterfeit songs using Singing Voice Conversion (SVC), along with the task of Singing Voice Deepfake Detection (SVDD) and associated model benchmarks. Subsequent work (Xie et al., 2024 Zang al. 2024a) combined Singing Voice Synthesis (SVS) with SVC to create phoneme- based songs, leading to specialized detection datasets. Methods in this area include convolutional networks for feature extraction followed by classification using graph neural network (Jung al.) 2022], wav2ve22-based extraction coupled with graph neural networks (Tak et al.] 2002). Linear- Frequency Cepstral Coefficients (LFCC) used with ResNet18 models (Zhang et al.] [02 and combination of music-specific models (MERT) & linguistic models (wav2vec2.0) with targeted augmentations (Chen al., 2024). However, SVC and SVS-based datasets retain original back- ground music. leading to a detectable "Karaoke effect" artifact. Recent end-to-end fake songs by Suno and Udio, can produce divergent fake songs where all musical components (e.g.. background music, styles, and lyrics) can be synthetic, presenting severe detection challenge. As shown in Ta- bleß models trained on SingFake dataset, perform poorly when tested on our end-to-end fake songs dataset, with significant drop in detection performance (FI score) ranging from 20% to 64%.
Synthetic Song Detection:
Songs exhibit long-range temporal patterns, such as repeated verses, rhythmss etc. setting them apart from speech (Albouy et al. 2024). Despite their potential to enhance detection performance, these patterns have been largely overlooked in existing methods (Xie et al. 2024 Zang et al., 2024b a). Mcanwhile, long audio classification remains relatively less explored area in audio research. Although automatic speech recognition handles long audio data (Koluguri et al. 2024). it struggles with end-to-end processing of extended audio due to its high computational cost, thus often uses sliding window techniques (Gulati et al. [2020] Radford et al. (003)) to manage costs. This further complicates leveraging long-context features for fake song detection.
Long Audio Classification:
METHODOLOGY
3.1 SONICS DATASET
The development of modern synthetic song detection system necessitates a dataset that meets sev- eral stringent criteria, which are conspicuously absent in existing music datasets. These criteria include: 1) songs where all components not just vocals can be Al-generated: 2) song lengths sufficient to capture long-term contextual relationships; 3) diverse spectrum of music-lyrics com- binations; and 4) quantity of data substantial enough to serve as a generative model benchmark. Addressing these needs, we introduce the SONICS dataset, comprehensive collection of end-to- end Al-generated songs produced using the latest audio generative models, spanning lengths from 32 to 240 sec and encompassing extensive array of music-lyrics styles. A detailed comparison of SONICS with existing datasets is presented in Table I It clearly illustrates that datasets such as FSD (Xie et al., 2024), SingFake (Zang et al. 2024b), and CtrSVDD (Zang et al.) 2024a) fall short of fulfilligg all the outlined criteria when juxtaposed with SONICS, Additionally, a comprehensive distribution summary of the SONICS dataset is provided in Table 3
3

---

Published as a conference paper at ICLR 2025
Table 3: Summary of our proposed SONICS dataset
Full Fake
Half Fake
Mostly Fake
Real
Split
Total
Suno v3.5
Suno x3
Su X2
Udio 32
Suno v3.5
Suno k3
V2
v3.5
Suno K3
S V2
Uddio 32
YouTube
Udio
Udio
Train
774
0
0
4558
11863
16172
0
0
0
32686
66709
Test
53
206
100
203
396
634
309
1117
1566
2797
1350
3970
13237
26015
# Valid
9
12
37
16
30
67
117
51
196
286
494
258
700
2167
4440
3.1.1
REAL SONGS
The dataset's Real songs segment comprises original compositions by human artists. This portion
of the dataset initially compiled by random sampling from the Genius Lyries Dataset (T. 2023). which provides metadata including lyrics, titles, and artist information. Subsequently, leveraging this metadata, a search was conducted dynamically on YouTube retrieve the corresponding audio files. This process yielded 48,090 songs performed by 9,096 artists.
3.1.2 FAKE SONGS
For generating end-to-end fake songs, we utilized the only two currently available audio genera- tive model families: Suno and Udio. Specifically, used three variants from Suno-Suno v2, v3, and v3.5. Suno v2, the earliest iteration, generates songs up to 80 seconds (sec.) in length. Suno v3 extends this capability to 120 see. The latest iteration, Suno v3.5, produces songs up to 240 sec. From the Udio family. we used Udio 32 and Udio 130, capable of generating songs of 32 and 130 sec, respectively. For generating and downloading songs from Suno, utilized the community API (gcui art 2024). In contrast, for Udio, we manually generated the songs from the website and then downloaded them using community API (Rieral 2024). Subsequently, used PyAnnote (Bredin & Laurent (021) to filter out audio files lacking vocal components, which were predominantly found among the Udio-generated song.
Topic
(Topic; Genre, Mood) Ginneatirr
Lyrics Jake
Random
Song fake
Prompt
Genre
Generntor
CPT-4e
Prompi
Song
Style
Mood
Edio
Figure I: End-to-end pipeline of Full Fake song generation process. Here a random combination of topic, genre, and mood is utilized to create fake lyrics and fake song styles by prompt engineering LLM GPT-4o. The generated lyrics and styles are utilized to create synthetic songs through audio- generative models from Suno and Udio.
The Fake songs are generated with Suno and Udio models by two primary text inputs: 1) lyrics and 2) song-style (e.g.. instruments, genre, vocal type) where for each combination of (lyrics, style) input, two songs are generated. Based on the nature of these inputs, the Fake songs are categorized into three distinct groups: i) Full Fake (FF) featuring both Al-generated lyrics and song-style; ii) Mostly Fake (MF)-where lyrics are Al-generated based on real lyrics features and song-style is derived from real songs; and fii) Half Fake (HF)-where where the lyrics are directly sourced from real songs, with song-style also extracted from real songs. To generate FF songs, we curated rich set of metadata, including broad topics (e.g., friendship. betrayal), 292 specific topics (e.g., star trek, pokemon), 49 music genres (e.g., rock, metal), and 72 moods (e.g., calm, angry). The full list can be found in the Appendix. Random combinations of these elements were used to generate final lyrics and styles via Large Language Model (LLM). The complete pipeline for FF song generation is illustrated in Fig. [ Here the prompt generator generates prompt for later stages by filling in the variable values into the placeholders of predefined prompt template (details in Appendix HH. In
4

---

Published as a conference paper at ICLR 2025
Igrics
Prenyt
Zyrice
Genemnor
Generatar
Prosept
Lyples Faieans
CPF-4a
Sind
Song
Pronye
Song
Generator
Stylevad
Fluck
Figure 2: End-to-end pipeline of Mostly Fake song generation process, Firstly, lyrics features are extracted using LLM from real song lyrics, which in turn is utilized create fake lyries through a second call to the LLM GPT-4o. Secondly, style information is extracted from real songs using multimodal Generative model Gemini 1.5 Flash. Finally, the generated lyrics and styles are utilized to create synthetic songs through audio-generative models from Suno and Udio.
contrast, the creation of MF and HF songs involved extracting song-styles from real songs using the multimodal LLM Gemini 1.5 Flash, which uniquely supports audio input. Additionally, for MF songs, lyrics were generated using the GPT-4o LLM, which extracts features from existing real song lyrics to produce new lyries with similar distribution, avoiding direct replication of existing content. The pipeline for generating MF songs is detailed in Fig. 2 The HF songs, differing only in the use of real lyrics directly sourced from existing songs, follow similar process MF songs. It is noteworthy that, Udio models cannot generate songs for lyrics from real songs; thus, HF songs contain only Suno models. In total, we generated 6, 6,132 HF songs, 40, 70,769 MF songs, and 2,173 FF songs using Suno and Udio models. The prompt templates used in the LLM to extract lyrics features and song styles, as well as to generate lyrics and styles, can be found in the Appendix.
3.2 MODEL
The architecture of an audio classification model depends on how the input audio is processed for feature extraction. In our benchmark, we use Mel-spectrogram due to its versatile usage and effective performance across various audio processing tasks (Gong et al. 202T Radford et al. 2023 Rouard et al. 2023 Niizumi et al.) 2024 Shul & Choil 2024). Spectrograms, resembling 2D shape of an image, allow the usage of image classification models for audio classification. Since image clas- sification models are generally categorized into CNN-based and Transformer-based architectures, we use popular models from both categories in our study, such as ConvNeXt (Liu et al., 2022b) and ViT (Dosovitskiy et al., 2020), for benchmarking. Additionally, jet al. 2023), a hybrid model that integrates both Transformer and CÑN components. However, these models encounter challenges when dealing with the inherent long-range temporal dependencies in songs. Specifically, ConvNeXt struggles to capture long-range dependencies due to its local recep- tive field, a characteristic inherited from CNNs. On the other hand, while ViT capable of capturing long-range dependencies, it incurs significant computational costs as the number of patches/tokens rapidly increases with longer audio inputs. Similarly, EfficientVTT, despite its linear global attention, becomes computationally expensive for long audio due to the large number of tokens combined with its multi-scale operations. To address this issue, we introduce SpecTTTra. which utilizes global at- tention similar to ViT but employs Spectro-Temppral Tokenizer to reduce the number of tokens considerably, thereby lowering computational costs and enhancing efficiency significantly.
employ Efficient VIT (Cai
3.2.1 SPECTRO-TEMPORAL TOKENS TRANSFORMER (SPECTTTRA)
As illustrated in Fig. 3 the proposed SpecTTTrra model begins by applying temporal and spectral slicing copies of the input spectrogram, generating distinct temporal and spectral clips/patches indicated by |6,] and |fi respectively where denotes the position of each clip. These clips are then processed by the Spectro- tokens (tz) and spectral clips as spectral tokens (f.). using separate tokenizers, where each token is represented as a vector of shape (1, embed dim). Subsequently, separate positional embeddings (p) for temporal and p for spectral) are added to the respective tokens. Separate embeddings are used because there is no positional relationship between temporal and spectral tokens.
Tokenizer, where temporal clips are embedded as temporal
The positionally aware tokens are then fed into a ViT-like Transformer encoder, where they are contextualized with one another through global attention. This process enables the temporal tokens
5

---

Published as a conference paper at ICLR 2025
Temporal
Speciro Temporal
Postional Embreding
Tokmss
Clips
Temporal Slicing
Temporal Tokenizer
Falso Song
Input
Average Pooling
Transformer Encoder
Transoso
Classiin
Spectral
Siicing
Song
Spectral
Tokenizer
Spectral Clips
Tokenizer
Tokens Transformer (SpecTTTra) model. First, the input mel- spectrogram undergoes separate temporal and spectral slicing to generate corresponding clips, which tokenized into temporal and spectral tokens using separate tokenizers. Next, separate positional embeddings are added to these tokens, which are then passed through Transformer encoder. The resulting globally contextualized features are then average pooled and finally passed the classifier.
Figure 3: Proposed
to become aware of both other temporal tokens and spectral tokens, and vice versa for spectral tokens. These globally aware features, now of shape (n_lokens, across the n_lokens dimension to aggregate the temporal and spectral information. Finally, the accumulated features are passed classifier, where they are classified as either real or fake songs.
are average-pooled
It is important to note that while previous work has attempted to utilize both spectral and temporal information, these approaches come with significant limitations. For instance, Yadav al. (2023); Gong et al. (2021) focuses solely on temporal tokens, neglecting the rich spectral information. On the other hand, methods like Zadeh al. (2019); (Shul Choi (2024) employ separate attentions for spectral and temporal information but with ViT-like tokens, which, as discussed in the follow- ing section, become highly inefficient and computationally expensive as they rapidly increase with longer audio inputs. In contrast, our approach disentangles spectral and temporal information the tokenization level and later contextualizes them with attention, leading to greater efficiency.
3.2.2 SPECTRO-TEMPORE TOKENIZATION
The ViT model creates grids small square patches by simultancously dividing a 2D spectrogram along both temporal and spectral dimensions, resulting in each patch having access only limited spectral and temporal information. More critically, as indicated in Eq. [ the number patches/- tokens (N.) increases rapidly with the temporal dimension T. For example, for a spectrogram of size F 128 (spectral dimension) with a short audio duration of sec (I' 128). ViT (patch size, p 16) generates tokens N, 64. However, for longer audio with 120 (T' 3744), the number of tokens surges to N, 1872, nearly 30x more than for the shorter audio. Since the computational cost for global attention in VIT scales quadratically with the number of tokens, this makes less practical for long audio classification. Similarly, EfficientViT, despite having linear global attention, multi-scale operations coupled with numerous tokens still result in high complexity.
In contrast, SpecTTTra performs slicing independently for temporal and spectral dimensions, as shown in Fig. This approach ensures that each temporal patch has access to the full spectral infor- mation, and each spectral patch contains the complete temporal information. This design leverages the observation that meaningful correlations can exist between distant temporal clips (e.g., between Itol and [tal. capturing repeated song verses) between distinct spectral clips (e.g.. between [fo] and l. capturing harmonics). Importantly, as demonstrated Eq. 2 due to the additive nature of token generation in SpecTTTra, the number of tokens grows more slowly compared to ViTs, signif-

---

Published as a conference paper at ICLR 2025
Equation for VIT Token Count
T and F denote the temporal and spec- tral dimensions of the spectrogram, respec- tively, and p the patch size, then the num- ber of patches (or tokens) generated by a VIT (N.) can be expressed as:
Number Tokens (N)
10a
3.44
F p
T P
N,
(1)
10a-
Equation for SpecTTTra Token Count If t and f represent the sizes of the tempo- ral and spectral clips, respectively, then the number of patches (or tokens) generated by SPecTTTraa denoted as No., is given by:
SpecTTTraaa
SpecTTTra-8
SpecTTIra-y
128
512
1024
2048
3744
4096
Time Frame (T)
Ny Nspectral + Ntesmporal
Comparison of the number of to- kens generated by the ViT model and the three SpecTTTra variants (a, 3, and 7) as a function of the number of time frames.
Figure 4:
F
T
(2)
+
icantly reducing computational costs. For instance, using the same parameters as before and setting the temporal and spectral clip sizes to t = 7 and f = 5, the number of tokens in TTea for short audio N₁ = 43, and for long audio, it increases
Nº 560, which approximately 3.4x fewer than in the VIT model. This substantial reduction in token count makes SpecTTTra significantly more computationally efficient. We further explore three variants of SpecTTTra, differentiated by the sizes of their temporal (1) and spectral (J) patches: SpecTTTrr-a (J 1, 3), Spec
(ʃ 3, = 5), and SpecTTTra-> (f 5, L 7). Fig. illustrates the rate which the number of tokens increases for different SpecTTTra variants and VIT the audio length grows.
The resulting spectral and temporal clips are processed by the Spectro-Tempor Tokenizer (STT) to create spectral and temporal tokens, respectively. The proposed STT block consists of separate Spectral and Temporal Tokenizers, which share identical architecture but differ in their objectives based their input. These tokenizers spectral/temporal tokens, followed by GELU activation, addition of learnable positional embeddings similar ViT, and finally, layer normalization. For efficiency, during implementation, the slicing and tokenization operations are merged using 1D CNN layer. Given an input spectrogram x E Fxx, the mathematical expression for tokenization is follows:
Linear layer map the spectral/temporal clips to
2G ConviD (z, F, D, t, =
(t) LayerNorm (GELU (x,) D')
(3)
ff CDnv (x .q T,C, = D, k = f-s=f)
(f) LayerNorm (GELU (x]) + +
Here, (t) E F and (f) F denote the temporal and spectral tokens, where D is the token embedding dimension. The parameters c,, Co, k, and represent the input channels, output channels, kernel size, and stride size of the CNN layers. Additionally, and T denote the learnable positional embeddings and Transpose operation. Further implementation details can be found in the Appendix.
EXPERIMENTS AND RESULTS
4.1 DATASET
We conduct all experiments using the proposed SONICS dataset, which is divided into train, valid, and test sets. To ensure comprehensive evaluation, the valid and test sets include cases with unseen algorithms (c.g., Suno v2, Suno v3, Udio 32) and unseen singers. We also prevent data leakage by ensuring that song pairs from the same (lyrics, style) inputs are exclusively either the training or valid-test sets, not in both. The distribution of the train, test, and valid sets is shown in Table

---

Published as a conference paper at ICLR 2025
4.2 IMPLEMENTATION DETAILS
To train models, we resampled both real and fake songs to 16kHz and generated spectrograms with n.fft= = win length 2048, pop_length- 512, and n memels = 128. yielding a 128 x 128 spectrogram for 5 sec and 128 x 3744 for 120 sec audio. Any song shorter than input length is sero-padded randomly, while for longer songs, a random crop is used. We also apply MixUp (Zhang 2017) and SpecAugment (Park et al. (2019) augmentations during training to improve generalization. However, during the test, to maintain determinism, padding is done on the right side and cropped segments taken from the middle. We conduct our training
NVIDIA A6000 GPU with 48GB RAM, using WandB for tracking. We use ViT-small (patch size- = 16) and ConvNeXt-tiny along with EfficientViT-BB from the timm [Wightman] 2019) library. In SpecTTTrra we use the same model configuration as ViT-small. We train all models for 50 epochs from scratch using Binary Cross- Entropy loss with 0.02 label smoothing (Szegedy
al.) 2016). Optimization is performed with AdamW (Loshchilov) 2017) and a cosine learning rate scheduler from timm, including a S-epoch warmn-u. While existing methods (Zang et al.) 2024b a Xie et al., 2024) use Equal Error Rate (EER) as metric, we prioritize the Fl score (binary average, threshold = 0.5) as our primary metric due to EER's 's susceptibility to class imbalance. We also evaluate Sensitivity (Sens.) and Specificity (Spec.) to assess performance across fake and real classes.
4.3
BENCHMARKS
4.3.1
AI BENCHMARK
The comparative analysis of the proposed SpecTTTra models against other existing models is pre- sented in Table The results reveal significant performance gain (6% for ConvNeXt, 8% for EfficientViT, 10% for ViT, and 17% for SpecITTTra-a) in the overall F1 score when using long songs. This finding substantiates our claim that leveraging long-context information is crucial for enhancing fake song detection. Additionally, the advantage of longer audio duration is more preva- lent in transformer-basee model, compared to the CNN-based ConvNeXt. Notably, the proposed SpecTTTra-a, while trail- ing Ne by 10% in the F1 score for short audio, outperforms it in long audio. This can be attributed to the global attention mechanism in transformer models, which effectively captures long- range dependencies within the input data. In contrast, models with CNN components tend to perform better on shorter audio. Specifically, ConvNeXt and EfficientViT achieve overall Fl scores of 90% and 87%, respectively, outperforming all transformer- based models on short audio. However, de- spite the absence of global attention, ConvNeXt demonstrates competitive performance compared to SpecTTTra-a on long audio and outperforms ViT, EfficientViT, and other SpecTTTra variants in both short and long audio scenarios. We hypothesize that this is due to the inherent inductive biases present in CNNs, which are lacking in transformers, leading the latter to require larger datasets to reach their true potential (Liu et al. [20T]] Dosovitskiy et al., 2020). Another intriguing observation is the performance of VIT. which, despite its large number of tokens (or patches), is outperformed by the a and / variants of SpecTTTra and is only on par with the y variant in terms of overall Fl score for long audio, reinforcing SpecTTTra's effectiveness. We hypothesize that this is due to an overload of redundant information from VIT's numerous patches, which may not contribute effec- tively to the detection task. Moreover, it can also be observed across all models that real songs are more easily identified than fake ones, as indicated by higher specificity and lower sensitivity scores.
models such as ViT and SpecTTTra, as well as in the hybrid EfficientViT
Diving deeper into different partitions of test data, we observe that all detection models achieve better performance on seen algorithms (Suno v3.5 and Udio 130) compared to unseen ones (Suno v2, Suno v3, and Udio 32). Particularly, they struggle more with the Udio algorithms, with the most pronounced difficulty observed for Udio 32. However, ConvNeXt and SpecTTTra-a perform rela- tively well in detecting the Udio 32 algorithm, achieving a sensitivity of 96% and 95% respectively. Interestingly, despite being an unseen algorithm, the detectors perform comparably well on Suno v3 as they do on the seen Suno v3.5 algorithm, suggesting a possible algorithmic similarity between the two. Conversely, for short audio samples, the detectors perform slightly better on songs with seen speakers than those with unseen speakers, a gap that diminishes when longer audio is used. Finally, in Fake Type partitions, all detectors excel in identifying HF songs, likely due to the exclusive pres- ence of Suno algorithms, where detectors generally perform better compared to Udio algorithms. Among MF and FF songs, the models exhibit slightly lower performance pattern on FF songs.
8

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison of SpecTTTra and conventional AI models on varying audio lengths, with F1 score as the primary evaluation metric. Here Real/Human and Fake/Al songs denoting Negative and Positive classes, respectively. indicates unseen algorithms during training.
Algorithm
Singer (Spec.)
Fake Type
Len. Model
(Sens.)
(Sens.)
Overall
(sec)
Suno v2
Suno v3
Suno
Udio 32
Udio
Half
Mostly Fake
Full Fake
v3.5
130
Seen
Unseen
Fake
F1
Sens.
Spec.
ConvNcXt
0.62
0.99
0.99
0.62
0.99
0.99
0.99
0.90
0.82
0.80
0.90
0.82
0.98
VIT
0.79
0.95
0.98
0.57
0.86
0.79
0.79
0.92
0.78
0.76
0.79
0.80
0.79
5
EfficientViT
0.66
0.98
0.99
0.49
0.97
0.99
0.98
0.90
0.76
0.74
0.87
0.78
0.98
Tpe:TTTTm-
0.51
0.98
0.99
0.10
0.99
0.98
0.97
0.87
0.61
0.62
0.76
0.63
0.98
Spe:TTTm-3
0.61
0.98
0.99
0.18
0.99
0.95
0.94
0.89
0.66
0.66
0.78
0.69
0.94
SpecTTTTa-
0.68
0.99
0.99
0.26
0.99
0.93
0.92
0.91
0.69
0.70
0.80
0.71
0.92
CunvNeXi
0.77
0,99
0.99
0.95
1.00
0.98
0.98
0.94
0.95
0.93
0.96
0.95
0.9%
ViT
0.82
0.99
1.00
0.53
0.99
0.98
0.80
0.80
0.89
0.82
0.98
EllicientViT
0.73
0.98
1.00
0.95
1.00
0.97
0.97
0.92
0.92
0.94
0.95
0.94
0.97
120
SpeeTT'Tra-
0.99
1.00
0.37
1.00
0.99
0.99
0.77
0.76
0.88
0.79
0.99
Spe
0.87
0.99
1.00
0.62
0.99
0.99
0.99
0.96
0.84
0.82
0.92
0.86
0.99
SpecTTTTraa
0.78
0.99
1.00
0.96
1.00
0.99
0.98
0.89
0.97
0.96
0.99
Table Comparison conventional models and TTeaa against human evaluators.
Table 6: Comparison of SpecTTTra against con- ventional models on efficiency related metrics.
ComNNcK
Efhiciet//
3pe
SpecTTh
Human
Len. (sec)
ConsNeXi
Model
Speed (A/S)
137
PLOPSs
(G) L
1.5
Mem. (GB)
0.1
Act. (M).
4
Param.
(M)
28
Partition
VTT
EffecentViT
VIT
156
55
0.6
1.1
0.2
2
17
22
SpeeITTra>>
154
Suno x2
0.05
0.80
0.63
0.53
0.64
0.72
0.09
SpeeTTTm..
152
17
Fake Type
Algurihm (Sens.)
(Sens.)
Suno v3.5
Udoo 32
Falr
Cdo 130
Full Fake
Suno v3
Masily
0.34
0.99
0.67
0.83
0:99
0.99
0.87
0.56
0.95
0.98
0158
0.98
0.98
8.78
0.96
0.99
0.64
0.18
0.98
0.99
0.23
0.99
0.99
0.98
0.99
0.33
0.99
0.72
0.99
0.71
00.25
0.82
0.75
0.55
0.66
0.63
SpecTTimaa
SpeeTTiha
SpeeTTTra-
EliicentViT
ConeNeXi
VIT
148
80
34
39
97
31.7
43.1
10.1
14.0
299
11.7
233
0.5
129
67
20
29
6
21
28
24
17
17
Real (Spec.)
0.58
0,80
0.98
0.97
0.95
0.94
0.78
SpeeTTTa-e
47
23.7
3.9
50
19
Take (Sens.)
0.35
0.82
0.82
0.66
0.72
0.75
0.66
Overall (PI)
0.2
0.8
0.57
0.78
0.50
0.8
0.71
4.3.2 HUMAN-AI BENCHMARK
To evaluate Human performance in comparison Al-based models, we selected a subset of 520 samples from our large test data. This evaluation employed a dynamic scoring system, similar to LMSYS (Chiang et al. 2024). allowing public participation and live leaderboard updates, which will be made publicly available after decision of this paper. Three human participants were involved in this benchmark, with their performance summarized Table 5 In contrast to the Al benchmark using short (5 sec) or long (120 sec) audio samples, this human benchmark employed 25 sec clips. This choice stems from the observation that short clips hinder human identification due to subtle inaudible artifacts easily detected by AI, while longer clips do not necessarily improve human per- formance due to how difficult it is to notice long-range temporal dependencies.
As shown in Table Al-based methods consistently outperform human participants across all test partitions. However, both humans and AI models struggle most with Udio algorithms, particularly Udio 32, where human sensitivity dropped to 23%. Conversely, Suno algorithms, especially Suno v3.5, are easier to detect, with a human sensitivity of 82%. This mirrors the findings in the AI benchmark, where models demonstrated higher specificity than sensitivity, indicating greater ease in identifying real songs compared to fake ones. Further analysis revealed distinct patterns within real and fake songs. For instance, Suno algorithms often produced synthetic or mechanical-soune sounding vocals, while Udio 32 algorithm occasionally created the " "Karaoke effect." Furthermore, Udio al- gorithms demonstrated the ability to create songs with multiple voices and higher notes, a feature absent in Suno algorithms. On the other hand, real songs exhibit unique features such wide note range, diverse timbre, complex rhythms, clear vocals, and unique sounds like flutes and finger snaps.
4.3.3 EFFICIENCY BENCHMARK
To comprehensively evaluate the efficiency of the proposed SpecTTTr model alongside other meth- ods, measure various metrics across different song lengths using P100 16GB GPU. The metrics

---

Published as conference paper at ICLR 2025
Table Ablation analysis of temporal and spectral tokens on model performance.
120 sec
Temp. Clip
# Temp.
Spec. Clip
# Spec.
Temp. Clip
Temp.
Spec. Clip
Spec. Tok. (F/n)
Size (1)
Tok. (T/t)
Size (f)
Tok. (F/f)
Size (1)
Tok. (T/t)
Size (f)
3
128
0
0.76
1248
0,91
3
128
5
25
0.78
1248
0.94
3
1248
0,96
3
128
1
128
0.80
1248
128
0,97
0
1
128
0.75
0
1
128
0,92
considered include Speed (A/S - Audio per Second), Floating Point Operations (FLOPs), GPU Memory Consumption (Mem.) during the forward pass with a batch size of 12, activation count (# Act.), and parameter count (# Param.). The results are summarized in Table O Our analy- sis reveals that while ViT is the fastest model for 5 sec songs, it becomes the slowest for 120 sec songs (SpecTTTra-c is 38% faster) and exhibits significant memory consumption (SpecTTTraae uses 26% less memory), rendering less practical for longer sequences. However, ViT remains the most efficient in terms of parameter count across both short and long songs. On the other hand, ConvNeXt, despite its strong detection performance, becomes very resource- rrsource-intensive for longer se- quences. It consumes large amount of memory and has the highest FLOPs and parameter count in that category. EfficientVVT shows decent performance but with a surprisingly slow speed for short songs, which over 2x slower than other models. However, and lesser FLOPs than VIT and ConvNeXt but has the largest memory requirement and activation count. contrast, the SpecTTTra model variants excel in their efficiency without compromising competitive performance in longer sequences. For example, in 120 sec songs, the SpecTTTra-> variant emerges as the fastest and most memory-efficient model, being nearly 3x faster and compu- tationally more economical than VIT while showing competitive performance to it. Similarly, the SpecTTTra-BB variant is more than 2x faster than ViT and uses less memory, all while achiev- ing 3% higher performance. Performance increase culminates in the SpecTTTra-a variant, which outperforms ConvNeXt and EfficinetViT by 1% and 2% respectively, and achieves the highest score of 97%. It achieves this by being 20% and 9% faster while using nearly 67% and 74% less memory, respectively. Therefore, SpecTTTra has the overall best performance while also being the most efficient model in the detection benchmark.
long songs, it shows better speed
4.4 ABLATION STUDY
We conduct an ablation study to highlight the importance of both temporal and spectral tokens, with the findings summarized Table 7 Additionally, we vary the number of spectral tokens independently of temporal tokens to evaluate their impact performance. Specifically, we change the clip size (f, f) relative to our best- pest-performing model, SpecTTTra-a' (1 = 3, f 1), to derive further insights. Notably, while it is possible to classify real and fake songs using only temporal tokens (F/f = 0) or only spectral tokens (T't = 0), the combination of both clearly yields the best performance, underscoring their complementary nature. Furthermore, increasing the song duration consistently enhance performance for both spectral and temporal tokens, reinforcing our assertion about the significance of long-context information.
5 CONCLUSION
In this paper, we introduced SONICS, comprehensive dataset for end-to-end synthetic song de- tection, addressing limitations in existing datasets, such as lack of music diversity, short duration, and most importantly, the absence of end-to-end Al-generated songs. Moreover, we proposed the SpecTTTra model, which efficiently captures long-range temporal relationships in songs, achiev- ing comparable performance existing popular models while reducing computational costs sig- nificantly. Through extensive experiments, we established both Al-based and human benchmarks, demonstrating the dataset's effectiveness in advancing synthetic song detection research. This work paves the way for future research to more effectively distinguish Al-generated music, thereby aiding in the preservation of human musical artistry.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
The dataset incorporates copyrighted song data from YouTube. To comply with legal standards, we provide only YouTube links to the original songs and will publicly release only the Al Al-generated songs. Given that the generative models used to generate these fake songs may have been trained on copyrighted songs, there could be potential concerns regarding the copyright status of our dataset. However, even if the generative models were trained using copyrighted data, our dataset falls under the fair use policy U.S. Code (2023) according research criteria. The same is true for additional metadata that was used to generate the dataset, including lyrics, style, etc. Thus, the use of these models and relevant metadata for generating our dataset is justified. Furthermore, the practice of using generative models (which likely have been trained on copyrighted data) to create large-scale datasets has been documented in the literature and published in neer-reviewed, widely accepted conferences. Notable examples include LLVV--Instruct1558
(from GPT-4) Liu et al. (2024), Gpt4tools (from ChaiGPT) Yang et al. (2024). Camel (from GPT-3.5 Turbo) [i al. (2023), the Baize Dataset (from ChatGPT) Xu et al. (2023), and the JourneyBench Dataset (which uses GPT- 4V and GPT-4O) Wang et al. (2024). among others.
Additionally, there might be concerns that these generated models regurgitate copyrighted music. To appease these concerns and verify whether the generated songs are identical to existing real songs, compared all the generated songs with real songs in pairwise manner using the cosine similarity metric with EfficiintNerB0 embeddings as representations. Among the top 50 songs with the highest similarity real songs, manually inspected each one and found no exact matches. All these songs exhibited variation in elements such as music style, vocals, instrumentation, or other features, including the "half fake" subset, where only the lyrics were shared. Given this variation, our work also falls under fair use policies U.S. Code (003). Finally, as these fake songs are generated through paid subseriptions that allow for the use and sharing of content, our dataset will be made publicly available under a CC BY-NC 4. 0 license.
We also acknowledge issues related to bias and fairness. The dataset is currently limited English- language songs, which affects its global applicability. Future work will address this by expanding the dataset include more languages. Notably, a gender bias is evident, with male singers dominating the song styles, trend that may stem from either the real songs the Gemini 1.5 Flash model that was used to extract song styles (Half Fake and Mostly Fake songs), or GPT-4 that was used to generate song styles (Full Fake Songs). Addressing this gender bias is beyond the scope of our study, and we leave it to the community to tackle future research.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have made extensive efforts to document and share all necessary de- tails. First, we provide the complete dataset generation process, including the end-to-end pipelines. The Appendix offers additional information, such as dataset statistics, to help better understand the data. Second, the pseudo-code for the Spectro-Temporal Tokenizer of the SpecITTra model is presented in the Appendix. All hyperparameters, training setups, and augmentation methods are detailed in the "Implementation Details" section of both the main paper and the Appendix. Third, include all assumptions and configurations for the benchmarks, which are available in both the main paper and the Appendix. Finally, the source code is provided in the supplementary materials, which contains detailed configurations for training, model parameters, and profiling.
REFERENCES
Philippe Albouy, Samuel A Mehr, Roxane S Hoyer, Jérémie Ginzburg, Yi Du, and Robert J Za- torre. Spectro-temporal acoustical markers differentiate speech from song across cultures, Nature Communications, 1511)5 2024.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 12449-12460, 2020.
Sarah Barrington, Romit Barua, Gautham Koorma, and Hany Farid. Single and multi-speaker cloned voice detection: from perceptual to learned features. In 2023 IEEE International Workshop on Information Forensicy and Security (WIFS), pp. 1-6. IEFE, 2023.
11

---

Published as a conference paper at ICLR 2025
Hervé Bredin and Antoine Laurent. End-to-an end speaker segmentation for overlap-aware resegmen- tation. arXiv preprint arXiv:2104.04045, 2021.
Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scal attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17302-17313, 2023.
Xuanjun Chen, Haibin Wu, Jyh-Shing Roger Jang, and Hung-yi Lee. Singing voice graph modeling for singfake detection. In INTERSPEECH 2024, August 2024. doi: 10448500/2Xiv.24000.1111
Saheb Chhabra, Kartik Thakral, Surbhi Mittal, Mayank Vatsa, and Richa Singh. Low quality deep- fake detection via unseen artifacts. IEEE Transactions on Artificial Intelligence, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.077 2023.
Victoria Derbyshire, Ellie Jacobs, and Tim Dodd. ai threat music. BBC News, 2023. URL sttps://www.bbc.com comp:ews/
Hozier would consider strike over
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
David C Epstein, Ishan Jain, Oliver Wang, and Richard Zhang. Online detection of ai-generated images. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 382-392, 2023.
FAIR. fvcore: A light-weight core library that provides the most common and essential functionality shared various computer vision frameworks developed at facebook ai research. https://
2023. Accessed: 2024-08-04.
geui art. Suno-api. 06-27.
art suno api 2024. Accessed: 2024-
Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 202/, pp. 571-575, 2021. doi: (0214577hterrpeech.2011 202teessspeecch..21699999
Anmol Gulati, James Qin, Chung-Cheni Chiu, Niki Parmar, Zhang. Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmente transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Carlos G. D. C. J. tion.
lyrics with language informa- https: www. kaggle, com/ httooo/ddaatttss///oooo cattts//e/roogd/e//
Genius song
genius song lyrics with witth-languape information 2023. 06-05.
Accessed: 2024-
Tejas Jayashankar, Jilong Wu, Leda Sari, David Kant, Vimal Manohar, and Qing He. Self-supervised representations for singing voice conversion. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans. Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 6367-6371. IEEE, 2022.
12

---

Published as a conference paper at ICLR 2025
Piotr Kawa, Marcin Plata, Michal Czuba, Piotr Szymański, and Piotr Syga. Improved deepfake detection using whisper features. arXiv preprint 2tti:23300002228,, 2023.
Nithin Rao Koluguri, Samuel Kriman, Georgy Zelenfroind. Somshubra Majumdar, Dima Rekesh, Vahid Noroozi, Jagadeesh Balam, and Boris Ginsburg. Investigating end-to-end ast architec- tures for long form audio transcription. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 13366-13370. IEFE, 2024.
Galina Lavrentyeva, Sergey Novoselov, Andzhukaev Tseren, Marina Volkova, Artem Gorlanov, and Alexandr Kozlov. Stc antispoofing systems for the asvspoof2019 challenge. arXiv preprint arXiv: (0tv:/9944.000 05576, 2019.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:5 16:51191-520000 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffisinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 11020-11028, 2022a.
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai, Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 34:23818 23830, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11 1196,, 2022b.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
Liv McMahon. Billie eilish and nicki minaj want stop to 'predatory' music ai. BBC News, 2024.
https:// /www.www.m bbc. com nwws/ wttps///tccoooooooo6611111
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked modeling duo: Towards universal audio pre-training framework. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24480-24489, 2023.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint wtwp:::1900007777000 2019.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-288 28518. PMLR, 2023.
Mare Riera. Udiowrapper. htttp::/ wttp:/uw...oooffoo////eeo// com/ http://///.oooeeeeeeoooooaaaaapa 2024. Ac- cessed: 2024-07-09.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference Acoustics. Speech and Sig- nal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yusun Shul and Jung- Woo Choi. Cst-former: Transformer with chamnel-specttt--eepporr attention for sound event localization and detection. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8686-8666 IEFF, 2024.
13

---

Published as a conference paper at ICLR 2025
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-end anti-spoofing with rawnet2. In ICASSP 2021-2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6369-6373. IERE, 2021.
Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee- ween Jung, Junichi Yamagishi, and Nicholas Evans. Automatic speaker verification spoofing and deepfake detection using wav2vee 2.0 and data augmentation. arXiv preprint arXiv:2202. /2233, 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference machine learning, pp. 6105-6114, 14. PMLR. 2019.
U.S. Code. 17 u.s.c. 107 limitations on exclusive rights: Fair use, 2023. https: https://uscode: /uscode. house..ov/ /wwow//iww..... wwt.ew.wwhtmllmmm wwtml http:///ww....aaaaa////////////.
Tao Wang, Ruibo Fu, Jiangyan Yi, Jianhua Tao, Zhengqi Wen, Chunyu Qiang, and Shiming Wang- Prosody and voice factorization for few-shot speaker adaptation in the challenge m2voc 2021. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing IEEE, 2021.
(ICASSP), pp.
Zhecan Wang, Junzhang Liu, Chia- Wei Tang, Hani Alomari, Anushka Sivakumar, Rui Sun, Wen- hao Li, Md Atabuzzaman, Hammad Ayyubi, Haoxuan You, et al. Journeybench: A challeng- ing one-stop vision-language understanding benchmark of generated images. arXiv preprint 22X1::2409.12933, 2024.
Ross Wightman. Pytorch image models. https://github. com rwightman. gyytorch-image-odde
2019. Accessed: 2024-08804.
Yuankun Xie, Jingjing Zhou, Xiaolin Lu, Zhenghao Jiang, Yuxin Yang. Haonan Cheng, and Long Ye. Fsd: An initial chinese dataset for fake song detection. In ICASSP 2024-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4605 4609. IEEE, 2024.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-effiiient tuning self-chat data. arXiv preprint arXiv:2304.01196 2023.
Amit Kumar Singh Yadav, Emily R Bartusiak, Kratika Bhagtani, and Edward J Delp. Synthetic speech attribution using self supervised audio spectrogram transformer. Electronic Imaging, 35: 1-11, 2023.
Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcending forgery speci- ficity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8984 8994, 2024.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via seff-instruction. Advances in Neural Information Processing Systems, 36, 2024.
Amir Zadeh, Tianjun Ma, Soujanya Poria, and Louis-Philippe Morency. Wildmix dataset and speptro-temporal transformer model for monoaural audio source separation. arXiv preprint 2019.
Yongyi Zang. Jiatong Shi, You Zhang, Ryuichi Yamamoto, Jionghao Han, Yuxun Tang, Shengyuan Xu, Wenxiao Zhao, Jing Guo, Tomoki Toda, et al. Ctrsvdd: A benchmark dataset and baseline analysis for controlled singing voice deepfake detection. arXiv preprint 2xXiv22406.02438, 2024a.
Yongyi Zang, You Zhang. Mojtaba Heydari, and Zhiyao Duan. Singfake: Singing voice deepfake detection. In ICASSP 2024-2024 IEÉE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12156-12160. IEFE, 2024h.
14

---

Published as a conference paper at ICLR 2025
Hongyi Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411 2017.
You Zhang, Fei Jiang, and Zhiyao Duan. One-class learning towards synthetic voice spoofing de- tection. IEEE Signal Processing Letters, 28:937-941, 2021a.
Zhenyu Zhang, Xiaowei Yi, and Xianfeng Zhao. Fake speech detection using residual network with transformer encoder. In Proceedings of the 2021 ACM workshop on information hiding and multimedia security, pp. 22, 2021b.
Yi Zhao, Wen-Chin Huang, Xiaohai Tian, Junichi Yamagishi, Rohan Kumar Das, Tomi Kinnunen, Zhenhua Ling, and Tomoki Toda. Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. arXiv preprint arXiv:2008. /2527, 2020.
15

---

Published as a conference paper at ICLR 2025
Appendix
Contents
Appendix A. Dataset Characteristics
17
Long Form Correlations
17
Song Duration
17
Genre Distribution
17
Embedding Space
17
Song Style
19
Appendix B. Dataset Quality Evaluation
19
Appendix C. Implemenaation Details
19
Dataset Cost Analysis
19
SingFake Training
19
Augmentation
20
Benchmarking
20
Model
20
Appendix D. Benchmark
Generalization Test
Human-Al AI Benchmark
Appendix E. Result Analysis
Appendix F. Comparison with Related Works
23
Appendix G. Limitations and Future Work
23
Appendix H. Prompt Engineering
24
Selection of LLMs
24
Half Fake
24
Mostly Fake
24
Full Fake
24
16

---

Published as a conference paper at ICLR 2025
A DATASET CHARACTERISTICS
Long Form Correlations: An example of the long context correlation can observed on Fig. 5 where a real song would stay consistent throughout the repetition of the phrases, refrains, rythms, etc. In comparison, synthetic generation methods can have difficulty generating consistent refrains, due to the long form context that might be required model this information. Therefore, synthetic song detection model that can capture the nuances of long context correlations present songs would be able better differentiate real and synthetic songs utilizing this property.
Frequency (Hz) Amplitude
Mali F Near, far, wherever you are believe that the heart does go on
4096
2048
1024
512
40.0
90.0
Time (sec)
140.0
Figure 5: Long Context Correlation. The red highlighted regions in the spectrogram and raw au- dio indicate repetition of the same verse ("Near, far, wherever you are, I believe that the heart does go on "),",, rhythms, and music. Such consistency is characteristic of real songs and can challenging for synthetic generation methods to replicate.
The distribution of the song duration across training and test sets is shown on Fig. 6 From the figure, it can be observed that aside from a few minor outliers, the distribution of fake and real song duration across both training and test sets similar.
Song Duration:
Fig. reveals that, despite slight imbalances, both real and fake songs are represented across all genres. This observation also indicates that the detectors are not overfitting to specific song genres. In other words, the models are not relying on genre information to determine whether song is real fake. Because if the detectors cation as a shortcut for detecting fake songs, their performance would have deteriorated. However, the strong performance of all methods demonstrates that our models are effectively learning features beyond genre classification. This finding provides confidence that the detection capability is robust and not dependent on genre-specific cues.
Genre Distribution:
attempting to perform genre classifi-
Embedding Space: The t-SNE plots presented in Fig. 8 illustrate the data distributions within the embedding space, generated using an EfficientNetBO (Tan & Le 2019) encoder with Mel Spectro- gram audio inputs.
Subfigure (a) shows a significant overlap between the training (Green) and testing (Orange) datasets. However, distinct clusters along the edges, predominantly composed of training data, suggest that the training set more diverse than the test set. This diversity is observed despite the test set containing previously unseen algorithms and speaker scenarios.
Subfigure (b) indicates a substantial overlap between real (Red) and fake (Blue) songs, with a few regions exclusively occupied by fake data or outliers. This highlights the inher- ent challenge of synthetic song detection within this dataset.
Subfigure (c) shows that the embedding space is primarily occupied by Suno v3.5 (Green) and Udio 133 (Orange), which are seen algorithms. Despite Suno v2 (Blue), Suno v3 (Red), and Udio 32 (Violet) being unseen algorithms, their embeddings fall within regions covered by seen algorithms, demonstrating underlying similarities. noteworthy obser- vation is the considerable overlap between Suno v3 (Red) and Suno v3.5 (Green), which likely contributes the similar detection performance in these algorithms, as indicated by T
Subfigure (d) depicts the distribution of fake types. Although Half Fake (Orange) songs form discernible clusters, Full Fake (Red) songs exhibit scattered distribution, in-
17

---

Published as a conference paper at ICLR 2025
(a) Train
10a
real Take
Frequency
10¹
10¹
10a
100
30
60
120
240
Length (sec)
(b) Test
real fake
10a
Frequency
102
10a
10
30
60
120
240
Length (sec)
Figure 6: Duration distribution between Training and Testing sets.
Genre Distribution of Real vs Fake Songs
25k
25000
Songs
Real Songs
20000
15000
13k
10000
7k
5000
41
3k
0.7 6.7ka.5t 0.4k0.8k
1k
0.3g 0.8k
0,2g0.4k
0.1ko.ok 1.100.2k 0.100 Ik
lock
folk
pop
hip hop
metal
electronic
dessical
country
rebb
blues
gospal
Jazz
regnae
soul
Genre
Figure 7: Genre Distribution between Real and Fake Songs.
dicating a lack of common features. Conversely, Mostly Fake (Green) songs are spread broadly across the embedding space, reflecting their diverse characteristics.
No Daste buttaa of Data S2it
Dr Detribation Saal Dete
AD Dutriblkknn of Tube Toes
Figure 8: Data distributions in the embedding space. This plot illustrates the clustering of different data categories such target, algorithms, fake types, and data split.
18

---

Published as a conference paper at ICLR 2025
Song Style: In order to imitate the song style observed in real songs, we had LLMs perform stylistic analysis of the real song lyrics (prompt shown in Table [2]. The wide variety of styles extracted from real songs are utilized to generate the synthetic songs. It is important to have the distribution of synthetic song style as close to the real song styles in the dataset, since it would be better indicator of real life cases. A wide diversity of synthetic song style in training data is also required so that the models trained on this dataset do not rely on the distinct characteristics of any particular song style as feature to detect synthetic songs, Additionally, is also important to evaluate performance of the trained model on testing set across wide variety of song style as well, so that the generalization capability of the detection model across multiple styles evaluated. A word cloud representation of the song styles that have been extracted from real songs and utilized to generate songs is show on Fig. The figure indicates that there are diverse song styles across both training and testing sets.
(a) Train Song-Style Distribution Rimospheric accussscaccuutt cic
(b) Test Song-Style Distribution
me Fallow
dance
li
melancholic
S acoustic
synth
ma e folk mencanho Indie
anthemic
upbeat T
eecttri
onthenic
bass
plano
alterrative
syynhh
rectrome
maLe
Dtrospher
guil
gut
Rellow
e
dince
pop
tta
upbeat t
RIP
blues
ind
folk
è eecctrnt C -pp sneggetic
dark
um
Temale
piano
energer
amiiti getic
Fast
fast! moderata
bass
Figure 9: Data distributions of song-style between train and test split in the word cloud.
B DATASET QUALITY EVALUATION
Given the large scale of the datasets, exhaustive manual verification not feasible. However, we have rigorously evaluated our pipeline and found that any potential errors have minimal impact on the dataset's overall quality. For instance, to assess PyAnnotate's performance in detecting non- vocal segments, we manually analyzed 50 random samples from our dataset. The results showed that the tool is highly accurate in distinguishing between vocals and non-vocals. While some errors may exist, their impact is negligible. Specifically, human evaluators reported instances of songs containing only background music without vocals (i.e., no false negatives). Furthermore, the total number of samples classified non-vocal (false positives) by Aynootaee is only 3% of the dataset. Even if we conservatively assume that all of these are false positives-though this is not the case based on our evaluation -heooon--thhhaao overall impact on the dataset remains minimal. Additionally, assess the accuracy of the extracted song styles, we manually analyzed 50 randomly selected real songs and compared their Al-generated styles (genre, instruments, mood, and vocal type) with their actual styles, as retrieved from the internet. Our analysis showed that the generated styles were highly accurate for genre, instruments, and mood. However, occasional inaccuracies were observed in the vocal type classification, which will not impact the primary goal of our work-fake song detection.
C IMPLEMENTATION DETAILS
Dataset Cost Analysis: Generating the SONICS dataset costs S1,055, allocated across GPT-4o (S405), Gemini 1.5 Flash (S80). Suno ($390), and Udio ($180).
SingFake Training: To demonstrate the limitations of SVDD (Singing Voice Deepfake Detection) models in detecting end-to-end fake songs, we first trained the models on the SingFake dataset, which consists of 597 training songs and 480 testing/validation songs. We then evaluated the per- formance of these models on our SONICS-test dataset, with the results summarized in Table is important to note that due to some inactive links in the SingFake dataset, we were able to download only 1077 songs, as opposed to the 1305 songs originally reported in the paper. For training, we maintained the same model configuration the 128 x 128 (5 sec) Mel spectrograms to 224 x 224 to leverage the pretrained ImageNet weights, compensating for the smaller size of the SingFake dataset.
in other experiments, with one exception: resized
19

---

Published as a conference paper at ICLR 2025
Augmentation: To enhance the robustness of our models, apply MixUp Zhang 2017) augmen- tation with a = 2.5 and a 50% probability. Additionally, we utilize SpecAugment (Park et al. 2019). applying two time masks of size and one frequency mask of size 8, each with 50% probability.
Benchmark: For the efficiency benchmark, we utilized a single PI00 GPU for all experiments. To measure the inference time of each model, performed warm-up runs followed by 100 test runs with a batch size of to record the processing time. The results were averaged and then inverted to determine the inference speed of each model. To compute GPU memory consumption, used batch size of 14 across all models, measuring the peak memory usage during a single forward pass. For calculating FLOPs, we employed the fvcore (FAIR) 2023) library.
Model: For the proposed SpecTTTra model, we use the mi_small_pathhh. configuration from timm (Wightman 2019) library with an embedding dimension of 384, 6 attention heads, 12 Trans- former layers, and an MLP ratio of 2.67.
To clarify the core components of the SpecTTTra model, we provide PyTorch-like code for the Tokenizer (st.okenizer) below. Note that while the code is presented in a func- tional format for clarity, the actual implementation follows an object-oriented approach.
import torch import torch. nn as nn
def st_tokenizer (x, t_clip, f_clip, embed_din)
B, F, x.size()
Temporal tokens
tokens = tokenizer (x, F, embed dim, t_iiip, T // t_clip)
Spectral tokens tokenns - tokenizer(
x.transpose 2)
T,
embed dim, f_clip, 7 // c_ipp
rpempro-l tokens st st_tckens torch.cat ((t_tokens, f_tokens), dim-1) return st tokens
def tokenizer (x, input_dim, token_din, clip_size, n_clips):
slicing and Tokenization convid nn. Convid in ih_channell=-inptt_din oor_chaaeelsttoen_.diii dim, kernel_size-cciipsirre stride=clip_size bias-False,
x convld(x) gelu transpose 2)
Positional Ambedding
_os_embeds nn. Parameter (torch. randn (1, n_clips, (oken_dim) 0.02)
pos embeds
Layer Normalization Layerrooom (toke__dd eps-1e-6) (x)
return
20

---

Published as a conference paper at ICLR 2025
In this code, the tokenizer function represents the spectral temporal tokenizer used embed spectral or temporal clips (patches) into tokens. Here, f f-iip and t.clip denote the sizes of the spectral and temporal clips, respectively, while embed dim signifies the feature dimension of each token. The dimensions 7 and F correspond to the temporal and spectral dimensions the input spectrogram.
D BENCHMARK
D.1 GENERALIZATION TEST
Although generalization is not the primary focus of our work, it remains crucial for evaluating the broader applicability of dataset and methods. To assess generalization, trained on the SONICS dataset using out-of-distributie SkyMusic and SeedMusie. Due the lack of an API for SkyMusic and a platform for SeedMusic, we manually collected 394 and 36 songs, respectively, from their publicly available demo websites. While the sample size is limited, these tests offer preliminary insights into generalization perfor- mance. The results, summarized in Table 8 reveal the following key observations:
evaluated models (OOD) songs from two external generators:
1. Performance Decline: A consistent decline in performance is evident across all models, as shown in Table g This trend aligns with findings in media forensies Ojha et al. (2023);
Epstein et al. 2023); Yan et al. (2024), underscoring the challenges of generalization. For
instance, Ceny which performed exceptionally well on the SONICS dataset, exhib- ited significant performance drops, ranking the lowest across both sources and durations.
2. SpecTTTra's Robustness:
SpecTTTra-7 achieved the highest Fl scores (80% on Seed- Music and 60% on SkyMusic), outperforming all other models. However, larger variants, such as SpecTTTra-a, showed susceptibility overfitting, particularly with long-duration songs.
3. Transformer Advantage: Models incorporating transformers (e.g., SpecTTTra, VIT. Effi- cientViT) demonstrated greater resilience OOD songs compared to CNN-based models, reaffirming their suitability for addressing generalization challenges.
The observed performance declines on OOD songs consistent with prior findings Ojha et al. (2023); [psttin et al.] (2023); Yan et al. (2024), emphasizing the necessity of advanced general- ization strategies. Despite the absence such strategies, some models achieved promising OOD results, highlighting the potential of the SONICS dataset for advancing song forensics. Notably, for validation purposes, we also evaluated all models with randomly initialized weights. None of these models exceeded an Fl score of 0.51, further reinforcing the significance of learned representations.
D.2 HUMAN-AL BENCHMARK
To assess human performance in synthetic song detection, we developed a Huggingface space called "Song Arena" illustrated in Fig. TO In this space, users evaluate whether randomly selected song from a subset of the proposed dataset (comprising 520 samples) is synthetic or not. The space also features leaderboard (shown in Fig. [1] that records human detection performance for songs generated by different algorithms and generation methods. The evaluation metrics used to assess the detectability of synthetic songs include the F1 score, Sensitivity (True Positive Rate), and Specificity (True Negative Rate). These metrics provide a comprehensive measure of the difficulty humans face detecting synthetic songs generated by various algorithms.
E RESULT ANALYSIS
Our analysis of the SpecTTTra-o model's results reveals audible and perceptible artifacts in both successful and failed cases of real and fake songs. True Negative cases, we find distinct patterns in correctly classified real songs. These include characteristics such unpredictability, dynamic variation, and unexpected changes that often absent in fake songs. Examples include non non-standard pitch variations, intricate rhythmic complexity, and expressive techniques like melismatic phrasing, sudden tempo changes, or improvisational segments, all of which showcase the nuanced artistry of human performance. Conversely, in True Positive cases, detect specific audible artifacts in
21

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of SpecTTTra and Existing Methods on Unseen Generators (SkyMusic and SeedMusic). AASIST and its variants face out-of-memory enror while training for 120s audio sam- ples resulting in missing performance scores (-).
Len. (sec)
Model
Seed Music
Sky Music
AASIST ResNet Spec.
0.65
0.56
0.52
0.48
ResNet + LFCC
0.54
0.51
Wav2Vec2 + AASIST
0.69
0.53
ConvNeXt
0.28
0.21
5
ViT
0.68
0.57
EfficientViT
0.67
0.54
SpecTTTrae
0.32
0.33
SpecITTra-$
0.32
0.41
SpecTTTra--o
0.47
0.57
AASIST
ResNet Spec.
0.53
0.35
ResNet LFCC
0.56
0.40
Wav2 + AASIST
ConvNeXt
0.36
0.19
120
ViT
0.64
0.53
EfficientViT
0.71
0.55
SpecITTra-y
0.80
0.60
SpecTTTra--8
0.53
0.28
SpecTTTra-a
0.56
0.25
Sing Gar mrotion
Leadwebord
Song Arena
desect sorgl gane need
Click Now Boand for anangg!
100d
A
Z Hew Ø
Figure 11: Song Arena Leaderboard on Hugging- face.
Figure 10: Song Arena on Huggingface.
correctly classified fake songs. These artifacts include mechanical or robotic vocal qualities, unclear vocal articulation, predictable rhythmic structures, and limited pitch variability. Such fake songs also lack the emotive expressiveness and complexity we consistently find in real music, making them notably distinct. In False Negative cases, we observe that fake songs detected as such lack the typical artifacts seen in true positive fake songs. Instead, these cases often incorporate features that mimic the unpredictability and nuanced variation of real songs. For instance, some include spoken interludes or conversational segments before the singing starts, creating deceptive resemblance to genuine music. In False Positive cases, we find that real songs misclassified as fake share characteristics with Al-generated music. These include unclear vocals, less rhythmic complexity, and noticeable absence of the unexpected changes that typically distinguishes human- created performances. Finally, encounter instances where artifacts. This suggests the presence subtle, imperceptible, or inaudible artifacts (Barrington et al. 2023) akin invisible artifacts (Chhabra al., 2023) in synthetic images.
unable detect any audible
22

---

Published as a conference paper at ICLR 2025
F COMPARISON WITH RELATED WORKS
To comprchensively evaluate our proposed method, we compared our SpecTTTra models with var- ious related works in fake song detection and provided findings in Table 9 Specifically, benchmarked our method against all the approaches mentioned in Fake Zang et al. (2024b).
including AASIST Jung al. (2022), Wav2Vec (Baevski al. (2000), and ResNet al. (20T6) variants. Our analysis demonstrates that the SpecTTTra models outperform the compared methods terms of effectiveness while maintaining efficiency. For example, although the AASIST models deliver promising results for short songs. they are computationally intensive, reflected by their high time and memory consumption. Additionally, AASIST models fail to process long songs of 120 seconds, resulting in out-of-memory (OOM) errors due to its extreme computational complex- ity. On the other hand, while ResNet variants exhibit high efficiency, they lack the capacity to detect fake songs effectively as SpecTTTra, particularly with long-duration songs. In contrast, the SpecTTTra models strike a balance between efficiency and effectiveness. As shown in the Ta- ble SPecTTTra-a achieves the highest FI score of 0.97 for long-duration songs while maintaining manageable computational requirements. These results establish the superiority our method in both detecting fake songs and handling longer audio inputs compared to other existing methods.
Table 9: Performance and efficiency comparison with existing works in fake song detection. AA- SIST and its variants face out-of-memory error while training for 120s audio samples resulting in missing performance scores (-). Their FLOPs and activation count were also unable to be deter- mined by the fvcore library.
Len. (sec)
Speed (A/S) f
FLOPs
Mem.
Act.
# Param. (M)
Model
F1
(G)
(GB) +
(M) 1
AASIST ResNet + Spec.
55
12
6
96
0.3
0.91
354
0.6
0.1
0.8
11
0.86
ResNet LFCC
331
0.6
0.1
0.8
11
0.88
Wav2Vec2 + AASIST
42
*
1.3
*
0.90
SpecTTTra--
154
0.7
0.1
2
17
0.76
SpecTTTra-83
152
1.1
0.2
2
17
0.78
SpecITTTra-a
148
2.9
0.5
6
17
0.80
AASIST
295
OOM
2393
0.3
ResNet + Spec.
146
17.2
2.3
24
11
0.90
ResNet LFCC
144
17.2
2.3
25
11
0.91
120
Wav2Vec2 + AASIST
*
00M
*
SpeeTTTra->
97
10.1
1.6
20
24
0.88
SpecTTTra-3
80
14.0
2.3
29
21
0.92
SpecITTra-a
47
23.7
3,9
50
19
0.97
G LIMITATIONS AND FUTURE WORK
The proposed SONICS dataset contains real songs dynamically queried from YouTube using their titles and artist names, which sometimes result in incorrect audio retrieval. A manual analysis of 600 random samples suggests that this issue affects approximately 0.5% of the dataset. To address this minor noise, we utilized label smoothing (Szegedy et al. 2016) during training. Another limi- tation is that the fake songs generated by the Udio platform cannot include lyrics from real songs, limiting the comprehensive evaluation in the Half Fake songs category only those generated by the Suno platform. Our current benchmarks are based solcly on Mel Spectrogram inputs: hence, we aim incorporate raw audio and explore other feature extraction methods, such as LFCC and MFCC, to enhance the robustness of our evaluations. Due to resource constraints, we trained and compared only smaller versions of all models. the future, we plan to compare larger versions of all models. Furthermore, we trained all models from scratch to ensure fair comparison, because the proposed SpecTT model is designed specifically for audio, while other models like ConvNeXt and ViT only have pretrained weights available for images (ImageNet). In the future, plan to pretrain all models on an large-scale audio dataset from scratch before training them on our proposed dataset for maximizing the performance.
23

---

Published as a conference paper at ICLR 2025
H PROMPT ENGINEERING
Selection of LLMs: For lyrics and song-style generation, as well as lyrics feature extraction, we evaluated several proprietary LLMs, including GPT-4o, Claude 3, and Gemini 1.5, along with open- source models like LLama 3, Gemma 2, and Mistral Large. Among these models, GPT-4o demon- strated superior performance, particularly in maintaining rhythm and coherence and accurately fol- lowing the content of prompts. Based on these qualities, we selected GPT-4o for both lyrics and song-style generation tasks.
For song-style analysis, only the proprietary models Gemini 1.5 Pro and Gemini 1.5 Flash, as well as the open-source model Qwen-Audio (Chu et al. 2023). are equipped to process audio inputs. Our evaluations indicated that both Gemini 1.5 Pro and Gemini 1.5 Flash models deliver similarly accurate performance. In contrast, Qwen-Audio frequently struggled to follow instructions correctly. Considering that the Gemini 1.5 Flash model is ten times more cost-effective than the Gemini 1.5 Pro, we selected the Gemini 1.5 Flash model for song-style analysis.
These songs are generated using lyrics and song style extracted from real songs. To extract the song style, the prompt template mentioned Table [2] is used. This template extracts song style information such vocal type, musical instruments, mood, etc.
Half Fake:
These songs are generated similarly to Half Fake songs, except that the lyrics are AI- generated. The lyrics created using an LLM (Large Language Model) with the prompt template shown in Table [T] where lyrics features are used as input. These lyrics features are also extracted from real song lyrics using the prompt template shown in Table TO The use of lyries features instead of direct lyrics prevents the L.LM from copying the original content, encouraging it to generate lyrics with a similar distribution rather than duplicating them.
Mostly Fake:
Full Fake: These songs are generated using Al-generated lyrics and song style, created using the prompt template provided in Table 13 In this process, the genre, topic, and mood were selected randomly from the lists provided below:
List of Genres:
alternative, baroque, blues, bollywood, c popo celtic, christian rock, classical, country, crunk, dance, dancehall, disco, doom metal, electronic, folk, funk, fusion, gospel, gothic, grime, grunge, hard rock, heavy metal, hip hop, indie rock, j-pop, jazz, k k-pop, lo-fi, lounge, metal, metal core, new age, opera, orchestral, pop, pop rock, progressive metal, progressive rock, punk, råb, rap, reggae, salsa, smooth jazz, soul, sufi, world music.
adventurous, ambivalent, amused, angry, anxious, apathetic, bittersweet, blissful, calm, carefree, cautious, chaotic, confident, confused, curious, desperate, determined, disenchanted, distracted, drained, dreamy, empathetic, enchanted, energetic, exhilarated, focused, forgiving, frustrated, gloomy, grateful, hateful, humble, inspired, intspecetivve, jealous, joyful, liberated, lonely, loving, melancholic, mischievous, motivated, mournful, mysterious, nostalgic, optimistic, passionate, pensive, pessimistic, playful, powerless, proud, rebellious, regretful, reluctant, restless, romantic, sarcastic, satisfied, shocked, skeptical, submissive, sympathetic, tense, timid, trapped, uninspired, vengeful, vulnerable, whimsical, yearning, zealous.
List of Moods:
List of Broad Topics: alien invasion, ancient civilizations, augmented reality, betrayal, childhood memories, climate change, cyber crime, dimensional portals, dreams and aspirations, dystopian future, empowerment, endangered species, extraterreatrial contact, family, fashion, financial struggles, first kiss, forgiveness, friendship, futuristic cities, generation gap, grief and loss, heartbreak, social media anxiety, interstellar travel,
24

---

Published as a conference paper at ICLR 2025
loneliness in a crowd, long-distance relationships, love, love at first sight, lunar colonization, nanotechnology, nature's beauty, nostalgia, ocean exploration, overcoming adversity, pandemic experiences, parallel universes, political revolution, politics, quantum physics, reincarnation, never give up, road trip adventures, robotic emotions, save the planet, sibling rivalry, social influencers, social justice, space exploration, space tourism, survival in the wild, technology addiction, time capsules, time paradoxes, time travel, unconditional love, work-life balance.
List of Specific Topics: elon musk vs yann lecun (AI), yann lecun vs geoffrey hinton (AI), convolution vs transformer (AI), gan vs diffusion models (AI), tensorflow vs pytorch (AI), pytorch vs jax (AT), twilight zone (TV), star trek (TV), game of thrones (TV), breaking bad (TV), stranger things (TV), big bang theory (TV), friends (TV), simpsons (TV), house of cards (TV), how met your mother (TV), the office US (TV), sherlock (TV), avatar the last airbender (TV), pokemon (anime), dragon ball 2 (anime), naruto (anime), one piece (anime), attack on titan (anime), my hero academia (anime), death note (anine), jujustu kaisen (anime), fullmetal alchemist (anime), demon slayer (anime), neanderthals (anthropology), pyramids of giza (archaeology), machu picchu (archaeology), stonehenge (archaeology), egyptian mummies (archaeology), giza sphinx (archaeology), notre dame cathedral (erchitecture), london bridge (architecture), big ben (architecture), eiffel tower (architecture), versailles (architecture), arc de triomphe (architecture), mona lisa (art), van gogh's s starry night (art), sistine chapel (art), solar eclipses (astronomy), black holes (astronomy), big bang theory (astronomy), supernovas (astronony), dark matter (astronomy), (astrconmmy), elon musk vs bezos (business), taylor swift (celebrity), tom cruise (celebrity), brad pitt (celebrity), angelina jolie (celebrity), lecnardo dicaprio (celebrity), michael jackson (celebrity),
andromeda galaxy
jennifer aniston (celebrity),
meryl streep (celebrity),
robert de niro (celebrity), matt damon vs jimmy kimmel (celebrity),
jimmy kimmel vs marvel vs de (comics),
jimmy fallon (celebrity), vs superman (comics), ther vs hulk (comics), iron man captain america (comics), batman va joker (comics), john constantine (comics), marvel de comics (comics), (cosmology), russian ballet (culture), bollywood (culture), hollywood (culture), the great hack (documentary), coachella (event), rose bowl parade (event), (event), kentucky derby (event), tokyo olympics (event), beijing olympics (event), columbus (exploration),
batman justice league vs avengers (comics),
universe (comics),
big bang theory
alphago the movie (documentary),
walt disney (entertainment),
wimbledon rio olympics (event),
nike vs adidas (fashion), world war III (fiction), pepsi vs coke (food), pele vs maradona (football), brazil vs argentina (football), monopoly (game), chess (game), go (game), dungeons and dragons (game), minecraft (game), fortnite (game), call of duty (game), mario (game), pac-man (game), sonic the hedgehog (game), fifa (game), cyberpunk 2077 (game), grand theft auto (game), the last of us (game), assassin's creed (game), resident evil (game), halo (game),
marco polo (expioration).
messi vs ronaldo (football),
25

---

Published as a conference paper at ICLR 2025
the witcher 3 (qame), god of war (game). alphago vs lee sedol (games), deep blue vs kasparov (ganes), playstation vs xbox (gaming), mount everest (geography), grand canyon (geography), dead sea (geography), ring of fire (geography), venetian canals (geography), volcanic eruptions (geology), ice age (geology), mariana trench (geology), great depression (history), moon landing (history), titanic (history), viking explorers (history), stone age (history). aztec empire (history), mayan calendar (history), vikings (history), cold war (history), space race (history), moon landing (history), fall of the berlin wall (history), industrial revolution (history), edison's light bulb (invention), the wright brothers (invention), tesla vs edison (inventors), taj mahal (landmark), great wall of china (landmark), central park (landmark), mount fuji (landmark), fifa world cup (sports), statue of liberty (landmark), golden gate bridge (landmark), niagara falls (landmark), times square (landmark), king arthur (legend), robin hood (legend), loch ness monster (legend), yeti (legend), camelot (legend), shakespearean sonnets (Literature), harry potter (literature), the hobbit (literature),
rocky mountains (geography),
empire state building (landmark), hollywood sign (landmark),
lord of the rings (literature), edgar allan poe (literature), charles dickens (literature), dracula (literature), frankenstein (literature), the great gatsby (literature), pride and prejudice (literature), alice in wonderland (literature), moby dick (literature), war and peace (literature), little women (literature), treasure island (literature), oliver twist narnia (literature),
romeo and juliet (literatuze),
(literature), peter pan (literature), aesop's s fables (literature), sherlock holmes (literature), the da vinci code (Literature), CNN vs Fox News (media), godzilla (movie), inception (movie), matrix (movie), interstellar (novie), john wick (movie), jason bourne (movie), james bond (novie), spiderman (movie), dark knight (movie), avengers (novie), star wars (movie), indiana jones (movie), back to the future (movie), jurassic park (movie), avatar (movie), wizard of oz (movie), star wars vs star trek (movies), star wars (movies), fight club (movies), the dark knight (movies), the green mile (movies), gladiator (movies), the departed (movies), the lion king (movies), aladdin (movies), beauty and the beast (movies), little mermaid (movies), frozen (movies), tangled (movies), mulan (movies), sleeping beauty (movies), cinderella (novies), snow white (movies), the social network (movies), the beatles (music). bob dylan (music), bermuda triangle (mystery), crop circles (mystery), atlantis (myth), hercules (myth), perseus (myth), pandora's box (myth), trojan war (myth), achilles (myth), hades (myth), olympus (myth), zeus (myth), hera (myth), apollo (myth), artemis (myth), athena (myth), poseidon (myth), phoenix (mythical creature), medusa (mythical creature), greek mythology (mythology), yellowstone (national park), yosemite (national park), grand tetons (national park), amazon rainforest (nature), sahara desert , barrier reef (nature), victoria falls (nature), niagara falls (nature), northern lights (nature), southern lights (nature), bioluminescent bays (nature), blue holes (nature), dinosaurs (paleontology), democrats vs republicans (politics), scientist vs engineer (profession), python vs c++ (progranming), java vs python (programming), javaseript vs
indiana jones (movies),
titanic (movies),
26

---

Published as a conference paper at ICLR 2025
Instructions:
Analyze provided song lyries extract following elements:
1. Subject Matter: Write what the song is about by providing summary of the story, narrative, central topic, events discussed in the song lyrics.
Theme: Identify the main theme message (emotion, life experience, social commentary, or philosophical concept).
Target Audience: Define the intended audience (age group, cultural background, or specific interests).
4. Narrative: The story and point of view (e.g., first person second person).
Character Analysis: Main characters (e.g., Protagonist, Antagonist) and their traits.
Song Structure: Outline the structure (number of verses, choruses, bridges, intros, outros) and note any unique elements or deviations.
7. Mood:
Describe the overall mood (upheat, melancholic, introspective, etc.). Identify any cultural, social, time, place, contextual references.
Reference:
How to respond:
You should provide your answer below after the "Answer" section. You are not allowed to use any text formatting (bold, italic, etc.) and narrative ('Here is the answer', 'Below is the response", etc.) your answer. Only answer using the following format: subject_matter.
theme: "...."
audgene.
* narrative:
character analysis:
song. structure
mood: "...."
reference: "...."
Lyrics: [lyrics]
Answer:
Table 10: Prompt template for extracting lyrics features (e.g subject matter, theme, mood) from real songs. Here, (lyries) indicates placeholder for input lyrics.
java (programming), paper book vs e-book (reading), physics (science), chemistry (science), biology (science), astronomy (science), ma mathematics (science), albert einstein (scientist), isaac newton (scientist), charles darwin (scientist), marie summer winter (season), twitter facebook (social media), olympics (sports), worllds) cup (sports), super bowl (sports), (sports), nba finals (sports), nfl playoffs (sports), elon musk vs mark zuckerberg (tech), ai human intelligence (tech), avidia va amd (tech), intel vs amd (tech), nvidia vs intel (tech), google vs openai (tech), iphone vs android (tech), mac va pe (tech), ai revolution (tech), rubik's cube (toy), barbie (toy), lego (toys), europe vs asia (travel), usa vs canada (travel), australia vs new zealand (travel), usa vs europe (travel), switzerland vs sweden (travel), beach vs mountains (vacation), city countryside (vacation).
curie (scientist),
tour de france (sports),
wimbledon
27

---

Published as a conference paper at ICLR 2025
Task:
You talented songwriter tasked with creating song based the following lyries features. The song include the features described below in the "Lyrics Features" section. The song should not be long; rather, should be modium-length
Lyrics Features: feature)
Instructions:
You should write the song with metatags following the song structure from "Lyrics Features". In some very rare cases, you can also scarcely include ad libs, or non-lexical vocables.
You add metatags your lyries top of section in [square brackets] that will create certain styles. Some examples of metatags are [verse], [Chorus], [Hridge] [Solo], [o), [Pre-Chorus], [Bridge), [Hook [Cpening), [Intro], Instrumentl, [Build], [Drep], [Breakdown). [Refrain], [Spoken.), [Interlude], [Prelude]. (Samplel. etc. Adding blank newline between sec- tions yields the best results.
In some very rare cases, you can also scarcely use Ad libs (vocal embellishments) to your prompts in hesrentheses), only when necessary, Examples include (yoah), (alright), (come on), (whoa), etc. Ad libs tend to work best the end of line but can also work mid-line. Unlike metatags, ad libs sung/verbalized.
* In some very rare cases, you can also scarcely use non-lexical vocables, only when necessary. Examples include la 1a 1a, na na na, sha na na.
Example: [verse]
I've been tryna call
I've been on own for long enough
Maybe can show me how to love, maybe
[Chorus]
I said, och, I'n blinded by the lights No, I can't sleep until I feel your touch said, och, I'n drowning in the night Oh, when I'm like this, you' ze the one I trust Hey, hey, hey
Write the lyries helow after the "Lyrics:" section. You are not allowed to add any narrative text before after your response such as "Here's your answer" or "Below is the response".
Lyrics:
Table 11:
Prompt template for generating song lyries from lyrics features. Here, flyrtus indicates placeholder for input lyrics features.
28

---

Published as a conference paper at ICLR 2025
Task:
Given a song. you need to conduct comprehensive stylistic analysis and extract all relevant information about the song's s style. This includes identifying characteristies such as instruments used (e.g., guitar, drums, piano, violin, synthesizers, bass, orchestral, solo, acoustic, trumpet, saxophone, flute, cello, harmonica, banjo, accordion, etc.), vocals types (male female), genres (e-g., rock, pop, pop rock, indie rock, hard rock, metal, heavy metal, r&c, electronic, soul, jazz, country, reggae, classical, hip hop, blues, folk, punk, funk, disco, alternative, grunge, etc.), tempo (slow, moderate, fast), mood (e.g., melancholic, upbeat, aggressive. melodic, sad, happy. excited, nostalgic, mellow, serene, joyful, dark, gothic, etc.), and any other stylistic elements (e.g.. dance, party, cinematic, drcamy, energetic, relaxing, anthemic, atmospheric, groovy, etc.) that contribute to the overall vibe, environment, or atmosphere of the song.
Your response should be structured as follows:
Answer:
<start>
stylel, style2, style3,
styleN
<end>
For example:
Answer:
<start>
male vocals, electronic, guitar, piano, energetic, pop rock, violin, upbeat, dance, synth, sad, soul, trumpet, reggae <end>
Please note that the list of style elements should be comprchensive and cover all relevant aspects of the song's style. Ensure that your response follows strictly to the specified formatting, includ- ing the use of angle brackets, commas, and space separating each element written in lowercase. There must not be any narrative or text in the answer (e.g.. 'Here's your answer' or 'Below is response"), ), only the listed style elements.
{song]
Table 12: Prompt template for extracting song style (e.g. vocal type, instruments) from audio song.
Here, {song) indicates placeholder for input audio song.
29

---

Published as a conference paper at ICLR 2025
Task:
You are a talented songwriter, music director, and composer. Your task is to compose {genre] genre song about (topic} with a (mood) mood. Provide the lyrics and style of the song after the "Answer" section. Follow the step-by-step instructions provided in the "Instructions" section and respond using the format given in the "How to Answer" section.
Instructions:
1. Before you write the song, you need to plan what you will write about the song, then synthesize the features of the song lyrics mentioned below.
* Subject Matter: Write what the song about by providing summary of the story, narrative, central topic, or events discussed in song lyries.
Theme: Identify the main theme or message (emotion, life experience, social commentary, or philo- sophical concept).
Target Audience: Define the intended audience (age group, cultural background, specific inter- ests).
Narrative: The story and point view (e.g.. first person or second person).
Character Analysis: Main characters (e.g., Protagonist, Antagonist) and their traits.
Song Structure: Outline the structure (number of verses, choruses, bridges, intros, and outros) and note any unique elements deviations.
* Mood:
Describe the overall mood (upbeat, melancholic, introspective, etc.).
Reference: Identify any cultural, social, time, place, or contextual referenees.
2. Then, you need to write song lyries that include all the features described in the "Lyrics Features" section. The song should not be long; rather, it should be medium-length You should also write the lyrics with metatags following the song structure from "Lyries Features." In some very cases, you can also scarcely include libs, or non-lexical vocables. Here are the detailed instructions:
You can add metatags to your lyrics on top of a section in [square brackets] that will ere- ate certain styles. Some examples of metatags are [Verse], [Chorus], Bridge), [Solo], [Outrol. [Fre-Chorus]. Briddee.. Hook], 10peningl, [Introl. [Instrument). [Buila], Dropp:: [Breakcown]. [Reirain], [Spoken]. [Interlude]. [Prelude]. [Samp] , etc. Adding a blank newline between sections yields the hest results.
In some very rare cases, you can also scarcely use Ad libs (vocal emhellishments)) to your prompts in (parentheser heses), only when necessary. Examples include (yeah), (alright (come on), (whoa), etc. Ad libs tend to work best at the end of line but can also work mid-line. Unlike metatags, libs are sung/verbalized.
In some very rare cases, you can also scarcely use non-lexical vocables, only when necessary. Ex- amples include la la 1a, na na na, sha na na.
3. Finally, you need to compose the song by synthesizing all relevant and detailed information about the song's style. This includes identifying characteristics such as instruments used (e.g., guitar, drums, piano, violin, synthesizers, bass, orchestral, solo, acoustic, trumpet, saxophone, flute, cello, harmonica, banjo, accordion, etc.), vocals types (female or male), genres (e.g.. rock, pop. pop rock, indie rock, hard rock, metal, heavy metal, r&b, electronic, soul, jazz, country, reggae, elassical, hip hop, blues, folk, punk, funk, disco, alternative, grunge, etc.), tempo (slow, moderate, fast), mood (c.g., melancholie, upbeat, aggressive, melodic, sad, happy, excited, nostalgic, mellow, serene, joyful. dark, gothic, etc.). and any other stylistic elements (e.g.. dance, party, cinematic, dreamy, energetic, relaxing, anthemic, atmospheric, groovy, etc.) that contribute to the overall vibe, environment, or atmosphere of the song. The name of the style must be written in lowercase and separated by commas. Finally, choose title for the song that best suits lyrics and style.
Prompt continued on next page...
30

---

Published as a conference paper at ICLR 2025
..continued from previous page
How to answer:
You need to provide your answer after the "Answer" section below, while strictly following the format below. Also in your answer, you are not allowed to use any text formatting (bold face, italic, etc.) narrative (Here's your answer, Answer is below, etc.). Just provide your answer using the below format:
Lyrics Feature:
<featurc>
subject_matter:
theme: "....
target augset-addiece::
narrative: "...."
character.analysis::
song-structure:
mood: ".... reference:
</ /feature>>
Song Lyries:
<lyrics>
[Verse 1]
In small town by the sea, where the waves kiss the shore, Lives dreamer with heart, always yearning for mere. With netebook in his hand, and vision in his eyes, He paints the world in colors, beneath the endless skies.
[Pre-Chorus|
Through the struggles and the trials, he keeps his head up high, With a song within his soul, knows he' touch the sky.
</lyrics>
Song Style:
<style>
male vocals, electronic, guitar, piano, energetic, pop rock, violin, upbeat, dance, synth, sad, soul, trumpet, reggae, </style>
Song Title:
<title>
Chasing the </title>
Answer:
Prompt template for generating song lyrics and style from genre, topic and mood. also generates lyrics feature and song title by-product. Here, (genre), (topic), {mood) indicates placeholders for input genre, topic and mood of the song.
Table 13:
31


Reviews:
[{"review_id": "qMl6lOeL21", "reviewer": "Reviewer_hEps", "summary": "This paper presented a dataset for detecting AI generated music from well-known AI generated music services, Suno and Udio. The authors basically configured two group of music which are fake and real songs. For real songs, they retrieved 48,090 real songs from YouTube and for fake songs, they generated music from Suno and Udio. Finally, they trained a model for detecting whether the song is real or fake. To do this, SpecTTTra model is proposed. In this model, the authors proposed to use spectral and temporal patches (full-size in one dimension) to construct spectral and temporal tokens for their transformer-based classification model. The results showed that the proposed model is highly effective in detection. Also, the authors compared the model performance with Human performance.", "questions": "I think the title of the paper and the term real and fake are a bit too generalized terms. \nFor example, the proposed dataset is about detection on Suno and Udio generated contents. (not counterfeit songs, maybe synthetic is okay) \nI think the authors can carefully suggest a better title.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-written and easy-to-read, and the details of the proposed method, experimental design, the dataset, and the experimental results are well addressed. I think the following researchers will be easily follow and use the proposed dataset, and the models as the authors described them well.", "weaknesses": "It is good to see some research paper in the direction of detecting AI generated content in the domain of music. However, I think the paper lacks of analysis part. The scores that's shown in result tables seem to prove the effectiveness of the proposed method, however, without such in-depth analysis, the proposed dataset and the proposed method might not be useful in the future if the algorithms are updated. Therefore, I think having some analysis section would be beneficial for further future researches. (I think leaving some comments that is related to more high-level explanation on why the proposed model is detecting the AI generated content well, e.g. vocal characteristics, temporal/timbral characteristics, etc) At least, some additional case study on Appendix would be nice. For example, If some real song is detected as a fake, then try to give some analysis on why it's been detected as fake or vice versa.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KKV1JW0lg8", "reviewer": "Reviewer_m36c", "summary": "This paper introduces a dataset of real and \"fake\" music recordings, where the fake recordings have been generated using commercial music generation systems. An attempt has been made to match some characteristics between the real and fake recordings. In addition, a new transformer architecture is proposed that is aimed at capturing longer-term temporal context in music audio classification. Finally, results on the fake vs. real music classification task are presented.", "questions": "Major questions/comments:\n- Line 81f: \"the fake songs in our dataset are free from copyright issues, as they are generated through paid subscriptions that provide a license to use and share the content legally\" --- Given that the generation models used are very likely to have been trained on copyrighted data, how can you ensure that the fake songs in your dataset are free from copyright issues? What if generations are identical or similar to existing real music recordings - in particular, when real lyrics and styles are used for generation (cf. \"half fake\" subset).\n- Several steps in the dataset preparation pipeline will probably produce very noisy or incorrect results, but their performance is never evaluated. This is concerning, given that the dataset is the main contribution of this paper. In particular:\n  - How well does PyAnnotate work for vocal detection on this data (line 189)?\n  - How accurate are the extracted song-styles? This, in particular, I would expect to work rather poorly. Vocal type classification from music audio, for example, is a hard problem.\n  - How accurate are the transcribed lyrics?\n  - In the cases were generated instead of transcribed lyrics were used: How often do these regurgitate the lyrics from real songs?\n- In the proposed SpecTTTra model, I am unconvinced that the proposed spectral clips can provide very helpful information for long songs. The resulting spectral tokens would contain something similar to an average spectrum, which would not be very informative for real vs. fake classification. Indeed, increasing the number $f$ of spectral clips does not improve results in Table 4. However, this was not independently evaluated from the choice of $t$.\n- Following up on the previous comment, it seems like results for the SpecTTTra-$\\alpha$ model are best, even though this utilizes the proposed architecture the least (low $f$ and $t$). \n\nMinor questions/comments:\n- In Table 1, consider adding \"Total hours\".\n- Line 142f: \"Meanwhile, long audio classification remains an uncharted area in audio research.\" --- This statement seems too strong and should be removed. A wide range of music audio classification tasks involve long audio classification, for example: genre classification, global key detection, global tempo estimation, etc.\n- Line 211f: How was the list of topics for the full fake generation obtained? The list given in the appendix seems very biased. Is \"matt damon vs jimmy kimmel\" really a representative example of a topic for a real song?\n- Line 288: Why were significantly more MF songs than HF or FF songs included?\n- Line 471: What was the musical experience level of the human annotators? Did they have access to spectral analysis tools (which could be helpful for the fake vs. real classification task)?\n\nAdditional comments:\n- Line 12: \"these inventions democratize music creation\" --- This is a highly questionable, opinionated slogan that has its place in marketing but not in science. It should be removed.\n- Line 68 (and others): Please do not use the bibliography to refer to companies such as Suno or Udio. A company is not a scientific publication. Same, e.g., for wandb (line 392).\n- Table 4: Consider clarifying in the caption what the positive and negative classes are (i.e. what is real and what is fake). Furthermore, indicate which algorithms are seen and unseen during training.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction of a dataset for fake vs. real music classification is a welcome and very timely contribution. Different systems by two major commercial providers are considered and the dataset is separated in terms of different levels of fake (fully, mostly, half), which provides interesting insights - notably the half-fake songs with real lyrics seem to be easier to recognize. In the provided transformer architecture, I believe it makes a lot of sense to have one token per time step rather than several tokens for different frequency patches.", "weaknesses": "Unfortunately, the steps of the dataset preparation pipeline do not seem to be evaluated rigorously, which limits the usefulness of the resulting dataset. I am also not very convinced by the usefulness of the proposed spectral tokens.\nOverall, because of these and some other issues (see \"Questions\"), I recommend to reject this paper, although this is a borderline recommendation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vwWOk24EDk", "reviewer": "Reviewer_LAdp", "summary": "The paper introduces SONICS (Synthetic Or Not - Identifying Counterfeit Songs), featuring the largest synthetic song dataset to date with over 97,000 songs (4,751 hours), including 49,074 AI-generated songs from Suno and Udio platforms. SONICS includes end-to-end generated content with both music and lyrics, and features longer duration songs (averaging 176 seconds). To effectively detect these synthetic songs, the authors propose SpecTTTra (Spectro-Temporal Tokens Transformer), a novel architecture specifically designed for capturing long-range patterns in music. This new model demonstrates superior efficiency compared to traditional approaches, running 38% faster than ViT while using 26% less memory and achieving better detection performance, particularly with longer audio samples. This work provides essential tools for detecting AI-generated music as these technologies become increasingly sophisticated and widespread.", "questions": "- Could the authors provide a representative set of audio samples from different categories (Full Fake, Half Fake, Mostly Fake) and platforms (Suno, Udio)?\n- What quality control measures were used to ensure the generated songs are suitable for the dataset?\n- Could the authors provide ablation studies demonstrating how the separate temporal and spectral tokenization contributes to capturing long-range patterns?\n- What was the rationale behind the specific clip sizes chosen for the three variants (α, β, γ)? And how sensitive is the model performance to these hyperparameters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a clear and reproducible workflow for synthetic song generation, with detailed documentation of the prompt engineering process, generation parameters, and data processing steps. \n- The generated song dataset can be an important addition to SVS and SVDD community.\n- The Spectro-Temporal tokenizer uses a simple but seemingly effective way to capture the long-range information.", "weaknesses": "- The paper lacks sample demonstrations of the generated synthetic songs for evaluation. Without access to audio examples, it is difficult for reviewers and readers to assess the quality and diversity of the generated content, as well as validate the claims about different types of synthetic artifacts (e.g., \"Karaoke effect\"). The authors should provide a representative set of samples across different generation categories (Full Fake, Half Fake, Mostly Fake) and generation platforms (Suno, Udio).\n- The naming \"SpecTTTra\" (Spectro-Temporal Tokens Transformer) is potentially misleading as the architecture only employs a transformer encoder component rather than a full transformer architecture. Furthermore, while the model claims to better capture long-range dependencies, the paper lacks ablation studies to demonstrate this capability. The authors should provide specific experiments showing how the separate temporal and spectral tokenization contributes to capturing long-range patterns in music.\n- The performance improvements over baseline models, particularly ConvNeXt, are modest given similar computational costs. The F1 score gain of 1% with 20% speed improvement doesn't strongly justify the introduction of a new architecture. The authors should either provide additional experiments demonstrating clear advantages of SpecTTTra (e.g., ablation studies, analysis of captured patterns), or consider separating the dataset contribution from the model architecture into two papers.\n- The quotation mark is used incorrectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v1BC4BU1So", "reviewer": "Reviewer_znTb", "summary": "This paper introduces SONICS, a large-scale dataset designed for end-to-end synthetic song detection, addressing the limitations of existing datasets such as lack of musical diversity, short song durations, and absence of fully AI-generated music. They also propose a novel model, SpecTTTra, which excels in capturing long-range temporal dependencies within songs, outperforming current models in terms of speed and memory efficiency. The paper establishes both human and AI benchmarks to advance research in distinguishing AI-generated music, thereby protecting artistic integrity.", "questions": "1. The dataset's limited scope to Suno and Udio-generated songs may introduce bias, overlooking other AI music tools like SeedMusic and SkyMusic.\n2. The model's high accuracy might reflect genre classification rather than distinguishing real from fake songs due to the focused style of Suno and Udio.\n3. The study lacks evidence to confirm the model's ability to identify fake songs independently of audio quality differences.\n4. The paper's experimental tables fail to highlight the best-performing metrics, hindering clear comparison.\n5. An Ablation Study is missing to assess the model's reliance on Spectral or Temporal information alone.\n6. The paper does not compare the model's performance with other methods from the Synthetic Song Detection literature.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper presents three data acquisition pipelines for fake songs, significantly expanding the dataset for synthetic song detection and providing valuable reference for peers looking to augment this dataset. The paper also introduces a new model for synthetic song detection that achieves better detection results while reducing training costs, offering a new reference method for the field.", "weaknesses": "1. The data collection for fake songs is not comprehensive, leading to potential biases. Only songs generated by Suno and Udio were collected, while other AI music generation tools such as SeedMusic and SkyMusic exist in the market.\n2. Music generated by AI tools from the same company tends to have a consistent style. The authors need to demonstrate how they can distinguish between real and fake songs, rather than just identifying the music styles of Suno and Udio. Suno and Udio's song styles are concentrated in certain genres that, while present in real-world music, are less common. For instance, if the model achieves an 80% accuracy rate, it might mean that 20% of the ground truth song collection consists of pop and rock styles generated by Suno and Udio, while the remaining 80% are styles rarely generated by these tools. Thus, the model might be performing genre classification rather than accurately distinguishing between real and fake songs.\n   I suggest that to reduce this interference and truly identify AI and non-AI differences, rather than stylistic differences, the authors could introduce songs generated by other AI tools, different from the training data, as a test set. For example, if the model is trained to distinguish between real and fake songs using data from Suno and Udio, it could be tested with songs generated by SkyMusic, SeedMusic, and other models. If the model can still identify songs from SkyMusic and SeedMusic as fake, it would demonstrate the model's effectiveness.\n\n3. The authors need to prove that they are identifying real songs versus fake songs, not just differences in audio quality. The dataset includes real songs and AI-generated songs. Real songs are sourced from YouTube, where the sampling rate is generally higher and the audio quality is better. In contrast, AI-generated songs mentioned in the paper have a sampling rate of 16kHz/32kHz, which is of poorer quality, and even if upscaled to match YouTube's quality, there would be significant loss in the high-frequency spectrum.\n   The paper does not explore or explain this aspect, nor does it mention whether the audio quality was normalized before training and testing. It is unclear whether the model has truly learned to distinguish AI-generated songs or if it has been judging audio quality.\n\n   Whether it's distinguishing between 16kHz resolution songs and high-resolution songs or differentiating between up-sampled 16kHz songs and original high-resolution songs, it's clear that these are audio quality tasks, not real song vs fake song recognition tasks. Only by reducing all to the same low-quality audio can the bias of audio quality on the model be avoided.\n\n4. The experimental tables in the paper do not indicate which metrics are the best, making it difficult to intuitively compare the effectiveness from the table.\n5. There is a lack of Ablation Study to show the model's performance using only Spectral or Temporal information.\n6. There is no experimental comparison with other models and methods used in Synthetic Song Detection papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)