Detecting Synthetic Lyrics with Few-Shot Inference
Authors: Yanis Labrak, Gabriel Meseguer-Brocal, Elena V. Epure


Reviewer paper summaries: This paper introduce a synthetic lyrics generation pipeline, and proposes several statistical and embedding based synthetic data detection methods for lyrics. The study emphasizes the effectiveness of these detectors in generalizing to new artists and models, and their robustness in identifying automatic paraphrasing. A comprehensive human evaluation component is included to ground the model performance. Overall, the research addresses the challenge of distinguishing human-written lyrics from machine-generated ones.
Summary of strengths: - The paper introduces a new dataset for synthetic lyrics.

- The paper examines a comprehensive set of statistical classification methods.

- The paper reports human evaluation in detail and offers insights into how the decision making in models and humans are different.

- The authors open-sourced their code and data.
Summary of weaknesses: 1. Line 027: it is important to clarify that the method is robust against “automatic post-generation paraphrasing” rather than just any type of post-hoc paraphrasing.

2. Line 292: I am not convinced by this modeling choice of only using 1 model, as the trained detector could pick up model attributes irrelevant to the “humanness” of the generation. It would be important to have a comparison of sampling from 1 model versus sampling from all models.

3. Line 410: I do not agree with this statement: “the higher the perplexity, the less likely it is that the lyrics are human-written.” There is no experimental evidence or citations backing up the claim. In fact, there are prior works showing the LLM-generated text are usually lower in perplexity. 

4. Line 439: using the same model for the embeddings as the semantic similarity embedding used in the generation stage (line 281) seems a bit circular, as the generation step already filters out sentences that are out of distribution from the human-written lyrics. 

5. Line 602: The fact that TinyLLaMa is the easiest to detect for humans and hardest to detect for models does not necessarily “reinforces the idea that humans and machines assess lyrics very differently”. If anything, these results suggests that the features selected in the methods section are somehow biased toward detecting generations of other models. For example, the use of Llama-2-7b to calculate log-likelihood possibly biases the method to better detect Llama-2-13b.

6. Section 5.4: the investigation of paraphrasing attacks is interesting but very limited in this paper. The authors only use one model: Mistral 7B, which is not shown to be particularly good at paraphrasing lyrics. A few other models could strengthen the results. A strong baseline of human post-hoc paraphrasing would also better contextualize the results.

Overall weaknesses:

- I am doubtful about the novelty of the work. The proposed generation and detection methods are not specific to lyrics. The paper lacks investigation for lyrics-specific features such as rhyme schemes, meter and rhythm, repetition, phonemic patterns, etc.

- The introduced dataset is poorly validated and contains limited genre, number of artists and size, and is not balanced in terms of gender. 

- There lacks a simple baseline of training a neural based classifier. It is hard to gauge the effectiveness of the methods, especially when the synthetic datasets are not rigorously evaluated. The Random baseline seems unnecessary and does not provide any information.
Comments, suggestions and typos: - Line 255: “various regular expressions developed along the process and based on the model’s output” - this is extremely vague. What exactly was done for this normalization step? How are the regular expressions extracted?

- Line 271: “human-written lyrics is represented in box plots” - please use standard and accurate language to describe the sampling process. For example, “remove outliers that are outside of n standard deviations of the human-generated lyrics.”

- Section 4.3 4.4: the model used for computing the perplexity and log-likelihood is not specified. 

- Line 205, 347, 348: use `` and '' for double quotation marks 

- Line 372: ungrammatical sentence.

- Line 393: unclear what this means

- Line 396: token-level negative log-likelihood? Why are you using the maximum token? What are the motivations and justification for this decision?

- Line 470-477: seems redundant, space can be better used to justify prior design decisions

- Table 2: please consider rearranging the rows to match the order in Section 4. Statistics Methods —> Statistical Methods

- Line 620: The confidence scores response from the participants is interesting, but seems a bit out-of-place for the main focus of the paper.
Overall assessment scores: 2.5