Learning to Rewrite: Generalized LLM-Generated Text Detection
Authors: Wei Hao, Ran Li, Weiliang Zhao, Junfeng Yang, Chengzhi Mao


Reviewer paper summaries: This paper aims to detect AI-generated text by AI-rewriting, based on the premise that human-written and AI-generated text would cause a different amount of rewrites. They fine-tuned a LLM to make more rewrites to AI text than to human text. Experiments shows the superiority over baselines and zero-shot AI-rewriting approach. They also provide a new AI-rewrite dataset for training and testing.
Summary of strengths: - AI detection problem is important.
- The experiments show the superiority of this method.
- The paper is well written and easy to follow.
Summary of weaknesses: - Novelty: The premise of this paper, stating that human-written and AI-generated text would cause a different amount of rewrites, is proposed by previous work[1]. While the authors apply this idea to fine-tune their model, the resulting performance boost is predictable.
- Fairness of evaluation.
    - Different training set: For the proposed L2R, the training data is from the same dataset as testing (section 5.2). However, for training-based baseline GPT2 detector, the training dataset is from a different dataset (webtext)[2]. The training set difference may hurt the evaluation quality.
    - Considering the distance function as a part, the L2R method is functionally equivalent to training a binary classifier. However, the GPT-2 detector is RoBERTa-based[2], which is much smaller than a Llama3-8B. A baseline of a naive classification approach fine-tuning Llama3 should be considered.
- Some adversarial test should be considered, such as [3].

### REF

[1] Chengzhi Mao, Carl Vondrick, Hao Wang, Junfeng Yang: Raidar: geneRative AI Detection viA Rewriting. ICLR. 2024

[2] https://github.com/openai/gpt-2-output-dataset

[3] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, Mohit Iyyer: Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. NeurIPS 2023.
Comments, suggestions and typos: Refer to weakness
Overall assessment scores: 2
Metareview score: 2
Metareview: This paper proposes a method to classify whether a given text is human-written or machine-generated by analyzing the distance between the input text and the text rewritten by a large language model (LLM).
Summary of reasons to publish: 1. The paper addresses a critical and timely issue in LLMs.
2. The method of using the distance between the original text and the LLM-rewritten text is novel.
3. The paper is well-written, easy to follow, and the presentation is good.