XPrompt: Complex-Prompted AI-Generated Text Detection
Authors: Yuanzhen Hao, Desheng Wu


Reviewer paper summaries: This paper evaluates the robustness of AI-Generated Text (AIGT) detectors trained on simple prompts to realistic scenarios where they should detect AIGTs from complex prompts. To this end, the authors construct a novel test benchmark that consists of generations from three LLMs and three complex prompting methods, which are detailed persona prompts, example prompts, and mixed prompts. Detectors simply fine-tuned on AIGTs from simple prompts did not generalize well on this benchmark.

The authors address this issue with a novel training framework named XPrompt. It utilizes POS tags to train detectors on text structure and frequently used words for each category explicitly. Detectors trained with XPrompt show improved performances against AIGTs from complex prompts without losing their effectiveness on generations from simple prompts.
Summary of strengths: 1. XPrompt is effective in improving the reliabilities of detectors and is easily applicable as the additional training processes only require a POS tagger.

2. The XPrompt framework is clearly stated and each component is described in detail. The figures in the paper also effectively illustrate the contents of the paper.
Summary of weaknesses: 1. LLMs are sensitive to small changes in prompts, and different models are suited for different prompts. The paper should reveal detailed information about the prompt choices such as the template for each prompting method, and the phrasing of example personas and in-context examples.

2. Many important configurations are missing regarding the AI generations. AIGT detectors are known to be affected by the generation hyperparameters, i.e. temperature, top-p, top-k, etc., and the length of input text. Such information should be provided in the paper.

3. Previous works regarding the impact of prompt choices on AIGT detection are not mentioned in Related Work. For example, [1, 2] investigates the robustness of existing detectors against prompt variations. Input perturbation attacks [3,4] find adversarial in-context examples that effectively deceive detectors.

4. Previous works already leveraged linguistic style features to detect AIGTs [5, 6]. The authors should compare the features utilized in XPrompt with features from such works.

5. Fig 3. shows that the similarity between AIGT and human text structures is lower overall, but this result changes if we separate mixed prompts from other prompts. The description of Fig 3. is unclear in the paper, so I assume that Fig 3. represents the number of common text structures among 20 frequent ones. Then, the mixed prompt has less common structures to other AIGTs than human writings, contradictory to the analysis in the paper. The authors need other evidence to support that text structure features help AIGT detection.

[1] Koike, R., Kaneko, M., & Okazaki, N. (2023). How you prompt matters! even task-oriented constraints in instructions affect llm-generated text detection. arXiv preprint arXiv:2311.08369.

[2] Taguchi, K., Gu, Y., & Sakurai, K. (2024). The Impact of Prompts on Zero-Shot Detection of AI-Generated Text. arXiv preprint arXiv:2403.20127.

[3] Ning Lu, Shengcai Liu, Rui He, Yew-Soon Ong, Qi Wang, & Ke Tang (2024). Large Language Models can be Guided to Evade AI-generated Text Detection*. Transactions on Machine Learning Research*.

[4] Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K. W., & Hsieh, C. J. (2024). Red teaming language model detectors with language models. Transactions of the Association for Computational Linguistics, 12, 174-189.

[5] Aich, A., Bhattacharya, S., & Parde, N. (2022, October). Demystifying neural fake news via linguistic feature-based interpretation. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 6586-6599).

[6] Shah, A., Ranka, P., Dedhia, U., Prasad, S., Muni, S., & Bhowmick, K. (2023). Detecting and Unmasking AI-Generated Texts through Explainable Artificial Intelligence using Stylistic Features. International Journal of Advanced Computer Science and Applications, 14(10).
Comments, suggestions and typos: 1. The paper focuses on the robustness of different prompts, but the additional elements in the proposed framework are not specialized in generalization to prompt variation. Hence, XPrompt can improve detection robustness in other types of domain shifts. Measuring the robustness of XPrompt detectors against them would benefit this paper. 

2. Word clouds are effective visualizations, but they do not convey detailed statistical information. Readers can comprehend the results better if they are accompanied with complementary figures or tables presenting statistical values.

3. There are multiple typos, including:
    
    L43: LLMs generation 
    
    L69: to find
    
    No space before “(” in multiple lines.
Overall assessment scores: 2