Measuring AI Detection Across Sources: Are Stochastic Parrots Red Herrings?
Authors: Creston Brooks, Samuel Eggert, Adele Goldberg, Denis Peskoff


Reviewer paper summaries: The paper evaluates three AI-generated text detection models -- GPTZero, Binoculars, and Ghostbuster -- using a curated dataset of press releases from United Nations country teams. The study identifies GPTZero as the most accurate tool and employs it to analyze the prevalence of AI-generated content in press releases before and after the release of ChatGPT, revealing a significant increase post-ChatGPT. Additionally, the paper examines AI-generated text in news articles and Reddit posts.
Summary of strengths: 1. Comprehensive comparison of three AI-generated text detection tools.
2. Creation of a novel dataset of UN press releases.
3. Exploration of the prevalence of AI-generated text across multiple platforms, including news and Reddit.
Summary of weaknesses: 1. Lack of depth in analysis -- concluding that AI-generated text increased post-ChatGPT release is predictable and not sufficiently examined.
2. Discrepancies between the results of Binoculars and GPTZero are not adequately analyzed or explained.
3. Limited news dataset raises questions about the reliability of the conclusion that no AI-generated text is present in news articles.

Overall, the paper explores an interesting and relevant direction in analyzing AI-generated content. However, it is still a work in progress and requires further depth and analysis before it is ready for publication.
Comments, suggestions and typos: N/A
Overall assessment scores: 2