PaperCheck AI-writing detector (model card)
The detector behind PaperCheck, a free tool that checks whole research papers, theses and project reports for AI-written text and for plagiarism against arXiv.
Weights are not published. This card documents how the model works and how it was tested. You can use it at https://paper-check.me (5 free checks a month, up to 20,000 words each), or from Claude, ChatGPT, Cursor and VS Code through MCP at https://paper-check.me/mcp.
Model
- Architecture: RoBERTa-base sequence classifier (human / AI), fine-tuned from a public AI-text detector.
- Training data: human academic and general writing from before 2022 (papers, Wikipedia, literature, open-access biomedical papers from many countries) and AI text from GPT, Claude, Gemini and open models, including student-style report sections. Train, validation and test are split by document.
- Input: each sentence is scored together with its neighbours (a 5-sentence window).
How a document is scored
- Running prose only: title pages, declarations, tables, lists and references are skipped.
- A sentence counts as AI when its score is above 0.995. The cutoff was chosen to keep false alarms on human writing low.
- The text is split into 250-word sections. A section counts as AI when 28% or more of its words are flagged.
- The headline number is the share of words in AI sections.
Evaluation (documents never used in training)
A document is "mostly AI" when the headline number is 50% or more.
| Test set | Documents | Called mostly AI |
|---|---|---|
| ELLIPSE learner essays (English learners) | 773 | 0 (0.0%) |
| Papers by authors in non-English-speaking countries (PMC) | 171 | 1 (0.6%) |
| Wikipedia, pre-2022 | 153 | 4 (2.6%) |
| Public-domain literature | 298 | 3 (1.0%) |
| Real-world Claude student-style reports | 30 | 27 caught (90%) |
| RAID AI text, no attack | 296 | 227 caught (76.7%) |
| RAID AI text, paraphrase attack | 296 | 199 caught (67.2%) |
| Humanizer-rewritten AI text | 111 | 40 caught (36.0%) |
The full per-document results for the public sets, with charts, are in the Kaggle notebook How an academic AI detector handles human, AI, attacked and humanized text.
Limitations
- No detector score is proof of how a text was written. Use it to find lines worth revising, not to judge a person.
- Paraphrasing and "humanizer" tools lower the catch rate, as they do for every detector.
- Older base models (for example GPT-2) are caught less often than modern chat models.
- English only. Very short texts are less reliable than full papers.
- PaperCheck is independent and not affiliated with Turnitin.
Links
- Website: https://paper-check.me
- Engineering write-up: https://paper-check.me/engineering
- Architecture: https://paper-check.me/architecture