Title: Doolittle: Benchmarks and Corpora for Academic Writing Formalization
Benchmark: Doolittle
Summary: The paper introduces Academic Writing Formalization (AWF), a paragraph‑level text‑refinement task that converts informal‑academic prose into formal‑academic prose, going beyond grammatical error correction to include word choice and structural improvements. To support the task, the authors release DOOLITTLE, a 68K‑paragraph corpus (55.6 K formal, 13.0 K informal) with expert rewrites for 930 test/dev paragraphs, and they benchmark nine systems, proposing metric‑oriented reinforcement learning (MORL) that lets smaller PLMs approach ChatGPT quality while still trailing human rewrites.
Contribution: - First large‑scale, paragraph‑level corpus targeting holistic academic‑style formalization.
- Crowdsourced formality ratings plus expert rewrites yield both non‑parallel and parallel data.
- Introduces MORL: PPO fine‑tuning where the reward is a weighted blend of automatic metrics (ACC‑aesw, PPL, SIM, BARTScore).
- Detailed evaluation with classical GEC, style‑transfer, ChatGPT, and MORL‑tuned BART‑Large / Galactica‑1.3B, plus GPT‑4 “LLM‑as‑judge” ratings.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: academic‑style formalization / text refinement
Phenomenon defined by authors: Yes
Phenomenon definition: In light of this, we propose the novel task of Academic Writing Formalization (AWF) that aims to generalize the scope of GEC for language refinement: given an informal-academic paragraph P, the objective of AWF is to refine the language of P to make it grammatically correct, concise, and fluent, while preserving its semantics.
Additionally, they clarify that AWF consists of three sub-objectives:
"(1) grammar correction, (2) word refinement, and (3) structure modification"
— to improve grammar, lexical precision, and sentence/paragraph conciseness respectively.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given an informal‑academic paragraph P, produce a semantically equivalent paragraph that is grammatically correct, uses precise vocabulary, and is stylistically concise and formal.
Task item definition: One row contains a source paragraph (informal or formal) and, in the dev/test splits, the corresponding expert rewrite; models must output a refined version of the source
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Paragraphs randomly sampled from the Semantic Scholar Open Research Corpus; AMT workers rated formality, and two native‑speaker experts rewrote 900+ informal paragraphs for gold references.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: Test: 415 informal-to-formal pairs (+415 formal controls)
Dataset size notes: Train: 68,600 non-parallel paragraphs; Validation: 465 parallel pairs
Dataset metadata available: Yes
Dataset metadata detail: Formality score, word & sentence counts, ACC/PPL/SIM stats per split.
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Distribution (perplexity, calibration, correlation), Semantic Similarity, BARTScore, Char-level edit distance
Metric definition detail: - The paper combines four automated metrics into a composite reward:
    1. Transfer Accuracy (ACC‑aesw) — soft classifier logits from a formality classifier fine-tuned on AESW
    2. Perplexity (PPL) — using a GPT‑2 model fine-tuned on formal academic text to assess fluency
    3. Semantic Similarity (SIM) — subword-level embedding similarity to original & reference
    4. BARTScore (BARTS) — generative likelihood from BART
- These metrics are not just used for evaluation, but also combined as a reward signal for reinforcement learning (MORL) via a manually weighted sum.
Metric aggregation: Simple Mean
Metric statistics: Simple mean, and for annotation agreement Cohen’s Kappa coefficient was used.
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Code is shared, dataset access needs to be requested via the form link given in the GitHub Repo
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors provide strong evidence for the validity of their benchmark through multiple evaluations. They report high inter-annotator agreement (Cohen’s Kappa = 0.657) on formality ratings, apply expert review to ensure the quality of formal rewrites, and show that these rewrites improve fluency, formality, and clarity without major semantic drift. Additionally, their ablation studies demonstrate that each evaluation metric meaningfully contributes to model performance, and GPT-4-based annotations confirm the benchmark’s ability to distinguish high-quality refinements, highlighting its construct validity and practical relevance.
Taxonomy: NLP / 
New taxonomy: Specific Application / Physical Sciences