anmol-unitmole/grammar-correction-flan-t5-base

Grammar Error Correction with FLAN-T5 Base

Fine-tuned FLAN-T5-base encoder-decoder model for English grammatical error correction (GEC), trained and selected through controlled experiments on BEA-2019 W&I+LOCNESS, JFLEG-based evaluation data, controlled synthetic error pairs, and leakage-safe identity examples.

The selected checkpoint is the Base Conservative 30% Identity model. It was chosen because it retained essentially the same ERRANT F0.5 as the strongest Base baseline while improving precision and materially reducing unnecessary edits on clean text.

Selected deployment model: 03c-base-conservative-30pct
Base architecture: google/flan-t5-base
Parameters: 247,577,856
Primary task: English grammar error correction / text-to-text generation

Live Resources

• Model Repository: https://huggingface.co/anmol-unitmole/grammar-correction-flan-t5-base
• Static Demo: https://huggingface.co/spaces/anmol-unitmole/05-grammar-error-correction-t5-encoder-decoder (deployment in progress)
• Base Model: https://huggingface.co/google/flan-t5-base

---

Evaluation Dashboard

The selected model was evaluated using a held-out grammar-correction set plus an independent clean-text over-correction benchmark.

Metric	Selected Model	What it measures
ERRANT F0.5	0.4527	Precision-weighted grammatical error correction quality
Precision	0.5053	Fraction of proposed edits that are supported by the reference
Recall	0.3195	Fraction of reference edits recovered
GLEU	0.7835	Reference-based grammatical correction quality
Semantic similarity	0.9384	Meaning preservation between source and corrected text
Sentence accuracy	0.0983	Exact whole-sentence reference match
Clean over-correction rate	25.73%	Clean sentences changed unnecessarily
Number preservation	99.72%	Preservation of numbers and quantities
Named-entity preservation	93.57%	Preservation of detected named entities
Technical-term preservation	99.63%	Preservation of technical terminology
Average batched latency	75.85 ms/item	Batched evaluation throughput per item
P95 batched latency	152.94 ms/item	95th-percentile batched evaluation latency

Evaluation Population

Evaluation component	Count
Combined evaluation records	2,487
Correction-bearing sentences	1,088
Clean sentences in metric partition	1,399
Dedicated leakage-safe clean benchmark	988

The clean metric partition includes the dedicated 988-example clean benchmark plus identity/correct examples already present in the main held-out test data.

---

Model Selection

Multiple controlled fine-tuning experiments were run rather than assuming that a larger model or more aggressive balancing would automatically perform better.

Fine-Tuned Experiment Comparison

Model	ERRANT F0.5	Precision	Recall	GLEU	Semantic Similarity	Clean Over-Correction
Small — 10% identity	0.3961	0.4714	0.2416	0.7510	0.9620	29.16%
Small — 20% identity	0.3939	0.4736	0.2354	0.7521	0.9606	26.38%
Small — 30% identity	0.3926	0.4851	0.2228	0.7516	0.9645	23.30%
Base — 10% identity	0.4530	0.4989	0.3311	0.7830	0.9359	29.38%
Base — 30% identity (selected)	0.4527	0.5053	0.3195	0.7835	0.9384	25.73%

Why Base 30% Was Selected

Compared with Base 10%, the selected Base 30% model:

• retained essentially identical ERRANT F0.5: 0.4530 → 0.4527;
• improved precision: 0.4989 → 0.5053;
• improved semantic similarity: 0.9359 → 0.9384;
• reduced clean over-correction: 29.38% → 25.73%;
• improved number preservation: 99.63% → 99.72%;
• slightly improved named-entity preservation: 93.20% → 93.57%.

The cost was a modest recall reduction from 0.3311 to 0.3195.

This made Base 30% the stronger quality-versus-conservatism trade-off for the final portfolio model.

---

Conservative Training Findings

Increasing the proportion of identity/correct examples consistently reduced unnecessary edits in the Small-model ablation:

Identity Ratio	ERRANT F0.5	Clean Over-Correction
10%	0.3961	29.16%
20%	0.3939	26.38%
30%	0.3926	23.30%

The Base experiment showed the same direction:

Base Variant	ERRANT F0.5	Precision	Clean Over-Correction
Base 10%	0.4530	0.4989	29.38%
Base 30%	0.4527	0.5053	25.73%

This demonstrated that identity pressure improved conservatism without materially sacrificing the primary F0.5 score.

---

Project Quality Gates

These are project-defined aspirational gates, not claims of external production certification.

Gate	Target	Selected Model	Status
Semantic similarity	≥ 0.90	0.9384	✅ Pass
Clean over-correction	≤ 8%	25.73%	⚠️ Not met
Number preservation	≥ 99.5%	99.72%	✅ Pass
Named-entity preservation	≥ 98%	93.57%	⚠️ Not met
Technical-term preservation	≥ 98%	99.63%	✅ Pass

The model is therefore presented as a measured portfolio/research model, not as a production-certified grammar checker.

---

Dataset and Training Setup

Final Conservative Training Dataset

Split / Property	Value
Final Base-30 training rows	38,825
Validation rows	3,259
Main held-out test rows	1,499
Dedicated clean benchmark	988
Identity/correct rows in Base-30 train	11,348
Identity ratio	~30%
Primary training error categories	61

The project uses:

• BEA-2019 W&I+LOCNESS grammar-error data;
• JFLEG multi-reference evaluation data;
• controlled synthetic grammar-error pairs;
• leakage-safe clean/identity examples;
• source-level split conflict resolution;
• dedicated clean-text over-correction evaluation.

The final audit reported zero source overlap between the training data and validation, test, and clean-test source sets.

---

Training Configuration

The selected model was fine-tuned locally on an NVIDIA RTX 5090.

Setting	Value
Base model	google/flan-t5-base
Epoch budget	4
Selected checkpoint	checkpoint-3000
Learning rate	1e-4
Per-device train batch size	8
Gradient accumulation	4
Effective batch size	32
Gradient checkpointing	Enabled
BF16	Enabled
TF32	Enabled
Max input length	384
Balanced sampling	Disabled for selected run
Training dataset	30% identity conservative set
Best validation loss	1.00873
Training duration	~44.8 minutes

---

Prompt Format

Training and inference use the same instruction-style prompt:

Task:
Correct the grammar, spelling, punctuation, and wording of the text while preserving the original meaning.

Rules:
- Preserve the original meaning.
- Do not add unsupported information.
- Do not remove important details.
- Keep named entities, numbers, technical terms, and product names unchanged unless clearly incorrect.
- Return only the corrected text.

Input:
{source_text}

Corrected Text:

Consistent prompting prevents a training/inference format mismatch.

---

Usage with Transformers

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

MODEL_ID = "anmol-unitmole/grammar-correction-flan-t5-base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

text = "She go to the office every day and complete her reports."

prompt = f"""Task:
Correct the grammar, spelling, punctuation, and wording of the text while preserving the original meaning.

Rules:
- Preserve the original meaning.
- Do not add unsupported information.
- Do not remove important details.
- Keep named entities, numbers, technical terms, and product names unchanged unless clearly incorrect.
- Return only the corrected text.

Input:
{text}

Corrected Text:"""

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=384,
).to(device)

with torch.inference_mode():
    generated = model.generate(
        **inputs,
        max_new_tokens=192,
        num_beams=4,
        early_stopping=True,
    )

correction = tokenizer.decode(generated[0], skip_special_tokens=True)
print(correction)

---

ONNX Optimization Findings

The selected PyTorch model was exported to ONNX and dynamically INT8-quantized as an engineering experiment.

Artifact Size

Runtime artifact	Total size
FP32 ONNX components	2.334 GB
INT8 ONNX components	1.075 GB
INT8 / FP32 size ratio	46.05%
Approx. aggregate size reduction	53.95%

100-Example Parity / Runtime Check

Runtime	Exact Output Match	Average Latency
PyTorch path	Reference	419.49 ms
INT8 ONNX path	50%	3343.95 ms

The parity test used a different single-example validation path from the batched evaluation latency shown earlier, so those latency figures should not be directly compared across tables.

Deployment decision: the INT8 ONNX experiment is retained as optimization evidence, but it is not claimed as the validated production/browser runtime because output parity and measured runtime were not strong enough.

---

What the Model Does Well

Measured strengths include:

• stronger correction quality than the Small FLAN-T5 experiments;
• high number preservation (99.72%);
• high technical-term preservation (99.63%);
• strong semantic similarity (93.84%);
• improved precision under conservative identity training;
• measurable reduction in unnecessary editing relative to Base 10%.

---

Known Limitations

• Clean-text over-correction remains substantial at 25.73%.
• Named-entity preservation is 93.57%, below the project's aspirational 98% gate.
• Exact sentence accuracy is intentionally strict and remains low.
• The model can miss valid corrections because conservative training trades some recall for precision.
• Multiple grammatical rewrites can be valid even when they differ from the benchmark reference.
• Long documents should be segmented rather than passed as one very long sequence.
• The INT8 ONNX experiment did not achieve sufficient parity/performance to be advertised as the validated live browser runtime.
• Human review is recommended for important professional text.

---

Intended Use

Suitable for:

• English grammar-correction experimentation;
• writing-assistance prototypes;
• sentence and paragraph correction;
• quality-report drafting support;
• customer-complaint narrative cleanup;
• root-cause and corrective-action writing assistance;
• professional communication demonstrations;
• encoder-decoder / FLAN-T5 portfolio work;
• GEC research and educational use.

---

Not Intended For

Do not use the model as the sole authority for:

• legal documents;
• medical documentation;
• financial or compliance-critical text;
• academic integrity decisions;
• safety-critical instructions;
• autonomous editing of confidential corporate text;
• any workflow where meaning or named entities must never change without human review.

---

Experimental Environment

Training and evaluation were executed locally using:

• GPU: NVIDIA GeForce RTX 5090
• VRAM: ~31.84 GB
• CUDA build: 13.0
• Python: 3.12.10
• PyTorch: 2.12.1 + CUDA 13.0
• Transformers: 4.57.6
• BF16: enabled
• TF32: enabled
• ERRANT: 3.0.2
• spaCy: 3.8.15

---

Reproducibility

The broader project contains reusable workflows for:

• BEA-2019 W&I+LOCNESS preparation;
• JFLEG preparation;
• controlled synthetic error generation;
• leakage auditing;
• token-length auditing;
• FLAN-T5-small/base fine-tuning;
• identity-ratio experiments;
• error-type balancing experiments;
• ERRANT evaluation;
• GLEU evaluation;
• semantic-similarity evaluation;
• preservation checks;
• clean over-correction benchmarking;
• latency benchmarking;
• ONNX export;
• INT8 quantization;
• ONNX parity validation;
• tracked edits;
• error categorization;
• batch correction;
• static portfolio deployment.

---

Responsible Use

Generated corrections can be incomplete, incorrect, overly aggressive, or meaning-changing.

Do not paste private, confidential, proprietary, customer, employee, or personally identifiable text into a public demonstration.

For important writing, review the original and corrected text before accepting changes.

---

License

This fine-tuned model is based on google/flan-t5-base, which is released under the Apache License 2.0. Users should also respect the licenses and usage terms of the datasets and other third-party resources used in the broader training/evaluation workflow.