MohammadKhosravi/roberta-large-cefr-classifier-JointLoss

RoBERTa Large - CEFR Classifier (Joint WCE+Ordinal Method)

This repository contains a publication-grade, fine-tuned transformer model optimized for mapping English student prose directly to the Common European Framework of Reference for Languages (CEFR) proficiency scales (A1–C2).

The model is built on top of the deep contextual architecture of FacebookAI/roberta-large and is fine-tuned on a massive, structural subset of the EFCAMDAT corpus comprising 104,125 rows. It implements a unique Joint Loss optimization framework designed to explicitly manage class imbalance and respect the continuous, ordinal sequence of language acquisition.

It achieves the following results on the evaluation set:

• Loss: 0.1880
• Strict Accuracy: 0.9833
• Adjacent Accuracy: 0.9943
• Macro F1: 0.9747
• Mae: 0.0240
• Qwk: 0.9863

Model description

Architectural Modification

The base model originates from FacebookAI/roberta-large (355M parameters). To repurpose the architecture for multi-class sequence classification, the pre-trained Masked Language Modeling (MLM) head (lm_head) was completely excised. It was replaced with a randomly initialized linear classification head mapped to 6 discrete output nodes corresponding to the target CEFR levels:

$$\text{Outputs}\in \{\text{A1},\text{A2},\text{B1},\text{B2},\text{C1},\text{C2}\}\to \{0,1,2,3,4,5\}\backslash text\{Outputs\}\; \backslash in\; \backslash \{\backslash text\{A1\},\; \backslash text\{A2\},\; \backslash text\{B1\},\; \backslash text\{B2\},\; \backslash text\{C1\},\; \backslash text\{C2\}\backslash \}\; \backslash rightarrow\; \backslash \{0,\; 1,\; 2,\; 3,\; 4,\; 5\backslash \}$$

The Joint Loss Framework

Standard sequence classifiers treat categorical labels as nominal variables, assuming the distance between any two mistakes is mathematically equal (e.g., misclassifying an A1 text as C2 carries the same penalty as misclassifying it as A2). Because CEFR proficiency scales operate on an ordinal ladder, this model utilizes a Joint Optimization Objective:

$${\mathcal{L}}_{\text{Total}}={\mathcal{L}}_{\text{WCE}}+\lambda {\mathcal{L}}_{\text{Ordinal}}\backslash mathcal\{L\}\_\{\backslash text\{Total\}\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{WCE\}\}\; +\; \backslash lambda\; \backslash mathcal\{L\}\_\{\backslash text\{Ordinal\}\}$$

Loss Function

The training objective combines Weighted Cross-Entropy with an Ordinal Distance Penalty to address both class imbalance and the ordinal nature of the classification task.

Weighted Cross-Entropy (L_WCE)

Corrects for severe data scarcity in advanced cohorts (C1/C2) by scaling the categorical cross-entropy loss using inverse class frequencies.

w_c = N_total / (C × N_c)

where:

• N_total is the total number of training samples.
• C is the number of classes.
• N_c is the number of samples belonging to class c.

Ordinal Distance Penalty (L_Ordinal)

Computes a continuous, differentiable expected class value from the softmax probability distribution over the ordered classes.

ŷ = Σ P(class_i) × i

The ordinal loss is then computed as the Mean Squared Error (MSE) between the expected prediction and the ground-truth label:

L_Ordinal = (1/B) Σ (ŷ_b − y_b)²

where:

• B is the batch size.
• ŷ_b is the expected class value for sample b.
• y_b is the corresponding ground-truth ordinal label.

For this training routine, the regularization weight was set to λ = 0.5, encouraging the model to resolve nearby decision boundaries sequentially while penalizing catastrophic multi-tier classification errors.

The final training objective is:

L = L_WCE + λ × L_Ordinal

Note: For a nominal baseline optimized exclusively through class weighting without distance constraints, please refer to the sister architecture: RoBERTa Large - CEFR Classifier (WCE Method).

Intended uses & limitations

Recommended Applications

• Automated Essay Scoring (AES): Instantaneous evaluation of continuous multi-paragraph student text blocks.
• Educational Analytics: Longitudinal progress tracking of language learners across digital learning platforms.
• Curriculum Alignment: Diagnostic screening of raw reading or writing materials to match specific target student profiles.

Technical Limitations

• Domain Specificity: The model was fine-tuned heavily on non-native English learner compositions. Performance may vary on native speech, fiction prose, or specialized academic/legal discourse.
• Context Length Constraints: Standard tokenization caps the input length at 512 subword tokens. Texts exceeding this threshold will undergo truncation, potentially missing macro-syntactic cues embedded at the tail end of longer essays.

Training and evaluation data

The final dataset consists of 104,125 rows with a heavily stratified 90/10 training and internal validation split:

• Level A1: 25,000 samples | 24.01% allocation
• Level A2: 25,000 samples | 24.01% allocation
• Level B1: 25,000 samples | 24.01% allocation
• Level B2: 23,529 samples | 22.60% allocation
• Level C1: 4,614 samples | 4.43% allocation
• Level C2: 982 samples | 0.94% allocation

---

Training procedure

Training hyperparameters

• Learning Rate: 2e-05
• Per-Device Train Batch Size: 128 (Optimized via full tensor parallel execution on NVIDIA A100 architecture)
• Per-Device Eval Batch Size: 128
• Total Seed Configuration: 42
• Optimizer: Fused AdamW (betas=(0.9,0.999), epsilon=1e-08)
• Total Training Epochs: 4
• Precision Mode: Native bf16 (Brain Floating Point 16)

Framework versions

• Transformers 5.12.0
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

Training results

Training Loss	Epoch	Step	Validation Loss	Strict Accuracy	Adjacent Accuracy	Macro F1	Mae	Qwk
0.4597	1.0	733	0.2285	0.9644	0.9908	0.9472	0.0482	0.9738
0.1328	2.0	1466	0.1902	0.9785	0.9947	0.9666	0.0285	0.9849
0.0370	3.0	2199	0.2034	0.9807	0.9935	0.9687	0.0279	0.9839
0.0168	4.0	2932	0.1880	0.9833	0.9943	0.9747	0.0240	0.9863

Per-Class Evaluation Distribution (Custom Table 2 Replication)

This table tracks the performance breakdown on the internal 10% stratified validation partition derived from the training inventory:

CEFR Level	Validation Count	Accuracy (%)	Error Rate (%)
A1	2,500	99.28%	0.72%
A2	2,500	98.48%	1.52%
B1	2,500	98.40%	1.60%
B2	2,353	97.96%	2.04%
C1	462	95.02%	4.98%
C2	98	92.86%	7.14%

---

🧪 Out-of-Distribution Robustness Benchmark (Balanced Unseen Dataset)

To establish true generalizability and rule out internal data leakage, the finalized model was subjected to an Independent Out-of-Distribution (OOD) Stress-Testing Routine.

Benchmark Methodology

The evaluation architecture was frozen in pure inference mode (model.eval(), torch.no_grad()) and fed an entirely separate, non-overlapping dataset slice from EFCAMDAT originally reserved for testing a distinct Plug-and-Play Language Model (PPLM) classifier module. To remove all baseline prevalence bias and simulate a strict diagnostic environment, the evaluation subset was forcefully downsampled into a perfectly balanced profile anchored to the absolute minority class floor ($N = 928$ rows per class across all 6 categories, creating a global verification volume of 5,568 samples).

Global Generalization Metrics

• Strict Accuracy: 97.74%
• Adjacent Accuracy (Distance $\leq$ 1): 98.83%
• Micro F1 Score: 0.9774
• Macro F1 Score: 0.9774
• Mean Absolute Error (MAE): 0.0383
• Quadratic Weighted Kappa (QWK): 0.9861

Per-Class Diagnostic Stress Test Breakdown

CEFR Level	Evaluated Samples	Accuracy (%)	Error Rate (%)
A1	928	99.68%	0.32%
A2	928	99.03%	0.97%
B1	928	98.38%	1.62%
B2	928	98.28%	1.72%
C1	928	95.69%	4.31%
C2	928	95.26%	4.74%

---

How to Use

You can easily instantiate this model for high-throughput batch inference using the code snippet below:

import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Define target repository path
MODEL_ID = "MohammadKhosravi/roberta-large-cefr-classifier-JointLoss"

# 1. Load native tokenizer and joint loss classifier
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)

# Place model on optimized GPU accelerator if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

# 2. Prepare sample student essays for diagnostic assessment
sample_texts = [
    "I like to play football with my friends on Sunday. It is very fun.",
    "Although the economic implications of the policy change are highly controversial, empirical data suggests a clear trend toward market stabilization."
]

# 3. Compute inference pipeline
inputs = tokenizer(sample_texts, padding=True, truncation=True, max_length=512, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1).cpu().numpy()

# 4. Map output tokens to clean strings
id2label = model.config.id2label
for text, pred_id in zip(sample_texts, predictions):
    print(f"\nText Sample: {text[:80]}...")
    print(f"Predicted CEFR Level: {id2label[pred_id]}")