Seronic/ambistory-e13

AmbiStory E13: DeBERTa-v3-large plausibility scorer (SemEval 2026 Task 5)

Rates how plausible (1–5) a given sense of a homonym is in a short story, the task of SemEval 2026 Task 5 (AmbiStory). Code, training scripts, evaluation and the Streamlit app: github.com/Seronic2001/ambistory.

Model

• microsoft/deberta-v3-large cross-encoder. Segment A is the story with the homonym quoted; segment B is homonym: gloss. Example: sentence.
• Mean pooling, then an ordinal (proportional-odds) head over the 5 ratings. The score is the expected rating.
• Trained on all of train+dev for 6 epochs with KL to the annotator distribution, a within-setup ranking loss and MSE, story-grouped batches and layer-wise LR decay. The checkpoint was chosen on an inner homonym-grouped split (kaggle/e13_train.py --deploy).
• Two seeds (model_seed42, model_seed43), stored as fp16 safetensors (828 MB each). The app averages them.

These are custom PyTorch modules, not a transformers pipeline. Load them with load_deployed from the repo:

from huggingface_hub import snapshot_download
import sys; sys.path[:0] = ["ambistory", "ambistory/kaggle"]        # a clone of the GitHub repo
from e13_train import load_deployed, build_inputs

path = snapshot_download("Seronic/ambistory-e13")
model, tok, meta = load_deployed(f"{path}/model_seed42")

Results (official 930-item test set)

	Spearman ρ	Accuracy within SD
2 seeds averaged	0.681	78.0%
2 seeds + within-setup soft-rank blend (as in the app)	0.702	80.1%

Accuracy counts a prediction as correct when |pred − mean| < max(annotator SD, 1), as in the official scorer. These scores were computed after the official evaluation period. The test set was never used for training or model selection. The cross-validated version of this model (5 folds × 3 seeds) reaches 0.719, and blending it with a zero-shot Qwen2.5-32B judge reaches 0.811 (see the GitHub README).

Limitations

Trained only on AmbiStory (English, short constructed stories, 275 homonyms). On CPU it takes about 1.2 s per story–meaning pair per model, and PyTorch dynamic int8 quantization of the feed-forward layers breaks it (correlation with the full-precision scores drops to ~0.1), so the weights are served unquantized.