# OGA vs dilemma: head-to-head on the manual gold layer

Question: should OGA (Opera Graeca Adnotata v0.2.0, Celano's all-automatic
annotation of ~40M tokens; Trankit morphosyntax + GreTa lemmatization)
replace or supplement dilemma as this repo's silver annotator? OGA is
already locus-joined to the corpus (the `oga-v1` export in the corpus
checkout, which dilemma consumes as attestation data), so the comparison is
cheap to run and was measured on 2026-07-13 with
`scripts/eval_oga_vs_dilemma.py` over 15 manual-gold works (260K scored
gold tokens; all four treebank families, prose and verse, Attic and
dialect). Full numbers: `data/reports/oga_eval_summary.json`, per-work
reports `data/reports/<urn>.{agreement,oga_agreement}.json`.

Reading the metrics honestly requires stripping THREE convention artifacts,
each applied identically to both sides:

- lemma_fold: case- and homograph-digit-insensitive (OGA lowercases every
  lemma);
- upos with NOUN/PROPN merged (PROPN derives from lemma capitalization,
  which OGA's lowercasing suppresses);
- upos_func: additionally merges PART/ADV/CCONJ/SCONJ into one bucket.
  This one matters most: the gold here is GLAUx-HARMONIZED postags, the
  convention dilemma trains on, while OGA trains on original AGDT
  conventions, and the two schemes genuinely disagree on how to class the
  highest-frequency particles (δέ, ἄν, γάρ, καί, οὖν...). Without the
  merge, dilemma gets a large home-convention advantage that says nothing
  about OGA's accuracy.

| aggregate | lemma_fold | upos (N/P merged) | upos (func merged) | feats | all exact |
|---|---|---|---|---|---|
| dilemma | 95.58 | 96.91 | 97.97 | 96.42 | 90.26 |
| OGA | 94.64 | 87.05 | 95.61 | 90.04 | 69.70 |

So the headline "10-point UPOS gap" is mostly scheme mismatch: it shrinks
to 2.4 points once the function-word cell is neutralized. The remaining
gaps are real, though:

- upos_func: dilemma leads on every family (2.4-3.9 points), including
  Gorman (99.16 vs 95.30), the one family where OGA trained on exactly
  these trees while dilemma held them out. (Even that comparison retains a
  residual convention confound, since the gold passed through GLAUx's
  harmonization; treat it as indicative, not exact.)
- feats: 96.42 vs 90.04, immune to the function-word cell (those classes
  carry no features under the scoring whitelist), so this is a genuine
  morphology gap.
- lemmas: near-parity on standard Attic prose (OGA even edges dilemma on
  Memorabilia, In Timarchum, the Apology), but OGA collapses on dialect
  and comedy (Theocritus 87.1 vs 94.2, Acharnians 90.0 vs 95.3,
  Agamemnon 91.9 vs 95.2).

Caveats: there is no OGA-clean holdout (its models train on the AGDT
family), so OGA's numbers are if anything flattered; and OGA's export
paired 57.7-100% of gold tokens per work (edition drift, worst on
Theocritus), so its scores cover the paired subset while dilemma is scored
on all gold tokens.

Conclusion: dilemma stays the sole silver annotator. On convention-neutral
metrics OGA is competitive but behind everywhere measured (morphology
clearly, lemmas on non-Attic material), and for this repo's sidecars
convention fit is itself decisive: silver must agree with the gold layer's
UD conventions, which dilemma is trained to. OGA remains valuable inside
dilemma as attestation data (where it already lives), not as a provenance
layer here. This matches dilemma's own earlier finding that OGA-derived
POS edges regressed its Byzantine benchmark.