# Sidecar schema

Version: `open-greek-corpus-annotations/1.0`

This repo stores token-level linguistic annotations (lemma, UPOS, UD
morphology) for the [Open Greek Corpus](https://github.com/open-greek/open-greek-corpus)
as sidecar files. The corpus repo owns the text; this repo owns the
annotations; the two are joined by content hashes so neither can silently
drift from the other.

## Files

One sidecar per corpus work, mirroring the corpus layout:

```
corpus:  data/corpus/<urn>.jsonl        (one record per citable passage)
sidecar: data/annotations/<urn>.jsonl   (one record per corpus record)
```

A sidecar is JSON Lines. The first line is a `_meta` header; every following
line is one annotation record corresponding to one corpus row.

## The `_meta` header

```json
{"_meta": {
  "schema": "open-greek-corpus-annotations/1.0",
  "urn": "aeschylus-tragedy.persae",
  "corpus": "github.com/open-greek/open-greek-corpus",
  "corpus_file": "data/corpus/aeschylus-tragedy.persae.jsonl",
  "edition": "1st1K-grc1",
  "text_license": "CC-BY-SA-4.0",
  "annotations_license": "CC-BY-SA-4.0",
  "tokenizer": {"name": "open-greek-corpus-annotations-whitespace-edgepunct",
                "version": "1.0.0",
                "compatible_with": "dilemma-tagger 0.7.x (grc/med profile, no MWT)"},
  "annotators": {"gold:glaux": {"...": "source, license, mapping version"},
                 "silver:dilemma-1.2.0": {"...": "versions, mode"}},
  "generated": "2026-07-10",
  "tool": "scripts/build_gold_glaux.py",
  "stats": {"...": "row/token/tier counts, alignment stats"}
}}
```

`annotators` must describe every provenance label used in the file: source,
license, and the exact versions (for silver: the dilemma package version, the
tagger module version, and the pinned tagger weights revision; for gold: the
treebank source, its license, and the postag-mapping version).

## Annotation records

One record per corpus row, keyed by `{urn, locus, text_sha256}`:

```json
{"urn": "aeschylus-tragedy.persae",
 "locus": "1",
 "text_sha256": "6acd5dce...",
 "tokens":     ["Τάδε", "μὲν", "Περσῶν", "τῶν", "οἰχομένων"],
 "offsets":    [[0,4], [5,8], [9,15], [16,19], [20,29]],
 "lemmas":     ["ὅδε", "μέν", "Πέρσης", "ὁ", "οἴχομαι"],
 "upos":       ["PRON", "PART", "PROPN", "DET", "VERB"],
 "feats":      ["Case=Nom|Gender=Neut|Number=Plur", "",
                "Case=Gen|Gender=Masc|Number=Plur",
                "Case=Gen|Gender=Masc|Number=Plur",
                "Case=Gen|Gender=Masc|Number=Plur|Tense=Pres|VerbForm=Part|Voice=Mid"],
 "provenance": ["gold:glaux", "gold:glaux", "gold:glaux", "gold:glaux", "gold:glaux"],
 "tier":       ["gold", "gold", "gold", "gold", "gold"]}
```

| field | meaning |
|---|---|
| `urn` | the corpus work slug (the corpus repo's primary work key) |
| `locus` | the corpus row's citation locus, verbatim |
| `text_sha256` | SHA-256 hex digest of the corpus row's `text` string (UTF-8 bytes, text exactly as served, no normalization) |
| `tokens` | token surface forms, in text order; each is an exact substring of the text |
| `offsets` | per token, `[start, end)` in Unicode code points into the text |
| `lemmas` | per token, NFC lemma (see lemma conventions below) |
| `upos` | per token, [UD UPOS](https://universaldependencies.org/u/pos/) tag |
| `feats` | per token, UD morphological features as a sorted `Key=Val\|Key=Val` string, `""` when none |
| `provenance` | per token, `gold:<source>`, `auto:<source>`, or `silver:<annotator>-<version>` (or `none` if a token could not be annotated) |
| `tier` | per token, confidence tier: `gold`, `auto`, `high`, `medium`, `low` |

The seven arrays are token-aligned and always the same length. All Greek text
in the annotation fields is NFC. Everything needed to interpret a token is
per-token, so records and even single tokens can be consumed independently.

### Keying and invalidation

`text_sha256` pins each record to the exact served text. When the corpus
corrects a passage (OCR fix, edition swap), that row's hash changes, the old
annotation record fails validation for precisely that row, and only that row
needs re-annotation. Nothing is keyed by file position or line number, so
corpus files can be regenerated, reordered, or extended without invalidating
untouched rows.

A locus is not assumed unique within a work: some works legitimately repeat a
locus across rows (e.g. split verse lines share a line number). The full key is
`(urn, locus, text_sha256)`, and the validator matches a record to the corpus
row with its locus **and** hash, so duplicate loci with distinct text are
valid.

## Tokenizer (versioned)

`scripts/ga_tokenizer.py`, name `open-greek-corpus-annotations-whitespace-edgepunct`,
version `1.0.0`. It reproduces the dilemma tagger's tokenization
(`dilemma.tagger` 0.7.x, [open-greek/dilemma](https://github.com/open-greek/dilemma)),
so silver annotations from the tagger map 1:1 onto sidecar tokens (the
builders assert this parity on every row). Rules:

1. words are runs of non-whitespace (any Unicode whitespace splits);
2. leading and trailing punctuation (Unicode category P) is split off each
   word into standalone tokens; runs of the same character stay one token
   (`"..."`), mixed runs split per character;
3. apostrophe-family marks (U+2019, U+02BC, U+1FBF, U+0027, U+2018, U+0384)
   never split: word-final ones are Greek elision, word-initial ones
   aphaeresis, and both stay attached to the word (U+1FBD and U+02B9 are not
   category P and also stay attached);
4. interior punctuation (hyphens, crasis marks) is never touched;
5. no multiword-token expansion (see open questions: MG `el` mode).

The tokenizer never inserts, deletes, or rewrites characters, which is what
makes the offset invariant possible.

### Validation invariant (offset round-trip)

For every record, against the corpus row with the same `(urn, locus)`:

- `sha256(text) == text_sha256`;
- for every token `i`: `text[offsets[i][0]:offsets[i][1]] == tokens[i]`;
- offsets are strictly increasing and non-overlapping;
- every character of the text outside all token spans is whitespace.

Therefore interleaving the inter-token gaps (all whitespace) with the tokens
reconstructs the text exactly. `scripts/validate_sidecar.py` enforces all of
this plus the structural schema; run it with `--cog-dir` for the round-trip.

## Provenance and confidence tiers

Three annotation classes, distinguishable per token (the middle class was
anticipated by open question 5 below and added 2026-07-13 when GLAUx's
automatic layer was split out of `gold:`):

- `gold:<source>`: the token was aligned to a token of a MANUALLY annotated
  (or manually homogenized) treebank, and lemma/UPOS/feats were taken from
  it. Sources: `gold:glaux` (only the sentences GLAUx marks manually
  corrected, `analysis="manual"`, in texts whose TREEBANK_ANNOTATIONS names
  an openly licensed manual layer: Perseus AGDT, Pedalion, Gorman via the
  perseids release, Harrington; see scripts/treebank_licenses.py) and
  `gold:macula` (MACULA Greek, the manually curated Nestle1904 NT). The
  gold/auto split is per SENTENCE, not per text: a treebank text yields gold
  only where GLAUx actually hand-corrected it.
- `auto:<source>`: the token was aligned to a token of an automatically
  annotated corpus. Source: `auto:glaux` - GLAUx's own transformer
  annotation (self-reported 97.2% morphology / 98.8% lemmas), covering both
  texts with no manual layer at all and the automatically tagged sentences
  inside treebank texts. Between gold and silver: not manual, but an
  independent second system aligned to our text.
- `silver:dilemma-<version>`: annotated by the dilemma tagger + lemmatizer
  (ONNX runtime). The `_meta.annotators` entry records the dilemma package
  version, tagger module version, pinned tagger weights revision, and mode
  (`grc`, `med`, or `el`).

Tiers:

| tier | meaning |
|---|---|
| `gold` | aligned to a gold treebank token (implies `gold:` provenance, and vice versa) |
| `auto` | aligned to an external automatic corpus token (implies `auto:` provenance, and vice versa) |
| `high` | silver: deterministic annotation (punctuation, numerals), or a Greek token whose chosen lemma is backed by an exact lookup hit in the dilemma lexicon |
| `medium` | silver: lemma produced by rules/normalization or the character-level model, or unverifiable against the candidate list |
| `low` | silver: identity-fallback lemma, or a token with no Greek letters (Latin editorial matter, apparatus, sigla) |

Tier reflects lemma confidence primarily; UPOS/feats confidence is captured
globally by the agreement report (see below), not per token.

### Non-lexical tokens (silver)

Silver runs dilemma's non-lexical classifier (`dilemma.classify_nonlexical`,
dilemma >= 1.2.2) over every token. Editorial/typographic residue that
pervades OCR'd lexica, scholia, and the Patrologia Graeca - variant marks
(`γρ`), Greek numerals (`κζ`), bracket references (`[76]`), citation
abbreviations (`fr.`, `Herod.`), sigla, and vowel-less fragments - is surfaced
as UPOS `X` (UD's non-lexical/other category) with the surface form as its own
lemma and tier `low`, instead of letting the tagger invent a `NOUN`/`PROPN`
reading or the lemmatizer manufacture a spurious lemma. Punctuation keeps the
tagger's `PUNCT` (finer than `X`) and its deterministic-high tier. The
classifier is conservative (a real Greek word always has a vowel; numerals are
unaccented with descending place-value), so real words - including elided
monosyllables and numeral-shaped words like `τε` - are untouched. Gold tokens
are never reclassified (they carry the treebank's own POS).

### Silver mode selection

The tagger mode comes from the work's era tag in the corpus source registry:

| era tag | mode |
|---|---|
| `era:archaic`, `era:classical`, `era:hellenistic`, `era:imperial`, `era:late-antique` | `grc` |
| `era:byzantine`, `era:early-modern` | `med` |
| (none) | `grc` |

`med` currently reuses the `grc` GreBerta weights (Byzantine literary Greek
is classicizing); it is a distinct label so the two can diverge later.

## Lemma conventions

- Gold lemmas are the source treebank's, NFC-normalized (GLAUx is NFD
  upstream). For GLAUx/AGDT this is broadly the LSJ-style treebank
  convention (articles to the article lemma, elided forms resolved).
- Silver lemmas are dilemma's Wiktionary-headword convention with
  `resolve_articles=True` (articles to the article lemma, pronoun clitics to
  the pronoun lemma), POS-aware disambiguation from the tagger's predicted
  UPOS, and a proper-noun guard: a lowercase token not tagged PROPN never
  receives the capitalized proper-noun twin of a common lemma.

The two conventions differ for a small set of words (verb stem families,
Ionic/Attic pairs). Agreement reports score both strict equality and
equivalence-group-adjusted equality to quantify exactly this gap.

## AGDT/GLAUx to UD mapping

Gold treebanks use the 9-position AGDT positional postag. `scripts/agdt_ud.py`
(version 1.0.0, vendored from dilemma's `convert_treebank.py` so gold and the
tagger's training convention are identical) maps it to UD as follows.

Position 0 (part of speech) to UPOS:

| AGDT | UPOS | note |
|---|---|---|
| `n` | `NOUN` | `PROPN` when the lemma starts with an uppercase Greek letter |
| `v` | `VERB` | |
| `a` | `ADJ` | |
| `d` | `ADV` | |
| `l` | `DET` | article |
| `c`, `b` | `CCONJ` | `SCONJ` when the lemma is a known subordinator (fixed list in `agdt_ud.py`); `b` is GLAUx's coordinator/particle code |
| `r` | `ADP` | |
| `p` | `PRON` | |
| `m` | `NUM` | |
| `i`, `e` | `INTJ` | |
| `g` | `PART` | |
| `u` | `PUNCT` | |
| `x`, other, or malformed postag | `X` | |

Positions 1-8 to UD features:

| position | values | UD |
|---|---|---|
| 1 person | `1/2/3` | `Person=1/2/3` |
| 2 number | `s/p/d` | `Number=Sing/Plur/Dual` |
| 3 tense | `p/i/r/l/f/a/t` | `Tense=Pres` / `Tense=Past,Aspect=Imp` / `Tense=Pres,Aspect=Perf` / `Tense=Pqp` / `Tense=Fut` / `Tense=Past,Aspect=Perf` / `Tense=Fut,Aspect=Perf` |
| 4 mood | `i/s/o/n/m/p` | `Mood=Ind/Sub/Opt` + `VerbForm=Fin`; `n` = `VerbForm=Inf`; `m` = `Mood=Imp,VerbForm=Fin`; `p` = `VerbForm=Part` |
| 5 voice | `a/p/m/e` | `Voice=Act/Pass/Mid/Mid,Pass` |
| 6 gender | `m/f/n` | `Gender=Masc/Fem/Neut` |
| 7 case | `n/g/d/a/v` | `Case=Nom/Gen/Dat/Acc/Voc` |
| 8 degree | `c/s` | `Degree=Cmp/Sup` |

Sidecars store the full mapped feature set. For agreement scoring only, both
sides are additionally projected through `normalize_feats` (a per-UPOS
whitelist matching what the tagger's training gold annotated: e.g. `Person`
is not annotated on pronouns, `Degree=Pos` is dropped, `Voice` is dropped on
the copula), so silver is scored against the convention it was trained on.

## Gold alignment (treebank -> corpus rows)

Implemented in `scripts/build_gold_glaux.py` (source selection and the
per-source coverage rationale are in [docs/GOLD_SOURCES.md](GOLD_SOURCES.md);
GLAUx was the only alignable gold source in the pilot; scripts/build_gold_macula.py now aligns MACULA the same way).

1. Locus join. A corpus row's locus is matched against the finest GLAUx
   citation level that has it, so both verse and prose join without knowing
   the scheme in advance: `div_section` / `div_chapter` / `div_book` for prose
   (GLAUx annotates these cumulatively, e.g. `div_section="1.1.1"`, which
   equals the corpus locus), then `line` for verse (treebank split lines like
   `909b` and corpus `book.line` loci fall back to the numeric line).
2. Token alignment within the row, in two tiers. First a
   `difflib.SequenceMatcher` skeleton over a diacritic-insensitive fold key
   (NFC, casefold, combining marks stripped, sigma folded, and the Greek high
   stop unified - First1KGreek prints ano teleia as a colon) anchors the two
   streams; being diacritic-insensitive it already absorbs accent, breathing,
   and iota-subscript variation. Then a second positional pass over the tokens
   difflib leaves in the gaps, gated by
   [prosodia](https://github.com/open-greek/ancient-greek-prosodia-engine)
   `is_compatible` (the shared Open Greek canon for SAFE alternations),
   recovers same-word pairs whose surface *letters* differ - movable nu
   (`πᾶσι`/`πᾶσιν`), elision (`ἀλλ'`/`ἀλλά`), lunate sigma, iota subscript vs
   adscript - which the skeleton key cannot bridge. The gate is strict: a
   lexical homograph (`τι`/`τί`, `εἰς`/`εἷς`) is never `is_compatible`, so gap
   repair only ever adds true same-word matches, never merges two words, and
   the result is a strict superset of the skeleton-only match. Matched pairs
   are classified `exact` (equal after NFC + apostrophe unification alone) or
   `variant` (equal only after folding). Artificial (elliptical) treebank
   tokens are never aligned; they have no surface text.
3. Whole-work global fallback. When the locus join covers under 85% of a
   work's tokens - GLAUx texts with no `div_*`/`line` metadata at all (e.g.
   most Aelius Aristides orations), or works whose corpus edition cites at a
   different granularity than GLAUx - the full corpus token stream is aligned
   against the full GLAUx token stream in document order (same folded-key
   `SequenceMatcher`), and adopted if it covers more. `stats.align_method`
   records `locus` or `global` per work. This recovers works that would
   otherwise align at ~0%.
4. Silver fill. Corpus tokens the gold cannot cover are left as provenance
   `none` by the pure-gold pass and filled by the silver pass (provenance
   says so per token). Punctuation aligned across editions keeps the served
   edition's character as its lemma.

Alignment quality is logged per work in `data/reports/<urn>.alignment.json`
(coverage %, method, match-level split, unmatched examples) and aggregated in
`data/coverage.json`. Residual failure classes: edition punctuation
differences, line-numbering drift in lyric, cross-edition textual variants
(fragmentary works can align low and fall through to silver), and treebank
sub-word splits (crasis).

## Versioning

- `schema`: bump on any record-shape change.
- `tokenizer.version`: bump on any tokenization behavior change; a bump
  invalidates all sidecars (offsets may differ).
- `agdt_ud.py` mapping version: bump on mapping changes; a bump invalidates
  gold/auto sidecars built from postag sources.
- `macula_ud.py` mapping version: bump on mapping changes; a bump
  invalidates `gold:macula` sidecars.
- Silver annotator versions are recorded per file in `_meta.annotators`;
  re-annotation with newer models produces a new `silver:...` label rather
  than silently overwriting provenance.

## Open questions for the full run

1. Multiword tokens in Modern Greek mode: the `el` tagger splits MWTs
   (`στο` -> `σ` + `το`), which breaks the substring invariant. Options: UD-style
   range tokens (a surface token that spans two syntactic words), or keep
   surface tokens and annotate the fused form. Not needed for `grc`/`med`.
2. Long prose rows: the tagger encoder context is 256 subwords; the builders
   chunk long rows at punctuation boundaries (soft cap 120 words). Proper
   sentence segmentation inside rows (offset-preserving) would tag with
   better context. Corpus rows are mostly passage-sized, so this matters for
   a minority of works.
3. Cross-line locus drift: the pilot joins strictly by locus; lyric passages
   where editions number cola differently lose a few percent of gold tokens.
   A work-level global token alignment (anchored by the locus join, allowed
   to spill across neighboring lines) would recover most of them.
4. Treebank crasis/sub-word splits: GLAUx sometimes splits crasis into
   multiple analyzed parts; the corpus keeps one surface token. A 1:N merge
   policy (store the head part's analysis? concatenate?) is undecided; the
   pilot falls back to silver for these.
5. Additional gold sources: AGDT originals (CC BY-SA 3.0 US) and Gorman
   (CC BY-SA 4.0, via the perseids-publications release) can be aligned
   directly where GLAUx lacks a work; Diorisis (CC BY 4.0, sentence-granular
   loci, Beta Code) is itself automatically annotated, so it belongs in a
   provenance class of its own between gold and silver rather than under
   `gold:`. (That class now exists: `auto:`, first used for GLAUx's
   automatic layer. Diorisis remains unused: dilemma already carries its
   signal.)
6. Per-token dependency syntax (head/deprel): both the treebanks and the
   dilemma tagger produce it. Left out of schema 1.0 because head indices
   are sentence-relative and corpus rows are not sentence-aligned; needs a
   sentence-span notion in the record.
7. Tier for numerals and sigla: numerals are tier `high` for the trivial
   identity lemma even though the tagger's UPOS on them is sometimes shaky;
   a UPOS-aware tier component may be worth adding.
8. Proper-noun case-twin guard (resolved upstream): the capitalization
   tiebreak that `silver._decap` applies (a lowercase non-PROPN token
   never gets the capitalized proper-noun twin of a common lemma) now
   lives inside dilemma itself, in `lemmatize_pos` / `lemmatize_batch_pos`
   (dilemma 1.2.1). The upstream ranking is strictly broader than the
   sidecar guard: it also fixes single-candidate lookups like ψυχῆς
   whose verbose list has only the capitalized `Ψυχή`, using the POS
   table's lowercase lemma - cases `_decap` could not reach. With the
   guard disabled and the upstream fix active, Persae lemma agreement is
   92.59% equiv-adjusted, matching the guarded number, so `_decap` is now
   redundant defense-in-depth and can eventually be dropped.