# Aligned-layer sources: what was ingested and why

The gold layer annotates a corpus token by aligning it to a token of a
manually annotated (or manually homogenized) treebank and taking that token's
lemma / UPOS / features. Only openly licensed layers are eligible, and the
gate checks the ANNOTATION layer's license, not just the source text's
(scripts/treebank_licenses.py): PROIEL (CC BY-NC-SA) and every other
NonCommercial layer are refused.

Revised 2026-07-13. The original 2026-07 run had two defects, both fixed:

1. Everything aligned from GLAUx was labeled `gold:glaux`, although GLAUx is
   mostly its own automatic annotation. Corrected in two steps. First, whole
   texts with no manual layer (TREEBANK_ANNOTATIONS = NA) were moved to the
   middle provenance class `auto:glaux` (see docs/SCHEMA.md), relabeled in
   place by `scripts/relabel_glaux_auto.py`. Then the split was pushed down to
   the SENTENCE: GLAUx hand-corrects only a minority of sentences even inside
   a treebank text and auto-tags the rest, so
   `build_gold_glaux.glaux_token_label` labels each token from its sentence's
   `analysis` status - `gold:glaux` for manually corrected sentences,
   `auto:glaux` for automatic ones. This moved ~2.5M tokens (the automatic
   sentences inside the treebank texts) out of the treebank-gold headline,
   from 3.6M to 1.08M.
2. The license gate read only GLAUx's SOURCE_LICENSE column (the TEXT
   license), so the PROIEL-derived layers of the 24 GLAUx NT texts and
   Herodotus (CC BY-NC-SA annotations on permissively licensed texts)
   shipped as gold. The gate now also refuses NC annotation layers; the NT
   was rebuilt from MACULA (below) and Herodotus from dilemma silver.

## GLAUx - the primary aligned source, split by layer class

[GLAUx](https://github.com/alekkeersmaekers/glaux) (CC BY-SA 4.0, a handful
of NonCommercial texts refused by the license gate) is the merged,
homogenized corpus that carries the openly licensed manual treebank layers
plus its own automatic annotation of everything else. The project's
multi-source rule is to prefer the merged/voted edition over any
single-source one, so GLAUx is the primary aligned source rather than the
raw per-project treebanks it subsumes.

Crosswalk: a served work's CTS TLG id (`urn:cts:greekLit:tlgAAAA.tlgWWW`)
maps to GLAUx's `AAAA-WWW` text id. Per GLAUx's metadata
TREEBANK_ANNOTATIONS column:

- `gold:glaux` (manually corrected sentences; 100 served works, 1.08M gold
  tokens): the hand-annotated treebank layers -
  Perseus AGDT (CC BY-SA 3.0 US), Pedalion Trees (CC BY-SA 4.0), Vanessa
  Gorman's prose treebanks (via the perseids-publications release, whose
  README says CC0 1.0 while its TREEBANK_LICENSE file carries CC BY-SA
  4.0; we record the stricter CC BY-SA reading. GLAUx links her personal
  repo, which is CC BY-NC-SA 4.0), and Harrington Trees (CC BY-SA 4.0). A
  treebank text contributes gold only on its manually corrected sentences,
  not its automatic gaps.
- `auto:glaux` (GLAUx's automatic annotation; 1,070 served works, 16.0M
  tokens): the 1,036 texts with no manual layer at all (TREEBANK_ANNOTATIONS
  = NA) plus the automatic sentences inside the 34 partly-treebanked texts.
  GLAUx's own transformer annotation (self-reported 97.2% morphology, 98.8%
  lemmas). Not manual, so not gold, but an independent second system aligned
  token-by-token to our served text.
- Refused: the 25 PROIEL-layered texts (CC BY-NC-SA 3.0).

Alignment (unchanged by the split): locus join where GLAUx carries
`line`/`div_*` metadata, whole-work global token alignment otherwise; the
tokens the alignment cannot cover are silver-filled. Per-work numbers in
`data/reports/<urn>.alignment.json`, aggregates in `data/coverage.json`.

## MACULA Greek - manual gold for the New Testament

[MACULA Greek](https://github.com/Clear-Bible/macula-greek) (CC BY 4.0,
Biblica) is manually curated token-level annotation of the Nestle 1904 NT:
lemma, word class, and decomposed morphology per word. It replaces the
refused PROIEL layer and extends gold to all 27 served NT works (the GLAUx
NT lacked 2 Peter and 1-2 John), 135K gold tokens.

- The corpus serves the NT from Perseus (perseus-grc2 = Westcott & Hort),
  so `build_gold_macula.py` does a cross-edition alignment: verse-level
  locus join (with a two-entry verse-alias map for Mark's shorter ending
  and the Matthew 23 Woes numbering), then the same two-tier token aligner
  as GLAUx. Gold covers 82-88% of corpus tokens per book; the remainder is
  almost entirely punctuation (MACULA does not tokenize it) plus W&H
  editorial brackets, all silver-filled.
- UPOS/feats come from `macula_ud.py`, which composes the 9-slot AGDT
  postag from MACULA's attributes and converts through the same `agdt_ud`
  mapping as every other source, so the UD convention is uniform.

## AGDT - subsumed by GLAUx

The [Perseus Ancient Greek Dependency Treebank](https://github.com/PerseusDL/treebank_data)
(CC BY-SA 3.0 US) is GLAUx's core manual layer. Of its 66 Greek `*.tb.xml`
files, all but one map to a GLAUx text already ingested. Ingesting AGDT
directly would mean preferring a single-source treebank over the merged one,
so it is used only where GLAUx has no coverage, which is one work. Not worth
a separate aligner; noted here for completeness.

## Gorman - ingested via GLAUx; openly licensed via perseids

The Gorman trees cover 158 files across Xenophon, Athenaeus, Demosthenes,
Polybius, Herodotus, Lysias, Antiphon, Aeschines, Thucydides, Plutarch, and
others; all authors GLAUx already covers (their GLAUx layers ARE Gorman's
trees, homogenized), so direct ingestion adds no gap works, and the raw
trees carry no CTS line/section citation anyway. License: her personal repo
(vgorman1/Greek-Dependency-Trees) says CC BY-NC-SA 4.0; the
perseids-publications/gorman-trees release of the same trees says CC0 1.0
in its README but ships the CC BY-SA 4.0 text in TREEBANK_LICENSE (an
upstream self-contradiction). This repo records the stricter CC BY-SA 4.0
reading; both readings are non-NC, so the gate passes either way.

## Diorisis - automatically annotated, and subsumed by dilemma

The [Diorisis corpus](https://figshare.com/articles/dataset/6187256)
(CC BY 4.0, Beta Code) is itself automatically lemmatized and tagged, so it
would be `auto:`, not `gold:`. Its loci are sentence-granular and its 23
gap works are silver-eligible anyway; dilemma already uses Diorisis as
training and attestation data, so its signal is not lost. Not ingested.

## OGA (Opera Graeca Adnotata) - automatic; evaluated head-to-head

[OGA](https://github.com/OperaGraecaAdnotata/OGA) v0.2.0 (CC BY-SA 4.0,
Celano) is an all-automatic annotation (Trankit morphosyntax + GreTa
lemmatization) of ~40M tokens of Perseus/First1KGreek/PTA texts. Like
Diorisis it is `auto:`-class by definition. It is already available
locus-joined in the corpus checkout (`data/annotations/oga/oga-v1`, the
export dilemma consumes for attestation), so instead of ingesting it as a
third layer we measured it head-to-head against dilemma on the manual gold
layer: see `scripts/eval_oga_vs_dilemma.py` and
`data/reports/oga_eval_summary.json`, summarized in [OGA_EVAL.md](OGA_EVAL.md).

## Net result

Aligned layers = GLAUx (`gold:glaux` on the hand-corrected sentences of 100
treebank works, `auto:glaux` on 1,070 works - the fully-automatic texts plus
the automatic sentences inside treebank works) + MACULA (27 NT works as
`gold:macula`). Everything
else is dilemma silver, including Herodotus (whose only external layer is
PROIEL). The non-ingested candidates are subsumed (AGDT, Gorman, PROIEL's
NT role via MACULA), unalignable at locus granularity (raw Gorman), or
automatic with no edge over dilemma (Diorisis, OGA - see the evaluation).