skykg-4b

A 4B search agent for aviation safety questions. It does not answer from its own knowledge — it is trained to find passages in a corpus through three tools and answer only from what it has opened, citing verbatim.

Given a student pilot's question, skykg-4b issues facet queries against an annotated corpus, reads the passages that come back, and either answers with citations or says plainly that the corpus does not cover the question.

Q: "...if I read back an instruction to cross the runway but aren't sure ATC caught it,
    am I required to stop and actively verify that clearance before crossing?"

→ search(index=situation, object=readback_verification, family=Aeroplane, op="Class C (ATC)")
→ open(FAA-H-8083-15B#570) · open(FAA-H-8083-16B#217)
→ neighbours(FAA-H-8083-15B#570, same_document)

A: "Yes — the corpus puts the burden on you to close the loop... The NAS chapter is explicit that
   ATC requires readbacks 'in order to verify that clearances and instructions are understood',
   and that 'in order to complete the communication loop, the controllers must also clearly
   understand the pilot's readback'..."

SOURCES
- FAA-H-8083-15B#570: "In order to complete the communication loop, the controllers must also
  clearly understand the pilot's readback and other responses."
- FAA-H-8083-16B#217: "When a pilot is uncertain of the clearance, ATC should be queried
  immediately as to what route of flight is preferred."

What is in here

The model and the graph it was trained against, so it runs end to end with no assembly:

model.safetensors-0000{1,2}-of-00002   the 4B policy (8.9 GB)
run_skykg.py                           the agent loop: vLLM, three tools, forced answer at budget
skykg/graph_env_full.py                the retrieval environment, as trained against
skykg/card_tools.py                    system card, observation renderer, SOURCES parser
skykg/rollout_tools.py                 think/tool-call parsing
skykg/reward.py                        the verifiable reward, to score your own runs
skykg/card.txt                         the frozen system card (the exact string trained with)
graph/graph.jsonl.gz                   205,656 annotated passages (89 MB)
graph/docmeta.json                     per-document metadata
kg/release/skyowl_states_v2.jsonl.gz   the states layer: 59,883 atoms, 591 causal pairs (20 MB)
KG.md                                  what the graph is, its facets, its provenance

skykg/ is the actual environment code, not a reimplementation. The model's behaviour is specific to these search semantics, so a rewritten index would change its results.

The three tools it emits calls against:

tool	contract
search(index, facets, k)	ranked passages with their annotation, not their text
open(uid)	the passage text
neighbours(uid, edge, k)	typed edges: same_act, same_document, same_causal_pair

search withholds the text on purpose. A model handed passages at search time has no reason to choose between them, and the trajectory collapses to one query and an answer.

The graph is 205,656 passages from FAA handbooks, accident reports, ASRS callbacks and type manuals, each annotated with an act (readback_verification, altitude_selection, …), a polarity (how it went wrong: omitted, late, insufficient, …), a flight phase, an operation and an aircraft family. It is included — KG.md documents the record schema, the facet vocabulary and the three jobs a field can do.

The one thing to understand before reading results: family is a scope, not a score. It filters candidates rather than ranking them. As a scoring field it could nudge a ranking but never exclude anything, and 43.3% of returned hits were the wrong aircraft family — a student asking about a DA40 was shown helicopter passages.

Running it

pip install "vllm==0.23.0"      # the version every number here was measured on

# the graph alone, no GPU: three example searches, an open, and a typed edge
python run_skykg.py --probe

# one question, printing each turn
python run_skykg.py --model . --question "Does waiting until touchdown to apply rudder work in a crosswind?"

# a batch, one atomic json trace per question
python run_skykg.py --model . --questions questions.jsonl --out traces/

Graph load is 7 s and ~2.1 GiB of host RAM, paid once per process — not a GPU cost. Point at a different graph with SKYKG_GRAPH= and STATES=.

Two things that will break it if you get them wrong

Do not set stop=["</tool_call>"]. The model is trained to issue several tool calls in one turn. That stop string truncates the generation after the first one — measured on ~2,000 turns, it forces tool-calls-per-turn to median 1, p90 1, max 1, which roughly doubles the number of turns a trajectory needs and halves throughput.

Give it room. Final contexts run to a median of ~7,500 tokens and p90 of ~10,000. Trained with max_model_len 32768; at 16,384 the context guard fires on about half of all trajectories and cuts them off mid-search.

Sensible defaults: temperature 0.6, max_tokens 1536 per turn, 10 tool-calling turns, then force an answer from what has been opened.

Training

Distilled from a 27B teacher by rejection sampling on a verifiable reward.

39,998 questions were back-translated from corpus passages, the teacher produced one trajectory each, and every trajectory was scored by inspection against the graph — no judge model. The reward's target is a carrier: any passage carrying the question's act, aircraft family and phase. Trajectories scoring below +0.30 were discarded.

39,998 teacher trajectories      55.1% self-answered · 37.6% forced at budget · 7.3% truncated
15,080 above reward +0.30        (the distribution is bimodal; any bar in [0.20, 0.50] gives this set)
 9,530 cited answers             all kept
 1,299 refusals                  downsampled to 12%, all self-terminated rather than budget-forced
10,548 rows trained              Qwen3.5-4B, ZeRO-2 on 4xH100, seq 16384, 1.39 epochs, lr 3e-5

Loss 0.774 → 0.344. Refusals were kept deliberately but held to 12%: a model that has learned to say "the corpus does not cover this" is useful, and one that says it by default is not.

Measured behaviour

On 200 held-out questions, scored against the same verifiable reward:

	opened a carrier	self-answered
skykg-4b	0.401	70.6%
27B teacher, same 200 questions	0.300	—
earlier 4B, unfiltered data	0.087	17.9%

It is above its 27B teacher on retrieval, which is what rejection sampling buys: it trained only on trajectories that found evidence, never on the teacher's failures.

Two cautions on that table. The teacher's rate over all 39,998 training questions is 0.377, a different question set — do not mix it into this column. And the 4.6× over the earlier 4B mostly measures a rebuilt retrieval environment rather than the model, since that one was trained on a hand-rolled JSON action format with a different system card: it is not a like-for-like comparison.

The noise floor is ~5 points at n=200. Two runs of an identical configuration gave 65.2% and 60.8% self-answered. Differences smaller than that are not resolvable — in this table or in your own measurements.

Current known limits

Attribution is weaker than retrieval. 31% of answers emit no SOURCES block. Of those, most do quote passages correctly in prose — it is a formatting failure more than a grounding one — but about 4% of all answers quote text not present in any passage they opened

It declines more than ideal. 27% of trajectories run to the action budget without committing to an answer.

Reward-shaped blind spot. The reward scores retrieval and citation fidelity, not whether the question was answered. A well-cited honest refusal can score above a correct answer, so the training signal does not fully distinguish them. Expect occasional over-hedging on questions the corpus can in fact settle.

Aviation only, English only. It is trained on one corpus with one annotation schema, and the facet vocabulary is closed — weight_and_balance, not "weight and balance". Queries outside the vocabulary match nothing.

Not for operational use. This is a research artefact for retrieval over a safety corpus. It is not an authority on flight operations, it is not a substitute for a flight instructor, certified publications or an operator's own procedures, and its answers must be checked against the cited source before being relied on for anything.