TensorGreed/brewslm-ntsb-accident-extractor

Qwen2.5-1.5B accident-report extractor (NTSB aviation narratives → JSON)

This is Qwen/Qwen2.5-1.5B-Instruct, fine-tuned with LoRA (merged into the weights) to turn a U.S. aviation accident narrative into one flat JSON record of 13 fields: aircraft, damage, injuries, operating rule, flight purpose, date, state and pilot hours. When the narrative doesn't state a field, it writes null instead of guessing.

For education only. This model is a demonstration of a training method. It is not an aviation-safety or accident-investigation expert, and it makes no claim to subject-matter expertise. It can be wrong, and it is not a source of NTSB findings. For any real purpose, use the official NTSB records at data.ntsb.gov. Don't use its output as evidence of what caused an accident (see Limitations).

Trained with BrewSLM, a platform for building small language models from your own data. Every step below (import, splits, training, evaluation, packaging) ran through it. How it was built, including where it still misses: Narrative in, JSON out: a 1.5B accident-report extractor that knows when to say null, and more on the BrewSLM blog. Non-commercial showcase; no warranty.

Result

1,234 held-out test narratives. Each output is parsed as JSON and compared field by field against the answer key. Comparison is typed: numbers as numbers, strings ignoring case and spacing, and null / missing / "" all count as null.

	Base model, same prompt	Base model, schema-aware prompt¹	This model
Output matches the schema (valid JSON, every field, right types and allowed values)	0.042	0.770	0.995
Fields exactly right (13 fields, all rows)	0.494	0.481	0.959
When it says null, the field really is unstated (null precision)	0.901	0.533	0.954
When a field is unstated, it says null (null recall)	0.545	0.371	0.976

¹ A 300-row subset of the test set. The prompt adds the chat template, the types and allowed values, and "use null when not stated". It fixes the format but makes the base model invent more, for example guessed dates and "Not specified" strings. Prompting alone doesn't give you this behaviour. (This one column was run by a separate script, scored with BrewSLM's own scorer; everything else ran inside BrewSLM.)

Per field (seed 42, test):

field	exact (all rows)	right when the text states it (rows)	null recall	base exact
aircraft_category	0.986	0.991 (1220)	0.571	0.120
aircraft_make	0.917	0.811 (508)	0.992	0.777
aircraft_model	0.752	0.388 (454)	0.964	0.702
damage	0.977	0.990 (1155)	0.785	0.490
event_date	0.994	0.996 (547)	0.991	0.796
flight_purpose	0.960	0.975 (563)	0.946	0.089
highest_injury	0.985	0.995 (567)	0.978	0.556
injuries_fatal	0.984	0.991 (567)	0.978	0.581
injuries_minor	0.977	0.982 (564)	0.973	0.507
injuries_serious	0.980	0.982 (567)	0.978	0.569
location_state	0.991	0.991 (555)	0.991	0.256
pilot_total_hours	0.970	0.765 (85)	0.985	0.574
regulation_part	0.991	0.993 (542)	0.990	0.406

How much to trust these numbers:

• The pass rule was written before any model was scored, and frozen with a checksum. It was amended twice before any evaluation ran: the paired gain is measured on per-row field F1, because per-row exact match isn't stored; and the published seed is the best on validation. Three seeds (13, 42, 1234) were trained on identical data, and each was evaluated on test once. All three had to pass every gate: schema-valid ≥ 0.95, overall field accuracy ≥ 0.85, null precision ≥ 0.95, null recall ≥ 0.90, per-field accuracy ≥ 0.70 and per-field null recall ≥ 0.80 on the 8 fields a narrative states directly, and a paired gain over the base model whose 95% CI excludes 0. The published seed is the one with the best validation score, chosen before any test result existed.

  Seed	Schema-valid	Field accuracy	Null precision	Null recall	Paired gain vs base, per-row field F1 (95% CI)	Val field accuracy
  13	0.994	0.958	0.955	0.976	+0.444 (+0.430 to +0.458)	0.9595
  42 (this model)	0.995	0.959	0.954	0.976	+0.444 (+0.430 to +0.458)	0.9614
  1234	0.997	0.957	0.952	0.977	+0.443 (+0.429 to +0.457)	0.9610

• What "right" means here. The answer key is the investigators' coded value, kept only when a fixed text check finds it stated in the narrative, and set to null otherwise. So the score measures "extracts what the text says". It doesn't measure "knows the true facts of the accident". A check can miss a value that is phrased unusually. The model is then marked wrong for extracting it, so the scores lean conservative.

• The 5 category fields aren't gated. aircraft_category, damage, highest_injury, regulation_part and flight_purpose map free wording onto fixed codes. They are reported above but weren't part of the per-field pass rule.

• No language model wrote any of the data, training or test. Every label comes from the NTSB's own coded fields and a deterministic text check.

How the data was made

• Source: the NTSB aviation accident database (bulk download from data.ntsb.gov/avdata, October 2026 snapshot).
• Population: accidents (not incidents) from 2015 on, with exactly one aircraft and a narrative that includes the probable cause, at most 14,000 characters: 12,421 events.
• Text: the published narrative sections joined and cleaned: the long narrative where there is one (70% of events), the summary narrative, and the probable-cause statement, with repeated paragraphs dropped.
• Answer key: for each field, the investigators' coded value, set to null unless a deterministic check finds it stated in the text. Examples: the make or model appears as written; the injury counts are stated; the date appears in the text (if the text's date is a day off the coded one, the text wins); total hours appear. Most narratives don't state most fields: only 7% state pilot total hours, and about 40% name the make.
• Splits: by a hash of the NTSB event id: 9,931 train, 1,256 validation, 1,234 test. No event is in two splits, and no narrative text is shared between splits.

Training

LoRA r 16, alpha 32, dropout 0.05, on every linear layer. bf16, AdamW, learning rate 2e-4 with cosine decay, warmup 3%, effective batch 16. Loss is on the JSON answer tokens only. Max sequence length is 4,096; the longest row is 3,366 tokens, so nothing is truncated. 1 epoch, 621 steps. Final train / eval loss for seed 13 is 0.010 / 0.008; for seed 42 (this model) 0.009 / 0.008. Trained on one NVIDIA GB10. The adapter was merged into the base weights.

Prompt format

The model was trained on a plain prompt with no chat template. Use it exactly; other layouts were never trained or tested.

Extract the following fields as JSON: aircraft_category, aircraft_make, aircraft_model, damage, event_date, flight_purpose, highest_injury, injuries_fatal, injuries_minor, injuries_serious, location_state, pilot_total_hours, regulation_part.
Reply with a single JSON object, nothing else.
Input: <narrative>
Output:

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "TensorGreed/brewslm-ntsb-accident-extractor"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

FIELDS = ("aircraft_category, aircraft_make, aircraft_model, damage, event_date, flight_purpose, highest_injury, "
          "injuries_fatal, injuries_minor, injuries_serious, location_state, pilot_total_hours, regulation_part")
narrative = "..."  # one accident narrative
prompt = (f"Extract the following fields as JSON: {FIELDS}.\nReply with a single JSON object, nothing else.\n"
          f"Input: {narrative}\nOutput:")
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
record = json.loads(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Output schema (allowed values; null is always allowed):

field	type	allowed values
aircraft_category	string	airplane, helicopter, glider, balloon, gyroplane, weight_shift, powered_parachute, other
aircraft_make, aircraft_model	string	as written in the narrative
damage	string	destroyed, substantial, minor, none
highest_injury	string	fatal, serious, minor, none
injuries_fatal / _serious / _minor	integer
regulation_part	string	part_91, part_121, part_135, part_137, other
flight_purpose	string	personal, instructional, aerial_application, business, positioning, other
event_date	string	YYYY-MM-DD
location_state	string	two-letter U.S. state code
pilot_total_hours	integer	rounded

Limitations

• One source, one writing style. It was trained and tested only on NTSB narratives written since 2015, for accidents involving one aircraft. Other agencies' reports, older reports, incidents and multi-aircraft accidents weren't tested.
• It extracts; it doesn't judge. The answer key covers what the narrative states, not what is true. A value the narrative gets wrong will be extracted wrong.
• Not evidence. Under 49 U.S.C. § 1154(b), NTSB accident reports can't be admitted as evidence in a civil action for damages. A model trained on them is no better: don't use its output to argue cause or liability.
• It leaves out aircraft models it should give. This is its biggest weakness. For 454 test narratives that state the model designator, it returned null for 232 and a different designator for 46. The make is missed less often: 88 of 508. It leans toward null. Across all fields it wrote null for a stated value 382 times, and invented a value for an unstated field 192 times. If you need the aircraft model, check null answers against the text.
• Null precision only just clears its bar. It is 0.954 against a floor of 0.95, and the lowest seed was 0.952. About 1 in 22 null answers hides a value the text does state.
• Whole records are right 62% of the time. Field accuracy is 0.959, but all 13 fields are right on 62% of test narratives (base model: 0%). Most of the gap is the aircraft model.
• Educational demonstration only, with no claim to aviation or investigation expertise. Use the official NTSB records for anything real.

Citation

Trained and evaluated with BrewSLM. The full story is in the build write-up on the BrewSLM blog. If you reuse the method, a link back is appreciated.

License

The model is Apache 2.0, the base model's license.

The training text is NTSB material: text prepared by U.S. Government employees as part of their official duties, which the NTSB states it "has placed … in the public domain" (NTSB policies; 17 U.S.C. § 105). Courtesy: National Transportation Safety Board. The answer key is a masked derivative of the NTSB's coded fields, not NTSB findings. The model is not endorsed by or affiliated with the NTSB.