FlashFacts-4B: Earnings Extraction That Says Null Instead of Guessing

FlashFacts-4B is a LoRA adaptation of google/gemma-3-4b-it that reads a real SEC earnings exhibit and returns the revenue figure, the period it covers, the units it is reported in, and the year-over-year change, as exact JSON. When the filing does not state a fact, it returns null.

Built for the Adaption AutoScientist Challenge, Part 2 (Market Analysis and News); the training corpus was co-optimized with Adaptive Data (Adaption Labs).

Trained on 13,976 rows, every one of them a real EX-99 filing. Zero synthetic rows.

Read this first: what kind of number follows

Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy against figures taken from the filing itself.

This entry is the one that can prove the difference matters, because it ran both. The identical corpus was trained on Adaption's own AutoScientist, and that platform reported a 95.48% win rate. We downloaded that adapter and scored it with our scorer on our slice: 62.0% exact-match accuracy. Same corpus, same base, same adapter. A win rate is a preference; an accuracy is a fact, and here they differ by 33 points.

what is measured	exact match against the figures in the filing
a language model judged something	never, not the labels, not the scoring
synthetic training rows	zero, all 13,976 are real SEC filings
held-out rows scored	650, across three independent slices
rows the base model wins	0 of 650
significance	p = 1.1e-109, exact McNemar, held-out
shortcut gain from column position	0.00, measured, not asserted
traceability	every row carries its SEC accession number and a SHA256 of the source

TL;DR

On held-out filings the base model scores 0.0% and FlashFacts-4B scores 90.8%. On an out-of-distribution anchor drawn from 10-Q filings, a different document type entirely, it scores 74.0%.

The number worth reading twice is not the headline. It is that this corpus has no positional shortcut, so the score cannot be earned by guessing.

The shortcut test

The obvious way to fake financial extraction is positional: always take the first numeric column. The build measures that directly, on the released corpus.

accuracy from column position alone	57.58%
majority-class baseline	57.58%
shortcut gain	0.00

A model that sees only which column a number sits in does exactly as well as always guessing the most common answer, and no better. An earlier build leaked 0.189 of shortcut gain and it was engineered out before training. Any score on this task was earned by reading the table.

Headline results

Base and tuned generated in one process under identical greedy decoding, scored by the released score_flashfacts.py, which a reviewer can run on a laptop with no GPU.

slice	rows	what it isolates	base	FlashFacts-4B
ff_held	400	held-out filings, companies and periods disjoint from training	0.0%	90.8%
ff_hard	150	scale and period disambiguation together	0.0%	91.3%
ff_ood	100	10-Q filings, a different document type	0.0%	74.0%

Significance, stated rather than assumed

Base and tuned answer the same rows in one process, so this is a paired design and the right test is an exact McNemar test on the discordant rows, those where exactly one model was right. The null is that the adapter is no better than the base.

slice	only tuned right	only base right	exact two-sided p
ff_held	363	0	1.1e-109
ff_hard	137	0	1.1e-41
ff_ood	74	0	1.1e-22

The base wins no row on any slice, including the out-of-distribution 10-Q anchor. Recompute with the released significance.py from the released generation files.

Abstention is graded, not assumed

Every row in the training corpus has at least one genuinely absent field. In this domain a system that invents a plausible revenue figure when the filing does not state one is worse than useless, so returning null correctly is scored as its own outcome rather than folded into accuracy.

	base	FlashFacts-4B
correctly returned null for an absent field	96.5%	99.9%
invented a value for an absent field	2.1%	0.1%
emitted parseable JSON	98.5%	100.0%

A second arm, trained on Adaption's own platform

The identical corpus was also trained through Adaption's AutoScientist, on their compute, and the resulting adapter was downloaded and evaluated on our slice with our scorer under identical greedy decoding, so the two arms are directly comparable.

arm	training	ff_held
ours	one A100, 3 epochs, lr 1e-4, max_len 2048	90.8%
platform	AutoScientist, 5 epochs, lr 1e-5, batch approximately 550	62.0%

The platform reported its own win rate of 95.48% on iteration 1 of 3. That figure is not an accuracy: a win rate is the share of rows where the tuned output beats the base, and this base scores near zero on its own distribution, so beating it is not the same as being right. Both numbers are published, and only the column measured with our scorer is comparable to anything else on this card.

Why this base

google/gemma-3-4b-it is the base the AutoScientist platform itself trains, which is what makes the platform arm above directly comparable: same base, same corpus, two training paths. It is also the base of every entry in this portfolio, fixed before any result existed.

Its 0.0% headline deserves one sentence of honesty in each direction. The base is not globally broken, and this card reports where it is competent: 96.5% correct nulls and 98.5% parseable JSON. What it cannot do is land all four fields of the strict contract at once, and that is the task. The claim here is the absolute score against that contract, not the ratio to the base, which is why the abstention and format tables report the base's real abilities beside the tuned model's.

Scope and operating notes

1. Scope. US registrants filing in English with the SEC, 2018 to 2024, EX-99 earnings exhibits. Revenue and period extraction specifically. Not guidance, not sentiment, not forward-looking statements, and not investment advice.
2. Token budget is real. The median row is 1,716 tokens and the maximum is 2,000. A 1024 window truncates 100% of rows and 1536 truncates 80.2%. This model requires a 2048 context; run it shorter and it will fail on the input, not on the task.
3. Decontamination has a declared gap. Checked clean against FinQA, ConvFinQA and TatQA across 3,250 reference questions, maximum 4-gram Jaccard 0.0151. FinanceBench was named in the build spec but was not available locally, so it is not covered. Stated rather than omitted.
4. 920 rows were dropped at build time for exceeding the token budget, rather than trained on truncated. Under completion-only masking a truncated completion is a silently poisoned label.

Training details

base	google/gemma-3-4b-it
method	LoRA r32 alpha64 on the language-model linears, completion-only masking
corpus	13,976 rows, 100% real SEC filings
epochs	3
max_len	2048, measured with the real tokenizer, not assumed
hardware	one A100-40GB, Spot

max_len is measured because two specs on this project were wrong about it and a truncated completion is a silently poisoned label. Batch size is 2 rather than the usual 4 for the same reason it has to be: at 2048 tokens, bsz 4 puts 8192 positions in a 262,145-wide loss tensor, which is the exact shape that ran this base model out of memory before.

Evaluation protocol

Base and tuned are generated in one process under identical greedy decoding (do_sample=False). Output length is compared base against tuned, and a tuned median below 0.6x the base median fails the run regardless of accuracy, which is a guard against buying a win rate with terseness. Every gate was proven able to fail by injecting deliberately corrupted rows before the corpus was trusted.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
                                             attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/flashfacts-4b").eval()

Try it, and everything that backs it

Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/flashfacts-demo Enter your own input and watch the base model and FlashFacts-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.

	Hugging Face	Kaggle
model	https://huggingface.co/Jainamshahhh/flashfacts-4b	https://www.kaggle.com/datasets/jainamshahhh/flashfacts-4b
dataset	https://huggingface.co/datasets/Jainamshahhh/flashfacts	https://www.kaggle.com/datasets/jainamshahhh/flashfacts-sec-earnings-extraction

Released on both platforms, with the scorer, every eval slice, the per row verdicts and significance.py alongside, so every number on this page can be recomputed rather than trusted.

Reproducibility and license

The released corpus, the scorer and every eval slice are published. Each row carries its SEC accession number and a SHA256 of the source exhibit, so any label can be traced to the exact document and rechecked. Apache-2.0, matching the base model. SEC filings are US government works in the public domain. Built with Adaptive Data by Adaption Labs.