danielscottsmith/schlaims-claim-selector

SchLAIMS Claim Selector

SchLAIMS Claim Selector is a Longformer-based paragraph selector. Given an ordered scientific paragraph, it returns the one sentence that states the paragraph's central claim, or null when no sentence does.

The model was trained on AI-consensus reference labels, not fully human-adjudicated labels. We used agents to create the training labels only after a hidden test showed that they could follow the human-written guidelines and usually select the same claims as human annotators. The agents never saw the human answers. Across 10 seeded low-reasoning GPT-5.4 mini runs, the retained human-consensus test had about 0.84 mean claim F1 and 0.948 mean sentence accuracy. The complete hand-labeled set, including seven paragraphs marked for adjudication, had about 0.69 mean claim F1 and 0.908 mean sentence accuracy. This validates the labeling procedure; it does not make every training or test label human gold. See the SchLAIMS validation pipeline.

Fixed Test Results

Both results below come from one inference pass and must be reported together.

Gold view	Scorable paragraphs	Precision	Recall	Claim F1
Full random-resolved	4,603	0.7032	0.7039	0.7035238841
Majority-only	4,538	0.7117	0.7106	0.7111252586

The full view includes deterministic selections from observed votes for three-way label disagreements. The majority view excludes those paragraphs, so it measures a different population. It is not simply a corrected score.

Use

Install the project environment and run its supported loader:

git clone https://github.com/danielscottsmith/SchLAIMS.git
cd SchLAIMS
conda env create -f 03_ml_training/environment.yml
conda run -n schlaims_ml python 03_ml_training/scripts/canonical_run.py predict \
  --input paragraphs.jsonl \
  --output predictions.jsonl \
  --model-repo danielscottsmith/schlaims-claim-selector \
  --revision v1

Each input row needs a unique paragraph_id and an ordered sentences list; each sentence needs consecutive sentence_n values starting at 1 and text. The output records the selected sentence or null plus every choice score. The custom sentence-selector head requires the SchLAIMS loader and is not a generic Transformers text-classification pipeline.

On the dataset card's example paragraph (triad_4498__original__p0037), this checkpoint selects sentence 1, matching the unanimous reference label.

Training And Limits

The checkpoint is run_20260711_1135: Longformer base 4096, seed 2027, learning rate 6e-6, weight decay 0.01, clip norm 1.0, 300 warmup updates, and a three-epoch horizon. Checkpoints were selected only on the full random-resolved development view, with 1,000-update checks, minimum improvement 0.002, patience 5, and a 21-check one-epoch floor.

The release has mixed but recorded preparation lineage: the first 150 triads were prepared and carried forward before the remaining 350 were added. One test paragraph was excluded because it exceeded model context. The benchmark is a retrospective baseline refined through iterative diagnostics, not a new prospective sealed test. The labels lack final human adjudication, and the model does not check truth, evidence quality, or scientific importance.

The checkpoint and selector head are released under Apache 2.0. Exact reproduction files for Version 1 are also available in the hashed canonical model archive.