scihigh2026-subtask1-bart-large-cnn

Fine-tuned facebook/bart-large-cnn for SciHigh-2026 Subtask 1: Research Highlight Generation — generating 3-5 short, telegraphic highlight sentences (~50-55 words total) from a scientific paper's abstract.

Task

Given a paper's abstract, generate a small set of highlight sentences that summarize the paper's key contributions/findings in a compressed, telegraphic style (not a paraphrase-style abstractive summary). Ranked by ROUGE-L F1 (primary metric), with ROUGE-1/2, METEOR, and BERTScore also reported.

Baseline: the FIRE-2025 shared-task's winning submission (fine-tuned Pegasus-large + NER features, 10 epochs) scored 23.45% ROUGE-L F1.

Training data

Fine-tuned on the MixSub-SciHigh dataset, an expanded pool of 15,960 (Abstract, Highlights) pairs: the official 2026 task's 10,000-row training split, plus 5,960 additional real training pairs recovered from the dataset's original source release (leakage-checked by exact Abstract-text match against the official validation/test splits before being used for training — zero overlap confirmed). Validated against the official held-out validation split (1,985 rows), which was not used for training, only for per-epoch checkpoint selection.

Training recipe

Setting	Value
Backbone	facebook/bart-large-cnn
Learning rate	2e-5 (Adafactor optimizer)
Batch size	2 (per device)
Warmup / label smoothing	none
Max input / output length	512 / 100 tokens
Beam width	4
Epoch ceiling	4, with load_best_model_at_end=True + EarlyStoppingCallback(patience=2) on validation ROUGE-L
Decode settings	no_repeat_ngram_size=3 (explicit); repetition_penalty and min_length left at bart-large-cnn's own shipped defaults (1.0 / 56)

Hyperparameters mirror the FIRE-2025 baseline recipe unchanged (a hyperparameter sweep over learning rate, warmup, and label smoothing found every variant flat-to-worse on top of this backbone). The epoch ceiling is a safety bound, not a fixed schedule — early stopping determines the actual stopping point.

In the training run that produced this checkpoint, validation ROUGE-L peaked after epoch 1 and did not improve in epochs 2 or 3; early stopping halted training after epoch 3 (before reaching the 4-epoch ceiling), and the epoch-1 checkpoint (this one) was restored as the final model.

Evaluation results

Evaluated on the full official validation split (1,985 rows):

Metric	Score	vs. FIRE-2025 baseline (23.45% ROUGE-L)
ROUGE-1	36.22%	—
ROUGE-2	13.17%	—
ROUGE-L	24.37%	+0.92 pts absolute (+3.9% relative)
ROUGE-Lsum	24.36%	—
METEOR	32.48%	—
BERTScore F1	87.49%	—

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "Yakk99/scihigh2026-subtask1-bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

abstract = "..."  # a scientific paper's abstract
inputs = tokenizer(abstract, max_length=512, truncation=True, return_tensors="pt")
output = model.generate(**inputs, max_length=100, num_beams=4)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])

max_length=100 is set explicitly to match the recipe's output-length budget — the checkpoint's own generation_config already carries this value, so a plain model.generate(**inputs) call (with no extra arguments) also reproduces the reported results.

Files

• train_final.py / train_final.ipynb — the standalone training script and a companion notebook documenting this exact recipe end-to-end (data, model, training configuration, decode settings, and the real reported results above).

Limitations

• Abstracts in the source dataset are frequently truncated mid-sentence (a scraped-preview artifact in the original data release), which imposes a hard ceiling on achievable ROUGE-L for the affected subset — some reference highlights cite facts genuinely absent from the (truncated) input.
• This model was selected and evaluated on ROUGE-L against a single validation split; it has not been evaluated for factual accuracy/ hallucination beyond standard n-gram/embedding-based metrics.