SigExt — 2500 samples, threshold 0.80 (wits-it, xlmr)

SigExt is a lightweight salient-information extractor: a Longformer fine-tuned for binary token classification that marks the sentences of a source document that should appear (in some form) in its summary. Extracted keyphrases steer a frozen generator LLM — this is the extractor component of the SM-SIP pipeline (Semantic & Multilingual Salient Information Prompting).

Language / corpus: Italian — Wikipedia-style articles (WITS, Italian)
Backbone: markussagen/xlm-roberta-longformer-base-4096 (multilingual XLM-RoBERTa extended to 4096-token contexts)
Training samples: 2500
Labeling threshold: 0.80 — training labels are silver labels: a source sentence is marked salient (1) if its Sentence-BERT embedding (sentence-transformers/paraphrase-multilingual-mpnet-base-v2) reaches cosine similarity > 0.80 with any summary sentence. Semantic supervision replaces the fuzzy lexical matching of prior work, which is what makes this work multilingual.
Task: token classification, 2 classes (salient / not salient)

Why

Generic LLM summarizers hallucinate and omit salient facts. SM-SIP decouples what to say (this extractor) from how to say it (a steered generator): keyphrases extracted by SigExt are injected into the generation prompt as hard constraints, grounding the LLM in the source's most informative content. On the 10k-sample WITS configuration the full pipeline reaches BERTScore 0.6245 / ROUGE-1 0.1772 / KIR 28.93% (Keyphrase Inclusion Rate, zero-shot steering of Llama-3-8B-Instruct).

Naming legend

sigext-<corpus>-<backbone>-<samples>-<threshold> — this model is one point of a grid swept over corpus (wits-it / arxiv-en), backbone (xlmr / allenai), training-set size (1k / 2500 / 5k) and S-BERT labeling threshold (0.60 / 0.70 / 0.80). See the collection for the other configurations.

Training

Weighted cross-entropy (salient class ×10 vs ×1) to fight the ~10% positive-class imbalance
max_length 2048, lr 2e-5, fp16, effective batch 8 (2 × grad-accum 4), 2 epochs
Labels projected onto the start of each sentence's token span (-100 elsewhere)

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

name = "LookUpMark/sigext-wits-it-xlmr-2500-080t"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name)

inputs = tok(document, return_tensors="pt", truncation=True, max_length=2048)
with torch.no_grad():
    preds = torch.argmax(model(**inputs).logits, dim=2)[0]

salient = [t for t, p in zip(inputs.tokens(), preds.tolist()) if p == 1]
print(tok.decode(tok.convert_tokens_to_ids(salient)))

Citation & links

Project write-up: SM-SIP implementation guide
Methodological reference: Xu et al., 2024 — industry-track paper on salient-information prompting for summarization (2024.emnlp-industry.4)

Research artifact: intermediate checkpoints of a course-project grid sweep, shared for reproducibility. Each configuration was trained with identical hyperparameters; only corpus, backbone, data size and labeling threshold differ.