ValantisAndreas/heva-baselines

HEVA baselines

Fine-tuned multilingual BERT baselines from HEVA: A Dataset, Toolkit and Benchmark for Heritage Value Extraction. Code, tasks and full results: https://github.com/zzbn12345/HEVA_toolkit.

• T1, value classification: predicts which of the eight Cultural Value Framework values a text expresses (multi-label). A value is predicted at probability 0.5.
• T2, value span extraction: labels each token with one of 17 BIO tags. Each word takes the prediction of its first subword.

Each task was trained with three seeds on one split:

Task	Seed	Subfolder
T1	13	t1/20261102-144928/model
T1	14	t1/20261102-152545/model
T1	15	t1/20261102-160931/model
T2	13	t2/20261102-134010/model
T2	14	t2/20261102-135455/model
T2	15	t2/20261102-140246/model

Results

Test set, mean ± standard deviation over the three seeds.

Task	Metric	Score
T1	Micro-F1	.747 ± .010
T1	Macro-F1	.727 ± .010
T2	Exact span micro-F1	.285 ± .016
T2	Overlap span F1	.551 ± .024
T2	Token F1	.542 ± .002

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "ValantisAndreas/heva-baselines"
folder = "t1/20261102-144928/model"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=folder)
model = AutoModelForSequenceClassification.from_pretrained(repo, subfolder=folder)

inputs = tokenizer("The old harbour is a landmark.", return_tensors="pt", truncation=True)
probabilities = model(**inputs).logits.sigmoid()[0]
values = [model.config.id2label[i] for i, p in enumerate(probabilities) if p >= 0.5]

For T2, use AutoModelForTokenClassification with a t2/... subfolder. Replace blank tokens such as \n with tokenizer.unk_token first, as the training code does; otherwise some predictions change.

Loading the tokenizer may print a warning about a regex pattern. It doesn't affect these models: the tokenizer gives the same encodings as the original mBERT tokenizer.

Limitations

The training data are dominated by one collection (Yazd), so T1 scores are lower on other collections (micro-F1 0.67 without Yazd). Span extraction is weak, especially for rare values.