ror-set-point-pair-scorer-baseline

This repository contains the baseline for ROR set point matching. The baseline uses the design of openalex-affiliation-matcher v3.0.0, retrained on the train split of affiliations_unseen_institutions. The models of the collection trained on the same strings.

Part of the ROR set point matching collection.

The pair scorer of this baseline learns from Jev's judgments: the teacher_judgments config of ror-set-point-matching-train-validation-test-openalex-jev-distill. The retriever and set model of the matcher learn from the judgments of StartLux-Decision v1.1 27B Instant.

What the baseline does

1. Candidate generators find records for the string: a lexical index, a dense index, an acronym rule and a table of aliases. The alias table comes from the training strings. This gives about 18 candidates.
2. A pair scorer gives each (string, record) pair a probability. The pair scorer is a cross-encoder: multilingual-e5-base with mean pooling and one output.
3. A chooser selects the set of records. The chooser is a gradient-boosted tree model with 35 features. It names a record when its score is 0.50 or more.

The baseline works with OpenAlex institution ids. Its answers are changed to ROR ids for evaluation.

Training data

Part	Config and split	Rows
Pair scorer: labelled pairs	affiliations_unseen_institutions train	16,605 positive and 270,134 negative pairs
Pair scorer: retention pairs	teacher_judgments train	321,999 pairs from 17,079 strings
Chooser	affiliations_unseen_institutions train	308,727 candidate rows

Retention pairs keep the knowledge of the released pair scorer. Teacher strings that the teacher links to a held-out institution are not used.

Training procedure

Pair scorer

The training started from the pair scorer of openalex-affiliation-matcher v3.0.0.

Setting	Value
Loss	Binary cross-entropy
Batch	64 pairs
Epochs	1 (9,511 steps for the final model)
Optimizer	AdamW, learning rate 2e-5, weight decay 0.01, linear warm-up for 5% of the steps, then linear decay
Maximum length	192 tokens
Precision	bf16 autocast
Seed	1363
Hardware	1 × NVIDIA A10G for each run, 31 to 35 minutes

Five fold runs give out-of-fold probabilities for the training strings. The chooser trains on these probabilities. A final run uses all rows and scores the evaluation strings.

Chooser

A HistGradientBoostingClassifier (scikit-learn 1.9.1) with 35 features. The threshold of 0.50 gives the best exact match on the out-of-fold answers of the training strings.

Evaluation

Exact match, precision and recall, in percent.

Split	Strings	Exact match	Precision	Recall
affiliations_unseen_institutions test	1,957	79.2	93.6	85.1
affiliations_unseen_institutions validation	1,914	75.0	94.8	84.8
affiliations validation	1,947	85.7	94.7	87.4

The card of ror-set-point-set-model-mmbert-base-mask compares this baseline with the set model.

For these comparisons, comparison/ holds the answers of two more systems:

• v3.0.0_local/: openalex-affiliation-matcher v3.0.0 with the parts anyone can run (lexical and dense candidates, the pair scorer and the chooser), on all six evaluation sets. It trained on some strings and institutions of affiliations_unseen_institutions.
• all_institutions/: this design, trained on the labelled strings of all institutions (affiliations train and openalex_sample train), on affiliations_test, datacite_test and openalex_sample_test. This repository does not hold that model.

These rows have OpenAlex institution ids (id, ids). The scorer of the training code changes them to ROR ids with data/ror_records of the dataset.

Files

Path	Content
model/	The final pair scorer: model.pt, student.json and tokenizer files
chooser/	The chooser model (model_names.pkl, scikit-learn 1.9.1), its summary, and the ROR registry file that its features read
evaluation/predictions/<set>.jsonl	The answers (id, ror_ids) for unseen_validation, unseen_test and affiliations_validation. id is the id of the dataset row
evaluation/probabilities/<set>.jsonl	id, and p: the pair scorer's probability for each candidate, as [OpenAlex id, p]
comparison/<system>/<set>.jsonl	The answers (id, ids in OpenAlex ids) of v3.0.0 (v3.0.0_local) and of this design trained on all institutions (all_institutions)
training_data/	The pair-scorer rows (train.jsonl.gz), the pairs to score (score_*.jsonl.gz), the scores of each run (student_scores_*.jsonl.gz) and manifest.json
candidates/	The candidate pools with features and pair-scorer probabilities, in OpenAlex ids
code/	The code of the chooser and the code that it needs
logs/	The logs and summaries of the six pair-scorer runs

The names unseen_validation and unseen_test are the validation and test splits of affiliations_unseen_institutions; affiliations_validation is the affiliations validation split. The files in candidates/, code/ and logs/ use the names of the training run.

To run the baseline, use the code of openalex-affiliation-matcher at commit 3ead9a7 with the files in code/.

Licence

The pair scorer comes from openalex-affiliation-matcher (MIT) and multilingual-e5-base (MIT).

Related repositories

• Dataset: ror-set-point-matching-train-validation-test.
• Collection: ROR set point matching.
• Retriever: ror-set-point-retriever-e5-base. Set model: ror-set-point-set-model-mmbert-base-mask. Baseline: ror-set-point-pair-scorer-baseline.