SCPO Reward Model — Canada

A scalar reward model fine-tuned on PRISM preference pairs from participants residing in Canada. Higher scores indicate a preferred response for the same prompt. Scores are not probabilities and are not calibrated across models.

Paper · Code

Recorded experiment results

Measurement	Accuracy
Starting global model, overall	60.4046%
SCPO model, overall	66.1850%

These are original in-memory post-training experiment scores, with no model-mode or random-state reset before scoring. A fresh eval() inference session disables dropout and may produce different accuracy. validation.json checks loading and finite, repeatable inference on generic probes; it is not a benchmark result.

Configurations were tuned on a fixed test split; these checkpoint results differ from the paper’s three-run averages. Reproduction details are in provenance.json.

Training and reproduction

Full-model fine-tuning from OpenAssistant/reward-model-deberta-v3-base.
Base revision: 7dab4156e63944834c3aae169c3fe83955672cff.
PRISM revision: 18ab5cfb37456f4ec8cbc00212ce54cf7b1239f6.
Learning rate: 1e-06; batch size: 8; optimizer steps: 1024.
Filtering tau: 0.7; weighting beta: 1.1.
Data shuffle seed: 0; survey split/model seed: 0; Trainer seed: 42; Trainer data seed: unset.
Survey users split 85/15; human-rating ties excluded from preference pairs.
Original training inputs use literal prompt + " [SEP] " + response, capped and padded to 512 tokens. Original scoring uses tokenizer prompt/response pairs in interaction batches of at most eight with no explicit truncation.

See release-settings.json, provenance.json, and requirements-pw-lock.txt. The exact country script and unchanged trainer/collator are included under reproduction/. With the recorded environment installed, run:

python reproduction/preferences.py ./reproduction-output

Original loaders use upstream main; verify the recorded model/data revisions before rerunning. No private dataset records are bundled.

Inference

The following uses standard evaluation-mode inference.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "MulticulturalAlignment/scpo-reward-model-canada"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
inputs = tokenizer("How can I disagree respectfully?",
                   "Listen carefully and explain your reasons calmly.",
                   return_tensors="pt")
with torch.inference_mode():
    reward = model(**inputs).logits.item()
print(reward)

Intended use and limitations

Research on preference modeling and pluralistic alignment. Country labels reflect participants' residence, not a representative account of a country's culture or every resident's preferences. The English-dominant sample and small country test sets limit generalization. This reward model does not generate text and is not validated for consequential decisions about people.

Attribution and release license

These fine-tuned weights are released under the MIT License. The OpenAssistant base model is marked MIT. The PRISM dataset card lists human text as CC-BY-4.0 and model responses as CC-BY-NC-4.0, subject to original provider terms. This package contains no PRISM records.

Citation

@misc{scpo,
      title={Steerable Cultural Preference Optimization of Reward Models},
      author={Minsik Oh and Advit Deepak and Sophie Wu and Douwe Kiela and Ekaterina Shutova},
      year={2026},
      eprint={2606.18606},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.18606},
}