areai51/propensity-one-lite

Propensity One lite

Research preview. Propensity One lite ranks headline variants for one story by how likely each one is to get the higher click-through rate in a randomised A/B test. It is the text-only CPU part of Propensity One, trained only on the Upworthy Research Archive (CC BY 4.0). Because of that data's licence it can be public.

Try it in the browser: spaces/areai51/propensity-one.

What it is, and what it isn't

• What it is: a ridge regression on word 1-2-gram and character 2-5-gram TF-IDF (78,452 + 63,910 features), trained on within-test contrasts. It learns what makes one headline beat another for the same story, not which stories are popular.
• Not the full Propensity One. The full model adds a fine-tuned ModernBERT-large encoder and, for video ads, cover-image features. It is trained partly on TikTok ad data licensed for research only, so it is not public.
• Trained on news and social-issue headlines from upworthy.com, 2013-2015. It hasn't been validated on product ads, other sites, other audiences or other languages.
• Scores only rank variants against each other. One score on its own means nothing.

Use

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("areai51/propensity-one-lite")
sys.path.insert(0, path)
from propensity_one_lite import PropensityOneLite

model = PropensityOneLite(path)
for r in model.rank(["Headline A", "Headline B", "Headline C"], excerpts="Optional excerpt shown under each"):
    print(r["rank"], round(r["score"], 3), r["p_top_beats"], r["headline"])

It needs numpy 2.4.6, scipy 1.17.1, scikit-learn 1.9.1 and safetensors 0.8.0 (other versions may work). The model loads without pickle: the vocabularies are JSON and the weights are safetensors. The Space runs a JavaScript port of the same scorer in the browser (propensity_one_lite.js). Before release, its scores were checked against Python on every validation and test headline (maximum difference below 1e-9).

Evaluation

These numbers are for the test split of upworthy_brx-pipe-0.2.0: 1,945 randomised tests with 8,654 headline variants. Splits are by family, which groups variants of one story with exact and near-duplicate headlines. So no test story or test headline appears in training. The ridge alpha (30.0) was chosen on validation, and the test numbers were measured once. A sealed holdout is kept for the final release check.

metric (test)	Propensity One lite	random
pairwise accuracy	0.589 [0.577, 0.600]	0.5
pairwise accuracy, weighted by how certain each observed winner is	0.627	0.5
top-1: the top pick is the observed best	0.345	0.242
top-1 on the 323 tests with a clear winner (P(best beats runner-up) >= 0.95)	0.486	0.266

Pairwise accuracy is the share of within-test pairs in which the higher-scored headline had the higher observed click-through rate. On the same groups, a plain TF-IDF ridge scores 0.572, LightGBM on headline features 0.539 and a random score 0.503. For context, the full (private) Propensity One stack scores 0.600 there, and GPT-6.1 Sol, zero-shot, 0.596.

Chance the top pick wins. p_top_beats is sigmoid(k x score difference), with k fitted on validation pairs. On test pairs, the predicted chance that the model's preferred headline wins averages 0.596, and it won 0.596 of the time. Log loss is 0.6583, against 0.6931 for always saying 50%. Most headline differences are small, so the probabilities stay near 50%. That reflects real uncertainty; it isn't a bug.

Training data

The Upworthy Research Archive: randomised headline tests run on upworthy.com in 2013-2015, with impressions and clicks for each variant. We keep tests with at least two headline variants. We drop 2,007 tests where the teaser image varied but wasn't crossed with the headline, so the image could have caused the difference. That leaves 19,268 tests; 61,925 variants are in the training split. The target is the logit of each variant's posterior-mean CTR minus its test's mean, weighted by how many clicks back it up.

Licence and attribution

The model is licensed CC BY 4.0, like its training data. The data is from Matias, J. N., Munger, K., Aubin Le Quere, M., & Ebersole, C. (2021), The Upworthy Research Archive, a time series of 32,487 experiments in U.S. media, Scientific Data 8, 195.

Files

• model.safetensors: idf weights per vectorizer, ridge coefficients and intercept.
• vocab_word.json, vocab_char.json: the vocabularies, in column order.
• config.json: vectorizer settings, pairwise calibration, training snapshot hashes and library versions.
• eval.json: the full validation and test report.
• propensity_one_lite.py: the loader.