PairwisePM — a mechanical pairwise judge for GenAI product decisions

Two competing GenAI product ideas enter on one questionnaire, on one screen, in one sitting. Out comes the winner, a win probability with an interval, and the specific levers that would flip the verdict. It removes the Tuesdays: the same PM, the same two ideas, the same answer — every day of the week.

▶ Read the visual explainer — brettleehari.github.io/PairwisePM

The whole project and its statistics, explained visually with a live model you can drag: GitHub Pages · mirror on Hugging Face Spaces

Download / mirror : huggingface.co/brettleehari/PairwisePM · Source : github.com/brettleehari/PairwisePM

→ The statistics behind this, explained one concept at a time — noise vs bias, why pairwise beats 1-to-10 scoring, Bradley-Terry as logistic regression on differences, why the default weights are deliberately not fitted, Goldberg's model-of-the-judge, shrinkage centered on the prior, Kendall's ζ, Brier, and the reason the c-statistic is not in the product yet.

Epistemics (read this before trusting a number)

• Priors are a stated, editable policy, not fitted values. The factor weights ship in configs/*.yaml as an explicit, human-readable policy. Edit the YAML, not the code. The model applies your policy more consistently than you can — it does not know better than you.
• 0→1 mode makes no accuracy claim. 0→1 mode structures your search. It does not predict product–market fit. There is no dataset of ideas like yours, so this tool makes no claim about which idea will succeed. The permanent in-app banner says exactly this, and it is not dismissible.
• The LLM's job was at build time — the runtime is weights and a bias term. And the bias term is constrained to zero. No LLM, no API key, no network call at decision time. Runs entirely in this app: no API key, no server, nothing stored anywhere but your own log file.

What it does

• Two modes (a toggle, one screen): 1→N allocation on an established product, and 0→1 search for product–market fit — each with a GenAI-specific factor schema (capability feasibility as eval pass rate, unit economics vs. inference cost, capability-trajectory exposure, …).
• The regression ladder: M0 scores with your stated prior weights from day one; M1 — a ridge logistic Bradley–Terry "model of the judge", shrunk toward your stated weights, not zero — activates at your 10th logged decision and shows you what your revealed policy weighs vs. what you say you weigh.
• Honesty by construction: every probability carries a 90% bootstrap-over-factors interval; an interval straddling 50% renders "too close to call — gather evidence", never a forced winner; evaluative inputs (strategic fit, brand) are captured and displayed but never scored; the 0→1 Ceiling flag routes to human portfolio judgment and never enters the score.
• The leverage panel: signed per-factor contributions, path-to-parity in raw units (interviews, person-weeks, pass-rate steps) proposed only through lever factors and only within plausible ranges, "loses on fundamentals" when no working-harder path exists, and a fragility flag when the margin rests on one soft estimate.
• Your log, your custody: decisions append to a local JSONL you download and keep. The Space stores nothing server-side. ζ (transitivity) and the Brier score accumulate from decision 1; upload your log next session and your history comes back. decisions.sample.jsonl is a worked example log generated by the real engine — upload it to see a populated audit view (it is sample data, not your history).

Install & run locally

pip install .[app]        # engine deps are numpy + PyYAML; gradio is the UI extra
python app.py             # same UI as the Space, fully offline
pip install .[test] && pytest   # engine invariants (no gradio needed)

Repository map

pairwisepm/     engine package: config.py, engine.py, log.py, strings.py
configs/        one_to_n.yaml, zero_to_one.yaml — weights are config, not code
app.py          Gradio UI (all math lives in the package, zero math here)
tests/          pytest suite for the engine invariants + the compliance spec
decisions.sample.jsonl   worked example log (real engine output, sample data)

Log record schema (v1)

One JSON object per line: schema_version, timestamp, mode, config_fingerprint (sha256 of the canonical factor config, so audits can be partitioned by policy version), both ideas' raw and z factor vectors, evaluative (captured, never scored), flags, pick, model (M0 and M1 probabilities + intervals + verdict), override_rationale, and an outcome slot that stays null until you label it later — the schema is ready for outcome-based calibration (M2) with no migration. Readers must ignore unknown keys; records without config_fingerprint (pre-v1.1) remain valid.