Skill: leooooooow/ab-interpreter (version 1.1.0)

--- name: A/B Interpreter description: Interpret A/B test results for ecommerce campaigns and pages by checking statistical significance, practical effect size, and next-step recommendations. --- # A/B Interpreter Most ecommerce A/B tests get called too early, too late, or on the wrong metric — and the team ships whichever variant "looks like" it won without a clean read on whether the lift was real, large enough to matter, or durable past the novelty window. This skill turns raw test results into a disciplined go or no-go verdict with statistical significance, practical effect size, segment checks, and a prescribed next step so every test actually moves the business. ## Quick Reference | Decision | Strong signal | Acceptable | Weak / Redesign | |---|---|---|---| | Statistical significance | p < 0.05, two-tailed, sample powered to 80% | p between 0.05 and 0.10 with large sample | p > 0.10 or sample underpowered | | Practical effect size | Lift exceeds MDE and the lower bound of the 95% CI is positive | Lift exceeds MDE but CI barely crosses zero | Lift below MDE even if "significant" | | Test duration | Ran across at least two full weekly cycles | Ran 10 to 14 days, no major anomalies | Under 7 days or overlaps a holiday/promo | | Sample ratio | Observed split within 1% of planned | Split within 2% | SRM > 2% — suspect tracking | | Guardrail metrics | All guardrails flat or positive | One guardrail flat, one minor regression | Any revenue, refund, or AOV guardrail regressed | | Segment stability | Lift positive across all key segments | Lift positive in 3 of 4 segments | Contradicting segments (new vs returning, device, geo) | | Novelty risk | Lift holds in final week of test | Lift decays mildly over time | Lift front-loaded and decays past week two | ## Solves - Teams ship winners that were noise, then can't reproduce the lift in production - Stakeholders disagree on whether a 3% lift with p = 0.08 is a "win" - Growth managers call tests after 3 days on a heavy-traffic weekend and miss weekday reality - Novelty-effect lifts decay in production and no one notices until the quarter closes - Segment-level contradictions (tablet users regress) get averaged away in the topline - Guardrail metrics like refund rate silently offset the headline conversion win - Test readouts lack a prescribed next step, so learnings do not compound ## Workflow 1. **Ingest the test setup.** Read the hypothesis, variants, traffic split, start and end dates, primary metric, minimum detectable effect (MDE), and guardrail list. Flag any required field that is missing and stop if the primary metric is not defined in advance. 2. **Sanity-check the data.** Confirm the observed traffic split matches the planned split (SRM check). Look for outlier days, tracking gaps, or bot spikes. If the split is off by more than 2%, mark the test inconclusive and recommend re-running. 3. **Run the core statistical check.** Use a two-proportion z-test for conversion rate, Welch's t-test for continuous revenue metrics. Compute the p-value, the observed lift, and a 95% confidence interval around the lift. 4. **Compare lift against MDE.** A significant result that falls below MDE is practically meaningless; a non-significant result that exceeds MDE suggests under-powering and an extension. The lower bound of the CI must exceed zero for a confident ship call. 5. **Check segments and guardrails.** Split by new vs. returning, device, traffic source, and geography. Any segment that contradicts the aggregate direction should be called out. Any guardrail regression (bounce rate, refund rate, add-to-cart) should be weighed against the primary lift. 6. **Assess novelty and seasonality.** If the test ran under 14 days, flag novelty risk. Compare week one to week two. If lift concentrated in week one, recommend extending or running a repeat test after a cooldown. 7. **Deliver the verdict and next step.** Write a plain-language verdict (ship, kill, extend, redesign), the reasoning a skeptical analyst would demand, and the prescribed follow-up test or monitoring metric. Every readout ends with one concrete action. ## Example 1: Shopify PDP button copy test **Inputs.** Hypothesis: "Buy Now" outperforms "Add to Cart" on the product page. 14-day test, 50/50 split, 48,200 visitors each arm. Control conversions: 1,446 (3.00%). Variant conversions: 1,640 (3.40%). MDE set at 0.25 percentage points. Guardrails: refund rate and AOV. **Analysis.** Two-proportion z-test gives p = 0.02 with a 95% CI on the lift of +0.08 to +0.72 pp. Observed lift of 0.40 pp exceeds MDE of 0.25. Sample split was 50.1 / 49.9. New vs. returning both show directional lift. Refund rate flat. AOV down 1.2% (not significant, CI crosses zero). Week one lift 0.45, week two 0.35 — mild decay, within tolerance. **Verdict.** Ship. The result is statistically and practically significant; guardrails are intact. Next step: monitor AOV weekly for four weeks post-rollout and re-test "Buy Now" against a bundled offer CTA in Q3. ## Example 2: Email subject line test with under-powered readout **Inputs.** Two subject lines sent to 8,000 recipients each. Opens: A = 1,440 (18.0%), B = 1,560 (19.5%). MDE of 2 percentage points was required to justify the winner for list-wide rollout. **Analysis.** Two-proportion z-test p = 0.022 — "significant". Observed lift of 1.5 pp is below the 2.0 pp MDE. 95% CI on the lift is +0.2 to +2.8, straddling the MDE. No segment breakdown provided. **Verdict.** Do not ship yet. The result is statistically significant but falls short of the practical threshold the team pre-committed to. Extend the test or re-run with a larger list to narrow the CI. Do not rationalize the "win" just because p < 0.05 — the pre-committed MDE exists precisely to prevent that. ## Common mistakes - **Peeking.** Calling the test early the moment p drops below 0.05 inflates false positive rates dramatically. Pre-commit to a sample size and honor it. - **Ignoring MDE.** A statistically significant but practically tiny lift does not justify a rollout. The MDE is the business's pre-committed threshold. - **No SRM check.** A broken randomizer can fabricate lifts. Always verify the observed split matches the planned split. - **Averaging away segments.** A +2% topline with a -5% tablet regression is not a clean win. Pull the segment cuts before celebrating. - **One-week tests.** Weekday vs. weekend behavior differs, and novelty effects dominate week one. Run at least two full business cycles. - **Mixing up absolute and relative lift.** "20% lift" on a 3% base rate means +0.6 pp, not +20 pp. Always state which you are reporting. - **Treating p-values as probability the hypothesis is true.** A p-value is the probability of the observed data under the null, not the probability the variant is better. - **No guardrails declared upfront.** Retrofitting guardrails after seeing results invites cherry-picking. Name them before the test starts. - **Ignoring post-ship drift.** Novelty decay and seasonality can erode a shipped win. Always prescribe a post-ship monitoring window. ## Resources - `references/output-template.md` — The four-block readout structure - `references/statistical-tests.md` — Choosing the right test for the metric type - `references/segment-playbook.md` — Standard segments to cut and what each contradiction signals - `assets/ab-readout-checklist.md` — Pre-flight checklist for every readout you deliver

path: references/output-template.md
content: # A/B Test Readout Template

Every readout has four blocks. Keep each block short, numbered, and scannable so a skeptical executive can read it in under two minutes.

## Block 1: Verdict

One line. Ship, Kill, Extend, or Redesign. Followed by the single most important reason.

- **Ship** — Significant, above MDE, guardrails intact, segments consistent, novelty decay acceptable
- **Kill** — Significant but regresses a guardrail, or contradicts key segment, or below MDE
- **Extend** — Directionally positive but underpowered or too short
- **Redesign** — Flat or mixed result that suggests the hypothesis was wrong

Example: "Ship. Variant B beats control by 4.1% on conversion with a 95% CI of +1.8% to +6.5%; guardrails intact; all segments align."

## Block 2: Statistics

Report the numbers a peer reviewer would ask for:

- Observed lift (absolute and relative)
- 95% confidence interval on the lift
- p-value and the test used (two-proportion z, Welch's t, chi-squared)
- Sample size per arm and the power actually achieved
- Minimum detectable effect set before the test began
- Duration and whether it covered two full business cycles

## Block 3: Risk

List anything that would give a skeptic pause:

- Sample ratio mismatch (observed vs. planned split)
- Segments that contradict the aggregate direction
- Guardrail regressions, even if not statistically significant
- Novelty risk if lift was front-loaded
- Seasonality, promo overlap, or tracking outages during the window
- External factors: competitor moves, press coverage, paid media spikes

## Block 4: Next step

One concrete action. Never end a readout without one.

Patterns:
- "Roll out to 100% over 48 hours; monitor AOV and refund rate weekly for four weeks"
- "Extend the test to 21 days to narrow the CI below the MDE"
- "Redesign the hypothesis around [specific segment] where lift was concentrated"
- "Kill the variant; queue a follow-up test on [revised hypothesis]"

## Tone notes

Avoid hedging language that obscures the call. "Potentially a modest lift" is not a verdict. "Ship" and "Kill" are. If you cannot commit to a verdict, say "Extend" and state the exact trigger for a re-read.


path: references/statistical-tests.md
content: # Choosing the Right Statistical Test

The test you run depends on the metric shape. Mismatches produce confident-sounding numbers that mean nothing. Use this guide to pick correctly before computing anything.

## Conversion rate, click-through, signup rate

These are proportions — each visitor either converts or does not. Use a **two-proportion z-test** for the difference in rates. Pair it with a Wilson or Newcombe confidence interval on the lift rather than a naive normal approximation, especially for sub-1% rates.

- Required inputs: per-arm visitor count and conversion count
- Reports: absolute lift, relative lift, p-value, CI
- Watch out for: extremely low base rates (under 0.5%) where normal approximations break down

## Revenue per user, average order value

These are continuous and often heavily skewed by a small number of large orders. Use **Welch's t-test** (unequal variances) on the mean per visitor, not per converter. Report median and a bootstrap CI alongside the mean when the distribution has a long tail.

- Required inputs: per-user revenue series for each arm (not just summaries)
- Reports: mean lift, median lift, bootstrap CI, t-test p-value
- Watch out for: one whale order flipping the call — bootstrap helps detect this

## Session length, pages per session, engagement

Also continuous but typically less skewed than revenue. Welch's t-test works. If the distribution is bimodal (bouncers vs. deep engagers), report the breakdown separately before pooling.

## Count data: reviews per purchase, items per cart

Use a **Poisson regression** or a negative binomial test if the variance exceeds the mean. A simple t-test on means is acceptable for large samples but understates variance for small ones.

## Survival / time-to-event

For metrics like time-to-first-purchase or churn, use a **log-rank test** and report a Kaplan-Meier curve. Do not use a t-test on censored data.

## Multiple metrics at once

If you look at 5 metrics, the probability at least one returns p < 0.05 by chance is over 22%. Either pre-commit to a single primary metric, apply a Bonferroni or Benjamini-Hochberg correction, or declare guardrails as non-decision metrics where a regression is disqualifying but a "win" cannot rescue a flat primary.

## Multiple variants

Running 4 variants creates 6 pairwise comparisons. Use a Tukey HSD or Dunnett's test against control rather than running independent two-sample tests.

## When to use Bayesian

If stakeholders want statements like "probability B beats A is 94%," a Bayesian framework with a weakly informative prior gives more intuitive outputs than a p-value. The decision cost structure remains the same — you still need an MDE and guardrails.

## Sample size sanity check

After the test, confirm the achieved power was at least 80% at the MDE. An underpowered test that shows "no significant difference" does not mean the variants are equal — it means you could not detect the effect you cared about.


path: references/segment-playbook.md
content: # Segment Playbook

A topline win that contradicts within a segment is not a clean win. Always cut the following segments and interpret the contradictions.

## Standard cuts to run on every ecommerce test

1. **New vs. returning visitors.** Returning visitors have stronger priors about your brand and respond differently to copy changes. A lift concentrated in new visitors often decays as the audience mix stabilizes.
2. **Device class.** Mobile, tablet, desktop behave differently. A mobile-first test must not regress desktop AOV, and vice versa.
3. **Traffic source.** Paid search, paid social, organic, email, and direct each carry different intent. A lift that only appears in paid social may be a paid-media artifact.
4. **Geography.** Country and — for large enough samples — major metro area. International traffic can have different price sensitivity and payment-method familiarity.
5. **Logged-in vs. guest checkout.** Logged-in users have stored payment methods and prior purchase history.
6. **New product vs. established SKU.** A copy change on a brand-new product may lift for different reasons than on a catalog staple.

## What each contradiction signals

- **Lift in new, flat in returning.** Change is attractive to unfamiliar visitors but neutral to people who already know your brand. Consider rolling out only to new traffic.
- **Lift in mobile, regression on desktop.** Usually a visual or layout issue. Check that your variant renders correctly at every breakpoint.
- **Lift in one traffic source only.** Your test may be picking up a shift in attribution or a difference in landing pages. Rerun with a consistent referrer.
- **Lift in one geography only.** Check currency display, payment methods, and shipping copy. A lift from fixing a localization bug does not generalize.
- **Lift in guests, flat for logged-in.** The change probably reduced friction on a path that logged-in users already skipped. Worth shipping but do not expect equal impact across the base.

## Segment significance thresholds

A segment cut multiplies the number of comparisons. A "regression" in a small segment may just be noise. Apply a Bonferroni correction across your segment tests: divide your alpha (0.05) by the number of segments. Or treat segment cuts as directional guidance rather than hypothesis tests — they should raise flags but not overturn a clean topline call on their own.

## Reporting segments

Present segment results as a small table with observed lift, sample size per arm, and CI. Highlight any segment where the lift direction contradicts the aggregate. Never hide a contradicting segment to make the story cleaner.

## When to stop cutting

If you cut more than six segments you will start finding spurious contradictions. Pre-commit to which segments matter and stop there. Additional cuts can be explored as hypothesis generators for follow-up tests, not as gates on the current ship decision.


path: assets/ab-readout-checklist.md
content: # A/B Readout Quality Checklist

Run this before you deliver any readout. If you cannot check a box, either fix the gap or name it explicitly in the Risk block.

## Setup completeness

- [ ] Hypothesis was written before the test launched
- [ ] Primary metric was named before launch
- [ ] Minimum detectable effect was set before launch
- [ ] Guardrail metrics were listed before launch
- [ ] Sample size target was computed from MDE and power
- [ ] Test duration covers at least two full weekly cycles

## Data hygiene

- [ ] Observed traffic split within 1% of planned (SRM check)
- [ ] No tracking outages during the window
- [ ] Bot traffic filtered
- [ ] Deduplicated across visits (one assignment per visitor)
- [ ] No external promo or press spike overlapped the test
- [ ] Sample size achieved the pre-committed target

## Statistical rigor

- [ ] Correct test type chosen for the metric shape
- [ ] p-value reported with the test name
- [ ] 95% CI on the lift is provided (not just the point estimate)
- [ ] Achieved power at the observed lift is at least 80%
- [ ] Absolute and relative lift both reported, not conflated
- [ ] Multiple-metric correction applied if relevant

## Practical significance

- [ ] Observed lift compared against pre-committed MDE
- [ ] Lower bound of the CI assessed, not just the midpoint
- [ ] Revenue impact estimated under current traffic volume
- [ ] Cost of rollout (engineering, merch) noted if non-trivial

## Segments and guardrails

- [ ] New vs. returning cut
- [ ] Device class cut
- [ ] Traffic source cut
- [ ] Geography cut where sample allows
- [ ] Every guardrail checked and reported
- [ ] Any contradicting segment called out explicitly

## Novelty and durability

- [ ] Week-one vs. week-two lift compared
- [ ] Front-loaded decay flagged if present
- [ ] Seasonality or promo overlap noted
- [ ] Post-ship monitoring window and metric prescribed

## Delivery

- [ ] Verdict is one line: Ship, Kill, Extend, or Redesign
- [ ] Reasoning a skeptic would demand is written out
- [ ] Next step is concrete and owner-assigned
- [ ] Readout is scannable in under two minutes
- [ ] Source data or dashboard link attached for reproducibility


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)