Skill: quochungto/bookforge-value-testing-technique-selection (version 1.0.0)

--- name: value-testing-technique-selection description: "Select and execute the right value testing technique for product discovery. Use when you have a prototype and need to know if customers will actually choose or buy the product, when deciding between a fake door test, usability test, A/B test, or invite-only program, when someone asks 'how do we test if users want this?' or 'should we run an A/B test?', or when setting up analytics instrumentation before launch. Also use when validating demand before building anything, when choosing between qualitative vs. quantitative value testing, or when the team is unsure whether 'people said they liked it' is enough evidence. Covers the 3-level value testing hierarchy (demand / qualitative / quantitative), the usability-then-value session protocol, and the analytics instrumentation checklist. For prototype selection, use discovery-prototype-selection. For finding product-market fit with reference customers, use customer-discovery-program. For stakeholder viability sign-off, use business-viability-stakeholder-testing." version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/inspired-how-to-create-tech-products/skills/value-testing-technique-selection metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: published source-books: - id: inspired-how-to-create-tech-products title: "INSPIRED: How to Create Tech Products Customers Love" authors: ["Marty Cagan"] chapters: [50, 51, 52, 53, 54] tags: [product-management, product-discovery, user-research, testing] depends-on: [product-discovery-risk-assessment, discovery-prototype-selection] execution: tier: 1 mode: hybrid inputs: - type: document description: "Risk assessment output or product/feature description including traffic volume, risk tolerance, and prototype readiness" tools-required: [Read, Write] tools-optional: [] mcps-required: [] environment: "Document-based product management environment" discovery: goal: "Select the correct value testing technique and produce an executable test plan" tasks: - "Apply the 3-level value testing hierarchy to select the appropriate technique" - "Design and run a usability test session as a prerequisite to value testing" - "Execute the specific value test (demand, qualitative, or quantitative)" - "Select among A/B, invite-only, and customer discovery program for quantitative work" - "Define the analytics instrumentation plan for the feature" audience: roles: [product-manager, product-designer, tech-lead, startup-founder] experience: any when_to_use: triggers: - "You have a high-fidelity prototype and need to know if customers will choose the product" - "Risk assessment has identified value risk as Medium or High" - "You need to decide whether to use A/B testing, invite-only, or customer discovery program" - "You are launching a new feature and need to define what analytics to instrument" - "Sales is hearing demand signals but you have no quantitative evidence" prerequisites: - "product-discovery-risk-assessment completed (value risk identified)" - "discovery-prototype-selection completed (prototype type selected)" not_for: - "Usability testing alone — this skill uses usability testing as a prerequisite step, not the goal" - "Feasibility testing — use engineer-led feasibility spikes" - "Business viability review — use stakeholder review process" environment: codebase_required: false codebase_helpful: false works_offline: true quality: scores: with_skill: 0 baseline: 0 delta: 0 tested_at: "" eval_count: 0 assertion_count: 0 iterations_needed: 0 --- # Value Testing Technique Selection ## When to Use Apply this skill when you have evidence of a product idea worth testing and need to determine whether customers will actually choose it. This skill answers three questions: 1. Which level of value testing does this situation call for — demand, qualitative, or quantitative? 2. How do I execute the right session protocol? 3. What analytics must be instrumented before or alongside any test? Do NOT use this skill as a substitute for usability testing alone, feasibility spikes, or business viability stakeholder review. Those are handled by separate skills. This skill assumes a prototype exists or can be built quickly. **The core insight:** Just because someone can use your product doesn't mean they will choose to use it. Feature parity is not enough. Customers must perceive your product as substantially better to endure the switching costs from their current solution. ## The 3-Level Value Testing Hierarchy Before selecting any technique, apply this hierarchy in order. Start at Level 1 and only advance if the prior level is satisfied. | Level | Question Answered | Technique | When to Use | |-------|------------------|-----------|-------------| | **1 — Demand** | Do customers care about this problem enough to act? | Fake door test / landing page demand test | Unknown if customers want this at all; new product or feature; before building anything | | **2 — Qualitative** | Do customers see real value when they experience it? Why or why not? | Interview + usability + specific value tests | You know demand exists but need to understand whether your solution is compelling and why | | **3 — Quantitative** | Is there statistically meaningful behavioral evidence of value? | A/B test / invite-only / customer discovery program | Usability and qualitative value are confirmed; you need evidence to justify full build investment | **Why this ordering matters:** Level 1 prevents wasted engineering effort on solutions nobody wants. Level 2 explains what Level 3 will measure and provides the "why" that quantitative data alone cannot. Skipping to Level 3 without Level 2 leaves you with data that tells you something is wrong but not how to fix it. ## Step 1: Determine Which Level to Enter Read the risk assessment and answer these questions in order: **Is demand established?** - No clear evidence anyone wants this → start at Level 1 (demand testing) - Demand signals exist (sales reports, customer interviews, competitor market) → skip to Level 2 **Is qualitative value confirmed?** - Have fewer than 5 customers experienced the solution and responded positively → start at Level 2 - 5+ users have tested the prototype and shown genuine value signals → consider Level 3 **What is the risk/traffic profile?** - New product or startup with low traffic → Level 1 or Level 2; skip quantitative until demand exists - Established product with traffic but high risk sensitivity → Level 3 invite-only or customer discovery program - Established product with significant traffic and moderate risk tolerance → Level 3 A/B test **WHY:** Jumping to A/B testing before qualitative value is confirmed produces data that shows whether a product works, but provides no insight into why it doesn't work or how to fix it. Conversely, stopping at qualitative testing without moving to quantitative leaves the team making claims that cannot be defended to stakeholders. ## Step 2: Execute the Selected Level ### Level 1 — Demand Testing Use a fake door test (for features on an existing product) or a landing page demand test (for new products). **Fake door test:** 1. Add the button or menu item to the product interface exactly where users would expect to find it 2. When clicked, redirect to a page explaining you are studying the possibility of adding this feature and inviting them to volunteer for a conversation 3. Collect email addresses or phone numbers from volunteers 4. Measure the click-through rate and compare to baseline feature interaction rates 5. Follow up with volunteers for qualitative interviews **Landing page demand test:** 1. Build a landing page that describes the new offering exactly as if it were live 2. Drive traffic via existing channels, search engine marketing, or email 3. When users click the call to action, redirect to a page explaining you are studying this offering and would like to speak with them 4. Collect volunteer contact information and measure click-through rate **What success looks like:** Meaningful click-through rate relative to other features/products, and a list of users actively willing to discuss the problem. The demand test does not prove customers will pay — only that they care enough to investigate. **WHY:** Building a complete product only to discover nobody wants it is the most avoidable failure in product development. Demand testing takes hours to days, not months. The fake door specifically reveals whether real users in actual context care about this, not just users asked hypothetically in a survey. See `references/qualitative-value-test-protocol.md` for the follow-up interview protocol after collecting volunteer contacts. ### Level 2 — Qualitative Value Testing This is the single most important discovery activity for most product teams. Run at minimum two to three qualitative value sessions per week during active discovery. **The session structure has four parts, always in this sequence:** **Part A — Interview First (5–10 minutes)** Confirm the user has the problem you think they have. Ask how they solve it today and what it would take for them to switch. This grounds the value test in actual context. **WHY:** If the user does not actually have the problem, their response to your solution is meaningless. The interview prevents you from treating a non-customer as a customer. **Part B — Usability Test (15–20 minutes)** Before testing value, the user must understand how to use the product. Run a full usability test first. The value test is only valid once the user has operated the prototype and understands what it does. **WHY:** Without usability testing first, a value session becomes a focus group where users comment hypothetically on something they have never operated. Focus groups produce polite opinions that do not predict behavior. Operate the prototype first, then test value. See `references/usability-test-protocol.md` for the complete four-phase usability test protocol. **Part C — Specific Value Tests (10–15 minutes)** After usability, apply one or more of these four value tests. They are designed to detect genuine enthusiasm, not polite agreeableness. | Value Test | What to Do | Signal | |-----------|-----------|--------| | **Money test** | Ask if they would pay for it right now; request credit card or letter of intent | Willingness to make a financial commitment | | **Time test** | Ask if they would schedule significant time with you to work on this | Willingness to invest their most scarce resource | | **Access test** | Ask for login credentials to the competing product they would switch from | Willingness to commit to switching right then | | **Referral test** | Ask for Net Promoter Score (0–10 likelihood to recommend); ask for their boss's or colleague's email | Willingness to stake their reputation on this | Use whichever tests fit the context. For consumer products, money and referral tests are most practical. For business products, all four apply. The goal is not to collect the credit card or email — it is to observe whether the user is willing to make a commitment, which reveals genuine perceived value versus politeness. **Part D — Iterate the Prototype** As soon as you see a problem or want to try a different approach, update the prototype and test it in the next session. There is no law that says you must keep the test identical across all subjects. Qualitative testing is about rapid learning, not proving anything. **Cadence:** Target two to three sessions per week throughout discovery. As product manager, you must be present at every session. Do not delegate this. ### Level 3 — Quantitative Value Testing Quantitative testing collects behavioral evidence. Use it to prove that a solution works, not to discover what to build. The technique depends on your traffic volume and risk tolerance. **Technique selection:** | Situation | Technique | Why | |-----------|-----------|-----| | High traffic, moderate risk tolerance | **A/B test at 1% exposure** | Gold standard; users don't know which version they see; most predictive data | | Lower traffic or higher risk aversion | **Invite-only test** | Identifies a willing set of early adopters; less predictive due to opt-in bias, but generates real usage data | | Maximum risk sensitivity (regulated, enterprise, brand-sensitive) | **Customer discovery program** | Participants under non-disclosure agreement; pre-existing close relationship; most conservative | **A/B testing (discovery variant):** Run the live-data prototype against the current product. Show the live-data prototype to 1% or fewer of users. Monitor closely. This differs from optimization A/B testing (which tests surface-level changes at 50/50 splits). Discovery A/B testing tests fundamentally different approaches with minimal user exposure. **Invite-only testing:** Identify users to contact directly and invite them to try an experimental version. They know it is experimental, so they are effectively opting in. The data is less predictive than a blind A/B test because early adopters are not representative of the full user base. When quantitative results are negative, follow up immediately with qualitative sessions to understand why. **Customer discovery program:** Use your existing customer discovery program members. They have already opted in to testing new versions and are under a non-disclosure agreement. For business-to-business products, this is the primary recommended quantitative technique during discovery. Compare their usage data to the broader customer base. **WHY for technique selection:** A/B tests produce the most predictive data because users are blind to the test. But they require sufficient traffic to achieve statistical significance, and they expose some users to potentially inferior experiences. Invite-only and customer discovery program reduce exposure risk at the cost of predictive power. Match the technique to what your traffic and risk profile can actually support. ## Step 3: Analytics Instrumentation (Non-Negotiable) Every feature shipped must have usage analytics instrumented before or at launch. Not doing this is the "flying blind" anti-pattern — a non-negotiable failure. **The flying blind anti-pattern:** Teams that ship features without instrumentation cannot know whether those features are working, whether users are adopting them, or whether they should be removed. This prevents data-informed decisions and makes roadmap prioritization guesswork. **Rule:** If you are not willing to instrument a feature to know immediately whether it is working and whether there are significant unintended consequences, do not ship it. **Minimum instrumentation checklist for any new feature:** - [ ] Feature adoption rate (are users finding and using it?) - [ ] Task completion rate (are users completing the intended workflow?) - [ ] Error/abandonment points (where are users dropping off?) - [ ] Impact on key business metric (conversion, retention, revenue, or the metric the feature was designed to move) **Broader analytics strategy:** Product managers must track analytics across all seven categories. Most teams only track user behavior and miss the rest. | Category | Examples | |----------|---------| | User behavior | Click paths, feature engagement, session flow | | Business | Active users, conversion rate, lifetime value, retention | | Financial | Average selling price, billings, time to close | | Performance | Load time, uptime, latency | | Operational | Storage, hosting costs | | Go-to-market | Acquisition cost, cost of sales | | Sentiment | Net Promoter Score, customer satisfaction, survey responses | See `references/analytics-strategy.md` for the full five uses of analytics and implementation guidance. ## Outputs After completing this skill, produce: ``` # Value Test Plan: [Feature / Product Name] ## Level Selected [ ] Level 1 — Demand [ ] Level 2 — Qualitative [ ] Level 3 — Quantitative Rationale: [why this level] ## Technique Selected [Fake door / Landing page / Qualitative session / A/B test / Invite-only / Customer discovery program] ## Test Plan [For demand: where the fake door appears, what the redirect page says, how CTR will be measured] [For qualitative: session structure, value tests selected, target users, weekly cadence] [For quantitative: traffic %, exposure duration, primary metric, statistical significance threshold or evidence threshold] ## Usability Test Required First? [ ] Yes — complete usability-test-protocol before value test [ ] No — user already understands the product ## Value Tests to Apply (qualitative) [ ] Money test [ ] Time test [ ] Access test [ ] Referral test ## Analytics Instrumentation Plan [List the specific events and metrics to instrument before launch] ## Success Criteria [What signal means this test produced sufficient evidence to move to next level or to full build?] ## Next Step [If positive: proceed to Level 3 / proceed to full build / present to stakeholders] [If negative: iterate prototype / reconsider problem framing / stop and preserve capacity] ``` ## Key Anti-Patterns **Prototype-as-value-proof:** Showing a high-fidelity prototype to users who say they love it does not validate value. People are polite. Use the specific value tests (money, time, access, referral) to detect genuine commitment, not stated enthusiasm. **Flying blind:** Shipping features without instrumentation. Non-negotiable failure. Every feature needs basic usage analytics before launch. **Skipping qualitative before quantitative:** Running an A/B test before qualitative value is confirmed produces data that tells you something is wrong but not how to fix it. **Focus group substitution:** Asking users to comment on a prototype they have never operated produces hypothetical opinions, not behavioral insight. Always run usability testing before value testing. **Delegating qualitative sessions:** As product manager, you must attend every qualitative value test. The firsthand exposure to customer reactions is a core part of the role and cannot be substituted with written summaries. ## References - `references/usability-test-protocol.md` — Complete four-phase usability test protocol (recruit, prepare, test, summarize) with the parrot technique and use-mode guidance - `references/qualitative-value-test-protocol.md` — Full qualitative value session structure including interview-first approach, all four specific value tests, and iteration cadence - `references/analytics-strategy.md` — Five uses of analytics in product teams, seven analytics categories, flying blind anti-pattern detail, and instrumentation implementation guidance ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — INSPIRED: How to Create Tech Products Customers Love by Marty Cagan. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-product-discovery-risk-assessment` - `clawhub install bookforge-discovery-prototype-selection` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/analytics-strategy.md
content: # Analytics Strategy for Product Teams

Product managers today are expected to be comfortable with data and understand how to leverage analytics to learn and improve quickly. This is not optional. The teams that fly blind — shipping features without instrumentation — consistently make worse product decisions and cannot justify their choices to stakeholders.

## The Flying Blind Anti-Pattern

Remarkably, too many product teams either do not instrument their products to collect analytics at all, or they do it at such a minor level that they cannot determine if and how their product is being used.

**The anti-pattern:** Ship a feature, see no instrumentation data, guess whether it is working, argue about it in meetings, and eventually either leave it forever or remove it based on opinion rather than evidence.

**The consequence:** Without usage data, you cannot:
- Know whether a feature is being adopted
- Identify which features are unused and should be removed
- Understand whether a change improved or hurt the metric it was meant to affect
- Defend roadmap decisions to leadership and stakeholders
- Know what actually happened when something goes wrong

**The rule:** If you would not be willing to instrument a feature to know immediately whether it is working and whether it is producing significant unintended consequences, do not ship it. Start from the position that you must have this data, then work backward to figure out the best way to get it.

For every new feature, ensure instrumentation is in place before launch. For most cloud-based products, this is straightforward. For installed desktop or mobile applications, the product calls home periodically with anonymized, aggregated usage data. Always anonymize and aggregate to avoid personally identifiable information.

## The Five Uses of Analytics

Strong product teams use analytics for five distinct purposes. Many teams only use the first. Using all five represents a significant competitive advantage.

### Use 1: Understand User and Customer Behavior

When most people think of analytics, they think of user behavior analytics. This is the most common use, but it is only the beginning.

The goal is to understand how users actually use your products versus how you assumed they would use them. The gap between assumption and reality is where product insight lives.

Applications:
- Identify features that are not being used (candidates for removal or redesign)
- Confirm that features are being used as expected
- Understand click paths through complex workflows
- See the difference between what users say they do and what they actually do

This is one of the very few non-negotiables in product: if you add a feature, you need to put in at least basic usage analytics for that feature. How else will you know if it is working?

### Use 2: Measure Product Progress

Rather than measuring progress by features shipped (output), measure progress by business outcomes (outcomes).

Strong product teams receive business objectives with measurable goals. They use analytics to track whether they are making progress against those objectives. This keeps teams focused on outcomes rather than shipping lists.

Application: Define the metric you are trying to move with each initiative. Track that metric before, during, and after the change to measure whether the initiative worked.

### Use 3: Prove Whether Product Ideas Work

For products with sufficient traffic, analytics can isolate the contribution of specific features, workflow changes, or design decisions.

This is the role of A/B testing in the discovery context — not optimization testing (testing button colors at 50/50 splits), but discovery testing (showing a live-data prototype to 1% of users to measure behavioral evidence of value).

Even when traffic volume makes statistically significant results difficult to achieve, actual usage data from live-data prototypes produces better-informed decisions than opinions alone.

Application: For high-risk or high-cost initiatives, require quantitative evidence before committing full engineering investment.

### Use 4: Inform Product Decisions

The worst product decisions are made based on opinion, and typically the most senior person's opinion wins. Data beats opinions.

When a stakeholder argues that a feature must be built because customers have asked for it, analytics can show how many customers actually use the related capabilities today. When someone argues that removing a feature will cause customer outrage, analytics can show how many customers have used that feature in the last 90 days.

Data does not make decisions for you. But it changes the quality of decisions by replacing assertion with evidence.

### Use 5: Inspire Product Work

This is often the most underappreciated use of analytics. The data itself — when explored with curiosity — can reveal product opportunities that no customer interview or stakeholder request would surface.

When you examine how users actually use your product, you encounter surprises. You discover workflows that contradict your design assumptions. You find features used in ways you never intended. You see patterns that suggest entirely new capabilities.

Some of the best product ideas are not requested by customers or proposed by engineering — they emerge from studying the data and asking "why is this happening?"

## The Seven Analytics Categories

Most teams track user behavior and one or two other categories. Strong product managers track across all seven.

| Category | What It Measures | Examples |
|----------|-----------------|---------|
| **User behavior** | How users interact with your product | Click paths, feature engagement, session flow, task completion |
| **Business** | Product-level business health | Active users, conversion rate, lifetime value, retention, churn |
| **Financial** | Revenue and deal metrics | Average selling price, billings, time to close, expansion revenue |
| **Performance** | Technical product quality | Load time, uptime, latency, error rate |
| **Operational** | Infrastructure costs | Storage consumption, hosting costs, API call costs |
| **Go-to-market** | Customer acquisition efficiency | Acquisition cost, cost of sales, program ROI |
| **Sentiment** | Customer perception and satisfaction | Net Promoter Score, customer satisfaction surveys, support ticket themes |

Most product managers have too narrow a view, focusing only on user behavior and business metrics. The other five categories often contain signals that explain why business metrics are moving in unexpected directions.

For example: a drop in conversion rate (business metric) might be explained by a spike in load time (performance metric) or a change in acquisition source (go-to-market metric). Without all seven categories, you see the symptom but not the cause.

## Minimum Instrumentation for a New Feature

Before launching any feature, confirm instrumentation is in place for:

1. **Adoption:** Is the feature being discovered and used? (feature exposure rate, activation rate)
2. **Task completion:** Are users completing the intended workflow? (completion rate, abandonment point)
3. **Error / friction:** Where do users encounter problems? (error events, rage clicks, abandonment steps)
4. **Outcome impact:** Is this feature moving the metric it was designed to affect? (the specific business, financial, or behavioral metric the feature was meant to influence)

For anything with higher risk or higher deployment cost, also add:
- Comparison baseline (what was the metric before the change?)
- Unintended consequence monitoring (are any adjacent metrics moving unexpectedly?)

## Analytics and Qualitative Testing: The Complete Loop

Analytics tells you what is happening. Qualitative testing tells you why.

These two types of learning are not alternatives — they are complements. The complete loop:

1. Analytics reveals an unexpected pattern (adoption is lower than expected; a workflow step has high abandonment; a feature is unused)
2. Qualitative testing (user interviews, value test sessions, usability sessions) explains why
3. The team forms a hypothesis, updates the prototype, and tests it
4. Analytics measures whether the change fixed the problem

Neither analytics alone nor qualitative testing alone is sufficient. Product teams that rely only on analytics optimize metrics without understanding customer motivation. Teams that rely only on qualitative testing cannot validate whether their changes actually worked at scale.

## Implementation Notes

- Analytics are often referred to as key performance indicators (KPIs) in organizational contexts. The categories above map to KPIs used in executive reporting.
- For strict firewall environments (regulated industries, enterprise on-premise software), products can generate periodic usage reports that are reviewed and approved before being forwarded electronically or in print. Even the most conservative environments can implement some form of product instrumentation.
- Use existing web analytics tools where applicable. For cases requiring custom tracking, the process is: (1) define what you need to know, (2) instrument the product to collect it, (3) generate reports for ongoing review.
- The first thing most strong product managers do in the morning is check the analytics from the preceding night's test results. This cadence — daily review of active experiments — is how they stay informed and responsive.


path: references/qualitative-value-test-protocol.md
content: # Qualitative Value Test Protocol

Qualitative value testing is, in Cagan's view, probably the single most important discovery activity for a product team. The goal is rapid learning and big insights — understanding whether customers see real value in your solution, and why or why not.

Quantitative testing tells you what is happening. Qualitative testing tells you why. Both are necessary. Qualitative testing comes first.

**Cadence target:** At minimum two to three qualitative value test sessions per week during active discovery.

**Who must attend:** Product manager, product designer, and at minimum one rotating engineer. The product manager must be present at every session without exception. Do not delegate this. Do not hire a firm to do this for you. The firsthand exposure to user reactions is a core competency of the product manager role.

## The Problem: Politeness Bias

The central challenge in qualitative value testing is that people are generally nice. When sitting face to face with someone who has built something, most people are not willing to tell you what they really think. They give polite, encouraging feedback that does not reflect their actual likelihood to use or pay for the product.

All value tests in this protocol are designed to cut through politeness bias by asking users to make a commitment — not just state an opinion. Commitments (money, time, access, reputation) reveal genuine value perception far more accurately than stated enthusiasm.

## Session Structure

Run each session in four sequential parts. Do not reorder them.

### Part 1: Interview First (5–10 minutes)

Open every session with a short customer interview before showing the prototype.

**Goal:** Confirm that this user actually has the problem you are solving, understand how they currently solve it, and learn what it would take for them to switch.

**Questions to ask:**
- "Tell me about how you currently [handle the problem area]."
- "What's the most frustrating part of that process?"
- "What tools or approaches have you tried?"
- "What would make you consider switching to something new?"

**Why this comes first:** If the user does not actually have the problem, their response to your solution is not useful data. You need to know whether you are talking to a genuine potential customer before interpreting anything they say about the product. This also creates context — after discussing their current approach, users evaluate your solution relative to their actual situation rather than abstractly.

**If the user does not have the problem:** Note this, thank them for their time, and consider whether your recruitment screening criteria need adjustment. Do not proceed with the value test — the data would be invalid.

### Part 2: Usability Test (15–25 minutes)

Before testing value, the user must understand how to use the product. Run a full usability test on the prototype.

**Why usability testing precedes value testing:**

If you attempt value testing without usability testing first, the session becomes a focus group. The user comments hypothetically on something they have never operated — imagining how it might work, asking clarifying questions, making up scenarios. Focus groups provide market insight but do not predict whether your product will work. You need users to have actually operated the product to have a meaningful conversation about value.

After a usability test, the user knows what the product is, how it works, and what it is meant to do. Only then can you have a credible conversation about whether they see value in it.

Follow the usability test protocol in `usability-test-protocol.md`. Use a high-fidelity prototype — value testing requires realism. A live-data prototype or hybrid prototype also works.

**Transition:** After the usability test, the user is in a position to have an informed conversation about the product. Move immediately to specific value tests.

### Part 3: Specific Value Tests (10–15 minutes)

Apply one or more of the four specific value tests. Choose based on what fits the context (consumer vs. business, product type, relationship).

**Value Test 1 — Money**

Ask the user if they would be willing to pay for the product right now.

For consumer products or lower-cost software: "Would you be willing to put in your credit card right now? We don't actually need it, but I'm curious if this is something you'd pay for."

For expensive business products (beyond what an individual would put on a credit card): "Would you be willing to sign a non-binding letter of intent to purchase this if we build it?"

**What the signal means:** You are not collecting the credit card or the letter. You are observing whether the user is willing to make a financial commitment. Willingness to do so — even when told it's not required — reveals genuine perceived value. Refusal or hesitation, even after expressed enthusiasm, reveals politeness bias at work.

**Value Test 2 — Time**

Especially effective for business products:

"Would you be willing to schedule significant time with us — say, a few hours per week over the next month — to work on this with us? We don't necessarily need it, but I'm curious if this is important enough for you to invest that kind of time."

**What the signal means:** Time is the scarcest resource in business. A user's willingness to commit time — even hypothetically — is a strong indicator of how much they value the solution. Businesses that actually have the problem will recognize the value of solving it and will be willing to invest.

**Value Test 3 — Access**

"Would you be willing to give us the login credentials for the product you're currently using for this? We have a migration utility that would help import your data. We don't really need it now, but I'm curious if you'd be ready to switch."

**What the signal means:** Handing over access credentials to a product they are actively using is a significant commitment. It reveals genuine readiness to switch, not just openness to switching someday. Users who are only mildly interested will not do this. Users who genuinely see this as a substantially better solution will.

**Value Test 4 — Referral**

Two variants, use one or both:

Net Promoter Score variant: "On a scale of 0 to 10, how likely would you be to recommend this product to a friend or colleague?"

Commitment variant: "Would you be willing to enter your boss's email address or a colleague's email address so we could tell them about this? We wouldn't actually email them without asking you first — I'm just curious if this is something you'd be willing to put your name behind."

**What the signal means:** Willingness to share on social media, provide a boss's email, or give a high Net Promoter Score indicates that the user is willing to stake their reputation on recommending this product. This cuts through politeness more effectively than asking "Would you use this?" — people are far more careful when their professional reputation is involved.

### Part 4: Iterate the Prototype

After each session (or set of sessions), update the prototype based on what you learned.

**There is no requirement to keep the test identical across all subjects.** The goal is rapid learning, not statistical proof. If session two reveals that users are confused by the navigation, fix the navigation before session three.

**Decision rules:**

- If responses are substantially different across users, investigate why. You may have different customer segments with different problems, or your solution may work for some use cases but not others.
- If you cannot get users interested in the problem or cannot make the solution usable enough that they perceive value, consider stopping and putting the idea on the shelf. This is not failure — it is the discovery process preventing engineering investment in something customers will not value.
- If users consistently show genuine value signals (willingness to pay, invest time, provide access, or recommend), proceed to the next steps: quantitative validation or full build planning.

## Session Notes Template

Use this template to capture findings immediately after each session:

```
Session Date: 
User Profile: [role, company size/type, current solution]
Problem Confirmed: [ ] Yes  [ ] No  [ ] Partial

Usability Issues Observed:
- [Issue 1 — where it occurred, what the user did]
- [Issue 2...]

Value Test Results:
- Money test: [ ] Strong  [ ] Lukewarm  [ ] Declined  [ ] Not administered
- Time test: [ ] Strong  [ ] Lukewarm  [ ] Declined  [ ] Not administered
- Access test: [ ] Strong  [ ] Lukewarm  [ ] Declined  [ ] Not administered
- Referral test: NPS score: __  Willing to provide email: [ ] Yes  [ ] No
  
Verbal Signals: [What did the user say that was most revealing?]
Behavioral Signals: [What did the user do that was most revealing?]
Body Language / Tone: [Did enthusiasm match stated opinion?]
Surprising Observations: 
Prototype Changes Before Next Session:
Overall Value Signal: [ ] Strong  [ ] Mixed  [ ] Weak  [ ] Negative
```

## Email Summary Format

After each session or set of sessions, the product manager or designer sends a brief email to the product team:

**Subject:** Value test learnings — [Feature Name] — [Date]

**Body:**
- What worked (tasks completed, strong value signals)
- What didn't work (friction points, weak value signals)
- What surprised us
- Changes being made to the prototype
- Next session scheduled: [date/time]

Keep it to one page or less. Long reports are obsolete before they are read.


path: references/usability-test-protocol.md
content: # Usability Test Protocol

This document provides the complete four-phase protocol for running usability tests as part of product discovery. In the value testing context, usability testing is always a prerequisite to qualitative value testing — users must understand how to operate the product before their value reactions are meaningful.

## Why Usability Testing in Discovery

Traditional usability testing happened at the end of development, after the product was built. By then, correcting serious issues required significant waste or produced a permanently flawed product.

Modern discovery practice runs usability testing on prototypes — before anything is built — so that issues are corrected at near-zero cost. The main difference is the timing: discovery usability testing uses prototypes, not products.

## Phase 1 — Recruit

You need at minimum 5 test subjects. Research consistently shows that 5 users reveal the majority of serious usability issues.

**Recruitment channels (in order of preference):**

1. **Customer discovery program members** — Pre-qualified, already opted in, best for business products. Use this if you have one.
2. **Email list selection** — If you have a user email list, work with your product marketing manager to select a screened subset matching your target customer profile.
3. **Company website volunteer form** — Solicit test subject volunteers. Screen all volunteers with a phone call to confirm they are in your target market.
4. **Craigslist or search engine marketing campaign** — Especially effective for reaching users who are actively in the moment of experiencing the problem your product solves (for example, a Google AdWords campaign targeting people searching for your problem category).
5. **Intercept at locations where target users congregate** — Trade shows for business software, shopping centers for e-commerce, sports bars for fantasy sports. Bring thank-you gifts.

**Compensation:** If asking users to come to your location, compensate them for their time. A common approach is to meet at a mutually convenient location — a coffee shop works well. This is so common it is called Starbucks testing.

**Screening requirement:** Always screen volunteers by phone before the session. Confirm they match your target customer profile. Volume is less important than quality — five well-matched users are more valuable than fifteen mismatched ones.

## Phase 2 — Prepare

**Prototype:** Use a high-fidelity user prototype for both usability testing and value testing. You can get useful usability feedback from lower-fidelity prototypes, but the value testing that follows requires realism. Build to the fidelity level that the downstream value test demands.

**Attendees:** Product manager, product designer, and at minimum one engineer (rotating among team members). All three must be present. The product manager and designer must attend every session without exception. Engineer attendance is strongly encouraged — the direct exposure to customer behavior produces learning that written summaries cannot replicate.

**Role division:**
- One person administers the test (interacts with the user)
- One person takes notes
- After the session, both debrief to confirm they observed the same things and reached the same conclusions

**Task definition:** Define in advance the specific tasks you want to test. Focus on the primary tasks — the ones users will do most of the time. There may also be less-frequent but important tasks; include those if they are risky. Do not try to test everything in one session.

**Example task list for a report generation feature:**
- Generate a report for last quarter's sales
- Change the date range on a report
- Share a report with a colleague
- Export a report to a spreadsheet

**Venue:**
- Coffee shop table — informal, user feels less like a lab rat, works well for most situations
- Customer's office — excellent for business products; environmental cues (monitor size, desk setup, coworker interruptions) reveal additional context
- Formal testing lab with two-way mirror — useful when available, but not required
- Remote tools — acceptable supplement for usability testing, but less suitable for the value testing that follows (value testing benefits from in-person presence)

## Phase 3 — Test

### Opening the Session

Before presenting any tasks:

1. **Set expectations:** Tell the user this is a prototype, it is a very early product idea, and it is not real. Explain that you are testing the ideas in the prototype, not testing them. They cannot pass or fail — only the prototype can. You want candid feedback, good or bad.

2. **Landing page cold read:** Before jumping into tasks, ask the user to look at the landing page or home screen of your prototype and tell you what they think it does and what might be valuable or appealing to them. Do this before they start interacting with tasks — once they are in task mode, you lose the first-time visitor perspective. Landing pages are critical for bridging the gap between user expectations and what the product actually does.

### During the Test

**Keep users in use mode, not critique mode.**

The goal is to observe what users can and cannot do. It does not matter whether the user thinks something is ugly or that a button is in the wrong place. What matters is whether they can complete the tasks they need to do.

Avoid asking questions like "What three things would you change about this page?" Unless the user is a product designer, their answer to that question is not useful. Watch what they do more than what they say.

**Stay quiet.**

The hardest skill in usability testing is silence. When a user struggles, the natural instinct is to help. Suppress that instinct. Get comfortable with silence — it is your most important tool. Struggle reveals friction; helping the user removes the signal.

**The three outcomes for each task:**
1. User completed the task with no problem and no help — good
2. User struggled but eventually completed the task — friction detected
3. User gave up — failure; note whether they truly abandoned or were about to leave the product entirely

**The parrot technique:**

When you feel compelled to say something (either because the silence is uncomfortable or because you want to prompt the user), parroting is the tool:

- If the user is scrolling and looking confused, say: "I see you're looking at the list on the right." This prompts them to tell you what they are trying to find without leading them toward an answer.
- If the user asks "Will clicking this create a new entry?", respond: "You're wondering if clicking this will create a new entry?" The user will typically answer their own question. This avoids leading the witness.
- Use parroting instead of value judgments. Instead of "Great!" when the user completes a task, say "You created a new entry." This avoids leading value judgments while still acknowledging their action.
- Parroting gives the note-taker more time to write down observations.

**Avoid leading the witness:**

Do not give hints. Do not redirect attention. If you see the user scrolling past the right answer, do not help. What you are observing is exactly the information you need — that the affordance is not discoverable.

**Reading non-verbal signals:**

Users communicate through body language and tone. Genuine enthusiasm is visible — they lean forward, ask follow-up questions, and often ask when the product will be available. Lack of genuine enthusiasm is also visible even when users say polite things. Watch for the gap between what users say and what their body language and engagement level communicate.

## Phase 4 — Summarize

**Immediately fix issues you identify.**

There is no rule that says you must keep the test identical across all subjects. Qualitative usability testing is about rapid learning, not proving anything statistically. As soon as you identify an issue, fix it in the prototype before the next session. This is the power of discovery usability testing — fast iteration at low cost.

**The email summary:**

After each test session (or after each set of sessions if running multiple in a day), the product manager or designer writes a short email summary of key learnings and sends it to the product team. Keep it brief — one page or less. Long reports that take hours to write and days to read are obsolete by the time they are delivered. The prototype will have moved past what was tested.

The email summary should include:
- What tasks users completed successfully
- Where users struggled or failed
- What surprised you
- What changes are being made to the prototype before the next session

**What you are looking for:**

The goal is to identify places in the prototype where the model the software presents is inconsistent or incompatible with how the user is thinking about the problem. When you find this mismatch, the fix is often straightforward — nomenclature, flow, or visual hierarchy — and fixing it can be a significant win.

## Common Mistakes

| Mistake | Why It Fails | Correction |
|---------|-------------|------------|
| Running usability test at the end of development | Issues found too late to fix without significant waste | Run on prototypes during discovery |
| Product manager or designer not attending | Secondhand summaries miss critical nuance; learning must be firsthand | Attend every session — non-negotiable |
| Asking for design opinions ("What would you change?") | Users cannot specify what to build; their opinions predict their own behavior poorly | Watch what they do, not what they say |
| Helping users when they struggle | Removes the friction signal | Stay silent; use the parrot technique |
| Writing long formal reports | Reports are obsolete before they are read | Write brief email summaries immediately after each session |
| Testing too few users | Missing major issues | Target minimum 5 users per round of testing |
| Using internal employees as test subjects | Employees know the product, think like the team, and are not target customers | Always use actual target customers |


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)